2024-03-23T03:33:42.769 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-23T03:33:42.769 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/quota} --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617203 --name yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi -- /home/teuthworker/archive/yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617203/orig.config.yaml 2024-03-23T03:33:42.813 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-23T03:33:42.905 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617203 branch: wip-yuri11-testing-2024-03-21-0851-reef description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/quota} email: yweinste@redhat.com first_in_suite: false job_id: '7617203' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri11-testing-2024-03-21-0851-reef ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 82316b04821a6c2c1d458a881920454bbadb8837 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 82316b04821a6c2c1d458a881920454bbadb8837 kclient: syntax: v1 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri11-testing-2024-03-21-0851-reef sha1: 82316b04821a6c2c1d458a881920454bbadb8837 owner: scheduled_yuriw@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 1919 sha1: 82316b04821a6c2c1d458a881920454bbadb8837 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri11-testing-2024-03-21-0851-reef suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_82316b04821a6c2c1d458a881920454bbadb8837/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 82316b04821a6c2c1d458a881920454bbadb8837 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi102.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMTizH0wUqZ4U/b89MXw+0NgcpY4R8KU/gvKM8U+SnddorpI5l/0MjELps6rL3KkXG2xlUc9NiXI5iVz1sRTpFs= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDYfGpry3ANlMxAACapxS822P0GSJelsL/67tVqmYt77USJcr2GPvIQKVoKwjDDWC8XLqyAmDUKqDa+hhPpnnWg= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_quota teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_13:15:24 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-23T03:33:42.905 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_82316b04821a6c2c1d458a881920454bbadb8837/qa; will attempt to use it 2024-03-23T03:33:42.906 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_82316b04821a6c2c1d458a881920454bbadb8837/qa/tasks 2024-03-23T03:33:42.906 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-23T03:33:42.908 INFO:teuthology.task.internal:Checking packages... 2024-03-23T03:33:42.932 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '82316b04821a6c2c1d458a881920454bbadb8837' 2024-03-23T03:33:42.932 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-23T03:33:42.932 INFO:teuthology.packaging:ref: None 2024-03-23T03:33:42.932 INFO:teuthology.packaging:tag: None 2024-03-23T03:33:42.932 INFO:teuthology.packaging:branch: wip-yuri11-testing-2024-03-21-0851-reef 2024-03-23T03:33:42.932 INFO:teuthology.packaging:sha1: 82316b04821a6c2c1d458a881920454bbadb8837 2024-03-23T03:33:42.932 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri11-testing-2024-03-21-0851-reef 2024-03-23T03:33:43.156 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-954.g82316b04 2024-03-23T03:33:43.157 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-23T03:33:43.164 INFO:teuthology.task.internal:no buildpackages task found 2024-03-23T03:33:43.164 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-23T03:33:43.170 INFO:teuthology.task.internal:Saving configuration 2024-03-23T03:33:43.187 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-23T03:33:43.193 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T03:33:43.219 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617203', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 03:28:43.692644', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMTizH0wUqZ4U/b89MXw+0NgcpY4R8KU/gvKM8U+SnddorpI5l/0MjELps6rL3KkXG2xlUc9NiXI5iVz1sRTpFs='} 2024-03-23T03:33:43.241 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617203', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 03:28:43.690835', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDYfGpry3ANlMxAACapxS822P0GSJelsL/67tVqmYt77USJcr2GPvIQKVoKwjDDWC8XLqyAmDUKqDa+hhPpnnWg='} 2024-03-23T03:33:43.242 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-23T03:33:43.248 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-23T03:33:43.248 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-23T03:33:43.248 INFO:teuthology.run_tasks:Running task console_log... 2024-03-23T03:33:43.319 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2b7b994160>, signals=[15]) 2024-03-23T03:33:43.320 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-23T03:33:43.326 INFO:teuthology.task.internal:Opening connections... 2024-03-23T03:33:43.326 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:33:43.328 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T03:33:43.414 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:33:43.415 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T03:33:43.500 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-23T03:33:43.507 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2024-03-23T03:33:43.524 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2024-03-23T03:33:43.524 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2024-03-23T03:33:43.580 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Stream" 2024-03-23T03:33:43.580 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="9" 2024-03-23T03:33:43.580 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos" 2024-03-23T03:33:43.580 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora" 2024-03-23T03:33:43.580 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="9" 2024-03-23T03:33:43.580 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el9" 2024-03-23T03:33:43.581 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-23T03:33:43.581 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2024-03-23T03:33:43.581 INFO:teuthology.orchestra.run.smithi102.stdout:LOGO="fedora-logo-icon" 2024-03-23T03:33:43.581 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-23T03:33:43.581 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/" 2024-03-23T03:33:43.581 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T03:33:43.581 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-23T03:33:43.581 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T03:33:43.581 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2024-03-23T03:33:43.603 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-03-23T03:33:43.621 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-03-23T03:33:43.621 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-03-23T03:33:43.677 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-03-23T03:33:43.677 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-03-23T03:33:43.677 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-03-23T03:33:43.677 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-03-23T03:33:43.677 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-03-23T03:33:43.677 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-03-23T03:33:43.677 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-23T03:33:43.677 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-03-23T03:33:43.677 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-03-23T03:33:43.678 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-23T03:33:43.678 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-03-23T03:33:43.678 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T03:33:43.678 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-23T03:33:43.678 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T03:33:43.678 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-03-23T03:33:43.695 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-23T03:33:43.704 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-23T03:33:43.710 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-23T03:33:43.710 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2024-03-23T03:33:43.712 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-03-23T03:33:43.731 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-23T03:33:43.737 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-23T03:33:43.738 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2024-03-23T03:33:43.768 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-03-23T03:33:43.870 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-23T03:33:43.927 INFO:teuthology.run_tasks:Running task kernel... 2024-03-23T03:33:43.942 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-23T03:33:43.942 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2024-03-23T03:33:43.942 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2024-03-23T03:33:43.943 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T03:33:43.943 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T03:33:43.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:33:43.959 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-23T03:33:43.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:33:43.963 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-23T03:33:44.198 DEBUG:teuthology.task.kernel:sha1 for client.0 is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T03:33:44.198 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-23T03:33:44.198 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-03-23T03:33:44.208 DEBUG:teuthology.task.kernel:sha1 for client.1 is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T03:33:44.208 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-23T03:33:44.208 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-03-23T03:33:44.215 INFO:teuthology.orchestra.run.smithi102.stdout:5.14.0-361.el9.x86_64 2024-03-23T03:33:44.215 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T03:33:44.215 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-23T03:33:44.216 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum upgrade -y linux-firmware 2024-03-23T03:33:44.225 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-361.el9.x86_64 2024-03-23T03:33:44.225 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T03:33:44.225 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-23T03:33:44.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum upgrade -y linux-firmware 2024-03-23T03:33:46.527 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-23T03:33:46.554 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-23T03:33:48.020 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-23T03:33:48.219 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - BaseOS 3.3 MB/s | 2.0 MB 00:00 2024-03-23T03:33:49.002 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-23T03:33:49.144 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-23T03:33:51.340 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-03-23T03:33:51.636 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-23T03:33:55.196 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-23T03:33:55.687 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-23T03:33:56.772 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-03-23T03:33:57.288 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-23T03:33:59.856 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 8.4 MB/s | 21 MB 00:02 2024-03-23T03:34:00.298 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 6.1 MB/s | 21 MB 00:03 2024-03-23T03:34:04.781 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-23T03:34:05.264 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-23T03:34:06.966 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-23T03:34:06.978 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-23T03:34:06.979 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-23T03:34:06.979 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-23T03:34:06.979 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2024-03-23T03:34:06.979 INFO:teuthology.orchestra.run.smithi102.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-23T03:34:06.979 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:34:06.979 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-23T03:34:06.979 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2024-03-23T03:34:06.979 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 1 Package 2024-03-23T03:34:06.979 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:34:06.979 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 375 M 2024-03-23T03:34:06.979 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-23T03:34:07.490 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-23T03:34:07.498 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-23T03:34:07.498 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-23T03:34:07.498 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-23T03:34:07.498 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-03-23T03:34:07.498 INFO:teuthology.orchestra.run.smithi173.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-23T03:34:07.498 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:34:07.498 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-23T03:34:07.498 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-23T03:34:07.499 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 1 Package 2024-03-23T03:34:07.499 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:34:07.499 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 375 M 2024-03-23T03:34:07.499 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-23T03:34:18.756 INFO:teuthology.orchestra.run.smithi173.stdout:linux-firmware-20240219-143.el9.noarch.rpm 33 MB/s | 375 MB 00:11 2024-03-23T03:34:18.757 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-23T03:34:18.757 INFO:teuthology.orchestra.run.smithi173.stdout:Total 33 MB/s | 375 MB 00:11 2024-03-23T03:34:18.757 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-23T03:34:18.841 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-23T03:34:18.841 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-23T03:34:21.232 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-23T03:34:21.232 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-23T03:34:22.906 INFO:teuthology.orchestra.run.smithi102.stdout:linux-firmware-20240219-143.el9.noarch.rpm 24 MB/s | 375 MB 00:15 2024-03-23T03:34:22.907 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-23T03:34:22.907 INFO:teuthology.orchestra.run.smithi102.stdout:Total 24 MB/s | 375 MB 00:15 2024-03-23T03:34:22.908 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-23T03:34:23.001 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-23T03:34:23.001 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-23T03:34:25.514 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-23T03:34:25.514 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-23T03:34:27.693 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-23T03:34:27.990 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-23T03:34:28.064 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T03:34:29.332 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T03:34:29.332 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-23T03:34:29.724 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T03:34:29.724 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:34:29.724 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-03-23T03:34:29.724 INFO:teuthology.orchestra.run.smithi173.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-23T03:34:29.724 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:34:29.724 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-23T03:34:30.035 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on client.1... 2024-03-23T03:34:30.035 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T03:34:30.186 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ 2024-03-23T03:34:30.187 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-23T03:34:30.189 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi173.front.sepia.ceph.com to complete... 2024-03-23T03:34:32.002 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-23T03:34:32.260 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-23T03:34:32.534 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T03:34:33.569 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T03:34:33.569 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-23T03:34:33.997 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T03:34:33.998 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:34:33.998 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2024-03-23T03:34:33.998 INFO:teuthology.orchestra.run.smithi102.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-23T03:34:33.998 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:34:33.998 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-23T03:34:34.286 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on client.0... 2024-03-23T03:34:34.286 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T03:34:34.438 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ 2024-03-23T03:34:34.438 DEBUG:teuthology.orchestra.run.smithi102:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-23T03:34:34.440 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi102.front.sepia.ceph.com to complete... 2024-03-23T03:34:46.744 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23 03:34:46 URL:https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm [336309687/336309687] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-23T03:34:46.745 INFO:teuthology.orchestra.run.smithi173.stderr:FINISHED --2024-03-23 03:34:46-- 2024-03-23T03:34:46.745 INFO:teuthology.orchestra.run.smithi173.stderr:Total wall clock time: 17s 2024-03-23T03:34:46.745 INFO:teuthology.orchestra.run.smithi173.stderr:Downloaded: 1 files, 321M in 16s (19.7 MB/s) 2024-03-23T03:34:46.747 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on client.1... 2024-03-23T03:34:46.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-23T03:34:48.687 INFO:teuthology.orchestra.run.smithi173.stdout:Verifying... ######################################## 2024-03-23T03:34:48.810 INFO:teuthology.orchestra.run.smithi173.stdout:Preparing... ######################################## 2024-03-23T03:34:48.889 INFO:teuthology.orchestra.run.smithi173.stdout:Updating / installing... 2024-03-23T03:34:56.377 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-6.8.0_g5ec3e6e88b2e-1 ######################################## 2024-03-23T03:35:12.743 INFO:teuthology.orchestra.run.smithi173.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-23T03:35:16.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23 03:35:16 URL:https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm [336309687/336309687] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-23T03:35:16.811 INFO:teuthology.orchestra.run.smithi102.stderr:FINISHED --2024-03-23 03:35:16-- 2024-03-23T03:35:16.811 INFO:teuthology.orchestra.run.smithi102.stderr:Total wall clock time: 42s 2024-03-23T03:35:16.812 INFO:teuthology.orchestra.run.smithi102.stderr:Downloaded: 1 files, 321M in 42s (7.68 MB/s) 2024-03-23T03:35:16.814 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on client.0... 2024-03-23T03:35:16.814 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-23T03:35:18.751 INFO:teuthology.orchestra.run.smithi102.stdout:Verifying... ######################################## 2024-03-23T03:35:18.829 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing... ######################################## 2024-03-23T03:35:18.909 INFO:teuthology.orchestra.run.smithi102.stdout:Updating / installing... 2024-03-23T03:35:26.407 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-6.8.0_g5ec3e6e88b2e-1 ######################################## 2024-03-23T03:35:35.225 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-23T03:35:35.225 DEBUG:teuthology.orchestra.run.smithi173:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-23T03:35:35.304 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e 2024-03-23T03:35:35.305 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/System.map 2024-03-23T03:35:35.305 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/config 2024-03-23T03:35:35.305 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel 2024-03-23T03:35:35.305 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch 2024-03-23T03:35:35.305 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86 2024-03-23T03:35:35.305 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto 2024-03-23T03:35:35.305 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-23T03:35:35.305 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-23T03:35:35.305 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-23T03:35:35.305 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-23T03:35:35.305 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-23T03:35:35.306 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-23T03:35:35.306 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-23T03:35:35.306 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-23T03:35:35.306 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-23T03:35:35.306 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-23T03:35:35.306 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-23T03:35:35.306 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-23T03:35:35.306 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-23T03:35:35.306 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-23T03:35:35.306 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-23T03:35:35.307 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-23T03:35:35.307 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-23T03:35:35.307 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-23T03:35:35.307 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-23T03:35:35.307 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-23T03:35:35.307 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel 2024-03-23T03:35:35.307 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu 2024-03-23T03:35:35.307 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce 2024-03-23T03:35:35.307 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-23T03:35:35.307 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm 2024-03-23T03:35:35.307 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-23T03:35:35.307 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-23T03:35:35.307 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-23T03:35:35.307 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto 2024-03-23T03:35:35.307 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-23T03:35:35.308 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-23T03:35:35.308 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-23T03:35:35.308 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx 2024-03-23T03:35:35.308 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-23T03:35:35.308 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-23T03:35:35.308 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-23T03:35:35.308 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-23T03:35:35.308 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-23T03:35:35.308 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-23T03:35:35.308 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-23T03:35:35.308 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-23T03:35:35.308 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-23T03:35:35.308 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-23T03:35:35.308 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-23T03:35:35.309 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-23T03:35:35.309 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-23T03:35:35.309 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-23T03:35:35.309 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-23T03:35:35.309 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-23T03:35:35.309 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-23T03:35:35.309 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-23T03:35:35.309 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-23T03:35:35.309 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-23T03:35:35.309 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-23T03:35:35.309 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-23T03:35:35.309 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-23T03:35:35.309 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-23T03:35:35.309 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-23T03:35:35.310 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-23T03:35:35.310 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-23T03:35:35.310 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-23T03:35:35.310 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-23T03:35:35.310 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-23T03:35:35.310 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-23T03:35:35.310 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-23T03:35:35.310 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-23T03:35:35.310 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-23T03:35:35.310 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-23T03:35:35.310 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-23T03:35:35.310 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-23T03:35:35.311 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-23T03:35:35.311 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-23T03:35:35.311 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-23T03:35:35.311 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-23T03:35:35.311 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-23T03:35:35.311 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-23T03:35:35.311 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-23T03:35:35.311 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-23T03:35:35.311 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers 2024-03-23T03:35:35.311 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi 2024-03-23T03:35:35.311 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-23T03:35:35.311 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-23T03:35:35.311 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/custom_method.ko 2024-03-23T03:35:35.312 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-23T03:35:35.312 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-23T03:35:35.312 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-23T03:35:35.312 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-23T03:35:35.312 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-23T03:35:35.312 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata 2024-03-23T03:35:35.312 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-23T03:35:35.312 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-23T03:35:35.312 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ata_generic.ko 2024-03-23T03:35:35.312 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-23T03:35:35.312 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-23T03:35:35.312 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-23T03:35:35.312 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-23T03:35:35.313 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-23T03:35:35.313 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-23T03:35:35.313 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-23T03:35:35.313 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-23T03:35:35.313 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-23T03:35:35.313 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-23T03:35:35.313 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-23T03:35:35.313 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-23T03:35:35.313 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-23T03:35:35.313 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-23T03:35:35.313 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-23T03:35:35.313 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-23T03:35:35.313 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-23T03:35:35.313 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-23T03:35:35.313 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-23T03:35:35.313 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-23T03:35:35.314 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-23T03:35:35.314 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-23T03:35:35.314 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-23T03:35:35.314 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-23T03:35:35.314 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-23T03:35:35.314 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-23T03:35:35.314 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-23T03:35:35.314 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-23T03:35:35.314 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-23T03:35:35.314 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-23T03:35:35.314 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-23T03:35:35.314 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-23T03:35:35.314 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-23T03:35:35.314 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-23T03:35:35.315 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-23T03:35:35.315 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sis.ko 2024-03-23T03:35:35.315 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-23T03:35:35.315 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-23T03:35:35.315 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-23T03:35:35.315 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-23T03:35:35.315 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-23T03:35:35.315 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-23T03:35:35.315 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-23T03:35:35.315 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-23T03:35:35.315 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-23T03:35:35.315 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-23T03:35:35.316 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-23T03:35:35.316 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-23T03:35:35.316 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-23T03:35:35.316 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-23T03:35:35.316 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-23T03:35:35.316 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-23T03:35:35.316 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-23T03:35:35.316 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay 2024-03-23T03:35:35.316 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-23T03:35:35.316 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base 2024-03-23T03:35:35.316 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap 2024-03-23T03:35:35.316 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-23T03:35:35.316 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-23T03:35:35.316 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma 2024-03-23T03:35:35.316 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-23T03:35:35.316 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block 2024-03-23T03:35:35.317 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe 2024-03-23T03:35:35.317 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-23T03:35:35.317 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/brd.ko 2024-03-23T03:35:35.317 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd 2024-03-23T03:35:35.317 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-23T03:35:35.317 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-23T03:35:35.317 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/loop.ko 2024-03-23T03:35:35.317 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx 2024-03-23T03:35:35.317 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-23T03:35:35.317 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-23T03:35:35.317 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk 2024-03-23T03:35:35.317 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-23T03:35:35.317 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-23T03:35:35.317 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-23T03:35:35.317 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/virtio_blk.ko 2024-03-23T03:35:35.318 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback 2024-03-23T03:35:35.318 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-23T03:35:35.318 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkfront.ko 2024-03-23T03:35:35.318 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram 2024-03-23T03:35:35.318 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-23T03:35:35.318 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char 2024-03-23T03:35:35.319 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-23T03:35:35.319 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random 2024-03-23T03:35:35.320 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-23T03:35:35.320 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-23T03:35:35.320 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-23T03:35:35.320 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-23T03:35:35.320 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-23T03:35:35.320 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi 2024-03-23T03:35:35.320 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-23T03:35:35.320 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-23T03:35:35.320 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-23T03:35:35.320 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-23T03:35:35.320 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-23T03:35:35.320 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-23T03:35:35.320 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-23T03:35:35.320 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave 2024-03-23T03:35:35.320 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-23T03:35:35.321 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-23T03:35:35.321 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-23T03:35:35.321 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm 2024-03-23T03:35:35.321 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm.ko 2024-03-23T03:35:35.321 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-23T03:35:35.321 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-23T03:35:35.321 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-23T03:35:35.321 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-23T03:35:35.321 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-23T03:35:35.321 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/uv_mmtimer.ko 2024-03-23T03:35:35.321 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/virtio_console.ko 2024-03-23T03:35:35.321 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq 2024-03-23T03:35:35.321 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-23T03:35:35.321 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-23T03:35:35.322 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-23T03:35:35.322 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-23T03:35:35.322 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-23T03:35:35.322 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-23T03:35:35.322 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto 2024-03-23T03:35:35.322 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp 2024-03-23T03:35:35.322 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-23T03:35:35.322 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-23T03:35:35.322 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel 2024-03-23T03:35:35.322 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat 2024-03-23T03:35:35.322 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common 2024-03-23T03:35:35.322 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-23T03:35:35.322 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-23T03:35:35.322 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-23T03:35:35.323 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-23T03:35:35.323 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-23T03:35:35.323 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio 2024-03-23T03:35:35.323 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-23T03:35:35.323 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca 2024-03-23T03:35:35.323 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-23T03:35:35.323 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq 2024-03-23T03:35:35.323 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-23T03:35:35.323 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma 2024-03-23T03:35:35.323 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw 2024-03-23T03:35:35.323 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-23T03:35:35.323 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat 2024-03-23T03:35:35.324 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-23T03:35:35.324 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac 2024-03-23T03:35:35.324 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-23T03:35:35.324 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-23T03:35:35.324 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-23T03:35:35.324 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-23T03:35:35.324 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-23T03:35:35.324 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-23T03:35:35.324 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-23T03:35:35.324 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-23T03:35:35.324 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-23T03:35:35.324 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-23T03:35:35.324 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-23T03:35:35.325 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-23T03:35:35.325 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-23T03:35:35.325 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware 2024-03-23T03:35:35.325 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/edd.ko 2024-03-23T03:35:35.325 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-23T03:35:35.325 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio 2024-03-23T03:35:35.325 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-23T03:35:35.325 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-23T03:35:35.325 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid 2024-03-23T03:35:35.325 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-23T03:35:35.325 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-23T03:35:35.325 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-23T03:35:35.325 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-23T03:35:35.325 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-23T03:35:35.326 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-23T03:35:35.326 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-23T03:35:35.326 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-23T03:35:35.326 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-23T03:35:35.326 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-23T03:35:35.326 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-23T03:35:35.326 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-23T03:35:35.326 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-23T03:35:35.326 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-23T03:35:35.326 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-23T03:35:35.326 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-23T03:35:35.326 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-23T03:35:35.327 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-23T03:35:35.327 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-23T03:35:35.327 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-23T03:35:35.327 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-23T03:35:35.327 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-23T03:35:35.327 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-23T03:35:35.327 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-23T03:35:35.327 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-23T03:35:35.327 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-23T03:35:35.327 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-23T03:35:35.327 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-23T03:35:35.327 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-23T03:35:35.328 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-23T03:35:35.328 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-23T03:35:35.328 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-23T03:35:35.328 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-23T03:35:35.328 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-23T03:35:35.328 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-23T03:35:35.328 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-23T03:35:35.328 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-23T03:35:35.328 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-23T03:35:35.328 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-23T03:35:35.328 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-23T03:35:35.328 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-23T03:35:35.329 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-23T03:35:35.329 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-23T03:35:35.329 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-23T03:35:35.329 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-23T03:35:35.329 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-23T03:35:35.329 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-23T03:35:35.329 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-23T03:35:35.329 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-23T03:35:35.329 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-23T03:35:35.329 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-23T03:35:35.329 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-23T03:35:35.329 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-23T03:35:35.330 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-23T03:35:35.330 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-23T03:35:35.330 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-23T03:35:35.330 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-23T03:35:35.330 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-23T03:35:35.330 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-23T03:35:35.330 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-23T03:35:35.330 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-23T03:35:35.330 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-23T03:35:35.330 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-23T03:35:35.330 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-23T03:35:35.330 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv 2024-03-23T03:35:35.330 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-23T03:35:35.331 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-23T03:35:35.331 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-23T03:35:35.331 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon 2024-03-23T03:35:35.331 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-23T03:35:35.331 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-23T03:35:35.331 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-23T03:35:35.331 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-23T03:35:35.331 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-23T03:35:35.331 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-23T03:35:35.331 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-23T03:35:35.331 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-23T03:35:35.331 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-23T03:35:35.331 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-23T03:35:35.331 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-23T03:35:35.331 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-23T03:35:35.332 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-23T03:35:35.332 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-23T03:35:35.332 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-23T03:35:35.332 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-23T03:35:35.332 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-23T03:35:35.332 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-23T03:35:35.332 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-23T03:35:35.332 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-23T03:35:35.332 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-23T03:35:35.332 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-23T03:35:35.332 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-23T03:35:35.332 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-23T03:35:35.332 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-23T03:35:35.333 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-23T03:35:35.333 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-23T03:35:35.333 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-23T03:35:35.333 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-23T03:35:35.333 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-23T03:35:35.333 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-23T03:35:35.333 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-23T03:35:35.333 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-23T03:35:35.333 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-23T03:35:35.333 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-23T03:35:35.333 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-23T03:35:35.333 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-23T03:35:35.333 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-23T03:35:35.333 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-23T03:35:35.334 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-23T03:35:35.334 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-23T03:35:35.334 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-23T03:35:35.334 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-23T03:35:35.334 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-23T03:35:35.334 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-23T03:35:35.334 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-23T03:35:35.334 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-23T03:35:35.334 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-23T03:35:35.334 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-23T03:35:35.334 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-23T03:35:35.334 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-23T03:35:35.334 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-23T03:35:35.334 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-23T03:35:35.335 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-23T03:35:35.335 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-23T03:35:35.335 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-23T03:35:35.335 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-23T03:35:35.335 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-23T03:35:35.335 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-23T03:35:35.335 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-23T03:35:35.335 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-23T03:35:35.335 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-23T03:35:35.335 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-23T03:35:35.335 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-23T03:35:35.335 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-23T03:35:35.335 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-23T03:35:35.335 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-23T03:35:35.335 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-23T03:35:35.335 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-23T03:35:35.336 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-23T03:35:35.336 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-23T03:35:35.336 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-23T03:35:35.336 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-23T03:35:35.336 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-23T03:35:35.336 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-23T03:35:35.336 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-23T03:35:35.336 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-23T03:35:35.336 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-23T03:35:35.336 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-23T03:35:35.336 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-23T03:35:35.336 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-23T03:35:35.336 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-23T03:35:35.336 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-23T03:35:35.336 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-23T03:35:35.336 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-23T03:35:35.337 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-23T03:35:35.337 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-23T03:35:35.337 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-23T03:35:35.337 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus 2024-03-23T03:35:35.337 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-23T03:35:35.337 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-23T03:35:35.337 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-23T03:35:35.337 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-23T03:35:35.337 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-23T03:35:35.337 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-23T03:35:35.337 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-23T03:35:35.337 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-23T03:35:35.337 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-23T03:35:35.337 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-23T03:35:35.337 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-23T03:35:35.337 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-23T03:35:35.338 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-23T03:35:35.338 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-23T03:35:35.338 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-23T03:35:35.338 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-23T03:35:35.338 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-23T03:35:35.338 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-23T03:35:35.338 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-23T03:35:35.338 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-23T03:35:35.338 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-23T03:35:35.338 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-23T03:35:35.338 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-23T03:35:35.338 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-23T03:35:35.339 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-23T03:35:35.339 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-23T03:35:35.339 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-23T03:35:35.339 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-23T03:35:35.339 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-23T03:35:35.339 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-23T03:35:35.339 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-23T03:35:35.339 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-23T03:35:35.339 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-23T03:35:35.339 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-23T03:35:35.339 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-23T03:35:35.339 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-23T03:35:35.339 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-23T03:35:35.339 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-23T03:35:35.339 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-23T03:35:35.339 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-23T03:35:35.339 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-23T03:35:35.340 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c 2024-03-23T03:35:35.340 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos 2024-03-23T03:35:35.340 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-23T03:35:35.340 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-23T03:35:35.340 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses 2024-03-23T03:35:35.340 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-23T03:35:35.340 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-23T03:35:35.340 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-23T03:35:35.340 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-23T03:35:35.340 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-23T03:35:35.340 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-23T03:35:35.340 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-23T03:35:35.340 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-23T03:35:35.340 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-23T03:35:35.340 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-23T03:35:35.340 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-23T03:35:35.341 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-23T03:35:35.341 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-23T03:35:35.341 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-23T03:35:35.341 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-23T03:35:35.341 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-23T03:35:35.341 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-23T03:35:35.341 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-23T03:35:35.341 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-23T03:35:35.341 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-23T03:35:35.341 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-23T03:35:35.341 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-23T03:35:35.341 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-23T03:35:35.341 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-dev.ko 2024-03-23T03:35:35.341 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-23T03:35:35.341 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-23T03:35:35.341 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-23T03:35:35.342 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-23T03:35:35.342 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband 2024-03-23T03:35:35.342 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core 2024-03-23T03:35:35.342 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-23T03:35:35.342 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-23T03:35:35.342 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-23T03:35:35.342 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-23T03:35:35.342 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-23T03:35:35.342 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-23T03:35:35.342 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-23T03:35:35.342 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw 2024-03-23T03:35:35.342 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4 2024-03-23T03:35:35.342 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-23T03:35:35.343 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4 2024-03-23T03:35:35.343 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-23T03:35:35.343 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5 2024-03-23T03:35:35.343 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-23T03:35:35.343 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca 2024-03-23T03:35:35.343 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-23T03:35:35.343 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp 2024-03-23T03:35:35.343 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib 2024-03-23T03:35:35.343 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-23T03:35:35.343 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser 2024-03-23T03:35:35.343 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-23T03:35:35.343 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert 2024-03-23T03:35:35.343 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-23T03:35:35.343 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp 2024-03-23T03:35:35.343 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-23T03:35:35.344 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt 2024-03-23T03:35:35.344 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-23T03:35:35.344 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input 2024-03-23T03:35:35.344 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport 2024-03-23T03:35:35.344 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-23T03:35:35.344 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-23T03:35:35.344 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-23T03:35:35.344 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-23T03:35:35.344 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard 2024-03-23T03:35:35.344 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-23T03:35:35.344 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse 2024-03-23T03:35:35.344 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-23T03:35:35.344 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-23T03:35:35.344 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-23T03:35:35.345 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-23T03:35:35.345 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-23T03:35:35.345 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-23T03:35:35.345 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-23T03:35:35.345 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-23T03:35:35.345 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4 2024-03-23T03:35:35.345 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-23T03:35:35.345 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio 2024-03-23T03:35:35.345 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-23T03:35:35.345 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-23T03:35:35.345 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-23T03:35:35.345 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-23T03:35:35.345 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-23T03:35:35.345 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds 2024-03-23T03:35:35.345 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-23T03:35:35.346 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-23T03:35:35.346 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-23T03:35:35.346 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-23T03:35:35.346 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger 2024-03-23T03:35:35.346 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-23T03:35:35.346 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-23T03:35:35.346 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-23T03:35:35.346 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-23T03:35:35.346 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-23T03:35:35.346 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-23T03:35:35.346 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-23T03:35:35.346 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-23T03:35:35.346 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-23T03:35:35.346 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md 2024-03-23T03:35:35.347 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache 2024-03-23T03:35:35.347 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-23T03:35:35.347 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-23T03:35:35.347 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-23T03:35:35.347 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-23T03:35:35.347 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-23T03:35:35.347 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-23T03:35:35.347 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-23T03:35:35.347 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-23T03:35:35.347 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-23T03:35:35.347 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-23T03:35:35.347 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-23T03:35:35.347 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-23T03:35:35.347 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-23T03:35:35.347 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-23T03:35:35.348 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-23T03:35:35.348 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-23T03:35:35.348 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data 2024-03-23T03:35:35.348 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-23T03:35:35.348 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-23T03:35:35.348 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-23T03:35:35.348 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-23T03:35:35.348 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-23T03:35:35.348 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message 2024-03-23T03:35:35.348 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion 2024-03-23T03:35:35.348 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-23T03:35:35.348 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-23T03:35:35.348 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-23T03:35:35.348 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-23T03:35:35.348 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-23T03:35:35.349 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-23T03:35:35.349 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-23T03:35:35.349 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd 2024-03-23T03:35:35.349 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-23T03:35:35.349 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-23T03:35:35.349 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mfd-core.ko 2024-03-23T03:35:35.349 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-23T03:35:35.349 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-23T03:35:35.349 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-23T03:35:35.349 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-23T03:35:35.349 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc 2024-03-23T03:35:35.349 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl 2024-03-23T03:35:35.349 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-23T03:35:35.349 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-23T03:35:35.349 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-23T03:35:35.350 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-23T03:35:35.350 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710 2024-03-23T03:35:35.350 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-23T03:35:35.350 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom 2024-03-23T03:35:35.350 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-23T03:35:35.350 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-23T03:35:35.350 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-23T03:35:35.350 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-23T03:35:35.350 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-23T03:35:35.350 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-23T03:35:35.350 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-23T03:35:35.350 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d 2024-03-23T03:35:35.350 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-23T03:35:35.350 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-23T03:35:35.351 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei 2024-03-23T03:35:35.351 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-23T03:35:35.351 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-23T03:35:35.351 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-23T03:35:35.351 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru 2024-03-23T03:35:35.351 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-23T03:35:35.351 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp 2024-03-23T03:35:35.351 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-23T03:35:35.351 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-23T03:35:35.351 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-23T03:35:35.351 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-23T03:35:35.351 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-23T03:35:35.351 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-23T03:35:35.351 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-23T03:35:35.351 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci 2024-03-23T03:35:35.352 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-23T03:35:35.352 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net 2024-03-23T03:35:35.352 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding 2024-03-23T03:35:35.352 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-23T03:35:35.352 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa 2024-03-23T03:35:35.352 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53 2024-03-23T03:35:35.352 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-23T03:35:35.352 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-23T03:35:35.352 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-23T03:35:35.352 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx 2024-03-23T03:35:35.352 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-23T03:35:35.352 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-23T03:35:35.352 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-23T03:35:35.352 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet 2024-03-23T03:35:35.353 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com 2024-03-23T03:35:35.353 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-23T03:35:35.353 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-23T03:35:35.353 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-23T03:35:35.353 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-23T03:35:35.353 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390 2024-03-23T03:35:35.353 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-23T03:35:35.353 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-23T03:35:35.353 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-23T03:35:35.353 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-23T03:35:35.353 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec 2024-03-23T03:35:35.353 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-23T03:35:35.353 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere 2024-03-23T03:35:35.353 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-23T03:35:35.353 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon 2024-03-23T03:35:35.354 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-23T03:35:35.354 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera 2024-03-23T03:35:35.354 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-23T03:35:35.354 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd 2024-03-23T03:35:35.354 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-23T03:35:35.354 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-23T03:35:35.354 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-23T03:35:35.354 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros 2024-03-23T03:35:35.354 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx 2024-03-23T03:35:35.354 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-23T03:35:35.354 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-23T03:35:35.354 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-23T03:35:35.355 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-23T03:35:35.355 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-23T03:35:35.355 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx 2024-03-23T03:35:35.355 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-23T03:35:35.355 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-23T03:35:35.355 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom 2024-03-23T03:35:35.355 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-23T03:35:35.355 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-23T03:35:35.355 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-23T03:35:35.355 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-23T03:35:35.355 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-23T03:35:35.355 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet 2024-03-23T03:35:35.355 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-23T03:35:35.355 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-23T03:35:35.355 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade 2024-03-23T03:35:35.356 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna 2024-03-23T03:35:35.356 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-23T03:35:35.356 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio 2024-03-23T03:35:35.356 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-23T03:35:35.356 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-23T03:35:35.356 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-23T03:35:35.356 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-23T03:35:35.356 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-23T03:35:35.356 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-23T03:35:35.356 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-23T03:35:35.356 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-23T03:35:35.356 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-23T03:35:35.356 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-23T03:35:35.356 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco 2024-03-23T03:35:35.356 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic 2024-03-23T03:35:35.356 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-23T03:35:35.357 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec 2024-03-23T03:35:35.357 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip 2024-03-23T03:35:35.357 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-23T03:35:35.357 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-23T03:35:35.357 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-23T03:35:35.357 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-23T03:35:35.357 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-23T03:35:35.357 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-23T03:35:35.357 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink 2024-03-23T03:35:35.357 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-23T03:35:35.357 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-23T03:35:35.357 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-23T03:35:35.357 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex 2024-03-23T03:35:35.357 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet 2024-03-23T03:35:35.357 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-23T03:35:35.357 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-23T03:35:35.357 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-23T03:35:35.358 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel 2024-03-23T03:35:35.358 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-23T03:35:35.358 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000 2024-03-23T03:35:35.358 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-23T03:35:35.358 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e 2024-03-23T03:35:35.358 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-23T03:35:35.358 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k 2024-03-23T03:35:35.358 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-23T03:35:35.358 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e 2024-03-23T03:35:35.358 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-23T03:35:35.358 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf 2024-03-23T03:35:35.358 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-23T03:35:35.358 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb 2024-03-23T03:35:35.359 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-23T03:35:35.359 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf 2024-03-23T03:35:35.359 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-23T03:35:35.359 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-23T03:35:35.359 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-23T03:35:35.359 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-23T03:35:35.359 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-23T03:35:35.359 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-23T03:35:35.359 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell 2024-03-23T03:35:35.359 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-23T03:35:35.359 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-23T03:35:35.359 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-23T03:35:35.360 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox 2024-03-23T03:35:35.360 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-23T03:35:35.360 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-23T03:35:35.360 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-23T03:35:35.360 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-23T03:35:35.360 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-23T03:35:35.360 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-23T03:35:35.360 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel 2024-03-23T03:35:35.360 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-23T03:35:35.360 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom 2024-03-23T03:35:35.360 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-23T03:35:35.360 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-23T03:35:35.360 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi 2024-03-23T03:35:35.360 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-23T03:35:35.361 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-23T03:35:35.361 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion 2024-03-23T03:35:35.361 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-23T03:35:35.361 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia 2024-03-23T03:35:35.361 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-23T03:35:35.361 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines 2024-03-23T03:35:35.361 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-23T03:35:35.361 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-23T03:35:35.361 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic 2024-03-23T03:35:35.361 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-23T03:35:35.361 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-23T03:35:35.361 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-23T03:35:35.361 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-23T03:35:35.362 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-23T03:35:35.362 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc 2024-03-23T03:35:35.362 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-23T03:35:35.362 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek 2024-03-23T03:35:35.362 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-23T03:35:35.362 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-23T03:35:35.362 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-23T03:35:35.362 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-23T03:35:35.362 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker 2024-03-23T03:35:35.362 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-23T03:35:35.362 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc 2024-03-23T03:35:35.362 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-23T03:35:35.362 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan 2024-03-23T03:35:35.362 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-23T03:35:35.363 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis 2024-03-23T03:35:35.363 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-23T03:35:35.363 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-23T03:35:35.363 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc 2024-03-23T03:35:35.363 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-23T03:35:35.363 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-23T03:35:35.363 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-23T03:35:35.363 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-23T03:35:35.363 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro 2024-03-23T03:35:35.363 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-23T03:35:35.363 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-23T03:35:35.363 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-23T03:35:35.363 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun 2024-03-23T03:35:35.364 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-23T03:35:35.364 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-23T03:35:35.364 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-23T03:35:35.364 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-23T03:35:35.364 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti 2024-03-23T03:35:35.364 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-23T03:35:35.364 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti 2024-03-23T03:35:35.364 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-23T03:35:35.364 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via 2024-03-23T03:35:35.364 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-23T03:35:35.364 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-23T03:35:35.364 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet 2024-03-23T03:35:35.364 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-23T03:35:35.365 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-23T03:35:35.365 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom 2024-03-23T03:35:35.365 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-23T03:35:35.365 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-23T03:35:35.365 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv 2024-03-23T03:35:35.365 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-23T03:35:35.365 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154 2024-03-23T03:35:35.365 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-23T03:35:35.365 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-23T03:35:35.365 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan 2024-03-23T03:35:35.365 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-23T03:35:35.365 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-23T03:35:35.365 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-23T03:35:35.365 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio 2024-03-23T03:35:35.366 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-23T03:35:35.366 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-23T03:35:35.366 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-23T03:35:35.366 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-23T03:35:35.366 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-23T03:35:35.366 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/net_failover.ko 2024-03-23T03:35:35.366 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-23T03:35:35.366 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-23T03:35:35.366 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs 2024-03-23T03:35:35.366 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-23T03:35:35.366 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-23T03:35:35.366 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy 2024-03-23T03:35:35.367 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-23T03:35:35.367 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-23T03:35:35.367 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-23T03:35:35.367 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-23T03:35:35.367 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-23T03:35:35.367 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-23T03:35:35.367 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-23T03:35:35.367 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-23T03:35:35.367 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/dp83640.ko 2024-03-23T03:35:35.367 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-23T03:35:35.367 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-23T03:35:35.367 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-23T03:35:35.367 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-23T03:35:35.368 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-23T03:35:35.368 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-23T03:35:35.368 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-23T03:35:35.368 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-23T03:35:35.368 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-23T03:35:35.368 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-23T03:35:35.368 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-23T03:35:35.368 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-23T03:35:35.368 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp 2024-03-23T03:35:35.368 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-23T03:35:35.368 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-23T03:35:35.368 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-23T03:35:35.368 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-23T03:35:35.369 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-23T03:35:35.369 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-23T03:35:35.369 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-23T03:35:35.369 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-23T03:35:35.369 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-23T03:35:35.369 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip 2024-03-23T03:35:35.369 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slhc.ko 2024-03-23T03:35:35.369 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-23T03:35:35.369 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-23T03:35:35.369 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-23T03:35:35.369 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team 2024-03-23T03:35:35.369 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-23T03:35:35.369 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-23T03:35:35.370 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-23T03:35:35.370 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-23T03:35:35.370 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-23T03:35:35.370 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-23T03:35:35.370 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tun.ko 2024-03-23T03:35:35.370 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-23T03:35:35.370 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/virtio_net.ko 2024-03-23T03:35:35.370 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3 2024-03-23T03:35:35.370 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-23T03:35:35.370 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan 2024-03-23T03:35:35.370 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-23T03:35:35.370 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback 2024-03-23T03:35:35.370 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-23T03:35:35.371 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netfront.ko 2024-03-23T03:35:35.371 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb 2024-03-23T03:35:35.371 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-23T03:35:35.371 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme 2024-03-23T03:35:35.371 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host 2024-03-23T03:35:35.371 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-23T03:35:35.371 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-23T03:35:35.371 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport 2024-03-23T03:35:35.371 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-23T03:35:35.371 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-23T03:35:35.371 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-23T03:35:35.371 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-23T03:35:35.371 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci 2024-03-23T03:35:35.371 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug 2024-03-23T03:35:35.372 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-23T03:35:35.372 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie 2024-03-23T03:35:35.372 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-23T03:35:35.372 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-23T03:35:35.372 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia 2024-03-23T03:35:35.372 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-23T03:35:35.372 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-23T03:35:35.372 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-23T03:35:35.372 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl 2024-03-23T03:35:35.372 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel 2024-03-23T03:35:35.372 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-23T03:35:35.372 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform 2024-03-23T03:35:35.372 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86 2024-03-23T03:35:35.372 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-23T03:35:35.373 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-23T03:35:35.373 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-23T03:35:35.373 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-23T03:35:35.373 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-23T03:35:35.373 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-23T03:35:35.373 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-23T03:35:35.373 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-23T03:35:35.373 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-23T03:35:35.373 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-23T03:35:35.373 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-23T03:35:35.373 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-23T03:35:35.373 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel 2024-03-23T03:35:35.373 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-23T03:35:35.373 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-23T03:35:35.374 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-23T03:35:35.374 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-23T03:35:35.374 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-23T03:35:35.374 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-23T03:35:35.374 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-23T03:35:35.374 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-23T03:35:35.374 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-23T03:35:35.374 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-23T03:35:35.374 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-23T03:35:35.374 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-23T03:35:35.374 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap 2024-03-23T03:35:35.374 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-23T03:35:35.374 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-23T03:35:35.375 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps 2024-03-23T03:35:35.375 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients 2024-03-23T03:35:35.375 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-23T03:35:35.375 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-23T03:35:35.375 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-23T03:35:35.375 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-23T03:35:35.375 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp 2024-03-23T03:35:35.375 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-23T03:35:35.375 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-23T03:35:35.375 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc 2024-03-23T03:35:35.375 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-23T03:35:35.375 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-23T03:35:35.375 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-23T03:35:35.375 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-23T03:35:35.375 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-23T03:35:35.376 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-23T03:35:35.376 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-23T03:35:35.376 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-23T03:35:35.376 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-23T03:35:35.376 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-23T03:35:35.376 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-23T03:35:35.376 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-23T03:35:35.376 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-23T03:35:35.376 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-23T03:35:35.376 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-23T03:35:35.376 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-23T03:35:35.376 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-23T03:35:35.376 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-23T03:35:35.376 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-23T03:35:35.376 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-23T03:35:35.377 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-23T03:35:35.377 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-23T03:35:35.377 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-23T03:35:35.377 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-23T03:35:35.377 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-23T03:35:35.377 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-23T03:35:35.377 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-23T03:35:35.377 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-23T03:35:35.377 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-23T03:35:35.377 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-23T03:35:35.377 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-23T03:35:35.377 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi 2024-03-23T03:35:35.377 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-23T03:35:35.377 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-23T03:35:35.378 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-23T03:35:35.378 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-23T03:35:35.378 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-23T03:35:35.378 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid 2024-03-23T03:35:35.378 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-23T03:35:35.378 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-23T03:35:35.378 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx 2024-03-23T03:35:35.378 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-23T03:35:35.378 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-23T03:35:35.378 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-23T03:35:35.378 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr 2024-03-23T03:35:35.378 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-23T03:35:35.378 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-23T03:35:35.378 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi 2024-03-23T03:35:35.378 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-23T03:35:35.379 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa 2024-03-23T03:35:35.379 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-23T03:35:35.379 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc 2024-03-23T03:35:35.379 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-23T03:35:35.379 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i 2024-03-23T03:35:35.379 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-23T03:35:35.379 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-23T03:35:35.379 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor 2024-03-23T03:35:35.379 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-23T03:35:35.379 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi 2024-03-23T03:35:35.379 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-23T03:35:35.379 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-23T03:35:35.379 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-23T03:35:35.379 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-23T03:35:35.379 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-23T03:35:35.380 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-23T03:35:35.380 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler 2024-03-23T03:35:35.380 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-23T03:35:35.380 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-23T03:35:35.380 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-23T03:35:35.380 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-23T03:35:35.380 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-23T03:35:35.380 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r 2024-03-23T03:35:35.380 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-23T03:35:35.380 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-23T03:35:35.380 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe 2024-03-23T03:35:35.380 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-23T03:35:35.380 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-23T03:35:35.381 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic 2024-03-23T03:35:35.381 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-23T03:35:35.381 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-23T03:35:35.381 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-23T03:35:35.381 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-23T03:35:35.381 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-23T03:35:35.381 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-23T03:35:35.381 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-23T03:35:35.381 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci 2024-03-23T03:35:35.381 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-23T03:35:35.381 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-23T03:35:35.381 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-23T03:35:35.381 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc 2024-03-23T03:35:35.381 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-23T03:35:35.381 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-23T03:35:35.382 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-23T03:35:35.382 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas 2024-03-23T03:35:35.382 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-23T03:35:35.382 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc 2024-03-23T03:35:35.382 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-23T03:35:35.382 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid 2024-03-23T03:35:35.382 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-23T03:35:35.382 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-23T03:35:35.382 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-23T03:35:35.382 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-23T03:35:35.382 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas 2024-03-23T03:35:35.382 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-23T03:35:35.382 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas 2024-03-23T03:35:35.383 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-23T03:35:35.383 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-23T03:35:35.383 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001 2024-03-23T03:35:35.383 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-23T03:35:35.383 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-23T03:35:35.383 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-23T03:35:35.383 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx 2024-03-23T03:35:35.383 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-23T03:35:35.383 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-23T03:35:35.383 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx 2024-03-23T03:35:35.383 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-23T03:35:35.383 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-23T03:35:35.383 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-23T03:35:35.383 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-23T03:35:35.383 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-23T03:35:35.384 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-23T03:35:35.384 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-23T03:35:35.384 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-23T03:35:35.384 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-23T03:35:35.384 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-23T03:35:35.384 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-23T03:35:35.384 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2 2024-03-23T03:35:35.384 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-23T03:35:35.384 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-23T03:35:35.384 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-23T03:35:35.384 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-23T03:35:35.384 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb 2024-03-23T03:35:35.384 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-23T03:35:35.384 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target 2024-03-23T03:35:35.384 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi 2024-03-23T03:35:35.384 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-23T03:35:35.385 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback 2024-03-23T03:35:35.385 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-23T03:35:35.385 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-23T03:35:35.385 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-23T03:35:35.385 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-23T03:35:35.385 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-23T03:35:35.385 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-23T03:35:35.385 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc 2024-03-23T03:35:35.385 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-23T03:35:35.385 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal 2024-03-23T03:35:35.385 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel 2024-03-23T03:35:35.385 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal 2024-03-23T03:35:35.385 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-23T03:35:35.385 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-23T03:35:35.386 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-23T03:35:35.386 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-23T03:35:35.386 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-23T03:35:35.386 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-23T03:35:35.386 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-23T03:35:35.386 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-23T03:35:35.386 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-23T03:35:35.386 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-23T03:35:35.386 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-23T03:35:35.386 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-23T03:35:35.386 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-23T03:35:35.386 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-23T03:35:35.386 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-23T03:35:35.386 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-23T03:35:35.386 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-23T03:35:35.387 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-23T03:35:35.387 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty 2024-03-23T03:35:35.387 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless 2024-03-23T03:35:35.387 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-23T03:35:35.387 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-23T03:35:35.387 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-23T03:35:35.387 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-23T03:35:35.387 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial 2024-03-23T03:35:35.387 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250 2024-03-23T03:35:35.387 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-23T03:35:35.387 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-23T03:35:35.387 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm 2024-03-23T03:35:35.387 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-23T03:35:35.387 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs 2024-03-23T03:35:35.387 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core 2024-03-23T03:35:35.388 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-23T03:35:35.388 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host 2024-03-23T03:35:35.388 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-23T03:35:35.388 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio 2024-03-23T03:35:35.388 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-23T03:35:35.388 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-23T03:35:35.388 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-23T03:35:35.388 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-23T03:35:35.388 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-23T03:35:35.388 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb 2024-03-23T03:35:35.388 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class 2024-03-23T03:35:35.388 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-23T03:35:35.388 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-23T03:35:35.388 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-23T03:35:35.389 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-23T03:35:35.389 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host 2024-03-23T03:35:35.389 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-23T03:35:35.389 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image 2024-03-23T03:35:35.389 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-23T03:35:35.389 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-23T03:35:35.389 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc 2024-03-23T03:35:35.389 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-23T03:35:35.389 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-23T03:35:35.389 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-23T03:35:35.389 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-23T03:35:35.389 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-23T03:35:35.389 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-23T03:35:35.389 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-23T03:35:35.389 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-23T03:35:35.390 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-23T03:35:35.390 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-23T03:35:35.390 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga 2024-03-23T03:35:35.390 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-23T03:35:35.390 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-23T03:35:35.390 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-23T03:35:35.390 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-23T03:35:35.390 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-23T03:35:35.390 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-23T03:35:35.390 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-23T03:35:35.390 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy 2024-03-23T03:35:35.390 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-23T03:35:35.390 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial 2024-03-23T03:35:35.390 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-23T03:35:35.390 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-23T03:35:35.391 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-23T03:35:35.391 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-23T03:35:35.391 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-23T03:35:35.391 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-23T03:35:35.391 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-23T03:35:35.391 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-23T03:35:35.391 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-23T03:35:35.391 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-23T03:35:35.391 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-23T03:35:35.391 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-23T03:35:35.391 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-23T03:35:35.391 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-23T03:35:35.391 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-23T03:35:35.391 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-23T03:35:35.392 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-23T03:35:35.392 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-23T03:35:35.392 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-23T03:35:35.392 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-23T03:35:35.392 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-23T03:35:35.392 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-23T03:35:35.392 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-23T03:35:35.392 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-23T03:35:35.392 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-23T03:35:35.392 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-23T03:35:35.392 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-23T03:35:35.392 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-23T03:35:35.392 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-23T03:35:35.392 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-23T03:35:35.392 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-23T03:35:35.393 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-23T03:35:35.393 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-23T03:35:35.393 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-23T03:35:35.393 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-23T03:35:35.393 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-23T03:35:35.393 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-23T03:35:35.393 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-23T03:35:35.393 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-23T03:35:35.393 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-23T03:35:35.393 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-23T03:35:35.393 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-23T03:35:35.393 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-23T03:35:35.393 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-23T03:35:35.393 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-23T03:35:35.393 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage 2024-03-23T03:35:35.393 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-23T03:35:35.394 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-23T03:35:35.394 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-23T03:35:35.394 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-23T03:35:35.394 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-23T03:35:35.394 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-23T03:35:35.394 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-23T03:35:35.394 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-23T03:35:35.394 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-23T03:35:35.394 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-23T03:35:35.394 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-23T03:35:35.394 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-23T03:35:35.394 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-23T03:35:35.395 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-23T03:35:35.395 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-23T03:35:35.395 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip 2024-03-23T03:35:35.395 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-23T03:35:35.395 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-23T03:35:35.395 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-23T03:35:35.395 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio 2024-03-23T03:35:35.395 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci 2024-03-23T03:35:35.395 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-23T03:35:35.395 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-23T03:35:35.395 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-23T03:35:35.395 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-23T03:35:35.395 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost 2024-03-23T03:35:35.395 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-23T03:35:35.395 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-23T03:35:35.396 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-23T03:35:35.396 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-23T03:35:35.396 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video 2024-03-23T03:35:35.396 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight 2024-03-23T03:35:35.396 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-23T03:35:35.396 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-23T03:35:35.396 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-23T03:35:35.396 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio 2024-03-23T03:35:35.396 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio.ko 2024-03-23T03:35:35.396 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_balloon.ko 2024-03-23T03:35:35.396 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_mmio.ko 2024-03-23T03:35:35.396 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci.ko 2024-03-23T03:35:35.396 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-23T03:35:35.396 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-23T03:35:35.396 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_ring.ko 2024-03-23T03:35:35.396 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1 2024-03-23T03:35:35.397 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters 2024-03-23T03:35:35.397 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-23T03:35:35.397 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-23T03:35:35.397 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves 2024-03-23T03:35:35.397 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-23T03:35:35.397 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-23T03:35:35.397 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-23T03:35:35.397 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-23T03:35:35.397 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-23T03:35:35.397 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-23T03:35:35.397 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-23T03:35:35.397 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-23T03:35:35.397 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-23T03:35:35.397 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-23T03:35:35.397 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-23T03:35:35.397 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-23T03:35:35.398 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog 2024-03-23T03:35:35.398 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-23T03:35:35.398 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-23T03:35:35.398 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-23T03:35:35.398 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-23T03:35:35.398 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-23T03:35:35.398 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-23T03:35:35.398 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-23T03:35:35.398 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-23T03:35:35.398 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-23T03:35:35.398 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-23T03:35:35.398 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-23T03:35:35.398 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-23T03:35:35.398 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-23T03:35:35.398 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-23T03:35:35.399 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-23T03:35:35.399 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-23T03:35:35.399 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-23T03:35:35.399 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-23T03:35:35.399 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-23T03:35:35.399 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-23T03:35:35.399 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-23T03:35:35.399 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-23T03:35:35.399 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-23T03:35:35.399 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-23T03:35:35.399 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-23T03:35:35.399 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-23T03:35:35.400 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen 2024-03-23T03:35:35.400 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-23T03:35:35.400 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-23T03:35:35.400 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-23T03:35:35.400 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-23T03:35:35.400 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback 2024-03-23T03:35:35.400 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-23T03:35:35.400 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-23T03:35:35.400 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs 2024-03-23T03:35:35.400 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-23T03:35:35.400 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs 2024-03-23T03:35:35.400 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p 2024-03-23T03:35:35.400 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-23T03:35:35.400 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs 2024-03-23T03:35:35.401 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-23T03:35:35.401 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs 2024-03-23T03:35:35.401 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-23T03:35:35.401 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-23T03:35:35.401 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs 2024-03-23T03:35:35.401 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-23T03:35:35.401 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles 2024-03-23T03:35:35.401 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-23T03:35:35.401 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph 2024-03-23T03:35:35.401 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-23T03:35:35.401 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda 2024-03-23T03:35:35.401 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-23T03:35:35.401 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs 2024-03-23T03:35:35.402 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-23T03:35:35.402 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm 2024-03-23T03:35:35.402 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-23T03:35:35.402 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs 2024-03-23T03:35:35.402 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-23T03:35:35.402 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs 2024-03-23T03:35:35.402 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-23T03:35:35.402 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat 2024-03-23T03:35:35.402 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/fat.ko 2024-03-23T03:35:35.402 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-23T03:35:35.402 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/vfat.ko 2024-03-23T03:35:35.402 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse 2024-03-23T03:35:35.402 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-23T03:35:35.402 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/fuse.ko 2024-03-23T03:35:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2 2024-03-23T03:35:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-23T03:35:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs 2024-03-23T03:35:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-23T03:35:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus 2024-03-23T03:35:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-23T03:35:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs 2024-03-23T03:35:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-23T03:35:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs 2024-03-23T03:35:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-23T03:35:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd 2024-03-23T03:35:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-23T03:35:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix 2024-03-23T03:35:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-23T03:35:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs 2024-03-23T03:35:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-23T03:35:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs 2024-03-23T03:35:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout 2024-03-23T03:35:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-23T03:35:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout 2024-03-23T03:35:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-23T03:35:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout 2024-03-23T03:35:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-23T03:35:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-23T03:35:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-23T03:35:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-23T03:35:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common 2024-03-23T03:35:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-23T03:35:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-23T03:35:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd 2024-03-23T03:35:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-23T03:35:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2 2024-03-23T03:35:35.405 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-23T03:35:35.405 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls 2024-03-23T03:35:35.405 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-23T03:35:35.405 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-23T03:35:35.405 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-23T03:35:35.405 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-23T03:35:35.405 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-23T03:35:35.405 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-23T03:35:35.405 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-23T03:35:35.405 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-23T03:35:35.405 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-23T03:35:35.405 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-23T03:35:35.405 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-23T03:35:35.405 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-23T03:35:35.405 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-23T03:35:35.406 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-23T03:35:35.406 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-23T03:35:35.406 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-23T03:35:35.406 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-23T03:35:35.406 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-23T03:35:35.406 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-23T03:35:35.406 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-23T03:35:35.406 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-23T03:35:35.406 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-23T03:35:35.406 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-23T03:35:35.406 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-23T03:35:35.406 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-23T03:35:35.406 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-23T03:35:35.406 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-23T03:35:35.407 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-23T03:35:35.407 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-23T03:35:35.407 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-23T03:35:35.407 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-23T03:35:35.407 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-23T03:35:35.407 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-23T03:35:35.407 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-23T03:35:35.407 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-23T03:35:35.407 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-23T03:35:35.407 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-23T03:35:35.407 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-23T03:35:35.407 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-23T03:35:35.407 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-23T03:35:35.407 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-23T03:35:35.407 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-23T03:35:35.408 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-23T03:35:35.408 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-23T03:35:35.408 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-23T03:35:35.408 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-23T03:35:35.408 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-23T03:35:35.408 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-23T03:35:35.408 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-23T03:35:35.408 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-23T03:35:35.408 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2 2024-03-23T03:35:35.408 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster 2024-03-23T03:35:35.408 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-23T03:35:35.408 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm 2024-03-23T03:35:35.409 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-23T03:35:35.409 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs 2024-03-23T03:35:35.409 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-23T03:35:35.409 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-23T03:35:35.409 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-23T03:35:35.409 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-23T03:35:35.409 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-23T03:35:35.409 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs 2024-03-23T03:35:35.409 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-23T03:35:35.409 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore 2024-03-23T03:35:35.409 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-23T03:35:35.409 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs 2024-03-23T03:35:35.409 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-23T03:35:35.410 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs 2024-03-23T03:35:35.410 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-23T03:35:35.410 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb 2024-03-23T03:35:35.410 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client 2024-03-23T03:35:35.410 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-23T03:35:35.410 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common 2024-03-23T03:35:35.410 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-23T03:35:35.410 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-23T03:35:35.410 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs 2024-03-23T03:35:35.410 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-23T03:35:35.410 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv 2024-03-23T03:35:35.410 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-23T03:35:35.410 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf 2024-03-23T03:35:35.410 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-23T03:35:35.410 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs 2024-03-23T03:35:35.411 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-23T03:35:35.411 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs 2024-03-23T03:35:35.411 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-23T03:35:35.411 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel 2024-03-23T03:35:35.411 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu 2024-03-23T03:35:35.411 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu/rcutorture.ko 2024-03-23T03:35:35.411 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-23T03:35:35.411 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace 2024-03-23T03:35:35.411 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-23T03:35:35.411 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib 2024-03-23T03:35:35.411 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/asn1_encoder.ko 2024-03-23T03:35:35.411 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-23T03:35:35.411 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-23T03:35:35.411 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto 2024-03-23T03:35:35.411 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-23T03:35:35.411 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-23T03:35:35.412 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-23T03:35:35.412 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-23T03:35:35.412 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4 2024-03-23T03:35:35.412 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-23T03:35:35.412 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-23T03:35:35.412 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math 2024-03-23T03:35:35.412 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-23T03:35:35.412 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6 2024-03-23T03:35:35.412 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-23T03:35:35.412 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon 2024-03-23T03:35:35.412 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-23T03:35:35.412 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-23T03:35:35.412 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-23T03:35:35.412 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-23T03:35:35.412 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd 2024-03-23T03:35:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd/zstd_compress.ko 2024-03-23T03:35:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm 2024-03-23T03:35:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-23T03:35:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net 2024-03-23T03:35:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan 2024-03-23T03:35:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-23T03:35:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-23T03:35:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-23T03:35:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-23T03:35:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-23T03:35:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-23T03:35:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-23T03:35:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-23T03:35:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802 2024-03-23T03:35:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-23T03:35:35.413 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-23T03:35:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-23T03:35:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-23T03:35:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-23T03:35:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q 2024-03-23T03:35:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-23T03:35:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p 2024-03-23T03:35:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-23T03:35:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-23T03:35:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-23T03:35:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-23T03:35:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv 2024-03-23T03:35:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-23T03:35:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge 2024-03-23T03:35:35.414 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-23T03:35:35.415 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-23T03:35:35.415 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter 2024-03-23T03:35:35.415 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-23T03:35:35.415 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-23T03:35:35.415 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-23T03:35:35.415 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-23T03:35:35.415 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-23T03:35:35.415 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-23T03:35:35.415 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-23T03:35:35.415 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-23T03:35:35.415 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-23T03:35:35.415 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-23T03:35:35.415 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-23T03:35:35.415 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-23T03:35:35.415 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-23T03:35:35.415 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-23T03:35:35.416 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-23T03:35:35.416 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-23T03:35:35.416 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-23T03:35:35.416 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-23T03:35:35.416 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-23T03:35:35.416 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-23T03:35:35.416 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-23T03:35:35.416 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-23T03:35:35.416 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-23T03:35:35.416 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-23T03:35:35.416 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph 2024-03-23T03:35:35.416 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-23T03:35:35.416 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core 2024-03-23T03:35:35.416 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/failover.ko 2024-03-23T03:35:35.416 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-23T03:35:35.417 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp 2024-03-23T03:35:35.417 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-23T03:35:35.417 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-23T03:35:35.417 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-23T03:35:35.417 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-23T03:35:35.417 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver 2024-03-23T03:35:35.417 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver/dns_resolver.ko 2024-03-23T03:35:35.417 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa 2024-03-23T03:35:35.417 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-23T03:35:35.417 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-23T03:35:35.417 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-23T03:35:35.417 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-23T03:35:35.417 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-23T03:35:35.417 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154 2024-03-23T03:35:35.417 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan 2024-03-23T03:35:35.417 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-23T03:35:35.418 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-23T03:35:35.418 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-23T03:35:35.418 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4 2024-03-23T03:35:35.418 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-23T03:35:35.418 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-23T03:35:35.418 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-23T03:35:35.418 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-23T03:35:35.418 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-23T03:35:35.418 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-23T03:35:35.418 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-23T03:35:35.418 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-23T03:35:35.418 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-23T03:35:35.418 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-23T03:35:35.418 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter 2024-03-23T03:35:35.418 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-23T03:35:35.419 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-23T03:35:35.419 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-23T03:35:35.419 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-23T03:35:35.419 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-23T03:35:35.419 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-23T03:35:35.419 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-23T03:35:35.419 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-23T03:35:35.419 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-23T03:35:35.419 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-23T03:35:35.419 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-23T03:35:35.419 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-23T03:35:35.419 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-23T03:35:35.419 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-23T03:35:35.419 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-23T03:35:35.419 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-23T03:35:35.419 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-23T03:35:35.420 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-23T03:35:35.420 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-23T03:35:35.420 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-23T03:35:35.420 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-23T03:35:35.420 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-23T03:35:35.420 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-23T03:35:35.420 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-23T03:35:35.420 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-23T03:35:35.420 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-23T03:35:35.420 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-23T03:35:35.420 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-23T03:35:35.420 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-23T03:35:35.420 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-23T03:35:35.420 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-23T03:35:35.420 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-23T03:35:35.420 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-23T03:35:35.420 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-23T03:35:35.421 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-23T03:35:35.421 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-23T03:35:35.421 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-23T03:35:35.421 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-23T03:35:35.421 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6 2024-03-23T03:35:35.421 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-23T03:35:35.421 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-23T03:35:35.421 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-23T03:35:35.421 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-23T03:35:35.421 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-23T03:35:35.421 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-23T03:35:35.421 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-23T03:35:35.422 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter 2024-03-23T03:35:35.422 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-23T03:35:35.422 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-23T03:35:35.422 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-23T03:35:35.422 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-23T03:35:35.422 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-23T03:35:35.422 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-23T03:35:35.422 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-23T03:35:35.422 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-23T03:35:35.422 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-23T03:35:35.422 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-23T03:35:35.422 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-23T03:35:35.422 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-23T03:35:35.423 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-23T03:35:35.423 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-23T03:35:35.423 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-23T03:35:35.423 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-23T03:35:35.423 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-23T03:35:35.423 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-23T03:35:35.423 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-23T03:35:35.423 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-23T03:35:35.423 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-23T03:35:35.423 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-23T03:35:35.423 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-23T03:35:35.423 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-23T03:35:35.423 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-23T03:35:35.423 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-23T03:35:35.424 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-23T03:35:35.424 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key 2024-03-23T03:35:35.424 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-23T03:35:35.424 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp 2024-03-23T03:35:35.424 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-23T03:35:35.424 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-23T03:35:35.424 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-23T03:35:35.424 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-23T03:35:35.424 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-23T03:35:35.424 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-23T03:35:35.424 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-23T03:35:35.424 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc 2024-03-23T03:35:35.424 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-23T03:35:35.424 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154 2024-03-23T03:35:35.424 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-23T03:35:35.425 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls 2024-03-23T03:35:35.425 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-23T03:35:35.425 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter 2024-03-23T03:35:35.425 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset 2024-03-23T03:35:35.425 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-23T03:35:35.425 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-23T03:35:35.425 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-23T03:35:35.425 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-23T03:35:35.425 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-23T03:35:35.425 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-23T03:35:35.425 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-23T03:35:35.425 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-23T03:35:35.426 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-23T03:35:35.426 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-23T03:35:35.426 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-23T03:35:35.426 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-23T03:35:35.426 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-23T03:35:35.426 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-23T03:35:35.426 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-23T03:35:35.426 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-23T03:35:35.426 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs 2024-03-23T03:35:35.426 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-23T03:35:35.426 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-23T03:35:35.426 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-23T03:35:35.427 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-23T03:35:35.427 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-23T03:35:35.427 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-23T03:35:35.427 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-23T03:35:35.427 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-23T03:35:35.427 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-23T03:35:35.427 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-23T03:35:35.427 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-23T03:35:35.427 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-23T03:35:35.427 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-23T03:35:35.427 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-23T03:35:35.427 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-23T03:35:35.427 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-23T03:35:35.428 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-23T03:35:35.428 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-23T03:35:35.428 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-23T03:35:35.428 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-23T03:35:35.428 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-23T03:35:35.428 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-23T03:35:35.428 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-23T03:35:35.428 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-23T03:35:35.428 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-23T03:35:35.428 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-23T03:35:35.428 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-23T03:35:35.428 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-23T03:35:35.428 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-23T03:35:35.428 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-23T03:35:35.429 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-23T03:35:35.429 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-23T03:35:35.429 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-23T03:35:35.429 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-23T03:35:35.429 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-23T03:35:35.429 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-23T03:35:35.429 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-23T03:35:35.429 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-23T03:35:35.429 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-23T03:35:35.429 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-23T03:35:35.429 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-23T03:35:35.429 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-23T03:35:35.430 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-23T03:35:35.430 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-23T03:35:35.430 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-23T03:35:35.430 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-23T03:35:35.430 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-23T03:35:35.430 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-23T03:35:35.430 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-23T03:35:35.430 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-23T03:35:35.430 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-23T03:35:35.430 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-23T03:35:35.430 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-23T03:35:35.430 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-23T03:35:35.431 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-23T03:35:35.431 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-23T03:35:35.431 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-23T03:35:35.431 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-23T03:35:35.431 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_quota.ko 2024-03-23T03:35:35.431 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-23T03:35:35.431 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-23T03:35:35.431 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-23T03:35:35.431 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-23T03:35:35.431 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tunnel.ko 2024-03-23T03:35:35.431 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-23T03:35:35.431 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-23T03:35:35.431 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-23T03:35:35.431 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-23T03:35:35.432 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-23T03:35:35.432 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-23T03:35:35.432 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-23T03:35:35.432 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-23T03:35:35.432 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-23T03:35:35.432 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-23T03:35:35.432 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-23T03:35:35.432 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-23T03:35:35.432 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-23T03:35:35.432 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-23T03:35:35.432 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-23T03:35:35.432 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-23T03:35:35.432 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-23T03:35:35.433 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-23T03:35:35.433 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-23T03:35:35.433 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-23T03:35:35.433 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-23T03:35:35.433 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-23T03:35:35.433 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-23T03:35:35.433 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-23T03:35:35.433 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-23T03:35:35.433 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-23T03:35:35.433 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-23T03:35:35.433 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-23T03:35:35.433 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-23T03:35:35.433 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-23T03:35:35.434 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-23T03:35:35.434 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-23T03:35:35.434 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-23T03:35:35.434 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-23T03:35:35.434 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-23T03:35:35.434 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-23T03:35:35.434 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-23T03:35:35.434 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-23T03:35:35.434 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-23T03:35:35.434 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-23T03:35:35.434 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-23T03:35:35.434 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-23T03:35:35.434 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-23T03:35:35.435 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-23T03:35:35.435 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-23T03:35:35.435 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-23T03:35:35.435 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-23T03:35:35.435 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-23T03:35:35.435 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-23T03:35:35.435 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-23T03:35:35.435 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-23T03:35:35.435 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-23T03:35:35.435 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-23T03:35:35.435 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-23T03:35:35.435 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-23T03:35:35.435 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-23T03:35:35.436 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-23T03:35:35.436 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-23T03:35:35.436 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-23T03:35:35.436 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-23T03:35:35.436 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-23T03:35:35.436 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-23T03:35:35.436 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-23T03:35:35.436 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-23T03:35:35.436 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-23T03:35:35.436 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-23T03:35:35.436 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-23T03:35:35.436 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-23T03:35:35.436 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-23T03:35:35.437 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-23T03:35:35.437 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-23T03:35:35.437 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink 2024-03-23T03:35:35.437 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-23T03:35:35.437 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh 2024-03-23T03:35:35.437 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-23T03:35:35.437 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch 2024-03-23T03:35:35.437 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-23T03:35:35.437 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-23T03:35:35.437 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-23T03:35:35.437 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-23T03:35:35.437 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet 2024-03-23T03:35:35.438 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-23T03:35:35.438 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds 2024-03-23T03:35:35.438 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-23T03:35:35.438 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-23T03:35:35.438 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-23T03:35:35.438 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched 2024-03-23T03:35:35.438 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-23T03:35:35.438 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-23T03:35:35.438 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-23T03:35:35.438 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-23T03:35:35.438 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-23T03:35:35.438 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-23T03:35:35.438 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-23T03:35:35.439 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-23T03:35:35.439 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-23T03:35:35.439 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-23T03:35:35.439 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-23T03:35:35.439 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-23T03:35:35.439 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-23T03:35:35.439 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-23T03:35:35.439 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-23T03:35:35.439 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-23T03:35:35.439 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-23T03:35:35.439 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-23T03:35:35.439 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-23T03:35:35.439 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-23T03:35:35.440 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-23T03:35:35.440 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-23T03:35:35.440 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-23T03:35:35.440 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-23T03:35:35.440 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-23T03:35:35.440 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-23T03:35:35.440 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-23T03:35:35.440 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-23T03:35:35.440 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-23T03:35:35.440 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-23T03:35:35.440 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-23T03:35:35.440 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-23T03:35:35.440 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-23T03:35:35.441 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-23T03:35:35.441 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-23T03:35:35.441 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-23T03:35:35.441 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-23T03:35:35.441 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-23T03:35:35.441 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-23T03:35:35.441 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-23T03:35:35.441 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-23T03:35:35.441 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-23T03:35:35.441 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-23T03:35:35.441 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp 2024-03-23T03:35:35.441 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-23T03:35:35.441 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-23T03:35:35.442 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc 2024-03-23T03:35:35.442 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss 2024-03-23T03:35:35.442 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-23T03:35:35.442 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-23T03:35:35.442 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-23T03:35:35.442 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma 2024-03-23T03:35:35.442 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-23T03:35:35.442 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc 2024-03-23T03:35:35.442 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-23T03:35:35.442 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-23T03:35:35.442 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix 2024-03-23T03:35:35.442 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-23T03:35:35.442 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock 2024-03-23T03:35:35.442 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-23T03:35:35.443 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-23T03:35:35.443 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-23T03:35:35.443 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-23T03:35:35.443 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-23T03:35:35.443 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm 2024-03-23T03:35:35.443 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-23T03:35:35.443 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security 2024-03-23T03:35:35.443 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys 2024-03-23T03:35:35.443 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys 2024-03-23T03:35:35.443 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-23T03:35:35.443 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys 2024-03-23T03:35:35.443 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys/trusted.ko 2024-03-23T03:35:35.443 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt 2024-03-23T03:35:35.443 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib 2024-03-23T03:35:35.444 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-23T03:35:35.444 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-23T03:35:35.444 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-23T03:35:35.444 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-23T03:35:35.444 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.alias.bin 2024-03-23T03:35:35.444 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-23T03:35:35.444 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-23T03:35:35.444 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-23T03:35:35.444 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-23T03:35:35.444 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-23T03:35:35.444 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-23T03:35:35.444 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-23T03:35:35.444 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-23T03:35:35.445 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-23T03:35:35.445 INFO:teuthology.orchestra.run.smithi173.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-23T03:35:35.448 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-23T03:35:35.449 DEBUG:teuthology.orchestra.run.smithi173:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-23T03:35:35.472 INFO:teuthology.orchestra.run.smithi173.stdout:postinstall scriptlet (using /bin/sh): 2024-03-23T03:35:35.473 INFO:teuthology.orchestra.run.smithi173.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-23T03:35:35.473 INFO:teuthology.orchestra.run.smithi173.stdout: /usr/bin/kernel-install add 6.8.0-g5ec3e6e88b2e /lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-23T03:35:35.473 INFO:teuthology.orchestra.run.smithi173.stdout:fi 2024-03-23T03:35:35.473 INFO:teuthology.orchestra.run.smithi173.stdout:for file in vmlinuz System.map config; do 2024-03-23T03:35:35.473 INFO:teuthology.orchestra.run.smithi173.stdout: if ! cmp --silent "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e"; then 2024-03-23T03:35:35.473 INFO:teuthology.orchestra.run.smithi173.stdout: cp "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e" 2024-03-23T03:35:35.473 INFO:teuthology.orchestra.run.smithi173.stdout: fi 2024-03-23T03:35:35.473 INFO:teuthology.orchestra.run.smithi173.stdout:done 2024-03-23T03:35:35.473 INFO:teuthology.orchestra.run.smithi173.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-23T03:35:35.473 INFO:teuthology.orchestra.run.smithi173.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-23T03:35:35.473 INFO:teuthology.orchestra.run.smithi173.stdout:new-kernel-pkg --remove 6.8.0-g5ec3e6e88b2e --rminitrd --initrdfile=/boot/initramfs-6.8.0-g5ec3e6e88b2e.img 2024-03-23T03:35:35.474 INFO:teuthology.orchestra.run.smithi173.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-23T03:35:35.474 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-install remove 6.8.0-g5ec3e6e88b2e 2024-03-23T03:35:35.474 INFO:teuthology.orchestra.run.smithi173.stdout:fi 2024-03-23T03:35:35.474 INFO:teuthology.orchestra.run.smithi173.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-23T03:35:35.474 INFO:teuthology.orchestra.run.smithi173.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-23T03:35:35.474 INFO:teuthology.orchestra.run.smithi173.stdout:/sbin/update-bootloader --remove 6.8.0-g5ec3e6e88b2e 2024-03-23T03:35:35.474 INFO:teuthology.orchestra.run.smithi173.stdout:fi 2024-03-23T03:35:35.475 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -qi grub2-tools 2024-03-23T03:35:35.551 INFO:teuthology.orchestra.run.smithi173.stdout:Name : grub2-tools 2024-03-23T03:35:35.551 INFO:teuthology.orchestra.run.smithi173.stdout:Epoch : 1 2024-03-23T03:35:35.551 INFO:teuthology.orchestra.run.smithi173.stdout:Version : 2.06 2024-03-23T03:35:35.551 INFO:teuthology.orchestra.run.smithi173.stdout:Release : 68.el9 2024-03-23T03:35:35.551 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: x86_64 2024-03-23T03:35:35.552 INFO:teuthology.orchestra.run.smithi173.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-23T03:35:35.552 INFO:teuthology.orchestra.run.smithi173.stdout:Group : Unspecified 2024-03-23T03:35:35.552 INFO:teuthology.orchestra.run.smithi173.stdout:Size : 8264254 2024-03-23T03:35:35.552 INFO:teuthology.orchestra.run.smithi173.stdout:License : GPLv3+ 2024-03-23T03:35:35.552 INFO:teuthology.orchestra.run.smithi173.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-23T03:35:35.552 INFO:teuthology.orchestra.run.smithi173.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-23T03:35:35.552 INFO:teuthology.orchestra.run.smithi173.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-23T03:35:35.552 INFO:teuthology.orchestra.run.smithi173.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-23T03:35:35.552 INFO:teuthology.orchestra.run.smithi173.stdout:Packager : builder@centos.org 2024-03-23T03:35:35.552 INFO:teuthology.orchestra.run.smithi173.stdout:Vendor : CentOS 2024-03-23T03:35:35.552 INFO:teuthology.orchestra.run.smithi173.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-23T03:35:35.553 INFO:teuthology.orchestra.run.smithi173.stdout:Summary : Support tools for GRUB. 2024-03-23T03:35:35.553 INFO:teuthology.orchestra.run.smithi173.stdout:Description : 2024-03-23T03:35:35.553 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:35:35.553 INFO:teuthology.orchestra.run.smithi173.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-23T03:35:35.553 INFO:teuthology.orchestra.run.smithi173.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-23T03:35:35.553 INFO:teuthology.orchestra.run.smithi173.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-23T03:35:35.553 INFO:teuthology.orchestra.run.smithi173.stdout:hardware devices. 2024-03-23T03:35:35.553 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:35:35.553 INFO:teuthology.orchestra.run.smithi173.stdout:This subpackage provides tools for support of all platforms. 2024-03-23T03:35:35.554 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-23T03:35:35.554 INFO:teuthology.task.kernel:Updating grub on smithi173 to boot 6.8.0-g5ec3e6e88b2e 2024-03-23T03:35:35.554 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-23T03:35:36.129 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2024-03-23T03:35:37.539 INFO:teuthology.orchestra.run.smithi173.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-23T03:35:37.574 INFO:teuthology.orchestra.run.smithi173.stderr:done 2024-03-23T03:35:37.577 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-03-23T03:35:37.592 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.kMs0OFPkGC 2024-03-23T03:35:37.592 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kMs0OFPkGC 2024-03-23T03:35:37.684 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.kMs0OFPkGC 2024-03-23T03:35:37.788 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.kMs0OFPkGC is 6KB 2024-03-23T03:35:37.836 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.kMs0OFPkGC 2024-03-23T03:35:37.850 DEBUG:teuthology.orchestra.run.smithi173:> sudo /bin/ls /boot/loader/entries || true 2024-03-23T03:35:37.916 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-23T03:35:37.916 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-23T03:35:37.917 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-23T03:35:37.917 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e.conf 2024-03-23T03:35:37.918 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e 2024-03-23T03:35:38.071 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2024-03-23T03:35:40.857 INFO:teuthology.orchestra.run.smithi102.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-23T03:36:03.257 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi102.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-23T03:36:03.258 DEBUG:teuthology.orchestra.run.smithi102:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-23T03:36:03.325 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e 2024-03-23T03:36:03.325 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/System.map 2024-03-23T03:36:03.325 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/config 2024-03-23T03:36:03.325 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel 2024-03-23T03:36:03.325 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch 2024-03-23T03:36:03.325 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86 2024-03-23T03:36:03.325 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto 2024-03-23T03:36:03.325 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-23T03:36:03.325 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-23T03:36:03.325 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-23T03:36:03.325 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-23T03:36:03.325 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-23T03:36:03.326 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-23T03:36:03.326 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-23T03:36:03.326 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-23T03:36:03.326 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-23T03:36:03.326 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-23T03:36:03.326 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-23T03:36:03.326 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-23T03:36:03.326 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-23T03:36:03.326 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-23T03:36:03.326 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-23T03:36:03.326 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-23T03:36:03.326 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-23T03:36:03.326 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-23T03:36:03.326 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-23T03:36:03.327 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-23T03:36:03.327 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel 2024-03-23T03:36:03.327 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu 2024-03-23T03:36:03.327 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce 2024-03-23T03:36:03.327 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-23T03:36:03.327 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm 2024-03-23T03:36:03.327 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-23T03:36:03.327 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-23T03:36:03.327 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-23T03:36:03.327 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto 2024-03-23T03:36:03.327 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-23T03:36:03.327 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-23T03:36:03.327 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-23T03:36:03.327 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx 2024-03-23T03:36:03.328 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-23T03:36:03.328 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-23T03:36:03.328 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-23T03:36:03.328 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-23T03:36:03.328 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-23T03:36:03.328 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-23T03:36:03.328 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-23T03:36:03.328 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-23T03:36:03.328 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-23T03:36:03.328 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-23T03:36:03.328 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-23T03:36:03.328 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-23T03:36:03.328 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-23T03:36:03.328 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-23T03:36:03.329 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-23T03:36:03.329 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-23T03:36:03.329 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-23T03:36:03.329 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-23T03:36:03.329 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-23T03:36:03.329 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-23T03:36:03.329 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-23T03:36:03.329 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-23T03:36:03.329 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-23T03:36:03.329 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-23T03:36:03.329 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-23T03:36:03.329 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-23T03:36:03.329 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-23T03:36:03.329 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-23T03:36:03.330 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-23T03:36:03.330 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-23T03:36:03.330 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-23T03:36:03.330 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-23T03:36:03.330 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-23T03:36:03.330 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-23T03:36:03.330 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-23T03:36:03.330 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-23T03:36:03.330 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-23T03:36:03.330 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-23T03:36:03.330 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-23T03:36:03.330 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-23T03:36:03.330 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-23T03:36:03.331 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-23T03:36:03.331 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-23T03:36:03.331 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-23T03:36:03.331 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-23T03:36:03.331 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers 2024-03-23T03:36:03.331 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi 2024-03-23T03:36:03.331 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-23T03:36:03.331 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-23T03:36:03.331 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/custom_method.ko 2024-03-23T03:36:03.331 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-23T03:36:03.331 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-23T03:36:03.331 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-23T03:36:03.331 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-23T03:36:03.331 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-23T03:36:03.331 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata 2024-03-23T03:36:03.332 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-23T03:36:03.332 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-23T03:36:03.332 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ata_generic.ko 2024-03-23T03:36:03.332 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-23T03:36:03.332 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-23T03:36:03.332 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-23T03:36:03.332 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-23T03:36:03.332 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-23T03:36:03.332 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-23T03:36:03.332 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-23T03:36:03.332 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-23T03:36:03.332 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-23T03:36:03.332 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-23T03:36:03.332 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-23T03:36:03.332 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-23T03:36:03.333 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-23T03:36:03.333 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-23T03:36:03.333 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-23T03:36:03.333 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-23T03:36:03.333 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-23T03:36:03.333 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-23T03:36:03.333 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-23T03:36:03.333 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-23T03:36:03.333 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-23T03:36:03.333 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-23T03:36:03.333 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-23T03:36:03.333 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-23T03:36:03.333 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-23T03:36:03.333 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-23T03:36:03.334 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-23T03:36:03.334 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-23T03:36:03.334 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-23T03:36:03.334 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-23T03:36:03.334 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-23T03:36:03.334 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-23T03:36:03.334 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-23T03:36:03.334 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-23T03:36:03.334 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-23T03:36:03.334 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sis.ko 2024-03-23T03:36:03.334 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-23T03:36:03.334 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-23T03:36:03.334 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-23T03:36:03.335 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-23T03:36:03.335 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-23T03:36:03.335 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-23T03:36:03.335 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-23T03:36:03.335 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-23T03:36:03.335 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-23T03:36:03.335 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-23T03:36:03.335 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-23T03:36:03.335 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-23T03:36:03.335 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-23T03:36:03.335 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-23T03:36:03.335 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-23T03:36:03.335 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-23T03:36:03.335 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-23T03:36:03.335 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay 2024-03-23T03:36:03.336 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-23T03:36:03.336 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base 2024-03-23T03:36:03.336 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap 2024-03-23T03:36:03.336 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-23T03:36:03.336 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-23T03:36:03.336 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma 2024-03-23T03:36:03.336 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-23T03:36:03.336 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block 2024-03-23T03:36:03.336 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe 2024-03-23T03:36:03.336 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-23T03:36:03.336 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/brd.ko 2024-03-23T03:36:03.336 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd 2024-03-23T03:36:03.336 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-23T03:36:03.336 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-23T03:36:03.337 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/loop.ko 2024-03-23T03:36:03.337 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx 2024-03-23T03:36:03.337 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-23T03:36:03.337 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-23T03:36:03.337 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk 2024-03-23T03:36:03.337 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-23T03:36:03.337 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-23T03:36:03.337 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-23T03:36:03.337 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/virtio_blk.ko 2024-03-23T03:36:03.337 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback 2024-03-23T03:36:03.337 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-23T03:36:03.337 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkfront.ko 2024-03-23T03:36:03.337 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram 2024-03-23T03:36:03.337 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-23T03:36:03.337 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char 2024-03-23T03:36:03.338 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-23T03:36:03.338 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random 2024-03-23T03:36:03.338 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-23T03:36:03.338 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-23T03:36:03.338 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-23T03:36:03.338 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-23T03:36:03.338 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-23T03:36:03.338 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi 2024-03-23T03:36:03.338 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-23T03:36:03.338 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-23T03:36:03.338 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-23T03:36:03.338 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-23T03:36:03.338 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-23T03:36:03.339 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-23T03:36:03.339 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-23T03:36:03.339 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave 2024-03-23T03:36:03.339 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-23T03:36:03.339 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-23T03:36:03.339 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-23T03:36:03.339 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm 2024-03-23T03:36:03.339 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm.ko 2024-03-23T03:36:03.339 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-23T03:36:03.339 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-23T03:36:03.339 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-23T03:36:03.339 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-23T03:36:03.339 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-23T03:36:03.339 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/uv_mmtimer.ko 2024-03-23T03:36:03.339 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/virtio_console.ko 2024-03-23T03:36:03.340 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq 2024-03-23T03:36:03.340 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-23T03:36:03.340 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-23T03:36:03.340 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-23T03:36:03.340 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-23T03:36:03.340 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-23T03:36:03.340 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-23T03:36:03.340 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto 2024-03-23T03:36:03.340 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp 2024-03-23T03:36:03.340 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-23T03:36:03.340 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-23T03:36:03.340 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel 2024-03-23T03:36:03.340 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat 2024-03-23T03:36:03.340 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common 2024-03-23T03:36:03.340 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-23T03:36:03.341 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-23T03:36:03.341 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-23T03:36:03.341 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-23T03:36:03.341 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-23T03:36:03.341 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio 2024-03-23T03:36:03.341 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-23T03:36:03.341 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca 2024-03-23T03:36:03.341 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-23T03:36:03.341 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq 2024-03-23T03:36:03.341 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-23T03:36:03.341 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma 2024-03-23T03:36:03.341 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw 2024-03-23T03:36:03.341 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-23T03:36:03.341 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat 2024-03-23T03:36:03.341 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-23T03:36:03.341 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac 2024-03-23T03:36:03.342 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-23T03:36:03.342 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-23T03:36:03.342 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-23T03:36:03.342 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-23T03:36:03.342 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-23T03:36:03.342 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-23T03:36:03.342 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-23T03:36:03.342 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-23T03:36:03.342 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-23T03:36:03.342 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-23T03:36:03.342 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-23T03:36:03.342 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-23T03:36:03.342 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-23T03:36:03.343 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware 2024-03-23T03:36:03.343 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/edd.ko 2024-03-23T03:36:03.343 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-23T03:36:03.343 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio 2024-03-23T03:36:03.343 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-23T03:36:03.343 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-23T03:36:03.343 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid 2024-03-23T03:36:03.343 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-23T03:36:03.343 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-23T03:36:03.343 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-23T03:36:03.343 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-23T03:36:03.343 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-23T03:36:03.343 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-23T03:36:03.343 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-23T03:36:03.344 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-23T03:36:03.344 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-23T03:36:03.344 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-23T03:36:03.344 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-23T03:36:03.344 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-23T03:36:03.344 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-23T03:36:03.344 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-23T03:36:03.344 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-23T03:36:03.344 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-23T03:36:03.344 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-23T03:36:03.344 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-23T03:36:03.344 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-23T03:36:03.344 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-23T03:36:03.344 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-23T03:36:03.345 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-23T03:36:03.345 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-23T03:36:03.345 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-23T03:36:03.345 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-23T03:36:03.345 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-23T03:36:03.345 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-23T03:36:03.345 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-23T03:36:03.345 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-23T03:36:03.345 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-23T03:36:03.345 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-23T03:36:03.345 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-23T03:36:03.345 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-23T03:36:03.345 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-23T03:36:03.345 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-23T03:36:03.345 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-23T03:36:03.346 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-23T03:36:03.346 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-23T03:36:03.346 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-23T03:36:03.346 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-23T03:36:03.346 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-23T03:36:03.346 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-23T03:36:03.346 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-23T03:36:03.346 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-23T03:36:03.346 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-23T03:36:03.346 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-23T03:36:03.346 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-23T03:36:03.346 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-23T03:36:03.346 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-23T03:36:03.346 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-23T03:36:03.346 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-23T03:36:03.347 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-23T03:36:03.347 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-23T03:36:03.347 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-23T03:36:03.347 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-23T03:36:03.347 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-23T03:36:03.347 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-23T03:36:03.347 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-23T03:36:03.347 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-23T03:36:03.347 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-23T03:36:03.347 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-23T03:36:03.347 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-23T03:36:03.347 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-23T03:36:03.347 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-23T03:36:03.347 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv 2024-03-23T03:36:03.348 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-23T03:36:03.348 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-23T03:36:03.348 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-23T03:36:03.348 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon 2024-03-23T03:36:03.348 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-23T03:36:03.348 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-23T03:36:03.348 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-23T03:36:03.348 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-23T03:36:03.348 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-23T03:36:03.348 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-23T03:36:03.348 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-23T03:36:03.348 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-23T03:36:03.348 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-23T03:36:03.348 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-23T03:36:03.348 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-23T03:36:03.348 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-23T03:36:03.349 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-23T03:36:03.349 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-23T03:36:03.349 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-23T03:36:03.349 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-23T03:36:03.349 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-23T03:36:03.349 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-23T03:36:03.349 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-23T03:36:03.349 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-23T03:36:03.349 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-23T03:36:03.349 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-23T03:36:03.349 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-23T03:36:03.349 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-23T03:36:03.349 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-23T03:36:03.349 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-23T03:36:03.350 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-23T03:36:03.350 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-23T03:36:03.350 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-23T03:36:03.350 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-23T03:36:03.350 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-23T03:36:03.350 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-23T03:36:03.350 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-23T03:36:03.350 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-23T03:36:03.350 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-23T03:36:03.350 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-23T03:36:03.350 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-23T03:36:03.350 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-23T03:36:03.350 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-23T03:36:03.350 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-23T03:36:03.351 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-23T03:36:03.351 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-23T03:36:03.351 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-23T03:36:03.351 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-23T03:36:03.351 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-23T03:36:03.351 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-23T03:36:03.351 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-23T03:36:03.351 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-23T03:36:03.351 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-23T03:36:03.351 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-23T03:36:03.351 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-23T03:36:03.351 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-23T03:36:03.351 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-23T03:36:03.351 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-23T03:36:03.351 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-23T03:36:03.351 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-23T03:36:03.352 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-23T03:36:03.352 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-23T03:36:03.352 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-23T03:36:03.352 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-23T03:36:03.352 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-23T03:36:03.352 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-23T03:36:03.352 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-23T03:36:03.352 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-23T03:36:03.352 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-23T03:36:03.352 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-23T03:36:03.352 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-23T03:36:03.352 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-23T03:36:03.352 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-23T03:36:03.352 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-23T03:36:03.352 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-23T03:36:03.352 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-23T03:36:03.353 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-23T03:36:03.353 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-23T03:36:03.353 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-23T03:36:03.353 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-23T03:36:03.353 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-23T03:36:03.353 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-23T03:36:03.353 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-23T03:36:03.353 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-23T03:36:03.353 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-23T03:36:03.353 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-23T03:36:03.353 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-23T03:36:03.353 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-23T03:36:03.353 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-23T03:36:03.353 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-23T03:36:03.354 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-23T03:36:03.354 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-23T03:36:03.354 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus 2024-03-23T03:36:03.354 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-23T03:36:03.354 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-23T03:36:03.354 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-23T03:36:03.354 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-23T03:36:03.354 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-23T03:36:03.354 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-23T03:36:03.354 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-23T03:36:03.354 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-23T03:36:03.354 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-23T03:36:03.354 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-23T03:36:03.354 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-23T03:36:03.355 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-23T03:36:03.355 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-23T03:36:03.355 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-23T03:36:03.355 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-23T03:36:03.355 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-23T03:36:03.355 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-23T03:36:03.355 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-23T03:36:03.355 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-23T03:36:03.355 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-23T03:36:03.355 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-23T03:36:03.355 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-23T03:36:03.355 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-23T03:36:03.355 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-23T03:36:03.355 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-23T03:36:03.355 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-23T03:36:03.356 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-23T03:36:03.356 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-23T03:36:03.356 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-23T03:36:03.356 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-23T03:36:03.356 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-23T03:36:03.356 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-23T03:36:03.356 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-23T03:36:03.356 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-23T03:36:03.356 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-23T03:36:03.356 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-23T03:36:03.356 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-23T03:36:03.356 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-23T03:36:03.356 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-23T03:36:03.356 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-23T03:36:03.357 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-23T03:36:03.357 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c 2024-03-23T03:36:03.357 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos 2024-03-23T03:36:03.357 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-23T03:36:03.357 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-23T03:36:03.357 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses 2024-03-23T03:36:03.357 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-23T03:36:03.357 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-23T03:36:03.357 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-23T03:36:03.357 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-23T03:36:03.357 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-23T03:36:03.357 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-23T03:36:03.357 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-23T03:36:03.357 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-23T03:36:03.357 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-23T03:36:03.358 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-23T03:36:03.358 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-23T03:36:03.358 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-23T03:36:03.358 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-23T03:36:03.358 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-23T03:36:03.358 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-23T03:36:03.358 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-23T03:36:03.358 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-23T03:36:03.358 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-23T03:36:03.358 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-23T03:36:03.358 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-23T03:36:03.358 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-23T03:36:03.358 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-23T03:36:03.358 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-23T03:36:03.359 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-dev.ko 2024-03-23T03:36:03.359 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-23T03:36:03.359 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-23T03:36:03.359 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-23T03:36:03.359 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-23T03:36:03.359 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband 2024-03-23T03:36:03.359 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core 2024-03-23T03:36:03.359 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-23T03:36:03.359 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-23T03:36:03.359 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-23T03:36:03.359 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-23T03:36:03.359 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-23T03:36:03.359 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-23T03:36:03.359 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-23T03:36:03.359 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw 2024-03-23T03:36:03.360 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4 2024-03-23T03:36:03.360 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-23T03:36:03.360 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4 2024-03-23T03:36:03.360 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-23T03:36:03.360 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5 2024-03-23T03:36:03.360 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-23T03:36:03.360 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca 2024-03-23T03:36:03.360 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-23T03:36:03.360 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp 2024-03-23T03:36:03.360 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib 2024-03-23T03:36:03.360 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-23T03:36:03.360 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser 2024-03-23T03:36:03.360 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-23T03:36:03.360 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert 2024-03-23T03:36:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-23T03:36:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp 2024-03-23T03:36:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-23T03:36:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt 2024-03-23T03:36:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-23T03:36:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input 2024-03-23T03:36:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport 2024-03-23T03:36:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-23T03:36:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-23T03:36:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-23T03:36:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-23T03:36:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard 2024-03-23T03:36:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-23T03:36:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse 2024-03-23T03:36:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-23T03:36:03.361 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-23T03:36:03.362 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-23T03:36:03.362 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-23T03:36:03.362 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-23T03:36:03.362 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-23T03:36:03.362 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-23T03:36:03.362 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-23T03:36:03.362 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4 2024-03-23T03:36:03.362 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-23T03:36:03.362 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio 2024-03-23T03:36:03.362 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-23T03:36:03.362 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-23T03:36:03.362 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-23T03:36:03.362 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-23T03:36:03.362 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-23T03:36:03.363 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds 2024-03-23T03:36:03.363 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-23T03:36:03.363 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-23T03:36:03.363 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-23T03:36:03.363 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-23T03:36:03.363 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger 2024-03-23T03:36:03.363 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-23T03:36:03.363 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-23T03:36:03.363 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-23T03:36:03.363 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-23T03:36:03.363 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-23T03:36:03.363 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-23T03:36:03.363 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-23T03:36:03.363 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-23T03:36:03.363 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-23T03:36:03.364 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md 2024-03-23T03:36:03.364 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache 2024-03-23T03:36:03.364 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-23T03:36:03.364 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-23T03:36:03.364 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-23T03:36:03.364 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-23T03:36:03.364 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-23T03:36:03.364 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-23T03:36:03.364 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-23T03:36:03.364 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-23T03:36:03.364 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-23T03:36:03.364 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-23T03:36:03.364 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-23T03:36:03.364 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-23T03:36:03.364 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-23T03:36:03.365 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-23T03:36:03.365 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-23T03:36:03.365 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-23T03:36:03.365 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data 2024-03-23T03:36:03.365 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-23T03:36:03.365 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-23T03:36:03.365 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-23T03:36:03.365 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-23T03:36:03.365 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-23T03:36:03.365 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message 2024-03-23T03:36:03.365 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion 2024-03-23T03:36:03.365 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-23T03:36:03.365 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-23T03:36:03.365 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-23T03:36:03.365 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-23T03:36:03.366 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-23T03:36:03.366 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-23T03:36:03.366 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-23T03:36:03.366 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd 2024-03-23T03:36:03.366 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-23T03:36:03.366 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-23T03:36:03.366 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mfd-core.ko 2024-03-23T03:36:03.366 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-23T03:36:03.366 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-23T03:36:03.366 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-23T03:36:03.366 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-23T03:36:03.366 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc 2024-03-23T03:36:03.366 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl 2024-03-23T03:36:03.366 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-23T03:36:03.366 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-23T03:36:03.367 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-23T03:36:03.367 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-23T03:36:03.367 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710 2024-03-23T03:36:03.367 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-23T03:36:03.367 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom 2024-03-23T03:36:03.367 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-23T03:36:03.367 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-23T03:36:03.367 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-23T03:36:03.367 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-23T03:36:03.367 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-23T03:36:03.367 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-23T03:36:03.367 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-23T03:36:03.367 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d 2024-03-23T03:36:03.367 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-23T03:36:03.367 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-23T03:36:03.367 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei 2024-03-23T03:36:03.368 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-23T03:36:03.368 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-23T03:36:03.368 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-23T03:36:03.368 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru 2024-03-23T03:36:03.368 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-23T03:36:03.368 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp 2024-03-23T03:36:03.368 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-23T03:36:03.368 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-23T03:36:03.368 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-23T03:36:03.368 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-23T03:36:03.368 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-23T03:36:03.368 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-23T03:36:03.368 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-23T03:36:03.368 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci 2024-03-23T03:36:03.369 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-23T03:36:03.369 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net 2024-03-23T03:36:03.369 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding 2024-03-23T03:36:03.369 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-23T03:36:03.369 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa 2024-03-23T03:36:03.369 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53 2024-03-23T03:36:03.369 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-23T03:36:03.369 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-23T03:36:03.369 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-23T03:36:03.369 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx 2024-03-23T03:36:03.369 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-23T03:36:03.369 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-23T03:36:03.369 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-23T03:36:03.369 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet 2024-03-23T03:36:03.369 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com 2024-03-23T03:36:03.370 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-23T03:36:03.370 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-23T03:36:03.370 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-23T03:36:03.370 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-23T03:36:03.370 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390 2024-03-23T03:36:03.370 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-23T03:36:03.370 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-23T03:36:03.370 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-23T03:36:03.370 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-23T03:36:03.370 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec 2024-03-23T03:36:03.370 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-23T03:36:03.370 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere 2024-03-23T03:36:03.370 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-23T03:36:03.370 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon 2024-03-23T03:36:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-23T03:36:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera 2024-03-23T03:36:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-23T03:36:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd 2024-03-23T03:36:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-23T03:36:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-23T03:36:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-23T03:36:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros 2024-03-23T03:36:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx 2024-03-23T03:36:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-23T03:36:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-23T03:36:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-23T03:36:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-23T03:36:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-23T03:36:03.371 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx 2024-03-23T03:36:03.372 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-23T03:36:03.372 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-23T03:36:03.372 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom 2024-03-23T03:36:03.372 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-23T03:36:03.372 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-23T03:36:03.372 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-23T03:36:03.372 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-23T03:36:03.372 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-23T03:36:03.372 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet 2024-03-23T03:36:03.372 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-23T03:36:03.372 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-23T03:36:03.372 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade 2024-03-23T03:36:03.372 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna 2024-03-23T03:36:03.372 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-23T03:36:03.372 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio 2024-03-23T03:36:03.372 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-23T03:36:03.373 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-23T03:36:03.373 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-23T03:36:03.373 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-23T03:36:03.373 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-23T03:36:03.373 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-23T03:36:03.373 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-23T03:36:03.373 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-23T03:36:03.373 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-23T03:36:03.373 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-23T03:36:03.373 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco 2024-03-23T03:36:03.373 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic 2024-03-23T03:36:03.373 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-23T03:36:03.373 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec 2024-03-23T03:36:03.373 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip 2024-03-23T03:36:03.373 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-23T03:36:03.374 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-23T03:36:03.374 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-23T03:36:03.374 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-23T03:36:03.374 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-23T03:36:03.374 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-23T03:36:03.374 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink 2024-03-23T03:36:03.374 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-23T03:36:03.374 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-23T03:36:03.374 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-23T03:36:03.374 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex 2024-03-23T03:36:03.374 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet 2024-03-23T03:36:03.374 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-23T03:36:03.374 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-23T03:36:03.374 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-23T03:36:03.375 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel 2024-03-23T03:36:03.375 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-23T03:36:03.375 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000 2024-03-23T03:36:03.375 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-23T03:36:03.375 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e 2024-03-23T03:36:03.375 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-23T03:36:03.375 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k 2024-03-23T03:36:03.375 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-23T03:36:03.375 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e 2024-03-23T03:36:03.375 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-23T03:36:03.375 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf 2024-03-23T03:36:03.375 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-23T03:36:03.375 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb 2024-03-23T03:36:03.375 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-23T03:36:03.376 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf 2024-03-23T03:36:03.376 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-23T03:36:03.376 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-23T03:36:03.376 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-23T03:36:03.376 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-23T03:36:03.376 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-23T03:36:03.376 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-23T03:36:03.376 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell 2024-03-23T03:36:03.376 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-23T03:36:03.376 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-23T03:36:03.376 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-23T03:36:03.376 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox 2024-03-23T03:36:03.376 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-23T03:36:03.376 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-23T03:36:03.376 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-23T03:36:03.376 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-23T03:36:03.377 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-23T03:36:03.377 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-23T03:36:03.377 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel 2024-03-23T03:36:03.377 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-23T03:36:03.377 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom 2024-03-23T03:36:03.377 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-23T03:36:03.377 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-23T03:36:03.377 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi 2024-03-23T03:36:03.377 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-23T03:36:03.377 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-23T03:36:03.377 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion 2024-03-23T03:36:03.377 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-23T03:36:03.377 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia 2024-03-23T03:36:03.377 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-23T03:36:03.377 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines 2024-03-23T03:36:03.377 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-23T03:36:03.378 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-23T03:36:03.378 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic 2024-03-23T03:36:03.378 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-23T03:36:03.378 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-23T03:36:03.378 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-23T03:36:03.378 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-23T03:36:03.378 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-23T03:36:03.378 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc 2024-03-23T03:36:03.378 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-23T03:36:03.378 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek 2024-03-23T03:36:03.378 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-23T03:36:03.378 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-23T03:36:03.378 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-23T03:36:03.378 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-23T03:36:03.379 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker 2024-03-23T03:36:03.379 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-23T03:36:03.379 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc 2024-03-23T03:36:03.379 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-23T03:36:03.379 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan 2024-03-23T03:36:03.379 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-23T03:36:03.379 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis 2024-03-23T03:36:03.379 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-23T03:36:03.379 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-23T03:36:03.379 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc 2024-03-23T03:36:03.379 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-23T03:36:03.379 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-23T03:36:03.379 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-23T03:36:03.379 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-23T03:36:03.379 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro 2024-03-23T03:36:03.380 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-23T03:36:03.380 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-23T03:36:03.380 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-23T03:36:03.380 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun 2024-03-23T03:36:03.380 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-23T03:36:03.380 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-23T03:36:03.380 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-23T03:36:03.380 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-23T03:36:03.380 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti 2024-03-23T03:36:03.380 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-23T03:36:03.380 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti 2024-03-23T03:36:03.380 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-23T03:36:03.380 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via 2024-03-23T03:36:03.380 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-23T03:36:03.380 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-23T03:36:03.380 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet 2024-03-23T03:36:03.381 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-23T03:36:03.381 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-23T03:36:03.381 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom 2024-03-23T03:36:03.381 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-23T03:36:03.381 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-23T03:36:03.381 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv 2024-03-23T03:36:03.381 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-23T03:36:03.381 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154 2024-03-23T03:36:03.381 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-23T03:36:03.381 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-23T03:36:03.381 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan 2024-03-23T03:36:03.381 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-23T03:36:03.381 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-23T03:36:03.381 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-23T03:36:03.381 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio 2024-03-23T03:36:03.381 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-23T03:36:03.382 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-23T03:36:03.382 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-23T03:36:03.382 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-23T03:36:03.382 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-23T03:36:03.382 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/net_failover.ko 2024-03-23T03:36:03.382 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-23T03:36:03.382 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-23T03:36:03.382 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs 2024-03-23T03:36:03.382 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-23T03:36:03.382 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-23T03:36:03.382 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy 2024-03-23T03:36:03.382 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-23T03:36:03.382 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-23T03:36:03.382 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-23T03:36:03.383 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-23T03:36:03.383 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-23T03:36:03.383 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-23T03:36:03.383 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-23T03:36:03.383 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-23T03:36:03.383 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/dp83640.ko 2024-03-23T03:36:03.383 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-23T03:36:03.383 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-23T03:36:03.383 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-23T03:36:03.383 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-23T03:36:03.383 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-23T03:36:03.383 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-23T03:36:03.383 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-23T03:36:03.383 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-23T03:36:03.383 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-23T03:36:03.383 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-23T03:36:03.384 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-23T03:36:03.384 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-23T03:36:03.384 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp 2024-03-23T03:36:03.384 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-23T03:36:03.384 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-23T03:36:03.384 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-23T03:36:03.384 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-23T03:36:03.384 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-23T03:36:03.384 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-23T03:36:03.384 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-23T03:36:03.384 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-23T03:36:03.384 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-23T03:36:03.384 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip 2024-03-23T03:36:03.384 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slhc.ko 2024-03-23T03:36:03.384 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-23T03:36:03.385 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-23T03:36:03.385 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-23T03:36:03.385 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team 2024-03-23T03:36:03.385 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-23T03:36:03.385 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-23T03:36:03.385 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-23T03:36:03.385 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-23T03:36:03.385 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-23T03:36:03.385 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-23T03:36:03.385 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tun.ko 2024-03-23T03:36:03.385 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-23T03:36:03.385 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/virtio_net.ko 2024-03-23T03:36:03.385 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3 2024-03-23T03:36:03.385 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-23T03:36:03.386 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan 2024-03-23T03:36:03.386 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-23T03:36:03.386 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback 2024-03-23T03:36:03.386 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-23T03:36:03.386 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netfront.ko 2024-03-23T03:36:03.386 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb 2024-03-23T03:36:03.386 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-23T03:36:03.386 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme 2024-03-23T03:36:03.386 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host 2024-03-23T03:36:03.386 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-23T03:36:03.386 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-23T03:36:03.386 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport 2024-03-23T03:36:03.386 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-23T03:36:03.387 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-23T03:36:03.387 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-23T03:36:03.387 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-23T03:36:03.387 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci 2024-03-23T03:36:03.387 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug 2024-03-23T03:36:03.387 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-23T03:36:03.387 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie 2024-03-23T03:36:03.387 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-23T03:36:03.387 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-23T03:36:03.387 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia 2024-03-23T03:36:03.387 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-23T03:36:03.387 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-23T03:36:03.387 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-23T03:36:03.387 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl 2024-03-23T03:36:03.388 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel 2024-03-23T03:36:03.388 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-23T03:36:03.388 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform 2024-03-23T03:36:03.388 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86 2024-03-23T03:36:03.388 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-23T03:36:03.388 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-23T03:36:03.388 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-23T03:36:03.388 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-23T03:36:03.388 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-23T03:36:03.388 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-23T03:36:03.388 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-23T03:36:03.388 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-23T03:36:03.388 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-23T03:36:03.388 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-23T03:36:03.388 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-23T03:36:03.389 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-23T03:36:03.389 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel 2024-03-23T03:36:03.389 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-23T03:36:03.389 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-23T03:36:03.389 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-23T03:36:03.389 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-23T03:36:03.389 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-23T03:36:03.389 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-23T03:36:03.389 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-23T03:36:03.389 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-23T03:36:03.389 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-23T03:36:03.389 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-23T03:36:03.389 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-23T03:36:03.389 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-23T03:36:03.389 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap 2024-03-23T03:36:03.390 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-23T03:36:03.390 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-23T03:36:03.390 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps 2024-03-23T03:36:03.390 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients 2024-03-23T03:36:03.390 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-23T03:36:03.390 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-23T03:36:03.390 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-23T03:36:03.390 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-23T03:36:03.390 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp 2024-03-23T03:36:03.390 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-23T03:36:03.390 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-23T03:36:03.390 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc 2024-03-23T03:36:03.390 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-23T03:36:03.391 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-23T03:36:03.391 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-23T03:36:03.391 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-23T03:36:03.391 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-23T03:36:03.391 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-23T03:36:03.391 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-23T03:36:03.391 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-23T03:36:03.391 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-23T03:36:03.391 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-23T03:36:03.391 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-23T03:36:03.391 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-23T03:36:03.391 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-23T03:36:03.391 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-23T03:36:03.391 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-23T03:36:03.392 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-23T03:36:03.392 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-23T03:36:03.392 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-23T03:36:03.392 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-23T03:36:03.392 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-23T03:36:03.392 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-23T03:36:03.392 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-23T03:36:03.392 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-23T03:36:03.392 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-23T03:36:03.392 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-23T03:36:03.392 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-23T03:36:03.392 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-23T03:36:03.392 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-23T03:36:03.392 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-23T03:36:03.392 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-23T03:36:03.392 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-23T03:36:03.393 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi 2024-03-23T03:36:03.393 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-23T03:36:03.393 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-23T03:36:03.393 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-23T03:36:03.393 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-23T03:36:03.393 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-23T03:36:03.393 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid 2024-03-23T03:36:03.393 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-23T03:36:03.393 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-23T03:36:03.393 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx 2024-03-23T03:36:03.393 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-23T03:36:03.393 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-23T03:36:03.393 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-23T03:36:03.393 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr 2024-03-23T03:36:03.393 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-23T03:36:03.394 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-23T03:36:03.394 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi 2024-03-23T03:36:03.394 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-23T03:36:03.394 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa 2024-03-23T03:36:03.394 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-23T03:36:03.394 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc 2024-03-23T03:36:03.394 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-23T03:36:03.394 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i 2024-03-23T03:36:03.394 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-23T03:36:03.394 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-23T03:36:03.394 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor 2024-03-23T03:36:03.394 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-23T03:36:03.394 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi 2024-03-23T03:36:03.394 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-23T03:36:03.395 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-23T03:36:03.395 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-23T03:36:03.395 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-23T03:36:03.395 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-23T03:36:03.395 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-23T03:36:03.395 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler 2024-03-23T03:36:03.395 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-23T03:36:03.395 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-23T03:36:03.395 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-23T03:36:03.395 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-23T03:36:03.395 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-23T03:36:03.395 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r 2024-03-23T03:36:03.395 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-23T03:36:03.395 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-23T03:36:03.396 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe 2024-03-23T03:36:03.396 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-23T03:36:03.396 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-23T03:36:03.396 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic 2024-03-23T03:36:03.396 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-23T03:36:03.396 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-23T03:36:03.396 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-23T03:36:03.396 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-23T03:36:03.396 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-23T03:36:03.396 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-23T03:36:03.396 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-23T03:36:03.396 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci 2024-03-23T03:36:03.396 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-23T03:36:03.396 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-23T03:36:03.396 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-23T03:36:03.396 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc 2024-03-23T03:36:03.397 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-23T03:36:03.397 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-23T03:36:03.397 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-23T03:36:03.397 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas 2024-03-23T03:36:03.397 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-23T03:36:03.397 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc 2024-03-23T03:36:03.397 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-23T03:36:03.397 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid 2024-03-23T03:36:03.397 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-23T03:36:03.397 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-23T03:36:03.397 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-23T03:36:03.397 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-23T03:36:03.397 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas 2024-03-23T03:36:03.397 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-23T03:36:03.397 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas 2024-03-23T03:36:03.397 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-23T03:36:03.398 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-23T03:36:03.398 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001 2024-03-23T03:36:03.398 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-23T03:36:03.398 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-23T03:36:03.398 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-23T03:36:03.398 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx 2024-03-23T03:36:03.398 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-23T03:36:03.398 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-23T03:36:03.398 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx 2024-03-23T03:36:03.398 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-23T03:36:03.398 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-23T03:36:03.398 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-23T03:36:03.398 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-23T03:36:03.398 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-23T03:36:03.398 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-23T03:36:03.399 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-23T03:36:03.399 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-23T03:36:03.399 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-23T03:36:03.399 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-23T03:36:03.399 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-23T03:36:03.399 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2 2024-03-23T03:36:03.399 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-23T03:36:03.399 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-23T03:36:03.399 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-23T03:36:03.399 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-23T03:36:03.399 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb 2024-03-23T03:36:03.399 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-23T03:36:03.399 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target 2024-03-23T03:36:03.399 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi 2024-03-23T03:36:03.400 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-23T03:36:03.400 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback 2024-03-23T03:36:03.400 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-23T03:36:03.400 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-23T03:36:03.400 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-23T03:36:03.400 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-23T03:36:03.400 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-23T03:36:03.400 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-23T03:36:03.400 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc 2024-03-23T03:36:03.400 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-23T03:36:03.400 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal 2024-03-23T03:36:03.400 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel 2024-03-23T03:36:03.400 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal 2024-03-23T03:36:03.400 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-23T03:36:03.400 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-23T03:36:03.401 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-23T03:36:03.401 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-23T03:36:03.401 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-23T03:36:03.401 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-23T03:36:03.401 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-23T03:36:03.401 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-23T03:36:03.401 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-23T03:36:03.401 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-23T03:36:03.401 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-23T03:36:03.401 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-23T03:36:03.401 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-23T03:36:03.401 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-23T03:36:03.401 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-23T03:36:03.401 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-23T03:36:03.401 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-23T03:36:03.401 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-23T03:36:03.402 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty 2024-03-23T03:36:03.402 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless 2024-03-23T03:36:03.402 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-23T03:36:03.402 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-23T03:36:03.402 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-23T03:36:03.402 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-23T03:36:03.402 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial 2024-03-23T03:36:03.402 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250 2024-03-23T03:36:03.402 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-23T03:36:03.402 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-23T03:36:03.402 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm 2024-03-23T03:36:03.402 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-23T03:36:03.402 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs 2024-03-23T03:36:03.402 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core 2024-03-23T03:36:03.402 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-23T03:36:03.402 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host 2024-03-23T03:36:03.403 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-23T03:36:03.403 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio 2024-03-23T03:36:03.403 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-23T03:36:03.403 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-23T03:36:03.403 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-23T03:36:03.403 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-23T03:36:03.403 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-23T03:36:03.403 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb 2024-03-23T03:36:03.403 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class 2024-03-23T03:36:03.403 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-23T03:36:03.403 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-23T03:36:03.403 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-23T03:36:03.403 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-23T03:36:03.403 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host 2024-03-23T03:36:03.403 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-23T03:36:03.403 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image 2024-03-23T03:36:03.404 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-23T03:36:03.404 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-23T03:36:03.404 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc 2024-03-23T03:36:03.404 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-23T03:36:03.404 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-23T03:36:03.404 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-23T03:36:03.404 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-23T03:36:03.404 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-23T03:36:03.404 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-23T03:36:03.404 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-23T03:36:03.404 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-23T03:36:03.404 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-23T03:36:03.404 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-23T03:36:03.404 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga 2024-03-23T03:36:03.404 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-23T03:36:03.404 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-23T03:36:03.405 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-23T03:36:03.405 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-23T03:36:03.405 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-23T03:36:03.405 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-23T03:36:03.405 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-23T03:36:03.405 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy 2024-03-23T03:36:03.405 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-23T03:36:03.405 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial 2024-03-23T03:36:03.405 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-23T03:36:03.405 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-23T03:36:03.405 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-23T03:36:03.405 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-23T03:36:03.405 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-23T03:36:03.405 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-23T03:36:03.405 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-23T03:36:03.406 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-23T03:36:03.406 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-23T03:36:03.406 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-23T03:36:03.406 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-23T03:36:03.406 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-23T03:36:03.406 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-23T03:36:03.406 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-23T03:36:03.406 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-23T03:36:03.406 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-23T03:36:03.406 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-23T03:36:03.406 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-23T03:36:03.406 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-23T03:36:03.406 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-23T03:36:03.406 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-23T03:36:03.406 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-23T03:36:03.406 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-23T03:36:03.407 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-23T03:36:03.407 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-23T03:36:03.407 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-23T03:36:03.407 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-23T03:36:03.407 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-23T03:36:03.407 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-23T03:36:03.407 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-23T03:36:03.407 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-23T03:36:03.407 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-23T03:36:03.407 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-23T03:36:03.407 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-23T03:36:03.407 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-23T03:36:03.407 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-23T03:36:03.407 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-23T03:36:03.407 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-23T03:36:03.407 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-23T03:36:03.408 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-23T03:36:03.408 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-23T03:36:03.408 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-23T03:36:03.408 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-23T03:36:03.408 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-23T03:36:03.408 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-23T03:36:03.408 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage 2024-03-23T03:36:03.408 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-23T03:36:03.408 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-23T03:36:03.408 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-23T03:36:03.408 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-23T03:36:03.408 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-23T03:36:03.408 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-23T03:36:03.408 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-23T03:36:03.408 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-23T03:36:03.409 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-23T03:36:03.409 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-23T03:36:03.409 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-23T03:36:03.409 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-23T03:36:03.409 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-23T03:36:03.409 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-23T03:36:03.409 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-23T03:36:03.409 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip 2024-03-23T03:36:03.409 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-23T03:36:03.409 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-23T03:36:03.409 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-23T03:36:03.409 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio 2024-03-23T03:36:03.409 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci 2024-03-23T03:36:03.409 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-23T03:36:03.409 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-23T03:36:03.409 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-23T03:36:03.410 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-23T03:36:03.410 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost 2024-03-23T03:36:03.410 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-23T03:36:03.410 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-23T03:36:03.410 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-23T03:36:03.410 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-23T03:36:03.410 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video 2024-03-23T03:36:03.410 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight 2024-03-23T03:36:03.410 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-23T03:36:03.410 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-23T03:36:03.410 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-23T03:36:03.410 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio 2024-03-23T03:36:03.410 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio.ko 2024-03-23T03:36:03.410 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_balloon.ko 2024-03-23T03:36:03.411 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_mmio.ko 2024-03-23T03:36:03.411 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci.ko 2024-03-23T03:36:03.411 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-23T03:36:03.411 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-23T03:36:03.411 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_ring.ko 2024-03-23T03:36:03.411 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1 2024-03-23T03:36:03.411 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters 2024-03-23T03:36:03.411 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-23T03:36:03.411 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-23T03:36:03.411 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves 2024-03-23T03:36:03.411 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-23T03:36:03.411 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-23T03:36:03.411 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-23T03:36:03.411 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-23T03:36:03.411 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-23T03:36:03.411 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-23T03:36:03.412 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-23T03:36:03.412 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-23T03:36:03.412 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-23T03:36:03.412 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-23T03:36:03.412 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-23T03:36:03.412 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-23T03:36:03.412 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog 2024-03-23T03:36:03.412 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-23T03:36:03.412 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-23T03:36:03.412 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-23T03:36:03.412 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-23T03:36:03.412 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-23T03:36:03.412 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-23T03:36:03.412 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-23T03:36:03.412 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-23T03:36:03.412 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-23T03:36:03.413 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-23T03:36:03.413 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-23T03:36:03.413 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-23T03:36:03.413 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-23T03:36:03.413 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-23T03:36:03.413 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-23T03:36:03.413 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-23T03:36:03.413 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-23T03:36:03.413 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-23T03:36:03.413 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-23T03:36:03.413 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-23T03:36:03.413 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-23T03:36:03.413 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-23T03:36:03.413 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-23T03:36:03.413 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-23T03:36:03.413 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-23T03:36:03.414 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-23T03:36:03.414 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen 2024-03-23T03:36:03.414 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-23T03:36:03.414 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-23T03:36:03.414 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-23T03:36:03.414 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-23T03:36:03.414 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback 2024-03-23T03:36:03.414 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-23T03:36:03.414 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-23T03:36:03.414 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs 2024-03-23T03:36:03.414 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-23T03:36:03.414 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs 2024-03-23T03:36:03.414 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p 2024-03-23T03:36:03.414 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-23T03:36:03.414 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs 2024-03-23T03:36:03.414 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-23T03:36:03.415 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs 2024-03-23T03:36:03.415 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-23T03:36:03.415 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-23T03:36:03.415 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs 2024-03-23T03:36:03.415 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-23T03:36:03.415 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles 2024-03-23T03:36:03.415 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-23T03:36:03.415 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph 2024-03-23T03:36:03.415 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-23T03:36:03.415 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda 2024-03-23T03:36:03.415 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-23T03:36:03.415 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs 2024-03-23T03:36:03.415 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-23T03:36:03.415 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm 2024-03-23T03:36:03.415 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-23T03:36:03.416 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs 2024-03-23T03:36:03.416 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-23T03:36:03.416 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs 2024-03-23T03:36:03.416 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-23T03:36:03.416 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat 2024-03-23T03:36:03.416 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/fat.ko 2024-03-23T03:36:03.416 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-23T03:36:03.416 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/vfat.ko 2024-03-23T03:36:03.416 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse 2024-03-23T03:36:03.416 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-23T03:36:03.416 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/fuse.ko 2024-03-23T03:36:03.416 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2 2024-03-23T03:36:03.416 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-23T03:36:03.416 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs 2024-03-23T03:36:03.416 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-23T03:36:03.416 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus 2024-03-23T03:36:03.417 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-23T03:36:03.417 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs 2024-03-23T03:36:03.417 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-23T03:36:03.417 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs 2024-03-23T03:36:03.417 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-23T03:36:03.417 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd 2024-03-23T03:36:03.417 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-23T03:36:03.417 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix 2024-03-23T03:36:03.417 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-23T03:36:03.417 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs 2024-03-23T03:36:03.417 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-23T03:36:03.417 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs 2024-03-23T03:36:03.417 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout 2024-03-23T03:36:03.417 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-23T03:36:03.417 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout 2024-03-23T03:36:03.417 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-23T03:36:03.418 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout 2024-03-23T03:36:03.418 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-23T03:36:03.418 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-23T03:36:03.418 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-23T03:36:03.418 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-23T03:36:03.418 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common 2024-03-23T03:36:03.418 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-23T03:36:03.418 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-23T03:36:03.418 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd 2024-03-23T03:36:03.418 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-23T03:36:03.418 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2 2024-03-23T03:36:03.418 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-23T03:36:03.418 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls 2024-03-23T03:36:03.419 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-23T03:36:03.419 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-23T03:36:03.419 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-23T03:36:03.419 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-23T03:36:03.419 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-23T03:36:03.419 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-23T03:36:03.419 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-23T03:36:03.419 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-23T03:36:03.419 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-23T03:36:03.419 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-23T03:36:03.419 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-23T03:36:03.419 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-23T03:36:03.419 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-23T03:36:03.419 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-23T03:36:03.419 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-23T03:36:03.419 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-23T03:36:03.419 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-23T03:36:03.420 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-23T03:36:03.420 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-23T03:36:03.420 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-23T03:36:03.420 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-23T03:36:03.420 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-23T03:36:03.420 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-23T03:36:03.420 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-23T03:36:03.420 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-23T03:36:03.420 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-23T03:36:03.420 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-23T03:36:03.420 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-23T03:36:03.420 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-23T03:36:03.420 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-23T03:36:03.420 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-23T03:36:03.420 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-23T03:36:03.421 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-23T03:36:03.421 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-23T03:36:03.421 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-23T03:36:03.421 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-23T03:36:03.421 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-23T03:36:03.421 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-23T03:36:03.421 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-23T03:36:03.421 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-23T03:36:03.421 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-23T03:36:03.421 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-23T03:36:03.421 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-23T03:36:03.421 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-23T03:36:03.421 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-23T03:36:03.421 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-23T03:36:03.421 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-23T03:36:03.421 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-23T03:36:03.422 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-23T03:36:03.422 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-23T03:36:03.422 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2 2024-03-23T03:36:03.422 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster 2024-03-23T03:36:03.422 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-23T03:36:03.422 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm 2024-03-23T03:36:03.422 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-23T03:36:03.422 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs 2024-03-23T03:36:03.422 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-23T03:36:03.422 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-23T03:36:03.422 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-23T03:36:03.422 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-23T03:36:03.422 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-23T03:36:03.422 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs 2024-03-23T03:36:03.422 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-23T03:36:03.423 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore 2024-03-23T03:36:03.423 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-23T03:36:03.423 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs 2024-03-23T03:36:03.423 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-23T03:36:03.423 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs 2024-03-23T03:36:03.423 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-23T03:36:03.423 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb 2024-03-23T03:36:03.423 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client 2024-03-23T03:36:03.423 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-23T03:36:03.423 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common 2024-03-23T03:36:03.423 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-23T03:36:03.423 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-23T03:36:03.423 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs 2024-03-23T03:36:03.423 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-23T03:36:03.423 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv 2024-03-23T03:36:03.423 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-23T03:36:03.424 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf 2024-03-23T03:36:03.424 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-23T03:36:03.424 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs 2024-03-23T03:36:03.424 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-23T03:36:03.424 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs 2024-03-23T03:36:03.424 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-23T03:36:03.424 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel 2024-03-23T03:36:03.424 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu 2024-03-23T03:36:03.424 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu/rcutorture.ko 2024-03-23T03:36:03.424 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-23T03:36:03.424 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace 2024-03-23T03:36:03.424 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-23T03:36:03.424 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib 2024-03-23T03:36:03.424 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/asn1_encoder.ko 2024-03-23T03:36:03.424 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-23T03:36:03.424 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-23T03:36:03.425 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto 2024-03-23T03:36:03.425 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-23T03:36:03.425 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-23T03:36:03.425 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-23T03:36:03.425 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-23T03:36:03.425 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4 2024-03-23T03:36:03.425 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-23T03:36:03.425 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-23T03:36:03.425 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math 2024-03-23T03:36:03.425 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-23T03:36:03.425 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6 2024-03-23T03:36:03.425 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-23T03:36:03.425 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon 2024-03-23T03:36:03.425 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-23T03:36:03.425 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-23T03:36:03.425 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-23T03:36:03.426 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-23T03:36:03.426 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd 2024-03-23T03:36:03.426 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd/zstd_compress.ko 2024-03-23T03:36:03.426 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm 2024-03-23T03:36:03.426 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-23T03:36:03.426 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net 2024-03-23T03:36:03.426 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan 2024-03-23T03:36:03.426 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-23T03:36:03.426 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-23T03:36:03.426 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-23T03:36:03.426 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-23T03:36:03.426 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-23T03:36:03.426 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-23T03:36:03.426 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-23T03:36:03.427 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-23T03:36:03.427 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802 2024-03-23T03:36:03.427 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-23T03:36:03.427 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-23T03:36:03.427 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-23T03:36:03.427 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-23T03:36:03.427 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-23T03:36:03.427 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q 2024-03-23T03:36:03.427 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-23T03:36:03.427 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p 2024-03-23T03:36:03.427 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-23T03:36:03.427 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-23T03:36:03.427 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-23T03:36:03.427 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-23T03:36:03.427 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv 2024-03-23T03:36:03.427 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-23T03:36:03.428 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge 2024-03-23T03:36:03.428 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-23T03:36:03.428 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-23T03:36:03.428 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter 2024-03-23T03:36:03.428 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-23T03:36:03.428 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-23T03:36:03.428 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-23T03:36:03.428 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-23T03:36:03.428 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-23T03:36:03.428 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-23T03:36:03.428 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-23T03:36:03.428 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-23T03:36:03.428 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-23T03:36:03.428 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-23T03:36:03.428 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-23T03:36:03.428 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-23T03:36:03.429 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-23T03:36:03.429 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-23T03:36:03.429 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-23T03:36:03.429 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-23T03:36:03.429 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-23T03:36:03.429 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-23T03:36:03.429 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-23T03:36:03.429 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-23T03:36:03.429 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-23T03:36:03.429 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-23T03:36:03.429 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-23T03:36:03.429 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-23T03:36:03.429 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph 2024-03-23T03:36:03.429 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-23T03:36:03.429 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core 2024-03-23T03:36:03.429 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/failover.ko 2024-03-23T03:36:03.430 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-23T03:36:03.430 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp 2024-03-23T03:36:03.430 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-23T03:36:03.430 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-23T03:36:03.430 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-23T03:36:03.430 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-23T03:36:03.430 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver 2024-03-23T03:36:03.430 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver/dns_resolver.ko 2024-03-23T03:36:03.430 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa 2024-03-23T03:36:03.430 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-23T03:36:03.430 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-23T03:36:03.430 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-23T03:36:03.430 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-23T03:36:03.430 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-23T03:36:03.430 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154 2024-03-23T03:36:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan 2024-03-23T03:36:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-23T03:36:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-23T03:36:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-23T03:36:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4 2024-03-23T03:36:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-23T03:36:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-23T03:36:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-23T03:36:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-23T03:36:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-23T03:36:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-23T03:36:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-23T03:36:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-23T03:36:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-23T03:36:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-23T03:36:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter 2024-03-23T03:36:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-23T03:36:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-23T03:36:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-23T03:36:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-23T03:36:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-23T03:36:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-23T03:36:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-23T03:36:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-23T03:36:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-23T03:36:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-23T03:36:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-23T03:36:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-23T03:36:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-23T03:36:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-23T03:36:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-23T03:36:03.433 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-23T03:36:03.433 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-23T03:36:03.433 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-23T03:36:03.433 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-23T03:36:03.433 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-23T03:36:03.433 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-23T03:36:03.433 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-23T03:36:03.433 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-23T03:36:03.433 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-23T03:36:03.433 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-23T03:36:03.433 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-23T03:36:03.433 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-23T03:36:03.433 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-23T03:36:03.433 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-23T03:36:03.433 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-23T03:36:03.433 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-23T03:36:03.434 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-23T03:36:03.434 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-23T03:36:03.434 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-23T03:36:03.434 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-23T03:36:03.434 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-23T03:36:03.434 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-23T03:36:03.434 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-23T03:36:03.434 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6 2024-03-23T03:36:03.434 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-23T03:36:03.434 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-23T03:36:03.434 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-23T03:36:03.434 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-23T03:36:03.434 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-23T03:36:03.434 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-23T03:36:03.434 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-23T03:36:03.434 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter 2024-03-23T03:36:03.435 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-23T03:36:03.435 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-23T03:36:03.435 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-23T03:36:03.435 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-23T03:36:03.435 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-23T03:36:03.435 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-23T03:36:03.435 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-23T03:36:03.435 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-23T03:36:03.435 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-23T03:36:03.435 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-23T03:36:03.435 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-23T03:36:03.435 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-23T03:36:03.435 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-23T03:36:03.435 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-23T03:36:03.435 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-23T03:36:03.436 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-23T03:36:03.436 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-23T03:36:03.436 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-23T03:36:03.436 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-23T03:36:03.436 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-23T03:36:03.436 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-23T03:36:03.436 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-23T03:36:03.436 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-23T03:36:03.436 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-23T03:36:03.436 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-23T03:36:03.436 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-23T03:36:03.436 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-23T03:36:03.436 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key 2024-03-23T03:36:03.436 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-23T03:36:03.437 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp 2024-03-23T03:36:03.437 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-23T03:36:03.437 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-23T03:36:03.437 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-23T03:36:03.437 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-23T03:36:03.437 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-23T03:36:03.437 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-23T03:36:03.437 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-23T03:36:03.437 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc 2024-03-23T03:36:03.437 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-23T03:36:03.437 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154 2024-03-23T03:36:03.437 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-23T03:36:03.437 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls 2024-03-23T03:36:03.437 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-23T03:36:03.437 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter 2024-03-23T03:36:03.438 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset 2024-03-23T03:36:03.438 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-23T03:36:03.438 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-23T03:36:03.438 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-23T03:36:03.438 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-23T03:36:03.438 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-23T03:36:03.438 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-23T03:36:03.438 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-23T03:36:03.438 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-23T03:36:03.438 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-23T03:36:03.438 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-23T03:36:03.438 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-23T03:36:03.438 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-23T03:36:03.438 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-23T03:36:03.438 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-23T03:36:03.439 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-23T03:36:03.439 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-23T03:36:03.439 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs 2024-03-23T03:36:03.439 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-23T03:36:03.439 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-23T03:36:03.439 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-23T03:36:03.439 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-23T03:36:03.439 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-23T03:36:03.439 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-23T03:36:03.439 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-23T03:36:03.439 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-23T03:36:03.439 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-23T03:36:03.439 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-23T03:36:03.439 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-23T03:36:03.440 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-23T03:36:03.440 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-23T03:36:03.440 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-23T03:36:03.440 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-23T03:36:03.440 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-23T03:36:03.440 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-23T03:36:03.440 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-23T03:36:03.440 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-23T03:36:03.440 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-23T03:36:03.440 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-23T03:36:03.440 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-23T03:36:03.440 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-23T03:36:03.440 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-23T03:36:03.440 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-23T03:36:03.441 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-23T03:36:03.441 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-23T03:36:03.441 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-23T03:36:03.441 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-23T03:36:03.441 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-23T03:36:03.441 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-23T03:36:03.441 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-23T03:36:03.441 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-23T03:36:03.441 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-23T03:36:03.441 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-23T03:36:03.441 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-23T03:36:03.441 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-23T03:36:03.441 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-23T03:36:03.441 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-23T03:36:03.442 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-23T03:36:03.442 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-23T03:36:03.442 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-23T03:36:03.442 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-23T03:36:03.442 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-23T03:36:03.442 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-23T03:36:03.442 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-23T03:36:03.442 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-23T03:36:03.442 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-23T03:36:03.442 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-23T03:36:03.442 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-23T03:36:03.442 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-23T03:36:03.442 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-23T03:36:03.442 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-23T03:36:03.443 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-23T03:36:03.443 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-23T03:36:03.443 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-23T03:36:03.443 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-23T03:36:03.443 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-23T03:36:03.443 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_quota.ko 2024-03-23T03:36:03.443 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-23T03:36:03.443 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-23T03:36:03.443 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-23T03:36:03.443 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-23T03:36:03.443 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tunnel.ko 2024-03-23T03:36:03.443 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-23T03:36:03.443 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-23T03:36:03.443 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-23T03:36:03.443 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-23T03:36:03.443 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-23T03:36:03.444 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-23T03:36:03.444 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-23T03:36:03.444 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-23T03:36:03.444 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-23T03:36:03.444 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-23T03:36:03.444 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-23T03:36:03.444 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-23T03:36:03.444 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-23T03:36:03.444 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-23T03:36:03.444 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-23T03:36:03.444 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-23T03:36:03.444 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-23T03:36:03.444 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-23T03:36:03.444 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-23T03:36:03.444 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-23T03:36:03.444 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-23T03:36:03.445 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-23T03:36:03.445 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-23T03:36:03.445 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-23T03:36:03.445 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-23T03:36:03.445 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-23T03:36:03.445 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-23T03:36:03.445 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-23T03:36:03.445 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-23T03:36:03.445 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-23T03:36:03.445 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-23T03:36:03.445 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-23T03:36:03.445 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-23T03:36:03.445 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-23T03:36:03.445 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-23T03:36:03.445 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-23T03:36:03.445 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-23T03:36:03.446 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-23T03:36:03.446 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-23T03:36:03.446 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-23T03:36:03.446 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-23T03:36:03.446 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-23T03:36:03.446 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-23T03:36:03.446 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-23T03:36:03.446 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-23T03:36:03.446 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-23T03:36:03.446 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-23T03:36:03.446 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-23T03:36:03.446 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-23T03:36:03.446 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-23T03:36:03.446 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-23T03:36:03.447 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-23T03:36:03.447 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-23T03:36:03.447 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-23T03:36:03.447 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-23T03:36:03.447 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-23T03:36:03.447 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-23T03:36:03.447 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-23T03:36:03.447 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-23T03:36:03.447 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-23T03:36:03.447 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-23T03:36:03.447 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-23T03:36:03.447 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-23T03:36:03.447 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-23T03:36:03.447 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-23T03:36:03.447 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-23T03:36:03.448 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-23T03:36:03.448 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-23T03:36:03.448 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-23T03:36:03.448 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-23T03:36:03.448 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-23T03:36:03.448 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink 2024-03-23T03:36:03.448 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-23T03:36:03.448 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh 2024-03-23T03:36:03.448 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-23T03:36:03.448 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch 2024-03-23T03:36:03.448 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-23T03:36:03.448 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-23T03:36:03.448 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-23T03:36:03.448 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-23T03:36:03.449 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet 2024-03-23T03:36:03.449 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-23T03:36:03.449 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds 2024-03-23T03:36:03.449 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-23T03:36:03.449 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-23T03:36:03.449 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-23T03:36:03.449 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched 2024-03-23T03:36:03.449 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-23T03:36:03.449 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-23T03:36:03.449 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-23T03:36:03.449 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-23T03:36:03.449 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-23T03:36:03.449 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-23T03:36:03.449 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-23T03:36:03.449 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-23T03:36:03.449 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-23T03:36:03.450 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-23T03:36:03.450 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-23T03:36:03.450 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-23T03:36:03.450 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-23T03:36:03.450 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-23T03:36:03.450 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-23T03:36:03.450 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-23T03:36:03.450 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-23T03:36:03.450 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-23T03:36:03.450 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-23T03:36:03.450 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-23T03:36:03.450 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-23T03:36:03.450 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-23T03:36:03.450 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-23T03:36:03.450 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-23T03:36:03.451 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-23T03:36:03.451 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-23T03:36:03.451 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-23T03:36:03.451 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-23T03:36:03.451 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-23T03:36:03.451 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-23T03:36:03.451 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-23T03:36:03.451 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-23T03:36:03.451 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-23T03:36:03.451 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-23T03:36:03.451 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-23T03:36:03.451 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-23T03:36:03.451 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-23T03:36:03.451 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-23T03:36:03.451 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-23T03:36:03.452 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-23T03:36:03.452 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-23T03:36:03.452 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-23T03:36:03.452 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-23T03:36:03.452 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp 2024-03-23T03:36:03.452 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-23T03:36:03.452 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-23T03:36:03.452 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc 2024-03-23T03:36:03.452 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss 2024-03-23T03:36:03.452 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-23T03:36:03.452 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-23T03:36:03.452 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-23T03:36:03.452 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma 2024-03-23T03:36:03.452 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-23T03:36:03.452 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc 2024-03-23T03:36:03.453 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-23T03:36:03.453 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-23T03:36:03.453 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix 2024-03-23T03:36:03.453 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-23T03:36:03.453 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock 2024-03-23T03:36:03.453 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-23T03:36:03.453 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-23T03:36:03.453 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-23T03:36:03.453 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-23T03:36:03.453 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-23T03:36:03.453 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm 2024-03-23T03:36:03.453 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-23T03:36:03.453 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security 2024-03-23T03:36:03.453 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys 2024-03-23T03:36:03.453 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys 2024-03-23T03:36:03.453 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-23T03:36:03.454 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys 2024-03-23T03:36:03.454 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys/trusted.ko 2024-03-23T03:36:03.454 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt 2024-03-23T03:36:03.454 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib 2024-03-23T03:36:03.454 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-23T03:36:03.454 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-23T03:36:03.454 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-23T03:36:03.454 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-23T03:36:03.454 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.alias.bin 2024-03-23T03:36:03.454 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-23T03:36:03.454 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-23T03:36:03.454 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-23T03:36:03.454 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-23T03:36:03.454 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-23T03:36:03.455 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-23T03:36:03.455 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-23T03:36:03.455 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-23T03:36:03.455 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-23T03:36:03.455 INFO:teuthology.orchestra.run.smithi102.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-23T03:36:03.458 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-23T03:36:03.458 DEBUG:teuthology.orchestra.run.smithi102:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-23T03:36:03.480 INFO:teuthology.orchestra.run.smithi102.stdout:postinstall scriptlet (using /bin/sh): 2024-03-23T03:36:03.480 INFO:teuthology.orchestra.run.smithi102.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-23T03:36:03.480 INFO:teuthology.orchestra.run.smithi102.stdout: /usr/bin/kernel-install add 6.8.0-g5ec3e6e88b2e /lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-23T03:36:03.480 INFO:teuthology.orchestra.run.smithi102.stdout:fi 2024-03-23T03:36:03.480 INFO:teuthology.orchestra.run.smithi102.stdout:for file in vmlinuz System.map config; do 2024-03-23T03:36:03.480 INFO:teuthology.orchestra.run.smithi102.stdout: if ! cmp --silent "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e"; then 2024-03-23T03:36:03.480 INFO:teuthology.orchestra.run.smithi102.stdout: cp "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e" 2024-03-23T03:36:03.481 INFO:teuthology.orchestra.run.smithi102.stdout: fi 2024-03-23T03:36:03.481 INFO:teuthology.orchestra.run.smithi102.stdout:done 2024-03-23T03:36:03.481 INFO:teuthology.orchestra.run.smithi102.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-23T03:36:03.481 INFO:teuthology.orchestra.run.smithi102.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-23T03:36:03.481 INFO:teuthology.orchestra.run.smithi102.stdout:new-kernel-pkg --remove 6.8.0-g5ec3e6e88b2e --rminitrd --initrdfile=/boot/initramfs-6.8.0-g5ec3e6e88b2e.img 2024-03-23T03:36:03.481 INFO:teuthology.orchestra.run.smithi102.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-23T03:36:03.481 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-install remove 6.8.0-g5ec3e6e88b2e 2024-03-23T03:36:03.481 INFO:teuthology.orchestra.run.smithi102.stdout:fi 2024-03-23T03:36:03.481 INFO:teuthology.orchestra.run.smithi102.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-23T03:36:03.481 INFO:teuthology.orchestra.run.smithi102.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-23T03:36:03.481 INFO:teuthology.orchestra.run.smithi102.stdout:/sbin/update-bootloader --remove 6.8.0-g5ec3e6e88b2e 2024-03-23T03:36:03.481 INFO:teuthology.orchestra.run.smithi102.stdout:fi 2024-03-23T03:36:03.482 DEBUG:teuthology.orchestra.run.smithi102:> sudo rpm -qi grub2-tools 2024-03-23T03:36:03.562 INFO:teuthology.orchestra.run.smithi102.stdout:Name : grub2-tools 2024-03-23T03:36:03.563 INFO:teuthology.orchestra.run.smithi102.stdout:Epoch : 1 2024-03-23T03:36:03.563 INFO:teuthology.orchestra.run.smithi102.stdout:Version : 2.06 2024-03-23T03:36:03.563 INFO:teuthology.orchestra.run.smithi102.stdout:Release : 68.el9 2024-03-23T03:36:03.563 INFO:teuthology.orchestra.run.smithi102.stdout:Architecture: x86_64 2024-03-23T03:36:03.563 INFO:teuthology.orchestra.run.smithi102.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-23T03:36:03.563 INFO:teuthology.orchestra.run.smithi102.stdout:Group : Unspecified 2024-03-23T03:36:03.563 INFO:teuthology.orchestra.run.smithi102.stdout:Size : 8264254 2024-03-23T03:36:03.563 INFO:teuthology.orchestra.run.smithi102.stdout:License : GPLv3+ 2024-03-23T03:36:03.563 INFO:teuthology.orchestra.run.smithi102.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-23T03:36:03.563 INFO:teuthology.orchestra.run.smithi102.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-23T03:36:03.563 INFO:teuthology.orchestra.run.smithi102.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-23T03:36:03.563 INFO:teuthology.orchestra.run.smithi102.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-23T03:36:03.563 INFO:teuthology.orchestra.run.smithi102.stdout:Packager : builder@centos.org 2024-03-23T03:36:03.563 INFO:teuthology.orchestra.run.smithi102.stdout:Vendor : CentOS 2024-03-23T03:36:03.563 INFO:teuthology.orchestra.run.smithi102.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-23T03:36:03.564 INFO:teuthology.orchestra.run.smithi102.stdout:Summary : Support tools for GRUB. 2024-03-23T03:36:03.564 INFO:teuthology.orchestra.run.smithi102.stdout:Description : 2024-03-23T03:36:03.564 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:36:03.564 INFO:teuthology.orchestra.run.smithi102.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-23T03:36:03.564 INFO:teuthology.orchestra.run.smithi102.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-23T03:36:03.564 INFO:teuthology.orchestra.run.smithi102.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-23T03:36:03.564 INFO:teuthology.orchestra.run.smithi102.stdout:hardware devices. 2024-03-23T03:36:03.564 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:36:03.564 INFO:teuthology.orchestra.run.smithi102.stdout:This subpackage provides tools for support of all platforms. 2024-03-23T03:36:03.565 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-23T03:36:03.565 INFO:teuthology.task.kernel:Updating grub on smithi102 to boot 6.8.0-g5ec3e6e88b2e 2024-03-23T03:36:03.565 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-23T03:36:04.225 INFO:teuthology.orchestra.run.smithi102.stderr:Generating grub configuration file ... 2024-03-23T03:36:05.672 INFO:teuthology.orchestra.run.smithi102.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-23T03:36:05.700 INFO:teuthology.orchestra.run.smithi102.stderr:done 2024-03-23T03:36:05.702 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-03-23T03:36:05.717 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.72XrzG6Rty 2024-03-23T03:36:05.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.72XrzG6Rty 2024-03-23T03:36:05.811 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.72XrzG6Rty 2024-03-23T03:36:05.912 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.72XrzG6Rty is 6KB 2024-03-23T03:36:05.931 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.72XrzG6Rty 2024-03-23T03:36:05.945 DEBUG:teuthology.orchestra.run.smithi102:> sudo /bin/ls /boot/loader/entries || true 2024-03-23T03:36:06.012 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-23T03:36:06.012 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-23T03:36:06.012 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-23T03:36:06.012 INFO:teuthology.orchestra.run.smithi102.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e.conf 2024-03-23T03:36:06.014 DEBUG:teuthology.orchestra.run.smithi102:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e 2024-03-23T03:36:06.175 DEBUG:teuthology.orchestra.run.smithi102:> sudo shutdown -r now 2024-03-23T03:36:08.076 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2024-03-23T03:36:08.076 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-23T03:36:08.077 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T03:36:26.399 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-03-23T03:36:35.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-23T03:36:35.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T03:36:36.179 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-23T03:36:36.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-23T03:36:36.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T03:36:54.558 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-03-23T03:37:03.568 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-23T03:37:03.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T03:37:06.622 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-03-23T03:37:18.624 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-23T03:37:18.625 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T03:37:35.429 DEBUG:teuthology.orchestra.remote:timed out 2024-03-23T03:37:37.054 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.102 2024-03-23T03:37:47.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-23T03:37:47.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T03:37:47.954 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-03-23T03:37:48.629 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-23T03:37:48.629 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.8.0-g5ec3e6e88b2e"... 2024-03-23T03:37:48.629 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-03-23T03:37:48.679 INFO:teuthology.orchestra.run.smithi173.stdout:6.8.0-g5ec3e6e88b2e 2024-03-23T03:37:48.679 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-23T03:37:48.680 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-23T03:37:48.680 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-23T03:37:49.680 DEBUG:teuthology.parallel:result is None 2024-03-23T03:37:52.055 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-23T03:37:52.056 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T03:37:52.431 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-03-23T03:37:53.073 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-23T03:37:53.073 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g5ec3e6e88b2e"... 2024-03-23T03:37:53.073 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-03-23T03:37:53.122 INFO:teuthology.orchestra.run.smithi102.stdout:6.8.0-g5ec3e6e88b2e 2024-03-23T03:37:53.122 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-23T03:37:53.122 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-23T03:37:53.122 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-23T03:37:54.124 DEBUG:teuthology.parallel:result is None 2024-03-23T03:37:54.124 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-23T03:37:54.135 INFO:teuthology.task.internal:Creating test directory... 2024-03-23T03:37:54.135 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T03:37:54.141 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T03:37:54.197 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-23T03:37:54.203 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-23T03:37:54.212 INFO:teuthology.task.internal:Creating archive directory... 2024-03-23T03:37:54.213 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T03:37:54.227 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T03:37:54.295 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-23T03:37:54.302 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-23T03:37:54.302 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T03:37:54.307 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T03:37:54.432 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T03:37:54.445 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T03:37:54.510 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T03:37:54.519 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T03:37:54.524 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-23T03:37:54.531 INFO:teuthology.task.internal:Configuring sudo... 2024-03-23T03:37:54.531 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T03:37:54.535 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T03:37:54.634 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-23T03:37:54.643 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-23T03:37:54.644 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T03:37:54.648 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T03:37:54.700 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T03:37:54.795 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T03:37:54.890 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T03:37:54.957 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T03:37:55.048 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:37:55.048 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T03:37:55.140 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T03:37:55.226 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T03:37:55.333 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T03:37:55.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T03:37:55.513 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:37:55.514 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T03:37:55.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2024-03-23T03:37:55.609 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-03-23T03:37:55.714 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T03:37:55.772 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T03:37:55.949 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-23T03:37:55.957 INFO:teuthology.task.internal:Starting timer... 2024-03-23T03:37:55.957 INFO:teuthology.run_tasks:Running task pcp... 2024-03-23T03:37:55.971 INFO:teuthology.run_tasks:Running task selinux... 2024-03-23T03:37:55.979 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2024-03-23T03:37:56.122 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: 2024-03-23T03:37:56.126 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-03-23T03:37:56.249 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-03-23T03:37:56.254 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-23T03:37:56.254 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2024-03-23T03:37:56.299 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2024-03-23T03:37:56.299 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-03-23T03:37:56.362 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-03-23T03:37:56.363 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi102.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-03-23T03:37:56.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-23T03:37:56.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:37:56.436 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-23T03:37:56.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:37:56.524 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-23T03:37:56.525 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2024-03-23T03:37:56.637 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-03-23T03:37:56.717 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-23T03:37:56.727 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-23T03:37:56.745 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-23T03:37:56.746 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi102.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-23T03:43:49.270 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi102.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-03-23T03:43:49.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-23T03:43:49.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T03:43:49.415 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-03-23T03:43:49.574 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-23T03:43:49.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-23T03:43:49.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T03:43:49.716 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-03-23T03:43:49.869 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-23T03:43:49.870 INFO:teuthology.run_tasks:Running task clock... 2024-03-23T03:43:49.885 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-23T03:43:49.886 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-23T03:43:49.886 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T03:43:49.895 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-23T03:43:49.895 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T03:43:49.983 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-23T03:43:49.996 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-23T03:43:50.034 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-23T03:43:50.049 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-23T03:43:50.126 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2024-03-23T03:43:50.144 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-03-23T03:43:50.171 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2024-03-23T03:43:50.185 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-03-23T03:43:50.222 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-23T03:43:50.236 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-23T03:43:50.271 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-23T03:43:50.289 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-23T03:43:50.395 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-03-23T03:43:50.406 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T03:43:50.406 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-03-23T03:43:50.406 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T03:43:50.406 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T03:43:50.406 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T03:43:50.407 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T03:43:50.414 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-03-23T03:43:50.423 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T03:43:50.423 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-03-23T03:43:50.423 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T03:43:50.424 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T03:43:50.424 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T03:43:50.424 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T03:43:50.425 INFO:teuthology.run_tasks:Running task install... 2024-03-23T03:43:50.437 DEBUG:teuthology.task.install:project ceph 2024-03-23T03:43:50.437 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'}} 2024-03-23T03:43:50.437 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'} 2024-03-23T03:43:50.437 INFO:teuthology.task.install:Using flavor: default 2024-03-23T03:43:50.443 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-23T03:43:50.443 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-03-23T03:43:50.443 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837', 'tag': None, 'wait_for_package': False} 2024-03-23T03:43:50.444 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=82316b04821a6c2c1d458a881920454bbadb8837 2024-03-23T03:43:50.446 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837', 'tag': None, 'wait_for_package': False} 2024-03-23T03:43:50.446 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=82316b04821a6c2c1d458a881920454bbadb8837 2024-03-23T03:43:50.597 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/centos/9/flavors/default/ 2024-03-23T03:43:50.597 INFO:teuthology.task.install.rpm:Package version is 18.2.2-954.g82316b04 2024-03-23T03:43:50.668 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/centos/9/flavors/default/ 2024-03-23T03:43:50.668 INFO:teuthology.task.install.rpm:Package version is 18.2.2-954.g82316b04 2024-03-23T03:43:50.717 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-23T03:43:50.717 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:43:50.717 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-23T03:43:50.818 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-23T03:43:50.819 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/82316b04821a6c2c1d458a881920454bbadb8837/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-23T03:43:50.844 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-yuri11-testing-2024-03-21-0851-reef/82316b04821a6c2c1d458a881920454bbadb8837/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-23T03:43:50.844 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:43:50.844 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-23T03:43:50.943 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-23T03:43:50.943 DEBUG:teuthology.orchestra.run.smithi102:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/82316b04821a6c2c1d458a881920454bbadb8837/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-23T03:43:50.958 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-23T03:43:51.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-23T03:43:51.090 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-23T03:43:51.178 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-03-23T03:43:51.183 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-03-23T03:43:51.208 DEBUG:teuthology.orchestra.run.smithi102:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-23T03:43:51.296 INFO:teuthology.orchestra.run.smithi102.stdout:check_obsoletes = 1 2024-03-23T03:43:51.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum clean all 2024-03-23T03:43:51.532 INFO:teuthology.orchestra.run.smithi173.stdout:74 files removed 2024-03-23T03:43:51.571 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-23T03:43:51.655 INFO:teuthology.orchestra.run.smithi102.stdout:74 files removed 2024-03-23T03:43:51.694 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-23T03:43:52.633 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 138 kB/s | 77 kB 00:00 2024-03-23T03:43:52.672 INFO:teuthology.orchestra.run.smithi102.stdout:ceph packages for x86_64 161 kB/s | 77 kB 00:00 2024-03-23T03:43:53.125 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 25 kB/s | 9.9 kB 00:00 2024-03-23T03:43:53.197 INFO:teuthology.orchestra.run.smithi102.stdout:ceph noarch packages 23 kB/s | 9.9 kB 00:00 2024-03-23T03:43:53.639 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 4.3 kB/s | 1.9 kB 00:00 2024-03-23T03:43:53.639 INFO:teuthology.orchestra.run.smithi102.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-23T03:43:54.288 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-23T03:43:54.477 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - AppStream 8.7 MB/s | 6.7 MB 00:00 2024-03-23T03:43:56.013 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - BaseOS 4.5 MB/s | 2.0 MB 00:00 2024-03-23T03:43:56.085 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-23T03:43:57.618 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.0 MB 00:01 2024-03-23T03:43:57.690 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 6.6 MB/s | 8.0 MB 00:01 2024-03-23T03:43:59.649 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-03-23T03:43:59.735 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-23T03:44:05.150 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.1 MB 00:01 2024-03-23T03:44:05.370 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.1 MB 00:01 2024-03-23T03:44:06.987 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-03-23T03:44:07.070 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-03-23T03:44:08.019 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 27 MB/s | 21 MB 00:00 2024-03-23T03:44:08.897 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-23T03:44:13.728 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-23T03:44:14.614 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-23T03:44:16.374 INFO:teuthology.orchestra.run.smithi102.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T03:44:16.376 INFO:teuthology.orchestra.run.smithi102.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T03:44:16.395 INFO:teuthology.orchestra.run.smithi102.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-23T03:44:16.397 INFO:teuthology.orchestra.run.smithi102.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-23T03:44:16.398 INFO:teuthology.orchestra.run.smithi102.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-23T03:44:16.399 INFO:teuthology.orchestra.run.smithi102.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-23T03:44:16.402 INFO:teuthology.orchestra.run.smithi102.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-23T03:44:16.406 INFO:teuthology.orchestra.run.smithi102.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-23T03:44:16.407 INFO:teuthology.orchestra.run.smithi102.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-23T03:44:16.490 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-23T03:44:16.518 INFO:teuthology.orchestra.run.smithi102.stdout:=========================================================================================== 2024-03-23T03:44:16.518 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-23T03:44:16.519 INFO:teuthology.orchestra.run.smithi102.stdout:=========================================================================================== 2024-03-23T03:44:16.519 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-23T03:44:16.519 INFO:teuthology.orchestra.run.smithi102.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-23T03:44:16.519 INFO:teuthology.orchestra.run.smithi102.stdout: ceph x86_64 2:18.2.2-954.g82316b04.el9 ceph 6.5 k 2024-03-23T03:44:16.519 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base x86_64 2:18.2.2-954.g82316b04.el9 ceph 5.1 M 2024-03-23T03:44:16.519 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse x86_64 2:18.2.2-954.g82316b04.el9 ceph 843 k 2024-03-23T03:44:16.519 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-954.g82316b04.el9 ceph 143 k 2024-03-23T03:44:16.519 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr x86_64 2:18.2.2-954.g82316b04.el9 ceph 1.4 M 2024-03-23T03:44:16.519 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 138 k 2024-03-23T03:44:16.519 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 1.7 M 2024-03-23T03:44:16.519 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 7.4 M 2024-03-23T03:44:16.520 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 49 k 2024-03-23T03:44:16.520 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw x86_64 2:18.2.2-954.g82316b04.el9 ceph 7.7 M 2024-03-23T03:44:16.520 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test x86_64 2:18.2.2-954.g82316b04.el9 ceph 40 M 2024-03-23T03:44:16.520 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 264 k 2024-03-23T03:44:16.520 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 224 k 2024-03-23T03:44:16.520 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-mirror x86_64 2:18.2.2-954.g82316b04.el9 ceph 221 k 2024-03-23T03:44:16.520 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-top noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 18 k 2024-03-23T03:44:16.520 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-23T03:44:16.520 INFO:teuthology.orchestra.run.smithi102.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-23T03:44:16.520 INFO:teuthology.orchestra.run.smithi102.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-23T03:44:16.520 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel x86_64 2:18.2.2-954.g82316b04.el9 ceph 31 k 2024-03-23T03:44:16.520 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2 x86_64 2:18.2.2-954.g82316b04.el9 ceph 704 k 2024-03-23T03:44:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel x86_64 2:18.2.2-954.g82316b04.el9 ceph 127 k 2024-03-23T03:44:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-23T03:44:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-23T03:44:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-23T03:44:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-23T03:44:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-23T03:44:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-23T03:44:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs x86_64 2:18.2.2-954.g82316b04.el9 ceph 161 k 2024-03-23T03:44:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados x86_64 2:18.2.2-954.g82316b04.el9 ceph 320 k 2024-03-23T03:44:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd x86_64 2:18.2.2-954.g82316b04.el9 ceph 297 k 2024-03-23T03:44:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw x86_64 2:18.2.2-954.g82316b04.el9 ceph 99 k 2024-03-23T03:44:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse x86_64 2:18.2.2-954.g82316b04.el9 ceph 87 k 2024-03-23T03:44:16.521 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror x86_64 2:18.2.2-954.g82316b04.el9 ceph 3.0 M 2024-03-23T03:44:16.522 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd x86_64 2:18.2.2-954.g82316b04.el9 ceph 172 k 2024-03-23T03:44:16.522 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-23T03:44:16.522 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2024-03-23T03:44:16.522 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-23T03:44:16.522 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-23T03:44:16.522 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-23T03:44:16.522 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-23T03:44:16.522 INFO:teuthology.orchestra.run.smithi102.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-23T03:44:16.522 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-23T03:44:16.522 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-23T03:44:16.522 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-23T03:44:16.522 INFO:teuthology.orchestra.run.smithi102.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-23T03:44:16.523 INFO:teuthology.orchestra.run.smithi102.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-23T03:44:16.523 INFO:teuthology.orchestra.run.smithi102.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-23T03:44:16.523 INFO:teuthology.orchestra.run.smithi102.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-23T03:44:16.523 INFO:teuthology.orchestra.run.smithi102.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-23T03:44:16.523 INFO:teuthology.orchestra.run.smithi102.stdout: librados2 x86_64 2:18.2.2-954.g82316b04.el9 ceph 3.3 M 2024-03-23T03:44:16.523 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1 x86_64 2:18.2.2-954.g82316b04.el9 ceph 3.0 M 2024-03-23T03:44:16.523 INFO:teuthology.orchestra.run.smithi102.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-23T03:44:16.523 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-23T03:44:16.523 INFO:teuthology.orchestra.run.smithi102.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-23T03:44:16.523 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-23T03:44:16.524 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-23T03:44:16.524 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-23T03:44:16.524 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-23T03:44:16.524 INFO:teuthology.orchestra.run.smithi102.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-23T03:44:16.524 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-23T03:44:16.524 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-23T03:44:16.524 INFO:teuthology.orchestra.run.smithi102.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-23T03:44:16.524 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-23T03:44:16.524 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common x86_64 2:18.2.2-954.g82316b04.el9 ceph 18 M 2024-03-23T03:44:16.524 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 23 k 2024-03-23T03:44:16.524 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds x86_64 2:18.2.2-954.g82316b04.el9 ceph 2.1 M 2024-03-23T03:44:16.524 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 245 k 2024-03-23T03:44:16.525 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon x86_64 2:18.2.2-954.g82316b04.el9 ceph 4.7 M 2024-03-23T03:44:16.525 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd x86_64 2:18.2.2-954.g82316b04.el9 ceph 16 M 2024-03-23T03:44:16.525 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 15 k 2024-03-23T03:44:16.525 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux x86_64 2:18.2.2-954.g82316b04.el9 ceph 25 k 2024-03-23T03:44:16.525 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-23T03:44:16.525 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-23T03:44:16.525 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-23T03:44:16.525 INFO:teuthology.orchestra.run.smithi102.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-23T03:44:16.525 INFO:teuthology.orchestra.run.smithi102.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-23T03:44:16.525 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-23T03:44:16.525 INFO:teuthology.orchestra.run.smithi102.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-23T03:44:16.525 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-23T03:44:16.525 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-23T03:44:16.526 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite x86_64 2:18.2.2-954.g82316b04.el9 ceph 166 k 2024-03-23T03:44:16.526 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-23T03:44:16.526 INFO:teuthology.orchestra.run.smithi102.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-23T03:44:16.526 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-23T03:44:16.526 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-23T03:44:16.526 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-23T03:44:16.526 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1 x86_64 2:18.2.2-954.g82316b04.el9 ceph 474 k 2024-03-23T03:44:16.526 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-23T03:44:16.526 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2 x86_64 2:18.2.2-954.g82316b04.el9 ceph 4.5 M 2024-03-23T03:44:16.526 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-23T03:44:16.526 INFO:teuthology.orchestra.run.smithi102.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-23T03:44:16.526 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-23T03:44:16.526 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-23T03:44:16.527 INFO:teuthology.orchestra.run.smithi102.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-23T03:44:16.527 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-23T03:44:16.527 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-23T03:44:16.527 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-23T03:44:16.527 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-23T03:44:16.527 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-23T03:44:16.527 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-23T03:44:16.527 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-23T03:44:16.527 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse x86_64 2:18.2.2-954.g82316b04.el9 ceph 45 k 2024-03-23T03:44:16.527 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common x86_64 2:18.2.2-954.g82316b04.el9 ceph 127 k 2024-03-23T03:44:16.527 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-23T03:44:16.527 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-23T03:44:16.527 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-23T03:44:16.528 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-23T03:44:16.528 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-23T03:44:16.528 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-23T03:44:16.528 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-23T03:44:16.528 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-23T03:44:16.528 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-23T03:44:16.528 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-23T03:44:16.528 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-23T03:44:16.528 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-23T03:44:16.528 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-23T03:44:16.528 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-23T03:44:16.528 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-23T03:44:16.528 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-23T03:44:16.528 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-23T03:44:16.529 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-23T03:44:16.529 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-23T03:44:16.529 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-23T03:44:16.529 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-23T03:44:16.529 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-23T03:44:16.529 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-23T03:44:16.529 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-23T03:44:16.529 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-23T03:44:16.529 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-23T03:44:16.529 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-23T03:44:16.529 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-23T03:44:16.529 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-23T03:44:16.529 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-23T03:44:16.530 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-23T03:44:16.530 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-23T03:44:16.530 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-23T03:44:16.530 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-23T03:44:16.530 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-23T03:44:16.530 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-23T03:44:16.530 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-23T03:44:16.530 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-23T03:44:16.530 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-23T03:44:16.530 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-23T03:44:16.530 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-23T03:44:16.530 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-23T03:44:16.531 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-23T03:44:16.531 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-23T03:44:16.531 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-23T03:44:16.531 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-23T03:44:16.531 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-23T03:44:16.531 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-23T03:44:16.531 INFO:teuthology.orchestra.run.smithi102.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-23T03:44:16.531 INFO:teuthology.orchestra.run.smithi102.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-23T03:44:16.531 INFO:teuthology.orchestra.run.smithi102.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-23T03:44:16.531 INFO:teuthology.orchestra.run.smithi102.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-23T03:44:16.531 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-23T03:44:16.531 INFO:teuthology.orchestra.run.smithi102.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-23T03:44:16.531 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:44:16.531 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-23T03:44:16.531 INFO:teuthology.orchestra.run.smithi102.stdout:=========================================================================================== 2024-03-23T03:44:16.532 INFO:teuthology.orchestra.run.smithi102.stdout:Install 132 Packages 2024-03-23T03:44:16.532 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 24 Packages 2024-03-23T03:44:16.532 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:44:16.532 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 204 M 2024-03-23T03:44:16.532 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-23T03:44:17.143 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T03:44:17.144 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T03:44:17.161 INFO:teuthology.orchestra.run.smithi173.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-23T03:44:17.163 INFO:teuthology.orchestra.run.smithi173.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-23T03:44:17.164 INFO:teuthology.orchestra.run.smithi173.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-23T03:44:17.165 INFO:teuthology.orchestra.run.smithi173.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-23T03:44:17.168 INFO:teuthology.orchestra.run.smithi173.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-23T03:44:17.171 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-23T03:44:17.173 INFO:teuthology.orchestra.run.smithi173.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-23T03:44:17.259 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-23T03:44:17.274 INFO:teuthology.orchestra.run.smithi173.stdout:=========================================================================================== 2024-03-23T03:44:17.274 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-23T03:44:17.274 INFO:teuthology.orchestra.run.smithi173.stdout:=========================================================================================== 2024-03-23T03:44:17.274 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-23T03:44:17.274 INFO:teuthology.orchestra.run.smithi173.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-23T03:44:17.274 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:18.2.2-954.g82316b04.el9 ceph 6.5 k 2024-03-23T03:44:17.275 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:18.2.2-954.g82316b04.el9 ceph 5.1 M 2024-03-23T03:44:17.275 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:18.2.2-954.g82316b04.el9 ceph 843 k 2024-03-23T03:44:17.275 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-954.g82316b04.el9 ceph 143 k 2024-03-23T03:44:17.275 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:18.2.2-954.g82316b04.el9 ceph 1.4 M 2024-03-23T03:44:17.275 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 138 k 2024-03-23T03:44:17.275 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 1.7 M 2024-03-23T03:44:17.275 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 7.4 M 2024-03-23T03:44:17.275 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 49 k 2024-03-23T03:44:17.275 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:18.2.2-954.g82316b04.el9 ceph 7.7 M 2024-03-23T03:44:17.275 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:18.2.2-954.g82316b04.el9 ceph 40 M 2024-03-23T03:44:17.275 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 264 k 2024-03-23T03:44:17.275 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 224 k 2024-03-23T03:44:17.275 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror x86_64 2:18.2.2-954.g82316b04.el9 ceph 221 k 2024-03-23T03:44:17.275 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 18 k 2024-03-23T03:44:17.276 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-23T03:44:17.276 INFO:teuthology.orchestra.run.smithi173.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-23T03:44:17.276 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-23T03:44:17.276 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:18.2.2-954.g82316b04.el9 ceph 31 k 2024-03-23T03:44:17.276 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:18.2.2-954.g82316b04.el9 ceph 704 k 2024-03-23T03:44:17.276 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:18.2.2-954.g82316b04.el9 ceph 127 k 2024-03-23T03:44:17.276 INFO:teuthology.orchestra.run.smithi173.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-23T03:44:17.276 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-23T03:44:17.276 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-23T03:44:17.276 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-23T03:44:17.276 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-23T03:44:17.276 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-23T03:44:17.276 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:18.2.2-954.g82316b04.el9 ceph 161 k 2024-03-23T03:44:17.277 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:18.2.2-954.g82316b04.el9 ceph 320 k 2024-03-23T03:44:17.277 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:18.2.2-954.g82316b04.el9 ceph 297 k 2024-03-23T03:44:17.277 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:18.2.2-954.g82316b04.el9 ceph 99 k 2024-03-23T03:44:17.277 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:18.2.2-954.g82316b04.el9 ceph 87 k 2024-03-23T03:44:17.277 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:18.2.2-954.g82316b04.el9 ceph 3.0 M 2024-03-23T03:44:17.277 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:18.2.2-954.g82316b04.el9 ceph 172 k 2024-03-23T03:44:17.277 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-23T03:44:17.277 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-03-23T03:44:17.277 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-23T03:44:17.277 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-23T03:44:17.277 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-23T03:44:17.277 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-23T03:44:17.278 INFO:teuthology.orchestra.run.smithi173.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-23T03:44:17.278 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-23T03:44:17.278 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-23T03:44:17.278 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-23T03:44:17.278 INFO:teuthology.orchestra.run.smithi173.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-23T03:44:17.278 INFO:teuthology.orchestra.run.smithi173.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-23T03:44:17.278 INFO:teuthology.orchestra.run.smithi173.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-23T03:44:17.278 INFO:teuthology.orchestra.run.smithi173.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-23T03:44:17.278 INFO:teuthology.orchestra.run.smithi173.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-23T03:44:17.278 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:18.2.2-954.g82316b04.el9 ceph 3.3 M 2024-03-23T03:44:17.278 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:18.2.2-954.g82316b04.el9 ceph 3.0 M 2024-03-23T03:44:17.278 INFO:teuthology.orchestra.run.smithi173.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-23T03:44:17.278 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-23T03:44:17.279 INFO:teuthology.orchestra.run.smithi173.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-23T03:44:17.279 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-23T03:44:17.279 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-23T03:44:17.279 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-23T03:44:17.279 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-23T03:44:17.279 INFO:teuthology.orchestra.run.smithi173.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-23T03:44:17.279 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-23T03:44:17.279 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-23T03:44:17.279 INFO:teuthology.orchestra.run.smithi173.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-23T03:44:17.279 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-23T03:44:17.279 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:18.2.2-954.g82316b04.el9 ceph 18 M 2024-03-23T03:44:17.279 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 23 k 2024-03-23T03:44:17.279 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:18.2.2-954.g82316b04.el9 ceph 2.1 M 2024-03-23T03:44:17.280 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 245 k 2024-03-23T03:44:17.280 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:18.2.2-954.g82316b04.el9 ceph 4.7 M 2024-03-23T03:44:17.280 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:18.2.2-954.g82316b04.el9 ceph 16 M 2024-03-23T03:44:17.280 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:18.2.2-954.g82316b04.el9 ceph-noarch 15 k 2024-03-23T03:44:17.280 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:18.2.2-954.g82316b04.el9 ceph 25 k 2024-03-23T03:44:17.280 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-23T03:44:17.280 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-23T03:44:17.280 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-23T03:44:17.280 INFO:teuthology.orchestra.run.smithi173.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-03-23T03:44:17.280 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-23T03:44:17.280 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-23T03:44:17.280 INFO:teuthology.orchestra.run.smithi173.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-23T03:44:17.280 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-23T03:44:17.281 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-23T03:44:17.281 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:18.2.2-954.g82316b04.el9 ceph 166 k 2024-03-23T03:44:17.281 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-23T03:44:17.281 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-23T03:44:17.281 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-23T03:44:17.281 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-23T03:44:17.281 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-23T03:44:17.281 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:18.2.2-954.g82316b04.el9 ceph 474 k 2024-03-23T03:44:17.281 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-23T03:44:17.281 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:18.2.2-954.g82316b04.el9 ceph 4.5 M 2024-03-23T03:44:17.281 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-23T03:44:17.281 INFO:teuthology.orchestra.run.smithi173.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-23T03:44:17.282 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-23T03:44:17.282 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-23T03:44:17.282 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-23T03:44:17.282 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-23T03:44:17.282 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-23T03:44:17.282 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-23T03:44:17.282 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-23T03:44:17.282 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-23T03:44:17.282 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-23T03:44:17.282 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-23T03:44:17.282 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:18.2.2-954.g82316b04.el9 ceph 45 k 2024-03-23T03:44:17.282 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:18.2.2-954.g82316b04.el9 ceph 127 k 2024-03-23T03:44:17.282 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-23T03:44:17.283 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-23T03:44:17.283 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-23T03:44:17.283 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-23T03:44:17.283 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-23T03:44:17.283 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-23T03:44:17.283 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-23T03:44:17.283 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-23T03:44:17.283 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-23T03:44:17.283 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-23T03:44:17.283 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-23T03:44:17.283 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-23T03:44:17.283 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-23T03:44:17.283 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-23T03:44:17.283 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-23T03:44:17.284 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-23T03:44:17.284 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-23T03:44:17.284 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-23T03:44:17.284 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-23T03:44:17.284 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-23T03:44:17.284 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-23T03:44:17.284 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-23T03:44:17.284 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-23T03:44:17.284 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-23T03:44:17.284 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-23T03:44:17.284 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-23T03:44:17.284 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-23T03:44:17.284 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-23T03:44:17.284 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-23T03:44:17.284 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-23T03:44:17.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-23T03:44:17.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-23T03:44:17.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-23T03:44:17.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-23T03:44:17.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-23T03:44:17.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-23T03:44:17.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-23T03:44:17.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-23T03:44:17.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-23T03:44:17.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-23T03:44:17.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-23T03:44:17.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-23T03:44:17.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-23T03:44:17.285 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-23T03:44:17.286 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-23T03:44:17.286 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-23T03:44:17.286 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-23T03:44:17.286 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-23T03:44:17.286 INFO:teuthology.orchestra.run.smithi173.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-23T03:44:17.286 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-23T03:44:17.286 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-23T03:44:17.286 INFO:teuthology.orchestra.run.smithi173.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-23T03:44:17.286 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-23T03:44:17.286 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-23T03:44:17.286 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:44:17.286 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-23T03:44:17.286 INFO:teuthology.orchestra.run.smithi173.stdout:=========================================================================================== 2024-03-23T03:44:17.287 INFO:teuthology.orchestra.run.smithi173.stdout:Install 132 Packages 2024-03-23T03:44:17.287 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 24 Packages 2024-03-23T03:44:17.287 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:44:17.287 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 204 M 2024-03-23T03:44:17.287 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-23T03:44:17.504 INFO:teuthology.orchestra.run.smithi102.stdout:(1/156): ceph-18.2.2-954.g82316b04.el9.x86_64.r 20 kB/s | 6.5 kB 00:00 2024-03-23T03:44:17.796 INFO:teuthology.orchestra.run.smithi102.stdout:(2/156): ceph-base-18.2.2-954.g82316b04.el9.x86 8.2 MB/s | 5.1 MB 00:00 2024-03-23T03:44:17.855 INFO:teuthology.orchestra.run.smithi102.stdout:(3/156): ceph-immutable-object-cache-18.2.2-954 2.4 MB/s | 143 kB 00:00 2024-03-23T03:44:18.030 INFO:teuthology.orchestra.run.smithi102.stdout:(4/156): ceph-mds-18.2.2-954.g82316b04.el9.x86_ 12 MB/s | 2.1 MB 00:00 2024-03-23T03:44:18.122 INFO:teuthology.orchestra.run.smithi102.stdout:(5/156): ceph-fuse-18.2.2-954.g82316b04.el9.x86 1.3 MB/s | 843 kB 00:00 2024-03-23T03:44:18.207 INFO:teuthology.orchestra.run.smithi173.stdout:(1/156): ceph-18.2.2-954.g82316b04.el9.x86_64.r 19 kB/s | 6.5 kB 00:00 2024-03-23T03:44:18.272 INFO:teuthology.orchestra.run.smithi102.stdout:(6/156): ceph-mgr-18.2.2-954.g82316b04.el9.x86_ 6.0 MB/s | 1.4 MB 00:00 2024-03-23T03:44:19.324 INFO:teuthology.orchestra.run.smithi173.stdout:(2/156): ceph-fuse-18.2.2-954.g82316b04.el9.x86 756 kB/s | 843 kB 00:01 2024-03-23T03:44:19.331 INFO:teuthology.orchestra.run.smithi102.stdout:(7/156): ceph-osd-18.2.2-954.g82316b04.el9.x86_ 16 MB/s | 16 MB 00:01 2024-03-23T03:44:19.466 INFO:teuthology.orchestra.run.smithi173.stdout:(3/156): ceph-immutable-object-cache-18.2.2-954 1.0 MB/s | 143 kB 00:00 2024-03-23T03:44:19.990 INFO:teuthology.orchestra.run.smithi102.stdout:(8/156): ceph-radosgw-18.2.2-954.g82316b04.el9. 12 MB/s | 7.7 MB 00:00 2024-03-23T03:44:20.048 INFO:teuthology.orchestra.run.smithi102.stdout:(9/156): ceph-selinux-18.2.2-954.g82316b04.el9. 440 kB/s | 25 kB 00:00 2024-03-23T03:44:20.281 INFO:teuthology.orchestra.run.smithi102.stdout:(10/156): ceph-mon-18.2.2-954.g82316b04.el9.x86 2.2 MB/s | 4.7 MB 00:02 2024-03-23T03:44:20.338 INFO:teuthology.orchestra.run.smithi173.stdout:(4/156): ceph-base-18.2.2-954.g82316b04.el9.x86 2.1 MB/s | 5.1 MB 00:02 2024-03-23T03:44:20.465 INFO:teuthology.orchestra.run.smithi102.stdout:(11/156): cephfs-mirror-18.2.2-954.g82316b04.el 1.2 MB/s | 221 kB 00:00 2024-03-23T03:44:20.598 INFO:teuthology.orchestra.run.smithi102.stdout:(12/156): libcephfs-devel-18.2.2-954.g82316b04. 235 kB/s | 31 kB 00:00 2024-03-23T03:44:20.850 INFO:teuthology.orchestra.run.smithi173.stdout:(5/156): ceph-mgr-18.2.2-954.g82316b04.el9.x86_ 2.8 MB/s | 1.4 MB 00:00 2024-03-23T03:44:21.032 INFO:teuthology.orchestra.run.smithi102.stdout:(13/156): libcephfs2-18.2.2-954.g82316b04.el9.x 1.6 MB/s | 704 kB 00:00 2024-03-23T03:44:21.149 INFO:teuthology.orchestra.run.smithi102.stdout:(14/156): libcephsqlite-18.2.2-954.g82316b04.el 1.4 MB/s | 166 kB 00:00 2024-03-23T03:44:21.291 INFO:teuthology.orchestra.run.smithi102.stdout:(15/156): librados-devel-18.2.2-954.g82316b04.e 899 kB/s | 127 kB 00:00 2024-03-23T03:44:21.600 INFO:teuthology.orchestra.run.smithi173.stdout:(6/156): ceph-mds-18.2.2-954.g82316b04.el9.x86_ 1.0 MB/s | 2.1 MB 00:02 2024-03-23T03:44:21.692 INFO:teuthology.orchestra.run.smithi102.stdout:(16/156): libradosstriper1-18.2.2-954.g82316b04 1.2 MB/s | 474 kB 00:00 2024-03-23T03:44:21.917 INFO:teuthology.orchestra.run.smithi173.stdout:(7/156): ceph-mon-18.2.2-954.g82316b04.el9.x86_ 4.4 MB/s | 4.7 MB 00:01 2024-03-23T03:44:22.084 INFO:teuthology.orchestra.run.smithi102.stdout:(17/156): ceph-common-18.2.2-954.g82316b04.el9. 3.7 MB/s | 18 MB 00:04 2024-03-23T03:44:22.192 INFO:teuthology.orchestra.run.smithi102.stdout:(18/156): python3-ceph-argparse-18.2.2-954.g823 420 kB/s | 45 kB 00:00 2024-03-23T03:44:22.292 INFO:teuthology.orchestra.run.smithi102.stdout:(19/156): python3-ceph-common-18.2.2-954.g82316 1.3 MB/s | 127 kB 00:00 2024-03-23T03:44:22.393 INFO:teuthology.orchestra.run.smithi102.stdout:(20/156): python3-cephfs-18.2.2-954.g82316b04.e 1.6 MB/s | 161 kB 00:00 2024-03-23T03:44:22.467 INFO:teuthology.orchestra.run.smithi173.stdout:(8/156): ceph-common-18.2.2-954.g82316b04.el9.x 4.0 MB/s | 18 MB 00:04 2024-03-23T03:44:22.534 INFO:teuthology.orchestra.run.smithi173.stdout:(9/156): ceph-selinux-18.2.2-954.g82316b04.el9. 383 kB/s | 25 kB 00:00 2024-03-23T03:44:22.535 INFO:teuthology.orchestra.run.smithi102.stdout:(21/156): python3-rados-18.2.2-954.g82316b04.el 2.2 MB/s | 320 kB 00:00 2024-03-23T03:44:23.068 INFO:teuthology.orchestra.run.smithi102.stdout:(22/156): ceph-test-18.2.2-954.g82316b04.el9.x8 13 MB/s | 40 MB 00:03 2024-03-23T03:44:23.102 INFO:teuthology.orchestra.run.smithi102.stdout:(23/156): python3-rbd-18.2.2-954.g82316b04.el9. 525 kB/s | 297 kB 00:00 2024-03-23T03:44:23.127 INFO:teuthology.orchestra.run.smithi102.stdout:(24/156): python3-rgw-18.2.2-954.g82316b04.el9. 1.7 MB/s | 99 kB 00:00 2024-03-23T03:44:23.134 INFO:teuthology.orchestra.run.smithi173.stdout:(10/156): ceph-radosgw-18.2.2-954.g82316b04.el9 6.3 MB/s | 7.7 MB 00:01 2024-03-23T03:44:23.210 INFO:teuthology.orchestra.run.smithi173.stdout:(11/156): cephfs-mirror-18.2.2-954.g82316b04.el 2.9 MB/s | 221 kB 00:00 2024-03-23T03:44:23.219 INFO:teuthology.orchestra.run.smithi102.stdout:(25/156): rbd-fuse-18.2.2-954.g82316b04.el9.x86 747 kB/s | 87 kB 00:00 2024-03-23T03:44:23.277 INFO:teuthology.orchestra.run.smithi173.stdout:(12/156): libcephfs-devel-18.2.2-954.g82316b04. 472 kB/s | 31 kB 00:00 2024-03-23T03:44:23.386 INFO:teuthology.orchestra.run.smithi102.stdout:(26/156): rbd-nbd-18.2.2-954.g82316b04.el9.x86_ 1.0 MB/s | 172 kB 00:00 2024-03-23T03:44:23.462 INFO:teuthology.orchestra.run.smithi102.stdout:(27/156): rbd-mirror-18.2.2-954.g82316b04.el9.x 9.0 MB/s | 3.0 MB 00:00 2024-03-23T03:44:23.462 INFO:teuthology.orchestra.run.smithi173.stdout:(13/156): libcephfs2-18.2.2-954.g82316b04.el9.x 3.8 MB/s | 704 kB 00:00 2024-03-23T03:44:23.478 INFO:teuthology.orchestra.run.smithi102.stdout:(28/156): ceph-grafana-dashboards-18.2.2-954.g8 257 kB/s | 23 kB 00:00 2024-03-23T03:44:23.520 INFO:teuthology.orchestra.run.smithi102.stdout:(29/156): ceph-mgr-cephadm-18.2.2-954.g82316b04 2.3 MB/s | 138 kB 00:00 2024-03-23T03:44:23.544 INFO:teuthology.orchestra.run.smithi173.stdout:(14/156): libcephsqlite-18.2.2-954.g82316b04.el 2.0 MB/s | 166 kB 00:00 2024-03-23T03:44:23.636 INFO:teuthology.orchestra.run.smithi173.stdout:(15/156): librados-devel-18.2.2-954.g82316b04.e 1.4 MB/s | 127 kB 00:00 2024-03-23T03:44:23.770 INFO:teuthology.orchestra.run.smithi173.stdout:(16/156): libradosstriper1-18.2.2-954.g82316b04 3.5 MB/s | 474 kB 00:00 2024-03-23T03:44:23.871 INFO:teuthology.orchestra.run.smithi102.stdout:(30/156): librgw2-18.2.2-954.g82316b04.el9.x86_ 2.0 MB/s | 4.5 MB 00:02 2024-03-23T03:44:23.972 INFO:teuthology.orchestra.run.smithi102.stdout:(31/156): ceph-mgr-dashboard-18.2.2-954.g82316b 3.5 MB/s | 1.7 MB 00:00 2024-03-23T03:44:24.073 INFO:teuthology.orchestra.run.smithi102.stdout:(32/156): ceph-mgr-modules-core-18.2.2-954.g823 1.2 MB/s | 245 kB 00:00 2024-03-23T03:44:24.105 INFO:teuthology.orchestra.run.smithi102.stdout:(33/156): ceph-mgr-rook-18.2.2-954.g82316b04.el 372 kB/s | 49 kB 00:00 2024-03-23T03:44:24.238 INFO:teuthology.orchestra.run.smithi102.stdout:(34/156): ceph-mgr-diskprediction-local-18.2.2- 10 MB/s | 7.4 MB 00:00 2024-03-23T03:44:24.264 INFO:teuthology.orchestra.run.smithi102.stdout:(35/156): ceph-prometheus-alerts-18.2.2-954.g82 78 kB/s | 15 kB 00:00 2024-03-23T03:44:24.297 INFO:teuthology.orchestra.run.smithi102.stdout:(36/156): ceph-volume-18.2.2-954.g82316b04.el9. 1.3 MB/s | 264 kB 00:00 2024-03-23T03:44:24.323 INFO:teuthology.orchestra.run.smithi102.stdout:(37/156): cephfs-top-18.2.2-954.g82316b04.el9.n 315 kB/s | 18 kB 00:00 2024-03-23T03:44:24.478 INFO:teuthology.orchestra.run.smithi173.stdout:(17/156): librgw2-18.2.2-954.g82316b04.el9.x86_ 6.3 MB/s | 4.5 MB 00:00 2024-03-23T03:44:24.489 INFO:teuthology.orchestra.run.smithi102.stdout:(38/156): cephadm-18.2.2-954.g82316b04.el9.noar 897 kB/s | 224 kB 00:00 2024-03-23T03:44:24.545 INFO:teuthology.orchestra.run.smithi173.stdout:(18/156): python3-ceph-argparse-18.2.2-954.g823 686 kB/s | 45 kB 00:00 2024-03-23T03:44:24.610 INFO:teuthology.orchestra.run.smithi102.stdout:(39/156): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 662 kB 00:00 2024-03-23T03:44:24.620 INFO:teuthology.orchestra.run.smithi173.stdout:(19/156): python3-ceph-common-18.2.2-954.g82316 1.7 MB/s | 127 kB 00:00 2024-03-23T03:44:24.657 INFO:teuthology.orchestra.run.smithi102.stdout:(40/156): bison-3.7.4-5.el9.x86_64.rpm 2.7 MB/s | 920 kB 00:00 2024-03-23T03:44:24.682 INFO:teuthology.orchestra.run.smithi102.stdout:(41/156): boost-program-options-1.75.0-8.el9.x8 557 kB/s | 106 kB 00:00 2024-03-23T03:44:24.687 INFO:teuthology.orchestra.run.smithi173.stdout:(20/156): python3-cephfs-18.2.2-954.g82316b04.e 2.4 MB/s | 161 kB 00:00 2024-03-23T03:44:24.715 INFO:teuthology.orchestra.run.smithi102.stdout:(42/156): flex-2.6.4-9.el9.x86_64.rpm 3.0 MB/s | 307 kB 00:00 2024-03-23T03:44:24.749 INFO:teuthology.orchestra.run.smithi102.stdout:(43/156): flexiblas-3.0.4-8.el9.x86_64.rpm 499 kB/s | 32 kB 00:00 2024-03-23T03:44:24.779 INFO:teuthology.orchestra.run.smithi173.stdout:(21/156): python3-rados-18.2.2-954.g82316b04.el 3.4 MB/s | 320 kB 00:00 2024-03-23T03:44:24.858 INFO:teuthology.orchestra.run.smithi102.stdout:(44/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 17 MB/s | 3.0 MB 00:00 2024-03-23T03:44:24.863 INFO:teuthology.orchestra.run.smithi173.stdout:(22/156): python3-rbd-18.2.2-954.g82316b04.el9. 3.5 MB/s | 297 kB 00:00 2024-03-23T03:44:24.883 INFO:teuthology.orchestra.run.smithi102.stdout:(45/156): flexiblas-openblas-openmp-3.0.4-8.el9 110 kB/s | 17 kB 00:00 2024-03-23T03:44:24.908 INFO:teuthology.orchestra.run.smithi102.stdout:(46/156): libaio-devel-0.3.111-13.el9.x86_64.rp 70 kB/s | 10 kB 00:00 2024-03-23T03:44:24.942 INFO:teuthology.orchestra.run.smithi102.stdout:(47/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-23T03:44:24.955 INFO:teuthology.orchestra.run.smithi173.stdout:(23/156): python3-rgw-18.2.2-954.g82316b04.el9. 1.1 MB/s | 99 kB 00:00 2024-03-23T03:44:24.967 INFO:teuthology.orchestra.run.smithi102.stdout:(48/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 536 kB/s | 44 kB 00:00 2024-03-23T03:44:25.009 INFO:teuthology.orchestra.run.smithi102.stdout:(49/156): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 661 kB 00:00 2024-03-23T03:44:25.051 INFO:teuthology.orchestra.run.smithi102.stdout:(50/156): libtool-2.4.6-45.el9.x86_64.rpm 5.2 MB/s | 578 kB 00:00 2024-03-23T03:44:25.055 INFO:teuthology.orchestra.run.smithi173.stdout:(24/156): rbd-fuse-18.2.2-954.g82316b04.el9.x86 872 kB/s | 87 kB 00:00 2024-03-23T03:44:25.076 INFO:teuthology.orchestra.run.smithi102.stdout:(51/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 450 kB/s | 46 kB 00:00 2024-03-23T03:44:25.110 INFO:teuthology.orchestra.run.smithi102.stdout:(52/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.9 MB/s | 282 kB 00:00 2024-03-23T03:44:25.144 INFO:teuthology.orchestra.run.smithi102.stdout:(53/156): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.3 MB/s | 217 kB 00:00 2024-03-23T03:44:25.169 INFO:teuthology.orchestra.run.smithi102.stdout:(54/156): openblas-0.3.21-2.el9.x86_64.rpm 383 kB/s | 34 kB 00:00 2024-03-23T03:44:25.436 INFO:teuthology.orchestra.run.smithi102.stdout:(55/156): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-03-23T03:44:25.503 INFO:teuthology.orchestra.run.smithi102.stdout:(56/156): postgresql-13.11-1.el9.x86_64.rpm 4.2 MB/s | 1.5 MB 00:00 2024-03-23T03:44:25.545 INFO:teuthology.orchestra.run.smithi102.stdout:(57/156): postgresql-contrib-13.11-1.el9.x86_64 2.1 MB/s | 811 kB 00:00 2024-03-23T03:44:25.612 INFO:teuthology.orchestra.run.smithi102.stdout:(58/156): postgresql-private-libs-13.11-1.el9.x 841 kB/s | 134 kB 00:00 2024-03-23T03:44:25.730 INFO:teuthology.orchestra.run.smithi173.stdout:(25/156): rbd-mirror-18.2.2-954.g82316b04.el9.x 4.4 MB/s | 3.0 MB 00:00 2024-03-23T03:44:25.822 INFO:teuthology.orchestra.run.smithi173.stdout:(26/156): rbd-nbd-18.2.2-954.g82316b04.el9.x86_ 1.9 MB/s | 172 kB 00:00 2024-03-23T03:44:25.896 INFO:teuthology.orchestra.run.smithi102.stdout:(59/156): postgresql-server-13.11-1.el9.x86_64. 15 MB/s | 5.7 MB 00:00 2024-03-23T03:44:25.898 INFO:teuthology.orchestra.run.smithi173.stdout:(27/156): ceph-grafana-dashboards-18.2.2-954.g8 315 kB/s | 23 kB 00:00 2024-03-23T03:44:25.946 INFO:teuthology.orchestra.run.smithi102.stdout:(60/156): python3-jsonpatch-1.21-16.el9.noarch. 78 kB/s | 24 kB 00:00 2024-03-23T03:44:25.965 INFO:teuthology.orchestra.run.smithi173.stdout:(28/156): ceph-mgr-cephadm-18.2.2-954.g82316b04 2.0 MB/s | 138 kB 00:00 2024-03-23T03:44:26.063 INFO:teuthology.orchestra.run.smithi102.stdout:(61/156): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2024-03-23T03:44:26.079 INFO:teuthology.orchestra.run.smithi102.stdout:(62/156): python3-jsonpointer-2.0-4.el9.noarch. 106 kB/s | 18 kB 00:00 2024-03-23T03:44:26.122 INFO:teuthology.orchestra.run.smithi102.stdout:(63/156): python3-mako-1.1.4-6.el9.noarch.rpm 895 kB/s | 144 kB 00:00 2024-03-23T03:44:26.147 INFO:teuthology.orchestra.run.smithi102.stdout:(64/156): python3-markupsafe-1.1.1-12.el9.x86_6 489 kB/s | 32 kB 00:00 2024-03-23T03:44:26.397 INFO:teuthology.orchestra.run.smithi102.stdout:(65/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 15 MB/s | 4.8 MB 00:00 2024-03-23T03:44:26.422 INFO:teuthology.orchestra.run.smithi102.stdout:(66/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 725 kB/s | 217 kB 00:00 2024-03-23T03:44:26.447 INFO:teuthology.orchestra.run.smithi102.stdout:(67/156): python3-oauthlib-3.1.1-5.el9.noarch.r 593 kB/s | 174 kB 00:00 2024-03-23T03:44:26.506 INFO:teuthology.orchestra.run.smithi102.stdout:(68/156): python3-packaging-20.9-5.el9.noarch.r 1.2 MB/s | 69 kB 00:00 2024-03-23T03:44:26.507 INFO:teuthology.orchestra.run.smithi173.stdout:(29/156): ceph-test-18.2.2-954.g82316b04.el9.x8 10 MB/s | 40 MB 00:03 2024-03-23T03:44:26.548 INFO:teuthology.orchestra.run.smithi102.stdout:(69/156): python3-prettytable-0.7.2-27.el9.noar 418 kB/s | 41 kB 00:00 2024-03-23T03:44:26.573 INFO:teuthology.orchestra.run.smithi102.stdout:(70/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 132 kB 00:00 2024-03-23T03:44:26.623 INFO:teuthology.orchestra.run.smithi102.stdout:(71/156): python3-pytz-2021.1-5.el9.noarch.rpm 977 kB/s | 47 kB 00:00 2024-03-23T03:44:26.649 INFO:teuthology.orchestra.run.smithi102.stdout:(72/156): python3-requests-oauthlib-1.3.0-12.el 581 kB/s | 43 kB 00:00 2024-03-23T03:44:26.682 INFO:teuthology.orchestra.run.smithi102.stdout:(73/156): python3-pyasn1-modules-0.4.8-6.el9.no 1.9 MB/s | 211 kB 00:00 2024-03-23T03:44:26.805 INFO:teuthology.orchestra.run.smithi173.stdout:(30/156): ceph-mgr-dashboard-18.2.2-954.g82316b 2.1 MB/s | 1.7 MB 00:00 2024-03-23T03:44:26.949 INFO:teuthology.orchestra.run.smithi173.stdout:(31/156): ceph-mgr-modules-core-18.2.2-954.g823 1.7 MB/s | 245 kB 00:00 2024-03-23T03:44:27.008 INFO:teuthology.orchestra.run.smithi102.stdout:(74/156): python3-toml-0.10.2-6.el9.noarch.rpm 108 kB/s | 39 kB 00:00 2024-03-23T03:44:27.049 INFO:teuthology.orchestra.run.smithi173.stdout:(32/156): ceph-mgr-rook-18.2.2-954.g82316b04.el 497 kB/s | 49 kB 00:00 2024-03-23T03:44:27.050 INFO:teuthology.orchestra.run.smithi102.stdout:(75/156): socat-1.7.4.1-5.el9.x86_64.rpm 846 kB/s | 300 kB 00:00 2024-03-23T03:44:27.191 INFO:teuthology.orchestra.run.smithi173.stdout:(33/156): ceph-mgr-diskprediction-local-18.2.2- 11 MB/s | 7.4 MB 00:00 2024-03-23T03:44:27.217 INFO:teuthology.orchestra.run.smithi173.stdout:(34/156): ceph-prometheus-alerts-18.2.2-954.g82 89 kB/s | 15 kB 00:00 2024-03-23T03:44:27.267 INFO:teuthology.orchestra.run.smithi173.stdout:(35/156): cephadm-18.2.2-954.g82316b04.el9.noar 4.4 MB/s | 224 kB 00:00 2024-03-23T03:44:27.292 INFO:teuthology.orchestra.run.smithi173.stdout:(36/156): ceph-volume-18.2.2-954.g82316b04.el9. 2.6 MB/s | 264 kB 00:00 2024-03-23T03:44:27.318 INFO:teuthology.orchestra.run.smithi173.stdout:(37/156): cephfs-top-18.2.2-954.g82316b04.el9.n 369 kB/s | 18 kB 00:00 2024-03-23T03:44:27.350 INFO:teuthology.orchestra.run.smithi102.stdout:(76/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-03-23T03:44:27.375 INFO:teuthology.orchestra.run.smithi102.stdout:(77/156): uuid-1.6.2-55.el9.x86_64.rpm 155 kB/s | 56 kB 00:00 2024-03-23T03:44:27.400 INFO:teuthology.orchestra.run.smithi102.stdout:(78/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 186 kB/s | 63 kB 00:00 2024-03-23T03:44:27.442 INFO:teuthology.orchestra.run.smithi102.stdout:(79/156): fuse-2.9.9-15.el9.x86_64.rpm 1.9 MB/s | 79 kB 00:00 2024-03-23T03:44:27.467 INFO:teuthology.orchestra.run.smithi102.stdout:(80/156): zlib-devel-1.2.11-40.el9.x86_64.rpm 671 kB/s | 44 kB 00:00 2024-03-23T03:44:27.509 INFO:teuthology.orchestra.run.smithi102.stdout:(81/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.2 MB/s | 241 kB 00:00 2024-03-23T03:44:27.559 INFO:teuthology.orchestra.run.smithi173.stdout:(38/156): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2024-03-23T03:44:27.568 INFO:teuthology.orchestra.run.smithi102.stdout:(82/156): python3-cryptography-36.0.1-4.el9.x86 9.4 MB/s | 1.1 MB 00:00 2024-03-23T03:44:27.602 INFO:teuthology.orchestra.run.smithi102.stdout:(83/156): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-23T03:44:27.618 INFO:teuthology.orchestra.run.smithi173.stdout:(39/156): bison-3.7.4-5.el9.x86_64.rpm 3.0 MB/s | 920 kB 00:00 2024-03-23T03:44:27.627 INFO:teuthology.orchestra.run.smithi102.stdout:(84/156): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 124 kB 00:00 2024-03-23T03:44:27.635 INFO:teuthology.orchestra.run.smithi173.stdout:(40/156): boost-program-options-1.75.0-8.el9.x8 1.5 MB/s | 106 kB 00:00 2024-03-23T03:44:27.661 INFO:teuthology.orchestra.run.smithi102.stdout:(85/156): python3-urllib3-1.26.5-5.el9.noarch.r 2.0 MB/s | 187 kB 00:00 2024-03-23T03:44:27.694 INFO:teuthology.orchestra.run.smithi173.stdout:(41/156): flex-2.6.4-9.el9.x86_64.rpm 4.1 MB/s | 307 kB 00:00 2024-03-23T03:44:27.719 INFO:teuthology.orchestra.run.smithi173.stdout:(42/156): flexiblas-3.0.4-8.el9.x86_64.rpm 380 kB/s | 32 kB 00:00 2024-03-23T03:44:27.753 INFO:teuthology.orchestra.run.smithi102.stdout:(86/156): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-03-23T03:44:27.786 INFO:teuthology.orchestra.run.smithi102.stdout:(87/156): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-23T03:44:27.828 INFO:teuthology.orchestra.run.smithi102.stdout:(88/156): libgfortran-11.4.1-3.el9.x86_64.rpm 4.0 MB/s | 803 kB 00:00 2024-03-23T03:44:27.870 INFO:teuthology.orchestra.run.smithi173.stdout:(43/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 17 MB/s | 3.0 MB 00:00 2024-03-23T03:44:27.878 INFO:teuthology.orchestra.run.smithi102.stdout:(89/156): python3-requests-2.25.1-8.el9.noarch. 1.0 MB/s | 125 kB 00:00 2024-03-23T03:44:27.887 INFO:teuthology.orchestra.run.smithi173.stdout:(44/156): flexiblas-openblas-openmp-3.0.4-8.el9 107 kB/s | 17 kB 00:00 2024-03-23T03:44:27.903 INFO:teuthology.orchestra.run.smithi173.stdout:(45/156): libaio-devel-0.3.111-13.el9.x86_64.rp 309 kB/s | 10 kB 00:00 2024-03-23T03:44:27.929 INFO:teuthology.orchestra.run.smithi173.stdout:(46/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2024-03-23T03:44:27.954 INFO:teuthology.orchestra.run.smithi173.stdout:(47/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 918 kB/s | 44 kB 00:00 2024-03-23T03:44:28.005 INFO:teuthology.orchestra.run.smithi173.stdout:(48/156): libtool-2.4.6-45.el9.x86_64.rpm 11 MB/s | 578 kB 00:00 2024-03-23T03:44:28.038 INFO:teuthology.orchestra.run.smithi173.stdout:(49/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-23T03:44:28.080 INFO:teuthology.orchestra.run.smithi173.stdout:(50/156): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 661 kB 00:00 2024-03-23T03:44:28.120 INFO:teuthology.orchestra.run.smithi102.stdout:(90/156): elfutils-libelf-devel-0.190-2.el9.x86 64 kB/s | 21 kB 00:00 2024-03-23T03:44:28.130 INFO:teuthology.orchestra.run.smithi173.stdout:(51/156): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-23T03:44:28.137 INFO:teuthology.orchestra.run.smithi102.stdout:(91/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 83 kB/s | 25 kB 00:00 2024-03-23T03:44:28.164 INFO:teuthology.orchestra.run.smithi173.stdout:(52/156): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-23T03:44:28.198 INFO:teuthology.orchestra.run.smithi173.stdout:(53/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 282 kB 00:00 2024-03-23T03:44:28.279 INFO:teuthology.orchestra.run.smithi102.stdout:(92/156): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-23T03:44:28.313 INFO:teuthology.orchestra.run.smithi102.stdout:(93/156): python3-jinja2-2.11.3-5.el9.noarch.rp 1.4 MB/s | 248 kB 00:00 2024-03-23T03:44:28.371 INFO:teuthology.orchestra.run.smithi102.stdout:(94/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.0 MB/s | 94 kB 00:00 2024-03-23T03:44:28.381 INFO:teuthology.orchestra.run.smithi173.stdout:(54/156): openblas-openmp-0.3.21-2.el9.x86_64.r 22 MB/s | 4.7 MB 00:00 2024-03-23T03:44:28.448 INFO:teuthology.orchestra.run.smithi173.stdout:(55/156): postgresql-13.11-1.el9.x86_64.rpm 6.4 MB/s | 1.5 MB 00:00 2024-03-23T03:44:28.499 INFO:teuthology.orchestra.run.smithi173.stdout:(56/156): postgresql-contrib-13.11-1.el9.x86_64 7.6 MB/s | 811 kB 00:00 2024-03-23T03:44:28.524 INFO:teuthology.orchestra.run.smithi173.stdout:(57/156): postgresql-private-libs-13.11-1.el9.x 1.9 MB/s | 134 kB 00:00 2024-03-23T03:44:28.680 INFO:teuthology.orchestra.run.smithi102.stdout:(95/156): openssl-devel-3.0.7-27.el9.x86_64.rpm 5.2 MB/s | 4.1 MB 00:00 2024-03-23T03:44:28.849 INFO:teuthology.orchestra.run.smithi173.stdout:(58/156): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-23T03:44:28.966 INFO:teuthology.orchestra.run.smithi173.stdout:(59/156): postgresql-server-13.11-1.el9.x86_64. 12 MB/s | 5.7 MB 00:00 2024-03-23T03:44:28.991 INFO:teuthology.orchestra.run.smithi173.stdout:(60/156): python3-jsonpatch-1.21-16.el9.noarch. 185 kB/s | 24 kB 00:00 2024-03-23T03:44:29.017 INFO:teuthology.orchestra.run.smithi173.stdout:(61/156): python3-jsonpointer-2.0-4.el9.noarch. 369 kB/s | 18 kB 00:00 2024-03-23T03:44:29.030 INFO:teuthology.orchestra.run.smithi102.stdout:(96/156): fmt-8.1.1-5.el9.x86_64.rpm 154 kB/s | 111 kB 00:00 2024-03-23T03:44:29.042 INFO:teuthology.orchestra.run.smithi173.stdout:(62/156): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 144 kB 00:00 2024-03-23T03:44:29.067 INFO:teuthology.orchestra.run.smithi173.stdout:(63/156): python3-markupsafe-1.1.1-12.el9.x86_6 655 kB/s | 32 kB 00:00 2024-03-23T03:44:29.193 INFO:teuthology.orchestra.run.smithi173.stdout:(64/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-23T03:44:29.218 INFO:teuthology.orchestra.run.smithi173.stdout:(65/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 217 kB 00:00 2024-03-23T03:44:29.239 INFO:teuthology.orchestra.run.smithi102.stdout:(97/156): libarrow-doc-9.0.0-10.el9.noarch.rpm 125 kB/s | 26 kB 00:00 2024-03-23T03:44:29.260 INFO:teuthology.orchestra.run.smithi173.stdout:(66/156): python3-oauthlib-3.1.1-5.el9.noarch.r 4.2 MB/s | 174 kB 00:00 2024-03-23T03:44:29.285 INFO:teuthology.orchestra.run.smithi173.stdout:(67/156): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-23T03:44:29.311 INFO:teuthology.orchestra.run.smithi173.stdout:(68/156): python3-prettytable-0.7.2-27.el9.noar 838 kB/s | 41 kB 00:00 2024-03-23T03:44:29.336 INFO:teuthology.orchestra.run.smithi173.stdout:(69/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-23T03:44:29.356 INFO:teuthology.orchestra.run.smithi102.stdout:(98/156): liboath-2.6.7-2.el9.x86_64.rpm 419 kB/s | 49 kB 00:00 2024-03-23T03:44:29.378 INFO:teuthology.orchestra.run.smithi173.stdout:(70/156): python3-pyasn1-modules-0.4.8-6.el9.no 3.2 MB/s | 211 kB 00:00 2024-03-23T03:44:29.396 INFO:teuthology.orchestra.run.smithi173.stdout:(71/156): python3-pytz-2021.1-5.el9.noarch.rpm 844 kB/s | 47 kB 00:00 2024-03-23T03:44:29.406 INFO:teuthology.orchestra.run.smithi102.stdout:(99/156): gperftools-libs-2.9.1-2.el9.x86_64.rp 299 kB/s | 309 kB 00:01 2024-03-23T03:44:29.412 INFO:teuthology.orchestra.run.smithi173.stdout:(72/156): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-23T03:44:29.571 INFO:teuthology.orchestra.run.smithi173.stdout:(73/156): python3-toml-0.10.2-6.el9.noarch.rpm 270 kB/s | 39 kB 00:00 2024-03-23T03:44:29.598 INFO:teuthology.orchestra.run.smithi102.stdout:(100/156): libunwind-1.6.2-1.el9.x86_64.rpm 279 kB/s | 67 kB 00:00 2024-03-23T03:44:29.771 INFO:teuthology.orchestra.run.smithi173.stdout:(74/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-23T03:44:29.805 INFO:teuthology.orchestra.run.smithi173.stdout:(75/156): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 300 kB 00:00 2024-03-23T03:44:29.822 INFO:teuthology.orchestra.run.smithi173.stdout:(76/156): uuid-1.6.2-55.el9.x86_64.rpm 1.2 MB/s | 56 kB 00:00 2024-03-23T03:44:29.847 INFO:teuthology.orchestra.run.smithi173.stdout:(77/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-23T03:44:29.864 INFO:teuthology.orchestra.run.smithi173.stdout:(78/156): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-03-23T03:44:29.898 INFO:teuthology.orchestra.run.smithi173.stdout:(79/156): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-03-23T03:44:29.931 INFO:teuthology.orchestra.run.smithi173.stdout:(80/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.6 MB/s | 241 kB 00:00 2024-03-23T03:44:29.982 INFO:teuthology.orchestra.run.smithi173.stdout:(81/156): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-23T03:44:30.009 INFO:teuthology.orchestra.run.smithi173.stdout:(82/156): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-23T03:44:30.032 INFO:teuthology.orchestra.run.smithi173.stdout:(83/156): python3-pycparser-2.20-6.el9.noarch.r 2.5 MB/s | 124 kB 00:00 2024-03-23T03:44:30.066 INFO:teuthology.orchestra.run.smithi173.stdout:(84/156): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-03-23T03:44:30.375 INFO:teuthology.orchestra.run.smithi102.stdout:(101/156): python3-asyncssh-2.13.2-5.el9.noarch 706 kB/s | 548 kB 00:00 2024-03-23T03:44:30.383 INFO:teuthology.orchestra.run.smithi173.stdout:(85/156): jq-1.6-16.el9.x86_64.rpm 538 kB/s | 187 kB 00:00 2024-03-23T03:44:30.425 INFO:teuthology.orchestra.run.smithi173.stdout:(86/156): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-03-23T03:44:30.432 INFO:teuthology.orchestra.run.smithi102.stdout:(102/156): parquet-libs-9.0.0-10.el9.x86_64.rpm 818 kB/s | 838 kB 00:01 2024-03-23T03:44:30.475 INFO:teuthology.orchestra.run.smithi173.stdout:(87/156): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-03-23T03:44:30.482 INFO:teuthology.orchestra.run.smithi102.stdout:(103/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 408 kB/s | 43 kB 00:00 2024-03-23T03:44:30.509 INFO:teuthology.orchestra.run.smithi173.stdout:(88/156): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-03-23T03:44:30.541 INFO:teuthology.orchestra.run.smithi102.stdout:(104/156): python3-cachetools-4.2.4-1.el9.noarc 298 kB/s | 32 kB 00:00 2024-03-23T03:44:30.551 INFO:teuthology.orchestra.run.smithi173.stdout:(89/156): elfutils-libelf-devel-0.190-2.el9.x86 288 kB/s | 21 kB 00:00 2024-03-23T03:44:30.576 INFO:teuthology.orchestra.run.smithi173.stdout:(90/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 384 kB/s | 25 kB 00:00 2024-03-23T03:44:30.591 INFO:teuthology.orchestra.run.smithi102.stdout:(105/156): python3-certifi-2021.10.8-2.el9.noar 139 kB/s | 15 kB 00:00 2024-03-23T03:44:30.676 INFO:teuthology.orchestra.run.smithi173.stdout:(91/156): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-23T03:44:30.752 INFO:teuthology.orchestra.run.smithi173.stdout:(92/156): python3-jinja2-2.11.3-5.el9.noarch.rp 3.3 MB/s | 248 kB 00:00 2024-03-23T03:44:30.841 INFO:teuthology.orchestra.run.smithi102.stdout:(106/156): python3-cheroot-8.6.0-4.el9.noarch.r 573 kB/s | 172 kB 00:00 2024-03-23T03:44:30.844 INFO:teuthology.orchestra.run.smithi173.stdout:(93/156): openssl-devel-3.0.7-27.el9.x86_64.rpm 14 MB/s | 4.1 MB 00:00 2024-03-23T03:44:30.894 INFO:teuthology.orchestra.run.smithi173.stdout:(94/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 661 kB/s | 94 kB 00:00 2024-03-23T03:44:31.133 INFO:teuthology.orchestra.run.smithi102.stdout:(107/156): python3-cherrypy-18.6.1-2.el9.noarch 662 kB/s | 358 kB 00:00 2024-03-23T03:44:31.158 INFO:teuthology.orchestra.run.smithi102.stdout:(108/156): python3-google-auth-2.28.2-1.el9.noa 641 kB/s | 203 kB 00:00 2024-03-23T03:44:31.242 INFO:teuthology.orchestra.run.smithi102.stdout:(109/156): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-03-23T03:44:31.267 INFO:teuthology.orchestra.run.smithi102.stdout:(110/156): python3-jaraco-classes-3.2.1-5.el9.n 164 kB/s | 18 kB 00:00 2024-03-23T03:44:31.351 INFO:teuthology.orchestra.run.smithi102.stdout:(111/156): python3-jaraco-collections-3.0.0-8.e 214 kB/s | 23 kB 00:00 2024-03-23T03:44:31.377 INFO:teuthology.orchestra.run.smithi102.stdout:(112/156): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-03-23T03:44:31.420 INFO:teuthology.orchestra.run.smithi173.stdout:(95/156): fmt-8.1.1-5.el9.x86_64.rpm 193 kB/s | 111 kB 00:00 2024-03-23T03:44:31.460 INFO:teuthology.orchestra.run.smithi102.stdout:(113/156): python3-jaraco-text-3.2.0-6.el9.noar 182 kB/s | 20 kB 00:00 2024-03-23T03:44:31.577 INFO:teuthology.orchestra.run.smithi102.stdout:(114/156): python3-logutils-0.3.5-21.el9.noarch 398 kB/s | 46 kB 00:00 2024-03-23T03:44:31.794 INFO:teuthology.orchestra.run.smithi102.stdout:(115/156): python3-more-itertools-8.12.0-2.el9. 365 kB/s | 79 kB 00:00 2024-03-23T03:44:31.903 INFO:teuthology.orchestra.run.smithi102.stdout:(116/156): python3-natsort-7.1.1-5.el9.noarch.r 536 kB/s | 58 kB 00:00 2024-03-23T03:44:31.937 INFO:teuthology.orchestra.run.smithi173.stdout:(96/156): ceph-osd-18.2.2-954.g82316b04.el9.x86 1.6 MB/s | 16 MB 00:10 2024-03-23T03:44:31.970 INFO:teuthology.orchestra.run.smithi173.stdout:(97/156): gperftools-libs-2.9.1-2.el9.x86_64.rp 287 kB/s | 309 kB 00:01 2024-03-23T03:44:32.095 INFO:teuthology.orchestra.run.smithi173.stdout:(98/156): liboath-2.6.7-2.el9.x86_64.rpm 391 kB/s | 49 kB 00:00 2024-03-23T03:44:32.212 INFO:teuthology.orchestra.run.smithi173.stdout:(99/156): libunwind-1.6.2-1.el9.x86_64.rpm 580 kB/s | 67 kB 00:00 2024-03-23T03:44:32.238 INFO:teuthology.orchestra.run.smithi173.stdout:(100/156): libarrow-doc-9.0.0-10.el9.noarch.rpm 87 kB/s | 26 kB 00:00 2024-03-23T03:44:32.311 INFO:teuthology.orchestra.run.smithi102.stdout:(117/156): python3-pecan-1.4.2-3.el9.noarch.rpm 667 kB/s | 272 kB 00:00 2024-03-23T03:44:32.428 INFO:teuthology.orchestra.run.smithi102.stdout:(118/156): python3-portend-3.1.0-2.el9.noarch.r 142 kB/s | 16 kB 00:00 2024-03-23T03:44:32.620 INFO:teuthology.orchestra.run.smithi102.stdout:(119/156): python3-kubernetes-26.1.0-2.el9.noar 841 kB/s | 1.0 MB 00:01 2024-03-23T03:44:32.645 INFO:teuthology.orchestra.run.smithi102.stdout:(120/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 416 kB/s | 90 kB 00:00 2024-03-23T03:44:32.762 INFO:teuthology.orchestra.run.smithi102.stdout:(121/156): python3-repoze-lru-0.7-16.el9.noarch 218 kB/s | 31 kB 00:00 2024-03-23T03:44:32.879 INFO:teuthology.orchestra.run.smithi102.stdout:(122/156): python3-rsa-4.9-2.el9.noarch.rpm 509 kB/s | 59 kB 00:00 2024-03-23T03:44:32.938 INFO:teuthology.orchestra.run.smithi102.stdout:(123/156): python3-routes-2.5.1-5.el9.noarch.rp 646 kB/s | 188 kB 00:00 2024-03-23T03:44:32.988 INFO:teuthology.orchestra.run.smithi102.stdout:(124/156): python3-tempora-5.0.0-2.el9.noarch.r 331 kB/s | 36 kB 00:00 2024-03-23T03:44:33.047 INFO:teuthology.orchestra.run.smithi102.stdout:(125/156): python3-typing-extensions-4.4.0-2.el 472 kB/s | 51 kB 00:00 2024-03-23T03:44:33.246 INFO:teuthology.orchestra.run.smithi173.stdout:(101/156): parquet-libs-9.0.0-10.el9.x86_64.rpm 811 kB/s | 838 kB 00:01 2024-03-23T03:44:33.280 INFO:teuthology.orchestra.run.smithi173.stdout:(102/156): python3-asyncssh-2.13.2-5.el9.noarch 526 kB/s | 548 kB 00:01 2024-03-23T03:44:33.280 INFO:teuthology.orchestra.run.smithi102.stdout:(126/156): python3-websocket-client-1.2.3-2.el9 385 kB/s | 90 kB 00:00 2024-03-23T03:44:33.314 INFO:teuthology.orchestra.run.smithi102.stdout:(127/156): python3-webob-1.8.7-6.el9.noarch.rpm 706 kB/s | 230 kB 00:00 2024-03-23T03:44:33.380 INFO:teuthology.orchestra.run.smithi173.stdout:(103/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 327 kB/s | 43 kB 00:00 2024-03-23T03:44:33.405 INFO:teuthology.orchestra.run.smithi173.stdout:(104/156): python3-cachetools-4.2.4-1.el9.noarc 258 kB/s | 32 kB 00:00 2024-03-23T03:44:33.431 INFO:teuthology.orchestra.run.smithi102.stdout:(128/156): python3-zc-lockfile-2.0-10.el9.noarc 172 kB/s | 20 kB 00:00 2024-03-23T03:44:33.506 INFO:teuthology.orchestra.run.smithi173.stdout:(105/156): python3-certifi-2021.10.8-2.el9.noar 120 kB/s | 15 kB 00:00 2024-03-23T03:44:33.714 INFO:teuthology.orchestra.run.smithi173.stdout:(106/156): python3-cheroot-8.6.0-4.el9.noarch.r 558 kB/s | 172 kB 00:00 2024-03-23T03:44:33.740 INFO:teuthology.orchestra.run.smithi102.stdout:(129/156): re2-20211101-3.el9.x86_64.rpm 630 kB/s | 192 kB 00:00 2024-03-23T03:44:33.848 INFO:teuthology.orchestra.run.smithi102.stdout:(130/156): python3-werkzeug-2.0.3-3.el9.1.noarc 754 kB/s | 427 kB 00:00 2024-03-23T03:44:34.023 INFO:teuthology.orchestra.run.smithi173.stdout:(107/156): python3-google-auth-2.28.2-1.el9.noa 659 kB/s | 203 kB 00:00 2024-03-23T03:44:34.056 INFO:teuthology.orchestra.run.smithi173.stdout:(108/156): python3-cherrypy-18.6.1-2.el9.noarch 652 kB/s | 358 kB 00:00 2024-03-23T03:44:34.132 INFO:teuthology.orchestra.run.smithi173.stdout:(109/156): python3-jaraco-8.2.1-3.el9.noarch.rp 99 kB/s | 11 kB 00:00 2024-03-23T03:44:34.165 INFO:teuthology.orchestra.run.smithi173.stdout:(110/156): python3-jaraco-classes-3.2.1-5.el9.n 164 kB/s | 18 kB 00:00 2024-03-23T03:44:34.257 INFO:teuthology.orchestra.run.smithi102.stdout:(131/156): libarrow-9.0.0-10.el9.x86_64.rpm 811 kB/s | 4.4 MB 00:05 2024-03-23T03:44:34.282 INFO:teuthology.orchestra.run.smithi173.stdout:(111/156): python3-jaraco-collections-3.0.0-8.e 155 kB/s | 23 kB 00:00 2024-03-23T03:44:34.308 INFO:teuthology.orchestra.run.smithi173.stdout:(112/156): python3-jaraco-functools-3.5.0-2.el9 137 kB/s | 19 kB 00:00 2024-03-23T03:44:34.400 INFO:teuthology.orchestra.run.smithi173.stdout:(113/156): python3-jaraco-text-3.2.0-6.el9.noar 169 kB/s | 20 kB 00:00 2024-03-23T03:44:34.517 INFO:teuthology.orchestra.run.smithi173.stdout:(114/156): python3-logutils-0.3.5-21.el9.noarch 399 kB/s | 46 kB 00:00 2024-03-23T03:44:34.549 INFO:teuthology.orchestra.run.smithi102.stdout:(132/156): librbd1-18.2.2-954.g82316b04.el9.x86 10 MB/s | 3.0 MB 00:00 2024-03-23T03:44:34.608 INFO:teuthology.orchestra.run.smithi102.stdout:(133/156): xfsdump-3.1.12-4.el9.x86_64.rpm 5.6 MB/s | 321 kB 00:00 2024-03-23T03:44:34.633 INFO:teuthology.orchestra.run.smithi173.stdout:(115/156): python3-more-itertools-8.12.0-2.el9. 679 kB/s | 79 kB 00:00 2024-03-23T03:44:34.750 INFO:teuthology.orchestra.run.smithi173.stdout:(116/156): python3-natsort-7.1.1-5.el9.noarch.r 496 kB/s | 58 kB 00:00 2024-03-23T03:44:34.783 INFO:teuthology.orchestra.run.smithi102.stdout:(134/156): NetworkManager-1.47.2-1.el9.x86_64.r 13 MB/s | 2.3 MB 00:00 2024-03-23T03:44:34.883 INFO:teuthology.orchestra.run.smithi102.stdout:(135/156): librados2-18.2.2-954.g82316b04.el9.x 3.2 MB/s | 3.3 MB 00:01 2024-03-23T03:44:34.950 INFO:teuthology.orchestra.run.smithi102.stdout:(136/156): NetworkManager-team-1.47.2-1.el9.x86 641 kB/s | 42 kB 00:00 2024-03-23T03:44:35.050 INFO:teuthology.orchestra.run.smithi102.stdout:(137/156): NetworkManager-libnm-1.47.2-1.el9.x8 6.9 MB/s | 1.8 MB 00:00 2024-03-23T03:44:35.093 INFO:teuthology.orchestra.run.smithi102.stdout:(138/156): NetworkManager-tui-1.47.2-1.el9.x86_ 1.7 MB/s | 248 kB 00:00 2024-03-23T03:44:35.140 INFO:teuthology.orchestra.run.smithi102.stdout:(139/156): acl-2.3.1-4.el9.x86_64.rpm 806 kB/s | 71 kB 00:00 2024-03-23T03:44:35.143 INFO:teuthology.orchestra.run.smithi173.stdout:(117/156): python3-pecan-1.4.2-3.el9.noarch.rpm 696 kB/s | 272 kB 00:00 2024-03-23T03:44:35.190 INFO:teuthology.orchestra.run.smithi102.stdout:(140/156): elfutils-debuginfod-client-0.190-2.e 377 kB/s | 36 kB 00:00 2024-03-23T03:44:35.249 INFO:teuthology.orchestra.run.smithi102.stdout:(141/156): elfutils-libelf-0.190-2.el9.x86_64.r 1.7 MB/s | 192 kB 00:00 2024-03-23T03:44:35.268 INFO:teuthology.orchestra.run.smithi173.stdout:(118/156): python3-portend-3.1.0-2.el9.noarch.r 132 kB/s | 16 kB 00:00 2024-03-23T03:44:35.282 INFO:teuthology.orchestra.run.smithi102.stdout:(142/156): elfutils-libs-0.190-2.el9.x86_64.rpm 2.8 MB/s | 256 kB 00:00 2024-03-23T03:44:35.325 INFO:teuthology.orchestra.run.smithi102.stdout:(143/156): libacl-2.3.1-4.el9.x86_64.rpm 308 kB/s | 23 kB 00:00 2024-03-23T03:44:35.375 INFO:teuthology.orchestra.run.smithi102.stdout:(144/156): libblkid-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 108 kB 00:00 2024-03-23T03:44:35.425 INFO:teuthology.orchestra.run.smithi102.stdout:(145/156): libfdisk-2.37.4-18.el9.x86_64.rpm 1.5 MB/s | 155 kB 00:00 2024-03-23T03:44:35.476 INFO:teuthology.orchestra.run.smithi102.stdout:(146/156): libmount-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 136 kB 00:00 2024-03-23T03:44:35.476 INFO:teuthology.orchestra.run.smithi173.stdout:(119/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 433 kB/s | 90 kB 00:00 2024-03-23T03:44:35.501 INFO:teuthology.orchestra.run.smithi102.stdout:(147/156): libsmartcols-2.37.4-18.el9.x86_64.rp 860 kB/s | 64 kB 00:00 2024-03-23T03:44:35.551 INFO:teuthology.orchestra.run.smithi102.stdout:(148/156): libuuid-2.37.4-18.el9.x86_64.rpm 384 kB/s | 29 kB 00:00 2024-03-23T03:44:35.610 INFO:teuthology.orchestra.run.smithi173.stdout:(120/156): python3-kubernetes-26.1.0-2.el9.noar 803 kB/s | 1.0 MB 00:01 2024-03-23T03:44:35.636 INFO:teuthology.orchestra.run.smithi173.stdout:(121/156): python3-repoze-lru-0.7-16.el9.noarch 194 kB/s | 31 kB 00:00 2024-03-23T03:44:35.685 INFO:teuthology.orchestra.run.smithi102.stdout:(149/156): openssl-3.0.7-27.el9.x86_64.rpm 6.6 MB/s | 1.2 MB 00:00 2024-03-23T03:44:35.761 INFO:teuthology.orchestra.run.smithi173.stdout:(122/156): python3-rsa-4.9-2.el9.noarch.rpm 475 kB/s | 59 kB 00:00 2024-03-23T03:44:35.785 INFO:teuthology.orchestra.run.smithi102.stdout:(150/156): openssl-libs-3.0.7-27.el9.x86_64.rpm 9.3 MB/s | 2.2 MB 00:00 2024-03-23T03:44:35.836 INFO:teuthology.orchestra.run.smithi102.stdout:(151/156): thrift-0.15.0-2.el9.x86_64.rpm 773 kB/s | 1.6 MB 00:02 2024-03-23T03:44:35.878 INFO:teuthology.orchestra.run.smithi173.stdout:(123/156): python3-tempora-5.0.0-2.el9.noarch.r 308 kB/s | 36 kB 00:00 2024-03-23T03:44:35.894 INFO:teuthology.orchestra.run.smithi102.stdout:(152/156): util-linux-user-2.37.4-18.el9.x86_64 543 kB/s | 31 kB 00:00 2024-03-23T03:44:35.912 INFO:teuthology.orchestra.run.smithi173.stdout:(124/156): python3-routes-2.5.1-5.el9.noarch.rp 628 kB/s | 188 kB 00:00 2024-03-23T03:44:35.944 INFO:teuthology.orchestra.run.smithi102.stdout:(153/156): util-linux-core-2.37.4-18.el9.x86_64 2.9 MB/s | 465 kB 00:00 2024-03-23T03:44:35.995 INFO:teuthology.orchestra.run.smithi173.stdout:(125/156): python3-typing-extensions-4.4.0-2.el 438 kB/s | 51 kB 00:00 2024-03-23T03:44:36.020 INFO:teuthology.orchestra.run.smithi102.stdout:(154/156): util-linux-2.37.4-18.el9.x86_64.rpm 6.9 MB/s | 2.3 MB 00:00 2024-03-23T03:44:36.170 INFO:teuthology.orchestra.run.smithi102.stdout:(155/156): xfsprogs-6.3.0-1.el9.x86_64.rpm 3.9 MB/s | 1.1 MB 00:00 2024-03-23T03:44:36.204 INFO:teuthology.orchestra.run.smithi173.stdout:(126/156): python3-websocket-client-1.2.3-2.el9 432 kB/s | 90 kB 00:00 2024-03-23T03:44:36.238 INFO:teuthology.orchestra.run.smithi173.stdout:(127/156): python3-webob-1.8.7-6.el9.noarch.rpm 706 kB/s | 230 kB 00:00 2024-03-23T03:44:36.253 INFO:teuthology.orchestra.run.smithi102.stdout:(156/156): libacl-devel-2.3.1-4.el9.x86_64.rpm 242 kB/s | 74 kB 00:00 2024-03-23T03:44:36.263 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-23T03:44:36.263 INFO:teuthology.orchestra.run.smithi102.stdout:Total 10 MB/s | 204 MB 00:19 2024-03-23T03:44:36.346 INFO:teuthology.orchestra.run.smithi173.stdout:(128/156): python3-zc-lockfile-2.0-10.el9.noarc 185 kB/s | 20 kB 00:00 2024-03-23T03:44:36.655 INFO:teuthology.orchestra.run.smithi173.stdout:(129/156): libarrow-9.0.0-10.el9.x86_64.rpm 864 kB/s | 4.4 MB 00:05 2024-03-23T03:44:36.688 INFO:teuthology.orchestra.run.smithi173.stdout:(130/156): re2-20211101-3.el9.x86_64.rpm 561 kB/s | 192 kB 00:00 2024-03-23T03:44:36.722 INFO:teuthology.orchestra.run.smithi173.stdout:(131/156): python3-werkzeug-2.0.3-3.el9.1.noarc 826 kB/s | 427 kB 00:00 2024-03-23T03:44:37.456 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-23T03:44:37.472 INFO:teuthology.orchestra.run.smithi173.stdout:(132/156): librbd1-18.2.2-954.g82316b04.el9.x86 4.0 MB/s | 3.0 MB 00:00 2024-03-23T03:44:37.540 INFO:teuthology.orchestra.run.smithi173.stdout:(133/156): xfsdump-3.1.12-4.el9.x86_64.rpm 4.8 MB/s | 321 kB 00:00 2024-03-23T03:44:37.820 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-23T03:44:37.821 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-23T03:44:37.865 INFO:teuthology.orchestra.run.smithi173.stdout:(134/156): librados2-18.2.2-954.g82316b04.el9.x 2.8 MB/s | 3.3 MB 00:01 2024-03-23T03:44:38.023 INFO:teuthology.orchestra.run.smithi173.stdout:(135/156): NetworkManager-1.47.2-1.el9.x86_64.r 4.8 MB/s | 2.3 MB 00:00 2024-03-23T03:44:38.090 INFO:teuthology.orchestra.run.smithi173.stdout:(136/156): NetworkManager-team-1.47.2-1.el9.x86 658 kB/s | 42 kB 00:00 2024-03-23T03:44:38.166 INFO:teuthology.orchestra.run.smithi173.stdout:(137/156): NetworkManager-tui-1.47.2-1.el9.x86_ 3.3 MB/s | 248 kB 00:00 2024-03-23T03:44:38.249 INFO:teuthology.orchestra.run.smithi173.stdout:(138/156): NetworkManager-libnm-1.47.2-1.el9.x8 4.8 MB/s | 1.8 MB 00:00 2024-03-23T03:44:38.274 INFO:teuthology.orchestra.run.smithi173.stdout:(139/156): acl-2.3.1-4.el9.x86_64.rpm 662 kB/s | 71 kB 00:00 2024-03-23T03:44:38.325 INFO:teuthology.orchestra.run.smithi173.stdout:(140/156): thrift-0.15.0-2.el9.x86_64.rpm 972 kB/s | 1.6 MB 00:01 2024-03-23T03:44:38.350 INFO:teuthology.orchestra.run.smithi173.stdout:(141/156): elfutils-debuginfod-client-0.190-2.e 365 kB/s | 36 kB 00:00 2024-03-23T03:44:38.383 INFO:teuthology.orchestra.run.smithi173.stdout:(142/156): elfutils-libelf-0.190-2.el9.x86_64.r 1.7 MB/s | 192 kB 00:00 2024-03-23T03:44:38.442 INFO:teuthology.orchestra.run.smithi173.stdout:(143/156): elfutils-libs-0.190-2.el9.x86_64.rpm 2.1 MB/s | 256 kB 00:00 2024-03-23T03:44:38.467 INFO:teuthology.orchestra.run.smithi173.stdout:(144/156): libacl-2.3.1-4.el9.x86_64.rpm 198 kB/s | 23 kB 00:00 2024-03-23T03:44:38.501 INFO:teuthology.orchestra.run.smithi173.stdout:(145/156): libblkid-2.37.4-18.el9.x86_64.rpm 931 kB/s | 108 kB 00:00 2024-03-23T03:44:38.535 INFO:teuthology.orchestra.run.smithi173.stdout:(146/156): libfdisk-2.37.4-18.el9.x86_64.rpm 1.7 MB/s | 155 kB 00:00 2024-03-23T03:44:38.568 INFO:teuthology.orchestra.run.smithi173.stdout:(147/156): libmount-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 136 kB 00:00 2024-03-23T03:44:38.594 INFO:teuthology.orchestra.run.smithi173.stdout:(148/156): libsmartcols-2.37.4-18.el9.x86_64.rp 693 kB/s | 64 kB 00:00 2024-03-23T03:44:38.619 INFO:teuthology.orchestra.run.smithi173.stdout:(149/156): libuuid-2.37.4-18.el9.x86_64.rpm 345 kB/s | 29 kB 00:00 2024-03-23T03:44:38.694 INFO:teuthology.orchestra.run.smithi173.stdout:(150/156): openssl-3.0.7-27.el9.x86_64.rpm 9.7 MB/s | 1.2 MB 00:00 2024-03-23T03:44:38.837 INFO:teuthology.orchestra.run.smithi173.stdout:(151/156): openssl-libs-3.0.7-27.el9.x86_64.rpm 8.9 MB/s | 2.2 MB 00:00 2024-03-23T03:44:38.887 INFO:teuthology.orchestra.run.smithi173.stdout:(152/156): util-linux-core-2.37.4-18.el9.x86_64 3.6 MB/s | 465 kB 00:00 2024-03-23T03:44:38.929 INFO:teuthology.orchestra.run.smithi173.stdout:(153/156): util-linux-user-2.37.4-18.el9.x86_64 341 kB/s | 31 kB 00:00 2024-03-23T03:44:39.004 INFO:teuthology.orchestra.run.smithi173.stdout:(154/156): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.2 MB/s | 1.1 MB 00:00 2024-03-23T03:44:39.029 INFO:teuthology.orchestra.run.smithi173.stdout:(155/156): libacl-devel-2.3.1-4.el9.x86_64.rpm 748 kB/s | 74 kB 00:00 2024-03-23T03:44:39.113 INFO:teuthology.orchestra.run.smithi173.stdout:(156/156): util-linux-2.37.4-18.el9.x86_64.rpm 4.6 MB/s | 2.3 MB 00:00 2024-03-23T03:44:39.126 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-23T03:44:39.127 INFO:teuthology.orchestra.run.smithi173.stdout:Total 9.3 MB/s | 204 MB 00:21 2024-03-23T03:44:39.995 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-23T03:44:39.995 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-23T03:44:40.381 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-23T03:44:40.749 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-23T03:44:40.749 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-23T03:44:42.908 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-23T03:44:42.908 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-23T03:44:46.761 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-23T03:44:46.870 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/180 2024-03-23T03:44:46.918 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/180 2024-03-23T03:44:46.956 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/180 2024-03-23T03:44:46.999 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/180 2024-03-23T03:44:47.095 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/180 2024-03-23T03:44:47.194 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/180 2024-03-23T03:44:47.248 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/180 2024-03-23T03:44:47.279 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/180 2024-03-23T03:44:47.366 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/180 2024-03-23T03:44:47.758 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/180 2024-03-23T03:44:47.803 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librados2-2:18.2.2-954.g82316b04.el9.x86_64 10/180 2024-03-23T03:44:47.958 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:18.2.2-954.g82316b04.el9.x86_64 10/180 2024-03-23T03:44:47.987 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs2-2:18.2.2-954.g82316b04.el9.x86_64 11/180 2024-03-23T03:44:48.109 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs2-2:18.2.2-954.g82316b04.el9.x86_64 11/180 2024-03-23T03:44:48.142 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rados-2:18.2.2-954.g82316b04.el9.x86_64 12/180 2024-03-23T03:44:48.201 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/180 2024-03-23T03:44:48.486 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/180 2024-03-23T03:44:48.543 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/180 2024-03-23T03:44:48.569 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephsqlite-2:18.2.2-954.g82316b04.el9.x86_64 16/180 2024-03-23T03:44:48.732 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephsqlite-2:18.2.2-954.g82316b04.el9.x86_64 16/180 2024-03-23T03:44:48.761 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libradosstriper1-2:18.2.2-954.g82316b04.el9.x86_ 17/180 2024-03-23T03:44:48.851 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libradosstriper1-2:18.2.2-954.g82316b04.el9.x86_ 17/180 2024-03-23T03:44:48.906 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/180 2024-03-23T03:44:48.953 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 19/180 2024-03-23T03:44:49.262 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 20/180 2024-03-23T03:44:49.435 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 21/180 2024-03-23T03:44:49.471 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-23T03:44:49.515 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 21/180 2024-03-23T03:44:49.598 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/180 2024-03-23T03:44:49.645 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/180 2024-03-23T03:44:49.691 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/180 2024-03-23T03:44:49.703 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 22/180 2024-03-23T03:44:49.736 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/180 2024-03-23T03:44:49.762 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 22/180 2024-03-23T03:44:49.820 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/180 2024-03-23T03:44:49.914 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/180 2024-03-23T03:44:49.964 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/180 2024-03-23T03:44:49.998 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/180 2024-03-23T03:44:50.174 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 22/180 2024-03-23T03:44:50.218 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 23/180 2024-03-23T03:44:50.255 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : liboath-2.6.7-2.el9.x86_64 24/180 2024-03-23T03:44:50.282 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/180 2024-03-23T03:44:50.315 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 25/180 2024-03-23T03:44:50.420 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 26/180 2024-03-23T03:44:50.484 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/180 2024-03-23T03:44:50.573 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/180 2024-03-23T03:44:50.671 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 29/180 2024-03-23T03:44:50.705 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/180 2024-03-23T03:44:50.715 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 30/180 2024-03-23T03:44:50.749 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 31/180 2024-03-23T03:44:50.780 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:18.2.2-954.g82316b04.el9.x86_64 10/180 2024-03-23T03:44:50.855 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 32/180 2024-03-23T03:44:50.895 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-common-2:18.2.2-954.g82316b04.el9.x 33/180 2024-03-23T03:44:50.948 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-argparse-2:18.2.2-954.g82316b04.el9 34/180 2024-03-23T03:44:50.957 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:18.2.2-954.g82316b04.el9.x86_64 10/180 2024-03-23T03:44:50.989 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:18.2.2-954.g82316b04.el9.x86_64 11/180 2024-03-23T03:44:51.059 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cephfs-2:18.2.2-954.g82316b04.el9.x86_64 35/180 2024-03-23T03:44:51.123 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:18.2.2-954.g82316b04.el9.x86_64 11/180 2024-03-23T03:44:51.157 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:18.2.2-954.g82316b04.el9.x86_64 12/180 2024-03-23T03:44:51.220 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/180 2024-03-23T03:44:51.234 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/180 2024-03-23T03:44:51.299 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/180 2024-03-23T03:44:51.500 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/180 2024-03-23T03:44:51.553 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/180 2024-03-23T03:44:51.578 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:18.2.2-954.g82316b04.el9.x86_64 16/180 2024-03-23T03:44:51.701 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:18.2.2-954.g82316b04.el9.x86_64 16/180 2024-03-23T03:44:51.727 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:18.2.2-954.g82316b04.el9.x86_ 17/180 2024-03-23T03:44:51.799 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:18.2.2-954.g82316b04.el9.x86_ 17/180 2024-03-23T03:44:51.847 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/180 2024-03-23T03:44:51.893 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 19/180 2024-03-23T03:44:51.977 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/180 2024-03-23T03:44:52.079 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/180 2024-03-23T03:44:52.185 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/180 2024-03-23T03:44:52.235 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 20/180 2024-03-23T03:44:52.239 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 41/180 2024-03-23T03:44:52.279 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 41/180 2024-03-23T03:44:52.346 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 42/180 2024-03-23T03:44:52.396 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados-devel-2:18.2.2-954.g82316b04.el9.x86_64 43/180 2024-03-23T03:44:52.419 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 21/180 2024-03-23T03:44:52.437 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/180 2024-03-23T03:44:52.491 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 21/180 2024-03-23T03:44:52.679 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 22/180 2024-03-23T03:44:52.730 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 22/180 2024-03-23T03:44:52.890 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 45/180 2024-03-23T03:44:53.106 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 46/180 2024-03-23T03:44:53.179 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 47/180 2024-03-23T03:44:53.194 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 22/180 2024-03-23T03:44:53.235 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 23/180 2024-03-23T03:44:53.237 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 48/180 2024-03-23T03:44:53.273 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 24/180 2024-03-23T03:44:53.350 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 25/180 2024-03-23T03:44:53.438 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/180 2024-03-23T03:44:53.471 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 26/180 2024-03-23T03:44:53.503 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 50/180 2024-03-23T03:44:53.534 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/180 2024-03-23T03:44:53.585 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/180 2024-03-23T03:44:53.652 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : re2-1:20211101-3.el9.x86_64 51/180 2024-03-23T03:44:53.685 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 29/180 2024-03-23T03:44:53.698 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 52/180 2024-03-23T03:44:53.757 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 30/180 2024-03-23T03:44:53.768 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 53/180 2024-03-23T03:44:53.792 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 31/180 2024-03-23T03:44:53.860 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/180 2024-03-23T03:44:53.899 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/180 2024-03-23T03:44:53.903 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 32/180 2024-03-23T03:44:53.930 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/180 2024-03-23T03:44:53.946 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:18.2.2-954.g82316b04.el9.x 33/180 2024-03-23T03:44:53.996 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:18.2.2-954.g82316b04.el9 34/180 2024-03-23T03:44:53.997 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/180 2024-03-23T03:44:54.053 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/180 2024-03-23T03:44:54.101 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:18.2.2-954.g82316b04.el9.x86_64 35/180 2024-03-23T03:44:54.112 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 59/180 2024-03-23T03:44:54.272 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/180 2024-03-23T03:44:54.305 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 60/180 2024-03-23T03:44:54.330 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/180 2024-03-23T03:44:54.347 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 61/180 2024-03-23T03:44:54.383 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 62/180 2024-03-23T03:44:54.528 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 63/180 2024-03-23T03:44:54.588 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 64/180 2024-03-23T03:44:54.635 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 65/180 2024-03-23T03:44:54.671 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 66/180 2024-03-23T03:44:54.710 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 67/180 2024-03-23T03:44:54.874 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fmt-8.1.1-5.el9.x86_64 68/180 2024-03-23T03:44:54.944 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 69/180 2024-03-23T03:44:55.014 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ply-3.11-14.el9.noarch 70/180 2024-03-23T03:44:55.022 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/180 2024-03-23T03:44:55.093 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 71/180 2024-03-23T03:44:55.122 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/180 2024-03-23T03:44:55.229 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 40/180 2024-03-23T03:44:55.288 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 41/180 2024-03-23T03:44:55.320 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 41/180 2024-03-23T03:44:55.341 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 72/180 2024-03-23T03:44:55.384 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 42/180 2024-03-23T03:44:55.418 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 73/180 2024-03-23T03:44:55.434 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:18.2.2-954.g82316b04.el9.x86_64 43/180 2024-03-23T03:44:55.477 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/180 2024-03-23T03:44:55.593 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 74/180 2024-03-23T03:44:55.636 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 75/180 2024-03-23T03:44:55.667 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 76/180 2024-03-23T03:44:55.697 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fuse-2.9.9-15.el9.x86_64 77/180 2024-03-23T03:44:55.736 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 78/180 2024-03-23T03:44:55.772 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 79/180 2024-03-23T03:44:55.823 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : uuid-1.6.2-55.el9.x86_64 80/180 2024-03-23T03:44:55.866 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 81/180 2024-03-23T03:44:55.902 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 45/180 2024-03-23T03:44:55.992 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 82/180 2024-03-23T03:44:56.039 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 83/180 2024-03-23T03:44:56.073 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 84/180 2024-03-23T03:44:56.107 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 46/180 2024-03-23T03:44:56.130 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 85/180 2024-03-23T03:44:56.168 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 86/180 2024-03-23T03:44:56.183 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 47/180 2024-03-23T03:44:56.247 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 48/180 2024-03-23T03:44:56.309 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 87/180 2024-03-23T03:44:56.370 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 88/180 2024-03-23T03:44:56.438 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 89/180 2024-03-23T03:44:56.441 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 49/180 2024-03-23T03:44:56.493 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 50/180 2024-03-23T03:44:56.586 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-packaging-20.9-5.el9.noarch 90/180 2024-03-23T03:44:56.633 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : re2-1:20211101-3.el9.x86_64 51/180 2024-03-23T03:44:56.649 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 91/180 2024-03-23T03:44:56.672 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 52/180 2024-03-23T03:44:56.745 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 53/180 2024-03-23T03:44:56.831 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 54/180 2024-03-23T03:44:56.880 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 55/180 2024-03-23T03:44:56.919 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 56/180 2024-03-23T03:44:56.987 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 57/180 2024-03-23T03:44:57.043 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 58/180 2024-03-23T03:44:57.102 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 59/180 2024-03-23T03:44:57.295 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 60/180 2024-03-23T03:44:57.345 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 61/180 2024-03-23T03:44:57.398 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 62/180 2024-03-23T03:44:57.526 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 63/180 2024-03-23T03:44:57.569 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 64/180 2024-03-23T03:44:57.599 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/180 2024-03-23T03:44:57.614 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 65/180 2024-03-23T03:44:57.651 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 66/180 2024-03-23T03:44:57.655 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 93/180 2024-03-23T03:44:57.688 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 94/180 2024-03-23T03:44:57.690 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 67/180 2024-03-23T03:44:57.721 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 95/180 2024-03-23T03:44:57.854 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fmt-8.1.1-5.el9.x86_64 68/180 2024-03-23T03:44:57.908 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 69/180 2024-03-23T03:44:57.966 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-0.3.21-2.el9.x86_64 96/180 2024-03-23T03:44:57.977 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 70/180 2024-03-23T03:44:58.001 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 97/180 2024-03-23T03:44:58.064 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 71/180 2024-03-23T03:44:58.100 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 98/180 2024-03-23T03:44:58.309 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 72/180 2024-03-23T03:44:58.370 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 73/180 2024-03-23T03:44:58.544 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 74/180 2024-03-23T03:44:58.592 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 75/180 2024-03-23T03:44:58.631 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 76/180 2024-03-23T03:44:58.670 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-15.el9.x86_64 77/180 2024-03-23T03:44:58.710 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 78/180 2024-03-23T03:44:58.764 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 79/180 2024-03-23T03:44:58.817 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : uuid-1.6.2-55.el9.x86_64 80/180 2024-03-23T03:44:58.865 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 81/180 2024-03-23T03:44:58.884 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 99/180 2024-03-23T03:44:58.991 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 82/180 2024-03-23T03:44:59.022 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 100/180 2024-03-23T03:44:59.045 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 83/180 2024-03-23T03:44:59.113 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 84/180 2024-03-23T03:44:59.181 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 85/180 2024-03-23T03:44:59.227 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 86/180 2024-03-23T03:44:59.373 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 87/180 2024-03-23T03:44:59.423 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 88/180 2024-03-23T03:44:59.474 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 89/180 2024-03-23T03:44:59.619 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-packaging-20.9-5.el9.noarch 90/180 2024-03-23T03:44:59.679 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 91/180 2024-03-23T03:45:00.535 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 101/180 2024-03-23T03:45:00.623 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 102/180 2024-03-23T03:45:00.637 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/180 2024-03-23T03:45:00.661 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 103/180 2024-03-23T03:45:00.692 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : jq-1.6-16.el9.x86_64 104/180 2024-03-23T03:45:00.694 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 93/180 2024-03-23T03:45:00.728 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 94/180 2024-03-23T03:45:00.732 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 105/180 2024-03-23T03:45:00.761 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 95/180 2024-03-23T03:45:01.013 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.21-2.el9.x86_64 96/180 2024-03-23T03:45:01.049 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 97/180 2024-03-23T03:45:01.090 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 106/180 2024-03-23T03:45:01.122 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librbd1-2:18.2.2-954.g82316b04.el9.x86_64 107/180 2024-03-23T03:45:01.140 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 98/180 2024-03-23T03:45:01.242 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:18.2.2-954.g82316b04.el9.x86_64 107/180 2024-03-23T03:45:01.270 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rbd-2:18.2.2-954.g82316b04.el9.x86_64 108/180 2024-03-23T03:45:01.324 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 109/180 2024-03-23T03:45:01.789 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 110/180 2024-03-23T03:45:01.817 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librgw2-2:18.2.2-954.g82316b04.el9.x86_64 111/180 2024-03-23T03:45:01.891 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librgw2-2:18.2.2-954.g82316b04.el9.x86_64 111/180 2024-03-23T03:45:01.910 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rgw-2:18.2.2-954.g82316b04.el9.x86_64 112/180 2024-03-23T03:45:01.924 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 99/180 2024-03-23T03:45:02.051 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 100/180 2024-03-23T03:45:03.571 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 101/180 2024-03-23T03:45:03.661 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 102/180 2024-03-23T03:45:03.699 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 103/180 2024-03-23T03:45:03.730 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : jq-1.6-16.el9.x86_64 104/180 2024-03-23T03:45:03.770 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 105/180 2024-03-23T03:45:04.138 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 106/180 2024-03-23T03:45:04.176 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:18.2.2-954.g82316b04.el9.x86_64 107/180 2024-03-23T03:45:04.297 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:18.2.2-954.g82316b04.el9.x86_64 107/180 2024-03-23T03:45:04.320 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:18.2.2-954.g82316b04.el9.x86_64 113/180 2024-03-23T03:45:04.324 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:18.2.2-954.g82316b04.el9.x86_64 108/180 2024-03-23T03:45:04.348 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-common-2:18.2.2-954.g82316b04.el9.x86_64 113/180 2024-03-23T03:45:04.379 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 109/180 2024-03-23T03:45:04.839 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 110/180 2024-03-23T03:45:04.872 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:18.2.2-954.g82316b04.el9.x86_64 111/180 2024-03-23T03:45:04.953 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:18.2.2-954.g82316b04.el9.x86_64 111/180 2024-03-23T03:45:04.968 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:18.2.2-954.g82316b04.el9.x86_64 113/180 2024-03-23T03:45:04.981 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:18.2.2-954.g82316b04.el9.x86_64 112/180 2024-03-23T03:45:05.294 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-base-2:18.2.2-954.g82316b04.el9.x86_64 114/180 2024-03-23T03:45:05.443 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-base-2:18.2.2-954.g82316b04.el9.x86_64 114/180 2024-03-23T03:45:05.444 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-23T03:45:05.444 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-23T03:45:05.444 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:05.486 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-selinux-2:18.2.2-954.g82316b04.el9.x86_64 115/180 2024-03-23T03:45:07.448 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:18.2.2-954.g82316b04.el9.x86_64 113/180 2024-03-23T03:45:07.486 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:18.2.2-954.g82316b04.el9.x86_64 113/180 2024-03-23T03:45:08.153 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:18.2.2-954.g82316b04.el9.x86_64 113/180 2024-03-23T03:45:08.198 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:18.2.2-954.g82316b04.el9.x86_64 114/180 2024-03-23T03:45:08.360 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:18.2.2-954.g82316b04.el9.x86_64 114/180 2024-03-23T03:45:08.360 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-23T03:45:08.360 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-23T03:45:08.360 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:08.407 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:18.2.2-954.g82316b04.el9.x86_64 115/180 2024-03-23T03:45:18.073 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-selinux-2:18.2.2-954.g82316b04.el9.x86_64 115/180 2024-03-23T03:45:18.074 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /sys 2024-03-23T03:45:18.074 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /proc 2024-03-23T03:45:18.074 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /mnt 2024-03-23T03:45:18.074 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /var/tmp 2024-03-23T03:45:18.074 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /home 2024-03-23T03:45:18.074 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /root 2024-03-23T03:45:18.074 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /tmp 2024-03-23T03:45:18.074 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:18.320 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-volume-2:18.2.2-954.g82316b04.el9.noarch 116/180 2024-03-23T03:45:18.369 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-volume-2:18.2.2-954.g82316b04.el9.noarch 116/180 2024-03-23T03:45:18.369 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:18.369 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-23T03:45:18.369 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:19.919 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-osd-2:18.2.2-954.g82316b04.el9.x86_64 117/180 2024-03-23T03:45:19.996 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-osd-2:18.2.2-954.g82316b04.el9.x86_64 117/180 2024-03-23T03:45:19.997 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:19.997 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-23T03:45:19.997 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T03:45:19.997 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T03:45:19.997 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:20.249 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mds-2:18.2.2-954.g82316b04.el9.x86_64 118/180 2024-03-23T03:45:20.331 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mds-2:18.2.2-954.g82316b04.el9.x86_64 118/180 2024-03-23T03:45:20.331 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:20.332 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-23T03:45:20.332 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T03:45:20.332 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T03:45:20.332 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:20.845 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mon-2:18.2.2-954.g82316b04.el9.x86_64 119/180 2024-03-23T03:45:20.918 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mon-2:18.2.2-954.g82316b04.el9.x86_64 119/180 2024-03-23T03:45:20.918 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:20.918 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-23T03:45:20.918 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T03:45:20.918 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T03:45:20.918 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:21.030 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : automake-1.16.2-8.el9.noarch 120/180 2024-03-23T03:45:21.461 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:18.2.2-954.g82316b04.el9.x86_64 115/180 2024-03-23T03:45:21.462 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-03-23T03:45:21.462 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-03-23T03:45:21.462 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-03-23T03:45:21.462 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-03-23T03:45:21.462 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-03-23T03:45:21.462 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-03-23T03:45:21.462 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-03-23T03:45:21.462 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:21.682 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:18.2.2-954.g82316b04.el9.noarch 121/180 2024-03-23T03:45:21.689 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-volume-2:18.2.2-954.g82316b04.el9.noarch 116/180 2024-03-23T03:45:21.715 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephadm-2:18.2.2-954.g82316b04.el9.noarch 121/180 2024-03-23T03:45:21.736 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-volume-2:18.2.2-954.g82316b04.el9.noarch 116/180 2024-03-23T03:45:21.737 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:21.737 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-23T03:45:21.737 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:21.760 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-954.g82316b04.el 122/180 2024-03-23T03:45:21.864 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-954.g82316b04.e 123/180 2024-03-23T03:45:21.893 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-954.g82316b04.el9.noar 124/180 2024-03-23T03:45:22.214 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-954.g82316b04.el9.noar 124/180 2024-03-23T03:45:22.252 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-954.g82316b04.el9.no 125/180 2024-03-23T03:45:23.339 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:18.2.2-954.g82316b04.el9.x86_64 117/180 2024-03-23T03:45:23.370 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-954.g82316b04.el9.no 125/180 2024-03-23T03:45:23.403 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-954.g8231 126/180 2024-03-23T03:45:23.424 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:18.2.2-954.g82316b04.el9.x86_64 117/180 2024-03-23T03:45:23.424 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:23.424 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-23T03:45:23.424 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T03:45:23.424 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T03:45:23.424 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:23.543 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-954.g8231 126/180 2024-03-23T03:45:23.648 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:18.2.2-954.g82316b04.el9.x86_64 118/180 2024-03-23T03:45:23.709 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-954.g82316b04.el9 127/180 2024-03-23T03:45:23.720 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:18.2.2-954.g82316b04.el9.x86_64 118/180 2024-03-23T03:45:23.720 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:23.721 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-23T03:45:23.721 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T03:45:23.721 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T03:45:23.721 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:23.746 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-2:18.2.2-954.g82316b04.el9.x86_64 128/180 2024-03-23T03:45:23.824 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-2:18.2.2-954.g82316b04.el9.x86_64 128/180 2024-03-23T03:45:23.824 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:23.824 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-23T03:45:23.824 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T03:45:23.824 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T03:45:23.824 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:23.881 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-rook-2:18.2.2-954.g82316b04.el9.noarch 129/180 2024-03-23T03:45:23.919 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-954.g82316b04.el9.noarch 129/180 2024-03-23T03:45:24.016 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-2:18.2.2-954.g82316b04.el9.x86_64 130/180 2024-03-23T03:45:24.207 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:18.2.2-954.g82316b04.el9.x86_64 119/180 2024-03-23T03:45:24.282 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:18.2.2-954.g82316b04.el9.x86_64 119/180 2024-03-23T03:45:24.282 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:24.282 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-23T03:45:24.282 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T03:45:24.282 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T03:45:24.282 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:24.392 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : automake-1.16.2-8.el9.noarch 120/180 2024-03-23T03:45:24.873 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libtool-2.4.6-45.el9.x86_64 131/180 2024-03-23T03:45:24.908 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-radosgw-2:18.2.2-954.g82316b04.el9.x86_64 132/180 2024-03-23T03:45:24.981 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-954.g82316b04.el9.x86_64 132/180 2024-03-23T03:45:24.982 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:24.982 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-23T03:45:24.982 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T03:45:24.982 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T03:45:24.982 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:24.986 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:18.2.2-954.g82316b04.el9.noarch 121/180 2024-03-23T03:45:25.019 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:18.2.2-954.g82316b04.el9.noarch 121/180 2024-03-23T03:45:25.050 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-954.g82316b 133/180 2024-03-23T03:45:25.065 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-954.g82316b04.el 122/180 2024-03-23T03:45:25.141 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-954.g82316b 133/180 2024-03-23T03:45:25.141 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:25.141 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-23T03:45:25.141 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:25.179 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-954.g82316b04.e 123/180 2024-03-23T03:45:25.226 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-mirror-2:18.2.2-954.g82316b04.el9.x86_64 134/180 2024-03-23T03:45:25.230 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-954.g82316b04.el9.noar 124/180 2024-03-23T03:45:25.326 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-954.g82316b04.el9.x86_64 134/180 2024-03-23T03:45:25.326 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:25.326 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-23T03:45:25.327 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T03:45:25.327 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T03:45:25.327 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:25.560 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-954.g82316b04.el9.noar 124/180 2024-03-23T03:45:25.598 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-954.g82316b04.el9.no 125/180 2024-03-23T03:45:25.677 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-mirror-2:18.2.2-954.g82316b04.el9.x86_64 135/180 2024-03-23T03:45:25.754 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-mirror-2:18.2.2-954.g82316b04.el9.x86_64 135/180 2024-03-23T03:45:25.754 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:25.754 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-23T03:45:25.754 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T03:45:25.754 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T03:45:25.754 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:26.740 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-954.g82316b04.el9.no 125/180 2024-03-23T03:45:26.773 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-954.g8231 126/180 2024-03-23T03:45:26.912 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-954.g8231 126/180 2024-03-23T03:45:27.094 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-954.g82316b04.el9 127/180 2024-03-23T03:45:27.125 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:18.2.2-954.g82316b04.el9.x86_64 128/180 2024-03-23T03:45:27.194 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:18.2.2-954.g82316b04.el9.x86_64 128/180 2024-03-23T03:45:27.194 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:27.194 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-23T03:45:27.194 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T03:45:27.194 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T03:45:27.195 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:27.251 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:18.2.2-954.g82316b04.el9.noarch 129/180 2024-03-23T03:45:27.289 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-954.g82316b04.el9.noarch 129/180 2024-03-23T03:45:27.344 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:18.2.2-954.g82316b04.el9.x86_64 130/180 2024-03-23T03:45:28.135 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libtool-2.4.6-45.el9.x86_64 131/180 2024-03-23T03:45:28.170 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:18.2.2-954.g82316b04.el9.x86_64 132/180 2024-03-23T03:45:28.238 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-954.g82316b04.el9.x86_64 132/180 2024-03-23T03:45:28.238 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:28.239 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-23T03:45:28.239 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T03:45:28.239 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T03:45:28.239 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:28.287 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-954.g82316b 133/180 2024-03-23T03:45:28.353 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-954.g82316b 133/180 2024-03-23T03:45:28.353 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:28.353 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-23T03:45:28.353 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:28.429 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-mirror-2:18.2.2-954.g82316b04.el9.x86_64 134/180 2024-03-23T03:45:28.506 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephfs-mirror-2:18.2.2-954.g82316b04.el9.x86_64 134/180 2024-03-23T03:45:28.506 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:28.506 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-23T03:45:28.506 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T03:45:28.507 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T03:45:28.507 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:28.871 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:18.2.2-954.g82316b04.el9.x86_64 135/180 2024-03-23T03:45:28.940 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:18.2.2-954.g82316b04.el9.x86_64 135/180 2024-03-23T03:45:28.940 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T03:45:28.940 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-23T03:45:28.940 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T03:45:28.940 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T03:45:28.940 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:29.700 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-test-2:18.2.2-954.g82316b04.el9.x86_64 136/180 2024-03-23T03:45:29.755 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-fuse-2:18.2.2-954.g82316b04.el9.x86_64 137/180 2024-03-23T03:45:29.788 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-nbd-2:18.2.2-954.g82316b04.el9.x86_64 138/180 2024-03-23T03:45:30.067 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 139/180 2024-03-23T03:45:30.194 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 140/180 2024-03-23T03:45:30.241 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-fuse-2:18.2.2-954.g82316b04.el9.x86_64 141/180 2024-03-23T03:45:30.296 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/180 2024-03-23T03:45:30.364 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 143/180 2024-03-23T03:45:30.408 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 144/180 2024-03-23T03:45:30.444 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs-devel-2:18.2.2-954.g82316b04.el9.x86_6 145/180 2024-03-23T03:45:30.489 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 146/180 2024-03-23T03:45:30.542 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 147/180 2024-03-23T03:45:30.629 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 148/180 2024-03-23T03:45:30.741 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 149/180 2024-03-23T03:45:30.806 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 150/180 2024-03-23T03:45:31.679 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 151/180 2024-03-23T03:45:31.721 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 151/180 2024-03-23T03:45:31.781 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 151/180 2024-03-23T03:45:32.987 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:18.2.2-954.g82316b04.el9.x86_64 136/180 2024-03-23T03:45:33.043 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:18.2.2-954.g82316b04.el9.x86_64 137/180 2024-03-23T03:45:33.075 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:18.2.2-954.g82316b04.el9.x86_64 138/180 2024-03-23T03:45:33.109 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-top-2:18.2.2-954.g82316b04.el9.noarch 152/180 2024-03-23T03:45:33.201 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 153/180 2024-03-23T03:45:33.249 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/180 2024-03-23T03:45:33.360 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 139/180 2024-03-23T03:45:33.377 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/180 2024-03-23T03:45:33.412 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/180 2024-03-23T03:45:33.468 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 157/180 2024-03-23T03:45:33.484 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 140/180 2024-03-23T03:45:33.534 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:18.2.2-954.g82316b04.el9.x86_64 141/180 2024-03-23T03:45:33.591 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/180 2024-03-23T03:45:33.668 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 143/180 2024-03-23T03:45:33.713 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 144/180 2024-03-23T03:45:33.732 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:18.2.2-954.g82316b04.el9.x86_6 145/180 2024-03-23T03:45:33.775 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 146/180 2024-03-23T03:45:33.826 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 147/180 2024-03-23T03:45:33.916 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 148/180 2024-03-23T03:45:33.965 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 158/180 2024-03-23T03:45:34.007 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 159/180 2024-03-23T03:45:34.007 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/180 2024-03-23T03:45:34.020 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 149/180 2024-03-23T03:45:34.050 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 150/180 2024-03-23T03:45:34.150 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/180 2024-03-23T03:45:34.200 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 161/180 2024-03-23T03:45:34.200 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 162/180 2024-03-23T03:45:34.329 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 162/180 2024-03-23T03:45:34.388 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 163/180 2024-03-23T03:45:34.433 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 164/180 2024-03-23T03:45:34.470 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 165/180 2024-03-23T03:45:34.502 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 166/180 2024-03-23T03:45:34.571 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 167/180 2024-03-23T03:45:34.631 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 168/180 2024-03-23T03:45:34.691 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 169/180 2024-03-23T03:45:34.725 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 170/180 2024-03-23T03:45:34.757 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 171/180 2024-03-23T03:45:34.791 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 172/180 2024-03-23T03:45:34.824 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 173/180 2024-03-23T03:45:34.861 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 174/180 2024-03-23T03:45:34.862 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 174/180 2024-03-23T03:45:34.942 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 151/180 2024-03-23T03:45:34.990 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 151/180 2024-03-23T03:45:35.048 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 151/180 2024-03-23T03:45:35.191 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 174/180 2024-03-23T03:45:35.192 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 175/180 2024-03-23T03:45:35.314 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 175/180 2024-03-23T03:45:35.344 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 176/180 2024-03-23T03:45:35.377 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 177/180 2024-03-23T03:45:35.414 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 178/180 2024-03-23T03:45:35.453 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 179/180 2024-03-23T03:45:35.574 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 180/180 2024-03-23T03:45:35.848 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 180/180 2024-03-23T03:45:35.848 INFO:teuthology.orchestra.run.smithi102.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-23T03:45:35.848 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:36.388 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephfs-top-2:18.2.2-954.g82316b04.el9.noarch 152/180 2024-03-23T03:45:36.478 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 153/180 2024-03-23T03:45:36.524 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/180 2024-03-23T03:45:36.653 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/180 2024-03-23T03:45:36.688 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/180 2024-03-23T03:45:36.722 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 157/180 2024-03-23T03:45:36.756 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 158/180 2024-03-23T03:45:36.796 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 159/180 2024-03-23T03:45:36.796 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 160/180 2024-03-23T03:45:36.951 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 160/180 2024-03-23T03:45:36.994 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 161/180 2024-03-23T03:45:36.994 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 162/180 2024-03-23T03:45:37.123 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 162/180 2024-03-23T03:45:37.168 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 163/180 2024-03-23T03:45:37.222 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 164/180 2024-03-23T03:45:37.255 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 165/180 2024-03-23T03:45:37.289 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 166/180 2024-03-23T03:45:37.358 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 167/180 2024-03-23T03:45:37.427 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 168/180 2024-03-23T03:45:37.469 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 169/180 2024-03-23T03:45:37.502 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 170/180 2024-03-23T03:45:37.535 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 171/180 2024-03-23T03:45:37.569 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 172/180 2024-03-23T03:45:37.603 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 173/180 2024-03-23T03:45:37.639 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 174/180 2024-03-23T03:45:37.639 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 174/180 2024-03-23T03:45:37.969 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 174/180 2024-03-23T03:45:37.970 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 175/180 2024-03-23T03:45:38.109 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 175/180 2024-03-23T03:45:38.139 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 176/180 2024-03-23T03:45:38.172 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 177/180 2024-03-23T03:45:38.217 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 178/180 2024-03-23T03:45:38.265 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 179/180 2024-03-23T03:45:38.386 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 180/180 2024-03-23T03:45:38.648 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 180/180 2024-03-23T03:45:38.648 INFO:teuthology.orchestra.run.smithi173.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-23T03:45:38.648 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:40.167 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-2:18.2.2-954.g82316b04.el9.x86_64 1/180 2024-03-23T03:45:40.168 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-base-2:18.2.2-954.g82316b04.el9.x86_64 2/180 2024-03-23T03:45:40.168 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-common-2:18.2.2-954.g82316b04.el9.x86_64 3/180 2024-03-23T03:45:40.168 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-fuse-2:18.2.2-954.g82316b04.el9.x86_64 4/180 2024-03-23T03:45:40.168 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-954.g82316b 5/180 2024-03-23T03:45:40.168 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mds-2:18.2.2-954.g82316b04.el9.x86_64 6/180 2024-03-23T03:45:40.168 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-2:18.2.2-954.g82316b04.el9.x86_64 7/180 2024-03-23T03:45:40.168 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mon-2:18.2.2-954.g82316b04.el9.x86_64 8/180 2024-03-23T03:45:40.168 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-osd-2:18.2.2-954.g82316b04.el9.x86_64 9/180 2024-03-23T03:45:40.168 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-radosgw-2:18.2.2-954.g82316b04.el9.x86_64 10/180 2024-03-23T03:45:40.168 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-selinux-2:18.2.2-954.g82316b04.el9.x86_64 11/180 2024-03-23T03:45:40.168 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-test-2:18.2.2-954.g82316b04.el9.x86_64 12/180 2024-03-23T03:45:40.168 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-mirror-2:18.2.2-954.g82316b04.el9.x86_64 13/180 2024-03-23T03:45:40.168 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs-devel-2:18.2.2-954.g82316b04.el9.x86_6 14/180 2024-03-23T03:45:40.168 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs2-2:18.2.2-954.g82316b04.el9.x86_64 15/180 2024-03-23T03:45:40.169 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephsqlite-2:18.2.2-954.g82316b04.el9.x86_64 16/180 2024-03-23T03:45:40.169 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados-devel-2:18.2.2-954.g82316b04.el9.x86_64 17/180 2024-03-23T03:45:40.169 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libradosstriper1-2:18.2.2-954.g82316b04.el9.x86_ 18/180 2024-03-23T03:45:40.169 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librgw2-2:18.2.2-954.g82316b04.el9.x86_64 19/180 2024-03-23T03:45:40.169 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-argparse-2:18.2.2-954.g82316b04.el9 20/180 2024-03-23T03:45:40.169 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-common-2:18.2.2-954.g82316b04.el9.x 21/180 2024-03-23T03:45:40.169 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cephfs-2:18.2.2-954.g82316b04.el9.x86_64 22/180 2024-03-23T03:45:40.169 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rados-2:18.2.2-954.g82316b04.el9.x86_64 23/180 2024-03-23T03:45:40.169 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rbd-2:18.2.2-954.g82316b04.el9.x86_64 24/180 2024-03-23T03:45:40.169 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rgw-2:18.2.2-954.g82316b04.el9.x86_64 25/180 2024-03-23T03:45:40.169 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-fuse-2:18.2.2-954.g82316b04.el9.x86_64 26/180 2024-03-23T03:45:40.169 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-mirror-2:18.2.2-954.g82316b04.el9.x86_64 27/180 2024-03-23T03:45:40.169 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-nbd-2:18.2.2-954.g82316b04.el9.x86_64 28/180 2024-03-23T03:45:40.183 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-954.g82316b04.e 29/180 2024-03-23T03:45:40.183 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-954.g82316b04.el9.noar 30/180 2024-03-23T03:45:40.183 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-954.g82316b04.el9.no 31/180 2024-03-23T03:45:40.183 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-954.g8231 32/180 2024-03-23T03:45:40.183 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-954.g82316b04.el9 33/180 2024-03-23T03:45:40.184 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-rook-2:18.2.2-954.g82316b04.el9.noarch 34/180 2024-03-23T03:45:40.184 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-954.g82316b04.el 35/180 2024-03-23T03:45:40.184 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-volume-2:18.2.2-954.g82316b04.el9.noarch 36/180 2024-03-23T03:45:40.184 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephadm-2:18.2.2-954.g82316b04.el9.noarch 37/180 2024-03-23T03:45:40.184 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-top-2:18.2.2-954.g82316b04.el9.noarch 38/180 2024-03-23T03:45:40.184 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/180 2024-03-23T03:45:40.184 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/180 2024-03-23T03:45:40.184 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/180 2024-03-23T03:45:40.184 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flex-2.6.4-9.el9.x86_64 42/180 2024-03-23T03:45:40.184 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 43/180 2024-03-23T03:45:40.184 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 44/180 2024-03-23T03:45:40.184 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 45/180 2024-03-23T03:45:40.184 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 46/180 2024-03-23T03:45:40.184 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 47/180 2024-03-23T03:45:40.184 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 48/180 2024-03-23T03:45:40.185 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 49/180 2024-03-23T03:45:40.185 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 50/180 2024-03-23T03:45:40.185 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 51/180 2024-03-23T03:45:40.185 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 52/180 2024-03-23T03:45:40.185 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 53/180 2024-03-23T03:45:40.185 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 54/180 2024-03-23T03:45:40.185 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 55/180 2024-03-23T03:45:40.185 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-13.11-1.el9.x86_64 56/180 2024-03-23T03:45:40.185 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 57/180 2024-03-23T03:45:40.185 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 58/180 2024-03-23T03:45:40.185 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 59/180 2024-03-23T03:45:40.185 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/180 2024-03-23T03:45:40.185 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/180 2024-03-23T03:45:40.186 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/180 2024-03-23T03:45:40.186 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/180 2024-03-23T03:45:40.186 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/180 2024-03-23T03:45:40.186 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/180 2024-03-23T03:45:40.186 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/180 2024-03-23T03:45:40.186 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/180 2024-03-23T03:45:40.186 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/180 2024-03-23T03:45:40.186 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/180 2024-03-23T03:45:40.186 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 70/180 2024-03-23T03:45:40.186 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 71/180 2024-03-23T03:45:40.186 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 72/180 2024-03-23T03:45:40.186 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 73/180 2024-03-23T03:45:40.186 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 74/180 2024-03-23T03:45:40.187 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 75/180 2024-03-23T03:45:40.187 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 76/180 2024-03-23T03:45:40.187 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 77/180 2024-03-23T03:45:40.187 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 78/180 2024-03-23T03:45:40.187 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 79/180 2024-03-23T03:45:40.187 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 80/180 2024-03-23T03:45:40.187 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 81/180 2024-03-23T03:45:40.187 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 82/180 2024-03-23T03:45:40.187 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ply-3.11-14.el9.noarch 83/180 2024-03-23T03:45:40.187 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 84/180 2024-03-23T03:45:40.187 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 85/180 2024-03-23T03:45:40.187 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : jq-1.6-16.el9.x86_64 86/180 2024-03-23T03:45:40.187 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 87/180 2024-03-23T03:45:40.188 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 88/180 2024-03-23T03:45:40.188 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 89/180 2024-03-23T03:45:40.188 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 90/180 2024-03-23T03:45:40.188 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 91/180 2024-03-23T03:45:40.188 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 92/180 2024-03-23T03:45:40.188 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 93/180 2024-03-23T03:45:40.188 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 94/180 2024-03-23T03:45:40.188 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 95/180 2024-03-23T03:45:40.188 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 96/180 2024-03-23T03:45:40.188 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/180 2024-03-23T03:45:40.188 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 98/180 2024-03-23T03:45:40.188 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 99/180 2024-03-23T03:45:40.188 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/180 2024-03-23T03:45:40.189 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/180 2024-03-23T03:45:40.189 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 102/180 2024-03-23T03:45:40.189 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 103/180 2024-03-23T03:45:40.189 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 104/180 2024-03-23T03:45:40.189 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 105/180 2024-03-23T03:45:40.189 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 106/180 2024-03-23T03:45:40.189 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 107/180 2024-03-23T03:45:40.189 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 108/180 2024-03-23T03:45:40.189 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 109/180 2024-03-23T03:45:40.189 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 110/180 2024-03-23T03:45:40.189 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 111/180 2024-03-23T03:45:40.189 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 112/180 2024-03-23T03:45:40.190 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 113/180 2024-03-23T03:45:40.190 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 114/180 2024-03-23T03:45:40.190 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 115/180 2024-03-23T03:45:40.190 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 116/180 2024-03-23T03:45:40.190 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 117/180 2024-03-23T03:45:40.190 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 118/180 2024-03-23T03:45:40.190 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 119/180 2024-03-23T03:45:40.190 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 120/180 2024-03-23T03:45:40.190 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/180 2024-03-23T03:45:40.190 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 122/180 2024-03-23T03:45:40.190 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 123/180 2024-03-23T03:45:40.190 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 124/180 2024-03-23T03:45:40.191 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 125/180 2024-03-23T03:45:40.191 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 126/180 2024-03-23T03:45:40.191 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 127/180 2024-03-23T03:45:40.191 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 128/180 2024-03-23T03:45:40.191 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 129/180 2024-03-23T03:45:40.191 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 130/180 2024-03-23T03:45:40.191 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : re2-1:20211101-3.el9.x86_64 131/180 2024-03-23T03:45:40.191 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 132/180 2024-03-23T03:45:40.191 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:18.2.2-954.g82316b04.el9.x86_64 133/180 2024-03-23T03:45:40.191 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 134/180 2024-03-23T03:45:40.191 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:18.2.2-954.g82316b04.el9.x86_64 135/180 2024-03-23T03:45:40.191 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 136/180 2024-03-23T03:45:40.191 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 137/180 2024-03-23T03:45:40.192 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 138/180 2024-03-23T03:45:40.192 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 139/180 2024-03-23T03:45:40.192 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 140/180 2024-03-23T03:45:40.192 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 141/180 2024-03-23T03:45:40.192 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 142/180 2024-03-23T03:45:40.192 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 143/180 2024-03-23T03:45:40.192 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 144/180 2024-03-23T03:45:40.192 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 145/180 2024-03-23T03:45:40.192 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 146/180 2024-03-23T03:45:40.192 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : acl-2.3.1-4.el9.x86_64 147/180 2024-03-23T03:45:40.192 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : acl-2.3.1-3.el9.x86_64 148/180 2024-03-23T03:45:40.192 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 149/180 2024-03-23T03:45:40.192 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 150/180 2024-03-23T03:45:40.193 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 151/180 2024-03-23T03:45:40.193 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 152/180 2024-03-23T03:45:40.193 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 153/180 2024-03-23T03:45:40.193 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 154/180 2024-03-23T03:45:40.193 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 155/180 2024-03-23T03:45:40.193 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 156/180 2024-03-23T03:45:40.193 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 157/180 2024-03-23T03:45:40.193 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 158/180 2024-03-23T03:45:40.193 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 159/180 2024-03-23T03:45:40.193 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 160/180 2024-03-23T03:45:40.193 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 161/180 2024-03-23T03:45:40.193 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 162/180 2024-03-23T03:45:40.193 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 163/180 2024-03-23T03:45:40.194 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 164/180 2024-03-23T03:45:40.194 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 165/180 2024-03-23T03:45:40.194 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 166/180 2024-03-23T03:45:40.194 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 167/180 2024-03-23T03:45:40.194 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 168/180 2024-03-23T03:45:40.194 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 169/180 2024-03-23T03:45:40.194 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 170/180 2024-03-23T03:45:40.194 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 171/180 2024-03-23T03:45:40.194 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 172/180 2024-03-23T03:45:40.194 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 173/180 2024-03-23T03:45:40.194 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 174/180 2024-03-23T03:45:40.194 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 175/180 2024-03-23T03:45:40.194 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 176/180 2024-03-23T03:45:40.194 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 177/180 2024-03-23T03:45:40.195 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 178/180 2024-03-23T03:45:40.195 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 179/180 2024-03-23T03:45:42.453 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 180/180 2024-03-23T03:45:42.453 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:42.453 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2024-03-23T03:45:42.453 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-23T03:45:42.453 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-23T03:45:42.454 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-23T03:45:42.454 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-23T03:45:42.454 INFO:teuthology.orchestra.run.smithi102.stdout: acl-2.3.1-4.el9.x86_64 2024-03-23T03:45:42.454 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-23T03:45:42.454 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-23T03:45:42.454 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-23T03:45:42.454 INFO:teuthology.orchestra.run.smithi102.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-23T03:45:42.454 INFO:teuthology.orchestra.run.smithi102.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-23T03:45:42.454 INFO:teuthology.orchestra.run.smithi102.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-23T03:45:42.454 INFO:teuthology.orchestra.run.smithi102.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-23T03:45:42.454 INFO:teuthology.orchestra.run.smithi102.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-23T03:45:42.454 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.454 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.454 INFO:teuthology.orchestra.run.smithi102.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-23T03:45:42.455 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-23T03:45:42.455 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-23T03:45:42.455 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-23T03:45:42.455 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-23T03:45:42.455 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-23T03:45:42.455 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-23T03:45:42.455 INFO:teuthology.orchestra.run.smithi102.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-23T03:45:42.455 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-23T03:45:42.455 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-23T03:45:42.455 INFO:teuthology.orchestra.run.smithi102.stdout: automake-1.16.2-8.el9.noarch 2024-03-23T03:45:42.455 INFO:teuthology.orchestra.run.smithi102.stdout: bison-3.7.4-5.el9.x86_64 2024-03-23T03:45:42.455 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-23T03:45:42.455 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.455 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.456 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.456 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.456 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:42.456 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.456 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.456 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.456 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:42.456 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:42.456 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:42.456 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:42.456 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:42.456 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.456 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.457 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:42.457 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.457 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.457 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.457 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:42.457 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:42.457 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-mirror-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.457 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-top-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:42.457 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-23T03:45:42.457 INFO:teuthology.orchestra.run.smithi102.stdout: flex-2.6.4-9.el9.x86_64 2024-03-23T03:45:42.457 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-23T03:45:42.457 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-23T03:45:42.457 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-23T03:45:42.457 INFO:teuthology.orchestra.run.smithi102.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-23T03:45:42.458 INFO:teuthology.orchestra.run.smithi102.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-23T03:45:42.458 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-23T03:45:42.458 INFO:teuthology.orchestra.run.smithi102.stdout: jq-1.6-16.el9.x86_64 2024-03-23T03:45:42.458 INFO:teuthology.orchestra.run.smithi102.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-23T03:45:42.458 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-23T03:45:42.458 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-23T03:45:42.458 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.458 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.458 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.458 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-23T03:45:42.458 INFO:teuthology.orchestra.run.smithi102.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-23T03:45:42.458 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-23T03:45:42.458 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-23T03:45:42.459 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-23T03:45:42.459 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.459 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.459 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-23T03:45:42.459 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.459 INFO:teuthology.orchestra.run.smithi102.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-23T03:45:42.459 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-23T03:45:42.459 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-23T03:45:42.459 INFO:teuthology.orchestra.run.smithi102.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-23T03:45:42.459 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-23T03:45:42.459 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-23T03:45:42.460 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-23T03:45:42.460 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-23T03:45:42.460 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-23T03:45:42.460 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-23T03:45:42.460 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-13.11-1.el9.x86_64 2024-03-23T03:45:42.460 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-23T03:45:42.460 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-23T03:45:42.460 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-23T03:45:42.460 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-23T03:45:42.460 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-23T03:45:42.460 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-23T03:45:42.460 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-23T03:45:42.461 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.461 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.461 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.461 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-23T03:45:42.461 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-23T03:45:42.461 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-23T03:45:42.461 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-23T03:45:42.461 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-23T03:45:42.461 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-23T03:45:42.461 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-23T03:45:42.461 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-23T03:45:42.461 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-23T03:45:42.461 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-23T03:45:42.462 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-23T03:45:42.462 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-23T03:45:42.462 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-23T03:45:42.462 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-23T03:45:42.462 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-23T03:45:42.462 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-23T03:45:42.462 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-23T03:45:42.462 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-23T03:45:42.462 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-23T03:45:42.462 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-23T03:45:42.462 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-23T03:45:42.462 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-23T03:45:42.462 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-23T03:45:42.462 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-23T03:45:42.463 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-23T03:45:42.463 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-23T03:45:42.463 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply-3.11-14.el9.noarch 2024-03-23T03:45:42.463 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-23T03:45:42.463 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-23T03:45:42.463 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-23T03:45:42.463 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-23T03:45:42.463 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-23T03:45:42.463 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-23T03:45:42.463 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-23T03:45:42.463 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.463 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.463 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-23T03:45:42.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-23T03:45:42.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-23T03:45:42.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-23T03:45:42.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-23T03:45:42.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-23T03:45:42.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-23T03:45:42.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-23T03:45:42.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-23T03:45:42.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-23T03:45:42.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-23T03:45:42.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-23T03:45:42.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-23T03:45:42.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-23T03:45:42.465 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.465 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.465 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:42.465 INFO:teuthology.orchestra.run.smithi102.stdout: re2-1:20211101-3.el9.x86_64 2024-03-23T03:45:42.465 INFO:teuthology.orchestra.run.smithi102.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-23T03:45:42.465 INFO:teuthology.orchestra.run.smithi102.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-23T03:45:42.465 INFO:teuthology.orchestra.run.smithi102.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-23T03:45:42.465 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-23T03:45:42.465 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-23T03:45:42.465 INFO:teuthology.orchestra.run.smithi102.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-23T03:45:42.466 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:42.466 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-23T03:45:43.117 DEBUG:teuthology.parallel:result is None 2024-03-23T03:45:43.275 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:18.2.2-954.g82316b04.el9.x86_64 1/180 2024-03-23T03:45:43.275 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:18.2.2-954.g82316b04.el9.x86_64 2/180 2024-03-23T03:45:43.275 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:18.2.2-954.g82316b04.el9.x86_64 3/180 2024-03-23T03:45:43.275 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:18.2.2-954.g82316b04.el9.x86_64 4/180 2024-03-23T03:45:43.275 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-954.g82316b 5/180 2024-03-23T03:45:43.275 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:18.2.2-954.g82316b04.el9.x86_64 6/180 2024-03-23T03:45:43.275 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:18.2.2-954.g82316b04.el9.x86_64 7/180 2024-03-23T03:45:43.275 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:18.2.2-954.g82316b04.el9.x86_64 8/180 2024-03-23T03:45:43.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:18.2.2-954.g82316b04.el9.x86_64 9/180 2024-03-23T03:45:43.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:18.2.2-954.g82316b04.el9.x86_64 10/180 2024-03-23T03:45:43.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:18.2.2-954.g82316b04.el9.x86_64 11/180 2024-03-23T03:45:43.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:18.2.2-954.g82316b04.el9.x86_64 12/180 2024-03-23T03:45:43.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-mirror-2:18.2.2-954.g82316b04.el9.x86_64 13/180 2024-03-23T03:45:43.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:18.2.2-954.g82316b04.el9.x86_6 14/180 2024-03-23T03:45:43.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:18.2.2-954.g82316b04.el9.x86_64 15/180 2024-03-23T03:45:43.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:18.2.2-954.g82316b04.el9.x86_64 16/180 2024-03-23T03:45:43.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:18.2.2-954.g82316b04.el9.x86_64 17/180 2024-03-23T03:45:43.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:18.2.2-954.g82316b04.el9.x86_ 18/180 2024-03-23T03:45:43.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:18.2.2-954.g82316b04.el9.x86_64 19/180 2024-03-23T03:45:43.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:18.2.2-954.g82316b04.el9 20/180 2024-03-23T03:45:43.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:18.2.2-954.g82316b04.el9.x 21/180 2024-03-23T03:45:43.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:18.2.2-954.g82316b04.el9.x86_64 22/180 2024-03-23T03:45:43.276 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:18.2.2-954.g82316b04.el9.x86_64 23/180 2024-03-23T03:45:43.277 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:18.2.2-954.g82316b04.el9.x86_64 24/180 2024-03-23T03:45:43.277 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:18.2.2-954.g82316b04.el9.x86_64 25/180 2024-03-23T03:45:43.277 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:18.2.2-954.g82316b04.el9.x86_64 26/180 2024-03-23T03:45:43.277 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:18.2.2-954.g82316b04.el9.x86_64 27/180 2024-03-23T03:45:43.277 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:18.2.2-954.g82316b04.el9.x86_64 28/180 2024-03-23T03:45:43.277 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-954.g82316b04.e 29/180 2024-03-23T03:45:43.277 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-954.g82316b04.el9.noar 30/180 2024-03-23T03:45:43.277 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-954.g82316b04.el9.no 31/180 2024-03-23T03:45:43.277 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-954.g8231 32/180 2024-03-23T03:45:43.277 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-954.g82316b04.el9 33/180 2024-03-23T03:45:43.277 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:18.2.2-954.g82316b04.el9.noarch 34/180 2024-03-23T03:45:43.277 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-954.g82316b04.el 35/180 2024-03-23T03:45:43.277 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-volume-2:18.2.2-954.g82316b04.el9.noarch 36/180 2024-03-23T03:45:43.277 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:18.2.2-954.g82316b04.el9.noarch 37/180 2024-03-23T03:45:43.278 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephfs-top-2:18.2.2-954.g82316b04.el9.noarch 38/180 2024-03-23T03:45:43.278 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/180 2024-03-23T03:45:43.278 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/180 2024-03-23T03:45:43.278 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/180 2024-03-23T03:45:43.278 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flex-2.6.4-9.el9.x86_64 42/180 2024-03-23T03:45:43.278 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 43/180 2024-03-23T03:45:43.278 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 44/180 2024-03-23T03:45:43.278 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 45/180 2024-03-23T03:45:43.278 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 46/180 2024-03-23T03:45:43.278 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 47/180 2024-03-23T03:45:43.278 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 48/180 2024-03-23T03:45:43.278 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 49/180 2024-03-23T03:45:43.278 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 50/180 2024-03-23T03:45:43.279 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 51/180 2024-03-23T03:45:43.279 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 52/180 2024-03-23T03:45:43.279 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 53/180 2024-03-23T03:45:43.279 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 54/180 2024-03-23T03:45:43.279 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 55/180 2024-03-23T03:45:43.279 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-13.11-1.el9.x86_64 56/180 2024-03-23T03:45:43.279 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 57/180 2024-03-23T03:45:43.279 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 58/180 2024-03-23T03:45:43.279 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 59/180 2024-03-23T03:45:43.279 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 60/180 2024-03-23T03:45:43.279 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/180 2024-03-23T03:45:43.279 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/180 2024-03-23T03:45:43.279 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/180 2024-03-23T03:45:43.280 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/180 2024-03-23T03:45:43.280 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/180 2024-03-23T03:45:43.280 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/180 2024-03-23T03:45:43.280 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/180 2024-03-23T03:45:43.280 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 68/180 2024-03-23T03:45:43.280 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 69/180 2024-03-23T03:45:43.280 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 70/180 2024-03-23T03:45:43.280 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 71/180 2024-03-23T03:45:43.280 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 72/180 2024-03-23T03:45:43.280 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 73/180 2024-03-23T03:45:43.280 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 74/180 2024-03-23T03:45:43.280 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 75/180 2024-03-23T03:45:43.280 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 76/180 2024-03-23T03:45:43.280 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 77/180 2024-03-23T03:45:43.281 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 78/180 2024-03-23T03:45:43.281 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 79/180 2024-03-23T03:45:43.281 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 80/180 2024-03-23T03:45:43.281 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 81/180 2024-03-23T03:45:43.281 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 82/180 2024-03-23T03:45:43.281 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 83/180 2024-03-23T03:45:43.281 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 84/180 2024-03-23T03:45:43.281 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 85/180 2024-03-23T03:45:43.281 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : jq-1.6-16.el9.x86_64 86/180 2024-03-23T03:45:43.281 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 87/180 2024-03-23T03:45:43.281 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 88/180 2024-03-23T03:45:43.281 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 89/180 2024-03-23T03:45:43.281 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 90/180 2024-03-23T03:45:43.281 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 91/180 2024-03-23T03:45:43.281 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 92/180 2024-03-23T03:45:43.282 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 93/180 2024-03-23T03:45:43.282 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 94/180 2024-03-23T03:45:43.282 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 95/180 2024-03-23T03:45:43.282 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 96/180 2024-03-23T03:45:43.282 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/180 2024-03-23T03:45:43.282 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 98/180 2024-03-23T03:45:43.282 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 99/180 2024-03-23T03:45:43.282 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/180 2024-03-23T03:45:43.282 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/180 2024-03-23T03:45:43.282 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 102/180 2024-03-23T03:45:43.282 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 103/180 2024-03-23T03:45:43.282 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 104/180 2024-03-23T03:45:43.282 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 105/180 2024-03-23T03:45:43.283 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 106/180 2024-03-23T03:45:43.283 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 107/180 2024-03-23T03:45:43.283 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 108/180 2024-03-23T03:45:43.283 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 109/180 2024-03-23T03:45:43.283 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 110/180 2024-03-23T03:45:43.283 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 111/180 2024-03-23T03:45:43.283 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 112/180 2024-03-23T03:45:43.283 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 113/180 2024-03-23T03:45:43.283 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 114/180 2024-03-23T03:45:43.283 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 115/180 2024-03-23T03:45:43.283 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 116/180 2024-03-23T03:45:43.283 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 117/180 2024-03-23T03:45:43.283 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 118/180 2024-03-23T03:45:43.283 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 119/180 2024-03-23T03:45:43.284 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 120/180 2024-03-23T03:45:43.284 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/180 2024-03-23T03:45:43.284 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 122/180 2024-03-23T03:45:43.284 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 123/180 2024-03-23T03:45:43.284 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 124/180 2024-03-23T03:45:43.284 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 125/180 2024-03-23T03:45:43.284 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 126/180 2024-03-23T03:45:43.284 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 127/180 2024-03-23T03:45:43.284 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 128/180 2024-03-23T03:45:43.284 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 129/180 2024-03-23T03:45:43.284 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 130/180 2024-03-23T03:45:43.284 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : re2-1:20211101-3.el9.x86_64 131/180 2024-03-23T03:45:43.284 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 132/180 2024-03-23T03:45:43.284 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:18.2.2-954.g82316b04.el9.x86_64 133/180 2024-03-23T03:45:43.285 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 134/180 2024-03-23T03:45:43.285 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:18.2.2-954.g82316b04.el9.x86_64 135/180 2024-03-23T03:45:43.285 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 136/180 2024-03-23T03:45:43.285 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 137/180 2024-03-23T03:45:43.285 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 138/180 2024-03-23T03:45:43.285 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 139/180 2024-03-23T03:45:43.285 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 140/180 2024-03-23T03:45:43.285 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 141/180 2024-03-23T03:45:43.285 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 142/180 2024-03-23T03:45:43.285 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 143/180 2024-03-23T03:45:43.285 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 144/180 2024-03-23T03:45:43.285 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 145/180 2024-03-23T03:45:43.285 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 146/180 2024-03-23T03:45:43.285 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : acl-2.3.1-4.el9.x86_64 147/180 2024-03-23T03:45:43.286 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : acl-2.3.1-3.el9.x86_64 148/180 2024-03-23T03:45:43.286 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 149/180 2024-03-23T03:45:43.286 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 150/180 2024-03-23T03:45:43.286 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 151/180 2024-03-23T03:45:43.286 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 152/180 2024-03-23T03:45:43.286 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 153/180 2024-03-23T03:45:43.286 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 154/180 2024-03-23T03:45:43.286 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 155/180 2024-03-23T03:45:43.286 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 156/180 2024-03-23T03:45:43.286 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 157/180 2024-03-23T03:45:43.286 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 158/180 2024-03-23T03:45:43.286 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 159/180 2024-03-23T03:45:43.286 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 160/180 2024-03-23T03:45:43.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 161/180 2024-03-23T03:45:43.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 162/180 2024-03-23T03:45:43.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 163/180 2024-03-23T03:45:43.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 164/180 2024-03-23T03:45:43.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 165/180 2024-03-23T03:45:43.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 166/180 2024-03-23T03:45:43.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 167/180 2024-03-23T03:45:43.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 168/180 2024-03-23T03:45:43.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 169/180 2024-03-23T03:45:43.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 170/180 2024-03-23T03:45:43.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 171/180 2024-03-23T03:45:43.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 172/180 2024-03-23T03:45:43.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 173/180 2024-03-23T03:45:43.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 174/180 2024-03-23T03:45:43.287 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 175/180 2024-03-23T03:45:43.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 176/180 2024-03-23T03:45:43.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 177/180 2024-03-23T03:45:43.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 178/180 2024-03-23T03:45:43.288 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 179/180 2024-03-23T03:45:45.551 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 180/180 2024-03-23T03:45:45.551 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:45.551 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-03-23T03:45:45.551 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-23T03:45:45.551 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-23T03:45:45.551 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-23T03:45:45.551 INFO:teuthology.orchestra.run.smithi173.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-23T03:45:45.552 INFO:teuthology.orchestra.run.smithi173.stdout: acl-2.3.1-4.el9.x86_64 2024-03-23T03:45:45.552 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-23T03:45:45.552 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-23T03:45:45.552 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-23T03:45:45.552 INFO:teuthology.orchestra.run.smithi173.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-23T03:45:45.552 INFO:teuthology.orchestra.run.smithi173.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-23T03:45:45.552 INFO:teuthology.orchestra.run.smithi173.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-23T03:45:45.552 INFO:teuthology.orchestra.run.smithi173.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-23T03:45:45.552 INFO:teuthology.orchestra.run.smithi173.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-23T03:45:45.552 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.552 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.552 INFO:teuthology.orchestra.run.smithi173.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-23T03:45:45.552 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-23T03:45:45.553 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-23T03:45:45.553 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-23T03:45:45.553 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-23T03:45:45.553 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-23T03:45:45.553 INFO:teuthology.orchestra.run.smithi173.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-23T03:45:45.553 INFO:teuthology.orchestra.run.smithi173.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-23T03:45:45.553 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-23T03:45:45.553 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-23T03:45:45.553 INFO:teuthology.orchestra.run.smithi173.stdout: automake-1.16.2-8.el9.noarch 2024-03-23T03:45:45.553 INFO:teuthology.orchestra.run.smithi173.stdout: bison-3.7.4-5.el9.x86_64 2024-03-23T03:45:45.553 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-23T03:45:45.553 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.553 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.554 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.554 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.554 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:45.554 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.554 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.554 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.554 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:45.554 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:45.554 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:45.554 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:45.554 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:45.554 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.554 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.555 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:45.555 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.555 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.555 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.555 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:45.555 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:45.555 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-mirror-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.555 INFO:teuthology.orchestra.run.smithi173.stdout: cephfs-top-2:18.2.2-954.g82316b04.el9.noarch 2024-03-23T03:45:45.555 INFO:teuthology.orchestra.run.smithi173.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-23T03:45:45.555 INFO:teuthology.orchestra.run.smithi173.stdout: flex-2.6.4-9.el9.x86_64 2024-03-23T03:45:45.555 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-23T03:45:45.555 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-23T03:45:45.555 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-23T03:45:45.555 INFO:teuthology.orchestra.run.smithi173.stdout: fmt-8.1.1-5.el9.x86_64 2024-03-23T03:45:45.555 INFO:teuthology.orchestra.run.smithi173.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-23T03:45:45.556 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-23T03:45:45.556 INFO:teuthology.orchestra.run.smithi173.stdout: jq-1.6-16.el9.x86_64 2024-03-23T03:45:45.556 INFO:teuthology.orchestra.run.smithi173.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-23T03:45:45.556 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-23T03:45:45.556 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-23T03:45:45.556 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.556 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.556 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.556 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-23T03:45:45.556 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-23T03:45:45.556 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-23T03:45:45.556 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-23T03:45:45.556 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-23T03:45:45.556 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.557 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.557 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-23T03:45:45.557 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.557 INFO:teuthology.orchestra.run.smithi173.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-23T03:45:45.557 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-23T03:45:45.557 INFO:teuthology.orchestra.run.smithi173.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-23T03:45:45.557 INFO:teuthology.orchestra.run.smithi173.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-23T03:45:45.557 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-23T03:45:45.557 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-23T03:45:45.557 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-23T03:45:45.557 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-23T03:45:45.558 INFO:teuthology.orchestra.run.smithi173.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-23T03:45:45.558 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-23T03:45:45.558 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-13.11-1.el9.x86_64 2024-03-23T03:45:45.558 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-23T03:45:45.558 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-23T03:45:45.558 INFO:teuthology.orchestra.run.smithi173.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-23T03:45:45.558 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-23T03:45:45.559 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-23T03:45:45.559 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-23T03:45:45.559 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-23T03:45:45.559 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.559 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.559 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.559 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-23T03:45:45.559 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-23T03:45:45.559 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-23T03:45:45.559 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-23T03:45:45.560 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-23T03:45:45.560 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-23T03:45:45.560 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-23T03:45:45.560 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-23T03:45:45.560 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-23T03:45:45.560 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-23T03:45:45.560 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-23T03:45:45.560 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-23T03:45:45.560 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-23T03:45:45.560 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-23T03:45:45.560 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-23T03:45:45.561 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-23T03:45:45.561 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-23T03:45:45.561 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-23T03:45:45.561 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-23T03:45:45.561 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-23T03:45:45.561 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-23T03:45:45.561 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-23T03:45:45.561 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-23T03:45:45.561 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-23T03:45:45.561 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-23T03:45:45.561 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-23T03:45:45.561 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-03-23T03:45:45.561 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-23T03:45:45.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-23T03:45:45.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-23T03:45:45.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-23T03:45:45.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-23T03:45:45.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-23T03:45:45.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-23T03:45:45.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-23T03:45:45.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-23T03:45:45.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-23T03:45:45.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-23T03:45:45.562 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-23T03:45:45.563 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-23T03:45:45.563 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-23T03:45:45.563 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-23T03:45:45.563 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-23T03:45:45.563 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-23T03:45:45.563 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-23T03:45:45.563 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-23T03:45:45.563 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-23T03:45:45.563 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-23T03:45:45.563 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.563 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.563 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:18.2.2-954.g82316b04.el9.x86_64 2024-03-23T03:45:45.563 INFO:teuthology.orchestra.run.smithi173.stdout: re2-1:20211101-3.el9.x86_64 2024-03-23T03:45:45.563 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-23T03:45:45.564 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-23T03:45:45.564 INFO:teuthology.orchestra.run.smithi173.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-23T03:45:45.564 INFO:teuthology.orchestra.run.smithi173.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-23T03:45:45.564 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-23T03:45:45.564 INFO:teuthology.orchestra.run.smithi173.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-23T03:45:45.564 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:45.564 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-23T03:45:46.212 DEBUG:teuthology.parallel:result is None 2024-03-23T03:45:46.212 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=82316b04821a6c2c1d458a881920454bbadb8837 2024-03-23T03:45:46.355 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-23T03:45:46.409 INFO:teuthology.orchestra.run.smithi102.stdout:18.2.2-954.g82316b04.el9 2024-03-23T03:45:46.409 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-954.g82316b04.el9 2024-03-23T03:45:46.410 INFO:teuthology.task.install:The correct ceph version 18.2.2-954.g82316b04 is installed. 2024-03-23T03:45:46.412 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=82316b04821a6c2c1d458a881920454bbadb8837 2024-03-23T03:45:46.562 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-23T03:45:46.630 INFO:teuthology.orchestra.run.smithi173.stdout:18.2.2-954.g82316b04.el9 2024-03-23T03:45:46.631 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-954.g82316b04.el9 2024-03-23T03:45:46.631 INFO:teuthology.task.install:The correct ceph version 18.2.2-954.g82316b04 is installed. 2024-03-23T03:45:46.633 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-23T03:45:46.633 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:45:46.633 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-23T03:45:46.715 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:45:46.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-23T03:45:46.817 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-23T03:45:46.817 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:45:46.818 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/daemon-helper 2024-03-23T03:45:46.896 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-23T03:45:47.011 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:45:47.011 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-03-23T03:45:47.086 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-23T03:45:47.202 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-23T03:45:47.202 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:45:47.202 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-23T03:45:47.278 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-23T03:45:47.426 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:45:47.426 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-23T03:45:47.501 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-23T03:45:47.614 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-23T03:45:47.614 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:45:47.614 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/stdin-killer 2024-03-23T03:45:47.696 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-23T03:45:47.811 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:45:47.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-03-23T03:45:47.882 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-23T03:45:48.011 INFO:teuthology.run_tasks:Running task ceph... 2024-03-23T03:45:48.082 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-23T03:45:48.082 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /var/log/ceph 2024-03-23T03:45:48.088 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /var/log/ceph 2024-03-23T03:45:48.163 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-23T03:45:48.163 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-23T03:45:48.168 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-23T03:45:48.242 INFO:tasks.ceph:Creating extra log directories... 2024-03-23T03:45:48.243 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-23T03:45:48.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-23T03:45:48.328 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-23T03:45:48.328 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-23T03:45:48.328 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-23T03:45:48.328 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:45:48.328 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-23T03:45:48.377 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-23T03:45:48.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-23T03:45:48.736 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:45:48.736 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-23T03:45:48.781 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-23T03:45:49.005 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-23T03:45:49.082 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-23T03:45:49.082 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-23T03:45:49.082 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617203', 'branch': 'wip-yuri11-testing-2024-03-21-0851-reef', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/quota}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7617203', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri11-testing-2024-03-21-0851-reef'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'}}, 'kclient': {'syntax': 'v1'}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri11-testing-2024-03-21-0851-reef', 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 1919, 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'fs', 'suite_branch': 'wip-yuri11-testing-2024-03-21-0851-reef', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_82316b04821a6c2c1d458a881920454bbadb8837/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '82316b04821a6c2c1d458a881920454bbadb8837', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi102.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMTizH0wUqZ4U/b89MXw+0NgcpY4R8KU/gvKM8U+SnddorpI5l/0MjELps6rL3KkXG2xlUc9NiXI5iVz1sRTpFs=', 'smithi173.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDYfGpry3ANlMxAACapxS822P0GSJelsL/67tVqmYt77USJcr2GPvIQKVoKwjDDWC8XLqyAmDUKqDa+hhPpnnWg='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '82316b04821a6c2c1d458a881920454bbadb8837', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_quota']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-22_13:15:24', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-23T03:45:49.083 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-23T03:45:49.090 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-23T03:45:49.139 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m0777 -- /var/run/ceph 2024-03-23T03:45:49.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m0777 -- /var/run/ceph 2024-03-23T03:45:49.242 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:45:49.243 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-03-23T03:45:49.294 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-23T03:45:49.294 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 2024-03-23T03:45:49.338 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-23T03:45:49.338 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T03:45:49.338 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 631 Links: 1 2024-03-23T03:45:49.338 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T03:45:49.338 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T03:45:49.339 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-23 03:45:36.262631757 +0000 2024-03-23T03:45:49.339 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-23 03:43:17.441947405 +0000 2024-03-23T03:45:49.339 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-23 03:43:17.441947405 +0000 2024-03-23T03:45:49.339 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-23 03:43:17.441947405 +0000 2024-03-23T03:45:49.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-23T03:45:49.469 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-23T03:45:49.469 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-23T03:45:49.469 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000325559 s, 1.6 MB/s 2024-03-23T03:45:49.473 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-23T03:45:49.529 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 2024-03-23T03:45:49.577 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-23T03:45:49.577 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T03:45:49.577 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 653 Links: 1 2024-03-23T03:45:49.577 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T03:45:49.577 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T03:45:49.577 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-23 03:45:36.263631736 +0000 2024-03-23T03:45:49.577 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-23 03:43:17.987936311 +0000 2024-03-23T03:45:49.577 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-23 03:43:17.987936311 +0000 2024-03-23T03:45:49.577 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-23 03:43:17.987936311 +0000 2024-03-23T03:45:49.578 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-23T03:45:49.678 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-23T03:45:49.679 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-23T03:45:49.679 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000342697 s, 1.5 MB/s 2024-03-23T03:45:49.683 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-23T03:45:49.733 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 2024-03-23T03:45:49.777 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-23T03:45:49.777 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T03:45:49.777 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-03-23T03:45:49.777 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T03:45:49.777 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T03:45:49.777 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-23 03:45:36.264631716 +0000 2024-03-23T03:45:49.777 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-23 03:43:18.581924243 +0000 2024-03-23T03:45:49.777 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-23 03:43:18.581924243 +0000 2024-03-23T03:45:49.777 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-23 03:43:18.581924243 +0000 2024-03-23T03:45:49.778 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-23T03:45:49.899 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-23T03:45:49.899 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-23T03:45:49.899 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000270512 s, 1.9 MB/s 2024-03-23T03:45:49.903 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-23T03:45:49.954 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 2024-03-23T03:45:49.997 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-23T03:45:49.997 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T03:45:49.997 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 688 Links: 1 2024-03-23T03:45:49.997 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T03:45:49.997 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T03:45:49.997 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-23 03:45:36.265631696 +0000 2024-03-23T03:45:49.997 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-23 03:43:19.142912841 +0000 2024-03-23T03:45:49.997 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-23 03:43:19.142912841 +0000 2024-03-23T03:45:49.998 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-23 03:43:19.142912841 +0000 2024-03-23T03:45:49.998 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-23T03:45:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-23T03:45:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-23T03:45:50.121 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000343352 s, 1.5 MB/s 2024-03-23T03:45:50.125 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-23T03:45:50.171 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T03:45:50.171 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:45:50.171 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-03-23T03:45:50.215 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-23T03:45:50.215 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-03-23T03:45:50.295 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-23T03:45:50.295 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T03:45:50.295 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 632 Links: 1 2024-03-23T03:45:50.295 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T03:45:50.295 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T03:45:50.295 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-23 03:45:39.080880421 +0000 2024-03-23T03:45:50.295 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-23 03:43:17.442966262 +0000 2024-03-23T03:45:50.295 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-23 03:43:17.442966262 +0000 2024-03-23T03:45:50.295 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-23 03:43:17.442966262 +0000 2024-03-23T03:45:50.296 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-23T03:45:50.404 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-23T03:45:50.450 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-23T03:45:50.450 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000350266 s, 1.5 MB/s 2024-03-23T03:45:50.451 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-23T03:45:50.493 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-03-23T03:45:50.576 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-23T03:45:50.576 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T03:45:50.576 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 651 Links: 1 2024-03-23T03:45:50.576 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T03:45:50.576 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T03:45:50.577 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-23 03:45:39.080880421 +0000 2024-03-23T03:45:50.577 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-23 03:43:18.017953404 +0000 2024-03-23T03:45:50.577 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-23 03:43:18.017953404 +0000 2024-03-23T03:45:50.577 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-23 03:43:18.017953404 +0000 2024-03-23T03:45:50.577 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-23T03:45:50.689 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-23T03:45:50.689 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-23T03:45:50.689 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000327065 s, 1.6 MB/s 2024-03-23T03:45:50.692 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-23T03:45:50.733 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-03-23T03:45:50.812 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-23T03:45:50.812 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T03:45:50.812 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 671 Links: 1 2024-03-23T03:45:50.812 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T03:45:50.813 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T03:45:50.813 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-23 03:45:39.081880399 +0000 2024-03-23T03:45:50.813 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-23 03:43:18.596940457 +0000 2024-03-23T03:45:50.813 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-23 03:43:18.596940457 +0000 2024-03-23T03:45:50.813 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-23 03:43:18.596940457 +0000 2024-03-23T03:45:50.813 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-23T03:45:50.919 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-23T03:45:50.919 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-23T03:45:50.919 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.00031414 s, 1.6 MB/s 2024-03-23T03:45:50.922 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-23T03:45:50.962 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-03-23T03:45:51.042 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-23T03:45:51.042 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T03:45:51.042 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 689 Links: 1 2024-03-23T03:45:51.042 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T03:45:51.042 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T03:45:51.042 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-23 03:45:39.082880378 +0000 2024-03-23T03:45:51.042 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-23 03:43:19.168927667 +0000 2024-03-23T03:45:51.042 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-23 03:43:19.168927667 +0000 2024-03-23T03:45:51.042 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-23 03:43:19.168927667 +0000 2024-03-23T03:45:51.043 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-23T03:45:51.147 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-23T03:45:51.147 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-23T03:45:51.147 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000266751 s, 1.9 MB/s 2024-03-23T03:45:51.150 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-23T03:45:51.193 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T03:45:51.193 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi102.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi173.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-23T03:45:51.193 INFO:tasks.ceph:Generating config... 2024-03-23T03:45:51.194 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-23T03:45:51.194 INFO:tasks.ceph:[client] debug client = 20 2024-03-23T03:45:51.194 INFO:tasks.ceph:[client] debug ms = 1 2024-03-23T03:45:51.194 INFO:tasks.ceph:[client] pid file = 2024-03-23T03:45:51.194 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-23T03:45:51.194 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-23T03:45:51.194 INFO:tasks.ceph:[global] ms die on skipped message = False 2024-03-23T03:45:51.195 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-23T03:45:51.195 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-23T03:45:51.195 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-23T03:45:51.195 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-23T03:45:51.195 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-23T03:45:51.195 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-23T03:45:51.195 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-23T03:45:51.195 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-23T03:45:51.195 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-23T03:45:51.195 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-23T03:45:51.195 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-23T03:45:51.195 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-23T03:45:51.195 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-23T03:45:51.195 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-23T03:45:51.195 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-23T03:45:51.196 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-23T03:45:51.196 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-23T03:45:51.196 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-23T03:45:51.196 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-23T03:45:51.196 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-23T03:45:51.196 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-23T03:45:51.196 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-23T03:45:51.196 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-23T03:45:51.196 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-23T03:45:51.196 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-23T03:45:51.196 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-23T03:45:51.196 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-23T03:45:51.196 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-23T03:45:51.196 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-23T03:45:51.197 INFO:tasks.ceph:Setting up mon.a... 2024-03-23T03:45:51.197 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-23T03:45:51.312 INFO:teuthology.orchestra.run.smithi102.stdout:creating /etc/ceph/ceph.keyring 2024-03-23T03:45:51.320 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-23T03:45:51.442 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-23T03:45:51.525 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.102'), ('mon.b', '172.21.15.173'), ('mon.c', '[v2:172.21.15.173:3301,v1:172.21.15.173:6790]')] 2024-03-23T03:45:51.525 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.102,172.21.15.173,[v2:172.21.15.173:3301,v1:172.21.15.173:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-23T03:45:51.526 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:45:51.526 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-23T03:45:51.574 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.102 --add b 172.21.15.173 --addv c '[v2:172.21.15.173:3301,v1:172.21.15.173:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-23T03:45:51.724 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-23T03:45:51.724 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool: generated fsid 776b2820-05f2-4e50-8d40-18dd76c5f6a0 2024-03-23T03:45:51.724 INFO:teuthology.orchestra.run.smithi102.stdout:setting min_mon_release = pacific 2024-03-23T03:45:51.724 INFO:teuthology.orchestra.run.smithi102.stdout:epoch 0 2024-03-23T03:45:51.724 INFO:teuthology.orchestra.run.smithi102.stdout:fsid 776b2820-05f2-4e50-8d40-18dd76c5f6a0 2024-03-23T03:45:51.724 INFO:teuthology.orchestra.run.smithi102.stdout:last_changed 2024-03-23T03:45:51.724715+0000 2024-03-23T03:45:51.724 INFO:teuthology.orchestra.run.smithi102.stdout:created 2024-03-23T03:45:51.724715+0000 2024-03-23T03:45:51.725 INFO:teuthology.orchestra.run.smithi102.stdout:min_mon_release 16 (pacific) 2024-03-23T03:45:51.725 INFO:teuthology.orchestra.run.smithi102.stdout:election_strategy: 1 2024-03-23T03:45:51.725 INFO:teuthology.orchestra.run.smithi102.stdout:0: [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon.a 2024-03-23T03:45:51.725 INFO:teuthology.orchestra.run.smithi102.stdout:1: [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon.b 2024-03-23T03:45:51.725 INFO:teuthology.orchestra.run.smithi102.stdout:2: [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] mon.c 2024-03-23T03:45:51.725 INFO:teuthology.orchestra.run.smithi102.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-23T03:45:51.727 DEBUG:teuthology.orchestra.run.smithi102:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-23T03:45:51.780 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 776b2820-05f2-4e50-8d40-18dd76c5f6a0... 2024-03-23T03:45:51.781 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-23T03:45:51.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-23T03:45:51.951 INFO:teuthology.orchestra.run.smithi102.stdout:[global] 2024-03-23T03:45:51.951 INFO:teuthology.orchestra.run.smithi102.stdout: chdir = "" 2024-03-23T03:45:51.951 INFO:teuthology.orchestra.run.smithi102.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T03:45:51.951 INFO:teuthology.orchestra.run.smithi102.stdout: auth supported = cephx 2024-03-23T03:45:51.951 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.951 INFO:teuthology.orchestra.run.smithi102.stdout: filestore xattr use omap = true 2024-03-23T03:45:51.951 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.951 INFO:teuthology.orchestra.run.smithi102.stdout: mon clock drift allowed = 1.000 2024-03-23T03:45:51.951 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.952 INFO:teuthology.orchestra.run.smithi102.stdout: osd crush chooseleaf type = 0 2024-03-23T03:45:51.952 INFO:teuthology.orchestra.run.smithi102.stdout: auth debug = true 2024-03-23T03:45:51.952 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.952 INFO:teuthology.orchestra.run.smithi102.stdout: ms die on old message = true 2024-03-23T03:45:51.952 INFO:teuthology.orchestra.run.smithi102.stdout: ms die on bug = true 2024-03-23T03:45:51.952 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.952 INFO:teuthology.orchestra.run.smithi102.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T03:45:51.952 INFO:teuthology.orchestra.run.smithi102.stdout: mon pg warn max object skew = 0 2024-03-23T03:45:51.952 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.952 INFO:teuthology.orchestra.run.smithi102.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T03:45:51.952 INFO:teuthology.orchestra.run.smithi102.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T03:45:51.952 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.952 INFO:teuthology.orchestra.run.smithi102.stdout: osd pool default size = 2 2024-03-23T03:45:51.953 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.953 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd allow primary affinity = true 2024-03-23T03:45:51.953 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd allow pg remap = true 2024-03-23T03:45:51.953 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on legacy crush tunables = false 2024-03-23T03:45:51.953 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on crush straw calc version zero = false 2024-03-23T03:45:51.953 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on no sortbitwise = false 2024-03-23T03:45:51.953 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on osd down out interval zero = false 2024-03-23T03:45:51.953 INFO:teuthology.orchestra.run.smithi102.stdout: mon warn on too few osds = false 2024-03-23T03:45:51.953 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T03:45:51.953 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T03:45:51.953 INFO:teuthology.orchestra.run.smithi102.stdout: mon_allow_pool_size_one = true 2024-03-23T03:45:51.954 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.954 INFO:teuthology.orchestra.run.smithi102.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T03:45:51.954 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.954 INFO:teuthology.orchestra.run.smithi102.stdout: osd default data pool replay window = 5 2024-03-23T03:45:51.954 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.954 INFO:teuthology.orchestra.run.smithi102.stdout: mon allow pool delete = true 2024-03-23T03:45:51.954 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.954 INFO:teuthology.orchestra.run.smithi102.stdout: mon cluster log file level = debug 2024-03-23T03:45:51.954 INFO:teuthology.orchestra.run.smithi102.stdout: debug asserts on shutdown = true 2024-03-23T03:45:51.954 INFO:teuthology.orchestra.run.smithi102.stdout: mon health detail to clog = false 2024-03-23T03:45:51.954 INFO:teuthology.orchestra.run.smithi102.stdout: mon host = "172.21.15.102,172.21.15.173,[v2:172.21.15.173:3301,v1:172.21.15.173:6790]" 2024-03-23T03:45:51.955 INFO:teuthology.orchestra.run.smithi102.stdout: ms die on skipped message = False 2024-03-23T03:45:51.955 INFO:teuthology.orchestra.run.smithi102.stdout: fsid = 776b2820-05f2-4e50-8d40-18dd76c5f6a0 2024-03-23T03:45:51.955 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.955 INFO:teuthology.orchestra.run.smithi102.stdout:[osd] 2024-03-23T03:45:51.955 INFO:teuthology.orchestra.run.smithi102.stdout: osd journal size = 100 2024-03-23T03:45:51.955 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.955 INFO:teuthology.orchestra.run.smithi102.stdout: osd scrub load threshold = 5.0 2024-03-23T03:45:51.955 INFO:teuthology.orchestra.run.smithi102.stdout: osd scrub max interval = 600 2024-03-23T03:45:51.955 INFO:teuthology.orchestra.run.smithi102.stdout: osd mclock profile = high_recovery_ops 2024-03-23T03:45:51.955 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.955 INFO:teuthology.orchestra.run.smithi102.stdout: osd recover clone overlap = true 2024-03-23T03:45:51.955 INFO:teuthology.orchestra.run.smithi102.stdout: osd recovery max chunk = 1048576 2024-03-23T03:45:51.955 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.956 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug shutdown = true 2024-03-23T03:45:51.956 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug op order = true 2024-03-23T03:45:51.956 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug verify stray on activate = true 2024-03-23T03:45:51.956 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.956 INFO:teuthology.orchestra.run.smithi102.stdout: osd open classes on start = true 2024-03-23T03:45:51.956 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug pg log writeout = true 2024-03-23T03:45:51.956 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.956 INFO:teuthology.orchestra.run.smithi102.stdout: osd deep scrub update digest min age = 30 2024-03-23T03:45:51.956 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.956 INFO:teuthology.orchestra.run.smithi102.stdout: osd map max advance = 10 2024-03-23T03:45:51.956 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.956 INFO:teuthology.orchestra.run.smithi102.stdout: journal zero on create = true 2024-03-23T03:45:51.956 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.956 INFO:teuthology.orchestra.run.smithi102.stdout: filestore ondisk finisher threads = 3 2024-03-23T03:45:51.957 INFO:teuthology.orchestra.run.smithi102.stdout: filestore apply finisher threads = 3 2024-03-23T03:45:51.957 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.957 INFO:teuthology.orchestra.run.smithi102.stdout: bdev debug aio = true 2024-03-23T03:45:51.957 INFO:teuthology.orchestra.run.smithi102.stdout: osd debug misdirected ops = true 2024-03-23T03:45:51.957 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore block size = 96636764160 2024-03-23T03:45:51.957 INFO:teuthology.orchestra.run.smithi102.stdout: bluestore fsck on mount = True 2024-03-23T03:45:51.957 INFO:teuthology.orchestra.run.smithi102.stdout: debug bluefs = 20 2024-03-23T03:45:51.957 INFO:teuthology.orchestra.run.smithi102.stdout: debug bluestore = 20 2024-03-23T03:45:51.957 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-23T03:45:51.957 INFO:teuthology.orchestra.run.smithi102.stdout: debug osd = 20 2024-03-23T03:45:51.957 INFO:teuthology.orchestra.run.smithi102.stdout: debug rocksdb = 10 2024-03-23T03:45:51.957 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T03:45:51.957 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd full ratio = 0.9 2024-03-23T03:45:51.957 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd nearfull ratio = 0.8 2024-03-23T03:45:51.958 INFO:teuthology.orchestra.run.smithi102.stdout: osd failsafe full ratio = 0.95 2024-03-23T03:45:51.958 INFO:teuthology.orchestra.run.smithi102.stdout: osd objectstore = bluestore 2024-03-23T03:45:51.958 INFO:teuthology.orchestra.run.smithi102.stdout: osd op complaint time = 180 2024-03-23T03:45:51.958 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.958 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr] 2024-03-23T03:45:51.958 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-23T03:45:51.958 INFO:teuthology.orchestra.run.smithi102.stdout: debug mgr = 20 2024-03-23T03:45:51.958 INFO:teuthology.orchestra.run.smithi102.stdout: debug mon = 20 2024-03-23T03:45:51.958 INFO:teuthology.orchestra.run.smithi102.stdout: debug auth = 20 2024-03-23T03:45:51.958 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min pgs per osd = 4 2024-03-23T03:45:51.958 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min bytes per osd = 10 2024-03-23T03:45:51.958 INFO:teuthology.orchestra.run.smithi102.stdout: mgr/telemetry/nag = false 2024-03-23T03:45:51.958 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.959 INFO:teuthology.orchestra.run.smithi102.stdout:[mon] 2024-03-23T03:45:51.959 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-23T03:45:51.959 INFO:teuthology.orchestra.run.smithi102.stdout: debug mon = 20 2024-03-23T03:45:51.959 INFO:teuthology.orchestra.run.smithi102.stdout: debug paxos = 20 2024-03-23T03:45:51.959 INFO:teuthology.orchestra.run.smithi102.stdout: debug auth = 20 2024-03-23T03:45:51.959 INFO:teuthology.orchestra.run.smithi102.stdout: mon data avail warn = 5 2024-03-23T03:45:51.960 INFO:teuthology.orchestra.run.smithi102.stdout: mon mgr mkfs grace = 240 2024-03-23T03:45:51.960 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min pgs per osd = 4 2024-03-23T03:45:51.960 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd reporter subtree level = osd 2024-03-23T03:45:51.960 INFO:teuthology.orchestra.run.smithi102.stdout: mon osd prime pg temp = true 2024-03-23T03:45:51.960 INFO:teuthology.orchestra.run.smithi102.stdout: mon reweight min bytes per osd = 10 2024-03-23T03:45:51.960 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.960 INFO:teuthology.orchestra.run.smithi102.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T03:45:51.960 INFO:teuthology.orchestra.run.smithi102.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T03:45:51.961 INFO:teuthology.orchestra.run.smithi102.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T03:45:51.961 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.961 INFO:teuthology.orchestra.run.smithi102.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T03:45:51.961 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T03:45:51.961 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T03:45:51.961 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.961 INFO:teuthology.orchestra.run.smithi102.stdout: # 1m isn't quite enough 2024-03-23T03:45:51.961 INFO:teuthology.orchestra.run.smithi102.stdout: mon_down_mkfs_grace = 2m 2024-03-23T03:45:51.961 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.961 INFO:teuthology.orchestra.run.smithi102.stdout: mon_warn_on_filestore_osds = false 2024-03-23T03:45:51.961 INFO:teuthology.orchestra.run.smithi102.stdout: mon op complaint time = 120 2024-03-23T03:45:51.961 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:45:51.962 INFO:teuthology.orchestra.run.smithi102.stdout:[client] 2024-03-23T03:45:51.962 INFO:teuthology.orchestra.run.smithi102.stdout: rgw cache enabled = true 2024-03-23T03:45:51.962 INFO:teuthology.orchestra.run.smithi102.stdout: rgw enable ops log = true 2024-03-23T03:45:51.962 INFO:teuthology.orchestra.run.smithi102.stdout: rgw enable usage log = true 2024-03-23T03:45:51.962 INFO:teuthology.orchestra.run.smithi102.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T03:45:51.962 INFO:teuthology.orchestra.run.smithi102.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T03:45:51.962 INFO:teuthology.orchestra.run.smithi102.stdout: client mount timeout = 600 2024-03-23T03:45:51.962 INFO:teuthology.orchestra.run.smithi102.stdout: debug client = 20 2024-03-23T03:45:51.962 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-23T03:45:51.962 INFO:teuthology.orchestra.run.smithi102.stdout: pid file = "" 2024-03-23T03:45:51.962 INFO:teuthology.orchestra.run.smithi102.stdout: rados mon op timeout = 900 2024-03-23T03:45:51.962 INFO:teuthology.orchestra.run.smithi102.stdout: rados osd op timeout = 900 2024-03-23T03:45:51.963 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.a] 2024-03-23T03:45:51.963 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.b] 2024-03-23T03:45:51.963 INFO:teuthology.orchestra.run.smithi102.stdout:[mon.c] 2024-03-23T03:45:51.963 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.a] 2024-03-23T03:45:51.963 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.c] 2024-03-23T03:45:51.963 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.b] 2024-03-23T03:45:51.963 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.d] 2024-03-23T03:45:51.963 INFO:teuthology.orchestra.run.smithi102.stdout:[mds] 2024-03-23T03:45:51.963 INFO:teuthology.orchestra.run.smithi102.stdout: debug mds = 20 2024-03-23T03:45:51.963 INFO:teuthology.orchestra.run.smithi102.stdout: debug mds balancer = 20 2024-03-23T03:45:51.964 INFO:teuthology.orchestra.run.smithi102.stdout: debug ms = 1 2024-03-23T03:45:51.964 INFO:teuthology.orchestra.run.smithi102.stdout: mds debug frag = True 2024-03-23T03:45:51.964 INFO:teuthology.orchestra.run.smithi102.stdout: mds debug scatterstat = True 2024-03-23T03:45:51.964 INFO:teuthology.orchestra.run.smithi102.stdout: mds op complaint time = 180 2024-03-23T03:45:51.964 INFO:teuthology.orchestra.run.smithi102.stdout: mds verify scatter = True 2024-03-23T03:45:51.964 INFO:teuthology.orchestra.run.smithi102.stdout: osd op complaint time = 180 2024-03-23T03:45:51.964 INFO:teuthology.orchestra.run.smithi102.stdout: rados mon op timeout = 900 2024-03-23T03:45:51.964 INFO:teuthology.orchestra.run.smithi102.stdout: rados osd op timeout = 900 2024-03-23T03:45:51.964 INFO:teuthology.orchestra.run.smithi173.stdout:[global] 2024-03-23T03:45:51.965 INFO:teuthology.orchestra.run.smithi173.stdout: chdir = "" 2024-03-23T03:45:51.965 INFO:teuthology.orchestra.run.smithi173.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T03:45:51.965 INFO:teuthology.orchestra.run.smithi173.stdout: auth supported = cephx 2024-03-23T03:45:51.965 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.965 INFO:teuthology.orchestra.run.smithi173.stdout: filestore xattr use omap = true 2024-03-23T03:45:51.965 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.965 INFO:teuthology.orchestra.run.smithi173.stdout: mon clock drift allowed = 1.000 2024-03-23T03:45:51.965 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.965 INFO:teuthology.orchestra.run.smithi173.stdout: osd crush chooseleaf type = 0 2024-03-23T03:45:51.965 INFO:teuthology.orchestra.run.smithi173.stdout: auth debug = true 2024-03-23T03:45:51.965 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.965 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on old message = true 2024-03-23T03:45:51.966 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on bug = true 2024-03-23T03:45:51.966 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.966 INFO:teuthology.orchestra.run.smithi173.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T03:45:51.966 INFO:teuthology.orchestra.run.smithi173.stdout: mon pg warn max object skew = 0 2024-03-23T03:45:51.966 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.966 INFO:teuthology.orchestra.run.smithi173.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T03:45:51.966 INFO:teuthology.orchestra.run.smithi173.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T03:45:51.966 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.966 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default size = 2 2024-03-23T03:45:51.966 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.966 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow primary affinity = true 2024-03-23T03:45:51.966 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd allow pg remap = true 2024-03-23T03:45:51.967 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on legacy crush tunables = false 2024-03-23T03:45:51.967 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on crush straw calc version zero = false 2024-03-23T03:45:51.967 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on no sortbitwise = false 2024-03-23T03:45:51.967 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on osd down out interval zero = false 2024-03-23T03:45:51.967 INFO:teuthology.orchestra.run.smithi173.stdout: mon warn on too few osds = false 2024-03-23T03:45:51.967 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T03:45:51.967 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T03:45:51.967 INFO:teuthology.orchestra.run.smithi173.stdout: mon_allow_pool_size_one = true 2024-03-23T03:45:51.967 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.967 INFO:teuthology.orchestra.run.smithi173.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T03:45:51.967 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.968 INFO:teuthology.orchestra.run.smithi173.stdout: osd default data pool replay window = 5 2024-03-23T03:45:51.968 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.968 INFO:teuthology.orchestra.run.smithi173.stdout: mon allow pool delete = true 2024-03-23T03:45:51.968 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.968 INFO:teuthology.orchestra.run.smithi173.stdout: mon cluster log file level = debug 2024-03-23T03:45:51.968 INFO:teuthology.orchestra.run.smithi173.stdout: debug asserts on shutdown = true 2024-03-23T03:45:51.968 INFO:teuthology.orchestra.run.smithi173.stdout: mon health detail to clog = false 2024-03-23T03:45:51.968 INFO:teuthology.orchestra.run.smithi173.stdout: mon host = "172.21.15.102,172.21.15.173,[v2:172.21.15.173:3301,v1:172.21.15.173:6790]" 2024-03-23T03:45:51.968 INFO:teuthology.orchestra.run.smithi173.stdout: ms die on skipped message = False 2024-03-23T03:45:51.968 INFO:teuthology.orchestra.run.smithi173.stdout: fsid = 776b2820-05f2-4e50-8d40-18dd76c5f6a0 2024-03-23T03:45:51.968 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.968 INFO:teuthology.orchestra.run.smithi173.stdout:[osd] 2024-03-23T03:45:51.969 INFO:teuthology.orchestra.run.smithi173.stdout: osd journal size = 100 2024-03-23T03:45:51.969 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.969 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub load threshold = 5.0 2024-03-23T03:45:51.969 INFO:teuthology.orchestra.run.smithi173.stdout: osd scrub max interval = 600 2024-03-23T03:45:51.969 INFO:teuthology.orchestra.run.smithi173.stdout: osd mclock profile = high_recovery_ops 2024-03-23T03:45:51.969 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.969 INFO:teuthology.orchestra.run.smithi173.stdout: osd recover clone overlap = true 2024-03-23T03:45:51.969 INFO:teuthology.orchestra.run.smithi173.stdout: osd recovery max chunk = 1048576 2024-03-23T03:45:51.969 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.969 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug shutdown = true 2024-03-23T03:45:51.969 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug op order = true 2024-03-23T03:45:51.969 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug verify stray on activate = true 2024-03-23T03:45:51.970 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.970 INFO:teuthology.orchestra.run.smithi173.stdout: osd open classes on start = true 2024-03-23T03:45:51.970 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug pg log writeout = true 2024-03-23T03:45:51.970 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.970 INFO:teuthology.orchestra.run.smithi173.stdout: osd deep scrub update digest min age = 30 2024-03-23T03:45:51.970 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.970 INFO:teuthology.orchestra.run.smithi173.stdout: osd map max advance = 10 2024-03-23T03:45:51.970 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.970 INFO:teuthology.orchestra.run.smithi173.stdout: journal zero on create = true 2024-03-23T03:45:51.970 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.970 INFO:teuthology.orchestra.run.smithi173.stdout: filestore ondisk finisher threads = 3 2024-03-23T03:45:51.970 INFO:teuthology.orchestra.run.smithi173.stdout: filestore apply finisher threads = 3 2024-03-23T03:45:51.971 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.971 INFO:teuthology.orchestra.run.smithi173.stdout: bdev debug aio = true 2024-03-23T03:45:51.971 INFO:teuthology.orchestra.run.smithi173.stdout: osd debug misdirected ops = true 2024-03-23T03:45:51.971 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore block size = 96636764160 2024-03-23T03:45:51.971 INFO:teuthology.orchestra.run.smithi173.stdout: bluestore fsck on mount = True 2024-03-23T03:45:51.971 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluefs = 20 2024-03-23T03:45:51.971 INFO:teuthology.orchestra.run.smithi173.stdout: debug bluestore = 20 2024-03-23T03:45:51.971 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-03-23T03:45:51.971 INFO:teuthology.orchestra.run.smithi173.stdout: debug osd = 20 2024-03-23T03:45:51.971 INFO:teuthology.orchestra.run.smithi173.stdout: debug rocksdb = 10 2024-03-23T03:45:51.971 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T03:45:51.971 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd full ratio = 0.9 2024-03-23T03:45:51.972 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd nearfull ratio = 0.8 2024-03-23T03:45:51.972 INFO:teuthology.orchestra.run.smithi173.stdout: osd failsafe full ratio = 0.95 2024-03-23T03:45:51.972 INFO:teuthology.orchestra.run.smithi173.stdout: osd objectstore = bluestore 2024-03-23T03:45:51.972 INFO:teuthology.orchestra.run.smithi173.stdout: osd op complaint time = 180 2024-03-23T03:45:51.972 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.972 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr] 2024-03-23T03:45:51.972 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-03-23T03:45:51.972 INFO:teuthology.orchestra.run.smithi173.stdout: debug mgr = 20 2024-03-23T03:45:51.972 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2024-03-23T03:45:51.972 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2024-03-23T03:45:51.972 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2024-03-23T03:45:51.972 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2024-03-23T03:45:51.972 INFO:teuthology.orchestra.run.smithi173.stdout: mgr/telemetry/nag = false 2024-03-23T03:45:51.973 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.973 INFO:teuthology.orchestra.run.smithi173.stdout:[mon] 2024-03-23T03:45:51.973 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-03-23T03:45:51.973 INFO:teuthology.orchestra.run.smithi173.stdout: debug mon = 20 2024-03-23T03:45:51.973 INFO:teuthology.orchestra.run.smithi173.stdout: debug paxos = 20 2024-03-23T03:45:51.973 INFO:teuthology.orchestra.run.smithi173.stdout: debug auth = 20 2024-03-23T03:45:51.973 INFO:teuthology.orchestra.run.smithi173.stdout: mon data avail warn = 5 2024-03-23T03:45:51.973 INFO:teuthology.orchestra.run.smithi173.stdout: mon mgr mkfs grace = 240 2024-03-23T03:45:51.973 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min pgs per osd = 4 2024-03-23T03:45:51.973 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd reporter subtree level = osd 2024-03-23T03:45:51.973 INFO:teuthology.orchestra.run.smithi173.stdout: mon osd prime pg temp = true 2024-03-23T03:45:51.973 INFO:teuthology.orchestra.run.smithi173.stdout: mon reweight min bytes per osd = 10 2024-03-23T03:45:51.973 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.974 INFO:teuthology.orchestra.run.smithi173.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T03:45:51.974 INFO:teuthology.orchestra.run.smithi173.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T03:45:51.974 INFO:teuthology.orchestra.run.smithi173.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T03:45:51.974 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.974 INFO:teuthology.orchestra.run.smithi173.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T03:45:51.974 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T03:45:51.974 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T03:45:51.974 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.974 INFO:teuthology.orchestra.run.smithi173.stdout: # 1m isn't quite enough 2024-03-23T03:45:51.974 INFO:teuthology.orchestra.run.smithi173.stdout: mon_down_mkfs_grace = 2m 2024-03-23T03:45:51.974 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.974 INFO:teuthology.orchestra.run.smithi173.stdout: mon_warn_on_filestore_osds = false 2024-03-23T03:45:51.975 INFO:teuthology.orchestra.run.smithi173.stdout: mon op complaint time = 120 2024-03-23T03:45:51.975 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-23T03:45:51.975 INFO:teuthology.orchestra.run.smithi173.stdout:[client] 2024-03-23T03:45:51.975 INFO:teuthology.orchestra.run.smithi173.stdout: rgw cache enabled = true 2024-03-23T03:45:51.975 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable ops log = true 2024-03-23T03:45:51.975 INFO:teuthology.orchestra.run.smithi173.stdout: rgw enable usage log = true 2024-03-23T03:45:51.975 INFO:teuthology.orchestra.run.smithi173.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T03:45:51.975 INFO:teuthology.orchestra.run.smithi173.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T03:45:51.975 INFO:teuthology.orchestra.run.smithi173.stdout: client mount timeout = 600 2024-03-23T03:45:51.975 INFO:teuthology.orchestra.run.smithi173.stdout: debug client = 20 2024-03-23T03:45:51.975 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-03-23T03:45:51.975 INFO:teuthology.orchestra.run.smithi173.stdout: pid file = "" 2024-03-23T03:45:51.975 INFO:teuthology.orchestra.run.smithi173.stdout: rados mon op timeout = 900 2024-03-23T03:45:51.975 INFO:teuthology.orchestra.run.smithi173.stdout: rados osd op timeout = 900 2024-03-23T03:45:51.976 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.a] 2024-03-23T03:45:51.976 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.b] 2024-03-23T03:45:51.976 INFO:teuthology.orchestra.run.smithi173.stdout:[mon.c] 2024-03-23T03:45:51.976 INFO:teuthology.orchestra.run.smithi173.stdout:[mds.a] 2024-03-23T03:45:51.976 INFO:teuthology.orchestra.run.smithi173.stdout:[mds.c] 2024-03-23T03:45:51.976 INFO:teuthology.orchestra.run.smithi173.stdout:[mds.b] 2024-03-23T03:45:51.976 INFO:teuthology.orchestra.run.smithi173.stdout:[mds.d] 2024-03-23T03:45:51.976 INFO:teuthology.orchestra.run.smithi173.stdout:[mds] 2024-03-23T03:45:51.976 INFO:teuthology.orchestra.run.smithi173.stdout: debug mds = 20 2024-03-23T03:45:51.976 INFO:teuthology.orchestra.run.smithi173.stdout: debug mds balancer = 20 2024-03-23T03:45:51.976 INFO:teuthology.orchestra.run.smithi173.stdout: debug ms = 1 2024-03-23T03:45:51.976 INFO:teuthology.orchestra.run.smithi173.stdout: mds debug frag = True 2024-03-23T03:45:51.977 INFO:teuthology.orchestra.run.smithi173.stdout: mds debug scatterstat = True 2024-03-23T03:45:51.977 INFO:teuthology.orchestra.run.smithi173.stdout: mds op complaint time = 180 2024-03-23T03:45:51.977 INFO:teuthology.orchestra.run.smithi173.stdout: mds verify scatter = True 2024-03-23T03:45:51.977 INFO:teuthology.orchestra.run.smithi173.stdout: osd op complaint time = 180 2024-03-23T03:45:51.977 INFO:teuthology.orchestra.run.smithi173.stdout: rados mon op timeout = 900 2024-03-23T03:45:51.977 INFO:teuthology.orchestra.run.smithi173.stdout: rados osd op timeout = 900 2024-03-23T03:45:51.995 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-23T03:45:51.995 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-23T03:45:52.139 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-23T03:45:52.139 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:45:52.139 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-23T03:45:52.191 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:45:52.191 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-23T03:45:52.238 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:45:52.238 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:45:52.238 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-23T03:45:52.238 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-23T03:45:52.430 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:45:52.430 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-23T03:45:52.526 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:45:52.526 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:45:52.526 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-23T03:45:52.526 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-23T03:45:52.682 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:45:52.682 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-23T03:45:52.777 INFO:tasks.ceph:Setting up mon nodes... 2024-03-23T03:45:52.778 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-23T03:45:52.778 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-23T03:45:52.962 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-23T03:45:52.972 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-23T03:45:53.169 INFO:teuthology.orchestra.run.smithi173.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-23T03:45:53.180 INFO:tasks.ceph:Setting up mds nodes... 2024-03-23T03:45:53.180 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-23T03:45:53.366 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-23T03:45:53.375 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-23T03:45:53.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-23T03:45:53.644 INFO:teuthology.orchestra.run.smithi102.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-23T03:45:53.653 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-23T03:45:53.747 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-23T03:45:53.933 INFO:teuthology.orchestra.run.smithi173.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-23T03:45:53.942 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-23T03:45:54.034 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-23T03:45:54.205 INFO:teuthology.orchestra.run.smithi173.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-23T03:45:54.214 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-23T03:45:54.304 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-23T03:45:54.305 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-23T03:45:54.434 INFO:teuthology.orchestra.run.smithi102.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-23T03:45:54.482 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-23T03:45:54.605 INFO:teuthology.orchestra.run.smithi102.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-23T03:45:54.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-23T03:45:54.780 INFO:teuthology.orchestra.run.smithi173.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-23T03:45:54.836 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-23T03:45:54.961 INFO:teuthology.orchestra.run.smithi173.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-23T03:45:55.016 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-23T03:45:55.016 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi102.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi173.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-23T03:45:55.016 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-23T03:45:55.117 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T03:45:55.117 INFO:tasks.ceph:role: osd.0 2024-03-23T03:45:55.117 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:45:55.118 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-23T03:45:55.215 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T03:45:55.215 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T03:45:55.215 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T03:45:55.215 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T03:45:55.216 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T03:45:55.216 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-03-23T03:45:55.216 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T03:45:55.216 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T03:45:55.216 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T03:45:55.216 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T03:45:55.736 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-03-23T03:45:55.742 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-03-23T03:45:55.743 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-23T03:45:55.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-23T03:45:56.014 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-23T03:45:56.099 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T03:45:56.099 INFO:tasks.ceph:role: osd.1 2024-03-23T03:45:56.099 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:45:56.099 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-23T03:45:56.195 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T03:45:56.195 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T03:45:56.195 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T03:45:56.195 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T03:45:56.195 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T03:45:56.195 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-03-23T03:45:56.195 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T03:45:56.195 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T03:45:56.195 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T03:45:56.195 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T03:45:56.677 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-03-23T03:45:56.683 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-03-23T03:45:56.684 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-23T03:45:56.830 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-23T03:45:56.920 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-23T03:45:57.010 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T03:45:57.010 INFO:tasks.ceph:role: osd.2 2024-03-23T03:45:57.010 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:45:57.010 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-23T03:45:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T03:45:57.108 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T03:45:57.108 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T03:45:57.108 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T03:45:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T03:45:57.108 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-03-23T03:45:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T03:45:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T03:45:57.108 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T03:45:57.108 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T03:45:57.626 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-03-23T03:45:57.633 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-03-23T03:45:57.633 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-23T03:45:57.826 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-23T03:45:57.928 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-23T03:45:58.022 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T03:45:58.023 INFO:tasks.ceph:role: osd.3 2024-03-23T03:45:58.023 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:45:58.023 DEBUG:teuthology.orchestra.run.smithi102:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-23T03:45:58.121 INFO:teuthology.orchestra.run.smithi102.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T03:45:58.122 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T03:45:58.122 INFO:teuthology.orchestra.run.smithi102.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T03:45:58.122 INFO:teuthology.orchestra.run.smithi102.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T03:45:58.122 INFO:teuthology.orchestra.run.smithi102.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T03:45:58.122 INFO:teuthology.orchestra.run.smithi102.stdout: = sunit=0 swidth=0 blks 2024-03-23T03:45:58.122 INFO:teuthology.orchestra.run.smithi102.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T03:45:58.122 INFO:teuthology.orchestra.run.smithi102.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T03:45:58.122 INFO:teuthology.orchestra.run.smithi102.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T03:45:58.122 INFO:teuthology.orchestra.run.smithi102.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T03:45:58.586 INFO:teuthology.orchestra.run.smithi102.stdout:Discarding blocks...Done. 2024-03-23T03:45:58.594 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi102.front.sepia.ceph.com -o noatime 2024-03-23T03:45:58.594 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-23T03:45:58.721 DEBUG:teuthology.orchestra.run.smithi102:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-23T03:45:58.815 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T03:45:58.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:45:58.966+0000 7f31bd2494c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-23T03:45:58.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:45:58.972+0000 7f31bd2494c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-23T03:45:58.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:45:58.972+0000 7f31bd2494c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-23T03:45:58.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:45:58.972+0000 7f31bd2494c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-23T03:45:58.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:45:58.972+0000 7f31bd2494c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-23T03:45:58.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:45:58.975+0000 7f31bd2494c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-23T03:46:01.902 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-23T03:46:01.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T03:46:02.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:02.130+0000 7f1f8b5944c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-23T03:46:02.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:02.130+0000 7f1f8b5944c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-23T03:46:02.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:02.130+0000 7f1f8b5944c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-23T03:46:02.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:02.130+0000 7f1f8b5944c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-23T03:46:02.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:02.130+0000 7f1f8b5944c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-23T03:46:02.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:02.133+0000 7f1f8b5944c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-23T03:46:05.308 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-23T03:46:05.408 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T03:46:05.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:05.546+0000 7f5603cda4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-23T03:46:05.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:05.546+0000 7f5603cda4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-23T03:46:05.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:05.546+0000 7f5603cda4c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-23T03:46:05.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:05.546+0000 7f5603cda4c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-23T03:46:05.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:05.546+0000 7f5603cda4c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-23T03:46:05.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:05.549+0000 7f5603cda4c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-23T03:46:08.483 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-23T03:46:08.580 DEBUG:teuthology.orchestra.run.smithi102:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T03:46:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:08.714+0000 7f08f9a1f4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-23T03:46:08.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:08.715+0000 7f08f9a1f4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-23T03:46:08.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:08.715+0000 7f08f9a1f4c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-23T03:46:08.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:08.715+0000 7f08f9a1f4c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-23T03:46:08.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:08.715+0000 7f08f9a1f4c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-23T03:46:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:08.718+0000 7f08f9a1f4c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-23T03:46:11.660 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-23T03:46:11.758 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-23T03:46:11.854 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T03:46:11.855 INFO:tasks.ceph:role: osd.4 2024-03-23T03:46:11.855 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:46:11.855 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-23T03:46:11.954 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T03:46:11.955 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T03:46:11.955 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T03:46:11.955 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T03:46:11.955 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T03:46:11.955 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-03-23T03:46:11.955 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T03:46:11.955 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T03:46:11.955 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T03:46:11.955 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T03:46:12.466 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-03-23T03:46:12.473 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-03-23T03:46:12.473 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-23T03:46:12.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-23T03:46:12.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-23T03:46:12.858 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T03:46:12.859 INFO:tasks.ceph:role: osd.5 2024-03-23T03:46:12.859 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:46:12.859 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-23T03:46:12.948 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T03:46:12.948 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T03:46:12.948 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T03:46:12.948 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T03:46:12.948 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T03:46:12.948 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-03-23T03:46:12.948 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T03:46:12.948 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T03:46:12.948 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T03:46:12.948 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T03:46:13.421 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-03-23T03:46:13.426 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-03-23T03:46:13.427 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-23T03:46:13.562 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-23T03:46:13.650 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-23T03:46:13.736 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T03:46:13.736 INFO:tasks.ceph:role: osd.6 2024-03-23T03:46:13.736 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:46:13.736 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-23T03:46:13.820 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T03:46:13.820 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T03:46:13.820 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T03:46:13.820 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T03:46:13.820 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T03:46:13.820 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-03-23T03:46:13.820 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T03:46:13.820 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T03:46:13.820 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T03:46:13.821 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T03:46:14.332 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-03-23T03:46:14.336 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-03-23T03:46:14.336 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-23T03:46:14.514 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-23T03:46:14.612 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-23T03:46:14.704 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T03:46:14.704 INFO:tasks.ceph:role: osd.7 2024-03-23T03:46:14.704 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:46:14.704 DEBUG:teuthology.orchestra.run.smithi173:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-23T03:46:14.801 INFO:teuthology.orchestra.run.smithi173.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T03:46:14.801 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T03:46:14.801 INFO:teuthology.orchestra.run.smithi173.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T03:46:14.801 INFO:teuthology.orchestra.run.smithi173.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T03:46:14.801 INFO:teuthology.orchestra.run.smithi173.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T03:46:14.801 INFO:teuthology.orchestra.run.smithi173.stdout: = sunit=0 swidth=0 blks 2024-03-23T03:46:14.801 INFO:teuthology.orchestra.run.smithi173.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T03:46:14.801 INFO:teuthology.orchestra.run.smithi173.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T03:46:14.802 INFO:teuthology.orchestra.run.smithi173.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T03:46:14.802 INFO:teuthology.orchestra.run.smithi173.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T03:46:15.258 INFO:teuthology.orchestra.run.smithi173.stdout:Discarding blocks...Done. 2024-03-23T03:46:15.264 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi173.front.sepia.ceph.com -o noatime 2024-03-23T03:46:15.264 DEBUG:teuthology.orchestra.run.smithi173:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-23T03:46:15.389 DEBUG:teuthology.orchestra.run.smithi173:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-23T03:46:15.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T03:46:15.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:15.627+0000 7f3f5ce184c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-23T03:46:15.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:15.628+0000 7f3f5ce184c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-23T03:46:15.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:15.628+0000 7f3f5ce184c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-23T03:46:15.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:15.628+0000 7f3f5ce184c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-23T03:46:15.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:15.628+0000 7f3f5ce184c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-23T03:46:15.638 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:15.638+0000 7f3f5ce184c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-23T03:46:18.825 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-23T03:46:18.923 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T03:46:19.057 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:19.057+0000 7f121b5e24c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-23T03:46:19.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:19.057+0000 7f121b5e24c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-23T03:46:19.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:19.058+0000 7f121b5e24c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-23T03:46:19.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:19.058+0000 7f121b5e24c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-23T03:46:19.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:19.058+0000 7f121b5e24c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-23T03:46:19.060 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:19.060+0000 7f121b5e24c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-23T03:46:19.084 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:46:19.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:46:22.238 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-23T03:46:22.330 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T03:46:22.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:22.465+0000 7f7a5da144c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-23T03:46:22.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:22.465+0000 7f7a5da144c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-23T03:46:22.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:22.465+0000 7f7a5da144c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-23T03:46:22.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:22.465+0000 7f7a5da144c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-23T03:46:22.467 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:22.465+0000 7f7a5da144c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-23T03:46:22.470 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:22.468+0000 7f7a5da144c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-23T03:46:25.418 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-23T03:46:25.516 DEBUG:teuthology.orchestra.run.smithi173:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T03:46:25.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:25.656+0000 7fb0f228c4c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-23T03:46:25.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:25.656+0000 7fb0f228c4c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-23T03:46:25.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:25.656+0000 7fb0f228c4c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-23T03:46:25.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:25.656+0000 7fb0f228c4c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-23T03:46:25.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:25.656+0000 7fb0f228c4c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-23T03:46:25.659 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:25.658+0000 7fb0f228c4c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-23T03:46:28.828 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-23T03:46:28.925 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-23T03:46:28.925 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:46:28.925 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-23T03:46:29.024 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:46:29.024 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-23T03:46:29.118 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:46:29.118 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-23T03:46:29.214 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:46:29.214 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-23T03:46:29.309 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:46:29.309 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-23T03:46:29.406 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:46:29.406 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-23T03:46:29.499 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:46:29.499 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-23T03:46:29.594 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:46:29.594 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-23T03:46:29.682 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:46:29.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-23T03:46:29.775 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:46:29.775 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-23T03:46:29.870 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:46:29.870 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-23T03:46:29.963 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:46:29.963 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-23T03:46:30.056 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:46:30.056 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-23T03:46:30.150 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:46:30.151 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-23T03:46:30.245 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:46:30.245 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-23T03:46:30.299 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:46:30.299 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-23T03:46:30.349 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:46:30.349 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-23T03:46:30.405 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:46:30.405 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-23T03:46:30.460 INFO:tasks.ceph:Adding keys to all mons... 2024-03-23T03:46:30.461 DEBUG:teuthology.orchestra.run.smithi102:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-23T03:46:30.467 DEBUG:teuthology.orchestra.run.smithi173:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-23T03:46:30.552 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr.y] 2024-03-23T03:46:30.552 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBwUP5lK59iORAACuBF0JaQ8epyu3Mq3slzjg== 2024-03-23T03:46:30.552 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.a] 2024-03-23T03:46:30.553 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBxUP5loBTZFRAA7/tTYyZwrDmwKwiU+dtCEw== 2024-03-23T03:46:30.553 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.c] 2024-03-23T03:46:30.553 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBxUP5l0b92JhAA5rvs+mdgmrQX/9lI3GW2rg== 2024-03-23T03:46:30.553 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.0] 2024-03-23T03:46:30.553 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQB2UP5lPW6uORAAejiQReOF95oFliFmk/vdQQ== 2024-03-23T03:46:30.553 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.1] 2024-03-23T03:46:30.553 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQB6UP5lFbbUBxAAIl0ZEYtJ+/oi3W6a4OwMhg== 2024-03-23T03:46:30.553 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.2] 2024-03-23T03:46:30.553 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQB9UP5l2LaQIBAAKJv7jWJAjlUJrm7+KTXi6Q== 2024-03-23T03:46:30.553 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.3] 2024-03-23T03:46:30.553 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCAUP5lxjWsKhAAzVO8CXUd+4LLvv4f/5xMOg== 2024-03-23T03:46:30.553 INFO:teuthology.orchestra.run.smithi102.stdout:[mgr.x] 2024-03-23T03:46:30.553 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBxUP5lQXojChAAzepTDh0Pk3fuBZYsFt/+DQ== 2024-03-23T03:46:30.554 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.b] 2024-03-23T03:46:30.554 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQBxUP5lJKKlNxAAj5+j99K79IhbeEm7/qC+wg== 2024-03-23T03:46:30.554 INFO:teuthology.orchestra.run.smithi102.stdout:[mds.d] 2024-03-23T03:46:30.554 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQByUP5ltjNMDBAAD1h5Nlv2Cp7mJt+DMlBtzw== 2024-03-23T03:46:30.554 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.4] 2024-03-23T03:46:30.554 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCHUP5lHcd2JRAAwID1TPb+Z9ViH51B0Mv/kQ== 2024-03-23T03:46:30.554 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.5] 2024-03-23T03:46:30.554 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCLUP5lApp3AxAADslbuzTe/bf5FML1ckE05w== 2024-03-23T03:46:30.554 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.6] 2024-03-23T03:46:30.554 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCOUP5lsi7VGxAArhWvXQ1VzJ8oSiOXzq8VsQ== 2024-03-23T03:46:30.554 INFO:teuthology.orchestra.run.smithi102.stdout:[osd.7] 2024-03-23T03:46:30.554 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQCRUP5lu182JxAA5LaDXfrBkOXEUNAJXFDCJQ== 2024-03-23T03:46:30.554 INFO:teuthology.orchestra.run.smithi102.stdout:[client.0] 2024-03-23T03:46:30.555 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQByUP5lxTfoGRAA3WIQ8p1rGSAyJkLZJMk3dA== 2024-03-23T03:46:30.555 INFO:teuthology.orchestra.run.smithi102.stdout:[client.2] 2024-03-23T03:46:30.555 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQByUP5lOIklJBAAhx2d0Xy/j1PG0qsvC3nMqA== 2024-03-23T03:46:30.555 INFO:teuthology.orchestra.run.smithi102.stdout:[client.1] 2024-03-23T03:46:30.555 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQByUP5lneSHLhAAwn6ULOajHXrOM650HUcLbw== 2024-03-23T03:46:30.555 INFO:teuthology.orchestra.run.smithi102.stdout:[client.3] 2024-03-23T03:46:30.555 INFO:teuthology.orchestra.run.smithi102.stdout: key = AQByUP5l+7BVORAA7PBYS6mkgLA8b4sLCAYgBA== 2024-03-23T03:46:30.559 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr.y] 2024-03-23T03:46:30.559 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQBwUP5lK59iORAACuBF0JaQ8epyu3Mq3slzjg== 2024-03-23T03:46:30.559 INFO:teuthology.orchestra.run.smithi173.stdout:[mds.a] 2024-03-23T03:46:30.559 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQBxUP5loBTZFRAA7/tTYyZwrDmwKwiU+dtCEw== 2024-03-23T03:46:30.560 INFO:teuthology.orchestra.run.smithi173.stdout:[mds.c] 2024-03-23T03:46:30.560 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQBxUP5l0b92JhAA5rvs+mdgmrQX/9lI3GW2rg== 2024-03-23T03:46:30.560 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.0] 2024-03-23T03:46:30.560 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQB2UP5lPW6uORAAejiQReOF95oFliFmk/vdQQ== 2024-03-23T03:46:30.560 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.1] 2024-03-23T03:46:30.560 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQB6UP5lFbbUBxAAIl0ZEYtJ+/oi3W6a4OwMhg== 2024-03-23T03:46:30.560 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.2] 2024-03-23T03:46:30.560 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQB9UP5l2LaQIBAAKJv7jWJAjlUJrm7+KTXi6Q== 2024-03-23T03:46:30.560 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.3] 2024-03-23T03:46:30.560 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCAUP5lxjWsKhAAzVO8CXUd+4LLvv4f/5xMOg== 2024-03-23T03:46:30.560 INFO:teuthology.orchestra.run.smithi173.stdout:[mgr.x] 2024-03-23T03:46:30.560 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQBxUP5lQXojChAAzepTDh0Pk3fuBZYsFt/+DQ== 2024-03-23T03:46:30.560 INFO:teuthology.orchestra.run.smithi173.stdout:[mds.b] 2024-03-23T03:46:30.560 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQBxUP5lJKKlNxAAj5+j99K79IhbeEm7/qC+wg== 2024-03-23T03:46:30.561 INFO:teuthology.orchestra.run.smithi173.stdout:[mds.d] 2024-03-23T03:46:30.561 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQByUP5ltjNMDBAAD1h5Nlv2Cp7mJt+DMlBtzw== 2024-03-23T03:46:30.561 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.4] 2024-03-23T03:46:30.561 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCHUP5lHcd2JRAAwID1TPb+Z9ViH51B0Mv/kQ== 2024-03-23T03:46:30.561 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.5] 2024-03-23T03:46:30.561 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCLUP5lApp3AxAADslbuzTe/bf5FML1ckE05w== 2024-03-23T03:46:30.561 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.6] 2024-03-23T03:46:30.561 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCOUP5lsi7VGxAArhWvXQ1VzJ8oSiOXzq8VsQ== 2024-03-23T03:46:30.561 INFO:teuthology.orchestra.run.smithi173.stdout:[osd.7] 2024-03-23T03:46:30.561 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCRUP5lu182JxAA5LaDXfrBkOXEUNAJXFDCJQ== 2024-03-23T03:46:30.561 INFO:teuthology.orchestra.run.smithi173.stdout:[client.0] 2024-03-23T03:46:30.561 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQByUP5lxTfoGRAA3WIQ8p1rGSAyJkLZJMk3dA== 2024-03-23T03:46:30.561 INFO:teuthology.orchestra.run.smithi173.stdout:[client.2] 2024-03-23T03:46:30.561 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQByUP5lOIklJBAAhx2d0Xy/j1PG0qsvC3nMqA== 2024-03-23T03:46:30.562 INFO:teuthology.orchestra.run.smithi173.stdout:[client.1] 2024-03-23T03:46:30.562 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQByUP5lneSHLhAAwn6ULOajHXrOM650HUcLbw== 2024-03-23T03:46:30.562 INFO:teuthology.orchestra.run.smithi173.stdout:[client.3] 2024-03-23T03:46:30.562 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQByUP5l+7BVORAA7PBYS6mkgLA8b4sLCAYgBA== 2024-03-23T03:46:30.564 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T03:46:30.604 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T03:46:30.749 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T03:46:30.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T03:46:30.895 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T03:46:30.902 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T03:46:31.044 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T03:46:31.048 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T03:46:31.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T03:46:31.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T03:46:31.331 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T03:46:31.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T03:46:31.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T03:46:31.626 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T03:46:31.765 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T03:46:31.771 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T03:46:31.912 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T03:46:31.917 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T03:46:32.056 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T03:46:32.063 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T03:46:32.205 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T03:46:32.211 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T03:46:32.354 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T03:46:32.361 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T03:46:32.499 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T03:46:32.504 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T03:46:32.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T03:46:32.649 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T03:46:32.787 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T03:46:32.795 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T03:46:32.938 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T03:46:32.943 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T03:46:33.086 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T03:46:33.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T03:46:33.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T03:46:33.239 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T03:46:33.382 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-23T03:46:33.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-23T03:46:33.478 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-23T03:46:33.630 DEBUG:teuthology.orchestra.run.smithi102:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-23T03:46:33.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-23T03:46:33.805 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-23T03:46:33.968 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-23T03:46:34.044 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-23T03:46:34.137 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-23T03:46:34.290 DEBUG:teuthology.orchestra.run.smithi173:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-23T03:46:34.368 DEBUG:teuthology.orchestra.run.smithi102:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-23T03:46:34.376 DEBUG:teuthology.orchestra.run.smithi173:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-23T03:46:34.430 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-23T03:46:34.430 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-23T03:46:34.431 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-23T03:46:34.435 INFO:tasks.ceph.mon.a:Started 2024-03-23T03:46:34.435 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-23T03:46:34.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-23T03:46:34.440 INFO:tasks.ceph.mon.b:Started 2024-03-23T03:46:34.440 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-23T03:46:34.440 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-23T03:46:34.445 INFO:tasks.ceph.mon.c:Started 2024-03-23T03:46:34.445 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-23T03:46:34.445 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-23T03:46:34.445 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-23T03:46:34.450 INFO:tasks.ceph.mgr.y:Started 2024-03-23T03:46:34.450 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-23T03:46:34.450 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-23T03:46:34.454 INFO:tasks.ceph.mgr.x:Started 2024-03-23T03:46:34.454 INFO:tasks.ceph:Setting crush tunables to default 2024-03-23T03:46:34.455 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph osd crush tunables default 2024-03-23T03:46:34.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.627+0000 7f159a200640 1 Processor -- start 2024-03-23T03:46:34.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.627+0000 7f159a200640 1 -- start start 2024-03-23T03:46:34.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.628+0000 7f159a200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940b84b0 0x7f15940d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:34.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.628+0000 7f159a200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15940d1280 con 0x7f15940b5aa0 2024-03-23T03:46:34.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.628+0000 7f159a200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15940d20c0 con 0x7f15940cd740 2024-03-23T03:46:34.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.628+0000 7f159a200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15940b8910 con 0x7f15940b84b0 2024-03-23T03:46:34.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.628+0000 7f1593e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f15940b5aa0 0x7f15940b5f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59474/0 (socket says 172.21.15.102:59474) 2024-03-23T03:46:34.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.628+0000 7f1593e00640 1 -- 172.21.15.102:0/613528110 learned_addr learned my addr 172.21.15.102:0/613528110 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:46:34.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.629+0000 7f1592a00640 1 --2- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940b84b0 0x7f15940d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:34.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.629+0000 7f1592a00640 1 -- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940b84b0 msgr2=0x7f15940d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T03:46:34.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.629+0000 7f1592a00640 1 -- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940b84b0 msgr2=0x7f15940d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T03:46:34.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.629+0000 7f1592a00640 1 --2- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940b84b0 0x7f15940d0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T03:46:34.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.629+0000 7f1592a00640 1 --2- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940b84b0 0x7f15940d0d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T03:46:34.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.830+0000 7f1592a00640 1 --2- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940b84b0 0x7f15940d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:34.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.831+0000 7f1592a00640 1 -- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940b84b0 msgr2=0x7f15940d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T03:46:34.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.831+0000 7f1592a00640 1 -- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940b84b0 msgr2=0x7f15940d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T03:46:34.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.831+0000 7f1592a00640 1 --2- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940b84b0 0x7f15940d0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T03:46:34.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.831+0000 7f1592a00640 1 --2- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940b84b0 0x7f15940d0d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T03:46:34.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.880+0000 7f1592000640 1 -- 172.21.15.102:0/613528110 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406982676 0 0) 0x7f15940d20c0 con 0x7f15940cd740 2024-03-23T03:46:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.880+0000 7f1592000640 1 -- 172.21.15.102:0/613528110 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1578003520 con 0x7f15940cd740 2024-03-23T03:46:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.881+0000 7f1592000640 1 -- 172.21.15.102:0/613528110 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053181804 0 0) 0x7f15940d1280 con 0x7f15940b5aa0 2024-03-23T03:46:34.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.881+0000 7f1592000640 1 -- 172.21.15.102:0/613528110 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15940d20c0 con 0x7f15940b5aa0 2024-03-23T03:46:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.882+0000 7f1592000640 1 -- 172.21.15.102:0/613528110 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 177680724 0 0) 0x7f15940d20c0 con 0x7f15940b5aa0 2024-03-23T03:46:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.882+0000 7f1592000640 1 -- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940b84b0 msgr2=0x7f15940d0d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T03:46:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.882+0000 7f1592000640 1 --2- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940b84b0 0x7f15940d0d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.882+0000 7f1592000640 1 -- 172.21.15.102:0/613528110 >> v1:172.21.15.173:6789/0 conn(0x7f15940cd740 legacy=0x7f15940cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.882+0000 7f1592000640 1 -- 172.21.15.102:0/613528110 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15940d14b0 con 0x7f15940b5aa0 2024-03-23T03:46:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.882+0000 7f1592000640 1 -- 172.21.15.102:0/613528110 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f158c0019f0 con 0x7f15940b5aa0 2024-03-23T03:46:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.882+0000 7f1592000640 1 -- 172.21.15.102:0/613528110 >> v1:172.21.15.102:6789/0 conn(0x7f15940b5aa0 legacy=0x7f15940b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:34.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.882+0000 7f1592000640 1 --2- 172.21.15.102:0/613528110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1578003b30 0x7f1578003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:34.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.882+0000 7f1592000640 1 --2- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940b84b0 0x7f15780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:34.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.883+0000 7f1592000640 1 --2- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1578004d00 0x7f1578005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:34.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.883+0000 7f1592000640 1 -- 172.21.15.102:0/613528110 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15940b8910 con 0x7f1578003b30 2024-03-23T03:46:34.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.883+0000 7f1592000640 1 -- 172.21.15.102:0/613528110 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157800bb10 con 0x7f1578004d00 2024-03-23T03:46:34.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.883+0000 7f1592000640 1 -- 172.21.15.102:0/613528110 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f157800bcd0 con 0x7f15940b84b0 2024-03-23T03:46:34.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.884+0000 7f1593400640 1 --2- 172.21.15.102:0/613528110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1578003b30 0x7f1578003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.884+0000 7f1592a00640 1 --2- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940b84b0 0x7f15780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.884+0000 7f1593e00640 1 --2- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1578004d00 0x7f1578005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.885+0000 7f1593400640 1 -- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940b84b0 msgr2=0x7f15780047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.885+0000 7f1593400640 1 --2- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940b84b0 0x7f15780047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.885+0000 7f1593400640 1 -- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1578004d00 msgr2=0x7f1578005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.885+0000 7f1593400640 1 --2- 172.21.15.102:0/613528110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1578004d00 0x7f1578005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:34.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.885+0000 7f1593400640 1 -- 172.21.15.102:0/613528110 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15940d20c0 con 0x7f1578003b30 2024-03-23T03:46:34.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.887+0000 7f1593400640 1 --2- 172.21.15.102:0/613528110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1578003b30 0x7f1578003f30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1584001900 tx=0x7f15780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=294a374569bd383e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:34.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.887+0000 7f1592000640 1 -- 172.21.15.102:0/613528110 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15840052d0 con 0x7f1578003b30 2024-03-23T03:46:34.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.887+0000 7f1592000640 1 -- 172.21.15.102:0/613528110 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1584005450 con 0x7f1578003b30 2024-03-23T03:46:34.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.887+0000 7f1592000640 1 -- 172.21.15.102:0/613528110 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1584003830 con 0x7f1578003b30 2024-03-23T03:46:34.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.889+0000 7f159a200640 1 -- 172.21.15.102:0/613528110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1578003b30 msgr2=0x7f1578003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:34.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.889+0000 7f159a200640 1 --2- 172.21.15.102:0/613528110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1578003b30 0x7f1578003f30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1584001900 tx=0x7f15780057f0 comp rx=0 tx=0).stop 2024-03-23T03:46:34.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.889+0000 7f1593e00640 1 -- 172.21.15.102:0/613528110 reap_dead start 2024-03-23T03:46:34.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.889+0000 7f159a200640 1 -- 172.21.15.102:0/613528110 shutdown_connections 2024-03-23T03:46:34.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.889+0000 7f159a200640 1 -- 172.21.15.102:0/613528110 >> 172.21.15.102:0/613528110 conn(0x7f15940cf5d0 msgr2=0x7f15940cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:34.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.889+0000 7f159a200640 1 -- 172.21.15.102:0/613528110 shutdown_connections 2024-03-23T03:46:34.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.889+0000 7f159a200640 1 -- 172.21.15.102:0/613528110 wait complete. 2024-03-23T03:46:34.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.893+0000 7f159a200640 1 Processor -- start 2024-03-23T03:46:34.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.893+0000 7f159a200640 1 -- start start 2024-03-23T03:46:34.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.894+0000 7f159a200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1578003b30 0x7f1594152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:34.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.894+0000 7f159a200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15940caca0 0x7f15940ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:34.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.894+0000 7f159a200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1594153f20 0x7f15941543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:34.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.894+0000 7f159a200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1594153430 con 0x7f1578003b30 2024-03-23T03:46:34.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.894+0000 7f159a200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15941548e0 con 0x7f15940caca0 2024-03-23T03:46:34.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.894+0000 7f159a200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15940cb210 con 0x7f1594153f20 2024-03-23T03:46:34.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.895+0000 7f1593400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1578003b30 0x7f1594152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:34.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.895+0000 7f1593400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1578003b30 0x7f1594152ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33138/0 (socket says 172.21.15.102:33138) 2024-03-23T03:46:34.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.895+0000 7f1593400640 1 -- 172.21.15.102:0/3449390473 learned_addr learned my addr 172.21.15.102:0/3449390473 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:34.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.895+0000 7f1592a00640 1 --2- 172.21.15.102:0/3449390473 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15940caca0 0x7f15940ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:34.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.895+0000 7f1593e00640 1 --2- 172.21.15.102:0/3449390473 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1594153f20 0x7f15941543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:34.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.896+0000 7f1593400640 1 -- 172.21.15.102:0/3449390473 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1594153f20 msgr2=0x7f15941543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:34.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.896+0000 7f1593400640 1 --2- 172.21.15.102:0/3449390473 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1594153f20 0x7f15941543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:34.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.896+0000 7f1593400640 1 -- 172.21.15.102:0/3449390473 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15940caca0 msgr2=0x7f15940ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:34.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.896+0000 7f1593400640 1 --2- 172.21.15.102:0/3449390473 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15940caca0 0x7f15940ca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:34.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.896+0000 7f1593400640 1 -- 172.21.15.102:0/3449390473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15940cb4f0 con 0x7f1578003b30 2024-03-23T03:46:34.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.896+0000 7f1593400640 1 --2- 172.21.15.102:0/3449390473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1578003b30 0x7f1594152ef0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1584035040 tx=0x7f15840012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:34.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.897+0000 7f158b400640 1 -- 172.21.15.102:0/3449390473 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1584006be0 con 0x7f1578003b30 2024-03-23T03:46:34.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.897+0000 7f158b400640 1 -- 172.21.15.102:0/3449390473 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1584006d40 con 0x7f1578003b30 2024-03-23T03:46:34.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.897+0000 7f158b400640 1 -- 172.21.15.102:0/3449390473 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15840037f0 con 0x7f1578003b30 2024-03-23T03:46:34.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.897+0000 7f159a200640 1 -- 172.21.15.102:0/3449390473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15940cb7d0 con 0x7f1578003b30 2024-03-23T03:46:34.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.898+0000 7f159a200640 1 -- 172.21.15.102:0/3449390473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15940cbd90 con 0x7f1578003b30 2024-03-23T03:46:34.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.898+0000 7f158b400640 1 -- 172.21.15.102:0/3449390473 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1584003c50 con 0x7f1578003b30 2024-03-23T03:46:34.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.898+0000 7f158b400640 1 -- 172.21.15.102:0/3449390473 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1584032af0 con 0x7f1578003b30 2024-03-23T03:46:34.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.899+0000 7f159a200640 1 -- 172.21.15.102:0/3449390473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1594073390 con 0x7f1578003b30 2024-03-23T03:46:34.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.903+0000 7f158b400640 1 -- 172.21.15.102:0/3449390473 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f158404b060 con 0x7f1578003b30 2024-03-23T03:46:34.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.977+0000 7f159a200640 1 -- 172.21.15.102:0/3449390473 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f15940b20b0 con 0x7f1578003b30 2024-03-23T03:46:34.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.981+0000 7f158b400640 1 -- 172.21.15.102:0/3449390473 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1584041070 con 0x7f1578003b30 2024-03-23T03:46:34.982 INFO:teuthology.orchestra.run.smithi102.stderr:adjusted tunables profile to default 2024-03-23T03:46:34.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.983+0000 7f159a200640 1 -- 172.21.15.102:0/3449390473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1578003b30 msgr2=0x7f1594152ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:34.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.983+0000 7f159a200640 1 --2- 172.21.15.102:0/3449390473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1578003b30 0x7f1594152ef0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1584035040 tx=0x7f15840012d0 comp rx=0 tx=0).stop 2024-03-23T03:46:34.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.983+0000 7f159a200640 1 -- 172.21.15.102:0/3449390473 shutdown_connections 2024-03-23T03:46:34.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.983+0000 7f159a200640 1 --2- 172.21.15.102:0/3449390473 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1594153f20 0x7f15941543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:34.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.983+0000 7f159a200640 1 --2- 172.21.15.102:0/3449390473 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f15940caca0 0x7f15940ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:34.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.983+0000 7f159a200640 1 --2- 172.21.15.102:0/3449390473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1578003b30 0x7f1594152ef0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:34.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.983+0000 7f159a200640 1 -- 172.21.15.102:0/3449390473 >> 172.21.15.102:0/3449390473 conn(0x7f15940cf5d0 msgr2=0x7f15940b14a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:34.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.983+0000 7f159a200640 1 -- 172.21.15.102:0/3449390473 shutdown_connections 2024-03-23T03:46:34.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:34.983+0000 7f159a200640 1 -- 172.21.15.102:0/3449390473 wait complete. 2024-03-23T03:46:35.001 INFO:tasks.ceph:check_enable_crimson: False 2024-03-23T03:46:35.001 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-23T03:46:35.001 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:46:35.001 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-23T03:46:35.071 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:46:35.071 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-23T03:46:35.185 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:46:35.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-23T03:46:35.311 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:46:35.311 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-23T03:46:35.438 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:46:35.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-23T03:46:35.454 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:35.453+0000 7faa1321e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-23T03:46:35.516 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:35.516+0000 7faa1321e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-23T03:46:35.525 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:46:35.525 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-23T03:46:35.600 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:46:35.600 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-23T03:46:35.728 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:46:35.728 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-23T03:46:35.787 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:35.787+0000 7f7d4922c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-23T03:46:35.853 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:35.853+0000 7f7d4922c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-23T03:46:35.855 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 2c69740f-05ae-4bab-abe9-08e4e449e501 0 2024-03-23T03:46:36.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.086+0000 7f21a6800640 1 Processor -- start 2024-03-23T03:46:36.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.087+0000 7f21a6800640 1 -- start start 2024-03-23T03:46:36.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.088+0000 7f21a6800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f21a00cfc20 0x7f21a00b5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:36.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.088+0000 7f21a6800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a0160ae0 con 0x7f21a00b5b90 2024-03-23T03:46:36.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.088+0000 7f21a6800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a00b6510 con 0x7f21a015cc30 2024-03-23T03:46:36.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.088+0000 7f21a6800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a00d0c00 con 0x7f21a00cfc20 2024-03-23T03:46:36.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.088+0000 7f21a5400640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f21a015cc30 0x7f21a00d16a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56248/0 (socket says 172.21.15.173:56248) 2024-03-23T03:46:36.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.088+0000 7f21a5400640 1 -- 172.21.15.173:0/3754868793 learned_addr learned my addr 172.21.15.173:0/3754868793 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-23T03:46:36.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f21a5e00640 1 --2- 172.21.15.173:0/3754868793 >> v2:172.21.15.102:3300/0 conn(0x7f21a00cfc20 0x7f21a00b5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:36.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f219fe00640 1 -- 172.21.15.173:0/3754868793 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729501780 0 0) 0x7f2194001110 con 0x7f21a015cc30 2024-03-23T03:46:36.089 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f219fe00640 1 -- 172.21.15.173:0/3754868793 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2188003520 con 0x7f21a015cc30 2024-03-23T03:46:36.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f219fe00640 1 -- 172.21.15.173:0/3754868793 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3214960260 0 0) 0x7f2188003520 con 0x7f21a015cc30 2024-03-23T03:46:36.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f219fe00640 1 -- 172.21.15.173:0/3754868793 >> v2:172.21.15.102:3300/0 conn(0x7f21a00cfc20 msgr2=0x7f21a00b5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:36.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f219fe00640 1 --2- 172.21.15.173:0/3754868793 >> v2:172.21.15.102:3300/0 conn(0x7f21a00cfc20 0x7f21a00b5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f219fe00640 1 -- 172.21.15.173:0/3754868793 >> v1:172.21.15.102:6789/0 conn(0x7f21a00b5b90 legacy=0x7f21a00cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:36.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f219fe00640 1 -- 172.21.15.173:0/3754868793 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a00cdde0 con 0x7f21a015cc30 2024-03-23T03:46:36.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f219fe00640 1 -- 172.21.15.173:0/3754868793 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f2194001800 con 0x7f21a015cc30 2024-03-23T03:46:36.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f219fe00640 1 -- 172.21.15.173:0/3754868793 >> v1:172.21.15.173:6789/0 conn(0x7f21a015cc30 legacy=0x7f21a00d16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:36.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f219fe00640 1 --2- 172.21.15.173:0/3754868793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21880039c0 0x7f2188003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:36.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f219fe00640 1 --2- 172.21.15.173:0/3754868793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21880047c0 0x7f2188004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:36.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f219fe00640 1 --2- 172.21.15.173:0/3754868793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2188005180 0x7f218800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:36.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f219fe00640 1 -- 172.21.15.173:0/3754868793 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a00d0c00 con 0x7f2188005180 2024-03-23T03:46:36.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f219fe00640 1 -- 172.21.15.173:0/3754868793 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218800c080 con 0x7f21880047c0 2024-03-23T03:46:36.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f219fe00640 1 -- 172.21.15.173:0/3754868793 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f218800c240 con 0x7f21880039c0 2024-03-23T03:46:36.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f21a5400640 1 --2- 172.21.15.173:0/3754868793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21880039c0 0x7f2188003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:36.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.089+0000 7f21a4a00640 1 --2- 172.21.15.173:0/3754868793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21880047c0 0x7f2188004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:36.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.090+0000 7f21a5e00640 1 --2- 172.21.15.173:0/3754868793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2188005180 0x7f218800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:36.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.090+0000 7f21a4a00640 1 -- 172.21.15.173:0/3754868793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21880039c0 msgr2=0x7f2188003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:36.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.090+0000 7f21a4a00640 1 --2- 172.21.15.173:0/3754868793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21880039c0 0x7f2188003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.090+0000 7f21a4a00640 1 -- 172.21.15.173:0/3754868793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2188005180 msgr2=0x7f218800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:36.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.090+0000 7f21a4a00640 1 --2- 172.21.15.173:0/3754868793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2188005180 0x7f218800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.090+0000 7f21a4a00640 1 -- 172.21.15.173:0/3754868793 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2188003520 con 0x7f21880047c0 2024-03-23T03:46:36.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.090+0000 7f21a5e00640 1 -- 172.21.15.173:0/3754868793 reap_dead start 2024-03-23T03:46:36.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.091+0000 7f21a4a00640 1 --2- 172.21.15.173:0/3754868793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21880047c0 0x7f2188004c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f218c001750 tx=0x7f21880057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8c231f5e7b6b187 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:36.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.091+0000 7f219fe00640 1 -- 172.21.15.173:0/3754868793 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218c0052d0 con 0x7f21880047c0 2024-03-23T03:46:36.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.091+0000 7f219fe00640 1 -- 172.21.15.173:0/3754868793 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218c02e050 con 0x7f21880047c0 2024-03-23T03:46:36.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.091+0000 7f219fe00640 1 -- 172.21.15.173:0/3754868793 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f218c003830 con 0x7f21880047c0 2024-03-23T03:46:36.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.092+0000 7f21a6800640 1 -- 172.21.15.173:0/3754868793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21880047c0 msgr2=0x7f2188004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:36.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.092+0000 7f21a6800640 1 --2- 172.21.15.173:0/3754868793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21880047c0 0x7f2188004c40 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f218c001750 tx=0x7f21880057f0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.092+0000 7f21a6800640 1 -- 172.21.15.173:0/3754868793 shutdown_connections 2024-03-23T03:46:36.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.092+0000 7f21a6800640 1 --2- 172.21.15.173:0/3754868793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21880047c0 0x7f2188004c40 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.092+0000 7f21a6800640 1 -- 172.21.15.173:0/3754868793 >> 172.21.15.173:0/3754868793 conn(0x7f21a006b100 msgr2=0x7f21a006b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:36.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.092+0000 7f21a6800640 1 -- 172.21.15.173:0/3754868793 shutdown_connections 2024-03-23T03:46:36.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.092+0000 7f21a6800640 1 -- 172.21.15.173:0/3754868793 wait complete. 2024-03-23T03:46:36.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.094+0000 7f21a6800640 1 Processor -- start 2024-03-23T03:46:36.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.094+0000 7f21a6800640 1 -- start start 2024-03-23T03:46:36.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.095+0000 7f21a6800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21880047c0 0x7f21a00b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:36.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.095+0000 7f21a6800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21a0151c10 0x7f21a0152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:36.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.095+0000 7f21a6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21a01571b0 0x7f21a0157660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:36.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.095+0000 7f21a6800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a0157ba0 con 0x7f21880047c0 2024-03-23T03:46:36.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.095+0000 7f21a6800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a01f37a0 con 0x7f21a01571b0 2024-03-23T03:46:36.096 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.095+0000 7f21a6800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a01f2b90 con 0x7f21a0151c10 2024-03-23T03:46:36.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.095+0000 7f21a4a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21a0151c10 0x7f21a0152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:36.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.095+0000 7f21a5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21a01571b0 0x7f21a0157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:36.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.095+0000 7f21a5e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21a01571b0 0x7f21a0157660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:39586/0 (socket says 172.21.15.173:39586) 2024-03-23T03:46:36.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.095+0000 7f21a5e00640 1 -- 172.21.15.173:0/3486718627 learned_addr learned my addr 172.21.15.173:0/3486718627 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-23T03:46:36.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.095+0000 7f21a5400640 1 --2- 172.21.15.173:0/3486718627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21880047c0 0x7f21a00b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:36.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.095+0000 7f21a5e00640 1 -- 172.21.15.173:0/3486718627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21a0151c10 msgr2=0x7f21a0152090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:36.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.095+0000 7f21a5e00640 1 --2- 172.21.15.173:0/3486718627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21a0151c10 0x7f21a0152090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.095+0000 7f21a5e00640 1 -- 172.21.15.173:0/3486718627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21880047c0 msgr2=0x7f21a00b2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:36.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.095+0000 7f21a5e00640 1 --2- 172.21.15.173:0/3486718627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21880047c0 0x7f21a00b2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.095+0000 7f21a5e00640 1 -- 172.21.15.173:0/3486718627 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a01f2e70 con 0x7f21a01571b0 2024-03-23T03:46:36.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.096+0000 7f21a5e00640 1 --2- 172.21.15.173:0/3486718627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21a01571b0 0x7f21a0157660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f21a006bf80 tx=0x7f21980115a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:36.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.096+0000 7f219d600640 1 -- 172.21.15.173:0/3486718627 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219800e070 con 0x7f21a01571b0 2024-03-23T03:46:36.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.096+0000 7f219d600640 1 -- 172.21.15.173:0/3486718627 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2198011de0 con 0x7f21a01571b0 2024-03-23T03:46:36.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.096+0000 7f21a6800640 1 -- 172.21.15.173:0/3486718627 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a01f3150 con 0x7f21a01571b0 2024-03-23T03:46:36.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.096+0000 7f219d600640 1 -- 172.21.15.173:0/3486718627 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198003990 con 0x7f21a01571b0 2024-03-23T03:46:36.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.096+0000 7f21a6800640 1 -- 172.21.15.173:0/3486718627 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a01f28a0 con 0x7f21a01571b0 2024-03-23T03:46:36.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.097+0000 7f219d600640 1 -- 172.21.15.173:0/3486718627 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2198004030 con 0x7f21a01571b0 2024-03-23T03:46:36.098 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.097+0000 7f219d600640 1 -- 172.21.15.173:0/3486718627 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2198016590 con 0x7f21a01571b0 2024-03-23T03:46:36.099 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.098+0000 7f21a6800640 1 -- 172.21.15.173:0/3486718627 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a004e030 con 0x7f21a01571b0 2024-03-23T03:46:36.100 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.100+0000 7f219d600640 1 -- 172.21.15.173:0/3486718627 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f2198004470 con 0x7f21a01571b0 2024-03-23T03:46:36.106 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:36.106+0000 7faa1321e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-23T03:46:36.167 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:36.166+0000 7faa1321e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-23T03:46:36.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.178+0000 7f21a6800640 1 -- 172.21.15.173:0/3486718627 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2c69740f-05ae-4bab-abe9-08e4e449e501", "id": 0} v 0) v1 -- 0x7f21a00d0710 con 0x7f21a01571b0 2024-03-23T03:46:36.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.187+0000 7f219d600640 1 -- 172.21.15.173:0/3486718627 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2c69740f-05ae-4bab-abe9-08e4e449e501", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2198004610 con 0x7f21a01571b0 2024-03-23T03:46:36.187 INFO:teuthology.orchestra.run.smithi173.stdout:0 2024-03-23T03:46:36.189 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.188+0000 7f21a6800640 1 -- 172.21.15.173:0/3486718627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21a01571b0 msgr2=0x7f21a0157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:36.189 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.188+0000 7f21a6800640 1 --2- 172.21.15.173:0/3486718627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21a01571b0 0x7f21a0157660 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f21a006bf80 tx=0x7f21980115a0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.189 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.189+0000 7f21a6800640 1 -- 172.21.15.173:0/3486718627 shutdown_connections 2024-03-23T03:46:36.189 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.189+0000 7f21a6800640 1 --2- 172.21.15.173:0/3486718627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21a01571b0 0x7f21a0157660 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.189 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.189+0000 7f21a6800640 1 --2- 172.21.15.173:0/3486718627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21a0151c10 0x7f21a0152090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.189 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.189+0000 7f21a6800640 1 --2- 172.21.15.173:0/3486718627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21880047c0 0x7f21a00b2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.189 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.189+0000 7f21a6800640 1 -- 172.21.15.173:0/3486718627 >> 172.21.15.173:0/3486718627 conn(0x7f21a006b100 msgr2=0x7f21a00c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:36.190 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.189+0000 7f21a6800640 1 -- 172.21.15.173:0/3486718627 shutdown_connections 2024-03-23T03:46:36.190 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.189+0000 7f21a6800640 1 -- 172.21.15.173:0/3486718627 wait complete. 2024-03-23T03:46:36.205 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new e23064a9-e519-4076-a8c3-a36a6e595694 1 2024-03-23T03:46:36.218 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:36.217+0000 7faa1321e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-23T03:46:36.279 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:36.279+0000 7faa1321e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-23T03:46:36.329 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:36.328+0000 7faa1321e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-23T03:46:36.392 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.391+0000 7f554b200640 1 Processor -- start 2024-03-23T03:46:36.392 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.392+0000 7f554b200640 1 -- start start 2024-03-23T03:46:36.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.393+0000 7f554b200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f554c0b61e0 0x7f554c0ceae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:36.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.393+0000 7f554b200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554c0b3410 con 0x7f554c15a830 2024-03-23T03:46:36.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.393+0000 7f554b200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554c0b36f0 con 0x7f554c0ce160 2024-03-23T03:46:36.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.393+0000 7f554b200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f554c0b39d0 con 0x7f554c0b61e0 2024-03-23T03:46:36.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.393+0000 7f5543e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f554c0b61e0 0x7f554c0ceae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:36.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.393+0000 7f5543e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f554c0b61e0 0x7f554c0ceae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:39588/0 (socket says 172.21.15.173:39588) 2024-03-23T03:46:36.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.393+0000 7f5543e00640 1 -- 172.21.15.173:0/881097698 learned_addr learned my addr 172.21.15.173:0/881097698 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-23T03:46:36.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.394+0000 7f5543e00640 1 -- 172.21.15.173:0/881097698 >> v1:172.21.15.173:6789/0 conn(0x7f554c0ce160 legacy=0x7f554c1643d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:36.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.394+0000 7f5543e00640 1 -- 172.21.15.173:0/881097698 >> v1:172.21.15.102:6789/0 conn(0x7f554c15a830 legacy=0x7f554c0cbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:36.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.394+0000 7f5543e00640 1 -- 172.21.15.173:0/881097698 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c0b3cb0 con 0x7f554c0b61e0 2024-03-23T03:46:36.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.395+0000 7f5543e00640 1 --2- 172.21.15.173:0/881097698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f554c0b61e0 0x7f554c0ceae0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5534009870 tx=0x7f553402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cb32b1a9ad9b64f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:36.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.395+0000 7f5549400640 1 -- 172.21.15.173:0/881097698 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5534031870 con 0x7f554c0b61e0 2024-03-23T03:46:36.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.395+0000 7f5549400640 1 -- 172.21.15.173:0/881097698 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55340319d0 con 0x7f554c0b61e0 2024-03-23T03:46:36.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.395+0000 7f5549400640 1 -- 172.21.15.173:0/881097698 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5534031d60 con 0x7f554c0b61e0 2024-03-23T03:46:36.396 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.396+0000 7f554b200640 1 -- 172.21.15.173:0/881097698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f554c0b61e0 msgr2=0x7f554c0ceae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:36.396 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.396+0000 7f554b200640 1 --2- 172.21.15.173:0/881097698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f554c0b61e0 0x7f554c0ceae0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5534009870 tx=0x7f553402b910 comp rx=0 tx=0).stop 2024-03-23T03:46:36.396 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.396+0000 7f554b200640 1 -- 172.21.15.173:0/881097698 shutdown_connections 2024-03-23T03:46:36.396 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.396+0000 7f554b200640 1 --2- 172.21.15.173:0/881097698 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f554c0b61e0 0x7f554c0ceae0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.396 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.396+0000 7f554b200640 1 -- 172.21.15.173:0/881097698 >> 172.21.15.173:0/881097698 conn(0x7f554c0b2010 msgr2=0x7f554c0b2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:36.396 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.396+0000 7f554b200640 1 -- 172.21.15.173:0/881097698 shutdown_connections 2024-03-23T03:46:36.396 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.396+0000 7f554b200640 1 -- 172.21.15.173:0/881097698 wait complete. 2024-03-23T03:46:36.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.397+0000 7f554b200640 1 Processor -- start 2024-03-23T03:46:36.398 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.398+0000 7f554b200640 1 -- start start 2024-03-23T03:46:36.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.398+0000 7f554b200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f554c0b61e0 0x7f554c1575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:36.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.399+0000 7f554b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f554c0ce160 0x7f554c0b5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:36.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.399+0000 7f554b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f554c15a830 0x7f554c0c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:36.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.399+0000 7f554b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554c0c89a0 con 0x7f554c15a830 2024-03-23T03:46:36.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.399+0000 7f554b200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554c09e1a0 con 0x7f554c0ce160 2024-03-23T03:46:36.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.399+0000 7f554b200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554c0b5990 con 0x7f554c0b61e0 2024-03-23T03:46:36.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.399+0000 7f5543e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f554c0ce160 0x7f554c0b5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:36.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.399+0000 7f5549e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f554c0b61e0 0x7f554c1575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:36.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.399+0000 7f5543e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f554c0ce160 0x7f554c0b5450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:39598/0 (socket says 172.21.15.173:39598) 2024-03-23T03:46:36.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.399+0000 7f5543e00640 1 -- 172.21.15.173:0/3063468884 learned_addr learned my addr 172.21.15.173:0/3063468884 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-23T03:46:36.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.399+0000 7f554a800640 1 --2- 172.21.15.173:0/3063468884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f554c15a830 0x7f554c0c9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:36.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.399+0000 7f5543e00640 1 -- 172.21.15.173:0/3063468884 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f554c0b61e0 msgr2=0x7f554c1575e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:36.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.399+0000 7f5543e00640 1 --2- 172.21.15.173:0/3063468884 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f554c0b61e0 0x7f554c1575e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.399+0000 7f5543e00640 1 -- 172.21.15.173:0/3063468884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f554c15a830 msgr2=0x7f554c0c9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:36.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.399+0000 7f5543e00640 1 --2- 172.21.15.173:0/3063468884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f554c15a830 0x7f554c0c9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.401 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.399+0000 7f5543e00640 1 -- 172.21.15.173:0/3063468884 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55340094d0 con 0x7f554c0ce160 2024-03-23T03:46:36.402 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.399+0000 7f554a800640 1 --2- 172.21.15.173:0/3063468884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f554c15a830 0x7f554c0c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:46:36.402 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.399+0000 7f5549e00640 1 --2- 172.21.15.173:0/3063468884 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f554c0b61e0 0x7f554c1575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:46:36.402 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.400+0000 7f5543e00640 1 --2- 172.21.15.173:0/3063468884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f554c0ce160 0x7f554c0b5450 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5534009870 tx=0x7f55340046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:36.402 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.400+0000 7f5542000640 1 -- 172.21.15.173:0/3063468884 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5534031820 con 0x7f554c0ce160 2024-03-23T03:46:36.402 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.400+0000 7f5542000640 1 -- 172.21.15.173:0/3063468884 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55340443f0 con 0x7f554c0ce160 2024-03-23T03:46:36.402 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.400+0000 7f554b200640 1 -- 172.21.15.173:0/3063468884 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f554c06b2b0 con 0x7f554c0ce160 2024-03-23T03:46:36.402 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.400+0000 7f5542000640 1 -- 172.21.15.173:0/3063468884 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55340447a0 con 0x7f554c0ce160 2024-03-23T03:46:36.402 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.400+0000 7f554b200640 1 -- 172.21.15.173:0/3063468884 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f554c0c86c0 con 0x7f554c0ce160 2024-03-23T03:46:36.402 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.400+0000 7f5542000640 1 -- 172.21.15.173:0/3063468884 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5534044c00 con 0x7f554c0ce160 2024-03-23T03:46:36.402 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.400+0000 7f5542000640 1 -- 172.21.15.173:0/3063468884 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f553402d040 con 0x7f554c0ce160 2024-03-23T03:46:36.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.401+0000 7f554b200640 1 -- 172.21.15.173:0/3063468884 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f554c06b440 con 0x7f554c0ce160 2024-03-23T03:46:36.403 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.403+0000 7f5542000640 1 -- 172.21.15.173:0/3063468884 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f5534035020 con 0x7f554c0ce160 2024-03-23T03:46:36.458 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:36.457+0000 7faa1321e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-23T03:46:36.469 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:36.469+0000 7f7d4922c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-23T03:46:36.482 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.481+0000 7f554b200640 1 -- 172.21.15.173:0/3063468884 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e23064a9-e519-4076-a8c3-a36a6e595694", "id": 1} v 0) v1 -- 0x7f554c04e030 con 0x7f554c0ce160 2024-03-23T03:46:36.491 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.491+0000 7f5542000640 1 -- 172.21.15.173:0/3063468884 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e23064a9-e519-4076-a8c3-a36a6e595694", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f553404dc70 con 0x7f554c0ce160 2024-03-23T03:46:36.491 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-03-23T03:46:36.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.492+0000 7f554b200640 1 -- 172.21.15.173:0/3063468884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f554c0ce160 msgr2=0x7f554c0b5450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:36.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.492+0000 7f554b200640 1 --2- 172.21.15.173:0/3063468884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f554c0ce160 0x7f554c0b5450 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5534009870 tx=0x7f55340046c0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.493+0000 7f554b200640 1 -- 172.21.15.173:0/3063468884 shutdown_connections 2024-03-23T03:46:36.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.493+0000 7f554b200640 1 --2- 172.21.15.173:0/3063468884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f554c15a830 0x7f554c0c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.493+0000 7f554b200640 1 --2- 172.21.15.173:0/3063468884 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f554c0ce160 0x7f554c0b5450 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.493+0000 7f554b200640 1 --2- 172.21.15.173:0/3063468884 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f554c0b61e0 0x7f554c1575e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.493+0000 7f554b200640 1 -- 172.21.15.173:0/3063468884 >> 172.21.15.173:0/3063468884 conn(0x7f554c0b2010 msgr2=0x7f554c15d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:36.493 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.493+0000 7f554b200640 1 -- 172.21.15.173:0/3063468884 shutdown_connections 2024-03-23T03:46:36.494 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.493+0000 7f554b200640 1 -- 172.21.15.173:0/3063468884 wait complete. 2024-03-23T03:46:36.509 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new d5c691e5-4b59-4203-9483-09d0a433e855 2 2024-03-23T03:46:36.535 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:36.535+0000 7f7d4922c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-23T03:46:36.536 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:36.536+0000 7faa1321e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-23T03:46:36.589 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:36.589+0000 7f7d4922c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-23T03:46:36.589 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:36.588+0000 7faa1321e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-23T03:46:36.648 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:36.648+0000 7faa1321e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-23T03:46:36.654 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:36.654+0000 7f7d4922c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-23T03:46:36.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.696+0000 7f355ca00640 1 Processor -- start 2024-03-23T03:46:36.697 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.697+0000 7f355ca00640 1 -- start start 2024-03-23T03:46:36.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.698+0000 7f355ca00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f355815cc30 0x7f35580d16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:36.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.698+0000 7f355ca00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f35580b62a0 0x7f35580cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:36.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.698+0000 7f355ca00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3558160ae0 con 0x7f35580d1be0 2024-03-23T03:46:36.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.698+0000 7f355ca00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35580d0c00 con 0x7f355815cc30 2024-03-23T03:46:36.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.698+0000 7f355ca00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35580cdde0 con 0x7f35580b62a0 2024-03-23T03:46:36.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.699+0000 7f3557400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f355815cc30 0x7f35580d16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:36.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.699+0000 7f3557400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f355815cc30 0x7f35580d16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:39614/0 (socket says 172.21.15.173:39614) 2024-03-23T03:46:36.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.699+0000 7f3557400640 1 -- 172.21.15.173:0/2316773504 learned_addr learned my addr 172.21.15.173:0/2316773504 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-23T03:46:36.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.699+0000 7f3557e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f35580b62a0 0x7f35580cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:36.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.699+0000 7f3557400640 1 -- 172.21.15.173:0/2316773504 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f35580b62a0 msgr2=0x7f35580cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:36.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.699+0000 7f3557400640 1 --2- 172.21.15.173:0/2316773504 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f35580b62a0 0x7f35580cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.699+0000 7f3557400640 1 -- 172.21.15.173:0/2316773504 >> v1:172.21.15.102:6789/0 conn(0x7f35580d1be0 legacy=0x7f35580b5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:36.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.699+0000 7f3557400640 1 -- 172.21.15.173:0/2316773504 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35580d0ea0 con 0x7f355815cc30 2024-03-23T03:46:36.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.700+0000 7f3557400640 1 --2- 172.21.15.173:0/2316773504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f355815cc30 0x7f35580d16a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3548009870 tx=0x7f354802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df251e47ff3d0e31 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:36.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.700+0000 7f3556000640 1 -- 172.21.15.173:0/2316773504 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3548031640 con 0x7f355815cc30 2024-03-23T03:46:36.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.700+0000 7f3556000640 1 -- 172.21.15.173:0/2316773504 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35480317a0 con 0x7f355815cc30 2024-03-23T03:46:36.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.700+0000 7f3556000640 1 -- 172.21.15.173:0/2316773504 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3548031b30 con 0x7f355815cc30 2024-03-23T03:46:36.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.701+0000 7f355ca00640 1 -- 172.21.15.173:0/2316773504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f355815cc30 msgr2=0x7f35580d16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:36.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.701+0000 7f355ca00640 1 --2- 172.21.15.173:0/2316773504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f355815cc30 0x7f35580d16a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3548009870 tx=0x7f354802b8f0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.701+0000 7f355ca00640 1 -- 172.21.15.173:0/2316773504 shutdown_connections 2024-03-23T03:46:36.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.701+0000 7f355ca00640 1 --2- 172.21.15.173:0/2316773504 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f35580b62a0 0x7f35580cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.701+0000 7f355ca00640 1 --2- 172.21.15.173:0/2316773504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f355815cc30 0x7f35580d16a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.701+0000 7f355ca00640 1 -- 172.21.15.173:0/2316773504 >> 172.21.15.173:0/2316773504 conn(0x7f355806b100 msgr2=0x7f355806b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:36.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.701+0000 7f355ca00640 1 -- 172.21.15.173:0/2316773504 shutdown_connections 2024-03-23T03:46:36.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.701+0000 7f355ca00640 1 -- 172.21.15.173:0/2316773504 wait complete. 2024-03-23T03:46:36.703 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:36.702+0000 7faa1321e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-23T03:46:36.707 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.703+0000 7f355ca00640 1 Processor -- start 2024-03-23T03:46:36.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.703+0000 7f355ca00640 1 -- start start 2024-03-23T03:46:36.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.704+0000 7f355ca00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f35580b62a0 0x7f3558159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:36.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.704+0000 7f355ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35580d1be0 0x7f35581f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:36.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.704+0000 7f355ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f355815cc30 0x7f355815e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:36.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.704+0000 7f355ca00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35580a9390 con 0x7f355815cc30 2024-03-23T03:46:36.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.704+0000 7f355ca00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35581f2f20 con 0x7f35580d1be0 2024-03-23T03:46:36.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.704+0000 7f355ca00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f355815d350 con 0x7f35580b62a0 2024-03-23T03:46:36.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.704+0000 7f3557400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f35580b62a0 0x7f3558159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:36.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.704+0000 7f3556a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35580d1be0 0x7f35581f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:36.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.704+0000 7f3557400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f35580b62a0 0x7f3558159f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.173:42552/0 (socket says 172.21.15.173:42552) 2024-03-23T03:46:36.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.704+0000 7f3557400640 1 -- 172.21.15.173:0/926908612 learned_addr learned my addr 172.21.15.173:0/926908612 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-23T03:46:36.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.704+0000 7f3557e00640 1 --2- 172.21.15.173:0/926908612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f355815cc30 0x7f355815e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:36.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.704+0000 7f3557400640 1 -- 172.21.15.173:0/926908612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35580d1be0 msgr2=0x7f35581f29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:36.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.704+0000 7f3557400640 1 --2- 172.21.15.173:0/926908612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35580d1be0 0x7f35581f29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.704+0000 7f3557400640 1 -- 172.21.15.173:0/926908612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f355815cc30 msgr2=0x7f355815e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:36.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.704+0000 7f3557400640 1 --2- 172.21.15.173:0/926908612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f355815cc30 0x7f355815e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.704+0000 7f3557400640 1 -- 172.21.15.173:0/926908612 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35480094d0 con 0x7f35580b62a0 2024-03-23T03:46:36.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.704+0000 7f3556a00640 1 --2- 172.21.15.173:0/926908612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35580d1be0 0x7f35581f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:46:36.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.705+0000 7f3557400640 1 --2- 172.21.15.173:0/926908612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f35580b62a0 0x7f3558159f20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3548009870 tx=0x7f35480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:36.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.705+0000 7f354f400640 1 -- 172.21.15.173:0/926908612 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354803a930 con 0x7f35580b62a0 2024-03-23T03:46:36.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.705+0000 7f354f400640 1 -- 172.21.15.173:0/926908612 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354803aa90 con 0x7f35580b62a0 2024-03-23T03:46:36.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.705+0000 7f355ca00640 1 -- 172.21.15.173:0/926908612 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355815d630 con 0x7f35580b62a0 2024-03-23T03:46:36.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.705+0000 7f354f400640 1 -- 172.21.15.173:0/926908612 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354803ae40 con 0x7f35580b62a0 2024-03-23T03:46:36.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.705+0000 7f355ca00640 1 -- 172.21.15.173:0/926908612 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355815dbf0 con 0x7f35580b62a0 2024-03-23T03:46:36.712 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.705+0000 7f354f400640 1 -- 172.21.15.173:0/926908612 <== mon.2 v2:172.21.15.173:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f354802f050 con 0x7f35580b62a0 2024-03-23T03:46:36.712 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.706+0000 7f354f400640 1 -- 172.21.15.173:0/926908612 <== mon.2 v2:172.21.15.173:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f3548044950 con 0x7f35580b62a0 2024-03-23T03:46:36.712 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.706+0000 7f355ca00640 1 -- 172.21.15.173:0/926908612 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355804e030 con 0x7f35580b62a0 2024-03-23T03:46:36.713 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:36.708+0000 7f7d4922c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-23T03:46:36.713 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.711+0000 7f354f400640 1 -- 172.21.15.173:0/926908612 <== mon.2 v2:172.21.15.173:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f3548035020 con 0x7f35580b62a0 2024-03-23T03:46:36.789 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.788+0000 7f355ca00640 1 -- 172.21.15.173:0/926908612 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d5c691e5-4b59-4203-9483-09d0a433e855", "id": 2} v 0) v1 -- 0x7f35580d0710 con 0x7f35580b62a0 2024-03-23T03:46:36.796 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.796+0000 7f354f400640 1 -- 172.21.15.173:0/926908612 <== mon.2 v2:172.21.15.173:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d5c691e5-4b59-4203-9483-09d0a433e855", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f354802d040 con 0x7f35580b62a0 2024-03-23T03:46:36.796 INFO:teuthology.orchestra.run.smithi173.stdout:2 2024-03-23T03:46:36.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.797+0000 7f355ca00640 1 -- 172.21.15.173:0/926908612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f35580b62a0 msgr2=0x7f3558159f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:36.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.797+0000 7f355ca00640 1 --2- 172.21.15.173:0/926908612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f35580b62a0 0x7f3558159f20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3548009870 tx=0x7f35480046c0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.797+0000 7f355ca00640 1 -- 172.21.15.173:0/926908612 shutdown_connections 2024-03-23T03:46:36.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.797+0000 7f355ca00640 1 --2- 172.21.15.173:0/926908612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f355815cc30 0x7f355815e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.797+0000 7f355ca00640 1 --2- 172.21.15.173:0/926908612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35580d1be0 0x7f35581f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.797+0000 7f355ca00640 1 --2- 172.21.15.173:0/926908612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f35580b62a0 0x7f3558159f20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:36.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.797+0000 7f355ca00640 1 -- 172.21.15.173:0/926908612 >> 172.21.15.173:0/926908612 conn(0x7f355806b100 msgr2=0x7f35580c6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:36.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.798+0000 7f355ca00640 1 -- 172.21.15.173:0/926908612 shutdown_connections 2024-03-23T03:46:36.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:36.798+0000 7f355ca00640 1 -- 172.21.15.173:0/926908612 wait complete. 2024-03-23T03:46:36.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 69470c30-bacd-4c78-a774-0a17ad1608b1 3 2024-03-23T03:46:36.846 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:36.846+0000 7f7d4922c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-23T03:46:36.932 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:36.932+0000 7f7d4922c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-23T03:46:36.988 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:36.988+0000 7f7d4922c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-23T03:46:37.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.037+0000 7fd0a9600640 1 Processor -- start 2024-03-23T03:46:37.038 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.037+0000 7fd0a9600640 1 -- start start 2024-03-23T03:46:37.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.039+0000 7fd0a9600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0a40b84b0 0x7fd0a40d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.040 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.039+0000 7fd0a9600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a40d1280 con 0x7fd0a40b5aa0 2024-03-23T03:46:37.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.039+0000 7fd0a9600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a40d20c0 con 0x7fd0a40cd740 2024-03-23T03:46:37.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.039+0000 7fd0a9600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a40b8910 con 0x7fd0a40b84b0 2024-03-23T03:46:37.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.039+0000 7fd0a2a00640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fd0a40cd740 0x7fd0a40cdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56278/0 (socket says 172.21.15.173:56278) 2024-03-23T03:46:37.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.039+0000 7fd0a2a00640 1 -- 172.21.15.173:0/4289852181 learned_addr learned my addr 172.21.15.173:0/4289852181 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-23T03:46:37.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.039+0000 7fd09be00640 1 --2- 172.21.15.173:0/4289852181 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0a40b84b0 0x7fd0a40d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.041 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.040+0000 7fd0a2000640 1 -- 172.21.15.173:0/4289852181 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2130012412 0 0) 0x7fd0a40d20c0 con 0x7fd0a40cd740 2024-03-23T03:46:37.042 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.040+0000 7fd09be00640 1 -- 172.21.15.173:0/4289852181 >> v1:172.21.15.173:6789/0 conn(0x7fd0a40cd740 legacy=0x7fd0a40cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.042 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.040+0000 7fd09be00640 1 -- 172.21.15.173:0/4289852181 >> v1:172.21.15.102:6789/0 conn(0x7fd0a40b5aa0 legacy=0x7fd0a40b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:37.042 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.040+0000 7fd09be00640 1 -- 172.21.15.173:0/4289852181 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a40d14b0 con 0x7fd0a40b84b0 2024-03-23T03:46:37.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.041+0000 7fd09be00640 1 --2- 172.21.15.173:0/4289852181 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0a40b84b0 0x7fd0a40d0d40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd08c0097d0 tx=0x7fd08c02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=429f6e4757858075 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:37.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.041+0000 7fd0a2000640 1 -- 172.21.15.173:0/4289852181 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c031800 con 0x7fd0a40b84b0 2024-03-23T03:46:37.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.041+0000 7fd0a2000640 1 -- 172.21.15.173:0/4289852181 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08c031960 con 0x7fd0a40b84b0 2024-03-23T03:46:37.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.041+0000 7fd0a2000640 1 -- 172.21.15.173:0/4289852181 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c031cf0 con 0x7fd0a40b84b0 2024-03-23T03:46:37.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.042+0000 7fd0a9600640 1 -- 172.21.15.173:0/4289852181 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0a40b84b0 msgr2=0x7fd0a40d0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.042+0000 7fd0a9600640 1 --2- 172.21.15.173:0/4289852181 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0a40b84b0 0x7fd0a40d0d40 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd08c0097d0 tx=0x7fd08c02b8a0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.042+0000 7fd0a9600640 1 -- 172.21.15.173:0/4289852181 shutdown_connections 2024-03-23T03:46:37.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.042+0000 7fd0a9600640 1 --2- 172.21.15.173:0/4289852181 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0a40b84b0 0x7fd0a40d0d40 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.042+0000 7fd0a9600640 1 -- 172.21.15.173:0/4289852181 >> 172.21.15.173:0/4289852181 conn(0x7fd0a40cf5d0 msgr2=0x7fd0a40cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:37.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.042+0000 7fd0a9600640 1 -- 172.21.15.173:0/4289852181 shutdown_connections 2024-03-23T03:46:37.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.042+0000 7fd0a9600640 1 -- 172.21.15.173:0/4289852181 wait complete. 2024-03-23T03:46:37.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.044+0000 7fd0a9600640 1 Processor -- start 2024-03-23T03:46:37.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.044+0000 7fd0a9600640 1 -- start start 2024-03-23T03:46:37.047 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd0a9600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0a40b5aa0 0x7fd0a40b4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd0a9600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0a40b84b0 0x7fd0a41544e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd0a9600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0a40cd740 0x7fd0a4154a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd0a9600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a40c6460 con 0x7fd0a40b84b0 2024-03-23T03:46:37.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd0a9600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a40c6740 con 0x7fd0a40cd740 2024-03-23T03:46:37.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd0a9600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a40c6a20 con 0x7fd0a40b5aa0 2024-03-23T03:46:37.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd0a2a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0a40b5aa0 0x7fd0a40b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd0a3400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0a40cd740 0x7fd0a4154a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd0a2a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0a40b5aa0 0x7fd0a40b4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.173:42572/0 (socket says 172.21.15.173:42572) 2024-03-23T03:46:37.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd0a2a00640 1 -- 172.21.15.173:0/2275239955 learned_addr learned my addr 172.21.15.173:0/2275239955 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-23T03:46:37.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd09be00640 1 --2- 172.21.15.173:0/2275239955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0a40b84b0 0x7fd0a41544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd0a3400640 1 -- 172.21.15.173:0/2275239955 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0a40b5aa0 msgr2=0x7fd0a40b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd0a3400640 1 --2- 172.21.15.173:0/2275239955 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0a40b5aa0 0x7fd0a40b4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.048 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd0a3400640 1 -- 172.21.15.173:0/2275239955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0a40b84b0 msgr2=0x7fd0a41544e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd0a3400640 1 --2- 172.21.15.173:0/2275239955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0a40b84b0 0x7fd0a41544e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd0a3400640 1 -- 172.21.15.173:0/2275239955 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08c009400 con 0x7fd0a40cd740 2024-03-23T03:46:37.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd0a2a00640 1 --2- 172.21.15.173:0/2275239955 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0a40b5aa0 0x7fd0a40b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:46:37.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.045+0000 7fd09be00640 1 --2- 172.21.15.173:0/2275239955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0a40b84b0 0x7fd0a41544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:46:37.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.046+0000 7fd0a3400640 1 --2- 172.21.15.173:0/2275239955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0a40cd740 0x7fd0a4154a20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd09c00c780 tx=0x7fd09c00cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:37.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.046+0000 7fd09aa00640 1 -- 172.21.15.173:0/2275239955 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c00aa50 con 0x7fd0a40cd740 2024-03-23T03:46:37.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.046+0000 7fd09aa00640 1 -- 172.21.15.173:0/2275239955 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09c00abb0 con 0x7fd0a40cd740 2024-03-23T03:46:37.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.046+0000 7fd0a9600640 1 -- 172.21.15.173:0/2275239955 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a40c6d00 con 0x7fd0a40cd740 2024-03-23T03:46:37.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.046+0000 7fd09aa00640 1 -- 172.21.15.173:0/2275239955 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c00ce30 con 0x7fd0a40cd740 2024-03-23T03:46:37.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.046+0000 7fd0a9600640 1 -- 172.21.15.173:0/2275239955 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a40c72c0 con 0x7fd0a40cd740 2024-03-23T03:46:37.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.046+0000 7fd09aa00640 1 -- 172.21.15.173:0/2275239955 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd09c004bf0 con 0x7fd0a40cd740 2024-03-23T03:46:37.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.046+0000 7fd09aa00640 1 -- 172.21.15.173:0/2275239955 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fd09c0054b0 con 0x7fd0a40cd740 2024-03-23T03:46:37.049 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.047+0000 7fd0a9600640 1 -- 172.21.15.173:0/2275239955 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0a4073390 con 0x7fd0a40cd740 2024-03-23T03:46:37.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.052+0000 7fd09aa00640 1 -- 172.21.15.173:0/2275239955 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fd09c039b50 con 0x7fd0a40cd740 2024-03-23T03:46:37.054 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:37.054+0000 7f7d4922c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-23T03:46:37.111 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:37.111+0000 7f7d4922c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-23T03:46:37.130 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.130+0000 7fd0a9600640 1 -- 172.21.15.173:0/2275239955 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd new", "uuid": "69470c30-bacd-4c78-a774-0a17ad1608b1", "id": 3} v 0) v1 -- 0x7fd0a40b3230 con 0x7fd0a40cd740 2024-03-23T03:46:37.140 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.140+0000 7fd09aa00640 1 -- 172.21.15.173:0/2275239955 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "69470c30-bacd-4c78-a774-0a17ad1608b1", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd09c028070 con 0x7fd0a40cd740 2024-03-23T03:46:37.141 INFO:teuthology.orchestra.run.smithi173.stdout:3 2024-03-23T03:46:37.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.141+0000 7fd0a9600640 1 -- 172.21.15.173:0/2275239955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0a40cd740 msgr2=0x7fd0a4154a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.141+0000 7fd0a9600640 1 --2- 172.21.15.173:0/2275239955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0a40cd740 0x7fd0a4154a20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd09c00c780 tx=0x7fd09c00cc40 comp rx=0 tx=0).stop 2024-03-23T03:46:37.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.142+0000 7fd0a9600640 1 -- 172.21.15.173:0/2275239955 shutdown_connections 2024-03-23T03:46:37.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.142+0000 7fd0a9600640 1 --2- 172.21.15.173:0/2275239955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0a40cd740 0x7fd0a4154a20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.142+0000 7fd0a9600640 1 --2- 172.21.15.173:0/2275239955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0a40b84b0 0x7fd0a41544e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.142 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.142+0000 7fd0a9600640 1 --2- 172.21.15.173:0/2275239955 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0a40b5aa0 0x7fd0a40b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.142+0000 7fd0a9600640 1 -- 172.21.15.173:0/2275239955 >> 172.21.15.173:0/2275239955 conn(0x7fd0a40cf5d0 msgr2=0x7fd0a415eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:37.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.142+0000 7fd0a9600640 1 -- 172.21.15.173:0/2275239955 shutdown_connections 2024-03-23T03:46:37.143 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.142+0000 7fd0a9600640 1 -- 172.21.15.173:0/2275239955 wait complete. 2024-03-23T03:46:37.149 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:37.148+0000 7faa1321e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-23T03:46:37.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 939c44ed-ffc0-4066-b4a8-667943081e0c 4 2024-03-23T03:46:37.306 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:37.306+0000 7faa1321e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-23T03:46:37.353 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.353+0000 7fc94aa00640 1 Processor -- start 2024-03-23T03:46:37.353 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.353+0000 7fc94aa00640 1 -- start start 2024-03-23T03:46:37.355 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.354+0000 7fc94aa00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9440cd740 0x7fc9440b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.355 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.354+0000 7fc94aa00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc9440b5aa0 0x7fc9440b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.355 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.354+0000 7fc94aa00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9440d20c0 con 0x7fc9440d0d40 2024-03-23T03:46:37.355 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.354+0000 7fc94aa00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9440d14b0 con 0x7fc9440b5aa0 2024-03-23T03:46:37.355 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.354+0000 7fc94aa00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9440d1790 con 0x7fc9440cd740 2024-03-23T03:46:37.355 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.354+0000 7fc943e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9440cd740 0x7fc9440b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.354+0000 7fc943e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9440cd740 0x7fc9440b84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.173:42578/0 (socket says 172.21.15.173:42578) 2024-03-23T03:46:37.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.354+0000 7fc948a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc9440b5aa0 0x7fc9440b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.354+0000 7fc943e00640 1 -- 172.21.15.173:0/1796157704 learned_addr learned my addr 172.21.15.173:0/1796157704 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-23T03:46:37.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.354+0000 7fc93be00640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fc9440d0d40 0x7fc9440cdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56294/0 (socket says 172.21.15.173:56294) 2024-03-23T03:46:37.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.355+0000 7fc943e00640 1 -- 172.21.15.173:0/1796157704 >> v2:172.21.15.173:3300/0 conn(0x7fc9440b5aa0 msgr2=0x7fc9440b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:37.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.355+0000 7fc943e00640 1 --2- 172.21.15.173:0/1796157704 >> v2:172.21.15.173:3300/0 conn(0x7fc9440b5aa0 0x7fc9440b5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.355+0000 7fc943e00640 1 -- 172.21.15.173:0/1796157704 >> v1:172.21.15.173:6789/0 conn(0x7fc9440d0d40 legacy=0x7fc9440cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.355+0000 7fc943e00640 1 -- 172.21.15.173:0/1796157704 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9440d1a70 con 0x7fc9440cd740 2024-03-23T03:46:37.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.355+0000 7fc943400640 1 -- 172.21.15.173:0/1796157704 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138139430 0 0) 0x7fc9440d20c0 con 0x7fc9440d0d40 2024-03-23T03:46:37.356 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.355+0000 7fc948a00640 1 --2- 172.21.15.173:0/1796157704 >> v2:172.21.15.173:3300/0 conn(0x7fc9440b5aa0 0x7fc9440b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:46:37.357 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.356+0000 7fc943e00640 1 --2- 172.21.15.173:0/1796157704 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9440cd740 0x7fc9440b84b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc934009870 tx=0x7fc93402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=601f276c3a2f540f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:37.357 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.356+0000 7fc943400640 1 -- 172.21.15.173:0/1796157704 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934030840 con 0x7fc9440cd740 2024-03-23T03:46:37.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.356+0000 7fc943400640 1 -- 172.21.15.173:0/1796157704 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9340309a0 con 0x7fc9440cd740 2024-03-23T03:46:37.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.356+0000 7fc943400640 1 -- 172.21.15.173:0/1796157704 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc934030d30 con 0x7fc9440cd740 2024-03-23T03:46:37.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.357+0000 7fc94aa00640 1 -- 172.21.15.173:0/1796157704 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9440cd740 msgr2=0x7fc9440b84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.357+0000 7fc94aa00640 1 --2- 172.21.15.173:0/1796157704 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9440cd740 0x7fc9440b84b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc934009870 tx=0x7fc93402b980 comp rx=0 tx=0).stop 2024-03-23T03:46:37.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.357+0000 7fc94aa00640 1 -- 172.21.15.173:0/1796157704 shutdown_connections 2024-03-23T03:46:37.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.357+0000 7fc94aa00640 1 --2- 172.21.15.173:0/1796157704 >> v2:172.21.15.173:3300/0 conn(0x7fc9440b5aa0 0x7fc9440b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.357+0000 7fc94aa00640 1 --2- 172.21.15.173:0/1796157704 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9440cd740 0x7fc9440b84b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.357+0000 7fc94aa00640 1 -- 172.21.15.173:0/1796157704 >> 172.21.15.173:0/1796157704 conn(0x7fc9440cf5d0 msgr2=0x7fc9440cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:37.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.357+0000 7fc94aa00640 1 -- 172.21.15.173:0/1796157704 shutdown_connections 2024-03-23T03:46:37.358 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.357+0000 7fc94aa00640 1 -- 172.21.15.173:0/1796157704 wait complete. 2024-03-23T03:46:37.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.359+0000 7fc94aa00640 1 Processor -- start 2024-03-23T03:46:37.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.359+0000 7fc94aa00640 1 -- start start 2024-03-23T03:46:37.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.360+0000 7fc94aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9440b5aa0 0x7fc94414f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.360+0000 7fc94aa00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9440cd740 0x7fc94414e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.360+0000 7fc94aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9440d0d40 0x7fc9440cbc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.360+0000 7fc94aa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94414fc20 con 0x7fc9440b5aa0 2024-03-23T03:46:37.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.360+0000 7fc94aa00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94414d200 con 0x7fc9440d0d40 2024-03-23T03:46:37.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.360+0000 7fc94aa00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc94414d4e0 con 0x7fc9440cd740 2024-03-23T03:46:37.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.360+0000 7fc93be00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9440cd740 0x7fc94414e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.360+0000 7fc93be00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9440cd740 0x7fc94414e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.173:42580/0 (socket says 172.21.15.173:42580) 2024-03-23T03:46:37.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.360+0000 7fc93be00640 1 -- 172.21.15.173:0/1811538230 learned_addr learned my addr 172.21.15.173:0/1811538230 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-23T03:46:37.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.360+0000 7fc948a00640 1 --2- 172.21.15.173:0/1811538230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9440d0d40 0x7fc9440cbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.360+0000 7fc943e00640 1 --2- 172.21.15.173:0/1811538230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9440b5aa0 0x7fc94414f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.361+0000 7fc948a00640 1 -- 172.21.15.173:0/1811538230 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9440cd740 msgr2=0x7fc94414e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.361+0000 7fc948a00640 1 --2- 172.21.15.173:0/1811538230 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9440cd740 0x7fc94414e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.361+0000 7fc948a00640 1 -- 172.21.15.173:0/1811538230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9440b5aa0 msgr2=0x7fc94414f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.361+0000 7fc948a00640 1 --2- 172.21.15.173:0/1811538230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9440b5aa0 0x7fc94414f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.361+0000 7fc948a00640 1 -- 172.21.15.173:0/1811538230 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93c009a70 con 0x7fc9440d0d40 2024-03-23T03:46:37.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.361+0000 7fc93be00640 1 --2- 172.21.15.173:0/1811538230 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9440cd740 0x7fc94414e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:46:37.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.361+0000 7fc948a00640 1 --2- 172.21.15.173:0/1811538230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9440d0d40 0x7fc9440cbc10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc93c00efc0 tx=0x7fc93c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:37.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.361+0000 7fc940c00640 1 -- 172.21.15.173:0/1811538230 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c00cd30 con 0x7fc9440d0d40 2024-03-23T03:46:37.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.361+0000 7fc940c00640 1 -- 172.21.15.173:0/1811538230 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc93c00f040 con 0x7fc9440d0d40 2024-03-23T03:46:37.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.361+0000 7fc940c00640 1 -- 172.21.15.173:0/1811538230 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc93c003bd0 con 0x7fc9440d0d40 2024-03-23T03:46:37.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.361+0000 7fc94aa00640 1 -- 172.21.15.173:0/1811538230 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9340094d0 con 0x7fc9440d0d40 2024-03-23T03:46:37.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.361+0000 7fc94aa00640 1 -- 172.21.15.173:0/1811538230 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94414daa0 con 0x7fc9440d0d40 2024-03-23T03:46:37.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.362+0000 7fc940c00640 1 -- 172.21.15.173:0/1811538230 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc93c014780 con 0x7fc9440d0d40 2024-03-23T03:46:37.363 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.362+0000 7fc940c00640 1 -- 172.21.15.173:0/1811538230 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fc93c01d410 con 0x7fc9440d0d40 2024-03-23T03:46:37.364 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.362+0000 7fc94aa00640 1 -- 172.21.15.173:0/1811538230 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9440d2530 con 0x7fc9440d0d40 2024-03-23T03:46:37.365 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.365+0000 7fc940c00640 1 -- 172.21.15.173:0/1811538230 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fc9440d2530 con 0x7fc9440d0d40 2024-03-23T03:46:37.443 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.442+0000 7fc94aa00640 1 -- 172.21.15.173:0/1811538230 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd new", "uuid": "939c44ed-ffc0-4066-b4a8-667943081e0c", "id": 4} v 0) v1 -- 0x7fc944073390 con 0x7fc9440d0d40 2024-03-23T03:46:37.445 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:37.445+0000 7faa1321e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-23T03:46:37.453 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.453+0000 7fc940c00640 1 -- 172.21.15.173:0/1811538230 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "939c44ed-ffc0-4066-b4a8-667943081e0c", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc93c018030 con 0x7fc9440d0d40 2024-03-23T03:46:37.453 INFO:teuthology.orchestra.run.smithi173.stdout:4 2024-03-23T03:46:37.454 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.454+0000 7fc94aa00640 1 -- 172.21.15.173:0/1811538230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9440d0d40 msgr2=0x7fc9440cbc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.455 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.454+0000 7fc94aa00640 1 --2- 172.21.15.173:0/1811538230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9440d0d40 0x7fc9440cbc10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc93c00efc0 tx=0x7fc93c00c490 comp rx=0 tx=0).stop 2024-03-23T03:46:37.455 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.454+0000 7fc94aa00640 1 -- 172.21.15.173:0/1811538230 shutdown_connections 2024-03-23T03:46:37.455 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.454+0000 7fc94aa00640 1 --2- 172.21.15.173:0/1811538230 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc9440d0d40 0x7fc9440cbc10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.455 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.454+0000 7fc94aa00640 1 --2- 172.21.15.173:0/1811538230 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9440cd740 0x7fc94414e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.455 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.454+0000 7fc94aa00640 1 --2- 172.21.15.173:0/1811538230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc9440b5aa0 0x7fc94414f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.455 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.454+0000 7fc94aa00640 1 -- 172.21.15.173:0/1811538230 >> 172.21.15.173:0/1811538230 conn(0x7fc9440cf5d0 msgr2=0x7fc9440cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:37.455 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.454+0000 7fc94aa00640 1 -- 172.21.15.173:0/1811538230 shutdown_connections 2024-03-23T03:46:37.455 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.454+0000 7fc94aa00640 1 -- 172.21.15.173:0/1811538230 wait complete. 2024-03-23T03:46:37.470 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 7f78f3b3-7b77-45fc-b705-6637c50bf18e 5 2024-03-23T03:46:37.498 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:37.497+0000 7faa1321e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-23T03:46:37.593 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:37.591+0000 7f7d4922c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-23T03:46:37.622 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:37.621+0000 7faa1321e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-23T03:46:37.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.660+0000 7fc496000640 1 Processor -- start 2024-03-23T03:46:37.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.661+0000 7fc496000640 1 -- start start 2024-03-23T03:46:37.662 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.662+0000 7fc496000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc4900b47d0 0x7fc4900b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.662+0000 7fc496000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4901617d0 con 0x7fc49015cd10 2024-03-23T03:46:37.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.662+0000 7fc496000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4900b5150 con 0x7fc4900d0f20 2024-03-23T03:46:37.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.662+0000 7fc496000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc490160c50 con 0x7fc4900b47d0 2024-03-23T03:46:37.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.662+0000 7fc48fe00640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fc4900d0f20 0x7fc4900d1390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56308/0 (socket says 172.21.15.173:56308) 2024-03-23T03:46:37.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.662+0000 7fc48fe00640 1 -- 172.21.15.173:0/3800971279 learned_addr learned my addr 172.21.15.173:0/3800971279 (peer_addr_for_me v1:172.21.15.173:0/0) 2024-03-23T03:46:37.663 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.662+0000 7fc487e00640 1 --2- 172.21.15.173:0/3800971279 >> v2:172.21.15.102:3300/0 conn(0x7fc4900b47d0 0x7fc4900b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.663+0000 7fc48ea00640 1 -- 172.21.15.173:0/3800971279 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3710178831 0 0) 0x7fc4900b5150 con 0x7fc4900d0f20 2024-03-23T03:46:37.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.663+0000 7fc48ea00640 1 -- 172.21.15.173:0/3800971279 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc474003520 con 0x7fc4900d0f20 2024-03-23T03:46:37.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.663+0000 7fc487e00640 1 -- 172.21.15.173:0/3800971279 >> v1:172.21.15.173:6789/0 conn(0x7fc4900d0f20 legacy=0x7fc4900d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.663+0000 7fc48ea00640 1 -- 172.21.15.173:0/3800971279 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 692573118 0 0) 0x7fc474003520 con 0x7fc4900d0f20 2024-03-23T03:46:37.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.663+0000 7fc487e00640 1 -- 172.21.15.173:0/3800971279 >> v1:172.21.15.102:6789/0 conn(0x7fc49015cd10 legacy=0x7fc4900cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.663+0000 7fc487e00640 1 -- 172.21.15.173:0/3800971279 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc490160f30 con 0x7fc4900b47d0 2024-03-23T03:46:37.664 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.664+0000 7fc487e00640 1 --2- 172.21.15.173:0/3800971279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4900b47d0 0x7fc4900b4c10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc478009720 tx=0x7fc47802b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf82bda762daee43 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:37.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.665+0000 7fc48ea00640 1 -- 172.21.15.173:0/3800971279 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4880019f0 con 0x7fc4900b47d0 2024-03-23T03:46:37.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.665+0000 7fc48ea00640 1 -- 172.21.15.173:0/3800971279 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc478031520 con 0x7fc4900b47d0 2024-03-23T03:46:37.665 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.665+0000 7fc48ea00640 1 -- 172.21.15.173:0/3800971279 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4780316b0 con 0x7fc4900b47d0 2024-03-23T03:46:37.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.665+0000 7fc496000640 1 -- 172.21.15.173:0/3800971279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4900b47d0 msgr2=0x7fc4900b4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.665+0000 7fc496000640 1 --2- 172.21.15.173:0/3800971279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4900b47d0 0x7fc4900b4c10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc478009720 tx=0x7fc47802b7c0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.665+0000 7fc496000640 1 -- 172.21.15.173:0/3800971279 shutdown_connections 2024-03-23T03:46:37.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.665+0000 7fc496000640 1 --2- 172.21.15.173:0/3800971279 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4900b47d0 0x7fc4900b4c10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.665+0000 7fc496000640 1 -- 172.21.15.173:0/3800971279 >> 172.21.15.173:0/3800971279 conn(0x7fc4900cf830 msgr2=0x7fc4900cfc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:37.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.666+0000 7fc496000640 1 -- 172.21.15.173:0/3800971279 shutdown_connections 2024-03-23T03:46:37.666 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.666+0000 7fc496000640 1 -- 172.21.15.173:0/3800971279 wait complete. 2024-03-23T03:46:37.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.667+0000 7fc496000640 1 Processor -- start 2024-03-23T03:46:37.667 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.667+0000 7fc496000640 1 -- start start 2024-03-23T03:46:37.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.668+0000 7fc496000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc4900b47d0 0x7fc49015d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.668+0000 7fc496000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4900d0f20 0x7fc490153d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.668+0000 7fc496000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc49015cd10 0x7fc49015a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.668+0000 7fc496000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4901f36f0 con 0x7fc49015cd10 2024-03-23T03:46:37.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.668+0000 7fc496000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc49015de60 con 0x7fc4900d0f20 2024-03-23T03:46:37.670 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.668+0000 7fc496000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc49015e140 con 0x7fc4900b47d0 2024-03-23T03:46:37.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.668+0000 7fc487e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4900d0f20 0x7fc490153d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.668+0000 7fc48f400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc4900b47d0 0x7fc49015d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.669+0000 7fc487e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4900d0f20 0x7fc490153d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:39662/0 (socket says 172.21.15.173:39662) 2024-03-23T03:46:37.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.669+0000 7fc48f400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc4900b47d0 0x7fc49015d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.173:42596/0 (socket says 172.21.15.173:42596) 2024-03-23T03:46:37.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.669+0000 7fc487e00640 1 -- 172.21.15.173:0/3874749706 learned_addr learned my addr 172.21.15.173:0/3874749706 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-23T03:46:37.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.669+0000 7fc48fe00640 1 --2- 172.21.15.173:0/3874749706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc49015cd10 0x7fc49015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.669+0000 7fc48f400640 1 -- 172.21.15.173:0/3874749706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4900d0f20 msgr2=0x7fc490153d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.669+0000 7fc48f400640 1 --2- 172.21.15.173:0/3874749706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4900d0f20 0x7fc490153d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.669+0000 7fc48f400640 1 -- 172.21.15.173:0/3874749706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc49015cd10 msgr2=0x7fc49015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.669+0000 7fc48f400640 1 --2- 172.21.15.173:0/3874749706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc49015cd10 0x7fc49015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.669+0000 7fc48f400640 1 -- 172.21.15.173:0/3874749706 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc478009400 con 0x7fc4900b47d0 2024-03-23T03:46:37.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.669+0000 7fc48f400640 1 --2- 172.21.15.173:0/3874749706 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc4900b47d0 0x7fc49015d610 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc48000c390 tx=0x7fc48000c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:37.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.669+0000 7fc487400640 1 -- 172.21.15.173:0/3874749706 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4800052d0 con 0x7fc4900b47d0 2024-03-23T03:46:37.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.670+0000 7fc487400640 1 -- 172.21.15.173:0/3874749706 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc480005430 con 0x7fc4900b47d0 2024-03-23T03:46:37.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.670+0000 7fc487400640 1 -- 172.21.15.173:0/3874749706 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc480003830 con 0x7fc4900b47d0 2024-03-23T03:46:37.671 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.670+0000 7fc496000640 1 -- 172.21.15.173:0/3874749706 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc49015e420 con 0x7fc4900b47d0 2024-03-23T03:46:37.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.670+0000 7fc496000640 1 -- 172.21.15.173:0/3874749706 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc49015db50 con 0x7fc4900b47d0 2024-03-23T03:46:37.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.670+0000 7fc487400640 1 -- 172.21.15.173:0/3874749706 <== mon.2 v2:172.21.15.173:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc480003c90 con 0x7fc4900b47d0 2024-03-23T03:46:37.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.670+0000 7fc487400640 1 -- 172.21.15.173:0/3874749706 <== mon.2 v2:172.21.15.173:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fc480011e00 con 0x7fc4900b47d0 2024-03-23T03:46:37.672 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.671+0000 7fc496000640 1 -- 172.21.15.173:0/3874749706 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4901f3430 con 0x7fc4900b47d0 2024-03-23T03:46:37.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.673+0000 7fc487400640 1 -- 172.21.15.173:0/3874749706 <== mon.2 v2:172.21.15.173:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fc480017070 con 0x7fc4900b47d0 2024-03-23T03:46:37.732 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:37.731+0000 7faa1321e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-23T03:46:37.751 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.751+0000 7fc496000640 1 -- 172.21.15.173:0/3874749706 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7f78f3b3-7b77-45fc-b705-6637c50bf18e", "id": 5} v 0) v1 -- 0x7fc490066bb0 con 0x7fc4900b47d0 2024-03-23T03:46:37.758 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:37.758+0000 7f7d4922c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-23T03:46:37.760 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.760+0000 7fc487400640 1 -- 172.21.15.173:0/3874749706 <== mon.2 v2:172.21.15.173:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7f78f3b3-7b77-45fc-b705-6637c50bf18e", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc480012420 con 0x7fc4900b47d0 2024-03-23T03:46:37.761 INFO:teuthology.orchestra.run.smithi173.stdout:5 2024-03-23T03:46:37.762 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.761+0000 7fc496000640 1 -- 172.21.15.173:0/3874749706 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc4900b47d0 msgr2=0x7fc49015d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.762 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.761+0000 7fc496000640 1 --2- 172.21.15.173:0/3874749706 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc4900b47d0 0x7fc49015d610 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc48000c390 tx=0x7fc48000c850 comp rx=0 tx=0).stop 2024-03-23T03:46:37.762 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.762+0000 7fc496000640 1 -- 172.21.15.173:0/3874749706 shutdown_connections 2024-03-23T03:46:37.762 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.762+0000 7fc496000640 1 --2- 172.21.15.173:0/3874749706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc49015cd10 0x7fc49015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.762 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.762+0000 7fc496000640 1 --2- 172.21.15.173:0/3874749706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4900d0f20 0x7fc490153d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.763 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.762+0000 7fc496000640 1 --2- 172.21.15.173:0/3874749706 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc4900b47d0 0x7fc49015d610 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.763 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.762+0000 7fc496000640 1 -- 172.21.15.173:0/3874749706 >> 172.21.15.173:0/3874749706 conn(0x7fc4900cf830 msgr2=0x7fc4900cfc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:37.763 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.762+0000 7fc496000640 1 -- 172.21.15.173:0/3874749706 shutdown_connections 2024-03-23T03:46:37.763 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.762+0000 7fc496000640 1 -- 172.21.15.173:0/3874749706 wait complete. 2024-03-23T03:46:37.779 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new 300271a9-e4bd-41cc-a4b5-7ad4e466f6c5 6 2024-03-23T03:46:37.907 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:37.906+0000 7f7d4922c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-23T03:46:37.907 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:37.906+0000 7faa1321e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-23T03:46:37.956 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:37.955+0000 7faa1321e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-23T03:46:37.962 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:37.962+0000 7f7d4922c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-23T03:46:37.968 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.968+0000 7f75c0a00640 1 Processor -- start 2024-03-23T03:46:37.969 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.968+0000 7f75c0a00640 1 -- start start 2024-03-23T03:46:37.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.970+0000 7f75c0a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f75bc15cc30 0x7f75bc0d16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.970 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.970+0000 7f75c0a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f75bc0b62a0 0x7f75bc0cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.971 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.970+0000 7f75c0a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75bc160ae0 con 0x7f75bc0d1be0 2024-03-23T03:46:37.971 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.970+0000 7f75c0a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75bc0d0c00 con 0x7f75bc15cc30 2024-03-23T03:46:37.971 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.970+0000 7f75c0a00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75bc0cdde0 con 0x7f75bc0b62a0 2024-03-23T03:46:37.971 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.970+0000 7f75bb200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f75bc0b62a0 0x7f75bc0cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.971 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.970+0000 7f75bb200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f75bc0b62a0 0x7f75bc0cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:39676/0 (socket says 172.21.15.173:39676) 2024-03-23T03:46:37.971 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.970+0000 7f75b3e00640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f75bc0d1be0 0x7f75bc0b5b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.173:56310/0 (socket says 172.21.15.173:56310) 2024-03-23T03:46:37.971 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.970+0000 7f75bb200640 1 -- 172.21.15.173:0/3544501679 learned_addr learned my addr 172.21.15.173:0/3544501679 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-23T03:46:37.971 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.970+0000 7f75ba800640 1 --2- 172.21.15.173:0/3544501679 >> v2:172.21.15.102:3300/0 conn(0x7f75bc15cc30 0x7f75bc0d16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.971+0000 7f75bb200640 1 -- 172.21.15.173:0/3544501679 >> v2:172.21.15.102:3300/0 conn(0x7f75bc15cc30 msgr2=0x7f75bc0d16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:37.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.971+0000 7f75bb200640 1 --2- 172.21.15.173:0/3544501679 >> v2:172.21.15.102:3300/0 conn(0x7f75bc15cc30 0x7f75bc0d16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.971+0000 7f75bb200640 1 -- 172.21.15.173:0/3544501679 >> v1:172.21.15.173:6789/0 conn(0x7f75bc0d1be0 legacy=0x7f75bc0b5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.971+0000 7f75bb200640 1 -- 172.21.15.173:0/3544501679 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75bc0d0ea0 con 0x7f75bc0b62a0 2024-03-23T03:46:37.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.971+0000 7f75b9e00640 1 -- 172.21.15.173:0/3544501679 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000737077 0 0) 0x7f75bc160ae0 con 0x7f75bc0d1be0 2024-03-23T03:46:37.972 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.972+0000 7f75bb200640 1 --2- 172.21.15.173:0/3544501679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75bc0b62a0 0x7f75bc0cf510 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0d22e0 tx=0x7f75b402bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6912ff3698185d47 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:37.973 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.972+0000 7f75b9e00640 1 -- 172.21.15.173:0/3544501679 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b402cb20 con 0x7f75bc0b62a0 2024-03-23T03:46:37.973 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.972+0000 7f75b9e00640 1 -- 172.21.15.173:0/3544501679 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b402cc80 con 0x7f75bc0b62a0 2024-03-23T03:46:37.973 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.972+0000 7f75b9e00640 1 -- 172.21.15.173:0/3544501679 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b40343f0 con 0x7f75bc0b62a0 2024-03-23T03:46:37.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.973+0000 7f75c0a00640 1 -- 172.21.15.173:0/3544501679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75bc0b62a0 msgr2=0x7f75bc0cf510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.973+0000 7f75c0a00640 1 --2- 172.21.15.173:0/3544501679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75bc0b62a0 0x7f75bc0cf510 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0d22e0 tx=0x7f75b402bf80 comp rx=0 tx=0).stop 2024-03-23T03:46:37.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.973+0000 7f75c0a00640 1 -- 172.21.15.173:0/3544501679 shutdown_connections 2024-03-23T03:46:37.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.973+0000 7f75c0a00640 1 --2- 172.21.15.173:0/3544501679 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75bc0b62a0 0x7f75bc0cf510 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.973+0000 7f75c0a00640 1 --2- 172.21.15.173:0/3544501679 >> v2:172.21.15.102:3300/0 conn(0x7f75bc15cc30 0x7f75bc0d16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.973+0000 7f75c0a00640 1 -- 172.21.15.173:0/3544501679 >> 172.21.15.173:0/3544501679 conn(0x7f75bc06b100 msgr2=0x7f75bc06b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:37.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.973+0000 7f75c0a00640 1 -- 172.21.15.173:0/3544501679 shutdown_connections 2024-03-23T03:46:37.974 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.973+0000 7f75c0a00640 1 -- 172.21.15.173:0/3544501679 wait complete. 2024-03-23T03:46:37.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.975+0000 7f75c0a00640 1 Processor -- start 2024-03-23T03:46:37.975 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.975+0000 7f75c0a00640 1 -- start start 2024-03-23T03:46:37.977 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.976+0000 7f75c0a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75bc0b62a0 0x7f75bc158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.976+0000 7f75c0a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f75bc0d1be0 0x7f75bc0ca460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.976+0000 7f75c0a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75bc15cc30 0x7f75bc0cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.976+0000 7f75c0a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc0b31f0 con 0x7f75bc15cc30 2024-03-23T03:46:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.976+0000 7f75c0a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc0ccd80 con 0x7f75bc0b62a0 2024-03-23T03:46:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.976+0000 7f75c0a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75bc0cd5c0 con 0x7f75bc0d1be0 2024-03-23T03:46:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.976+0000 7f75ba800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75bc0b62a0 0x7f75bc158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.976+0000 7f75b3e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f75bc0d1be0 0x7f75bc0ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.976+0000 7f75ba800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75bc0b62a0 0x7f75bc158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:39686/0 (socket says 172.21.15.173:39686) 2024-03-23T03:46:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.976+0000 7f75ba800640 1 -- 172.21.15.173:0/4015735799 learned_addr learned my addr 172.21.15.173:0/4015735799 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-23T03:46:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.976+0000 7f75bb200640 1 --2- 172.21.15.173:0/4015735799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75bc15cc30 0x7f75bc0cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.976+0000 7f75ba800640 1 -- 172.21.15.173:0/4015735799 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f75bc0d1be0 msgr2=0x7f75bc0ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.976+0000 7f75ba800640 1 --2- 172.21.15.173:0/4015735799 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f75bc0d1be0 0x7f75bc0ca460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.978 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.976+0000 7f75ba800640 1 -- 172.21.15.173:0/4015735799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75bc15cc30 msgr2=0x7f75bc0cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:37.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.976+0000 7f75ba800640 1 --2- 172.21.15.173:0/4015735799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75bc15cc30 0x7f75bc0cc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:37.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.976+0000 7f75ba800640 1 -- 172.21.15.173:0/4015735799 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b4009b30 con 0x7f75bc0b62a0 2024-03-23T03:46:37.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.976+0000 7f75bb200640 1 --2- 172.21.15.173:0/4015735799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75bc15cc30 0x7f75bc0cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:46:37.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.977+0000 7f75ba800640 1 --2- 172.21.15.173:0/4015735799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75bc0b62a0 0x7f75bc158d80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00d5f0 tx=0x7f75ac00dab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:37.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.977+0000 7f75b2a00640 1 -- 172.21.15.173:0/4015735799 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac0125f0 con 0x7f75bc0b62a0 2024-03-23T03:46:37.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.977+0000 7f75b2a00640 1 -- 172.21.15.173:0/4015735799 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ac012750 con 0x7f75bc0b62a0 2024-03-23T03:46:37.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.977+0000 7f75b2a00640 1 -- 172.21.15.173:0/4015735799 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75ac012b00 con 0x7f75bc0b62a0 2024-03-23T03:46:37.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.977+0000 7f75c0a00640 1 -- 172.21.15.173:0/4015735799 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75bc0cd000 con 0x7f75bc0b62a0 2024-03-23T03:46:37.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.977+0000 7f75c0a00640 1 -- 172.21.15.173:0/4015735799 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75bc0b2e80 con 0x7f75bc0b62a0 2024-03-23T03:46:37.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.977+0000 7f75b2a00640 1 -- 172.21.15.173:0/4015735799 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f75ac01e460 con 0x7f75bc0b62a0 2024-03-23T03:46:37.979 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.977+0000 7f75b2a00640 1 -- 172.21.15.173:0/4015735799 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f75ac014ac0 con 0x7f75bc0b62a0 2024-03-23T03:46:37.980 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.978+0000 7f75c0a00640 1 -- 172.21.15.173:0/4015735799 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75bc0734a0 con 0x7f75bc0b62a0 2024-03-23T03:46:37.980 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:37.980+0000 7f75b2a00640 1 -- 172.21.15.173:0/4015735799 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f75ac014d10 con 0x7f75bc0b62a0 2024-03-23T03:46:38.058 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.058+0000 7f75c0a00640 1 -- 172.21.15.173:0/4015735799 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd new", "uuid": "300271a9-e4bd-41cc-a4b5-7ad4e466f6c5", "id": 6} v 0) v1 -- 0x7f75bc0ceca0 con 0x7f75bc0b62a0 2024-03-23T03:46:38.068 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.067+0000 7f75b2a00640 1 -- 172.21.15.173:0/4015735799 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "300271a9-e4bd-41cc-a4b5-7ad4e466f6c5", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f75ac0144d0 con 0x7f75bc0b62a0 2024-03-23T03:46:38.068 INFO:teuthology.orchestra.run.smithi173.stdout:6 2024-03-23T03:46:38.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.069+0000 7f75c0a00640 1 -- 172.21.15.173:0/4015735799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75bc0b62a0 msgr2=0x7f75bc158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:38.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.069+0000 7f75c0a00640 1 --2- 172.21.15.173:0/4015735799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75bc0b62a0 0x7f75bc158d80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f75ac00d5f0 tx=0x7f75ac00dab0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.069 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.069+0000 7f75c0a00640 1 -- 172.21.15.173:0/4015735799 shutdown_connections 2024-03-23T03:46:38.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.069+0000 7f75c0a00640 1 --2- 172.21.15.173:0/4015735799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75bc15cc30 0x7f75bc0cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.069+0000 7f75c0a00640 1 --2- 172.21.15.173:0/4015735799 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f75bc0d1be0 0x7f75bc0ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.069+0000 7f75c0a00640 1 --2- 172.21.15.173:0/4015735799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f75bc0b62a0 0x7f75bc158d80 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.069+0000 7f75c0a00640 1 -- 172.21.15.173:0/4015735799 >> 172.21.15.173:0/4015735799 conn(0x7f75bc06b100 msgr2=0x7f75bc0c6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:38.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.069+0000 7f75c0a00640 1 -- 172.21.15.173:0/4015735799 shutdown_connections 2024-03-23T03:46:38.070 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.069+0000 7f75c0a00640 1 -- 172.21.15.173:0/4015735799 wait complete. 2024-03-23T03:46:38.085 DEBUG:teuthology.orchestra.run.smithi173:> sudo ceph --cluster ceph osd new d242af13-3439-4b6b-8c84-08a5a27e639c 7 2024-03-23T03:46:38.097 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:38.097+0000 7f7d4922c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-23T03:46:38.128 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:38.127+0000 7faa1321e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-23T03:46:38.214 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:38.214+0000 7f7d4922c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-23T03:46:38.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.272+0000 7f9267000640 1 Processor -- start 2024-03-23T03:46:38.273 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.273+0000 7f9267000640 1 -- start start 2024-03-23T03:46:38.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.274+0000 7f9267000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f92600b84b0 0x7f92600d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:38.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.274+0000 7f9267000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92600b5aa0 0x7f92600b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:38.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.274+0000 7f9267000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92600d1280 con 0x7f92600cd740 2024-03-23T03:46:38.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.274+0000 7f9267000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92600d20c0 con 0x7f92600b84b0 2024-03-23T03:46:38.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.274+0000 7f9267000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92600b8940 con 0x7f92600b5aa0 2024-03-23T03:46:38.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.274+0000 7f9265000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92600b5aa0 0x7f92600b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:38.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.274+0000 7f9265000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92600b5aa0 0x7f92600b5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.173:42622/0 (socket says 172.21.15.173:42622) 2024-03-23T03:46:38.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.274+0000 7f9265000640 1 -- 172.21.15.173:0/1181644622 learned_addr learned my addr 172.21.15.173:0/1181644622 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-23T03:46:38.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.274+0000 7f9257e00640 1 --2- 172.21.15.173:0/1181644622 >> v2:172.21.15.102:3300/0 conn(0x7f92600b84b0 0x7f92600d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:38.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.275+0000 7f9265000640 1 -- 172.21.15.173:0/1181644622 >> v2:172.21.15.102:3300/0 conn(0x7f92600b84b0 msgr2=0x7f92600d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:38.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.275+0000 7f9265000640 1 --2- 172.21.15.173:0/1181644622 >> v2:172.21.15.102:3300/0 conn(0x7f92600b84b0 0x7f92600d0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.275+0000 7f9265000640 1 -- 172.21.15.173:0/1181644622 >> v1:172.21.15.102:6789/0 conn(0x7f92600cd740 legacy=0x7f92600cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:38.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.275+0000 7f9265000640 1 -- 172.21.15.173:0/1181644622 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92600d14b0 con 0x7f92600b5aa0 2024-03-23T03:46:38.276 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.276+0000 7f9265000640 1 --2- 172.21.15.173:0/1181644622 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92600b5aa0 0x7f92600b5f30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9258009e50 tx=0x7f925802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39b0c6f822eb6bcf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:38.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.276+0000 7f925f400640 1 -- 172.21.15.173:0/1181644622 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925802cb00 con 0x7f92600b5aa0 2024-03-23T03:46:38.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.276+0000 7f925f400640 1 -- 172.21.15.173:0/1181644622 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925802cc60 con 0x7f92600b5aa0 2024-03-23T03:46:38.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.276+0000 7f925f400640 1 -- 172.21.15.173:0/1181644622 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92580343f0 con 0x7f92600b5aa0 2024-03-23T03:46:38.277 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.277+0000 7f9267000640 1 -- 172.21.15.173:0/1181644622 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92600b5aa0 msgr2=0x7f92600b5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:38.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.277+0000 7f9267000640 1 --2- 172.21.15.173:0/1181644622 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92600b5aa0 0x7f92600b5f30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9258009e50 tx=0x7f925802bf60 comp rx=0 tx=0).stop 2024-03-23T03:46:38.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.277+0000 7f9267000640 1 -- 172.21.15.173:0/1181644622 shutdown_connections 2024-03-23T03:46:38.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.277+0000 7f9267000640 1 --2- 172.21.15.173:0/1181644622 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92600b5aa0 0x7f92600b5f30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.277+0000 7f9267000640 1 --2- 172.21.15.173:0/1181644622 >> v2:172.21.15.102:3300/0 conn(0x7f92600b84b0 0x7f92600d0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.277+0000 7f9267000640 1 -- 172.21.15.173:0/1181644622 >> 172.21.15.173:0/1181644622 conn(0x7f92600cf5d0 msgr2=0x7f92600cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:38.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.277+0000 7f9267000640 1 -- 172.21.15.173:0/1181644622 shutdown_connections 2024-03-23T03:46:38.278 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.277+0000 7f9267000640 1 -- 172.21.15.173:0/1181644622 wait complete. 2024-03-23T03:46:38.279 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.279+0000 7f9267000640 1 Processor -- start 2024-03-23T03:46:38.279 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.279+0000 7f9267000640 1 -- start start 2024-03-23T03:46:38.282 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.280+0000 7f9267000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92600b5aa0 0x7f9260156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:38.282 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.280+0000 7f9267000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92600b84b0 0x7f92601f2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:38.282 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.280+0000 7f9267000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92600cd740 0x7f9260153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:38.282 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.280+0000 7f9267000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92600b38a0 con 0x7f92600cd740 2024-03-23T03:46:38.282 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.280+0000 7f9267000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92601f3140 con 0x7f92600b5aa0 2024-03-23T03:46:38.282 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.280+0000 7f9267000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92601f2b80 con 0x7f92600b84b0 2024-03-23T03:46:38.282 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.280+0000 7f9257e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92600b84b0 0x7f92601f2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:38.282 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.280+0000 7f9257e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92600b84b0 0x7f92601f2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.173:42628/0 (socket says 172.21.15.173:42628) 2024-03-23T03:46:38.282 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.280+0000 7f9257e00640 1 -- 172.21.15.173:0/2130858734 learned_addr learned my addr 172.21.15.173:0/2130858734 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-23T03:46:38.282 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.280+0000 7f925fe00640 1 --2- 172.21.15.173:0/2130858734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92600b5aa0 0x7f9260156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:38.282 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.280+0000 7f9265000640 1 --2- 172.21.15.173:0/2130858734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92600cd740 0x7f9260153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:38.282 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.281+0000 7f9257e00640 1 -- 172.21.15.173:0/2130858734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92600b5aa0 msgr2=0x7f9260156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:38.282 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.281+0000 7f9257e00640 1 --2- 172.21.15.173:0/2130858734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92600b5aa0 0x7f9260156410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.281+0000 7f9257e00640 1 -- 172.21.15.173:0/2130858734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92600cd740 msgr2=0x7f9260153ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:38.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.281+0000 7f9257e00640 1 --2- 172.21.15.173:0/2130858734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92600cd740 0x7f9260153ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.281+0000 7f9257e00640 1 -- 172.21.15.173:0/2130858734 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9258009b30 con 0x7f92600b84b0 2024-03-23T03:46:38.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.281+0000 7f9257e00640 1 --2- 172.21.15.173:0/2130858734 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92600b84b0 0x7f92601f2100 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f924800e9b0 tx=0x7f924800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:38.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.281+0000 7f925cc00640 1 -- 172.21.15.173:0/2130858734 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9248009670 con 0x7f92600b84b0 2024-03-23T03:46:38.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.281+0000 7f925cc00640 1 -- 172.21.15.173:0/2130858734 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924800c9a0 con 0x7f92600b84b0 2024-03-23T03:46:38.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.281+0000 7f925cc00640 1 -- 172.21.15.173:0/2130858734 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924800cd50 con 0x7f92600b84b0 2024-03-23T03:46:38.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.281+0000 7f9267000640 1 -- 172.21.15.173:0/2130858734 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92601f2e60 con 0x7f92600b84b0 2024-03-23T03:46:38.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.281+0000 7f9267000640 1 -- 172.21.15.173:0/2130858734 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9260156950 con 0x7f92600b84b0 2024-03-23T03:46:38.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.282+0000 7f925cc00640 1 -- 172.21.15.173:0/2130858734 <== mon.2 v2:172.21.15.173:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9248005c90 con 0x7f92600b84b0 2024-03-23T03:46:38.283 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.282+0000 7f925cc00640 1 -- 172.21.15.173:0/2130858734 <== mon.2 v2:172.21.15.173:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f9248013470 con 0x7f92600b84b0 2024-03-23T03:46:38.284 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.282+0000 7f9267000640 1 -- 172.21.15.173:0/2130858734 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92600b3320 con 0x7f92600b84b0 2024-03-23T03:46:38.285 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.285+0000 7f925cc00640 1 -- 172.21.15.173:0/2130858734 <== mon.2 v2:172.21.15.173:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7f92600b3320 con 0x7f92600b84b0 2024-03-23T03:46:38.362 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.362+0000 7f9267000640 1 -- 172.21.15.173:0/2130858734 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d242af13-3439-4b6b-8c84-08a5a27e639c", "id": 7} v 0) v1 -- 0x7f92600b39e0 con 0x7f92600b84b0 2024-03-23T03:46:38.372 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.372+0000 7f925cc00640 1 -- 172.21.15.173:0/2130858734 <== mon.2 v2:172.21.15.173:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d242af13-3439-4b6b-8c84-08a5a27e639c", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f92600b39e0 con 0x7f92600b84b0 2024-03-23T03:46:38.372 INFO:teuthology.orchestra.run.smithi173.stdout:7 2024-03-23T03:46:38.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.373+0000 7f9267000640 1 -- 172.21.15.173:0/2130858734 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92600b84b0 msgr2=0x7f92601f2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:38.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.373+0000 7f9267000640 1 --2- 172.21.15.173:0/2130858734 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92600b84b0 0x7f92601f2100 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f924800e9b0 tx=0x7f924800ee70 comp rx=0 tx=0).stop 2024-03-23T03:46:38.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.373+0000 7f9267000640 1 -- 172.21.15.173:0/2130858734 shutdown_connections 2024-03-23T03:46:38.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.373+0000 7f9267000640 1 --2- 172.21.15.173:0/2130858734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92600cd740 0x7f9260153ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.374+0000 7f9267000640 1 --2- 172.21.15.173:0/2130858734 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92600b84b0 0x7f92601f2100 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.374+0000 7f9267000640 1 --2- 172.21.15.173:0/2130858734 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92600b5aa0 0x7f9260156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.374+0000 7f9267000640 1 -- 172.21.15.173:0/2130858734 >> 172.21.15.173:0/2130858734 conn(0x7f92600cf5d0 msgr2=0x7f92600b1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:38.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.374+0000 7f9267000640 1 -- 172.21.15.173:0/2130858734 shutdown_connections 2024-03-23T03:46:38.374 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-23T03:46:38.374+0000 7f9267000640 1 -- 172.21.15.173:0/2130858734 wait complete. 2024-03-23T03:46:38.389 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-23T03:46:38.389 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-23T03:46:38.396 INFO:tasks.ceph.osd.0:Started 2024-03-23T03:46:38.396 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-23T03:46:38.396 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-23T03:46:38.401 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:38.400+0000 7f7d4922c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-23T03:46:38.403 INFO:tasks.ceph.osd.1:Started 2024-03-23T03:46:38.403 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-23T03:46:38.403 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-23T03:46:38.410 INFO:tasks.ceph.osd.2:Started 2024-03-23T03:46:38.410 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-23T03:46:38.410 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-23T03:46:38.415 INFO:tasks.ceph.osd.3:Started 2024-03-23T03:46:38.415 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-23T03:46:38.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-23T03:46:38.422 INFO:tasks.ceph.osd.4:Started 2024-03-23T03:46:38.422 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-23T03:46:38.422 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-23T03:46:38.429 INFO:tasks.ceph.osd.5:Started 2024-03-23T03:46:38.429 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-23T03:46:38.429 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-23T03:46:38.434 INFO:tasks.ceph.osd.6:Started 2024-03-23T03:46:38.434 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-23T03:46:38.435 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-23T03:46:38.439 INFO:tasks.ceph.osd.7:Started 2024-03-23T03:46:38.440 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:46:38.457 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:38.456+0000 7f7d4922c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-23T03:46:38.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.644+0000 7fbb6c800640 1 Processor -- start 2024-03-23T03:46:38.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.644+0000 7fbb6c800640 1 -- start start 2024-03-23T03:46:38.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.645+0000 7fbb6c800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fbb6415cda0 0x7fbb640cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:38.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.645+0000 7fbb6c800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb640b5a30 0x7fbb640b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:38.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.645+0000 7fbb6c800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb640d1830 con 0x7fbb640b4350 2024-03-23T03:46:38.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.645+0000 7fbb6c800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb640b1f20 con 0x7fbb6415cda0 2024-03-23T03:46:38.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.645+0000 7fbb6c800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640d0c20 con 0x7fbb640b5a30 2024-03-23T03:46:38.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.645+0000 7fbb69e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fbb6415cda0 0x7fbb640cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:38.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.645+0000 7fbb69400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb640b5a30 0x7fbb640b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:38.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.646+0000 7fbb69e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fbb6415cda0 0x7fbb640cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:37826/0 (socket says 172.21.15.102:37826) 2024-03-23T03:46:38.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.646+0000 7fbb69e00640 1 -- 172.21.15.102:0/2268212442 learned_addr learned my addr 172.21.15.102:0/2268212442 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:38.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.646+0000 7fbb69e00640 1 -- 172.21.15.102:0/2268212442 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb640b5a30 msgr2=0x7fbb640b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:38.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.646+0000 7fbb69e00640 1 --2- 172.21.15.102:0/2268212442 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb640b5a30 0x7fbb640b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.646+0000 7fbb69e00640 1 -- 172.21.15.102:0/2268212442 >> v1:172.21.15.173:6789/0 conn(0x7fbb640b4350 legacy=0x7fbb640d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:38.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.646+0000 7fbb69e00640 1 -- 172.21.15.102:0/2268212442 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb640d0f00 con 0x7fbb6415cda0 2024-03-23T03:46:38.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.647+0000 7fbb69e00640 1 --2- 172.21.15.102:0/2268212442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb6415cda0 0x7fbb640cdb40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbb58009870 tx=0x7fbb5802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77b61104c5587f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:38.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.647+0000 7fbb68a00640 1 -- 172.21.15.102:0/2268212442 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb58030840 con 0x7fbb6415cda0 2024-03-23T03:46:38.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.647+0000 7fbb68a00640 1 -- 172.21.15.102:0/2268212442 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb580309a0 con 0x7fbb6415cda0 2024-03-23T03:46:38.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.647+0000 7fbb68a00640 1 -- 172.21.15.102:0/2268212442 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb58030d30 con 0x7fbb6415cda0 2024-03-23T03:46:38.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.648+0000 7fbb6c800640 1 -- 172.21.15.102:0/2268212442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb6415cda0 msgr2=0x7fbb640cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:38.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.648+0000 7fbb6c800640 1 --2- 172.21.15.102:0/2268212442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb6415cda0 0x7fbb640cdb40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbb58009870 tx=0x7fbb5802b980 comp rx=0 tx=0).stop 2024-03-23T03:46:38.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.648+0000 7fbb6c800640 1 -- 172.21.15.102:0/2268212442 shutdown_connections 2024-03-23T03:46:38.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.648+0000 7fbb6c800640 1 --2- 172.21.15.102:0/2268212442 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb640b5a30 0x7fbb640b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.648+0000 7fbb6c800640 1 --2- 172.21.15.102:0/2268212442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb6415cda0 0x7fbb640cdb40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.648+0000 7fbb6c800640 1 -- 172.21.15.102:0/2268212442 >> 172.21.15.102:0/2268212442 conn(0x7fbb640cf350 msgr2=0x7fbb640cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:38.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.648+0000 7fbb6c800640 1 -- 172.21.15.102:0/2268212442 shutdown_connections 2024-03-23T03:46:38.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.648+0000 7fbb6c800640 1 -- 172.21.15.102:0/2268212442 wait complete. 2024-03-23T03:46:38.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.650+0000 7fbb6c800640 1 Processor -- start 2024-03-23T03:46:38.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.650+0000 7fbb6c800640 1 -- start start 2024-03-23T03:46:38.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.651+0000 7fbb6c800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb640b4350 0x7fbb640ca870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:38.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.651+0000 7fbb6c800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb640b5a30 0x7fbb64156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:38.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.651+0000 7fbb6c800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb6415cda0 0x7fbb6415e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:38.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.651+0000 7fbb6c800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640cb0c0 con 0x7fbb640b5a30 2024-03-23T03:46:38.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.651+0000 7fbb6c800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640d3180 con 0x7fbb6415cda0 2024-03-23T03:46:38.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.651+0000 7fbb6c800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640d3610 con 0x7fbb640b4350 2024-03-23T03:46:38.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.651+0000 7fbb69e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb640b4350 0x7fbb640ca870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:38.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.651+0000 7fbb69400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb640b5a30 0x7fbb64156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:38.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.651+0000 7fbb69e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb640b4350 0x7fbb640ca870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:33024/0 (socket says 172.21.15.102:33024) 2024-03-23T03:46:38.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.651+0000 7fbb69e00640 1 -- 172.21.15.102:0/1669404732 learned_addr learned my addr 172.21.15.102:0/1669404732 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:38.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.651+0000 7fbb6a800640 1 --2- 172.21.15.102:0/1669404732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb6415cda0 0x7fbb6415e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:38.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.651+0000 7fbb69e00640 1 -- 172.21.15.102:0/1669404732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb6415cda0 msgr2=0x7fbb6415e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:38.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.651+0000 7fbb69e00640 1 --2- 172.21.15.102:0/1669404732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb6415cda0 0x7fbb6415e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.651+0000 7fbb69e00640 1 -- 172.21.15.102:0/1669404732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb640b5a30 msgr2=0x7fbb64156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:38.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.651+0000 7fbb69e00640 1 --2- 172.21.15.102:0/1669404732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb640b5a30 0x7fbb64156d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.651+0000 7fbb69e00640 1 -- 172.21.15.102:0/1669404732 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb580094d0 con 0x7fbb640b4350 2024-03-23T03:46:38.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.652+0000 7fbb69e00640 1 --2- 172.21.15.102:0/1669404732 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb640b4350 0x7fbb640ca870 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbb58000c00 tx=0x7fbb58004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:38.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.652+0000 7fbb5e000640 1 -- 172.21.15.102:0/1669404732 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb58039a90 con 0x7fbb640b4350 2024-03-23T03:46:38.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.652+0000 7fbb5e000640 1 -- 172.21.15.102:0/1669404732 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb58039c10 con 0x7fbb640b4350 2024-03-23T03:46:38.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.652+0000 7fbb5e000640 1 -- 172.21.15.102:0/1669404732 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb58043410 con 0x7fbb640b4350 2024-03-23T03:46:38.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.652+0000 7fbb6c800640 1 -- 172.21.15.102:0/1669404732 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb640d2850 con 0x7fbb640b4350 2024-03-23T03:46:38.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.652+0000 7fbb6c800640 1 -- 172.21.15.102:0/1669404732 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb640d2e10 con 0x7fbb640b4350 2024-03-23T03:46:38.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.652+0000 7fbb5e000640 1 -- 172.21.15.102:0/1669404732 <== mon.2 v2:172.21.15.173:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fbb58043870 con 0x7fbb640b4350 2024-03-23T03:46:38.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.653+0000 7fbb5e000640 1 -- 172.21.15.102:0/1669404732 <== mon.2 v2:172.21.15.173:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fbb5804c3f0 con 0x7fbb640b4350 2024-03-23T03:46:38.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.653+0000 7fbb6c800640 1 -- 172.21.15.102:0/1669404732 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb64073390 con 0x7fbb640b4350 2024-03-23T03:46:38.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.656+0000 7fbb5e000640 1 -- 172.21.15.102:0/1669404732 <== mon.2 v2:172.21.15.173:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fbb58048070 con 0x7fbb640b4350 2024-03-23T03:46:38.670 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:38.670+0000 7f7d4922c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-23T03:46:38.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.719+0000 7fbb6c800640 1 -- 172.21.15.102:0/1669404732 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb640cadb0 con 0x7fbb640b4350 2024-03-23T03:46:38.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.721+0000 7fbb5e000640 1 -- 172.21.15.102:0/1669404732 <== mon.2 v2:172.21.15.173:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fbb58034070 con 0x7fbb640b4350 2024-03-23T03:46:38.721 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:46:38.721 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":10,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:38.365403+0000","last_up_change":"0.000000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:46:38.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.722+0000 7fbb6c800640 1 -- 172.21.15.102:0/1669404732 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb640b4350 msgr2=0x7fbb640ca870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:38.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.722+0000 7fbb6c800640 1 --2- 172.21.15.102:0/1669404732 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb640b4350 0x7fbb640ca870 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbb58000c00 tx=0x7fbb58004730 comp rx=0 tx=0).stop 2024-03-23T03:46:38.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.722+0000 7fbb6c800640 1 -- 172.21.15.102:0/1669404732 shutdown_connections 2024-03-23T03:46:38.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.722+0000 7fbb6c800640 1 --2- 172.21.15.102:0/1669404732 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb6415cda0 0x7fbb6415e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.722+0000 7fbb6c800640 1 --2- 172.21.15.102:0/1669404732 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb640b5a30 0x7fbb64156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.722+0000 7fbb6c800640 1 --2- 172.21.15.102:0/1669404732 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb640b4350 0x7fbb640ca870 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:38.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.722+0000 7fbb6c800640 1 -- 172.21.15.102:0/1669404732 >> 172.21.15.102:0/1669404732 conn(0x7fbb640cf350 msgr2=0x7fbb640cc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:38.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.722+0000 7fbb6c800640 1 -- 172.21.15.102:0/1669404732 shutdown_connections 2024-03-23T03:46:38.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:38.722+0000 7fbb6c800640 1 -- 172.21.15.102:0/1669404732 wait complete. 2024-03-23T03:46:38.737 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-23T03:46:38.738 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-23T03:46:39.108 INFO:tasks.ceph.osd.5.smithi173.stderr:2024-03-23T03:46:39.108+0000 7fbd14c164c0 -1 Falling back to public interface 2024-03-23T03:46:39.110 INFO:tasks.ceph.osd.2.smithi102.stderr:2024-03-23T03:46:39.109+0000 7f8079e0d4c0 -1 Falling back to public interface 2024-03-23T03:46:39.113 INFO:tasks.ceph.osd.6.smithi173.stderr:2024-03-23T03:46:39.113+0000 7f46d940d4c0 -1 Falling back to public interface 2024-03-23T03:46:39.115 INFO:tasks.ceph.osd.3.smithi102.stderr:2024-03-23T03:46:39.114+0000 7fe72740c4c0 -1 Falling back to public interface 2024-03-23T03:46:39.128 INFO:tasks.ceph.osd.0.smithi102.stderr:2024-03-23T03:46:39.127+0000 7ffac6b2b4c0 -1 Falling back to public interface 2024-03-23T03:46:39.141 INFO:tasks.ceph.osd.1.smithi102.stderr:2024-03-23T03:46:39.141+0000 7f33fb5294c0 -1 Falling back to public interface 2024-03-23T03:46:39.148 INFO:tasks.ceph.osd.4.smithi173.stderr:2024-03-23T03:46:39.148+0000 7f7fcaa274c0 -1 Falling back to public interface 2024-03-23T03:46:39.149 INFO:tasks.ceph.osd.7.smithi173.stderr:2024-03-23T03:46:39.149+0000 7f45384674c0 -1 Falling back to public interface 2024-03-23T03:46:39.197 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:39.196+0000 7faa1321e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-23T03:46:39.499 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:39.498+0000 7faa1321e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-23T03:46:39.544 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-23T03:46:39.558 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:39.557+0000 7faa1321e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-23T03:46:39.695 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:39.694+0000 7faa1321e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-23T03:46:39.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.709+0000 7fb08ea00640 1 Processor -- start 2024-03-23T03:46:39.710 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.709+0000 7fb08ea00640 1 -- start start 2024-03-23T03:46:39.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.710+0000 7fb08ea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb0880cd8c0 0x7fb0880b83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:39.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.711+0000 7fb08ea00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb08815d210 con 0x7fb08815ce30 2024-03-23T03:46:39.711 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.711+0000 7fb08ea00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0880b88e0 con 0x7fb0880b4af0 2024-03-23T03:46:39.712 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.711+0000 7fb08ea00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0880b2180 con 0x7fb0880cd8c0 2024-03-23T03:46:39.712 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.711+0000 7fb087e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb08815ce30 0x7fb0880b43e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:59500/0 (socket says 172.21.15.102:59500) 2024-03-23T03:46:39.712 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.711+0000 7fb087e00640 1 -- 172.21.15.102:0/1218062463 learned_addr learned my addr 172.21.15.102:0/1218062463 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:46:39.712 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.711+0000 7fb08ca00640 1 --2- 172.21.15.102:0/1218062463 >> v2:172.21.15.102:3300/0 conn(0x7fb0880cd8c0 0x7fb0880b83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:39.712 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.711+0000 7fb086a00640 1 -- 172.21.15.102:0/1218062463 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658278238 0 0) 0x7fb078001110 con 0x7fb08815ce30 2024-03-23T03:46:39.713 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.711+0000 7fb086a00640 1 -- 172.21.15.102:0/1218062463 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb06c003520 con 0x7fb08815ce30 2024-03-23T03:46:39.713 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.711+0000 7fb08ca00640 1 -- 172.21.15.102:0/1218062463 >> v1:172.21.15.173:6789/0 conn(0x7fb0880b4af0 legacy=0x7fb0880b5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:39.713 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.711+0000 7fb08ca00640 1 -- 172.21.15.102:0/1218062463 >> v1:172.21.15.102:6789/0 conn(0x7fb08815ce30 legacy=0x7fb0880b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:39.713 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.711+0000 7fb08ca00640 1 -- 172.21.15.102:0/1218062463 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0880cddf0 con 0x7fb0880cd8c0 2024-03-23T03:46:39.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.713+0000 7fb08ca00640 1 --2- 172.21.15.102:0/1218062463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0880cd8c0 0x7fb0880b83a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb080009d90 tx=0x7fb08002bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec3dd41fed6f75b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:39.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.713+0000 7fb086a00640 1 -- 172.21.15.102:0/1218062463 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08002cd80 con 0x7fb0880cd8c0 2024-03-23T03:46:39.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.713+0000 7fb086a00640 1 -- 172.21.15.102:0/1218062463 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb080034440 con 0x7fb0880cd8c0 2024-03-23T03:46:39.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.713+0000 7fb086a00640 1 -- 172.21.15.102:0/1218062463 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0800347b0 con 0x7fb0880cd8c0 2024-03-23T03:46:39.714 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.714+0000 7fb08ea00640 1 -- 172.21.15.102:0/1218062463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0880cd8c0 msgr2=0x7fb0880b83a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:39.715 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.714+0000 7fb08ea00640 1 --2- 172.21.15.102:0/1218062463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0880cd8c0 0x7fb0880b83a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb080009d90 tx=0x7fb08002bfd0 comp rx=0 tx=0).stop 2024-03-23T03:46:39.715 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.714+0000 7fb08ea00640 1 -- 172.21.15.102:0/1218062463 shutdown_connections 2024-03-23T03:46:39.715 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.714+0000 7fb08ea00640 1 --2- 172.21.15.102:0/1218062463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0880cd8c0 0x7fb0880b83a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:39.715 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.714+0000 7fb08ea00640 1 -- 172.21.15.102:0/1218062463 >> 172.21.15.102:0/1218062463 conn(0x7fb0880cf3e0 msgr2=0x7fb0880cf7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:39.715 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.715+0000 7fb08ea00640 1 -- 172.21.15.102:0/1218062463 shutdown_connections 2024-03-23T03:46:39.716 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.715+0000 7fb08ea00640 1 -- 172.21.15.102:0/1218062463 wait complete. 2024-03-23T03:46:39.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.716+0000 7fb08ea00640 1 Processor -- start 2024-03-23T03:46:39.717 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.717+0000 7fb08ea00640 1 -- start start 2024-03-23T03:46:39.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.717+0000 7fb08ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0880b4af0 0x7fb0880c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:39.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.718+0000 7fb08ea00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0880cd8c0 0x7fb0881563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:39.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.718+0000 7fb08ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb08815ce30 0x7fb0880cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:39.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.718+0000 7fb08ea00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0880ca010 con 0x7fb08815ce30 2024-03-23T03:46:39.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.718+0000 7fb08ea00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0880ca2f0 con 0x7fb0880b4af0 2024-03-23T03:46:39.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.718+0000 7fb08ea00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0880ca5d0 con 0x7fb0880cd8c0 2024-03-23T03:46:39.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.718+0000 7fb08ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb08815ce30 0x7fb0880cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:39.719 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.718+0000 7fb087e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0880b4af0 0x7fb0880c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:39.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.718+0000 7fb08ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb08815ce30 0x7fb0880cb0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33192/0 (socket says 172.21.15.102:33192) 2024-03-23T03:46:39.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.718+0000 7fb08ca00640 1 -- 172.21.15.102:0/2319927052 learned_addr learned my addr 172.21.15.102:0/2319927052 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:39.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.718+0000 7fb087400640 1 --2- 172.21.15.102:0/2319927052 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0880cd8c0 0x7fb0881563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:39.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.718+0000 7fb08ca00640 1 -- 172.21.15.102:0/2319927052 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0880cd8c0 msgr2=0x7fb0881563f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:39.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.718+0000 7fb08ca00640 1 --2- 172.21.15.102:0/2319927052 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0880cd8c0 0x7fb0881563f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:39.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.718+0000 7fb08ca00640 1 -- 172.21.15.102:0/2319927052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0880b4af0 msgr2=0x7fb0880c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:39.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.718+0000 7fb08ca00640 1 --2- 172.21.15.102:0/2319927052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0880b4af0 0x7fb0880c9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:39.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.718+0000 7fb08ca00640 1 -- 172.21.15.102:0/2319927052 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb080009a70 con 0x7fb08815ce30 2024-03-23T03:46:39.720 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.718+0000 7fb087400640 1 --2- 172.21.15.102:0/2319927052 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0880cd8c0 0x7fb0881563f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:46:39.721 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.719+0000 7fb08ca00640 1 --2- 172.21.15.102:0/2319927052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb08815ce30 0x7fb0880cb0b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb080037040 tx=0x7fb080004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:39.721 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.719+0000 7fb07fe00640 1 -- 172.21.15.102:0/2319927052 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080004250 con 0x7fb08815ce30 2024-03-23T03:46:39.721 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.719+0000 7fb07fe00640 1 -- 172.21.15.102:0/2319927052 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0800043d0 con 0x7fb08815ce30 2024-03-23T03:46:39.721 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.719+0000 7fb07fe00640 1 -- 172.21.15.102:0/2319927052 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080004780 con 0x7fb08815ce30 2024-03-23T03:46:39.721 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.719+0000 7fb08ea00640 1 -- 172.21.15.102:0/2319927052 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0880ca8b0 con 0x7fb08815ce30 2024-03-23T03:46:39.721 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.720+0000 7fb08ea00640 1 -- 172.21.15.102:0/2319927052 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0881570e0 con 0x7fb08815ce30 2024-03-23T03:46:39.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.721+0000 7fb07fe00640 1 -- 172.21.15.102:0/2319927052 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 2) v1 ==== 44520+0+0 (secure 0 0 0) 0x7fb0800345b0 con 0x7fb08815ce30 2024-03-23T03:46:39.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.721+0000 7fb08ea00640 1 -- 172.21.15.102:0/2319927052 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0880670d0 con 0x7fb08815ce30 2024-03-23T03:46:39.722 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.721+0000 7fb07fe00640 1 -- 172.21.15.102:0/2319927052 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fb08006bdc0 con 0x7fb08815ce30 2024-03-23T03:46:39.725 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.724+0000 7fb07fe00640 1 -- 172.21.15.102:0/2319927052 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72508 (secure 0 0 0) 0x7fb0800487c0 con 0x7fb08815ce30 2024-03-23T03:46:39.748 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:39.748+0000 7f7d4922c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-23T03:46:39.793 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.792+0000 7fb08ea00640 1 -- 172.21.15.102:0/2319927052 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0880cadd0 con 0x7fb08815ce30 2024-03-23T03:46:39.793 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.793+0000 7fb07fe00640 1 -- 172.21.15.102:0/2319927052 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fb0880cadd0 con 0x7fb08815ce30 2024-03-23T03:46:39.794 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-23T03:46:39.794 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":10,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:38.365403+0000","last_up_change":"0.000000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:46:39.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.794+0000 7fb08ea00640 1 -- 172.21.15.102:0/2319927052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb08815ce30 msgr2=0x7fb0880cb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:39.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.794+0000 7fb08ea00640 1 --2- 172.21.15.102:0/2319927052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb08815ce30 0x7fb0880cb0b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb080037040 tx=0x7fb080004a40 comp rx=0 tx=0).stop 2024-03-23T03:46:39.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.794+0000 7fb08ea00640 1 -- 172.21.15.102:0/2319927052 shutdown_connections 2024-03-23T03:46:39.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.794+0000 7fb08ea00640 1 --2- 172.21.15.102:0/2319927052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb08815ce30 0x7fb0880cb0b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:39.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.794+0000 7fb08ea00640 1 --2- 172.21.15.102:0/2319927052 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0880cd8c0 0x7fb0881563f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:39.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.794+0000 7fb08ea00640 1 --2- 172.21.15.102:0/2319927052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0880b4af0 0x7fb0880c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:39.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.794+0000 7fb08ea00640 1 -- 172.21.15.102:0/2319927052 >> 172.21.15.102:0/2319927052 conn(0x7fb0880cf3e0 msgr2=0x7fb08815f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:39.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.794+0000 7fb08ea00640 1 -- 172.21.15.102:0/2319927052 shutdown_connections 2024-03-23T03:46:39.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:39.794+0000 7fb08ea00640 1 -- 172.21.15.102:0/2319927052 wait complete. 2024-03-23T03:46:39.806 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-23T03:46:40.059 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:40.059+0000 7f7d4922c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-23T03:46:40.118 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:40.117+0000 7f7d4922c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-23T03:46:40.247 INFO:tasks.ceph.mgr.x.smithi173.stderr:2024-03-23T03:46:40.247+0000 7f7d4922c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-23T03:46:40.736 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:40.735+0000 7fa9b3a00640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-23T03:46:40.736 INFO:tasks.ceph.mgr.y.smithi102.stderr:2024-03-23T03:46:40.736+0000 7fa9b3a00640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-23T03:46:42.632 INFO:tasks.ceph.osd.5.smithi173.stderr:2024-03-23T03:46:42.631+0000 7fbd14c164c0 -1 osd.5 0 log_to_monitors true 2024-03-23T03:46:42.695 INFO:tasks.ceph.osd.0.smithi102.stderr:2024-03-23T03:46:42.694+0000 7ffac6b2b4c0 -1 osd.0 0 log_to_monitors true 2024-03-23T03:46:42.698 INFO:tasks.ceph.osd.1.smithi102.stderr:2024-03-23T03:46:42.698+0000 7f33fb5294c0 -1 osd.1 0 log_to_monitors true 2024-03-23T03:46:42.700 INFO:tasks.ceph.osd.3.smithi102.stderr:2024-03-23T03:46:42.700+0000 7fe72740c4c0 -1 osd.3 0 log_to_monitors true 2024-03-23T03:46:42.723 INFO:tasks.ceph.osd.7.smithi173.stderr:2024-03-23T03:46:42.724+0000 7f45384674c0 -1 osd.7 0 log_to_monitors true 2024-03-23T03:46:42.891 INFO:tasks.ceph.osd.2.smithi102.stderr:2024-03-23T03:46:42.890+0000 7f8079e0d4c0 -1 osd.2 0 log_to_monitors true 2024-03-23T03:46:42.943 INFO:tasks.ceph.osd.6.smithi173.stderr:2024-03-23T03:46:42.943+0000 7f46d940d4c0 -1 osd.6 0 log_to_monitors true 2024-03-23T03:46:42.979 INFO:tasks.ceph.osd.4.smithi173.stderr:2024-03-23T03:46:42.978+0000 7f7fcaa274c0 -1 osd.4 0 log_to_monitors true 2024-03-23T03:46:45.320 INFO:tasks.ceph.osd.5.smithi173.stderr:2024-03-23T03:46:45.320+0000 7fbd0f600640 -1 osd.5 0 waiting for initial osdmap 2024-03-23T03:46:45.341 INFO:tasks.ceph.osd.5.smithi173.stderr:2024-03-23T03:46:45.341+0000 7fbd05600640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T03:46:45.427 INFO:tasks.ceph.osd.7.smithi173.stderr:2024-03-23T03:46:45.427+0000 7f4533e00640 -1 osd.7 0 waiting for initial osdmap 2024-03-23T03:46:45.540 INFO:tasks.ceph.osd.7.smithi173.stderr:2024-03-23T03:46:45.540+0000 7f4529400640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T03:46:45.770 INFO:tasks.ceph.osd.1.smithi102.stderr:2024-03-23T03:46:45.770+0000 7f33f6400640 -1 osd.1 0 waiting for initial osdmap 2024-03-23T03:46:45.799 INFO:tasks.ceph.osd.1.smithi102.stderr:2024-03-23T03:46:45.795+0000 7f33ec400640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T03:46:45.879 INFO:tasks.ceph.osd.3.smithi102.stderr:2024-03-23T03:46:45.878+0000 7fe722600640 -1 osd.3 0 waiting for initial osdmap 2024-03-23T03:46:45.887 INFO:tasks.ceph.osd.3.smithi102.stderr:2024-03-23T03:46:45.887+0000 7fe717c00640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T03:46:45.890 INFO:tasks.ceph.osd.0.smithi102.stderr:2024-03-23T03:46:45.890+0000 7ffac1a00640 -1 osd.0 0 waiting for initial osdmap 2024-03-23T03:46:45.896 INFO:tasks.ceph.osd.0.smithi102.stderr:2024-03-23T03:46:45.896+0000 7ffab7a00640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T03:46:46.203 INFO:tasks.ceph.osd.6.smithi173.stderr:2024-03-23T03:46:46.203+0000 7f46d3c00640 -1 osd.6 0 waiting for initial osdmap 2024-03-23T03:46:46.206 INFO:tasks.ceph.osd.2.smithi102.stderr:2024-03-23T03:46:46.206+0000 7f8074600640 -1 osd.2 0 waiting for initial osdmap 2024-03-23T03:46:46.210 INFO:tasks.ceph.osd.6.smithi173.stderr:2024-03-23T03:46:46.210+0000 7f46c9c00640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T03:46:46.213 INFO:tasks.ceph.osd.4.smithi173.stderr:2024-03-23T03:46:46.213+0000 7f7fc5400640 -1 osd.4 0 waiting for initial osdmap 2024-03-23T03:46:46.214 INFO:tasks.ceph.osd.2.smithi102.stderr:2024-03-23T03:46:46.214+0000 7f806a600640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T03:46:46.228 INFO:tasks.ceph.osd.4.smithi173.stderr:2024-03-23T03:46:46.228+0000 7f7fbb400640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T03:46:46.612 DEBUG:teuthology.orchestra.run.smithi102:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-23T03:46:46.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.794+0000 7fe51e000640 1 Processor -- start 2024-03-23T03:46:46.795 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.794+0000 7fe51e000640 1 -- start start 2024-03-23T03:46:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.796+0000 7fe51e000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5180afcc0 0x7fe5181407d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.796+0000 7fe51e000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe5180b04b0 0x7fe5180b0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.796+0000 7fe51e000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe5180cd410 0x7fe5180cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.796+0000 7fe51e000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5180b0e50 con 0x7fe5180cd410 2024-03-23T03:46:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.796+0000 7fe51e000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe518140d10 con 0x7fe5180b04b0 2024-03-23T03:46:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.796+0000 7fe51e000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe51815bbd0 con 0x7fe5180afcc0 2024-03-23T03:46:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.796+0000 7fe51d600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe5180cd410 0x7fe5180cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.796+0000 7fe51d600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe5180cd410 0x7fe5180cd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52464/0 (socket says 172.21.15.102:52464) 2024-03-23T03:46:46.797 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.796+0000 7fe51d600640 1 -- 172.21.15.102:0/2753595274 learned_addr learned my addr 172.21.15.102:0/2753595274 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:46.798 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.797+0000 7fe51cc00640 1 --2- 172.21.15.102:0/2753595274 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5180afcc0 0x7fe5181407d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:46.799 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.797+0000 7fe51d600640 1 -- 172.21.15.102:0/2753595274 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5180afcc0 msgr2=0x7fe5181407d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:46.799 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.797+0000 7fe51d600640 1 --2- 172.21.15.102:0/2753595274 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5180afcc0 0x7fe5181407d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:46.799 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.797+0000 7fe51d600640 1 -- 172.21.15.102:0/2753595274 >> v2:172.21.15.173:3300/0 conn(0x7fe5180b04b0 msgr2=0x7fe5180b0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:46.799 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.797+0000 7fe50fe00640 1 --2- 172.21.15.102:0/2753595274 >> v2:172.21.15.173:3300/0 conn(0x7fe5180b04b0 0x7fe5180b0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:46.799 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.797+0000 7fe51d600640 1 --2- 172.21.15.102:0/2753595274 >> v2:172.21.15.173:3300/0 conn(0x7fe5180b04b0 0x7fe5180b0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:46.799 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.797+0000 7fe51d600640 1 -- 172.21.15.102:0/2753595274 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5180b00a0 con 0x7fe5180cd410 2024-03-23T03:46:46.799 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.797+0000 7fe51cc00640 1 --2- 172.21.15.102:0/2753595274 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5180afcc0 0x7fe5181407d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:46:46.799 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.798+0000 7fe51d600640 1 --2- 172.21.15.102:0/2753595274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5180cd410 0x7fe5180cd880 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe5180ce530 tx=0x7fe51002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8124946b8c883996 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:46.799 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.798+0000 7fe517e00640 1 -- 172.21.15.102:0/2753595274 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe51002ca70 con 0x7fe5180cd410 2024-03-23T03:46:46.799 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.798+0000 7fe517e00640 1 -- 172.21.15.102:0/2753595274 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe51002cbd0 con 0x7fe5180cd410 2024-03-23T03:46:46.799 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.798+0000 7fe517e00640 1 -- 172.21.15.102:0/2753595274 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5100343f0 con 0x7fe5180cd410 2024-03-23T03:46:46.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.799+0000 7fe51e000640 1 -- 172.21.15.102:0/2753595274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5180cd410 msgr2=0x7fe5180cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:46.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.799+0000 7fe51e000640 1 --2- 172.21.15.102:0/2753595274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5180cd410 0x7fe5180cd880 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe5180ce530 tx=0x7fe51002bed0 comp rx=0 tx=0).stop 2024-03-23T03:46:46.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.799+0000 7fe51e000640 1 -- 172.21.15.102:0/2753595274 shutdown_connections 2024-03-23T03:46:46.800 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.799+0000 7fe51e000640 1 --2- 172.21.15.102:0/2753595274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5180cd410 0x7fe5180cd880 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:46.801 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.799+0000 7fe51e000640 1 --2- 172.21.15.102:0/2753595274 >> v2:172.21.15.173:3300/0 conn(0x7fe5180b04b0 0x7fe5180b0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:46.801 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.799+0000 7fe51e000640 1 --2- 172.21.15.102:0/2753595274 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5180afcc0 0x7fe5181407d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:46.801 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.799+0000 7fe51e000640 1 -- 172.21.15.102:0/2753595274 >> 172.21.15.102:0/2753595274 conn(0x7fe518066170 msgr2=0x7fe518069c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:46.801 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.799+0000 7fe51e000640 1 -- 172.21.15.102:0/2753595274 shutdown_connections 2024-03-23T03:46:46.801 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.799+0000 7fe51e000640 1 -- 172.21.15.102:0/2753595274 wait complete. 2024-03-23T03:46:46.801 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.800+0000 7fe51e000640 1 Processor -- start 2024-03-23T03:46:46.801 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.801+0000 7fe51e000640 1 -- start start 2024-03-23T03:46:46.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.801+0000 7fe51e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5180afcc0 0x7fe51812e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:46.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.801+0000 7fe51e000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5180b04b0 0x7fe51813c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:46.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.801+0000 7fe51e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5180cd410 0x7fe51812d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:46.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.801+0000 7fe51e000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51812ee50 con 0x7fe5180afcc0 2024-03-23T03:46:46.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.801+0000 7fe51e000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51812c410 con 0x7fe5180cd410 2024-03-23T03:46:46.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.801+0000 7fe51e000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe51812c6f0 con 0x7fe5180b04b0 2024-03-23T03:46:46.803 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.802+0000 7fe51cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5180afcc0 0x7fe51812e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:46.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.802+0000 7fe51cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5180afcc0 0x7fe51812e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52480/0 (socket says 172.21.15.102:52480) 2024-03-23T03:46:46.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.802+0000 7fe51cc00640 1 -- 172.21.15.102:0/1871340814 learned_addr learned my addr 172.21.15.102:0/1871340814 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:46.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.802+0000 7fe51cc00640 1 -- 172.21.15.102:0/1871340814 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5180b04b0 msgr2=0x7fe51813c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:46.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.802+0000 7fe51cc00640 1 --2- 172.21.15.102:0/1871340814 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5180b04b0 0x7fe51813c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:46.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.802+0000 7fe51cc00640 1 -- 172.21.15.102:0/1871340814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5180cd410 msgr2=0x7fe51812d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:46.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.802+0000 7fe51cc00640 1 --2- 172.21.15.102:0/1871340814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5180cd410 0x7fe51812d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:46.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.802+0000 7fe51cc00640 1 -- 172.21.15.102:0/1871340814 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe510009b30 con 0x7fe5180afcc0 2024-03-23T03:46:46.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.802+0000 7fe51cc00640 1 --2- 172.21.15.102:0/1871340814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5180afcc0 0x7fe51812e910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe50800c7f0 tx=0x7fe50800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:46.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.802+0000 7fe515600640 1 -- 172.21.15.102:0/1871340814 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe508007820 con 0x7fe5180afcc0 2024-03-23T03:46:46.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.803+0000 7fe515600640 1 -- 172.21.15.102:0/1871340814 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe508007980 con 0x7fe5180afcc0 2024-03-23T03:46:46.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.803+0000 7fe51e000640 1 -- 172.21.15.102:0/1871340814 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe51812c9d0 con 0x7fe5180afcc0 2024-03-23T03:46:46.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.803+0000 7fe515600640 1 -- 172.21.15.102:0/1871340814 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe508007cf0 con 0x7fe5180afcc0 2024-03-23T03:46:46.804 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.803+0000 7fe51e000640 1 -- 172.21.15.102:0/1871340814 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe51812cf90 con 0x7fe5180afcc0 2024-03-23T03:46:46.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.804+0000 7fe51e000640 1 -- 172.21.15.102:0/1871340814 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5180670d0 con 0x7fe5180afcc0 2024-03-23T03:46:46.805 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.805+0000 7fe515600640 1 -- 172.21.15.102:0/1871340814 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7fe5080113f0 con 0x7fe5180afcc0 2024-03-23T03:46:46.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.805+0000 7fe515600640 1 --2- 172.21.15.102:0/1871340814 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe4f406ab70 0x7fe4f406d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:46.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.805+0000 7fe515600640 1 -- 172.21.15.102:0/1871340814 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fe5080890e0 con 0x7fe5180afcc0 2024-03-23T03:46:46.806 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.806+0000 7fe50fe00640 1 --2- 172.21.15.102:0/1871340814 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe4f406ab70 0x7fe4f406d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:46.809 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.806+0000 7fe50fe00640 1 --2- 172.21.15.102:0/1871340814 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe4f406ab70 0x7fe4f406d030 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe500006b10 tx=0x7fe500008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:46.811 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.811+0000 7fe515600640 1 -- 172.21.15.102:0/1871340814 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe508054860 con 0x7fe5180afcc0 2024-03-23T03:46:46.993 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.993+0000 7fe51e000640 1 -- 172.21.15.102:0/1871340814 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe51813d8f0 con 0x7fe5180afcc0 2024-03-23T03:46:46.994 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.994+0000 7fe515600640 1 -- 172.21.15.102:0/1871340814 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10488 (secure 0 0 0) 0x7fe508087e60 con 0x7fe5180afcc0 2024-03-23T03:46:46.995 INFO:teuthology.misc.health.smithi102.stdout: 2024-03-23T03:46:46.995 INFO:teuthology.misc.health.smithi102.stdout:{"epoch":14,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:46.328969+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:46:46.997 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.996+0000 7fe51e000640 1 -- 172.21.15.102:0/1871340814 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe4f406ab70 msgr2=0x7fe4f406d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:46.997 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.996+0000 7fe51e000640 1 --2- 172.21.15.102:0/1871340814 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe4f406ab70 0x7fe4f406d030 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe500006b10 tx=0x7fe500008040 comp rx=0 tx=0).stop 2024-03-23T03:46:46.997 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.996+0000 7fe51e000640 1 -- 172.21.15.102:0/1871340814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5180afcc0 msgr2=0x7fe51812e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:46.997 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.996+0000 7fe51e000640 1 --2- 172.21.15.102:0/1871340814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5180afcc0 0x7fe51812e910 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe50800c7f0 tx=0x7fe50800ccb0 comp rx=0 tx=0).stop 2024-03-23T03:46:46.997 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.996+0000 7fe51e000640 1 -- 172.21.15.102:0/1871340814 shutdown_connections 2024-03-23T03:46:46.997 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.996+0000 7fe51e000640 1 --2- 172.21.15.102:0/1871340814 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe4f406ab70 0x7fe4f406d030 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:46.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.996+0000 7fe51e000640 1 --2- 172.21.15.102:0/1871340814 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5180cd410 0x7fe51812d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:46.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.996+0000 7fe51e000640 1 --2- 172.21.15.102:0/1871340814 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5180b04b0 0x7fe51813c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:46.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.996+0000 7fe51e000640 1 --2- 172.21.15.102:0/1871340814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5180afcc0 0x7fe51812e910 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:46.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.996+0000 7fe51e000640 1 -- 172.21.15.102:0/1871340814 >> 172.21.15.102:0/1871340814 conn(0x7fe518066170 msgr2=0x7fe518154220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:46.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.997+0000 7fe51e000640 1 -- 172.21.15.102:0/1871340814 shutdown_connections 2024-03-23T03:46:46.998 INFO:teuthology.misc.health.smithi102.stderr:2024-03-23T03:46:46.997+0000 7fe51e000640 1 -- 172.21.15.102:0/1871340814 wait complete. 2024-03-23T03:46:47.010 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-23T03:46:47.010 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-23T03:46:47.011 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T03:46:47.011 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T03:46:47.018 INFO:tasks.ceph.mds.a:Started 2024-03-23T03:46:47.018 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T03:46:47.018 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T03:46:47.025 INFO:tasks.ceph.mds.c:Started 2024-03-23T03:46:47.025 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T03:46:47.025 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T03:46:47.033 INFO:tasks.ceph.mds.b:Started 2024-03-23T03:46:47.033 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T03:46:47.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T03:46:47.040 INFO:tasks.ceph.mds.d:Started 2024-03-23T03:46:47.040 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-23T03:46:47.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:46:47.164 INFO:tasks.ceph.mds.c.smithi102.stdout:starting mds.c at 2024-03-23T03:46:47.165 INFO:tasks.ceph.mds.a.smithi102.stdout:starting mds.a at 2024-03-23T03:46:47.196 INFO:tasks.ceph.mds.d.smithi173.stdout:starting mds.d at 2024-03-23T03:46:47.197 INFO:tasks.ceph.mds.b.smithi173.stdout:starting mds.b at 2024-03-23T03:46:47.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.219+0000 7fdbb2400640 1 Processor -- start 2024-03-23T03:46:47.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.219+0000 7fdbb2400640 1 -- start start 2024-03-23T03:46:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.220+0000 7fdbb2400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdbac0b5a30 0x7fdbac0b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.220+0000 7fdbb2400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbac0d1830 con 0x7fdbac0b4350 2024-03-23T03:46:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.220+0000 7fdbb2400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbac0b1f20 con 0x7fdbac15cda0 2024-03-23T03:46:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.220+0000 7fdbb2400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbac0d0c20 con 0x7fdbac0b5a30 2024-03-23T03:46:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.220+0000 7fdbaaa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdbac0b5a30 0x7fdbac0b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.220+0000 7fdbaaa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdbac0b5a30 0x7fdbac0b5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52528/0 (socket says 172.21.15.102:52528) 2024-03-23T03:46:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.220+0000 7fdbaaa00640 1 -- 172.21.15.102:0/2090382300 learned_addr learned my addr 172.21.15.102:0/2090382300 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.220+0000 7fdbaaa00640 1 -- 172.21.15.102:0/2090382300 >> v1:172.21.15.173:6789/0 conn(0x7fdbac15cda0 legacy=0x7fdbac0cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.220+0000 7fdbaaa00640 1 -- 172.21.15.102:0/2090382300 >> v1:172.21.15.102:6789/0 conn(0x7fdbac0b4350 legacy=0x7fdbac0d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:47.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.220+0000 7fdbaaa00640 1 -- 172.21.15.102:0/2090382300 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbac0d0f00 con 0x7fdbac0b5a30 2024-03-23T03:46:47.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.221+0000 7fdbaaa00640 1 --2- 172.21.15.102:0/2090382300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbac0b5a30 0x7fdbac0b5e90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdb98009870 tx=0x7fdb9802ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c49c9482f561031 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:47.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.222+0000 7fdbaa000640 1 -- 172.21.15.102:0/2090382300 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9802f860 con 0x7fdbac0b5a30 2024-03-23T03:46:47.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.222+0000 7fdbaa000640 1 -- 172.21.15.102:0/2090382300 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9802f9c0 con 0x7fdbac0b5a30 2024-03-23T03:46:47.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.222+0000 7fdbaa000640 1 -- 172.21.15.102:0/2090382300 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9802fd50 con 0x7fdbac0b5a30 2024-03-23T03:46:47.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.222+0000 7fdbb2400640 1 -- 172.21.15.102:0/2090382300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbac0b5a30 msgr2=0x7fdbac0b5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:47.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.222+0000 7fdbb2400640 1 --2- 172.21.15.102:0/2090382300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbac0b5a30 0x7fdbac0b5e90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdb98009870 tx=0x7fdb9802ba40 comp rx=0 tx=0).stop 2024-03-23T03:46:47.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.222+0000 7fdbb2400640 1 -- 172.21.15.102:0/2090382300 shutdown_connections 2024-03-23T03:46:47.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.222+0000 7fdbb2400640 1 --2- 172.21.15.102:0/2090382300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbac0b5a30 0x7fdbac0b5e90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:47.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.222+0000 7fdbb2400640 1 -- 172.21.15.102:0/2090382300 >> 172.21.15.102:0/2090382300 conn(0x7fdbac0cf350 msgr2=0x7fdbac0cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:47.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.222+0000 7fdbb2400640 1 -- 172.21.15.102:0/2090382300 shutdown_connections 2024-03-23T03:46:47.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.222+0000 7fdbb2400640 1 -- 172.21.15.102:0/2090382300 wait complete. 2024-03-23T03:46:47.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.224+0000 7fdbb2400640 1 Processor -- start 2024-03-23T03:46:47.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.224+0000 7fdbb2400640 1 -- start start 2024-03-23T03:46:47.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.225+0000 7fdbb2400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdbac0b4350 0x7fdbac0cac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:47.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.225+0000 7fdbb2400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbac0b5a30 0x7fdbac1f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:47.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.225+0000 7fdbb2400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbac15cda0 0x7fdbac0c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:47.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.225+0000 7fdbb2400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbac15e660 con 0x7fdbac15cda0 2024-03-23T03:46:47.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.225+0000 7fdbb2400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbac15e940 con 0x7fdbac0b5a30 2024-03-23T03:46:47.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.225+0000 7fdbb2400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbac15ec20 con 0x7fdbac0b4350 2024-03-23T03:46:47.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.225+0000 7fdbabe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbac15cda0 0x7fdbac0c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:47.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.225+0000 7fdbabe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbac15cda0 0x7fdbac0c8d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52530/0 (socket says 172.21.15.102:52530) 2024-03-23T03:46:47.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.225+0000 7fdbabe00640 1 -- 172.21.15.102:0/1390613829 learned_addr learned my addr 172.21.15.102:0/1390613829 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:47.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.225+0000 7fdbabe00640 1 -- 172.21.15.102:0/1390613829 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdbac0b4350 msgr2=0x7fdbac0cac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:47.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.225+0000 7fdbabe00640 1 --2- 172.21.15.102:0/1390613829 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdbac0b4350 0x7fdbac0cac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:47.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.225+0000 7fdbabe00640 1 -- 172.21.15.102:0/1390613829 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbac0b5a30 msgr2=0x7fdbac1f28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:47.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.225+0000 7fdbabe00640 1 --2- 172.21.15.102:0/1390613829 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbac0b5a30 0x7fdbac1f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:47.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.225+0000 7fdbabe00640 1 -- 172.21.15.102:0/1390613829 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb980094d0 con 0x7fdbac15cda0 2024-03-23T03:46:47.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.226+0000 7fdbabe00640 1 --2- 172.21.15.102:0/1390613829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbac15cda0 0x7fdbac0c8d20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdba400c7a0 tx=0x7fdba400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:47.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.226+0000 7fdba3400640 1 -- 172.21.15.102:0/1390613829 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba4003b70 con 0x7fdbac15cda0 2024-03-23T03:46:47.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.226+0000 7fdba3400640 1 -- 172.21.15.102:0/1390613829 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdba4003cf0 con 0x7fdbac15cda0 2024-03-23T03:46:47.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.226+0000 7fdba3400640 1 -- 172.21.15.102:0/1390613829 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdba4003ea0 con 0x7fdbac15cda0 2024-03-23T03:46:47.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.226+0000 7fdbb2400640 1 -- 172.21.15.102:0/1390613829 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbac15ef00 con 0x7fdbac15cda0 2024-03-23T03:46:47.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.226+0000 7fdbb2400640 1 -- 172.21.15.102:0/1390613829 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbac15e260 con 0x7fdbac15cda0 2024-03-23T03:46:47.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.227+0000 7fdbb2400640 1 -- 172.21.15.102:0/1390613829 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbac073390 con 0x7fdbac15cda0 2024-03-23T03:46:47.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.228+0000 7fdba3400640 1 -- 172.21.15.102:0/1390613829 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7fdba400aa90 con 0x7fdbac15cda0 2024-03-23T03:46:47.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.228+0000 7fdba3400640 1 --2- 172.21.15.102:0/1390613829 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdb8c06ab70 0x7fdb8c06d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:47.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.228+0000 7fdba3400640 1 -- 172.21.15.102:0/1390613829 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fdba4088be0 con 0x7fdbac15cda0 2024-03-23T03:46:47.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.229+0000 7fdbab400640 1 --2- 172.21.15.102:0/1390613829 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdb8c06ab70 0x7fdb8c06d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:47.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.229+0000 7fdbab400640 1 --2- 172.21.15.102:0/1390613829 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdb8c06ab70 0x7fdb8c06d030 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c000f60 tx=0x7fdb9c0079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:47.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.233+0000 7fdba3400640 1 -- 172.21.15.102:0/1390613829 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdba4054360 con 0x7fdbac15cda0 2024-03-23T03:46:47.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.414+0000 7fdbb2400640 1 -- 172.21.15.102:0/1390613829 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbac0c6160 con 0x7fdbac15cda0 2024-03-23T03:46:47.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.415+0000 7fdba3400640 1 -- 172.21.15.102:0/1390613829 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+10672 (secure 0 0 0) 0x7fdba406be60 con 0x7fdbac15cda0 2024-03-23T03:46:47.416 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:46:47.416 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":15,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:47.333822+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:46:47.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.418+0000 7fdbb2400640 1 -- 172.21.15.102:0/1390613829 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdb8c06ab70 msgr2=0x7fdb8c06d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:47.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.418+0000 7fdbb2400640 1 --2- 172.21.15.102:0/1390613829 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdb8c06ab70 0x7fdb8c06d030 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c000f60 tx=0x7fdb9c0079b0 comp rx=0 tx=0).stop 2024-03-23T03:46:47.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.418+0000 7fdbb2400640 1 -- 172.21.15.102:0/1390613829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbac15cda0 msgr2=0x7fdbac0c8d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:47.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.418+0000 7fdbb2400640 1 --2- 172.21.15.102:0/1390613829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbac15cda0 0x7fdbac0c8d20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdba400c7a0 tx=0x7fdba400cc60 comp rx=0 tx=0).stop 2024-03-23T03:46:47.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.418+0000 7fdbb2400640 1 -- 172.21.15.102:0/1390613829 shutdown_connections 2024-03-23T03:46:47.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.418+0000 7fdbb2400640 1 --2- 172.21.15.102:0/1390613829 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdb8c06ab70 0x7fdb8c06d030 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:47.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.418+0000 7fdbb2400640 1 --2- 172.21.15.102:0/1390613829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbac15cda0 0x7fdbac0c8d20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:47.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.418+0000 7fdbb2400640 1 --2- 172.21.15.102:0/1390613829 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbac0b5a30 0x7fdbac1f28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:47.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.418+0000 7fdbb2400640 1 --2- 172.21.15.102:0/1390613829 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdbac0b4350 0x7fdbac0cac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:47.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.418+0000 7fdbb2400640 1 -- 172.21.15.102:0/1390613829 >> 172.21.15.102:0/1390613829 conn(0x7fdbac0cf350 msgr2=0x7fdbac0b2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:47.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.418+0000 7fdbb2400640 1 -- 172.21.15.102:0/1390613829 shutdown_connections 2024-03-23T03:46:47.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.418+0000 7fdbb2400640 1 -- 172.21.15.102:0/1390613829 wait complete. 2024-03-23T03:46:47.434 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-03-23T03:46:47.434 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:46:47.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.627+0000 7fb6e2200640 1 Processor -- start 2024-03-23T03:46:47.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.628+0000 7fb6e2200640 1 -- start start 2024-03-23T03:46:47.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.629+0000 7fb6e2200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6dc15cd70 0x7fb6dc0d0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:47.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.629+0000 7fb6e2200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb6dc0b4a50 0x7fb6dc0b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:47.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.629+0000 7fb6e2200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc0d17d0 con 0x7fb6dc0d1330 2024-03-23T03:46:47.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.629+0000 7fb6e2200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6dc15d150 con 0x7fb6dc0b4a50 2024-03-23T03:46:47.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.629+0000 7fb6e2200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc0d1ab0 con 0x7fb6dc15cd70 2024-03-23T03:46:47.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.629+0000 7fb6dbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb6dc0b4a50 0x7fb6dc0b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:47.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.629+0000 7fb6dbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb6dc0b4a50 0x7fb6dc0b5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52542/0 (socket says 172.21.15.102:52542) 2024-03-23T03:46:47.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.629+0000 7fb6dbe00640 1 -- 172.21.15.102:0/4224477408 learned_addr learned my addr 172.21.15.102:0/4224477408 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:47.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.630+0000 7fb6db400640 1 --2- 172.21.15.102:0/4224477408 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6dc15cd70 0x7fb6dc0d0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:47.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.630+0000 7fb6dbe00640 1 -- 172.21.15.102:0/4224477408 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6dc15cd70 msgr2=0x7fb6dc0d0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:47.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.630+0000 7fb6dbe00640 1 --2- 172.21.15.102:0/4224477408 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6dc15cd70 0x7fb6dc0d0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:47.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.630+0000 7fb6dbe00640 1 -- 172.21.15.102:0/4224477408 >> v1:172.21.15.102:6789/0 conn(0x7fb6dc0d1330 legacy=0x7fb6dc0b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:47.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.630+0000 7fb6dbe00640 1 -- 172.21.15.102:0/4224477408 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc0d1d90 con 0x7fb6dc0b4a50 2024-03-23T03:46:47.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.631+0000 7fb6dbe00640 1 --2- 172.21.15.102:0/4224477408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6dc0b4a50 0x7fb6dc0b5ba0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4009e50 tx=0x7fb6d402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e7d24e59a10267a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:47.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.631+0000 7fb6daa00640 1 -- 172.21.15.102:0/4224477408 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d402cb00 con 0x7fb6dc0b4a50 2024-03-23T03:46:47.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.631+0000 7fb6daa00640 1 -- 172.21.15.102:0/4224477408 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d402cc60 con 0x7fb6dc0b4a50 2024-03-23T03:46:47.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.631+0000 7fb6daa00640 1 -- 172.21.15.102:0/4224477408 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d40343f0 con 0x7fb6dc0b4a50 2024-03-23T03:46:47.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.632+0000 7fb6e2200640 1 -- 172.21.15.102:0/4224477408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6dc0b4a50 msgr2=0x7fb6dc0b5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:47.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.632+0000 7fb6e2200640 1 --2- 172.21.15.102:0/4224477408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6dc0b4a50 0x7fb6dc0b5ba0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4009e50 tx=0x7fb6d402bf60 comp rx=0 tx=0).stop 2024-03-23T03:46:47.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.632+0000 7fb6e2200640 1 -- 172.21.15.102:0/4224477408 shutdown_connections 2024-03-23T03:46:47.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.632+0000 7fb6e2200640 1 --2- 172.21.15.102:0/4224477408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6dc0b4a50 0x7fb6dc0b5ba0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:47.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.632+0000 7fb6e2200640 1 --2- 172.21.15.102:0/4224477408 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6dc15cd70 0x7fb6dc0d0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:47.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.632+0000 7fb6e2200640 1 -- 172.21.15.102:0/4224477408 >> 172.21.15.102:0/4224477408 conn(0x7fb6dc0cf340 msgr2=0x7fb6dc0cf750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:47.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.632+0000 7fb6e2200640 1 -- 172.21.15.102:0/4224477408 shutdown_connections 2024-03-23T03:46:47.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.632+0000 7fb6e2200640 1 -- 172.21.15.102:0/4224477408 wait complete. 2024-03-23T03:46:47.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.634+0000 7fb6e2200640 1 Processor -- start 2024-03-23T03:46:47.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.634+0000 7fb6e2200640 1 -- start start 2024-03-23T03:46:47.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.634+0000 7fb6e2200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6dc0b4a50 0x7fb6dc0a9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:47.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.634+0000 7fb6e2200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6dc0d1330 0x7fb6dc14e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:47.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.635+0000 7fb6e2200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6dc15cd70 0x7fb6dc14edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:47.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.635+0000 7fb6e2200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc14f300 con 0x7fb6dc15cd70 2024-03-23T03:46:47.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.635+0000 7fb6e2200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc14f650 con 0x7fb6dc0d1330 2024-03-23T03:46:47.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.635+0000 7fb6e2200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc0ca8c0 con 0x7fb6dc0b4a50 2024-03-23T03:46:47.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.635+0000 7fb6dbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6dc15cd70 0x7fb6dc14edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:47.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.635+0000 7fb6dbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6dc15cd70 0x7fb6dc14edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52558/0 (socket says 172.21.15.102:52558) 2024-03-23T03:46:47.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.635+0000 7fb6dbe00640 1 -- 172.21.15.102:0/1381688849 learned_addr learned my addr 172.21.15.102:0/1381688849 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:47.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.635+0000 7fb6dbe00640 1 -- 172.21.15.102:0/1381688849 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6dc0b4a50 msgr2=0x7fb6dc0a9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:47.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.635+0000 7fb6dbe00640 1 --2- 172.21.15.102:0/1381688849 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6dc0b4a50 0x7fb6dc0a9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:47.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.635+0000 7fb6dbe00640 1 -- 172.21.15.102:0/1381688849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6dc0d1330 msgr2=0x7fb6dc14e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:47.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.635+0000 7fb6d2a00640 1 --2- 172.21.15.102:0/1381688849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6dc0d1330 0x7fb6dc14e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:47.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.635+0000 7fb6dbe00640 1 --2- 172.21.15.102:0/1381688849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6dc0d1330 0x7fb6dc14e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:47.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.635+0000 7fb6dbe00640 1 -- 172.21.15.102:0/1381688849 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d4009b30 con 0x7fb6dc15cd70 2024-03-23T03:46:47.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.636+0000 7fb6dbe00640 1 --2- 172.21.15.102:0/1381688849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6dc15cd70 0x7fb6dc14edc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4036040 tx=0x7fb6d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:47.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.636+0000 7fb6d3e00640 1 -- 172.21.15.102:0/1381688849 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4009510 con 0x7fb6dc15cd70 2024-03-23T03:46:47.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.636+0000 7fb6d3e00640 1 -- 172.21.15.102:0/1381688849 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d403e450 con 0x7fb6dc15cd70 2024-03-23T03:46:47.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.636+0000 7fb6e2200640 1 -- 172.21.15.102:0/1381688849 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6dc0cab90 con 0x7fb6dc15cd70 2024-03-23T03:46:47.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.636+0000 7fb6d3e00640 1 -- 172.21.15.102:0/1381688849 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d403e800 con 0x7fb6dc15cd70 2024-03-23T03:46:47.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.636+0000 7fb6e2200640 1 -- 172.21.15.102:0/1381688849 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6dc0cb3d0 con 0x7fb6dc15cd70 2024-03-23T03:46:47.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.637+0000 7fb6e2200640 1 -- 172.21.15.102:0/1381688849 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6dc06c240 con 0x7fb6dc15cd70 2024-03-23T03:46:47.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.638+0000 7fb6d3e00640 1 -- 172.21.15.102:0/1381688849 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7fb6d403e960 con 0x7fb6dc15cd70 2024-03-23T03:46:47.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.639+0000 7fb6d3e00640 1 --2- 172.21.15.102:0/1381688849 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6b806aba0 0x7fb6b806d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:47.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.639+0000 7fb6d3e00640 1 -- 172.21.15.102:0/1381688849 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4471+0+0 (secure 0 0 0) 0x7fb6d40abb50 con 0x7fb6dc15cd70 2024-03-23T03:46:47.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.640+0000 7fb6db400640 1 --2- 172.21.15.102:0/1381688849 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6b806aba0 0x7fb6b806d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:47.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.640+0000 7fb6db400640 1 --2- 172.21.15.102:0/1381688849 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6b806aba0 0x7fb6b806d060 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc006fd0 tx=0x7fb6cc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:47.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.644+0000 7fb6d3e00640 1 -- 172.21.15.102:0/1381688849 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb6d40ade60 con 0x7fb6dc15cd70 2024-03-23T03:46:47.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.883+0000 7fb6e2200640 1 -- 172.21.15.102:0/1381688849 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6dc073390 con 0x7fb6dc15cd70 2024-03-23T03:46:47.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.885+0000 7fb6d3e00640 1 -- 172.21.15.102:0/1381688849 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1256 (secure 0 0 0) 0x7fb6d4043500 con 0x7fb6dc15cd70 2024-03-23T03:46:47.885 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:46:47.886 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/891818172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":891818172},{"type":"v1","addr":"172.21.15.102:6836","nonce":891818172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2024-03-23T03:46:47.886 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 2 2024-03-23T03:46:47.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.887+0000 7fb6e2200640 1 -- 172.21.15.102:0/1381688849 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6b806aba0 msgr2=0x7fb6b806d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:47.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.887+0000 7fb6e2200640 1 --2- 172.21.15.102:0/1381688849 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6b806aba0 0x7fb6b806d060 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc006fd0 tx=0x7fb6cc008040 comp rx=0 tx=0).stop 2024-03-23T03:46:47.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.887+0000 7fb6e2200640 1 -- 172.21.15.102:0/1381688849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6dc15cd70 msgr2=0x7fb6dc14edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:47.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.887+0000 7fb6e2200640 1 --2- 172.21.15.102:0/1381688849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6dc15cd70 0x7fb6dc14edc0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4036040 tx=0x7fb6d4004a40 comp rx=0 tx=0).stop 2024-03-23T03:46:47.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.887+0000 7fb6e2200640 1 -- 172.21.15.102:0/1381688849 shutdown_connections 2024-03-23T03:46:47.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.887+0000 7fb6e2200640 1 --2- 172.21.15.102:0/1381688849 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6b806aba0 0x7fb6b806d060 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:47.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.887+0000 7fb6e2200640 1 --2- 172.21.15.102:0/1381688849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6dc15cd70 0x7fb6dc14edc0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:47.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.887+0000 7fb6e2200640 1 --2- 172.21.15.102:0/1381688849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6dc0d1330 0x7fb6dc14e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:47.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.887+0000 7fb6e2200640 1 --2- 172.21.15.102:0/1381688849 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6dc0b4a50 0x7fb6dc0a9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:47.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.887+0000 7fb6e2200640 1 -- 172.21.15.102:0/1381688849 >> 172.21.15.102:0/1381688849 conn(0x7fb6dc0cf340 msgr2=0x7fb6dc15f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:47.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.888+0000 7fb6e2200640 1 -- 172.21.15.102:0/1381688849 shutdown_connections 2024-03-23T03:46:47.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:47.888+0000 7fb6e2200640 1 -- 172.21.15.102:0/1381688849 wait complete. 2024-03-23T03:46:49.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:46:49.249 DEBUG:teuthology.orchestra.run.smithi173:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:46:49.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:46:50.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.101+0000 7f21bfa00640 1 Processor -- start 2024-03-23T03:46:50.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.101+0000 7f21bfa00640 1 -- start start 2024-03-23T03:46:50.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.103+0000 7f21bfa00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f21b815cda0 0x7f21b80cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:50.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.103+0000 7f21bfa00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21b80b5a30 0x7f21b80b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:50.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.103+0000 7f21bfa00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21b80d1830 con 0x7f21b80b4350 2024-03-23T03:46:50.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.103+0000 7f21bfa00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21b80b1f20 con 0x7f21b815cda0 2024-03-23T03:46:50.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.103+0000 7f21bfa00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b80d0c20 con 0x7f21b80b5a30 2024-03-23T03:46:50.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.104+0000 7f21bda00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f21b80b4350 0x7f21b80d1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40958/0 (socket says 172.21.15.102:40958) 2024-03-23T03:46:50.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.104+0000 7f21bda00640 1 -- 172.21.15.102:0/3048537705 learned_addr learned my addr 172.21.15.102:0/3048537705 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:46:50.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.104+0000 7f21bd000640 1 --2- 172.21.15.102:0/3048537705 >> v2:172.21.15.173:3300/0 conn(0x7f21b815cda0 0x7f21b80cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:50.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.104+0000 7f21b7e00640 1 --2- 172.21.15.102:0/3048537705 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21b80b5a30 0x7f21b80b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:50.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.104+0000 7f21b7400640 1 -- 172.21.15.102:0/3048537705 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352815291 0 0) 0x7f21b80d1830 con 0x7f21b80b4350 2024-03-23T03:46:50.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.104+0000 7f21b7400640 1 -- 172.21.15.102:0/3048537705 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21a0003520 con 0x7f21b80b4350 2024-03-23T03:46:50.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.105+0000 7f21b7400640 1 -- 172.21.15.102:0/3048537705 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1262238204 0 0) 0x7f21a0003520 con 0x7f21b80b4350 2024-03-23T03:46:50.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.105+0000 7f21b7400640 1 -- 172.21.15.102:0/3048537705 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21b80b5a30 msgr2=0x7f21b80b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:50.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.105+0000 7f21b7400640 1 --2- 172.21.15.102:0/3048537705 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21b80b5a30 0x7f21b80b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.105+0000 7f21b7400640 1 -- 172.21.15.102:0/3048537705 >> v2:172.21.15.173:3300/0 conn(0x7f21b815cda0 msgr2=0x7f21b80cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:50.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.105+0000 7f21b7400640 1 --2- 172.21.15.102:0/3048537705 >> v2:172.21.15.173:3300/0 conn(0x7f21b815cda0 0x7f21b80cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.105+0000 7f21b7400640 1 -- 172.21.15.102:0/3048537705 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b80d0f00 con 0x7f21b80b4350 2024-03-23T03:46:50.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.105+0000 7f21b7400640 1 -- 172.21.15.102:0/3048537705 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f21b00019f0 con 0x7f21b80b4350 2024-03-23T03:46:50.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.105+0000 7f21b7400640 1 -- 172.21.15.102:0/3048537705 >> v1:172.21.15.102:6789/0 conn(0x7f21b80b4350 legacy=0x7f21b80d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:50.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.105+0000 7f21b7400640 1 --2- 172.21.15.102:0/3048537705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21a0003970 0x7f21a0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:50.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.105+0000 7f21b7400640 1 --2- 172.21.15.102:0/3048537705 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21b80b5a30 0x7f21a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:50.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.105+0000 7f21b7400640 1 --2- 172.21.15.102:0/3048537705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21a0004d00 0x7f21a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:50.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.105+0000 7f21b7400640 1 -- 172.21.15.102:0/3048537705 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b80b1f20 con 0x7f21a0004d00 2024-03-23T03:46:50.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.105+0000 7f21b7400640 1 -- 172.21.15.102:0/3048537705 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b80d0c20 con 0x7f21a0003970 2024-03-23T03:46:50.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.105+0000 7f21b7400640 1 -- 172.21.15.102:0/3048537705 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a000b980 con 0x7f21b80b5a30 2024-03-23T03:46:50.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.105+0000 7f21bda00640 1 --2- 172.21.15.102:0/3048537705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21a0004d00 0x7f21a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:50.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.106+0000 7f21bda00640 1 -- 172.21.15.102:0/3048537705 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21b80b5a30 msgr2=0x7f21a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:50.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.106+0000 7f21bda00640 1 --2- 172.21.15.102:0/3048537705 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21b80b5a30 0x7f21a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.106+0000 7f21bda00640 1 -- 172.21.15.102:0/3048537705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21a0003970 msgr2=0x7f21a0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:50.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.106+0000 7f21bda00640 1 --2- 172.21.15.102:0/3048537705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21a0003970 0x7f21a0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.106+0000 7f21bda00640 1 -- 172.21.15.102:0/3048537705 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b80d0f00 con 0x7f21a0004d00 2024-03-23T03:46:50.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.107+0000 7f21bda00640 1 --2- 172.21.15.102:0/3048537705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21a0004d00 0x7f21a0005180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21b000a970 tx=0x7f21a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4935704f83c9b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:50.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.107+0000 7f21b7400640 1 -- 172.21.15.102:0/3048537705 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b0034630 con 0x7f21a0004d00 2024-03-23T03:46:50.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.107+0000 7f21b7400640 1 -- 172.21.15.102:0/3048537705 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b002f070 con 0x7f21a0004d00 2024-03-23T03:46:50.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.107+0000 7f21b7400640 1 -- 172.21.15.102:0/3048537705 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b00349f0 con 0x7f21a0004d00 2024-03-23T03:46:50.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.108+0000 7f21bfa00640 1 -- 172.21.15.102:0/3048537705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21a0004d00 msgr2=0x7f21a0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:50.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.108+0000 7f21bfa00640 1 --2- 172.21.15.102:0/3048537705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21a0004d00 0x7f21a0005180 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f21b000a970 tx=0x7f21a00057f0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.108+0000 7f21bda00640 1 -- 172.21.15.102:0/3048537705 reap_dead start 2024-03-23T03:46:50.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.108+0000 7f21bfa00640 1 -- 172.21.15.102:0/3048537705 shutdown_connections 2024-03-23T03:46:50.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.108+0000 7f21bfa00640 1 -- 172.21.15.102:0/3048537705 >> 172.21.15.102:0/3048537705 conn(0x7f21b80cf350 msgr2=0x7f21b80cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:50.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.108+0000 7f21bfa00640 1 -- 172.21.15.102:0/3048537705 shutdown_connections 2024-03-23T03:46:50.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.108+0000 7f21bfa00640 1 -- 172.21.15.102:0/3048537705 wait complete. 2024-03-23T03:46:50.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.110+0000 7f21bfa00640 1 Processor -- start 2024-03-23T03:46:50.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.110+0000 7f21bfa00640 1 -- start start 2024-03-23T03:46:50.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.112+0000 7f21bfa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21a0004d00 0x7f21b8158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:50.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.112+0000 7f21bfa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b80c8690 0x7f21b80c8af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:50.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.112+0000 7f21bfa00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21b806b140 0x7f21b806b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:50.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.112+0000 7f21bfa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b815dc30 con 0x7f21a0004d00 2024-03-23T03:46:50.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.112+0000 7f21bfa00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21b80c9030 con 0x7f21b80c8690 2024-03-23T03:46:50.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.112+0000 7f21bfa00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21b806bb00 con 0x7f21b806b140 2024-03-23T03:46:50.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.112+0000 7f21bd000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21a0004d00 0x7f21b8158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:50.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.112+0000 7f21bd000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21a0004d00 0x7f21b8158f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52602/0 (socket says 172.21.15.102:52602) 2024-03-23T03:46:50.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.112+0000 7f21bd000640 1 -- 172.21.15.102:0/593994468 learned_addr learned my addr 172.21.15.102:0/593994468 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:50.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.112+0000 7f21b7e00640 1 --2- 172.21.15.102:0/593994468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b80c8690 0x7f21b80c8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:50.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.112+0000 7f21bda00640 1 --2- 172.21.15.102:0/593994468 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21b806b140 0x7f21b806b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:50.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.113+0000 7f21bd000640 1 -- 172.21.15.102:0/593994468 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21b806b140 msgr2=0x7f21b806b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:50.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.113+0000 7f21bd000640 1 --2- 172.21.15.102:0/593994468 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21b806b140 0x7f21b806b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.113+0000 7f21bd000640 1 -- 172.21.15.102:0/593994468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b80c8690 msgr2=0x7f21b80c8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:50.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.113+0000 7f21bd000640 1 --2- 172.21.15.102:0/593994468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b80c8690 0x7f21b80c8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.113+0000 7f21bd000640 1 -- 172.21.15.102:0/593994468 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21b81594a0 con 0x7f21a0004d00 2024-03-23T03:46:50.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.113+0000 7f21bd000640 1 --2- 172.21.15.102:0/593994468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21a0004d00 0x7f21b8158f60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00e7f0 tx=0x7f21ac00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:50.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.113+0000 7f21b4c00640 1 -- 172.21.15.102:0/593994468 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac00c8e0 con 0x7f21a0004d00 2024-03-23T03:46:50.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.113+0000 7f21b4c00640 1 -- 172.21.15.102:0/593994468 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ac00ca40 con 0x7f21a0004d00 2024-03-23T03:46:50.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.113+0000 7f21b4c00640 1 -- 172.21.15.102:0/593994468 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21ac00cdf0 con 0x7f21a0004d00 2024-03-23T03:46:50.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.113+0000 7f21bfa00640 1 -- 172.21.15.102:0/593994468 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21b80c9210 con 0x7f21a0004d00 2024-03-23T03:46:50.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.114+0000 7f21bfa00640 1 -- 172.21.15.102:0/593994468 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21b815de10 con 0x7f21a0004d00 2024-03-23T03:46:50.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.115+0000 7f21bfa00640 1 -- 172.21.15.102:0/593994468 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21b8073390 con 0x7f21a0004d00 2024-03-23T03:46:50.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.116+0000 7f21b4c00640 1 -- 172.21.15.102:0/593994468 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7f21ac013470 con 0x7f21a0004d00 2024-03-23T03:46:50.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.117+0000 7f21b4c00640 1 --2- 172.21.15.102:0/593994468 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f219806aba0 0x7f219806d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:50.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.117+0000 7f21b4c00640 1 -- 172.21.15.102:0/593994468 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f21ac088fb0 con 0x7f21a0004d00 2024-03-23T03:46:50.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.117+0000 7f21b7e00640 1 --2- 172.21.15.102:0/593994468 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f219806aba0 0x7f219806d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:50.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.118+0000 7f21b7e00640 1 --2- 172.21.15.102:0/593994468 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f219806aba0 0x7f219806d060 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f21a4004770 tx=0x7f21a4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:50.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.122+0000 7f21b4c00640 1 -- 172.21.15.102:0/593994468 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f21ac0bc580 con 0x7f21a0004d00 2024-03-23T03:46:50.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.359+0000 7f21bfa00640 1 -- 172.21.15.102:0/593994468 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21b80b34f0 con 0x7f21a0004d00 2024-03-23T03:46:50.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.360+0000 7f21b4c00640 1 -- 172.21.15.102:0/593994468 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3533 (secure 0 0 0) 0x7f21ac017de0 con 0x7f21a0004d00 2024-03-23T03:46:50.360 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:46:50.360 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/891818172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":891818172},{"type":"v1","addr":"172.21.15.102:6836","nonce":891818172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2404489233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2404489233},{"type":"v1","addr":"172.21.15.102:6837","nonce":2404489233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/1839967745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":1839967745},{"type":"v1","addr":"172.21.15.173:6835","nonce":1839967745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6834/1462637509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1462637509},{"type":"v1","addr":"172.21.15.173:6834","nonce":1462637509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-03-23T03:46:50.361 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 3 2024-03-23T03:46:50.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.362+0000 7f21bfa00640 1 -- 172.21.15.102:0/593994468 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f219806aba0 msgr2=0x7f219806d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:50.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.362+0000 7f21bfa00640 1 --2- 172.21.15.102:0/593994468 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f219806aba0 0x7f219806d060 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f21a4004770 tx=0x7f21a4008040 comp rx=0 tx=0).stop 2024-03-23T03:46:50.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.362+0000 7f21bfa00640 1 -- 172.21.15.102:0/593994468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21a0004d00 msgr2=0x7f21b8158f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:50.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.362+0000 7f21bfa00640 1 --2- 172.21.15.102:0/593994468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21a0004d00 0x7f21b8158f60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f21ac00e7f0 tx=0x7f21ac00ecb0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.362+0000 7f21bfa00640 1 -- 172.21.15.102:0/593994468 shutdown_connections 2024-03-23T03:46:50.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.362+0000 7f21bfa00640 1 --2- 172.21.15.102:0/593994468 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f219806aba0 0x7f219806d060 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.362+0000 7f21bfa00640 1 --2- 172.21.15.102:0/593994468 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21b806b140 0x7f21b806b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.362+0000 7f21bfa00640 1 --2- 172.21.15.102:0/593994468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21b80c8690 0x7f21b80c8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.362+0000 7f21bfa00640 1 --2- 172.21.15.102:0/593994468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21a0004d00 0x7f21b8158f60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.362+0000 7f21bfa00640 1 -- 172.21.15.102:0/593994468 >> 172.21.15.102:0/593994468 conn(0x7f21b80cf350 msgr2=0x7f21b80ce200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:50.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.362+0000 7f21bfa00640 1 -- 172.21.15.102:0/593994468 shutdown_connections 2024-03-23T03:46:50.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.362+0000 7f21bfa00640 1 -- 172.21.15.102:0/593994468 wait complete. 2024-03-23T03:46:50.380 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:46:50.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.570+0000 7f88c8400640 1 Processor -- start 2024-03-23T03:46:50.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.571+0000 7f88c8400640 1 -- start start 2024-03-23T03:46:50.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.572+0000 7f88c8400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f88c015cda0 0x7f88c00cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:50.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.572+0000 7f88c8400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88c00b5a30 0x7f88c00b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:50.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.572+0000 7f88c8400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c00d1830 con 0x7f88c00b4350 2024-03-23T03:46:50.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.572+0000 7f88c8400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c00b1f20 con 0x7f88c015cda0 2024-03-23T03:46:50.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.572+0000 7f88c8400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c00d0c20 con 0x7f88c00b5a30 2024-03-23T03:46:50.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.572+0000 7f88c5a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f88c015cda0 0x7f88c00cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:50.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.572+0000 7f88c6400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f88c00b4350 0x7f88c00d1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40968/0 (socket says 172.21.15.102:40968) 2024-03-23T03:46:50.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.572+0000 7f88c6400640 1 -- 172.21.15.102:0/944804524 learned_addr learned my addr 172.21.15.102:0/944804524 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:46:50.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.572+0000 7f88c5000640 1 --2- 172.21.15.102:0/944804524 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88c00b5a30 0x7f88c00b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:50.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.573+0000 7f88c5a00640 1 -- 172.21.15.102:0/944804524 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88c00b5a30 msgr2=0x7f88c00b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:50.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.573+0000 7f88c5a00640 1 --2- 172.21.15.102:0/944804524 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88c00b5a30 0x7f88c00b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.573+0000 7f88c5a00640 1 -- 172.21.15.102:0/944804524 >> v1:172.21.15.102:6789/0 conn(0x7f88c00b4350 legacy=0x7f88c00d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:50.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.573+0000 7f88c5a00640 1 -- 172.21.15.102:0/944804524 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c00d0f00 con 0x7f88c015cda0 2024-03-23T03:46:50.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.573+0000 7f88c5000640 1 --2- 172.21.15.102:0/944804524 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88c00b5a30 0x7f88c00b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:46:50.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.574+0000 7f88c5a00640 1 --2- 172.21.15.102:0/944804524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88c015cda0 0x7f88c00cdb40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f88b40097a0 tx=0x7f88b402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7290ba2f2fc0d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:50.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.574+0000 7f88bfe00640 1 -- 172.21.15.102:0/944804524 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b402d040 con 0x7f88c015cda0 2024-03-23T03:46:50.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.574+0000 7f88bfe00640 1 -- 172.21.15.102:0/944804524 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b4031810 con 0x7f88c015cda0 2024-03-23T03:46:50.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.574+0000 7f88bfe00640 1 -- 172.21.15.102:0/944804524 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b4031ba0 con 0x7f88c015cda0 2024-03-23T03:46:50.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.575+0000 7f88c8400640 1 -- 172.21.15.102:0/944804524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88c015cda0 msgr2=0x7f88c00cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:50.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.575+0000 7f88c8400640 1 --2- 172.21.15.102:0/944804524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88c015cda0 0x7f88c00cdb40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f88b40097a0 tx=0x7f88b402b8b0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.575+0000 7f88c8400640 1 -- 172.21.15.102:0/944804524 shutdown_connections 2024-03-23T03:46:50.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.575+0000 7f88c8400640 1 --2- 172.21.15.102:0/944804524 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88c00b5a30 0x7f88c00b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.575+0000 7f88c8400640 1 --2- 172.21.15.102:0/944804524 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88c015cda0 0x7f88c00cdb40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.575+0000 7f88c8400640 1 -- 172.21.15.102:0/944804524 >> 172.21.15.102:0/944804524 conn(0x7f88c00cf350 msgr2=0x7f88c00cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:50.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.575+0000 7f88c8400640 1 -- 172.21.15.102:0/944804524 shutdown_connections 2024-03-23T03:46:50.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.575+0000 7f88c8400640 1 -- 172.21.15.102:0/944804524 wait complete. 2024-03-23T03:46:50.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.577+0000 7f88c8400640 1 Processor -- start 2024-03-23T03:46:50.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.577+0000 7f88c8400640 1 -- start start 2024-03-23T03:46:50.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.578+0000 7f88c8400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88c00b4350 0x7f88c01f2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:50.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.578+0000 7f88c8400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88c00b5a30 0x7f88c00c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:50.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.578+0000 7f88c8400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88c015cda0 0x7f88c006b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:50.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.578+0000 7f88c8400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c015d940 con 0x7f88c015cda0 2024-03-23T03:46:50.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.578+0000 7f88c8400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c01f2900 con 0x7f88c00b5a30 2024-03-23T03:46:50.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.578+0000 7f88c8400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c01f2be0 con 0x7f88c00b4350 2024-03-23T03:46:50.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.578+0000 7f88c6400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88c015cda0 0x7f88c006b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:50.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.578+0000 7f88c6400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88c015cda0 0x7f88c006b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52624/0 (socket says 172.21.15.102:52624) 2024-03-23T03:46:50.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.578+0000 7f88c6400640 1 -- 172.21.15.102:0/2699675485 learned_addr learned my addr 172.21.15.102:0/2699675485 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:50.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.578+0000 7f88c5000640 1 --2- 172.21.15.102:0/2699675485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88c00b5a30 0x7f88c00c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:50.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.578+0000 7f88c5a00640 1 --2- 172.21.15.102:0/2699675485 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88c00b4350 0x7f88c01f2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:50.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.578+0000 7f88c6400640 1 -- 172.21.15.102:0/2699675485 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88c00b4350 msgr2=0x7f88c01f2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:50.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.578+0000 7f88c6400640 1 --2- 172.21.15.102:0/2699675485 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88c00b4350 0x7f88c01f2080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.578+0000 7f88c6400640 1 -- 172.21.15.102:0/2699675485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88c00b5a30 msgr2=0x7f88c00c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:50.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.578+0000 7f88c6400640 1 --2- 172.21.15.102:0/2699675485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88c00b5a30 0x7f88c00c79e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.579+0000 7f88c6400640 1 -- 172.21.15.102:0/2699675485 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b4009400 con 0x7f88c015cda0 2024-03-23T03:46:50.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.579+0000 7f88c6400640 1 --2- 172.21.15.102:0/2699675485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88c015cda0 0x7f88c006b140 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f88b800cbc0 tx=0x7f88b800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:50.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.579+0000 7f88bd600640 1 -- 172.21.15.102:0/2699675485 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b800ad40 con 0x7f88c015cda0 2024-03-23T03:46:50.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.579+0000 7f88bd600640 1 -- 172.21.15.102:0/2699675485 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b8012400 con 0x7f88c015cda0 2024-03-23T03:46:50.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.579+0000 7f88c8400640 1 -- 172.21.15.102:0/2699675485 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c01f2ec0 con 0x7f88c015cda0 2024-03-23T03:46:50.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.579+0000 7f88bd600640 1 -- 172.21.15.102:0/2699675485 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b800aea0 con 0x7f88c015cda0 2024-03-23T03:46:50.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.579+0000 7f88c8400640 1 -- 172.21.15.102:0/2699675485 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c01f25c0 con 0x7f88c015cda0 2024-03-23T03:46:50.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.580+0000 7f88c8400640 1 -- 172.21.15.102:0/2699675485 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c01f30b0 con 0x7f88c015cda0 2024-03-23T03:46:50.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.581+0000 7f88bd600640 1 -- 172.21.15.102:0/2699675485 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7f88b8007070 con 0x7f88c015cda0 2024-03-23T03:46:50.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.582+0000 7f88bd600640 1 --2- 172.21.15.102:0/2699675485 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f88a406ab70 0x7f88a406d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:50.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.582+0000 7f88bd600640 1 -- 172.21.15.102:0/2699675485 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f88b8088df0 con 0x7f88c015cda0 2024-03-23T03:46:50.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.583+0000 7f88c5a00640 1 --2- 172.21.15.102:0/2699675485 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f88a406ab70 0x7f88a406d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:50.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.583+0000 7f88c5a00640 1 --2- 172.21.15.102:0/2699675485 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f88a406ab70 0x7f88a406d030 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f88b4036000 tx=0x7f88b403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:50.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.591+0000 7f88bd600640 1 -- 172.21.15.102:0/2699675485 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f88c01f30b0 con 0x7f88c015cda0 2024-03-23T03:46:50.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.789+0000 7f88c8400640 1 -- 172.21.15.102:0/2699675485 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88c01f30b0 con 0x7f88c015cda0 2024-03-23T03:46:50.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.791+0000 7f88bd600640 1 -- 172.21.15.102:0/2699675485 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12430 (secure 0 0 0) 0x7f88c01f30b0 con 0x7f88c015cda0 2024-03-23T03:46:50.791 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:46:50.792 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":18,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:50.617151+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:46:50.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.793+0000 7f88c8400640 1 -- 172.21.15.102:0/2699675485 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f88a406ab70 msgr2=0x7f88a406d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:50.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.793+0000 7f88c8400640 1 --2- 172.21.15.102:0/2699675485 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f88a406ab70 0x7f88a406d030 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f88b4036000 tx=0x7f88b403c040 comp rx=0 tx=0).stop 2024-03-23T03:46:50.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.793+0000 7f88c8400640 1 -- 172.21.15.102:0/2699675485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88c015cda0 msgr2=0x7f88c006b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:50.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.793+0000 7f88c8400640 1 --2- 172.21.15.102:0/2699675485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88c015cda0 0x7f88c006b140 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f88b800cbc0 tx=0x7f88b800a430 comp rx=0 tx=0).stop 2024-03-23T03:46:50.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.794+0000 7f88c8400640 1 -- 172.21.15.102:0/2699675485 shutdown_connections 2024-03-23T03:46:50.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.794+0000 7f88c8400640 1 --2- 172.21.15.102:0/2699675485 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f88a406ab70 0x7f88a406d030 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.794+0000 7f88c8400640 1 --2- 172.21.15.102:0/2699675485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88c015cda0 0x7f88c006b140 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.794+0000 7f88c8400640 1 --2- 172.21.15.102:0/2699675485 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88c00b5a30 0x7f88c00c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.794+0000 7f88c8400640 1 --2- 172.21.15.102:0/2699675485 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88c00b4350 0x7f88c01f2080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:50.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.794+0000 7f88c8400640 1 -- 172.21.15.102:0/2699675485 >> 172.21.15.102:0/2699675485 conn(0x7f88c00cf350 msgr2=0x7f88c00cc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:50.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.794+0000 7f88c8400640 1 -- 172.21.15.102:0/2699675485 shutdown_connections 2024-03-23T03:46:50.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:50.794+0000 7f88c8400640 1 -- 172.21.15.102:0/2699675485 wait complete. 2024-03-23T03:46:50.812 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:46:50.812 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:46:51.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.010+0000 7fcf4b000640 1 Processor -- start 2024-03-23T03:46:51.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.011+0000 7fcf4b000640 1 -- start start 2024-03-23T03:46:51.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.012+0000 7fcf4b000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf4415ad10 0x7fcf440b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.012+0000 7fcf4b000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fcf440ae830 0x7fcf4415d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.012+0000 7fcf4b000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf440b2ad0 con 0x7fcf4415df10 2024-03-23T03:46:51.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.012+0000 7fcf4b000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf4415d5b0 con 0x7fcf440ae830 2024-03-23T03:46:51.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.012+0000 7fcf4b000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf440b2d00 con 0x7fcf4415ad10 2024-03-23T03:46:51.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.013+0000 7fcf43e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf4415ad10 0x7fcf440b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.013+0000 7fcf43e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf4415ad10 0x7fcf440b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:57368/0 (socket says 172.21.15.102:57368) 2024-03-23T03:46:51.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.013+0000 7fcf43e00640 1 -- 172.21.15.102:0/4139461398 learned_addr learned my addr 172.21.15.102:0/4139461398 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:51.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.013+0000 7fcf3be00640 1 --2- 172.21.15.102:0/4139461398 >> v2:172.21.15.173:3300/0 conn(0x7fcf440ae830 0x7fcf4415d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.014+0000 7fcf43e00640 1 -- 172.21.15.102:0/4139461398 >> v2:172.21.15.173:3300/0 conn(0x7fcf440ae830 msgr2=0x7fcf4415d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:51.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.014+0000 7fcf43e00640 1 --2- 172.21.15.102:0/4139461398 >> v2:172.21.15.173:3300/0 conn(0x7fcf440ae830 0x7fcf4415d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.014+0000 7fcf43e00640 1 -- 172.21.15.102:0/4139461398 >> v1:172.21.15.173:6789/0 conn(0x7fcf4415df10 legacy=0x7fcf440b38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:51.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.014+0000 7fcf43e00640 1 -- 172.21.15.102:0/4139461398 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf440b2fe0 con 0x7fcf4415ad10 2024-03-23T03:46:51.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.015+0000 7fcf43e00640 1 --2- 172.21.15.102:0/4139461398 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf4415ad10 0x7fcf440b2590 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcf34009870 tx=0x7fcf3402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a2accdcf735b1f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:51.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.015+0000 7fcf43400640 1 -- 172.21.15.102:0/4139461398 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf34030840 con 0x7fcf4415ad10 2024-03-23T03:46:51.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.015+0000 7fcf43400640 1 -- 172.21.15.102:0/4139461398 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf340309a0 con 0x7fcf4415ad10 2024-03-23T03:46:51.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.015+0000 7fcf43400640 1 -- 172.21.15.102:0/4139461398 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf34030d30 con 0x7fcf4415ad10 2024-03-23T03:46:51.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.016+0000 7fcf4b000640 1 -- 172.21.15.102:0/4139461398 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf4415ad10 msgr2=0x7fcf440b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:51.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.016+0000 7fcf4b000640 1 --2- 172.21.15.102:0/4139461398 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf4415ad10 0x7fcf440b2590 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcf34009870 tx=0x7fcf3402b980 comp rx=0 tx=0).stop 2024-03-23T03:46:51.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.016+0000 7fcf4b000640 1 -- 172.21.15.102:0/4139461398 shutdown_connections 2024-03-23T03:46:51.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.016+0000 7fcf4b000640 1 --2- 172.21.15.102:0/4139461398 >> v2:172.21.15.173:3300/0 conn(0x7fcf440ae830 0x7fcf4415d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.016+0000 7fcf4b000640 1 --2- 172.21.15.102:0/4139461398 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf4415ad10 0x7fcf440b2590 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.016+0000 7fcf4b000640 1 -- 172.21.15.102:0/4139461398 >> 172.21.15.102:0/4139461398 conn(0x7fcf440b4c90 msgr2=0x7fcf440b50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:51.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.016+0000 7fcf4b000640 1 -- 172.21.15.102:0/4139461398 shutdown_connections 2024-03-23T03:46:51.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.016+0000 7fcf4b000640 1 -- 172.21.15.102:0/4139461398 wait complete. 2024-03-23T03:46:51.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.018+0000 7fcf4b000640 1 Processor -- start 2024-03-23T03:46:51.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.019+0000 7fcf4b000640 1 -- start start 2024-03-23T03:46:51.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.019+0000 7fcf4b000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf440ae830 0x7fcf44145d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.019+0000 7fcf4b000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf4415ad10 0x7fcf4415cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.019+0000 7fcf4b000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf4415df10 0x7fcf44166b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.019+0000 7fcf4b000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf44166710 con 0x7fcf4415df10 2024-03-23T03:46:51.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.019+0000 7fcf4b000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf44165a90 con 0x7fcf4415ad10 2024-03-23T03:46:51.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.019+0000 7fcf4b000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf44165d70 con 0x7fcf440ae830 2024-03-23T03:46:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.020+0000 7fcf49000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf4415df10 0x7fcf44166b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.020+0000 7fcf49000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf4415df10 0x7fcf44166b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52632/0 (socket says 172.21.15.102:52632) 2024-03-23T03:46:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.020+0000 7fcf49000640 1 -- 172.21.15.102:0/177976369 learned_addr learned my addr 172.21.15.102:0/177976369 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.020+0000 7fcf43e00640 1 --2- 172.21.15.102:0/177976369 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf440ae830 0x7fcf44145d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.020+0000 7fcf3be00640 1 --2- 172.21.15.102:0/177976369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf4415ad10 0x7fcf4415cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.020+0000 7fcf49000640 1 -- 172.21.15.102:0/177976369 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf440ae830 msgr2=0x7fcf44145d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.020+0000 7fcf49000640 1 --2- 172.21.15.102:0/177976369 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf440ae830 0x7fcf44145d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.020+0000 7fcf49000640 1 -- 172.21.15.102:0/177976369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf4415ad10 msgr2=0x7fcf4415cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.020+0000 7fcf49000640 1 --2- 172.21.15.102:0/177976369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf4415ad10 0x7fcf4415cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.020+0000 7fcf49000640 1 -- 172.21.15.102:0/177976369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf340094d0 con 0x7fcf4415df10 2024-03-23T03:46:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.020+0000 7fcf49000640 1 --2- 172.21.15.102:0/177976369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf4415df10 0x7fcf44166b30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00c6b0 tx=0x7fcf3c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.021+0000 7fcf40c00640 1 -- 172.21.15.102:0/177976369 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c003b70 con 0x7fcf4415df10 2024-03-23T03:46:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.021+0000 7fcf40c00640 1 -- 172.21.15.102:0/177976369 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3c003cf0 con 0x7fcf4415df10 2024-03-23T03:46:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.021+0000 7fcf40c00640 1 -- 172.21.15.102:0/177976369 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c003ea0 con 0x7fcf4415df10 2024-03-23T03:46:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.021+0000 7fcf4b000640 1 -- 172.21.15.102:0/177976369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf44166050 con 0x7fcf4415df10 2024-03-23T03:46:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.021+0000 7fcf4b000640 1 -- 172.21.15.102:0/177976369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf44146240 con 0x7fcf4415df10 2024-03-23T03:46:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.022+0000 7fcf4b000640 1 -- 172.21.15.102:0/177976369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf44166850 con 0x7fcf4415df10 2024-03-23T03:46:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.023+0000 7fcf40c00640 1 -- 172.21.15.102:0/177976369 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7fcf3c005310 con 0x7fcf4415df10 2024-03-23T03:46:51.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.024+0000 7fcf40c00640 1 --2- 172.21.15.102:0/177976369 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcf2006ab70 0x7fcf2006d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.024+0000 7fcf40c00640 1 -- 172.21.15.102:0/177976369 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fcf3c088c20 con 0x7fcf4415df10 2024-03-23T03:46:51.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.024+0000 7fcf43e00640 1 --2- 172.21.15.102:0/177976369 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcf2006ab70 0x7fcf2006d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.025+0000 7fcf43e00640 1 --2- 172.21.15.102:0/177976369 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcf2006ab70 0x7fcf2006d030 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcf34036000 tx=0x7fcf34008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:51.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.029+0000 7fcf40c00640 1 -- 172.21.15.102:0/177976369 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcf44166850 con 0x7fcf4415df10 2024-03-23T03:46:51.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.222+0000 7fcf4b000640 1 -- 172.21.15.102:0/177976369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcf44166850 con 0x7fcf4415df10 2024-03-23T03:46:51.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.223+0000 7fcf40c00640 1 -- 172.21.15.102:0/177976369 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf44166850 con 0x7fcf4415df10 2024-03-23T03:46:51.224 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:46:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.226+0000 7fcf4b000640 1 -- 172.21.15.102:0/177976369 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcf2006ab70 msgr2=0x7fcf2006d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.226+0000 7fcf4b000640 1 --2- 172.21.15.102:0/177976369 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcf2006ab70 0x7fcf2006d030 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcf34036000 tx=0x7fcf34008030 comp rx=0 tx=0).stop 2024-03-23T03:46:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.226+0000 7fcf4b000640 1 -- 172.21.15.102:0/177976369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf4415df10 msgr2=0x7fcf44166b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.226+0000 7fcf4b000640 1 --2- 172.21.15.102:0/177976369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf4415df10 0x7fcf44166b30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c00c6b0 tx=0x7fcf3c00cb70 comp rx=0 tx=0).stop 2024-03-23T03:46:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.226+0000 7fcf4b000640 1 -- 172.21.15.102:0/177976369 shutdown_connections 2024-03-23T03:46:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.226+0000 7fcf4b000640 1 --2- 172.21.15.102:0/177976369 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcf2006ab70 0x7fcf2006d030 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.226+0000 7fcf4b000640 1 --2- 172.21.15.102:0/177976369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf4415df10 0x7fcf44166b30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.226+0000 7fcf4b000640 1 --2- 172.21.15.102:0/177976369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf4415ad10 0x7fcf4415cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.226+0000 7fcf4b000640 1 --2- 172.21.15.102:0/177976369 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf440ae830 0x7fcf44145d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.226+0000 7fcf4b000640 1 -- 172.21.15.102:0/177976369 >> 172.21.15.102:0/177976369 conn(0x7fcf440b4c90 msgr2=0x7fcf44069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:51.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.226+0000 7fcf4b000640 1 -- 172.21.15.102:0/177976369 shutdown_connections 2024-03-23T03:46:51.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.226+0000 7fcf4b000640 1 -- 172.21.15.102:0/177976369 wait complete. 2024-03-23T03:46:51.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T03:46:51.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.439+0000 7fa406c00640 1 Processor -- start 2024-03-23T03:46:51.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.439+0000 7fa406c00640 1 -- start start 2024-03-23T03:46:51.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.441+0000 7fa406c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa40015cda0 0x7fa4000cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.441+0000 7fa406c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa4000b4a60 0x7fa4000b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.441+0000 7fa406c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4000d1830 con 0x7fa4000b5a30 2024-03-23T03:46:51.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.441+0000 7fa406c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4000b1f20 con 0x7fa40015cda0 2024-03-23T03:46:51.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.441+0000 7fa406c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4000d0c20 con 0x7fa4000b4a60 2024-03-23T03:46:51.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.441+0000 7fa3ffe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa40015cda0 0x7fa4000cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.441+0000 7fa3ffe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa40015cda0 0x7fa4000cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52640/0 (socket says 172.21.15.102:52640) 2024-03-23T03:46:51.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.441+0000 7fa3ffe00640 1 -- 172.21.15.102:0/3786970066 learned_addr learned my addr 172.21.15.102:0/3786970066 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:51.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.441+0000 7fa404c00640 1 --2- 172.21.15.102:0/3786970066 >> v2:172.21.15.173:3300/0 conn(0x7fa4000b4a60 0x7fa4000b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.442+0000 7fa3ffe00640 1 -- 172.21.15.102:0/3786970066 >> v2:172.21.15.173:3300/0 conn(0x7fa4000b4a60 msgr2=0x7fa4000b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:51.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.442+0000 7fa3ffe00640 1 --2- 172.21.15.102:0/3786970066 >> v2:172.21.15.173:3300/0 conn(0x7fa4000b4a60 0x7fa4000b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.442+0000 7fa3ffe00640 1 -- 172.21.15.102:0/3786970066 >> v1:172.21.15.173:6789/0 conn(0x7fa4000b5a30 legacy=0x7fa4000b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:51.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.442+0000 7fa3ffe00640 1 -- 172.21.15.102:0/3786970066 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4000d0f00 con 0x7fa40015cda0 2024-03-23T03:46:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.443+0000 7fa3ffe00640 1 --2- 172.21.15.102:0/3786970066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa40015cda0 0x7fa4000cdb40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa4000b8460 tx=0x7fa3f802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=940bb7404a2b5d8c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.443+0000 7fa3fea00640 1 -- 172.21.15.102:0/3786970066 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f802ca80 con 0x7fa40015cda0 2024-03-23T03:46:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.443+0000 7fa3fea00640 1 -- 172.21.15.102:0/3786970066 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f802cbe0 con 0x7fa40015cda0 2024-03-23T03:46:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.443+0000 7fa3fea00640 1 -- 172.21.15.102:0/3786970066 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f80343f0 con 0x7fa40015cda0 2024-03-23T03:46:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.444+0000 7fa406c00640 1 -- 172.21.15.102:0/3786970066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa40015cda0 msgr2=0x7fa4000cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.444+0000 7fa406c00640 1 --2- 172.21.15.102:0/3786970066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa40015cda0 0x7fa4000cdb40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa4000b8460 tx=0x7fa3f802bee0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.444+0000 7fa406c00640 1 -- 172.21.15.102:0/3786970066 shutdown_connections 2024-03-23T03:46:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.444+0000 7fa406c00640 1 --2- 172.21.15.102:0/3786970066 >> v2:172.21.15.173:3300/0 conn(0x7fa4000b4a60 0x7fa4000b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.444+0000 7fa406c00640 1 --2- 172.21.15.102:0/3786970066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa40015cda0 0x7fa4000cdb40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.444+0000 7fa406c00640 1 -- 172.21.15.102:0/3786970066 >> 172.21.15.102:0/3786970066 conn(0x7fa4000cf350 msgr2=0x7fa4000cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:51.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.444+0000 7fa406c00640 1 -- 172.21.15.102:0/3786970066 shutdown_connections 2024-03-23T03:46:51.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.444+0000 7fa406c00640 1 -- 172.21.15.102:0/3786970066 wait complete. 2024-03-23T03:46:51.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.446+0000 7fa406c00640 1 Processor -- start 2024-03-23T03:46:51.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.446+0000 7fa406c00640 1 -- start start 2024-03-23T03:46:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.447+0000 7fa406c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4000b4a60 0x7fa40014a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.447+0000 7fa406c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4000b5a30 0x7fa4000caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.447+0000 7fa406c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa40015cda0 0x7fa4000b7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.447+0000 7fa406c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4000cafa0 con 0x7fa4000b4a60 2024-03-23T03:46:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.447+0000 7fa406c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4000cb270 con 0x7fa4000b5a30 2024-03-23T03:46:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.447+0000 7fa406c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4000cb550 con 0x7fa40015cda0 2024-03-23T03:46:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.447+0000 7fa3ffe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4000b4a60 0x7fa40014a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.447+0000 7fa3ffe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4000b4a60 0x7fa40014a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52654/0 (socket says 172.21.15.102:52654) 2024-03-23T03:46:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.447+0000 7fa3ffe00640 1 -- 172.21.15.102:0/1183731009 learned_addr learned my addr 172.21.15.102:0/1183731009 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.447+0000 7fa404c00640 1 --2- 172.21.15.102:0/1183731009 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa40015cda0 0x7fa4000b7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.447+0000 7fa3ff400640 1 --2- 172.21.15.102:0/1183731009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4000b5a30 0x7fa4000caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.448+0000 7fa3ffe00640 1 -- 172.21.15.102:0/1183731009 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa40015cda0 msgr2=0x7fa4000b7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.448+0000 7fa3ffe00640 1 --2- 172.21.15.102:0/1183731009 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa40015cda0 0x7fa4000b7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.448+0000 7fa3ffe00640 1 -- 172.21.15.102:0/1183731009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4000b5a30 msgr2=0x7fa4000caa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:51.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.448+0000 7fa3ffe00640 1 --2- 172.21.15.102:0/1183731009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4000b5a30 0x7fa4000caa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.448+0000 7fa3ffe00640 1 -- 172.21.15.102:0/1183731009 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f8009b30 con 0x7fa4000b4a60 2024-03-23T03:46:51.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.448+0000 7fa404c00640 1 --2- 172.21.15.102:0/1183731009 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa40015cda0 0x7fa4000b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:46:51.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.448+0000 7fa3ff400640 1 --2- 172.21.15.102:0/1183731009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4000b5a30 0x7fa4000caa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:46:51.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.448+0000 7fa3ffe00640 1 --2- 172.21.15.102:0/1183731009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4000b4a60 0x7fa40014a880 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8000f80 tx=0x7fa3f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:51.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.448+0000 7fa3f7e00640 1 -- 172.21.15.102:0/1183731009 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f8009510 con 0x7fa4000b4a60 2024-03-23T03:46:51.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.448+0000 7fa3f7e00640 1 -- 172.21.15.102:0/1183731009 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f803e450 con 0x7fa4000b4a60 2024-03-23T03:46:51.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.448+0000 7fa406c00640 1 -- 172.21.15.102:0/1183731009 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4000cb830 con 0x7fa4000b4a60 2024-03-23T03:46:51.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.448+0000 7fa3f7e00640 1 -- 172.21.15.102:0/1183731009 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f803e800 con 0x7fa4000b4a60 2024-03-23T03:46:51.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.449+0000 7fa406c00640 1 -- 172.21.15.102:0/1183731009 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4000cbdf0 con 0x7fa4000b4a60 2024-03-23T03:46:51.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.450+0000 7fa406c00640 1 -- 172.21.15.102:0/1183731009 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa400147c70 con 0x7fa4000b4a60 2024-03-23T03:46:51.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.451+0000 7fa3f7e00640 1 -- 172.21.15.102:0/1183731009 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7fa3f802cd50 con 0x7fa4000b4a60 2024-03-23T03:46:51.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.452+0000 7fa3f7e00640 1 --2- 172.21.15.102:0/1183731009 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa3dc06ab70 0x7fa3dc06d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.452+0000 7fa3f7e00640 1 -- 172.21.15.102:0/1183731009 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa3f80abb60 con 0x7fa4000b4a60 2024-03-23T03:46:51.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.452+0000 7fa3ff400640 1 --2- 172.21.15.102:0/1183731009 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa3dc06ab70 0x7fa3dc06d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.453+0000 7fa3ff400640 1 --2- 172.21.15.102:0/1183731009 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa3dc06ab70 0x7fa3dc06d030 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa4000d3620 tx=0x7fa3e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:51.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.456+0000 7fa3f7e00640 1 -- 172.21.15.102:0/1183731009 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa400147c70 con 0x7fa4000b4a60 2024-03-23T03:46:51.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.665+0000 7fa406c00640 1 -- 172.21.15.102:0/1183731009 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa4000cbf80 con 0x7fa4000b4a60 2024-03-23T03:46:51.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.666+0000 7fa3f7e00640 1 -- 172.21.15.102:0/1183731009 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa3f80770d0 con 0x7fa4000b4a60 2024-03-23T03:46:51.667 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:46:51.667 INFO:teuthology.orchestra.run.smithi102.stdout:[ 2024-03-23T03:46:51.667 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:46:51.667 INFO:teuthology.orchestra.run.smithi102.stdout: "poolnum": 1, 2024-03-23T03:46:51.667 INFO:teuthology.orchestra.run.smithi102.stdout: "poolname": ".mgr" 2024-03-23T03:46:51.667 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:46:51.667 INFO:teuthology.orchestra.run.smithi102.stdout:] 2024-03-23T03:46:51.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.669+0000 7fa406c00640 1 -- 172.21.15.102:0/1183731009 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa3dc06ab70 msgr2=0x7fa3dc06d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:51.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.669+0000 7fa406c00640 1 --2- 172.21.15.102:0/1183731009 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa3dc06ab70 0x7fa3dc06d030 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa4000d3620 tx=0x7fa3e8009040 comp rx=0 tx=0).stop 2024-03-23T03:46:51.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.669+0000 7fa406c00640 1 -- 172.21.15.102:0/1183731009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4000b4a60 msgr2=0x7fa40014a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:51.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.669+0000 7fa406c00640 1 --2- 172.21.15.102:0/1183731009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4000b4a60 0x7fa40014a880 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8000f80 tx=0x7fa3f8004a40 comp rx=0 tx=0).stop 2024-03-23T03:46:51.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.669+0000 7fa406c00640 1 -- 172.21.15.102:0/1183731009 shutdown_connections 2024-03-23T03:46:51.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.669+0000 7fa406c00640 1 --2- 172.21.15.102:0/1183731009 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa3dc06ab70 0x7fa3dc06d030 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.669+0000 7fa406c00640 1 --2- 172.21.15.102:0/1183731009 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa40015cda0 0x7fa4000b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.669+0000 7fa406c00640 1 --2- 172.21.15.102:0/1183731009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa4000b5a30 0x7fa4000caa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.669+0000 7fa406c00640 1 --2- 172.21.15.102:0/1183731009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4000b4a60 0x7fa40014a880 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.669+0000 7fa406c00640 1 -- 172.21.15.102:0/1183731009 >> 172.21.15.102:0/1183731009 conn(0x7fa4000cf350 msgr2=0x7fa4000cc350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:51.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.669+0000 7fa406c00640 1 -- 172.21.15.102:0/1183731009 shutdown_connections 2024-03-23T03:46:51.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.669+0000 7fa406c00640 1 -- 172.21.15.102:0/1183731009 wait complete. 2024-03-23T03:46:51.687 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T03:46:51.688 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T03:46:51.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.897+0000 7f6ea4e00640 1 Processor -- start 2024-03-23T03:46:51.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.898+0000 7f6ea4e00640 1 -- start start 2024-03-23T03:46:51.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.899+0000 7f6ea4e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ea015c9e0 0x7f6ea015dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.899+0000 7f6ea4e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6ea00cb420 0x7f6ea0160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.899+0000 7f6ea4e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea0160530 con 0x7f6ea00b12d0 2024-03-23T03:46:51.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.899+0000 7f6ea4e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ea0160af0 con 0x7f6ea00cb420 2024-03-23T03:46:51.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.899+0000 7f6ea4e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea01614c0 con 0x7f6ea015c9e0 2024-03-23T03:46:51.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.899+0000 7f6e9fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6ea00cb420 0x7f6ea0160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.899+0000 7f6e9fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6ea00cb420 0x7f6ea0160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52660/0 (socket says 172.21.15.102:52660) 2024-03-23T03:46:51.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.899+0000 7f6e9fe00640 1 -- 172.21.15.102:0/1680368262 learned_addr learned my addr 172.21.15.102:0/1680368262 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:51.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.899+0000 7f6e9f400640 1 --2- 172.21.15.102:0/1680368262 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ea015c9e0 0x7f6ea015dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.900+0000 7f6e9fe00640 1 -- 172.21.15.102:0/1680368262 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ea015c9e0 msgr2=0x7f6ea015dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:51.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.900+0000 7f6e9fe00640 1 --2- 172.21.15.102:0/1680368262 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ea015c9e0 0x7f6ea015dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.900+0000 7f6e9fe00640 1 -- 172.21.15.102:0/1680368262 >> v1:172.21.15.173:6789/0 conn(0x7f6ea00b12d0 legacy=0x7f6ea00cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:51.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.900+0000 7f6e9fe00640 1 -- 172.21.15.102:0/1680368262 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea015cdc0 con 0x7f6ea00cb420 2024-03-23T03:46:51.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.901+0000 7f6e9fe00640 1 --2- 172.21.15.102:0/1680368262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea00cb420 0x7f6ea0160f80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009e50 tx=0x7f6e9802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ce33ce26217584 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:51.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.902+0000 7f6e9e000640 1 -- 172.21.15.102:0/1680368262 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9802cb00 con 0x7f6ea00cb420 2024-03-23T03:46:51.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.902+0000 7f6e9e000640 1 -- 172.21.15.102:0/1680368262 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9802cc60 con 0x7f6ea00cb420 2024-03-23T03:46:51.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.902+0000 7f6e9e000640 1 -- 172.21.15.102:0/1680368262 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e980343f0 con 0x7f6ea00cb420 2024-03-23T03:46:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.902+0000 7f6ea4e00640 1 -- 172.21.15.102:0/1680368262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea00cb420 msgr2=0x7f6ea0160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.902+0000 7f6ea4e00640 1 --2- 172.21.15.102:0/1680368262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea00cb420 0x7f6ea0160f80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009e50 tx=0x7f6e9802bf60 comp rx=0 tx=0).stop 2024-03-23T03:46:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.902+0000 7f6ea4e00640 1 -- 172.21.15.102:0/1680368262 shutdown_connections 2024-03-23T03:46:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.902+0000 7f6ea4e00640 1 --2- 172.21.15.102:0/1680368262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea00cb420 0x7f6ea0160f80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.902+0000 7f6ea4e00640 1 --2- 172.21.15.102:0/1680368262 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ea015c9e0 0x7f6ea015dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.902+0000 7f6ea4e00640 1 -- 172.21.15.102:0/1680368262 >> 172.21.15.102:0/1680368262 conn(0x7f6ea00b01b0 msgr2=0x7f6ea00b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.902+0000 7f6ea4e00640 1 -- 172.21.15.102:0/1680368262 shutdown_connections 2024-03-23T03:46:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.902+0000 7f6ea4e00640 1 -- 172.21.15.102:0/1680368262 wait complete. 2024-03-23T03:46:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.904+0000 7f6ea4e00640 1 Processor -- start 2024-03-23T03:46:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.904+0000 7f6ea4e00640 1 -- start start 2024-03-23T03:46:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.905+0000 7f6ea4e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ea00b12d0 0x7f6ea00ceda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.905+0000 7f6ea4e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea00cb420 0x7f6ea0158a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.905+0000 7f6ea4e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ea015c9e0 0x7f6ea0147690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.905+0000 7f6ea4e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea00cd110 con 0x7f6ea00cb420 2024-03-23T03:46:51.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.905+0000 7f6ea4e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea00cdfe0 con 0x7f6ea015c9e0 2024-03-23T03:46:51.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.905+0000 7f6ea4e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea00ce2c0 con 0x7f6ea00b12d0 2024-03-23T03:46:51.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.905+0000 7f6e9ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea00cb420 0x7f6ea0158a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.905+0000 7f6e9ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea00cb420 0x7f6ea0158a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52674/0 (socket says 172.21.15.102:52674) 2024-03-23T03:46:51.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.905+0000 7f6e9ea00640 1 -- 172.21.15.102:0/3132673532 learned_addr learned my addr 172.21.15.102:0/3132673532 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:51.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.905+0000 7f6e9f400640 1 --2- 172.21.15.102:0/3132673532 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ea00b12d0 0x7f6ea00ceda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.906+0000 7f6e9fe00640 1 --2- 172.21.15.102:0/3132673532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ea015c9e0 0x7f6ea0147690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.906+0000 7f6e9ea00640 1 -- 172.21.15.102:0/3132673532 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ea00b12d0 msgr2=0x7f6ea00ceda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:51.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.906+0000 7f6e9ea00640 1 --2- 172.21.15.102:0/3132673532 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ea00b12d0 0x7f6ea00ceda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.906+0000 7f6e9ea00640 1 -- 172.21.15.102:0/3132673532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ea015c9e0 msgr2=0x7f6ea0147690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:51.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.906+0000 7f6e9ea00640 1 --2- 172.21.15.102:0/3132673532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ea015c9e0 0x7f6ea0147690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:51.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.906+0000 7f6e9ea00640 1 -- 172.21.15.102:0/3132673532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e98009b30 con 0x7f6ea00cb420 2024-03-23T03:46:51.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.906+0000 7f6e9ea00640 1 --2- 172.21.15.102:0/3132673532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea00cb420 0x7f6ea0158a80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c00c400 tx=0x7f6e8c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:51.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.906+0000 7f6e97400640 1 -- 172.21.15.102:0/3132673532 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c0052d0 con 0x7f6ea00cb420 2024-03-23T03:46:51.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.906+0000 7f6e97400640 1 -- 172.21.15.102:0/3132673532 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e8c005430 con 0x7f6ea00cb420 2024-03-23T03:46:51.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.906+0000 7f6e97400640 1 -- 172.21.15.102:0/3132673532 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c003830 con 0x7f6ea00cb420 2024-03-23T03:46:51.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.906+0000 7f6ea4e00640 1 -- 172.21.15.102:0/3132673532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea00ce5a0 con 0x7f6ea00cb420 2024-03-23T03:46:51.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.907+0000 7f6ea4e00640 1 -- 172.21.15.102:0/3132673532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea0147bd0 con 0x7f6ea00cb420 2024-03-23T03:46:51.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.908+0000 7f6ea4e00640 1 -- 172.21.15.102:0/3132673532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea004e030 con 0x7f6ea00cb420 2024-03-23T03:46:51.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.909+0000 7f6e97400640 1 -- 172.21.15.102:0/3132673532 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 4) v1 ==== 89081+0+0 (secure 0 0 0) 0x7f6e8c003990 con 0x7f6ea00cb420 2024-03-23T03:46:51.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.909+0000 7f6e97400640 1 --2- 172.21.15.102:0/3132673532 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6e8006ab70 0x7f6e8006d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:51.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.910+0000 7f6e97400640 1 -- 172.21.15.102:0/3132673532 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6e8c088330 con 0x7f6ea00cb420 2024-03-23T03:46:51.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.910+0000 7f6e9f400640 1 --2- 172.21.15.102:0/3132673532 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6e8006ab70 0x7f6e8006d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:51.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.910+0000 7f6e9f400640 1 --2- 172.21.15.102:0/3132673532 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6e8006ab70 0x7f6e8006d030 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009b40 tx=0x7f6e90009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:51.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:51.914+0000 7f6e97400640 1 -- 172.21.15.102:0/3132673532 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6e8c053a80 con 0x7f6ea00cb420 2024-03-23T03:46:52.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.109+0000 7f6ea4e00640 1 -- 172.21.15.102:0/3132673532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f6ea00caaf0 con 0x7f6ea00cb420 2024-03-23T03:46:52.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.419+0000 7f6e97400640 1 -- 172.21.15.102:0/3132673532 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6e8c088080 con 0x7f6ea00cb420 2024-03-23T03:46:52.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.420+0000 7f6e97400640 1 -- 172.21.15.102:0/3132673532 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6e8c050f30 con 0x7f6ea00cb420 2024-03-23T03:46:52.421 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' created 2024-03-23T03:46:52.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.426+0000 7f6ea4e00640 1 -- 172.21.15.102:0/3132673532 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6e8006ab70 msgr2=0x7f6e8006d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:52.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.426+0000 7f6ea4e00640 1 --2- 172.21.15.102:0/3132673532 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6e8006ab70 0x7f6e8006d030 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009b40 tx=0x7f6e90009190 comp rx=0 tx=0).stop 2024-03-23T03:46:52.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.426+0000 7f6ea4e00640 1 -- 172.21.15.102:0/3132673532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea00cb420 msgr2=0x7f6ea0158a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:52.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.426+0000 7f6ea4e00640 1 --2- 172.21.15.102:0/3132673532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea00cb420 0x7f6ea0158a80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c00c400 tx=0x7f6e8c00c8c0 comp rx=0 tx=0).stop 2024-03-23T03:46:52.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.427+0000 7f6ea4e00640 1 -- 172.21.15.102:0/3132673532 shutdown_connections 2024-03-23T03:46:52.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.427+0000 7f6ea4e00640 1 --2- 172.21.15.102:0/3132673532 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6e8006ab70 0x7f6e8006d030 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:52.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.427+0000 7f6ea4e00640 1 --2- 172.21.15.102:0/3132673532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ea015c9e0 0x7f6ea0147690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:52.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.427+0000 7f6ea4e00640 1 --2- 172.21.15.102:0/3132673532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ea00cb420 0x7f6ea0158a80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:52.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.427+0000 7f6ea4e00640 1 --2- 172.21.15.102:0/3132673532 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ea00b12d0 0x7f6ea00ceda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:52.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.427+0000 7f6ea4e00640 1 -- 172.21.15.102:0/3132673532 >> 172.21.15.102:0/3132673532 conn(0x7f6ea00b01b0 msgr2=0x7f6ea015e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:52.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.427+0000 7f6ea4e00640 1 -- 172.21.15.102:0/3132673532 shutdown_connections 2024-03-23T03:46:52.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.427+0000 7f6ea4e00640 1 -- 172.21.15.102:0/3132673532 wait complete. 2024-03-23T03:46:52.448 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-23T03:46:52.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.649+0000 7f24d9200640 1 Processor -- start 2024-03-23T03:46:52.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.650+0000 7f24d9200640 1 -- start start 2024-03-23T03:46:52.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.651+0000 7f24d9200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24d40cd830 0x7f24d4160e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:52.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.651+0000 7f24d9200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d41613d0 con 0x7f24d415c820 2024-03-23T03:46:52.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.651+0000 7f24d9200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24d40ccd90 con 0x7f24d40cf7c0 2024-03-23T03:46:52.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.651+0000 7f24d9200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24d40b7fc0 con 0x7f24d40cd830 2024-03-23T03:46:52.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.651+0000 7f24d3400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f24d415c820 0x7f24d40cf0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40974/0 (socket says 172.21.15.102:40974) 2024-03-23T03:46:52.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.651+0000 7f24d3400640 1 -- 172.21.15.102:0/2132698573 learned_addr learned my addr 172.21.15.102:0/2132698573 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:46:52.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.652+0000 7f24d3e00640 1 --2- 172.21.15.102:0/2132698573 >> v2:172.21.15.102:3300/0 conn(0x7f24d40cd830 0x7f24d4160e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:52.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.652+0000 7f24d3e00640 1 -- 172.21.15.102:0/2132698573 >> v1:172.21.15.173:6789/0 conn(0x7f24d40cf7c0 legacy=0x7f24d40d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:52.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.652+0000 7f24d3e00640 1 -- 172.21.15.102:0/2132698573 >> v1:172.21.15.102:6789/0 conn(0x7f24d415c820 legacy=0x7f24d40cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:52.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.652+0000 7f24d3e00640 1 -- 172.21.15.102:0/2132698573 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d40b82a0 con 0x7f24d40cd830 2024-03-23T03:46:52.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.652+0000 7f24d2a00640 1 -- 172.21.15.102:0/2132698573 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188052788 0 0) 0x7f24c4001110 con 0x7f24d415c820 2024-03-23T03:46:52.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.653+0000 7f24d3e00640 1 --2- 172.21.15.102:0/2132698573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24d40cd830 0x7f24d4160e90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f24cc009d90 tx=0x7f24cc02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1551ca0c840924a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:52.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.653+0000 7f24d2a00640 1 -- 172.21.15.102:0/2132698573 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24cc02ccf0 con 0x7f24d40cd830 2024-03-23T03:46:52.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.654+0000 7f24d2a00640 1 -- 172.21.15.102:0/2132698573 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24cc02ce50 con 0x7f24d40cd830 2024-03-23T03:46:52.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.654+0000 7f24d2a00640 1 -- 172.21.15.102:0/2132698573 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24cc034660 con 0x7f24d40cd830 2024-03-23T03:46:52.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.654+0000 7f24d9200640 1 -- 172.21.15.102:0/2132698573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24d40cd830 msgr2=0x7f24d4160e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:52.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.654+0000 7f24d9200640 1 --2- 172.21.15.102:0/2132698573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24d40cd830 0x7f24d4160e90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f24cc009d90 tx=0x7f24cc02bf40 comp rx=0 tx=0).stop 2024-03-23T03:46:52.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.654+0000 7f24d9200640 1 -- 172.21.15.102:0/2132698573 shutdown_connections 2024-03-23T03:46:52.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.654+0000 7f24d9200640 1 --2- 172.21.15.102:0/2132698573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24d40cd830 0x7f24d4160e90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:52.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.654+0000 7f24d9200640 1 -- 172.21.15.102:0/2132698573 >> 172.21.15.102:0/2132698573 conn(0x7f24d406b050 msgr2=0x7f24d406b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:52.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.654+0000 7f24d9200640 1 -- 172.21.15.102:0/2132698573 shutdown_connections 2024-03-23T03:46:52.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.654+0000 7f24d9200640 1 -- 172.21.15.102:0/2132698573 wait complete. 2024-03-23T03:46:52.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.656+0000 7f24d9200640 1 Processor -- start 2024-03-23T03:46:52.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.656+0000 7f24d9200640 1 -- start start 2024-03-23T03:46:52.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.657+0000 7f24d9200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24d40cd830 0x7f24d41544e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:52.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.657+0000 7f24d9200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d40cf7c0 0x7f24d41e8d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:52.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.657+0000 7f24d9200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f24d415c820 0x7f24d41e7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:52.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.657+0000 7f24d9200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d40b3cf0 con 0x7f24d40cd830 2024-03-23T03:46:52.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.657+0000 7f24d9200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d4157960 con 0x7f24d40cf7c0 2024-03-23T03:46:52.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.657+0000 7f24d9200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d41e84c0 con 0x7f24d415c820 2024-03-23T03:46:52.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.657+0000 7f24d3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24d40cd830 0x7f24d41544e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:52.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.657+0000 7f24d3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24d40cd830 0x7f24d41544e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52702/0 (socket says 172.21.15.102:52702) 2024-03-23T03:46:52.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.657+0000 7f24d3400640 1 -- 172.21.15.102:0/182308592 learned_addr learned my addr 172.21.15.102:0/182308592 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:52.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.658+0000 7f24d3e00640 1 --2- 172.21.15.102:0/182308592 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f24d415c820 0x7f24d41e7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:52.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.658+0000 7f24cbe00640 1 --2- 172.21.15.102:0/182308592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d40cf7c0 0x7f24d41e8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:52.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.658+0000 7f24d3400640 1 -- 172.21.15.102:0/182308592 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f24d415c820 msgr2=0x7f24d41e7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:52.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.658+0000 7f24d3400640 1 --2- 172.21.15.102:0/182308592 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f24d415c820 0x7f24d41e7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:52.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.658+0000 7f24d3400640 1 -- 172.21.15.102:0/182308592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d40cf7c0 msgr2=0x7f24d41e8d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:52.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.658+0000 7f24d3400640 1 --2- 172.21.15.102:0/182308592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d40cf7c0 0x7f24d41e8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:52.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.658+0000 7f24d3400640 1 -- 172.21.15.102:0/182308592 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24cc009a70 con 0x7f24d40cd830 2024-03-23T03:46:52.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.658+0000 7f24d3400640 1 --2- 172.21.15.102:0/182308592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24d40cd830 0x7f24d41544e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f24c400c460 tx=0x7f24c400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:52.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.658+0000 7f24cb400640 1 -- 172.21.15.102:0/182308592 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c40074e0 con 0x7f24d40cd830 2024-03-23T03:46:52.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.658+0000 7f24cb400640 1 -- 172.21.15.102:0/182308592 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c4007660 con 0x7f24d40cd830 2024-03-23T03:46:52.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.658+0000 7f24d9200640 1 -- 172.21.15.102:0/182308592 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d41e8220 con 0x7f24d40cd830 2024-03-23T03:46:52.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.659+0000 7f24cb400640 1 -- 172.21.15.102:0/182308592 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c4004a20 con 0x7f24d40cd830 2024-03-23T03:46:52.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.659+0000 7f24d9200640 1 -- 172.21.15.102:0/182308592 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d40b36a0 con 0x7f24d40cd830 2024-03-23T03:46:52.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.660+0000 7f24d9200640 1 -- 172.21.15.102:0/182308592 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d404e030 con 0x7f24d40cd830 2024-03-23T03:46:52.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.661+0000 7f24cb400640 1 -- 172.21.15.102:0/182308592 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f24c4004b80 con 0x7f24d40cd830 2024-03-23T03:46:52.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.662+0000 7f24cb400640 1 --2- 172.21.15.102:0/182308592 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f24b006ac00 0x7f24b006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:52.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.662+0000 7f24cb400640 1 -- 172.21.15.102:0/182308592 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f24c4088580 con 0x7f24d40cd830 2024-03-23T03:46:52.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.662+0000 7f24cbe00640 1 --2- 172.21.15.102:0/182308592 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f24b006ac00 0x7f24b006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:52.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.662+0000 7f24cbe00640 1 --2- 172.21.15.102:0/182308592 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f24b006ac00 0x7f24b006d0c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f24d40b5170 tx=0x7f24bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:52.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.667+0000 7f24cb400640 1 -- 172.21.15.102:0/182308592 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f24c40538a0 con 0x7f24d40cd830 2024-03-23T03:46:52.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:52.864+0000 7f24d9200640 1 -- 172.21.15.102:0/182308592 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f24d41613d0 con 0x7f24d40cd830 2024-03-23T03:46:53.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.437+0000 7f24cb400640 1 -- 172.21.15.102:0/182308592 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v20) v1 ==== 171+0+0 (secure 0 0 0) 0x7f24d41613d0 con 0x7f24d40cd830 2024-03-23T03:46:53.442 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' created 2024-03-23T03:46:53.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.448+0000 7f24d9200640 1 -- 172.21.15.102:0/182308592 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f24b006ac00 msgr2=0x7f24b006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:53.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.448+0000 7f24d9200640 1 --2- 172.21.15.102:0/182308592 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f24b006ac00 0x7f24b006d0c0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f24d40b5170 tx=0x7f24bc009040 comp rx=0 tx=0).stop 2024-03-23T03:46:53.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.448+0000 7f24d9200640 1 -- 172.21.15.102:0/182308592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24d40cd830 msgr2=0x7f24d41544e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:53.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.448+0000 7f24d9200640 1 --2- 172.21.15.102:0/182308592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24d40cd830 0x7f24d41544e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f24c400c460 tx=0x7f24c400c920 comp rx=0 tx=0).stop 2024-03-23T03:46:53.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.448+0000 7f24d9200640 1 -- 172.21.15.102:0/182308592 shutdown_connections 2024-03-23T03:46:53.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.448+0000 7f24d9200640 1 --2- 172.21.15.102:0/182308592 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f24b006ac00 0x7f24b006d0c0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:53.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.448+0000 7f24d9200640 1 --2- 172.21.15.102:0/182308592 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f24d415c820 0x7f24d41e7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:53.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.448+0000 7f24d9200640 1 --2- 172.21.15.102:0/182308592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24d40cf7c0 0x7f24d41e8d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:53.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.448+0000 7f24d9200640 1 --2- 172.21.15.102:0/182308592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24d40cd830 0x7f24d41544e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:53.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.448+0000 7f24d9200640 1 -- 172.21.15.102:0/182308592 >> 172.21.15.102:0/182308592 conn(0x7f24d406b050 msgr2=0x7f24d40d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:53.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.449+0000 7f24d9200640 1 -- 172.21.15.102:0/182308592 shutdown_connections 2024-03-23T03:46:53.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.450+0000 7f24d9200640 1 -- 172.21.15.102:0/182308592 wait complete. 2024-03-23T03:46:53.479 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T03:46:53.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.660+0000 7f0a9f200640 1 Processor -- start 2024-03-23T03:46:53.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.660+0000 7f0a9f200640 1 -- start start 2024-03-23T03:46:53.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.660+0000 7f0a9f200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0aa015a850 0x7f0aa00cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:53.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.660+0000 7f0a9f200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f0aa00b22b0 0x7f0aa015eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:53.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.660+0000 7f0a9f200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aa015e320 con 0x7f0aa015f020 2024-03-23T03:46:53.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.660+0000 7f0a9f200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa015ac30 con 0x7f0aa015a850 2024-03-23T03:46:53.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.660+0000 7f0a9f200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa00cda20 con 0x7f0aa00b22b0 2024-03-23T03:46:53.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.661+0000 7f0a9de00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0aa015a850 0x7f0aa00cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:53.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.661+0000 7f0a9de00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0aa015a850 0x7f0aa00cd1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50664/0 (socket says 172.21.15.102:50664) 2024-03-23T03:46:53.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.661+0000 7f0a9de00640 1 -- 172.21.15.102:0/3744557412 learned_addr learned my addr 172.21.15.102:0/3744557412 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:53.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.661+0000 7f0a9d400640 1 --2- 172.21.15.102:0/3744557412 >> v2:172.21.15.173:3300/0 conn(0x7f0aa00b22b0 0x7f0aa015eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:53.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.661+0000 7f0a9de00640 1 -- 172.21.15.102:0/3744557412 >> v2:172.21.15.173:3300/0 conn(0x7f0aa00b22b0 msgr2=0x7f0aa015eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:53.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.661+0000 7f0a9de00640 1 --2- 172.21.15.102:0/3744557412 >> v2:172.21.15.173:3300/0 conn(0x7f0aa00b22b0 0x7f0aa015eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:53.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.661+0000 7f0a9de00640 1 -- 172.21.15.102:0/3744557412 >> v1:172.21.15.173:6789/0 conn(0x7f0aa015f020 legacy=0x7f0aa00cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:53.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.661+0000 7f0a9de00640 1 -- 172.21.15.102:0/3744557412 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa00cdd00 con 0x7f0aa015a850 2024-03-23T03:46:53.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.662+0000 7f0a9de00640 1 --2- 172.21.15.102:0/3744557412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa015a850 0x7f0aa00cd1d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0aa006acc0 tx=0x7f0a9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9077ee1e21868e67 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:53.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.663+0000 7f0a9ca00640 1 -- 172.21.15.102:0/3744557412 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a90031870 con 0x7f0aa015a850 2024-03-23T03:46:53.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.663+0000 7f0a9ca00640 1 -- 172.21.15.102:0/3744557412 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a900319d0 con 0x7f0aa015a850 2024-03-23T03:46:53.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.663+0000 7f0a9ca00640 1 -- 172.21.15.102:0/3744557412 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a90031d60 con 0x7f0aa015a850 2024-03-23T03:46:53.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.663+0000 7f0a9f200640 1 -- 172.21.15.102:0/3744557412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa015a850 msgr2=0x7f0aa00cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:53.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.663+0000 7f0a9f200640 1 --2- 172.21.15.102:0/3744557412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa015a850 0x7f0aa00cd1d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0aa006acc0 tx=0x7f0a9002b910 comp rx=0 tx=0).stop 2024-03-23T03:46:53.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.663+0000 7f0a9f200640 1 -- 172.21.15.102:0/3744557412 shutdown_connections 2024-03-23T03:46:53.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.663+0000 7f0a9f200640 1 --2- 172.21.15.102:0/3744557412 >> v2:172.21.15.173:3300/0 conn(0x7f0aa00b22b0 0x7f0aa015eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.663+0000 7f0a9f200640 1 --2- 172.21.15.102:0/3744557412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa015a850 0x7f0aa00cd1d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.663+0000 7f0a9f200640 1 -- 172.21.15.102:0/3744557412 >> 172.21.15.102:0/3744557412 conn(0x7f0aa00b01b0 msgr2=0x7f0aa00b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.663+0000 7f0a9f200640 1 -- 172.21.15.102:0/3744557412 shutdown_connections 2024-03-23T03:46:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.663+0000 7f0a9f200640 1 -- 172.21.15.102:0/3744557412 wait complete. 2024-03-23T03:46:53.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.666+0000 7f0a9f200640 1 Processor -- start 2024-03-23T03:46:53.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.666+0000 7f0a9f200640 1 -- start start 2024-03-23T03:46:53.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.666+0000 7f0a9f200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0aa00b22b0 0x7f0aa0151930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:53.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.666+0000 7f0a9f200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0aa015a850 0x7f0aa015b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:53.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.666+0000 7f0a9f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa015f020 0x7f0aa0158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:53.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.666+0000 7f0a9f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa0151e70 con 0x7f0aa015f020 2024-03-23T03:46:53.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.666+0000 7f0a9f200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa015c060 con 0x7f0aa015a850 2024-03-23T03:46:53.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.666+0000 7f0a9f200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa015c340 con 0x7f0aa00b22b0 2024-03-23T03:46:53.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.666+0000 7f0a9e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa015f020 0x7f0aa0158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:53.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.666+0000 7f0a9e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa015f020 0x7f0aa0158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50670/0 (socket says 172.21.15.102:50670) 2024-03-23T03:46:53.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.666+0000 7f0a9e800640 1 -- 172.21.15.102:0/1513537706 learned_addr learned my addr 172.21.15.102:0/1513537706 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:53.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.667+0000 7f0a9de00640 1 --2- 172.21.15.102:0/1513537706 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0aa00b22b0 0x7f0aa0151930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:53.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.667+0000 7f0a9d400640 1 --2- 172.21.15.102:0/1513537706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0aa015a850 0x7f0aa015b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:53.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.667+0000 7f0a9e800640 1 -- 172.21.15.102:0/1513537706 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0aa00b22b0 msgr2=0x7f0aa0151930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:53.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.667+0000 7f0a9e800640 1 --2- 172.21.15.102:0/1513537706 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0aa00b22b0 0x7f0aa0151930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:53.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.667+0000 7f0a9e800640 1 -- 172.21.15.102:0/1513537706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0aa015a850 msgr2=0x7f0aa015b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:53.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.667+0000 7f0a9e800640 1 --2- 172.21.15.102:0/1513537706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0aa015a850 0x7f0aa015b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:53.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.667+0000 7f0a9e800640 1 -- 172.21.15.102:0/1513537706 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a900094d0 con 0x7f0aa015f020 2024-03-23T03:46:53.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.667+0000 7f0a9de00640 1 --2- 172.21.15.102:0/1513537706 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0aa00b22b0 0x7f0aa0151930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:46:53.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.667+0000 7f0a9d400640 1 --2- 172.21.15.102:0/1513537706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0aa015a850 0x7f0aa015b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:46:53.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.667+0000 7f0a9e800640 1 --2- 172.21.15.102:0/1513537706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa015f020 0x7f0aa0158ac0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0a9800b970 tx=0x7f0a9800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:53.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.667+0000 7f0a96000640 1 -- 172.21.15.102:0/1513537706 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9800cb40 con 0x7f0aa015f020 2024-03-23T03:46:53.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.667+0000 7f0a96000640 1 -- 172.21.15.102:0/1513537706 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9800cca0 con 0x7f0aa015f020 2024-03-23T03:46:53.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.667+0000 7f0a96000640 1 -- 172.21.15.102:0/1513537706 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a98014680 con 0x7f0aa015f020 2024-03-23T03:46:53.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.667+0000 7f0a9f200640 1 -- 172.21.15.102:0/1513537706 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa015c620 con 0x7f0aa015f020 2024-03-23T03:46:53.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.668+0000 7f0a9f200640 1 -- 172.21.15.102:0/1513537706 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa015cbe0 con 0x7f0aa015f020 2024-03-23T03:46:53.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.669+0000 7f0a9f200640 1 -- 172.21.15.102:0/1513537706 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa015ce00 con 0x7f0aa015f020 2024-03-23T03:46:53.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.670+0000 7f0a96000640 1 -- 172.21.15.102:0/1513537706 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0a9801d3b0 con 0x7f0aa015f020 2024-03-23T03:46:53.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.671+0000 7f0a96000640 1 --2- 172.21.15.102:0/1513537706 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0a8006ac00 0x7f0a8006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:53.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.671+0000 7f0a96000640 1 -- 172.21.15.102:0/1513537706 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f0a980891a0 con 0x7f0aa015f020 2024-03-23T03:46:53.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.671+0000 7f0a9de00640 1 --2- 172.21.15.102:0/1513537706 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0a8006ac00 0x7f0a8006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:53.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.671+0000 7f0a9de00640 1 --2- 172.21.15.102:0/1513537706 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0a8006ac00 0x7f0a8006d0c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0a90036000 tx=0x7f0a9003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:53.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.676+0000 7f0a96000640 1 -- 172.21.15.102:0/1513537706 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0a980543d0 con 0x7f0aa015f020 2024-03-23T03:46:53.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.910+0000 7f0a9f200640 1 -- 172.21.15.102:0/1513537706 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0aa004e030 con 0x7f0aa015f020 2024-03-23T03:46:53.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.919+0000 7f0a96000640 1 -- 172.21.15.102:0/1513537706 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f0a980579f0 con 0x7f0aa015f020 2024-03-23T03:46:53.920 INFO:teuthology.orchestra.run.smithi102.stderr:new fs with metadata pool 2 and data pool 3 2024-03-23T03:46:53.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.923+0000 7f0a9f200640 1 -- 172.21.15.102:0/1513537706 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0a8006ac00 msgr2=0x7f0a8006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:53.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.923+0000 7f0a9f200640 1 --2- 172.21.15.102:0/1513537706 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0a8006ac00 0x7f0a8006d0c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0a90036000 tx=0x7f0a9003c040 comp rx=0 tx=0).stop 2024-03-23T03:46:53.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.923+0000 7f0a9f200640 1 -- 172.21.15.102:0/1513537706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa015f020 msgr2=0x7f0aa0158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:53.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.923+0000 7f0a9f200640 1 --2- 172.21.15.102:0/1513537706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa015f020 0x7f0aa0158ac0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0a9800b970 tx=0x7f0a9800be30 comp rx=0 tx=0).stop 2024-03-23T03:46:53.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.923+0000 7f0a9f200640 1 -- 172.21.15.102:0/1513537706 shutdown_connections 2024-03-23T03:46:53.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.923+0000 7f0a9f200640 1 --2- 172.21.15.102:0/1513537706 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0a8006ac00 0x7f0a8006d0c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:53.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.923+0000 7f0a9f200640 1 --2- 172.21.15.102:0/1513537706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa015f020 0x7f0aa0158ac0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:53.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.923+0000 7f0a9f200640 1 --2- 172.21.15.102:0/1513537706 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0aa015a850 0x7f0aa015b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:53.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.923+0000 7f0a9f200640 1 --2- 172.21.15.102:0/1513537706 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0aa00b22b0 0x7f0aa0151930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:53.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.923+0000 7f0a9f200640 1 -- 172.21.15.102:0/1513537706 >> 172.21.15.102:0/1513537706 conn(0x7f0aa00b01b0 msgr2=0x7f0aa00aef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:53.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.923+0000 7f0a9f200640 1 -- 172.21.15.102:0/1513537706 shutdown_connections 2024-03-23T03:46:53.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:53.924+0000 7f0a9f200640 1 -- 172.21.15.102:0/1513537706 wait complete. 2024-03-23T03:46:53.957 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-23T03:46:53.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-23T03:46:54.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.139+0000 7f97a2000640 1 Processor -- start 2024-03-23T03:46:54.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.139+0000 7f97a2000640 1 -- start start 2024-03-23T03:46:54.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.140+0000 7f97a2000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f979c15a850 0x7f979c0cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:54.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.140+0000 7f97a2000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f979c0b22b0 0x7f979c15eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:54.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.140+0000 7f97a2000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f979c15e320 con 0x7f979c15f020 2024-03-23T03:46:54.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.140+0000 7f97a2000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f979c15ac30 con 0x7f979c0b22b0 2024-03-23T03:46:54.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.140+0000 7f97a2000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f979c0cda20 con 0x7f979c15a850 2024-03-23T03:46:54.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.141+0000 7f97a1600640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f979c15f020 0x7f979c0cacf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38072/0 (socket says 172.21.15.102:38072) 2024-03-23T03:46:54.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.141+0000 7f97a1600640 1 -- 172.21.15.102:0/3835061326 learned_addr learned my addr 172.21.15.102:0/3835061326 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:46:54.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.141+0000 7f9793e00640 1 --2- 172.21.15.102:0/3835061326 >> v2:172.21.15.102:3300/0 conn(0x7f979c0b22b0 0x7f979c15eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:54.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.141+0000 7f97a0c00640 1 --2- 172.21.15.102:0/3835061326 >> v2:172.21.15.173:3300/0 conn(0x7f979c15a850 0x7f979c0cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:54.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.141+0000 7f979be00640 1 -- 172.21.15.102:0/3835061326 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594964531 0 0) 0x7f979c15e320 con 0x7f979c15f020 2024-03-23T03:46:54.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.141+0000 7f979be00640 1 -- 172.21.15.102:0/3835061326 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9780003520 con 0x7f979c15f020 2024-03-23T03:46:54.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.141+0000 7f9793e00640 1 -- 172.21.15.102:0/3835061326 >> v2:172.21.15.173:3300/0 conn(0x7f979c15a850 msgr2=0x7f979c0cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:54.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.141+0000 7f9793e00640 1 --2- 172.21.15.102:0/3835061326 >> v2:172.21.15.173:3300/0 conn(0x7f979c15a850 0x7f979c0cd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:54.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.141+0000 7f9793e00640 1 -- 172.21.15.102:0/3835061326 >> v1:172.21.15.102:6789/0 conn(0x7f979c15f020 legacy=0x7f979c0cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:54.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.141+0000 7f9793e00640 1 -- 172.21.15.102:0/3835061326 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c0cdd00 con 0x7f979c0b22b0 2024-03-23T03:46:54.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.143+0000 7f9793e00640 1 --2- 172.21.15.102:0/3835061326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f979c0b22b0 0x7f979c15eae0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f979c06adb0 tx=0x7f978402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fac072aef66dea2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:54.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.143+0000 7f979be00640 1 -- 172.21.15.102:0/3835061326 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784031830 con 0x7f979c0b22b0 2024-03-23T03:46:54.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.143+0000 7f979be00640 1 -- 172.21.15.102:0/3835061326 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9784031990 con 0x7f979c0b22b0 2024-03-23T03:46:54.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.143+0000 7f979be00640 1 -- 172.21.15.102:0/3835061326 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9784031d20 con 0x7f979c0b22b0 2024-03-23T03:46:54.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.144+0000 7f97a2000640 1 -- 172.21.15.102:0/3835061326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f979c0b22b0 msgr2=0x7f979c15eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:54.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.144+0000 7f97a2000640 1 --2- 172.21.15.102:0/3835061326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f979c0b22b0 0x7f979c15eae0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f979c06adb0 tx=0x7f978402b8d0 comp rx=0 tx=0).stop 2024-03-23T03:46:54.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.144+0000 7f97a2000640 1 -- 172.21.15.102:0/3835061326 shutdown_connections 2024-03-23T03:46:54.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.144+0000 7f97a2000640 1 --2- 172.21.15.102:0/3835061326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f979c0b22b0 0x7f979c15eae0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:54.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.144+0000 7f97a2000640 1 --2- 172.21.15.102:0/3835061326 >> v2:172.21.15.173:3300/0 conn(0x7f979c15a850 0x7f979c0cd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:54.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.144+0000 7f97a2000640 1 -- 172.21.15.102:0/3835061326 >> 172.21.15.102:0/3835061326 conn(0x7f979c0b01b0 msgr2=0x7f979c0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:54.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.144+0000 7f97a2000640 1 -- 172.21.15.102:0/3835061326 shutdown_connections 2024-03-23T03:46:54.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.144+0000 7f97a2000640 1 -- 172.21.15.102:0/3835061326 wait complete. 2024-03-23T03:46:54.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.146+0000 7f97a2000640 1 Processor -- start 2024-03-23T03:46:54.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.146+0000 7f97a2000640 1 -- start start 2024-03-23T03:46:54.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.147+0000 7f97a2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f979c0b22b0 0x7f979c1f19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:54.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.147+0000 7f97a2000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f979c15a850 0x7f979c0c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:54.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.147+0000 7f97a2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f979c15f020 0x7f979c0c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:54.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.147+0000 7f97a2000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979c159a30 con 0x7f979c15f020 2024-03-23T03:46:54.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.147+0000 7f97a2000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979c159ec0 con 0x7f979c0b22b0 2024-03-23T03:46:54.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.147+0000 7f97a2000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f979c15bc40 con 0x7f979c15a850 2024-03-23T03:46:54.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.147+0000 7f97a1600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f979c15f020 0x7f979c0c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:54.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.147+0000 7f97a1600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f979c15f020 0x7f979c0c7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50688/0 (socket says 172.21.15.102:50688) 2024-03-23T03:46:54.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.147+0000 7f97a1600640 1 -- 172.21.15.102:0/1820035504 learned_addr learned my addr 172.21.15.102:0/1820035504 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:54.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.147+0000 7f97a0c00640 1 --2- 172.21.15.102:0/1820035504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f979c0b22b0 0x7f979c1f19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:54.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.147+0000 7f9793e00640 1 --2- 172.21.15.102:0/1820035504 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f979c15a850 0x7f979c0c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:54.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.147+0000 7f97a1600640 1 -- 172.21.15.102:0/1820035504 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f979c15a850 msgr2=0x7f979c0c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:54.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.148+0000 7f97a1600640 1 --2- 172.21.15.102:0/1820035504 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f979c15a850 0x7f979c0c7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:54.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.148+0000 7f97a1600640 1 -- 172.21.15.102:0/1820035504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f979c0b22b0 msgr2=0x7f979c1f19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:54.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.148+0000 7f97a1600640 1 --2- 172.21.15.102:0/1820035504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f979c0b22b0 0x7f979c1f19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:54.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.148+0000 7f97a1600640 1 -- 172.21.15.102:0/1820035504 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9784009400 con 0x7f979c15f020 2024-03-23T03:46:54.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.148+0000 7f97a0c00640 1 --2- 172.21.15.102:0/1820035504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f979c0b22b0 0x7f979c1f19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:46:54.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.148+0000 7f97a1600640 1 --2- 172.21.15.102:0/1820035504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f979c15f020 0x7f979c0c7b90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f979400caa0 tx=0x7f979400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:54.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.148+0000 7f9799600640 1 -- 172.21.15.102:0/1820035504 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979400ac00 con 0x7f979c15f020 2024-03-23T03:46:54.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.148+0000 7f9799600640 1 -- 172.21.15.102:0/1820035504 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979400ad60 con 0x7f979c15f020 2024-03-23T03:46:54.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.148+0000 7f97a2000640 1 -- 172.21.15.102:0/1820035504 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f979c159100 con 0x7f979c15f020 2024-03-23T03:46:54.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.148+0000 7f9799600640 1 -- 172.21.15.102:0/1820035504 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9794012620 con 0x7f979c15f020 2024-03-23T03:46:54.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.148+0000 7f97a2000640 1 -- 172.21.15.102:0/1820035504 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f979c1596c0 con 0x7f979c15f020 2024-03-23T03:46:54.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.150+0000 7f97a2000640 1 -- 172.21.15.102:0/1820035504 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f979c159ba0 con 0x7f979c15f020 2024-03-23T03:46:54.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.150+0000 7f9799600640 1 -- 172.21.15.102:0/1820035504 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9794007070 con 0x7f979c15f020 2024-03-23T03:46:54.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.151+0000 7f9799600640 1 --2- 172.21.15.102:0/1820035504 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f977806ac30 0x7f977806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:54.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.151+0000 7f9799600640 1 -- 172.21.15.102:0/1820035504 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f97940892d0 con 0x7f979c15f020 2024-03-23T03:46:54.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.151+0000 7f97a0c00640 1 --2- 172.21.15.102:0/1820035504 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f977806ac30 0x7f977806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:54.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.152+0000 7f97a0c00640 1 --2- 172.21.15.102:0/1820035504 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f977806ac30 0x7f977806d0f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f978c006fd0 tx=0x7f978c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:54.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.156+0000 7f9799600640 1 -- 172.21.15.102:0/1820035504 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f97940544b0 con 0x7f979c15f020 2024-03-23T03:46:54.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.356+0000 7f97a2000640 1 -- 172.21.15.102:0/1820035504 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f979c04e030 con 0x7f979c15f020 2024-03-23T03:46:54.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.926+0000 7f9799600640 1 -- 172.21.15.102:0/1820035504 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v22) v1 ==== 157+0+0 (secure 0 0 0) 0x7f9794057ad0 con 0x7f979c15f020 2024-03-23T03:46:54.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.930+0000 7f97a2000640 1 -- 172.21.15.102:0/1820035504 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f977806ac30 msgr2=0x7f977806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:54.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.930+0000 7f97a2000640 1 --2- 172.21.15.102:0/1820035504 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f977806ac30 0x7f977806d0f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f978c006fd0 tx=0x7f978c008040 comp rx=0 tx=0).stop 2024-03-23T03:46:54.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.930+0000 7f97a2000640 1 -- 172.21.15.102:0/1820035504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f979c15f020 msgr2=0x7f979c0c7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:54.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.930+0000 7f97a2000640 1 --2- 172.21.15.102:0/1820035504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f979c15f020 0x7f979c0c7b90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f979400caa0 tx=0x7f979400cf60 comp rx=0 tx=0).stop 2024-03-23T03:46:54.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.930+0000 7f97a2000640 1 -- 172.21.15.102:0/1820035504 shutdown_connections 2024-03-23T03:46:54.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.930+0000 7f97a2000640 1 --2- 172.21.15.102:0/1820035504 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f977806ac30 0x7f977806d0f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:54.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.930+0000 7f97a2000640 1 --2- 172.21.15.102:0/1820035504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f979c15f020 0x7f979c0c7b90 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:54.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.930+0000 7f97a2000640 1 --2- 172.21.15.102:0/1820035504 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f979c15a850 0x7f979c0c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:54.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.930+0000 7f97a2000640 1 --2- 172.21.15.102:0/1820035504 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f979c0b22b0 0x7f979c1f19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:54.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.930+0000 7f97a2000640 1 -- 172.21.15.102:0/1820035504 >> 172.21.15.102:0/1820035504 conn(0x7f979c0b01b0 msgr2=0x7f979c158190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:54.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.930+0000 7f97a2000640 1 -- 172.21.15.102:0/1820035504 shutdown_connections 2024-03-23T03:46:54.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:54.930+0000 7f97a2000640 1 -- 172.21.15.102:0/1820035504 wait complete. 2024-03-23T03:46:54.949 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T03:46:55.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.171+0000 7f2fc9800640 1 Processor -- start 2024-03-23T03:46:55.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.171+0000 7f2fc9800640 1 -- start start 2024-03-23T03:46:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.173+0000 7f2fc9800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fc415c890 0x7f2fc40b7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.173+0000 7f2fc9800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f2fc40b84b0 0x7f2fc40b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.173+0000 7f2fc9800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc415cc70 con 0x7f2fc40b46f0 2024-03-23T03:46:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.173+0000 7f2fc9800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc40b4b80 con 0x7f2fc40b84b0 2024-03-23T03:46:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.173+0000 7f2fc9800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc40b7b40 con 0x7f2fc415c890 2024-03-23T03:46:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.173+0000 7f2fc8e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2fc40b46f0 0x7f2fc4160380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38084/0 (socket says 172.21.15.102:38084) 2024-03-23T03:46:55.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.173+0000 7f2fc8e00640 1 -- 172.21.15.102:0/3629564802 learned_addr learned my addr 172.21.15.102:0/3629564802 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:46:55.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc2a00640 1 -- 172.21.15.102:0/3629564802 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648131445 0 0) 0x7f2fc415cc70 con 0x7f2fc40b46f0 2024-03-23T03:46:55.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc3e00640 1 --2- 172.21.15.102:0/3629564802 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fc415c890 0x7f2fc40b7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:55.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc3400640 1 --2- 172.21.15.102:0/3629564802 >> v2:172.21.15.173:3300/0 conn(0x7f2fc40b84b0 0x7f2fc40b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:55.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc2a00640 1 -- 172.21.15.102:0/3629564802 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa8003520 con 0x7f2fc40b46f0 2024-03-23T03:46:55.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc2a00640 1 -- 172.21.15.102:0/3629564802 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 252999761 0 0) 0x7f2fa8003520 con 0x7f2fc40b46f0 2024-03-23T03:46:55.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc2a00640 1 -- 172.21.15.102:0/3629564802 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fc415c890 msgr2=0x7f2fc40b7f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:55.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc2a00640 1 --2- 172.21.15.102:0/3629564802 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fc415c890 0x7f2fc40b7f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:55.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc2a00640 1 -- 172.21.15.102:0/3629564802 >> v2:172.21.15.173:3300/0 conn(0x7f2fc40b84b0 msgr2=0x7f2fc40b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:55.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc2a00640 1 --2- 172.21.15.102:0/3629564802 >> v2:172.21.15.173:3300/0 conn(0x7f2fc40b84b0 0x7f2fc40b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:55.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc2a00640 1 -- 172.21.15.102:0/3629564802 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc40b4f90 con 0x7f2fc40b46f0 2024-03-23T03:46:55.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc2a00640 1 -- 172.21.15.102:0/3629564802 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f2fbc0019f0 con 0x7f2fc40b46f0 2024-03-23T03:46:55.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc2a00640 1 -- 172.21.15.102:0/3629564802 >> v1:172.21.15.102:6789/0 conn(0x7f2fc40b46f0 legacy=0x7f2fc4160380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:55.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc2a00640 1 --2- 172.21.15.102:0/3629564802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa8003970 0x7f2fa8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:55.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc2a00640 1 --2- 172.21.15.102:0/3629564802 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fa80047c0 0x7f2fa8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:55.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc2a00640 1 --2- 172.21.15.102:0/3629564802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fa8005180 0x7f2fa800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:55.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc2a00640 1 -- 172.21.15.102:0/3629564802 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc40b4b80 con 0x7f2fa8003970 2024-03-23T03:46:55.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc2a00640 1 -- 172.21.15.102:0/3629564802 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc40b7b40 con 0x7f2fa8005180 2024-03-23T03:46:55.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.174+0000 7f2fc2a00640 1 -- 172.21.15.102:0/3629564802 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa800bef0 con 0x7f2fa80047c0 2024-03-23T03:46:55.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.175+0000 7f2fc3e00640 1 --2- 172.21.15.102:0/3629564802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa8003970 0x7f2fa8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:55.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.175+0000 7f2fc3400640 1 --2- 172.21.15.102:0/3629564802 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fa80047c0 0x7f2fa8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:55.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.175+0000 7f2fc8e00640 1 --2- 172.21.15.102:0/3629564802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fa8005180 0x7f2fa800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:55.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.175+0000 7f2fc3e00640 1 -- 172.21.15.102:0/3629564802 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fa80047c0 msgr2=0x7f2fa8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:55.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.175+0000 7f2fc3e00640 1 --2- 172.21.15.102:0/3629564802 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fa80047c0 0x7f2fa8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:55.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.175+0000 7f2fc3e00640 1 -- 172.21.15.102:0/3629564802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fa8005180 msgr2=0x7f2fa800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:55.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.175+0000 7f2fc3e00640 1 --2- 172.21.15.102:0/3629564802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fa8005180 0x7f2fa800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:55.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.175+0000 7f2fc3e00640 1 -- 172.21.15.102:0/3629564802 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa8003520 con 0x7f2fa8003970 2024-03-23T03:46:55.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.176+0000 7f2fc3e00640 1 --2- 172.21.15.102:0/3629564802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa8003970 0x7f2fa8003d90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40cf6d0 tx=0x7f2fa80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fdd7c8eecc58d73 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:55.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.176+0000 7f2fc2a00640 1 -- 172.21.15.102:0/3629564802 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4034610 con 0x7f2fa8003970 2024-03-23T03:46:55.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.176+0000 7f2fc2a00640 1 -- 172.21.15.102:0/3629564802 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb402e070 con 0x7f2fa8003970 2024-03-23T03:46:55.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.176+0000 7f2fc2a00640 1 -- 172.21.15.102:0/3629564802 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4034970 con 0x7f2fa8003970 2024-03-23T03:46:55.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.177+0000 7f2fc9800640 1 -- 172.21.15.102:0/3629564802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa8003970 msgr2=0x7f2fa8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:55.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.177+0000 7f2fc9800640 1 --2- 172.21.15.102:0/3629564802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa8003970 0x7f2fa8003d90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40cf6d0 tx=0x7f2fa80057f0 comp rx=0 tx=0).stop 2024-03-23T03:46:55.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.177+0000 7f2fc8e00640 1 -- 172.21.15.102:0/3629564802 reap_dead start 2024-03-23T03:46:55.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.177+0000 7f2fc9800640 1 -- 172.21.15.102:0/3629564802 shutdown_connections 2024-03-23T03:46:55.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.177+0000 7f2fc9800640 1 -- 172.21.15.102:0/3629564802 >> 172.21.15.102:0/3629564802 conn(0x7f2fc40717e0 msgr2=0x7f2fc4071bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:55.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.177+0000 7f2fc9800640 1 -- 172.21.15.102:0/3629564802 shutdown_connections 2024-03-23T03:46:55.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.177+0000 7f2fc9800640 1 -- 172.21.15.102:0/3629564802 wait complete. 2024-03-23T03:46:55.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.179+0000 7f2fc9800640 1 Processor -- start 2024-03-23T03:46:55.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.179+0000 7f2fc9800640 1 -- start start 2024-03-23T03:46:55.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.180+0000 7f2fc9800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa8003970 0x7f2fc415e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:55.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.180+0000 7f2fc9800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fc415f880 0x7f2fc40c61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:55.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.180+0000 7f2fc9800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fc4157360 0x7f2fc41488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:55.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.180+0000 7f2fc9800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc40c7c60 con 0x7f2fa8003970 2024-03-23T03:46:55.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.180+0000 7f2fc9800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc415f3f0 con 0x7f2fc4157360 2024-03-23T03:46:55.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.180+0000 7f2fc9800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc415e7e0 con 0x7f2fc415f880 2024-03-23T03:46:55.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.180+0000 7f2fc3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa8003970 0x7f2fc415e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:55.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.180+0000 7f2fc3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa8003970 0x7f2fc415e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50702/0 (socket says 172.21.15.102:50702) 2024-03-23T03:46:55.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.180+0000 7f2fc3e00640 1 -- 172.21.15.102:0/2430417493 learned_addr learned my addr 172.21.15.102:0/2430417493 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:55.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.180+0000 7f2fc3e00640 1 -- 172.21.15.102:0/2430417493 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fc415f880 msgr2=0x7f2fc40c61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:55.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.180+0000 7f2fc3e00640 1 --2- 172.21.15.102:0/2430417493 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fc415f880 0x7f2fc40c61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:55.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.180+0000 7f2fc3e00640 1 -- 172.21.15.102:0/2430417493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fc4157360 msgr2=0x7f2fc41488c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:55.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.180+0000 7f2fc3e00640 1 --2- 172.21.15.102:0/2430417493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fc4157360 0x7f2fc41488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:55.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.180+0000 7f2fc3e00640 1 -- 172.21.15.102:0/2430417493 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc415eac0 con 0x7f2fa8003970 2024-03-23T03:46:55.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.181+0000 7f2fc3e00640 1 --2- 172.21.15.102:0/2430417493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa8003970 0x7f2fc415e200 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2fb402be60 tx=0x7f2fb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:55.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.181+0000 7f2fbbe00640 1 -- 172.21.15.102:0/2430417493 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb403d840 con 0x7f2fa8003970 2024-03-23T03:46:55.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.181+0000 7f2fbbe00640 1 -- 172.21.15.102:0/2430417493 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb4034e90 con 0x7f2fa8003970 2024-03-23T03:46:55.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.181+0000 7f2fbbe00640 1 -- 172.21.15.102:0/2430417493 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb403dbc0 con 0x7f2fa8003970 2024-03-23T03:46:55.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.181+0000 7f2fc9800640 1 -- 172.21.15.102:0/2430417493 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc415eda0 con 0x7f2fa8003970 2024-03-23T03:46:55.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.181+0000 7f2fc9800640 1 -- 172.21.15.102:0/2430417493 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc40c6730 con 0x7f2fa8003970 2024-03-23T03:46:55.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.182+0000 7f2fc9800640 1 -- 172.21.15.102:0/2430417493 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc404e030 con 0x7f2fa8003970 2024-03-23T03:46:55.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.183+0000 7f2fbbe00640 1 -- 172.21.15.102:0/2430417493 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2fb403dd20 con 0x7f2fa8003970 2024-03-23T03:46:55.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.184+0000 7f2fbbe00640 1 --2- 172.21.15.102:0/2430417493 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fa006ac00 0x7f2fa006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:55.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.184+0000 7f2fbbe00640 1 -- 172.21.15.102:0/2430417493 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5935+0+0 (secure 0 0 0) 0x7f2fb40ab5a0 con 0x7f2fa8003970 2024-03-23T03:46:55.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.184+0000 7f2fc3400640 1 --2- 172.21.15.102:0/2430417493 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fa006ac00 0x7f2fa006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:55.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.185+0000 7f2fc3400640 1 --2- 172.21.15.102:0/2430417493 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fa006ac00 0x7f2fa006d0c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40c69b0 tx=0x7f2fac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:55.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.189+0000 7f2fbbe00640 1 -- 172.21.15.102:0/2430417493 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2fb407a090 con 0x7f2fa8003970 2024-03-23T03:46:55.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:55.383+0000 7f2fc9800640 1 -- 172.21.15.102:0/2430417493 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2fc40b6600 con 0x7f2fa8003970 2024-03-23T03:46:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:56.934+0000 7f2fbbe00640 1 -- 172.21.15.102:0/2430417493 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v24) v1 ==== 228+0+0 (secure 0 0 0) 0x7f2fb4079c20 con 0x7f2fa8003970 2024-03-23T03:46:56.935 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data_ec' created 2024-03-23T03:46:56.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:56.939+0000 7f2fc9800640 1 -- 172.21.15.102:0/2430417493 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fa006ac00 msgr2=0x7f2fa006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:56.939+0000 7f2fc9800640 1 --2- 172.21.15.102:0/2430417493 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fa006ac00 0x7f2fa006d0c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40c69b0 tx=0x7f2fac008040 comp rx=0 tx=0).stop 2024-03-23T03:46:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:56.939+0000 7f2fc9800640 1 -- 172.21.15.102:0/2430417493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa8003970 msgr2=0x7f2fc415e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:56.939+0000 7f2fc9800640 1 --2- 172.21.15.102:0/2430417493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa8003970 0x7f2fc415e200 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2fb402be60 tx=0x7f2fb4004730 comp rx=0 tx=0).stop 2024-03-23T03:46:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:56.939+0000 7f2fc9800640 1 -- 172.21.15.102:0/2430417493 shutdown_connections 2024-03-23T03:46:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:56.939+0000 7f2fc9800640 1 --2- 172.21.15.102:0/2430417493 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fa006ac00 0x7f2fa006d0c0 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2fc40c69b0 tx=0x7f2fac008040 comp rx=0 tx=0).stop 2024-03-23T03:46:56.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:56.939+0000 7f2fc9800640 1 --2- 172.21.15.102:0/2430417493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fc4157360 0x7f2fc41488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:56.939+0000 7f2fc9800640 1 --2- 172.21.15.102:0/2430417493 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fc415f880 0x7f2fc40c61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:56.939+0000 7f2fc9800640 1 --2- 172.21.15.102:0/2430417493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa8003970 0x7f2fc415e200 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:56.939+0000 7f2fc9800640 1 -- 172.21.15.102:0/2430417493 >> 172.21.15.102:0/2430417493 conn(0x7f2fc40717e0 msgr2=0x7f2fc40d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:56.940+0000 7f2fc9800640 1 -- 172.21.15.102:0/2430417493 shutdown_connections 2024-03-23T03:46:56.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:56.940+0000 7f2fc9800640 1 -- 172.21.15.102:0/2430417493 wait complete. 2024-03-23T03:46:56.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-23T03:46:57.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.154+0000 7fc0d2000640 1 Processor -- start 2024-03-23T03:46:57.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.154+0000 7fc0d2000640 1 -- start start 2024-03-23T03:46:57.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.156+0000 7fc0d2000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc0cc0ce930 0x7fc0cc0add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:57.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.156+0000 7fc0d2000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc0cc0cd3e0 0x7fc0cc0ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:57.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.156+0000 7fc0d2000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0cc0cf250 con 0x7fc0cc06ce90 2024-03-23T03:46:57.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.156+0000 7fc0d2000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0cc06c760 con 0x7fc0cc0ce930 2024-03-23T03:46:57.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.156+0000 7fc0d2000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc0b42d0 con 0x7fc0cc0cd3e0 2024-03-23T03:46:57.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.156+0000 7fc0cbe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc0cc06ce90 0x7fc0cc15e290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38086/0 (socket says 172.21.15.102:38086) 2024-03-23T03:46:57.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.156+0000 7fc0cbe00640 1 -- 172.21.15.102:0/3757666227 learned_addr learned my addr 172.21.15.102:0/3757666227 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:46:57.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0cb400640 1 --2- 172.21.15.102:0/3757666227 >> v2:172.21.15.173:3300/0 conn(0x7fc0cc0ce930 0x7fc0cc0add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:57.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0c3e00640 1 --2- 172.21.15.102:0/3757666227 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc0cc0cd3e0 0x7fc0cc0ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:57.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0caa00640 1 -- 172.21.15.102:0/3757666227 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744644616 0 0) 0x7fc0cc0cf250 con 0x7fc0cc06ce90 2024-03-23T03:46:57.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0caa00640 1 -- 172.21.15.102:0/3757666227 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0ac003520 con 0x7fc0cc06ce90 2024-03-23T03:46:57.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0caa00640 1 -- 172.21.15.102:0/3757666227 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1487429212 0 0) 0x7fc0ac003520 con 0x7fc0cc06ce90 2024-03-23T03:46:57.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0caa00640 1 -- 172.21.15.102:0/3757666227 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc0cc0cd3e0 msgr2=0x7fc0cc0ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:57.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0caa00640 1 --2- 172.21.15.102:0/3757666227 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc0cc0cd3e0 0x7fc0cc0ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:57.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0caa00640 1 -- 172.21.15.102:0/3757666227 >> v2:172.21.15.173:3300/0 conn(0x7fc0cc0ce930 msgr2=0x7fc0cc0add30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:57.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0caa00640 1 --2- 172.21.15.102:0/3757666227 >> v2:172.21.15.173:3300/0 conn(0x7fc0cc0ce930 0x7fc0cc0add30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:57.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0caa00640 1 -- 172.21.15.102:0/3757666227 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0cc0cdba0 con 0x7fc0cc06ce90 2024-03-23T03:46:57.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0caa00640 1 -- 172.21.15.102:0/3757666227 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fc0c40019f0 con 0x7fc0cc06ce90 2024-03-23T03:46:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0caa00640 1 -- 172.21.15.102:0/3757666227 >> v1:172.21.15.102:6789/0 conn(0x7fc0cc06ce90 legacy=0x7fc0cc15e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0caa00640 1 --2- 172.21.15.102:0/3757666227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0ac003970 0x7fc0ac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0caa00640 1 --2- 172.21.15.102:0/3757666227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0ac0047c0 0x7fc0ac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0caa00640 1 --2- 172.21.15.102:0/3757666227 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc0cc0cd3e0 0x7fc0ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0caa00640 1 -- 172.21.15.102:0/3757666227 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc06c760 con 0x7fc0ac0047c0 2024-03-23T03:46:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0caa00640 1 -- 172.21.15.102:0/3757666227 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc0b42d0 con 0x7fc0ac003970 2024-03-23T03:46:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.157+0000 7fc0caa00640 1 -- 172.21.15.102:0/3757666227 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0ac00b980 con 0x7fc0cc0cd3e0 2024-03-23T03:46:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.158+0000 7fc0c3e00640 1 --2- 172.21.15.102:0/3757666227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0ac0047c0 0x7fc0ac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.158+0000 7fc0cb400640 1 --2- 172.21.15.102:0/3757666227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0ac003970 0x7fc0ac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.158+0000 7fc0cbe00640 1 --2- 172.21.15.102:0/3757666227 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc0cc0cd3e0 0x7fc0ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.158+0000 7fc0c3e00640 1 -- 172.21.15.102:0/3757666227 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc0cc0cd3e0 msgr2=0x7fc0ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.158+0000 7fc0c3e00640 1 --2- 172.21.15.102:0/3757666227 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc0cc0cd3e0 0x7fc0ac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.158+0000 7fc0c3e00640 1 -- 172.21.15.102:0/3757666227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0ac003970 msgr2=0x7fc0ac003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:57.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.158+0000 7fc0c3e00640 1 --2- 172.21.15.102:0/3757666227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0ac003970 0x7fc0ac003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:57.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.158+0000 7fc0c3e00640 1 -- 172.21.15.102:0/3757666227 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ac003520 con 0x7fc0ac0047c0 2024-03-23T03:46:57.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.158+0000 7fc0cbe00640 1 --2- 172.21.15.102:0/3757666227 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc0cc0cd3e0 0x7fc0ac005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:46:57.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.159+0000 7fc0c3e00640 1 --2- 172.21.15.102:0/3757666227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0ac0047c0 0x7fc0ac004c40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4009ac0 tx=0x7fc0ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfe2a0988fb3db4f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:57.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.159+0000 7fc0caa00640 1 -- 172.21.15.102:0/3757666227 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b402c6b0 con 0x7fc0ac0047c0 2024-03-23T03:46:57.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.159+0000 7fc0caa00640 1 -- 172.21.15.102:0/3757666227 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b402c810 con 0x7fc0ac0047c0 2024-03-23T03:46:57.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.159+0000 7fc0caa00640 1 -- 172.21.15.102:0/3757666227 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b402cba0 con 0x7fc0ac0047c0 2024-03-23T03:46:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.160+0000 7fc0d2000640 1 -- 172.21.15.102:0/3757666227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0ac0047c0 msgr2=0x7fc0ac004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.160+0000 7fc0d2000640 1 --2- 172.21.15.102:0/3757666227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0ac0047c0 0x7fc0ac004c40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4009ac0 tx=0x7fc0ac0057f0 comp rx=0 tx=0).stop 2024-03-23T03:46:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.160+0000 7fc0cbe00640 1 -- 172.21.15.102:0/3757666227 reap_dead start 2024-03-23T03:46:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.160+0000 7fc0d2000640 1 -- 172.21.15.102:0/3757666227 shutdown_connections 2024-03-23T03:46:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.160+0000 7fc0d2000640 1 -- 172.21.15.102:0/3757666227 >> 172.21.15.102:0/3757666227 conn(0x7fc0cc06a0b0 msgr2=0x7fc0cc06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.160+0000 7fc0d2000640 1 -- 172.21.15.102:0/3757666227 shutdown_connections 2024-03-23T03:46:57.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.160+0000 7fc0d2000640 1 -- 172.21.15.102:0/3757666227 wait complete. 2024-03-23T03:46:57.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.162+0000 7fc0d2000640 1 Processor -- start 2024-03-23T03:46:57.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.162+0000 7fc0d2000640 1 -- start start 2024-03-23T03:46:57.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.163+0000 7fc0d2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0ac0047c0 0x7fc0cc0ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:57.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.163+0000 7fc0d2000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc0cc0c3f10 0x7fc0cc0acda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.163+0000 7fc0d2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0cc0c5580 0x7fc0cc0c5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.163+0000 7fc0d2000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc0c5f50 con 0x7fc0ac0047c0 2024-03-23T03:46:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.163+0000 7fc0d2000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc15afe0 con 0x7fc0cc0c5580 2024-03-23T03:46:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.163+0000 7fc0d2000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0cc0c43d0 con 0x7fc0cc0c3f10 2024-03-23T03:46:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.163+0000 7fc0cb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0ac0047c0 0x7fc0cc0ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.163+0000 7fc0cb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0ac0047c0 0x7fc0cc0ca0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50718/0 (socket says 172.21.15.102:50718) 2024-03-23T03:46:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.163+0000 7fc0cb400640 1 -- 172.21.15.102:0/3746779596 learned_addr learned my addr 172.21.15.102:0/3746779596 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.164+0000 7fc0cb400640 1 -- 172.21.15.102:0/3746779596 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc0cc0c3f10 msgr2=0x7fc0cc0acda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.164+0000 7fc0cbe00640 1 --2- 172.21.15.102:0/3746779596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0cc0c5580 0x7fc0cc0c5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.164+0000 7fc0c3e00640 1 --2- 172.21.15.102:0/3746779596 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc0cc0c3f10 0x7fc0cc0acda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.164+0000 7fc0cb400640 1 --2- 172.21.15.102:0/3746779596 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc0cc0c3f10 0x7fc0cc0acda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.164+0000 7fc0cb400640 1 -- 172.21.15.102:0/3746779596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0cc0c5580 msgr2=0x7fc0cc0c5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:57.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.164+0000 7fc0cb400640 1 --2- 172.21.15.102:0/3746779596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0cc0c5580 0x7fc0cc0c5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.164+0000 7fc0cb400640 1 -- 172.21.15.102:0/3746779596 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0cc0c46b0 con 0x7fc0ac0047c0 2024-03-23T03:46:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.164+0000 7fc0cb400640 1 --2- 172.21.15.102:0/3746779596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0ac0047c0 0x7fc0cc0ca0b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc0116d0 tx=0x7fc0bc011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.164+0000 7fc0c3400640 1 -- 172.21.15.102:0/3746779596 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc00f9c0 con 0x7fc0ac0047c0 2024-03-23T03:46:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.164+0000 7fc0c3400640 1 -- 172.21.15.102:0/3746779596 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0bc00fb20 con 0x7fc0ac0047c0 2024-03-23T03:46:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.164+0000 7fc0c3400640 1 -- 172.21.15.102:0/3746779596 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc00fe90 con 0x7fc0ac0047c0 2024-03-23T03:46:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.165+0000 7fc0d2000640 1 -- 172.21.15.102:0/3746779596 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0cc0c4990 con 0x7fc0ac0047c0 2024-03-23T03:46:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.165+0000 7fc0d2000640 1 -- 172.21.15.102:0/3746779596 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0cc150540 con 0x7fc0ac0047c0 2024-03-23T03:46:57.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.166+0000 7fc0d2000640 1 -- 172.21.15.102:0/3746779596 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0cc0cf600 con 0x7fc0ac0047c0 2024-03-23T03:46:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.167+0000 7fc0c3400640 1 -- 172.21.15.102:0/3746779596 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc0bc015460 con 0x7fc0ac0047c0 2024-03-23T03:46:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.168+0000 7fc0c3400640 1 --2- 172.21.15.102:0/3746779596 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc0a806ac00 0x7fc0a806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.168+0000 7fc0c3400640 1 -- 172.21.15.102:0/3746779596 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6428+0+0 (secure 0 0 0) 0x7fc0bc089c90 con 0x7fc0ac0047c0 2024-03-23T03:46:57.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.168+0000 7fc0c3e00640 1 --2- 172.21.15.102:0/3746779596 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc0a806ac00 0x7fc0a806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:57.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.169+0000 7fc0c3e00640 1 --2- 172.21.15.102:0/3746779596 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc0a806ac00 0x7fc0a806d0c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4036000 tx=0x7fc0b4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:57.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.173+0000 7fc0c3400640 1 -- 172.21.15.102:0/3746779596 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc0bc054b00 con 0x7fc0ac0047c0 2024-03-23T03:46:57.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.367+0000 7fc0d2000640 1 -- 172.21.15.102:0/3746779596 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7fc0cc073390 con 0x7fc0ac0047c0 2024-03-23T03:46:57.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.939+0000 7fc0c3400640 1 -- 172.21.15.102:0/3746779596 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc0bc0220c0 con 0x7fc0ac0047c0 2024-03-23T03:46:57.940 INFO:teuthology.orchestra.run.smithi102.stderr:set pool 4 allow_ec_overwrites to true 2024-03-23T03:46:57.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.944+0000 7fc0d2000640 1 -- 172.21.15.102:0/3746779596 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc0a806ac00 msgr2=0x7fc0a806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:57.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.944+0000 7fc0d2000640 1 --2- 172.21.15.102:0/3746779596 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc0a806ac00 0x7fc0a806d0c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4036000 tx=0x7fc0b4030000 comp rx=0 tx=0).stop 2024-03-23T03:46:57.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.944+0000 7fc0d2000640 1 -- 172.21.15.102:0/3746779596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0ac0047c0 msgr2=0x7fc0cc0ca0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:57.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.944+0000 7fc0d2000640 1 --2- 172.21.15.102:0/3746779596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0ac0047c0 0x7fc0cc0ca0b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc0116d0 tx=0x7fc0bc011b90 comp rx=0 tx=0).stop 2024-03-23T03:46:57.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.945+0000 7fc0d2000640 1 -- 172.21.15.102:0/3746779596 shutdown_connections 2024-03-23T03:46:57.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.945+0000 7fc0d2000640 1 --2- 172.21.15.102:0/3746779596 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc0a806ac00 0x7fc0a806d0c0 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4036000 tx=0x7fc0b4030000 comp rx=0 tx=0).stop 2024-03-23T03:46:57.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.945+0000 7fc0d2000640 1 --2- 172.21.15.102:0/3746779596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc0cc0c5580 0x7fc0cc0c5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:57.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.945+0000 7fc0d2000640 1 --2- 172.21.15.102:0/3746779596 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc0cc0c3f10 0x7fc0cc0acda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:57.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.945+0000 7fc0d2000640 1 --2- 172.21.15.102:0/3746779596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0ac0047c0 0x7fc0cc0ca0b0 secure :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc0116d0 tx=0x7fc0bc011b90 comp rx=0 tx=0).stop 2024-03-23T03:46:57.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.945+0000 7fc0d2000640 1 -- 172.21.15.102:0/3746779596 >> 172.21.15.102:0/3746779596 conn(0x7fc0cc06a0b0 msgr2=0x7fc0cc0b16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:57.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.948+0000 7fc0d2000640 1 -- 172.21.15.102:0/3746779596 shutdown_connections 2024-03-23T03:46:57.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:57.949+0000 7fc0d2000640 1 -- 172.21.15.102:0/3746779596 wait complete. 2024-03-23T03:46:57.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-23T03:46:58.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.161+0000 7f1737e00640 1 Processor -- start 2024-03-23T03:46:58.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.161+0000 7f1737e00640 1 -- start start 2024-03-23T03:46:58.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.162+0000 7f1737e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f173815cab0 0x7f17380b8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:58.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.162+0000 7f1737e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f17380ceeb0 0x7f17380cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:58.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.162+0000 7f1737e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f17380b4120 0x7f17380b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:58.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.162+0000 7f1737e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17380cf850 con 0x7f17380b4120 2024-03-23T03:46:58.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.162+0000 7f1737e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f173815ce90 con 0x7f17380ceeb0 2024-03-23T03:46:58.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.162+0000 7f1737e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17380cddf0 con 0x7f173815cab0 2024-03-23T03:46:58.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.163+0000 7f1737400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f17380b4120 0x7f17380b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:58.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.163+0000 7f1737400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f17380b4120 0x7f17380b45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50734/0 (socket says 172.21.15.102:50734) 2024-03-23T03:46:58.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.163+0000 7f1737400640 1 -- 172.21.15.102:0/1593271959 learned_addr learned my addr 172.21.15.102:0/1593271959 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:58.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.163+0000 7f1736000640 1 --2- 172.21.15.102:0/1593271959 >> v2:172.21.15.173:3300/0 conn(0x7f17380ceeb0 0x7f17380cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:58.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.163+0000 7f1736a00640 1 --2- 172.21.15.102:0/1593271959 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f173815cab0 0x7f17380b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:58.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.163+0000 7f1737400640 1 -- 172.21.15.102:0/1593271959 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f173815cab0 msgr2=0x7f17380b8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:58.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.163+0000 7f1737400640 1 --2- 172.21.15.102:0/1593271959 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f173815cab0 0x7f17380b8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.163+0000 7f1737400640 1 -- 172.21.15.102:0/1593271959 >> v2:172.21.15.173:3300/0 conn(0x7f17380ceeb0 msgr2=0x7f17380cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:58.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.163+0000 7f1737400640 1 --2- 172.21.15.102:0/1593271959 >> v2:172.21.15.173:3300/0 conn(0x7f17380ceeb0 0x7f17380cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.163+0000 7f1737400640 1 -- 172.21.15.102:0/1593271959 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17380cd570 con 0x7f17380b4120 2024-03-23T03:46:58.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.164+0000 7f1737400640 1 --2- 172.21.15.102:0/1593271959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17380b4120 0x7f17380b45b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f17380b51a0 tx=0x7f173002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a976ad88ba7d2d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:58.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.165+0000 7f1735600640 1 -- 172.21.15.102:0/1593271959 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f173002ca70 con 0x7f17380b4120 2024-03-23T03:46:58.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.165+0000 7f1735600640 1 -- 172.21.15.102:0/1593271959 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f173002cbd0 con 0x7f17380b4120 2024-03-23T03:46:58.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.165+0000 7f1735600640 1 -- 172.21.15.102:0/1593271959 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17300343f0 con 0x7f17380b4120 2024-03-23T03:46:58.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.165+0000 7f1737e00640 1 -- 172.21.15.102:0/1593271959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17380b4120 msgr2=0x7f17380b45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:58.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.165+0000 7f1737e00640 1 --2- 172.21.15.102:0/1593271959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17380b4120 0x7f17380b45b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f17380b51a0 tx=0x7f173002bed0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.165+0000 7f1737e00640 1 -- 172.21.15.102:0/1593271959 shutdown_connections 2024-03-23T03:46:58.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.165+0000 7f1737e00640 1 --2- 172.21.15.102:0/1593271959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17380b4120 0x7f17380b45b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.165+0000 7f1737e00640 1 --2- 172.21.15.102:0/1593271959 >> v2:172.21.15.173:3300/0 conn(0x7f17380ceeb0 0x7f17380cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.165+0000 7f1737e00640 1 --2- 172.21.15.102:0/1593271959 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f173815cab0 0x7f17380b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.165+0000 7f1737e00640 1 -- 172.21.15.102:0/1593271959 >> 172.21.15.102:0/1593271959 conn(0x7f173806ae90 msgr2=0x7f173806b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:58.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.165+0000 7f1737e00640 1 -- 172.21.15.102:0/1593271959 shutdown_connections 2024-03-23T03:46:58.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.165+0000 7f1737e00640 1 -- 172.21.15.102:0/1593271959 wait complete. 2024-03-23T03:46:58.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.167+0000 7f1737e00640 1 Processor -- start 2024-03-23T03:46:58.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.167+0000 7f1737e00640 1 -- start start 2024-03-23T03:46:58.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.168+0000 7f1737e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17380b4120 0x7f17380ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:58.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.168+0000 7f1737e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f17380ceeb0 0x7f17380b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:58.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.168+0000 7f1737e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f173815cab0 0x7f173815d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:58.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.168+0000 7f1737e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17380cbb90 con 0x7f17380b4120 2024-03-23T03:46:58.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.168+0000 7f1737e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17380b0f30 con 0x7f173815cab0 2024-03-23T03:46:58.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.168+0000 7f1737e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17380caf80 con 0x7f17380ceeb0 2024-03-23T03:46:58.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.168+0000 7f1736a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17380b4120 0x7f17380ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:58.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.168+0000 7f1736a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17380b4120 0x7f17380ca9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50736/0 (socket says 172.21.15.102:50736) 2024-03-23T03:46:58.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.168+0000 7f1736a00640 1 -- 172.21.15.102:0/3559863902 learned_addr learned my addr 172.21.15.102:0/3559863902 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:58.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.169+0000 7f1737400640 1 --2- 172.21.15.102:0/3559863902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f173815cab0 0x7f173815d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:58.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.169+0000 7f1736a00640 1 -- 172.21.15.102:0/3559863902 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f17380ceeb0 msgr2=0x7f17380b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:58.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.169+0000 7f1736a00640 1 --2- 172.21.15.102:0/3559863902 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f17380ceeb0 0x7f17380b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.169+0000 7f1736a00640 1 -- 172.21.15.102:0/3559863902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f173815cab0 msgr2=0x7f173815d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:58.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.169+0000 7f1736a00640 1 --2- 172.21.15.102:0/3559863902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f173815cab0 0x7f173815d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.169+0000 7f1736a00640 1 -- 172.21.15.102:0/3559863902 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1730009b30 con 0x7f17380b4120 2024-03-23T03:46:58.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.169+0000 7f1736a00640 1 --2- 172.21.15.102:0/3559863902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17380b4120 0x7f17380ca9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f172800c820 tx=0x7f172800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:58.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.169+0000 7f172ea00640 1 -- 172.21.15.102:0/3559863902 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1728004220 con 0x7f17380b4120 2024-03-23T03:46:58.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.169+0000 7f172ea00640 1 -- 172.21.15.102:0/3559863902 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17280043a0 con 0x7f17380b4120 2024-03-23T03:46:58.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.169+0000 7f1737e00640 1 -- 172.21.15.102:0/3559863902 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17380cb260 con 0x7f17380b4120 2024-03-23T03:46:58.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.169+0000 7f172ea00640 1 -- 172.21.15.102:0/3559863902 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1728007a70 con 0x7f17380b4120 2024-03-23T03:46:58.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.169+0000 7f1737e00640 1 -- 172.21.15.102:0/3559863902 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17380cb820 con 0x7f17380b4120 2024-03-23T03:46:58.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.170+0000 7f1737e00640 1 -- 172.21.15.102:0/3559863902 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f173804e030 con 0x7f17380b4120 2024-03-23T03:46:58.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.171+0000 7f172ea00640 1 -- 172.21.15.102:0/3559863902 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1728007bd0 con 0x7f17380b4120 2024-03-23T03:46:58.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.172+0000 7f172ea00640 1 --2- 172.21.15.102:0/3559863902 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f171406ac30 0x7f171406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:58.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.172+0000 7f172ea00640 1 -- 172.21.15.102:0/3559863902 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6428+0+0 (secure 0 0 0) 0x7f17280893b0 con 0x7f17380b4120 2024-03-23T03:46:58.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.172+0000 7f1736000640 1 --2- 172.21.15.102:0/3559863902 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f171406ac30 0x7f171406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:58.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.173+0000 7f1736000640 1 --2- 172.21.15.102:0/3559863902 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f171406ac30 0x7f171406d0f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f173815ffd0 tx=0x7f1720008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:58.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.177+0000 7f172ea00640 1 -- 172.21.15.102:0/3559863902 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f17280542b0 con 0x7f17380b4120 2024-03-23T03:46:58.194 INFO:tasks.ceph.mds.d.smithi173.stderr:2024-03-23T03:46:58.194+0000 7fc643000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T03:46:58.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.408+0000 7f1737e00640 1 -- 172.21.15.102:0/3559863902 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f17380d02d0 con 0x7f17380b4120 2024-03-23T03:46:58.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.416+0000 7f172ea00640 1 -- 172.21.15.102:0/3559863902 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v6) v1 ==== 137+0+0 (secure 0 0 0) 0x7f17280578d0 con 0x7f17380b4120 2024-03-23T03:46:58.417 INFO:teuthology.orchestra.run.smithi102.stderr:added data pool 4 to fsmap 2024-03-23T03:46:58.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.418+0000 7f1737e00640 1 -- 172.21.15.102:0/3559863902 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f171406ac30 msgr2=0x7f171406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:58.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.418+0000 7f1737e00640 1 --2- 172.21.15.102:0/3559863902 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f171406ac30 0x7f171406d0f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f173815ffd0 tx=0x7f1720008040 comp rx=0 tx=0).stop 2024-03-23T03:46:58.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.418+0000 7f1737e00640 1 -- 172.21.15.102:0/3559863902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17380b4120 msgr2=0x7f17380ca9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:58.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.418+0000 7f1737e00640 1 --2- 172.21.15.102:0/3559863902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17380b4120 0x7f17380ca9a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f172800c820 tx=0x7f172800cce0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.419+0000 7f1737e00640 1 -- 172.21.15.102:0/3559863902 shutdown_connections 2024-03-23T03:46:58.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.419+0000 7f1737e00640 1 --2- 172.21.15.102:0/3559863902 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f171406ac30 0x7f171406d0f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.419+0000 7f1737e00640 1 --2- 172.21.15.102:0/3559863902 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f173815cab0 0x7f173815d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.419+0000 7f1737e00640 1 --2- 172.21.15.102:0/3559863902 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f17380ceeb0 0x7f17380b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.419+0000 7f1737e00640 1 --2- 172.21.15.102:0/3559863902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17380b4120 0x7f17380ca9a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.419+0000 7f1737e00640 1 -- 172.21.15.102:0/3559863902 >> 172.21.15.102:0/3559863902 conn(0x7f173806ae90 msgr2=0x7f17380c8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:58.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.419+0000 7f1737e00640 1 -- 172.21.15.102:0/3559863902 shutdown_connections 2024-03-23T03:46:58.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.419+0000 7f1737e00640 1 -- 172.21.15.102:0/3559863902 wait complete. 2024-03-23T03:46:58.437 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:46:58.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.631+0000 7fd290800640 1 Processor -- start 2024-03-23T03:46:58.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.631+0000 7fd290800640 1 -- start start 2024-03-23T03:46:58.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.633+0000 7fd290800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd28815b560 0x7fd28815b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:58.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.633+0000 7fd290800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd2880b39b0 0x7fd2880b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:58.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.633+0000 7fd290800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd28815e500 0x7fd28815e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:58.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.633+0000 7fd290800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2880b4370 con 0x7fd28815e500 2024-03-23T03:46:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.633+0000 7fd290800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd28806c6f0 con 0x7fd2880b39b0 2024-03-23T03:46:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.633+0000 7fd290800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28806c9d0 con 0x7fd28815b560 2024-03-23T03:46:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.633+0000 7fd28e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd28815e500 0x7fd28815e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.633+0000 7fd28e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd28815e500 0x7fd28815e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50752/0 (socket says 172.21.15.102:50752) 2024-03-23T03:46:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.633+0000 7fd28e800640 1 -- 172.21.15.102:0/3143582902 learned_addr learned my addr 172.21.15.102:0/3143582902 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.633+0000 7fd28de00640 1 --2- 172.21.15.102:0/3143582902 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd28815b560 0x7fd28815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.634+0000 7fd28d400640 1 --2- 172.21.15.102:0/3143582902 >> v2:172.21.15.173:3300/0 conn(0x7fd2880b39b0 0x7fd2880b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.634+0000 7fd28e800640 1 -- 172.21.15.102:0/3143582902 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd28815b560 msgr2=0x7fd28815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.634+0000 7fd28e800640 1 --2- 172.21.15.102:0/3143582902 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd28815b560 0x7fd28815b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.634+0000 7fd28e800640 1 -- 172.21.15.102:0/3143582902 >> v2:172.21.15.173:3300/0 conn(0x7fd2880b39b0 msgr2=0x7fd2880b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.634+0000 7fd28e800640 1 --2- 172.21.15.102:0/3143582902 >> v2:172.21.15.173:3300/0 conn(0x7fd2880b39b0 0x7fd2880b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.634+0000 7fd28e800640 1 -- 172.21.15.102:0/3143582902 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28806ccb0 con 0x7fd28815e500 2024-03-23T03:46:58.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.635+0000 7fd28e800640 1 --2- 172.21.15.102:0/3143582902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28815e500 0x7fd28815e990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd288071510 tx=0x7fd28402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6c983d314b87063 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:58.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.635+0000 7fd28ca00640 1 -- 172.21.15.102:0/3143582902 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28402ca70 con 0x7fd28815e500 2024-03-23T03:46:58.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.635+0000 7fd28ca00640 1 -- 172.21.15.102:0/3143582902 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28402cbd0 con 0x7fd28815e500 2024-03-23T03:46:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.635+0000 7fd28ca00640 1 -- 172.21.15.102:0/3143582902 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2840343f0 con 0x7fd28815e500 2024-03-23T03:46:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.636+0000 7fd290800640 1 -- 172.21.15.102:0/3143582902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28815e500 msgr2=0x7fd28815e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.636+0000 7fd290800640 1 --2- 172.21.15.102:0/3143582902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28815e500 0x7fd28815e990 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd288071510 tx=0x7fd28402bed0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.636+0000 7fd290800640 1 -- 172.21.15.102:0/3143582902 shutdown_connections 2024-03-23T03:46:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.636+0000 7fd290800640 1 --2- 172.21.15.102:0/3143582902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28815e500 0x7fd28815e990 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.636+0000 7fd290800640 1 --2- 172.21.15.102:0/3143582902 >> v2:172.21.15.173:3300/0 conn(0x7fd2880b39b0 0x7fd2880b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.636+0000 7fd290800640 1 --2- 172.21.15.102:0/3143582902 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd28815b560 0x7fd28815b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.636+0000 7fd290800640 1 -- 172.21.15.102:0/3143582902 >> 172.21.15.102:0/3143582902 conn(0x7fd28806a3d0 msgr2=0x7fd2880b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.636+0000 7fd290800640 1 -- 172.21.15.102:0/3143582902 shutdown_connections 2024-03-23T03:46:58.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.636+0000 7fd290800640 1 -- 172.21.15.102:0/3143582902 wait complete. 2024-03-23T03:46:58.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.638+0000 7fd290800640 1 Processor -- start 2024-03-23T03:46:58.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.638+0000 7fd290800640 1 -- start start 2024-03-23T03:46:58.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.639+0000 7fd290800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2880b39b0 0x7fd288153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.639+0000 7fd290800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd28815b560 0x7fd28813d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.639+0000 7fd290800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28815e500 0x7fd2880b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.639+0000 7fd290800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd288154410 con 0x7fd28815e500 2024-03-23T03:46:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.639+0000 7fd290800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2880b1d60 con 0x7fd28815b560 2024-03-23T03:46:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.639+0000 7fd290800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2880b2040 con 0x7fd2880b39b0 2024-03-23T03:46:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.639+0000 7fd28e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28815e500 0x7fd2880b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.639+0000 7fd28e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28815e500 0x7fd2880b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50766/0 (socket says 172.21.15.102:50766) 2024-03-23T03:46:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.639+0000 7fd28e800640 1 -- 172.21.15.102:0/2626952986 learned_addr learned my addr 172.21.15.102:0/2626952986 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.639+0000 7fd28de00640 1 --2- 172.21.15.102:0/2626952986 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2880b39b0 0x7fd288153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.639+0000 7fd28d400640 1 --2- 172.21.15.102:0/2626952986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd28815b560 0x7fd28813d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.639+0000 7fd28e800640 1 -- 172.21.15.102:0/2626952986 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2880b39b0 msgr2=0x7fd288153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:58.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.639+0000 7fd28e800640 1 --2- 172.21.15.102:0/2626952986 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2880b39b0 0x7fd288153ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.639+0000 7fd28e800640 1 -- 172.21.15.102:0/2626952986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd28815b560 msgr2=0x7fd28813d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.639+0000 7fd28e800640 1 --2- 172.21.15.102:0/2626952986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd28815b560 0x7fd28813d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.639+0000 7fd28e800640 1 -- 172.21.15.102:0/2626952986 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd284009b30 con 0x7fd28815e500 2024-03-23T03:46:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.639+0000 7fd28d400640 1 --2- 172.21.15.102:0/2626952986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd28815b560 0x7fd28813d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:46:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.640+0000 7fd28e800640 1 --2- 172.21.15.102:0/2626952986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28815e500 0x7fd2880b2e00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd284036040 tx=0x7fd284004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.640+0000 7fd282000640 1 -- 172.21.15.102:0/2626952986 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd284009510 con 0x7fd28815e500 2024-03-23T03:46:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.640+0000 7fd282000640 1 -- 172.21.15.102:0/2626952986 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28403e450 con 0x7fd28815e500 2024-03-23T03:46:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.640+0000 7fd282000640 1 -- 172.21.15.102:0/2626952986 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28403e800 con 0x7fd28815e500 2024-03-23T03:46:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.640+0000 7fd290800640 1 -- 172.21.15.102:0/2626952986 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2880b2320 con 0x7fd28815e500 2024-03-23T03:46:58.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.640+0000 7fd290800640 1 -- 172.21.15.102:0/2626952986 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2880b28e0 con 0x7fd28815e500 2024-03-23T03:46:58.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.641+0000 7fd290800640 1 -- 172.21.15.102:0/2626952986 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd28806ac20 con 0x7fd28815e500 2024-03-23T03:46:58.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.642+0000 7fd282000640 1 -- 172.21.15.102:0/2626952986 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd28403e960 con 0x7fd28815e500 2024-03-23T03:46:58.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.643+0000 7fd282000640 1 --2- 172.21.15.102:0/2626952986 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd26c06ac00 0x7fd26c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:58.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.643+0000 7fd282000640 1 -- 172.21.15.102:0/2626952986 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd2840ac4f0 con 0x7fd28815e500 2024-03-23T03:46:58.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.643+0000 7fd28de00640 1 --2- 172.21.15.102:0/2626952986 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd26c06ac00 0x7fd26c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:58.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.644+0000 7fd28de00640 1 --2- 172.21.15.102:0/2626952986 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd26c06ac00 0x7fd26c06d0c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd27c006fd0 tx=0x7fd27c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:58.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.647+0000 7fd282000640 1 -- 172.21.15.102:0/2626952986 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd284077340 con 0x7fd28815e500 2024-03-23T03:46:58.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.860+0000 7fd290800640 1 -- 172.21.15.102:0/2626952986 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd288073390 con 0x7fd28815e500 2024-03-23T03:46:58.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.861+0000 7fd282000640 1 -- 172.21.15.102:0/2626952986 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4739 (secure 0 0 0) 0x7fd28407a960 con 0x7fd28815e500 2024-03-23T03:46:58.862 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:46:58.862 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/891818172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":891818172},{"type":"v1","addr":"172.21.15.102:6836","nonce":891818172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2404489233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2404489233},{"type":"v1","addr":"172.21.15.102:6837","nonce":2404489233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/1839967745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":1839967745},{"type":"v1","addr":"172.21.15.173:6835","nonce":1839967745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:46:53.912287+0000","modified":"2024-03-23T03:46:58.411062+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6834/1462637509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1462637509},{"type":"v1","addr":"172.21.15.173:6834","nonce":1462637509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T03:46:58.862 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 6 2024-03-23T03:46:58.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.864+0000 7fd290800640 1 -- 172.21.15.102:0/2626952986 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd26c06ac00 msgr2=0x7fd26c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:58.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.864+0000 7fd290800640 1 --2- 172.21.15.102:0/2626952986 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd26c06ac00 0x7fd26c06d0c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd27c006fd0 tx=0x7fd27c008040 comp rx=0 tx=0).stop 2024-03-23T03:46:58.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.864+0000 7fd290800640 1 -- 172.21.15.102:0/2626952986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28815e500 msgr2=0x7fd2880b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:58.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.864+0000 7fd290800640 1 --2- 172.21.15.102:0/2626952986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28815e500 0x7fd2880b2e00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd284036040 tx=0x7fd284004a40 comp rx=0 tx=0).stop 2024-03-23T03:46:58.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.864+0000 7fd290800640 1 -- 172.21.15.102:0/2626952986 shutdown_connections 2024-03-23T03:46:58.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.864+0000 7fd290800640 1 --2- 172.21.15.102:0/2626952986 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd26c06ac00 0x7fd26c06d0c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.864+0000 7fd290800640 1 --2- 172.21.15.102:0/2626952986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd28815e500 0x7fd2880b2e00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.864+0000 7fd290800640 1 --2- 172.21.15.102:0/2626952986 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd28815b560 0x7fd28813d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.864+0000 7fd290800640 1 --2- 172.21.15.102:0/2626952986 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2880b39b0 0x7fd288153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:58.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.864+0000 7fd290800640 1 -- 172.21.15.102:0/2626952986 >> 172.21.15.102:0/2626952986 conn(0x7fd28806a3d0 msgr2=0x7fd288142af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:58.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.864+0000 7fd290800640 1 -- 172.21.15.102:0/2626952986 shutdown_connections 2024-03-23T03:46:58.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:58.864+0000 7fd290800640 1 -- 172.21.15.102:0/2626952986 wait complete. 2024-03-23T03:46:58.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:46:59.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.074+0000 7f9425200640 1 Processor -- start 2024-03-23T03:46:59.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.075+0000 7f9425200640 1 -- start start 2024-03-23T03:46:59.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.076+0000 7f9425200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f94200b5a30 0x7f94200b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.076+0000 7f9425200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94200d1830 con 0x7f94200b4350 2024-03-23T03:46:59.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.076+0000 7f9425200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94200b1f20 con 0x7f942015cda0 2024-03-23T03:46:59.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.076+0000 7f9425200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94200d0c20 con 0x7f94200b5a30 2024-03-23T03:46:59.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.076+0000 7f941e800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f94200b4350 0x7f94200d1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38100/0 (socket says 172.21.15.102:38100) 2024-03-23T03:46:59.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.076+0000 7f941e800640 1 -- 172.21.15.102:0/561442508 learned_addr learned my addr 172.21.15.102:0/561442508 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:46:59.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.076+0000 7f941d400640 1 --2- 172.21.15.102:0/561442508 >> v2:172.21.15.173:3300/0 conn(0x7f94200b5a30 0x7f94200b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.077+0000 7f941ca00640 1 -- 172.21.15.102:0/561442508 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046792902 0 0) 0x7f94200d1830 con 0x7f94200b4350 2024-03-23T03:46:59.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.077+0000 7f941ca00640 1 -- 172.21.15.102:0/561442508 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9404003520 con 0x7f94200b4350 2024-03-23T03:46:59.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.077+0000 7f941ca00640 1 -- 172.21.15.102:0/561442508 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 950464957 0 0) 0x7f9404003520 con 0x7f94200b4350 2024-03-23T03:46:59.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.077+0000 7f941ca00640 1 -- 172.21.15.102:0/561442508 >> v2:172.21.15.173:3300/0 conn(0x7f94200b5a30 msgr2=0x7f94200b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:59.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.077+0000 7f941ca00640 1 --2- 172.21.15.102:0/561442508 >> v2:172.21.15.173:3300/0 conn(0x7f94200b5a30 0x7f94200b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.077+0000 7f941ca00640 1 -- 172.21.15.102:0/561442508 >> v1:172.21.15.173:6789/0 conn(0x7f942015cda0 legacy=0x7f94200cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:59.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.077+0000 7f941ca00640 1 -- 172.21.15.102:0/561442508 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94200d0f00 con 0x7f94200b4350 2024-03-23T03:46:59.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.077+0000 7f941ca00640 1 -- 172.21.15.102:0/561442508 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f94180019f0 con 0x7f94200b4350 2024-03-23T03:46:59.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.077+0000 7f941ca00640 1 -- 172.21.15.102:0/561442508 >> v1:172.21.15.102:6789/0 conn(0x7f94200b4350 legacy=0x7f94200d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:59.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.077+0000 7f941ca00640 1 --2- 172.21.15.102:0/561442508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9404003970 0x7f9404003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.077+0000 7f941ca00640 1 --2- 172.21.15.102:0/561442508 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94040047c0 0x7f9404004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.077+0000 7f941ca00640 1 --2- 172.21.15.102:0/561442508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9404005180 0x7f940400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.077+0000 7f941ca00640 1 -- 172.21.15.102:0/561442508 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94200d0c20 con 0x7f9404003970 2024-03-23T03:46:59.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.077+0000 7f941ca00640 1 -- 172.21.15.102:0/561442508 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940400c020 con 0x7f9404005180 2024-03-23T03:46:59.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.077+0000 7f941ca00640 1 -- 172.21.15.102:0/561442508 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f940400c1e0 con 0x7f94040047c0 2024-03-23T03:46:59.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.077+0000 7f941de00640 1 --2- 172.21.15.102:0/561442508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9404003970 0x7f9404003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.078+0000 7f941d400640 1 --2- 172.21.15.102:0/561442508 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94040047c0 0x7f9404004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.078+0000 7f941de00640 1 -- 172.21.15.102:0/561442508 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94040047c0 msgr2=0x7f9404004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:59.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.078+0000 7f941de00640 1 --2- 172.21.15.102:0/561442508 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94040047c0 0x7f9404004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.078+0000 7f941de00640 1 -- 172.21.15.102:0/561442508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9404005180 msgr2=0x7f940400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:59.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.078+0000 7f941de00640 1 --2- 172.21.15.102:0/561442508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9404005180 0x7f940400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.078+0000 7f941de00640 1 -- 172.21.15.102:0/561442508 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9404003520 con 0x7f9404003970 2024-03-23T03:46:59.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.078+0000 7f941e800640 1 -- 172.21.15.102:0/561442508 reap_dead start 2024-03-23T03:46:59.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.079+0000 7f941de00640 1 --2- 172.21.15.102:0/561442508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9404003970 0x7f9404003d90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9410001750 tx=0x7f94040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=276c8bb570deeeb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:59.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.079+0000 7f941ca00640 1 -- 172.21.15.102:0/561442508 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94100052d0 con 0x7f9404003970 2024-03-23T03:46:59.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.079+0000 7f941ca00640 1 -- 172.21.15.102:0/561442508 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f941002e050 con 0x7f9404003970 2024-03-23T03:46:59.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.079+0000 7f941ca00640 1 -- 172.21.15.102:0/561442508 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9410003830 con 0x7f9404003970 2024-03-23T03:46:59.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.079+0000 7f9425200640 1 -- 172.21.15.102:0/561442508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9404003970 msgr2=0x7f9404003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:59.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.079+0000 7f9425200640 1 --2- 172.21.15.102:0/561442508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9404003970 0x7f9404003d90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9410001750 tx=0x7f94040057f0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.080+0000 7f9425200640 1 -- 172.21.15.102:0/561442508 shutdown_connections 2024-03-23T03:46:59.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.080+0000 7f9425200640 1 --2- 172.21.15.102:0/561442508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9404003970 0x7f9404003d90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.080+0000 7f9425200640 1 -- 172.21.15.102:0/561442508 >> 172.21.15.102:0/561442508 conn(0x7f94200cf350 msgr2=0x7f94200cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:59.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.080+0000 7f9425200640 1 -- 172.21.15.102:0/561442508 shutdown_connections 2024-03-23T03:46:59.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.080+0000 7f9425200640 1 -- 172.21.15.102:0/561442508 wait complete. 2024-03-23T03:46:59.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.081+0000 7f9425200640 1 Processor -- start 2024-03-23T03:46:59.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.081+0000 7f9425200640 1 -- start start 2024-03-23T03:46:59.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.082+0000 7f9425200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9404003970 0x7f942015d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.082+0000 7f9425200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94200b0d10 0x7f94200c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.082+0000 7f9425200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94200ccf00 0x7f94200cd3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.082+0000 7f9425200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94200cd8f0 con 0x7f94200b0d10 2024-03-23T03:46:59.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.082+0000 7f9425200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94200a9270 con 0x7f9404003970 2024-03-23T03:46:59.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.082+0000 7f9425200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94200b1170 con 0x7f94200ccf00 2024-03-23T03:46:59.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.083+0000 7f941d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94200b0d10 0x7f94200c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.083+0000 7f941d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94200b0d10 0x7f94200c82d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50786/0 (socket says 172.21.15.102:50786) 2024-03-23T03:46:59.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.083+0000 7f941d400640 1 -- 172.21.15.102:0/3377676132 learned_addr learned my addr 172.21.15.102:0/3377676132 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:59.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.083+0000 7f941de00640 1 --2- 172.21.15.102:0/3377676132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9404003970 0x7f942015d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.083+0000 7f941d400640 1 -- 172.21.15.102:0/3377676132 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94200ccf00 msgr2=0x7f94200cd3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:59.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.083+0000 7f941d400640 1 --2- 172.21.15.102:0/3377676132 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94200ccf00 0x7f94200cd3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.083+0000 7f941d400640 1 -- 172.21.15.102:0/3377676132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9404003970 msgr2=0x7f942015d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:59.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.083+0000 7f941d400640 1 --2- 172.21.15.102:0/3377676132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9404003970 0x7f942015d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.083+0000 7f941d400640 1 -- 172.21.15.102:0/3377676132 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94200c8810 con 0x7f94200b0d10 2024-03-23T03:46:59.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.083+0000 7f941de00640 1 --2- 172.21.15.102:0/3377676132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9404003970 0x7f942015d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:46:59.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.083+0000 7f941d400640 1 --2- 172.21.15.102:0/3377676132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94200b0d10 0x7f94200c82d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9408010790 tx=0x7f9408010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:59.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.083+0000 7f9416000640 1 -- 172.21.15.102:0/3377676132 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408009630 con 0x7f94200b0d10 2024-03-23T03:46:59.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.084+0000 7f9416000640 1 -- 172.21.15.102:0/3377676132 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f940800eb60 con 0x7f94200b0d10 2024-03-23T03:46:59.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.084+0000 7f9425200640 1 -- 172.21.15.102:0/3377676132 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f942015d840 con 0x7f94200b0d10 2024-03-23T03:46:59.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.084+0000 7f9416000640 1 -- 172.21.15.102:0/3377676132 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408010ea0 con 0x7f94200b0d10 2024-03-23T03:46:59.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.084+0000 7f9425200640 1 -- 172.21.15.102:0/3377676132 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f942015de00 con 0x7f94200b0d10 2024-03-23T03:46:59.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.085+0000 7f9425200640 1 -- 172.21.15.102:0/3377676132 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9420073390 con 0x7f94200b0d10 2024-03-23T03:46:59.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.086+0000 7f9416000640 1 -- 172.21.15.102:0/3377676132 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f940801c400 con 0x7f94200b0d10 2024-03-23T03:46:59.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.086+0000 7f9416000640 1 --2- 172.21.15.102:0/3377676132 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93fc06ac30 0x7f93fc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.086+0000 7f9416000640 1 -- 172.21.15.102:0/3377676132 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9408089b00 con 0x7f94200b0d10 2024-03-23T03:46:59.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.087+0000 7f941de00640 1 --2- 172.21.15.102:0/3377676132 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93fc06ac30 0x7f93fc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.087+0000 7f941de00640 1 --2- 172.21.15.102:0/3377676132 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93fc06ac30 0x7f93fc06d0f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9410033000 tx=0x7f9410009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:59.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.091+0000 7f9416000640 1 -- 172.21.15.102:0/3377676132 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f940808be60 con 0x7f94200b0d10 2024-03-23T03:46:59.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.284+0000 7f9425200640 1 -- 172.21.15.102:0/3377676132 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94200b12b0 con 0x7f94200b0d10 2024-03-23T03:46:59.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.287+0000 7f9416000640 1 -- 172.21.15.102:0/3377676132 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f9408054a00 con 0x7f94200b0d10 2024-03-23T03:46:59.288 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:46:59.288 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:58.410551+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:46:59.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.289+0000 7f9425200640 1 -- 172.21.15.102:0/3377676132 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93fc06ac30 msgr2=0x7f93fc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:59.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.289+0000 7f9425200640 1 --2- 172.21.15.102:0/3377676132 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93fc06ac30 0x7f93fc06d0f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9410033000 tx=0x7f9410009030 comp rx=0 tx=0).stop 2024-03-23T03:46:59.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.289+0000 7f9425200640 1 -- 172.21.15.102:0/3377676132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94200b0d10 msgr2=0x7f94200c82d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:59.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.289+0000 7f9425200640 1 --2- 172.21.15.102:0/3377676132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94200b0d10 0x7f94200c82d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9408010790 tx=0x7f9408010c50 comp rx=0 tx=0).stop 2024-03-23T03:46:59.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.289+0000 7f9425200640 1 -- 172.21.15.102:0/3377676132 shutdown_connections 2024-03-23T03:46:59.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.290+0000 7f9425200640 1 --2- 172.21.15.102:0/3377676132 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93fc06ac30 0x7f93fc06d0f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.290+0000 7f9425200640 1 --2- 172.21.15.102:0/3377676132 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94200ccf00 0x7f94200cd3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.290+0000 7f9425200640 1 --2- 172.21.15.102:0/3377676132 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94200b0d10 0x7f94200c82d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.290+0000 7f9425200640 1 --2- 172.21.15.102:0/3377676132 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9404003970 0x7f942015d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.290+0000 7f9425200640 1 -- 172.21.15.102:0/3377676132 >> 172.21.15.102:0/3377676132 conn(0x7f94200cf350 msgr2=0x7f94200cb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:59.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.290+0000 7f9425200640 1 -- 172.21.15.102:0/3377676132 shutdown_connections 2024-03-23T03:46:59.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.290+0000 7f9425200640 1 -- 172.21.15.102:0/3377676132 wait complete. 2024-03-23T03:46:59.308 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:46:59.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.517+0000 7feb0d200640 1 Processor -- start 2024-03-23T03:46:59.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.517+0000 7feb0d200640 1 -- start start 2024-03-23T03:46:59.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.518+0000 7feb0d200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7feb08143930 0x7feb080cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.518+0000 7feb0d200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feb080cdc50 0x7feb08070020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.518+0000 7feb0d200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0815b2b0 con 0x7feb08143460 2024-03-23T03:46:59.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.518+0000 7feb0d200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb080cc6b0 con 0x7feb080cdc50 2024-03-23T03:46:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.518+0000 7feb0d200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb080cc950 con 0x7feb08143930 2024-03-23T03:46:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.519+0000 7feb07e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feb080cdc50 0x7feb08070020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.519+0000 7feb07e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feb080cdc50 0x7feb08070020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50802/0 (socket says 172.21.15.102:50802) 2024-03-23T03:46:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.519+0000 7feb07e00640 1 -- 172.21.15.102:0/3326682702 learned_addr learned my addr 172.21.15.102:0/3326682702 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.519+0000 7feafea00640 1 --2- 172.21.15.102:0/3326682702 >> v2:172.21.15.173:3300/0 conn(0x7feb08143930 0x7feb080cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.519+0000 7feb07e00640 1 -- 172.21.15.102:0/3326682702 >> v2:172.21.15.173:3300/0 conn(0x7feb08143930 msgr2=0x7feb080cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.519+0000 7feb07e00640 1 --2- 172.21.15.102:0/3326682702 >> v2:172.21.15.173:3300/0 conn(0x7feb08143930 0x7feb080cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.519+0000 7feb07e00640 1 -- 172.21.15.102:0/3326682702 >> v1:172.21.15.173:6789/0 conn(0x7feb08143460 legacy=0x7feb0815aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:59.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.519+0000 7feb07e00640 1 -- 172.21.15.102:0/3326682702 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb080ccc30 con 0x7feb080cdc50 2024-03-23T03:46:59.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.520+0000 7feb07e00640 1 --2- 172.21.15.102:0/3326682702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb080cdc50 0x7feb08070020 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7feb0806ace0 tx=0x7feb0002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87b4c69e3be69a6a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:59.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.520+0000 7feb06a00640 1 -- 172.21.15.102:0/3326682702 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0002cb20 con 0x7feb080cdc50 2024-03-23T03:46:59.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.521+0000 7feb06a00640 1 -- 172.21.15.102:0/3326682702 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0002cc80 con 0x7feb080cdc50 2024-03-23T03:46:59.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.521+0000 7feb06a00640 1 -- 172.21.15.102:0/3326682702 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb000343f0 con 0x7feb080cdc50 2024-03-23T03:46:59.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.521+0000 7feb0d200640 1 -- 172.21.15.102:0/3326682702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb080cdc50 msgr2=0x7feb08070020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:59.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.521+0000 7feb0d200640 1 --2- 172.21.15.102:0/3326682702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb080cdc50 0x7feb08070020 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7feb0806ace0 tx=0x7feb0002bf80 comp rx=0 tx=0).stop 2024-03-23T03:46:59.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.521+0000 7feb0d200640 1 -- 172.21.15.102:0/3326682702 shutdown_connections 2024-03-23T03:46:59.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.521+0000 7feb0d200640 1 --2- 172.21.15.102:0/3326682702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb080cdc50 0x7feb08070020 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.521+0000 7feb0d200640 1 --2- 172.21.15.102:0/3326682702 >> v2:172.21.15.173:3300/0 conn(0x7feb08143930 0x7feb080cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.521+0000 7feb0d200640 1 -- 172.21.15.102:0/3326682702 >> 172.21.15.102:0/3326682702 conn(0x7feb08065df0 msgr2=0x7feb08069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:59.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.521+0000 7feb0d200640 1 -- 172.21.15.102:0/3326682702 shutdown_connections 2024-03-23T03:46:59.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.521+0000 7feb0d200640 1 -- 172.21.15.102:0/3326682702 wait complete. 2024-03-23T03:46:59.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.523+0000 7feb0d200640 1 Processor -- start 2024-03-23T03:46:59.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.523+0000 7feb0d200640 1 -- start start 2024-03-23T03:46:59.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.524+0000 7feb0d200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feb080cdc50 0x7feb08142880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.524+0000 7feb0d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb08143460 0x7feb081598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.524+0000 7feb0d200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb08143930 0x7feb0813e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.524+0000 7feb0d200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0813a070 con 0x7feb08143930 2024-03-23T03:46:59.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.524+0000 7feb0d200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb08141c20 con 0x7feb08143460 2024-03-23T03:46:59.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.524+0000 7feb0d200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0815a3a0 con 0x7feb080cdc50 2024-03-23T03:46:59.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.524+0000 7feb07e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb08143930 0x7feb0813e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.524+0000 7feb07e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb08143930 0x7feb0813e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50816/0 (socket says 172.21.15.102:50816) 2024-03-23T03:46:59.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.524+0000 7feb07e00640 1 -- 172.21.15.102:0/1328299690 learned_addr learned my addr 172.21.15.102:0/1328299690 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:59.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.525+0000 7feafea00640 1 --2- 172.21.15.102:0/1328299690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb08143460 0x7feb081598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.525+0000 7feb07400640 1 --2- 172.21.15.102:0/1328299690 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feb080cdc50 0x7feb08142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.525+0000 7feb07e00640 1 -- 172.21.15.102:0/1328299690 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feb080cdc50 msgr2=0x7feb08142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:59.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.525+0000 7feb07e00640 1 --2- 172.21.15.102:0/1328299690 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feb080cdc50 0x7feb08142880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.525+0000 7feb07e00640 1 -- 172.21.15.102:0/1328299690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb08143460 msgr2=0x7feb081598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:59.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.525+0000 7feb07e00640 1 --2- 172.21.15.102:0/1328299690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb08143460 0x7feb081598a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.525+0000 7feb07e00640 1 -- 172.21.15.102:0/1328299690 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb00009b30 con 0x7feb08143930 2024-03-23T03:46:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.525+0000 7feb07e00640 1 --2- 172.21.15.102:0/1328299690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb08143930 0x7feb0813e520 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7feb00009f40 tx=0x7feb00004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.525+0000 7feaffe00640 1 -- 172.21.15.102:0/1328299690 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb00009510 con 0x7feb08143930 2024-03-23T03:46:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.525+0000 7feaffe00640 1 -- 172.21.15.102:0/1328299690 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb00007e50 con 0x7feb08143930 2024-03-23T03:46:59.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.525+0000 7feaffe00640 1 -- 172.21.15.102:0/1328299690 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0003e670 con 0x7feb08143930 2024-03-23T03:46:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.525+0000 7feb0d200640 1 -- 172.21.15.102:0/1328299690 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb08141010 con 0x7feb08143930 2024-03-23T03:46:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.526+0000 7feb0d200640 1 -- 172.21.15.102:0/1328299690 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb081415d0 con 0x7feb08143930 2024-03-23T03:46:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.527+0000 7feb0d200640 1 -- 172.21.15.102:0/1328299690 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb081373e0 con 0x7feb08143930 2024-03-23T03:46:59.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.528+0000 7feaffe00640 1 -- 172.21.15.102:0/1328299690 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7feb0003e7d0 con 0x7feb08143930 2024-03-23T03:46:59.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.528+0000 7feaffe00640 1 --2- 172.21.15.102:0/1328299690 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7feae406ac00 0x7feae406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.529+0000 7feaffe00640 1 -- 172.21.15.102:0/1328299690 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7feb000ac110 con 0x7feb08143930 2024-03-23T03:46:59.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.529+0000 7feb07400640 1 --2- 172.21.15.102:0/1328299690 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7feae406ac00 0x7feae406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.529+0000 7feb07400640 1 --2- 172.21.15.102:0/1328299690 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7feae406ac00 0x7feae406d0c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7feaf80015d0 tx=0x7feaf8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:59.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.535+0000 7feaffe00640 1 -- 172.21.15.102:0/1328299690 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feb081373e0 con 0x7feb08143930 2024-03-23T03:46:59.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.734+0000 7feb0d200640 1 -- 172.21.15.102:0/1328299690 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb081373e0 con 0x7feb08143930 2024-03-23T03:46:59.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.737+0000 7feaffe00640 1 -- 172.21.15.102:0/1328299690 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17926 (secure 0 0 0) 0x7feb081373e0 con 0x7feb08143930 2024-03-23T03:46:59.739 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:46:59.739 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:58.410551+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:46:59.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.740+0000 7feb0d200640 1 -- 172.21.15.102:0/1328299690 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7feae406ac00 msgr2=0x7feae406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:59.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.740+0000 7feb0d200640 1 --2- 172.21.15.102:0/1328299690 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7feae406ac00 0x7feae406d0c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7feaf80015d0 tx=0x7feaf8009040 comp rx=0 tx=0).stop 2024-03-23T03:46:59.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.740+0000 7feb0d200640 1 -- 172.21.15.102:0/1328299690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb08143930 msgr2=0x7feb0813e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:59.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.741+0000 7feb0d200640 1 --2- 172.21.15.102:0/1328299690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb08143930 0x7feb0813e520 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7feb00009f40 tx=0x7feb00004a40 comp rx=0 tx=0).stop 2024-03-23T03:46:59.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.741+0000 7feb0d200640 1 -- 172.21.15.102:0/1328299690 shutdown_connections 2024-03-23T03:46:59.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.741+0000 7feb0d200640 1 --2- 172.21.15.102:0/1328299690 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7feae406ac00 0x7feae406d0c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.741+0000 7feb0d200640 1 --2- 172.21.15.102:0/1328299690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb08143930 0x7feb0813e520 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.741+0000 7feb0d200640 1 --2- 172.21.15.102:0/1328299690 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb08143460 0x7feb081598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.741+0000 7feb0d200640 1 --2- 172.21.15.102:0/1328299690 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feb080cdc50 0x7feb08142880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.741+0000 7feb0d200640 1 -- 172.21.15.102:0/1328299690 >> 172.21.15.102:0/1328299690 conn(0x7feb08065df0 msgr2=0x7feb080aefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:59.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.741+0000 7feb0d200640 1 -- 172.21.15.102:0/1328299690 shutdown_connections 2024-03-23T03:46:59.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.741+0000 7feb0d200640 1 -- 172.21.15.102:0/1328299690 wait complete. 2024-03-23T03:46:59.774 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:46:59.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.976+0000 7f0ec6c00640 1 Processor -- start 2024-03-23T03:46:59.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.977+0000 7f0ec6c00640 1 -- start start 2024-03-23T03:46:59.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.978+0000 7f0ec6c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0ec015cda0 0x7f0ec00cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.978+0000 7f0ec6c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0ec00b5a30 0x7f0ec00b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.978+0000 7f0ec6c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f0ec00b4350 0x7f0ec00b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.978+0000 7f0ec6c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec00b4d20 con 0x7f0ec00b5a30 2024-03-23T03:46:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.978+0000 7f0ec6c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ec00d1830 con 0x7f0ec00b4350 2024-03-23T03:46:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.978+0000 7f0ec6c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec00b1f20 con 0x7f0ec015cda0 2024-03-23T03:46:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.978+0000 7f0ebf400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0ec00b5a30 0x7f0ec00b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.978+0000 7f0ebf400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0ec00b5a30 0x7f0ec00b5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50820/0 (socket says 172.21.15.102:50820) 2024-03-23T03:46:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.978+0000 7f0ebf400640 1 -- 172.21.15.102:0/2955132969 learned_addr learned my addr 172.21.15.102:0/2955132969 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.979+0000 7f0ec4c00640 1 --2- 172.21.15.102:0/2955132969 >> v2:172.21.15.173:3300/0 conn(0x7f0ec00b4350 0x7f0ec00b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.979+0000 7f0ebfe00640 1 --2- 172.21.15.102:0/2955132969 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0ec015cda0 0x7f0ec00cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.979+0000 7f0ebf400640 1 -- 172.21.15.102:0/2955132969 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0ec015cda0 msgr2=0x7f0ec00cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.979+0000 7f0ebf400640 1 --2- 172.21.15.102:0/2955132969 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0ec015cda0 0x7f0ec00cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.979+0000 7f0ebf400640 1 -- 172.21.15.102:0/2955132969 >> v2:172.21.15.173:3300/0 conn(0x7f0ec00b4350 msgr2=0x7f0ec00b47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:46:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.979+0000 7f0ebf400640 1 --2- 172.21.15.102:0/2955132969 >> v2:172.21.15.173:3300/0 conn(0x7f0ec00b4350 0x7f0ec00b47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.979+0000 7f0ebf400640 1 -- 172.21.15.102:0/2955132969 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec00d0c20 con 0x7f0ec00b5a30 2024-03-23T03:46:59.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.979+0000 7f0ebfe00640 1 --2- 172.21.15.102:0/2955132969 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0ec015cda0 0x7f0ec00cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:46:59.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.979+0000 7f0ec4c00640 1 --2- 172.21.15.102:0/2955132969 >> v2:172.21.15.173:3300/0 conn(0x7f0ec00b4350 0x7f0ec00b47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:46:59.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.980+0000 7f0ebf400640 1 --2- 172.21.15.102:0/2955132969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ec00b5a30 0x7f0ec00b5e90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0ec008bc00 tx=0x7f0ea802b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daad0d0ec3b7e422 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:59.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.980+0000 7f0ebea00640 1 -- 172.21.15.102:0/2955132969 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea8031880 con 0x7f0ec00b5a30 2024-03-23T03:46:59.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.980+0000 7f0ebea00640 1 -- 172.21.15.102:0/2955132969 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea80319e0 con 0x7f0ec00b5a30 2024-03-23T03:46:59.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.980+0000 7f0ebea00640 1 -- 172.21.15.102:0/2955132969 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea8031d70 con 0x7f0ec00b5a30 2024-03-23T03:46:59.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.981+0000 7f0ec6c00640 1 -- 172.21.15.102:0/2955132969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ec00b5a30 msgr2=0x7f0ec00b5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:59.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.981+0000 7f0ec6c00640 1 --2- 172.21.15.102:0/2955132969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ec00b5a30 0x7f0ec00b5e90 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f0ec008bc00 tx=0x7f0ea802b920 comp rx=0 tx=0).stop 2024-03-23T03:46:59.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.981+0000 7f0ec6c00640 1 -- 172.21.15.102:0/2955132969 shutdown_connections 2024-03-23T03:46:59.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.981+0000 7f0ec6c00640 1 --2- 172.21.15.102:0/2955132969 >> v2:172.21.15.173:3300/0 conn(0x7f0ec00b4350 0x7f0ec00b47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.981+0000 7f0ec6c00640 1 --2- 172.21.15.102:0/2955132969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ec00b5a30 0x7f0ec00b5e90 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.981+0000 7f0ec6c00640 1 --2- 172.21.15.102:0/2955132969 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0ec015cda0 0x7f0ec00cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.981+0000 7f0ec6c00640 1 -- 172.21.15.102:0/2955132969 >> 172.21.15.102:0/2955132969 conn(0x7f0ec00cf350 msgr2=0x7f0ec00cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:46:59.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.981+0000 7f0ec6c00640 1 -- 172.21.15.102:0/2955132969 shutdown_connections 2024-03-23T03:46:59.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.981+0000 7f0ec6c00640 1 -- 172.21.15.102:0/2955132969 wait complete. 2024-03-23T03:46:59.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.983+0000 7f0ec6c00640 1 Processor -- start 2024-03-23T03:46:59.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.983+0000 7f0ec6c00640 1 -- start start 2024-03-23T03:46:59.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.984+0000 7f0ec6c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ec00b4350 0x7f0ec00c6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.984+0000 7f0ec6c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0ec00b5a30 0x7f0ec01f19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.984+0000 7f0ec6c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ec015cda0 0x7f0ec00ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.984+0000 7f0ec6c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec0157d40 con 0x7f0ec015cda0 2024-03-23T03:46:59.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.984+0000 7f0ec6c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec00c8d50 con 0x7f0ec00b4350 2024-03-23T03:46:59.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.984+0000 7f0ec6c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec00c9030 con 0x7f0ec00b5a30 2024-03-23T03:46:59.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.984+0000 7f0ec4c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ec015cda0 0x7f0ec00ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.984+0000 7f0ec4c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ec015cda0 0x7f0ec00ca9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50828/0 (socket says 172.21.15.102:50828) 2024-03-23T03:46:59.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.984+0000 7f0ec4c00640 1 -- 172.21.15.102:0/3563103789 learned_addr learned my addr 172.21.15.102:0/3563103789 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:46:59.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.984+0000 7f0ebf400640 1 --2- 172.21.15.102:0/3563103789 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0ec00b5a30 0x7f0ec01f19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.984+0000 7f0ebfe00640 1 --2- 172.21.15.102:0/3563103789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ec00b4350 0x7f0ec00c6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.985+0000 7f0ec4c00640 1 -- 172.21.15.102:0/3563103789 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0ec00b5a30 msgr2=0x7f0ec01f19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:59.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.985+0000 7f0ec4c00640 1 --2- 172.21.15.102:0/3563103789 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0ec00b5a30 0x7f0ec01f19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.985+0000 7f0ec4c00640 1 -- 172.21.15.102:0/3563103789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ec00b4350 msgr2=0x7f0ec00c6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:46:59.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.985+0000 7f0ec4c00640 1 --2- 172.21.15.102:0/3563103789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ec00b4350 0x7f0ec00c6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:46:59.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.985+0000 7f0ec4c00640 1 -- 172.21.15.102:0/3563103789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea80094d0 con 0x7f0ec015cda0 2024-03-23T03:46:59.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.985+0000 7f0ebf400640 1 --2- 172.21.15.102:0/3563103789 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0ec00b5a30 0x7f0ec01f19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:46:59.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.985+0000 7f0ec4c00640 1 --2- 172.21.15.102:0/3563103789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ec015cda0 0x7f0ec00ca9f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800cd10 tx=0x7f0eb800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:59.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.985+0000 7f0eb7e00640 1 -- 172.21.15.102:0/3563103789 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb800f680 con 0x7f0ec015cda0 2024-03-23T03:46:59.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.985+0000 7f0eb7e00640 1 -- 172.21.15.102:0/3563103789 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0eb800f800 con 0x7f0ec015cda0 2024-03-23T03:46:59.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.985+0000 7f0eb7e00640 1 -- 172.21.15.102:0/3563103789 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb800fbb0 con 0x7f0ec015cda0 2024-03-23T03:46:59.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.985+0000 7f0ec6c00640 1 -- 172.21.15.102:0/3563103789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec00c9310 con 0x7f0ec015cda0 2024-03-23T03:46:59.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.985+0000 7f0ec6c00640 1 -- 172.21.15.102:0/3563103789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec01f1ee0 con 0x7f0ec015cda0 2024-03-23T03:46:59.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.986+0000 7f0ec6c00640 1 -- 172.21.15.102:0/3563103789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ec0157e80 con 0x7f0ec015cda0 2024-03-23T03:46:59.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.987+0000 7f0eb7e00640 1 -- 172.21.15.102:0/3563103789 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0eb800b040 con 0x7f0ec015cda0 2024-03-23T03:46:59.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.988+0000 7f0eb7e00640 1 --2- 172.21.15.102:0/3563103789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0e9c06ac00 0x7f0e9c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:46:59.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.988+0000 7f0eb7e00640 1 -- 172.21.15.102:0/3563103789 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0eb808a060 con 0x7f0ec015cda0 2024-03-23T03:46:59.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.988+0000 7f0ebfe00640 1 --2- 172.21.15.102:0/3563103789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0e9c06ac00 0x7f0e9c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:46:59.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.989+0000 7f0ebfe00640 1 --2- 172.21.15.102:0/3563103789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0e9c06ac00 0x7f0e9c06d0c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0008850 tx=0x7f0eb0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:46:59.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:46:59.993+0000 7f0eb7e00640 1 -- 172.21.15.102:0/3563103789 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0eb80bd580 con 0x7f0ec015cda0 2024-03-23T03:47:00.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.223+0000 7f0ec6c00640 1 -- 172.21.15.102:0/3563103789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ec0073390 con 0x7f0ec015cda0 2024-03-23T03:47:00.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.225+0000 7f0eb7e00640 1 -- 172.21.15.102:0/3563103789 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4739 (secure 0 0 0) 0x7f0eb801d020 con 0x7f0ec015cda0 2024-03-23T03:47:00.225 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:00.226 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/891818172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":891818172},{"type":"v1","addr":"172.21.15.102:6836","nonce":891818172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2404489233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2404489233},{"type":"v1","addr":"172.21.15.102:6837","nonce":2404489233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/1839967745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":1839967745},{"type":"v1","addr":"172.21.15.173:6835","nonce":1839967745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:46:53.912287+0000","modified":"2024-03-23T03:46:58.411062+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6834/1462637509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1462637509},{"type":"v1","addr":"172.21.15.173:6834","nonce":1462637509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-23T03:47:00.226 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 6 2024-03-23T03:47:00.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.228+0000 7f0ec6c00640 1 -- 172.21.15.102:0/3563103789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0e9c06ac00 msgr2=0x7f0e9c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:00.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.228+0000 7f0ec6c00640 1 --2- 172.21.15.102:0/3563103789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0e9c06ac00 0x7f0e9c06d0c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0008850 tx=0x7f0eb0008040 comp rx=0 tx=0).stop 2024-03-23T03:47:00.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.228+0000 7f0ec6c00640 1 -- 172.21.15.102:0/3563103789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ec015cda0 msgr2=0x7f0ec00ca9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:00.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.228+0000 7f0ec6c00640 1 --2- 172.21.15.102:0/3563103789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ec015cda0 0x7f0ec00ca9f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0eb800cd10 tx=0x7f0eb800a650 comp rx=0 tx=0).stop 2024-03-23T03:47:00.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.228+0000 7f0ec6c00640 1 -- 172.21.15.102:0/3563103789 shutdown_connections 2024-03-23T03:47:00.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.228+0000 7f0ec6c00640 1 --2- 172.21.15.102:0/3563103789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0e9c06ac00 0x7f0e9c06d0c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:00.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.228+0000 7f0ec6c00640 1 --2- 172.21.15.102:0/3563103789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ec015cda0 0x7f0ec00ca9f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:00.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.228+0000 7f0ec6c00640 1 --2- 172.21.15.102:0/3563103789 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0ec00b5a30 0x7f0ec01f19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:00.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.228+0000 7f0ec6c00640 1 --2- 172.21.15.102:0/3563103789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ec00b4350 0x7f0ec00c6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:00.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.228+0000 7f0ec6c00640 1 -- 172.21.15.102:0/3563103789 >> 172.21.15.102:0/3563103789 conn(0x7f0ec00cf350 msgr2=0x7f0ec00cc350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:00.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.228+0000 7f0ec6c00640 1 -- 172.21.15.102:0/3563103789 shutdown_connections 2024-03-23T03:47:00.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.228+0000 7f0ec6c00640 1 -- 172.21.15.102:0/3563103789 wait complete. 2024-03-23T03:47:00.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:00.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.448+0000 7f4a5aa00640 1 Processor -- start 2024-03-23T03:47:00.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.449+0000 7f4a5aa00640 1 -- start start 2024-03-23T03:47:00.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.450+0000 7f4a5aa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4a54098a40 0x7f4a54098e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:00.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.450+0000 7f4a5aa00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f4a5415e420 0x7f4a5406b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:00.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.450+0000 7f4a5aa00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a540ce980 con 0x7f4a5415cd10 2024-03-23T03:47:00.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.450+0000 7f4a5aa00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a540cef20 con 0x7f4a54098a40 2024-03-23T03:47:00.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.450+0000 7f4a5aa00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a54099360 con 0x7f4a5415e420 2024-03-23T03:47:00.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.451+0000 7f4a53e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4a54098a40 0x7f4a54098e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:00.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.451+0000 7f4a53e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4a54098a40 0x7f4a54098e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50842/0 (socket says 172.21.15.102:50842) 2024-03-23T03:47:00.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.451+0000 7f4a53e00640 1 -- 172.21.15.102:0/944077354 learned_addr learned my addr 172.21.15.102:0/944077354 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:00.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.451+0000 7f4a58a00640 1 --2- 172.21.15.102:0/944077354 >> v2:172.21.15.173:3300/0 conn(0x7f4a5415e420 0x7f4a5406b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:00.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.451+0000 7f4a53e00640 1 -- 172.21.15.102:0/944077354 >> v2:172.21.15.173:3300/0 conn(0x7f4a5415e420 msgr2=0x7f4a5406b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:00.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.451+0000 7f4a53e00640 1 --2- 172.21.15.102:0/944077354 >> v2:172.21.15.173:3300/0 conn(0x7f4a5415e420 0x7f4a5406b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:00.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.451+0000 7f4a53e00640 1 -- 172.21.15.102:0/944077354 >> v1:172.21.15.173:6789/0 conn(0x7f4a5415cd10 legacy=0x7f4a5406ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:00.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.451+0000 7f4a53e00640 1 -- 172.21.15.102:0/944077354 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5406bbf0 con 0x7f4a54098a40 2024-03-23T03:47:00.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.452+0000 7f4a53e00640 1 --2- 172.21.15.102:0/944077354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a54098a40 0x7f4a54098e20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4a540d0650 tx=0x7f4a4402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cd071203e286c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:00.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.453+0000 7f4a52a00640 1 -- 172.21.15.102:0/944077354 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a44031870 con 0x7f4a54098a40 2024-03-23T03:47:00.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.453+0000 7f4a52a00640 1 -- 172.21.15.102:0/944077354 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a440319d0 con 0x7f4a54098a40 2024-03-23T03:47:00.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.453+0000 7f4a52a00640 1 -- 172.21.15.102:0/944077354 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a44031d60 con 0x7f4a54098a40 2024-03-23T03:47:00.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.454+0000 7f4a5aa00640 1 -- 172.21.15.102:0/944077354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a54098a40 msgr2=0x7f4a54098e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:00.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.454+0000 7f4a5aa00640 1 --2- 172.21.15.102:0/944077354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a54098a40 0x7f4a54098e20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4a540d0650 tx=0x7f4a4402b910 comp rx=0 tx=0).stop 2024-03-23T03:47:00.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.454+0000 7f4a5aa00640 1 -- 172.21.15.102:0/944077354 shutdown_connections 2024-03-23T03:47:00.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.454+0000 7f4a5aa00640 1 --2- 172.21.15.102:0/944077354 >> v2:172.21.15.173:3300/0 conn(0x7f4a5415e420 0x7f4a5406b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:00.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.454+0000 7f4a5aa00640 1 --2- 172.21.15.102:0/944077354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a54098a40 0x7f4a54098e20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:00.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.454+0000 7f4a5aa00640 1 -- 172.21.15.102:0/944077354 >> 172.21.15.102:0/944077354 conn(0x7f4a540b21a0 msgr2=0x7f4a540b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:00.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.454+0000 7f4a5aa00640 1 -- 172.21.15.102:0/944077354 shutdown_connections 2024-03-23T03:47:00.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.454+0000 7f4a5aa00640 1 -- 172.21.15.102:0/944077354 wait complete. 2024-03-23T03:47:00.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.455+0000 7f4a5aa00640 1 Processor -- start 2024-03-23T03:47:00.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.456+0000 7f4a5aa00640 1 -- start start 2024-03-23T03:47:00.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.456+0000 7f4a5aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a54098a40 0x7f4a540c5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:00.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.456+0000 7f4a5aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a5415cd10 0x7f4a54150a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:00.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.456+0000 7f4a5aa00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4a5415e420 0x7f4a5409c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:00.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.456+0000 7f4a5aa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a54158130 con 0x7f4a54098a40 2024-03-23T03:47:00.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.456+0000 7f4a5aa00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a540c41e0 con 0x7f4a5415cd10 2024-03-23T03:47:00.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.456+0000 7f4a5aa00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a540c44c0 con 0x7f4a5415e420 2024-03-23T03:47:00.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.457+0000 7f4a53e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a54098a40 0x7f4a540c5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:00.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.457+0000 7f4a53e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a54098a40 0x7f4a540c5560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50844/0 (socket says 172.21.15.102:50844) 2024-03-23T03:47:00.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.457+0000 7f4a53e00640 1 -- 172.21.15.102:0/3201037234 learned_addr learned my addr 172.21.15.102:0/3201037234 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:00.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.457+0000 7f4a58a00640 1 --2- 172.21.15.102:0/3201037234 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4a5415e420 0x7f4a5409c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:00.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.457+0000 7f4a53400640 1 --2- 172.21.15.102:0/3201037234 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a5415cd10 0x7f4a54150a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:00.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.457+0000 7f4a53e00640 1 -- 172.21.15.102:0/3201037234 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4a5415e420 msgr2=0x7f4a5409c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:00.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.457+0000 7f4a53e00640 1 --2- 172.21.15.102:0/3201037234 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4a5415e420 0x7f4a5409c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:00.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.457+0000 7f4a53e00640 1 -- 172.21.15.102:0/3201037234 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a5415cd10 msgr2=0x7f4a54150a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:00.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.457+0000 7f4a53e00640 1 --2- 172.21.15.102:0/3201037234 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a5415cd10 0x7f4a54150a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:00.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.457+0000 7f4a53e00640 1 -- 172.21.15.102:0/3201037234 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a440094d0 con 0x7f4a54098a40 2024-03-23T03:47:00.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.458+0000 7f4a53e00640 1 --2- 172.21.15.102:0/3201037234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a54098a40 0x7f4a540c5560 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4a44000c00 tx=0x7f4a440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:00.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.458+0000 7f4a4be00640 1 -- 172.21.15.102:0/3201037234 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a4403abe0 con 0x7f4a54098a40 2024-03-23T03:47:00.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.458+0000 7f4a4be00640 1 -- 172.21.15.102:0/3201037234 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a4403ad40 con 0x7f4a54098a40 2024-03-23T03:47:00.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.458+0000 7f4a5aa00640 1 -- 172.21.15.102:0/3201037234 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a540c47a0 con 0x7f4a54098a40 2024-03-23T03:47:00.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.458+0000 7f4a4be00640 1 -- 172.21.15.102:0/3201037234 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a44044610 con 0x7f4a54098a40 2024-03-23T03:47:00.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.458+0000 7f4a5aa00640 1 -- 172.21.15.102:0/3201037234 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a540c4d60 con 0x7f4a54098a40 2024-03-23T03:47:00.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.459+0000 7f4a5aa00640 1 -- 172.21.15.102:0/3201037234 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a54066bb0 con 0x7f4a54098a40 2024-03-23T03:47:00.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.460+0000 7f4a4be00640 1 -- 172.21.15.102:0/3201037234 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4a44044770 con 0x7f4a54098a40 2024-03-23T03:47:00.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.461+0000 7f4a4be00640 1 --2- 172.21.15.102:0/3201037234 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4a3006ac00 0x7f4a3006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:00.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.461+0000 7f4a4be00640 1 -- 172.21.15.102:0/3201037234 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4a4402d040 con 0x7f4a54098a40 2024-03-23T03:47:00.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.461+0000 7f4a53400640 1 --2- 172.21.15.102:0/3201037234 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4a3006ac00 0x7f4a3006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:00.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.462+0000 7f4a53400640 1 --2- 172.21.15.102:0/3201037234 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4a3006ac00 0x7f4a3006d0c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4a5409bff0 tx=0x7f4a3c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:00.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.466+0000 7f4a4be00640 1 -- 172.21.15.102:0/3201037234 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4a44076bb0 con 0x7f4a54098a40 2024-03-23T03:47:00.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.660+0000 7f4a5aa00640 1 -- 172.21.15.102:0/3201037234 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a540b2f90 con 0x7f4a54098a40 2024-03-23T03:47:00.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.663+0000 7f4a4be00640 1 -- 172.21.15.102:0/3201037234 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f4a4407a1d0 con 0x7f4a54098a40 2024-03-23T03:47:00.664 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:00.664 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:58.410551+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:47:00.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.665+0000 7f4a5aa00640 1 -- 172.21.15.102:0/3201037234 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4a3006ac00 msgr2=0x7f4a3006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:00.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.665+0000 7f4a5aa00640 1 --2- 172.21.15.102:0/3201037234 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4a3006ac00 0x7f4a3006d0c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4a5409bff0 tx=0x7f4a3c009040 comp rx=0 tx=0).stop 2024-03-23T03:47:00.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.665+0000 7f4a5aa00640 1 -- 172.21.15.102:0/3201037234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a54098a40 msgr2=0x7f4a540c5560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:00.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.665+0000 7f4a5aa00640 1 --2- 172.21.15.102:0/3201037234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a54098a40 0x7f4a540c5560 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4a44000c00 tx=0x7f4a440046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:00.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.666+0000 7f4a5aa00640 1 -- 172.21.15.102:0/3201037234 shutdown_connections 2024-03-23T03:47:00.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.666+0000 7f4a5aa00640 1 --2- 172.21.15.102:0/3201037234 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4a3006ac00 0x7f4a3006d0c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:00.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.666+0000 7f4a5aa00640 1 --2- 172.21.15.102:0/3201037234 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4a5415e420 0x7f4a5409c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:00.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.666+0000 7f4a5aa00640 1 --2- 172.21.15.102:0/3201037234 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4a5415cd10 0x7f4a54150a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:00.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.666+0000 7f4a5aa00640 1 --2- 172.21.15.102:0/3201037234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a54098a40 0x7f4a540c5560 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:00.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.666+0000 7f4a5aa00640 1 -- 172.21.15.102:0/3201037234 >> 172.21.15.102:0/3201037234 conn(0x7f4a540b21a0 msgr2=0x7f4a540cd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:00.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.666+0000 7f4a5aa00640 1 -- 172.21.15.102:0/3201037234 shutdown_connections 2024-03-23T03:47:00.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:00.666+0000 7f4a5aa00640 1 -- 172.21.15.102:0/3201037234 wait complete. 2024-03-23T03:47:01.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T03:46:53.912287+0000', 'modified': '2024-03-23T03:46:58.411062+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4270': {'gid': 4270, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.173:6834/1462637509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6832', 'nonce': 1462637509}, {'type': 'v1', 'addr': '172.21.15.173:6834', 'nonce': 1462637509}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-03-23T03:47:01.087 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T03:47:01.087 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-23T03:47:01.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.303+0000 7f328b600640 1 Processor -- start 2024-03-23T03:47:01.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.304+0000 7f328b600640 1 -- start start 2024-03-23T03:47:01.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.305+0000 7f328b600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f328413fa80 0x7f328413ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.305+0000 7f328b600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f32840cf040 0x7f32841422d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.305+0000 7f328b600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32841404a0 con 0x7f3284160dd0 2024-03-23T03:47:01.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.305+0000 7f328b600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328406cdc0 con 0x7f328413fa80 2024-03-23T03:47:01.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.305+0000 7f328b600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328406d060 con 0x7f32840cf040 2024-03-23T03:47:01.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.305+0000 7f3288c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f328413fa80 0x7f328413ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.305+0000 7f3288c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f328413fa80 0x7f328413ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50848/0 (socket says 172.21.15.102:50848) 2024-03-23T03:47:01.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.305+0000 7f3288c00640 1 -- 172.21.15.102:0/1738837501 learned_addr learned my addr 172.21.15.102:0/1738837501 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:01.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.306+0000 7f3289600640 1 --2- 172.21.15.102:0/1738837501 >> v2:172.21.15.173:3300/0 conn(0x7f32840cf040 0x7f32841422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.306+0000 7f3288c00640 1 -- 172.21.15.102:0/1738837501 >> v2:172.21.15.173:3300/0 conn(0x7f32840cf040 msgr2=0x7f32841422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:01.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.306+0000 7f3288c00640 1 --2- 172.21.15.102:0/1738837501 >> v2:172.21.15.173:3300/0 conn(0x7f32840cf040 0x7f32841422d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.306+0000 7f3288c00640 1 -- 172.21.15.102:0/1738837501 >> v1:172.21.15.173:6789/0 conn(0x7f3284160dd0 legacy=0x7f32840ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:01.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.306+0000 7f3288c00640 1 -- 172.21.15.102:0/1738837501 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328406d340 con 0x7f328413fa80 2024-03-23T03:47:01.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.307+0000 7f3288c00640 1 --2- 172.21.15.102:0/1738837501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f328413fa80 0x7f328413ff60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f328406c850 tx=0x7f326c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c03612013ae75c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:01.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.307+0000 7f3283e00640 1 -- 172.21.15.102:0/1738837501 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c031870 con 0x7f328413fa80 2024-03-23T03:47:01.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.307+0000 7f3283e00640 1 -- 172.21.15.102:0/1738837501 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326c0319d0 con 0x7f328413fa80 2024-03-23T03:47:01.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.307+0000 7f3283e00640 1 -- 172.21.15.102:0/1738837501 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c031d60 con 0x7f328413fa80 2024-03-23T03:47:01.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.308+0000 7f328b600640 1 -- 172.21.15.102:0/1738837501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f328413fa80 msgr2=0x7f328413ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:01.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.308+0000 7f328b600640 1 --2- 172.21.15.102:0/1738837501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f328413fa80 0x7f328413ff60 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f328406c850 tx=0x7f326c02b910 comp rx=0 tx=0).stop 2024-03-23T03:47:01.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.308+0000 7f328b600640 1 -- 172.21.15.102:0/1738837501 shutdown_connections 2024-03-23T03:47:01.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.308+0000 7f328b600640 1 --2- 172.21.15.102:0/1738837501 >> v2:172.21.15.173:3300/0 conn(0x7f32840cf040 0x7f32841422d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.308+0000 7f328b600640 1 --2- 172.21.15.102:0/1738837501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f328413fa80 0x7f328413ff60 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.308+0000 7f328b600640 1 -- 172.21.15.102:0/1738837501 >> 172.21.15.102:0/1738837501 conn(0x7f328406a0b0 msgr2=0x7f328406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:01.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.308+0000 7f328b600640 1 -- 172.21.15.102:0/1738837501 shutdown_connections 2024-03-23T03:47:01.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.308+0000 7f328b600640 1 -- 172.21.15.102:0/1738837501 wait complete. 2024-03-23T03:47:01.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.311+0000 7f328b600640 1 Processor -- start 2024-03-23T03:47:01.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.311+0000 7f328b600640 1 -- start start 2024-03-23T03:47:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.312+0000 7f328b600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32840cf040 0x7f328413e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.312+0000 7f328b600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f328413fa80 0x7f32840b0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.312+0000 7f328b600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3284160dd0 0x7f32840b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.312+0000 7f328b600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32840b0df0 con 0x7f3284160dd0 2024-03-23T03:47:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.312+0000 7f328b600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32840b12c0 con 0x7f32840cf040 2024-03-23T03:47:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.312+0000 7f328b600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32841e7020 con 0x7f328413fa80 2024-03-23T03:47:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.312+0000 7f3289600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3284160dd0 0x7f32840b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.313+0000 7f3289600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3284160dd0 0x7f32840b08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50862/0 (socket says 172.21.15.102:50862) 2024-03-23T03:47:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.313+0000 7f3289600640 1 -- 172.21.15.102:0/3552538134 learned_addr learned my addr 172.21.15.102:0/3552538134 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.313+0000 7f3288c00640 1 --2- 172.21.15.102:0/3552538134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32840cf040 0x7f328413e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.313+0000 7f327be00640 1 --2- 172.21.15.102:0/3552538134 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f328413fa80 0x7f32840b0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.313+0000 7f3289600640 1 -- 172.21.15.102:0/3552538134 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f328413fa80 msgr2=0x7f32840b0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.313+0000 7f3289600640 1 --2- 172.21.15.102:0/3552538134 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f328413fa80 0x7f32840b0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.313+0000 7f3289600640 1 -- 172.21.15.102:0/3552538134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32840cf040 msgr2=0x7f328413e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.313+0000 7f3289600640 1 --2- 172.21.15.102:0/3552538134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32840cf040 0x7f328413e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.313+0000 7f3289600640 1 -- 172.21.15.102:0/3552538134 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326c0094d0 con 0x7f3284160dd0 2024-03-23T03:47:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.313+0000 7f3289600640 1 --2- 172.21.15.102:0/3552538134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3284160dd0 0x7f32840b08b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f327c007e60 tx=0x7f327c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.313+0000 7f3281600640 1 -- 172.21.15.102:0/3552538134 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c014620 con 0x7f3284160dd0 2024-03-23T03:47:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.313+0000 7f3281600640 1 -- 172.21.15.102:0/3552538134 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327c014780 con 0x7f3284160dd0 2024-03-23T03:47:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.313+0000 7f3281600640 1 -- 172.21.15.102:0/3552538134 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c014b30 con 0x7f3284160dd0 2024-03-23T03:47:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.314+0000 7f328b600640 1 -- 172.21.15.102:0/3552538134 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32841e6a10 con 0x7f3284160dd0 2024-03-23T03:47:01.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.314+0000 7f328b600640 1 -- 172.21.15.102:0/3552538134 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32841e6cb0 con 0x7f3284160dd0 2024-03-23T03:47:01.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.316+0000 7f3281600640 1 -- 172.21.15.102:0/3552538134 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f327c014c90 con 0x7f3284160dd0 2024-03-23T03:47:01.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.317+0000 7f3281600640 1 --2- 172.21.15.102:0/3552538134 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f326006ac00 0x7f326006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.317+0000 7f3281600640 1 -- 172.21.15.102:0/3552538134 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f327c08a080 con 0x7f3284160dd0 2024-03-23T03:47:01.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.317+0000 7f3288c00640 1 --2- 172.21.15.102:0/3552538134 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f326006ac00 0x7f326006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.318+0000 7f328e25e740 1 Processor -- start 2024-03-23T03:47:01.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.318+0000 7f328e25e740 1 -- start start 2024-03-23T03:47:01.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.318+0000 7f3288c00640 1 --2- 172.21.15.102:0/3552538134 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f326006ac00 0x7f326006d0c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f326c036000 tx=0x7f326c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:01.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.319+0000 7f328e25e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08ad00 0x56262c05dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.319+0000 7f328e25e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08b0e0 0x56262c08da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.319+0000 7f328e25e740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c05e1e0 0x56262c08dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.319+0000 7f328e25e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262c0392c0 con 0x56262c08ad00 2024-03-23T03:47:01.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.319+0000 7f328e25e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262bfce8f0 con 0x56262c08b0e0 2024-03-23T03:47:01.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.319+0000 7f328e25e740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x56262bfcc080 con 0x56262c05e1e0 2024-03-23T03:47:01.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.319+0000 7f3288c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08ad00 0x56262c05dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.319+0000 7f3288c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08ad00 0x56262c05dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50864/0 (socket says 172.21.15.102:50864) 2024-03-23T03:47:01.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.319+0000 7f3288c00640 1 -- 172.21.15.102:0/2340127806 learned_addr learned my addr 172.21.15.102:0/2340127806 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:01.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.320+0000 7f3289600640 1 --2- 172.21.15.102:0/2340127806 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c05e1e0 0x56262c08dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.320+0000 7f327be00640 1 --2- 172.21.15.102:0/2340127806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08b0e0 0x56262c08da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.320+0000 7f3288c00640 1 -- 172.21.15.102:0/2340127806 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c05e1e0 msgr2=0x56262c08dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:01.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.320+0000 7f3288c00640 1 --2- 172.21.15.102:0/2340127806 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c05e1e0 0x56262c08dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.320+0000 7f3288c00640 1 -- 172.21.15.102:0/2340127806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08b0e0 msgr2=0x56262c08da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:01.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.320+0000 7f3288c00640 1 --2- 172.21.15.102:0/2340127806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08b0e0 0x56262c08da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.320+0000 7f3288c00640 1 -- 172.21.15.102:0/2340127806 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56262bf92a10 con 0x56262c08ad00 2024-03-23T03:47:01.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.320+0000 7f3288c00640 1 --2- 172.21.15.102:0/2340127806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08ad00 0x56262c05dca0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f326c009930 tx=0x7f326c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1475d27634c253c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:01.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.320+0000 7f3279600640 1 -- 172.21.15.102:0/2340127806 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c04a070 con 0x56262c08ad00 2024-03-23T03:47:01.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.320+0000 7f3279600640 1 -- 172.21.15.102:0/2340127806 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326c03ae70 con 0x56262c08ad00 2024-03-23T03:47:01.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.320+0000 7f3279600640 1 -- 172.21.15.102:0/2340127806 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c04f470 con 0x56262c08ad00 2024-03-23T03:47:01.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.321+0000 7f328e25e740 1 -- 172.21.15.102:0/2340127806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08ad00 msgr2=0x56262c05dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:01.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.321+0000 7f328e25e740 1 --2- 172.21.15.102:0/2340127806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08ad00 0x56262c05dca0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f326c009930 tx=0x7f326c030000 comp rx=0 tx=0).stop 2024-03-23T03:47:01.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.321+0000 7f328e25e740 1 -- 172.21.15.102:0/2340127806 shutdown_connections 2024-03-23T03:47:01.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.321+0000 7f328e25e740 1 --2- 172.21.15.102:0/2340127806 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c05e1e0 0x56262c08dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.321+0000 7f328e25e740 1 --2- 172.21.15.102:0/2340127806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08b0e0 0x56262c08da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.321+0000 7f328e25e740 1 --2- 172.21.15.102:0/2340127806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08ad00 0x56262c05dca0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.321+0000 7f328e25e740 1 -- 172.21.15.102:0/2340127806 >> 172.21.15.102:0/2340127806 conn(0x56262c05ff10 msgr2=0x56262c0a8a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:01.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.321+0000 7f328e25e740 1 -- 172.21.15.102:0/2340127806 shutdown_connections 2024-03-23T03:47:01.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.321+0000 7f328e25e740 1 -- 172.21.15.102:0/2340127806 wait complete. 2024-03-23T03:47:01.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.322+0000 7f328e25e740 1 -- start start 2024-03-23T03:47:01.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.322+0000 7f328e25e740 1 Processor -- start 2024-03-23T03:47:01.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.323+0000 7f328e25e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08ad00 0x56262c07e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.323+0000 7f328e25e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08b0e0 0x56262c0a26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.323+0000 7f328e25e740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c0abfa0 0x56262c0ac420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.323+0000 7f328e25e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262bfc7970 con 0x56262c08ad00 2024-03-23T03:47:01.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.323+0000 7f328e25e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262bf690a0 con 0x56262c08b0e0 2024-03-23T03:47:01.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.323+0000 7f328e25e740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x56262bf68d40 con 0x56262c0abfa0 2024-03-23T03:47:01.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.323+0000 7f328e25e740 10 client.0 _ll_register_callbacks cb 0x56262bfc0fe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T03:47:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.323+0000 7f3288c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08ad00 0x56262c07e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.323+0000 7f3288c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08ad00 0x56262c07e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50880/0 (socket says 172.21.15.102:50880) 2024-03-23T03:47:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.323+0000 7f3288c00640 1 -- 172.21.15.102:0/1161679040 learned_addr learned my addr 172.21.15.102:0/1161679040 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.324+0000 7f327be00640 1 --2- 172.21.15.102:0/1161679040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08b0e0 0x56262c0a26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.324+0000 7f3288c00640 1 -- 172.21.15.102:0/1161679040 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c0abfa0 msgr2=0x56262c0ac420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.324+0000 7f3289600640 1 --2- 172.21.15.102:0/1161679040 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c0abfa0 0x56262c0ac420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.324+0000 7f3288c00640 1 --2- 172.21.15.102:0/1161679040 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c0abfa0 0x56262c0ac420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.324+0000 7f3288c00640 1 -- 172.21.15.102:0/1161679040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08b0e0 msgr2=0x56262c0a26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.324+0000 7f3288c00640 1 --2- 172.21.15.102:0/1161679040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08b0e0 0x56262c0a26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.324+0000 7f3288c00640 1 -- 172.21.15.102:0/1161679040 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328406d340 con 0x56262c08ad00 2024-03-23T03:47:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.324+0000 7f327be00640 1 --2- 172.21.15.102:0/1161679040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08b0e0 0x56262c0a26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.324+0000 7f3288c00640 1 --2- 172.21.15.102:0/1161679040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08ad00 0x56262c07e740 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f326c052040 tx=0x7f326c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1488d06bf26d099 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.324+0000 7f3272000640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-23T03:47:01.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.324+0000 7f3272000640 1 -- 172.21.15.102:0/1161679040 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f326c04a050 con 0x56262c08ad00 2024-03-23T03:47:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.324+0000 7f328e25e740 1 -- 172.21.15.102:0/1161679040 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56262bff1b10 con 0x56262c08ad00 2024-03-23T03:47:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.324+0000 7f3272000640 1 -- 172.21.15.102:0/1161679040 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f326c03f070 con 0x56262c08ad00 2024-03-23T03:47:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.325+0000 7f3272000640 1 -- 172.21.15.102:0/1161679040 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f326c04fcd0 con 0x56262c08ad00 2024-03-23T03:47:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.325+0000 7f328e25e740 10 client.4404 fetch_fsmap learned FSMap version 6 2024-03-23T03:47:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.325+0000 7f328e25e740 1 -- 172.21.15.102:0/1161679040 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x56262bfc5ff0 con 0x56262c08ad00 2024-03-23T03:47:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.325+0000 7f3272000640 1 -- 172.21.15.102:0/1161679040 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56262bff1b10 con 0x56262c08ad00 2024-03-23T03:47:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.325+0000 7f3272000640 1 -- 172.21.15.102:0/1161679040 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 6) v1 ==== 3122+0+0 (secure 0 0 0) 0x7f326c062350 con 0x56262c08ad00 2024-03-23T03:47:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.325+0000 7f328e25e740 10 client.4404 fetch_fsmap finished waiting for FSMap version 6 2024-03-23T03:47:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.325+0000 7f328e25e740 10 client.4404 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-23T03:47:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.325+0000 7f328e25e740 20 client.4404 populate_metadata read hostname 'smithi102' 2024-03-23T03:47:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.325+0000 7f328e25e740 1 --2- 172.21.15.102:0/1161679040 >> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] conn(0x56262c07d380 0x56262c07d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.325+0000 7f328e25e740 4 client.4404 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T03:47:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.325+0000 7f328e25e740 1 -- 172.21.15.102:0/1161679040 --> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56262bcbb3c0 con 0x56262c07d380 2024-03-23T03:47:01.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.326+0000 7f327be00640 1 --2- 172.21.15.102:0/1161679040 >> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] conn(0x56262c07d380 0x56262c07d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.326+0000 7f327be00640 1 --2- 172.21.15.102:0/1161679040 >> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] conn(0x56262c07d380 0x56262c07d760 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aff010d834062818 server_cookie=1fc50e4f8549755e in_seq=0 out_seq=0 2024-03-23T03:47:01.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.326+0000 7f3272000640 10 client.4404 ms_handle_connect on v2:172.21.15.173:6832/1462637509 2024-03-23T03:47:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.329+0000 7f3272000640 1 -- 172.21.15.102:0/1161679040 <== mds.0 v2:172.21.15.173:6832/1462637509 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56262bcbb3c0 con 0x56262c07d380 2024-03-23T03:47:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.329+0000 7f3272000640 10 client.4404 handle_command_reply: tid=0 2024-03-23T03:47:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.329+0000 7f328e25e740 1 client.4404 shutdown 2024-03-23T03:47:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.329+0000 7f328e25e740 1 -- 172.21.15.102:0/1161679040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08ad00 msgr2=0x56262c07e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.329+0000 7f328e25e740 1 --2- 172.21.15.102:0/1161679040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08ad00 0x56262c07e740 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f326c052040 tx=0x7f326c004730 comp rx=0 tx=0).stop 2024-03-23T03:47:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.329+0000 7f328e25e740 1 -- 172.21.15.102:0/1161679040 shutdown_connections 2024-03-23T03:47:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.329+0000 7f328e25e740 1 --2- 172.21.15.102:0/1161679040 >> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] conn(0x56262c07d380 0x56262c07d760 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.329+0000 7f328e25e740 1 --2- 172.21.15.102:0/1161679040 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c0abfa0 0x56262c0ac420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.329+0000 7f328e25e740 1 --2- 172.21.15.102:0/1161679040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08b0e0 0x56262c0a26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.329+0000 7f328e25e740 1 --2- 172.21.15.102:0/1161679040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08ad00 0x56262c07e740 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.329+0000 7f3272000640 0 client.4404 ms_handle_reset on v2:172.21.15.173:6832/1462637509 2024-03-23T03:47:01.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.329+0000 7f328e25e740 1 -- 172.21.15.102:0/1161679040 >> 172.21.15.102:0/1161679040 conn(0x56262c05e1e0 msgr2=0x56262c0a9ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:01.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.329+0000 7f328e25e740 1 -- 172.21.15.102:0/1161679040 shutdown_connections 2024-03-23T03:47:01.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.329+0000 7f328e25e740 1 -- 172.21.15.102:0/1161679040 wait complete. 2024-03-23T03:47:01.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.330+0000 7f328e25e740 20 client.4404 trim_cache size 0 max 16384 2024-03-23T03:47:01.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.345+0000 7f328e25e740 1 Processor -- start 2024-03-23T03:47:01.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.346+0000 7f328e25e740 1 -- start start 2024-03-23T03:47:01.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.346+0000 7f328e25e740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c08ad00 0x56262c0a6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.346+0000 7f328e25e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08b0e0 0x56262c05cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.346+0000 7f328e25e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c0abfa0 0x56262c05d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.346+0000 7f328e25e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262bd2ef20 con 0x56262c0abfa0 2024-03-23T03:47:01.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.346+0000 7f328e25e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262c0a5080 con 0x56262c08b0e0 2024-03-23T03:47:01.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.346+0000 7f328e25e740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x56262c0ac430 con 0x56262c08ad00 2024-03-23T03:47:01.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.347+0000 7f3289600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c0abfa0 0x56262c05d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.347+0000 7f3289600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c0abfa0 0x56262c05d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50884/0 (socket says 172.21.15.102:50884) 2024-03-23T03:47:01.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.347+0000 7f3289600640 1 -- 172.21.15.102:0/1659543076 learned_addr learned my addr 172.21.15.102:0/1659543076 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:01.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.347+0000 7f3289600640 1 -- 172.21.15.102:0/1659543076 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c08ad00 msgr2=0x56262c0a6130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:01.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.347+0000 7f3289600640 1 --2- 172.21.15.102:0/1659543076 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c08ad00 0x56262c0a6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.347+0000 7f3289600640 1 -- 172.21.15.102:0/1659543076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08b0e0 msgr2=0x56262c05cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:01.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.347+0000 7f3289600640 1 --2- 172.21.15.102:0/1659543076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08b0e0 0x56262c05cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.347+0000 7f3289600640 1 -- 172.21.15.102:0/1659543076 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56262bffb7c0 con 0x56262c0abfa0 2024-03-23T03:47:01.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.347+0000 7f3289600640 1 --2- 172.21.15.102:0/1659543076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c0abfa0 0x56262c05d1e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f327c060a50 tx=0x7f327c0112e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5200a3dc0b8cdd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:01.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.347+0000 7f327a000640 1 -- 172.21.15.102:0/1659543076 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c058b90 con 0x56262c0abfa0 2024-03-23T03:47:01.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.347+0000 7f327a000640 1 -- 172.21.15.102:0/1659543076 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327c05c9c0 con 0x56262c0abfa0 2024-03-23T03:47:01.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.347+0000 7f327a000640 1 -- 172.21.15.102:0/1659543076 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327c060f80 con 0x56262c0abfa0 2024-03-23T03:47:01.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.348+0000 7f328e25e740 1 -- 172.21.15.102:0/1659543076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c0abfa0 msgr2=0x56262c05d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:01.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.348+0000 7f328e25e740 1 --2- 172.21.15.102:0/1659543076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c0abfa0 0x56262c05d1e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f327c060a50 tx=0x7f327c0112e0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.348+0000 7f328e25e740 1 -- 172.21.15.102:0/1659543076 shutdown_connections 2024-03-23T03:47:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.348+0000 7f328e25e740 1 --2- 172.21.15.102:0/1659543076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c0abfa0 0x56262c05d1e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.348+0000 7f328e25e740 1 --2- 172.21.15.102:0/1659543076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08b0e0 0x56262c05cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.348+0000 7f328e25e740 1 --2- 172.21.15.102:0/1659543076 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c08ad00 0x56262c0a6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.348+0000 7f328e25e740 1 -- 172.21.15.102:0/1659543076 >> 172.21.15.102:0/1659543076 conn(0x56262c05e1e0 msgr2=0x56262c0595c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.348+0000 7f328e25e740 1 -- 172.21.15.102:0/1659543076 shutdown_connections 2024-03-23T03:47:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.348+0000 7f328e25e740 1 -- 172.21.15.102:0/1659543076 wait complete. 2024-03-23T03:47:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.349+0000 7f328e25e740 1 -- start start 2024-03-23T03:47:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.349+0000 7f328e25e740 1 Processor -- start 2024-03-23T03:47:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.350+0000 7f328e25e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08ad00 0x56262c0a2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.350+0000 7f328e25e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08b0e0 0x56262c0a2980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.350+0000 7f328e25e740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c0abfa0 0x56262c0faab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.350+0000 7f328e25e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262c0fbbc0 con 0x56262c08b0e0 2024-03-23T03:47:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.350+0000 7f328e25e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56262c100330 con 0x56262c08ad00 2024-03-23T03:47:01.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.350+0000 7f328e25e740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x56262c0755e0 con 0x56262c0abfa0 2024-03-23T03:47:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.350+0000 7f328e25e740 10 client.0 _ll_register_callbacks cb 0x7f326c052070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T03:47:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.350+0000 7f327be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08b0e0 0x56262c0a2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.350+0000 7f327be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08b0e0 0x56262c0a2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50892/0 (socket says 172.21.15.102:50892) 2024-03-23T03:47:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.350+0000 7f327be00640 1 -- 172.21.15.102:0/126791749 learned_addr learned my addr 172.21.15.102:0/126791749 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.350+0000 7f3289600640 1 --2- 172.21.15.102:0/126791749 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c0abfa0 0x56262c0faab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.350+0000 7f3288c00640 1 --2- 172.21.15.102:0/126791749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08ad00 0x56262c0a2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.351+0000 7f327be00640 1 -- 172.21.15.102:0/126791749 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c0abfa0 msgr2=0x56262c0faab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.351+0000 7f327be00640 1 --2- 172.21.15.102:0/126791749 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c0abfa0 0x56262c0faab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.351+0000 7f327be00640 1 -- 172.21.15.102:0/126791749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08ad00 msgr2=0x56262c0a2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.351+0000 7f327be00640 1 --2- 172.21.15.102:0/126791749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08ad00 0x56262c0a2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.351+0000 7f327be00640 1 -- 172.21.15.102:0/126791749 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32841e6cb0 con 0x56262c08b0e0 2024-03-23T03:47:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.351+0000 7f3289600640 1 --2- 172.21.15.102:0/126791749 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c0abfa0 0x56262c0faab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.351+0000 7f327be00640 1 --2- 172.21.15.102:0/126791749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08b0e0 0x56262c0a2980 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3274004730 tx=0x7f327400bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50492d1aa766bed server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:01.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.351+0000 7f3272000640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-23T03:47:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.351+0000 7f3272000640 1 -- 172.21.15.102:0/126791749 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3274016630 con 0x56262c08b0e0 2024-03-23T03:47:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.351+0000 7f3272000640 1 -- 172.21.15.102:0/126791749 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3274016790 con 0x56262c08b0e0 2024-03-23T03:47:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.351+0000 7f328e25e740 1 -- 172.21.15.102:0/126791749 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f326c03ae70 con 0x56262c08b0e0 2024-03-23T03:47:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.351+0000 7f3272000640 1 -- 172.21.15.102:0/126791749 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3274019440 con 0x56262c08b0e0 2024-03-23T03:47:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.351+0000 7f328e25e740 10 client.4410 fetch_fsmap learned FSMap version 6 2024-03-23T03:47:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.351+0000 7f328e25e740 1 -- 172.21.15.102:0/126791749 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) v3 -- 0x56262bf682b0 con 0x56262c08b0e0 2024-03-23T03:47:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.351+0000 7f3272000640 1 -- 172.21.15.102:0/126791749 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326c03ae70 con 0x56262c08b0e0 2024-03-23T03:47:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.352+0000 7f3272000640 1 -- 172.21.15.102:0/126791749 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 6) v1 ==== 3122+0+0 (secure 0 0 0) 0x7f3274016950 con 0x56262c08b0e0 2024-03-23T03:47:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.352+0000 7f328e25e740 10 client.4410 fetch_fsmap finished waiting for FSMap version 6 2024-03-23T03:47:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.352+0000 7f328e25e740 10 client.4410 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-23T03:47:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.352+0000 7f328e25e740 20 client.4410 populate_metadata read hostname 'smithi102' 2024-03-23T03:47:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.352+0000 7f328e25e740 1 --2- 172.21.15.102:0/126791749 >> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] conn(0x56262c0faff0 0x56262c0fc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.352+0000 7f328e25e740 4 client.4410 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-23T03:47:01.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.352+0000 7f328e25e740 1 -- 172.21.15.102:0/126791749 --> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56262bccc6c0 con 0x56262c0faff0 2024-03-23T03:47:01.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.352+0000 7f3288c00640 1 --2- 172.21.15.102:0/126791749 >> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] conn(0x56262c0faff0 0x56262c0fc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.353+0000 7f3288c00640 1 --2- 172.21.15.102:0/126791749 >> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] conn(0x56262c0faff0 0x56262c0fc0f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33114ee61e61382a server_cookie=573419975e294c57 in_seq=0 out_seq=0 2024-03-23T03:47:01.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.353+0000 7f3272000640 10 client.4410 ms_handle_connect on v2:172.21.15.173:6832/1462637509 2024-03-23T03:47:01.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.353+0000 7f3272000640 1 -- 172.21.15.102:0/126791749 <== mds.0 v2:172.21.15.173:6832/1462637509 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x56262bccc6c0 con 0x56262c0faff0 2024-03-23T03:47:01.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.353+0000 7f3272000640 10 client.4410 handle_command_reply: tid=0 2024-03-23T03:47:01.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.353+0000 7f328e25e740 1 client.4410 shutdown 2024-03-23T03:47:01.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.354+0000 7f328e25e740 1 -- 172.21.15.102:0/126791749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08b0e0 msgr2=0x56262c0a2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:01.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.354+0000 7f328e25e740 1 --2- 172.21.15.102:0/126791749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08b0e0 0x56262c0a2980 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3274004730 tx=0x7f327400bf60 comp rx=0 tx=0).stop 2024-03-23T03:47:01.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.354+0000 7f328e25e740 1 -- 172.21.15.102:0/126791749 shutdown_connections 2024-03-23T03:47:01.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.354+0000 7f328e25e740 1 --2- 172.21.15.102:0/126791749 >> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] conn(0x56262c0faff0 0x56262c0fc0f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.354+0000 7f328e25e740 1 --2- 172.21.15.102:0/126791749 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56262c0abfa0 0x56262c0faab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.354+0000 7f328e25e740 1 --2- 172.21.15.102:0/126791749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56262c08b0e0 0x56262c0a2980 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.354+0000 7f3272000640 0 client.4410 ms_handle_reset on v2:172.21.15.173:6832/1462637509 2024-03-23T03:47:01.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.354+0000 7f328e25e740 1 --2- 172.21.15.102:0/126791749 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56262c08ad00 0x56262c0a2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.354+0000 7f328e25e740 1 -- 172.21.15.102:0/126791749 >> 172.21.15.102:0/126791749 conn(0x56262c05e1e0 msgr2=0x56262c0595c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:01.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.354+0000 7f328e25e740 1 -- 172.21.15.102:0/126791749 shutdown_connections 2024-03-23T03:47:01.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.354+0000 7f328e25e740 1 -- 172.21.15.102:0/126791749 wait complete. 2024-03-23T03:47:01.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.354+0000 7f328e25e740 20 client.4410 trim_cache size 0 max 16384 2024-03-23T03:47:01.359 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:47:01.359 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "776b2820-05f2-4e50-8d40-18dd76c5f6a0", 2024-03-23T03:47:01.359 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 0, 2024-03-23T03:47:01.359 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 4270, 2024-03-23T03:47:01.359 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-03-23T03:47:01.359 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-03-23T03:47:01.359 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-03-23T03:47:01.359 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 7.4352347249999999, 2024-03-23T03:47:01.359 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 6, 2024-03-23T03:47:01.359 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 26, 2024-03-23T03:47:01.359 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 25, 2024-03-23T03:47:01.359 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 14.153969824000001 2024-03-23T03:47:01.359 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:47:01.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.354+0000 7f328b600640 1 -- 172.21.15.102:0/3552538134 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f326006ac00 msgr2=0x7f326006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:01.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.354+0000 7f328b600640 1 --2- 172.21.15.102:0/3552538134 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f326006ac00 0x7f326006d0c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f326c036000 tx=0x7f326c03c040 comp rx=0 tx=0).stop 2024-03-23T03:47:01.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.354+0000 7f328b600640 1 -- 172.21.15.102:0/3552538134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3284160dd0 msgr2=0x7f32840b08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:01.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.354+0000 7f328b600640 1 --2- 172.21.15.102:0/3552538134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3284160dd0 0x7f32840b08b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f327c007e60 tx=0x7f327c00e4b0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.355+0000 7f328b600640 1 -- 172.21.15.102:0/3552538134 shutdown_connections 2024-03-23T03:47:01.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.355+0000 7f328b600640 1 --2- 172.21.15.102:0/3552538134 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f326006ac00 0x7f326006d0c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.355+0000 7f328b600640 1 --2- 172.21.15.102:0/3552538134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3284160dd0 0x7f32840b08b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.355+0000 7f328b600640 1 --2- 172.21.15.102:0/3552538134 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f328413fa80 0x7f32840b0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.355+0000 7f328b600640 1 --2- 172.21.15.102:0/3552538134 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32840cf040 0x7f328413e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.355+0000 7f328b600640 1 -- 172.21.15.102:0/3552538134 >> 172.21.15.102:0/3552538134 conn(0x7f328406a0b0 msgr2=0x7f32840ac5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:01.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.355+0000 7f328b600640 1 -- 172.21.15.102:0/3552538134 shutdown_connections 2024-03-23T03:47:01.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.355+0000 7f328b600640 1 -- 172.21.15.102:0/3552538134 wait complete. 2024-03-23T03:47:01.372 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-23T03:47:01.373 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-23T03:47:01.373 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:47:01.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.571+0000 7f3ba8000640 1 Processor -- start 2024-03-23T03:47:01.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.571+0000 7f3ba8000640 1 -- start start 2024-03-23T03:47:01.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.573+0000 7f3ba8000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f3ba00adbc0 0x7f3ba00b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.573+0000 7f3ba8000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3ba00cbab0 0x7f3ba00b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.573+0000 7f3ba8000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba00b16a0 con 0x7f3ba006d390 2024-03-23T03:47:01.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.573+0000 7f3ba8000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ba00b4700 con 0x7f3ba00adbc0 2024-03-23T03:47:01.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.573+0000 7f3ba8000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba00cbfb0 con 0x7f3ba00cbab0 2024-03-23T03:47:01.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.574+0000 7f3ba5600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f3ba00adbc0 0x7f3ba00b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.574+0000 7f3ba4c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3ba00cbab0 0x7f3ba00b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.574+0000 7f3ba4c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3ba00cbab0 0x7f3ba00b20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:58632/0 (socket says 172.21.15.102:58632) 2024-03-23T03:47:01.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.574+0000 7f3ba4c00640 1 -- 172.21.15.102:0/2626557945 learned_addr learned my addr 172.21.15.102:0/2626557945 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:01.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.574+0000 7f3ba6000640 1 --1- 172.21.15.102:0/2626557945 >> v1:172.21.15.173:6789/0 conn(0x7f3ba006d390 0x7f3ba00b2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:55752/0 (socket says 172.21.15.102:55752) 2024-03-23T03:47:01.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.575+0000 7f3ba5600640 1 -- 172.21.15.102:0/2626557945 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3ba00cbab0 msgr2=0x7f3ba00b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:01.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.575+0000 7f3ba5600640 1 --2- 172.21.15.102:0/2626557945 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3ba00cbab0 0x7f3ba00b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.575+0000 7f3ba5600640 1 -- 172.21.15.102:0/2626557945 >> v1:172.21.15.173:6789/0 conn(0x7f3ba006d390 legacy=0x7f3ba00b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:01.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.575+0000 7f3ba5600640 1 -- 172.21.15.102:0/2626557945 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba00cc290 con 0x7f3ba00adbc0 2024-03-23T03:47:01.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.575+0000 7f3ba4c00640 1 --2- 172.21.15.102:0/2626557945 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3ba00cbab0 0x7f3ba00b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:47:01.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.576+0000 7f3ba5600640 1 --2- 172.21.15.102:0/2626557945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ba00adbc0 0x7f3ba00b41c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3b940097a0 tx=0x7f3b9402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cb17d2d1e5ee3ed server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:01.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.576+0000 7f3b9fe00640 1 -- 172.21.15.102:0/2626557945 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b94031780 con 0x7f3ba00adbc0 2024-03-23T03:47:01.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.576+0000 7f3b9fe00640 1 -- 172.21.15.102:0/2626557945 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b940318e0 con 0x7f3ba00adbc0 2024-03-23T03:47:01.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.577+0000 7f3b9fe00640 1 -- 172.21.15.102:0/2626557945 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b94031c70 con 0x7f3ba00adbc0 2024-03-23T03:47:01.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.577+0000 7f3ba8000640 1 -- 172.21.15.102:0/2626557945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ba00adbc0 msgr2=0x7f3ba00b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:01.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.577+0000 7f3ba8000640 1 --2- 172.21.15.102:0/2626557945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ba00adbc0 0x7f3ba00b41c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3b940097a0 tx=0x7f3b9402b820 comp rx=0 tx=0).stop 2024-03-23T03:47:01.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.577+0000 7f3ba8000640 1 -- 172.21.15.102:0/2626557945 shutdown_connections 2024-03-23T03:47:01.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.577+0000 7f3ba8000640 1 --2- 172.21.15.102:0/2626557945 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3ba00cbab0 0x7f3ba00b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.577+0000 7f3ba8000640 1 --2- 172.21.15.102:0/2626557945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ba00adbc0 0x7f3ba00b41c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.577+0000 7f3ba8000640 1 -- 172.21.15.102:0/2626557945 >> 172.21.15.102:0/2626557945 conn(0x7f3ba006a3d0 msgr2=0x7f3ba00b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:01.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.577+0000 7f3ba8000640 1 -- 172.21.15.102:0/2626557945 shutdown_connections 2024-03-23T03:47:01.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.577+0000 7f3ba8000640 1 -- 172.21.15.102:0/2626557945 wait complete. 2024-03-23T03:47:01.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.579+0000 7f3ba8000640 1 Processor -- start 2024-03-23T03:47:01.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.580+0000 7f3ba8000640 1 -- start start 2024-03-23T03:47:01.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.581+0000 7f3ba8000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ba006d390 0x7f3ba0159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.581+0000 7f3ba8000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3ba00adbc0 0x7f3ba0153250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.581+0000 7f3ba8000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ba00cbab0 0x7f3ba015a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.581+0000 7f3ba8000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba015ce80 con 0x7f3ba00cbab0 2024-03-23T03:47:01.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.581+0000 7f3ba8000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba013b3c0 con 0x7f3ba006d390 2024-03-23T03:47:01.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.581+0000 7f3ba8000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba013c1e0 con 0x7f3ba00adbc0 2024-03-23T03:47:01.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.581+0000 7f3ba6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ba00cbab0 0x7f3ba015a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.581+0000 7f3ba6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ba00cbab0 0x7f3ba015a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50900/0 (socket says 172.21.15.102:50900) 2024-03-23T03:47:01.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.581+0000 7f3ba6000640 1 -- 172.21.15.102:0/577943150 learned_addr learned my addr 172.21.15.102:0/577943150 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:01.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.581+0000 7f3ba5600640 1 --2- 172.21.15.102:0/577943150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ba006d390 0x7f3ba0159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.581+0000 7f3ba4c00640 1 --2- 172.21.15.102:0/577943150 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3ba00adbc0 0x7f3ba0153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.582+0000 7f3ba6000640 1 -- 172.21.15.102:0/577943150 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3ba00adbc0 msgr2=0x7f3ba0153250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:01.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.582+0000 7f3ba6000640 1 --2- 172.21.15.102:0/577943150 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3ba00adbc0 0x7f3ba0153250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.582+0000 7f3ba6000640 1 -- 172.21.15.102:0/577943150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ba006d390 msgr2=0x7f3ba0159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:01.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.582+0000 7f3ba6000640 1 --2- 172.21.15.102:0/577943150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ba006d390 0x7f3ba0159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.582+0000 7f3ba6000640 1 -- 172.21.15.102:0/577943150 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b8c0094d0 con 0x7f3ba00cbab0 2024-03-23T03:47:01.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.582+0000 7f3ba6000640 1 --2- 172.21.15.102:0/577943150 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ba00cbab0 0x7f3ba015a0b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3b9800c730 tx=0x7f3b9800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:01.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.582+0000 7f3b9d600640 1 -- 172.21.15.102:0/577943150 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9800ce40 con 0x7f3ba00cbab0 2024-03-23T03:47:01.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.582+0000 7f3b9d600640 1 -- 172.21.15.102:0/577943150 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9800a640 con 0x7f3ba00cbab0 2024-03-23T03:47:01.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.582+0000 7f3b9d600640 1 -- 172.21.15.102:0/577943150 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b9800a9f0 con 0x7f3ba00cbab0 2024-03-23T03:47:01.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.582+0000 7f3ba8000640 1 -- 172.21.15.102:0/577943150 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b94009400 con 0x7f3ba00cbab0 2024-03-23T03:47:01.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.582+0000 7f3ba8000640 1 -- 172.21.15.102:0/577943150 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba013bb90 con 0x7f3ba00cbab0 2024-03-23T03:47:01.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.583+0000 7f3ba8000640 1 -- 172.21.15.102:0/577943150 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ba006ac20 con 0x7f3ba00cbab0 2024-03-23T03:47:01.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.584+0000 7f3b9d600640 1 -- 172.21.15.102:0/577943150 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3b9800ab50 con 0x7f3ba00cbab0 2024-03-23T03:47:01.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.585+0000 7f3b9d600640 1 --2- 172.21.15.102:0/577943150 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b8006ac30 0x7f3b8006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:01.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.585+0000 7f3b9d600640 1 -- 172.21.15.102:0/577943150 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3b98088db0 con 0x7f3ba00cbab0 2024-03-23T03:47:01.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.585+0000 7f3ba5600640 1 --2- 172.21.15.102:0/577943150 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b8006ac30 0x7f3b8006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:01.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.586+0000 7f3ba5600640 1 --2- 172.21.15.102:0/577943150 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b8006ac30 0x7f3b8006d0f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3b94036000 tx=0x7f3b9403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:01.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.590+0000 7f3b9d600640 1 -- 172.21.15.102:0/577943150 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3b9808d070 con 0x7f3ba00cbab0 2024-03-23T03:47:01.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.825+0000 7f3ba8000640 1 -- 172.21.15.102:0/577943150 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3ba013c350 con 0x7f3ba00cbab0 2024-03-23T03:47:01.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.826+0000 7f3b9d600640 1 -- 172.21.15.102:0/577943150 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) v1 ==== 53+0+88 (secure 0 0 0) 0x7f3b98017020 con 0x7f3ba00cbab0 2024-03-23T03:47:01.827 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-23T03:47:01.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.829+0000 7f3ba8000640 1 -- 172.21.15.102:0/577943150 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b8006ac30 msgr2=0x7f3b8006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:01.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.829+0000 7f3ba8000640 1 --2- 172.21.15.102:0/577943150 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b8006ac30 0x7f3b8006d0f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3b94036000 tx=0x7f3b9403d040 comp rx=0 tx=0).stop 2024-03-23T03:47:01.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.829+0000 7f3ba8000640 1 -- 172.21.15.102:0/577943150 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ba00cbab0 msgr2=0x7f3ba015a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:01.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.829+0000 7f3ba8000640 1 --2- 172.21.15.102:0/577943150 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ba00cbab0 0x7f3ba015a0b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3b9800c730 tx=0x7f3b9800cbf0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.829+0000 7f3ba8000640 1 -- 172.21.15.102:0/577943150 shutdown_connections 2024-03-23T03:47:01.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.829+0000 7f3ba8000640 1 --2- 172.21.15.102:0/577943150 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b8006ac30 0x7f3b8006d0f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.829+0000 7f3ba8000640 1 --2- 172.21.15.102:0/577943150 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ba00cbab0 0x7f3ba015a0b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.829+0000 7f3ba8000640 1 --2- 172.21.15.102:0/577943150 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3ba00adbc0 0x7f3ba0153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.829+0000 7f3ba8000640 1 --2- 172.21.15.102:0/577943150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ba006d390 0x7f3ba0159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:01.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.829+0000 7f3ba8000640 1 -- 172.21.15.102:0/577943150 >> 172.21.15.102:0/577943150 conn(0x7f3ba006a3d0 msgr2=0x7f3ba0142bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:01.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.829+0000 7f3ba8000640 1 -- 172.21.15.102:0/577943150 shutdown_connections 2024-03-23T03:47:01.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:01.829+0000 7f3ba8000640 1 -- 172.21.15.102:0/577943150 wait complete. 2024-03-23T03:47:01.850 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-23T03:47:01.850 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:47:01.850 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2024-03-23T03:47:01.850 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-23T03:47:01.850 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-23T03:47:01.850 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-23T03:47:01.851 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T03:47:01.851 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:47:01.851 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-23T03:47:01.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:01.851 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2024-03-23T03:47:01.902 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:47:01.903 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:47:01.903 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:47:01.903 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:01.903 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-03-23T03:47:01.903 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:01.903 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:47:01.903 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:01.903 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f0 2024-03-23T03:47:01.903 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:47:01.903 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:01.903 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f1 2024-03-23T03:47:01.903 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:47:01.903 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:01.903 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:47:01.903 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:01.904 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:47:01.904 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 42649sec preferred_lft 42649sec 2024-03-23T03:47:01.904 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-03-23T03:47:01.904 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:01.905 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-23T03:47:01.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:01.905 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:47:01.905 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:47:01.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add name ceph-brx type bridge 2024-03-23T03:47:01.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr flush dev ceph-brx 2024-03-23T03:47:01.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set ceph-brx up 2024-03-23T03:47:01.906 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-23T03:47:01.906 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:47:02.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:02.326 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:47:02.404 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-23T03:47:02.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:02.408 DEBUG:teuthology.orchestra.run.smithi102:> route 2024-03-23T03:47:02.467 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2024-03-23T03:47:02.467 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:47:02.467 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:47:02.467 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:47:02.467 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:47:02.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:02.467 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:47:02.468 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:47:02.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:47:02.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:47:02.468 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:47:02.468 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:47:03.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:03.145 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-23T03:47:03.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:03.199 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list-id 2024-03-23T03:47:03.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:03.254 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:47:03.254 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:47:03.254 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-23T03:47:03.254 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-23T03:47:03.254 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:47:03.387 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-23T03:47:03.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:03.388 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:47:03.388 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:47:03.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-23T03:47:03.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-23T03:47:03.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-23T03:47:03.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-23T03:47:03.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-23T03:47:03.388 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:47:03.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:03.669 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:47:03.669 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:47:03.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.0 up 2024-03-23T03:47:03.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set dev brx.0 master ceph-brx 2024-03-23T03:47:03.670 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:47:03.827 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-23T03:47:03.827 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:47:03.827 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-23T03:47:03.870 INFO:teuthology.orchestra.run.smithi102.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-23T03:47:03.870 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:47:03.870 DEBUG:teuthology.orchestra.run.smithi102:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-23T03:47:03.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo modprobe fuse 2024-03-23T03:47:04.059 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T03:47:04.059 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T03:47:04.141 INFO:teuthology.orchestra.run.smithi102.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T03:47:04.144 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T03:47:04.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:04.145 DEBUG:teuthology.orchestra.run.smithi102:> ls /sys/fs/fuse/connections 2024-03-23T03:47:04.189 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T03:47:04.189 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-23T03:47:04.237 DEBUG:teuthology.orchestra.run.smithi102:> sudo modprobe fuse 2024-03-23T03:47:04.319 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T03:47:04.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T03:47:04.395 INFO:teuthology.orchestra.run.smithi102.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T03:47:04.398 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T03:47:04.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:04.398 DEBUG:teuthology.orchestra.run.smithi102:> ls /sys/fs/fuse/connections 2024-03-23T03:47:04.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi102.stderr:2024-03-23T03:47:04.429+0000 7f55e7038480 -1 init, newargv = 0x5623b8093030 newargc=15 2024-03-23T03:47:04.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi102.stderr:ceph-fuse[43309]: starting ceph client 2024-03-23T03:47:04.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi102.stderr:ceph-fuse[43309]: starting fuse 2024-03-23T03:47:05.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo modprobe fuse 2024-03-23T03:47:05.641 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-03-23T03:47:05.641 DEBUG:teuthology.orchestra.run.smithi102:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-03-23T03:47:05.730 INFO:teuthology.orchestra.run.smithi102.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-03-23T03:47:05.733 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-03-23T03:47:05.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:05.734 DEBUG:teuthology.orchestra.run.smithi102:> ls /sys/fs/fuse/connections 2024-03-23T03:47:05.786 INFO:teuthology.orchestra.run.smithi102.stdout:40 2024-03-23T03:47:05.786 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2024-03-23T03:47:05.786 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T03:47:05.786 DEBUG:teuthology.orchestra.run.smithi102:> import glob 2024-03-23T03:47:05.786 DEBUG:teuthology.orchestra.run.smithi102:> import re 2024-03-23T03:47:05.786 DEBUG:teuthology.orchestra.run.smithi102:> import os 2024-03-23T03:47:05.786 DEBUG:teuthology.orchestra.run.smithi102:> import subprocess 2024-03-23T03:47:05.786 DEBUG:teuthology.orchestra.run.smithi102:> 2024-03-23T03:47:05.787 DEBUG:teuthology.orchestra.run.smithi102:> def _find_admin_socket(client_name): 2024-03-23T03:47:05.787 DEBUG:teuthology.orchestra.run.smithi102:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-23T03:47:05.787 DEBUG:teuthology.orchestra.run.smithi102:> files = glob.glob(asok_path) 2024-03-23T03:47:05.787 DEBUG:teuthology.orchestra.run.smithi102:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-23T03:47:05.787 DEBUG:teuthology.orchestra.run.smithi102:> 2024-03-23T03:47:05.787 DEBUG:teuthology.orchestra.run.smithi102:> # Given a non-glob path, it better be there 2024-03-23T03:47:05.787 DEBUG:teuthology.orchestra.run.smithi102:> if "*" not in asok_path: 2024-03-23T03:47:05.787 DEBUG:teuthology.orchestra.run.smithi102:> assert(len(files) == 1) 2024-03-23T03:47:05.787 DEBUG:teuthology.orchestra.run.smithi102:> return files[0] 2024-03-23T03:47:05.787 DEBUG:teuthology.orchestra.run.smithi102:> 2024-03-23T03:47:05.787 DEBUG:teuthology.orchestra.run.smithi102:> for f in files: 2024-03-23T03:47:05.787 DEBUG:teuthology.orchestra.run.smithi102:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T03:47:05.787 DEBUG:teuthology.orchestra.run.smithi102:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T03:47:05.787 DEBUG:teuthology.orchestra.run.smithi102:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T03:47:05.787 DEBUG:teuthology.orchestra.run.smithi102:> contents = proc_f.read() 2024-03-23T03:47:05.788 DEBUG:teuthology.orchestra.run.smithi102:> if mountpoint in contents: 2024-03-23T03:47:05.788 DEBUG:teuthology.orchestra.run.smithi102:> return f 2024-03-23T03:47:05.788 DEBUG:teuthology.orchestra.run.smithi102:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T03:47:05.788 DEBUG:teuthology.orchestra.run.smithi102:> 2024-03-23T03:47:05.788 DEBUG:teuthology.orchestra.run.smithi102:> print(_find_admin_socket("client.admin")) 2024-03-23T03:47:05.788 DEBUG:teuthology.orchestra.run.smithi102:> ' 2024-03-23T03:47:05.957 INFO:teuthology.orchestra.run.smithi102.stdout:/var/run/ceph/ceph-client.admin.43309.asok 2024-03-23T03:47:06.139 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.43309.asok 2024-03-23T03:47:06.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:06.140 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.43309.asok status 2024-03-23T03:47:06.376 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:47:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: "metadata": { 2024-03-23T03:47:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_sha1": "82316b04821a6c2c1d458a881920454bbadb8837", 2024-03-23T03:47:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_version": "ceph version 18.2.2-954-g82316b04 (82316b04821a6c2c1d458a881920454bbadb8837) reef (stable)", 2024-03-23T03:47:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: "entity_id": "admin", 2024-03-23T03:47:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: "hostname": "smithi102", 2024-03-23T03:47:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-23T03:47:06.376 INFO:teuthology.orchestra.run.smithi102.stdout: "pid": "43309", 2024-03-23T03:47:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "root": "/" 2024-03-23T03:47:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:47:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "dentry_count": 0, 2024-03-23T03:47:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "dentry_pinned_count": 0, 2024-03-23T03:47:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 4419, 2024-03-23T03:47:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "inst": { 2024-03-23T03:47:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "name": { 2024-03-23T03:47:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "client", 2024-03-23T03:47:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "num": 4419 2024-03-23T03:47:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:47:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": { 2024-03-23T03:47:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2024-03-23T03:47:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "192.168.0.1:0", 2024-03-23T03:47:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 830498779 2024-03-23T03:47:06.377 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:47:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:47:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": { 2024-03-23T03:47:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "type": "v1", 2024-03-23T03:47:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "addr": "192.168.0.1:0", 2024-03-23T03:47:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "nonce": 830498779 2024-03-23T03:47:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:47:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "inst_str": "client.4419 192.168.0.1:0/830498779", 2024-03-23T03:47:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "addr_str": "192.168.0.1:0/830498779", 2024-03-23T03:47:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "inode_count": 1, 2024-03-23T03:47:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_epoch": 6, 2024-03-23T03:47:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_epoch": 26, 2024-03-23T03:47:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_epoch_barrier": 0, 2024-03-23T03:47:06.378 INFO:teuthology.orchestra.run.smithi102.stdout: "blocklisted": false, 2024-03-23T03:47:06.379 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs" 2024-03-23T03:47:06.379 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:47:06.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:06.391 DEBUG:teuthology.orchestra.run.smithi102:> stat --file-system '--printf=%T 2024-03-23T03:47:06.391 DEBUG:teuthology.orchestra.run.smithi102:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-23T03:47:06.440 INFO:teuthology.orchestra.run.smithi102.stdout:fuseblk 2024-03-23T03:47:06.440 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-23T03:47:06.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:06.441 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-23T03:47:06.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:06.577 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-23T03:47:06.672 INFO:teuthology.orchestra.run.smithi102.stdout:# file: . 2024-03-23T03:47:06.672 INFO:teuthology.orchestra.run.smithi102.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-23T03:47:06.672 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:06.673 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:47:06.673 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:47:06.728 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:06.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.939+0000 7f0831200640 1 Processor -- start 2024-03-23T03:47:06.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.939+0000 7f0831200640 1 -- start start 2024-03-23T03:47:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.941+0000 7f0831200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f082c15cd70 0x7f082c0d0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.941+0000 7f0831200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f082c0d1330 0x7f082c0b4340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.941+0000 7f0831200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f082c0d17d0 con 0x7f082c0b4880 2024-03-23T03:47:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.941+0000 7f0831200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f082c15d150 con 0x7f082c15cd70 2024-03-23T03:47:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.941+0000 7f0831200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f082c0b4d10 con 0x7f082c0d1330 2024-03-23T03:47:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.941+0000 7f0829e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f082c15cd70 0x7f082c0d0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.941+0000 7f0829e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f082c15cd70 0x7f082c0d0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53612/0 (socket says 172.21.15.102:53612) 2024-03-23T03:47:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.941+0000 7f082a800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f082c0b4880 0x7f082c06b120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46702/0 (socket says 172.21.15.102:46702) 2024-03-23T03:47:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.941+0000 7f0829e00640 1 -- 172.21.15.102:0/3280721011 learned_addr learned my addr 172.21.15.102:0/3280721011 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.941+0000 7f0823e00640 1 --2- 172.21.15.102:0/3280721011 >> v2:172.21.15.173:3300/0 conn(0x7f082c0d1330 0x7f082c0b4340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:06.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.942+0000 7f0829e00640 1 -- 172.21.15.102:0/3280721011 >> v2:172.21.15.173:3300/0 conn(0x7f082c0d1330 msgr2=0x7f082c0b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:06.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.942+0000 7f0829e00640 1 --2- 172.21.15.102:0/3280721011 >> v2:172.21.15.173:3300/0 conn(0x7f082c0d1330 0x7f082c0b4340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:06.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.942+0000 7f0829e00640 1 -- 172.21.15.102:0/3280721011 >> v1:172.21.15.102:6789/0 conn(0x7f082c0b4880 legacy=0x7f082c06b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:06.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.942+0000 7f0829e00640 1 -- 172.21.15.102:0/3280721011 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c0d1ab0 con 0x7f082c15cd70 2024-03-23T03:47:06.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.942+0000 7f0829400640 1 -- 172.21.15.102:0/3280721011 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418624116 0 0) 0x7f082c0d17d0 con 0x7f082c0b4880 2024-03-23T03:47:06.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.943+0000 7f0829e00640 1 --2- 172.21.15.102:0/3280721011 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f082c15cd70 0x7f082c0d0df0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f082c0c9940 tx=0x7f081c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d2da4f64a9c3256 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:06.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.943+0000 7f0829400640 1 -- 172.21.15.102:0/3280721011 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081c030860 con 0x7f082c15cd70 2024-03-23T03:47:06.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.943+0000 7f0829400640 1 -- 172.21.15.102:0/3280721011 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f081c0309c0 con 0x7f082c15cd70 2024-03-23T03:47:06.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.943+0000 7f0829400640 1 -- 172.21.15.102:0/3280721011 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081c030d50 con 0x7f082c15cd70 2024-03-23T03:47:06.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.944+0000 7f0831200640 1 -- 172.21.15.102:0/3280721011 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f082c15cd70 msgr2=0x7f082c0d0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:06.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.944+0000 7f0831200640 1 --2- 172.21.15.102:0/3280721011 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f082c15cd70 0x7f082c0d0df0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f082c0c9940 tx=0x7f081c02b9a0 comp rx=0 tx=0).stop 2024-03-23T03:47:06.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.944+0000 7f0831200640 1 -- 172.21.15.102:0/3280721011 shutdown_connections 2024-03-23T03:47:06.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.944+0000 7f0831200640 1 --2- 172.21.15.102:0/3280721011 >> v2:172.21.15.173:3300/0 conn(0x7f082c0d1330 0x7f082c0b4340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:06.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.944+0000 7f0831200640 1 --2- 172.21.15.102:0/3280721011 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f082c15cd70 0x7f082c0d0df0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:06.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.944+0000 7f0831200640 1 -- 172.21.15.102:0/3280721011 >> 172.21.15.102:0/3280721011 conn(0x7f082c0cf340 msgr2=0x7f082c0cf750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:06.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.944+0000 7f0831200640 1 -- 172.21.15.102:0/3280721011 shutdown_connections 2024-03-23T03:47:06.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.944+0000 7f0831200640 1 -- 172.21.15.102:0/3280721011 wait complete. 2024-03-23T03:47:06.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.946+0000 7f0831200640 1 Processor -- start 2024-03-23T03:47:06.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.946+0000 7f0831200640 1 -- start start 2024-03-23T03:47:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.947+0000 7f0831200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f082c0b4880 0x7f082c0cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.947+0000 7f0831200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f082c0d1330 0x7f082c142950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.947+0000 7f0831200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f082c15cd70 0x7f082c155480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.947+0000 7f0831200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082c15e2f0 con 0x7f082c0d1330 2024-03-23T03:47:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.947+0000 7f0831200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082c142e90 con 0x7f082c15cd70 2024-03-23T03:47:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.947+0000 7f0831200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f082c155df0 con 0x7f082c0b4880 2024-03-23T03:47:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.947+0000 7f0823e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f082c0d1330 0x7f082c142950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.947+0000 7f0823e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f082c0d1330 0x7f082c142950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53620/0 (socket says 172.21.15.102:53620) 2024-03-23T03:47:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.947+0000 7f0823e00640 1 -- 172.21.15.102:0/2704611557 learned_addr learned my addr 172.21.15.102:0/2704611557 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.947+0000 7f0829e00640 1 --2- 172.21.15.102:0/2704611557 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f082c0b4880 0x7f082c0cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.947+0000 7f0823e00640 1 -- 172.21.15.102:0/2704611557 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f082c0b4880 msgr2=0x7f082c0cc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.947+0000 7f0823e00640 1 --2- 172.21.15.102:0/2704611557 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f082c0b4880 0x7f082c0cc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.947+0000 7f0823e00640 1 -- 172.21.15.102:0/2704611557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f082c15cd70 msgr2=0x7f082c155480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.947+0000 7f0823e00640 1 --2- 172.21.15.102:0/2704611557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f082c15cd70 0x7f082c155480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.947+0000 7f0823e00640 1 -- 172.21.15.102:0/2704611557 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081c0094d0 con 0x7f082c0d1330 2024-03-23T03:47:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.948+0000 7f0823e00640 1 --2- 172.21.15.102:0/2704611557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f082c0d1330 0x7f082c142950 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f081400d7d0 tx=0x7f081400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:06.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.948+0000 7f0822000640 1 -- 172.21.15.102:0/2704611557 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081400b6c0 con 0x7f082c0d1330 2024-03-23T03:47:06.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.948+0000 7f0822000640 1 -- 172.21.15.102:0/2704611557 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f081400b820 con 0x7f082c0d1330 2024-03-23T03:47:06.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.948+0000 7f0822000640 1 -- 172.21.15.102:0/2704611557 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081400bbb0 con 0x7f082c0d1330 2024-03-23T03:47:06.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.948+0000 7f0831200640 1 -- 172.21.15.102:0/2704611557 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f082c15e540 con 0x7f082c0d1330 2024-03-23T03:47:06.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.948+0000 7f0831200640 1 -- 172.21.15.102:0/2704611557 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f082c15eb00 con 0x7f082c0d1330 2024-03-23T03:47:06.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.949+0000 7f0831200640 1 -- 172.21.15.102:0/2704611557 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f082c15ecc0 con 0x7f082c0d1330 2024-03-23T03:47:06.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.950+0000 7f0822000640 1 -- 172.21.15.102:0/2704611557 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f081400bd10 con 0x7f082c0d1330 2024-03-23T03:47:06.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.951+0000 7f0822000640 1 --2- 172.21.15.102:0/2704611557 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f080806ac00 0x7f080806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:06.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.951+0000 7f0822000640 1 -- 172.21.15.102:0/2704611557 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0814089ba0 con 0x7f082c0d1330 2024-03-23T03:47:06.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.951+0000 7f0829e00640 1 --2- 172.21.15.102:0/2704611557 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f080806ac00 0x7f080806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:06.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.952+0000 7f0829e00640 1 --2- 172.21.15.102:0/2704611557 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f080806ac00 0x7f080806d0c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f081c036000 tx=0x7f081c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:06.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:06.956+0000 7f0822000640 1 -- 172.21.15.102:0/2704611557 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0814054a80 con 0x7f082c0d1330 2024-03-23T03:47:07.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.154+0000 7f0831200640 1 -- 172.21.15.102:0/2704611557 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f082c073390 con 0x7f082c0d1330 2024-03-23T03:47:07.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.157+0000 7f0822000640 1 -- 172.21.15.102:0/2704611557 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f08140580a0 con 0x7f082c0d1330 2024-03-23T03:47:07.158 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:07.158 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:58.410551+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:47:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.160+0000 7f0831200640 1 -- 172.21.15.102:0/2704611557 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f080806ac00 msgr2=0x7f080806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.160+0000 7f0831200640 1 --2- 172.21.15.102:0/2704611557 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f080806ac00 0x7f080806d0c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f081c036000 tx=0x7f081c03b040 comp rx=0 tx=0).stop 2024-03-23T03:47:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.160+0000 7f0831200640 1 -- 172.21.15.102:0/2704611557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f082c0d1330 msgr2=0x7f082c142950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.160+0000 7f0831200640 1 --2- 172.21.15.102:0/2704611557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f082c0d1330 0x7f082c142950 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f081400d7d0 tx=0x7f081400dc90 comp rx=0 tx=0).stop 2024-03-23T03:47:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.160+0000 7f0831200640 1 -- 172.21.15.102:0/2704611557 shutdown_connections 2024-03-23T03:47:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.160+0000 7f0831200640 1 --2- 172.21.15.102:0/2704611557 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f080806ac00 0x7f080806d0c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.160+0000 7f0831200640 1 --2- 172.21.15.102:0/2704611557 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f082c15cd70 0x7f082c155480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.160+0000 7f0831200640 1 --2- 172.21.15.102:0/2704611557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f082c0d1330 0x7f082c142950 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.160+0000 7f0831200640 1 --2- 172.21.15.102:0/2704611557 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f082c0b4880 0x7f082c0cc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.160+0000 7f0831200640 1 -- 172.21.15.102:0/2704611557 >> 172.21.15.102:0/2704611557 conn(0x7f082c0cf340 msgr2=0x7f082c0b3d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.160+0000 7f0831200640 1 -- 172.21.15.102:0/2704611557 shutdown_connections 2024-03-23T03:47:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.160+0000 7f0831200640 1 -- 172.21.15.102:0/2704611557 wait complete. 2024-03-23T03:47:07.180 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:46:52.158650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:46:52.914275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-23T03:46:55.983754+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-23T03:47:07.180 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:47:07.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.387+0000 7f73e0200640 1 Processor -- start 2024-03-23T03:47:07.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.387+0000 7f73e0200640 1 -- start start 2024-03-23T03:47:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.388+0000 7f73e0200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f73d8160d20 0x7f73d8164920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.388+0000 7f73e0200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f73d80ca7b0 0x7f73d80ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.388+0000 7f73e0200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d8165830 con 0x7f73d80cee70 2024-03-23T03:47:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.389+0000 7f73e0200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73d80cf300 con 0x7f73d80ca7b0 2024-03-23T03:47:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.389+0000 7f73e0200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d8164f90 con 0x7f73d8160d20 2024-03-23T03:47:07.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.389+0000 7f73dce00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f73d80ca7b0 0x7f73d80ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:07.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.389+0000 7f73dd800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f73d8160d20 0x7f73d8164920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:07.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.389+0000 7f73dce00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f73d80ca7b0 0x7f73d80ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:36698/0 (socket says 172.21.15.102:36698) 2024-03-23T03:47:07.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.389+0000 7f73dd800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f73d8160d20 0x7f73d8164920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:36922/0 (socket says 172.21.15.102:36922) 2024-03-23T03:47:07.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.389+0000 7f73dce00640 1 -- 172.21.15.102:0/2342840297 learned_addr learned my addr 172.21.15.102:0/2342840297 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:07.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.389+0000 7f73de200640 1 --1- 172.21.15.102:0/2342840297 >> v1:172.21.15.173:6789/0 conn(0x7f73d80cee70 0x7f73d8070150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:53204/0 (socket says 172.21.15.102:53204) 2024-03-23T03:47:07.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.390+0000 7f73dce00640 1 -- 172.21.15.102:0/2342840297 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f73d8160d20 msgr2=0x7f73d8164920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:07.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.390+0000 7f73dce00640 1 --2- 172.21.15.102:0/2342840297 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f73d8160d20 0x7f73d8164920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.390+0000 7f73dce00640 1 -- 172.21.15.102:0/2342840297 >> v1:172.21.15.173:6789/0 conn(0x7f73d80cee70 legacy=0x7f73d8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:07.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.390+0000 7f73dce00640 1 -- 172.21.15.102:0/2342840297 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d8165270 con 0x7f73d80ca7b0 2024-03-23T03:47:07.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.390+0000 7f73dd800640 1 --2- 172.21.15.102:0/2342840297 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f73d8160d20 0x7f73d8164920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:47:07.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.391+0000 7f73dce00640 1 --2- 172.21.15.102:0/2342840297 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73d80ca7b0 0x7f73d80ce930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f73c4009870 tx=0x7f73c402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9cfd86061ce07df5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:07.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.392+0000 7f73d7e00640 1 -- 172.21.15.102:0/2342840297 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c4031640 con 0x7f73d80ca7b0 2024-03-23T03:47:07.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.392+0000 7f73d7e00640 1 -- 172.21.15.102:0/2342840297 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c40317a0 con 0x7f73d80ca7b0 2024-03-23T03:47:07.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.392+0000 7f73d7e00640 1 -- 172.21.15.102:0/2342840297 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c4031b30 con 0x7f73d80ca7b0 2024-03-23T03:47:07.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.393+0000 7f73e0200640 1 -- 172.21.15.102:0/2342840297 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73d80ca7b0 msgr2=0x7f73d80ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:07.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.393+0000 7f73e0200640 1 --2- 172.21.15.102:0/2342840297 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73d80ca7b0 0x7f73d80ce930 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f73c4009870 tx=0x7f73c402b8f0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.393+0000 7f73e0200640 1 -- 172.21.15.102:0/2342840297 shutdown_connections 2024-03-23T03:47:07.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.393+0000 7f73e0200640 1 --2- 172.21.15.102:0/2342840297 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73d80ca7b0 0x7f73d80ce930 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.393+0000 7f73e0200640 1 --2- 172.21.15.102:0/2342840297 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f73d8160d20 0x7f73d8164920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.393+0000 7f73e0200640 1 -- 172.21.15.102:0/2342840297 >> 172.21.15.102:0/2342840297 conn(0x7f73d806a0b0 msgr2=0x7f73d806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:07.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.393+0000 7f73e0200640 1 -- 172.21.15.102:0/2342840297 shutdown_connections 2024-03-23T03:47:07.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.393+0000 7f73e0200640 1 -- 172.21.15.102:0/2342840297 wait complete. 2024-03-23T03:47:07.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.395+0000 7f73e0200640 1 Processor -- start 2024-03-23T03:47:07.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.395+0000 7f73e0200640 1 -- start start 2024-03-23T03:47:07.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.396+0000 7f73e0200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73d80ca7b0 0x7f73d814bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:07.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.396+0000 7f73e0200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73d80cee70 0x7f73d80cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:07.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.396+0000 7f73e0200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f73d8160d20 0x7f73d80b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:07.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.396+0000 7f73e0200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d814c010 con 0x7f73d80ca7b0 2024-03-23T03:47:07.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.396+0000 7f73e0200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d80cce40 con 0x7f73d80cee70 2024-03-23T03:47:07.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.396+0000 7f73e0200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d80cd120 con 0x7f73d8160d20 2024-03-23T03:47:07.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.396+0000 7f73dd800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73d80ca7b0 0x7f73d814bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:07.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.396+0000 7f73dd800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73d80ca7b0 0x7f73d814bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53622/0 (socket says 172.21.15.102:53622) 2024-03-23T03:47:07.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.396+0000 7f73dd800640 1 -- 172.21.15.102:0/2575641159 learned_addr learned my addr 172.21.15.102:0/2575641159 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:07.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.396+0000 7f73dce00640 1 --2- 172.21.15.102:0/2575641159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73d80cee70 0x7f73d80cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:07.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.396+0000 7f73dd800640 1 -- 172.21.15.102:0/2575641159 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f73d8160d20 msgr2=0x7f73d80b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:07.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.396+0000 7f73dd800640 1 --2- 172.21.15.102:0/2575641159 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f73d8160d20 0x7f73d80b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.396+0000 7f73dd800640 1 -- 172.21.15.102:0/2575641159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73d80cee70 msgr2=0x7f73d80cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:07.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.396+0000 7f73dd800640 1 --2- 172.21.15.102:0/2575641159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73d80cee70 0x7f73d80cc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.396+0000 7f73dd800640 1 -- 172.21.15.102:0/2575641159 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73cc009400 con 0x7f73d80ca7b0 2024-03-23T03:47:07.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.397+0000 7f73dd800640 1 --2- 172.21.15.102:0/2575641159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73d80ca7b0 0x7f73d814bad0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f73cc00ebe0 tx=0x7f73cc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:07.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.397+0000 7f73d5600640 1 -- 172.21.15.102:0/2575641159 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc009760 con 0x7f73d80ca7b0 2024-03-23T03:47:07.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.397+0000 7f73d5600640 1 -- 172.21.15.102:0/2575641159 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73cc00cc30 con 0x7f73d80ca7b0 2024-03-23T03:47:07.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.397+0000 7f73e0200640 1 -- 172.21.15.102:0/2575641159 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c40094d0 con 0x7f73d80ca7b0 2024-03-23T03:47:07.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.397+0000 7f73d5600640 1 -- 172.21.15.102:0/2575641159 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73cc00eee0 con 0x7f73d80ca7b0 2024-03-23T03:47:07.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.397+0000 7f73e0200640 1 -- 172.21.15.102:0/2575641159 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d80cd6e0 con 0x7f73d80ca7b0 2024-03-23T03:47:07.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.398+0000 7f73e0200640 1 -- 172.21.15.102:0/2575641159 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d8073390 con 0x7f73d80ca7b0 2024-03-23T03:47:07.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.399+0000 7f73d5600640 1 -- 172.21.15.102:0/2575641159 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f73cc013470 con 0x7f73d80ca7b0 2024-03-23T03:47:07.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.400+0000 7f73d5600640 1 --2- 172.21.15.102:0/2575641159 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f73b806ac00 0x7f73b806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:07.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.400+0000 7f73d5600640 1 -- 172.21.15.102:0/2575641159 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f73cc0899f0 con 0x7f73d80ca7b0 2024-03-23T03:47:07.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.400+0000 7f73dce00640 1 --2- 172.21.15.102:0/2575641159 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f73b806ac00 0x7f73b806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:07.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.401+0000 7f73dce00640 1 --2- 172.21.15.102:0/2575641159 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f73b806ac00 0x7f73b806d0c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f73c4036000 tx=0x7f73c403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:07.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.404+0000 7f73d5600640 1 -- 172.21.15.102:0/2575641159 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f73cc054840 con 0x7f73d80ca7b0 2024-03-23T03:47:07.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.593+0000 7f73e0200640 1 -- 172.21.15.102:0/2575641159 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f73d80b27d0 con 0x7f73d80ca7b0 2024-03-23T03:47:07.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.594+0000 7f73d5600640 1 -- 172.21.15.102:0/2575641159 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f73cc057e60 con 0x7f73d80ca7b0 2024-03-23T03:47:07.595 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:47:07.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.596+0000 7f73e0200640 1 -- 172.21.15.102:0/2575641159 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f73b806ac00 msgr2=0x7f73b806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:07.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.596+0000 7f73e0200640 1 --2- 172.21.15.102:0/2575641159 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f73b806ac00 0x7f73b806d0c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f73c4036000 tx=0x7f73c403c040 comp rx=0 tx=0).stop 2024-03-23T03:47:07.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.596+0000 7f73e0200640 1 -- 172.21.15.102:0/2575641159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73d80ca7b0 msgr2=0x7f73d814bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:07.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.596+0000 7f73e0200640 1 --2- 172.21.15.102:0/2575641159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73d80ca7b0 0x7f73d814bad0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f73cc00ebe0 tx=0x7f73cc00c520 comp rx=0 tx=0).stop 2024-03-23T03:47:07.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.596+0000 7f73e0200640 1 -- 172.21.15.102:0/2575641159 shutdown_connections 2024-03-23T03:47:07.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.596+0000 7f73e0200640 1 --2- 172.21.15.102:0/2575641159 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f73b806ac00 0x7f73b806d0c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.596+0000 7f73e0200640 1 --2- 172.21.15.102:0/2575641159 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f73d8160d20 0x7f73d80b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.596+0000 7f73e0200640 1 --2- 172.21.15.102:0/2575641159 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73d80cee70 0x7f73d80cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.596+0000 7f73e0200640 1 --2- 172.21.15.102:0/2575641159 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73d80ca7b0 0x7f73d814bad0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.596+0000 7f73e0200640 1 -- 172.21.15.102:0/2575641159 >> 172.21.15.102:0/2575641159 conn(0x7f73d806a0b0 msgr2=0x7f73d80b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:07.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.597+0000 7f73e0200640 1 -- 172.21.15.102:0/2575641159 shutdown_connections 2024-03-23T03:47:07.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.597+0000 7f73e0200640 1 -- 172.21.15.102:0/2575641159 wait complete. 2024-03-23T03:47:07.613 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:47:07.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.811+0000 7f013dc00640 1 Processor -- start 2024-03-23T03:47:07.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.811+0000 7f013dc00640 1 -- start start 2024-03-23T03:47:07.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.812+0000 7f013dc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0138159830 0x7f01380cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:07.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.812+0000 7f013dc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f01380ce930 0x7f01380cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:07.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.812+0000 7f013dc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0138159c10 con 0x7f01380ca7b0 2024-03-23T03:47:07.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.812+0000 7f013dc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f013806de50 con 0x7f01380ce930 2024-03-23T03:47:07.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.812+0000 7f013dc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01380cf300 con 0x7f0138159830 2024-03-23T03:47:07.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.812+0000 7f012fe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f01380ca7b0 0x7f01380cdcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46708/0 (socket says 172.21.15.102:46708) 2024-03-23T03:47:07.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.813+0000 7f012fe00640 1 -- 172.21.15.102:0/2095200620 learned_addr learned my addr 172.21.15.102:0/2095200620 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:07.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.813+0000 7f0137400640 1 --2- 172.21.15.102:0/2095200620 >> v2:172.21.15.102:3300/0 conn(0x7f01380ce930 0x7f01380cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:07.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.813+0000 7f0137400640 1 -- 172.21.15.102:0/2095200620 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0138159830 msgr2=0x7f01380cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:07.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.813+0000 7f0136a00640 1 --2- 172.21.15.102:0/2095200620 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0138159830 0x7f01380cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:07.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.813+0000 7f0136000640 1 -- 172.21.15.102:0/2095200620 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1506153779 0 0) 0x7f0138159c10 con 0x7f01380ca7b0 2024-03-23T03:47:07.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.813+0000 7f0137400640 1 --2- 172.21.15.102:0/2095200620 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0138159830 0x7f01380cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.813+0000 7f0137400640 1 -- 172.21.15.102:0/2095200620 >> v1:172.21.15.102:6789/0 conn(0x7f01380ca7b0 legacy=0x7f01380cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:07.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.813+0000 7f0137400640 1 -- 172.21.15.102:0/2095200620 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01380cd840 con 0x7f01380ce930 2024-03-23T03:47:07.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.813+0000 7f0136a00640 1 --2- 172.21.15.102:0/2095200620 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0138159830 0x7f01380cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:07.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.814+0000 7f0137400640 1 --2- 172.21.15.102:0/2095200620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01380ce930 0x7f01380cedc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0130009d90 tx=0x7f013002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf2ab9191f145f27 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:07.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.814+0000 7f0136000640 1 -- 172.21.15.102:0/2095200620 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013002cce0 con 0x7f01380ce930 2024-03-23T03:47:07.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.814+0000 7f0136000640 1 -- 172.21.15.102:0/2095200620 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013002ce40 con 0x7f01380ce930 2024-03-23T03:47:07.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.814+0000 7f0136000640 1 -- 172.21.15.102:0/2095200620 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0130034660 con 0x7f01380ce930 2024-03-23T03:47:07.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.815+0000 7f013dc00640 1 -- 172.21.15.102:0/2095200620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01380ce930 msgr2=0x7f01380cedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:07.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.815+0000 7f013dc00640 1 --2- 172.21.15.102:0/2095200620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01380ce930 0x7f01380cedc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0130009d90 tx=0x7f013002bf30 comp rx=0 tx=0).stop 2024-03-23T03:47:07.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.815+0000 7f013dc00640 1 -- 172.21.15.102:0/2095200620 shutdown_connections 2024-03-23T03:47:07.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.815+0000 7f013dc00640 1 --2- 172.21.15.102:0/2095200620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01380ce930 0x7f01380cedc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.815+0000 7f013dc00640 1 --2- 172.21.15.102:0/2095200620 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0138159830 0x7f01380cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.815+0000 7f013dc00640 1 -- 172.21.15.102:0/2095200620 >> 172.21.15.102:0/2095200620 conn(0x7f013806a0b0 msgr2=0x7f013806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:07.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.815+0000 7f013dc00640 1 -- 172.21.15.102:0/2095200620 shutdown_connections 2024-03-23T03:47:07.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.815+0000 7f013dc00640 1 -- 172.21.15.102:0/2095200620 wait complete. 2024-03-23T03:47:07.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.817+0000 7f013dc00640 1 Processor -- start 2024-03-23T03:47:07.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.817+0000 7f013dc00640 1 -- start start 2024-03-23T03:47:07.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.818+0000 7f013dc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f01380ca7b0 0x7f01381491b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:07.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.818+0000 7f013dc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01380ce930 0x7f01381524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:07.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.818+0000 7f013dc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0138159830 0x7f013815c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.818+0000 7f013dc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01381496f0 con 0x7f01380ce930 2024-03-23T03:47:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.818+0000 7f013dc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f013815b070 con 0x7f0138159830 2024-03-23T03:47:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.818+0000 7f013dc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f013815b350 con 0x7f01380ca7b0 2024-03-23T03:47:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.818+0000 7f012fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01380ce930 0x7f01381524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.818+0000 7f012fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01380ce930 0x7f01381524b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53644/0 (socket says 172.21.15.102:53644) 2024-03-23T03:47:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.818+0000 7f012fe00640 1 -- 172.21.15.102:0/438631158 learned_addr learned my addr 172.21.15.102:0/438631158 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.818+0000 7f0136a00640 1 --2- 172.21.15.102:0/438631158 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f01380ca7b0 0x7f01381491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.818+0000 7f0137400640 1 --2- 172.21.15.102:0/438631158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0138159830 0x7f013815c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.818+0000 7f012fe00640 1 -- 172.21.15.102:0/438631158 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f01380ca7b0 msgr2=0x7f01381491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.818+0000 7f012fe00640 1 --2- 172.21.15.102:0/438631158 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f01380ca7b0 0x7f01381491b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.818+0000 7f012fe00640 1 -- 172.21.15.102:0/438631158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0138159830 msgr2=0x7f013815c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.818+0000 7f012fe00640 1 --2- 172.21.15.102:0/438631158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0138159830 0x7f013815c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.818+0000 7f012fe00640 1 -- 172.21.15.102:0/438631158 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0130009a70 con 0x7f01380ce930 2024-03-23T03:47:07.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.818+0000 7f0136a00640 1 --2- 172.21.15.102:0/438631158 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f01380ca7b0 0x7f01381491b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:07.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.819+0000 7f012fe00640 1 --2- 172.21.15.102:0/438631158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01380ce930 0x7f01381524b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f012000c4d0 tx=0x7f012000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:07.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.819+0000 7f012ea00640 1 -- 172.21.15.102:0/438631158 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01200074e0 con 0x7f01380ce930 2024-03-23T03:47:07.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.819+0000 7f012ea00640 1 -- 172.21.15.102:0/438631158 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0120007660 con 0x7f01380ce930 2024-03-23T03:47:07.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.819+0000 7f013dc00640 1 -- 172.21.15.102:0/438631158 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f013815b630 con 0x7f01380ce930 2024-03-23T03:47:07.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.819+0000 7f012ea00640 1 -- 172.21.15.102:0/438631158 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0120004a20 con 0x7f01380ce930 2024-03-23T03:47:07.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.819+0000 7f013dc00640 1 -- 172.21.15.102:0/438631158 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f013815bbf0 con 0x7f01380ce930 2024-03-23T03:47:07.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.820+0000 7f013dc00640 1 -- 172.21.15.102:0/438631158 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f013815be10 con 0x7f01380ce930 2024-03-23T03:47:07.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.821+0000 7f012ea00640 1 -- 172.21.15.102:0/438631158 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0120004b80 con 0x7f01380ce930 2024-03-23T03:47:07.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.822+0000 7f012ea00640 1 --2- 172.21.15.102:0/438631158 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f011406ac00 0x7f011406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:07.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.822+0000 7f012ea00640 1 -- 172.21.15.102:0/438631158 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0120088b50 con 0x7f01380ce930 2024-03-23T03:47:07.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.822+0000 7f0136a00640 1 --2- 172.21.15.102:0/438631158 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f011406ac00 0x7f011406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:07.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.823+0000 7f0136a00640 1 --2- 172.21.15.102:0/438631158 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f011406ac00 0x7f011406d0c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0128006fd0 tx=0x7f0128008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:07.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:07.826+0000 7f012ea00640 1 -- 172.21.15.102:0/438631158 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0120053a30 con 0x7f01380ce930 2024-03-23T03:47:08.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.019+0000 7f013dc00640 1 -- 172.21.15.102:0/438631158 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0138073390 con 0x7f01380ce930 2024-03-23T03:47:08.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.020+0000 7f012ea00640 1 -- 172.21.15.102:0/438631158 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f01200180a0 con 0x7f01380ce930 2024-03-23T03:47:08.021 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:08.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.023+0000 7f013dc00640 1 -- 172.21.15.102:0/438631158 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f011406ac00 msgr2=0x7f011406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:08.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.023+0000 7f013dc00640 1 --2- 172.21.15.102:0/438631158 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f011406ac00 0x7f011406d0c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0128006fd0 tx=0x7f0128008040 comp rx=0 tx=0).stop 2024-03-23T03:47:08.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.023+0000 7f013dc00640 1 -- 172.21.15.102:0/438631158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01380ce930 msgr2=0x7f01381524b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:08.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.023+0000 7f013dc00640 1 --2- 172.21.15.102:0/438631158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01380ce930 0x7f01381524b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f012000c4d0 tx=0x7f012000c990 comp rx=0 tx=0).stop 2024-03-23T03:47:08.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.023+0000 7f013dc00640 1 -- 172.21.15.102:0/438631158 shutdown_connections 2024-03-23T03:47:08.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.023+0000 7f013dc00640 1 --2- 172.21.15.102:0/438631158 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f011406ac00 0x7f011406d0c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.023+0000 7f013dc00640 1 --2- 172.21.15.102:0/438631158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0138159830 0x7f013815c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.023+0000 7f013dc00640 1 --2- 172.21.15.102:0/438631158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01380ce930 0x7f01381524b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.023+0000 7f013dc00640 1 --2- 172.21.15.102:0/438631158 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f01380ca7b0 0x7f01381491b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.023+0000 7f013dc00640 1 -- 172.21.15.102:0/438631158 >> 172.21.15.102:0/438631158 conn(0x7f013806a0b0 msgr2=0x7f01380b1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:08.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.023+0000 7f013dc00640 1 -- 172.21.15.102:0/438631158 shutdown_connections 2024-03-23T03:47:08.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.023+0000 7f013dc00640 1 -- 172.21.15.102:0/438631158 wait complete. 2024-03-23T03:47:08.042 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:47:08.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.243+0000 7f07a9600640 1 Processor -- start 2024-03-23T03:47:08.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.243+0000 7f07a9600640 1 -- start start 2024-03-23T03:47:08.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.244+0000 7f07a9600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f07a4070150 0x7f07a40705b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:08.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.244+0000 7f07a9600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a4070af0 con 0x7f07a40cc810 2024-03-23T03:47:08.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.245+0000 7f07a9600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a40caec0 con 0x7f07a4158db0 2024-03-23T03:47:08.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.245+0000 7f07a9600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a40cba90 con 0x7f07a4070150 2024-03-23T03:47:08.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.245+0000 7f07a3400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f07a40cc810 0x7f07a40ccc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46718/0 (socket says 172.21.15.102:46718) 2024-03-23T03:47:08.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.245+0000 7f07a3400640 1 -- 172.21.15.102:0/870694671 learned_addr learned my addr 172.21.15.102:0/870694671 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:08.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.245+0000 7f079a000640 1 --2- 172.21.15.102:0/870694671 >> v2:172.21.15.173:3300/0 conn(0x7f07a4070150 0x7f07a40705b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:08.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.245+0000 7f07a2000640 1 -- 172.21.15.102:0/870694671 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350031477 0 0) 0x7f07a4070af0 con 0x7f07a40cc810 2024-03-23T03:47:08.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.245+0000 7f07a2000640 1 -- 172.21.15.102:0/870694671 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0788004630 con 0x7f07a40cc810 2024-03-23T03:47:08.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2000640 1 -- 172.21.15.102:0/870694671 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 281035340 0 0) 0x7f0788004630 con 0x7f07a40cc810 2024-03-23T03:47:08.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2000640 1 -- 172.21.15.102:0/870694671 >> v2:172.21.15.173:3300/0 conn(0x7f07a4070150 msgr2=0x7f07a40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:08.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2000640 1 --2- 172.21.15.102:0/870694671 >> v2:172.21.15.173:3300/0 conn(0x7f07a4070150 0x7f07a40705b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2000640 1 -- 172.21.15.102:0/870694671 >> v1:172.21.15.173:6789/0 conn(0x7f07a4158db0 legacy=0x7f07a415d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:08.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2000640 1 -- 172.21.15.102:0/870694671 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a4159190 con 0x7f07a40cc810 2024-03-23T03:47:08.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2000640 1 -- 172.21.15.102:0/870694671 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f079c0019f0 con 0x7f07a40cc810 2024-03-23T03:47:08.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2000640 1 -- 172.21.15.102:0/870694671 >> v1:172.21.15.102:6789/0 conn(0x7f07a40cc810 legacy=0x7f07a40ccc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:08.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2000640 1 --2- 172.21.15.102:0/870694671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0788004a80 0x7f0788004ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:08.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2000640 1 --2- 172.21.15.102:0/870694671 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f07880058d0 0x7f0788005d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:08.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2000640 1 --2- 172.21.15.102:0/870694671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0788006290 0x7f078800ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:08.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2000640 1 -- 172.21.15.102:0/870694671 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a40cba90 con 0x7f0788004a80 2024-03-23T03:47:08.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2000640 1 -- 172.21.15.102:0/870694671 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078800d130 con 0x7f0788006290 2024-03-23T03:47:08.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2000640 1 -- 172.21.15.102:0/870694671 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078800d2f0 con 0x7f07880058d0 2024-03-23T03:47:08.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2a00640 1 --2- 172.21.15.102:0/870694671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0788004a80 0x7f0788004ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:08.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a3400640 1 --2- 172.21.15.102:0/870694671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0788006290 0x7f078800ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:08.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2a00640 1 -- 172.21.15.102:0/870694671 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f07880058d0 msgr2=0x7f0788005d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:08.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2a00640 1 --2- 172.21.15.102:0/870694671 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f07880058d0 0x7f0788005d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2a00640 1 -- 172.21.15.102:0/870694671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0788006290 msgr2=0x7f078800ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:08.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2a00640 1 --2- 172.21.15.102:0/870694671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0788006290 0x7f078800ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a2a00640 1 -- 172.21.15.102:0/870694671 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0788004630 con 0x7f0788004a80 2024-03-23T03:47:08.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.246+0000 7f07a3400640 1 -- 172.21.15.102:0/870694671 reap_dead start 2024-03-23T03:47:08.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.247+0000 7f07a2a00640 1 --2- 172.21.15.102:0/870694671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0788004a80 0x7f0788004ea0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0794001750 tx=0x7f0788006900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=572a054132d76971 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:08.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.248+0000 7f07a2000640 1 -- 172.21.15.102:0/870694671 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07940052d0 con 0x7f0788004a80 2024-03-23T03:47:08.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.248+0000 7f07a2000640 1 -- 172.21.15.102:0/870694671 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079402e050 con 0x7f0788004a80 2024-03-23T03:47:08.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.248+0000 7f07a2000640 1 -- 172.21.15.102:0/870694671 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794003830 con 0x7f0788004a80 2024-03-23T03:47:08.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.248+0000 7f07a9600640 1 -- 172.21.15.102:0/870694671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0788004a80 msgr2=0x7f0788004ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:08.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.248+0000 7f07a9600640 1 --2- 172.21.15.102:0/870694671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0788004a80 0x7f0788004ea0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0794001750 tx=0x7f0788006900 comp rx=0 tx=0).stop 2024-03-23T03:47:08.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.248+0000 7f07a9600640 1 -- 172.21.15.102:0/870694671 shutdown_connections 2024-03-23T03:47:08.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.248+0000 7f07a9600640 1 --2- 172.21.15.102:0/870694671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0788004a80 0x7f0788004ea0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.248+0000 7f07a9600640 1 -- 172.21.15.102:0/870694671 >> 172.21.15.102:0/870694671 conn(0x7f07a406a0b0 msgr2=0x7f07a406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:08.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.248+0000 7f07a9600640 1 -- 172.21.15.102:0/870694671 shutdown_connections 2024-03-23T03:47:08.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.248+0000 7f07a9600640 1 -- 172.21.15.102:0/870694671 wait complete. 2024-03-23T03:47:08.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.250+0000 7f07a9600640 1 Processor -- start 2024-03-23T03:47:08.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.250+0000 7f07a9600640 1 -- start start 2024-03-23T03:47:08.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.251+0000 7f07a9600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0788004a80 0x7f07a4157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:08.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.251+0000 7f07a9600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f07a40ad510 0x7f07a4158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:08.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.251+0000 7f07a9600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07a4150f00 0x7f07a4151380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:08.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.251+0000 7f07a9600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a41586a0 con 0x7f0788004a80 2024-03-23T03:47:08.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.251+0000 7f07a9600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a4140c70 con 0x7f07a4150f00 2024-03-23T03:47:08.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.251+0000 7f07a9600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a4151c30 con 0x7f07a40ad510 2024-03-23T03:47:08.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.251+0000 7f07a2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0788004a80 0x7f07a4157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:08.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.251+0000 7f07a2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0788004a80 0x7f07a4157c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53662/0 (socket says 172.21.15.102:53662) 2024-03-23T03:47:08.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.251+0000 7f07a2a00640 1 -- 172.21.15.102:0/172601037 learned_addr learned my addr 172.21.15.102:0/172601037 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:08.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.252+0000 7f07a2a00640 1 -- 172.21.15.102:0/172601037 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f07a40ad510 msgr2=0x7f07a4158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:08.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.252+0000 7f07a2a00640 1 --2- 172.21.15.102:0/172601037 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f07a40ad510 0x7f07a4158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.252+0000 7f07a2a00640 1 -- 172.21.15.102:0/172601037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07a4150f00 msgr2=0x7f07a4151380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:08.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.252+0000 7f07a3400640 1 --2- 172.21.15.102:0/172601037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07a4150f00 0x7f07a4151380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:08.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.252+0000 7f07a2a00640 1 --2- 172.21.15.102:0/172601037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07a4150f00 0x7f07a4151380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.252+0000 7f07a2a00640 1 -- 172.21.15.102:0/172601037 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a4141030 con 0x7f0788004a80 2024-03-23T03:47:08.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.252+0000 7f07a2a00640 1 --2- 172.21.15.102:0/172601037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0788004a80 0x7f07a4157c20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f079402bc00 tx=0x7f0794001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:08.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.252+0000 7f079b400640 1 -- 172.21.15.102:0/172601037 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07940076c0 con 0x7f0788004a80 2024-03-23T03:47:08.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.252+0000 7f079b400640 1 -- 172.21.15.102:0/172601037 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0794007a30 con 0x7f0788004a80 2024-03-23T03:47:08.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.252+0000 7f079b400640 1 -- 172.21.15.102:0/172601037 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794006c20 con 0x7f0788004a80 2024-03-23T03:47:08.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.252+0000 7f07a9600640 1 -- 172.21.15.102:0/172601037 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a4141310 con 0x7f0788004a80 2024-03-23T03:47:08.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.252+0000 7f07a9600640 1 -- 172.21.15.102:0/172601037 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a41418d0 con 0x7f0788004a80 2024-03-23T03:47:08.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.253+0000 7f07a9600640 1 -- 172.21.15.102:0/172601037 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a4073390 con 0x7f0788004a80 2024-03-23T03:47:08.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.254+0000 7f079b400640 1 -- 172.21.15.102:0/172601037 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f07940035f0 con 0x7f0788004a80 2024-03-23T03:47:08.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.255+0000 7f079b400640 1 --2- 172.21.15.102:0/172601037 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f078006ac00 0x7f078006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:08.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.255+0000 7f079b400640 1 -- 172.21.15.102:0/172601037 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f07940ab270 con 0x7f0788004a80 2024-03-23T03:47:08.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.255+0000 7f079a000640 1 --2- 172.21.15.102:0/172601037 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f078006ac00 0x7f078006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:08.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.256+0000 7f079a000640 1 --2- 172.21.15.102:0/172601037 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f078006ac00 0x7f078006d0c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f078c0093d0 tx=0x7f078c009300 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:08.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.259+0000 7f079b400640 1 -- 172.21.15.102:0/172601037 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0794079be0 con 0x7f0788004a80 2024-03-23T03:47:08.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.448+0000 7f07a9600640 1 -- 172.21.15.102:0/172601037 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f07a415c690 con 0x7f0788004a80 2024-03-23T03:47:08.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.449+0000 7f079b400640 1 -- 172.21.15.102:0/172601037 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0794079770 con 0x7f0788004a80 2024-03-23T03:47:08.450 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:47:08.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.451+0000 7f07a9600640 1 -- 172.21.15.102:0/172601037 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f078006ac00 msgr2=0x7f078006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:08.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.451+0000 7f07a9600640 1 --2- 172.21.15.102:0/172601037 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f078006ac00 0x7f078006d0c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f078c0093d0 tx=0x7f078c009300 comp rx=0 tx=0).stop 2024-03-23T03:47:08.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.451+0000 7f07a9600640 1 -- 172.21.15.102:0/172601037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0788004a80 msgr2=0x7f07a4157c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:08.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.451+0000 7f07a9600640 1 --2- 172.21.15.102:0/172601037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0788004a80 0x7f07a4157c20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f079402bc00 tx=0x7f0794001120 comp rx=0 tx=0).stop 2024-03-23T03:47:08.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.452+0000 7f07a9600640 1 -- 172.21.15.102:0/172601037 shutdown_connections 2024-03-23T03:47:08.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.452+0000 7f07a9600640 1 --2- 172.21.15.102:0/172601037 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f078006ac00 0x7f078006d0c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.452+0000 7f07a9600640 1 --2- 172.21.15.102:0/172601037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07a4150f00 0x7f07a4151380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.452+0000 7f07a9600640 1 --2- 172.21.15.102:0/172601037 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f07a40ad510 0x7f07a4158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.452+0000 7f07a9600640 1 --2- 172.21.15.102:0/172601037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0788004a80 0x7f07a4157c20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.452+0000 7f07a9600640 1 -- 172.21.15.102:0/172601037 >> 172.21.15.102:0/172601037 conn(0x7f07a406a0b0 msgr2=0x7f07a406a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:08.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.452+0000 7f07a9600640 1 -- 172.21.15.102:0/172601037 shutdown_connections 2024-03-23T03:47:08.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.452+0000 7f07a9600640 1 -- 172.21.15.102:0/172601037 wait complete. 2024-03-23T03:47:08.469 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-23T03:47:08.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.674+0000 7fd340200640 1 Processor -- start 2024-03-23T03:47:08.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.674+0000 7fd340200640 1 -- start start 2024-03-23T03:47:08.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.676+0000 7fd340200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd3380cd740 0x7fd3380b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:08.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.676+0000 7fd340200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd3380b5aa0 0x7fd3380b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:08.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.676+0000 7fd340200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3380d20c0 con 0x7fd3380d0d40 2024-03-23T03:47:08.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.676+0000 7fd340200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3380d14b0 con 0x7fd3380b5aa0 2024-03-23T03:47:08.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.676+0000 7fd340200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3380d1790 con 0x7fd3380cd740 2024-03-23T03:47:08.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.676+0000 7fd33e200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd3380b5aa0 0x7fd3380b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.676+0000 7fd33e200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd3380b5aa0 0x7fd3380b5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53674/0 (socket says 172.21.15.102:53674) 2024-03-23T03:47:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.676+0000 7fd33e200640 1 -- 172.21.15.102:0/2054538481 learned_addr learned my addr 172.21.15.102:0/2054538481 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.676+0000 7fd33ce00640 1 --1- 172.21.15.102:0/2054538481 >> v1:172.21.15.102:6789/0 conn(0x7fd3380d0d40 0x7fd3380cdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46728/0 (socket says 172.21.15.102:46728) 2024-03-23T03:47:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.677+0000 7fd33d800640 1 --2- 172.21.15.102:0/2054538481 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd3380cd740 0x7fd3380b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.677+0000 7fd33e200640 1 -- 172.21.15.102:0/2054538481 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd3380cd740 msgr2=0x7fd3380b84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.677+0000 7fd33e200640 1 --2- 172.21.15.102:0/2054538481 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd3380cd740 0x7fd3380b84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.677+0000 7fd33e200640 1 -- 172.21.15.102:0/2054538481 >> v1:172.21.15.102:6789/0 conn(0x7fd3380d0d40 legacy=0x7fd3380cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.677+0000 7fd337e00640 1 -- 172.21.15.102:0/2054538481 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228561682 0 0) 0x7fd3380d20c0 con 0x7fd3380d0d40 2024-03-23T03:47:08.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.677+0000 7fd33e200640 1 -- 172.21.15.102:0/2054538481 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3380d1a70 con 0x7fd3380b5aa0 2024-03-23T03:47:08.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.678+0000 7fd33e200640 1 --2- 172.21.15.102:0/2054538481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3380b5aa0 0x7fd3380b5f30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd330009870 tx=0x7fd33002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b2a6d5fb0621fb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:08.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.678+0000 7fd337e00640 1 -- 172.21.15.102:0/2054538481 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd330030840 con 0x7fd3380b5aa0 2024-03-23T03:47:08.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.678+0000 7fd337e00640 1 -- 172.21.15.102:0/2054538481 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3300309a0 con 0x7fd3380b5aa0 2024-03-23T03:47:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.678+0000 7fd337e00640 1 -- 172.21.15.102:0/2054538481 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd330030d30 con 0x7fd3380b5aa0 2024-03-23T03:47:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.679+0000 7fd340200640 1 -- 172.21.15.102:0/2054538481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3380b5aa0 msgr2=0x7fd3380b5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.679+0000 7fd340200640 1 --2- 172.21.15.102:0/2054538481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3380b5aa0 0x7fd3380b5f30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd330009870 tx=0x7fd33002b980 comp rx=0 tx=0).stop 2024-03-23T03:47:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.679+0000 7fd340200640 1 -- 172.21.15.102:0/2054538481 shutdown_connections 2024-03-23T03:47:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.679+0000 7fd340200640 1 --2- 172.21.15.102:0/2054538481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3380b5aa0 0x7fd3380b5f30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.679+0000 7fd340200640 1 --2- 172.21.15.102:0/2054538481 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd3380cd740 0x7fd3380b84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.679+0000 7fd340200640 1 -- 172.21.15.102:0/2054538481 >> 172.21.15.102:0/2054538481 conn(0x7fd3380cf5d0 msgr2=0x7fd3380cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.679+0000 7fd340200640 1 -- 172.21.15.102:0/2054538481 shutdown_connections 2024-03-23T03:47:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.679+0000 7fd340200640 1 -- 172.21.15.102:0/2054538481 wait complete. 2024-03-23T03:47:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.681+0000 7fd340200640 1 Processor -- start 2024-03-23T03:47:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.681+0000 7fd340200640 1 -- start start 2024-03-23T03:47:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.682+0000 7fd340200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd3380b5aa0 0x7fd3380cc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.682+0000 7fd340200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3380cd740 0x7fd3380c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.682+0000 7fd340200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3380d0d40 0x7fd338159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.682+0000 7fd340200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3380cc920 con 0x7fd3380cd740 2024-03-23T03:47:08.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.682+0000 7fd340200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3381546a0 con 0x7fd3380d0d40 2024-03-23T03:47:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.682+0000 7fd340200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd338158760 con 0x7fd3380b5aa0 2024-03-23T03:47:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.682+0000 7fd33ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3380cd740 0x7fd3380c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.682+0000 7fd33ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3380cd740 0x7fd3380c7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53676/0 (socket says 172.21.15.102:53676) 2024-03-23T03:47:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.682+0000 7fd33ce00640 1 -- 172.21.15.102:0/2833872342 learned_addr learned my addr 172.21.15.102:0/2833872342 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.682+0000 7fd33ce00640 1 -- 172.21.15.102:0/2833872342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd3380b5aa0 msgr2=0x7fd3380cc3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.682+0000 7fd33ce00640 1 --2- 172.21.15.102:0/2833872342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd3380b5aa0 0x7fd3380cc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.682+0000 7fd33ce00640 1 -- 172.21.15.102:0/2833872342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3380d0d40 msgr2=0x7fd338159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.682+0000 7fd33ce00640 1 --2- 172.21.15.102:0/2833872342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3380d0d40 0x7fd338159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.682+0000 7fd33ce00640 1 -- 172.21.15.102:0/2833872342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3300094d0 con 0x7fd3380cd740 2024-03-23T03:47:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.683+0000 7fd33ce00640 1 --2- 172.21.15.102:0/2833872342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3380cd740 0x7fd3380c7f20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd32800c7c0 tx=0x7fd32800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.683+0000 7fd335600640 1 -- 172.21.15.102:0/2833872342 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32800a640 con 0x7fd3380cd740 2024-03-23T03:47:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.683+0000 7fd335600640 1 -- 172.21.15.102:0/2833872342 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32800a7c0 con 0x7fd3380cd740 2024-03-23T03:47:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.683+0000 7fd340200640 1 -- 172.21.15.102:0/2833872342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd338158a40 con 0x7fd3380cd740 2024-03-23T03:47:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.683+0000 7fd335600640 1 -- 172.21.15.102:0/2833872342 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32800ab70 con 0x7fd3380cd740 2024-03-23T03:47:08.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.683+0000 7fd340200640 1 -- 172.21.15.102:0/2833872342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd338159000 con 0x7fd3380cd740 2024-03-23T03:47:08.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.684+0000 7fd340200640 1 -- 172.21.15.102:0/2833872342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd338073390 con 0x7fd3380cd740 2024-03-23T03:47:08.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.685+0000 7fd335600640 1 -- 172.21.15.102:0/2833872342 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd32800acd0 con 0x7fd3380cd740 2024-03-23T03:47:08.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.686+0000 7fd335600640 1 --2- 172.21.15.102:0/2833872342 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd31c06ac00 0x7fd31c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:08.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.686+0000 7fd335600640 1 -- 172.21.15.102:0/2833872342 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd328088f20 con 0x7fd3380cd740 2024-03-23T03:47:08.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.686+0000 7fd33d800640 1 --2- 172.21.15.102:0/2833872342 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd31c06ac00 0x7fd31c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:08.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.687+0000 7fd33d800640 1 --2- 172.21.15.102:0/2833872342 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd31c06ac00 0x7fd31c06d0c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0098f0 tx=0x7fd32c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:08.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.691+0000 7fd335600640 1 -- 172.21.15.102:0/2833872342 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd328053d70 con 0x7fd3380cd740 2024-03-23T03:47:08.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.883+0000 7fd340200640 1 -- 172.21.15.102:0/2833872342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd3380b20b0 con 0x7fd3380cd740 2024-03-23T03:47:08.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.884+0000 7fd335600640 1 -- 172.21.15.102:0/2833872342 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd328018020 con 0x7fd3380cd740 2024-03-23T03:47:08.887 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:08.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.888+0000 7fd340200640 1 -- 172.21.15.102:0/2833872342 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd31c06ac00 msgr2=0x7fd31c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:08.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.888+0000 7fd340200640 1 --2- 172.21.15.102:0/2833872342 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd31c06ac00 0x7fd31c06d0c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd32c0098f0 tx=0x7fd32c008040 comp rx=0 tx=0).stop 2024-03-23T03:47:08.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.888+0000 7fd340200640 1 -- 172.21.15.102:0/2833872342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3380cd740 msgr2=0x7fd3380c7f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:08.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.888+0000 7fd340200640 1 --2- 172.21.15.102:0/2833872342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3380cd740 0x7fd3380c7f20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd32800c7c0 tx=0x7fd32800cc80 comp rx=0 tx=0).stop 2024-03-23T03:47:08.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.888+0000 7fd340200640 1 -- 172.21.15.102:0/2833872342 shutdown_connections 2024-03-23T03:47:08.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.888+0000 7fd340200640 1 --2- 172.21.15.102:0/2833872342 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd31c06ac00 0x7fd31c06d0c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.888+0000 7fd340200640 1 --2- 172.21.15.102:0/2833872342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd3380d0d40 0x7fd338159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.888+0000 7fd340200640 1 --2- 172.21.15.102:0/2833872342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3380cd740 0x7fd3380c7f20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.888+0000 7fd340200640 1 --2- 172.21.15.102:0/2833872342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd3380b5aa0 0x7fd3380cc3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:08.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.888+0000 7fd340200640 1 -- 172.21.15.102:0/2833872342 >> 172.21.15.102:0/2833872342 conn(0x7fd3380cf5d0 msgr2=0x7fd3380726a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:08.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.888+0000 7fd340200640 1 -- 172.21.15.102:0/2833872342 shutdown_connections 2024-03-23T03:47:08.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:08.888+0000 7fd340200640 1 -- 172.21.15.102:0/2833872342 wait complete. 2024-03-23T03:47:08.907 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:47:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.111+0000 7f5e4b200640 1 Processor -- start 2024-03-23T03:47:09.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.111+0000 7f5e4b200640 1 -- start start 2024-03-23T03:47:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.113+0000 7f5e4b200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e4c15c820 0x7f5e4c0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.113+0000 7f5e4b200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e4c0ccd90 con 0x7f5e4c160e90 2024-03-23T03:47:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.113+0000 7f5e4b200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e4c0b7fc0 con 0x7f5e4c0cf7c0 2024-03-23T03:47:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.113+0000 7f5e4b200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c0cdd70 con 0x7f5e4c15c820 2024-03-23T03:47:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.113+0000 7f5e49400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5e4c160e90 0x7f5e4c0cf0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46738/0 (socket says 172.21.15.102:46738) 2024-03-23T03:47:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.113+0000 7f5e49400640 1 -- 172.21.15.102:0/3383719481 learned_addr learned my addr 172.21.15.102:0/3383719481 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:09.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.113+0000 7f5e49e00640 1 --2- 172.21.15.102:0/3383719481 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e4c15c820 0x7f5e4c0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.114+0000 7f5e48a00640 1 -- 172.21.15.102:0/3383719481 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824270625 0 0) 0x7f5e4c0ccd90 con 0x7f5e4c160e90 2024-03-23T03:47:09.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.114+0000 7f5e48a00640 1 -- 172.21.15.102:0/3383719481 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e2c003520 con 0x7f5e4c160e90 2024-03-23T03:47:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.114+0000 7f5e48a00640 1 -- 172.21.15.102:0/3383719481 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2052639643 0 0) 0x7f5e2c003520 con 0x7f5e4c160e90 2024-03-23T03:47:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.114+0000 7f5e48a00640 1 -- 172.21.15.102:0/3383719481 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e4c15c820 msgr2=0x7f5e4c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.114+0000 7f5e48a00640 1 --2- 172.21.15.102:0/3383719481 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e4c15c820 0x7f5e4c0cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.114+0000 7f5e48a00640 1 -- 172.21.15.102:0/3383719481 >> v1:172.21.15.173:6789/0 conn(0x7f5e4c0cf7c0 legacy=0x7f5e4c0d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.114+0000 7f5e48a00640 1 -- 172.21.15.102:0/3383719481 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4c0b82a0 con 0x7f5e4c160e90 2024-03-23T03:47:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.114+0000 7f5e48a00640 1 -- 172.21.15.102:0/3383719481 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f5e340019f0 con 0x7f5e4c160e90 2024-03-23T03:47:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.114+0000 7f5e48a00640 1 -- 172.21.15.102:0/3383719481 >> v1:172.21.15.102:6789/0 conn(0x7f5e4c160e90 legacy=0x7f5e4c0cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.114+0000 7f5e48a00640 1 --2- 172.21.15.102:0/3383719481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e2c0039a0 0x7f5e2c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.114+0000 7f5e48a00640 1 --2- 172.21.15.102:0/3383719481 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e4c15c820 0x7f5e2c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.114+0000 7f5e48a00640 1 --2- 172.21.15.102:0/3383719481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e2c004d00 0x7f5e2c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.114+0000 7f5e48a00640 1 -- 172.21.15.102:0/3383719481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c0cdd70 con 0x7f5e2c004d00 2024-03-23T03:47:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.114+0000 7f5e48a00640 1 -- 172.21.15.102:0/3383719481 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c00bac0 con 0x7f5e2c0039a0 2024-03-23T03:47:09.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.114+0000 7f5e48a00640 1 -- 172.21.15.102:0/3383719481 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2c00bc80 con 0x7f5e4c15c820 2024-03-23T03:47:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.115+0000 7f5e4a800640 1 --2- 172.21.15.102:0/3383719481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e2c004d00 0x7f5e2c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.115+0000 7f5e49400640 1 --2- 172.21.15.102:0/3383719481 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e4c15c820 0x7f5e2c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.115+0000 7f5e49e00640 1 --2- 172.21.15.102:0/3383719481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e2c0039a0 0x7f5e2c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.115+0000 7f5e4a800640 1 -- 172.21.15.102:0/3383719481 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e4c15c820 msgr2=0x7f5e2c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.115+0000 7f5e4a800640 1 --2- 172.21.15.102:0/3383719481 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e4c15c820 0x7f5e2c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.115+0000 7f5e4a800640 1 -- 172.21.15.102:0/3383719481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e2c0039a0 msgr2=0x7f5e2c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.115+0000 7f5e4a800640 1 --2- 172.21.15.102:0/3383719481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e2c0039a0 0x7f5e2c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.115+0000 7f5e4a800640 1 -- 172.21.15.102:0/3383719481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e2c003520 con 0x7f5e2c004d00 2024-03-23T03:47:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.116+0000 7f5e4a800640 1 --2- 172.21.15.102:0/3383719481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e2c004d00 0x7f5e2c005180 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5e44001720 tx=0x7f5e2c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf8306bb6854b655 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.116+0000 7f5e48a00640 1 -- 172.21.15.102:0/3383719481 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e440052d0 con 0x7f5e2c004d00 2024-03-23T03:47:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.116+0000 7f5e48a00640 1 -- 172.21.15.102:0/3383719481 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4402e050 con 0x7f5e2c004d00 2024-03-23T03:47:09.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.116+0000 7f5e48a00640 1 -- 172.21.15.102:0/3383719481 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e440035f0 con 0x7f5e2c004d00 2024-03-23T03:47:09.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.117+0000 7f5e4b200640 1 -- 172.21.15.102:0/3383719481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e2c004d00 msgr2=0x7f5e2c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:09.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.117+0000 7f5e4b200640 1 --2- 172.21.15.102:0/3383719481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e2c004d00 0x7f5e2c005180 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5e44001720 tx=0x7f5e2c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.117+0000 7f5e4a800640 1 -- 172.21.15.102:0/3383719481 reap_dead start 2024-03-23T03:47:09.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.117+0000 7f5e4b200640 1 -- 172.21.15.102:0/3383719481 shutdown_connections 2024-03-23T03:47:09.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.117+0000 7f5e4b200640 1 -- 172.21.15.102:0/3383719481 >> 172.21.15.102:0/3383719481 conn(0x7f5e4c06b070 msgr2=0x7f5e4c06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:09.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.117+0000 7f5e4b200640 1 -- 172.21.15.102:0/3383719481 shutdown_connections 2024-03-23T03:47:09.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.117+0000 7f5e4b200640 1 -- 172.21.15.102:0/3383719481 wait complete. 2024-03-23T03:47:09.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.119+0000 7f5e4b200640 1 Processor -- start 2024-03-23T03:47:09.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.119+0000 7f5e4b200640 1 -- start start 2024-03-23T03:47:09.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.120+0000 7f5e4b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e2c004d00 0x7f5e4c0b6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:09.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.120+0000 7f5e4b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e4c147880 0x7f5e4c147ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:09.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.120+0000 7f5e4b200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e4c0c7de0 0x7f5e4c0c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:09.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.120+0000 7f5e4b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c0c87a0 con 0x7f5e4c147880 2024-03-23T03:47:09.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.120+0000 7f5e4b200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c148340 con 0x7f5e2c004d00 2024-03-23T03:47:09.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.120+0000 7f5e4b200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c148620 con 0x7f5e4c0c7de0 2024-03-23T03:47:09.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.120+0000 7f5e49400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e4c147880 0x7f5e4c147ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:09.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.120+0000 7f5e49400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e4c147880 0x7f5e4c147ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53686/0 (socket says 172.21.15.102:53686) 2024-03-23T03:47:09.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.120+0000 7f5e49400640 1 -- 172.21.15.102:0/841383338 learned_addr learned my addr 172.21.15.102:0/841383338 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:09.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.120+0000 7f5e4a800640 1 --2- 172.21.15.102:0/841383338 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e4c0c7de0 0x7f5e4c0c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:09.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.120+0000 7f5e49e00640 1 --2- 172.21.15.102:0/841383338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e2c004d00 0x7f5e4c0b6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:09.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.120+0000 7f5e49400640 1 -- 172.21.15.102:0/841383338 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e4c0c7de0 msgr2=0x7f5e4c0c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:09.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.120+0000 7f5e49400640 1 --2- 172.21.15.102:0/841383338 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e4c0c7de0 0x7f5e4c0c8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.120+0000 7f5e49400640 1 -- 172.21.15.102:0/841383338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e2c004d00 msgr2=0x7f5e4c0b6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:09.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.120+0000 7f5e49400640 1 --2- 172.21.15.102:0/841383338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e2c004d00 0x7f5e4c0b6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.120+0000 7f5e49400640 1 -- 172.21.15.102:0/841383338 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4c157b70 con 0x7f5e4c147880 2024-03-23T03:47:09.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.121+0000 7f5e49400640 1 --2- 172.21.15.102:0/841383338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e4c147880 0x7f5e4c147ce0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5e34005fd0 tx=0x7f5e3400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:09.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.121+0000 7f5e42000640 1 -- 172.21.15.102:0/841383338 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e3400ee50 con 0x7f5e4c147880 2024-03-23T03:47:09.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.121+0000 7f5e42000640 1 -- 172.21.15.102:0/841383338 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e34013420 con 0x7f5e4c147880 2024-03-23T03:47:09.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.121+0000 7f5e42000640 1 -- 172.21.15.102:0/841383338 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e340137b0 con 0x7f5e4c147880 2024-03-23T03:47:09.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.121+0000 7f5e4b200640 1 -- 172.21.15.102:0/841383338 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e4c157e50 con 0x7f5e4c147880 2024-03-23T03:47:09.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.121+0000 7f5e4b200640 1 -- 172.21.15.102:0/841383338 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e4c155b50 con 0x7f5e4c147880 2024-03-23T03:47:09.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.122+0000 7f5e4b200640 1 -- 172.21.15.102:0/841383338 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e4c1487e0 con 0x7f5e4c147880 2024-03-23T03:47:09.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.123+0000 7f5e42000640 1 -- 172.21.15.102:0/841383338 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5e34013910 con 0x7f5e4c147880 2024-03-23T03:47:09.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.124+0000 7f5e42000640 1 --2- 172.21.15.102:0/841383338 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e2806ac30 0x7f5e2806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:09.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.124+0000 7f5e42000640 1 -- 172.21.15.102:0/841383338 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5e34089f10 con 0x7f5e4c147880 2024-03-23T03:47:09.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.124+0000 7f5e49e00640 1 --2- 172.21.15.102:0/841383338 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e2806ac30 0x7f5e2806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:09.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.125+0000 7f5e49e00640 1 --2- 172.21.15.102:0/841383338 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e2806ac30 0x7f5e2806d0f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009950 tx=0x7f5e3c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:09.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.128+0000 7f5e42000640 1 -- 172.21.15.102:0/841383338 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5e34054d60 con 0x7f5e4c147880 2024-03-23T03:47:09.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.333+0000 7f5e4b200640 1 -- 172.21.15.102:0/841383338 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5e4c158040 con 0x7f5e4c147880 2024-03-23T03:47:09.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.334+0000 7f5e42000640 1 -- 172.21.15.102:0/841383338 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5e34058380 con 0x7f5e4c147880 2024-03-23T03:47:09.335 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:47:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.337+0000 7f5e4b200640 1 -- 172.21.15.102:0/841383338 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e2806ac30 msgr2=0x7f5e2806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.337+0000 7f5e4b200640 1 --2- 172.21.15.102:0/841383338 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e2806ac30 0x7f5e2806d0f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c009950 tx=0x7f5e3c008040 comp rx=0 tx=0).stop 2024-03-23T03:47:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.337+0000 7f5e4b200640 1 -- 172.21.15.102:0/841383338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e4c147880 msgr2=0x7f5e4c147ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.337+0000 7f5e4b200640 1 --2- 172.21.15.102:0/841383338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e4c147880 0x7f5e4c147ce0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5e34005fd0 tx=0x7f5e3400e530 comp rx=0 tx=0).stop 2024-03-23T03:47:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.337+0000 7f5e4b200640 1 -- 172.21.15.102:0/841383338 shutdown_connections 2024-03-23T03:47:09.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.337+0000 7f5e4b200640 1 --2- 172.21.15.102:0/841383338 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e2806ac30 0x7f5e2806d0f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.337+0000 7f5e4b200640 1 --2- 172.21.15.102:0/841383338 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e4c0c7de0 0x7f5e4c0c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.337+0000 7f5e4b200640 1 --2- 172.21.15.102:0/841383338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e4c147880 0x7f5e4c147ce0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.337+0000 7f5e4b200640 1 --2- 172.21.15.102:0/841383338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e2c004d00 0x7f5e4c0b6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.337+0000 7f5e4b200640 1 -- 172.21.15.102:0/841383338 >> 172.21.15.102:0/841383338 conn(0x7f5e4c06b070 msgr2=0x7f5e4c1603f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.338+0000 7f5e4b200640 1 -- 172.21.15.102:0/841383338 shutdown_connections 2024-03-23T03:47:09.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.338+0000 7f5e4b200640 1 -- 172.21.15.102:0/841383338 wait complete. 2024-03-23T03:47:09.356 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:47:09.356 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:47:09.410 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:09.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.619+0000 7fa550c00640 1 Processor -- start 2024-03-23T03:47:09.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.620+0000 7fa550c00640 1 -- start start 2024-03-23T03:47:09.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.621+0000 7fa550c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa54c0b37f0 0x7fa54c0cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:09.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.621+0000 7fa550c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa54c0ccec0 0x7fa54c06c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:09.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.621+0000 7fa550c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54c06da70 con 0x7fa54c14aa40 2024-03-23T03:47:09.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.621+0000 7fa550c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa54c06d7d0 con 0x7fa54c0ccec0 2024-03-23T03:47:09.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.621+0000 7fa550c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c0b1d00 con 0x7fa54c0b37f0 2024-03-23T03:47:09.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.622+0000 7fa549e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa54c14aa40 0x7fa54c0cc7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46744/0 (socket says 172.21.15.102:46744) 2024-03-23T03:47:09.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.622+0000 7fa549e00640 1 -- 172.21.15.102:0/1829677241 learned_addr learned my addr 172.21.15.102:0/1829677241 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:09.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.622+0000 7fa541400640 1 --2- 172.21.15.102:0/1829677241 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa54c0b37f0 0x7fa54c0cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:09.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.622+0000 7fa54a800640 1 --2- 172.21.15.102:0/1829677241 >> v2:172.21.15.173:3300/0 conn(0x7fa54c0ccec0 0x7fa54c06c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:09.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.622+0000 7fa549400640 1 -- 172.21.15.102:0/1829677241 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313808865 0 0) 0x7fa54c06da70 con 0x7fa54c14aa40 2024-03-23T03:47:09.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.622+0000 7fa549400640 1 -- 172.21.15.102:0/1829677241 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa530003520 con 0x7fa54c14aa40 2024-03-23T03:47:09.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.622+0000 7fa549400640 1 -- 172.21.15.102:0/1829677241 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 142749741 0 0) 0x7fa530003520 con 0x7fa54c14aa40 2024-03-23T03:47:09.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.622+0000 7fa549400640 1 -- 172.21.15.102:0/1829677241 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa54c0b37f0 msgr2=0x7fa54c0cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:09.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.622+0000 7fa549400640 1 --2- 172.21.15.102:0/1829677241 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa54c0b37f0 0x7fa54c0cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.623+0000 7fa549400640 1 -- 172.21.15.102:0/1829677241 >> v2:172.21.15.173:3300/0 conn(0x7fa54c0ccec0 msgr2=0x7fa54c06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:09.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.623+0000 7fa549400640 1 --2- 172.21.15.102:0/1829677241 >> v2:172.21.15.173:3300/0 conn(0x7fa54c0ccec0 0x7fa54c06c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.623+0000 7fa549400640 1 -- 172.21.15.102:0/1829677241 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54c0b1fe0 con 0x7fa54c14aa40 2024-03-23T03:47:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.623+0000 7fa549400640 1 -- 172.21.15.102:0/1829677241 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fa53c001670 con 0x7fa54c14aa40 2024-03-23T03:47:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.623+0000 7fa549400640 1 -- 172.21.15.102:0/1829677241 >> v1:172.21.15.102:6789/0 conn(0x7fa54c14aa40 legacy=0x7fa54c0cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.623+0000 7fa549400640 1 --2- 172.21.15.102:0/1829677241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa530003970 0x7fa530003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.623+0000 7fa549400640 1 --2- 172.21.15.102:0/1829677241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5300047c0 0x7fa530004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.623+0000 7fa549400640 1 --2- 172.21.15.102:0/1829677241 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa54c0b37f0 0x7fa530005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.623+0000 7fa549400640 1 -- 172.21.15.102:0/1829677241 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c06d7d0 con 0x7fa5300047c0 2024-03-23T03:47:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.623+0000 7fa549400640 1 -- 172.21.15.102:0/1829677241 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c0b1d00 con 0x7fa530003970 2024-03-23T03:47:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.623+0000 7fa549400640 1 -- 172.21.15.102:0/1829677241 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa53000b980 con 0x7fa54c0b37f0 2024-03-23T03:47:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.623+0000 7fa541400640 1 --2- 172.21.15.102:0/1829677241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5300047c0 0x7fa530004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.623+0000 7fa549e00640 1 --2- 172.21.15.102:0/1829677241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa530003970 0x7fa530003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.623+0000 7fa54a800640 1 --2- 172.21.15.102:0/1829677241 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa54c0b37f0 0x7fa530005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:09.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.623+0000 7fa541400640 1 -- 172.21.15.102:0/1829677241 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa54c0b37f0 msgr2=0x7fa530005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:09.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.624+0000 7fa541400640 1 --2- 172.21.15.102:0/1829677241 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa54c0b37f0 0x7fa530005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.624+0000 7fa541400640 1 -- 172.21.15.102:0/1829677241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa530003970 msgr2=0x7fa530003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:09.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.624+0000 7fa541400640 1 --2- 172.21.15.102:0/1829677241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa530003970 0x7fa530003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.624+0000 7fa541400640 1 -- 172.21.15.102:0/1829677241 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa530003520 con 0x7fa5300047c0 2024-03-23T03:47:09.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.624+0000 7fa54a800640 1 --2- 172.21.15.102:0/1829677241 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa54c0b37f0 0x7fa530005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:09.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.625+0000 7fa541400640 1 --2- 172.21.15.102:0/1829677241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5300047c0 0x7fa530004c40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa534009af0 tx=0x7fa5300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc5bc244072e578 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:09.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.626+0000 7fa549400640 1 -- 172.21.15.102:0/1829677241 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53402c6b0 con 0x7fa5300047c0 2024-03-23T03:47:09.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.626+0000 7fa549400640 1 -- 172.21.15.102:0/1829677241 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53402c810 con 0x7fa5300047c0 2024-03-23T03:47:09.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.626+0000 7fa549400640 1 -- 172.21.15.102:0/1829677241 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53402cba0 con 0x7fa5300047c0 2024-03-23T03:47:09.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.626+0000 7fa550c00640 1 -- 172.21.15.102:0/1829677241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5300047c0 msgr2=0x7fa530004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:09.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.626+0000 7fa550c00640 1 --2- 172.21.15.102:0/1829677241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5300047c0 0x7fa530004c40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa534009af0 tx=0x7fa5300057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.626+0000 7fa54a800640 1 -- 172.21.15.102:0/1829677241 reap_dead start 2024-03-23T03:47:09.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.626+0000 7fa550c00640 1 -- 172.21.15.102:0/1829677241 shutdown_connections 2024-03-23T03:47:09.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.626+0000 7fa550c00640 1 -- 172.21.15.102:0/1829677241 >> 172.21.15.102:0/1829677241 conn(0x7fa54c06a3d0 msgr2=0x7fa54c0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:09.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.627+0000 7fa550c00640 1 -- 172.21.15.102:0/1829677241 shutdown_connections 2024-03-23T03:47:09.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.627+0000 7fa550c00640 1 -- 172.21.15.102:0/1829677241 wait complete. 2024-03-23T03:47:09.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.628+0000 7fa550c00640 1 Processor -- start 2024-03-23T03:47:09.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.629+0000 7fa550c00640 1 -- start start 2024-03-23T03:47:09.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.629+0000 7fa550c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5300047c0 0x7fa54c140250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:09.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.629+0000 7fa550c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa54c0af230 0x7fa54c0af690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:09.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.629+0000 7fa550c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa54c142e20 0x7fa54c1432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:09.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.629+0000 7fa550c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c13e3a0 con 0x7fa54c142e20 2024-03-23T03:47:09.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.629+0000 7fa550c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c0afbd0 con 0x7fa5300047c0 2024-03-23T03:47:09.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.629+0000 7fa550c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c140790 con 0x7fa54c0af230 2024-03-23T03:47:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.630+0000 7fa54a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa54c142e20 0x7fa54c1432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.630+0000 7fa54a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa54c142e20 0x7fa54c1432a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53708/0 (socket says 172.21.15.102:53708) 2024-03-23T03:47:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.630+0000 7fa54a800640 1 -- 172.21.15.102:0/1428275333 learned_addr learned my addr 172.21.15.102:0/1428275333 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.630+0000 7fa541400640 1 --2- 172.21.15.102:0/1428275333 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa54c0af230 0x7fa54c0af690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.630+0000 7fa549e00640 1 --2- 172.21.15.102:0/1428275333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5300047c0 0x7fa54c140250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.630+0000 7fa54a800640 1 -- 172.21.15.102:0/1428275333 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa54c0af230 msgr2=0x7fa54c0af690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.630+0000 7fa54a800640 1 --2- 172.21.15.102:0/1428275333 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa54c0af230 0x7fa54c0af690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.630+0000 7fa54a800640 1 -- 172.21.15.102:0/1428275333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5300047c0 msgr2=0x7fa54c140250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.630+0000 7fa54a800640 1 --2- 172.21.15.102:0/1428275333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5300047c0 0x7fa54c140250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.630+0000 7fa54a800640 1 -- 172.21.15.102:0/1428275333 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54c143b50 con 0x7fa54c142e20 2024-03-23T03:47:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.631+0000 7fa54a800640 1 --2- 172.21.15.102:0/1428275333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa54c142e20 0x7fa54c1432a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0adae0 tx=0x7fa54400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:09.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.631+0000 7fa542a00640 1 -- 172.21.15.102:0/1428275333 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa544016620 con 0x7fa54c142e20 2024-03-23T03:47:09.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.631+0000 7fa542a00640 1 -- 172.21.15.102:0/1428275333 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa544016780 con 0x7fa54c142e20 2024-03-23T03:47:09.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.631+0000 7fa542a00640 1 -- 172.21.15.102:0/1428275333 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa544016b10 con 0x7fa54c142e20 2024-03-23T03:47:09.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.631+0000 7fa550c00640 1 -- 172.21.15.102:0/1428275333 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa54c143e30 con 0x7fa54c142e20 2024-03-23T03:47:09.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.631+0000 7fa550c00640 1 -- 172.21.15.102:0/1428275333 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54c1443f0 con 0x7fa54c142e20 2024-03-23T03:47:09.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.632+0000 7fa550c00640 1 -- 172.21.15.102:0/1428275333 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa54c13e4e0 con 0x7fa54c142e20 2024-03-23T03:47:09.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.633+0000 7fa542a00640 1 -- 172.21.15.102:0/1428275333 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa544016c70 con 0x7fa54c142e20 2024-03-23T03:47:09.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.634+0000 7fa542a00640 1 --2- 172.21.15.102:0/1428275333 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa52806ac30 0x7fa52806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:09.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.634+0000 7fa542a00640 1 -- 172.21.15.102:0/1428275333 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa54408a3b0 con 0x7fa54c142e20 2024-03-23T03:47:09.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.635+0000 7fa549e00640 1 --2- 172.21.15.102:0/1428275333 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa52806ac30 0x7fa52806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:09.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.635+0000 7fa549e00640 1 --2- 172.21.15.102:0/1428275333 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa52806ac30 0x7fa52806d0f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa53c0059d0 tx=0x7fa53c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:09.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.639+0000 7fa542a00640 1 -- 172.21.15.102:0/1428275333 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa54c13e4e0 con 0x7fa54c142e20 2024-03-23T03:47:09.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.834+0000 7fa550c00640 1 -- 172.21.15.102:0/1428275333 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa54c13e4e0 con 0x7fa54c142e20 2024-03-23T03:47:09.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.836+0000 7fa542a00640 1 -- 172.21.15.102:0/1428275333 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17926 (secure 0 0 0) 0x7fa54c13e4e0 con 0x7fa54c142e20 2024-03-23T03:47:09.838 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:09.838 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:58.410551+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:47:09.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.839+0000 7fa550c00640 1 -- 172.21.15.102:0/1428275333 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa52806ac30 msgr2=0x7fa52806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:09.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.839+0000 7fa550c00640 1 --2- 172.21.15.102:0/1428275333 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa52806ac30 0x7fa52806d0f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa53c0059d0 tx=0x7fa53c009040 comp rx=0 tx=0).stop 2024-03-23T03:47:09.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.839+0000 7fa550c00640 1 -- 172.21.15.102:0/1428275333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa54c142e20 msgr2=0x7fa54c1432a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:09.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.839+0000 7fa550c00640 1 --2- 172.21.15.102:0/1428275333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa54c142e20 0x7fa54c1432a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa54c0adae0 tx=0x7fa54400f520 comp rx=0 tx=0).stop 2024-03-23T03:47:09.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.839+0000 7fa550c00640 1 -- 172.21.15.102:0/1428275333 shutdown_connections 2024-03-23T03:47:09.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.839+0000 7fa550c00640 1 --2- 172.21.15.102:0/1428275333 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa52806ac30 0x7fa52806d0f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.839+0000 7fa550c00640 1 --2- 172.21.15.102:0/1428275333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa54c142e20 0x7fa54c1432a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.839+0000 7fa550c00640 1 --2- 172.21.15.102:0/1428275333 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa54c0af230 0x7fa54c0af690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.839+0000 7fa550c00640 1 --2- 172.21.15.102:0/1428275333 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5300047c0 0x7fa54c140250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:09.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.839+0000 7fa550c00640 1 -- 172.21.15.102:0/1428275333 >> 172.21.15.102:0/1428275333 conn(0x7fa54c06a3d0 msgr2=0x7fa54c149440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:09.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.839+0000 7fa550c00640 1 -- 172.21.15.102:0/1428275333 shutdown_connections 2024-03-23T03:47:09.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:09.839+0000 7fa550c00640 1 -- 172.21.15.102:0/1428275333 wait complete. 2024-03-23T03:47:09.858 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:46:52.158650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:46:52.914275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-23T03:46:55.983754+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-23T03:47:09.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:47:10.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.064+0000 7f7433200640 1 Processor -- start 2024-03-23T03:47:10.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.064+0000 7f7433200640 1 -- start start 2024-03-23T03:47:10.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.066+0000 7f7433200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f742c0add30 0x7f742c0ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.066+0000 7f7433200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f742c06ce90 0x7f742c06d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.066+0000 7f7433200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f742c0cf250 con 0x7f742c0ce930 2024-03-23T03:47:10.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.066+0000 7f7433200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f742c0ae1c0 con 0x7f742c0add30 2024-03-23T03:47:10.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.066+0000 7f7433200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742c06c760 con 0x7f742c06ce90 2024-03-23T03:47:10.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.066+0000 7f742b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f742c0add30 0x7f742c0ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.066+0000 7f742be00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f742c0ce930 0x7f742c0cd3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46758/0 (socket says 172.21.15.102:46758) 2024-03-23T03:47:10.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.066+0000 7f742be00640 1 -- 172.21.15.102:0/2018026366 learned_addr learned my addr 172.21.15.102:0/2018026366 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.067+0000 7f7431200640 1 --2- 172.21.15.102:0/2018026366 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f742c06ce90 0x7f742c06d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.067+0000 7f742b400640 1 -- 172.21.15.102:0/2018026366 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f742c06ce90 msgr2=0x7f742c06d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.067+0000 7f742b400640 1 --2- 172.21.15.102:0/2018026366 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f742c06ce90 0x7f742c06d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.067+0000 7f742aa00640 1 -- 172.21.15.102:0/2018026366 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581708979 0 0) 0x7f742c0cf250 con 0x7f742c0ce930 2024-03-23T03:47:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.067+0000 7f742b400640 1 -- 172.21.15.102:0/2018026366 >> v1:172.21.15.102:6789/0 conn(0x7f742c0ce930 legacy=0x7f742c0cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.067+0000 7f742b400640 1 -- 172.21.15.102:0/2018026366 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742c0cdba0 con 0x7f742c0add30 2024-03-23T03:47:10.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.068+0000 7f742b400640 1 --2- 172.21.15.102:0/2018026366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f742c0add30 0x7f742c0ced10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f74140097a0 tx=0x7f741402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f2f90299a89856 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:10.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.068+0000 7f742aa00640 1 -- 172.21.15.102:0/2018026366 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414031830 con 0x7f742c0add30 2024-03-23T03:47:10.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.068+0000 7f742aa00640 1 -- 172.21.15.102:0/2018026366 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7414031990 con 0x7f742c0add30 2024-03-23T03:47:10.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.068+0000 7f742aa00640 1 -- 172.21.15.102:0/2018026366 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7414031d20 con 0x7f742c0add30 2024-03-23T03:47:10.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.069+0000 7f7433200640 1 -- 172.21.15.102:0/2018026366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f742c0add30 msgr2=0x7f742c0ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:10.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.069+0000 7f7433200640 1 --2- 172.21.15.102:0/2018026366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f742c0add30 0x7f742c0ced10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f74140097a0 tx=0x7f741402b8d0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.069+0000 7f7433200640 1 -- 172.21.15.102:0/2018026366 shutdown_connections 2024-03-23T03:47:10.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.069+0000 7f7433200640 1 --2- 172.21.15.102:0/2018026366 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f742c06ce90 0x7f742c06d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.069+0000 7f7433200640 1 --2- 172.21.15.102:0/2018026366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f742c0add30 0x7f742c0ced10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.069+0000 7f7433200640 1 -- 172.21.15.102:0/2018026366 >> 172.21.15.102:0/2018026366 conn(0x7f742c06a0b0 msgr2=0x7f742c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:10.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.069+0000 7f7433200640 1 -- 172.21.15.102:0/2018026366 shutdown_connections 2024-03-23T03:47:10.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.069+0000 7f7433200640 1 -- 172.21.15.102:0/2018026366 wait complete. 2024-03-23T03:47:10.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.071+0000 7f7433200640 1 Processor -- start 2024-03-23T03:47:10.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.071+0000 7f7433200640 1 -- start start 2024-03-23T03:47:10.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.072+0000 7f7433200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f742c06ce90 0x7f742c1ef950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.072+0000 7f7433200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f742c0add30 0x7f742c151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.072+0000 7f7433200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f742c0ce930 0x7f742c157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.072+0000 7f7433200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742c151400 con 0x7f742c06ce90 2024-03-23T03:47:10.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.072+0000 7f7433200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742c1efe90 con 0x7f742c0add30 2024-03-23T03:47:10.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.072+0000 7f7433200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742c151dd0 con 0x7f742c0ce930 2024-03-23T03:47:10.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.072+0000 7f742be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f742c06ce90 0x7f742c1ef950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.072+0000 7f742be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f742c06ce90 0x7f742c1ef950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53722/0 (socket says 172.21.15.102:53722) 2024-03-23T03:47:10.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.072+0000 7f742be00640 1 -- 172.21.15.102:0/1983091253 learned_addr learned my addr 172.21.15.102:0/1983091253 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:10.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.073+0000 7f742b400640 1 --2- 172.21.15.102:0/1983091253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f742c0add30 0x7f742c151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.073+0000 7f7431200640 1 --2- 172.21.15.102:0/1983091253 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f742c0ce930 0x7f742c157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.073+0000 7f742be00640 1 -- 172.21.15.102:0/1983091253 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f742c0ce930 msgr2=0x7f742c157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:10.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.073+0000 7f742be00640 1 --2- 172.21.15.102:0/1983091253 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f742c0ce930 0x7f742c157320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.073+0000 7f742be00640 1 -- 172.21.15.102:0/1983091253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f742c0add30 msgr2=0x7f742c151890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:10.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.073+0000 7f742be00640 1 --2- 172.21.15.102:0/1983091253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f742c0add30 0x7f742c151890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.073+0000 7f742be00640 1 -- 172.21.15.102:0/1983091253 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7414009400 con 0x7f742c06ce90 2024-03-23T03:47:10.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.073+0000 7f742b400640 1 --2- 172.21.15.102:0/1983091253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f742c0add30 0x7f742c151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:10.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.073+0000 7f742be00640 1 --2- 172.21.15.102:0/1983091253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f742c06ce90 0x7f742c1ef950 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f741c00c430 tx=0x7f741c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:10.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.073+0000 7f7423e00640 1 -- 172.21.15.102:0/1983091253 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c0074e0 con 0x7f742c06ce90 2024-03-23T03:47:10.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.073+0000 7f7423e00640 1 -- 172.21.15.102:0/1983091253 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741c007660 con 0x7f742c06ce90 2024-03-23T03:47:10.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.073+0000 7f7423e00640 1 -- 172.21.15.102:0/1983091253 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c004a20 con 0x7f742c06ce90 2024-03-23T03:47:10.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.073+0000 7f7433200640 1 -- 172.21.15.102:0/1983091253 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742c1507f0 con 0x7f742c06ce90 2024-03-23T03:47:10.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.074+0000 7f7433200640 1 -- 172.21.15.102:0/1983091253 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f742c150db0 con 0x7f742c06ce90 2024-03-23T03:47:10.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.075+0000 7f7433200640 1 -- 172.21.15.102:0/1983091253 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f742c06a9f0 con 0x7f742c06ce90 2024-03-23T03:47:10.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.075+0000 7f7423e00640 1 -- 172.21.15.102:0/1983091253 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f741c004b80 con 0x7f742c06ce90 2024-03-23T03:47:10.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.076+0000 7f7423e00640 1 --2- 172.21.15.102:0/1983091253 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f740806ac00 0x7f740806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.076+0000 7f7423e00640 1 -- 172.21.15.102:0/1983091253 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f741c089520 con 0x7f742c06ce90 2024-03-23T03:47:10.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.077+0000 7f742b400640 1 --2- 172.21.15.102:0/1983091253 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f740806ac00 0x7f740806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.077+0000 7f742b400640 1 --2- 172.21.15.102:0/1983091253 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f740806ac00 0x7f740806d0c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7414036000 tx=0x7f7414008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:10.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.081+0000 7f7423e00640 1 -- 172.21.15.102:0/1983091253 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f741c054370 con 0x7f742c06ce90 2024-03-23T03:47:10.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.275+0000 7f7433200640 1 -- 172.21.15.102:0/1983091253 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f742c073390 con 0x7f742c06ce90 2024-03-23T03:47:10.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.275+0000 7f7423e00640 1 -- 172.21.15.102:0/1983091253 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f741c057990 con 0x7f742c06ce90 2024-03-23T03:47:10.276 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:47:10.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.278+0000 7f7433200640 1 -- 172.21.15.102:0/1983091253 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f740806ac00 msgr2=0x7f740806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:10.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.278+0000 7f7433200640 1 --2- 172.21.15.102:0/1983091253 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f740806ac00 0x7f740806d0c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7414036000 tx=0x7f7414008030 comp rx=0 tx=0).stop 2024-03-23T03:47:10.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.278+0000 7f7433200640 1 -- 172.21.15.102:0/1983091253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f742c06ce90 msgr2=0x7f742c1ef950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:10.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.278+0000 7f7433200640 1 --2- 172.21.15.102:0/1983091253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f742c06ce90 0x7f742c1ef950 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f741c00c430 tx=0x7f741c00c8f0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.278+0000 7f7433200640 1 -- 172.21.15.102:0/1983091253 shutdown_connections 2024-03-23T03:47:10.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.278+0000 7f7433200640 1 --2- 172.21.15.102:0/1983091253 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f740806ac00 0x7f740806d0c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.278+0000 7f7433200640 1 --2- 172.21.15.102:0/1983091253 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f742c0ce930 0x7f742c157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.278+0000 7f7433200640 1 --2- 172.21.15.102:0/1983091253 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f742c0add30 0x7f742c151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.278+0000 7f7433200640 1 --2- 172.21.15.102:0/1983091253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f742c06ce90 0x7f742c1ef950 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.278+0000 7f7433200640 1 -- 172.21.15.102:0/1983091253 >> 172.21.15.102:0/1983091253 conn(0x7f742c06a0b0 msgr2=0x7f742c0b0b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:10.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.279+0000 7f7433200640 1 -- 172.21.15.102:0/1983091253 shutdown_connections 2024-03-23T03:47:10.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.279+0000 7f7433200640 1 -- 172.21.15.102:0/1983091253 wait complete. 2024-03-23T03:47:10.298 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:47:10.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.494+0000 7fc22b800640 1 Processor -- start 2024-03-23T03:47:10.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.495+0000 7fc22b800640 1 -- start start 2024-03-23T03:47:10.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.496+0000 7fc22b800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2240cd740 0x7fc2240b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.496+0000 7fc22b800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc2240d0d40 0x7fc2240cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.496+0000 7fc22b800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc22406b0a0 0x7fc22406b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.496+0000 7fc22b800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2240d20c0 con 0x7fc2240cd740 2024-03-23T03:47:10.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.496+0000 7fc22b800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2240ce060 con 0x7fc2240d0d40 2024-03-23T03:47:10.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.496+0000 7fc22b800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2240d14b0 con 0x7fc22406b0a0 2024-03-23T03:47:10.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.496+0000 7fc228e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2240cd740 0x7fc2240b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.497+0000 7fc228e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2240cd740 0x7fc2240b84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53728/0 (socket says 172.21.15.102:53728) 2024-03-23T03:47:10.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.497+0000 7fc228e00640 1 -- 172.21.15.102:0/418271801 learned_addr learned my addr 172.21.15.102:0/418271801 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:10.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.497+0000 7fc223e00640 1 --2- 172.21.15.102:0/418271801 >> v2:172.21.15.173:3300/0 conn(0x7fc2240d0d40 0x7fc2240cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.497+0000 7fc229800640 1 --2- 172.21.15.102:0/418271801 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc22406b0a0 0x7fc22406b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.497+0000 7fc228e00640 1 -- 172.21.15.102:0/418271801 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc22406b0a0 msgr2=0x7fc22406b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:10.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.497+0000 7fc228e00640 1 --2- 172.21.15.102:0/418271801 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc22406b0a0 0x7fc22406b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.497+0000 7fc228e00640 1 -- 172.21.15.102:0/418271801 >> v2:172.21.15.173:3300/0 conn(0x7fc2240d0d40 msgr2=0x7fc2240cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:10.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.497+0000 7fc228e00640 1 --2- 172.21.15.102:0/418271801 >> v2:172.21.15.173:3300/0 conn(0x7fc2240d0d40 0x7fc2240cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.497+0000 7fc228e00640 1 -- 172.21.15.102:0/418271801 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2240d1790 con 0x7fc2240cd740 2024-03-23T03:47:10.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.498+0000 7fc228e00640 1 --2- 172.21.15.102:0/418271801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2240cd740 0x7fc2240b84b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc2240b8050 tx=0x7fc21402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675b45ec952bee20 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:10.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.498+0000 7fc223400640 1 -- 172.21.15.102:0/418271801 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc214031870 con 0x7fc2240cd740 2024-03-23T03:47:10.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.498+0000 7fc223400640 1 -- 172.21.15.102:0/418271801 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2140319d0 con 0x7fc2240cd740 2024-03-23T03:47:10.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.498+0000 7fc223400640 1 -- 172.21.15.102:0/418271801 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc214031d60 con 0x7fc2240cd740 2024-03-23T03:47:10.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.499+0000 7fc22b800640 1 -- 172.21.15.102:0/418271801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2240cd740 msgr2=0x7fc2240b84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:10.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.499+0000 7fc22b800640 1 --2- 172.21.15.102:0/418271801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2240cd740 0x7fc2240b84b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc2240b8050 tx=0x7fc21402b910 comp rx=0 tx=0).stop 2024-03-23T03:47:10.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.499+0000 7fc22b800640 1 -- 172.21.15.102:0/418271801 shutdown_connections 2024-03-23T03:47:10.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.499+0000 7fc22b800640 1 --2- 172.21.15.102:0/418271801 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc22406b0a0 0x7fc22406b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.499+0000 7fc22b800640 1 --2- 172.21.15.102:0/418271801 >> v2:172.21.15.173:3300/0 conn(0x7fc2240d0d40 0x7fc2240cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.499+0000 7fc22b800640 1 --2- 172.21.15.102:0/418271801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2240cd740 0x7fc2240b84b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.499+0000 7fc22b800640 1 -- 172.21.15.102:0/418271801 >> 172.21.15.102:0/418271801 conn(0x7fc2240cf5d0 msgr2=0x7fc2240cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:10.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.499+0000 7fc22b800640 1 -- 172.21.15.102:0/418271801 shutdown_connections 2024-03-23T03:47:10.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.499+0000 7fc22b800640 1 -- 172.21.15.102:0/418271801 wait complete. 2024-03-23T03:47:10.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.502+0000 7fc22b800640 1 Processor -- start 2024-03-23T03:47:10.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.502+0000 7fc22b800640 1 -- start start 2024-03-23T03:47:10.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.503+0000 7fc22b800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc22406b0a0 0x7fc2240cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.503+0000 7fc22b800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2240cd740 0x7fc2240c9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.503+0000 7fc22b800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2240d0d40 0x7fc2241f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.503+0000 7fc22b800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2240ca500 con 0x7fc2240d0d40 2024-03-23T03:47:10.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.503+0000 7fc22b800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2240ca7e0 con 0x7fc2240cd740 2024-03-23T03:47:10.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.503+0000 7fc22b800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2240caac0 con 0x7fc22406b0a0 2024-03-23T03:47:10.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.503+0000 7fc229800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2240d0d40 0x7fc2241f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.503+0000 7fc229800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2240d0d40 0x7fc2241f20e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53730/0 (socket says 172.21.15.102:53730) 2024-03-23T03:47:10.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.503+0000 7fc229800640 1 -- 172.21.15.102:0/2537628194 learned_addr learned my addr 172.21.15.102:0/2537628194 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:10.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.504+0000 7fc228e00640 1 --2- 172.21.15.102:0/2537628194 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc22406b0a0 0x7fc2240cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.504+0000 7fc223e00640 1 --2- 172.21.15.102:0/2537628194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2240cd740 0x7fc2240c9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.504+0000 7fc229800640 1 -- 172.21.15.102:0/2537628194 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc22406b0a0 msgr2=0x7fc2240cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:10.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.504+0000 7fc229800640 1 --2- 172.21.15.102:0/2537628194 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc22406b0a0 0x7fc2240cb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.504+0000 7fc229800640 1 -- 172.21.15.102:0/2537628194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2240cd740 msgr2=0x7fc2240c9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:10.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.504+0000 7fc229800640 1 --2- 172.21.15.102:0/2537628194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2240cd740 0x7fc2240c9a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.504+0000 7fc229800640 1 -- 172.21.15.102:0/2537628194 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2140094d0 con 0x7fc2240d0d40 2024-03-23T03:47:10.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.504+0000 7fc229800640 1 --2- 172.21.15.102:0/2537628194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2240d0d40 0x7fc2241f20e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc21c00ccf0 tx=0x7fc21c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:10.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.504+0000 7fc220c00640 1 -- 172.21.15.102:0/2537628194 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21c003b90 con 0x7fc2240d0d40 2024-03-23T03:47:10.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.504+0000 7fc220c00640 1 -- 172.21.15.102:0/2537628194 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21c003cf0 con 0x7fc2240d0d40 2024-03-23T03:47:10.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.504+0000 7fc220c00640 1 -- 172.21.15.102:0/2537628194 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21c004240 con 0x7fc2240d0d40 2024-03-23T03:47:10.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.504+0000 7fc22b800640 1 -- 172.21.15.102:0/2537628194 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2240cada0 con 0x7fc2240d0d40 2024-03-23T03:47:10.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.505+0000 7fc22b800640 1 -- 172.21.15.102:0/2537628194 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2240aa4b0 con 0x7fc2240d0d40 2024-03-23T03:47:10.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.506+0000 7fc22b800640 1 -- 172.21.15.102:0/2537628194 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2240cb2c0 con 0x7fc2240d0d40 2024-03-23T03:47:10.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.507+0000 7fc220c00640 1 -- 172.21.15.102:0/2537628194 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc21c0043a0 con 0x7fc2240d0d40 2024-03-23T03:47:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.507+0000 7fc220c00640 1 --2- 172.21.15.102:0/2537628194 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc20006ac00 0x7fc20006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.508+0000 7fc220c00640 1 -- 172.21.15.102:0/2537628194 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc21c012070 con 0x7fc2240d0d40 2024-03-23T03:47:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.508+0000 7fc228e00640 1 --2- 172.21.15.102:0/2537628194 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc20006ac00 0x7fc20006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.509+0000 7fc228e00640 1 --2- 172.21.15.102:0/2537628194 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc20006ac00 0x7fc20006d0c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc214036000 tx=0x7fc21403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:10.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.512+0000 7fc220c00640 1 -- 172.21.15.102:0/2537628194 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc2240cb2c0 con 0x7fc2240d0d40 2024-03-23T03:47:10.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.709+0000 7fc22b800640 1 -- 172.21.15.102:0/2537628194 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc2240cb2c0 con 0x7fc2240d0d40 2024-03-23T03:47:10.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.710+0000 7fc220c00640 1 -- 172.21.15.102:0/2537628194 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc2240cb2c0 con 0x7fc2240d0d40 2024-03-23T03:47:10.711 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:10.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.713+0000 7fc22b800640 1 -- 172.21.15.102:0/2537628194 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc20006ac00 msgr2=0x7fc20006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:10.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.713+0000 7fc22b800640 1 --2- 172.21.15.102:0/2537628194 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc20006ac00 0x7fc20006d0c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc214036000 tx=0x7fc21403c040 comp rx=0 tx=0).stop 2024-03-23T03:47:10.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.713+0000 7fc22b800640 1 -- 172.21.15.102:0/2537628194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2240d0d40 msgr2=0x7fc2241f20e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:10.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.713+0000 7fc22b800640 1 --2- 172.21.15.102:0/2537628194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2240d0d40 0x7fc2241f20e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc21c00ccf0 tx=0x7fc21c00a660 comp rx=0 tx=0).stop 2024-03-23T03:47:10.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.714+0000 7fc22b800640 1 -- 172.21.15.102:0/2537628194 shutdown_connections 2024-03-23T03:47:10.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.714+0000 7fc22b800640 1 --2- 172.21.15.102:0/2537628194 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc20006ac00 0x7fc20006d0c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.714+0000 7fc22b800640 1 --2- 172.21.15.102:0/2537628194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2240d0d40 0x7fc2241f20e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.714+0000 7fc22b800640 1 --2- 172.21.15.102:0/2537628194 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2240cd740 0x7fc2240c9a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.714+0000 7fc22b800640 1 --2- 172.21.15.102:0/2537628194 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc22406b0a0 0x7fc2240cb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.714+0000 7fc22b800640 1 -- 172.21.15.102:0/2537628194 >> 172.21.15.102:0/2537628194 conn(0x7fc2240cf5d0 msgr2=0x7fc22406b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:10.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.714+0000 7fc22b800640 1 -- 172.21.15.102:0/2537628194 shutdown_connections 2024-03-23T03:47:10.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.714+0000 7fc22b800640 1 -- 172.21.15.102:0/2537628194 wait complete. 2024-03-23T03:47:10.734 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:47:10.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.936+0000 7fd59c400640 1 Processor -- start 2024-03-23T03:47:10.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.936+0000 7fd59c400640 1 -- start start 2024-03-23T03:47:10.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.937+0000 7fd59c400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd5941456a0 0x7fd5940ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.937+0000 7fd59c400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd594145c20 0x7fd5940cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.937+0000 7fd59c400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59406d390 con 0x7fd594070cd0 2024-03-23T03:47:10.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.937+0000 7fd59c400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5940ad890 con 0x7fd5941456a0 2024-03-23T03:47:10.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.937+0000 7fd59c400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd594145a80 con 0x7fd594145c20 2024-03-23T03:47:10.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.938+0000 7fd599a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd5941456a0 0x7fd5940ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.938+0000 7fd599a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd5941456a0 0x7fd5940ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53736/0 (socket says 172.21.15.102:53736) 2024-03-23T03:47:10.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.938+0000 7fd599a00640 1 -- 172.21.15.102:0/1419241676 learned_addr learned my addr 172.21.15.102:0/1419241676 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:10.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.938+0000 7fd599000640 1 --1- 172.21.15.102:0/1419241676 >> v1:172.21.15.102:6789/0 conn(0x7fd594070cd0 0x7fd5940ab8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46772/0 (socket says 172.21.15.102:46772) 2024-03-23T03:47:10.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.938+0000 7fd59a400640 1 --2- 172.21.15.102:0/1419241676 >> v2:172.21.15.173:3300/0 conn(0x7fd594145c20 0x7fd5940cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.938+0000 7fd599a00640 1 -- 172.21.15.102:0/1419241676 >> v2:172.21.15.173:3300/0 conn(0x7fd594145c20 msgr2=0x7fd5940cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:10.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.938+0000 7fd599a00640 1 --2- 172.21.15.102:0/1419241676 >> v2:172.21.15.173:3300/0 conn(0x7fd594145c20 0x7fd5940cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.938+0000 7fd599a00640 1 -- 172.21.15.102:0/1419241676 >> v1:172.21.15.102:6789/0 conn(0x7fd594070cd0 legacy=0x7fd5940ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:10.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.938+0000 7fd599a00640 1 -- 172.21.15.102:0/1419241676 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59406d670 con 0x7fd5941456a0 2024-03-23T03:47:10.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.938+0000 7fd59a400640 1 --2- 172.21.15.102:0/1419241676 >> v2:172.21.15.173:3300/0 conn(0x7fd594145c20 0x7fd5940cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:10.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.939+0000 7fd599a00640 1 --2- 172.21.15.102:0/1419241676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5941456a0 0x7fd5940ce930 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd5940ce100 tx=0x7fd58802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e547a0c7edfe8f22 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:10.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.940+0000 7fd593e00640 1 -- 172.21.15.102:0/1419241676 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd588030860 con 0x7fd5941456a0 2024-03-23T03:47:10.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.940+0000 7fd593e00640 1 -- 172.21.15.102:0/1419241676 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5880309c0 con 0x7fd5941456a0 2024-03-23T03:47:10.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.940+0000 7fd593e00640 1 -- 172.21.15.102:0/1419241676 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd588030d50 con 0x7fd5941456a0 2024-03-23T03:47:10.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.940+0000 7fd59c400640 1 -- 172.21.15.102:0/1419241676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5941456a0 msgr2=0x7fd5940ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:10.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.940+0000 7fd59c400640 1 --2- 172.21.15.102:0/1419241676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5941456a0 0x7fd5940ce930 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd5940ce100 tx=0x7fd58802b9a0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.940+0000 7fd59c400640 1 -- 172.21.15.102:0/1419241676 shutdown_connections 2024-03-23T03:47:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.940+0000 7fd59c400640 1 --2- 172.21.15.102:0/1419241676 >> v2:172.21.15.173:3300/0 conn(0x7fd594145c20 0x7fd5940cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.940+0000 7fd59c400640 1 --2- 172.21.15.102:0/1419241676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5941456a0 0x7fd5940ce930 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.940+0000 7fd59c400640 1 -- 172.21.15.102:0/1419241676 >> 172.21.15.102:0/1419241676 conn(0x7fd59406a0b0 msgr2=0x7fd59406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.940+0000 7fd59c400640 1 -- 172.21.15.102:0/1419241676 shutdown_connections 2024-03-23T03:47:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.940+0000 7fd59c400640 1 -- 172.21.15.102:0/1419241676 wait complete. 2024-03-23T03:47:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.942+0000 7fd59c400640 1 Processor -- start 2024-03-23T03:47:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.942+0000 7fd59c400640 1 -- start start 2024-03-23T03:47:10.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.943+0000 7fd59c400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd594070cd0 0x7fd59414f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.943+0000 7fd59c400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5941456a0 0x7fd5941512e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.943+0000 7fd59c400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd594145c20 0x7fd59415e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.943+0000 7fd59c400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd594150e80 con 0x7fd594145c20 2024-03-23T03:47:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.943+0000 7fd59c400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd594150200 con 0x7fd594070cd0 2024-03-23T03:47:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.943+0000 7fd59c400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5941504e0 con 0x7fd5941456a0 2024-03-23T03:47:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.943+0000 7fd59a400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd594145c20 0x7fd59415e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.943+0000 7fd59a400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd594145c20 0x7fd59415e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53748/0 (socket says 172.21.15.102:53748) 2024-03-23T03:47:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.943+0000 7fd59a400640 1 -- 172.21.15.102:0/3387358468 learned_addr learned my addr 172.21.15.102:0/3387358468 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.944+0000 7fd599a00640 1 --2- 172.21.15.102:0/3387358468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd594070cd0 0x7fd59414f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.944+0000 7fd599000640 1 --2- 172.21.15.102:0/3387358468 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5941456a0 0x7fd5941512e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.944+0000 7fd59a400640 1 -- 172.21.15.102:0/3387358468 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5941456a0 msgr2=0x7fd5941512e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.944+0000 7fd59a400640 1 --2- 172.21.15.102:0/3387358468 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5941456a0 0x7fd5941512e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.944+0000 7fd59a400640 1 -- 172.21.15.102:0/3387358468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd594070cd0 msgr2=0x7fd59414f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.944+0000 7fd59a400640 1 --2- 172.21.15.102:0/3387358468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd594070cd0 0x7fd59414f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.944+0000 7fd59a400640 1 -- 172.21.15.102:0/3387358468 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5880094d0 con 0x7fd594145c20 2024-03-23T03:47:10.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.944+0000 7fd599000640 1 --2- 172.21.15.102:0/3387358468 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5941456a0 0x7fd5941512e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:10.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.944+0000 7fd59a400640 1 --2- 172.21.15.102:0/3387358468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd594145c20 0x7fd59415e9c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd58c00cd50 tx=0x7fd58c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:10.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.944+0000 7fd591600640 1 -- 172.21.15.102:0/3387358468 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c00f680 con 0x7fd594145c20 2024-03-23T03:47:10.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.945+0000 7fd591600640 1 -- 172.21.15.102:0/3387358468 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58c00f800 con 0x7fd594145c20 2024-03-23T03:47:10.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.945+0000 7fd591600640 1 -- 172.21.15.102:0/3387358468 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c00fbb0 con 0x7fd594145c20 2024-03-23T03:47:10.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.945+0000 7fd59c400640 1 -- 172.21.15.102:0/3387358468 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5941507c0 con 0x7fd594145c20 2024-03-23T03:47:10.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.945+0000 7fd59c400640 1 -- 172.21.15.102:0/3387358468 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59414fed0 con 0x7fd594145c20 2024-03-23T03:47:10.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.946+0000 7fd59c400640 1 -- 172.21.15.102:0/3387358468 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5940b1a00 con 0x7fd594145c20 2024-03-23T03:47:10.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.947+0000 7fd591600640 1 -- 172.21.15.102:0/3387358468 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd58c00b040 con 0x7fd594145c20 2024-03-23T03:47:10.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.947+0000 7fd591600640 1 --2- 172.21.15.102:0/3387358468 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd57406ac00 0x7fd57406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:10.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.948+0000 7fd591600640 1 -- 172.21.15.102:0/3387358468 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd58c08a0c0 con 0x7fd594145c20 2024-03-23T03:47:10.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.948+0000 7fd599a00640 1 --2- 172.21.15.102:0/3387358468 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd57406ac00 0x7fd57406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:10.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.948+0000 7fd599a00640 1 --2- 172.21.15.102:0/3387358468 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd57406ac00 0x7fd57406d0c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd588036000 tx=0x7fd588008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:10.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:10.952+0000 7fd591600640 1 -- 172.21.15.102:0/3387358468 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd58c054f10 con 0x7fd594145c20 2024-03-23T03:47:11.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.145+0000 7fd59c400640 1 -- 172.21.15.102:0/3387358468 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd5940acb50 con 0x7fd594145c20 2024-03-23T03:47:11.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.146+0000 7fd591600640 1 -- 172.21.15.102:0/3387358468 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd58c01d0a0 con 0x7fd594145c20 2024-03-23T03:47:11.147 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:47:11.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.148+0000 7fd59c400640 1 -- 172.21.15.102:0/3387358468 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd57406ac00 msgr2=0x7fd57406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.148+0000 7fd59c400640 1 --2- 172.21.15.102:0/3387358468 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd57406ac00 0x7fd57406d0c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd588036000 tx=0x7fd588008030 comp rx=0 tx=0).stop 2024-03-23T03:47:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.148+0000 7fd59c400640 1 -- 172.21.15.102:0/3387358468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd594145c20 msgr2=0x7fd59415e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.149+0000 7fd59c400640 1 --2- 172.21.15.102:0/3387358468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd594145c20 0x7fd59415e9c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd58c00cd50 tx=0x7fd58c00a650 comp rx=0 tx=0).stop 2024-03-23T03:47:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.149+0000 7fd59c400640 1 -- 172.21.15.102:0/3387358468 shutdown_connections 2024-03-23T03:47:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.149+0000 7fd59c400640 1 --2- 172.21.15.102:0/3387358468 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd57406ac00 0x7fd57406d0c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.149+0000 7fd59c400640 1 --2- 172.21.15.102:0/3387358468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd594145c20 0x7fd59415e9c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.149+0000 7fd59c400640 1 --2- 172.21.15.102:0/3387358468 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5941456a0 0x7fd5941512e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.149+0000 7fd59c400640 1 --2- 172.21.15.102:0/3387358468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd594070cd0 0x7fd59414f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.149+0000 7fd59c400640 1 -- 172.21.15.102:0/3387358468 >> 172.21.15.102:0/3387358468 conn(0x7fd59406a0b0 msgr2=0x7fd5940ab8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.149+0000 7fd59c400640 1 -- 172.21.15.102:0/3387358468 shutdown_connections 2024-03-23T03:47:11.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.149+0000 7fd59c400640 1 -- 172.21.15.102:0/3387358468 wait complete. 2024-03-23T03:47:11.169 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-23T03:47:11.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.381+0000 7f3923000640 1 Processor -- start 2024-03-23T03:47:11.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.382+0000 7f3923000640 1 -- start start 2024-03-23T03:47:11.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.383+0000 7f3923000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f391c0ce930 0x7f391c0add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:11.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.383+0000 7f3923000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f391c0cd3e0 0x7f391c0ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:11.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.383+0000 7f3923000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f391c06ce90 0x7f391c06d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:11.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.383+0000 7f3923000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f391c06c760 con 0x7f391c0cd3e0 2024-03-23T03:47:11.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.383+0000 7f3923000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f391c0b42d0 con 0x7f391c06ce90 2024-03-23T03:47:11.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.383+0000 7f3923000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f391c0cf250 con 0x7f391c0ce930 2024-03-23T03:47:11.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.383+0000 7f3913400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f391c0cd3e0 0x7f391c0ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:11.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.383+0000 7f3913400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f391c0cd3e0 0x7f391c0ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53750/0 (socket says 172.21.15.102:53750) 2024-03-23T03:47:11.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.383+0000 7f3913400640 1 -- 172.21.15.102:0/2522650089 learned_addr learned my addr 172.21.15.102:0/2522650089 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:11.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.384+0000 7f3921000640 1 --2- 172.21.15.102:0/2522650089 >> v2:172.21.15.173:3300/0 conn(0x7f391c06ce90 0x7f391c06d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:11.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.384+0000 7f391be00640 1 --2- 172.21.15.102:0/2522650089 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f391c0ce930 0x7f391c0add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:11.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.384+0000 7f3913400640 1 -- 172.21.15.102:0/2522650089 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f391c0ce930 msgr2=0x7f391c0add30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:11.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.384+0000 7f3913400640 1 --2- 172.21.15.102:0/2522650089 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f391c0ce930 0x7f391c0add30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.384+0000 7f3913400640 1 -- 172.21.15.102:0/2522650089 >> v2:172.21.15.173:3300/0 conn(0x7f391c06ce90 msgr2=0x7f391c06d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:11.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.384+0000 7f3913400640 1 --2- 172.21.15.102:0/2522650089 >> v2:172.21.15.173:3300/0 conn(0x7f391c06ce90 0x7f391c06d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.384+0000 7f3913400640 1 -- 172.21.15.102:0/2522650089 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f391c0cdba0 con 0x7f391c0cd3e0 2024-03-23T03:47:11.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.384+0000 7f391be00640 1 --2- 172.21.15.102:0/2522650089 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f391c0ce930 0x7f391c0add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:11.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.385+0000 7f3913400640 1 --2- 172.21.15.102:0/2522650089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f391c0cd3e0 0x7f391c0ced10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f391c0ccc70 tx=0x7f390402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265a9708f179ee3e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:11.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.385+0000 7f391b400640 1 -- 172.21.15.102:0/2522650089 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3904031860 con 0x7f391c0cd3e0 2024-03-23T03:47:11.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.385+0000 7f391b400640 1 -- 172.21.15.102:0/2522650089 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39040319c0 con 0x7f391c0cd3e0 2024-03-23T03:47:11.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.385+0000 7f391b400640 1 -- 172.21.15.102:0/2522650089 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3904031d50 con 0x7f391c0cd3e0 2024-03-23T03:47:11.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.386+0000 7f3923000640 1 -- 172.21.15.102:0/2522650089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f391c0cd3e0 msgr2=0x7f391c0ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:11.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.386+0000 7f3923000640 1 --2- 172.21.15.102:0/2522650089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f391c0cd3e0 0x7f391c0ced10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f391c0ccc70 tx=0x7f390402b900 comp rx=0 tx=0).stop 2024-03-23T03:47:11.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.386+0000 7f3923000640 1 -- 172.21.15.102:0/2522650089 shutdown_connections 2024-03-23T03:47:11.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.386+0000 7f3923000640 1 --2- 172.21.15.102:0/2522650089 >> v2:172.21.15.173:3300/0 conn(0x7f391c06ce90 0x7f391c06d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.386+0000 7f3923000640 1 --2- 172.21.15.102:0/2522650089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f391c0cd3e0 0x7f391c0ced10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.386+0000 7f3923000640 1 --2- 172.21.15.102:0/2522650089 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f391c0ce930 0x7f391c0add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.386+0000 7f3923000640 1 -- 172.21.15.102:0/2522650089 >> 172.21.15.102:0/2522650089 conn(0x7f391c06a0b0 msgr2=0x7f391c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:11.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.386+0000 7f3923000640 1 -- 172.21.15.102:0/2522650089 shutdown_connections 2024-03-23T03:47:11.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.386+0000 7f3923000640 1 -- 172.21.15.102:0/2522650089 wait complete. 2024-03-23T03:47:11.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.388+0000 7f3923000640 1 Processor -- start 2024-03-23T03:47:11.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.388+0000 7f3923000640 1 -- start start 2024-03-23T03:47:11.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.389+0000 7f3923000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f391c06ce90 0x7f391c1f6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:11.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.389+0000 7f3923000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f391c0cd3e0 0x7f391c14e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:11.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.389+0000 7f3923000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f391c0ce930 0x7f391c14f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:11.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.389+0000 7f3923000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391c1f6850 con 0x7f391c0cd3e0 2024-03-23T03:47:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.389+0000 7f3923000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f391c0b36d0 con 0x7f391c0ce930 2024-03-23T03:47:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.389+0000 7f3923000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f391c151cf0 con 0x7f391c06ce90 2024-03-23T03:47:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.389+0000 7f3913400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f391c0cd3e0 0x7f391c14e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.389+0000 7f3913400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f391c0cd3e0 0x7f391c14e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53762/0 (socket says 172.21.15.102:53762) 2024-03-23T03:47:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.389+0000 7f3913400640 1 -- 172.21.15.102:0/310788469 learned_addr learned my addr 172.21.15.102:0/310788469 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.390+0000 7f3921000640 1 --2- 172.21.15.102:0/310788469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f391c0ce930 0x7f391c14f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.390+0000 7f391be00640 1 --2- 172.21.15.102:0/310788469 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f391c06ce90 0x7f391c1f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.390+0000 7f3913400640 1 -- 172.21.15.102:0/310788469 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f391c06ce90 msgr2=0x7f391c1f6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.390+0000 7f3913400640 1 --2- 172.21.15.102:0/310788469 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f391c06ce90 0x7f391c1f6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.390+0000 7f3913400640 1 -- 172.21.15.102:0/310788469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f391c0ce930 msgr2=0x7f391c14f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.390+0000 7f3913400640 1 --2- 172.21.15.102:0/310788469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f391c0ce930 0x7f391c14f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.390+0000 7f3913400640 1 -- 172.21.15.102:0/310788469 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39040094d0 con 0x7f391c0cd3e0 2024-03-23T03:47:11.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.390+0000 7f3913400640 1 --2- 172.21.15.102:0/310788469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f391c0cd3e0 0x7f391c14e250 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f390402b940 tx=0x7f39040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:11.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.390+0000 7f3918c00640 1 -- 172.21.15.102:0/310788469 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f390403ad30 con 0x7f391c0cd3e0 2024-03-23T03:47:11.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.391+0000 7f3918c00640 1 -- 172.21.15.102:0/310788469 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39040443f0 con 0x7f391c0cd3e0 2024-03-23T03:47:11.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.391+0000 7f3918c00640 1 -- 172.21.15.102:0/310788469 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39040447a0 con 0x7f391c0cd3e0 2024-03-23T03:47:11.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.391+0000 7f3923000640 1 -- 172.21.15.102:0/310788469 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f391c0ae420 con 0x7f391c0cd3e0 2024-03-23T03:47:11.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.391+0000 7f3923000640 1 -- 172.21.15.102:0/310788469 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f391c0ae9e0 con 0x7f391c0cd3e0 2024-03-23T03:47:11.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.392+0000 7f3923000640 1 -- 172.21.15.102:0/310788469 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f391c073390 con 0x7f391c0cd3e0 2024-03-23T03:47:11.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.393+0000 7f3918c00640 1 -- 172.21.15.102:0/310788469 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3904044900 con 0x7f391c0cd3e0 2024-03-23T03:47:11.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.394+0000 7f3918c00640 1 --2- 172.21.15.102:0/310788469 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f38f806ac00 0x7f38f806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:11.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.394+0000 7f3918c00640 1 -- 172.21.15.102:0/310788469 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f39040abde0 con 0x7f391c0cd3e0 2024-03-23T03:47:11.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.394+0000 7f391be00640 1 --2- 172.21.15.102:0/310788469 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f38f806ac00 0x7f38f806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:11.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.395+0000 7f391be00640 1 --2- 172.21.15.102:0/310788469 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f38f806ac00 0x7f38f806d0c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f390c00a910 tx=0x7f390c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:11.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.398+0000 7f3918c00640 1 -- 172.21.15.102:0/310788469 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3904076de0 con 0x7f391c0cd3e0 2024-03-23T03:47:11.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.592+0000 7f3923000640 1 -- 172.21.15.102:0/310788469 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f391c0b1f60 con 0x7f391c0cd3e0 2024-03-23T03:47:11.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.592+0000 7f3918c00640 1 -- 172.21.15.102:0/310788469 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3904035070 con 0x7f391c0cd3e0 2024-03-23T03:47:11.593 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:11.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.595+0000 7f3923000640 1 -- 172.21.15.102:0/310788469 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f38f806ac00 msgr2=0x7f38f806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:11.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.595+0000 7f3923000640 1 --2- 172.21.15.102:0/310788469 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f38f806ac00 0x7f38f806d0c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f390c00a910 tx=0x7f390c008040 comp rx=0 tx=0).stop 2024-03-23T03:47:11.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.595+0000 7f3923000640 1 -- 172.21.15.102:0/310788469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f391c0cd3e0 msgr2=0x7f391c14e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:11.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.595+0000 7f3923000640 1 --2- 172.21.15.102:0/310788469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f391c0cd3e0 0x7f391c14e250 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f390402b940 tx=0x7f39040046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.595+0000 7f3923000640 1 -- 172.21.15.102:0/310788469 shutdown_connections 2024-03-23T03:47:11.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.595+0000 7f3923000640 1 --2- 172.21.15.102:0/310788469 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f38f806ac00 0x7f38f806d0c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.595+0000 7f3923000640 1 --2- 172.21.15.102:0/310788469 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f391c0ce930 0x7f391c14f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.595+0000 7f3923000640 1 --2- 172.21.15.102:0/310788469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f391c0cd3e0 0x7f391c14e250 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.595+0000 7f3923000640 1 --2- 172.21.15.102:0/310788469 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f391c06ce90 0x7f391c1f6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.595+0000 7f3923000640 1 -- 172.21.15.102:0/310788469 >> 172.21.15.102:0/310788469 conn(0x7f391c06a0b0 msgr2=0x7f391c0b16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:11.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.595+0000 7f3923000640 1 -- 172.21.15.102:0/310788469 shutdown_connections 2024-03-23T03:47:11.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.595+0000 7f3923000640 1 -- 172.21.15.102:0/310788469 wait complete. 2024-03-23T03:47:11.616 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:47:11.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.829+0000 7f25abe00640 1 Processor -- start 2024-03-23T03:47:11.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.829+0000 7f25abe00640 1 -- start start 2024-03-23T03:47:11.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.831+0000 7f25abe00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f25a40cb660 0x7f25a40cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:11.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.831+0000 7f25abe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f25a406c2b0 0x7f25a406c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:11.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.831+0000 7f25abe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25a415ae60 con 0x7f25a40ce930 2024-03-23T03:47:11.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.831+0000 7f25abe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a415bae0 con 0x7f25a406c2b0 2024-03-23T03:47:11.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.831+0000 7f25abe00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25a415b140 con 0x7f25a40cb660 2024-03-23T03:47:11.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.831+0000 7f25a9400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f25a40ce930 0x7f25a40ced10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:46784/0 (socket says 172.21.15.102:46784) 2024-03-23T03:47:11.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.831+0000 7f25a9400640 1 -- 172.21.15.102:0/2988078000 learned_addr learned my addr 172.21.15.102:0/2988078000 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:11.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.831+0000 7f25a9e00640 1 --2- 172.21.15.102:0/2988078000 >> v2:172.21.15.102:3300/0 conn(0x7f25a406c2b0 0x7f25a406c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.831+0000 7f25a8a00640 1 --2- 172.21.15.102:0/2988078000 >> v2:172.21.15.173:3300/0 conn(0x7f25a40cb660 0x7f25a40cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.831+0000 7f25a3e00640 1 -- 172.21.15.102:0/2988078000 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504703582 0 0) 0x7f25a415ae60 con 0x7f25a40ce930 2024-03-23T03:47:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.832+0000 7f25a3e00640 1 -- 172.21.15.102:0/2988078000 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f258c003520 con 0x7f25a40ce930 2024-03-23T03:47:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.832+0000 7f25a9e00640 1 -- 172.21.15.102:0/2988078000 >> v2:172.21.15.173:3300/0 conn(0x7f25a40cb660 msgr2=0x7f25a40cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.832+0000 7f25a9e00640 1 --2- 172.21.15.102:0/2988078000 >> v2:172.21.15.173:3300/0 conn(0x7f25a40cb660 0x7f25a40cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.832+0000 7f25a9e00640 1 -- 172.21.15.102:0/2988078000 >> v1:172.21.15.102:6789/0 conn(0x7f25a40ce930 legacy=0x7f25a40ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:11.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.832+0000 7f25a9e00640 1 -- 172.21.15.102:0/2988078000 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a415b420 con 0x7f25a406c2b0 2024-03-23T03:47:11.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.833+0000 7f25a9e00640 1 --2- 172.21.15.102:0/2988078000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25a406c2b0 0x7f25a406c720 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f25a406fe60 tx=0x7f259c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b5e33bd3db55b6a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:11.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.833+0000 7f25a3e00640 1 -- 172.21.15.102:0/2988078000 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c02cc80 con 0x7f25a406c2b0 2024-03-23T03:47:11.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.833+0000 7f25a3e00640 1 -- 172.21.15.102:0/2988078000 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c02cde0 con 0x7f25a406c2b0 2024-03-23T03:47:11.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.833+0000 7f25a3e00640 1 -- 172.21.15.102:0/2988078000 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c034640 con 0x7f25a406c2b0 2024-03-23T03:47:11.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.834+0000 7f25abe00640 1 -- 172.21.15.102:0/2988078000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25a406c2b0 msgr2=0x7f25a406c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:11.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.834+0000 7f25abe00640 1 --2- 172.21.15.102:0/2988078000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25a406c2b0 0x7f25a406c720 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f25a406fe60 tx=0x7f259c02bed0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.834+0000 7f25abe00640 1 -- 172.21.15.102:0/2988078000 shutdown_connections 2024-03-23T03:47:11.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.834+0000 7f25abe00640 1 --2- 172.21.15.102:0/2988078000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25a406c2b0 0x7f25a406c720 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.834+0000 7f25abe00640 1 --2- 172.21.15.102:0/2988078000 >> v2:172.21.15.173:3300/0 conn(0x7f25a40cb660 0x7f25a40cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.834+0000 7f25abe00640 1 -- 172.21.15.102:0/2988078000 >> 172.21.15.102:0/2988078000 conn(0x7f25a406a0b0 msgr2=0x7f25a406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:11.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.834+0000 7f25abe00640 1 -- 172.21.15.102:0/2988078000 shutdown_connections 2024-03-23T03:47:11.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.834+0000 7f25abe00640 1 -- 172.21.15.102:0/2988078000 wait complete. 2024-03-23T03:47:11.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.835+0000 7f25abe00640 1 Processor -- start 2024-03-23T03:47:11.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.836+0000 7f25abe00640 1 -- start start 2024-03-23T03:47:11.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.836+0000 7f25abe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25a406c2b0 0x7f25a41f17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:11.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.836+0000 7f25abe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25a40cb660 0x7f25a4156fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:11.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.837+0000 7f25abe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a40ce930 0x7f25a4157510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:11.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.837+0000 7f25abe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a40b16c0 con 0x7f25a406c2b0 2024-03-23T03:47:11.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.837+0000 7f25abe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25a40ae630 con 0x7f25a40ce930 2024-03-23T03:47:11.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.837+0000 7f25abe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25a40ae910 con 0x7f25a40cb660 2024-03-23T03:47:11.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.837+0000 7f25a9400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25a406c2b0 0x7f25a41f17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:11.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.837+0000 7f25a9400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25a406c2b0 0x7f25a41f17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53772/0 (socket says 172.21.15.102:53772) 2024-03-23T03:47:11.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.837+0000 7f25a9400640 1 -- 172.21.15.102:0/1384382302 learned_addr learned my addr 172.21.15.102:0/1384382302 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:11.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.837+0000 7f25a9e00640 1 --2- 172.21.15.102:0/1384382302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a40ce930 0x7f25a4157510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:11.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.837+0000 7f25a9400640 1 -- 172.21.15.102:0/1384382302 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25a40cb660 msgr2=0x7f25a4156fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:11.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.837+0000 7f25a8a00640 1 --2- 172.21.15.102:0/1384382302 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25a40cb660 0x7f25a4156fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:11.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.837+0000 7f25a9400640 1 --2- 172.21.15.102:0/1384382302 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25a40cb660 0x7f25a4156fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.837+0000 7f25a9400640 1 -- 172.21.15.102:0/1384382302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a40ce930 msgr2=0x7f25a4157510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:11.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.837+0000 7f25a9400640 1 --2- 172.21.15.102:0/1384382302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a40ce930 0x7f25a4157510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:11.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.837+0000 7f25a9400640 1 -- 172.21.15.102:0/1384382302 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259c009a70 con 0x7f25a406c2b0 2024-03-23T03:47:11.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.838+0000 7f25a9400640 1 --2- 172.21.15.102:0/1384382302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25a406c2b0 0x7f25a41f17b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f259800c390 tx=0x7f259800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:11.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.838+0000 7f25a1600640 1 -- 172.21.15.102:0/1384382302 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2598005ce0 con 0x7f25a406c2b0 2024-03-23T03:47:11.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.838+0000 7f25a1600640 1 -- 172.21.15.102:0/1384382302 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2598005e40 con 0x7f25a406c2b0 2024-03-23T03:47:11.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.838+0000 7f25abe00640 1 -- 172.21.15.102:0/1384382302 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25a4155280 con 0x7f25a406c2b0 2024-03-23T03:47:11.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.838+0000 7f25a1600640 1 -- 172.21.15.102:0/1384382302 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25980072f0 con 0x7f25a406c2b0 2024-03-23T03:47:11.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.838+0000 7f25abe00640 1 -- 172.21.15.102:0/1384382302 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25a40aeed0 con 0x7f25a406c2b0 2024-03-23T03:47:11.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.839+0000 7f25abe00640 1 -- 172.21.15.102:0/1384382302 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25a4073390 con 0x7f25a406c2b0 2024-03-23T03:47:11.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.840+0000 7f25a1600640 1 -- 172.21.15.102:0/1384382302 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2598007450 con 0x7f25a406c2b0 2024-03-23T03:47:11.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.841+0000 7f25a1600640 1 --2- 172.21.15.102:0/1384382302 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f258406ac00 0x7f258406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:11.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.841+0000 7f25a1600640 1 -- 172.21.15.102:0/1384382302 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2598088c60 con 0x7f25a406c2b0 2024-03-23T03:47:11.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.841+0000 7f25a8a00640 1 --2- 172.21.15.102:0/1384382302 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f258406ac00 0x7f258406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:11.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.842+0000 7f25a8a00640 1 --2- 172.21.15.102:0/1384382302 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f258406ac00 0x7f258406d0c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2590006fd0 tx=0x7f2590008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:11.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:11.845+0000 7f25a1600640 1 -- 172.21.15.102:0/1384382302 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2598053b40 con 0x7f25a406c2b0 2024-03-23T03:47:12.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.048+0000 7f25abe00640 1 -- 172.21.15.102:0/1384382302 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f25a40af060 con 0x7f25a406c2b0 2024-03-23T03:47:12.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.049+0000 7f25a1600640 1 -- 172.21.15.102:0/1384382302 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v26) v1 ==== 80+0+0 (secure 0 0 0) 0x7f2598017020 con 0x7f25a406c2b0 2024-03-23T03:47:12.050 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:47:12.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.051+0000 7f25abe00640 1 -- 172.21.15.102:0/1384382302 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f258406ac00 msgr2=0x7f258406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:12.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.051+0000 7f25abe00640 1 --2- 172.21.15.102:0/1384382302 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f258406ac00 0x7f258406d0c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2590006fd0 tx=0x7f2590008040 comp rx=0 tx=0).stop 2024-03-23T03:47:12.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.051+0000 7f25abe00640 1 -- 172.21.15.102:0/1384382302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25a406c2b0 msgr2=0x7f25a41f17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:12.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.051+0000 7f25abe00640 1 --2- 172.21.15.102:0/1384382302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25a406c2b0 0x7f25a41f17b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f259800c390 tx=0x7f259800c850 comp rx=0 tx=0).stop 2024-03-23T03:47:12.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.051+0000 7f25abe00640 1 -- 172.21.15.102:0/1384382302 shutdown_connections 2024-03-23T03:47:12.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.051+0000 7f25abe00640 1 --2- 172.21.15.102:0/1384382302 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f258406ac00 0x7f258406d0c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.051+0000 7f25abe00640 1 --2- 172.21.15.102:0/1384382302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25a40ce930 0x7f25a4157510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.051+0000 7f25abe00640 1 --2- 172.21.15.102:0/1384382302 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25a40cb660 0x7f25a4156fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.051+0000 7f25abe00640 1 --2- 172.21.15.102:0/1384382302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25a406c2b0 0x7f25a41f17b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.051+0000 7f25abe00640 1 -- 172.21.15.102:0/1384382302 >> 172.21.15.102:0/1384382302 conn(0x7f25a406a0b0 msgr2=0x7f25a415a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:12.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.052+0000 7f25abe00640 1 -- 172.21.15.102:0/1384382302 shutdown_connections 2024-03-23T03:47:12.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.052+0000 7f25abe00640 1 -- 172.21.15.102:0/1384382302 wait complete. 2024-03-23T03:47:12.070 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi102.front.sepia.ceph.com... 2024-03-23T03:47:12.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:12.070 DEBUG:teuthology.orchestra.run.smithi102:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-23T03:47:12.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi102.stderr:ceph-fuse[43309]: fuse finished with error 0 and tester_r 0 2024-03-23T03:47:12.153 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:47:12.225 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:47:12.226 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:12.226 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-23T03:47:12.283 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:12.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.485+0000 7f3e7ec00640 1 Processor -- start 2024-03-23T03:47:12.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.485+0000 7f3e7ec00640 1 -- start start 2024-03-23T03:47:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.486+0000 7f3e7ec00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e78143930 0x7f3e780cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.486+0000 7f3e7ec00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3e780cdc50 0x7f3e78070020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.486+0000 7f3e7ec00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e7815b2b0 con 0x7f3e78143460 2024-03-23T03:47:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.486+0000 7f3e7ec00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e780cc6b0 con 0x7f3e780cdc50 2024-03-23T03:47:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.486+0000 7f3e7ec00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e780cc950 con 0x7f3e78143930 2024-03-23T03:47:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.487+0000 7f3e7e200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3e780cdc50 0x7f3e78070020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.487+0000 7f3e7e200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3e780cdc50 0x7f3e78070020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53778/0 (socket says 172.21.15.102:53778) 2024-03-23T03:47:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.487+0000 7f3e7e200640 1 -- 172.21.15.102:0/1736510913 learned_addr learned my addr 172.21.15.102:0/1736510913 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.487+0000 7f3e7ce00640 1 --2- 172.21.15.102:0/1736510913 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e78143930 0x7f3e780cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.487+0000 7f3e7e200640 1 -- 172.21.15.102:0/1736510913 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e78143930 msgr2=0x7f3e780cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:12.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.487+0000 7f3e7e200640 1 --2- 172.21.15.102:0/1736510913 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e78143930 0x7f3e780cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.487+0000 7f3e7e200640 1 -- 172.21.15.102:0/1736510913 >> v1:172.21.15.173:6789/0 conn(0x7f3e78143460 legacy=0x7f3e7815aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:12.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.487+0000 7f3e7e200640 1 -- 172.21.15.102:0/1736510913 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e780ccc30 con 0x7f3e780cdc50 2024-03-23T03:47:12.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.488+0000 7f3e7e200640 1 --2- 172.21.15.102:0/1736510913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e780cdc50 0x7f3e78070020 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3e70009e50 tx=0x7f3e7002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26721adc80bb43f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:12.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.489+0000 7f3e77e00640 1 -- 172.21.15.102:0/1736510913 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7002cb00 con 0x7f3e780cdc50 2024-03-23T03:47:12.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.489+0000 7f3e77e00640 1 -- 172.21.15.102:0/1736510913 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7002cc60 con 0x7f3e780cdc50 2024-03-23T03:47:12.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.489+0000 7f3e77e00640 1 -- 172.21.15.102:0/1736510913 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e700343f0 con 0x7f3e780cdc50 2024-03-23T03:47:12.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.489+0000 7f3e7ec00640 1 -- 172.21.15.102:0/1736510913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e780cdc50 msgr2=0x7f3e78070020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:12.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.489+0000 7f3e7ec00640 1 --2- 172.21.15.102:0/1736510913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e780cdc50 0x7f3e78070020 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3e70009e50 tx=0x7f3e7002bf60 comp rx=0 tx=0).stop 2024-03-23T03:47:12.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.489+0000 7f3e7ec00640 1 -- 172.21.15.102:0/1736510913 shutdown_connections 2024-03-23T03:47:12.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.489+0000 7f3e7ec00640 1 --2- 172.21.15.102:0/1736510913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e780cdc50 0x7f3e78070020 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.489+0000 7f3e7ec00640 1 --2- 172.21.15.102:0/1736510913 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e78143930 0x7f3e780cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.489+0000 7f3e7ec00640 1 -- 172.21.15.102:0/1736510913 >> 172.21.15.102:0/1736510913 conn(0x7f3e78065df0 msgr2=0x7f3e78069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.489+0000 7f3e7ec00640 1 -- 172.21.15.102:0/1736510913 shutdown_connections 2024-03-23T03:47:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.489+0000 7f3e7ec00640 1 -- 172.21.15.102:0/1736510913 wait complete. 2024-03-23T03:47:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.491+0000 7f3e7ec00640 1 Processor -- start 2024-03-23T03:47:12.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.491+0000 7f3e7ec00640 1 -- start start 2024-03-23T03:47:12.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.492+0000 7f3e7ec00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e780cdc50 0x7f3e781f18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:12.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.492+0000 7f3e7ec00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e78143460 0x7f3e7815cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:12.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.492+0000 7f3e7ec00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e78143930 0x7f3e7815d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:12.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.492+0000 7f3e7ec00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e7806d9a0 con 0x7f3e78143930 2024-03-23T03:47:12.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.492+0000 7f3e7ec00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e781f1e20 con 0x7f3e78143460 2024-03-23T03:47:12.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.492+0000 7f3e7ec00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e780adcd0 con 0x7f3e780cdc50 2024-03-23T03:47:12.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.492+0000 7f3e7e200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e78143930 0x7f3e7815d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:12.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.492+0000 7f3e7e200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e78143930 0x7f3e7815d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53794/0 (socket says 172.21.15.102:53794) 2024-03-23T03:47:12.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.492+0000 7f3e7e200640 1 -- 172.21.15.102:0/2961217571 learned_addr learned my addr 172.21.15.102:0/2961217571 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:12.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.493+0000 7f3e7ce00640 1 --2- 172.21.15.102:0/2961217571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e78143460 0x7f3e7815cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:12.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.493+0000 7f3e7d800640 1 --2- 172.21.15.102:0/2961217571 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e780cdc50 0x7f3e781f18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:12.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.493+0000 7f3e7e200640 1 -- 172.21.15.102:0/2961217571 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e780cdc50 msgr2=0x7f3e781f18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:12.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.493+0000 7f3e7e200640 1 --2- 172.21.15.102:0/2961217571 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e780cdc50 0x7f3e781f18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.493+0000 7f3e7e200640 1 -- 172.21.15.102:0/2961217571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e78143460 msgr2=0x7f3e7815cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:12.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.493+0000 7f3e7e200640 1 --2- 172.21.15.102:0/2961217571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e78143460 0x7f3e7815cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.493+0000 7f3e7e200640 1 -- 172.21.15.102:0/2961217571 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e70009b30 con 0x7f3e78143930 2024-03-23T03:47:12.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.493+0000 7f3e7ce00640 1 --2- 172.21.15.102:0/2961217571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e78143460 0x7f3e7815cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:12.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.493+0000 7f3e7d800640 1 --2- 172.21.15.102:0/2961217571 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e780cdc50 0x7f3e781f18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:12.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.493+0000 7f3e7e200640 1 --2- 172.21.15.102:0/2961217571 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e78143930 0x7f3e7815d170 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3e70009e50 tx=0x7f3e70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:12.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.493+0000 7f3e75600640 1 -- 172.21.15.102:0/2961217571 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e70009510 con 0x7f3e78143930 2024-03-23T03:47:12.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.493+0000 7f3e75600640 1 -- 172.21.15.102:0/2961217571 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7003e450 con 0x7f3e78143930 2024-03-23T03:47:12.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.493+0000 7f3e75600640 1 -- 172.21.15.102:0/2961217571 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7003e800 con 0x7f3e78143930 2024-03-23T03:47:12.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.493+0000 7f3e7ec00640 1 -- 172.21.15.102:0/2961217571 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e7815a3b0 con 0x7f3e78143930 2024-03-23T03:47:12.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.493+0000 7f3e7ec00640 1 -- 172.21.15.102:0/2961217571 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e7806d2e0 con 0x7f3e78143930 2024-03-23T03:47:12.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.494+0000 7f3e7ec00640 1 -- 172.21.15.102:0/2961217571 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e780adf50 con 0x7f3e78143930 2024-03-23T03:47:12.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.495+0000 7f3e75600640 1 -- 172.21.15.102:0/2961217571 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3e7003e960 con 0x7f3e78143930 2024-03-23T03:47:12.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.496+0000 7f3e75600640 1 --2- 172.21.15.102:0/2961217571 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e5806ac00 0x7f3e5806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:12.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.496+0000 7f3e75600640 1 -- 172.21.15.102:0/2961217571 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3e700ac410 con 0x7f3e78143930 2024-03-23T03:47:12.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.496+0000 7f3e7d800640 1 --2- 172.21.15.102:0/2961217571 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e5806ac00 0x7f3e5806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:12.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.497+0000 7f3e7d800640 1 --2- 172.21.15.102:0/2961217571 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e5806ac00 0x7f3e5806d0c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c003eb0 tx=0x7f3e6c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:12.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.501+0000 7f3e75600640 1 -- 172.21.15.102:0/2961217571 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3e700772f0 con 0x7f3e78143930 2024-03-23T03:47:12.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.693+0000 7f3e7ec00640 1 -- 172.21.15.102:0/2961217571 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e7806dae0 con 0x7f3e78143930 2024-03-23T03:47:12.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.695+0000 7f3e75600640 1 -- 172.21.15.102:0/2961217571 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f3e7806dae0 con 0x7f3e78143930 2024-03-23T03:47:12.696 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:12.696 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:58.410551+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:47:12.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.697+0000 7f3e7ec00640 1 -- 172.21.15.102:0/2961217571 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e5806ac00 msgr2=0x7f3e5806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.697+0000 7f3e7ec00640 1 --2- 172.21.15.102:0/2961217571 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e5806ac00 0x7f3e5806d0c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c003eb0 tx=0x7f3e6c009040 comp rx=0 tx=0).stop 2024-03-23T03:47:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.697+0000 7f3e7ec00640 1 -- 172.21.15.102:0/2961217571 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e78143930 msgr2=0x7f3e7815d170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.697+0000 7f3e7ec00640 1 --2- 172.21.15.102:0/2961217571 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e78143930 0x7f3e7815d170 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3e70009e50 tx=0x7f3e70004a40 comp rx=0 tx=0).stop 2024-03-23T03:47:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.698+0000 7f3e7ec00640 1 -- 172.21.15.102:0/2961217571 shutdown_connections 2024-03-23T03:47:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.698+0000 7f3e7ec00640 1 --2- 172.21.15.102:0/2961217571 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e5806ac00 0x7f3e5806d0c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.698+0000 7f3e7ec00640 1 --2- 172.21.15.102:0/2961217571 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e78143930 0x7f3e7815d170 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.698+0000 7f3e7ec00640 1 --2- 172.21.15.102:0/2961217571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e78143460 0x7f3e7815cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.698+0000 7f3e7ec00640 1 --2- 172.21.15.102:0/2961217571 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e780cdc50 0x7f3e781f18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.698+0000 7f3e7ec00640 1 -- 172.21.15.102:0/2961217571 >> 172.21.15.102:0/2961217571 conn(0x7f3e78065df0 msgr2=0x7f3e780ae0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.698+0000 7f3e7ec00640 1 -- 172.21.15.102:0/2961217571 shutdown_connections 2024-03-23T03:47:12.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.698+0000 7f3e7ec00640 1 -- 172.21.15.102:0/2961217571 wait complete. 2024-03-23T03:47:12.716 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:12.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.925+0000 7efc80000640 1 Processor -- start 2024-03-23T03:47:12.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.925+0000 7efc80000640 1 -- start start 2024-03-23T03:47:12.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.926+0000 7efc80000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7efc780cbab0 0x7efc780b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:12.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.926+0000 7efc80000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc780b16a0 con 0x7efc7806d390 2024-03-23T03:47:12.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.926+0000 7efc80000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc780cbfb0 con 0x7efc780adbc0 2024-03-23T03:47:12.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.926+0000 7efc80000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc780cc290 con 0x7efc780cbab0 2024-03-23T03:47:12.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.927+0000 7efc7e000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7efc7806d390 0x7efc780b2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41072/0 (socket says 172.21.15.102:41072) 2024-03-23T03:47:12.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.927+0000 7efc7e000640 1 -- 172.21.15.102:0/2175073550 learned_addr learned my addr 172.21.15.102:0/2175073550 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:12.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.927+0000 7efc7cc00640 1 --2- 172.21.15.102:0/2175073550 >> v2:172.21.15.173:3300/0 conn(0x7efc780cbab0 0x7efc780b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:12.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.927+0000 7efc77e00640 1 -- 172.21.15.102:0/2175073550 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491217830 0 0) 0x7efc780b16a0 con 0x7efc7806d390 2024-03-23T03:47:12.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.927+0000 7efc77e00640 1 -- 172.21.15.102:0/2175073550 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc60005730 con 0x7efc7806d390 2024-03-23T03:47:12.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.928+0000 7efc77e00640 1 -- 172.21.15.102:0/2175073550 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 566415071 0 0) 0x7efc60005730 con 0x7efc7806d390 2024-03-23T03:47:12.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.928+0000 7efc77e00640 1 -- 172.21.15.102:0/2175073550 >> v2:172.21.15.173:3300/0 conn(0x7efc780cbab0 msgr2=0x7efc780b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:12.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.928+0000 7efc77e00640 1 --2- 172.21.15.102:0/2175073550 >> v2:172.21.15.173:3300/0 conn(0x7efc780cbab0 0x7efc780b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.928+0000 7efc77e00640 1 -- 172.21.15.102:0/2175073550 >> v1:172.21.15.173:6789/0 conn(0x7efc780adbc0 legacy=0x7efc780b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:12.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.928+0000 7efc77e00640 1 -- 172.21.15.102:0/2175073550 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc780b18f0 con 0x7efc7806d390 2024-03-23T03:47:12.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.928+0000 7efc77e00640 1 -- 172.21.15.102:0/2175073550 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7efc700019f0 con 0x7efc7806d390 2024-03-23T03:47:12.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.928+0000 7efc77e00640 1 -- 172.21.15.102:0/2175073550 >> v1:172.21.15.102:6789/0 conn(0x7efc7806d390 legacy=0x7efc780b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:12.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.928+0000 7efc77e00640 1 --2- 172.21.15.102:0/2175073550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc60005b80 0x7efc60005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:12.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.928+0000 7efc77e00640 1 --2- 172.21.15.102:0/2175073550 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc600069d0 0x7efc60006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:12.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.928+0000 7efc77e00640 1 --2- 172.21.15.102:0/2175073550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc60007390 0x7efc6000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:12.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.928+0000 7efc77e00640 1 -- 172.21.15.102:0/2175073550 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc780cc290 con 0x7efc60005b80 2024-03-23T03:47:12.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.928+0000 7efc77e00640 1 -- 172.21.15.102:0/2175073550 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc6000e330 con 0x7efc60007390 2024-03-23T03:47:12.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.928+0000 7efc77e00640 1 -- 172.21.15.102:0/2175073550 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc6000e4f0 con 0x7efc600069d0 2024-03-23T03:47:12.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.928+0000 7efc7d600640 1 --2- 172.21.15.102:0/2175073550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc60005b80 0x7efc60005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:12.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.928+0000 7efc7cc00640 1 --2- 172.21.15.102:0/2175073550 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc600069d0 0x7efc60006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:12.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.929+0000 7efc7e000640 1 --2- 172.21.15.102:0/2175073550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc60007390 0x7efc6000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:12.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.929+0000 7efc7d600640 1 -- 172.21.15.102:0/2175073550 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc600069d0 msgr2=0x7efc60006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:12.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.929+0000 7efc7d600640 1 --2- 172.21.15.102:0/2175073550 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc600069d0 0x7efc60006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.929+0000 7efc7d600640 1 -- 172.21.15.102:0/2175073550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc60007390 msgr2=0x7efc6000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:12.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.929+0000 7efc7d600640 1 --2- 172.21.15.102:0/2175073550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc60007390 0x7efc6000db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.929+0000 7efc7e000640 1 -- 172.21.15.102:0/2175073550 reap_dead start 2024-03-23T03:47:12.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.929+0000 7efc7d600640 1 -- 172.21.15.102:0/2175073550 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc60005730 con 0x7efc60005b80 2024-03-23T03:47:12.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.930+0000 7efc7d600640 1 --2- 172.21.15.102:0/2175073550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc60005b80 0x7efc60005fa0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efc6c001750 tx=0x7efc60007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73528eaf14e28a1f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:12.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.930+0000 7efc77e00640 1 -- 172.21.15.102:0/2175073550 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c0052d0 con 0x7efc60005b80 2024-03-23T03:47:12.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.930+0000 7efc77e00640 1 -- 172.21.15.102:0/2175073550 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6c02e050 con 0x7efc60005b80 2024-03-23T03:47:12.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.930+0000 7efc77e00640 1 -- 172.21.15.102:0/2175073550 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6c003830 con 0x7efc60005b80 2024-03-23T03:47:12.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.931+0000 7efc80000640 1 -- 172.21.15.102:0/2175073550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc60005b80 msgr2=0x7efc60005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:12.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.931+0000 7efc80000640 1 --2- 172.21.15.102:0/2175073550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc60005b80 0x7efc60005fa0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efc6c001750 tx=0x7efc60007a00 comp rx=0 tx=0).stop 2024-03-23T03:47:12.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.931+0000 7efc80000640 1 -- 172.21.15.102:0/2175073550 shutdown_connections 2024-03-23T03:47:12.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.931+0000 7efc80000640 1 --2- 172.21.15.102:0/2175073550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc60005b80 0x7efc60005fa0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.931+0000 7efc80000640 1 -- 172.21.15.102:0/2175073550 >> 172.21.15.102:0/2175073550 conn(0x7efc7806a3d0 msgr2=0x7efc780b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:12.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.931+0000 7efc80000640 1 -- 172.21.15.102:0/2175073550 shutdown_connections 2024-03-23T03:47:12.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.931+0000 7efc80000640 1 -- 172.21.15.102:0/2175073550 wait complete. 2024-03-23T03:47:12.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.933+0000 7efc80000640 1 Processor -- start 2024-03-23T03:47:12.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.933+0000 7efc80000640 1 -- start start 2024-03-23T03:47:12.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.934+0000 7efc80000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc60005b80 0x7efc781593d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:12.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.934+0000 7efc80000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc780ad410 0x7efc781f5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:12.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.934+0000 7efc80000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc781f3040 0x7efc780aef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:12.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.934+0000 7efc80000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc781f6340 con 0x7efc780ad410 2024-03-23T03:47:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.934+0000 7efc80000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc781f65c0 con 0x7efc781f3040 2024-03-23T03:47:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.934+0000 7efc80000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc781f68a0 con 0x7efc60005b80 2024-03-23T03:47:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.934+0000 7efc7cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc780ad410 0x7efc781f5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.934+0000 7efc7cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc780ad410 0x7efc781f5e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37004/0 (socket says 172.21.15.102:37004) 2024-03-23T03:47:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.934+0000 7efc7cc00640 1 -- 172.21.15.102:0/2076982210 learned_addr learned my addr 172.21.15.102:0/2076982210 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.934+0000 7efc7d600640 1 --2- 172.21.15.102:0/2076982210 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc60005b80 0x7efc781593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.935+0000 7efc7e000640 1 --2- 172.21.15.102:0/2076982210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc781f3040 0x7efc780aef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.935+0000 7efc7cc00640 1 -- 172.21.15.102:0/2076982210 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc60005b80 msgr2=0x7efc781593d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:12.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.935+0000 7efc7cc00640 1 --2- 172.21.15.102:0/2076982210 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc60005b80 0x7efc781593d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.935+0000 7efc7cc00640 1 -- 172.21.15.102:0/2076982210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc781f3040 msgr2=0x7efc780aef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:12.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.935+0000 7efc7cc00640 1 --2- 172.21.15.102:0/2076982210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc781f3040 0x7efc780aef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:12.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.935+0000 7efc7cc00640 1 -- 172.21.15.102:0/2076982210 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc781f6b80 con 0x7efc780ad410 2024-03-23T03:47:12.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.935+0000 7efc7cc00640 1 --2- 172.21.15.102:0/2076982210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc780ad410 0x7efc781f5e00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efc640114c0 tx=0x7efc64011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:12.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.935+0000 7efc75600640 1 -- 172.21.15.102:0/2076982210 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc64015870 con 0x7efc780ad410 2024-03-23T03:47:12.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.935+0000 7efc75600640 1 -- 172.21.15.102:0/2076982210 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc640159d0 con 0x7efc780ad410 2024-03-23T03:47:12.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.935+0000 7efc75600640 1 -- 172.21.15.102:0/2076982210 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc64015d40 con 0x7efc780ad410 2024-03-23T03:47:12.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.935+0000 7efc80000640 1 -- 172.21.15.102:0/2076982210 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc781f2b20 con 0x7efc780ad410 2024-03-23T03:47:12.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.936+0000 7efc80000640 1 -- 172.21.15.102:0/2076982210 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc781f34f0 con 0x7efc780ad410 2024-03-23T03:47:12.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.937+0000 7efc80000640 1 -- 172.21.15.102:0/2076982210 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc781f3770 con 0x7efc780ad410 2024-03-23T03:47:12.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.937+0000 7efc75600640 1 -- 172.21.15.102:0/2076982210 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7efc64013400 con 0x7efc780ad410 2024-03-23T03:47:12.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.938+0000 7efc75600640 1 --2- 172.21.15.102:0/2076982210 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efc5806ac30 0x7efc5806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:12.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.938+0000 7efc75600640 1 -- 172.21.15.102:0/2076982210 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7efc64089a10 con 0x7efc780ad410 2024-03-23T03:47:12.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.938+0000 7efc7d600640 1 --2- 172.21.15.102:0/2076982210 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efc5806ac30 0x7efc5806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:12.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.939+0000 7efc7d600640 1 --2- 172.21.15.102:0/2076982210 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efc5806ac30 0x7efc5806d0f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efc6c033000 tx=0x7efc6c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:12.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:12.942+0000 7efc75600640 1 -- 172.21.15.102:0/2076982210 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efc64054860 con 0x7efc780ad410 2024-03-23T03:47:13.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.135+0000 7efc80000640 1 -- 172.21.15.102:0/2076982210 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc7806ac20 con 0x7efc780ad410 2024-03-23T03:47:13.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.139+0000 7efc75600640 1 -- 172.21.15.102:0/2076982210 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17926 (secure 0 0 0) 0x7efc64057e80 con 0x7efc780ad410 2024-03-23T03:47:13.140 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:13.140 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:58.410551+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:47:13.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.141+0000 7efc80000640 1 -- 172.21.15.102:0/2076982210 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efc5806ac30 msgr2=0x7efc5806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:13.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.141+0000 7efc80000640 1 --2- 172.21.15.102:0/2076982210 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efc5806ac30 0x7efc5806d0f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7efc6c033000 tx=0x7efc6c009030 comp rx=0 tx=0).stop 2024-03-23T03:47:13.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.141+0000 7efc80000640 1 -- 172.21.15.102:0/2076982210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc780ad410 msgr2=0x7efc781f5e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:13.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.141+0000 7efc80000640 1 --2- 172.21.15.102:0/2076982210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc780ad410 0x7efc781f5e00 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7efc640114c0 tx=0x7efc64011980 comp rx=0 tx=0).stop 2024-03-23T03:47:13.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.141+0000 7efc80000640 1 -- 172.21.15.102:0/2076982210 shutdown_connections 2024-03-23T03:47:13.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.141+0000 7efc80000640 1 --2- 172.21.15.102:0/2076982210 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efc5806ac30 0x7efc5806d0f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:13.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.141+0000 7efc80000640 1 --2- 172.21.15.102:0/2076982210 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc781f3040 0x7efc780aef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:13.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.141+0000 7efc80000640 1 --2- 172.21.15.102:0/2076982210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc780ad410 0x7efc781f5e00 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:13.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.141+0000 7efc80000640 1 --2- 172.21.15.102:0/2076982210 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc60005b80 0x7efc781593d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:13.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.141+0000 7efc80000640 1 -- 172.21.15.102:0/2076982210 >> 172.21.15.102:0/2076982210 conn(0x7efc7806a3d0 msgr2=0x7efc780ae360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:13.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.142+0000 7efc80000640 1 -- 172.21.15.102:0/2076982210 shutdown_connections 2024-03-23T03:47:13.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.142+0000 7efc80000640 1 -- 172.21.15.102:0/2076982210 wait complete. 2024-03-23T03:47:13.161 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T03:47:13.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.374+0000 7f8319a00640 1 Processor -- start 2024-03-23T03:47:13.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.374+0000 7f8319a00640 1 -- start start 2024-03-23T03:47:13.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.375+0000 7f8319a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f83140b84b0 0x7f83140d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:13.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.375+0000 7f8319a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f83140b5aa0 0x7f83140b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:13.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.375+0000 7f8319a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83140d1280 con 0x7f83140cd740 2024-03-23T03:47:13.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.375+0000 7f8319a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83140d20c0 con 0x7f83140b5aa0 2024-03-23T03:47:13.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.375+0000 7f8319a00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83140b8940 con 0x7f83140b84b0 2024-03-23T03:47:13.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.376+0000 7f8312a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f83140cd740 0x7f83140cdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41088/0 (socket says 172.21.15.102:41088) 2024-03-23T03:47:13.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.376+0000 7f8312a00640 1 -- 172.21.15.102:0/1264178656 learned_addr learned my addr 172.21.15.102:0/1264178656 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:13.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.376+0000 7f8313400640 1 --2- 172.21.15.102:0/1264178656 >> v2:172.21.15.102:3300/0 conn(0x7f83140b5aa0 0x7f83140b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:13.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.376+0000 7f830a000640 1 --2- 172.21.15.102:0/1264178656 >> v2:172.21.15.173:3300/0 conn(0x7f83140b84b0 0x7f83140d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:13.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.376+0000 7f8313400640 1 -- 172.21.15.102:0/1264178656 >> v2:172.21.15.173:3300/0 conn(0x7f83140b84b0 msgr2=0x7f83140d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:13.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.376+0000 7f8313400640 1 --2- 172.21.15.102:0/1264178656 >> v2:172.21.15.173:3300/0 conn(0x7f83140b84b0 0x7f83140d0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:13.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.376+0000 7f8313400640 1 -- 172.21.15.102:0/1264178656 >> v1:172.21.15.102:6789/0 conn(0x7f83140cd740 legacy=0x7f83140cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:13.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.376+0000 7f8313400640 1 -- 172.21.15.102:0/1264178656 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83140d14b0 con 0x7f83140b5aa0 2024-03-23T03:47:13.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.376+0000 7f8312000640 1 -- 172.21.15.102:0/1264178656 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2002580551 0 0) 0x7f83140d1280 con 0x7f83140cd740 2024-03-23T03:47:13.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.377+0000 7f8313400640 1 --2- 172.21.15.102:0/1264178656 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83140b5aa0 0x7f83140b5f30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f83140b8050 tx=0x7f830c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a91721f1b727f1b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:13.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.378+0000 7f8312000640 1 -- 172.21.15.102:0/1264178656 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830c031830 con 0x7f83140b5aa0 2024-03-23T03:47:13.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.378+0000 7f8312000640 1 -- 172.21.15.102:0/1264178656 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f830c031990 con 0x7f83140b5aa0 2024-03-23T03:47:13.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.378+0000 7f8312000640 1 -- 172.21.15.102:0/1264178656 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f830c031d20 con 0x7f83140b5aa0 2024-03-23T03:47:13.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.378+0000 7f8319a00640 1 -- 172.21.15.102:0/1264178656 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83140b5aa0 msgr2=0x7f83140b5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:13.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.378+0000 7f8319a00640 1 --2- 172.21.15.102:0/1264178656 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83140b5aa0 0x7f83140b5f30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f83140b8050 tx=0x7f830c02b8d0 comp rx=0 tx=0).stop 2024-03-23T03:47:13.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.378+0000 7f8319a00640 1 -- 172.21.15.102:0/1264178656 shutdown_connections 2024-03-23T03:47:13.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.379+0000 7f8319a00640 1 --2- 172.21.15.102:0/1264178656 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83140b5aa0 0x7f83140b5f30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:13.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.379+0000 7f8319a00640 1 --2- 172.21.15.102:0/1264178656 >> v2:172.21.15.173:3300/0 conn(0x7f83140b84b0 0x7f83140d0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:13.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.379+0000 7f8319a00640 1 -- 172.21.15.102:0/1264178656 >> 172.21.15.102:0/1264178656 conn(0x7f83140cf5d0 msgr2=0x7f83140cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:13.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.379+0000 7f8319a00640 1 -- 172.21.15.102:0/1264178656 shutdown_connections 2024-03-23T03:47:13.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.379+0000 7f8319a00640 1 -- 172.21.15.102:0/1264178656 wait complete. 2024-03-23T03:47:13.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.381+0000 7f8319a00640 1 Processor -- start 2024-03-23T03:47:13.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.381+0000 7f8319a00640 1 -- start start 2024-03-23T03:47:13.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.382+0000 7f8319a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f83140b5aa0 0x7f8314157890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:13.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.382+0000 7f8319a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83140b84b0 0x7f831415ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:13.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.382+0000 7f8319a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83140cd740 0x7f831415efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:13.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.382+0000 7f8319a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83141f23f0 con 0x7f83140b84b0 2024-03-23T03:47:13.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.382+0000 7f8319a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8314158270 con 0x7f83140cd740 2024-03-23T03:47:13.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.382+0000 7f8319a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8314158550 con 0x7f83140b5aa0 2024-03-23T03:47:13.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.383+0000 7f830a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83140b84b0 0x7f831415ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:13.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.383+0000 7f830a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83140b84b0 0x7f831415ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37024/0 (socket says 172.21.15.102:37024) 2024-03-23T03:47:13.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.383+0000 7f830a000640 1 -- 172.21.15.102:0/788782634 learned_addr learned my addr 172.21.15.102:0/788782634 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:13.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.383+0000 7f8313400640 1 --2- 172.21.15.102:0/788782634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83140cd740 0x7f831415efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:13.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.383+0000 7f8312a00640 1 --2- 172.21.15.102:0/788782634 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f83140b5aa0 0x7f8314157890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:13.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.383+0000 7f830a000640 1 -- 172.21.15.102:0/788782634 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f83140b5aa0 msgr2=0x7f8314157890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:13.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.383+0000 7f830a000640 1 --2- 172.21.15.102:0/788782634 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f83140b5aa0 0x7f8314157890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:13.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.383+0000 7f830a000640 1 -- 172.21.15.102:0/788782634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83140cd740 msgr2=0x7f831415efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:13.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.383+0000 7f830a000640 1 --2- 172.21.15.102:0/788782634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83140cd740 0x7f831415efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:13.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.383+0000 7f830a000640 1 -- 172.21.15.102:0/788782634 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830c009400 con 0x7f83140b84b0 2024-03-23T03:47:13.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.383+0000 7f8312a00640 1 --2- 172.21.15.102:0/788782634 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f83140b5aa0 0x7f8314157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:13.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.383+0000 7f8313400640 1 --2- 172.21.15.102:0/788782634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83140cd740 0x7f831415efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:13.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.383+0000 7f830a000640 1 --2- 172.21.15.102:0/788782634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83140b84b0 0x7f831415ea70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f82fc00d800 tx=0x7f82fc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:13.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.384+0000 7f830b400640 1 -- 172.21.15.102:0/788782634 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc00b6c0 con 0x7f83140b84b0 2024-03-23T03:47:13.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.384+0000 7f830b400640 1 -- 172.21.15.102:0/788782634 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82fc00b820 con 0x7f83140b84b0 2024-03-23T03:47:13.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.384+0000 7f830b400640 1 -- 172.21.15.102:0/788782634 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82fc00bbb0 con 0x7f83140b84b0 2024-03-23T03:47:13.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.384+0000 7f8319a00640 1 -- 172.21.15.102:0/788782634 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8314158830 con 0x7f83140b84b0 2024-03-23T03:47:13.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.384+0000 7f8319a00640 1 -- 172.21.15.102:0/788782634 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8314157530 con 0x7f83140b84b0 2024-03-23T03:47:13.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.385+0000 7f8319a00640 1 -- 172.21.15.102:0/788782634 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83141576c0 con 0x7f83140b84b0 2024-03-23T03:47:13.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.386+0000 7f830b400640 1 -- 172.21.15.102:0/788782634 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f82fc00bd10 con 0x7f83140b84b0 2024-03-23T03:47:13.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.387+0000 7f830b400640 1 --2- 172.21.15.102:0/788782634 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82f006ac00 0x7f82f006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:13.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.387+0000 7f830b400640 1 -- 172.21.15.102:0/788782634 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f82fc089ba0 con 0x7f83140b84b0 2024-03-23T03:47:13.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.387+0000 7f8312a00640 1 --2- 172.21.15.102:0/788782634 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82f006ac00 0x7f82f006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:13.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.388+0000 7f8312a00640 1 --2- 172.21.15.102:0/788782634 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82f006ac00 0x7f82f006d0c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8304007fd0 tx=0x7f8304009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:13.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.391+0000 7f830b400640 1 -- 172.21.15.102:0/788782634 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f82fc054a80 con 0x7f83140b84b0 2024-03-23T03:47:13.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.619+0000 7f8319a00640 1 -- 172.21.15.102:0/788782634 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8314158a20 con 0x7f83140b84b0 2024-03-23T03:47:13.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.814+0000 7f830b400640 1 -- 172.21.15.102:0/788782634 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f82fc0580a0 con 0x7f83140b84b0 2024-03-23T03:47:13.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.819+0000 7f8319a00640 1 -- 172.21.15.102:0/788782634 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82f006ac00 msgr2=0x7f82f006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:13.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.819+0000 7f8319a00640 1 --2- 172.21.15.102:0/788782634 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82f006ac00 0x7f82f006d0c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8304007fd0 tx=0x7f8304009040 comp rx=0 tx=0).stop 2024-03-23T03:47:13.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.819+0000 7f8319a00640 1 -- 172.21.15.102:0/788782634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83140b84b0 msgr2=0x7f831415ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:13.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.819+0000 7f8319a00640 1 --2- 172.21.15.102:0/788782634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83140b84b0 0x7f831415ea70 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f82fc00d800 tx=0x7f82fc00dcc0 comp rx=0 tx=0).stop 2024-03-23T03:47:13.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.819+0000 7f8319a00640 1 -- 172.21.15.102:0/788782634 shutdown_connections 2024-03-23T03:47:13.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.819+0000 7f8319a00640 1 --2- 172.21.15.102:0/788782634 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82f006ac00 0x7f82f006d0c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:13.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.819+0000 7f8319a00640 1 --2- 172.21.15.102:0/788782634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f83140cd740 0x7f831415efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:13.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.819+0000 7f8319a00640 1 --2- 172.21.15.102:0/788782634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83140b84b0 0x7f831415ea70 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:13.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.819+0000 7f8319a00640 1 --2- 172.21.15.102:0/788782634 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f83140b5aa0 0x7f8314157890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:13.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.819+0000 7f8319a00640 1 -- 172.21.15.102:0/788782634 >> 172.21.15.102:0/788782634 conn(0x7f83140cf5d0 msgr2=0x7f83140b5e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:13.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.819+0000 7f8319a00640 1 -- 172.21.15.102:0/788782634 shutdown_connections 2024-03-23T03:47:13.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:13.819+0000 7f8319a00640 1 -- 172.21.15.102:0/788782634 wait complete. 2024-03-23T03:47:13.840 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-03-23T03:47:13.841 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T03:47:14.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.054+0000 7f5e65400640 1 Processor -- start 2024-03-23T03:47:14.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.054+0000 7f5e65400640 1 -- start start 2024-03-23T03:47:14.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.055+0000 7f5e65400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5e60159830 0x7f5e600cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:14.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.055+0000 7f5e65400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e600ca7b0 0x7f5e600cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:14.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.055+0000 7f5e65400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f5e600ce930 0x7f5e600cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:14.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.055+0000 7f5e65400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e6006de50 con 0x7f5e60159830 2024-03-23T03:47:14.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.055+0000 7f5e65400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e600cf300 con 0x7f5e600ce930 2024-03-23T03:47:14.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.055+0000 7f5e65400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e60159c10 con 0x7f5e600ca7b0 2024-03-23T03:47:14.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.056+0000 7f5e5e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5e60159830 0x7f5e600cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:14.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.056+0000 7f5e5e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5e60159830 0x7f5e600cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37036/0 (socket says 172.21.15.102:37036) 2024-03-23T03:47:14.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.056+0000 7f5e5e800640 1 -- 172.21.15.102:0/1875065401 learned_addr learned my addr 172.21.15.102:0/1875065401 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.056+0000 7f5e5f200640 1 --2- 172.21.15.102:0/1875065401 >> v2:172.21.15.173:3300/0 conn(0x7f5e600ce930 0x7f5e600cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.056+0000 7f5e5e800640 1 -- 172.21.15.102:0/1875065401 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e600ca7b0 msgr2=0x7f5e600cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.056+0000 7f5e5e800640 1 --2- 172.21.15.102:0/1875065401 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e600ca7b0 0x7f5e600cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.056+0000 7f5e5e800640 1 -- 172.21.15.102:0/1875065401 >> v2:172.21.15.173:3300/0 conn(0x7f5e600ce930 msgr2=0x7f5e600cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.056+0000 7f5e5e800640 1 --2- 172.21.15.102:0/1875065401 >> v2:172.21.15.173:3300/0 conn(0x7f5e600ce930 0x7f5e600cedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.056+0000 7f5e5e800640 1 -- 172.21.15.102:0/1875065401 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e600cd840 con 0x7f5e60159830 2024-03-23T03:47:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.057+0000 7f5e5e800640 1 --2- 172.21.15.102:0/1875065401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e60159830 0x7f5e600cc3c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5e600cb030 tx=0x7f5e5002e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b25176c33e8e1b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.058+0000 7f5e5de00640 1 -- 172.21.15.102:0/1875065401 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50036440 con 0x7f5e60159830 2024-03-23T03:47:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.058+0000 7f5e5de00640 1 -- 172.21.15.102:0/1875065401 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e500365a0 con 0x7f5e60159830 2024-03-23T03:47:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.058+0000 7f5e5de00640 1 -- 172.21.15.102:0/1875065401 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e50036930 con 0x7f5e60159830 2024-03-23T03:47:14.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.058+0000 7f5e65400640 1 -- 172.21.15.102:0/1875065401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e60159830 msgr2=0x7f5e600cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:14.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.058+0000 7f5e65400640 1 --2- 172.21.15.102:0/1875065401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e60159830 0x7f5e600cc3c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5e600cb030 tx=0x7f5e5002e170 comp rx=0 tx=0).stop 2024-03-23T03:47:14.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.058+0000 7f5e65400640 1 -- 172.21.15.102:0/1875065401 shutdown_connections 2024-03-23T03:47:14.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.058+0000 7f5e65400640 1 --2- 172.21.15.102:0/1875065401 >> v2:172.21.15.173:3300/0 conn(0x7f5e600ce930 0x7f5e600cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:14.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.058+0000 7f5e65400640 1 --2- 172.21.15.102:0/1875065401 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e600ca7b0 0x7f5e600cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:14.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.058+0000 7f5e65400640 1 --2- 172.21.15.102:0/1875065401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e60159830 0x7f5e600cc3c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:14.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.058+0000 7f5e65400640 1 -- 172.21.15.102:0/1875065401 >> 172.21.15.102:0/1875065401 conn(0x7f5e6006a0b0 msgr2=0x7f5e6006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:14.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.058+0000 7f5e65400640 1 -- 172.21.15.102:0/1875065401 shutdown_connections 2024-03-23T03:47:14.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.058+0000 7f5e65400640 1 -- 172.21.15.102:0/1875065401 wait complete. 2024-03-23T03:47:14.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.060+0000 7f5e65400640 1 Processor -- start 2024-03-23T03:47:14.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.060+0000 7f5e65400640 1 -- start start 2024-03-23T03:47:14.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.061+0000 7f5e65400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e600ca7b0 0x7f5e60151820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:14.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.061+0000 7f5e65400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e600ce930 0x7f5e6014f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:14.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.061+0000 7f5e65400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e60159830 0x7f5e600b2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:14.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.061+0000 7f5e65400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e60159e80 con 0x7f5e60159830 2024-03-23T03:47:14.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.061+0000 7f5e65400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6014f8d0 con 0x7f5e600ce930 2024-03-23T03:47:14.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.061+0000 7f5e65400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6014fbb0 con 0x7f5e600ca7b0 2024-03-23T03:47:14.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.061+0000 7f5e5f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e60159830 0x7f5e600b2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:14.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.061+0000 7f5e5f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e60159830 0x7f5e600b2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37044/0 (socket says 172.21.15.102:37044) 2024-03-23T03:47:14.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.061+0000 7f5e5f200640 1 -- 172.21.15.102:0/2036144224 learned_addr learned my addr 172.21.15.102:0/2036144224 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:14.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.062+0000 7f5e5e800640 1 --2- 172.21.15.102:0/2036144224 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e600ca7b0 0x7f5e60151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:14.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.062+0000 7f5e55e00640 1 --2- 172.21.15.102:0/2036144224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e600ce930 0x7f5e6014f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:14.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.062+0000 7f5e5f200640 1 -- 172.21.15.102:0/2036144224 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e600ca7b0 msgr2=0x7f5e60151820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:14.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.062+0000 7f5e5f200640 1 --2- 172.21.15.102:0/2036144224 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e600ca7b0 0x7f5e60151820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:14.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.062+0000 7f5e5f200640 1 -- 172.21.15.102:0/2036144224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e600ce930 msgr2=0x7f5e6014f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:14.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.062+0000 7f5e5f200640 1 --2- 172.21.15.102:0/2036144224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e600ce930 0x7f5e6014f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:14.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.062+0000 7f5e5f200640 1 -- 172.21.15.102:0/2036144224 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e5000b770 con 0x7f5e60159830 2024-03-23T03:47:14.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.062+0000 7f5e55e00640 1 --2- 172.21.15.102:0/2036144224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e600ce930 0x7f5e6014f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:14.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.062+0000 7f5e5f200640 1 --2- 172.21.15.102:0/2036144224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e60159830 0x7f5e600b2770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5e5800cd70 tx=0x7f5e5800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:14.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.062+0000 7f5e57400640 1 -- 172.21.15.102:0/2036144224 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e58003b90 con 0x7f5e60159830 2024-03-23T03:47:14.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.062+0000 7f5e57400640 1 -- 172.21.15.102:0/2036144224 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e58003cf0 con 0x7f5e60159830 2024-03-23T03:47:14.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.062+0000 7f5e57400640 1 -- 172.21.15.102:0/2036144224 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e58004240 con 0x7f5e60159830 2024-03-23T03:47:14.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.062+0000 7f5e65400640 1 -- 172.21.15.102:0/2036144224 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e6014fe90 con 0x7f5e60159830 2024-03-23T03:47:14.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.063+0000 7f5e65400640 1 -- 172.21.15.102:0/2036144224 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e6014f5e0 con 0x7f5e60159830 2024-03-23T03:47:14.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.064+0000 7f5e65400640 1 -- 172.21.15.102:0/2036144224 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e6015b050 con 0x7f5e60159830 2024-03-23T03:47:14.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.065+0000 7f5e57400640 1 -- 172.21.15.102:0/2036144224 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5e580043a0 con 0x7f5e60159830 2024-03-23T03:47:14.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.065+0000 7f5e57400640 1 --2- 172.21.15.102:0/2036144224 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e3c06ac00 0x7f5e3c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:14.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.066+0000 7f5e57400640 1 -- 172.21.15.102:0/2036144224 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5e58012070 con 0x7f5e60159830 2024-03-23T03:47:14.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.066+0000 7f5e5e800640 1 --2- 172.21.15.102:0/2036144224 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e3c06ac00 0x7f5e3c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:14.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.066+0000 7f5e5e800640 1 --2- 172.21.15.102:0/2036144224 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e3c06ac00 0x7f5e3c06d0c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5e50038000 tx=0x7f5e50032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:14.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.070+0000 7f5e57400640 1 -- 172.21.15.102:0/2036144224 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5e6015b050 con 0x7f5e60159830 2024-03-23T03:47:14.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.305+0000 7f5e65400640 1 -- 172.21.15.102:0/2036144224 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f5e6015b050 con 0x7f5e60159830 2024-03-23T03:47:14.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.832+0000 7f5e57400640 1 -- 172.21.15.102:0/2036144224 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5e6015b050 con 0x7f5e60159830 2024-03-23T03:47:14.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.836+0000 7f5e65400640 1 -- 172.21.15.102:0/2036144224 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e3c06ac00 msgr2=0x7f5e3c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:14.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.836+0000 7f5e65400640 1 --2- 172.21.15.102:0/2036144224 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e3c06ac00 0x7f5e3c06d0c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5e50038000 tx=0x7f5e50032000 comp rx=0 tx=0).stop 2024-03-23T03:47:14.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.836+0000 7f5e65400640 1 -- 172.21.15.102:0/2036144224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e60159830 msgr2=0x7f5e600b2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:14.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.836+0000 7f5e65400640 1 --2- 172.21.15.102:0/2036144224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e60159830 0x7f5e600b2770 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5e5800cd70 tx=0x7f5e5800a660 comp rx=0 tx=0).stop 2024-03-23T03:47:14.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.836+0000 7f5e65400640 1 -- 172.21.15.102:0/2036144224 shutdown_connections 2024-03-23T03:47:14.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.836+0000 7f5e65400640 1 --2- 172.21.15.102:0/2036144224 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e3c06ac00 0x7f5e3c06d0c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:14.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.836+0000 7f5e65400640 1 --2- 172.21.15.102:0/2036144224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e60159830 0x7f5e600b2770 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:14.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.836+0000 7f5e65400640 1 --2- 172.21.15.102:0/2036144224 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e600ce930 0x7f5e6014f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:14.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.836+0000 7f5e65400640 1 --2- 172.21.15.102:0/2036144224 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e600ca7b0 0x7f5e60151820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:14.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.836+0000 7f5e65400640 1 -- 172.21.15.102:0/2036144224 >> 172.21.15.102:0/2036144224 conn(0x7f5e6006a0b0 msgr2=0x7f5e600b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:14.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.836+0000 7f5e65400640 1 -- 172.21.15.102:0/2036144224 shutdown_connections 2024-03-23T03:47:14.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:14.836+0000 7f5e65400640 1 -- 172.21.15.102:0/2036144224 wait complete. 2024-03-23T03:47:14.860 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T03:47:14.860 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:47:15.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.067+0000 7fdf8c400640 1 Processor -- start 2024-03-23T03:47:15.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.067+0000 7fdf8c400640 1 -- start start 2024-03-23T03:47:15.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.068+0000 7fdf8c400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fdf840adbc0 0x7fdf840b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.068+0000 7fdf8c400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf840b16a0 con 0x7fdf8406d390 2024-03-23T03:47:15.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.068+0000 7fdf8c400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf840b4700 con 0x7fdf840cbab0 2024-03-23T03:47:15.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.069+0000 7fdf8c400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf840cbfb0 con 0x7fdf840adbc0 2024-03-23T03:47:15.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.069+0000 7fdf8a400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdf8406d390 0x7fdf840b2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41104/0 (socket says 172.21.15.102:41104) 2024-03-23T03:47:15.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.069+0000 7fdf8a400640 1 -- 172.21.15.102:0/135732661 learned_addr learned my addr 172.21.15.102:0/135732661 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.069+0000 7fdf89a00640 1 --2- 172.21.15.102:0/135732661 >> v2:172.21.15.173:3300/0 conn(0x7fdf840adbc0 0x7fdf840b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.069+0000 7fdf83e00640 1 -- 172.21.15.102:0/135732661 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2175431501 0 0) 0x7fdf840b16a0 con 0x7fdf8406d390 2024-03-23T03:47:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.069+0000 7fdf83e00640 1 -- 172.21.15.102:0/135732661 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf6c003520 con 0x7fdf8406d390 2024-03-23T03:47:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.070+0000 7fdf83e00640 1 -- 172.21.15.102:0/135732661 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3117627526 0 0) 0x7fdf6c003520 con 0x7fdf8406d390 2024-03-23T03:47:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.070+0000 7fdf83e00640 1 -- 172.21.15.102:0/135732661 >> v2:172.21.15.173:3300/0 conn(0x7fdf840adbc0 msgr2=0x7fdf840b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.070+0000 7fdf83e00640 1 --2- 172.21.15.102:0/135732661 >> v2:172.21.15.173:3300/0 conn(0x7fdf840adbc0 0x7fdf840b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.070+0000 7fdf83e00640 1 -- 172.21.15.102:0/135732661 >> v1:172.21.15.173:6789/0 conn(0x7fdf840cbab0 legacy=0x7fdf840b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.070+0000 7fdf83e00640 1 -- 172.21.15.102:0/135732661 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf840cc290 con 0x7fdf8406d390 2024-03-23T03:47:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.070+0000 7fdf83e00640 1 -- 172.21.15.102:0/135732661 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fdf7c0019f0 con 0x7fdf8406d390 2024-03-23T03:47:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.070+0000 7fdf83e00640 1 -- 172.21.15.102:0/135732661 >> v1:172.21.15.102:6789/0 conn(0x7fdf8406d390 legacy=0x7fdf840b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.070+0000 7fdf83e00640 1 --2- 172.21.15.102:0/135732661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf6c0039c0 0x7fdf6c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.070+0000 7fdf83e00640 1 --2- 172.21.15.102:0/135732661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf6c0047c0 0x7fdf6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.070+0000 7fdf83e00640 1 --2- 172.21.15.102:0/135732661 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdf6c0051d0 0x7fdf6c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.070+0000 7fdf83e00640 1 -- 172.21.15.102:0/135732661 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf840cbfb0 con 0x7fdf6c0047c0 2024-03-23T03:47:15.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.070+0000 7fdf83e00640 1 -- 172.21.15.102:0/135732661 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c00e330 con 0x7fdf6c0039c0 2024-03-23T03:47:15.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.070+0000 7fdf83e00640 1 -- 172.21.15.102:0/135732661 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c00e4f0 con 0x7fdf6c0051d0 2024-03-23T03:47:15.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.070+0000 7fdf89000640 1 --2- 172.21.15.102:0/135732661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf6c0047c0 0x7fdf6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.070+0000 7fdf89000640 1 -- 172.21.15.102:0/135732661 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdf6c0051d0 msgr2=0x7fdf6c00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:15.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.071+0000 7fdf89000640 1 --2- 172.21.15.102:0/135732661 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdf6c0051d0 0x7fdf6c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.071+0000 7fdf89a00640 1 --2- 172.21.15.102:0/135732661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf6c0039c0 0x7fdf6c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.071+0000 7fdf89000640 1 -- 172.21.15.102:0/135732661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf6c0039c0 msgr2=0x7fdf6c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:15.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.071+0000 7fdf89000640 1 --2- 172.21.15.102:0/135732661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf6c0039c0 0x7fdf6c003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.071+0000 7fdf89000640 1 -- 172.21.15.102:0/135732661 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c003520 con 0x7fdf6c0047c0 2024-03-23T03:47:15.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.071+0000 7fdf8a400640 1 -- 172.21.15.102:0/135732661 reap_dead start 2024-03-23T03:47:15.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.072+0000 7fdf89000640 1 --2- 172.21.15.102:0/135732661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf6c0047c0 0x7fdf6c004c40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdf840cbee0 tx=0x7fdf6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=108623a1eb02856e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:15.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.072+0000 7fdf83e00640 1 -- 172.21.15.102:0/135732661 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf700052d0 con 0x7fdf6c0047c0 2024-03-23T03:47:15.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.072+0000 7fdf83e00640 1 -- 172.21.15.102:0/135732661 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7002e050 con 0x7fdf6c0047c0 2024-03-23T03:47:15.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.072+0000 7fdf83e00640 1 -- 172.21.15.102:0/135732661 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf70003830 con 0x7fdf6c0047c0 2024-03-23T03:47:15.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.072+0000 7fdf8c400640 1 -- 172.21.15.102:0/135732661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf6c0047c0 msgr2=0x7fdf6c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:15.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.072+0000 7fdf8c400640 1 --2- 172.21.15.102:0/135732661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf6c0047c0 0x7fdf6c004c40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fdf840cbee0 tx=0x7fdf6c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.073+0000 7fdf8c400640 1 -- 172.21.15.102:0/135732661 shutdown_connections 2024-03-23T03:47:15.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.073+0000 7fdf8c400640 1 --2- 172.21.15.102:0/135732661 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf6c0047c0 0x7fdf6c004c40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.073+0000 7fdf8c400640 1 -- 172.21.15.102:0/135732661 >> 172.21.15.102:0/135732661 conn(0x7fdf8406a3d0 msgr2=0x7fdf840b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:15.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.073+0000 7fdf8c400640 1 -- 172.21.15.102:0/135732661 shutdown_connections 2024-03-23T03:47:15.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.073+0000 7fdf8c400640 1 -- 172.21.15.102:0/135732661 wait complete. 2024-03-23T03:47:15.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.074+0000 7fdf8c400640 1 Processor -- start 2024-03-23T03:47:15.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.075+0000 7fdf8c400640 1 -- start start 2024-03-23T03:47:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.075+0000 7fdf8c400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf6c0047c0 0x7fdf84159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.075+0000 7fdf8c400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf84139fb0 0x7fdf84136200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.075+0000 7fdf8c400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdf841ee8b0 0x7fdf8413aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.075+0000 7fdf8c400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8413b6b0 con 0x7fdf6c0047c0 2024-03-23T03:47:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.075+0000 7fdf8c400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8415b9f0 con 0x7fdf84139fb0 2024-03-23T03:47:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.075+0000 7fdf8c400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8415bc90 con 0x7fdf841ee8b0 2024-03-23T03:47:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.076+0000 7fdf89a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf6c0047c0 0x7fdf84159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.076+0000 7fdf89a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf6c0047c0 0x7fdf84159090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37068/0 (socket says 172.21.15.102:37068) 2024-03-23T03:47:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.076+0000 7fdf89a00640 1 -- 172.21.15.102:0/2677582092 learned_addr learned my addr 172.21.15.102:0/2677582092 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.076+0000 7fdf89000640 1 --2- 172.21.15.102:0/2677582092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf84139fb0 0x7fdf84136200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.076+0000 7fdf89a00640 1 -- 172.21.15.102:0/2677582092 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdf841ee8b0 msgr2=0x7fdf8413aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:15.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.076+0000 7fdf8a400640 1 --2- 172.21.15.102:0/2677582092 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdf841ee8b0 0x7fdf8413aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.076+0000 7fdf89a00640 1 --2- 172.21.15.102:0/2677582092 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdf841ee8b0 0x7fdf8413aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.076+0000 7fdf89a00640 1 -- 172.21.15.102:0/2677582092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf84139fb0 msgr2=0x7fdf84136200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.076+0000 7fdf89a00640 1 --2- 172.21.15.102:0/2677582092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf84139fb0 0x7fdf84136200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.076+0000 7fdf89a00640 1 -- 172.21.15.102:0/2677582092 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf8415bf70 con 0x7fdf6c0047c0 2024-03-23T03:47:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.076+0000 7fdf89000640 1 --2- 172.21.15.102:0/2677582092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf84139fb0 0x7fdf84136200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.076+0000 7fdf89a00640 1 --2- 172.21.15.102:0/2677582092 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf6c0047c0 0x7fdf84159090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdf78010790 tx=0x7fdf78010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.077+0000 7fdf81600640 1 -- 172.21.15.102:0/2677582092 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf78009630 con 0x7fdf6c0047c0 2024-03-23T03:47:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.077+0000 7fdf81600640 1 -- 172.21.15.102:0/2677582092 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7800eb60 con 0x7fdf6c0047c0 2024-03-23T03:47:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.077+0000 7fdf81600640 1 -- 172.21.15.102:0/2677582092 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf78010ea0 con 0x7fdf6c0047c0 2024-03-23T03:47:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.077+0000 7fdf8c400640 1 -- 172.21.15.102:0/2677582092 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf84136740 con 0x7fdf6c0047c0 2024-03-23T03:47:15.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.077+0000 7fdf8c400640 1 -- 172.21.15.102:0/2677582092 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf8413b920 con 0x7fdf6c0047c0 2024-03-23T03:47:15.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.078+0000 7fdf8c400640 1 -- 172.21.15.102:0/2677582092 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8413a390 con 0x7fdf6c0047c0 2024-03-23T03:47:15.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.079+0000 7fdf81600640 1 -- 172.21.15.102:0/2677582092 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdf7801c400 con 0x7fdf6c0047c0 2024-03-23T03:47:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.080+0000 7fdf81600640 1 --2- 172.21.15.102:0/2677582092 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdf6406ac30 0x7fdf6406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.080+0000 7fdf81600640 1 -- 172.21.15.102:0/2677582092 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fdf78089c00 con 0x7fdf6c0047c0 2024-03-23T03:47:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.080+0000 7fdf89000640 1 --2- 172.21.15.102:0/2677582092 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdf6406ac30 0x7fdf6406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.081+0000 7fdf89000640 1 --2- 172.21.15.102:0/2677582092 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdf6406ac30 0x7fdf6406d0f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdf70033000 tx=0x7fdf70009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:15.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.085+0000 7fdf81600640 1 -- 172.21.15.102:0/2677582092 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdf78054a50 con 0x7fdf6c0047c0 2024-03-23T03:47:15.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.319+0000 7fdf8c400640 1 -- 172.21.15.102:0/2677582092 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf8415c100 con 0x7fdf6c0047c0 2024-03-23T03:47:15.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.320+0000 7fdf81600640 1 -- 172.21.15.102:0/2677582092 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4739 (secure 0 0 0) 0x7fdf78058070 con 0x7fdf6c0047c0 2024-03-23T03:47:15.321 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:15.321 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/891818172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":891818172},{"type":"v1","addr":"172.21.15.102:6836","nonce":891818172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2404489233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2404489233},{"type":"v1","addr":"172.21.15.102:6837","nonce":2404489233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/1839967745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":1839967745},{"type":"v1","addr":"172.21.15.173:6835","nonce":1839967745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:46:53.912287+0000","modified":"2024-03-23T03:47:14.818670+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6834/1462637509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1462637509},{"type":"v1","addr":"172.21.15.173:6834","nonce":1462637509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-23T03:47:15.321 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 8 2024-03-23T03:47:15.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.323+0000 7fdf8c400640 1 -- 172.21.15.102:0/2677582092 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdf6406ac30 msgr2=0x7fdf6406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:15.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.323+0000 7fdf8c400640 1 --2- 172.21.15.102:0/2677582092 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdf6406ac30 0x7fdf6406d0f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fdf70033000 tx=0x7fdf70009030 comp rx=0 tx=0).stop 2024-03-23T03:47:15.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.323+0000 7fdf8c400640 1 -- 172.21.15.102:0/2677582092 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf6c0047c0 msgr2=0x7fdf84159090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:15.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.323+0000 7fdf8c400640 1 --2- 172.21.15.102:0/2677582092 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf6c0047c0 0x7fdf84159090 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdf78010790 tx=0x7fdf78010c50 comp rx=0 tx=0).stop 2024-03-23T03:47:15.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.323+0000 7fdf8c400640 1 -- 172.21.15.102:0/2677582092 shutdown_connections 2024-03-23T03:47:15.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.323+0000 7fdf8c400640 1 --2- 172.21.15.102:0/2677582092 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdf6406ac30 0x7fdf6406d0f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.323+0000 7fdf8c400640 1 --2- 172.21.15.102:0/2677582092 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdf841ee8b0 0x7fdf8413aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.323+0000 7fdf8c400640 1 --2- 172.21.15.102:0/2677582092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf84139fb0 0x7fdf84136200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.323+0000 7fdf8c400640 1 --2- 172.21.15.102:0/2677582092 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf6c0047c0 0x7fdf84159090 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.323+0000 7fdf8c400640 1 -- 172.21.15.102:0/2677582092 >> 172.21.15.102:0/2677582092 conn(0x7fdf8406a3d0 msgr2=0x7fdf840cda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:15.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.323+0000 7fdf8c400640 1 -- 172.21.15.102:0/2677582092 shutdown_connections 2024-03-23T03:47:15.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.323+0000 7fdf8c400640 1 -- 172.21.15.102:0/2677582092 wait complete. 2024-03-23T03:47:15.343 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:15.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.544+0000 7f345b400640 1 Processor -- start 2024-03-23T03:47:15.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.544+0000 7f345b400640 1 -- start start 2024-03-23T03:47:15.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.545+0000 7f345b400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f345415b560 0x7f345415b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.545+0000 7f345b400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f34540b39b0 0x7f34540b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.545+0000 7f345b400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f345406c6f0 con 0x7f345415e500 2024-03-23T03:47:15.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.545+0000 7f345b400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34540b4370 con 0x7f34540b39b0 2024-03-23T03:47:15.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.545+0000 7f345b400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f345406c9d0 con 0x7f345415b560 2024-03-23T03:47:15.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.546+0000 7f3459400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f345415e500 0x7f345415e990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41110/0 (socket says 172.21.15.102:41110) 2024-03-23T03:47:15.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.546+0000 7f3459400640 1 -- 172.21.15.102:0/438258945 learned_addr learned my addr 172.21.15.102:0/438258945 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:15.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.546+0000 7f3458a00640 1 --2- 172.21.15.102:0/438258945 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f345415b560 0x7f345415b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.546+0000 7f3453e00640 1 -- 172.21.15.102:0/438258945 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172838403 0 0) 0x7f345406c6f0 con 0x7f345415e500 2024-03-23T03:47:15.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.546+0000 7f344be00640 1 --2- 172.21.15.102:0/438258945 >> v2:172.21.15.173:3300/0 conn(0x7f34540b39b0 0x7f34540b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.546+0000 7f3453e00640 1 -- 172.21.15.102:0/438258945 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3438003520 con 0x7f345415e500 2024-03-23T03:47:15.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.546+0000 7f3453e00640 1 -- 172.21.15.102:0/438258945 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1677327747 0 0) 0x7f3438003520 con 0x7f345415e500 2024-03-23T03:47:15.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.546+0000 7f3453e00640 1 -- 172.21.15.102:0/438258945 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f345415b560 msgr2=0x7f345415b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:15.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.546+0000 7f3453e00640 1 --2- 172.21.15.102:0/438258945 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f345415b560 0x7f345415b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.546+0000 7f3453e00640 1 -- 172.21.15.102:0/438258945 >> v2:172.21.15.173:3300/0 conn(0x7f34540b39b0 msgr2=0x7f34540b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:15.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.546+0000 7f3453e00640 1 --2- 172.21.15.102:0/438258945 >> v2:172.21.15.173:3300/0 conn(0x7f34540b39b0 0x7f34540b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.546+0000 7f3453e00640 1 -- 172.21.15.102:0/438258945 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345406ccb0 con 0x7f345415e500 2024-03-23T03:47:15.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.546+0000 7f3453e00640 1 -- 172.21.15.102:0/438258945 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f344c0019f0 con 0x7f345415e500 2024-03-23T03:47:15.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.547+0000 7f3453e00640 1 -- 172.21.15.102:0/438258945 >> v1:172.21.15.102:6789/0 conn(0x7f345415e500 legacy=0x7f345415e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:15.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.547+0000 7f3453e00640 1 --2- 172.21.15.102:0/438258945 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3438003970 0x7f3438003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.547+0000 7f3453e00640 1 --2- 172.21.15.102:0/438258945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34380047c0 0x7f3438004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.547+0000 7f3453e00640 1 --2- 172.21.15.102:0/438258945 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f345415b560 0x7f3438005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.547+0000 7f3453e00640 1 -- 172.21.15.102:0/438258945 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34540b4370 con 0x7f3438003970 2024-03-23T03:47:15.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.547+0000 7f3453e00640 1 -- 172.21.15.102:0/438258945 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f345406c9d0 con 0x7f34380047c0 2024-03-23T03:47:15.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.547+0000 7f3453e00640 1 -- 172.21.15.102:0/438258945 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343800ba30 con 0x7f345415b560 2024-03-23T03:47:15.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.547+0000 7f3458a00640 1 --2- 172.21.15.102:0/438258945 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3438003970 0x7f3438003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.547+0000 7f3458a00640 1 -- 172.21.15.102:0/438258945 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f345415b560 msgr2=0x7f3438005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:15.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.547+0000 7f3458a00640 1 --2- 172.21.15.102:0/438258945 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f345415b560 0x7f3438005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.547+0000 7f3458a00640 1 -- 172.21.15.102:0/438258945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34380047c0 msgr2=0x7f3438004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:15.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.547+0000 7f3458a00640 1 --2- 172.21.15.102:0/438258945 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34380047c0 0x7f3438004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.547+0000 7f3458a00640 1 -- 172.21.15.102:0/438258945 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3438003520 con 0x7f3438003970 2024-03-23T03:47:15.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.548+0000 7f3458a00640 1 --2- 172.21.15.102:0/438258945 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3438003970 0x7f3438003d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f344400ba00 tx=0x7f34380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf671f5618db2407 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:15.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.549+0000 7f3453e00640 1 -- 172.21.15.102:0/438258945 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3444032630 con 0x7f3438003970 2024-03-23T03:47:15.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.549+0000 7f3453e00640 1 -- 172.21.15.102:0/438258945 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3444032790 con 0x7f3438003970 2024-03-23T03:47:15.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.549+0000 7f3453e00640 1 -- 172.21.15.102:0/438258945 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3444032b20 con 0x7f3438003970 2024-03-23T03:47:15.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.549+0000 7f345b400640 1 -- 172.21.15.102:0/438258945 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3438003970 msgr2=0x7f3438003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:15.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.549+0000 7f345b400640 1 --2- 172.21.15.102:0/438258945 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3438003970 0x7f3438003d90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f344400ba00 tx=0x7f34380057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.549+0000 7f3459400640 1 -- 172.21.15.102:0/438258945 reap_dead start 2024-03-23T03:47:15.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.549+0000 7f345b400640 1 -- 172.21.15.102:0/438258945 shutdown_connections 2024-03-23T03:47:15.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.549+0000 7f345b400640 1 -- 172.21.15.102:0/438258945 >> 172.21.15.102:0/438258945 conn(0x7f345406a3d0 msgr2=0x7f34540b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:15.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.549+0000 7f345b400640 1 -- 172.21.15.102:0/438258945 shutdown_connections 2024-03-23T03:47:15.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.549+0000 7f345b400640 1 -- 172.21.15.102:0/438258945 wait complete. 2024-03-23T03:47:15.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.551+0000 7f345b400640 1 Processor -- start 2024-03-23T03:47:15.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.551+0000 7f345b400640 1 -- start start 2024-03-23T03:47:15.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.552+0000 7f345b400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3438003970 0x7f34540b0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.552+0000 7f345b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3454151fa0 0x7f3454138f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.552+0000 7f345b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3454153740 0x7f345415d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.552+0000 7f345b400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3454152400 con 0x7f3454153740 2024-03-23T03:47:15.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.552+0000 7f345b400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34541526a0 con 0x7f3454151fa0 2024-03-23T03:47:15.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.552+0000 7f345b400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3454152980 con 0x7f3438003970 2024-03-23T03:47:15.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.552+0000 7f3459400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3454153740 0x7f345415d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.552+0000 7f3459400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3454153740 0x7f345415d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37086/0 (socket says 172.21.15.102:37086) 2024-03-23T03:47:15.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.552+0000 7f3459400640 1 -- 172.21.15.102:0/1638757306 learned_addr learned my addr 172.21.15.102:0/1638757306 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:15.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.553+0000 7f3458a00640 1 --2- 172.21.15.102:0/1638757306 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3438003970 0x7f34540b0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.553+0000 7f344be00640 1 --2- 172.21.15.102:0/1638757306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3454151fa0 0x7f3454138f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.553+0000 7f3459400640 1 -- 172.21.15.102:0/1638757306 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3438003970 msgr2=0x7f34540b0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:15.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.553+0000 7f3459400640 1 --2- 172.21.15.102:0/1638757306 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3438003970 0x7f34540b0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.553+0000 7f3459400640 1 -- 172.21.15.102:0/1638757306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3454151fa0 msgr2=0x7f3454138f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:15.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.553+0000 7f3459400640 1 --2- 172.21.15.102:0/1638757306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3454151fa0 0x7f3454138f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.553+0000 7f3459400640 1 -- 172.21.15.102:0/1638757306 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3454152c60 con 0x7f3454153740 2024-03-23T03:47:15.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.553+0000 7f3459400640 1 --2- 172.21.15.102:0/1638757306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3454153740 0x7f345415d500 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f344c0013d0 tx=0x7f344c00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:15.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.553+0000 7f3451600640 1 -- 172.21.15.102:0/1638757306 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c00a610 con 0x7f3454153740 2024-03-23T03:47:15.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.553+0000 7f345b400640 1 -- 172.21.15.102:0/1638757306 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3454152f40 con 0x7f3454153740 2024-03-23T03:47:15.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.553+0000 7f3451600640 1 -- 172.21.15.102:0/1638757306 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344c00a790 con 0x7f3454153740 2024-03-23T03:47:15.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.553+0000 7f3451600640 1 -- 172.21.15.102:0/1638757306 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344c00ab40 con 0x7f3454153740 2024-03-23T03:47:15.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.554+0000 7f345b400640 1 -- 172.21.15.102:0/1638757306 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34541394a0 con 0x7f3454153740 2024-03-23T03:47:15.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.555+0000 7f345b400640 1 -- 172.21.15.102:0/1638757306 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3454139630 con 0x7f3454153740 2024-03-23T03:47:15.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.555+0000 7f3451600640 1 -- 172.21.15.102:0/1638757306 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f344c00aca0 con 0x7f3454153740 2024-03-23T03:47:15.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.556+0000 7f3451600640 1 --2- 172.21.15.102:0/1638757306 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f343006ac30 0x7f343006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.556+0000 7f3451600640 1 -- 172.21.15.102:0/1638757306 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f344c088c10 con 0x7f3454153740 2024-03-23T03:47:15.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.557+0000 7f3458a00640 1 --2- 172.21.15.102:0/1638757306 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f343006ac30 0x7f343006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.557+0000 7f3458a00640 1 --2- 172.21.15.102:0/1638757306 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f343006ac30 0x7f343006d0f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f34380057a0 tx=0x7f3444030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:15.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.561+0000 7f3451600640 1 -- 172.21.15.102:0/1638757306 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3454139630 con 0x7f3454153740 2024-03-23T03:47:15.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.744+0000 7f345b400640 1 -- 172.21.15.102:0/1638757306 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3454153460 con 0x7f3454153740 2024-03-23T03:47:15.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.747+0000 7f3451600640 1 -- 172.21.15.102:0/1638757306 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f3454153460 con 0x7f3454153740 2024-03-23T03:47:15.748 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:15.748 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:58.410551+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:47:15.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.750+0000 7f345b400640 1 -- 172.21.15.102:0/1638757306 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f343006ac30 msgr2=0x7f343006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:15.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.750+0000 7f345b400640 1 --2- 172.21.15.102:0/1638757306 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f343006ac30 0x7f343006d0f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f34380057a0 tx=0x7f3444030000 comp rx=0 tx=0).stop 2024-03-23T03:47:15.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.750+0000 7f345b400640 1 -- 172.21.15.102:0/1638757306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3454153740 msgr2=0x7f345415d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:15.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.750+0000 7f345b400640 1 --2- 172.21.15.102:0/1638757306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3454153740 0x7f345415d500 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f344c0013d0 tx=0x7f344c00ccd0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.750+0000 7f345b400640 1 -- 172.21.15.102:0/1638757306 shutdown_connections 2024-03-23T03:47:15.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.750+0000 7f345b400640 1 --2- 172.21.15.102:0/1638757306 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f343006ac30 0x7f343006d0f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.750+0000 7f345b400640 1 --2- 172.21.15.102:0/1638757306 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3454153740 0x7f345415d500 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.750+0000 7f345b400640 1 --2- 172.21.15.102:0/1638757306 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3454151fa0 0x7f3454138f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.750+0000 7f345b400640 1 --2- 172.21.15.102:0/1638757306 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3438003970 0x7f34540b0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.750+0000 7f345b400640 1 -- 172.21.15.102:0/1638757306 >> 172.21.15.102:0/1638757306 conn(0x7f345406a3d0 msgr2=0x7f34540cb5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:15.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.750+0000 7f345b400640 1 -- 172.21.15.102:0/1638757306 shutdown_connections 2024-03-23T03:47:15.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.750+0000 7f345b400640 1 -- 172.21.15.102:0/1638757306 wait complete. 2024-03-23T03:47:15.767 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T03:47:15.767 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:47:15.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.976+0000 7f13aea00640 1 Processor -- start 2024-03-23T03:47:15.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.976+0000 7f13aea00640 1 -- start start 2024-03-23T03:47:15.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.978+0000 7f13aea00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13a80b39b0 0x7f13a80b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.978+0000 7f13aea00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f13a815c400 0x7f13a815c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.978+0000 7f13aea00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a806c6f0 con 0x7f13a815b560 2024-03-23T03:47:15.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.978+0000 7f13aea00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a80b4370 con 0x7f13a815c400 2024-03-23T03:47:15.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.978+0000 7f13aea00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a806c9d0 con 0x7f13a80b39b0 2024-03-23T03:47:15.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.978+0000 7f13aca00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f13a815c400 0x7f13a815c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.978+0000 7f139f400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13a80b39b0 0x7f13a80b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.978+0000 7f13aca00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f13a815c400 0x7f13a815c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:42090/0 (socket says 172.21.15.102:42090) 2024-03-23T03:47:15.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.978+0000 7f13aca00640 1 -- 172.21.15.102:0/3075722053 learned_addr learned my addr 172.21.15.102:0/3075722053 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:15.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.979+0000 7f13aca00640 1 -- 172.21.15.102:0/3075722053 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13a80b39b0 msgr2=0x7f13a80b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:15.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.979+0000 7f13aca00640 1 --2- 172.21.15.102:0/3075722053 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13a80b39b0 0x7f13a80b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.979+0000 7f13aca00640 1 -- 172.21.15.102:0/3075722053 >> v1:172.21.15.173:6789/0 conn(0x7f13a815b560 legacy=0x7f13a815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:15.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.979+0000 7f13aca00640 1 -- 172.21.15.102:0/3075722053 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a806ccb0 con 0x7f13a815c400 2024-03-23T03:47:15.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.980+0000 7f13aca00640 1 --2- 172.21.15.102:0/3075722053 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13a815c400 0x7f13a815c890 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009e50 tx=0x7f13a002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67a2fe3cdd382589 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:15.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.981+0000 7f13a7400640 1 -- 172.21.15.102:0/3075722053 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a002cb00 con 0x7f13a815c400 2024-03-23T03:47:15.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.981+0000 7f13a7400640 1 -- 172.21.15.102:0/3075722053 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a002cc60 con 0x7f13a815c400 2024-03-23T03:47:15.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.981+0000 7f13a7400640 1 -- 172.21.15.102:0/3075722053 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a00343f0 con 0x7f13a815c400 2024-03-23T03:47:15.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.982+0000 7f13aea00640 1 -- 172.21.15.102:0/3075722053 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13a815c400 msgr2=0x7f13a815c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:15.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.982+0000 7f13aea00640 1 --2- 172.21.15.102:0/3075722053 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13a815c400 0x7f13a815c890 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f13a0009e50 tx=0x7f13a002bf60 comp rx=0 tx=0).stop 2024-03-23T03:47:15.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.982+0000 7f13aea00640 1 -- 172.21.15.102:0/3075722053 shutdown_connections 2024-03-23T03:47:15.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.982+0000 7f13aea00640 1 --2- 172.21.15.102:0/3075722053 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13a815c400 0x7f13a815c890 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.982+0000 7f13aea00640 1 --2- 172.21.15.102:0/3075722053 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13a80b39b0 0x7f13a80b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.982+0000 7f13aea00640 1 -- 172.21.15.102:0/3075722053 >> 172.21.15.102:0/3075722053 conn(0x7f13a806a3d0 msgr2=0x7f13a80b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:15.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.982+0000 7f13aea00640 1 -- 172.21.15.102:0/3075722053 shutdown_connections 2024-03-23T03:47:15.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.982+0000 7f13aea00640 1 -- 172.21.15.102:0/3075722053 wait complete. 2024-03-23T03:47:15.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.983+0000 7f13aea00640 1 Processor -- start 2024-03-23T03:47:15.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.984+0000 7f13aea00640 1 -- start start 2024-03-23T03:47:15.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.984+0000 7f13aea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13a80b39b0 0x7f13a80af5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.984+0000 7f13aea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13a815b560 0x7f13a8157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.984+0000 7f13aea00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13a815c400 0x7f13a81575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.984+0000 7f13aea00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a8157bc0 con 0x7f13a80b39b0 2024-03-23T03:47:15.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.985+0000 7f13aea00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a8157ea0 con 0x7f13a815b560 2024-03-23T03:47:15.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.985+0000 7f13aea00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a8158180 con 0x7f13a815c400 2024-03-23T03:47:15.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.985+0000 7f13a7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13a80b39b0 0x7f13a80af5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.985+0000 7f13a7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13a80b39b0 0x7f13a80af5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37090/0 (socket says 172.21.15.102:37090) 2024-03-23T03:47:15.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.985+0000 7f13a7e00640 1 -- 172.21.15.102:0/1085239953 learned_addr learned my addr 172.21.15.102:0/1085239953 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:15.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.985+0000 7f139f400640 1 --2- 172.21.15.102:0/1085239953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13a815b560 0x7f13a8157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.985+0000 7f13aca00640 1 --2- 172.21.15.102:0/1085239953 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13a815c400 0x7f13a81575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.985+0000 7f13a7e00640 1 -- 172.21.15.102:0/1085239953 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13a815c400 msgr2=0x7f13a81575b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:15.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.985+0000 7f13a7e00640 1 --2- 172.21.15.102:0/1085239953 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13a815c400 0x7f13a81575b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.985+0000 7f13a7e00640 1 -- 172.21.15.102:0/1085239953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13a815b560 msgr2=0x7f13a8157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:15.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.985+0000 7f13a7e00640 1 --2- 172.21.15.102:0/1085239953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13a815b560 0x7f13a8157070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:15.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.985+0000 7f13a7e00640 1 -- 172.21.15.102:0/1085239953 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a0009b30 con 0x7f13a80b39b0 2024-03-23T03:47:15.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.986+0000 7f13a7e00640 1 --2- 172.21.15.102:0/1085239953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13a80b39b0 0x7f13a80af5d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f139800c400 tx=0x7f139800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:15.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.986+0000 7f13a4c00640 1 -- 172.21.15.102:0/1085239953 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13980052d0 con 0x7f13a80b39b0 2024-03-23T03:47:15.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.986+0000 7f13a4c00640 1 -- 172.21.15.102:0/1085239953 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1398005450 con 0x7f13a80b39b0 2024-03-23T03:47:15.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.986+0000 7f13aea00640 1 -- 172.21.15.102:0/1085239953 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a8151fa0 con 0x7f13a80b39b0 2024-03-23T03:47:15.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.986+0000 7f13a4c00640 1 -- 172.21.15.102:0/1085239953 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1398007260 con 0x7f13a80b39b0 2024-03-23T03:47:15.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.986+0000 7f13aea00640 1 -- 172.21.15.102:0/1085239953 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a80b3210 con 0x7f13a80b39b0 2024-03-23T03:47:15.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.987+0000 7f13aea00640 1 -- 172.21.15.102:0/1085239953 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a806ac20 con 0x7f13a80b39b0 2024-03-23T03:47:15.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.988+0000 7f13a4c00640 1 -- 172.21.15.102:0/1085239953 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f13980073c0 con 0x7f13a80b39b0 2024-03-23T03:47:15.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.989+0000 7f13a4c00640 1 --2- 172.21.15.102:0/1085239953 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f138406ac00 0x7f138406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:15.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.989+0000 7f13a4c00640 1 -- 172.21.15.102:0/1085239953 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1398088a90 con 0x7f13a80b39b0 2024-03-23T03:47:15.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.989+0000 7f139f400640 1 --2- 172.21.15.102:0/1085239953 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f138406ac00 0x7f138406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:15.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.990+0000 7f139f400640 1 --2- 172.21.15.102:0/1085239953 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f138406ac00 0x7f138406d0c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f13a80b2a90 tx=0x7f1390008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:15.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:15.994+0000 7f13a4c00640 1 -- 172.21.15.102:0/1085239953 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1398053980 con 0x7f13a80b39b0 2024-03-23T03:47:16.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.187+0000 7f13aea00640 1 -- 172.21.15.102:0/1085239953 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f13a815de40 con 0x7f138406ac00 2024-03-23T03:47:16.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.206+0000 7f13a4c00640 1 -- 172.21.15.102:0/1085239953 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345116 (secure 0 0 0) 0x7f13a815de40 con 0x7f138406ac00 2024-03-23T03:47:16.213 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:16.215 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:47:16.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.214+0000 7f13aea00640 1 -- 172.21.15.102:0/1085239953 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f138406ac00 msgr2=0x7f138406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:16.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.214+0000 7f13aea00640 1 --2- 172.21.15.102:0/1085239953 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f138406ac00 0x7f138406d0c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f13a80b2a90 tx=0x7f1390008040 comp rx=0 tx=0).stop 2024-03-23T03:47:16.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.214+0000 7f13aea00640 1 -- 172.21.15.102:0/1085239953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13a80b39b0 msgr2=0x7f13a80af5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:16.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.214+0000 7f13aea00640 1 --2- 172.21.15.102:0/1085239953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13a80b39b0 0x7f13a80af5d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f139800c400 tx=0x7f139800c8c0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.214+0000 7f13aea00640 1 -- 172.21.15.102:0/1085239953 shutdown_connections 2024-03-23T03:47:16.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.214+0000 7f13aea00640 1 --2- 172.21.15.102:0/1085239953 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f138406ac00 0x7f138406d0c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.214+0000 7f13aea00640 1 --2- 172.21.15.102:0/1085239953 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13a815c400 0x7f13a81575b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.215+0000 7f13aea00640 1 --2- 172.21.15.102:0/1085239953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13a815b560 0x7f13a8157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.215+0000 7f13aea00640 1 --2- 172.21.15.102:0/1085239953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13a80b39b0 0x7f13a80af5d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.215+0000 7f13aea00640 1 -- 172.21.15.102:0/1085239953 >> 172.21.15.102:0/1085239953 conn(0x7f13a806a3d0 msgr2=0x7f13a81593b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:16.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.215+0000 7f13aea00640 1 -- 172.21.15.102:0/1085239953 shutdown_connections 2024-03-23T03:47:16.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.215+0000 7f13aea00640 1 -- 172.21.15.102:0/1085239953 wait complete. 2024-03-23T03:47:16.235 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-23T03:47:15.721801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594820,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219556,"kb_used_data":3368,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755164,"statfs":{"total":773094113280,"available":772869287936,"internally_reserved":0,"allocated":3448832,"data_stored":2050424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":221236794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2090,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004284"},"pg_stats":[{"pgid":"2.1f","version":"21'4","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.954535+0000","last_change":"2024-03-23T03:46:53.465534+0000","last_active":"2024-03-23T03:46:57.954535+0000","last_peered":"2024-03-23T03:46:57.954535+0000","last_clean":"2024-03-23T03:46:57.954535+0000","last_became_active":"2024-03-23T03:46:53.465309+0000","last_became_peered":"2024-03-23T03:46:53.465309+0000","last_unstale":"2024-03-23T03:46:57.954535+0000","last_undegraded":"2024-03-23T03:46:57.954535+0000","last_fullsized":"2024-03-23T03:46:57.954535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:52:38.737363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951985+0000","last_change":"2024-03-23T03:46:53.963693+0000","last_active":"2024-03-23T03:46:57.951985+0000","last_peered":"2024-03-23T03:46:57.951985+0000","last_clean":"2024-03-23T03:46:57.951985+0000","last_became_active":"2024-03-23T03:46:53.963435+0000","last_became_peered":"2024-03-23T03:46:53.963435+0000","last_unstale":"2024-03-23T03:46:57.951985+0000","last_undegraded":"2024-03-23T03:46:57.951985+0000","last_fullsized":"2024-03-23T03:46:57.951985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:51.878165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.544002+0000","last_change":"2024-03-23T03:46:58.018608+0000","last_active":"2024-03-23T03:46:59.544002+0000","last_peered":"2024-03-23T03:46:59.544002+0000","last_clean":"2024-03-23T03:46:59.544002+0000","last_became_active":"2024-03-23T03:46:58.018216+0000","last_became_peered":"2024-03-23T03:46:58.018216+0000","last_unstale":"2024-03-23T03:46:59.544002+0000","last_undegraded":"2024-03-23T03:46:59.544002+0000","last_fullsized":"2024-03-23T03:46:59.544002+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:41:36.624562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952201+0000","last_change":"2024-03-23T03:46:53.462106+0000","last_active":"2024-03-23T03:46:57.952201+0000","last_peered":"2024-03-23T03:46:57.952201+0000","last_clean":"2024-03-23T03:46:57.952201+0000","last_became_active":"2024-03-23T03:46:53.461757+0000","last_became_peered":"2024-03-23T03:46:53.461757+0000","last_unstale":"2024-03-23T03:46:57.952201+0000","last_undegraded":"2024-03-23T03:46:57.952201+0000","last_fullsized":"2024-03-23T03:46:57.952201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:20:09.953919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.954319+0000","last_change":"2024-03-23T03:46:53.959076+0000","last_active":"2024-03-23T03:46:57.954319+0000","last_peered":"2024-03-23T03:46:57.954319+0000","last_clean":"2024-03-23T03:46:57.954319+0000","last_became_active":"2024-03-23T03:46:53.958351+0000","last_became_peered":"2024-03-23T03:46:53.958351+0000","last_unstale":"2024-03-23T03:46:57.954319+0000","last_undegraded":"2024-03-23T03:46:57.954319+0000","last_fullsized":"2024-03-23T03:46:57.954319+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:55.749189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.423330+0000","last_change":"2024-03-23T03:46:57.979717+0000","last_active":"2024-03-23T03:46:58.423330+0000","last_peered":"2024-03-23T03:46:58.423330+0000","last_clean":"2024-03-23T03:46:58.423330+0000","last_became_active":"2024-03-23T03:46:57.975153+0000","last_became_peered":"2024-03-23T03:46:57.975153+0000","last_unstale":"2024-03-23T03:46:58.423330+0000","last_undegraded":"2024-03-23T03:46:58.423330+0000","last_fullsized":"2024-03-23T03:46:58.423330+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:24:30.082562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017773+0000","last_change":"2024-03-23T03:46:53.480716+0000","last_active":"2024-03-23T03:46:58.017773+0000","last_peered":"2024-03-23T03:46:58.017773+0000","last_clean":"2024-03-23T03:46:58.017773+0000","last_became_active":"2024-03-23T03:46:53.479510+0000","last_became_peered":"2024-03-23T03:46:53.479510+0000","last_unstale":"2024-03-23T03:46:58.017773+0000","last_undegraded":"2024-03-23T03:46:58.017773+0000","last_fullsized":"2024-03-23T03:46:58.017773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:40:20.814830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970572+0000","last_change":"2024-03-23T03:46:53.947108+0000","last_active":"2024-03-23T03:46:57.970572+0000","last_peered":"2024-03-23T03:46:57.970572+0000","last_clean":"2024-03-23T03:46:57.970572+0000","last_became_active":"2024-03-23T03:46:53.946415+0000","last_became_peered":"2024-03-23T03:46:53.946415+0000","last_unstale":"2024-03-23T03:46:57.970572+0000","last_undegraded":"2024-03-23T03:46:57.970572+0000","last_fullsized":"2024-03-23T03:46:57.970572+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:38:13.205855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.426555+0000","last_change":"2024-03-23T03:46:57.968557+0000","last_active":"2024-03-23T03:46:58.426555+0000","last_peered":"2024-03-23T03:46:58.426555+0000","last_clean":"2024-03-23T03:46:58.426555+0000","last_became_active":"2024-03-23T03:46:57.968085+0000","last_became_peered":"2024-03-23T03:46:57.968085+0000","last_unstale":"2024-03-23T03:46:58.426555+0000","last_undegraded":"2024-03-23T03:46:58.426555+0000","last_fullsized":"2024-03-23T03:46:58.426555+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:26:09.015758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.014925+0000","last_change":"2024-03-23T03:46:53.481389+0000","last_active":"2024-03-23T03:46:58.014925+0000","last_peered":"2024-03-23T03:46:58.014925+0000","last_clean":"2024-03-23T03:46:58.014925+0000","last_became_active":"2024-03-23T03:46:53.481119+0000","last_became_peered":"2024-03-23T03:46:53.481119+0000","last_unstale":"2024-03-23T03:46:58.014925+0000","last_undegraded":"2024-03-23T03:46:58.014925+0000","last_fullsized":"2024-03-23T03:46:58.014925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:23:34.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970863+0000","last_change":"2024-03-23T03:46:53.952464+0000","last_active":"2024-03-23T03:46:57.970863+0000","last_peered":"2024-03-23T03:46:57.970863+0000","last_clean":"2024-03-23T03:46:57.970863+0000","last_became_active":"2024-03-23T03:46:53.951343+0000","last_became_peered":"2024-03-23T03:46:53.951343+0000","last_unstale":"2024-03-23T03:46:57.970863+0000","last_undegraded":"2024-03-23T03:46:57.970863+0000","last_fullsized":"2024-03-23T03:46:57.970863+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:35:49.855085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.423616+0000","last_change":"2024-03-23T03:46:57.974943+0000","last_active":"2024-03-23T03:46:58.423616+0000","last_peered":"2024-03-23T03:46:58.423616+0000","last_clean":"2024-03-23T03:46:58.423616+0000","last_became_active":"2024-03-23T03:46:57.974037+0000","last_became_peered":"2024-03-23T03:46:57.974037+0000","last_unstale":"2024-03-23T03:46:58.423616+0000","last_undegraded":"2024-03-23T03:46:58.423616+0000","last_fullsized":"2024-03-23T03:46:58.423616+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:24:17.331178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952188+0000","last_change":"2024-03-23T03:46:53.961283+0000","last_active":"2024-03-23T03:46:57.952188+0000","last_peered":"2024-03-23T03:46:57.952188+0000","last_clean":"2024-03-23T03:46:57.952188+0000","last_became_active":"2024-03-23T03:46:53.961004+0000","last_became_peered":"2024-03-23T03:46:53.961004+0000","last_unstale":"2024-03-23T03:46:57.952188+0000","last_undegraded":"2024-03-23T03:46:57.952188+0000","last_fullsized":"2024-03-23T03:46:57.952188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:30:06.938168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017032+0000","last_change":"2024-03-23T03:46:53.480802+0000","last_active":"2024-03-23T03:46:58.017032+0000","last_peered":"2024-03-23T03:46:58.017032+0000","last_clean":"2024-03-23T03:46:58.017032+0000","last_became_active":"2024-03-23T03:46:53.480500+0000","last_became_peered":"2024-03-23T03:46:53.480500+0000","last_unstale":"2024-03-23T03:46:58.017032+0000","last_undegraded":"2024-03-23T03:46:58.017032+0000","last_fullsized":"2024-03-23T03:46:58.017032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:38:29.605938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.424746+0000","last_change":"2024-03-23T03:46:57.972616+0000","last_active":"2024-03-23T03:46:58.424746+0000","last_peered":"2024-03-23T03:46:58.424746+0000","last_clean":"2024-03-23T03:46:58.424746+0000","last_became_active":"2024-03-23T03:46:57.972028+0000","last_became_peered":"2024-03-23T03:46:57.972028+0000","last_unstale":"2024-03-23T03:46:58.424746+0000","last_undegraded":"2024-03-23T03:46:58.424746+0000","last_fullsized":"2024-03-23T03:46:58.424746+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:55:53.867281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952016+0000","last_change":"2024-03-23T03:46:53.960010+0000","last_active":"2024-03-23T03:46:57.952016+0000","last_peered":"2024-03-23T03:46:57.952016+0000","last_clean":"2024-03-23T03:46:57.952016+0000","last_became_active":"2024-03-23T03:46:53.959761+0000","last_became_peered":"2024-03-23T03:46:53.959761+0000","last_unstale":"2024-03-23T03:46:57.952016+0000","last_undegraded":"2024-03-23T03:46:57.952016+0000","last_fullsized":"2024-03-23T03:46:57.952016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:49.329280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.946127+0000","last_change":"2024-03-23T03:46:53.460246+0000","last_active":"2024-03-23T03:46:57.946127+0000","last_peered":"2024-03-23T03:46:57.946127+0000","last_clean":"2024-03-23T03:46:57.946127+0000","last_became_active":"2024-03-23T03:46:53.459900+0000","last_became_peered":"2024-03-23T03:46:53.459900+0000","last_unstale":"2024-03-23T03:46:57.946127+0000","last_undegraded":"2024-03-23T03:46:57.946127+0000","last_fullsized":"2024-03-23T03:46:57.946127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:23:34.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.424823+0000","last_change":"2024-03-23T03:46:57.976006+0000","last_active":"2024-03-23T03:46:58.424823+0000","last_peered":"2024-03-23T03:46:58.424823+0000","last_clean":"2024-03-23T03:46:58.424823+0000","last_became_active":"2024-03-23T03:46:57.975320+0000","last_became_peered":"2024-03-23T03:46:57.975320+0000","last_unstale":"2024-03-23T03:46:58.424823+0000","last_undegraded":"2024-03-23T03:46:58.424823+0000","last_fullsized":"2024-03-23T03:46:58.424823+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:54.573575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.960391+0000","last_change":"2024-03-23T03:46:53.954675+0000","last_active":"2024-03-23T03:46:57.960391+0000","last_peered":"2024-03-23T03:46:57.960391+0000","last_clean":"2024-03-23T03:46:57.960391+0000","last_became_active":"2024-03-23T03:46:53.954433+0000","last_became_peered":"2024-03-23T03:46:53.954433+0000","last_unstale":"2024-03-23T03:46:57.960391+0000","last_undegraded":"2024-03-23T03:46:57.960391+0000","last_fullsized":"2024-03-23T03:46:57.960391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:27:03.917141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943505+0000","last_change":"2024-03-23T03:46:53.463665+0000","last_active":"2024-03-23T03:46:57.943505+0000","last_peered":"2024-03-23T03:46:57.943505+0000","last_clean":"2024-03-23T03:46:57.943505+0000","last_became_active":"2024-03-23T03:46:53.463379+0000","last_became_peered":"2024-03-23T03:46:53.463379+0000","last_unstale":"2024-03-23T03:46:57.943505+0000","last_undegraded":"2024-03-23T03:46:57.943505+0000","last_fullsized":"2024-03-23T03:46:57.943505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:39:13.578639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545667+0000","last_change":"2024-03-23T03:46:58.017005+0000","last_active":"2024-03-23T03:46:59.545667+0000","last_peered":"2024-03-23T03:46:59.545667+0000","last_clean":"2024-03-23T03:46:59.545667+0000","last_became_active":"2024-03-23T03:46:58.016393+0000","last_became_peered":"2024-03-23T03:46:58.016393+0000","last_unstale":"2024-03-23T03:46:59.545667+0000","last_undegraded":"2024-03-23T03:46:59.545667+0000","last_fullsized":"2024-03-23T03:46:59.545667+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:54:23.785439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.959120+0000","last_change":"2024-03-23T03:46:53.941019+0000","last_active":"2024-03-23T03:46:57.959120+0000","last_peered":"2024-03-23T03:46:57.959120+0000","last_clean":"2024-03-23T03:46:57.959120+0000","last_became_active":"2024-03-23T03:46:53.940265+0000","last_became_peered":"2024-03-23T03:46:53.940265+0000","last_unstale":"2024-03-23T03:46:57.959120+0000","last_undegraded":"2024-03-23T03:46:57.959120+0000","last_fullsized":"2024-03-23T03:46:57.959120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:04:08.642972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.012311+0000","last_change":"2024-03-23T03:46:53.471334+0000","last_active":"2024-03-23T03:46:58.012311+0000","last_peered":"2024-03-23T03:46:58.012311+0000","last_clean":"2024-03-23T03:46:58.012311+0000","last_became_active":"2024-03-23T03:46:53.470996+0000","last_became_peered":"2024-03-23T03:46:53.470996+0000","last_unstale":"2024-03-23T03:46:58.012311+0000","last_undegraded":"2024-03-23T03:46:58.012311+0000","last_fullsized":"2024-03-23T03:46:58.012311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:55.074632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.425416+0000","last_change":"2024-03-23T03:46:57.975950+0000","last_active":"2024-03-23T03:46:58.425416+0000","last_peered":"2024-03-23T03:46:58.425416+0000","last_clean":"2024-03-23T03:46:58.425416+0000","last_became_active":"2024-03-23T03:46:57.975504+0000","last_became_peered":"2024-03-23T03:46:57.975504+0000","last_unstale":"2024-03-23T03:46:58.425416+0000","last_undegraded":"2024-03-23T03:46:58.425416+0000","last_fullsized":"2024-03-23T03:46:58.425416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:21.492278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.955743+0000","last_change":"2024-03-23T03:46:53.957169+0000","last_active":"2024-03-23T03:46:57.955743+0000","last_peered":"2024-03-23T03:46:57.955743+0000","last_clean":"2024-03-23T03:46:57.955743+0000","last_became_active":"2024-03-23T03:46:53.956617+0000","last_became_peered":"2024-03-23T03:46:53.956617+0000","last_unstale":"2024-03-23T03:46:57.955743+0000","last_undegraded":"2024-03-23T03:46:57.955743+0000","last_fullsized":"2024-03-23T03:46:57.955743+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:50:57.246940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.953730+0000","last_change":"2024-03-23T03:46:53.475395+0000","last_active":"2024-03-23T03:46:57.953730+0000","last_peered":"2024-03-23T03:46:57.953730+0000","last_clean":"2024-03-23T03:46:57.953730+0000","last_became_active":"2024-03-23T03:46:53.475130+0000","last_became_peered":"2024-03-23T03:46:53.475130+0000","last_unstale":"2024-03-23T03:46:57.953730+0000","last_undegraded":"2024-03-23T03:46:57.953730+0000","last_fullsized":"2024-03-23T03:46:57.953730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:19:02.377973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.423957+0000","last_change":"2024-03-23T03:46:58.019438+0000","last_active":"2024-03-23T03:46:58.423957+0000","last_peered":"2024-03-23T03:46:58.423957+0000","last_clean":"2024-03-23T03:46:58.423957+0000","last_became_active":"2024-03-23T03:46:58.019018+0000","last_became_peered":"2024-03-23T03:46:58.019018+0000","last_unstale":"2024-03-23T03:46:58.423957+0000","last_undegraded":"2024-03-23T03:46:58.423957+0000","last_fullsized":"2024-03-23T03:46:58.423957+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:53:39.544425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.971959+0000","last_change":"2024-03-23T03:46:53.950103+0000","last_active":"2024-03-23T03:46:57.971959+0000","last_peered":"2024-03-23T03:46:57.971959+0000","last_clean":"2024-03-23T03:46:57.971959+0000","last_became_active":"2024-03-23T03:46:53.947870+0000","last_became_peered":"2024-03-23T03:46:53.947870+0000","last_unstale":"2024-03-23T03:46:57.971959+0000","last_undegraded":"2024-03-23T03:46:57.971959+0000","last_fullsized":"2024-03-23T03:46:57.971959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:56:29.065234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943640+0000","last_change":"2024-03-23T03:46:53.476543+0000","last_active":"2024-03-23T03:46:57.943640+0000","last_peered":"2024-03-23T03:46:57.943640+0000","last_clean":"2024-03-23T03:46:57.943640+0000","last_became_active":"2024-03-23T03:46:53.476227+0000","last_became_peered":"2024-03-23T03:46:53.476227+0000","last_unstale":"2024-03-23T03:46:57.943640+0000","last_undegraded":"2024-03-23T03:46:57.943640+0000","last_fullsized":"2024-03-23T03:46:57.943640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:52:38.737363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.543994+0000","last_change":"2024-03-23T03:46:58.013935+0000","last_active":"2024-03-23T03:46:59.543994+0000","last_peered":"2024-03-23T03:46:59.543994+0000","last_clean":"2024-03-23T03:46:59.543994+0000","last_became_active":"2024-03-23T03:46:58.012631+0000","last_became_peered":"2024-03-23T03:46:58.012631+0000","last_unstale":"2024-03-23T03:46:59.543994+0000","last_undegraded":"2024-03-23T03:46:59.543994+0000","last_fullsized":"2024-03-23T03:46:59.543994+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:42:09.297861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.950778+0000","last_change":"2024-03-23T03:46:53.945184+0000","last_active":"2024-03-23T03:46:57.950778+0000","last_peered":"2024-03-23T03:46:57.950778+0000","last_clean":"2024-03-23T03:46:57.950778+0000","last_became_active":"2024-03-23T03:46:53.943550+0000","last_became_peered":"2024-03-23T03:46:53.943550+0000","last_unstale":"2024-03-23T03:46:57.950778+0000","last_undegraded":"2024-03-23T03:46:57.950778+0000","last_fullsized":"2024-03-23T03:46:57.950778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:01:31.453936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.018003+0000","last_change":"2024-03-23T03:46:53.472577+0000","last_active":"2024-03-23T03:46:58.018003+0000","last_peered":"2024-03-23T03:46:58.018003+0000","last_clean":"2024-03-23T03:46:58.018003+0000","last_became_active":"2024-03-23T03:46:53.472218+0000","last_became_peered":"2024-03-23T03:46:53.472218+0000","last_unstale":"2024-03-23T03:46:58.018003+0000","last_undegraded":"2024-03-23T03:46:58.018003+0000","last_fullsized":"2024-03-23T03:46:58.018003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:21:03.254658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411414+0000","last_change":"2024-03-23T03:46:58.013198+0000","last_active":"2024-03-23T03:46:59.411414+0000","last_peered":"2024-03-23T03:46:59.411414+0000","last_clean":"2024-03-23T03:46:59.411414+0000","last_became_active":"2024-03-23T03:46:58.012474+0000","last_became_peered":"2024-03-23T03:46:58.012474+0000","last_unstale":"2024-03-23T03:46:59.411414+0000","last_undegraded":"2024-03-23T03:46:59.411414+0000","last_fullsized":"2024-03-23T03:46:59.411414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:22:28.017991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970873+0000","last_change":"2024-03-23T03:46:53.955218+0000","last_active":"2024-03-23T03:46:57.970873+0000","last_peered":"2024-03-23T03:46:57.970873+0000","last_clean":"2024-03-23T03:46:57.970873+0000","last_became_active":"2024-03-23T03:46:53.954977+0000","last_became_peered":"2024-03-23T03:46:53.954977+0000","last_unstale":"2024-03-23T03:46:57.970873+0000","last_undegraded":"2024-03-23T03:46:57.970873+0000","last_fullsized":"2024-03-23T03:46:57.970873+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:12.278522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.947128+0000","last_change":"2024-03-23T03:46:53.456663+0000","last_active":"2024-03-23T03:46:57.947128+0000","last_peered":"2024-03-23T03:46:57.947128+0000","last_clean":"2024-03-23T03:46:57.947128+0000","last_became_active":"2024-03-23T03:46:53.456393+0000","last_became_peered":"2024-03-23T03:46:53.456393+0000","last_unstale":"2024-03-23T03:46:57.947128+0000","last_undegraded":"2024-03-23T03:46:57.947128+0000","last_fullsized":"2024-03-23T03:46:57.947128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:38:29.605938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.544795+0000","last_change":"2024-03-23T03:46:57.975493+0000","last_active":"2024-03-23T03:46:59.544795+0000","last_peered":"2024-03-23T03:46:59.544795+0000","last_clean":"2024-03-23T03:46:59.544795+0000","last_became_active":"2024-03-23T03:46:57.974943+0000","last_became_peered":"2024-03-23T03:46:57.974943+0000","last_unstale":"2024-03-23T03:46:59.544795+0000","last_undegraded":"2024-03-23T03:46:59.544795+0000","last_fullsized":"2024-03-23T03:46:59.544795+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:56.623233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.960709+0000","last_change":"2024-03-23T03:46:53.963022+0000","last_active":"2024-03-23T03:46:57.960709+0000","last_peered":"2024-03-23T03:46:57.960709+0000","last_clean":"2024-03-23T03:46:57.960709+0000","last_became_active":"2024-03-23T03:46:53.962270+0000","last_became_peered":"2024-03-23T03:46:53.962270+0000","last_unstale":"2024-03-23T03:46:57.960709+0000","last_undegraded":"2024-03-23T03:46:57.960709+0000","last_fullsized":"2024-03-23T03:46:57.960709+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:55:23.986885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970281+0000","last_change":"2024-03-23T03:46:53.456217+0000","last_active":"2024-03-23T03:46:57.970281+0000","last_peered":"2024-03-23T03:46:57.970281+0000","last_clean":"2024-03-23T03:46:57.970281+0000","last_became_active":"2024-03-23T03:46:53.455825+0000","last_became_peered":"2024-03-23T03:46:53.455825+0000","last_unstale":"2024-03-23T03:46:57.970281+0000","last_undegraded":"2024-03-23T03:46:57.970281+0000","last_fullsized":"2024-03-23T03:46:57.970281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:20:09.953919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.544875+0000","last_change":"2024-03-23T03:46:57.978215+0000","last_active":"2024-03-23T03:46:59.544875+0000","last_peered":"2024-03-23T03:46:59.544875+0000","last_clean":"2024-03-23T03:46:59.544875+0000","last_became_active":"2024-03-23T03:46:57.977737+0000","last_became_peered":"2024-03-23T03:46:57.977737+0000","last_unstale":"2024-03-23T03:46:59.544875+0000","last_undegraded":"2024-03-23T03:46:59.544875+0000","last_fullsized":"2024-03-23T03:46:59.544875+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:09:10.919635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943234+0000","last_change":"2024-03-23T03:46:53.949224+0000","last_active":"2024-03-23T03:46:57.943234+0000","last_peered":"2024-03-23T03:46:57.943234+0000","last_clean":"2024-03-23T03:46:57.943234+0000","last_became_active":"2024-03-23T03:46:53.948869+0000","last_became_peered":"2024-03-23T03:46:53.948869+0000","last_unstale":"2024-03-23T03:46:57.943234+0000","last_undegraded":"2024-03-23T03:46:57.943234+0000","last_fullsized":"2024-03-23T03:46:57.943234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:49:34.782399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.948932+0000","last_change":"2024-03-23T03:46:53.461930+0000","last_active":"2024-03-23T03:46:57.948932+0000","last_peered":"2024-03-23T03:46:57.948932+0000","last_clean":"2024-03-23T03:46:57.948932+0000","last_became_active":"2024-03-23T03:46:53.461500+0000","last_became_peered":"2024-03-23T03:46:53.461500+0000","last_unstale":"2024-03-23T03:46:57.948932+0000","last_undegraded":"2024-03-23T03:46:57.948932+0000","last_fullsized":"2024-03-23T03:46:57.948932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:12:50.664977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545763+0000","last_change":"2024-03-23T03:46:57.978014+0000","last_active":"2024-03-23T03:46:59.545763+0000","last_peered":"2024-03-23T03:46:59.545763+0000","last_clean":"2024-03-23T03:46:59.545763+0000","last_became_active":"2024-03-23T03:46:57.977584+0000","last_became_peered":"2024-03-23T03:46:57.977584+0000","last_unstale":"2024-03-23T03:46:59.545763+0000","last_undegraded":"2024-03-23T03:46:59.545763+0000","last_fullsized":"2024-03-23T03:46:59.545763+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:17:26.245417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.950704+0000","last_change":"2024-03-23T03:46:53.968396+0000","last_active":"2024-03-23T03:46:57.950704+0000","last_peered":"2024-03-23T03:46:57.950704+0000","last_clean":"2024-03-23T03:46:57.950704+0000","last_became_active":"2024-03-23T03:46:53.968127+0000","last_became_peered":"2024-03-23T03:46:53.968127+0000","last_unstale":"2024-03-23T03:46:57.950704+0000","last_undegraded":"2024-03-23T03:46:57.950704+0000","last_fullsized":"2024-03-23T03:46:57.950704+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:56:29.065234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.012423+0000","last_change":"2024-03-23T03:46:53.472587+0000","last_active":"2024-03-23T03:46:58.012423+0000","last_peered":"2024-03-23T03:46:58.012423+0000","last_clean":"2024-03-23T03:46:58.012423+0000","last_became_active":"2024-03-23T03:46:53.472235+0000","last_became_peered":"2024-03-23T03:46:53.472235+0000","last_unstale":"2024-03-23T03:46:58.012423+0000","last_undegraded":"2024-03-23T03:46:58.012423+0000","last_fullsized":"2024-03-23T03:46:58.012423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:56.874824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411831+0000","last_change":"2024-03-23T03:46:58.016199+0000","last_active":"2024-03-23T03:46:59.411831+0000","last_peered":"2024-03-23T03:46:59.411831+0000","last_clean":"2024-03-23T03:46:59.411831+0000","last_became_active":"2024-03-23T03:46:58.015478+0000","last_became_peered":"2024-03-23T03:46:58.015478+0000","last_unstale":"2024-03-23T03:46:59.411831+0000","last_undegraded":"2024-03-23T03:46:59.411831+0000","last_fullsized":"2024-03-23T03:46:59.411831+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:56.623233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951639+0000","last_change":"2024-03-23T03:46:53.955152+0000","last_active":"2024-03-23T03:46:57.951639+0000","last_peered":"2024-03-23T03:46:57.951639+0000","last_clean":"2024-03-23T03:46:57.951639+0000","last_became_active":"2024-03-23T03:46:53.953527+0000","last_became_peered":"2024-03-23T03:46:53.953527+0000","last_unstale":"2024-03-23T03:46:57.951639+0000","last_undegraded":"2024-03-23T03:46:57.951639+0000","last_fullsized":"2024-03-23T03:46:57.951639+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:01:31.453936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970187+0000","last_change":"2024-03-23T03:46:53.462993+0000","last_active":"2024-03-23T03:46:57.970187+0000","last_peered":"2024-03-23T03:46:57.970187+0000","last_clean":"2024-03-23T03:46:57.970187+0000","last_became_active":"2024-03-23T03:46:53.462438+0000","last_became_peered":"2024-03-23T03:46:53.462438+0000","last_unstale":"2024-03-23T03:46:57.970187+0000","last_undegraded":"2024-03-23T03:46:57.970187+0000","last_fullsized":"2024-03-23T03:46:57.970187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:52:38.737363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":167,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.012529+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:46:58.012529+0000","last_peered":"2024-03-23T03:46:58.012529+0000","last_clean":"2024-03-23T03:46:58.012529+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:46:58.012529+0000","last_undegraded":"2024-03-23T03:46:58.012529+0000","last_fullsized":"2024-03-23T03:46:58.012529+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.547057+0000","last_change":"2024-03-23T03:46:58.018456+0000","last_active":"2024-03-23T03:46:59.547057+0000","last_peered":"2024-03-23T03:46:59.547057+0000","last_clean":"2024-03-23T03:46:59.547057+0000","last_became_active":"2024-03-23T03:46:58.018035+0000","last_became_peered":"2024-03-23T03:46:58.018035+0000","last_unstale":"2024-03-23T03:46:59.547057+0000","last_undegraded":"2024-03-23T03:46:59.547057+0000","last_fullsized":"2024-03-23T03:46:59.547057+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:44:17.122381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017645+0000","last_change":"2024-03-23T03:46:53.970202+0000","last_active":"2024-03-23T03:46:58.017645+0000","last_peered":"2024-03-23T03:46:58.017645+0000","last_clean":"2024-03-23T03:46:58.017645+0000","last_became_active":"2024-03-23T03:46:53.969976+0000","last_became_peered":"2024-03-23T03:46:53.969976+0000","last_unstale":"2024-03-23T03:46:58.017645+0000","last_undegraded":"2024-03-23T03:46:58.017645+0000","last_fullsized":"2024-03-23T03:46:58.017645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:59.541399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.947920+0000","last_change":"2024-03-23T03:46:53.478374+0000","last_active":"2024-03-23T03:46:57.947920+0000","last_peered":"2024-03-23T03:46:57.947920+0000","last_clean":"2024-03-23T03:46:57.947920+0000","last_became_active":"2024-03-23T03:46:53.478063+0000","last_became_peered":"2024-03-23T03:46:53.478063+0000","last_unstale":"2024-03-23T03:46:57.947920+0000","last_undegraded":"2024-03-23T03:46:57.947920+0000","last_fullsized":"2024-03-23T03:46:57.947920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:28:03.413525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.546398+0000","last_change":"2024-03-23T03:46:58.013147+0000","last_active":"2024-03-23T03:46:59.546398+0000","last_peered":"2024-03-23T03:46:59.546398+0000","last_clean":"2024-03-23T03:46:59.546398+0000","last_became_active":"2024-03-23T03:46:58.012728+0000","last_became_peered":"2024-03-23T03:46:58.012728+0000","last_unstale":"2024-03-23T03:46:59.546398+0000","last_undegraded":"2024-03-23T03:46:59.546398+0000","last_fullsized":"2024-03-23T03:46:59.546398+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:28:49.474199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970654+0000","last_change":"2024-03-23T03:46:53.956675+0000","last_active":"2024-03-23T03:46:57.970654+0000","last_peered":"2024-03-23T03:46:57.970654+0000","last_clean":"2024-03-23T03:46:57.970654+0000","last_became_active":"2024-03-23T03:46:53.956420+0000","last_became_peered":"2024-03-23T03:46:53.956420+0000","last_unstale":"2024-03-23T03:46:57.970654+0000","last_undegraded":"2024-03-23T03:46:57.970654+0000","last_fullsized":"2024-03-23T03:46:57.970654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:16:39.131183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943199+0000","last_change":"2024-03-23T03:46:53.467369+0000","last_active":"2024-03-23T03:46:57.943199+0000","last_peered":"2024-03-23T03:46:57.943199+0000","last_clean":"2024-03-23T03:46:57.943199+0000","last_became_active":"2024-03-23T03:46:53.465916+0000","last_became_peered":"2024-03-23T03:46:53.465916+0000","last_unstale":"2024-03-23T03:46:57.943199+0000","last_undegraded":"2024-03-23T03:46:57.943199+0000","last_fullsized":"2024-03-23T03:46:57.943199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:56.874824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411552+0000","last_change":"2024-03-23T03:46:57.974919+0000","last_active":"2024-03-23T03:46:59.411552+0000","last_peered":"2024-03-23T03:46:59.411552+0000","last_clean":"2024-03-23T03:46:59.411552+0000","last_became_active":"2024-03-23T03:46:57.972661+0000","last_became_peered":"2024-03-23T03:46:57.972661+0000","last_unstale":"2024-03-23T03:46:59.411552+0000","last_undegraded":"2024-03-23T03:46:59.411552+0000","last_fullsized":"2024-03-23T03:46:59.411552+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:27:37.992019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.018541+0000","last_change":"2024-03-23T03:46:53.965749+0000","last_active":"2024-03-23T03:46:58.018541+0000","last_peered":"2024-03-23T03:46:58.018541+0000","last_clean":"2024-03-23T03:46:58.018541+0000","last_became_active":"2024-03-23T03:46:53.965362+0000","last_became_peered":"2024-03-23T03:46:53.965362+0000","last_unstale":"2024-03-23T03:46:58.018541+0000","last_undegraded":"2024-03-23T03:46:58.018541+0000","last_fullsized":"2024-03-23T03:46:58.018541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:38:13.205855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.948005+0000","last_change":"2024-03-23T03:46:53.460818+0000","last_active":"2024-03-23T03:46:57.948005+0000","last_peered":"2024-03-23T03:46:57.948005+0000","last_clean":"2024-03-23T03:46:57.948005+0000","last_became_active":"2024-03-23T03:46:53.460547+0000","last_became_peered":"2024-03-23T03:46:53.460547+0000","last_unstale":"2024-03-23T03:46:57.948005+0000","last_undegraded":"2024-03-23T03:46:57.948005+0000","last_fullsized":"2024-03-23T03:46:57.948005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:49:24.952855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.424397+0000","last_change":"2024-03-23T03:46:58.014145+0000","last_active":"2024-03-23T03:46:58.424397+0000","last_peered":"2024-03-23T03:46:58.424397+0000","last_clean":"2024-03-23T03:46:58.424397+0000","last_became_active":"2024-03-23T03:46:58.013649+0000","last_became_peered":"2024-03-23T03:46:58.013649+0000","last_unstale":"2024-03-23T03:46:58.424397+0000","last_undegraded":"2024-03-23T03:46:58.424397+0000","last_fullsized":"2024-03-23T03:46:58.424397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:56.623233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017437+0000","last_change":"2024-03-23T03:46:53.966787+0000","last_active":"2024-03-23T03:46:58.017437+0000","last_peered":"2024-03-23T03:46:58.017437+0000","last_clean":"2024-03-23T03:46:58.017437+0000","last_became_active":"2024-03-23T03:46:53.966135+0000","last_became_peered":"2024-03-23T03:46:53.966135+0000","last_unstale":"2024-03-23T03:46:58.017437+0000","last_undegraded":"2024-03-23T03:46:58.017437+0000","last_fullsized":"2024-03-23T03:46:58.017437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:12.278522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.964838+0000","last_change":"2024-03-23T03:46:53.473134+0000","last_active":"2024-03-23T03:46:57.964838+0000","last_peered":"2024-03-23T03:46:57.964838+0000","last_clean":"2024-03-23T03:46:57.964838+0000","last_became_active":"2024-03-23T03:46:53.472902+0000","last_became_peered":"2024-03-23T03:46:53.472902+0000","last_unstale":"2024-03-23T03:46:57.964838+0000","last_undegraded":"2024-03-23T03:46:57.964838+0000","last_fullsized":"2024-03-23T03:46:57.964838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:42:16.552281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.423299+0000","last_change":"2024-03-23T03:46:57.976918+0000","last_active":"2024-03-23T03:46:58.423299+0000","last_peered":"2024-03-23T03:46:58.423299+0000","last_clean":"2024-03-23T03:46:58.423299+0000","last_became_active":"2024-03-23T03:46:57.976542+0000","last_became_peered":"2024-03-23T03:46:57.976542+0000","last_unstale":"2024-03-23T03:46:58.423299+0000","last_undegraded":"2024-03-23T03:46:58.423299+0000","last_fullsized":"2024-03-23T03:46:58.423299+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:49:33.207155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017030+0000","last_change":"2024-03-23T03:46:53.964676+0000","last_active":"2024-03-23T03:46:58.017030+0000","last_peered":"2024-03-23T03:46:58.017030+0000","last_clean":"2024-03-23T03:46:58.017030+0000","last_became_active":"2024-03-23T03:46:53.964382+0000","last_became_peered":"2024-03-23T03:46:53.964382+0000","last_unstale":"2024-03-23T03:46:58.017030+0000","last_undegraded":"2024-03-23T03:46:58.017030+0000","last_fullsized":"2024-03-23T03:46:58.017030+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:34:41.509208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.947240+0000","last_change":"2024-03-23T03:46:53.459242+0000","last_active":"2024-03-23T03:46:57.947240+0000","last_peered":"2024-03-23T03:46:57.947240+0000","last_clean":"2024-03-23T03:46:57.947240+0000","last_became_active":"2024-03-23T03:46:53.458136+0000","last_became_peered":"2024-03-23T03:46:53.458136+0000","last_unstale":"2024-03-23T03:46:57.947240+0000","last_undegraded":"2024-03-23T03:46:57.947240+0000","last_fullsized":"2024-03-23T03:46:57.947240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:42:13.208524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411874+0000","last_change":"2024-03-23T03:46:57.968727+0000","last_active":"2024-03-23T03:46:59.411874+0000","last_peered":"2024-03-23T03:46:59.411874+0000","last_clean":"2024-03-23T03:46:59.411874+0000","last_became_active":"2024-03-23T03:46:57.968336+0000","last_became_peered":"2024-03-23T03:46:57.968336+0000","last_unstale":"2024-03-23T03:46:59.411874+0000","last_undegraded":"2024-03-23T03:46:59.411874+0000","last_fullsized":"2024-03-23T03:46:59.411874+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:17:26.245417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951907+0000","last_change":"2024-03-23T03:46:53.967303+0000","last_active":"2024-03-23T03:46:57.951907+0000","last_peered":"2024-03-23T03:46:57.951907+0000","last_clean":"2024-03-23T03:46:57.951907+0000","last_became_active":"2024-03-23T03:46:53.966903+0000","last_became_peered":"2024-03-23T03:46:53.966903+0000","last_unstale":"2024-03-23T03:46:57.951907+0000","last_undegraded":"2024-03-23T03:46:57.951907+0000","last_fullsized":"2024-03-23T03:46:57.951907+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:18:17.089475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951906+0000","last_change":"2024-03-23T03:46:53.479049+0000","last_active":"2024-03-23T03:46:57.951906+0000","last_peered":"2024-03-23T03:46:57.951906+0000","last_clean":"2024-03-23T03:46:57.951906+0000","last_became_active":"2024-03-23T03:46:53.478779+0000","last_became_peered":"2024-03-23T03:46:53.478779+0000","last_unstale":"2024-03-23T03:46:57.951906+0000","last_undegraded":"2024-03-23T03:46:57.951906+0000","last_fullsized":"2024-03-23T03:46:57.951906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:28:03.413525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.547121+0000","last_change":"2024-03-23T03:46:58.018779+0000","last_active":"2024-03-23T03:46:59.547121+0000","last_peered":"2024-03-23T03:46:59.547121+0000","last_clean":"2024-03-23T03:46:59.547121+0000","last_became_active":"2024-03-23T03:46:58.018342+0000","last_became_peered":"2024-03-23T03:46:58.018342+0000","last_unstale":"2024-03-23T03:46:59.547121+0000","last_undegraded":"2024-03-23T03:46:59.547121+0000","last_fullsized":"2024-03-23T03:46:59.547121+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:49:30.580961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.016869+0000","last_change":"2024-03-23T03:46:53.966087+0000","last_active":"2024-03-23T03:46:58.016869+0000","last_peered":"2024-03-23T03:46:58.016869+0000","last_clean":"2024-03-23T03:46:58.016869+0000","last_became_active":"2024-03-23T03:46:53.965556+0000","last_became_peered":"2024-03-23T03:46:53.965556+0000","last_unstale":"2024-03-23T03:46:58.016869+0000","last_undegraded":"2024-03-23T03:46:58.016869+0000","last_fullsized":"2024-03-23T03:46:58.016869+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:35:49.855085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970252+0000","last_change":"2024-03-23T03:46:53.468499+0000","last_active":"2024-03-23T03:46:57.970252+0000","last_peered":"2024-03-23T03:46:57.970252+0000","last_clean":"2024-03-23T03:46:57.970252+0000","last_became_active":"2024-03-23T03:46:53.466565+0000","last_became_peered":"2024-03-23T03:46:53.466565+0000","last_unstale":"2024-03-23T03:46:57.970252+0000","last_undegraded":"2024-03-23T03:46:57.970252+0000","last_fullsized":"2024-03-23T03:46:57.970252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:38:29.605938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.546719+0000","last_change":"2024-03-23T03:46:58.013721+0000","last_active":"2024-03-23T03:46:59.546719+0000","last_peered":"2024-03-23T03:46:59.546719+0000","last_clean":"2024-03-23T03:46:59.546719+0000","last_became_active":"2024-03-23T03:46:58.012503+0000","last_became_peered":"2024-03-23T03:46:58.012503+0000","last_unstale":"2024-03-23T03:46:59.546719+0000","last_undegraded":"2024-03-23T03:46:59.546719+0000","last_fullsized":"2024-03-23T03:46:59.546719+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:54:55.461256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951186+0000","last_change":"2024-03-23T03:46:53.952982+0000","last_active":"2024-03-23T03:46:57.951186+0000","last_peered":"2024-03-23T03:46:57.951186+0000","last_clean":"2024-03-23T03:46:57.951186+0000","last_became_active":"2024-03-23T03:46:53.952707+0000","last_became_peered":"2024-03-23T03:46:53.952707+0000","last_unstale":"2024-03-23T03:46:57.951186+0000","last_undegraded":"2024-03-23T03:46:57.951186+0000","last_fullsized":"2024-03-23T03:46:57.951186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:01:24.719476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952254+0000","last_change":"2024-03-23T03:46:53.472959+0000","last_active":"2024-03-23T03:46:57.952254+0000","last_peered":"2024-03-23T03:46:57.952254+0000","last_clean":"2024-03-23T03:46:57.952254+0000","last_became_active":"2024-03-23T03:46:53.472652+0000","last_became_peered":"2024-03-23T03:46:53.472652+0000","last_unstale":"2024-03-23T03:46:57.952254+0000","last_undegraded":"2024-03-23T03:46:57.952254+0000","last_fullsized":"2024-03-23T03:46:57.952254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:48:00.747717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.426089+0000","last_change":"2024-03-23T03:46:58.019621+0000","last_active":"2024-03-23T03:46:58.426089+0000","last_peered":"2024-03-23T03:46:58.426089+0000","last_clean":"2024-03-23T03:46:58.426089+0000","last_became_active":"2024-03-23T03:46:58.018726+0000","last_became_peered":"2024-03-23T03:46:58.018726+0000","last_unstale":"2024-03-23T03:46:58.426089+0000","last_undegraded":"2024-03-23T03:46:58.426089+0000","last_fullsized":"2024-03-23T03:46:58.426089+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:18:24.184664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017563+0000","last_change":"2024-03-23T03:46:53.954977+0000","last_active":"2024-03-23T03:46:58.017563+0000","last_peered":"2024-03-23T03:46:58.017563+0000","last_clean":"2024-03-23T03:46:58.017563+0000","last_became_active":"2024-03-23T03:46:53.954667+0000","last_became_peered":"2024-03-23T03:46:53.954667+0000","last_unstale":"2024-03-23T03:46:58.017563+0000","last_undegraded":"2024-03-23T03:46:58.017563+0000","last_fullsized":"2024-03-23T03:46:58.017563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:16.232161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:47:04.440126+0000","last_change":"2024-03-23T03:46:53.469497+0000","last_active":"2024-03-23T03:47:04.440126+0000","last_peered":"2024-03-23T03:47:04.440126+0000","last_clean":"2024-03-23T03:47:04.440126+0000","last_became_active":"2024-03-23T03:46:53.469139+0000","last_became_peered":"2024-03-23T03:46:53.469139+0000","last_unstale":"2024-03-23T03:47:04.440126+0000","last_undegraded":"2024-03-23T03:47:04.440126+0000","last_fullsized":"2024-03-23T03:47:04.440126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:42:13.208524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3654,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545062+0000","last_change":"2024-03-23T03:46:57.965959+0000","last_active":"2024-03-23T03:46:59.545062+0000","last_peered":"2024-03-23T03:46:59.545062+0000","last_clean":"2024-03-23T03:46:59.545062+0000","last_became_active":"2024-03-23T03:46:57.965418+0000","last_became_peered":"2024-03-23T03:46:57.965418+0000","last_unstale":"2024-03-23T03:46:59.545062+0000","last_undegraded":"2024-03-23T03:46:59.545062+0000","last_fullsized":"2024-03-23T03:46:59.545062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:57:19.038735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.962037+0000","last_change":"2024-03-23T03:46:53.946234+0000","last_active":"2024-03-23T03:46:57.962037+0000","last_peered":"2024-03-23T03:46:57.962037+0000","last_clean":"2024-03-23T03:46:57.962037+0000","last_became_active":"2024-03-23T03:46:53.945188+0000","last_became_peered":"2024-03-23T03:46:53.945188+0000","last_unstale":"2024-03-23T03:46:57.962037+0000","last_undegraded":"2024-03-23T03:46:57.962037+0000","last_fullsized":"2024-03-23T03:46:57.962037+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:29:34.917375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943105+0000","last_change":"2024-03-23T03:46:53.474846+0000","last_active":"2024-03-23T03:46:57.943105+0000","last_peered":"2024-03-23T03:46:57.943105+0000","last_clean":"2024-03-23T03:46:57.943105+0000","last_became_active":"2024-03-23T03:46:53.474561+0000","last_became_peered":"2024-03-23T03:46:53.474561+0000","last_unstale":"2024-03-23T03:46:57.943105+0000","last_undegraded":"2024-03-23T03:46:57.943105+0000","last_fullsized":"2024-03-23T03:46:57.943105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:40:20.814830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.424404+0000","last_change":"2024-03-23T03:46:57.974187+0000","last_active":"2024-03-23T03:46:58.424404+0000","last_peered":"2024-03-23T03:46:58.424404+0000","last_clean":"2024-03-23T03:46:58.424404+0000","last_became_active":"2024-03-23T03:46:57.973629+0000","last_became_peered":"2024-03-23T03:46:57.973629+0000","last_unstale":"2024-03-23T03:46:58.424404+0000","last_undegraded":"2024-03-23T03:46:58.424404+0000","last_fullsized":"2024-03-23T03:46:58.424404+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:32:46.731709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.954425+0000","last_change":"2024-03-23T03:46:53.958810+0000","last_active":"2024-03-23T03:46:57.954425+0000","last_peered":"2024-03-23T03:46:57.954425+0000","last_clean":"2024-03-23T03:46:57.954425+0000","last_became_active":"2024-03-23T03:46:53.958534+0000","last_became_peered":"2024-03-23T03:46:53.958534+0000","last_unstale":"2024-03-23T03:46:57.954425+0000","last_undegraded":"2024-03-23T03:46:57.954425+0000","last_fullsized":"2024-03-23T03:46:57.954425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:01:31.453936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970435+0000","last_change":"2024-03-23T03:46:53.472164+0000","last_active":"2024-03-23T03:46:57.970435+0000","last_peered":"2024-03-23T03:46:57.970435+0000","last_clean":"2024-03-23T03:46:57.970435+0000","last_became_active":"2024-03-23T03:46:53.471488+0000","last_became_peered":"2024-03-23T03:46:53.471488+0000","last_unstale":"2024-03-23T03:46:57.970435+0000","last_undegraded":"2024-03-23T03:46:57.970435+0000","last_fullsized":"2024-03-23T03:46:57.970435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:56.874824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.544253+0000","last_change":"2024-03-23T03:46:57.967928+0000","last_active":"2024-03-23T03:46:59.544253+0000","last_peered":"2024-03-23T03:46:59.544253+0000","last_clean":"2024-03-23T03:46:59.544253+0000","last_became_active":"2024-03-23T03:46:57.967377+0000","last_became_peered":"2024-03-23T03:46:57.967377+0000","last_unstale":"2024-03-23T03:46:59.544253+0000","last_undegraded":"2024-03-23T03:46:59.544253+0000","last_fullsized":"2024-03-23T03:46:59.544253+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:58:12.948095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970395+0000","last_change":"2024-03-23T03:46:53.954230+0000","last_active":"2024-03-23T03:46:57.970395+0000","last_peered":"2024-03-23T03:46:57.970395+0000","last_clean":"2024-03-23T03:46:57.970395+0000","last_became_active":"2024-03-23T03:46:53.953991+0000","last_became_peered":"2024-03-23T03:46:53.953991+0000","last_unstale":"2024-03-23T03:46:57.970395+0000","last_undegraded":"2024-03-23T03:46:57.970395+0000","last_fullsized":"2024-03-23T03:46:57.970395+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:16.232161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.953457+0000","last_change":"2024-03-23T03:46:53.475516+0000","last_active":"2024-03-23T03:46:57.953457+0000","last_peered":"2024-03-23T03:46:57.953457+0000","last_clean":"2024-03-23T03:46:57.953457+0000","last_became_active":"2024-03-23T03:46:53.475185+0000","last_became_peered":"2024-03-23T03:46:53.475185+0000","last_unstale":"2024-03-23T03:46:57.953457+0000","last_undegraded":"2024-03-23T03:46:57.953457+0000","last_fullsized":"2024-03-23T03:46:57.953457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:05:58.548539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545880+0000","last_change":"2024-03-23T03:46:58.018887+0000","last_active":"2024-03-23T03:46:59.545880+0000","last_peered":"2024-03-23T03:46:59.545880+0000","last_clean":"2024-03-23T03:46:59.545880+0000","last_became_active":"2024-03-23T03:46:58.018514+0000","last_became_peered":"2024-03-23T03:46:58.018514+0000","last_unstale":"2024-03-23T03:46:59.545880+0000","last_undegraded":"2024-03-23T03:46:59.545880+0000","last_fullsized":"2024-03-23T03:46:59.545880+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:51:12.276163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411517+0000","last_change":"2024-03-23T03:46:58.015847+0000","last_active":"2024-03-23T03:46:59.411517+0000","last_peered":"2024-03-23T03:46:59.411517+0000","last_clean":"2024-03-23T03:46:59.411517+0000","last_became_active":"2024-03-23T03:46:58.015471+0000","last_became_peered":"2024-03-23T03:46:58.015471+0000","last_unstale":"2024-03-23T03:46:59.411517+0000","last_undegraded":"2024-03-23T03:46:59.411517+0000","last_fullsized":"2024-03-23T03:46:59.411517+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:34:34.288987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943156+0000","last_change":"2024-03-23T03:46:53.952670+0000","last_active":"2024-03-23T03:46:57.943156+0000","last_peered":"2024-03-23T03:46:57.943156+0000","last_clean":"2024-03-23T03:46:57.943156+0000","last_became_active":"2024-03-23T03:46:53.951520+0000","last_became_peered":"2024-03-23T03:46:53.951520+0000","last_unstale":"2024-03-23T03:46:57.943156+0000","last_undegraded":"2024-03-23T03:46:57.943156+0000","last_fullsized":"2024-03-23T03:46:57.943156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:55.749189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.968564+0000","last_change":"2024-03-23T03:46:53.475560+0000","last_active":"2024-03-23T03:46:57.968564+0000","last_peered":"2024-03-23T03:46:57.968564+0000","last_clean":"2024-03-23T03:46:57.968564+0000","last_became_active":"2024-03-23T03:46:53.468090+0000","last_became_peered":"2024-03-23T03:46:53.468090+0000","last_unstale":"2024-03-23T03:46:57.968564+0000","last_undegraded":"2024-03-23T03:46:57.968564+0000","last_fullsized":"2024-03-23T03:46:57.968564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:23:34.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.548770+0000","last_change":"2024-03-23T03:46:57.974834+0000","last_active":"2024-03-23T03:46:59.548770+0000","last_peered":"2024-03-23T03:46:59.548770+0000","last_clean":"2024-03-23T03:46:59.548770+0000","last_became_active":"2024-03-23T03:46:57.974450+0000","last_became_peered":"2024-03-23T03:46:57.974450+0000","last_unstale":"2024-03-23T03:46:59.548770+0000","last_undegraded":"2024-03-23T03:46:59.548770+0000","last_fullsized":"2024-03-23T03:46:59.548770+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:01.676599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.954004+0000","last_change":"2024-03-23T03:46:53.963896+0000","last_active":"2024-03-23T03:46:57.954004+0000","last_peered":"2024-03-23T03:46:57.954004+0000","last_clean":"2024-03-23T03:46:57.954004+0000","last_became_active":"2024-03-23T03:46:53.963072+0000","last_became_peered":"2024-03-23T03:46:53.963072+0000","last_unstale":"2024-03-23T03:46:57.954004+0000","last_undegraded":"2024-03-23T03:46:57.954004+0000","last_fullsized":"2024-03-23T03:46:57.954004+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:49:34.782399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.961727+0000","last_change":"2024-03-23T03:46:53.475321+0000","last_active":"2024-03-23T03:46:57.961727+0000","last_peered":"2024-03-23T03:46:57.961727+0000","last_clean":"2024-03-23T03:46:57.961727+0000","last_became_active":"2024-03-23T03:46:53.475119+0000","last_became_peered":"2024-03-23T03:46:53.475119+0000","last_unstale":"2024-03-23T03:46:57.961727+0000","last_undegraded":"2024-03-23T03:46:57.961727+0000","last_fullsized":"2024-03-23T03:46:57.961727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:48:00.747717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952537+0000","last_change":"2024-03-23T03:46:53.475209+0000","last_active":"2024-03-23T03:46:57.952537+0000","last_peered":"2024-03-23T03:46:57.952537+0000","last_clean":"2024-03-23T03:46:57.952537+0000","last_became_active":"2024-03-23T03:46:53.474046+0000","last_became_peered":"2024-03-23T03:46:53.474046+0000","last_unstale":"2024-03-23T03:46:57.952537+0000","last_undegraded":"2024-03-23T03:46:57.952537+0000","last_fullsized":"2024-03-23T03:46:57.952537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:04:07.631198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952167+0000","last_change":"2024-03-23T03:46:53.961251+0000","last_active":"2024-03-23T03:46:57.952167+0000","last_peered":"2024-03-23T03:46:57.952167+0000","last_clean":"2024-03-23T03:46:57.952167+0000","last_became_active":"2024-03-23T03:46:53.960887+0000","last_became_peered":"2024-03-23T03:46:53.960887+0000","last_unstale":"2024-03-23T03:46:57.952167+0000","last_undegraded":"2024-03-23T03:46:57.952167+0000","last_fullsized":"2024-03-23T03:46:57.952167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:51.878165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545740+0000","last_change":"2024-03-23T03:46:57.969555+0000","last_active":"2024-03-23T03:46:59.545740+0000","last_peered":"2024-03-23T03:46:59.545740+0000","last_clean":"2024-03-23T03:46:59.545740+0000","last_became_active":"2024-03-23T03:46:57.969163+0000","last_became_peered":"2024-03-23T03:46:57.969163+0000","last_unstale":"2024-03-23T03:46:59.545740+0000","last_undegraded":"2024-03-23T03:46:59.545740+0000","last_fullsized":"2024-03-23T03:46:59.545740+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:09.677468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952350+0000","last_change":"2024-03-23T03:46:53.463135+0000","last_active":"2024-03-23T03:46:57.952350+0000","last_peered":"2024-03-23T03:46:57.952350+0000","last_clean":"2024-03-23T03:46:57.952350+0000","last_became_active":"2024-03-23T03:46:53.462844+0000","last_became_peered":"2024-03-23T03:46:53.462844+0000","last_unstale":"2024-03-23T03:46:57.952350+0000","last_undegraded":"2024-03-23T03:46:57.952350+0000","last_fullsized":"2024-03-23T03:46:57.952350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:39:13.578639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.960303+0000","last_change":"2024-03-23T03:46:53.935183+0000","last_active":"2024-03-23T03:46:57.960303+0000","last_peered":"2024-03-23T03:46:57.960303+0000","last_clean":"2024-03-23T03:46:57.960303+0000","last_became_active":"2024-03-23T03:46:53.934902+0000","last_became_peered":"2024-03-23T03:46:53.934902+0000","last_unstale":"2024-03-23T03:46:57.960303+0000","last_undegraded":"2024-03-23T03:46:57.960303+0000","last_fullsized":"2024-03-23T03:46:57.960303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:19:03.389747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.546902+0000","last_change":"2024-03-23T03:46:57.971524+0000","last_active":"2024-03-23T03:46:59.546902+0000","last_peered":"2024-03-23T03:46:59.546902+0000","last_clean":"2024-03-23T03:46:59.546902+0000","last_became_active":"2024-03-23T03:46:57.967540+0000","last_became_peered":"2024-03-23T03:46:57.967540+0000","last_unstale":"2024-03-23T03:46:59.546902+0000","last_undegraded":"2024-03-23T03:46:59.546902+0000","last_fullsized":"2024-03-23T03:46:59.546902+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:36:58.375101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951620+0000","last_change":"2024-03-23T03:46:53.966166+0000","last_active":"2024-03-23T03:46:57.951620+0000","last_peered":"2024-03-23T03:46:57.951620+0000","last_clean":"2024-03-23T03:46:57.951620+0000","last_became_active":"2024-03-23T03:46:53.965850+0000","last_became_peered":"2024-03-23T03:46:53.965850+0000","last_unstale":"2024-03-23T03:46:57.951620+0000","last_undegraded":"2024-03-23T03:46:57.951620+0000","last_fullsized":"2024-03-23T03:46:57.951620+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:35:38.384483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.948826+0000","last_change":"2024-03-23T03:46:53.954723+0000","last_active":"2024-03-23T03:46:57.948826+0000","last_peered":"2024-03-23T03:46:57.948826+0000","last_clean":"2024-03-23T03:46:57.948826+0000","last_became_active":"2024-03-23T03:46:53.954419+0000","last_became_peered":"2024-03-23T03:46:53.954419+0000","last_unstale":"2024-03-23T03:46:57.948826+0000","last_undegraded":"2024-03-23T03:46:57.948826+0000","last_fullsized":"2024-03-23T03:46:57.948826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:55:23.986885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970164+0000","last_change":"2024-03-23T03:46:53.950337+0000","last_active":"2024-03-23T03:46:57.970164+0000","last_peered":"2024-03-23T03:46:57.970164+0000","last_clean":"2024-03-23T03:46:57.970164+0000","last_became_active":"2024-03-23T03:46:53.949979+0000","last_became_peered":"2024-03-23T03:46:53.949979+0000","last_unstale":"2024-03-23T03:46:57.970164+0000","last_undegraded":"2024-03-23T03:46:57.970164+0000","last_fullsized":"2024-03-23T03:46:57.970164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:36.598727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952086+0000","last_change":"2024-03-23T03:46:53.957785+0000","last_active":"2024-03-23T03:46:57.952086+0000","last_peered":"2024-03-23T03:46:57.952086+0000","last_clean":"2024-03-23T03:46:57.952086+0000","last_became_active":"2024-03-23T03:46:53.956251+0000","last_became_peered":"2024-03-23T03:46:53.956251+0000","last_unstale":"2024-03-23T03:46:57.952086+0000","last_undegraded":"2024-03-23T03:46:57.952086+0000","last_fullsized":"2024-03-23T03:46:57.952086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:49:34.782399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.964730+0000","last_change":"2024-03-23T03:46:53.950745+0000","last_active":"2024-03-23T03:46:57.964730+0000","last_peered":"2024-03-23T03:46:57.964730+0000","last_clean":"2024-03-23T03:46:57.964730+0000","last_became_active":"2024-03-23T03:46:53.950592+0000","last_became_peered":"2024-03-23T03:46:53.950592+0000","last_unstale":"2024-03-23T03:46:57.964730+0000","last_undegraded":"2024-03-23T03:46:57.964730+0000","last_fullsized":"2024-03-23T03:46:57.964730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:16:39.131183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.942799+0000","last_change":"2024-03-23T03:46:53.949596+0000","last_active":"2024-03-23T03:46:57.942799+0000","last_peered":"2024-03-23T03:46:57.942799+0000","last_clean":"2024-03-23T03:46:57.942799+0000","last_became_active":"2024-03-23T03:46:53.949298+0000","last_became_peered":"2024-03-23T03:46:53.949298+0000","last_unstale":"2024-03-23T03:46:57.942799+0000","last_undegraded":"2024-03-23T03:46:57.942799+0000","last_fullsized":"2024-03-23T03:46:57.942799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:47.801092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952735+0000","last_change":"2024-03-23T03:46:53.961179+0000","last_active":"2024-03-23T03:46:57.952735+0000","last_peered":"2024-03-23T03:46:57.952735+0000","last_clean":"2024-03-23T03:46:57.952735+0000","last_became_active":"2024-03-23T03:46:53.960616+0000","last_became_peered":"2024-03-23T03:46:53.960616+0000","last_unstale":"2024-03-23T03:46:57.952735+0000","last_undegraded":"2024-03-23T03:46:57.952735+0000","last_fullsized":"2024-03-23T03:46:57.952735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:51:48.524978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.971878+0000","last_change":"2024-03-23T03:46:53.951801+0000","last_active":"2024-03-23T03:46:57.971878+0000","last_peered":"2024-03-23T03:46:57.971878+0000","last_clean":"2024-03-23T03:46:57.971878+0000","last_became_active":"2024-03-23T03:46:53.951529+0000","last_became_peered":"2024-03-23T03:46:53.951529+0000","last_unstale":"2024-03-23T03:46:57.971878+0000","last_undegraded":"2024-03-23T03:46:57.971878+0000","last_fullsized":"2024-03-23T03:46:57.971878+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:36.598727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.968476+0000","last_change":"2024-03-23T03:46:53.950898+0000","last_active":"2024-03-23T03:46:57.968476+0000","last_peered":"2024-03-23T03:46:57.968476+0000","last_clean":"2024-03-23T03:46:57.968476+0000","last_became_active":"2024-03-23T03:46:53.950232+0000","last_became_peered":"2024-03-23T03:46:53.950232+0000","last_unstale":"2024-03-23T03:46:57.968476+0000","last_undegraded":"2024-03-23T03:46:57.968476+0000","last_fullsized":"2024-03-23T03:46:57.968476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:51.878165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.961869+0000","last_change":"2024-03-23T03:46:53.939297+0000","last_active":"2024-03-23T03:46:57.961869+0000","last_peered":"2024-03-23T03:46:57.961869+0000","last_clean":"2024-03-23T03:46:57.961869+0000","last_became_active":"2024-03-23T03:46:53.939052+0000","last_became_peered":"2024-03-23T03:46:53.939052+0000","last_unstale":"2024-03-23T03:46:57.961869+0000","last_undegraded":"2024-03-23T03:46:57.961869+0000","last_fullsized":"2024-03-23T03:46:57.961869+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:16.232161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970145+0000","last_change":"2024-03-23T03:46:53.953978+0000","last_active":"2024-03-23T03:46:57.970145+0000","last_peered":"2024-03-23T03:46:57.970145+0000","last_clean":"2024-03-23T03:46:57.970145+0000","last_became_active":"2024-03-23T03:46:53.953703+0000","last_became_peered":"2024-03-23T03:46:53.953703+0000","last_unstale":"2024-03-23T03:46:57.970145+0000","last_undegraded":"2024-03-23T03:46:57.970145+0000","last_fullsized":"2024-03-23T03:46:57.970145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:50:10.900452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951969+0000","last_change":"2024-03-23T03:46:53.969252+0000","last_active":"2024-03-23T03:46:57.951969+0000","last_peered":"2024-03-23T03:46:57.951969+0000","last_clean":"2024-03-23T03:46:57.951969+0000","last_became_active":"2024-03-23T03:46:53.968966+0000","last_became_peered":"2024-03-23T03:46:53.968966+0000","last_unstale":"2024-03-23T03:46:57.951969+0000","last_undegraded":"2024-03-23T03:46:57.951969+0000","last_fullsized":"2024-03-23T03:46:57.951969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:49.329280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.960849+0000","last_change":"2024-03-23T03:46:53.965172+0000","last_active":"2024-03-23T03:46:57.960849+0000","last_peered":"2024-03-23T03:46:57.960849+0000","last_clean":"2024-03-23T03:46:57.960849+0000","last_became_active":"2024-03-23T03:46:53.963673+0000","last_became_peered":"2024-03-23T03:46:53.963673+0000","last_unstale":"2024-03-23T03:46:57.960849+0000","last_undegraded":"2024-03-23T03:46:57.960849+0000","last_fullsized":"2024-03-23T03:46:57.960849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:52:35.361467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017879+0000","last_change":"2024-03-23T03:46:53.952664+0000","last_active":"2024-03-23T03:46:58.017879+0000","last_peered":"2024-03-23T03:46:58.017879+0000","last_clean":"2024-03-23T03:46:58.017879+0000","last_became_active":"2024-03-23T03:46:53.952264+0000","last_became_peered":"2024-03-23T03:46:53.952264+0000","last_unstale":"2024-03-23T03:46:58.017879+0000","last_undegraded":"2024-03-23T03:46:58.017879+0000","last_fullsized":"2024-03-23T03:46:58.017879+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:51.878165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.947849+0000","last_change":"2024-03-23T03:46:53.965686+0000","last_active":"2024-03-23T03:46:57.947849+0000","last_peered":"2024-03-23T03:46:57.947849+0000","last_clean":"2024-03-23T03:46:57.947849+0000","last_became_active":"2024-03-23T03:46:53.965321+0000","last_became_peered":"2024-03-23T03:46:53.965321+0000","last_unstale":"2024-03-23T03:46:57.947849+0000","last_undegraded":"2024-03-23T03:46:57.947849+0000","last_fullsized":"2024-03-23T03:46:57.947849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:29:34.917375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970735+0000","last_change":"2024-03-23T03:46:53.973376+0000","last_active":"2024-03-23T03:46:57.970735+0000","last_peered":"2024-03-23T03:46:57.970735+0000","last_clean":"2024-03-23T03:46:57.970735+0000","last_became_active":"2024-03-23T03:46:53.973131+0000","last_became_peered":"2024-03-23T03:46:53.973131+0000","last_unstale":"2024-03-23T03:46:57.970735+0000","last_undegraded":"2024-03-23T03:46:57.970735+0000","last_fullsized":"2024-03-23T03:46:57.970735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:01:04.097184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.968336+0000","last_change":"2024-03-23T03:46:53.964244+0000","last_active":"2024-03-23T03:46:57.968336+0000","last_peered":"2024-03-23T03:46:57.968336+0000","last_clean":"2024-03-23T03:46:57.968336+0000","last_became_active":"2024-03-23T03:46:53.963926+0000","last_became_peered":"2024-03-23T03:46:53.963926+0000","last_unstale":"2024-03-23T03:46:57.968336+0000","last_undegraded":"2024-03-23T03:46:57.968336+0000","last_fullsized":"2024-03-23T03:46:57.968336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:49.329280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943357+0000","last_change":"2024-03-23T03:46:53.938653+0000","last_active":"2024-03-23T03:46:57.943357+0000","last_peered":"2024-03-23T03:46:57.943357+0000","last_clean":"2024-03-23T03:46:57.943357+0000","last_became_active":"2024-03-23T03:46:53.938323+0000","last_became_peered":"2024-03-23T03:46:53.938323+0000","last_unstale":"2024-03-23T03:46:57.943357+0000","last_undegraded":"2024-03-23T03:46:57.943357+0000","last_fullsized":"2024-03-23T03:46:57.943357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:49.329280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951110+0000","last_change":"2024-03-23T03:46:53.956639+0000","last_active":"2024-03-23T03:46:57.951110+0000","last_peered":"2024-03-23T03:46:57.951110+0000","last_clean":"2024-03-23T03:46:57.951110+0000","last_became_active":"2024-03-23T03:46:53.956097+0000","last_became_peered":"2024-03-23T03:46:53.956097+0000","last_unstale":"2024-03-23T03:46:57.951110+0000","last_undegraded":"2024-03-23T03:46:57.951110+0000","last_fullsized":"2024-03-23T03:46:57.951110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:46.978451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951858+0000","last_change":"2024-03-23T03:46:53.964641+0000","last_active":"2024-03-23T03:46:57.951858+0000","last_peered":"2024-03-23T03:46:57.951858+0000","last_clean":"2024-03-23T03:46:57.951858+0000","last_became_active":"2024-03-23T03:46:53.964167+0000","last_became_peered":"2024-03-23T03:46:53.964167+0000","last_unstale":"2024-03-23T03:46:57.951858+0000","last_undegraded":"2024-03-23T03:46:57.951858+0000","last_fullsized":"2024-03-23T03:46:57.951858+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:55.749189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.965170+0000","last_change":"2024-03-23T03:46:53.962737+0000","last_active":"2024-03-23T03:46:57.965170+0000","last_peered":"2024-03-23T03:46:57.965170+0000","last_clean":"2024-03-23T03:46:57.965170+0000","last_became_active":"2024-03-23T03:46:53.962229+0000","last_became_peered":"2024-03-23T03:46:53.962229+0000","last_unstale":"2024-03-23T03:46:57.965170+0000","last_undegraded":"2024-03-23T03:46:57.965170+0000","last_fullsized":"2024-03-23T03:46:57.965170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:59.999542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.953048+0000","last_change":"2024-03-23T03:46:53.971706+0000","last_active":"2024-03-23T03:46:57.953048+0000","last_peered":"2024-03-23T03:46:57.953048+0000","last_clean":"2024-03-23T03:46:57.953048+0000","last_became_active":"2024-03-23T03:46:53.971525+0000","last_became_peered":"2024-03-23T03:46:53.971525+0000","last_unstale":"2024-03-23T03:46:57.953048+0000","last_undegraded":"2024-03-23T03:46:57.953048+0000","last_fullsized":"2024-03-23T03:46:57.953048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:55:54.119888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952505+0000","last_change":"2024-03-23T03:46:53.970600+0000","last_active":"2024-03-23T03:46:57.952505+0000","last_peered":"2024-03-23T03:46:57.952505+0000","last_clean":"2024-03-23T03:46:57.952505+0000","last_became_active":"2024-03-23T03:46:53.970340+0000","last_became_peered":"2024-03-23T03:46:53.970340+0000","last_unstale":"2024-03-23T03:46:57.952505+0000","last_undegraded":"2024-03-23T03:46:57.952505+0000","last_fullsized":"2024-03-23T03:46:57.952505+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:48:32.346809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017965+0000","last_change":"2024-03-23T03:46:53.971448+0000","last_active":"2024-03-23T03:46:58.017965+0000","last_peered":"2024-03-23T03:46:58.017965+0000","last_clean":"2024-03-23T03:46:58.017965+0000","last_became_active":"2024-03-23T03:46:53.971152+0000","last_became_peered":"2024-03-23T03:46:53.971152+0000","last_unstale":"2024-03-23T03:46:58.017965+0000","last_undegraded":"2024-03-23T03:46:58.017965+0000","last_fullsized":"2024-03-23T03:46:58.017965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:36.598727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.953820+0000","last_change":"2024-03-23T03:46:53.944836+0000","last_active":"2024-03-23T03:46:57.953820+0000","last_peered":"2024-03-23T03:46:57.953820+0000","last_clean":"2024-03-23T03:46:57.953820+0000","last_became_active":"2024-03-23T03:46:53.944614+0000","last_became_peered":"2024-03-23T03:46:53.944614+0000","last_unstale":"2024-03-23T03:46:57.953820+0000","last_undegraded":"2024-03-23T03:46:57.953820+0000","last_fullsized":"2024-03-23T03:46:57.953820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:01:24.719476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.013281+0000","last_change":"2024-03-23T03:46:53.957372+0000","last_active":"2024-03-23T03:46:58.013281+0000","last_peered":"2024-03-23T03:46:58.013281+0000","last_clean":"2024-03-23T03:46:58.013281+0000","last_became_active":"2024-03-23T03:46:53.956119+0000","last_became_peered":"2024-03-23T03:46:53.956119+0000","last_unstale":"2024-03-23T03:46:58.013281+0000","last_undegraded":"2024-03-23T03:46:58.013281+0000","last_fullsized":"2024-03-23T03:46:58.013281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:16:39.131183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.968302+0000","last_change":"2024-03-23T03:46:53.954765+0000","last_active":"2024-03-23T03:46:57.968302+0000","last_peered":"2024-03-23T03:46:57.968302+0000","last_clean":"2024-03-23T03:46:57.968302+0000","last_became_active":"2024-03-23T03:46:53.954424+0000","last_became_peered":"2024-03-23T03:46:53.954424+0000","last_unstale":"2024-03-23T03:46:57.968302+0000","last_undegraded":"2024-03-23T03:46:57.968302+0000","last_fullsized":"2024-03-23T03:46:57.968302+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:56:29.065234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.016764+0000","last_change":"2024-03-23T03:46:53.949427+0000","last_active":"2024-03-23T03:46:58.016764+0000","last_peered":"2024-03-23T03:46:58.016764+0000","last_clean":"2024-03-23T03:46:58.016764+0000","last_became_active":"2024-03-23T03:46:53.949114+0000","last_became_peered":"2024-03-23T03:46:53.949114+0000","last_unstale":"2024-03-23T03:46:58.016764+0000","last_undegraded":"2024-03-23T03:46:58.016764+0000","last_fullsized":"2024-03-23T03:46:58.016764+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:50:10.900452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970488+0000","last_change":"2024-03-23T03:46:53.956260+0000","last_active":"2024-03-23T03:46:57.970488+0000","last_peered":"2024-03-23T03:46:57.970488+0000","last_clean":"2024-03-23T03:46:57.970488+0000","last_became_active":"2024-03-23T03:46:53.955509+0000","last_became_peered":"2024-03-23T03:46:53.955509+0000","last_unstale":"2024-03-23T03:46:57.970488+0000","last_undegraded":"2024-03-23T03:46:57.970488+0000","last_fullsized":"2024-03-23T03:46:57.970488+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:34:41.509208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951671+0000","last_change":"2024-03-23T03:46:53.957607+0000","last_active":"2024-03-23T03:46:57.951671+0000","last_peered":"2024-03-23T03:46:57.951671+0000","last_clean":"2024-03-23T03:46:57.951671+0000","last_became_active":"2024-03-23T03:46:53.956236+0000","last_became_peered":"2024-03-23T03:46:53.956236+0000","last_unstale":"2024-03-23T03:46:57.951671+0000","last_undegraded":"2024-03-23T03:46:57.951671+0000","last_fullsized":"2024-03-23T03:46:57.951671+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:27:03.917141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943481+0000","last_change":"2024-03-23T03:46:53.960428+0000","last_active":"2024-03-23T03:46:57.943481+0000","last_peered":"2024-03-23T03:46:57.943481+0000","last_clean":"2024-03-23T03:46:57.943481+0000","last_became_active":"2024-03-23T03:46:53.960148+0000","last_became_peered":"2024-03-23T03:46:53.960148+0000","last_unstale":"2024-03-23T03:46:57.943481+0000","last_undegraded":"2024-03-23T03:46:57.943481+0000","last_fullsized":"2024-03-23T03:46:57.943481+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:01:24.719476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017880+0000","last_change":"2024-03-23T03:46:53.966166+0000","last_active":"2024-03-23T03:46:58.017880+0000","last_peered":"2024-03-23T03:46:58.017880+0000","last_clean":"2024-03-23T03:46:58.017880+0000","last_became_active":"2024-03-23T03:46:53.965871+0000","last_became_peered":"2024-03-23T03:46:53.965871+0000","last_unstale":"2024-03-23T03:46:58.017880+0000","last_undegraded":"2024-03-23T03:46:58.017880+0000","last_fullsized":"2024-03-23T03:46:58.017880+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:01:04.097184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4452,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":698548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1060000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.276}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0569999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":111252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.589}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3939999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4670000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":107688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":107724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2689999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.345}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":111252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2050000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4359999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0569999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.341}]}]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":107598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9119999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.97}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8360000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8120000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":107758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3080000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.871}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":880640,"data_stored":698604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.659}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.829}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8160000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:47:16.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:47:16.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.485+0000 7f7302200640 1 Processor -- start 2024-03-23T03:47:16.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.485+0000 7f7302200640 1 -- start start 2024-03-23T03:47:16.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.486+0000 7f7302200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f72fc15b560 0x7f72fc15b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:16.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.487+0000 7f7302200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc15f320 con 0x7f72fc15d410 2024-03-23T03:47:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.487+0000 7f7302200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc06c6f0 con 0x7f72fc0b39b0 2024-03-23T03:47:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.487+0000 7f7302200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72fc06c9d0 con 0x7f72fc15b560 2024-03-23T03:47:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.487+0000 7f72fbe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f72fc15d410 0x7f72fc15ec10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41120/0 (socket says 172.21.15.102:41120) 2024-03-23T03:47:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.487+0000 7f72fbe00640 1 -- 172.21.15.102:0/1597514756 learned_addr learned my addr 172.21.15.102:0/1597514756 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.487+0000 7f72fb400640 1 --2- 172.21.15.102:0/1597514756 >> v2:172.21.15.102:3300/0 conn(0x7f72fc15b560 0x7f72fc15b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:16.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.487+0000 7f72faa00640 1 -- 172.21.15.102:0/1597514756 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102816901 0 0) 0x7f72fc15f320 con 0x7f72fc15d410 2024-03-23T03:47:16.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.487+0000 7f72faa00640 1 -- 172.21.15.102:0/1597514756 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e0003520 con 0x7f72fc15d410 2024-03-23T03:47:16.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.487+0000 7f72fb400640 1 -- 172.21.15.102:0/1597514756 >> v1:172.21.15.173:6789/0 conn(0x7f72fc0b39b0 legacy=0x7f72fc15e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:16.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.487+0000 7f72fb400640 1 -- 172.21.15.102:0/1597514756 >> v1:172.21.15.102:6789/0 conn(0x7f72fc15d410 legacy=0x7f72fc15ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:16.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.487+0000 7f72fb400640 1 -- 172.21.15.102:0/1597514756 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc06ccb0 con 0x7f72fc15b560 2024-03-23T03:47:16.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.489+0000 7f72fb400640 1 --2- 172.21.15.102:0/1597514756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72fc15b560 0x7f72fc15b940 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0097a0 tx=0x7f72ec02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933fcd19b667fa1a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:16.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.489+0000 7f72faa00640 1 -- 172.21.15.102:0/1597514756 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec02e070 con 0x7f72fc15b560 2024-03-23T03:47:16.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.489+0000 7f72faa00640 1 -- 172.21.15.102:0/1597514756 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ec0315a0 con 0x7f72fc15b560 2024-03-23T03:47:16.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.489+0000 7f72faa00640 1 -- 172.21.15.102:0/1597514756 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec031930 con 0x7f72fc15b560 2024-03-23T03:47:16.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.489+0000 7f7302200640 1 -- 172.21.15.102:0/1597514756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72fc15b560 msgr2=0x7f72fc15b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:16.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.489+0000 7f7302200640 1 --2- 172.21.15.102:0/1597514756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72fc15b560 0x7f72fc15b940 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0097a0 tx=0x7f72ec02b840 comp rx=0 tx=0).stop 2024-03-23T03:47:16.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.490+0000 7f7302200640 1 -- 172.21.15.102:0/1597514756 shutdown_connections 2024-03-23T03:47:16.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.490+0000 7f7302200640 1 --2- 172.21.15.102:0/1597514756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72fc15b560 0x7f72fc15b940 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.490+0000 7f7302200640 1 -- 172.21.15.102:0/1597514756 >> 172.21.15.102:0/1597514756 conn(0x7f72fc06a3d0 msgr2=0x7f72fc0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:16.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.490+0000 7f7302200640 1 -- 172.21.15.102:0/1597514756 shutdown_connections 2024-03-23T03:47:16.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.490+0000 7f7302200640 1 -- 172.21.15.102:0/1597514756 wait complete. 2024-03-23T03:47:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.491+0000 7f7302200640 1 Processor -- start 2024-03-23T03:47:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.492+0000 7f7302200640 1 -- start start 2024-03-23T03:47:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.492+0000 7f7302200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72fc0b39b0 0x7f72fc1324f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.492+0000 7f7302200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72fc15b560 0x7f72fc135960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.492+0000 7f7302200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72fc15d410 0x7f72fc158290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.492+0000 7f7302200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc1547b0 con 0x7f72fc0b39b0 2024-03-23T03:47:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.492+0000 7f7302200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc157840 con 0x7f72fc15d410 2024-03-23T03:47:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.492+0000 7f7302200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc1549b0 con 0x7f72fc15b560 2024-03-23T03:47:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.493+0000 7f72fb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72fc0b39b0 0x7f72fc1324f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.493+0000 7f72fb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72fc0b39b0 0x7f72fc1324f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37116/0 (socket says 172.21.15.102:37116) 2024-03-23T03:47:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.493+0000 7f72fb400640 1 -- 172.21.15.102:0/2497726580 learned_addr learned my addr 172.21.15.102:0/2497726580 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.493+0000 7f72fbe00640 1 --2- 172.21.15.102:0/2497726580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72fc15d410 0x7f72fc158290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:16.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.493+0000 7f72fb400640 1 -- 172.21.15.102:0/2497726580 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72fc15b560 msgr2=0x7f72fc135960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:16.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.493+0000 7f72fb400640 1 --2- 172.21.15.102:0/2497726580 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72fc15b560 0x7f72fc135960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.493+0000 7f72fb400640 1 -- 172.21.15.102:0/2497726580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72fc15d410 msgr2=0x7f72fc158290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:16.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.493+0000 7f72fb400640 1 --2- 172.21.15.102:0/2497726580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72fc15d410 0x7f72fc158290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.493+0000 7f72fb400640 1 -- 172.21.15.102:0/2497726580 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec009400 con 0x7f72fc0b39b0 2024-03-23T03:47:16.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.493+0000 7f72fb400640 1 --2- 172.21.15.102:0/2497726580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72fc0b39b0 0x7f72fc1324f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f72ec034040 tx=0x7f72ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.494+0000 7f72f3e00640 1 -- 172.21.15.102:0/2497726580 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec032840 con 0x7f72fc0b39b0 2024-03-23T03:47:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.494+0000 7f72f3e00640 1 -- 172.21.15.102:0/2497726580 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ec0329c0 con 0x7f72fc0b39b0 2024-03-23T03:47:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.494+0000 7f72f3e00640 1 -- 172.21.15.102:0/2497726580 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72ec032d70 con 0x7f72fc0b39b0 2024-03-23T03:47:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.494+0000 7f7302200640 1 -- 172.21.15.102:0/2497726580 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72fc154c90 con 0x7f72fc0b39b0 2024-03-23T03:47:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.494+0000 7f7302200640 1 -- 172.21.15.102:0/2497726580 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72fc157d70 con 0x7f72fc0b39b0 2024-03-23T03:47:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.495+0000 7f7302200640 1 -- 172.21.15.102:0/2497726580 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72fc06ac20 con 0x7f72fc0b39b0 2024-03-23T03:47:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.496+0000 7f72f3e00640 1 -- 172.21.15.102:0/2497726580 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f72ec03b3f0 con 0x7f72fc0b39b0 2024-03-23T03:47:16.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.497+0000 7f72f3e00640 1 --2- 172.21.15.102:0/2497726580 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f72d806ac00 0x7f72d806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:16.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.497+0000 7f72f3e00640 1 -- 172.21.15.102:0/2497726580 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f72ec0ab5f0 con 0x7f72fc0b39b0 2024-03-23T03:47:16.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.497+0000 7f72f2a00640 1 --2- 172.21.15.102:0/2497726580 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f72d806ac00 0x7f72d806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:16.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.498+0000 7f72f2a00640 1 --2- 172.21.15.102:0/2497726580 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f72d806ac00 0x7f72d806d0c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0afb60 tx=0x7f72e40079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:16.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.502+0000 7f72f3e00640 1 -- 172.21.15.102:0/2497726580 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f72ec076440 con 0x7f72fc0b39b0 2024-03-23T03:47:16.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.692+0000 7f7302200640 1 -- 172.21.15.102:0/2497726580 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f72fc06dd80 con 0x7f72d806ac00 2024-03-23T03:47:16.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.717+0000 7f72f3e00640 1 -- 172.21.15.102:0/2497726580 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345116 (secure 0 0 0) 0x7f72e4005030 con 0x7f72d806ac00 2024-03-23T03:47:16.722 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:16.725 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:47:16.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.726+0000 7f7302200640 1 -- 172.21.15.102:0/2497726580 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f72d806ac00 msgr2=0x7f72d806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:16.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.726+0000 7f7302200640 1 --2- 172.21.15.102:0/2497726580 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f72d806ac00 0x7f72d806d0c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f72fc0afb60 tx=0x7f72e40079b0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.726+0000 7f7302200640 1 -- 172.21.15.102:0/2497726580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72fc0b39b0 msgr2=0x7f72fc1324f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:16.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.726+0000 7f7302200640 1 --2- 172.21.15.102:0/2497726580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72fc0b39b0 0x7f72fc1324f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f72ec034040 tx=0x7f72ec0046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.726+0000 7f7302200640 1 -- 172.21.15.102:0/2497726580 shutdown_connections 2024-03-23T03:47:16.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.726+0000 7f7302200640 1 --2- 172.21.15.102:0/2497726580 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f72d806ac00 0x7f72d806d0c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.726+0000 7f7302200640 1 --2- 172.21.15.102:0/2497726580 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72fc15d410 0x7f72fc158290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.726+0000 7f7302200640 1 --2- 172.21.15.102:0/2497726580 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72fc15b560 0x7f72fc135960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.726+0000 7f7302200640 1 --2- 172.21.15.102:0/2497726580 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72fc0b39b0 0x7f72fc1324f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.726+0000 7f7302200640 1 -- 172.21.15.102:0/2497726580 >> 172.21.15.102:0/2497726580 conn(0x7f72fc06a3d0 msgr2=0x7f72fc0aecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:16.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.727+0000 7f7302200640 1 -- 172.21.15.102:0/2497726580 shutdown_connections 2024-03-23T03:47:16.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.727+0000 7f7302200640 1 -- 172.21.15.102:0/2497726580 wait complete. 2024-03-23T03:47:16.747 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-23T03:47:15.721801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594820,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":206,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":158,"ondisk_log_size":158,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219556,"kb_used_data":3368,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755164,"statfs":{"total":773094113280,"available":772869287936,"internally_reserved":0,"allocated":3448832,"data_stored":2050424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":221236794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2090,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004284"},"pg_stats":[{"pgid":"2.1f","version":"21'4","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.954535+0000","last_change":"2024-03-23T03:46:53.465534+0000","last_active":"2024-03-23T03:46:57.954535+0000","last_peered":"2024-03-23T03:46:57.954535+0000","last_clean":"2024-03-23T03:46:57.954535+0000","last_became_active":"2024-03-23T03:46:53.465309+0000","last_became_peered":"2024-03-23T03:46:53.465309+0000","last_unstale":"2024-03-23T03:46:57.954535+0000","last_undegraded":"2024-03-23T03:46:57.954535+0000","last_fullsized":"2024-03-23T03:46:57.954535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:52:38.737363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951985+0000","last_change":"2024-03-23T03:46:53.963693+0000","last_active":"2024-03-23T03:46:57.951985+0000","last_peered":"2024-03-23T03:46:57.951985+0000","last_clean":"2024-03-23T03:46:57.951985+0000","last_became_active":"2024-03-23T03:46:53.963435+0000","last_became_peered":"2024-03-23T03:46:53.963435+0000","last_unstale":"2024-03-23T03:46:57.951985+0000","last_undegraded":"2024-03-23T03:46:57.951985+0000","last_fullsized":"2024-03-23T03:46:57.951985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:51.878165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.544002+0000","last_change":"2024-03-23T03:46:58.018608+0000","last_active":"2024-03-23T03:46:59.544002+0000","last_peered":"2024-03-23T03:46:59.544002+0000","last_clean":"2024-03-23T03:46:59.544002+0000","last_became_active":"2024-03-23T03:46:58.018216+0000","last_became_peered":"2024-03-23T03:46:58.018216+0000","last_unstale":"2024-03-23T03:46:59.544002+0000","last_undegraded":"2024-03-23T03:46:59.544002+0000","last_fullsized":"2024-03-23T03:46:59.544002+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:41:36.624562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952201+0000","last_change":"2024-03-23T03:46:53.462106+0000","last_active":"2024-03-23T03:46:57.952201+0000","last_peered":"2024-03-23T03:46:57.952201+0000","last_clean":"2024-03-23T03:46:57.952201+0000","last_became_active":"2024-03-23T03:46:53.461757+0000","last_became_peered":"2024-03-23T03:46:53.461757+0000","last_unstale":"2024-03-23T03:46:57.952201+0000","last_undegraded":"2024-03-23T03:46:57.952201+0000","last_fullsized":"2024-03-23T03:46:57.952201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:20:09.953919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.954319+0000","last_change":"2024-03-23T03:46:53.959076+0000","last_active":"2024-03-23T03:46:57.954319+0000","last_peered":"2024-03-23T03:46:57.954319+0000","last_clean":"2024-03-23T03:46:57.954319+0000","last_became_active":"2024-03-23T03:46:53.958351+0000","last_became_peered":"2024-03-23T03:46:53.958351+0000","last_unstale":"2024-03-23T03:46:57.954319+0000","last_undegraded":"2024-03-23T03:46:57.954319+0000","last_fullsized":"2024-03-23T03:46:57.954319+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:55.749189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.423330+0000","last_change":"2024-03-23T03:46:57.979717+0000","last_active":"2024-03-23T03:46:58.423330+0000","last_peered":"2024-03-23T03:46:58.423330+0000","last_clean":"2024-03-23T03:46:58.423330+0000","last_became_active":"2024-03-23T03:46:57.975153+0000","last_became_peered":"2024-03-23T03:46:57.975153+0000","last_unstale":"2024-03-23T03:46:58.423330+0000","last_undegraded":"2024-03-23T03:46:58.423330+0000","last_fullsized":"2024-03-23T03:46:58.423330+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:24:30.082562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017773+0000","last_change":"2024-03-23T03:46:53.480716+0000","last_active":"2024-03-23T03:46:58.017773+0000","last_peered":"2024-03-23T03:46:58.017773+0000","last_clean":"2024-03-23T03:46:58.017773+0000","last_became_active":"2024-03-23T03:46:53.479510+0000","last_became_peered":"2024-03-23T03:46:53.479510+0000","last_unstale":"2024-03-23T03:46:58.017773+0000","last_undegraded":"2024-03-23T03:46:58.017773+0000","last_fullsized":"2024-03-23T03:46:58.017773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:40:20.814830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970572+0000","last_change":"2024-03-23T03:46:53.947108+0000","last_active":"2024-03-23T03:46:57.970572+0000","last_peered":"2024-03-23T03:46:57.970572+0000","last_clean":"2024-03-23T03:46:57.970572+0000","last_became_active":"2024-03-23T03:46:53.946415+0000","last_became_peered":"2024-03-23T03:46:53.946415+0000","last_unstale":"2024-03-23T03:46:57.970572+0000","last_undegraded":"2024-03-23T03:46:57.970572+0000","last_fullsized":"2024-03-23T03:46:57.970572+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:38:13.205855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.426555+0000","last_change":"2024-03-23T03:46:57.968557+0000","last_active":"2024-03-23T03:46:58.426555+0000","last_peered":"2024-03-23T03:46:58.426555+0000","last_clean":"2024-03-23T03:46:58.426555+0000","last_became_active":"2024-03-23T03:46:57.968085+0000","last_became_peered":"2024-03-23T03:46:57.968085+0000","last_unstale":"2024-03-23T03:46:58.426555+0000","last_undegraded":"2024-03-23T03:46:58.426555+0000","last_fullsized":"2024-03-23T03:46:58.426555+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:26:09.015758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.014925+0000","last_change":"2024-03-23T03:46:53.481389+0000","last_active":"2024-03-23T03:46:58.014925+0000","last_peered":"2024-03-23T03:46:58.014925+0000","last_clean":"2024-03-23T03:46:58.014925+0000","last_became_active":"2024-03-23T03:46:53.481119+0000","last_became_peered":"2024-03-23T03:46:53.481119+0000","last_unstale":"2024-03-23T03:46:58.014925+0000","last_undegraded":"2024-03-23T03:46:58.014925+0000","last_fullsized":"2024-03-23T03:46:58.014925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:23:34.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970863+0000","last_change":"2024-03-23T03:46:53.952464+0000","last_active":"2024-03-23T03:46:57.970863+0000","last_peered":"2024-03-23T03:46:57.970863+0000","last_clean":"2024-03-23T03:46:57.970863+0000","last_became_active":"2024-03-23T03:46:53.951343+0000","last_became_peered":"2024-03-23T03:46:53.951343+0000","last_unstale":"2024-03-23T03:46:57.970863+0000","last_undegraded":"2024-03-23T03:46:57.970863+0000","last_fullsized":"2024-03-23T03:46:57.970863+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:35:49.855085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.423616+0000","last_change":"2024-03-23T03:46:57.974943+0000","last_active":"2024-03-23T03:46:58.423616+0000","last_peered":"2024-03-23T03:46:58.423616+0000","last_clean":"2024-03-23T03:46:58.423616+0000","last_became_active":"2024-03-23T03:46:57.974037+0000","last_became_peered":"2024-03-23T03:46:57.974037+0000","last_unstale":"2024-03-23T03:46:58.423616+0000","last_undegraded":"2024-03-23T03:46:58.423616+0000","last_fullsized":"2024-03-23T03:46:58.423616+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:24:17.331178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952188+0000","last_change":"2024-03-23T03:46:53.961283+0000","last_active":"2024-03-23T03:46:57.952188+0000","last_peered":"2024-03-23T03:46:57.952188+0000","last_clean":"2024-03-23T03:46:57.952188+0000","last_became_active":"2024-03-23T03:46:53.961004+0000","last_became_peered":"2024-03-23T03:46:53.961004+0000","last_unstale":"2024-03-23T03:46:57.952188+0000","last_undegraded":"2024-03-23T03:46:57.952188+0000","last_fullsized":"2024-03-23T03:46:57.952188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:30:06.938168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017032+0000","last_change":"2024-03-23T03:46:53.480802+0000","last_active":"2024-03-23T03:46:58.017032+0000","last_peered":"2024-03-23T03:46:58.017032+0000","last_clean":"2024-03-23T03:46:58.017032+0000","last_became_active":"2024-03-23T03:46:53.480500+0000","last_became_peered":"2024-03-23T03:46:53.480500+0000","last_unstale":"2024-03-23T03:46:58.017032+0000","last_undegraded":"2024-03-23T03:46:58.017032+0000","last_fullsized":"2024-03-23T03:46:58.017032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:38:29.605938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.424746+0000","last_change":"2024-03-23T03:46:57.972616+0000","last_active":"2024-03-23T03:46:58.424746+0000","last_peered":"2024-03-23T03:46:58.424746+0000","last_clean":"2024-03-23T03:46:58.424746+0000","last_became_active":"2024-03-23T03:46:57.972028+0000","last_became_peered":"2024-03-23T03:46:57.972028+0000","last_unstale":"2024-03-23T03:46:58.424746+0000","last_undegraded":"2024-03-23T03:46:58.424746+0000","last_fullsized":"2024-03-23T03:46:58.424746+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:55:53.867281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952016+0000","last_change":"2024-03-23T03:46:53.960010+0000","last_active":"2024-03-23T03:46:57.952016+0000","last_peered":"2024-03-23T03:46:57.952016+0000","last_clean":"2024-03-23T03:46:57.952016+0000","last_became_active":"2024-03-23T03:46:53.959761+0000","last_became_peered":"2024-03-23T03:46:53.959761+0000","last_unstale":"2024-03-23T03:46:57.952016+0000","last_undegraded":"2024-03-23T03:46:57.952016+0000","last_fullsized":"2024-03-23T03:46:57.952016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:49.329280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.946127+0000","last_change":"2024-03-23T03:46:53.460246+0000","last_active":"2024-03-23T03:46:57.946127+0000","last_peered":"2024-03-23T03:46:57.946127+0000","last_clean":"2024-03-23T03:46:57.946127+0000","last_became_active":"2024-03-23T03:46:53.459900+0000","last_became_peered":"2024-03-23T03:46:53.459900+0000","last_unstale":"2024-03-23T03:46:57.946127+0000","last_undegraded":"2024-03-23T03:46:57.946127+0000","last_fullsized":"2024-03-23T03:46:57.946127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:23:34.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.424823+0000","last_change":"2024-03-23T03:46:57.976006+0000","last_active":"2024-03-23T03:46:58.424823+0000","last_peered":"2024-03-23T03:46:58.424823+0000","last_clean":"2024-03-23T03:46:58.424823+0000","last_became_active":"2024-03-23T03:46:57.975320+0000","last_became_peered":"2024-03-23T03:46:57.975320+0000","last_unstale":"2024-03-23T03:46:58.424823+0000","last_undegraded":"2024-03-23T03:46:58.424823+0000","last_fullsized":"2024-03-23T03:46:58.424823+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:54.573575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.960391+0000","last_change":"2024-03-23T03:46:53.954675+0000","last_active":"2024-03-23T03:46:57.960391+0000","last_peered":"2024-03-23T03:46:57.960391+0000","last_clean":"2024-03-23T03:46:57.960391+0000","last_became_active":"2024-03-23T03:46:53.954433+0000","last_became_peered":"2024-03-23T03:46:53.954433+0000","last_unstale":"2024-03-23T03:46:57.960391+0000","last_undegraded":"2024-03-23T03:46:57.960391+0000","last_fullsized":"2024-03-23T03:46:57.960391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:27:03.917141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943505+0000","last_change":"2024-03-23T03:46:53.463665+0000","last_active":"2024-03-23T03:46:57.943505+0000","last_peered":"2024-03-23T03:46:57.943505+0000","last_clean":"2024-03-23T03:46:57.943505+0000","last_became_active":"2024-03-23T03:46:53.463379+0000","last_became_peered":"2024-03-23T03:46:53.463379+0000","last_unstale":"2024-03-23T03:46:57.943505+0000","last_undegraded":"2024-03-23T03:46:57.943505+0000","last_fullsized":"2024-03-23T03:46:57.943505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:39:13.578639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545667+0000","last_change":"2024-03-23T03:46:58.017005+0000","last_active":"2024-03-23T03:46:59.545667+0000","last_peered":"2024-03-23T03:46:59.545667+0000","last_clean":"2024-03-23T03:46:59.545667+0000","last_became_active":"2024-03-23T03:46:58.016393+0000","last_became_peered":"2024-03-23T03:46:58.016393+0000","last_unstale":"2024-03-23T03:46:59.545667+0000","last_undegraded":"2024-03-23T03:46:59.545667+0000","last_fullsized":"2024-03-23T03:46:59.545667+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:54:23.785439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.959120+0000","last_change":"2024-03-23T03:46:53.941019+0000","last_active":"2024-03-23T03:46:57.959120+0000","last_peered":"2024-03-23T03:46:57.959120+0000","last_clean":"2024-03-23T03:46:57.959120+0000","last_became_active":"2024-03-23T03:46:53.940265+0000","last_became_peered":"2024-03-23T03:46:53.940265+0000","last_unstale":"2024-03-23T03:46:57.959120+0000","last_undegraded":"2024-03-23T03:46:57.959120+0000","last_fullsized":"2024-03-23T03:46:57.959120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:04:08.642972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.012311+0000","last_change":"2024-03-23T03:46:53.471334+0000","last_active":"2024-03-23T03:46:58.012311+0000","last_peered":"2024-03-23T03:46:58.012311+0000","last_clean":"2024-03-23T03:46:58.012311+0000","last_became_active":"2024-03-23T03:46:53.470996+0000","last_became_peered":"2024-03-23T03:46:53.470996+0000","last_unstale":"2024-03-23T03:46:58.012311+0000","last_undegraded":"2024-03-23T03:46:58.012311+0000","last_fullsized":"2024-03-23T03:46:58.012311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:55.074632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.425416+0000","last_change":"2024-03-23T03:46:57.975950+0000","last_active":"2024-03-23T03:46:58.425416+0000","last_peered":"2024-03-23T03:46:58.425416+0000","last_clean":"2024-03-23T03:46:58.425416+0000","last_became_active":"2024-03-23T03:46:57.975504+0000","last_became_peered":"2024-03-23T03:46:57.975504+0000","last_unstale":"2024-03-23T03:46:58.425416+0000","last_undegraded":"2024-03-23T03:46:58.425416+0000","last_fullsized":"2024-03-23T03:46:58.425416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:21.492278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.955743+0000","last_change":"2024-03-23T03:46:53.957169+0000","last_active":"2024-03-23T03:46:57.955743+0000","last_peered":"2024-03-23T03:46:57.955743+0000","last_clean":"2024-03-23T03:46:57.955743+0000","last_became_active":"2024-03-23T03:46:53.956617+0000","last_became_peered":"2024-03-23T03:46:53.956617+0000","last_unstale":"2024-03-23T03:46:57.955743+0000","last_undegraded":"2024-03-23T03:46:57.955743+0000","last_fullsized":"2024-03-23T03:46:57.955743+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:50:57.246940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.953730+0000","last_change":"2024-03-23T03:46:53.475395+0000","last_active":"2024-03-23T03:46:57.953730+0000","last_peered":"2024-03-23T03:46:57.953730+0000","last_clean":"2024-03-23T03:46:57.953730+0000","last_became_active":"2024-03-23T03:46:53.475130+0000","last_became_peered":"2024-03-23T03:46:53.475130+0000","last_unstale":"2024-03-23T03:46:57.953730+0000","last_undegraded":"2024-03-23T03:46:57.953730+0000","last_fullsized":"2024-03-23T03:46:57.953730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:19:02.377973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.423957+0000","last_change":"2024-03-23T03:46:58.019438+0000","last_active":"2024-03-23T03:46:58.423957+0000","last_peered":"2024-03-23T03:46:58.423957+0000","last_clean":"2024-03-23T03:46:58.423957+0000","last_became_active":"2024-03-23T03:46:58.019018+0000","last_became_peered":"2024-03-23T03:46:58.019018+0000","last_unstale":"2024-03-23T03:46:58.423957+0000","last_undegraded":"2024-03-23T03:46:58.423957+0000","last_fullsized":"2024-03-23T03:46:58.423957+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:53:39.544425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.971959+0000","last_change":"2024-03-23T03:46:53.950103+0000","last_active":"2024-03-23T03:46:57.971959+0000","last_peered":"2024-03-23T03:46:57.971959+0000","last_clean":"2024-03-23T03:46:57.971959+0000","last_became_active":"2024-03-23T03:46:53.947870+0000","last_became_peered":"2024-03-23T03:46:53.947870+0000","last_unstale":"2024-03-23T03:46:57.971959+0000","last_undegraded":"2024-03-23T03:46:57.971959+0000","last_fullsized":"2024-03-23T03:46:57.971959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:56:29.065234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943640+0000","last_change":"2024-03-23T03:46:53.476543+0000","last_active":"2024-03-23T03:46:57.943640+0000","last_peered":"2024-03-23T03:46:57.943640+0000","last_clean":"2024-03-23T03:46:57.943640+0000","last_became_active":"2024-03-23T03:46:53.476227+0000","last_became_peered":"2024-03-23T03:46:53.476227+0000","last_unstale":"2024-03-23T03:46:57.943640+0000","last_undegraded":"2024-03-23T03:46:57.943640+0000","last_fullsized":"2024-03-23T03:46:57.943640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:52:38.737363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.543994+0000","last_change":"2024-03-23T03:46:58.013935+0000","last_active":"2024-03-23T03:46:59.543994+0000","last_peered":"2024-03-23T03:46:59.543994+0000","last_clean":"2024-03-23T03:46:59.543994+0000","last_became_active":"2024-03-23T03:46:58.012631+0000","last_became_peered":"2024-03-23T03:46:58.012631+0000","last_unstale":"2024-03-23T03:46:59.543994+0000","last_undegraded":"2024-03-23T03:46:59.543994+0000","last_fullsized":"2024-03-23T03:46:59.543994+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:42:09.297861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.950778+0000","last_change":"2024-03-23T03:46:53.945184+0000","last_active":"2024-03-23T03:46:57.950778+0000","last_peered":"2024-03-23T03:46:57.950778+0000","last_clean":"2024-03-23T03:46:57.950778+0000","last_became_active":"2024-03-23T03:46:53.943550+0000","last_became_peered":"2024-03-23T03:46:53.943550+0000","last_unstale":"2024-03-23T03:46:57.950778+0000","last_undegraded":"2024-03-23T03:46:57.950778+0000","last_fullsized":"2024-03-23T03:46:57.950778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:01:31.453936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.018003+0000","last_change":"2024-03-23T03:46:53.472577+0000","last_active":"2024-03-23T03:46:58.018003+0000","last_peered":"2024-03-23T03:46:58.018003+0000","last_clean":"2024-03-23T03:46:58.018003+0000","last_became_active":"2024-03-23T03:46:53.472218+0000","last_became_peered":"2024-03-23T03:46:53.472218+0000","last_unstale":"2024-03-23T03:46:58.018003+0000","last_undegraded":"2024-03-23T03:46:58.018003+0000","last_fullsized":"2024-03-23T03:46:58.018003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:21:03.254658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411414+0000","last_change":"2024-03-23T03:46:58.013198+0000","last_active":"2024-03-23T03:46:59.411414+0000","last_peered":"2024-03-23T03:46:59.411414+0000","last_clean":"2024-03-23T03:46:59.411414+0000","last_became_active":"2024-03-23T03:46:58.012474+0000","last_became_peered":"2024-03-23T03:46:58.012474+0000","last_unstale":"2024-03-23T03:46:59.411414+0000","last_undegraded":"2024-03-23T03:46:59.411414+0000","last_fullsized":"2024-03-23T03:46:59.411414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:22:28.017991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970873+0000","last_change":"2024-03-23T03:46:53.955218+0000","last_active":"2024-03-23T03:46:57.970873+0000","last_peered":"2024-03-23T03:46:57.970873+0000","last_clean":"2024-03-23T03:46:57.970873+0000","last_became_active":"2024-03-23T03:46:53.954977+0000","last_became_peered":"2024-03-23T03:46:53.954977+0000","last_unstale":"2024-03-23T03:46:57.970873+0000","last_undegraded":"2024-03-23T03:46:57.970873+0000","last_fullsized":"2024-03-23T03:46:57.970873+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:12.278522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.947128+0000","last_change":"2024-03-23T03:46:53.456663+0000","last_active":"2024-03-23T03:46:57.947128+0000","last_peered":"2024-03-23T03:46:57.947128+0000","last_clean":"2024-03-23T03:46:57.947128+0000","last_became_active":"2024-03-23T03:46:53.456393+0000","last_became_peered":"2024-03-23T03:46:53.456393+0000","last_unstale":"2024-03-23T03:46:57.947128+0000","last_undegraded":"2024-03-23T03:46:57.947128+0000","last_fullsized":"2024-03-23T03:46:57.947128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:38:29.605938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.544795+0000","last_change":"2024-03-23T03:46:57.975493+0000","last_active":"2024-03-23T03:46:59.544795+0000","last_peered":"2024-03-23T03:46:59.544795+0000","last_clean":"2024-03-23T03:46:59.544795+0000","last_became_active":"2024-03-23T03:46:57.974943+0000","last_became_peered":"2024-03-23T03:46:57.974943+0000","last_unstale":"2024-03-23T03:46:59.544795+0000","last_undegraded":"2024-03-23T03:46:59.544795+0000","last_fullsized":"2024-03-23T03:46:59.544795+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:56.623233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.960709+0000","last_change":"2024-03-23T03:46:53.963022+0000","last_active":"2024-03-23T03:46:57.960709+0000","last_peered":"2024-03-23T03:46:57.960709+0000","last_clean":"2024-03-23T03:46:57.960709+0000","last_became_active":"2024-03-23T03:46:53.962270+0000","last_became_peered":"2024-03-23T03:46:53.962270+0000","last_unstale":"2024-03-23T03:46:57.960709+0000","last_undegraded":"2024-03-23T03:46:57.960709+0000","last_fullsized":"2024-03-23T03:46:57.960709+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:55:23.986885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970281+0000","last_change":"2024-03-23T03:46:53.456217+0000","last_active":"2024-03-23T03:46:57.970281+0000","last_peered":"2024-03-23T03:46:57.970281+0000","last_clean":"2024-03-23T03:46:57.970281+0000","last_became_active":"2024-03-23T03:46:53.455825+0000","last_became_peered":"2024-03-23T03:46:53.455825+0000","last_unstale":"2024-03-23T03:46:57.970281+0000","last_undegraded":"2024-03-23T03:46:57.970281+0000","last_fullsized":"2024-03-23T03:46:57.970281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:20:09.953919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.544875+0000","last_change":"2024-03-23T03:46:57.978215+0000","last_active":"2024-03-23T03:46:59.544875+0000","last_peered":"2024-03-23T03:46:59.544875+0000","last_clean":"2024-03-23T03:46:59.544875+0000","last_became_active":"2024-03-23T03:46:57.977737+0000","last_became_peered":"2024-03-23T03:46:57.977737+0000","last_unstale":"2024-03-23T03:46:59.544875+0000","last_undegraded":"2024-03-23T03:46:59.544875+0000","last_fullsized":"2024-03-23T03:46:59.544875+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:09:10.919635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943234+0000","last_change":"2024-03-23T03:46:53.949224+0000","last_active":"2024-03-23T03:46:57.943234+0000","last_peered":"2024-03-23T03:46:57.943234+0000","last_clean":"2024-03-23T03:46:57.943234+0000","last_became_active":"2024-03-23T03:46:53.948869+0000","last_became_peered":"2024-03-23T03:46:53.948869+0000","last_unstale":"2024-03-23T03:46:57.943234+0000","last_undegraded":"2024-03-23T03:46:57.943234+0000","last_fullsized":"2024-03-23T03:46:57.943234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:49:34.782399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.948932+0000","last_change":"2024-03-23T03:46:53.461930+0000","last_active":"2024-03-23T03:46:57.948932+0000","last_peered":"2024-03-23T03:46:57.948932+0000","last_clean":"2024-03-23T03:46:57.948932+0000","last_became_active":"2024-03-23T03:46:53.461500+0000","last_became_peered":"2024-03-23T03:46:53.461500+0000","last_unstale":"2024-03-23T03:46:57.948932+0000","last_undegraded":"2024-03-23T03:46:57.948932+0000","last_fullsized":"2024-03-23T03:46:57.948932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:12:50.664977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545763+0000","last_change":"2024-03-23T03:46:57.978014+0000","last_active":"2024-03-23T03:46:59.545763+0000","last_peered":"2024-03-23T03:46:59.545763+0000","last_clean":"2024-03-23T03:46:59.545763+0000","last_became_active":"2024-03-23T03:46:57.977584+0000","last_became_peered":"2024-03-23T03:46:57.977584+0000","last_unstale":"2024-03-23T03:46:59.545763+0000","last_undegraded":"2024-03-23T03:46:59.545763+0000","last_fullsized":"2024-03-23T03:46:59.545763+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:17:26.245417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.950704+0000","last_change":"2024-03-23T03:46:53.968396+0000","last_active":"2024-03-23T03:46:57.950704+0000","last_peered":"2024-03-23T03:46:57.950704+0000","last_clean":"2024-03-23T03:46:57.950704+0000","last_became_active":"2024-03-23T03:46:53.968127+0000","last_became_peered":"2024-03-23T03:46:53.968127+0000","last_unstale":"2024-03-23T03:46:57.950704+0000","last_undegraded":"2024-03-23T03:46:57.950704+0000","last_fullsized":"2024-03-23T03:46:57.950704+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:56:29.065234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.012423+0000","last_change":"2024-03-23T03:46:53.472587+0000","last_active":"2024-03-23T03:46:58.012423+0000","last_peered":"2024-03-23T03:46:58.012423+0000","last_clean":"2024-03-23T03:46:58.012423+0000","last_became_active":"2024-03-23T03:46:53.472235+0000","last_became_peered":"2024-03-23T03:46:53.472235+0000","last_unstale":"2024-03-23T03:46:58.012423+0000","last_undegraded":"2024-03-23T03:46:58.012423+0000","last_fullsized":"2024-03-23T03:46:58.012423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:56.874824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411831+0000","last_change":"2024-03-23T03:46:58.016199+0000","last_active":"2024-03-23T03:46:59.411831+0000","last_peered":"2024-03-23T03:46:59.411831+0000","last_clean":"2024-03-23T03:46:59.411831+0000","last_became_active":"2024-03-23T03:46:58.015478+0000","last_became_peered":"2024-03-23T03:46:58.015478+0000","last_unstale":"2024-03-23T03:46:59.411831+0000","last_undegraded":"2024-03-23T03:46:59.411831+0000","last_fullsized":"2024-03-23T03:46:59.411831+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:56.623233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951639+0000","last_change":"2024-03-23T03:46:53.955152+0000","last_active":"2024-03-23T03:46:57.951639+0000","last_peered":"2024-03-23T03:46:57.951639+0000","last_clean":"2024-03-23T03:46:57.951639+0000","last_became_active":"2024-03-23T03:46:53.953527+0000","last_became_peered":"2024-03-23T03:46:53.953527+0000","last_unstale":"2024-03-23T03:46:57.951639+0000","last_undegraded":"2024-03-23T03:46:57.951639+0000","last_fullsized":"2024-03-23T03:46:57.951639+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:01:31.453936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970187+0000","last_change":"2024-03-23T03:46:53.462993+0000","last_active":"2024-03-23T03:46:57.970187+0000","last_peered":"2024-03-23T03:46:57.970187+0000","last_clean":"2024-03-23T03:46:57.970187+0000","last_became_active":"2024-03-23T03:46:53.462438+0000","last_became_peered":"2024-03-23T03:46:53.462438+0000","last_unstale":"2024-03-23T03:46:57.970187+0000","last_undegraded":"2024-03-23T03:46:57.970187+0000","last_fullsized":"2024-03-23T03:46:57.970187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:52:38.737363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":167,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.012529+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:46:58.012529+0000","last_peered":"2024-03-23T03:46:58.012529+0000","last_clean":"2024-03-23T03:46:58.012529+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:46:58.012529+0000","last_undegraded":"2024-03-23T03:46:58.012529+0000","last_fullsized":"2024-03-23T03:46:58.012529+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.547057+0000","last_change":"2024-03-23T03:46:58.018456+0000","last_active":"2024-03-23T03:46:59.547057+0000","last_peered":"2024-03-23T03:46:59.547057+0000","last_clean":"2024-03-23T03:46:59.547057+0000","last_became_active":"2024-03-23T03:46:58.018035+0000","last_became_peered":"2024-03-23T03:46:58.018035+0000","last_unstale":"2024-03-23T03:46:59.547057+0000","last_undegraded":"2024-03-23T03:46:59.547057+0000","last_fullsized":"2024-03-23T03:46:59.547057+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:44:17.122381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017645+0000","last_change":"2024-03-23T03:46:53.970202+0000","last_active":"2024-03-23T03:46:58.017645+0000","last_peered":"2024-03-23T03:46:58.017645+0000","last_clean":"2024-03-23T03:46:58.017645+0000","last_became_active":"2024-03-23T03:46:53.969976+0000","last_became_peered":"2024-03-23T03:46:53.969976+0000","last_unstale":"2024-03-23T03:46:58.017645+0000","last_undegraded":"2024-03-23T03:46:58.017645+0000","last_fullsized":"2024-03-23T03:46:58.017645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:59.541399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.947920+0000","last_change":"2024-03-23T03:46:53.478374+0000","last_active":"2024-03-23T03:46:57.947920+0000","last_peered":"2024-03-23T03:46:57.947920+0000","last_clean":"2024-03-23T03:46:57.947920+0000","last_became_active":"2024-03-23T03:46:53.478063+0000","last_became_peered":"2024-03-23T03:46:53.478063+0000","last_unstale":"2024-03-23T03:46:57.947920+0000","last_undegraded":"2024-03-23T03:46:57.947920+0000","last_fullsized":"2024-03-23T03:46:57.947920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:28:03.413525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.546398+0000","last_change":"2024-03-23T03:46:58.013147+0000","last_active":"2024-03-23T03:46:59.546398+0000","last_peered":"2024-03-23T03:46:59.546398+0000","last_clean":"2024-03-23T03:46:59.546398+0000","last_became_active":"2024-03-23T03:46:58.012728+0000","last_became_peered":"2024-03-23T03:46:58.012728+0000","last_unstale":"2024-03-23T03:46:59.546398+0000","last_undegraded":"2024-03-23T03:46:59.546398+0000","last_fullsized":"2024-03-23T03:46:59.546398+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:28:49.474199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970654+0000","last_change":"2024-03-23T03:46:53.956675+0000","last_active":"2024-03-23T03:46:57.970654+0000","last_peered":"2024-03-23T03:46:57.970654+0000","last_clean":"2024-03-23T03:46:57.970654+0000","last_became_active":"2024-03-23T03:46:53.956420+0000","last_became_peered":"2024-03-23T03:46:53.956420+0000","last_unstale":"2024-03-23T03:46:57.970654+0000","last_undegraded":"2024-03-23T03:46:57.970654+0000","last_fullsized":"2024-03-23T03:46:57.970654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:16:39.131183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943199+0000","last_change":"2024-03-23T03:46:53.467369+0000","last_active":"2024-03-23T03:46:57.943199+0000","last_peered":"2024-03-23T03:46:57.943199+0000","last_clean":"2024-03-23T03:46:57.943199+0000","last_became_active":"2024-03-23T03:46:53.465916+0000","last_became_peered":"2024-03-23T03:46:53.465916+0000","last_unstale":"2024-03-23T03:46:57.943199+0000","last_undegraded":"2024-03-23T03:46:57.943199+0000","last_fullsized":"2024-03-23T03:46:57.943199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:56.874824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411552+0000","last_change":"2024-03-23T03:46:57.974919+0000","last_active":"2024-03-23T03:46:59.411552+0000","last_peered":"2024-03-23T03:46:59.411552+0000","last_clean":"2024-03-23T03:46:59.411552+0000","last_became_active":"2024-03-23T03:46:57.972661+0000","last_became_peered":"2024-03-23T03:46:57.972661+0000","last_unstale":"2024-03-23T03:46:59.411552+0000","last_undegraded":"2024-03-23T03:46:59.411552+0000","last_fullsized":"2024-03-23T03:46:59.411552+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:27:37.992019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.018541+0000","last_change":"2024-03-23T03:46:53.965749+0000","last_active":"2024-03-23T03:46:58.018541+0000","last_peered":"2024-03-23T03:46:58.018541+0000","last_clean":"2024-03-23T03:46:58.018541+0000","last_became_active":"2024-03-23T03:46:53.965362+0000","last_became_peered":"2024-03-23T03:46:53.965362+0000","last_unstale":"2024-03-23T03:46:58.018541+0000","last_undegraded":"2024-03-23T03:46:58.018541+0000","last_fullsized":"2024-03-23T03:46:58.018541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:38:13.205855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.948005+0000","last_change":"2024-03-23T03:46:53.460818+0000","last_active":"2024-03-23T03:46:57.948005+0000","last_peered":"2024-03-23T03:46:57.948005+0000","last_clean":"2024-03-23T03:46:57.948005+0000","last_became_active":"2024-03-23T03:46:53.460547+0000","last_became_peered":"2024-03-23T03:46:53.460547+0000","last_unstale":"2024-03-23T03:46:57.948005+0000","last_undegraded":"2024-03-23T03:46:57.948005+0000","last_fullsized":"2024-03-23T03:46:57.948005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:49:24.952855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.424397+0000","last_change":"2024-03-23T03:46:58.014145+0000","last_active":"2024-03-23T03:46:58.424397+0000","last_peered":"2024-03-23T03:46:58.424397+0000","last_clean":"2024-03-23T03:46:58.424397+0000","last_became_active":"2024-03-23T03:46:58.013649+0000","last_became_peered":"2024-03-23T03:46:58.013649+0000","last_unstale":"2024-03-23T03:46:58.424397+0000","last_undegraded":"2024-03-23T03:46:58.424397+0000","last_fullsized":"2024-03-23T03:46:58.424397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:56.623233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017437+0000","last_change":"2024-03-23T03:46:53.966787+0000","last_active":"2024-03-23T03:46:58.017437+0000","last_peered":"2024-03-23T03:46:58.017437+0000","last_clean":"2024-03-23T03:46:58.017437+0000","last_became_active":"2024-03-23T03:46:53.966135+0000","last_became_peered":"2024-03-23T03:46:53.966135+0000","last_unstale":"2024-03-23T03:46:58.017437+0000","last_undegraded":"2024-03-23T03:46:58.017437+0000","last_fullsized":"2024-03-23T03:46:58.017437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:12.278522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.964838+0000","last_change":"2024-03-23T03:46:53.473134+0000","last_active":"2024-03-23T03:46:57.964838+0000","last_peered":"2024-03-23T03:46:57.964838+0000","last_clean":"2024-03-23T03:46:57.964838+0000","last_became_active":"2024-03-23T03:46:53.472902+0000","last_became_peered":"2024-03-23T03:46:53.472902+0000","last_unstale":"2024-03-23T03:46:57.964838+0000","last_undegraded":"2024-03-23T03:46:57.964838+0000","last_fullsized":"2024-03-23T03:46:57.964838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:42:16.552281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.423299+0000","last_change":"2024-03-23T03:46:57.976918+0000","last_active":"2024-03-23T03:46:58.423299+0000","last_peered":"2024-03-23T03:46:58.423299+0000","last_clean":"2024-03-23T03:46:58.423299+0000","last_became_active":"2024-03-23T03:46:57.976542+0000","last_became_peered":"2024-03-23T03:46:57.976542+0000","last_unstale":"2024-03-23T03:46:58.423299+0000","last_undegraded":"2024-03-23T03:46:58.423299+0000","last_fullsized":"2024-03-23T03:46:58.423299+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:49:33.207155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017030+0000","last_change":"2024-03-23T03:46:53.964676+0000","last_active":"2024-03-23T03:46:58.017030+0000","last_peered":"2024-03-23T03:46:58.017030+0000","last_clean":"2024-03-23T03:46:58.017030+0000","last_became_active":"2024-03-23T03:46:53.964382+0000","last_became_peered":"2024-03-23T03:46:53.964382+0000","last_unstale":"2024-03-23T03:46:58.017030+0000","last_undegraded":"2024-03-23T03:46:58.017030+0000","last_fullsized":"2024-03-23T03:46:58.017030+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:34:41.509208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.947240+0000","last_change":"2024-03-23T03:46:53.459242+0000","last_active":"2024-03-23T03:46:57.947240+0000","last_peered":"2024-03-23T03:46:57.947240+0000","last_clean":"2024-03-23T03:46:57.947240+0000","last_became_active":"2024-03-23T03:46:53.458136+0000","last_became_peered":"2024-03-23T03:46:53.458136+0000","last_unstale":"2024-03-23T03:46:57.947240+0000","last_undegraded":"2024-03-23T03:46:57.947240+0000","last_fullsized":"2024-03-23T03:46:57.947240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:42:13.208524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411874+0000","last_change":"2024-03-23T03:46:57.968727+0000","last_active":"2024-03-23T03:46:59.411874+0000","last_peered":"2024-03-23T03:46:59.411874+0000","last_clean":"2024-03-23T03:46:59.411874+0000","last_became_active":"2024-03-23T03:46:57.968336+0000","last_became_peered":"2024-03-23T03:46:57.968336+0000","last_unstale":"2024-03-23T03:46:59.411874+0000","last_undegraded":"2024-03-23T03:46:59.411874+0000","last_fullsized":"2024-03-23T03:46:59.411874+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:17:26.245417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951907+0000","last_change":"2024-03-23T03:46:53.967303+0000","last_active":"2024-03-23T03:46:57.951907+0000","last_peered":"2024-03-23T03:46:57.951907+0000","last_clean":"2024-03-23T03:46:57.951907+0000","last_became_active":"2024-03-23T03:46:53.966903+0000","last_became_peered":"2024-03-23T03:46:53.966903+0000","last_unstale":"2024-03-23T03:46:57.951907+0000","last_undegraded":"2024-03-23T03:46:57.951907+0000","last_fullsized":"2024-03-23T03:46:57.951907+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:18:17.089475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951906+0000","last_change":"2024-03-23T03:46:53.479049+0000","last_active":"2024-03-23T03:46:57.951906+0000","last_peered":"2024-03-23T03:46:57.951906+0000","last_clean":"2024-03-23T03:46:57.951906+0000","last_became_active":"2024-03-23T03:46:53.478779+0000","last_became_peered":"2024-03-23T03:46:53.478779+0000","last_unstale":"2024-03-23T03:46:57.951906+0000","last_undegraded":"2024-03-23T03:46:57.951906+0000","last_fullsized":"2024-03-23T03:46:57.951906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:28:03.413525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.547121+0000","last_change":"2024-03-23T03:46:58.018779+0000","last_active":"2024-03-23T03:46:59.547121+0000","last_peered":"2024-03-23T03:46:59.547121+0000","last_clean":"2024-03-23T03:46:59.547121+0000","last_became_active":"2024-03-23T03:46:58.018342+0000","last_became_peered":"2024-03-23T03:46:58.018342+0000","last_unstale":"2024-03-23T03:46:59.547121+0000","last_undegraded":"2024-03-23T03:46:59.547121+0000","last_fullsized":"2024-03-23T03:46:59.547121+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:49:30.580961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.016869+0000","last_change":"2024-03-23T03:46:53.966087+0000","last_active":"2024-03-23T03:46:58.016869+0000","last_peered":"2024-03-23T03:46:58.016869+0000","last_clean":"2024-03-23T03:46:58.016869+0000","last_became_active":"2024-03-23T03:46:53.965556+0000","last_became_peered":"2024-03-23T03:46:53.965556+0000","last_unstale":"2024-03-23T03:46:58.016869+0000","last_undegraded":"2024-03-23T03:46:58.016869+0000","last_fullsized":"2024-03-23T03:46:58.016869+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:35:49.855085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970252+0000","last_change":"2024-03-23T03:46:53.468499+0000","last_active":"2024-03-23T03:46:57.970252+0000","last_peered":"2024-03-23T03:46:57.970252+0000","last_clean":"2024-03-23T03:46:57.970252+0000","last_became_active":"2024-03-23T03:46:53.466565+0000","last_became_peered":"2024-03-23T03:46:53.466565+0000","last_unstale":"2024-03-23T03:46:57.970252+0000","last_undegraded":"2024-03-23T03:46:57.970252+0000","last_fullsized":"2024-03-23T03:46:57.970252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:38:29.605938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.546719+0000","last_change":"2024-03-23T03:46:58.013721+0000","last_active":"2024-03-23T03:46:59.546719+0000","last_peered":"2024-03-23T03:46:59.546719+0000","last_clean":"2024-03-23T03:46:59.546719+0000","last_became_active":"2024-03-23T03:46:58.012503+0000","last_became_peered":"2024-03-23T03:46:58.012503+0000","last_unstale":"2024-03-23T03:46:59.546719+0000","last_undegraded":"2024-03-23T03:46:59.546719+0000","last_fullsized":"2024-03-23T03:46:59.546719+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:54:55.461256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951186+0000","last_change":"2024-03-23T03:46:53.952982+0000","last_active":"2024-03-23T03:46:57.951186+0000","last_peered":"2024-03-23T03:46:57.951186+0000","last_clean":"2024-03-23T03:46:57.951186+0000","last_became_active":"2024-03-23T03:46:53.952707+0000","last_became_peered":"2024-03-23T03:46:53.952707+0000","last_unstale":"2024-03-23T03:46:57.951186+0000","last_undegraded":"2024-03-23T03:46:57.951186+0000","last_fullsized":"2024-03-23T03:46:57.951186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:01:24.719476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952254+0000","last_change":"2024-03-23T03:46:53.472959+0000","last_active":"2024-03-23T03:46:57.952254+0000","last_peered":"2024-03-23T03:46:57.952254+0000","last_clean":"2024-03-23T03:46:57.952254+0000","last_became_active":"2024-03-23T03:46:53.472652+0000","last_became_peered":"2024-03-23T03:46:53.472652+0000","last_unstale":"2024-03-23T03:46:57.952254+0000","last_undegraded":"2024-03-23T03:46:57.952254+0000","last_fullsized":"2024-03-23T03:46:57.952254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:48:00.747717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.426089+0000","last_change":"2024-03-23T03:46:58.019621+0000","last_active":"2024-03-23T03:46:58.426089+0000","last_peered":"2024-03-23T03:46:58.426089+0000","last_clean":"2024-03-23T03:46:58.426089+0000","last_became_active":"2024-03-23T03:46:58.018726+0000","last_became_peered":"2024-03-23T03:46:58.018726+0000","last_unstale":"2024-03-23T03:46:58.426089+0000","last_undegraded":"2024-03-23T03:46:58.426089+0000","last_fullsized":"2024-03-23T03:46:58.426089+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:18:24.184664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017563+0000","last_change":"2024-03-23T03:46:53.954977+0000","last_active":"2024-03-23T03:46:58.017563+0000","last_peered":"2024-03-23T03:46:58.017563+0000","last_clean":"2024-03-23T03:46:58.017563+0000","last_became_active":"2024-03-23T03:46:53.954667+0000","last_became_peered":"2024-03-23T03:46:53.954667+0000","last_unstale":"2024-03-23T03:46:58.017563+0000","last_undegraded":"2024-03-23T03:46:58.017563+0000","last_fullsized":"2024-03-23T03:46:58.017563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:16.232161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'8","reported_seq":23,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:47:04.440126+0000","last_change":"2024-03-23T03:46:53.469497+0000","last_active":"2024-03-23T03:47:04.440126+0000","last_peered":"2024-03-23T03:47:04.440126+0000","last_clean":"2024-03-23T03:47:04.440126+0000","last_became_active":"2024-03-23T03:46:53.469139+0000","last_became_peered":"2024-03-23T03:46:53.469139+0000","last_unstale":"2024-03-23T03:47:04.440126+0000","last_undegraded":"2024-03-23T03:47:04.440126+0000","last_fullsized":"2024-03-23T03:47:04.440126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:42:13.208524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3654,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545062+0000","last_change":"2024-03-23T03:46:57.965959+0000","last_active":"2024-03-23T03:46:59.545062+0000","last_peered":"2024-03-23T03:46:59.545062+0000","last_clean":"2024-03-23T03:46:59.545062+0000","last_became_active":"2024-03-23T03:46:57.965418+0000","last_became_peered":"2024-03-23T03:46:57.965418+0000","last_unstale":"2024-03-23T03:46:59.545062+0000","last_undegraded":"2024-03-23T03:46:59.545062+0000","last_fullsized":"2024-03-23T03:46:59.545062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:57:19.038735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.962037+0000","last_change":"2024-03-23T03:46:53.946234+0000","last_active":"2024-03-23T03:46:57.962037+0000","last_peered":"2024-03-23T03:46:57.962037+0000","last_clean":"2024-03-23T03:46:57.962037+0000","last_became_active":"2024-03-23T03:46:53.945188+0000","last_became_peered":"2024-03-23T03:46:53.945188+0000","last_unstale":"2024-03-23T03:46:57.962037+0000","last_undegraded":"2024-03-23T03:46:57.962037+0000","last_fullsized":"2024-03-23T03:46:57.962037+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:29:34.917375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943105+0000","last_change":"2024-03-23T03:46:53.474846+0000","last_active":"2024-03-23T03:46:57.943105+0000","last_peered":"2024-03-23T03:46:57.943105+0000","last_clean":"2024-03-23T03:46:57.943105+0000","last_became_active":"2024-03-23T03:46:53.474561+0000","last_became_peered":"2024-03-23T03:46:53.474561+0000","last_unstale":"2024-03-23T03:46:57.943105+0000","last_undegraded":"2024-03-23T03:46:57.943105+0000","last_fullsized":"2024-03-23T03:46:57.943105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:40:20.814830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.424404+0000","last_change":"2024-03-23T03:46:57.974187+0000","last_active":"2024-03-23T03:46:58.424404+0000","last_peered":"2024-03-23T03:46:58.424404+0000","last_clean":"2024-03-23T03:46:58.424404+0000","last_became_active":"2024-03-23T03:46:57.973629+0000","last_became_peered":"2024-03-23T03:46:57.973629+0000","last_unstale":"2024-03-23T03:46:58.424404+0000","last_undegraded":"2024-03-23T03:46:58.424404+0000","last_fullsized":"2024-03-23T03:46:58.424404+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:32:46.731709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.954425+0000","last_change":"2024-03-23T03:46:53.958810+0000","last_active":"2024-03-23T03:46:57.954425+0000","last_peered":"2024-03-23T03:46:57.954425+0000","last_clean":"2024-03-23T03:46:57.954425+0000","last_became_active":"2024-03-23T03:46:53.958534+0000","last_became_peered":"2024-03-23T03:46:53.958534+0000","last_unstale":"2024-03-23T03:46:57.954425+0000","last_undegraded":"2024-03-23T03:46:57.954425+0000","last_fullsized":"2024-03-23T03:46:57.954425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:01:31.453936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970435+0000","last_change":"2024-03-23T03:46:53.472164+0000","last_active":"2024-03-23T03:46:57.970435+0000","last_peered":"2024-03-23T03:46:57.970435+0000","last_clean":"2024-03-23T03:46:57.970435+0000","last_became_active":"2024-03-23T03:46:53.471488+0000","last_became_peered":"2024-03-23T03:46:53.471488+0000","last_unstale":"2024-03-23T03:46:57.970435+0000","last_undegraded":"2024-03-23T03:46:57.970435+0000","last_fullsized":"2024-03-23T03:46:57.970435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:56.874824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.544253+0000","last_change":"2024-03-23T03:46:57.967928+0000","last_active":"2024-03-23T03:46:59.544253+0000","last_peered":"2024-03-23T03:46:59.544253+0000","last_clean":"2024-03-23T03:46:59.544253+0000","last_became_active":"2024-03-23T03:46:57.967377+0000","last_became_peered":"2024-03-23T03:46:57.967377+0000","last_unstale":"2024-03-23T03:46:59.544253+0000","last_undegraded":"2024-03-23T03:46:59.544253+0000","last_fullsized":"2024-03-23T03:46:59.544253+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:58:12.948095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970395+0000","last_change":"2024-03-23T03:46:53.954230+0000","last_active":"2024-03-23T03:46:57.970395+0000","last_peered":"2024-03-23T03:46:57.970395+0000","last_clean":"2024-03-23T03:46:57.970395+0000","last_became_active":"2024-03-23T03:46:53.953991+0000","last_became_peered":"2024-03-23T03:46:53.953991+0000","last_unstale":"2024-03-23T03:46:57.970395+0000","last_undegraded":"2024-03-23T03:46:57.970395+0000","last_fullsized":"2024-03-23T03:46:57.970395+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:16.232161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.953457+0000","last_change":"2024-03-23T03:46:53.475516+0000","last_active":"2024-03-23T03:46:57.953457+0000","last_peered":"2024-03-23T03:46:57.953457+0000","last_clean":"2024-03-23T03:46:57.953457+0000","last_became_active":"2024-03-23T03:46:53.475185+0000","last_became_peered":"2024-03-23T03:46:53.475185+0000","last_unstale":"2024-03-23T03:46:57.953457+0000","last_undegraded":"2024-03-23T03:46:57.953457+0000","last_fullsized":"2024-03-23T03:46:57.953457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:05:58.548539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545880+0000","last_change":"2024-03-23T03:46:58.018887+0000","last_active":"2024-03-23T03:46:59.545880+0000","last_peered":"2024-03-23T03:46:59.545880+0000","last_clean":"2024-03-23T03:46:59.545880+0000","last_became_active":"2024-03-23T03:46:58.018514+0000","last_became_peered":"2024-03-23T03:46:58.018514+0000","last_unstale":"2024-03-23T03:46:59.545880+0000","last_undegraded":"2024-03-23T03:46:59.545880+0000","last_fullsized":"2024-03-23T03:46:59.545880+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:51:12.276163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411517+0000","last_change":"2024-03-23T03:46:58.015847+0000","last_active":"2024-03-23T03:46:59.411517+0000","last_peered":"2024-03-23T03:46:59.411517+0000","last_clean":"2024-03-23T03:46:59.411517+0000","last_became_active":"2024-03-23T03:46:58.015471+0000","last_became_peered":"2024-03-23T03:46:58.015471+0000","last_unstale":"2024-03-23T03:46:59.411517+0000","last_undegraded":"2024-03-23T03:46:59.411517+0000","last_fullsized":"2024-03-23T03:46:59.411517+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:34:34.288987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943156+0000","last_change":"2024-03-23T03:46:53.952670+0000","last_active":"2024-03-23T03:46:57.943156+0000","last_peered":"2024-03-23T03:46:57.943156+0000","last_clean":"2024-03-23T03:46:57.943156+0000","last_became_active":"2024-03-23T03:46:53.951520+0000","last_became_peered":"2024-03-23T03:46:53.951520+0000","last_unstale":"2024-03-23T03:46:57.943156+0000","last_undegraded":"2024-03-23T03:46:57.943156+0000","last_fullsized":"2024-03-23T03:46:57.943156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:55.749189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.968564+0000","last_change":"2024-03-23T03:46:53.475560+0000","last_active":"2024-03-23T03:46:57.968564+0000","last_peered":"2024-03-23T03:46:57.968564+0000","last_clean":"2024-03-23T03:46:57.968564+0000","last_became_active":"2024-03-23T03:46:53.468090+0000","last_became_peered":"2024-03-23T03:46:53.468090+0000","last_unstale":"2024-03-23T03:46:57.968564+0000","last_undegraded":"2024-03-23T03:46:57.968564+0000","last_fullsized":"2024-03-23T03:46:57.968564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:23:34.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.548770+0000","last_change":"2024-03-23T03:46:57.974834+0000","last_active":"2024-03-23T03:46:59.548770+0000","last_peered":"2024-03-23T03:46:59.548770+0000","last_clean":"2024-03-23T03:46:59.548770+0000","last_became_active":"2024-03-23T03:46:57.974450+0000","last_became_peered":"2024-03-23T03:46:57.974450+0000","last_unstale":"2024-03-23T03:46:59.548770+0000","last_undegraded":"2024-03-23T03:46:59.548770+0000","last_fullsized":"2024-03-23T03:46:59.548770+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:01.676599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.954004+0000","last_change":"2024-03-23T03:46:53.963896+0000","last_active":"2024-03-23T03:46:57.954004+0000","last_peered":"2024-03-23T03:46:57.954004+0000","last_clean":"2024-03-23T03:46:57.954004+0000","last_became_active":"2024-03-23T03:46:53.963072+0000","last_became_peered":"2024-03-23T03:46:53.963072+0000","last_unstale":"2024-03-23T03:46:57.954004+0000","last_undegraded":"2024-03-23T03:46:57.954004+0000","last_fullsized":"2024-03-23T03:46:57.954004+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:49:34.782399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.961727+0000","last_change":"2024-03-23T03:46:53.475321+0000","last_active":"2024-03-23T03:46:57.961727+0000","last_peered":"2024-03-23T03:46:57.961727+0000","last_clean":"2024-03-23T03:46:57.961727+0000","last_became_active":"2024-03-23T03:46:53.475119+0000","last_became_peered":"2024-03-23T03:46:53.475119+0000","last_unstale":"2024-03-23T03:46:57.961727+0000","last_undegraded":"2024-03-23T03:46:57.961727+0000","last_fullsized":"2024-03-23T03:46:57.961727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:48:00.747717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952537+0000","last_change":"2024-03-23T03:46:53.475209+0000","last_active":"2024-03-23T03:46:57.952537+0000","last_peered":"2024-03-23T03:46:57.952537+0000","last_clean":"2024-03-23T03:46:57.952537+0000","last_became_active":"2024-03-23T03:46:53.474046+0000","last_became_peered":"2024-03-23T03:46:53.474046+0000","last_unstale":"2024-03-23T03:46:57.952537+0000","last_undegraded":"2024-03-23T03:46:57.952537+0000","last_fullsized":"2024-03-23T03:46:57.952537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:04:07.631198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952167+0000","last_change":"2024-03-23T03:46:53.961251+0000","last_active":"2024-03-23T03:46:57.952167+0000","last_peered":"2024-03-23T03:46:57.952167+0000","last_clean":"2024-03-23T03:46:57.952167+0000","last_became_active":"2024-03-23T03:46:53.960887+0000","last_became_peered":"2024-03-23T03:46:53.960887+0000","last_unstale":"2024-03-23T03:46:57.952167+0000","last_undegraded":"2024-03-23T03:46:57.952167+0000","last_fullsized":"2024-03-23T03:46:57.952167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:51.878165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545740+0000","last_change":"2024-03-23T03:46:57.969555+0000","last_active":"2024-03-23T03:46:59.545740+0000","last_peered":"2024-03-23T03:46:59.545740+0000","last_clean":"2024-03-23T03:46:59.545740+0000","last_became_active":"2024-03-23T03:46:57.969163+0000","last_became_peered":"2024-03-23T03:46:57.969163+0000","last_unstale":"2024-03-23T03:46:59.545740+0000","last_undegraded":"2024-03-23T03:46:59.545740+0000","last_fullsized":"2024-03-23T03:46:59.545740+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:09.677468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952350+0000","last_change":"2024-03-23T03:46:53.463135+0000","last_active":"2024-03-23T03:46:57.952350+0000","last_peered":"2024-03-23T03:46:57.952350+0000","last_clean":"2024-03-23T03:46:57.952350+0000","last_became_active":"2024-03-23T03:46:53.462844+0000","last_became_peered":"2024-03-23T03:46:53.462844+0000","last_unstale":"2024-03-23T03:46:57.952350+0000","last_undegraded":"2024-03-23T03:46:57.952350+0000","last_fullsized":"2024-03-23T03:46:57.952350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:39:13.578639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.960303+0000","last_change":"2024-03-23T03:46:53.935183+0000","last_active":"2024-03-23T03:46:57.960303+0000","last_peered":"2024-03-23T03:46:57.960303+0000","last_clean":"2024-03-23T03:46:57.960303+0000","last_became_active":"2024-03-23T03:46:53.934902+0000","last_became_peered":"2024-03-23T03:46:53.934902+0000","last_unstale":"2024-03-23T03:46:57.960303+0000","last_undegraded":"2024-03-23T03:46:57.960303+0000","last_fullsized":"2024-03-23T03:46:57.960303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:19:03.389747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.546902+0000","last_change":"2024-03-23T03:46:57.971524+0000","last_active":"2024-03-23T03:46:59.546902+0000","last_peered":"2024-03-23T03:46:59.546902+0000","last_clean":"2024-03-23T03:46:59.546902+0000","last_became_active":"2024-03-23T03:46:57.967540+0000","last_became_peered":"2024-03-23T03:46:57.967540+0000","last_unstale":"2024-03-23T03:46:59.546902+0000","last_undegraded":"2024-03-23T03:46:59.546902+0000","last_fullsized":"2024-03-23T03:46:59.546902+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:36:58.375101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951620+0000","last_change":"2024-03-23T03:46:53.966166+0000","last_active":"2024-03-23T03:46:57.951620+0000","last_peered":"2024-03-23T03:46:57.951620+0000","last_clean":"2024-03-23T03:46:57.951620+0000","last_became_active":"2024-03-23T03:46:53.965850+0000","last_became_peered":"2024-03-23T03:46:53.965850+0000","last_unstale":"2024-03-23T03:46:57.951620+0000","last_undegraded":"2024-03-23T03:46:57.951620+0000","last_fullsized":"2024-03-23T03:46:57.951620+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:35:38.384483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.948826+0000","last_change":"2024-03-23T03:46:53.954723+0000","last_active":"2024-03-23T03:46:57.948826+0000","last_peered":"2024-03-23T03:46:57.948826+0000","last_clean":"2024-03-23T03:46:57.948826+0000","last_became_active":"2024-03-23T03:46:53.954419+0000","last_became_peered":"2024-03-23T03:46:53.954419+0000","last_unstale":"2024-03-23T03:46:57.948826+0000","last_undegraded":"2024-03-23T03:46:57.948826+0000","last_fullsized":"2024-03-23T03:46:57.948826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:55:23.986885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970164+0000","last_change":"2024-03-23T03:46:53.950337+0000","last_active":"2024-03-23T03:46:57.970164+0000","last_peered":"2024-03-23T03:46:57.970164+0000","last_clean":"2024-03-23T03:46:57.970164+0000","last_became_active":"2024-03-23T03:46:53.949979+0000","last_became_peered":"2024-03-23T03:46:53.949979+0000","last_unstale":"2024-03-23T03:46:57.970164+0000","last_undegraded":"2024-03-23T03:46:57.970164+0000","last_fullsized":"2024-03-23T03:46:57.970164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:36.598727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952086+0000","last_change":"2024-03-23T03:46:53.957785+0000","last_active":"2024-03-23T03:46:57.952086+0000","last_peered":"2024-03-23T03:46:57.952086+0000","last_clean":"2024-03-23T03:46:57.952086+0000","last_became_active":"2024-03-23T03:46:53.956251+0000","last_became_peered":"2024-03-23T03:46:53.956251+0000","last_unstale":"2024-03-23T03:46:57.952086+0000","last_undegraded":"2024-03-23T03:46:57.952086+0000","last_fullsized":"2024-03-23T03:46:57.952086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:49:34.782399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.964730+0000","last_change":"2024-03-23T03:46:53.950745+0000","last_active":"2024-03-23T03:46:57.964730+0000","last_peered":"2024-03-23T03:46:57.964730+0000","last_clean":"2024-03-23T03:46:57.964730+0000","last_became_active":"2024-03-23T03:46:53.950592+0000","last_became_peered":"2024-03-23T03:46:53.950592+0000","last_unstale":"2024-03-23T03:46:57.964730+0000","last_undegraded":"2024-03-23T03:46:57.964730+0000","last_fullsized":"2024-03-23T03:46:57.964730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:16:39.131183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.942799+0000","last_change":"2024-03-23T03:46:53.949596+0000","last_active":"2024-03-23T03:46:57.942799+0000","last_peered":"2024-03-23T03:46:57.942799+0000","last_clean":"2024-03-23T03:46:57.942799+0000","last_became_active":"2024-03-23T03:46:53.949298+0000","last_became_peered":"2024-03-23T03:46:53.949298+0000","last_unstale":"2024-03-23T03:46:57.942799+0000","last_undegraded":"2024-03-23T03:46:57.942799+0000","last_fullsized":"2024-03-23T03:46:57.942799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:47.801092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952735+0000","last_change":"2024-03-23T03:46:53.961179+0000","last_active":"2024-03-23T03:46:57.952735+0000","last_peered":"2024-03-23T03:46:57.952735+0000","last_clean":"2024-03-23T03:46:57.952735+0000","last_became_active":"2024-03-23T03:46:53.960616+0000","last_became_peered":"2024-03-23T03:46:53.960616+0000","last_unstale":"2024-03-23T03:46:57.952735+0000","last_undegraded":"2024-03-23T03:46:57.952735+0000","last_fullsized":"2024-03-23T03:46:57.952735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:51:48.524978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.971878+0000","last_change":"2024-03-23T03:46:53.951801+0000","last_active":"2024-03-23T03:46:57.971878+0000","last_peered":"2024-03-23T03:46:57.971878+0000","last_clean":"2024-03-23T03:46:57.971878+0000","last_became_active":"2024-03-23T03:46:53.951529+0000","last_became_peered":"2024-03-23T03:46:53.951529+0000","last_unstale":"2024-03-23T03:46:57.971878+0000","last_undegraded":"2024-03-23T03:46:57.971878+0000","last_fullsized":"2024-03-23T03:46:57.971878+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:36.598727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.968476+0000","last_change":"2024-03-23T03:46:53.950898+0000","last_active":"2024-03-23T03:46:57.968476+0000","last_peered":"2024-03-23T03:46:57.968476+0000","last_clean":"2024-03-23T03:46:57.968476+0000","last_became_active":"2024-03-23T03:46:53.950232+0000","last_became_peered":"2024-03-23T03:46:53.950232+0000","last_unstale":"2024-03-23T03:46:57.968476+0000","last_undegraded":"2024-03-23T03:46:57.968476+0000","last_fullsized":"2024-03-23T03:46:57.968476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:51.878165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.961869+0000","last_change":"2024-03-23T03:46:53.939297+0000","last_active":"2024-03-23T03:46:57.961869+0000","last_peered":"2024-03-23T03:46:57.961869+0000","last_clean":"2024-03-23T03:46:57.961869+0000","last_became_active":"2024-03-23T03:46:53.939052+0000","last_became_peered":"2024-03-23T03:46:53.939052+0000","last_unstale":"2024-03-23T03:46:57.961869+0000","last_undegraded":"2024-03-23T03:46:57.961869+0000","last_fullsized":"2024-03-23T03:46:57.961869+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:16.232161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970145+0000","last_change":"2024-03-23T03:46:53.953978+0000","last_active":"2024-03-23T03:46:57.970145+0000","last_peered":"2024-03-23T03:46:57.970145+0000","last_clean":"2024-03-23T03:46:57.970145+0000","last_became_active":"2024-03-23T03:46:53.953703+0000","last_became_peered":"2024-03-23T03:46:53.953703+0000","last_unstale":"2024-03-23T03:46:57.970145+0000","last_undegraded":"2024-03-23T03:46:57.970145+0000","last_fullsized":"2024-03-23T03:46:57.970145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:50:10.900452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951969+0000","last_change":"2024-03-23T03:46:53.969252+0000","last_active":"2024-03-23T03:46:57.951969+0000","last_peered":"2024-03-23T03:46:57.951969+0000","last_clean":"2024-03-23T03:46:57.951969+0000","last_became_active":"2024-03-23T03:46:53.968966+0000","last_became_peered":"2024-03-23T03:46:53.968966+0000","last_unstale":"2024-03-23T03:46:57.951969+0000","last_undegraded":"2024-03-23T03:46:57.951969+0000","last_fullsized":"2024-03-23T03:46:57.951969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:49.329280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.960849+0000","last_change":"2024-03-23T03:46:53.965172+0000","last_active":"2024-03-23T03:46:57.960849+0000","last_peered":"2024-03-23T03:46:57.960849+0000","last_clean":"2024-03-23T03:46:57.960849+0000","last_became_active":"2024-03-23T03:46:53.963673+0000","last_became_peered":"2024-03-23T03:46:53.963673+0000","last_unstale":"2024-03-23T03:46:57.960849+0000","last_undegraded":"2024-03-23T03:46:57.960849+0000","last_fullsized":"2024-03-23T03:46:57.960849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:52:35.361467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017879+0000","last_change":"2024-03-23T03:46:53.952664+0000","last_active":"2024-03-23T03:46:58.017879+0000","last_peered":"2024-03-23T03:46:58.017879+0000","last_clean":"2024-03-23T03:46:58.017879+0000","last_became_active":"2024-03-23T03:46:53.952264+0000","last_became_peered":"2024-03-23T03:46:53.952264+0000","last_unstale":"2024-03-23T03:46:58.017879+0000","last_undegraded":"2024-03-23T03:46:58.017879+0000","last_fullsized":"2024-03-23T03:46:58.017879+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:51.878165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.947849+0000","last_change":"2024-03-23T03:46:53.965686+0000","last_active":"2024-03-23T03:46:57.947849+0000","last_peered":"2024-03-23T03:46:57.947849+0000","last_clean":"2024-03-23T03:46:57.947849+0000","last_became_active":"2024-03-23T03:46:53.965321+0000","last_became_peered":"2024-03-23T03:46:53.965321+0000","last_unstale":"2024-03-23T03:46:57.947849+0000","last_undegraded":"2024-03-23T03:46:57.947849+0000","last_fullsized":"2024-03-23T03:46:57.947849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:29:34.917375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970735+0000","last_change":"2024-03-23T03:46:53.973376+0000","last_active":"2024-03-23T03:46:57.970735+0000","last_peered":"2024-03-23T03:46:57.970735+0000","last_clean":"2024-03-23T03:46:57.970735+0000","last_became_active":"2024-03-23T03:46:53.973131+0000","last_became_peered":"2024-03-23T03:46:53.973131+0000","last_unstale":"2024-03-23T03:46:57.970735+0000","last_undegraded":"2024-03-23T03:46:57.970735+0000","last_fullsized":"2024-03-23T03:46:57.970735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:01:04.097184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.968336+0000","last_change":"2024-03-23T03:46:53.964244+0000","last_active":"2024-03-23T03:46:57.968336+0000","last_peered":"2024-03-23T03:46:57.968336+0000","last_clean":"2024-03-23T03:46:57.968336+0000","last_became_active":"2024-03-23T03:46:53.963926+0000","last_became_peered":"2024-03-23T03:46:53.963926+0000","last_unstale":"2024-03-23T03:46:57.968336+0000","last_undegraded":"2024-03-23T03:46:57.968336+0000","last_fullsized":"2024-03-23T03:46:57.968336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:49.329280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943357+0000","last_change":"2024-03-23T03:46:53.938653+0000","last_active":"2024-03-23T03:46:57.943357+0000","last_peered":"2024-03-23T03:46:57.943357+0000","last_clean":"2024-03-23T03:46:57.943357+0000","last_became_active":"2024-03-23T03:46:53.938323+0000","last_became_peered":"2024-03-23T03:46:53.938323+0000","last_unstale":"2024-03-23T03:46:57.943357+0000","last_undegraded":"2024-03-23T03:46:57.943357+0000","last_fullsized":"2024-03-23T03:46:57.943357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:49.329280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951110+0000","last_change":"2024-03-23T03:46:53.956639+0000","last_active":"2024-03-23T03:46:57.951110+0000","last_peered":"2024-03-23T03:46:57.951110+0000","last_clean":"2024-03-23T03:46:57.951110+0000","last_became_active":"2024-03-23T03:46:53.956097+0000","last_became_peered":"2024-03-23T03:46:53.956097+0000","last_unstale":"2024-03-23T03:46:57.951110+0000","last_undegraded":"2024-03-23T03:46:57.951110+0000","last_fullsized":"2024-03-23T03:46:57.951110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:46.978451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951858+0000","last_change":"2024-03-23T03:46:53.964641+0000","last_active":"2024-03-23T03:46:57.951858+0000","last_peered":"2024-03-23T03:46:57.951858+0000","last_clean":"2024-03-23T03:46:57.951858+0000","last_became_active":"2024-03-23T03:46:53.964167+0000","last_became_peered":"2024-03-23T03:46:53.964167+0000","last_unstale":"2024-03-23T03:46:57.951858+0000","last_undegraded":"2024-03-23T03:46:57.951858+0000","last_fullsized":"2024-03-23T03:46:57.951858+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:55.749189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.965170+0000","last_change":"2024-03-23T03:46:53.962737+0000","last_active":"2024-03-23T03:46:57.965170+0000","last_peered":"2024-03-23T03:46:57.965170+0000","last_clean":"2024-03-23T03:46:57.965170+0000","last_became_active":"2024-03-23T03:46:53.962229+0000","last_became_peered":"2024-03-23T03:46:53.962229+0000","last_unstale":"2024-03-23T03:46:57.965170+0000","last_undegraded":"2024-03-23T03:46:57.965170+0000","last_fullsized":"2024-03-23T03:46:57.965170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:59.999542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.953048+0000","last_change":"2024-03-23T03:46:53.971706+0000","last_active":"2024-03-23T03:46:57.953048+0000","last_peered":"2024-03-23T03:46:57.953048+0000","last_clean":"2024-03-23T03:46:57.953048+0000","last_became_active":"2024-03-23T03:46:53.971525+0000","last_became_peered":"2024-03-23T03:46:53.971525+0000","last_unstale":"2024-03-23T03:46:57.953048+0000","last_undegraded":"2024-03-23T03:46:57.953048+0000","last_fullsized":"2024-03-23T03:46:57.953048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:55:54.119888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952505+0000","last_change":"2024-03-23T03:46:53.970600+0000","last_active":"2024-03-23T03:46:57.952505+0000","last_peered":"2024-03-23T03:46:57.952505+0000","last_clean":"2024-03-23T03:46:57.952505+0000","last_became_active":"2024-03-23T03:46:53.970340+0000","last_became_peered":"2024-03-23T03:46:53.970340+0000","last_unstale":"2024-03-23T03:46:57.952505+0000","last_undegraded":"2024-03-23T03:46:57.952505+0000","last_fullsized":"2024-03-23T03:46:57.952505+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:48:32.346809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017965+0000","last_change":"2024-03-23T03:46:53.971448+0000","last_active":"2024-03-23T03:46:58.017965+0000","last_peered":"2024-03-23T03:46:58.017965+0000","last_clean":"2024-03-23T03:46:58.017965+0000","last_became_active":"2024-03-23T03:46:53.971152+0000","last_became_peered":"2024-03-23T03:46:53.971152+0000","last_unstale":"2024-03-23T03:46:58.017965+0000","last_undegraded":"2024-03-23T03:46:58.017965+0000","last_fullsized":"2024-03-23T03:46:58.017965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:36.598727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.953820+0000","last_change":"2024-03-23T03:46:53.944836+0000","last_active":"2024-03-23T03:46:57.953820+0000","last_peered":"2024-03-23T03:46:57.953820+0000","last_clean":"2024-03-23T03:46:57.953820+0000","last_became_active":"2024-03-23T03:46:53.944614+0000","last_became_peered":"2024-03-23T03:46:53.944614+0000","last_unstale":"2024-03-23T03:46:57.953820+0000","last_undegraded":"2024-03-23T03:46:57.953820+0000","last_fullsized":"2024-03-23T03:46:57.953820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:01:24.719476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.013281+0000","last_change":"2024-03-23T03:46:53.957372+0000","last_active":"2024-03-23T03:46:58.013281+0000","last_peered":"2024-03-23T03:46:58.013281+0000","last_clean":"2024-03-23T03:46:58.013281+0000","last_became_active":"2024-03-23T03:46:53.956119+0000","last_became_peered":"2024-03-23T03:46:53.956119+0000","last_unstale":"2024-03-23T03:46:58.013281+0000","last_undegraded":"2024-03-23T03:46:58.013281+0000","last_fullsized":"2024-03-23T03:46:58.013281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:16:39.131183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.968302+0000","last_change":"2024-03-23T03:46:53.954765+0000","last_active":"2024-03-23T03:46:57.968302+0000","last_peered":"2024-03-23T03:46:57.968302+0000","last_clean":"2024-03-23T03:46:57.968302+0000","last_became_active":"2024-03-23T03:46:53.954424+0000","last_became_peered":"2024-03-23T03:46:53.954424+0000","last_unstale":"2024-03-23T03:46:57.968302+0000","last_undegraded":"2024-03-23T03:46:57.968302+0000","last_fullsized":"2024-03-23T03:46:57.968302+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:56:29.065234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.016764+0000","last_change":"2024-03-23T03:46:53.949427+0000","last_active":"2024-03-23T03:46:58.016764+0000","last_peered":"2024-03-23T03:46:58.016764+0000","last_clean":"2024-03-23T03:46:58.016764+0000","last_became_active":"2024-03-23T03:46:53.949114+0000","last_became_peered":"2024-03-23T03:46:53.949114+0000","last_unstale":"2024-03-23T03:46:58.016764+0000","last_undegraded":"2024-03-23T03:46:58.016764+0000","last_fullsized":"2024-03-23T03:46:58.016764+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:50:10.900452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970488+0000","last_change":"2024-03-23T03:46:53.956260+0000","last_active":"2024-03-23T03:46:57.970488+0000","last_peered":"2024-03-23T03:46:57.970488+0000","last_clean":"2024-03-23T03:46:57.970488+0000","last_became_active":"2024-03-23T03:46:53.955509+0000","last_became_peered":"2024-03-23T03:46:53.955509+0000","last_unstale":"2024-03-23T03:46:57.970488+0000","last_undegraded":"2024-03-23T03:46:57.970488+0000","last_fullsized":"2024-03-23T03:46:57.970488+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:34:41.509208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951671+0000","last_change":"2024-03-23T03:46:53.957607+0000","last_active":"2024-03-23T03:46:57.951671+0000","last_peered":"2024-03-23T03:46:57.951671+0000","last_clean":"2024-03-23T03:46:57.951671+0000","last_became_active":"2024-03-23T03:46:53.956236+0000","last_became_peered":"2024-03-23T03:46:53.956236+0000","last_unstale":"2024-03-23T03:46:57.951671+0000","last_undegraded":"2024-03-23T03:46:57.951671+0000","last_fullsized":"2024-03-23T03:46:57.951671+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:27:03.917141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943481+0000","last_change":"2024-03-23T03:46:53.960428+0000","last_active":"2024-03-23T03:46:57.943481+0000","last_peered":"2024-03-23T03:46:57.943481+0000","last_clean":"2024-03-23T03:46:57.943481+0000","last_became_active":"2024-03-23T03:46:53.960148+0000","last_became_peered":"2024-03-23T03:46:53.960148+0000","last_unstale":"2024-03-23T03:46:57.943481+0000","last_undegraded":"2024-03-23T03:46:57.943481+0000","last_fullsized":"2024-03-23T03:46:57.943481+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:01:24.719476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017880+0000","last_change":"2024-03-23T03:46:53.966166+0000","last_active":"2024-03-23T03:46:58.017880+0000","last_peered":"2024-03-23T03:46:58.017880+0000","last_clean":"2024-03-23T03:46:58.017880+0000","last_became_active":"2024-03-23T03:46:53.965871+0000","last_became_peered":"2024-03-23T03:46:53.965871+0000","last_unstale":"2024-03-23T03:46:58.017880+0000","last_undegraded":"2024-03-23T03:46:58.017880+0000","last_fullsized":"2024-03-23T03:46:58.017880+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:01:04.097184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4452,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":8904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":698548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1060000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.276}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0569999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3819999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542150,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":111252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2609999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.589}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3939999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4670000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4259999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542150,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":107688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.119}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2230000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2709999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":107724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.113}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2689999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.345}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2949999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":111252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.753}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2050000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4359999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0569999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.341}]}]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":107598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1859999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1439999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.468}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9119999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.97}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8360000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8120000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":107758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0960000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3080000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3420000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5229999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.871}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":880640,"data_stored":698604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3540000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.659}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.829}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8160000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:47:16.752 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T03:47:16.752 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-23T03:47:16.752 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-23T03:47:16.753 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-23T03:47:16.753 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-23T03:47:16.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.992+0000 7fab35e00640 1 Processor -- start 2024-03-23T03:47:16.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.992+0000 7fab35e00640 1 -- start start 2024-03-23T03:47:16.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.993+0000 7fab35e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fab300cbab0 0x7fab300b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:16.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.993+0000 7fab35e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3006d390 0x7fab3006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:16.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.994+0000 7fab35e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab300b16a0 con 0x7fab300adbc0 2024-03-23T03:47:16.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.994+0000 7fab35e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab3006dd60 con 0x7fab300cbab0 2024-03-23T03:47:16.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.994+0000 7fab35e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab300cbfb0 con 0x7fab3006d390 2024-03-23T03:47:16.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.994+0000 7fab2ea00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fab300adbc0 0x7fab300b41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41132/0 (socket says 172.21.15.102:41132) 2024-03-23T03:47:16.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.994+0000 7fab2ea00640 1 -- 172.21.15.102:0/3049012398 learned_addr learned my addr 172.21.15.102:0/3049012398 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:16.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.994+0000 7fab2e000640 1 -- 172.21.15.102:0/3049012398 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201055591 0 0) 0x7fab300b16a0 con 0x7fab300adbc0 2024-03-23T03:47:16.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.994+0000 7fab2e000640 1 -- 172.21.15.102:0/3049012398 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab14003520 con 0x7fab300adbc0 2024-03-23T03:47:16.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.994+0000 7fab27e00640 1 --2- 172.21.15.102:0/3049012398 >> v2:172.21.15.173:3300/0 conn(0x7fab300cbab0 0x7fab300b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.994+0000 7fab2f400640 1 --2- 172.21.15.102:0/3049012398 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3006d390 0x7fab3006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.994+0000 7fab2e000640 1 -- 172.21.15.102:0/3049012398 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 642353713 0 0) 0x7fab14003520 con 0x7fab300adbc0 2024-03-23T03:47:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2e000640 1 -- 172.21.15.102:0/3049012398 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3006d390 msgr2=0x7fab3006d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2e000640 1 --2- 172.21.15.102:0/3049012398 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3006d390 0x7fab3006d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2e000640 1 -- 172.21.15.102:0/3049012398 >> v2:172.21.15.173:3300/0 conn(0x7fab300cbab0 msgr2=0x7fab300b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2e000640 1 --2- 172.21.15.102:0/3049012398 >> v2:172.21.15.173:3300/0 conn(0x7fab300cbab0 0x7fab300b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2e000640 1 -- 172.21.15.102:0/3049012398 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab300cc290 con 0x7fab300adbc0 2024-03-23T03:47:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2e000640 1 -- 172.21.15.102:0/3049012398 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fab20001670 con 0x7fab300adbc0 2024-03-23T03:47:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2e000640 1 -- 172.21.15.102:0/3049012398 >> v1:172.21.15.102:6789/0 conn(0x7fab300adbc0 legacy=0x7fab300b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2e000640 1 --2- 172.21.15.102:0/3049012398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab140039a0 0x7fab14003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2e000640 1 --2- 172.21.15.102:0/3049012398 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3006d390 0x7fab140047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:16.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2e000640 1 --2- 172.21.15.102:0/3049012398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab14004d00 0x7fab14005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:16.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2e000640 1 -- 172.21.15.102:0/3049012398 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3006dd60 con 0x7fab140039a0 2024-03-23T03:47:16.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2e000640 1 -- 172.21.15.102:0/3049012398 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab300cbfb0 con 0x7fab14004d00 2024-03-23T03:47:16.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2e000640 1 -- 172.21.15.102:0/3049012398 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab1400bb00 con 0x7fab3006d390 2024-03-23T03:47:16.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2ea00640 1 --2- 172.21.15.102:0/3049012398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab140039a0 0x7fab14003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:16.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2f400640 1 --2- 172.21.15.102:0/3049012398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab14004d00 0x7fab14005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:16.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab27e00640 1 --2- 172.21.15.102:0/3049012398 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3006d390 0x7fab140047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:16.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2ea00640 1 -- 172.21.15.102:0/3049012398 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3006d390 msgr2=0x7fab140047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:16.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2ea00640 1 --2- 172.21.15.102:0/3049012398 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3006d390 0x7fab140047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2ea00640 1 -- 172.21.15.102:0/3049012398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab14004d00 msgr2=0x7fab14005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:16.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2ea00640 1 --2- 172.21.15.102:0/3049012398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab14004d00 0x7fab14005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:16.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.995+0000 7fab2ea00640 1 -- 172.21.15.102:0/3049012398 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab14003520 con 0x7fab140039a0 2024-03-23T03:47:16.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.996+0000 7fab2f400640 1 --2- 172.21.15.102:0/3049012398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab14004d00 0x7fab14005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:16.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.997+0000 7fab2ea00640 1 --2- 172.21.15.102:0/3049012398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab140039a0 0x7fab14003dc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fab2000c240 tx=0x7fab140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd32b09a2109f204 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:16.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.997+0000 7fab2e000640 1 -- 172.21.15.102:0/3049012398 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab20005ad0 con 0x7fab140039a0 2024-03-23T03:47:17.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.997+0000 7fab2e000640 1 -- 172.21.15.102:0/3049012398 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab20005c30 con 0x7fab140039a0 2024-03-23T03:47:17.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.997+0000 7fab2e000640 1 -- 172.21.15.102:0/3049012398 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2002c8b0 con 0x7fab140039a0 2024-03-23T03:47:17.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.997+0000 7fab35e00640 1 -- 172.21.15.102:0/3049012398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab140039a0 msgr2=0x7fab14003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:17.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.997+0000 7fab35e00640 1 --2- 172.21.15.102:0/3049012398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab140039a0 0x7fab14003dc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fab2000c240 tx=0x7fab140057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.997+0000 7fab2f400640 1 -- 172.21.15.102:0/3049012398 reap_dead start 2024-03-23T03:47:17.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.997+0000 7fab35e00640 1 -- 172.21.15.102:0/3049012398 shutdown_connections 2024-03-23T03:47:17.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.997+0000 7fab35e00640 1 -- 172.21.15.102:0/3049012398 >> 172.21.15.102:0/3049012398 conn(0x7fab3006a3d0 msgr2=0x7fab300b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:17.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.998+0000 7fab35e00640 1 -- 172.21.15.102:0/3049012398 shutdown_connections 2024-03-23T03:47:17.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.998+0000 7fab35e00640 1 -- 172.21.15.102:0/3049012398 wait complete. 2024-03-23T03:47:17.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.999+0000 7fab35e00640 1 Processor -- start 2024-03-23T03:47:17.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:16.999+0000 7fab35e00640 1 -- start start 2024-03-23T03:47:17.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.000+0000 7fab35e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab140039a0 0x7fab3015af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.000+0000 7fab35e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3015b4c0 0x7fab3006c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.000+0000 7fab35e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3006c7c0 0x7fab3013dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.000+0000 7fab35e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3006cc40 con 0x7fab3006c7c0 2024-03-23T03:47:17.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.000+0000 7fab35e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3015bfe0 con 0x7fab140039a0 2024-03-23T03:47:17.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.000+0000 7fab35e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3015ba20 con 0x7fab3015b4c0 2024-03-23T03:47:17.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.001+0000 7fab2f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3006c7c0 0x7fab3013dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.001+0000 7fab2f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3006c7c0 0x7fab3013dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37144/0 (socket says 172.21.15.102:37144) 2024-03-23T03:47:17.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.001+0000 7fab2f400640 1 -- 172.21.15.102:0/112482873 learned_addr learned my addr 172.21.15.102:0/112482873 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:17.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.001+0000 7fab2ea00640 1 --2- 172.21.15.102:0/112482873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab140039a0 0x7fab3015af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.001+0000 7fab27e00640 1 --2- 172.21.15.102:0/112482873 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3015b4c0 0x7fab3006c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.001+0000 7fab2f400640 1 -- 172.21.15.102:0/112482873 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3015b4c0 msgr2=0x7fab3006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:17.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.001+0000 7fab2f400640 1 --2- 172.21.15.102:0/112482873 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3015b4c0 0x7fab3006c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.001+0000 7fab2f400640 1 -- 172.21.15.102:0/112482873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab140039a0 msgr2=0x7fab3015af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:17.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.001+0000 7fab2f400640 1 --2- 172.21.15.102:0/112482873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab140039a0 0x7fab3015af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.001+0000 7fab2f400640 1 -- 172.21.15.102:0/112482873 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3015bd00 con 0x7fab3006c7c0 2024-03-23T03:47:17.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.001+0000 7fab2f400640 1 --2- 172.21.15.102:0/112482873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3006c7c0 0x7fab3013dc40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fab30071440 tx=0x7fab28009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:17.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.002+0000 7fab27400640 1 -- 172.21.15.102:0/112482873 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab28007cc0 con 0x7fab3006c7c0 2024-03-23T03:47:17.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.002+0000 7fab27400640 1 -- 172.21.15.102:0/112482873 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab28007e40 con 0x7fab3006c7c0 2024-03-23T03:47:17.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.002+0000 7fab35e00640 1 -- 172.21.15.102:0/112482873 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab3013d440 con 0x7fab3006c7c0 2024-03-23T03:47:17.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.002+0000 7fab27400640 1 -- 172.21.15.102:0/112482873 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab28014640 con 0x7fab3006c7c0 2024-03-23T03:47:17.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.002+0000 7fab35e00640 1 -- 172.21.15.102:0/112482873 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3015ca90 con 0x7fab3006c7c0 2024-03-23T03:47:17.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.003+0000 7fab35e00640 1 -- 172.21.15.102:0/112482873 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3013d960 con 0x7fab3006c7c0 2024-03-23T03:47:17.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.004+0000 7fab27400640 1 -- 172.21.15.102:0/112482873 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fab2801d3b0 con 0x7fab3006c7c0 2024-03-23T03:47:17.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.004+0000 7fab27400640 1 --2- 172.21.15.102:0/112482873 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab0c06ac30 0x7fab0c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.005+0000 7fab27400640 1 -- 172.21.15.102:0/112482873 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fab28089320 con 0x7fab3006c7c0 2024-03-23T03:47:17.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.005+0000 7fab2ea00640 1 --2- 172.21.15.102:0/112482873 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab0c06ac30 0x7fab0c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.005+0000 7fab2ea00640 1 --2- 172.21.15.102:0/112482873 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab0c06ac30 0x7fab0c06d0f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fab20034000 tx=0x7fab20009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:17.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.009+0000 7fab27400640 1 -- 172.21.15.102:0/112482873 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fab3013d960 con 0x7fab3006c7c0 2024-03-23T03:47:17.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.252+0000 7fab35e00640 1 -- 172.21.15.102:0/112482873 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fab3013d960 con 0x7fab3006c7c0 2024-03-23T03:47:17.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.258+0000 7fab27400640 1 -- 172.21.15.102:0/112482873 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170932 (secure 0 0 0) 0x7fab3013d960 con 0x7fab3006c7c0 2024-03-23T03:47:17.261 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.262+0000 7fab35e00640 1 -- 172.21.15.102:0/112482873 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab0c06ac30 msgr2=0x7fab0c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.262+0000 7fab35e00640 1 --2- 172.21.15.102:0/112482873 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab0c06ac30 0x7fab0c06d0f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fab20034000 tx=0x7fab20009030 comp rx=0 tx=0).stop 2024-03-23T03:47:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.262+0000 7fab35e00640 1 -- 172.21.15.102:0/112482873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3006c7c0 msgr2=0x7fab3013dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.262+0000 7fab35e00640 1 --2- 172.21.15.102:0/112482873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3006c7c0 0x7fab3013dc40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fab30071440 tx=0x7fab28009cb0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.262+0000 7fab35e00640 1 -- 172.21.15.102:0/112482873 shutdown_connections 2024-03-23T03:47:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.262+0000 7fab35e00640 1 --2- 172.21.15.102:0/112482873 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab0c06ac30 0x7fab0c06d0f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.262+0000 7fab35e00640 1 --2- 172.21.15.102:0/112482873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3006c7c0 0x7fab3013dc40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.262+0000 7fab35e00640 1 --2- 172.21.15.102:0/112482873 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3015b4c0 0x7fab3006c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.262+0000 7fab35e00640 1 --2- 172.21.15.102:0/112482873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab140039a0 0x7fab3015af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.262+0000 7fab35e00640 1 -- 172.21.15.102:0/112482873 >> 172.21.15.102:0/112482873 conn(0x7fab3006a3d0 msgr2=0x7fab30138790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.263+0000 7fab35e00640 1 -- 172.21.15.102:0/112482873 shutdown_connections 2024-03-23T03:47:17.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.263+0000 7fab35e00640 1 -- 172.21.15.102:0/112482873 wait complete. 2024-03-23T03:47:17.281 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":5,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":37097},{"type":"v1","addr":"172.21.15.102:6833","nonce":37097}]},"active_addr":"172.21.15.102:6833/37097","active_change":"2024-03-23T03:46:39.697872+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4114,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":842277532}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":4001941115}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":521195975}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.102:0","nonce":3036292470}]}]} 2024-03-23T03:47:17.286 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-23T03:47:17.286 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-23T03:47:17.286 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:17.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.499+0000 7fa2dda00640 1 Processor -- start 2024-03-23T03:47:17.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.500+0000 7fa2dda00640 1 -- start start 2024-03-23T03:47:17.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.501+0000 7fa2dda00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa2d80adbc0 0x7fa2d80b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.501+0000 7fa2dda00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2d806d390 0x7fa2d806d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.501+0000 7fa2dda00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2d80b16a0 con 0x7fa2d80cbab0 2024-03-23T03:47:17.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.501+0000 7fa2dda00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2d806dd40 con 0x7fa2d80adbc0 2024-03-23T03:47:17.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.501+0000 7fa2dda00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d80b4700 con 0x7fa2d806d390 2024-03-23T03:47:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.502+0000 7fa2d6a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa2d80adbc0 0x7fa2d80b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.502+0000 7fa2d6a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa2d80adbc0 0x7fa2d80b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:42134/0 (socket says 172.21.15.102:42134) 2024-03-23T03:47:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.502+0000 7fa2d6a00640 1 -- 172.21.15.102:0/4198883348 learned_addr learned my addr 172.21.15.102:0/4198883348 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.502+0000 7fa2d7400640 1 --2- 172.21.15.102:0/4198883348 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2d806d390 0x7fa2d806d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.502+0000 7fa2d6a00640 1 -- 172.21.15.102:0/4198883348 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2d806d390 msgr2=0x7fa2d806d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.502+0000 7fa2d6a00640 1 --2- 172.21.15.102:0/4198883348 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2d806d390 0x7fa2d806d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.502+0000 7fa2d6a00640 1 -- 172.21.15.102:0/4198883348 >> v1:172.21.15.173:6789/0 conn(0x7fa2d80cbab0 legacy=0x7fa2d80b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:17.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.502+0000 7fa2d6a00640 1 -- 172.21.15.102:0/4198883348 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d80cbfb0 con 0x7fa2d80adbc0 2024-03-23T03:47:17.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.504+0000 7fa2d6a00640 1 --2- 172.21.15.102:0/4198883348 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2d80adbc0 0x7fa2d80b41c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009870 tx=0x7fa2c802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb657f0f1b5e7665 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:17.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.504+0000 7fa2d5600640 1 -- 172.21.15.102:0/4198883348 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8031640 con 0x7fa2d80adbc0 2024-03-23T03:47:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.504+0000 7fa2d5600640 1 -- 172.21.15.102:0/4198883348 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2c80317a0 con 0x7fa2d80adbc0 2024-03-23T03:47:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.504+0000 7fa2d5600640 1 -- 172.21.15.102:0/4198883348 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2c8031b30 con 0x7fa2d80adbc0 2024-03-23T03:47:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.505+0000 7fa2dda00640 1 -- 172.21.15.102:0/4198883348 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2d80adbc0 msgr2=0x7fa2d80b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:17.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.505+0000 7fa2dda00640 1 --2- 172.21.15.102:0/4198883348 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2d80adbc0 0x7fa2d80b41c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8009870 tx=0x7fa2c802b8f0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.505+0000 7fa2dda00640 1 -- 172.21.15.102:0/4198883348 shutdown_connections 2024-03-23T03:47:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.505+0000 7fa2dda00640 1 --2- 172.21.15.102:0/4198883348 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2d806d390 0x7fa2d806d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.505+0000 7fa2dda00640 1 --2- 172.21.15.102:0/4198883348 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2d80adbc0 0x7fa2d80b41c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.505+0000 7fa2dda00640 1 -- 172.21.15.102:0/4198883348 >> 172.21.15.102:0/4198883348 conn(0x7fa2d806a3d0 msgr2=0x7fa2d80b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.505+0000 7fa2dda00640 1 -- 172.21.15.102:0/4198883348 shutdown_connections 2024-03-23T03:47:17.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.505+0000 7fa2dda00640 1 -- 172.21.15.102:0/4198883348 wait complete. 2024-03-23T03:47:17.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.507+0000 7fa2dda00640 1 Processor -- start 2024-03-23T03:47:17.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.507+0000 7fa2dda00640 1 -- start start 2024-03-23T03:47:17.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.508+0000 7fa2dda00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2d806d390 0x7fa2d80a74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.508+0000 7fa2dda00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2d80adbc0 0x7fa2d81326b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.508+0000 7fa2dda00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2d80cbab0 0x7fa2d80a9070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.508+0000 7fa2dda00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d8132bf0 con 0x7fa2d80cbab0 2024-03-23T03:47:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.508+0000 7fa2dda00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d80a7a80 con 0x7fa2d806d390 2024-03-23T03:47:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.508+0000 7fa2dda00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d80a7d60 con 0x7fa2d80adbc0 2024-03-23T03:47:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.509+0000 7fa2d7400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2d80cbab0 0x7fa2d80a9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.509+0000 7fa2d7400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2d80cbab0 0x7fa2d80a9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37148/0 (socket says 172.21.15.102:37148) 2024-03-23T03:47:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.509+0000 7fa2d7400640 1 -- 172.21.15.102:0/3866650162 learned_addr learned my addr 172.21.15.102:0/3866650162 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.509+0000 7fa2d6a00640 1 --2- 172.21.15.102:0/3866650162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2d806d390 0x7fa2d80a74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.509+0000 7fa2d6000640 1 --2- 172.21.15.102:0/3866650162 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2d80adbc0 0x7fa2d81326b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.509+0000 7fa2d7400640 1 -- 172.21.15.102:0/3866650162 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2d80adbc0 msgr2=0x7fa2d81326b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.509+0000 7fa2d7400640 1 --2- 172.21.15.102:0/3866650162 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2d80adbc0 0x7fa2d81326b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.509+0000 7fa2d7400640 1 -- 172.21.15.102:0/3866650162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2d806d390 msgr2=0x7fa2d80a74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.509+0000 7fa2d7400640 1 --2- 172.21.15.102:0/3866650162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2d806d390 0x7fa2d80a74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.509+0000 7fa2d7400640 1 -- 172.21.15.102:0/3866650162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c80094d0 con 0x7fa2d80cbab0 2024-03-23T03:47:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.510+0000 7fa2d7400640 1 --2- 172.21.15.102:0/3866650162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2d80cbab0 0x7fa2d80a9070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa2d000efc0 tx=0x7fa2d000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:17.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.510+0000 7fa2cea00640 1 -- 172.21.15.102:0/3866650162 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d0009ba0 con 0x7fa2d80cbab0 2024-03-23T03:47:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.510+0000 7fa2cea00640 1 -- 172.21.15.102:0/3866650162 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d000f040 con 0x7fa2d80cbab0 2024-03-23T03:47:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.510+0000 7fa2cea00640 1 -- 172.21.15.102:0/3866650162 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d00146c0 con 0x7fa2d80cbab0 2024-03-23T03:47:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.510+0000 7fa2dda00640 1 -- 172.21.15.102:0/3866650162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d80a8040 con 0x7fa2d80cbab0 2024-03-23T03:47:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.510+0000 7fa2dda00640 1 -- 172.21.15.102:0/3866650162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d80a8600 con 0x7fa2d80cbab0 2024-03-23T03:47:17.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.511+0000 7fa2dda00640 1 -- 172.21.15.102:0/3866650162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d806ac20 con 0x7fa2d80cbab0 2024-03-23T03:47:17.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.512+0000 7fa2cea00640 1 -- 172.21.15.102:0/3866650162 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa2d0014820 con 0x7fa2d80cbab0 2024-03-23T03:47:17.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.513+0000 7fa2cea00640 1 --2- 172.21.15.102:0/3866650162 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa2b806ac30 0x7fa2b806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.513+0000 7fa2cea00640 1 -- 172.21.15.102:0/3866650162 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa2d008a0d0 con 0x7fa2d80cbab0 2024-03-23T03:47:17.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.513+0000 7fa2d6a00640 1 --2- 172.21.15.102:0/3866650162 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa2b806ac30 0x7fa2b806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.514+0000 7fa2d6a00640 1 --2- 172.21.15.102:0/3866650162 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa2b806ac30 0x7fa2b806d0f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8036000 tx=0x7fa2c803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:17.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.518+0000 7fa2cea00640 1 -- 172.21.15.102:0/3866650162 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa2d0055050 con 0x7fa2d80cbab0 2024-03-23T03:47:17.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.711+0000 7fa2dda00640 1 -- 172.21.15.102:0/3866650162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2d8073390 con 0x7fa2d80cbab0 2024-03-23T03:47:17.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.714+0000 7fa2cea00640 1 -- 172.21.15.102:0/3866650162 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17926 (secure 0 0 0) 0x7fa2d0058670 con 0x7fa2d80cbab0 2024-03-23T03:47:17.715 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:17.715 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:58.410551+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:47:17.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.716+0000 7fa2dda00640 1 -- 172.21.15.102:0/3866650162 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa2b806ac30 msgr2=0x7fa2b806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:17.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.716+0000 7fa2dda00640 1 --2- 172.21.15.102:0/3866650162 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa2b806ac30 0x7fa2b806d0f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa2c8036000 tx=0x7fa2c803c040 comp rx=0 tx=0).stop 2024-03-23T03:47:17.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.716+0000 7fa2dda00640 1 -- 172.21.15.102:0/3866650162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2d80cbab0 msgr2=0x7fa2d80a9070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:17.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.716+0000 7fa2dda00640 1 --2- 172.21.15.102:0/3866650162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2d80cbab0 0x7fa2d80a9070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa2d000efc0 tx=0x7fa2d000c490 comp rx=0 tx=0).stop 2024-03-23T03:47:17.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.716+0000 7fa2dda00640 1 -- 172.21.15.102:0/3866650162 shutdown_connections 2024-03-23T03:47:17.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.716+0000 7fa2dda00640 1 --2- 172.21.15.102:0/3866650162 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa2b806ac30 0x7fa2b806d0f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.716+0000 7fa2dda00640 1 --2- 172.21.15.102:0/3866650162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2d80cbab0 0x7fa2d80a9070 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.716+0000 7fa2dda00640 1 --2- 172.21.15.102:0/3866650162 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2d80adbc0 0x7fa2d81326b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.716+0000 7fa2dda00640 1 --2- 172.21.15.102:0/3866650162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2d806d390 0x7fa2d80a74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.716+0000 7fa2dda00640 1 -- 172.21.15.102:0/3866650162 >> 172.21.15.102:0/3866650162 conn(0x7fa2d806a3d0 msgr2=0x7fa2d80b2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.717+0000 7fa2dda00640 1 -- 172.21.15.102:0/3866650162 shutdown_connections 2024-03-23T03:47:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.717+0000 7fa2dda00640 1 -- 172.21.15.102:0/3866650162 wait complete. 2024-03-23T03:47:17.736 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-23T03:47:17.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:17.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.945+0000 7fea45600640 1 Processor -- start 2024-03-23T03:47:17.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.946+0000 7fea45600640 1 -- start start 2024-03-23T03:47:17.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.947+0000 7fea45600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fea40143460 0x7fea4015aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.947+0000 7fea45600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fea400cde20 0x7fea40070020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.947+0000 7fea45600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea400cc6b0 con 0x7fea40143930 2024-03-23T03:47:17.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.947+0000 7fea45600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea400cc950 con 0x7fea400cde20 2024-03-23T03:47:17.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.947+0000 7fea45600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea400ccc30 con 0x7fea40143460 2024-03-23T03:47:17.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.948+0000 7fea3f400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fea40143930 0x7fea400cd710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41138/0 (socket says 172.21.15.102:41138) 2024-03-23T03:47:17.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.948+0000 7fea3f400640 1 -- 172.21.15.102:0/3630634590 learned_addr learned my addr 172.21.15.102:0/3630634590 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:17.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.948+0000 7fea3fe00640 1 --2- 172.21.15.102:0/3630634590 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fea40143460 0x7fea4015aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.948+0000 7fea44c00640 1 --2- 172.21.15.102:0/3630634590 >> v2:172.21.15.173:3300/0 conn(0x7fea400cde20 0x7fea40070020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.948+0000 7fea3ea00640 1 -- 172.21.15.102:0/3630634590 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461142001 0 0) 0x7fea400cc6b0 con 0x7fea40143930 2024-03-23T03:47:17.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.948+0000 7fea3ea00640 1 -- 172.21.15.102:0/3630634590 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea24003520 con 0x7fea40143930 2024-03-23T03:47:17.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.948+0000 7fea3ea00640 1 -- 172.21.15.102:0/3630634590 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4023537654 0 0) 0x7fea24003520 con 0x7fea40143930 2024-03-23T03:47:17.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.948+0000 7fea3ea00640 1 -- 172.21.15.102:0/3630634590 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fea40143460 msgr2=0x7fea4015aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:17.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.948+0000 7fea3ea00640 1 --2- 172.21.15.102:0/3630634590 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fea40143460 0x7fea4015aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.948+0000 7fea3ea00640 1 -- 172.21.15.102:0/3630634590 >> v2:172.21.15.173:3300/0 conn(0x7fea400cde20 msgr2=0x7fea40070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:17.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.948+0000 7fea3ea00640 1 --2- 172.21.15.102:0/3630634590 >> v2:172.21.15.173:3300/0 conn(0x7fea400cde20 0x7fea40070020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.948+0000 7fea3ea00640 1 -- 172.21.15.102:0/3630634590 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea400ccf10 con 0x7fea40143930 2024-03-23T03:47:17.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.948+0000 7fea3ea00640 1 -- 172.21.15.102:0/3630634590 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fea2c001670 con 0x7fea40143930 2024-03-23T03:47:17.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.949+0000 7fea3ea00640 1 -- 172.21.15.102:0/3630634590 >> v1:172.21.15.102:6789/0 conn(0x7fea40143930 legacy=0x7fea400cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:17.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.949+0000 7fea3ea00640 1 --2- 172.21.15.102:0/3630634590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea24003970 0x7fea24003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.949+0000 7fea3ea00640 1 --2- 172.21.15.102:0/3630634590 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fea40143460 0x7fea240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.949+0000 7fea3ea00640 1 --2- 172.21.15.102:0/3630634590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea24004d00 0x7fea24005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.949+0000 7fea3ea00640 1 -- 172.21.15.102:0/3630634590 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea400cc950 con 0x7fea24004d00 2024-03-23T03:47:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.949+0000 7fea3ea00640 1 -- 172.21.15.102:0/3630634590 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea400ccc30 con 0x7fea24003970 2024-03-23T03:47:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.949+0000 7fea3ea00640 1 -- 172.21.15.102:0/3630634590 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea2400b980 con 0x7fea40143460 2024-03-23T03:47:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.949+0000 7fea44c00640 1 --2- 172.21.15.102:0/3630634590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea24004d00 0x7fea24005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.949+0000 7fea44c00640 1 -- 172.21.15.102:0/3630634590 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fea40143460 msgr2=0x7fea240047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.949+0000 7fea3f400640 1 --2- 172.21.15.102:0/3630634590 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fea40143460 0x7fea240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.949+0000 7fea44c00640 1 --2- 172.21.15.102:0/3630634590 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fea40143460 0x7fea240047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.949+0000 7fea3fe00640 1 --2- 172.21.15.102:0/3630634590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea24003970 0x7fea24003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.949+0000 7fea44c00640 1 -- 172.21.15.102:0/3630634590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea24003970 msgr2=0x7fea24003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.949+0000 7fea44c00640 1 --2- 172.21.15.102:0/3630634590 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea24003970 0x7fea24003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.950+0000 7fea44c00640 1 -- 172.21.15.102:0/3630634590 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea24003520 con 0x7fea24004d00 2024-03-23T03:47:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.951+0000 7fea44c00640 1 --2- 172.21.15.102:0/3630634590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea24004d00 0x7fea24005180 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fea38009fc0 tx=0x7fea240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df161b26cc4c4398 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.951+0000 7fea3ea00640 1 -- 172.21.15.102:0/3630634590 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3802cc60 con 0x7fea24004d00 2024-03-23T03:47:17.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.951+0000 7fea3ea00640 1 -- 172.21.15.102:0/3630634590 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3800f040 con 0x7fea24004d00 2024-03-23T03:47:17.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.951+0000 7fea3ea00640 1 -- 172.21.15.102:0/3630634590 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea38034440 con 0x7fea24004d00 2024-03-23T03:47:17.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.951+0000 7fea45600640 1 -- 172.21.15.102:0/3630634590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea24004d00 msgr2=0x7fea24005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:17.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.951+0000 7fea45600640 1 --2- 172.21.15.102:0/3630634590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea24004d00 0x7fea24005180 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fea38009fc0 tx=0x7fea240057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.952+0000 7fea44c00640 1 -- 172.21.15.102:0/3630634590 reap_dead start 2024-03-23T03:47:17.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.952+0000 7fea45600640 1 -- 172.21.15.102:0/3630634590 shutdown_connections 2024-03-23T03:47:17.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.952+0000 7fea45600640 1 -- 172.21.15.102:0/3630634590 >> 172.21.15.102:0/3630634590 conn(0x7fea40065df0 msgr2=0x7fea40069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:17.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.952+0000 7fea45600640 1 -- 172.21.15.102:0/3630634590 shutdown_connections 2024-03-23T03:47:17.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.952+0000 7fea45600640 1 -- 172.21.15.102:0/3630634590 wait complete. 2024-03-23T03:47:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.954+0000 7fea45600640 1 Processor -- start 2024-03-23T03:47:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.955+0000 7fea45600640 1 -- start start 2024-03-23T03:47:17.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.956+0000 7fea45600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fea24004d00 0x7fea400cac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.956+0000 7fea45600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea4013ad50 0x7fea401413f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.956+0000 7fea45600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea4013b1b0 0x7fea4014bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.956+0000 7fea45600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea40140f60 con 0x7fea4013ad50 2024-03-23T03:47:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.956+0000 7fea45600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea40140350 con 0x7fea4013b1b0 2024-03-23T03:47:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.956+0000 7fea45600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea40140630 con 0x7fea24004d00 2024-03-23T03:47:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.956+0000 7fea3f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea4013ad50 0x7fea401413f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.956+0000 7fea3f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea4013ad50 0x7fea401413f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37168/0 (socket says 172.21.15.102:37168) 2024-03-23T03:47:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.956+0000 7fea3f400640 1 -- 172.21.15.102:0/1798998211 learned_addr learned my addr 172.21.15.102:0/1798998211 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.956+0000 7fea3fe00640 1 --2- 172.21.15.102:0/1798998211 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fea24004d00 0x7fea400cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.956+0000 7fea44c00640 1 --2- 172.21.15.102:0/1798998211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea4013b1b0 0x7fea4014bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.956+0000 7fea3f400640 1 -- 172.21.15.102:0/1798998211 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fea24004d00 msgr2=0x7fea400cac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.956+0000 7fea3f400640 1 --2- 172.21.15.102:0/1798998211 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fea24004d00 0x7fea400cac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.956+0000 7fea3f400640 1 -- 172.21.15.102:0/1798998211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea4013b1b0 msgr2=0x7fea4014bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.956+0000 7fea3f400640 1 --2- 172.21.15.102:0/1798998211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea4013b1b0 0x7fea4014bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.956+0000 7fea3f400640 1 -- 172.21.15.102:0/1798998211 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea40140910 con 0x7fea4013ad50 2024-03-23T03:47:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.957+0000 7fea3f400640 1 --2- 172.21.15.102:0/1798998211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea4013ad50 0x7fea401413f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fea2c00d810 tx=0x7fea2c00dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.957+0000 7fea37e00640 1 -- 172.21.15.102:0/1798998211 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c0074e0 con 0x7fea4013ad50 2024-03-23T03:47:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.957+0000 7fea37e00640 1 -- 172.21.15.102:0/1798998211 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea2c007640 con 0x7fea4013ad50 2024-03-23T03:47:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.957+0000 7fea37e00640 1 -- 172.21.15.102:0/1798998211 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2c0049e0 con 0x7fea4013ad50 2024-03-23T03:47:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.957+0000 7fea45600640 1 -- 172.21.15.102:0/1798998211 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea40140bf0 con 0x7fea4013ad50 2024-03-23T03:47:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.957+0000 7fea45600640 1 -- 172.21.15.102:0/1798998211 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea400c3a40 con 0x7fea4013ad50 2024-03-23T03:47:17.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.958+0000 7fea45600640 1 -- 172.21.15.102:0/1798998211 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea401410d0 con 0x7fea4013ad50 2024-03-23T03:47:17.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.959+0000 7fea37e00640 1 -- 172.21.15.102:0/1798998211 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fea2c004b40 con 0x7fea4013ad50 2024-03-23T03:47:17.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.960+0000 7fea37e00640 1 --2- 172.21.15.102:0/1798998211 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fea2006ac00 0x7fea2006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:17.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.960+0000 7fea37e00640 1 -- 172.21.15.102:0/1798998211 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fea2c088fb0 con 0x7fea4013ad50 2024-03-23T03:47:17.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.961+0000 7fea3fe00640 1 --2- 172.21.15.102:0/1798998211 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fea2006ac00 0x7fea2006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:17.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.961+0000 7fea3fe00640 1 --2- 172.21.15.102:0/1798998211 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fea2006ac00 0x7fea2006d0c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fea30009910 tx=0x7fea30008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:17.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:17.965+0000 7fea37e00640 1 -- 172.21.15.102:0/1798998211 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fea2c053e90 con 0x7fea4013ad50 2024-03-23T03:47:18.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.158+0000 7fea45600640 1 -- 172.21.15.102:0/1798998211 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea4004e030 con 0x7fea4013ad50 2024-03-23T03:47:18.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.161+0000 7fea37e00640 1 -- 172.21.15.102:0/1798998211 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17926 (secure 0 0 0) 0x7fea2c019020 con 0x7fea4013ad50 2024-03-23T03:47:18.162 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:18.162 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:58.410551+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:47:18.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.164+0000 7fea45600640 1 -- 172.21.15.102:0/1798998211 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fea2006ac00 msgr2=0x7fea2006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.164+0000 7fea45600640 1 --2- 172.21.15.102:0/1798998211 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fea2006ac00 0x7fea2006d0c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fea30009910 tx=0x7fea30008040 comp rx=0 tx=0).stop 2024-03-23T03:47:18.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.164+0000 7fea45600640 1 -- 172.21.15.102:0/1798998211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea4013ad50 msgr2=0x7fea401413f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.164+0000 7fea45600640 1 --2- 172.21.15.102:0/1798998211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea4013ad50 0x7fea401413f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fea2c00d810 tx=0x7fea2c00dcd0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.164+0000 7fea45600640 1 -- 172.21.15.102:0/1798998211 shutdown_connections 2024-03-23T03:47:18.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.164+0000 7fea45600640 1 --2- 172.21.15.102:0/1798998211 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fea2006ac00 0x7fea2006d0c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.164+0000 7fea45600640 1 --2- 172.21.15.102:0/1798998211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea4013b1b0 0x7fea4014bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.164+0000 7fea45600640 1 --2- 172.21.15.102:0/1798998211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea4013ad50 0x7fea401413f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.164+0000 7fea45600640 1 --2- 172.21.15.102:0/1798998211 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fea24004d00 0x7fea400cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.164+0000 7fea45600640 1 -- 172.21.15.102:0/1798998211 >> 172.21.15.102:0/1798998211 conn(0x7fea40065df0 msgr2=0x7fea4015a080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.164+0000 7fea45600640 1 -- 172.21.15.102:0/1798998211 shutdown_connections 2024-03-23T03:47:18.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.164+0000 7fea45600640 1 -- 172.21.15.102:0/1798998211 wait complete. 2024-03-23T03:47:18.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-23T03:47:18.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-23T03:47:18.185 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-23T03:47:18.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-23T03:47:18.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-23T03:47:18.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-23T03:47:18.187 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-23T03:47:18.187 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-23T03:47:18.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.473+0000 7f235e200640 1 Processor -- start 2024-03-23T03:47:18.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.473+0000 7f235e200640 1 -- start start 2024-03-23T03:47:18.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.475+0000 7f235e200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f23580cdaa0 0x7f23581603d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.475+0000 7f235e200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f23580b7fe0 0x7f23580d10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.475+0000 7f235e200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2358161350 con 0x7f235815c7b0 2024-03-23T03:47:18.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.475+0000 7f235e200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23580cfcf0 con 0x7f23580b7fe0 2024-03-23T03:47:18.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.475+0000 7f235e200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2358160ab0 con 0x7f23580cdaa0 2024-03-23T03:47:18.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.475+0000 7f235ce00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f235815c7b0 0x7f23580cd390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41144/0 (socket says 172.21.15.102:41144) 2024-03-23T03:47:18.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.475+0000 7f235ce00640 1 -- 172.21.15.102:0/1308875210 learned_addr learned my addr 172.21.15.102:0/1308875210 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:18.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.476+0000 7f235d800640 1 --2- 172.21.15.102:0/1308875210 >> v2:172.21.15.102:3300/0 conn(0x7f23580b7fe0 0x7f23580d10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.476+0000 7f2357e00640 1 --2- 172.21.15.102:0/1308875210 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f23580cdaa0 0x7f23581603d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.477+0000 7f235d800640 1 -- 172.21.15.102:0/1308875210 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f23580cdaa0 msgr2=0x7f23581603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.477+0000 7f235d800640 1 --2- 172.21.15.102:0/1308875210 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f23580cdaa0 0x7f23581603d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.477+0000 7f235d800640 1 -- 172.21.15.102:0/1308875210 >> v1:172.21.15.102:6789/0 conn(0x7f235815c7b0 legacy=0x7f23580cd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.477+0000 7f2357400640 1 -- 172.21.15.102:0/1308875210 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771968983 0 0) 0x7f2358161350 con 0x7f235815c7b0 2024-03-23T03:47:18.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.477+0000 7f235d800640 1 -- 172.21.15.102:0/1308875210 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2358160d90 con 0x7f23580b7fe0 2024-03-23T03:47:18.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.478+0000 7fb670a00640 1 Processor -- start 2024-03-23T03:47:18.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.479+0000 7f235d800640 1 --2- 172.21.15.102:0/1308875210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23580b7fe0 0x7f23580d10e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f23480097a0 tx=0x7f234802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f073cd9943b2c3a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.479+0000 7fb670a00640 1 -- start start 2024-03-23T03:47:18.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.479+0000 7f2357400640 1 -- 172.21.15.102:0/1308875210 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234802e070 con 0x7f23580b7fe0 2024-03-23T03:47:18.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.480+0000 7f2357400640 1 -- 172.21.15.102:0/1308875210 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23480315a0 con 0x7f23580b7fe0 2024-03-23T03:47:18.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.480+0000 7f2357400640 1 -- 172.21.15.102:0/1308875210 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f234802e070 con 0x7f23580b7fe0 2024-03-23T03:47:18.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.479+0000 7f4c18400640 1 Processor -- start 2024-03-23T03:47:18.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.479+0000 7f4c18400640 1 -- start start 2024-03-23T03:47:18.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.481+0000 7f4c18400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4c100cefe0 0x7f4c1006c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.481+0000 7f4c18400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f4c1006d330 0x7f4c101648d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.481+0000 7f4c18400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c1006cee0 con 0x7f4c10160c90 2024-03-23T03:47:18.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.481+0000 7f235e200640 1 -- 172.21.15.102:0/1308875210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23580b7fe0 msgr2=0x7f23580d10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.481+0000 7f235e200640 1 --2- 172.21.15.102:0/1308875210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23580b7fe0 0x7f23580d10e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f23480097a0 tx=0x7f234802b840 comp rx=0 tx=0).stop 2024-03-23T03:47:18.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.482+0000 7fb670a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb66c15e7a0 0x7fb66c0b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.482+0000 7f4c16400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f4c1006d330 0x7f4c101648d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.482+0000 7f4c16400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f4c1006d330 0x7f4c101648d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:42196/0 (socket says 172.21.15.102:42196) 2024-03-23T03:47:18.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.482+0000 7f4c16400640 1 -- 172.21.15.102:0/709886957 learned_addr learned my addr 172.21.15.102:0/709886957 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.482+0000 7f4c18400640 1 -- 172.21.15.102:0/709886957 --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c100ae1b0 con 0x7f4c100cefe0 2024-03-23T03:47:18.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.482+0000 7f4c18400640 1 -- 172.21.15.102:0/709886957 --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c100b15c0 con 0x7f4c1006d330 2024-03-23T03:47:18.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.482+0000 7f4c15000640 1 --2- 172.21.15.102:0/709886957 >> v2:172.21.15.102:3300/0 conn(0x7f4c100cefe0 0x7f4c1006c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.482+0000 7fb669e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb66c15e7a0 0x7fb66c0b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.483+0000 7f235e200640 1 -- 172.21.15.102:0/1308875210 shutdown_connections 2024-03-23T03:47:18.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.483+0000 7f235e200640 1 --2- 172.21.15.102:0/1308875210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23580b7fe0 0x7f23580d10e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.483+0000 7f235e200640 1 --2- 172.21.15.102:0/1308875210 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f23580cdaa0 0x7f23581603d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.483+0000 7f235e200640 1 -- 172.21.15.102:0/1308875210 >> 172.21.15.102:0/1308875210 conn(0x7f235806ae90 msgr2=0x7f235806b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.483+0000 7f4c0fe00640 1 -- 172.21.15.102:0/709886957 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493068772 0 0) 0x7f4c08001490 con 0x7f4c10160c90 2024-03-23T03:47:18.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.483+0000 7f235e200640 1 -- 172.21.15.102:0/1308875210 shutdown_connections 2024-03-23T03:47:18.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.483+0000 7f4c0fe00640 1 -- 172.21.15.102:0/709886957 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bf8003520 con 0x7f4c10160c90 2024-03-23T03:47:18.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.483+0000 7f4c16400640 1 -- 172.21.15.102:0/709886957 >> v2:172.21.15.102:3300/0 conn(0x7f4c100cefe0 msgr2=0x7f4c1006c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.483+0000 7fb669400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb66c0adc60 0x7fb66c0b2b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41152/0 (socket says 172.21.15.102:41152) 2024-03-23T03:47:18.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.483+0000 7fb669e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb66c15e7a0 0x7fb66c0b1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:54318/0 (socket says 172.21.15.102:54318) 2024-03-23T03:47:18.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.483+0000 7fb669e00640 1 -- 172.21.15.102:0/3200057276 learned_addr learned my addr 172.21.15.102:0/3200057276 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.483+0000 7fb670a00640 1 -- 172.21.15.102:0/3200057276 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66c15eb80 con 0x7fb66c0adc60 2024-03-23T03:47:18.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.483+0000 7fb670a00640 1 -- 172.21.15.102:0/3200057276 --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66c0b24a0 con 0x7fb66c06da40 2024-03-23T03:47:18.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.483+0000 7fb670a00640 1 -- 172.21.15.102:0/3200057276 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c0b3a20 con 0x7fb66c15e7a0 2024-03-23T03:47:18.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.484+0000 7fb669e00640 1 -- 172.21.15.102:0/3200057276 >> v1:172.21.15.173:6789/0 conn(0x7fb66c06da40 legacy=0x7fb66c1646f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T03:47:18.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.484+0000 7fb669e00640 1 -- 172.21.15.102:0/3200057276 >> v1:172.21.15.102:6789/0 conn(0x7fb66c0adc60 legacy=0x7fb66c0b2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.484+0000 7fb669e00640 1 -- 172.21.15.102:0/3200057276 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c0b3460 con 0x7fb66c15e7a0 2024-03-23T03:47:18.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.484+0000 7f4c0fe00640 1 -- 172.21.15.102:0/709886957 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1821099312 0 0) 0x7f4bf8003520 con 0x7f4c10160c90 2024-03-23T03:47:18.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.483+0000 7f235e200640 1 -- 172.21.15.102:0/1308875210 wait complete. 2024-03-23T03:47:18.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.484+0000 7f4c16400640 1 --2- 172.21.15.102:0/709886957 >> v2:172.21.15.102:3300/0 conn(0x7f4c100cefe0 0x7f4c1006c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.484+0000 7f4c16400640 1 -- 172.21.15.102:0/709886957 >> v1:172.21.15.173:6789/0 conn(0x7f4c10160c90 legacy=0x7f4c100ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.484+0000 7f4c16400640 1 -- 172.21.15.102:0/709886957 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c1006d860 con 0x7f4c1006d330 2024-03-23T03:47:18.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.485+0000 7f4c15000640 1 --2- 172.21.15.102:0/709886957 >> v2:172.21.15.102:3300/0 conn(0x7f4c100cefe0 0x7f4c1006c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:47:18.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.486+0000 7f4c16400640 1 --2- 172.21.15.102:0/709886957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c1006d330 0x7f4c101648d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4c100b3750 tx=0x7f4bfc02b800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88a5486213d36a09 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.486+0000 7f235e200640 1 Processor -- start 2024-03-23T03:47:18.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.486+0000 7f235e200640 1 -- start start 2024-03-23T03:47:18.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.487+0000 7f4c0fe00640 1 -- 172.21.15.102:0/709886957 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c08001b80 con 0x7f4c1006d330 2024-03-23T03:47:18.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.487+0000 7fb669e00640 1 --2- 172.21.15.102:0/3200057276 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb66c15e7a0 0x7fb66c0b1f60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0098a0 tx=0x7fb65c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eeea0eeb554e99c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.487+0000 7f4c0fe00640 1 -- 172.21.15.102:0/709886957 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bfc02e050 con 0x7f4c1006d330 2024-03-23T03:47:18.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.487+0000 7fb668a00640 1 -- 172.21.15.102:0/3200057276 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c031640 con 0x7fb66c15e7a0 2024-03-23T03:47:18.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.487+0000 7fb668a00640 1 -- 172.21.15.102:0/3200057276 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65c0317a0 con 0x7fb66c15e7a0 2024-03-23T03:47:18.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.487+0000 7f235e200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23580b7fe0 0x7f23580c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.488+0000 7fb668a00640 1 -- 172.21.15.102:0/3200057276 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c031b30 con 0x7fb66c15e7a0 2024-03-23T03:47:18.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.488+0000 7f4c0fe00640 1 -- 172.21.15.102:0/709886957 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c08001b80 con 0x7f4c1006d330 2024-03-23T03:47:18.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.487+0000 7f235e200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23580cdaa0 0x7f2358148890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.487+0000 7f235e200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f235815c7b0 0x7f23580b4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.487+0000 7f235e200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23580b3710 con 0x7f23580b7fe0 2024-03-23T03:47:18.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.487+0000 7f235e200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23580b4290 con 0x7f23580cdaa0 2024-03-23T03:47:18.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.487+0000 7f235e200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23580b3960 con 0x7f235815c7b0 2024-03-23T03:47:18.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.488+0000 7f4c18400640 1 -- 172.21.15.102:0/709886957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c1006d330 msgr2=0x7f4c101648d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.488+0000 7f4c18400640 1 --2- 172.21.15.102:0/709886957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c1006d330 0x7f4c101648d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4c100b3750 tx=0x7f4bfc02b800 comp rx=0 tx=0).stop 2024-03-23T03:47:18.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.488+0000 7fae28e00640 1 Processor -- start 2024-03-23T03:47:18.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.488+0000 7f235ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23580b7fe0 0x7f23580c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.488+0000 7f235ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23580b7fe0 0x7f23580c8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37190/0 (socket says 172.21.15.102:37190) 2024-03-23T03:47:18.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.488+0000 7f235ce00640 1 -- 172.21.15.102:0/2644647259 learned_addr learned my addr 172.21.15.102:0/2644647259 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.488+0000 7f4c18400640 1 -- 172.21.15.102:0/709886957 shutdown_connections 2024-03-23T03:47:18.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.488+0000 7f4c18400640 1 --2- 172.21.15.102:0/709886957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c1006d330 0x7f4c101648d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.488+0000 7f4c18400640 1 --2- 172.21.15.102:0/709886957 >> v2:172.21.15.102:3300/0 conn(0x7f4c100cefe0 0x7f4c1006c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.488+0000 7f4c18400640 1 -- 172.21.15.102:0/709886957 >> 172.21.15.102:0/709886957 conn(0x7f4c10069c90 msgr2=0x7f4c1006a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.489+0000 7f2357e00640 1 --2- 172.21.15.102:0/2644647259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23580cdaa0 0x7f2358148890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.489+0000 7f235ce00640 1 -- 172.21.15.102:0/2644647259 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f235815c7b0 msgr2=0x7f23580b4720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-23T03:47:18.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.489+0000 7fb670a00640 1 -- 172.21.15.102:0/3200057276 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb66c15e7a0 msgr2=0x7fb66c0b1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.489+0000 7fb670a00640 1 --2- 172.21.15.102:0/3200057276 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb66c15e7a0 0x7fb66c0b1f60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb65c0098a0 tx=0x7fb65c02b8f0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.489+0000 7fb670a00640 1 -- 172.21.15.102:0/3200057276 shutdown_connections 2024-03-23T03:47:18.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.489+0000 7fb670a00640 1 --2- 172.21.15.102:0/3200057276 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb66c15e7a0 0x7fb66c0b1f60 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.489+0000 7fb670a00640 1 -- 172.21.15.102:0/3200057276 >> 172.21.15.102:0/3200057276 conn(0x7fb66c066000 msgr2=0x7fb66c069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.489+0000 7f4c18400640 1 -- 172.21.15.102:0/709886957 shutdown_connections 2024-03-23T03:47:18.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.489+0000 7f4c18400640 1 -- 172.21.15.102:0/709886957 wait complete. 2024-03-23T03:47:18.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.489+0000 7f235d800640 1 --2- 172.21.15.102:0/2644647259 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f235815c7b0 0x7f23580b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.490+0000 7f235ce00640 1 --2- 172.21.15.102:0/2644647259 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f235815c7b0 0x7f23580b4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.490+0000 7fb670a00640 1 -- 172.21.15.102:0/3200057276 shutdown_connections 2024-03-23T03:47:18.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.490+0000 7f235ce00640 1 -- 172.21.15.102:0/2644647259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23580cdaa0 msgr2=0x7f2358148890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.490+0000 7f235ce00640 1 --2- 172.21.15.102:0/2644647259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23580cdaa0 0x7f2358148890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.490+0000 7f235ce00640 1 -- 172.21.15.102:0/2644647259 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2348009400 con 0x7f23580b7fe0 2024-03-23T03:47:18.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.490+0000 7f2357e00640 1 --2- 172.21.15.102:0/2644647259 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f23580cdaa0 0x7f2358148890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:18.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.490+0000 7f235ce00640 1 --2- 172.21.15.102:0/2644647259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23580b7fe0 0x7f23580c8c90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f235000c7b0 tx=0x7f235000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.490+0000 7fb670a00640 1 -- 172.21.15.102:0/3200057276 wait complete. 2024-03-23T03:47:18.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.488+0000 7fae28e00640 1 -- start start 2024-03-23T03:47:18.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.491+0000 7fae28e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fae241627c0 0x7fae2406d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.491+0000 7fae28e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae241657c0 con 0x7fae24165340 2024-03-23T03:47:18.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.491+0000 7fae28e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae2406cdc0 con 0x7fae240ce930 2024-03-23T03:47:18.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.491+0000 7fae28e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae24161ab0 con 0x7fae241627c0 2024-03-23T03:47:18.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.491+0000 7f018ae00640 1 Processor -- start 2024-03-23T03:47:18.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.490+0000 7f2354c00640 1 -- 172.21.15.102:0/2644647259 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235000a640 con 0x7f23580b7fe0 2024-03-23T03:47:18.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.490+0000 7f2354c00640 1 -- 172.21.15.102:0/2644647259 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235000a7a0 con 0x7f23580b7fe0 2024-03-23T03:47:18.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.490+0000 7f2354c00640 1 -- 172.21.15.102:0/2644647259 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f235000ab90 con 0x7f23580b7fe0 2024-03-23T03:47:18.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.491+0000 7f235e200640 1 -- 172.21.15.102:0/2644647259 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23580b3c40 con 0x7f23580b7fe0 2024-03-23T03:47:18.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.491+0000 7f235e200640 1 -- 172.21.15.102:0/2644647259 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2358148dd0 con 0x7f23580b7fe0 2024-03-23T03:47:18.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.491+0000 7f018ae00640 1 -- start start 2024-03-23T03:47:18.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.492+0000 7f4c18400640 1 Processor -- start 2024-03-23T03:47:18.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.492+0000 7f4c18400640 1 -- start start 2024-03-23T03:47:18.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.493+0000 7fb670a00640 1 Processor -- start 2024-03-23T03:47:18.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.493+0000 7f018ae00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f01840b4e60 0x7f0184167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.491+0000 7fae22800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fae241627c0 0x7fae2406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.491+0000 7fae22800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fae241627c0 0x7fae2406d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:42208/0 (socket says 172.21.15.102:42208) 2024-03-23T03:47:18.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.491+0000 7fae22800640 1 -- 172.21.15.102:0/2124763685 learned_addr learned my addr 172.21.15.102:0/2124763685 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.492+0000 7fae22800640 1 -- 172.21.15.102:0/2124763685 >> v1:172.21.15.173:6789/0 conn(0x7fae240ce930 legacy=0x7fae241612d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.492+0000 7fae22800640 1 -- 172.21.15.102:0/2124763685 >> v1:172.21.15.102:6789/0 conn(0x7fae24165340 legacy=0x7fae240ced10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T03:47:18.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.493+0000 7f018ae00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01841607d0 con 0x7f01840b1f20 2024-03-23T03:47:18.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.493+0000 7f018ae00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0184160a70 con 0x7f018415cc70 2024-03-23T03:47:18.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.493+0000 7f018ae00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0184160d50 con 0x7f01840b4e60 2024-03-23T03:47:18.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.494+0000 7fae22800640 1 -- 172.21.15.102:0/2124763685 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae24161d90 con 0x7fae241627c0 2024-03-23T03:47:18.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.494+0000 7f235e200640 1 -- 172.21.15.102:0/2644647259 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f23580d0bc0 con 0x7f23580b7fe0 2024-03-23T03:47:18.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.494+0000 7f4c18400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c1006d330 0x7f4c10148910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.494+0000 7f4c18400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c100cefe0 0x7f4c100cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.494+0000 7f0183e00640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f018415cc70 0x7f01840cdba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:58860/0 (socket says 172.21.15.102:58860) 2024-03-23T03:47:18.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.494+0000 7f0183e00640 1 -- 172.21.15.102:0/2536715823 learned_addr learned my addr 172.21.15.102:0/2536715823 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:18.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.494+0000 7f4c18400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c10160c90 0x7f4c10159d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.494+0000 7f4c18400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c10159950 con 0x7f4c100cefe0 2024-03-23T03:47:18.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.494+0000 7f4c18400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1015db00 con 0x7f4c1006d330 2024-03-23T03:47:18.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.494+0000 7f4c18400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c10158390 con 0x7f4c10160c90 2024-03-23T03:47:18.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.495+0000 7f2354c00640 1 -- 172.21.15.102:0/2644647259 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f235000acf0 con 0x7f23580b7fe0 2024-03-23T03:47:18.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.494+0000 7fb670a00640 1 -- start start 2024-03-23T03:47:18.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.496+0000 7f4c15a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c1006d330 0x7f4c10148910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.496+0000 7fae22800640 1 --2- 172.21.15.102:0/2124763685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae241627c0 0x7fae2406d530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fae1c009e50 tx=0x7fae1c02bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4198824e00c4de6e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.496+0000 7f2354c00640 1 --2- 172.21.15.102:0/2644647259 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f233806ad00 0x7f233806d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.496+0000 7f2354c00640 1 -- 172.21.15.102:0/2644647259 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2350088f00 con 0x7f23580b7fe0 2024-03-23T03:47:18.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.496+0000 7f4c15000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c100cefe0 0x7f4c100cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.496+0000 7f4c15a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c1006d330 0x7f4c10148910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:42210/0 (socket says 172.21.15.102:42210) 2024-03-23T03:47:18.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.496+0000 7f4c15a00640 1 -- 172.21.15.102:0/713833806 learned_addr learned my addr 172.21.15.102:0/713833806 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.496+0000 7f0182a00640 1 -- 172.21.15.102:0/2536715823 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113450107 0 0) 0x7f01841607d0 con 0x7f01840b1f20 2024-03-23T03:47:18.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.496+0000 7f0182a00640 1 -- 172.21.15.102:0/2536715823 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0168003520 con 0x7f01840b1f20 2024-03-23T03:47:18.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.496+0000 7f0182a00640 1 -- 172.21.15.102:0/2536715823 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278353189 0 0) 0x7f0184160a70 con 0x7f018415cc70 2024-03-23T03:47:18.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.496+0000 7f0182a00640 1 -- 172.21.15.102:0/2536715823 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01680048a0 con 0x7f018415cc70 2024-03-23T03:47:18.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.496+0000 7f0188e00640 1 --2- 172.21.15.102:0/2536715823 >> v2:172.21.15.102:3300/0 conn(0x7f01840b4e60 0x7f0184167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.496+0000 7f4c16400640 1 --2- 172.21.15.102:0/713833806 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c10160c90 0x7f4c10159d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.496+0000 7f2354c00640 1 --2- 172.21.15.102:0/2644647259 >> [v2:172.21.15.102:6800/37350,v1:172.21.15.102:6801/37350] conn(0x7f2338074340 0x7f2338076780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.496+0000 7f2354c00640 1 -- 172.21.15.102:0/2644647259 --> [v2:172.21.15.102:6800/37350,v1:172.21.15.102:6801/37350] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2338076e50 con 0x7f2338074340 2024-03-23T03:47:18.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.496+0000 7f2354c00640 1 -- 172.21.15.102:0/2644647259 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f235008b520 con 0x7f23580b7fe0 2024-03-23T03:47:18.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7fae20a00640 1 -- 172.21.15.102:0/2124763685 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c02ca80 con 0x7fae241627c0 2024-03-23T03:47:18.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7f0188e00640 1 -- 172.21.15.102:0/2536715823 >> v1:172.21.15.173:6789/0 conn(0x7f018415cc70 legacy=0x7f01840cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7f0188e00640 1 -- 172.21.15.102:0/2536715823 >> v1:172.21.15.102:6789/0 conn(0x7f01840b1f20 legacy=0x7f01840b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7f0188e00640 1 -- 172.21.15.102:0/2536715823 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0184161030 con 0x7f01840b4e60 2024-03-23T03:47:18.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.496+0000 7fa6c2800640 1 Processor -- start 2024-03-23T03:47:18.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7fa6c2800640 1 -- start start 2024-03-23T03:47:18.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7fae20a00640 1 -- 172.21.15.102:0/2124763685 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1c02cbe0 con 0x7fae241627c0 2024-03-23T03:47:18.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7fae20a00640 1 -- 172.21.15.102:0/2124763685 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1c0343f0 con 0x7fae241627c0 2024-03-23T03:47:18.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7fb670a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb66c06da40 0x7fb66c159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7f2357e00640 1 --2- 172.21.15.102:0/2644647259 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f233806ad00 0x7f233806d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7f4c15a00640 1 -- 172.21.15.102:0/713833806 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c10160c90 msgr2=0x7f4c10159d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7f4c15a00640 1 --2- 172.21.15.102:0/713833806 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c10160c90 0x7f4c10159d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7f4c15a00640 1 -- 172.21.15.102:0/713833806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c100cefe0 msgr2=0x7f4c100cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7f4c15a00640 1 --2- 172.21.15.102:0/713833806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c100cefe0 0x7f4c100cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7f4c15a00640 1 -- 172.21.15.102:0/713833806 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc0094d0 con 0x7f4c1006d330 2024-03-23T03:47:18.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7fb670a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb66c0adc60 0x7fb66c1526e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7fb670a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb66c15e7a0 0x7fb66c152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7fb670a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c153160 con 0x7fb66c0adc60 2024-03-23T03:47:18.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7fb670a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c1ec970 con 0x7fb66c15e7a0 2024-03-23T03:47:18.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.497+0000 7fb670a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c1ec0d0 con 0x7fb66c06da40 2024-03-23T03:47:18.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.498+0000 7f4c15a00640 1 --2- 172.21.15.102:0/713833806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c1006d330 0x7f4c10148910 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4c0800c940 tx=0x7f4c0800ce00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.498+0000 7f235d800640 1 --2- 172.21.15.102:0/2644647259 >> [v2:172.21.15.102:6800/37350,v1:172.21.15.102:6801/37350] conn(0x7f2338074340 0x7f2338076780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.498+0000 7fb669e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb66c06da40 0x7fb66c159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.498+0000 7fd7ea800640 1 Processor -- start 2024-03-23T03:47:18.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.498+0000 7fb669e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb66c06da40 0x7fb66c159690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:54342/0 (socket says 172.21.15.102:54342) 2024-03-23T03:47:18.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.498+0000 7fb669e00640 1 -- 172.21.15.102:0/1590126895 learned_addr learned my addr 172.21.15.102:0/1590126895 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.498+0000 7fa6c2800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6bc0b4e60 0x7fa6bc167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.498+0000 7fa6c2800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6bc1607d0 con 0x7fa6bc0b1f20 2024-03-23T03:47:18.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.498+0000 7fb669400640 1 --2- 172.21.15.102:0/1590126895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb66c0adc60 0x7fb66c1526e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.499+0000 7fb66a800640 1 --2- 172.21.15.102:0/1590126895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb66c15e7a0 0x7fb66c152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.498+0000 7fd7ea800640 1 -- start start 2024-03-23T03:47:18.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.499+0000 7fa6c2800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6bc160a70 con 0x7fa6bc15cc70 2024-03-23T03:47:18.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.499+0000 7fa6c2800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc160d50 con 0x7fa6bc0b4e60 2024-03-23T03:47:18.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.499+0000 7fa6baa00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa6bc0b1f20 0x7fa6bc0b4750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41176/0 (socket says 172.21.15.102:41176) 2024-03-23T03:47:18.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.499+0000 7fa6baa00640 1 -- 172.21.15.102:0/2121539875 learned_addr learned my addr 172.21.15.102:0/2121539875 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:18.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.499+0000 7fb669e00640 1 -- 172.21.15.102:0/1590126895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb66c15e7a0 msgr2=0x7fb66c152c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.499+0000 7f2357e00640 1 --2- 172.21.15.102:0/2644647259 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f233806ad00 0x7f233806d1c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2358159750 tx=0x7f2344008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.499+0000 7fb669e00640 1 --2- 172.21.15.102:0/1590126895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb66c15e7a0 0x7fb66c152c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.499+0000 7fb669e00640 1 -- 172.21.15.102:0/1590126895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb66c0adc60 msgr2=0x7fb66c1526e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.499+0000 7fb669e00640 1 --2- 172.21.15.102:0/1590126895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb66c0adc60 0x7fb66c1526e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.498+0000 7fae28e00640 1 -- 172.21.15.102:0/2124763685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae241627c0 msgr2=0x7fae2406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.498+0000 7fae28e00640 1 --2- 172.21.15.102:0/2124763685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae241627c0 0x7fae2406d530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fae1c009e50 tx=0x7fae1c02bee0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.498+0000 7fae28e00640 1 -- 172.21.15.102:0/2124763685 shutdown_connections 2024-03-23T03:47:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.498+0000 7fae28e00640 1 --2- 172.21.15.102:0/2124763685 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae241627c0 0x7fae2406d530 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.498+0000 7fae28e00640 1 -- 172.21.15.102:0/2124763685 >> 172.21.15.102:0/2124763685 conn(0x7fae2406a0b0 msgr2=0x7fae2406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.499+0000 7fb669e00640 1 -- 172.21.15.102:0/1590126895 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65c0094d0 con 0x7fb66c06da40 2024-03-23T03:47:18.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.499+0000 7f0188e00640 1 --2- 172.21.15.102:0/2536715823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01840b4e60 0x7f0184167b00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f017c009d90 tx=0x7f017c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab90dc60b8e1023 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.499+0000 7fb669400640 1 --2- 172.21.15.102:0/1590126895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb66c0adc60 0x7fb66c1526e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:47:18.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.499+0000 7fb66a800640 1 --2- 172.21.15.102:0/1590126895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb66c15e7a0 0x7fb66c152c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:18.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.500+0000 7f235d800640 1 --2- 172.21.15.102:0/2644647259 >> [v2:172.21.15.102:6800/37350,v1:172.21.15.102:6801/37350] conn(0x7f2338074340 0x7f2338076780 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.500+0000 7fa6ba000640 1 -- 172.21.15.102:0/2121539875 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2505824740 0 0) 0x7fa6bc160a70 con 0x7fa6bc15cc70 2024-03-23T03:47:18.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.500+0000 7fa6ba000640 1 -- 172.21.15.102:0/2121539875 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a0003520 con 0x7fa6bc15cc70 2024-03-23T03:47:18.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.500+0000 7fa6ba000640 1 -- 172.21.15.102:0/2121539875 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104093080 0 0) 0x7fa6bc1607d0 con 0x7fa6bc0b1f20 2024-03-23T03:47:18.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.500+0000 7fb669e00640 1 --2- 172.21.15.102:0/1590126895 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb66c06da40 0x7fb66c159690 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb65c000c00 tx=0x7fb65c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.500+0000 7fa6ba000640 1 -- 172.21.15.102:0/2121539875 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6bc160a70 con 0x7fa6bc0b1f20 2024-03-23T03:47:18.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.500+0000 7f4c0d600640 1 -- 172.21.15.102:0/713833806 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0800aab0 con 0x7f4c1006d330 2024-03-23T03:47:18.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.500+0000 7f4c0d600640 1 -- 172.21.15.102:0/713833806 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0800ac10 con 0x7f4c1006d330 2024-03-23T03:47:18.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.500+0000 7f4c0d600640 1 -- 172.21.15.102:0/713833806 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c08012420 con 0x7f4c1006d330 2024-03-23T03:47:18.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.500+0000 7f4c18400640 1 -- 172.21.15.102:0/713833806 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c10158670 con 0x7f4c1006d330 2024-03-23T03:47:18.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.500+0000 7fae28e00640 1 -- 172.21.15.102:0/2124763685 shutdown_connections 2024-03-23T03:47:18.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.500+0000 7fb662000640 1 -- 172.21.15.102:0/1590126895 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c03ac60 con 0x7fb66c06da40 2024-03-23T03:47:18.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.500+0000 7fb662000640 1 -- 172.21.15.102:0/1590126895 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65c03adc0 con 0x7fb66c06da40 2024-03-23T03:47:18.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.500+0000 7fb662000640 1 -- 172.21.15.102:0/1590126895 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c044610 con 0x7fb66c06da40 2024-03-23T03:47:18.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fb670a00640 1 -- 172.21.15.102:0/1590126895 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb66c1ec3b0 con 0x7fb66c06da40 2024-03-23T03:47:18.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fb670a00640 1 -- 172.21.15.102:0/1590126895 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb66c154770 con 0x7fb66c06da40 2024-03-23T03:47:18.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.500+0000 7f4c18400640 1 -- 172.21.15.102:0/713833806 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c10159570 con 0x7f4c1006d330 2024-03-23T03:47:18.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7f0182a00640 1 -- 172.21.15.102:0/2536715823 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c02cc60 con 0x7f01840b4e60 2024-03-23T03:47:18.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7f0182a00640 1 -- 172.21.15.102:0/2536715823 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f017c02cdc0 con 0x7f01840b4e60 2024-03-23T03:47:18.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7f0182a00640 1 -- 172.21.15.102:0/2536715823 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c034660 con 0x7f01840b4e60 2024-03-23T03:47:18.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fd7ea800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd7e40ce930 0x7fd7e4165340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fd7ea800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7e41612d0 0x7fd7e40ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fd7ea800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7e406cdc0 con 0x7fd7e41627c0 2024-03-23T03:47:18.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fd7ea800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7e40cf250 con 0x7fd7e40ce930 2024-03-23T03:47:18.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fd7ea800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e4161ab0 con 0x7fd7e41612d0 2024-03-23T03:47:18.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.500+0000 7fae28e00640 1 -- 172.21.15.102:0/2124763685 wait complete. 2024-03-23T03:47:18.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fa6bbe00640 1 --2- 172.21.15.102:0/2121539875 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6bc0b4e60 0x7fa6bc167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fa6ba000640 1 -- 172.21.15.102:0/2121539875 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1102101433 0 0) 0x7fa6a0003520 con 0x7fa6bc15cc70 2024-03-23T03:47:18.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fa6ba000640 1 -- 172.21.15.102:0/2121539875 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6bc0b4e60 msgr2=0x7fa6bc167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fa6ba000640 1 --2- 172.21.15.102:0/2121539875 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6bc0b4e60 0x7fa6bc167b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fa6ba000640 1 -- 172.21.15.102:0/2121539875 >> v1:172.21.15.102:6789/0 conn(0x7fa6bc0b1f20 legacy=0x7fa6bc0b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fa6ba000640 1 -- 172.21.15.102:0/2121539875 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc161030 con 0x7fa6bc15cc70 2024-03-23T03:47:18.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fa6ba000640 1 -- 172.21.15.102:0/2121539875 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fa6ac001670 con 0x7fa6bc15cc70 2024-03-23T03:47:18.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fa6ba000640 1 -- 172.21.15.102:0/2121539875 >> v1:172.21.15.173:6789/0 conn(0x7fa6bc15cc70 legacy=0x7fa6bc0cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fa6ba000640 1 --2- 172.21.15.102:0/2121539875 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6a0003b30 0x7fa6a0003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fa6ba000640 1 --2- 172.21.15.102:0/2121539875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6a00047c0 0x7fa6a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fa6ba000640 1 --2- 172.21.15.102:0/2121539875 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6a0005180 0x7fa6a0014190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fa6ba000640 1 -- 172.21.15.102:0/2121539875 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a8001230 con 0x7fa6a00047c0 2024-03-23T03:47:18.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fa6ba000640 1 -- 172.21.15.102:0/2121539875 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc160d50 con 0x7fa6a0005180 2024-03-23T03:47:18.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.501+0000 7fa6ba000640 1 -- 172.21.15.102:0/2121539875 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a0005a00 con 0x7fa6a0003b30 2024-03-23T03:47:18.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.502+0000 7f4c02a00640 1 -- 172.21.15.102:0/713833806 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4be4000fc0 con 0x7f4c1006d330 2024-03-23T03:47:18.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.502+0000 7fd7e3e00640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fd7e41627c0 0x7fd7e406d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:58886/0 (socket says 172.21.15.102:58886) 2024-03-23T03:47:18.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.502+0000 7fd7e3e00640 1 -- 172.21.15.102:0/638176719 learned_addr learned my addr 172.21.15.102:0/638176719 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:18.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.502+0000 7fd7e3400640 1 --2- 172.21.15.102:0/638176719 >> v2:172.21.15.173:3300/0 conn(0x7fd7e40ce930 0x7fd7e4165340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.502+0000 7f2354c00640 1 -- 172.21.15.102:0/2644647259 <== osd.2 v2:172.21.15.102:6800/37350 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f2338076e50 con 0x7f2338074340 2024-03-23T03:47:18.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.502+0000 7fd7daa00640 1 --2- 172.21.15.102:0/638176719 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7e41612d0 0x7fd7e40ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.502+0000 7fb670a00640 1 -- 172.21.15.102:0/1590126895 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb66c0b1a40 con 0x7fb66c06da40 2024-03-23T03:47:18.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.503+0000 7fd7e3400640 1 -- 172.21.15.102:0/638176719 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7e41612d0 msgr2=0x7fd7e40ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.503+0000 7fd7e3400640 1 --2- 172.21.15.102:0/638176719 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7e41612d0 0x7fd7e40ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.503+0000 7fd7e3400640 1 -- 172.21.15.102:0/638176719 >> v1:172.21.15.173:6789/0 conn(0x7fd7e41627c0 legacy=0x7fd7e406d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.503+0000 7fd7e3400640 1 -- 172.21.15.102:0/638176719 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e4161d90 con 0x7fd7e40ce930 2024-03-23T03:47:18.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.503+0000 7f018ae00640 1 -- 172.21.15.102:0/2536715823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01840b4e60 msgr2=0x7f0184167b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.503+0000 7f018ae00640 1 --2- 172.21.15.102:0/2536715823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01840b4e60 0x7f0184167b00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f017c009d90 tx=0x7f017c02beb0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.503+0000 7fa6baa00640 1 --2- 172.21.15.102:0/2121539875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6a00047c0 0x7fa6a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.503+0000 7fa6bb400640 1 --2- 172.21.15.102:0/2121539875 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6a0003b30 0x7fa6a0003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.503+0000 7fa6bbe00640 1 --2- 172.21.15.102:0/2121539875 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6a0005180 0x7fa6a0014190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.503+0000 7fae28e00640 1 Processor -- start 2024-03-23T03:47:18.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.504+0000 7fae28e00640 1 -- start start 2024-03-23T03:47:18.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.504+0000 7f018ae00640 1 -- 172.21.15.102:0/2536715823 shutdown_connections 2024-03-23T03:47:18.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.504+0000 7f018ae00640 1 --2- 172.21.15.102:0/2536715823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f01840b4e60 0x7f0184167b00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.504+0000 7f018ae00640 1 -- 172.21.15.102:0/2536715823 >> 172.21.15.102:0/2536715823 conn(0x7f01840cf790 msgr2=0x7f01840cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.504+0000 7f018ae00640 1 -- 172.21.15.102:0/2536715823 shutdown_connections 2024-03-23T03:47:18.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.504+0000 7fa6bb400640 1 -- 172.21.15.102:0/2121539875 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6a0005180 msgr2=0x7fa6a0014190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.504+0000 7fa6bb400640 1 --2- 172.21.15.102:0/2121539875 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6a0005180 0x7fa6a0014190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.504+0000 7fa6bb400640 1 -- 172.21.15.102:0/2121539875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6a00047c0 msgr2=0x7fa6a0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.504+0000 7fa6bb400640 1 --2- 172.21.15.102:0/2121539875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6a00047c0 0x7fa6a0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.504+0000 7fa6bb400640 1 -- 172.21.15.102:0/2121539875 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc161030 con 0x7fa6a0003b30 2024-03-23T03:47:18.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.504+0000 7fa6baa00640 1 --2- 172.21.15.102:0/2121539875 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6a00047c0 0x7fa6a0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:47:18.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.504+0000 7f018ae00640 1 -- 172.21.15.102:0/2536715823 wait complete. 2024-03-23T03:47:18.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.506+0000 7fd7e3400640 1 --2- 172.21.15.102:0/638176719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7e40ce930 0x7fd7e4165340 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009d90 tx=0x7fd7d402bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84e589a65fcc0fd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.506+0000 7fa6bb400640 1 --2- 172.21.15.102:0/2121539875 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6a0003b30 0x7fa6a0003f10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac00c160 tx=0x7fa6a00057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6e5101f3579e277 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.506+0000 7fae28e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae240ce930 0x7fae2414d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.506+0000 7fae28e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae241627c0 0x7fae241f6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.506+0000 7fae28e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fae24165340 0x7fae2414cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.506+0000 7fae28e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae240cd150 con 0x7fae241627c0 2024-03-23T03:47:18.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.506+0000 7fae28e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae24157010 con 0x7fae240ce930 2024-03-23T03:47:18.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.506+0000 7fae28e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae241f6fe0 con 0x7fae24165340 2024-03-23T03:47:18.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.506+0000 7fb662000640 1 -- 172.21.15.102:0/1590126895 <== mon.2 v2:172.21.15.173:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb65c02f050 con 0x7fb66c06da40 2024-03-23T03:47:18.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.507+0000 7f018ae00640 1 Processor -- start 2024-03-23T03:47:18.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.507+0000 7fae21e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae240ce930 0x7fae2414d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.507+0000 7fa6ba000640 1 -- 172.21.15.102:0/2121539875 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac02c9c0 con 0x7fa6a0003b30 2024-03-23T03:47:18.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.507+0000 7fa6ba000640 1 -- 172.21.15.102:0/2121539875 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ac02cb40 con 0x7fa6a0003b30 2024-03-23T03:47:18.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.507+0000 7fa6ba000640 1 -- 172.21.15.102:0/2121539875 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac034440 con 0x7fa6a0003b30 2024-03-23T03:47:18.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.507+0000 7f522d800640 1 Processor -- start 2024-03-23T03:47:18.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.507+0000 7f522d800640 1 -- start start 2024-03-23T03:47:18.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.507+0000 7fae21400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae241627c0 0x7fae241f6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.507+0000 7fae21400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae241627c0 0x7fae241f6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37238/0 (socket says 172.21.15.102:37238) 2024-03-23T03:47:18.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.507+0000 7fae21400640 1 -- 172.21.15.102:0/2752044212 learned_addr learned my addr 172.21.15.102:0/2752044212 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7fa6c2800640 1 -- 172.21.15.102:0/2121539875 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6a0003b30 msgr2=0x7fa6a0003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7f018ae00640 1 -- start start 2024-03-23T03:47:18.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7fa6c2800640 1 --2- 172.21.15.102:0/2121539875 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6a0003b30 0x7fa6a0003f10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac00c160 tx=0x7fa6a00057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.507+0000 7fd7e2a00640 1 -- 172.21.15.102:0/638176719 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d402ca10 con 0x7fd7e40ce930 2024-03-23T03:47:18.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.507+0000 7fd7e2a00640 1 -- 172.21.15.102:0/638176719 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d402cb70 con 0x7fd7e40ce930 2024-03-23T03:47:18.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.507+0000 7fd7e2a00640 1 -- 172.21.15.102:0/638176719 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4034440 con 0x7fd7e40ce930 2024-03-23T03:47:18.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7fd7ea800640 1 -- 172.21.15.102:0/638176719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7e40ce930 msgr2=0x7fd7e4165340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7fd7ea800640 1 --2- 172.21.15.102:0/638176719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7e40ce930 0x7fd7e4165340 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009d90 tx=0x7fd7d402bce0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7fb662000640 1 --2- 172.21.15.102:0/1590126895 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb64c06ac30 0x7fb64c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7fb662000640 1 -- 172.21.15.102:0/1590126895 <== mon.2 v2:172.21.15.173:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb65c0abad0 con 0x7fb66c06da40 2024-03-23T03:47:18.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7fb662000640 1 --2- 172.21.15.102:0/1590126895 >> [v2:172.21.15.102:6816/37351,v1:172.21.15.102:6817/37351] conn(0x7fb64c0741f0 0x7fb64c076630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7fb662000640 1 -- 172.21.15.102:0/1590126895 --> [v2:172.21.15.102:6816/37351,v1:172.21.15.102:6817/37351] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb64c076d00 con 0x7fb64c0741f0 2024-03-23T03:47:18.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7fa6bbe00640 1 -- 172.21.15.102:0/2121539875 reap_dead start 2024-03-23T03:47:18.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7fb662000640 1 -- 172.21.15.102:0/1590126895 <== mon.2 v2:172.21.15.173:3301/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb65c0abe40 con 0x7fb66c06da40 2024-03-23T03:47:18.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7fa6c2800640 1 -- 172.21.15.102:0/2121539875 shutdown_connections 2024-03-23T03:47:18.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.509+0000 7fa6c2800640 1 -- 172.21.15.102:0/2121539875 >> 172.21.15.102:0/2121539875 conn(0x7fa6bc0cf790 msgr2=0x7fa6bc0cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.509+0000 7f522d800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f522815cd90 0x7f52280b8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.509+0000 7f522d800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f52280cda90 0x7f52280d1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.509+0000 7f522d800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522806b840 con 0x7f52280cd6b0 2024-03-23T03:47:18.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.509+0000 7f522d800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f522806c0e0 con 0x7f522815cd90 2024-03-23T03:47:18.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.509+0000 7f522d800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f522806bb20 con 0x7f52280cda90 2024-03-23T03:47:18.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7fae21400640 1 -- 172.21.15.102:0/2752044212 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fae24165340 msgr2=0x7fae2414cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7fae21400640 1 --2- 172.21.15.102:0/2752044212 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fae24165340 0x7fae2414cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7fae21400640 1 -- 172.21.15.102:0/2752044212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae240ce930 msgr2=0x7fae2414d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7fae21400640 1 --2- 172.21.15.102:0/2752044212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae240ce930 0x7fae2414d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7fae21400640 1 -- 172.21.15.102:0/2752044212 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c009b30 con 0x7fae241627c0 2024-03-23T03:47:18.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.508+0000 7f4c0d600640 1 -- 172.21.15.102:0/713833806 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4c08012580 con 0x7f4c1006d330 2024-03-23T03:47:18.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.509+0000 7fae21e00640 1 --2- 172.21.15.102:0/2752044212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae240ce930 0x7fae2414d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:47:18.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.509+0000 7fb66a800640 1 --2- 172.21.15.102:0/1590126895 >> [v2:172.21.15.102:6816/37351,v1:172.21.15.102:6817/37351] conn(0x7fb64c0741f0 0x7fb64c076630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.509+0000 7f5226000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f522815cd90 0x7f52280b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.509+0000 7fd7ea800640 1 -- 172.21.15.102:0/638176719 shutdown_connections 2024-03-23T03:47:18.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.509+0000 7fd7ea800640 1 --2- 172.21.15.102:0/638176719 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7e41612d0 0x7fd7e40ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.509+0000 7fd7ea800640 1 --2- 172.21.15.102:0/638176719 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7e40ce930 0x7fd7e4165340 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.509+0000 7fd7ea800640 1 -- 172.21.15.102:0/638176719 >> 172.21.15.102:0/638176719 conn(0x7fd7e406a0b0 msgr2=0x7fd7e406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.509+0000 7f5226000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f522815cd90 0x7f52280b8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:42252/0 (socket says 172.21.15.102:42252) 2024-03-23T03:47:18.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.509+0000 7f5226000640 1 -- 172.21.15.102:0/1800263646 learned_addr learned my addr 172.21.15.102:0/1800263646 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7f5227400640 1 --2- 172.21.15.102:0/1800263646 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f52280cda90 0x7f52280d1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7fd7ea800640 1 -- 172.21.15.102:0/638176719 shutdown_connections 2024-03-23T03:47:18.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7fb669400640 1 --2- 172.21.15.102:0/1590126895 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb64c06ac30 0x7fb64c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7f5226000640 1 -- 172.21.15.102:0/1800263646 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f52280cda90 msgr2=0x7f52280d1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7f5226000640 1 --2- 172.21.15.102:0/1800263646 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f52280cda90 0x7f52280d1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7f5226000640 1 -- 172.21.15.102:0/1800263646 >> v1:172.21.15.173:6789/0 conn(0x7f52280cd6b0 legacy=0x7f522806b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7f5226000640 1 -- 172.21.15.102:0/1800263646 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522806be00 con 0x7f522815cd90 2024-03-23T03:47:18.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7fb66a800640 1 --2- 172.21.15.102:0/1590126895 >> [v2:172.21.15.102:6816/37351,v1:172.21.15.102:6817/37351] conn(0x7fb64c0741f0 0x7fb64c076630 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7fa6c2800640 1 -- 172.21.15.102:0/2121539875 shutdown_connections 2024-03-23T03:47:18.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7f018ae00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01840b1f20 0x7f0184155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7f018ae00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f01840b4e60 0x7f01840b5b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7f018ae00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f018415cc70 0x7f01840c6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7f018ae00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01840c99c0 con 0x7f018415cc70 2024-03-23T03:47:18.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7f018ae00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01841555d0 con 0x7f01840b1f20 2024-03-23T03:47:18.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7f018ae00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01840b6370 con 0x7f01840b4e60 2024-03-23T03:47:18.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7fae21400640 1 --2- 172.21.15.102:0/2752044212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae241627c0 0x7fae241f6aa0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fae1000b3f0 tx=0x7fae1000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7fd7ea800640 1 -- 172.21.15.102:0/638176719 wait complete. 2024-03-23T03:47:18.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.510+0000 7fa6c2800640 1 -- 172.21.15.102:0/2121539875 wait complete. 2024-03-23T03:47:18.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.511+0000 7fae1a000640 1 -- 172.21.15.102:0/2752044212 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae10011830 con 0x7fae241627c0 2024-03-23T03:47:18.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.511+0000 7fae1a000640 1 -- 172.21.15.102:0/2752044212 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae100119b0 con 0x7fae241627c0 2024-03-23T03:47:18.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.511+0000 7fae1a000640 1 -- 172.21.15.102:0/2752044212 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae10011d60 con 0x7fae241627c0 2024-03-23T03:47:18.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.511+0000 7fae28e00640 1 -- 172.21.15.102:0/2752044212 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae2414d0b0 con 0x7fae241627c0 2024-03-23T03:47:18.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.511+0000 7f0183400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f01840b4e60 0x7f01840b5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.511+0000 7f0188e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f018415cc70 0x7f01840c6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.511+0000 7f0188e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f018415cc70 0x7f01840c6ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37252/0 (socket says 172.21.15.102:37252) 2024-03-23T03:47:18.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.511+0000 7f0183400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f01840b4e60 0x7f01840b5b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:54408/0 (socket says 172.21.15.102:54408) 2024-03-23T03:47:18.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.511+0000 7f0188e00640 1 -- 172.21.15.102:0/1418855141 learned_addr learned my addr 172.21.15.102:0/1418855141 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.511+0000 7f0183e00640 1 --2- 172.21.15.102:0/1418855141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01840b1f20 0x7f0184155090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.512+0000 7f0188e00640 1 -- 172.21.15.102:0/1418855141 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f01840b4e60 msgr2=0x7f01840b5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.512+0000 7f0188e00640 1 --2- 172.21.15.102:0/1418855141 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f01840b4e60 0x7f01840b5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.512+0000 7f0188e00640 1 -- 172.21.15.102:0/1418855141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01840b1f20 msgr2=0x7f0184155090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.512+0000 7f0188e00640 1 --2- 172.21.15.102:0/1418855141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01840b1f20 0x7f0184155090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.512+0000 7f0188e00640 1 -- 172.21.15.102:0/1418855141 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017c009a70 con 0x7f018415cc70 2024-03-23T03:47:18.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.512+0000 7fb662000640 1 -- 172.21.15.102:0/1590126895 <== osd.0 v2:172.21.15.102:6816/37351 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fb64c076d00 con 0x7fb64c0741f0 2024-03-23T03:47:18.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.512+0000 7fd7ea800640 1 Processor -- start 2024-03-23T03:47:18.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.513+0000 7f5226000640 1 --2- 172.21.15.102:0/1800263646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f522815cd90 0x7f52280b8490 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5218009870 tx=0x7f521802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a378f1fa24e32cd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.513+0000 7fa6c2800640 1 Processor -- start 2024-03-23T03:47:18.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.513+0000 7fd7ea800640 1 -- start start 2024-03-23T03:47:18.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.512+0000 7fae28e00640 1 -- 172.21.15.102:0/2752044212 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae240ccd70 con 0x7fae241627c0 2024-03-23T03:47:18.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.513+0000 7fae28e00640 1 -- 172.21.15.102:0/2752044212 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fae2406d010 con 0x7fae241627c0 2024-03-23T03:47:18.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.513+0000 7f0188e00640 1 --2- 172.21.15.102:0/1418855141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f018415cc70 0x7f01840c6ab0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f017c009eb0 tx=0x7f017c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.513+0000 7f017be00640 1 -- 172.21.15.102:0/1418855141 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c03e610 con 0x7f018415cc70 2024-03-23T03:47:18.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.513+0000 7f017be00640 1 -- 172.21.15.102:0/1418855141 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f017c03e770 con 0x7f018415cc70 2024-03-23T03:47:18.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.513+0000 7f017be00640 1 -- 172.21.15.102:0/1418855141 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017c03e610 con 0x7f018415cc70 2024-03-23T03:47:18.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.514+0000 7f018ae00640 1 -- 172.21.15.102:0/1418855141 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01840b6650 con 0x7f018415cc70 2024-03-23T03:47:18.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.515+0000 7f5225600640 1 -- 172.21.15.102:0/1800263646 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5218031640 con 0x7f522815cd90 2024-03-23T03:47:18.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.513+0000 7fa6c2800640 1 -- start start 2024-03-23T03:47:18.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.515+0000 7fa6c2800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6bc0707d0 0x7fa6bc0ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.515+0000 7fa6c2800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6bc070bb0 0x7fa6bc158610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.515+0000 7fa6c2800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6bc0cc5b0 0x7fa6bc158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.515+0000 7fd7ea800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7e40ce930 0x7fd7e415c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.515+0000 7fd7ea800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7e41612d0 0x7fd7e4152eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.515+0000 7fd7ea800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7e41627c0 0x7fd7e4158d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.514+0000 7f018ae00640 1 -- 172.21.15.102:0/1418855141 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01840c9bd0 con 0x7f018415cc70 2024-03-23T03:47:18.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.515+0000 7f5225600640 1 -- 172.21.15.102:0/1800263646 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52180317a0 con 0x7f522815cd90 2024-03-23T03:47:18.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.515+0000 7f5225600640 1 -- 172.21.15.102:0/1800263646 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5218031b30 con 0x7f522815cd90 2024-03-23T03:47:18.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.515+0000 7fd7ea800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e41542a0 con 0x7fd7e40ce930 2024-03-23T03:47:18.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.515+0000 7fa6c2800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc0cc120 con 0x7fa6bc0cc5b0 2024-03-23T03:47:18.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.515+0000 7fa6c2800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc0cb550 con 0x7fa6bc070bb0 2024-03-23T03:47:18.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.515+0000 7fa6c2800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6bc0cb7f0 con 0x7fa6bc0707d0 2024-03-23T03:47:18.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.515+0000 7fd7e3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7e40ce930 0x7fd7e415c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.516+0000 7fd7e3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7e40ce930 0x7fd7e415c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37268/0 (socket says 172.21.15.102:37268) 2024-03-23T03:47:18.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.516+0000 7fd7e3400640 1 -- 172.21.15.102:0/1648165928 learned_addr learned my addr 172.21.15.102:0/1648165928 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.516+0000 7fd7ea800640 1 -- 172.21.15.102:0/1648165928 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e41533f0 con 0x7fd7e41627c0 2024-03-23T03:47:18.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.516+0000 7f522d800640 1 -- 172.21.15.102:0/1800263646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f522815cd90 msgr2=0x7f52280b8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.516+0000 7f522d800640 1 --2- 172.21.15.102:0/1800263646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f522815cd90 0x7f52280b8490 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5218009870 tx=0x7f521802b8f0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.516+0000 7fd7daa00640 1 --2- 172.21.15.102:0/1648165928 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7e41612d0 0x7fd7e4152eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.515+0000 7fa6bb400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6bc0707d0 0x7fa6bc0ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.515+0000 7fa6baa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6bc070bb0 0x7fa6bc158610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.516+0000 7fa6baa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6bc070bb0 0x7fa6bc158610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:42284/0 (socket says 172.21.15.102:42284) 2024-03-23T03:47:18.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.516+0000 7fa6baa00640 1 -- 172.21.15.102:0/2621027432 learned_addr learned my addr 172.21.15.102:0/2621027432 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.516+0000 7fd7e3e00640 1 --2- 172.21.15.102:0/1648165928 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7e41627c0 0x7fd7e4158d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.516+0000 7fd7ea800640 1 -- 172.21.15.102:0/1648165928 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e4153690 con 0x7fd7e41612d0 2024-03-23T03:47:18.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.516+0000 7f4c0d600640 1 --2- 172.21.15.102:0/713833806 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4bf006ac00 0x7f4bf006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.516+0000 7f522d800640 1 -- 172.21.15.102:0/1800263646 shutdown_connections 2024-03-23T03:47:18.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.516+0000 7f522d800640 1 --2- 172.21.15.102:0/1800263646 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f52280cda90 0x7f52280d1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.516+0000 7f522d800640 1 --2- 172.21.15.102:0/1800263646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f522815cd90 0x7f52280b8490 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.516+0000 7f522d800640 1 -- 172.21.15.102:0/1800263646 >> 172.21.15.102:0/1800263646 conn(0x7f52280cf780 msgr2=0x7f52280cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.516+0000 7f018ae00640 1 -- 172.21.15.102:0/1418855141 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f018415d050 con 0x7f018415cc70 2024-03-23T03:47:18.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.517+0000 7fa6bbe00640 1 --2- 172.21.15.102:0/2621027432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6bc0cc5b0 0x7fa6bc158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.517+0000 7f522d800640 1 -- 172.21.15.102:0/1800263646 shutdown_connections 2024-03-23T03:47:18.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.517+0000 7f522d800640 1 -- 172.21.15.102:0/1800263646 wait complete. 2024-03-23T03:47:18.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.517+0000 7fa6baa00640 1 -- 172.21.15.102:0/2621027432 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6bc0707d0 msgr2=0x7fa6bc0ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.517+0000 7fa6baa00640 1 --2- 172.21.15.102:0/2621027432 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6bc0707d0 0x7fa6bc0ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.517+0000 7fa6baa00640 1 -- 172.21.15.102:0/2621027432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6bc0cc5b0 msgr2=0x7fa6bc158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.517+0000 7fa6baa00640 1 --2- 172.21.15.102:0/2621027432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6bc0cc5b0 0x7fa6bc158b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.517+0000 7fa6baa00640 1 -- 172.21.15.102:0/2621027432 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc0cbad0 con 0x7fa6bc070bb0 2024-03-23T03:47:18.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.517+0000 7fd7e3400640 1 -- 172.21.15.102:0/1648165928 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7e41612d0 msgr2=0x7fd7e4152eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.517+0000 7fd7e3400640 1 --2- 172.21.15.102:0/1648165928 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7e41612d0 0x7fd7e4152eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.517+0000 7fd7e3400640 1 -- 172.21.15.102:0/1648165928 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7e41627c0 msgr2=0x7fd7e4158d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.517+0000 7fd7e3400640 1 --2- 172.21.15.102:0/1648165928 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7e41627c0 0x7fd7e4158d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.517+0000 7fd7e3400640 1 -- 172.21.15.102:0/1648165928 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d4009a70 con 0x7fd7e40ce930 2024-03-23T03:47:18.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.518+0000 7f017be00640 1 -- 172.21.15.102:0/1418855141 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f017c03b050 con 0x7f018415cc70 2024-03-23T03:47:18.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.518+0000 7fa6baa00640 1 --2- 172.21.15.102:0/2621027432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6bc070bb0 0x7fa6bc158610 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa6a80072c0 tx=0x7fa6a8007780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.519+0000 7fa6b3400640 1 -- 172.21.15.102:0/2621027432 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a800fa80 con 0x7fa6bc070bb0 2024-03-23T03:47:18.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.519+0000 7fa6b3400640 1 -- 172.21.15.102:0/2621027432 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a800fbe0 con 0x7fa6bc070bb0 2024-03-23T03:47:18.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.519+0000 7fa6b3400640 1 -- 172.21.15.102:0/2621027432 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a80133f0 con 0x7fa6bc070bb0 2024-03-23T03:47:18.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.519+0000 7f017be00640 1 --2- 172.21.15.102:0/1418855141 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f016006ac90 0x7f016006d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.519+0000 7fb669400640 1 --2- 172.21.15.102:0/1590126895 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb64c06ac30 0x7fb64c06d0f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0cdbb0 tx=0x7fb658009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.519+0000 7f017be00640 1 -- 172.21.15.102:0/1418855141 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f017c0ab610 con 0x7f018415cc70 2024-03-23T03:47:18.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.519+0000 7fa6c2800640 1 -- 172.21.15.102:0/2621027432 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6bc0cbdb0 con 0x7fa6bc070bb0 2024-03-23T03:47:18.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.519+0000 7fa6c2800640 1 -- 172.21.15.102:0/2621027432 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6bc0ca8d0 con 0x7fa6bc070bb0 2024-03-23T03:47:18.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.518+0000 7fae1a000640 1 -- 172.21.15.102:0/2752044212 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fae1001a400 con 0x7fae241627c0 2024-03-23T03:47:18.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.519+0000 7f017be00640 1 --2- 172.21.15.102:0/1418855141 >> [v2:172.21.15.173:6816/37509,v1:172.21.15.173:6817/37509] conn(0x7f01600741d0 0x7f0160076610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.519+0000 7f017be00640 1 -- 172.21.15.102:0/1418855141 --> [v2:172.21.15.173:6816/37509,v1:172.21.15.173:6817/37509] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0160076ce0 con 0x7f01600741d0 2024-03-23T03:47:18.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.519+0000 7f017be00640 1 -- 172.21.15.102:0/1418855141 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f017c0765b0 con 0x7f018415cc70 2024-03-23T03:47:18.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.519+0000 7f4c0d600640 1 -- 172.21.15.102:0/713833806 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4c08089c10 con 0x7f4c1006d330 2024-03-23T03:47:18.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.520+0000 7f4c15000640 1 --2- 172.21.15.102:0/713833806 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4bf006ac00 0x7f4bf006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.520+0000 7f4c0d600640 1 --2- 172.21.15.102:0/713833806 >> [v2:172.21.15.102:6824/37352,v1:172.21.15.102:6825/37352] conn(0x7f4bf00741c0 0x7f4bf0076600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.520+0000 7f4c0d600640 1 -- 172.21.15.102:0/713833806 --> [v2:172.21.15.102:6824/37352,v1:172.21.15.102:6825/37352] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4bf0076cd0 con 0x7f4bf00741c0 2024-03-23T03:47:18.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.520+0000 7f4c0d600640 1 -- 172.21.15.102:0/713833806 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c0808a060 con 0x7f4c1006d330 2024-03-23T03:47:18.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.519+0000 7fd7e3400640 1 --2- 172.21.15.102:0/1648165928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7e40ce930 0x7fd7e415c450 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009eb0 tx=0x7fd7d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.520+0000 7f0183400640 1 --2- 172.21.15.102:0/1418855141 >> [v2:172.21.15.173:6816/37509,v1:172.21.15.173:6817/37509] conn(0x7f01600741d0 0x7f0160076610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.520+0000 7fae1a000640 1 --2- 172.21.15.102:0/2752044212 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fae0406ac50 0x7fae0406d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.520+0000 7fa6c2800640 1 -- 172.21.15.102:0/2621027432 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa694000fc0 con 0x7fa6bc070bb0 2024-03-23T03:47:18.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.521+0000 7f4c15000640 1 --2- 172.21.15.102:0/713833806 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4bf006ac00 0x7f4bf006d0c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4c10153850 tx=0x7f4c04008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.521+0000 7f0183e00640 1 --2- 172.21.15.102:0/1418855141 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f016006ac90 0x7f016006d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.521+0000 7fd7dbe00640 1 -- 172.21.15.102:0/1648165928 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d403e610 con 0x7fd7e40ce930 2024-03-23T03:47:18.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.521+0000 7fd7dbe00640 1 -- 172.21.15.102:0/1648165928 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7d403e770 con 0x7fd7e40ce930 2024-03-23T03:47:18.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.521+0000 7fd7dbe00640 1 -- 172.21.15.102:0/1648165928 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7d403eb20 con 0x7fd7e40ce930 2024-03-23T03:47:18.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.521+0000 7fd7ea800640 1 -- 172.21.15.102:0/1648165928 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e4153970 con 0x7fd7e40ce930 2024-03-23T03:47:18.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.521+0000 7fd7ea800640 1 -- 172.21.15.102:0/1648165928 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e4153f30 con 0x7fd7e40ce930 2024-03-23T03:47:18.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.522+0000 7f522d800640 1 Processor -- start 2024-03-23T03:47:18.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.522+0000 7f522d800640 1 -- start start 2024-03-23T03:47:18.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.522+0000 7f0183400640 1 --2- 172.21.15.102:0/1418855141 >> [v2:172.21.15.173:6816/37509,v1:172.21.15.173:6817/37509] conn(0x7f01600741d0 0x7f0160076610 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.521+0000 7fae1a000640 1 -- 172.21.15.102:0/2752044212 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fae100889e0 con 0x7fae241627c0 2024-03-23T03:47:18.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.521+0000 7fae21e00640 1 --2- 172.21.15.102:0/2752044212 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fae0406ac50 0x7fae0406d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.521+0000 7fae1a000640 1 --2- 172.21.15.102:0/2752044212 >> [v2:172.21.15.102:6808/37353,v1:172.21.15.102:6809/37353] conn(0x7fae04074290 0x7fae040766d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.521+0000 7fae1a000640 1 -- 172.21.15.102:0/2752044212 --> [v2:172.21.15.102:6808/37353,v1:172.21.15.102:6809/37353] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae04076da0 con 0x7fae04074290 2024-03-23T03:47:18.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.521+0000 7fae1a000640 1 -- 172.21.15.102:0/2752044212 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae10088d60 con 0x7fae241627c0 2024-03-23T03:47:18.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.523+0000 7fae21e00640 1 --2- 172.21.15.102:0/2752044212 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fae0406ac50 0x7fae0406d110 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fae2406c5b0 tx=0x7fae14009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.522+0000 7fd7ea800640 1 -- 172.21.15.102:0/1648165928 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd7e4161750 con 0x7fd7e40ce930 2024-03-23T03:47:18.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.523+0000 7fa6b3400640 1 -- 172.21.15.102:0/2621027432 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa6a8013550 con 0x7fa6bc070bb0 2024-03-23T03:47:18.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.523+0000 7fae22800640 1 --2- 172.21.15.102:0/2752044212 >> [v2:172.21.15.102:6808/37353,v1:172.21.15.102:6809/37353] conn(0x7fae04074290 0x7fae040766d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.524+0000 7f522d800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52280cd6b0 0x7f5228157ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.524+0000 7f522d800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f52280cda90 0x7f52281e8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.524+0000 7f522d800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f522815cd90 0x7f52280b6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.524+0000 7f4c16400640 1 --2- 172.21.15.102:0/713833806 >> [v2:172.21.15.102:6824/37352,v1:172.21.15.102:6825/37352] conn(0x7f4bf00741c0 0x7f4bf0076600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.524+0000 7f522d800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52281f23a0 con 0x7f522815cd90 2024-03-23T03:47:18.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.524+0000 7f522d800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52281f2650 con 0x7f52280cd6b0 2024-03-23T03:47:18.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.524+0000 7f522d800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52281f2930 con 0x7f52280cda90 2024-03-23T03:47:18.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.524+0000 7f017be00640 1 -- 172.21.15.102:0/1418855141 <== osd.4 v2:172.21.15.173:6816/37509 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f0160076ce0 con 0x7f01600741d0 2024-03-23T03:47:18.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.524+0000 7f0183e00640 1 --2- 172.21.15.102:0/1418855141 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f016006ac90 0x7f016006d150 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f01840cc9c0 tx=0x7f016c007be0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.524+0000 7fae22800640 1 --2- 172.21.15.102:0/2752044212 >> [v2:172.21.15.102:6808/37353,v1:172.21.15.102:6809/37353] conn(0x7fae04074290 0x7fae040766d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.524+0000 7fa6b3400640 1 --2- 172.21.15.102:0/2621027432 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa69c06ac30 0x7fa69c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.525+0000 7fa6b3400640 1 -- 172.21.15.102:0/2621027432 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa6a808a320 con 0x7fa6bc070bb0 2024-03-23T03:47:18.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.525+0000 7fa6b3400640 1 --2- 172.21.15.102:0/2621027432 >> [v2:172.21.15.173:6800/37507,v1:172.21.15.173:6801/37507] conn(0x7fa69c074270 0x7fa69c0766b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.525+0000 7fa6b3400640 1 -- 172.21.15.102:0/2621027432 --> [v2:172.21.15.173:6800/37507,v1:172.21.15.173:6801/37507] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa69c076d80 con 0x7fa69c074270 2024-03-23T03:47:18.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.525+0000 7fa6b3400640 1 -- 172.21.15.102:0/2621027432 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6a808a6d0 con 0x7fa6bc070bb0 2024-03-23T03:47:18.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.525+0000 7fa6bb400640 1 --2- 172.21.15.102:0/2621027432 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa69c06ac30 0x7fa69c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.525+0000 7f5226a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52280cd6b0 0x7f5228157ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.525+0000 7f5226000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f52280cda90 0x7f52281e8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.525+0000 7f5226a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52280cd6b0 0x7f5228157ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:42312/0 (socket says 172.21.15.102:42312) 2024-03-23T03:47:18.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.525+0000 7f5226a00640 1 -- 172.21.15.102:0/206965771 learned_addr learned my addr 172.21.15.102:0/206965771 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.526+0000 7f5226a00640 1 -- 172.21.15.102:0/206965771 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f52280cda90 msgr2=0x7f52281e8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.526+0000 7f5226a00640 1 --2- 172.21.15.102:0/206965771 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f52280cda90 0x7f52281e8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.526+0000 7f5226a00640 1 -- 172.21.15.102:0/206965771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f522815cd90 msgr2=0x7f52280b6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.526+0000 7f5226a00640 1 --2- 172.21.15.102:0/206965771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f522815cd90 0x7f52280b6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.526+0000 7f5226a00640 1 -- 172.21.15.102:0/206965771 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52180094d0 con 0x7f52280cd6b0 2024-03-23T03:47:18.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.526+0000 7fae1a000640 1 -- 172.21.15.102:0/2752044212 <== osd.3 v2:172.21.15.102:6808/37353 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fae04076da0 con 0x7fae04074290 2024-03-23T03:47:18.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.526+0000 7fd7dbe00640 1 -- 172.21.15.102:0/1648165928 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd7d403b050 con 0x7fd7e40ce930 2024-03-23T03:47:18.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.526+0000 7f5226a00640 1 --2- 172.21.15.102:0/206965771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52280cd6b0 0x7f5228157ca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f521000c410 tx=0x7f521000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.527+0000 7f521ea00640 1 -- 172.21.15.102:0/206965771 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52100035f0 con 0x7f52280cd6b0 2024-03-23T03:47:18.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.527+0000 7f522d800640 1 -- 172.21.15.102:0/206965771 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52281f2c10 con 0x7f52280cd6b0 2024-03-23T03:47:18.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.527+0000 7f522d800640 1 -- 172.21.15.102:0/206965771 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52281f31d0 con 0x7f52280cd6b0 2024-03-23T03:47:18.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.527+0000 7f4c16400640 1 --2- 172.21.15.102:0/713833806 >> [v2:172.21.15.102:6824/37352,v1:172.21.15.102:6825/37352] conn(0x7f4bf00741c0 0x7f4bf0076600 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.526+0000 7f235e200640 1 -- 172.21.15.102:0/2644647259 --> [v2:172.21.15.102:6800/37350,v1:172.21.15.102:6801/37350] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f23580b3aa0 con 0x7f2338074340 2024-03-23T03:47:18.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.527+0000 7f2354c00640 1 -- 172.21.15.102:0/2644647259 <== osd.2 v2:172.21.15.102:6800/37350 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f23580b3aa0 con 0x7f2338074340 2024-03-23T03:47:18.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.527+0000 7f235e200640 1 -- 172.21.15.102:0/2644647259 >> [v2:172.21.15.102:6800/37350,v1:172.21.15.102:6801/37350] conn(0x7f2338074340 msgr2=0x7f2338076780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.527+0000 7f235e200640 1 --2- 172.21.15.102:0/2644647259 >> [v2:172.21.15.102:6800/37350,v1:172.21.15.102:6801/37350] conn(0x7f2338074340 0x7f2338076780 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.528+0000 7f521ea00640 1 -- 172.21.15.102:0/206965771 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5210003750 con 0x7f52280cd6b0 2024-03-23T03:47:18.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.528+0000 7f521ea00640 1 -- 172.21.15.102:0/206965771 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52100035f0 con 0x7f52280cd6b0 2024-03-23T03:47:18.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.528+0000 7f522d800640 1 -- 172.21.15.102:0/206965771 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5200000fc0 con 0x7f52280cd6b0 2024-03-23T03:47:18.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.527+0000 7fd7dbe00640 1 --2- 172.21.15.102:0/1648165928 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd7c006ac50 0x7fd7c006d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.527+0000 7fd7dbe00640 1 -- 172.21.15.102:0/1648165928 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd7d40aba90 con 0x7fd7e40ce930 2024-03-23T03:47:18.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.528+0000 7fd7dbe00640 1 --2- 172.21.15.102:0/1648165928 >> [v2:172.21.15.173:6818/37510,v1:172.21.15.173:6821/37510] conn(0x7fd7c0074210 0x7fd7c0076650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.528+0000 7fd7dbe00640 1 -- 172.21.15.102:0/1648165928 --> [v2:172.21.15.173:6818/37510,v1:172.21.15.173:6821/37510] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd7c0076d20 con 0x7fd7c0074210 2024-03-23T03:47:18.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.528+0000 7fd7daa00640 1 --2- 172.21.15.102:0/1648165928 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd7c006ac50 0x7fd7c006d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.528+0000 7fa6bb400640 1 --2- 172.21.15.102:0/2621027432 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa69c06ac30 0x7fa69c06d0f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac036000 tx=0x7fa6ac009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.528+0000 7fa6bbe00640 1 --2- 172.21.15.102:0/2621027432 >> [v2:172.21.15.173:6800/37507,v1:172.21.15.173:6801/37507] conn(0x7fa69c074270 0x7fa69c0766b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.529+0000 7f235e200640 1 -- 172.21.15.102:0/2644647259 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f233806ad00 msgr2=0x7f233806d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.529+0000 7f235e200640 1 --2- 172.21.15.102:0/2644647259 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f233806ad00 0x7f233806d1c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2358159750 tx=0x7f2344008040 comp rx=0 tx=0).stop 2024-03-23T03:47:18.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.529+0000 7f235e200640 1 -- 172.21.15.102:0/2644647259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23580b7fe0 msgr2=0x7f23580c8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.529+0000 7f235e200640 1 --2- 172.21.15.102:0/2644647259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23580b7fe0 0x7f23580c8c90 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f235000c7b0 tx=0x7f235000cc70 comp rx=0 tx=0).stop 2024-03-23T03:47:18.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.529+0000 7f235d800640 1 -- 172.21.15.102:0/2644647259 reap_dead start 2024-03-23T03:47:18.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.529+0000 7f235e200640 1 -- 172.21.15.102:0/2644647259 shutdown_connections 2024-03-23T03:47:18.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.529+0000 7f235e200640 1 -- 172.21.15.102:0/2644647259 >> 172.21.15.102:0/2644647259 conn(0x7f235806ae90 msgr2=0x7f235815db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.529+0000 7fd7dbe00640 1 -- 172.21.15.102:0/1648165928 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7d4076aa0 con 0x7fd7e40ce930 2024-03-23T03:47:18.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.529+0000 7f4c0d600640 1 -- 172.21.15.102:0/713833806 <== osd.1 v2:172.21.15.102:6824/37352 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4bf0076cd0 con 0x7f4bf00741c0 2024-03-23T03:47:18.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.529+0000 7fa6bbe00640 1 --2- 172.21.15.102:0/2621027432 >> [v2:172.21.15.173:6800/37507,v1:172.21.15.173:6801/37507] conn(0x7fa69c074270 0x7fa69c0766b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.529+0000 7f235e200640 1 -- 172.21.15.102:0/2644647259 shutdown_connections 2024-03-23T03:47:18.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.529+0000 7f235e200640 1 -- 172.21.15.102:0/2644647259 wait complete. 2024-03-23T03:47:18.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.529+0000 7fd7e3e00640 1 --2- 172.21.15.102:0/1648165928 >> [v2:172.21.15.173:6818/37510,v1:172.21.15.173:6821/37510] conn(0x7fd7c0074210 0x7fd7c0076650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.530+0000 7fd7daa00640 1 --2- 172.21.15.102:0/1648165928 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd7c006ac50 0x7fd7c006d110 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd7e41f98a0 tx=0x7fd7cc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.531+0000 7fd7e3e00640 1 --2- 172.21.15.102:0/1648165928 >> [v2:172.21.15.173:6818/37510,v1:172.21.15.173:6821/37510] conn(0x7fd7c0074210 0x7fd7c0076650 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.532+0000 7f521ea00640 1 -- 172.21.15.102:0/206965771 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f52100038c0 con 0x7f52280cd6b0 2024-03-23T03:47:18.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.533+0000 7fa6b3400640 1 -- 172.21.15.102:0/2621027432 <== osd.5 v2:172.21.15.173:6800/37507 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fa69c076d80 con 0x7fa69c074270 2024-03-23T03:47:18.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.534+0000 7f521ea00640 1 --2- 172.21.15.102:0/206965771 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f520406ac60 0x7f520406d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.534+0000 7fd7dbe00640 1 -- 172.21.15.102:0/1648165928 <== osd.7 v2:172.21.15.173:6818/37510 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7fd7c0076d20 con 0x7fd7c0074210 2024-03-23T03:47:18.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.534+0000 7f521ea00640 1 -- 172.21.15.102:0/206965771 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5210088c00 con 0x7f52280cd6b0 2024-03-23T03:47:18.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.534+0000 7f521ea00640 1 --2- 172.21.15.102:0/206965771 >> [v2:172.21.15.173:6808/37508,v1:172.21.15.173:6809/37508] conn(0x7f52040742a0 0x7f52040766e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.534+0000 7f5226000640 1 --2- 172.21.15.102:0/206965771 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f520406ac60 0x7f520406d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.534+0000 7f521ea00640 1 -- 172.21.15.102:0/206965771 --> [v2:172.21.15.173:6808/37508,v1:172.21.15.173:6809/37508] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5204076db0 con 0x7f52040742a0 2024-03-23T03:47:18.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.534+0000 7f521ea00640 1 -- 172.21.15.102:0/206965771 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5210088fb0 con 0x7f52280cd6b0 2024-03-23T03:47:18.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.535+0000 7f5227400640 1 --2- 172.21.15.102:0/206965771 >> [v2:172.21.15.173:6808/37508,v1:172.21.15.173:6809/37508] conn(0x7f52040742a0 0x7f52040766e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.536+0000 7f5226000640 1 --2- 172.21.15.102:0/206965771 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f520406ac60 0x7f520406d120 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5218036000 tx=0x7f521803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.536+0000 7f5227400640 1 --2- 172.21.15.102:0/206965771 >> [v2:172.21.15.173:6808/37508,v1:172.21.15.173:6809/37508] conn(0x7f52040742a0 0x7f52040766e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.538+0000 7fb670a00640 1 -- 172.21.15.102:0/1590126895 --> [v2:172.21.15.102:6816/37351,v1:172.21.15.102:6817/37351] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb66c0b1760 con 0x7fb64c0741f0 2024-03-23T03:47:18.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.538+0000 7fb662000640 1 -- 172.21.15.102:0/1590126895 <== osd.0 v2:172.21.15.102:6816/37351 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb664007a20 con 0x7fb64c0741f0 2024-03-23T03:47:18.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.539+0000 7f521ea00640 1 -- 172.21.15.102:0/206965771 <== osd.6 v2:172.21.15.173:6808/37508 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f5204076db0 con 0x7f52040742a0 2024-03-23T03:47:18.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.539+0000 7fb670a00640 1 -- 172.21.15.102:0/1590126895 >> [v2:172.21.15.102:6816/37351,v1:172.21.15.102:6817/37351] conn(0x7fb64c0741f0 msgr2=0x7fb64c076630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.539+0000 7fb670a00640 1 --2- 172.21.15.102:0/1590126895 >> [v2:172.21.15.102:6816/37351,v1:172.21.15.102:6817/37351] conn(0x7fb64c0741f0 0x7fb64c076630 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.539+0000 7fb670a00640 1 -- 172.21.15.102:0/1590126895 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb64c06ac30 msgr2=0x7fb64c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.540+0000 7fb670a00640 1 --2- 172.21.15.102:0/1590126895 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb64c06ac30 0x7fb64c06d0f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0cdbb0 tx=0x7fb658009040 comp rx=0 tx=0).stop 2024-03-23T03:47:18.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.540+0000 7fb670a00640 1 -- 172.21.15.102:0/1590126895 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb66c06da40 msgr2=0x7fb66c159690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.540+0000 7fb670a00640 1 --2- 172.21.15.102:0/1590126895 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb66c06da40 0x7fb66c159690 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb65c000c00 tx=0x7fb65c0046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.540+0000 7fb66a800640 1 -- 172.21.15.102:0/1590126895 reap_dead start 2024-03-23T03:47:18.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.540+0000 7fb670a00640 1 -- 172.21.15.102:0/1590126895 shutdown_connections 2024-03-23T03:47:18.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.540+0000 7fb670a00640 1 -- 172.21.15.102:0/1590126895 >> 172.21.15.102:0/1590126895 conn(0x7fb66c066000 msgr2=0x7fb66c0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.540+0000 7fb670a00640 1 -- 172.21.15.102:0/1590126895 shutdown_connections 2024-03-23T03:47:18.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.540+0000 7fb670a00640 1 -- 172.21.15.102:0/1590126895 wait complete. 2024-03-23T03:47:18.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.553+0000 7fa6c2800640 1 -- 172.21.15.102:0/2621027432 --> [v2:172.21.15.173:6800/37507,v1:172.21.15.173:6801/37507] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa694002cb0 con 0x7fa69c074270 2024-03-23T03:47:18.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.553+0000 7f018ae00640 1 -- 172.21.15.102:0/1418855141 --> [v2:172.21.15.173:6816/37509,v1:172.21.15.173:6817/37509] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f01841615f0 con 0x7f01600741d0 2024-03-23T03:47:18.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.555+0000 7f017be00640 1 -- 172.21.15.102:0/1418855141 <== osd.4 v2:172.21.15.173:6816/37509 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0160076ce0 con 0x7f01600741d0 2024-03-23T03:47:18.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.556+0000 7f018ae00640 1 -- 172.21.15.102:0/1418855141 >> [v2:172.21.15.173:6816/37509,v1:172.21.15.173:6817/37509] conn(0x7f01600741d0 msgr2=0x7f0160076610 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.556+0000 7f018ae00640 1 --2- 172.21.15.102:0/1418855141 >> [v2:172.21.15.173:6816/37509,v1:172.21.15.173:6817/37509] conn(0x7f01600741d0 0x7f0160076610 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.556+0000 7f018ae00640 1 -- 172.21.15.102:0/1418855141 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f016006ac90 msgr2=0x7f016006d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.556+0000 7f018ae00640 1 --2- 172.21.15.102:0/1418855141 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f016006ac90 0x7f016006d150 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f01840cc9c0 tx=0x7f016c007be0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.556+0000 7f018ae00640 1 -- 172.21.15.102:0/1418855141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f018415cc70 msgr2=0x7f01840c6ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.556+0000 7f018ae00640 1 --2- 172.21.15.102:0/1418855141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f018415cc70 0x7f01840c6ab0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f017c009eb0 tx=0x7f017c004a40 comp rx=0 tx=0).stop 2024-03-23T03:47:18.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.557+0000 7f0188e00640 1 -- 172.21.15.102:0/1418855141 reap_dead start 2024-03-23T03:47:18.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.554+0000 7fa6b3400640 1 -- 172.21.15.102:0/2621027432 <== osd.5 v2:172.21.15.173:6800/37507 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa694002cb0 con 0x7fa69c074270 2024-03-23T03:47:18.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.555+0000 7fa6c2800640 1 -- 172.21.15.102:0/2621027432 >> [v2:172.21.15.173:6800/37507,v1:172.21.15.173:6801/37507] conn(0x7fa69c074270 msgr2=0x7fa69c0766b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.555+0000 7fa6c2800640 1 --2- 172.21.15.102:0/2621027432 >> [v2:172.21.15.173:6800/37507,v1:172.21.15.173:6801/37507] conn(0x7fa69c074270 0x7fa69c0766b0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.555+0000 7fa6c2800640 1 -- 172.21.15.102:0/2621027432 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa69c06ac30 msgr2=0x7fa69c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.555+0000 7fa6c2800640 1 --2- 172.21.15.102:0/2621027432 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa69c06ac30 0x7fa69c06d0f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac036000 tx=0x7fa6ac009030 comp rx=0 tx=0).stop 2024-03-23T03:47:18.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.555+0000 7fa6c2800640 1 -- 172.21.15.102:0/2621027432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6bc070bb0 msgr2=0x7fa6bc158610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.555+0000 7fa6c2800640 1 --2- 172.21.15.102:0/2621027432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6bc070bb0 0x7fa6bc158610 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa6a80072c0 tx=0x7fa6a8007780 comp rx=0 tx=0).stop 2024-03-23T03:47:18.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.555+0000 7fa6bbe00640 1 -- 172.21.15.102:0/2621027432 reap_dead start 2024-03-23T03:47:18.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.556+0000 7fa6c2800640 1 -- 172.21.15.102:0/2621027432 shutdown_connections 2024-03-23T03:47:18.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.556+0000 7fa6c2800640 1 -- 172.21.15.102:0/2621027432 >> 172.21.15.102:0/2621027432 conn(0x7fa6bc0cf790 msgr2=0x7fa6bc0cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.556+0000 7fa6c2800640 1 -- 172.21.15.102:0/2621027432 shutdown_connections 2024-03-23T03:47:18.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.556+0000 7fa6c2800640 1 -- 172.21.15.102:0/2621027432 wait complete. 2024-03-23T03:47:18.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.557+0000 7f018ae00640 1 -- 172.21.15.102:0/1418855141 shutdown_connections 2024-03-23T03:47:18.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.557+0000 7f018ae00640 1 -- 172.21.15.102:0/1418855141 >> 172.21.15.102:0/1418855141 conn(0x7f01840cf790 msgr2=0x7f01840b4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.582 INFO:teuthology.orchestra.run.smithi102.stdout:60129542153 2024-03-23T03:47:18.582 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-23T03:47:18.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.557+0000 7f018ae00640 1 -- 172.21.15.102:0/1418855141 shutdown_connections 2024-03-23T03:47:18.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.558+0000 7f018ae00640 1 -- 172.21.15.102:0/1418855141 wait complete. 2024-03-23T03:47:18.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.561+0000 7fae28e00640 1 -- 172.21.15.102:0/2752044212 --> [v2:172.21.15.102:6808/37353,v1:172.21.15.102:6809/37353] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fae240cd290 con 0x7fae04074290 2024-03-23T03:47:18.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.562+0000 7fae1a000640 1 -- 172.21.15.102:0/2752044212 <== osd.3 v2:172.21.15.102:6808/37353 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fae240cd290 con 0x7fae04074290 2024-03-23T03:47:18.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.562+0000 7fae28e00640 1 -- 172.21.15.102:0/2752044212 >> [v2:172.21.15.102:6808/37353,v1:172.21.15.102:6809/37353] conn(0x7fae04074290 msgr2=0x7fae040766d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.562+0000 7fae28e00640 1 --2- 172.21.15.102:0/2752044212 >> [v2:172.21.15.102:6808/37353,v1:172.21.15.102:6809/37353] conn(0x7fae04074290 0x7fae040766d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.563+0000 7fae28e00640 1 -- 172.21.15.102:0/2752044212 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fae0406ac50 msgr2=0x7fae0406d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.563+0000 7fae28e00640 1 --2- 172.21.15.102:0/2752044212 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fae0406ac50 0x7fae0406d110 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fae2406c5b0 tx=0x7fae14009040 comp rx=0 tx=0).stop 2024-03-23T03:47:18.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.563+0000 7fae28e00640 1 -- 172.21.15.102:0/2752044212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae241627c0 msgr2=0x7fae241f6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.563+0000 7fae28e00640 1 --2- 172.21.15.102:0/2752044212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae241627c0 0x7fae241f6aa0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fae1000b3f0 tx=0x7fae1000b8b0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.563+0000 7fae22800640 1 -- 172.21.15.102:0/2752044212 reap_dead start 2024-03-23T03:47:18.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.563+0000 7fae28e00640 1 -- 172.21.15.102:0/2752044212 shutdown_connections 2024-03-23T03:47:18.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.563+0000 7fae28e00640 1 -- 172.21.15.102:0/2752044212 >> 172.21.15.102:0/2752044212 conn(0x7fae2406a0b0 msgr2=0x7fae2406d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.563+0000 7fae28e00640 1 -- 172.21.15.102:0/2752044212 shutdown_connections 2024-03-23T03:47:18.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.564+0000 7fae28e00640 1 -- 172.21.15.102:0/2752044212 wait complete. 2024-03-23T03:47:18.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.564+0000 7f4c02a00640 1 -- 172.21.15.102:0/713833806 --> [v2:172.21.15.102:6824/37352,v1:172.21.15.102:6825/37352] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4be4002cb0 con 0x7f4bf00741c0 2024-03-23T03:47:18.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.565+0000 7f4c0d600640 1 -- 172.21.15.102:0/713833806 <== osd.1 v2:172.21.15.102:6824/37352 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4be4002cb0 con 0x7f4bf00741c0 2024-03-23T03:47:18.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.566+0000 7fd7ea800640 1 -- 172.21.15.102:0/1648165928 --> [v2:172.21.15.173:6818/37510,v1:172.21.15.173:6821/37510] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd7e406d010 con 0x7fd7c0074210 2024-03-23T03:47:18.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.567+0000 7f4c02a00640 1 -- 172.21.15.102:0/713833806 >> [v2:172.21.15.102:6824/37352,v1:172.21.15.102:6825/37352] conn(0x7f4bf00741c0 msgr2=0x7f4bf0076600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.567+0000 7f4c02a00640 1 --2- 172.21.15.102:0/713833806 >> [v2:172.21.15.102:6824/37352,v1:172.21.15.102:6825/37352] conn(0x7f4bf00741c0 0x7f4bf0076600 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.567+0000 7f4c02a00640 1 -- 172.21.15.102:0/713833806 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4bf006ac00 msgr2=0x7f4bf006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.567+0000 7f4c02a00640 1 --2- 172.21.15.102:0/713833806 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4bf006ac00 0x7f4bf006d0c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4c10153850 tx=0x7f4c04008040 comp rx=0 tx=0).stop 2024-03-23T03:47:18.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.567+0000 7f4c02a00640 1 -- 172.21.15.102:0/713833806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c1006d330 msgr2=0x7f4c10148910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.567+0000 7f4c02a00640 1 --2- 172.21.15.102:0/713833806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c1006d330 0x7f4c10148910 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4c0800c940 tx=0x7f4c0800ce00 comp rx=0 tx=0).stop 2024-03-23T03:47:18.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.567+0000 7f4c16400640 1 -- 172.21.15.102:0/713833806 reap_dead start 2024-03-23T03:47:18.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.567+0000 7f4c02a00640 1 -- 172.21.15.102:0/713833806 shutdown_connections 2024-03-23T03:47:18.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.567+0000 7f4c02a00640 1 -- 172.21.15.102:0/713833806 >> 172.21.15.102:0/713833806 conn(0x7f4c10069c90 msgr2=0x7f4c1015f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.567+0000 7fd7dbe00640 1 -- 172.21.15.102:0/1648165928 <== osd.7 v2:172.21.15.173:6818/37510 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd7dc007300 con 0x7fd7c0074210 2024-03-23T03:47:18.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.568+0000 7fd7ea800640 1 -- 172.21.15.102:0/1648165928 >> [v2:172.21.15.173:6818/37510,v1:172.21.15.173:6821/37510] conn(0x7fd7c0074210 msgr2=0x7fd7c0076650 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.568+0000 7fd7ea800640 1 --2- 172.21.15.102:0/1648165928 >> [v2:172.21.15.173:6818/37510,v1:172.21.15.173:6821/37510] conn(0x7fd7c0074210 0x7fd7c0076650 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.568+0000 7fd7ea800640 1 -- 172.21.15.102:0/1648165928 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd7c006ac50 msgr2=0x7fd7c006d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.568+0000 7fd7ea800640 1 --2- 172.21.15.102:0/1648165928 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd7c006ac50 0x7fd7c006d110 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd7e41f98a0 tx=0x7fd7cc008040 comp rx=0 tx=0).stop 2024-03-23T03:47:18.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.568+0000 7fd7ea800640 1 -- 172.21.15.102:0/1648165928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7e40ce930 msgr2=0x7fd7e415c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.568+0000 7fd7ea800640 1 --2- 172.21.15.102:0/1648165928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7e40ce930 0x7fd7e415c450 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4009eb0 tx=0x7fd7d4004a40 comp rx=0 tx=0).stop 2024-03-23T03:47:18.589 INFO:teuthology.orchestra.run.smithi102.stdout:60129542152 2024-03-23T03:47:18.590 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-23T03:47:18.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.569+0000 7f4c02a00640 1 -- 172.21.15.102:0/713833806 shutdown_connections 2024-03-23T03:47:18.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.569+0000 7f4c02a00640 1 -- 172.21.15.102:0/713833806 wait complete. 2024-03-23T03:47:18.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.568+0000 7fd7e3e00640 1 -- 172.21.15.102:0/1648165928 reap_dead start 2024-03-23T03:47:18.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.568+0000 7f522d800640 1 -- 172.21.15.102:0/206965771 --> [v2:172.21.15.173:6808/37508,v1:172.21.15.173:6809/37508] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5200002cb0 con 0x7f52040742a0 2024-03-23T03:47:18.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.569+0000 7fd7ea800640 1 -- 172.21.15.102:0/1648165928 shutdown_connections 2024-03-23T03:47:18.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.569+0000 7fd7ea800640 1 -- 172.21.15.102:0/1648165928 >> 172.21.15.102:0/1648165928 conn(0x7fd7e406a0b0 msgr2=0x7fd7e4164080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.569+0000 7fd7ea800640 1 -- 172.21.15.102:0/1648165928 shutdown_connections 2024-03-23T03:47:18.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.569+0000 7fd7ea800640 1 -- 172.21.15.102:0/1648165928 wait complete. 2024-03-23T03:47:18.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.570+0000 7f521ea00640 1 -- 172.21.15.102:0/206965771 <== osd.6 v2:172.21.15.173:6808/37508 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5200002cb0 con 0x7f52040742a0 2024-03-23T03:47:18.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.571+0000 7f522d800640 1 -- 172.21.15.102:0/206965771 >> [v2:172.21.15.173:6808/37508,v1:172.21.15.173:6809/37508] conn(0x7f52040742a0 msgr2=0x7f52040766e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.571+0000 7f522d800640 1 --2- 172.21.15.102:0/206965771 >> [v2:172.21.15.173:6808/37508,v1:172.21.15.173:6809/37508] conn(0x7f52040742a0 0x7f52040766e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.571+0000 7f522d800640 1 -- 172.21.15.102:0/206965771 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f520406ac60 msgr2=0x7f520406d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.571+0000 7f522d800640 1 --2- 172.21.15.102:0/206965771 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f520406ac60 0x7f520406d120 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5218036000 tx=0x7f521803c040 comp rx=0 tx=0).stop 2024-03-23T03:47:18.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.571+0000 7f522d800640 1 -- 172.21.15.102:0/206965771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52280cd6b0 msgr2=0x7f5228157ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.571+0000 7f522d800640 1 --2- 172.21.15.102:0/206965771 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f52280cd6b0 0x7f5228157ca0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f521000c410 tx=0x7f521000c8d0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.572+0000 7f5227400640 1 -- 172.21.15.102:0/206965771 reap_dead start 2024-03-23T03:47:18.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.572+0000 7f522d800640 1 -- 172.21.15.102:0/206965771 shutdown_connections 2024-03-23T03:47:18.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.572+0000 7f522d800640 1 -- 172.21.15.102:0/206965771 >> 172.21.15.102:0/206965771 conn(0x7f52280cf780 msgr2=0x7f522806b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.572+0000 7f522d800640 1 -- 172.21.15.102:0/206965771 shutdown_connections 2024-03-23T03:47:18.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.572+0000 7f522d800640 1 -- 172.21.15.102:0/206965771 wait complete. 2024-03-23T03:47:18.594 INFO:teuthology.orchestra.run.smithi102.stdout:60129542152 2024-03-23T03:47:18.595 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-23T03:47:18.596 INFO:teuthology.orchestra.run.smithi102.stdout:60129542153 2024-03-23T03:47:18.596 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-23T03:47:18.597 INFO:teuthology.orchestra.run.smithi102.stdout:60129542153 2024-03-23T03:47:18.597 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-23T03:47:18.598 INFO:teuthology.orchestra.run.smithi102.stdout:60129542152 2024-03-23T03:47:18.598 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-23T03:47:18.599 INFO:teuthology.orchestra.run.smithi102.stdout:60129542152 2024-03-23T03:47:18.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-23T03:47:18.600 INFO:teuthology.orchestra.run.smithi102.stdout:60129542154 2024-03-23T03:47:18.600 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-23T03:47:18.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.921+0000 7f9138e00640 1 Processor -- start 2024-03-23T03:47:18.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.922+0000 7f9138e00640 1 -- start start 2024-03-23T03:47:18.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.924+0000 7f9138e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91340cd6b0 0x7f91340b8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.924+0000 7f9138e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f913415cd90 0x7f913406b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.925+0000 7f9138e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913406b5a0 con 0x7f91340cda90 2024-03-23T03:47:18.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.925+0000 7f9132800640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f91340cda90 0x7f91341606c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:58910/0 (socket says 172.21.15.102:58910) 2024-03-23T03:47:18.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.925+0000 7f9132800640 1 -- 172.21.15.102:0/2550132926 learned_addr learned my addr 172.21.15.102:0/2550132926 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:18.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.925+0000 7f9131400640 1 --2- 172.21.15.102:0/2550132926 >> v2:172.21.15.173:3300/0 conn(0x7f913415cd90 0x7f913406b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.925+0000 7f9131e00640 1 --2- 172.21.15.102:0/2550132926 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91340cd6b0 0x7f91340b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.925+0000 7f9138e00640 1 -- 172.21.15.102:0/2550132926 --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f913406c0e0 con 0x7f913415cd90 2024-03-23T03:47:18.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.925+0000 7f9138e00640 1 -- 172.21.15.102:0/2550132926 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913406b840 con 0x7f91340cd6b0 2024-03-23T03:47:18.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.926+0000 7f8526400640 1 Processor -- start 2024-03-23T03:47:18.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.926+0000 7f8526400640 1 -- start start 2024-03-23T03:47:18.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.926+0000 7f9130a00640 1 -- 172.21.15.102:0/2550132926 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722770618 0 0) 0x7f913406b5a0 con 0x7f91340cda90 2024-03-23T03:47:18.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.926+0000 7f9130a00640 1 -- 172.21.15.102:0/2550132926 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9118003520 con 0x7f91340cda90 2024-03-23T03:47:18.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.926+0000 7f9131400640 1 -- 172.21.15.102:0/2550132926 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91340cd6b0 msgr2=0x7f91340b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.926+0000 7f9131400640 1 --2- 172.21.15.102:0/2550132926 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91340cd6b0 0x7f91340b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.926+0000 7f9131400640 1 -- 172.21.15.102:0/2550132926 >> v1:172.21.15.173:6789/0 conn(0x7f91340cda90 legacy=0x7f91341606c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.926+0000 7f9131400640 1 -- 172.21.15.102:0/2550132926 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913406bb20 con 0x7f913415cd90 2024-03-23T03:47:18.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.928+0000 7f9636800640 1 Processor -- start 2024-03-23T03:47:18.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.928+0000 7f8526400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f85200cebb0 0x7f852006d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.928+0000 7f8526400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f852006d7b0 0x7f85201612c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.928+0000 7f8526400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f852015ed20 con 0x7f852015e940 2024-03-23T03:47:18.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.928+0000 7f8526400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85200b2000 con 0x7f852006d7b0 2024-03-23T03:47:18.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.928+0000 7f8526400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852006dfa0 con 0x7f85200cebb0 2024-03-23T03:47:18.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.928+0000 7f9636800640 1 -- start start 2024-03-23T03:47:18.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.929+0000 7f851f400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f852015e940 0x7f85200ce4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41182/0 (socket says 172.21.15.102:41182) 2024-03-23T03:47:18.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.929+0000 7f851f400640 1 -- 172.21.15.102:0/651099303 learned_addr learned my addr 172.21.15.102:0/651099303 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:18.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.929+0000 7f851ea00640 1 --2- 172.21.15.102:0/651099303 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f85200cebb0 0x7f852006d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.929+0000 7f9131400640 1 --2- 172.21.15.102:0/2550132926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f913415cd90 0x7f913406b060 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f91240097a0 tx=0x7f912402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaffa1a83da175bf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.930+0000 7f851fe00640 1 --2- 172.21.15.102:0/651099303 >> v2:172.21.15.102:3300/0 conn(0x7f852006d7b0 0x7f85201612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.930+0000 7f9130a00640 1 -- 172.21.15.102:0/2550132926 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912402d040 con 0x7f913415cd90 2024-03-23T03:47:18.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.930+0000 7f9130a00640 1 -- 172.21.15.102:0/2550132926 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9124031810 con 0x7f913415cd90 2024-03-23T03:47:18.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.931+0000 7f9636800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f963015a2f0 0x7f9630160f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.931+0000 7f9636800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f963015a6d0 0x7f96300ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.931+0000 7f9636800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f963015cf20 0x7f96300ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.931+0000 7f9636800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9630161460 con 0x7f963015a6d0 2024-03-23T03:47:18.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.931+0000 7f9636800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f963006d1c0 con 0x7f963015a2f0 2024-03-23T03:47:18.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.931+0000 7f9636800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963015d4a0 con 0x7f963015cf20 2024-03-23T03:47:18.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.930+0000 7f851ea00640 1 -- 172.21.15.102:0/651099303 >> v2:172.21.15.102:3300/0 conn(0x7f852006d7b0 msgr2=0x7f85201612c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.930+0000 7f851ea00640 1 --2- 172.21.15.102:0/651099303 >> v2:172.21.15.102:3300/0 conn(0x7f852006d7b0 0x7f85201612c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.930+0000 7f851ea00640 1 -- 172.21.15.102:0/651099303 >> v1:172.21.15.102:6789/0 conn(0x7f852015e940 legacy=0x7f85200ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.930+0000 7f851ea00640 1 -- 172.21.15.102:0/651099303 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85200b1660 con 0x7f85200cebb0 2024-03-23T03:47:18.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.932+0000 7f962ea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f963015a6d0 0x7f96300ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.932+0000 7f962ea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f963015a6d0 0x7f96300ce990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37304/0 (socket says 172.21.15.102:37304) 2024-03-23T03:47:18.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.932+0000 7f962ea00640 1 -- 172.21.15.102:0/1008750998 learned_addr learned my addr 172.21.15.102:0/1008750998 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.931+0000 7f9130a00640 1 -- 172.21.15.102:0/2550132926 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912402d040 con 0x7f913415cd90 2024-03-23T03:47:18.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.932+0000 7f962f400640 1 --2- 172.21.15.102:0/1008750998 >> v2:172.21.15.173:3300/0 conn(0x7f963015a2f0 0x7f9630160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.932+0000 7f851ea00640 1 --2- 172.21.15.102:0/651099303 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f85200cebb0 0x7f852006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f850c0097a0 tx=0x7f850c02b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4a83f82f125bf14 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.932+0000 7f962ea00640 1 -- 172.21.15.102:0/1008750998 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f963015cf20 msgr2=0x7f96300ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.932+0000 7f962fe00640 1 --2- 172.21.15.102:0/1008750998 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f963015cf20 0x7f96300ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.933+0000 7f9138e00640 1 -- 172.21.15.102:0/2550132926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f913415cd90 msgr2=0x7f913406b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.933+0000 7f9138e00640 1 --2- 172.21.15.102:0/2550132926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f913415cd90 0x7f913406b060 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f91240097a0 tx=0x7f912402b8b0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.933+0000 7f9138e00640 1 -- 172.21.15.102:0/2550132926 shutdown_connections 2024-03-23T03:47:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.933+0000 7f9138e00640 1 --2- 172.21.15.102:0/2550132926 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f913415cd90 0x7f913406b060 secure :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f91240097a0 tx=0x7f912402b8b0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.933+0000 7f9138e00640 1 --2- 172.21.15.102:0/2550132926 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91340cd6b0 0x7f91340b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.933+0000 7f9138e00640 1 -- 172.21.15.102:0/2550132926 >> 172.21.15.102:0/2550132926 conn(0x7f91340cf780 msgr2=0x7f91340cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.933+0000 7f9138e00640 1 -- 172.21.15.102:0/2550132926 shutdown_connections 2024-03-23T03:47:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.933+0000 7f9138e00640 1 -- 172.21.15.102:0/2550132926 wait complete. 2024-03-23T03:47:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.933+0000 7f962ea00640 1 --2- 172.21.15.102:0/1008750998 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f963015cf20 0x7f96300ceed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.933+0000 7f962ea00640 1 -- 172.21.15.102:0/1008750998 >> v2:172.21.15.173:3300/0 conn(0x7f963015a2f0 msgr2=0x7f9630160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.933+0000 7f962ea00640 1 --2- 172.21.15.102:0/1008750998 >> v2:172.21.15.173:3300/0 conn(0x7f963015a2f0 0x7f9630160f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.933+0000 7f962ea00640 1 -- 172.21.15.102:0/1008750998 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963015d780 con 0x7f963015a6d0 2024-03-23T03:47:18.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.933+0000 7f962fe00640 1 --2- 172.21.15.102:0/1008750998 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f963015cf20 0x7f96300ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:18.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.934+0000 7f851e000640 1 -- 172.21.15.102:0/651099303 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c031520 con 0x7f85200cebb0 2024-03-23T03:47:18.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.934+0000 7f851e000640 1 -- 172.21.15.102:0/651099303 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850c031680 con 0x7f85200cebb0 2024-03-23T03:47:18.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.934+0000 7f962ea00640 1 --2- 172.21.15.102:0/1008750998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963015a6d0 0x7f96300ce990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f963006c980 tx=0x7f962002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b007207987b0515f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.935+0000 7f8526400640 1 -- 172.21.15.102:0/651099303 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f85200cebb0 msgr2=0x7f852006d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.935+0000 7f8526400640 1 --2- 172.21.15.102:0/651099303 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f85200cebb0 0x7f852006d270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f850c0097a0 tx=0x7f850c02b7c0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.935+0000 7f962e000640 1 -- 172.21.15.102:0/1008750998 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9620031870 con 0x7f963015a6d0 2024-03-23T03:47:18.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.935+0000 7f962e000640 1 -- 172.21.15.102:0/1008750998 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96200319d0 con 0x7f963015a6d0 2024-03-23T03:47:18.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.935+0000 7f962e000640 1 -- 172.21.15.102:0/1008750998 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9620031870 con 0x7f963015a6d0 2024-03-23T03:47:18.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.935+0000 7f8526400640 1 -- 172.21.15.102:0/651099303 shutdown_connections 2024-03-23T03:47:18.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.935+0000 7f8526400640 1 --2- 172.21.15.102:0/651099303 >> v2:172.21.15.102:3300/0 conn(0x7f852006d7b0 0x7f85201612c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.935+0000 7f8526400640 1 --2- 172.21.15.102:0/651099303 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f85200cebb0 0x7f852006d270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.935+0000 7f8526400640 1 -- 172.21.15.102:0/651099303 >> 172.21.15.102:0/651099303 conn(0x7f8520069c90 msgr2=0x7f852006a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.936+0000 7f9138e00640 1 Processor -- start 2024-03-23T03:47:18.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.936+0000 7f9636800640 1 -- 172.21.15.102:0/1008750998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963015a6d0 msgr2=0x7f96300ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.936+0000 7f9138e00640 1 -- start start 2024-03-23T03:47:18.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.936+0000 7f9636800640 1 --2- 172.21.15.102:0/1008750998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963015a6d0 0x7f96300ce990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f963006c980 tx=0x7f962002b910 comp rx=0 tx=0).stop 2024-03-23T03:47:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.937+0000 7f9138e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91340cd6b0 0x7f91340c4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.937+0000 7f9138e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91340cda90 0x7f91340ca3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.937+0000 7f9138e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91340ca900 0x7f9134070320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.937+0000 7f8526400640 1 -- 172.21.15.102:0/651099303 shutdown_connections 2024-03-23T03:47:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.937+0000 7f9138e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9134158810 con 0x7f91340cd6b0 2024-03-23T03:47:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.937+0000 7f9138e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91341590b0 con 0x7f91340ca900 2024-03-23T03:47:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.937+0000 7f9138e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91340c5170 con 0x7f91340cda90 2024-03-23T03:47:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.938+0000 7f9131e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91340cd6b0 0x7f91340c4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.938+0000 7f9131e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91340cd6b0 0x7f91340c4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37308/0 (socket says 172.21.15.102:37308) 2024-03-23T03:47:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.938+0000 7f9131e00640 1 -- 172.21.15.102:0/4034551163 learned_addr learned my addr 172.21.15.102:0/4034551163 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.938+0000 7f8526400640 1 -- 172.21.15.102:0/651099303 wait complete. 2024-03-23T03:47:18.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.938+0000 7f9131400640 1 --2- 172.21.15.102:0/4034551163 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91340cda90 0x7f91340ca3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.938+0000 7f9132800640 1 --2- 172.21.15.102:0/4034551163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91340ca900 0x7f9134070320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.939+0000 7f9131e00640 1 -- 172.21.15.102:0/4034551163 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91340cda90 msgr2=0x7f91340ca3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.939+0000 7f9131e00640 1 --2- 172.21.15.102:0/4034551163 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91340cda90 0x7f91340ca3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.939+0000 7f9131e00640 1 -- 172.21.15.102:0/4034551163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91340ca900 msgr2=0x7f9134070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.939+0000 7f9131e00640 1 --2- 172.21.15.102:0/4034551163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91340ca900 0x7f9134070320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.939+0000 7f9131e00640 1 -- 172.21.15.102:0/4034551163 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9124009400 con 0x7f91340cd6b0 2024-03-23T03:47:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.939+0000 7f9132800640 1 --2- 172.21.15.102:0/4034551163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91340ca900 0x7f9134070320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.939+0000 7f9131e00640 1 --2- 172.21.15.102:0/4034551163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91340cd6b0 0x7f91340c4c30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f912c00efc0 tx=0x7f912c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.940+0000 7f912a000640 1 -- 172.21.15.102:0/4034551163 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c014480 con 0x7f91340cd6b0 2024-03-23T03:47:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.940+0000 7f912a000640 1 -- 172.21.15.102:0/4034551163 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912c00f040 con 0x7f91340cd6b0 2024-03-23T03:47:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.940+0000 7f912a000640 1 -- 172.21.15.102:0/4034551163 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c0147e0 con 0x7f91340cd6b0 2024-03-23T03:47:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.940+0000 7f9636800640 1 -- 172.21.15.102:0/1008750998 shutdown_connections 2024-03-23T03:47:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.940+0000 7f9636800640 1 --2- 172.21.15.102:0/1008750998 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f963015cf20 0x7f96300ceed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.940+0000 7f9636800640 1 --2- 172.21.15.102:0/1008750998 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963015a6d0 0x7f96300ce990 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.940+0000 7f9636800640 1 --2- 172.21.15.102:0/1008750998 >> v2:172.21.15.173:3300/0 conn(0x7f963015a2f0 0x7f9630160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.940+0000 7f9636800640 1 -- 172.21.15.102:0/1008750998 >> 172.21.15.102:0/1008750998 conn(0x7f9630069c90 msgr2=0x7f963006a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.940+0000 7f9138e00640 1 -- 172.21.15.102:0/4034551163 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91340cae40 con 0x7f91340cd6b0 2024-03-23T03:47:18.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.940+0000 7f9138e00640 1 -- 172.21.15.102:0/4034551163 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9134158d40 con 0x7f91340cd6b0 2024-03-23T03:47:18.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.940+0000 7f8526400640 1 Processor -- start 2024-03-23T03:47:18.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.941+0000 7f8526400640 1 -- start start 2024-03-23T03:47:18.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.940+0000 7f9636800640 1 -- 172.21.15.102:0/1008750998 shutdown_connections 2024-03-23T03:47:18.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.940+0000 7f9636800640 1 -- 172.21.15.102:0/1008750998 wait complete. 2024-03-23T03:47:18.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.942+0000 7f9138e00640 1 -- 172.21.15.102:0/4034551163 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9134073390 con 0x7f91340cd6b0 2024-03-23T03:47:18.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.942+0000 7f9636800640 1 Processor -- start 2024-03-23T03:47:18.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.943+0000 7fd98a800640 1 Processor -- start 2024-03-23T03:47:18.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.943+0000 7f8526400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f852006d7b0 0x7f85200ac100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.943+0000 7f8526400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85200cebb0 0x7f85200a9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.943+0000 7f8526400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f852015e940 0x7f85200a95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.943+0000 7f9636800640 1 -- start start 2024-03-23T03:47:18.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.944+0000 7f8526400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85200abce0 con 0x7f852006d7b0 2024-03-23T03:47:18.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.944+0000 7f8526400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85200ab0c0 con 0x7f85200cebb0 2024-03-23T03:47:18.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.944+0000 7f851ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85200cebb0 0x7f85200a9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.944+0000 7f851ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85200cebb0 0x7f85200a9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:42348/0 (socket says 172.21.15.102:42348) 2024-03-23T03:47:18.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.944+0000 7f851ea00640 1 -- 172.21.15.102:0/2245842818 learned_addr learned my addr 172.21.15.102:0/2245842818 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.944+0000 7f851fe00640 1 --2- 172.21.15.102:0/2245842818 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f852015e940 0x7f85200a95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.944+0000 7fd98a800640 1 -- start start 2024-03-23T03:47:18.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.944+0000 7f8526400640 1 -- 172.21.15.102:0/2245842818 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85200a9c00 con 0x7f852015e940 2024-03-23T03:47:18.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.945+0000 7f9636800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f963015a2f0 0x7f9630143d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.945+0000 7f9636800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963015a6d0 0x7f96301443f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.945+0000 7f9636800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f963015cf20 0x7f9630158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.945+0000 7f9636800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300b2d80 con 0x7f963015a6d0 2024-03-23T03:47:18.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.945+0000 7f9636800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96300b3060 con 0x7f963015cf20 2024-03-23T03:47:18.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.945+0000 7f9636800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96300cbf10 con 0x7f963015a2f0 2024-03-23T03:47:18.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.944+0000 7f851f400640 1 --2- 172.21.15.102:0/2245842818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f852006d7b0 0x7f85200ac100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.944+0000 7f912a000640 1 -- 172.21.15.102:0/4034551163 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f912c014940 con 0x7f91340cd6b0 2024-03-23T03:47:18.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.945+0000 7f912a000640 1 --2- 172.21.15.102:0/4034551163 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f911406ac50 0x7f911406d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.946+0000 7f912a000640 1 -- 172.21.15.102:0/4034551163 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f912c08a1b0 con 0x7f91340cd6b0 2024-03-23T03:47:18.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.946+0000 7f9131400640 1 --2- 172.21.15.102:0/4034551163 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f911406ac50 0x7f911406d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.947+0000 7f962f400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f963015a2f0 0x7f9630143d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.947+0000 7f962f400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f963015a2f0 0x7f9630143d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:54486/0 (socket says 172.21.15.102:54486) 2024-03-23T03:47:18.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.947+0000 7f962f400640 1 -- 172.21.15.102:0/525328657 learned_addr learned my addr 172.21.15.102:0/525328657 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.947+0000 7fd98a800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9840cb480 0x7fd9840cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.947+0000 7fd98a800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9840cca00 0x7fd9840b24a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.947+0000 7fd98a800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9840cd1b0 con 0x7fd984160ee0 2024-03-23T03:47:18.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.947+0000 7fd98a800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9840cd490 con 0x7fd9840cca00 2024-03-23T03:47:18.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.947+0000 7fd98a800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9840b1ca0 con 0x7fd9840cb480 2024-03-23T03:47:18.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.946+0000 7f851ea00640 1 -- 172.21.15.102:0/2245842818 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f852015e940 msgr2=0x7f85200a95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.946+0000 7f851ea00640 1 --2- 172.21.15.102:0/2245842818 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f852015e940 0x7f85200a95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.946+0000 7f851ea00640 1 -- 172.21.15.102:0/2245842818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f852006d7b0 msgr2=0x7f85200ac100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.946+0000 7f851ea00640 1 --2- 172.21.15.102:0/2245842818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f852006d7b0 0x7f85200ac100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.946+0000 7f851ea00640 1 -- 172.21.15.102:0/2245842818 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c009400 con 0x7f85200cebb0 2024-03-23T03:47:18.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.947+0000 7f962fe00640 1 --2- 172.21.15.102:0/525328657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f963015cf20 0x7f9630158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7f962f400640 1 -- 172.21.15.102:0/525328657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f963015cf20 msgr2=0x7f9630158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7f962f400640 1 --2- 172.21.15.102:0/525328657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f963015cf20 0x7f9630158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7f9131400640 1 --2- 172.21.15.102:0/4034551163 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f911406ac50 0x7f911406d110 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9124036000 tx=0x7f9124008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.947+0000 7fd989e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9840cca00 0x7fd9840b24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7fd989e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9840cca00 0x7fd9840b24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37320/0 (socket says 172.21.15.102:37320) 2024-03-23T03:47:18.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7fd989e00640 1 -- 172.21.15.102:0/2385675202 learned_addr learned my addr 172.21.15.102:0/2385675202 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7f63aa200640 1 Processor -- start 2024-03-23T03:47:18.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7f851ea00640 1 --2- 172.21.15.102:0/2245842818 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85200cebb0 0x7f85200a9090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f850c02bdd0 tx=0x7f850c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7f962f400640 1 -- 172.21.15.102:0/525328657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963015a6d0 msgr2=0x7f96301443f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7f962f400640 1 --2- 172.21.15.102:0/525328657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963015a6d0 0x7f96301443f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7f962f400640 1 -- 172.21.15.102:0/525328657 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96200094d0 con 0x7f963015a2f0 2024-03-23T03:47:18.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7f63aa200640 1 -- start start 2024-03-23T03:47:18.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7fd989e00640 1 -- 172.21.15.102:0/2385675202 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9840cb480 msgr2=0x7fd9840cc4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T03:47:18.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7fd989e00640 1 --2- 172.21.15.102:0/2385675202 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9840cb480 0x7fd9840cc4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7fd989e00640 1 -- 172.21.15.102:0/2385675202 >> v1:172.21.15.102:6789/0 conn(0x7fd984160ee0 legacy=0x7fd9840cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7fd989e00640 1 -- 172.21.15.102:0/2385675202 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9840b1f80 con 0x7fd9840cca00 2024-03-23T03:47:18.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7f962f400640 1 --2- 172.21.15.102:0/525328657 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f963015a2f0 0x7f9630143d80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f962800efc0 tx=0x7f962800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7f8517400640 1 -- 172.21.15.102:0/2245842818 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c03d410 con 0x7f85200cebb0 2024-03-23T03:47:18.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7f8517400640 1 -- 172.21.15.102:0/2245842818 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850c03d570 con 0x7f85200cebb0 2024-03-23T03:47:18.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.948+0000 7f8517400640 1 -- 172.21.15.102:0/2245842818 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c03d900 con 0x7f85200cebb0 2024-03-23T03:47:18.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.949+0000 7f8526400640 1 -- 172.21.15.102:0/2245842818 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85200ab340 con 0x7f85200cebb0 2024-03-23T03:47:18.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.949+0000 7f5bb7c00640 1 Processor -- start 2024-03-23T03:47:18.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.949+0000 7f9627400640 1 -- 172.21.15.102:0/525328657 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9628014480 con 0x7f963015a2f0 2024-03-23T03:47:18.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.949+0000 7f9627400640 1 -- 172.21.15.102:0/525328657 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962800f040 con 0x7f963015a2f0 2024-03-23T03:47:18.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.949+0000 7f9627400640 1 -- 172.21.15.102:0/525328657 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9628014800 con 0x7f963015a2f0 2024-03-23T03:47:18.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.949+0000 7f5bb7c00640 1 -- start start 2024-03-23T03:47:18.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.949+0000 7f8526400640 1 -- 172.21.15.102:0/2245842818 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85200ab900 con 0x7f85200cebb0 2024-03-23T03:47:18.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.950+0000 7f63aa200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f63a40b4e60 0x7f63a4167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.950+0000 7f63aa200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a41607d0 con 0x7f63a415cc70 2024-03-23T03:47:18.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.950+0000 7f63aa200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63a4160a70 con 0x7f63a40b1f20 2024-03-23T03:47:18.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.950+0000 7f63aa200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a4160d50 con 0x7f63a40b4e60 2024-03-23T03:47:18.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.950+0000 7f9636800640 1 -- 172.21.15.102:0/525328657 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96300cc1f0 con 0x7f963015a2f0 2024-03-23T03:47:18.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.950+0000 7f9636800640 1 -- 172.21.15.102:0/525328657 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9630158a50 con 0x7f963015a2f0 2024-03-23T03:47:18.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.951+0000 7f63a3e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f63a40b4e60 0x7f63a4167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.951+0000 7f63a3e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f63a40b4e60 0x7f63a4167b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:54488/0 (socket says 172.21.15.102:54488) 2024-03-23T03:47:18.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.951+0000 7f63a3e00640 1 -- 172.21.15.102:0/2194710285 learned_addr learned my addr 172.21.15.102:0/2194710285 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.951+0000 7fd989e00640 1 --2- 172.21.15.102:0/2385675202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9840cca00 0x7fd9840b24a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd970009870 tx=0x7fd97002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75687a23e6b57b15 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.951+0000 7f8526400640 1 -- 172.21.15.102:0/2245842818 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f8005250 con 0x7f85200cebb0 2024-03-23T03:47:18.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.952+0000 7f5bb7c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f5bb0165340 0x7f5bb00ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.951+0000 7fb34f200640 1 Processor -- start 2024-03-23T03:47:18.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.951+0000 7fb34f200640 1 -- start start 2024-03-23T03:47:18.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.952+0000 7f5bb7c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5bb01627c0 0x7f5bb006d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.952+0000 7f63a2000640 1 -- 172.21.15.102:0/2194710285 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139662226 0 0) 0x7f63a4160a70 con 0x7f63a40b1f20 2024-03-23T03:47:18.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.952+0000 7f63a2000640 1 -- 172.21.15.102:0/2194710285 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6388003520 con 0x7f63a40b1f20 2024-03-23T03:47:18.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.952+0000 7f9636800640 1 -- 172.21.15.102:0/525328657 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96300b3280 con 0x7f963015a2f0 2024-03-23T03:47:18.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.952+0000 7f5bb7c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb01657c0 con 0x7f5bb00ce930 2024-03-23T03:47:18.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.952+0000 7f5bb7c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb006cdc0 con 0x7f5bb01627c0 2024-03-23T03:47:18.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.952+0000 7f5bb7c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb00cf250 con 0x7f5bb0165340 2024-03-23T03:47:18.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.952+0000 7f63a3e00640 1 -- 172.21.15.102:0/2194710285 >> v1:172.21.15.173:6789/0 conn(0x7f63a40b1f20 legacy=0x7f63a40b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.952+0000 7f63a3e00640 1 -- 172.21.15.102:0/2194710285 >> v1:172.21.15.102:6789/0 conn(0x7f63a415cc70 legacy=0x7f63a40cdba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.952+0000 7f63a3e00640 1 -- 172.21.15.102:0/2194710285 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a4161030 con 0x7f63a40b4e60 2024-03-23T03:47:18.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.953+0000 7fb34f200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb3500707e0 0x7fb35015c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.953+0000 7fb34f200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb35006dfc0 0x7fb3500b1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.953+0000 7fb34f200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb35015d260 con 0x7fb3500adac0 2024-03-23T03:47:18.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.953+0000 7fb34f200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3500adea0 con 0x7fb35006dfc0 2024-03-23T03:47:18.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.953+0000 7fb34f200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb35006e3f0 con 0x7fb3500707e0 2024-03-23T03:47:18.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.953+0000 7f5bb5c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5bb01627c0 0x7f5bb006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.953+0000 7f5bafe00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f5bb0165340 0x7f5bb00ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.953+0000 7f5bb5200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5bb00ce930 0x7f5bb01612d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41212/0 (socket says 172.21.15.102:41212) 2024-03-23T03:47:18.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.953+0000 7f5bb5200640 1 -- 172.21.15.102:0/2226672440 learned_addr learned my addr 172.21.15.102:0/2226672440 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:18.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.954+0000 7fb34e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb35006dfc0 0x7fb3500b1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.954+0000 7f8517400640 1 -- 172.21.15.102:0/2245842818 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f850c03a070 con 0x7f85200cebb0 2024-03-23T03:47:18.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.954+0000 7f63a3e00640 1 --2- 172.21.15.102:0/2194710285 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f63a40b4e60 0x7f63a4167b00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f639c009e80 tx=0x7f639c02bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=316a1d8a15cadde0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.954+0000 7f5bafe00640 1 -- 172.21.15.102:0/2226672440 >> v2:172.21.15.102:3300/0 conn(0x7f5bb01627c0 msgr2=0x7f5bb006d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.954+0000 7f5bafe00640 1 --2- 172.21.15.102:0/2226672440 >> v2:172.21.15.102:3300/0 conn(0x7f5bb01627c0 0x7f5bb006d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.954+0000 7f5bafe00640 1 -- 172.21.15.102:0/2226672440 >> v1:172.21.15.102:6789/0 conn(0x7f5bb00ce930 legacy=0x7f5bb01612d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.954+0000 7f5bafe00640 1 -- 172.21.15.102:0/2226672440 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb0161ab0 con 0x7f5bb0165340 2024-03-23T03:47:18.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.954+0000 7f5bb5c00640 1 --2- 172.21.15.102:0/2226672440 >> v2:172.21.15.102:3300/0 conn(0x7f5bb01627c0 0x7f5bb006d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:47:18.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.955+0000 7f9627400640 1 -- 172.21.15.102:0/525328657 <== mon.2 v2:172.21.15.173:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9628014960 con 0x7f963015a2f0 2024-03-23T03:47:18.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.954+0000 7fb34e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb35006dfc0 0x7fb3500b1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37344/0 (socket says 172.21.15.102:37344) 2024-03-23T03:47:18.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.954+0000 7fb34e800640 1 -- 172.21.15.102:0/2526249476 learned_addr learned my addr 172.21.15.102:0/2526249476 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.955+0000 7fb34d400640 1 --2- 172.21.15.102:0/2526249476 >> v2:172.21.15.173:3300/0 conn(0x7fb3500707e0 0x7fb35015c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.955+0000 7fb34e800640 1 -- 172.21.15.102:0/2526249476 >> v2:172.21.15.173:3300/0 conn(0x7fb3500707e0 msgr2=0x7fb35015c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.955+0000 7fb34e800640 1 --2- 172.21.15.102:0/2526249476 >> v2:172.21.15.173:3300/0 conn(0x7fb3500707e0 0x7fb35015c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.955+0000 7fb34e800640 1 -- 172.21.15.102:0/2526249476 >> v1:172.21.15.173:6789/0 conn(0x7fb3500adac0 legacy=0x7fb3500700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.955+0000 7fb34e800640 1 -- 172.21.15.102:0/2526249476 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb35015c9c0 con 0x7fb35006dfc0 2024-03-23T03:47:18.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.955+0000 7fb34d400640 1 --2- 172.21.15.102:0/2526249476 >> v2:172.21.15.173:3300/0 conn(0x7fb3500707e0 0x7fb35015c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:18.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.955+0000 7f8517400640 1 --2- 172.21.15.102:0/2245842818 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f850006ac50 0x7f850006d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.955+0000 7f8517400640 1 -- 172.21.15.102:0/2245842818 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f850c0ab520 con 0x7f85200cebb0 2024-03-23T03:47:18.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.956+0000 7f851f400640 1 --2- 172.21.15.102:0/2245842818 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f850006ac50 0x7f850006d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.956+0000 7f9627400640 1 --2- 172.21.15.102:0/525328657 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f961006ac50 0x7f961006d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.956+0000 7f5bafe00640 1 --2- 172.21.15.102:0/2226672440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb0165340 0x7f5bb00ced10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5bb006ff50 tx=0x7f5ba802bec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26516be13dbdfca4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.956+0000 7f9627400640 1 -- 172.21.15.102:0/525328657 <== mon.2 v2:172.21.15.173:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f962808a8a0 con 0x7f963015a2f0 2024-03-23T03:47:18.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.956+0000 7fd983e00640 1 -- 172.21.15.102:0/2385675202 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970031870 con 0x7fd9840cca00 2024-03-23T03:47:18.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.956+0000 7fd983e00640 1 -- 172.21.15.102:0/2385675202 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9700319d0 con 0x7fd9840cca00 2024-03-23T03:47:18.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.956+0000 7fd983e00640 1 -- 172.21.15.102:0/2385675202 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970031d60 con 0x7fd9840cca00 2024-03-23T03:47:18.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.957+0000 7f851f400640 1 --2- 172.21.15.102:0/2245842818 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f850006ac50 0x7f850006d110 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8518005c00 tx=0x7f8518009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.957+0000 7fb34e800640 1 --2- 172.21.15.102:0/2526249476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb35006dfc0 0x7fb3500b1ba0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb3500714d0 tx=0x7fb3480346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65dd58e9367c9f15 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.957+0000 7f912a000640 1 -- 172.21.15.102:0/4034551163 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f912c055210 con 0x7f91340cd6b0 2024-03-23T03:47:18.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.957+0000 7fc4cb400640 1 Processor -- start 2024-03-23T03:47:18.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.957+0000 7f962ea00640 1 --2- 172.21.15.102:0/525328657 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f961006ac50 0x7f961006d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.957+0000 7fb34ca00640 1 -- 172.21.15.102:0/2526249476 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34803b600 con 0x7fb35006dfc0 2024-03-23T03:47:18.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.957+0000 7fb34ca00640 1 -- 172.21.15.102:0/2526249476 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb348037050 con 0x7fb35006dfc0 2024-03-23T03:47:18.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.957+0000 7fb34ca00640 1 -- 172.21.15.102:0/2526249476 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34803b980 con 0x7fb35006dfc0 2024-03-23T03:47:18.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.957+0000 7fd98a800640 1 -- 172.21.15.102:0/2385675202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9840cca00 msgr2=0x7fd9840b24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.957+0000 7fd98a800640 1 --2- 172.21.15.102:0/2385675202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9840cca00 0x7fd9840b24a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fd970009870 tx=0x7fd97002b910 comp rx=0 tx=0).stop 2024-03-23T03:47:18.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.957+0000 7fc4cb400640 1 -- start start 2024-03-23T03:47:18.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.958+0000 7fd98a800640 1 -- 172.21.15.102:0/2385675202 shutdown_connections 2024-03-23T03:47:18.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.958+0000 7fd98a800640 1 --2- 172.21.15.102:0/2385675202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9840cca00 0x7fd9840b24a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.958+0000 7fd98a800640 1 --2- 172.21.15.102:0/2385675202 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9840cb480 0x7fd9840cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.958+0000 7fd98a800640 1 -- 172.21.15.102:0/2385675202 >> 172.21.15.102:0/2385675202 conn(0x7fd984065e20 msgr2=0x7fd984069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.958+0000 7fd98a800640 1 -- 172.21.15.102:0/2385675202 shutdown_connections 2024-03-23T03:47:18.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.958+0000 7f962ea00640 1 --2- 172.21.15.102:0/525328657 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f961006ac50 0x7f961006d110 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9620036000 tx=0x7f9620030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.958+0000 7fb34f200640 1 -- 172.21.15.102:0/2526249476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb35006dfc0 msgr2=0x7fb3500b1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.958+0000 7fb34f200640 1 --2- 172.21.15.102:0/2526249476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb35006dfc0 0x7fb3500b1ba0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb3500714d0 tx=0x7fb3480346f0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.958+0000 7fb34f200640 1 -- 172.21.15.102:0/2526249476 shutdown_connections 2024-03-23T03:47:18.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.958+0000 7fb34f200640 1 --2- 172.21.15.102:0/2526249476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb35006dfc0 0x7fb3500b1ba0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.958+0000 7fb34f200640 1 --2- 172.21.15.102:0/2526249476 >> v2:172.21.15.173:3300/0 conn(0x7fb3500707e0 0x7fb35015c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.958+0000 7fb34f200640 1 -- 172.21.15.102:0/2526249476 >> 172.21.15.102:0/2526249476 conn(0x7fb3500b0250 msgr2=0x7fb3500b0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.958+0000 7fd98a800640 1 -- 172.21.15.102:0/2385675202 wait complete. 2024-03-23T03:47:18.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.958+0000 7fb34f200640 1 -- 172.21.15.102:0/2526249476 shutdown_connections 2024-03-23T03:47:18.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.958+0000 7fb34f200640 1 -- 172.21.15.102:0/2526249476 wait complete. 2024-03-23T03:47:18.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.959+0000 7fc4cb400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc4c40cd6b0 0x7fc4c40b8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.959+0000 7fc4cb400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc4c415cd90 0x7fc4c406b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.959+0000 7fc4cb400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc4c40cda90 0x7fc4c41606c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.959+0000 7fc4cb400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4c406c0e0 con 0x7fc4c415cd90 2024-03-23T03:47:18.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.959+0000 7fc4cb400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4c406b5a0 con 0x7fc4c40cd6b0 2024-03-23T03:47:18.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.959+0000 7fc4cb400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c406b840 con 0x7fc4c40cda90 2024-03-23T03:47:18.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.960+0000 7fc4c8a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc4c40cd6b0 0x7fc4c40b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.960+0000 7fc4c8a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc4c40cd6b0 0x7fc4c40b8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:42390/0 (socket says 172.21.15.102:42390) 2024-03-23T03:47:18.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.960+0000 7fc4c8a00640 1 -- 172.21.15.102:0/2265721833 learned_addr learned my addr 172.21.15.102:0/2265721833 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.960+0000 7fc4c3e00640 1 --2- 172.21.15.102:0/2265721833 >> v2:172.21.15.102:3300/0 conn(0x7fc4c415cd90 0x7fc4c406b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.960+0000 7fd98a800640 1 Processor -- start 2024-03-23T03:47:18.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.960+0000 7fd98a800640 1 -- start start 2024-03-23T03:47:18.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.960+0000 7fc4c8a00640 1 -- 172.21.15.102:0/2265721833 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc4c40cda90 msgr2=0x7fc4c41606c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.960+0000 7fc4c8a00640 1 --2- 172.21.15.102:0/2265721833 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc4c40cda90 0x7fc4c41606c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.960+0000 7fc4c8a00640 1 -- 172.21.15.102:0/2265721833 >> v2:172.21.15.102:3300/0 conn(0x7fc4c415cd90 msgr2=0x7fc4c406b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:18.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.961+0000 7fc4c8a00640 1 --2- 172.21.15.102:0/2265721833 >> v2:172.21.15.102:3300/0 conn(0x7fc4c415cd90 0x7fc4c406b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.961+0000 7fc4c8a00640 1 -- 172.21.15.102:0/2265721833 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c406bb20 con 0x7fc4c40cd6b0 2024-03-23T03:47:18.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.961+0000 7fc4c3e00640 1 --2- 172.21.15.102:0/2265721833 >> v2:172.21.15.102:3300/0 conn(0x7fc4c415cd90 0x7fc4c406b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:47:18.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.961+0000 7fb34f200640 1 Processor -- start 2024-03-23T03:47:18.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.961+0000 7fb34f200640 1 -- start start 2024-03-23T03:47:18.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.961+0000 7fd98a800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9840cb480 0x7fd9841f90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.961+0000 7fd98a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9840cca00 0x7fd98414f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.961+0000 7fd98a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd984160ee0 0x7fd9840ae000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.961+0000 7fd98a800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd984162f40 con 0x7fd984160ee0 2024-03-23T03:47:18.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.961+0000 7fd98a800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd984150020 con 0x7fd9840cca00 2024-03-23T03:47:18.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.961+0000 7fd98a800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd98414fd40 con 0x7fd9840cb480 2024-03-23T03:47:18.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.962+0000 7fd989e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd984160ee0 0x7fd9840ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.962+0000 7fd989e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd984160ee0 0x7fd9840ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37362/0 (socket says 172.21.15.102:37362) 2024-03-23T03:47:18.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.962+0000 7fd989e00640 1 -- 172.21.15.102:0/2466815195 learned_addr learned my addr 172.21.15.102:0/2466815195 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.962+0000 7fd988a00640 1 --2- 172.21.15.102:0/2466815195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9840cca00 0x7fd98414f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.962+0000 7fc4c8a00640 1 --2- 172.21.15.102:0/2265721833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4c40cd6b0 0x7fc4c40b8490 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40b46b0 tx=0x7fc4b402bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bf6a277ca6fc39e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.962+0000 7fd989e00640 1 -- 172.21.15.102:0/2466815195 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9840cb480 msgr2=0x7fd9841f90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.962+0000 7fd989e00640 1 --2- 172.21.15.102:0/2466815195 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9840cb480 0x7fd9841f90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.962+0000 7fd989e00640 1 -- 172.21.15.102:0/2466815195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9840cca00 msgr2=0x7fd98414f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.962+0000 7fd989e00640 1 --2- 172.21.15.102:0/2466815195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9840cca00 0x7fd98414f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.962+0000 7fd989e00640 1 -- 172.21.15.102:0/2466815195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9700094d0 con 0x7fd984160ee0 2024-03-23T03:47:18.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.963+0000 7fd989e00640 1 --2- 172.21.15.102:0/2466815195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd984160ee0 0x7fd9840ae000 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd97002b950 tx=0x7fd9700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.962+0000 7fb34f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb35006dfc0 0x7fb350156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.963+0000 7fb34f200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3500707e0 0x7fb3501ee220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.963+0000 7fb34f200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3500adac0 0x7fb3501ee760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.963+0000 7fb34f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb350158a30 con 0x7fb35006dfc0 2024-03-23T03:47:18.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.963+0000 7fb34f200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb350157490 con 0x7fb3500adac0 2024-03-23T03:47:18.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.963+0000 7fb34f200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb350157770 con 0x7fb3500707e0 2024-03-23T03:47:18.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.963+0000 7fd981600640 1 -- 172.21.15.102:0/2466815195 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9700443f0 con 0x7fd984160ee0 2024-03-23T03:47:18.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.963+0000 7fd981600640 1 -- 172.21.15.102:0/2466815195 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd970044570 con 0x7fd984160ee0 2024-03-23T03:47:18.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.963+0000 7fd981600640 1 -- 172.21.15.102:0/2466815195 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd970044920 con 0x7fd984160ee0 2024-03-23T03:47:18.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.963+0000 7fd98a800640 1 -- 172.21.15.102:0/2466815195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9841628f0 con 0x7fd984160ee0 2024-03-23T03:47:18.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.963+0000 7fb34de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb35006dfc0 0x7fb350156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.963+0000 7fb34d400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3500707e0 0x7fb3501ee220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.963+0000 7fd98a800640 1 -- 172.21.15.102:0/2466815195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9840a7650 con 0x7fd984160ee0 2024-03-23T03:47:18.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.963+0000 7fb34de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb35006dfc0 0x7fb350156bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37364/0 (socket says 172.21.15.102:37364) 2024-03-23T03:47:18.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.963+0000 7fb34de00640 1 -- 172.21.15.102:0/1984133221 learned_addr learned my addr 172.21.15.102:0/1984133221 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:18.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.963+0000 7fb34e800640 1 --2- 172.21.15.102:0/1984133221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3500adac0 0x7fb3501ee760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.964+0000 7fb34de00640 1 -- 172.21.15.102:0/1984133221 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3500707e0 msgr2=0x7fb3501ee220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.964+0000 7fb34de00640 1 --2- 172.21.15.102:0/1984133221 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3500707e0 0x7fb3501ee220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.964+0000 7fb34de00640 1 -- 172.21.15.102:0/1984133221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3500adac0 msgr2=0x7fb3501ee760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.964+0000 7fb34de00640 1 --2- 172.21.15.102:0/1984133221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3500adac0 0x7fb3501ee760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.964+0000 7fb34de00640 1 -- 172.21.15.102:0/1984133221 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb348009b30 con 0x7fb35006dfc0 2024-03-23T03:47:18.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.964+0000 7fb34d400640 1 --2- 172.21.15.102:0/1984133221 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3500707e0 0x7fb3501ee220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:47:18.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.964+0000 7fd98a800640 1 -- 172.21.15.102:0/2466815195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd98404e030 con 0x7fd984160ee0 2024-03-23T03:47:18.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.965+0000 7fb34de00640 1 --2- 172.21.15.102:0/1984133221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb35006dfc0 0x7fb350156bd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb34000c450 tx=0x7fb34000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.965+0000 7fb346000640 1 -- 172.21.15.102:0/1984133221 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3400052d0 con 0x7fb35006dfc0 2024-03-23T03:47:18.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.965+0000 7fb346000640 1 -- 172.21.15.102:0/1984133221 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb340005430 con 0x7fb35006dfc0 2024-03-23T03:47:18.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.965+0000 7fb346000640 1 -- 172.21.15.102:0/1984133221 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb340012870 con 0x7fb35006dfc0 2024-03-23T03:47:18.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.965+0000 7fd981600640 1 -- 172.21.15.102:0/2466815195 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd97004d450 con 0x7fd984160ee0 2024-03-23T03:47:18.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.966+0000 7fb34f200640 1 -- 172.21.15.102:0/1984133221 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb350158c80 con 0x7fb35006dfc0 2024-03-23T03:47:18.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.966+0000 7fb34f200640 1 -- 172.21.15.102:0/1984133221 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb350152aa0 con 0x7fb35006dfc0 2024-03-23T03:47:18.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.966+0000 7fd981600640 1 --2- 172.21.15.102:0/2466815195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd96406ac50 0x7fd96406d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.967+0000 7fd981600640 1 -- 172.21.15.102:0/2466815195 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd97002d040 con 0x7fd984160ee0 2024-03-23T03:47:18.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.967+0000 7fd989400640 1 --2- 172.21.15.102:0/2466815195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd96406ac50 0x7fd96406d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.967+0000 7fb34f200640 1 -- 172.21.15.102:0/1984133221 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3501591a0 con 0x7fb35006dfc0 2024-03-23T03:47:18.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.968+0000 7fd989400640 1 --2- 172.21.15.102:0/2466815195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd96406ac50 0x7fd96406d110 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00abb0 tx=0x7fd97c009250 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.971+0000 7fc4c3400640 1 -- 172.21.15.102:0/2265721833 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b402ca80 con 0x7fc4c40cd6b0 2024-03-23T03:47:18.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.972+0000 7f5baf400640 1 -- 172.21.15.102:0/2226672440 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba802cc70 con 0x7f5bb0165340 2024-03-23T03:47:18.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.972+0000 7fc4c3400640 1 -- 172.21.15.102:0/2265721833 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b402cbe0 con 0x7fc4c40cd6b0 2024-03-23T03:47:18.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.972+0000 7f5baf400640 1 -- 172.21.15.102:0/2226672440 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba802cdd0 con 0x7f5bb0165340 2024-03-23T03:47:18.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.972+0000 7f5baf400640 1 -- 172.21.15.102:0/2226672440 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba8034660 con 0x7f5bb0165340 2024-03-23T03:47:18.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.972+0000 7fd981600640 1 -- 172.21.15.102:0/2466815195 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd970076d20 con 0x7fd984160ee0 2024-03-23T03:47:18.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.972+0000 7fc4c3400640 1 -- 172.21.15.102:0/2265721833 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b40343f0 con 0x7fc4c40cd6b0 2024-03-23T03:47:18.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.972+0000 7f63a2000640 1 -- 172.21.15.102:0/2194710285 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c02cb90 con 0x7f63a40b4e60 2024-03-23T03:47:18.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.972+0000 7f63a2000640 1 -- 172.21.15.102:0/2194710285 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c02ccf0 con 0x7f63a40b4e60 2024-03-23T03:47:18.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.972+0000 7f63a2000640 1 -- 172.21.15.102:0/2194710285 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c02cea0 con 0x7f63a40b4e60 2024-03-23T03:47:18.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.973+0000 7f5bb7c00640 1 -- 172.21.15.102:0/2226672440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb0165340 msgr2=0x7f5bb00ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.973+0000 7f5bb7c00640 1 --2- 172.21.15.102:0/2226672440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb0165340 0x7f5bb00ced10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5bb006ff50 tx=0x7f5ba802bec0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.973+0000 7f9627400640 1 -- 172.21.15.102:0/525328657 <== mon.2 v2:172.21.15.173:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9628055800 con 0x7f963015a2f0 2024-03-23T03:47:18.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.973+0000 7f5bb7c00640 1 -- 172.21.15.102:0/2226672440 shutdown_connections 2024-03-23T03:47:18.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.973+0000 7f5bb7c00640 1 --2- 172.21.15.102:0/2226672440 >> v2:172.21.15.102:3300/0 conn(0x7f5bb01627c0 0x7f5bb006d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.973+0000 7f5bb7c00640 1 --2- 172.21.15.102:0/2226672440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb0165340 0x7f5bb00ced10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.973+0000 7f63aa200640 1 -- 172.21.15.102:0/2194710285 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f63a40b4e60 msgr2=0x7f63a4167b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.973+0000 7f63aa200640 1 --2- 172.21.15.102:0/2194710285 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f63a40b4e60 0x7f63a4167b00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f639c009e80 tx=0x7f639c02bff0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.973+0000 7f63aa200640 1 -- 172.21.15.102:0/2194710285 shutdown_connections 2024-03-23T03:47:18.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.973+0000 7fc4cb400640 1 -- 172.21.15.102:0/2265721833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4c40cd6b0 msgr2=0x7fc4c40b8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:18.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.974+0000 7fc4cb400640 1 --2- 172.21.15.102:0/2265721833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4c40cd6b0 0x7fc4c40b8490 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40b46b0 tx=0x7fc4b402bee0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.974+0000 7f63aa200640 1 --2- 172.21.15.102:0/2194710285 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f63a40b4e60 0x7f63a4167b00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.974+0000 7f63aa200640 1 -- 172.21.15.102:0/2194710285 >> 172.21.15.102:0/2194710285 conn(0x7f63a40cf790 msgr2=0x7f63a40cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.974+0000 7f63aa200640 1 -- 172.21.15.102:0/2194710285 shutdown_connections 2024-03-23T03:47:18.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.973+0000 7f5bb7c00640 1 -- 172.21.15.102:0/2226672440 >> 172.21.15.102:0/2226672440 conn(0x7f5bb006a0b0 msgr2=0x7f5bb006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.974+0000 7f63aa200640 1 -- 172.21.15.102:0/2194710285 wait complete. 2024-03-23T03:47:18.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.974+0000 7fb346000640 1 -- 172.21.15.102:0/1984133221 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb3400129d0 con 0x7fb35006dfc0 2024-03-23T03:47:18.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.974+0000 7f8517400640 1 -- 172.21.15.102:0/2245842818 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f850c076750 con 0x7f85200cebb0 2024-03-23T03:47:18.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.975+0000 7f5bb7c00640 1 -- 172.21.15.102:0/2226672440 shutdown_connections 2024-03-23T03:47:18.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.977+0000 7f63aa200640 1 Processor -- start 2024-03-23T03:47:18.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.978+0000 7f63aa200640 1 -- start start 2024-03-23T03:47:18.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.975+0000 7f5bb7c00640 1 -- 172.21.15.102:0/2226672440 wait complete. 2024-03-23T03:47:18.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.978+0000 7f5bb7c00640 1 Processor -- start 2024-03-23T03:47:18.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.978+0000 7f5bb7c00640 1 -- start start 2024-03-23T03:47:18.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.976+0000 7fc4cb400640 1 -- 172.21.15.102:0/2265721833 shutdown_connections 2024-03-23T03:47:18.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.976+0000 7fc4cb400640 1 --2- 172.21.15.102:0/2265721833 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc4c40cda90 0x7fc4c41606c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.976+0000 7fc4cb400640 1 --2- 172.21.15.102:0/2265721833 >> v2:172.21.15.102:3300/0 conn(0x7fc4c415cd90 0x7fc4c406b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.976+0000 7fc4cb400640 1 --2- 172.21.15.102:0/2265721833 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4c40cd6b0 0x7fc4c40b8490 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:18.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.976+0000 7fc4cb400640 1 -- 172.21.15.102:0/2265721833 >> 172.21.15.102:0/2265721833 conn(0x7fc4c40cf780 msgr2=0x7fc4c40cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:18.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.976+0000 7fb346000640 1 --2- 172.21.15.102:0/1984133221 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb33006ac00 0x7fb33006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.976+0000 7fb346000640 1 -- 172.21.15.102:0/1984133221 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb340089cd0 con 0x7fb35006dfc0 2024-03-23T03:47:18.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.977+0000 7fb34d400640 1 --2- 172.21.15.102:0/1984133221 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb33006ac00 0x7fb33006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.978+0000 7fb34d400640 1 --2- 172.21.15.102:0/1984133221 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb33006ac00 0x7fb33006d0c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb350151710 tx=0x7fb33c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:18.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.979+0000 7fc4cb400640 1 -- 172.21.15.102:0/2265721833 shutdown_connections 2024-03-23T03:47:18.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.979+0000 7fc4cb400640 1 -- 172.21.15.102:0/2265721833 wait complete. 2024-03-23T03:47:18.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.980+0000 7f63aa200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f63a40b1f20 0x7f63a41533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.980+0000 7f63aa200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63a40b4e60 0x7f63a41f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.980+0000 7f63aa200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63a415cc70 0x7f63a40c5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.980+0000 7f5bb7c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb00ce930 0x7f5bb01f5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.980+0000 7f63aa200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a40b6b30 con 0x7f63a415cc70 2024-03-23T03:47:18.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.980+0000 7f63aa200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63a41f6280 con 0x7f63a40b4e60 2024-03-23T03:47:18.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.980+0000 7f63aa200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63a414d5e0 con 0x7f63a40b1f20 2024-03-23T03:47:18.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.980+0000 7f5bb7c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bb01627c0 0x7f5bb0157980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.980+0000 7f63a2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63a40b4e60 0x7f63a41f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.981+0000 7f5bb5200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb00ce930 0x7f5bb01f5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.981+0000 7f63a3400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f63a40b1f20 0x7f63a41533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:18.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.981+0000 7f5bb7c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5bb0165340 0x7f5bb015b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:18.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.981+0000 7f5bb7c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb015e020 con 0x7f5bb01627c0 2024-03-23T03:47:18.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.981+0000 7f5bb7c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb015f3e0 con 0x7f5bb00ce930 2024-03-23T03:47:19.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.981+0000 7f5bb7c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb015c910 con 0x7f5bb0165340 2024-03-23T03:47:19.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.981+0000 7f63a2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63a40b4e60 0x7f63a41f5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:42426/0 (socket says 172.21.15.102:42426) 2024-03-23T03:47:19.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.981+0000 7f63a2a00640 1 -- 172.21.15.102:0/3148479416 learned_addr learned my addr 172.21.15.102:0/3148479416 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:19.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.981+0000 7f5bb5200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb00ce930 0x7f5bb01f5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:42442/0 (socket says 172.21.15.102:42442) 2024-03-23T03:47:19.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.981+0000 7f5bb5200640 1 -- 172.21.15.102:0/246790882 learned_addr learned my addr 172.21.15.102:0/246790882 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:19.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.981+0000 7f5bb5c00640 1 --2- 172.21.15.102:0/246790882 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5bb0165340 0x7f5bb015b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:19.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.982+0000 7f5bafe00640 1 --2- 172.21.15.102:0/246790882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bb01627c0 0x7f5bb0157980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:19.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.982+0000 7f5bb5c00640 1 -- 172.21.15.102:0/246790882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb00ce930 msgr2=0x7f5bb01f5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.982+0000 7f5bb5c00640 1 --2- 172.21.15.102:0/246790882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb00ce930 0x7f5bb01f5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.982+0000 7f5bb5c00640 1 -- 172.21.15.102:0/246790882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bb01627c0 msgr2=0x7f5bb0157980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.982+0000 7f5bb5c00640 1 --2- 172.21.15.102:0/246790882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bb01627c0 0x7f5bb0157980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.982+0000 7f5bb5c00640 1 -- 172.21.15.102:0/246790882 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba8009a70 con 0x7f5bb0165340 2024-03-23T03:47:19.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.982+0000 7f63a3e00640 1 --2- 172.21.15.102:0/3148479416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63a415cc70 0x7f63a40c5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:19.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.983+0000 7f5bb5c00640 1 --2- 172.21.15.102:0/246790882 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5bb0165340 0x7f5bb015b890 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000e560 tx=0x7f5ba000ea20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:19.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.983+0000 7fb346000640 1 -- 172.21.15.102:0/1984133221 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb340054ba0 con 0x7fb35006dfc0 2024-03-23T03:47:19.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.983+0000 7f63a3e00640 1 -- 172.21.15.102:0/3148479416 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f63a40b1f20 msgr2=0x7f63a41533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.983+0000 7f63a3e00640 1 --2- 172.21.15.102:0/3148479416 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f63a40b1f20 0x7f63a41533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.983+0000 7f63a3e00640 1 -- 172.21.15.102:0/3148479416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63a40b4e60 msgr2=0x7f63a41f5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.983+0000 7f63a3e00640 1 --2- 172.21.15.102:0/3148479416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63a40b4e60 0x7f63a41f5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.983+0000 7f63a3e00640 1 -- 172.21.15.102:0/3148479416 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639c009b30 con 0x7f63a415cc70 2024-03-23T03:47:19.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.983+0000 7f5bacc00640 1 -- 172.21.15.102:0/246790882 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0013610 con 0x7f5bb0165340 2024-03-23T03:47:19.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.983+0000 7f5bacc00640 1 -- 172.21.15.102:0/246790882 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba0013770 con 0x7f5bb0165340 2024-03-23T03:47:19.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.983+0000 7f5bacc00640 1 -- 172.21.15.102:0/246790882 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba0013ae0 con 0x7f5bb0165340 2024-03-23T03:47:19.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.982+0000 7fc4cb400640 1 Processor -- start 2024-03-23T03:47:19.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.984+0000 7fc4cb400640 1 -- start start 2024-03-23T03:47:19.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.985+0000 7f5bb7c00640 1 -- 172.21.15.102:0/246790882 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb015c350 con 0x7f5bb0165340 2024-03-23T03:47:19.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.985+0000 7f5bb7c00640 1 -- 172.21.15.102:0/246790882 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb015f8f0 con 0x7f5bb0165340 2024-03-23T03:47:19.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.985+0000 7f63a3e00640 1 --2- 172.21.15.102:0/3148479416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63a415cc70 0x7f63a40c5370 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f639c032000 tx=0x7f639c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:19.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.986+0000 7f639b400640 1 -- 172.21.15.102:0/3148479416 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c03e450 con 0x7f63a415cc70 2024-03-23T03:47:19.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.986+0000 7f639b400640 1 -- 172.21.15.102:0/3148479416 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f639c03e5d0 con 0x7f63a415cc70 2024-03-23T03:47:19.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.986+0000 7f639b400640 1 -- 172.21.15.102:0/3148479416 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f639c03e980 con 0x7f63a415cc70 2024-03-23T03:47:19.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.986+0000 7f63aa200640 1 -- 172.21.15.102:0/3148479416 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63a40b5fb0 con 0x7f63a415cc70 2024-03-23T03:47:19.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.986+0000 7f63aa200640 1 -- 172.21.15.102:0/3148479416 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63a40b6570 con 0x7f63a415cc70 2024-03-23T03:47:19.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.986+0000 7fc4cb400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc4c40cd6b0 0x7fc4c4154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:19.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.987+0000 7fc4cb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4c40cda90 0x7fc4c415f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:19.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.987+0000 7fc4cb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4c415cd90 0x7fc4c406f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:19.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.987+0000 7fc4cb400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c41f4fb0 con 0x7fc4c415cd90 2024-03-23T03:47:19.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.987+0000 7fc4cb400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c41f5220 con 0x7fc4c40cda90 2024-03-23T03:47:19.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.987+0000 7fc4cb400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c41f5500 con 0x7fc4c40cd6b0 2024-03-23T03:47:19.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.987+0000 7fc4c3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4c40cda90 0x7fc4c415f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:19.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.988+0000 7fc4c3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4c40cda90 0x7fc4c415f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:42454/0 (socket says 172.21.15.102:42454) 2024-03-23T03:47:19.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.988+0000 7fc4c3e00640 1 -- 172.21.15.102:0/776251565 learned_addr learned my addr 172.21.15.102:0/776251565 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:19.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.988+0000 7fc4c8a00640 1 --2- 172.21.15.102:0/776251565 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc4c40cd6b0 0x7fc4c4154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:19.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.987+0000 7f5bb7c00640 1 -- 172.21.15.102:0/246790882 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb0073390 con 0x7f5bb0165340 2024-03-23T03:47:19.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.989+0000 7f5bacc00640 1 -- 172.21.15.102:0/246790882 <== mon.2 v2:172.21.15.173:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5ba0013c40 con 0x7f5bb0165340 2024-03-23T03:47:19.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.990+0000 7f5bacc00640 1 --2- 172.21.15.102:0/246790882 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b9006ad00 0x7f5b9006d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:19.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.990+0000 7f5bacc00640 1 -- 172.21.15.102:0/246790882 <== mon.2 v2:172.21.15.173:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5ba0089280 con 0x7f5bb0165340 2024-03-23T03:47:19.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.989+0000 7fc4c3e00640 1 -- 172.21.15.102:0/776251565 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc4c40cd6b0 msgr2=0x7fc4c4154440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.989+0000 7fc4c3e00640 1 --2- 172.21.15.102:0/776251565 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc4c40cd6b0 0x7fc4c4154440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.989+0000 7fc4c3e00640 1 -- 172.21.15.102:0/776251565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4c415cd90 msgr2=0x7fc4c406f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.989+0000 7fc4c3e00640 1 --2- 172.21.15.102:0/776251565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4c415cd90 0x7fc4c406f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.989+0000 7fc4c3e00640 1 -- 172.21.15.102:0/776251565 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ac009400 con 0x7fc4c40cda90 2024-03-23T03:47:19.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.991+0000 7f5bb5200640 1 --2- 172.21.15.102:0/246790882 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b9006ad00 0x7f5b9006d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:19.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.990+0000 7f639b400640 1 -- 172.21.15.102:0/3148479416 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f639c03eae0 con 0x7f63a415cc70 2024-03-23T03:47:19.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.991+0000 7fc4c3e00640 1 --2- 172.21.15.102:0/776251565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4c40cda90 0x7fc4c415f100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac00e8d0 tx=0x7fc4ac00ed90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:19.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.991+0000 7f639b400640 1 --2- 172.21.15.102:0/3148479416 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f638406acd0 0x7f638406d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:19.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.991+0000 7f639b400640 1 -- 172.21.15.102:0/3148479416 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f639c0ac200 con 0x7f63a415cc70 2024-03-23T03:47:19.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.992+0000 7f63aa200640 1 -- 172.21.15.102:0/3148479416 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63a4066bb0 con 0x7f63a415cc70 2024-03-23T03:47:19.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.992+0000 7f5bb5200640 1 --2- 172.21.15.102:0/246790882 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b9006ad00 0x7f5b9006d1c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c000f60 tx=0x7f5b9c007f10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:19.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.992+0000 7fc4c0c00640 1 -- 172.21.15.102:0/776251565 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac00c940 con 0x7fc4c40cda90 2024-03-23T03:47:19.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.992+0000 7fc4c0c00640 1 -- 172.21.15.102:0/776251565 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4ac00caa0 con 0x7fc4c40cda90 2024-03-23T03:47:19.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.992+0000 7fc4c0c00640 1 -- 172.21.15.102:0/776251565 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4ac00ce50 con 0x7fc4c40cda90 2024-03-23T03:47:19.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.992+0000 7fc4cb400640 1 -- 172.21.15.102:0/776251565 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b4009b30 con 0x7fc4c40cda90 2024-03-23T03:47:19.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.992+0000 7fc4cb400640 1 -- 172.21.15.102:0/776251565 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c406cfb0 con 0x7fc4c40cda90 2024-03-23T03:47:19.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.992+0000 7f63a3400640 1 --2- 172.21.15.102:0/3148479416 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f638406acd0 0x7f638406d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:19.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.996+0000 7f63a3400640 1 --2- 172.21.15.102:0/3148479416 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f638406acd0 0x7f638406d190 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6394000f60 tx=0x7f6394009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:19.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.995+0000 7fc4cb400640 1 -- 172.21.15.102:0/776251565 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c41f5a20 con 0x7fc4c40cda90 2024-03-23T03:47:19.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.997+0000 7fc4c0c00640 1 -- 172.21.15.102:0/776251565 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc4ac013470 con 0x7fc4c40cda90 2024-03-23T03:47:19.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.998+0000 7fc4c0c00640 1 --2- 172.21.15.102:0/776251565 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc4a006ac00 0x7fc4a006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:19.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.998+0000 7fc4c0c00640 1 -- 172.21.15.102:0/776251565 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc4ac089460 con 0x7fc4c40cda90 2024-03-23T03:47:19.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:18.999+0000 7fc4c8a00640 1 --2- 172.21.15.102:0/776251565 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc4a006ac00 0x7fc4a006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:19.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.000+0000 7fc4c8a00640 1 --2- 172.21.15.102:0/776251565 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc4a006ac00 0x7fc4a006d0c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4035000 tx=0x7fc4b4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:19.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.000+0000 7f639b400640 1 -- 172.21.15.102:0/3148479416 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f639c077180 con 0x7f63a415cc70 2024-03-23T03:47:19.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.003+0000 7f5bacc00640 1 -- 172.21.15.102:0/246790882 <== mon.2 v2:172.21.15.173:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5ba0054200 con 0x7f5bb0165340 2024-03-23T03:47:19.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.012+0000 7fc4c0c00640 1 -- 172.21.15.102:0/776251565 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc4c41f5a20 con 0x7fc4c40cda90 2024-03-23T03:47:19.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.356+0000 7f9138e00640 1 -- 172.21.15.102:0/4034551163 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f91340c9240 con 0x7f91340cd6b0 2024-03-23T03:47:19.359 INFO:teuthology.orchestra.run.smithi102.stdout:60129542152 2024-03-23T03:47:19.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.357+0000 7f912a000640 1 -- 172.21.15.102:0/4034551163 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f912c058830 con 0x7f91340cd6b0 2024-03-23T03:47:19.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.360+0000 7f9138e00640 1 -- 172.21.15.102:0/4034551163 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f911406ac50 msgr2=0x7f911406d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.361+0000 7f9138e00640 1 --2- 172.21.15.102:0/4034551163 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f911406ac50 0x7f911406d110 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9124036000 tx=0x7f9124008030 comp rx=0 tx=0).stop 2024-03-23T03:47:19.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.361+0000 7f9138e00640 1 -- 172.21.15.102:0/4034551163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91340cd6b0 msgr2=0x7f91340c4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.361+0000 7f9138e00640 1 --2- 172.21.15.102:0/4034551163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91340cd6b0 0x7f91340c4c30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f912c00efc0 tx=0x7f912c00c490 comp rx=0 tx=0).stop 2024-03-23T03:47:19.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.362+0000 7f9138e00640 1 -- 172.21.15.102:0/4034551163 shutdown_connections 2024-03-23T03:47:19.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.362+0000 7f9138e00640 1 --2- 172.21.15.102:0/4034551163 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f911406ac50 0x7f911406d110 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.362+0000 7f9138e00640 1 --2- 172.21.15.102:0/4034551163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91340ca900 0x7f9134070320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.362+0000 7f9138e00640 1 --2- 172.21.15.102:0/4034551163 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91340cda90 0x7f91340ca3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.362+0000 7f9138e00640 1 --2- 172.21.15.102:0/4034551163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91340cd6b0 0x7f91340c4c30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.362+0000 7f9138e00640 1 -- 172.21.15.102:0/4034551163 >> 172.21.15.102:0/4034551163 conn(0x7f91340cf780 msgr2=0x7f91340b5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:19.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.363+0000 7f9138e00640 1 -- 172.21.15.102:0/4034551163 shutdown_connections 2024-03-23T03:47:19.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.363+0000 7f9138e00640 1 -- 172.21.15.102:0/4034551163 wait complete. 2024-03-23T03:47:19.385 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:47:19.393 DEBUG:teuthology.orchestra.run.smithi173:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:47:19.394 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.2 2024-03-23T03:47:19.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.392+0000 7f8526400640 1 -- 172.21.15.102:0/2245842818 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f84f80057d0 con 0x7f85200cebb0 2024-03-23T03:47:19.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.392+0000 7fd98a800640 1 -- 172.21.15.102:0/2466815195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fd984067680 con 0x7fd984160ee0 2024-03-23T03:47:19.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.393+0000 7fd981600640 1 -- 172.21.15.102:0/2466815195 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd97007a340 con 0x7fd984160ee0 2024-03-23T03:47:19.397 INFO:teuthology.orchestra.run.smithi102.stdout:60129542151 2024-03-23T03:47:19.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.394+0000 7f8517400640 1 -- 172.21.15.102:0/2245842818 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f850c079d70 con 0x7f85200cebb0 2024-03-23T03:47:19.398 INFO:teuthology.orchestra.run.smithi102.stdout:60129542152 2024-03-23T03:47:19.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.397+0000 7fd98a800640 1 -- 172.21.15.102:0/2466815195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd96406ac50 msgr2=0x7fd96406d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.397+0000 7fd98a800640 1 --2- 172.21.15.102:0/2466815195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd96406ac50 0x7fd96406d110 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00abb0 tx=0x7fd97c009250 comp rx=0 tx=0).stop 2024-03-23T03:47:19.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.397+0000 7fd98a800640 1 -- 172.21.15.102:0/2466815195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd984160ee0 msgr2=0x7fd9840ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.397+0000 7fd98a800640 1 --2- 172.21.15.102:0/2466815195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd984160ee0 0x7fd9840ae000 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd97002b950 tx=0x7fd9700046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.397+0000 7f8526400640 1 -- 172.21.15.102:0/2245842818 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f850006ac50 msgr2=0x7f850006d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.397+0000 7f8526400640 1 --2- 172.21.15.102:0/2245842818 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f850006ac50 0x7f850006d110 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8518005c00 tx=0x7f8518009040 comp rx=0 tx=0).stop 2024-03-23T03:47:19.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.397+0000 7f8526400640 1 -- 172.21.15.102:0/2245842818 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85200cebb0 msgr2=0x7f85200a9090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.397+0000 7f8526400640 1 --2- 172.21.15.102:0/2245842818 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85200cebb0 0x7f85200a9090 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f850c02bdd0 tx=0x7f850c0046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.398+0000 7f8526400640 1 -- 172.21.15.102:0/2245842818 shutdown_connections 2024-03-23T03:47:19.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.398+0000 7f8526400640 1 --2- 172.21.15.102:0/2245842818 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f850006ac50 0x7f850006d110 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.398+0000 7f8526400640 1 --2- 172.21.15.102:0/2245842818 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f852015e940 0x7f85200a95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.398+0000 7f8526400640 1 --2- 172.21.15.102:0/2245842818 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85200cebb0 0x7f85200a9090 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.398+0000 7f8526400640 1 --2- 172.21.15.102:0/2245842818 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f852006d7b0 0x7f85200ac100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.398+0000 7f8526400640 1 -- 172.21.15.102:0/2245842818 >> 172.21.15.102:0/2245842818 conn(0x7f8520069c90 msgr2=0x7f85200b03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:19.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.398+0000 7f8526400640 1 -- 172.21.15.102:0/2245842818 shutdown_connections 2024-03-23T03:47:19.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.398+0000 7f8526400640 1 -- 172.21.15.102:0/2245842818 wait complete. 2024-03-23T03:47:19.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.399+0000 7fd98a800640 1 -- 172.21.15.102:0/2466815195 shutdown_connections 2024-03-23T03:47:19.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.399+0000 7fd98a800640 1 --2- 172.21.15.102:0/2466815195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd96406ac50 0x7fd96406d110 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.399+0000 7fd98a800640 1 --2- 172.21.15.102:0/2466815195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd984160ee0 0x7fd9840ae000 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.399+0000 7fd98a800640 1 --2- 172.21.15.102:0/2466815195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9840cca00 0x7fd98414f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.399+0000 7fd98a800640 1 --2- 172.21.15.102:0/2466815195 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9840cb480 0x7fd9841f90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.399+0000 7fd98a800640 1 -- 172.21.15.102:0/2466815195 >> 172.21.15.102:0/2466815195 conn(0x7fd984065e20 msgr2=0x7fd9840af430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:19.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.399+0000 7f9636800640 1 -- 172.21.15.102:0/525328657 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f96300cc050 con 0x7f963015a2f0 2024-03-23T03:47:19.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.399+0000 7fd98a800640 1 -- 172.21.15.102:0/2466815195 shutdown_connections 2024-03-23T03:47:19.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.399+0000 7fd98a800640 1 -- 172.21.15.102:0/2466815195 wait complete. 2024-03-23T03:47:19.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.401+0000 7f9627400640 1 -- 172.21.15.102:0/525328657 <== mon.2 v2:172.21.15.173:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f96300cc050 con 0x7f963015a2f0 2024-03-23T03:47:19.407 INFO:teuthology.orchestra.run.smithi102.stdout:60129542151 2024-03-23T03:47:19.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.407+0000 7f9636800640 1 -- 172.21.15.102:0/525328657 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f961006ac50 msgr2=0x7f961006d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.407+0000 7f9636800640 1 --2- 172.21.15.102:0/525328657 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f961006ac50 0x7f961006d110 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9620036000 tx=0x7f9620030000 comp rx=0 tx=0).stop 2024-03-23T03:47:19.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.407+0000 7f9636800640 1 -- 172.21.15.102:0/525328657 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f963015a2f0 msgr2=0x7f9630143d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.407+0000 7f9636800640 1 --2- 172.21.15.102:0/525328657 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f963015a2f0 0x7f9630143d80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f962800efc0 tx=0x7f962800c490 comp rx=0 tx=0).stop 2024-03-23T03:47:19.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.407+0000 7f9636800640 1 -- 172.21.15.102:0/525328657 shutdown_connections 2024-03-23T03:47:19.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.407+0000 7f9636800640 1 --2- 172.21.15.102:0/525328657 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f961006ac50 0x7f961006d110 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9620036000 tx=0x7f9620030000 comp rx=0 tx=0).stop 2024-03-23T03:47:19.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.407+0000 7f9636800640 1 --2- 172.21.15.102:0/525328657 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f963015cf20 0x7f9630158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.407+0000 7f9636800640 1 --2- 172.21.15.102:0/525328657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963015a6d0 0x7f96301443f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.407+0000 7f9636800640 1 --2- 172.21.15.102:0/525328657 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f963015a2f0 0x7f9630143d80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.407+0000 7f9636800640 1 -- 172.21.15.102:0/525328657 >> 172.21.15.102:0/525328657 conn(0x7f9630069c90 msgr2=0x7f96300abcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:19.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.408+0000 7f9636800640 1 -- 172.21.15.102:0/525328657 shutdown_connections 2024-03-23T03:47:19.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.408+0000 7f9636800640 1 -- 172.21.15.102:0/525328657 wait complete. 2024-03-23T03:47:19.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.409+0000 7fb34f200640 1 -- 172.21.15.102:0/1984133221 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fb350066500 con 0x7fb35006dfc0 2024-03-23T03:47:19.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.410+0000 7fb346000640 1 -- 172.21.15.102:0/1984133221 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb3400581c0 con 0x7fb35006dfc0 2024-03-23T03:47:19.411 INFO:teuthology.orchestra.run.smithi102.stdout:60129542152 2024-03-23T03:47:19.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.414+0000 7fb34f200640 1 -- 172.21.15.102:0/1984133221 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb33006ac00 msgr2=0x7fb33006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.414+0000 7fb34f200640 1 --2- 172.21.15.102:0/1984133221 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb33006ac00 0x7fb33006d0c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fb350151710 tx=0x7fb33c008040 comp rx=0 tx=0).stop 2024-03-23T03:47:19.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.414+0000 7fb34f200640 1 -- 172.21.15.102:0/1984133221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb35006dfc0 msgr2=0x7fb350156bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.414+0000 7fb34f200640 1 --2- 172.21.15.102:0/1984133221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb35006dfc0 0x7fb350156bd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb34000c450 tx=0x7fb34000c910 comp rx=0 tx=0).stop 2024-03-23T03:47:19.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.415+0000 7fb34f200640 1 -- 172.21.15.102:0/1984133221 shutdown_connections 2024-03-23T03:47:19.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.415+0000 7fb34f200640 1 --2- 172.21.15.102:0/1984133221 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb33006ac00 0x7fb33006d0c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.415+0000 7fb34f200640 1 --2- 172.21.15.102:0/1984133221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3500adac0 0x7fb3501ee760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.415+0000 7fb34f200640 1 --2- 172.21.15.102:0/1984133221 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3500707e0 0x7fb3501ee220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.415+0000 7fb34f200640 1 --2- 172.21.15.102:0/1984133221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb35006dfc0 0x7fb350156bd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.415+0000 7fb34f200640 1 -- 172.21.15.102:0/1984133221 >> 172.21.15.102:0/1984133221 conn(0x7fb3500b0250 msgr2=0x7fb3500b2e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:19.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.416+0000 7fb34f200640 1 -- 172.21.15.102:0/1984133221 shutdown_connections 2024-03-23T03:47:19.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.416+0000 7fb34f200640 1 -- 172.21.15.102:0/1984133221 wait complete. 2024-03-23T03:47:19.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.424+0000 7f63aa200640 1 -- 172.21.15.102:0/3148479416 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f63a40d0600 con 0x7f63a415cc70 2024-03-23T03:47:19.427 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.5 2024-03-23T03:47:19.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.426+0000 7f639b400640 1 -- 172.21.15.102:0/3148479416 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f639c043070 con 0x7f63a415cc70 2024-03-23T03:47:19.427 INFO:teuthology.orchestra.run.smithi102.stdout:60129542151 2024-03-23T03:47:19.428 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.3 2024-03-23T03:47:19.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.427+0000 7f5bb7c00640 1 -- 172.21.15.102:0/246790882 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5bb006cba0 con 0x7f5bb0165340 2024-03-23T03:47:19.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.427+0000 7f5bacc00640 1 -- 172.21.15.102:0/246790882 <== mon.2 v2:172.21.15.173:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5ba0057820 con 0x7f5bb0165340 2024-03-23T03:47:19.429 INFO:teuthology.orchestra.run.smithi102.stdout:60129542151 2024-03-23T03:47:19.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.429+0000 7f63aa200640 1 -- 172.21.15.102:0/3148479416 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f638406acd0 msgr2=0x7f638406d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.429+0000 7f63aa200640 1 --2- 172.21.15.102:0/3148479416 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f638406acd0 0x7f638406d190 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6394000f60 tx=0x7f6394009040 comp rx=0 tx=0).stop 2024-03-23T03:47:19.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.429+0000 7f63aa200640 1 -- 172.21.15.102:0/3148479416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63a415cc70 msgr2=0x7f63a40c5370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.429+0000 7f63aa200640 1 --2- 172.21.15.102:0/3148479416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63a415cc70 0x7f63a40c5370 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f639c032000 tx=0x7f639c004a40 comp rx=0 tx=0).stop 2024-03-23T03:47:19.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.429+0000 7f63aa200640 1 -- 172.21.15.102:0/3148479416 shutdown_connections 2024-03-23T03:47:19.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.429+0000 7f63aa200640 1 --2- 172.21.15.102:0/3148479416 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f638406acd0 0x7f638406d190 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.429+0000 7f63aa200640 1 --2- 172.21.15.102:0/3148479416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63a415cc70 0x7f63a40c5370 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.429+0000 7f63aa200640 1 --2- 172.21.15.102:0/3148479416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f63a40b4e60 0x7f63a41f5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.429+0000 7f63aa200640 1 --2- 172.21.15.102:0/3148479416 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f63a40b1f20 0x7f63a41533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.429+0000 7f63aa200640 1 -- 172.21.15.102:0/3148479416 >> 172.21.15.102:0/3148479416 conn(0x7f63a40cf790 msgr2=0x7f63a40b28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:19.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.430+0000 7f63aa200640 1 -- 172.21.15.102:0/3148479416 shutdown_connections 2024-03-23T03:47:19.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.430+0000 7f63aa200640 1 -- 172.21.15.102:0/3148479416 wait complete. 2024-03-23T03:47:19.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.431+0000 7f5bb7c00640 1 -- 172.21.15.102:0/246790882 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b9006ad00 msgr2=0x7f5b9006d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.431+0000 7f5bb7c00640 1 --2- 172.21.15.102:0/246790882 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b9006ad00 0x7f5b9006d1c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5b9c000f60 tx=0x7f5b9c007f10 comp rx=0 tx=0).stop 2024-03-23T03:47:19.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.431+0000 7f5bb7c00640 1 -- 172.21.15.102:0/246790882 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5bb0165340 msgr2=0x7f5bb015b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.431+0000 7f5bb7c00640 1 --2- 172.21.15.102:0/246790882 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5bb0165340 0x7f5bb015b890 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5ba000e560 tx=0x7f5ba000ea20 comp rx=0 tx=0).stop 2024-03-23T03:47:19.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.431+0000 7f5bb7c00640 1 -- 172.21.15.102:0/246790882 shutdown_connections 2024-03-23T03:47:19.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.431+0000 7f5bb7c00640 1 --2- 172.21.15.102:0/246790882 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b9006ad00 0x7f5b9006d1c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.431+0000 7f5bb7c00640 1 --2- 172.21.15.102:0/246790882 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5bb0165340 0x7f5bb015b890 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.431+0000 7f5bb7c00640 1 --2- 172.21.15.102:0/246790882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5bb01627c0 0x7f5bb0157980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.431+0000 7f5bb7c00640 1 --2- 172.21.15.102:0/246790882 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb00ce930 0x7f5bb01f5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.431+0000 7f5bb7c00640 1 -- 172.21.15.102:0/246790882 >> 172.21.15.102:0/246790882 conn(0x7f5bb006a0b0 msgr2=0x7f5bb006a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:19.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.432+0000 7fc4cb400640 1 -- 172.21.15.102:0/776251565 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fc4c41f5a20 con 0x7fc4c40cda90 2024-03-23T03:47:19.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.432+0000 7fc4c0c00640 1 -- 172.21.15.102:0/776251565 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc4c41f5a20 con 0x7fc4c40cda90 2024-03-23T03:47:19.433 INFO:teuthology.orchestra.run.smithi102.stdout:60129542153 2024-03-23T03:47:19.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.433+0000 7f5bb7c00640 1 -- 172.21.15.102:0/246790882 shutdown_connections 2024-03-23T03:47:19.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.434+0000 7f5bb7c00640 1 -- 172.21.15.102:0/246790882 wait complete. 2024-03-23T03:47:19.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.435+0000 7fc4cb400640 1 -- 172.21.15.102:0/776251565 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc4a006ac00 msgr2=0x7fc4a006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.435+0000 7fc4cb400640 1 --2- 172.21.15.102:0/776251565 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc4a006ac00 0x7fc4a006d0c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fc4b4035000 tx=0x7fc4b4030000 comp rx=0 tx=0).stop 2024-03-23T03:47:19.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.435+0000 7fc4cb400640 1 -- 172.21.15.102:0/776251565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4c40cda90 msgr2=0x7fc4c415f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:19.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.435+0000 7fc4cb400640 1 --2- 172.21.15.102:0/776251565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4c40cda90 0x7fc4c415f100 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc4ac00e8d0 tx=0x7fc4ac00ed90 comp rx=0 tx=0).stop 2024-03-23T03:47:19.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.436+0000 7fc4cb400640 1 -- 172.21.15.102:0/776251565 shutdown_connections 2024-03-23T03:47:19.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.436+0000 7fc4cb400640 1 --2- 172.21.15.102:0/776251565 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc4a006ac00 0x7fc4a006d0c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.436+0000 7fc4cb400640 1 --2- 172.21.15.102:0/776251565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4c415cd90 0x7fc4c406f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.436+0000 7fc4cb400640 1 --2- 172.21.15.102:0/776251565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc4c40cda90 0x7fc4c415f100 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.436+0000 7fc4cb400640 1 --2- 172.21.15.102:0/776251565 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc4c40cd6b0 0x7fc4c4154440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:19.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.436+0000 7fc4cb400640 1 -- 172.21.15.102:0/776251565 >> 172.21.15.102:0/776251565 conn(0x7fc4c40cf780 msgr2=0x7fc4c40b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:19.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.436+0000 7fc4cb400640 1 -- 172.21.15.102:0/776251565 shutdown_connections 2024-03-23T03:47:19.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:19.436+0000 7fc4cb400640 1 -- 172.21.15.102:0/776251565 wait complete. 2024-03-23T03:47:19.438 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.0 2024-03-23T03:47:19.439 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542152 for osd.4 2024-03-23T03:47:19.450 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.6 2024-03-23T03:47:19.454 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542153 for osd.7 2024-03-23T03:47:19.455 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.1 2024-03-23T03:47:20.395 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-23T03:47:20.428 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-23T03:47:20.429 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-23T03:47:20.439 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-23T03:47:20.440 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-23T03:47:20.450 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-23T03:47:20.455 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-23T03:47:20.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-23T03:47:20.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.665+0000 7f1efca00640 1 Processor -- start 2024-03-23T03:47:20.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.668+0000 7f1efca00640 1 -- start start 2024-03-23T03:47:20.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.670+0000 7f1efca00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1ef815ca90 0x7f1ef80d2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.670+0000 7f1efca00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1ef80d3340 0x7f1ef80ceea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.670+0000 7f1efca00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef80c6370 con 0x7f1ef80cf3e0 2024-03-23T03:47:20.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.670+0000 7f1efca00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ef80cf890 con 0x7f1ef80d3340 2024-03-23T03:47:20.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.670+0000 7f1efca00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ef80ce590 con 0x7f1ef815ca90 2024-03-23T03:47:20.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.671+0000 7f1ef5e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1ef815ca90 0x7f1ef80d2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.671+0000 7f1ef5e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1ef815ca90 0x7f1ef80d2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:42466/0 (socket says 172.21.15.102:42466) 2024-03-23T03:47:20.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.671+0000 7f1ef5e00640 1 -- 172.21.15.102:0/174059730 learned_addr learned my addr 172.21.15.102:0/174059730 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:20.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.672+0000 7f1ef5e00640 1 -- 172.21.15.102:0/174059730 >> v2:172.21.15.102:3300/0 conn(0x7f1ef80d3340 msgr2=0x7f1ef80ceea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T03:47:20.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.672+0000 7f1ef5e00640 1 --2- 172.21.15.102:0/174059730 >> v2:172.21.15.102:3300/0 conn(0x7f1ef80d3340 0x7f1ef80ceea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.672+0000 7f1ef5e00640 1 -- 172.21.15.102:0/174059730 >> v1:172.21.15.173:6789/0 conn(0x7f1ef80cf3e0 legacy=0x7f1ef80d0770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T03:47:20.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.672+0000 7f1ef5e00640 1 -- 172.21.15.102:0/174059730 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef80cdcf0 con 0x7f1ef815ca90 2024-03-23T03:47:20.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.679+0000 7f1ef5e00640 1 --2- 172.21.15.102:0/174059730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ef815ca90 0x7f1ef80d2e00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1ef808b060 tx=0x7f1ee403c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbb2e1988b29b16d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.683+0000 7f1ef4a00640 1 -- 172.21.15.102:0/174059730 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4041800 con 0x7f1ef815ca90 2024-03-23T03:47:20.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.683+0000 7f1ef4a00640 1 -- 172.21.15.102:0/174059730 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee4041960 con 0x7f1ef815ca90 2024-03-23T03:47:20.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.683+0000 7f1ef4a00640 1 -- 172.21.15.102:0/174059730 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4041cf0 con 0x7f1ef815ca90 2024-03-23T03:47:20.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.688+0000 7f1efca00640 1 -- 172.21.15.102:0/174059730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ef815ca90 msgr2=0x7f1ef80d2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.688+0000 7f1efca00640 1 --2- 172.21.15.102:0/174059730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ef815ca90 0x7f1ef80d2e00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1ef808b060 tx=0x7f1ee403c940 comp rx=0 tx=0).stop 2024-03-23T03:47:20.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.688+0000 7f1efca00640 1 -- 172.21.15.102:0/174059730 shutdown_connections 2024-03-23T03:47:20.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.688+0000 7f1efca00640 1 --2- 172.21.15.102:0/174059730 >> v2:172.21.15.102:3300/0 conn(0x7f1ef80d3340 0x7f1ef80ceea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.688+0000 7f1efca00640 1 --2- 172.21.15.102:0/174059730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ef815ca90 0x7f1ef80d2e00 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1ef808b060 tx=0x7f1ee403c940 comp rx=0 tx=0).stop 2024-03-23T03:47:20.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.689+0000 7f1efca00640 1 -- 172.21.15.102:0/174059730 >> 172.21.15.102:0/174059730 conn(0x7f1ef8089fd0 msgr2=0x7f1ef80d26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:20.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.690+0000 7f1efca00640 1 -- 172.21.15.102:0/174059730 shutdown_connections 2024-03-23T03:47:20.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.690+0000 7f1efca00640 1 -- 172.21.15.102:0/174059730 wait complete. 2024-03-23T03:47:20.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.699+0000 7f1efca00640 1 Processor -- start 2024-03-23T03:47:20.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.700+0000 7f1efca00640 1 -- start start 2024-03-23T03:47:20.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.701+0000 7f1efca00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ef80cf3e0 0x7f1ef81586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.701+0000 7f1efca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ef80d3340 0x7f1ef80cc010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.701+0000 7f1efca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ef80cb230 0x7f1ef8156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.701+0000 7f1efca00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef8158f10 con 0x7f1ef80d3340 2024-03-23T03:47:20.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.701+0000 7f1efca00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef81591f0 con 0x7f1ef80cb230 2024-03-23T03:47:20.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.701+0000 7f1efca00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef81594d0 con 0x7f1ef80cf3e0 2024-03-23T03:47:20.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.705+0000 7f1ef5400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ef80d3340 0x7f1ef80cc010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.705+0000 7f1ef5e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ef80cf3e0 0x7f1ef81586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.705+0000 7f1ef5e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ef80cf3e0 0x7f1ef81586e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:54556/0 (socket says 172.21.15.102:54556) 2024-03-23T03:47:20.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.705+0000 7f1ef5e00640 1 -- 172.21.15.102:0/1788793517 learned_addr learned my addr 172.21.15.102:0/1788793517 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:20.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.705+0000 7f1ef6800640 1 --2- 172.21.15.102:0/1788793517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ef80cb230 0x7f1ef8156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.706+0000 7f1ef5e00640 1 -- 172.21.15.102:0/1788793517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ef80cb230 msgr2=0x7f1ef8156a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.706+0000 7f1ef5e00640 1 --2- 172.21.15.102:0/1788793517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ef80cb230 0x7f1ef8156a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.706+0000 7f1ef5e00640 1 -- 172.21.15.102:0/1788793517 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ef80d3340 msgr2=0x7f1ef80cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.707+0000 7f1ef5e00640 1 --2- 172.21.15.102:0/1788793517 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ef80d3340 0x7f1ef80cc010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.707+0000 7f1ef5e00640 1 -- 172.21.15.102:0/1788793517 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee40094d0 con 0x7f1ef80cf3e0 2024-03-23T03:47:20.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.707+0000 7f1ef5400640 1 --2- 172.21.15.102:0/1788793517 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ef80d3340 0x7f1ef80cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:47:20.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.712+0000 7f1ef5e00640 1 --2- 172.21.15.102:0/1788793517 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ef80cf3e0 0x7f1ef81586e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4000c00 tx=0x7f1ee4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.712+0000 7f1eee000640 1 -- 172.21.15.102:0/1788793517 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee404ad60 con 0x7f1ef80cf3e0 2024-03-23T03:47:20.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.713+0000 7f1eee000640 1 -- 172.21.15.102:0/1788793517 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee4054400 con 0x7f1ef80cf3e0 2024-03-23T03:47:20.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.713+0000 7f1eee000640 1 -- 172.21.15.102:0/1788793517 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4054790 con 0x7f1ef80cf3e0 2024-03-23T03:47:20.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.713+0000 7f1efca00640 1 -- 172.21.15.102:0/1788793517 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef81597b0 con 0x7f1ef80cf3e0 2024-03-23T03:47:20.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.713+0000 7f1efca00640 1 -- 172.21.15.102:0/1788793517 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef8158c20 con 0x7f1ef80cf3e0 2024-03-23T03:47:20.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.718+0000 7f1efca00640 1 -- 172.21.15.102:0/1788793517 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef8159cd0 con 0x7f1ef80cf3e0 2024-03-23T03:47:20.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.720+0000 7f1eee000640 1 -- 172.21.15.102:0/1788793517 <== mon.2 v2:172.21.15.173:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1ee40548f0 con 0x7f1ef80cf3e0 2024-03-23T03:47:20.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.721+0000 7f1eee000640 1 --2- 172.21.15.102:0/1788793517 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1ed806ac00 0x7f1ed806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.722+0000 7f1eee000640 1 -- 172.21.15.102:0/1788793517 <== mon.2 v2:172.21.15.173:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1ee404f070 con 0x7f1ef80cf3e0 2024-03-23T03:47:20.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.723+0000 7f1ef5400640 1 --2- 172.21.15.102:0/1788793517 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1ed806ac00 0x7f1ed806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.724+0000 7f1ef5400640 1 --2- 172.21.15.102:0/1788793517 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1ed806ac00 0x7f1ed806d0c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1ef81f1af0 tx=0x7f1ee8009060 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.728+0000 7f7f1b200640 1 Processor -- start 2024-03-23T03:47:20.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.728+0000 7f7f1b200640 1 -- start start 2024-03-23T03:47:20.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.730+0000 7f7f1b200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f7f1c06c610 0x7f7f1c06c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.730+0000 7f7f1b200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7f1c06cf30 0x7f7f1c0cc9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.731+0000 7f7f19400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7f1c06cf30 0x7f7f1c0cc9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.731+0000 7f7f19400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7f1c06cf30 0x7f7f1c0cc9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37384/0 (socket says 172.21.15.102:37384) 2024-03-23T03:47:20.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.731+0000 7f7f1a800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7f1c0ccee0 0x7f7f1c0b1930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41218/0 (socket says 172.21.15.102:41218) 2024-03-23T03:47:20.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.732+0000 7f7f19400640 1 -- 172.21.15.102:0/749950600 learned_addr learned my addr 172.21.15.102:0/749950600 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:20.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.732+0000 7f7f1b200640 1 -- 172.21.15.102:0/749950600 --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f1c158b90 con 0x7f7f1c0ccee0 2024-03-23T03:47:20.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.732+0000 7f7f1b200640 1 -- 172.21.15.102:0/749950600 --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f1c0ce2c0 con 0x7f7f1c06cf30 2024-03-23T03:47:20.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.732+0000 7f7f1b200640 1 -- 172.21.15.102:0/749950600 --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f1c0cd740 con 0x7f7f1c06c610 2024-03-23T03:47:20.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.732+0000 7f7f19e00640 1 --2- 172.21.15.102:0/749950600 >> v2:172.21.15.173:3300/0 conn(0x7f7f1c06c610 0x7f7f1c06c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.732+0000 7f7f18a00640 1 -- 172.21.15.102:0/749950600 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984010773 0 0) 0x7f7f1c158b90 con 0x7f7f1c0ccee0 2024-03-23T03:47:20.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.735+0000 7f7f19400640 1 -- 172.21.15.102:0/749950600 >> v2:172.21.15.173:3300/0 conn(0x7f7f1c06c610 msgr2=0x7f7f1c06c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:20.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.735+0000 7f7f19400640 1 --2- 172.21.15.102:0/749950600 >> v2:172.21.15.173:3300/0 conn(0x7f7f1c06c610 0x7f7f1c06c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.735+0000 7f7f19400640 1 -- 172.21.15.102:0/749950600 >> v1:172.21.15.102:6789/0 conn(0x7f7f1c0ccee0 legacy=0x7f7f1c0b1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.735+0000 7f7f19400640 1 -- 172.21.15.102:0/749950600 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f1c0cda20 con 0x7f7f1c06cf30 2024-03-23T03:47:20.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.736+0000 7f1eee000640 1 -- 172.21.15.102:0/1788793517 <== mon.2 v2:172.21.15.173:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1ef8159cd0 con 0x7f1ef80cf3e0 2024-03-23T03:47:20.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.737+0000 7f7f19e00640 1 --2- 172.21.15.102:0/749950600 >> v2:172.21.15.173:3300/0 conn(0x7f7f1c06c610 0x7f7f1c06c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:20.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.737+0000 7f88f3c00640 1 Processor -- start 2024-03-23T03:47:20.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.738+0000 7f7f19400640 1 --2- 172.21.15.102:0/749950600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f1c06cf30 0x7f7f1c0cc9a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c06fcb0 tx=0x7f7f0802cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17856ca393296b8a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.739+0000 7f7f18a00640 1 -- 172.21.15.102:0/749950600 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08008df0 con 0x7f7f1c06cf30 2024-03-23T03:47:20.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.739+0000 7f450da00640 1 Processor -- start 2024-03-23T03:47:20.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.739+0000 7f450da00640 1 -- start start 2024-03-23T03:47:20.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.741+0000 7f450da00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f45080cd3e0 0x7f450806b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.741+0000 7f450da00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45081614c0 0x7f4508161920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.741+0000 7f450da00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450806b520 con 0x7f45080b4730 2024-03-23T03:47:20.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.741+0000 7f450da00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4508161fe0 con 0x7f45080cd3e0 2024-03-23T03:47:20.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.741+0000 7f450da00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45081622c0 con 0x7f45081614c0 2024-03-23T03:47:20.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.739+0000 7f7f18a00640 1 -- 172.21.15.102:0/749950600 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0800a0d0 con 0x7f7f1c06cf30 2024-03-23T03:47:20.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.739+0000 7f7f18a00640 1 -- 172.21.15.102:0/749950600 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0800a4a0 con 0x7f7f1c06cf30 2024-03-23T03:47:20.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.740+0000 7f7f1b200640 1 -- 172.21.15.102:0/749950600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f1c06cf30 msgr2=0x7f7f1c0cc9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.740+0000 7f7f1b200640 1 --2- 172.21.15.102:0/749950600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f1c06cf30 0x7f7f1c0cc9a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7f1c06fcb0 tx=0x7f7f0802cae0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.741+0000 7f7f1b200640 1 -- 172.21.15.102:0/749950600 shutdown_connections 2024-03-23T03:47:20.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.741+0000 7f7f1b200640 1 --2- 172.21.15.102:0/749950600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f1c06cf30 0x7f7f1c0cc9a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.741+0000 7f7f1b200640 1 --2- 172.21.15.102:0/749950600 >> v2:172.21.15.173:3300/0 conn(0x7f7f1c06c610 0x7f7f1c06c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.741+0000 7f7f1b200640 1 -- 172.21.15.102:0/749950600 >> 172.21.15.102:0/749950600 conn(0x7f7f1c069c90 msgr2=0x7f7f1c06a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:20.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.741+0000 7f7f1b200640 1 -- 172.21.15.102:0/749950600 shutdown_connections 2024-03-23T03:47:20.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.741+0000 7f7f1b200640 1 -- 172.21.15.102:0/749950600 wait complete. 2024-03-23T03:47:20.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.738+0000 7f88f3c00640 1 -- start start 2024-03-23T03:47:20.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.740+0000 7f88f3c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f88ec1627c0 0x7f88ec06d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.740+0000 7f88f3c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88ec1657c0 con 0x7f88ec165340 2024-03-23T03:47:20.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.740+0000 7f88f3c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88ec06cdc0 con 0x7f88ec0ce930 2024-03-23T03:47:20.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.740+0000 7f88f3c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88ec161ab0 con 0x7f88ec1627c0 2024-03-23T03:47:20.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.742+0000 7f4506000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45081614c0 0x7f4508161920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.743+0000 7ff00f000640 1 Processor -- start 2024-03-23T03:47:20.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.743+0000 7f88f1200640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f88ec0ce930 0x7f88ec1612d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:58940/0 (socket says 172.21.15.102:58940) 2024-03-23T03:47:20.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.744+0000 7f88f1200640 1 -- 172.21.15.102:0/951484703 learned_addr learned my addr 172.21.15.102:0/951484703 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:20.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.744+0000 7f88f1c00640 1 --2- 172.21.15.102:0/951484703 >> v2:172.21.15.102:3300/0 conn(0x7f88ec1627c0 0x7f88ec06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.743+0000 7ff00f000640 1 -- start start 2024-03-23T03:47:20.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.745+0000 7ff00f000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff00815e7a0 0x7ff0080b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.745+0000 7ff00f000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff00815eb80 con 0x7ff0080adc60 2024-03-23T03:47:20.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.745+0000 7ff00f000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0080b24a0 con 0x7ff00806da40 2024-03-23T03:47:20.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.745+0000 7ff00f000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0080b3a20 con 0x7ff00815e7a0 2024-03-23T03:47:20.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.745+0000 7f88f1c00640 1 -- 172.21.15.102:0/951484703 >> v1:172.21.15.173:6789/0 conn(0x7f88ec0ce930 legacy=0x7f88ec1612d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:20.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.745+0000 7f88f1c00640 1 -- 172.21.15.102:0/951484703 >> v1:172.21.15.102:6789/0 conn(0x7f88ec165340 legacy=0x7f88ec0ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.746+0000 7f88f1c00640 1 -- 172.21.15.102:0/951484703 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec161d90 con 0x7f88ec1627c0 2024-03-23T03:47:20.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.746+0000 7f88f1200640 1 --1- 172.21.15.102:0/951484703 >> v1:172.21.15.173:6789/0 conn(0x7f88ec0ce930 0x7f88ec1612d0 :-1 s=CLOSED pgs=0 cs=0 l=0).handle_server_banner_and_identify state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-23T03:47:20.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.743+0000 7f4507400640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f45080b4730 0x7f45080d04c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:58936/0 (socket says 172.21.15.102:58936) 2024-03-23T03:47:20.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.743+0000 7f4507400640 1 -- 172.21.15.102:0/160644895 learned_addr learned my addr 172.21.15.102:0/160644895 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:20.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.743+0000 7f4506a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f45080cd3e0 0x7f450806b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.744+0000 7f4506a00640 1 -- 172.21.15.102:0/160644895 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45081614c0 msgr2=0x7f4508161920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:20.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.744+0000 7f4506a00640 1 --2- 172.21.15.102:0/160644895 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45081614c0 0x7f4508161920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.744+0000 7f4506a00640 1 -- 172.21.15.102:0/160644895 >> v1:172.21.15.173:6789/0 conn(0x7f45080b4730 legacy=0x7f45080d04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.744+0000 7f4506a00640 1 -- 172.21.15.102:0/160644895 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450806b1b0 con 0x7f45080cd3e0 2024-03-23T03:47:20.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.746+0000 7ff007400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff0080adc60 0x7ff0080b2b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41246/0 (socket says 172.21.15.102:41246) 2024-03-23T03:47:20.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.746+0000 7ff007e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff00815e7a0 0x7ff0080b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.746+0000 7ff007e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff00815e7a0 0x7ff0080b1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:54562/0 (socket says 172.21.15.102:54562) 2024-03-23T03:47:20.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.746+0000 7ff007e00640 1 -- 172.21.15.102:0/1689529552 learned_addr learned my addr 172.21.15.102:0/1689529552 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:20.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.747+0000 7f4506a00640 1 --2- 172.21.15.102:0/160644895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45080cd3e0 0x7f450806b9b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f44f80097a0 tx=0x7f44f802b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d566b97ac09ef9f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.747+0000 7ff006a00640 1 -- 172.21.15.102:0/1689529552 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288410550 0 0) 0x7ff0080b24a0 con 0x7ff00806da40 2024-03-23T03:47:20.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.747+0000 7f4505600640 1 -- 172.21.15.102:0/160644895 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f8030710 con 0x7f45080cd3e0 2024-03-23T03:47:20.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.748+0000 7f4505600640 1 -- 172.21.15.102:0/160644895 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f802e050 con 0x7f45080cd3e0 2024-03-23T03:47:20.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.748+0000 7f4505600640 1 -- 172.21.15.102:0/160644895 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f8030ad0 con 0x7f45080cd3e0 2024-03-23T03:47:20.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.748+0000 7ff006a00640 1 -- 172.21.15.102:0/1689529552 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefec003520 con 0x7ff00806da40 2024-03-23T03:47:20.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.748+0000 7ff006a00640 1 -- 172.21.15.102:0/1689529552 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196850783 0 0) 0x7feff8001110 con 0x7ff0080adc60 2024-03-23T03:47:20.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.748+0000 7ff006a00640 1 -- 172.21.15.102:0/1689529552 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0080b24a0 con 0x7ff0080adc60 2024-03-23T03:47:20.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7f450da00640 1 -- 172.21.15.102:0/160644895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45080cd3e0 msgr2=0x7f450806b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7f450da00640 1 --2- 172.21.15.102:0/160644895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45080cd3e0 0x7f450806b9b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f44f80097a0 tx=0x7f44f802b730 comp rx=0 tx=0).stop 2024-03-23T03:47:20.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7ff006a00640 1 -- 172.21.15.102:0/1689529552 <== mon.1 v1:172.21.15.173:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1394647854 0 0) 0x7fefec003520 con 0x7ff00806da40 2024-03-23T03:47:20.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7ff006a00640 1 -- 172.21.15.102:0/1689529552 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff00815e7a0 msgr2=0x7ff0080b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:20.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7ff006a00640 1 --2- 172.21.15.102:0/1689529552 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff00815e7a0 0x7ff0080b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7f450da00640 1 -- 172.21.15.102:0/160644895 shutdown_connections 2024-03-23T03:47:20.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7f450da00640 1 --2- 172.21.15.102:0/160644895 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45081614c0 0x7f4508161920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7f450da00640 1 --2- 172.21.15.102:0/160644895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45080cd3e0 0x7f450806b9b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7f450da00640 1 -- 172.21.15.102:0/160644895 >> 172.21.15.102:0/160644895 conn(0x7f45080cf790 msgr2=0x7f45080cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:20.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.750+0000 7f450da00640 1 -- 172.21.15.102:0/160644895 shutdown_connections 2024-03-23T03:47:20.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.750+0000 7f450da00640 1 -- 172.21.15.102:0/160644895 wait complete. 2024-03-23T03:47:20.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7ff006a00640 1 -- 172.21.15.102:0/1689529552 >> v1:172.21.15.102:6789/0 conn(0x7ff0080adc60 legacy=0x7ff0080b2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7ff006a00640 1 -- 172.21.15.102:0/1689529552 --> v1:172.21.15.173:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0080b3460 con 0x7ff00806da40 2024-03-23T03:47:20.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7ff006a00640 1 -- 172.21.15.102:0/1689529552 <== mon.1 v1:172.21.15.173:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7ff0000019f0 con 0x7ff00806da40 2024-03-23T03:47:20.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7ff006a00640 1 -- 172.21.15.102:0/1689529552 >> v1:172.21.15.173:6789/0 conn(0x7ff00806da40 legacy=0x7ff0081646f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7ff006a00640 1 --2- 172.21.15.102:0/1689529552 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fefec003b00 0x7fefec003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.750+0000 7f7f1b200640 1 Processor -- start 2024-03-23T03:47:20.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7ff006a00640 1 --2- 172.21.15.102:0/1689529552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fefec007960 0x7fefec009d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7ff006a00640 1 --2- 172.21.15.102:0/1689529552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefec00a290 0x7fefec00c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7ff006a00640 1 -- 172.21.15.102:0/1689529552 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0080b3a20 con 0x7fefec007960 2024-03-23T03:47:20.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7ff006a00640 1 -- 172.21.15.102:0/1689529552 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec00cf40 con 0x7fefec00a290 2024-03-23T03:47:20.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.749+0000 7ff006a00640 1 -- 172.21.15.102:0/1689529552 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefec00d100 con 0x7fefec003b00 2024-03-23T03:47:20.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.751+0000 7ff00d000640 1 --2- 172.21.15.102:0/1689529552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefec00a290 0x7fefec00c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.751+0000 7f88f1c00640 1 --2- 172.21.15.102:0/951484703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88ec1627c0 0x7f88ec06d530 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009d90 tx=0x7f88e402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af76dab60a491bc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.751+0000 7f88eb400640 1 -- 172.21.15.102:0/951484703 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e402cc60 con 0x7f88ec1627c0 2024-03-23T03:47:20.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.752+0000 7f88eb400640 1 -- 172.21.15.102:0/951484703 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e402cdc0 con 0x7f88ec1627c0 2024-03-23T03:47:20.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.752+0000 7f88eb400640 1 -- 172.21.15.102:0/951484703 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e402cc60 con 0x7f88ec1627c0 2024-03-23T03:47:20.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.751+0000 7f7f1b200640 1 -- start start 2024-03-23T03:47:20.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.751+0000 7ff007400640 1 --2- 172.21.15.102:0/1689529552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fefec007960 0x7fefec009d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.752+0000 7ff007400640 1 -- 172.21.15.102:0/1689529552 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fefec003b00 msgr2=0x7fefec003f00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T03:47:20.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.753+0000 7f450da00640 1 Processor -- start 2024-03-23T03:47:20.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.754+0000 7f7f1b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f1c06c610 0x7f7f1c14e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.754+0000 7f450da00640 1 -- start start 2024-03-23T03:47:20.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.754+0000 7f7f1b200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7f1c06cf30 0x7f7f1c15d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.754+0000 7f7f1b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c0ccee0 0x7f7f1c0a9920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.753+0000 7ff007400640 1 --2- 172.21.15.102:0/1689529552 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fefec003b00 0x7fefec003f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.753+0000 7ff007400640 1 -- 172.21.15.102:0/1689529552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefec00a290 msgr2=0x7fefec00c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:20.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.753+0000 7ff007400640 1 --2- 172.21.15.102:0/1689529552 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefec00a290 0x7fefec00c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.753+0000 7ff007400640 1 -- 172.21.15.102:0/1689529552 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0080b3460 con 0x7fefec007960 2024-03-23T03:47:20.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.753+0000 7f88f3c00640 1 -- 172.21.15.102:0/951484703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88ec1627c0 msgr2=0x7f88ec06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.753+0000 7f88f3c00640 1 --2- 172.21.15.102:0/951484703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88ec1627c0 0x7f88ec06d530 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009d90 tx=0x7f88e402beb0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.754+0000 7f7f1b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c14ebc0 con 0x7f7f1c06c610 2024-03-23T03:47:20.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.754+0000 7f7f1b200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c15d9b0 con 0x7f7f1c0ccee0 2024-03-23T03:47:20.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.754+0000 7f7f1b200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c0a7950 con 0x7f7f1c06cf30 2024-03-23T03:47:20.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.754+0000 7f7f19e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f1c06c610 0x7f7f1c14e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.755+0000 7ff007400640 1 --2- 172.21.15.102:0/1689529552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fefec007960 0x7fefec009d50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7feff80018c0 tx=0x7fefec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779f8fa83aa48328 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.755+0000 7f7f19400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7f1c06cf30 0x7f7f1c15d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.755+0000 7f7f1a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c0ccee0 0x7f7f1c0a9920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.755+0000 7f7f19400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7f1c06cf30 0x7f7f1c15d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:54580/0 (socket says 172.21.15.102:54580) 2024-03-23T03:47:20.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.755+0000 7f7f19400640 1 -- 172.21.15.102:0/498533704 learned_addr learned my addr 172.21.15.102:0/498533704 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:20.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.755+0000 7f88f3c00640 1 -- 172.21.15.102:0/951484703 shutdown_connections 2024-03-23T03:47:20.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.755+0000 7f88f3c00640 1 --2- 172.21.15.102:0/951484703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88ec1627c0 0x7f88ec06d530 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.755+0000 7f88f3c00640 1 -- 172.21.15.102:0/951484703 >> 172.21.15.102:0/951484703 conn(0x7f88ec06a0b0 msgr2=0x7f88ec06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:20.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.756+0000 7ff006a00640 1 -- 172.21.15.102:0/1689529552 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff80072d0 con 0x7fefec007960 2024-03-23T03:47:20.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.756+0000 7f7f19400640 1 -- 172.21.15.102:0/498533704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c0ccee0 msgr2=0x7f7f1c0a9920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.756+0000 7ff006a00640 1 -- 172.21.15.102:0/1689529552 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff8007430 con 0x7fefec007960 2024-03-23T03:47:20.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.756+0000 7f7f19400640 1 --2- 172.21.15.102:0/498533704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c0ccee0 0x7f7f1c0a9920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.756+0000 7ff006a00640 1 -- 172.21.15.102:0/1689529552 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff80047e0 con 0x7fefec007960 2024-03-23T03:47:20.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.756+0000 7f88f3c00640 1 -- 172.21.15.102:0/951484703 shutdown_connections 2024-03-23T03:47:20.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.756+0000 7f88f3c00640 1 -- 172.21.15.102:0/951484703 wait complete. 2024-03-23T03:47:20.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.756+0000 7f450da00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45080b4730 0x7f45080b5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.757+0000 7f450da00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45080cd3e0 0x7f45080b70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.757+0000 7ff00f000640 1 -- 172.21.15.102:0/1689529552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fefec007960 msgr2=0x7fefec009d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.757+0000 7ff00f000640 1 --2- 172.21.15.102:0/1689529552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fefec007960 0x7fefec009d50 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7feff80018c0 tx=0x7fefec0057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.757+0000 7f7f19400640 1 -- 172.21.15.102:0/498533704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f1c06c610 msgr2=0x7f7f1c14e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.757+0000 7f7f19400640 1 --2- 172.21.15.102:0/498533704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f1c06c610 0x7f7f1c14e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.757+0000 7f7f19400640 1 -- 172.21.15.102:0/498533704 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f08007040 con 0x7f7f1c06cf30 2024-03-23T03:47:20.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.757+0000 7ff00d000640 1 -- 172.21.15.102:0/1689529552 reap_dead start 2024-03-23T03:47:20.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.757+0000 7f7f19e00640 1 --2- 172.21.15.102:0/498533704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f1c06c610 0x7f7f1c14e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:47:20.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.757+0000 7ff00f000640 1 -- 172.21.15.102:0/1689529552 shutdown_connections 2024-03-23T03:47:20.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.758+0000 7ff00f000640 1 -- 172.21.15.102:0/1689529552 >> 172.21.15.102:0/1689529552 conn(0x7ff008066000 msgr2=0x7ff008069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:20.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.758+0000 7ff00f000640 1 -- 172.21.15.102:0/1689529552 shutdown_connections 2024-03-23T03:47:20.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.758+0000 7ff00f000640 1 -- 172.21.15.102:0/1689529552 wait complete. 2024-03-23T03:47:20.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.758+0000 7f450da00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45081614c0 0x7f45080d1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.758+0000 7f4506000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45080cd3e0 0x7f45080b70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.758+0000 7f4506000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45080cd3e0 0x7f45080b70c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:54586/0 (socket says 172.21.15.102:54586) 2024-03-23T03:47:20.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.759+0000 7f4506000640 1 -- 172.21.15.102:0/143855158 learned_addr learned my addr 172.21.15.102:0/143855158 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:20.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.759+0000 7f7f19400640 1 --2- 172.21.15.102:0/498533704 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7f1c06cf30 0x7f7f1c15d470 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7f08000c00 tx=0x7f7f080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.759+0000 7f4507400640 1 --2- 172.21.15.102:0/143855158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45081614c0 0x7f45080d1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.759+0000 7f450da00640 1 -- 172.21.15.102:0/143855158 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45080c3fa0 con 0x7f45080b4730 2024-03-23T03:47:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.759+0000 7f450da00640 1 -- 172.21.15.102:0/143855158 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45080b79a0 con 0x7f45081614c0 2024-03-23T03:47:20.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.759+0000 7f7f12000640 1 -- 172.21.15.102:0/498533704 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0803d070 con 0x7f7f1c06cf30 2024-03-23T03:47:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.759+0000 7f450da00640 1 -- 172.21.15.102:0/143855158 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45080b7c80 con 0x7f45080cd3e0 2024-03-23T03:47:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.760+0000 7f4506a00640 1 --2- 172.21.15.102:0/143855158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45080b4730 0x7f45080b5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.760+0000 7f7f1b200640 1 -- 172.21.15.102:0/498533704 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1c0a81f0 con 0x7f7f1c06cf30 2024-03-23T03:47:20.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.761+0000 7f4507400640 1 -- 172.21.15.102:0/143855158 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45080cd3e0 msgr2=0x7f45080b70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.761+0000 7f4507400640 1 --2- 172.21.15.102:0/143855158 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45080cd3e0 0x7f45080b70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.761+0000 7f4507400640 1 -- 172.21.15.102:0/143855158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45080b4730 msgr2=0x7f45080b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.761+0000 7f4507400640 1 --2- 172.21.15.102:0/143855158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45080b4730 0x7f45080b5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.761+0000 7f4507400640 1 -- 172.21.15.102:0/143855158 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f8009400 con 0x7f45081614c0 2024-03-23T03:47:20.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.760+0000 7f7f1b200640 1 -- 172.21.15.102:0/498533704 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f1c0a7ba0 con 0x7f7f1c06cf30 2024-03-23T03:47:20.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.761+0000 7f7f12000640 1 -- 172.21.15.102:0/498533704 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f0800a380 con 0x7f7f1c06cf30 2024-03-23T03:47:20.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.761+0000 7ff00f000640 1 Processor -- start 2024-03-23T03:47:20.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.761+0000 7f4506a00640 1 --2- 172.21.15.102:0/143855158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45080b4730 0x7f45080b5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:47:20.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.762+0000 7f7f1b200640 1 -- 172.21.15.102:0/498533704 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1c0a8380 con 0x7f7f1c06cf30 2024-03-23T03:47:20.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.762+0000 7f7f12000640 1 -- 172.21.15.102:0/498533704 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f0800a710 con 0x7f7f1c06cf30 2024-03-23T03:47:20.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.762+0000 7f4507400640 1 --2- 172.21.15.102:0/143855158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45081614c0 0x7f45080d1e80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f450000cac0 tx=0x7f450000cf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.762+0000 7ff00f000640 1 -- start start 2024-03-23T03:47:20.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.761+0000 7f88f3c00640 1 Processor -- start 2024-03-23T03:47:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.763+0000 7f44fea00640 1 -- 172.21.15.102:0/143855158 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450000a640 con 0x7f45081614c0 2024-03-23T03:47:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.763+0000 7f44fea00640 1 -- 172.21.15.102:0/143855158 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450000a7a0 con 0x7f45081614c0 2024-03-23T03:47:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.763+0000 7f450da00640 1 -- 172.21.15.102:0/143855158 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45080b7f60 con 0x7f45081614c0 2024-03-23T03:47:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.763+0000 7f450da00640 1 -- 172.21.15.102:0/143855158 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45080b7600 con 0x7f45081614c0 2024-03-23T03:47:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.764+0000 7f44fea00640 1 -- 172.21.15.102:0/143855158 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450000ab10 con 0x7f45081614c0 2024-03-23T03:47:20.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.763+0000 7f88f3c00640 1 -- start start 2024-03-23T03:47:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.764+0000 7ff00f000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefec007960 0x7ff008159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.764+0000 7ff00f000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff0080ace70 0x7ff0080af210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.764+0000 7ff00f000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0080af750 0x7ff0081ea910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.765+0000 7ff007e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefec007960 0x7ff008159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.765+0000 7ff007e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefec007960 0x7ff008159780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:42526/0 (socket says 172.21.15.102:42526) 2024-03-23T03:47:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.765+0000 7ff007400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff0080ace70 0x7ff0080af210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.765+0000 7ff007400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff0080ace70 0x7ff0080af210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:54588/0 (socket says 172.21.15.102:54588) 2024-03-23T03:47:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.765+0000 7ff00f000640 1 -- 172.21.15.102:0/2068796012 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0080afbd0 con 0x7ff0080af750 2024-03-23T03:47:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.765+0000 7ff007400640 1 -- 172.21.15.102:0/2068796012 learned_addr learned my addr 172.21.15.102:0/2068796012 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.765+0000 7ff00f000640 1 -- 172.21.15.102:0/2068796012 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff008159cc0 con 0x7fefec007960 2024-03-23T03:47:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.765+0000 7ff00f000640 1 -- 172.21.15.102:0/2068796012 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff008160ea0 con 0x7ff0080ace70 2024-03-23T03:47:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.765+0000 7ff00d000640 1 --2- 172.21.15.102:0/2068796012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0080af750 0x7ff0081ea910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.766+0000 7ff00d000640 1 -- 172.21.15.102:0/2068796012 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff0080ace70 msgr2=0x7ff0080af210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.766+0000 7ff00d000640 1 --2- 172.21.15.102:0/2068796012 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff0080ace70 0x7ff0080af210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.766+0000 7ff00d000640 1 -- 172.21.15.102:0/2068796012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefec007960 msgr2=0x7ff008159780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.766+0000 7ff00d000640 1 --2- 172.21.15.102:0/2068796012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefec007960 0x7ff008159780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.766+0000 7ff00d000640 1 -- 172.21.15.102:0/2068796012 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0081618b0 con 0x7ff0080af750 2024-03-23T03:47:20.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.767+0000 7f450da00640 1 -- 172.21.15.102:0/143855158 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45080c4150 con 0x7f45081614c0 2024-03-23T03:47:20.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.767+0000 7f88f3c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88ec0ce930 0x7f88ec1faa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.767+0000 7f88f3c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88ec1627c0 0x7f88ec0cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.767+0000 7f88f3c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88ec165340 0x7f88ec140820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.767+0000 7f88f3c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec0cce80 con 0x7f88ec165340 2024-03-23T03:47:20.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.767+0000 7f88f3c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec0cbc60 con 0x7f88ec1627c0 2024-03-23T03:47:20.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.767+0000 7f88f3c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec0cca80 con 0x7f88ec0ce930 2024-03-23T03:47:20.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.767+0000 7ff00d000640 1 --2- 172.21.15.102:0/2068796012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0080af750 0x7ff0081ea910 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff00806a850 tx=0x7ff000001ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.768+0000 7fefffe00640 1 -- 172.21.15.102:0/2068796012 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff000006c10 con 0x7ff0080af750 2024-03-23T03:47:20.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.769+0000 7fefffe00640 1 -- 172.21.15.102:0/2068796012 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff000006d70 con 0x7ff0080af750 2024-03-23T03:47:20.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.769+0000 7fefffe00640 1 -- 172.21.15.102:0/2068796012 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff000006c10 con 0x7ff0080af750 2024-03-23T03:47:20.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.769+0000 7ff00f000640 1 -- 172.21.15.102:0/2068796012 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff008161b90 con 0x7ff0080af750 2024-03-23T03:47:20.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.767+0000 7f7f12000640 1 -- 172.21.15.102:0/498533704 <== mon.2 v2:172.21.15.173:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7f08041070 con 0x7f7f1c06cf30 2024-03-23T03:47:20.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.769+0000 7f7f12000640 1 --2- 172.21.15.102:0/498533704 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7efc06ac00 0x7f7efc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.769+0000 7f7f12000640 1 -- 172.21.15.102:0/498533704 <== mon.2 v2:172.21.15.173:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7f08033030 con 0x7f7f1c06cf30 2024-03-23T03:47:20.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.769+0000 7ff00f000640 1 -- 172.21.15.102:0/2068796012 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff008163fe0 con 0x7ff0080af750 2024-03-23T03:47:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.768+0000 7f88f1200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88ec0ce930 0x7f88ec1faa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.768+0000 7f88f1200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88ec0ce930 0x7f88ec1faa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:54604/0 (socket says 172.21.15.102:54604) 2024-03-23T03:47:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.768+0000 7f88f1200640 1 -- 172.21.15.102:0/611438510 learned_addr learned my addr 172.21.15.102:0/611438510 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.768+0000 7f88f1c00640 1 --2- 172.21.15.102:0/611438510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88ec165340 0x7f88ec140820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.770+0000 7f88ebe00640 1 --2- 172.21.15.102:0/611438510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88ec1627c0 0x7f88ec0cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.770+0000 7f88f1200640 1 -- 172.21.15.102:0/611438510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88ec1627c0 msgr2=0x7f88ec0cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.770+0000 7f88f1200640 1 --2- 172.21.15.102:0/611438510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88ec1627c0 0x7f88ec0cc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.770+0000 7f88f1200640 1 -- 172.21.15.102:0/611438510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88ec165340 msgr2=0x7f88ec140820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.770+0000 7f88f1200640 1 --2- 172.21.15.102:0/611438510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88ec165340 0x7f88ec140820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.770+0000 7f88f1200640 1 -- 172.21.15.102:0/611438510 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e4009a70 con 0x7f88ec0ce930 2024-03-23T03:47:20.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.771+0000 7f88f1200640 1 --2- 172.21.15.102:0/611438510 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88ec0ce930 0x7f88ec1faa50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f88e000c460 tx=0x7f88e000c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.772+0000 7feffd600640 1 -- 172.21.15.102:0/2068796012 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefdc005250 con 0x7ff0080af750 2024-03-23T03:47:20.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.772+0000 7f44fea00640 1 -- 172.21.15.102:0/143855158 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f450000ac70 con 0x7f45081614c0 2024-03-23T03:47:20.780 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.772+0000 7f7f19e00640 1 --2- 172.21.15.102:0/498533704 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7efc06ac00 0x7f7efc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.773+0000 7f7f19e00640 1 --2- 172.21.15.102:0/498533704 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7efc06ac00 0x7f7efc06d0c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7f14007fd0 tx=0x7f7f14007810 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.773+0000 7f44fea00640 1 --2- 172.21.15.102:0/143855158 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f44e806ac00 0x7f44e806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.774+0000 7f44fea00640 1 -- 172.21.15.102:0/143855158 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4500088f80 con 0x7f45081614c0 2024-03-23T03:47:20.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.774+0000 7f4506a00640 1 --2- 172.21.15.102:0/143855158 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f44e806ac00 0x7f44e806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.774+0000 7fefffe00640 1 -- 172.21.15.102:0/2068796012 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff00000db10 con 0x7ff0080af750 2024-03-23T03:47:20.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.775+0000 7fefffe00640 1 --2- 172.21.15.102:0/2068796012 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fefe806ac90 0x7fefe806d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.776+0000 7fefffe00640 1 -- 172.21.15.102:0/2068796012 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff000088790 con 0x7ff0080af750 2024-03-23T03:47:20.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.776+0000 7f4506a00640 1 --2- 172.21.15.102:0/143855158 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f44e806ac00 0x7f44e806d0c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f44f8034000 tx=0x7f44f8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.776+0000 7ff007e00640 1 --2- 172.21.15.102:0/2068796012 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fefe806ac90 0x7fefe806d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.777+0000 7ff007e00640 1 --2- 172.21.15.102:0/2068796012 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fefe806ac90 0x7fefe806d150 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff0080b3c40 tx=0x7feff4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.778+0000 7f88e8c00640 1 -- 172.21.15.102:0/611438510 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e00074e0 con 0x7f88ec0ce930 2024-03-23T03:47:20.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.778+0000 7f88e8c00640 1 -- 172.21.15.102:0/611438510 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88e0007660 con 0x7f88ec0ce930 2024-03-23T03:47:20.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.778+0000 7f88e8c00640 1 -- 172.21.15.102:0/611438510 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88e0004a20 con 0x7f88ec0ce930 2024-03-23T03:47:20.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.778+0000 7f88f3c00640 1 -- 172.21.15.102:0/611438510 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ec0cb330 con 0x7f88ec0ce930 2024-03-23T03:47:20.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.778+0000 7f88f3c00640 1 -- 172.21.15.102:0/611438510 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ec0cb8f0 con 0x7f88ec0ce930 2024-03-23T03:47:20.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.779+0000 7f7f12000640 1 -- 172.21.15.102:0/498533704 <== mon.2 v2:172.21.15.173:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7f080aae60 con 0x7f7f1c06cf30 2024-03-23T03:47:20.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.780+0000 7f88f3c00640 1 -- 172.21.15.102:0/611438510 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88ec073390 con 0x7f88ec0ce930 2024-03-23T03:47:20.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.783+0000 7f88e8c00640 1 -- 172.21.15.102:0/611438510 <== mon.2 v2:172.21.15.173:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f88e0004b80 con 0x7f88ec0ce930 2024-03-23T03:47:20.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.784+0000 7f88e8c00640 1 --2- 172.21.15.102:0/611438510 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f88cc06ac00 0x7f88cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.784+0000 7f88e8c00640 1 -- 172.21.15.102:0/611438510 <== mon.2 v2:172.21.15.173:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f88e0088800 con 0x7f88ec0ce930 2024-03-23T03:47:20.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.785+0000 7f88ebe00640 1 --2- 172.21.15.102:0/611438510 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f88cc06ac00 0x7f88cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.786+0000 7fefffe00640 1 -- 172.21.15.102:0/2068796012 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff000053720 con 0x7ff0080af750 2024-03-23T03:47:20.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.787+0000 7f88ebe00640 1 --2- 172.21.15.102:0/611438510 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f88cc06ac00 0x7f88cc06d0c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f88ec06be70 tx=0x7f88d8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.788+0000 7f44fea00640 1 -- 172.21.15.102:0/143855158 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f45080c4150 con 0x7f45081614c0 2024-03-23T03:47:20.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.801+0000 7f88e8c00640 1 -- 172.21.15.102:0/611438510 <== mon.2 v2:172.21.15.173:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f88e0053790 con 0x7f88ec0ce930 2024-03-23T03:47:20.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.818+0000 7f0161000640 1 Processor -- start 2024-03-23T03:47:20.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.818+0000 7f0161000640 1 -- start start 2024-03-23T03:47:20.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.820+0000 7f0161000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f015c070520 0x7f015c06d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.820+0000 7f0161000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f015c070900 0x7f015c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.820+0000 7f0161000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f015c06c340 con 0x7f015c06e6e0 2024-03-23T03:47:20.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.820+0000 7f0161000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f015c156840 con 0x7f015c070520 2024-03-23T03:47:20.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.820+0000 7f0161000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f015c06c590 con 0x7f015c070900 2024-03-23T03:47:20.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.822+0000 7f0159e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f015c070520 0x7f015c06d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.822+0000 7f0159e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f015c070520 0x7f015c06d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37440/0 (socket says 172.21.15.102:37440) 2024-03-23T03:47:20.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.822+0000 7f0159400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f015c070900 0x7f015c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.823+0000 7f0159400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f015c070900 0x7f015c06e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:42554/0 (socket says 172.21.15.102:42554) 2024-03-23T03:47:20.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.823+0000 7f0159400640 1 -- 172.21.15.102:0/2047294963 learned_addr learned my addr 172.21.15.102:0/2047294963 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:20.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.824+0000 7f0159e00640 1 -- 172.21.15.102:0/2047294963 >> v2:172.21.15.173:3300/0 conn(0x7f015c070900 msgr2=0x7f015c06e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:20.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.824+0000 7f0159e00640 1 --2- 172.21.15.102:0/2047294963 >> v2:172.21.15.173:3300/0 conn(0x7f015c070900 0x7f015c06e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.824+0000 7f0159e00640 1 -- 172.21.15.102:0/2047294963 >> v1:172.21.15.102:6789/0 conn(0x7f015c06e6e0 legacy=0x7f015c0cc5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:20.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.825+0000 7f0159e00640 1 -- 172.21.15.102:0/2047294963 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015c06c870 con 0x7f015c070520 2024-03-23T03:47:20.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.825+0000 7f0159400640 1 --2- 172.21.15.102:0/2047294963 >> v2:172.21.15.173:3300/0 conn(0x7f015c070900 0x7f015c06e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:47:20.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.827+0000 7f0159e00640 1 --2- 172.21.15.102:0/2047294963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f015c070520 0x7f015c06d350 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f015c071320 tx=0x7f014802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf11ab4ae5c64992 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.827+0000 7f0158a00640 1 -- 172.21.15.102:0/2047294963 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0148031740 con 0x7f015c070520 2024-03-23T03:47:20.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.828+0000 7f0158a00640 1 -- 172.21.15.102:0/2047294963 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01480318a0 con 0x7f015c070520 2024-03-23T03:47:20.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.828+0000 7f0158a00640 1 -- 172.21.15.102:0/2047294963 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0148031c70 con 0x7f015c070520 2024-03-23T03:47:20.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.831+0000 7f0161000640 1 -- 172.21.15.102:0/2047294963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f015c070520 msgr2=0x7f015c06d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.831+0000 7f0161000640 1 --2- 172.21.15.102:0/2047294963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f015c070520 0x7f015c06d350 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f015c071320 tx=0x7f014802b7e0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.832+0000 7f0161000640 1 -- 172.21.15.102:0/2047294963 shutdown_connections 2024-03-23T03:47:20.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.832+0000 7f0161000640 1 --2- 172.21.15.102:0/2047294963 >> v2:172.21.15.173:3300/0 conn(0x7f015c070900 0x7f015c06e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.832+0000 7f0161000640 1 --2- 172.21.15.102:0/2047294963 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f015c070520 0x7f015c06d350 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.832+0000 7f0161000640 1 -- 172.21.15.102:0/2047294963 >> 172.21.15.102:0/2047294963 conn(0x7f015c069e70 msgr2=0x7f015c06a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:20.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.832+0000 7f0161000640 1 -- 172.21.15.102:0/2047294963 shutdown_connections 2024-03-23T03:47:20.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.832+0000 7f0161000640 1 -- 172.21.15.102:0/2047294963 wait complete. 2024-03-23T03:47:20.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.835+0000 7f0161000640 1 Processor -- start 2024-03-23T03:47:20.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.835+0000 7f0161000640 1 -- start start 2024-03-23T03:47:20.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.837+0000 7f0161000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f015c06e6e0 0x7f015c14fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.837+0000 7f0161000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f015c070520 0x7f015c14be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.837+0000 7f0161000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f015c070900 0x7f015c151760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.837+0000 7f0161000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c152f30 con 0x7f015c06e6e0 2024-03-23T03:47:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.837+0000 7f0161000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c1f10e0 con 0x7f015c070900 2024-03-23T03:47:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.837+0000 7f0161000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f015c1f0560 con 0x7f015c070520 2024-03-23T03:47:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.838+0000 7f0159400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f015c070520 0x7f015c14be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.838+0000 7f0159400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f015c070520 0x7f015c14be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:54606/0 (socket says 172.21.15.102:54606) 2024-03-23T03:47:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.838+0000 7f0159400640 1 -- 172.21.15.102:0/1362949860 learned_addr learned my addr 172.21.15.102:0/1362949860 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.838+0000 7f015a800640 1 --2- 172.21.15.102:0/1362949860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f015c070900 0x7f015c151760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.838+0000 7f0159e00640 1 --2- 172.21.15.102:0/1362949860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f015c06e6e0 0x7f015c14fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.838+0000 7f0159400640 1 -- 172.21.15.102:0/1362949860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f015c070900 msgr2=0x7f015c151760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.838+0000 7f0159400640 1 --2- 172.21.15.102:0/1362949860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f015c070900 0x7f015c151760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.838+0000 7f0159400640 1 -- 172.21.15.102:0/1362949860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f015c06e6e0 msgr2=0x7f015c14fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.838+0000 7f0159400640 1 --2- 172.21.15.102:0/1362949860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f015c06e6e0 0x7f015c14fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.838+0000 7f0159400640 1 -- 172.21.15.102:0/1362949860 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c0094d0 con 0x7f015c070520 2024-03-23T03:47:20.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.839+0000 7f0159e00640 1 --2- 172.21.15.102:0/1362949860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f015c06e6e0 0x7f015c14fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:20.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.840+0000 7f0159400640 1 --2- 172.21.15.102:0/1362949860 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f015c070520 0x7f015c14be60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f014c00e9d0 tx=0x7f014c00ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.840+0000 7f7909800640 1 Processor -- start 2024-03-23T03:47:20.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.840+0000 7f0152000640 1 -- 172.21.15.102:0/1362949860 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c00ca60 con 0x7f015c070520 2024-03-23T03:47:20.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.840+0000 7f0152000640 1 -- 172.21.15.102:0/1362949860 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014c00cbc0 con 0x7f015c070520 2024-03-23T03:47:20.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.840+0000 7f0152000640 1 -- 172.21.15.102:0/1362949860 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014c013470 con 0x7f015c070520 2024-03-23T03:47:20.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.841+0000 7f0161000640 1 -- 172.21.15.102:0/1362949860 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0148009400 con 0x7f015c070520 2024-03-23T03:47:20.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.841+0000 7f0161000640 1 -- 172.21.15.102:0/1362949860 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015c1f0b20 con 0x7f015c070520 2024-03-23T03:47:20.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.840+0000 7f7909800640 1 -- start start 2024-03-23T03:47:20.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.842+0000 7f7909800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f79040b4300 0x7f790406bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.842+0000 7f7909800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f79040cd0e0 0x7f79040b46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.842+0000 7f7909800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790406bff0 con 0x7f7904160660 2024-03-23T03:47:20.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.842+0000 7f7909800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79040ce040 con 0x7f79040b4300 2024-03-23T03:47:20.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.842+0000 7f7909800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79040d2620 con 0x7f79040cd0e0 2024-03-23T03:47:20.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.842+0000 7f7902a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f79040b4300 0x7f790406bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.843+0000 7f0147400640 1 -- 172.21.15.102:0/1362949860 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0130005250 con 0x7f015c070520 2024-03-23T03:47:20.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.842+0000 7f7902a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f79040b4300 0x7f790406bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37464/0 (socket says 172.21.15.102:37464) 2024-03-23T03:47:20.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.843+0000 7f7902a00640 1 -- 172.21.15.102:0/4220971681 learned_addr learned my addr 172.21.15.102:0/4220971681 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:20.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.843+0000 7f7902a00640 1 -- 172.21.15.102:0/4220971681 >> v2:172.21.15.173:3300/0 conn(0x7f79040cd0e0 msgr2=0x7f79040b46e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T03:47:20.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.843+0000 7f7902a00640 1 --2- 172.21.15.102:0/4220971681 >> v2:172.21.15.173:3300/0 conn(0x7f79040cd0e0 0x7f79040b46e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.843+0000 7f7902a00640 1 -- 172.21.15.102:0/4220971681 >> v1:172.21.15.173:6789/0 conn(0x7f7904160660 legacy=0x7f7904160ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:20.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.843+0000 7f7902a00640 1 -- 172.21.15.102:0/4220971681 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79040b4c20 con 0x7f79040b4300 2024-03-23T03:47:20.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.847+0000 7f7902a00640 1 --2- 172.21.15.102:0/4220971681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79040b4300 0x7f790406bab0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f79040d3890 tx=0x7f78f002bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c2589e9986e633b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.848+0000 7f7901600640 1 -- 172.21.15.102:0/4220971681 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f002d6b0 con 0x7f79040b4300 2024-03-23T03:47:20.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.848+0000 7f7901600640 1 -- 172.21.15.102:0/4220971681 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78f002d810 con 0x7f79040b4300 2024-03-23T03:47:20.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.848+0000 7f7901600640 1 -- 172.21.15.102:0/4220971681 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f002dba0 con 0x7f79040b4300 2024-03-23T03:47:20.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.849+0000 7f7909800640 1 -- 172.21.15.102:0/4220971681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79040b4300 msgr2=0x7f790406bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.848+0000 7f0152000640 1 -- 172.21.15.102:0/1362949860 <== mon.2 v2:172.21.15.173:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f014c005990 con 0x7f015c070520 2024-03-23T03:47:20.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.849+0000 7f0152000640 1 --2- 172.21.15.102:0/1362949860 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f013c06ac30 0x7f013c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.849+0000 7f7909800640 1 --2- 172.21.15.102:0/4220971681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79040b4300 0x7f790406bab0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f79040d3890 tx=0x7f78f002bb00 comp rx=0 tx=0).stop 2024-03-23T03:47:20.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.850+0000 7f0159e00640 1 --2- 172.21.15.102:0/1362949860 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f013c06ac30 0x7f013c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.851+0000 7efd1da00640 1 Processor -- start 2024-03-23T03:47:20.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.851+0000 7f0152000640 1 -- 172.21.15.102:0/1362949860 <== mon.2 v2:172.21.15.173:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f014c058610 con 0x7f015c070520 2024-03-23T03:47:20.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.851+0000 7f7909800640 1 -- 172.21.15.102:0/4220971681 shutdown_connections 2024-03-23T03:47:20.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.851+0000 7f7909800640 1 --2- 172.21.15.102:0/4220971681 >> v2:172.21.15.173:3300/0 conn(0x7f79040cd0e0 0x7f79040b46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.851+0000 7f7909800640 1 --2- 172.21.15.102:0/4220971681 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79040b4300 0x7f790406bab0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.851+0000 7f7909800640 1 -- 172.21.15.102:0/4220971681 >> 172.21.15.102:0/4220971681 conn(0x7f79040cf300 msgr2=0x7f79040cf710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:20.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.852+0000 7f0159e00640 1 --2- 172.21.15.102:0/1362949860 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f013c06ac30 0x7f013c06d0f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0148035000 tx=0x7f0148008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.851+0000 7efd1da00640 1 -- start start 2024-03-23T03:47:20.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.853+0000 7efd1da00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efd1815e550 0x7efd1806de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.853+0000 7efd1da00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7efd18161780 0x7efd18070530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.853+0000 7efd1da00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd18070a70 con 0x7efd180700d0 2024-03-23T03:47:20.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.853+0000 7efd1da00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd18161c50 con 0x7efd18161780 2024-03-23T03:47:20.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.853+0000 7efd1da00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd18161f30 con 0x7efd1815e550 2024-03-23T03:47:20.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.854+0000 7f7909800640 1 -- 172.21.15.102:0/4220971681 shutdown_connections 2024-03-23T03:47:20.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.854+0000 7f7909800640 1 -- 172.21.15.102:0/4220971681 wait complete. 2024-03-23T03:47:20.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.854+0000 7efd17400640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7efd180700d0 0x7efd18161070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:58968/0 (socket says 172.21.15.102:58968) 2024-03-23T03:47:20.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.854+0000 7efd17400640 1 -- 172.21.15.102:0/926423701 learned_addr learned my addr 172.21.15.102:0/926423701 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:20.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.854+0000 7efd17e00640 1 --2- 172.21.15.102:0/926423701 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efd1815e550 0x7efd1806de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.855+0000 7efd17e00640 1 -- 172.21.15.102:0/926423701 >> v2:172.21.15.173:3300/0 conn(0x7efd18161780 msgr2=0x7efd18070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:20.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.855+0000 7efd17e00640 1 --2- 172.21.15.102:0/926423701 >> v2:172.21.15.173:3300/0 conn(0x7efd18161780 0x7efd18070530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.855+0000 7efd17e00640 1 -- 172.21.15.102:0/926423701 >> v1:172.21.15.173:6789/0 conn(0x7efd180700d0 legacy=0x7efd18161070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.855+0000 7efd17e00640 1 -- 172.21.15.102:0/926423701 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1806d350 con 0x7efd1815e550 2024-03-23T03:47:20.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.857+0000 7efd17e00640 1 --2- 172.21.15.102:0/926423701 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efd1815e550 0x7efd1806de30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efd040097d0 tx=0x7efd0402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fde277b846c5099 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.857+0000 7f7909800640 1 Processor -- start 2024-03-23T03:47:20.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.857+0000 7f7909800640 1 -- start start 2024-03-23T03:47:20.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.859+0000 7f7909800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79040b4300 0x7f79040c7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.859+0000 7f7909800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79040cd0e0 0x7f79040c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.860+0000 7f7909800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7904160660 0x7f79041fef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.860+0000 7efd16a00640 1 -- 172.21.15.102:0/926423701 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0402d040 con 0x7efd1815e550 2024-03-23T03:47:20.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.860+0000 7efd16a00640 1 -- 172.21.15.102:0/926423701 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd04031810 con 0x7efd1815e550 2024-03-23T03:47:20.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.860+0000 7efd16a00640 1 -- 172.21.15.102:0/926423701 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd04031ba0 con 0x7efd1815e550 2024-03-23T03:47:20.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.860+0000 7f7909800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79041ff4a0 con 0x7f79040cd0e0 2024-03-23T03:47:20.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.860+0000 7f7909800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7904153420 con 0x7f79040b4300 2024-03-23T03:47:20.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.860+0000 7f7909800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79040c5620 con 0x7f7904160660 2024-03-23T03:47:20.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.861+0000 7f0152000640 1 -- 172.21.15.102:0/1362949860 <== mon.2 v2:172.21.15.173:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f014c0550e0 con 0x7f015c070520 2024-03-23T03:47:20.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.862+0000 7efd1da00640 1 -- 172.21.15.102:0/926423701 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efd1815e550 msgr2=0x7efd1806de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.862+0000 7efd1da00640 1 --2- 172.21.15.102:0/926423701 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efd1815e550 0x7efd1806de30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efd040097d0 tx=0x7efd0402b8b0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.860+0000 7f7902a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79040b4300 0x7f79040c7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.860+0000 7f7903400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7904160660 0x7f79041fef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.861+0000 7f7903400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7904160660 0x7f79041fef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:54632/0 (socket says 172.21.15.102:54632) 2024-03-23T03:47:20.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.861+0000 7f7903400640 1 -- 172.21.15.102:0/1522305541 learned_addr learned my addr 172.21.15.102:0/1522305541 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:20.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.862+0000 7f7903400640 1 -- 172.21.15.102:0/1522305541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79040b4300 msgr2=0x7f79040c7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.862+0000 7f7903400640 1 --2- 172.21.15.102:0/1522305541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79040b4300 0x7f79040c7930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.862+0000 7f7903400640 1 -- 172.21.15.102:0/1522305541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79040cd0e0 msgr2=0x7f79040c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.862+0000 7f7903400640 1 --2- 172.21.15.102:0/1522305541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79040cd0e0 0x7f79040c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.862+0000 7f7903400640 1 -- 172.21.15.102:0/1522305541 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f00094d0 con 0x7f7904160660 2024-03-23T03:47:20.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.863+0000 7efd1da00640 1 -- 172.21.15.102:0/926423701 shutdown_connections 2024-03-23T03:47:20.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.863+0000 7efd1da00640 1 --2- 172.21.15.102:0/926423701 >> v2:172.21.15.173:3300/0 conn(0x7efd18161780 0x7efd18070530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.863+0000 7efd1da00640 1 --2- 172.21.15.102:0/926423701 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efd1815e550 0x7efd1806de30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.863+0000 7efd1da00640 1 -- 172.21.15.102:0/926423701 >> 172.21.15.102:0/926423701 conn(0x7efd180b0250 msgr2=0x7efd180b0630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:20.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.864+0000 7efd1da00640 1 -- 172.21.15.102:0/926423701 shutdown_connections 2024-03-23T03:47:20.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.864+0000 7efd1da00640 1 -- 172.21.15.102:0/926423701 wait complete. 2024-03-23T03:47:20.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.866+0000 7efd1da00640 1 Processor -- start 2024-03-23T03:47:20.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.867+0000 7f7903400640 1 --2- 172.21.15.102:0/1522305541 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7904160660 0x7f79041fef60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00c780 tx=0x7f78fc00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.867+0000 7efd1da00640 1 -- start start 2024-03-23T03:47:20.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.869+0000 7efd1da00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd180700d0 0x7efd180af570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.869+0000 7efd1da00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd1815e550 0x7efd18156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.870+0000 7efd17e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd180700d0 0x7efd180af570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.870+0000 7efd17e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd180700d0 0x7efd180af570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:42604/0 (socket says 172.21.15.102:42604) 2024-03-23T03:47:20.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.870+0000 7efd17e00640 1 -- 172.21.15.102:0/1228381018 learned_addr learned my addr 172.21.15.102:0/1228381018 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:20.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.870+0000 7efd1da00640 1 --2- 172.21.15.102:0/1228381018 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efd18161780 0x7efd180ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.870+0000 7f78faa00640 1 -- 172.21.15.102:0/1522305541 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc003d80 con 0x7f7904160660 2024-03-23T03:47:20.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.871+0000 7f78faa00640 1 -- 172.21.15.102:0/1522305541 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78fc00ce90 con 0x7f7904160660 2024-03-23T03:47:20.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.871+0000 7f78faa00640 1 -- 172.21.15.102:0/1522305541 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc0052b0 con 0x7f7904160660 2024-03-23T03:47:20.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.871+0000 7efd1da00640 1 -- 172.21.15.102:0/1228381018 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd180add70 con 0x7efd1815e550 2024-03-23T03:47:20.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.871+0000 7f7909800640 1 -- 172.21.15.102:0/1522305541 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7904152b10 con 0x7f7904160660 2024-03-23T03:47:20.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.871+0000 7f7909800640 1 -- 172.21.15.102:0/1522305541 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7904152dd0 con 0x7f7904160660 2024-03-23T03:47:20.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.871+0000 7efd1d000640 1 --2- 172.21.15.102:0/1228381018 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efd18161780 0x7efd180ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.871+0000 7efd1da00640 1 -- 172.21.15.102:0/1228381018 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd180ae010 con 0x7efd180700d0 2024-03-23T03:47:20.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.872+0000 7efd1da00640 1 -- 172.21.15.102:0/1228381018 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd180afab0 con 0x7efd18161780 2024-03-23T03:47:20.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.873+0000 7efd17400640 1 --2- 172.21.15.102:0/1228381018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd1815e550 0x7efd18156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.873+0000 7efd1d000640 1 -- 172.21.15.102:0/1228381018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd180700d0 msgr2=0x7efd180af570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.873+0000 7efd1d000640 1 --2- 172.21.15.102:0/1228381018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd180700d0 0x7efd180af570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.873+0000 7efd1d000640 1 -- 172.21.15.102:0/1228381018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd1815e550 msgr2=0x7efd18156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:20.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.874+0000 7efd17e00640 1 --2- 172.21.15.102:0/1228381018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd180700d0 0x7efd180af570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:47:20.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.874+0000 7efd1d000640 1 --2- 172.21.15.102:0/1228381018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd1815e550 0x7efd18156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:20.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.874+0000 7efd1d000640 1 -- 172.21.15.102:0/1228381018 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd04009400 con 0x7efd18161780 2024-03-23T03:47:20.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.874+0000 7efd17400640 1 --2- 172.21.15.102:0/1228381018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd1815e550 0x7efd18156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:20.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.875+0000 7efd1d000640 1 --2- 172.21.15.102:0/1228381018 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efd18161780 0x7efd180ae4a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efd1000be20 tx=0x7efd1000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.873+0000 7f7909800640 1 -- 172.21.15.102:0/1522305541 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79041535b0 con 0x7f7904160660 2024-03-23T03:47:20.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.879+0000 7f78faa00640 1 -- 172.21.15.102:0/1522305541 <== mon.2 v2:172.21.15.173:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f78fc006b10 con 0x7f7904160660 2024-03-23T03:47:20.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.881+0000 7f78faa00640 1 --2- 172.21.15.102:0/1522305541 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78e406ac30 0x7f78e406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.881+0000 7f78faa00640 1 -- 172.21.15.102:0/1522305541 <== mon.2 v2:172.21.15.173:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f78fc088c70 con 0x7f7904160660 2024-03-23T03:47:20.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.885+0000 7f7902a00640 1 --2- 172.21.15.102:0/1522305541 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78e406ac30 0x7f78e406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.886+0000 7f7902a00640 1 --2- 172.21.15.102:0/1522305541 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78e406ac30 0x7f78e406d0f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f78f0036000 tx=0x7f78f0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.888+0000 7f78faa00640 1 -- 172.21.15.102:0/1522305541 <== mon.2 v2:172.21.15.173:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f78fc053b40 con 0x7f7904160660 2024-03-23T03:47:20.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.888+0000 7efd0fe00640 1 -- 172.21.15.102:0/1228381018 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1000ce70 con 0x7efd18161780 2024-03-23T03:47:20.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.888+0000 7efd1da00640 1 -- 172.21.15.102:0/1228381018 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd18147c40 con 0x7efd18161780 2024-03-23T03:47:20.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.888+0000 7efd1da00640 1 -- 172.21.15.102:0/1228381018 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd18148200 con 0x7efd18161780 2024-03-23T03:47:20.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.889+0000 7efd0fe00640 1 -- 172.21.15.102:0/1228381018 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd10014440 con 0x7efd18161780 2024-03-23T03:47:20.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.889+0000 7efd0fe00640 1 -- 172.21.15.102:0/1228381018 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1000ce70 con 0x7efd18161780 2024-03-23T03:47:20.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.891+0000 7efd1da00640 1 -- 172.21.15.102:0/1228381018 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcf0005250 con 0x7efd18161780 2024-03-23T03:47:20.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.894+0000 7efd0fe00640 1 -- 172.21.15.102:0/1228381018 <== mon.2 v2:172.21.15.173:3301/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7efd100145d0 con 0x7efd18161780 2024-03-23T03:47:20.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.896+0000 7efd0fe00640 1 --2- 172.21.15.102:0/1228381018 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efcf806ac60 0x7efcf806d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:20.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.896+0000 7efd0fe00640 1 -- 172.21.15.102:0/1228381018 <== mon.2 v2:172.21.15.173:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7efd10089570 con 0x7efd18161780 2024-03-23T03:47:20.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.897+0000 7efd17e00640 1 --2- 172.21.15.102:0/1228381018 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efcf806ac60 0x7efcf806d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:20.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.898+0000 7efd17e00640 1 --2- 172.21.15.102:0/1228381018 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efcf806ac60 0x7efcf806d120 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efd0402b8f0 tx=0x7efd0403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:20.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:20.908+0000 7efd0fe00640 1 -- 172.21.15.102:0/1228381018 <== mon.2 v2:172.21.15.173:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efd10054580 con 0x7efd18161780 2024-03-23T03:47:21.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.171+0000 7f1efca00640 1 -- 172.21.15.102:0/1788793517 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1ef8159cd0 con 0x7f1ef80cf3e0 2024-03-23T03:47:21.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.174+0000 7f1eee000640 1 -- 172.21.15.102:0/1788793517 <== mon.2 v2:172.21.15.173:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1ef8159cd0 con 0x7f1ef80cf3e0 2024-03-23T03:47:21.181 INFO:teuthology.orchestra.run.smithi102.stdout:60129542153 2024-03-23T03:47:21.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.181+0000 7f1efca00640 1 -- 172.21.15.102:0/1788793517 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1ed806ac00 msgr2=0x7f1ed806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.181+0000 7f1efca00640 1 --2- 172.21.15.102:0/1788793517 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1ed806ac00 0x7f1ed806d0c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1ef81f1af0 tx=0x7f1ee8009060 comp rx=0 tx=0).stop 2024-03-23T03:47:21.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.181+0000 7f1efca00640 1 -- 172.21.15.102:0/1788793517 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ef80cf3e0 msgr2=0x7f1ef81586e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.181+0000 7f1efca00640 1 --2- 172.21.15.102:0/1788793517 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ef80cf3e0 0x7f1ef81586e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4000c00 tx=0x7f1ee4004730 comp rx=0 tx=0).stop 2024-03-23T03:47:21.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.184+0000 7feffd600640 1 -- 172.21.15.102:0/2068796012 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fefdc005500 con 0x7ff0080af750 2024-03-23T03:47:21.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.185+0000 7f1efca00640 1 -- 172.21.15.102:0/1788793517 shutdown_connections 2024-03-23T03:47:21.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.185+0000 7f1efca00640 1 --2- 172.21.15.102:0/1788793517 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1ed806ac00 0x7f1ed806d0c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.185+0000 7f1efca00640 1 --2- 172.21.15.102:0/1788793517 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ef80cb230 0x7f1ef8156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.185+0000 7f1efca00640 1 --2- 172.21.15.102:0/1788793517 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ef80d3340 0x7f1ef80cc010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.185+0000 7f1efca00640 1 --2- 172.21.15.102:0/1788793517 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ef80cf3e0 0x7f1ef81586e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.185+0000 7f1efca00640 1 -- 172.21.15.102:0/1788793517 >> 172.21.15.102:0/1788793517 conn(0x7f1ef8089fd0 msgr2=0x7f1ef80c7be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:21.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.186+0000 7fefffe00640 1 -- 172.21.15.102:0/2068796012 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff000056d40 con 0x7ff0080af750 2024-03-23T03:47:21.187 INFO:teuthology.orchestra.run.smithi102.stdout:60129542153 2024-03-23T03:47:21.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.189+0000 7f1efca00640 1 -- 172.21.15.102:0/1788793517 shutdown_connections 2024-03-23T03:47:21.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.189+0000 7f1efca00640 1 -- 172.21.15.102:0/1788793517 wait complete. 2024-03-23T03:47:21.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.190+0000 7feffd600640 1 -- 172.21.15.102:0/2068796012 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fefe806ac90 msgr2=0x7fefe806d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.190+0000 7feffd600640 1 --2- 172.21.15.102:0/2068796012 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fefe806ac90 0x7fefe806d150 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff0080b3c40 tx=0x7feff4008040 comp rx=0 tx=0).stop 2024-03-23T03:47:21.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.190+0000 7feffd600640 1 -- 172.21.15.102:0/2068796012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0080af750 msgr2=0x7ff0081ea910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.190+0000 7feffd600640 1 --2- 172.21.15.102:0/2068796012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0080af750 0x7ff0081ea910 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff00806a850 tx=0x7ff000001ae0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.191+0000 7feffd600640 1 -- 172.21.15.102:0/2068796012 shutdown_connections 2024-03-23T03:47:21.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.191+0000 7feffd600640 1 --2- 172.21.15.102:0/2068796012 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fefe806ac90 0x7fefe806d150 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.191+0000 7feffd600640 1 --2- 172.21.15.102:0/2068796012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0080af750 0x7ff0081ea910 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.191+0000 7feffd600640 1 --2- 172.21.15.102:0/2068796012 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff0080ace70 0x7ff0080af210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.191+0000 7feffd600640 1 --2- 172.21.15.102:0/2068796012 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fefec007960 0x7ff008159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.191+0000 7feffd600640 1 -- 172.21.15.102:0/2068796012 >> 172.21.15.102:0/2068796012 conn(0x7ff008066000 msgr2=0x7ff0080b2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:21.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.191+0000 7feffd600640 1 -- 172.21.15.102:0/2068796012 shutdown_connections 2024-03-23T03:47:21.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.191+0000 7feffd600640 1 -- 172.21.15.102:0/2068796012 wait complete. 2024-03-23T03:47:21.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.197+0000 7f88f3c00640 1 -- 172.21.15.102:0/611438510 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f88ec15ad80 con 0x7f88ec0ce930 2024-03-23T03:47:21.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.202+0000 7f88e8c00640 1 -- 172.21.15.102:0/611438510 <== mon.2 v2:172.21.15.173:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f88e0018090 con 0x7f88ec0ce930 2024-03-23T03:47:21.205 INFO:teuthology.orchestra.run.smithi102.stdout:60129542152 2024-03-23T03:47:21.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.206+0000 7f450da00640 1 -- 172.21.15.102:0/143855158 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f45080c4150 con 0x7f45081614c0 2024-03-23T03:47:21.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.208+0000 7f88f3c00640 1 -- 172.21.15.102:0/611438510 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f88cc06ac00 msgr2=0x7f88cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.208+0000 7f88f3c00640 1 --2- 172.21.15.102:0/611438510 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f88cc06ac00 0x7f88cc06d0c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f88ec06be70 tx=0x7f88d8009040 comp rx=0 tx=0).stop 2024-03-23T03:47:21.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.208+0000 7f88f3c00640 1 -- 172.21.15.102:0/611438510 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88ec0ce930 msgr2=0x7f88ec1faa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.208+0000 7f88f3c00640 1 --2- 172.21.15.102:0/611438510 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88ec0ce930 0x7f88ec1faa50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f88e000c460 tx=0x7f88e000c920 comp rx=0 tx=0).stop 2024-03-23T03:47:21.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.209+0000 7f88f3c00640 1 -- 172.21.15.102:0/611438510 shutdown_connections 2024-03-23T03:47:21.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.209+0000 7f88f3c00640 1 --2- 172.21.15.102:0/611438510 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f88cc06ac00 0x7f88cc06d0c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.209+0000 7f88f3c00640 1 --2- 172.21.15.102:0/611438510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88ec165340 0x7f88ec140820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.209+0000 7f88f3c00640 1 --2- 172.21.15.102:0/611438510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88ec1627c0 0x7f88ec0cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.209+0000 7f88f3c00640 1 --2- 172.21.15.102:0/611438510 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f88ec0ce930 0x7f88ec1faa50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.209+0000 7f88f3c00640 1 -- 172.21.15.102:0/611438510 >> 172.21.15.102:0/611438510 conn(0x7f88ec06a0b0 msgr2=0x7f88ec162c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:21.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.209+0000 7f88f3c00640 1 -- 172.21.15.102:0/611438510 shutdown_connections 2024-03-23T03:47:21.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.209+0000 7f88f3c00640 1 -- 172.21.15.102:0/611438510 wait complete. 2024-03-23T03:47:21.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.210+0000 7f44fea00640 1 -- 172.21.15.102:0/143855158 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f45080c4150 con 0x7f45081614c0 2024-03-23T03:47:21.211 INFO:teuthology.orchestra.run.smithi102.stdout:60129542152 2024-03-23T03:47:21.220 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.3 2024-03-23T03:47:21.220 DEBUG:teuthology.parallel:result is None 2024-03-23T03:47:21.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.219+0000 7f450da00640 1 -- 172.21.15.102:0/143855158 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f44e806ac00 msgr2=0x7f44e806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.219+0000 7f450da00640 1 --2- 172.21.15.102:0/143855158 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f44e806ac00 0x7f44e806d0c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f44f8034000 tx=0x7f44f8008030 comp rx=0 tx=0).stop 2024-03-23T03:47:21.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.219+0000 7f450da00640 1 -- 172.21.15.102:0/143855158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45081614c0 msgr2=0x7f45080d1e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.219+0000 7f450da00640 1 --2- 172.21.15.102:0/143855158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45081614c0 0x7f45080d1e80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f450000cac0 tx=0x7f450000cf80 comp rx=0 tx=0).stop 2024-03-23T03:47:21.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.220+0000 7f450da00640 1 -- 172.21.15.102:0/143855158 shutdown_connections 2024-03-23T03:47:21.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.220+0000 7f450da00640 1 --2- 172.21.15.102:0/143855158 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f44e806ac00 0x7f44e806d0c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.220+0000 7f450da00640 1 --2- 172.21.15.102:0/143855158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45081614c0 0x7f45080d1e80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.220+0000 7f450da00640 1 --2- 172.21.15.102:0/143855158 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45080cd3e0 0x7f45080b70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.220+0000 7f450da00640 1 --2- 172.21.15.102:0/143855158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45080b4730 0x7f45080b5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.220+0000 7f450da00640 1 -- 172.21.15.102:0/143855158 >> 172.21.15.102:0/143855158 conn(0x7f45080cf790 msgr2=0x7f45080b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:21.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.220+0000 7f450da00640 1 -- 172.21.15.102:0/143855158 shutdown_connections 2024-03-23T03:47:21.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.220+0000 7f450da00640 1 -- 172.21.15.102:0/143855158 wait complete. 2024-03-23T03:47:21.225 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.2 2024-03-23T03:47:21.225 DEBUG:teuthology.parallel:result is None 2024-03-23T03:47:21.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.228+0000 7f7f1b200640 1 -- 172.21.15.102:0/498533704 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f7f1c071b10 con 0x7f7f1c06cf30 2024-03-23T03:47:21.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.229+0000 7f7f12000640 1 -- 172.21.15.102:0/498533704 <== mon.2 v2:172.21.15.173:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7f08048020 con 0x7f7f1c06cf30 2024-03-23T03:47:21.230 INFO:teuthology.orchestra.run.smithi102.stdout:60129542153 2024-03-23T03:47:21.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.233+0000 7f7f1b200640 1 -- 172.21.15.102:0/498533704 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7efc06ac00 msgr2=0x7f7efc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.233+0000 7f7f1b200640 1 --2- 172.21.15.102:0/498533704 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7efc06ac00 0x7f7efc06d0c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7f14007fd0 tx=0x7f7f14007810 comp rx=0 tx=0).stop 2024-03-23T03:47:21.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.233+0000 7f7f1b200640 1 -- 172.21.15.102:0/498533704 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7f1c06cf30 msgr2=0x7f7f1c15d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.233+0000 7f7f1b200640 1 --2- 172.21.15.102:0/498533704 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7f1c06cf30 0x7f7f1c15d470 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7f08000c00 tx=0x7f7f080046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.233+0000 7f7f1b200640 1 -- 172.21.15.102:0/498533704 shutdown_connections 2024-03-23T03:47:21.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.233+0000 7f7f1b200640 1 --2- 172.21.15.102:0/498533704 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7efc06ac00 0x7f7efc06d0c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.233+0000 7f7f1b200640 1 --2- 172.21.15.102:0/498533704 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f1c0ccee0 0x7f7f1c0a9920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.235 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.5 2024-03-23T03:47:21.235 DEBUG:teuthology.parallel:result is None 2024-03-23T03:47:21.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.233+0000 7f7f1b200640 1 --2- 172.21.15.102:0/498533704 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7f1c06cf30 0x7f7f1c15d470 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.233+0000 7f7f1b200640 1 --2- 172.21.15.102:0/498533704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7f1c06c610 0x7f7f1c14e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.233+0000 7f7f1b200640 1 -- 172.21.15.102:0/498533704 >> 172.21.15.102:0/498533704 conn(0x7f7f1c069c90 msgr2=0x7f7f1c1580d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:21.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.233+0000 7f7f1b200640 1 -- 172.21.15.102:0/498533704 shutdown_connections 2024-03-23T03:47:21.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.233+0000 7f7f1b200640 1 -- 172.21.15.102:0/498533704 wait complete. 2024-03-23T03:47:21.246 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.0 2024-03-23T03:47:21.247 DEBUG:teuthology.parallel:result is None 2024-03-23T03:47:21.252 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.4 2024-03-23T03:47:21.252 DEBUG:teuthology.parallel:result is None 2024-03-23T03:47:21.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.261+0000 7f0147400640 1 -- 172.21.15.102:0/1362949860 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f01300057d0 con 0x7f015c070520 2024-03-23T03:47:21.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.262+0000 7f0152000640 1 -- 172.21.15.102:0/1362949860 <== mon.2 v2:172.21.15.173:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f014c017030 con 0x7f015c070520 2024-03-23T03:47:21.263 INFO:teuthology.orchestra.run.smithi102.stdout:60129542152 2024-03-23T03:47:21.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.264+0000 7f0147400640 1 -- 172.21.15.102:0/1362949860 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f013c06ac30 msgr2=0x7f013c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.264+0000 7f0147400640 1 --2- 172.21.15.102:0/1362949860 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f013c06ac30 0x7f013c06d0f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0148035000 tx=0x7f0148008030 comp rx=0 tx=0).stop 2024-03-23T03:47:21.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.264+0000 7f0147400640 1 -- 172.21.15.102:0/1362949860 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f015c070520 msgr2=0x7f015c14be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.264+0000 7f0147400640 1 --2- 172.21.15.102:0/1362949860 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f015c070520 0x7f015c14be60 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f014c00e9d0 tx=0x7f014c00ee90 comp rx=0 tx=0).stop 2024-03-23T03:47:21.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.264+0000 7f0147400640 1 -- 172.21.15.102:0/1362949860 shutdown_connections 2024-03-23T03:47:21.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.264+0000 7f0147400640 1 --2- 172.21.15.102:0/1362949860 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f013c06ac30 0x7f013c06d0f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.264+0000 7f0147400640 1 --2- 172.21.15.102:0/1362949860 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f015c070900 0x7f015c151760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.264+0000 7f0147400640 1 --2- 172.21.15.102:0/1362949860 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f015c070520 0x7f015c14be60 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.264+0000 7f0147400640 1 --2- 172.21.15.102:0/1362949860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f015c06e6e0 0x7f015c14fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.264+0000 7f0147400640 1 -- 172.21.15.102:0/1362949860 >> 172.21.15.102:0/1362949860 conn(0x7f015c069e70 msgr2=0x7f015c06d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:21.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.264+0000 7f0147400640 1 -- 172.21.15.102:0/1362949860 shutdown_connections 2024-03-23T03:47:21.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.264+0000 7f0147400640 1 -- 172.21.15.102:0/1362949860 wait complete. 2024-03-23T03:47:21.281 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.6 2024-03-23T03:47:21.281 DEBUG:teuthology.parallel:result is None 2024-03-23T03:47:21.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.345+0000 7efd1da00640 1 -- 172.21.15.102:0/1228381018 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7efcf00057d0 con 0x7efd18161780 2024-03-23T03:47:21.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.346+0000 7efd0fe00640 1 -- 172.21.15.102:0/1228381018 <== mon.2 v2:172.21.15.173:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efd10057ba0 con 0x7efd18161780 2024-03-23T03:47:21.347 INFO:teuthology.orchestra.run.smithi102.stdout:60129542152 2024-03-23T03:47:21.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.348+0000 7efd1da00640 1 -- 172.21.15.102:0/1228381018 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efcf806ac60 msgr2=0x7efcf806d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.348+0000 7efd1da00640 1 --2- 172.21.15.102:0/1228381018 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efcf806ac60 0x7efcf806d120 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7efd0402b8f0 tx=0x7efd0403c040 comp rx=0 tx=0).stop 2024-03-23T03:47:21.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.348+0000 7efd1da00640 1 -- 172.21.15.102:0/1228381018 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efd18161780 msgr2=0x7efd180ae4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.348+0000 7efd1da00640 1 --2- 172.21.15.102:0/1228381018 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efd18161780 0x7efd180ae4a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efd1000be20 tx=0x7efd1000bf10 comp rx=0 tx=0).stop 2024-03-23T03:47:21.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.349+0000 7efd1da00640 1 -- 172.21.15.102:0/1228381018 shutdown_connections 2024-03-23T03:47:21.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.349+0000 7efd1da00640 1 --2- 172.21.15.102:0/1228381018 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efcf806ac60 0x7efcf806d120 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.349+0000 7efd1da00640 1 --2- 172.21.15.102:0/1228381018 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efd18161780 0x7efd180ae4a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.349+0000 7efd1da00640 1 --2- 172.21.15.102:0/1228381018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd1815e550 0x7efd18156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.349+0000 7efd1da00640 1 --2- 172.21.15.102:0/1228381018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd180700d0 0x7efd180af570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.349+0000 7efd1da00640 1 -- 172.21.15.102:0/1228381018 >> 172.21.15.102:0/1228381018 conn(0x7efd180b0250 msgr2=0x7efd181623c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:21.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.349+0000 7efd1da00640 1 -- 172.21.15.102:0/1228381018 shutdown_connections 2024-03-23T03:47:21.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.349+0000 7efd1da00640 1 -- 172.21.15.102:0/1228381018 wait complete. 2024-03-23T03:47:21.364 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.1 2024-03-23T03:47:21.364 DEBUG:teuthology.parallel:result is None 2024-03-23T03:47:21.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.374+0000 7f7909800640 1 -- 172.21.15.102:0/1522305541 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f7904073390 con 0x7f7904160660 2024-03-23T03:47:21.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.375+0000 7f78faa00640 1 -- 172.21.15.102:0/1522305541 <== mon.2 v2:172.21.15.173:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f78fc017090 con 0x7f7904160660 2024-03-23T03:47:21.376 INFO:teuthology.orchestra.run.smithi102.stdout:60129542154 2024-03-23T03:47:21.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.377+0000 7f7909800640 1 -- 172.21.15.102:0/1522305541 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78e406ac30 msgr2=0x7f78e406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.377+0000 7f7909800640 1 --2- 172.21.15.102:0/1522305541 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78e406ac30 0x7f78e406d0f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f78f0036000 tx=0x7f78f0008030 comp rx=0 tx=0).stop 2024-03-23T03:47:21.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.377+0000 7f7909800640 1 -- 172.21.15.102:0/1522305541 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7904160660 msgr2=0x7f79041fef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.377+0000 7f7909800640 1 --2- 172.21.15.102:0/1522305541 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7904160660 0x7f79041fef60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00c780 tx=0x7f78fc00cc40 comp rx=0 tx=0).stop 2024-03-23T03:47:21.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.377+0000 7f7909800640 1 -- 172.21.15.102:0/1522305541 shutdown_connections 2024-03-23T03:47:21.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.377+0000 7f7909800640 1 --2- 172.21.15.102:0/1522305541 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78e406ac30 0x7f78e406d0f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.377+0000 7f7909800640 1 --2- 172.21.15.102:0/1522305541 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7904160660 0x7f79041fef60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.377+0000 7f7909800640 1 --2- 172.21.15.102:0/1522305541 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f79040cd0e0 0x7f79040c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.377+0000 7f7909800640 1 --2- 172.21.15.102:0/1522305541 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79040b4300 0x7f79040c7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.377+0000 7f7909800640 1 -- 172.21.15.102:0/1522305541 >> 172.21.15.102:0/1522305541 conn(0x7f79040cf300 msgr2=0x7f790415c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:21.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.378+0000 7f7909800640 1 -- 172.21.15.102:0/1522305541 shutdown_connections 2024-03-23T03:47:21.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.378+0000 7f7909800640 1 -- 172.21.15.102:0/1522305541 wait complete. 2024-03-23T03:47:21.395 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542154 got 60129542154 for osd.7 2024-03-23T03:47:21.395 DEBUG:teuthology.parallel:result is None 2024-03-23T03:47:21.395 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-23T03:47:21.396 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:47:21.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.611+0000 7f6641400640 1 Processor -- start 2024-03-23T03:47:21.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.611+0000 7f6641400640 1 -- start start 2024-03-23T03:47:21.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.612+0000 7f6641400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f663c143460 0x7f663c15aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:21.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.613+0000 7f6641400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663c0cc6b0 con 0x7f663c0cde20 2024-03-23T03:47:21.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.613+0000 7f6641400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663c0cc950 con 0x7f663c143930 2024-03-23T03:47:21.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.613+0000 7f6641400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663c0ccc30 con 0x7f663c143460 2024-03-23T03:47:21.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.613+0000 7f6640a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f663c0cde20 0x7f663c070020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41260/0 (socket says 172.21.15.102:41260) 2024-03-23T03:47:21.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.613+0000 7f6640a00640 1 -- 172.21.15.102:0/3435362440 learned_addr learned my addr 172.21.15.102:0/3435362440 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:21.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.613+0000 7f663be00640 1 --2- 172.21.15.102:0/3435362440 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f663c143460 0x7f663c15aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:21.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.613+0000 7f663b400640 1 -- 172.21.15.102:0/3435362440 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789242977 0 0) 0x7f663c0cc6b0 con 0x7f663c0cde20 2024-03-23T03:47:21.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.613+0000 7f663b400640 1 -- 172.21.15.102:0/3435362440 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6620003520 con 0x7f663c0cde20 2024-03-23T03:47:21.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f663b400640 1 -- 172.21.15.102:0/3435362440 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1155574473 0 0) 0x7f6620003520 con 0x7f663c0cde20 2024-03-23T03:47:21.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f663b400640 1 -- 172.21.15.102:0/3435362440 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f663c143460 msgr2=0x7f663c15aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:21.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f663b400640 1 --2- 172.21.15.102:0/3435362440 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f663c143460 0x7f663c15aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f663b400640 1 -- 172.21.15.102:0/3435362440 >> v1:172.21.15.173:6789/0 conn(0x7f663c143930 legacy=0x7f663c0cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f663b400640 1 -- 172.21.15.102:0/3435362440 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c0ccf10 con 0x7f663c0cde20 2024-03-23T03:47:21.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f663b400640 1 -- 172.21.15.102:0/3435362440 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f66340019f0 con 0x7f663c0cde20 2024-03-23T03:47:21.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f663b400640 1 -- 172.21.15.102:0/3435362440 >> v1:172.21.15.102:6789/0 conn(0x7f663c0cde20 legacy=0x7f663c070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f663b400640 1 --2- 172.21.15.102:0/3435362440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66200039a0 0x7f6620003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:21.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f663b400640 1 --2- 172.21.15.102:0/3435362440 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66200047c0 0x7f6620004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:21.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f663b400640 1 --2- 172.21.15.102:0/3435362440 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f663c143460 0x7f6620005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:21.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f663b400640 1 -- 172.21.15.102:0/3435362440 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c0ccc30 con 0x7f66200047c0 2024-03-23T03:47:21.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f663b400640 1 -- 172.21.15.102:0/3435362440 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662000baf0 con 0x7f66200039a0 2024-03-23T03:47:21.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f663b400640 1 -- 172.21.15.102:0/3435362440 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662000bcb0 con 0x7f663c143460 2024-03-23T03:47:21.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f6633e00640 1 --2- 172.21.15.102:0/3435362440 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66200047c0 0x7f6620004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:21.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f663be00640 1 --2- 172.21.15.102:0/3435362440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66200039a0 0x7f6620003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:21.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f6640a00640 1 --2- 172.21.15.102:0/3435362440 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f663c143460 0x7f6620005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:21.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f6633e00640 1 -- 172.21.15.102:0/3435362440 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f663c143460 msgr2=0x7f6620005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:21.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f6633e00640 1 --2- 172.21.15.102:0/3435362440 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f663c143460 0x7f6620005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.614+0000 7f6633e00640 1 -- 172.21.15.102:0/3435362440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66200039a0 msgr2=0x7f6620003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:21.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.615+0000 7f6633e00640 1 --2- 172.21.15.102:0/3435362440 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66200039a0 0x7f6620003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.615+0000 7f6633e00640 1 -- 172.21.15.102:0/3435362440 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6620003520 con 0x7f66200047c0 2024-03-23T03:47:21.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.615+0000 7f6640a00640 1 --2- 172.21.15.102:0/3435362440 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f663c143460 0x7f6620005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:21.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.616+0000 7f6633e00640 1 --2- 172.21.15.102:0/3435362440 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66200047c0 0x7f6620004c40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6624001720 tx=0x7f66200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eed44dc6ad2ed5fe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:21.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.616+0000 7f663b400640 1 -- 172.21.15.102:0/3435362440 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66240052d0 con 0x7f66200047c0 2024-03-23T03:47:21.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.616+0000 7f663b400640 1 -- 172.21.15.102:0/3435362440 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662402e050 con 0x7f66200047c0 2024-03-23T03:47:21.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.616+0000 7f663b400640 1 -- 172.21.15.102:0/3435362440 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66240035f0 con 0x7f66200047c0 2024-03-23T03:47:21.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.616+0000 7f6641400640 1 -- 172.21.15.102:0/3435362440 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66200047c0 msgr2=0x7f6620004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.616+0000 7f6641400640 1 --2- 172.21.15.102:0/3435362440 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66200047c0 0x7f6620004c40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6624001720 tx=0x7f66200057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.616+0000 7f6640a00640 1 -- 172.21.15.102:0/3435362440 reap_dead start 2024-03-23T03:47:21.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.616+0000 7f6641400640 1 -- 172.21.15.102:0/3435362440 shutdown_connections 2024-03-23T03:47:21.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.616+0000 7f6641400640 1 -- 172.21.15.102:0/3435362440 >> 172.21.15.102:0/3435362440 conn(0x7f663c065df0 msgr2=0x7f663c069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:21.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.616+0000 7f6641400640 1 -- 172.21.15.102:0/3435362440 shutdown_connections 2024-03-23T03:47:21.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.616+0000 7f6641400640 1 -- 172.21.15.102:0/3435362440 wait complete. 2024-03-23T03:47:21.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.618+0000 7f6641400640 1 Processor -- start 2024-03-23T03:47:21.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.618+0000 7f6641400640 1 -- start start 2024-03-23T03:47:21.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.619+0000 7f6641400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f66200047c0 0x7f663c1563e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:21.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.619+0000 7f6641400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f663c15a160 0x7f663c15b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:21.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.619+0000 7f6641400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f663c13a240 0x7f663c13a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:21.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.619+0000 7f6641400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c13ac30 con 0x7f663c15a160 2024-03-23T03:47:21.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.619+0000 7f6641400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663c156920 con 0x7f663c13a240 2024-03-23T03:47:21.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.619+0000 7f6641400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663c15cac0 con 0x7f66200047c0 2024-03-23T03:47:21.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.619+0000 7f6633e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f663c15a160 0x7f663c15b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:21.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.619+0000 7f6633e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f663c15a160 0x7f663c15b0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37492/0 (socket says 172.21.15.102:37492) 2024-03-23T03:47:21.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.619+0000 7f6633e00640 1 -- 172.21.15.102:0/2717966203 learned_addr learned my addr 172.21.15.102:0/2717966203 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:21.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.620+0000 7f663be00640 1 --2- 172.21.15.102:0/2717966203 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f66200047c0 0x7f663c1563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:21.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.620+0000 7f6640a00640 1 --2- 172.21.15.102:0/2717966203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f663c13a240 0x7f663c13a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:21.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.620+0000 7f6633e00640 1 -- 172.21.15.102:0/2717966203 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f66200047c0 msgr2=0x7f663c1563e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.620+0000 7f6633e00640 1 --2- 172.21.15.102:0/2717966203 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f66200047c0 0x7f663c1563e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.620+0000 7f6633e00640 1 -- 172.21.15.102:0/2717966203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f663c13a240 msgr2=0x7f663c13a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.620+0000 7f6633e00640 1 --2- 172.21.15.102:0/2717966203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f663c13a240 0x7f663c13a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.620+0000 7f6633e00640 1 -- 172.21.15.102:0/2717966203 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663c15ba20 con 0x7f663c15a160 2024-03-23T03:47:21.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.620+0000 7f6633e00640 1 --2- 172.21.15.102:0/2717966203 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f663c15a160 0x7f663c15b0b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6624008fd0 tx=0x7f6624001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:21.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.620+0000 7f6638c00640 1 -- 172.21.15.102:0/2717966203 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66240076c0 con 0x7f663c15a160 2024-03-23T03:47:21.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.620+0000 7f6638c00640 1 -- 172.21.15.102:0/2717966203 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6624007a30 con 0x7f663c15a160 2024-03-23T03:47:21.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.620+0000 7f6638c00640 1 -- 172.21.15.102:0/2717966203 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6624006c00 con 0x7f663c15a160 2024-03-23T03:47:21.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.620+0000 7f6641400640 1 -- 172.21.15.102:0/2717966203 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f663c15bd00 con 0x7f663c15a160 2024-03-23T03:47:21.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.621+0000 7f6641400640 1 -- 172.21.15.102:0/2717966203 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f663c15c2c0 con 0x7f663c15a160 2024-03-23T03:47:21.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.622+0000 7f6641400640 1 -- 172.21.15.102:0/2717966203 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663c04e030 con 0x7f663c15a160 2024-03-23T03:47:21.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.622+0000 7f6638c00640 1 -- 172.21.15.102:0/2717966203 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f66240035f0 con 0x7f663c15a160 2024-03-23T03:47:21.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.623+0000 7f6638c00640 1 --2- 172.21.15.102:0/2717966203 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f661806ac00 0x7f661806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:21.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.623+0000 7f6638c00640 1 -- 172.21.15.102:0/2717966203 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f66240ab830 con 0x7f663c15a160 2024-03-23T03:47:21.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.623+0000 7f663be00640 1 --2- 172.21.15.102:0/2717966203 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f661806ac00 0x7f661806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:21.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.624+0000 7f663be00640 1 --2- 172.21.15.102:0/2717966203 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f661806ac00 0x7f661806d0c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f662c009a10 tx=0x7f662c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:21.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.627+0000 7f6638c00640 1 -- 172.21.15.102:0/2717966203 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6624076680 con 0x7f663c15a160 2024-03-23T03:47:21.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.806+0000 7f6641400640 1 -- 172.21.15.102:0/2717966203 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f663c144080 con 0x7f661806ac00 2024-03-23T03:47:21.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.822+0000 7f6638c00640 1 -- 172.21.15.102:0/2717966203 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345005 (secure 0 0 0) 0x7f662c013a10 con 0x7f661806ac00 2024-03-23T03:47:21.825 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:21.830 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:47:21.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.830+0000 7f6641400640 1 -- 172.21.15.102:0/2717966203 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f661806ac00 msgr2=0x7f661806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.830+0000 7f6641400640 1 --2- 172.21.15.102:0/2717966203 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f661806ac00 0x7f661806d0c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f662c009a10 tx=0x7f662c008040 comp rx=0 tx=0).stop 2024-03-23T03:47:21.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.830+0000 7f6641400640 1 -- 172.21.15.102:0/2717966203 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f663c15a160 msgr2=0x7f663c15b0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:21.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.830+0000 7f6641400640 1 --2- 172.21.15.102:0/2717966203 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f663c15a160 0x7f663c15b0b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f6624008fd0 tx=0x7f6624001120 comp rx=0 tx=0).stop 2024-03-23T03:47:21.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.830+0000 7f6641400640 1 -- 172.21.15.102:0/2717966203 shutdown_connections 2024-03-23T03:47:21.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.830+0000 7f6641400640 1 --2- 172.21.15.102:0/2717966203 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f661806ac00 0x7f661806d0c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.830+0000 7f6641400640 1 --2- 172.21.15.102:0/2717966203 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f663c13a240 0x7f663c13a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.830+0000 7f6641400640 1 --2- 172.21.15.102:0/2717966203 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f663c15a160 0x7f663c15b0b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.830+0000 7f6641400640 1 --2- 172.21.15.102:0/2717966203 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f66200047c0 0x7f663c1563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:21.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.830+0000 7f6641400640 1 -- 172.21.15.102:0/2717966203 >> 172.21.15.102:0/2717966203 conn(0x7f663c065df0 msgr2=0x7f663c0b1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:21.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.830+0000 7f6641400640 1 -- 172.21.15.102:0/2717966203 shutdown_connections 2024-03-23T03:47:21.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:21.830+0000 7f6641400640 1 -- 172.21.15.102:0/2717966203 wait complete. 2024-03-23T03:47:21.848 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-23T03:47:21.723580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594961,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219556,"kb_used_data":3368,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755164,"statfs":{"total":773094113280,"available":772869287936,"internally_reserved":0,"allocated":3448832,"data_stored":2050706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":221236794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003753"},"pg_stats":[{"pgid":"2.1f","version":"21'4","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.954535+0000","last_change":"2024-03-23T03:46:53.465534+0000","last_active":"2024-03-23T03:46:57.954535+0000","last_peered":"2024-03-23T03:46:57.954535+0000","last_clean":"2024-03-23T03:46:57.954535+0000","last_became_active":"2024-03-23T03:46:53.465309+0000","last_became_peered":"2024-03-23T03:46:53.465309+0000","last_unstale":"2024-03-23T03:46:57.954535+0000","last_undegraded":"2024-03-23T03:46:57.954535+0000","last_fullsized":"2024-03-23T03:46:57.954535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:52:38.737363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951985+0000","last_change":"2024-03-23T03:46:53.963693+0000","last_active":"2024-03-23T03:46:57.951985+0000","last_peered":"2024-03-23T03:46:57.951985+0000","last_clean":"2024-03-23T03:46:57.951985+0000","last_became_active":"2024-03-23T03:46:53.963435+0000","last_became_peered":"2024-03-23T03:46:53.963435+0000","last_unstale":"2024-03-23T03:46:57.951985+0000","last_undegraded":"2024-03-23T03:46:57.951985+0000","last_fullsized":"2024-03-23T03:46:57.951985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:51.878165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.544002+0000","last_change":"2024-03-23T03:46:58.018608+0000","last_active":"2024-03-23T03:46:59.544002+0000","last_peered":"2024-03-23T03:46:59.544002+0000","last_clean":"2024-03-23T03:46:59.544002+0000","last_became_active":"2024-03-23T03:46:58.018216+0000","last_became_peered":"2024-03-23T03:46:58.018216+0000","last_unstale":"2024-03-23T03:46:59.544002+0000","last_undegraded":"2024-03-23T03:46:59.544002+0000","last_fullsized":"2024-03-23T03:46:59.544002+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:41:36.624562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952201+0000","last_change":"2024-03-23T03:46:53.462106+0000","last_active":"2024-03-23T03:46:57.952201+0000","last_peered":"2024-03-23T03:46:57.952201+0000","last_clean":"2024-03-23T03:46:57.952201+0000","last_became_active":"2024-03-23T03:46:53.461757+0000","last_became_peered":"2024-03-23T03:46:53.461757+0000","last_unstale":"2024-03-23T03:46:57.952201+0000","last_undegraded":"2024-03-23T03:46:57.952201+0000","last_fullsized":"2024-03-23T03:46:57.952201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:20:09.953919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.954319+0000","last_change":"2024-03-23T03:46:53.959076+0000","last_active":"2024-03-23T03:46:57.954319+0000","last_peered":"2024-03-23T03:46:57.954319+0000","last_clean":"2024-03-23T03:46:57.954319+0000","last_became_active":"2024-03-23T03:46:53.958351+0000","last_became_peered":"2024-03-23T03:46:53.958351+0000","last_unstale":"2024-03-23T03:46:57.954319+0000","last_undegraded":"2024-03-23T03:46:57.954319+0000","last_fullsized":"2024-03-23T03:46:57.954319+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:55.749189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.423330+0000","last_change":"2024-03-23T03:46:57.979717+0000","last_active":"2024-03-23T03:46:58.423330+0000","last_peered":"2024-03-23T03:46:58.423330+0000","last_clean":"2024-03-23T03:46:58.423330+0000","last_became_active":"2024-03-23T03:46:57.975153+0000","last_became_peered":"2024-03-23T03:46:57.975153+0000","last_unstale":"2024-03-23T03:46:58.423330+0000","last_undegraded":"2024-03-23T03:46:58.423330+0000","last_fullsized":"2024-03-23T03:46:58.423330+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:24:30.082562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017773+0000","last_change":"2024-03-23T03:46:53.480716+0000","last_active":"2024-03-23T03:46:58.017773+0000","last_peered":"2024-03-23T03:46:58.017773+0000","last_clean":"2024-03-23T03:46:58.017773+0000","last_became_active":"2024-03-23T03:46:53.479510+0000","last_became_peered":"2024-03-23T03:46:53.479510+0000","last_unstale":"2024-03-23T03:46:58.017773+0000","last_undegraded":"2024-03-23T03:46:58.017773+0000","last_fullsized":"2024-03-23T03:46:58.017773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:40:20.814830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970572+0000","last_change":"2024-03-23T03:46:53.947108+0000","last_active":"2024-03-23T03:46:57.970572+0000","last_peered":"2024-03-23T03:46:57.970572+0000","last_clean":"2024-03-23T03:46:57.970572+0000","last_became_active":"2024-03-23T03:46:53.946415+0000","last_became_peered":"2024-03-23T03:46:53.946415+0000","last_unstale":"2024-03-23T03:46:57.970572+0000","last_undegraded":"2024-03-23T03:46:57.970572+0000","last_fullsized":"2024-03-23T03:46:57.970572+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:38:13.205855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.426555+0000","last_change":"2024-03-23T03:46:57.968557+0000","last_active":"2024-03-23T03:46:58.426555+0000","last_peered":"2024-03-23T03:46:58.426555+0000","last_clean":"2024-03-23T03:46:58.426555+0000","last_became_active":"2024-03-23T03:46:57.968085+0000","last_became_peered":"2024-03-23T03:46:57.968085+0000","last_unstale":"2024-03-23T03:46:58.426555+0000","last_undegraded":"2024-03-23T03:46:58.426555+0000","last_fullsized":"2024-03-23T03:46:58.426555+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:26:09.015758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.014925+0000","last_change":"2024-03-23T03:46:53.481389+0000","last_active":"2024-03-23T03:46:58.014925+0000","last_peered":"2024-03-23T03:46:58.014925+0000","last_clean":"2024-03-23T03:46:58.014925+0000","last_became_active":"2024-03-23T03:46:53.481119+0000","last_became_peered":"2024-03-23T03:46:53.481119+0000","last_unstale":"2024-03-23T03:46:58.014925+0000","last_undegraded":"2024-03-23T03:46:58.014925+0000","last_fullsized":"2024-03-23T03:46:58.014925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:23:34.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970863+0000","last_change":"2024-03-23T03:46:53.952464+0000","last_active":"2024-03-23T03:46:57.970863+0000","last_peered":"2024-03-23T03:46:57.970863+0000","last_clean":"2024-03-23T03:46:57.970863+0000","last_became_active":"2024-03-23T03:46:53.951343+0000","last_became_peered":"2024-03-23T03:46:53.951343+0000","last_unstale":"2024-03-23T03:46:57.970863+0000","last_undegraded":"2024-03-23T03:46:57.970863+0000","last_fullsized":"2024-03-23T03:46:57.970863+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:35:49.855085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.423616+0000","last_change":"2024-03-23T03:46:57.974943+0000","last_active":"2024-03-23T03:46:58.423616+0000","last_peered":"2024-03-23T03:46:58.423616+0000","last_clean":"2024-03-23T03:46:58.423616+0000","last_became_active":"2024-03-23T03:46:57.974037+0000","last_became_peered":"2024-03-23T03:46:57.974037+0000","last_unstale":"2024-03-23T03:46:58.423616+0000","last_undegraded":"2024-03-23T03:46:58.423616+0000","last_fullsized":"2024-03-23T03:46:58.423616+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:24:17.331178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952188+0000","last_change":"2024-03-23T03:46:53.961283+0000","last_active":"2024-03-23T03:46:57.952188+0000","last_peered":"2024-03-23T03:46:57.952188+0000","last_clean":"2024-03-23T03:46:57.952188+0000","last_became_active":"2024-03-23T03:46:53.961004+0000","last_became_peered":"2024-03-23T03:46:53.961004+0000","last_unstale":"2024-03-23T03:46:57.952188+0000","last_undegraded":"2024-03-23T03:46:57.952188+0000","last_fullsized":"2024-03-23T03:46:57.952188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:30:06.938168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017032+0000","last_change":"2024-03-23T03:46:53.480802+0000","last_active":"2024-03-23T03:46:58.017032+0000","last_peered":"2024-03-23T03:46:58.017032+0000","last_clean":"2024-03-23T03:46:58.017032+0000","last_became_active":"2024-03-23T03:46:53.480500+0000","last_became_peered":"2024-03-23T03:46:53.480500+0000","last_unstale":"2024-03-23T03:46:58.017032+0000","last_undegraded":"2024-03-23T03:46:58.017032+0000","last_fullsized":"2024-03-23T03:46:58.017032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:38:29.605938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.424746+0000","last_change":"2024-03-23T03:46:57.972616+0000","last_active":"2024-03-23T03:46:58.424746+0000","last_peered":"2024-03-23T03:46:58.424746+0000","last_clean":"2024-03-23T03:46:58.424746+0000","last_became_active":"2024-03-23T03:46:57.972028+0000","last_became_peered":"2024-03-23T03:46:57.972028+0000","last_unstale":"2024-03-23T03:46:58.424746+0000","last_undegraded":"2024-03-23T03:46:58.424746+0000","last_fullsized":"2024-03-23T03:46:58.424746+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:55:53.867281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952016+0000","last_change":"2024-03-23T03:46:53.960010+0000","last_active":"2024-03-23T03:46:57.952016+0000","last_peered":"2024-03-23T03:46:57.952016+0000","last_clean":"2024-03-23T03:46:57.952016+0000","last_became_active":"2024-03-23T03:46:53.959761+0000","last_became_peered":"2024-03-23T03:46:53.959761+0000","last_unstale":"2024-03-23T03:46:57.952016+0000","last_undegraded":"2024-03-23T03:46:57.952016+0000","last_fullsized":"2024-03-23T03:46:57.952016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:49.329280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.946127+0000","last_change":"2024-03-23T03:46:53.460246+0000","last_active":"2024-03-23T03:46:57.946127+0000","last_peered":"2024-03-23T03:46:57.946127+0000","last_clean":"2024-03-23T03:46:57.946127+0000","last_became_active":"2024-03-23T03:46:53.459900+0000","last_became_peered":"2024-03-23T03:46:53.459900+0000","last_unstale":"2024-03-23T03:46:57.946127+0000","last_undegraded":"2024-03-23T03:46:57.946127+0000","last_fullsized":"2024-03-23T03:46:57.946127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:23:34.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.424823+0000","last_change":"2024-03-23T03:46:57.976006+0000","last_active":"2024-03-23T03:46:58.424823+0000","last_peered":"2024-03-23T03:46:58.424823+0000","last_clean":"2024-03-23T03:46:58.424823+0000","last_became_active":"2024-03-23T03:46:57.975320+0000","last_became_peered":"2024-03-23T03:46:57.975320+0000","last_unstale":"2024-03-23T03:46:58.424823+0000","last_undegraded":"2024-03-23T03:46:58.424823+0000","last_fullsized":"2024-03-23T03:46:58.424823+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:54.573575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.960391+0000","last_change":"2024-03-23T03:46:53.954675+0000","last_active":"2024-03-23T03:46:57.960391+0000","last_peered":"2024-03-23T03:46:57.960391+0000","last_clean":"2024-03-23T03:46:57.960391+0000","last_became_active":"2024-03-23T03:46:53.954433+0000","last_became_peered":"2024-03-23T03:46:53.954433+0000","last_unstale":"2024-03-23T03:46:57.960391+0000","last_undegraded":"2024-03-23T03:46:57.960391+0000","last_fullsized":"2024-03-23T03:46:57.960391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:27:03.917141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943505+0000","last_change":"2024-03-23T03:46:53.463665+0000","last_active":"2024-03-23T03:46:57.943505+0000","last_peered":"2024-03-23T03:46:57.943505+0000","last_clean":"2024-03-23T03:46:57.943505+0000","last_became_active":"2024-03-23T03:46:53.463379+0000","last_became_peered":"2024-03-23T03:46:53.463379+0000","last_unstale":"2024-03-23T03:46:57.943505+0000","last_undegraded":"2024-03-23T03:46:57.943505+0000","last_fullsized":"2024-03-23T03:46:57.943505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:39:13.578639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545667+0000","last_change":"2024-03-23T03:46:58.017005+0000","last_active":"2024-03-23T03:46:59.545667+0000","last_peered":"2024-03-23T03:46:59.545667+0000","last_clean":"2024-03-23T03:46:59.545667+0000","last_became_active":"2024-03-23T03:46:58.016393+0000","last_became_peered":"2024-03-23T03:46:58.016393+0000","last_unstale":"2024-03-23T03:46:59.545667+0000","last_undegraded":"2024-03-23T03:46:59.545667+0000","last_fullsized":"2024-03-23T03:46:59.545667+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:54:23.785439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.959120+0000","last_change":"2024-03-23T03:46:53.941019+0000","last_active":"2024-03-23T03:46:57.959120+0000","last_peered":"2024-03-23T03:46:57.959120+0000","last_clean":"2024-03-23T03:46:57.959120+0000","last_became_active":"2024-03-23T03:46:53.940265+0000","last_became_peered":"2024-03-23T03:46:53.940265+0000","last_unstale":"2024-03-23T03:46:57.959120+0000","last_undegraded":"2024-03-23T03:46:57.959120+0000","last_fullsized":"2024-03-23T03:46:57.959120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:04:08.642972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.012311+0000","last_change":"2024-03-23T03:46:53.471334+0000","last_active":"2024-03-23T03:46:58.012311+0000","last_peered":"2024-03-23T03:46:58.012311+0000","last_clean":"2024-03-23T03:46:58.012311+0000","last_became_active":"2024-03-23T03:46:53.470996+0000","last_became_peered":"2024-03-23T03:46:53.470996+0000","last_unstale":"2024-03-23T03:46:58.012311+0000","last_undegraded":"2024-03-23T03:46:58.012311+0000","last_fullsized":"2024-03-23T03:46:58.012311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:55.074632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.425416+0000","last_change":"2024-03-23T03:46:57.975950+0000","last_active":"2024-03-23T03:46:58.425416+0000","last_peered":"2024-03-23T03:46:58.425416+0000","last_clean":"2024-03-23T03:46:58.425416+0000","last_became_active":"2024-03-23T03:46:57.975504+0000","last_became_peered":"2024-03-23T03:46:57.975504+0000","last_unstale":"2024-03-23T03:46:58.425416+0000","last_undegraded":"2024-03-23T03:46:58.425416+0000","last_fullsized":"2024-03-23T03:46:58.425416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:21.492278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.955743+0000","last_change":"2024-03-23T03:46:53.957169+0000","last_active":"2024-03-23T03:46:57.955743+0000","last_peered":"2024-03-23T03:46:57.955743+0000","last_clean":"2024-03-23T03:46:57.955743+0000","last_became_active":"2024-03-23T03:46:53.956617+0000","last_became_peered":"2024-03-23T03:46:53.956617+0000","last_unstale":"2024-03-23T03:46:57.955743+0000","last_undegraded":"2024-03-23T03:46:57.955743+0000","last_fullsized":"2024-03-23T03:46:57.955743+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:50:57.246940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.953730+0000","last_change":"2024-03-23T03:46:53.475395+0000","last_active":"2024-03-23T03:46:57.953730+0000","last_peered":"2024-03-23T03:46:57.953730+0000","last_clean":"2024-03-23T03:46:57.953730+0000","last_became_active":"2024-03-23T03:46:53.475130+0000","last_became_peered":"2024-03-23T03:46:53.475130+0000","last_unstale":"2024-03-23T03:46:57.953730+0000","last_undegraded":"2024-03-23T03:46:57.953730+0000","last_fullsized":"2024-03-23T03:46:57.953730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:19:02.377973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.423957+0000","last_change":"2024-03-23T03:46:58.019438+0000","last_active":"2024-03-23T03:46:58.423957+0000","last_peered":"2024-03-23T03:46:58.423957+0000","last_clean":"2024-03-23T03:46:58.423957+0000","last_became_active":"2024-03-23T03:46:58.019018+0000","last_became_peered":"2024-03-23T03:46:58.019018+0000","last_unstale":"2024-03-23T03:46:58.423957+0000","last_undegraded":"2024-03-23T03:46:58.423957+0000","last_fullsized":"2024-03-23T03:46:58.423957+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:53:39.544425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.971959+0000","last_change":"2024-03-23T03:46:53.950103+0000","last_active":"2024-03-23T03:46:57.971959+0000","last_peered":"2024-03-23T03:46:57.971959+0000","last_clean":"2024-03-23T03:46:57.971959+0000","last_became_active":"2024-03-23T03:46:53.947870+0000","last_became_peered":"2024-03-23T03:46:53.947870+0000","last_unstale":"2024-03-23T03:46:57.971959+0000","last_undegraded":"2024-03-23T03:46:57.971959+0000","last_fullsized":"2024-03-23T03:46:57.971959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:56:29.065234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943640+0000","last_change":"2024-03-23T03:46:53.476543+0000","last_active":"2024-03-23T03:46:57.943640+0000","last_peered":"2024-03-23T03:46:57.943640+0000","last_clean":"2024-03-23T03:46:57.943640+0000","last_became_active":"2024-03-23T03:46:53.476227+0000","last_became_peered":"2024-03-23T03:46:53.476227+0000","last_unstale":"2024-03-23T03:46:57.943640+0000","last_undegraded":"2024-03-23T03:46:57.943640+0000","last_fullsized":"2024-03-23T03:46:57.943640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:52:38.737363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.543994+0000","last_change":"2024-03-23T03:46:58.013935+0000","last_active":"2024-03-23T03:46:59.543994+0000","last_peered":"2024-03-23T03:46:59.543994+0000","last_clean":"2024-03-23T03:46:59.543994+0000","last_became_active":"2024-03-23T03:46:58.012631+0000","last_became_peered":"2024-03-23T03:46:58.012631+0000","last_unstale":"2024-03-23T03:46:59.543994+0000","last_undegraded":"2024-03-23T03:46:59.543994+0000","last_fullsized":"2024-03-23T03:46:59.543994+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:42:09.297861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.950778+0000","last_change":"2024-03-23T03:46:53.945184+0000","last_active":"2024-03-23T03:46:57.950778+0000","last_peered":"2024-03-23T03:46:57.950778+0000","last_clean":"2024-03-23T03:46:57.950778+0000","last_became_active":"2024-03-23T03:46:53.943550+0000","last_became_peered":"2024-03-23T03:46:53.943550+0000","last_unstale":"2024-03-23T03:46:57.950778+0000","last_undegraded":"2024-03-23T03:46:57.950778+0000","last_fullsized":"2024-03-23T03:46:57.950778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:01:31.453936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.018003+0000","last_change":"2024-03-23T03:46:53.472577+0000","last_active":"2024-03-23T03:46:58.018003+0000","last_peered":"2024-03-23T03:46:58.018003+0000","last_clean":"2024-03-23T03:46:58.018003+0000","last_became_active":"2024-03-23T03:46:53.472218+0000","last_became_peered":"2024-03-23T03:46:53.472218+0000","last_unstale":"2024-03-23T03:46:58.018003+0000","last_undegraded":"2024-03-23T03:46:58.018003+0000","last_fullsized":"2024-03-23T03:46:58.018003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:21:03.254658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411414+0000","last_change":"2024-03-23T03:46:58.013198+0000","last_active":"2024-03-23T03:46:59.411414+0000","last_peered":"2024-03-23T03:46:59.411414+0000","last_clean":"2024-03-23T03:46:59.411414+0000","last_became_active":"2024-03-23T03:46:58.012474+0000","last_became_peered":"2024-03-23T03:46:58.012474+0000","last_unstale":"2024-03-23T03:46:59.411414+0000","last_undegraded":"2024-03-23T03:46:59.411414+0000","last_fullsized":"2024-03-23T03:46:59.411414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:22:28.017991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970873+0000","last_change":"2024-03-23T03:46:53.955218+0000","last_active":"2024-03-23T03:46:57.970873+0000","last_peered":"2024-03-23T03:46:57.970873+0000","last_clean":"2024-03-23T03:46:57.970873+0000","last_became_active":"2024-03-23T03:46:53.954977+0000","last_became_peered":"2024-03-23T03:46:53.954977+0000","last_unstale":"2024-03-23T03:46:57.970873+0000","last_undegraded":"2024-03-23T03:46:57.970873+0000","last_fullsized":"2024-03-23T03:46:57.970873+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:12.278522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.947128+0000","last_change":"2024-03-23T03:46:53.456663+0000","last_active":"2024-03-23T03:46:57.947128+0000","last_peered":"2024-03-23T03:46:57.947128+0000","last_clean":"2024-03-23T03:46:57.947128+0000","last_became_active":"2024-03-23T03:46:53.456393+0000","last_became_peered":"2024-03-23T03:46:53.456393+0000","last_unstale":"2024-03-23T03:46:57.947128+0000","last_undegraded":"2024-03-23T03:46:57.947128+0000","last_fullsized":"2024-03-23T03:46:57.947128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:38:29.605938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.544795+0000","last_change":"2024-03-23T03:46:57.975493+0000","last_active":"2024-03-23T03:46:59.544795+0000","last_peered":"2024-03-23T03:46:59.544795+0000","last_clean":"2024-03-23T03:46:59.544795+0000","last_became_active":"2024-03-23T03:46:57.974943+0000","last_became_peered":"2024-03-23T03:46:57.974943+0000","last_unstale":"2024-03-23T03:46:59.544795+0000","last_undegraded":"2024-03-23T03:46:59.544795+0000","last_fullsized":"2024-03-23T03:46:59.544795+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:56.623233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.960709+0000","last_change":"2024-03-23T03:46:53.963022+0000","last_active":"2024-03-23T03:46:57.960709+0000","last_peered":"2024-03-23T03:46:57.960709+0000","last_clean":"2024-03-23T03:46:57.960709+0000","last_became_active":"2024-03-23T03:46:53.962270+0000","last_became_peered":"2024-03-23T03:46:53.962270+0000","last_unstale":"2024-03-23T03:46:57.960709+0000","last_undegraded":"2024-03-23T03:46:57.960709+0000","last_fullsized":"2024-03-23T03:46:57.960709+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:55:23.986885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970281+0000","last_change":"2024-03-23T03:46:53.456217+0000","last_active":"2024-03-23T03:46:57.970281+0000","last_peered":"2024-03-23T03:46:57.970281+0000","last_clean":"2024-03-23T03:46:57.970281+0000","last_became_active":"2024-03-23T03:46:53.455825+0000","last_became_peered":"2024-03-23T03:46:53.455825+0000","last_unstale":"2024-03-23T03:46:57.970281+0000","last_undegraded":"2024-03-23T03:46:57.970281+0000","last_fullsized":"2024-03-23T03:46:57.970281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:20:09.953919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.544875+0000","last_change":"2024-03-23T03:46:57.978215+0000","last_active":"2024-03-23T03:46:59.544875+0000","last_peered":"2024-03-23T03:46:59.544875+0000","last_clean":"2024-03-23T03:46:59.544875+0000","last_became_active":"2024-03-23T03:46:57.977737+0000","last_became_peered":"2024-03-23T03:46:57.977737+0000","last_unstale":"2024-03-23T03:46:59.544875+0000","last_undegraded":"2024-03-23T03:46:59.544875+0000","last_fullsized":"2024-03-23T03:46:59.544875+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:09:10.919635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943234+0000","last_change":"2024-03-23T03:46:53.949224+0000","last_active":"2024-03-23T03:46:57.943234+0000","last_peered":"2024-03-23T03:46:57.943234+0000","last_clean":"2024-03-23T03:46:57.943234+0000","last_became_active":"2024-03-23T03:46:53.948869+0000","last_became_peered":"2024-03-23T03:46:53.948869+0000","last_unstale":"2024-03-23T03:46:57.943234+0000","last_undegraded":"2024-03-23T03:46:57.943234+0000","last_fullsized":"2024-03-23T03:46:57.943234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:49:34.782399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.948932+0000","last_change":"2024-03-23T03:46:53.461930+0000","last_active":"2024-03-23T03:46:57.948932+0000","last_peered":"2024-03-23T03:46:57.948932+0000","last_clean":"2024-03-23T03:46:57.948932+0000","last_became_active":"2024-03-23T03:46:53.461500+0000","last_became_peered":"2024-03-23T03:46:53.461500+0000","last_unstale":"2024-03-23T03:46:57.948932+0000","last_undegraded":"2024-03-23T03:46:57.948932+0000","last_fullsized":"2024-03-23T03:46:57.948932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:12:50.664977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545763+0000","last_change":"2024-03-23T03:46:57.978014+0000","last_active":"2024-03-23T03:46:59.545763+0000","last_peered":"2024-03-23T03:46:59.545763+0000","last_clean":"2024-03-23T03:46:59.545763+0000","last_became_active":"2024-03-23T03:46:57.977584+0000","last_became_peered":"2024-03-23T03:46:57.977584+0000","last_unstale":"2024-03-23T03:46:59.545763+0000","last_undegraded":"2024-03-23T03:46:59.545763+0000","last_fullsized":"2024-03-23T03:46:59.545763+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:17:26.245417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.950704+0000","last_change":"2024-03-23T03:46:53.968396+0000","last_active":"2024-03-23T03:46:57.950704+0000","last_peered":"2024-03-23T03:46:57.950704+0000","last_clean":"2024-03-23T03:46:57.950704+0000","last_became_active":"2024-03-23T03:46:53.968127+0000","last_became_peered":"2024-03-23T03:46:53.968127+0000","last_unstale":"2024-03-23T03:46:57.950704+0000","last_undegraded":"2024-03-23T03:46:57.950704+0000","last_fullsized":"2024-03-23T03:46:57.950704+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:56:29.065234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.012423+0000","last_change":"2024-03-23T03:46:53.472587+0000","last_active":"2024-03-23T03:46:58.012423+0000","last_peered":"2024-03-23T03:46:58.012423+0000","last_clean":"2024-03-23T03:46:58.012423+0000","last_became_active":"2024-03-23T03:46:53.472235+0000","last_became_peered":"2024-03-23T03:46:53.472235+0000","last_unstale":"2024-03-23T03:46:58.012423+0000","last_undegraded":"2024-03-23T03:46:58.012423+0000","last_fullsized":"2024-03-23T03:46:58.012423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:56.874824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411831+0000","last_change":"2024-03-23T03:46:58.016199+0000","last_active":"2024-03-23T03:46:59.411831+0000","last_peered":"2024-03-23T03:46:59.411831+0000","last_clean":"2024-03-23T03:46:59.411831+0000","last_became_active":"2024-03-23T03:46:58.015478+0000","last_became_peered":"2024-03-23T03:46:58.015478+0000","last_unstale":"2024-03-23T03:46:59.411831+0000","last_undegraded":"2024-03-23T03:46:59.411831+0000","last_fullsized":"2024-03-23T03:46:59.411831+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:56.623233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951639+0000","last_change":"2024-03-23T03:46:53.955152+0000","last_active":"2024-03-23T03:46:57.951639+0000","last_peered":"2024-03-23T03:46:57.951639+0000","last_clean":"2024-03-23T03:46:57.951639+0000","last_became_active":"2024-03-23T03:46:53.953527+0000","last_became_peered":"2024-03-23T03:46:53.953527+0000","last_unstale":"2024-03-23T03:46:57.951639+0000","last_undegraded":"2024-03-23T03:46:57.951639+0000","last_fullsized":"2024-03-23T03:46:57.951639+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:01:31.453936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970187+0000","last_change":"2024-03-23T03:46:53.462993+0000","last_active":"2024-03-23T03:46:57.970187+0000","last_peered":"2024-03-23T03:46:57.970187+0000","last_clean":"2024-03-23T03:46:57.970187+0000","last_became_active":"2024-03-23T03:46:53.462438+0000","last_became_peered":"2024-03-23T03:46:53.462438+0000","last_unstale":"2024-03-23T03:46:57.970187+0000","last_undegraded":"2024-03-23T03:46:57.970187+0000","last_fullsized":"2024-03-23T03:46:57.970187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:52:38.737363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":167,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.012529+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:46:58.012529+0000","last_peered":"2024-03-23T03:46:58.012529+0000","last_clean":"2024-03-23T03:46:58.012529+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:46:58.012529+0000","last_undegraded":"2024-03-23T03:46:58.012529+0000","last_fullsized":"2024-03-23T03:46:58.012529+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.547057+0000","last_change":"2024-03-23T03:46:58.018456+0000","last_active":"2024-03-23T03:46:59.547057+0000","last_peered":"2024-03-23T03:46:59.547057+0000","last_clean":"2024-03-23T03:46:59.547057+0000","last_became_active":"2024-03-23T03:46:58.018035+0000","last_became_peered":"2024-03-23T03:46:58.018035+0000","last_unstale":"2024-03-23T03:46:59.547057+0000","last_undegraded":"2024-03-23T03:46:59.547057+0000","last_fullsized":"2024-03-23T03:46:59.547057+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:44:17.122381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017645+0000","last_change":"2024-03-23T03:46:53.970202+0000","last_active":"2024-03-23T03:46:58.017645+0000","last_peered":"2024-03-23T03:46:58.017645+0000","last_clean":"2024-03-23T03:46:58.017645+0000","last_became_active":"2024-03-23T03:46:53.969976+0000","last_became_peered":"2024-03-23T03:46:53.969976+0000","last_unstale":"2024-03-23T03:46:58.017645+0000","last_undegraded":"2024-03-23T03:46:58.017645+0000","last_fullsized":"2024-03-23T03:46:58.017645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:59.541399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.947920+0000","last_change":"2024-03-23T03:46:53.478374+0000","last_active":"2024-03-23T03:46:57.947920+0000","last_peered":"2024-03-23T03:46:57.947920+0000","last_clean":"2024-03-23T03:46:57.947920+0000","last_became_active":"2024-03-23T03:46:53.478063+0000","last_became_peered":"2024-03-23T03:46:53.478063+0000","last_unstale":"2024-03-23T03:46:57.947920+0000","last_undegraded":"2024-03-23T03:46:57.947920+0000","last_fullsized":"2024-03-23T03:46:57.947920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:28:03.413525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.546398+0000","last_change":"2024-03-23T03:46:58.013147+0000","last_active":"2024-03-23T03:46:59.546398+0000","last_peered":"2024-03-23T03:46:59.546398+0000","last_clean":"2024-03-23T03:46:59.546398+0000","last_became_active":"2024-03-23T03:46:58.012728+0000","last_became_peered":"2024-03-23T03:46:58.012728+0000","last_unstale":"2024-03-23T03:46:59.546398+0000","last_undegraded":"2024-03-23T03:46:59.546398+0000","last_fullsized":"2024-03-23T03:46:59.546398+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:28:49.474199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970654+0000","last_change":"2024-03-23T03:46:53.956675+0000","last_active":"2024-03-23T03:46:57.970654+0000","last_peered":"2024-03-23T03:46:57.970654+0000","last_clean":"2024-03-23T03:46:57.970654+0000","last_became_active":"2024-03-23T03:46:53.956420+0000","last_became_peered":"2024-03-23T03:46:53.956420+0000","last_unstale":"2024-03-23T03:46:57.970654+0000","last_undegraded":"2024-03-23T03:46:57.970654+0000","last_fullsized":"2024-03-23T03:46:57.970654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:16:39.131183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943199+0000","last_change":"2024-03-23T03:46:53.467369+0000","last_active":"2024-03-23T03:46:57.943199+0000","last_peered":"2024-03-23T03:46:57.943199+0000","last_clean":"2024-03-23T03:46:57.943199+0000","last_became_active":"2024-03-23T03:46:53.465916+0000","last_became_peered":"2024-03-23T03:46:53.465916+0000","last_unstale":"2024-03-23T03:46:57.943199+0000","last_undegraded":"2024-03-23T03:46:57.943199+0000","last_fullsized":"2024-03-23T03:46:57.943199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:56.874824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411552+0000","last_change":"2024-03-23T03:46:57.974919+0000","last_active":"2024-03-23T03:46:59.411552+0000","last_peered":"2024-03-23T03:46:59.411552+0000","last_clean":"2024-03-23T03:46:59.411552+0000","last_became_active":"2024-03-23T03:46:57.972661+0000","last_became_peered":"2024-03-23T03:46:57.972661+0000","last_unstale":"2024-03-23T03:46:59.411552+0000","last_undegraded":"2024-03-23T03:46:59.411552+0000","last_fullsized":"2024-03-23T03:46:59.411552+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:27:37.992019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.018541+0000","last_change":"2024-03-23T03:46:53.965749+0000","last_active":"2024-03-23T03:46:58.018541+0000","last_peered":"2024-03-23T03:46:58.018541+0000","last_clean":"2024-03-23T03:46:58.018541+0000","last_became_active":"2024-03-23T03:46:53.965362+0000","last_became_peered":"2024-03-23T03:46:53.965362+0000","last_unstale":"2024-03-23T03:46:58.018541+0000","last_undegraded":"2024-03-23T03:46:58.018541+0000","last_fullsized":"2024-03-23T03:46:58.018541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:38:13.205855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.948005+0000","last_change":"2024-03-23T03:46:53.460818+0000","last_active":"2024-03-23T03:46:57.948005+0000","last_peered":"2024-03-23T03:46:57.948005+0000","last_clean":"2024-03-23T03:46:57.948005+0000","last_became_active":"2024-03-23T03:46:53.460547+0000","last_became_peered":"2024-03-23T03:46:53.460547+0000","last_unstale":"2024-03-23T03:46:57.948005+0000","last_undegraded":"2024-03-23T03:46:57.948005+0000","last_fullsized":"2024-03-23T03:46:57.948005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:49:24.952855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.424397+0000","last_change":"2024-03-23T03:46:58.014145+0000","last_active":"2024-03-23T03:46:58.424397+0000","last_peered":"2024-03-23T03:46:58.424397+0000","last_clean":"2024-03-23T03:46:58.424397+0000","last_became_active":"2024-03-23T03:46:58.013649+0000","last_became_peered":"2024-03-23T03:46:58.013649+0000","last_unstale":"2024-03-23T03:46:58.424397+0000","last_undegraded":"2024-03-23T03:46:58.424397+0000","last_fullsized":"2024-03-23T03:46:58.424397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:56.623233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017437+0000","last_change":"2024-03-23T03:46:53.966787+0000","last_active":"2024-03-23T03:46:58.017437+0000","last_peered":"2024-03-23T03:46:58.017437+0000","last_clean":"2024-03-23T03:46:58.017437+0000","last_became_active":"2024-03-23T03:46:53.966135+0000","last_became_peered":"2024-03-23T03:46:53.966135+0000","last_unstale":"2024-03-23T03:46:58.017437+0000","last_undegraded":"2024-03-23T03:46:58.017437+0000","last_fullsized":"2024-03-23T03:46:58.017437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:12.278522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.964838+0000","last_change":"2024-03-23T03:46:53.473134+0000","last_active":"2024-03-23T03:46:57.964838+0000","last_peered":"2024-03-23T03:46:57.964838+0000","last_clean":"2024-03-23T03:46:57.964838+0000","last_became_active":"2024-03-23T03:46:53.472902+0000","last_became_peered":"2024-03-23T03:46:53.472902+0000","last_unstale":"2024-03-23T03:46:57.964838+0000","last_undegraded":"2024-03-23T03:46:57.964838+0000","last_fullsized":"2024-03-23T03:46:57.964838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:42:16.552281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.423299+0000","last_change":"2024-03-23T03:46:57.976918+0000","last_active":"2024-03-23T03:46:58.423299+0000","last_peered":"2024-03-23T03:46:58.423299+0000","last_clean":"2024-03-23T03:46:58.423299+0000","last_became_active":"2024-03-23T03:46:57.976542+0000","last_became_peered":"2024-03-23T03:46:57.976542+0000","last_unstale":"2024-03-23T03:46:58.423299+0000","last_undegraded":"2024-03-23T03:46:58.423299+0000","last_fullsized":"2024-03-23T03:46:58.423299+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:49:33.207155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017030+0000","last_change":"2024-03-23T03:46:53.964676+0000","last_active":"2024-03-23T03:46:58.017030+0000","last_peered":"2024-03-23T03:46:58.017030+0000","last_clean":"2024-03-23T03:46:58.017030+0000","last_became_active":"2024-03-23T03:46:53.964382+0000","last_became_peered":"2024-03-23T03:46:53.964382+0000","last_unstale":"2024-03-23T03:46:58.017030+0000","last_undegraded":"2024-03-23T03:46:58.017030+0000","last_fullsized":"2024-03-23T03:46:58.017030+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:34:41.509208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.947240+0000","last_change":"2024-03-23T03:46:53.459242+0000","last_active":"2024-03-23T03:46:57.947240+0000","last_peered":"2024-03-23T03:46:57.947240+0000","last_clean":"2024-03-23T03:46:57.947240+0000","last_became_active":"2024-03-23T03:46:53.458136+0000","last_became_peered":"2024-03-23T03:46:53.458136+0000","last_unstale":"2024-03-23T03:46:57.947240+0000","last_undegraded":"2024-03-23T03:46:57.947240+0000","last_fullsized":"2024-03-23T03:46:57.947240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:42:13.208524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411874+0000","last_change":"2024-03-23T03:46:57.968727+0000","last_active":"2024-03-23T03:46:59.411874+0000","last_peered":"2024-03-23T03:46:59.411874+0000","last_clean":"2024-03-23T03:46:59.411874+0000","last_became_active":"2024-03-23T03:46:57.968336+0000","last_became_peered":"2024-03-23T03:46:57.968336+0000","last_unstale":"2024-03-23T03:46:59.411874+0000","last_undegraded":"2024-03-23T03:46:59.411874+0000","last_fullsized":"2024-03-23T03:46:59.411874+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:17:26.245417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951907+0000","last_change":"2024-03-23T03:46:53.967303+0000","last_active":"2024-03-23T03:46:57.951907+0000","last_peered":"2024-03-23T03:46:57.951907+0000","last_clean":"2024-03-23T03:46:57.951907+0000","last_became_active":"2024-03-23T03:46:53.966903+0000","last_became_peered":"2024-03-23T03:46:53.966903+0000","last_unstale":"2024-03-23T03:46:57.951907+0000","last_undegraded":"2024-03-23T03:46:57.951907+0000","last_fullsized":"2024-03-23T03:46:57.951907+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:18:17.089475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951906+0000","last_change":"2024-03-23T03:46:53.479049+0000","last_active":"2024-03-23T03:46:57.951906+0000","last_peered":"2024-03-23T03:46:57.951906+0000","last_clean":"2024-03-23T03:46:57.951906+0000","last_became_active":"2024-03-23T03:46:53.478779+0000","last_became_peered":"2024-03-23T03:46:53.478779+0000","last_unstale":"2024-03-23T03:46:57.951906+0000","last_undegraded":"2024-03-23T03:46:57.951906+0000","last_fullsized":"2024-03-23T03:46:57.951906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:28:03.413525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.547121+0000","last_change":"2024-03-23T03:46:58.018779+0000","last_active":"2024-03-23T03:46:59.547121+0000","last_peered":"2024-03-23T03:46:59.547121+0000","last_clean":"2024-03-23T03:46:59.547121+0000","last_became_active":"2024-03-23T03:46:58.018342+0000","last_became_peered":"2024-03-23T03:46:58.018342+0000","last_unstale":"2024-03-23T03:46:59.547121+0000","last_undegraded":"2024-03-23T03:46:59.547121+0000","last_fullsized":"2024-03-23T03:46:59.547121+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:49:30.580961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.016869+0000","last_change":"2024-03-23T03:46:53.966087+0000","last_active":"2024-03-23T03:46:58.016869+0000","last_peered":"2024-03-23T03:46:58.016869+0000","last_clean":"2024-03-23T03:46:58.016869+0000","last_became_active":"2024-03-23T03:46:53.965556+0000","last_became_peered":"2024-03-23T03:46:53.965556+0000","last_unstale":"2024-03-23T03:46:58.016869+0000","last_undegraded":"2024-03-23T03:46:58.016869+0000","last_fullsized":"2024-03-23T03:46:58.016869+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:35:49.855085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970252+0000","last_change":"2024-03-23T03:46:53.468499+0000","last_active":"2024-03-23T03:46:57.970252+0000","last_peered":"2024-03-23T03:46:57.970252+0000","last_clean":"2024-03-23T03:46:57.970252+0000","last_became_active":"2024-03-23T03:46:53.466565+0000","last_became_peered":"2024-03-23T03:46:53.466565+0000","last_unstale":"2024-03-23T03:46:57.970252+0000","last_undegraded":"2024-03-23T03:46:57.970252+0000","last_fullsized":"2024-03-23T03:46:57.970252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:38:29.605938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.546719+0000","last_change":"2024-03-23T03:46:58.013721+0000","last_active":"2024-03-23T03:46:59.546719+0000","last_peered":"2024-03-23T03:46:59.546719+0000","last_clean":"2024-03-23T03:46:59.546719+0000","last_became_active":"2024-03-23T03:46:58.012503+0000","last_became_peered":"2024-03-23T03:46:58.012503+0000","last_unstale":"2024-03-23T03:46:59.546719+0000","last_undegraded":"2024-03-23T03:46:59.546719+0000","last_fullsized":"2024-03-23T03:46:59.546719+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:54:55.461256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951186+0000","last_change":"2024-03-23T03:46:53.952982+0000","last_active":"2024-03-23T03:46:57.951186+0000","last_peered":"2024-03-23T03:46:57.951186+0000","last_clean":"2024-03-23T03:46:57.951186+0000","last_became_active":"2024-03-23T03:46:53.952707+0000","last_became_peered":"2024-03-23T03:46:53.952707+0000","last_unstale":"2024-03-23T03:46:57.951186+0000","last_undegraded":"2024-03-23T03:46:57.951186+0000","last_fullsized":"2024-03-23T03:46:57.951186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:01:24.719476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952254+0000","last_change":"2024-03-23T03:46:53.472959+0000","last_active":"2024-03-23T03:46:57.952254+0000","last_peered":"2024-03-23T03:46:57.952254+0000","last_clean":"2024-03-23T03:46:57.952254+0000","last_became_active":"2024-03-23T03:46:53.472652+0000","last_became_peered":"2024-03-23T03:46:53.472652+0000","last_unstale":"2024-03-23T03:46:57.952254+0000","last_undegraded":"2024-03-23T03:46:57.952254+0000","last_fullsized":"2024-03-23T03:46:57.952254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:48:00.747717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.426089+0000","last_change":"2024-03-23T03:46:58.019621+0000","last_active":"2024-03-23T03:46:58.426089+0000","last_peered":"2024-03-23T03:46:58.426089+0000","last_clean":"2024-03-23T03:46:58.426089+0000","last_became_active":"2024-03-23T03:46:58.018726+0000","last_became_peered":"2024-03-23T03:46:58.018726+0000","last_unstale":"2024-03-23T03:46:58.426089+0000","last_undegraded":"2024-03-23T03:46:58.426089+0000","last_fullsized":"2024-03-23T03:46:58.426089+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:18:24.184664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017563+0000","last_change":"2024-03-23T03:46:53.954977+0000","last_active":"2024-03-23T03:46:58.017563+0000","last_peered":"2024-03-23T03:46:58.017563+0000","last_clean":"2024-03-23T03:46:58.017563+0000","last_became_active":"2024-03-23T03:46:53.954667+0000","last_became_peered":"2024-03-23T03:46:53.954667+0000","last_unstale":"2024-03-23T03:46:58.017563+0000","last_undegraded":"2024-03-23T03:46:58.017563+0000","last_fullsized":"2024-03-23T03:46:58.017563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:16.232161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'10","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:47:12.152821+0000","last_change":"2024-03-23T03:46:53.469497+0000","last_active":"2024-03-23T03:47:12.152821+0000","last_peered":"2024-03-23T03:47:12.152821+0000","last_clean":"2024-03-23T03:47:12.152821+0000","last_became_active":"2024-03-23T03:46:53.469139+0000","last_became_peered":"2024-03-23T03:46:53.469139+0000","last_unstale":"2024-03-23T03:47:12.152821+0000","last_undegraded":"2024-03-23T03:47:12.152821+0000","last_fullsized":"2024-03-23T03:47:12.152821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:42:13.208524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3795,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545062+0000","last_change":"2024-03-23T03:46:57.965959+0000","last_active":"2024-03-23T03:46:59.545062+0000","last_peered":"2024-03-23T03:46:59.545062+0000","last_clean":"2024-03-23T03:46:59.545062+0000","last_became_active":"2024-03-23T03:46:57.965418+0000","last_became_peered":"2024-03-23T03:46:57.965418+0000","last_unstale":"2024-03-23T03:46:59.545062+0000","last_undegraded":"2024-03-23T03:46:59.545062+0000","last_fullsized":"2024-03-23T03:46:59.545062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:57:19.038735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.962037+0000","last_change":"2024-03-23T03:46:53.946234+0000","last_active":"2024-03-23T03:46:57.962037+0000","last_peered":"2024-03-23T03:46:57.962037+0000","last_clean":"2024-03-23T03:46:57.962037+0000","last_became_active":"2024-03-23T03:46:53.945188+0000","last_became_peered":"2024-03-23T03:46:53.945188+0000","last_unstale":"2024-03-23T03:46:57.962037+0000","last_undegraded":"2024-03-23T03:46:57.962037+0000","last_fullsized":"2024-03-23T03:46:57.962037+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:29:34.917375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943105+0000","last_change":"2024-03-23T03:46:53.474846+0000","last_active":"2024-03-23T03:46:57.943105+0000","last_peered":"2024-03-23T03:46:57.943105+0000","last_clean":"2024-03-23T03:46:57.943105+0000","last_became_active":"2024-03-23T03:46:53.474561+0000","last_became_peered":"2024-03-23T03:46:53.474561+0000","last_unstale":"2024-03-23T03:46:57.943105+0000","last_undegraded":"2024-03-23T03:46:57.943105+0000","last_fullsized":"2024-03-23T03:46:57.943105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:40:20.814830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.424404+0000","last_change":"2024-03-23T03:46:57.974187+0000","last_active":"2024-03-23T03:46:58.424404+0000","last_peered":"2024-03-23T03:46:58.424404+0000","last_clean":"2024-03-23T03:46:58.424404+0000","last_became_active":"2024-03-23T03:46:57.973629+0000","last_became_peered":"2024-03-23T03:46:57.973629+0000","last_unstale":"2024-03-23T03:46:58.424404+0000","last_undegraded":"2024-03-23T03:46:58.424404+0000","last_fullsized":"2024-03-23T03:46:58.424404+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:32:46.731709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.954425+0000","last_change":"2024-03-23T03:46:53.958810+0000","last_active":"2024-03-23T03:46:57.954425+0000","last_peered":"2024-03-23T03:46:57.954425+0000","last_clean":"2024-03-23T03:46:57.954425+0000","last_became_active":"2024-03-23T03:46:53.958534+0000","last_became_peered":"2024-03-23T03:46:53.958534+0000","last_unstale":"2024-03-23T03:46:57.954425+0000","last_undegraded":"2024-03-23T03:46:57.954425+0000","last_fullsized":"2024-03-23T03:46:57.954425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:01:31.453936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970435+0000","last_change":"2024-03-23T03:46:53.472164+0000","last_active":"2024-03-23T03:46:57.970435+0000","last_peered":"2024-03-23T03:46:57.970435+0000","last_clean":"2024-03-23T03:46:57.970435+0000","last_became_active":"2024-03-23T03:46:53.471488+0000","last_became_peered":"2024-03-23T03:46:53.471488+0000","last_unstale":"2024-03-23T03:46:57.970435+0000","last_undegraded":"2024-03-23T03:46:57.970435+0000","last_fullsized":"2024-03-23T03:46:57.970435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:56.874824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.544253+0000","last_change":"2024-03-23T03:46:57.967928+0000","last_active":"2024-03-23T03:46:59.544253+0000","last_peered":"2024-03-23T03:46:59.544253+0000","last_clean":"2024-03-23T03:46:59.544253+0000","last_became_active":"2024-03-23T03:46:57.967377+0000","last_became_peered":"2024-03-23T03:46:57.967377+0000","last_unstale":"2024-03-23T03:46:59.544253+0000","last_undegraded":"2024-03-23T03:46:59.544253+0000","last_fullsized":"2024-03-23T03:46:59.544253+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:58:12.948095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970395+0000","last_change":"2024-03-23T03:46:53.954230+0000","last_active":"2024-03-23T03:46:57.970395+0000","last_peered":"2024-03-23T03:46:57.970395+0000","last_clean":"2024-03-23T03:46:57.970395+0000","last_became_active":"2024-03-23T03:46:53.953991+0000","last_became_peered":"2024-03-23T03:46:53.953991+0000","last_unstale":"2024-03-23T03:46:57.970395+0000","last_undegraded":"2024-03-23T03:46:57.970395+0000","last_fullsized":"2024-03-23T03:46:57.970395+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:16.232161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.953457+0000","last_change":"2024-03-23T03:46:53.475516+0000","last_active":"2024-03-23T03:46:57.953457+0000","last_peered":"2024-03-23T03:46:57.953457+0000","last_clean":"2024-03-23T03:46:57.953457+0000","last_became_active":"2024-03-23T03:46:53.475185+0000","last_became_peered":"2024-03-23T03:46:53.475185+0000","last_unstale":"2024-03-23T03:46:57.953457+0000","last_undegraded":"2024-03-23T03:46:57.953457+0000","last_fullsized":"2024-03-23T03:46:57.953457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:05:58.548539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545880+0000","last_change":"2024-03-23T03:46:58.018887+0000","last_active":"2024-03-23T03:46:59.545880+0000","last_peered":"2024-03-23T03:46:59.545880+0000","last_clean":"2024-03-23T03:46:59.545880+0000","last_became_active":"2024-03-23T03:46:58.018514+0000","last_became_peered":"2024-03-23T03:46:58.018514+0000","last_unstale":"2024-03-23T03:46:59.545880+0000","last_undegraded":"2024-03-23T03:46:59.545880+0000","last_fullsized":"2024-03-23T03:46:59.545880+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:51:12.276163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411517+0000","last_change":"2024-03-23T03:46:58.015847+0000","last_active":"2024-03-23T03:46:59.411517+0000","last_peered":"2024-03-23T03:46:59.411517+0000","last_clean":"2024-03-23T03:46:59.411517+0000","last_became_active":"2024-03-23T03:46:58.015471+0000","last_became_peered":"2024-03-23T03:46:58.015471+0000","last_unstale":"2024-03-23T03:46:59.411517+0000","last_undegraded":"2024-03-23T03:46:59.411517+0000","last_fullsized":"2024-03-23T03:46:59.411517+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:34:34.288987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943156+0000","last_change":"2024-03-23T03:46:53.952670+0000","last_active":"2024-03-23T03:46:57.943156+0000","last_peered":"2024-03-23T03:46:57.943156+0000","last_clean":"2024-03-23T03:46:57.943156+0000","last_became_active":"2024-03-23T03:46:53.951520+0000","last_became_peered":"2024-03-23T03:46:53.951520+0000","last_unstale":"2024-03-23T03:46:57.943156+0000","last_undegraded":"2024-03-23T03:46:57.943156+0000","last_fullsized":"2024-03-23T03:46:57.943156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:55.749189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.968564+0000","last_change":"2024-03-23T03:46:53.475560+0000","last_active":"2024-03-23T03:46:57.968564+0000","last_peered":"2024-03-23T03:46:57.968564+0000","last_clean":"2024-03-23T03:46:57.968564+0000","last_became_active":"2024-03-23T03:46:53.468090+0000","last_became_peered":"2024-03-23T03:46:53.468090+0000","last_unstale":"2024-03-23T03:46:57.968564+0000","last_undegraded":"2024-03-23T03:46:57.968564+0000","last_fullsized":"2024-03-23T03:46:57.968564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:23:34.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.548770+0000","last_change":"2024-03-23T03:46:57.974834+0000","last_active":"2024-03-23T03:46:59.548770+0000","last_peered":"2024-03-23T03:46:59.548770+0000","last_clean":"2024-03-23T03:46:59.548770+0000","last_became_active":"2024-03-23T03:46:57.974450+0000","last_became_peered":"2024-03-23T03:46:57.974450+0000","last_unstale":"2024-03-23T03:46:59.548770+0000","last_undegraded":"2024-03-23T03:46:59.548770+0000","last_fullsized":"2024-03-23T03:46:59.548770+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:01.676599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.954004+0000","last_change":"2024-03-23T03:46:53.963896+0000","last_active":"2024-03-23T03:46:57.954004+0000","last_peered":"2024-03-23T03:46:57.954004+0000","last_clean":"2024-03-23T03:46:57.954004+0000","last_became_active":"2024-03-23T03:46:53.963072+0000","last_became_peered":"2024-03-23T03:46:53.963072+0000","last_unstale":"2024-03-23T03:46:57.954004+0000","last_undegraded":"2024-03-23T03:46:57.954004+0000","last_fullsized":"2024-03-23T03:46:57.954004+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:49:34.782399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.961727+0000","last_change":"2024-03-23T03:46:53.475321+0000","last_active":"2024-03-23T03:46:57.961727+0000","last_peered":"2024-03-23T03:46:57.961727+0000","last_clean":"2024-03-23T03:46:57.961727+0000","last_became_active":"2024-03-23T03:46:53.475119+0000","last_became_peered":"2024-03-23T03:46:53.475119+0000","last_unstale":"2024-03-23T03:46:57.961727+0000","last_undegraded":"2024-03-23T03:46:57.961727+0000","last_fullsized":"2024-03-23T03:46:57.961727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:48:00.747717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952537+0000","last_change":"2024-03-23T03:46:53.475209+0000","last_active":"2024-03-23T03:46:57.952537+0000","last_peered":"2024-03-23T03:46:57.952537+0000","last_clean":"2024-03-23T03:46:57.952537+0000","last_became_active":"2024-03-23T03:46:53.474046+0000","last_became_peered":"2024-03-23T03:46:53.474046+0000","last_unstale":"2024-03-23T03:46:57.952537+0000","last_undegraded":"2024-03-23T03:46:57.952537+0000","last_fullsized":"2024-03-23T03:46:57.952537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:04:07.631198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952167+0000","last_change":"2024-03-23T03:46:53.961251+0000","last_active":"2024-03-23T03:46:57.952167+0000","last_peered":"2024-03-23T03:46:57.952167+0000","last_clean":"2024-03-23T03:46:57.952167+0000","last_became_active":"2024-03-23T03:46:53.960887+0000","last_became_peered":"2024-03-23T03:46:53.960887+0000","last_unstale":"2024-03-23T03:46:57.952167+0000","last_undegraded":"2024-03-23T03:46:57.952167+0000","last_fullsized":"2024-03-23T03:46:57.952167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:51.878165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545740+0000","last_change":"2024-03-23T03:46:57.969555+0000","last_active":"2024-03-23T03:46:59.545740+0000","last_peered":"2024-03-23T03:46:59.545740+0000","last_clean":"2024-03-23T03:46:59.545740+0000","last_became_active":"2024-03-23T03:46:57.969163+0000","last_became_peered":"2024-03-23T03:46:57.969163+0000","last_unstale":"2024-03-23T03:46:59.545740+0000","last_undegraded":"2024-03-23T03:46:59.545740+0000","last_fullsized":"2024-03-23T03:46:59.545740+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:09.677468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952350+0000","last_change":"2024-03-23T03:46:53.463135+0000","last_active":"2024-03-23T03:46:57.952350+0000","last_peered":"2024-03-23T03:46:57.952350+0000","last_clean":"2024-03-23T03:46:57.952350+0000","last_became_active":"2024-03-23T03:46:53.462844+0000","last_became_peered":"2024-03-23T03:46:53.462844+0000","last_unstale":"2024-03-23T03:46:57.952350+0000","last_undegraded":"2024-03-23T03:46:57.952350+0000","last_fullsized":"2024-03-23T03:46:57.952350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:39:13.578639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.960303+0000","last_change":"2024-03-23T03:46:53.935183+0000","last_active":"2024-03-23T03:46:57.960303+0000","last_peered":"2024-03-23T03:46:57.960303+0000","last_clean":"2024-03-23T03:46:57.960303+0000","last_became_active":"2024-03-23T03:46:53.934902+0000","last_became_peered":"2024-03-23T03:46:53.934902+0000","last_unstale":"2024-03-23T03:46:57.960303+0000","last_undegraded":"2024-03-23T03:46:57.960303+0000","last_fullsized":"2024-03-23T03:46:57.960303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:19:03.389747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.546902+0000","last_change":"2024-03-23T03:46:57.971524+0000","last_active":"2024-03-23T03:46:59.546902+0000","last_peered":"2024-03-23T03:46:59.546902+0000","last_clean":"2024-03-23T03:46:59.546902+0000","last_became_active":"2024-03-23T03:46:57.967540+0000","last_became_peered":"2024-03-23T03:46:57.967540+0000","last_unstale":"2024-03-23T03:46:59.546902+0000","last_undegraded":"2024-03-23T03:46:59.546902+0000","last_fullsized":"2024-03-23T03:46:59.546902+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:36:58.375101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951620+0000","last_change":"2024-03-23T03:46:53.966166+0000","last_active":"2024-03-23T03:46:57.951620+0000","last_peered":"2024-03-23T03:46:57.951620+0000","last_clean":"2024-03-23T03:46:57.951620+0000","last_became_active":"2024-03-23T03:46:53.965850+0000","last_became_peered":"2024-03-23T03:46:53.965850+0000","last_unstale":"2024-03-23T03:46:57.951620+0000","last_undegraded":"2024-03-23T03:46:57.951620+0000","last_fullsized":"2024-03-23T03:46:57.951620+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:35:38.384483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.948826+0000","last_change":"2024-03-23T03:46:53.954723+0000","last_active":"2024-03-23T03:46:57.948826+0000","last_peered":"2024-03-23T03:46:57.948826+0000","last_clean":"2024-03-23T03:46:57.948826+0000","last_became_active":"2024-03-23T03:46:53.954419+0000","last_became_peered":"2024-03-23T03:46:53.954419+0000","last_unstale":"2024-03-23T03:46:57.948826+0000","last_undegraded":"2024-03-23T03:46:57.948826+0000","last_fullsized":"2024-03-23T03:46:57.948826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:55:23.986885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970164+0000","last_change":"2024-03-23T03:46:53.950337+0000","last_active":"2024-03-23T03:46:57.970164+0000","last_peered":"2024-03-23T03:46:57.970164+0000","last_clean":"2024-03-23T03:46:57.970164+0000","last_became_active":"2024-03-23T03:46:53.949979+0000","last_became_peered":"2024-03-23T03:46:53.949979+0000","last_unstale":"2024-03-23T03:46:57.970164+0000","last_undegraded":"2024-03-23T03:46:57.970164+0000","last_fullsized":"2024-03-23T03:46:57.970164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:36.598727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952086+0000","last_change":"2024-03-23T03:46:53.957785+0000","last_active":"2024-03-23T03:46:57.952086+0000","last_peered":"2024-03-23T03:46:57.952086+0000","last_clean":"2024-03-23T03:46:57.952086+0000","last_became_active":"2024-03-23T03:46:53.956251+0000","last_became_peered":"2024-03-23T03:46:53.956251+0000","last_unstale":"2024-03-23T03:46:57.952086+0000","last_undegraded":"2024-03-23T03:46:57.952086+0000","last_fullsized":"2024-03-23T03:46:57.952086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:49:34.782399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.964730+0000","last_change":"2024-03-23T03:46:53.950745+0000","last_active":"2024-03-23T03:46:57.964730+0000","last_peered":"2024-03-23T03:46:57.964730+0000","last_clean":"2024-03-23T03:46:57.964730+0000","last_became_active":"2024-03-23T03:46:53.950592+0000","last_became_peered":"2024-03-23T03:46:53.950592+0000","last_unstale":"2024-03-23T03:46:57.964730+0000","last_undegraded":"2024-03-23T03:46:57.964730+0000","last_fullsized":"2024-03-23T03:46:57.964730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:16:39.131183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.942799+0000","last_change":"2024-03-23T03:46:53.949596+0000","last_active":"2024-03-23T03:46:57.942799+0000","last_peered":"2024-03-23T03:46:57.942799+0000","last_clean":"2024-03-23T03:46:57.942799+0000","last_became_active":"2024-03-23T03:46:53.949298+0000","last_became_peered":"2024-03-23T03:46:53.949298+0000","last_unstale":"2024-03-23T03:46:57.942799+0000","last_undegraded":"2024-03-23T03:46:57.942799+0000","last_fullsized":"2024-03-23T03:46:57.942799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:47.801092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952735+0000","last_change":"2024-03-23T03:46:53.961179+0000","last_active":"2024-03-23T03:46:57.952735+0000","last_peered":"2024-03-23T03:46:57.952735+0000","last_clean":"2024-03-23T03:46:57.952735+0000","last_became_active":"2024-03-23T03:46:53.960616+0000","last_became_peered":"2024-03-23T03:46:53.960616+0000","last_unstale":"2024-03-23T03:46:57.952735+0000","last_undegraded":"2024-03-23T03:46:57.952735+0000","last_fullsized":"2024-03-23T03:46:57.952735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:51:48.524978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.971878+0000","last_change":"2024-03-23T03:46:53.951801+0000","last_active":"2024-03-23T03:46:57.971878+0000","last_peered":"2024-03-23T03:46:57.971878+0000","last_clean":"2024-03-23T03:46:57.971878+0000","last_became_active":"2024-03-23T03:46:53.951529+0000","last_became_peered":"2024-03-23T03:46:53.951529+0000","last_unstale":"2024-03-23T03:46:57.971878+0000","last_undegraded":"2024-03-23T03:46:57.971878+0000","last_fullsized":"2024-03-23T03:46:57.971878+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:36.598727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.968476+0000","last_change":"2024-03-23T03:46:53.950898+0000","last_active":"2024-03-23T03:46:57.968476+0000","last_peered":"2024-03-23T03:46:57.968476+0000","last_clean":"2024-03-23T03:46:57.968476+0000","last_became_active":"2024-03-23T03:46:53.950232+0000","last_became_peered":"2024-03-23T03:46:53.950232+0000","last_unstale":"2024-03-23T03:46:57.968476+0000","last_undegraded":"2024-03-23T03:46:57.968476+0000","last_fullsized":"2024-03-23T03:46:57.968476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:51.878165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.961869+0000","last_change":"2024-03-23T03:46:53.939297+0000","last_active":"2024-03-23T03:46:57.961869+0000","last_peered":"2024-03-23T03:46:57.961869+0000","last_clean":"2024-03-23T03:46:57.961869+0000","last_became_active":"2024-03-23T03:46:53.939052+0000","last_became_peered":"2024-03-23T03:46:53.939052+0000","last_unstale":"2024-03-23T03:46:57.961869+0000","last_undegraded":"2024-03-23T03:46:57.961869+0000","last_fullsized":"2024-03-23T03:46:57.961869+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:16.232161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970145+0000","last_change":"2024-03-23T03:46:53.953978+0000","last_active":"2024-03-23T03:46:57.970145+0000","last_peered":"2024-03-23T03:46:57.970145+0000","last_clean":"2024-03-23T03:46:57.970145+0000","last_became_active":"2024-03-23T03:46:53.953703+0000","last_became_peered":"2024-03-23T03:46:53.953703+0000","last_unstale":"2024-03-23T03:46:57.970145+0000","last_undegraded":"2024-03-23T03:46:57.970145+0000","last_fullsized":"2024-03-23T03:46:57.970145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:50:10.900452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951969+0000","last_change":"2024-03-23T03:46:53.969252+0000","last_active":"2024-03-23T03:46:57.951969+0000","last_peered":"2024-03-23T03:46:57.951969+0000","last_clean":"2024-03-23T03:46:57.951969+0000","last_became_active":"2024-03-23T03:46:53.968966+0000","last_became_peered":"2024-03-23T03:46:53.968966+0000","last_unstale":"2024-03-23T03:46:57.951969+0000","last_undegraded":"2024-03-23T03:46:57.951969+0000","last_fullsized":"2024-03-23T03:46:57.951969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:49.329280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.960849+0000","last_change":"2024-03-23T03:46:53.965172+0000","last_active":"2024-03-23T03:46:57.960849+0000","last_peered":"2024-03-23T03:46:57.960849+0000","last_clean":"2024-03-23T03:46:57.960849+0000","last_became_active":"2024-03-23T03:46:53.963673+0000","last_became_peered":"2024-03-23T03:46:53.963673+0000","last_unstale":"2024-03-23T03:46:57.960849+0000","last_undegraded":"2024-03-23T03:46:57.960849+0000","last_fullsized":"2024-03-23T03:46:57.960849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:52:35.361467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017879+0000","last_change":"2024-03-23T03:46:53.952664+0000","last_active":"2024-03-23T03:46:58.017879+0000","last_peered":"2024-03-23T03:46:58.017879+0000","last_clean":"2024-03-23T03:46:58.017879+0000","last_became_active":"2024-03-23T03:46:53.952264+0000","last_became_peered":"2024-03-23T03:46:53.952264+0000","last_unstale":"2024-03-23T03:46:58.017879+0000","last_undegraded":"2024-03-23T03:46:58.017879+0000","last_fullsized":"2024-03-23T03:46:58.017879+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:51.878165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.947849+0000","last_change":"2024-03-23T03:46:53.965686+0000","last_active":"2024-03-23T03:46:57.947849+0000","last_peered":"2024-03-23T03:46:57.947849+0000","last_clean":"2024-03-23T03:46:57.947849+0000","last_became_active":"2024-03-23T03:46:53.965321+0000","last_became_peered":"2024-03-23T03:46:53.965321+0000","last_unstale":"2024-03-23T03:46:57.947849+0000","last_undegraded":"2024-03-23T03:46:57.947849+0000","last_fullsized":"2024-03-23T03:46:57.947849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:29:34.917375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970735+0000","last_change":"2024-03-23T03:46:53.973376+0000","last_active":"2024-03-23T03:46:57.970735+0000","last_peered":"2024-03-23T03:46:57.970735+0000","last_clean":"2024-03-23T03:46:57.970735+0000","last_became_active":"2024-03-23T03:46:53.973131+0000","last_became_peered":"2024-03-23T03:46:53.973131+0000","last_unstale":"2024-03-23T03:46:57.970735+0000","last_undegraded":"2024-03-23T03:46:57.970735+0000","last_fullsized":"2024-03-23T03:46:57.970735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:01:04.097184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.968336+0000","last_change":"2024-03-23T03:46:53.964244+0000","last_active":"2024-03-23T03:46:57.968336+0000","last_peered":"2024-03-23T03:46:57.968336+0000","last_clean":"2024-03-23T03:46:57.968336+0000","last_became_active":"2024-03-23T03:46:53.963926+0000","last_became_peered":"2024-03-23T03:46:53.963926+0000","last_unstale":"2024-03-23T03:46:57.968336+0000","last_undegraded":"2024-03-23T03:46:57.968336+0000","last_fullsized":"2024-03-23T03:46:57.968336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:49.329280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943357+0000","last_change":"2024-03-23T03:46:53.938653+0000","last_active":"2024-03-23T03:46:57.943357+0000","last_peered":"2024-03-23T03:46:57.943357+0000","last_clean":"2024-03-23T03:46:57.943357+0000","last_became_active":"2024-03-23T03:46:53.938323+0000","last_became_peered":"2024-03-23T03:46:53.938323+0000","last_unstale":"2024-03-23T03:46:57.943357+0000","last_undegraded":"2024-03-23T03:46:57.943357+0000","last_fullsized":"2024-03-23T03:46:57.943357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:49.329280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951110+0000","last_change":"2024-03-23T03:46:53.956639+0000","last_active":"2024-03-23T03:46:57.951110+0000","last_peered":"2024-03-23T03:46:57.951110+0000","last_clean":"2024-03-23T03:46:57.951110+0000","last_became_active":"2024-03-23T03:46:53.956097+0000","last_became_peered":"2024-03-23T03:46:53.956097+0000","last_unstale":"2024-03-23T03:46:57.951110+0000","last_undegraded":"2024-03-23T03:46:57.951110+0000","last_fullsized":"2024-03-23T03:46:57.951110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:46.978451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951858+0000","last_change":"2024-03-23T03:46:53.964641+0000","last_active":"2024-03-23T03:46:57.951858+0000","last_peered":"2024-03-23T03:46:57.951858+0000","last_clean":"2024-03-23T03:46:57.951858+0000","last_became_active":"2024-03-23T03:46:53.964167+0000","last_became_peered":"2024-03-23T03:46:53.964167+0000","last_unstale":"2024-03-23T03:46:57.951858+0000","last_undegraded":"2024-03-23T03:46:57.951858+0000","last_fullsized":"2024-03-23T03:46:57.951858+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:55.749189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.965170+0000","last_change":"2024-03-23T03:46:53.962737+0000","last_active":"2024-03-23T03:46:57.965170+0000","last_peered":"2024-03-23T03:46:57.965170+0000","last_clean":"2024-03-23T03:46:57.965170+0000","last_became_active":"2024-03-23T03:46:53.962229+0000","last_became_peered":"2024-03-23T03:46:53.962229+0000","last_unstale":"2024-03-23T03:46:57.965170+0000","last_undegraded":"2024-03-23T03:46:57.965170+0000","last_fullsized":"2024-03-23T03:46:57.965170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:59.999542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.953048+0000","last_change":"2024-03-23T03:46:53.971706+0000","last_active":"2024-03-23T03:46:57.953048+0000","last_peered":"2024-03-23T03:46:57.953048+0000","last_clean":"2024-03-23T03:46:57.953048+0000","last_became_active":"2024-03-23T03:46:53.971525+0000","last_became_peered":"2024-03-23T03:46:53.971525+0000","last_unstale":"2024-03-23T03:46:57.953048+0000","last_undegraded":"2024-03-23T03:46:57.953048+0000","last_fullsized":"2024-03-23T03:46:57.953048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:55:54.119888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952505+0000","last_change":"2024-03-23T03:46:53.970600+0000","last_active":"2024-03-23T03:46:57.952505+0000","last_peered":"2024-03-23T03:46:57.952505+0000","last_clean":"2024-03-23T03:46:57.952505+0000","last_became_active":"2024-03-23T03:46:53.970340+0000","last_became_peered":"2024-03-23T03:46:53.970340+0000","last_unstale":"2024-03-23T03:46:57.952505+0000","last_undegraded":"2024-03-23T03:46:57.952505+0000","last_fullsized":"2024-03-23T03:46:57.952505+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:48:32.346809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017965+0000","last_change":"2024-03-23T03:46:53.971448+0000","last_active":"2024-03-23T03:46:58.017965+0000","last_peered":"2024-03-23T03:46:58.017965+0000","last_clean":"2024-03-23T03:46:58.017965+0000","last_became_active":"2024-03-23T03:46:53.971152+0000","last_became_peered":"2024-03-23T03:46:53.971152+0000","last_unstale":"2024-03-23T03:46:58.017965+0000","last_undegraded":"2024-03-23T03:46:58.017965+0000","last_fullsized":"2024-03-23T03:46:58.017965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:36.598727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.953820+0000","last_change":"2024-03-23T03:46:53.944836+0000","last_active":"2024-03-23T03:46:57.953820+0000","last_peered":"2024-03-23T03:46:57.953820+0000","last_clean":"2024-03-23T03:46:57.953820+0000","last_became_active":"2024-03-23T03:46:53.944614+0000","last_became_peered":"2024-03-23T03:46:53.944614+0000","last_unstale":"2024-03-23T03:46:57.953820+0000","last_undegraded":"2024-03-23T03:46:57.953820+0000","last_fullsized":"2024-03-23T03:46:57.953820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:01:24.719476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.013281+0000","last_change":"2024-03-23T03:46:53.957372+0000","last_active":"2024-03-23T03:46:58.013281+0000","last_peered":"2024-03-23T03:46:58.013281+0000","last_clean":"2024-03-23T03:46:58.013281+0000","last_became_active":"2024-03-23T03:46:53.956119+0000","last_became_peered":"2024-03-23T03:46:53.956119+0000","last_unstale":"2024-03-23T03:46:58.013281+0000","last_undegraded":"2024-03-23T03:46:58.013281+0000","last_fullsized":"2024-03-23T03:46:58.013281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:16:39.131183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.968302+0000","last_change":"2024-03-23T03:46:53.954765+0000","last_active":"2024-03-23T03:46:57.968302+0000","last_peered":"2024-03-23T03:46:57.968302+0000","last_clean":"2024-03-23T03:46:57.968302+0000","last_became_active":"2024-03-23T03:46:53.954424+0000","last_became_peered":"2024-03-23T03:46:53.954424+0000","last_unstale":"2024-03-23T03:46:57.968302+0000","last_undegraded":"2024-03-23T03:46:57.968302+0000","last_fullsized":"2024-03-23T03:46:57.968302+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:56:29.065234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.016764+0000","last_change":"2024-03-23T03:46:53.949427+0000","last_active":"2024-03-23T03:46:58.016764+0000","last_peered":"2024-03-23T03:46:58.016764+0000","last_clean":"2024-03-23T03:46:58.016764+0000","last_became_active":"2024-03-23T03:46:53.949114+0000","last_became_peered":"2024-03-23T03:46:53.949114+0000","last_unstale":"2024-03-23T03:46:58.016764+0000","last_undegraded":"2024-03-23T03:46:58.016764+0000","last_fullsized":"2024-03-23T03:46:58.016764+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:50:10.900452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970488+0000","last_change":"2024-03-23T03:46:53.956260+0000","last_active":"2024-03-23T03:46:57.970488+0000","last_peered":"2024-03-23T03:46:57.970488+0000","last_clean":"2024-03-23T03:46:57.970488+0000","last_became_active":"2024-03-23T03:46:53.955509+0000","last_became_peered":"2024-03-23T03:46:53.955509+0000","last_unstale":"2024-03-23T03:46:57.970488+0000","last_undegraded":"2024-03-23T03:46:57.970488+0000","last_fullsized":"2024-03-23T03:46:57.970488+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:34:41.509208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951671+0000","last_change":"2024-03-23T03:46:53.957607+0000","last_active":"2024-03-23T03:46:57.951671+0000","last_peered":"2024-03-23T03:46:57.951671+0000","last_clean":"2024-03-23T03:46:57.951671+0000","last_became_active":"2024-03-23T03:46:53.956236+0000","last_became_peered":"2024-03-23T03:46:53.956236+0000","last_unstale":"2024-03-23T03:46:57.951671+0000","last_undegraded":"2024-03-23T03:46:57.951671+0000","last_fullsized":"2024-03-23T03:46:57.951671+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:27:03.917141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943481+0000","last_change":"2024-03-23T03:46:53.960428+0000","last_active":"2024-03-23T03:46:57.943481+0000","last_peered":"2024-03-23T03:46:57.943481+0000","last_clean":"2024-03-23T03:46:57.943481+0000","last_became_active":"2024-03-23T03:46:53.960148+0000","last_became_peered":"2024-03-23T03:46:53.960148+0000","last_unstale":"2024-03-23T03:46:57.943481+0000","last_undegraded":"2024-03-23T03:46:57.943481+0000","last_fullsized":"2024-03-23T03:46:57.943481+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:01:24.719476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017880+0000","last_change":"2024-03-23T03:46:53.966166+0000","last_active":"2024-03-23T03:46:58.017880+0000","last_peered":"2024-03-23T03:46:58.017880+0000","last_clean":"2024-03-23T03:46:58.017880+0000","last_became_active":"2024-03-23T03:46:53.965871+0000","last_became_peered":"2024-03-23T03:46:53.965871+0000","last_unstale":"2024-03-23T03:46:58.017880+0000","last_undegraded":"2024-03-23T03:46:58.017880+0000","last_fullsized":"2024-03-23T03:46:58.017880+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:01:04.097184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4593,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542154,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":698548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4730000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4079999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":111393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.258}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]}]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":107688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6419999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.54}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8500000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.877}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.964}]}]},{"osd":4,"up_from":14,"seq":60129542153,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":107724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.383}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.343}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":111393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.7859999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542153,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":107598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.498}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7050000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5800000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":107758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.488}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.595}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6359999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":880640,"data_stored":698604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.163}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.367}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:47:21.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:47:22.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.096+0000 7ff2f6600640 1 Processor -- start 2024-03-23T03:47:22.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.096+0000 7ff2f6600640 1 -- start start 2024-03-23T03:47:22.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.097+0000 7ff2f6600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff2f00cbab0 0x7ff2f00b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:22.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.097+0000 7ff2f6600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff2f006d390 0x7ff2f006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:22.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.097+0000 7ff2f6600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f00b16a0 con 0x7ff2f00adbc0 2024-03-23T03:47:22.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.097+0000 7ff2f6600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f006dd60 con 0x7ff2f006d390 2024-03-23T03:47:22.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.097+0000 7ff2f6600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f00cbfb0 con 0x7ff2f00cbab0 2024-03-23T03:47:22.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.098+0000 7ff2efe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff2f006d390 0x7ff2f006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:22.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.098+0000 7ff2ef400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff2f00adbc0 0x7ff2f00b41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41266/0 (socket says 172.21.15.102:41266) 2024-03-23T03:47:22.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.098+0000 7ff2ef400640 1 -- 172.21.15.102:0/3611424034 learned_addr learned my addr 172.21.15.102:0/3611424034 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:22.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.098+0000 7ff2efe00640 1 -- 172.21.15.102:0/3611424034 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff2f00cbab0 msgr2=0x7ff2f00b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:22.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.098+0000 7ff2e6a00640 1 --2- 172.21.15.102:0/3611424034 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff2f00cbab0 0x7ff2f00b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:22.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.098+0000 7ff2efe00640 1 --2- 172.21.15.102:0/3611424034 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff2f00cbab0 0x7ff2f00b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.098+0000 7ff2eea00640 1 -- 172.21.15.102:0/3611424034 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1171810410 0 0) 0x7ff2f00b16a0 con 0x7ff2f00adbc0 2024-03-23T03:47:22.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.098+0000 7ff2efe00640 1 -- 172.21.15.102:0/3611424034 >> v1:172.21.15.102:6789/0 conn(0x7ff2f00adbc0 legacy=0x7ff2f00b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:22.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.098+0000 7ff2efe00640 1 -- 172.21.15.102:0/3611424034 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f00cc290 con 0x7ff2f006d390 2024-03-23T03:47:22.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.098+0000 7ff2e6a00640 1 --2- 172.21.15.102:0/3611424034 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff2f00cbab0 0x7ff2f00b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:22.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.099+0000 7ff2efe00640 1 --2- 172.21.15.102:0/3611424034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2f006d390 0x7ff2f006d820 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8009d90 tx=0x7ff2e802be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea1ccc9c690afd04 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:22.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.099+0000 7ff2eea00640 1 -- 172.21.15.102:0/3611424034 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e802cbc0 con 0x7ff2f006d390 2024-03-23T03:47:22.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.099+0000 7ff2eea00640 1 -- 172.21.15.102:0/3611424034 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e802cd20 con 0x7ff2f006d390 2024-03-23T03:47:22.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.099+0000 7ff2eea00640 1 -- 172.21.15.102:0/3611424034 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8034660 con 0x7ff2f006d390 2024-03-23T03:47:22.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.100+0000 7ff2f6600640 1 -- 172.21.15.102:0/3611424034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2f006d390 msgr2=0x7ff2f006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:22.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.100+0000 7ff2f6600640 1 --2- 172.21.15.102:0/3611424034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2f006d390 0x7ff2f006d820 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8009d90 tx=0x7ff2e802be10 comp rx=0 tx=0).stop 2024-03-23T03:47:22.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.100+0000 7ff2f6600640 1 -- 172.21.15.102:0/3611424034 shutdown_connections 2024-03-23T03:47:22.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.100+0000 7ff2f6600640 1 --2- 172.21.15.102:0/3611424034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2f006d390 0x7ff2f006d820 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.100+0000 7ff2f6600640 1 --2- 172.21.15.102:0/3611424034 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff2f00cbab0 0x7ff2f00b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.100+0000 7ff2f6600640 1 -- 172.21.15.102:0/3611424034 >> 172.21.15.102:0/3611424034 conn(0x7ff2f006a3d0 msgr2=0x7ff2f00b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:22.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.100+0000 7ff2f6600640 1 -- 172.21.15.102:0/3611424034 shutdown_connections 2024-03-23T03:47:22.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.100+0000 7ff2f6600640 1 -- 172.21.15.102:0/3611424034 wait complete. 2024-03-23T03:47:22.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.102+0000 7ff2f6600640 1 Processor -- start 2024-03-23T03:47:22.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.102+0000 7ff2f6600640 1 -- start start 2024-03-23T03:47:22.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.103+0000 7ff2f6600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2f006d390 0x7ff2f00c3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:22.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.103+0000 7ff2f6600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff2f00adbc0 0x7ff2f01ef9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:22.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.103+0000 7ff2f6600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2f00cbab0 0x7ff2f0134880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:22.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.103+0000 7ff2f6600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f00c3b50 con 0x7ff2f00cbab0 2024-03-23T03:47:22.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.103+0000 7ff2f6600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f00abac0 con 0x7ff2f006d390 2024-03-23T03:47:22.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.103+0000 7ff2f6600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f00abda0 con 0x7ff2f00adbc0 2024-03-23T03:47:22.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.103+0000 7ff2efe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2f00cbab0 0x7ff2f0134880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:22.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.103+0000 7ff2efe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2f00cbab0 0x7ff2f0134880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37514/0 (socket says 172.21.15.102:37514) 2024-03-23T03:47:22.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.103+0000 7ff2efe00640 1 -- 172.21.15.102:0/557969297 learned_addr learned my addr 172.21.15.102:0/557969297 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:22.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.103+0000 7ff2efe00640 1 -- 172.21.15.102:0/557969297 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff2f00adbc0 msgr2=0x7ff2f01ef9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:22.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.103+0000 7ff2e6a00640 1 --2- 172.21.15.102:0/557969297 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff2f00adbc0 0x7ff2f01ef9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:22.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.103+0000 7ff2ef400640 1 --2- 172.21.15.102:0/557969297 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2f006d390 0x7ff2f00c3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:22.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.103+0000 7ff2efe00640 1 --2- 172.21.15.102:0/557969297 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff2f00adbc0 0x7ff2f01ef9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.103+0000 7ff2efe00640 1 -- 172.21.15.102:0/557969297 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2f006d390 msgr2=0x7ff2f00c3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:22.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.103+0000 7ff2efe00640 1 --2- 172.21.15.102:0/557969297 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2f006d390 0x7ff2f00c3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.103+0000 7ff2efe00640 1 -- 172.21.15.102:0/557969297 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e8009a70 con 0x7ff2f00cbab0 2024-03-23T03:47:22.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.104+0000 7ff2efe00640 1 --2- 172.21.15.102:0/557969297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2f00cbab0 0x7ff2f0134880 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8009d90 tx=0x7ff2e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.104+0000 7ff2e7e00640 1 -- 172.21.15.102:0/557969297 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8003b90 con 0x7ff2f00cbab0 2024-03-23T03:47:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.104+0000 7ff2e7e00640 1 -- 172.21.15.102:0/557969297 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e802ce90 con 0x7ff2f00cbab0 2024-03-23T03:47:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.104+0000 7ff2f6600640 1 -- 172.21.15.102:0/557969297 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f00ac080 con 0x7ff2f00cbab0 2024-03-23T03:47:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.104+0000 7ff2e7e00640 1 -- 172.21.15.102:0/557969297 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e8004030 con 0x7ff2f00cbab0 2024-03-23T03:47:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.104+0000 7ff2f6600640 1 -- 172.21.15.102:0/557969297 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f00ac640 con 0x7ff2f00cbab0 2024-03-23T03:47:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.105+0000 7ff2f6600640 1 -- 172.21.15.102:0/557969297 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f006ac20 con 0x7ff2f00cbab0 2024-03-23T03:47:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.106+0000 7ff2e7e00640 1 -- 172.21.15.102:0/557969297 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff2e803b050 con 0x7ff2f00cbab0 2024-03-23T03:47:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.107+0000 7ff2e7e00640 1 --2- 172.21.15.102:0/557969297 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff2cc06ac00 0x7ff2cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.107+0000 7ff2e7e00640 1 -- 172.21.15.102:0/557969297 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff2e80abc10 con 0x7ff2f00cbab0 2024-03-23T03:47:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.107+0000 7ff2ef400640 1 --2- 172.21.15.102:0/557969297 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff2cc06ac00 0x7ff2cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.108+0000 7ff2ef400640 1 --2- 172.21.15.102:0/557969297 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff2cc06ac00 0x7ff2cc06d0c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0007fd0 tx=0x7ff2e0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:22.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.111+0000 7ff2e7e00640 1 -- 172.21.15.102:0/557969297 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff2e8076a60 con 0x7ff2f00cbab0 2024-03-23T03:47:22.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.287+0000 7ff2f6600640 1 -- 172.21.15.102:0/557969297 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff2f00b1bd0 con 0x7ff2cc06ac00 2024-03-23T03:47:22.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.313+0000 7ff2e7e00640 1 -- 172.21.15.102:0/557969297 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345005 (secure 0 0 0) 0x7ff2e0001c70 con 0x7ff2cc06ac00 2024-03-23T03:47:22.318 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:22.320 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:47:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.320+0000 7ff2f6600640 1 -- 172.21.15.102:0/557969297 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff2cc06ac00 msgr2=0x7ff2cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.320+0000 7ff2f6600640 1 --2- 172.21.15.102:0/557969297 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff2cc06ac00 0x7ff2cc06d0c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff2e0007fd0 tx=0x7ff2e0009040 comp rx=0 tx=0).stop 2024-03-23T03:47:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.320+0000 7ff2f6600640 1 -- 172.21.15.102:0/557969297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2f00cbab0 msgr2=0x7ff2f0134880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.320+0000 7ff2f6600640 1 --2- 172.21.15.102:0/557969297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2f00cbab0 0x7ff2f0134880 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8009d90 tx=0x7ff2e8004a40 comp rx=0 tx=0).stop 2024-03-23T03:47:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.320+0000 7ff2f6600640 1 -- 172.21.15.102:0/557969297 shutdown_connections 2024-03-23T03:47:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.320+0000 7ff2f6600640 1 --2- 172.21.15.102:0/557969297 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff2cc06ac00 0x7ff2cc06d0c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.320+0000 7ff2f6600640 1 --2- 172.21.15.102:0/557969297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2f00cbab0 0x7ff2f0134880 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.320+0000 7ff2f6600640 1 --2- 172.21.15.102:0/557969297 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff2f00adbc0 0x7ff2f01ef9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.320+0000 7ff2f6600640 1 --2- 172.21.15.102:0/557969297 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff2f006d390 0x7ff2f00c3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.320+0000 7ff2f6600640 1 -- 172.21.15.102:0/557969297 >> 172.21.15.102:0/557969297 conn(0x7ff2f006a3d0 msgr2=0x7ff2f0142bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:22.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.321+0000 7ff2f6600640 1 -- 172.21.15.102:0/557969297 shutdown_connections 2024-03-23T03:47:22.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.321+0000 7ff2f6600640 1 -- 172.21.15.102:0/557969297 wait complete. 2024-03-23T03:47:22.339 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-23T03:47:21.723580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594961,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":208,"num_write_kb":2435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160,"ondisk_log_size":160,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219556,"kb_used_data":3368,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755164,"statfs":{"total":773094113280,"available":772869287936,"internally_reserved":0,"allocated":3448832,"data_stored":2050706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":221236794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":141,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003753"},"pg_stats":[{"pgid":"2.1f","version":"21'4","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.954535+0000","last_change":"2024-03-23T03:46:53.465534+0000","last_active":"2024-03-23T03:46:57.954535+0000","last_peered":"2024-03-23T03:46:57.954535+0000","last_clean":"2024-03-23T03:46:57.954535+0000","last_became_active":"2024-03-23T03:46:53.465309+0000","last_became_peered":"2024-03-23T03:46:53.465309+0000","last_unstale":"2024-03-23T03:46:57.954535+0000","last_undegraded":"2024-03-23T03:46:57.954535+0000","last_fullsized":"2024-03-23T03:46:57.954535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:52:38.737363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951985+0000","last_change":"2024-03-23T03:46:53.963693+0000","last_active":"2024-03-23T03:46:57.951985+0000","last_peered":"2024-03-23T03:46:57.951985+0000","last_clean":"2024-03-23T03:46:57.951985+0000","last_became_active":"2024-03-23T03:46:53.963435+0000","last_became_peered":"2024-03-23T03:46:53.963435+0000","last_unstale":"2024-03-23T03:46:57.951985+0000","last_undegraded":"2024-03-23T03:46:57.951985+0000","last_fullsized":"2024-03-23T03:46:57.951985+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:51.878165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.544002+0000","last_change":"2024-03-23T03:46:58.018608+0000","last_active":"2024-03-23T03:46:59.544002+0000","last_peered":"2024-03-23T03:46:59.544002+0000","last_clean":"2024-03-23T03:46:59.544002+0000","last_became_active":"2024-03-23T03:46:58.018216+0000","last_became_peered":"2024-03-23T03:46:58.018216+0000","last_unstale":"2024-03-23T03:46:59.544002+0000","last_undegraded":"2024-03-23T03:46:59.544002+0000","last_fullsized":"2024-03-23T03:46:59.544002+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:41:36.624562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952201+0000","last_change":"2024-03-23T03:46:53.462106+0000","last_active":"2024-03-23T03:46:57.952201+0000","last_peered":"2024-03-23T03:46:57.952201+0000","last_clean":"2024-03-23T03:46:57.952201+0000","last_became_active":"2024-03-23T03:46:53.461757+0000","last_became_peered":"2024-03-23T03:46:53.461757+0000","last_unstale":"2024-03-23T03:46:57.952201+0000","last_undegraded":"2024-03-23T03:46:57.952201+0000","last_fullsized":"2024-03-23T03:46:57.952201+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:20:09.953919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.954319+0000","last_change":"2024-03-23T03:46:53.959076+0000","last_active":"2024-03-23T03:46:57.954319+0000","last_peered":"2024-03-23T03:46:57.954319+0000","last_clean":"2024-03-23T03:46:57.954319+0000","last_became_active":"2024-03-23T03:46:53.958351+0000","last_became_peered":"2024-03-23T03:46:53.958351+0000","last_unstale":"2024-03-23T03:46:57.954319+0000","last_undegraded":"2024-03-23T03:46:57.954319+0000","last_fullsized":"2024-03-23T03:46:57.954319+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:55.749189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.423330+0000","last_change":"2024-03-23T03:46:57.979717+0000","last_active":"2024-03-23T03:46:58.423330+0000","last_peered":"2024-03-23T03:46:58.423330+0000","last_clean":"2024-03-23T03:46:58.423330+0000","last_became_active":"2024-03-23T03:46:57.975153+0000","last_became_peered":"2024-03-23T03:46:57.975153+0000","last_unstale":"2024-03-23T03:46:58.423330+0000","last_undegraded":"2024-03-23T03:46:58.423330+0000","last_fullsized":"2024-03-23T03:46:58.423330+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:24:30.082562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017773+0000","last_change":"2024-03-23T03:46:53.480716+0000","last_active":"2024-03-23T03:46:58.017773+0000","last_peered":"2024-03-23T03:46:58.017773+0000","last_clean":"2024-03-23T03:46:58.017773+0000","last_became_active":"2024-03-23T03:46:53.479510+0000","last_became_peered":"2024-03-23T03:46:53.479510+0000","last_unstale":"2024-03-23T03:46:58.017773+0000","last_undegraded":"2024-03-23T03:46:58.017773+0000","last_fullsized":"2024-03-23T03:46:58.017773+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:40:20.814830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970572+0000","last_change":"2024-03-23T03:46:53.947108+0000","last_active":"2024-03-23T03:46:57.970572+0000","last_peered":"2024-03-23T03:46:57.970572+0000","last_clean":"2024-03-23T03:46:57.970572+0000","last_became_active":"2024-03-23T03:46:53.946415+0000","last_became_peered":"2024-03-23T03:46:53.946415+0000","last_unstale":"2024-03-23T03:46:57.970572+0000","last_undegraded":"2024-03-23T03:46:57.970572+0000","last_fullsized":"2024-03-23T03:46:57.970572+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:38:13.205855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.426555+0000","last_change":"2024-03-23T03:46:57.968557+0000","last_active":"2024-03-23T03:46:58.426555+0000","last_peered":"2024-03-23T03:46:58.426555+0000","last_clean":"2024-03-23T03:46:58.426555+0000","last_became_active":"2024-03-23T03:46:57.968085+0000","last_became_peered":"2024-03-23T03:46:57.968085+0000","last_unstale":"2024-03-23T03:46:58.426555+0000","last_undegraded":"2024-03-23T03:46:58.426555+0000","last_fullsized":"2024-03-23T03:46:58.426555+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:26:09.015758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.014925+0000","last_change":"2024-03-23T03:46:53.481389+0000","last_active":"2024-03-23T03:46:58.014925+0000","last_peered":"2024-03-23T03:46:58.014925+0000","last_clean":"2024-03-23T03:46:58.014925+0000","last_became_active":"2024-03-23T03:46:53.481119+0000","last_became_peered":"2024-03-23T03:46:53.481119+0000","last_unstale":"2024-03-23T03:46:58.014925+0000","last_undegraded":"2024-03-23T03:46:58.014925+0000","last_fullsized":"2024-03-23T03:46:58.014925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:23:34.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970863+0000","last_change":"2024-03-23T03:46:53.952464+0000","last_active":"2024-03-23T03:46:57.970863+0000","last_peered":"2024-03-23T03:46:57.970863+0000","last_clean":"2024-03-23T03:46:57.970863+0000","last_became_active":"2024-03-23T03:46:53.951343+0000","last_became_peered":"2024-03-23T03:46:53.951343+0000","last_unstale":"2024-03-23T03:46:57.970863+0000","last_undegraded":"2024-03-23T03:46:57.970863+0000","last_fullsized":"2024-03-23T03:46:57.970863+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:35:49.855085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.423616+0000","last_change":"2024-03-23T03:46:57.974943+0000","last_active":"2024-03-23T03:46:58.423616+0000","last_peered":"2024-03-23T03:46:58.423616+0000","last_clean":"2024-03-23T03:46:58.423616+0000","last_became_active":"2024-03-23T03:46:57.974037+0000","last_became_peered":"2024-03-23T03:46:57.974037+0000","last_unstale":"2024-03-23T03:46:58.423616+0000","last_undegraded":"2024-03-23T03:46:58.423616+0000","last_fullsized":"2024-03-23T03:46:58.423616+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:24:17.331178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952188+0000","last_change":"2024-03-23T03:46:53.961283+0000","last_active":"2024-03-23T03:46:57.952188+0000","last_peered":"2024-03-23T03:46:57.952188+0000","last_clean":"2024-03-23T03:46:57.952188+0000","last_became_active":"2024-03-23T03:46:53.961004+0000","last_became_peered":"2024-03-23T03:46:53.961004+0000","last_unstale":"2024-03-23T03:46:57.952188+0000","last_undegraded":"2024-03-23T03:46:57.952188+0000","last_fullsized":"2024-03-23T03:46:57.952188+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:30:06.938168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017032+0000","last_change":"2024-03-23T03:46:53.480802+0000","last_active":"2024-03-23T03:46:58.017032+0000","last_peered":"2024-03-23T03:46:58.017032+0000","last_clean":"2024-03-23T03:46:58.017032+0000","last_became_active":"2024-03-23T03:46:53.480500+0000","last_became_peered":"2024-03-23T03:46:53.480500+0000","last_unstale":"2024-03-23T03:46:58.017032+0000","last_undegraded":"2024-03-23T03:46:58.017032+0000","last_fullsized":"2024-03-23T03:46:58.017032+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:38:29.605938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.424746+0000","last_change":"2024-03-23T03:46:57.972616+0000","last_active":"2024-03-23T03:46:58.424746+0000","last_peered":"2024-03-23T03:46:58.424746+0000","last_clean":"2024-03-23T03:46:58.424746+0000","last_became_active":"2024-03-23T03:46:57.972028+0000","last_became_peered":"2024-03-23T03:46:57.972028+0000","last_unstale":"2024-03-23T03:46:58.424746+0000","last_undegraded":"2024-03-23T03:46:58.424746+0000","last_fullsized":"2024-03-23T03:46:58.424746+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:55:53.867281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952016+0000","last_change":"2024-03-23T03:46:53.960010+0000","last_active":"2024-03-23T03:46:57.952016+0000","last_peered":"2024-03-23T03:46:57.952016+0000","last_clean":"2024-03-23T03:46:57.952016+0000","last_became_active":"2024-03-23T03:46:53.959761+0000","last_became_peered":"2024-03-23T03:46:53.959761+0000","last_unstale":"2024-03-23T03:46:57.952016+0000","last_undegraded":"2024-03-23T03:46:57.952016+0000","last_fullsized":"2024-03-23T03:46:57.952016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:49.329280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.946127+0000","last_change":"2024-03-23T03:46:53.460246+0000","last_active":"2024-03-23T03:46:57.946127+0000","last_peered":"2024-03-23T03:46:57.946127+0000","last_clean":"2024-03-23T03:46:57.946127+0000","last_became_active":"2024-03-23T03:46:53.459900+0000","last_became_peered":"2024-03-23T03:46:53.459900+0000","last_unstale":"2024-03-23T03:46:57.946127+0000","last_undegraded":"2024-03-23T03:46:57.946127+0000","last_fullsized":"2024-03-23T03:46:57.946127+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:23:34.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.424823+0000","last_change":"2024-03-23T03:46:57.976006+0000","last_active":"2024-03-23T03:46:58.424823+0000","last_peered":"2024-03-23T03:46:58.424823+0000","last_clean":"2024-03-23T03:46:58.424823+0000","last_became_active":"2024-03-23T03:46:57.975320+0000","last_became_peered":"2024-03-23T03:46:57.975320+0000","last_unstale":"2024-03-23T03:46:58.424823+0000","last_undegraded":"2024-03-23T03:46:58.424823+0000","last_fullsized":"2024-03-23T03:46:58.424823+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:04:54.573575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.960391+0000","last_change":"2024-03-23T03:46:53.954675+0000","last_active":"2024-03-23T03:46:57.960391+0000","last_peered":"2024-03-23T03:46:57.960391+0000","last_clean":"2024-03-23T03:46:57.960391+0000","last_became_active":"2024-03-23T03:46:53.954433+0000","last_became_peered":"2024-03-23T03:46:53.954433+0000","last_unstale":"2024-03-23T03:46:57.960391+0000","last_undegraded":"2024-03-23T03:46:57.960391+0000","last_fullsized":"2024-03-23T03:46:57.960391+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:27:03.917141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943505+0000","last_change":"2024-03-23T03:46:53.463665+0000","last_active":"2024-03-23T03:46:57.943505+0000","last_peered":"2024-03-23T03:46:57.943505+0000","last_clean":"2024-03-23T03:46:57.943505+0000","last_became_active":"2024-03-23T03:46:53.463379+0000","last_became_peered":"2024-03-23T03:46:53.463379+0000","last_unstale":"2024-03-23T03:46:57.943505+0000","last_undegraded":"2024-03-23T03:46:57.943505+0000","last_fullsized":"2024-03-23T03:46:57.943505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:39:13.578639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545667+0000","last_change":"2024-03-23T03:46:58.017005+0000","last_active":"2024-03-23T03:46:59.545667+0000","last_peered":"2024-03-23T03:46:59.545667+0000","last_clean":"2024-03-23T03:46:59.545667+0000","last_became_active":"2024-03-23T03:46:58.016393+0000","last_became_peered":"2024-03-23T03:46:58.016393+0000","last_unstale":"2024-03-23T03:46:59.545667+0000","last_undegraded":"2024-03-23T03:46:59.545667+0000","last_fullsized":"2024-03-23T03:46:59.545667+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:54:23.785439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.959120+0000","last_change":"2024-03-23T03:46:53.941019+0000","last_active":"2024-03-23T03:46:57.959120+0000","last_peered":"2024-03-23T03:46:57.959120+0000","last_clean":"2024-03-23T03:46:57.959120+0000","last_became_active":"2024-03-23T03:46:53.940265+0000","last_became_peered":"2024-03-23T03:46:53.940265+0000","last_unstale":"2024-03-23T03:46:57.959120+0000","last_undegraded":"2024-03-23T03:46:57.959120+0000","last_fullsized":"2024-03-23T03:46:57.959120+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:04:08.642972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.012311+0000","last_change":"2024-03-23T03:46:53.471334+0000","last_active":"2024-03-23T03:46:58.012311+0000","last_peered":"2024-03-23T03:46:58.012311+0000","last_clean":"2024-03-23T03:46:58.012311+0000","last_became_active":"2024-03-23T03:46:53.470996+0000","last_became_peered":"2024-03-23T03:46:53.470996+0000","last_unstale":"2024-03-23T03:46:58.012311+0000","last_undegraded":"2024-03-23T03:46:58.012311+0000","last_fullsized":"2024-03-23T03:46:58.012311+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:55.074632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.425416+0000","last_change":"2024-03-23T03:46:57.975950+0000","last_active":"2024-03-23T03:46:58.425416+0000","last_peered":"2024-03-23T03:46:58.425416+0000","last_clean":"2024-03-23T03:46:58.425416+0000","last_became_active":"2024-03-23T03:46:57.975504+0000","last_became_peered":"2024-03-23T03:46:57.975504+0000","last_unstale":"2024-03-23T03:46:58.425416+0000","last_undegraded":"2024-03-23T03:46:58.425416+0000","last_fullsized":"2024-03-23T03:46:58.425416+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:21.492278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.955743+0000","last_change":"2024-03-23T03:46:53.957169+0000","last_active":"2024-03-23T03:46:57.955743+0000","last_peered":"2024-03-23T03:46:57.955743+0000","last_clean":"2024-03-23T03:46:57.955743+0000","last_became_active":"2024-03-23T03:46:53.956617+0000","last_became_peered":"2024-03-23T03:46:53.956617+0000","last_unstale":"2024-03-23T03:46:57.955743+0000","last_undegraded":"2024-03-23T03:46:57.955743+0000","last_fullsized":"2024-03-23T03:46:57.955743+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:50:57.246940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.953730+0000","last_change":"2024-03-23T03:46:53.475395+0000","last_active":"2024-03-23T03:46:57.953730+0000","last_peered":"2024-03-23T03:46:57.953730+0000","last_clean":"2024-03-23T03:46:57.953730+0000","last_became_active":"2024-03-23T03:46:53.475130+0000","last_became_peered":"2024-03-23T03:46:53.475130+0000","last_unstale":"2024-03-23T03:46:57.953730+0000","last_undegraded":"2024-03-23T03:46:57.953730+0000","last_fullsized":"2024-03-23T03:46:57.953730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:19:02.377973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.423957+0000","last_change":"2024-03-23T03:46:58.019438+0000","last_active":"2024-03-23T03:46:58.423957+0000","last_peered":"2024-03-23T03:46:58.423957+0000","last_clean":"2024-03-23T03:46:58.423957+0000","last_became_active":"2024-03-23T03:46:58.019018+0000","last_became_peered":"2024-03-23T03:46:58.019018+0000","last_unstale":"2024-03-23T03:46:58.423957+0000","last_undegraded":"2024-03-23T03:46:58.423957+0000","last_fullsized":"2024-03-23T03:46:58.423957+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:53:39.544425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.971959+0000","last_change":"2024-03-23T03:46:53.950103+0000","last_active":"2024-03-23T03:46:57.971959+0000","last_peered":"2024-03-23T03:46:57.971959+0000","last_clean":"2024-03-23T03:46:57.971959+0000","last_became_active":"2024-03-23T03:46:53.947870+0000","last_became_peered":"2024-03-23T03:46:53.947870+0000","last_unstale":"2024-03-23T03:46:57.971959+0000","last_undegraded":"2024-03-23T03:46:57.971959+0000","last_fullsized":"2024-03-23T03:46:57.971959+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:56:29.065234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"24'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943640+0000","last_change":"2024-03-23T03:46:53.476543+0000","last_active":"2024-03-23T03:46:57.943640+0000","last_peered":"2024-03-23T03:46:57.943640+0000","last_clean":"2024-03-23T03:46:57.943640+0000","last_became_active":"2024-03-23T03:46:53.476227+0000","last_became_peered":"2024-03-23T03:46:53.476227+0000","last_unstale":"2024-03-23T03:46:57.943640+0000","last_undegraded":"2024-03-23T03:46:57.943640+0000","last_fullsized":"2024-03-23T03:46:57.943640+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:52:38.737363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.543994+0000","last_change":"2024-03-23T03:46:58.013935+0000","last_active":"2024-03-23T03:46:59.543994+0000","last_peered":"2024-03-23T03:46:59.543994+0000","last_clean":"2024-03-23T03:46:59.543994+0000","last_became_active":"2024-03-23T03:46:58.012631+0000","last_became_peered":"2024-03-23T03:46:58.012631+0000","last_unstale":"2024-03-23T03:46:59.543994+0000","last_undegraded":"2024-03-23T03:46:59.543994+0000","last_fullsized":"2024-03-23T03:46:59.543994+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:42:09.297861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.950778+0000","last_change":"2024-03-23T03:46:53.945184+0000","last_active":"2024-03-23T03:46:57.950778+0000","last_peered":"2024-03-23T03:46:57.950778+0000","last_clean":"2024-03-23T03:46:57.950778+0000","last_became_active":"2024-03-23T03:46:53.943550+0000","last_became_peered":"2024-03-23T03:46:53.943550+0000","last_unstale":"2024-03-23T03:46:57.950778+0000","last_undegraded":"2024-03-23T03:46:57.950778+0000","last_fullsized":"2024-03-23T03:46:57.950778+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:01:31.453936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.018003+0000","last_change":"2024-03-23T03:46:53.472577+0000","last_active":"2024-03-23T03:46:58.018003+0000","last_peered":"2024-03-23T03:46:58.018003+0000","last_clean":"2024-03-23T03:46:58.018003+0000","last_became_active":"2024-03-23T03:46:53.472218+0000","last_became_peered":"2024-03-23T03:46:53.472218+0000","last_unstale":"2024-03-23T03:46:58.018003+0000","last_undegraded":"2024-03-23T03:46:58.018003+0000","last_fullsized":"2024-03-23T03:46:58.018003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:21:03.254658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411414+0000","last_change":"2024-03-23T03:46:58.013198+0000","last_active":"2024-03-23T03:46:59.411414+0000","last_peered":"2024-03-23T03:46:59.411414+0000","last_clean":"2024-03-23T03:46:59.411414+0000","last_became_active":"2024-03-23T03:46:58.012474+0000","last_became_peered":"2024-03-23T03:46:58.012474+0000","last_unstale":"2024-03-23T03:46:59.411414+0000","last_undegraded":"2024-03-23T03:46:59.411414+0000","last_fullsized":"2024-03-23T03:46:59.411414+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:22:28.017991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970873+0000","last_change":"2024-03-23T03:46:53.955218+0000","last_active":"2024-03-23T03:46:57.970873+0000","last_peered":"2024-03-23T03:46:57.970873+0000","last_clean":"2024-03-23T03:46:57.970873+0000","last_became_active":"2024-03-23T03:46:53.954977+0000","last_became_peered":"2024-03-23T03:46:53.954977+0000","last_unstale":"2024-03-23T03:46:57.970873+0000","last_undegraded":"2024-03-23T03:46:57.970873+0000","last_fullsized":"2024-03-23T03:46:57.970873+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:12.278522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.947128+0000","last_change":"2024-03-23T03:46:53.456663+0000","last_active":"2024-03-23T03:46:57.947128+0000","last_peered":"2024-03-23T03:46:57.947128+0000","last_clean":"2024-03-23T03:46:57.947128+0000","last_became_active":"2024-03-23T03:46:53.456393+0000","last_became_peered":"2024-03-23T03:46:53.456393+0000","last_unstale":"2024-03-23T03:46:57.947128+0000","last_undegraded":"2024-03-23T03:46:57.947128+0000","last_fullsized":"2024-03-23T03:46:57.947128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:38:29.605938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.544795+0000","last_change":"2024-03-23T03:46:57.975493+0000","last_active":"2024-03-23T03:46:59.544795+0000","last_peered":"2024-03-23T03:46:59.544795+0000","last_clean":"2024-03-23T03:46:59.544795+0000","last_became_active":"2024-03-23T03:46:57.974943+0000","last_became_peered":"2024-03-23T03:46:57.974943+0000","last_unstale":"2024-03-23T03:46:59.544795+0000","last_undegraded":"2024-03-23T03:46:59.544795+0000","last_fullsized":"2024-03-23T03:46:59.544795+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:56.623233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.960709+0000","last_change":"2024-03-23T03:46:53.963022+0000","last_active":"2024-03-23T03:46:57.960709+0000","last_peered":"2024-03-23T03:46:57.960709+0000","last_clean":"2024-03-23T03:46:57.960709+0000","last_became_active":"2024-03-23T03:46:53.962270+0000","last_became_peered":"2024-03-23T03:46:53.962270+0000","last_unstale":"2024-03-23T03:46:57.960709+0000","last_undegraded":"2024-03-23T03:46:57.960709+0000","last_fullsized":"2024-03-23T03:46:57.960709+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:55:23.986885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970281+0000","last_change":"2024-03-23T03:46:53.456217+0000","last_active":"2024-03-23T03:46:57.970281+0000","last_peered":"2024-03-23T03:46:57.970281+0000","last_clean":"2024-03-23T03:46:57.970281+0000","last_became_active":"2024-03-23T03:46:53.455825+0000","last_became_peered":"2024-03-23T03:46:53.455825+0000","last_unstale":"2024-03-23T03:46:57.970281+0000","last_undegraded":"2024-03-23T03:46:57.970281+0000","last_fullsized":"2024-03-23T03:46:57.970281+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:20:09.953919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.544875+0000","last_change":"2024-03-23T03:46:57.978215+0000","last_active":"2024-03-23T03:46:59.544875+0000","last_peered":"2024-03-23T03:46:59.544875+0000","last_clean":"2024-03-23T03:46:59.544875+0000","last_became_active":"2024-03-23T03:46:57.977737+0000","last_became_peered":"2024-03-23T03:46:57.977737+0000","last_unstale":"2024-03-23T03:46:59.544875+0000","last_undegraded":"2024-03-23T03:46:59.544875+0000","last_fullsized":"2024-03-23T03:46:59.544875+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:09:10.919635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943234+0000","last_change":"2024-03-23T03:46:53.949224+0000","last_active":"2024-03-23T03:46:57.943234+0000","last_peered":"2024-03-23T03:46:57.943234+0000","last_clean":"2024-03-23T03:46:57.943234+0000","last_became_active":"2024-03-23T03:46:53.948869+0000","last_became_peered":"2024-03-23T03:46:53.948869+0000","last_unstale":"2024-03-23T03:46:57.943234+0000","last_undegraded":"2024-03-23T03:46:57.943234+0000","last_fullsized":"2024-03-23T03:46:57.943234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:49:34.782399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.948932+0000","last_change":"2024-03-23T03:46:53.461930+0000","last_active":"2024-03-23T03:46:57.948932+0000","last_peered":"2024-03-23T03:46:57.948932+0000","last_clean":"2024-03-23T03:46:57.948932+0000","last_became_active":"2024-03-23T03:46:53.461500+0000","last_became_peered":"2024-03-23T03:46:53.461500+0000","last_unstale":"2024-03-23T03:46:57.948932+0000","last_undegraded":"2024-03-23T03:46:57.948932+0000","last_fullsized":"2024-03-23T03:46:57.948932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:12:50.664977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545763+0000","last_change":"2024-03-23T03:46:57.978014+0000","last_active":"2024-03-23T03:46:59.545763+0000","last_peered":"2024-03-23T03:46:59.545763+0000","last_clean":"2024-03-23T03:46:59.545763+0000","last_became_active":"2024-03-23T03:46:57.977584+0000","last_became_peered":"2024-03-23T03:46:57.977584+0000","last_unstale":"2024-03-23T03:46:59.545763+0000","last_undegraded":"2024-03-23T03:46:59.545763+0000","last_fullsized":"2024-03-23T03:46:59.545763+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:17:26.245417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.950704+0000","last_change":"2024-03-23T03:46:53.968396+0000","last_active":"2024-03-23T03:46:57.950704+0000","last_peered":"2024-03-23T03:46:57.950704+0000","last_clean":"2024-03-23T03:46:57.950704+0000","last_became_active":"2024-03-23T03:46:53.968127+0000","last_became_peered":"2024-03-23T03:46:53.968127+0000","last_unstale":"2024-03-23T03:46:57.950704+0000","last_undegraded":"2024-03-23T03:46:57.950704+0000","last_fullsized":"2024-03-23T03:46:57.950704+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:56:29.065234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.012423+0000","last_change":"2024-03-23T03:46:53.472587+0000","last_active":"2024-03-23T03:46:58.012423+0000","last_peered":"2024-03-23T03:46:58.012423+0000","last_clean":"2024-03-23T03:46:58.012423+0000","last_became_active":"2024-03-23T03:46:53.472235+0000","last_became_peered":"2024-03-23T03:46:53.472235+0000","last_unstale":"2024-03-23T03:46:58.012423+0000","last_undegraded":"2024-03-23T03:46:58.012423+0000","last_fullsized":"2024-03-23T03:46:58.012423+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:56.874824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411831+0000","last_change":"2024-03-23T03:46:58.016199+0000","last_active":"2024-03-23T03:46:59.411831+0000","last_peered":"2024-03-23T03:46:59.411831+0000","last_clean":"2024-03-23T03:46:59.411831+0000","last_became_active":"2024-03-23T03:46:58.015478+0000","last_became_peered":"2024-03-23T03:46:58.015478+0000","last_unstale":"2024-03-23T03:46:59.411831+0000","last_undegraded":"2024-03-23T03:46:59.411831+0000","last_fullsized":"2024-03-23T03:46:59.411831+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:56.623233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951639+0000","last_change":"2024-03-23T03:46:53.955152+0000","last_active":"2024-03-23T03:46:57.951639+0000","last_peered":"2024-03-23T03:46:57.951639+0000","last_clean":"2024-03-23T03:46:57.951639+0000","last_became_active":"2024-03-23T03:46:53.953527+0000","last_became_peered":"2024-03-23T03:46:53.953527+0000","last_unstale":"2024-03-23T03:46:57.951639+0000","last_undegraded":"2024-03-23T03:46:57.951639+0000","last_fullsized":"2024-03-23T03:46:57.951639+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:01:31.453936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970187+0000","last_change":"2024-03-23T03:46:53.462993+0000","last_active":"2024-03-23T03:46:57.970187+0000","last_peered":"2024-03-23T03:46:57.970187+0000","last_clean":"2024-03-23T03:46:57.970187+0000","last_became_active":"2024-03-23T03:46:53.462438+0000","last_became_peered":"2024-03-23T03:46:53.462438+0000","last_unstale":"2024-03-23T03:46:57.970187+0000","last_undegraded":"2024-03-23T03:46:57.970187+0000","last_fullsized":"2024-03-23T03:46:57.970187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:52:38.737363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":167,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.012529+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:46:58.012529+0000","last_peered":"2024-03-23T03:46:58.012529+0000","last_clean":"2024-03-23T03:46:58.012529+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:46:58.012529+0000","last_undegraded":"2024-03-23T03:46:58.012529+0000","last_fullsized":"2024-03-23T03:46:58.012529+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.547057+0000","last_change":"2024-03-23T03:46:58.018456+0000","last_active":"2024-03-23T03:46:59.547057+0000","last_peered":"2024-03-23T03:46:59.547057+0000","last_clean":"2024-03-23T03:46:59.547057+0000","last_became_active":"2024-03-23T03:46:58.018035+0000","last_became_peered":"2024-03-23T03:46:58.018035+0000","last_unstale":"2024-03-23T03:46:59.547057+0000","last_undegraded":"2024-03-23T03:46:59.547057+0000","last_fullsized":"2024-03-23T03:46:59.547057+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:44:17.122381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017645+0000","last_change":"2024-03-23T03:46:53.970202+0000","last_active":"2024-03-23T03:46:58.017645+0000","last_peered":"2024-03-23T03:46:58.017645+0000","last_clean":"2024-03-23T03:46:58.017645+0000","last_became_active":"2024-03-23T03:46:53.969976+0000","last_became_peered":"2024-03-23T03:46:53.969976+0000","last_unstale":"2024-03-23T03:46:58.017645+0000","last_undegraded":"2024-03-23T03:46:58.017645+0000","last_fullsized":"2024-03-23T03:46:58.017645+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:43:59.541399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.947920+0000","last_change":"2024-03-23T03:46:53.478374+0000","last_active":"2024-03-23T03:46:57.947920+0000","last_peered":"2024-03-23T03:46:57.947920+0000","last_clean":"2024-03-23T03:46:57.947920+0000","last_became_active":"2024-03-23T03:46:53.478063+0000","last_became_peered":"2024-03-23T03:46:53.478063+0000","last_unstale":"2024-03-23T03:46:57.947920+0000","last_undegraded":"2024-03-23T03:46:57.947920+0000","last_fullsized":"2024-03-23T03:46:57.947920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:28:03.413525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.546398+0000","last_change":"2024-03-23T03:46:58.013147+0000","last_active":"2024-03-23T03:46:59.546398+0000","last_peered":"2024-03-23T03:46:59.546398+0000","last_clean":"2024-03-23T03:46:59.546398+0000","last_became_active":"2024-03-23T03:46:58.012728+0000","last_became_peered":"2024-03-23T03:46:58.012728+0000","last_unstale":"2024-03-23T03:46:59.546398+0000","last_undegraded":"2024-03-23T03:46:59.546398+0000","last_fullsized":"2024-03-23T03:46:59.546398+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:28:49.474199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970654+0000","last_change":"2024-03-23T03:46:53.956675+0000","last_active":"2024-03-23T03:46:57.970654+0000","last_peered":"2024-03-23T03:46:57.970654+0000","last_clean":"2024-03-23T03:46:57.970654+0000","last_became_active":"2024-03-23T03:46:53.956420+0000","last_became_peered":"2024-03-23T03:46:53.956420+0000","last_unstale":"2024-03-23T03:46:57.970654+0000","last_undegraded":"2024-03-23T03:46:57.970654+0000","last_fullsized":"2024-03-23T03:46:57.970654+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:16:39.131183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943199+0000","last_change":"2024-03-23T03:46:53.467369+0000","last_active":"2024-03-23T03:46:57.943199+0000","last_peered":"2024-03-23T03:46:57.943199+0000","last_clean":"2024-03-23T03:46:57.943199+0000","last_became_active":"2024-03-23T03:46:53.465916+0000","last_became_peered":"2024-03-23T03:46:53.465916+0000","last_unstale":"2024-03-23T03:46:57.943199+0000","last_undegraded":"2024-03-23T03:46:57.943199+0000","last_fullsized":"2024-03-23T03:46:57.943199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:56.874824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411552+0000","last_change":"2024-03-23T03:46:57.974919+0000","last_active":"2024-03-23T03:46:59.411552+0000","last_peered":"2024-03-23T03:46:59.411552+0000","last_clean":"2024-03-23T03:46:59.411552+0000","last_became_active":"2024-03-23T03:46:57.972661+0000","last_became_peered":"2024-03-23T03:46:57.972661+0000","last_unstale":"2024-03-23T03:46:59.411552+0000","last_undegraded":"2024-03-23T03:46:59.411552+0000","last_fullsized":"2024-03-23T03:46:59.411552+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:27:37.992019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.018541+0000","last_change":"2024-03-23T03:46:53.965749+0000","last_active":"2024-03-23T03:46:58.018541+0000","last_peered":"2024-03-23T03:46:58.018541+0000","last_clean":"2024-03-23T03:46:58.018541+0000","last_became_active":"2024-03-23T03:46:53.965362+0000","last_became_peered":"2024-03-23T03:46:53.965362+0000","last_unstale":"2024-03-23T03:46:58.018541+0000","last_undegraded":"2024-03-23T03:46:58.018541+0000","last_fullsized":"2024-03-23T03:46:58.018541+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:38:13.205855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.948005+0000","last_change":"2024-03-23T03:46:53.460818+0000","last_active":"2024-03-23T03:46:57.948005+0000","last_peered":"2024-03-23T03:46:57.948005+0000","last_clean":"2024-03-23T03:46:57.948005+0000","last_became_active":"2024-03-23T03:46:53.460547+0000","last_became_peered":"2024-03-23T03:46:53.460547+0000","last_unstale":"2024-03-23T03:46:57.948005+0000","last_undegraded":"2024-03-23T03:46:57.948005+0000","last_fullsized":"2024-03-23T03:46:57.948005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:49:24.952855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.424397+0000","last_change":"2024-03-23T03:46:58.014145+0000","last_active":"2024-03-23T03:46:58.424397+0000","last_peered":"2024-03-23T03:46:58.424397+0000","last_clean":"2024-03-23T03:46:58.424397+0000","last_became_active":"2024-03-23T03:46:58.013649+0000","last_became_peered":"2024-03-23T03:46:58.013649+0000","last_unstale":"2024-03-23T03:46:58.424397+0000","last_undegraded":"2024-03-23T03:46:58.424397+0000","last_fullsized":"2024-03-23T03:46:58.424397+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:24:56.623233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017437+0000","last_change":"2024-03-23T03:46:53.966787+0000","last_active":"2024-03-23T03:46:58.017437+0000","last_peered":"2024-03-23T03:46:58.017437+0000","last_clean":"2024-03-23T03:46:58.017437+0000","last_became_active":"2024-03-23T03:46:53.966135+0000","last_became_peered":"2024-03-23T03:46:53.966135+0000","last_unstale":"2024-03-23T03:46:58.017437+0000","last_undegraded":"2024-03-23T03:46:58.017437+0000","last_fullsized":"2024-03-23T03:46:58.017437+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:28:12.278522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.964838+0000","last_change":"2024-03-23T03:46:53.473134+0000","last_active":"2024-03-23T03:46:57.964838+0000","last_peered":"2024-03-23T03:46:57.964838+0000","last_clean":"2024-03-23T03:46:57.964838+0000","last_became_active":"2024-03-23T03:46:53.472902+0000","last_became_peered":"2024-03-23T03:46:53.472902+0000","last_unstale":"2024-03-23T03:46:57.964838+0000","last_undegraded":"2024-03-23T03:46:57.964838+0000","last_fullsized":"2024-03-23T03:46:57.964838+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:42:16.552281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.423299+0000","last_change":"2024-03-23T03:46:57.976918+0000","last_active":"2024-03-23T03:46:58.423299+0000","last_peered":"2024-03-23T03:46:58.423299+0000","last_clean":"2024-03-23T03:46:58.423299+0000","last_became_active":"2024-03-23T03:46:57.976542+0000","last_became_peered":"2024-03-23T03:46:57.976542+0000","last_unstale":"2024-03-23T03:46:58.423299+0000","last_undegraded":"2024-03-23T03:46:58.423299+0000","last_fullsized":"2024-03-23T03:46:58.423299+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:49:33.207155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017030+0000","last_change":"2024-03-23T03:46:53.964676+0000","last_active":"2024-03-23T03:46:58.017030+0000","last_peered":"2024-03-23T03:46:58.017030+0000","last_clean":"2024-03-23T03:46:58.017030+0000","last_became_active":"2024-03-23T03:46:53.964382+0000","last_became_peered":"2024-03-23T03:46:53.964382+0000","last_unstale":"2024-03-23T03:46:58.017030+0000","last_undegraded":"2024-03-23T03:46:58.017030+0000","last_fullsized":"2024-03-23T03:46:58.017030+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:34:41.509208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.947240+0000","last_change":"2024-03-23T03:46:53.459242+0000","last_active":"2024-03-23T03:46:57.947240+0000","last_peered":"2024-03-23T03:46:57.947240+0000","last_clean":"2024-03-23T03:46:57.947240+0000","last_became_active":"2024-03-23T03:46:53.458136+0000","last_became_peered":"2024-03-23T03:46:53.458136+0000","last_unstale":"2024-03-23T03:46:57.947240+0000","last_undegraded":"2024-03-23T03:46:57.947240+0000","last_fullsized":"2024-03-23T03:46:57.947240+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:42:13.208524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411874+0000","last_change":"2024-03-23T03:46:57.968727+0000","last_active":"2024-03-23T03:46:59.411874+0000","last_peered":"2024-03-23T03:46:59.411874+0000","last_clean":"2024-03-23T03:46:59.411874+0000","last_became_active":"2024-03-23T03:46:57.968336+0000","last_became_peered":"2024-03-23T03:46:57.968336+0000","last_unstale":"2024-03-23T03:46:59.411874+0000","last_undegraded":"2024-03-23T03:46:59.411874+0000","last_fullsized":"2024-03-23T03:46:59.411874+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:17:26.245417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951907+0000","last_change":"2024-03-23T03:46:53.967303+0000","last_active":"2024-03-23T03:46:57.951907+0000","last_peered":"2024-03-23T03:46:57.951907+0000","last_clean":"2024-03-23T03:46:57.951907+0000","last_became_active":"2024-03-23T03:46:53.966903+0000","last_became_peered":"2024-03-23T03:46:53.966903+0000","last_unstale":"2024-03-23T03:46:57.951907+0000","last_undegraded":"2024-03-23T03:46:57.951907+0000","last_fullsized":"2024-03-23T03:46:57.951907+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:18:17.089475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951906+0000","last_change":"2024-03-23T03:46:53.479049+0000","last_active":"2024-03-23T03:46:57.951906+0000","last_peered":"2024-03-23T03:46:57.951906+0000","last_clean":"2024-03-23T03:46:57.951906+0000","last_became_active":"2024-03-23T03:46:53.478779+0000","last_became_peered":"2024-03-23T03:46:53.478779+0000","last_unstale":"2024-03-23T03:46:57.951906+0000","last_undegraded":"2024-03-23T03:46:57.951906+0000","last_fullsized":"2024-03-23T03:46:57.951906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:28:03.413525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.547121+0000","last_change":"2024-03-23T03:46:58.018779+0000","last_active":"2024-03-23T03:46:59.547121+0000","last_peered":"2024-03-23T03:46:59.547121+0000","last_clean":"2024-03-23T03:46:59.547121+0000","last_became_active":"2024-03-23T03:46:58.018342+0000","last_became_peered":"2024-03-23T03:46:58.018342+0000","last_unstale":"2024-03-23T03:46:59.547121+0000","last_undegraded":"2024-03-23T03:46:59.547121+0000","last_fullsized":"2024-03-23T03:46:59.547121+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:49:30.580961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.016869+0000","last_change":"2024-03-23T03:46:53.966087+0000","last_active":"2024-03-23T03:46:58.016869+0000","last_peered":"2024-03-23T03:46:58.016869+0000","last_clean":"2024-03-23T03:46:58.016869+0000","last_became_active":"2024-03-23T03:46:53.965556+0000","last_became_peered":"2024-03-23T03:46:53.965556+0000","last_unstale":"2024-03-23T03:46:58.016869+0000","last_undegraded":"2024-03-23T03:46:58.016869+0000","last_fullsized":"2024-03-23T03:46:58.016869+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:35:49.855085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970252+0000","last_change":"2024-03-23T03:46:53.468499+0000","last_active":"2024-03-23T03:46:57.970252+0000","last_peered":"2024-03-23T03:46:57.970252+0000","last_clean":"2024-03-23T03:46:57.970252+0000","last_became_active":"2024-03-23T03:46:53.466565+0000","last_became_peered":"2024-03-23T03:46:53.466565+0000","last_unstale":"2024-03-23T03:46:57.970252+0000","last_undegraded":"2024-03-23T03:46:57.970252+0000","last_fullsized":"2024-03-23T03:46:57.970252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:38:29.605938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.546719+0000","last_change":"2024-03-23T03:46:58.013721+0000","last_active":"2024-03-23T03:46:59.546719+0000","last_peered":"2024-03-23T03:46:59.546719+0000","last_clean":"2024-03-23T03:46:59.546719+0000","last_became_active":"2024-03-23T03:46:58.012503+0000","last_became_peered":"2024-03-23T03:46:58.012503+0000","last_unstale":"2024-03-23T03:46:59.546719+0000","last_undegraded":"2024-03-23T03:46:59.546719+0000","last_fullsized":"2024-03-23T03:46:59.546719+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:54:55.461256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951186+0000","last_change":"2024-03-23T03:46:53.952982+0000","last_active":"2024-03-23T03:46:57.951186+0000","last_peered":"2024-03-23T03:46:57.951186+0000","last_clean":"2024-03-23T03:46:57.951186+0000","last_became_active":"2024-03-23T03:46:53.952707+0000","last_became_peered":"2024-03-23T03:46:53.952707+0000","last_unstale":"2024-03-23T03:46:57.951186+0000","last_undegraded":"2024-03-23T03:46:57.951186+0000","last_fullsized":"2024-03-23T03:46:57.951186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:01:24.719476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952254+0000","last_change":"2024-03-23T03:46:53.472959+0000","last_active":"2024-03-23T03:46:57.952254+0000","last_peered":"2024-03-23T03:46:57.952254+0000","last_clean":"2024-03-23T03:46:57.952254+0000","last_became_active":"2024-03-23T03:46:53.472652+0000","last_became_peered":"2024-03-23T03:46:53.472652+0000","last_unstale":"2024-03-23T03:46:57.952254+0000","last_undegraded":"2024-03-23T03:46:57.952254+0000","last_fullsized":"2024-03-23T03:46:57.952254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:48:00.747717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.426089+0000","last_change":"2024-03-23T03:46:58.019621+0000","last_active":"2024-03-23T03:46:58.426089+0000","last_peered":"2024-03-23T03:46:58.426089+0000","last_clean":"2024-03-23T03:46:58.426089+0000","last_became_active":"2024-03-23T03:46:58.018726+0000","last_became_peered":"2024-03-23T03:46:58.018726+0000","last_unstale":"2024-03-23T03:46:58.426089+0000","last_undegraded":"2024-03-23T03:46:58.426089+0000","last_fullsized":"2024-03-23T03:46:58.426089+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:18:24.184664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017563+0000","last_change":"2024-03-23T03:46:53.954977+0000","last_active":"2024-03-23T03:46:58.017563+0000","last_peered":"2024-03-23T03:46:58.017563+0000","last_clean":"2024-03-23T03:46:58.017563+0000","last_became_active":"2024-03-23T03:46:53.954667+0000","last_became_peered":"2024-03-23T03:46:53.954667+0000","last_unstale":"2024-03-23T03:46:58.017563+0000","last_undegraded":"2024-03-23T03:46:58.017563+0000","last_fullsized":"2024-03-23T03:46:58.017563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:16.232161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"26'10","reported_seq":25,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:47:12.152821+0000","last_change":"2024-03-23T03:46:53.469497+0000","last_active":"2024-03-23T03:47:12.152821+0000","last_peered":"2024-03-23T03:47:12.152821+0000","last_clean":"2024-03-23T03:47:12.152821+0000","last_became_active":"2024-03-23T03:46:53.469139+0000","last_became_peered":"2024-03-23T03:46:53.469139+0000","last_unstale":"2024-03-23T03:47:12.152821+0000","last_undegraded":"2024-03-23T03:47:12.152821+0000","last_fullsized":"2024-03-23T03:47:12.152821+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:42:13.208524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3795,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545062+0000","last_change":"2024-03-23T03:46:57.965959+0000","last_active":"2024-03-23T03:46:59.545062+0000","last_peered":"2024-03-23T03:46:59.545062+0000","last_clean":"2024-03-23T03:46:59.545062+0000","last_became_active":"2024-03-23T03:46:57.965418+0000","last_became_peered":"2024-03-23T03:46:57.965418+0000","last_unstale":"2024-03-23T03:46:59.545062+0000","last_undegraded":"2024-03-23T03:46:59.545062+0000","last_fullsized":"2024-03-23T03:46:59.545062+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:57:19.038735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.962037+0000","last_change":"2024-03-23T03:46:53.946234+0000","last_active":"2024-03-23T03:46:57.962037+0000","last_peered":"2024-03-23T03:46:57.962037+0000","last_clean":"2024-03-23T03:46:57.962037+0000","last_became_active":"2024-03-23T03:46:53.945188+0000","last_became_peered":"2024-03-23T03:46:53.945188+0000","last_unstale":"2024-03-23T03:46:57.962037+0000","last_undegraded":"2024-03-23T03:46:57.962037+0000","last_fullsized":"2024-03-23T03:46:57.962037+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:29:34.917375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943105+0000","last_change":"2024-03-23T03:46:53.474846+0000","last_active":"2024-03-23T03:46:57.943105+0000","last_peered":"2024-03-23T03:46:57.943105+0000","last_clean":"2024-03-23T03:46:57.943105+0000","last_became_active":"2024-03-23T03:46:53.474561+0000","last_became_peered":"2024-03-23T03:46:53.474561+0000","last_unstale":"2024-03-23T03:46:57.943105+0000","last_undegraded":"2024-03-23T03:46:57.943105+0000","last_fullsized":"2024-03-23T03:46:57.943105+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:40:20.814830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.424404+0000","last_change":"2024-03-23T03:46:57.974187+0000","last_active":"2024-03-23T03:46:58.424404+0000","last_peered":"2024-03-23T03:46:58.424404+0000","last_clean":"2024-03-23T03:46:58.424404+0000","last_became_active":"2024-03-23T03:46:57.973629+0000","last_became_peered":"2024-03-23T03:46:57.973629+0000","last_unstale":"2024-03-23T03:46:58.424404+0000","last_undegraded":"2024-03-23T03:46:58.424404+0000","last_fullsized":"2024-03-23T03:46:58.424404+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:32:46.731709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.954425+0000","last_change":"2024-03-23T03:46:53.958810+0000","last_active":"2024-03-23T03:46:57.954425+0000","last_peered":"2024-03-23T03:46:57.954425+0000","last_clean":"2024-03-23T03:46:57.954425+0000","last_became_active":"2024-03-23T03:46:53.958534+0000","last_became_peered":"2024-03-23T03:46:53.958534+0000","last_unstale":"2024-03-23T03:46:57.954425+0000","last_undegraded":"2024-03-23T03:46:57.954425+0000","last_fullsized":"2024-03-23T03:46:57.954425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:01:31.453936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970435+0000","last_change":"2024-03-23T03:46:53.472164+0000","last_active":"2024-03-23T03:46:57.970435+0000","last_peered":"2024-03-23T03:46:57.970435+0000","last_clean":"2024-03-23T03:46:57.970435+0000","last_became_active":"2024-03-23T03:46:53.471488+0000","last_became_peered":"2024-03-23T03:46:53.471488+0000","last_unstale":"2024-03-23T03:46:57.970435+0000","last_undegraded":"2024-03-23T03:46:57.970435+0000","last_fullsized":"2024-03-23T03:46:57.970435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:56.874824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.544253+0000","last_change":"2024-03-23T03:46:57.967928+0000","last_active":"2024-03-23T03:46:59.544253+0000","last_peered":"2024-03-23T03:46:59.544253+0000","last_clean":"2024-03-23T03:46:59.544253+0000","last_became_active":"2024-03-23T03:46:57.967377+0000","last_became_peered":"2024-03-23T03:46:57.967377+0000","last_unstale":"2024-03-23T03:46:59.544253+0000","last_undegraded":"2024-03-23T03:46:59.544253+0000","last_fullsized":"2024-03-23T03:46:59.544253+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:58:12.948095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970395+0000","last_change":"2024-03-23T03:46:53.954230+0000","last_active":"2024-03-23T03:46:57.970395+0000","last_peered":"2024-03-23T03:46:57.970395+0000","last_clean":"2024-03-23T03:46:57.970395+0000","last_became_active":"2024-03-23T03:46:53.953991+0000","last_became_peered":"2024-03-23T03:46:53.953991+0000","last_unstale":"2024-03-23T03:46:57.970395+0000","last_undegraded":"2024-03-23T03:46:57.970395+0000","last_fullsized":"2024-03-23T03:46:57.970395+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:16.232161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.953457+0000","last_change":"2024-03-23T03:46:53.475516+0000","last_active":"2024-03-23T03:46:57.953457+0000","last_peered":"2024-03-23T03:46:57.953457+0000","last_clean":"2024-03-23T03:46:57.953457+0000","last_became_active":"2024-03-23T03:46:53.475185+0000","last_became_peered":"2024-03-23T03:46:53.475185+0000","last_unstale":"2024-03-23T03:46:57.953457+0000","last_undegraded":"2024-03-23T03:46:57.953457+0000","last_fullsized":"2024-03-23T03:46:57.953457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:05:58.548539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545880+0000","last_change":"2024-03-23T03:46:58.018887+0000","last_active":"2024-03-23T03:46:59.545880+0000","last_peered":"2024-03-23T03:46:59.545880+0000","last_clean":"2024-03-23T03:46:59.545880+0000","last_became_active":"2024-03-23T03:46:58.018514+0000","last_became_peered":"2024-03-23T03:46:58.018514+0000","last_unstale":"2024-03-23T03:46:59.545880+0000","last_undegraded":"2024-03-23T03:46:59.545880+0000","last_fullsized":"2024-03-23T03:46:59.545880+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:51:12.276163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.411517+0000","last_change":"2024-03-23T03:46:58.015847+0000","last_active":"2024-03-23T03:46:59.411517+0000","last_peered":"2024-03-23T03:46:59.411517+0000","last_clean":"2024-03-23T03:46:59.411517+0000","last_became_active":"2024-03-23T03:46:58.015471+0000","last_became_peered":"2024-03-23T03:46:58.015471+0000","last_unstale":"2024-03-23T03:46:59.411517+0000","last_undegraded":"2024-03-23T03:46:59.411517+0000","last_fullsized":"2024-03-23T03:46:59.411517+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:34:34.288987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943156+0000","last_change":"2024-03-23T03:46:53.952670+0000","last_active":"2024-03-23T03:46:57.943156+0000","last_peered":"2024-03-23T03:46:57.943156+0000","last_clean":"2024-03-23T03:46:57.943156+0000","last_became_active":"2024-03-23T03:46:53.951520+0000","last_became_peered":"2024-03-23T03:46:53.951520+0000","last_unstale":"2024-03-23T03:46:57.943156+0000","last_undegraded":"2024-03-23T03:46:57.943156+0000","last_fullsized":"2024-03-23T03:46:57.943156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:55.749189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.968564+0000","last_change":"2024-03-23T03:46:53.475560+0000","last_active":"2024-03-23T03:46:57.968564+0000","last_peered":"2024-03-23T03:46:57.968564+0000","last_clean":"2024-03-23T03:46:57.968564+0000","last_became_active":"2024-03-23T03:46:53.468090+0000","last_became_peered":"2024-03-23T03:46:53.468090+0000","last_unstale":"2024-03-23T03:46:57.968564+0000","last_undegraded":"2024-03-23T03:46:57.968564+0000","last_fullsized":"2024-03-23T03:46:57.968564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:23:34.472803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.548770+0000","last_change":"2024-03-23T03:46:57.974834+0000","last_active":"2024-03-23T03:46:59.548770+0000","last_peered":"2024-03-23T03:46:59.548770+0000","last_clean":"2024-03-23T03:46:59.548770+0000","last_became_active":"2024-03-23T03:46:57.974450+0000","last_became_peered":"2024-03-23T03:46:57.974450+0000","last_unstale":"2024-03-23T03:46:59.548770+0000","last_undegraded":"2024-03-23T03:46:59.548770+0000","last_fullsized":"2024-03-23T03:46:59.548770+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:01.676599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.954004+0000","last_change":"2024-03-23T03:46:53.963896+0000","last_active":"2024-03-23T03:46:57.954004+0000","last_peered":"2024-03-23T03:46:57.954004+0000","last_clean":"2024-03-23T03:46:57.954004+0000","last_became_active":"2024-03-23T03:46:53.963072+0000","last_became_peered":"2024-03-23T03:46:53.963072+0000","last_unstale":"2024-03-23T03:46:57.954004+0000","last_undegraded":"2024-03-23T03:46:57.954004+0000","last_fullsized":"2024-03-23T03:46:57.954004+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:49:34.782399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":15,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.961727+0000","last_change":"2024-03-23T03:46:53.475321+0000","last_active":"2024-03-23T03:46:57.961727+0000","last_peered":"2024-03-23T03:46:57.961727+0000","last_clean":"2024-03-23T03:46:57.961727+0000","last_became_active":"2024-03-23T03:46:53.475119+0000","last_became_peered":"2024-03-23T03:46:53.475119+0000","last_unstale":"2024-03-23T03:46:57.961727+0000","last_undegraded":"2024-03-23T03:46:57.961727+0000","last_fullsized":"2024-03-23T03:46:57.961727+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:48:00.747717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952537+0000","last_change":"2024-03-23T03:46:53.475209+0000","last_active":"2024-03-23T03:46:57.952537+0000","last_peered":"2024-03-23T03:46:57.952537+0000","last_clean":"2024-03-23T03:46:57.952537+0000","last_became_active":"2024-03-23T03:46:53.474046+0000","last_became_peered":"2024-03-23T03:46:53.474046+0000","last_unstale":"2024-03-23T03:46:57.952537+0000","last_undegraded":"2024-03-23T03:46:57.952537+0000","last_fullsized":"2024-03-23T03:46:57.952537+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:04:07.631198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952167+0000","last_change":"2024-03-23T03:46:53.961251+0000","last_active":"2024-03-23T03:46:57.952167+0000","last_peered":"2024-03-23T03:46:57.952167+0000","last_clean":"2024-03-23T03:46:57.952167+0000","last_became_active":"2024-03-23T03:46:53.960887+0000","last_became_peered":"2024-03-23T03:46:53.960887+0000","last_unstale":"2024-03-23T03:46:57.952167+0000","last_undegraded":"2024-03-23T03:46:57.952167+0000","last_fullsized":"2024-03-23T03:46:57.952167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:51.878165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.545740+0000","last_change":"2024-03-23T03:46:57.969555+0000","last_active":"2024-03-23T03:46:59.545740+0000","last_peered":"2024-03-23T03:46:59.545740+0000","last_clean":"2024-03-23T03:46:59.545740+0000","last_became_active":"2024-03-23T03:46:57.969163+0000","last_became_peered":"2024-03-23T03:46:57.969163+0000","last_unstale":"2024-03-23T03:46:59.545740+0000","last_undegraded":"2024-03-23T03:46:59.545740+0000","last_fullsized":"2024-03-23T03:46:59.545740+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:09.677468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":16,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952350+0000","last_change":"2024-03-23T03:46:53.463135+0000","last_active":"2024-03-23T03:46:57.952350+0000","last_peered":"2024-03-23T03:46:57.952350+0000","last_clean":"2024-03-23T03:46:57.952350+0000","last_became_active":"2024-03-23T03:46:53.462844+0000","last_became_peered":"2024-03-23T03:46:53.462844+0000","last_unstale":"2024-03-23T03:46:57.952350+0000","last_undegraded":"2024-03-23T03:46:57.952350+0000","last_fullsized":"2024-03-23T03:46:57.952350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:52.401840+0000","last_clean_scrub_stamp":"2024-03-23T03:46:52.401840+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:39:13.578639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.960303+0000","last_change":"2024-03-23T03:46:53.935183+0000","last_active":"2024-03-23T03:46:57.960303+0000","last_peered":"2024-03-23T03:46:57.960303+0000","last_clean":"2024-03-23T03:46:57.960303+0000","last_became_active":"2024-03-23T03:46:53.934902+0000","last_became_peered":"2024-03-23T03:46:53.934902+0000","last_unstale":"2024-03-23T03:46:57.960303+0000","last_undegraded":"2024-03-23T03:46:57.960303+0000","last_fullsized":"2024-03-23T03:46:57.960303+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:19:03.389747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-03-23T03:46:59.546902+0000","last_change":"2024-03-23T03:46:57.971524+0000","last_active":"2024-03-23T03:46:59.546902+0000","last_peered":"2024-03-23T03:46:59.546902+0000","last_clean":"2024-03-23T03:46:59.546902+0000","last_became_active":"2024-03-23T03:46:57.967540+0000","last_became_peered":"2024-03-23T03:46:57.967540+0000","last_unstale":"2024-03-23T03:46:59.546902+0000","last_undegraded":"2024-03-23T03:46:59.546902+0000","last_fullsized":"2024-03-23T03:46:59.546902+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:56.926084+0000","last_clean_scrub_stamp":"2024-03-23T03:46:56.926084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:36:58.375101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951620+0000","last_change":"2024-03-23T03:46:53.966166+0000","last_active":"2024-03-23T03:46:57.951620+0000","last_peered":"2024-03-23T03:46:57.951620+0000","last_clean":"2024-03-23T03:46:57.951620+0000","last_became_active":"2024-03-23T03:46:53.965850+0000","last_became_peered":"2024-03-23T03:46:53.965850+0000","last_unstale":"2024-03-23T03:46:57.951620+0000","last_undegraded":"2024-03-23T03:46:57.951620+0000","last_fullsized":"2024-03-23T03:46:57.951620+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:35:38.384483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.948826+0000","last_change":"2024-03-23T03:46:53.954723+0000","last_active":"2024-03-23T03:46:57.948826+0000","last_peered":"2024-03-23T03:46:57.948826+0000","last_clean":"2024-03-23T03:46:57.948826+0000","last_became_active":"2024-03-23T03:46:53.954419+0000","last_became_peered":"2024-03-23T03:46:53.954419+0000","last_unstale":"2024-03-23T03:46:57.948826+0000","last_undegraded":"2024-03-23T03:46:57.948826+0000","last_fullsized":"2024-03-23T03:46:57.948826+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:55:23.986885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970164+0000","last_change":"2024-03-23T03:46:53.950337+0000","last_active":"2024-03-23T03:46:57.970164+0000","last_peered":"2024-03-23T03:46:57.970164+0000","last_clean":"2024-03-23T03:46:57.970164+0000","last_became_active":"2024-03-23T03:46:53.949979+0000","last_became_peered":"2024-03-23T03:46:53.949979+0000","last_unstale":"2024-03-23T03:46:57.970164+0000","last_undegraded":"2024-03-23T03:46:57.970164+0000","last_fullsized":"2024-03-23T03:46:57.970164+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:36.598727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952086+0000","last_change":"2024-03-23T03:46:53.957785+0000","last_active":"2024-03-23T03:46:57.952086+0000","last_peered":"2024-03-23T03:46:57.952086+0000","last_clean":"2024-03-23T03:46:57.952086+0000","last_became_active":"2024-03-23T03:46:53.956251+0000","last_became_peered":"2024-03-23T03:46:53.956251+0000","last_unstale":"2024-03-23T03:46:57.952086+0000","last_undegraded":"2024-03-23T03:46:57.952086+0000","last_fullsized":"2024-03-23T03:46:57.952086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:49:34.782399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.964730+0000","last_change":"2024-03-23T03:46:53.950745+0000","last_active":"2024-03-23T03:46:57.964730+0000","last_peered":"2024-03-23T03:46:57.964730+0000","last_clean":"2024-03-23T03:46:57.964730+0000","last_became_active":"2024-03-23T03:46:53.950592+0000","last_became_peered":"2024-03-23T03:46:53.950592+0000","last_unstale":"2024-03-23T03:46:57.964730+0000","last_undegraded":"2024-03-23T03:46:57.964730+0000","last_fullsized":"2024-03-23T03:46:57.964730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:16:39.131183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.942799+0000","last_change":"2024-03-23T03:46:53.949596+0000","last_active":"2024-03-23T03:46:57.942799+0000","last_peered":"2024-03-23T03:46:57.942799+0000","last_clean":"2024-03-23T03:46:57.942799+0000","last_became_active":"2024-03-23T03:46:53.949298+0000","last_became_peered":"2024-03-23T03:46:53.949298+0000","last_unstale":"2024-03-23T03:46:57.942799+0000","last_undegraded":"2024-03-23T03:46:57.942799+0000","last_fullsized":"2024-03-23T03:46:57.942799+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:47.801092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952735+0000","last_change":"2024-03-23T03:46:53.961179+0000","last_active":"2024-03-23T03:46:57.952735+0000","last_peered":"2024-03-23T03:46:57.952735+0000","last_clean":"2024-03-23T03:46:57.952735+0000","last_became_active":"2024-03-23T03:46:53.960616+0000","last_became_peered":"2024-03-23T03:46:53.960616+0000","last_unstale":"2024-03-23T03:46:57.952735+0000","last_undegraded":"2024-03-23T03:46:57.952735+0000","last_fullsized":"2024-03-23T03:46:57.952735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:51:48.524978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.971878+0000","last_change":"2024-03-23T03:46:53.951801+0000","last_active":"2024-03-23T03:46:57.971878+0000","last_peered":"2024-03-23T03:46:57.971878+0000","last_clean":"2024-03-23T03:46:57.971878+0000","last_became_active":"2024-03-23T03:46:53.951529+0000","last_became_peered":"2024-03-23T03:46:53.951529+0000","last_unstale":"2024-03-23T03:46:57.971878+0000","last_undegraded":"2024-03-23T03:46:57.971878+0000","last_fullsized":"2024-03-23T03:46:57.971878+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:36.598727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.968476+0000","last_change":"2024-03-23T03:46:53.950898+0000","last_active":"2024-03-23T03:46:57.968476+0000","last_peered":"2024-03-23T03:46:57.968476+0000","last_clean":"2024-03-23T03:46:57.968476+0000","last_became_active":"2024-03-23T03:46:53.950232+0000","last_became_peered":"2024-03-23T03:46:53.950232+0000","last_unstale":"2024-03-23T03:46:57.968476+0000","last_undegraded":"2024-03-23T03:46:57.968476+0000","last_fullsized":"2024-03-23T03:46:57.968476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:51.878165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.961869+0000","last_change":"2024-03-23T03:46:53.939297+0000","last_active":"2024-03-23T03:46:57.961869+0000","last_peered":"2024-03-23T03:46:57.961869+0000","last_clean":"2024-03-23T03:46:57.961869+0000","last_became_active":"2024-03-23T03:46:53.939052+0000","last_became_peered":"2024-03-23T03:46:53.939052+0000","last_unstale":"2024-03-23T03:46:57.961869+0000","last_undegraded":"2024-03-23T03:46:57.961869+0000","last_fullsized":"2024-03-23T03:46:57.961869+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:16.232161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970145+0000","last_change":"2024-03-23T03:46:53.953978+0000","last_active":"2024-03-23T03:46:57.970145+0000","last_peered":"2024-03-23T03:46:57.970145+0000","last_clean":"2024-03-23T03:46:57.970145+0000","last_became_active":"2024-03-23T03:46:53.953703+0000","last_became_peered":"2024-03-23T03:46:53.953703+0000","last_unstale":"2024-03-23T03:46:57.970145+0000","last_undegraded":"2024-03-23T03:46:57.970145+0000","last_fullsized":"2024-03-23T03:46:57.970145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:50:10.900452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951969+0000","last_change":"2024-03-23T03:46:53.969252+0000","last_active":"2024-03-23T03:46:57.951969+0000","last_peered":"2024-03-23T03:46:57.951969+0000","last_clean":"2024-03-23T03:46:57.951969+0000","last_became_active":"2024-03-23T03:46:53.968966+0000","last_became_peered":"2024-03-23T03:46:53.968966+0000","last_unstale":"2024-03-23T03:46:57.951969+0000","last_undegraded":"2024-03-23T03:46:57.951969+0000","last_fullsized":"2024-03-23T03:46:57.951969+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:49.329280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.960849+0000","last_change":"2024-03-23T03:46:53.965172+0000","last_active":"2024-03-23T03:46:57.960849+0000","last_peered":"2024-03-23T03:46:57.960849+0000","last_clean":"2024-03-23T03:46:57.960849+0000","last_became_active":"2024-03-23T03:46:53.963673+0000","last_became_peered":"2024-03-23T03:46:53.963673+0000","last_unstale":"2024-03-23T03:46:57.960849+0000","last_undegraded":"2024-03-23T03:46:57.960849+0000","last_fullsized":"2024-03-23T03:46:57.960849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:52:35.361467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017879+0000","last_change":"2024-03-23T03:46:53.952664+0000","last_active":"2024-03-23T03:46:58.017879+0000","last_peered":"2024-03-23T03:46:58.017879+0000","last_clean":"2024-03-23T03:46:58.017879+0000","last_became_active":"2024-03-23T03:46:53.952264+0000","last_became_peered":"2024-03-23T03:46:53.952264+0000","last_unstale":"2024-03-23T03:46:58.017879+0000","last_undegraded":"2024-03-23T03:46:58.017879+0000","last_fullsized":"2024-03-23T03:46:58.017879+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:41:51.878165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.947849+0000","last_change":"2024-03-23T03:46:53.965686+0000","last_active":"2024-03-23T03:46:57.947849+0000","last_peered":"2024-03-23T03:46:57.947849+0000","last_clean":"2024-03-23T03:46:57.947849+0000","last_became_active":"2024-03-23T03:46:53.965321+0000","last_became_peered":"2024-03-23T03:46:53.965321+0000","last_unstale":"2024-03-23T03:46:57.947849+0000","last_undegraded":"2024-03-23T03:46:57.947849+0000","last_fullsized":"2024-03-23T03:46:57.947849+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:29:34.917375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970735+0000","last_change":"2024-03-23T03:46:53.973376+0000","last_active":"2024-03-23T03:46:57.970735+0000","last_peered":"2024-03-23T03:46:57.970735+0000","last_clean":"2024-03-23T03:46:57.970735+0000","last_became_active":"2024-03-23T03:46:53.973131+0000","last_became_peered":"2024-03-23T03:46:53.973131+0000","last_unstale":"2024-03-23T03:46:57.970735+0000","last_undegraded":"2024-03-23T03:46:57.970735+0000","last_fullsized":"2024-03-23T03:46:57.970735+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:01:04.097184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.968336+0000","last_change":"2024-03-23T03:46:53.964244+0000","last_active":"2024-03-23T03:46:57.968336+0000","last_peered":"2024-03-23T03:46:57.968336+0000","last_clean":"2024-03-23T03:46:57.968336+0000","last_became_active":"2024-03-23T03:46:53.963926+0000","last_became_peered":"2024-03-23T03:46:53.963926+0000","last_unstale":"2024-03-23T03:46:57.968336+0000","last_undegraded":"2024-03-23T03:46:57.968336+0000","last_fullsized":"2024-03-23T03:46:57.968336+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:49.329280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943357+0000","last_change":"2024-03-23T03:46:53.938653+0000","last_active":"2024-03-23T03:46:57.943357+0000","last_peered":"2024-03-23T03:46:57.943357+0000","last_clean":"2024-03-23T03:46:57.943357+0000","last_became_active":"2024-03-23T03:46:53.938323+0000","last_became_peered":"2024-03-23T03:46:53.938323+0000","last_unstale":"2024-03-23T03:46:57.943357+0000","last_undegraded":"2024-03-23T03:46:57.943357+0000","last_fullsized":"2024-03-23T03:46:57.943357+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:49.329280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951110+0000","last_change":"2024-03-23T03:46:53.956639+0000","last_active":"2024-03-23T03:46:57.951110+0000","last_peered":"2024-03-23T03:46:57.951110+0000","last_clean":"2024-03-23T03:46:57.951110+0000","last_became_active":"2024-03-23T03:46:53.956097+0000","last_became_peered":"2024-03-23T03:46:53.956097+0000","last_unstale":"2024-03-23T03:46:57.951110+0000","last_undegraded":"2024-03-23T03:46:57.951110+0000","last_fullsized":"2024-03-23T03:46:57.951110+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:03:46.978451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951858+0000","last_change":"2024-03-23T03:46:53.964641+0000","last_active":"2024-03-23T03:46:57.951858+0000","last_peered":"2024-03-23T03:46:57.951858+0000","last_clean":"2024-03-23T03:46:57.951858+0000","last_became_active":"2024-03-23T03:46:53.964167+0000","last_became_peered":"2024-03-23T03:46:53.964167+0000","last_unstale":"2024-03-23T03:46:57.951858+0000","last_undegraded":"2024-03-23T03:46:57.951858+0000","last_fullsized":"2024-03-23T03:46:57.951858+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:55.749189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.965170+0000","last_change":"2024-03-23T03:46:53.962737+0000","last_active":"2024-03-23T03:46:57.965170+0000","last_peered":"2024-03-23T03:46:57.965170+0000","last_clean":"2024-03-23T03:46:57.965170+0000","last_became_active":"2024-03-23T03:46:53.962229+0000","last_became_peered":"2024-03-23T03:46:53.962229+0000","last_unstale":"2024-03-23T03:46:57.965170+0000","last_undegraded":"2024-03-23T03:46:57.965170+0000","last_fullsized":"2024-03-23T03:46:57.965170+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:59.999542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.953048+0000","last_change":"2024-03-23T03:46:53.971706+0000","last_active":"2024-03-23T03:46:57.953048+0000","last_peered":"2024-03-23T03:46:57.953048+0000","last_clean":"2024-03-23T03:46:57.953048+0000","last_became_active":"2024-03-23T03:46:53.971525+0000","last_became_peered":"2024-03-23T03:46:53.971525+0000","last_unstale":"2024-03-23T03:46:57.953048+0000","last_undegraded":"2024-03-23T03:46:57.953048+0000","last_fullsized":"2024-03-23T03:46:57.953048+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:55:54.119888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.952505+0000","last_change":"2024-03-23T03:46:53.970600+0000","last_active":"2024-03-23T03:46:57.952505+0000","last_peered":"2024-03-23T03:46:57.952505+0000","last_clean":"2024-03-23T03:46:57.952505+0000","last_became_active":"2024-03-23T03:46:53.970340+0000","last_became_peered":"2024-03-23T03:46:53.970340+0000","last_unstale":"2024-03-23T03:46:57.952505+0000","last_undegraded":"2024-03-23T03:46:57.952505+0000","last_fullsized":"2024-03-23T03:46:57.952505+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:48:32.346809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017965+0000","last_change":"2024-03-23T03:46:53.971448+0000","last_active":"2024-03-23T03:46:58.017965+0000","last_peered":"2024-03-23T03:46:58.017965+0000","last_clean":"2024-03-23T03:46:58.017965+0000","last_became_active":"2024-03-23T03:46:53.971152+0000","last_became_peered":"2024-03-23T03:46:53.971152+0000","last_unstale":"2024-03-23T03:46:58.017965+0000","last_undegraded":"2024-03-23T03:46:58.017965+0000","last_fullsized":"2024-03-23T03:46:58.017965+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:36.598727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.953820+0000","last_change":"2024-03-23T03:46:53.944836+0000","last_active":"2024-03-23T03:46:57.953820+0000","last_peered":"2024-03-23T03:46:57.953820+0000","last_clean":"2024-03-23T03:46:57.953820+0000","last_became_active":"2024-03-23T03:46:53.944614+0000","last_became_peered":"2024-03-23T03:46:53.944614+0000","last_unstale":"2024-03-23T03:46:57.953820+0000","last_undegraded":"2024-03-23T03:46:57.953820+0000","last_fullsized":"2024-03-23T03:46:57.953820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:01:24.719476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.013281+0000","last_change":"2024-03-23T03:46:53.957372+0000","last_active":"2024-03-23T03:46:58.013281+0000","last_peered":"2024-03-23T03:46:58.013281+0000","last_clean":"2024-03-23T03:46:58.013281+0000","last_became_active":"2024-03-23T03:46:53.956119+0000","last_became_peered":"2024-03-23T03:46:53.956119+0000","last_unstale":"2024-03-23T03:46:58.013281+0000","last_undegraded":"2024-03-23T03:46:58.013281+0000","last_fullsized":"2024-03-23T03:46:58.013281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:16:39.131183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.968302+0000","last_change":"2024-03-23T03:46:53.954765+0000","last_active":"2024-03-23T03:46:57.968302+0000","last_peered":"2024-03-23T03:46:57.968302+0000","last_clean":"2024-03-23T03:46:57.968302+0000","last_became_active":"2024-03-23T03:46:53.954424+0000","last_became_peered":"2024-03-23T03:46:53.954424+0000","last_unstale":"2024-03-23T03:46:57.968302+0000","last_undegraded":"2024-03-23T03:46:57.968302+0000","last_fullsized":"2024-03-23T03:46:57.968302+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:56:29.065234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.016764+0000","last_change":"2024-03-23T03:46:53.949427+0000","last_active":"2024-03-23T03:46:58.016764+0000","last_peered":"2024-03-23T03:46:58.016764+0000","last_clean":"2024-03-23T03:46:58.016764+0000","last_became_active":"2024-03-23T03:46:53.949114+0000","last_became_peered":"2024-03-23T03:46:53.949114+0000","last_unstale":"2024-03-23T03:46:58.016764+0000","last_undegraded":"2024-03-23T03:46:58.016764+0000","last_fullsized":"2024-03-23T03:46:58.016764+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:50:10.900452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.970488+0000","last_change":"2024-03-23T03:46:53.956260+0000","last_active":"2024-03-23T03:46:57.970488+0000","last_peered":"2024-03-23T03:46:57.970488+0000","last_clean":"2024-03-23T03:46:57.970488+0000","last_became_active":"2024-03-23T03:46:53.955509+0000","last_became_peered":"2024-03-23T03:46:53.955509+0000","last_unstale":"2024-03-23T03:46:57.970488+0000","last_undegraded":"2024-03-23T03:46:57.970488+0000","last_fullsized":"2024-03-23T03:46:57.970488+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:34:41.509208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":14,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.951671+0000","last_change":"2024-03-23T03:46:53.957607+0000","last_active":"2024-03-23T03:46:57.951671+0000","last_peered":"2024-03-23T03:46:57.951671+0000","last_clean":"2024-03-23T03:46:57.951671+0000","last_became_active":"2024-03-23T03:46:53.956236+0000","last_became_peered":"2024-03-23T03:46:53.956236+0000","last_unstale":"2024-03-23T03:46:57.951671+0000","last_undegraded":"2024-03-23T03:46:57.951671+0000","last_fullsized":"2024-03-23T03:46:57.951671+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:27:03.917141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:57.943481+0000","last_change":"2024-03-23T03:46:53.960428+0000","last_active":"2024-03-23T03:46:57.943481+0000","last_peered":"2024-03-23T03:46:57.943481+0000","last_clean":"2024-03-23T03:46:57.943481+0000","last_became_active":"2024-03-23T03:46:53.960148+0000","last_became_peered":"2024-03-23T03:46:53.960148+0000","last_unstale":"2024-03-23T03:46:57.943481+0000","last_undegraded":"2024-03-23T03:46:57.943481+0000","last_fullsized":"2024-03-23T03:46:57.943481+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:01:24.719476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-03-23T03:46:58.017880+0000","last_change":"2024-03-23T03:46:53.966166+0000","last_active":"2024-03-23T03:46:58.017880+0000","last_peered":"2024-03-23T03:46:58.017880+0000","last_clean":"2024-03-23T03:46:58.017880+0000","last_became_active":"2024-03-23T03:46:53.965871+0000","last_became_peered":"2024-03-23T03:46:53.965871+0000","last_unstale":"2024-03-23T03:46:58.017880+0000","last_undegraded":"2024-03-23T03:46:58.017880+0000","last_fullsized":"2024-03-23T03:46:58.017880+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:53.413613+0000","last_clean_scrub_stamp":"2024-03-23T03:46:53.413613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:01:04.097184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4593,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542154,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27880,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343960,"statfs":{"total":96636764160,"available":96608215040,"internally_reserved":0,"allocated":872448,"data_stored":698548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4730000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4079999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.524}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2749999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542152,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":111393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.179}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.258}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2370000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.109}]}]},{"osd":5,"up_from":14,"seq":60129542152,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":278528,"data_stored":107688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.458}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6419999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.54}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8500000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.877}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.964}]}]},{"osd":4,"up_from":14,"seq":60129542153,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27300,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344540,"statfs":{"total":96636764160,"available":96608808960,"internally_reserved":0,"allocated":286720,"data_stored":107724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.383}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.133}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.343}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":286720,"data_stored":111393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.282}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.7859999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3180000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542153,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":107598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.014}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5309999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6890000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.498}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7050000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5800000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27304,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344536,"statfs":{"total":96636764160,"available":96608804864,"internally_reserved":0,"allocated":282624,"data_stored":107758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.488}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.595}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6359999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27888,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343952,"statfs":{"total":96636764160,"available":96608206848,"internally_reserved":0,"allocated":880640,"data_stored":698604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.163}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.367}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1519999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2030000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:47:22.344 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-23T03:47:22.344 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-23T03:47:22.344 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-23T03:47:22.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-23T03:47:22.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.585+0000 7fbe1d200640 1 Processor -- start 2024-03-23T03:47:22.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.585+0000 7fbe1d200640 1 -- start start 2024-03-23T03:47:22.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.587+0000 7fbe1d200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbe1815b560 0x7fbe1815b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:22.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.587+0000 7fbe1d200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fbe180b39b0 0x7fbe180b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:22.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.587+0000 7fbe1d200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1806c6f0 con 0x7fbe1815e500 2024-03-23T03:47:22.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.587+0000 7fbe1d200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe180b4370 con 0x7fbe180b39b0 2024-03-23T03:47:22.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.587+0000 7fbe1d200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1806c9d0 con 0x7fbe1815b560 2024-03-23T03:47:22.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.587+0000 7fbe16800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbe1815e500 0x7fbe1815e990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41282/0 (socket says 172.21.15.102:41282) 2024-03-23T03:47:22.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.587+0000 7fbe16800640 1 -- 172.21.15.102:0/3953228172 learned_addr learned my addr 172.21.15.102:0/3953228172 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:22.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe15e00640 1 --2- 172.21.15.102:0/3953228172 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbe1815b560 0x7fbe1815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:22.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe0d400640 1 --2- 172.21.15.102:0/3953228172 >> v2:172.21.15.173:3300/0 conn(0x7fbe180b39b0 0x7fbe180b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:22.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe15400640 1 -- 172.21.15.102:0/3953228172 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936595024 0 0) 0x7fbe1806c6f0 con 0x7fbe1815e500 2024-03-23T03:47:22.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe15400640 1 -- 172.21.15.102:0/3953228172 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdfc003520 con 0x7fbe1815e500 2024-03-23T03:47:22.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe15400640 1 -- 172.21.15.102:0/3953228172 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 90370370 0 0) 0x7fbdfc003520 con 0x7fbe1815e500 2024-03-23T03:47:22.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe15400640 1 -- 172.21.15.102:0/3953228172 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbe1815b560 msgr2=0x7fbe1815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:22.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe15400640 1 --2- 172.21.15.102:0/3953228172 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbe1815b560 0x7fbe1815b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe15400640 1 -- 172.21.15.102:0/3953228172 >> v2:172.21.15.173:3300/0 conn(0x7fbe180b39b0 msgr2=0x7fbe180b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:22.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe15400640 1 --2- 172.21.15.102:0/3953228172 >> v2:172.21.15.173:3300/0 conn(0x7fbe180b39b0 0x7fbe180b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe15400640 1 -- 172.21.15.102:0/3953228172 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1806ccb0 con 0x7fbe1815e500 2024-03-23T03:47:22.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe15400640 1 -- 172.21.15.102:0/3953228172 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fbe100019f0 con 0x7fbe1815e500 2024-03-23T03:47:22.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe15400640 1 -- 172.21.15.102:0/3953228172 >> v1:172.21.15.102:6789/0 conn(0x7fbe1815e500 legacy=0x7fbe1815e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:22.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe15400640 1 --2- 172.21.15.102:0/3953228172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdfc003970 0x7fbdfc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:22.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe15400640 1 --2- 172.21.15.102:0/3953228172 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbe1815b560 0x7fbdfc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:22.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe15400640 1 --2- 172.21.15.102:0/3953228172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdfc004d00 0x7fbdfc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:22.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe15400640 1 -- 172.21.15.102:0/3953228172 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe180b4370 con 0x7fbdfc003970 2024-03-23T03:47:22.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe15400640 1 -- 172.21.15.102:0/3953228172 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1806c9d0 con 0x7fbdfc004d00 2024-03-23T03:47:22.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.588+0000 7fbe15400640 1 -- 172.21.15.102:0/3953228172 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdfc00b980 con 0x7fbe1815b560 2024-03-23T03:47:22.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.589+0000 7fbe15e00640 1 --2- 172.21.15.102:0/3953228172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdfc003970 0x7fbdfc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:22.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.589+0000 7fbe0d400640 1 --2- 172.21.15.102:0/3953228172 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbe1815b560 0x7fbdfc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:22.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.589+0000 7fbe16800640 1 --2- 172.21.15.102:0/3953228172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdfc004d00 0x7fbdfc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:22.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.589+0000 7fbe15e00640 1 -- 172.21.15.102:0/3953228172 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbe1815b560 msgr2=0x7fbdfc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:22.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.589+0000 7fbe15e00640 1 --2- 172.21.15.102:0/3953228172 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbe1815b560 0x7fbdfc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.589+0000 7fbe15e00640 1 -- 172.21.15.102:0/3953228172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdfc004d00 msgr2=0x7fbdfc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:22.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.589+0000 7fbe15e00640 1 --2- 172.21.15.102:0/3953228172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdfc004d00 0x7fbdfc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.589+0000 7fbe15e00640 1 -- 172.21.15.102:0/3953228172 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdfc003520 con 0x7fbdfc003970 2024-03-23T03:47:22.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.590+0000 7fbe15e00640 1 --2- 172.21.15.102:0/3953228172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdfc003970 0x7fbdfc003d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbe08009ac0 tx=0x7fbdfc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8b6f92f266a72f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:22.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.590+0000 7fbe15400640 1 -- 172.21.15.102:0/3953228172 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0802c6b0 con 0x7fbdfc003970 2024-03-23T03:47:22.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.590+0000 7fbe15400640 1 -- 172.21.15.102:0/3953228172 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe0802c810 con 0x7fbdfc003970 2024-03-23T03:47:22.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.591+0000 7fbe15400640 1 -- 172.21.15.102:0/3953228172 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0802cba0 con 0x7fbdfc003970 2024-03-23T03:47:22.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.591+0000 7fbe1d200640 1 -- 172.21.15.102:0/3953228172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdfc003970 msgr2=0x7fbdfc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:22.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.591+0000 7fbe1d200640 1 --2- 172.21.15.102:0/3953228172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdfc003970 0x7fbdfc003d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbe08009ac0 tx=0x7fbdfc0057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.591+0000 7fbe16800640 1 -- 172.21.15.102:0/3953228172 reap_dead start 2024-03-23T03:47:22.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.591+0000 7fbe1d200640 1 -- 172.21.15.102:0/3953228172 shutdown_connections 2024-03-23T03:47:22.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.591+0000 7fbe1d200640 1 -- 172.21.15.102:0/3953228172 >> 172.21.15.102:0/3953228172 conn(0x7fbe1806a3d0 msgr2=0x7fbe180b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:22.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.591+0000 7fbe1d200640 1 -- 172.21.15.102:0/3953228172 shutdown_connections 2024-03-23T03:47:22.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.591+0000 7fbe1d200640 1 -- 172.21.15.102:0/3953228172 wait complete. 2024-03-23T03:47:22.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.593+0000 7fbe1d200640 1 Processor -- start 2024-03-23T03:47:22.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.593+0000 7fbe1d200640 1 -- start start 2024-03-23T03:47:22.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.594+0000 7fbe1d200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdfc003970 0x7fbe181f50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:22.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.594+0000 7fbe1d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe181f3450 0x7fbe181f22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:22.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.594+0000 7fbe1d200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbe181f2810 0x7fbe18159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:22.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.594+0000 7fbe1d200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe180b2280 con 0x7fbdfc003970 2024-03-23T03:47:22.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.594+0000 7fbe1d200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe181f5630 con 0x7fbe181f3450 2024-03-23T03:47:22.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.594+0000 7fbe1d200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe180b1c40 con 0x7fbe181f2810 2024-03-23T03:47:22.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.594+0000 7fbe15e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdfc003970 0x7fbe181f50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:22.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.594+0000 7fbe15e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdfc003970 0x7fbe181f50f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:37532/0 (socket says 172.21.15.102:37532) 2024-03-23T03:47:22.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.594+0000 7fbe15e00640 1 -- 172.21.15.102:0/1175271350 learned_addr learned my addr 172.21.15.102:0/1175271350 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:22.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.594+0000 7fbe0d400640 1 --2- 172.21.15.102:0/1175271350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe181f3450 0x7fbe181f22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:22.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.595+0000 7fbe15e00640 1 -- 172.21.15.102:0/1175271350 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbe181f2810 msgr2=0x7fbe18159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:22.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.595+0000 7fbe16800640 1 --2- 172.21.15.102:0/1175271350 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbe181f2810 0x7fbe18159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:22.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.595+0000 7fbe15e00640 1 --2- 172.21.15.102:0/1175271350 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbe181f2810 0x7fbe18159930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.595+0000 7fbe15e00640 1 -- 172.21.15.102:0/1175271350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe181f3450 msgr2=0x7fbe181f22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:22.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.595+0000 7fbe15e00640 1 --2- 172.21.15.102:0/1175271350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe181f3450 0x7fbe181f22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.595+0000 7fbe15e00640 1 -- 172.21.15.102:0/1175271350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe181f2e00 con 0x7fbdfc003970 2024-03-23T03:47:22.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.595+0000 7fbe0d400640 1 --2- 172.21.15.102:0/1175271350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe181f3450 0x7fbe181f22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:22.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.595+0000 7fbe15e00640 1 --2- 172.21.15.102:0/1175271350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdfc003970 0x7fbe181f50f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbe08037040 tx=0x7fbe08004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:22.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.595+0000 7fbe0ea00640 1 -- 172.21.15.102:0/1175271350 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe08003810 con 0x7fbdfc003970 2024-03-23T03:47:22.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.595+0000 7fbe0ea00640 1 -- 172.21.15.102:0/1175271350 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe08003970 con 0x7fbdfc003970 2024-03-23T03:47:22.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.595+0000 7fbe0ea00640 1 -- 172.21.15.102:0/1175271350 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe08003ce0 con 0x7fbdfc003970 2024-03-23T03:47:22.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.595+0000 7fbe1d200640 1 -- 172.21.15.102:0/1175271350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe181f30e0 con 0x7fbdfc003970 2024-03-23T03:47:22.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.595+0000 7fbe1d200640 1 -- 172.21.15.102:0/1175271350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe1813a670 con 0x7fbdfc003970 2024-03-23T03:47:22.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.596+0000 7fbe1d200640 1 -- 172.21.15.102:0/1175271350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe1806ac20 con 0x7fbdfc003970 2024-03-23T03:47:22.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.597+0000 7fbe0ea00640 1 -- 172.21.15.102:0/1175271350 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbe08034440 con 0x7fbdfc003970 2024-03-23T03:47:22.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.598+0000 7fbe0ea00640 1 --2- 172.21.15.102:0/1175271350 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbdf406ac00 0x7fbdf406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:22.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.598+0000 7fbe0ea00640 1 -- 172.21.15.102:0/1175271350 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbe080ab8d0 con 0x7fbdfc003970 2024-03-23T03:47:22.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.598+0000 7fbe0d400640 1 --2- 172.21.15.102:0/1175271350 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbdf406ac00 0x7fbdf406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:22.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.599+0000 7fbe0d400640 1 --2- 172.21.15.102:0/1175271350 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbdf406ac00 0x7fbdf406d0c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbe181f3870 tx=0x7fbe00008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:22.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.602+0000 7fbe0ea00640 1 -- 172.21.15.102:0/1175271350 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbe08076980 con 0x7fbdfc003970 2024-03-23T03:47:22.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.839+0000 7fbe1d200640 1 -- 172.21.15.102:0/1175271350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbe18073390 con 0x7fbdfc003970 2024-03-23T03:47:22.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.840+0000 7fbe0ea00640 1 -- 172.21.15.102:0/1175271350 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fbe080044e0 con 0x7fbdfc003970 2024-03-23T03:47:22.841 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:22.841 INFO:teuthology.orchestra.run.smithi102.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-23T03:47:22.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.843+0000 7fbe1d200640 1 -- 172.21.15.102:0/1175271350 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbdf406ac00 msgr2=0x7fbdf406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:22.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.843+0000 7fbe1d200640 1 --2- 172.21.15.102:0/1175271350 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbdf406ac00 0x7fbdf406d0c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbe181f3870 tx=0x7fbe00008040 comp rx=0 tx=0).stop 2024-03-23T03:47:22.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.843+0000 7fbe1d200640 1 -- 172.21.15.102:0/1175271350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdfc003970 msgr2=0x7fbe181f50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:22.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.843+0000 7fbe1d200640 1 --2- 172.21.15.102:0/1175271350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdfc003970 0x7fbe181f50f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbe08037040 tx=0x7fbe08004730 comp rx=0 tx=0).stop 2024-03-23T03:47:22.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.843+0000 7fbe1d200640 1 -- 172.21.15.102:0/1175271350 shutdown_connections 2024-03-23T03:47:22.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.843+0000 7fbe1d200640 1 --2- 172.21.15.102:0/1175271350 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbdf406ac00 0x7fbdf406d0c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.843+0000 7fbe1d200640 1 --2- 172.21.15.102:0/1175271350 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbe181f2810 0x7fbe18159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.843+0000 7fbe1d200640 1 --2- 172.21.15.102:0/1175271350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe181f3450 0x7fbe181f22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.843+0000 7fbe1d200640 1 --2- 172.21.15.102:0/1175271350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdfc003970 0x7fbe181f50f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:22.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.843+0000 7fbe1d200640 1 -- 172.21.15.102:0/1175271350 >> 172.21.15.102:0/1175271350 conn(0x7fbe1806a3d0 msgr2=0x7fbe1815c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:22.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.844+0000 7fbe1d200640 1 -- 172.21.15.102:0/1175271350 shutdown_connections 2024-03-23T03:47:22.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:22.844+0000 7fbe1d200640 1 -- 172.21.15.102:0/1175271350 wait complete. 2024-03-23T03:47:22.863 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-23T03:47:22.863 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:23.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.067+0000 7f4f5be00640 1 Processor -- start 2024-03-23T03:47:23.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.067+0000 7f4f5be00640 1 -- start start 2024-03-23T03:47:23.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.068+0000 7f4f5be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4f5c15cab0 0x7f4f5c0b8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.068+0000 7f4f5be00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5c15ce90 con 0x7f4f5c0d1c30 2024-03-23T03:47:23.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.069+0000 7f4f5be00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5c0cddf0 con 0x7f4f5c0ceeb0 2024-03-23T03:47:23.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.069+0000 7f4f5be00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f5c0b4f40 con 0x7f4f5c15cab0 2024-03-23T03:47:23.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.069+0000 7f4f5aa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4f5c15cab0 0x7f4f5c0b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.069+0000 7f4f5b400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4f5c0d1c30 0x7f4f5c0b4830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52786/0 (socket says 172.21.15.102:52786) 2024-03-23T03:47:23.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.069+0000 7f4f5b400640 1 -- 172.21.15.102:0/365205376 learned_addr learned my addr 172.21.15.102:0/365205376 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:23.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.069+0000 7f4f5aa00640 1 -- 172.21.15.102:0/365205376 >> v1:172.21.15.173:6789/0 conn(0x7f4f5c0ceeb0 legacy=0x7f4f5c0b4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:23.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.069+0000 7f4f5aa00640 1 -- 172.21.15.102:0/365205376 >> v1:172.21.15.102:6789/0 conn(0x7f4f5c0d1c30 legacy=0x7f4f5c0b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:23.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.069+0000 7f4f5aa00640 1 -- 172.21.15.102:0/365205376 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5c0cd570 con 0x7f4f5c15cab0 2024-03-23T03:47:23.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.069+0000 7f4f59600640 1 -- 172.21.15.102:0/365205376 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108032244 0 0) 0x7f4f5c15ce90 con 0x7f4f5c0d1c30 2024-03-23T03:47:23.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.070+0000 7f4f5aa00640 1 --2- 172.21.15.102:0/365205376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f5c15cab0 0x7f4f5c0b8050 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c0097a0 tx=0x7f4f4c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c8ed262b7265a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:23.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.071+0000 7f4f59600640 1 -- 172.21.15.102:0/365205376 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c02e070 con 0x7f4f5c15cab0 2024-03-23T03:47:23.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.071+0000 7f4f59600640 1 -- 172.21.15.102:0/365205376 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f4c0315a0 con 0x7f4f5c15cab0 2024-03-23T03:47:23.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.071+0000 7f4f59600640 1 -- 172.21.15.102:0/365205376 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4c031930 con 0x7f4f5c15cab0 2024-03-23T03:47:23.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.071+0000 7f4f5be00640 1 -- 172.21.15.102:0/365205376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f5c15cab0 msgr2=0x7f4f5c0b8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:23.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.071+0000 7f4f5be00640 1 --2- 172.21.15.102:0/365205376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f5c15cab0 0x7f4f5c0b8050 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c0097a0 tx=0x7f4f4c02b840 comp rx=0 tx=0).stop 2024-03-23T03:47:23.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.071+0000 7f4f5be00640 1 -- 172.21.15.102:0/365205376 shutdown_connections 2024-03-23T03:47:23.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.071+0000 7f4f5be00640 1 --2- 172.21.15.102:0/365205376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f5c15cab0 0x7f4f5c0b8050 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.071+0000 7f4f5be00640 1 -- 172.21.15.102:0/365205376 >> 172.21.15.102:0/365205376 conn(0x7f4f5c06ae90 msgr2=0x7f4f5c06b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:23.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.071+0000 7f4f5be00640 1 -- 172.21.15.102:0/365205376 shutdown_connections 2024-03-23T03:47:23.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.072+0000 7f4f5be00640 1 -- 172.21.15.102:0/365205376 wait complete. 2024-03-23T03:47:23.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.073+0000 7f4f5be00640 1 Processor -- start 2024-03-23T03:47:23.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.073+0000 7f4f5be00640 1 -- start start 2024-03-23T03:47:23.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.074+0000 7f4f5be00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f5c0ceeb0 0x7f4f5c15e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.074+0000 7f4f5be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f5c0d1c30 0x7f4f5c15f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.074+0000 7f4f5be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f5c15cab0 0x7f4f5c156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.074+0000 7f4f5be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c15fed0 con 0x7f4f5c15cab0 2024-03-23T03:47:23.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.074+0000 7f4f5be00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c157030 con 0x7f4f5c0d1c30 2024-03-23T03:47:23.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.074+0000 7f4f5be00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c15eff0 con 0x7f4f5c0ceeb0 2024-03-23T03:47:23.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.074+0000 7f4f5b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f5c15cab0 0x7f4f5c156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.074+0000 7f4f5b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f5c15cab0 0x7f4f5c156af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33140/0 (socket says 172.21.15.102:33140) 2024-03-23T03:47:23.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.074+0000 7f4f5b400640 1 -- 172.21.15.102:0/3145392235 learned_addr learned my addr 172.21.15.102:0/3145392235 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:23.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.075+0000 7f4f5aa00640 1 --2- 172.21.15.102:0/3145392235 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f5c0ceeb0 0x7f4f5c15e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.075+0000 7f4f5a000640 1 --2- 172.21.15.102:0/3145392235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f5c0d1c30 0x7f4f5c15f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.075+0000 7f4f5b400640 1 -- 172.21.15.102:0/3145392235 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f5c0ceeb0 msgr2=0x7f4f5c15e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:23.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.075+0000 7f4f5b400640 1 --2- 172.21.15.102:0/3145392235 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f5c0ceeb0 0x7f4f5c15e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.075+0000 7f4f5b400640 1 -- 172.21.15.102:0/3145392235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f5c0d1c30 msgr2=0x7f4f5c15f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:23.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.075+0000 7f4f5b400640 1 --2- 172.21.15.102:0/3145392235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f5c0d1c30 0x7f4f5c15f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.075+0000 7f4f5b400640 1 -- 172.21.15.102:0/3145392235 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c009400 con 0x7f4f5c15cab0 2024-03-23T03:47:23.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.075+0000 7f4f5b400640 1 --2- 172.21.15.102:0/3145392235 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f5c15cab0 0x7f4f5c156af0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4f5400c7e0 tx=0x7f4f5400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:23.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.075+0000 7f4f52a00640 1 -- 172.21.15.102:0/3145392235 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5400a640 con 0x7f4f5c15cab0 2024-03-23T03:47:23.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.075+0000 7f4f52a00640 1 -- 172.21.15.102:0/3145392235 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5400a7c0 con 0x7f4f5c15cab0 2024-03-23T03:47:23.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.075+0000 7f4f52a00640 1 -- 172.21.15.102:0/3145392235 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5400ab70 con 0x7f4f5c15cab0 2024-03-23T03:47:23.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.075+0000 7f4f5be00640 1 -- 172.21.15.102:0/3145392235 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f5c0c6ec0 con 0x7f4f5c15cab0 2024-03-23T03:47:23.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.076+0000 7f4f5be00640 1 -- 172.21.15.102:0/3145392235 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5c0c7480 con 0x7f4f5c15cab0 2024-03-23T03:47:23.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.076+0000 7f4f5be00640 1 -- 172.21.15.102:0/3145392235 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f5c0b8440 con 0x7f4f5c15cab0 2024-03-23T03:47:23.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.077+0000 7f4f52a00640 1 -- 172.21.15.102:0/3145392235 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4f5400acd0 con 0x7f4f5c15cab0 2024-03-23T03:47:23.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.078+0000 7f4f52a00640 1 --2- 172.21.15.102:0/3145392235 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f3c06ac00 0x7f4f3c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.078+0000 7f4f52a00640 1 -- 172.21.15.102:0/3145392235 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4f54088cb0 con 0x7f4f5c15cab0 2024-03-23T03:47:23.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.079+0000 7f4f5aa00640 1 --2- 172.21.15.102:0/3145392235 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f3c06ac00 0x7f4f3c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.079+0000 7f4f5aa00640 1 --2- 172.21.15.102:0/3145392235 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f3c06ac00 0x7f4f3c06d0c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c036000 tx=0x7f4f4c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:23.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.083+0000 7f4f52a00640 1 -- 172.21.15.102:0/3145392235 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4f54053b00 con 0x7f4f5c15cab0 2024-03-23T03:47:23.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.275+0000 7f4f5be00640 1 -- 172.21.15.102:0/3145392235 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f5c0735a0 con 0x7f4f5c15cab0 2024-03-23T03:47:23.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.278+0000 7f4f52a00640 1 -- 172.21.15.102:0/3145392235 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f4f54018070 con 0x7f4f5c15cab0 2024-03-23T03:47:23.279 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:23.280 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:58.410551+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:47:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.281+0000 7f4f5be00640 1 -- 172.21.15.102:0/3145392235 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f3c06ac00 msgr2=0x7f4f3c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.281+0000 7f4f5be00640 1 --2- 172.21.15.102:0/3145392235 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f3c06ac00 0x7f4f3c06d0c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4f4c036000 tx=0x7f4f4c03d040 comp rx=0 tx=0).stop 2024-03-23T03:47:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.281+0000 7f4f5be00640 1 -- 172.21.15.102:0/3145392235 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f5c15cab0 msgr2=0x7f4f5c156af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.281+0000 7f4f5be00640 1 --2- 172.21.15.102:0/3145392235 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f5c15cab0 0x7f4f5c156af0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4f5400c7e0 tx=0x7f4f5400cca0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.282+0000 7f4f5be00640 1 -- 172.21.15.102:0/3145392235 shutdown_connections 2024-03-23T03:47:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.282+0000 7f4f5be00640 1 --2- 172.21.15.102:0/3145392235 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f3c06ac00 0x7f4f3c06d0c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.282+0000 7f4f5be00640 1 --2- 172.21.15.102:0/3145392235 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f5c15cab0 0x7f4f5c156af0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.282+0000 7f4f5be00640 1 --2- 172.21.15.102:0/3145392235 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f5c0d1c30 0x7f4f5c15f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.282+0000 7f4f5be00640 1 --2- 172.21.15.102:0/3145392235 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f5c0ceeb0 0x7f4f5c15e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.282+0000 7f4f5be00640 1 -- 172.21.15.102:0/3145392235 >> 172.21.15.102:0/3145392235 conn(0x7f4f5c06ae90 msgr2=0x7f4f5c160710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.282+0000 7f4f5be00640 1 -- 172.21.15.102:0/3145392235 shutdown_connections 2024-03-23T03:47:23.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.282+0000 7f4f5be00640 1 -- 172.21.15.102:0/3145392235 wait complete. 2024-03-23T03:47:23.302 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:46:52.158650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:46:52.914275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-23T03:46:55.983754+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-23T03:47:23.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:47:23.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.510+0000 7f722c200640 1 Processor -- start 2024-03-23T03:47:23.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.510+0000 7f722c200640 1 -- start start 2024-03-23T03:47:23.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.511+0000 7f722c200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7224159830 0x7f72240cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.511+0000 7f722c200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72240ca7b0 0x7f72240cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.511+0000 7f722c200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7224159c10 con 0x7f72240ce930 2024-03-23T03:47:23.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.511+0000 7f722c200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f722406de50 con 0x7f7224159830 2024-03-23T03:47:23.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.511+0000 7f722c200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72240cd840 con 0x7f72240ca7b0 2024-03-23T03:47:23.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.512+0000 7f7229800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7224159830 0x7f72240cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.512+0000 7f7229800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7224159830 0x7f72240cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33148/0 (socket says 172.21.15.102:33148) 2024-03-23T03:47:23.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.512+0000 7f7229800640 1 -- 172.21.15.102:0/840787061 learned_addr learned my addr 172.21.15.102:0/840787061 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:23.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.512+0000 7f7228e00640 1 --2- 172.21.15.102:0/840787061 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72240ca7b0 0x7f72240cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.512+0000 7f7229800640 1 -- 172.21.15.102:0/840787061 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72240ca7b0 msgr2=0x7f72240cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:23.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.512+0000 7f7229800640 1 --2- 172.21.15.102:0/840787061 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72240ca7b0 0x7f72240cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.512+0000 7f7229800640 1 -- 172.21.15.102:0/840787061 >> v1:172.21.15.173:6789/0 conn(0x7f72240ce930 legacy=0x7f72240b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:23.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.512+0000 7f7229800640 1 -- 172.21.15.102:0/840787061 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72240ccc30 con 0x7f7224159830 2024-03-23T03:47:23.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.512+0000 7f7228e00640 1 --2- 172.21.15.102:0/840787061 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72240ca7b0 0x7f72240cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:23.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.513+0000 7f7229800640 1 --2- 172.21.15.102:0/840787061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7224159830 0x7f72240cc3c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7214009870 tx=0x7f721402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d0266a6fd76bb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:23.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.514+0000 7f7223e00640 1 -- 172.21.15.102:0/840787061 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7214030840 con 0x7f7224159830 2024-03-23T03:47:23.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.514+0000 7f7223e00640 1 -- 172.21.15.102:0/840787061 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72140309a0 con 0x7f7224159830 2024-03-23T03:47:23.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.514+0000 7f7223e00640 1 -- 172.21.15.102:0/840787061 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7214030d30 con 0x7f7224159830 2024-03-23T03:47:23.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.514+0000 7f722c200640 1 -- 172.21.15.102:0/840787061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7224159830 msgr2=0x7f72240cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:23.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.514+0000 7f722c200640 1 --2- 172.21.15.102:0/840787061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7224159830 0x7f72240cc3c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7214009870 tx=0x7f721402b980 comp rx=0 tx=0).stop 2024-03-23T03:47:23.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.514+0000 7f722c200640 1 -- 172.21.15.102:0/840787061 shutdown_connections 2024-03-23T03:47:23.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.514+0000 7f722c200640 1 --2- 172.21.15.102:0/840787061 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72240ca7b0 0x7f72240cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.514+0000 7f722c200640 1 --2- 172.21.15.102:0/840787061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7224159830 0x7f72240cc3c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.514+0000 7f722c200640 1 -- 172.21.15.102:0/840787061 >> 172.21.15.102:0/840787061 conn(0x7f722406a0b0 msgr2=0x7f722406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:23.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.515+0000 7f722c200640 1 -- 172.21.15.102:0/840787061 shutdown_connections 2024-03-23T03:47:23.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.515+0000 7f722c200640 1 -- 172.21.15.102:0/840787061 wait complete. 2024-03-23T03:47:23.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.516+0000 7f722c200640 1 Processor -- start 2024-03-23T03:47:23.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.516+0000 7f722c200640 1 -- start start 2024-03-23T03:47:23.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.517+0000 7f722c200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72240ca7b0 0x7f72241f2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.517+0000 7f722c200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72240ce930 0x7f72240acbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.517+0000 7f722c200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7224159830 0x7f72240ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.517+0000 7f722c200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72240ab8b0 con 0x7f72240ca7b0 2024-03-23T03:47:23.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.517+0000 7f722c200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72241f3450 con 0x7f7224159830 2024-03-23T03:47:23.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.517+0000 7f722c200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72240abb50 con 0x7f72240ce930 2024-03-23T03:47:23.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.517+0000 7f7229800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72240ca7b0 0x7f72241f2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.518+0000 7f7229800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72240ca7b0 0x7f72241f2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33156/0 (socket says 172.21.15.102:33156) 2024-03-23T03:47:23.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.518+0000 7f7229800640 1 -- 172.21.15.102:0/2172171862 learned_addr learned my addr 172.21.15.102:0/2172171862 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:23.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.518+0000 7f7228e00640 1 --2- 172.21.15.102:0/2172171862 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72240ce930 0x7f72240acbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.518+0000 7f722a200640 1 --2- 172.21.15.102:0/2172171862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7224159830 0x7f72240ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.518+0000 7f7229800640 1 -- 172.21.15.102:0/2172171862 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72240ce930 msgr2=0x7f72240acbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:23.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.518+0000 7f7229800640 1 --2- 172.21.15.102:0/2172171862 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72240ce930 0x7f72240acbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.518+0000 7f7229800640 1 -- 172.21.15.102:0/2172171862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7224159830 msgr2=0x7f72240ae200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:23.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.518+0000 7f7229800640 1 --2- 172.21.15.102:0/2172171862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7224159830 0x7f72240ae200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.518+0000 7f7229800640 1 -- 172.21.15.102:0/2172171862 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72140094d0 con 0x7f72240ca7b0 2024-03-23T03:47:23.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.518+0000 7f7229800640 1 --2- 172.21.15.102:0/2172171862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72240ca7b0 0x7f72241f2f10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7214009870 tx=0x7f7214004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:23.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.519+0000 7f7221600640 1 -- 172.21.15.102:0/2172171862 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72140307f0 con 0x7f72240ca7b0 2024-03-23T03:47:23.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.519+0000 7f7221600640 1 -- 172.21.15.102:0/2172171862 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7214043400 con 0x7f72240ca7b0 2024-03-23T03:47:23.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.519+0000 7f7221600640 1 -- 172.21.15.102:0/2172171862 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72140437b0 con 0x7f72240ca7b0 2024-03-23T03:47:23.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.519+0000 7f722c200640 1 -- 172.21.15.102:0/2172171862 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72240abe30 con 0x7f72240ca7b0 2024-03-23T03:47:23.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.519+0000 7f722c200640 1 -- 172.21.15.102:0/2172171862 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72240ac3f0 con 0x7f72240ca7b0 2024-03-23T03:47:23.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.520+0000 7f722c200640 1 -- 172.21.15.102:0/2172171862 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7224073390 con 0x7f72240ca7b0 2024-03-23T03:47:23.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.521+0000 7f7221600640 1 -- 172.21.15.102:0/2172171862 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7214043910 con 0x7f72240ca7b0 2024-03-23T03:47:23.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.522+0000 7f7221600640 1 --2- 172.21.15.102:0/2172171862 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f720406ac00 0x7f720406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.522+0000 7f7221600640 1 -- 172.21.15.102:0/2172171862 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f721403e070 con 0x7f72240ca7b0 2024-03-23T03:47:23.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.523+0000 7f7228e00640 1 --2- 172.21.15.102:0/2172171862 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f720406ac00 0x7f720406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.523+0000 7f7228e00640 1 --2- 172.21.15.102:0/2172171862 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f720406ac00 0x7f720406d0c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f72241f6b80 tx=0x7f7210008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:23.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.527+0000 7f7221600640 1 -- 172.21.15.102:0/2172171862 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f72140777b0 con 0x7f72240ca7b0 2024-03-23T03:47:23.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.719+0000 7f722c200640 1 -- 172.21.15.102:0/2172171862 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f72241557d0 con 0x7f72240ca7b0 2024-03-23T03:47:23.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.720+0000 7f7221600640 1 -- 172.21.15.102:0/2172171862 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f721407add0 con 0x7f72240ca7b0 2024-03-23T03:47:23.720 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:47:23.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.723+0000 7f722c200640 1 -- 172.21.15.102:0/2172171862 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f720406ac00 msgr2=0x7f720406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:23.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.723+0000 7f722c200640 1 --2- 172.21.15.102:0/2172171862 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f720406ac00 0x7f720406d0c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f72241f6b80 tx=0x7f7210008040 comp rx=0 tx=0).stop 2024-03-23T03:47:23.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.723+0000 7f722c200640 1 -- 172.21.15.102:0/2172171862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72240ca7b0 msgr2=0x7f72241f2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:23.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.723+0000 7f722c200640 1 --2- 172.21.15.102:0/2172171862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72240ca7b0 0x7f72241f2f10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7214009870 tx=0x7f7214004730 comp rx=0 tx=0).stop 2024-03-23T03:47:23.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.723+0000 7f722c200640 1 -- 172.21.15.102:0/2172171862 shutdown_connections 2024-03-23T03:47:23.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.723+0000 7f722c200640 1 --2- 172.21.15.102:0/2172171862 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f720406ac00 0x7f720406d0c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.723+0000 7f722c200640 1 --2- 172.21.15.102:0/2172171862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7224159830 0x7f72240ae200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.723+0000 7f722c200640 1 --2- 172.21.15.102:0/2172171862 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72240ce930 0x7f72240acbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.723+0000 7f722c200640 1 --2- 172.21.15.102:0/2172171862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72240ca7b0 0x7f72241f2f10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.723+0000 7f722c200640 1 -- 172.21.15.102:0/2172171862 >> 172.21.15.102:0/2172171862 conn(0x7f722406a0b0 msgr2=0x7f72241579e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:23.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.723+0000 7f722c200640 1 -- 172.21.15.102:0/2172171862 shutdown_connections 2024-03-23T03:47:23.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.723+0000 7f722c200640 1 -- 172.21.15.102:0/2172171862 wait complete. 2024-03-23T03:47:23.743 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:47:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.953+0000 7f1aef000640 1 Processor -- start 2024-03-23T03:47:23.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.953+0000 7f1aef000640 1 -- start start 2024-03-23T03:47:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.955+0000 7f1aef000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ae80b84b0 0x7f1ae80d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.955+0000 7f1aef000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1ae80b5aa0 0x7f1ae80b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.955+0000 7f1aef000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae80d1280 con 0x7f1ae80cd740 2024-03-23T03:47:23.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.955+0000 7f1aef000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae80d20c0 con 0x7f1ae80b5aa0 2024-03-23T03:47:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.955+0000 7f1aef000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80b8910 con 0x7f1ae80b84b0 2024-03-23T03:47:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.955+0000 7f1ae7e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1ae80cd740 0x7f1ae80cdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52792/0 (socket says 172.21.15.102:52792) 2024-03-23T03:47:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.955+0000 7f1ae7e00640 1 -- 172.21.15.102:0/4120958062 learned_addr learned my addr 172.21.15.102:0/4120958062 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1ae7400640 1 --2- 172.21.15.102:0/4120958062 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ae80b84b0 0x7f1ae80d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1ae6a00640 1 -- 172.21.15.102:0/4120958062 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581531555 0 0) 0x7f1ae80d1280 con 0x7f1ae80cd740 2024-03-23T03:47:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1aed000640 1 --2- 172.21.15.102:0/4120958062 >> v2:172.21.15.173:3300/0 conn(0x7f1ae80b5aa0 0x7f1ae80b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1ae6a00640 1 -- 172.21.15.102:0/4120958062 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1acc003520 con 0x7f1ae80cd740 2024-03-23T03:47:23.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1ae6a00640 1 -- 172.21.15.102:0/4120958062 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2346609023 0 0) 0x7f1acc003520 con 0x7f1ae80cd740 2024-03-23T03:47:23.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1ae6a00640 1 -- 172.21.15.102:0/4120958062 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ae80b84b0 msgr2=0x7f1ae80d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:23.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1ae6a00640 1 --2- 172.21.15.102:0/4120958062 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ae80b84b0 0x7f1ae80d0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1ae6a00640 1 -- 172.21.15.102:0/4120958062 >> v2:172.21.15.173:3300/0 conn(0x7f1ae80b5aa0 msgr2=0x7f1ae80b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:23.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1ae6a00640 1 --2- 172.21.15.102:0/4120958062 >> v2:172.21.15.173:3300/0 conn(0x7f1ae80b5aa0 0x7f1ae80b5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1ae6a00640 1 -- 172.21.15.102:0/4120958062 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae80d14b0 con 0x7f1ae80cd740 2024-03-23T03:47:23.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1ae6a00640 1 -- 172.21.15.102:0/4120958062 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f1ae00019f0 con 0x7f1ae80cd740 2024-03-23T03:47:23.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1ae6a00640 1 -- 172.21.15.102:0/4120958062 >> v1:172.21.15.102:6789/0 conn(0x7f1ae80cd740 legacy=0x7f1ae80cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:23.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1ae6a00640 1 --2- 172.21.15.102:0/4120958062 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ae80b84b0 0x7f1acc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1ae6a00640 1 --2- 172.21.15.102:0/4120958062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1acc003ef0 0x7f1acc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1ae6a00640 1 --2- 172.21.15.102:0/4120958062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1acc004d00 0x7f1acc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1ae6a00640 1 -- 172.21.15.102:0/4120958062 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80d20c0 con 0x7f1acc003ef0 2024-03-23T03:47:23.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1ae6a00640 1 -- 172.21.15.102:0/4120958062 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80b8910 con 0x7f1acc004d00 2024-03-23T03:47:23.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.956+0000 7f1ae6a00640 1 -- 172.21.15.102:0/4120958062 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1acc00b980 con 0x7f1ae80b84b0 2024-03-23T03:47:23.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.957+0000 7f1ae7400640 1 --2- 172.21.15.102:0/4120958062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1acc003ef0 0x7f1acc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.957+0000 7f1ae7e00640 1 --2- 172.21.15.102:0/4120958062 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ae80b84b0 0x7f1acc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.957+0000 7f1ae7400640 1 -- 172.21.15.102:0/4120958062 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ae80b84b0 msgr2=0x7f1acc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:23.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.957+0000 7f1ae7400640 1 --2- 172.21.15.102:0/4120958062 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ae80b84b0 0x7f1acc003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.957+0000 7f1ae7400640 1 -- 172.21.15.102:0/4120958062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1acc004d00 msgr2=0x7f1acc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:23.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.957+0000 7f1aed000640 1 --2- 172.21.15.102:0/4120958062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1acc004d00 0x7f1acc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.957+0000 7f1ae7400640 1 --2- 172.21.15.102:0/4120958062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1acc004d00 0x7f1acc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.957+0000 7f1ae7400640 1 -- 172.21.15.102:0/4120958062 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1acc003520 con 0x7f1acc003ef0 2024-03-23T03:47:23.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.957+0000 7f1ae7e00640 1 --2- 172.21.15.102:0/4120958062 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ae80b84b0 0x7f1acc003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:23.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.957+0000 7f1aed000640 1 --2- 172.21.15.102:0/4120958062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1acc004d00 0x7f1acc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:23.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.958+0000 7f1ae7400640 1 --2- 172.21.15.102:0/4120958062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1acc003ef0 0x7f1acc0047c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0009ac0 tx=0x7f1acc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b1aeecefdfe4879 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:23.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.958+0000 7f1ae6a00640 1 -- 172.21.15.102:0/4120958062 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad002c6b0 con 0x7f1acc003ef0 2024-03-23T03:47:23.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.958+0000 7f1ae6a00640 1 -- 172.21.15.102:0/4120958062 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad002c810 con 0x7f1acc003ef0 2024-03-23T03:47:23.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.958+0000 7f1ae6a00640 1 -- 172.21.15.102:0/4120958062 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad002cba0 con 0x7f1acc003ef0 2024-03-23T03:47:23.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.959+0000 7f1aef000640 1 -- 172.21.15.102:0/4120958062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1acc003ef0 msgr2=0x7f1acc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:23.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.959+0000 7f1aef000640 1 --2- 172.21.15.102:0/4120958062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1acc003ef0 0x7f1acc0047c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0009ac0 tx=0x7f1acc0057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.959+0000 7f1aed000640 1 -- 172.21.15.102:0/4120958062 reap_dead start 2024-03-23T03:47:23.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.959+0000 7f1aef000640 1 -- 172.21.15.102:0/4120958062 shutdown_connections 2024-03-23T03:47:23.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.959+0000 7f1aef000640 1 -- 172.21.15.102:0/4120958062 >> 172.21.15.102:0/4120958062 conn(0x7f1ae80cf5d0 msgr2=0x7f1ae80cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:23.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.959+0000 7f1aef000640 1 -- 172.21.15.102:0/4120958062 shutdown_connections 2024-03-23T03:47:23.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.959+0000 7f1aef000640 1 -- 172.21.15.102:0/4120958062 wait complete. 2024-03-23T03:47:23.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.961+0000 7f1aef000640 1 Processor -- start 2024-03-23T03:47:23.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.961+0000 7f1aef000640 1 -- start start 2024-03-23T03:47:23.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.962+0000 7f1aef000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1acc003ef0 0x7f1ae8156410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.962+0000 7f1aef000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae81572d0 0x7f1ae81f5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.962+0000 7f1aef000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ae81f5a40 0x7f1ae81f1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.962+0000 7f1aef000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae8156950 con 0x7f1ae81572d0 2024-03-23T03:47:23.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.962+0000 7f1aef000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae81f5ef0 con 0x7f1acc003ef0 2024-03-23T03:47:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.962+0000 7f1aef000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae80cbff0 con 0x7f1ae81f5a40 2024-03-23T03:47:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.962+0000 7f1ae7400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae81572d0 0x7f1ae81f5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.962+0000 7f1ae7400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae81572d0 0x7f1ae81f5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33174/0 (socket says 172.21.15.102:33174) 2024-03-23T03:47:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.962+0000 7f1ae7400640 1 -- 172.21.15.102:0/311142766 learned_addr learned my addr 172.21.15.102:0/311142766 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.963+0000 7f1ae7e00640 1 --2- 172.21.15.102:0/311142766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1acc003ef0 0x7f1ae8156410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.963+0000 7f1aed000640 1 --2- 172.21.15.102:0/311142766 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ae81f5a40 0x7f1ae81f1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.963+0000 7f1ae7400640 1 -- 172.21.15.102:0/311142766 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ae81f5a40 msgr2=0x7f1ae81f1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.963+0000 7f1ae7400640 1 --2- 172.21.15.102:0/311142766 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ae81f5a40 0x7f1ae81f1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.963+0000 7f1ae7400640 1 -- 172.21.15.102:0/311142766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1acc003ef0 msgr2=0x7f1ae8156410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.963+0000 7f1ae7400640 1 --2- 172.21.15.102:0/311142766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1acc003ef0 0x7f1ae8156410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.963+0000 7f1ae7400640 1 -- 172.21.15.102:0/311142766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae80cc830 con 0x7f1ae81572d0 2024-03-23T03:47:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.963+0000 7f1ae7400640 1 --2- 172.21.15.102:0/311142766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae81572d0 0x7f1ae81f5500 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1ad002c2b0 tx=0x7f1ad0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.963+0000 7f1adfe00640 1 -- 172.21.15.102:0/311142766 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0003810 con 0x7f1ae81572d0 2024-03-23T03:47:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.963+0000 7f1adfe00640 1 -- 172.21.15.102:0/311142766 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad0003970 con 0x7f1ae81572d0 2024-03-23T03:47:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.963+0000 7f1adfe00640 1 -- 172.21.15.102:0/311142766 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad0003ce0 con 0x7f1ae81572d0 2024-03-23T03:47:23.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.963+0000 7f1aef000640 1 -- 172.21.15.102:0/311142766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae80cc270 con 0x7f1ae81572d0 2024-03-23T03:47:23.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.964+0000 7f1aef000640 1 -- 172.21.15.102:0/311142766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae80b3450 con 0x7f1ae81572d0 2024-03-23T03:47:23.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.965+0000 7f1aef000640 1 -- 172.21.15.102:0/311142766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae8073390 con 0x7f1ae81572d0 2024-03-23T03:47:23.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.966+0000 7f1adfe00640 1 -- 172.21.15.102:0/311142766 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1ad0034440 con 0x7f1ae81572d0 2024-03-23T03:47:23.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.966+0000 7f1adfe00640 1 --2- 172.21.15.102:0/311142766 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1ac406ac00 0x7f1ac406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:23.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.967+0000 7f1adfe00640 1 -- 172.21.15.102:0/311142766 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1ad00ab890 con 0x7f1ae81572d0 2024-03-23T03:47:23.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.967+0000 7f1ae7e00640 1 --2- 172.21.15.102:0/311142766 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1ac406ac00 0x7f1ac406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:23.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.967+0000 7f1ae7e00640 1 --2- 172.21.15.102:0/311142766 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1ac406ac00 0x7f1ac406d0c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0001ba0 tx=0x7f1ae0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:23.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:23.971+0000 7f1adfe00640 1 -- 172.21.15.102:0/311142766 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1ad0076940 con 0x7f1ae81572d0 2024-03-23T03:47:24.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.163+0000 7f1aef000640 1 -- 172.21.15.102:0/311142766 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1ae80b20b0 con 0x7f1ae81572d0 2024-03-23T03:47:24.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.164+0000 7f1adfe00640 1 -- 172.21.15.102:0/311142766 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1ad00044e0 con 0x7f1ae81572d0 2024-03-23T03:47:24.164 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:24.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.166+0000 7f1aef000640 1 -- 172.21.15.102:0/311142766 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1ac406ac00 msgr2=0x7f1ac406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:24.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.166+0000 7f1aef000640 1 --2- 172.21.15.102:0/311142766 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1ac406ac00 0x7f1ac406d0c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0001ba0 tx=0x7f1ae0009040 comp rx=0 tx=0).stop 2024-03-23T03:47:24.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.166+0000 7f1aef000640 1 -- 172.21.15.102:0/311142766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae81572d0 msgr2=0x7f1ae81f5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:24.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.166+0000 7f1aef000640 1 --2- 172.21.15.102:0/311142766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae81572d0 0x7f1ae81f5500 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1ad002c2b0 tx=0x7f1ad0004730 comp rx=0 tx=0).stop 2024-03-23T03:47:24.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.166+0000 7f1aef000640 1 -- 172.21.15.102:0/311142766 shutdown_connections 2024-03-23T03:47:24.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.166+0000 7f1aef000640 1 --2- 172.21.15.102:0/311142766 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1ac406ac00 0x7f1ac406d0c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.166+0000 7f1aef000640 1 --2- 172.21.15.102:0/311142766 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ae81f5a40 0x7f1ae81f1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.166+0000 7f1aef000640 1 --2- 172.21.15.102:0/311142766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae81572d0 0x7f1ae81f5500 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.166+0000 7f1aef000640 1 --2- 172.21.15.102:0/311142766 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1acc003ef0 0x7f1ae8156410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.166+0000 7f1aef000640 1 -- 172.21.15.102:0/311142766 >> 172.21.15.102:0/311142766 conn(0x7f1ae80cf5d0 msgr2=0x7f1ae80d2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:24.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.166+0000 7f1aef000640 1 -- 172.21.15.102:0/311142766 shutdown_connections 2024-03-23T03:47:24.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.166+0000 7f1aef000640 1 -- 172.21.15.102:0/311142766 wait complete. 2024-03-23T03:47:24.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:47:24.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.397+0000 7f5a9f200640 1 Processor -- start 2024-03-23T03:47:24.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.398+0000 7f5a9f200640 1 -- start start 2024-03-23T03:47:24.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.399+0000 7f5a9f200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f5a98160d20 0x7f5a98164920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:24.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.399+0000 7f5a9f200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5a980ca7b0 0x7f5a980ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:24.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.399+0000 7f5a9f200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5a980cee70 0x7f5a98070150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:24.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.399+0000 7f5a9f200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a980cf300 con 0x7f5a980cee70 2024-03-23T03:47:24.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.399+0000 7f5a9f200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a98165830 con 0x7f5a98160d20 2024-03-23T03:47:24.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.400+0000 7f5a9f200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a98164f90 con 0x7f5a980ca7b0 2024-03-23T03:47:24.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.400+0000 7f5a9d200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5a980cee70 0x7f5a98070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:24.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.400+0000 7f5a9d200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5a980cee70 0x7f5a98070150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33190/0 (socket says 172.21.15.102:33190) 2024-03-23T03:47:24.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.400+0000 7f5a9d200640 1 -- 172.21.15.102:0/2434385583 learned_addr learned my addr 172.21.15.102:0/2434385583 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:24.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.400+0000 7f5a8f400640 1 --2- 172.21.15.102:0/2434385583 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5a980ca7b0 0x7f5a980ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:24.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.400+0000 7f5a97e00640 1 --2- 172.21.15.102:0/2434385583 >> v2:172.21.15.173:3300/0 conn(0x7f5a98160d20 0x7f5a98164920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:24.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.400+0000 7f5a9d200640 1 -- 172.21.15.102:0/2434385583 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5a980ca7b0 msgr2=0x7f5a980ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:24.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.400+0000 7f5a9d200640 1 --2- 172.21.15.102:0/2434385583 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5a980ca7b0 0x7f5a980ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.400+0000 7f5a9d200640 1 -- 172.21.15.102:0/2434385583 >> v2:172.21.15.173:3300/0 conn(0x7f5a98160d20 msgr2=0x7f5a98164920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:24.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.400+0000 7f5a9d200640 1 --2- 172.21.15.102:0/2434385583 >> v2:172.21.15.173:3300/0 conn(0x7f5a98160d20 0x7f5a98164920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.400+0000 7f5a9d200640 1 -- 172.21.15.102:0/2434385583 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a98165270 con 0x7f5a980cee70 2024-03-23T03:47:24.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.401+0000 7f5a9d200640 1 --2- 172.21.15.102:0/2434385583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a980cee70 0x7f5a98070150 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5a980cf4c0 tx=0x7f5a9002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=412d09d2e30cd7cc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:24.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.402+0000 7f5a97400640 1 -- 172.21.15.102:0/2434385583 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9002c9e0 con 0x7f5a980cee70 2024-03-23T03:47:24.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.402+0000 7f5a97400640 1 -- 172.21.15.102:0/2434385583 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a9002cb40 con 0x7f5a980cee70 2024-03-23T03:47:24.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.402+0000 7f5a97400640 1 -- 172.21.15.102:0/2434385583 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a900343f0 con 0x7f5a980cee70 2024-03-23T03:47:24.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.402+0000 7f5a9f200640 1 -- 172.21.15.102:0/2434385583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a980cee70 msgr2=0x7f5a98070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:24.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.402+0000 7f5a9f200640 1 --2- 172.21.15.102:0/2434385583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a980cee70 0x7f5a98070150 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5a980cf4c0 tx=0x7f5a9002be40 comp rx=0 tx=0).stop 2024-03-23T03:47:24.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.403+0000 7f5a9f200640 1 -- 172.21.15.102:0/2434385583 shutdown_connections 2024-03-23T03:47:24.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.403+0000 7f5a9f200640 1 --2- 172.21.15.102:0/2434385583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a980cee70 0x7f5a98070150 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.403+0000 7f5a9f200640 1 --2- 172.21.15.102:0/2434385583 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5a980ca7b0 0x7f5a980ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.403+0000 7f5a9f200640 1 --2- 172.21.15.102:0/2434385583 >> v2:172.21.15.173:3300/0 conn(0x7f5a98160d20 0x7f5a98164920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.403+0000 7f5a9f200640 1 -- 172.21.15.102:0/2434385583 >> 172.21.15.102:0/2434385583 conn(0x7f5a9806a0b0 msgr2=0x7f5a9806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:24.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.403+0000 7f5a9f200640 1 -- 172.21.15.102:0/2434385583 shutdown_connections 2024-03-23T03:47:24.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.403+0000 7f5a9f200640 1 -- 172.21.15.102:0/2434385583 wait complete. 2024-03-23T03:47:24.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.405+0000 7f5a9f200640 1 Processor -- start 2024-03-23T03:47:24.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.405+0000 7f5a9f200640 1 -- start start 2024-03-23T03:47:24.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.406+0000 7f5a9f200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a980ca7b0 0x7f5a980cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:24.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.406+0000 7f5a9f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a980cee70 0x7f5a9815c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:24.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.406+0000 7f5a9f200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5a98160d20 0x7f5a980af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:24.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.406+0000 7f5a9f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9815cee0 con 0x7f5a980cee70 2024-03-23T03:47:24.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.406+0000 7f5a9f200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9815d1c0 con 0x7f5a980ca7b0 2024-03-23T03:47:24.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.406+0000 7f5a9f200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9815d4a0 con 0x7f5a98160d20 2024-03-23T03:47:24.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.406+0000 7f5a8f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a980cee70 0x7f5a9815c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:24.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.406+0000 7f5a8f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a980cee70 0x7f5a9815c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33204/0 (socket says 172.21.15.102:33204) 2024-03-23T03:47:24.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.406+0000 7f5a8f400640 1 -- 172.21.15.102:0/597962872 learned_addr learned my addr 172.21.15.102:0/597962872 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:24.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.406+0000 7f5a8f400640 1 -- 172.21.15.102:0/597962872 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5a98160d20 msgr2=0x7f5a980af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:24.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.406+0000 7f5a97e00640 1 --2- 172.21.15.102:0/597962872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a980ca7b0 0x7f5a980cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:24.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.406+0000 7f5a8f400640 1 --2- 172.21.15.102:0/597962872 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5a98160d20 0x7f5a980af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.406+0000 7f5a8f400640 1 -- 172.21.15.102:0/597962872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a980ca7b0 msgr2=0x7f5a980cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:24.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.406+0000 7f5a8f400640 1 --2- 172.21.15.102:0/597962872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a980ca7b0 0x7f5a980cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.406+0000 7f5a8f400640 1 -- 172.21.15.102:0/597962872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a90009b30 con 0x7f5a980cee70 2024-03-23T03:47:24.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.407+0000 7f5a8f400640 1 --2- 172.21.15.102:0/597962872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a980cee70 0x7f5a9815c690 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5a8000d5f0 tx=0x7f5a8000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:24.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.407+0000 7f5a94c00640 1 -- 172.21.15.102:0/597962872 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a800125f0 con 0x7f5a980cee70 2024-03-23T03:47:24.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.407+0000 7f5a94c00640 1 -- 172.21.15.102:0/597962872 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a80012750 con 0x7f5a980cee70 2024-03-23T03:47:24.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.407+0000 7f5a94c00640 1 -- 172.21.15.102:0/597962872 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a80012b00 con 0x7f5a980cee70 2024-03-23T03:47:24.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.407+0000 7f5a9f200640 1 -- 172.21.15.102:0/597962872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a9815d780 con 0x7f5a980cee70 2024-03-23T03:47:24.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.407+0000 7f5a9f200640 1 -- 172.21.15.102:0/597962872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a9815cbd0 con 0x7f5a980cee70 2024-03-23T03:47:24.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.408+0000 7f5a9f200640 1 -- 172.21.15.102:0/597962872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a98073390 con 0x7f5a980cee70 2024-03-23T03:47:24.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.409+0000 7f5a94c00640 1 -- 172.21.15.102:0/597962872 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5a80012c60 con 0x7f5a980cee70 2024-03-23T03:47:24.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.411+0000 7f5a94c00640 1 --2- 172.21.15.102:0/597962872 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5a7406ac00 0x7f5a7406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:24.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.411+0000 7f5a94c00640 1 -- 172.21.15.102:0/597962872 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5a80089b30 con 0x7f5a980cee70 2024-03-23T03:47:24.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.411+0000 7f5a97e00640 1 --2- 172.21.15.102:0/597962872 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5a7406ac00 0x7f5a7406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:24.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.412+0000 7f5a97e00640 1 --2- 172.21.15.102:0/597962872 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5a7406ac00 0x7f5a7406d0c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5a88006fd0 tx=0x7f5a88008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:24.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.415+0000 7f5a94c00640 1 -- 172.21.15.102:0/597962872 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5a80054a20 con 0x7f5a980cee70 2024-03-23T03:47:24.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.608+0000 7f5a9f200640 1 -- 172.21.15.102:0/597962872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5a981648b0 con 0x7f5a980cee70 2024-03-23T03:47:24.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.609+0000 7f5a94c00640 1 -- 172.21.15.102:0/597962872 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5a80058040 con 0x7f5a980cee70 2024-03-23T03:47:24.610 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:47:24.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.611+0000 7f5a9f200640 1 -- 172.21.15.102:0/597962872 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5a7406ac00 msgr2=0x7f5a7406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:24.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.611+0000 7f5a9f200640 1 --2- 172.21.15.102:0/597962872 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5a7406ac00 0x7f5a7406d0c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5a88006fd0 tx=0x7f5a88008040 comp rx=0 tx=0).stop 2024-03-23T03:47:24.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.611+0000 7f5a9f200640 1 -- 172.21.15.102:0/597962872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a980cee70 msgr2=0x7f5a9815c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:24.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.611+0000 7f5a9f200640 1 --2- 172.21.15.102:0/597962872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a980cee70 0x7f5a9815c690 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5a8000d5f0 tx=0x7f5a8000dab0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.612+0000 7f5a9f200640 1 -- 172.21.15.102:0/597962872 shutdown_connections 2024-03-23T03:47:24.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.612+0000 7f5a9f200640 1 --2- 172.21.15.102:0/597962872 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5a7406ac00 0x7f5a7406d0c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.612+0000 7f5a9f200640 1 --2- 172.21.15.102:0/597962872 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5a98160d20 0x7f5a980af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.612+0000 7f5a9f200640 1 --2- 172.21.15.102:0/597962872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a980cee70 0x7f5a9815c690 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.612+0000 7f5a9f200640 1 --2- 172.21.15.102:0/597962872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a980ca7b0 0x7f5a980cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.612+0000 7f5a9f200640 1 -- 172.21.15.102:0/597962872 >> 172.21.15.102:0/597962872 conn(0x7f5a9806a0b0 msgr2=0x7f5a98161980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:24.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.612+0000 7f5a9f200640 1 -- 172.21.15.102:0/597962872 shutdown_connections 2024-03-23T03:47:24.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.612+0000 7f5a9f200640 1 -- 172.21.15.102:0/597962872 wait complete. 2024-03-23T03:47:24.630 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-23T03:47:24.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.834+0000 7f5e89e00640 1 Processor -- start 2024-03-23T03:47:24.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.834+0000 7f5e89e00640 1 -- start start 2024-03-23T03:47:24.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.836+0000 7f5e89e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5e840ce930 0x7f5e840add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.836+0000 7f5e89e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e840cd3e0 0x7f5e840ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.836+0000 7f5e89e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f5e8406ce90 0x7f5e8406d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.836+0000 7f5e89e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e8406c760 con 0x7f5e840ce930 2024-03-23T03:47:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.836+0000 7f5e89e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e840b42d0 con 0x7f5e8406ce90 2024-03-23T03:47:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.836+0000 7f5e89e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e840cf250 con 0x7f5e840cd3e0 2024-03-23T03:47:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.836+0000 7f5e82a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5e840ce930 0x7f5e840add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.837+0000 7f5e82a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5e840ce930 0x7f5e840add30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33212/0 (socket says 172.21.15.102:33212) 2024-03-23T03:47:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.837+0000 7f5e82a00640 1 -- 172.21.15.102:0/3721486327 learned_addr learned my addr 172.21.15.102:0/3721486327 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.837+0000 7f5e82000640 1 --2- 172.21.15.102:0/3721486327 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e840cd3e0 0x7f5e840ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.837+0000 7f5e83400640 1 --2- 172.21.15.102:0/3721486327 >> v2:172.21.15.173:3300/0 conn(0x7f5e8406ce90 0x7f5e8406d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.837+0000 7f5e82a00640 1 -- 172.21.15.102:0/3721486327 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e840cd3e0 msgr2=0x7f5e840ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.837+0000 7f5e82a00640 1 --2- 172.21.15.102:0/3721486327 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e840cd3e0 0x7f5e840ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.837+0000 7f5e82a00640 1 -- 172.21.15.102:0/3721486327 >> v2:172.21.15.173:3300/0 conn(0x7f5e8406ce90 msgr2=0x7f5e8406d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:24.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.837+0000 7f5e82a00640 1 --2- 172.21.15.102:0/3721486327 >> v2:172.21.15.173:3300/0 conn(0x7f5e8406ce90 0x7f5e8406d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.837+0000 7f5e82a00640 1 -- 172.21.15.102:0/3721486327 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e840cdba0 con 0x7f5e840ce930 2024-03-23T03:47:24.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.837+0000 7f5e82000640 1 --2- 172.21.15.102:0/3721486327 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e840cd3e0 0x7f5e840ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:47:24.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.839+0000 7f5e82a00640 1 --2- 172.21.15.102:0/3721486327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e840ce930 0x7f5e840add30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5e840ccc70 tx=0x7f5e7402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85c2c7d15691d5c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:24.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.839+0000 7f5e81600640 1 -- 172.21.15.102:0/3721486327 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e74031860 con 0x7f5e840ce930 2024-03-23T03:47:24.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.839+0000 7f5e81600640 1 -- 172.21.15.102:0/3721486327 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e740319c0 con 0x7f5e840ce930 2024-03-23T03:47:24.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.839+0000 7f5e81600640 1 -- 172.21.15.102:0/3721486327 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e74031d50 con 0x7f5e840ce930 2024-03-23T03:47:24.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.840+0000 7f5e89e00640 1 -- 172.21.15.102:0/3721486327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e840ce930 msgr2=0x7f5e840add30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:24.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.840+0000 7f5e89e00640 1 --2- 172.21.15.102:0/3721486327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e840ce930 0x7f5e840add30 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5e840ccc70 tx=0x7f5e7402b900 comp rx=0 tx=0).stop 2024-03-23T03:47:24.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.840+0000 7f5e89e00640 1 -- 172.21.15.102:0/3721486327 shutdown_connections 2024-03-23T03:47:24.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.840+0000 7f5e89e00640 1 --2- 172.21.15.102:0/3721486327 >> v2:172.21.15.173:3300/0 conn(0x7f5e8406ce90 0x7f5e8406d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.840+0000 7f5e89e00640 1 --2- 172.21.15.102:0/3721486327 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e840cd3e0 0x7f5e840ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.840+0000 7f5e89e00640 1 --2- 172.21.15.102:0/3721486327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e840ce930 0x7f5e840add30 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.840+0000 7f5e89e00640 1 -- 172.21.15.102:0/3721486327 >> 172.21.15.102:0/3721486327 conn(0x7f5e8406a0b0 msgr2=0x7f5e8406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:24.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.840+0000 7f5e89e00640 1 -- 172.21.15.102:0/3721486327 shutdown_connections 2024-03-23T03:47:24.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.840+0000 7f5e89e00640 1 -- 172.21.15.102:0/3721486327 wait complete. 2024-03-23T03:47:24.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.842+0000 7f5e89e00640 1 Processor -- start 2024-03-23T03:47:24.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.842+0000 7f5e89e00640 1 -- start start 2024-03-23T03:47:24.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.844+0000 7f5e89e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e8406ce90 0x7f5e8414af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:24.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.844+0000 7f5e89e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e840cd3e0 0x7f5e840b1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:24.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.844+0000 7f5e89e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e840ce930 0x7f5e840b1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:24.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.844+0000 7f5e89e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8414e010 con 0x7f5e8406ce90 2024-03-23T03:47:24.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.844+0000 7f5e89e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8414e2f0 con 0x7f5e840cd3e0 2024-03-23T03:47:24.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.844+0000 7f5e89e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8414e5d0 con 0x7f5e840ce930 2024-03-23T03:47:24.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.844+0000 7f5e82a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e8406ce90 0x7f5e8414af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:24.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.844+0000 7f5e82a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e8406ce90 0x7f5e8414af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33222/0 (socket says 172.21.15.102:33222) 2024-03-23T03:47:24.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.844+0000 7f5e82a00640 1 -- 172.21.15.102:0/1815884777 learned_addr learned my addr 172.21.15.102:0/1815884777 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:24.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.844+0000 7f5e82a00640 1 -- 172.21.15.102:0/1815884777 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e840ce930 msgr2=0x7f5e840b1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:24.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.844+0000 7f5e82a00640 1 --2- 172.21.15.102:0/1815884777 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e840ce930 0x7f5e840b1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.844+0000 7f5e82a00640 1 -- 172.21.15.102:0/1815884777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e840cd3e0 msgr2=0x7f5e840b1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:24.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.844+0000 7f5e82a00640 1 --2- 172.21.15.102:0/1815884777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e840cd3e0 0x7f5e840b1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:24.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.844+0000 7f5e82a00640 1 -- 172.21.15.102:0/1815884777 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e740094d0 con 0x7f5e8406ce90 2024-03-23T03:47:24.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.845+0000 7f5e82a00640 1 --2- 172.21.15.102:0/1815884777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e8406ce90 0x7f5e8414af60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5e7402b940 tx=0x7f5e740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:24.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.845+0000 7f5e7b400640 1 -- 172.21.15.102:0/1815884777 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7403ad30 con 0x7f5e8406ce90 2024-03-23T03:47:24.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.845+0000 7f5e7b400640 1 -- 172.21.15.102:0/1815884777 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e740443f0 con 0x7f5e8406ce90 2024-03-23T03:47:24.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.845+0000 7f5e7b400640 1 -- 172.21.15.102:0/1815884777 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e740447a0 con 0x7f5e8406ce90 2024-03-23T03:47:24.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.845+0000 7f5e89e00640 1 -- 172.21.15.102:0/1815884777 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e8414e8b0 con 0x7f5e8406ce90 2024-03-23T03:47:24.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.845+0000 7f5e89e00640 1 -- 172.21.15.102:0/1815884777 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e8414b4a0 con 0x7f5e8406ce90 2024-03-23T03:47:24.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.846+0000 7f5e89e00640 1 -- 172.21.15.102:0/1815884777 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e8414edd0 con 0x7f5e8406ce90 2024-03-23T03:47:24.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.847+0000 7f5e7b400640 1 -- 172.21.15.102:0/1815884777 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5e7404d450 con 0x7f5e8406ce90 2024-03-23T03:47:24.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.848+0000 7f5e7b400640 1 --2- 172.21.15.102:0/1815884777 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e6406ac00 0x7f5e6406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:24.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.848+0000 7f5e7b400640 1 -- 172.21.15.102:0/1815884777 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5e740abc00 con 0x7f5e8406ce90 2024-03-23T03:47:24.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.849+0000 7f5e82000640 1 --2- 172.21.15.102:0/1815884777 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e6406ac00 0x7f5e6406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:24.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.849+0000 7f5e82000640 1 --2- 172.21.15.102:0/1815884777 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e6406ac00 0x7f5e6406d0c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5e70006e80 tx=0x7f5e70008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:24.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:24.853+0000 7f5e7b400640 1 -- 172.21.15.102:0/1815884777 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5e8414edd0 con 0x7f5e8406ce90 2024-03-23T03:47:25.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.050+0000 7f5e89e00640 1 -- 172.21.15.102:0/1815884777 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5e8414edd0 con 0x7f5e8406ce90 2024-03-23T03:47:25.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.051+0000 7f5e7b400640 1 -- 172.21.15.102:0/1815884777 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5e8414edd0 con 0x7f5e8406ce90 2024-03-23T03:47:25.052 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:25.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.054+0000 7f5e89e00640 1 -- 172.21.15.102:0/1815884777 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e6406ac00 msgr2=0x7f5e6406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:25.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.054+0000 7f5e89e00640 1 --2- 172.21.15.102:0/1815884777 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e6406ac00 0x7f5e6406d0c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5e70006e80 tx=0x7f5e70008040 comp rx=0 tx=0).stop 2024-03-23T03:47:25.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.054+0000 7f5e89e00640 1 -- 172.21.15.102:0/1815884777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e8406ce90 msgr2=0x7f5e8414af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:25.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.054+0000 7f5e89e00640 1 --2- 172.21.15.102:0/1815884777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e8406ce90 0x7f5e8414af60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5e7402b940 tx=0x7f5e740046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.054+0000 7f5e89e00640 1 -- 172.21.15.102:0/1815884777 shutdown_connections 2024-03-23T03:47:25.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.054+0000 7f5e89e00640 1 --2- 172.21.15.102:0/1815884777 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5e6406ac00 0x7f5e6406d0c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.054+0000 7f5e89e00640 1 --2- 172.21.15.102:0/1815884777 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5e840ce930 0x7f5e840b1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.054+0000 7f5e89e00640 1 --2- 172.21.15.102:0/1815884777 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e840cd3e0 0x7f5e840b1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.054+0000 7f5e89e00640 1 --2- 172.21.15.102:0/1815884777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e8406ce90 0x7f5e8414af60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.054+0000 7f5e89e00640 1 -- 172.21.15.102:0/1815884777 >> 172.21.15.102:0/1815884777 conn(0x7f5e8406a0b0 msgr2=0x7f5e840b0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:25.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.054+0000 7f5e89e00640 1 -- 172.21.15.102:0/1815884777 shutdown_connections 2024-03-23T03:47:25.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.054+0000 7f5e89e00640 1 -- 172.21.15.102:0/1815884777 wait complete. 2024-03-23T03:47:25.072 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:47:25.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.273+0000 7f1114a00640 1 Processor -- start 2024-03-23T03:47:25.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.273+0000 7f1114a00640 1 -- start start 2024-03-23T03:47:25.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.275+0000 7f1114a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1110160800 0x7f11100b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:25.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.275+0000 7f1114a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f11100b1be0 0x7f111006c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:25.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.275+0000 7f1114a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11100b2030 con 0x7f111013faa0 2024-03-23T03:47:25.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.275+0000 7f1114a00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1110161640 con 0x7f11100b1be0 2024-03-23T03:47:25.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.275+0000 7f1114a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11100b38a0 con 0x7f1110160800 2024-03-23T03:47:25.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.276+0000 7f1105400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1110160800 0x7f11100b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:25.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.276+0000 7f110e800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f11100b1be0 0x7f111006c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:25.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.276+0000 7f1105400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1110160800 0x7f11100b16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:57706/0 (socket says 172.21.15.102:57706) 2024-03-23T03:47:25.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.276+0000 7f1105400640 1 -- 172.21.15.102:0/2778400880 learned_addr learned my addr 172.21.15.102:0/2778400880 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:25.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.276+0000 7f1105400640 1 -- 172.21.15.102:0/2778400880 >> v2:172.21.15.173:3300/0 conn(0x7f11100b1be0 msgr2=0x7f111006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:25.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.276+0000 7f1105400640 1 --2- 172.21.15.102:0/2778400880 >> v2:172.21.15.173:3300/0 conn(0x7f11100b1be0 0x7f111006c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.276+0000 7f1105400640 1 -- 172.21.15.102:0/2778400880 >> v1:172.21.15.173:6789/0 conn(0x7f111013faa0 legacy=0x7f1110160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:25.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.277+0000 7f1105400640 1 -- 172.21.15.102:0/2778400880 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11100b3b80 con 0x7f1110160800 2024-03-23T03:47:25.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.278+0000 7f1105400640 1 --2- 172.21.15.102:0/2778400880 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1110160800 0x7f11100b16a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009870 tx=0x7f10f802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a29ee6f4d220942c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:25.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.278+0000 7f110d400640 1 -- 172.21.15.102:0/2778400880 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f8030840 con 0x7f1110160800 2024-03-23T03:47:25.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.278+0000 7f110d400640 1 -- 172.21.15.102:0/2778400880 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f80309a0 con 0x7f1110160800 2024-03-23T03:47:25.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.278+0000 7f110d400640 1 -- 172.21.15.102:0/2778400880 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f8030d30 con 0x7f1110160800 2024-03-23T03:47:25.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.279+0000 7f1114a00640 1 -- 172.21.15.102:0/2778400880 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1110160800 msgr2=0x7f11100b16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:25.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.279+0000 7f1114a00640 1 --2- 172.21.15.102:0/2778400880 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1110160800 0x7f11100b16a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f10f8009870 tx=0x7f10f802b980 comp rx=0 tx=0).stop 2024-03-23T03:47:25.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.279+0000 7f1114a00640 1 -- 172.21.15.102:0/2778400880 shutdown_connections 2024-03-23T03:47:25.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.279+0000 7f1114a00640 1 --2- 172.21.15.102:0/2778400880 >> v2:172.21.15.173:3300/0 conn(0x7f11100b1be0 0x7f111006c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.279+0000 7f1114a00640 1 --2- 172.21.15.102:0/2778400880 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1110160800 0x7f11100b16a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.279+0000 7f1114a00640 1 -- 172.21.15.102:0/2778400880 >> 172.21.15.102:0/2778400880 conn(0x7f111006a3d0 msgr2=0x7f11100b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:25.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.279+0000 7f1114a00640 1 -- 172.21.15.102:0/2778400880 shutdown_connections 2024-03-23T03:47:25.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.279+0000 7f1114a00640 1 -- 172.21.15.102:0/2778400880 wait complete. 2024-03-23T03:47:25.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.281+0000 7f1114a00640 1 Processor -- start 2024-03-23T03:47:25.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.281+0000 7f1114a00640 1 -- start start 2024-03-23T03:47:25.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.282+0000 7f1114a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f11100b1be0 0x7f11100ca890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:25.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.283+0000 7f1114a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f111013faa0 0x7f11100af360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:25.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.283+0000 7f1114a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1110160800 0x7f11100af8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:25.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.283+0000 7f1114a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11100cba80 con 0x7f111013faa0 2024-03-23T03:47:25.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.283+0000 7f1114a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11100cae70 con 0x7f1110160800 2024-03-23T03:47:25.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.283+0000 7f1114a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11100cb150 con 0x7f11100b1be0 2024-03-23T03:47:25.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.283+0000 7f1105400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f111013faa0 0x7f11100af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:25.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.283+0000 7f1105400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f111013faa0 0x7f11100af360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33224/0 (socket says 172.21.15.102:33224) 2024-03-23T03:47:25.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.283+0000 7f1105400640 1 -- 172.21.15.102:0/1209700376 learned_addr learned my addr 172.21.15.102:0/1209700376 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:25.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.283+0000 7f110de00640 1 --2- 172.21.15.102:0/1209700376 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f11100b1be0 0x7f11100ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:25.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.283+0000 7f110e800640 1 --2- 172.21.15.102:0/1209700376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1110160800 0x7f11100af8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:25.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.283+0000 7f1105400640 1 -- 172.21.15.102:0/1209700376 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f11100b1be0 msgr2=0x7f11100ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:25.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.283+0000 7f1105400640 1 --2- 172.21.15.102:0/1209700376 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f11100b1be0 0x7f11100ca890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.283+0000 7f1105400640 1 -- 172.21.15.102:0/1209700376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1110160800 msgr2=0x7f11100af8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:25.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.283+0000 7f1105400640 1 --2- 172.21.15.102:0/1209700376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1110160800 0x7f11100af8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.283+0000 7f1105400640 1 -- 172.21.15.102:0/1209700376 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f80094d0 con 0x7f111013faa0 2024-03-23T03:47:25.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.284+0000 7f1105400640 1 --2- 172.21.15.102:0/1209700376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f111013faa0 0x7f11100af360 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f10f802b9c0 tx=0x7f10f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:25.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.284+0000 7f1106a00640 1 -- 172.21.15.102:0/1209700376 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f803e070 con 0x7f111013faa0 2024-03-23T03:47:25.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.284+0000 7f1106a00640 1 -- 172.21.15.102:0/1209700376 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10f8039a10 con 0x7f111013faa0 2024-03-23T03:47:25.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.284+0000 7f1114a00640 1 -- 172.21.15.102:0/1209700376 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11100cb430 con 0x7f111013faa0 2024-03-23T03:47:25.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.284+0000 7f1106a00640 1 -- 172.21.15.102:0/1209700376 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10f8039de0 con 0x7f111013faa0 2024-03-23T03:47:25.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.284+0000 7f1114a00640 1 -- 172.21.15.102:0/1209700376 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11100b03f0 con 0x7f111013faa0 2024-03-23T03:47:25.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.286+0000 7f1114a00640 1 -- 172.21.15.102:0/1209700376 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11100cbbf0 con 0x7f111013faa0 2024-03-23T03:47:25.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.286+0000 7f1106a00640 1 -- 172.21.15.102:0/1209700376 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f10f804b400 con 0x7f111013faa0 2024-03-23T03:47:25.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.287+0000 7f1106a00640 1 --2- 172.21.15.102:0/1209700376 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f10ec06ac00 0x7f10ec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:25.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.287+0000 7f1106a00640 1 -- 172.21.15.102:0/1209700376 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f10f80abba0 con 0x7f111013faa0 2024-03-23T03:47:25.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.287+0000 7f110de00640 1 --2- 172.21.15.102:0/1209700376 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f10ec06ac00 0x7f10ec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:25.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.288+0000 7f110de00640 1 --2- 172.21.15.102:0/1209700376 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f10ec06ac00 0x7f10ec06d0c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1100007fd0 tx=0x7f1100009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:25.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.292+0000 7f1106a00640 1 -- 172.21.15.102:0/1209700376 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f10f8076a80 con 0x7f111013faa0 2024-03-23T03:47:25.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.525+0000 7f1114a00640 1 -- 172.21.15.102:0/1209700376 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1110073390 con 0x7f111013faa0 2024-03-23T03:47:25.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.527+0000 7f1106a00640 1 -- 172.21.15.102:0/1209700376 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4739 (secure 0 0 0) 0x7f10f8034020 con 0x7f111013faa0 2024-03-23T03:47:25.527 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:25.527 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/891818172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":891818172},{"type":"v1","addr":"172.21.15.102:6836","nonce":891818172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2404489233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2404489233},{"type":"v1","addr":"172.21.15.102:6837","nonce":2404489233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/1839967745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":1839967745},{"type":"v1","addr":"172.21.15.173:6835","nonce":1839967745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:46:53.912287+0000","modified":"2024-03-23T03:47:14.818670+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6834/1462637509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1462637509},{"type":"v1","addr":"172.21.15.173:6834","nonce":1462637509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-23T03:47:25.528 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 8 2024-03-23T03:47:25.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.529+0000 7f1114a00640 1 -- 172.21.15.102:0/1209700376 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f10ec06ac00 msgr2=0x7f10ec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:25.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.529+0000 7f1114a00640 1 --2- 172.21.15.102:0/1209700376 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f10ec06ac00 0x7f10ec06d0c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1100007fd0 tx=0x7f1100009040 comp rx=0 tx=0).stop 2024-03-23T03:47:25.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.529+0000 7f1114a00640 1 -- 172.21.15.102:0/1209700376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f111013faa0 msgr2=0x7f11100af360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:25.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.529+0000 7f1114a00640 1 --2- 172.21.15.102:0/1209700376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f111013faa0 0x7f11100af360 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f10f802b9c0 tx=0x7f10f80046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.529+0000 7f1114a00640 1 -- 172.21.15.102:0/1209700376 shutdown_connections 2024-03-23T03:47:25.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.529+0000 7f1114a00640 1 --2- 172.21.15.102:0/1209700376 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f10ec06ac00 0x7f10ec06d0c0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.529+0000 7f1114a00640 1 --2- 172.21.15.102:0/1209700376 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1110160800 0x7f11100af8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.529+0000 7f1114a00640 1 --2- 172.21.15.102:0/1209700376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f111013faa0 0x7f11100af360 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.529+0000 7f1114a00640 1 --2- 172.21.15.102:0/1209700376 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f11100b1be0 0x7f11100ca890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.529+0000 7f1114a00640 1 -- 172.21.15.102:0/1209700376 >> 172.21.15.102:0/1209700376 conn(0x7f111006a3d0 msgr2=0x7f111006c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:25.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.530+0000 7f1114a00640 1 -- 172.21.15.102:0/1209700376 shutdown_connections 2024-03-23T03:47:25.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.530+0000 7f1114a00640 1 -- 172.21.15.102:0/1209700376 wait complete. 2024-03-23T03:47:25.548 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:25.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.751+0000 7f9845000640 1 Processor -- start 2024-03-23T03:47:25.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.751+0000 7f9845000640 1 -- start start 2024-03-23T03:47:25.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.753+0000 7f9845000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f98400ccec0 0x7f98400cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:25.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.753+0000 7f9845000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984006da70 con 0x7f98400b37f0 2024-03-23T03:47:25.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.753+0000 7f9845000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98400b1d00 con 0x7f984014aa40 2024-03-23T03:47:25.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.753+0000 7f9845000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98400b1fe0 con 0x7f98400ccec0 2024-03-23T03:47:25.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.753+0000 7f983d400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f98400b37f0 0x7f984006c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52802/0 (socket says 172.21.15.102:52802) 2024-03-23T03:47:25.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.753+0000 7f983d400640 1 -- 172.21.15.102:0/3815763586 learned_addr learned my addr 172.21.15.102:0/3815763586 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:25.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.754+0000 7f983e800640 1 --2- 172.21.15.102:0/3815763586 >> v2:172.21.15.173:3300/0 conn(0x7f98400ccec0 0x7f98400cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:25.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.754+0000 7f983ca00640 1 -- 172.21.15.102:0/3815763586 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4177367150 0 0) 0x7f984006da70 con 0x7f98400b37f0 2024-03-23T03:47:25.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.754+0000 7f983ca00640 1 -- 172.21.15.102:0/3815763586 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9824003520 con 0x7f98400b37f0 2024-03-23T03:47:25.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.754+0000 7f983ca00640 1 -- 172.21.15.102:0/3815763586 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2217514792 0 0) 0x7f9824003520 con 0x7f98400b37f0 2024-03-23T03:47:25.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.754+0000 7f983ca00640 1 -- 172.21.15.102:0/3815763586 >> v2:172.21.15.173:3300/0 conn(0x7f98400ccec0 msgr2=0x7f98400cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:25.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.754+0000 7f983ca00640 1 --2- 172.21.15.102:0/3815763586 >> v2:172.21.15.173:3300/0 conn(0x7f98400ccec0 0x7f98400cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.754+0000 7f983ca00640 1 -- 172.21.15.102:0/3815763586 >> v1:172.21.15.173:6789/0 conn(0x7f984014aa40 legacy=0x7f98400cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:25.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.754+0000 7f983ca00640 1 -- 172.21.15.102:0/3815763586 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98400b22c0 con 0x7f98400b37f0 2024-03-23T03:47:25.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.754+0000 7f983ca00640 1 -- 172.21.15.102:0/3815763586 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f982c001670 con 0x7f98400b37f0 2024-03-23T03:47:25.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.754+0000 7f983ca00640 1 -- 172.21.15.102:0/3815763586 >> v1:172.21.15.102:6789/0 conn(0x7f98400b37f0 legacy=0x7f984006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:25.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.754+0000 7f983ca00640 1 --2- 172.21.15.102:0/3815763586 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9824003970 0x7f9824003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:25.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.754+0000 7f983ca00640 1 --2- 172.21.15.102:0/3815763586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98240047c0 0x7f9824004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:25.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.754+0000 7f983ca00640 1 --2- 172.21.15.102:0/3815763586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9824005180 0x7f982400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:25.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.754+0000 7f983ca00640 1 -- 172.21.15.102:0/3815763586 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98400b1fe0 con 0x7f9824005180 2024-03-23T03:47:25.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.754+0000 7f983ca00640 1 -- 172.21.15.102:0/3815763586 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982400c020 con 0x7f98240047c0 2024-03-23T03:47:25.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.754+0000 7f983ca00640 1 -- 172.21.15.102:0/3815763586 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982400c1e0 con 0x7f9824003970 2024-03-23T03:47:25.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.755+0000 7f983e800640 1 --2- 172.21.15.102:0/3815763586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9824005180 0x7f982400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:25.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.755+0000 7f983d400640 1 --2- 172.21.15.102:0/3815763586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98240047c0 0x7f9824004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:25.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.755+0000 7f983de00640 1 --2- 172.21.15.102:0/3815763586 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9824003970 0x7f9824003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:25.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.755+0000 7f983e800640 1 -- 172.21.15.102:0/3815763586 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9824003970 msgr2=0x7f9824003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:25.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.755+0000 7f983e800640 1 --2- 172.21.15.102:0/3815763586 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9824003970 0x7f9824003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.755+0000 7f983e800640 1 -- 172.21.15.102:0/3815763586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98240047c0 msgr2=0x7f9824004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:25.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.755+0000 7f983e800640 1 --2- 172.21.15.102:0/3815763586 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98240047c0 0x7f9824004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.755+0000 7f983e800640 1 -- 172.21.15.102:0/3815763586 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9824003520 con 0x7f9824005180 2024-03-23T03:47:25.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.755+0000 7f983de00640 1 --2- 172.21.15.102:0/3815763586 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9824003970 0x7f9824003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:25.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.756+0000 7f983e800640 1 -- 172.21.15.102:0/3815763586 reap_dead start 2024-03-23T03:47:25.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.756+0000 7f983e800640 1 --2- 172.21.15.102:0/3815763586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9824005180 0x7f982400b980 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9838009fc0 tx=0x7f98240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38d53c7b8bc552e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:25.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.756+0000 7f983ca00640 1 -- 172.21.15.102:0/3815763586 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983802cc60 con 0x7f9824005180 2024-03-23T03:47:25.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.756+0000 7f983ca00640 1 -- 172.21.15.102:0/3815763586 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983800f040 con 0x7f9824005180 2024-03-23T03:47:25.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.756+0000 7f983ca00640 1 -- 172.21.15.102:0/3815763586 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9838003990 con 0x7f9824005180 2024-03-23T03:47:25.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.757+0000 7f9845000640 1 -- 172.21.15.102:0/3815763586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9824005180 msgr2=0x7f982400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:25.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.757+0000 7f9845000640 1 --2- 172.21.15.102:0/3815763586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9824005180 0x7f982400b980 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9838009fc0 tx=0x7f98240057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.757+0000 7f9845000640 1 -- 172.21.15.102:0/3815763586 shutdown_connections 2024-03-23T03:47:25.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.757+0000 7f9845000640 1 --2- 172.21.15.102:0/3815763586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9824005180 0x7f982400b980 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.757+0000 7f9845000640 1 -- 172.21.15.102:0/3815763586 >> 172.21.15.102:0/3815763586 conn(0x7f984006a3d0 msgr2=0x7f98400b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:25.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.757+0000 7f9845000640 1 -- 172.21.15.102:0/3815763586 shutdown_connections 2024-03-23T03:47:25.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.757+0000 7f9845000640 1 -- 172.21.15.102:0/3815763586 wait complete. 2024-03-23T03:47:25.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.759+0000 7f9845000640 1 Processor -- start 2024-03-23T03:47:25.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.759+0000 7f9845000640 1 -- start start 2024-03-23T03:47:25.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f9845000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9824005180 0x7f9840160630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:25.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f9845000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9840140360 0x7f98401407e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f9845000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9840143680 0x7f9840143b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f9845000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9840140d20 con 0x7f9840140360 2024-03-23T03:47:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f9845000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9840163370 con 0x7f9840143680 2024-03-23T03:47:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f9845000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9840162760 con 0x7f9824005180 2024-03-23T03:47:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f983d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9840140360 0x7f98401407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f983d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9840140360 0x7f98401407e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33244/0 (socket says 172.21.15.102:33244) 2024-03-23T03:47:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f983d400640 1 -- 172.21.15.102:0/565973957 learned_addr learned my addr 172.21.15.102:0/565973957 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f983de00640 1 --2- 172.21.15.102:0/565973957 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9824005180 0x7f9840160630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f983e800640 1 --2- 172.21.15.102:0/565973957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9840143680 0x7f9840143b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f983d400640 1 -- 172.21.15.102:0/565973957 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9824005180 msgr2=0x7f9840160630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f983d400640 1 --2- 172.21.15.102:0/565973957 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9824005180 0x7f9840160630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f983d400640 1 -- 172.21.15.102:0/565973957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9840143680 msgr2=0x7f9840143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f983d400640 1 --2- 172.21.15.102:0/565973957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9840143680 0x7f9840143b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f983d400640 1 -- 172.21.15.102:0/565973957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9840162a40 con 0x7f9840140360 2024-03-23T03:47:25.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f983e800640 1 --2- 172.21.15.102:0/565973957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9840143680 0x7f9840143b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:25.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.760+0000 7f983de00640 1 --2- 172.21.15.102:0/565973957 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9824005180 0x7f9840160630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:25.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.761+0000 7f983d400640 1 --2- 172.21.15.102:0/565973957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9840140360 0x7f98401407e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f982c001050 tx=0x7f982c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:25.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.761+0000 7f9836000640 1 -- 172.21.15.102:0/565973957 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c00ee60 con 0x7f9840140360 2024-03-23T03:47:25.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.761+0000 7f9836000640 1 -- 172.21.15.102:0/565973957 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982c00c8a0 con 0x7f9840140360 2024-03-23T03:47:25.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.761+0000 7f9845000640 1 -- 172.21.15.102:0/565973957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9840162d20 con 0x7f9840140360 2024-03-23T03:47:25.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.761+0000 7f9836000640 1 -- 172.21.15.102:0/565973957 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982c00cc50 con 0x7f9840140360 2024-03-23T03:47:25.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.761+0000 7f9845000640 1 -- 172.21.15.102:0/565973957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9840160b70 con 0x7f9840140360 2024-03-23T03:47:25.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.762+0000 7f9845000640 1 -- 172.21.15.102:0/565973957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f984006ac20 con 0x7f9840140360 2024-03-23T03:47:25.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.763+0000 7f9836000640 1 -- 172.21.15.102:0/565973957 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f982c01d460 con 0x7f9840140360 2024-03-23T03:47:25.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.764+0000 7f9836000640 1 --2- 172.21.15.102:0/565973957 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f982006ac00 0x7f982006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:25.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.764+0000 7f9836000640 1 -- 172.21.15.102:0/565973957 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f982c089610 con 0x7f9840140360 2024-03-23T03:47:25.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.764+0000 7f983de00640 1 --2- 172.21.15.102:0/565973957 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f982006ac00 0x7f982006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:25.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.765+0000 7f983de00640 1 --2- 172.21.15.102:0/565973957 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f982006ac00 0x7f982006d0c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9830007fd0 tx=0x7f9830009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:25.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.769+0000 7f9836000640 1 -- 172.21.15.102:0/565973957 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f982c054500 con 0x7f9840140360 2024-03-23T03:47:25.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.955+0000 7f9845000640 1 -- 172.21.15.102:0/565973957 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9840073390 con 0x7f9840140360 2024-03-23T03:47:25.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.959+0000 7f9836000640 1 -- 172.21.15.102:0/565973957 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f982c00cdb0 con 0x7f9840140360 2024-03-23T03:47:25.960 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:25.960 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:58.410551+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:47:25.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.962+0000 7f9845000640 1 -- 172.21.15.102:0/565973957 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f982006ac00 msgr2=0x7f982006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:25.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.962+0000 7f9845000640 1 --2- 172.21.15.102:0/565973957 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f982006ac00 0x7f982006d0c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9830007fd0 tx=0x7f9830009040 comp rx=0 tx=0).stop 2024-03-23T03:47:25.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.962+0000 7f9845000640 1 -- 172.21.15.102:0/565973957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9840140360 msgr2=0x7f98401407e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:25.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.962+0000 7f9845000640 1 --2- 172.21.15.102:0/565973957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9840140360 0x7f98401407e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f982c001050 tx=0x7f982c00ec10 comp rx=0 tx=0).stop 2024-03-23T03:47:25.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.962+0000 7f9845000640 1 -- 172.21.15.102:0/565973957 shutdown_connections 2024-03-23T03:47:25.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.962+0000 7f9845000640 1 --2- 172.21.15.102:0/565973957 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f982006ac00 0x7f982006d0c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.962+0000 7f9845000640 1 --2- 172.21.15.102:0/565973957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9840143680 0x7f9840143b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.962+0000 7f9845000640 1 --2- 172.21.15.102:0/565973957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9840140360 0x7f98401407e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.962+0000 7f9845000640 1 --2- 172.21.15.102:0/565973957 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9824005180 0x7f9840160630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:25.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.962+0000 7f9845000640 1 -- 172.21.15.102:0/565973957 >> 172.21.15.102:0/565973957 conn(0x7f984006a3d0 msgr2=0x7f98401498e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:25.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.962+0000 7f9845000640 1 -- 172.21.15.102:0/565973957 shutdown_connections 2024-03-23T03:47:25.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:25.962+0000 7f9845000640 1 -- 172.21.15.102:0/565973957 wait complete. 2024-03-23T03:47:25.981 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:46:52.158650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:46:52.914275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-23T03:46:55.983754+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-23T03:47:25.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:47:26.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.184+0000 7f04afe00640 1 Processor -- start 2024-03-23T03:47:26.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.184+0000 7f04afe00640 1 -- start start 2024-03-23T03:47:26.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.185+0000 7f04afe00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f04b0159cd0 0x7f04b00ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.185+0000 7f04afe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f04b00ad7a0 0x7f04b015a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.185+0000 7f04afe00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b00c8c90 con 0x7f04b00cc200 2024-03-23T03:47:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.185+0000 7f04afe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b00cc6c0 con 0x7f04b00ad7a0 2024-03-23T03:47:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.185+0000 7f04afe00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b00cc9a0 con 0x7f04b0159cd0 2024-03-23T03:47:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.186+0000 7f04af400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f04b00ad7a0 0x7f04b015a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.186+0000 7f04af400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f04b00ad7a0 0x7f04b015a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33252/0 (socket says 172.21.15.102:33252) 2024-03-23T03:47:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.186+0000 7f04af400640 1 -- 172.21.15.102:0/2288081343 learned_addr learned my addr 172.21.15.102:0/2288081343 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.186+0000 7f04aea00640 1 --2- 172.21.15.102:0/2288081343 >> v2:172.21.15.173:3300/0 conn(0x7f04b0159cd0 0x7f04b00ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.186+0000 7f04af400640 1 -- 172.21.15.102:0/2288081343 >> v2:172.21.15.173:3300/0 conn(0x7f04b0159cd0 msgr2=0x7f04b00ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.186+0000 7f04af400640 1 --2- 172.21.15.102:0/2288081343 >> v2:172.21.15.173:3300/0 conn(0x7f04b0159cd0 0x7f04b00ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.186+0000 7f04af400640 1 -- 172.21.15.102:0/2288081343 >> v1:172.21.15.173:6789/0 conn(0x7f04b00cc200 legacy=0x7f04b00cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:26.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.186+0000 7f04af400640 1 -- 172.21.15.102:0/2288081343 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b00ccc80 con 0x7f04b00ad7a0 2024-03-23T03:47:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.187+0000 7f04af400640 1 --2- 172.21.15.102:0/2288081343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04b00ad7a0 0x7f04b015a0b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f04b006e4f0 tx=0x7f04a802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b414c6c2b03dd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.187+0000 7f04ae000640 1 -- 172.21.15.102:0/2288081343 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a802c9f0 con 0x7f04b00ad7a0 2024-03-23T03:47:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.188+0000 7f04ae000640 1 -- 172.21.15.102:0/2288081343 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a802cb50 con 0x7f04b00ad7a0 2024-03-23T03:47:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.188+0000 7f04ae000640 1 -- 172.21.15.102:0/2288081343 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a80343f0 con 0x7f04b00ad7a0 2024-03-23T03:47:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.188+0000 7f04afe00640 1 -- 172.21.15.102:0/2288081343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04b00ad7a0 msgr2=0x7f04b015a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.188+0000 7f04afe00640 1 --2- 172.21.15.102:0/2288081343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04b00ad7a0 0x7f04b015a0b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f04b006e4f0 tx=0x7f04a802be50 comp rx=0 tx=0).stop 2024-03-23T03:47:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.188+0000 7f04afe00640 1 -- 172.21.15.102:0/2288081343 shutdown_connections 2024-03-23T03:47:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.188+0000 7f04afe00640 1 --2- 172.21.15.102:0/2288081343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04b00ad7a0 0x7f04b015a0b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.188+0000 7f04afe00640 1 --2- 172.21.15.102:0/2288081343 >> v2:172.21.15.173:3300/0 conn(0x7f04b0159cd0 0x7f04b00ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.188+0000 7f04afe00640 1 -- 172.21.15.102:0/2288081343 >> 172.21.15.102:0/2288081343 conn(0x7f04b00b01b0 msgr2=0x7f04b00b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.188+0000 7f04afe00640 1 -- 172.21.15.102:0/2288081343 shutdown_connections 2024-03-23T03:47:26.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.188+0000 7f04afe00640 1 -- 172.21.15.102:0/2288081343 wait complete. 2024-03-23T03:47:26.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.190+0000 7f04afe00640 1 Processor -- start 2024-03-23T03:47:26.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.190+0000 7f04afe00640 1 -- start start 2024-03-23T03:47:26.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.191+0000 7f04afe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f04b00ad7a0 0x7f04b01f1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:26.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.191+0000 7f04afe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04b00cc200 0x7f04b0070e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:26.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.191+0000 7f04afe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04b0159cd0 0x7f04b00713d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:26.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.191+0000 7f04afe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b01f2280 con 0x7f04b00cc200 2024-03-23T03:47:26.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.191+0000 7f04afe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b009ad70 con 0x7f04b0159cd0 2024-03-23T03:47:26.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.191+0000 7f04afe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b009afa0 con 0x7f04b00ad7a0 2024-03-23T03:47:26.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.191+0000 7f04a7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04b00cc200 0x7f04b0070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:26.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.191+0000 7f04a7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04b00cc200 0x7f04b0070e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33262/0 (socket says 172.21.15.102:33262) 2024-03-23T03:47:26.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.191+0000 7f04a7e00640 1 -- 172.21.15.102:0/3977842806 learned_addr learned my addr 172.21.15.102:0/3977842806 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:26.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.192+0000 7f04af400640 1 --2- 172.21.15.102:0/3977842806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04b0159cd0 0x7f04b00713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:26.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.192+0000 7f04aea00640 1 --2- 172.21.15.102:0/3977842806 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f04b00ad7a0 0x7f04b01f1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:26.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.192+0000 7f04a7e00640 1 -- 172.21.15.102:0/3977842806 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f04b00ad7a0 msgr2=0x7f04b01f1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:26.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.192+0000 7f04a7e00640 1 --2- 172.21.15.102:0/3977842806 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f04b00ad7a0 0x7f04b01f1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:26.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.192+0000 7f04a7e00640 1 -- 172.21.15.102:0/3977842806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04b0159cd0 msgr2=0x7f04b00713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:26.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.192+0000 7f04a7e00640 1 --2- 172.21.15.102:0/3977842806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04b0159cd0 0x7f04b00713d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:26.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.192+0000 7f04a7e00640 1 -- 172.21.15.102:0/3977842806 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a8009b30 con 0x7f04b00cc200 2024-03-23T03:47:26.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.192+0000 7f04af400640 1 --2- 172.21.15.102:0/3977842806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04b0159cd0 0x7f04b00713d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:26.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.192+0000 7f04a7e00640 1 --2- 172.21.15.102:0/3977842806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04b00cc200 0x7f04b0070e90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f049800c410 tx=0x7f049800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:26.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.192+0000 7f04a6a00640 1 -- 172.21.15.102:0/3977842806 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04980035f0 con 0x7f04b00cc200 2024-03-23T03:47:26.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.192+0000 7f04a6a00640 1 -- 172.21.15.102:0/3977842806 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0498003750 con 0x7f04b00cc200 2024-03-23T03:47:26.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.192+0000 7f04afe00640 1 -- 172.21.15.102:0/3977842806 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b009b280 con 0x7f04b00cc200 2024-03-23T03:47:26.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.192+0000 7f04a6a00640 1 -- 172.21.15.102:0/3977842806 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0498003b00 con 0x7f04b00cc200 2024-03-23T03:47:26.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.193+0000 7f04afe00640 1 -- 172.21.15.102:0/3977842806 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b009b840 con 0x7f04b00cc200 2024-03-23T03:47:26.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.194+0000 7f04afe00640 1 -- 172.21.15.102:0/3977842806 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b004e030 con 0x7f04b00cc200 2024-03-23T03:47:26.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.195+0000 7f04a6a00640 1 -- 172.21.15.102:0/3977842806 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0498003c60 con 0x7f04b00cc200 2024-03-23T03:47:26.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.196+0000 7f04a6a00640 1 --2- 172.21.15.102:0/3977842806 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f048c06ac00 0x7f048c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:26.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.196+0000 7f04a6a00640 1 -- 172.21.15.102:0/3977842806 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f04980888f0 con 0x7f04b00cc200 2024-03-23T03:47:26.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.196+0000 7f04aea00640 1 --2- 172.21.15.102:0/3977842806 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f048c06ac00 0x7f048c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:26.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.197+0000 7f04aea00640 1 --2- 172.21.15.102:0/3977842806 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f048c06ac00 0x7f048c06d0c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f04a0006fd0 tx=0x7f04a0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:26.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.200+0000 7f04a6a00640 1 -- 172.21.15.102:0/3977842806 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0498053740 con 0x7f04b00cc200 2024-03-23T03:47:26.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.393+0000 7f04afe00640 1 -- 172.21.15.102:0/3977842806 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f04b006e290 con 0x7f04b00cc200 2024-03-23T03:47:26.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.394+0000 7f04a6a00640 1 -- 172.21.15.102:0/3977842806 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v26) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0498056d60 con 0x7f04b00cc200 2024-03-23T03:47:26.395 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:47:26.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.398+0000 7f04afe00640 1 -- 172.21.15.102:0/3977842806 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f048c06ac00 msgr2=0x7f048c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:26.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.398+0000 7f04afe00640 1 --2- 172.21.15.102:0/3977842806 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f048c06ac00 0x7f048c06d0c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f04a0006fd0 tx=0x7f04a0008040 comp rx=0 tx=0).stop 2024-03-23T03:47:26.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.398+0000 7f04afe00640 1 -- 172.21.15.102:0/3977842806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04b00cc200 msgr2=0x7f04b0070e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:26.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.398+0000 7f04afe00640 1 --2- 172.21.15.102:0/3977842806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04b00cc200 0x7f04b0070e90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f049800c410 tx=0x7f049800c8d0 comp rx=0 tx=0).stop 2024-03-23T03:47:26.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.398+0000 7f04afe00640 1 -- 172.21.15.102:0/3977842806 shutdown_connections 2024-03-23T03:47:26.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.398+0000 7f04afe00640 1 --2- 172.21.15.102:0/3977842806 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f048c06ac00 0x7f048c06d0c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:26.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.398+0000 7f04afe00640 1 --2- 172.21.15.102:0/3977842806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04b0159cd0 0x7f04b00713d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:26.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.398+0000 7f04afe00640 1 --2- 172.21.15.102:0/3977842806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04b00cc200 0x7f04b0070e90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:26.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.398+0000 7f04afe00640 1 --2- 172.21.15.102:0/3977842806 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f04b00ad7a0 0x7f04b01f1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:26.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.398+0000 7f04afe00640 1 -- 172.21.15.102:0/3977842806 >> 172.21.15.102:0/3977842806 conn(0x7f04b00b01b0 msgr2=0x7f04b0069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:26.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.398+0000 7f04afe00640 1 -- 172.21.15.102:0/3977842806 shutdown_connections 2024-03-23T03:47:26.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.398+0000 7f04afe00640 1 -- 172.21.15.102:0/3977842806 wait complete. 2024-03-23T03:47:26.419 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:47:26.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.632+0000 7f2983000640 1 Processor -- start 2024-03-23T03:47:26.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.633+0000 7f2983000640 1 -- start start 2024-03-23T03:47:26.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.634+0000 7f2983000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f297c0ce930 0x7f297c0add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:26.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.634+0000 7f2983000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f297c0cdba0 con 0x7f297c0cd3e0 2024-03-23T03:47:26.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.634+0000 7f2983000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f297c0cde80 con 0x7f297c06ce90 2024-03-23T03:47:26.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.634+0000 7f2983000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f297c06c760 con 0x7f297c0ce930 2024-03-23T03:47:26.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.635+0000 7f2973400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f297c0cd3e0 0x7f297c0ced10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52806/0 (socket says 172.21.15.102:52806) 2024-03-23T03:47:26.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.635+0000 7f2973400640 1 -- 172.21.15.102:0/2865370632 learned_addr learned my addr 172.21.15.102:0/2865370632 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:26.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.635+0000 7f297be00640 1 --2- 172.21.15.102:0/2865370632 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f297c0ce930 0x7f297c0add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:26.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.635+0000 7f297b400640 1 -- 172.21.15.102:0/2865370632 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929349716 0 0) 0x7f297c0cdba0 con 0x7f297c0cd3e0 2024-03-23T03:47:26.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.635+0000 7f297b400640 1 -- 172.21.15.102:0/2865370632 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2960003520 con 0x7f297c0cd3e0 2024-03-23T03:47:26.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.636+0000 7f297b400640 1 -- 172.21.15.102:0/2865370632 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460891639 0 0) 0x7f297c0cde80 con 0x7f297c06ce90 2024-03-23T03:47:26.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.636+0000 7f297b400640 1 -- 172.21.15.102:0/2865370632 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f297c0cdba0 con 0x7f297c06ce90 2024-03-23T03:47:26.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.636+0000 7f297be00640 1 -- 172.21.15.102:0/2865370632 >> v1:172.21.15.173:6789/0 conn(0x7f297c06ce90 legacy=0x7f297c15e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:26.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.636+0000 7f297be00640 1 -- 172.21.15.102:0/2865370632 >> v1:172.21.15.102:6789/0 conn(0x7f297c0cd3e0 legacy=0x7f297c0ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:26.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.636+0000 7f297be00640 1 -- 172.21.15.102:0/2865370632 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297c0ce160 con 0x7f297c0ce930 2024-03-23T03:47:26.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.637+0000 7f297be00640 1 --2- 172.21.15.102:0/2865370632 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f297c0ce930 0x7f297c0add30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f29640097d0 tx=0x7f296402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e3ed22c23b64983 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:26.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.637+0000 7f297b400640 1 -- 172.21.15.102:0/2865370632 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2964031890 con 0x7f297c0ce930 2024-03-23T03:47:26.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.638+0000 7f297b400640 1 -- 172.21.15.102:0/2865370632 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29640319f0 con 0x7f297c0ce930 2024-03-23T03:47:26.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.638+0000 7f297b400640 1 -- 172.21.15.102:0/2865370632 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2964031d60 con 0x7f297c0ce930 2024-03-23T03:47:26.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.638+0000 7f2983000640 1 -- 172.21.15.102:0/2865370632 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f297c0ce930 msgr2=0x7f297c0add30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:26.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.638+0000 7f2983000640 1 --2- 172.21.15.102:0/2865370632 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f297c0ce930 0x7f297c0add30 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f29640097d0 tx=0x7f296402b930 comp rx=0 tx=0).stop 2024-03-23T03:47:26.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.638+0000 7f2983000640 1 -- 172.21.15.102:0/2865370632 shutdown_connections 2024-03-23T03:47:26.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.638+0000 7f2983000640 1 --2- 172.21.15.102:0/2865370632 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f297c0ce930 0x7f297c0add30 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:26.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.638+0000 7f2983000640 1 -- 172.21.15.102:0/2865370632 >> 172.21.15.102:0/2865370632 conn(0x7f297c06a0b0 msgr2=0x7f297c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:26.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.638+0000 7f2983000640 1 -- 172.21.15.102:0/2865370632 shutdown_connections 2024-03-23T03:47:26.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.638+0000 7f2983000640 1 -- 172.21.15.102:0/2865370632 wait complete. 2024-03-23T03:47:26.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.640+0000 7f2983000640 1 Processor -- start 2024-03-23T03:47:26.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.640+0000 7f2983000640 1 -- start start 2024-03-23T03:47:26.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.641+0000 7f2983000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f297c06ce90 0x7f297c152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:26.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.641+0000 7f2983000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f297c0cd3e0 0x7f297c14d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:26.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.641+0000 7f2983000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f297c0ce930 0x7f297c06d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:26.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.641+0000 7f2983000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297c153ed0 con 0x7f297c0ce930 2024-03-23T03:47:26.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.641+0000 7f2983000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f297c1551e0 con 0x7f297c0cd3e0 2024-03-23T03:47:26.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.641+0000 7f2983000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f297c153bf0 con 0x7f297c06ce90 2024-03-23T03:47:26.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.642+0000 7f2981000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f297c0ce930 0x7f297c06d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:26.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.642+0000 7f2981000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f297c0ce930 0x7f297c06d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33272/0 (socket says 172.21.15.102:33272) 2024-03-23T03:47:26.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.642+0000 7f2981000640 1 -- 172.21.15.102:0/2719715367 learned_addr learned my addr 172.21.15.102:0/2719715367 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:26.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.642+0000 7f297be00640 1 --2- 172.21.15.102:0/2719715367 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f297c06ce90 0x7f297c152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:26.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.642+0000 7f2973400640 1 --2- 172.21.15.102:0/2719715367 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f297c0cd3e0 0x7f297c14d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:26.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.642+0000 7f2981000640 1 -- 172.21.15.102:0/2719715367 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f297c06ce90 msgr2=0x7f297c152540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:26.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.642+0000 7f2981000640 1 --2- 172.21.15.102:0/2719715367 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f297c06ce90 0x7f297c152540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:26.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.642+0000 7f2981000640 1 -- 172.21.15.102:0/2719715367 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f297c0cd3e0 msgr2=0x7f297c14d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:26.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.642+0000 7f2981000640 1 --2- 172.21.15.102:0/2719715367 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f297c0cd3e0 0x7f297c14d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:26.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.642+0000 7f2981000640 1 -- 172.21.15.102:0/2719715367 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2964009400 con 0x7f297c0ce930 2024-03-23T03:47:26.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.642+0000 7f2981000640 1 --2- 172.21.15.102:0/2719715367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f297c0ce930 0x7f297c06d320 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f297400c790 tx=0x7f297400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:26.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.643+0000 7f2978c00640 1 -- 172.21.15.102:0/2719715367 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297400cea0 con 0x7f297c0ce930 2024-03-23T03:47:26.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.643+0000 7f2978c00640 1 -- 172.21.15.102:0/2719715367 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2974003b70 con 0x7f297c0ce930 2024-03-23T03:47:26.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.643+0000 7f2978c00640 1 -- 172.21.15.102:0/2719715367 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297400a850 con 0x7f297c0ce930 2024-03-23T03:47:26.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.643+0000 7f2983000640 1 -- 172.21.15.102:0/2719715367 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f297c155480 con 0x7f297c0ce930 2024-03-23T03:47:26.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.643+0000 7f2983000640 1 -- 172.21.15.102:0/2719715367 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f297c152a80 con 0x7f297c0ce930 2024-03-23T03:47:26.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.644+0000 7f2983000640 1 -- 172.21.15.102:0/2719715367 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f297c155040 con 0x7f297c0ce930 2024-03-23T03:47:26.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.645+0000 7f2978c00640 1 -- 172.21.15.102:0/2719715367 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f297400a9b0 con 0x7f297c0ce930 2024-03-23T03:47:26.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.646+0000 7f2978c00640 1 --2- 172.21.15.102:0/2719715367 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f295806ac00 0x7f295806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:26.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.646+0000 7f2978c00640 1 -- 172.21.15.102:0/2719715367 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2974089620 con 0x7f297c0ce930 2024-03-23T03:47:26.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.646+0000 7f297be00640 1 --2- 172.21.15.102:0/2719715367 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f295806ac00 0x7f295806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:26.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.647+0000 7f297be00640 1 --2- 172.21.15.102:0/2719715367 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f295806ac00 0x7f295806d0c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2964036000 tx=0x7f2964008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:26.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.650+0000 7f2978c00640 1 -- 172.21.15.102:0/2719715367 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f297c155040 con 0x7f297c0ce930 2024-03-23T03:47:26.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.844+0000 7f2983000640 1 -- 172.21.15.102:0/2719715367 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f297c1559a0 con 0x7f297c0ce930 2024-03-23T03:47:26.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.845+0000 7f2978c00640 1 -- 172.21.15.102:0/2719715367 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v26) v1 ==== 104+0+11 (secure 0 0 0) 0x7f297c1559a0 con 0x7f297c0ce930 2024-03-23T03:47:26.846 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.848+0000 7f2983000640 1 -- 172.21.15.102:0/2719715367 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f295806ac00 msgr2=0x7f295806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.848+0000 7f2983000640 1 --2- 172.21.15.102:0/2719715367 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f295806ac00 0x7f295806d0c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f2964036000 tx=0x7f2964008030 comp rx=0 tx=0).stop 2024-03-23T03:47:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.848+0000 7f2983000640 1 -- 172.21.15.102:0/2719715367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f297c0ce930 msgr2=0x7f297c06d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.848+0000 7f2983000640 1 --2- 172.21.15.102:0/2719715367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f297c0ce930 0x7f297c06d320 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f297400c790 tx=0x7f297400cc50 comp rx=0 tx=0).stop 2024-03-23T03:47:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.848+0000 7f2983000640 1 -- 172.21.15.102:0/2719715367 shutdown_connections 2024-03-23T03:47:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.848+0000 7f2983000640 1 --2- 172.21.15.102:0/2719715367 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f295806ac00 0x7f295806d0c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.848+0000 7f2983000640 1 --2- 172.21.15.102:0/2719715367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f297c0ce930 0x7f297c06d320 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.848+0000 7f2983000640 1 --2- 172.21.15.102:0/2719715367 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f297c0cd3e0 0x7f297c14d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.848+0000 7f2983000640 1 --2- 172.21.15.102:0/2719715367 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f297c06ce90 0x7f297c152540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.848+0000 7f2983000640 1 -- 172.21.15.102:0/2719715367 >> 172.21.15.102:0/2719715367 conn(0x7f297c06a0b0 msgr2=0x7f297c06a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.848+0000 7f2983000640 1 -- 172.21.15.102:0/2719715367 shutdown_connections 2024-03-23T03:47:26.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:26.848+0000 7f2983000640 1 -- 172.21.15.102:0/2719715367 wait complete. 2024-03-23T03:47:26.867 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:47:27.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.068+0000 7f6755c00640 1 Processor -- start 2024-03-23T03:47:27.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.069+0000 7f6755c00640 1 -- start start 2024-03-23T03:47:27.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.070+0000 7f6755c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6750098a40 0x7f6750098e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.070+0000 7f6755c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6750160d90 0x7f675015c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.070+0000 7f6755c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67500d0280 con 0x7f675006b2e0 2024-03-23T03:47:27.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.070+0000 7f6755c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6750099360 con 0x7f6750160d90 2024-03-23T03:47:27.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.070+0000 7f6755c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f675015d9e0 con 0x7f6750098a40 2024-03-23T03:47:27.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.070+0000 7f674f400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6750160d90 0x7f675015c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.070+0000 7f674f400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6750160d90 0x7f675015c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33282/0 (socket says 172.21.15.102:33282) 2024-03-23T03:47:27.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.070+0000 7f674f400640 1 -- 172.21.15.102:0/2559924913 learned_addr learned my addr 172.21.15.102:0/2559924913 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:27.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.071+0000 7f674ea00640 1 --2- 172.21.15.102:0/2559924913 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6750098a40 0x7f6750098e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.071+0000 7f674f400640 1 -- 172.21.15.102:0/2559924913 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6750098a40 msgr2=0x7f6750098e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:27.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.071+0000 7f674f400640 1 --2- 172.21.15.102:0/2559924913 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6750098a40 0x7f6750098e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.071+0000 7f674f400640 1 -- 172.21.15.102:0/2559924913 >> v1:172.21.15.173:6789/0 conn(0x7f675006b2e0 legacy=0x7f6750160680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:27.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.071+0000 7f674f400640 1 -- 172.21.15.102:0/2559924913 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675015dcc0 con 0x7f6750160d90 2024-03-23T03:47:27.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.071+0000 7f674ea00640 1 --2- 172.21.15.102:0/2559924913 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6750098a40 0x7f6750098e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:27.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.072+0000 7f674f400640 1 --2- 172.21.15.102:0/2559924913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6750160d90 0x7f675015c300 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6748009870 tx=0x7f674802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=617a7e1cdbf2b5f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:27.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.072+0000 7f674d600640 1 -- 172.21.15.102:0/2559924913 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6748031640 con 0x7f6750160d90 2024-03-23T03:47:27.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.072+0000 7f674d600640 1 -- 172.21.15.102:0/2559924913 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67480317a0 con 0x7f6750160d90 2024-03-23T03:47:27.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.072+0000 7f674d600640 1 -- 172.21.15.102:0/2559924913 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6748031b30 con 0x7f6750160d90 2024-03-23T03:47:27.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.073+0000 7f6755c00640 1 -- 172.21.15.102:0/2559924913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6750160d90 msgr2=0x7f675015c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:27.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.073+0000 7f6755c00640 1 --2- 172.21.15.102:0/2559924913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6750160d90 0x7f675015c300 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6748009870 tx=0x7f674802b8f0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.073+0000 7f6755c00640 1 -- 172.21.15.102:0/2559924913 shutdown_connections 2024-03-23T03:47:27.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.073+0000 7f6755c00640 1 --2- 172.21.15.102:0/2559924913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6750160d90 0x7f675015c300 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.073+0000 7f6755c00640 1 --2- 172.21.15.102:0/2559924913 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6750098a40 0x7f6750098e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.073+0000 7f6755c00640 1 -- 172.21.15.102:0/2559924913 >> 172.21.15.102:0/2559924913 conn(0x7f67500b2180 msgr2=0x7f67500b2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:27.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.073+0000 7f6755c00640 1 -- 172.21.15.102:0/2559924913 shutdown_connections 2024-03-23T03:47:27.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.073+0000 7f6755c00640 1 -- 172.21.15.102:0/2559924913 wait complete. 2024-03-23T03:47:27.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.075+0000 7f6755c00640 1 Processor -- start 2024-03-23T03:47:27.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.075+0000 7f6755c00640 1 -- start start 2024-03-23T03:47:27.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.076+0000 7f6755c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f675006b2e0 0x7f675009d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.076+0000 7f6755c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6750098a40 0x7f67500c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.076+0000 7f6755c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6750160d90 0x7f67500c63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.076+0000 7f6755c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f675009a930 con 0x7f6750160d90 2024-03-23T03:47:27.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.076+0000 7f6755c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67500c6bd0 con 0x7f6750098a40 2024-03-23T03:47:27.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.076+0000 7f6755c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f675009abf0 con 0x7f675006b2e0 2024-03-23T03:47:27.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.076+0000 7f674f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6750160d90 0x7f67500c63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.076+0000 7f674f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6750160d90 0x7f67500c63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33286/0 (socket says 172.21.15.102:33286) 2024-03-23T03:47:27.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.076+0000 7f674f400640 1 -- 172.21.15.102:0/2943768136 learned_addr learned my addr 172.21.15.102:0/2943768136 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:27.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.077+0000 7f674ea00640 1 --2- 172.21.15.102:0/2943768136 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f675006b2e0 0x7f675009d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.077+0000 7f674e000640 1 --2- 172.21.15.102:0/2943768136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6750098a40 0x7f67500c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.077+0000 7f674f400640 1 -- 172.21.15.102:0/2943768136 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f675006b2e0 msgr2=0x7f675009d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:27.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.077+0000 7f674f400640 1 --2- 172.21.15.102:0/2943768136 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f675006b2e0 0x7f675009d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.077+0000 7f674f400640 1 -- 172.21.15.102:0/2943768136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6750098a40 msgr2=0x7f67500c5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:27.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.077+0000 7f674f400640 1 --2- 172.21.15.102:0/2943768136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6750098a40 0x7f67500c5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.077+0000 7f674f400640 1 -- 172.21.15.102:0/2943768136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67480094d0 con 0x7f6750160d90 2024-03-23T03:47:27.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.077+0000 7f674e000640 1 --2- 172.21.15.102:0/2943768136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6750098a40 0x7f67500c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:27.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.077+0000 7f674f400640 1 --2- 172.21.15.102:0/2943768136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6750160d90 0x7f67500c63b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f674802b930 tx=0x7f67480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:27.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.077+0000 7f6746a00640 1 -- 172.21.15.102:0/2943768136 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674803ac60 con 0x7f6750160d90 2024-03-23T03:47:27.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.078+0000 7f6746a00640 1 -- 172.21.15.102:0/2943768136 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f674803ade0 con 0x7f6750160d90 2024-03-23T03:47:27.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.078+0000 7f6746a00640 1 -- 172.21.15.102:0/2943768136 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6748044630 con 0x7f6750160d90 2024-03-23T03:47:27.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.078+0000 7f6755c00640 1 -- 172.21.15.102:0/2943768136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f675009aed0 con 0x7f6750160d90 2024-03-23T03:47:27.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.078+0000 7f6755c00640 1 -- 172.21.15.102:0/2943768136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67500c68f0 con 0x7f6750160d90 2024-03-23T03:47:27.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.079+0000 7f6755c00640 1 -- 172.21.15.102:0/2943768136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675009b3f0 con 0x7f6750160d90 2024-03-23T03:47:27.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.080+0000 7f6746a00640 1 -- 172.21.15.102:0/2943768136 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f674802f050 con 0x7f6750160d90 2024-03-23T03:47:27.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.081+0000 7f6746a00640 1 --2- 172.21.15.102:0/2943768136 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f673006ac00 0x7f673006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.081+0000 7f6746a00640 1 -- 172.21.15.102:0/2943768136 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f67480ab850 con 0x7f6750160d90 2024-03-23T03:47:27.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.081+0000 7f674ea00640 1 --2- 172.21.15.102:0/2943768136 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f673006ac00 0x7f673006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.082+0000 7f674ea00640 1 --2- 172.21.15.102:0/2943768136 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f673006ac00 0x7f673006d0c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f67400075b0 tx=0x7f6740007540 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:27.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.085+0000 7f6746a00640 1 -- 172.21.15.102:0/2943768136 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f675009b3f0 con 0x7f6750160d90 2024-03-23T03:47:27.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.282+0000 7f6755c00640 1 -- 172.21.15.102:0/2943768136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f675009b3f0 con 0x7f6750160d90 2024-03-23T03:47:27.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.282+0000 7f6746a00640 1 -- 172.21.15.102:0/2943768136 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v26) v1 ==== 100+0+11 (secure 0 0 0) 0x7f675009b3f0 con 0x7f6750160d90 2024-03-23T03:47:27.283 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:47:27.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.285+0000 7f6755c00640 1 -- 172.21.15.102:0/2943768136 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f673006ac00 msgr2=0x7f673006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:27.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.285+0000 7f6755c00640 1 --2- 172.21.15.102:0/2943768136 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f673006ac00 0x7f673006d0c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f67400075b0 tx=0x7f6740007540 comp rx=0 tx=0).stop 2024-03-23T03:47:27.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.285+0000 7f6755c00640 1 -- 172.21.15.102:0/2943768136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6750160d90 msgr2=0x7f67500c63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:27.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.285+0000 7f6755c00640 1 --2- 172.21.15.102:0/2943768136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6750160d90 0x7f67500c63b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f674802b930 tx=0x7f67480046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.285+0000 7f6755c00640 1 -- 172.21.15.102:0/2943768136 shutdown_connections 2024-03-23T03:47:27.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.285+0000 7f6755c00640 1 --2- 172.21.15.102:0/2943768136 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f673006ac00 0x7f673006d0c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.285+0000 7f6755c00640 1 --2- 172.21.15.102:0/2943768136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6750160d90 0x7f67500c63b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.285+0000 7f6755c00640 1 --2- 172.21.15.102:0/2943768136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6750098a40 0x7f67500c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.285+0000 7f6755c00640 1 --2- 172.21.15.102:0/2943768136 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f675006b2e0 0x7f675009d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.285+0000 7f6755c00640 1 -- 172.21.15.102:0/2943768136 >> 172.21.15.102:0/2943768136 conn(0x7f67500b2180 msgr2=0x7f675006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:27.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.285+0000 7f6755c00640 1 -- 172.21.15.102:0/2943768136 shutdown_connections 2024-03-23T03:47:27.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.285+0000 7f6755c00640 1 -- 172.21.15.102:0/2943768136 wait complete. 2024-03-23T03:47:27.304 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-23T03:47:27.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.508+0000 7f2fdc000640 1 Processor -- start 2024-03-23T03:47:27.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.508+0000 7f2fdc000640 1 -- start start 2024-03-23T03:47:27.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.509+0000 7f2fdc000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2fd40add30 0x7f2fd40ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.509+0000 7f2fdc000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fd406ce90 0x7f2fd406d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.509+0000 7f2fdc000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fd40cf250 con 0x7f2fd40ce930 2024-03-23T03:47:27.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.509+0000 7f2fdc000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fd40ae1c0 con 0x7f2fd40add30 2024-03-23T03:47:27.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.509+0000 7f2fdc000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd406c760 con 0x7f2fd406ce90 2024-03-23T03:47:27.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.510+0000 7f2fd8c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2fd40add30 0x7f2fd40ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.510+0000 7f2fd8c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2fd40add30 0x7f2fd40ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33302/0 (socket says 172.21.15.102:33302) 2024-03-23T03:47:27.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.510+0000 7f2fd8c00640 1 -- 172.21.15.102:0/1246552527 learned_addr learned my addr 172.21.15.102:0/1246552527 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:27.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.510+0000 7f2fda000640 1 --2- 172.21.15.102:0/1246552527 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fd406ce90 0x7f2fd406d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.510+0000 7f2fd8c00640 1 -- 172.21.15.102:0/1246552527 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fd406ce90 msgr2=0x7f2fd406d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:27.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.510+0000 7f2fd8c00640 1 --2- 172.21.15.102:0/1246552527 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fd406ce90 0x7f2fd406d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.510+0000 7f2fd8c00640 1 -- 172.21.15.102:0/1246552527 >> v1:172.21.15.173:6789/0 conn(0x7f2fd40ce930 legacy=0x7f2fd40cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:27.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.510+0000 7f2fd8c00640 1 -- 172.21.15.102:0/1246552527 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd40cdba0 con 0x7f2fd40add30 2024-03-23T03:47:27.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.512+0000 7f2fd8c00640 1 --2- 172.21.15.102:0/1246552527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd40add30 0x7f2fd40ced10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4009870 tx=0x7f2fc402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227fab72cda3cd48 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:27.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.512+0000 7f2fd3e00640 1 -- 172.21.15.102:0/1246552527 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4030840 con 0x7f2fd40add30 2024-03-23T03:47:27.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.512+0000 7f2fd3e00640 1 -- 172.21.15.102:0/1246552527 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc40309a0 con 0x7f2fd40add30 2024-03-23T03:47:27.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.512+0000 7f2fd3e00640 1 -- 172.21.15.102:0/1246552527 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4030d30 con 0x7f2fd40add30 2024-03-23T03:47:27.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.513+0000 7f2fdc000640 1 -- 172.21.15.102:0/1246552527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd40add30 msgr2=0x7f2fd40ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:27.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.513+0000 7f2fdc000640 1 --2- 172.21.15.102:0/1246552527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd40add30 0x7f2fd40ced10 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4009870 tx=0x7f2fc402b980 comp rx=0 tx=0).stop 2024-03-23T03:47:27.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.513+0000 7f2fdc000640 1 -- 172.21.15.102:0/1246552527 shutdown_connections 2024-03-23T03:47:27.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.513+0000 7f2fdc000640 1 --2- 172.21.15.102:0/1246552527 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fd406ce90 0x7f2fd406d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.513+0000 7f2fdc000640 1 --2- 172.21.15.102:0/1246552527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd40add30 0x7f2fd40ced10 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.513+0000 7f2fdc000640 1 -- 172.21.15.102:0/1246552527 >> 172.21.15.102:0/1246552527 conn(0x7f2fd406a0b0 msgr2=0x7f2fd406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:27.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.513+0000 7f2fdc000640 1 -- 172.21.15.102:0/1246552527 shutdown_connections 2024-03-23T03:47:27.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.513+0000 7f2fdc000640 1 -- 172.21.15.102:0/1246552527 wait complete. 2024-03-23T03:47:27.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.515+0000 7f2fdc000640 1 Processor -- start 2024-03-23T03:47:27.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.515+0000 7f2fdc000640 1 -- start start 2024-03-23T03:47:27.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.516+0000 7f2fdc000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fd406ce90 0x7f2fd4155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.516+0000 7f2fdc000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fd40add30 0x7f2fd41501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.516+0000 7f2fdc000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd40ce930 0x7f2fd40b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.516+0000 7f2fdc000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd4154c20 con 0x7f2fd40ce930 2024-03-23T03:47:27.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.516+0000 7f2fdc000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd4154f00 con 0x7f2fd40add30 2024-03-23T03:47:27.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.516+0000 7f2fdc000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd41551e0 con 0x7f2fd406ce90 2024-03-23T03:47:27.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.516+0000 7f2fda000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd40ce930 0x7f2fd40b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.516+0000 7f2fda000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd40ce930 0x7f2fd40b2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33306/0 (socket says 172.21.15.102:33306) 2024-03-23T03:47:27.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.516+0000 7f2fda000640 1 -- 172.21.15.102:0/2664629151 learned_addr learned my addr 172.21.15.102:0/2664629151 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:27.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.517+0000 7f2fd9600640 1 --2- 172.21.15.102:0/2664629151 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fd406ce90 0x7f2fd4155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.517+0000 7f2fd8c00640 1 --2- 172.21.15.102:0/2664629151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fd40add30 0x7f2fd41501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.517+0000 7f2fda000640 1 -- 172.21.15.102:0/2664629151 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fd406ce90 msgr2=0x7f2fd4155cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:27.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.517+0000 7f2fda000640 1 --2- 172.21.15.102:0/2664629151 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fd406ce90 0x7f2fd4155cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.517+0000 7f2fda000640 1 -- 172.21.15.102:0/2664629151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fd40add30 msgr2=0x7f2fd41501c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:27.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.517+0000 7f2fda000640 1 --2- 172.21.15.102:0/2664629151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fd40add30 0x7f2fd41501c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.517+0000 7f2fda000640 1 -- 172.21.15.102:0/2664629151 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc40094d0 con 0x7f2fd40ce930 2024-03-23T03:47:27.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.517+0000 7f2fda000640 1 --2- 172.21.15.102:0/2664629151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd40ce930 0x7f2fd40b2cb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc007e40 tx=0x7f2fcc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:27.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.517+0000 7f2fd1600640 1 -- 172.21.15.102:0/2664629151 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc014620 con 0x7f2fd40ce930 2024-03-23T03:47:27.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.517+0000 7f2fd1600640 1 -- 172.21.15.102:0/2664629151 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fcc00ee90 con 0x7f2fd40ce930 2024-03-23T03:47:27.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.517+0000 7f2fd1600640 1 -- 172.21.15.102:0/2664629151 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc0149e0 con 0x7f2fd40ce930 2024-03-23T03:47:27.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.517+0000 7f2fdc000640 1 -- 172.21.15.102:0/2664629151 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd41554c0 con 0x7f2fd40ce930 2024-03-23T03:47:27.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.518+0000 7f2fdc000640 1 -- 172.21.15.102:0/2664629151 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd4150700 con 0x7f2fd40ce930 2024-03-23T03:47:27.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.519+0000 7f2fdc000640 1 -- 172.21.15.102:0/2664629151 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd41559e0 con 0x7f2fd40ce930 2024-03-23T03:47:27.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.520+0000 7f2fd1600640 1 -- 172.21.15.102:0/2664629151 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2fcc014b40 con 0x7f2fd40ce930 2024-03-23T03:47:27.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.520+0000 7f2fd1600640 1 --2- 172.21.15.102:0/2664629151 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fb806ac00 0x7f2fb806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.521+0000 7f2fd1600640 1 -- 172.21.15.102:0/2664629151 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2fcc089ff0 con 0x7f2fd40ce930 2024-03-23T03:47:27.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.521+0000 7f2fd9600640 1 --2- 172.21.15.102:0/2664629151 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fb806ac00 0x7f2fb806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.521+0000 7f2fd9600640 1 --2- 172.21.15.102:0/2664629151 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fb806ac00 0x7f2fb806d0c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8000f60 tx=0x7f2fc8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:27.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.525+0000 7f2fd1600640 1 -- 172.21.15.102:0/2664629151 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2fcc0bd580 con 0x7f2fd40ce930 2024-03-23T03:47:27.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.717+0000 7f2fdc000640 1 -- 172.21.15.102:0/2664629151 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2fd4073390 con 0x7f2fd40ce930 2024-03-23T03:47:27.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.717+0000 7f2fd1600640 1 -- 172.21.15.102:0/2664629151 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v26) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2fd41559e0 con 0x7f2fd40ce930 2024-03-23T03:47:27.718 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:27.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.720+0000 7f2fdc000640 1 -- 172.21.15.102:0/2664629151 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fb806ac00 msgr2=0x7f2fb806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:27.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.720+0000 7f2fdc000640 1 --2- 172.21.15.102:0/2664629151 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fb806ac00 0x7f2fb806d0c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2fc8000f60 tx=0x7f2fc8009040 comp rx=0 tx=0).stop 2024-03-23T03:47:27.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.720+0000 7f2fdc000640 1 -- 172.21.15.102:0/2664629151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd40ce930 msgr2=0x7f2fd40b2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:27.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.720+0000 7f2fdc000640 1 --2- 172.21.15.102:0/2664629151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd40ce930 0x7f2fd40b2cb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc007e40 tx=0x7f2fcc00e4b0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.720+0000 7f2fdc000640 1 -- 172.21.15.102:0/2664629151 shutdown_connections 2024-03-23T03:47:27.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.720+0000 7f2fdc000640 1 --2- 172.21.15.102:0/2664629151 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fb806ac00 0x7f2fb806d0c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.720+0000 7f2fdc000640 1 --2- 172.21.15.102:0/2664629151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fd40ce930 0x7f2fd40b2cb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.720+0000 7f2fdc000640 1 --2- 172.21.15.102:0/2664629151 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fd40add30 0x7f2fd41501c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.720+0000 7f2fdc000640 1 --2- 172.21.15.102:0/2664629151 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fd406ce90 0x7f2fd4155cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.720+0000 7f2fdc000640 1 -- 172.21.15.102:0/2664629151 >> 172.21.15.102:0/2664629151 conn(0x7f2fd406a0b0 msgr2=0x7f2fd40cbc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:27.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.720+0000 7f2fdc000640 1 -- 172.21.15.102:0/2664629151 shutdown_connections 2024-03-23T03:47:27.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.720+0000 7f2fdc000640 1 -- 172.21.15.102:0/2664629151 wait complete. 2024-03-23T03:47:27.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:47:27.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.944+0000 7fc6fbc00640 1 Processor -- start 2024-03-23T03:47:27.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.944+0000 7fc6fbc00640 1 -- start start 2024-03-23T03:47:27.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.945+0000 7fc6fbc00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc6f40adbc0 0x7fc6f40b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.945+0000 7fc6fbc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc6f40cbab0 0x7fc6f40b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.945+0000 7fc6fbc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc6f406d390 0x7fc6f406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.945+0000 7fc6fbc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f406dd60 con 0x7fc6f406d390 2024-03-23T03:47:27.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.945+0000 7fc6fbc00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f40b16a0 con 0x7fc6f40adbc0 2024-03-23T03:47:27.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.945+0000 7fc6fbc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f40b4700 con 0x7fc6f40cbab0 2024-03-23T03:47:27.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.946+0000 7fc6f9c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc6f406d390 0x7fc6f406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.946+0000 7fc6f9c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc6f406d390 0x7fc6f406d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33308/0 (socket says 172.21.15.102:33308) 2024-03-23T03:47:27.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.946+0000 7fc6f9c00640 1 -- 172.21.15.102:0/2372570448 learned_addr learned my addr 172.21.15.102:0/2372570448 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:27.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.946+0000 7fc6f9200640 1 --2- 172.21.15.102:0/2372570448 >> v2:172.21.15.173:3300/0 conn(0x7fc6f40adbc0 0x7fc6f40b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.946+0000 7fc6f3e00640 1 --2- 172.21.15.102:0/2372570448 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc6f40cbab0 0x7fc6f40b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.946+0000 7fc6f9c00640 1 -- 172.21.15.102:0/2372570448 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc6f40cbab0 msgr2=0x7fc6f40b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:27.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.946+0000 7fc6f9c00640 1 --2- 172.21.15.102:0/2372570448 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc6f40cbab0 0x7fc6f40b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.946+0000 7fc6f9c00640 1 -- 172.21.15.102:0/2372570448 >> v2:172.21.15.173:3300/0 conn(0x7fc6f40adbc0 msgr2=0x7fc6f40b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:27.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.946+0000 7fc6f9c00640 1 --2- 172.21.15.102:0/2372570448 >> v2:172.21.15.173:3300/0 conn(0x7fc6f40adbc0 0x7fc6f40b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.946+0000 7fc6f9c00640 1 -- 172.21.15.102:0/2372570448 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f40cbfb0 con 0x7fc6f406d390 2024-03-23T03:47:27.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.947+0000 7fc6f9c00640 1 --2- 172.21.15.102:0/2372570448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f406d390 0x7fc6f406d820 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc6f406e020 tx=0x7fc6ec02e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b900119e8315fe9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:27.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.948+0000 7fc6f3400640 1 -- 172.21.15.102:0/2372570448 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec02ea00 con 0x7fc6f406d390 2024-03-23T03:47:27.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.948+0000 7fc6f3400640 1 -- 172.21.15.102:0/2372570448 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6ec02eb60 con 0x7fc6f406d390 2024-03-23T03:47:27.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.948+0000 7fc6f3400640 1 -- 172.21.15.102:0/2372570448 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec009600 con 0x7fc6f406d390 2024-03-23T03:47:27.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.948+0000 7fc6fbc00640 1 -- 172.21.15.102:0/2372570448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f406d390 msgr2=0x7fc6f406d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:27.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.948+0000 7fc6fbc00640 1 --2- 172.21.15.102:0/2372570448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f406d390 0x7fc6f406d820 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc6f406e020 tx=0x7fc6ec02e060 comp rx=0 tx=0).stop 2024-03-23T03:47:27.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.949+0000 7fc6fbc00640 1 -- 172.21.15.102:0/2372570448 shutdown_connections 2024-03-23T03:47:27.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.949+0000 7fc6fbc00640 1 --2- 172.21.15.102:0/2372570448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f406d390 0x7fc6f406d820 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.949+0000 7fc6fbc00640 1 --2- 172.21.15.102:0/2372570448 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc6f40cbab0 0x7fc6f40b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.949+0000 7fc6fbc00640 1 --2- 172.21.15.102:0/2372570448 >> v2:172.21.15.173:3300/0 conn(0x7fc6f40adbc0 0x7fc6f40b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.949+0000 7fc6fbc00640 1 -- 172.21.15.102:0/2372570448 >> 172.21.15.102:0/2372570448 conn(0x7fc6f406a3d0 msgr2=0x7fc6f40b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:27.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.949+0000 7fc6fbc00640 1 -- 172.21.15.102:0/2372570448 shutdown_connections 2024-03-23T03:47:27.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.949+0000 7fc6fbc00640 1 -- 172.21.15.102:0/2372570448 wait complete. 2024-03-23T03:47:27.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.951+0000 7fc6fbc00640 1 Processor -- start 2024-03-23T03:47:27.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.951+0000 7fc6fbc00640 1 -- start start 2024-03-23T03:47:27.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.952+0000 7fc6fbc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc6f406d390 0x7fc6f413a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.952+0000 7fc6fbc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f40adbc0 0x7fc6f415c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.952+0000 7fc6fbc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6f40cbab0 0x7fc6f415c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.952+0000 7fc6fbc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f413e110 con 0x7fc6f40adbc0 2024-03-23T03:47:27.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.952+0000 7fc6fbc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f413e3f0 con 0x7fc6f40cbab0 2024-03-23T03:47:27.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.952+0000 7fc6fbc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f413e6d0 con 0x7fc6f406d390 2024-03-23T03:47:27.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.952+0000 7fc6f3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f40adbc0 0x7fc6f415c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.953+0000 7fc6f3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f40adbc0 0x7fc6f415c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33312/0 (socket says 172.21.15.102:33312) 2024-03-23T03:47:27.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.953+0000 7fc6f3e00640 1 -- 172.21.15.102:0/2653003677 learned_addr learned my addr 172.21.15.102:0/2653003677 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:27.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.953+0000 7fc6f9c00640 1 --2- 172.21.15.102:0/2653003677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6f40cbab0 0x7fc6f415c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.953+0000 7fc6f9200640 1 --2- 172.21.15.102:0/2653003677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc6f406d390 0x7fc6f413a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.953+0000 7fc6f3e00640 1 -- 172.21.15.102:0/2653003677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc6f406d390 msgr2=0x7fc6f413a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:27.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.953+0000 7fc6f3e00640 1 --2- 172.21.15.102:0/2653003677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc6f406d390 0x7fc6f413a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.953+0000 7fc6f3e00640 1 -- 172.21.15.102:0/2653003677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6f40cbab0 msgr2=0x7fc6f415c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:27.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.953+0000 7fc6f3e00640 1 --2- 172.21.15.102:0/2653003677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6f40cbab0 0x7fc6f415c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:27.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.953+0000 7fc6f3e00640 1 -- 172.21.15.102:0/2653003677 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ec00bd40 con 0x7fc6f40adbc0 2024-03-23T03:47:27.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.953+0000 7fc6f9c00640 1 --2- 172.21.15.102:0/2653003677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6f40cbab0 0x7fc6f415c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:47:27.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.953+0000 7fc6f3e00640 1 --2- 172.21.15.102:0/2653003677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f40adbc0 0x7fc6f415c100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000d800 tx=0x7fc6e000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:27.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.954+0000 7fc6f0c00640 1 -- 172.21.15.102:0/2653003677 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e000b6c0 con 0x7fc6f40adbc0 2024-03-23T03:47:27.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.954+0000 7fc6f0c00640 1 -- 172.21.15.102:0/2653003677 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e000b820 con 0x7fc6f40adbc0 2024-03-23T03:47:27.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.954+0000 7fc6f0c00640 1 -- 172.21.15.102:0/2653003677 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e000bbb0 con 0x7fc6f40adbc0 2024-03-23T03:47:27.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.954+0000 7fc6fbc00640 1 -- 172.21.15.102:0/2653003677 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f41407e0 con 0x7fc6f40adbc0 2024-03-23T03:47:27.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.954+0000 7fc6fbc00640 1 -- 172.21.15.102:0/2653003677 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f4140da0 con 0x7fc6f40adbc0 2024-03-23T03:47:27.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.955+0000 7fc6fbc00640 1 -- 172.21.15.102:0/2653003677 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f4140f60 con 0x7fc6f40adbc0 2024-03-23T03:47:27.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.956+0000 7fc6f0c00640 1 -- 172.21.15.102:0/2653003677 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc6e000bd10 con 0x7fc6f40adbc0 2024-03-23T03:47:27.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.957+0000 7fc6f0c00640 1 --2- 172.21.15.102:0/2653003677 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc6d406ac30 0x7fc6d406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:27.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.957+0000 7fc6f0c00640 1 -- 172.21.15.102:0/2653003677 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc6e0089b90 con 0x7fc6f40adbc0 2024-03-23T03:47:27.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.957+0000 7fc6f9200640 1 --2- 172.21.15.102:0/2653003677 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc6d406ac30 0x7fc6d406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:27.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.958+0000 7fc6f9200640 1 --2- 172.21.15.102:0/2653003677 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc6d406ac30 0x7fc6d406d0f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8006fd0 tx=0x7fc6e8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:27.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:27.961+0000 7fc6f0c00640 1 -- 172.21.15.102:0/2653003677 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc6e0054a80 con 0x7fc6f40adbc0 2024-03-23T03:47:28.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.193+0000 7fc6fbc00640 1 -- 172.21.15.102:0/2653003677 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6f406ac20 con 0x7fc6f40adbc0 2024-03-23T03:47:28.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.194+0000 7fc6f0c00640 1 -- 172.21.15.102:0/2653003677 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4739 (secure 0 0 0) 0x7fc6e00580a0 con 0x7fc6f40adbc0 2024-03-23T03:47:28.195 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:28.195 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/891818172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":891818172},{"type":"v1","addr":"172.21.15.102:6836","nonce":891818172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2404489233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2404489233},{"type":"v1","addr":"172.21.15.102:6837","nonce":2404489233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/1839967745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":1839967745},{"type":"v1","addr":"172.21.15.173:6835","nonce":1839967745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:46:53.912287+0000","modified":"2024-03-23T03:47:14.818670+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6834/1462637509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1462637509},{"type":"v1","addr":"172.21.15.173:6834","nonce":1462637509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-23T03:47:28.196 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 8 2024-03-23T03:47:28.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.197+0000 7fc6fbc00640 1 -- 172.21.15.102:0/2653003677 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc6d406ac30 msgr2=0x7fc6d406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:28.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.197+0000 7fc6fbc00640 1 --2- 172.21.15.102:0/2653003677 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc6d406ac30 0x7fc6d406d0f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8006fd0 tx=0x7fc6e8008040 comp rx=0 tx=0).stop 2024-03-23T03:47:28.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.197+0000 7fc6fbc00640 1 -- 172.21.15.102:0/2653003677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f40adbc0 msgr2=0x7fc6f415c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:28.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.197+0000 7fc6fbc00640 1 --2- 172.21.15.102:0/2653003677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f40adbc0 0x7fc6f415c100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc6e000d800 tx=0x7fc6e000dcc0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.197+0000 7fc6fbc00640 1 -- 172.21.15.102:0/2653003677 shutdown_connections 2024-03-23T03:47:28.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.197+0000 7fc6fbc00640 1 --2- 172.21.15.102:0/2653003677 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc6d406ac30 0x7fc6d406d0f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.197+0000 7fc6fbc00640 1 --2- 172.21.15.102:0/2653003677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6f40cbab0 0x7fc6f415c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.197+0000 7fc6fbc00640 1 --2- 172.21.15.102:0/2653003677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6f40adbc0 0x7fc6f415c100 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.197+0000 7fc6fbc00640 1 --2- 172.21.15.102:0/2653003677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc6f406d390 0x7fc6f413a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.197+0000 7fc6fbc00640 1 -- 172.21.15.102:0/2653003677 >> 172.21.15.102:0/2653003677 conn(0x7fc6f406a3d0 msgr2=0x7fc6f40affb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:28.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.197+0000 7fc6fbc00640 1 -- 172.21.15.102:0/2653003677 shutdown_connections 2024-03-23T03:47:28.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.197+0000 7fc6fbc00640 1 -- 172.21.15.102:0/2653003677 wait complete. 2024-03-23T03:47:28.217 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:28.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.422+0000 7f3b14e00640 1 Processor -- start 2024-03-23T03:47:28.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.422+0000 7f3b14e00640 1 -- start start 2024-03-23T03:47:28.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.424+0000 7f3b14e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f3b1006d390 0x7f3b1006d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:28.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.424+0000 7f3b14e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b100b16a0 con 0x7f3b100cbab0 2024-03-23T03:47:28.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.424+0000 7f3b14e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b100cbfb0 con 0x7f3b100adbc0 2024-03-23T03:47:28.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.424+0000 7f3b14e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b1006dd40 con 0x7f3b1006d390 2024-03-23T03:47:28.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.424+0000 7f3b0d400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3b100cbab0 0x7f3b100b20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52816/0 (socket says 172.21.15.102:52816) 2024-03-23T03:47:28.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.424+0000 7f3b0d400640 1 -- 172.21.15.102:0/1089435510 learned_addr learned my addr 172.21.15.102:0/1089435510 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:28.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.424+0000 7f3b0e800640 1 --2- 172.21.15.102:0/1089435510 >> v2:172.21.15.173:3300/0 conn(0x7f3b1006d390 0x7f3b1006d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:28.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.424+0000 7f3b0ca00640 1 -- 172.21.15.102:0/1089435510 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672666368 0 0) 0x7f3b100b16a0 con 0x7f3b100cbab0 2024-03-23T03:47:28.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.424+0000 7f3b0ca00640 1 -- 172.21.15.102:0/1089435510 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3af4003520 con 0x7f3b100cbab0 2024-03-23T03:47:28.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.425+0000 7f3b0ca00640 1 -- 172.21.15.102:0/1089435510 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1968019251 0 0) 0x7f3af4003520 con 0x7f3b100cbab0 2024-03-23T03:47:28.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.425+0000 7f3b0ca00640 1 -- 172.21.15.102:0/1089435510 >> v2:172.21.15.173:3300/0 conn(0x7f3b1006d390 msgr2=0x7f3b1006d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:28.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.425+0000 7f3b0ca00640 1 --2- 172.21.15.102:0/1089435510 >> v2:172.21.15.173:3300/0 conn(0x7f3b1006d390 0x7f3b1006d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.425+0000 7f3b0ca00640 1 -- 172.21.15.102:0/1089435510 >> v1:172.21.15.173:6789/0 conn(0x7f3b100adbc0 legacy=0x7f3b100b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:28.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.425+0000 7f3b0ca00640 1 -- 172.21.15.102:0/1089435510 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b100cc290 con 0x7f3b100cbab0 2024-03-23T03:47:28.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.425+0000 7f3b0ca00640 1 -- 172.21.15.102:0/1089435510 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f3afc001670 con 0x7f3b100cbab0 2024-03-23T03:47:28.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.425+0000 7f3b0ca00640 1 -- 172.21.15.102:0/1089435510 >> v1:172.21.15.102:6789/0 conn(0x7f3b100cbab0 legacy=0x7f3b100b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:28.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.425+0000 7f3b0ca00640 1 --2- 172.21.15.102:0/1089435510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3af4003970 0x7f3af4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:28.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.425+0000 7f3b0ca00640 1 --2- 172.21.15.102:0/1089435510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af40047c0 0x7f3af4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:28.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.425+0000 7f3b0ca00640 1 --2- 172.21.15.102:0/1089435510 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3af4005180 0x7f3af400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:28.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.425+0000 7f3b0ca00640 1 -- 172.21.15.102:0/1089435510 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1006dd40 con 0x7f3af40047c0 2024-03-23T03:47:28.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.425+0000 7f3b0ca00640 1 -- 172.21.15.102:0/1089435510 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af400c120 con 0x7f3af4003970 2024-03-23T03:47:28.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.425+0000 7f3b0ca00640 1 -- 172.21.15.102:0/1089435510 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af400c2e0 con 0x7f3af4005180 2024-03-23T03:47:28.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.425+0000 7f3b0d400640 1 --2- 172.21.15.102:0/1089435510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af40047c0 0x7f3af4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:28.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.426+0000 7f3b0d400640 1 -- 172.21.15.102:0/1089435510 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3af4005180 msgr2=0x7f3af400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:28.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.426+0000 7f3b0d400640 1 --2- 172.21.15.102:0/1089435510 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3af4005180 0x7f3af400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.426+0000 7f3b0de00640 1 --2- 172.21.15.102:0/1089435510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3af4003970 0x7f3af4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:28.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.426+0000 7f3b0d400640 1 -- 172.21.15.102:0/1089435510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3af4003970 msgr2=0x7f3af4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:28.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.426+0000 7f3b0d400640 1 --2- 172.21.15.102:0/1089435510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3af4003970 0x7f3af4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.426+0000 7f3b0d400640 1 -- 172.21.15.102:0/1089435510 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af4003520 con 0x7f3af40047c0 2024-03-23T03:47:28.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.426+0000 7f3b0e800640 1 -- 172.21.15.102:0/1089435510 reap_dead start 2024-03-23T03:47:28.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.426+0000 7f3b0de00640 1 --2- 172.21.15.102:0/1089435510 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3af4003970 0x7f3af4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.427+0000 7f3b0d400640 1 --2- 172.21.15.102:0/1089435510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af40047c0 0x7f3af4004c40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3afc00c210 tx=0x7f3af40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=942be0d0c8bc52f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.427+0000 7f3b0ca00640 1 -- 172.21.15.102:0/1089435510 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc005ad0 con 0x7f3af40047c0 2024-03-23T03:47:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.427+0000 7f3b0ca00640 1 -- 172.21.15.102:0/1089435510 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3afc005c50 con 0x7f3af40047c0 2024-03-23T03:47:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.427+0000 7f3b0ca00640 1 -- 172.21.15.102:0/1089435510 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc02c850 con 0x7f3af40047c0 2024-03-23T03:47:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.427+0000 7f3b14e00640 1 -- 172.21.15.102:0/1089435510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af40047c0 msgr2=0x7f3af4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.427+0000 7f3b14e00640 1 --2- 172.21.15.102:0/1089435510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af40047c0 0x7f3af4004c40 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3afc00c210 tx=0x7f3af40057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.428+0000 7f3b14e00640 1 -- 172.21.15.102:0/1089435510 shutdown_connections 2024-03-23T03:47:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.428+0000 7f3b14e00640 1 --2- 172.21.15.102:0/1089435510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af40047c0 0x7f3af4004c40 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.428+0000 7f3b14e00640 1 -- 172.21.15.102:0/1089435510 >> 172.21.15.102:0/1089435510 conn(0x7f3b1006a3d0 msgr2=0x7f3b100b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.428+0000 7f3b14e00640 1 -- 172.21.15.102:0/1089435510 shutdown_connections 2024-03-23T03:47:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.428+0000 7f3b14e00640 1 -- 172.21.15.102:0/1089435510 wait complete. 2024-03-23T03:47:28.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.429+0000 7f3b14e00640 1 Processor -- start 2024-03-23T03:47:28.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.429+0000 7f3b14e00640 1 -- start start 2024-03-23T03:47:28.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.430+0000 7f3b14e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af40047c0 0x7f3b10135380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:28.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.430+0000 7f3b14e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b10138f60 0x7f3b10133a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:28.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.430+0000 7f3b14e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b10167ab0 0x7f3b1015af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:28.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.430+0000 7f3b14e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b101358c0 con 0x7f3af40047c0 2024-03-23T03:47:28.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.430+0000 7f3b14e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b101342e0 con 0x7f3b10138f60 2024-03-23T03:47:28.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.430+0000 7f3b14e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b101345c0 con 0x7f3b10167ab0 2024-03-23T03:47:28.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.430+0000 7f3b0de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af40047c0 0x7f3b10135380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:28.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.431+0000 7f3b0de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af40047c0 0x7f3b10135380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33336/0 (socket says 172.21.15.102:33336) 2024-03-23T03:47:28.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.431+0000 7f3b0de00640 1 -- 172.21.15.102:0/2074009292 learned_addr learned my addr 172.21.15.102:0/2074009292 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:28.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.431+0000 7f3b0d400640 1 --2- 172.21.15.102:0/2074009292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b10138f60 0x7f3b10133a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:28.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.431+0000 7f3b0e800640 1 --2- 172.21.15.102:0/2074009292 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b10167ab0 0x7f3b1015af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:28.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.431+0000 7f3b0de00640 1 -- 172.21.15.102:0/2074009292 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b10167ab0 msgr2=0x7f3b1015af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:28.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.431+0000 7f3b0de00640 1 --2- 172.21.15.102:0/2074009292 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b10167ab0 0x7f3b1015af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.431+0000 7f3b0de00640 1 -- 172.21.15.102:0/2074009292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b10138f60 msgr2=0x7f3b10133a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:28.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.431+0000 7f3b0de00640 1 --2- 172.21.15.102:0/2074009292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b10138f60 0x7f3b10133a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.431+0000 7f3b0de00640 1 -- 172.21.15.102:0/2074009292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b101348a0 con 0x7f3af40047c0 2024-03-23T03:47:28.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.431+0000 7f3b0d400640 1 --2- 172.21.15.102:0/2074009292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b10138f60 0x7f3b10133a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:28.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.431+0000 7f3b0de00640 1 --2- 172.21.15.102:0/2074009292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af40047c0 0x7f3b10135380 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000d480 tx=0x7f3b0000d940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:28.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.432+0000 7f3b06000640 1 -- 172.21.15.102:0/2074009292 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00005480 con 0x7f3af40047c0 2024-03-23T03:47:28.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.432+0000 7f3b06000640 1 -- 172.21.15.102:0/2074009292 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b00003610 con 0x7f3af40047c0 2024-03-23T03:47:28.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.432+0000 7f3b06000640 1 -- 172.21.15.102:0/2074009292 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b00003a00 con 0x7f3af40047c0 2024-03-23T03:47:28.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.432+0000 7f3b14e00640 1 -- 172.21.15.102:0/2074009292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b10134b80 con 0x7f3af40047c0 2024-03-23T03:47:28.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.432+0000 7f3b14e00640 1 -- 172.21.15.102:0/2074009292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b10133fd0 con 0x7f3af40047c0 2024-03-23T03:47:28.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.433+0000 7f3b14e00640 1 -- 172.21.15.102:0/2074009292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1006ac20 con 0x7f3af40047c0 2024-03-23T03:47:28.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.434+0000 7f3b06000640 1 -- 172.21.15.102:0/2074009292 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3b00003b60 con 0x7f3af40047c0 2024-03-23T03:47:28.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.435+0000 7f3b06000640 1 --2- 172.21.15.102:0/2074009292 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3af006ac00 0x7f3af006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:28.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.435+0000 7f3b06000640 1 -- 172.21.15.102:0/2074009292 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3b00089550 con 0x7f3af40047c0 2024-03-23T03:47:28.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.435+0000 7f3b0d400640 1 --2- 172.21.15.102:0/2074009292 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3af006ac00 0x7f3af006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:28.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.436+0000 7f3b0d400640 1 --2- 172.21.15.102:0/2074009292 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3af006ac00 0x7f3af006d0c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3afc034000 tx=0x7f3afc009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:28.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.439+0000 7f3b06000640 1 -- 172.21.15.102:0/2074009292 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3b000544f0 con 0x7f3af40047c0 2024-03-23T03:47:28.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.634+0000 7f3b14e00640 1 -- 172.21.15.102:0/2074009292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b10073390 con 0x7f3af40047c0 2024-03-23T03:47:28.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.637+0000 7f3b06000640 1 -- 172.21.15.102:0/2074009292 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+17926 (secure 0 0 0) 0x7f3b00057b10 con 0x7f3af40047c0 2024-03-23T03:47:28.638 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:28.638 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:46:58.410551+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:47:28.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.640+0000 7f3b14e00640 1 -- 172.21.15.102:0/2074009292 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3af006ac00 msgr2=0x7f3af006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:28.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.640+0000 7f3b14e00640 1 --2- 172.21.15.102:0/2074009292 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3af006ac00 0x7f3af006d0c0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3afc034000 tx=0x7f3afc009030 comp rx=0 tx=0).stop 2024-03-23T03:47:28.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.640+0000 7f3b14e00640 1 -- 172.21.15.102:0/2074009292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af40047c0 msgr2=0x7f3b10135380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:28.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.640+0000 7f3b14e00640 1 --2- 172.21.15.102:0/2074009292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af40047c0 0x7f3b10135380 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3b0000d480 tx=0x7f3b0000d940 comp rx=0 tx=0).stop 2024-03-23T03:47:28.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.640+0000 7f3b14e00640 1 -- 172.21.15.102:0/2074009292 shutdown_connections 2024-03-23T03:47:28.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.640+0000 7f3b14e00640 1 --2- 172.21.15.102:0/2074009292 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3af006ac00 0x7f3af006d0c0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.640+0000 7f3b14e00640 1 --2- 172.21.15.102:0/2074009292 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b10167ab0 0x7f3b1015af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.640+0000 7f3b14e00640 1 --2- 172.21.15.102:0/2074009292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b10138f60 0x7f3b10133a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.640+0000 7f3b14e00640 1 --2- 172.21.15.102:0/2074009292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3af40047c0 0x7f3b10135380 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.640+0000 7f3b14e00640 1 -- 172.21.15.102:0/2074009292 >> 172.21.15.102:0/2074009292 conn(0x7f3b1006a3d0 msgr2=0x7f3b100b3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:28.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.640+0000 7f3b14e00640 1 -- 172.21.15.102:0/2074009292 shutdown_connections 2024-03-23T03:47:28.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.640+0000 7f3b14e00640 1 -- 172.21.15.102:0/2074009292 wait complete. 2024-03-23T03:47:28.660 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:47:28.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.868+0000 7fb0bfe00640 1 Processor -- start 2024-03-23T03:47:28.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.868+0000 7fb0bfe00640 1 -- start start 2024-03-23T03:47:28.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.870+0000 7fb0bfe00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb0c0156e30 0x7fb0c006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:28.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.870+0000 7fb0bfe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0c00cb540 0x7fb0c00b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:28.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.870+0000 7fb0bfe00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c00cc8c0 con 0x7fb0c00b3310 2024-03-23T03:47:28.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.870+0000 7fb0bfe00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c006d630 con 0x7fb0c0156e30 2024-03-23T03:47:28.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.870+0000 7fb0bfe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c015ce60 con 0x7fb0c00cb540 2024-03-23T03:47:28.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.870+0000 7fb0bea00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb0c0156e30 0x7fb0c006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:28.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.870+0000 7fb0bf400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0c00cb540 0x7fb0c00b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:28.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.870+0000 7fb0bea00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb0c0156e30 0x7fb0c006d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:59486/0 (socket says 172.21.15.102:59486) 2024-03-23T03:47:28.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.870+0000 7fb0bea00640 1 -- 172.21.15.102:0/3430140358 learned_addr learned my addr 172.21.15.102:0/3430140358 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:28.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.871+0000 7fb0bea00640 1 -- 172.21.15.102:0/3430140358 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0c00cb540 msgr2=0x7fb0c00b1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:28.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.871+0000 7fb0bea00640 1 --2- 172.21.15.102:0/3430140358 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0c00cb540 0x7fb0c00b1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.871+0000 7fb0bea00640 1 -- 172.21.15.102:0/3430140358 >> v1:172.21.15.173:6789/0 conn(0x7fb0c00b3310 legacy=0x7fb0c00cae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:28.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.871+0000 7fb0bea00640 1 -- 172.21.15.102:0/3430140358 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c015d120 con 0x7fb0c0156e30 2024-03-23T03:47:28.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.872+0000 7fb0bea00640 1 --2- 172.21.15.102:0/3430140358 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0c0156e30 0x7fb0c006d0f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009870 tx=0x7fb0b002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4915f557dce64501 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:28.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.873+0000 7fb0be000640 1 -- 172.21.15.102:0/3430140358 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0030840 con 0x7fb0c0156e30 2024-03-23T03:47:28.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.873+0000 7fb0be000640 1 -- 172.21.15.102:0/3430140358 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b00309a0 con 0x7fb0c0156e30 2024-03-23T03:47:28.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.873+0000 7fb0be000640 1 -- 172.21.15.102:0/3430140358 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0030d30 con 0x7fb0c0156e30 2024-03-23T03:47:28.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.873+0000 7fb0bfe00640 1 -- 172.21.15.102:0/3430140358 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0c0156e30 msgr2=0x7fb0c006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:28.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.873+0000 7fb0bfe00640 1 --2- 172.21.15.102:0/3430140358 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0c0156e30 0x7fb0c006d0f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0009870 tx=0x7fb0b002b980 comp rx=0 tx=0).stop 2024-03-23T03:47:28.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.874+0000 7fb0bfe00640 1 -- 172.21.15.102:0/3430140358 shutdown_connections 2024-03-23T03:47:28.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.874+0000 7fb0bfe00640 1 --2- 172.21.15.102:0/3430140358 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0c00cb540 0x7fb0c00b1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.874+0000 7fb0bfe00640 1 --2- 172.21.15.102:0/3430140358 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0c0156e30 0x7fb0c006d0f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.874+0000 7fb0bfe00640 1 -- 172.21.15.102:0/3430140358 >> 172.21.15.102:0/3430140358 conn(0x7fb0c0065df0 msgr2=0x7fb0c0069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:28.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.874+0000 7fb0bfe00640 1 -- 172.21.15.102:0/3430140358 shutdown_connections 2024-03-23T03:47:28.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.874+0000 7fb0bfe00640 1 -- 172.21.15.102:0/3430140358 wait complete. 2024-03-23T03:47:28.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.876+0000 7fb0bfe00640 1 Processor -- start 2024-03-23T03:47:28.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.876+0000 7fb0bfe00640 1 -- start start 2024-03-23T03:47:28.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.877+0000 7fb0bfe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0c00b3310 0x7fb0c01515a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:28.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.877+0000 7fb0bfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c00cb540 0x7fb0c01f58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:28.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.877+0000 7fb0bfe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0c0156e30 0x7fb0c0144ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:28.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.877+0000 7fb0bfe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c01f22c0 con 0x7fb0c00cb540 2024-03-23T03:47:28.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.877+0000 7fb0bfe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c01f5df0 con 0x7fb0c00b3310 2024-03-23T03:47:28.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.877+0000 7fb0bfe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c01f6020 con 0x7fb0c0156e30 2024-03-23T03:47:28.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.877+0000 7fb0b7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c00cb540 0x7fb0c01f58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:28.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.877+0000 7fb0b7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c00cb540 0x7fb0c01f58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33346/0 (socket says 172.21.15.102:33346) 2024-03-23T03:47:28.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.877+0000 7fb0b7e00640 1 -- 172.21.15.102:0/2772597823 learned_addr learned my addr 172.21.15.102:0/2772597823 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:28.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.878+0000 7fb0bea00640 1 --2- 172.21.15.102:0/2772597823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0c00b3310 0x7fb0c01515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:28.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.878+0000 7fb0bf400640 1 --2- 172.21.15.102:0/2772597823 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0c0156e30 0x7fb0c0144ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:28.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.878+0000 7fb0b7e00640 1 -- 172.21.15.102:0/2772597823 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0c0156e30 msgr2=0x7fb0c0144ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:28.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.878+0000 7fb0b7e00640 1 --2- 172.21.15.102:0/2772597823 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0c0156e30 0x7fb0c0144ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.878+0000 7fb0b7e00640 1 -- 172.21.15.102:0/2772597823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0c00b3310 msgr2=0x7fb0c01515a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:28.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.878+0000 7fb0b7e00640 1 --2- 172.21.15.102:0/2772597823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0c00b3310 0x7fb0c01515a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:28.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.878+0000 7fb0b7e00640 1 -- 172.21.15.102:0/2772597823 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b00094d0 con 0x7fb0c00cb540 2024-03-23T03:47:28.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.878+0000 7fb0b7e00640 1 --2- 172.21.15.102:0/2772597823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c00cb540 0x7fb0c01f58b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800c430 tx=0x7fb0a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:28.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.879+0000 7fb0b6a00640 1 -- 172.21.15.102:0/2772597823 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a80052d0 con 0x7fb0c00cb540 2024-03-23T03:47:28.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.879+0000 7fb0b6a00640 1 -- 172.21.15.102:0/2772597823 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a8005430 con 0x7fb0c00cb540 2024-03-23T03:47:28.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.879+0000 7fb0bfe00640 1 -- 172.21.15.102:0/2772597823 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c01f6300 con 0x7fb0c00cb540 2024-03-23T03:47:28.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.879+0000 7fb0b6a00640 1 -- 172.21.15.102:0/2772597823 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8003830 con 0x7fb0c00cb540 2024-03-23T03:47:28.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.879+0000 7fb0bfe00640 1 -- 172.21.15.102:0/2772597823 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c01f24b0 con 0x7fb0c00cb540 2024-03-23T03:47:28.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.880+0000 7fb0bfe00640 1 -- 172.21.15.102:0/2772597823 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c01f6820 con 0x7fb0c00cb540 2024-03-23T03:47:28.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.881+0000 7fb0b6a00640 1 -- 172.21.15.102:0/2772597823 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb0a8003990 con 0x7fb0c00cb540 2024-03-23T03:47:28.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.882+0000 7fb0b6a00640 1 --2- 172.21.15.102:0/2772597823 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb09c06ac00 0x7fb09c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:28.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.882+0000 7fb0b6a00640 1 -- 172.21.15.102:0/2772597823 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb0a8088c00 con 0x7fb0c00cb540 2024-03-23T03:47:28.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.882+0000 7fb0bea00640 1 --2- 172.21.15.102:0/2772597823 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb09c06ac00 0x7fb09c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:28.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.883+0000 7fb0bea00640 1 --2- 172.21.15.102:0/2772597823 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb09c06ac00 0x7fb09c06d0c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0036000 tx=0x7fb0b003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:28.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:28.886+0000 7fb0b6a00640 1 -- 172.21.15.102:0/2772597823 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb0a8053ae0 con 0x7fb0c00cb540 2024-03-23T03:47:29.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.119+0000 7fb0bfe00640 1 -- 172.21.15.102:0/2772597823 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0c006d800 con 0x7fb0c00cb540 2024-03-23T03:47:29.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.121+0000 7fb0b6a00640 1 -- 172.21.15.102:0/2772597823 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4739 (secure 0 0 0) 0x7fb0a8017090 con 0x7fb0c00cb540 2024-03-23T03:47:29.122 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:29.122 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/891818172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":891818172},{"type":"v1","addr":"172.21.15.102:6836","nonce":891818172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2404489233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2404489233},{"type":"v1","addr":"172.21.15.102:6837","nonce":2404489233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/1839967745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":1839967745},{"type":"v1","addr":"172.21.15.173:6835","nonce":1839967745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:46:53.912287+0000","modified":"2024-03-23T03:47:14.818670+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6834/1462637509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1462637509},{"type":"v1","addr":"172.21.15.173:6834","nonce":1462637509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-23T03:47:29.122 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 8 2024-03-23T03:47:29.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.123+0000 7fb0bfe00640 1 -- 172.21.15.102:0/2772597823 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb09c06ac00 msgr2=0x7fb09c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:29.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.123+0000 7fb0bfe00640 1 --2- 172.21.15.102:0/2772597823 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb09c06ac00 0x7fb09c06d0c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb0b0036000 tx=0x7fb0b003b040 comp rx=0 tx=0).stop 2024-03-23T03:47:29.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.123+0000 7fb0bfe00640 1 -- 172.21.15.102:0/2772597823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c00cb540 msgr2=0x7fb0c01f58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:29.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.123+0000 7fb0bfe00640 1 --2- 172.21.15.102:0/2772597823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c00cb540 0x7fb0c01f58b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb0a800c430 tx=0x7fb0a800c8f0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.124+0000 7fb0bfe00640 1 -- 172.21.15.102:0/2772597823 shutdown_connections 2024-03-23T03:47:29.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.124+0000 7fb0bfe00640 1 --2- 172.21.15.102:0/2772597823 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb09c06ac00 0x7fb09c06d0c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.124+0000 7fb0bfe00640 1 --2- 172.21.15.102:0/2772597823 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0c0156e30 0x7fb0c0144ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.124+0000 7fb0bfe00640 1 --2- 172.21.15.102:0/2772597823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0c00cb540 0x7fb0c01f58b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.124+0000 7fb0bfe00640 1 --2- 172.21.15.102:0/2772597823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0c00b3310 0x7fb0c01515a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.124+0000 7fb0bfe00640 1 -- 172.21.15.102:0/2772597823 >> 172.21.15.102:0/2772597823 conn(0x7fb0c0065df0 msgr2=0x7fb0c0069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:29.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.124+0000 7fb0bfe00640 1 -- 172.21.15.102:0/2772597823 shutdown_connections 2024-03-23T03:47:29.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.124+0000 7fb0bfe00640 1 -- 172.21.15.102:0/2772597823 wait complete. 2024-03-23T03:47:29.545 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T03:46:53.912287+0000', 'modified': '2024-03-23T03:47:14.818670+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4270}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4270': {'gid': 4270, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.173:6834/1462637509', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6832', 'nonce': 1462637509}, {'type': 'v1', 'addr': '172.21.15.173:6834', 'nonce': 1462637509}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T03:47:29.545 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T03:47:29.545 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-23T03:47:29.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.758+0000 7f992f400640 1 Processor -- start 2024-03-23T03:47:29.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.759+0000 7f992f400640 1 -- start start 2024-03-23T03:47:29.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.760+0000 7f992f400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f99280cbae0 0x7f992806c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.760+0000 7f992f400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99280cbf50 con 0x7f99280ce930 2024-03-23T03:47:29.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.760+0000 7f992f400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992815ae60 con 0x7f99280cb660 2024-03-23T03:47:29.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.760+0000 7f992f400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f992815bae0 con 0x7f99280cbae0 2024-03-23T03:47:29.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.760+0000 7f992ca00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f99280ce930 0x7f99280ced10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52828/0 (socket says 172.21.15.102:52828) 2024-03-23T03:47:29.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.760+0000 7f992ca00640 1 -- 172.21.15.102:0/1575972494 learned_addr learned my addr 172.21.15.102:0/1575972494 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:29.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.761+0000 7f992d400640 1 --2- 172.21.15.102:0/1575972494 >> v2:172.21.15.173:3300/0 conn(0x7f99280cbae0 0x7f992806c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.761+0000 7f9927400640 1 -- 172.21.15.102:0/1575972494 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2629848430 0 0) 0x7f9910001110 con 0x7f99280ce930 2024-03-23T03:47:29.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.761+0000 7f9927400640 1 -- 172.21.15.102:0/1575972494 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f990c003520 con 0x7f99280ce930 2024-03-23T03:47:29.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.761+0000 7f9927400640 1 -- 172.21.15.102:0/1575972494 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1435873858 0 0) 0x7f990c003520 con 0x7f99280ce930 2024-03-23T03:47:29.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.761+0000 7f9927400640 1 -- 172.21.15.102:0/1575972494 >> v2:172.21.15.173:3300/0 conn(0x7f99280cbae0 msgr2=0x7f992806c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:29.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.761+0000 7f9927400640 1 --2- 172.21.15.102:0/1575972494 >> v2:172.21.15.173:3300/0 conn(0x7f99280cbae0 0x7f992806c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.761+0000 7f9927400640 1 -- 172.21.15.102:0/1575972494 >> v1:172.21.15.173:6789/0 conn(0x7f99280cb660 legacy=0x7f992806c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:29.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.761+0000 7f9927400640 1 -- 172.21.15.102:0/1575972494 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992815b140 con 0x7f99280ce930 2024-03-23T03:47:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.761+0000 7f9927400640 1 -- 172.21.15.102:0/1575972494 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f9910001800 con 0x7f99280ce930 2024-03-23T03:47:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.761+0000 7f9927400640 1 -- 172.21.15.102:0/1575972494 >> v1:172.21.15.102:6789/0 conn(0x7f99280ce930 legacy=0x7f99280ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.761+0000 7f9927400640 1 --2- 172.21.15.102:0/1575972494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f990c0039c0 0x7f990c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.761+0000 7f9927400640 1 --2- 172.21.15.102:0/1575972494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f990c0047c0 0x7f990c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.761+0000 7f9927400640 1 --2- 172.21.15.102:0/1575972494 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f990c005180 0x7f990c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.761+0000 7f9927400640 1 -- 172.21.15.102:0/1575972494 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992815bae0 con 0x7f990c0039c0 2024-03-23T03:47:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.761+0000 7f9927400640 1 -- 172.21.15.102:0/1575972494 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c00c080 con 0x7f990c0047c0 2024-03-23T03:47:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.761+0000 7f9927400640 1 -- 172.21.15.102:0/1575972494 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f990c00c240 con 0x7f990c005180 2024-03-23T03:47:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.762+0000 7f992ca00640 1 --2- 172.21.15.102:0/1575972494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f990c0039c0 0x7f990c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.762+0000 7f9927e00640 1 --2- 172.21.15.102:0/1575972494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f990c0047c0 0x7f990c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.762+0000 7f992ca00640 1 -- 172.21.15.102:0/1575972494 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f990c005180 msgr2=0x7f990c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.762+0000 7f992ca00640 1 --2- 172.21.15.102:0/1575972494 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f990c005180 0x7f990c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.762+0000 7f992ca00640 1 -- 172.21.15.102:0/1575972494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f990c0047c0 msgr2=0x7f990c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.762+0000 7f992ca00640 1 --2- 172.21.15.102:0/1575972494 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f990c0047c0 0x7f990c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.762+0000 7f992ca00640 1 -- 172.21.15.102:0/1575972494 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c003520 con 0x7f990c0039c0 2024-03-23T03:47:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.762+0000 7f992d400640 1 -- 172.21.15.102:0/1575972494 reap_dead start 2024-03-23T03:47:29.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.763+0000 7f992ca00640 1 --2- 172.21.15.102:0/1575972494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f990c0039c0 0x7f990c003de0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9910007fc0 tx=0x7f990c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15340afcc2aafb89 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.763+0000 7f9927400640 1 -- 172.21.15.102:0/1575972494 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9910005ad0 con 0x7f990c0039c0 2024-03-23T03:47:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.763+0000 7f9927400640 1 -- 172.21.15.102:0/1575972494 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9910005c50 con 0x7f990c0039c0 2024-03-23T03:47:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.763+0000 7f9927400640 1 -- 172.21.15.102:0/1575972494 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991002c7c0 con 0x7f990c0039c0 2024-03-23T03:47:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.764+0000 7f992f400640 1 -- 172.21.15.102:0/1575972494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f990c0039c0 msgr2=0x7f990c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.764+0000 7f992f400640 1 --2- 172.21.15.102:0/1575972494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f990c0039c0 0x7f990c003de0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9910007fc0 tx=0x7f990c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.764+0000 7f992f400640 1 -- 172.21.15.102:0/1575972494 shutdown_connections 2024-03-23T03:47:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.764+0000 7f992f400640 1 --2- 172.21.15.102:0/1575972494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f990c0039c0 0x7f990c003de0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.764+0000 7f992f400640 1 -- 172.21.15.102:0/1575972494 >> 172.21.15.102:0/1575972494 conn(0x7f992806a0b0 msgr2=0x7f992806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.764+0000 7f992f400640 1 -- 172.21.15.102:0/1575972494 shutdown_connections 2024-03-23T03:47:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.764+0000 7f992f400640 1 -- 172.21.15.102:0/1575972494 wait complete. 2024-03-23T03:47:29.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.766+0000 7f992f400640 1 Processor -- start 2024-03-23T03:47:29.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.766+0000 7f992f400640 1 -- start start 2024-03-23T03:47:29.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.767+0000 7f992f400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f990c0039c0 0x7f9928136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.767+0000 7f992f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99280b2b50 0x7f99281373b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.767+0000 7f992f400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9928139e50 0x7f992813a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.767+0000 7f992f400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99281378f0 con 0x7f99280b2b50 2024-03-23T03:47:29.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.767+0000 7f992f400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9928138240 con 0x7f990c0039c0 2024-03-23T03:47:29.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.767+0000 7f992f400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99280b3110 con 0x7f9928139e50 2024-03-23T03:47:29.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.767+0000 7f9927e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99280b2b50 0x7f99281373b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.767+0000 7f9927e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99280b2b50 0x7f99281373b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33366/0 (socket says 172.21.15.102:33366) 2024-03-23T03:47:29.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.767+0000 7f9927e00640 1 -- 172.21.15.102:0/4025324692 learned_addr learned my addr 172.21.15.102:0/4025324692 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:29.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.767+0000 7f992ca00640 1 --2- 172.21.15.102:0/4025324692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f990c0039c0 0x7f9928136e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.767+0000 7f992d400640 1 --2- 172.21.15.102:0/4025324692 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9928139e50 0x7f992813a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.768+0000 7f9927e00640 1 -- 172.21.15.102:0/4025324692 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9928139e50 msgr2=0x7f992813a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:29.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.768+0000 7f9927e00640 1 --2- 172.21.15.102:0/4025324692 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9928139e50 0x7f992813a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.768+0000 7f9927e00640 1 -- 172.21.15.102:0/4025324692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f990c0039c0 msgr2=0x7f9928136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:29.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.768+0000 7f9927e00640 1 --2- 172.21.15.102:0/4025324692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f990c0039c0 0x7f9928136e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.768+0000 7f9927e00640 1 -- 172.21.15.102:0/4025324692 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9928155280 con 0x7f99280b2b50 2024-03-23T03:47:29.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.768+0000 7f9927e00640 1 --2- 172.21.15.102:0/4025324692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99280b2b50 0x7f99281373b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f991800e920 tx=0x7f991800ede0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:29.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.768+0000 7f9924c00640 1 -- 172.21.15.102:0/4025324692 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9918003b80 con 0x7f99280b2b50 2024-03-23T03:47:29.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.768+0000 7f9924c00640 1 -- 172.21.15.102:0/4025324692 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9918003ce0 con 0x7f99280b2b50 2024-03-23T03:47:29.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.768+0000 7f9924c00640 1 -- 172.21.15.102:0/4025324692 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99180052f0 con 0x7f99280b2b50 2024-03-23T03:47:29.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.768+0000 7f992f400640 1 -- 172.21.15.102:0/4025324692 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9928137bf0 con 0x7f99280b2b50 2024-03-23T03:47:29.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.768+0000 7f992f400640 1 -- 172.21.15.102:0/4025324692 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99280b33f0 con 0x7f99280b2b50 2024-03-23T03:47:29.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.771+0000 7f9924c00640 1 -- 172.21.15.102:0/4025324692 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9918005450 con 0x7f99280b2b50 2024-03-23T03:47:29.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.772+0000 7f9924c00640 1 --2- 172.21.15.102:0/4025324692 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f990406ac00 0x7f990406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.772+0000 7f9924c00640 1 -- 172.21.15.102:0/4025324692 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9918089720 con 0x7f99280b2b50 2024-03-23T03:47:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.772+0000 7f992ca00640 1 --2- 172.21.15.102:0/4025324692 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f990406ac00 0x7f990406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.773+0000 7f9931f99740 1 Processor -- start 2024-03-23T03:47:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.773+0000 7f992ca00640 1 --2- 172.21.15.102:0/4025324692 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f990406ac00 0x7f990406d0c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9910034000 tx=0x7f9910009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.773+0000 7f9931f99740 1 -- start start 2024-03-23T03:47:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.774+0000 7f9931f99740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06ed00 0x55bdfe041ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.774+0000 7f9931f99740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06f0e0 0x55bdfe071a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.774+0000 7f9931f99740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe0421e0 0x55bdfe071fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.774+0000 7f9931f99740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdfe01d2c0 con 0x55bdfe06ed00 2024-03-23T03:47:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.774+0000 7f9931f99740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdfdfb28f0 con 0x55bdfe06f0e0 2024-03-23T03:47:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.774+0000 7f9931f99740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bdfdfb0080 con 0x55bdfe0421e0 2024-03-23T03:47:29.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.774+0000 7f992ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06ed00 0x55bdfe041ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.774+0000 7f992ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06ed00 0x55bdfe041ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33372/0 (socket says 172.21.15.102:33372) 2024-03-23T03:47:29.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.774+0000 7f992ca00640 1 -- 172.21.15.102:0/4005200167 learned_addr learned my addr 172.21.15.102:0/4005200167 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:29.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.774+0000 7f9927e00640 1 --2- 172.21.15.102:0/4005200167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06f0e0 0x55bdfe071a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.774+0000 7f992d400640 1 --2- 172.21.15.102:0/4005200167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe0421e0 0x55bdfe071fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.775+0000 7f992ca00640 1 -- 172.21.15.102:0/4005200167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe0421e0 msgr2=0x55bdfe071fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:29.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.775+0000 7f992ca00640 1 --2- 172.21.15.102:0/4005200167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe0421e0 0x55bdfe071fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.775+0000 7f992ca00640 1 -- 172.21.15.102:0/4005200167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06f0e0 msgr2=0x55bdfe071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:29.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.775+0000 7f992ca00640 1 --2- 172.21.15.102:0/4005200167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06f0e0 0x55bdfe071a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.775+0000 7f992ca00640 1 -- 172.21.15.102:0/4005200167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdfdf76a10 con 0x55bdfe06ed00 2024-03-23T03:47:29.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.775+0000 7f9927e00640 1 --2- 172.21.15.102:0/4005200167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06f0e0 0x55bdfe071a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:29.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.775+0000 7f992ca00640 1 --2- 172.21.15.102:0/4005200167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06ed00 0x55bdfe041ca0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f991002c600 tx=0x7f9910007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea2e3923589d711c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:29.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.775+0000 7f991d600640 1 -- 172.21.15.102:0/4005200167 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991003c040 con 0x55bdfe06ed00 2024-03-23T03:47:29.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.775+0000 7f991d600640 1 -- 172.21.15.102:0/4005200167 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9910001600 con 0x55bdfe06ed00 2024-03-23T03:47:29.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.775+0000 7f991d600640 1 -- 172.21.15.102:0/4005200167 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9910004e70 con 0x55bdfe06ed00 2024-03-23T03:47:29.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.776+0000 7f9931f99740 1 -- 172.21.15.102:0/4005200167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06ed00 msgr2=0x55bdfe041ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:29.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.776+0000 7f9931f99740 1 --2- 172.21.15.102:0/4005200167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06ed00 0x55bdfe041ca0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f991002c600 tx=0x7f9910007000 comp rx=0 tx=0).stop 2024-03-23T03:47:29.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.776+0000 7f9931f99740 1 -- 172.21.15.102:0/4005200167 shutdown_connections 2024-03-23T03:47:29.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.776+0000 7f9931f99740 1 --2- 172.21.15.102:0/4005200167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe0421e0 0x55bdfe071fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.776+0000 7f9931f99740 1 --2- 172.21.15.102:0/4005200167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06f0e0 0x55bdfe071a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.776+0000 7f9931f99740 1 --2- 172.21.15.102:0/4005200167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06ed00 0x55bdfe041ca0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.776+0000 7f9931f99740 1 -- 172.21.15.102:0/4005200167 >> 172.21.15.102:0/4005200167 conn(0x55bdfe043f10 msgr2=0x55bdfe08ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:29.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.776+0000 7f9931f99740 1 -- 172.21.15.102:0/4005200167 shutdown_connections 2024-03-23T03:47:29.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.776+0000 7f9931f99740 1 -- 172.21.15.102:0/4005200167 wait complete. 2024-03-23T03:47:29.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.777+0000 7f9931f99740 1 -- start start 2024-03-23T03:47:29.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.777+0000 7f9931f99740 1 Processor -- start 2024-03-23T03:47:29.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.778+0000 7f9931f99740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06ed00 0x55bdfe05e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.778+0000 7f9931f99740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe06f0e0 0x55bdfe08e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.778+0000 7f9931f99740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06e240 0x55bdfe08eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.778+0000 7f9931f99740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdfdfab970 con 0x55bdfe06e240 2024-03-23T03:47:29.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.778+0000 7f9931f99740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdfdf4d0a0 con 0x55bdfe06ed00 2024-03-23T03:47:29.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.778+0000 7f9931f99740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bdfdf4cd40 con 0x55bdfe06f0e0 2024-03-23T03:47:29.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.778+0000 7f9931f99740 10 client.0 _ll_register_callbacks cb 0x55bdfdfa4fe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T03:47:29.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.778+0000 7f992d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06e240 0x55bdfe08eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.778+0000 7f992d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06e240 0x55bdfe08eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33388/0 (socket says 172.21.15.102:33388) 2024-03-23T03:47:29.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.778+0000 7f992d400640 1 -- 172.21.15.102:0/4221946161 learned_addr learned my addr 172.21.15.102:0/4221946161 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:29.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.778+0000 7f992ca00640 1 --2- 172.21.15.102:0/4221946161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06ed00 0x55bdfe05e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.778+0000 7f9927e00640 1 --2- 172.21.15.102:0/4221946161 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe06f0e0 0x55bdfe08e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.778+0000 7f992d400640 1 -- 172.21.15.102:0/4221946161 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe06f0e0 msgr2=0x55bdfe08e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:29.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.779+0000 7f992d400640 1 --2- 172.21.15.102:0/4221946161 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe06f0e0 0x55bdfe08e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.779+0000 7f992d400640 1 -- 172.21.15.102:0/4221946161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06ed00 msgr2=0x55bdfe05e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.779+0000 7f992d400640 1 --2- 172.21.15.102:0/4221946161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06ed00 0x55bdfe05e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.779+0000 7f992d400640 1 -- 172.21.15.102:0/4221946161 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f990c003520 con 0x55bdfe06e240 2024-03-23T03:47:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.779+0000 7f9927e00640 1 --2- 172.21.15.102:0/4221946161 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe06f0e0 0x55bdfe08e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.779+0000 7f992ca00640 1 --2- 172.21.15.102:0/4221946161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06ed00 0x55bdfe05e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.779+0000 7f992d400640 1 --2- 172.21.15.102:0/4221946161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06e240 0x55bdfe08eca0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f99200122f0 tx=0x7f99200127b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77d997fefde3c1e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.779+0000 7f9916000640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-23T03:47:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.779+0000 7f9916000640 1 -- 172.21.15.102:0/4221946161 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9920015860 con 0x55bdfe06e240 2024-03-23T03:47:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.779+0000 7f9916000640 1 -- 172.21.15.102:0/4221946161 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f99200159c0 con 0x55bdfe06e240 2024-03-23T03:47:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.779+0000 7f9931f99740 1 -- 172.21.15.102:0/4221946161 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bdfdfd5b10 con 0x55bdfe06e240 2024-03-23T03:47:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.779+0000 7f9916000640 1 -- 172.21.15.102:0/4221946161 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9920012ea0 con 0x55bdfe06e240 2024-03-23T03:47:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.779+0000 7f9931f99740 10 client.4803 fetch_fsmap learned FSMap version 8 2024-03-23T03:47:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.779+0000 7f9931f99740 1 -- 172.21.15.102:0/4221946161 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55bdfdfa9ff0 con 0x55bdfe06e240 2024-03-23T03:47:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.780+0000 7f9916000640 1 -- 172.21.15.102:0/4221946161 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bdfdfd5b10 con 0x55bdfe06e240 2024-03-23T03:47:29.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.780+0000 7f9916000640 1 -- 172.21.15.102:0/4221946161 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 8) v1 ==== 3122+0+0 (secure 0 0 0) 0x7f9920015b80 con 0x55bdfe06e240 2024-03-23T03:47:29.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.780+0000 7f9931f99740 10 client.4803 fetch_fsmap finished waiting for FSMap version 8 2024-03-23T03:47:29.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.780+0000 7f9931f99740 10 client.4803 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-23T03:47:29.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.780+0000 7f9931f99740 20 client.4803 populate_metadata read hostname 'smithi102' 2024-03-23T03:47:29.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.780+0000 7f9931f99740 1 --2- 172.21.15.102:0/4221946161 >> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] conn(0x55bdfe08c670 0x55bdfe08ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.780+0000 7f9931f99740 4 client.4803 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T03:47:29.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.780+0000 7f9931f99740 1 -- 172.21.15.102:0/4221946161 --> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bdfdc9f3c0 con 0x55bdfe08c670 2024-03-23T03:47:29.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.780+0000 7f992ca00640 1 --2- 172.21.15.102:0/4221946161 >> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] conn(0x55bdfe08c670 0x55bdfe08ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.781+0000 7f992ca00640 1 --2- 172.21.15.102:0/4221946161 >> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] conn(0x55bdfe08c670 0x55bdfe08ca50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcd7b4ec77f220c2 server_cookie=b531c7fe59bdd21e in_seq=0 out_seq=0 2024-03-23T03:47:29.785 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.781+0000 7f9916000640 10 client.4803 ms_handle_connect on v2:172.21.15.173:6832/1462637509 2024-03-23T03:47:29.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.783+0000 7f9916000640 1 -- 172.21.15.102:0/4221946161 <== mds.0 v2:172.21.15.173:6832/1462637509 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55bdfdc9f3c0 con 0x55bdfe08c670 2024-03-23T03:47:29.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.783+0000 7f9916000640 10 client.4803 handle_command_reply: tid=0 2024-03-23T03:47:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.783+0000 7f9931f99740 1 client.4803 shutdown 2024-03-23T03:47:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.783+0000 7f9931f99740 1 -- 172.21.15.102:0/4221946161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06e240 msgr2=0x55bdfe08eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.783+0000 7f9931f99740 1 --2- 172.21.15.102:0/4221946161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06e240 0x55bdfe08eca0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f99200122f0 tx=0x7f99200127b0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.784+0000 7f9931f99740 1 -- 172.21.15.102:0/4221946161 shutdown_connections 2024-03-23T03:47:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.784+0000 7f9931f99740 1 --2- 172.21.15.102:0/4221946161 >> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] conn(0x55bdfe08c670 0x55bdfe08ca50 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.784+0000 7f9931f99740 1 --2- 172.21.15.102:0/4221946161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06e240 0x55bdfe08eca0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.784+0000 7f9931f99740 1 --2- 172.21.15.102:0/4221946161 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe06f0e0 0x55bdfe08e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.784+0000 7f9916000640 0 client.4803 ms_handle_reset on v2:172.21.15.173:6832/1462637509 2024-03-23T03:47:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.784+0000 7f9931f99740 1 --2- 172.21.15.102:0/4221946161 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06ed00 0x55bdfe05e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.784+0000 7f9931f99740 1 -- 172.21.15.102:0/4221946161 >> 172.21.15.102:0/4221946161 conn(0x55bdfe0421e0 msgr2=0x55bdfe045cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.784+0000 7f9931f99740 1 -- 172.21.15.102:0/4221946161 shutdown_connections 2024-03-23T03:47:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.784+0000 7f9931f99740 1 -- 172.21.15.102:0/4221946161 wait complete. 2024-03-23T03:47:29.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.784+0000 7f9931f99740 20 client.4803 trim_cache size 0 max 16384 2024-03-23T03:47:29.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.800+0000 7f9931f99740 1 Processor -- start 2024-03-23T03:47:29.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.800+0000 7f9931f99740 1 -- start start 2024-03-23T03:47:29.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f9931f99740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06e240 0x55bdfe03d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f9931f99740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06ed00 0x55bdfe03d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f9931f99740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe06f0e0 0x55bdfe05ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f9931f99740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdfdd12f20 con 0x55bdfe06ed00 2024-03-23T03:47:29.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f9931f99740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdfe06e0e0 con 0x55bdfe06e240 2024-03-23T03:47:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f9931f99740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bdfe06f850 con 0x55bdfe06f0e0 2024-03-23T03:47:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f9927e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06ed00 0x55bdfe03d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f9927e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06ed00 0x55bdfe03d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33390/0 (socket says 172.21.15.102:33390) 2024-03-23T03:47:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f9927e00640 1 -- 172.21.15.102:0/2063366905 learned_addr learned my addr 172.21.15.102:0/2063366905 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f992d400640 1 --2- 172.21.15.102:0/2063366905 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe06f0e0 0x55bdfe05ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f992ca00640 1 --2- 172.21.15.102:0/2063366905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06e240 0x55bdfe03d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f9927e00640 1 -- 172.21.15.102:0/2063366905 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe06f0e0 msgr2=0x55bdfe05ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f9927e00640 1 --2- 172.21.15.102:0/2063366905 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe06f0e0 0x55bdfe05ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f9927e00640 1 -- 172.21.15.102:0/2063366905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06e240 msgr2=0x55bdfe03d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f9927e00640 1 --2- 172.21.15.102:0/2063366905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06e240 0x55bdfe03d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f9927e00640 1 -- 172.21.15.102:0/2063366905 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdfdfdf7c0 con 0x55bdfe06ed00 2024-03-23T03:47:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f992d400640 1 --2- 172.21.15.102:0/2063366905 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe06f0e0 0x55bdfe05ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:29.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.801+0000 7f992ca00640 1 --2- 172.21.15.102:0/2063366905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06e240 0x55bdfe03d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.802+0000 7f9927e00640 1 --2- 172.21.15.102:0/2063366905 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06ed00 0x55bdfe03d970 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9918058250 tx=0x7f991801a310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d671ec8649137ec server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.802+0000 7f991e000640 1 -- 172.21.15.102:0/2063366905 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991805c0f0 con 0x55bdfe06ed00 2024-03-23T03:47:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.802+0000 7f991e000640 1 -- 172.21.15.102:0/2063366905 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99180606b0 con 0x55bdfe06ed00 2024-03-23T03:47:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.802+0000 7f991e000640 1 -- 172.21.15.102:0/2063366905 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9918005700 con 0x55bdfe06ed00 2024-03-23T03:47:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.802+0000 7f9931f99740 1 -- 172.21.15.102:0/2063366905 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06ed00 msgr2=0x55bdfe03d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.802+0000 7f9931f99740 1 --2- 172.21.15.102:0/2063366905 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06ed00 0x55bdfe03d970 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9918058250 tx=0x7f991801a310 comp rx=0 tx=0).stop 2024-03-23T03:47:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.803+0000 7f9931f99740 1 -- 172.21.15.102:0/2063366905 shutdown_connections 2024-03-23T03:47:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.803+0000 7f9931f99740 1 --2- 172.21.15.102:0/2063366905 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe06f0e0 0x55bdfe05ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.803+0000 7f9931f99740 1 --2- 172.21.15.102:0/2063366905 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06ed00 0x55bdfe03d970 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.803+0000 7f9931f99740 1 --2- 172.21.15.102:0/2063366905 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06e240 0x55bdfe03d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.803+0000 7f9931f99740 1 -- 172.21.15.102:0/2063366905 >> 172.21.15.102:0/2063366905 conn(0x55bdfe0421e0 msgr2=0x55bdfe09b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.803+0000 7f9931f99740 1 -- 172.21.15.102:0/2063366905 shutdown_connections 2024-03-23T03:47:29.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.803+0000 7f9931f99740 1 -- 172.21.15.102:0/2063366905 wait complete. 2024-03-23T03:47:29.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.803+0000 7f9931f99740 1 -- start start 2024-03-23T03:47:29.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.803+0000 7f9931f99740 1 Processor -- start 2024-03-23T03:47:29.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.804+0000 7f9931f99740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06e240 0x55bdfe0fad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.804+0000 7f9931f99740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06ed00 0x55bdfe072830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.804+0000 7f9931f99740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe06f0e0 0x55bdfe072d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.804+0000 7f9931f99740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdfe08ec70 con 0x55bdfe06e240 2024-03-23T03:47:29.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.804+0000 7f9931f99740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdfe043ac0 con 0x55bdfe06ed00 2024-03-23T03:47:29.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.804+0000 7f9931f99740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bdfe0fb240 con 0x55bdfe06f0e0 2024-03-23T03:47:29.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.804+0000 7f9931f99740 10 client.0 _ll_register_callbacks cb 0x7f9920009ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T03:47:29.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.805+0000 7f992ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06e240 0x55bdfe0fad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.805+0000 7f992ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06e240 0x55bdfe0fad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33404/0 (socket says 172.21.15.102:33404) 2024-03-23T03:47:29.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.805+0000 7f992ca00640 1 -- 172.21.15.102:0/1556955030 learned_addr learned my addr 172.21.15.102:0/1556955030 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:29.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.805+0000 7f992ca00640 1 -- 172.21.15.102:0/1556955030 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe06f0e0 msgr2=0x55bdfe072d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:29.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.805+0000 7f992ca00640 1 --2- 172.21.15.102:0/1556955030 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe06f0e0 0x55bdfe072d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.805+0000 7f992ca00640 1 -- 172.21.15.102:0/1556955030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06ed00 msgr2=0x55bdfe072830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:29.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.805+0000 7f992ca00640 1 --2- 172.21.15.102:0/1556955030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06ed00 0x55bdfe072830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.805+0000 7f992ca00640 1 -- 172.21.15.102:0/1556955030 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99280b33f0 con 0x55bdfe06e240 2024-03-23T03:47:29.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.805+0000 7f992ca00640 1 --2- 172.21.15.102:0/1556955030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06e240 0x55bdfe0fad00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f991004d040 tx=0x7f991002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d96755aab8bedb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:29.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.805+0000 7f9916000640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-23T03:47:29.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.805+0000 7f9916000640 1 -- 172.21.15.102:0/1556955030 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991003c040 con 0x55bdfe06e240 2024-03-23T03:47:29.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.805+0000 7f9916000640 1 -- 172.21.15.102:0/1556955030 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9910058370 con 0x55bdfe06e240 2024-03-23T03:47:29.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.805+0000 7f9931f99740 1 -- 172.21.15.102:0/1556955030 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9910001600 con 0x55bdfe06e240 2024-03-23T03:47:29.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.806+0000 7f9916000640 1 -- 172.21.15.102:0/1556955030 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9910058960 con 0x55bdfe06e240 2024-03-23T03:47:29.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.806+0000 7f9931f99740 10 client.4809 fetch_fsmap learned FSMap version 8 2024-03-23T03:47:29.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.806+0000 7f9931f99740 1 -- 172.21.15.102:0/1556955030 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x55bdfdf4c2b0 con 0x55bdfe06e240 2024-03-23T03:47:29.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.806+0000 7f9916000640 1 -- 172.21.15.102:0/1556955030 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9910001600 con 0x55bdfe06e240 2024-03-23T03:47:29.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.806+0000 7f9916000640 1 -- 172.21.15.102:0/1556955030 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 8) v1 ==== 3122+0+0 (secure 0 0 0) 0x7f991003f070 con 0x55bdfe06e240 2024-03-23T03:47:29.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.806+0000 7f9931f99740 10 client.4809 fetch_fsmap finished waiting for FSMap version 8 2024-03-23T03:47:29.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.806+0000 7f9931f99740 10 client.4809 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-23T03:47:29.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.806+0000 7f9931f99740 20 client.4809 populate_metadata read hostname 'smithi102' 2024-03-23T03:47:29.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.806+0000 7f9931f99740 1 --2- 172.21.15.102:0/1556955030 >> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] conn(0x55bdfe05d490 0x55bdfe05d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:29.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.806+0000 7f9931f99740 4 client.4809 mds_command: new command op to 4270 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-23T03:47:29.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.806+0000 7f9931f99740 1 -- 172.21.15.102:0/1556955030 --> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bdfdcb06c0 con 0x55bdfe05d490 2024-03-23T03:47:29.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.807+0000 7f9927e00640 1 --2- 172.21.15.102:0/1556955030 >> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] conn(0x55bdfe05d490 0x55bdfe05d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:29.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.808+0000 7f9927e00640 1 --2- 172.21.15.102:0/1556955030 >> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] conn(0x55bdfe05d490 0x55bdfe05d870 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ae0d252909ec9a2 server_cookie=7c470ff7c9b84d69 in_seq=0 out_seq=0 2024-03-23T03:47:29.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.808+0000 7f9916000640 10 client.4809 ms_handle_connect on v2:172.21.15.173:6832/1462637509 2024-03-23T03:47:29.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.808+0000 7f9916000640 1 -- 172.21.15.102:0/1556955030 <== mds.0 v2:172.21.15.173:6832/1462637509 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x55bdfdcb06c0 con 0x55bdfe05d490 2024-03-23T03:47:29.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.808+0000 7f9916000640 10 client.4809 handle_command_reply: tid=0 2024-03-23T03:47:29.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.808+0000 7f9931f99740 1 client.4809 shutdown 2024-03-23T03:47:29.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.808+0000 7f9931f99740 1 -- 172.21.15.102:0/1556955030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06e240 msgr2=0x55bdfe0fad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:29.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.808+0000 7f9931f99740 1 --2- 172.21.15.102:0/1556955030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06e240 0x55bdfe0fad00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f991004d040 tx=0x7f991002e070 comp rx=0 tx=0).stop 2024-03-23T03:47:29.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.808+0000 7f9931f99740 1 -- 172.21.15.102:0/1556955030 shutdown_connections 2024-03-23T03:47:29.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.808+0000 7f9931f99740 1 --2- 172.21.15.102:0/1556955030 >> [v2:172.21.15.173:6832/1462637509,v1:172.21.15.173:6834/1462637509] conn(0x55bdfe05d490 0x55bdfe05d870 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.808+0000 7f9931f99740 1 --2- 172.21.15.102:0/1556955030 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bdfe06f0e0 0x55bdfe072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.808+0000 7f9931f99740 1 --2- 172.21.15.102:0/1556955030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bdfe06ed00 0x55bdfe072830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.808+0000 7f9916000640 0 client.4809 ms_handle_reset on v2:172.21.15.173:6832/1462637509 2024-03-23T03:47:29.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.808+0000 7f9931f99740 1 --2- 172.21.15.102:0/1556955030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdfe06e240 0x55bdfe0fad00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.808+0000 7f9931f99740 1 -- 172.21.15.102:0/1556955030 >> 172.21.15.102:0/1556955030 conn(0x55bdfe0421e0 msgr2=0x55bdfe071a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:29.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.809+0000 7f9931f99740 1 -- 172.21.15.102:0/1556955030 shutdown_connections 2024-03-23T03:47:29.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.809+0000 7f9931f99740 1 -- 172.21.15.102:0/1556955030 wait complete. 2024-03-23T03:47:29.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.809+0000 7f9931f99740 20 client.4809 trim_cache size 0 max 16384 2024-03-23T03:47:29.813 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:47:29.813 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "776b2820-05f2-4e50-8d40-18dd76c5f6a0", 2024-03-23T03:47:29.813 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 0, 2024-03-23T03:47:29.813 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 4270, 2024-03-23T03:47:29.814 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-03-23T03:47:29.814 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-03-23T03:47:29.814 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-03-23T03:47:29.814 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 35.889834364000002, 2024-03-23T03:47:29.814 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 8, 2024-03-23T03:47:29.814 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 26, 2024-03-23T03:47:29.814 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 26, 2024-03-23T03:47:29.814 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 42.608570892000003 2024-03-23T03:47:29.814 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:47:29.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.809+0000 7f992f400640 1 -- 172.21.15.102:0/4025324692 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f990406ac00 msgr2=0x7f990406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:29.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.809+0000 7f992f400640 1 --2- 172.21.15.102:0/4025324692 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f990406ac00 0x7f990406d0c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9910034000 tx=0x7f9910009030 comp rx=0 tx=0).stop 2024-03-23T03:47:29.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.809+0000 7f992f400640 1 -- 172.21.15.102:0/4025324692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99280b2b50 msgr2=0x7f99281373b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:29.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.809+0000 7f992f400640 1 --2- 172.21.15.102:0/4025324692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99280b2b50 0x7f99281373b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f991800e920 tx=0x7f991800ede0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.809+0000 7f992f400640 1 -- 172.21.15.102:0/4025324692 shutdown_connections 2024-03-23T03:47:29.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.809+0000 7f992f400640 1 --2- 172.21.15.102:0/4025324692 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f990406ac00 0x7f990406d0c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.809+0000 7f992f400640 1 --2- 172.21.15.102:0/4025324692 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9928139e50 0x7f992813a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.809+0000 7f992f400640 1 --2- 172.21.15.102:0/4025324692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99280b2b50 0x7f99281373b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.809+0000 7f992f400640 1 --2- 172.21.15.102:0/4025324692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f990c0039c0 0x7f9928136e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:29.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.809+0000 7f992f400640 1 -- 172.21.15.102:0/4025324692 >> 172.21.15.102:0/4025324692 conn(0x7f992806a0b0 msgr2=0x7f992815d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:29.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.809+0000 7f992f400640 1 -- 172.21.15.102:0/4025324692 shutdown_connections 2024-03-23T03:47:29.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:29.810+0000 7f992f400640 1 -- 172.21.15.102:0/4025324692 wait complete. 2024-03-23T03:47:29.826 INFO:teuthology.run_tasks:Running task kclient... 2024-03-23T03:47:29.839 INFO:tasks.kclient:Mounting kernel clients... 2024-03-23T03:47:29.840 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-23T03:47:29.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:29.840 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-23T03:47:29.880 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-23T03:47:29.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:29.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-23T03:47:30.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:30.020 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link delete ceph-brx 2024-03-23T03:47:30.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:30.131 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-23T03:47:30.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:30.170 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link delete ceph-brx 2024-03-23T03:47:30.299 INFO:teuthology.orchestra.run.smithi102.stderr:Cannot find device "ceph-brx" 2024-03-23T03:47:30.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:47:30.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:30.304 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list 2024-03-23T03:47:30.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:30.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link delete ceph-brx 2024-03-23T03:47:30.463 INFO:teuthology.orchestra.run.smithi173.stderr:Cannot find device "ceph-brx" 2024-03-23T03:47:30.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:47:30.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:30.468 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list 2024-03-23T03:47:30.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:30.523 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link delete ceph-brx 2024-03-23T03:47:30.625 INFO:teuthology.orchestra.run.smithi173.stderr:Cannot find device "ceph-brx" 2024-03-23T03:47:30.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:47:30.630 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-03-23T03:47:30.630 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-23T03:47:30.630 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T03:47:30.631 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:47:30.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.841+0000 7fbb1f200640 1 Processor -- start 2024-03-23T03:47:30.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.841+0000 7fbb1f200640 1 -- start start 2024-03-23T03:47:30.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.842+0000 7fbb1f200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbb1813faa0 0x7fbb18160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:30.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.842+0000 7fbb1f200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb18160800 0x7fbb180b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:30.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.842+0000 7fbb1f200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb180b2030 con 0x7fbb180b1be0 2024-03-23T03:47:30.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.842+0000 7fbb1f200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb180b38a0 con 0x7fbb1813faa0 2024-03-23T03:47:30.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.842+0000 7fbb1f200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb180b3b80 con 0x7fbb18160800 2024-03-23T03:47:30.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.843+0000 7fbb17e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbb1813faa0 0x7fbb18160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:30.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.843+0000 7fbb17e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbb1813faa0 0x7fbb18160f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33410/0 (socket says 172.21.15.102:33410) 2024-03-23T03:47:30.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.843+0000 7fbb17e00640 1 -- 172.21.15.102:0/933423566 learned_addr learned my addr 172.21.15.102:0/933423566 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:30.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.843+0000 7fbb0f400640 1 --2- 172.21.15.102:0/933423566 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb18160800 0x7fbb180b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:30.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.843+0000 7fbb17e00640 1 -- 172.21.15.102:0/933423566 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb18160800 msgr2=0x7fbb180b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:30.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.843+0000 7fbb17e00640 1 --2- 172.21.15.102:0/933423566 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb18160800 0x7fbb180b16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:30.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.843+0000 7fbb17e00640 1 -- 172.21.15.102:0/933423566 >> v1:172.21.15.173:6789/0 conn(0x7fbb180b1be0 legacy=0x7fbb181422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:30.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.843+0000 7fbb17e00640 1 -- 172.21.15.102:0/933423566 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb180b3e60 con 0x7fbb1813faa0 2024-03-23T03:47:30.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.843+0000 7fbb0f400640 1 --2- 172.21.15.102:0/933423566 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb18160800 0x7fbb180b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:30.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.844+0000 7fbb17e00640 1 --2- 172.21.15.102:0/933423566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb1813faa0 0x7fbb18160f30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbb08009870 tx=0x7fbb0802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c0548d0ade4759 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:30.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.844+0000 7fbb17400640 1 -- 172.21.15.102:0/933423566 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08031640 con 0x7fbb1813faa0 2024-03-23T03:47:30.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.844+0000 7fbb17400640 1 -- 172.21.15.102:0/933423566 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb080317a0 con 0x7fbb1813faa0 2024-03-23T03:47:30.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.845+0000 7fbb17400640 1 -- 172.21.15.102:0/933423566 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08031b30 con 0x7fbb1813faa0 2024-03-23T03:47:30.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.845+0000 7fbb1f200640 1 -- 172.21.15.102:0/933423566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb1813faa0 msgr2=0x7fbb18160f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:30.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.845+0000 7fbb1f200640 1 --2- 172.21.15.102:0/933423566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb1813faa0 0x7fbb18160f30 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbb08009870 tx=0x7fbb0802b8f0 comp rx=0 tx=0).stop 2024-03-23T03:47:30.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.845+0000 7fbb1f200640 1 -- 172.21.15.102:0/933423566 shutdown_connections 2024-03-23T03:47:30.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.845+0000 7fbb1f200640 1 --2- 172.21.15.102:0/933423566 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb18160800 0x7fbb180b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:30.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.845+0000 7fbb1f200640 1 --2- 172.21.15.102:0/933423566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb1813faa0 0x7fbb18160f30 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:30.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.845+0000 7fbb1f200640 1 -- 172.21.15.102:0/933423566 >> 172.21.15.102:0/933423566 conn(0x7fbb1806a3d0 msgr2=0x7fbb180b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:30.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.845+0000 7fbb1f200640 1 -- 172.21.15.102:0/933423566 shutdown_connections 2024-03-23T03:47:30.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.845+0000 7fbb1f200640 1 -- 172.21.15.102:0/933423566 wait complete. 2024-03-23T03:47:30.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.847+0000 7fbb1f200640 1 Processor -- start 2024-03-23T03:47:30.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.847+0000 7fbb1f200640 1 -- start start 2024-03-23T03:47:30.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.848+0000 7fbb1f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb180b1be0 0x7fbb1813bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.848+0000 7fbb1f200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb1813faa0 0x7fbb18142ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.848+0000 7fbb1f200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb18160800 0x7fbb180ac900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.848+0000 7fbb1f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1813c4f0 con 0x7fbb180b1be0 2024-03-23T03:47:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.848+0000 7fbb1f200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb181f32f0 con 0x7fbb18160800 2024-03-23T03:47:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.848+0000 7fbb1f200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb181f34e0 con 0x7fbb1813faa0 2024-03-23T03:47:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.848+0000 7fbb17e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb180b1be0 0x7fbb1813bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.848+0000 7fbb17e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb180b1be0 0x7fbb1813bfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33414/0 (socket says 172.21.15.102:33414) 2024-03-23T03:47:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.848+0000 7fbb17e00640 1 -- 172.21.15.102:0/1141213546 learned_addr learned my addr 172.21.15.102:0/1141213546 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.849+0000 7fbb1d200640 1 --2- 172.21.15.102:0/1141213546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb18160800 0x7fbb180ac900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.849+0000 7fbb17e00640 1 -- 172.21.15.102:0/1141213546 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb1813faa0 msgr2=0x7fbb18142ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.849+0000 7fbb0f400640 1 --2- 172.21.15.102:0/1141213546 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb1813faa0 0x7fbb18142ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:30.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.849+0000 7fbb17e00640 1 --2- 172.21.15.102:0/1141213546 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb1813faa0 0x7fbb18142ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:30.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.849+0000 7fbb17e00640 1 -- 172.21.15.102:0/1141213546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb18160800 msgr2=0x7fbb180ac900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:30.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.849+0000 7fbb17e00640 1 --2- 172.21.15.102:0/1141213546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb18160800 0x7fbb180ac900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:30.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.849+0000 7fbb17e00640 1 -- 172.21.15.102:0/1141213546 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb080094d0 con 0x7fbb180b1be0 2024-03-23T03:47:30.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.849+0000 7fbb1d200640 1 --2- 172.21.15.102:0/1141213546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb18160800 0x7fbb180ac900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:30.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.849+0000 7fbb17e00640 1 --2- 172.21.15.102:0/1141213546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb180b1be0 0x7fbb1813bfb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbb08000c00 tx=0x7fbb080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:30.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.849+0000 7fbb14c00640 1 -- 172.21.15.102:0/1141213546 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0803ac60 con 0x7fbb180b1be0 2024-03-23T03:47:30.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.850+0000 7fbb14c00640 1 -- 172.21.15.102:0/1141213546 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0803ade0 con 0x7fbb180b1be0 2024-03-23T03:47:30.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.850+0000 7fbb14c00640 1 -- 172.21.15.102:0/1141213546 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb08044630 con 0x7fbb180b1be0 2024-03-23T03:47:30.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.850+0000 7fbb1f200640 1 -- 172.21.15.102:0/1141213546 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb181f2770 con 0x7fbb180b1be0 2024-03-23T03:47:30.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.850+0000 7fbb1f200640 1 -- 172.21.15.102:0/1141213546 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb181f2d30 con 0x7fbb180b1be0 2024-03-23T03:47:30.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.851+0000 7fbb1f200640 1 -- 172.21.15.102:0/1141213546 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb18073390 con 0x7fbb180b1be0 2024-03-23T03:47:30.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.852+0000 7fbb14c00640 1 -- 172.21.15.102:0/1141213546 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbb0802f050 con 0x7fbb180b1be0 2024-03-23T03:47:30.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.852+0000 7fbb14c00640 1 --2- 172.21.15.102:0/1141213546 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbaf406ac00 0x7fbaf406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:30.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.853+0000 7fbb14c00640 1 -- 172.21.15.102:0/1141213546 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbb080ab6e0 con 0x7fbb180b1be0 2024-03-23T03:47:30.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.853+0000 7fbb0f400640 1 --2- 172.21.15.102:0/1141213546 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbaf406ac00 0x7fbaf406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:30.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.853+0000 7fbb0f400640 1 --2- 172.21.15.102:0/1141213546 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbaf406ac00 0x7fbaf406d0c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbb180ad9c0 tx=0x7fbb00008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:30.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:30.857+0000 7fbb14c00640 1 -- 172.21.15.102:0/1141213546 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbb080765f0 con 0x7fbb180b1be0 2024-03-23T03:47:31.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:31.087+0000 7fbb1f200640 1 -- 172.21.15.102:0/1141213546 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbb180b2030 con 0x7fbb180b1be0 2024-03-23T03:47:31.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:31.088+0000 7fbb14c00640 1 -- 172.21.15.102:0/1141213546 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7fbb180b2030 con 0x7fbb180b1be0 2024-03-23T03:47:31.088 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-23T03:47:31.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:31.090+0000 7fbb1f200640 1 -- 172.21.15.102:0/1141213546 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbaf406ac00 msgr2=0x7fbaf406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:31.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:31.090+0000 7fbb1f200640 1 --2- 172.21.15.102:0/1141213546 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbaf406ac00 0x7fbaf406d0c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbb180ad9c0 tx=0x7fbb00008040 comp rx=0 tx=0).stop 2024-03-23T03:47:31.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:31.090+0000 7fbb1f200640 1 -- 172.21.15.102:0/1141213546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb180b1be0 msgr2=0x7fbb1813bfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:31.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:31.090+0000 7fbb1f200640 1 --2- 172.21.15.102:0/1141213546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb180b1be0 0x7fbb1813bfb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fbb08000c00 tx=0x7fbb080046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:31.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:31.090+0000 7fbb1f200640 1 -- 172.21.15.102:0/1141213546 shutdown_connections 2024-03-23T03:47:31.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:31.090+0000 7fbb1f200640 1 --2- 172.21.15.102:0/1141213546 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbaf406ac00 0x7fbaf406d0c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:31.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:31.090+0000 7fbb1f200640 1 --2- 172.21.15.102:0/1141213546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb18160800 0x7fbb180ac900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:31.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:31.090+0000 7fbb1f200640 1 --2- 172.21.15.102:0/1141213546 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb1813faa0 0x7fbb18142ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:31.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:31.090+0000 7fbb1f200640 1 --2- 172.21.15.102:0/1141213546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb180b1be0 0x7fbb1813bfb0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:31.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:31.090+0000 7fbb1f200640 1 -- 172.21.15.102:0/1141213546 >> 172.21.15.102:0/1141213546 conn(0x7fbb1806a3d0 msgr2=0x7fbb180aeb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:31.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:31.090+0000 7fbb1f200640 1 -- 172.21.15.102:0/1141213546 shutdown_connections 2024-03-23T03:47:31.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:31.090+0000 7fbb1f200640 1 -- 172.21.15.102:0/1141213546 wait complete. 2024-03-23T03:47:31.107 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T03:47:31.107 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:47:31.107 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2024-03-23T03:47:31.107 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T03:47:31.107 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T03:47:31.107 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T03:47:31.107 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T03:47:31.107 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:47:31.107 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:47:31.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:31.108 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2024-03-23T03:47:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:47:31.152 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:47:31.152 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:47:31.152 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:31.152 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-03-23T03:47:31.152 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:47:31.152 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:31.152 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f0 2024-03-23T03:47:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:47:31.152 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:31.152 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f1 2024-03-23T03:47:31.152 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:47:31.152 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:31.153 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:47:31.153 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:31.153 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:47:31.153 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 42619sec preferred_lft 42619sec 2024-03-23T03:47:31.153 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-03-23T03:47:31.153 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:31.153 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-23T03:47:31.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:31.153 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:47:31.153 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:47:31.153 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add name ceph-brx type bridge 2024-03-23T03:47:31.153 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr flush dev ceph-brx 2024-03-23T03:47:31.153 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set ceph-brx up 2024-03-23T03:47:31.154 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-23T03:47:31.154 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:47:31.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:31.414 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:47:31.485 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-23T03:47:31.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:31.489 DEBUG:teuthology.orchestra.run.smithi102:> route 2024-03-23T03:47:31.539 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2024-03-23T03:47:31.540 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:47:31.540 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:47:31.540 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:47:31.540 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:47:31.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:31.540 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:47:31.540 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:47:31.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:47:31.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:47:31.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:47:31.541 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:47:31.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:31.704 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-23T03:47:31.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:31.753 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list-id 2024-03-23T03:47:31.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:31.803 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:47:31.803 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:47:31.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T03:47:31.803 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-23T03:47:31.803 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:47:31.938 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-23T03:47:31.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:31.938 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:47:31.938 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:47:31.939 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-23T03:47:31.939 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-23T03:47:31.939 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-23T03:47:31.939 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-23T03:47:31.939 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-23T03:47:31.939 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:47:32.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:32.181 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:47:32.181 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:47:32.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.0 up 2024-03-23T03:47:32.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set dev brx.0 master ceph-brx 2024-03-23T03:47:32.182 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:47:32.329 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:47:32.329 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T03:47:32.374 INFO:teuthology.orchestra.run.smithi102.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:47:32.374 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:47:32.374 DEBUG:teuthology.orchestra.run.smithi102:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T03:47:32.465 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-23T03:47:32.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:32.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:47:32.584 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:47:32.584 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:47:32.584 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:47:33.205 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:47:33.205 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:47:33.205 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:47:33.205 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:47:33.206 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:47:33.237 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:47:33.237 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:47:33.237 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:47:33.237 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:47:33.237 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:47:33.237 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790" 2024-03-23T03:47:33.237 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/ 2024-03-23T03:47:33.238 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=776b2820-05f2-4e50-8d40-18dd76c5f6a0" will pass to kernel 2024-03-23T03:47:33.242 INFO:teuthology.orchestra.run.smithi102.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-23T03:47:33.242 INFO:teuthology.orchestra.run.smithi102.stdout: You just mounted a file system that supports labels which does not 2024-03-23T03:47:33.243 INFO:teuthology.orchestra.run.smithi102.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T03:47:33.243 INFO:teuthology.orchestra.run.smithi102.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T03:47:33.243 INFO:teuthology.orchestra.run.smithi102.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T03:47:33.246 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T03:47:33.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:33.247 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T03:47:33.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:33.324 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:47:33.374 INFO:teuthology.orchestra.run.smithi102.stdout:client4490 2024-03-23T03:47:33.375 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:47:33.375 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:47:33.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:33.421 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:47:33.513 INFO:teuthology.orchestra.run.smithi102.stdout:client4490 2024-03-23T03:47:33.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:33.514 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:47:33.558 INFO:teuthology.orchestra.run.smithi102.stdout:client4490 2024-03-23T03:47:33.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:33.558 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/776b2820-05f2-4e50-8d40-18dd76c5f6a0.client4490/status') 2024-03-23T03:47:33.681 INFO:teuthology.orchestra.run.smithi102.stdout:instance: client.4490 (3)172.21.15.102:0/320595563 2024-03-23T03:47:33.681 INFO:teuthology.orchestra.run.smithi102.stdout:blocklisted: false 2024-03-23T03:47:33.681 INFO:teuthology.orchestra.run.smithi102.stderr:0+1 records in 2024-03-23T03:47:33.681 INFO:teuthology.orchestra.run.smithi102.stderr:0+1 records out 2024-03-23T03:47:33.681 INFO:teuthology.orchestra.run.smithi102.stderr:70 bytes copied, 6.812e-05 s, 1.0 MB/s 2024-03-23T03:47:33.684 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2024-03-23T03:47:33.685 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-23T03:47:33.685 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-23T03:47:33.685 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:47:33.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.887+0000 7fd1b9600640 1 Processor -- start 2024-03-23T03:47:33.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.887+0000 7fd1b9600640 1 -- start start 2024-03-23T03:47:33.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.889+0000 7fd1b9600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd1b40c8de0 0x7fd1b40cd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:33.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.889+0000 7fd1b9600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd1b4069c90 0x7fd1b406a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:33.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.889+0000 7fd1b9600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b4160970 con 0x7fd1b415ca90 2024-03-23T03:47:33.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.889+0000 7fd1b9600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b406a620 con 0x7fd1b40c8de0 2024-03-23T03:47:33.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.889+0000 7fd1b9600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b406ab50 con 0x7fd1b4069c90 2024-03-23T03:47:33.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.889+0000 7fd1b3400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd1b40c8de0 0x7fd1b40cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:33.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.889+0000 7fd1b8c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd1b4069c90 0x7fd1b406a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:33.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.889+0000 7fd1b3400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd1b40c8de0 0x7fd1b40cd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:52080/0 (socket says 172.21.15.102:52080) 2024-03-23T03:47:33.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.889+0000 7fd1b3400640 1 -- 172.21.15.102:0/2558520673 learned_addr learned my addr 172.21.15.102:0/2558520673 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:33.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.890+0000 7fd1b3400640 1 -- 172.21.15.102:0/2558520673 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd1b4069c90 msgr2=0x7fd1b406a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:33.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.890+0000 7fd1b3400640 1 --2- 172.21.15.102:0/2558520673 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd1b4069c90 0x7fd1b406a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:33.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.890+0000 7fd1b3400640 1 -- 172.21.15.102:0/2558520673 >> v1:172.21.15.173:6789/0 conn(0x7fd1b415ca90 legacy=0x7fd1b40c86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:33.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.890+0000 7fd1b3400640 1 -- 172.21.15.102:0/2558520673 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b40cd8d0 con 0x7fd1b40c8de0 2024-03-23T03:47:33.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.892+0000 7fd1b3400640 1 --2- 172.21.15.102:0/2558520673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1b40c8de0 0x7fd1b40cd390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0009870 tx=0x7fd1a002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7a986b876f4b820 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:33.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.892+0000 7fd1b2a00640 1 -- 172.21.15.102:0/2558520673 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0030840 con 0x7fd1b40c8de0 2024-03-23T03:47:33.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.892+0000 7fd1b2a00640 1 -- 172.21.15.102:0/2558520673 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a00309a0 con 0x7fd1b40c8de0 2024-03-23T03:47:33.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.892+0000 7fd1b2a00640 1 -- 172.21.15.102:0/2558520673 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a0030d30 con 0x7fd1b40c8de0 2024-03-23T03:47:33.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.893+0000 7fd1b9600640 1 -- 172.21.15.102:0/2558520673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1b40c8de0 msgr2=0x7fd1b40cd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:33.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.893+0000 7fd1b9600640 1 --2- 172.21.15.102:0/2558520673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1b40c8de0 0x7fd1b40cd390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0009870 tx=0x7fd1a002b980 comp rx=0 tx=0).stop 2024-03-23T03:47:33.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.893+0000 7fd1b9600640 1 -- 172.21.15.102:0/2558520673 shutdown_connections 2024-03-23T03:47:33.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.893+0000 7fd1b9600640 1 --2- 172.21.15.102:0/2558520673 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd1b4069c90 0x7fd1b406a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:33.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.893+0000 7fd1b9600640 1 --2- 172.21.15.102:0/2558520673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1b40c8de0 0x7fd1b40cd390 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:33.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.893+0000 7fd1b9600640 1 -- 172.21.15.102:0/2558520673 >> 172.21.15.102:0/2558520673 conn(0x7fd1b4098a40 msgr2=0x7fd1b40b1050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:33.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.893+0000 7fd1b9600640 1 -- 172.21.15.102:0/2558520673 shutdown_connections 2024-03-23T03:47:33.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.893+0000 7fd1b9600640 1 -- 172.21.15.102:0/2558520673 wait complete. 2024-03-23T03:47:33.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.895+0000 7fd1b9600640 1 Processor -- start 2024-03-23T03:47:33.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.895+0000 7fd1b9600640 1 -- start start 2024-03-23T03:47:33.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.896+0000 7fd1b9600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b4069c90 0x7fd1b414b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:33.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.896+0000 7fd1b9600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd1b40c8de0 0x7fd1b40ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:33.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.896+0000 7fd1b9600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1b415ca90 0x7fd1b40ad730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:33.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.896+0000 7fd1b9600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b4157680 con 0x7fd1b4069c90 2024-03-23T03:47:33.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.896+0000 7fd1b9600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b4156d50 con 0x7fd1b415ca90 2024-03-23T03:47:33.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.896+0000 7fd1b9600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b4157030 con 0x7fd1b40c8de0 2024-03-23T03:47:33.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.896+0000 7fd1b3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b4069c90 0x7fd1b414b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:33.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.896+0000 7fd1b3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b4069c90 0x7fd1b414b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53742/0 (socket says 172.21.15.102:53742) 2024-03-23T03:47:33.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.896+0000 7fd1b3e00640 1 -- 172.21.15.102:0/2476391624 learned_addr learned my addr 172.21.15.102:0/2476391624 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:33.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.896+0000 7fd1b8c00640 1 --2- 172.21.15.102:0/2476391624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1b415ca90 0x7fd1b40ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:33.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.897+0000 7fd1b3400640 1 --2- 172.21.15.102:0/2476391624 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd1b40c8de0 0x7fd1b40ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:33.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.897+0000 7fd1b3e00640 1 -- 172.21.15.102:0/2476391624 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd1b40c8de0 msgr2=0x7fd1b40ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:33.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.897+0000 7fd1b3e00640 1 --2- 172.21.15.102:0/2476391624 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd1b40c8de0 0x7fd1b40ad1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:33.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.897+0000 7fd1b3e00640 1 -- 172.21.15.102:0/2476391624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1b415ca90 msgr2=0x7fd1b40ad730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:33.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.897+0000 7fd1b3e00640 1 --2- 172.21.15.102:0/2476391624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1b415ca90 0x7fd1b40ad730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:33.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.897+0000 7fd1b3e00640 1 -- 172.21.15.102:0/2476391624 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a00094d0 con 0x7fd1b4069c90 2024-03-23T03:47:33.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.897+0000 7fd1b3e00640 1 --2- 172.21.15.102:0/2476391624 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b4069c90 0x7fd1b414b520 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd1a400c3c0 tx=0x7fd1a400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:33.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.897+0000 7fd1abe00640 1 -- 172.21.15.102:0/2476391624 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a40052d0 con 0x7fd1b4069c90 2024-03-23T03:47:33.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.897+0000 7fd1abe00640 1 -- 172.21.15.102:0/2476391624 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a4005450 con 0x7fd1b4069c90 2024-03-23T03:47:33.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.897+0000 7fd1abe00640 1 -- 172.21.15.102:0/2476391624 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a4007260 con 0x7fd1b4069c90 2024-03-23T03:47:33.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.897+0000 7fd1b9600640 1 -- 172.21.15.102:0/2476391624 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b4157310 con 0x7fd1b4069c90 2024-03-23T03:47:33.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.898+0000 7fd1b9600640 1 -- 172.21.15.102:0/2476391624 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b4148920 con 0x7fd1b4069c90 2024-03-23T03:47:33.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.898+0000 7fd1b9600640 1 -- 172.21.15.102:0/2476391624 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b404e030 con 0x7fd1b4069c90 2024-03-23T03:47:33.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.899+0000 7fd1abe00640 1 -- 172.21.15.102:0/2476391624 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd1a40073c0 con 0x7fd1b4069c90 2024-03-23T03:47:33.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.900+0000 7fd1abe00640 1 --2- 172.21.15.102:0/2476391624 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd19406ac00 0x7fd19406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:33.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.900+0000 7fd1abe00640 1 -- 172.21.15.102:0/2476391624 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd1a4088ab0 con 0x7fd1b4069c90 2024-03-23T03:47:33.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.900+0000 7fd1b3400640 1 --2- 172.21.15.102:0/2476391624 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd19406ac00 0x7fd19406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:33.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.901+0000 7fd1b3400640 1 --2- 172.21.15.102:0/2476391624 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd19406ac00 0x7fd19406d0c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0036000 tx=0x7fd1a0008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:33.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:33.904+0000 7fd1abe00640 1 -- 172.21.15.102:0/2476391624 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd1a4053900 con 0x7fd1b4069c90 2024-03-23T03:47:34.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:34.162+0000 7fd1b9600640 1 -- 172.21.15.102:0/2476391624 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd1b406ca20 con 0x7fd1b4069c90 2024-03-23T03:47:34.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:34.163+0000 7fd1abe00640 1 -- 172.21.15.102:0/2476391624 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7fd1a4056f20 con 0x7fd1b4069c90 2024-03-23T03:47:34.164 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-23T03:47:34.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:34.166+0000 7fd1b9600640 1 -- 172.21.15.102:0/2476391624 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd19406ac00 msgr2=0x7fd19406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:34.166+0000 7fd1b9600640 1 --2- 172.21.15.102:0/2476391624 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd19406ac00 0x7fd19406d0c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd1a0036000 tx=0x7fd1a0008030 comp rx=0 tx=0).stop 2024-03-23T03:47:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:34.166+0000 7fd1b9600640 1 -- 172.21.15.102:0/2476391624 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b4069c90 msgr2=0x7fd1b414b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:34.166+0000 7fd1b9600640 1 --2- 172.21.15.102:0/2476391624 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b4069c90 0x7fd1b414b520 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd1a400c3c0 tx=0x7fd1a400c880 comp rx=0 tx=0).stop 2024-03-23T03:47:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:34.166+0000 7fd1b9600640 1 -- 172.21.15.102:0/2476391624 shutdown_connections 2024-03-23T03:47:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:34.166+0000 7fd1b9600640 1 --2- 172.21.15.102:0/2476391624 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd19406ac00 0x7fd19406d0c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:34.166+0000 7fd1b9600640 1 --2- 172.21.15.102:0/2476391624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1b415ca90 0x7fd1b40ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:34.166+0000 7fd1b9600640 1 --2- 172.21.15.102:0/2476391624 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd1b40c8de0 0x7fd1b40ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:34.166+0000 7fd1b9600640 1 --2- 172.21.15.102:0/2476391624 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1b4069c90 0x7fd1b414b520 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:34.166+0000 7fd1b9600640 1 -- 172.21.15.102:0/2476391624 >> 172.21.15.102:0/2476391624 conn(0x7fd1b4098a40 msgr2=0x7fd1b406d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:34.166+0000 7fd1b9600640 1 -- 172.21.15.102:0/2476391624 shutdown_connections 2024-03-23T03:47:34.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:34.166+0000 7fd1b9600640 1 -- 172.21.15.102:0/2476391624 wait complete. 2024-03-23T03:47:34.184 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T03:47:34.185 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:47:34.185 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2024-03-23T03:47:34.185 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-23T03:47:34.185 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-23T03:47:34.185 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T03:47:34.185 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T03:47:34.185 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:47:34.185 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-23T03:47:34.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:34.185 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2024-03-23T03:47:34.236 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:47:34.236 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:47:34.236 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:47:34.236 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:34.237 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-03-23T03:47:34.237 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:34.237 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:47:34.237 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:34.237 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f0 2024-03-23T03:47:34.237 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:47:34.237 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:34.237 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f1 2024-03-23T03:47:34.237 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:47:34.237 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:34.237 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:47:34.238 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:34.238 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:47:34.238 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 42616sec preferred_lft 42616sec 2024-03-23T03:47:34.238 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-03-23T03:47:34.238 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:34.238 INFO:teuthology.orchestra.run.smithi102.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:47:34.238 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether aa:17:e3:39:b0:56 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:34.238 INFO:teuthology.orchestra.run.smithi102.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T03:47:34.238 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:34.238 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::5cce:1fff:fe41:5a05/64 scope link 2024-03-23T03:47:34.238 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:34.238 INFO:teuthology.orchestra.run.smithi102.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:47:34.239 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether aa:17:e3:39:b0:56 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T03:47:34.239 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::a817:e3ff:fe39:b056/64 scope link 2024-03-23T03:47:34.239 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:34.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:34.240 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:47:34.322 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-23T03:47:34.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:34.327 DEBUG:teuthology.orchestra.run.smithi102:> route 2024-03-23T03:47:34.380 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2024-03-23T03:47:34.381 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:47:34.381 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:47:34.381 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:47:34.381 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:47:34.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:34.381 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:47:34.382 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:47:34.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:47:34.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:47:34.382 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:47:34.382 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:47:34.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:34.562 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-23T03:47:34.611 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T03:47:34.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:34.611 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list-id 2024-03-23T03:47:34.661 INFO:teuthology.orchestra.run.smithi102.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-23T03:47:34.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:34.662 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:47:34.662 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:47:34.662 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T03:47:34.662 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-23T03:47:34.663 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:47:34.801 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:34.801 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-23T03:47:34.888 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:47:34.888 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:47:34.888 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:47:34.888 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:34.888 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-03-23T03:47:34.888 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:34.888 INFO:teuthology.orchestra.run.smithi102.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:47:34.888 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 52:c7:d8:1d:c5:98 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-23T03:47:34.888 INFO:teuthology.orchestra.run.smithi102.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-23T03:47:34.888 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:34.889 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::50c7:d8ff:fe1d:c598/64 scope link 2024-03-23T03:47:34.889 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:34.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:34.892 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-23T03:47:34.976 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:47:34.976 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:47:34.976 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:47:34.976 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:34.977 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-03-23T03:47:34.977 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:34.977 INFO:teuthology.orchestra.run.smithi102.stdout:2: veth0@if9: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:47:34.977 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 52:c7:d8:1d:c5:98 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-23T03:47:34.977 INFO:teuthology.orchestra.run.smithi102.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-23T03:47:34.977 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:34.977 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::50c7:d8ff:fe1d:c598/64 scope link 2024-03-23T03:47:34.977 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:34.980 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-23T03:47:34.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:34.980 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:47:34.980 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:47:34.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-23T03:47:34.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-23T03:47:34.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-23T03:47:34.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-23T03:47:34.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-23T03:47:34.980 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:47:35.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:35.244 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:47:35.244 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:47:35.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.1 up 2024-03-23T03:47:35.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set dev brx.1 master ceph-brx 2024-03-23T03:47:35.244 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:47:35.406 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:47:35.406 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-23T03:47:35.452 INFO:teuthology.orchestra.run.smithi102.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-23T03:47:35.452 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:47:35.452 DEBUG:teuthology.orchestra.run.smithi102:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-23T03:47:35.546 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-23T03:47:35.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:35.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-03-23T03:47:35.687 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-03-23T03:47:35.687 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-03-23T03:47:35.687 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:47:35.709 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-03-23T03:47:35.710 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-03-23T03:47:35.710 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:47:35.710 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:47:35.710 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:47:35.738 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2024-03-23T03:47:35.738 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2024-03-23T03:47:35.738 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:47:35.738 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:47:35.738 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:47:35.738 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790" 2024-03-23T03:47:35.738 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/ 2024-03-23T03:47:35.738 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=776b2820-05f2-4e50-8d40-18dd76c5f6a0" will pass to kernel 2024-03-23T03:47:35.739 INFO:teuthology.orchestra.run.smithi102.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2024-03-23T03:47:35.739 INFO:teuthology.orchestra.run.smithi102.stdout: You just mounted a file system that supports labels which does not 2024-03-23T03:47:35.740 INFO:teuthology.orchestra.run.smithi102.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T03:47:35.740 INFO:teuthology.orchestra.run.smithi102.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T03:47:35.740 INFO:teuthology.orchestra.run.smithi102.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T03:47:35.743 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T03:47:35.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:35.744 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-23T03:47:35.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:35.826 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:47:35.883 INFO:teuthology.orchestra.run.smithi102.stdout:client4499 2024-03-23T03:47:35.883 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:47:35.883 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:47:35.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:35.932 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:47:36.034 INFO:teuthology.orchestra.run.smithi102.stdout:client4499 2024-03-23T03:47:36.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:36.034 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:47:36.087 INFO:teuthology.orchestra.run.smithi102.stdout:client4499 2024-03-23T03:47:36.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:36.087 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/776b2820-05f2-4e50-8d40-18dd76c5f6a0.client4499/status') 2024-03-23T03:47:36.173 INFO:teuthology.orchestra.run.smithi102.stdout:instance: client.4499 (3)172.21.15.102:0/3631857762 2024-03-23T03:47:36.173 INFO:teuthology.orchestra.run.smithi102.stdout:blocklisted: false 2024-03-23T03:47:36.173 INFO:teuthology.orchestra.run.smithi102.stderr:0+1 records in 2024-03-23T03:47:36.173 INFO:teuthology.orchestra.run.smithi102.stderr:0+1 records out 2024-03-23T03:47:36.173 INFO:teuthology.orchestra.run.smithi102.stderr:71 bytes copied, 7.8682e-05 s, 902 kB/s 2024-03-23T03:47:36.177 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-03-23T03:47:36.177 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-23T03:47:36.177 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-23T03:47:36.177 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:47:36.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.377+0000 7f8a5ca00640 1 Processor -- start 2024-03-23T03:47:36.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.377+0000 7f8a5ca00640 1 -- start start 2024-03-23T03:47:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.378+0000 7f8a5ca00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8a58160ca0 0x7f8a58161ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.378+0000 7f8a5ca00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8a5806c080 0x7f8a5806ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.379+0000 7f8a5ca00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a580cda90 con 0x7f8a5806cf40 2024-03-23T03:47:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.379+0000 7f8a5ca00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a580ce280 con 0x7f8a58160ca0 2024-03-23T03:47:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.379+0000 7f8a5ca00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5806d3b0 con 0x7f8a5806c080 2024-03-23T03:47:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.379+0000 7f8a56800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8a58160ca0 0x7f8a58161ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.379+0000 7f8a56800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8a58160ca0 0x7f8a58161ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53752/0 (socket says 172.21.15.102:53752) 2024-03-23T03:47:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.379+0000 7f8a56800640 1 -- 172.21.15.102:0/1647430383 learned_addr learned my addr 172.21.15.102:0/1647430383 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.379+0000 7f8a4de00640 1 --2- 172.21.15.102:0/1647430383 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8a5806c080 0x7f8a5806ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.379+0000 7f8a56800640 1 -- 172.21.15.102:0/1647430383 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8a5806c080 msgr2=0x7f8a5806ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.379+0000 7f8a56800640 1 --2- 172.21.15.102:0/1647430383 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8a5806c080 0x7f8a5806ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.379+0000 7f8a56800640 1 -- 172.21.15.102:0/1647430383 >> v1:172.21.15.173:6789/0 conn(0x7f8a5806cf40 legacy=0x7f8a58070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:36.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.379+0000 7f8a56800640 1 -- 172.21.15.102:0/1647430383 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a58162220 con 0x7f8a58160ca0 2024-03-23T03:47:36.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.380+0000 7f8a56800640 1 --2- 172.21.15.102:0/1647430383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a58160ca0 0x7f8a58161ce0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8a480098a0 tx=0x7f8a4802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53e8b46955977faa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:36.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.381+0000 7f8a55e00640 1 -- 172.21.15.102:0/1647430383 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a480317c0 con 0x7f8a58160ca0 2024-03-23T03:47:36.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.381+0000 7f8a55e00640 1 -- 172.21.15.102:0/1647430383 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a48031920 con 0x7f8a58160ca0 2024-03-23T03:47:36.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.381+0000 7f8a55e00640 1 -- 172.21.15.102:0/1647430383 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a48031cb0 con 0x7f8a58160ca0 2024-03-23T03:47:36.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.381+0000 7f8a5ca00640 1 -- 172.21.15.102:0/1647430383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a58160ca0 msgr2=0x7f8a58161ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:36.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.381+0000 7f8a5ca00640 1 --2- 172.21.15.102:0/1647430383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a58160ca0 0x7f8a58161ce0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8a480098a0 tx=0x7f8a4802b860 comp rx=0 tx=0).stop 2024-03-23T03:47:36.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.382+0000 7f8a5ca00640 1 -- 172.21.15.102:0/1647430383 shutdown_connections 2024-03-23T03:47:36.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.382+0000 7f8a5ca00640 1 --2- 172.21.15.102:0/1647430383 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8a5806c080 0x7f8a5806ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:36.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.382+0000 7f8a5ca00640 1 --2- 172.21.15.102:0/1647430383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a58160ca0 0x7f8a58161ce0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:36.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.382+0000 7f8a5ca00640 1 -- 172.21.15.102:0/1647430383 >> 172.21.15.102:0/1647430383 conn(0x7f8a58065df0 msgr2=0x7f8a58069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:36.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.382+0000 7f8a5ca00640 1 -- 172.21.15.102:0/1647430383 shutdown_connections 2024-03-23T03:47:36.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.382+0000 7f8a5ca00640 1 -- 172.21.15.102:0/1647430383 wait complete. 2024-03-23T03:47:36.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.383+0000 7f8a5ca00640 1 Processor -- start 2024-03-23T03:47:36.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.383+0000 7f8a5ca00640 1 -- start start 2024-03-23T03:47:36.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.384+0000 7f8a5ca00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8a5806c080 0x7f8a581f5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:36.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.384+0000 7f8a5ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a5806cf40 0x7f8a580aca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:36.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.384+0000 7f8a5ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a58160ca0 0x7f8a580aa610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:36.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.384+0000 7f8a5ca00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a581f62d0 con 0x7f8a5806cf40 2024-03-23T03:47:36.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.384+0000 7f8a5ca00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a580acfd0 con 0x7f8a58160ca0 2024-03-23T03:47:36.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.384+0000 7f8a5ca00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a580a7f60 con 0x7f8a5806c080 2024-03-23T03:47:36.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.385+0000 7f8a4de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a5806cf40 0x7f8a580aca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:36.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.385+0000 7f8a4de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a5806cf40 0x7f8a580aca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53754/0 (socket says 172.21.15.102:53754) 2024-03-23T03:47:36.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.385+0000 7f8a4de00640 1 -- 172.21.15.102:0/1781104355 learned_addr learned my addr 172.21.15.102:0/1781104355 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:36.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.385+0000 7f8a57200640 1 --2- 172.21.15.102:0/1781104355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a58160ca0 0x7f8a580aa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:36.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.385+0000 7f8a56800640 1 --2- 172.21.15.102:0/1781104355 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8a5806c080 0x7f8a581f5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:36.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.385+0000 7f8a4de00640 1 -- 172.21.15.102:0/1781104355 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8a5806c080 msgr2=0x7f8a581f5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:36.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.385+0000 7f8a4de00640 1 --2- 172.21.15.102:0/1781104355 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8a5806c080 0x7f8a581f5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:36.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.385+0000 7f8a4de00640 1 -- 172.21.15.102:0/1781104355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a58160ca0 msgr2=0x7f8a580aa610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:36.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.385+0000 7f8a4de00640 1 --2- 172.21.15.102:0/1781104355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a58160ca0 0x7f8a580aa610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:36.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.385+0000 7f8a4de00640 1 -- 172.21.15.102:0/1781104355 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a480094d0 con 0x7f8a5806cf40 2024-03-23T03:47:36.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.385+0000 7f8a56800640 1 --2- 172.21.15.102:0/1781104355 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8a5806c080 0x7f8a581f5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:36.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.385+0000 7f8a4de00640 1 --2- 172.21.15.102:0/1781104355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a5806cf40 0x7f8a580aca90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000c820 tx=0x7f8a4000cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:36.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.386+0000 7f8a4f400640 1 -- 172.21.15.102:0/1781104355 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a40004220 con 0x7f8a5806cf40 2024-03-23T03:47:36.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.386+0000 7f8a4f400640 1 -- 172.21.15.102:0/1781104355 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a400043a0 con 0x7f8a5806cf40 2024-03-23T03:47:36.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.386+0000 7f8a4f400640 1 -- 172.21.15.102:0/1781104355 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a40007a70 con 0x7f8a5806cf40 2024-03-23T03:47:36.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.386+0000 7f8a5ca00640 1 -- 172.21.15.102:0/1781104355 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a580aab50 con 0x7f8a5806cf40 2024-03-23T03:47:36.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.386+0000 7f8a5ca00640 1 -- 172.21.15.102:0/1781104355 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a580ad200 con 0x7f8a5806cf40 2024-03-23T03:47:36.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.387+0000 7f8a5ca00640 1 -- 172.21.15.102:0/1781104355 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5804e030 con 0x7f8a5806cf40 2024-03-23T03:47:36.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.388+0000 7f8a4f400640 1 -- 172.21.15.102:0/1781104355 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8a40007bd0 con 0x7f8a5806cf40 2024-03-23T03:47:36.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.389+0000 7f8a4f400640 1 --2- 172.21.15.102:0/1781104355 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8a3406ac30 0x7f8a3406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:36.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.389+0000 7f8a4f400640 1 -- 172.21.15.102:0/1781104355 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f8a40089460 con 0x7f8a5806cf40 2024-03-23T03:47:36.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.390+0000 7f8a56800640 1 --2- 172.21.15.102:0/1781104355 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8a3406ac30 0x7f8a3406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:36.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.390+0000 7f8a56800640 1 --2- 172.21.15.102:0/1781104355 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8a3406ac30 0x7f8a3406d0f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8a48036000 tx=0x7f8a4803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:36.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.394+0000 7f8a4f400640 1 -- 172.21.15.102:0/1781104355 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8a400542b0 con 0x7f8a5806cf40 2024-03-23T03:47:36.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.626+0000 7f8a5ca00640 1 -- 172.21.15.102:0/1781104355 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8a580a80a0 con 0x7f8a5806cf40 2024-03-23T03:47:36.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.627+0000 7f8a4f400640 1 -- 172.21.15.102:0/1781104355 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7f8a400578d0 con 0x7f8a5806cf40 2024-03-23T03:47:36.628 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-23T03:47:36.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.630+0000 7f8a5ca00640 1 -- 172.21.15.102:0/1781104355 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8a3406ac30 msgr2=0x7f8a3406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:36.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.630+0000 7f8a5ca00640 1 --2- 172.21.15.102:0/1781104355 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8a3406ac30 0x7f8a3406d0f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8a48036000 tx=0x7f8a4803d040 comp rx=0 tx=0).stop 2024-03-23T03:47:36.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.630+0000 7f8a5ca00640 1 -- 172.21.15.102:0/1781104355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a5806cf40 msgr2=0x7f8a580aca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.630+0000 7f8a5ca00640 1 --2- 172.21.15.102:0/1781104355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a5806cf40 0x7f8a580aca90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000c820 tx=0x7f8a4000cce0 comp rx=0 tx=0).stop 2024-03-23T03:47:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.630+0000 7f8a5ca00640 1 -- 172.21.15.102:0/1781104355 shutdown_connections 2024-03-23T03:47:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.630+0000 7f8a5ca00640 1 --2- 172.21.15.102:0/1781104355 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8a3406ac30 0x7f8a3406d0f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.630+0000 7f8a5ca00640 1 --2- 172.21.15.102:0/1781104355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a58160ca0 0x7f8a580aa610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.630+0000 7f8a5ca00640 1 --2- 172.21.15.102:0/1781104355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a5806cf40 0x7f8a580aca90 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.630+0000 7f8a5ca00640 1 --2- 172.21.15.102:0/1781104355 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8a5806c080 0x7f8a581f5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.630+0000 7f8a5ca00640 1 -- 172.21.15.102:0/1781104355 >> 172.21.15.102:0/1781104355 conn(0x7f8a58065df0 msgr2=0x7f8a580aea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.630+0000 7f8a5ca00640 1 -- 172.21.15.102:0/1781104355 shutdown_connections 2024-03-23T03:47:36.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:36.630+0000 7f8a5ca00640 1 -- 172.21.15.102:0/1781104355 wait complete. 2024-03-23T03:47:36.649 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T03:47:36.649 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:47:36.649 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi173.front.sepia.ceph.com 2024-03-23T03:47:36.649 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-23T03:47:36.649 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-23T03:47:36.649 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T03:47:36.649 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T03:47:36.649 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:47:36.649 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:47:36.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:36.650 DEBUG:teuthology.orchestra.run.smithi173:> ip addr 2024-03-23T03:47:36.706 INFO:teuthology.orchestra.run.smithi173.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:47:36.707 INFO:teuthology.orchestra.run.smithi173.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:47:36.707 INFO:teuthology.orchestra.run.smithi173.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:47:36.707 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:36.707 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 ::1/128 scope host 2024-03-23T03:47:36.707 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:36.707 INFO:teuthology.orchestra.run.smithi173.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:47:36.707 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6a brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:36.707 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f0 2024-03-23T03:47:36.707 INFO:teuthology.orchestra.run.smithi173.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:47:36.707 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6b brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:36.707 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f1 2024-03-23T03:47:36.707 INFO:teuthology.orchestra.run.smithi173.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:47:36.707 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:58 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:36.707 INFO:teuthology.orchestra.run.smithi173.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:47:36.708 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:59 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:36.708 INFO:teuthology.orchestra.run.smithi173.stdout: inet 172.21.15.173/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:47:36.708 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft 42585sec preferred_lft 42585sec 2024-03-23T03:47:36.708 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::ec4:7aff:fe88:7259/64 scope link 2024-03-23T03:47:36.708 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:36.709 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-23T03:47:36.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:36.709 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-03-23T03:47:36.709 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-03-23T03:47:36.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link add name ceph-brx type bridge 2024-03-23T03:47:36.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip addr flush dev ceph-brx 2024-03-23T03:47:36.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link set ceph-brx up 2024-03-23T03:47:36.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-23T03:47:36.709 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-03-23T03:47:37.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:37.122 DEBUG:teuthology.orchestra.run.smithi173:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:47:37.193 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-03-23T03:47:37.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:37.196 DEBUG:teuthology.orchestra.run.smithi173:> route 2024-03-23T03:47:37.250 INFO:teuthology.orchestra.run.smithi173.stdout:Kernel IP routing table 2024-03-23T03:47:37.250 INFO:teuthology.orchestra.run.smithi173.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:47:37.250 INFO:teuthology.orchestra.run.smithi173.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:47:37.250 INFO:teuthology.orchestra.run.smithi173.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:47:37.250 INFO:teuthology.orchestra.run.smithi173.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:47:37.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:37.251 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-03-23T03:47:37.251 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-03-23T03:47:37.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:47:37.251 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:47:37.252 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:47:37.252 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-03-23T03:47:37.937 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:37.937 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list 2024-03-23T03:47:37.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:37.992 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list-id 2024-03-23T03:47:38.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:38.040 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-03-23T03:47:38.040 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-03-23T03:47:38.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T03:47:38.040 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-23T03:47:38.040 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-03-23T03:47:38.172 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-23T03:47:38.172 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:38.172 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-03-23T03:47:38.172 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-03-23T03:47:38.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-23T03:47:38.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-23T03:47:38.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-23T03:47:38.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-23T03:47:38.172 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-23T03:47:38.172 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-03-23T03:47:38.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:38.447 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-03-23T03:47:38.447 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-03-23T03:47:38.447 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link set brx.0 up 2024-03-23T03:47:38.447 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link set dev brx.0 master ceph-brx 2024-03-23T03:47:38.447 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-03-23T03:47:38.612 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:47:38.612 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-23T03:47:38.660 INFO:teuthology.orchestra.run.smithi173.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:47:38.660 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:47:38.660 DEBUG:teuthology.orchestra.run.smithi173:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-23T03:47:38.709 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-23T03:47:38.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:38.709 DEBUG:teuthology.orchestra.run.smithi173:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:47:38.837 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:47:38.837 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:47:38.837 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:47:39.435 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:47:39.435 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:47:39.435 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:47:39.436 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:47:39.436 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:47:39.474 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:47:39.474 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:47:39.474 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:47:39.474 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:47:39.474 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:47:39.474 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790" 2024-03-23T03:47:39.474 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/ 2024-03-23T03:47:39.474 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=776b2820-05f2-4e50-8d40-18dd76c5f6a0" will pass to kernel 2024-03-23T03:47:39.479 INFO:teuthology.orchestra.run.smithi173.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-23T03:47:39.479 INFO:teuthology.orchestra.run.smithi173.stdout: You just mounted a file system that supports labels which does not 2024-03-23T03:47:39.479 INFO:teuthology.orchestra.run.smithi173.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T03:47:39.479 INFO:teuthology.orchestra.run.smithi173.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T03:47:39.479 INFO:teuthology.orchestra.run.smithi173.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T03:47:39.484 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T03:47:39.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:39.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-23T03:47:39.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:39.576 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:47:39.632 INFO:teuthology.orchestra.run.smithi173.stdout:client4669 2024-03-23T03:47:39.632 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:47:39.632 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:47:39.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:39.680 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:47:39.781 INFO:teuthology.orchestra.run.smithi173.stdout:client4669 2024-03-23T03:47:39.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:39.781 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:47:39.840 INFO:teuthology.orchestra.run.smithi173.stdout:client4669 2024-03-23T03:47:39.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:39.840 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/776b2820-05f2-4e50-8d40-18dd76c5f6a0.client4669/status') 2024-03-23T03:47:39.931 INFO:teuthology.orchestra.run.smithi173.stdout:instance: client.4669 (3)192.168.0.1:0/3354320279 2024-03-23T03:47:39.931 INFO:teuthology.orchestra.run.smithi173.stdout:blocklisted: false 2024-03-23T03:47:39.931 INFO:teuthology.orchestra.run.smithi173.stderr:0+1 records in 2024-03-23T03:47:39.931 INFO:teuthology.orchestra.run.smithi173.stderr:0+1 records out 2024-03-23T03:47:39.932 INFO:teuthology.orchestra.run.smithi173.stderr:69 bytes copied, 7.701e-05 s, 896 kB/s 2024-03-23T03:47:39.935 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2024-03-23T03:47:39.935 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-23T03:47:39.935 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-23T03:47:39.935 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:47:40.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.151+0000 7f1346600640 1 Processor -- start 2024-03-23T03:47:40.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.151+0000 7f1346600640 1 -- start start 2024-03-23T03:47:40.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.153+0000 7f1346600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f134015b560 0x7f134015b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:40.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.153+0000 7f1346600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f13400b39b0 0x7f13400b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:40.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.153+0000 7f1346600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f134015e500 0x7f134015e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:40.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.153+0000 7f1346600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13400b4370 con 0x7f134015e500 2024-03-23T03:47:40.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.153+0000 7f1346600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f134006c6f0 con 0x7f13400b39b0 2024-03-23T03:47:40.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.153+0000 7f1346600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f134006c9d0 con 0x7f134015b560 2024-03-23T03:47:40.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.153+0000 7f133fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f134015e500 0x7f134015e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:40.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.153+0000 7f133fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f134015e500 0x7f134015e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53760/0 (socket says 172.21.15.102:53760) 2024-03-23T03:47:40.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.153+0000 7f133fe00640 1 -- 172.21.15.102:0/3004655884 learned_addr learned my addr 172.21.15.102:0/3004655884 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:40.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.153+0000 7f133f400640 1 --2- 172.21.15.102:0/3004655884 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f134015b560 0x7f134015b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:40.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.153+0000 7f133ea00640 1 --2- 172.21.15.102:0/3004655884 >> v2:172.21.15.173:3300/0 conn(0x7f13400b39b0 0x7f13400b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:40.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.154+0000 7f133fe00640 1 -- 172.21.15.102:0/3004655884 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f134015b560 msgr2=0x7f134015b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:40.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.154+0000 7f133fe00640 1 --2- 172.21.15.102:0/3004655884 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f134015b560 0x7f134015b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:40.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.154+0000 7f133fe00640 1 -- 172.21.15.102:0/3004655884 >> v2:172.21.15.173:3300/0 conn(0x7f13400b39b0 msgr2=0x7f13400b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:40.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.154+0000 7f133fe00640 1 --2- 172.21.15.102:0/3004655884 >> v2:172.21.15.173:3300/0 conn(0x7f13400b39b0 0x7f13400b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:40.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.154+0000 7f133fe00640 1 -- 172.21.15.102:0/3004655884 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134006ccb0 con 0x7f134015e500 2024-03-23T03:47:40.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.155+0000 7f133fe00640 1 --2- 172.21.15.102:0/3004655884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f134015e500 0x7f134015e990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1340071510 tx=0x7f133802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fedee9fabe2eeb20 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:40.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.155+0000 7f133e000640 1 -- 172.21.15.102:0/3004655884 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1338031860 con 0x7f134015e500 2024-03-23T03:47:40.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.155+0000 7f133e000640 1 -- 172.21.15.102:0/3004655884 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13380319c0 con 0x7f134015e500 2024-03-23T03:47:40.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.155+0000 7f133e000640 1 -- 172.21.15.102:0/3004655884 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1338031d50 con 0x7f134015e500 2024-03-23T03:47:40.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.156+0000 7f1346600640 1 -- 172.21.15.102:0/3004655884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f134015e500 msgr2=0x7f134015e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:40.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.156+0000 7f1346600640 1 --2- 172.21.15.102:0/3004655884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f134015e500 0x7f134015e990 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f1340071510 tx=0x7f133802b900 comp rx=0 tx=0).stop 2024-03-23T03:47:40.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.156+0000 7f1346600640 1 -- 172.21.15.102:0/3004655884 shutdown_connections 2024-03-23T03:47:40.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.156+0000 7f1346600640 1 --2- 172.21.15.102:0/3004655884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f134015e500 0x7f134015e990 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:40.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.156+0000 7f1346600640 1 --2- 172.21.15.102:0/3004655884 >> v2:172.21.15.173:3300/0 conn(0x7f13400b39b0 0x7f13400b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:40.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.156+0000 7f1346600640 1 --2- 172.21.15.102:0/3004655884 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f134015b560 0x7f134015b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:40.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.156+0000 7f1346600640 1 -- 172.21.15.102:0/3004655884 >> 172.21.15.102:0/3004655884 conn(0x7f134006a3d0 msgr2=0x7f13400b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:40.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.156+0000 7f1346600640 1 -- 172.21.15.102:0/3004655884 shutdown_connections 2024-03-23T03:47:40.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.156+0000 7f1346600640 1 -- 172.21.15.102:0/3004655884 wait complete. 2024-03-23T03:47:40.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.158+0000 7f1346600640 1 Processor -- start 2024-03-23T03:47:40.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.158+0000 7f1346600640 1 -- start start 2024-03-23T03:47:40.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.159+0000 7f1346600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13400b39b0 0x7f134013f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:40.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.159+0000 7f1346600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f134015b560 0x7f1340150fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:40.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.159+0000 7f1346600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f134015e500 0x7f13400aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:40.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.159+0000 7f1346600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1340151520 con 0x7f134015b560 2024-03-23T03:47:40.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.159+0000 7f1346600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13400af260 con 0x7f13400b39b0 2024-03-23T03:47:40.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.159+0000 7f1346600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13401f0500 con 0x7f134015e500 2024-03-23T03:47:40.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.159+0000 7f133ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f134015b560 0x7f1340150fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:40.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.159+0000 7f133ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f134015b560 0x7f1340150fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53770/0 (socket says 172.21.15.102:53770) 2024-03-23T03:47:40.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.159+0000 7f133ea00640 1 -- 172.21.15.102:0/4254041744 learned_addr learned my addr 172.21.15.102:0/4254041744 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:40.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.160+0000 7f133f400640 1 --2- 172.21.15.102:0/4254041744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13400b39b0 0x7f134013f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:40.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.160+0000 7f133fe00640 1 --2- 172.21.15.102:0/4254041744 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f134015e500 0x7f13400aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:40.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.160+0000 7f133ea00640 1 -- 172.21.15.102:0/4254041744 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f134015e500 msgr2=0x7f13400aed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:40.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.160+0000 7f133ea00640 1 --2- 172.21.15.102:0/4254041744 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f134015e500 0x7f13400aed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:40.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.160+0000 7f133ea00640 1 -- 172.21.15.102:0/4254041744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13400b39b0 msgr2=0x7f134013f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:40.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.160+0000 7f133ea00640 1 --2- 172.21.15.102:0/4254041744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13400b39b0 0x7f134013f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:40.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.160+0000 7f133ea00640 1 -- 172.21.15.102:0/4254041744 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13380094d0 con 0x7f134015b560 2024-03-23T03:47:40.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.160+0000 7f133ea00640 1 --2- 172.21.15.102:0/4254041744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f134015b560 0x7f1340150fe0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f132c007e40 tx=0x7f132c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:40.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.160+0000 7f1337400640 1 -- 172.21.15.102:0/4254041744 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c014620 con 0x7f134015b560 2024-03-23T03:47:40.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.161+0000 7f1337400640 1 -- 172.21.15.102:0/4254041744 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132c0147a0 con 0x7f134015b560 2024-03-23T03:47:40.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.161+0000 7f1337400640 1 -- 172.21.15.102:0/4254041744 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f132c014b50 con 0x7f134015b560 2024-03-23T03:47:40.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.161+0000 7f1346600640 1 -- 172.21.15.102:0/4254041744 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13400af470 con 0x7f134015b560 2024-03-23T03:47:40.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.161+0000 7f1346600640 1 -- 172.21.15.102:0/4254041744 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13400afa30 con 0x7f134015b560 2024-03-23T03:47:40.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.162+0000 7f1346600640 1 -- 172.21.15.102:0/4254041744 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f134006ac20 con 0x7f134015b560 2024-03-23T03:47:40.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.163+0000 7f1337400640 1 -- 172.21.15.102:0/4254041744 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f132c014cb0 con 0x7f134015b560 2024-03-23T03:47:40.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.164+0000 7f1337400640 1 --2- 172.21.15.102:0/4254041744 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f132006ac30 0x7f132006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:40.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.164+0000 7f1337400640 1 -- 172.21.15.102:0/4254041744 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f132c089ed0 con 0x7f134015b560 2024-03-23T03:47:40.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.164+0000 7f133f400640 1 --2- 172.21.15.102:0/4254041744 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f132006ac30 0x7f132006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:40.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.165+0000 7f133f400640 1 --2- 172.21.15.102:0/4254041744 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f132006ac30 0x7f132006d0f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1330006fd0 tx=0x7f1330008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:40.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.168+0000 7f1337400640 1 -- 172.21.15.102:0/4254041744 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f132c054e30 con 0x7f134015b560 2024-03-23T03:47:40.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.402+0000 7f1346600640 1 -- 172.21.15.102:0/4254041744 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f13401552a0 con 0x7f134015b560 2024-03-23T03:47:40.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.402+0000 7f1337400640 1 -- 172.21.15.102:0/4254041744 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+88 (secure 0 0 0) 0x7f132c058450 con 0x7f134015b560 2024-03-23T03:47:40.403 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-23T03:47:40.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.405+0000 7f1346600640 1 -- 172.21.15.102:0/4254041744 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f132006ac30 msgr2=0x7f132006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:40.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.405+0000 7f1346600640 1 --2- 172.21.15.102:0/4254041744 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f132006ac30 0x7f132006d0f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1330006fd0 tx=0x7f1330008040 comp rx=0 tx=0).stop 2024-03-23T03:47:40.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.405+0000 7f1346600640 1 -- 172.21.15.102:0/4254041744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f134015b560 msgr2=0x7f1340150fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:40.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.405+0000 7f1346600640 1 --2- 172.21.15.102:0/4254041744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f134015b560 0x7f1340150fe0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f132c007e40 tx=0x7f132c00e4b0 comp rx=0 tx=0).stop 2024-03-23T03:47:40.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.405+0000 7f1346600640 1 -- 172.21.15.102:0/4254041744 shutdown_connections 2024-03-23T03:47:40.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.405+0000 7f1346600640 1 --2- 172.21.15.102:0/4254041744 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f132006ac30 0x7f132006d0f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:40.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.405+0000 7f1346600640 1 --2- 172.21.15.102:0/4254041744 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f134015e500 0x7f13400aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:40.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.405+0000 7f1346600640 1 --2- 172.21.15.102:0/4254041744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f134015b560 0x7f1340150fe0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:40.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.405+0000 7f1346600640 1 --2- 172.21.15.102:0/4254041744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13400b39b0 0x7f134013f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:40.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.405+0000 7f1346600640 1 -- 172.21.15.102:0/4254041744 >> 172.21.15.102:0/4254041744 conn(0x7f134006a3d0 msgr2=0x7f13400cc690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:40.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.406+0000 7f1346600640 1 -- 172.21.15.102:0/4254041744 shutdown_connections 2024-03-23T03:47:40.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:40.406+0000 7f1346600640 1 -- 172.21.15.102:0/4254041744 wait complete. 2024-03-23T03:47:40.423 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T03:47:40.424 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:47:40.424 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi173.front.sepia.ceph.com 2024-03-23T03:47:40.424 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-23T03:47:40.424 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-23T03:47:40.424 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T03:47:40.424 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T03:47:40.424 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:47:40.424 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-23T03:47:40.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:40.424 DEBUG:teuthology.orchestra.run.smithi173:> ip addr 2024-03-23T03:47:40.483 INFO:teuthology.orchestra.run.smithi173.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:47:40.483 INFO:teuthology.orchestra.run.smithi173.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:47:40.483 INFO:teuthology.orchestra.run.smithi173.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:47:40.483 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:40.483 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 ::1/128 scope host 2024-03-23T03:47:40.483 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:40.483 INFO:teuthology.orchestra.run.smithi173.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:47:40.484 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6a brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:40.484 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f0 2024-03-23T03:47:40.484 INFO:teuthology.orchestra.run.smithi173.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:47:40.484 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6b brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:40.484 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f1 2024-03-23T03:47:40.484 INFO:teuthology.orchestra.run.smithi173.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:47:40.484 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:58 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:40.484 INFO:teuthology.orchestra.run.smithi173.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:47:40.484 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:59 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:40.484 INFO:teuthology.orchestra.run.smithi173.stdout: inet 172.21.15.173/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:47:40.484 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft 42581sec preferred_lft 42581sec 2024-03-23T03:47:40.484 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::ec4:7aff:fe88:7259/64 scope link 2024-03-23T03:47:40.485 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:40.485 INFO:teuthology.orchestra.run.smithi173.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:47:40.485 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether a2:27:ac:42:68:ce brd ff:ff:ff:ff:ff:ff 2024-03-23T03:47:40.485 INFO:teuthology.orchestra.run.smithi173.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T03:47:40.485 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:40.485 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::6894:e9ff:fe5d:1492/64 scope link 2024-03-23T03:47:40.485 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:40.485 INFO:teuthology.orchestra.run.smithi173.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:47:40.485 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether a2:27:ac:42:68:ce brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T03:47:40.485 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::a027:acff:fe42:68ce/64 scope link 2024-03-23T03:47:40.485 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:40.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:40.486 DEBUG:teuthology.orchestra.run.smithi173:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:47:40.574 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-03-23T03:47:40.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:40.579 DEBUG:teuthology.orchestra.run.smithi173:> route 2024-03-23T03:47:40.630 INFO:teuthology.orchestra.run.smithi173.stdout:Kernel IP routing table 2024-03-23T03:47:40.630 INFO:teuthology.orchestra.run.smithi173.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:47:40.630 INFO:teuthology.orchestra.run.smithi173.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:47:40.631 INFO:teuthology.orchestra.run.smithi173.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:47:40.631 INFO:teuthology.orchestra.run.smithi173.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:47:40.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:40.632 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-03-23T03:47:40.632 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-03-23T03:47:40.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:47:40.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:47:40.632 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:47:40.632 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-03-23T03:47:40.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:40.794 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list 2024-03-23T03:47:40.840 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-23T03:47:40.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:40.841 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list-id 2024-03-23T03:47:40.935 INFO:teuthology.orchestra.run.smithi173.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-23T03:47:40.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:40.936 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-03-23T03:47:40.936 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-03-23T03:47:40.936 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-23T03:47:40.936 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-23T03:47:40.936 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-03-23T03:47:41.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:41.080 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-23T03:47:41.166 INFO:teuthology.orchestra.run.smithi173.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:47:41.166 INFO:teuthology.orchestra.run.smithi173.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:47:41.166 INFO:teuthology.orchestra.run.smithi173.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:47:41.166 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:41.166 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 ::1/128 scope host 2024-03-23T03:47:41.166 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:41.166 INFO:teuthology.orchestra.run.smithi173.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:47:41.166 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 76:92:10:49:74:67 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-23T03:47:41.166 INFO:teuthology.orchestra.run.smithi173.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-23T03:47:41.166 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:41.166 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::7492:10ff:fe49:7467/64 scope link 2024-03-23T03:47:41.167 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:41.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:41.170 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-23T03:47:41.260 INFO:teuthology.orchestra.run.smithi173.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:47:41.260 INFO:teuthology.orchestra.run.smithi173.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:47:41.260 INFO:teuthology.orchestra.run.smithi173.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:47:41.260 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:41.260 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 ::1/128 scope host 2024-03-23T03:47:41.260 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:41.260 INFO:teuthology.orchestra.run.smithi173.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:47:41.260 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 76:92:10:49:74:67 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-23T03:47:41.260 INFO:teuthology.orchestra.run.smithi173.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-23T03:47:41.261 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:41.261 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::7492:10ff:fe49:7467/64 scope link 2024-03-23T03:47:41.261 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:47:41.265 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-23T03:47:41.265 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:41.265 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-03-23T03:47:41.265 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-03-23T03:47:41.265 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-23T03:47:41.265 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-23T03:47:41.265 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-23T03:47:41.265 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-23T03:47:41.265 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-23T03:47:41.266 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-03-23T03:47:41.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:41.529 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-03-23T03:47:41.529 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-03-23T03:47:41.529 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link set brx.1 up 2024-03-23T03:47:41.529 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link set dev brx.1 master ceph-brx 2024-03-23T03:47:41.529 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-03-23T03:47:41.698 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:47:41.698 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-23T03:47:41.743 INFO:teuthology.orchestra.run.smithi173.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-23T03:47:41.743 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:47:41.743 DEBUG:teuthology.orchestra.run.smithi173:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-23T03:47:41.840 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-23T03:47:41.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:41.840 DEBUG:teuthology.orchestra.run.smithi173:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-03-23T03:47:41.953 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-03-23T03:47:41.953 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-03-23T03:47:41.953 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:47:41.973 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-03-23T03:47:41.974 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-03-23T03:47:41.974 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:47:41.974 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:47:41.974 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:47:41.998 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2024-03-23T03:47:41.998 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2024-03-23T03:47:41.998 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:47:41.998 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:47:41.998 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:47:41.998 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790" 2024-03-23T03:47:41.998 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/ 2024-03-23T03:47:41.998 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=776b2820-05f2-4e50-8d40-18dd76c5f6a0" will pass to kernel 2024-03-23T03:47:42.000 INFO:teuthology.orchestra.run.smithi173.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2024-03-23T03:47:42.000 INFO:teuthology.orchestra.run.smithi173.stdout: You just mounted a file system that supports labels which does not 2024-03-23T03:47:42.000 INFO:teuthology.orchestra.run.smithi173.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T03:47:42.000 INFO:teuthology.orchestra.run.smithi173.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T03:47:42.000 INFO:teuthology.orchestra.run.smithi173.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T03:47:42.004 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T03:47:42.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:42.004 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-23T03:47:42.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:42.085 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:47:42.137 INFO:teuthology.orchestra.run.smithi173.stdout:client4505 2024-03-23T03:47:42.137 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:47:42.137 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:47:42.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:42.188 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:47:42.288 INFO:teuthology.orchestra.run.smithi173.stdout:client4505 2024-03-23T03:47:42.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:42.289 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:47:42.341 INFO:teuthology.orchestra.run.smithi173.stdout:client4505 2024-03-23T03:47:42.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:42.341 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/776b2820-05f2-4e50-8d40-18dd76c5f6a0.client4505/status') 2024-03-23T03:47:42.429 INFO:teuthology.orchestra.run.smithi173.stdout:instance: client.4505 (3)192.168.0.2:0/1508013574 2024-03-23T03:47:42.430 INFO:teuthology.orchestra.run.smithi173.stdout:blocklisted: false 2024-03-23T03:47:42.430 INFO:teuthology.orchestra.run.smithi173.stderr:0+1 records in 2024-03-23T03:47:42.430 INFO:teuthology.orchestra.run.smithi173.stderr:0+1 records out 2024-03-23T03:47:42.430 INFO:teuthology.orchestra.run.smithi173.stderr:69 bytes copied, 6.4537e-05 s, 1.1 MB/s 2024-03-23T03:47:42.434 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-23T03:47:42.472 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:42.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.692+0000 7f09c4c00640 1 Processor -- start 2024-03-23T03:47:42.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.692+0000 7f09c4c00640 1 -- start start 2024-03-23T03:47:42.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.692+0000 7f09c4c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f09c0069c90 0x7f09c006a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:42.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.692+0000 7f09c4c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09c00b43b0 0x7f09c00acdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:42.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.692+0000 7f09c4c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f09c0145b40 0x7f09c00ad2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:42.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.692+0000 7f09c4c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c00b23c0 con 0x7f09c0145b40 2024-03-23T03:47:42.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.692+0000 7f09c4c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09c006a5b0 con 0x7f09c0069c90 2024-03-23T03:47:42.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.692+0000 7f09c4c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c00b2610 con 0x7f09c00b43b0 2024-03-23T03:47:42.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.692+0000 7f09bfe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f09c0145b40 0x7f09c00ad2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:42.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.693+0000 7f09bfe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f09c0145b40 0x7f09c00ad2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53776/0 (socket says 172.21.15.102:53776) 2024-03-23T03:47:42.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.693+0000 7f09bfe00640 1 -- 172.21.15.102:0/1226517954 learned_addr learned my addr 172.21.15.102:0/1226517954 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:42.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.693+0000 7f09bfe00640 1 -- 172.21.15.102:0/1226517954 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09c00b43b0 msgr2=0x7f09c00acdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:42.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.693+0000 7f09bfe00640 1 --2- 172.21.15.102:0/1226517954 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09c00b43b0 0x7f09c00acdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:42.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.693+0000 7f09bfe00640 1 -- 172.21.15.102:0/1226517954 >> v2:172.21.15.173:3300/0 conn(0x7f09c0069c90 msgr2=0x7f09c006a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:42.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.693+0000 7f09bfe00640 1 --2- 172.21.15.102:0/1226517954 >> v2:172.21.15.173:3300/0 conn(0x7f09c0069c90 0x7f09c006a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:42.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.693+0000 7f09bfe00640 1 -- 172.21.15.102:0/1226517954 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c006eda0 con 0x7f09c0145b40 2024-03-23T03:47:42.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.694+0000 7f09bfe00640 1 --2- 172.21.15.102:0/1226517954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09c0145b40 0x7f09c00ad2f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f09c006d0a0 tx=0x7f09b802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f58a7fde597b9fd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:42.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.694+0000 7f09be000640 1 -- 172.21.15.102:0/1226517954 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b802c9e0 con 0x7f09c0145b40 2024-03-23T03:47:42.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.694+0000 7f09be000640 1 -- 172.21.15.102:0/1226517954 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b802cb40 con 0x7f09c0145b40 2024-03-23T03:47:42.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.694+0000 7f09be000640 1 -- 172.21.15.102:0/1226517954 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b80343f0 con 0x7f09c0145b40 2024-03-23T03:47:42.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.695+0000 7f09c4c00640 1 -- 172.21.15.102:0/1226517954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09c0145b40 msgr2=0x7f09c00ad2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:42.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.695+0000 7f09c4c00640 1 --2- 172.21.15.102:0/1226517954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09c0145b40 0x7f09c00ad2f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f09c006d0a0 tx=0x7f09b802be40 comp rx=0 tx=0).stop 2024-03-23T03:47:42.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.695+0000 7f09c4c00640 1 -- 172.21.15.102:0/1226517954 shutdown_connections 2024-03-23T03:47:42.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.695+0000 7f09c4c00640 1 --2- 172.21.15.102:0/1226517954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09c0145b40 0x7f09c00ad2f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:42.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.695+0000 7f09c4c00640 1 --2- 172.21.15.102:0/1226517954 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09c00b43b0 0x7f09c00acdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:42.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.695+0000 7f09c4c00640 1 --2- 172.21.15.102:0/1226517954 >> v2:172.21.15.173:3300/0 conn(0x7f09c0069c90 0x7f09c006a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:42.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.695+0000 7f09c4c00640 1 -- 172.21.15.102:0/1226517954 >> 172.21.15.102:0/1226517954 conn(0x7f09c00b48e0 msgr2=0x7f09c00b4cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:42.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.695+0000 7f09c4c00640 1 -- 172.21.15.102:0/1226517954 shutdown_connections 2024-03-23T03:47:42.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.695+0000 7f09c4c00640 1 -- 172.21.15.102:0/1226517954 wait complete. 2024-03-23T03:47:42.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.697+0000 7f09c4c00640 1 Processor -- start 2024-03-23T03:47:42.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.697+0000 7f09c4c00640 1 -- start start 2024-03-23T03:47:42.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.698+0000 7f09c4c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09c0069c90 0x7f09c00cf7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:42.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.698+0000 7f09c4c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09c00b43b0 0x7f09c00cfcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:42.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.698+0000 7f09c4c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09c0145b40 0x7f09c01f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:42.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.698+0000 7f09c4c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c00d0230 con 0x7f09c0145b40 2024-03-23T03:47:42.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.698+0000 7f09c4c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09c014fb90 con 0x7f09c0069c90 2024-03-23T03:47:42.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.698+0000 7f09c4c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09c0099ba0 con 0x7f09c00b43b0 2024-03-23T03:47:42.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.698+0000 7f09bfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09c0145b40 0x7f09c01f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:42.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.698+0000 7f09bfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09c0145b40 0x7f09c01f2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53780/0 (socket says 172.21.15.102:53780) 2024-03-23T03:47:42.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.698+0000 7f09bfe00640 1 -- 172.21.15.102:0/2645030773 learned_addr learned my addr 172.21.15.102:0/2645030773 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:42.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.699+0000 7f09bea00640 1 --2- 172.21.15.102:0/2645030773 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09c00b43b0 0x7f09c00cfcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:42.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.699+0000 7f09bfe00640 1 -- 172.21.15.102:0/2645030773 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09c00b43b0 msgr2=0x7f09c00cfcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:42.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.699+0000 7f09bf400640 1 --2- 172.21.15.102:0/2645030773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09c0069c90 0x7f09c00cf7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:42.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.699+0000 7f09bfe00640 1 --2- 172.21.15.102:0/2645030773 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09c00b43b0 0x7f09c00cfcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:42.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.699+0000 7f09bfe00640 1 -- 172.21.15.102:0/2645030773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09c0069c90 msgr2=0x7f09c00cf7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:42.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.699+0000 7f09bfe00640 1 --2- 172.21.15.102:0/2645030773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09c0069c90 0x7f09c00cf7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:42.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.699+0000 7f09bfe00640 1 -- 172.21.15.102:0/2645030773 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b8009b30 con 0x7f09c0145b40 2024-03-23T03:47:42.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.699+0000 7f09bea00640 1 --2- 172.21.15.102:0/2645030773 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09c00b43b0 0x7f09c00cfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:42.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.699+0000 7f09bf400640 1 --2- 172.21.15.102:0/2645030773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09c0069c90 0x7f09c00cf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:42.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.699+0000 7f09bfe00640 1 --2- 172.21.15.102:0/2645030773 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09c0145b40 0x7f09c01f2250 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f09b8036040 tx=0x7f09b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:42.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.699+0000 7f09b7400640 1 -- 172.21.15.102:0/2645030773 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b803e450 con 0x7f09c0145b40 2024-03-23T03:47:42.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.699+0000 7f09b7400640 1 -- 172.21.15.102:0/2645030773 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09b803e5d0 con 0x7f09c0145b40 2024-03-23T03:47:42.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.699+0000 7f09c4c00640 1 -- 172.21.15.102:0/2645030773 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09c014f610 con 0x7f09c0145b40 2024-03-23T03:47:42.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.699+0000 7f09b7400640 1 -- 172.21.15.102:0/2645030773 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09b803e980 con 0x7f09c0145b40 2024-03-23T03:47:42.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.700+0000 7f09c4c00640 1 -- 172.21.15.102:0/2645030773 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09c00995e0 con 0x7f09c0145b40 2024-03-23T03:47:42.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.701+0000 7f09c4c00640 1 -- 172.21.15.102:0/2645030773 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09c004e030 con 0x7f09c0145b40 2024-03-23T03:47:42.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.701+0000 7f09b7400640 1 -- 172.21.15.102:0/2645030773 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f09b803eae0 con 0x7f09c0145b40 2024-03-23T03:47:42.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.702+0000 7f09b7400640 1 --2- 172.21.15.102:0/2645030773 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f09a006ac00 0x7f09a006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:42.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.702+0000 7f09b7400640 1 -- 172.21.15.102:0/2645030773 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f09b80ac5c0 con 0x7f09c0145b40 2024-03-23T03:47:42.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.703+0000 7f09bf400640 1 --2- 172.21.15.102:0/2645030773 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f09a006ac00 0x7f09a006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:42.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.703+0000 7f09bf400640 1 --2- 172.21.15.102:0/2645030773 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f09a006ac00 0x7f09a006d0c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f09c014ca90 tx=0x7f09b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:42.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.708+0000 7f09b7400640 1 -- 172.21.15.102:0/2645030773 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f09b8077480 con 0x7f09c0145b40 2024-03-23T03:47:42.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.904+0000 7f09c4c00640 1 -- 172.21.15.102:0/2645030773 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09c00ad070 con 0x7f09c0145b40 2024-03-23T03:47:42.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.907+0000 7f09b7400640 1 -- 172.21.15.102:0/2645030773 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18260 (secure 0 0 0) 0x7f09b807aaa0 con 0x7f09c0145b40 2024-03-23T03:47:42.908 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:42.908 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":28,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:47:41.647786+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:47:42.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.909+0000 7f09c4c00640 1 -- 172.21.15.102:0/2645030773 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f09a006ac00 msgr2=0x7f09a006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:42.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.909+0000 7f09c4c00640 1 --2- 172.21.15.102:0/2645030773 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f09a006ac00 0x7f09a006d0c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f09c014ca90 tx=0x7f09b0008040 comp rx=0 tx=0).stop 2024-03-23T03:47:42.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.909+0000 7f09c4c00640 1 -- 172.21.15.102:0/2645030773 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09c0145b40 msgr2=0x7f09c01f2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:42.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.909+0000 7f09c4c00640 1 --2- 172.21.15.102:0/2645030773 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09c0145b40 0x7f09c01f2250 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f09b8036040 tx=0x7f09b8004a40 comp rx=0 tx=0).stop 2024-03-23T03:47:42.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.909+0000 7f09c4c00640 1 -- 172.21.15.102:0/2645030773 shutdown_connections 2024-03-23T03:47:42.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.909+0000 7f09c4c00640 1 --2- 172.21.15.102:0/2645030773 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f09a006ac00 0x7f09a006d0c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:42.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.909+0000 7f09c4c00640 1 --2- 172.21.15.102:0/2645030773 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09c0145b40 0x7f09c01f2250 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:42.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.909+0000 7f09c4c00640 1 --2- 172.21.15.102:0/2645030773 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09c00b43b0 0x7f09c00cfcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:42.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.909+0000 7f09c4c00640 1 --2- 172.21.15.102:0/2645030773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09c0069c90 0x7f09c00cf7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:42.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.909+0000 7f09c4c00640 1 -- 172.21.15.102:0/2645030773 >> 172.21.15.102:0/2645030773 conn(0x7f09c00b48e0 msgr2=0x7f09c00ad8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:42.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.910+0000 7f09c4c00640 1 -- 172.21.15.102:0/2645030773 shutdown_connections 2024-03-23T03:47:42.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:42.910+0000 7f09c4c00640 1 -- 172.21.15.102:0/2645030773 wait complete. 2024-03-23T03:47:42.929 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:46:52.158650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:46:52.914275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-23T03:46:55.983754+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-23T03:47:42.929 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:47:43.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.142+0000 7fd680600640 1 Processor -- start 2024-03-23T03:47:43.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.142+0000 7fd680600640 1 -- start start 2024-03-23T03:47:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.143+0000 7fd680600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd6780ce930 0x7fd6780add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.143+0000 7fd680600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd6780cd3e0 0x7fd6780ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.144+0000 7fd680600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd67806ce90 0x7fd67806d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.144+0000 7fd680600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd67806c760 con 0x7fd6780cd3e0 2024-03-23T03:47:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.144+0000 7fd680600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6780b42d0 con 0x7fd6780ce930 2024-03-23T03:47:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.144+0000 7fd680600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6780cf250 con 0x7fd67806ce90 2024-03-23T03:47:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.144+0000 7fd67d200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd6780cd3e0 0x7fd6780ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.144+0000 7fd67d200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd6780cd3e0 0x7fd6780ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33136/0 (socket says 172.21.15.102:33136) 2024-03-23T03:47:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.144+0000 7fd67d200640 1 -- 172.21.15.102:0/2523489808 learned_addr learned my addr 172.21.15.102:0/2523489808 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.144+0000 7fd67d200640 1 -- 172.21.15.102:0/2523489808 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd67806ce90 msgr2=0x7fd67806d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:43.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.144+0000 7fd67d200640 1 --2- 172.21.15.102:0/2523489808 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd67806ce90 0x7fd67806d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.144+0000 7fd67d200640 1 -- 172.21.15.102:0/2523489808 >> v2:172.21.15.173:3300/0 conn(0x7fd6780ce930 msgr2=0x7fd6780add30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:43.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.144+0000 7fd67d200640 1 --2- 172.21.15.102:0/2523489808 >> v2:172.21.15.173:3300/0 conn(0x7fd6780ce930 0x7fd6780add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.144+0000 7fd67d200640 1 -- 172.21.15.102:0/2523489808 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6780cdba0 con 0x7fd6780cd3e0 2024-03-23T03:47:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.145+0000 7fd67d200640 1 --2- 172.21.15.102:0/2523489808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6780cd3e0 0x7fd6780ced10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd6780cf540 tx=0x7fd66402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc3d5828cddb4521 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.146+0000 7fd677e00640 1 -- 172.21.15.102:0/2523489808 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664031860 con 0x7fd6780cd3e0 2024-03-23T03:47:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.146+0000 7fd677e00640 1 -- 172.21.15.102:0/2523489808 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6640319c0 con 0x7fd6780cd3e0 2024-03-23T03:47:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.146+0000 7fd677e00640 1 -- 172.21.15.102:0/2523489808 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664031d50 con 0x7fd6780cd3e0 2024-03-23T03:47:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.146+0000 7fd680600640 1 -- 172.21.15.102:0/2523489808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6780cd3e0 msgr2=0x7fd6780ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.146+0000 7fd680600640 1 --2- 172.21.15.102:0/2523489808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6780cd3e0 0x7fd6780ced10 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd6780cf540 tx=0x7fd66402b900 comp rx=0 tx=0).stop 2024-03-23T03:47:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.146+0000 7fd680600640 1 -- 172.21.15.102:0/2523489808 shutdown_connections 2024-03-23T03:47:43.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.146+0000 7fd680600640 1 --2- 172.21.15.102:0/2523489808 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd67806ce90 0x7fd67806d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.146+0000 7fd680600640 1 --2- 172.21.15.102:0/2523489808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6780cd3e0 0x7fd6780ced10 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.146+0000 7fd680600640 1 --2- 172.21.15.102:0/2523489808 >> v2:172.21.15.173:3300/0 conn(0x7fd6780ce930 0x7fd6780add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.146+0000 7fd680600640 1 -- 172.21.15.102:0/2523489808 >> 172.21.15.102:0/2523489808 conn(0x7fd67806a0b0 msgr2=0x7fd67806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:43.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.146+0000 7fd680600640 1 -- 172.21.15.102:0/2523489808 shutdown_connections 2024-03-23T03:47:43.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.146+0000 7fd680600640 1 -- 172.21.15.102:0/2523489808 wait complete. 2024-03-23T03:47:43.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.148+0000 7fd680600640 1 Processor -- start 2024-03-23T03:47:43.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.148+0000 7fd680600640 1 -- start start 2024-03-23T03:47:43.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.149+0000 7fd680600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd67806ce90 0x7fd6780afac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:43.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.149+0000 7fd680600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6780cd3e0 0x7fd678145e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:43.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.149+0000 7fd680600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd6780ce930 0x7fd678146380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:43.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.149+0000 7fd680600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6781f07d0 con 0x7fd6780cd3e0 2024-03-23T03:47:43.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.149+0000 7fd680600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6781468c0 con 0x7fd67806ce90 2024-03-23T03:47:43.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.149+0000 7fd680600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd678154a80 con 0x7fd6780ce930 2024-03-23T03:47:43.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.149+0000 7fd67d200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6780cd3e0 0x7fd678145e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.149+0000 7fd67d200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6780cd3e0 0x7fd678145e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33144/0 (socket says 172.21.15.102:33144) 2024-03-23T03:47:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.149+0000 7fd67d200640 1 -- 172.21.15.102:0/4071823572 learned_addr learned my addr 172.21.15.102:0/4071823572 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.150+0000 7fd67dc00640 1 --2- 172.21.15.102:0/4071823572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd67806ce90 0x7fd6780afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.150+0000 7fd67e600640 1 --2- 172.21.15.102:0/4071823572 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd6780ce930 0x7fd678146380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.150+0000 7fd67d200640 1 -- 172.21.15.102:0/4071823572 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd6780ce930 msgr2=0x7fd678146380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.150+0000 7fd67d200640 1 --2- 172.21.15.102:0/4071823572 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd6780ce930 0x7fd678146380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.150+0000 7fd67d200640 1 -- 172.21.15.102:0/4071823572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd67806ce90 msgr2=0x7fd6780afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.150+0000 7fd67d200640 1 --2- 172.21.15.102:0/4071823572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd67806ce90 0x7fd6780afac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.150+0000 7fd67d200640 1 -- 172.21.15.102:0/4071823572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6640094d0 con 0x7fd6780cd3e0 2024-03-23T03:47:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.150+0000 7fd67dc00640 1 --2- 172.21.15.102:0/4071823572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd67806ce90 0x7fd6780afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.150+0000 7fd67d200640 1 --2- 172.21.15.102:0/4071823572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6780cd3e0 0x7fd678145e40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd664000c00 tx=0x7fd6640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.150+0000 7fd675600640 1 -- 172.21.15.102:0/4071823572 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664031830 con 0x7fd6780cd3e0 2024-03-23T03:47:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.151+0000 7fd675600640 1 -- 172.21.15.102:0/4071823572 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6640443f0 con 0x7fd6780cd3e0 2024-03-23T03:47:43.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.151+0000 7fd675600640 1 -- 172.21.15.102:0/4071823572 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6640447a0 con 0x7fd6780cd3e0 2024-03-23T03:47:43.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.151+0000 7fd680600640 1 -- 172.21.15.102:0/4071823572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd678154d60 con 0x7fd6780cd3e0 2024-03-23T03:47:43.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.151+0000 7fd680600640 1 -- 172.21.15.102:0/4071823572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd678155320 con 0x7fd6780cd3e0 2024-03-23T03:47:43.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.152+0000 7fd680600640 1 -- 172.21.15.102:0/4071823572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd678073390 con 0x7fd6780cd3e0 2024-03-23T03:47:43.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.153+0000 7fd675600640 1 -- 172.21.15.102:0/4071823572 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd664044900 con 0x7fd6780cd3e0 2024-03-23T03:47:43.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.153+0000 7fd675600640 1 --2- 172.21.15.102:0/4071823572 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd65806ac00 0x7fd65806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:43.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.154+0000 7fd675600640 1 -- 172.21.15.102:0/4071823572 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd6640abf30 con 0x7fd6780cd3e0 2024-03-23T03:47:43.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.154+0000 7fd67dc00640 1 --2- 172.21.15.102:0/4071823572 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd65806ac00 0x7fd65806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:43.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.154+0000 7fd67dc00640 1 --2- 172.21.15.102:0/4071823572 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd65806ac00 0x7fd65806d0c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd66c006fd0 tx=0x7fd66c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:43.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.158+0000 7fd675600640 1 -- 172.21.15.102:0/4071823572 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd664076cb0 con 0x7fd6780cd3e0 2024-03-23T03:47:43.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.351+0000 7fd680600640 1 -- 172.21.15.102:0/4071823572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd67806c540 con 0x7fd6780cd3e0 2024-03-23T03:47:43.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.352+0000 7fd675600640 1 -- 172.21.15.102:0/4071823572 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd66407a2d0 con 0x7fd6780cd3e0 2024-03-23T03:47:43.353 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:47:43.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.354+0000 7fd680600640 1 -- 172.21.15.102:0/4071823572 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd65806ac00 msgr2=0x7fd65806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:43.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.354+0000 7fd680600640 1 --2- 172.21.15.102:0/4071823572 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd65806ac00 0x7fd65806d0c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd66c006fd0 tx=0x7fd66c008040 comp rx=0 tx=0).stop 2024-03-23T03:47:43.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.354+0000 7fd680600640 1 -- 172.21.15.102:0/4071823572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6780cd3e0 msgr2=0x7fd678145e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:43.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.355+0000 7fd680600640 1 --2- 172.21.15.102:0/4071823572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6780cd3e0 0x7fd678145e40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd664000c00 tx=0x7fd6640046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.355+0000 7fd680600640 1 -- 172.21.15.102:0/4071823572 shutdown_connections 2024-03-23T03:47:43.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.355+0000 7fd680600640 1 --2- 172.21.15.102:0/4071823572 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd65806ac00 0x7fd65806d0c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.355+0000 7fd680600640 1 --2- 172.21.15.102:0/4071823572 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd6780ce930 0x7fd678146380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.355+0000 7fd680600640 1 --2- 172.21.15.102:0/4071823572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd6780cd3e0 0x7fd678145e40 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.355+0000 7fd680600640 1 --2- 172.21.15.102:0/4071823572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd67806ce90 0x7fd6780afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.355+0000 7fd680600640 1 -- 172.21.15.102:0/4071823572 >> 172.21.15.102:0/4071823572 conn(0x7fd67806a0b0 msgr2=0x7fd67815e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:43.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.355+0000 7fd680600640 1 -- 172.21.15.102:0/4071823572 shutdown_connections 2024-03-23T03:47:43.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.355+0000 7fd680600640 1 -- 172.21.15.102:0/4071823572 wait complete. 2024-03-23T03:47:43.373 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:47:43.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.584+0000 7f047f200640 1 Processor -- start 2024-03-23T03:47:43.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.584+0000 7f047f200640 1 -- start start 2024-03-23T03:47:43.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.585+0000 7f047f200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f04800b22b0 0x7f048015eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:43.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.586+0000 7f047f200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f048015e320 con 0x7f048015a850 2024-03-23T03:47:43.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.586+0000 7f047f200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04800cda20 con 0x7f048015f020 2024-03-23T03:47:43.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.586+0000 7f047f200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048015ac30 con 0x7f04800b22b0 2024-03-23T03:47:43.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.586+0000 7f047de00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f048015a850 0x7f04800cd1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47116/0 (socket says 172.21.15.102:47116) 2024-03-23T03:47:43.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.586+0000 7f047de00640 1 -- 172.21.15.102:0/164717812 learned_addr learned my addr 172.21.15.102:0/164717812 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:43.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.586+0000 7f047d400640 1 -- 172.21.15.102:0/164717812 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517683439 0 0) 0x7f048015e320 con 0x7f048015a850 2024-03-23T03:47:43.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.586+0000 7f0475400640 1 --2- 172.21.15.102:0/164717812 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f04800b22b0 0x7f048015eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:43.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.586+0000 7f047d400640 1 -- 172.21.15.102:0/164717812 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0464003520 con 0x7f048015a850 2024-03-23T03:47:43.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.587+0000 7f047d400640 1 -- 172.21.15.102:0/164717812 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2287914090 0 0) 0x7f0464003520 con 0x7f048015a850 2024-03-23T03:47:43.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.587+0000 7f047d400640 1 -- 172.21.15.102:0/164717812 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f04800b22b0 msgr2=0x7f048015eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:43.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.587+0000 7f047d400640 1 --2- 172.21.15.102:0/164717812 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f04800b22b0 0x7f048015eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.587+0000 7f047d400640 1 -- 172.21.15.102:0/164717812 >> v1:172.21.15.173:6789/0 conn(0x7f048015f020 legacy=0x7f04800cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:43.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.587+0000 7f047d400640 1 -- 172.21.15.102:0/164717812 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04800cdd00 con 0x7f048015a850 2024-03-23T03:47:43.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.587+0000 7f047d400640 1 -- 172.21.15.102:0/164717812 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f0470001670 con 0x7f048015a850 2024-03-23T03:47:43.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.587+0000 7f047d400640 1 -- 172.21.15.102:0/164717812 >> v1:172.21.15.102:6789/0 conn(0x7f048015a850 legacy=0x7f04800cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:43.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.587+0000 7f047d400640 1 --2- 172.21.15.102:0/164717812 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04640039a0 0x7f0464003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.587+0000 7f047d400640 1 --2- 172.21.15.102:0/164717812 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f04800b22b0 0x7f04640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.587+0000 7f047d400640 1 --2- 172.21.15.102:0/164717812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0464004d00 0x7f0464005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.587+0000 7f047d400640 1 -- 172.21.15.102:0/164717812 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048015ac30 con 0x7f04640039a0 2024-03-23T03:47:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.587+0000 7f047d400640 1 -- 172.21.15.102:0/164717812 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046400ba30 con 0x7f0464004d00 2024-03-23T03:47:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.587+0000 7f047d400640 1 -- 172.21.15.102:0/164717812 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046400bbf0 con 0x7f04800b22b0 2024-03-23T03:47:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.587+0000 7f047de00640 1 --2- 172.21.15.102:0/164717812 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04640039a0 0x7f0464003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.587+0000 7f0475400640 1 --2- 172.21.15.102:0/164717812 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f04800b22b0 0x7f04640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.588+0000 7f047e800640 1 --2- 172.21.15.102:0/164717812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0464004d00 0x7f0464005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.588+0000 7f047de00640 1 -- 172.21.15.102:0/164717812 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f04800b22b0 msgr2=0x7f04640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.588+0000 7f047de00640 1 --2- 172.21.15.102:0/164717812 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f04800b22b0 0x7f04640047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.588+0000 7f047de00640 1 -- 172.21.15.102:0/164717812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0464004d00 msgr2=0x7f0464005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.588+0000 7f047de00640 1 --2- 172.21.15.102:0/164717812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0464004d00 0x7f0464005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.588+0000 7f047de00640 1 -- 172.21.15.102:0/164717812 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0464003520 con 0x7f04640039a0 2024-03-23T03:47:43.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.588+0000 7f047e800640 1 --2- 172.21.15.102:0/164717812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0464004d00 0x7f0464005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:43.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.589+0000 7f047de00640 1 --2- 172.21.15.102:0/164717812 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04640039a0 0x7f0464003dc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0470007fd0 tx=0x7f04640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7000307c7906f28 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:43.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.589+0000 7f047d400640 1 -- 172.21.15.102:0/164717812 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0470005ad0 con 0x7f04640039a0 2024-03-23T03:47:43.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.589+0000 7f047d400640 1 -- 172.21.15.102:0/164717812 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0470005c30 con 0x7f04640039a0 2024-03-23T03:47:43.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.589+0000 7f047d400640 1 -- 172.21.15.102:0/164717812 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f047002c870 con 0x7f04640039a0 2024-03-23T03:47:43.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.590+0000 7f047f200640 1 -- 172.21.15.102:0/164717812 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04640039a0 msgr2=0x7f0464003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:43.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.590+0000 7f047f200640 1 --2- 172.21.15.102:0/164717812 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04640039a0 0x7f0464003dc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0470007fd0 tx=0x7f04640057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.590+0000 7f047e800640 1 -- 172.21.15.102:0/164717812 reap_dead start 2024-03-23T03:47:43.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.590+0000 7f047f200640 1 -- 172.21.15.102:0/164717812 shutdown_connections 2024-03-23T03:47:43.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.590+0000 7f047f200640 1 -- 172.21.15.102:0/164717812 >> 172.21.15.102:0/164717812 conn(0x7f04800b01b0 msgr2=0x7f04800b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:43.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.590+0000 7f047f200640 1 -- 172.21.15.102:0/164717812 shutdown_connections 2024-03-23T03:47:43.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.590+0000 7f047f200640 1 -- 172.21.15.102:0/164717812 wait complete. 2024-03-23T03:47:43.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.591+0000 7f047f200640 1 Processor -- start 2024-03-23T03:47:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.592+0000 7f047f200640 1 -- start start 2024-03-23T03:47:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.592+0000 7f047f200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04640039a0 0x7f048014d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.593+0000 7f047f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f048015ae80 0x7f0480149ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.593+0000 7f047f200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f048014a410 0x7f0480071440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.593+0000 7f047f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048014d900 con 0x7f048015ae80 2024-03-23T03:47:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.593+0000 7f047f200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048014a920 con 0x7f04640039a0 2024-03-23T03:47:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.593+0000 7f047f200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f048014abc0 con 0x7f048014a410 2024-03-23T03:47:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.593+0000 7f0475400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f048015ae80 0x7f0480149ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.593+0000 7f0475400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f048015ae80 0x7f0480149ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33164/0 (socket says 172.21.15.102:33164) 2024-03-23T03:47:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.593+0000 7f0475400640 1 -- 172.21.15.102:0/1268315124 learned_addr learned my addr 172.21.15.102:0/1268315124 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:43.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.593+0000 7f0475400640 1 -- 172.21.15.102:0/1268315124 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f048014a410 msgr2=0x7f0480071440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.593+0000 7f0475400640 1 --2- 172.21.15.102:0/1268315124 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f048014a410 0x7f0480071440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.593+0000 7f0475400640 1 -- 172.21.15.102:0/1268315124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04640039a0 msgr2=0x7f048014d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.593+0000 7f0475400640 1 --2- 172.21.15.102:0/1268315124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04640039a0 0x7f048014d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.593+0000 7f0475400640 1 -- 172.21.15.102:0/1268315124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f048014aea0 con 0x7f048015ae80 2024-03-23T03:47:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.594+0000 7f0475400640 1 --2- 172.21.15.102:0/1268315124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f048015ae80 0x7f0480149ed0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0468011600 tx=0x7f0468011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.594+0000 7f0476a00640 1 -- 172.21.15.102:0/1268315124 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046800f920 con 0x7f048015ae80 2024-03-23T03:47:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.594+0000 7f0476a00640 1 -- 172.21.15.102:0/1268315124 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046800fa80 con 0x7f048015ae80 2024-03-23T03:47:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.594+0000 7f047f200640 1 -- 172.21.15.102:0/1268315124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048014b180 con 0x7f048015ae80 2024-03-23T03:47:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.594+0000 7f0476a00640 1 -- 172.21.15.102:0/1268315124 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046800fdf0 con 0x7f048015ae80 2024-03-23T03:47:43.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.594+0000 7f047f200640 1 -- 172.21.15.102:0/1268315124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f048014b740 con 0x7f048015ae80 2024-03-23T03:47:43.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.595+0000 7f047f200640 1 -- 172.21.15.102:0/1268315124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f048004e030 con 0x7f048015ae80 2024-03-23T03:47:43.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.596+0000 7f0476a00640 1 -- 172.21.15.102:0/1268315124 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0468015460 con 0x7f048015ae80 2024-03-23T03:47:43.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.597+0000 7f0476a00640 1 --2- 172.21.15.102:0/1268315124 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f045c06ac00 0x7f045c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:43.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.597+0000 7f0476a00640 1 -- 172.21.15.102:0/1268315124 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0468089b10 con 0x7f048015ae80 2024-03-23T03:47:43.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.597+0000 7f047de00640 1 --2- 172.21.15.102:0/1268315124 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f045c06ac00 0x7f045c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:43.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.598+0000 7f047de00640 1 --2- 172.21.15.102:0/1268315124 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f045c06ac00 0x7f045c06d0c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0470034000 tx=0x7f0470039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:43.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.601+0000 7f0476a00640 1 -- 172.21.15.102:0/1268315124 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0468054920 con 0x7f048015ae80 2024-03-23T03:47:43.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.797+0000 7f047f200640 1 -- 172.21.15.102:0/1268315124 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f04800720e0 con 0x7f048015ae80 2024-03-23T03:47:43.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.798+0000 7f0476a00640 1 -- 172.21.15.102:0/1268315124 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0468057f40 con 0x7f048015ae80 2024-03-23T03:47:43.797 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:43.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.801+0000 7f047f200640 1 -- 172.21.15.102:0/1268315124 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f045c06ac00 msgr2=0x7f045c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:43.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.801+0000 7f047f200640 1 --2- 172.21.15.102:0/1268315124 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f045c06ac00 0x7f045c06d0c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0470034000 tx=0x7f0470039040 comp rx=0 tx=0).stop 2024-03-23T03:47:43.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.801+0000 7f047f200640 1 -- 172.21.15.102:0/1268315124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f048015ae80 msgr2=0x7f0480149ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:43.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.801+0000 7f047f200640 1 --2- 172.21.15.102:0/1268315124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f048015ae80 0x7f0480149ed0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0468011600 tx=0x7f0468011ac0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.801+0000 7f047f200640 1 -- 172.21.15.102:0/1268315124 shutdown_connections 2024-03-23T03:47:43.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.801+0000 7f047f200640 1 --2- 172.21.15.102:0/1268315124 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f045c06ac00 0x7f045c06d0c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.801+0000 7f047f200640 1 --2- 172.21.15.102:0/1268315124 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f048014a410 0x7f0480071440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.801+0000 7f047f200640 1 --2- 172.21.15.102:0/1268315124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f048015ae80 0x7f0480149ed0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.801+0000 7f047f200640 1 --2- 172.21.15.102:0/1268315124 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f04640039a0 0x7f048014d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:43.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.801+0000 7f047f200640 1 -- 172.21.15.102:0/1268315124 >> 172.21.15.102:0/1268315124 conn(0x7f04800b01b0 msgr2=0x7f04800b1500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:43.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.801+0000 7f047f200640 1 -- 172.21.15.102:0/1268315124 shutdown_connections 2024-03-23T03:47:43.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:43.801+0000 7f047f200640 1 -- 172.21.15.102:0/1268315124 wait complete. 2024-03-23T03:47:43.819 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:47:44.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.030+0000 7fc908000640 1 Processor -- start 2024-03-23T03:47:44.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.031+0000 7fc908000640 1 -- start start 2024-03-23T03:47:44.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.032+0000 7fc908000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9000cd740 0x7fc9000b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.032+0000 7fc908000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000d20c0 con 0x7fc9000b5aa0 2024-03-23T03:47:44.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.032+0000 7fc908000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000d14b0 con 0x7fc9000d0d40 2024-03-23T03:47:44.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.032+0000 7fc908000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000d1790 con 0x7fc9000cd740 2024-03-23T03:47:44.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.032+0000 7fc906000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc9000b5aa0 0x7fc9000b5f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47120/0 (socket says 172.21.15.102:47120) 2024-03-23T03:47:44.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.032+0000 7fc906000640 1 -- 172.21.15.102:0/1865602302 learned_addr learned my addr 172.21.15.102:0/1865602302 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:44.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.033+0000 7fc905600640 1 --2- 172.21.15.102:0/1865602302 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9000cd740 0x7fc9000b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.033+0000 7fc8ffe00640 1 -- 172.21.15.102:0/1865602302 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332815850 0 0) 0x7fc9000d20c0 con 0x7fc9000b5aa0 2024-03-23T03:47:44.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.033+0000 7fc8ffe00640 1 -- 172.21.15.102:0/1865602302 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e4003520 con 0x7fc9000b5aa0 2024-03-23T03:47:44.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.033+0000 7fc8ffe00640 1 -- 172.21.15.102:0/1865602302 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3087483055 0 0) 0x7fc8e4003520 con 0x7fc9000b5aa0 2024-03-23T03:47:44.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.033+0000 7fc8ffe00640 1 -- 172.21.15.102:0/1865602302 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9000cd740 msgr2=0x7fc9000b84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:44.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.033+0000 7fc8ffe00640 1 --2- 172.21.15.102:0/1865602302 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9000cd740 0x7fc9000b84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.033+0000 7fc8ffe00640 1 -- 172.21.15.102:0/1865602302 >> v1:172.21.15.173:6789/0 conn(0x7fc9000d0d40 legacy=0x7fc9000cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:44.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.033+0000 7fc8ffe00640 1 -- 172.21.15.102:0/1865602302 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9000d1a70 con 0x7fc9000b5aa0 2024-03-23T03:47:44.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.033+0000 7fc8ffe00640 1 -- 172.21.15.102:0/1865602302 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fc8f80019f0 con 0x7fc9000b5aa0 2024-03-23T03:47:44.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.033+0000 7fc8ffe00640 1 -- 172.21.15.102:0/1865602302 >> v1:172.21.15.102:6789/0 conn(0x7fc9000b5aa0 legacy=0x7fc9000b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:44.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.033+0000 7fc8ffe00640 1 --2- 172.21.15.102:0/1865602302 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9000cd740 0x7fc8e40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.033+0000 7fc8ffe00640 1 --2- 172.21.15.102:0/1865602302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8e4003f70 0x7fc8e40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.033+0000 7fc8ffe00640 1 --2- 172.21.15.102:0/1865602302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e4004d00 0x7fc8e4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.033+0000 7fc8ffe00640 1 -- 172.21.15.102:0/1865602302 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000d1790 con 0x7fc8e4004d00 2024-03-23T03:47:44.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.033+0000 7fc8ffe00640 1 -- 172.21.15.102:0/1865602302 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e400bac0 con 0x7fc8e4003f70 2024-03-23T03:47:44.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.033+0000 7fc8ffe00640 1 -- 172.21.15.102:0/1865602302 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e400bc80 con 0x7fc9000cd740 2024-03-23T03:47:44.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.034+0000 7fc906000640 1 --2- 172.21.15.102:0/1865602302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e4004d00 0x7fc8e4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.034+0000 7fc905600640 1 --2- 172.21.15.102:0/1865602302 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9000cd740 0x7fc8e40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.034+0000 7fc904c00640 1 --2- 172.21.15.102:0/1865602302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8e4003f70 0x7fc8e40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.034+0000 7fc906000640 1 -- 172.21.15.102:0/1865602302 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9000cd740 msgr2=0x7fc8e40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:44.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.034+0000 7fc906000640 1 --2- 172.21.15.102:0/1865602302 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc9000cd740 0x7fc8e40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.034+0000 7fc906000640 1 -- 172.21.15.102:0/1865602302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8e4003f70 msgr2=0x7fc8e40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:44.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.034+0000 7fc906000640 1 --2- 172.21.15.102:0/1865602302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8e4003f70 0x7fc8e40047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.034+0000 7fc906000640 1 -- 172.21.15.102:0/1865602302 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e4003520 con 0x7fc8e4004d00 2024-03-23T03:47:44.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.035+0000 7fc906000640 1 --2- 172.21.15.102:0/1865602302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e4004d00 0x7fc8e4005180 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800a760 tx=0x7fc8e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d477275863971382 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:44.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.035+0000 7fc8ffe00640 1 -- 172.21.15.102:0/1865602302 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8034630 con 0x7fc8e4004d00 2024-03-23T03:47:44.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.035+0000 7fc8ffe00640 1 -- 172.21.15.102:0/1865602302 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f802f070 con 0x7fc8e4004d00 2024-03-23T03:47:44.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.035+0000 7fc8ffe00640 1 -- 172.21.15.102:0/1865602302 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8034a50 con 0x7fc8e4004d00 2024-03-23T03:47:44.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.036+0000 7fc908000640 1 -- 172.21.15.102:0/1865602302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e4004d00 msgr2=0x7fc8e4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:44.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.036+0000 7fc908000640 1 --2- 172.21.15.102:0/1865602302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8e4004d00 0x7fc8e4005180 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc8f800a760 tx=0x7fc8e40057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.036+0000 7fc906000640 1 -- 172.21.15.102:0/1865602302 reap_dead start 2024-03-23T03:47:44.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.036+0000 7fc908000640 1 -- 172.21.15.102:0/1865602302 shutdown_connections 2024-03-23T03:47:44.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.036+0000 7fc908000640 1 -- 172.21.15.102:0/1865602302 >> 172.21.15.102:0/1865602302 conn(0x7fc9000cf5d0 msgr2=0x7fc9000cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:44.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.036+0000 7fc908000640 1 -- 172.21.15.102:0/1865602302 shutdown_connections 2024-03-23T03:47:44.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.036+0000 7fc908000640 1 -- 172.21.15.102:0/1865602302 wait complete. 2024-03-23T03:47:44.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.038+0000 7fc908000640 1 Processor -- start 2024-03-23T03:47:44.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.038+0000 7fc908000640 1 -- start start 2024-03-23T03:47:44.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.039+0000 7fc908000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8e4004d00 0x7fc9001463a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.039+0000 7fc908000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc900147750 0x7fc900144ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.039+0000 7fc908000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc900146db0 0x7fc9000c7870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.039+0000 7fc908000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9001468e0 con 0x7fc900147750 2024-03-23T03:47:44.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.039+0000 7fc908000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc900145300 con 0x7fc8e4004d00 2024-03-23T03:47:44.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.039+0000 7fc908000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9001455e0 con 0x7fc900146db0 2024-03-23T03:47:44.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.039+0000 7fc904c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc900147750 0x7fc900144ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.039+0000 7fc904c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc900147750 0x7fc900144ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33186/0 (socket says 172.21.15.102:33186) 2024-03-23T03:47:44.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.039+0000 7fc904c00640 1 -- 172.21.15.102:0/4183344767 learned_addr learned my addr 172.21.15.102:0/4183344767 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:44.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.039+0000 7fc904c00640 1 -- 172.21.15.102:0/4183344767 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc900146db0 msgr2=0x7fc9000c7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:44.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.040+0000 7fc906000640 1 --2- 172.21.15.102:0/4183344767 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc900146db0 0x7fc9000c7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.040+0000 7fc904c00640 1 --2- 172.21.15.102:0/4183344767 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc900146db0 0x7fc9000c7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.040+0000 7fc905600640 1 --2- 172.21.15.102:0/4183344767 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8e4004d00 0x7fc9001463a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.040+0000 7fc904c00640 1 -- 172.21.15.102:0/4183344767 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8e4004d00 msgr2=0x7fc9001463a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:44.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.040+0000 7fc904c00640 1 --2- 172.21.15.102:0/4183344767 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8e4004d00 0x7fc9001463a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.040+0000 7fc904c00640 1 -- 172.21.15.102:0/4183344767 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9001458c0 con 0x7fc900147750 2024-03-23T03:47:44.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.040+0000 7fc906000640 1 --2- 172.21.15.102:0/4183344767 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc900146db0 0x7fc9000c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:44.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.040+0000 7fc904c00640 1 --2- 172.21.15.102:0/4183344767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc900147750 0x7fc900144ab0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec001eb0 tx=0x7fc8ec00d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:44.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.040+0000 7fc8fd600640 1 -- 172.21.15.102:0/4183344767 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec005480 con 0x7fc900147750 2024-03-23T03:47:44.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.040+0000 7fc8fd600640 1 -- 172.21.15.102:0/4183344767 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ec003610 con 0x7fc900147750 2024-03-23T03:47:44.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.040+0000 7fc8fd600640 1 -- 172.21.15.102:0/4183344767 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ec003980 con 0x7fc900147750 2024-03-23T03:47:44.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.040+0000 7fc908000640 1 -- 172.21.15.102:0/4183344767 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc900145ba0 con 0x7fc900147750 2024-03-23T03:47:44.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.040+0000 7fc908000640 1 -- 172.21.15.102:0/4183344767 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc900144ff0 con 0x7fc900147750 2024-03-23T03:47:44.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.042+0000 7fc908000640 1 -- 172.21.15.102:0/4183344767 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc900073390 con 0x7fc900147750 2024-03-23T03:47:44.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.042+0000 7fc8fd600640 1 -- 172.21.15.102:0/4183344767 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc8ec003ae0 con 0x7fc900147750 2024-03-23T03:47:44.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.043+0000 7fc8fd600640 1 --2- 172.21.15.102:0/4183344767 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc8e006ac00 0x7fc8e006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.043+0000 7fc8fd600640 1 -- 172.21.15.102:0/4183344767 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc8ec089630 con 0x7fc900147750 2024-03-23T03:47:44.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.044+0000 7fc905600640 1 --2- 172.21.15.102:0/4183344767 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc8e006ac00 0x7fc8e006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.044+0000 7fc905600640 1 --2- 172.21.15.102:0/4183344767 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc8e006ac00 0x7fc8e006d0c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0009950 tx=0x7fc8f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:44.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.048+0000 7fc8fd600640 1 -- 172.21.15.102:0/4183344767 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc8ec054430 con 0x7fc900147750 2024-03-23T03:47:44.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.244+0000 7fc908000640 1 -- 172.21.15.102:0/4183344767 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc9000cdfe0 con 0x7fc900147750 2024-03-23T03:47:44.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.245+0000 7fc8fd600640 1 -- 172.21.15.102:0/4183344767 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc8ec057a50 con 0x7fc900147750 2024-03-23T03:47:44.244 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:47:44.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.247+0000 7fc908000640 1 -- 172.21.15.102:0/4183344767 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc8e006ac00 msgr2=0x7fc8e006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:44.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.247+0000 7fc908000640 1 --2- 172.21.15.102:0/4183344767 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc8e006ac00 0x7fc8e006d0c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc8f0009950 tx=0x7fc8f0008040 comp rx=0 tx=0).stop 2024-03-23T03:47:44.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.247+0000 7fc908000640 1 -- 172.21.15.102:0/4183344767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc900147750 msgr2=0x7fc900144ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:44.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.247+0000 7fc908000640 1 --2- 172.21.15.102:0/4183344767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc900147750 0x7fc900144ab0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec001eb0 tx=0x7fc8ec00d9d0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.247+0000 7fc908000640 1 -- 172.21.15.102:0/4183344767 shutdown_connections 2024-03-23T03:47:44.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.247+0000 7fc908000640 1 --2- 172.21.15.102:0/4183344767 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc8e006ac00 0x7fc8e006d0c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.247+0000 7fc908000640 1 --2- 172.21.15.102:0/4183344767 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc900146db0 0x7fc9000c7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.247+0000 7fc908000640 1 --2- 172.21.15.102:0/4183344767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc900147750 0x7fc900144ab0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.247+0000 7fc908000640 1 --2- 172.21.15.102:0/4183344767 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8e4004d00 0x7fc9001463a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.248+0000 7fc908000640 1 -- 172.21.15.102:0/4183344767 >> 172.21.15.102:0/4183344767 conn(0x7fc9000cf5d0 msgr2=0x7fc9000b6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:44.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.248+0000 7fc908000640 1 -- 172.21.15.102:0/4183344767 shutdown_connections 2024-03-23T03:47:44.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.248+0000 7fc908000640 1 -- 172.21.15.102:0/4183344767 wait complete. 2024-03-23T03:47:44.265 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-23T03:47:44.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.465+0000 7f3081600640 1 Processor -- start 2024-03-23T03:47:44.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.466+0000 7f3081600640 1 -- start start 2024-03-23T03:47:44.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.467+0000 7f3081600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f307c159cd0 0x7f307c0ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.467+0000 7f3081600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f307c0ad7a0 0x7f307c15a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.467+0000 7f3081600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f307c0c8c90 con 0x7f307c0cc200 2024-03-23T03:47:44.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.467+0000 7f3081600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f307c0cc6c0 con 0x7f307c159cd0 2024-03-23T03:47:44.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.467+0000 7f3081600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307c0cc9a0 con 0x7f307c0ad7a0 2024-03-23T03:47:44.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.468+0000 7f3080c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f307c0ad7a0 0x7f307c15a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.468+0000 7f307be00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f307c159cd0 0x7f307c0ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.468+0000 7f3080c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f307c0ad7a0 0x7f307c15a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:49756/0 (socket says 172.21.15.102:49756) 2024-03-23T03:47:44.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.468+0000 7f3080c00640 1 -- 172.21.15.102:0/925358866 learned_addr learned my addr 172.21.15.102:0/925358866 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:44.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.469+0000 7f307be00640 1 -- 172.21.15.102:0/925358866 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f307c0ad7a0 msgr2=0x7f307c15a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:44.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.469+0000 7f307be00640 1 --2- 172.21.15.102:0/925358866 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f307c0ad7a0 0x7f307c15a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.469+0000 7f307be00640 1 -- 172.21.15.102:0/925358866 >> v1:172.21.15.173:6789/0 conn(0x7f307c0cc200 legacy=0x7f307c0cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:44.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.469+0000 7f307be00640 1 -- 172.21.15.102:0/925358866 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307c0ccc80 con 0x7f307c159cd0 2024-03-23T03:47:44.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.470+0000 7f307be00640 1 --2- 172.21.15.102:0/925358866 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f307c159cd0 0x7f307c0ad260 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f306c0097a0 tx=0x7f306c02dae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e617cc70b0836ed server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:44.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.470+0000 7f307b400640 1 -- 172.21.15.102:0/925358866 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306c02f8a0 con 0x7f307c159cd0 2024-03-23T03:47:44.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.471+0000 7f307b400640 1 -- 172.21.15.102:0/925358866 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306c02fa00 con 0x7f307c159cd0 2024-03-23T03:47:44.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.471+0000 7f307b400640 1 -- 172.21.15.102:0/925358866 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f306c02fd90 con 0x7f307c159cd0 2024-03-23T03:47:44.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.471+0000 7f3081600640 1 -- 172.21.15.102:0/925358866 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f307c159cd0 msgr2=0x7f307c0ad260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:44.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.471+0000 7f3081600640 1 --2- 172.21.15.102:0/925358866 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f307c159cd0 0x7f307c0ad260 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f306c0097a0 tx=0x7f306c02dae0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.471+0000 7f3081600640 1 -- 172.21.15.102:0/925358866 shutdown_connections 2024-03-23T03:47:44.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.471+0000 7f3081600640 1 --2- 172.21.15.102:0/925358866 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f307c0ad7a0 0x7f307c15a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.471+0000 7f3081600640 1 --2- 172.21.15.102:0/925358866 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f307c159cd0 0x7f307c0ad260 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.471+0000 7f3081600640 1 -- 172.21.15.102:0/925358866 >> 172.21.15.102:0/925358866 conn(0x7f307c0b01b0 msgr2=0x7f307c0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:44.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.471+0000 7f3081600640 1 -- 172.21.15.102:0/925358866 shutdown_connections 2024-03-23T03:47:44.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.471+0000 7f3081600640 1 -- 172.21.15.102:0/925358866 wait complete. 2024-03-23T03:47:44.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.473+0000 7f3081600640 1 Processor -- start 2024-03-23T03:47:44.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.473+0000 7f3081600640 1 -- start start 2024-03-23T03:47:44.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.474+0000 7f3081600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f307c0ad7a0 0x7f307c1f0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.474+0000 7f3081600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f307c0cc200 0x7f307c157fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.474+0000 7f3081600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f307c159cd0 0x7f307c158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.474+0000 7f3081600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c156b30 con 0x7f307c159cd0 2024-03-23T03:47:44.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.474+0000 7f3081600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c156e10 con 0x7f307c0cc200 2024-03-23T03:47:44.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.474+0000 7f3081600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307c1570f0 con 0x7f307c0ad7a0 2024-03-23T03:47:44.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.475+0000 7f3080c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f307c159cd0 0x7f307c158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.475+0000 7f3080c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f307c159cd0 0x7f307c158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33194/0 (socket says 172.21.15.102:33194) 2024-03-23T03:47:44.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.475+0000 7f3080c00640 1 -- 172.21.15.102:0/2817356744 learned_addr learned my addr 172.21.15.102:0/2817356744 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:44.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.475+0000 7f307be00640 1 --2- 172.21.15.102:0/2817356744 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f307c0ad7a0 0x7f307c1f0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.475+0000 7f3073400640 1 --2- 172.21.15.102:0/2817356744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f307c0cc200 0x7f307c157fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.475+0000 7f3080c00640 1 -- 172.21.15.102:0/2817356744 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f307c0ad7a0 msgr2=0x7f307c1f0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:44.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.475+0000 7f3080c00640 1 --2- 172.21.15.102:0/2817356744 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f307c0ad7a0 0x7f307c1f0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.475+0000 7f3080c00640 1 -- 172.21.15.102:0/2817356744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f307c0cc200 msgr2=0x7f307c157fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:44.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.475+0000 7f3080c00640 1 --2- 172.21.15.102:0/2817356744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f307c0cc200 0x7f307c157fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.475+0000 7f3080c00640 1 -- 172.21.15.102:0/2817356744 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f306c009400 con 0x7f307c159cd0 2024-03-23T03:47:44.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.475+0000 7f307be00640 1 --2- 172.21.15.102:0/2817356744 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f307c0ad7a0 0x7f307c1f0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:44.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.475+0000 7f3073400640 1 --2- 172.21.15.102:0/2817356744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f307c0cc200 0x7f307c157fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:44.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.475+0000 7f3080c00640 1 --2- 172.21.15.102:0/2817356744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f307c159cd0 0x7f307c158500 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f307400efc0 tx=0x7f307400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:44.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.476+0000 7f3078c00640 1 -- 172.21.15.102:0/2817356744 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f307400cd20 con 0x7f307c159cd0 2024-03-23T03:47:44.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.476+0000 7f3078c00640 1 -- 172.21.15.102:0/2817356744 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f307400f040 con 0x7f307c159cd0 2024-03-23T03:47:44.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.476+0000 7f3078c00640 1 -- 172.21.15.102:0/2817356744 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3074003bd0 con 0x7f307c159cd0 2024-03-23T03:47:44.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.476+0000 7f3081600640 1 -- 172.21.15.102:0/2817356744 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f307c1573d0 con 0x7f307c159cd0 2024-03-23T03:47:44.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.476+0000 7f3081600640 1 -- 172.21.15.102:0/2817356744 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307c1f11b0 con 0x7f307c159cd0 2024-03-23T03:47:44.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.477+0000 7f3081600640 1 -- 172.21.15.102:0/2817356744 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307c1578f0 con 0x7f307c159cd0 2024-03-23T03:47:44.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.478+0000 7f3078c00640 1 -- 172.21.15.102:0/2817356744 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3074003d30 con 0x7f307c159cd0 2024-03-23T03:47:44.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.479+0000 7f3078c00640 1 --2- 172.21.15.102:0/2817356744 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f305806ac00 0x7f305806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.479+0000 7f3078c00640 1 -- 172.21.15.102:0/2817356744 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3074089d90 con 0x7f307c159cd0 2024-03-23T03:47:44.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.479+0000 7f307be00640 1 --2- 172.21.15.102:0/2817356744 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f305806ac00 0x7f305806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.480+0000 7f307be00640 1 --2- 172.21.15.102:0/2817356744 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f305806ac00 0x7f305806d0c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f306c038000 tx=0x7f306c03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:44.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.483+0000 7f3078c00640 1 -- 172.21.15.102:0/2817356744 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f307c1578f0 con 0x7f307c159cd0 2024-03-23T03:47:44.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.679+0000 7f3081600640 1 -- 172.21.15.102:0/2817356744 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f307c1578f0 con 0x7f307c159cd0 2024-03-23T03:47:44.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.680+0000 7f3078c00640 1 -- 172.21.15.102:0/2817356744 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f307c1578f0 con 0x7f307c159cd0 2024-03-23T03:47:44.680 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:44.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.685+0000 7f3081600640 1 -- 172.21.15.102:0/2817356744 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f305806ac00 msgr2=0x7f305806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:44.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.685+0000 7f3081600640 1 --2- 172.21.15.102:0/2817356744 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f305806ac00 0x7f305806d0c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f306c038000 tx=0x7f306c03b040 comp rx=0 tx=0).stop 2024-03-23T03:47:44.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.685+0000 7f3081600640 1 -- 172.21.15.102:0/2817356744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f307c159cd0 msgr2=0x7f307c158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:44.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.685+0000 7f3081600640 1 --2- 172.21.15.102:0/2817356744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f307c159cd0 0x7f307c158500 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f307400efc0 tx=0x7f307400c490 comp rx=0 tx=0).stop 2024-03-23T03:47:44.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.685+0000 7f3081600640 1 -- 172.21.15.102:0/2817356744 shutdown_connections 2024-03-23T03:47:44.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.685+0000 7f3081600640 1 --2- 172.21.15.102:0/2817356744 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f305806ac00 0x7f305806d0c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.685+0000 7f3081600640 1 --2- 172.21.15.102:0/2817356744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f307c159cd0 0x7f307c158500 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.685+0000 7f3081600640 1 --2- 172.21.15.102:0/2817356744 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f307c0cc200 0x7f307c157fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.685+0000 7f3081600640 1 --2- 172.21.15.102:0/2817356744 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f307c0ad7a0 0x7f307c1f0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.685+0000 7f3081600640 1 -- 172.21.15.102:0/2817356744 >> 172.21.15.102:0/2817356744 conn(0x7f307c0b01b0 msgr2=0x7f307c0b0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:44.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.685+0000 7f3081600640 1 -- 172.21.15.102:0/2817356744 shutdown_connections 2024-03-23T03:47:44.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.686+0000 7f3081600640 1 -- 172.21.15.102:0/2817356744 wait complete. 2024-03-23T03:47:44.706 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:44.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.911+0000 7ff7cea00640 1 Processor -- start 2024-03-23T03:47:44.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.911+0000 7ff7cea00640 1 -- start start 2024-03-23T03:47:44.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.913+0000 7ff7cea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff7c806a3a0 0x7ff7c80b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.913+0000 7ff7cea00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c80b4200 con 0x7ff7c815cdc0 2024-03-23T03:47:44.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.913+0000 7ff7cea00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c80b44e0 con 0x7ff7c815dcb0 2024-03-23T03:47:44.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.913+0000 7ff7cea00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c80b47c0 con 0x7ff7c806a3a0 2024-03-23T03:47:44.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.913+0000 7ff7c7e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff7c815cdc0 0x7ff7c80ae9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47132/0 (socket says 172.21.15.102:47132) 2024-03-23T03:47:44.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.913+0000 7ff7c7e00640 1 -- 172.21.15.102:0/1717444855 learned_addr learned my addr 172.21.15.102:0/1717444855 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:44.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.913+0000 7ff7cca00640 1 --2- 172.21.15.102:0/1717444855 >> v2:172.21.15.102:3300/0 conn(0x7ff7c806a3a0 0x7ff7c80b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.913+0000 7ff7c7400640 1 -- 172.21.15.102:0/1717444855 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564503744 0 0) 0x7ff7c80b4200 con 0x7ff7c815cdc0 2024-03-23T03:47:44.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.913+0000 7ff7c7400640 1 -- 172.21.15.102:0/1717444855 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7ac003520 con 0x7ff7c815cdc0 2024-03-23T03:47:44.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.913+0000 7ff7cca00640 1 -- 172.21.15.102:0/1717444855 >> v1:172.21.15.173:6789/0 conn(0x7ff7c815dcb0 legacy=0x7ff7c8069c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:44.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.913+0000 7ff7cca00640 1 -- 172.21.15.102:0/1717444855 >> v1:172.21.15.102:6789/0 conn(0x7ff7c815cdc0 legacy=0x7ff7c80ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:44.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.913+0000 7ff7cca00640 1 -- 172.21.15.102:0/1717444855 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c815d4b0 con 0x7ff7c806a3a0 2024-03-23T03:47:44.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.915+0000 7ff7cca00640 1 --2- 172.21.15.102:0/1717444855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7c806a3a0 0x7ff7c80b2670 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0009d90 tx=0x7ff7c002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5123bdbe3e6eae65 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:44.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.915+0000 7ff7c7400640 1 -- 172.21.15.102:0/1717444855 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c002cbd0 con 0x7ff7c806a3a0 2024-03-23T03:47:44.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.915+0000 7ff7c7400640 1 -- 172.21.15.102:0/1717444855 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c002cd30 con 0x7ff7c806a3a0 2024-03-23T03:47:44.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.915+0000 7ff7c7400640 1 -- 172.21.15.102:0/1717444855 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c002cea0 con 0x7ff7c806a3a0 2024-03-23T03:47:44.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.915+0000 7ff7cea00640 1 -- 172.21.15.102:0/1717444855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7c806a3a0 msgr2=0x7ff7c80b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:44.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.915+0000 7ff7cea00640 1 --2- 172.21.15.102:0/1717444855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7c806a3a0 0x7ff7c80b2670 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0009d90 tx=0x7ff7c002be20 comp rx=0 tx=0).stop 2024-03-23T03:47:44.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.916+0000 7ff7cea00640 1 -- 172.21.15.102:0/1717444855 shutdown_connections 2024-03-23T03:47:44.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.916+0000 7ff7cea00640 1 --2- 172.21.15.102:0/1717444855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7c806a3a0 0x7ff7c80b2670 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.916+0000 7ff7cea00640 1 -- 172.21.15.102:0/1717444855 >> 172.21.15.102:0/1717444855 conn(0x7ff7c80b1370 msgr2=0x7ff7c80b1780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:44.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.916+0000 7ff7cea00640 1 -- 172.21.15.102:0/1717444855 shutdown_connections 2024-03-23T03:47:44.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.916+0000 7ff7cea00640 1 -- 172.21.15.102:0/1717444855 wait complete. 2024-03-23T03:47:44.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.917+0000 7ff7cea00640 1 Processor -- start 2024-03-23T03:47:44.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.917+0000 7ff7cea00640 1 -- start start 2024-03-23T03:47:44.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.918+0000 7ff7cea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7c806a3a0 0x7ff7c8147d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.918+0000 7ff7cea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7c815cdc0 0x7ff7c80cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.918+0000 7ff7cea00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff7c815dcb0 0x7ff7c80a74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.918+0000 7ff7cea00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8149780 con 0x7ff7c815cdc0 2024-03-23T03:47:44.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.918+0000 7ff7cea00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8148b00 con 0x7ff7c806a3a0 2024-03-23T03:47:44.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.918+0000 7ff7cea00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8148de0 con 0x7ff7c815dcb0 2024-03-23T03:47:44.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.919+0000 7ff7bfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7c815cdc0 0x7ff7c80cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.919+0000 7ff7bfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7c815cdc0 0x7ff7c80cbc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33214/0 (socket says 172.21.15.102:33214) 2024-03-23T03:47:44.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.919+0000 7ff7bfe00640 1 -- 172.21.15.102:0/1605590120 learned_addr learned my addr 172.21.15.102:0/1605590120 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:44.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.919+0000 7ff7bfe00640 1 -- 172.21.15.102:0/1605590120 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff7c815dcb0 msgr2=0x7ff7c80a74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:44.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.919+0000 7ff7cca00640 1 --2- 172.21.15.102:0/1605590120 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff7c815dcb0 0x7ff7c80a74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.919+0000 7ff7bfe00640 1 --2- 172.21.15.102:0/1605590120 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff7c815dcb0 0x7ff7c80a74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.919+0000 7ff7bfe00640 1 -- 172.21.15.102:0/1605590120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7c806a3a0 msgr2=0x7ff7c8147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:44.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.919+0000 7ff7c7e00640 1 --2- 172.21.15.102:0/1605590120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7c806a3a0 0x7ff7c8147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.919+0000 7ff7bfe00640 1 --2- 172.21.15.102:0/1605590120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7c806a3a0 0x7ff7c8147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:44.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.919+0000 7ff7bfe00640 1 -- 172.21.15.102:0/1605590120 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c0009a70 con 0x7ff7c815cdc0 2024-03-23T03:47:44.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.919+0000 7ff7c7e00640 1 --2- 172.21.15.102:0/1605590120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7c806a3a0 0x7ff7c8147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:44.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.919+0000 7ff7bfe00640 1 --2- 172.21.15.102:0/1605590120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7c815cdc0 0x7ff7c80cbc70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff7b000c430 tx=0x7ff7b000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:44.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.920+0000 7ff7c4c00640 1 -- 172.21.15.102:0/1605590120 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0011790 con 0x7ff7c815cdc0 2024-03-23T03:47:44.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.920+0000 7ff7c4c00640 1 -- 172.21.15.102:0/1605590120 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b00118f0 con 0x7ff7c815cdc0 2024-03-23T03:47:44.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.920+0000 7ff7cea00640 1 -- 172.21.15.102:0/1605590120 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c81490c0 con 0x7ff7c815cdc0 2024-03-23T03:47:44.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.920+0000 7ff7c4c00640 1 -- 172.21.15.102:0/1605590120 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0011ce0 con 0x7ff7c815cdc0 2024-03-23T03:47:44.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.920+0000 7ff7cea00640 1 -- 172.21.15.102:0/1605590120 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c81482d0 con 0x7ff7c815cdc0 2024-03-23T03:47:44.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.921+0000 7ff7cea00640 1 -- 172.21.15.102:0/1605590120 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c8073390 con 0x7ff7c815cdc0 2024-03-23T03:47:44.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.922+0000 7ff7c4c00640 1 -- 172.21.15.102:0/1605590120 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff7b001c460 con 0x7ff7c815cdc0 2024-03-23T03:47:44.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.922+0000 7ff7c4c00640 1 --2- 172.21.15.102:0/1605590120 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff7a406ac00 0x7ff7a406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:44.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.923+0000 7ff7c4c00640 1 -- 172.21.15.102:0/1605590120 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff7b0089740 con 0x7ff7c815cdc0 2024-03-23T03:47:44.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.923+0000 7ff7c7e00640 1 --2- 172.21.15.102:0/1605590120 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff7a406ac00 0x7ff7a406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:44.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.924+0000 7ff7c7e00640 1 --2- 172.21.15.102:0/1605590120 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff7a406ac00 0x7ff7a406d0c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff7b80011e0 tx=0x7ff7b8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:44.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:44.928+0000 7ff7c4c00640 1 -- 172.21.15.102:0/1605590120 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff7b00544d0 con 0x7ff7c815cdc0 2024-03-23T03:47:45.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.122+0000 7ff7cea00640 1 -- 172.21.15.102:0/1605590120 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7c806ede0 con 0x7ff7c815cdc0 2024-03-23T03:47:45.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.125+0000 7ff7c4c00640 1 -- 172.21.15.102:0/1605590120 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18260 (secure 0 0 0) 0x7ff7b0057af0 con 0x7ff7c815cdc0 2024-03-23T03:47:45.126 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:45.126 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":28,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:47:41.647786+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:47:45.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.127+0000 7ff7cea00640 1 -- 172.21.15.102:0/1605590120 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff7a406ac00 msgr2=0x7ff7a406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:45.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.127+0000 7ff7cea00640 1 --2- 172.21.15.102:0/1605590120 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff7a406ac00 0x7ff7a406d0c0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff7b80011e0 tx=0x7ff7b8009040 comp rx=0 tx=0).stop 2024-03-23T03:47:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.127+0000 7ff7cea00640 1 -- 172.21.15.102:0/1605590120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7c815cdc0 msgr2=0x7ff7c80cbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.127+0000 7ff7cea00640 1 --2- 172.21.15.102:0/1605590120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7c815cdc0 0x7ff7c80cbc70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff7b000c430 tx=0x7ff7b000c8f0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.128+0000 7ff7cea00640 1 -- 172.21.15.102:0/1605590120 shutdown_connections 2024-03-23T03:47:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.128+0000 7ff7cea00640 1 --2- 172.21.15.102:0/1605590120 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff7a406ac00 0x7ff7a406d0c0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.128+0000 7ff7cea00640 1 --2- 172.21.15.102:0/1605590120 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff7c815dcb0 0x7ff7c80a74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.128+0000 7ff7cea00640 1 --2- 172.21.15.102:0/1605590120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7c815cdc0 0x7ff7c80cbc70 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.128+0000 7ff7cea00640 1 --2- 172.21.15.102:0/1605590120 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff7c806a3a0 0x7ff7c8147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.128+0000 7ff7cea00640 1 -- 172.21.15.102:0/1605590120 >> 172.21.15.102:0/1605590120 conn(0x7ff7c80b1370 msgr2=0x7ff7c815a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.128+0000 7ff7cea00640 1 -- 172.21.15.102:0/1605590120 shutdown_connections 2024-03-23T03:47:45.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.128+0000 7ff7cea00640 1 -- 172.21.15.102:0/1605590120 wait complete. 2024-03-23T03:47:45.146 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:46:52.158650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:46:52.914275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-23T03:46:55.983754+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-23T03:47:45.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:47:45.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.347+0000 7f7f1ee00640 1 Processor -- start 2024-03-23T03:47:45.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.347+0000 7f7f1ee00640 1 -- start start 2024-03-23T03:47:45.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.349+0000 7f7f1ee00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f7f180ce930 0x7f7f180cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:45.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.349+0000 7f7f1ee00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f18159c10 con 0x7f7f180ca7b0 2024-03-23T03:47:45.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.349+0000 7f7f1ee00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f180cd840 con 0x7f7f18159830 2024-03-23T03:47:45.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.349+0000 7f7f1ee00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f1806de50 con 0x7f7f180ce930 2024-03-23T03:47:45.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.350+0000 7f7f17400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7f180ca7b0 0x7f7f180cdcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47144/0 (socket says 172.21.15.102:47144) 2024-03-23T03:47:45.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.350+0000 7f7f17400640 1 -- 172.21.15.102:0/1665020636 learned_addr learned my addr 172.21.15.102:0/1665020636 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:45.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.350+0000 7f7f1ce00640 1 --2- 172.21.15.102:0/1665020636 >> v2:172.21.15.173:3300/0 conn(0x7f7f180ce930 0x7f7f180cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:45.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.350+0000 7f7f16a00640 1 -- 172.21.15.102:0/1665020636 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 325695459 0 0) 0x7f7f18159c10 con 0x7f7f180ca7b0 2024-03-23T03:47:45.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.350+0000 7f7f16a00640 1 -- 172.21.15.102:0/1665020636 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ef8005730 con 0x7f7f180ca7b0 2024-03-23T03:47:45.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f16a00640 1 -- 172.21.15.102:0/1665020636 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273532797 0 0) 0x7f7f180cd840 con 0x7f7f18159830 2024-03-23T03:47:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f16a00640 1 -- 172.21.15.102:0/1665020636 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f18159c10 con 0x7f7f18159830 2024-03-23T03:47:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f16a00640 1 -- 172.21.15.102:0/1665020636 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 256875070 0 0) 0x7f7ef8005730 con 0x7f7f180ca7b0 2024-03-23T03:47:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f16a00640 1 -- 172.21.15.102:0/1665020636 >> v2:172.21.15.173:3300/0 conn(0x7f7f180ce930 msgr2=0x7f7f180cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f16a00640 1 --2- 172.21.15.102:0/1665020636 >> v2:172.21.15.173:3300/0 conn(0x7f7f180ce930 0x7f7f180cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f16a00640 1 -- 172.21.15.102:0/1665020636 >> v1:172.21.15.173:6789/0 conn(0x7f7f18159830 legacy=0x7f7f180cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f16a00640 1 -- 172.21.15.102:0/1665020636 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f180ccc30 con 0x7f7f180ca7b0 2024-03-23T03:47:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f16a00640 1 -- 172.21.15.102:0/1665020636 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f7f000019f0 con 0x7f7f180ca7b0 2024-03-23T03:47:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f16a00640 1 -- 172.21.15.102:0/1665020636 >> v1:172.21.15.102:6789/0 conn(0x7f7f180ca7b0 legacy=0x7f7f180cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f16a00640 1 --2- 172.21.15.102:0/1665020636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ef8005d40 0x7f7ef8006140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f16a00640 1 --2- 172.21.15.102:0/1665020636 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7ef80069d0 0x7f7ef8006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f16a00640 1 --2- 172.21.15.102:0/1665020636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef8007390 0x7f7ef800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f16a00640 1 -- 172.21.15.102:0/1665020636 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef800e180 con 0x7f7ef8007390 2024-03-23T03:47:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f16a00640 1 -- 172.21.15.102:0/1665020636 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef800e2f0 con 0x7f7ef8005d40 2024-03-23T03:47:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f16a00640 1 -- 172.21.15.102:0/1665020636 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ef800e4b0 con 0x7f7ef80069d0 2024-03-23T03:47:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f1ce00640 1 --2- 172.21.15.102:0/1665020636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef8007390 0x7f7ef800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:45.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f1ce00640 1 -- 172.21.15.102:0/1665020636 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7ef80069d0 msgr2=0x7f7ef8006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:45.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f17400640 1 --2- 172.21.15.102:0/1665020636 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7ef80069d0 0x7f7ef8006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:45.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f1ce00640 1 --2- 172.21.15.102:0/1665020636 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7ef80069d0 0x7f7ef8006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.351+0000 7f7f1ce00640 1 -- 172.21.15.102:0/1665020636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ef8005d40 msgr2=0x7f7ef8006140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:45.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.352+0000 7f7f1ce00640 1 --2- 172.21.15.102:0/1665020636 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ef8005d40 0x7f7ef8006140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.352+0000 7f7f1ce00640 1 -- 172.21.15.102:0/1665020636 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef8005730 con 0x7f7ef8007390 2024-03-23T03:47:45.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.352+0000 7f7f1ce00640 1 -- 172.21.15.102:0/1665020636 reap_dead start 2024-03-23T03:47:45.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.353+0000 7f7f1ce00640 1 --2- 172.21.15.102:0/1665020636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef8007390 0x7f7ef800db90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7f10009ac0 tx=0x7f7ef8007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40b3e8aec43c9e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:45.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.353+0000 7f7f16a00640 1 -- 172.21.15.102:0/1665020636 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f10004020 con 0x7f7ef8007390 2024-03-23T03:47:45.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.353+0000 7f7f16a00640 1 -- 172.21.15.102:0/1665020636 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f10004180 con 0x7f7ef8007390 2024-03-23T03:47:45.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.353+0000 7f7f16a00640 1 -- 172.21.15.102:0/1665020636 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1002c710 con 0x7f7ef8007390 2024-03-23T03:47:45.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.353+0000 7f7f1ee00640 1 -- 172.21.15.102:0/1665020636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef8007390 msgr2=0x7f7ef800db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:45.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.353+0000 7f7f1ee00640 1 --2- 172.21.15.102:0/1665020636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef8007390 0x7f7ef800db90 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f7f10009ac0 tx=0x7f7ef8007a00 comp rx=0 tx=0).stop 2024-03-23T03:47:45.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.353+0000 7f7f1ee00640 1 -- 172.21.15.102:0/1665020636 shutdown_connections 2024-03-23T03:47:45.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.353+0000 7f7f1ee00640 1 --2- 172.21.15.102:0/1665020636 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef8007390 0x7f7ef800db90 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.353+0000 7f7f1ee00640 1 -- 172.21.15.102:0/1665020636 >> 172.21.15.102:0/1665020636 conn(0x7f7f1806a0b0 msgr2=0x7f7f1806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:45.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.354+0000 7f7f1ee00640 1 -- 172.21.15.102:0/1665020636 shutdown_connections 2024-03-23T03:47:45.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.354+0000 7f7f1ee00640 1 -- 172.21.15.102:0/1665020636 wait complete. 2024-03-23T03:47:45.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.355+0000 7f7f1ee00640 1 Processor -- start 2024-03-23T03:47:45.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.355+0000 7f7f1ee00640 1 -- start start 2024-03-23T03:47:45.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.356+0000 7f7f1ee00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef8007390 0x7f7f180acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:45.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.356+0000 7f7f1ee00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f181edf50 0x7f7f18157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:45.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.356+0000 7f7f1ee00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7f18157db0 0x7f7f18153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:45.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.356+0000 7f7f1ee00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f181ee3d0 con 0x7f7ef8007390 2024-03-23T03:47:45.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.356+0000 7f7f1ee00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f181ee6b0 con 0x7f7f181edf50 2024-03-23T03:47:45.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.356+0000 7f7f1ee00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f181ee990 con 0x7f7f18157db0 2024-03-23T03:47:45.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.357+0000 7f7f17e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef8007390 0x7f7f180acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:45.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.357+0000 7f7f17e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef8007390 0x7f7f180acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33226/0 (socket says 172.21.15.102:33226) 2024-03-23T03:47:45.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.357+0000 7f7f17e00640 1 -- 172.21.15.102:0/3371776794 learned_addr learned my addr 172.21.15.102:0/3371776794 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:45.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.357+0000 7f7f1ce00640 1 --2- 172.21.15.102:0/3371776794 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7f18157db0 0x7f7f18153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:45.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.357+0000 7f7f17400640 1 --2- 172.21.15.102:0/3371776794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f181edf50 0x7f7f18157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:45.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.357+0000 7f7f17e00640 1 -- 172.21.15.102:0/3371776794 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7f18157db0 msgr2=0x7f7f18153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:45.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.357+0000 7f7f17e00640 1 --2- 172.21.15.102:0/3371776794 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7f18157db0 0x7f7f18153030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.357+0000 7f7f17e00640 1 -- 172.21.15.102:0/3371776794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f181edf50 msgr2=0x7f7f18157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:45.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.357+0000 7f7f17e00640 1 --2- 172.21.15.102:0/3371776794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f181edf50 0x7f7f18157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.357+0000 7f7f17e00640 1 -- 172.21.15.102:0/3371776794 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f181eec70 con 0x7f7ef8007390 2024-03-23T03:47:45.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.357+0000 7f7f1ce00640 1 --2- 172.21.15.102:0/3371776794 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7f18157db0 0x7f7f18153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:47:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.358+0000 7f7f17e00640 1 --2- 172.21.15.102:0/3371776794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef8007390 0x7f7f180acdd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800c520 tx=0x7f7f0800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.358+0000 7f7f0fe00640 1 -- 172.21.15.102:0/3371776794 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f080052d0 con 0x7f7ef8007390 2024-03-23T03:47:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.358+0000 7f7f0fe00640 1 -- 172.21.15.102:0/3371776794 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f08005430 con 0x7f7ef8007390 2024-03-23T03:47:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.358+0000 7f7f0fe00640 1 -- 172.21.15.102:0/3371776794 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f08012870 con 0x7f7ef8007390 2024-03-23T03:47:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.358+0000 7f7f1ee00640 1 -- 172.21.15.102:0/3371776794 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f180ad310 con 0x7f7ef8007390 2024-03-23T03:47:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.358+0000 7f7f1ee00640 1 -- 172.21.15.102:0/3371776794 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f18153750 con 0x7f7ef8007390 2024-03-23T03:47:45.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.359+0000 7f7f1ee00640 1 -- 172.21.15.102:0/3371776794 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f180ad500 con 0x7f7ef8007390 2024-03-23T03:47:45.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.360+0000 7f7f0fe00640 1 -- 172.21.15.102:0/3371776794 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7f080035f0 con 0x7f7ef8007390 2024-03-23T03:47:45.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.361+0000 7f7f0fe00640 1 --2- 172.21.15.102:0/3371776794 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7ef406ac00 0x7f7ef406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:45.361 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.361+0000 7f7f0fe00640 1 -- 172.21.15.102:0/3371776794 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7f08088d30 con 0x7f7ef8007390 2024-03-23T03:47:45.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.361+0000 7f7f17400640 1 --2- 172.21.15.102:0/3371776794 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7ef406ac00 0x7f7ef406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:45.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.362+0000 7f7f17400640 1 --2- 172.21.15.102:0/3371776794 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7ef406ac00 0x7f7ef406d0c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7f00002230 tx=0x7f7f00009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:45.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.365+0000 7f7f0fe00640 1 -- 172.21.15.102:0/3371776794 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7f08053c00 con 0x7f7ef8007390 2024-03-23T03:47:45.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.557+0000 7f7f1ee00640 1 -- 172.21.15.102:0/3371776794 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7f180ae150 con 0x7f7ef8007390 2024-03-23T03:47:45.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.558+0000 7f7f0fe00640 1 -- 172.21.15.102:0/3371776794 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7f080170a0 con 0x7f7ef8007390 2024-03-23T03:47:45.559 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:47:45.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.561+0000 7f7f1ee00640 1 -- 172.21.15.102:0/3371776794 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7ef406ac00 msgr2=0x7f7ef406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:45.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.561+0000 7f7f1ee00640 1 --2- 172.21.15.102:0/3371776794 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7ef406ac00 0x7f7ef406d0c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7f00002230 tx=0x7f7f00009040 comp rx=0 tx=0).stop 2024-03-23T03:47:45.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.561+0000 7f7f1ee00640 1 -- 172.21.15.102:0/3371776794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef8007390 msgr2=0x7f7f180acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.561+0000 7f7f1ee00640 1 --2- 172.21.15.102:0/3371776794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef8007390 0x7f7f180acdd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7f0800c520 tx=0x7f7f0800c9e0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.561+0000 7f7f1ee00640 1 -- 172.21.15.102:0/3371776794 shutdown_connections 2024-03-23T03:47:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.561+0000 7f7f1ee00640 1 --2- 172.21.15.102:0/3371776794 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7ef406ac00 0x7f7ef406d0c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.561+0000 7f7f1ee00640 1 --2- 172.21.15.102:0/3371776794 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7f18157db0 0x7f7f18153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.561+0000 7f7f1ee00640 1 --2- 172.21.15.102:0/3371776794 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7f181edf50 0x7f7f18157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.561+0000 7f7f1ee00640 1 --2- 172.21.15.102:0/3371776794 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ef8007390 0x7f7f180acdd0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.561+0000 7f7f1ee00640 1 -- 172.21.15.102:0/3371776794 >> 172.21.15.102:0/3371776794 conn(0x7f7f1806a0b0 msgr2=0x7f7f180b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.561+0000 7f7f1ee00640 1 -- 172.21.15.102:0/3371776794 shutdown_connections 2024-03-23T03:47:45.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.561+0000 7f7f1ee00640 1 -- 172.21.15.102:0/3371776794 wait complete. 2024-03-23T03:47:45.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:47:45.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.792+0000 7f2ff6800640 1 Processor -- start 2024-03-23T03:47:45.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.792+0000 7f2ff6800640 1 -- start start 2024-03-23T03:47:45.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.794+0000 7f2ff6800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2ff00ce930 0x7f2ff00add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:45.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.794+0000 7f2ff6800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff00cd3e0 0x7f2ff00ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:45.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.794+0000 7f2ff6800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff00cf250 con 0x7f2ff006ce90 2024-03-23T03:47:45.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.794+0000 7f2ff6800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff006c760 con 0x7f2ff00ce930 2024-03-23T03:47:45.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.794+0000 7f2ff6800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff00b42d0 con 0x7f2ff00cd3e0 2024-03-23T03:47:45.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.794+0000 7f2fef400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2ff00ce930 0x7f2ff00add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:45.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.794+0000 7f2fefe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2ff006ce90 0x7f2ff015e290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47148/0 (socket says 172.21.15.102:47148) 2024-03-23T03:47:45.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.794+0000 7f2fefe00640 1 -- 172.21.15.102:0/4143290091 learned_addr learned my addr 172.21.15.102:0/4143290091 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:45.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.795+0000 7f2fe7e00640 1 --2- 172.21.15.102:0/4143290091 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff00cd3e0 0x7f2ff00ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:45.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.795+0000 7f2feea00640 1 -- 172.21.15.102:0/4143290091 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1246815416 0 0) 0x7f2ff00cf250 con 0x7f2ff006ce90 2024-03-23T03:47:45.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.795+0000 7f2feea00640 1 -- 172.21.15.102:0/4143290091 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fd4003520 con 0x7f2ff006ce90 2024-03-23T03:47:45.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.795+0000 7f2fef400640 1 -- 172.21.15.102:0/4143290091 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff00cd3e0 msgr2=0x7f2ff00ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:45.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.795+0000 7f2fef400640 1 --2- 172.21.15.102:0/4143290091 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff00cd3e0 0x7f2ff00ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.795+0000 7f2fef400640 1 -- 172.21.15.102:0/4143290091 >> v1:172.21.15.102:6789/0 conn(0x7f2ff006ce90 legacy=0x7f2ff015e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:45.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.795+0000 7f2fef400640 1 -- 172.21.15.102:0/4143290091 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff00cdba0 con 0x7f2ff00ce930 2024-03-23T03:47:45.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.796+0000 7f2fef400640 1 --2- 172.21.15.102:0/4143290091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00ce930 0x7f2ff00add30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00097a0 tx=0x7f2fe002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efbfd78bf8c4ba87 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:45.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.796+0000 7f2feea00640 1 -- 172.21.15.102:0/4143290091 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0031830 con 0x7f2ff00ce930 2024-03-23T03:47:45.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.796+0000 7f2feea00640 1 -- 172.21.15.102:0/4143290091 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe0031990 con 0x7f2ff00ce930 2024-03-23T03:47:45.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.797+0000 7f2feea00640 1 -- 172.21.15.102:0/4143290091 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0031d20 con 0x7f2ff00ce930 2024-03-23T03:47:45.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.797+0000 7f2ff6800640 1 -- 172.21.15.102:0/4143290091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00ce930 msgr2=0x7f2ff00add30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:45.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.797+0000 7f2ff6800640 1 --2- 172.21.15.102:0/4143290091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00ce930 0x7f2ff00add30 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2fe00097a0 tx=0x7f2fe002b8d0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.797+0000 7f2ff6800640 1 -- 172.21.15.102:0/4143290091 shutdown_connections 2024-03-23T03:47:45.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.797+0000 7f2ff6800640 1 --2- 172.21.15.102:0/4143290091 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff00cd3e0 0x7f2ff00ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.797+0000 7f2ff6800640 1 --2- 172.21.15.102:0/4143290091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00ce930 0x7f2ff00add30 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.797+0000 7f2ff6800640 1 -- 172.21.15.102:0/4143290091 >> 172.21.15.102:0/4143290091 conn(0x7f2ff006a0b0 msgr2=0x7f2ff006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:45.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.797+0000 7f2ff6800640 1 -- 172.21.15.102:0/4143290091 shutdown_connections 2024-03-23T03:47:45.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.797+0000 7f2ff6800640 1 -- 172.21.15.102:0/4143290091 wait complete. 2024-03-23T03:47:45.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.799+0000 7f2ff6800640 1 Processor -- start 2024-03-23T03:47:45.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.799+0000 7f2ff6800640 1 -- start start 2024-03-23T03:47:45.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.800+0000 7f2ff6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ff006ce90 0x7f2ff00b3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:45.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.800+0000 7f2ff6800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff00cd3e0 0x7f2ff015a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:45.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.800+0000 7f2ff6800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00ce930 0x7f2ff006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:45.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.800+0000 7f2ff6800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff0158e90 con 0x7f2ff00ce930 2024-03-23T03:47:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.800+0000 7f2ff6800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff0159cf0 con 0x7f2ff006ce90 2024-03-23T03:47:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.800+0000 7f2ff6800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff00b36a0 con 0x7f2ff00cd3e0 2024-03-23T03:47:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.800+0000 7f2fefe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00ce930 0x7f2ff006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.800+0000 7f2fefe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00ce930 0x7f2ff006d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33238/0 (socket says 172.21.15.102:33238) 2024-03-23T03:47:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.800+0000 7f2fefe00640 1 -- 172.21.15.102:0/3691192836 learned_addr learned my addr 172.21.15.102:0/3691192836 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.801+0000 7f2fe7e00640 1 --2- 172.21.15.102:0/3691192836 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff00cd3e0 0x7f2ff015a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.801+0000 7f2fefe00640 1 -- 172.21.15.102:0/3691192836 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff00cd3e0 msgr2=0x7f2ff015a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.801+0000 7f2fefe00640 1 --2- 172.21.15.102:0/3691192836 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff00cd3e0 0x7f2ff015a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.801+0000 7f2fefe00640 1 -- 172.21.15.102:0/3691192836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ff006ce90 msgr2=0x7f2ff00b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.801+0000 7f2fefe00640 1 --2- 172.21.15.102:0/3691192836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ff006ce90 0x7f2ff00b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.801+0000 7f2fefe00640 1 -- 172.21.15.102:0/3691192836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe0009400 con 0x7f2ff00ce930 2024-03-23T03:47:45.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.801+0000 7f2fefe00640 1 --2- 172.21.15.102:0/3691192836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00ce930 0x7f2ff006d570 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800cad0 tx=0x7f2fe800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:45.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.801+0000 7f2fe7400640 1 -- 172.21.15.102:0/3691192836 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe800ac30 con 0x7f2ff00ce930 2024-03-23T03:47:45.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.801+0000 7f2fe7400640 1 -- 172.21.15.102:0/3691192836 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe800ad90 con 0x7f2ff00ce930 2024-03-23T03:47:45.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.801+0000 7f2fe7400640 1 -- 172.21.15.102:0/3691192836 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe8012620 con 0x7f2ff00ce930 2024-03-23T03:47:45.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.801+0000 7f2ff6800640 1 -- 172.21.15.102:0/3691192836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff01590e0 con 0x7f2ff00ce930 2024-03-23T03:47:45.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.801+0000 7f2ff6800640 1 -- 172.21.15.102:0/3691192836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff01596a0 con 0x7f2ff00ce930 2024-03-23T03:47:45.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.803+0000 7f2ff6800640 1 -- 172.21.15.102:0/3691192836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff0073390 con 0x7f2ff00ce930 2024-03-23T03:47:45.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.803+0000 7f2fe7400640 1 -- 172.21.15.102:0/3691192836 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2fe8007070 con 0x7f2ff00ce930 2024-03-23T03:47:45.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.804+0000 7f2fe7400640 1 --2- 172.21.15.102:0/3691192836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fcc06ac00 0x7f2fcc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:45.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.804+0000 7f2fe7400640 1 -- 172.21.15.102:0/3691192836 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2fe8089770 con 0x7f2ff00ce930 2024-03-23T03:47:45.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.805+0000 7f2fef400640 1 --2- 172.21.15.102:0/3691192836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fcc06ac00 0x7f2fcc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:45.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.805+0000 7f2fef400640 1 --2- 172.21.15.102:0/3691192836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fcc06ac00 0x7f2fcc06d0c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0036000 tx=0x7f2fe0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:45.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:45.809+0000 7f2fe7400640 1 -- 172.21.15.102:0/3691192836 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2fe80544f0 con 0x7f2ff00ce930 2024-03-23T03:47:46.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.005+0000 7f2ff6800640 1 -- 172.21.15.102:0/3691192836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2ff0159e30 con 0x7f2ff00ce930 2024-03-23T03:47:46.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.006+0000 7f2fe7400640 1 -- 172.21.15.102:0/3691192836 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2fe8057b10 con 0x7f2ff00ce930 2024-03-23T03:47:46.007 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.009+0000 7f2ff6800640 1 -- 172.21.15.102:0/3691192836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fcc06ac00 msgr2=0x7f2fcc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.009+0000 7f2ff6800640 1 --2- 172.21.15.102:0/3691192836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fcc06ac00 0x7f2fcc06d0c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0036000 tx=0x7f2fe0030000 comp rx=0 tx=0).stop 2024-03-23T03:47:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.009+0000 7f2ff6800640 1 -- 172.21.15.102:0/3691192836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00ce930 msgr2=0x7f2ff006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.009+0000 7f2ff6800640 1 --2- 172.21.15.102:0/3691192836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00ce930 0x7f2ff006d570 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800cad0 tx=0x7f2fe800cf90 comp rx=0 tx=0).stop 2024-03-23T03:47:46.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.009+0000 7f2ff6800640 1 -- 172.21.15.102:0/3691192836 shutdown_connections 2024-03-23T03:47:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.009+0000 7f2ff6800640 1 --2- 172.21.15.102:0/3691192836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fcc06ac00 0x7f2fcc06d0c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.009+0000 7f2ff6800640 1 --2- 172.21.15.102:0/3691192836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00ce930 0x7f2ff006d570 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.009+0000 7f2ff6800640 1 --2- 172.21.15.102:0/3691192836 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff00cd3e0 0x7f2ff015a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.009+0000 7f2ff6800640 1 --2- 172.21.15.102:0/3691192836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ff006ce90 0x7f2ff00b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.009+0000 7f2ff6800640 1 -- 172.21.15.102:0/3691192836 >> 172.21.15.102:0/3691192836 conn(0x7f2ff006a0b0 msgr2=0x7f2ff00b00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.010+0000 7f2ff6800640 1 -- 172.21.15.102:0/3691192836 shutdown_connections 2024-03-23T03:47:46.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.010+0000 7f2ff6800640 1 -- 172.21.15.102:0/3691192836 wait complete. 2024-03-23T03:47:46.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:47:46.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.238+0000 7f5ad1000640 1 Processor -- start 2024-03-23T03:47:46.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.238+0000 7f5ad1000640 1 -- start start 2024-03-23T03:47:46.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.239+0000 7f5ad1000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5acc15a780 0x7f5acc099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:46.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.239+0000 7f5ad1000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5acc0cec50 0x7f5acc0cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:46.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.239+0000 7f5ad1000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5acc0ce560 con 0x7f5acc15ab60 2024-03-23T03:47:46.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.239+0000 7f5ad1000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5acc099f90 con 0x7f5acc15a780 2024-03-23T03:47:46.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.239+0000 7f5ad1000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc0cf5e0 con 0x7f5acc0cec50 2024-03-23T03:47:46.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.240+0000 7f5ac9400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5acc15ab60 0x7f5acc0cbcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47158/0 (socket says 172.21.15.102:47158) 2024-03-23T03:47:46.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.240+0000 7f5ac9400640 1 -- 172.21.15.102:0/646972353 learned_addr learned my addr 172.21.15.102:0/646972353 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:46.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.240+0000 7f5ac9e00640 1 --2- 172.21.15.102:0/646972353 >> v2:172.21.15.102:3300/0 conn(0x7f5acc15a780 0x7f5acc099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:46.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.240+0000 7f5aca800640 1 --2- 172.21.15.102:0/646972353 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5acc0cec50 0x7f5acc0cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:46.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.240+0000 7f5ac9e00640 1 -- 172.21.15.102:0/646972353 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5acc0cec50 msgr2=0x7f5acc0cf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:46.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.240+0000 7f5ac9e00640 1 --2- 172.21.15.102:0/646972353 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5acc0cec50 0x7f5acc0cf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.240+0000 7f5ac9e00640 1 -- 172.21.15.102:0/646972353 >> v1:172.21.15.102:6789/0 conn(0x7f5acc15ab60 legacy=0x7f5acc0cbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:46.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.240+0000 7f5ac9e00640 1 -- 172.21.15.102:0/646972353 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5acc098a40 con 0x7f5acc15a780 2024-03-23T03:47:46.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.242+0000 7f5ac9e00640 1 --2- 172.21.15.102:0/646972353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5acc15a780 0x7f5acc099a50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5abc0097a0 tx=0x7f5abc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd24cc9ac159e922 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:46.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.242+0000 7f5ac8a00640 1 -- 172.21.15.102:0/646972353 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc02d040 con 0x7f5acc15a780 2024-03-23T03:47:46.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.242+0000 7f5ac8a00640 1 -- 172.21.15.102:0/646972353 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5abc031810 con 0x7f5acc15a780 2024-03-23T03:47:46.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.242+0000 7f5ac8a00640 1 -- 172.21.15.102:0/646972353 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc031ba0 con 0x7f5acc15a780 2024-03-23T03:47:46.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.243+0000 7f5ad1000640 1 -- 172.21.15.102:0/646972353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5acc15a780 msgr2=0x7f5acc099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:46.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.243+0000 7f5ad1000640 1 --2- 172.21.15.102:0/646972353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5acc15a780 0x7f5acc099a50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5abc0097a0 tx=0x7f5abc02b8b0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.243+0000 7f5ad1000640 1 -- 172.21.15.102:0/646972353 shutdown_connections 2024-03-23T03:47:46.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.243+0000 7f5ad1000640 1 --2- 172.21.15.102:0/646972353 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5acc0cec50 0x7f5acc0cf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.243+0000 7f5ad1000640 1 --2- 172.21.15.102:0/646972353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5acc15a780 0x7f5acc099a50 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.243+0000 7f5ad1000640 1 -- 172.21.15.102:0/646972353 >> 172.21.15.102:0/646972353 conn(0x7f5acc0b2180 msgr2=0x7f5acc0b2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:46.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.243+0000 7f5ad1000640 1 -- 172.21.15.102:0/646972353 shutdown_connections 2024-03-23T03:47:46.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.243+0000 7f5ad1000640 1 -- 172.21.15.102:0/646972353 wait complete. 2024-03-23T03:47:46.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.244+0000 7f5ad1000640 1 Processor -- start 2024-03-23T03:47:46.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.245+0000 7f5ad1000640 1 -- start start 2024-03-23T03:47:46.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5ad1000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5acc0cec50 0x7f5acc14ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:46.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5ad1000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5acc15a780 0x7f5acc14a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:46.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5ad1000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5acc15ab60 0x7f5acc14ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:46.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5ad1000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc14b370 con 0x7f5acc15ab60 2024-03-23T03:47:46.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5ad1000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc0c44c0 con 0x7f5acc15a780 2024-03-23T03:47:46.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5ad1000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc15b270 con 0x7f5acc0cec50 2024-03-23T03:47:46.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5aca800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5acc15ab60 0x7f5acc14ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:46.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5aca800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5acc15ab60 0x7f5acc14ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33258/0 (socket says 172.21.15.102:33258) 2024-03-23T03:47:46.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5aca800640 1 -- 172.21.15.102:0/2995644378 learned_addr learned my addr 172.21.15.102:0/2995644378 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:46.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5ac9e00640 1 --2- 172.21.15.102:0/2995644378 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5acc0cec50 0x7f5acc14ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:46.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5aca800640 1 -- 172.21.15.102:0/2995644378 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5acc0cec50 msgr2=0x7f5acc14ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5aca800640 1 --2- 172.21.15.102:0/2995644378 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5acc0cec50 0x7f5acc14ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5aca800640 1 -- 172.21.15.102:0/2995644378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5acc15a780 msgr2=0x7f5acc14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5ac9400640 1 --2- 172.21.15.102:0/2995644378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5acc15a780 0x7f5acc14a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5aca800640 1 --2- 172.21.15.102:0/2995644378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5acc15a780 0x7f5acc14a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5aca800640 1 -- 172.21.15.102:0/2995644378 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5abc009400 con 0x7f5acc15ab60 2024-03-23T03:47:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5ac9e00640 1 --2- 172.21.15.102:0/2995644378 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5acc0cec50 0x7f5acc14ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.246+0000 7f5ac9400640 1 --2- 172.21.15.102:0/2995644378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5acc15a780 0x7f5acc14a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.247+0000 7f5aca800640 1 --2- 172.21.15.102:0/2995644378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5acc15ab60 0x7f5acc14ae30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400d800 tx=0x7f5ac400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.247+0000 7f5ac2000640 1 -- 172.21.15.102:0/2995644378 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac400b6c0 con 0x7f5acc15ab60 2024-03-23T03:47:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.247+0000 7f5ac2000640 1 -- 172.21.15.102:0/2995644378 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac400b840 con 0x7f5acc15ab60 2024-03-23T03:47:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.247+0000 7f5ac2000640 1 -- 172.21.15.102:0/2995644378 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac400bbf0 con 0x7f5acc15ab60 2024-03-23T03:47:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.247+0000 7f5ad1000640 1 -- 172.21.15.102:0/2995644378 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5acc0b5d40 con 0x7f5acc15ab60 2024-03-23T03:47:46.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.247+0000 7f5ad1000640 1 -- 172.21.15.102:0/2995644378 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5acc15b530 con 0x7f5acc15ab60 2024-03-23T03:47:46.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.248+0000 7f5ad1000640 1 -- 172.21.15.102:0/2995644378 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5acc0b6260 con 0x7f5acc15ab60 2024-03-23T03:47:46.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.249+0000 7f5ac2000640 1 -- 172.21.15.102:0/2995644378 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5ac400bd50 con 0x7f5acc15ab60 2024-03-23T03:47:46.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.250+0000 7f5ac2000640 1 --2- 172.21.15.102:0/2995644378 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5aa806ac00 0x7f5aa806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:46.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.251+0000 7f5ac2000640 1 -- 172.21.15.102:0/2995644378 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5ac4089ce0 con 0x7f5acc15ab60 2024-03-23T03:47:46.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.251+0000 7f5ac9e00640 1 --2- 172.21.15.102:0/2995644378 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5aa806ac00 0x7f5aa806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:46.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.252+0000 7f5ac9e00640 1 --2- 172.21.15.102:0/2995644378 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5aa806ac00 0x7f5aa806d0c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5abc036000 tx=0x7f5abc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:46.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.255+0000 7f5ac2000640 1 -- 172.21.15.102:0/2995644378 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5ac4090080 con 0x7f5acc15ab60 2024-03-23T03:47:46.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.452+0000 7f5ad1000640 1 -- 172.21.15.102:0/2995644378 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5acc0cf0d0 con 0x7f5acc15ab60 2024-03-23T03:47:46.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.453+0000 7f5ac2000640 1 -- 172.21.15.102:0/2995644378 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5ac4054a60 con 0x7f5acc15ab60 2024-03-23T03:47:46.454 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:47:46.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.455+0000 7f5ad1000640 1 -- 172.21.15.102:0/2995644378 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5aa806ac00 msgr2=0x7f5aa806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.455+0000 7f5ad1000640 1 --2- 172.21.15.102:0/2995644378 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5aa806ac00 0x7f5aa806d0c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5abc036000 tx=0x7f5abc03c040 comp rx=0 tx=0).stop 2024-03-23T03:47:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.455+0000 7f5ad1000640 1 -- 172.21.15.102:0/2995644378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5acc15ab60 msgr2=0x7f5acc14ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.455+0000 7f5ad1000640 1 --2- 172.21.15.102:0/2995644378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5acc15ab60 0x7f5acc14ae30 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5ac400d800 tx=0x7f5ac400dcc0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.456+0000 7f5ad1000640 1 -- 172.21.15.102:0/2995644378 shutdown_connections 2024-03-23T03:47:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.456+0000 7f5ad1000640 1 --2- 172.21.15.102:0/2995644378 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5aa806ac00 0x7f5aa806d0c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.456+0000 7f5ad1000640 1 --2- 172.21.15.102:0/2995644378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5acc15ab60 0x7f5acc14ae30 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.456+0000 7f5ad1000640 1 --2- 172.21.15.102:0/2995644378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5acc15a780 0x7f5acc14a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.456+0000 7f5ad1000640 1 --2- 172.21.15.102:0/2995644378 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5acc0cec50 0x7f5acc14ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.456+0000 7f5ad1000640 1 -- 172.21.15.102:0/2995644378 >> 172.21.15.102:0/2995644378 conn(0x7f5acc0b2180 msgr2=0x7f5acc0b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.456+0000 7f5ad1000640 1 -- 172.21.15.102:0/2995644378 shutdown_connections 2024-03-23T03:47:46.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.456+0000 7f5ad1000640 1 -- 172.21.15.102:0/2995644378 wait complete. 2024-03-23T03:47:46.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-23T03:47:46.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.675+0000 7f9203200640 1 Processor -- start 2024-03-23T03:47:46.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.676+0000 7f9203200640 1 -- start start 2024-03-23T03:47:46.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.677+0000 7f9203200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f91fc0cd740 0x7f91fc0b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:46.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.677+0000 7f9203200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91fc0d0d40 0x7f91fc0cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:46.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.677+0000 7f9203200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f91fc06b0a0 0x7f91fc06b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:46.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.677+0000 7f9203200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91fc0d20c0 con 0x7f91fc0cd740 2024-03-23T03:47:46.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.677+0000 7f9203200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91fc0ce060 con 0x7f91fc06b0a0 2024-03-23T03:47:46.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.677+0000 7f9203200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc0d14b0 con 0x7f91fc0d0d40 2024-03-23T03:47:46.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.677+0000 7f91fbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f91fc0cd740 0x7f91fc0b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:46.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.678+0000 7f91fbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f91fc0cd740 0x7f91fc0b84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33264/0 (socket says 172.21.15.102:33264) 2024-03-23T03:47:46.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.678+0000 7f91fbe00640 1 -- 172.21.15.102:0/4101001830 learned_addr learned my addr 172.21.15.102:0/4101001830 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:46.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.678+0000 7f9201200640 1 --2- 172.21.15.102:0/4101001830 >> v2:172.21.15.173:3300/0 conn(0x7f91fc06b0a0 0x7f91fc06b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:46.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.678+0000 7f91fb400640 1 --2- 172.21.15.102:0/4101001830 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91fc0d0d40 0x7f91fc0cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:46.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.678+0000 7f91fbe00640 1 -- 172.21.15.102:0/4101001830 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91fc0d0d40 msgr2=0x7f91fc0cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:46.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.678+0000 7f91fbe00640 1 --2- 172.21.15.102:0/4101001830 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91fc0d0d40 0x7f91fc0cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.678+0000 7f91fbe00640 1 -- 172.21.15.102:0/4101001830 >> v2:172.21.15.173:3300/0 conn(0x7f91fc06b0a0 msgr2=0x7f91fc06b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:46.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.678+0000 7f91fbe00640 1 --2- 172.21.15.102:0/4101001830 >> v2:172.21.15.173:3300/0 conn(0x7f91fc06b0a0 0x7f91fc06b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.678+0000 7f91fbe00640 1 -- 172.21.15.102:0/4101001830 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91fc0d1790 con 0x7f91fc0cd740 2024-03-23T03:47:46.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.679+0000 7f91fbe00640 1 --2- 172.21.15.102:0/4101001830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91fc0cd740 0x7f91fc0b84b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f91fc0c99c0 tx=0x7f91ec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0796f80072c7364 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:46.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.680+0000 7f91faa00640 1 -- 172.21.15.102:0/4101001830 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ec031870 con 0x7f91fc0cd740 2024-03-23T03:47:46.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.680+0000 7f91faa00640 1 -- 172.21.15.102:0/4101001830 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91ec0319d0 con 0x7f91fc0cd740 2024-03-23T03:47:46.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.680+0000 7f91faa00640 1 -- 172.21.15.102:0/4101001830 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ec031d60 con 0x7f91fc0cd740 2024-03-23T03:47:46.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.680+0000 7f9203200640 1 -- 172.21.15.102:0/4101001830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91fc0cd740 msgr2=0x7f91fc0b84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:46.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.680+0000 7f9203200640 1 --2- 172.21.15.102:0/4101001830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91fc0cd740 0x7f91fc0b84b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f91fc0c99c0 tx=0x7f91ec02b910 comp rx=0 tx=0).stop 2024-03-23T03:47:46.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.680+0000 7f9203200640 1 -- 172.21.15.102:0/4101001830 shutdown_connections 2024-03-23T03:47:46.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.680+0000 7f9203200640 1 --2- 172.21.15.102:0/4101001830 >> v2:172.21.15.173:3300/0 conn(0x7f91fc06b0a0 0x7f91fc06b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.680+0000 7f9203200640 1 --2- 172.21.15.102:0/4101001830 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91fc0d0d40 0x7f91fc0cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.680+0000 7f9203200640 1 --2- 172.21.15.102:0/4101001830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91fc0cd740 0x7f91fc0b84b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.680+0000 7f9203200640 1 -- 172.21.15.102:0/4101001830 >> 172.21.15.102:0/4101001830 conn(0x7f91fc0cf5d0 msgr2=0x7f91fc0cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:46.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.680+0000 7f9203200640 1 -- 172.21.15.102:0/4101001830 shutdown_connections 2024-03-23T03:47:46.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.680+0000 7f9203200640 1 -- 172.21.15.102:0/4101001830 wait complete. 2024-03-23T03:47:46.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.682+0000 7f9203200640 1 Processor -- start 2024-03-23T03:47:46.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.682+0000 7f9203200640 1 -- start start 2024-03-23T03:47:46.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.683+0000 7f9203200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91fc06b0a0 0x7f91fc14a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:46.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.683+0000 7f9203200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91fc0cd740 0x7f91fc0cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:46.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.683+0000 7f9203200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91fc0d0d40 0x7f91fc0b7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:46.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.683+0000 7f9203200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc0cc6c0 con 0x7f91fc06b0a0 2024-03-23T03:47:46.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.683+0000 7f9203200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc0cbab0 con 0x7f91fc0d0d40 2024-03-23T03:47:46.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.683+0000 7f9203200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc0cbd90 con 0x7f91fc0cd740 2024-03-23T03:47:46.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.683+0000 7f91fbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91fc06b0a0 0x7f91fc14a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:46.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.683+0000 7f91fbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91fc06b0a0 0x7f91fc14a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33280/0 (socket says 172.21.15.102:33280) 2024-03-23T03:47:46.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.683+0000 7f91fbe00640 1 -- 172.21.15.102:0/26429123 learned_addr learned my addr 172.21.15.102:0/26429123 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:46.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.684+0000 7f9201200640 1 --2- 172.21.15.102:0/26429123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91fc0d0d40 0x7f91fc0b7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:46.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.684+0000 7f91fb400640 1 --2- 172.21.15.102:0/26429123 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91fc0cd740 0x7f91fc0cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:46.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.684+0000 7f91fbe00640 1 -- 172.21.15.102:0/26429123 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91fc0cd740 msgr2=0x7f91fc0cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:46.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.684+0000 7f91fbe00640 1 --2- 172.21.15.102:0/26429123 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91fc0cd740 0x7f91fc0cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.684+0000 7f91fbe00640 1 -- 172.21.15.102:0/26429123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91fc0d0d40 msgr2=0x7f91fc0b7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:46.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.684+0000 7f91fbe00640 1 --2- 172.21.15.102:0/26429123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91fc0d0d40 0x7f91fc0b7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.684+0000 7f91fbe00640 1 -- 172.21.15.102:0/26429123 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ec0094d0 con 0x7f91fc06b0a0 2024-03-23T03:47:46.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.684+0000 7f91fbe00640 1 --2- 172.21.15.102:0/26429123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91fc06b0a0 0x7f91fc14a240 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f91ec02b950 tx=0x7f91ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:46.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.684+0000 7f91f3e00640 1 -- 172.21.15.102:0/26429123 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ec03ad30 con 0x7f91fc06b0a0 2024-03-23T03:47:46.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.685+0000 7f91f3e00640 1 -- 172.21.15.102:0/26429123 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91ec0443f0 con 0x7f91fc06b0a0 2024-03-23T03:47:46.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.685+0000 7f91f3e00640 1 -- 172.21.15.102:0/26429123 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91ec0447a0 con 0x7f91fc06b0a0 2024-03-23T03:47:46.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.685+0000 7f9203200640 1 -- 172.21.15.102:0/26429123 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91fc0cc070 con 0x7f91fc06b0a0 2024-03-23T03:47:46.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.685+0000 7f9203200640 1 -- 172.21.15.102:0/26429123 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91fc14a780 con 0x7f91fc06b0a0 2024-03-23T03:47:46.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.686+0000 7f9203200640 1 -- 172.21.15.102:0/26429123 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91fc073390 con 0x7f91fc06b0a0 2024-03-23T03:47:46.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.687+0000 7f91f3e00640 1 -- 172.21.15.102:0/26429123 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f91ec04d450 con 0x7f91fc06b0a0 2024-03-23T03:47:46.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.687+0000 7f91f3e00640 1 --2- 172.21.15.102:0/26429123 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f91d806ac00 0x7f91d806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:46.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.688+0000 7f91f3e00640 1 -- 172.21.15.102:0/26429123 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f91ec02d040 con 0x7f91fc06b0a0 2024-03-23T03:47:46.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.688+0000 7f91fb400640 1 --2- 172.21.15.102:0/26429123 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f91d806ac00 0x7f91d806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:46.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.688+0000 7f91fb400640 1 --2- 172.21.15.102:0/26429123 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f91d806ac00 0x7f91d806d0c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f91e4006fd0 tx=0x7f91e4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:46.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.692+0000 7f91f3e00640 1 -- 172.21.15.102:0/26429123 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f91ec076be0 con 0x7f91fc06b0a0 2024-03-23T03:47:46.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.890+0000 7f9203200640 1 -- 172.21.15.102:0/26429123 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f91fc0b49c0 con 0x7f91fc06b0a0 2024-03-23T03:47:46.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.890+0000 7f91f3e00640 1 -- 172.21.15.102:0/26429123 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f91ec035030 con 0x7f91fc06b0a0 2024-03-23T03:47:46.891 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:46.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.893+0000 7f9203200640 1 -- 172.21.15.102:0/26429123 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f91d806ac00 msgr2=0x7f91d806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:46.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.893+0000 7f9203200640 1 --2- 172.21.15.102:0/26429123 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f91d806ac00 0x7f91d806d0c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f91e4006fd0 tx=0x7f91e4008040 comp rx=0 tx=0).stop 2024-03-23T03:47:46.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.893+0000 7f9203200640 1 -- 172.21.15.102:0/26429123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91fc06b0a0 msgr2=0x7f91fc14a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:46.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.893+0000 7f9203200640 1 --2- 172.21.15.102:0/26429123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91fc06b0a0 0x7f91fc14a240 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f91ec02b950 tx=0x7f91ec0046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.893+0000 7f9203200640 1 -- 172.21.15.102:0/26429123 shutdown_connections 2024-03-23T03:47:46.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.893+0000 7f9203200640 1 --2- 172.21.15.102:0/26429123 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f91d806ac00 0x7f91d806d0c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.893+0000 7f9203200640 1 --2- 172.21.15.102:0/26429123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91fc0d0d40 0x7f91fc0b7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.893+0000 7f9203200640 1 --2- 172.21.15.102:0/26429123 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91fc0cd740 0x7f91fc0cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.893+0000 7f9203200640 1 --2- 172.21.15.102:0/26429123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91fc06b0a0 0x7f91fc14a240 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:46.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.893+0000 7f9203200640 1 -- 172.21.15.102:0/26429123 >> 172.21.15.102:0/26429123 conn(0x7f91fc0cf5d0 msgr2=0x7f91fc0cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:46.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.893+0000 7f9203200640 1 -- 172.21.15.102:0/26429123 shutdown_connections 2024-03-23T03:47:46.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:46.893+0000 7f9203200640 1 -- 172.21.15.102:0/26429123 wait complete. 2024-03-23T03:47:46.913 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:47.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.116+0000 7fad68400640 1 Processor -- start 2024-03-23T03:47:47.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.116+0000 7fad68400640 1 -- start start 2024-03-23T03:47:47.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.118+0000 7fad68400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fad600b5a30 0x7fad600b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:47.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.118+0000 7fad68400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fad600b4350 0x7fad600b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:47.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.118+0000 7fad68400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad600d1830 con 0x7fad6015cda0 2024-03-23T03:47:47.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.118+0000 7fad68400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad600b4d20 con 0x7fad600b4350 2024-03-23T03:47:47.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.118+0000 7fad68400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad600b1f20 con 0x7fad600b5a30 2024-03-23T03:47:47.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.118+0000 7fad66400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fad600b4350 0x7fad600b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:47.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.118+0000 7fad65a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fad6015cda0 0x7fad600cdb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47166/0 (socket says 172.21.15.102:47166) 2024-03-23T03:47:47.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.118+0000 7fad65a00640 1 -- 172.21.15.102:0/3386321795 learned_addr learned my addr 172.21.15.102:0/3386321795 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:47.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.119+0000 7fad65000640 1 --2- 172.21.15.102:0/3386321795 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fad600b5a30 0x7fad600b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:47.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.119+0000 7fad66400640 1 -- 172.21.15.102:0/3386321795 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fad600b5a30 msgr2=0x7fad600b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:47.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.119+0000 7fad66400640 1 --2- 172.21.15.102:0/3386321795 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fad600b5a30 0x7fad600b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.119+0000 7fad66400640 1 -- 172.21.15.102:0/3386321795 >> v1:172.21.15.102:6789/0 conn(0x7fad6015cda0 legacy=0x7fad600cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:47.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.119+0000 7fad66400640 1 -- 172.21.15.102:0/3386321795 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad600d0c20 con 0x7fad600b4350 2024-03-23T03:47:47.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.120+0000 7fad66400640 1 --2- 172.21.15.102:0/3386321795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad600b4350 0x7fad600b47e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fad58009d90 tx=0x7fad5802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fde44571338fa8c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.120+0000 7fad5fe00640 1 -- 172.21.15.102:0/3386321795 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad5802cc60 con 0x7fad600b4350 2024-03-23T03:47:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.120+0000 7fad5fe00640 1 -- 172.21.15.102:0/3386321795 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad5802cdc0 con 0x7fad600b4350 2024-03-23T03:47:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.120+0000 7fad5fe00640 1 -- 172.21.15.102:0/3386321795 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad58034660 con 0x7fad600b4350 2024-03-23T03:47:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.121+0000 7fad68400640 1 -- 172.21.15.102:0/3386321795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad600b4350 msgr2=0x7fad600b47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.121+0000 7fad68400640 1 --2- 172.21.15.102:0/3386321795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad600b4350 0x7fad600b47e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fad58009d90 tx=0x7fad5802beb0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.121+0000 7fad68400640 1 -- 172.21.15.102:0/3386321795 shutdown_connections 2024-03-23T03:47:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.121+0000 7fad68400640 1 --2- 172.21.15.102:0/3386321795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad600b4350 0x7fad600b47e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.121+0000 7fad68400640 1 --2- 172.21.15.102:0/3386321795 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fad600b5a30 0x7fad600b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.121+0000 7fad68400640 1 -- 172.21.15.102:0/3386321795 >> 172.21.15.102:0/3386321795 conn(0x7fad600cf350 msgr2=0x7fad600cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.121+0000 7fad68400640 1 -- 172.21.15.102:0/3386321795 shutdown_connections 2024-03-23T03:47:47.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.121+0000 7fad68400640 1 -- 172.21.15.102:0/3386321795 wait complete. 2024-03-23T03:47:47.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.123+0000 7fad68400640 1 Processor -- start 2024-03-23T03:47:47.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.123+0000 7fad68400640 1 -- start start 2024-03-23T03:47:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.124+0000 7fad68400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad600b4350 0x7fad601f5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.124+0000 7fad68400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fad600b5a30 0x7fad600b74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:47.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.124+0000 7fad68400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad6015cda0 0x7fad600b79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.124+0000 7fad68400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad601602c0 con 0x7fad600b4350 2024-03-23T03:47:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.124+0000 7fad68400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad601f5690 con 0x7fad6015cda0 2024-03-23T03:47:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.124+0000 7fad68400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad601f2510 con 0x7fad600b5a30 2024-03-23T03:47:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.124+0000 7fad65a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad600b4350 0x7fad601f5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.124+0000 7fad65a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad600b4350 0x7fad601f5150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33290/0 (socket says 172.21.15.102:33290) 2024-03-23T03:47:47.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.124+0000 7fad65a00640 1 -- 172.21.15.102:0/4127883797 learned_addr learned my addr 172.21.15.102:0/4127883797 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:47.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.124+0000 7fad65a00640 1 -- 172.21.15.102:0/4127883797 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fad600b5a30 msgr2=0x7fad600b74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:47.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.124+0000 7fad65000640 1 --2- 172.21.15.102:0/4127883797 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fad600b5a30 0x7fad600b74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:47.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.124+0000 7fad65a00640 1 --2- 172.21.15.102:0/4127883797 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fad600b5a30 0x7fad600b74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.124+0000 7fad65a00640 1 -- 172.21.15.102:0/4127883797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad6015cda0 msgr2=0x7fad600b79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:47.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.124+0000 7fad65a00640 1 --2- 172.21.15.102:0/4127883797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad6015cda0 0x7fad600b79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.124+0000 7fad65a00640 1 -- 172.21.15.102:0/4127883797 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad58009a70 con 0x7fad600b4350 2024-03-23T03:47:47.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.125+0000 7fad65a00640 1 --2- 172.21.15.102:0/4127883797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad600b4350 0x7fad601f5150 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fad5000c480 tx=0x7fad5000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:47.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.125+0000 7fad5d600640 1 -- 172.21.15.102:0/4127883797 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad50005ce0 con 0x7fad600b4350 2024-03-23T03:47:47.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.125+0000 7fad5d600640 1 -- 172.21.15.102:0/4127883797 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad50005e40 con 0x7fad600b4350 2024-03-23T03:47:47.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.125+0000 7fad5d600640 1 -- 172.21.15.102:0/4127883797 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad500072f0 con 0x7fad600b4350 2024-03-23T03:47:47.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.125+0000 7fad68400640 1 -- 172.21.15.102:0/4127883797 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad601f27f0 con 0x7fad600b4350 2024-03-23T03:47:47.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.125+0000 7fad68400640 1 -- 172.21.15.102:0/4127883797 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad601f2db0 con 0x7fad600b4350 2024-03-23T03:47:47.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.126+0000 7fad68400640 1 -- 172.21.15.102:0/4127883797 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad601f2fd0 con 0x7fad600b4350 2024-03-23T03:47:47.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.127+0000 7fad5d600640 1 -- 172.21.15.102:0/4127883797 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fad50007450 con 0x7fad600b4350 2024-03-23T03:47:47.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.128+0000 7fad5d600640 1 --2- 172.21.15.102:0/4127883797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fad4006ac30 0x7fad4006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:47.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.128+0000 7fad5d600640 1 -- 172.21.15.102:0/4127883797 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fad50088ab0 con 0x7fad600b4350 2024-03-23T03:47:47.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.128+0000 7fad65000640 1 --2- 172.21.15.102:0/4127883797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fad4006ac30 0x7fad4006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:47.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.129+0000 7fad65000640 1 --2- 172.21.15.102:0/4127883797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fad4006ac30 0x7fad4006d0f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fad600cd520 tx=0x7fad4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:47.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.133+0000 7fad5d600640 1 -- 172.21.15.102:0/4127883797 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fad50053a10 con 0x7fad600b4350 2024-03-23T03:47:47.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.326+0000 7fad68400640 1 -- 172.21.15.102:0/4127883797 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad60073390 con 0x7fad600b4350 2024-03-23T03:47:47.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.329+0000 7fad5d600640 1 -- 172.21.15.102:0/4127883797 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18260 (secure 0 0 0) 0x7fad50017020 con 0x7fad600b4350 2024-03-23T03:47:47.330 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:47.330 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":28,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:47:41.647786+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:47:47.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.331+0000 7fad68400640 1 -- 172.21.15.102:0/4127883797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fad4006ac30 msgr2=0x7fad4006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:47.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.331+0000 7fad68400640 1 --2- 172.21.15.102:0/4127883797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fad4006ac30 0x7fad4006d0f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fad600cd520 tx=0x7fad4c008040 comp rx=0 tx=0).stop 2024-03-23T03:47:47.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.331+0000 7fad68400640 1 -- 172.21.15.102:0/4127883797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad600b4350 msgr2=0x7fad601f5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:47.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.331+0000 7fad68400640 1 --2- 172.21.15.102:0/4127883797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad600b4350 0x7fad601f5150 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fad5000c480 tx=0x7fad5000c940 comp rx=0 tx=0).stop 2024-03-23T03:47:47.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.332+0000 7fad68400640 1 -- 172.21.15.102:0/4127883797 shutdown_connections 2024-03-23T03:47:47.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.332+0000 7fad68400640 1 --2- 172.21.15.102:0/4127883797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fad4006ac30 0x7fad4006d0f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.332+0000 7fad68400640 1 --2- 172.21.15.102:0/4127883797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad6015cda0 0x7fad600b79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.332+0000 7fad68400640 1 --2- 172.21.15.102:0/4127883797 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fad600b5a30 0x7fad600b74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.332+0000 7fad68400640 1 --2- 172.21.15.102:0/4127883797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad600b4350 0x7fad601f5150 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.332+0000 7fad68400640 1 -- 172.21.15.102:0/4127883797 >> 172.21.15.102:0/4127883797 conn(0x7fad600cf350 msgr2=0x7fad600b2560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:47.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.332+0000 7fad68400640 1 -- 172.21.15.102:0/4127883797 shutdown_connections 2024-03-23T03:47:47.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.332+0000 7fad68400640 1 -- 172.21.15.102:0/4127883797 wait complete. 2024-03-23T03:47:47.352 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:46:52.158650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:46:52.914275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-23T03:46:55.983754+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-23T03:47:47.352 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:47:47.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.562+0000 7fabcf200640 1 Processor -- start 2024-03-23T03:47:47.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.562+0000 7fabcf200640 1 -- start start 2024-03-23T03:47:47.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.564+0000 7fabcf200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fabc815c820 0x7fabc80cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:47.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.564+0000 7fabcf200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fabc8160e90 0x7fabc80cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:47.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.564+0000 7fabcf200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc80cfa30 con 0x7fabc80cf5f0 2024-03-23T03:47:47.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.564+0000 7fabcf200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc80cdd70 con 0x7fabc8160e90 2024-03-23T03:47:47.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.564+0000 7fabcf200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc80ccd90 con 0x7fabc815c820 2024-03-23T03:47:47.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.564+0000 7fabcd400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fabc8160e90 0x7fabc80cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:47.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.565+0000 7fabcd400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fabc8160e90 0x7fabc80cf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33310/0 (socket says 172.21.15.102:33310) 2024-03-23T03:47:47.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.565+0000 7fabcd400640 1 -- 172.21.15.102:0/2176359336 learned_addr learned my addr 172.21.15.102:0/2176359336 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:47.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.565+0000 7fabcde00640 1 --2- 172.21.15.102:0/2176359336 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fabc815c820 0x7fabc80cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:47.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.565+0000 7fabcd400640 1 -- 172.21.15.102:0/2176359336 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fabc815c820 msgr2=0x7fabc80cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:47.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.565+0000 7fabcd400640 1 --2- 172.21.15.102:0/2176359336 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fabc815c820 0x7fabc80cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.565+0000 7fabcd400640 1 -- 172.21.15.102:0/2176359336 >> v1:172.21.15.173:6789/0 conn(0x7fabc80cf5f0 legacy=0x7fabc80d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:47.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.565+0000 7fabcd400640 1 -- 172.21.15.102:0/2176359336 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc80b7fc0 con 0x7fabc8160e90 2024-03-23T03:47:47.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.566+0000 7fabcd400640 1 --2- 172.21.15.102:0/2176359336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabc8160e90 0x7fabc80cf0b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fabb8009870 tx=0x7fabb802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ba42d5740aaa8dd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:47.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.566+0000 7fabcca00640 1 -- 172.21.15.102:0/2176359336 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb8030840 con 0x7fabc8160e90 2024-03-23T03:47:47.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.567+0000 7fabcca00640 1 -- 172.21.15.102:0/2176359336 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb80309a0 con 0x7fabc8160e90 2024-03-23T03:47:47.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.567+0000 7fabcca00640 1 -- 172.21.15.102:0/2176359336 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb8030d30 con 0x7fabc8160e90 2024-03-23T03:47:47.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.567+0000 7fabcf200640 1 -- 172.21.15.102:0/2176359336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabc8160e90 msgr2=0x7fabc80cf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:47.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.567+0000 7fabcf200640 1 --2- 172.21.15.102:0/2176359336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabc8160e90 0x7fabc80cf0b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fabb8009870 tx=0x7fabb802b980 comp rx=0 tx=0).stop 2024-03-23T03:47:47.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.567+0000 7fabcf200640 1 -- 172.21.15.102:0/2176359336 shutdown_connections 2024-03-23T03:47:47.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.567+0000 7fabcf200640 1 --2- 172.21.15.102:0/2176359336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabc8160e90 0x7fabc80cf0b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.567+0000 7fabcf200640 1 --2- 172.21.15.102:0/2176359336 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fabc815c820 0x7fabc80cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.567+0000 7fabcf200640 1 -- 172.21.15.102:0/2176359336 >> 172.21.15.102:0/2176359336 conn(0x7fabc806b050 msgr2=0x7fabc806b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:47.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.567+0000 7fabcf200640 1 -- 172.21.15.102:0/2176359336 shutdown_connections 2024-03-23T03:47:47.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.567+0000 7fabcf200640 1 -- 172.21.15.102:0/2176359336 wait complete. 2024-03-23T03:47:47.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.569+0000 7fabcf200640 1 Processor -- start 2024-03-23T03:47:47.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.570+0000 7fabcf200640 1 -- start start 2024-03-23T03:47:47.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.571+0000 7fabcf200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fabc80cf5f0 0x7fabc815bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:47.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.571+0000 7fabcf200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabc815c820 0x7fabc80ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:47.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.571+0000 7fabcf200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabc8160e90 0x7fabc80ca930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:47.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.571+0000 7fabcf200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc81f32a0 con 0x7fabc8160e90 2024-03-23T03:47:47.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.571+0000 7fabcf200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc81f2360 con 0x7fabc815c820 2024-03-23T03:47:47.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.571+0000 7fabcf200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc81f2620 con 0x7fabc80cf5f0 2024-03-23T03:47:47.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.571+0000 7fabce800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabc8160e90 0x7fabc80ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:47.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.571+0000 7fabce800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabc8160e90 0x7fabc80ca930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33314/0 (socket says 172.21.15.102:33314) 2024-03-23T03:47:47.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.571+0000 7fabce800640 1 -- 172.21.15.102:0/4201269893 learned_addr learned my addr 172.21.15.102:0/4201269893 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:47.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.571+0000 7fabcde00640 1 --2- 172.21.15.102:0/4201269893 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fabc80cf5f0 0x7fabc815bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:47.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.572+0000 7fabcd400640 1 --2- 172.21.15.102:0/4201269893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabc815c820 0x7fabc80ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:47.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.572+0000 7fabce800640 1 -- 172.21.15.102:0/4201269893 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fabc80cf5f0 msgr2=0x7fabc815bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:47.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.572+0000 7fabce800640 1 --2- 172.21.15.102:0/4201269893 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fabc80cf5f0 0x7fabc815bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.572+0000 7fabce800640 1 -- 172.21.15.102:0/4201269893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabc815c820 msgr2=0x7fabc80ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:47.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.572+0000 7fabce800640 1 --2- 172.21.15.102:0/4201269893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabc815c820 0x7fabc80ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.572+0000 7fabce800640 1 -- 172.21.15.102:0/4201269893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb80094d0 con 0x7fabc8160e90 2024-03-23T03:47:47.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.572+0000 7fabcd400640 1 --2- 172.21.15.102:0/4201269893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabc815c820 0x7fabc80ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:47.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.572+0000 7fabce800640 1 --2- 172.21.15.102:0/4201269893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabc8160e90 0x7fabc80ca930 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fabc400b970 tx=0x7fabc400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:47.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.572+0000 7fabc2000640 1 -- 172.21.15.102:0/4201269893 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc400cb40 con 0x7fabc8160e90 2024-03-23T03:47:47.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.572+0000 7fabc2000640 1 -- 172.21.15.102:0/4201269893 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc400cca0 con 0x7fabc8160e90 2024-03-23T03:47:47.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.572+0000 7fabc2000640 1 -- 172.21.15.102:0/4201269893 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc4014680 con 0x7fabc8160e90 2024-03-23T03:47:47.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.572+0000 7fabcf200640 1 -- 172.21.15.102:0/4201269893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc81f2900 con 0x7fabc8160e90 2024-03-23T03:47:47.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.572+0000 7fabcf200640 1 -- 172.21.15.102:0/4201269893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc81f2ec0 con 0x7fabc8160e90 2024-03-23T03:47:47.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.574+0000 7fabcf200640 1 -- 172.21.15.102:0/4201269893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc80d2480 con 0x7fabc8160e90 2024-03-23T03:47:47.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.575+0000 7fabc2000640 1 -- 172.21.15.102:0/4201269893 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fabc401d3b0 con 0x7fabc8160e90 2024-03-23T03:47:47.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.575+0000 7fabc2000640 1 --2- 172.21.15.102:0/4201269893 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fabac06ac00 0x7fabac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:47.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.575+0000 7fabc2000640 1 -- 172.21.15.102:0/4201269893 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fabc4089600 con 0x7fabc8160e90 2024-03-23T03:47:47.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.576+0000 7fabcde00640 1 --2- 172.21.15.102:0/4201269893 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fabac06ac00 0x7fabac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:47.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.576+0000 7fabcde00640 1 --2- 172.21.15.102:0/4201269893 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fabac06ac00 0x7fabac06d0c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fabbc006fd0 tx=0x7fabbc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:47.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.580+0000 7fabc2000640 1 -- 172.21.15.102:0/4201269893 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fabc4054380 con 0x7fabc8160e90 2024-03-23T03:47:47.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.774+0000 7fabcf200640 1 -- 172.21.15.102:0/4201269893 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fabc804e030 con 0x7fabc8160e90 2024-03-23T03:47:47.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.775+0000 7fabc2000640 1 -- 172.21.15.102:0/4201269893 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7fabc40579a0 con 0x7fabc8160e90 2024-03-23T03:47:47.776 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:47:47.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.777+0000 7fabcf200640 1 -- 172.21.15.102:0/4201269893 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fabac06ac00 msgr2=0x7fabac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:47.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.777+0000 7fabcf200640 1 --2- 172.21.15.102:0/4201269893 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fabac06ac00 0x7fabac06d0c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fabbc006fd0 tx=0x7fabbc008040 comp rx=0 tx=0).stop 2024-03-23T03:47:47.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.777+0000 7fabcf200640 1 -- 172.21.15.102:0/4201269893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabc8160e90 msgr2=0x7fabc80ca930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:47.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.777+0000 7fabcf200640 1 --2- 172.21.15.102:0/4201269893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabc8160e90 0x7fabc80ca930 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fabc400b970 tx=0x7fabc400be30 comp rx=0 tx=0).stop 2024-03-23T03:47:47.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.777+0000 7fabcf200640 1 -- 172.21.15.102:0/4201269893 shutdown_connections 2024-03-23T03:47:47.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.777+0000 7fabcf200640 1 --2- 172.21.15.102:0/4201269893 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fabac06ac00 0x7fabac06d0c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.778+0000 7fabcf200640 1 --2- 172.21.15.102:0/4201269893 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabc8160e90 0x7fabc80ca930 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.778+0000 7fabcf200640 1 --2- 172.21.15.102:0/4201269893 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fabc815c820 0x7fabc80ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.778+0000 7fabcf200640 1 --2- 172.21.15.102:0/4201269893 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fabc80cf5f0 0x7fabc815bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:47.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.778+0000 7fabcf200640 1 -- 172.21.15.102:0/4201269893 >> 172.21.15.102:0/4201269893 conn(0x7fabc806b050 msgr2=0x7fabc806b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:47.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.778+0000 7fabcf200640 1 -- 172.21.15.102:0/4201269893 shutdown_connections 2024-03-23T03:47:47.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.778+0000 7fabcf200640 1 -- 172.21.15.102:0/4201269893 wait complete. 2024-03-23T03:47:47.796 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:47:47.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.998+0000 7f3b18e00640 1 Processor -- start 2024-03-23T03:47:47.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:47.998+0000 7f3b18e00640 1 -- start start 2024-03-23T03:47:48.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.000+0000 7f3b18e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3b140b84b0 0x7f3b140d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.000+0000 7f3b18e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b140b5aa0 0x7f3b140b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.000+0000 7f3b18e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b140d1280 con 0x7f3b140cd740 2024-03-23T03:47:48.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.000+0000 7f3b18e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b140d20c0 con 0x7f3b140b84b0 2024-03-23T03:47:48.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.000+0000 7f3b18e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b140b8940 con 0x7f3b140b5aa0 2024-03-23T03:47:48.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.000+0000 7f3b11400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3b140b84b0 0x7f3b140d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.000+0000 7f3b11400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3b140b84b0 0x7f3b140d0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33322/0 (socket says 172.21.15.102:33322) 2024-03-23T03:47:48.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.000+0000 7f3b11400640 1 -- 172.21.15.102:0/3960806712 learned_addr learned my addr 172.21.15.102:0/3960806712 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:48.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.000+0000 7f3b11400640 1 -- 172.21.15.102:0/3960806712 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b140b5aa0 msgr2=0x7f3b140b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:48.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.000+0000 7f3b12800640 1 --2- 172.21.15.102:0/3960806712 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b140b5aa0 0x7f3b140b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.000+0000 7f3b11400640 1 --2- 172.21.15.102:0/3960806712 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b140b5aa0 0x7f3b140b5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.000+0000 7f3b11400640 1 -- 172.21.15.102:0/3960806712 >> v1:172.21.15.173:6789/0 conn(0x7f3b140cd740 legacy=0x7f3b140cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:48.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.001+0000 7f3b11400640 1 -- 172.21.15.102:0/3960806712 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b140d14b0 con 0x7f3b140b84b0 2024-03-23T03:47:48.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.001+0000 7f3b11400640 1 --2- 172.21.15.102:0/3960806712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b140b84b0 0x7f3b140d0d40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009870 tx=0x7f3afc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=412a797c92731cee server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:48.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.002+0000 7f3b10a00640 1 -- 172.21.15.102:0/3960806712 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc031640 con 0x7f3b140b84b0 2024-03-23T03:47:48.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.002+0000 7f3b10a00640 1 -- 172.21.15.102:0/3960806712 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3afc0317a0 con 0x7f3b140b84b0 2024-03-23T03:47:48.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.002+0000 7f3b10a00640 1 -- 172.21.15.102:0/3960806712 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc031b30 con 0x7f3b140b84b0 2024-03-23T03:47:48.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.002+0000 7f3b18e00640 1 -- 172.21.15.102:0/3960806712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b140b84b0 msgr2=0x7f3b140d0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:48.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.002+0000 7f3b18e00640 1 --2- 172.21.15.102:0/3960806712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b140b84b0 0x7f3b140d0d40 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009870 tx=0x7f3afc02b8f0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.002+0000 7f3b18e00640 1 -- 172.21.15.102:0/3960806712 shutdown_connections 2024-03-23T03:47:48.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.002+0000 7f3b18e00640 1 --2- 172.21.15.102:0/3960806712 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b140b5aa0 0x7f3b140b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.002+0000 7f3b18e00640 1 --2- 172.21.15.102:0/3960806712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b140b84b0 0x7f3b140d0d40 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.002+0000 7f3b18e00640 1 -- 172.21.15.102:0/3960806712 >> 172.21.15.102:0/3960806712 conn(0x7f3b140cf5d0 msgr2=0x7f3b140cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:48.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.003+0000 7f3b18e00640 1 -- 172.21.15.102:0/3960806712 shutdown_connections 2024-03-23T03:47:48.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.003+0000 7f3b18e00640 1 -- 172.21.15.102:0/3960806712 wait complete. 2024-03-23T03:47:48.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.004+0000 7f3b18e00640 1 Processor -- start 2024-03-23T03:47:48.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.004+0000 7f3b18e00640 1 -- start start 2024-03-23T03:47:48.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.005+0000 7f3b18e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b140b5aa0 0x7f3b140cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.005+0000 7f3b18e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b140b84b0 0x7f3b141f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.005+0000 7f3b18e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b140cd740 0x7f3b14157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.005+0000 7f3b18e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b14157a40 con 0x7f3b140b84b0 2024-03-23T03:47:48.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.005+0000 7f3b18e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b14157d00 con 0x7f3b140cd740 2024-03-23T03:47:48.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.005+0000 7f3b18e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b14157fe0 con 0x7f3b140b5aa0 2024-03-23T03:47:48.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.005+0000 7f3b11400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b140b84b0 0x7f3b141f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.005+0000 7f3b11400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b140b84b0 0x7f3b141f2f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33332/0 (socket says 172.21.15.102:33332) 2024-03-23T03:47:48.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.005+0000 7f3b11400640 1 -- 172.21.15.102:0/990745102 learned_addr learned my addr 172.21.15.102:0/990745102 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:48.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.006+0000 7f3b11e00640 1 --2- 172.21.15.102:0/990745102 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b140b5aa0 0x7f3b140cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.006+0000 7f3b12800640 1 --2- 172.21.15.102:0/990745102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b140cd740 0x7f3b14157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.006+0000 7f3b11400640 1 -- 172.21.15.102:0/990745102 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b140b5aa0 msgr2=0x7f3b140cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:48.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.006+0000 7f3b11400640 1 --2- 172.21.15.102:0/990745102 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b140b5aa0 0x7f3b140cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.006+0000 7f3b11400640 1 -- 172.21.15.102:0/990745102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b140cd740 msgr2=0x7f3b14157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:48.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.006+0000 7f3b11400640 1 --2- 172.21.15.102:0/990745102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b140cd740 0x7f3b14157500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.006+0000 7f3b11400640 1 -- 172.21.15.102:0/990745102 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3afc0094d0 con 0x7f3b140b84b0 2024-03-23T03:47:48.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.006+0000 7f3b11400640 1 --2- 172.21.15.102:0/990745102 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b140b84b0 0x7f3b141f2f20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009870 tx=0x7f3afc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:48.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.007+0000 7f3b0a000640 1 -- 172.21.15.102:0/990745102 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc03ac60 con 0x7f3b140b84b0 2024-03-23T03:47:48.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.007+0000 7f3b0a000640 1 -- 172.21.15.102:0/990745102 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3afc03ade0 con 0x7f3b140b84b0 2024-03-23T03:47:48.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.007+0000 7f3b0a000640 1 -- 172.21.15.102:0/990745102 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc044630 con 0x7f3b140b84b0 2024-03-23T03:47:48.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.007+0000 7f3b18e00640 1 -- 172.21.15.102:0/990745102 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b141582c0 con 0x7f3b140b84b0 2024-03-23T03:47:48.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.007+0000 7f3b18e00640 1 -- 172.21.15.102:0/990745102 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b14158880 con 0x7f3b140b84b0 2024-03-23T03:47:48.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.008+0000 7f3b18e00640 1 -- 172.21.15.102:0/990745102 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b14073390 con 0x7f3b140b84b0 2024-03-23T03:47:48.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.009+0000 7f3b0a000640 1 -- 172.21.15.102:0/990745102 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3afc02f050 con 0x7f3b140b84b0 2024-03-23T03:47:48.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.009+0000 7f3b0a000640 1 --2- 172.21.15.102:0/990745102 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3af006ac00 0x7f3af006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.010+0000 7f3b0a000640 1 -- 172.21.15.102:0/990745102 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3afc0ab9a0 con 0x7f3b140b84b0 2024-03-23T03:47:48.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.010+0000 7f3b11e00640 1 --2- 172.21.15.102:0/990745102 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3af006ac00 0x7f3af006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.011+0000 7f3b11e00640 1 --2- 172.21.15.102:0/990745102 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3af006ac00 0x7f3af006d0c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3b04007fd0 tx=0x7f3b04009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:48.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.014+0000 7f3b0a000640 1 -- 172.21.15.102:0/990745102 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3afc0768e0 con 0x7f3b140b84b0 2024-03-23T03:47:48.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.208+0000 7f3b18e00640 1 -- 172.21.15.102:0/990745102 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3b140b20b0 con 0x7f3b140b84b0 2024-03-23T03:47:48.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.209+0000 7f3b0a000640 1 -- 172.21.15.102:0/990745102 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3afc079f00 con 0x7f3b140b84b0 2024-03-23T03:47:48.209 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:48.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.211+0000 7f3b18e00640 1 -- 172.21.15.102:0/990745102 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3af006ac00 msgr2=0x7f3af006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:48.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.211+0000 7f3b18e00640 1 --2- 172.21.15.102:0/990745102 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3af006ac00 0x7f3af006d0c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f3b04007fd0 tx=0x7f3b04009040 comp rx=0 tx=0).stop 2024-03-23T03:47:48.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.211+0000 7f3b18e00640 1 -- 172.21.15.102:0/990745102 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b140b84b0 msgr2=0x7f3b141f2f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:48.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.211+0000 7f3b18e00640 1 --2- 172.21.15.102:0/990745102 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b140b84b0 0x7f3b141f2f20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009870 tx=0x7f3afc0046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.211+0000 7f3b18e00640 1 -- 172.21.15.102:0/990745102 shutdown_connections 2024-03-23T03:47:48.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.211+0000 7f3b18e00640 1 --2- 172.21.15.102:0/990745102 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3af006ac00 0x7f3af006d0c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.211+0000 7f3b18e00640 1 --2- 172.21.15.102:0/990745102 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b140cd740 0x7f3b14157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.211+0000 7f3b18e00640 1 --2- 172.21.15.102:0/990745102 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b140b84b0 0x7f3b141f2f20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.211+0000 7f3b18e00640 1 --2- 172.21.15.102:0/990745102 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b140b5aa0 0x7f3b140cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.211+0000 7f3b18e00640 1 -- 172.21.15.102:0/990745102 >> 172.21.15.102:0/990745102 conn(0x7f3b140cf5d0 msgr2=0x7f3b140d2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:48.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.212+0000 7f3b18e00640 1 -- 172.21.15.102:0/990745102 shutdown_connections 2024-03-23T03:47:48.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.212+0000 7f3b18e00640 1 -- 172.21.15.102:0/990745102 wait complete. 2024-03-23T03:47:48.230 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:47:48.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.437+0000 7fa278000640 1 Processor -- start 2024-03-23T03:47:48.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.437+0000 7fa278000640 1 -- start start 2024-03-23T03:47:48.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.438+0000 7fa278000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa270159830 0x7fa2700cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.438+0000 7fa278000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2700ca7b0 0x7fa2700cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.438+0000 7fa278000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa270159c10 con 0x7fa2700ce930 2024-03-23T03:47:48.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.438+0000 7fa278000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa27006de50 con 0x7fa270159830 2024-03-23T03:47:48.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.438+0000 7fa278000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2700cd840 con 0x7fa2700ca7b0 2024-03-23T03:47:48.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.439+0000 7fa276000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa2700ce930 0x7fa2700b18d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47176/0 (socket says 172.21.15.102:47176) 2024-03-23T03:47:48.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.439+0000 7fa276000640 1 -- 172.21.15.102:0/1379624403 learned_addr learned my addr 172.21.15.102:0/1379624403 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:48.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.439+0000 7fa274c00640 1 --2- 172.21.15.102:0/1379624403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2700ca7b0 0x7fa2700cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.439+0000 7fa275600640 1 --2- 172.21.15.102:0/1379624403 >> v2:172.21.15.173:3300/0 conn(0x7fa270159830 0x7fa2700cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.439+0000 7fa26fe00640 1 -- 172.21.15.102:0/1379624403 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2777189180 0 0) 0x7fa270159c10 con 0x7fa2700ce930 2024-03-23T03:47:48.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.439+0000 7fa26fe00640 1 -- 172.21.15.102:0/1379624403 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa258003520 con 0x7fa2700ce930 2024-03-23T03:47:48.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.439+0000 7fa26fe00640 1 -- 172.21.15.102:0/1379624403 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2563562503 0 0) 0x7fa258003520 con 0x7fa2700ce930 2024-03-23T03:47:48.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.439+0000 7fa26fe00640 1 -- 172.21.15.102:0/1379624403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2700ca7b0 msgr2=0x7fa2700cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:48.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.439+0000 7fa26fe00640 1 --2- 172.21.15.102:0/1379624403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2700ca7b0 0x7fa2700cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.439+0000 7fa26fe00640 1 -- 172.21.15.102:0/1379624403 >> v2:172.21.15.173:3300/0 conn(0x7fa270159830 msgr2=0x7fa2700cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:48.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.439+0000 7fa26fe00640 1 --2- 172.21.15.102:0/1379624403 >> v2:172.21.15.173:3300/0 conn(0x7fa270159830 0x7fa2700cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.439+0000 7fa26fe00640 1 -- 172.21.15.102:0/1379624403 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2700ccc30 con 0x7fa2700ce930 2024-03-23T03:47:48.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.439+0000 7fa26fe00640 1 -- 172.21.15.102:0/1379624403 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fa2680019f0 con 0x7fa2700ce930 2024-03-23T03:47:48.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.439+0000 7fa275600640 1 --2- 172.21.15.102:0/1379624403 >> v2:172.21.15.173:3300/0 conn(0x7fa270159830 0x7fa2700cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:47:48.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.439+0000 7fa274c00640 1 --2- 172.21.15.102:0/1379624403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2700ca7b0 0x7fa2700cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:47:48.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.440+0000 7fa26fe00640 1 -- 172.21.15.102:0/1379624403 >> v1:172.21.15.102:6789/0 conn(0x7fa2700ce930 legacy=0x7fa2700b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:48.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.440+0000 7fa26fe00640 1 --2- 172.21.15.102:0/1379624403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2700ca7b0 0x7fa258003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.440+0000 7fa26fe00640 1 --2- 172.21.15.102:0/1379624403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa258003fb0 0x7fa2580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.440+0000 7fa26fe00640 1 --2- 172.21.15.102:0/1379624403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa258004d00 0x7fa258005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.440+0000 7fa26fe00640 1 -- 172.21.15.102:0/1379624403 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2700cd840 con 0x7fa258004d00 2024-03-23T03:47:48.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.440+0000 7fa26fe00640 1 -- 172.21.15.102:0/1379624403 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25800bac0 con 0x7fa258003fb0 2024-03-23T03:47:48.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.440+0000 7fa26fe00640 1 -- 172.21.15.102:0/1379624403 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa25800bc80 con 0x7fa2700ca7b0 2024-03-23T03:47:48.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.440+0000 7fa276000640 1 --2- 172.21.15.102:0/1379624403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa258004d00 0x7fa258005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.440+0000 7fa275600640 1 --2- 172.21.15.102:0/1379624403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2700ca7b0 0x7fa258003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.440+0000 7fa274c00640 1 --2- 172.21.15.102:0/1379624403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa258003fb0 0x7fa2580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.440+0000 7fa276000640 1 -- 172.21.15.102:0/1379624403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2700ca7b0 msgr2=0x7fa258003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:48.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.440+0000 7fa276000640 1 --2- 172.21.15.102:0/1379624403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2700ca7b0 0x7fa258003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.440+0000 7fa276000640 1 -- 172.21.15.102:0/1379624403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa258003fb0 msgr2=0x7fa2580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:48.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.440+0000 7fa276000640 1 --2- 172.21.15.102:0/1379624403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa258003fb0 0x7fa2580047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.440+0000 7fa276000640 1 -- 172.21.15.102:0/1379624403 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa258003520 con 0x7fa258004d00 2024-03-23T03:47:48.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.441+0000 7fa276000640 1 --2- 172.21.15.102:0/1379624403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa258004d00 0x7fa258005180 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa26800a790 tx=0x7fa2580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb98863dc6d5d92 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:48.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.442+0000 7fa26fe00640 1 -- 172.21.15.102:0/1379624403 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268034630 con 0x7fa258004d00 2024-03-23T03:47:48.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.442+0000 7fa26fe00640 1 -- 172.21.15.102:0/1379624403 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa26802f070 con 0x7fa258004d00 2024-03-23T03:47:48.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.442+0000 7fa26fe00640 1 -- 172.21.15.102:0/1379624403 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268034a30 con 0x7fa258004d00 2024-03-23T03:47:48.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.442+0000 7fa278000640 1 -- 172.21.15.102:0/1379624403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa258004d00 msgr2=0x7fa258005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:48.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.442+0000 7fa278000640 1 --2- 172.21.15.102:0/1379624403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa258004d00 0x7fa258005180 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa26800a790 tx=0x7fa2580057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.442+0000 7fa276000640 1 -- 172.21.15.102:0/1379624403 reap_dead start 2024-03-23T03:47:48.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.442+0000 7fa278000640 1 -- 172.21.15.102:0/1379624403 shutdown_connections 2024-03-23T03:47:48.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.442+0000 7fa278000640 1 -- 172.21.15.102:0/1379624403 >> 172.21.15.102:0/1379624403 conn(0x7fa27006a0b0 msgr2=0x7fa27006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:48.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.443+0000 7fa278000640 1 -- 172.21.15.102:0/1379624403 shutdown_connections 2024-03-23T03:47:48.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.443+0000 7fa278000640 1 -- 172.21.15.102:0/1379624403 wait complete. 2024-03-23T03:47:48.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.444+0000 7fa278000640 1 Processor -- start 2024-03-23T03:47:48.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.445+0000 7fa278000640 1 -- start start 2024-03-23T03:47:48.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.445+0000 7fa278000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2701f1ce0 0x7fa27015ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.445+0000 7fa278000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2701f2fa0 0x7fa2701e5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.445+0000 7fa278000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2701e5eb0 0x7fa2700b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.445+0000 7fa278000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2701e6330 con 0x7fa2701e5eb0 2024-03-23T03:47:48.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.445+0000 7fa278000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2701f5920 con 0x7fa2701f1ce0 2024-03-23T03:47:48.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.445+0000 7fa278000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2701f5bc0 con 0x7fa2701f2fa0 2024-03-23T03:47:48.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.446+0000 7fa276000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2701e5eb0 0x7fa2700b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.446+0000 7fa276000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2701e5eb0 0x7fa2700b23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33340/0 (socket says 172.21.15.102:33340) 2024-03-23T03:47:48.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.446+0000 7fa276000640 1 -- 172.21.15.102:0/2237213597 learned_addr learned my addr 172.21.15.102:0/2237213597 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:48.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.446+0000 7fa274c00640 1 --2- 172.21.15.102:0/2237213597 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2701f2fa0 0x7fa2701e5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.446+0000 7fa275600640 1 --2- 172.21.15.102:0/2237213597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2701f1ce0 0x7fa27015ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.446+0000 7fa276000640 1 -- 172.21.15.102:0/2237213597 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2701f2fa0 msgr2=0x7fa2701e5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:48.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.446+0000 7fa276000640 1 --2- 172.21.15.102:0/2237213597 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2701f2fa0 0x7fa2701e5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.446+0000 7fa276000640 1 -- 172.21.15.102:0/2237213597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2701f1ce0 msgr2=0x7fa27015ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:48.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.446+0000 7fa276000640 1 --2- 172.21.15.102:0/2237213597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2701f1ce0 0x7fa27015ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.446+0000 7fa276000640 1 -- 172.21.15.102:0/2237213597 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2701f5ea0 con 0x7fa2701e5eb0 2024-03-23T03:47:48.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.447+0000 7fa276000640 1 --2- 172.21.15.102:0/2237213597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2701e5eb0 0x7fa2700b23e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa26800a8b0 tx=0x7fa268001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:48.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.447+0000 7fa26d600640 1 -- 172.21.15.102:0/2237213597 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268034840 con 0x7fa2701e5eb0 2024-03-23T03:47:48.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.447+0000 7fa26d600640 1 -- 172.21.15.102:0/2237213597 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2680019e0 con 0x7fa2701e5eb0 2024-03-23T03:47:48.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.447+0000 7fa26d600640 1 -- 172.21.15.102:0/2237213597 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa268034be0 con 0x7fa2701e5eb0 2024-03-23T03:47:48.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.447+0000 7fa278000640 1 -- 172.21.15.102:0/2237213597 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2701f6180 con 0x7fa2701e5eb0 2024-03-23T03:47:48.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.447+0000 7fa278000640 1 -- 172.21.15.102:0/2237213597 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2701f20c0 con 0x7fa2701e5eb0 2024-03-23T03:47:48.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.448+0000 7fa278000640 1 -- 172.21.15.102:0/2237213597 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa270073390 con 0x7fa2701e5eb0 2024-03-23T03:47:48.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.449+0000 7fa26d600640 1 -- 172.21.15.102:0/2237213597 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa268031030 con 0x7fa2701e5eb0 2024-03-23T03:47:48.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.450+0000 7fa26d600640 1 --2- 172.21.15.102:0/2237213597 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa25006ac00 0x7fa25006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.450+0000 7fa26d600640 1 -- 172.21.15.102:0/2237213597 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa2680ab980 con 0x7fa2701e5eb0 2024-03-23T03:47:48.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.451+0000 7fa275600640 1 --2- 172.21.15.102:0/2237213597 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa25006ac00 0x7fa25006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.451+0000 7fa275600640 1 --2- 172.21.15.102:0/2237213597 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa25006ac00 0x7fa25006d0c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa264006fd0 tx=0x7fa264008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:48.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.455+0000 7fa26d600640 1 -- 172.21.15.102:0/2237213597 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa26807a3e0 con 0x7fa2701e5eb0 2024-03-23T03:47:48.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.649+0000 7fa278000640 1 -- 172.21.15.102:0/2237213597 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa270073390 con 0x7fa2701e5eb0 2024-03-23T03:47:48.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.650+0000 7fa26d600640 1 -- 172.21.15.102:0/2237213597 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa268001f00 con 0x7fa2701e5eb0 2024-03-23T03:47:48.650 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:47:48.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.652+0000 7fa278000640 1 -- 172.21.15.102:0/2237213597 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa25006ac00 msgr2=0x7fa25006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:48.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.652+0000 7fa278000640 1 --2- 172.21.15.102:0/2237213597 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa25006ac00 0x7fa25006d0c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa264006fd0 tx=0x7fa264008040 comp rx=0 tx=0).stop 2024-03-23T03:47:48.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.652+0000 7fa278000640 1 -- 172.21.15.102:0/2237213597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2701e5eb0 msgr2=0x7fa2700b23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:48.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.652+0000 7fa278000640 1 --2- 172.21.15.102:0/2237213597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2701e5eb0 0x7fa2700b23e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa26800a8b0 tx=0x7fa268001cb0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.653+0000 7fa278000640 1 -- 172.21.15.102:0/2237213597 shutdown_connections 2024-03-23T03:47:48.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.653+0000 7fa278000640 1 --2- 172.21.15.102:0/2237213597 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa25006ac00 0x7fa25006d0c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.653+0000 7fa278000640 1 --2- 172.21.15.102:0/2237213597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2701e5eb0 0x7fa2700b23e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.653+0000 7fa278000640 1 --2- 172.21.15.102:0/2237213597 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa2701f2fa0 0x7fa2701e5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.653+0000 7fa278000640 1 --2- 172.21.15.102:0/2237213597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa2701f1ce0 0x7fa27015ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.653+0000 7fa278000640 1 -- 172.21.15.102:0/2237213597 >> 172.21.15.102:0/2237213597 conn(0x7fa27006a0b0 msgr2=0x7fa270158420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:48.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.653+0000 7fa278000640 1 -- 172.21.15.102:0/2237213597 shutdown_connections 2024-03-23T03:47:48.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.653+0000 7fa278000640 1 -- 172.21.15.102:0/2237213597 wait complete. 2024-03-23T03:47:48.672 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-23T03:47:48.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.875+0000 7f27f1000640 1 Processor -- start 2024-03-23T03:47:48.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.876+0000 7f27f1000640 1 -- start start 2024-03-23T03:47:48.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.877+0000 7f27f1000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f27ec06afb0 0x7f27ec06be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.877+0000 7f27f1000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ec0cf880 con 0x7f27ec15cb60 2024-03-23T03:47:48.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.877+0000 7f27f1000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27ec0cec70 con 0x7f27ec098a40 2024-03-23T03:47:48.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.877+0000 7f27f1000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27ec0cef50 con 0x7f27ec06afb0 2024-03-23T03:47:48.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.877+0000 7f27e1400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f27ec06afb0 0x7f27ec06be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.877+0000 7f27e9e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f27ec15cb60 0x7f27ec0cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47190/0 (socket says 172.21.15.102:47190) 2024-03-23T03:47:48.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.877+0000 7f27e9e00640 1 -- 172.21.15.102:0/1620780602 learned_addr learned my addr 172.21.15.102:0/1620780602 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:48.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.878+0000 7f27e1400640 1 -- 172.21.15.102:0/1620780602 >> v1:172.21.15.173:6789/0 conn(0x7f27ec098a40 legacy=0x7f27ec0cace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:48.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.878+0000 7f27e1400640 1 -- 172.21.15.102:0/1620780602 >> v1:172.21.15.102:6789/0 conn(0x7f27ec15cb60 legacy=0x7f27ec0cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:48.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.878+0000 7f27e1400640 1 -- 172.21.15.102:0/1620780602 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ec0cf230 con 0x7f27ec06afb0 2024-03-23T03:47:48.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.879+0000 7f27e1400640 1 --2- 172.21.15.102:0/1620780602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27ec06afb0 0x7f27ec06be90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f27d40097a0 tx=0x7f27d402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b08239e9a949fc3d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:48.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.879+0000 7f27e9400640 1 -- 172.21.15.102:0/1620780602 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d4031830 con 0x7f27ec06afb0 2024-03-23T03:47:48.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.879+0000 7f27e9400640 1 -- 172.21.15.102:0/1620780602 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d4031990 con 0x7f27ec06afb0 2024-03-23T03:47:48.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.879+0000 7f27e9400640 1 -- 172.21.15.102:0/1620780602 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d4031d20 con 0x7f27ec06afb0 2024-03-23T03:47:48.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.880+0000 7f27f1000640 1 -- 172.21.15.102:0/1620780602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27ec06afb0 msgr2=0x7f27ec06be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:48.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.880+0000 7f27f1000640 1 --2- 172.21.15.102:0/1620780602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27ec06afb0 0x7f27ec06be90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f27d40097a0 tx=0x7f27d402b8d0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.880+0000 7f27f1000640 1 -- 172.21.15.102:0/1620780602 shutdown_connections 2024-03-23T03:47:48.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.880+0000 7f27f1000640 1 --2- 172.21.15.102:0/1620780602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27ec06afb0 0x7f27ec06be90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.880+0000 7f27f1000640 1 -- 172.21.15.102:0/1620780602 >> 172.21.15.102:0/1620780602 conn(0x7f27ec0b2180 msgr2=0x7f27ec0b2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:48.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.880+0000 7f27f1000640 1 -- 172.21.15.102:0/1620780602 shutdown_connections 2024-03-23T03:47:48.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.880+0000 7f27f1000640 1 -- 172.21.15.102:0/1620780602 wait complete. 2024-03-23T03:47:48.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.882+0000 7f27f1000640 1 Processor -- start 2024-03-23T03:47:48.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.882+0000 7f27f1000640 1 -- start start 2024-03-23T03:47:48.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.883+0000 7f27f1000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27ec06afb0 0x7f27ec0af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.883+0000 7f27f1000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27ec098a40 0x7f27ec0ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.883+0000 7f27f1000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f27ec15cb60 0x7f27ec0b4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.883+0000 7f27f1000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec154020 con 0x7f27ec098a40 2024-03-23T03:47:48.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.883+0000 7f27f1000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec0aee70 con 0x7f27ec06afb0 2024-03-23T03:47:48.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.883+0000 7f27f1000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec153a60 con 0x7f27ec15cb60 2024-03-23T03:47:48.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.883+0000 7f27e1400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27ec098a40 0x7f27ec0ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.883+0000 7f27e1400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27ec098a40 0x7f27ec0ae930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33368/0 (socket says 172.21.15.102:33368) 2024-03-23T03:47:48.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.883+0000 7f27e1400640 1 -- 172.21.15.102:0/1486987820 learned_addr learned my addr 172.21.15.102:0/1486987820 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:48.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.883+0000 7f27e1400640 1 -- 172.21.15.102:0/1486987820 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f27ec15cb60 msgr2=0x7f27ec0b4690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:48.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.883+0000 7f27e1400640 1 --2- 172.21.15.102:0/1486987820 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f27ec15cb60 0x7f27ec0b4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.883+0000 7f27e1400640 1 -- 172.21.15.102:0/1486987820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27ec06afb0 msgr2=0x7f27ec0af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:48.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.883+0000 7f27e1400640 1 --2- 172.21.15.102:0/1486987820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27ec06afb0 0x7f27ec0af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:48.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.883+0000 7f27e1400640 1 -- 172.21.15.102:0/1486987820 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d4009400 con 0x7f27ec098a40 2024-03-23T03:47:48.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.884+0000 7f27e1400640 1 --2- 172.21.15.102:0/1486987820 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27ec098a40 0x7f27ec0ae930 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f27d4034040 tx=0x7f27d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:48.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.884+0000 7f27e2a00640 1 -- 172.21.15.102:0/1486987820 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d403aa60 con 0x7f27ec098a40 2024-03-23T03:47:48.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.884+0000 7f27e2a00640 1 -- 172.21.15.102:0/1486987820 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d403abe0 con 0x7f27ec098a40 2024-03-23T03:47:48.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.884+0000 7f27e2a00640 1 -- 172.21.15.102:0/1486987820 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d4044400 con 0x7f27ec098a40 2024-03-23T03:47:48.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.884+0000 7f27f1000640 1 -- 172.21.15.102:0/1486987820 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ec153d40 con 0x7f27ec098a40 2024-03-23T03:47:48.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.884+0000 7f27f1000640 1 -- 172.21.15.102:0/1486987820 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27ec0c4610 con 0x7f27ec098a40 2024-03-23T03:47:48.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.885+0000 7f27f1000640 1 -- 172.21.15.102:0/1486987820 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27ec0c4190 con 0x7f27ec098a40 2024-03-23T03:47:48.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.886+0000 7f27e2a00640 1 -- 172.21.15.102:0/1486987820 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f27d404d440 con 0x7f27ec098a40 2024-03-23T03:47:48.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.887+0000 7f27e2a00640 1 --2- 172.21.15.102:0/1486987820 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f27c806ac00 0x7f27c806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.887+0000 7f27e2a00640 1 -- 172.21.15.102:0/1486987820 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f27d40aba10 con 0x7f27ec098a40 2024-03-23T03:47:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.887+0000 7f27e9e00640 1 --2- 172.21.15.102:0/1486987820 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f27c806ac00 0x7f27c806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:48.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.888+0000 7f27e9e00640 1 --2- 172.21.15.102:0/1486987820 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f27c806ac00 0x7f27c806d0c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f27dc0015f0 tx=0x7f27dc007a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:48.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:48.891+0000 7f27e2a00640 1 -- 172.21.15.102:0/1486987820 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f27ec0c4190 con 0x7f27ec098a40 2024-03-23T03:47:49.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.084+0000 7f27f1000640 1 -- 172.21.15.102:0/1486987820 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f27ec0c4190 con 0x7f27ec098a40 2024-03-23T03:47:49.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.085+0000 7f27e2a00640 1 -- 172.21.15.102:0/1486987820 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f27ec0c4190 con 0x7f27ec098a40 2024-03-23T03:47:49.085 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.087+0000 7f27f1000640 1 -- 172.21.15.102:0/1486987820 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f27c806ac00 msgr2=0x7f27c806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.087+0000 7f27f1000640 1 --2- 172.21.15.102:0/1486987820 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f27c806ac00 0x7f27c806d0c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f27dc0015f0 tx=0x7f27dc007a40 comp rx=0 tx=0).stop 2024-03-23T03:47:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.087+0000 7f27f1000640 1 -- 172.21.15.102:0/1486987820 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27ec098a40 msgr2=0x7f27ec0ae930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.087+0000 7f27f1000640 1 --2- 172.21.15.102:0/1486987820 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27ec098a40 0x7f27ec0ae930 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f27d4034040 tx=0x7f27d40046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.088+0000 7f27f1000640 1 -- 172.21.15.102:0/1486987820 shutdown_connections 2024-03-23T03:47:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.088+0000 7f27f1000640 1 --2- 172.21.15.102:0/1486987820 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f27c806ac00 0x7f27c806d0c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.088+0000 7f27f1000640 1 --2- 172.21.15.102:0/1486987820 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f27ec15cb60 0x7f27ec0b4690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.088+0000 7f27f1000640 1 --2- 172.21.15.102:0/1486987820 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27ec098a40 0x7f27ec0ae930 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.088+0000 7f27f1000640 1 --2- 172.21.15.102:0/1486987820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27ec06afb0 0x7f27ec0af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.088+0000 7f27f1000640 1 -- 172.21.15.102:0/1486987820 >> 172.21.15.102:0/1486987820 conn(0x7f27ec0b2180 msgr2=0x7f27ec06fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.088+0000 7f27f1000640 1 -- 172.21.15.102:0/1486987820 shutdown_connections 2024-03-23T03:47:49.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.088+0000 7f27f1000640 1 -- 172.21.15.102:0/1486987820 wait complete. 2024-03-23T03:47:49.106 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:49.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.311+0000 7f25b1600640 1 Processor -- start 2024-03-23T03:47:49.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.311+0000 7f25b1600640 1 -- start start 2024-03-23T03:47:49.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.312+0000 7f25b1600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f25ac0adbc0 0x7f25ac0b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:49.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.312+0000 7f25b1600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25ac06d390 0x7f25ac06d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:49.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.312+0000 7f25b1600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25ac0b16a0 con 0x7f25ac0cbab0 2024-03-23T03:47:49.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.312+0000 7f25b1600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25ac06dd40 con 0x7f25ac0adbc0 2024-03-23T03:47:49.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.312+0000 7f25b1600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac0b4700 con 0x7f25ac06d390 2024-03-23T03:47:49.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.313+0000 7f25ab400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25ac06d390 0x7f25ac06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:49.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.313+0000 7f25aaa00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f25ac0adbc0 0x7f25ac0b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:49.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.313+0000 7f25a2000640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f25ac0cbab0 0x7f25ac0b20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:35044/0 (socket says 172.21.15.102:35044) 2024-03-23T03:47:49.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.314+0000 7f25a2000640 1 -- 172.21.15.102:0/2469596082 learned_addr learned my addr 172.21.15.102:0/2469596082 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:49.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.314+0000 7f25aaa00640 1 -- 172.21.15.102:0/2469596082 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25ac06d390 msgr2=0x7f25ac06d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:49.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.314+0000 7f25aaa00640 1 --2- 172.21.15.102:0/2469596082 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25ac06d390 0x7f25ac06d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.314+0000 7f25aaa00640 1 -- 172.21.15.102:0/2469596082 >> v1:172.21.15.173:6789/0 conn(0x7f25ac0cbab0 legacy=0x7f25ac0b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:49.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.314+0000 7f25aaa00640 1 -- 172.21.15.102:0/2469596082 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25ac0cbfb0 con 0x7f25ac0adbc0 2024-03-23T03:47:49.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.314+0000 7f25ab400640 1 --2- 172.21.15.102:0/2469596082 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25ac06d390 0x7f25ac06d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:47:49.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.316+0000 7f25aaa00640 1 --2- 172.21.15.102:0/2469596082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25ac0adbc0 0x7f25ac0b41c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f259c0097a0 tx=0x7f259c02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a77a1a5464c792e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:49.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.316+0000 7f25aa000640 1 -- 172.21.15.102:0/2469596082 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c031780 con 0x7f25ac0adbc0 2024-03-23T03:47:49.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.316+0000 7f25aa000640 1 -- 172.21.15.102:0/2469596082 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f259c0318e0 con 0x7f25ac0adbc0 2024-03-23T03:47:49.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.316+0000 7f25aa000640 1 -- 172.21.15.102:0/2469596082 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f259c031c70 con 0x7f25ac0adbc0 2024-03-23T03:47:49.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.317+0000 7f25b1600640 1 -- 172.21.15.102:0/2469596082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25ac0adbc0 msgr2=0x7f25ac0b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:49.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.317+0000 7f25b1600640 1 --2- 172.21.15.102:0/2469596082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25ac0adbc0 0x7f25ac0b41c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f259c0097a0 tx=0x7f259c02b820 comp rx=0 tx=0).stop 2024-03-23T03:47:49.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.317+0000 7f25b1600640 1 -- 172.21.15.102:0/2469596082 shutdown_connections 2024-03-23T03:47:49.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.317+0000 7f25b1600640 1 --2- 172.21.15.102:0/2469596082 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25ac06d390 0x7f25ac06d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.317+0000 7f25b1600640 1 --2- 172.21.15.102:0/2469596082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25ac0adbc0 0x7f25ac0b41c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.317+0000 7f25b1600640 1 -- 172.21.15.102:0/2469596082 >> 172.21.15.102:0/2469596082 conn(0x7f25ac06a3d0 msgr2=0x7f25ac0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:49.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.317+0000 7f25b1600640 1 -- 172.21.15.102:0/2469596082 shutdown_connections 2024-03-23T03:47:49.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.317+0000 7f25b1600640 1 -- 172.21.15.102:0/2469596082 wait complete. 2024-03-23T03:47:49.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.319+0000 7f25b1600640 1 Processor -- start 2024-03-23T03:47:49.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.319+0000 7f25b1600640 1 -- start start 2024-03-23T03:47:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.321+0000 7f25b1600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25ac06d390 0x7f25ac158d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.321+0000 7f25b1600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ac0adbc0 0x7f25ac0b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.321+0000 7f25b1600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25ac0cbab0 0x7f25ac0b39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.321+0000 7f25b1600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac13e7d0 con 0x7f25ac0adbc0 2024-03-23T03:47:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.321+0000 7f25b1600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac13afa0 con 0x7f25ac0cbab0 2024-03-23T03:47:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.321+0000 7f25b1600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25ac13b280 con 0x7f25ac06d390 2024-03-23T03:47:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.321+0000 7f25a2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ac0adbc0 0x7f25ac0b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.321+0000 7f25a2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ac0adbc0 0x7f25ac0b3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33374/0 (socket says 172.21.15.102:33374) 2024-03-23T03:47:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.321+0000 7f25a2000640 1 -- 172.21.15.102:0/1498019669 learned_addr learned my addr 172.21.15.102:0/1498019669 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.321+0000 7f25aaa00640 1 --2- 172.21.15.102:0/1498019669 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25ac06d390 0x7f25ac158d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.321+0000 7f25ab400640 1 --2- 172.21.15.102:0/1498019669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25ac0cbab0 0x7f25ac0b39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.321+0000 7f25a2000640 1 -- 172.21.15.102:0/1498019669 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25ac06d390 msgr2=0x7f25ac158d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:49.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.321+0000 7f25a2000640 1 --2- 172.21.15.102:0/1498019669 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25ac06d390 0x7f25ac158d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.321+0000 7f25a2000640 1 -- 172.21.15.102:0/1498019669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25ac0cbab0 msgr2=0x7f25ac0b39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.321+0000 7f25a2000640 1 --2- 172.21.15.102:0/1498019669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25ac0cbab0 0x7f25ac0b39a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.322+0000 7f25a2000640 1 -- 172.21.15.102:0/1498019669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a4009a70 con 0x7f25ac0adbc0 2024-03-23T03:47:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.322+0000 7f25a2000640 1 --2- 172.21.15.102:0/1498019669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ac0adbc0 0x7f25ac0b3460 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f259400c500 tx=0x7f259400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.322+0000 7f25a3400640 1 -- 172.21.15.102:0/1498019669 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2594005ce0 con 0x7f25ac0adbc0 2024-03-23T03:47:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.322+0000 7f25a3400640 1 -- 172.21.15.102:0/1498019669 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2594005e40 con 0x7f25ac0adbc0 2024-03-23T03:47:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.322+0000 7f25a3400640 1 -- 172.21.15.102:0/1498019669 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25940072f0 con 0x7f25ac0adbc0 2024-03-23T03:47:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.322+0000 7f25b1600640 1 -- 172.21.15.102:0/1498019669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259c009400 con 0x7f25ac0adbc0 2024-03-23T03:47:49.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.322+0000 7f25b1600640 1 -- 172.21.15.102:0/1498019669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25ac13b840 con 0x7f25ac0adbc0 2024-03-23T03:47:49.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.323+0000 7f25b1600640 1 -- 172.21.15.102:0/1498019669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25ac06ac20 con 0x7f25ac0adbc0 2024-03-23T03:47:49.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.324+0000 7f25a3400640 1 -- 172.21.15.102:0/1498019669 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2594007510 con 0x7f25ac0adbc0 2024-03-23T03:47:49.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.326+0000 7f25a3400640 1 --2- 172.21.15.102:0/1498019669 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f258806ac00 0x7f258806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:49.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.326+0000 7f25a3400640 1 -- 172.21.15.102:0/1498019669 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2594088b20 con 0x7f25ac0adbc0 2024-03-23T03:47:49.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.326+0000 7f25aaa00640 1 --2- 172.21.15.102:0/1498019669 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f258806ac00 0x7f258806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:49.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.327+0000 7f25aaa00640 1 --2- 172.21.15.102:0/1498019669 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f258806ac00 0x7f258806d0c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f259c036000 tx=0x7f259c03d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:49.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.331+0000 7f25a3400640 1 -- 172.21.15.102:0/1498019669 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2594053a80 con 0x7f25ac0adbc0 2024-03-23T03:47:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.529+0000 7f25b1600640 1 -- 172.21.15.102:0/1498019669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25ac13ba30 con 0x7f25ac0adbc0 2024-03-23T03:47:49.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.532+0000 7f25a3400640 1 -- 172.21.15.102:0/1498019669 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18260 (secure 0 0 0) 0x7f2594017020 con 0x7f25ac0adbc0 2024-03-23T03:47:49.533 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:49.534 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":28,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:47:41.647786+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:47:49.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.535+0000 7f25b1600640 1 -- 172.21.15.102:0/1498019669 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f258806ac00 msgr2=0x7f258806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:49.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.535+0000 7f25b1600640 1 --2- 172.21.15.102:0/1498019669 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f258806ac00 0x7f258806d0c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f259c036000 tx=0x7f259c03d040 comp rx=0 tx=0).stop 2024-03-23T03:47:49.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.535+0000 7f25b1600640 1 -- 172.21.15.102:0/1498019669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ac0adbc0 msgr2=0x7f25ac0b3460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:49.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.535+0000 7f25b1600640 1 --2- 172.21.15.102:0/1498019669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ac0adbc0 0x7f25ac0b3460 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f259400c500 tx=0x7f259400c9c0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.536+0000 7f25b1600640 1 -- 172.21.15.102:0/1498019669 shutdown_connections 2024-03-23T03:47:49.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.536+0000 7f25b1600640 1 --2- 172.21.15.102:0/1498019669 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f258806ac00 0x7f258806d0c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.536+0000 7f25b1600640 1 --2- 172.21.15.102:0/1498019669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25ac0cbab0 0x7f25ac0b39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.536+0000 7f25b1600640 1 --2- 172.21.15.102:0/1498019669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25ac0adbc0 0x7f25ac0b3460 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.536+0000 7f25b1600640 1 --2- 172.21.15.102:0/1498019669 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25ac06d390 0x7f25ac158d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.536+0000 7f25b1600640 1 -- 172.21.15.102:0/1498019669 >> 172.21.15.102:0/1498019669 conn(0x7f25ac06a3d0 msgr2=0x7f25ac0b02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:49.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.536+0000 7f25b1600640 1 -- 172.21.15.102:0/1498019669 shutdown_connections 2024-03-23T03:47:49.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.536+0000 7f25b1600640 1 -- 172.21.15.102:0/1498019669 wait complete. 2024-03-23T03:47:49.538 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:47:49.544 DEBUG:teuthology.orchestra.run.smithi173:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:47:49.555 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:46:52.158650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:46:52.914275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-23T03:46:55.983754+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-23T03:47:49.555 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:47:49.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.794+0000 7f887ca00640 1 Processor -- start 2024-03-23T03:47:49.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.794+0000 7f887ca00640 1 -- start start 2024-03-23T03:47:49.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.795+0000 7f887ca00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f88780ce930 0x7f88780add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:49.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.796+0000 7f887ca00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f88780cd3e0 0x7f88780ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:49.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.796+0000 7f887ca00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88780cf250 con 0x7f887806ce90 2024-03-23T03:47:49.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.796+0000 7f887ca00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f887806c760 con 0x7f88780ce930 2024-03-23T03:47:49.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.796+0000 7f887ca00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88780b42d0 con 0x7f88780cd3e0 2024-03-23T03:47:49.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.796+0000 7f8875e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f88780ce930 0x7f88780add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:49.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.796+0000 7f8876800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f887806ce90 0x7f887815e290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47202/0 (socket says 172.21.15.102:47202) 2024-03-23T03:47:49.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.796+0000 7f8876800640 1 -- 172.21.15.102:0/680190380 learned_addr learned my addr 172.21.15.102:0/680190380 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:49.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.796+0000 7f886fe00640 1 --2- 172.21.15.102:0/680190380 >> v2:172.21.15.173:3300/0 conn(0x7f88780cd3e0 0x7f88780ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:49.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.797+0000 7f8875e00640 1 -- 172.21.15.102:0/680190380 >> v2:172.21.15.173:3300/0 conn(0x7f88780cd3e0 msgr2=0x7f88780ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:49.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.797+0000 7f8875e00640 1 --2- 172.21.15.102:0/680190380 >> v2:172.21.15.173:3300/0 conn(0x7f88780cd3e0 0x7f88780ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.797+0000 7f8875e00640 1 -- 172.21.15.102:0/680190380 >> v1:172.21.15.102:6789/0 conn(0x7f887806ce90 legacy=0x7f887815e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:49.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.797+0000 7f8875e00640 1 -- 172.21.15.102:0/680190380 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88780cdba0 con 0x7f88780ce930 2024-03-23T03:47:49.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.797+0000 7f8875400640 1 -- 172.21.15.102:0/680190380 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712884104 0 0) 0x7f88780cf250 con 0x7f887806ce90 2024-03-23T03:47:49.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.798+0000 7f8875e00640 1 --2- 172.21.15.102:0/680190380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88780ce930 0x7f88780add30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f88780cf4f0 tx=0x7f886802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f42939b5ac801a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:49.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.798+0000 7f8875400640 1 -- 172.21.15.102:0/680190380 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868031830 con 0x7f88780ce930 2024-03-23T03:47:49.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.798+0000 7f8875400640 1 -- 172.21.15.102:0/680190380 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8868031990 con 0x7f88780ce930 2024-03-23T03:47:49.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.798+0000 7f8875400640 1 -- 172.21.15.102:0/680190380 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8868031d20 con 0x7f88780ce930 2024-03-23T03:47:49.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.799+0000 7f887ca00640 1 -- 172.21.15.102:0/680190380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88780ce930 msgr2=0x7f88780add30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:49.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.799+0000 7f887ca00640 1 --2- 172.21.15.102:0/680190380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88780ce930 0x7f88780add30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f88780cf4f0 tx=0x7f886802b8d0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.799+0000 7f887ca00640 1 -- 172.21.15.102:0/680190380 shutdown_connections 2024-03-23T03:47:49.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.799+0000 7f887ca00640 1 --2- 172.21.15.102:0/680190380 >> v2:172.21.15.173:3300/0 conn(0x7f88780cd3e0 0x7f88780ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.799+0000 7f887ca00640 1 --2- 172.21.15.102:0/680190380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88780ce930 0x7f88780add30 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.799+0000 7f887ca00640 1 -- 172.21.15.102:0/680190380 >> 172.21.15.102:0/680190380 conn(0x7f887806a0b0 msgr2=0x7f887806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:49.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.799+0000 7f887ca00640 1 -- 172.21.15.102:0/680190380 shutdown_connections 2024-03-23T03:47:49.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.799+0000 7f887ca00640 1 -- 172.21.15.102:0/680190380 wait complete. 2024-03-23T03:47:49.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.801+0000 7f887ca00640 1 Processor -- start 2024-03-23T03:47:49.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.801+0000 7f887ca00640 1 -- start start 2024-03-23T03:47:49.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.802+0000 7f887ca00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f887806ce90 0x7f887813d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:49.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.802+0000 7f887ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88780cd3e0 0x7f8878158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:49.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.802+0000 7f887ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88780ce930 0x7f88781593d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:49.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.802+0000 7f887ca00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887813fa40 con 0x7f88780cd3e0 2024-03-23T03:47:49.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.802+0000 7f887ca00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f887813d930 con 0x7f88780ce930 2024-03-23T03:47:49.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.802+0000 7f887ca00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f887813f480 con 0x7f887806ce90 2024-03-23T03:47:49.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.802+0000 7f886fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88780cd3e0 0x7f8878158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:49.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.802+0000 7f886fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88780cd3e0 0x7f8878158e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33392/0 (socket says 172.21.15.102:33392) 2024-03-23T03:47:49.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.802+0000 7f886fe00640 1 -- 172.21.15.102:0/1551882164 learned_addr learned my addr 172.21.15.102:0/1551882164 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:49.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.803+0000 7f8875e00640 1 --2- 172.21.15.102:0/1551882164 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f887806ce90 0x7f887813d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:49.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.803+0000 7f886fe00640 1 -- 172.21.15.102:0/1551882164 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f887806ce90 msgr2=0x7f887813d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:49.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.803+0000 7f886fe00640 1 --2- 172.21.15.102:0/1551882164 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f887806ce90 0x7f887813d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.803+0000 7f886fe00640 1 -- 172.21.15.102:0/1551882164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88780ce930 msgr2=0x7f88781593d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:49.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.803+0000 7f886fe00640 1 --2- 172.21.15.102:0/1551882164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88780ce930 0x7f88781593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:49.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.803+0000 7f886fe00640 1 -- 172.21.15.102:0/1551882164 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8868009400 con 0x7f88780cd3e0 2024-03-23T03:47:49.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.803+0000 7f886fe00640 1 --2- 172.21.15.102:0/1551882164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88780cd3e0 0x7f8878158e90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f886000e9e0 tx=0x7f886000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:49.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.803+0000 7f886e000640 1 -- 172.21.15.102:0/1551882164 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8860009670 con 0x7f88780cd3e0 2024-03-23T03:47:49.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.803+0000 7f886e000640 1 -- 172.21.15.102:0/1551882164 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f886000c9d0 con 0x7f88780cd3e0 2024-03-23T03:47:49.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.803+0000 7f886e000640 1 -- 172.21.15.102:0/1551882164 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f886000cd80 con 0x7f88780cd3e0 2024-03-23T03:47:49.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.803+0000 7f887ca00640 1 -- 172.21.15.102:0/1551882164 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f887813f760 con 0x7f88780cd3e0 2024-03-23T03:47:49.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.803+0000 7f887ca00640 1 -- 172.21.15.102:0/1551882164 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f887815b8b0 con 0x7f88780cd3e0 2024-03-23T03:47:49.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.804+0000 7f887ca00640 1 -- 172.21.15.102:0/1551882164 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f887815b430 con 0x7f88780cd3e0 2024-03-23T03:47:49.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.805+0000 7f886e000640 1 -- 172.21.15.102:0/1551882164 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8860013470 con 0x7f88780cd3e0 2024-03-23T03:47:49.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.806+0000 7f886e000640 1 --2- 172.21.15.102:0/1551882164 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f885406ac00 0x7f885406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:49.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.806+0000 7f886e000640 1 -- 172.21.15.102:0/1551882164 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8860089a50 con 0x7f88780cd3e0 2024-03-23T03:47:49.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.807+0000 7f8875e00640 1 --2- 172.21.15.102:0/1551882164 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f885406ac00 0x7f885406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:49.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.807+0000 7f8875e00640 1 --2- 172.21.15.102:0/1551882164 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f885406ac00 0x7f885406d0c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8868036000 tx=0x7f886803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:49.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:49.811+0000 7f886e000640 1 -- 172.21.15.102:0/1551882164 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f887815b430 con 0x7f88780cd3e0 2024-03-23T03:47:50.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.010+0000 7f887ca00640 1 -- 172.21.15.102:0/1551882164 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f887815b430 con 0x7f88780cd3e0 2024-03-23T03:47:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.011+0000 7f886e000640 1 -- 172.21.15.102:0/1551882164 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f887815b430 con 0x7f88780cd3e0 2024-03-23T03:47:50.012 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:47:50.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.014+0000 7f887ca00640 1 -- 172.21.15.102:0/1551882164 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f885406ac00 msgr2=0x7f885406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.014+0000 7f887ca00640 1 --2- 172.21.15.102:0/1551882164 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f885406ac00 0x7f885406d0c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8868036000 tx=0x7f886803c040 comp rx=0 tx=0).stop 2024-03-23T03:47:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.014+0000 7f887ca00640 1 -- 172.21.15.102:0/1551882164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88780cd3e0 msgr2=0x7f8878158e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.014+0000 7f887ca00640 1 --2- 172.21.15.102:0/1551882164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88780cd3e0 0x7f8878158e90 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f886000e9e0 tx=0x7f886000eea0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.014+0000 7f887ca00640 1 -- 172.21.15.102:0/1551882164 shutdown_connections 2024-03-23T03:47:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.014+0000 7f887ca00640 1 --2- 172.21.15.102:0/1551882164 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f885406ac00 0x7f885406d0c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.014+0000 7f887ca00640 1 --2- 172.21.15.102:0/1551882164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f88780ce930 0x7f88781593d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.014+0000 7f887ca00640 1 --2- 172.21.15.102:0/1551882164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f88780cd3e0 0x7f8878158e90 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.014+0000 7f887ca00640 1 --2- 172.21.15.102:0/1551882164 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f887806ce90 0x7f887813d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.014+0000 7f887ca00640 1 -- 172.21.15.102:0/1551882164 >> 172.21.15.102:0/1551882164 conn(0x7f887806a0b0 msgr2=0x7f887806a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.014+0000 7f887ca00640 1 -- 172.21.15.102:0/1551882164 shutdown_connections 2024-03-23T03:47:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.014+0000 7f887ca00640 1 -- 172.21.15.102:0/1551882164 wait complete. 2024-03-23T03:47:50.032 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:47:50.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.236+0000 7f7887200640 1 Processor -- start 2024-03-23T03:47:50.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.236+0000 7f7887200640 1 -- start start 2024-03-23T03:47:50.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.237+0000 7f7887200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7880159830 0x7f78800cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:50.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.237+0000 7f7887200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7880159c10 con 0x7f78800ce930 2024-03-23T03:47:50.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.237+0000 7f7887200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78800cd840 con 0x7f78800ca7b0 2024-03-23T03:47:50.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.237+0000 7f7887200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788006de50 con 0x7f7880159830 2024-03-23T03:47:50.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.238+0000 7f787fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7880159830 0x7f78800cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:50.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.238+0000 7f7885200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f78800ce930 0x7f78800b18d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47208/0 (socket says 172.21.15.102:47208) 2024-03-23T03:47:50.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.238+0000 7f7885200640 1 -- 172.21.15.102:0/1094495553 learned_addr learned my addr 172.21.15.102:0/1094495553 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:50.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.238+0000 7f787fe00640 1 -- 172.21.15.102:0/1094495553 >> v1:172.21.15.173:6789/0 conn(0x7f78800ca7b0 legacy=0x7f78800cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:50.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.238+0000 7f787fe00640 1 -- 172.21.15.102:0/1094495553 >> v1:172.21.15.102:6789/0 conn(0x7f78800ce930 legacy=0x7f78800b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:50.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.238+0000 7f787fe00640 1 -- 172.21.15.102:0/1094495553 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78800ccc30 con 0x7f7880159830 2024-03-23T03:47:50.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.239+0000 7f787fe00640 1 --2- 172.21.15.102:0/1094495553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7880159830 0x7f78800cc3c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f78700097d0 tx=0x7f787002db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe769bd8400cad3c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:50.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.240+0000 7f787f400640 1 -- 172.21.15.102:0/1094495553 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787002e920 con 0x7f7880159830 2024-03-23T03:47:50.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.240+0000 7f787f400640 1 -- 172.21.15.102:0/1094495553 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787002ea80 con 0x7f7880159830 2024-03-23T03:47:50.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.240+0000 7f787f400640 1 -- 172.21.15.102:0/1094495553 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787002ee10 con 0x7f7880159830 2024-03-23T03:47:50.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.240+0000 7f7887200640 1 -- 172.21.15.102:0/1094495553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7880159830 msgr2=0x7f78800cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:50.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.240+0000 7f7887200640 1 --2- 172.21.15.102:0/1094495553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7880159830 0x7f78800cc3c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f78700097d0 tx=0x7f787002db70 comp rx=0 tx=0).stop 2024-03-23T03:47:50.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.240+0000 7f7887200640 1 -- 172.21.15.102:0/1094495553 shutdown_connections 2024-03-23T03:47:50.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.240+0000 7f7887200640 1 --2- 172.21.15.102:0/1094495553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7880159830 0x7f78800cc3c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.240+0000 7f7887200640 1 -- 172.21.15.102:0/1094495553 >> 172.21.15.102:0/1094495553 conn(0x7f788006a0b0 msgr2=0x7f788006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:50.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.240+0000 7f7887200640 1 -- 172.21.15.102:0/1094495553 shutdown_connections 2024-03-23T03:47:50.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.240+0000 7f7887200640 1 -- 172.21.15.102:0/1094495553 wait complete. 2024-03-23T03:47:50.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.242+0000 7f7887200640 1 Processor -- start 2024-03-23T03:47:50.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.242+0000 7f7887200640 1 -- start start 2024-03-23T03:47:50.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.243+0000 7f7887200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78800ca7b0 0x7f78801451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:50.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.243+0000 7f7887200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78800ce930 0x7f78800af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:50.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.243+0000 7f7887200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7880159830 0x7f78800ad1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:50.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.243+0000 7f7887200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78800af9f0 con 0x7f78800ce930 2024-03-23T03:47:50.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.243+0000 7f7887200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78800ac140 con 0x7f7880159830 2024-03-23T03:47:50.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.243+0000 7f7887200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78800ac420 con 0x7f78800ca7b0 2024-03-23T03:47:50.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.243+0000 7f7877e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78800ce930 0x7f78800af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:50.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.243+0000 7f7877e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78800ce930 0x7f78800af4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33406/0 (socket says 172.21.15.102:33406) 2024-03-23T03:47:50.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.243+0000 7f7877e00640 1 -- 172.21.15.102:0/354652987 learned_addr learned my addr 172.21.15.102:0/354652987 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:50.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.244+0000 7f7877e00640 1 -- 172.21.15.102:0/354652987 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78800ca7b0 msgr2=0x7f78801451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:50.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.244+0000 7f7885200640 1 --2- 172.21.15.102:0/354652987 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7880159830 0x7f78800ad1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:50.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.244+0000 7f7877e00640 1 --2- 172.21.15.102:0/354652987 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78800ca7b0 0x7f78801451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.244+0000 7f7877e00640 1 -- 172.21.15.102:0/354652987 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7880159830 msgr2=0x7f78800ad1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:50.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.244+0000 7f7877e00640 1 --2- 172.21.15.102:0/354652987 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7880159830 0x7f78800ad1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.244+0000 7f7877e00640 1 -- 172.21.15.102:0/354652987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7870009400 con 0x7f78800ce930 2024-03-23T03:47:50.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.244+0000 7f7885200640 1 --2- 172.21.15.102:0/354652987 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7880159830 0x7f78800ad1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:50.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.244+0000 7f7877e00640 1 --2- 172.21.15.102:0/354652987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78800ce930 0x7f78800af4b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f786800c430 tx=0x7f786800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:50.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.244+0000 7f787cc00640 1 -- 172.21.15.102:0/354652987 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7868011790 con 0x7f78800ce930 2024-03-23T03:47:50.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.244+0000 7f787cc00640 1 -- 172.21.15.102:0/354652987 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78680118f0 con 0x7f78800ce930 2024-03-23T03:47:50.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.244+0000 7f787cc00640 1 -- 172.21.15.102:0/354652987 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7868011ce0 con 0x7f78800ce930 2024-03-23T03:47:50.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.244+0000 7f7887200640 1 -- 172.21.15.102:0/354652987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78800ac700 con 0x7f78800ce930 2024-03-23T03:47:50.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.245+0000 7f7887200640 1 -- 172.21.15.102:0/354652987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78800accc0 con 0x7f78800ce930 2024-03-23T03:47:50.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.246+0000 7f7887200640 1 -- 172.21.15.102:0/354652987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7880073390 con 0x7f78800ce930 2024-03-23T03:47:50.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.247+0000 7f787cc00640 1 -- 172.21.15.102:0/354652987 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f786801c460 con 0x7f78800ce930 2024-03-23T03:47:50.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.248+0000 7f787cc00640 1 --2- 172.21.15.102:0/354652987 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f785c06ac00 0x7f785c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:50.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.248+0000 7f787cc00640 1 -- 172.21.15.102:0/354652987 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7868089640 con 0x7f78800ce930 2024-03-23T03:47:50.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.248+0000 7f787fe00640 1 --2- 172.21.15.102:0/354652987 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f785c06ac00 0x7f785c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:50.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.249+0000 7f787fe00640 1 --2- 172.21.15.102:0/354652987 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f785c06ac00 0x7f785c06d0c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7870038000 tx=0x7f7870030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:50.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.253+0000 7f787cc00640 1 -- 172.21.15.102:0/354652987 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f78680544f0 con 0x7f78800ce930 2024-03-23T03:47:50.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.446+0000 7f7887200640 1 -- 172.21.15.102:0/354652987 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f78800706f0 con 0x7f78800ce930 2024-03-23T03:47:50.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.447+0000 7f787cc00640 1 -- 172.21.15.102:0/354652987 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7868057b10 con 0x7f78800ce930 2024-03-23T03:47:50.448 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:50.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.450+0000 7f7887200640 1 -- 172.21.15.102:0/354652987 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f785c06ac00 msgr2=0x7f785c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:50.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.450+0000 7f7887200640 1 --2- 172.21.15.102:0/354652987 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f785c06ac00 0x7f785c06d0c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7870038000 tx=0x7f7870030000 comp rx=0 tx=0).stop 2024-03-23T03:47:50.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.450+0000 7f7887200640 1 -- 172.21.15.102:0/354652987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78800ce930 msgr2=0x7f78800af4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:50.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.450+0000 7f7887200640 1 --2- 172.21.15.102:0/354652987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78800ce930 0x7f78800af4b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f786800c430 tx=0x7f786800c8f0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.450+0000 7f7887200640 1 -- 172.21.15.102:0/354652987 shutdown_connections 2024-03-23T03:47:50.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.450+0000 7f7887200640 1 --2- 172.21.15.102:0/354652987 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f785c06ac00 0x7f785c06d0c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.450+0000 7f7887200640 1 --2- 172.21.15.102:0/354652987 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7880159830 0x7f78800ad1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.450+0000 7f7887200640 1 --2- 172.21.15.102:0/354652987 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78800ce930 0x7f78800af4b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.450+0000 7f7887200640 1 --2- 172.21.15.102:0/354652987 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78800ca7b0 0x7f78801451c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.450+0000 7f7887200640 1 -- 172.21.15.102:0/354652987 >> 172.21.15.102:0/354652987 conn(0x7f788006a0b0 msgr2=0x7f78800b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:50.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.450+0000 7f7887200640 1 -- 172.21.15.102:0/354652987 shutdown_connections 2024-03-23T03:47:50.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.451+0000 7f7887200640 1 -- 172.21.15.102:0/354652987 wait complete. 2024-03-23T03:47:50.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:47:50.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.671+0000 7fd28a800640 1 Processor -- start 2024-03-23T03:47:50.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.671+0000 7fd28a800640 1 -- start start 2024-03-23T03:47:50.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.672+0000 7fd28a800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd284143460 0x7fd28415aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:50.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.672+0000 7fd28a800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd284143930 0x7fd2840cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:50.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.673+0000 7fd28a800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2840cc6b0 con 0x7fd2840cdc50 2024-03-23T03:47:50.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.673+0000 7fd28a800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2840cc950 con 0x7fd284143460 2024-03-23T03:47:50.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.673+0000 7fd28a800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2840ccc30 con 0x7fd284143930 2024-03-23T03:47:50.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.673+0000 7fd288a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd284143930 0x7fd2840cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:50.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.673+0000 7fd289400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd284143460 0x7fd28415aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:50.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.673+0000 7fd288a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd284143930 0x7fd2840cd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:49998/0 (socket says 172.21.15.102:49998) 2024-03-23T03:47:50.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.673+0000 7fd289e00640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fd2840cdc50 0x7fd28415e1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:35048/0 (socket says 172.21.15.102:35048) 2024-03-23T03:47:50.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.673+0000 7fd288a00640 1 -- 172.21.15.102:0/2273006176 learned_addr learned my addr 172.21.15.102:0/2273006176 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:50.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.674+0000 7fd288a00640 1 -- 172.21.15.102:0/2273006176 >> v2:172.21.15.173:3300/0 conn(0x7fd284143460 msgr2=0x7fd28415aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:50.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.674+0000 7fd288a00640 1 --2- 172.21.15.102:0/2273006176 >> v2:172.21.15.173:3300/0 conn(0x7fd284143460 0x7fd28415aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.674+0000 7fd288a00640 1 -- 172.21.15.102:0/2273006176 >> v1:172.21.15.173:6789/0 conn(0x7fd2840cdc50 legacy=0x7fd28415e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:50.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.674+0000 7fd288a00640 1 -- 172.21.15.102:0/2273006176 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2840ccf10 con 0x7fd284143930 2024-03-23T03:47:50.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.674+0000 7fd289400640 1 --2- 172.21.15.102:0/2273006176 >> v2:172.21.15.173:3300/0 conn(0x7fd284143460 0x7fd28415aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:47:50.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.676+0000 7fd288a00640 1 --2- 172.21.15.102:0/2273006176 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd284143930 0x7fd2840cd710 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd270009870 tx=0x7fd27002ba20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f74d133e6d245b5f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:50.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.676+0000 7fd283e00640 1 -- 172.21.15.102:0/2273006176 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27002f840 con 0x7fd284143930 2024-03-23T03:47:50.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.676+0000 7fd283e00640 1 -- 172.21.15.102:0/2273006176 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd27002f9a0 con 0x7fd284143930 2024-03-23T03:47:50.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.676+0000 7fd283e00640 1 -- 172.21.15.102:0/2273006176 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27002fd30 con 0x7fd284143930 2024-03-23T03:47:50.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.677+0000 7fd28a800640 1 -- 172.21.15.102:0/2273006176 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd284143930 msgr2=0x7fd2840cd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:50.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.677+0000 7fd28a800640 1 --2- 172.21.15.102:0/2273006176 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd284143930 0x7fd2840cd710 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd270009870 tx=0x7fd27002ba20 comp rx=0 tx=0).stop 2024-03-23T03:47:50.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.677+0000 7fd28a800640 1 -- 172.21.15.102:0/2273006176 shutdown_connections 2024-03-23T03:47:50.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.677+0000 7fd28a800640 1 --2- 172.21.15.102:0/2273006176 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd284143930 0x7fd2840cd710 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.677+0000 7fd28a800640 1 --2- 172.21.15.102:0/2273006176 >> v2:172.21.15.173:3300/0 conn(0x7fd284143460 0x7fd28415aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.677+0000 7fd28a800640 1 -- 172.21.15.102:0/2273006176 >> 172.21.15.102:0/2273006176 conn(0x7fd284065df0 msgr2=0x7fd284069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:50.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.677+0000 7fd28a800640 1 -- 172.21.15.102:0/2273006176 shutdown_connections 2024-03-23T03:47:50.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.677+0000 7fd28a800640 1 -- 172.21.15.102:0/2273006176 wait complete. 2024-03-23T03:47:50.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.679+0000 7fd28a800640 1 Processor -- start 2024-03-23T03:47:50.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.680+0000 7fd28a800640 1 -- start start 2024-03-23T03:47:50.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.681+0000 7fd28a800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2840cdc50 0x7fd284134610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:50.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.681+0000 7fd28a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd284143460 0x7fd284139450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:50.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.681+0000 7fd28a800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd284143930 0x7fd28413f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:50.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.681+0000 7fd28a800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2841f0c00 con 0x7fd284143460 2024-03-23T03:47:50.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.681+0000 7fd28a800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2841f0080 con 0x7fd284143930 2024-03-23T03:47:50.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.681+0000 7fd28a800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2841f0360 con 0x7fd2840cdc50 2024-03-23T03:47:50.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.681+0000 7fd288a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd284143460 0x7fd284139450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:50.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.681+0000 7fd288a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd284143460 0x7fd284139450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33422/0 (socket says 172.21.15.102:33422) 2024-03-23T03:47:50.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.681+0000 7fd288a00640 1 -- 172.21.15.102:0/3031661039 learned_addr learned my addr 172.21.15.102:0/3031661039 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:50.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.681+0000 7fd289400640 1 --2- 172.21.15.102:0/3031661039 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2840cdc50 0x7fd284134610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:50.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.682+0000 7fd289e00640 1 --2- 172.21.15.102:0/3031661039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd284143930 0x7fd28413f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:50.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.682+0000 7fd288a00640 1 -- 172.21.15.102:0/3031661039 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2840cdc50 msgr2=0x7fd284134610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:50.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.682+0000 7fd288a00640 1 --2- 172.21.15.102:0/3031661039 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2840cdc50 0x7fd284134610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.682+0000 7fd288a00640 1 -- 172.21.15.102:0/3031661039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd284143930 msgr2=0x7fd28413f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:50.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.682+0000 7fd288a00640 1 --2- 172.21.15.102:0/3031661039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd284143930 0x7fd28413f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.682+0000 7fd288a00640 1 -- 172.21.15.102:0/3031661039 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd278009400 con 0x7fd284143460 2024-03-23T03:47:50.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.682+0000 7fd289e00640 1 --2- 172.21.15.102:0/3031661039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd284143930 0x7fd28413f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:50.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.682+0000 7fd288a00640 1 --2- 172.21.15.102:0/3031661039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd284143460 0x7fd284139450 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd270000c00 tx=0x7fd2700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:50.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.682+0000 7fd281600640 1 -- 172.21.15.102:0/3031661039 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27003b040 con 0x7fd284143460 2024-03-23T03:47:50.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.682+0000 7fd281600640 1 -- 172.21.15.102:0/3031661039 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd270037c40 con 0x7fd284143460 2024-03-23T03:47:50.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.682+0000 7fd281600640 1 -- 172.21.15.102:0/3031661039 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd270042480 con 0x7fd284143460 2024-03-23T03:47:50.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.682+0000 7fd28a800640 1 -- 172.21.15.102:0/3031661039 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2700094d0 con 0x7fd284143460 2024-03-23T03:47:50.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.683+0000 7fd28a800640 1 -- 172.21.15.102:0/3031661039 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2841f0920 con 0x7fd284143460 2024-03-23T03:47:50.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.684+0000 7fd28a800640 1 -- 172.21.15.102:0/3031661039 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2841f0e40 con 0x7fd284143460 2024-03-23T03:47:50.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.685+0000 7fd281600640 1 -- 172.21.15.102:0/3031661039 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd2700425e0 con 0x7fd284143460 2024-03-23T03:47:50.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.686+0000 7fd281600640 1 --2- 172.21.15.102:0/3031661039 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd26406ac30 0x7fd26406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:50.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.686+0000 7fd281600640 1 -- 172.21.15.102:0/3031661039 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd2700ac840 con 0x7fd284143460 2024-03-23T03:47:50.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.686+0000 7fd289400640 1 --2- 172.21.15.102:0/3031661039 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd26406ac30 0x7fd26406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:50.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.687+0000 7fd289400640 1 --2- 172.21.15.102:0/3031661039 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd26406ac30 0x7fd26406d0f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd28415abc0 tx=0x7fd278008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:50.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.690+0000 7fd281600640 1 -- 172.21.15.102:0/3031661039 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd270077810 con 0x7fd284143460 2024-03-23T03:47:50.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.885+0000 7fd28a800640 1 -- 172.21.15.102:0/3031661039 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd284144050 con 0x7fd284143460 2024-03-23T03:47:50.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.886+0000 7fd281600640 1 -- 172.21.15.102:0/3031661039 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd27007ae30 con 0x7fd284143460 2024-03-23T03:47:50.887 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:47:50.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.889+0000 7fd28a800640 1 -- 172.21.15.102:0/3031661039 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd26406ac30 msgr2=0x7fd26406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:50.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.889+0000 7fd28a800640 1 --2- 172.21.15.102:0/3031661039 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd26406ac30 0x7fd26406d0f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd28415abc0 tx=0x7fd278008040 comp rx=0 tx=0).stop 2024-03-23T03:47:50.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.889+0000 7fd28a800640 1 -- 172.21.15.102:0/3031661039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd284143460 msgr2=0x7fd284139450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:50.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.889+0000 7fd28a800640 1 --2- 172.21.15.102:0/3031661039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd284143460 0x7fd284139450 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd270000c00 tx=0x7fd2700046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.889+0000 7fd28a800640 1 -- 172.21.15.102:0/3031661039 shutdown_connections 2024-03-23T03:47:50.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.889+0000 7fd28a800640 1 --2- 172.21.15.102:0/3031661039 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd26406ac30 0x7fd26406d0f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.889+0000 7fd28a800640 1 --2- 172.21.15.102:0/3031661039 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd284143930 0x7fd28413f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.889+0000 7fd28a800640 1 --2- 172.21.15.102:0/3031661039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd284143460 0x7fd284139450 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.889+0000 7fd28a800640 1 --2- 172.21.15.102:0/3031661039 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2840cdc50 0x7fd284134610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:50.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.889+0000 7fd28a800640 1 -- 172.21.15.102:0/3031661039 >> 172.21.15.102:0/3031661039 conn(0x7fd284065df0 msgr2=0x7fd284142570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:50.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.889+0000 7fd28a800640 1 -- 172.21.15.102:0/3031661039 shutdown_connections 2024-03-23T03:47:50.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:50.889+0000 7fd28a800640 1 -- 172.21.15.102:0/3031661039 wait complete. 2024-03-23T03:47:50.908 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-23T03:47:51.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.102+0000 7f77ea800640 1 Processor -- start 2024-03-23T03:47:51.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.103+0000 7f77ea800640 1 -- start start 2024-03-23T03:47:51.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.104+0000 7f77ea800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77e40b84b0 0x7f77e40d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:51.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.104+0000 7f77ea800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f77e40b5aa0 0x7f77e40b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:51.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.104+0000 7f77ea800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e40d1280 con 0x7f77e40cd740 2024-03-23T03:47:51.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.104+0000 7f77ea800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e40d20c0 con 0x7f77e40b5aa0 2024-03-23T03:47:51.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.104+0000 7f77ea800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40b8910 con 0x7f77e40b84b0 2024-03-23T03:47:51.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.104+0000 7f77e3400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f77e40cd740 0x7f77e40cdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47212/0 (socket says 172.21.15.102:47212) 2024-03-23T03:47:51.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.104+0000 7f77e3400640 1 -- 172.21.15.102:0/3492672674 learned_addr learned my addr 172.21.15.102:0/3492672674 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:51.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e2a00640 1 --2- 172.21.15.102:0/3492672674 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77e40b84b0 0x7f77e40d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:51.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e3e00640 1 --2- 172.21.15.102:0/3492672674 >> v2:172.21.15.173:3300/0 conn(0x7f77e40b5aa0 0x7f77e40b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:51.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e2000640 1 -- 172.21.15.102:0/3492672674 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715319974 0 0) 0x7f77e40d1280 con 0x7f77e40cd740 2024-03-23T03:47:51.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e2000640 1 -- 172.21.15.102:0/3492672674 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77c8003520 con 0x7f77e40cd740 2024-03-23T03:47:51.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e2000640 1 -- 172.21.15.102:0/3492672674 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4042129250 0 0) 0x7f77c8003520 con 0x7f77e40cd740 2024-03-23T03:47:51.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e2000640 1 -- 172.21.15.102:0/3492672674 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77e40b84b0 msgr2=0x7f77e40d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:51.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e2000640 1 --2- 172.21.15.102:0/3492672674 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77e40b84b0 0x7f77e40d0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:51.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e2000640 1 -- 172.21.15.102:0/3492672674 >> v2:172.21.15.173:3300/0 conn(0x7f77e40b5aa0 msgr2=0x7f77e40b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:51.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e2000640 1 --2- 172.21.15.102:0/3492672674 >> v2:172.21.15.173:3300/0 conn(0x7f77e40b5aa0 0x7f77e40b5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:51.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e2000640 1 -- 172.21.15.102:0/3492672674 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e40d14b0 con 0x7f77e40cd740 2024-03-23T03:47:51.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e2000640 1 -- 172.21.15.102:0/3492672674 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f77d4001670 con 0x7f77e40cd740 2024-03-23T03:47:51.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e2000640 1 -- 172.21.15.102:0/3492672674 >> v1:172.21.15.102:6789/0 conn(0x7f77e40cd740 legacy=0x7f77e40cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:51.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e2000640 1 --2- 172.21.15.102:0/3492672674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77c8003970 0x7f77c8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:51.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e2000640 1 --2- 172.21.15.102:0/3492672674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77c80047c0 0x7f77c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:51.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e2000640 1 --2- 172.21.15.102:0/3492672674 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77e40b84b0 0x7f77c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:51.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e2000640 1 -- 172.21.15.102:0/3492672674 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40d20c0 con 0x7f77c8003970 2024-03-23T03:47:51.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e2000640 1 -- 172.21.15.102:0/3492672674 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40b8910 con 0x7f77c80047c0 2024-03-23T03:47:51.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.105+0000 7f77e2000640 1 -- 172.21.15.102:0/3492672674 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c800b980 con 0x7f77e40b84b0 2024-03-23T03:47:51.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.106+0000 7f77e3400640 1 --2- 172.21.15.102:0/3492672674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77c8003970 0x7f77c8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:51.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.106+0000 7f77e3e00640 1 --2- 172.21.15.102:0/3492672674 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77e40b84b0 0x7f77c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:51.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.106+0000 7f77e2a00640 1 --2- 172.21.15.102:0/3492672674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77c80047c0 0x7f77c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:51.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.106+0000 7f77e3400640 1 -- 172.21.15.102:0/3492672674 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77e40b84b0 msgr2=0x7f77c8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:51.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.106+0000 7f77e3400640 1 --2- 172.21.15.102:0/3492672674 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77e40b84b0 0x7f77c8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:51.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.106+0000 7f77e3400640 1 -- 172.21.15.102:0/3492672674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77c80047c0 msgr2=0x7f77c8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:51.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.106+0000 7f77e3400640 1 --2- 172.21.15.102:0/3492672674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77c80047c0 0x7f77c8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:51.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.106+0000 7f77e3400640 1 -- 172.21.15.102:0/3492672674 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c8003520 con 0x7f77c8003970 2024-03-23T03:47:51.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.106+0000 7f77e2a00640 1 --2- 172.21.15.102:0/3492672674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77c80047c0 0x7f77c8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:51.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.107+0000 7f77e3400640 1 --2- 172.21.15.102:0/3492672674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77c8003970 0x7f77c8003d90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f77d400c240 tx=0x7f77c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8246ca3361f0dcea server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:51.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.108+0000 7f77e2000640 1 -- 172.21.15.102:0/3492672674 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d402cb00 con 0x7f77c8003970 2024-03-23T03:47:51.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.108+0000 7f77e2000640 1 -- 172.21.15.102:0/3492672674 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d402cc60 con 0x7f77c8003970 2024-03-23T03:47:51.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.108+0000 7f77e2000640 1 -- 172.21.15.102:0/3492672674 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d4005af0 con 0x7f77c8003970 2024-03-23T03:47:51.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.108+0000 7f77ea800640 1 -- 172.21.15.102:0/3492672674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77c8003970 msgr2=0x7f77c8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:51.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.108+0000 7f77ea800640 1 --2- 172.21.15.102:0/3492672674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77c8003970 0x7f77c8003d90 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f77d400c240 tx=0x7f77c80057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:51.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.108+0000 7f77e3e00640 1 -- 172.21.15.102:0/3492672674 reap_dead start 2024-03-23T03:47:51.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.108+0000 7f77ea800640 1 -- 172.21.15.102:0/3492672674 shutdown_connections 2024-03-23T03:47:51.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.108+0000 7f77ea800640 1 -- 172.21.15.102:0/3492672674 >> 172.21.15.102:0/3492672674 conn(0x7f77e40cf5d0 msgr2=0x7f77e40cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:51.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.108+0000 7f77ea800640 1 -- 172.21.15.102:0/3492672674 shutdown_connections 2024-03-23T03:47:51.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.108+0000 7f77ea800640 1 -- 172.21.15.102:0/3492672674 wait complete. 2024-03-23T03:47:51.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.110+0000 7f77ea800640 1 Processor -- start 2024-03-23T03:47:51.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.110+0000 7f77ea800640 1 -- start start 2024-03-23T03:47:51.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.111+0000 7f77ea800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77c8003970 0x7f77e4153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:51.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.111+0000 7f77ea800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e41534b0 0x7f77e4149c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:51.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.111+0000 7f77ea800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77e40cc130 0x7f77e40b3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:51.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.111+0000 7f77ea800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e4154150 con 0x7f77e41534b0 2024-03-23T03:47:51.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.111+0000 7f77ea800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40cbca0 con 0x7f77c8003970 2024-03-23T03:47:51.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.111+0000 7f77ea800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e414a190 con 0x7f77e40cc130 2024-03-23T03:47:51.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.112+0000 7f77e2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e41534b0 0x7f77e4149c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:51.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.112+0000 7f77e2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e41534b0 0x7f77e4149c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33440/0 (socket says 172.21.15.102:33440) 2024-03-23T03:47:51.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.112+0000 7f77e2a00640 1 -- 172.21.15.102:0/3968334414 learned_addr learned my addr 172.21.15.102:0/3968334414 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:51.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.112+0000 7f77e3400640 1 --2- 172.21.15.102:0/3968334414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77c8003970 0x7f77e4153c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.112+0000 7f77e3e00640 1 --2- 172.21.15.102:0/3968334414 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77e40cc130 0x7f77e40b3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.112+0000 7f77e2a00640 1 -- 172.21.15.102:0/3968334414 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77e40cc130 msgr2=0x7f77e40b3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.112+0000 7f77e2a00640 1 --2- 172.21.15.102:0/3968334414 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77e40cc130 0x7f77e40b3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.112+0000 7f77e2a00640 1 -- 172.21.15.102:0/3968334414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77c8003970 msgr2=0x7f77e4153c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.112+0000 7f77e2a00640 1 --2- 172.21.15.102:0/3968334414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77c8003970 0x7f77e4153c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.112+0000 7f77e2a00640 1 -- 172.21.15.102:0/3968334414 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e40cb090 con 0x7f77e41534b0 2024-03-23T03:47:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.112+0000 7f77e2a00640 1 --2- 172.21.15.102:0/3968334414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e41534b0 0x7f77e4149c50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f77d0010890 tx=0x7f77d0010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.113+0000 7f77db400640 1 -- 172.21.15.102:0/3968334414 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d000eb00 con 0x7f77e41534b0 2024-03-23T03:47:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.113+0000 7f77db400640 1 -- 172.21.15.102:0/3968334414 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d000ec60 con 0x7f77e41534b0 2024-03-23T03:47:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.113+0000 7f77ea800640 1 -- 172.21.15.102:0/3968334414 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e40cb370 con 0x7f77e41534b0 2024-03-23T03:47:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.113+0000 7f77db400640 1 -- 172.21.15.102:0/3968334414 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d001c400 con 0x7f77e41534b0 2024-03-23T03:47:51.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.113+0000 7f77ea800640 1 -- 172.21.15.102:0/3968334414 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e40cb930 con 0x7f77e41534b0 2024-03-23T03:47:51.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.114+0000 7f77ea800640 1 -- 172.21.15.102:0/3968334414 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e4073390 con 0x7f77e41534b0 2024-03-23T03:47:51.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.115+0000 7f77db400640 1 -- 172.21.15.102:0/3968334414 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f77d001c560 con 0x7f77e41534b0 2024-03-23T03:47:51.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.116+0000 7f77db400640 1 --2- 172.21.15.102:0/3968334414 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f77c406ac30 0x7f77c406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:51.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.116+0000 7f77db400640 1 -- 172.21.15.102:0/3968334414 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f77d0089b90 con 0x7f77e41534b0 2024-03-23T03:47:51.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.116+0000 7f77e3400640 1 --2- 172.21.15.102:0/3968334414 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f77c406ac30 0x7f77c406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:51.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.117+0000 7f77e3400640 1 --2- 172.21.15.102:0/3968334414 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f77c406ac30 0x7f77c406d0f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f77d4034000 tx=0x7f77d4007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:51.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.121+0000 7f77db400640 1 -- 172.21.15.102:0/3968334414 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f77d0054a50 con 0x7f77e41534b0 2024-03-23T03:47:51.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.314+0000 7f77ea800640 1 -- 172.21.15.102:0/3968334414 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f77e40b20b0 con 0x7f77e41534b0 2024-03-23T03:47:51.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.315+0000 7f77db400640 1 -- 172.21.15.102:0/3968334414 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f77d0058070 con 0x7f77e41534b0 2024-03-23T03:47:51.316 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:51.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.318+0000 7f77ea800640 1 -- 172.21.15.102:0/3968334414 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f77c406ac30 msgr2=0x7f77c406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:51.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.318+0000 7f77ea800640 1 --2- 172.21.15.102:0/3968334414 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f77c406ac30 0x7f77c406d0f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f77d4034000 tx=0x7f77d4007000 comp rx=0 tx=0).stop 2024-03-23T03:47:51.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.318+0000 7f77ea800640 1 -- 172.21.15.102:0/3968334414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e41534b0 msgr2=0x7f77e4149c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:51.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.318+0000 7f77ea800640 1 --2- 172.21.15.102:0/3968334414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e41534b0 0x7f77e4149c50 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f77d0010890 tx=0x7f77d0010d50 comp rx=0 tx=0).stop 2024-03-23T03:47:51.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.318+0000 7f77ea800640 1 -- 172.21.15.102:0/3968334414 shutdown_connections 2024-03-23T03:47:51.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.318+0000 7f77ea800640 1 --2- 172.21.15.102:0/3968334414 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f77c406ac30 0x7f77c406d0f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:51.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.318+0000 7f77ea800640 1 --2- 172.21.15.102:0/3968334414 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77e40cc130 0x7f77e40b3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:51.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.318+0000 7f77ea800640 1 --2- 172.21.15.102:0/3968334414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e41534b0 0x7f77e4149c50 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:51.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.318+0000 7f77ea800640 1 --2- 172.21.15.102:0/3968334414 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f77c8003970 0x7f77e4153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:51.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.318+0000 7f77ea800640 1 -- 172.21.15.102:0/3968334414 >> 172.21.15.102:0/3968334414 conn(0x7f77e40cf5d0 msgr2=0x7f77e415d1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:51.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.318+0000 7f77ea800640 1 -- 172.21.15.102:0/3968334414 shutdown_connections 2024-03-23T03:47:51.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.318+0000 7f77ea800640 1 -- 172.21.15.102:0/3968334414 wait complete. 2024-03-23T03:47:51.343 INFO:tasks.cephfs_test_runner:Starting test: test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) test: 2024-03-23T03:47:51.343 INFO:tasks.cephfs_test_runner:test_disable_enable_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2024-03-23T03:47:51.343 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values' 2024-03-23T03:47:51.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.545+0000 7fb303000640 1 Processor -- start 2024-03-23T03:47:51.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.545+0000 7fb303000640 1 -- start start 2024-03-23T03:47:51.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.546+0000 7fb303000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb2fc0b3e30 0x7fb2fc15cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:51.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.546+0000 7fb303000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc06c6f0 con 0x7fb2fc0b39b0 2024-03-23T03:47:51.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.546+0000 7fb303000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2fc06c9d0 con 0x7fb2fc15b560 2024-03-23T03:47:51.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.546+0000 7fb303000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2fc06ccb0 con 0x7fb2fc0b3e30 2024-03-23T03:47:51.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.547+0000 7fb2fb400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb2fc0b39b0 0x7fb2fc15c400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47222/0 (socket says 172.21.15.102:47222) 2024-03-23T03:47:51.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.547+0000 7fb2fb400640 1 -- 172.21.15.102:0/2033162271 learned_addr learned my addr 172.21.15.102:0/2033162271 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:51.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.547+0000 7fb301000640 1 --2- 172.21.15.102:0/2033162271 >> v2:172.21.15.102:3300/0 conn(0x7fb2fc0b3e30 0x7fb2fc15cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:51.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.547+0000 7fb2faa00640 1 -- 172.21.15.102:0/2033162271 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134077198 0 0) 0x7fb2fc06c6f0 con 0x7fb2fc0b39b0 2024-03-23T03:47:51.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.547+0000 7fb2faa00640 1 -- 172.21.15.102:0/2033162271 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e0003520 con 0x7fb2fc0b39b0 2024-03-23T03:47:51.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.547+0000 7fb301000640 1 -- 172.21.15.102:0/2033162271 >> v1:172.21.15.173:6789/0 conn(0x7fb2fc15b560 legacy=0x7fb2fc15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:51.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.547+0000 7fb301000640 1 -- 172.21.15.102:0/2033162271 >> v1:172.21.15.102:6789/0 conn(0x7fb2fc0b39b0 legacy=0x7fb2fc15c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:51.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.547+0000 7fb301000640 1 -- 172.21.15.102:0/2033162271 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc06cf90 con 0x7fb2fc0b3e30 2024-03-23T03:47:51.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.549+0000 7fb301000640 1 --2- 172.21.15.102:0/2033162271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2fc0b3e30 0x7fb2fc15cb10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4009d90 tx=0x7fb2f402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f287704f69aba87 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:51.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.549+0000 7fb2faa00640 1 -- 172.21.15.102:0/2033162271 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f402cd00 con 0x7fb2fc0b3e30 2024-03-23T03:47:51.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.549+0000 7fb2faa00640 1 -- 172.21.15.102:0/2033162271 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f402ce60 con 0x7fb2fc0b3e30 2024-03-23T03:47:51.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.549+0000 7fb2faa00640 1 -- 172.21.15.102:0/2033162271 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4034660 con 0x7fb2fc0b3e30 2024-03-23T03:47:51.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.549+0000 7fb303000640 1 -- 172.21.15.102:0/2033162271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2fc0b3e30 msgr2=0x7fb2fc15cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:51.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.549+0000 7fb303000640 1 --2- 172.21.15.102:0/2033162271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2fc0b3e30 0x7fb2fc15cb10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb2f4009d90 tx=0x7fb2f402bf50 comp rx=0 tx=0).stop 2024-03-23T03:47:51.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.550+0000 7fb303000640 1 -- 172.21.15.102:0/2033162271 shutdown_connections 2024-03-23T03:47:51.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.550+0000 7fb303000640 1 --2- 172.21.15.102:0/2033162271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2fc0b3e30 0x7fb2fc15cb10 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:51.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.550+0000 7fb303000640 1 -- 172.21.15.102:0/2033162271 >> 172.21.15.102:0/2033162271 conn(0x7fb2fc06a3d0 msgr2=0x7fb2fc0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:51.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.550+0000 7fb303000640 1 -- 172.21.15.102:0/2033162271 shutdown_connections 2024-03-23T03:47:51.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.550+0000 7fb303000640 1 -- 172.21.15.102:0/2033162271 wait complete. 2024-03-23T03:47:51.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.551+0000 7fb303000640 1 Processor -- start 2024-03-23T03:47:51.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.552+0000 7fb303000640 1 -- start start 2024-03-23T03:47:51.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.552+0000 7fb303000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2fc0b39b0 0x7fb2fc145ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:51.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.552+0000 7fb303000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2fc0b3e30 0x7fb2fc153470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:51.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.552+0000 7fb303000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb2fc15b560 0x7fb2fc1f2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:51.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.552+0000 7fb303000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc0b1dd0 con 0x7fb2fc0b39b0 2024-03-23T03:47:51.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.552+0000 7fb303000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc152f50 con 0x7fb2fc0b3e30 2024-03-23T03:47:51.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.552+0000 7fb303000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2fc0b17a0 con 0x7fb2fc15b560 2024-03-23T03:47:51.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.553+0000 7fb2fbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2fc0b39b0 0x7fb2fc145ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:51.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.553+0000 7fb2fbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2fc0b39b0 0x7fb2fc145ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33448/0 (socket says 172.21.15.102:33448) 2024-03-23T03:47:51.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.553+0000 7fb2fbe00640 1 -- 172.21.15.102:0/1212261413 learned_addr learned my addr 172.21.15.102:0/1212261413 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:51.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.553+0000 7fb301000640 1 --2- 172.21.15.102:0/1212261413 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb2fc15b560 0x7fb2fc1f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:51.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.553+0000 7fb2fbe00640 1 -- 172.21.15.102:0/1212261413 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb2fc15b560 msgr2=0x7fb2fc1f2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:51.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.553+0000 7fb2fbe00640 1 --2- 172.21.15.102:0/1212261413 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb2fc15b560 0x7fb2fc1f2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:51.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.553+0000 7fb2fbe00640 1 -- 172.21.15.102:0/1212261413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2fc0b3e30 msgr2=0x7fb2fc153470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:51.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.553+0000 7fb2fb400640 1 --2- 172.21.15.102:0/1212261413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2fc0b3e30 0x7fb2fc153470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:51.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.553+0000 7fb2fbe00640 1 --2- 172.21.15.102:0/1212261413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2fc0b3e30 0x7fb2fc153470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:51.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.553+0000 7fb2fbe00640 1 -- 172.21.15.102:0/1212261413 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f4009a70 con 0x7fb2fc0b39b0 2024-03-23T03:47:51.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.553+0000 7fb2fbe00640 1 --2- 172.21.15.102:0/1212261413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2fc0b39b0 0x7fb2fc145ee0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec00c400 tx=0x7fb2ec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:51.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.554+0000 7fb2f3e00640 1 -- 172.21.15.102:0/1212261413 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec0052d0 con 0x7fb2fc0b39b0 2024-03-23T03:47:51.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.554+0000 7fb2f3e00640 1 -- 172.21.15.102:0/1212261413 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ec005430 con 0x7fb2fc0b39b0 2024-03-23T03:47:51.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.554+0000 7fb303000640 1 -- 172.21.15.102:0/1212261413 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2fc146420 con 0x7fb2fc0b39b0 2024-03-23T03:47:51.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.554+0000 7fb2f3e00640 1 -- 172.21.15.102:0/1212261413 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec003830 con 0x7fb2fc0b39b0 2024-03-23T03:47:51.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.554+0000 7fb303000640 1 -- 172.21.15.102:0/1212261413 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2fc0b19f0 con 0x7fb2fc0b39b0 2024-03-23T03:47:51.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.555+0000 7fb303000640 1 -- 172.21.15.102:0/1212261413 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2fc06ac20 con 0x7fb2fc0b39b0 2024-03-23T03:47:51.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.556+0000 7fb2f3e00640 1 -- 172.21.15.102:0/1212261413 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb2ec003990 con 0x7fb2fc0b39b0 2024-03-23T03:47:51.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.557+0000 7fb2f3e00640 1 --2- 172.21.15.102:0/1212261413 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2dc06ac00 0x7fb2dc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:51.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.558+0000 7fb2f3e00640 1 -- 172.21.15.102:0/1212261413 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb2ec088ca0 con 0x7fb2fc0b39b0 2024-03-23T03:47:51.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.558+0000 7fb2fb400640 1 --2- 172.21.15.102:0/1212261413 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2dc06ac00 0x7fb2dc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:51.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.558+0000 7fb2fb400640 1 --2- 172.21.15.102:0/1212261413 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2dc06ac00 0x7fb2dc06d0c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc14eb80 tx=0x7fb2e8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:51.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.562+0000 7fb2f3e00640 1 -- 172.21.15.102:0/1212261413 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb2ec053ab0 con 0x7fb2fc0b39b0 2024-03-23T03:47:51.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:51.823+0000 7fb303000640 1 -- 172.21.15.102:0/1212261413 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values"]} v 0) v1 -- 0x7fb2fc073390 con 0x7fb2fc0b39b0 2024-03-23T03:47:52.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.756+0000 7fb2f3e00640 1 -- 172.21.15.102:0/1212261413 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values"]}]=0 v71) v1 ==== 163+0+0 (secure 0 0 0) 0x7fb2ec0170e0 con 0x7fb2fc0b39b0 2024-03-23T03:47:52.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.760+0000 7fb303000640 1 -- 172.21.15.102:0/1212261413 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2dc06ac00 msgr2=0x7fb2dc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:52.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.760+0000 7fb303000640 1 --2- 172.21.15.102:0/1212261413 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2dc06ac00 0x7fb2dc06d0c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc14eb80 tx=0x7fb2e8009040 comp rx=0 tx=0).stop 2024-03-23T03:47:52.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.760+0000 7fb303000640 1 -- 172.21.15.102:0/1212261413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2fc0b39b0 msgr2=0x7fb2fc145ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:52.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.760+0000 7fb303000640 1 --2- 172.21.15.102:0/1212261413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2fc0b39b0 0x7fb2fc145ee0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec00c400 tx=0x7fb2ec00c8c0 comp rx=0 tx=0).stop 2024-03-23T03:47:52.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.760+0000 7fb303000640 1 -- 172.21.15.102:0/1212261413 shutdown_connections 2024-03-23T03:47:52.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.760+0000 7fb303000640 1 --2- 172.21.15.102:0/1212261413 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2dc06ac00 0x7fb2dc06d0c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:52.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.760+0000 7fb303000640 1 --2- 172.21.15.102:0/1212261413 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb2fc15b560 0x7fb2fc1f2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:52.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.760+0000 7fb303000640 1 --2- 172.21.15.102:0/1212261413 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb2fc0b3e30 0x7fb2fc153470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:52.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.760+0000 7fb303000640 1 --2- 172.21.15.102:0/1212261413 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2fc0b39b0 0x7fb2fc145ee0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:52.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.760+0000 7fb303000640 1 -- 172.21.15.102:0/1212261413 >> 172.21.15.102:0/1212261413 conn(0x7fb2fc06a3d0 msgr2=0x7fb2fc0aebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:52.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.761+0000 7fb303000640 1 -- 172.21.15.102:0/1212261413 shutdown_connections 2024-03-23T03:47:52.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.761+0000 7fb303000640 1 -- 172.21.15.102:0/1212261413 wait complete. 2024-03-23T03:47:52.782 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-23T03:47:52.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.997+0000 7fc2bb800640 1 Processor -- start 2024-03-23T03:47:52.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.998+0000 7fc2bb800640 1 -- start start 2024-03-23T03:47:53.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.999+0000 7fc2bb800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2b40cd740 0x7fc2b40b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:53.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.999+0000 7fc2bb800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc2b40b5aa0 0x7fc2b40b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:53.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.999+0000 7fc2bb800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2b40d20c0 con 0x7fc2b40d0d40 2024-03-23T03:47:53.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.999+0000 7fc2bb800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b40d14b0 con 0x7fc2b40cd740 2024-03-23T03:47:53.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.999+0000 7fc2bb800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2b40d1790 con 0x7fc2b40b5aa0 2024-03-23T03:47:53.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.999+0000 7fc2b3e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc2b40d0d40 0x7fc2b40cdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40968/0 (socket says 172.21.15.102:40968) 2024-03-23T03:47:53.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.999+0000 7fc2b3e00640 1 -- 172.21.15.102:0/2872666532 learned_addr learned my addr 172.21.15.102:0/2872666532 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:53.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:52.999+0000 7fc2b8e00640 1 --2- 172.21.15.102:0/2872666532 >> v2:172.21.15.102:3300/0 conn(0x7fc2b40cd740 0x7fc2b40b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:53.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.000+0000 7fc2b9800640 1 --2- 172.21.15.102:0/2872666532 >> v2:172.21.15.173:3300/0 conn(0x7fc2b40b5aa0 0x7fc2b40b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:53.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.000+0000 7fc2b3400640 1 -- 172.21.15.102:0/2872666532 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803788369 0 0) 0x7fc2b40d20c0 con 0x7fc2b40d0d40 2024-03-23T03:47:53.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.000+0000 7fc2b3400640 1 -- 172.21.15.102:0/2872666532 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc298003520 con 0x7fc2b40d0d40 2024-03-23T03:47:53.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.000+0000 7fc2b8e00640 1 -- 172.21.15.102:0/2872666532 >> v2:172.21.15.173:3300/0 conn(0x7fc2b40b5aa0 msgr2=0x7fc2b40b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:53.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.000+0000 7fc2b8e00640 1 --2- 172.21.15.102:0/2872666532 >> v2:172.21.15.173:3300/0 conn(0x7fc2b40b5aa0 0x7fc2b40b5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.000+0000 7fc2b8e00640 1 -- 172.21.15.102:0/2872666532 >> v1:172.21.15.102:6789/0 conn(0x7fc2b40d0d40 legacy=0x7fc2b40cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:53.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.000+0000 7fc2b8e00640 1 -- 172.21.15.102:0/2872666532 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b40d1a70 con 0x7fc2b40cd740 2024-03-23T03:47:53.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.000+0000 7fc2b9800640 1 --2- 172.21.15.102:0/2872666532 >> v2:172.21.15.173:3300/0 conn(0x7fc2b40b5aa0 0x7fc2b40b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:53.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.001+0000 7fc2b8e00640 1 --2- 172.21.15.102:0/2872666532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b40cd740 0x7fc2b40b84b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc2b40c99c0 tx=0x7fc2a402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80cd73eeddb8de1a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:53.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.001+0000 7fc2b3400640 1 -- 172.21.15.102:0/2872666532 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a402cc80 con 0x7fc2b40cd740 2024-03-23T03:47:53.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.001+0000 7fc2b3400640 1 -- 172.21.15.102:0/2872666532 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a402cde0 con 0x7fc2b40cd740 2024-03-23T03:47:53.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.001+0000 7fc2b3400640 1 -- 172.21.15.102:0/2872666532 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4034660 con 0x7fc2b40cd740 2024-03-23T03:47:53.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.002+0000 7fc2bb800640 1 -- 172.21.15.102:0/2872666532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b40cd740 msgr2=0x7fc2b40b84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:53.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.002+0000 7fc2bb800640 1 --2- 172.21.15.102:0/2872666532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b40cd740 0x7fc2b40b84b0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc2b40c99c0 tx=0x7fc2a402bed0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.002+0000 7fc2bb800640 1 -- 172.21.15.102:0/2872666532 shutdown_connections 2024-03-23T03:47:53.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.002+0000 7fc2bb800640 1 --2- 172.21.15.102:0/2872666532 >> v2:172.21.15.173:3300/0 conn(0x7fc2b40b5aa0 0x7fc2b40b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.002+0000 7fc2bb800640 1 --2- 172.21.15.102:0/2872666532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b40cd740 0x7fc2b40b84b0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.002+0000 7fc2bb800640 1 -- 172.21.15.102:0/2872666532 >> 172.21.15.102:0/2872666532 conn(0x7fc2b40cf5d0 msgr2=0x7fc2b40cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:53.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.002+0000 7fc2bb800640 1 -- 172.21.15.102:0/2872666532 shutdown_connections 2024-03-23T03:47:53.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.002+0000 7fc2bb800640 1 -- 172.21.15.102:0/2872666532 wait complete. 2024-03-23T03:47:53.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.004+0000 7fc2bb800640 1 Processor -- start 2024-03-23T03:47:53.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.004+0000 7fc2bb800640 1 -- start start 2024-03-23T03:47:53.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.005+0000 7fc2bb800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2b40b5aa0 0x7fc2b40ad190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:53.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.005+0000 7fc2bb800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b40cd740 0x7fc2b40a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:53.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.005+0000 7fc2bb800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2b40d0d40 0x7fc2b414c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:53.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.005+0000 7fc2bb800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b40aa7b0 con 0x7fc2b40cd740 2024-03-23T03:47:53.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.005+0000 7fc2bb800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b40aa4d0 con 0x7fc2b40b5aa0 2024-03-23T03:47:53.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.005+0000 7fc2bb800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2b414cf00 con 0x7fc2b40d0d40 2024-03-23T03:47:53.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.005+0000 7fc2b3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b40cd740 0x7fc2b40a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:53.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.005+0000 7fc2b3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b40cd740 0x7fc2b40a9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54038/0 (socket says 172.21.15.102:54038) 2024-03-23T03:47:53.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.005+0000 7fc2b3e00640 1 -- 172.21.15.102:0/2585425933 learned_addr learned my addr 172.21.15.102:0/2585425933 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:53.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.005+0000 7fc2b8e00640 1 --2- 172.21.15.102:0/2585425933 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2b40b5aa0 0x7fc2b40ad190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:53.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.006+0000 7fc2b9800640 1 --2- 172.21.15.102:0/2585425933 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2b40d0d40 0x7fc2b414c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:53.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.006+0000 7fc2b3e00640 1 -- 172.21.15.102:0/2585425933 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2b40d0d40 msgr2=0x7fc2b414c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:53.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.006+0000 7fc2b3e00640 1 --2- 172.21.15.102:0/2585425933 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2b40d0d40 0x7fc2b414c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.006+0000 7fc2b3e00640 1 -- 172.21.15.102:0/2585425933 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2b40b5aa0 msgr2=0x7fc2b40ad190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:53.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.006+0000 7fc2b3e00640 1 --2- 172.21.15.102:0/2585425933 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2b40b5aa0 0x7fc2b40ad190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.006+0000 7fc2b3e00640 1 -- 172.21.15.102:0/2585425933 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a4009a70 con 0x7fc2b40cd740 2024-03-23T03:47:53.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.006+0000 7fc2b8e00640 1 --2- 172.21.15.102:0/2585425933 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2b40b5aa0 0x7fc2b40ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:53.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.006+0000 7fc2b3e00640 1 --2- 172.21.15.102:0/2585425933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b40cd740 0x7fc2b40a9c80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc2a000c410 tx=0x7fc2a000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:53.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.006+0000 7fc2b0c00640 1 -- 172.21.15.102:0/2585425933 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0005ce0 con 0x7fc2b40cd740 2024-03-23T03:47:53.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.006+0000 7fc2b0c00640 1 -- 172.21.15.102:0/2585425933 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2a0005e40 con 0x7fc2b40cd740 2024-03-23T03:47:53.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.006+0000 7fc2b0c00640 1 -- 172.21.15.102:0/2585425933 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2a00072f0 con 0x7fc2b40cd740 2024-03-23T03:47:53.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.006+0000 7fc2bb800640 1 -- 172.21.15.102:0/2585425933 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b414d1e0 con 0x7fc2b40cd740 2024-03-23T03:47:53.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.006+0000 7fc2bb800640 1 -- 172.21.15.102:0/2585425933 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b40aa1c0 con 0x7fc2b40cd740 2024-03-23T03:47:53.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.007+0000 7fc2bb800640 1 -- 172.21.15.102:0/2585425933 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2b40d2530 con 0x7fc2b40cd740 2024-03-23T03:47:53.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.008+0000 7fc2b0c00640 1 -- 172.21.15.102:0/2585425933 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc2a0007450 con 0x7fc2b40cd740 2024-03-23T03:47:53.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.010+0000 7fc2b0c00640 1 --2- 172.21.15.102:0/2585425933 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc29406ac30 0x7fc29406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:53.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.010+0000 7fc2b0c00640 1 -- 172.21.15.102:0/2585425933 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc2a0088e00 con 0x7fc2b40cd740 2024-03-23T03:47:53.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.010+0000 7fc2b8e00640 1 --2- 172.21.15.102:0/2585425933 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc29406ac30 0x7fc29406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:53.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.010+0000 7fc2b8e00640 1 --2- 172.21.15.102:0/2585425933 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc29406ac30 0x7fc29406d0f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4036000 tx=0x7fc2a4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:53.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.014+0000 7fc2b0c00640 1 -- 172.21.15.102:0/2585425933 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc2b40d2530 con 0x7fc2b40cd740 2024-03-23T03:47:53.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.208+0000 7fc2bb800640 1 -- 172.21.15.102:0/2585425933 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc2b40d2530 con 0x7fc2b40cd740 2024-03-23T03:47:53.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.217+0000 7fc2b0c00640 1 -- 172.21.15.102:0/2585425933 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc2b40d2530 con 0x7fc2b40cd740 2024-03-23T03:47:53.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.219+0000 7fc2bb800640 1 -- 172.21.15.102:0/2585425933 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc29406ac30 msgr2=0x7fc29406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:53.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.219+0000 7fc2bb800640 1 --2- 172.21.15.102:0/2585425933 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc29406ac30 0x7fc29406d0f0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc2a4036000 tx=0x7fc2a4030000 comp rx=0 tx=0).stop 2024-03-23T03:47:53.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.220+0000 7fc2bb800640 1 -- 172.21.15.102:0/2585425933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b40cd740 msgr2=0x7fc2b40a9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:53.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.220+0000 7fc2bb800640 1 --2- 172.21.15.102:0/2585425933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b40cd740 0x7fc2b40a9c80 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fc2a000c410 tx=0x7fc2a000c8d0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.220+0000 7fc2bb800640 1 -- 172.21.15.102:0/2585425933 shutdown_connections 2024-03-23T03:47:53.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.220+0000 7fc2bb800640 1 --2- 172.21.15.102:0/2585425933 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc29406ac30 0x7fc29406d0f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.220+0000 7fc2bb800640 1 --2- 172.21.15.102:0/2585425933 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2b40d0d40 0x7fc2b414c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.220+0000 7fc2bb800640 1 --2- 172.21.15.102:0/2585425933 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2b40cd740 0x7fc2b40a9c80 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.220+0000 7fc2bb800640 1 --2- 172.21.15.102:0/2585425933 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2b40b5aa0 0x7fc2b40ad190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.220+0000 7fc2bb800640 1 -- 172.21.15.102:0/2585425933 >> 172.21.15.102:0/2585425933 conn(0x7fc2b40cf5d0 msgr2=0x7fc2b415e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:53.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.220+0000 7fc2bb800640 1 -- 172.21.15.102:0/2585425933 shutdown_connections 2024-03-23T03:47:53.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.220+0000 7fc2bb800640 1 -- 172.21.15.102:0/2585425933 wait complete. 2024-03-23T03:47:53.240 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T03:47:53.240 DEBUG:teuthology.orchestra.run.smithi102:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:47:53.353 DEBUG:teuthology.orchestra.run.smithi173:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:47:53.482 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T03:47:53.482 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:47:53.482 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:47:53.538 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:47:53.538 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:47:53.592 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:47:53.592 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:47:53.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:53.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.860+0000 7fd9ce200640 1 Processor -- start 2024-03-23T03:47:53.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.861+0000 7fd9ce200640 1 -- start start 2024-03-23T03:47:53.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.862+0000 7fd9ce200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c815b560 0x7fd9c815b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:53.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.862+0000 7fd9ce200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9c80b39b0 0x7fd9c80b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:53.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.862+0000 7fd9ce200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd9c815e500 0x7fd9c815e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:53.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.863+0000 7fd9ce200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9c80b4370 con 0x7fd9c80b39b0 2024-03-23T03:47:53.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.863+0000 7fd9ce200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9c806c6f0 con 0x7fd9c815e500 2024-03-23T03:47:53.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.863+0000 7fd9ce200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c806c9d0 con 0x7fd9c815b560 2024-03-23T03:47:53.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.863+0000 7fd9bea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9c80b39b0 0x7fd9c80b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:53.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.863+0000 7fd9bea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9c80b39b0 0x7fd9c80b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54054/0 (socket says 172.21.15.102:54054) 2024-03-23T03:47:53.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.863+0000 7fd9bea00640 1 -- 172.21.15.102:0/3773982370 learned_addr learned my addr 172.21.15.102:0/3773982370 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:53.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.863+0000 7fd9c7400640 1 --2- 172.21.15.102:0/3773982370 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c815b560 0x7fd9c815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:53.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.863+0000 7fd9bea00640 1 -- 172.21.15.102:0/3773982370 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c815b560 msgr2=0x7fd9c815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:53.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.863+0000 7fd9bea00640 1 --2- 172.21.15.102:0/3773982370 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c815b560 0x7fd9c815b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.863+0000 7fd9bea00640 1 -- 172.21.15.102:0/3773982370 >> v2:172.21.15.173:3300/0 conn(0x7fd9c815e500 msgr2=0x7fd9c815e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:53.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.863+0000 7fd9bea00640 1 --2- 172.21.15.102:0/3773982370 >> v2:172.21.15.173:3300/0 conn(0x7fd9c815e500 0x7fd9c815e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.863+0000 7fd9bea00640 1 -- 172.21.15.102:0/3773982370 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c806ccb0 con 0x7fd9c80b39b0 2024-03-23T03:47:53.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.864+0000 7fd9bea00640 1 --2- 172.21.15.102:0/3773982370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c80b39b0 0x7fd9c80b3e30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80cab40 tx=0x7fd9b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b91dd9d9c91e5217 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:53.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.865+0000 7fd9c6a00640 1 -- 172.21.15.102:0/3773982370 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0031870 con 0x7fd9c80b39b0 2024-03-23T03:47:53.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.865+0000 7fd9c6a00640 1 -- 172.21.15.102:0/3773982370 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b00319d0 con 0x7fd9c80b39b0 2024-03-23T03:47:53.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.865+0000 7fd9c6a00640 1 -- 172.21.15.102:0/3773982370 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0031d60 con 0x7fd9c80b39b0 2024-03-23T03:47:53.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.865+0000 7fd9ce200640 1 -- 172.21.15.102:0/3773982370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c80b39b0 msgr2=0x7fd9c80b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:53.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.865+0000 7fd9ce200640 1 --2- 172.21.15.102:0/3773982370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c80b39b0 0x7fd9c80b3e30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd9c80cab40 tx=0x7fd9b002b910 comp rx=0 tx=0).stop 2024-03-23T03:47:53.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.865+0000 7fd9ce200640 1 -- 172.21.15.102:0/3773982370 shutdown_connections 2024-03-23T03:47:53.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.865+0000 7fd9ce200640 1 --2- 172.21.15.102:0/3773982370 >> v2:172.21.15.173:3300/0 conn(0x7fd9c815e500 0x7fd9c815e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.865+0000 7fd9ce200640 1 --2- 172.21.15.102:0/3773982370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c80b39b0 0x7fd9c80b3e30 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.865+0000 7fd9ce200640 1 --2- 172.21.15.102:0/3773982370 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c815b560 0x7fd9c815b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.865+0000 7fd9ce200640 1 -- 172.21.15.102:0/3773982370 >> 172.21.15.102:0/3773982370 conn(0x7fd9c806a3d0 msgr2=0x7fd9c80b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:53.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.865+0000 7fd9ce200640 1 -- 172.21.15.102:0/3773982370 shutdown_connections 2024-03-23T03:47:53.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.866+0000 7fd9ce200640 1 -- 172.21.15.102:0/3773982370 wait complete. 2024-03-23T03:47:53.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.867+0000 7fd9ce200640 1 Processor -- start 2024-03-23T03:47:53.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.867+0000 7fd9ce200640 1 -- start start 2024-03-23T03:47:53.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.868+0000 7fd9ce200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c80b39b0 0x7fd9c81f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:53.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.868+0000 7fd9ce200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c815b560 0x7fd9c81f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:53.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.868+0000 7fd9ce200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9c815e500 0x7fd9c8151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:53.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.868+0000 7fd9ce200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c81516c0 con 0x7fd9c815b560 2024-03-23T03:47:53.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.868+0000 7fd9ce200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c80abd20 con 0x7fd9c815e500 2024-03-23T03:47:53.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.868+0000 7fd9ce200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c80ac000 con 0x7fd9c80b39b0 2024-03-23T03:47:53.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.868+0000 7fd9bea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c815b560 0x7fd9c81f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:53.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.869+0000 7fd9bea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c815b560 0x7fd9c81f38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54060/0 (socket says 172.21.15.102:54060) 2024-03-23T03:47:53.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.869+0000 7fd9bea00640 1 -- 172.21.15.102:0/185052710 learned_addr learned my addr 172.21.15.102:0/185052710 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:53.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.869+0000 7fd9c7400640 1 --2- 172.21.15.102:0/185052710 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c80b39b0 0x7fd9c81f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:53.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.869+0000 7fd9c7e00640 1 --2- 172.21.15.102:0/185052710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9c815e500 0x7fd9c8151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:53.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.869+0000 7fd9bea00640 1 -- 172.21.15.102:0/185052710 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c80b39b0 msgr2=0x7fd9c81f32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:53.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.869+0000 7fd9bea00640 1 --2- 172.21.15.102:0/185052710 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c80b39b0 0x7fd9c81f32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.869+0000 7fd9bea00640 1 -- 172.21.15.102:0/185052710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9c815e500 msgr2=0x7fd9c8151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:53.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.869+0000 7fd9bea00640 1 --2- 172.21.15.102:0/185052710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9c815e500 0x7fd9c8151180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:53.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.869+0000 7fd9bea00640 1 -- 172.21.15.102:0/185052710 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b00094d0 con 0x7fd9c815b560 2024-03-23T03:47:53.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.869+0000 7fd9bea00640 1 --2- 172.21.15.102:0/185052710 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c815b560 0x7fd9c81f38a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd9b002b950 tx=0x7fd9b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.870+0000 7fd9bfe00640 1 -- 172.21.15.102:0/185052710 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0031860 con 0x7fd9c815b560 2024-03-23T03:47:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.870+0000 7fd9bfe00640 1 -- 172.21.15.102:0/185052710 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b00443f0 con 0x7fd9c815b560 2024-03-23T03:47:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.870+0000 7fd9bfe00640 1 -- 172.21.15.102:0/185052710 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b00447a0 con 0x7fd9c815b560 2024-03-23T03:47:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.870+0000 7fd9ce200640 1 -- 172.21.15.102:0/185052710 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9c80ac2e0 con 0x7fd9c815b560 2024-03-23T03:47:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.870+0000 7fd9ce200640 1 -- 172.21.15.102:0/185052710 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9c80ab9d0 con 0x7fd9c815b560 2024-03-23T03:47:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.871+0000 7fd9ce200640 1 -- 172.21.15.102:0/185052710 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9c80ac800 con 0x7fd9c815b560 2024-03-23T03:47:53.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.872+0000 7fd9bfe00640 1 -- 172.21.15.102:0/185052710 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd9b004d450 con 0x7fd9c815b560 2024-03-23T03:47:53.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.873+0000 7fd9bfe00640 1 --2- 172.21.15.102:0/185052710 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9a406ac00 0x7fd9a406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:53.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.873+0000 7fd9bfe00640 1 -- 172.21.15.102:0/185052710 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd9b002d040 con 0x7fd9c815b560 2024-03-23T03:47:53.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.873+0000 7fd9c7400640 1 --2- 172.21.15.102:0/185052710 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9a406ac00 0x7fd9a406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:53.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.874+0000 7fd9c7400640 1 --2- 172.21.15.102:0/185052710 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9a406ac00 0x7fd9a406d0c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd9c81f5c30 tx=0x7fd9b8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:53.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:53.878+0000 7fd9bfe00640 1 -- 172.21.15.102:0/185052710 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd9c80ac800 con 0x7fd9c815b560 2024-03-23T03:47:54.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.079+0000 7fd9ce200640 1 -- 172.21.15.102:0/185052710 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9c80ac800 con 0x7fd9c815b560 2024-03-23T03:47:54.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.082+0000 7fd9bfe00640 1 -- 172.21.15.102:0/185052710 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18260 (secure 0 0 0) 0x7fd9c80ac800 con 0x7fd9c815b560 2024-03-23T03:47:54.083 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:54.083 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":28,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:47:41.647786+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:47:54.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.084+0000 7fd9ce200640 1 -- 172.21.15.102:0/185052710 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9a406ac00 msgr2=0x7fd9a406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:54.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.084+0000 7fd9ce200640 1 --2- 172.21.15.102:0/185052710 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9a406ac00 0x7fd9a406d0c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd9c81f5c30 tx=0x7fd9b8008040 comp rx=0 tx=0).stop 2024-03-23T03:47:54.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.084+0000 7fd9ce200640 1 -- 172.21.15.102:0/185052710 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c815b560 msgr2=0x7fd9c81f38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:54.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.084+0000 7fd9ce200640 1 --2- 172.21.15.102:0/185052710 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c815b560 0x7fd9c81f38a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd9b002b950 tx=0x7fd9b00046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.084+0000 7fd9ce200640 1 -- 172.21.15.102:0/185052710 shutdown_connections 2024-03-23T03:47:54.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.084+0000 7fd9ce200640 1 --2- 172.21.15.102:0/185052710 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9a406ac00 0x7fd9a406d0c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.084+0000 7fd9ce200640 1 --2- 172.21.15.102:0/185052710 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9c815e500 0x7fd9c8151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.084+0000 7fd9ce200640 1 --2- 172.21.15.102:0/185052710 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c815b560 0x7fd9c81f38a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.084+0000 7fd9ce200640 1 --2- 172.21.15.102:0/185052710 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c80b39b0 0x7fd9c81f32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.085+0000 7fd9ce200640 1 -- 172.21.15.102:0/185052710 >> 172.21.15.102:0/185052710 conn(0x7fd9c806a3d0 msgr2=0x7fd9c815c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:54.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.085+0000 7fd9ce200640 1 -- 172.21.15.102:0/185052710 shutdown_connections 2024-03-23T03:47:54.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.085+0000 7fd9ce200640 1 -- 172.21.15.102:0/185052710 wait complete. 2024-03-23T03:47:54.105 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:46:52.158650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:46:52.914275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-23T03:46:55.983754+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-23T03:47:54.106 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:47:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.315+0000 7f7dd8000640 1 Processor -- start 2024-03-23T03:47:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.315+0000 7f7dd8000640 1 -- start start 2024-03-23T03:47:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.316+0000 7f7dd8000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7dd00ca7b0 0x7f7dd00cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.316+0000 7f7dd8000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f7dd00ce930 0x7f7dd00cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.316+0000 7f7dd8000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd0159c10 con 0x7f7dd0159830 2024-03-23T03:47:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.316+0000 7f7dd8000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd006de50 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.316+0000 7f7dd8000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd00cf300 con 0x7f7dd00ce930 2024-03-23T03:47:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.317+0000 7f7dd4c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7dd00ca7b0 0x7f7dd00cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.317+0000 7f7dd4c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7dd00ca7b0 0x7f7dd00cdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54062/0 (socket says 172.21.15.102:54062) 2024-03-23T03:47:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.317+0000 7f7dd4c00640 1 -- 172.21.15.102:0/1831526918 learned_addr learned my addr 172.21.15.102:0/1831526918 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.317+0000 7f7dd4c00640 1 -- 172.21.15.102:0/1831526918 >> v2:172.21.15.173:3300/0 conn(0x7f7dd00ce930 msgr2=0x7f7dd00cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.317+0000 7f7dd4c00640 1 --2- 172.21.15.102:0/1831526918 >> v2:172.21.15.173:3300/0 conn(0x7f7dd00ce930 0x7f7dd00cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.317+0000 7f7dd4c00640 1 -- 172.21.15.102:0/1831526918 >> v1:172.21.15.173:6789/0 conn(0x7f7dd0159830 legacy=0x7f7dd00cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.317+0000 7f7dd4c00640 1 -- 172.21.15.102:0/1831526918 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd00cd840 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.318+0000 7f7dd4c00640 1 --2- 172.21.15.102:0/1831526918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dd00ca7b0 0x7f7dd00cdcd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc0098a0 tx=0x7f7dbc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ae6280c0d23d3f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:54.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.319+0000 7f7dcfe00640 1 -- 172.21.15.102:0/1831526918 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc030840 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.319+0000 7f7dcfe00640 1 -- 172.21.15.102:0/1831526918 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dbc0309a0 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.319+0000 7f7dcfe00640 1 -- 172.21.15.102:0/1831526918 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc030d30 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.319+0000 7f7dd8000640 1 -- 172.21.15.102:0/1831526918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dd00ca7b0 msgr2=0x7f7dd00cdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:54.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.319+0000 7f7dd8000640 1 --2- 172.21.15.102:0/1831526918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dd00ca7b0 0x7f7dd00cdcd0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc0098a0 tx=0x7f7dbc02b980 comp rx=0 tx=0).stop 2024-03-23T03:47:54.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.319+0000 7f7dd8000640 1 -- 172.21.15.102:0/1831526918 shutdown_connections 2024-03-23T03:47:54.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.319+0000 7f7dd8000640 1 --2- 172.21.15.102:0/1831526918 >> v2:172.21.15.173:3300/0 conn(0x7f7dd00ce930 0x7f7dd00cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.319+0000 7f7dd8000640 1 --2- 172.21.15.102:0/1831526918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dd00ca7b0 0x7f7dd00cdcd0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.319+0000 7f7dd8000640 1 -- 172.21.15.102:0/1831526918 >> 172.21.15.102:0/1831526918 conn(0x7f7dd006a0b0 msgr2=0x7f7dd006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:54.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.320+0000 7f7dd8000640 1 -- 172.21.15.102:0/1831526918 shutdown_connections 2024-03-23T03:47:54.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.320+0000 7f7dd8000640 1 -- 172.21.15.102:0/1831526918 wait complete. 2024-03-23T03:47:54.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.321+0000 7f7dd8000640 1 Processor -- start 2024-03-23T03:47:54.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.321+0000 7f7dd8000640 1 -- start start 2024-03-23T03:47:54.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.322+0000 7f7dd8000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dd00ca7b0 0x7f7dd015c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:54.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.322+0000 7f7dd8000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dd00ce930 0x7f7dd0142120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:54.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.322+0000 7f7dd8000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7dd0159830 0x7f7dd015ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:54.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.322+0000 7f7dd8000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd0142660 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.322+0000 7f7dd8000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd015b500 con 0x7f7dd0159830 2024-03-23T03:47:54.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.322+0000 7f7dd8000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd015b7e0 con 0x7f7dd00ce930 2024-03-23T03:47:54.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.322+0000 7f7dd5600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dd00ca7b0 0x7f7dd015c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:54.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.323+0000 7f7dd5600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dd00ca7b0 0x7f7dd015c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54070/0 (socket says 172.21.15.102:54070) 2024-03-23T03:47:54.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.323+0000 7f7dd5600640 1 -- 172.21.15.102:0/45552896 learned_addr learned my addr 172.21.15.102:0/45552896 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.323+0000 7f7dd5600640 1 -- 172.21.15.102:0/45552896 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dd00ce930 msgr2=0x7f7dd0142120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.323+0000 7f7dd5600640 1 --2- 172.21.15.102:0/45552896 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dd00ce930 0x7f7dd0142120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.323+0000 7f7dd5600640 1 -- 172.21.15.102:0/45552896 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7dd0159830 msgr2=0x7f7dd015ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.323+0000 7f7dd5600640 1 --2- 172.21.15.102:0/45552896 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7dd0159830 0x7f7dd015ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.323+0000 7f7dd5600640 1 -- 172.21.15.102:0/45552896 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc0094d0 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.323+0000 7f7dd5600640 1 --2- 172.21.15.102:0/45552896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dd00ca7b0 0x7f7dd015c5a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400c430 tx=0x7f7dc400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.323+0000 7f7dcd600640 1 -- 172.21.15.102:0/45552896 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4011790 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.324+0000 7f7dcd600640 1 -- 172.21.15.102:0/45552896 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc40118f0 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.324+0000 7f7dcd600640 1 -- 172.21.15.102:0/45552896 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4011ce0 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.324+0000 7f7dd8000640 1 -- 172.21.15.102:0/45552896 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd015bac0 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.324+0000 7f7dd8000640 1 -- 172.21.15.102:0/45552896 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd015c080 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.325+0000 7f7dd8000640 1 -- 172.21.15.102:0/45552896 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd0073390 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.326+0000 7f7dcd600640 1 -- 172.21.15.102:0/45552896 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7dc401c460 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.327+0000 7f7dcd600640 1 --2- 172.21.15.102:0/45552896 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7db006ac30 0x7f7db006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:54.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.327+0000 7f7dcd600640 1 -- 172.21.15.102:0/45552896 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7dc4089790 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.327+0000 7f7dd4c00640 1 --2- 172.21.15.102:0/45552896 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7db006ac30 0x7f7db006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:54.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.327+0000 7f7dd4c00640 1 --2- 172.21.15.102:0/45552896 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7db006ac30 0x7f7db006d0f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc036000 tx=0x7f7dbc03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:54.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.331+0000 7f7dcd600640 1 -- 172.21.15.102:0/45552896 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7dc4054520 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.522+0000 7f7dd8000640 1 -- 172.21.15.102:0/45552896 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7dd0156050 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.523+0000 7f7dcd600640 1 -- 172.21.15.102:0/45552896 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7dc4057b40 con 0x7f7dd00ca7b0 2024-03-23T03:47:54.524 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:47:54.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.526+0000 7f7dd8000640 1 -- 172.21.15.102:0/45552896 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7db006ac30 msgr2=0x7f7db006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.526+0000 7f7dd8000640 1 --2- 172.21.15.102:0/45552896 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7db006ac30 0x7f7db006d0f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc036000 tx=0x7f7dbc03b040 comp rx=0 tx=0).stop 2024-03-23T03:47:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.526+0000 7f7dd8000640 1 -- 172.21.15.102:0/45552896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dd00ca7b0 msgr2=0x7f7dd015c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.526+0000 7f7dd8000640 1 --2- 172.21.15.102:0/45552896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dd00ca7b0 0x7f7dd015c5a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400c430 tx=0x7f7dc400c8f0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.526+0000 7f7dd8000640 1 -- 172.21.15.102:0/45552896 shutdown_connections 2024-03-23T03:47:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.526+0000 7f7dd8000640 1 --2- 172.21.15.102:0/45552896 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7db006ac30 0x7f7db006d0f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.526+0000 7f7dd8000640 1 --2- 172.21.15.102:0/45552896 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7dd0159830 0x7f7dd015ac00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.526+0000 7f7dd8000640 1 --2- 172.21.15.102:0/45552896 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dd00ce930 0x7f7dd0142120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.526+0000 7f7dd8000640 1 --2- 172.21.15.102:0/45552896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dd00ca7b0 0x7f7dd015c5a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.526+0000 7f7dd8000640 1 -- 172.21.15.102:0/45552896 >> 172.21.15.102:0/45552896 conn(0x7f7dd006a0b0 msgr2=0x7f7dd0158a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.526+0000 7f7dd8000640 1 -- 172.21.15.102:0/45552896 shutdown_connections 2024-03-23T03:47:54.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.526+0000 7f7dd8000640 1 -- 172.21.15.102:0/45552896 wait complete. 2024-03-23T03:47:54.546 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:47:54.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.757+0000 7f847c800640 1 Processor -- start 2024-03-23T03:47:54.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.758+0000 7f847c800640 1 -- start start 2024-03-23T03:47:54.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.759+0000 7f847c800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f84740ce930 0x7f84740add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:54.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.759+0000 7f847c800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84740cdba0 con 0x7f84740cd3e0 2024-03-23T03:47:54.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.759+0000 7f847c800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84740cde80 con 0x7f847406ce90 2024-03-23T03:47:54.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.759+0000 7f847c800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f847406c760 con 0x7f84740ce930 2024-03-23T03:47:54.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.760+0000 7f8479400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f84740cd3e0 0x7f84740ced10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40970/0 (socket says 172.21.15.102:40970) 2024-03-23T03:47:54.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.760+0000 7f8479400640 1 -- 172.21.15.102:0/2539531198 learned_addr learned my addr 172.21.15.102:0/2539531198 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.760+0000 7f8478a00640 1 -- 172.21.15.102:0/2539531198 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235737859 0 0) 0x7f84740cdba0 con 0x7f84740cd3e0 2024-03-23T03:47:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.760+0000 7f8478a00640 1 -- 172.21.15.102:0/2539531198 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f845c003520 con 0x7f84740cd3e0 2024-03-23T03:47:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.760+0000 7f8479e00640 1 --2- 172.21.15.102:0/2539531198 >> v2:172.21.15.173:3300/0 conn(0x7f84740ce930 0x7f84740add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.760+0000 7f8478a00640 1 -- 172.21.15.102:0/2539531198 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 573739667 0 0) 0x7f845c003520 con 0x7f84740cd3e0 2024-03-23T03:47:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f8478a00640 1 -- 172.21.15.102:0/2539531198 >> v2:172.21.15.173:3300/0 conn(0x7f84740ce930 msgr2=0x7f84740add30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f8478a00640 1 --2- 172.21.15.102:0/2539531198 >> v2:172.21.15.173:3300/0 conn(0x7f84740ce930 0x7f84740add30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f8478a00640 1 -- 172.21.15.102:0/2539531198 >> v1:172.21.15.173:6789/0 conn(0x7f847406ce90 legacy=0x7f847415e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f8478a00640 1 -- 172.21.15.102:0/2539531198 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84740ce160 con 0x7f84740cd3e0 2024-03-23T03:47:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f8478a00640 1 -- 172.21.15.102:0/2539531198 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f8464001670 con 0x7f84740cd3e0 2024-03-23T03:47:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f8478a00640 1 -- 172.21.15.102:0/2539531198 >> v1:172.21.15.102:6789/0 conn(0x7f84740cd3e0 legacy=0x7f84740ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f8478a00640 1 --2- 172.21.15.102:0/2539531198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f845c003970 0x7f845c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f8478a00640 1 --2- 172.21.15.102:0/2539531198 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f845c0047c0 0x7f845c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f8478a00640 1 --2- 172.21.15.102:0/2539531198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f845c005180 0x7f845c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f8478a00640 1 -- 172.21.15.102:0/2539531198 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847406c760 con 0x7f845c003970 2024-03-23T03:47:54.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f8478a00640 1 -- 172.21.15.102:0/2539531198 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c00c0d0 con 0x7f845c005180 2024-03-23T03:47:54.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f8478a00640 1 -- 172.21.15.102:0/2539531198 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845c00c290 con 0x7f845c0047c0 2024-03-23T03:47:54.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f8479e00640 1 --2- 172.21.15.102:0/2539531198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f845c003970 0x7f845c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:54.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f8479e00640 1 -- 172.21.15.102:0/2539531198 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f845c0047c0 msgr2=0x7f845c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:54.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f8479400640 1 --2- 172.21.15.102:0/2539531198 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f845c0047c0 0x7f845c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:54.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f8479e00640 1 --2- 172.21.15.102:0/2539531198 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f845c0047c0 0x7f845c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f8479e00640 1 -- 172.21.15.102:0/2539531198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f845c005180 msgr2=0x7f845c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:54.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.761+0000 7f847a800640 1 --2- 172.21.15.102:0/2539531198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f845c005180 0x7f845c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:54.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.762+0000 7f8479e00640 1 --2- 172.21.15.102:0/2539531198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f845c005180 0x7f845c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.762+0000 7f8479e00640 1 -- 172.21.15.102:0/2539531198 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845c003520 con 0x7f845c003970 2024-03-23T03:47:54.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.762+0000 7f847a800640 1 --2- 172.21.15.102:0/2539531198 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f845c005180 0x7f845c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:54.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.762+0000 7f847a800640 1 -- 172.21.15.102:0/2539531198 reap_dead start 2024-03-23T03:47:54.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.762+0000 7f8479400640 1 --2- 172.21.15.102:0/2539531198 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f845c0047c0 0x7f845c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:54.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.763+0000 7f8479e00640 1 --2- 172.21.15.102:0/2539531198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f845c003970 0x7f845c003d90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f846800ba30 tx=0x7f845c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4af8172e9fbb8b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:54.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.763+0000 7f8478a00640 1 -- 172.21.15.102:0/2539531198 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8468032660 con 0x7f845c003970 2024-03-23T03:47:54.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.763+0000 7f8478a00640 1 -- 172.21.15.102:0/2539531198 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84680327c0 con 0x7f845c003970 2024-03-23T03:47:54.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.763+0000 7f8478a00640 1 -- 172.21.15.102:0/2539531198 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8468032b50 con 0x7f845c003970 2024-03-23T03:47:54.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.764+0000 7f847c800640 1 -- 172.21.15.102:0/2539531198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f845c003970 msgr2=0x7f845c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:54.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.764+0000 7f847c800640 1 --2- 172.21.15.102:0/2539531198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f845c003970 0x7f845c003d90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f846800ba30 tx=0x7f845c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.764+0000 7f847c800640 1 -- 172.21.15.102:0/2539531198 shutdown_connections 2024-03-23T03:47:54.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.764+0000 7f847c800640 1 --2- 172.21.15.102:0/2539531198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f845c003970 0x7f845c003d90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.764+0000 7f847c800640 1 -- 172.21.15.102:0/2539531198 >> 172.21.15.102:0/2539531198 conn(0x7f847406a0b0 msgr2=0x7f847406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:54.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.764+0000 7f847c800640 1 -- 172.21.15.102:0/2539531198 shutdown_connections 2024-03-23T03:47:54.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.764+0000 7f847c800640 1 -- 172.21.15.102:0/2539531198 wait complete. 2024-03-23T03:47:54.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.765+0000 7f847c800640 1 Processor -- start 2024-03-23T03:47:54.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.766+0000 7f847c800640 1 -- start start 2024-03-23T03:47:54.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.766+0000 7f847c800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f845c003970 0x7f847414f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:54.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.766+0000 7f847c800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84740b3200 0x7f8474144010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:54.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.767+0000 7f847c800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f847414dcf0 0x7f847414e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:54.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.767+0000 7f847c800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f847414e6b0 con 0x7f847414dcf0 2024-03-23T03:47:54.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.767+0000 7f847c800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84740b16c0 con 0x7f845c003970 2024-03-23T03:47:54.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.767+0000 7f847c800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8474151f50 con 0x7f84740b3200 2024-03-23T03:47:54.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.767+0000 7f847a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f847414dcf0 0x7f847414e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:54.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.767+0000 7f847a800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f847414dcf0 0x7f847414e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54098/0 (socket says 172.21.15.102:54098) 2024-03-23T03:47:54.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.767+0000 7f847a800640 1 -- 172.21.15.102:0/1669604506 learned_addr learned my addr 172.21.15.102:0/1669604506 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:54.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.767+0000 7f8479400640 1 --2- 172.21.15.102:0/1669604506 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84740b3200 0x7f8474144010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:54.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.767+0000 7f847a800640 1 -- 172.21.15.102:0/1669604506 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84740b3200 msgr2=0x7f8474144010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:54.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.767+0000 7f847a800640 1 --2- 172.21.15.102:0/1669604506 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84740b3200 0x7f8474144010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.767+0000 7f847a800640 1 -- 172.21.15.102:0/1669604506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f845c003970 msgr2=0x7f847414f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:54.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.767+0000 7f847a800640 1 --2- 172.21.15.102:0/1669604506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f845c003970 0x7f847414f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.767+0000 7f847a800640 1 -- 172.21.15.102:0/1669604506 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8474144550 con 0x7f847414dcf0 2024-03-23T03:47:54.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.767+0000 7f8479400640 1 --2- 172.21.15.102:0/1669604506 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84740b3200 0x7f8474144010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:54.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.768+0000 7f847a800640 1 --2- 172.21.15.102:0/1669604506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f847414dcf0 0x7f847414e170 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f84740ccb90 tx=0x7f847000d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:54.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.768+0000 7f846e000640 1 -- 172.21.15.102:0/1669604506 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8470017070 con 0x7f847414dcf0 2024-03-23T03:47:54.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.768+0000 7f846e000640 1 -- 172.21.15.102:0/1669604506 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8470003b80 con 0x7f847414dcf0 2024-03-23T03:47:54.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.768+0000 7f846e000640 1 -- 172.21.15.102:0/1669604506 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8470006b50 con 0x7f847414dcf0 2024-03-23T03:47:54.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.768+0000 7f847c800640 1 -- 172.21.15.102:0/1669604506 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f847414fe20 con 0x7f847414dcf0 2024-03-23T03:47:54.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.768+0000 7f847c800640 1 -- 172.21.15.102:0/1669604506 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84741524f0 con 0x7f847414dcf0 2024-03-23T03:47:54.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.769+0000 7f847c800640 1 -- 172.21.15.102:0/1669604506 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8474073390 con 0x7f847414dcf0 2024-03-23T03:47:54.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.770+0000 7f846e000640 1 -- 172.21.15.102:0/1669604506 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8470006cb0 con 0x7f847414dcf0 2024-03-23T03:47:54.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.771+0000 7f846e000640 1 --2- 172.21.15.102:0/1669604506 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f845806ac00 0x7f845806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:54.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.771+0000 7f846e000640 1 -- 172.21.15.102:0/1669604506 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8470088ea0 con 0x7f847414dcf0 2024-03-23T03:47:54.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.771+0000 7f8479e00640 1 --2- 172.21.15.102:0/1669604506 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f845806ac00 0x7f845806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:54.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.772+0000 7f8479e00640 1 --2- 172.21.15.102:0/1669604506 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f845806ac00 0x7f845806d0c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f846800ba00 tx=0x7f8468032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:54.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.776+0000 7f846e000640 1 -- 172.21.15.102:0/1669604506 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8470053c20 con 0x7f847414dcf0 2024-03-23T03:47:54.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.973+0000 7f847c800640 1 -- 172.21.15.102:0/1669604506 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f84740b1800 con 0x7f847414dcf0 2024-03-23T03:47:54.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.974+0000 7f846e000640 1 -- 172.21.15.102:0/1669604506 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8470057240 con 0x7f847414dcf0 2024-03-23T03:47:54.975 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:54.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.977+0000 7f847c800640 1 -- 172.21.15.102:0/1669604506 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f845806ac00 msgr2=0x7f845806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.977+0000 7f847c800640 1 --2- 172.21.15.102:0/1669604506 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f845806ac00 0x7f845806d0c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f846800ba00 tx=0x7f8468032000 comp rx=0 tx=0).stop 2024-03-23T03:47:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.977+0000 7f847c800640 1 -- 172.21.15.102:0/1669604506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f847414dcf0 msgr2=0x7f847414e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.977+0000 7f847c800640 1 --2- 172.21.15.102:0/1669604506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f847414dcf0 0x7f847414e170 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f84740ccb90 tx=0x7f847000d9d0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.977+0000 7f847c800640 1 -- 172.21.15.102:0/1669604506 shutdown_connections 2024-03-23T03:47:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.977+0000 7f847c800640 1 --2- 172.21.15.102:0/1669604506 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f845806ac00 0x7f845806d0c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.977+0000 7f847c800640 1 --2- 172.21.15.102:0/1669604506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f847414dcf0 0x7f847414e170 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.977+0000 7f847c800640 1 --2- 172.21.15.102:0/1669604506 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84740b3200 0x7f8474144010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.977+0000 7f847c800640 1 --2- 172.21.15.102:0/1669604506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f845c003970 0x7f847414f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.977+0000 7f847c800640 1 -- 172.21.15.102:0/1669604506 >> 172.21.15.102:0/1669604506 conn(0x7f847406a0b0 msgr2=0x7f84740cb470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.977+0000 7f847c800640 1 -- 172.21.15.102:0/1669604506 shutdown_connections 2024-03-23T03:47:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:54.977+0000 7f847c800640 1 -- 172.21.15.102:0/1669604506 wait complete. 2024-03-23T03:47:54.995 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:47:55.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.205+0000 7fc3fde00640 1 Processor -- start 2024-03-23T03:47:55.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.205+0000 7fc3fde00640 1 -- start start 2024-03-23T03:47:55.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.207+0000 7fc3fde00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc3f8159830 0x7fc3f80cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:55.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.207+0000 7fc3fde00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc3f80ca7b0 0x7fc3f80cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:55.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.207+0000 7fc3fde00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f8159c10 con 0x7fc3f80ce930 2024-03-23T03:47:55.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.207+0000 7fc3fde00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f806de50 con 0x7fc3f8159830 2024-03-23T03:47:55.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.207+0000 7fc3fde00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f80cd840 con 0x7fc3f80ca7b0 2024-03-23T03:47:55.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.207+0000 7fc3f6a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc3f8159830 0x7fc3f80cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:55.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.207+0000 7fc3f7400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc3f80ce930 0x7fc3f80b18d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40974/0 (socket says 172.21.15.102:40974) 2024-03-23T03:47:55.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.207+0000 7fc3f7400640 1 -- 172.21.15.102:0/3069377509 learned_addr learned my addr 172.21.15.102:0/3069377509 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:55.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.208+0000 7fc3f6a00640 1 -- 172.21.15.102:0/3069377509 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc3f80ca7b0 msgr2=0x7fc3f80cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:55.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.208+0000 7fc3f6000640 1 --2- 172.21.15.102:0/3069377509 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc3f80ca7b0 0x7fc3f80cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:55.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.208+0000 7fc3f6a00640 1 --2- 172.21.15.102:0/3069377509 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc3f80ca7b0 0x7fc3f80cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.208+0000 7fc3f6a00640 1 -- 172.21.15.102:0/3069377509 >> v1:172.21.15.102:6789/0 conn(0x7fc3f80ce930 legacy=0x7fc3f80b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:55.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.208+0000 7fc3f6a00640 1 -- 172.21.15.102:0/3069377509 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f80ccc30 con 0x7fc3f8159830 2024-03-23T03:47:55.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.208+0000 7fc3f6000640 1 --2- 172.21.15.102:0/3069377509 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc3f80ca7b0 0x7fc3f80cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:55.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.208+0000 7fc3f5600640 1 -- 172.21.15.102:0/3069377509 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566163516 0 0) 0x7fc3f8159c10 con 0x7fc3f80ce930 2024-03-23T03:47:55.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.209+0000 7fc3f6a00640 1 --2- 172.21.15.102:0/3069377509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3f8159830 0x7fc3f80cc3c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80097a0 tx=0x7fc3e802b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cbcb09cbcec780c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.209+0000 7fc3f5600640 1 -- 172.21.15.102:0/3069377509 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8030800 con 0x7fc3f8159830 2024-03-23T03:47:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.209+0000 7fc3f5600640 1 -- 172.21.15.102:0/3069377509 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e8030960 con 0x7fc3f8159830 2024-03-23T03:47:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.209+0000 7fc3f5600640 1 -- 172.21.15.102:0/3069377509 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8030cf0 con 0x7fc3f8159830 2024-03-23T03:47:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.210+0000 7fc3fde00640 1 -- 172.21.15.102:0/3069377509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3f8159830 msgr2=0x7fc3f80cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.210+0000 7fc3fde00640 1 --2- 172.21.15.102:0/3069377509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3f8159830 0x7fc3f80cc3c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80097a0 tx=0x7fc3e802b940 comp rx=0 tx=0).stop 2024-03-23T03:47:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.210+0000 7fc3fde00640 1 -- 172.21.15.102:0/3069377509 shutdown_connections 2024-03-23T03:47:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.210+0000 7fc3fde00640 1 --2- 172.21.15.102:0/3069377509 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc3f80ca7b0 0x7fc3f80cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.210+0000 7fc3fde00640 1 --2- 172.21.15.102:0/3069377509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3f8159830 0x7fc3f80cc3c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.210+0000 7fc3fde00640 1 -- 172.21.15.102:0/3069377509 >> 172.21.15.102:0/3069377509 conn(0x7fc3f806a0b0 msgr2=0x7fc3f806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.210+0000 7fc3fde00640 1 -- 172.21.15.102:0/3069377509 shutdown_connections 2024-03-23T03:47:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.210+0000 7fc3fde00640 1 -- 172.21.15.102:0/3069377509 wait complete. 2024-03-23T03:47:55.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.211+0000 7fc3fde00640 1 Processor -- start 2024-03-23T03:47:55.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.212+0000 7fc3fde00640 1 -- start start 2024-03-23T03:47:55.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.212+0000 7fc3fde00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc3f80ca7b0 0x7fc3f8140d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:55.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.212+0000 7fc3fde00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3f80ce930 0x7fc3f813da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:55.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.212+0000 7fc3fde00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3f8159830 0x7fc3f8159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.212+0000 7fc3fde00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f815a300 con 0x7fc3f80ce930 2024-03-23T03:47:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.212+0000 7fc3fde00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f815c550 con 0x7fc3f8159830 2024-03-23T03:47:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.212+0000 7fc3fde00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f80b0610 con 0x7fc3f80ca7b0 2024-03-23T03:47:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.213+0000 7fc3f6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3f80ce930 0x7fc3f813da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.213+0000 7fc3f6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3f80ce930 0x7fc3f813da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54114/0 (socket says 172.21.15.102:54114) 2024-03-23T03:47:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.213+0000 7fc3f6000640 1 -- 172.21.15.102:0/246794965 learned_addr learned my addr 172.21.15.102:0/246794965 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.213+0000 7fc3f6a00640 1 --2- 172.21.15.102:0/246794965 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc3f80ca7b0 0x7fc3f8140d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.213+0000 7fc3f6000640 1 -- 172.21.15.102:0/246794965 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc3f80ca7b0 msgr2=0x7fc3f8140d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.213+0000 7fc3f6000640 1 --2- 172.21.15.102:0/246794965 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc3f80ca7b0 0x7fc3f8140d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.213+0000 7fc3f6000640 1 -- 172.21.15.102:0/246794965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3f8159830 msgr2=0x7fc3f8159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.213+0000 7fc3f6000640 1 --2- 172.21.15.102:0/246794965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3f8159830 0x7fc3f8159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.213+0000 7fc3f6000640 1 -- 172.21.15.102:0/246794965 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e8009400 con 0x7fc3f80ce930 2024-03-23T03:47:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.214+0000 7fc3f6000640 1 --2- 172.21.15.102:0/246794965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3f80ce930 0x7fc3f813da60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400c7f0 tx=0x7fc3e400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.214+0000 7fc3ef400640 1 -- 172.21.15.102:0/246794965 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4007820 con 0x7fc3f80ce930 2024-03-23T03:47:55.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.214+0000 7fc3ef400640 1 -- 172.21.15.102:0/246794965 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e4007980 con 0x7fc3f80ce930 2024-03-23T03:47:55.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.214+0000 7fc3ef400640 1 -- 172.21.15.102:0/246794965 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4007cf0 con 0x7fc3f80ce930 2024-03-23T03:47:55.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.214+0000 7fc3fde00640 1 -- 172.21.15.102:0/246794965 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f815bc20 con 0x7fc3f80ce930 2024-03-23T03:47:55.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.214+0000 7fc3fde00640 1 -- 172.21.15.102:0/246794965 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f815c1e0 con 0x7fc3f80ce930 2024-03-23T03:47:55.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.215+0000 7fc3fde00640 1 -- 172.21.15.102:0/246794965 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f815c700 con 0x7fc3f80ce930 2024-03-23T03:47:55.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.216+0000 7fc3ef400640 1 -- 172.21.15.102:0/246794965 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc3e40113f0 con 0x7fc3f80ce930 2024-03-23T03:47:55.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.217+0000 7fc3ef400640 1 --2- 172.21.15.102:0/246794965 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc3d806ac30 0x7fc3d806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:55.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.217+0000 7fc3ef400640 1 -- 172.21.15.102:0/246794965 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc3e4089a00 con 0x7fc3f80ce930 2024-03-23T03:47:55.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.217+0000 7fc3f6a00640 1 --2- 172.21.15.102:0/246794965 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc3d806ac30 0x7fc3d806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:55.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.218+0000 7fc3f6a00640 1 --2- 172.21.15.102:0/246794965 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc3d806ac30 0x7fc3d806d0f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8036000 tx=0x7fc3e8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:55.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.224+0000 7fc3ef400640 1 -- 172.21.15.102:0/246794965 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc3f815c700 con 0x7fc3f80ce930 2024-03-23T03:47:55.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.419+0000 7fc3fde00640 1 -- 172.21.15.102:0/246794965 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc3f815c6c0 con 0x7fc3f80ce930 2024-03-23T03:47:55.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.420+0000 7fc3ef400640 1 -- 172.21.15.102:0/246794965 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc3e4054810 con 0x7fc3f80ce930 2024-03-23T03:47:55.421 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:47:55.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.423+0000 7fc3fde00640 1 -- 172.21.15.102:0/246794965 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc3d806ac30 msgr2=0x7fc3d806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:55.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.423+0000 7fc3fde00640 1 --2- 172.21.15.102:0/246794965 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc3d806ac30 0x7fc3d806d0f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8036000 tx=0x7fc3e8008030 comp rx=0 tx=0).stop 2024-03-23T03:47:55.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.423+0000 7fc3fde00640 1 -- 172.21.15.102:0/246794965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3f80ce930 msgr2=0x7fc3f813da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:55.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.423+0000 7fc3fde00640 1 --2- 172.21.15.102:0/246794965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3f80ce930 0x7fc3f813da60 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400c7f0 tx=0x7fc3e400ccb0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.424+0000 7fc3fde00640 1 -- 172.21.15.102:0/246794965 shutdown_connections 2024-03-23T03:47:55.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.424+0000 7fc3fde00640 1 --2- 172.21.15.102:0/246794965 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc3d806ac30 0x7fc3d806d0f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.424+0000 7fc3fde00640 1 --2- 172.21.15.102:0/246794965 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc3f8159830 0x7fc3f8159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.424+0000 7fc3fde00640 1 --2- 172.21.15.102:0/246794965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3f80ce930 0x7fc3f813da60 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.424+0000 7fc3fde00640 1 --2- 172.21.15.102:0/246794965 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc3f80ca7b0 0x7fc3f8140d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.424+0000 7fc3fde00640 1 -- 172.21.15.102:0/246794965 >> 172.21.15.102:0/246794965 conn(0x7fc3f806a0b0 msgr2=0x7fc3f8158a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:55.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.424+0000 7fc3fde00640 1 -- 172.21.15.102:0/246794965 shutdown_connections 2024-03-23T03:47:55.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.424+0000 7fc3fde00640 1 -- 172.21.15.102:0/246794965 wait complete. 2024-03-23T03:47:55.443 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-23T03:47:55.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.640+0000 7f8776e00640 1 Processor -- start 2024-03-23T03:47:55.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.641+0000 7f8776e00640 1 -- start start 2024-03-23T03:47:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.642+0000 7f8776e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87700d0d40 0x7f87700b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.642+0000 7f8776e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87700d20c0 con 0x7f87700cd740 2024-03-23T03:47:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.642+0000 7f8776e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87700b8940 con 0x7f87700b84b0 2024-03-23T03:47:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.642+0000 7f8776e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87700d14b0 con 0x7f87700d0d40 2024-03-23T03:47:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.642+0000 7f876fe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f87700cd740 0x7f87700cdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40976/0 (socket says 172.21.15.102:40976) 2024-03-23T03:47:55.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.642+0000 7f876fe00640 1 -- 172.21.15.102:0/2862107092 learned_addr learned my addr 172.21.15.102:0/2862107092 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.643+0000 7f876ea00640 1 -- 172.21.15.102:0/2862107092 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876214441 0 0) 0x7f87700d20c0 con 0x7f87700cd740 2024-03-23T03:47:55.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.643+0000 7f876ea00640 1 -- 172.21.15.102:0/2862107092 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8754003520 con 0x7f87700cd740 2024-03-23T03:47:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.643+0000 7f8774e00640 1 --2- 172.21.15.102:0/2862107092 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87700d0d40 0x7f87700b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.643+0000 7f876ea00640 1 -- 172.21.15.102:0/2862107092 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 115997873 0 0) 0x7f8754003520 con 0x7f87700cd740 2024-03-23T03:47:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.643+0000 7f876ea00640 1 -- 172.21.15.102:0/2862107092 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87700d0d40 msgr2=0x7f87700b61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.643+0000 7f876ea00640 1 --2- 172.21.15.102:0/2862107092 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87700d0d40 0x7f87700b61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.643+0000 7f876ea00640 1 -- 172.21.15.102:0/2862107092 >> v1:172.21.15.173:6789/0 conn(0x7f87700b84b0 legacy=0x7f87700b5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.643+0000 7f876ea00640 1 -- 172.21.15.102:0/2862107092 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87700d1790 con 0x7f87700cd740 2024-03-23T03:47:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.643+0000 7f876ea00640 1 -- 172.21.15.102:0/2862107092 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f87600019f0 con 0x7f87700cd740 2024-03-23T03:47:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.643+0000 7f876ea00640 1 -- 172.21.15.102:0/2862107092 >> v1:172.21.15.102:6789/0 conn(0x7f87700cd740 legacy=0x7f87700cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.643+0000 7f876ea00640 1 --2- 172.21.15.102:0/2862107092 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87540039a0 0x7f8754003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.643+0000 7f876ea00640 1 --2- 172.21.15.102:0/2862107092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87540047c0 0x7f8754004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.643+0000 7f876ea00640 1 --2- 172.21.15.102:0/2862107092 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8754005180 0x7f875400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:55.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.643+0000 7f876ea00640 1 -- 172.21.15.102:0/2862107092 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87700d14b0 con 0x7f87540039a0 2024-03-23T03:47:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.643+0000 7f876ea00640 1 -- 172.21.15.102:0/2862107092 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875400c020 con 0x7f87540047c0 2024-03-23T03:47:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.643+0000 7f876ea00640 1 -- 172.21.15.102:0/2862107092 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f875400c1e0 con 0x7f8754005180 2024-03-23T03:47:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.643+0000 7f8774e00640 1 --2- 172.21.15.102:0/2862107092 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87700d0d40 0x7f87700b61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:47:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.644+0000 7f876fe00640 1 --2- 172.21.15.102:0/2862107092 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87540039a0 0x7f8754003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.644+0000 7f876f400640 1 --2- 172.21.15.102:0/2862107092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87540047c0 0x7f8754004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.644+0000 7f876fe00640 1 -- 172.21.15.102:0/2862107092 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8754005180 msgr2=0x7f875400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.644+0000 7f876fe00640 1 --2- 172.21.15.102:0/2862107092 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8754005180 0x7f875400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.644+0000 7f876fe00640 1 -- 172.21.15.102:0/2862107092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87540047c0 msgr2=0x7f8754004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.644+0000 7f876fe00640 1 --2- 172.21.15.102:0/2862107092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87540047c0 0x7f8754004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.644+0000 7f876fe00640 1 -- 172.21.15.102:0/2862107092 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8754003520 con 0x7f87540039a0 2024-03-23T03:47:55.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.644+0000 7f876f400640 1 --2- 172.21.15.102:0/2862107092 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87540047c0 0x7f8754004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:55.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.645+0000 7f876fe00640 1 --2- 172.21.15.102:0/2862107092 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87540039a0 0x7f8754003dc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f876000a830 tx=0x7f87540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ba4e9616119bf49 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.645+0000 7f876ea00640 1 -- 172.21.15.102:0/2862107092 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87600072d0 con 0x7f87540039a0 2024-03-23T03:47:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.645+0000 7f876ea00640 1 -- 172.21.15.102:0/2862107092 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f876002f070 con 0x7f87540039a0 2024-03-23T03:47:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.645+0000 7f876ea00640 1 -- 172.21.15.102:0/2862107092 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87600076b0 con 0x7f87540039a0 2024-03-23T03:47:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.646+0000 7f8776e00640 1 -- 172.21.15.102:0/2862107092 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87540039a0 msgr2=0x7f8754003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.646+0000 7f8776e00640 1 --2- 172.21.15.102:0/2862107092 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87540039a0 0x7f8754003dc0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f876000a830 tx=0x7f87540057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.646+0000 7f8774e00640 1 -- 172.21.15.102:0/2862107092 reap_dead start 2024-03-23T03:47:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.646+0000 7f8776e00640 1 -- 172.21.15.102:0/2862107092 shutdown_connections 2024-03-23T03:47:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.646+0000 7f8776e00640 1 -- 172.21.15.102:0/2862107092 >> 172.21.15.102:0/2862107092 conn(0x7f87700cf5d0 msgr2=0x7f87700cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.646+0000 7f8776e00640 1 -- 172.21.15.102:0/2862107092 shutdown_connections 2024-03-23T03:47:55.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.646+0000 7f8776e00640 1 -- 172.21.15.102:0/2862107092 wait complete. 2024-03-23T03:47:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.648+0000 7f8776e00640 1 Processor -- start 2024-03-23T03:47:55.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.648+0000 7f8776e00640 1 -- start start 2024-03-23T03:47:55.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.649+0000 7f8776e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87700b7310 0x7f87700b0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:55.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.649+0000 7f8776e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87700ac910 0x7f877014fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:55.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.649+0000 7f8776e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87700b76f0 0x7f877014ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:55.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.649+0000 7f8776e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87700b10a0 con 0x7f87700ac910 2024-03-23T03:47:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.649+0000 7f8776e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87700b0dc0 con 0x7f87700b7310 2024-03-23T03:47:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.649+0000 7f8776e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8770150960 con 0x7f87700b76f0 2024-03-23T03:47:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.649+0000 7f876f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87700ac910 0x7f877014fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.649+0000 7f876f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87700ac910 0x7f877014fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54124/0 (socket says 172.21.15.102:54124) 2024-03-23T03:47:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.649+0000 7f876f400640 1 -- 172.21.15.102:0/1924443776 learned_addr learned my addr 172.21.15.102:0/1924443776 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.650+0000 7f876f400640 1 -- 172.21.15.102:0/1924443776 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87700b76f0 msgr2=0x7f877014ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.650+0000 7f876f400640 1 --2- 172.21.15.102:0/1924443776 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87700b76f0 0x7f877014ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.650+0000 7f876f400640 1 -- 172.21.15.102:0/1924443776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87700b7310 msgr2=0x7f87700b0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.650+0000 7f876f400640 1 --2- 172.21.15.102:0/1924443776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87700b7310 0x7f87700b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.650+0000 7f876f400640 1 -- 172.21.15.102:0/1924443776 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8770150c40 con 0x7f87700ac910 2024-03-23T03:47:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.650+0000 7f876f400640 1 --2- 172.21.15.102:0/1924443776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87700ac910 0x7f877014fa20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f875800d7a0 tx=0x7f875800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.650+0000 7f8767e00640 1 -- 172.21.15.102:0/1924443776 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758017070 con 0x7f87700ac910 2024-03-23T03:47:55.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.650+0000 7f8767e00640 1 -- 172.21.15.102:0/1924443776 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8758005070 con 0x7f87700ac910 2024-03-23T03:47:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.650+0000 7f8767e00640 1 -- 172.21.15.102:0/1924443776 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758005420 con 0x7f87700ac910 2024-03-23T03:47:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.650+0000 7f8776e00640 1 -- 172.21.15.102:0/1924443776 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8770150f20 con 0x7f87700ac910 2024-03-23T03:47:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.650+0000 7f8776e00640 1 -- 172.21.15.102:0/1924443776 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87700b0b70 con 0x7f87700ac910 2024-03-23T03:47:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.651+0000 7f8776e00640 1 -- 172.21.15.102:0/1924443776 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87701510b0 con 0x7f87700ac910 2024-03-23T03:47:55.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.652+0000 7f8767e00640 1 -- 172.21.15.102:0/1924443776 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8758007030 con 0x7f87700ac910 2024-03-23T03:47:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.653+0000 7f8767e00640 1 --2- 172.21.15.102:0/1924443776 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f874c06ac00 0x7f874c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.653+0000 7f8767e00640 1 -- 172.21.15.102:0/1924443776 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8758088fe0 con 0x7f87700ac910 2024-03-23T03:47:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.653+0000 7f876fe00640 1 --2- 172.21.15.102:0/1924443776 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f874c06ac00 0x7f874c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:55.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.654+0000 7f876fe00640 1 --2- 172.21.15.102:0/1924443776 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f874c06ac00 0x7f874c06d0c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8760034000 tx=0x7f876002f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:55.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.657+0000 7f8767e00640 1 -- 172.21.15.102:0/1924443776 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8758053e00 con 0x7f87700ac910 2024-03-23T03:47:55.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.853+0000 7f8776e00640 1 -- 172.21.15.102:0/1924443776 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8770073390 con 0x7f87700ac910 2024-03-23T03:47:55.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.854+0000 7f8767e00640 1 -- 172.21.15.102:0/1924443776 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8758057420 con 0x7f87700ac910 2024-03-23T03:47:55.855 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:55.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.856+0000 7f8776e00640 1 -- 172.21.15.102:0/1924443776 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f874c06ac00 msgr2=0x7f874c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:55.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.856+0000 7f8776e00640 1 --2- 172.21.15.102:0/1924443776 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f874c06ac00 0x7f874c06d0c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8760034000 tx=0x7f876002f000 comp rx=0 tx=0).stop 2024-03-23T03:47:55.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.856+0000 7f8776e00640 1 -- 172.21.15.102:0/1924443776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87700ac910 msgr2=0x7f877014fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:55.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.856+0000 7f8776e00640 1 --2- 172.21.15.102:0/1924443776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87700ac910 0x7f877014fa20 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f875800d7a0 tx=0x7f875800dc60 comp rx=0 tx=0).stop 2024-03-23T03:47:55.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.856+0000 7f8776e00640 1 -- 172.21.15.102:0/1924443776 shutdown_connections 2024-03-23T03:47:55.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.856+0000 7f8776e00640 1 --2- 172.21.15.102:0/1924443776 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f874c06ac00 0x7f874c06d0c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.856+0000 7f8776e00640 1 --2- 172.21.15.102:0/1924443776 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87700b76f0 0x7f877014ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.856+0000 7f8776e00640 1 --2- 172.21.15.102:0/1924443776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87700ac910 0x7f877014fa20 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.856+0000 7f8776e00640 1 --2- 172.21.15.102:0/1924443776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f87700b7310 0x7f87700b0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:55.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.856+0000 7f8776e00640 1 -- 172.21.15.102:0/1924443776 >> 172.21.15.102:0/1924443776 conn(0x7f87700cf5d0 msgr2=0x7f87700c8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:55.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.857+0000 7f8776e00640 1 -- 172.21.15.102:0/1924443776 shutdown_connections 2024-03-23T03:47:55.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:55.857+0000 7f8776e00640 1 -- 172.21.15.102:0/1924443776 wait complete. 2024-03-23T03:47:55.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:47:56.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.077+0000 7f8d38c00640 1 Processor -- start 2024-03-23T03:47:56.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.077+0000 7f8d38c00640 1 -- start start 2024-03-23T03:47:56.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.078+0000 7f8d38c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f8d340cb660 0x7f8d340cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:56.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.078+0000 7f8d38c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d3406c2b0 0x7f8d3406c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:56.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.078+0000 7f8d38c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3415ae60 con 0x7f8d340ce930 2024-03-23T03:47:56.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.078+0000 7f8d38c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d3415bae0 con 0x7f8d340cb660 2024-03-23T03:47:56.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.078+0000 7f8d38c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3415b140 con 0x7f8d3406c2b0 2024-03-23T03:47:56.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.079+0000 7f8d32800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d3406c2b0 0x7f8d3406c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:56.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.079+0000 7f8d32800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d3406c2b0 0x7f8d3406c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:35786/0 (socket says 172.21.15.102:35786) 2024-03-23T03:47:56.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.079+0000 7f8d32800640 1 -- 172.21.15.102:0/3352521961 learned_addr learned my addr 172.21.15.102:0/3352521961 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:56.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.079+0000 7f8d31400640 1 --2- 172.21.15.102:0/3352521961 >> v2:172.21.15.173:3300/0 conn(0x7f8d340cb660 0x7f8d340cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:56.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.080+0000 7f8d32800640 1 -- 172.21.15.102:0/3352521961 >> v2:172.21.15.173:3300/0 conn(0x7f8d340cb660 msgr2=0x7f8d340cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:56.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.080+0000 7f8d32800640 1 --2- 172.21.15.102:0/3352521961 >> v2:172.21.15.173:3300/0 conn(0x7f8d340cb660 0x7f8d340cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:56.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.080+0000 7f8d32800640 1 -- 172.21.15.102:0/3352521961 >> v1:172.21.15.173:6789/0 conn(0x7f8d340ce930 legacy=0x7f8d340ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:56.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.080+0000 7f8d32800640 1 -- 172.21.15.102:0/3352521961 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3415b420 con 0x7f8d3406c2b0 2024-03-23T03:47:56.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.081+0000 7f8d32800640 1 --2- 172.21.15.102:0/3352521961 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d3406c2b0 0x7f8d3406c740 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c009870 tx=0x7f8d2c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6131f15026ed5d11 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:56.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.082+0000 7f8d30a00640 1 -- 172.21.15.102:0/3352521961 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c030840 con 0x7f8d3406c2b0 2024-03-23T03:47:56.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.082+0000 7f8d30a00640 1 -- 172.21.15.102:0/3352521961 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2c0309a0 con 0x7f8d3406c2b0 2024-03-23T03:47:56.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.082+0000 7f8d30a00640 1 -- 172.21.15.102:0/3352521961 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c030d30 con 0x7f8d3406c2b0 2024-03-23T03:47:56.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.082+0000 7f8d38c00640 1 -- 172.21.15.102:0/3352521961 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d3406c2b0 msgr2=0x7f8d3406c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.082+0000 7f8d38c00640 1 --2- 172.21.15.102:0/3352521961 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d3406c2b0 0x7f8d3406c740 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c009870 tx=0x7f8d2c02b980 comp rx=0 tx=0).stop 2024-03-23T03:47:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.083+0000 7f8d38c00640 1 -- 172.21.15.102:0/3352521961 shutdown_connections 2024-03-23T03:47:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.083+0000 7f8d38c00640 1 --2- 172.21.15.102:0/3352521961 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d3406c2b0 0x7f8d3406c740 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.083+0000 7f8d38c00640 1 --2- 172.21.15.102:0/3352521961 >> v2:172.21.15.173:3300/0 conn(0x7f8d340cb660 0x7f8d340cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.083+0000 7f8d38c00640 1 -- 172.21.15.102:0/3352521961 >> 172.21.15.102:0/3352521961 conn(0x7f8d3406a0b0 msgr2=0x7f8d3406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.083+0000 7f8d38c00640 1 -- 172.21.15.102:0/3352521961 shutdown_connections 2024-03-23T03:47:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.083+0000 7f8d38c00640 1 -- 172.21.15.102:0/3352521961 wait complete. 2024-03-23T03:47:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.084+0000 7f8d38c00640 1 Processor -- start 2024-03-23T03:47:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.084+0000 7f8d38c00640 1 -- start start 2024-03-23T03:47:56.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.085+0000 7f8d38c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d3406c2b0 0x7f8d34157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:56.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.085+0000 7f8d38c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d340cb660 0x7f8d34157c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:56.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.085+0000 7f8d38c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d340ce930 0x7f8d3414dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:56.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.085+0000 7f8d38c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d34155410 con 0x7f8d3406c2b0 2024-03-23T03:47:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.085+0000 7f8d38c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d341556f0 con 0x7f8d340ce930 2024-03-23T03:47:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.085+0000 7f8d38c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3414eb50 con 0x7f8d340cb660 2024-03-23T03:47:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.086+0000 7f8d31e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d3406c2b0 0x7f8d34157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.086+0000 7f8d31e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d3406c2b0 0x7f8d34157710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54132/0 (socket says 172.21.15.102:54132) 2024-03-23T03:47:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.086+0000 7f8d31e00640 1 -- 172.21.15.102:0/993183373 learned_addr learned my addr 172.21.15.102:0/993183373 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.086+0000 7f8d31400640 1 --2- 172.21.15.102:0/993183373 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d340cb660 0x7f8d34157c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.086+0000 7f8d31e00640 1 -- 172.21.15.102:0/993183373 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d340cb660 msgr2=0x7f8d34157c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.086+0000 7f8d31e00640 1 --2- 172.21.15.102:0/993183373 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d340cb660 0x7f8d34157c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.086+0000 7f8d31e00640 1 -- 172.21.15.102:0/993183373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d340ce930 msgr2=0x7f8d3414dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.086+0000 7f8d32800640 1 --2- 172.21.15.102:0/993183373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d340ce930 0x7f8d3414dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.086+0000 7f8d31e00640 1 --2- 172.21.15.102:0/993183373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d340ce930 0x7f8d3414dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.086+0000 7f8d31e00640 1 -- 172.21.15.102:0/993183373 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c0094d0 con 0x7f8d3406c2b0 2024-03-23T03:47:56.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.086+0000 7f8d31e00640 1 --2- 172.21.15.102:0/993183373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d3406c2b0 0x7f8d34157710 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400c790 tx=0x7f8d2400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:56.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.087+0000 7f8d2a000640 1 -- 172.21.15.102:0/993183373 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2400cea0 con 0x7f8d3406c2b0 2024-03-23T03:47:56.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.087+0000 7f8d2a000640 1 -- 172.21.15.102:0/993183373 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d24006d50 con 0x7f8d3406c2b0 2024-03-23T03:47:56.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.087+0000 7f8d38c00640 1 -- 172.21.15.102:0/993183373 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d34155280 con 0x7f8d3406c2b0 2024-03-23T03:47:56.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.087+0000 7f8d2a000640 1 -- 172.21.15.102:0/993183373 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d24003bb0 con 0x7f8d3406c2b0 2024-03-23T03:47:56.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.087+0000 7f8d38c00640 1 -- 172.21.15.102:0/993183373 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3414f110 con 0x7f8d3406c2b0 2024-03-23T03:47:56.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.088+0000 7f8d38c00640 1 -- 172.21.15.102:0/993183373 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d3414f2a0 con 0x7f8d3406c2b0 2024-03-23T03:47:56.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.089+0000 7f8d2a000640 1 -- 172.21.15.102:0/993183373 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8d24003d10 con 0x7f8d3406c2b0 2024-03-23T03:47:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.090+0000 7f8d2a000640 1 --2- 172.21.15.102:0/993183373 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d1406ac00 0x7f8d1406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.090+0000 7f8d2a000640 1 -- 172.21.15.102:0/993183373 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8d24089350 con 0x7f8d3406c2b0 2024-03-23T03:47:56.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.090+0000 7f8d31400640 1 --2- 172.21.15.102:0/993183373 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d1406ac00 0x7f8d1406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:56.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.091+0000 7f8d31400640 1 --2- 172.21.15.102:0/993183373 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d1406ac00 0x7f8d1406d0c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8d341576c0 tx=0x7f8d20008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:56.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.094+0000 7f8d2a000640 1 -- 172.21.15.102:0/993183373 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8d24057bf0 con 0x7f8d3406c2b0 2024-03-23T03:47:56.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.285+0000 7f8d38c00640 1 -- 172.21.15.102:0/993183373 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8d34073390 con 0x7f8d3406c2b0 2024-03-23T03:47:56.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.286+0000 7f8d2a000640 1 -- 172.21.15.102:0/993183373 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8d24057780 con 0x7f8d3406c2b0 2024-03-23T03:47:56.287 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:47:56.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.289+0000 7f8d38c00640 1 -- 172.21.15.102:0/993183373 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d1406ac00 msgr2=0x7f8d1406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:56.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.289+0000 7f8d38c00640 1 --2- 172.21.15.102:0/993183373 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d1406ac00 0x7f8d1406d0c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8d341576c0 tx=0x7f8d20008040 comp rx=0 tx=0).stop 2024-03-23T03:47:56.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.289+0000 7f8d38c00640 1 -- 172.21.15.102:0/993183373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d3406c2b0 msgr2=0x7f8d34157710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:56.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.289+0000 7f8d38c00640 1 --2- 172.21.15.102:0/993183373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d3406c2b0 0x7f8d34157710 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8d2400c790 tx=0x7f8d2400cc50 comp rx=0 tx=0).stop 2024-03-23T03:47:56.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.289+0000 7f8d38c00640 1 -- 172.21.15.102:0/993183373 shutdown_connections 2024-03-23T03:47:56.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.289+0000 7f8d38c00640 1 --2- 172.21.15.102:0/993183373 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d1406ac00 0x7f8d1406d0c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:56.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.289+0000 7f8d38c00640 1 --2- 172.21.15.102:0/993183373 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d340ce930 0x7f8d3414dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:56.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.289+0000 7f8d38c00640 1 --2- 172.21.15.102:0/993183373 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d340cb660 0x7f8d34157c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:56.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.289+0000 7f8d38c00640 1 --2- 172.21.15.102:0/993183373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d3406c2b0 0x7f8d34157710 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:56.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.289+0000 7f8d38c00640 1 -- 172.21.15.102:0/993183373 >> 172.21.15.102:0/993183373 conn(0x7f8d3406a0b0 msgr2=0x7f8d3406a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:56.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.289+0000 7f8d38c00640 1 -- 172.21.15.102:0/993183373 shutdown_connections 2024-03-23T03:47:56.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.289+0000 7f8d38c00640 1 -- 172.21.15.102:0/993183373 wait complete. 2024-03-23T03:47:56.309 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-23T03:47:56.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:56.309 DEBUG:teuthology.orchestra.run.smithi102:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2024-03-23T03:47:56.397 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:47:56.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:56.398 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T03:47:56.442 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:47:56.442 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:47:56.500 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:47:56.500 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:47:56.555 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:47:56.555 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:47:56.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:56.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.825+0000 7fc8f3200640 1 Processor -- start 2024-03-23T03:47:56.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.825+0000 7fc8f3200640 1 -- start start 2024-03-23T03:47:56.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.827+0000 7fc8f3200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc8ec0b39b0 0x7fc8ec0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:56.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.827+0000 7fc8f3200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8ec06c6f0 con 0x7fc8ec15b560 2024-03-23T03:47:56.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.827+0000 7fc8f3200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8ec06c9d0 con 0x7fc8ec15c400 2024-03-23T03:47:56.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.827+0000 7fc8f3200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8ec0b4370 con 0x7fc8ec0b39b0 2024-03-23T03:47:56.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.827+0000 7fc8ebe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc8ec15b560 0x7fc8ec15b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:40988/0 (socket says 172.21.15.102:40988) 2024-03-23T03:47:56.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.827+0000 7fc8ebe00640 1 -- 172.21.15.102:0/3349253876 learned_addr learned my addr 172.21.15.102:0/3349253876 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:56.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.827+0000 7fc8eaa00640 1 -- 172.21.15.102:0/3349253876 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395457175 0 0) 0x7fc8ec06c6f0 con 0x7fc8ec15b560 2024-03-23T03:47:56.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.827+0000 7fc8eaa00640 1 -- 172.21.15.102:0/3349253876 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8d0003520 con 0x7fc8ec15b560 2024-03-23T03:47:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.827+0000 7fc8eb400640 1 --2- 172.21.15.102:0/3349253876 >> v2:172.21.15.173:3300/0 conn(0x7fc8ec0b39b0 0x7fc8ec0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.828+0000 7fc8eaa00640 1 -- 172.21.15.102:0/3349253876 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4260694059 0 0) 0x7fc8d0003520 con 0x7fc8ec15b560 2024-03-23T03:47:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.828+0000 7fc8eaa00640 1 -- 172.21.15.102:0/3349253876 >> v2:172.21.15.173:3300/0 conn(0x7fc8ec0b39b0 msgr2=0x7fc8ec0b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.828+0000 7fc8eaa00640 1 --2- 172.21.15.102:0/3349253876 >> v2:172.21.15.173:3300/0 conn(0x7fc8ec0b39b0 0x7fc8ec0b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.828+0000 7fc8eaa00640 1 -- 172.21.15.102:0/3349253876 >> v1:172.21.15.173:6789/0 conn(0x7fc8ec15c400 legacy=0x7fc8ec15c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.828+0000 7fc8eaa00640 1 -- 172.21.15.102:0/3349253876 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec06ccb0 con 0x7fc8ec15b560 2024-03-23T03:47:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.828+0000 7fc8eaa00640 1 -- 172.21.15.102:0/3349253876 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fc8dc001670 con 0x7fc8ec15b560 2024-03-23T03:47:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.828+0000 7fc8eaa00640 1 -- 172.21.15.102:0/3349253876 >> v1:172.21.15.102:6789/0 conn(0x7fc8ec15b560 legacy=0x7fc8ec15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.828+0000 7fc8eaa00640 1 --2- 172.21.15.102:0/3349253876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d00039c0 0x7fc8d0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.828+0000 7fc8eaa00640 1 --2- 172.21.15.102:0/3349253876 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8d00047c0 0x7fc8d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.828+0000 7fc8eaa00640 1 --2- 172.21.15.102:0/3349253876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8d00051d0 0x7fc8d000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.828+0000 7fc8eaa00640 1 -- 172.21.15.102:0/3349253876 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec0b4370 con 0x7fc8d00039c0 2024-03-23T03:47:56.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.828+0000 7fc8eaa00640 1 -- 172.21.15.102:0/3349253876 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d000c120 con 0x7fc8d00051d0 2024-03-23T03:47:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.828+0000 7fc8eaa00640 1 -- 172.21.15.102:0/3349253876 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d000c2e0 con 0x7fc8d00047c0 2024-03-23T03:47:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.828+0000 7fc8ebe00640 1 --2- 172.21.15.102:0/3349253876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d00039c0 0x7fc8d0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.829+0000 7fc8ebe00640 1 -- 172.21.15.102:0/3349253876 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8d00047c0 msgr2=0x7fc8d0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.829+0000 7fc8ebe00640 1 --2- 172.21.15.102:0/3349253876 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8d00047c0 0x7fc8d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.829+0000 7fc8ebe00640 1 -- 172.21.15.102:0/3349253876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8d00051d0 msgr2=0x7fc8d000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.829+0000 7fc8ebe00640 1 --2- 172.21.15.102:0/3349253876 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8d00051d0 0x7fc8d000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.829+0000 7fc8ebe00640 1 -- 172.21.15.102:0/3349253876 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d0003520 con 0x7fc8d00039c0 2024-03-23T03:47:56.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.829+0000 7fc8f1200640 1 -- 172.21.15.102:0/3349253876 reap_dead start 2024-03-23T03:47:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.830+0000 7fc8ebe00640 1 --2- 172.21.15.102:0/3349253876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d00039c0 0x7fc8d0003de0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc00c0e0 tx=0x7fc8d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732db03aac649ecc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.830+0000 7fc8eaa00640 1 -- 172.21.15.102:0/3349253876 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc005ad0 con 0x7fc8d00039c0 2024-03-23T03:47:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.830+0000 7fc8eaa00640 1 -- 172.21.15.102:0/3349253876 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8dc005c30 con 0x7fc8d00039c0 2024-03-23T03:47:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.830+0000 7fc8eaa00640 1 -- 172.21.15.102:0/3349253876 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8dc02c7d0 con 0x7fc8d00039c0 2024-03-23T03:47:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.830+0000 7fc8f3200640 1 -- 172.21.15.102:0/3349253876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d00039c0 msgr2=0x7fc8d0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.830+0000 7fc8f3200640 1 --2- 172.21.15.102:0/3349253876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d00039c0 0x7fc8d0003de0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc00c0e0 tx=0x7fc8d00057f0 comp rx=0 tx=0).stop 2024-03-23T03:47:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.830+0000 7fc8f3200640 1 -- 172.21.15.102:0/3349253876 shutdown_connections 2024-03-23T03:47:56.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.830+0000 7fc8f3200640 1 --2- 172.21.15.102:0/3349253876 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8d00039c0 0x7fc8d0003de0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:56.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.830+0000 7fc8f3200640 1 -- 172.21.15.102:0/3349253876 >> 172.21.15.102:0/3349253876 conn(0x7fc8ec06a3d0 msgr2=0x7fc8ec0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:56.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.830+0000 7fc8f3200640 1 -- 172.21.15.102:0/3349253876 shutdown_connections 2024-03-23T03:47:56.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.831+0000 7fc8f3200640 1 -- 172.21.15.102:0/3349253876 wait complete. 2024-03-23T03:47:56.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.832+0000 7fc8f3200640 1 Processor -- start 2024-03-23T03:47:56.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.832+0000 7fc8f3200640 1 -- start start 2024-03-23T03:47:56.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.833+0000 7fc8f3200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8d00039c0 0x7fc8ec13ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:56.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.833+0000 7fc8f3200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8ec155f30 0x7fc8ec1563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.833+0000 7fc8f3200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8ec15e4c0 0x7fc8ec15e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.833+0000 7fc8f3200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec15ee80 con 0x7fc8ec15e4c0 2024-03-23T03:47:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.833+0000 7fc8f3200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec15a0d0 con 0x7fc8d00039c0 2024-03-23T03:47:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.833+0000 7fc8f3200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ec156aa0 con 0x7fc8ec155f30 2024-03-23T03:47:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.834+0000 7fc8f1200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8ec15e4c0 0x7fc8ec15e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.834+0000 7fc8f1200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8ec15e4c0 0x7fc8ec15e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54150/0 (socket says 172.21.15.102:54150) 2024-03-23T03:47:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.834+0000 7fc8f1200640 1 -- 172.21.15.102:0/1388094932 learned_addr learned my addr 172.21.15.102:0/1388094932 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.834+0000 7fc8eb400640 1 --2- 172.21.15.102:0/1388094932 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8ec155f30 0x7fc8ec1563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.834+0000 7fc8ebe00640 1 --2- 172.21.15.102:0/1388094932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8d00039c0 0x7fc8ec13ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.834+0000 7fc8f1200640 1 -- 172.21.15.102:0/1388094932 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8ec155f30 msgr2=0x7fc8ec1563b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.834+0000 7fc8f1200640 1 --2- 172.21.15.102:0/1388094932 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8ec155f30 0x7fc8ec1563b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:56.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.834+0000 7fc8f1200640 1 -- 172.21.15.102:0/1388094932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8d00039c0 msgr2=0x7fc8ec13ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:56.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.834+0000 7fc8f1200640 1 --2- 172.21.15.102:0/1388094932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8d00039c0 0x7fc8ec13ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:56.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.834+0000 7fc8f1200640 1 -- 172.21.15.102:0/1388094932 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec156d80 con 0x7fc8ec15e4c0 2024-03-23T03:47:56.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.834+0000 7fc8ebe00640 1 --2- 172.21.15.102:0/1388094932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8d00039c0 0x7fc8ec13ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:56.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.834+0000 7fc8f1200640 1 --2- 172.21.15.102:0/1388094932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8ec15e4c0 0x7fc8ec15e940 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec158a70 tx=0x7fc8e400c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:56.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.834+0000 7fc8e3e00640 1 -- 172.21.15.102:0/1388094932 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4007070 con 0x7fc8ec15e4c0 2024-03-23T03:47:56.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.834+0000 7fc8e3e00640 1 -- 172.21.15.102:0/1388094932 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e40071d0 con 0x7fc8ec15e4c0 2024-03-23T03:47:56.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.834+0000 7fc8e3e00640 1 -- 172.21.15.102:0/1388094932 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4007580 con 0x7fc8ec15e4c0 2024-03-23T03:47:56.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.834+0000 7fc8f3200640 1 -- 172.21.15.102:0/1388094932 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ec157060 con 0x7fc8ec15e4c0 2024-03-23T03:47:56.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.835+0000 7fc8f3200640 1 -- 172.21.15.102:0/1388094932 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ec13b2b0 con 0x7fc8ec15e4c0 2024-03-23T03:47:56.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.836+0000 7fc8f3200640 1 -- 172.21.15.102:0/1388094932 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ec157860 con 0x7fc8ec15e4c0 2024-03-23T03:47:56.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.837+0000 7fc8e3e00640 1 -- 172.21.15.102:0/1388094932 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc8e40076e0 con 0x7fc8ec15e4c0 2024-03-23T03:47:56.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.838+0000 7fc8e3e00640 1 --2- 172.21.15.102:0/1388094932 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc8cc06ac00 0x7fc8cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:56.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.838+0000 7fc8e3e00640 1 -- 172.21.15.102:0/1388094932 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc8e4088c60 con 0x7fc8ec15e4c0 2024-03-23T03:47:56.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.838+0000 7fc8ebe00640 1 --2- 172.21.15.102:0/1388094932 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc8cc06ac00 0x7fc8cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:56.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.839+0000 7fc8ebe00640 1 --2- 172.21.15.102:0/1388094932 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc8cc06ac00 0x7fc8cc06d0c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc034000 tx=0x7fc8dc009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:56.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:56.842+0000 7fc8e3e00640 1 -- 172.21.15.102:0/1388094932 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc8e40539e0 con 0x7fc8ec15e4c0 2024-03-23T03:47:57.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.037+0000 7fc8f3200640 1 -- 172.21.15.102:0/1388094932 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8ec06ac20 con 0x7fc8ec15e4c0 2024-03-23T03:47:57.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.040+0000 7fc8e3e00640 1 -- 172.21.15.102:0/1388094932 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18260 (secure 0 0 0) 0x7fc8e4017070 con 0x7fc8ec15e4c0 2024-03-23T03:47:57.041 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:47:57.041 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":28,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:47:41.647786+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:47:57.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.042+0000 7fc8f3200640 1 -- 172.21.15.102:0/1388094932 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc8cc06ac00 msgr2=0x7fc8cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:57.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.042+0000 7fc8f3200640 1 --2- 172.21.15.102:0/1388094932 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc8cc06ac00 0x7fc8cc06d0c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc8dc034000 tx=0x7fc8dc009030 comp rx=0 tx=0).stop 2024-03-23T03:47:57.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.042+0000 7fc8f3200640 1 -- 172.21.15.102:0/1388094932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8ec15e4c0 msgr2=0x7fc8ec15e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.042+0000 7fc8f3200640 1 --2- 172.21.15.102:0/1388094932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8ec15e4c0 0x7fc8ec15e940 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec158a70 tx=0x7fc8e400c980 comp rx=0 tx=0).stop 2024-03-23T03:47:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.043+0000 7fc8f3200640 1 -- 172.21.15.102:0/1388094932 shutdown_connections 2024-03-23T03:47:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.043+0000 7fc8f3200640 1 --2- 172.21.15.102:0/1388094932 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc8cc06ac00 0x7fc8cc06d0c0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.043+0000 7fc8f3200640 1 --2- 172.21.15.102:0/1388094932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8ec15e4c0 0x7fc8ec15e940 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.043+0000 7fc8f3200640 1 --2- 172.21.15.102:0/1388094932 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8ec155f30 0x7fc8ec1563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.043+0000 7fc8f3200640 1 --2- 172.21.15.102:0/1388094932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8d00039c0 0x7fc8ec13ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.043+0000 7fc8f3200640 1 -- 172.21.15.102:0/1388094932 >> 172.21.15.102:0/1388094932 conn(0x7fc8ec06a3d0 msgr2=0x7fc8ec15c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.043+0000 7fc8f3200640 1 -- 172.21.15.102:0/1388094932 shutdown_connections 2024-03-23T03:47:57.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.043+0000 7fc8f3200640 1 -- 172.21.15.102:0/1388094932 wait complete. 2024-03-23T03:47:57.064 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:46:52.158650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:46:52.914275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-23T03:46:55.983754+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-23T03:47:57.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:47:57.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.276+0000 7f8a74400640 1 Processor -- start 2024-03-23T03:47:57.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.276+0000 7f8a74400640 1 -- start start 2024-03-23T03:47:57.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.278+0000 7f8a74400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8a6c06ce90 0x7f8a6c06d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:57.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.278+0000 7f8a74400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a6c0cdba0 con 0x7f8a6c0add30 2024-03-23T03:47:57.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.278+0000 7f8a74400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a6c0cde80 con 0x7f8a6c0ce930 2024-03-23T03:47:57.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.278+0000 7f8a74400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a6c0ae1c0 con 0x7f8a6c06ce90 2024-03-23T03:47:57.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.278+0000 7f8a72400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8a6c06ce90 0x7f8a6c06d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:57.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.278+0000 7f8a71000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8a6c0add30 0x7f8a6c0ced10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41002/0 (socket says 172.21.15.102:41002) 2024-03-23T03:47:57.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.278+0000 7f8a71000640 1 -- 172.21.15.102:0/875829607 learned_addr learned my addr 172.21.15.102:0/875829607 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:57.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.278+0000 7f8a72400640 1 -- 172.21.15.102:0/875829607 >> v1:172.21.15.173:6789/0 conn(0x7f8a6c0ce930 legacy=0x7f8a6c0cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:57.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.278+0000 7f8a72400640 1 -- 172.21.15.102:0/875829607 >> v1:172.21.15.102:6789/0 conn(0x7f8a6c0add30 legacy=0x7f8a6c0ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:57.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.279+0000 7f8a72400640 1 -- 172.21.15.102:0/875829607 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6c0ce160 con 0x7f8a6c06ce90 2024-03-23T03:47:57.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.279+0000 7f8a6be00640 1 -- 172.21.15.102:0/875829607 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125261392 0 0) 0x7f8a6c0cdba0 con 0x7f8a6c0add30 2024-03-23T03:47:57.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.280+0000 7f8a72400640 1 --2- 172.21.15.102:0/875829607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c06ce90 0x7f8a6c06d300 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8a64009d90 tx=0x7f8a6402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78168c10c032f357 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:57.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.280+0000 7f8a6be00640 1 -- 172.21.15.102:0/875829607 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6402cd00 con 0x7f8a6c06ce90 2024-03-23T03:47:57.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.280+0000 7f8a6be00640 1 -- 172.21.15.102:0/875829607 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6402ce60 con 0x7f8a6c06ce90 2024-03-23T03:47:57.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.280+0000 7f8a6be00640 1 -- 172.21.15.102:0/875829607 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a64034660 con 0x7f8a6c06ce90 2024-03-23T03:47:57.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.281+0000 7f8a74400640 1 -- 172.21.15.102:0/875829607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c06ce90 msgr2=0x7f8a6c06d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:57.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.281+0000 7f8a74400640 1 --2- 172.21.15.102:0/875829607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c06ce90 0x7f8a6c06d300 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8a64009d90 tx=0x7f8a6402bf50 comp rx=0 tx=0).stop 2024-03-23T03:47:57.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.281+0000 7f8a74400640 1 -- 172.21.15.102:0/875829607 shutdown_connections 2024-03-23T03:47:57.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.281+0000 7f8a74400640 1 --2- 172.21.15.102:0/875829607 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c06ce90 0x7f8a6c06d300 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.281+0000 7f8a74400640 1 -- 172.21.15.102:0/875829607 >> 172.21.15.102:0/875829607 conn(0x7f8a6c06a0b0 msgr2=0x7f8a6c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:57.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.281+0000 7f8a74400640 1 -- 172.21.15.102:0/875829607 shutdown_connections 2024-03-23T03:47:57.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.281+0000 7f8a74400640 1 -- 172.21.15.102:0/875829607 wait complete. 2024-03-23T03:47:57.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.283+0000 7f8a74400640 1 Processor -- start 2024-03-23T03:47:57.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.283+0000 7f8a74400640 1 -- start start 2024-03-23T03:47:57.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.284+0000 7f8a74400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a6c06ce90 0x7f8a6c14b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:57.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.284+0000 7f8a74400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c0add30 0x7f8a6c157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:57.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.284+0000 7f8a74400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8a6c0ce930 0x7f8a6c1575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:57.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.284+0000 7f8a74400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c0b2b30 con 0x7f8a6c0add30 2024-03-23T03:47:57.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.284+0000 7f8a74400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c0b2e10 con 0x7f8a6c06ce90 2024-03-23T03:47:57.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.284+0000 7f8a74400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c0b30f0 con 0x7f8a6c0ce930 2024-03-23T03:47:57.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.284+0000 7f8a71000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c0add30 0x7f8a6c157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:57.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.284+0000 7f8a71000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c0add30 0x7f8a6c157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54172/0 (socket says 172.21.15.102:54172) 2024-03-23T03:47:57.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.284+0000 7f8a71000640 1 -- 172.21.15.102:0/3879879800 learned_addr learned my addr 172.21.15.102:0/3879879800 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:57.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.284+0000 7f8a71000640 1 -- 172.21.15.102:0/3879879800 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8a6c0ce930 msgr2=0x7f8a6c1575a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:57.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.284+0000 7f8a72400640 1 --2- 172.21.15.102:0/3879879800 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8a6c0ce930 0x7f8a6c1575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:57.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.284+0000 7f8a71000640 1 --2- 172.21.15.102:0/3879879800 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8a6c0ce930 0x7f8a6c1575a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.284+0000 7f8a71000640 1 -- 172.21.15.102:0/3879879800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a6c06ce90 msgr2=0x7f8a6c14b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:57.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.284+0000 7f8a71a00640 1 --2- 172.21.15.102:0/3879879800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a6c06ce90 0x7f8a6c14b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:57.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.284+0000 7f8a71000640 1 --2- 172.21.15.102:0/3879879800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a6c06ce90 0x7f8a6c14b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.284+0000 7f8a71000640 1 -- 172.21.15.102:0/3879879800 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a64009a70 con 0x7f8a6c0add30 2024-03-23T03:47:57.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.285+0000 7f8a71000640 1 --2- 172.21.15.102:0/3879879800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c0add30 0x7f8a6c157060 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8a5800c4a0 tx=0x7f8a5800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:57.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.285+0000 7f8a69600640 1 -- 172.21.15.102:0/3879879800 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a580074e0 con 0x7f8a6c0add30 2024-03-23T03:47:57.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.285+0000 7f8a69600640 1 -- 172.21.15.102:0/3879879800 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a58007640 con 0x7f8a6c0add30 2024-03-23T03:47:57.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.285+0000 7f8a69600640 1 -- 172.21.15.102:0/3879879800 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a58004a00 con 0x7f8a6c0add30 2024-03-23T03:47:57.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.285+0000 7f8a74400640 1 -- 172.21.15.102:0/3879879800 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a6c158540 con 0x7f8a6c0add30 2024-03-23T03:47:57.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.285+0000 7f8a74400640 1 -- 172.21.15.102:0/3879879800 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a6c14b640 con 0x7f8a6c0add30 2024-03-23T03:47:57.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.286+0000 7f8a74400640 1 -- 172.21.15.102:0/3879879800 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6c15a250 con 0x7f8a6c0add30 2024-03-23T03:47:57.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.287+0000 7f8a69600640 1 -- 172.21.15.102:0/3879879800 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8a58004b60 con 0x7f8a6c0add30 2024-03-23T03:47:57.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.288+0000 7f8a69600640 1 --2- 172.21.15.102:0/3879879800 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8a4c06ac00 0x7f8a4c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:57.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.288+0000 7f8a69600640 1 -- 172.21.15.102:0/3879879800 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8a58088920 con 0x7f8a6c0add30 2024-03-23T03:47:57.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.289+0000 7f8a71a00640 1 --2- 172.21.15.102:0/3879879800 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8a4c06ac00 0x7f8a4c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:57.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.289+0000 7f8a71a00640 1 --2- 172.21.15.102:0/3879879800 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8a4c06ac00 0x7f8a4c06d0c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8a60000f60 tx=0x7f8a60009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:57.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.292+0000 7f8a69600640 1 -- 172.21.15.102:0/3879879800 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8a58053730 con 0x7f8a6c0add30 2024-03-23T03:47:57.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.486+0000 7f8a74400640 1 -- 172.21.15.102:0/3879879800 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8a6c158a60 con 0x7f8a6c0add30 2024-03-23T03:47:57.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.487+0000 7f8a69600640 1 -- 172.21.15.102:0/3879879800 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8a6c158a60 con 0x7f8a6c0add30 2024-03-23T03:47:57.488 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:47:57.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.490+0000 7f8a74400640 1 -- 172.21.15.102:0/3879879800 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8a4c06ac00 msgr2=0x7f8a4c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:57.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.490+0000 7f8a74400640 1 --2- 172.21.15.102:0/3879879800 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8a4c06ac00 0x7f8a4c06d0c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8a60000f60 tx=0x7f8a60009040 comp rx=0 tx=0).stop 2024-03-23T03:47:57.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.490+0000 7f8a74400640 1 -- 172.21.15.102:0/3879879800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c0add30 msgr2=0x7f8a6c157060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:57.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.490+0000 7f8a74400640 1 --2- 172.21.15.102:0/3879879800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c0add30 0x7f8a6c157060 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8a5800c4a0 tx=0x7f8a5800c960 comp rx=0 tx=0).stop 2024-03-23T03:47:57.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.490+0000 7f8a74400640 1 -- 172.21.15.102:0/3879879800 shutdown_connections 2024-03-23T03:47:57.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.490+0000 7f8a74400640 1 --2- 172.21.15.102:0/3879879800 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8a4c06ac00 0x7f8a4c06d0c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.490+0000 7f8a74400640 1 --2- 172.21.15.102:0/3879879800 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8a6c0ce930 0x7f8a6c1575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.490+0000 7f8a74400640 1 --2- 172.21.15.102:0/3879879800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a6c0add30 0x7f8a6c157060 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.490+0000 7f8a74400640 1 --2- 172.21.15.102:0/3879879800 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a6c06ce90 0x7f8a6c14b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.490+0000 7f8a74400640 1 -- 172.21.15.102:0/3879879800 >> 172.21.15.102:0/3879879800 conn(0x7f8a6c06a0b0 msgr2=0x7f8a6c15e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:57.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.490+0000 7f8a74400640 1 -- 172.21.15.102:0/3879879800 shutdown_connections 2024-03-23T03:47:57.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.490+0000 7f8a74400640 1 -- 172.21.15.102:0/3879879800 wait complete. 2024-03-23T03:47:57.510 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:47:57.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.719+0000 7fae97200640 1 Processor -- start 2024-03-23T03:47:57.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.720+0000 7fae97200640 1 -- start start 2024-03-23T03:47:57.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.721+0000 7fae97200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fae900b84b0 0x7fae900d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:57.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.721+0000 7fae97200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fae900b5aa0 0x7fae900b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:57.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.721+0000 7fae97200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae900d1280 con 0x7fae900cd740 2024-03-23T03:47:57.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.721+0000 7fae97200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae900d20c0 con 0x7fae900b84b0 2024-03-23T03:47:57.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.721+0000 7fae97200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae900b8940 con 0x7fae900b5aa0 2024-03-23T03:47:57.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.721+0000 7fae8f400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fae900b84b0 0x7fae900d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:57.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.721+0000 7fae8f400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fae900b84b0 0x7fae900d0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54176/0 (socket says 172.21.15.102:54176) 2024-03-23T03:47:57.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.721+0000 7fae8f400640 1 -- 172.21.15.102:0/394036407 learned_addr learned my addr 172.21.15.102:0/394036407 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:57.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.722+0000 7fae95200640 1 --2- 172.21.15.102:0/394036407 >> v2:172.21.15.173:3300/0 conn(0x7fae900b5aa0 0x7fae900b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:57.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.722+0000 7fae8f400640 1 -- 172.21.15.102:0/394036407 >> v2:172.21.15.173:3300/0 conn(0x7fae900b5aa0 msgr2=0x7fae900b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:57.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.722+0000 7fae8f400640 1 --2- 172.21.15.102:0/394036407 >> v2:172.21.15.173:3300/0 conn(0x7fae900b5aa0 0x7fae900b5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.722+0000 7fae8f400640 1 -- 172.21.15.102:0/394036407 >> v1:172.21.15.173:6789/0 conn(0x7fae900cd740 legacy=0x7fae900cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:57.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.722+0000 7fae8f400640 1 -- 172.21.15.102:0/394036407 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae900d14b0 con 0x7fae900b84b0 2024-03-23T03:47:57.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.723+0000 7fae8f400640 1 --2- 172.21.15.102:0/394036407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900b84b0 0x7fae900d0d40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fae900c99c0 tx=0x7fae7802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29f62be919a19ab1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:57.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.723+0000 7fae8ea00640 1 -- 172.21.15.102:0/394036407 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae78031870 con 0x7fae900b84b0 2024-03-23T03:47:57.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.723+0000 7fae8ea00640 1 -- 172.21.15.102:0/394036407 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae780319d0 con 0x7fae900b84b0 2024-03-23T03:47:57.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.723+0000 7fae8ea00640 1 -- 172.21.15.102:0/394036407 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae78031d60 con 0x7fae900b84b0 2024-03-23T03:47:57.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.724+0000 7fae97200640 1 -- 172.21.15.102:0/394036407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900b84b0 msgr2=0x7fae900d0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:57.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.724+0000 7fae97200640 1 --2- 172.21.15.102:0/394036407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900b84b0 0x7fae900d0d40 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fae900c99c0 tx=0x7fae7802b910 comp rx=0 tx=0).stop 2024-03-23T03:47:57.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.724+0000 7fae97200640 1 -- 172.21.15.102:0/394036407 shutdown_connections 2024-03-23T03:47:57.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.724+0000 7fae97200640 1 --2- 172.21.15.102:0/394036407 >> v2:172.21.15.173:3300/0 conn(0x7fae900b5aa0 0x7fae900b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.724+0000 7fae97200640 1 --2- 172.21.15.102:0/394036407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900b84b0 0x7fae900d0d40 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.724+0000 7fae97200640 1 -- 172.21.15.102:0/394036407 >> 172.21.15.102:0/394036407 conn(0x7fae900cf5d0 msgr2=0x7fae900cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:57.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.724+0000 7fae97200640 1 -- 172.21.15.102:0/394036407 shutdown_connections 2024-03-23T03:47:57.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.724+0000 7fae97200640 1 -- 172.21.15.102:0/394036407 wait complete. 2024-03-23T03:47:57.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.726+0000 7fae97200640 1 Processor -- start 2024-03-23T03:47:57.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.726+0000 7fae97200640 1 -- start start 2024-03-23T03:47:57.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.727+0000 7fae97200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fae900b5aa0 0x7fae9014c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:57.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.727+0000 7fae97200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900b84b0 0x7fae901f5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:57.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.727+0000 7fae97200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae900cd740 0x7fae901f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:57.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.727+0000 7fae97200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae900cb910 con 0x7fae900b84b0 2024-03-23T03:47:57.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.727+0000 7fae97200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae900cbbf0 con 0x7fae900cd740 2024-03-23T03:47:57.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.727+0000 7fae97200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae900cbed0 con 0x7fae900b5aa0 2024-03-23T03:47:57.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.727+0000 7fae8f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900b84b0 0x7fae901f5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:57.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.727+0000 7fae8f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900b84b0 0x7fae901f5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54180/0 (socket says 172.21.15.102:54180) 2024-03-23T03:47:57.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.727+0000 7fae8f400640 1 -- 172.21.15.102:0/3102983003 learned_addr learned my addr 172.21.15.102:0/3102983003 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:57.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.728+0000 7fae95200640 1 --2- 172.21.15.102:0/3102983003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae900cd740 0x7fae901f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:57.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.728+0000 7fae8fe00640 1 --2- 172.21.15.102:0/3102983003 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fae900b5aa0 0x7fae9014c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:57.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.728+0000 7fae8f400640 1 -- 172.21.15.102:0/3102983003 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fae900b5aa0 msgr2=0x7fae9014c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:57.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.728+0000 7fae8f400640 1 --2- 172.21.15.102:0/3102983003 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fae900b5aa0 0x7fae9014c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.728+0000 7fae8f400640 1 -- 172.21.15.102:0/3102983003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae900cd740 msgr2=0x7fae901f5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:57.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.728+0000 7fae8f400640 1 --2- 172.21.15.102:0/3102983003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae900cd740 0x7fae901f5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.728+0000 7fae8f400640 1 -- 172.21.15.102:0/3102983003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae780094d0 con 0x7fae900b84b0 2024-03-23T03:47:57.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.728+0000 7fae95200640 1 --2- 172.21.15.102:0/3102983003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae900cd740 0x7fae901f5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:57.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.728+0000 7fae8f400640 1 --2- 172.21.15.102:0/3102983003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900b84b0 0x7fae901f5a70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fae78000c00 tx=0x7fae780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:57.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.728+0000 7fae87e00640 1 -- 172.21.15.102:0/3102983003 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae78031840 con 0x7fae900b84b0 2024-03-23T03:47:57.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.728+0000 7fae87e00640 1 -- 172.21.15.102:0/3102983003 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae780443f0 con 0x7fae900b84b0 2024-03-23T03:47:57.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.728+0000 7fae87e00640 1 -- 172.21.15.102:0/3102983003 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae780447a0 con 0x7fae900b84b0 2024-03-23T03:47:57.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.728+0000 7fae97200640 1 -- 172.21.15.102:0/3102983003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae900cc1b0 con 0x7fae900b84b0 2024-03-23T03:47:57.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.728+0000 7fae97200640 1 -- 172.21.15.102:0/3102983003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae900cad70 con 0x7fae900b84b0 2024-03-23T03:47:57.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.730+0000 7fae97200640 1 -- 172.21.15.102:0/3102983003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae900cc340 con 0x7fae900b84b0 2024-03-23T03:47:57.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.731+0000 7fae87e00640 1 -- 172.21.15.102:0/3102983003 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fae7804d450 con 0x7fae900b84b0 2024-03-23T03:47:57.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.731+0000 7fae87e00640 1 --2- 172.21.15.102:0/3102983003 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fae6c06ac30 0x7fae6c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:57.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.732+0000 7fae87e00640 1 -- 172.21.15.102:0/3102983003 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fae7802d040 con 0x7fae900b84b0 2024-03-23T03:47:57.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.732+0000 7fae8fe00640 1 --2- 172.21.15.102:0/3102983003 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fae6c06ac30 0x7fae6c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:57.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.732+0000 7fae8fe00640 1 --2- 172.21.15.102:0/3102983003 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fae6c06ac30 0x7fae6c06d0f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fae80007bf0 tx=0x7fae80007240 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:57.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.736+0000 7fae87e00640 1 -- 172.21.15.102:0/3102983003 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fae78076d20 con 0x7fae900b84b0 2024-03-23T03:47:57.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.931+0000 7fae97200640 1 -- 172.21.15.102:0/3102983003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fae90073390 con 0x7fae900b84b0 2024-03-23T03:47:57.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.932+0000 7fae87e00640 1 -- 172.21.15.102:0/3102983003 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7fae780350a0 con 0x7fae900b84b0 2024-03-23T03:47:57.932 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:57.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.934+0000 7fae97200640 1 -- 172.21.15.102:0/3102983003 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fae6c06ac30 msgr2=0x7fae6c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:57.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.934+0000 7fae97200640 1 --2- 172.21.15.102:0/3102983003 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fae6c06ac30 0x7fae6c06d0f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fae80007bf0 tx=0x7fae80007240 comp rx=0 tx=0).stop 2024-03-23T03:47:57.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.935+0000 7fae97200640 1 -- 172.21.15.102:0/3102983003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900b84b0 msgr2=0x7fae901f5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:57.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.935+0000 7fae97200640 1 --2- 172.21.15.102:0/3102983003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900b84b0 0x7fae901f5a70 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fae78000c00 tx=0x7fae780046c0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.935+0000 7fae97200640 1 -- 172.21.15.102:0/3102983003 shutdown_connections 2024-03-23T03:47:57.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.935+0000 7fae97200640 1 --2- 172.21.15.102:0/3102983003 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fae6c06ac30 0x7fae6c06d0f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.935+0000 7fae97200640 1 --2- 172.21.15.102:0/3102983003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae900cd740 0x7fae901f5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.935+0000 7fae97200640 1 --2- 172.21.15.102:0/3102983003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fae900b84b0 0x7fae901f5a70 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.935+0000 7fae97200640 1 --2- 172.21.15.102:0/3102983003 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fae900b5aa0 0x7fae9014c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:57.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.935+0000 7fae97200640 1 -- 172.21.15.102:0/3102983003 >> 172.21.15.102:0/3102983003 conn(0x7fae900cf5d0 msgr2=0x7fae900b4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:57.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.935+0000 7fae97200640 1 -- 172.21.15.102:0/3102983003 shutdown_connections 2024-03-23T03:47:57.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:57.935+0000 7fae97200640 1 -- 172.21.15.102:0/3102983003 wait complete. 2024-03-23T03:47:57.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:47:58.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.152+0000 7f6b7a000640 1 Processor -- start 2024-03-23T03:47:58.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.152+0000 7f6b7a000640 1 -- start start 2024-03-23T03:47:58.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.154+0000 7f6b7a000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f6b74143460 0x7f6b7415aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:58.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.154+0000 7f6b7a000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6b74143930 0x7f6b740cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:58.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.154+0000 7f6b7a000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6b740cdc50 0x7f6b7415e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:58.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.154+0000 7f6b7a000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b740cc6b0 con 0x7f6b74143930 2024-03-23T03:47:58.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.154+0000 7f6b7a000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b740cc950 con 0x7f6b74143460 2024-03-23T03:47:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.154+0000 7f6b7a000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b740ccc30 con 0x7f6b740cdc50 2024-03-23T03:47:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.154+0000 7f6b73e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6b74143930 0x7f6b740cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.154+0000 7f6b73e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6b74143930 0x7f6b740cd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54184/0 (socket says 172.21.15.102:54184) 2024-03-23T03:47:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.154+0000 7f6b73e00640 1 -- 172.21.15.102:0/3521250176 learned_addr learned my addr 172.21.15.102:0/3521250176 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.154+0000 7f6b79600640 1 --2- 172.21.15.102:0/3521250176 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6b740cdc50 0x7f6b7415e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.154+0000 7f6b73e00640 1 -- 172.21.15.102:0/3521250176 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6b740cdc50 msgr2=0x7f6b7415e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.154+0000 7f6b73e00640 1 --2- 172.21.15.102:0/3521250176 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6b740cdc50 0x7f6b7415e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.154+0000 7f6b73e00640 1 -- 172.21.15.102:0/3521250176 >> v2:172.21.15.173:3300/0 conn(0x7f6b74143460 msgr2=0x7f6b7415aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.154+0000 7f6b78c00640 1 --2- 172.21.15.102:0/3521250176 >> v2:172.21.15.173:3300/0 conn(0x7f6b74143460 0x7f6b7415aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.155+0000 7f6b73e00640 1 --2- 172.21.15.102:0/3521250176 >> v2:172.21.15.173:3300/0 conn(0x7f6b74143460 0x7f6b7415aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.155+0000 7f6b73e00640 1 -- 172.21.15.102:0/3521250176 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b740ccf10 con 0x7f6b74143930 2024-03-23T03:47:58.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.155+0000 7f6b79600640 1 --2- 172.21.15.102:0/3521250176 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6b740cdc50 0x7f6b7415e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:58.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.156+0000 7f6b73e00640 1 --2- 172.21.15.102:0/3521250176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b74143930 0x7f6b740cd710 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6b7406e410 tx=0x7f6b6002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cecad5d2120fa94f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:58.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.156+0000 7f6b73400640 1 -- 172.21.15.102:0/3521250176 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60030860 con 0x7f6b74143930 2024-03-23T03:47:58.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.156+0000 7f6b73400640 1 -- 172.21.15.102:0/3521250176 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b600309c0 con 0x7f6b74143930 2024-03-23T03:47:58.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.156+0000 7f6b73400640 1 -- 172.21.15.102:0/3521250176 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b60030d50 con 0x7f6b74143930 2024-03-23T03:47:58.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.157+0000 7f6b7a000640 1 -- 172.21.15.102:0/3521250176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b74143930 msgr2=0x7f6b740cd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:58.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.157+0000 7f6b7a000640 1 --2- 172.21.15.102:0/3521250176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b74143930 0x7f6b740cd710 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6b7406e410 tx=0x7f6b6002b9a0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.157+0000 7f6b7a000640 1 -- 172.21.15.102:0/3521250176 shutdown_connections 2024-03-23T03:47:58.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.157+0000 7f6b7a000640 1 --2- 172.21.15.102:0/3521250176 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6b740cdc50 0x7f6b7415e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.157+0000 7f6b7a000640 1 --2- 172.21.15.102:0/3521250176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b74143930 0x7f6b740cd710 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.157+0000 7f6b7a000640 1 --2- 172.21.15.102:0/3521250176 >> v2:172.21.15.173:3300/0 conn(0x7f6b74143460 0x7f6b7415aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.157+0000 7f6b7a000640 1 -- 172.21.15.102:0/3521250176 >> 172.21.15.102:0/3521250176 conn(0x7f6b74065df0 msgr2=0x7f6b74069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:58.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.157+0000 7f6b7a000640 1 -- 172.21.15.102:0/3521250176 shutdown_connections 2024-03-23T03:47:58.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.157+0000 7f6b7a000640 1 -- 172.21.15.102:0/3521250176 wait complete. 2024-03-23T03:47:58.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.159+0000 7f6b7a000640 1 Processor -- start 2024-03-23T03:47:58.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.159+0000 7f6b7a000640 1 -- start start 2024-03-23T03:47:58.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.160+0000 7f6b7a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b740cdc50 0x7f6b74141690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:58.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.160+0000 7f6b7a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b74143460 0x7f6b74141bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:58.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.160+0000 7f6b7a000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6b74143930 0x7f6b740ac540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:58.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.160+0000 7f6b7a000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b7415be00 con 0x7f6b740cdc50 2024-03-23T03:47:58.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.160+0000 7f6b7a000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b74142110 con 0x7f6b74143460 2024-03-23T03:47:58.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.160+0000 7f6b7a000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b741423c0 con 0x7f6b74143930 2024-03-23T03:47:58.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.161+0000 7f6b78c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b740cdc50 0x7f6b74141690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:58.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.161+0000 7f6b78c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b740cdc50 0x7f6b74141690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54196/0 (socket says 172.21.15.102:54196) 2024-03-23T03:47:58.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.161+0000 7f6b78c00640 1 -- 172.21.15.102:0/1023405247 learned_addr learned my addr 172.21.15.102:0/1023405247 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:58.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.161+0000 7f6b79600640 1 --2- 172.21.15.102:0/1023405247 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6b74143930 0x7f6b740ac540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:58.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.161+0000 7f6b73e00640 1 --2- 172.21.15.102:0/1023405247 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b74143460 0x7f6b74141bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:58.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.161+0000 7f6b78c00640 1 -- 172.21.15.102:0/1023405247 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6b74143930 msgr2=0x7f6b740ac540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:58.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.161+0000 7f6b78c00640 1 --2- 172.21.15.102:0/1023405247 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6b74143930 0x7f6b740ac540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.161+0000 7f6b78c00640 1 -- 172.21.15.102:0/1023405247 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b74143460 msgr2=0x7f6b74141bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:58.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.161+0000 7f6b78c00640 1 --2- 172.21.15.102:0/1023405247 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b74143460 0x7f6b74141bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.161+0000 7f6b78c00640 1 -- 172.21.15.102:0/1023405247 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b600094d0 con 0x7f6b740cdc50 2024-03-23T03:47:58.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.161+0000 7f6b73e00640 1 --2- 172.21.15.102:0/1023405247 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b74143460 0x7f6b74141bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:58.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.162+0000 7f6b78c00640 1 --2- 172.21.15.102:0/1023405247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b740cdc50 0x7f6b74141690 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800c7f0 tx=0x7f6b6800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:58.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.162+0000 7f6b70c00640 1 -- 172.21.15.102:0/1023405247 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68004220 con 0x7f6b740cdc50 2024-03-23T03:47:58.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.162+0000 7f6b70c00640 1 -- 172.21.15.102:0/1023405247 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b680043a0 con 0x7f6b740cdc50 2024-03-23T03:47:58.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.162+0000 7f6b70c00640 1 -- 172.21.15.102:0/1023405247 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b68007860 con 0x7f6b740cdc50 2024-03-23T03:47:58.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.162+0000 7f6b7a000640 1 -- 172.21.15.102:0/1023405247 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7415b4d0 con 0x7f6b740cdc50 2024-03-23T03:47:58.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.162+0000 7f6b7a000640 1 -- 172.21.15.102:0/1023405247 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b7415ba90 con 0x7f6b740cdc50 2024-03-23T03:47:58.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.163+0000 7f6b7a000640 1 -- 172.21.15.102:0/1023405247 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b740cb9f0 con 0x7f6b740cdc50 2024-03-23T03:47:58.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.164+0000 7f6b70c00640 1 -- 172.21.15.102:0/1023405247 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6b68007bd0 con 0x7f6b740cdc50 2024-03-23T03:47:58.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.165+0000 7f6b70c00640 1 --2- 172.21.15.102:0/1023405247 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6b5406ac30 0x7f6b5406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:58.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.165+0000 7f6b70c00640 1 -- 172.21.15.102:0/1023405247 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f6b68089760 con 0x7f6b740cdc50 2024-03-23T03:47:58.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.165+0000 7f6b73e00640 1 --2- 172.21.15.102:0/1023405247 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6b5406ac30 0x7f6b5406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:58.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.166+0000 7f6b73e00640 1 --2- 172.21.15.102:0/1023405247 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6b5406ac30 0x7f6b5406d0f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6b60036000 tx=0x7f6b6003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:58.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.169+0000 7f6b70c00640 1 -- 172.21.15.102:0/1023405247 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6b680544e0 con 0x7f6b740cdc50 2024-03-23T03:47:58.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.364+0000 7f6b7a000640 1 -- 172.21.15.102:0/1023405247 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6b7404e030 con 0x7f6b740cdc50 2024-03-23T03:47:58.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.365+0000 7f6b70c00640 1 -- 172.21.15.102:0/1023405247 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6b68057b00 con 0x7f6b740cdc50 2024-03-23T03:47:58.366 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:47:58.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.368+0000 7f6b7a000640 1 -- 172.21.15.102:0/1023405247 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6b5406ac30 msgr2=0x7f6b5406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:58.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.368+0000 7f6b7a000640 1 --2- 172.21.15.102:0/1023405247 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6b5406ac30 0x7f6b5406d0f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6b60036000 tx=0x7f6b6003b040 comp rx=0 tx=0).stop 2024-03-23T03:47:58.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.368+0000 7f6b7a000640 1 -- 172.21.15.102:0/1023405247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b740cdc50 msgr2=0x7f6b74141690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:58.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.368+0000 7f6b7a000640 1 --2- 172.21.15.102:0/1023405247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b740cdc50 0x7f6b74141690 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f6b6800c7f0 tx=0x7f6b6800ccb0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.368+0000 7f6b7a000640 1 -- 172.21.15.102:0/1023405247 shutdown_connections 2024-03-23T03:47:58.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.368+0000 7f6b7a000640 1 --2- 172.21.15.102:0/1023405247 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6b5406ac30 0x7f6b5406d0f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.368+0000 7f6b7a000640 1 --2- 172.21.15.102:0/1023405247 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6b74143930 0x7f6b740ac540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.368+0000 7f6b7a000640 1 --2- 172.21.15.102:0/1023405247 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6b74143460 0x7f6b74141bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.368+0000 7f6b7a000640 1 --2- 172.21.15.102:0/1023405247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6b740cdc50 0x7f6b74141690 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.368+0000 7f6b7a000640 1 -- 172.21.15.102:0/1023405247 >> 172.21.15.102:0/1023405247 conn(0x7f6b74065df0 msgr2=0x7f6b74069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:58.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.368+0000 7f6b7a000640 1 -- 172.21.15.102:0/1023405247 shutdown_connections 2024-03-23T03:47:58.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.368+0000 7f6b7a000640 1 -- 172.21.15.102:0/1023405247 wait complete. 2024-03-23T03:47:58.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-23T03:47:58.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.586+0000 7f7799400640 1 Processor -- start 2024-03-23T03:47:58.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.586+0000 7f7799400640 1 -- start start 2024-03-23T03:47:58.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.587+0000 7f7799400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77940add30 0x7f77940ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:58.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.587+0000 7f7799400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f779406ce90 0x7f779406d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:58.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.587+0000 7f7799400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77940cf250 con 0x7f77940ce930 2024-03-23T03:47:58.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.587+0000 7f7799400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f779406c760 con 0x7f779406ce90 2024-03-23T03:47:58.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.587+0000 7f7799400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77940b42d0 con 0x7f77940add30 2024-03-23T03:47:58.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.588+0000 7f7793200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f779406ce90 0x7f779406d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:58.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.588+0000 7f7793200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f779406ce90 0x7f779406d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54208/0 (socket says 172.21.15.102:54208) 2024-03-23T03:47:58.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.588+0000 7f7793200640 1 -- 172.21.15.102:0/2967718986 learned_addr learned my addr 172.21.15.102:0/2967718986 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:58.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.588+0000 7f7791e00640 1 --2- 172.21.15.102:0/2967718986 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77940add30 0x7f77940ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:58.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.588+0000 7f7793200640 1 -- 172.21.15.102:0/2967718986 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77940add30 msgr2=0x7f77940ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:58.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.588+0000 7f7793200640 1 --2- 172.21.15.102:0/2967718986 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77940add30 0x7f77940ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.588+0000 7f7793200640 1 -- 172.21.15.102:0/2967718986 >> v1:172.21.15.173:6789/0 conn(0x7f77940ce930 legacy=0x7f77940cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:58.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.588+0000 7f7793200640 1 -- 172.21.15.102:0/2967718986 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77940cdba0 con 0x7f779406ce90 2024-03-23T03:47:58.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.589+0000 7f7793200640 1 --2- 172.21.15.102:0/2967718986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f779406ce90 0x7f779406d320 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f778c009e50 tx=0x7f778c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=369e7884e17ba0c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:58.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.590+0000 7f7791400640 1 -- 172.21.15.102:0/2967718986 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c02ca70 con 0x7f779406ce90 2024-03-23T03:47:58.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.590+0000 7f7791400640 1 -- 172.21.15.102:0/2967718986 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778c02cbd0 con 0x7f779406ce90 2024-03-23T03:47:58.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.590+0000 7f7791400640 1 -- 172.21.15.102:0/2967718986 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778c0343f0 con 0x7f779406ce90 2024-03-23T03:47:58.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.590+0000 7f7799400640 1 -- 172.21.15.102:0/2967718986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f779406ce90 msgr2=0x7f779406d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:58.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.590+0000 7f7799400640 1 --2- 172.21.15.102:0/2967718986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f779406ce90 0x7f779406d320 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f778c009e50 tx=0x7f778c02bed0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.590+0000 7f7799400640 1 -- 172.21.15.102:0/2967718986 shutdown_connections 2024-03-23T03:47:58.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.590+0000 7f7799400640 1 --2- 172.21.15.102:0/2967718986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f779406ce90 0x7f779406d320 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.590+0000 7f7799400640 1 --2- 172.21.15.102:0/2967718986 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77940add30 0x7f77940ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.590+0000 7f7799400640 1 -- 172.21.15.102:0/2967718986 >> 172.21.15.102:0/2967718986 conn(0x7f779406a0b0 msgr2=0x7f779406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:58.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.590+0000 7f7799400640 1 -- 172.21.15.102:0/2967718986 shutdown_connections 2024-03-23T03:47:58.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.590+0000 7f7799400640 1 -- 172.21.15.102:0/2967718986 wait complete. 2024-03-23T03:47:58.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.592+0000 7f7799400640 1 Processor -- start 2024-03-23T03:47:58.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.592+0000 7f7799400640 1 -- start start 2024-03-23T03:47:58.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.593+0000 7f7799400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f779406ce90 0x7f77940b0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:58.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.593+0000 7f7799400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77940add30 0x7f779414bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:58.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.593+0000 7f7799400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77940ce930 0x7f77941561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:58.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.593+0000 7f7799400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7794155490 con 0x7f77940add30 2024-03-23T03:47:58.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.593+0000 7f7799400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f779415a5b0 con 0x7f779406ce90 2024-03-23T03:47:58.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.593+0000 7f7799400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77941556e0 con 0x7f77940ce930 2024-03-23T03:47:58.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.593+0000 7f7791e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77940add30 0x7f779414bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:58.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.593+0000 7f7791e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77940add30 0x7f779414bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54216/0 (socket says 172.21.15.102:54216) 2024-03-23T03:47:58.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.593+0000 7f7791e00640 1 -- 172.21.15.102:0/2768511789 learned_addr learned my addr 172.21.15.102:0/2768511789 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:58.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.594+0000 7f7793200640 1 --2- 172.21.15.102:0/2768511789 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77940ce930 0x7f77941561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:58.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.594+0000 7f7792800640 1 --2- 172.21.15.102:0/2768511789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f779406ce90 0x7f77940b0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:58.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.594+0000 7f7791e00640 1 -- 172.21.15.102:0/2768511789 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77940ce930 msgr2=0x7f77941561c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:58.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.594+0000 7f7791e00640 1 --2- 172.21.15.102:0/2768511789 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77940ce930 0x7f77941561c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.594+0000 7f7791e00640 1 -- 172.21.15.102:0/2768511789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f779406ce90 msgr2=0x7f77940b0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:58.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.594+0000 7f7791e00640 1 --2- 172.21.15.102:0/2768511789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f779406ce90 0x7f77940b0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.594+0000 7f7791e00640 1 -- 172.21.15.102:0/2768511789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778c009b30 con 0x7f77940add30 2024-03-23T03:47:58.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.594+0000 7f7792800640 1 --2- 172.21.15.102:0/2768511789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f779406ce90 0x7f77940b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:58.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.594+0000 7f7791e00640 1 --2- 172.21.15.102:0/2768511789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77940add30 0x7f779414bd20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f778000c7f0 tx=0x7f778000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:58.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.594+0000 7f778aa00640 1 -- 172.21.15.102:0/2768511789 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7780004220 con 0x7f77940add30 2024-03-23T03:47:58.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.595+0000 7f778aa00640 1 -- 172.21.15.102:0/2768511789 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77800043a0 con 0x7f77940add30 2024-03-23T03:47:58.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.595+0000 7f7799400640 1 -- 172.21.15.102:0/2768511789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77941559c0 con 0x7f77940add30 2024-03-23T03:47:58.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.595+0000 7f778aa00640 1 -- 172.21.15.102:0/2768511789 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7780007860 con 0x7f77940add30 2024-03-23T03:47:58.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.595+0000 7f7799400640 1 -- 172.21.15.102:0/2768511789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f779414c260 con 0x7f77940add30 2024-03-23T03:47:58.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.596+0000 7f7799400640 1 -- 172.21.15.102:0/2768511789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7794155ee0 con 0x7f77940add30 2024-03-23T03:47:58.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.597+0000 7f778aa00640 1 -- 172.21.15.102:0/2768511789 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7780007bd0 con 0x7f77940add30 2024-03-23T03:47:58.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.598+0000 7f778aa00640 1 --2- 172.21.15.102:0/2768511789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f777406ac30 0x7f777406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:58.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.598+0000 7f778aa00640 1 -- 172.21.15.102:0/2768511789 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f77800897a0 con 0x7f77940add30 2024-03-23T03:47:58.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.598+0000 7f7792800640 1 --2- 172.21.15.102:0/2768511789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f777406ac30 0x7f777406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:58.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.599+0000 7f7792800640 1 --2- 172.21.15.102:0/2768511789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f777406ac30 0x7f777406d0f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f778400a830 tx=0x7f7784008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:58.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.602+0000 7f778aa00640 1 -- 172.21.15.102:0/2768511789 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7794155ee0 con 0x7f77940add30 2024-03-23T03:47:58.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.795+0000 7f7799400640 1 -- 172.21.15.102:0/2768511789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f7794155ee0 con 0x7f77940add30 2024-03-23T03:47:58.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.796+0000 7f778aa00640 1 -- 172.21.15.102:0/2768511789 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7794155ee0 con 0x7f77940add30 2024-03-23T03:47:58.797 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:47:58.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.799+0000 7f7799400640 1 -- 172.21.15.102:0/2768511789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f777406ac30 msgr2=0x7f777406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:58.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.799+0000 7f7799400640 1 --2- 172.21.15.102:0/2768511789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f777406ac30 0x7f777406d0f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f778400a830 tx=0x7f7784008040 comp rx=0 tx=0).stop 2024-03-23T03:47:58.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.799+0000 7f7799400640 1 -- 172.21.15.102:0/2768511789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77940add30 msgr2=0x7f779414bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:58.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.799+0000 7f7799400640 1 --2- 172.21.15.102:0/2768511789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77940add30 0x7f779414bd20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f778000c7f0 tx=0x7f778000ccb0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.799+0000 7f7799400640 1 -- 172.21.15.102:0/2768511789 shutdown_connections 2024-03-23T03:47:58.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.799+0000 7f7799400640 1 --2- 172.21.15.102:0/2768511789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f777406ac30 0x7f777406d0f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.799+0000 7f7799400640 1 --2- 172.21.15.102:0/2768511789 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f77940ce930 0x7f77941561c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.799+0000 7f7799400640 1 --2- 172.21.15.102:0/2768511789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77940add30 0x7f779414bd20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.799+0000 7f7799400640 1 --2- 172.21.15.102:0/2768511789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f779406ce90 0x7f77940b0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:58.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.799+0000 7f7799400640 1 -- 172.21.15.102:0/2768511789 >> 172.21.15.102:0/2768511789 conn(0x7f779406a0b0 msgr2=0x7f779406a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:58.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.800+0000 7f7799400640 1 -- 172.21.15.102:0/2768511789 shutdown_connections 2024-03-23T03:47:58.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:58.800+0000 7f7799400640 1 -- 172.21.15.102:0/2768511789 wait complete. 2024-03-23T03:47:58.818 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:47:59.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.028+0000 7f7214600640 1 Processor -- start 2024-03-23T03:47:59.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.028+0000 7f7214600640 1 -- start start 2024-03-23T03:47:59.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.030+0000 7f7214600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f720c160dd0 0x7f720c0ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:59.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.030+0000 7f7214600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720c0cf2c0 con 0x7f720c0cee70 2024-03-23T03:47:59.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.030+0000 7f7214600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f720c06cdc0 con 0x7f720c13fa80 2024-03-23T03:47:59.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.030+0000 7f7214600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f720c06d060 con 0x7f720c160dd0 2024-03-23T03:47:59.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.030+0000 7f7211200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f720c160dd0 0x7f720c0ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:59.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.030+0000 7f7212600640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f720c0cee70 0x7f720c1422d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41010/0 (socket says 172.21.15.102:41010) 2024-03-23T03:47:59.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.030+0000 7f7212600640 1 -- 172.21.15.102:0/2059415806 learned_addr learned my addr 172.21.15.102:0/2059415806 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:59.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.031+0000 7f7211200640 1 -- 172.21.15.102:0/2059415806 >> v1:172.21.15.173:6789/0 conn(0x7f720c13fa80 legacy=0x7f720c13ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:59.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.031+0000 7f7211200640 1 -- 172.21.15.102:0/2059415806 >> v1:172.21.15.102:6789/0 conn(0x7f720c0cee70 legacy=0x7f720c1422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:59.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.031+0000 7f7211200640 1 -- 172.21.15.102:0/2059415806 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f720c06d340 con 0x7f720c160dd0 2024-03-23T03:47:59.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.031+0000 7f720be00640 1 -- 172.21.15.102:0/2059415806 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739559509 0 0) 0x7f7204001490 con 0x7f720c0cee70 2024-03-23T03:47:59.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.032+0000 7f7211200640 1 --2- 172.21.15.102:0/2059415806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f720c160dd0 0x7f720c0ce930 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f71f80097a0 tx=0x7f71f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18cbd898fac34250 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:59.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.032+0000 7f720be00640 1 -- 172.21.15.102:0/2059415806 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8031830 con 0x7f720c160dd0 2024-03-23T03:47:59.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.032+0000 7f720be00640 1 -- 172.21.15.102:0/2059415806 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f8031990 con 0x7f720c160dd0 2024-03-23T03:47:59.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.032+0000 7f720be00640 1 -- 172.21.15.102:0/2059415806 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8031d20 con 0x7f720c160dd0 2024-03-23T03:47:59.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.033+0000 7f7214600640 1 -- 172.21.15.102:0/2059415806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f720c160dd0 msgr2=0x7f720c0ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:59.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.033+0000 7f7214600640 1 --2- 172.21.15.102:0/2059415806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f720c160dd0 0x7f720c0ce930 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f71f80097a0 tx=0x7f71f802b8d0 comp rx=0 tx=0).stop 2024-03-23T03:47:59.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.033+0000 7f7214600640 1 -- 172.21.15.102:0/2059415806 shutdown_connections 2024-03-23T03:47:59.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.033+0000 7f7214600640 1 --2- 172.21.15.102:0/2059415806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f720c160dd0 0x7f720c0ce930 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:59.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.033+0000 7f7214600640 1 -- 172.21.15.102:0/2059415806 >> 172.21.15.102:0/2059415806 conn(0x7f720c06a0b0 msgr2=0x7f720c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:59.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.033+0000 7f7214600640 1 -- 172.21.15.102:0/2059415806 shutdown_connections 2024-03-23T03:47:59.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.033+0000 7f7214600640 1 -- 172.21.15.102:0/2059415806 wait complete. 2024-03-23T03:47:59.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.035+0000 7f7214600640 1 Processor -- start 2024-03-23T03:47:59.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.035+0000 7f7214600640 1 -- start start 2024-03-23T03:47:59.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.036+0000 7f7214600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f720c0cee70 0x7f720c13ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:59.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.036+0000 7f7214600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f720c13fa80 0x7f720c156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:59.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.036+0000 7f7214600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f720c160dd0 0x7f720c156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:59.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.036+0000 7f7214600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720c0af360 con 0x7f720c160dd0 2024-03-23T03:47:59.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.036+0000 7f7214600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720c0af540 con 0x7f720c0cee70 2024-03-23T03:47:59.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.036+0000 7f7214600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f720c0af820 con 0x7f720c13fa80 2024-03-23T03:47:59.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.036+0000 7f7212600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f720c160dd0 0x7f720c156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:59.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.036+0000 7f7212600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f720c160dd0 0x7f720c156ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54234/0 (socket says 172.21.15.102:54234) 2024-03-23T03:47:59.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.036+0000 7f7212600640 1 -- 172.21.15.102:0/1686067195 learned_addr learned my addr 172.21.15.102:0/1686067195 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:59.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.036+0000 7f7212600640 1 -- 172.21.15.102:0/1686067195 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f720c13fa80 msgr2=0x7f720c156960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:59.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.036+0000 7f7211200640 1 --2- 172.21.15.102:0/1686067195 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f720c13fa80 0x7f720c156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:59.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.037+0000 7f7212600640 1 --2- 172.21.15.102:0/1686067195 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f720c13fa80 0x7f720c156960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:59.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.037+0000 7f7212600640 1 -- 172.21.15.102:0/1686067195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f720c0cee70 msgr2=0x7f720c13ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:59.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.037+0000 7f7211c00640 1 --2- 172.21.15.102:0/1686067195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f720c0cee70 0x7f720c13ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:59.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.037+0000 7f7212600640 1 --2- 172.21.15.102:0/1686067195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f720c0cee70 0x7f720c13ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:59.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.037+0000 7f7212600640 1 -- 172.21.15.102:0/1686067195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f8009400 con 0x7f720c160dd0 2024-03-23T03:47:59.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.037+0000 7f7211200640 1 --2- 172.21.15.102:0/1686067195 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f720c13fa80 0x7f720c156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:59.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.037+0000 7f7212600640 1 --2- 172.21.15.102:0/1686067195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f720c160dd0 0x7f720c156ea0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f720400c7d0 tx=0x7f720400cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:59.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.037+0000 7f7209600640 1 -- 172.21.15.102:0/1686067195 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720400a640 con 0x7f720c160dd0 2024-03-23T03:47:59.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.037+0000 7f7209600640 1 -- 172.21.15.102:0/1686067195 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f720400a7c0 con 0x7f720c160dd0 2024-03-23T03:47:59.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.037+0000 7f7209600640 1 -- 172.21.15.102:0/1686067195 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720400abb0 con 0x7f720c160dd0 2024-03-23T03:47:59.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.037+0000 7f7214600640 1 -- 172.21.15.102:0/1686067195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f720c0afb00 con 0x7f720c160dd0 2024-03-23T03:47:59.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.037+0000 7f7214600640 1 -- 172.21.15.102:0/1686067195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f720c0b0300 con 0x7f720c160dd0 2024-03-23T03:47:59.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.038+0000 7f7214600640 1 -- 172.21.15.102:0/1686067195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f720c0b0020 con 0x7f720c160dd0 2024-03-23T03:47:59.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.039+0000 7f7209600640 1 -- 172.21.15.102:0/1686067195 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f720400ad10 con 0x7f720c160dd0 2024-03-23T03:47:59.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.040+0000 7f7209600640 1 --2- 172.21.15.102:0/1686067195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f71ec06ac00 0x7f71ec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:59.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.040+0000 7f7209600640 1 -- 172.21.15.102:0/1686067195 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7204088bf0 con 0x7f720c160dd0 2024-03-23T03:47:59.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.041+0000 7f7211c00640 1 --2- 172.21.15.102:0/1686067195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f71ec06ac00 0x7f71ec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:59.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.041+0000 7f7211c00640 1 --2- 172.21.15.102:0/1686067195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f71ec06ac00 0x7f71ec06d0c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7200001650 tx=0x7f7200009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:59.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.045+0000 7f7209600640 1 -- 172.21.15.102:0/1686067195 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f720c0b0020 con 0x7f720c160dd0 2024-03-23T03:47:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.252+0000 7f7214600640 1 -- 172.21.15.102:0/1686067195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f720c0cf2c0 con 0x7f720c160dd0 2024-03-23T03:47:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.253+0000 7f7209600640 1 -- 172.21.15.102:0/1686067195 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) v1 ==== 80+0+0 (secure 0 0 0) 0x7f720c0cf2c0 con 0x7f720c160dd0 2024-03-23T03:47:59.253 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:47:59.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.255+0000 7f7214600640 1 -- 172.21.15.102:0/1686067195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f71ec06ac00 msgr2=0x7f71ec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:59.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.255+0000 7f7214600640 1 --2- 172.21.15.102:0/1686067195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f71ec06ac00 0x7f71ec06d0c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7200001650 tx=0x7f7200009040 comp rx=0 tx=0).stop 2024-03-23T03:47:59.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.255+0000 7f7214600640 1 -- 172.21.15.102:0/1686067195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f720c160dd0 msgr2=0x7f720c156ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:59.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.255+0000 7f7214600640 1 --2- 172.21.15.102:0/1686067195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f720c160dd0 0x7f720c156ea0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f720400c7d0 tx=0x7f720400cc90 comp rx=0 tx=0).stop 2024-03-23T03:47:59.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.256+0000 7f7214600640 1 -- 172.21.15.102:0/1686067195 shutdown_connections 2024-03-23T03:47:59.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.256+0000 7f7214600640 1 --2- 172.21.15.102:0/1686067195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f71ec06ac00 0x7f71ec06d0c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:59.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.256+0000 7f7214600640 1 --2- 172.21.15.102:0/1686067195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f720c160dd0 0x7f720c156ea0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:59.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.256+0000 7f7214600640 1 --2- 172.21.15.102:0/1686067195 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f720c13fa80 0x7f720c156960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:59.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.256+0000 7f7214600640 1 --2- 172.21.15.102:0/1686067195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f720c0cee70 0x7f720c13ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:59.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.256+0000 7f7214600640 1 -- 172.21.15.102:0/1686067195 >> 172.21.15.102:0/1686067195 conn(0x7f720c06a0b0 msgr2=0x7f720c13d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:59.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.256+0000 7f7214600640 1 -- 172.21.15.102:0/1686067195 shutdown_connections 2024-03-23T03:47:59.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.256+0000 7f7214600640 1 -- 172.21.15.102:0/1686067195 wait complete. 2024-03-23T03:47:59.273 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-23T03:47:59.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:59.273 DEBUG:teuthology.orchestra.run.smithi173:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2024-03-23T03:47:59.379 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:47:59.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:47:59.380 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T03:47:59.439 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:47:59.439 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:47:59.495 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:47:59.495 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:47:59.548 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:47:59.548 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:47:59.643 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:47:59.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.842+0000 7fcf0be00640 1 Processor -- start 2024-03-23T03:47:59.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.843+0000 7fcf0be00640 1 -- start start 2024-03-23T03:47:59.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.844+0000 7fcf0be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcf0c143460 0x7fcf0c15aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:59.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.844+0000 7fcf0be00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf0c0cc6b0 con 0x7fcf0c0cde20 2024-03-23T03:47:59.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.844+0000 7fcf0be00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf0c0cc950 con 0x7fcf0c143930 2024-03-23T03:47:59.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.844+0000 7fcf0be00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf0c0ce270 con 0x7fcf0c143460 2024-03-23T03:47:59.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.844+0000 7fcf0b400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fcf0c0cde20 0x7fcf0c070020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41026/0 (socket says 172.21.15.102:41026) 2024-03-23T03:47:59.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.844+0000 7fcf0b400640 1 -- 172.21.15.102:0/2125870114 learned_addr learned my addr 172.21.15.102:0/2125870114 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:47:59.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.844+0000 7fcf0aa00640 1 --2- 172.21.15.102:0/2125870114 >> v2:172.21.15.102:3300/0 conn(0x7fcf0c143460 0x7fcf0c15aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:59.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.845+0000 7fcf0a000640 1 -- 172.21.15.102:0/2125870114 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3990887267 0 0) 0x7fcf0c0cc6b0 con 0x7fcf0c0cde20 2024-03-23T03:47:59.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.845+0000 7fcf0a000640 1 -- 172.21.15.102:0/2125870114 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcef0003520 con 0x7fcf0c0cde20 2024-03-23T03:47:59.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.845+0000 7fcf0aa00640 1 -- 172.21.15.102:0/2125870114 >> v1:172.21.15.173:6789/0 conn(0x7fcf0c143930 legacy=0x7fcf0c0cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:47:59.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.845+0000 7fcf0aa00640 1 -- 172.21.15.102:0/2125870114 >> v1:172.21.15.102:6789/0 conn(0x7fcf0c0cde20 legacy=0x7fcf0c070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:59.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.845+0000 7fcf0aa00640 1 -- 172.21.15.102:0/2125870114 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0c0ccc30 con 0x7fcf0c143460 2024-03-23T03:47:59.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.846+0000 7fcf0aa00640 1 --2- 172.21.15.102:0/2125870114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf0c143460 0x7fcf0c15aba0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0097a0 tx=0x7fcefc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad14708c7d832cd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:59.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.846+0000 7fcf0a000640 1 -- 172.21.15.102:0/2125870114 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc031830 con 0x7fcf0c143460 2024-03-23T03:47:59.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.846+0000 7fcf0a000640 1 -- 172.21.15.102:0/2125870114 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcefc031990 con 0x7fcf0c143460 2024-03-23T03:47:59.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.846+0000 7fcf0a000640 1 -- 172.21.15.102:0/2125870114 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcefc031d20 con 0x7fcf0c143460 2024-03-23T03:47:59.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.847+0000 7fcf0be00640 1 -- 172.21.15.102:0/2125870114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf0c143460 msgr2=0x7fcf0c15aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:59.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.847+0000 7fcf0be00640 1 --2- 172.21.15.102:0/2125870114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf0c143460 0x7fcf0c15aba0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0097a0 tx=0x7fcefc02b8d0 comp rx=0 tx=0).stop 2024-03-23T03:47:59.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.847+0000 7fcf0be00640 1 -- 172.21.15.102:0/2125870114 shutdown_connections 2024-03-23T03:47:59.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.847+0000 7fcf0be00640 1 --2- 172.21.15.102:0/2125870114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf0c143460 0x7fcf0c15aba0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:59.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.847+0000 7fcf0be00640 1 -- 172.21.15.102:0/2125870114 >> 172.21.15.102:0/2125870114 conn(0x7fcf0c065df0 msgr2=0x7fcf0c069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:47:59.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.847+0000 7fcf0be00640 1 -- 172.21.15.102:0/2125870114 shutdown_connections 2024-03-23T03:47:59.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.847+0000 7fcf0be00640 1 -- 172.21.15.102:0/2125870114 wait complete. 2024-03-23T03:47:59.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.849+0000 7fcf0be00640 1 Processor -- start 2024-03-23T03:47:59.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.849+0000 7fcf0be00640 1 -- start start 2024-03-23T03:47:59.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.850+0000 7fcf0be00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf0c0cde20 0x7fcf0c15a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:59.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.850+0000 7fcf0be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf0c143460 0x7fcf0c137be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:59.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.850+0000 7fcf0be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf0c143930 0x7fcf0c06c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:59.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.850+0000 7fcf0be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c1390d0 con 0x7fcf0c143460 2024-03-23T03:47:59.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.850+0000 7fcf0be00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c138550 con 0x7fcf0c143930 2024-03-23T03:47:59.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.850+0000 7fcf0be00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c138830 con 0x7fcf0c0cde20 2024-03-23T03:47:59.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.850+0000 7fcf03e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf0c143460 0x7fcf0c137be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:59.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.850+0000 7fcf03e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf0c143460 0x7fcf0c137be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54254/0 (socket says 172.21.15.102:54254) 2024-03-23T03:47:59.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.850+0000 7fcf03e00640 1 -- 172.21.15.102:0/262215220 learned_addr learned my addr 172.21.15.102:0/262215220 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:47:59.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.850+0000 7fcf0aa00640 1 --2- 172.21.15.102:0/262215220 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf0c0cde20 0x7fcf0c15a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:59.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.850+0000 7fcf03e00640 1 -- 172.21.15.102:0/262215220 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf0c0cde20 msgr2=0x7fcf0c15a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:59.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.850+0000 7fcf0b400640 1 --2- 172.21.15.102:0/262215220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf0c143930 0x7fcf0c06c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:59.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.850+0000 7fcf03e00640 1 --2- 172.21.15.102:0/262215220 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf0c0cde20 0x7fcf0c15a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:59.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.850+0000 7fcf03e00640 1 -- 172.21.15.102:0/262215220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf0c143930 msgr2=0x7fcf0c06c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:47:59.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.850+0000 7fcf03e00640 1 --2- 172.21.15.102:0/262215220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf0c143930 0x7fcf0c06c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:47:59.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.850+0000 7fcf03e00640 1 -- 172.21.15.102:0/262215220 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc009400 con 0x7fcf0c143460 2024-03-23T03:47:59.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.850+0000 7fcf0b400640 1 --2- 172.21.15.102:0/262215220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf0c143930 0x7fcf0c06c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:47:59.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.851+0000 7fcf03e00640 1 --2- 172.21.15.102:0/262215220 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf0c143460 0x7fcf0c137be0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fcef400b3f0 tx=0x7fcef400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:59.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.851+0000 7fcf03400640 1 -- 172.21.15.102:0/262215220 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef4011830 con 0x7fcf0c143460 2024-03-23T03:47:59.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.851+0000 7fcf03400640 1 -- 172.21.15.102:0/262215220 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcef4011990 con 0x7fcf0c143460 2024-03-23T03:47:59.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.851+0000 7fcf03400640 1 -- 172.21.15.102:0/262215220 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcef4011d40 con 0x7fcf0c143460 2024-03-23T03:47:59.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.851+0000 7fcf0be00640 1 -- 172.21.15.102:0/262215220 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0c138b10 con 0x7fcf0c143460 2024-03-23T03:47:59.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.851+0000 7fcf0be00640 1 -- 172.21.15.102:0/262215220 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf0c138120 con 0x7fcf0c143460 2024-03-23T03:47:59.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.852+0000 7fcf0be00640 1 -- 172.21.15.102:0/262215220 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf0c04e030 con 0x7fcf0c143460 2024-03-23T03:47:59.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.853+0000 7fcf03400640 1 -- 172.21.15.102:0/262215220 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcef401a400 con 0x7fcf0c143460 2024-03-23T03:47:59.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.854+0000 7fcf03400640 1 --2- 172.21.15.102:0/262215220 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcee806ac00 0x7fcee806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:47:59.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.854+0000 7fcf03400640 1 -- 172.21.15.102:0/262215220 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fcef4089570 con 0x7fcf0c143460 2024-03-23T03:47:59.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.854+0000 7fcf0aa00640 1 --2- 172.21.15.102:0/262215220 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcee806ac00 0x7fcee806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:47:59.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.855+0000 7fcf0aa00640 1 --2- 172.21.15.102:0/262215220 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcee806ac00 0x7fcee806d0c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcefc036000 tx=0x7fcefc008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:47:59.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:47:59.859+0000 7fcf03400640 1 -- 172.21.15.102:0/262215220 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcef4054380 con 0x7fcf0c143460 2024-03-23T03:48:00.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.039+0000 7fcf0be00640 1 -- 172.21.15.102:0/262215220 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf0c15da50 con 0x7fcf0c143460 2024-03-23T03:48:00.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.043+0000 7fcf03400640 1 -- 172.21.15.102:0/262215220 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18260 (secure 0 0 0) 0x7fcef40579a0 con 0x7fcf0c143460 2024-03-23T03:48:00.044 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:00.044 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":28,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:47:41.647786+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:48:00.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.045+0000 7fcf0be00640 1 -- 172.21.15.102:0/262215220 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcee806ac00 msgr2=0x7fcee806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:00.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.045+0000 7fcf0be00640 1 --2- 172.21.15.102:0/262215220 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcee806ac00 0x7fcee806d0c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcefc036000 tx=0x7fcefc008030 comp rx=0 tx=0).stop 2024-03-23T03:48:00.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.045+0000 7fcf0be00640 1 -- 172.21.15.102:0/262215220 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf0c143460 msgr2=0x7fcf0c137be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:00.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.045+0000 7fcf0be00640 1 --2- 172.21.15.102:0/262215220 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf0c143460 0x7fcf0c137be0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fcef400b3f0 tx=0x7fcef400b8b0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.045+0000 7fcf0be00640 1 -- 172.21.15.102:0/262215220 shutdown_connections 2024-03-23T03:48:00.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.045+0000 7fcf0be00640 1 --2- 172.21.15.102:0/262215220 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcee806ac00 0x7fcee806d0c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.045+0000 7fcf0be00640 1 --2- 172.21.15.102:0/262215220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf0c143930 0x7fcf0c06c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.045+0000 7fcf0be00640 1 --2- 172.21.15.102:0/262215220 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf0c143460 0x7fcf0c137be0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.045+0000 7fcf0be00640 1 --2- 172.21.15.102:0/262215220 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf0c0cde20 0x7fcf0c15a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.045+0000 7fcf0be00640 1 -- 172.21.15.102:0/262215220 >> 172.21.15.102:0/262215220 conn(0x7fcf0c065df0 msgr2=0x7fcf0c0b1a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:00.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.045+0000 7fcf0be00640 1 -- 172.21.15.102:0/262215220 shutdown_connections 2024-03-23T03:48:00.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.045+0000 7fcf0be00640 1 -- 172.21.15.102:0/262215220 wait complete. 2024-03-23T03:48:00.064 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:46:52.158650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:46:52.914275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-23T03:46:55.983754+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-23T03:48:00.065 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:48:00.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.274+0000 7fe187e00640 1 Processor -- start 2024-03-23T03:48:00.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.274+0000 7fe187e00640 1 -- start start 2024-03-23T03:48:00.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.276+0000 7fe187e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe180159830 0x7fe1800cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:00.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.276+0000 7fe187e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe1800ca7b0 0x7fe1800cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:00.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.276+0000 7fe187e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe180159c10 con 0x7fe1800ce930 2024-03-23T03:48:00.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.276+0000 7fe187e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe18006de50 con 0x7fe1800ca7b0 2024-03-23T03:48:00.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.276+0000 7fe187e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1800cd840 con 0x7fe180159830 2024-03-23T03:48:00.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.276+0000 7fe185e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe1800ce930 0x7fe1800b18d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41034/0 (socket says 172.21.15.102:41034) 2024-03-23T03:48:00.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.276+0000 7fe185e00640 1 -- 172.21.15.102:0/97420677 learned_addr learned my addr 172.21.15.102:0/97420677 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:00.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.276+0000 7fe185400640 1 --2- 172.21.15.102:0/97420677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe180159830 0x7fe1800cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:00.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.277+0000 7fe184a00640 1 --2- 172.21.15.102:0/97420677 >> v2:172.21.15.173:3300/0 conn(0x7fe1800ca7b0 0x7fe1800cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:00.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.277+0000 7fe17fe00640 1 -- 172.21.15.102:0/97420677 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912585065 0 0) 0x7fe180159c10 con 0x7fe1800ce930 2024-03-23T03:48:00.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.277+0000 7fe17fe00640 1 -- 172.21.15.102:0/97420677 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe168003520 con 0x7fe1800ce930 2024-03-23T03:48:00.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.277+0000 7fe17fe00640 1 -- 172.21.15.102:0/97420677 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1777275142 0 0) 0x7fe168003520 con 0x7fe1800ce930 2024-03-23T03:48:00.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.277+0000 7fe17fe00640 1 -- 172.21.15.102:0/97420677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe180159830 msgr2=0x7fe1800cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:00.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.277+0000 7fe17fe00640 1 --2- 172.21.15.102:0/97420677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe180159830 0x7fe1800cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.277+0000 7fe17fe00640 1 -- 172.21.15.102:0/97420677 >> v2:172.21.15.173:3300/0 conn(0x7fe1800ca7b0 msgr2=0x7fe1800cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:00.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.277+0000 7fe17fe00640 1 --2- 172.21.15.102:0/97420677 >> v2:172.21.15.173:3300/0 conn(0x7fe1800ca7b0 0x7fe1800cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.277+0000 7fe17fe00640 1 -- 172.21.15.102:0/97420677 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1800ccc30 con 0x7fe1800ce930 2024-03-23T03:48:00.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.277+0000 7fe17fe00640 1 -- 172.21.15.102:0/97420677 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fe1780019f0 con 0x7fe1800ce930 2024-03-23T03:48:00.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.277+0000 7fe17fe00640 1 -- 172.21.15.102:0/97420677 >> v1:172.21.15.102:6789/0 conn(0x7fe1800ce930 legacy=0x7fe1800b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:00.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.277+0000 7fe17fe00640 1 --2- 172.21.15.102:0/97420677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe180159830 0x7fe168003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:00.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.277+0000 7fe17fe00640 1 --2- 172.21.15.102:0/97420677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe168003ef0 0x7fe1680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:00.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.277+0000 7fe17fe00640 1 --2- 172.21.15.102:0/97420677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe168004d00 0x7fe168005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:00.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.277+0000 7fe17fe00640 1 -- 172.21.15.102:0/97420677 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1800cd840 con 0x7fe168004d00 2024-03-23T03:48:00.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.277+0000 7fe17fe00640 1 -- 172.21.15.102:0/97420677 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe16800b980 con 0x7fe168003ef0 2024-03-23T03:48:00.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.277+0000 7fe17fe00640 1 -- 172.21.15.102:0/97420677 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe16800baf0 con 0x7fe180159830 2024-03-23T03:48:00.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.278+0000 7fe185e00640 1 --2- 172.21.15.102:0/97420677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe168004d00 0x7fe168005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:00.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.278+0000 7fe184a00640 1 --2- 172.21.15.102:0/97420677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe168003ef0 0x7fe1680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:00.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.278+0000 7fe185400640 1 --2- 172.21.15.102:0/97420677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe180159830 0x7fe168003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:00.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.278+0000 7fe185e00640 1 -- 172.21.15.102:0/97420677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe180159830 msgr2=0x7fe168003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:00.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.278+0000 7fe185e00640 1 --2- 172.21.15.102:0/97420677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe180159830 0x7fe168003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.278+0000 7fe185e00640 1 -- 172.21.15.102:0/97420677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe168003ef0 msgr2=0x7fe1680047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:00.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.278+0000 7fe185e00640 1 --2- 172.21.15.102:0/97420677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe168003ef0 0x7fe1680047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.278+0000 7fe185e00640 1 -- 172.21.15.102:0/97420677 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe168003520 con 0x7fe168004d00 2024-03-23T03:48:00.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.279+0000 7fe185e00640 1 --2- 172.21.15.102:0/97420677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe168004d00 0x7fe168005180 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe17800a860 tx=0x7fe1680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1cbcae48dbe46b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:00.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.280+0000 7fe17fe00640 1 -- 172.21.15.102:0/97420677 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178034630 con 0x7fe168004d00 2024-03-23T03:48:00.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.280+0000 7fe17fe00640 1 -- 172.21.15.102:0/97420677 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe17802f070 con 0x7fe168004d00 2024-03-23T03:48:00.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.280+0000 7fe17fe00640 1 -- 172.21.15.102:0/97420677 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178034a30 con 0x7fe168004d00 2024-03-23T03:48:00.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.280+0000 7fe187e00640 1 -- 172.21.15.102:0/97420677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe168004d00 msgr2=0x7fe168005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:00.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.280+0000 7fe187e00640 1 --2- 172.21.15.102:0/97420677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe168004d00 0x7fe168005180 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe17800a860 tx=0x7fe1680057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.280+0000 7fe185e00640 1 -- 172.21.15.102:0/97420677 reap_dead start 2024-03-23T03:48:00.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.280+0000 7fe187e00640 1 -- 172.21.15.102:0/97420677 shutdown_connections 2024-03-23T03:48:00.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.280+0000 7fe187e00640 1 -- 172.21.15.102:0/97420677 >> 172.21.15.102:0/97420677 conn(0x7fe18006a0b0 msgr2=0x7fe18006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:00.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.280+0000 7fe187e00640 1 -- 172.21.15.102:0/97420677 shutdown_connections 2024-03-23T03:48:00.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.281+0000 7fe187e00640 1 -- 172.21.15.102:0/97420677 wait complete. 2024-03-23T03:48:00.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.283+0000 7fe187e00640 1 Processor -- start 2024-03-23T03:48:00.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.283+0000 7fe187e00640 1 -- start start 2024-03-23T03:48:00.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.284+0000 7fe187e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe168004d00 0x7fe1800afc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:00.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.284+0000 7fe187e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe1800ae310 0x7fe180140750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:00.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.284+0000 7fe187e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe18014e160 0x7fe1800b1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:00.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.284+0000 7fe187e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe180167370 con 0x7fe18014e160 2024-03-23T03:48:00.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.284+0000 7fe187e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe180143790 con 0x7fe168004d00 2024-03-23T03:48:00.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.284+0000 7fe187e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1800ae790 con 0x7fe1800ae310 2024-03-23T03:48:00.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.284+0000 7fe185e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe18014e160 0x7fe1800b1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:00.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.285+0000 7fe185e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe18014e160 0x7fe1800b1880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54276/0 (socket says 172.21.15.102:54276) 2024-03-23T03:48:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.285+0000 7fe185e00640 1 -- 172.21.15.102:0/3123662615 learned_addr learned my addr 172.21.15.102:0/3123662615 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.285+0000 7fe184a00640 1 --2- 172.21.15.102:0/3123662615 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe1800ae310 0x7fe180140750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.285+0000 7fe185400640 1 --2- 172.21.15.102:0/3123662615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe168004d00 0x7fe1800afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.285+0000 7fe185e00640 1 -- 172.21.15.102:0/3123662615 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe1800ae310 msgr2=0x7fe180140750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.285+0000 7fe185e00640 1 --2- 172.21.15.102:0/3123662615 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe1800ae310 0x7fe180140750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.285+0000 7fe185e00640 1 -- 172.21.15.102:0/3123662615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe168004d00 msgr2=0x7fe1800afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.285+0000 7fe185e00640 1 --2- 172.21.15.102:0/3123662615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe168004d00 0x7fe1800afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.285+0000 7fe185e00640 1 -- 172.21.15.102:0/3123662615 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe180166db0 con 0x7fe18014e160 2024-03-23T03:48:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.285+0000 7fe185400640 1 --2- 172.21.15.102:0/3123662615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe168004d00 0x7fe1800afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.285+0000 7fe185e00640 1 --2- 172.21.15.102:0/3123662615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe18014e160 0x7fe1800b1880 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe17800a860 tx=0x7fe178001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.285+0000 7fe17d600640 1 -- 172.21.15.102:0/3123662615 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178034840 con 0x7fe18014e160 2024-03-23T03:48:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.285+0000 7fe17d600640 1 -- 172.21.15.102:0/3123662615 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1780349a0 con 0x7fe18014e160 2024-03-23T03:48:00.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.285+0000 7fe187e00640 1 -- 172.21.15.102:0/3123662615 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe180167090 con 0x7fe18014e160 2024-03-23T03:48:00.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.286+0000 7fe17d600640 1 -- 172.21.15.102:0/3123662615 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe178034d50 con 0x7fe18014e160 2024-03-23T03:48:00.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.286+0000 7fe187e00640 1 -- 172.21.15.102:0/3123662615 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe180140c90 con 0x7fe18014e160 2024-03-23T03:48:00.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.287+0000 7fe187e00640 1 -- 172.21.15.102:0/3123662615 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18014de80 con 0x7fe18014e160 2024-03-23T03:48:00.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.288+0000 7fe17d600640 1 -- 172.21.15.102:0/3123662615 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe178031030 con 0x7fe18014e160 2024-03-23T03:48:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.289+0000 7fe17d600640 1 --2- 172.21.15.102:0/3123662615 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe16006ac00 0x7fe16006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.289+0000 7fe17d600640 1 -- 172.21.15.102:0/3123662615 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe1780aadb0 con 0x7fe18014e160 2024-03-23T03:48:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.289+0000 7fe185400640 1 --2- 172.21.15.102:0/3123662615 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe16006ac00 0x7fe16006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:00.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.290+0000 7fe185400640 1 --2- 172.21.15.102:0/3123662615 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe16006ac00 0x7fe16006d0c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe174009ab0 tx=0x7fe174008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:00.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.294+0000 7fe17d600640 1 -- 172.21.15.102:0/3123662615 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe18014de80 con 0x7fe18014e160 2024-03-23T03:48:00.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.486+0000 7fe187e00640 1 -- 172.21.15.102:0/3123662615 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe18014de80 con 0x7fe18014e160 2024-03-23T03:48:00.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.487+0000 7fe17d600640 1 -- 172.21.15.102:0/3123662615 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe18014de80 con 0x7fe18014e160 2024-03-23T03:48:00.488 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:48:00.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.490+0000 7fe187e00640 1 -- 172.21.15.102:0/3123662615 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe16006ac00 msgr2=0x7fe16006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:00.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.490+0000 7fe187e00640 1 --2- 172.21.15.102:0/3123662615 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe16006ac00 0x7fe16006d0c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fe174009ab0 tx=0x7fe174008040 comp rx=0 tx=0).stop 2024-03-23T03:48:00.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.490+0000 7fe187e00640 1 -- 172.21.15.102:0/3123662615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe18014e160 msgr2=0x7fe1800b1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:00.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.490+0000 7fe187e00640 1 --2- 172.21.15.102:0/3123662615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe18014e160 0x7fe1800b1880 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe17800a860 tx=0x7fe178001d80 comp rx=0 tx=0).stop 2024-03-23T03:48:00.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.490+0000 7fe187e00640 1 -- 172.21.15.102:0/3123662615 shutdown_connections 2024-03-23T03:48:00.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.490+0000 7fe187e00640 1 --2- 172.21.15.102:0/3123662615 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe16006ac00 0x7fe16006d0c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.490+0000 7fe187e00640 1 --2- 172.21.15.102:0/3123662615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe18014e160 0x7fe1800b1880 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.490+0000 7fe187e00640 1 --2- 172.21.15.102:0/3123662615 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe1800ae310 0x7fe180140750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.490+0000 7fe187e00640 1 --2- 172.21.15.102:0/3123662615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe168004d00 0x7fe1800afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.490+0000 7fe187e00640 1 -- 172.21.15.102:0/3123662615 >> 172.21.15.102:0/3123662615 conn(0x7fe18006a0b0 msgr2=0x7fe18006d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:00.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.490+0000 7fe187e00640 1 -- 172.21.15.102:0/3123662615 shutdown_connections 2024-03-23T03:48:00.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.490+0000 7fe187e00640 1 -- 172.21.15.102:0/3123662615 wait complete. 2024-03-23T03:48:00.509 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:48:00.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.718+0000 7f914be00640 1 Processor -- start 2024-03-23T03:48:00.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.718+0000 7f914be00640 1 -- start start 2024-03-23T03:48:00.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.720+0000 7f914be00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f914406ce90 0x7f914406d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:00.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.720+0000 7f914be00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91440cdba0 con 0x7f91440ce930 2024-03-23T03:48:00.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.720+0000 7f914be00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91440cde80 con 0x7f91440add30 2024-03-23T03:48:00.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.720+0000 7f914be00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91440ae1c0 con 0x7f914406ce90 2024-03-23T03:48:00.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.720+0000 7f9149400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f91440ce930 0x7f91440cd3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41050/0 (socket says 172.21.15.102:41050) 2024-03-23T03:48:00.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.720+0000 7f9149400640 1 -- 172.21.15.102:0/3148821441 learned_addr learned my addr 172.21.15.102:0/3148821441 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:00.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.720+0000 7f9149e00640 1 --2- 172.21.15.102:0/3148821441 >> v2:172.21.15.173:3300/0 conn(0x7f914406ce90 0x7f914406d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:00.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.720+0000 7f9143e00640 1 -- 172.21.15.102:0/3148821441 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879849135 0 0) 0x7f91440cdba0 con 0x7f91440ce930 2024-03-23T03:48:00.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.720+0000 7f9143e00640 1 -- 172.21.15.102:0/3148821441 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f912c003520 con 0x7f91440ce930 2024-03-23T03:48:00.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.721+0000 7f9143e00640 1 -- 172.21.15.102:0/3148821441 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3285819750 0 0) 0x7f912c003520 con 0x7f91440ce930 2024-03-23T03:48:00.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.721+0000 7f9143e00640 1 -- 172.21.15.102:0/3148821441 >> v2:172.21.15.173:3300/0 conn(0x7f914406ce90 msgr2=0x7f914406d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:00.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.721+0000 7f9143e00640 1 --2- 172.21.15.102:0/3148821441 >> v2:172.21.15.173:3300/0 conn(0x7f914406ce90 0x7f914406d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.721+0000 7f9143e00640 1 -- 172.21.15.102:0/3148821441 >> v1:172.21.15.173:6789/0 conn(0x7f91440add30 legacy=0x7f91440ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:00.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.721+0000 7f9143e00640 1 -- 172.21.15.102:0/3148821441 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91440ce160 con 0x7f91440ce930 2024-03-23T03:48:00.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.721+0000 7f9143e00640 1 -- 172.21.15.102:0/3148821441 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f9138001670 con 0x7f91440ce930 2024-03-23T03:48:00.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.721+0000 7f9143e00640 1 -- 172.21.15.102:0/3148821441 >> v1:172.21.15.102:6789/0 conn(0x7f91440ce930 legacy=0x7f91440cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:00.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.721+0000 7f9143e00640 1 --2- 172.21.15.102:0/3148821441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f912c003970 0x7f912c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:00.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.721+0000 7f9143e00640 1 --2- 172.21.15.102:0/3148821441 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f912c0047c0 0x7f912c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:00.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.721+0000 7f9143e00640 1 --2- 172.21.15.102:0/3148821441 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f912c005180 0x7f912c00fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:00.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.721+0000 7f9143e00640 1 -- 172.21.15.102:0/3148821441 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912c010490 con 0x7f912c0047c0 2024-03-23T03:48:00.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.721+0000 7f9143e00640 1 -- 172.21.15.102:0/3148821441 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912c010600 con 0x7f912c003970 2024-03-23T03:48:00.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.721+0000 7f9143e00640 1 -- 172.21.15.102:0/3148821441 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912c0107c0 con 0x7f912c005180 2024-03-23T03:48:00.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.721+0000 7f9148a00640 1 --2- 172.21.15.102:0/3148821441 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f912c0047c0 0x7f912c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:00.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.721+0000 7f9149400640 1 --2- 172.21.15.102:0/3148821441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f912c003970 0x7f912c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:00.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.721+0000 7f9149e00640 1 --2- 172.21.15.102:0/3148821441 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f912c005180 0x7f912c00fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:00.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.722+0000 7f9148a00640 1 -- 172.21.15.102:0/3148821441 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f912c005180 msgr2=0x7f912c00fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:00.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.722+0000 7f9148a00640 1 --2- 172.21.15.102:0/3148821441 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f912c005180 0x7f912c00fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.722+0000 7f9148a00640 1 -- 172.21.15.102:0/3148821441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f912c003970 msgr2=0x7f912c003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:00.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.722+0000 7f9148a00640 1 --2- 172.21.15.102:0/3148821441 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f912c003970 0x7f912c003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.722+0000 7f9148a00640 1 -- 172.21.15.102:0/3148821441 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912c003520 con 0x7f912c0047c0 2024-03-23T03:48:00.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.722+0000 7f9149e00640 1 -- 172.21.15.102:0/3148821441 reap_dead start 2024-03-23T03:48:00.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.723+0000 7f9148a00640 1 --2- 172.21.15.102:0/3148821441 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f912c0047c0 0x7f912c004c40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9134003d80 tx=0x7f912c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f77ca2c6d156b285 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:00.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.723+0000 7f9143e00640 1 -- 172.21.15.102:0/3148821441 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9134032600 con 0x7f912c0047c0 2024-03-23T03:48:00.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.723+0000 7f9143e00640 1 -- 172.21.15.102:0/3148821441 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913402e050 con 0x7f912c0047c0 2024-03-23T03:48:00.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.723+0000 7f9143e00640 1 -- 172.21.15.102:0/3148821441 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91340329e0 con 0x7f912c0047c0 2024-03-23T03:48:00.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.724+0000 7f914be00640 1 -- 172.21.15.102:0/3148821441 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f912c0047c0 msgr2=0x7f912c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:00.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.724+0000 7f914be00640 1 --2- 172.21.15.102:0/3148821441 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f912c0047c0 0x7f912c004c40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9134003d80 tx=0x7f912c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.724+0000 7f914be00640 1 -- 172.21.15.102:0/3148821441 shutdown_connections 2024-03-23T03:48:00.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.724+0000 7f914be00640 1 --2- 172.21.15.102:0/3148821441 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f912c0047c0 0x7f912c004c40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.724+0000 7f914be00640 1 -- 172.21.15.102:0/3148821441 >> 172.21.15.102:0/3148821441 conn(0x7f914406a0b0 msgr2=0x7f914406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:00.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.724+0000 7f914be00640 1 -- 172.21.15.102:0/3148821441 shutdown_connections 2024-03-23T03:48:00.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.724+0000 7f914be00640 1 -- 172.21.15.102:0/3148821441 wait complete. 2024-03-23T03:48:00.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.726+0000 7f914be00640 1 Processor -- start 2024-03-23T03:48:00.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.726+0000 7f914be00640 1 -- start start 2024-03-23T03:48:00.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.727+0000 7f914be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f912c0047c0 0x7f914414ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:00.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.727+0000 7f914be00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91440cc040 0x7f91440cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:00.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.727+0000 7f914be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91440cc4c0 0x7f91440cb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:00.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.727+0000 7f914be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914415cf70 con 0x7f912c0047c0 2024-03-23T03:48:00.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.727+0000 7f914be00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91440aeaf0 con 0x7f91440cc4c0 2024-03-23T03:48:00.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.727+0000 7f914be00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914415c2f0 con 0x7f91440cc040 2024-03-23T03:48:00.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.727+0000 7f9149400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f912c0047c0 0x7f914414ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:00.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.727+0000 7f9149400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f912c0047c0 0x7f914414ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54286/0 (socket says 172.21.15.102:54286) 2024-03-23T03:48:00.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.727+0000 7f9149400640 1 -- 172.21.15.102:0/417938642 learned_addr learned my addr 172.21.15.102:0/417938642 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:00.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.727+0000 7f9148a00640 1 --2- 172.21.15.102:0/417938642 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91440cc040 0x7f91440cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:00.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.727+0000 7f9149400640 1 -- 172.21.15.102:0/417938642 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91440cc040 msgr2=0x7f91440cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:00.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.727+0000 7f9149400640 1 --2- 172.21.15.102:0/417938642 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91440cc040 0x7f91440cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.727+0000 7f9149400640 1 -- 172.21.15.102:0/417938642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91440cc4c0 msgr2=0x7f91440cb340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:00.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.727+0000 7f9149400640 1 --2- 172.21.15.102:0/417938642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91440cc4c0 0x7f91440cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.727+0000 7f9149400640 1 -- 172.21.15.102:0/417938642 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914415c5d0 con 0x7f912c0047c0 2024-03-23T03:48:00.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.728+0000 7f9149400640 1 --2- 172.21.15.102:0/417938642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f912c0047c0 0x7f914414ab10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9138001b70 tx=0x7f913800eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:00.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.728+0000 7f9141600640 1 -- 172.21.15.102:0/417938642 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913800c660 con 0x7f912c0047c0 2024-03-23T03:48:00.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.728+0000 7f9141600640 1 -- 172.21.15.102:0/417938642 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913800c7c0 con 0x7f912c0047c0 2024-03-23T03:48:00.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.728+0000 7f9141600640 1 -- 172.21.15.102:0/417938642 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913800cb70 con 0x7f912c0047c0 2024-03-23T03:48:00.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.728+0000 7f914be00640 1 -- 172.21.15.102:0/417938642 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914415c8b0 con 0x7f912c0047c0 2024-03-23T03:48:00.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.728+0000 7f914be00640 1 -- 172.21.15.102:0/417938642 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f914414b050 con 0x7f912c0047c0 2024-03-23T03:48:00.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.729+0000 7f914be00640 1 -- 172.21.15.102:0/417938642 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9144073390 con 0x7f912c0047c0 2024-03-23T03:48:00.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.731+0000 7f9141600640 1 -- 172.21.15.102:0/417938642 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f913800ccd0 con 0x7f912c0047c0 2024-03-23T03:48:00.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.731+0000 7f9141600640 1 --2- 172.21.15.102:0/417938642 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f912806ac00 0x7f912806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:00.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.732+0000 7f9141600640 1 -- 172.21.15.102:0/417938642 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f9138089650 con 0x7f912c0047c0 2024-03-23T03:48:00.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.732+0000 7f9148a00640 1 --2- 172.21.15.102:0/417938642 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f912806ac00 0x7f912806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:00.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.732+0000 7f9148a00640 1 --2- 172.21.15.102:0/417938642 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f912806ac00 0x7f912806d0c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9134034000 tx=0x7f9134009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:00.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.736+0000 7f9141600640 1 -- 172.21.15.102:0/417938642 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9138054470 con 0x7f912c0047c0 2024-03-23T03:48:00.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.930+0000 7f914be00640 1 -- 172.21.15.102:0/417938642 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f914406c540 con 0x7f912c0047c0 2024-03-23T03:48:00.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.931+0000 7f9141600640 1 -- 172.21.15.102:0/417938642 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9138057a90 con 0x7f912c0047c0 2024-03-23T03:48:00.932 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:48:00.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.934+0000 7f914be00640 1 -- 172.21.15.102:0/417938642 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f912806ac00 msgr2=0x7f912806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:00.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.934+0000 7f914be00640 1 --2- 172.21.15.102:0/417938642 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f912806ac00 0x7f912806d0c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9134034000 tx=0x7f9134009030 comp rx=0 tx=0).stop 2024-03-23T03:48:00.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.934+0000 7f914be00640 1 -- 172.21.15.102:0/417938642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f912c0047c0 msgr2=0x7f914414ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:00.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.934+0000 7f914be00640 1 --2- 172.21.15.102:0/417938642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f912c0047c0 0x7f914414ab10 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9138001b70 tx=0x7f913800eb50 comp rx=0 tx=0).stop 2024-03-23T03:48:00.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.934+0000 7f914be00640 1 -- 172.21.15.102:0/417938642 shutdown_connections 2024-03-23T03:48:00.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.934+0000 7f914be00640 1 --2- 172.21.15.102:0/417938642 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f912806ac00 0x7f912806d0c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.934+0000 7f914be00640 1 --2- 172.21.15.102:0/417938642 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91440cc4c0 0x7f91440cb340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.934+0000 7f914be00640 1 --2- 172.21.15.102:0/417938642 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91440cc040 0x7f91440cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.934+0000 7f914be00640 1 --2- 172.21.15.102:0/417938642 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f912c0047c0 0x7f914414ab10 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:00.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.934+0000 7f914be00640 1 -- 172.21.15.102:0/417938642 >> 172.21.15.102:0/417938642 conn(0x7f914406a0b0 msgr2=0x7f914414db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:00.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.934+0000 7f914be00640 1 -- 172.21.15.102:0/417938642 shutdown_connections 2024-03-23T03:48:00.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:00.934+0000 7f914be00640 1 -- 172.21.15.102:0/417938642 wait complete. 2024-03-23T03:48:00.954 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:48:01.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.168+0000 7fc794200640 1 Processor -- start 2024-03-23T03:48:01.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.168+0000 7fc794200640 1 -- start start 2024-03-23T03:48:01.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.170+0000 7fc794200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc78c1456a0 0x7fc78c0ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:01.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.170+0000 7fc794200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78c0ac6d0 con 0x7fc78c070cd0 2024-03-23T03:48:01.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.170+0000 7fc794200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc78c06d390 con 0x7fc78c145c20 2024-03-23T03:48:01.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.170+0000 7fc794200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0ad890 con 0x7fc78c1456a0 2024-03-23T03:48:01.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.170+0000 7fc790e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc78c070cd0 0x7fc78c0ab8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41064/0 (socket says 172.21.15.102:41064) 2024-03-23T03:48:01.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.170+0000 7fc790e00640 1 -- 172.21.15.102:0/3573055864 learned_addr learned my addr 172.21.15.102:0/3573055864 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:01.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.170+0000 7fc791800640 1 --2- 172.21.15.102:0/3573055864 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc78c1456a0 0x7fc78c0ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:01.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.171+0000 7fc78be00640 1 -- 172.21.15.102:0/3573055864 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726529194 0 0) 0x7fc78c0ac6d0 con 0x7fc78c070cd0 2024-03-23T03:48:01.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.171+0000 7fc78be00640 1 -- 172.21.15.102:0/3573055864 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc774003520 con 0x7fc78c070cd0 2024-03-23T03:48:01.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.171+0000 7fc78be00640 1 -- 172.21.15.102:0/3573055864 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1966859496 0 0) 0x7fc774003520 con 0x7fc78c070cd0 2024-03-23T03:48:01.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.171+0000 7fc78be00640 1 -- 172.21.15.102:0/3573055864 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc78c1456a0 msgr2=0x7fc78c0ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:01.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.171+0000 7fc78be00640 1 --2- 172.21.15.102:0/3573055864 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc78c1456a0 0x7fc78c0ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.171+0000 7fc78be00640 1 -- 172.21.15.102:0/3573055864 >> v1:172.21.15.173:6789/0 conn(0x7fc78c145c20 legacy=0x7fc78c0abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:01.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.171+0000 7fc78be00640 1 -- 172.21.15.102:0/3573055864 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c06d670 con 0x7fc78c070cd0 2024-03-23T03:48:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.171+0000 7fc78be00640 1 -- 172.21.15.102:0/3573055864 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fc778001670 con 0x7fc78c070cd0 2024-03-23T03:48:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.171+0000 7fc78be00640 1 -- 172.21.15.102:0/3573055864 >> v1:172.21.15.102:6789/0 conn(0x7fc78c070cd0 legacy=0x7fc78c0ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.171+0000 7fc78be00640 1 --2- 172.21.15.102:0/3573055864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7740039a0 0x7fc774003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.171+0000 7fc78be00640 1 --2- 172.21.15.102:0/3573055864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7740047c0 0x7fc774004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.171+0000 7fc78be00640 1 --2- 172.21.15.102:0/3573055864 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc78c1456a0 0x7fc774005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.171+0000 7fc78be00640 1 -- 172.21.15.102:0/3573055864 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c0ad890 con 0x7fc7740039a0 2024-03-23T03:48:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.171+0000 7fc78be00640 1 -- 172.21.15.102:0/3573055864 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77400ba30 con 0x7fc7740047c0 2024-03-23T03:48:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.171+0000 7fc78be00640 1 -- 172.21.15.102:0/3573055864 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc77400bba0 con 0x7fc78c1456a0 2024-03-23T03:48:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.172+0000 7fc791800640 1 --2- 172.21.15.102:0/3573055864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7740039a0 0x7fc774003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.172+0000 7fc790e00640 1 --2- 172.21.15.102:0/3573055864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7740047c0 0x7fc774004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.172+0000 7fc792200640 1 --2- 172.21.15.102:0/3573055864 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc78c1456a0 0x7fc774005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.172+0000 7fc791800640 1 -- 172.21.15.102:0/3573055864 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc78c1456a0 msgr2=0x7fc774005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.172+0000 7fc791800640 1 --2- 172.21.15.102:0/3573055864 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc78c1456a0 0x7fc774005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.172+0000 7fc791800640 1 -- 172.21.15.102:0/3573055864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7740047c0 msgr2=0x7fc774004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:01.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.172+0000 7fc791800640 1 --2- 172.21.15.102:0/3573055864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7740047c0 0x7fc774004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.172+0000 7fc791800640 1 -- 172.21.15.102:0/3573055864 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc774003520 con 0x7fc7740039a0 2024-03-23T03:48:01.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.174+0000 7fc791800640 1 --2- 172.21.15.102:0/3573055864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7740039a0 0x7fc774003d80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc780009f90 tx=0x7fc7740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b753b06cd17b8fbf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:01.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.174+0000 7fc78be00640 1 -- 172.21.15.102:0/3573055864 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78002cb90 con 0x7fc7740039a0 2024-03-23T03:48:01.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.174+0000 7fc78be00640 1 -- 172.21.15.102:0/3573055864 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78000f040 con 0x7fc7740039a0 2024-03-23T03:48:01.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.174+0000 7fc78be00640 1 -- 172.21.15.102:0/3573055864 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc780034440 con 0x7fc7740039a0 2024-03-23T03:48:01.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.175+0000 7fc794200640 1 -- 172.21.15.102:0/3573055864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7740039a0 msgr2=0x7fc774003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:01.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.175+0000 7fc794200640 1 --2- 172.21.15.102:0/3573055864 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7740039a0 0x7fc774003d80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc780009f90 tx=0x7fc7740057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.175+0000 7fc792200640 1 -- 172.21.15.102:0/3573055864 reap_dead start 2024-03-23T03:48:01.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.175+0000 7fc794200640 1 -- 172.21.15.102:0/3573055864 shutdown_connections 2024-03-23T03:48:01.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.175+0000 7fc794200640 1 -- 172.21.15.102:0/3573055864 >> 172.21.15.102:0/3573055864 conn(0x7fc78c06a0b0 msgr2=0x7fc78c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:01.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.175+0000 7fc794200640 1 -- 172.21.15.102:0/3573055864 shutdown_connections 2024-03-23T03:48:01.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.175+0000 7fc794200640 1 -- 172.21.15.102:0/3573055864 wait complete. 2024-03-23T03:48:01.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.177+0000 7fc794200640 1 Processor -- start 2024-03-23T03:48:01.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.177+0000 7fc794200640 1 -- start start 2024-03-23T03:48:01.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.178+0000 7fc794200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7740039a0 0x7fc78c138c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:01.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.178+0000 7fc794200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc78c1f1950 0x7fc78c1398c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:01.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.178+0000 7fc794200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc78c1415c0 0x7fc78c141a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:01.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.178+0000 7fc794200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c138800 con 0x7fc78c1f1950 2024-03-23T03:48:01.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.178+0000 7fc794200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c137b80 con 0x7fc78c1415c0 2024-03-23T03:48:01.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.178+0000 7fc794200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78c137e60 con 0x7fc7740039a0 2024-03-23T03:48:01.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.178+0000 7fc790e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc78c1f1950 0x7fc78c1398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:01.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.178+0000 7fc790e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc78c1f1950 0x7fc78c1398c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54304/0 (socket says 172.21.15.102:54304) 2024-03-23T03:48:01.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.178+0000 7fc790e00640 1 -- 172.21.15.102:0/2253430984 learned_addr learned my addr 172.21.15.102:0/2253430984 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:01.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.179+0000 7fc791800640 1 --2- 172.21.15.102:0/2253430984 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7740039a0 0x7fc78c138c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:01.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.179+0000 7fc792200640 1 --2- 172.21.15.102:0/2253430984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc78c1415c0 0x7fc78c141a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:01.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.179+0000 7fc790e00640 1 -- 172.21.15.102:0/2253430984 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7740039a0 msgr2=0x7fc78c138c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:01.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.179+0000 7fc790e00640 1 --2- 172.21.15.102:0/2253430984 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7740039a0 0x7fc78c138c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.179+0000 7fc790e00640 1 -- 172.21.15.102:0/2253430984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc78c1415c0 msgr2=0x7fc78c141a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:01.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.179+0000 7fc790e00640 1 --2- 172.21.15.102:0/2253430984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc78c1415c0 0x7fc78c141a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.179+0000 7fc790e00640 1 -- 172.21.15.102:0/2253430984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c138140 con 0x7fc78c1f1950 2024-03-23T03:48:01.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.179+0000 7fc790e00640 1 --2- 172.21.15.102:0/2253430984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc78c1f1950 0x7fc78c1398c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc778001050 tx=0x7fc77800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:01.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.179+0000 7fc789600640 1 -- 172.21.15.102:0/2253430984 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77800c870 con 0x7fc78c1f1950 2024-03-23T03:48:01.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.180+0000 7fc789600640 1 -- 172.21.15.102:0/2253430984 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc77800c9f0 con 0x7fc78c1f1950 2024-03-23T03:48:01.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.180+0000 7fc789600640 1 -- 172.21.15.102:0/2253430984 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77800cda0 con 0x7fc78c1f1950 2024-03-23T03:48:01.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.180+0000 7fc794200640 1 -- 172.21.15.102:0/2253430984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc78c138420 con 0x7fc78c1f1950 2024-03-23T03:48:01.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.180+0000 7fc794200640 1 -- 172.21.15.102:0/2253430984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc78c15c5c0 con 0x7fc78c1f1950 2024-03-23T03:48:01.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.181+0000 7fc794200640 1 -- 172.21.15.102:0/2253430984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc78c138940 con 0x7fc78c1f1950 2024-03-23T03:48:01.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.182+0000 7fc789600640 1 -- 172.21.15.102:0/2253430984 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc77801d460 con 0x7fc78c1f1950 2024-03-23T03:48:01.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.182+0000 7fc789600640 1 --2- 172.21.15.102:0/2253430984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc76c06ac30 0x7fc76c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:01.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.183+0000 7fc789600640 1 -- 172.21.15.102:0/2253430984 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc7780897c0 con 0x7fc78c1f1950 2024-03-23T03:48:01.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.183+0000 7fc791800640 1 --2- 172.21.15.102:0/2253430984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc76c06ac30 0x7fc76c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:01.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.183+0000 7fc791800640 1 --2- 172.21.15.102:0/2253430984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc76c06ac30 0x7fc76c06d0f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc780036000 tx=0x7fc78002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:01.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.187+0000 7fc789600640 1 -- 172.21.15.102:0/2253430984 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc78c138940 con 0x7fc78c1f1950 2024-03-23T03:48:01.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.379+0000 7fc794200640 1 -- 172.21.15.102:0/2253430984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc78c138940 con 0x7fc78c1f1950 2024-03-23T03:48:01.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.380+0000 7fc789600640 1 -- 172.21.15.102:0/2253430984 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc78c138940 con 0x7fc78c1f1950 2024-03-23T03:48:01.380 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:48:01.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.382+0000 7fc794200640 1 -- 172.21.15.102:0/2253430984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc76c06ac30 msgr2=0x7fc76c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:01.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.382+0000 7fc794200640 1 --2- 172.21.15.102:0/2253430984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc76c06ac30 0x7fc76c06d0f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc780036000 tx=0x7fc78002e000 comp rx=0 tx=0).stop 2024-03-23T03:48:01.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.382+0000 7fc794200640 1 -- 172.21.15.102:0/2253430984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc78c1f1950 msgr2=0x7fc78c1398c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:01.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.382+0000 7fc794200640 1 --2- 172.21.15.102:0/2253430984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc78c1f1950 0x7fc78c1398c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc778001050 tx=0x7fc77800ec60 comp rx=0 tx=0).stop 2024-03-23T03:48:01.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.382+0000 7fc794200640 1 -- 172.21.15.102:0/2253430984 shutdown_connections 2024-03-23T03:48:01.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.382+0000 7fc794200640 1 --2- 172.21.15.102:0/2253430984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc76c06ac30 0x7fc76c06d0f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.382+0000 7fc794200640 1 --2- 172.21.15.102:0/2253430984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc78c1415c0 0x7fc78c141a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.382+0000 7fc794200640 1 --2- 172.21.15.102:0/2253430984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc78c1f1950 0x7fc78c1398c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.382+0000 7fc794200640 1 --2- 172.21.15.102:0/2253430984 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7740039a0 0x7fc78c138c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.382+0000 7fc794200640 1 -- 172.21.15.102:0/2253430984 >> 172.21.15.102:0/2253430984 conn(0x7fc78c06a0b0 msgr2=0x7fc78c0b3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:01.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.383+0000 7fc794200640 1 -- 172.21.15.102:0/2253430984 shutdown_connections 2024-03-23T03:48:01.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.383+0000 7fc794200640 1 -- 172.21.15.102:0/2253430984 wait complete. 2024-03-23T03:48:01.401 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-23T03:48:01.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.599+0000 7f3ee0c00640 1 Processor -- start 2024-03-23T03:48:01.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.599+0000 7f3ee0c00640 1 -- start start 2024-03-23T03:48:01.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.601+0000 7f3ee0c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3edc15a850 0x7f3edc0cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:01.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.601+0000 7f3ee0c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3edc15e320 con 0x7f3edc0b22b0 2024-03-23T03:48:01.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.601+0000 7f3ee0c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3edc0cda20 con 0x7f3edc0cacf0 2024-03-23T03:48:01.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.601+0000 7f3ee0c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc15ac30 con 0x7f3edc15a850 2024-03-23T03:48:01.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.601+0000 7f3ed2a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3edc0b22b0 0x7f3edc15eae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41068/0 (socket says 172.21.15.102:41068) 2024-03-23T03:48:01.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.601+0000 7f3ed2a00640 1 -- 172.21.15.102:0/3221608889 learned_addr learned my addr 172.21.15.102:0/3221608889 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.601+0000 7f3edb400640 1 --2- 172.21.15.102:0/3221608889 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3edc15a850 0x7f3edc0cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.601+0000 7f3edaa00640 1 -- 172.21.15.102:0/3221608889 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080851447 0 0) 0x7f3edc15e320 con 0x7f3edc0b22b0 2024-03-23T03:48:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.602+0000 7f3edaa00640 1 -- 172.21.15.102:0/3221608889 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ebc003520 con 0x7f3edc0b22b0 2024-03-23T03:48:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.602+0000 7f3edaa00640 1 -- 172.21.15.102:0/3221608889 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3778173123 0 0) 0x7f3ebc003520 con 0x7f3edc0b22b0 2024-03-23T03:48:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.602+0000 7f3edaa00640 1 -- 172.21.15.102:0/3221608889 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3edc15a850 msgr2=0x7f3edc0cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.602+0000 7f3edaa00640 1 --2- 172.21.15.102:0/3221608889 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3edc15a850 0x7f3edc0cd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.602+0000 7f3edaa00640 1 -- 172.21.15.102:0/3221608889 >> v1:172.21.15.173:6789/0 conn(0x7f3edc0cacf0 legacy=0x7f3edc06d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.602+0000 7f3edaa00640 1 -- 172.21.15.102:0/3221608889 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc0cdd00 con 0x7f3edc0b22b0 2024-03-23T03:48:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.602+0000 7f3edaa00640 1 -- 172.21.15.102:0/3221608889 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f3ec4001670 con 0x7f3edc0b22b0 2024-03-23T03:48:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.602+0000 7f3edaa00640 1 -- 172.21.15.102:0/3221608889 >> v1:172.21.15.102:6789/0 conn(0x7f3edc0b22b0 legacy=0x7f3edc15eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.602+0000 7f3edaa00640 1 --2- 172.21.15.102:0/3221608889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ebc0039a0 0x7f3ebc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:01.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.602+0000 7f3edaa00640 1 --2- 172.21.15.102:0/3221608889 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3edc15a850 0x7f3ebc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.602+0000 7f3edaa00640 1 --2- 172.21.15.102:0/3221608889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ebc004d00 0x7f3ebc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.602+0000 7f3edaa00640 1 -- 172.21.15.102:0/3221608889 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc15ac30 con 0x7f3ebc004d00 2024-03-23T03:48:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.602+0000 7f3edaa00640 1 -- 172.21.15.102:0/3221608889 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc00ba60 con 0x7f3ebc0039a0 2024-03-23T03:48:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.602+0000 7f3edaa00640 1 -- 172.21.15.102:0/3221608889 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ebc00bc20 con 0x7f3edc15a850 2024-03-23T03:48:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.602+0000 7f3edbe00640 1 --2- 172.21.15.102:0/3221608889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ebc004d00 0x7f3ebc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.603+0000 7f3edb400640 1 --2- 172.21.15.102:0/3221608889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ebc0039a0 0x7f3ebc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.603+0000 7f3ed2a00640 1 --2- 172.21.15.102:0/3221608889 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3edc15a850 0x7f3ebc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.603+0000 7f3edbe00640 1 -- 172.21.15.102:0/3221608889 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3edc15a850 msgr2=0x7f3ebc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.603+0000 7f3edbe00640 1 --2- 172.21.15.102:0/3221608889 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3edc15a850 0x7f3ebc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.603+0000 7f3edbe00640 1 -- 172.21.15.102:0/3221608889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ebc0039a0 msgr2=0x7f3ebc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.603+0000 7f3edbe00640 1 --2- 172.21.15.102:0/3221608889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ebc0039a0 0x7f3ebc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.603+0000 7f3edbe00640 1 -- 172.21.15.102:0/3221608889 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ebc003520 con 0x7f3ebc004d00 2024-03-23T03:48:01.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.604+0000 7f3edbe00640 1 --2- 172.21.15.102:0/3221608889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ebc004d00 0x7f3ebc005180 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4001e20 tx=0x7f3ebc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc55d2281b5cebb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.604+0000 7f3edaa00640 1 -- 172.21.15.102:0/3221608889 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4003b70 con 0x7f3ebc004d00 2024-03-23T03:48:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.604+0000 7f3edaa00640 1 -- 172.21.15.102:0/3221608889 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed4003cd0 con 0x7f3ebc004d00 2024-03-23T03:48:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.604+0000 7f3edaa00640 1 -- 172.21.15.102:0/3221608889 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed4005310 con 0x7f3ebc004d00 2024-03-23T03:48:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.605+0000 7f3ee0c00640 1 -- 172.21.15.102:0/3221608889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ebc004d00 msgr2=0x7f3ebc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.605+0000 7f3ee0c00640 1 --2- 172.21.15.102:0/3221608889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3ebc004d00 0x7f3ebc005180 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4001e20 tx=0x7f3ebc0057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.605+0000 7f3edbe00640 1 -- 172.21.15.102:0/3221608889 reap_dead start 2024-03-23T03:48:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.605+0000 7f3ee0c00640 1 -- 172.21.15.102:0/3221608889 shutdown_connections 2024-03-23T03:48:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.605+0000 7f3ee0c00640 1 -- 172.21.15.102:0/3221608889 >> 172.21.15.102:0/3221608889 conn(0x7f3edc0b01b0 msgr2=0x7f3edc0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.605+0000 7f3ee0c00640 1 -- 172.21.15.102:0/3221608889 shutdown_connections 2024-03-23T03:48:01.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.605+0000 7f3ee0c00640 1 -- 172.21.15.102:0/3221608889 wait complete. 2024-03-23T03:48:01.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.607+0000 7f3ee0c00640 1 Processor -- start 2024-03-23T03:48:01.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.607+0000 7f3ee0c00640 1 -- start start 2024-03-23T03:48:01.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.608+0000 7f3ee0c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ebc004d00 0x7f3edc1f5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:01.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.608+0000 7f3ee0c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3edc14b690 0x7f3edc148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:01.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.608+0000 7f3ee0c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3edc069f00 0x7f3edc144e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:01.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.608+0000 7f3ee0c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc147fe0 con 0x7f3edc069f00 2024-03-23T03:48:01.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.608+0000 7f3ee0c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc09c330 con 0x7f3ebc004d00 2024-03-23T03:48:01.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.608+0000 7f3ee0c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc1f63f0 con 0x7f3edc14b690 2024-03-23T03:48:01.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.608+0000 7f3edbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3edc069f00 0x7f3edc144e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:01.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.608+0000 7f3edbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3edc069f00 0x7f3edc144e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54326/0 (socket says 172.21.15.102:54326) 2024-03-23T03:48:01.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.608+0000 7f3edbe00640 1 -- 172.21.15.102:0/627950152 learned_addr learned my addr 172.21.15.102:0/627950152 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:01.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.608+0000 7f3edb400640 1 --2- 172.21.15.102:0/627950152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ebc004d00 0x7f3edc1f5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:01.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.609+0000 7f3ed2a00640 1 --2- 172.21.15.102:0/627950152 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3edc14b690 0x7f3edc148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:01.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.609+0000 7f3edbe00640 1 -- 172.21.15.102:0/627950152 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3edc14b690 msgr2=0x7f3edc148400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:01.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.609+0000 7f3edbe00640 1 --2- 172.21.15.102:0/627950152 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3edc14b690 0x7f3edc148400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.609+0000 7f3edbe00640 1 -- 172.21.15.102:0/627950152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ebc004d00 msgr2=0x7f3edc1f5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:01.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.609+0000 7f3edbe00640 1 --2- 172.21.15.102:0/627950152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ebc004d00 0x7f3edc1f5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.609+0000 7f3edbe00640 1 -- 172.21.15.102:0/627950152 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc1f5e30 con 0x7f3edc069f00 2024-03-23T03:48:01.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.609+0000 7f3edbe00640 1 --2- 172.21.15.102:0/627950152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3edc069f00 0x7f3edc144e80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4001f40 tx=0x7f3ed40014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:01.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.609+0000 7f3ed3e00640 1 -- 172.21.15.102:0/627950152 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed402f640 con 0x7f3edc069f00 2024-03-23T03:48:01.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.610+0000 7f3ed3e00640 1 -- 172.21.15.102:0/627950152 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed402f7a0 con 0x7f3edc069f00 2024-03-23T03:48:01.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.610+0000 7f3ee0c00640 1 -- 172.21.15.102:0/627950152 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3edc1f6110 con 0x7f3edc069f00 2024-03-23T03:48:01.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.610+0000 7f3ed3e00640 1 -- 172.21.15.102:0/627950152 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed402fb50 con 0x7f3edc069f00 2024-03-23T03:48:01.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.610+0000 7f3ee0c00640 1 -- 172.21.15.102:0/627950152 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3edc1f5a90 con 0x7f3edc069f00 2024-03-23T03:48:01.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.611+0000 7f3ee0c00640 1 -- 172.21.15.102:0/627950152 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3edc04e030 con 0x7f3edc069f00 2024-03-23T03:48:01.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.612+0000 7f3ed3e00640 1 -- 172.21.15.102:0/627950152 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3ed402fcb0 con 0x7f3edc069f00 2024-03-23T03:48:01.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.613+0000 7f3ed3e00640 1 --2- 172.21.15.102:0/627950152 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3eb806ac00 0x7f3eb806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:01.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.613+0000 7f3ed3e00640 1 -- 172.21.15.102:0/627950152 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3ed40aaf50 con 0x7f3edc069f00 2024-03-23T03:48:01.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.613+0000 7f3edb400640 1 --2- 172.21.15.102:0/627950152 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3eb806ac00 0x7f3eb806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:01.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.614+0000 7f3edb400640 1 --2- 172.21.15.102:0/627950152 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3eb806ac00 0x7f3eb806d0c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc009ac0 tx=0x7f3ecc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:01.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.617+0000 7f3ed3e00640 1 -- 172.21.15.102:0/627950152 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3ed4075d80 con 0x7f3edc069f00 2024-03-23T03:48:01.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.813+0000 7f3ee0c00640 1 -- 172.21.15.102:0/627950152 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f3edc0ccaa0 con 0x7f3edc069f00 2024-03-23T03:48:01.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.814+0000 7f3ed3e00640 1 -- 172.21.15.102:0/627950152 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f3ed4047020 con 0x7f3edc069f00 2024-03-23T03:48:01.815 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:48:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.817+0000 7f3ee0c00640 1 -- 172.21.15.102:0/627950152 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3eb806ac00 msgr2=0x7f3eb806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.817+0000 7f3ee0c00640 1 --2- 172.21.15.102:0/627950152 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3eb806ac00 0x7f3eb806d0c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc009ac0 tx=0x7f3ecc008040 comp rx=0 tx=0).stop 2024-03-23T03:48:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.817+0000 7f3ee0c00640 1 -- 172.21.15.102:0/627950152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3edc069f00 msgr2=0x7f3edc144e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.817+0000 7f3ee0c00640 1 --2- 172.21.15.102:0/627950152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3edc069f00 0x7f3edc144e80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f3ed4001f40 tx=0x7f3ed40014a0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.817+0000 7f3ee0c00640 1 -- 172.21.15.102:0/627950152 shutdown_connections 2024-03-23T03:48:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.817+0000 7f3ee0c00640 1 --2- 172.21.15.102:0/627950152 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3eb806ac00 0x7f3eb806d0c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.817+0000 7f3ee0c00640 1 --2- 172.21.15.102:0/627950152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3edc069f00 0x7f3edc144e80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.817+0000 7f3ee0c00640 1 --2- 172.21.15.102:0/627950152 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3edc14b690 0x7f3edc148400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.817+0000 7f3ee0c00640 1 --2- 172.21.15.102:0/627950152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ebc004d00 0x7f3edc1f5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:01.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.817+0000 7f3ee0c00640 1 -- 172.21.15.102:0/627950152 >> 172.21.15.102:0/627950152 conn(0x7f3edc0b01b0 msgr2=0x7f3edc06dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:01.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.817+0000 7f3ee0c00640 1 -- 172.21.15.102:0/627950152 shutdown_connections 2024-03-23T03:48:01.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:01.817+0000 7f3ee0c00640 1 -- 172.21.15.102:0/627950152 wait complete. 2024-03-23T03:48:01.836 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:48:02.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.051+0000 7fc970c00640 1 Processor -- start 2024-03-23T03:48:02.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.051+0000 7fc970c00640 1 -- start start 2024-03-23T03:48:02.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.053+0000 7fc970c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc96c0cf7c0 0x7fc96c0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:02.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.053+0000 7fc970c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc96c160e90 0x7fc96c0d0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:02.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.053+0000 7fc970c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc96c0ccd90 con 0x7fc96c15c820 2024-03-23T03:48:02.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.053+0000 7fc970c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96c0cdd70 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.053+0000 7fc970c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc96c0b7fc0 con 0x7fc96c160e90 2024-03-23T03:48:02.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.053+0000 7fc96b400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc96c15c820 0x7fc96c0cf0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:41076/0 (socket says 172.21.15.102:41076) 2024-03-23T03:48:02.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.053+0000 7fc96b400640 1 -- 172.21.15.102:0/2831832604 learned_addr learned my addr 172.21.15.102:0/2831832604 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:02.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.053+0000 7fc96aa00640 1 --2- 172.21.15.102:0/2831832604 >> v2:172.21.15.102:3300/0 conn(0x7fc96c0cf7c0 0x7fc96c0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:02.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.053+0000 7fc96be00640 1 --2- 172.21.15.102:0/2831832604 >> v2:172.21.15.173:3300/0 conn(0x7fc96c160e90 0x7fc96c0d0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:02.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.053+0000 7fc96a000640 1 -- 172.21.15.102:0/2831832604 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242955293 0 0) 0x7fc96c0ccd90 con 0x7fc96c15c820 2024-03-23T03:48:02.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.053+0000 7fc96a000640 1 -- 172.21.15.102:0/2831832604 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc94c003520 con 0x7fc96c15c820 2024-03-23T03:48:02.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.054+0000 7fc96aa00640 1 -- 172.21.15.102:0/2831832604 >> v2:172.21.15.173:3300/0 conn(0x7fc96c160e90 msgr2=0x7fc96c0d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:02.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.054+0000 7fc96aa00640 1 --2- 172.21.15.102:0/2831832604 >> v2:172.21.15.173:3300/0 conn(0x7fc96c160e90 0x7fc96c0d0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:02.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.054+0000 7fc96aa00640 1 -- 172.21.15.102:0/2831832604 >> v1:172.21.15.102:6789/0 conn(0x7fc96c15c820 legacy=0x7fc96c0cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:02.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.054+0000 7fc96aa00640 1 -- 172.21.15.102:0/2831832604 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96c0b82a0 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.055+0000 7fc96aa00640 1 --2- 172.21.15.102:0/2831832604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc96c0cf7c0 0x7fc96c0cd830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc96c08b340 tx=0x7fc95402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595b3f3c85bd9f3e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:02.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.055+0000 7fc96a000640 1 -- 172.21.15.102:0/2831832604 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95402cd00 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.055+0000 7fc96a000640 1 -- 172.21.15.102:0/2831832604 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95402ce60 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.055+0000 7fc96a000640 1 -- 172.21.15.102:0/2831832604 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc954034660 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.055+0000 7fc970c00640 1 -- 172.21.15.102:0/2831832604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc96c0cf7c0 msgr2=0x7fc96c0cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:02.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.055+0000 7fc970c00640 1 --2- 172.21.15.102:0/2831832604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc96c0cf7c0 0x7fc96c0cd830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc96c08b340 tx=0x7fc95402bf50 comp rx=0 tx=0).stop 2024-03-23T03:48:02.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.056+0000 7fc970c00640 1 -- 172.21.15.102:0/2831832604 shutdown_connections 2024-03-23T03:48:02.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.056+0000 7fc970c00640 1 --2- 172.21.15.102:0/2831832604 >> v2:172.21.15.173:3300/0 conn(0x7fc96c160e90 0x7fc96c0d0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:02.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.056+0000 7fc970c00640 1 --2- 172.21.15.102:0/2831832604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc96c0cf7c0 0x7fc96c0cd830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:02.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.056+0000 7fc970c00640 1 -- 172.21.15.102:0/2831832604 >> 172.21.15.102:0/2831832604 conn(0x7fc96c06b070 msgr2=0x7fc96c06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:02.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.056+0000 7fc970c00640 1 -- 172.21.15.102:0/2831832604 shutdown_connections 2024-03-23T03:48:02.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.056+0000 7fc970c00640 1 -- 172.21.15.102:0/2831832604 wait complete. 2024-03-23T03:48:02.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.057+0000 7fc970c00640 1 Processor -- start 2024-03-23T03:48:02.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.057+0000 7fc970c00640 1 -- start start 2024-03-23T03:48:02.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.058+0000 7fc970c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc96c0cf7c0 0x7fc96c1f2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:02.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.058+0000 7fc970c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc96c15c820 0x7fc96c0b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:02.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.058+0000 7fc970c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc96c160e90 0x7fc96c15cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.058+0000 7fc970c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c0b0f80 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.058+0000 7fc970c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c0b2d30 con 0x7fc96c15c820 2024-03-23T03:48:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.058+0000 7fc970c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc96c0b3010 con 0x7fc96c160e90 2024-03-23T03:48:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.059+0000 7fc96b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc96c0cf7c0 0x7fc96c1f2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.059+0000 7fc96b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc96c0cf7c0 0x7fc96c1f2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54336/0 (socket says 172.21.15.102:54336) 2024-03-23T03:48:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.059+0000 7fc96b400640 1 -- 172.21.15.102:0/1523718156 learned_addr learned my addr 172.21.15.102:0/1523718156 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.059+0000 7fc96aa00640 1 --2- 172.21.15.102:0/1523718156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc96c15c820 0x7fc96c0b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.059+0000 7fc96b400640 1 -- 172.21.15.102:0/1523718156 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc96c160e90 msgr2=0x7fc96c15cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.059+0000 7fc96be00640 1 --2- 172.21.15.102:0/1523718156 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc96c160e90 0x7fc96c15cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.059+0000 7fc96b400640 1 --2- 172.21.15.102:0/1523718156 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc96c160e90 0x7fc96c15cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.059+0000 7fc96b400640 1 -- 172.21.15.102:0/1523718156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc96c15c820 msgr2=0x7fc96c0b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.059+0000 7fc96b400640 1 --2- 172.21.15.102:0/1523718156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc96c15c820 0x7fc96c0b0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.059+0000 7fc96b400640 1 -- 172.21.15.102:0/1523718156 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc954009a70 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.059+0000 7fc96aa00640 1 --2- 172.21.15.102:0/1523718156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc96c15c820 0x7fc96c0b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:02.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.059+0000 7fc96b400640 1 --2- 172.21.15.102:0/1523718156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc96c0cf7c0 0x7fc96c1f2b70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc95c00c500 tx=0x7fc95c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:02.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.060+0000 7fc963400640 1 -- 172.21.15.102:0/1523718156 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c005ce0 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.060+0000 7fc963400640 1 -- 172.21.15.102:0/1523718156 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc95c005e40 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.060+0000 7fc963400640 1 -- 172.21.15.102:0/1523718156 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95c007330 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.060+0000 7fc970c00640 1 -- 172.21.15.102:0/1523718156 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc96c0b32f0 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.060+0000 7fc970c00640 1 -- 172.21.15.102:0/1523718156 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc96c0b38b0 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.061+0000 7fc970c00640 1 -- 172.21.15.102:0/1523718156 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc96c0d2480 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.062+0000 7fc963400640 1 -- 172.21.15.102:0/1523718156 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc95c007550 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.063+0000 7fc963400640 1 --2- 172.21.15.102:0/1523718156 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc94806ac00 0x7fc94806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:02.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.064+0000 7fc963400640 1 -- 172.21.15.102:0/1523718156 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc95c088d80 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.064+0000 7fc96aa00640 1 --2- 172.21.15.102:0/1523718156 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc94806ac00 0x7fc94806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:02.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.064+0000 7fc96aa00640 1 --2- 172.21.15.102:0/1523718156 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc94806ac00 0x7fc94806d0c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc954036000 tx=0x7fc954030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:02.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.068+0000 7fc963400640 1 -- 172.21.15.102:0/1523718156 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc95c08c030 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.276+0000 7fc970c00640 1 -- 172.21.15.102:0/1523718156 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc96c04e030 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.276+0000 7fc963400640 1 -- 172.21.15.102:0/1523718156 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc95c017080 con 0x7fc96c0cf7c0 2024-03-23T03:48:02.277 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:48:02.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.279+0000 7fc970c00640 1 -- 172.21.15.102:0/1523718156 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc94806ac00 msgr2=0x7fc94806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:02.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.279+0000 7fc970c00640 1 --2- 172.21.15.102:0/1523718156 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc94806ac00 0x7fc94806d0c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc954036000 tx=0x7fc954030000 comp rx=0 tx=0).stop 2024-03-23T03:48:02.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.279+0000 7fc970c00640 1 -- 172.21.15.102:0/1523718156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc96c0cf7c0 msgr2=0x7fc96c1f2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:02.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.279+0000 7fc970c00640 1 --2- 172.21.15.102:0/1523718156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc96c0cf7c0 0x7fc96c1f2b70 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fc95c00c500 tx=0x7fc95c00c9c0 comp rx=0 tx=0).stop 2024-03-23T03:48:02.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.279+0000 7fc970c00640 1 -- 172.21.15.102:0/1523718156 shutdown_connections 2024-03-23T03:48:02.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.279+0000 7fc970c00640 1 --2- 172.21.15.102:0/1523718156 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc94806ac00 0x7fc94806d0c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:02.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.279+0000 7fc970c00640 1 --2- 172.21.15.102:0/1523718156 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc96c160e90 0x7fc96c15cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:02.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.279+0000 7fc970c00640 1 --2- 172.21.15.102:0/1523718156 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc96c15c820 0x7fc96c0b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:02.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.279+0000 7fc970c00640 1 --2- 172.21.15.102:0/1523718156 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc96c0cf7c0 0x7fc96c1f2b70 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:02.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.279+0000 7fc970c00640 1 -- 172.21.15.102:0/1523718156 >> 172.21.15.102:0/1523718156 conn(0x7fc96c06b070 msgr2=0x7fc96c0cbe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:02.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.279+0000 7fc970c00640 1 -- 172.21.15.102:0/1523718156 shutdown_connections 2024-03-23T03:48:02.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.279+0000 7fc970c00640 1 -- 172.21.15.102:0/1523718156 wait complete. 2024-03-23T03:48:02.300 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2024-03-23T03:48:02.300 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:02.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2024-03-23T03:48:02.391 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:48:02.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:02.391 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-23T03:48:02.439 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:48:02.440 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:48:02.496 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:48:02.496 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:48:02.552 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:48:02.552 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:48:02.602 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:48:02.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.821+0000 7f8dc1600640 1 Processor -- start 2024-03-23T03:48:02.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.821+0000 7f8dc1600640 1 -- start start 2024-03-23T03:48:02.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.822+0000 7f8dc1600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8dbc1571d0 0x7f8dbc0b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:02.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.822+0000 7f8dc1600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8dbc0cdc00 0x7f8dbc15aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:02.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.822+0000 7f8dc1600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dbc1575b0 con 0x7f8dbc15afe0 2024-03-23T03:48:02.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.822+0000 7f8dc1600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dbc159f20 con 0x7f8dbc0cdc00 2024-03-23T03:48:02.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.822+0000 7f8dc1600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc15a200 con 0x7f8dbc1571d0 2024-03-23T03:48:02.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.823+0000 7f8dbb400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8dbc0cdc00 0x7f8dbc15aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:02.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.823+0000 7f8dbb400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8dbc0cdc00 0x7f8dbc15aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54350/0 (socket says 172.21.15.102:54350) 2024-03-23T03:48:02.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.823+0000 7f8dbb400640 1 -- 172.21.15.102:0/1542243134 learned_addr learned my addr 172.21.15.102:0/1542243134 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:02.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.823+0000 7f8dbb400640 1 -- 172.21.15.102:0/1542243134 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8dbc1571d0 msgr2=0x7f8dbc0b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:02.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.823+0000 7f8dbb400640 1 --2- 172.21.15.102:0/1542243134 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8dbc1571d0 0x7f8dbc0b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:02.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.823+0000 7f8dbb400640 1 -- 172.21.15.102:0/1542243134 >> v1:172.21.15.173:6789/0 conn(0x7f8dbc15afe0 legacy=0x7f8dbc0cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:02.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.823+0000 7f8dbb400640 1 -- 172.21.15.102:0/1542243134 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dbc15a4e0 con 0x7f8dbc0cdc00 2024-03-23T03:48:02.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.824+0000 7f8dbb400640 1 --2- 172.21.15.102:0/1542243134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dbc0cdc00 0x7f8dbc15aaa0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8da8009870 tx=0x7f8da802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56287b731ee0e737 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:02.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.825+0000 7f8dbaa00640 1 -- 172.21.15.102:0/1542243134 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da8031640 con 0x7f8dbc0cdc00 2024-03-23T03:48:02.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.825+0000 7f8dbaa00640 1 -- 172.21.15.102:0/1542243134 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8da80317a0 con 0x7f8dbc0cdc00 2024-03-23T03:48:02.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.825+0000 7f8dbaa00640 1 -- 172.21.15.102:0/1542243134 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8da8031b30 con 0x7f8dbc0cdc00 2024-03-23T03:48:02.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.825+0000 7f8dc1600640 1 -- 172.21.15.102:0/1542243134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dbc0cdc00 msgr2=0x7f8dbc15aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:02.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.825+0000 7f8dc1600640 1 --2- 172.21.15.102:0/1542243134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dbc0cdc00 0x7f8dbc15aaa0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8da8009870 tx=0x7f8da802b8f0 comp rx=0 tx=0).stop 2024-03-23T03:48:02.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.825+0000 7f8dc1600640 1 -- 172.21.15.102:0/1542243134 shutdown_connections 2024-03-23T03:48:02.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.825+0000 7f8dc1600640 1 --2- 172.21.15.102:0/1542243134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dbc0cdc00 0x7f8dbc15aaa0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:02.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.825+0000 7f8dc1600640 1 --2- 172.21.15.102:0/1542243134 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8dbc1571d0 0x7f8dbc0b35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:02.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.825+0000 7f8dc1600640 1 -- 172.21.15.102:0/1542243134 >> 172.21.15.102:0/1542243134 conn(0x7f8dbc065df0 msgr2=0x7f8dbc069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:02.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.825+0000 7f8dc1600640 1 -- 172.21.15.102:0/1542243134 shutdown_connections 2024-03-23T03:48:02.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.825+0000 7f8dc1600640 1 -- 172.21.15.102:0/1542243134 wait complete. 2024-03-23T03:48:02.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.827+0000 7f8dc1600640 1 Processor -- start 2024-03-23T03:48:02.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.827+0000 7f8dc1600640 1 -- start start 2024-03-23T03:48:02.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.828+0000 7f8dc1600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8dbc0cdc00 0x7f8dbc06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:02.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.828+0000 7f8dc1600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8dbc1571d0 0x7f8dbc1f1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.828+0000 7f8dc1600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dbc15afe0 0x7f8dbc0cc2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.828+0000 7f8dc1600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc0c3710 con 0x7f8dbc15afe0 2024-03-23T03:48:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.828+0000 7f8dc1600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc06e0b0 con 0x7f8dbc0cdc00 2024-03-23T03:48:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.828+0000 7f8dc1600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dbc06cf50 con 0x7f8dbc1571d0 2024-03-23T03:48:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.828+0000 7f8dc0c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dbc15afe0 0x7f8dbc0cc2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.828+0000 7f8dc0c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dbc15afe0 0x7f8dbc0cc2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54356/0 (socket says 172.21.15.102:54356) 2024-03-23T03:48:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.828+0000 7f8dc0c00640 1 -- 172.21.15.102:0/3560392948 learned_addr learned my addr 172.21.15.102:0/3560392948 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.829+0000 7f8dbbe00640 1 --2- 172.21.15.102:0/3560392948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8dbc0cdc00 0x7f8dbc06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.829+0000 7f8dbb400640 1 --2- 172.21.15.102:0/3560392948 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8dbc1571d0 0x7f8dbc1f1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.829+0000 7f8dc0c00640 1 -- 172.21.15.102:0/3560392948 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8dbc1571d0 msgr2=0x7f8dbc1f1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.829+0000 7f8dc0c00640 1 --2- 172.21.15.102:0/3560392948 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8dbc1571d0 0x7f8dbc1f1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.829+0000 7f8dc0c00640 1 -- 172.21.15.102:0/3560392948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8dbc0cdc00 msgr2=0x7f8dbc06db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.829+0000 7f8dc0c00640 1 --2- 172.21.15.102:0/3560392948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8dbc0cdc00 0x7f8dbc06db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:02.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.829+0000 7f8dc0c00640 1 -- 172.21.15.102:0/3560392948 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da80094d0 con 0x7f8dbc15afe0 2024-03-23T03:48:02.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.829+0000 7f8dbb400640 1 --2- 172.21.15.102:0/3560392948 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8dbc1571d0 0x7f8dbc1f1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:02.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.829+0000 7f8dc0c00640 1 --2- 172.21.15.102:0/3560392948 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dbc15afe0 0x7f8dbc0cc2c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8db400c780 tx=0x7f8db400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:02.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.829+0000 7f8db3e00640 1 -- 172.21.15.102:0/3560392948 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db4003d80 con 0x7f8dbc15afe0 2024-03-23T03:48:02.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.829+0000 7f8db3e00640 1 -- 172.21.15.102:0/3560392948 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db400ce90 con 0x7f8dbc15afe0 2024-03-23T03:48:02.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.829+0000 7f8dc1600640 1 -- 172.21.15.102:0/3560392948 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dbc06d230 con 0x7f8dbc15afe0 2024-03-23T03:48:02.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.830+0000 7f8db3e00640 1 -- 172.21.15.102:0/3560392948 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db40052b0 con 0x7f8dbc15afe0 2024-03-23T03:48:02.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.830+0000 7f8dc1600640 1 -- 172.21.15.102:0/3560392948 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dbc0abc30 con 0x7f8dbc15afe0 2024-03-23T03:48:02.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.831+0000 7f8dc1600640 1 -- 172.21.15.102:0/3560392948 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dbc04e030 con 0x7f8dbc15afe0 2024-03-23T03:48:02.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.832+0000 7f8db3e00640 1 -- 172.21.15.102:0/3560392948 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8db4005410 con 0x7f8dbc15afe0 2024-03-23T03:48:02.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.832+0000 7f8db3e00640 1 --2- 172.21.15.102:0/3560392948 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d9c06ac00 0x7f8d9c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:02.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.833+0000 7f8db3e00640 1 -- 172.21.15.102:0/3560392948 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8db4088ff0 con 0x7f8dbc15afe0 2024-03-23T03:48:02.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.833+0000 7f8dbbe00640 1 --2- 172.21.15.102:0/3560392948 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d9c06ac00 0x7f8d9c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:02.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.833+0000 7f8dbbe00640 1 --2- 172.21.15.102:0/3560392948 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d9c06ac00 0x7f8d9c06d0c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc1f5400 tx=0x7f8dac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:02.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:02.837+0000 7f8db3e00640 1 -- 172.21.15.102:0/3560392948 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8db4053d70 con 0x7f8dbc15afe0 2024-03-23T03:48:03.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.030+0000 7f8dc1600640 1 -- 172.21.15.102:0/3560392948 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8dbc070d60 con 0x7f8dbc15afe0 2024-03-23T03:48:03.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.032+0000 7f8db3e00640 1 -- 172.21.15.102:0/3560392948 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18260 (secure 0 0 0) 0x7f8db4017070 con 0x7f8dbc15afe0 2024-03-23T03:48:03.032 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:03.033 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":28,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:47:41.647786+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:48:03.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.034+0000 7f8dc1600640 1 -- 172.21.15.102:0/3560392948 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d9c06ac00 msgr2=0x7f8d9c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:03.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.034+0000 7f8dc1600640 1 --2- 172.21.15.102:0/3560392948 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d9c06ac00 0x7f8d9c06d0c0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8dbc1f5400 tx=0x7f8dac008040 comp rx=0 tx=0).stop 2024-03-23T03:48:03.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.034+0000 7f8dc1600640 1 -- 172.21.15.102:0/3560392948 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dbc15afe0 msgr2=0x7f8dbc0cc2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:03.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.034+0000 7f8dc1600640 1 --2- 172.21.15.102:0/3560392948 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dbc15afe0 0x7f8dbc0cc2c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f8db400c780 tx=0x7f8db400cc40 comp rx=0 tx=0).stop 2024-03-23T03:48:03.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.035+0000 7f8dc1600640 1 -- 172.21.15.102:0/3560392948 shutdown_connections 2024-03-23T03:48:03.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.035+0000 7f8dc1600640 1 --2- 172.21.15.102:0/3560392948 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d9c06ac00 0x7f8d9c06d0c0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.035+0000 7f8dc1600640 1 --2- 172.21.15.102:0/3560392948 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8dbc15afe0 0x7f8dbc0cc2c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.035+0000 7f8dc1600640 1 --2- 172.21.15.102:0/3560392948 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8dbc1571d0 0x7f8dbc1f1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.035+0000 7f8dc1600640 1 --2- 172.21.15.102:0/3560392948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8dbc0cdc00 0x7f8dbc06db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.035+0000 7f8dc1600640 1 -- 172.21.15.102:0/3560392948 >> 172.21.15.102:0/3560392948 conn(0x7f8dbc065df0 msgr2=0x7f8dbc0cae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:03.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.035+0000 7f8dc1600640 1 -- 172.21.15.102:0/3560392948 shutdown_connections 2024-03-23T03:48:03.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.035+0000 7f8dc1600640 1 -- 172.21.15.102:0/3560392948 wait complete. 2024-03-23T03:48:03.054 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:46:52.158650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:46:52.914275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-23T03:46:55.983754+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-23T03:48:03.055 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:48:03.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.258+0000 7f6f49800640 1 Processor -- start 2024-03-23T03:48:03.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.258+0000 7f6f49800640 1 -- start start 2024-03-23T03:48:03.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.260+0000 7f6f49800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f6f44159830 0x7f6f440cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:03.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.260+0000 7f6f49800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6f440ce930 0x7f6f440cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:03.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.260+0000 7f6f49800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f44159c10 con 0x7f6f440ca7b0 2024-03-23T03:48:03.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.260+0000 7f6f49800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4406de50 con 0x7f6f44159830 2024-03-23T03:48:03.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.260+0000 7f6f49800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f440cf300 con 0x7f6f440ce930 2024-03-23T03:48:03.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.260+0000 7f6f3a000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6f440ca7b0 0x7f6f440cdcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45092/0 (socket says 172.21.15.102:45092) 2024-03-23T03:48:03.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.260+0000 7f6f3a000640 1 -- 172.21.15.102:0/526645098 learned_addr learned my addr 172.21.15.102:0/526645098 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:03.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.260+0000 7f6f43400640 1 --2- 172.21.15.102:0/526645098 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6f440ce930 0x7f6f440cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:03.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f42a00640 1 --2- 172.21.15.102:0/526645098 >> v2:172.21.15.173:3300/0 conn(0x7f6f44159830 0x7f6f440cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:03.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f42000640 1 -- 172.21.15.102:0/526645098 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2587166267 0 0) 0x7f6f44159c10 con 0x7f6f440ca7b0 2024-03-23T03:48:03.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f42000640 1 -- 172.21.15.102:0/526645098 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f28005730 con 0x7f6f440ca7b0 2024-03-23T03:48:03.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f42000640 1 -- 172.21.15.102:0/526645098 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 368845267 0 0) 0x7f6f28005730 con 0x7f6f440ca7b0 2024-03-23T03:48:03.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f42000640 1 -- 172.21.15.102:0/526645098 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6f440ce930 msgr2=0x7f6f440cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:03.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f42000640 1 --2- 172.21.15.102:0/526645098 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6f440ce930 0x7f6f440cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f42000640 1 -- 172.21.15.102:0/526645098 >> v2:172.21.15.173:3300/0 conn(0x7f6f44159830 msgr2=0x7f6f440cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:03.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f42000640 1 --2- 172.21.15.102:0/526645098 >> v2:172.21.15.173:3300/0 conn(0x7f6f44159830 0x7f6f440cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f42000640 1 -- 172.21.15.102:0/526645098 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f440cd840 con 0x7f6f440ca7b0 2024-03-23T03:48:03.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f42000640 1 -- 172.21.15.102:0/526645098 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f6f2c001670 con 0x7f6f440ca7b0 2024-03-23T03:48:03.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f42000640 1 -- 172.21.15.102:0/526645098 >> v1:172.21.15.102:6789/0 conn(0x7f6f440ca7b0 legacy=0x7f6f440cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:03.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f42000640 1 --2- 172.21.15.102:0/526645098 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6f440ce930 0x7f6f28005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:03.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f42000640 1 --2- 172.21.15.102:0/526645098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f28006100 0x7f6f280069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:03.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f42000640 1 --2- 172.21.15.102:0/526645098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f28006f10 0x7f6f28007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:03.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f42000640 1 -- 172.21.15.102:0/526645098 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f440cf300 con 0x7f6f28006100 2024-03-23T03:48:03.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f42000640 1 -- 172.21.15.102:0/526645098 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2800db90 con 0x7f6f28006f10 2024-03-23T03:48:03.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f42000640 1 -- 172.21.15.102:0/526645098 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2800dd00 con 0x7f6f440ce930 2024-03-23T03:48:03.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.261+0000 7f6f3a000640 1 --2- 172.21.15.102:0/526645098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f28006100 0x7f6f280069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:03.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.262+0000 7f6f43400640 1 --2- 172.21.15.102:0/526645098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f28006f10 0x7f6f28007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:03.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.262+0000 7f6f42a00640 1 --2- 172.21.15.102:0/526645098 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6f440ce930 0x7f6f28005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:03.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.262+0000 7f6f3a000640 1 -- 172.21.15.102:0/526645098 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6f440ce930 msgr2=0x7f6f28005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:03.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.262+0000 7f6f3a000640 1 --2- 172.21.15.102:0/526645098 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6f440ce930 0x7f6f28005b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.262+0000 7f6f3a000640 1 -- 172.21.15.102:0/526645098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f28006f10 msgr2=0x7f6f28007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:03.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.262+0000 7f6f3a000640 1 --2- 172.21.15.102:0/526645098 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f28006f10 0x7f6f28007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.262+0000 7f6f3a000640 1 -- 172.21.15.102:0/526645098 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f28005730 con 0x7f6f28006100 2024-03-23T03:48:03.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.263+0000 7f6f3a000640 1 --2- 172.21.15.102:0/526645098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f28006100 0x7f6f280069d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c00c240 tx=0x7f6f28007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e8d6f0834b3d2f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:03.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.263+0000 7f6f42000640 1 -- 172.21.15.102:0/526645098 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c02cb00 con 0x7f6f28006100 2024-03-23T03:48:03.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.263+0000 7f6f42000640 1 -- 172.21.15.102:0/526645098 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2c02cc60 con 0x7f6f28006100 2024-03-23T03:48:03.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.263+0000 7f6f42000640 1 -- 172.21.15.102:0/526645098 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c005af0 con 0x7f6f28006100 2024-03-23T03:48:03.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.264+0000 7f6f49800640 1 -- 172.21.15.102:0/526645098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f28006100 msgr2=0x7f6f280069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:03.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.264+0000 7f6f49800640 1 --2- 172.21.15.102:0/526645098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f28006100 0x7f6f280069d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c00c240 tx=0x7f6f28007a00 comp rx=0 tx=0).stop 2024-03-23T03:48:03.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.264+0000 7f6f43400640 1 -- 172.21.15.102:0/526645098 reap_dead start 2024-03-23T03:48:03.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.264+0000 7f6f49800640 1 -- 172.21.15.102:0/526645098 shutdown_connections 2024-03-23T03:48:03.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.264+0000 7f6f49800640 1 -- 172.21.15.102:0/526645098 >> 172.21.15.102:0/526645098 conn(0x7f6f4406a0b0 msgr2=0x7f6f4406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:03.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.264+0000 7f6f49800640 1 -- 172.21.15.102:0/526645098 shutdown_connections 2024-03-23T03:48:03.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.264+0000 7f6f49800640 1 -- 172.21.15.102:0/526645098 wait complete. 2024-03-23T03:48:03.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.266+0000 7f6f49800640 1 Processor -- start 2024-03-23T03:48:03.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.266+0000 7f6f49800640 1 -- start start 2024-03-23T03:48:03.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.267+0000 7f6f49800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6f28006100 0x7f6f4414c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:03.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.267+0000 7f6f49800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f440ae310 0x7f6f440b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:03.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.267+0000 7f6f49800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f4414a7a0 0x7f6f4414ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:03.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.267+0000 7f6f49800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4414c9e0 con 0x7f6f440ae310 2024-03-23T03:48:03.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.267+0000 7f6f49800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f441f5820 con 0x7f6f4414a7a0 2024-03-23T03:48:03.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.267+0000 7f6f49800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4414b670 con 0x7f6f28006100 2024-03-23T03:48:03.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.267+0000 7f6f3a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f440ae310 0x7f6f440b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:03.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.267+0000 7f6f3a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f440ae310 0x7f6f440b3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41306/0 (socket says 172.21.15.102:41306) 2024-03-23T03:48:03.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.267+0000 7f6f3a000640 1 -- 172.21.15.102:0/860435773 learned_addr learned my addr 172.21.15.102:0/860435773 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:03.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.267+0000 7f6f42a00640 1 --2- 172.21.15.102:0/860435773 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6f28006100 0x7f6f4414c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:03.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.268+0000 7f6f43400640 1 --2- 172.21.15.102:0/860435773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f4414a7a0 0x7f6f4414ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:03.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.268+0000 7f6f3a000640 1 -- 172.21.15.102:0/860435773 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6f28006100 msgr2=0x7f6f4414c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.268+0000 7f6f3a000640 1 --2- 172.21.15.102:0/860435773 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6f28006100 0x7f6f4414c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.268+0000 7f6f3a000640 1 -- 172.21.15.102:0/860435773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f4414a7a0 msgr2=0x7f6f4414ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.268+0000 7f6f3a000640 1 --2- 172.21.15.102:0/860435773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f4414a7a0 0x7f6f4414ac20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.268+0000 7f6f3a000640 1 -- 172.21.15.102:0/860435773 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f441f5a30 con 0x7f6f440ae310 2024-03-23T03:48:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.268+0000 7f6f42a00640 1 --2- 172.21.15.102:0/860435773 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6f28006100 0x7f6f4414c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.268+0000 7f6f3a000640 1 --2- 172.21.15.102:0/860435773 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f440ae310 0x7f6f440b3a20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c00c240 tx=0x7f6f2c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.268+0000 7f6f3b400640 1 -- 172.21.15.102:0/860435773 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c039070 con 0x7f6f440ae310 2024-03-23T03:48:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.268+0000 7f6f3b400640 1 -- 172.21.15.102:0/860435773 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2c005c80 con 0x7f6f440ae310 2024-03-23T03:48:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.268+0000 7f6f3b400640 1 -- 172.21.15.102:0/860435773 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c03c040 con 0x7f6f440ae310 2024-03-23T03:48:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.268+0000 7f6f49800640 1 -- 172.21.15.102:0/860435773 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f441f5d10 con 0x7f6f440ae310 2024-03-23T03:48:03.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.268+0000 7f6f49800640 1 -- 172.21.15.102:0/860435773 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f441f62d0 con 0x7f6f440ae310 2024-03-23T03:48:03.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.270+0000 7f6f49800640 1 -- 172.21.15.102:0/860435773 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f441f6490 con 0x7f6f440ae310 2024-03-23T03:48:03.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.270+0000 7f6f3b400640 1 -- 172.21.15.102:0/860435773 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6f2c032070 con 0x7f6f440ae310 2024-03-23T03:48:03.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.271+0000 7f6f3b400640 1 --2- 172.21.15.102:0/860435773 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6f2006ac30 0x7f6f2006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:03.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.271+0000 7f6f3b400640 1 -- 172.21.15.102:0/860435773 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f6f2c0aad50 con 0x7f6f440ae310 2024-03-23T03:48:03.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.271+0000 7f6f42a00640 1 --2- 172.21.15.102:0/860435773 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6f2006ac30 0x7f6f2006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:03.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.272+0000 7f6f42a00640 1 --2- 172.21.15.102:0/860435773 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6f2006ac30 0x7f6f2006d0f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6f34007460 tx=0x7f6f3400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:03.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.276+0000 7f6f3b400640 1 -- 172.21.15.102:0/860435773 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6f2c075ca0 con 0x7f6f440ae310 2024-03-23T03:48:03.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.468+0000 7f6f49800640 1 -- 172.21.15.102:0/860435773 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6f44073390 con 0x7f6f440ae310 2024-03-23T03:48:03.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.469+0000 7f6f3b400640 1 -- 172.21.15.102:0/860435773 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6f2c046020 con 0x7f6f440ae310 2024-03-23T03:48:03.470 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:48:03.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.471+0000 7f6f49800640 1 -- 172.21.15.102:0/860435773 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6f2006ac30 msgr2=0x7f6f2006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:03.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.471+0000 7f6f49800640 1 --2- 172.21.15.102:0/860435773 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6f2006ac30 0x7f6f2006d0f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6f34007460 tx=0x7f6f3400b040 comp rx=0 tx=0).stop 2024-03-23T03:48:03.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.471+0000 7f6f49800640 1 -- 172.21.15.102:0/860435773 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f440ae310 msgr2=0x7f6f440b3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:03.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.471+0000 7f6f49800640 1 --2- 172.21.15.102:0/860435773 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f440ae310 0x7f6f440b3a20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c00c240 tx=0x7f6f2c001a00 comp rx=0 tx=0).stop 2024-03-23T03:48:03.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.471+0000 7f6f49800640 1 -- 172.21.15.102:0/860435773 shutdown_connections 2024-03-23T03:48:03.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.471+0000 7f6f49800640 1 --2- 172.21.15.102:0/860435773 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6f2006ac30 0x7f6f2006d0f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.471+0000 7f6f49800640 1 --2- 172.21.15.102:0/860435773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6f4414a7a0 0x7f6f4414ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.471+0000 7f6f49800640 1 --2- 172.21.15.102:0/860435773 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f440ae310 0x7f6f440b3a20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.471+0000 7f6f49800640 1 --2- 172.21.15.102:0/860435773 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6f28006100 0x7f6f4414c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.471+0000 7f6f49800640 1 -- 172.21.15.102:0/860435773 >> 172.21.15.102:0/860435773 conn(0x7f6f4406a0b0 msgr2=0x7f6f440b0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:03.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.471+0000 7f6f49800640 1 -- 172.21.15.102:0/860435773 shutdown_connections 2024-03-23T03:48:03.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.472+0000 7f6f49800640 1 -- 172.21.15.102:0/860435773 wait complete. 2024-03-23T03:48:03.491 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:48:03.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.703+0000 7f7da6400640 1 Processor -- start 2024-03-23T03:48:03.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.703+0000 7f7da6400640 1 -- start start 2024-03-23T03:48:03.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.704+0000 7f7da6400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f7da01456a0 0x7f7da00ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:03.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.705+0000 7f7da6400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7da00ac6d0 con 0x7f7da0070cd0 2024-03-23T03:48:03.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.705+0000 7f7da6400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7da006d390 con 0x7f7da0145c20 2024-03-23T03:48:03.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.705+0000 7f7da6400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7da00ad890 con 0x7f7da01456a0 2024-03-23T03:48:03.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.705+0000 7f7d9ea00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7da0070cd0 0x7f7da00ab8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45100/0 (socket says 172.21.15.102:45100) 2024-03-23T03:48:03.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.705+0000 7f7d9ea00640 1 -- 172.21.15.102:0/312315052 learned_addr learned my addr 172.21.15.102:0/312315052 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:03.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.705+0000 7f7d9f400640 1 --2- 172.21.15.102:0/312315052 >> v2:172.21.15.173:3300/0 conn(0x7f7da01456a0 0x7f7da00ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:03.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.706+0000 7f7d9e000640 1 -- 172.21.15.102:0/312315052 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591561701 0 0) 0x7f7da00ac6d0 con 0x7f7da0070cd0 2024-03-23T03:48:03.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.706+0000 7f7d9e000640 1 -- 172.21.15.102:0/312315052 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d84003520 con 0x7f7da0070cd0 2024-03-23T03:48:03.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.706+0000 7f7d9e000640 1 -- 172.21.15.102:0/312315052 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2577479888 0 0) 0x7f7d84003520 con 0x7f7da0070cd0 2024-03-23T03:48:03.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.706+0000 7f7d9f400640 1 -- 172.21.15.102:0/312315052 >> v1:172.21.15.173:6789/0 conn(0x7f7da0145c20 legacy=0x7f7da00abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:03.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.706+0000 7f7d9f400640 1 -- 172.21.15.102:0/312315052 >> v1:172.21.15.102:6789/0 conn(0x7f7da0070cd0 legacy=0x7f7da00ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:03.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.706+0000 7f7d9f400640 1 -- 172.21.15.102:0/312315052 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da006d670 con 0x7f7da01456a0 2024-03-23T03:48:03.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.707+0000 7f7d9f400640 1 --2- 172.21.15.102:0/312315052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da01456a0 0x7f7da00ce930 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7d900097a0 tx=0x7f7d9002b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9385ecf02cf9ff2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:03.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.708+0000 7f7d9e000640 1 -- 172.21.15.102:0/312315052 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9002f050 con 0x7f7da01456a0 2024-03-23T03:48:03.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.708+0000 7f7d9e000640 1 -- 172.21.15.102:0/312315052 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d90031500 con 0x7f7da01456a0 2024-03-23T03:48:03.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.708+0000 7f7d9e000640 1 -- 172.21.15.102:0/312315052 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c001670 con 0x7f7da01456a0 2024-03-23T03:48:03.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.708+0000 7f7da6400640 1 -- 172.21.15.102:0/312315052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da01456a0 msgr2=0x7f7da00ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:03.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.708+0000 7f7da6400640 1 --2- 172.21.15.102:0/312315052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da01456a0 0x7f7da00ce930 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7d900097a0 tx=0x7f7d9002b820 comp rx=0 tx=0).stop 2024-03-23T03:48:03.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.709+0000 7f7da6400640 1 -- 172.21.15.102:0/312315052 shutdown_connections 2024-03-23T03:48:03.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.709+0000 7f7da6400640 1 --2- 172.21.15.102:0/312315052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da01456a0 0x7f7da00ce930 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.709+0000 7f7da6400640 1 -- 172.21.15.102:0/312315052 >> 172.21.15.102:0/312315052 conn(0x7f7da006a0b0 msgr2=0x7f7da006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:03.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.709+0000 7f7da6400640 1 -- 172.21.15.102:0/312315052 shutdown_connections 2024-03-23T03:48:03.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.709+0000 7f7da6400640 1 -- 172.21.15.102:0/312315052 wait complete. 2024-03-23T03:48:03.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.710+0000 7f7da6400640 1 Processor -- start 2024-03-23T03:48:03.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.710+0000 7f7da6400640 1 -- start start 2024-03-23T03:48:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.711+0000 7f7da6400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da0070cd0 0x7f7da01419d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.711+0000 7f7da6400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7da01456a0 0x7f7da013cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.711+0000 7f7da6400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7da0145c20 0x7f7da0143f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.711+0000 7f7da6400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da014c7a0 con 0x7f7da01456a0 2024-03-23T03:48:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.711+0000 7f7da6400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da014ca80 con 0x7f7da0070cd0 2024-03-23T03:48:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.711+0000 7f7da6400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da014cd60 con 0x7f7da0145c20 2024-03-23T03:48:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.712+0000 7f7d9ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7da01456a0 0x7f7da013cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.712+0000 7f7d9ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7da01456a0 0x7f7da013cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41310/0 (socket says 172.21.15.102:41310) 2024-03-23T03:48:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.712+0000 7f7d9ea00640 1 -- 172.21.15.102:0/2423662083 learned_addr learned my addr 172.21.15.102:0/2423662083 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.712+0000 7f7d9f400640 1 --2- 172.21.15.102:0/2423662083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da0070cd0 0x7f7da01419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.712+0000 7f7d9ea00640 1 -- 172.21.15.102:0/2423662083 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7da0145c20 msgr2=0x7f7da0143f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:03.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.712+0000 7f7d9fe00640 1 --2- 172.21.15.102:0/2423662083 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7da0145c20 0x7f7da0143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:03.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.712+0000 7f7d9ea00640 1 --2- 172.21.15.102:0/2423662083 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7da0145c20 0x7f7da0143f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.712+0000 7f7d9ea00640 1 -- 172.21.15.102:0/2423662083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da0070cd0 msgr2=0x7f7da01419d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:03.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.712+0000 7f7d9ea00640 1 --2- 172.21.15.102:0/2423662083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da0070cd0 0x7f7da01419d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.712+0000 7f7d9ea00640 1 -- 172.21.15.102:0/2423662083 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d90009400 con 0x7f7da01456a0 2024-03-23T03:48:03.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.712+0000 7f7d9f400640 1 --2- 172.21.15.102:0/2423662083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da0070cd0 0x7f7da01419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:03.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.712+0000 7f7d9ea00640 1 --2- 172.21.15.102:0/2423662083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7da01456a0 0x7f7da013cfd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c00c750 tx=0x7f7d8c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:03.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.713+0000 7f7d97400640 1 -- 172.21.15.102:0/2423662083 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c005ad0 con 0x7f7da01456a0 2024-03-23T03:48:03.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.713+0000 7f7d97400640 1 -- 172.21.15.102:0/2423662083 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8c005c30 con 0x7f7da01456a0 2024-03-23T03:48:03.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.713+0000 7f7da6400640 1 -- 172.21.15.102:0/2423662083 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7da014d040 con 0x7f7da01456a0 2024-03-23T03:48:03.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.713+0000 7f7d97400640 1 -- 172.21.15.102:0/2423662083 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8c007290 con 0x7f7da01456a0 2024-03-23T03:48:03.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.713+0000 7f7da6400640 1 -- 172.21.15.102:0/2423662083 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7da013d510 con 0x7f7da01456a0 2024-03-23T03:48:03.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.714+0000 7f7da6400640 1 -- 172.21.15.102:0/2423662083 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7da014d560 con 0x7f7da01456a0 2024-03-23T03:48:03.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.715+0000 7f7d97400640 1 -- 172.21.15.102:0/2423662083 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7d8c0073f0 con 0x7f7da01456a0 2024-03-23T03:48:03.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.716+0000 7f7d97400640 1 --2- 172.21.15.102:0/2423662083 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d8006ac30 0x7f7d8006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:03.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.716+0000 7f7d97400640 1 -- 172.21.15.102:0/2423662083 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7d8c088db0 con 0x7f7da01456a0 2024-03-23T03:48:03.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.716+0000 7f7d9f400640 1 --2- 172.21.15.102:0/2423662083 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d8006ac30 0x7f7d8006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:03.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.717+0000 7f7d9f400640 1 --2- 172.21.15.102:0/2423662083 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d8006ac30 0x7f7d8006d0f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7d90035000 tx=0x7f7d90008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:03.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.720+0000 7f7d97400640 1 -- 172.21.15.102:0/2423662083 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7da014d560 con 0x7f7da01456a0 2024-03-23T03:48:03.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.918+0000 7f7da6400640 1 -- 172.21.15.102:0/2423662083 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7da014d560 con 0x7f7da01456a0 2024-03-23T03:48:03.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.919+0000 7f7d97400640 1 -- 172.21.15.102:0/2423662083 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7da014d560 con 0x7f7da01456a0 2024-03-23T03:48:03.920 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:48:03.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.922+0000 7f7da6400640 1 -- 172.21.15.102:0/2423662083 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d8006ac30 msgr2=0x7f7d8006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:03.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.922+0000 7f7da6400640 1 --2- 172.21.15.102:0/2423662083 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d8006ac30 0x7f7d8006d0f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7d90035000 tx=0x7f7d90008030 comp rx=0 tx=0).stop 2024-03-23T03:48:03.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.922+0000 7f7da6400640 1 -- 172.21.15.102:0/2423662083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7da01456a0 msgr2=0x7f7da013cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:03.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.922+0000 7f7da6400640 1 --2- 172.21.15.102:0/2423662083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7da01456a0 0x7f7da013cfd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c00c750 tx=0x7f7d8c00cc10 comp rx=0 tx=0).stop 2024-03-23T03:48:03.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.922+0000 7f7da6400640 1 -- 172.21.15.102:0/2423662083 shutdown_connections 2024-03-23T03:48:03.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.922+0000 7f7da6400640 1 --2- 172.21.15.102:0/2423662083 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d8006ac30 0x7f7d8006d0f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.922+0000 7f7da6400640 1 --2- 172.21.15.102:0/2423662083 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7da0145c20 0x7f7da0143f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.922+0000 7f7da6400640 1 --2- 172.21.15.102:0/2423662083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7da01456a0 0x7f7da013cfd0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.922+0000 7f7da6400640 1 --2- 172.21.15.102:0/2423662083 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7da0070cd0 0x7f7da01419d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:03.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.922+0000 7f7da6400640 1 -- 172.21.15.102:0/2423662083 >> 172.21.15.102:0/2423662083 conn(0x7f7da006a0b0 msgr2=0x7f7da00ac8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:03.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.922+0000 7f7da6400640 1 -- 172.21.15.102:0/2423662083 shutdown_connections 2024-03-23T03:48:03.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:03.922+0000 7f7da6400640 1 -- 172.21.15.102:0/2423662083 wait complete. 2024-03-23T03:48:03.940 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:48:04.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.151+0000 7fc54a200640 1 Processor -- start 2024-03-23T03:48:04.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.151+0000 7fc54a200640 1 -- start start 2024-03-23T03:48:04.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.152+0000 7fc54a200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc5440add30 0x7fc5440ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:04.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.152+0000 7fc54a200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5440cf250 con 0x7fc5440ce930 2024-03-23T03:48:04.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.152+0000 7fc54a200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5440cdba0 con 0x7fc54406ce90 2024-03-23T03:48:04.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.152+0000 7fc54a200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5440ae1c0 con 0x7fc5440add30 2024-03-23T03:48:04.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.152+0000 7fc543400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc5440ce930 0x7fc5440cd3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45116/0 (socket says 172.21.15.102:45116) 2024-03-23T03:48:04.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.152+0000 7fc543400640 1 -- 172.21.15.102:0/3783414158 learned_addr learned my addr 172.21.15.102:0/3783414158 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:04.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.153+0000 7fc542a00640 1 --2- 172.21.15.102:0/3783414158 >> v2:172.21.15.173:3300/0 conn(0x7fc5440add30 0x7fc5440ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:04.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.153+0000 7fc542000640 1 -- 172.21.15.102:0/3783414158 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330019965 0 0) 0x7fc5440cf250 con 0x7fc5440ce930 2024-03-23T03:48:04.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.153+0000 7fc542000640 1 -- 172.21.15.102:0/3783414158 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc528003520 con 0x7fc5440ce930 2024-03-23T03:48:04.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.153+0000 7fc542000640 1 -- 172.21.15.102:0/3783414158 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3986894258 0 0) 0x7fc528003520 con 0x7fc5440ce930 2024-03-23T03:48:04.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.153+0000 7fc542000640 1 -- 172.21.15.102:0/3783414158 >> v2:172.21.15.173:3300/0 conn(0x7fc5440add30 msgr2=0x7fc5440ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:04.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.153+0000 7fc542000640 1 --2- 172.21.15.102:0/3783414158 >> v2:172.21.15.173:3300/0 conn(0x7fc5440add30 0x7fc5440ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.153+0000 7fc542000640 1 -- 172.21.15.102:0/3783414158 >> v1:172.21.15.173:6789/0 conn(0x7fc54406ce90 legacy=0x7fc54415e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:04.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.153+0000 7fc542000640 1 -- 172.21.15.102:0/3783414158 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5440cde80 con 0x7fc5440ce930 2024-03-23T03:48:04.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.153+0000 7fc542000640 1 -- 172.21.15.102:0/3783414158 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fc534001670 con 0x7fc5440ce930 2024-03-23T03:48:04.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.153+0000 7fc542000640 1 -- 172.21.15.102:0/3783414158 >> v1:172.21.15.102:6789/0 conn(0x7fc5440ce930 legacy=0x7fc5440cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:04.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.153+0000 7fc542000640 1 --2- 172.21.15.102:0/3783414158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc528003970 0x7fc528003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:04.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.153+0000 7fc542000640 1 --2- 172.21.15.102:0/3783414158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5280047c0 0x7fc528004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:04.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.153+0000 7fc542000640 1 --2- 172.21.15.102:0/3783414158 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc5280052a0 0x7fc52800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:04.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.153+0000 7fc542000640 1 -- 172.21.15.102:0/3783414158 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52800c120 con 0x7fc528003970 2024-03-23T03:48:04.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.153+0000 7fc542000640 1 -- 172.21.15.102:0/3783414158 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc52800c290 con 0x7fc5280047c0 2024-03-23T03:48:04.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.153+0000 7fc542000640 1 -- 172.21.15.102:0/3783414158 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc52800c450 con 0x7fc5280052a0 2024-03-23T03:48:04.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.154+0000 7fc543400640 1 --2- 172.21.15.102:0/3783414158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc528003970 0x7fc528003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:04.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.154+0000 7fc542a00640 1 --2- 172.21.15.102:0/3783414158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5280047c0 0x7fc528004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:04.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.154+0000 7fc543e00640 1 --2- 172.21.15.102:0/3783414158 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc5280052a0 0x7fc52800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:04.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.154+0000 7fc543400640 1 -- 172.21.15.102:0/3783414158 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc5280052a0 msgr2=0x7fc52800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:04.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.154+0000 7fc543400640 1 --2- 172.21.15.102:0/3783414158 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc5280052a0 0x7fc52800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.154+0000 7fc543400640 1 -- 172.21.15.102:0/3783414158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5280047c0 msgr2=0x7fc528004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:04.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.154+0000 7fc543400640 1 --2- 172.21.15.102:0/3783414158 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5280047c0 0x7fc528004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.154+0000 7fc543400640 1 -- 172.21.15.102:0/3783414158 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc528003520 con 0x7fc528003970 2024-03-23T03:48:04.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.154+0000 7fc543e00640 1 -- 172.21.15.102:0/3783414158 reap_dead start 2024-03-23T03:48:04.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.155+0000 7fc543400640 1 --2- 172.21.15.102:0/3783414158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc528003970 0x7fc528003d70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc534001eb0 tx=0x7fc5280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac2197d0f268932b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:04.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.155+0000 7fc542000640 1 -- 172.21.15.102:0/3783414158 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc534005ad0 con 0x7fc528003970 2024-03-23T03:48:04.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.156+0000 7fc542000640 1 -- 172.21.15.102:0/3783414158 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc534005c30 con 0x7fc528003970 2024-03-23T03:48:04.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.156+0000 7fc542000640 1 -- 172.21.15.102:0/3783414158 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53402c890 con 0x7fc528003970 2024-03-23T03:48:04.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.156+0000 7fc54a200640 1 -- 172.21.15.102:0/3783414158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc528003970 msgr2=0x7fc528003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:04.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.156+0000 7fc54a200640 1 --2- 172.21.15.102:0/3783414158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc528003970 0x7fc528003d70 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc534001eb0 tx=0x7fc5280057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.156+0000 7fc54a200640 1 -- 172.21.15.102:0/3783414158 shutdown_connections 2024-03-23T03:48:04.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.156+0000 7fc54a200640 1 --2- 172.21.15.102:0/3783414158 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc528003970 0x7fc528003d70 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.156+0000 7fc54a200640 1 -- 172.21.15.102:0/3783414158 >> 172.21.15.102:0/3783414158 conn(0x7fc54406a0b0 msgr2=0x7fc54406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:04.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.156+0000 7fc54a200640 1 -- 172.21.15.102:0/3783414158 shutdown_connections 2024-03-23T03:48:04.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.156+0000 7fc54a200640 1 -- 172.21.15.102:0/3783414158 wait complete. 2024-03-23T03:48:04.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.158+0000 7fc54a200640 1 Processor -- start 2024-03-23T03:48:04.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.158+0000 7fc54a200640 1 -- start start 2024-03-23T03:48:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.159+0000 7fc54a200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc528003970 0x7fc544150f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.159+0000 7fc54a200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5441ef5e0 0x7fc544151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.159+0000 7fc54a200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5441f55f0 0x7fc5441f5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.159+0000 7fc54a200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5441f6360 con 0x7fc5441f55f0 2024-03-23T03:48:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.159+0000 7fc54a200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5441f2da0 con 0x7fc5441ef5e0 2024-03-23T03:48:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.159+0000 7fc54a200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5441f1fe0 con 0x7fc528003970 2024-03-23T03:48:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.159+0000 7fc543e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5441f55f0 0x7fc5441f5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.159+0000 7fc543e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5441f55f0 0x7fc5441f5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41336/0 (socket says 172.21.15.102:41336) 2024-03-23T03:48:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.159+0000 7fc543e00640 1 -- 172.21.15.102:0/3405961429 learned_addr learned my addr 172.21.15.102:0/3405961429 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.160+0000 7fc543400640 1 --2- 172.21.15.102:0/3405961429 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc528003970 0x7fc544150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.160+0000 7fc542a00640 1 --2- 172.21.15.102:0/3405961429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5441ef5e0 0x7fc544151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.160+0000 7fc543e00640 1 -- 172.21.15.102:0/3405961429 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc528003970 msgr2=0x7fc544150f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:04.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.160+0000 7fc543e00640 1 --2- 172.21.15.102:0/3405961429 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc528003970 0x7fc544150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.160+0000 7fc543e00640 1 -- 172.21.15.102:0/3405961429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5441ef5e0 msgr2=0x7fc544151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:04.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.160+0000 7fc543e00640 1 --2- 172.21.15.102:0/3405961429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5441ef5e0 0x7fc544151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.160+0000 7fc543e00640 1 -- 172.21.15.102:0/3405961429 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5441f22c0 con 0x7fc5441f55f0 2024-03-23T03:48:04.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.160+0000 7fc543e00640 1 --2- 172.21.15.102:0/3405961429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5441f55f0 0x7fc5441f5a70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc5440ccb90 tx=0x7fc53c00d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:04.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.160+0000 7fc53b400640 1 -- 172.21.15.102:0/3405961429 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c012800 con 0x7fc5441f55f0 2024-03-23T03:48:04.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.160+0000 7fc53b400640 1 -- 172.21.15.102:0/3405961429 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc53c012980 con 0x7fc5441f55f0 2024-03-23T03:48:04.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.160+0000 7fc53b400640 1 -- 172.21.15.102:0/3405961429 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c012d30 con 0x7fc5441f55f0 2024-03-23T03:48:04.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.160+0000 7fc54a200640 1 -- 172.21.15.102:0/3405961429 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5441f25a0 con 0x7fc5441f55f0 2024-03-23T03:48:04.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.161+0000 7fc54a200640 1 -- 172.21.15.102:0/3405961429 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc544154b70 con 0x7fc5441f55f0 2024-03-23T03:48:04.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.162+0000 7fc54a200640 1 -- 172.21.15.102:0/3405961429 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc544073390 con 0x7fc5441f55f0 2024-03-23T03:48:04.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.163+0000 7fc53b400640 1 -- 172.21.15.102:0/3405961429 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc53c01e460 con 0x7fc5441f55f0 2024-03-23T03:48:04.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.164+0000 7fc53b400640 1 --2- 172.21.15.102:0/3405961429 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc52006ac30 0x7fc52006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.164+0000 7fc53b400640 1 -- 172.21.15.102:0/3405961429 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fc53c089400 con 0x7fc5441f55f0 2024-03-23T03:48:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.164+0000 7fc543400640 1 --2- 172.21.15.102:0/3405961429 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc52006ac30 0x7fc52006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.165+0000 7fc543400640 1 --2- 172.21.15.102:0/3405961429 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc52006ac30 0x7fc52006d0f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc534034000 tx=0x7fc534009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:04.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.169+0000 7fc53b400640 1 -- 172.21.15.102:0/3405961429 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc53c0542b0 con 0x7fc5441f55f0 2024-03-23T03:48:04.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.362+0000 7fc54a200640 1 -- 172.21.15.102:0/3405961429 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc54406c540 con 0x7fc5441f55f0 2024-03-23T03:48:04.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.363+0000 7fc53b400640 1 -- 172.21.15.102:0/3405961429 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc53c0578d0 con 0x7fc5441f55f0 2024-03-23T03:48:04.363 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:48:04.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.365+0000 7fc54a200640 1 -- 172.21.15.102:0/3405961429 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc52006ac30 msgr2=0x7fc52006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:04.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.365+0000 7fc54a200640 1 --2- 172.21.15.102:0/3405961429 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc52006ac30 0x7fc52006d0f0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc534034000 tx=0x7fc534009030 comp rx=0 tx=0).stop 2024-03-23T03:48:04.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.365+0000 7fc54a200640 1 -- 172.21.15.102:0/3405961429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5441f55f0 msgr2=0x7fc5441f5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:04.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.365+0000 7fc54a200640 1 --2- 172.21.15.102:0/3405961429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5441f55f0 0x7fc5441f5a70 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc5440ccb90 tx=0x7fc53c00d9b0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.365+0000 7fc54a200640 1 -- 172.21.15.102:0/3405961429 shutdown_connections 2024-03-23T03:48:04.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.365+0000 7fc54a200640 1 --2- 172.21.15.102:0/3405961429 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc52006ac30 0x7fc52006d0f0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.365+0000 7fc54a200640 1 --2- 172.21.15.102:0/3405961429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5441f55f0 0x7fc5441f5a70 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.365+0000 7fc54a200640 1 --2- 172.21.15.102:0/3405961429 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc5441ef5e0 0x7fc544151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.365+0000 7fc54a200640 1 --2- 172.21.15.102:0/3405961429 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc528003970 0x7fc544150f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.366+0000 7fc54a200640 1 -- 172.21.15.102:0/3405961429 >> 172.21.15.102:0/3405961429 conn(0x7fc54406a0b0 msgr2=0x7fc5440afa40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:04.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.366+0000 7fc54a200640 1 -- 172.21.15.102:0/3405961429 shutdown_connections 2024-03-23T03:48:04.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.366+0000 7fc54a200640 1 -- 172.21.15.102:0/3405961429 wait complete. 2024-03-23T03:48:04.385 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-23T03:48:04.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.597+0000 7f93cea00640 1 Processor -- start 2024-03-23T03:48:04.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.597+0000 7f93cea00640 1 -- start start 2024-03-23T03:48:04.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.598+0000 7f93cea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f93c80b2030 0x7f93c8069c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:04.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.598+0000 7f93cea00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c80b33b0 con 0x7f93c806a1d0 2024-03-23T03:48:04.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.598+0000 7f93cea00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c8159410 con 0x7f93c80b1c50 2024-03-23T03:48:04.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.598+0000 7f93cea00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c80b2f90 con 0x7f93c80b2030 2024-03-23T03:48:04.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.599+0000 7f93ccc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f93c80b2030 0x7f93c8069c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:04.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.599+0000 7f93ce000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f93c806a1d0 0x7f93c806ffe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45126/0 (socket says 172.21.15.102:45126) 2024-03-23T03:48:04.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.599+0000 7f93ce000640 1 -- 172.21.15.102:0/3426690461 learned_addr learned my addr 172.21.15.102:0/3426690461 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:04.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.599+0000 7f93ccc00640 1 -- 172.21.15.102:0/3426690461 >> v1:172.21.15.173:6789/0 conn(0x7f93c80b1c50 legacy=0x7f93c80ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:04.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.599+0000 7f93ccc00640 1 -- 172.21.15.102:0/3426690461 >> v1:172.21.15.102:6789/0 conn(0x7f93c806a1d0 legacy=0x7f93c806ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:04.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.599+0000 7f93ccc00640 1 -- 172.21.15.102:0/3426690461 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c80b2bd0 con 0x7f93c80b2030 2024-03-23T03:48:04.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.599+0000 7f93c7e00640 1 -- 172.21.15.102:0/3426690461 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692071775 0 0) 0x7f93c80b33b0 con 0x7f93c806a1d0 2024-03-23T03:48:04.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.600+0000 7f93ccc00640 1 --2- 172.21.15.102:0/3426690461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93c80b2030 0x7f93c8069c90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f93b40097a0 tx=0x7f93b402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f44f10ba946e1225 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:04.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.600+0000 7f93c7e00640 1 -- 172.21.15.102:0/3426690461 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b402e070 con 0x7f93c80b2030 2024-03-23T03:48:04.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.600+0000 7f93c7e00640 1 -- 172.21.15.102:0/3426690461 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b40315a0 con 0x7f93c80b2030 2024-03-23T03:48:04.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.600+0000 7f93c7e00640 1 -- 172.21.15.102:0/3426690461 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b4031930 con 0x7f93c80b2030 2024-03-23T03:48:04.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.601+0000 7f93cea00640 1 -- 172.21.15.102:0/3426690461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93c80b2030 msgr2=0x7f93c8069c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:04.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.601+0000 7f93cea00640 1 --2- 172.21.15.102:0/3426690461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93c80b2030 0x7f93c8069c90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f93b40097a0 tx=0x7f93b402b840 comp rx=0 tx=0).stop 2024-03-23T03:48:04.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.601+0000 7f93cea00640 1 -- 172.21.15.102:0/3426690461 shutdown_connections 2024-03-23T03:48:04.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.601+0000 7f93cea00640 1 --2- 172.21.15.102:0/3426690461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93c80b2030 0x7f93c8069c90 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.601+0000 7f93cea00640 1 -- 172.21.15.102:0/3426690461 >> 172.21.15.102:0/3426690461 conn(0x7f93c80b01b0 msgr2=0x7f93c80b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:04.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.601+0000 7f93cea00640 1 -- 172.21.15.102:0/3426690461 shutdown_connections 2024-03-23T03:48:04.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.601+0000 7f93cea00640 1 -- 172.21.15.102:0/3426690461 wait complete. 2024-03-23T03:48:04.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.603+0000 7f93cea00640 1 Processor -- start 2024-03-23T03:48:04.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.603+0000 7f93cea00640 1 -- start start 2024-03-23T03:48:04.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.604+0000 7f93cea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93c806a1d0 0x7f93c81f4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:04.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.604+0000 7f93cea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93c80b1c50 0x7f93c8150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:04.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.604+0000 7f93cea00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f93c80b2030 0x7f93c8142960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:04.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.604+0000 7f93cea00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c81512d0 con 0x7f93c80b1c50 2024-03-23T03:48:04.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.604+0000 7f93cea00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c81515b0 con 0x7f93c806a1d0 2024-03-23T03:48:04.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.604+0000 7f93cea00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c8151890 con 0x7f93c80b2030 2024-03-23T03:48:04.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.604+0000 7f93ccc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93c80b1c50 0x7f93c8150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:04.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.604+0000 7f93ccc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93c80b1c50 0x7f93c8150830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41350/0 (socket says 172.21.15.102:41350) 2024-03-23T03:48:04.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.604+0000 7f93ccc00640 1 -- 172.21.15.102:0/3296158484 learned_addr learned my addr 172.21.15.102:0/3296158484 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:04.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.604+0000 7f93ccc00640 1 -- 172.21.15.102:0/3296158484 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f93c80b2030 msgr2=0x7f93c8142960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:04.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.604+0000 7f93ccc00640 1 --2- 172.21.15.102:0/3296158484 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f93c80b2030 0x7f93c8142960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.605+0000 7f93ccc00640 1 -- 172.21.15.102:0/3296158484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93c806a1d0 msgr2=0x7f93c81f4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:04.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.605+0000 7f93cd600640 1 --2- 172.21.15.102:0/3296158484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93c806a1d0 0x7f93c81f4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:04.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.605+0000 7f93ccc00640 1 --2- 172.21.15.102:0/3296158484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93c806a1d0 0x7f93c81f4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.605+0000 7f93ccc00640 1 -- 172.21.15.102:0/3296158484 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b4009400 con 0x7f93c80b1c50 2024-03-23T03:48:04.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.605+0000 7f93cd600640 1 --2- 172.21.15.102:0/3296158484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93c806a1d0 0x7f93c81f4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:04.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.605+0000 7f93ccc00640 1 --2- 172.21.15.102:0/3296158484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93c80b1c50 0x7f93c8150830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f93b4034040 tx=0x7f93b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:04.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.605+0000 7f93c5600640 1 -- 172.21.15.102:0/3296158484 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b4032640 con 0x7f93c80b1c50 2024-03-23T03:48:04.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.605+0000 7f93c5600640 1 -- 172.21.15.102:0/3296158484 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b40327c0 con 0x7f93c80b1c50 2024-03-23T03:48:04.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.605+0000 7f93c5600640 1 -- 172.21.15.102:0/3296158484 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b4032b70 con 0x7f93c80b1c50 2024-03-23T03:48:04.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.605+0000 7f93cea00640 1 -- 172.21.15.102:0/3296158484 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c8142440 con 0x7f93c80b1c50 2024-03-23T03:48:04.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.606+0000 7f93cea00640 1 -- 172.21.15.102:0/3296158484 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c8151b70 con 0x7f93c80b1c50 2024-03-23T03:48:04.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.607+0000 7f93cea00640 1 -- 172.21.15.102:0/3296158484 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c8142650 con 0x7f93c80b1c50 2024-03-23T03:48:04.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.607+0000 7f93c5600640 1 -- 172.21.15.102:0/3296158484 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f93b4032cd0 con 0x7f93c80b1c50 2024-03-23T03:48:04.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.608+0000 7f93c5600640 1 --2- 172.21.15.102:0/3296158484 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93a806ac30 0x7f93a806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:04.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.608+0000 7f93c5600640 1 -- 172.21.15.102:0/3296158484 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f93b40aaf40 con 0x7f93c80b1c50 2024-03-23T03:48:04.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.609+0000 7f93cd600640 1 --2- 172.21.15.102:0/3296158484 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93a806ac30 0x7f93a806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:04.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.609+0000 7f93cd600640 1 --2- 172.21.15.102:0/3296158484 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93a806ac30 0x7f93a806d0f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f93bc003eb0 tx=0x7f93bc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:04.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.613+0000 7f93c5600640 1 -- 172.21.15.102:0/3296158484 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f93b4075d50 con 0x7f93c80b1c50 2024-03-23T03:48:04.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.810+0000 7f93cea00640 1 -- 172.21.15.102:0/3296158484 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f93c804e030 con 0x7f93c80b1c50 2024-03-23T03:48:04.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.816+0000 7f93c5600640 1 -- 172.21.15.102:0/3296158484 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f93b4079370 con 0x7f93c80b1c50 2024-03-23T03:48:04.817 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:48:04.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.819+0000 7f93cea00640 1 -- 172.21.15.102:0/3296158484 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93a806ac30 msgr2=0x7f93a806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:04.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.819+0000 7f93cea00640 1 --2- 172.21.15.102:0/3296158484 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93a806ac30 0x7f93a806d0f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f93bc003eb0 tx=0x7f93bc009040 comp rx=0 tx=0).stop 2024-03-23T03:48:04.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.819+0000 7f93cea00640 1 -- 172.21.15.102:0/3296158484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93c80b1c50 msgr2=0x7f93c8150830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:04.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.819+0000 7f93cea00640 1 --2- 172.21.15.102:0/3296158484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93c80b1c50 0x7f93c8150830 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f93b4034040 tx=0x7f93b40046c0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.819+0000 7f93cea00640 1 -- 172.21.15.102:0/3296158484 shutdown_connections 2024-03-23T03:48:04.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.819+0000 7f93cea00640 1 --2- 172.21.15.102:0/3296158484 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93a806ac30 0x7f93a806d0f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.819+0000 7f93cea00640 1 --2- 172.21.15.102:0/3296158484 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f93c80b2030 0x7f93c8142960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.819+0000 7f93cea00640 1 --2- 172.21.15.102:0/3296158484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93c80b1c50 0x7f93c8150830 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.819+0000 7f93cea00640 1 --2- 172.21.15.102:0/3296158484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93c806a1d0 0x7f93c81f4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:04.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.819+0000 7f93cea00640 1 -- 172.21.15.102:0/3296158484 >> 172.21.15.102:0/3296158484 conn(0x7f93c80b01b0 msgr2=0x7f93c80ca2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:04.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.819+0000 7f93cea00640 1 -- 172.21.15.102:0/3296158484 shutdown_connections 2024-03-23T03:48:04.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:04.819+0000 7f93cea00640 1 -- 172.21.15.102:0/3296158484 wait complete. 2024-03-23T03:48:04.837 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:48:05.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.037+0000 7f4d27600640 1 Processor -- start 2024-03-23T03:48:05.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.037+0000 7f4d27600640 1 -- start start 2024-03-23T03:48:05.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.038+0000 7f4d27600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4d200cbae0 0x7f4d2006c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:05.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.039+0000 7f4d27600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2015ae60 con 0x7f4d200ce930 2024-03-23T03:48:05.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.039+0000 7f4d27600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d2015b140 con 0x7f4d200cb660 2024-03-23T03:48:05.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.039+0000 7f4d27600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2015bae0 con 0x7f4d200cbae0 2024-03-23T03:48:05.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.039+0000 7f4d24c00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4d200ce930 0x7f4d200ced10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45138/0 (socket says 172.21.15.102:45138) 2024-03-23T03:48:05.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.039+0000 7f4d24c00640 1 -- 172.21.15.102:0/3306451571 learned_addr learned my addr 172.21.15.102:0/3306451571 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:05.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.039+0000 7f4d1fe00640 1 -- 172.21.15.102:0/3306451571 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196577878 0 0) 0x7f4d2015ae60 con 0x7f4d200ce930 2024-03-23T03:48:05.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.039+0000 7f4d25600640 1 --2- 172.21.15.102:0/3306451571 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4d200cbae0 0x7f4d2006c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:05.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.039+0000 7f4d1fe00640 1 -- 172.21.15.102:0/3306451571 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d00003520 con 0x7f4d200ce930 2024-03-23T03:48:05.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d1fe00640 1 -- 172.21.15.102:0/3306451571 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2335146211 0 0) 0x7f4d00003520 con 0x7f4d200ce930 2024-03-23T03:48:05.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d1fe00640 1 -- 172.21.15.102:0/3306451571 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4d200cbae0 msgr2=0x7f4d2006c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:05.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d1fe00640 1 --2- 172.21.15.102:0/3306451571 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4d200cbae0 0x7f4d2006c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d1fe00640 1 -- 172.21.15.102:0/3306451571 >> v1:172.21.15.173:6789/0 conn(0x7f4d200cb660 legacy=0x7f4d2006c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:05.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d1fe00640 1 -- 172.21.15.102:0/3306451571 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d2015b420 con 0x7f4d200ce930 2024-03-23T03:48:05.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d1fe00640 1 -- 172.21.15.102:0/3306451571 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f4d08001670 con 0x7f4d200ce930 2024-03-23T03:48:05.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d1fe00640 1 -- 172.21.15.102:0/3306451571 >> v1:172.21.15.102:6789/0 conn(0x7f4d200ce930 legacy=0x7f4d200ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:05.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d1fe00640 1 --2- 172.21.15.102:0/3306451571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d000039a0 0x7f4d00003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:05.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d1fe00640 1 --2- 172.21.15.102:0/3306451571 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4d200cbae0 0x7f4d000047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:05.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d1fe00640 1 --2- 172.21.15.102:0/3306451571 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d00004d00 0x7f4d00005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:05.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d1fe00640 1 -- 172.21.15.102:0/3306451571 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2015bae0 con 0x7f4d00004d00 2024-03-23T03:48:05.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d1fe00640 1 -- 172.21.15.102:0/3306451571 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0000ba30 con 0x7f4d000039a0 2024-03-23T03:48:05.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d1fe00640 1 -- 172.21.15.102:0/3306451571 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d0000bbf0 con 0x7f4d200cbae0 2024-03-23T03:48:05.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d25600640 1 --2- 172.21.15.102:0/3306451571 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d00004d00 0x7f4d00005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:05.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d25600640 1 -- 172.21.15.102:0/3306451571 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4d200cbae0 msgr2=0x7f4d000047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:05.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d25600640 1 --2- 172.21.15.102:0/3306451571 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4d200cbae0 0x7f4d000047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d25600640 1 -- 172.21.15.102:0/3306451571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d000039a0 msgr2=0x7f4d00003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:05.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d25600640 1 --2- 172.21.15.102:0/3306451571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d000039a0 0x7f4d00003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.040+0000 7f4d25600640 1 -- 172.21.15.102:0/3306451571 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d00003520 con 0x7f4d00004d00 2024-03-23T03:48:05.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.041+0000 7f4d25600640 1 --2- 172.21.15.102:0/3306451571 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d00004d00 0x7f4d00005180 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009f90 tx=0x7f4d000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a974ece751096ac8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:05.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.042+0000 7f4d1fe00640 1 -- 172.21.15.102:0/3306451571 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d1802cc20 con 0x7f4d00004d00 2024-03-23T03:48:05.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.042+0000 7f4d1fe00640 1 -- 172.21.15.102:0/3306451571 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d1800f040 con 0x7f4d00004d00 2024-03-23T03:48:05.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.042+0000 7f4d1fe00640 1 -- 172.21.15.102:0/3306451571 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18034440 con 0x7f4d00004d00 2024-03-23T03:48:05.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.042+0000 7f4d27600640 1 -- 172.21.15.102:0/3306451571 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d00004d00 msgr2=0x7f4d00005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:05.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.042+0000 7f4d27600640 1 --2- 172.21.15.102:0/3306451571 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d00004d00 0x7f4d00005180 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f4d18009f90 tx=0x7f4d000057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.042+0000 7f4d25600640 1 -- 172.21.15.102:0/3306451571 reap_dead start 2024-03-23T03:48:05.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.042+0000 7f4d27600640 1 -- 172.21.15.102:0/3306451571 shutdown_connections 2024-03-23T03:48:05.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.042+0000 7f4d27600640 1 -- 172.21.15.102:0/3306451571 >> 172.21.15.102:0/3306451571 conn(0x7f4d2006a0b0 msgr2=0x7f4d2006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:05.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.043+0000 7f4d27600640 1 -- 172.21.15.102:0/3306451571 shutdown_connections 2024-03-23T03:48:05.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.043+0000 7f4d27600640 1 -- 172.21.15.102:0/3306451571 wait complete. 2024-03-23T03:48:05.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.044+0000 7f4d27600640 1 Processor -- start 2024-03-23T03:48:05.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.044+0000 7f4d27600640 1 -- start start 2024-03-23T03:48:05.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.045+0000 7f4d27600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4d00004d00 0x7f4d2014eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:05.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.045+0000 7f4d27600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d200b31e0 0x7f4d200b16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:05.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.045+0000 7f4d27600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d200b1c00 0x7f4d201536d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:05.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.045+0000 7f4d27600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2014f2e0 con 0x7f4d200b1c00 2024-03-23T03:48:05.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.045+0000 7f4d27600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d200b2140 con 0x7f4d200b31e0 2024-03-23T03:48:05.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.045+0000 7f4d27600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d200b2420 con 0x7f4d00004d00 2024-03-23T03:48:05.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.046+0000 7f4d25600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d200b1c00 0x7f4d201536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:05.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.046+0000 7f4d25600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d200b1c00 0x7f4d201536d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41368/0 (socket says 172.21.15.102:41368) 2024-03-23T03:48:05.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.046+0000 7f4d25600640 1 -- 172.21.15.102:0/2397704457 learned_addr learned my addr 172.21.15.102:0/2397704457 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:05.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.046+0000 7f4d17e00640 1 --2- 172.21.15.102:0/2397704457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d200b31e0 0x7f4d200b16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:05.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.046+0000 7f4d24c00640 1 --2- 172.21.15.102:0/2397704457 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4d00004d00 0x7f4d2014eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:05.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.046+0000 7f4d25600640 1 -- 172.21.15.102:0/2397704457 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4d00004d00 msgr2=0x7f4d2014eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:05.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.046+0000 7f4d25600640 1 --2- 172.21.15.102:0/2397704457 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4d00004d00 0x7f4d2014eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.046+0000 7f4d25600640 1 -- 172.21.15.102:0/2397704457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d200b31e0 msgr2=0x7f4d200b16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:05.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.046+0000 7f4d25600640 1 --2- 172.21.15.102:0/2397704457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d200b31e0 0x7f4d200b16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.046+0000 7f4d25600640 1 -- 172.21.15.102:0/2397704457 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d20155280 con 0x7f4d200b1c00 2024-03-23T03:48:05.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.046+0000 7f4d24c00640 1 --2- 172.21.15.102:0/2397704457 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4d00004d00 0x7f4d2014eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:05.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.046+0000 7f4d25600640 1 --2- 172.21.15.102:0/2397704457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d200b1c00 0x7f4d201536d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4d18037040 tx=0x7f4d18009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:05.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.047+0000 7f4d1d600640 1 -- 172.21.15.102:0/2397704457 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18034d50 con 0x7f4d200b1c00 2024-03-23T03:48:05.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.047+0000 7f4d1d600640 1 -- 172.21.15.102:0/2397704457 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d18003990 con 0x7f4d200b1c00 2024-03-23T03:48:05.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.047+0000 7f4d1d600640 1 -- 172.21.15.102:0/2397704457 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d18003d20 con 0x7f4d200b1c00 2024-03-23T03:48:05.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.047+0000 7f4d27600640 1 -- 172.21.15.102:0/2397704457 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d200b2700 con 0x7f4d200b1c00 2024-03-23T03:48:05.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.047+0000 7f4d27600640 1 -- 172.21.15.102:0/2397704457 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d200b2cc0 con 0x7f4d200b1c00 2024-03-23T03:48:05.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.048+0000 7f4d27600640 1 -- 172.21.15.102:0/2397704457 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d200b4120 con 0x7f4d200b1c00 2024-03-23T03:48:05.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.049+0000 7f4d1d600640 1 -- 172.21.15.102:0/2397704457 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4d18004030 con 0x7f4d200b1c00 2024-03-23T03:48:05.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.050+0000 7f4d1d600640 1 --2- 172.21.15.102:0/2397704457 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4cfc06ac30 0x7f4cfc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:05.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.050+0000 7f4d1d600640 1 -- 172.21.15.102:0/2397704457 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f4d180ab170 con 0x7f4d200b1c00 2024-03-23T03:48:05.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.050+0000 7f4d24c00640 1 --2- 172.21.15.102:0/2397704457 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4cfc06ac30 0x7f4cfc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:05.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.051+0000 7f4d24c00640 1 --2- 172.21.15.102:0/2397704457 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4cfc06ac30 0x7f4cfc06d0f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4d08001eb0 tx=0x7f4d08009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:05.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.055+0000 7f4d1d600640 1 -- 172.21.15.102:0/2397704457 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4d18076040 con 0x7f4d200b1c00 2024-03-23T03:48:05.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.260+0000 7f4d27600640 1 -- 172.21.15.102:0/2397704457 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4d2014ff80 con 0x7f4d200b1c00 2024-03-23T03:48:05.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.261+0000 7f4d1d600640 1 -- 172.21.15.102:0/2397704457 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v28) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4d18079660 con 0x7f4d200b1c00 2024-03-23T03:48:05.261 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:48:05.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.263+0000 7f4d27600640 1 -- 172.21.15.102:0/2397704457 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4cfc06ac30 msgr2=0x7f4cfc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:05.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.263+0000 7f4d27600640 1 --2- 172.21.15.102:0/2397704457 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4cfc06ac30 0x7f4cfc06d0f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4d08001eb0 tx=0x7f4d08009040 comp rx=0 tx=0).stop 2024-03-23T03:48:05.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.263+0000 7f4d27600640 1 -- 172.21.15.102:0/2397704457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d200b1c00 msgr2=0x7f4d201536d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:05.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.263+0000 7f4d27600640 1 --2- 172.21.15.102:0/2397704457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d200b1c00 0x7f4d201536d0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4d18037040 tx=0x7f4d18009750 comp rx=0 tx=0).stop 2024-03-23T03:48:05.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.263+0000 7f4d27600640 1 -- 172.21.15.102:0/2397704457 shutdown_connections 2024-03-23T03:48:05.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.263+0000 7f4d27600640 1 --2- 172.21.15.102:0/2397704457 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4cfc06ac30 0x7f4cfc06d0f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.263+0000 7f4d27600640 1 --2- 172.21.15.102:0/2397704457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d200b1c00 0x7f4d201536d0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.263+0000 7f4d27600640 1 --2- 172.21.15.102:0/2397704457 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4d200b31e0 0x7f4d200b16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.263+0000 7f4d27600640 1 --2- 172.21.15.102:0/2397704457 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4d00004d00 0x7f4d2014eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.264+0000 7f4d27600640 1 -- 172.21.15.102:0/2397704457 >> 172.21.15.102:0/2397704457 conn(0x7f4d2006a0b0 msgr2=0x7f4d2015beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:05.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.264+0000 7f4d27600640 1 -- 172.21.15.102:0/2397704457 shutdown_connections 2024-03-23T03:48:05.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.264+0000 7f4d27600640 1 -- 172.21.15.102:0/2397704457 wait complete. 2024-03-23T03:48:05.283 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2024-03-23T03:48:05.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:05.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2024-03-23T03:48:05.389 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:48:05.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:05.390 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-23T03:48:05.441 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-23T03:48:05.441 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:48:05.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.647+0000 7f012e600640 1 Processor -- start 2024-03-23T03:48:05.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.648+0000 7f012e600640 1 -- start start 2024-03-23T03:48:05.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.649+0000 7f012e600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f012815a940 0x7f0128099450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:05.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.649+0000 7f012e600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01280cba30 con 0x7f01280cbec0 2024-03-23T03:48:05.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.649+0000 7f012e600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012815ad20 con 0x7f01280cc320 2024-03-23T03:48:05.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.649+0000 7f012e600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01280cb140 con 0x7f012815a940 2024-03-23T03:48:05.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.650+0000 7f0127400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f012815a940 0x7f0128099450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:05.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.650+0000 7f0126a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f01280cbec0 0x7f01280c4180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45144/0 (socket says 172.21.15.102:45144) 2024-03-23T03:48:05.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.650+0000 7f0126a00640 1 -- 172.21.15.102:0/1638962811 learned_addr learned my addr 172.21.15.102:0/1638962811 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:05.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.650+0000 7f0127400640 1 -- 172.21.15.102:0/1638962811 >> v1:172.21.15.173:6789/0 conn(0x7f01280cc320 legacy=0x7f012815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:05.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.650+0000 7f0127400640 1 -- 172.21.15.102:0/1638962811 >> v1:172.21.15.102:6789/0 conn(0x7f01280cbec0 legacy=0x7f01280c4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:05.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.650+0000 7f0127400640 1 -- 172.21.15.102:0/1638962811 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0128099ae0 con 0x7f012815a940 2024-03-23T03:48:05.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.650+0000 7f0126000640 1 -- 172.21.15.102:0/1638962811 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95014376 0 0) 0x7f01280cba30 con 0x7f01280cbec0 2024-03-23T03:48:05.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.651+0000 7f0127400640 1 --2- 172.21.15.102:0/1638962811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f012815a940 0x7f0128099450 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f01180097a0 tx=0x7f011802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce30957ff68f1afd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:05.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.652+0000 7f0126000640 1 -- 172.21.15.102:0/1638962811 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118031830 con 0x7f012815a940 2024-03-23T03:48:05.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.652+0000 7f0126000640 1 -- 172.21.15.102:0/1638962811 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0118031990 con 0x7f012815a940 2024-03-23T03:48:05.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.652+0000 7f0126000640 1 -- 172.21.15.102:0/1638962811 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0118031d20 con 0x7f012815a940 2024-03-23T03:48:05.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.652+0000 7f012e600640 1 -- 172.21.15.102:0/1638962811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f012815a940 msgr2=0x7f0128099450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:05.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.652+0000 7f012e600640 1 --2- 172.21.15.102:0/1638962811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f012815a940 0x7f0128099450 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f01180097a0 tx=0x7f011802b8d0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.652+0000 7f012e600640 1 -- 172.21.15.102:0/1638962811 shutdown_connections 2024-03-23T03:48:05.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.652+0000 7f012e600640 1 --2- 172.21.15.102:0/1638962811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f012815a940 0x7f0128099450 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.652+0000 7f012e600640 1 -- 172.21.15.102:0/1638962811 >> 172.21.15.102:0/1638962811 conn(0x7f01280b2120 msgr2=0x7f01280b2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:05.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.652+0000 7f012e600640 1 -- 172.21.15.102:0/1638962811 shutdown_connections 2024-03-23T03:48:05.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.652+0000 7f012e600640 1 -- 172.21.15.102:0/1638962811 wait complete. 2024-03-23T03:48:05.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.654+0000 7f012e600640 1 Processor -- start 2024-03-23T03:48:05.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.654+0000 7f012e600640 1 -- start start 2024-03-23T03:48:05.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.655+0000 7f012e600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01280cbec0 0x7f01281ef8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:05.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.655+0000 7f012e600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f012815a940 0x7f0128156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:05.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.655+0000 7f012e600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0128156580 0x7f01280ce5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:05.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.655+0000 7f012e600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0128145940 con 0x7f0128156580 2024-03-23T03:48:05.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.655+0000 7f012e600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0128145e10 con 0x7f01280cbec0 2024-03-23T03:48:05.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.655+0000 7f012e600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0128156be0 con 0x7f012815a940 2024-03-23T03:48:05.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.655+0000 7f0127e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0128156580 0x7f01280ce5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:05.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.655+0000 7f0127e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0128156580 0x7f01280ce5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41386/0 (socket says 172.21.15.102:41386) 2024-03-23T03:48:05.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.655+0000 7f0127e00640 1 -- 172.21.15.102:0/2387527539 learned_addr learned my addr 172.21.15.102:0/2387527539 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:05.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.656+0000 7f0127400640 1 --2- 172.21.15.102:0/2387527539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01280cbec0 0x7f01281ef8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.656+0000 7f0127e00640 1 -- 172.21.15.102:0/2387527539 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f012815a940 msgr2=0x7f0128156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.656+0000 7f0127e00640 1 --2- 172.21.15.102:0/2387527539 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f012815a940 0x7f0128156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.656+0000 7f0127e00640 1 -- 172.21.15.102:0/2387527539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01280cbec0 msgr2=0x7f01281ef8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.656+0000 7f0127e00640 1 --2- 172.21.15.102:0/2387527539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01280cbec0 0x7f01281ef8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.656+0000 7f0127e00640 1 -- 172.21.15.102:0/2387527539 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0118009400 con 0x7f0128156580 2024-03-23T03:48:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.656+0000 7f0127400640 1 --2- 172.21.15.102:0/2387527539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01280cbec0 0x7f01281ef8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.656+0000 7f0127e00640 1 --2- 172.21.15.102:0/2387527539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0128156580 0x7f01280ce5b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f012000c6d0 tx=0x7f012000cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.656+0000 7f011f400640 1 -- 172.21.15.102:0/2387527539 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012000a840 con 0x7f0128156580 2024-03-23T03:48:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.656+0000 7f011f400640 1 -- 172.21.15.102:0/2387527539 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f012000a9a0 con 0x7f0128156580 2024-03-23T03:48:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.656+0000 7f011f400640 1 -- 172.21.15.102:0/2387527539 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f012000ad90 con 0x7f0128156580 2024-03-23T03:48:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.656+0000 7f012e600640 1 -- 172.21.15.102:0/2387527539 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01280c77a0 con 0x7f0128156580 2024-03-23T03:48:05.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.657+0000 7f012e600640 1 -- 172.21.15.102:0/2387527539 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01280c7d60 con 0x7f0128156580 2024-03-23T03:48:05.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.658+0000 7f012e600640 1 -- 172.21.15.102:0/2387527539 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0128066bb0 con 0x7f0128156580 2024-03-23T03:48:05.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.659+0000 7f011f400640 1 -- 172.21.15.102:0/2387527539 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0120012630 con 0x7f0128156580 2024-03-23T03:48:05.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.659+0000 7f011f400640 1 --2- 172.21.15.102:0/2387527539 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f010806ac00 0x7f010806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:05.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.660+0000 7f011f400640 1 -- 172.21.15.102:0/2387527539 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0120089d20 con 0x7f0128156580 2024-03-23T03:48:05.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.660+0000 7f0127400640 1 --2- 172.21.15.102:0/2387527539 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f010806ac00 0x7f010806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:05.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.660+0000 7f0127400640 1 --2- 172.21.15.102:0/2387527539 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f010806ac00 0x7f010806d0c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0118036000 tx=0x7f011803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.664+0000 7f011f400640 1 -- 172.21.15.102:0/2387527539 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0120054c80 con 0x7f0128156580 2024-03-23T03:48:05.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.900+0000 7f012e600640 1 -- 172.21.15.102:0/2387527539 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01280b2e80 con 0x7f0128156580 2024-03-23T03:48:05.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.901+0000 7f011f400640 1 -- 172.21.15.102:0/2387527539 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4739 (secure 0 0 0) 0x7f0120016070 con 0x7f0128156580 2024-03-23T03:48:05.902 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:05.902 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/891818172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":891818172},{"type":"v1","addr":"172.21.15.102:6836","nonce":891818172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2404489233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2404489233},{"type":"v1","addr":"172.21.15.102:6837","nonce":2404489233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/1839967745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":1839967745},{"type":"v1","addr":"172.21.15.173:6835","nonce":1839967745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:46:53.912287+0000","modified":"2024-03-23T03:47:14.818670+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6834/1462637509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1462637509},{"type":"v1","addr":"172.21.15.173:6834","nonce":1462637509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-23T03:48:05.902 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 8 2024-03-23T03:48:05.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.904+0000 7f012e600640 1 -- 172.21.15.102:0/2387527539 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f010806ac00 msgr2=0x7f010806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:05.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.904+0000 7f012e600640 1 --2- 172.21.15.102:0/2387527539 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f010806ac00 0x7f010806d0c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0118036000 tx=0x7f011803c040 comp rx=0 tx=0).stop 2024-03-23T03:48:05.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.904+0000 7f012e600640 1 -- 172.21.15.102:0/2387527539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0128156580 msgr2=0x7f01280ce5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:05.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.904+0000 7f012e600640 1 --2- 172.21.15.102:0/2387527539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0128156580 0x7f01280ce5b0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f012000c6d0 tx=0x7f012000cb90 comp rx=0 tx=0).stop 2024-03-23T03:48:05.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.904+0000 7f012e600640 1 -- 172.21.15.102:0/2387527539 shutdown_connections 2024-03-23T03:48:05.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.904+0000 7f012e600640 1 --2- 172.21.15.102:0/2387527539 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f010806ac00 0x7f010806d0c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.904+0000 7f012e600640 1 --2- 172.21.15.102:0/2387527539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0128156580 0x7f01280ce5b0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.904+0000 7f012e600640 1 --2- 172.21.15.102:0/2387527539 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f012815a940 0x7f0128156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.904+0000 7f012e600640 1 --2- 172.21.15.102:0/2387527539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01280cbec0 0x7f01281ef8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:05.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.904+0000 7f012e600640 1 -- 172.21.15.102:0/2387527539 >> 172.21.15.102:0/2387527539 conn(0x7f01280b2120 msgr2=0x7f01280c4120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:05.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.905+0000 7f012e600640 1 -- 172.21.15.102:0/2387527539 shutdown_connections 2024-03-23T03:48:05.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:05.905+0000 7f012e600640 1 -- 172.21.15.102:0/2387527539 wait complete. 2024-03-23T03:48:05.924 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:48:06.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.117+0000 7f0bc7e00640 1 Processor -- start 2024-03-23T03:48:06.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.118+0000 7f0bc7e00640 1 -- start start 2024-03-23T03:48:06.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.119+0000 7f0bc7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0bc815cab0 0x7f0bc80b8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:06.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.119+0000 7f0bc7e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f0bc80ceeb0 0x7f0bc80cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:06.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.119+0000 7f0bc7e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc80cf850 con 0x7f0bc80b4120 2024-03-23T03:48:06.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.119+0000 7f0bc7e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc815ce90 con 0x7f0bc815cab0 2024-03-23T03:48:06.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.119+0000 7f0bc7e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc80cddf0 con 0x7f0bc80ceeb0 2024-03-23T03:48:06.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.119+0000 7f0bc6a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0bc815cab0 0x7f0bc80b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:06.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.119+0000 7f0bc6a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0bc815cab0 0x7f0bc80b8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41400/0 (socket says 172.21.15.102:41400) 2024-03-23T03:48:06.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.119+0000 7f0bc6a00640 1 -- 172.21.15.102:0/854577116 learned_addr learned my addr 172.21.15.102:0/854577116 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:06.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.120+0000 7f0bbfe00640 1 --2- 172.21.15.102:0/854577116 >> v2:172.21.15.173:3300/0 conn(0x7f0bc80ceeb0 0x7f0bc80cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:06.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.120+0000 7f0bc6a00640 1 -- 172.21.15.102:0/854577116 >> v2:172.21.15.173:3300/0 conn(0x7f0bc80ceeb0 msgr2=0x7f0bc80cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:06.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.120+0000 7f0bc6a00640 1 --2- 172.21.15.102:0/854577116 >> v2:172.21.15.173:3300/0 conn(0x7f0bc80ceeb0 0x7f0bc80cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.120+0000 7f0bc6a00640 1 -- 172.21.15.102:0/854577116 >> v1:172.21.15.173:6789/0 conn(0x7f0bc80b4120 legacy=0x7f0bc80b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:06.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.120+0000 7f0bc6a00640 1 -- 172.21.15.102:0/854577116 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc80cd570 con 0x7f0bc815cab0 2024-03-23T03:48:06.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.121+0000 7f0bc6a00640 1 --2- 172.21.15.102:0/854577116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0bc815cab0 0x7f0bc80b8050 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0bc80d3750 tx=0x7f0bb802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc53351e843ec4ae server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:06.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.121+0000 7f0bc6000640 1 -- 172.21.15.102:0/854577116 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8031870 con 0x7f0bc815cab0 2024-03-23T03:48:06.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.121+0000 7f0bc6000640 1 -- 172.21.15.102:0/854577116 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb80319d0 con 0x7f0bc815cab0 2024-03-23T03:48:06.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.121+0000 7f0bc6000640 1 -- 172.21.15.102:0/854577116 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb8031d60 con 0x7f0bc815cab0 2024-03-23T03:48:06.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.122+0000 7f0bc7e00640 1 -- 172.21.15.102:0/854577116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0bc815cab0 msgr2=0x7f0bc80b8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:06.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.122+0000 7f0bc7e00640 1 --2- 172.21.15.102:0/854577116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0bc815cab0 0x7f0bc80b8050 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0bc80d3750 tx=0x7f0bb802b910 comp rx=0 tx=0).stop 2024-03-23T03:48:06.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.122+0000 7f0bc7e00640 1 -- 172.21.15.102:0/854577116 shutdown_connections 2024-03-23T03:48:06.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.122+0000 7f0bc7e00640 1 --2- 172.21.15.102:0/854577116 >> v2:172.21.15.173:3300/0 conn(0x7f0bc80ceeb0 0x7f0bc80cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.122+0000 7f0bc7e00640 1 --2- 172.21.15.102:0/854577116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0bc815cab0 0x7f0bc80b8050 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.122+0000 7f0bc7e00640 1 -- 172.21.15.102:0/854577116 >> 172.21.15.102:0/854577116 conn(0x7f0bc806ae90 msgr2=0x7f0bc806b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:06.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.122+0000 7f0bc7e00640 1 -- 172.21.15.102:0/854577116 shutdown_connections 2024-03-23T03:48:06.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.122+0000 7f0bc7e00640 1 -- 172.21.15.102:0/854577116 wait complete. 2024-03-23T03:48:06.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.124+0000 7f0bc7e00640 1 Processor -- start 2024-03-23T03:48:06.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.124+0000 7f0bc7e00640 1 -- start start 2024-03-23T03:48:06.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.125+0000 7f0bc7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bc80b4120 0x7f0bc815f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:06.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.125+0000 7f0bc7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0bc80ceeb0 0x7f0bc815f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:06.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.125+0000 7f0bc7e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0bc815cab0 0x7f0bc80c7e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:06.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.125+0000 7f0bc7e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc80c6f40 con 0x7f0bc80ceeb0 2024-03-23T03:48:06.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.125+0000 7f0bc7e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc80c6370 con 0x7f0bc80b4120 2024-03-23T03:48:06.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.125+0000 7f0bc7e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc80c6610 con 0x7f0bc815cab0 2024-03-23T03:48:06.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.125+0000 7f0bbfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0bc80ceeb0 0x7f0bc815f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:06.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.125+0000 7f0bbfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0bc80ceeb0 0x7f0bc815f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41408/0 (socket says 172.21.15.102:41408) 2024-03-23T03:48:06.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.125+0000 7f0bbfe00640 1 -- 172.21.15.102:0/1115894930 learned_addr learned my addr 172.21.15.102:0/1115894930 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:06.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.125+0000 7f0bbfe00640 1 -- 172.21.15.102:0/1115894930 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0bc815cab0 msgr2=0x7f0bc80c7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:06.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.125+0000 7f0bbfe00640 1 --2- 172.21.15.102:0/1115894930 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0bc815cab0 0x7f0bc80c7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.125+0000 7f0bbfe00640 1 -- 172.21.15.102:0/1115894930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bc80b4120 msgr2=0x7f0bc815f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:06.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.125+0000 7f0bbfe00640 1 --2- 172.21.15.102:0/1115894930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bc80b4120 0x7f0bc815f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.126+0000 7f0bbfe00640 1 -- 172.21.15.102:0/1115894930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb80094d0 con 0x7f0bc80ceeb0 2024-03-23T03:48:06.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.126+0000 7f0bbfe00640 1 --2- 172.21.15.102:0/1115894930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0bc80ceeb0 0x7f0bc815f6e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0bb000d800 tx=0x7f0bb000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.126+0000 7f0bbea00640 1 -- 172.21.15.102:0/1115894930 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb000b6c0 con 0x7f0bc80ceeb0 2024-03-23T03:48:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.126+0000 7f0bbea00640 1 -- 172.21.15.102:0/1115894930 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb000b820 con 0x7f0bc80ceeb0 2024-03-23T03:48:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.126+0000 7f0bbea00640 1 -- 172.21.15.102:0/1115894930 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb000bbb0 con 0x7f0bc80ceeb0 2024-03-23T03:48:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.126+0000 7f0bc7e00640 1 -- 172.21.15.102:0/1115894930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc80c68f0 con 0x7f0bc80ceeb0 2024-03-23T03:48:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.126+0000 7f0bc7e00640 1 -- 172.21.15.102:0/1115894930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc81f26f0 con 0x7f0bc80ceeb0 2024-03-23T03:48:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.127+0000 7f0bc7e00640 1 -- 172.21.15.102:0/1115894930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc804e030 con 0x7f0bc80ceeb0 2024-03-23T03:48:06.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.128+0000 7f0bbea00640 1 -- 172.21.15.102:0/1115894930 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0bb000bd10 con 0x7f0bc80ceeb0 2024-03-23T03:48:06.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.129+0000 7f0bbea00640 1 --2- 172.21.15.102:0/1115894930 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0ba406ac00 0x7f0ba406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:06.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.129+0000 7f0bbea00640 1 -- 172.21.15.102:0/1115894930 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0bb0089cd0 con 0x7f0bc80ceeb0 2024-03-23T03:48:06.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.129+0000 7f0bc6a00640 1 --2- 172.21.15.102:0/1115894930 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0ba406ac00 0x7f0ba406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:06.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.130+0000 7f0bc6a00640 1 --2- 172.21.15.102:0/1115894930 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0ba406ac00 0x7f0ba406d0c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8036000 tx=0x7f0bb8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:06.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.133+0000 7f0bbea00640 1 -- 172.21.15.102:0/1115894930 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0bb0054ae0 con 0x7f0bc80ceeb0 2024-03-23T03:48:06.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.327+0000 7f0bc7e00640 1 -- 172.21.15.102:0/1115894930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bc80c7080 con 0x7f0bc80ceeb0 2024-03-23T03:48:06.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.329+0000 7f0bbea00640 1 -- 172.21.15.102:0/1115894930 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18260 (secure 0 0 0) 0x7f0bc80c7080 con 0x7f0bc80ceeb0 2024-03-23T03:48:06.331 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:06.331 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":28,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:47:41.647786+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:48:06.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.332+0000 7f0bc7e00640 1 -- 172.21.15.102:0/1115894930 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0ba406ac00 msgr2=0x7f0ba406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:06.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.332+0000 7f0bc7e00640 1 --2- 172.21.15.102:0/1115894930 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0ba406ac00 0x7f0ba406d0c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8036000 tx=0x7f0bb8030000 comp rx=0 tx=0).stop 2024-03-23T03:48:06.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.332+0000 7f0bc7e00640 1 -- 172.21.15.102:0/1115894930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0bc80ceeb0 msgr2=0x7f0bc815f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:06.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.332+0000 7f0bc7e00640 1 --2- 172.21.15.102:0/1115894930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0bc80ceeb0 0x7f0bc815f6e0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0bb000d800 tx=0x7f0bb000dcc0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.332+0000 7f0bc7e00640 1 -- 172.21.15.102:0/1115894930 shutdown_connections 2024-03-23T03:48:06.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.332+0000 7f0bc7e00640 1 --2- 172.21.15.102:0/1115894930 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0ba406ac00 0x7f0ba406d0c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.332+0000 7f0bc7e00640 1 --2- 172.21.15.102:0/1115894930 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0bc815cab0 0x7f0bc80c7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.332+0000 7f0bc7e00640 1 --2- 172.21.15.102:0/1115894930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0bc80ceeb0 0x7f0bc815f6e0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.332+0000 7f0bc7e00640 1 --2- 172.21.15.102:0/1115894930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0bc80b4120 0x7f0bc815f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.332+0000 7f0bc7e00640 1 -- 172.21.15.102:0/1115894930 >> 172.21.15.102:0/1115894930 conn(0x7f0bc806ae90 msgr2=0x7f0bc806b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:06.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.332+0000 7f0bc7e00640 1 -- 172.21.15.102:0/1115894930 shutdown_connections 2024-03-23T03:48:06.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.332+0000 7f0bc7e00640 1 -- 172.21.15.102:0/1115894930 wait complete. 2024-03-23T03:48:06.352 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:46:52.158650+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:46:52.914275+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-23T03:46:55.983754+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '26', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-23T03:48:06.353 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:48:06.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.554+0000 7f7d32e00640 1 Processor -- start 2024-03-23T03:48:06.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.554+0000 7f7d32e00640 1 -- start start 2024-03-23T03:48:06.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.556+0000 7f7d32e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7d2c158e20 0x7f7d2c0b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:06.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.556+0000 7f7d32e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d2c070860 0x7f7d2c06c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:06.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.556+0000 7f7d32e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d2c0b20d0 con 0x7f7d2c0b1b00 2024-03-23T03:48:06.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.556+0000 7f7d32e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d2c0cba90 con 0x7f7d2c158e20 2024-03-23T03:48:06.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.556+0000 7f7d32e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c06c3c0 con 0x7f7d2c070860 2024-03-23T03:48:06.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.556+0000 7f7d2be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7d2c158e20 0x7f7d2c0b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:06.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.556+0000 7f7d2be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7d2c158e20 0x7f7d2c0b15c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41412/0 (socket says 172.21.15.102:41412) 2024-03-23T03:48:06.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.556+0000 7f7d2be00640 1 -- 172.21.15.102:0/3861277073 learned_addr learned my addr 172.21.15.102:0/3861277073 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:06.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.556+0000 7f7d30e00640 1 --2- 172.21.15.102:0/3861277073 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d2c070860 0x7f7d2c06c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:06.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.557+0000 7f7d2be00640 1 -- 172.21.15.102:0/3861277073 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d2c070860 msgr2=0x7f7d2c06c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:06.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.557+0000 7f7d2be00640 1 --2- 172.21.15.102:0/3861277073 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d2c070860 0x7f7d2c06c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.557+0000 7f7d2be00640 1 -- 172.21.15.102:0/3861277073 >> v1:172.21.15.173:6789/0 conn(0x7f7d2c0b1b00 legacy=0x7f7d2c070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:06.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.557+0000 7f7d2be00640 1 -- 172.21.15.102:0/3861277073 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d2c159200 con 0x7f7d2c158e20 2024-03-23T03:48:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.558+0000 7f7d2be00640 1 --2- 172.21.15.102:0/3861277073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d2c158e20 0x7f7d2c0b15c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009870 tx=0x7f7d2402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81db06bd1dc4fa60 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.558+0000 7f7d2aa00640 1 -- 172.21.15.102:0/3861277073 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24030840 con 0x7f7d2c158e20 2024-03-23T03:48:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.558+0000 7f7d2aa00640 1 -- 172.21.15.102:0/3861277073 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d240309a0 con 0x7f7d2c158e20 2024-03-23T03:48:06.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.558+0000 7f7d2aa00640 1 -- 172.21.15.102:0/3861277073 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24030d30 con 0x7f7d2c158e20 2024-03-23T03:48:06.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.559+0000 7f7d32e00640 1 -- 172.21.15.102:0/3861277073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d2c158e20 msgr2=0x7f7d2c0b15c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:06.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.559+0000 7f7d32e00640 1 --2- 172.21.15.102:0/3861277073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d2c158e20 0x7f7d2c0b15c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009870 tx=0x7f7d2402b980 comp rx=0 tx=0).stop 2024-03-23T03:48:06.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.559+0000 7f7d32e00640 1 -- 172.21.15.102:0/3861277073 shutdown_connections 2024-03-23T03:48:06.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.559+0000 7f7d32e00640 1 --2- 172.21.15.102:0/3861277073 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d2c070860 0x7f7d2c06c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.559+0000 7f7d32e00640 1 --2- 172.21.15.102:0/3861277073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d2c158e20 0x7f7d2c0b15c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.559+0000 7f7d32e00640 1 -- 172.21.15.102:0/3861277073 >> 172.21.15.102:0/3861277073 conn(0x7f7d2c06a0b0 msgr2=0x7f7d2c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:06.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.559+0000 7f7d32e00640 1 -- 172.21.15.102:0/3861277073 shutdown_connections 2024-03-23T03:48:06.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.559+0000 7f7d32e00640 1 -- 172.21.15.102:0/3861277073 wait complete. 2024-03-23T03:48:06.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.561+0000 7f7d32e00640 1 Processor -- start 2024-03-23T03:48:06.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.561+0000 7f7d32e00640 1 -- start start 2024-03-23T03:48:06.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.562+0000 7f7d32e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d2c070860 0x7f7d2c1f20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:06.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.562+0000 7f7d32e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d2c0b1b00 0x7f7d2c0b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:06.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.562+0000 7f7d32e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d2c158e20 0x7f7d2c0b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:06.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.562+0000 7f7d32e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0b0330 con 0x7f7d2c070860 2024-03-23T03:48:06.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.562+0000 7f7d32e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c0b0050 con 0x7f7d2c0b1b00 2024-03-23T03:48:06.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.562+0000 7f7d32e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d2c155310 con 0x7f7d2c158e20 2024-03-23T03:48:06.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.562+0000 7f7d2be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d2c070860 0x7f7d2c1f20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:06.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.562+0000 7f7d2be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d2c070860 0x7f7d2c1f20b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41424/0 (socket says 172.21.15.102:41424) 2024-03-23T03:48:06.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.562+0000 7f7d2be00640 1 -- 172.21.15.102:0/1109822326 learned_addr learned my addr 172.21.15.102:0/1109822326 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:06.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.563+0000 7f7d30e00640 1 --2- 172.21.15.102:0/1109822326 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d2c158e20 0x7f7d2c0b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:06.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.563+0000 7f7d2be00640 1 -- 172.21.15.102:0/1109822326 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d2c158e20 msgr2=0x7f7d2c0b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:06.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.563+0000 7f7d2b400640 1 --2- 172.21.15.102:0/1109822326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d2c0b1b00 0x7f7d2c0b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:06.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.563+0000 7f7d2be00640 1 --2- 172.21.15.102:0/1109822326 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d2c158e20 0x7f7d2c0b3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.563+0000 7f7d2be00640 1 -- 172.21.15.102:0/1109822326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d2c0b1b00 msgr2=0x7f7d2c0b2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:06.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.563+0000 7f7d2be00640 1 --2- 172.21.15.102:0/1109822326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d2c0b1b00 0x7f7d2c0b2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.563+0000 7f7d2be00640 1 -- 172.21.15.102:0/1109822326 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d240094d0 con 0x7f7d2c070860 2024-03-23T03:48:06.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.563+0000 7f7d2be00640 1 --2- 172.21.15.102:0/1109822326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d2c070860 0x7f7d2c1f20b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7d24000c00 tx=0x7f7d24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:06.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.563+0000 7f7d23e00640 1 -- 172.21.15.102:0/1109822326 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24039a90 con 0x7f7d2c070860 2024-03-23T03:48:06.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.563+0000 7f7d23e00640 1 -- 172.21.15.102:0/1109822326 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d24039bf0 con 0x7f7d2c070860 2024-03-23T03:48:06.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.563+0000 7f7d23e00640 1 -- 172.21.15.102:0/1109822326 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d24043410 con 0x7f7d2c070860 2024-03-23T03:48:06.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.563+0000 7f7d32e00640 1 -- 172.21.15.102:0/1109822326 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d2c1555f0 con 0x7f7d2c070860 2024-03-23T03:48:06.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.564+0000 7f7d32e00640 1 -- 172.21.15.102:0/1109822326 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d2c0af360 con 0x7f7d2c070860 2024-03-23T03:48:06.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.565+0000 7f7d32e00640 1 -- 172.21.15.102:0/1109822326 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d2c155170 con 0x7f7d2c070860 2024-03-23T03:48:06.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.566+0000 7f7d23e00640 1 -- 172.21.15.102:0/1109822326 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7d2402f390 con 0x7f7d2c070860 2024-03-23T03:48:06.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.566+0000 7f7d23e00640 1 --2- 172.21.15.102:0/1109822326 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d0806ac30 0x7f7d0806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:06.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.567+0000 7f7d23e00640 1 -- 172.21.15.102:0/1109822326 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7d240abac0 con 0x7f7d2c070860 2024-03-23T03:48:06.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.567+0000 7f7d2b400640 1 --2- 172.21.15.102:0/1109822326 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d0806ac30 0x7f7d0806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:06.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.567+0000 7f7d2b400640 1 --2- 172.21.15.102:0/1109822326 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d0806ac30 0x7f7d0806d0f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7d14007fd0 tx=0x7f7d14009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:06.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.571+0000 7f7d23e00640 1 -- 172.21.15.102:0/1109822326 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7d2c155170 con 0x7f7d2c070860 2024-03-23T03:48:06.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.764+0000 7f7d32e00640 1 -- 172.21.15.102:0/1109822326 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d2c155b10 con 0x7f7d2c070860 2024-03-23T03:48:06.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.765+0000 7f7d23e00640 1 -- 172.21.15.102:0/1109822326 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v28) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7d2c155b10 con 0x7f7d2c070860 2024-03-23T03:48:06.766 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:48:06.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.768+0000 7f7d32e00640 1 -- 172.21.15.102:0/1109822326 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d0806ac30 msgr2=0x7f7d0806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:06.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.768+0000 7f7d32e00640 1 --2- 172.21.15.102:0/1109822326 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d0806ac30 0x7f7d0806d0f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7d14007fd0 tx=0x7f7d14009040 comp rx=0 tx=0).stop 2024-03-23T03:48:06.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.768+0000 7f7d32e00640 1 -- 172.21.15.102:0/1109822326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d2c070860 msgr2=0x7f7d2c1f20b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:06.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.768+0000 7f7d32e00640 1 --2- 172.21.15.102:0/1109822326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d2c070860 0x7f7d2c1f20b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7d24000c00 tx=0x7f7d24004730 comp rx=0 tx=0).stop 2024-03-23T03:48:06.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.768+0000 7f7d32e00640 1 -- 172.21.15.102:0/1109822326 shutdown_connections 2024-03-23T03:48:06.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.768+0000 7f7d32e00640 1 --2- 172.21.15.102:0/1109822326 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d0806ac30 0x7f7d0806d0f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.768+0000 7f7d32e00640 1 --2- 172.21.15.102:0/1109822326 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d2c158e20 0x7f7d2c0b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.768+0000 7f7d32e00640 1 --2- 172.21.15.102:0/1109822326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d2c0b1b00 0x7f7d2c0b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.768+0000 7f7d32e00640 1 --2- 172.21.15.102:0/1109822326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d2c070860 0x7f7d2c1f20b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:06.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.768+0000 7f7d32e00640 1 -- 172.21.15.102:0/1109822326 >> 172.21.15.102:0/1109822326 conn(0x7f7d2c06a0b0 msgr2=0x7f7d2c15bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:06.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.768+0000 7f7d32e00640 1 -- 172.21.15.102:0/1109822326 shutdown_connections 2024-03-23T03:48:06.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:06.768+0000 7f7d32e00640 1 -- 172.21.15.102:0/1109822326 wait complete. 2024-03-23T03:48:06.788 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:48:07.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.001+0000 7fca58c00640 1 Processor -- start 2024-03-23T03:48:07.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.001+0000 7fca58c00640 1 -- start start 2024-03-23T03:48:07.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.002+0000 7fca58c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fca54159830 0x7fca540cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.002+0000 7fca58c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca540ca7b0 0x7fca540cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.002+0000 7fca58c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fca540ce930 0x7fca540cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.002+0000 7fca58c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca5406de50 con 0x7fca54159830 2024-03-23T03:48:07.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.002+0000 7fca58c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca540cf300 con 0x7fca540ce930 2024-03-23T03:48:07.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.002+0000 7fca58c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca54159c10 con 0x7fca540ca7b0 2024-03-23T03:48:07.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.003+0000 7fca51e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fca54159830 0x7fca540cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.003+0000 7fca51e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fca54159830 0x7fca540cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41434/0 (socket says 172.21.15.102:41434) 2024-03-23T03:48:07.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.003+0000 7fca51e00640 1 -- 172.21.15.102:0/3638820809 learned_addr learned my addr 172.21.15.102:0/3638820809 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:07.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.003+0000 7fca52800640 1 --2- 172.21.15.102:0/3638820809 >> v2:172.21.15.173:3300/0 conn(0x7fca540ce930 0x7fca540cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.003+0000 7fca49400640 1 --2- 172.21.15.102:0/3638820809 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca540ca7b0 0x7fca540cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.003+0000 7fca51e00640 1 -- 172.21.15.102:0/3638820809 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca540ca7b0 msgr2=0x7fca540cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:07.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.003+0000 7fca51e00640 1 --2- 172.21.15.102:0/3638820809 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca540ca7b0 0x7fca540cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.003+0000 7fca51e00640 1 -- 172.21.15.102:0/3638820809 >> v2:172.21.15.173:3300/0 conn(0x7fca540ce930 msgr2=0x7fca540cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:07.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.003+0000 7fca51e00640 1 --2- 172.21.15.102:0/3638820809 >> v2:172.21.15.173:3300/0 conn(0x7fca540ce930 0x7fca540cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.003+0000 7fca51e00640 1 -- 172.21.15.102:0/3638820809 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca540cd840 con 0x7fca54159830 2024-03-23T03:48:07.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.004+0000 7fca51e00640 1 --2- 172.21.15.102:0/3638820809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca54159830 0x7fca540cc3c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fca540cb030 tx=0x7fca4402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bba7d13f1eac878 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:07.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.005+0000 7fca51400640 1 -- 172.21.15.102:0/3638820809 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca44030860 con 0x7fca54159830 2024-03-23T03:48:07.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.005+0000 7fca51400640 1 -- 172.21.15.102:0/3638820809 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca440309c0 con 0x7fca54159830 2024-03-23T03:48:07.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.005+0000 7fca51400640 1 -- 172.21.15.102:0/3638820809 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca44030d50 con 0x7fca54159830 2024-03-23T03:48:07.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.005+0000 7fca58c00640 1 -- 172.21.15.102:0/3638820809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca54159830 msgr2=0x7fca540cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:07.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.005+0000 7fca58c00640 1 --2- 172.21.15.102:0/3638820809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca54159830 0x7fca540cc3c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fca540cb030 tx=0x7fca4402b9a0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.006+0000 7fca58c00640 1 -- 172.21.15.102:0/3638820809 shutdown_connections 2024-03-23T03:48:07.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.006+0000 7fca58c00640 1 --2- 172.21.15.102:0/3638820809 >> v2:172.21.15.173:3300/0 conn(0x7fca540ce930 0x7fca540cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.006+0000 7fca58c00640 1 --2- 172.21.15.102:0/3638820809 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca540ca7b0 0x7fca540cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.006+0000 7fca58c00640 1 --2- 172.21.15.102:0/3638820809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca54159830 0x7fca540cc3c0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.006+0000 7fca58c00640 1 -- 172.21.15.102:0/3638820809 >> 172.21.15.102:0/3638820809 conn(0x7fca5406a0b0 msgr2=0x7fca5406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:07.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.006+0000 7fca58c00640 1 -- 172.21.15.102:0/3638820809 shutdown_connections 2024-03-23T03:48:07.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.006+0000 7fca58c00640 1 -- 172.21.15.102:0/3638820809 wait complete. 2024-03-23T03:48:07.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.007+0000 7fca58c00640 1 Processor -- start 2024-03-23T03:48:07.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.008+0000 7fca58c00640 1 -- start start 2024-03-23T03:48:07.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.008+0000 7fca58c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca540ca7b0 0x7fca5414f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.008+0000 7fca58c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca540ce930 0x7fca541f3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.008+0000 7fca58c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca54159830 0x7fca540ad730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.008+0000 7fca58c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca541576b0 con 0x7fca54159830 2024-03-23T03:48:07.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.008+0000 7fca58c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5414f7c0 con 0x7fca540ca7b0 2024-03-23T03:48:07.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.008+0000 7fca58c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca541578c0 con 0x7fca540ce930 2024-03-23T03:48:07.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.009+0000 7fca52800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca54159830 0x7fca540ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.009+0000 7fca52800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca54159830 0x7fca540ad730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41442/0 (socket says 172.21.15.102:41442) 2024-03-23T03:48:07.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.009+0000 7fca52800640 1 -- 172.21.15.102:0/1513441913 learned_addr learned my addr 172.21.15.102:0/1513441913 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:07.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.009+0000 7fca49400640 1 --2- 172.21.15.102:0/1513441913 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca540ce930 0x7fca541f3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.009+0000 7fca51e00640 1 --2- 172.21.15.102:0/1513441913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca540ca7b0 0x7fca5414f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.009+0000 7fca52800640 1 -- 172.21.15.102:0/1513441913 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca540ce930 msgr2=0x7fca541f3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:07.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.009+0000 7fca52800640 1 --2- 172.21.15.102:0/1513441913 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca540ce930 0x7fca541f3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.009+0000 7fca52800640 1 -- 172.21.15.102:0/1513441913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca540ca7b0 msgr2=0x7fca5414f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:07.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.009+0000 7fca52800640 1 --2- 172.21.15.102:0/1513441913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca540ca7b0 0x7fca5414f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.009+0000 7fca52800640 1 -- 172.21.15.102:0/1513441913 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca440094d0 con 0x7fca54159830 2024-03-23T03:48:07.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.009+0000 7fca49400640 1 --2- 172.21.15.102:0/1513441913 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca540ce930 0x7fca541f3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:07.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.009+0000 7fca52800640 1 --2- 172.21.15.102:0/1513441913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca54159830 0x7fca540ad730 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fca4c007e40 tx=0x7fca4c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:07.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.010+0000 7fca4aa00640 1 -- 172.21.15.102:0/1513441913 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c014620 con 0x7fca54159830 2024-03-23T03:48:07.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.010+0000 7fca4aa00640 1 -- 172.21.15.102:0/1513441913 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4c00ee90 con 0x7fca54159830 2024-03-23T03:48:07.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.010+0000 7fca4aa00640 1 -- 172.21.15.102:0/1513441913 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4c0149e0 con 0x7fca54159830 2024-03-23T03:48:07.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.010+0000 7fca58c00640 1 -- 172.21.15.102:0/1513441913 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca54157ba0 con 0x7fca54159830 2024-03-23T03:48:07.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.010+0000 7fca58c00640 1 -- 172.21.15.102:0/1513441913 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca54158160 con 0x7fca54159830 2024-03-23T03:48:07.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.011+0000 7fca58c00640 1 -- 172.21.15.102:0/1513441913 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca54158380 con 0x7fca54159830 2024-03-23T03:48:07.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.012+0000 7fca4aa00640 1 -- 172.21.15.102:0/1513441913 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fca4c014b40 con 0x7fca54159830 2024-03-23T03:48:07.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.013+0000 7fca4aa00640 1 --2- 172.21.15.102:0/1513441913 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fca3006ac00 0x7fca3006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.013+0000 7fca4aa00640 1 -- 172.21.15.102:0/1513441913 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fca4c08a0a0 con 0x7fca54159830 2024-03-23T03:48:07.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.013+0000 7fca51e00640 1 --2- 172.21.15.102:0/1513441913 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fca3006ac00 0x7fca3006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.014+0000 7fca51e00640 1 --2- 172.21.15.102:0/1513441913 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fca3006ac00 0x7fca3006d0c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fca44036000 tx=0x7fca4403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:07.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.017+0000 7fca4aa00640 1 -- 172.21.15.102:0/1513441913 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fca4c054e20 con 0x7fca54159830 2024-03-23T03:48:07.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.209+0000 7fca58c00640 1 -- 172.21.15.102:0/1513441913 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fca54073390 con 0x7fca54159830 2024-03-23T03:48:07.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.209+0000 7fca4aa00640 1 -- 172.21.15.102:0/1513441913 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v28) v1 ==== 104+0+11 (secure 0 0 0) 0x7fca4c058440 con 0x7fca54159830 2024-03-23T03:48:07.210 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:48:07.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.212+0000 7fca58c00640 1 -- 172.21.15.102:0/1513441913 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fca3006ac00 msgr2=0x7fca3006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:07.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.212+0000 7fca58c00640 1 --2- 172.21.15.102:0/1513441913 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fca3006ac00 0x7fca3006d0c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fca44036000 tx=0x7fca4403b040 comp rx=0 tx=0).stop 2024-03-23T03:48:07.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.212+0000 7fca58c00640 1 -- 172.21.15.102:0/1513441913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca54159830 msgr2=0x7fca540ad730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:07.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.212+0000 7fca58c00640 1 --2- 172.21.15.102:0/1513441913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca54159830 0x7fca540ad730 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fca4c007e40 tx=0x7fca4c00e4b0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.212+0000 7fca58c00640 1 -- 172.21.15.102:0/1513441913 shutdown_connections 2024-03-23T03:48:07.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.212+0000 7fca58c00640 1 --2- 172.21.15.102:0/1513441913 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fca3006ac00 0x7fca3006d0c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.212+0000 7fca58c00640 1 --2- 172.21.15.102:0/1513441913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca54159830 0x7fca540ad730 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.212+0000 7fca58c00640 1 --2- 172.21.15.102:0/1513441913 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca540ce930 0x7fca541f3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.212+0000 7fca58c00640 1 --2- 172.21.15.102:0/1513441913 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca540ca7b0 0x7fca5414f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.212+0000 7fca58c00640 1 -- 172.21.15.102:0/1513441913 >> 172.21.15.102:0/1513441913 conn(0x7fca5406a0b0 msgr2=0x7fca540b1420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:07.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.212+0000 7fca58c00640 1 -- 172.21.15.102:0/1513441913 shutdown_connections 2024-03-23T03:48:07.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.212+0000 7fca58c00640 1 -- 172.21.15.102:0/1513441913 wait complete. 2024-03-23T03:48:07.231 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:48:07.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.435+0000 7fbdee200640 1 Processor -- start 2024-03-23T03:48:07.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.435+0000 7fbdee200640 1 -- start start 2024-03-23T03:48:07.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.437+0000 7fbdee200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbde8164430 0x7fbde80ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.437+0000 7fbdee200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fbde80b1930 0x7fbde80cc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.437+0000 7fbdee200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde80b3920 con 0x7fbde80d0140 2024-03-23T03:48:07.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.437+0000 7fbdee200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde80b2f40 con 0x7fbde80b1930 2024-03-23T03:48:07.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.437+0000 7fbdee200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde8071b10 con 0x7fbde8164430 2024-03-23T03:48:07.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.438+0000 7fbde6a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbde80d0140 0x7fbde80b1220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45156/0 (socket says 172.21.15.102:45156) 2024-03-23T03:48:07.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.438+0000 7fbde6a00640 1 -- 172.21.15.102:0/4013536612 learned_addr learned my addr 172.21.15.102:0/4013536612 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:07.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.438+0000 7fbde7400640 1 --2- 172.21.15.102:0/4013536612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbde8164430 0x7fbde80ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.438+0000 7fbde7e00640 1 --2- 172.21.15.102:0/4013536612 >> v2:172.21.15.173:3300/0 conn(0x7fbde80b1930 0x7fbde80cc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.438+0000 7fbde6000640 1 -- 172.21.15.102:0/4013536612 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517201474 0 0) 0x7fbde80b3920 con 0x7fbde80d0140 2024-03-23T03:48:07.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.438+0000 7fbde6000640 1 -- 172.21.15.102:0/4013536612 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdcc003520 con 0x7fbde80d0140 2024-03-23T03:48:07.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.438+0000 7fbde6000640 1 -- 172.21.15.102:0/4013536612 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3355651640 0 0) 0x7fbdcc003520 con 0x7fbde80d0140 2024-03-23T03:48:07.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.438+0000 7fbde6000640 1 -- 172.21.15.102:0/4013536612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbde8164430 msgr2=0x7fbde80ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:07.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.438+0000 7fbde6000640 1 --2- 172.21.15.102:0/4013536612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbde8164430 0x7fbde80ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.438+0000 7fbde6000640 1 -- 172.21.15.102:0/4013536612 >> v2:172.21.15.173:3300/0 conn(0x7fbde80b1930 msgr2=0x7fbde80cc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:07.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.438+0000 7fbde6000640 1 --2- 172.21.15.102:0/4013536612 >> v2:172.21.15.173:3300/0 conn(0x7fbde80b1930 0x7fbde80cc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.438+0000 7fbde6000640 1 -- 172.21.15.102:0/4013536612 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde8164810 con 0x7fbde80d0140 2024-03-23T03:48:07.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.438+0000 7fbde6000640 1 -- 172.21.15.102:0/4013536612 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fbdd0001670 con 0x7fbde80d0140 2024-03-23T03:48:07.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.439+0000 7fbde6000640 1 -- 172.21.15.102:0/4013536612 >> v1:172.21.15.102:6789/0 conn(0x7fbde80d0140 legacy=0x7fbde80b1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:07.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.439+0000 7fbde6000640 1 --2- 172.21.15.102:0/4013536612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbde8164430 0x7fbdcc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.439+0000 7fbde6000640 1 --2- 172.21.15.102:0/4013536612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdcc003ef0 0x7fbdcc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.439+0000 7fbde6000640 1 --2- 172.21.15.102:0/4013536612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdcc004d00 0x7fbdcc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.439+0000 7fbde6000640 1 -- 172.21.15.102:0/4013536612 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde80b2f40 con 0x7fbdcc003ef0 2024-03-23T03:48:07.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.439+0000 7fbde6000640 1 -- 172.21.15.102:0/4013536612 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde8071b10 con 0x7fbdcc004d00 2024-03-23T03:48:07.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.439+0000 7fbde6000640 1 -- 172.21.15.102:0/4013536612 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdcc00b9b0 con 0x7fbde8164430 2024-03-23T03:48:07.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.439+0000 7fbde6a00640 1 --2- 172.21.15.102:0/4013536612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdcc003ef0 0x7fbdcc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.439+0000 7fbde7400640 1 --2- 172.21.15.102:0/4013536612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbde8164430 0x7fbdcc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.439+0000 7fbde6a00640 1 -- 172.21.15.102:0/4013536612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbde8164430 msgr2=0x7fbdcc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:07.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.439+0000 7fbde6a00640 1 --2- 172.21.15.102:0/4013536612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbde8164430 0x7fbdcc003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.439+0000 7fbde6a00640 1 -- 172.21.15.102:0/4013536612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdcc004d00 msgr2=0x7fbdcc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:07.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.439+0000 7fbde6a00640 1 --2- 172.21.15.102:0/4013536612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbdcc004d00 0x7fbdcc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.439+0000 7fbde6a00640 1 -- 172.21.15.102:0/4013536612 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdcc003520 con 0x7fbdcc003ef0 2024-03-23T03:48:07.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.440+0000 7fbde6a00640 1 --2- 172.21.15.102:0/4013536612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdcc003ef0 0x7fbdcc0047c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbdd000c290 tx=0x7fbdcc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4300e02550f50505 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:07.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.441+0000 7fbde6000640 1 -- 172.21.15.102:0/4013536612 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd0005ad0 con 0x7fbdcc003ef0 2024-03-23T03:48:07.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.441+0000 7fbde6000640 1 -- 172.21.15.102:0/4013536612 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd0005c30 con 0x7fbdcc003ef0 2024-03-23T03:48:07.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.441+0000 7fbde6000640 1 -- 172.21.15.102:0/4013536612 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd002c8d0 con 0x7fbdcc003ef0 2024-03-23T03:48:07.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.441+0000 7fbdee200640 1 -- 172.21.15.102:0/4013536612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdcc003ef0 msgr2=0x7fbdcc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:07.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.441+0000 7fbdee200640 1 --2- 172.21.15.102:0/4013536612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbdcc003ef0 0x7fbdcc0047c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbdd000c290 tx=0x7fbdcc0057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.441+0000 7fbde7e00640 1 -- 172.21.15.102:0/4013536612 reap_dead start 2024-03-23T03:48:07.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.441+0000 7fbdee200640 1 -- 172.21.15.102:0/4013536612 shutdown_connections 2024-03-23T03:48:07.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.441+0000 7fbdee200640 1 -- 172.21.15.102:0/4013536612 >> 172.21.15.102:0/4013536612 conn(0x7fbde80b4c90 msgr2=0x7fbde80b50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:07.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.441+0000 7fbdee200640 1 -- 172.21.15.102:0/4013536612 shutdown_connections 2024-03-23T03:48:07.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.441+0000 7fbdee200640 1 -- 172.21.15.102:0/4013536612 wait complete. 2024-03-23T03:48:07.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.443+0000 7fbdee200640 1 Processor -- start 2024-03-23T03:48:07.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.443+0000 7fbdee200640 1 -- start start 2024-03-23T03:48:07.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.444+0000 7fbdee200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbdcc003ef0 0x7fbde8146c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.444+0000 7fbdee200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbde814aae0 0x7fbde80cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.444+0000 7fbdee200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbde814af40 0x7fbde815cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.444+0000 7fbdee200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde81485a0 con 0x7fbde814af40 2024-03-23T03:48:07.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.444+0000 7fbdee200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde8147920 con 0x7fbde814aae0 2024-03-23T03:48:07.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.444+0000 7fbdee200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde8147c00 con 0x7fbdcc003ef0 2024-03-23T03:48:07.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.445+0000 7fbde7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbde814af40 0x7fbde815cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.445+0000 7fbde7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbde814af40 0x7fbde815cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41456/0 (socket says 172.21.15.102:41456) 2024-03-23T03:48:07.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.445+0000 7fbde7e00640 1 -- 172.21.15.102:0/1256291030 learned_addr learned my addr 172.21.15.102:0/1256291030 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:07.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.445+0000 7fbde7400640 1 --2- 172.21.15.102:0/1256291030 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbdcc003ef0 0x7fbde8146c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.445+0000 7fbde6a00640 1 --2- 172.21.15.102:0/1256291030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbde814aae0 0x7fbde80cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.445+0000 7fbde7e00640 1 -- 172.21.15.102:0/1256291030 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbdcc003ef0 msgr2=0x7fbde8146c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:07.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.445+0000 7fbde7e00640 1 --2- 172.21.15.102:0/1256291030 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbdcc003ef0 0x7fbde8146c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.445+0000 7fbde7e00640 1 -- 172.21.15.102:0/1256291030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbde814aae0 msgr2=0x7fbde80cdca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:07.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.445+0000 7fbde7e00640 1 --2- 172.21.15.102:0/1256291030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbde814aae0 0x7fbde80cdca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.445+0000 7fbde7e00640 1 -- 172.21.15.102:0/1256291030 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde8147ee0 con 0x7fbde814af40 2024-03-23T03:48:07.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.445+0000 7fbde7e00640 1 --2- 172.21.15.102:0/1256291030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbde814af40 0x7fbde815cf90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbde80716e0 tx=0x7fbde000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:07.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.446+0000 7fbddf400640 1 -- 172.21.15.102:0/1256291030 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde000c680 con 0x7fbde814af40 2024-03-23T03:48:07.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.446+0000 7fbddf400640 1 -- 172.21.15.102:0/1256291030 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde000c7e0 con 0x7fbde814af40 2024-03-23T03:48:07.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.446+0000 7fbddf400640 1 -- 172.21.15.102:0/1256291030 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde000cb90 con 0x7fbde814af40 2024-03-23T03:48:07.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.446+0000 7fbdee200640 1 -- 172.21.15.102:0/1256291030 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde81481c0 con 0x7fbde814af40 2024-03-23T03:48:07.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.446+0000 7fbdee200640 1 -- 172.21.15.102:0/1256291030 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde81471c0 con 0x7fbde814af40 2024-03-23T03:48:07.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.447+0000 7fbdee200640 1 -- 172.21.15.102:0/1256291030 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde81486e0 con 0x7fbde814af40 2024-03-23T03:48:07.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.448+0000 7fbddf400640 1 -- 172.21.15.102:0/1256291030 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbde000ccf0 con 0x7fbde814af40 2024-03-23T03:48:07.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.449+0000 7fbddf400640 1 --2- 172.21.15.102:0/1256291030 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbdc406ac00 0x7fbdc406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.449+0000 7fbddf400640 1 -- 172.21.15.102:0/1256291030 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fbde0089460 con 0x7fbde814af40 2024-03-23T03:48:07.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.449+0000 7fbde7400640 1 --2- 172.21.15.102:0/1256291030 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbdc406ac00 0x7fbdc406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.450+0000 7fbde7400640 1 --2- 172.21.15.102:0/1256291030 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbdc406ac00 0x7fbdc406d0c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8007440 tx=0x7fbdd800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:07.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.453+0000 7fbddf400640 1 -- 172.21.15.102:0/1256291030 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbde81486e0 con 0x7fbde814af40 2024-03-23T03:48:07.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.642+0000 7fbdee200640 1 -- 172.21.15.102:0/1256291030 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbde81486e0 con 0x7fbde814af40 2024-03-23T03:48:07.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.643+0000 7fbddf400640 1 -- 172.21.15.102:0/1256291030 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v28) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbde81486e0 con 0x7fbde814af40 2024-03-23T03:48:07.644 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:48:07.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.646+0000 7fbdee200640 1 -- 172.21.15.102:0/1256291030 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbdc406ac00 msgr2=0x7fbdc406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:07.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.646+0000 7fbdee200640 1 --2- 172.21.15.102:0/1256291030 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbdc406ac00 0x7fbdc406d0c0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8007440 tx=0x7fbdd800b040 comp rx=0 tx=0).stop 2024-03-23T03:48:07.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.646+0000 7fbdee200640 1 -- 172.21.15.102:0/1256291030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbde814af40 msgr2=0x7fbde815cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:07.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.646+0000 7fbdee200640 1 --2- 172.21.15.102:0/1256291030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbde814af40 0x7fbde815cf90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbde80716e0 tx=0x7fbde000ec00 comp rx=0 tx=0).stop 2024-03-23T03:48:07.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.646+0000 7fbdee200640 1 -- 172.21.15.102:0/1256291030 shutdown_connections 2024-03-23T03:48:07.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.646+0000 7fbdee200640 1 --2- 172.21.15.102:0/1256291030 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbdc406ac00 0x7fbdc406d0c0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.646+0000 7fbdee200640 1 --2- 172.21.15.102:0/1256291030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbde814af40 0x7fbde815cf90 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.646+0000 7fbdee200640 1 --2- 172.21.15.102:0/1256291030 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbde814aae0 0x7fbde80cdca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.646+0000 7fbdee200640 1 --2- 172.21.15.102:0/1256291030 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbdcc003ef0 0x7fbde8146c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.646+0000 7fbdee200640 1 -- 172.21.15.102:0/1256291030 >> 172.21.15.102:0/1256291030 conn(0x7fbde80b4c90 msgr2=0x7fbde80afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:07.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.646+0000 7fbdee200640 1 -- 172.21.15.102:0/1256291030 shutdown_connections 2024-03-23T03:48:07.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.646+0000 7fbdee200640 1 -- 172.21.15.102:0/1256291030 wait complete. 2024-03-23T03:48:07.664 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-23T03:48:07.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.870+0000 7f2f9be00640 1 Processor -- start 2024-03-23T03:48:07.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.870+0000 7f2f9be00640 1 -- start start 2024-03-23T03:48:07.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.871+0000 7f2f9be00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2f9c0b26c0 0x7f2f9c15a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.871+0000 7f2f9be00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c0c8c90 con 0x7f2f9c159cd0 2024-03-23T03:48:07.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.871+0000 7f2f9be00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f9c0cc6c0 con 0x7f2f9c0cc200 2024-03-23T03:48:07.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.871+0000 7f2f9be00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c0cc9a0 con 0x7f2f9c0b26c0 2024-03-23T03:48:07.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.872+0000 7f2f9aa00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2f9c159cd0 0x7f2f9c0ad260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45166/0 (socket says 172.21.15.102:45166) 2024-03-23T03:48:07.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.872+0000 7f2f9aa00640 1 -- 172.21.15.102:0/3233167047 learned_addr learned my addr 172.21.15.102:0/3233167047 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:07.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.872+0000 7f2f9a000640 1 -- 172.21.15.102:0/3233167047 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022541856 0 0) 0x7f2f9c0c8c90 con 0x7f2f9c159cd0 2024-03-23T03:48:07.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.872+0000 7f2f9a000640 1 -- 172.21.15.102:0/3233167047 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f80003520 con 0x7f2f9c159cd0 2024-03-23T03:48:07.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.872+0000 7f2f9b400640 1 --2- 172.21.15.102:0/3233167047 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2f9c0b26c0 0x7f2f9c15a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.872+0000 7f2f9a000640 1 -- 172.21.15.102:0/3233167047 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2922260259 0 0) 0x7f2f80003520 con 0x7f2f9c159cd0 2024-03-23T03:48:07.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.872+0000 7f2f9a000640 1 -- 172.21.15.102:0/3233167047 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2f9c0b26c0 msgr2=0x7f2f9c15a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:07.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.872+0000 7f2f9a000640 1 --2- 172.21.15.102:0/3233167047 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2f9c0b26c0 0x7f2f9c15a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.872+0000 7f2f9a000640 1 -- 172.21.15.102:0/3233167047 >> v1:172.21.15.173:6789/0 conn(0x7f2f9c0cc200 legacy=0x7f2f9c0cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:07.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.872+0000 7f2f9a000640 1 -- 172.21.15.102:0/3233167047 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c0ccc80 con 0x7f2f9c159cd0 2024-03-23T03:48:07.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.872+0000 7f2f9a000640 1 -- 172.21.15.102:0/3233167047 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f2f8c001670 con 0x7f2f9c159cd0 2024-03-23T03:48:07.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.872+0000 7f2f9a000640 1 -- 172.21.15.102:0/3233167047 >> v1:172.21.15.102:6789/0 conn(0x7f2f9c159cd0 legacy=0x7f2f9c0ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:07.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.872+0000 7f2f9a000640 1 --2- 172.21.15.102:0/3233167047 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2f800039a0 0x7f2f80003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.872+0000 7f2f9a000640 1 --2- 172.21.15.102:0/3233167047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f800047c0 0x7f2f80004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.872+0000 7f2f9a000640 1 --2- 172.21.15.102:0/3233167047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f80005180 0x7f2f8000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.873+0000 7f2f9a000640 1 -- 172.21.15.102:0/3233167047 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c0cc9a0 con 0x7f2f800047c0 2024-03-23T03:48:07.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.873+0000 7f2f9a000640 1 -- 172.21.15.102:0/3233167047 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8000e290 con 0x7f2f80005180 2024-03-23T03:48:07.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.873+0000 7f2f9a000640 1 -- 172.21.15.102:0/3233167047 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8000e450 con 0x7f2f800039a0 2024-03-23T03:48:07.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.873+0000 7f2f9b400640 1 --2- 172.21.15.102:0/3233167047 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2f9c0b26c0 0x7f2f9c15a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:48:07.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.873+0000 7f2f92000640 1 --2- 172.21.15.102:0/3233167047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f800047c0 0x7f2f80004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.873+0000 7f2f9aa00640 1 --2- 172.21.15.102:0/3233167047 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2f800039a0 0x7f2f80003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.873+0000 7f2f9b400640 1 --2- 172.21.15.102:0/3233167047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f80005180 0x7f2f8000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.873+0000 7f2f92000640 1 -- 172.21.15.102:0/3233167047 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2f800039a0 msgr2=0x7f2f80003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:07.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.873+0000 7f2f92000640 1 --2- 172.21.15.102:0/3233167047 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2f800039a0 0x7f2f80003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.873+0000 7f2f92000640 1 -- 172.21.15.102:0/3233167047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f80005180 msgr2=0x7f2f8000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:07.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.873+0000 7f2f92000640 1 --2- 172.21.15.102:0/3233167047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f80005180 0x7f2f8000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.873+0000 7f2f92000640 1 -- 172.21.15.102:0/3233167047 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f80003520 con 0x7f2f800047c0 2024-03-23T03:48:07.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.873+0000 7f2f9aa00640 1 --2- 172.21.15.102:0/3233167047 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2f800039a0 0x7f2f80003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:48:07.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.874+0000 7f2f92000640 1 --2- 172.21.15.102:0/3233167047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f800047c0 0x7f2f80004c40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2f84001750 tx=0x7f2f800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f917ae231efdf6b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:07.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.875+0000 7f2f9a000640 1 -- 172.21.15.102:0/3233167047 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f840052d0 con 0x7f2f800047c0 2024-03-23T03:48:07.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.875+0000 7f2f9a000640 1 -- 172.21.15.102:0/3233167047 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8402e050 con 0x7f2f800047c0 2024-03-23T03:48:07.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.875+0000 7f2f9a000640 1 -- 172.21.15.102:0/3233167047 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f84003830 con 0x7f2f800047c0 2024-03-23T03:48:07.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.875+0000 7f2f9be00640 1 -- 172.21.15.102:0/3233167047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f800047c0 msgr2=0x7f2f80004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:07.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.875+0000 7f2f9be00640 1 --2- 172.21.15.102:0/3233167047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f800047c0 0x7f2f80004c40 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f2f84001750 tx=0x7f2f800057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.875+0000 7f2f9b400640 1 -- 172.21.15.102:0/3233167047 reap_dead start 2024-03-23T03:48:07.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.875+0000 7f2f9be00640 1 -- 172.21.15.102:0/3233167047 shutdown_connections 2024-03-23T03:48:07.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.875+0000 7f2f9be00640 1 -- 172.21.15.102:0/3233167047 >> 172.21.15.102:0/3233167047 conn(0x7f2f9c0b01b0 msgr2=0x7f2f9c0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:07.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.875+0000 7f2f9be00640 1 -- 172.21.15.102:0/3233167047 shutdown_connections 2024-03-23T03:48:07.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.875+0000 7f2f9be00640 1 -- 172.21.15.102:0/3233167047 wait complete. 2024-03-23T03:48:07.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.877+0000 7f2f9be00640 1 Processor -- start 2024-03-23T03:48:07.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.877+0000 7f2f9be00640 1 -- start start 2024-03-23T03:48:07.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.878+0000 7f2f9be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f800047c0 0x7f2f9c0983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.878+0000 7f2f9be00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2f9c070f30 0x7f2f9c0af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.878+0000 7f2f9be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c1f48d0 0x7f2f9c1f4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.878+0000 7f2f9be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c099f00 con 0x7f2f9c1f48d0 2024-03-23T03:48:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.878+0000 7f2f9be00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c1f52c0 con 0x7f2f800047c0 2024-03-23T03:48:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.878+0000 7f2f9be00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c098900 con 0x7f2f9c070f30 2024-03-23T03:48:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.878+0000 7f2f9b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c1f48d0 0x7f2f9c1f4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.879+0000 7f2f9b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c1f48d0 0x7f2f9c1f4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41480/0 (socket says 172.21.15.102:41480) 2024-03-23T03:48:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.879+0000 7f2f9b400640 1 -- 172.21.15.102:0/2558802512 learned_addr learned my addr 172.21.15.102:0/2558802512 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.879+0000 7f2f92000640 1 --2- 172.21.15.102:0/2558802512 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2f9c070f30 0x7f2f9c0af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.879+0000 7f2f9aa00640 1 --2- 172.21.15.102:0/2558802512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f800047c0 0x7f2f9c0983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.879+0000 7f2f9b400640 1 -- 172.21.15.102:0/2558802512 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2f9c070f30 msgr2=0x7f2f9c0af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.879+0000 7f2f9b400640 1 --2- 172.21.15.102:0/2558802512 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2f9c070f30 0x7f2f9c0af1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.879+0000 7f2f9b400640 1 -- 172.21.15.102:0/2558802512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f800047c0 msgr2=0x7f2f9c0983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.879+0000 7f2f9b400640 1 --2- 172.21.15.102:0/2558802512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f800047c0 0x7f2f9c0983c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.879+0000 7f2f9b400640 1 -- 172.21.15.102:0/2558802512 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c0713c0 con 0x7f2f9c1f48d0 2024-03-23T03:48:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.879+0000 7f2f9b400640 1 --2- 172.21.15.102:0/2558802512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c1f48d0 0x7f2f9c1f4d80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0b2b50 tx=0x7f2f94007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.879+0000 7f2f93400640 1 -- 172.21.15.102:0/2558802512 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f940048a0 con 0x7f2f9c1f48d0 2024-03-23T03:48:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.880+0000 7f2f93400640 1 -- 172.21.15.102:0/2558802512 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9400f050 con 0x7f2f9c1f48d0 2024-03-23T03:48:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.880+0000 7f2f9be00640 1 -- 172.21.15.102:0/2558802512 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f9c0716a0 con 0x7f2f9c1f48d0 2024-03-23T03:48:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.880+0000 7f2f93400640 1 -- 172.21.15.102:0/2558802512 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f94014630 con 0x7f2f9c1f48d0 2024-03-23T03:48:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.880+0000 7f2f9be00640 1 -- 172.21.15.102:0/2558802512 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f9c071c60 con 0x7f2f9c1f48d0 2024-03-23T03:48:07.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.881+0000 7f2f9be00640 1 -- 172.21.15.102:0/2558802512 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f9c09a040 con 0x7f2f9c1f48d0 2024-03-23T03:48:07.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.882+0000 7f2f93400640 1 -- 172.21.15.102:0/2558802512 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2f94014790 con 0x7f2f9c1f48d0 2024-03-23T03:48:07.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.882+0000 7f2f93400640 1 --2- 172.21.15.102:0/2558802512 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2f7806ac00 0x7f2f7806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:07.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.883+0000 7f2f93400640 1 -- 172.21.15.102:0/2558802512 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2f94089df0 con 0x7f2f9c1f48d0 2024-03-23T03:48:07.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.883+0000 7f2f9aa00640 1 --2- 172.21.15.102:0/2558802512 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2f7806ac00 0x7f2f7806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:07.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.883+0000 7f2f9aa00640 1 --2- 172.21.15.102:0/2558802512 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2f7806ac00 0x7f2f7806d0c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c001c30 tx=0x7f2f8c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:07.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:07.887+0000 7f2f93400640 1 -- 172.21.15.102:0/2558802512 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2f9c09a040 con 0x7f2f9c1f48d0 2024-03-23T03:48:08.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.065+0000 7f2f9be00640 1 -- 172.21.15.102:0/2558802512 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2f9c09a040 con 0x7f2f9c1f48d0 2024-03-23T03:48:08.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.065+0000 7f2f93400640 1 -- 172.21.15.102:0/2558802512 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v28) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2f9c09a040 con 0x7f2f9c1f48d0 2024-03-23T03:48:08.066 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:48:08.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.068+0000 7f2f9be00640 1 -- 172.21.15.102:0/2558802512 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2f7806ac00 msgr2=0x7f2f7806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:08.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.068+0000 7f2f9be00640 1 --2- 172.21.15.102:0/2558802512 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2f7806ac00 0x7f2f7806d0c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c001c30 tx=0x7f2f8c009040 comp rx=0 tx=0).stop 2024-03-23T03:48:08.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.068+0000 7f2f9be00640 1 -- 172.21.15.102:0/2558802512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c1f48d0 msgr2=0x7f2f9c1f4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:08.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.068+0000 7f2f9be00640 1 --2- 172.21.15.102:0/2558802512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c1f48d0 0x7f2f9c1f4d80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0b2b50 tx=0x7f2f94007700 comp rx=0 tx=0).stop 2024-03-23T03:48:08.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.068+0000 7f2f9be00640 1 -- 172.21.15.102:0/2558802512 shutdown_connections 2024-03-23T03:48:08.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.068+0000 7f2f9be00640 1 --2- 172.21.15.102:0/2558802512 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2f7806ac00 0x7f2f7806d0c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.068+0000 7f2f9be00640 1 --2- 172.21.15.102:0/2558802512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2f9c1f48d0 0x7f2f9c1f4d80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.068+0000 7f2f9be00640 1 --2- 172.21.15.102:0/2558802512 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2f9c070f30 0x7f2f9c0af1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.068+0000 7f2f9be00640 1 --2- 172.21.15.102:0/2558802512 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2f800047c0 0x7f2f9c0983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.068+0000 7f2f9be00640 1 -- 172.21.15.102:0/2558802512 >> 172.21.15.102:0/2558802512 conn(0x7f2f9c0b01b0 msgr2=0x7f2f9c0ce200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:08.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.068+0000 7f2f9be00640 1 -- 172.21.15.102:0/2558802512 shutdown_connections 2024-03-23T03:48:08.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.068+0000 7f2f9be00640 1 -- 172.21.15.102:0/2558802512 wait complete. 2024-03-23T03:48:08.087 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:48:08.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.288+0000 7f13b3e00640 1 Processor -- start 2024-03-23T03:48:08.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.288+0000 7f13b3e00640 1 -- start start 2024-03-23T03:48:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.289+0000 7f13b3e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13b41571d0 0x7f13b40b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.289+0000 7f13b3e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f13b40cdc00 0x7f13b415aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.289+0000 7f13b3e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f13b415afe0 0x7f13b40cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.289+0000 7f13b3e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b41575b0 con 0x7f13b40cdc00 2024-03-23T03:48:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.289+0000 7f13b3e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13b4159f20 con 0x7f13b415afe0 2024-03-23T03:48:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.289+0000 7f13b3e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b415a200 con 0x7f13b41571d0 2024-03-23T03:48:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.289+0000 7f13abe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f13b40cdc00 0x7f13b415aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:08.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.290+0000 7f13abe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f13b40cdc00 0x7f13b415aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41494/0 (socket says 172.21.15.102:41494) 2024-03-23T03:48:08.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.290+0000 7f13abe00640 1 -- 172.21.15.102:0/1718513687 learned_addr learned my addr 172.21.15.102:0/1718513687 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:08.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.290+0000 7f13b2a00640 1 --2- 172.21.15.102:0/1718513687 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13b41571d0 0x7f13b40b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:08.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.290+0000 7f13abe00640 1 -- 172.21.15.102:0/1718513687 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13b41571d0 msgr2=0x7f13b40b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:08.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.290+0000 7f13abe00640 1 --2- 172.21.15.102:0/1718513687 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13b41571d0 0x7f13b40b35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.290+0000 7f13abe00640 1 -- 172.21.15.102:0/1718513687 >> v2:172.21.15.173:3300/0 conn(0x7f13b415afe0 msgr2=0x7f13b40cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:08.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.290+0000 7f13b3400640 1 --2- 172.21.15.102:0/1718513687 >> v2:172.21.15.173:3300/0 conn(0x7f13b415afe0 0x7f13b40cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:08.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.290+0000 7f13abe00640 1 --2- 172.21.15.102:0/1718513687 >> v2:172.21.15.173:3300/0 conn(0x7f13b415afe0 0x7f13b40cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.290+0000 7f13abe00640 1 -- 172.21.15.102:0/1718513687 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b415a4e0 con 0x7f13b40cdc00 2024-03-23T03:48:08.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.290+0000 7f13b3400640 1 --2- 172.21.15.102:0/1718513687 >> v2:172.21.15.173:3300/0 conn(0x7f13b415afe0 0x7f13b40cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:08.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.291+0000 7f13abe00640 1 --2- 172.21.15.102:0/1718513687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b40cdc00 0x7f13b415aaa0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f13b40adf30 tx=0x7f139c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1017c5a151011758 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:08.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.291+0000 7f13b2000640 1 -- 172.21.15.102:0/1718513687 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c031870 con 0x7f13b40cdc00 2024-03-23T03:48:08.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.291+0000 7f13b2000640 1 -- 172.21.15.102:0/1718513687 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139c0319d0 con 0x7f13b40cdc00 2024-03-23T03:48:08.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.292+0000 7f13b2000640 1 -- 172.21.15.102:0/1718513687 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c031d60 con 0x7f13b40cdc00 2024-03-23T03:48:08.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.292+0000 7f13b3e00640 1 -- 172.21.15.102:0/1718513687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b40cdc00 msgr2=0x7f13b415aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:08.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.292+0000 7f13b3e00640 1 --2- 172.21.15.102:0/1718513687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b40cdc00 0x7f13b415aaa0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f13b40adf30 tx=0x7f139c02b910 comp rx=0 tx=0).stop 2024-03-23T03:48:08.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.292+0000 7f13b3e00640 1 -- 172.21.15.102:0/1718513687 shutdown_connections 2024-03-23T03:48:08.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.292+0000 7f13b3e00640 1 --2- 172.21.15.102:0/1718513687 >> v2:172.21.15.173:3300/0 conn(0x7f13b415afe0 0x7f13b40cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.292+0000 7f13b3e00640 1 --2- 172.21.15.102:0/1718513687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b40cdc00 0x7f13b415aaa0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.292+0000 7f13b3e00640 1 --2- 172.21.15.102:0/1718513687 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13b41571d0 0x7f13b40b35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.292+0000 7f13b3e00640 1 -- 172.21.15.102:0/1718513687 >> 172.21.15.102:0/1718513687 conn(0x7f13b4065df0 msgr2=0x7f13b4069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:08.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.292+0000 7f13b3e00640 1 -- 172.21.15.102:0/1718513687 shutdown_connections 2024-03-23T03:48:08.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.292+0000 7f13b3e00640 1 -- 172.21.15.102:0/1718513687 wait complete. 2024-03-23T03:48:08.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.294+0000 7f13b3e00640 1 Processor -- start 2024-03-23T03:48:08.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.294+0000 7f13b3e00640 1 -- start start 2024-03-23T03:48:08.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.295+0000 7f13b3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b40cdc00 0x7f13b4158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:08.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.295+0000 7f13b3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13b41571d0 0x7f13b413f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:08.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.295+0000 7f13b3e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13b415afe0 0x7f13b415c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:08.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.295+0000 7f13b3e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b40ab770 con 0x7f13b40cdc00 2024-03-23T03:48:08.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.295+0000 7f13b3e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13b40ac920 con 0x7f13b41571d0 2024-03-23T03:48:08.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.295+0000 7f13b3e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b40acc00 con 0x7f13b415afe0 2024-03-23T03:48:08.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.295+0000 7f13b2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b40cdc00 0x7f13b4158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:08.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.295+0000 7f13b2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b40cdc00 0x7f13b4158990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41500/0 (socket says 172.21.15.102:41500) 2024-03-23T03:48:08.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.295+0000 7f13b2a00640 1 -- 172.21.15.102:0/2972510096 learned_addr learned my addr 172.21.15.102:0/2972510096 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:08.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.296+0000 7f13b3400640 1 --2- 172.21.15.102:0/2972510096 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13b415afe0 0x7f13b415c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:08.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.296+0000 7f13b2a00640 1 -- 172.21.15.102:0/2972510096 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13b415afe0 msgr2=0x7f13b415c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:08.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.296+0000 7f13b2a00640 1 --2- 172.21.15.102:0/2972510096 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13b415afe0 0x7f13b415c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.296+0000 7f13abe00640 1 --2- 172.21.15.102:0/2972510096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13b41571d0 0x7f13b413f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:08.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.296+0000 7f13b2a00640 1 -- 172.21.15.102:0/2972510096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13b41571d0 msgr2=0x7f13b413f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:08.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.296+0000 7f13b2a00640 1 --2- 172.21.15.102:0/2972510096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13b41571d0 0x7f13b413f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.296+0000 7f13b2a00640 1 -- 172.21.15.102:0/2972510096 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c0094d0 con 0x7f13b40cdc00 2024-03-23T03:48:08.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.296+0000 7f13b2a00640 1 --2- 172.21.15.102:0/2972510096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b40cdc00 0x7f13b4158990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f13a400d800 tx=0x7f13a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:08.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.296+0000 7f13ab400640 1 -- 172.21.15.102:0/2972510096 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a40096a0 con 0x7f13b40cdc00 2024-03-23T03:48:08.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.296+0000 7f13ab400640 1 -- 172.21.15.102:0/2972510096 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a4009800 con 0x7f13b40cdc00 2024-03-23T03:48:08.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.296+0000 7f13b3e00640 1 -- 172.21.15.102:0/2972510096 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13b40acee0 con 0x7f13b40cdc00 2024-03-23T03:48:08.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.296+0000 7f13ab400640 1 -- 172.21.15.102:0/2972510096 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a4009b90 con 0x7f13b40cdc00 2024-03-23T03:48:08.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.297+0000 7f13b3e00640 1 -- 172.21.15.102:0/2972510096 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13b4158ed0 con 0x7f13b40cdc00 2024-03-23T03:48:08.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.298+0000 7f13b3e00640 1 -- 172.21.15.102:0/2972510096 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13b40b2720 con 0x7f13b40cdc00 2024-03-23T03:48:08.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.299+0000 7f13ab400640 1 -- 172.21.15.102:0/2972510096 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f13a4009cf0 con 0x7f13b40cdc00 2024-03-23T03:48:08.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.300+0000 7f13ab400640 1 --2- 172.21.15.102:0/2972510096 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f139006ac00 0x7f139006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:08.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.300+0000 7f13ab400640 1 -- 172.21.15.102:0/2972510096 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f13a4089c20 con 0x7f13b40cdc00 2024-03-23T03:48:08.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.300+0000 7f13abe00640 1 --2- 172.21.15.102:0/2972510096 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f139006ac00 0x7f139006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:08.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.301+0000 7f13abe00640 1 --2- 172.21.15.102:0/2972510096 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f139006ac00 0x7f139006d0c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f139c036000 tx=0x7f139c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:08.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.304+0000 7f13ab400640 1 -- 172.21.15.102:0/2972510096 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f13a40549a0 con 0x7f13b40cdc00 2024-03-23T03:48:08.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.533+0000 7f13b3e00640 1 -- 172.21.15.102:0/2972510096 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13b404e030 con 0x7f13b40cdc00 2024-03-23T03:48:08.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.534+0000 7f13ab400640 1 -- 172.21.15.102:0/2972510096 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4739 (secure 0 0 0) 0x7f13a400b440 con 0x7f13b40cdc00 2024-03-23T03:48:08.535 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:08.535 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/891818172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":891818172},{"type":"v1","addr":"172.21.15.102:6836","nonce":891818172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2404489233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2404489233},{"type":"v1","addr":"172.21.15.102:6837","nonce":2404489233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/1839967745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":1839967745},{"type":"v1","addr":"172.21.15.173:6835","nonce":1839967745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:46:53.912287+0000","modified":"2024-03-23T03:47:14.818670+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6834/1462637509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1462637509},{"type":"v1","addr":"172.21.15.173:6834","nonce":1462637509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-23T03:48:08.535 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 8 2024-03-23T03:48:08.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.537+0000 7f13b3e00640 1 -- 172.21.15.102:0/2972510096 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f139006ac00 msgr2=0x7f139006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:08.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.537+0000 7f13b3e00640 1 --2- 172.21.15.102:0/2972510096 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f139006ac00 0x7f139006d0c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f139c036000 tx=0x7f139c03c040 comp rx=0 tx=0).stop 2024-03-23T03:48:08.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.537+0000 7f13b3e00640 1 -- 172.21.15.102:0/2972510096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b40cdc00 msgr2=0x7f13b4158990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:08.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.537+0000 7f13b3e00640 1 --2- 172.21.15.102:0/2972510096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b40cdc00 0x7f13b4158990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f13a400d800 tx=0x7f13a400dcc0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.537+0000 7f13b3e00640 1 -- 172.21.15.102:0/2972510096 shutdown_connections 2024-03-23T03:48:08.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.537+0000 7f13b3e00640 1 --2- 172.21.15.102:0/2972510096 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f139006ac00 0x7f139006d0c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.537+0000 7f13b3e00640 1 --2- 172.21.15.102:0/2972510096 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13b415afe0 0x7f13b415c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.537+0000 7f13b3e00640 1 --2- 172.21.15.102:0/2972510096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13b41571d0 0x7f13b413f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.537+0000 7f13b3e00640 1 --2- 172.21.15.102:0/2972510096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13b40cdc00 0x7f13b4158990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.537+0000 7f13b3e00640 1 -- 172.21.15.102:0/2972510096 >> 172.21.15.102:0/2972510096 conn(0x7f13b4065df0 msgr2=0x7f13b40cd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:08.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.538+0000 7f13b3e00640 1 -- 172.21.15.102:0/2972510096 shutdown_connections 2024-03-23T03:48:08.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.538+0000 7f13b3e00640 1 -- 172.21.15.102:0/2972510096 wait complete. 2024-03-23T03:48:08.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:48:08.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.757+0000 7fd84de00640 1 Processor -- start 2024-03-23T03:48:08.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.757+0000 7fd84de00640 1 -- start start 2024-03-23T03:48:08.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.759+0000 7fd84de00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd8480adbc0 0x7fd8480b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:08.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.759+0000 7fd84de00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd8480cbab0 0x7fd8480b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:08.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.759+0000 7fd84de00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd84806d390 0x7fd84806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:08.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.759+0000 7fd84de00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84806dd60 con 0x7fd8480adbc0 2024-03-23T03:48:08.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.759+0000 7fd84de00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8480b16a0 con 0x7fd84806d390 2024-03-23T03:48:08.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.759+0000 7fd84de00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8480b4700 con 0x7fd8480cbab0 2024-03-23T03:48:08.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.759+0000 7fd846a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd8480adbc0 0x7fd8480b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:08.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.759+0000 7fd846a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd8480adbc0 0x7fd8480b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41510/0 (socket says 172.21.15.102:41510) 2024-03-23T03:48:08.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.759+0000 7fd846a00640 1 -- 172.21.15.102:0/2830430461 learned_addr learned my addr 172.21.15.102:0/2830430461 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:08.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.759+0000 7fd846a00640 1 -- 172.21.15.102:0/2830430461 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd8480cbab0 msgr2=0x7fd8480b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:08.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.759+0000 7fd846a00640 1 --2- 172.21.15.102:0/2830430461 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd8480cbab0 0x7fd8480b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.759+0000 7fd846a00640 1 -- 172.21.15.102:0/2830430461 >> v2:172.21.15.173:3300/0 conn(0x7fd84806d390 msgr2=0x7fd84806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:08.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.759+0000 7fd846a00640 1 --2- 172.21.15.102:0/2830430461 >> v2:172.21.15.173:3300/0 conn(0x7fd84806d390 0x7fd84806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.759+0000 7fd846a00640 1 -- 172.21.15.102:0/2830430461 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8480cbfb0 con 0x7fd8480adbc0 2024-03-23T03:48:08.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.760+0000 7fd846a00640 1 --2- 172.21.15.102:0/2830430461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8480adbc0 0x7fd8480b41c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd8480cc650 tx=0x7fd83802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebe0468bf32c87 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:08.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.761+0000 7fd846000640 1 -- 172.21.15.102:0/2830430461 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8380317e0 con 0x7fd8480adbc0 2024-03-23T03:48:08.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.761+0000 7fd846000640 1 -- 172.21.15.102:0/2830430461 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd838031940 con 0x7fd8480adbc0 2024-03-23T03:48:08.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.761+0000 7fd846000640 1 -- 172.21.15.102:0/2830430461 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838031cd0 con 0x7fd8480adbc0 2024-03-23T03:48:08.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.761+0000 7fd84de00640 1 -- 172.21.15.102:0/2830430461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8480adbc0 msgr2=0x7fd8480b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:08.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.761+0000 7fd84de00640 1 --2- 172.21.15.102:0/2830430461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8480adbc0 0x7fd8480b41c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd8480cc650 tx=0x7fd83802b880 comp rx=0 tx=0).stop 2024-03-23T03:48:08.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.761+0000 7fd84de00640 1 -- 172.21.15.102:0/2830430461 shutdown_connections 2024-03-23T03:48:08.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.761+0000 7fd84de00640 1 --2- 172.21.15.102:0/2830430461 >> v2:172.21.15.173:3300/0 conn(0x7fd84806d390 0x7fd84806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.761+0000 7fd84de00640 1 --2- 172.21.15.102:0/2830430461 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd8480cbab0 0x7fd8480b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.761+0000 7fd84de00640 1 --2- 172.21.15.102:0/2830430461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8480adbc0 0x7fd8480b41c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.761+0000 7fd84de00640 1 -- 172.21.15.102:0/2830430461 >> 172.21.15.102:0/2830430461 conn(0x7fd84806a3d0 msgr2=0x7fd8480b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:08.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.762+0000 7fd84de00640 1 -- 172.21.15.102:0/2830430461 shutdown_connections 2024-03-23T03:48:08.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.762+0000 7fd84de00640 1 -- 172.21.15.102:0/2830430461 wait complete. 2024-03-23T03:48:08.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.763+0000 7fd84de00640 1 Processor -- start 2024-03-23T03:48:08.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.763+0000 7fd84de00640 1 -- start start 2024-03-23T03:48:08.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.764+0000 7fd84de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd84806d390 0x7fd848158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:08.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.764+0000 7fd84de00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd8480adbc0 0x7fd848140920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:08.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.764+0000 7fd84de00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8480cbab0 0x7fd848140e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:08.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.764+0000 7fd84de00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8481f2de0 con 0x7fd84806d390 2024-03-23T03:48:08.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.764+0000 7fd84de00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8481f30c0 con 0x7fd8480cbab0 2024-03-23T03:48:08.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.764+0000 7fd84de00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8481f33a0 con 0x7fd8480adbc0 2024-03-23T03:48:08.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.765+0000 7fd846a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd84806d390 0x7fd848158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:08.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.765+0000 7fd846a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd84806d390 0x7fd848158b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41516/0 (socket says 172.21.15.102:41516) 2024-03-23T03:48:08.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.765+0000 7fd846a00640 1 -- 172.21.15.102:0/115678076 learned_addr learned my addr 172.21.15.102:0/115678076 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:08.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.765+0000 7fd846a00640 1 -- 172.21.15.102:0/115678076 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd8480adbc0 msgr2=0x7fd848140920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:08.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.765+0000 7fd846a00640 1 --2- 172.21.15.102:0/115678076 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd8480adbc0 0x7fd848140920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.765+0000 7fd846a00640 1 -- 172.21.15.102:0/115678076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8480cbab0 msgr2=0x7fd848140e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:08.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.765+0000 7fd846a00640 1 --2- 172.21.15.102:0/115678076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8480cbab0 0x7fd848140e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.765+0000 7fd846a00640 1 -- 172.21.15.102:0/115678076 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8380094d0 con 0x7fd84806d390 2024-03-23T03:48:08.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.765+0000 7fd846a00640 1 --2- 172.21.15.102:0/115678076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd84806d390 0x7fd848158b80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd83802b8c0 tx=0x7fd8380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:08.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.765+0000 7fd83fe00640 1 -- 172.21.15.102:0/115678076 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838032940 con 0x7fd84806d390 2024-03-23T03:48:08.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.766+0000 7fd83fe00640 1 -- 172.21.15.102:0/115678076 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd838032aa0 con 0x7fd84806d390 2024-03-23T03:48:08.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.766+0000 7fd83fe00640 1 -- 172.21.15.102:0/115678076 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838032e50 con 0x7fd84806d390 2024-03-23T03:48:08.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.766+0000 7fd84de00640 1 -- 172.21.15.102:0/115678076 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8481f3680 con 0x7fd84806d390 2024-03-23T03:48:08.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.766+0000 7fd84de00640 1 -- 172.21.15.102:0/115678076 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8481590c0 con 0x7fd84806d390 2024-03-23T03:48:08.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.767+0000 7fd84de00640 1 -- 172.21.15.102:0/115678076 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd848159680 con 0x7fd84806d390 2024-03-23T03:48:08.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.768+0000 7fd83fe00640 1 -- 172.21.15.102:0/115678076 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd83804e450 con 0x7fd84806d390 2024-03-23T03:48:08.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.768+0000 7fd83fe00640 1 --2- 172.21.15.102:0/115678076 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd82406ac00 0x7fd82406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:08.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.769+0000 7fd83fe00640 1 -- 172.21.15.102:0/115678076 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd8380ab960 con 0x7fd84806d390 2024-03-23T03:48:08.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.769+0000 7fd83e000640 1 --2- 172.21.15.102:0/115678076 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd82406ac00 0x7fd82406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:08.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.770+0000 7fd83e000640 1 --2- 172.21.15.102:0/115678076 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd82406ac00 0x7fd82406d0c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd8480b1100 tx=0x7fd830009250 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:08.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.773+0000 7fd83fe00640 1 -- 172.21.15.102:0/115678076 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd838076780 con 0x7fd84806d390 2024-03-23T03:48:08.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.964+0000 7fd84de00640 1 -- 172.21.15.102:0/115678076 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd84806ac20 con 0x7fd84806d390 2024-03-23T03:48:08.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.967+0000 7fd83fe00640 1 -- 172.21.15.102:0/115678076 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18260 (secure 0 0 0) 0x7fd838079da0 con 0x7fd84806d390 2024-03-23T03:48:08.968 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:08.968 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":28,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:47:41.647786+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:48:08.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.970+0000 7fd84de00640 1 -- 172.21.15.102:0/115678076 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd82406ac00 msgr2=0x7fd82406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:08.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.970+0000 7fd84de00640 1 --2- 172.21.15.102:0/115678076 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd82406ac00 0x7fd82406d0c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd8480b1100 tx=0x7fd830009250 comp rx=0 tx=0).stop 2024-03-23T03:48:08.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.970+0000 7fd84de00640 1 -- 172.21.15.102:0/115678076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd84806d390 msgr2=0x7fd848158b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:08.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.970+0000 7fd84de00640 1 --2- 172.21.15.102:0/115678076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd84806d390 0x7fd848158b80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd83802b8c0 tx=0x7fd8380046c0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.970+0000 7fd84de00640 1 -- 172.21.15.102:0/115678076 shutdown_connections 2024-03-23T03:48:08.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.970+0000 7fd84de00640 1 --2- 172.21.15.102:0/115678076 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd82406ac00 0x7fd82406d0c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.970+0000 7fd84de00640 1 --2- 172.21.15.102:0/115678076 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8480cbab0 0x7fd848140e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.970+0000 7fd84de00640 1 --2- 172.21.15.102:0/115678076 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd8480adbc0 0x7fd848140920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.970+0000 7fd84de00640 1 --2- 172.21.15.102:0/115678076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd84806d390 0x7fd848158b80 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:08.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.970+0000 7fd84de00640 1 -- 172.21.15.102:0/115678076 >> 172.21.15.102:0/115678076 conn(0x7fd84806a3d0 msgr2=0x7fd8480ab8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:08.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.970+0000 7fd84de00640 1 -- 172.21.15.102:0/115678076 shutdown_connections 2024-03-23T03:48:08.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:08.970+0000 7fd84de00640 1 -- 172.21.15.102:0/115678076 wait complete. 2024-03-23T03:48:08.987 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T03:48:08.987 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:48:09.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.173+0000 7ffb25e00640 1 Processor -- start 2024-03-23T03:48:09.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.173+0000 7ffb25e00640 1 -- start start 2024-03-23T03:48:09.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.174+0000 7ffb25e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ffb200b39b0 0x7ffb200b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:09.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.174+0000 7ffb25e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2006c6f0 con 0x7ffb2015c400 2024-03-23T03:48:09.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.174+0000 7ffb25e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2006c9d0 con 0x7ffb2015b560 2024-03-23T03:48:09.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.174+0000 7ffb25e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb200b4370 con 0x7ffb200b39b0 2024-03-23T03:48:09.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.175+0000 7ffb1f400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ffb2015c400 0x7ffb2015c830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45180/0 (socket says 172.21.15.102:45180) 2024-03-23T03:48:09.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.175+0000 7ffb1f400640 1 -- 172.21.15.102:0/3101133327 learned_addr learned my addr 172.21.15.102:0/3101133327 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:09.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.175+0000 7ffb16000640 1 --2- 172.21.15.102:0/3101133327 >> v2:172.21.15.173:3300/0 conn(0x7ffb200b39b0 0x7ffb200b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:09.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.175+0000 7ffb1e000640 1 -- 172.21.15.102:0/3101133327 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527300409 0 0) 0x7ffb2006c6f0 con 0x7ffb2015c400 2024-03-23T03:48:09.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.175+0000 7ffb1e000640 1 -- 172.21.15.102:0/3101133327 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb04003520 con 0x7ffb2015c400 2024-03-23T03:48:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.176+0000 7ffb1e000640 1 -- 172.21.15.102:0/3101133327 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1829106179 0 0) 0x7ffb04003520 con 0x7ffb2015c400 2024-03-23T03:48:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.176+0000 7ffb1e000640 1 -- 172.21.15.102:0/3101133327 >> v2:172.21.15.173:3300/0 conn(0x7ffb200b39b0 msgr2=0x7ffb200b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.176+0000 7ffb1e000640 1 --2- 172.21.15.102:0/3101133327 >> v2:172.21.15.173:3300/0 conn(0x7ffb200b39b0 0x7ffb200b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.176+0000 7ffb1e000640 1 -- 172.21.15.102:0/3101133327 >> v1:172.21.15.173:6789/0 conn(0x7ffb2015b560 legacy=0x7ffb2015b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.176+0000 7ffb1e000640 1 -- 172.21.15.102:0/3101133327 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2006ccb0 con 0x7ffb2015c400 2024-03-23T03:48:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.176+0000 7ffb1e000640 1 -- 172.21.15.102:0/3101133327 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7ffb180019f0 con 0x7ffb2015c400 2024-03-23T03:48:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.176+0000 7ffb1e000640 1 -- 172.21.15.102:0/3101133327 >> v1:172.21.15.102:6789/0 conn(0x7ffb2015c400 legacy=0x7ffb2015c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.176+0000 7ffb1e000640 1 --2- 172.21.15.102:0/3101133327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb040039c0 0x7ffb04003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.176+0000 7ffb1e000640 1 --2- 172.21.15.102:0/3101133327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb040047c0 0x7ffb04004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.176+0000 7ffb1e000640 1 --2- 172.21.15.102:0/3101133327 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffb04005180 0x7ffb0400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.176+0000 7ffb1e000640 1 -- 172.21.15.102:0/3101133327 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb200b4370 con 0x7ffb040039c0 2024-03-23T03:48:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.176+0000 7ffb1e000640 1 -- 172.21.15.102:0/3101133327 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0400c080 con 0x7ffb040047c0 2024-03-23T03:48:09.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.176+0000 7ffb1e000640 1 -- 172.21.15.102:0/3101133327 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0400c240 con 0x7ffb04005180 2024-03-23T03:48:09.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.176+0000 7ffb1ea00640 1 --2- 172.21.15.102:0/3101133327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb040039c0 0x7ffb04003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:09.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.176+0000 7ffb16000640 1 --2- 172.21.15.102:0/3101133327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb040047c0 0x7ffb04004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:09.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.177+0000 7ffb1f400640 1 --2- 172.21.15.102:0/3101133327 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffb04005180 0x7ffb0400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:09.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.177+0000 7ffb1ea00640 1 -- 172.21.15.102:0/3101133327 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffb04005180 msgr2=0x7ffb0400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:09.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.177+0000 7ffb1ea00640 1 --2- 172.21.15.102:0/3101133327 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffb04005180 0x7ffb0400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.177+0000 7ffb1ea00640 1 -- 172.21.15.102:0/3101133327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb040047c0 msgr2=0x7ffb04004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:09.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.177+0000 7ffb1ea00640 1 --2- 172.21.15.102:0/3101133327 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb040047c0 0x7ffb04004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.177+0000 7ffb1ea00640 1 -- 172.21.15.102:0/3101133327 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb04003520 con 0x7ffb040039c0 2024-03-23T03:48:09.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.177+0000 7ffb1f400640 1 -- 172.21.15.102:0/3101133327 reap_dead start 2024-03-23T03:48:09.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.178+0000 7ffb1ea00640 1 --2- 172.21.15.102:0/3101133327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb040039c0 0x7ffb04003de0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ffb10001750 tx=0x7ffb040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58671c3ed143c87d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:09.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.178+0000 7ffb1e000640 1 -- 172.21.15.102:0/3101133327 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb100052d0 con 0x7ffb040039c0 2024-03-23T03:48:09.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.178+0000 7ffb1e000640 1 -- 172.21.15.102:0/3101133327 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb1002e050 con 0x7ffb040039c0 2024-03-23T03:48:09.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.178+0000 7ffb1e000640 1 -- 172.21.15.102:0/3101133327 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10003830 con 0x7ffb040039c0 2024-03-23T03:48:09.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.179+0000 7ffb25e00640 1 -- 172.21.15.102:0/3101133327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb040039c0 msgr2=0x7ffb04003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:09.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.179+0000 7ffb25e00640 1 --2- 172.21.15.102:0/3101133327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb040039c0 0x7ffb04003de0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7ffb10001750 tx=0x7ffb040057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.179+0000 7ffb25e00640 1 -- 172.21.15.102:0/3101133327 shutdown_connections 2024-03-23T03:48:09.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.179+0000 7ffb25e00640 1 --2- 172.21.15.102:0/3101133327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb040039c0 0x7ffb04003de0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.179+0000 7ffb25e00640 1 -- 172.21.15.102:0/3101133327 >> 172.21.15.102:0/3101133327 conn(0x7ffb2006a3d0 msgr2=0x7ffb200b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:09.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.179+0000 7ffb25e00640 1 -- 172.21.15.102:0/3101133327 shutdown_connections 2024-03-23T03:48:09.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.179+0000 7ffb25e00640 1 -- 172.21.15.102:0/3101133327 wait complete. 2024-03-23T03:48:09.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.181+0000 7ffb25e00640 1 Processor -- start 2024-03-23T03:48:09.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.181+0000 7ffb25e00640 1 -- start start 2024-03-23T03:48:09.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.182+0000 7ffb25e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb040039c0 0x7ffb201503f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:09.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.182+0000 7ffb25e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb2015e4b0 0x7ffb20155f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:09.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.182+0000 7ffb25e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffb200cc370 0x7ffb200cc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:09.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.182+0000 7ffb25e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb200ccd30 con 0x7ffb040039c0 2024-03-23T03:48:09.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.182+0000 7ffb25e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2015d240 con 0x7ffb2015e4b0 2024-03-23T03:48:09.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.182+0000 7ffb25e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb20155520 con 0x7ffb200cc370 2024-03-23T03:48:09.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.182+0000 7ffb1ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb040039c0 0x7ffb201503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:09.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.182+0000 7ffb1ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb040039c0 0x7ffb201503f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41534/0 (socket says 172.21.15.102:41534) 2024-03-23T03:48:09.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.182+0000 7ffb1ea00640 1 -- 172.21.15.102:0/2177243228 learned_addr learned my addr 172.21.15.102:0/2177243228 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:09.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.182+0000 7ffb1ea00640 1 -- 172.21.15.102:0/2177243228 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffb200cc370 msgr2=0x7ffb200cc7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:09.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.182+0000 7ffb1ea00640 1 --2- 172.21.15.102:0/2177243228 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffb200cc370 0x7ffb200cc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.182+0000 7ffb1ea00640 1 -- 172.21.15.102:0/2177243228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb2015e4b0 msgr2=0x7ffb20155f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:09.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.182+0000 7ffb1ea00640 1 --2- 172.21.15.102:0/2177243228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb2015e4b0 0x7ffb20155f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.182+0000 7ffb1ea00640 1 -- 172.21.15.102:0/2177243228 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2015d490 con 0x7ffb040039c0 2024-03-23T03:48:09.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.183+0000 7ffb1ea00640 1 --2- 172.21.15.102:0/2177243228 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb040039c0 0x7ffb201503f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ffb10005040 tx=0x7ffb10001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:09.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.183+0000 7ffb17e00640 1 -- 172.21.15.102:0/2177243228 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10007640 con 0x7ffb040039c0 2024-03-23T03:48:09.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.183+0000 7ffb17e00640 1 -- 172.21.15.102:0/2177243228 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb100069e0 con 0x7ffb040039c0 2024-03-23T03:48:09.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.183+0000 7ffb17e00640 1 -- 172.21.15.102:0/2177243228 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb10006d50 con 0x7ffb040039c0 2024-03-23T03:48:09.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.183+0000 7ffb25e00640 1 -- 172.21.15.102:0/2177243228 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb2015d770 con 0x7ffb040039c0 2024-03-23T03:48:09.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.183+0000 7ffb25e00640 1 -- 172.21.15.102:0/2177243228 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb20155a50 con 0x7ffb040039c0 2024-03-23T03:48:09.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.184+0000 7ffb25e00640 1 -- 172.21.15.102:0/2177243228 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb20151e10 con 0x7ffb040039c0 2024-03-23T03:48:09.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.185+0000 7ffb17e00640 1 -- 172.21.15.102:0/2177243228 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ffb100035f0 con 0x7ffb040039c0 2024-03-23T03:48:09.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.186+0000 7ffb17e00640 1 --2- 172.21.15.102:0/2177243228 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffafc06ac30 0x7ffafc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:09.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.186+0000 7ffb17e00640 1 -- 172.21.15.102:0/2177243228 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ffb100ab420 con 0x7ffb040039c0 2024-03-23T03:48:09.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.186+0000 7ffb16000640 1 --2- 172.21.15.102:0/2177243228 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffafc06ac30 0x7ffafc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:09.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.187+0000 7ffb16000640 1 --2- 172.21.15.102:0/2177243228 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffafc06ac30 0x7ffafc06d0f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ffb200b08d0 tx=0x7ffb08008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:09.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.190+0000 7ffb17e00640 1 -- 172.21.15.102:0/2177243228 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffb10076230 con 0x7ffb040039c0 2024-03-23T03:48:09.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.404+0000 7ffb25e00640 1 -- 172.21.15.102:0/2177243228 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb2006ac20 con 0x7ffb040039c0 2024-03-23T03:48:09.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.405+0000 7ffb17e00640 1 -- 172.21.15.102:0/2177243228 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4739 (secure 0 0 0) 0x7ffb10079850 con 0x7ffb040039c0 2024-03-23T03:48:09.406 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:09.406 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/891818172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":891818172},{"type":"v1","addr":"172.21.15.102:6836","nonce":891818172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2404489233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2404489233},{"type":"v1","addr":"172.21.15.102:6837","nonce":2404489233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/1839967745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":1839967745},{"type":"v1","addr":"172.21.15.173:6835","nonce":1839967745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:46:53.912287+0000","modified":"2024-03-23T03:47:14.818670+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6834/1462637509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1462637509},{"type":"v1","addr":"172.21.15.173:6834","nonce":1462637509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-23T03:48:09.406 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 8 2024-03-23T03:48:09.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.407+0000 7ffb25e00640 1 -- 172.21.15.102:0/2177243228 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffafc06ac30 msgr2=0x7ffafc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:09.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.407+0000 7ffb25e00640 1 --2- 172.21.15.102:0/2177243228 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffafc06ac30 0x7ffafc06d0f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ffb200b08d0 tx=0x7ffb08008040 comp rx=0 tx=0).stop 2024-03-23T03:48:09.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.407+0000 7ffb25e00640 1 -- 172.21.15.102:0/2177243228 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb040039c0 msgr2=0x7ffb201503f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:09.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.407+0000 7ffb25e00640 1 --2- 172.21.15.102:0/2177243228 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb040039c0 0x7ffb201503f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7ffb10005040 tx=0x7ffb10001120 comp rx=0 tx=0).stop 2024-03-23T03:48:09.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.407+0000 7ffb25e00640 1 -- 172.21.15.102:0/2177243228 shutdown_connections 2024-03-23T03:48:09.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.407+0000 7ffb25e00640 1 --2- 172.21.15.102:0/2177243228 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffafc06ac30 0x7ffafc06d0f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.407+0000 7ffb25e00640 1 --2- 172.21.15.102:0/2177243228 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffb200cc370 0x7ffb200cc7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.407+0000 7ffb25e00640 1 --2- 172.21.15.102:0/2177243228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb2015e4b0 0x7ffb20155f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.407+0000 7ffb25e00640 1 --2- 172.21.15.102:0/2177243228 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb040039c0 0x7ffb201503f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.407+0000 7ffb25e00640 1 -- 172.21.15.102:0/2177243228 >> 172.21.15.102:0/2177243228 conn(0x7ffb2006a3d0 msgr2=0x7ffb200b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:09.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.408+0000 7ffb25e00640 1 -- 172.21.15.102:0/2177243228 shutdown_connections 2024-03-23T03:48:09.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.408+0000 7ffb25e00640 1 -- 172.21.15.102:0/2177243228 wait complete. 2024-03-23T03:48:09.425 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:48:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.626+0000 7f746b400640 1 Processor -- start 2024-03-23T03:48:09.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.626+0000 7f746b400640 1 -- start start 2024-03-23T03:48:09.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.628+0000 7f746b400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f7464160800 0x7f74640b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:09.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.628+0000 7f746b400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74640b2030 con 0x7f74640b1be0 2024-03-23T03:48:09.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.628+0000 7f746b400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7464161640 con 0x7f746413faa0 2024-03-23T03:48:09.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.628+0000 7f746b400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74640b38a0 con 0x7f7464160800 2024-03-23T03:48:09.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.628+0000 7f7469400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f74640b1be0 0x7f746406c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45188/0 (socket says 172.21.15.102:45188) 2024-03-23T03:48:09.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.628+0000 7f7469400640 1 -- 172.21.15.102:0/3135290692 learned_addr learned my addr 172.21.15.102:0/3135290692 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:09.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.628+0000 7f7463e00640 1 --2- 172.21.15.102:0/3135290692 >> v2:172.21.15.173:3300/0 conn(0x7f7464160800 0x7f74640b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:09.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.629+0000 7f7463400640 1 -- 172.21.15.102:0/3135290692 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1062987164 0 0) 0x7f745c001490 con 0x7f74640b1be0 2024-03-23T03:48:09.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.629+0000 7f7463400640 1 -- 172.21.15.102:0/3135290692 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f744c003520 con 0x7f74640b1be0 2024-03-23T03:48:09.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.629+0000 7f7463400640 1 -- 172.21.15.102:0/3135290692 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 967290836 0 0) 0x7f744c003520 con 0x7f74640b1be0 2024-03-23T03:48:09.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.629+0000 7f7463400640 1 -- 172.21.15.102:0/3135290692 >> v2:172.21.15.173:3300/0 conn(0x7f7464160800 msgr2=0x7f74640b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:09.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.629+0000 7f7463400640 1 --2- 172.21.15.102:0/3135290692 >> v2:172.21.15.173:3300/0 conn(0x7f7464160800 0x7f74640b16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.629+0000 7f7463400640 1 -- 172.21.15.102:0/3135290692 >> v1:172.21.15.173:6789/0 conn(0x7f746413faa0 legacy=0x7f7464160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:09.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.629+0000 7f7463400640 1 -- 172.21.15.102:0/3135290692 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74640b3b80 con 0x7f74640b1be0 2024-03-23T03:48:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.629+0000 7f7463400640 1 -- 172.21.15.102:0/3135290692 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f745c001b80 con 0x7f74640b1be0 2024-03-23T03:48:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.629+0000 7f7463400640 1 -- 172.21.15.102:0/3135290692 >> v1:172.21.15.102:6789/0 conn(0x7f74640b1be0 legacy=0x7f746406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.629+0000 7f7463400640 1 --2- 172.21.15.102:0/3135290692 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f744c0039c0 0x7f744c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.629+0000 7f7463400640 1 --2- 172.21.15.102:0/3135290692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f744c0047c0 0x7f744c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.629+0000 7f7463400640 1 --2- 172.21.15.102:0/3135290692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f744c005180 0x7f744c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.629+0000 7f7463400640 1 -- 172.21.15.102:0/3135290692 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74640b38a0 con 0x7f744c0047c0 2024-03-23T03:48:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.629+0000 7f7463400640 1 -- 172.21.15.102:0/3135290692 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c00c080 con 0x7f744c005180 2024-03-23T03:48:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.629+0000 7f7463400640 1 -- 172.21.15.102:0/3135290692 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f744c00c240 con 0x7f744c0039c0 2024-03-23T03:48:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.630+0000 7f7463e00640 1 --2- 172.21.15.102:0/3135290692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f744c0047c0 0x7f744c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.630+0000 7f7468a00640 1 --2- 172.21.15.102:0/3135290692 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f744c0039c0 0x7f744c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.630+0000 7f7469400640 1 --2- 172.21.15.102:0/3135290692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f744c005180 0x7f744c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.630+0000 7f7463e00640 1 -- 172.21.15.102:0/3135290692 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f744c0039c0 msgr2=0x7f744c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.630+0000 7f7463e00640 1 --2- 172.21.15.102:0/3135290692 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f744c0039c0 0x7f744c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.630+0000 7f7463e00640 1 -- 172.21.15.102:0/3135290692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f744c005180 msgr2=0x7f744c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:09.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.630+0000 7f7463e00640 1 --2- 172.21.15.102:0/3135290692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f744c005180 0x7f744c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.630+0000 7f7463e00640 1 -- 172.21.15.102:0/3135290692 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f744c003520 con 0x7f744c0047c0 2024-03-23T03:48:09.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.630+0000 7f7469400640 1 --2- 172.21.15.102:0/3135290692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f744c005180 0x7f744c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:09.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.630+0000 7f7469400640 1 -- 172.21.15.102:0/3135290692 reap_dead start 2024-03-23T03:48:09.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.631+0000 7f7463e00640 1 --2- 172.21.15.102:0/3135290692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f744c0047c0 0x7f744c004c40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7450009ac0 tx=0x7f744c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ecb43db8ab4d2e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:09.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.631+0000 7f7463400640 1 -- 172.21.15.102:0/3135290692 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745002c6b0 con 0x7f744c0047c0 2024-03-23T03:48:09.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.631+0000 7f7463400640 1 -- 172.21.15.102:0/3135290692 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f745002c810 con 0x7f744c0047c0 2024-03-23T03:48:09.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.631+0000 7f7463400640 1 -- 172.21.15.102:0/3135290692 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745002cba0 con 0x7f744c0047c0 2024-03-23T03:48:09.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.633+0000 7f746b400640 1 -- 172.21.15.102:0/3135290692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f744c0047c0 msgr2=0x7f744c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:09.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.633+0000 7f746b400640 1 --2- 172.21.15.102:0/3135290692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f744c0047c0 0x7f744c004c40 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7450009ac0 tx=0x7f744c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.633+0000 7f746b400640 1 -- 172.21.15.102:0/3135290692 shutdown_connections 2024-03-23T03:48:09.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.633+0000 7f746b400640 1 --2- 172.21.15.102:0/3135290692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f744c0047c0 0x7f744c004c40 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.633+0000 7f746b400640 1 -- 172.21.15.102:0/3135290692 >> 172.21.15.102:0/3135290692 conn(0x7f746406a3d0 msgr2=0x7f74640b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:09.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.633+0000 7f746b400640 1 -- 172.21.15.102:0/3135290692 shutdown_connections 2024-03-23T03:48:09.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.633+0000 7f746b400640 1 -- 172.21.15.102:0/3135290692 wait complete. 2024-03-23T03:48:09.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.635+0000 7f746b400640 1 Processor -- start 2024-03-23T03:48:09.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.635+0000 7f746b400640 1 -- start start 2024-03-23T03:48:09.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.635+0000 7f746b400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f744c0047c0 0x7f7464142d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:09.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.636+0000 7f746b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74641f5990 0x7f74640cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:09.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.636+0000 7f746b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74641f23f0 0x7f74641f2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:09.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.636+0000 7f746b400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7464142bb0 con 0x7f74641f5990 2024-03-23T03:48:09.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.636+0000 7f746b400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74640cd340 con 0x7f74641f23f0 2024-03-23T03:48:09.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.636+0000 7f746b400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74641f6100 con 0x7f744c0047c0 2024-03-23T03:48:09.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.636+0000 7f7463e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74641f5990 0x7f74640cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:09.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.636+0000 7f7463e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74641f5990 0x7f74640cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41544/0 (socket says 172.21.15.102:41544) 2024-03-23T03:48:09.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.636+0000 7f7463e00640 1 -- 172.21.15.102:0/1987011171 learned_addr learned my addr 172.21.15.102:0/1987011171 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:09.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.636+0000 7f7463e00640 1 -- 172.21.15.102:0/1987011171 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f744c0047c0 msgr2=0x7f7464142d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:09.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.636+0000 7f7463e00640 1 --2- 172.21.15.102:0/1987011171 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f744c0047c0 0x7f7464142d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.636+0000 7f7463e00640 1 -- 172.21.15.102:0/1987011171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74641f23f0 msgr2=0x7f74641f2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:09.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.636+0000 7f7463e00640 1 --2- 172.21.15.102:0/1987011171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74641f23f0 0x7f74641f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.636+0000 7f7463e00640 1 -- 172.21.15.102:0/1987011171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74641f63e0 con 0x7f74641f5990 2024-03-23T03:48:09.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.637+0000 7f7463e00640 1 --2- 172.21.15.102:0/1987011171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74641f5990 0x7f74640cce00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f745002c250 tx=0x7f7450004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:09.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.637+0000 7f7460c00640 1 -- 172.21.15.102:0/1987011171 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7450003810 con 0x7f74641f5990 2024-03-23T03:48:09.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.637+0000 7f7460c00640 1 -- 172.21.15.102:0/1987011171 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7450003970 con 0x7f74641f5990 2024-03-23T03:48:09.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.637+0000 7f7460c00640 1 -- 172.21.15.102:0/1987011171 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7450003d00 con 0x7f74641f5990 2024-03-23T03:48:09.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.637+0000 7f746b400640 1 -- 172.21.15.102:0/1987011171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74641f66c0 con 0x7f74641f5990 2024-03-23T03:48:09.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.637+0000 7f746b400640 1 -- 172.21.15.102:0/1987011171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74641f5e10 con 0x7f74641f5990 2024-03-23T03:48:09.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.638+0000 7f746b400640 1 -- 172.21.15.102:0/1987011171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74641f6900 con 0x7f74641f5990 2024-03-23T03:48:09.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.639+0000 7f7460c00640 1 -- 172.21.15.102:0/1987011171 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7450034440 con 0x7f74641f5990 2024-03-23T03:48:09.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.640+0000 7f7460c00640 1 --2- 172.21.15.102:0/1987011171 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f744406ac00 0x7f744406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:09.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.640+0000 7f7460c00640 1 -- 172.21.15.102:0/1987011171 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f74500ab990 con 0x7f74641f5990 2024-03-23T03:48:09.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.640+0000 7f7468a00640 1 --2- 172.21.15.102:0/1987011171 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f744406ac00 0x7f744406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:09.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.641+0000 7f7468a00640 1 --2- 172.21.15.102:0/1987011171 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f744406ac00 0x7f744406d0c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7458007fd0 tx=0x7f7458009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:09.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.645+0000 7f7460c00640 1 -- 172.21.15.102:0/1987011171 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f74500768f0 con 0x7f74641f5990 2024-03-23T03:48:09.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.879+0000 7f746b400640 1 -- 172.21.15.102:0/1987011171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7464073390 con 0x7f74641f5990 2024-03-23T03:48:09.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.880+0000 7f7460c00640 1 -- 172.21.15.102:0/1987011171 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4739 (secure 0 0 0) 0x7f7450003e60 con 0x7f74641f5990 2024-03-23T03:48:09.881 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:09.881 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4245,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/891818172","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":891818172},{"type":"v1","addr":"172.21.15.102:6836","nonce":891818172}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4261,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/2404489233","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":2404489233},{"type":"v1","addr":"172.21.15.102:6837","nonce":2404489233}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4263,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/1839967745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":1839967745},{"type":"v1","addr":"172.21.15.173:6835","nonce":1839967745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:46:53.912287+0000","modified":"2024-03-23T03:47:14.818670+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4270},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4270":{"gid":4270,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6834/1462637509","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1462637509},{"type":"v1","addr":"172.21.15.173:6834","nonce":1462637509}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-23T03:48:09.881 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 8 2024-03-23T03:48:09.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.883+0000 7f746b400640 1 -- 172.21.15.102:0/1987011171 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f744406ac00 msgr2=0x7f744406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:09.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.883+0000 7f746b400640 1 --2- 172.21.15.102:0/1987011171 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f744406ac00 0x7f744406d0c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7458007fd0 tx=0x7f7458009040 comp rx=0 tx=0).stop 2024-03-23T03:48:09.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.883+0000 7f746b400640 1 -- 172.21.15.102:0/1987011171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74641f5990 msgr2=0x7f74640cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:09.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.883+0000 7f746b400640 1 --2- 172.21.15.102:0/1987011171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74641f5990 0x7f74640cce00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f745002c250 tx=0x7f7450004730 comp rx=0 tx=0).stop 2024-03-23T03:48:09.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.883+0000 7f746b400640 1 -- 172.21.15.102:0/1987011171 shutdown_connections 2024-03-23T03:48:09.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.883+0000 7f746b400640 1 --2- 172.21.15.102:0/1987011171 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f744406ac00 0x7f744406d0c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.883+0000 7f746b400640 1 --2- 172.21.15.102:0/1987011171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74641f23f0 0x7f74641f2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.883+0000 7f746b400640 1 --2- 172.21.15.102:0/1987011171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74641f5990 0x7f74640cce00 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.883+0000 7f746b400640 1 --2- 172.21.15.102:0/1987011171 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f744c0047c0 0x7f7464142d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:09.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.883+0000 7f746b400640 1 -- 172.21.15.102:0/1987011171 >> 172.21.15.102:0/1987011171 conn(0x7f746406a3d0 msgr2=0x7f74640af930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:09.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.883+0000 7f746b400640 1 -- 172.21.15.102:0/1987011171 shutdown_connections 2024-03-23T03:48:09.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:09.883+0000 7f746b400640 1 -- 172.21.15.102:0/1987011171 wait complete. 2024-03-23T03:48:09.903 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:48:10.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.117+0000 7f7c81000640 1 Processor -- start 2024-03-23T03:48:10.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.117+0000 7f7c81000640 1 -- start start 2024-03-23T03:48:10.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.118+0000 7f7c81000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c7c0adbc0 0x7f7c7c0b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:10.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.118+0000 7f7c81000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7c7c06d390 0x7f7c7c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:10.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.118+0000 7f7c81000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7c0b16a0 con 0x7f7c7c0cbab0 2024-03-23T03:48:10.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.118+0000 7f7c81000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c7c06dd60 con 0x7f7c7c06d390 2024-03-23T03:48:10.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.118+0000 7f7c81000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c0b4700 con 0x7f7c7c0adbc0 2024-03-23T03:48:10.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.119+0000 7f7c7a800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7c7c06d390 0x7f7c7c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:10.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.119+0000 7f7c7a800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7c7c06d390 0x7f7c7c06d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41550/0 (socket says 172.21.15.102:41550) 2024-03-23T03:48:10.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.119+0000 7f7c7a800640 1 -- 172.21.15.102:0/1170678024 learned_addr learned my addr 172.21.15.102:0/1170678024 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:10.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.119+0000 7f7c79e00640 1 --2- 172.21.15.102:0/1170678024 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c7c0adbc0 0x7f7c7c0b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:10.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.119+0000 7f7c7a800640 1 -- 172.21.15.102:0/1170678024 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c7c0adbc0 msgr2=0x7f7c7c0b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:10.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.119+0000 7f7c7a800640 1 --2- 172.21.15.102:0/1170678024 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c7c0adbc0 0x7f7c7c0b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.119+0000 7f7c7a800640 1 -- 172.21.15.102:0/1170678024 >> v1:172.21.15.173:6789/0 conn(0x7f7c7c0cbab0 legacy=0x7f7c7c0b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:10.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.119+0000 7f7c7a800640 1 -- 172.21.15.102:0/1170678024 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c0cbfb0 con 0x7f7c7c06d390 2024-03-23T03:48:10.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.120+0000 7f7c7a800640 1 --2- 172.21.15.102:0/1170678024 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7c06d390 0x7f7c7c06d820 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7c74009870 tx=0x7f7c7402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d986e591e87b81d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:10.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.121+0000 7f7c79400640 1 -- 172.21.15.102:0/1170678024 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c740317c0 con 0x7f7c7c06d390 2024-03-23T03:48:10.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.121+0000 7f7c79400640 1 -- 172.21.15.102:0/1170678024 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c74031920 con 0x7f7c7c06d390 2024-03-23T03:48:10.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.121+0000 7f7c79400640 1 -- 172.21.15.102:0/1170678024 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c74031cb0 con 0x7f7c7c06d390 2024-03-23T03:48:10.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.121+0000 7f7c81000640 1 -- 172.21.15.102:0/1170678024 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7c06d390 msgr2=0x7f7c7c06d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:10.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.121+0000 7f7c81000640 1 --2- 172.21.15.102:0/1170678024 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7c06d390 0x7f7c7c06d820 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7c74009870 tx=0x7f7c7402b860 comp rx=0 tx=0).stop 2024-03-23T03:48:10.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.122+0000 7f7c81000640 1 -- 172.21.15.102:0/1170678024 shutdown_connections 2024-03-23T03:48:10.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.122+0000 7f7c81000640 1 --2- 172.21.15.102:0/1170678024 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7c06d390 0x7f7c7c06d820 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.122+0000 7f7c81000640 1 --2- 172.21.15.102:0/1170678024 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c7c0adbc0 0x7f7c7c0b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.122+0000 7f7c81000640 1 -- 172.21.15.102:0/1170678024 >> 172.21.15.102:0/1170678024 conn(0x7f7c7c06a3d0 msgr2=0x7f7c7c0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:10.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.122+0000 7f7c81000640 1 -- 172.21.15.102:0/1170678024 shutdown_connections 2024-03-23T03:48:10.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.122+0000 7f7c81000640 1 -- 172.21.15.102:0/1170678024 wait complete. 2024-03-23T03:48:10.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.124+0000 7f7c81000640 1 Processor -- start 2024-03-23T03:48:10.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.124+0000 7f7c81000640 1 -- start start 2024-03-23T03:48:10.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.125+0000 7f7c81000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c7c06d390 0x7f7c7c137e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:10.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.125+0000 7f7c81000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7c0adbc0 0x7f7c7c15b260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:10.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.125+0000 7f7c81000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c7c0cbab0 0x7f7c7c1563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:10.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.125+0000 7f7c81000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c1594e0 con 0x7f7c7c0adbc0 2024-03-23T03:48:10.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.125+0000 7f7c81000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c1383d0 con 0x7f7c7c06d390 2024-03-23T03:48:10.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.125+0000 7f7c81000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7c137a00 con 0x7f7c7c0cbab0 2024-03-23T03:48:10.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.125+0000 7f7c71400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7c0adbc0 0x7f7c7c15b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:10.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.125+0000 7f7c71400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7c0adbc0 0x7f7c7c15b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41566/0 (socket says 172.21.15.102:41566) 2024-03-23T03:48:10.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.126+0000 7f7c71400640 1 -- 172.21.15.102:0/3770035372 learned_addr learned my addr 172.21.15.102:0/3770035372 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:10.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.126+0000 7f7c79e00640 1 --2- 172.21.15.102:0/3770035372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c7c06d390 0x7f7c7c137e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:10.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.126+0000 7f7c7a800640 1 --2- 172.21.15.102:0/3770035372 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c7c0cbab0 0x7f7c7c1563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:10.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.126+0000 7f7c71400640 1 -- 172.21.15.102:0/3770035372 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c7c0cbab0 msgr2=0x7f7c7c1563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:10.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.126+0000 7f7c71400640 1 --2- 172.21.15.102:0/3770035372 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c7c0cbab0 0x7f7c7c1563a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.126+0000 7f7c71400640 1 -- 172.21.15.102:0/3770035372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c7c06d390 msgr2=0x7f7c7c137e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:10.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.126+0000 7f7c71400640 1 --2- 172.21.15.102:0/3770035372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c7c06d390 0x7f7c7c137e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.126+0000 7f7c71400640 1 -- 172.21.15.102:0/3770035372 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c740094d0 con 0x7f7c7c0adbc0 2024-03-23T03:48:10.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.126+0000 7f7c71400640 1 --2- 172.21.15.102:0/3770035372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7c0adbc0 0x7f7c7c15b260 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400c410 tx=0x7f7c6400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:10.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.127+0000 7f7c72a00640 1 -- 172.21.15.102:0/3770035372 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c640035f0 con 0x7f7c7c0adbc0 2024-03-23T03:48:10.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.127+0000 7f7c72a00640 1 -- 172.21.15.102:0/3770035372 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c64003750 con 0x7f7c7c0adbc0 2024-03-23T03:48:10.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.127+0000 7f7c72a00640 1 -- 172.21.15.102:0/3770035372 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c64003b40 con 0x7f7c7c0adbc0 2024-03-23T03:48:10.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.127+0000 7f7c81000640 1 -- 172.21.15.102:0/3770035372 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7c136df0 con 0x7f7c7c0adbc0 2024-03-23T03:48:10.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.127+0000 7f7c81000640 1 -- 172.21.15.102:0/3770035372 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7c1373b0 con 0x7f7c7c0adbc0 2024-03-23T03:48:10.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.128+0000 7f7c81000640 1 -- 172.21.15.102:0/3770035372 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7c06ac20 con 0x7f7c7c0adbc0 2024-03-23T03:48:10.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.129+0000 7f7c72a00640 1 -- 172.21.15.102:0/3770035372 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7c64003ca0 con 0x7f7c7c0adbc0 2024-03-23T03:48:10.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.130+0000 7f7c72a00640 1 --2- 172.21.15.102:0/3770035372 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7c5806ac30 0x7f7c5806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:10.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.130+0000 7f7c72a00640 1 -- 172.21.15.102:0/3770035372 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7c64088870 con 0x7f7c7c0adbc0 2024-03-23T03:48:10.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.130+0000 7f7c79e00640 1 --2- 172.21.15.102:0/3770035372 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7c5806ac30 0x7f7c5806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:10.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.131+0000 7f7c79e00640 1 --2- 172.21.15.102:0/3770035372 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7c5806ac30 0x7f7c5806d0f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0070b0 tx=0x7f7c6c007040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:10.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.135+0000 7f7c72a00640 1 -- 172.21.15.102:0/3770035372 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7c64053690 con 0x7f7c7c0adbc0 2024-03-23T03:48:10.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.328+0000 7f7c81000640 1 -- 172.21.15.102:0/3770035372 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c7c073390 con 0x7f7c7c0adbc0 2024-03-23T03:48:10.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.330+0000 7f7c72a00640 1 -- 172.21.15.102:0/3770035372 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v28) v1 ==== 74+0+18260 (secure 0 0 0) 0x7f7c64056cb0 con 0x7f7c7c0adbc0 2024-03-23T03:48:10.331 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:10.331 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":28,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:47:41.647786+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:46:52.158650+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T03:46:52.914275+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-23T03:46:55.983754+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1d","mappings":[{"from":5,"to":2}]},{"pgid":"3.29","mappings":[{"from":3,"to":1}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"3.3f","mappings":[{"from":7,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.10","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:48:10.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.332+0000 7f7c81000640 1 -- 172.21.15.102:0/3770035372 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7c5806ac30 msgr2=0x7f7c5806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:10.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.332+0000 7f7c81000640 1 --2- 172.21.15.102:0/3770035372 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7c5806ac30 0x7f7c5806d0f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0070b0 tx=0x7f7c6c007040 comp rx=0 tx=0).stop 2024-03-23T03:48:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.332+0000 7f7c81000640 1 -- 172.21.15.102:0/3770035372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7c0adbc0 msgr2=0x7f7c7c15b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.332+0000 7f7c81000640 1 --2- 172.21.15.102:0/3770035372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7c0adbc0 0x7f7c7c15b260 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7c6400c410 tx=0x7f7c6400c8d0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.333+0000 7f7c81000640 1 -- 172.21.15.102:0/3770035372 shutdown_connections 2024-03-23T03:48:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.333+0000 7f7c81000640 1 --2- 172.21.15.102:0/3770035372 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7c5806ac30 0x7f7c5806d0f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.333+0000 7f7c81000640 1 --2- 172.21.15.102:0/3770035372 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c7c0cbab0 0x7f7c7c1563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.333+0000 7f7c81000640 1 --2- 172.21.15.102:0/3770035372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c7c0adbc0 0x7f7c7c15b260 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.333+0000 7f7c81000640 1 --2- 172.21.15.102:0/3770035372 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c7c06d390 0x7f7c7c137e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.333+0000 7f7c81000640 1 -- 172.21.15.102:0/3770035372 >> 172.21.15.102:0/3770035372 conn(0x7f7c7c06a3d0 msgr2=0x7f7c7c138ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.333+0000 7f7c81000640 1 -- 172.21.15.102:0/3770035372 shutdown_connections 2024-03-23T03:48:10.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.333+0000 7f7c81000640 1 -- 172.21.15.102:0/3770035372 wait complete. 2024-03-23T03:48:10.352 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T03:48:10.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.557+0000 7f3ddc200640 1 Processor -- start 2024-03-23T03:48:10.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.558+0000 7f3ddc200640 1 -- start start 2024-03-23T03:48:10.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.559+0000 7f3ddc200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3dd40ce930 0x7f3dd40ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.559+0000 7f3ddc200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3dd40cb660 0x7f3dd40cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.559+0000 7f3ddc200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd40cf250 con 0x7f3dd4070170 2024-03-23T03:48:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.559+0000 7f3ddc200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dd415bae0 con 0x7f3dd40ce930 2024-03-23T03:48:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.559+0000 7f3ddc200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd415ae60 con 0x7f3dd40cb660 2024-03-23T03:48:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.559+0000 7f3dd9800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3dd40ce930 0x7f3dd40ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.560+0000 7f3dd9800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3dd40ce930 0x7f3dd40ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41576/0 (socket says 172.21.15.102:41576) 2024-03-23T03:48:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.560+0000 7f3dd9800640 1 -- 172.21.15.102:0/2734384697 learned_addr learned my addr 172.21.15.102:0/2734384697 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.560+0000 7f3dd8e00640 1 --2- 172.21.15.102:0/2734384697 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3dd40cb660 0x7f3dd40cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.560+0000 7f3dd9800640 1 -- 172.21.15.102:0/2734384697 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3dd40cb660 msgr2=0x7f3dd40cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.560+0000 7f3dd9800640 1 --2- 172.21.15.102:0/2734384697 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3dd40cb660 0x7f3dd40cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.560+0000 7f3dd9800640 1 -- 172.21.15.102:0/2734384697 >> v1:172.21.15.173:6789/0 conn(0x7f3dd4070170 legacy=0x7f3dd4070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:10.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.560+0000 7f3dd9800640 1 -- 172.21.15.102:0/2734384697 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd415b140 con 0x7f3dd40ce930 2024-03-23T03:48:10.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.561+0000 7f3dd9800640 1 --2- 172.21.15.102:0/2734384697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3dd40ce930 0x7f3dd40ced10 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009870 tx=0x7f3dc402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a7d54034a3976ee server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:10.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.562+0000 7f3dd3e00640 1 -- 172.21.15.102:0/2734384697 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4031640 con 0x7f3dd40ce930 2024-03-23T03:48:10.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.562+0000 7f3dd3e00640 1 -- 172.21.15.102:0/2734384697 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc40317a0 con 0x7f3dd40ce930 2024-03-23T03:48:10.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.562+0000 7f3dd3e00640 1 -- 172.21.15.102:0/2734384697 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4031b30 con 0x7f3dd40ce930 2024-03-23T03:48:10.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.563+0000 7f3ddc200640 1 -- 172.21.15.102:0/2734384697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3dd40ce930 msgr2=0x7f3dd40ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:10.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.563+0000 7f3ddc200640 1 --2- 172.21.15.102:0/2734384697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3dd40ce930 0x7f3dd40ced10 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009870 tx=0x7f3dc402b8f0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.563+0000 7f3ddc200640 1 -- 172.21.15.102:0/2734384697 shutdown_connections 2024-03-23T03:48:10.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.563+0000 7f3ddc200640 1 --2- 172.21.15.102:0/2734384697 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3dd40cb660 0x7f3dd40cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.563+0000 7f3ddc200640 1 --2- 172.21.15.102:0/2734384697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3dd40ce930 0x7f3dd40ced10 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.563+0000 7f3ddc200640 1 -- 172.21.15.102:0/2734384697 >> 172.21.15.102:0/2734384697 conn(0x7f3dd406a0b0 msgr2=0x7f3dd406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:10.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.563+0000 7f3ddc200640 1 -- 172.21.15.102:0/2734384697 shutdown_connections 2024-03-23T03:48:10.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.563+0000 7f3ddc200640 1 -- 172.21.15.102:0/2734384697 wait complete. 2024-03-23T03:48:10.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.565+0000 7f3ddc200640 1 Processor -- start 2024-03-23T03:48:10.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.565+0000 7f3ddc200640 1 -- start start 2024-03-23T03:48:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.566+0000 7f3ddc200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3dd4070170 0x7f3dd414c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.566+0000 7f3ddc200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3dd40cb660 0x7f3dd4152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.566+0000 7f3ddc200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3dd40ce930 0x7f3dd4152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.566+0000 7f3ddc200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd40b2f30 con 0x7f3dd40cb660 2024-03-23T03:48:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.566+0000 7f3ddc200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd41535e0 con 0x7f3dd40ce930 2024-03-23T03:48:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.566+0000 7f3ddc200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd41538c0 con 0x7f3dd4070170 2024-03-23T03:48:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.566+0000 7f3dd8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3dd40cb660 0x7f3dd4152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.566+0000 7f3dd8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3dd40cb660 0x7f3dd4152a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41588/0 (socket says 172.21.15.102:41588) 2024-03-23T03:48:10.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.566+0000 7f3dd8e00640 1 -- 172.21.15.102:0/110150530 learned_addr learned my addr 172.21.15.102:0/110150530 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:10.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.567+0000 7f3dd9800640 1 --2- 172.21.15.102:0/110150530 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3dd4070170 0x7f3dd414c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:10.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.567+0000 7f3dda200640 1 --2- 172.21.15.102:0/110150530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3dd40ce930 0x7f3dd4152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:10.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.567+0000 7f3dd8e00640 1 -- 172.21.15.102:0/110150530 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3dd4070170 msgr2=0x7f3dd414c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:10.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.567+0000 7f3dd8e00640 1 --2- 172.21.15.102:0/110150530 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3dd4070170 0x7f3dd414c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.567+0000 7f3dd8e00640 1 -- 172.21.15.102:0/110150530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3dd40ce930 msgr2=0x7f3dd4152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:10.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.567+0000 7f3dd8e00640 1 --2- 172.21.15.102:0/110150530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3dd40ce930 0x7f3dd4152fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.567+0000 7f3dd8e00640 1 -- 172.21.15.102:0/110150530 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc40094d0 con 0x7f3dd40cb660 2024-03-23T03:48:10.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.567+0000 7f3dda200640 1 --2- 172.21.15.102:0/110150530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3dd40ce930 0x7f3dd4152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:10.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.567+0000 7f3dd8e00640 1 --2- 172.21.15.102:0/110150530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3dd40cb660 0x7f3dd4152a70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3dc000d800 tx=0x7f3dc000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:10.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.567+0000 7f3dd1600640 1 -- 172.21.15.102:0/110150530 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc000b6c0 con 0x7f3dd40cb660 2024-03-23T03:48:10.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.567+0000 7f3dd1600640 1 -- 172.21.15.102:0/110150530 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc000b820 con 0x7f3dd40cb660 2024-03-23T03:48:10.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.567+0000 7f3dd1600640 1 -- 172.21.15.102:0/110150530 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc000bbb0 con 0x7f3dd40cb660 2024-03-23T03:48:10.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.567+0000 7f3ddc200640 1 -- 172.21.15.102:0/110150530 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd4155280 con 0x7f3dd40cb660 2024-03-23T03:48:10.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.568+0000 7f3ddc200640 1 -- 172.21.15.102:0/110150530 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd40ae4f0 con 0x7f3dd40cb660 2024-03-23T03:48:10.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.569+0000 7f3ddc200640 1 -- 172.21.15.102:0/110150530 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd4073390 con 0x7f3dd40cb660 2024-03-23T03:48:10.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.570+0000 7f3dd1600640 1 -- 172.21.15.102:0/110150530 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3dc000bd10 con 0x7f3dd40cb660 2024-03-23T03:48:10.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.570+0000 7f3dd1600640 1 --2- 172.21.15.102:0/110150530 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3db406ac00 0x7f3db406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:10.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.571+0000 7f3dd1600640 1 -- 172.21.15.102:0/110150530 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3dc0089c70 con 0x7f3dd40cb660 2024-03-23T03:48:10.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.571+0000 7f3dd9800640 1 --2- 172.21.15.102:0/110150530 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3db406ac00 0x7f3db406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:10.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.572+0000 7f3dd9800640 1 --2- 172.21.15.102:0/110150530 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3db406ac00 0x7f3db406d0c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4036000 tx=0x7f3dc403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:10.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.575+0000 7f3dd1600640 1 -- 172.21.15.102:0/110150530 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3dc0054a80 con 0x7f3dd40cb660 2024-03-23T03:48:10.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.807+0000 7f3ddc200640 1 -- 172.21.15.102:0/110150530 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3dd4066ce0 con 0x7f3dd40cb660 2024-03-23T03:48:10.809 INFO:tasks.ceph.mon.a.smithi102.stderr:2024-03-23T03:48:10.809+0000 7f5aab200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T03:48:10.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.825+0000 7f3dd1600640 1 -- 172.21.15.102:0/110150530 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3dc00580a0 con 0x7f3dd40cb660 2024-03-23T03:48:10.826 INFO:teuthology.orchestra.run.smithi102.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T03:48:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.828+0000 7f3ddc200640 1 -- 172.21.15.102:0/110150530 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3db406ac00 msgr2=0x7f3db406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.828+0000 7f3ddc200640 1 --2- 172.21.15.102:0/110150530 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3db406ac00 0x7f3db406d0c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4036000 tx=0x7f3dc403c040 comp rx=0 tx=0).stop 2024-03-23T03:48:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.828+0000 7f3ddc200640 1 -- 172.21.15.102:0/110150530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3dd40cb660 msgr2=0x7f3dd4152a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.828+0000 7f3ddc200640 1 --2- 172.21.15.102:0/110150530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3dd40cb660 0x7f3dd4152a70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f3dc000d800 tx=0x7f3dc000dcc0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.828+0000 7f3ddc200640 1 -- 172.21.15.102:0/110150530 shutdown_connections 2024-03-23T03:48:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.828+0000 7f3ddc200640 1 --2- 172.21.15.102:0/110150530 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3db406ac00 0x7f3db406d0c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.828+0000 7f3ddc200640 1 --2- 172.21.15.102:0/110150530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3dd40ce930 0x7f3dd4152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.828+0000 7f3ddc200640 1 --2- 172.21.15.102:0/110150530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3dd40cb660 0x7f3dd4152a70 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.828+0000 7f3ddc200640 1 --2- 172.21.15.102:0/110150530 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3dd4070170 0x7f3dd414c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.828+0000 7f3ddc200640 1 -- 172.21.15.102:0/110150530 >> 172.21.15.102:0/110150530 conn(0x7f3dd406a0b0 msgr2=0x7f3dd406c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:10.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.828+0000 7f3ddc200640 1 -- 172.21.15.102:0/110150530 shutdown_connections 2024-03-23T03:48:10.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:10.828+0000 7f3ddc200640 1 -- 172.21.15.102:0/110150530 wait complete. 2024-03-23T03:48:10.845 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T03:48:10.857 INFO:tasks.ceph.mds.d.smithi173.stderr: -5511> 2024-03-23T03:46:58.194+0000 7fc643000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T03:48:10.936 INFO:tasks.ceph.mds.d.smithi173.stdout:starting mds.d at 2024-03-23T03:48:11.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.042+0000 7f7700200640 1 Processor -- start 2024-03-23T03:48:11.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.043+0000 7f7700200640 1 -- start start 2024-03-23T03:48:11.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.044+0000 7f7700200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f76f815ca20 0x7f76f81605d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:11.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.044+0000 7f7700200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f76f80cdd10 0x7f76f80d12c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:11.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.044+0000 7f7700200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76f80d0e30 con 0x7f76f80b8330 2024-03-23T03:48:11.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.044+0000 7f7700200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f80d0b30 con 0x7f76f80cdd10 2024-03-23T03:48:11.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.044+0000 7f7700200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76f8160eb0 con 0x7f76f815ca20 2024-03-23T03:48:11.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.044+0000 7f76fce00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f76f80b8330 0x7f76f80cd600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45196/0 (socket says 172.21.15.102:45196) 2024-03-23T03:48:11.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.044+0000 7f76fce00640 1 -- 172.21.15.102:0/565940786 learned_addr learned my addr 172.21.15.102:0/565940786 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:11.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.044+0000 7f76fe200640 1 --2- 172.21.15.102:0/565940786 >> v2:172.21.15.102:3300/0 conn(0x7f76f80cdd10 0x7f76f80d12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:11.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.045+0000 7f76fd800640 1 --2- 172.21.15.102:0/565940786 >> v2:172.21.15.173:3300/0 conn(0x7f76f815ca20 0x7f76f81605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:11.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.045+0000 7f76f7e00640 1 -- 172.21.15.102:0/565940786 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657916459 0 0) 0x7f76f80d0e30 con 0x7f76f80b8330 2024-03-23T03:48:11.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.045+0000 7f76f7e00640 1 -- 172.21.15.102:0/565940786 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76e0003520 con 0x7f76f80b8330 2024-03-23T03:48:11.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.045+0000 7f76fe200640 1 -- 172.21.15.102:0/565940786 >> v2:172.21.15.173:3300/0 conn(0x7f76f815ca20 msgr2=0x7f76f81605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:11.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.045+0000 7f76fe200640 1 --2- 172.21.15.102:0/565940786 >> v2:172.21.15.173:3300/0 conn(0x7f76f815ca20 0x7f76f81605d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:11.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.045+0000 7f76fe200640 1 -- 172.21.15.102:0/565940786 >> v1:172.21.15.102:6789/0 conn(0x7f76f80b8330 legacy=0x7f76f80cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:11.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.045+0000 7f76fe200640 1 -- 172.21.15.102:0/565940786 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f8161190 con 0x7f76f80cdd10 2024-03-23T03:48:11.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.046+0000 7f76fe200640 1 --2- 172.21.15.102:0/565940786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76f80cdd10 0x7f76f80d12c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f76f00097a0 tx=0x7f76f002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cdf90f07fccbf5a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:11.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.047+0000 7f76f7e00640 1 -- 172.21.15.102:0/565940786 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f0031830 con 0x7f76f80cdd10 2024-03-23T03:48:11.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.047+0000 7f76f7e00640 1 -- 172.21.15.102:0/565940786 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76f0031990 con 0x7f76f80cdd10 2024-03-23T03:48:11.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.047+0000 7f76f7e00640 1 -- 172.21.15.102:0/565940786 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76f0031d00 con 0x7f76f80cdd10 2024-03-23T03:48:11.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.047+0000 7f7700200640 1 -- 172.21.15.102:0/565940786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76f80cdd10 msgr2=0x7f76f80d12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:11.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.047+0000 7f7700200640 1 --2- 172.21.15.102:0/565940786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76f80cdd10 0x7f76f80d12c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f76f00097a0 tx=0x7f76f002b8d0 comp rx=0 tx=0).stop 2024-03-23T03:48:11.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.047+0000 7f7700200640 1 -- 172.21.15.102:0/565940786 shutdown_connections 2024-03-23T03:48:11.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.047+0000 7f7700200640 1 --2- 172.21.15.102:0/565940786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76f80cdd10 0x7f76f80d12c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:11.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.047+0000 7f7700200640 1 --2- 172.21.15.102:0/565940786 >> v2:172.21.15.173:3300/0 conn(0x7f76f815ca20 0x7f76f81605d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:11.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.047+0000 7f7700200640 1 -- 172.21.15.102:0/565940786 >> 172.21.15.102:0/565940786 conn(0x7f76f80cf7d0 msgr2=0x7f76f80cfbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:11.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.048+0000 7f7700200640 1 -- 172.21.15.102:0/565940786 shutdown_connections 2024-03-23T03:48:11.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.048+0000 7f7700200640 1 -- 172.21.15.102:0/565940786 wait complete. 2024-03-23T03:48:11.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.050+0000 7f7700200640 1 Processor -- start 2024-03-23T03:48:11.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.050+0000 7f7700200640 1 -- start start 2024-03-23T03:48:11.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.051+0000 7f7700200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76f80b8330 0x7f76f80cb010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:11.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.051+0000 7f7700200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76f80cdd10 0x7f76f81f56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:11.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.051+0000 7f7700200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f815ca20 0x7f76f80cc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:11.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.051+0000 7f7700200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f80cbef0 con 0x7f76f80cdd10 2024-03-23T03:48:11.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.051+0000 7f7700200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f80c6290 con 0x7f76f815ca20 2024-03-23T03:48:11.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.051+0000 7f7700200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76f80c6570 con 0x7f76f80b8330 2024-03-23T03:48:11.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.051+0000 7f76fce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76f80cdd10 0x7f76f81f56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:11.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.052+0000 7f76fce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76f80cdd10 0x7f76f81f56e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41606/0 (socket says 172.21.15.102:41606) 2024-03-23T03:48:11.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.052+0000 7f76fce00640 1 -- 172.21.15.102:0/1368699416 learned_addr learned my addr 172.21.15.102:0/1368699416 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:11.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.052+0000 7f76fd800640 1 --2- 172.21.15.102:0/1368699416 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76f80b8330 0x7f76f80cb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:11.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.052+0000 7f76fe200640 1 --2- 172.21.15.102:0/1368699416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f815ca20 0x7f76f80cc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:11.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.052+0000 7f76fce00640 1 -- 172.21.15.102:0/1368699416 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76f80b8330 msgr2=0x7f76f80cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:11.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.052+0000 7f76fce00640 1 --2- 172.21.15.102:0/1368699416 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76f80b8330 0x7f76f80cb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:11.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.052+0000 7f76fce00640 1 -- 172.21.15.102:0/1368699416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f815ca20 msgr2=0x7f76f80cc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:11.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.052+0000 7f76fce00640 1 --2- 172.21.15.102:0/1368699416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f815ca20 0x7f76f80cc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:11.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.052+0000 7f76fce00640 1 -- 172.21.15.102:0/1368699416 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f0009400 con 0x7f76f80cdd10 2024-03-23T03:48:11.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.052+0000 7f76fd800640 1 --2- 172.21.15.102:0/1368699416 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76f80b8330 0x7f76f80cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:11.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.052+0000 7f76fce00640 1 --2- 172.21.15.102:0/1368699416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76f80cdd10 0x7f76f81f56e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f76e800cb40 tx=0x7f76e800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:11.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.053+0000 7f76f5600640 1 -- 172.21.15.102:0/1368699416 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e800aca0 con 0x7f76f80cdd10 2024-03-23T03:48:11.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.053+0000 7f76f5600640 1 -- 172.21.15.102:0/1368699416 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e800ae00 con 0x7f76f80cdd10 2024-03-23T03:48:11.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.053+0000 7f7700200640 1 -- 172.21.15.102:0/1368699416 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f80cb830 con 0x7f76f80cdd10 2024-03-23T03:48:11.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.053+0000 7f76f5600640 1 -- 172.21.15.102:0/1368699416 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76e8012620 con 0x7f76f80cdd10 2024-03-23T03:48:11.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.053+0000 7f7700200640 1 -- 172.21.15.102:0/1368699416 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f80c6850 con 0x7f76f80cdd10 2024-03-23T03:48:11.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.054+0000 7f7700200640 1 -- 172.21.15.102:0/1368699416 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f8073390 con 0x7f76f80cdd10 2024-03-23T03:48:11.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.055+0000 7f76f5600640 1 -- 172.21.15.102:0/1368699416 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f76e8007070 con 0x7f76f80cdd10 2024-03-23T03:48:11.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.056+0000 7f76f5600640 1 --2- 172.21.15.102:0/1368699416 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f76dc06ac00 0x7f76dc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:11.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.056+0000 7f76f5600640 1 -- 172.21.15.102:0/1368699416 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f76e8089860 con 0x7f76f80cdd10 2024-03-23T03:48:11.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.056+0000 7f76fd800640 1 --2- 172.21.15.102:0/1368699416 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f76dc06ac00 0x7f76dc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:11.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.057+0000 7f76fd800640 1 --2- 172.21.15.102:0/1368699416 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f76dc06ac00 0x7f76dc06d0c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009a20 tx=0x7f76ec008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:11.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.060+0000 7f76f5600640 1 -- 172.21.15.102:0/1368699416 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f76e8054590 con 0x7f76f80cdd10 2024-03-23T03:48:11.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.276+0000 7f7700200640 1 -- 172.21.15.102:0/1368699416 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f76f80d0640 con 0x7f76f80cdd10 2024-03-23T03:48:11.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.861+0000 7f76f5600640 1 -- 172.21.15.102:0/1368699416 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f76e8001c00 con 0x7f76f80cdd10 2024-03-23T03:48:11.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.865+0000 7f7700200640 1 -- 172.21.15.102:0/1368699416 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f76dc06ac00 msgr2=0x7f76dc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:11.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.865+0000 7f7700200640 1 --2- 172.21.15.102:0/1368699416 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f76dc06ac00 0x7f76dc06d0c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f76ec009a20 tx=0x7f76ec008040 comp rx=0 tx=0).stop 2024-03-23T03:48:11.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.865+0000 7f7700200640 1 -- 172.21.15.102:0/1368699416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76f80cdd10 msgr2=0x7f76f81f56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:11.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.865+0000 7f7700200640 1 --2- 172.21.15.102:0/1368699416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76f80cdd10 0x7f76f81f56e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f76e800cb40 tx=0x7f76e800a060 comp rx=0 tx=0).stop 2024-03-23T03:48:11.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.865+0000 7f7700200640 1 -- 172.21.15.102:0/1368699416 shutdown_connections 2024-03-23T03:48:11.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.865+0000 7f7700200640 1 --2- 172.21.15.102:0/1368699416 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f76dc06ac00 0x7f76dc06d0c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:11.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.865+0000 7f7700200640 1 --2- 172.21.15.102:0/1368699416 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76f815ca20 0x7f76f80cc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:11.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.865+0000 7f7700200640 1 --2- 172.21.15.102:0/1368699416 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76f80cdd10 0x7f76f81f56e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:11.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.865+0000 7f7700200640 1 --2- 172.21.15.102:0/1368699416 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76f80b8330 0x7f76f80cb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:11.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.865+0000 7f7700200640 1 -- 172.21.15.102:0/1368699416 >> 172.21.15.102:0/1368699416 conn(0x7f76f80cf7d0 msgr2=0x7f76f806b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:11.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.865+0000 7f7700200640 1 -- 172.21.15.102:0/1368699416 shutdown_connections 2024-03-23T03:48:11.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:11.865+0000 7f7700200640 1 -- 172.21.15.102:0/1368699416 wait complete. 2024-03-23T03:48:11.887 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T03:48:12.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.098+0000 7effcb800640 1 Processor -- start 2024-03-23T03:48:12.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.098+0000 7effcb800640 1 -- start start 2024-03-23T03:48:12.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.100+0000 7effcb800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7effc40cf040 0x7effc41604c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:12.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.100+0000 7effcb800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7effc4070cd0 0x7effc4160a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:12.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.100+0000 7effcb800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc406d390 con 0x7effc41456a0 2024-03-23T03:48:12.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.100+0000 7effcb800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc40ad890 con 0x7effc40cf040 2024-03-23T03:48:12.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.100+0000 7effcb800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc4145a80 con 0x7effc4070cd0 2024-03-23T03:48:12.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.100+0000 7effc3e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7effc40cf040 0x7effc41604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:12.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.100+0000 7effc3e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7effc40cf040 0x7effc41604c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41610/0 (socket says 172.21.15.102:41610) 2024-03-23T03:48:12.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.100+0000 7effc3e00640 1 -- 172.21.15.102:0/1965026223 learned_addr learned my addr 172.21.15.102:0/1965026223 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:12.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.101+0000 7effc9800640 1 --2- 172.21.15.102:0/1965026223 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7effc4070cd0 0x7effc4160a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:12.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.101+0000 7effc3e00640 1 -- 172.21.15.102:0/1965026223 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7effc4070cd0 msgr2=0x7effc4160a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:12.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.101+0000 7effc3e00640 1 --2- 172.21.15.102:0/1965026223 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7effc4070cd0 0x7effc4160a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:12.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.101+0000 7effc3e00640 1 -- 172.21.15.102:0/1965026223 >> v1:172.21.15.173:6789/0 conn(0x7effc41456a0 legacy=0x7effc40ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:12.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.101+0000 7effc3e00640 1 -- 172.21.15.102:0/1965026223 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc406d670 con 0x7effc40cf040 2024-03-23T03:48:12.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.102+0000 7effc3e00640 1 --2- 172.21.15.102:0/1965026223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc40cf040 0x7effc41604c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7effc40ae2e0 tx=0x7effb002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab888292704972fd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:12.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.102+0000 7effc3400640 1 -- 172.21.15.102:0/1965026223 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0031830 con 0x7effc40cf040 2024-03-23T03:48:12.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.102+0000 7effc3400640 1 -- 172.21.15.102:0/1965026223 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb0031990 con 0x7effc40cf040 2024-03-23T03:48:12.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.102+0000 7effc3400640 1 -- 172.21.15.102:0/1965026223 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0031d20 con 0x7effc40cf040 2024-03-23T03:48:12.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.103+0000 7effcb800640 1 -- 172.21.15.102:0/1965026223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc40cf040 msgr2=0x7effc41604c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:12.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.103+0000 7effcb800640 1 --2- 172.21.15.102:0/1965026223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc40cf040 0x7effc41604c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7effc40ae2e0 tx=0x7effb002b8d0 comp rx=0 tx=0).stop 2024-03-23T03:48:12.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.103+0000 7effcb800640 1 -- 172.21.15.102:0/1965026223 shutdown_connections 2024-03-23T03:48:12.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.103+0000 7effcb800640 1 --2- 172.21.15.102:0/1965026223 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7effc4070cd0 0x7effc4160a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:12.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.103+0000 7effcb800640 1 --2- 172.21.15.102:0/1965026223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc40cf040 0x7effc41604c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:12.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.103+0000 7effcb800640 1 -- 172.21.15.102:0/1965026223 >> 172.21.15.102:0/1965026223 conn(0x7effc406a0b0 msgr2=0x7effc406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:12.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.103+0000 7effcb800640 1 -- 172.21.15.102:0/1965026223 shutdown_connections 2024-03-23T03:48:12.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.103+0000 7effcb800640 1 -- 172.21.15.102:0/1965026223 wait complete. 2024-03-23T03:48:12.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.105+0000 7effcb800640 1 Processor -- start 2024-03-23T03:48:12.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.105+0000 7effcb800640 1 -- start start 2024-03-23T03:48:12.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.106+0000 7effcb800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4070cd0 0x7effc415eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:12.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.106+0000 7effcb800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effc40cf040 0x7effc415bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:12.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.106+0000 7effcb800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7effc41456a0 0x7effc40c3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:12.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.106+0000 7effcb800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc415f8a0 con 0x7effc4070cd0 2024-03-23T03:48:12.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.106+0000 7effcb800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc415c260 con 0x7effc40cf040 2024-03-23T03:48:12.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.106+0000 7effcb800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc415ea30 con 0x7effc41456a0 2024-03-23T03:48:12.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.107+0000 7effc8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4070cd0 0x7effc415eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:12.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.107+0000 7effc8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4070cd0 0x7effc415eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41624/0 (socket says 172.21.15.102:41624) 2024-03-23T03:48:12.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.107+0000 7effc8e00640 1 -- 172.21.15.102:0/3855916402 learned_addr learned my addr 172.21.15.102:0/3855916402 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:12.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.107+0000 7effc3e00640 1 --2- 172.21.15.102:0/3855916402 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effc40cf040 0x7effc415bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:12.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.107+0000 7effc8e00640 1 -- 172.21.15.102:0/3855916402 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7effc41456a0 msgr2=0x7effc40c3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:12.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.107+0000 7effc8e00640 1 --2- 172.21.15.102:0/3855916402 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7effc41456a0 0x7effc40c3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:12.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.107+0000 7effc8e00640 1 -- 172.21.15.102:0/3855916402 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effc40cf040 msgr2=0x7effc415bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:12.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.107+0000 7effc8e00640 1 --2- 172.21.15.102:0/3855916402 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effc40cf040 0x7effc415bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:12.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.107+0000 7effc8e00640 1 -- 172.21.15.102:0/3855916402 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb00094d0 con 0x7effc4070cd0 2024-03-23T03:48:12.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.107+0000 7effc8e00640 1 --2- 172.21.15.102:0/3855916402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4070cd0 0x7effc415eec0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7effb400b3f0 tx=0x7effb400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:12.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.108+0000 7effc0c00640 1 -- 172.21.15.102:0/3855916402 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb4011830 con 0x7effc4070cd0 2024-03-23T03:48:12.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.108+0000 7effc0c00640 1 -- 172.21.15.102:0/3855916402 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb4011990 con 0x7effc4070cd0 2024-03-23T03:48:12.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.108+0000 7effc0c00640 1 -- 172.21.15.102:0/3855916402 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb4011d40 con 0x7effc4070cd0 2024-03-23T03:48:12.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.108+0000 7effcb800640 1 -- 172.21.15.102:0/3855916402 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc40c3a70 con 0x7effc4070cd0 2024-03-23T03:48:12.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.108+0000 7effcb800640 1 -- 172.21.15.102:0/3855916402 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc415fa80 con 0x7effc4070cd0 2024-03-23T03:48:12.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.109+0000 7effcb800640 1 -- 172.21.15.102:0/3855916402 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc4073390 con 0x7effc4070cd0 2024-03-23T03:48:12.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.110+0000 7effc0c00640 1 -- 172.21.15.102:0/3855916402 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7effb401a400 con 0x7effc4070cd0 2024-03-23T03:48:12.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.111+0000 7effc0c00640 1 --2- 172.21.15.102:0/3855916402 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7effa406ac30 0x7effa406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:12.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.111+0000 7effc0c00640 1 -- 172.21.15.102:0/3855916402 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6749+0+0 (secure 0 0 0) 0x7effb4089840 con 0x7effc4070cd0 2024-03-23T03:48:12.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.111+0000 7effc3e00640 1 --2- 172.21.15.102:0/3855916402 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7effa406ac30 0x7effa406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:12.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.112+0000 7effc3e00640 1 --2- 172.21.15.102:0/3855916402 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7effa406ac30 0x7effa406d0f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7effb0036000 tx=0x7effb003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:12.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.115+0000 7effc0c00640 1 -- 172.21.15.102:0/3855916402 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7effb4054600 con 0x7effc4070cd0 2024-03-23T03:48:12.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.309+0000 7effcb800640 1 -- 172.21.15.102:0/3855916402 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7effc406cbc0 con 0x7effc4070cd0 2024-03-23T03:48:12.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.865+0000 7effc0c00640 1 -- 172.21.15.102:0/3855916402 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v30) v1 ==== 181+0+0 (secure 0 0 0) 0x7effb4057c20 con 0x7effc4070cd0 2024-03-23T03:48:12.866 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' removed 2024-03-23T03:48:12.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.871+0000 7effcb800640 1 -- 172.21.15.102:0/3855916402 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7effa406ac30 msgr2=0x7effa406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:12.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.871+0000 7effcb800640 1 --2- 172.21.15.102:0/3855916402 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7effa406ac30 0x7effa406d0f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7effb0036000 tx=0x7effb003c040 comp rx=0 tx=0).stop 2024-03-23T03:48:12.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.871+0000 7effcb800640 1 -- 172.21.15.102:0/3855916402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4070cd0 msgr2=0x7effc415eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:12.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.871+0000 7effcb800640 1 --2- 172.21.15.102:0/3855916402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4070cd0 0x7effc415eec0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7effb400b3f0 tx=0x7effb400b8b0 comp rx=0 tx=0).stop 2024-03-23T03:48:12.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.872+0000 7effcb800640 1 -- 172.21.15.102:0/3855916402 shutdown_connections 2024-03-23T03:48:12.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.872+0000 7effcb800640 1 --2- 172.21.15.102:0/3855916402 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7effa406ac30 0x7effa406d0f0 secure :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7effb0036000 tx=0x7effb003c040 comp rx=0 tx=0).stop 2024-03-23T03:48:12.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.872+0000 7effcb800640 1 --2- 172.21.15.102:0/3855916402 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7effc41456a0 0x7effc40c3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:12.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.872+0000 7effcb800640 1 --2- 172.21.15.102:0/3855916402 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7effc40cf040 0x7effc415bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:12.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.872+0000 7effcb800640 1 --2- 172.21.15.102:0/3855916402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effc4070cd0 0x7effc415eec0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:12.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.872+0000 7effcb800640 1 -- 172.21.15.102:0/3855916402 >> 172.21.15.102:0/3855916402 conn(0x7effc406a0b0 msgr2=0x7effc41422b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:12.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.874+0000 7effcb800640 1 -- 172.21.15.102:0/3855916402 shutdown_connections 2024-03-23T03:48:12.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:12.874+0000 7effcb800640 1 -- 172.21.15.102:0/3855916402 wait complete. 2024-03-23T03:48:12.899 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T03:48:13.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.094+0000 7fa970400640 1 Processor -- start 2024-03-23T03:48:13.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.094+0000 7fa970400640 1 -- start start 2024-03-23T03:48:13.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.096+0000 7fa970400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa96815cb60 0x7fa96806f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:13.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.096+0000 7fa970400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa9680b3aa0 0x7fa96815cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:13.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.096+0000 7fa970400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9680cebc0 con 0x7fa96815d480 2024-03-23T03:48:13.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.096+0000 7fa970400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9680b22b0 con 0x7fa9680b3aa0 2024-03-23T03:48:13.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.096+0000 7fa970400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa96815d8f0 con 0x7fa96815cb60 2024-03-23T03:48:13.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.096+0000 7fa96d000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa9680b3aa0 0x7fa96815cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:13.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.096+0000 7fa96d000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa9680b3aa0 0x7fa96815cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43380/0 (socket says 172.21.15.102:43380) 2024-03-23T03:48:13.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.096+0000 7fa96d000640 1 -- 172.21.15.102:0/1902520436 learned_addr learned my addr 172.21.15.102:0/1902520436 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:13.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.096+0000 7fa96da00640 1 --2- 172.21.15.102:0/1902520436 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa96815cb60 0x7fa96806f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:13.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.097+0000 7fa96d000640 1 -- 172.21.15.102:0/1902520436 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa96815cb60 msgr2=0x7fa96806f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:13.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.097+0000 7fa96d000640 1 --2- 172.21.15.102:0/1902520436 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa96815cb60 0x7fa96806f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:13.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.097+0000 7fa96d000640 1 -- 172.21.15.102:0/1902520436 >> v1:172.21.15.173:6789/0 conn(0x7fa96815d480 legacy=0x7fa9680cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:13.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.097+0000 7fa96d000640 1 -- 172.21.15.102:0/1902520436 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9680ce600 con 0x7fa9680b3aa0 2024-03-23T03:48:13.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.098+0000 7fa96d000640 1 --2- 172.21.15.102:0/1902520436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9680b3aa0 0x7fa96815cf40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa954009870 tx=0x7fa95402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f72dab65d1efe2a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:13.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.098+0000 7fa967e00640 1 -- 172.21.15.102:0/1902520436 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa954030840 con 0x7fa9680b3aa0 2024-03-23T03:48:13.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.098+0000 7fa967e00640 1 -- 172.21.15.102:0/1902520436 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9540309a0 con 0x7fa9680b3aa0 2024-03-23T03:48:13.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.098+0000 7fa967e00640 1 -- 172.21.15.102:0/1902520436 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa954030d30 con 0x7fa9680b3aa0 2024-03-23T03:48:13.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.099+0000 7fa970400640 1 -- 172.21.15.102:0/1902520436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9680b3aa0 msgr2=0x7fa96815cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:13.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.099+0000 7fa970400640 1 --2- 172.21.15.102:0/1902520436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9680b3aa0 0x7fa96815cf40 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa954009870 tx=0x7fa95402b980 comp rx=0 tx=0).stop 2024-03-23T03:48:13.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.099+0000 7fa970400640 1 -- 172.21.15.102:0/1902520436 shutdown_connections 2024-03-23T03:48:13.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.099+0000 7fa970400640 1 --2- 172.21.15.102:0/1902520436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9680b3aa0 0x7fa96815cf40 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:13.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.099+0000 7fa970400640 1 --2- 172.21.15.102:0/1902520436 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa96815cb60 0x7fa96806f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:13.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.099+0000 7fa970400640 1 -- 172.21.15.102:0/1902520436 >> 172.21.15.102:0/1902520436 conn(0x7fa9680ced50 msgr2=0x7fa9680cf130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:13.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.099+0000 7fa970400640 1 -- 172.21.15.102:0/1902520436 shutdown_connections 2024-03-23T03:48:13.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.099+0000 7fa970400640 1 -- 172.21.15.102:0/1902520436 wait complete. 2024-03-23T03:48:13.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.101+0000 7fa970400640 1 Processor -- start 2024-03-23T03:48:13.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.101+0000 7fa970400640 1 -- start start 2024-03-23T03:48:13.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.102+0000 7fa970400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9680b3aa0 0x7fa968143940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:13.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.102+0000 7fa970400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa96815cb60 0x7fa9680afc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:13.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.102+0000 7fa970400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa96815d480 0x7fa9680b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:13.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.102+0000 7fa970400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa968156c20 con 0x7fa96815d480 2024-03-23T03:48:13.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.102+0000 7fa970400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa968156380 con 0x7fa9680b3aa0 2024-03-23T03:48:13.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.102+0000 7fa970400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa968156660 con 0x7fa96815cb60 2024-03-23T03:48:13.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.102+0000 7fa96e400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa96815d480 0x7fa9680b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:13.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.102+0000 7fa96e400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa96815d480 0x7fa9680b0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43396/0 (socket says 172.21.15.102:43396) 2024-03-23T03:48:13.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.102+0000 7fa96e400640 1 -- 172.21.15.102:0/3778710355 learned_addr learned my addr 172.21.15.102:0/3778710355 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:13.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.102+0000 7fa96d000640 1 --2- 172.21.15.102:0/3778710355 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa96815cb60 0x7fa9680afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:13.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.102+0000 7fa96da00640 1 --2- 172.21.15.102:0/3778710355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9680b3aa0 0x7fa968143940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:13.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.103+0000 7fa96e400640 1 -- 172.21.15.102:0/3778710355 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa96815cb60 msgr2=0x7fa9680afc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:13.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.103+0000 7fa96e400640 1 --2- 172.21.15.102:0/3778710355 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa96815cb60 0x7fa9680afc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:13.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.103+0000 7fa96e400640 1 -- 172.21.15.102:0/3778710355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9680b3aa0 msgr2=0x7fa968143940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:13.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.103+0000 7fa96e400640 1 --2- 172.21.15.102:0/3778710355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9680b3aa0 0x7fa968143940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:13.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.103+0000 7fa96e400640 1 -- 172.21.15.102:0/3778710355 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9540094d0 con 0x7fa96815d480 2024-03-23T03:48:13.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.103+0000 7fa96da00640 1 --2- 172.21.15.102:0/3778710355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9680b3aa0 0x7fa968143940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:13.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.103+0000 7fa96d000640 1 --2- 172.21.15.102:0/3778710355 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa96815cb60 0x7fa9680afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:13.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.103+0000 7fa96e400640 1 --2- 172.21.15.102:0/3778710355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa96815d480 0x7fa9680b0140 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa95800c390 tx=0x7fa95800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:13.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.103+0000 7fa965600640 1 -- 172.21.15.102:0/3778710355 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9580052d0 con 0x7fa96815d480 2024-03-23T03:48:13.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.103+0000 7fa965600640 1 -- 172.21.15.102:0/3778710355 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa958005430 con 0x7fa96815d480 2024-03-23T03:48:13.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.103+0000 7fa970400640 1 -- 172.21.15.102:0/3778710355 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa968156940 con 0x7fa96815d480 2024-03-23T03:48:13.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.103+0000 7fa965600640 1 -- 172.21.15.102:0/3778710355 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa958003810 con 0x7fa96815d480 2024-03-23T03:48:13.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.103+0000 7fa970400640 1 -- 172.21.15.102:0/3778710355 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa968143e80 con 0x7fa96815d480 2024-03-23T03:48:13.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.104+0000 7fa970400640 1 -- 172.21.15.102:0/3778710355 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa968066740 con 0x7fa96815d480 2024-03-23T03:48:13.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.105+0000 7fa965600640 1 -- 172.21.15.102:0/3778710355 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa958003970 con 0x7fa96815d480 2024-03-23T03:48:13.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.106+0000 7fa965600640 1 --2- 172.21.15.102:0/3778710355 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa94806ac30 0x7fa94806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:13.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.106+0000 7fa965600640 1 -- 172.21.15.102:0/3778710355 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6311+0+0 (secure 0 0 0) 0x7fa9580889f0 con 0x7fa96815d480 2024-03-23T03:48:13.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.107+0000 7fa96da00640 1 --2- 172.21.15.102:0/3778710355 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa94806ac30 0x7fa94806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:13.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.107+0000 7fa96da00640 1 --2- 172.21.15.102:0/3778710355 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa94806ac30 0x7fa94806d0f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa960007330 tx=0x7fa96000b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:13.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.111+0000 7fa965600640 1 -- 172.21.15.102:0/3778710355 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa958053a00 con 0x7fa96815d480 2024-03-23T03:48:13.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.303+0000 7fa970400640 1 -- 172.21.15.102:0/3778710355 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa968066bb0 con 0x7fa96815d480 2024-03-23T03:48:13.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.869+0000 7fa965600640 1 -- 172.21.15.102:0/3778710355 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v31) v1 ==== 169+0+0 (secure 0 0 0) 0x7fa9580170d0 con 0x7fa96815d480 2024-03-23T03:48:13.870 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' removed 2024-03-23T03:48:13.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.874+0000 7fa970400640 1 -- 172.21.15.102:0/3778710355 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa94806ac30 msgr2=0x7fa94806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:13.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.874+0000 7fa970400640 1 --2- 172.21.15.102:0/3778710355 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa94806ac30 0x7fa94806d0f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa960007330 tx=0x7fa96000b040 comp rx=0 tx=0).stop 2024-03-23T03:48:13.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.874+0000 7fa970400640 1 -- 172.21.15.102:0/3778710355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa96815d480 msgr2=0x7fa9680b0140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:13.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.874+0000 7fa970400640 1 --2- 172.21.15.102:0/3778710355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa96815d480 0x7fa9680b0140 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa95800c390 tx=0x7fa95800c850 comp rx=0 tx=0).stop 2024-03-23T03:48:13.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.876+0000 7fa970400640 1 -- 172.21.15.102:0/3778710355 shutdown_connections 2024-03-23T03:48:13.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.876+0000 7fa970400640 1 --2- 172.21.15.102:0/3778710355 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa94806ac30 0x7fa94806d0f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:13.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.876+0000 7fa970400640 1 --2- 172.21.15.102:0/3778710355 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa96815d480 0x7fa9680b0140 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:13.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.876+0000 7fa970400640 1 --2- 172.21.15.102:0/3778710355 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa96815cb60 0x7fa9680afc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:13.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.876+0000 7fa970400640 1 --2- 172.21.15.102:0/3778710355 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9680b3aa0 0x7fa968143940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:13.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.876+0000 7fa970400640 1 -- 172.21.15.102:0/3778710355 >> 172.21.15.102:0/3778710355 conn(0x7fa9680ced50 msgr2=0x7fa9680caf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:13.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.876+0000 7fa970400640 1 -- 172.21.15.102:0/3778710355 shutdown_connections 2024-03-23T03:48:13.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:13.877+0000 7fa970400640 1 -- 172.21.15.102:0/3778710355 wait complete. 2024-03-23T03:48:13.901 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-23T03:48:14.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.106+0000 7f86bce00640 1 Processor -- start 2024-03-23T03:48:14.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.106+0000 7f86bce00640 1 -- start start 2024-03-23T03:48:14.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.107+0000 7f86bce00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f86b8159830 0x7f86b80cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:14.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.107+0000 7f86bce00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f86b80ce930 0x7f86b80cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:14.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.107+0000 7f86bce00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b8159c10 con 0x7f86b80ca7b0 2024-03-23T03:48:14.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.108+0000 7f86bce00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86b806de50 con 0x7f86b80ce930 2024-03-23T03:48:14.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.108+0000 7f86bce00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b80cf300 con 0x7f86b8159830 2024-03-23T03:48:14.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.108+0000 7f86b6800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f86b80ce930 0x7f86b80cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:14.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.108+0000 7f86b6800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f86b80ce930 0x7f86b80cedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43398/0 (socket says 172.21.15.102:43398) 2024-03-23T03:48:14.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.108+0000 7f86b6800640 1 -- 172.21.15.102:0/81901520 learned_addr learned my addr 172.21.15.102:0/81901520 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:14.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.108+0000 7f86b6800640 1 -- 172.21.15.102:0/81901520 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f86b8159830 msgr2=0x7f86b80cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:14.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.108+0000 7f86b6800640 1 --2- 172.21.15.102:0/81901520 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f86b8159830 0x7f86b80cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:14.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.108+0000 7f86b6800640 1 -- 172.21.15.102:0/81901520 >> v1:172.21.15.173:6789/0 conn(0x7f86b80ca7b0 legacy=0x7f86b80cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:14.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.108+0000 7f86b6800640 1 -- 172.21.15.102:0/81901520 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b80cd840 con 0x7f86b80ce930 2024-03-23T03:48:14.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.109+0000 7f86b6800640 1 --2- 172.21.15.102:0/81901520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86b80ce930 0x7f86b80cedc0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f86b0009e50 tx=0x7f86b002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41f8523f241527b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:14.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.110+0000 7f86b5400640 1 -- 172.21.15.102:0/81901520 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b002cb00 con 0x7f86b80ce930 2024-03-23T03:48:14.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.110+0000 7f86b5400640 1 -- 172.21.15.102:0/81901520 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b002cc60 con 0x7f86b80ce930 2024-03-23T03:48:14.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.110+0000 7f86b5400640 1 -- 172.21.15.102:0/81901520 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b00343f0 con 0x7f86b80ce930 2024-03-23T03:48:14.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.110+0000 7f86bce00640 1 -- 172.21.15.102:0/81901520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86b80ce930 msgr2=0x7f86b80cedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:14.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.110+0000 7f86bce00640 1 --2- 172.21.15.102:0/81901520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86b80ce930 0x7f86b80cedc0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f86b0009e50 tx=0x7f86b002bf60 comp rx=0 tx=0).stop 2024-03-23T03:48:14.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.111+0000 7f86bce00640 1 -- 172.21.15.102:0/81901520 shutdown_connections 2024-03-23T03:48:14.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.111+0000 7f86bce00640 1 --2- 172.21.15.102:0/81901520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86b80ce930 0x7f86b80cedc0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:14.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.111+0000 7f86bce00640 1 --2- 172.21.15.102:0/81901520 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f86b8159830 0x7f86b80cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:14.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.111+0000 7f86bce00640 1 -- 172.21.15.102:0/81901520 >> 172.21.15.102:0/81901520 conn(0x7f86b806a0b0 msgr2=0x7f86b806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:14.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.111+0000 7f86bce00640 1 -- 172.21.15.102:0/81901520 shutdown_connections 2024-03-23T03:48:14.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.111+0000 7f86bce00640 1 -- 172.21.15.102:0/81901520 wait complete. 2024-03-23T03:48:14.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.112+0000 7f86bce00640 1 Processor -- start 2024-03-23T03:48:14.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.112+0000 7f86bce00640 1 -- start start 2024-03-23T03:48:14.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.113+0000 7f86bce00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86b80ca7b0 0x7f86b8141f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:14.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.113+0000 7f86bce00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f86b80ce930 0x7f86b814e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:14.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.113+0000 7f86bce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86b8159830 0x7f86b815bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:14.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.113+0000 7f86bce00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b814eb70 con 0x7f86b8159830 2024-03-23T03:48:14.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.113+0000 7f86bce00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b814ee50 con 0x7f86b80ca7b0 2024-03-23T03:48:14.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.113+0000 7f86bce00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b814f130 con 0x7f86b80ce930 2024-03-23T03:48:14.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.114+0000 7f86b6800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86b8159830 0x7f86b815bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:14.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.114+0000 7f86b6800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86b8159830 0x7f86b815bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43410/0 (socket says 172.21.15.102:43410) 2024-03-23T03:48:14.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.114+0000 7f86b6800640 1 -- 172.21.15.102:0/206597407 learned_addr learned my addr 172.21.15.102:0/206597407 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:14.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.114+0000 7f86b6800640 1 -- 172.21.15.102:0/206597407 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f86b80ce930 msgr2=0x7f86b814e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:14.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.114+0000 7f86b6800640 1 --2- 172.21.15.102:0/206597407 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f86b80ce930 0x7f86b814e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:14.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.114+0000 7f86b6800640 1 -- 172.21.15.102:0/206597407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86b80ca7b0 msgr2=0x7f86b8141f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:14.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.114+0000 7f86b6800640 1 --2- 172.21.15.102:0/206597407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86b80ca7b0 0x7f86b8141f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:14.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.114+0000 7f86b6800640 1 -- 172.21.15.102:0/206597407 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b0009b30 con 0x7f86b8159830 2024-03-23T03:48:14.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.114+0000 7f86b6800640 1 --2- 172.21.15.102:0/206597407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86b8159830 0x7f86b815bfe0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f86b0036040 tx=0x7f86b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:14.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.115+0000 7f86ae000640 1 -- 172.21.15.102:0/206597407 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b003e450 con 0x7f86b8159830 2024-03-23T03:48:14.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.115+0000 7f86ae000640 1 -- 172.21.15.102:0/206597407 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b003e5d0 con 0x7f86b8159830 2024-03-23T03:48:14.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.115+0000 7f86ae000640 1 -- 172.21.15.102:0/206597407 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b003e980 con 0x7f86b8159830 2024-03-23T03:48:14.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.115+0000 7f86bce00640 1 -- 172.21.15.102:0/206597407 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b814f410 con 0x7f86b8159830 2024-03-23T03:48:14.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.115+0000 7f86bce00640 1 -- 172.21.15.102:0/206597407 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b8143830 con 0x7f86b8159830 2024-03-23T03:48:14.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.116+0000 7f86bce00640 1 -- 172.21.15.102:0/206597407 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b814f930 con 0x7f86b8159830 2024-03-23T03:48:14.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.117+0000 7f86ae000640 1 -- 172.21.15.102:0/206597407 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f86b003eae0 con 0x7f86b8159830 2024-03-23T03:48:14.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.118+0000 7f86ae000640 1 --2- 172.21.15.102:0/206597407 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f869406ac30 0x7f869406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:14.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.118+0000 7f86ae000640 1 -- 172.21.15.102:0/206597407 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f86b00abcb0 con 0x7f86b8159830 2024-03-23T03:48:14.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.118+0000 7f86b5e00640 1 --2- 172.21.15.102:0/206597407 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f869406ac30 0x7f869406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:14.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.118+0000 7f86b5e00640 1 --2- 172.21.15.102:0/206597407 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f869406ac30 0x7f869406d0f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f86a8009670 tx=0x7f86a8006bf0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:14.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.123+0000 7f86ae000640 1 -- 172.21.15.102:0/206597407 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f86b814f930 con 0x7f86b8159830 2024-03-23T03:48:14.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.317+0000 7f86bce00640 1 -- 172.21.15.102:0/206597407 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f86b814f930 con 0x7f86b8159830 2024-03-23T03:48:14.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.883+0000 7f86ae000640 1 -- 172.21.15.102:0/206597407 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v32) v1 ==== 178+0+0 (secure 0 0 0) 0x7f86b814f930 con 0x7f86b8159830 2024-03-23T03:48:14.884 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data_ec' removed 2024-03-23T03:48:14.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.889+0000 7f86bce00640 1 -- 172.21.15.102:0/206597407 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f869406ac30 msgr2=0x7f869406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:14.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.889+0000 7f86bce00640 1 --2- 172.21.15.102:0/206597407 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f869406ac30 0x7f869406d0f0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f86a8009670 tx=0x7f86a8006bf0 comp rx=0 tx=0).stop 2024-03-23T03:48:14.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.889+0000 7f86bce00640 1 -- 172.21.15.102:0/206597407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86b8159830 msgr2=0x7f86b815bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:14.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.889+0000 7f86bce00640 1 --2- 172.21.15.102:0/206597407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86b8159830 0x7f86b815bfe0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f86b0036040 tx=0x7f86b0004a40 comp rx=0 tx=0).stop 2024-03-23T03:48:14.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.892+0000 7f86bce00640 1 -- 172.21.15.102:0/206597407 shutdown_connections 2024-03-23T03:48:14.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.892+0000 7f86bce00640 1 --2- 172.21.15.102:0/206597407 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f869406ac30 0x7f869406d0f0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:14.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.892+0000 7f86bce00640 1 --2- 172.21.15.102:0/206597407 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86b8159830 0x7f86b815bfe0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:14.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.892+0000 7f86bce00640 1 --2- 172.21.15.102:0/206597407 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f86b80ce930 0x7f86b814e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:14.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.892+0000 7f86bce00640 1 --2- 172.21.15.102:0/206597407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86b80ca7b0 0x7f86b8141f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:14.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.892+0000 7f86bce00640 1 -- 172.21.15.102:0/206597407 >> 172.21.15.102:0/206597407 conn(0x7f86b806a0b0 msgr2=0x7f86b80cc280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:14.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.895+0000 7f86bce00640 1 -- 172.21.15.102:0/206597407 shutdown_connections 2024-03-23T03:48:14.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:14.895+0000 7f86bce00640 1 -- 172.21.15.102:0/206597407 wait complete. 2024-03-23T03:48:14.925 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T03:48:14.925 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-23T03:48:14.925 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T03:48:14.925 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:48:14.925 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T03:48:14.926 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-23T03:48:14.926 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T03:48:14.926 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:48:14.926 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T03:48:14.926 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-23T03:48:14.926 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T03:48:14.926 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:48:14.926 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T03:48:14.927 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-23T03:48:14.927 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T03:48:14.927 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:48:14.945 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T03:48:14.945 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T03:48:14.947 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T03:48:14.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T03:48:14.947 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T03:48:14.948 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T03:48:14.953 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T03:48:14.953 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T03:48:14.994 INFO:tasks.ceph.mds.a:Started 2024-03-23T03:48:14.994 DEBUG:teuthology.parallel:result is None 2024-03-23T03:48:15.000 INFO:tasks.ceph.mds.b:Started 2024-03-23T03:48:15.000 DEBUG:teuthology.parallel:result is None 2024-03-23T03:48:15.001 INFO:tasks.ceph.mds.c:Started 2024-03-23T03:48:15.001 DEBUG:teuthology.parallel:result is None 2024-03-23T03:48:15.006 INFO:tasks.ceph.mds.d:Started 2024-03-23T03:48:15.006 DEBUG:teuthology.parallel:result is None 2024-03-23T03:48:15.007 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-23T03:48:15.148 INFO:tasks.ceph.mds.a.smithi102.stdout:starting mds.a at 2024-03-23T03:48:15.149 INFO:tasks.ceph.mds.c.smithi102.stdout:starting mds.c at 2024-03-23T03:48:15.170 INFO:tasks.ceph.mds.d.smithi173.stdout:starting mds.d at 2024-03-23T03:48:15.172 INFO:tasks.ceph.mds.b.smithi173.stdout:starting mds.b at 2024-03-23T03:48:15.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.196+0000 7fb030800640 1 Processor -- start 2024-03-23T03:48:15.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.196+0000 7fb030800640 1 -- start start 2024-03-23T03:48:15.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.197+0000 7fb030800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb02815cd90 0x7fb0280b8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:15.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.197+0000 7fb030800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb02806b840 con 0x7fb0280cda90 2024-03-23T03:48:15.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.197+0000 7fb030800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb02806bb20 con 0x7fb0280cd6b0 2024-03-23T03:48:15.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.197+0000 7fb030800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb02806c0e0 con 0x7fb02815cd90 2024-03-23T03:48:15.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.198+0000 7fb02d400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb02815cd90 0x7fb0280b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:15.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.198+0000 7fb02d400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb02815cd90 0x7fb0280b8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43450/0 (socket says 172.21.15.102:43450) 2024-03-23T03:48:15.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.198+0000 7fb02d400640 1 -- 172.21.15.102:0/693187432 learned_addr learned my addr 172.21.15.102:0/693187432 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:15.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.198+0000 7fb02d400640 1 -- 172.21.15.102:0/693187432 >> v1:172.21.15.173:6789/0 conn(0x7fb0280cd6b0 legacy=0x7fb02806b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:15.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.198+0000 7fb02d400640 1 -- 172.21.15.102:0/693187432 >> v1:172.21.15.102:6789/0 conn(0x7fb0280cda90 legacy=0x7fb0280d1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:15.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.198+0000 7fb02d400640 1 -- 172.21.15.102:0/693187432 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb02806be00 con 0x7fb02815cd90 2024-03-23T03:48:15.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.200+0000 7fb02d400640 1 --2- 172.21.15.102:0/693187432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb02815cd90 0x7fb0280b8490 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb018009870 tx=0x7fb01802ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb6d5cc8644d61d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:15.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.200+0000 7fb02ca00640 1 -- 172.21.15.102:0/693187432 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01802f840 con 0x7fb02815cd90 2024-03-23T03:48:15.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.200+0000 7fb02ca00640 1 -- 172.21.15.102:0/693187432 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01802f9a0 con 0x7fb02815cd90 2024-03-23T03:48:15.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.200+0000 7fb02ca00640 1 -- 172.21.15.102:0/693187432 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01802fd30 con 0x7fb02815cd90 2024-03-23T03:48:15.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.201+0000 7fb030800640 1 -- 172.21.15.102:0/693187432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb02815cd90 msgr2=0x7fb0280b8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:15.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.201+0000 7fb030800640 1 --2- 172.21.15.102:0/693187432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb02815cd90 0x7fb0280b8490 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fb018009870 tx=0x7fb01802ba20 comp rx=0 tx=0).stop 2024-03-23T03:48:15.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.201+0000 7fb030800640 1 -- 172.21.15.102:0/693187432 shutdown_connections 2024-03-23T03:48:15.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.201+0000 7fb030800640 1 --2- 172.21.15.102:0/693187432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb02815cd90 0x7fb0280b8490 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:15.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.201+0000 7fb030800640 1 -- 172.21.15.102:0/693187432 >> 172.21.15.102:0/693187432 conn(0x7fb0280cf780 msgr2=0x7fb0280cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:15.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.201+0000 7fb030800640 1 -- 172.21.15.102:0/693187432 shutdown_connections 2024-03-23T03:48:15.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.201+0000 7fb030800640 1 -- 172.21.15.102:0/693187432 wait complete. 2024-03-23T03:48:15.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.203+0000 7fb030800640 1 Processor -- start 2024-03-23T03:48:15.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.203+0000 7fb030800640 1 -- start start 2024-03-23T03:48:15.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.204+0000 7fb030800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0280cd6b0 0x7fb02815f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:15.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.204+0000 7fb030800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0280cda90 0x7fb0280cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:15.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.204+0000 7fb030800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb02815cd90 0x7fb0280b4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:15.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.204+0000 7fb030800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0280cc9b0 con 0x7fb02815cd90 2024-03-23T03:48:15.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.204+0000 7fb030800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02815d190 con 0x7fb0280cda90 2024-03-23T03:48:15.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.204+0000 7fb030800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb02815f7e0 con 0x7fb0280cd6b0 2024-03-23T03:48:15.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.204+0000 7fb02e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb02815cd90 0x7fb0280b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:15.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.204+0000 7fb02e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb02815cd90 0x7fb0280b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43456/0 (socket says 172.21.15.102:43456) 2024-03-23T03:48:15.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.204+0000 7fb02e800640 1 -- 172.21.15.102:0/3454593445 learned_addr learned my addr 172.21.15.102:0/3454593445 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:15.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.204+0000 7fb02de00640 1 --2- 172.21.15.102:0/3454593445 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0280cd6b0 0x7fb02815f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:15.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.204+0000 7fb02d400640 1 --2- 172.21.15.102:0/3454593445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0280cda90 0x7fb0280cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:15.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.205+0000 7fb02e800640 1 -- 172.21.15.102:0/3454593445 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0280cd6b0 msgr2=0x7fb02815f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:15.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.205+0000 7fb02e800640 1 --2- 172.21.15.102:0/3454593445 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0280cd6b0 0x7fb02815f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:15.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.205+0000 7fb02e800640 1 -- 172.21.15.102:0/3454593445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0280cda90 msgr2=0x7fb0280cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:15.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.205+0000 7fb02e800640 1 --2- 172.21.15.102:0/3454593445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0280cda90 0x7fb0280cc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:15.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.205+0000 7fb02e800640 1 -- 172.21.15.102:0/3454593445 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0180094d0 con 0x7fb02815cd90 2024-03-23T03:48:15.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.205+0000 7fb02e800640 1 --2- 172.21.15.102:0/3454593445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb02815cd90 0x7fb0280b4900 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb02400c7a0 tx=0x7fb02400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:15.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.205+0000 7fb01e000640 1 -- 172.21.15.102:0/3454593445 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb024003b70 con 0x7fb02815cd90 2024-03-23T03:48:15.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.205+0000 7fb01e000640 1 -- 172.21.15.102:0/3454593445 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb024003cf0 con 0x7fb02815cd90 2024-03-23T03:48:15.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.205+0000 7fb01e000640 1 -- 172.21.15.102:0/3454593445 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb024003ea0 con 0x7fb02815cd90 2024-03-23T03:48:15.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.205+0000 7fb030800640 1 -- 172.21.15.102:0/3454593445 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb02815fac0 con 0x7fb02815cd90 2024-03-23T03:48:15.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.206+0000 7fb030800640 1 -- 172.21.15.102:0/3454593445 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb02815d450 con 0x7fb02815cd90 2024-03-23T03:48:15.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.207+0000 7fb030800640 1 -- 172.21.15.102:0/3454593445 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb028073390 con 0x7fb02815cd90 2024-03-23T03:48:15.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.207+0000 7fb01e000640 1 -- 172.21.15.102:0/3454593445 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb02400aa90 con 0x7fb02815cd90 2024-03-23T03:48:15.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.208+0000 7fb01e000640 1 --2- 172.21.15.102:0/3454593445 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb00c06ac00 0x7fb00c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:15.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.208+0000 7fb01e000640 1 -- 172.21.15.102:0/3454593445 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fb024088ed0 con 0x7fb02815cd90 2024-03-23T03:48:15.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.209+0000 7fb02de00640 1 --2- 172.21.15.102:0/3454593445 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb00c06ac00 0x7fb00c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:15.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.209+0000 7fb02de00640 1 --2- 172.21.15.102:0/3454593445 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb00c06ac00 0x7fb00c06d0c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb020000f60 tx=0x7fb020009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:15.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.213+0000 7fb01e000640 1 -- 172.21.15.102:0/3454593445 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb024054310 con 0x7fb02815cd90 2024-03-23T03:48:15.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:15.420+0000 7fb030800640 1 -- 172.21.15.102:0/3454593445 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fb02806b160 con 0x7fb02815cd90 2024-03-23T03:48:16.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.209+0000 7fb01e000640 1 -- 172.21.15.102:0/3454593445 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v37) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb024057930 con 0x7fb02815cd90 2024-03-23T03:48:16.210 INFO:teuthology.orchestra.run.smithi102.stderr: removed all blocklist entries 2024-03-23T03:48:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.213+0000 7fb030800640 1 -- 172.21.15.102:0/3454593445 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb00c06ac00 msgr2=0x7fb00c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.213+0000 7fb030800640 1 --2- 172.21.15.102:0/3454593445 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb00c06ac00 0x7fb00c06d0c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb020000f60 tx=0x7fb020009040 comp rx=0 tx=0).stop 2024-03-23T03:48:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.213+0000 7fb030800640 1 -- 172.21.15.102:0/3454593445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb02815cd90 msgr2=0x7fb0280b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.213+0000 7fb030800640 1 --2- 172.21.15.102:0/3454593445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb02815cd90 0x7fb0280b4900 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fb02400c7a0 tx=0x7fb02400cc60 comp rx=0 tx=0).stop 2024-03-23T03:48:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.213+0000 7fb030800640 1 -- 172.21.15.102:0/3454593445 shutdown_connections 2024-03-23T03:48:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.213+0000 7fb030800640 1 --2- 172.21.15.102:0/3454593445 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb00c06ac00 0x7fb00c06d0c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.213+0000 7fb030800640 1 --2- 172.21.15.102:0/3454593445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb02815cd90 0x7fb0280b4900 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.213+0000 7fb030800640 1 --2- 172.21.15.102:0/3454593445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0280cda90 0x7fb0280cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.213+0000 7fb030800640 1 --2- 172.21.15.102:0/3454593445 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0280cd6b0 0x7fb02815f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.213+0000 7fb030800640 1 -- 172.21.15.102:0/3454593445 >> 172.21.15.102:0/3454593445 conn(0x7fb0280cf780 msgr2=0x7fb0280b68a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.213+0000 7fb030800640 1 -- 172.21.15.102:0/3454593445 shutdown_connections 2024-03-23T03:48:16.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.213+0000 7fb030800640 1 -- 172.21.15.102:0/3454593445 wait complete. 2024-03-23T03:48:16.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-23T03:48:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.434+0000 7fcc3ea00640 1 Processor -- start 2024-03-23T03:48:16.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.434+0000 7fcc3ea00640 1 -- start start 2024-03-23T03:48:16.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.435+0000 7fcc3ea00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fcc380cbab0 0x7fcc380b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:16.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.436+0000 7fcc3ea00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcc3806d390 0x7fcc3806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:16.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.436+0000 7fcc3ea00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc380b16a0 con 0x7fcc380adbc0 2024-03-23T03:48:16.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.436+0000 7fcc3ea00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc3806dd60 con 0x7fcc380cbab0 2024-03-23T03:48:16.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.436+0000 7fcc3ea00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc380cbfb0 con 0x7fcc3806d390 2024-03-23T03:48:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.436+0000 7fcc37e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fcc380adbc0 0x7fcc380b41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50364/0 (socket says 172.21.15.102:50364) 2024-03-23T03:48:16.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.436+0000 7fcc37e00640 1 -- 172.21.15.102:0/3487016204 learned_addr learned my addr 172.21.15.102:0/3487016204 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:16.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.436+0000 7fcc37400640 1 -- 172.21.15.102:0/3487016204 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216987573 0 0) 0x7fcc380b16a0 con 0x7fcc380adbc0 2024-03-23T03:48:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.436+0000 7fcc2f400640 1 --2- 172.21.15.102:0/3487016204 >> v2:172.21.15.173:3300/0 conn(0x7fcc380cbab0 0x7fcc380b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.436+0000 7fcc3ca00640 1 --2- 172.21.15.102:0/3487016204 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcc3806d390 0x7fcc3806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.436+0000 7fcc37400640 1 -- 172.21.15.102:0/3487016204 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc1c005730 con 0x7fcc380adbc0 2024-03-23T03:48:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.437+0000 7fcc37400640 1 -- 172.21.15.102:0/3487016204 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 453540003 0 0) 0x7fcc1c005730 con 0x7fcc380adbc0 2024-03-23T03:48:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.437+0000 7fcc37400640 1 -- 172.21.15.102:0/3487016204 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcc3806d390 msgr2=0x7fcc3806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.437+0000 7fcc37400640 1 --2- 172.21.15.102:0/3487016204 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcc3806d390 0x7fcc3806d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.437+0000 7fcc37400640 1 -- 172.21.15.102:0/3487016204 >> v2:172.21.15.173:3300/0 conn(0x7fcc380cbab0 msgr2=0x7fcc380b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.437+0000 7fcc37400640 1 --2- 172.21.15.102:0/3487016204 >> v2:172.21.15.173:3300/0 conn(0x7fcc380cbab0 0x7fcc380b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.437+0000 7fcc37400640 1 -- 172.21.15.102:0/3487016204 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc380cc290 con 0x7fcc380adbc0 2024-03-23T03:48:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.437+0000 7fcc37400640 1 -- 172.21.15.102:0/3487016204 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fcc28001670 con 0x7fcc380adbc0 2024-03-23T03:48:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.437+0000 7fcc37400640 1 -- 172.21.15.102:0/3487016204 >> v1:172.21.15.102:6789/0 conn(0x7fcc380adbc0 legacy=0x7fcc380b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.437+0000 7fcc37400640 1 --2- 172.21.15.102:0/3487016204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc1c005b80 0x7fcc1c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:16.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.437+0000 7fcc37400640 1 --2- 172.21.15.102:0/3487016204 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcc1c0069d0 0x7fcc1c006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.437+0000 7fcc37400640 1 --2- 172.21.15.102:0/3487016204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc1c007390 0x7fcc1c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.437+0000 7fcc37400640 1 -- 172.21.15.102:0/3487016204 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3806dd60 con 0x7fcc1c005b80 2024-03-23T03:48:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.437+0000 7fcc37400640 1 -- 172.21.15.102:0/3487016204 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc380cbfb0 con 0x7fcc1c007390 2024-03-23T03:48:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.437+0000 7fcc37400640 1 -- 172.21.15.102:0/3487016204 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1c00e2b0 con 0x7fcc1c0069d0 2024-03-23T03:48:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.437+0000 7fcc3ca00640 1 --2- 172.21.15.102:0/3487016204 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcc3806d390 0x7fcc3806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:48:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.437+0000 7fcc37e00640 1 --2- 172.21.15.102:0/3487016204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc1c005b80 0x7fcc1c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.438+0000 7fcc3ca00640 1 --2- 172.21.15.102:0/3487016204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc1c007390 0x7fcc1c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.438+0000 7fcc37e00640 1 -- 172.21.15.102:0/3487016204 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcc1c0069d0 msgr2=0x7fcc1c006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.438+0000 7fcc37e00640 1 --2- 172.21.15.102:0/3487016204 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcc1c0069d0 0x7fcc1c006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.438+0000 7fcc37e00640 1 -- 172.21.15.102:0/3487016204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc1c007390 msgr2=0x7fcc1c00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.438+0000 7fcc37e00640 1 --2- 172.21.15.102:0/3487016204 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc1c007390 0x7fcc1c00db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.438+0000 7fcc37e00640 1 -- 172.21.15.102:0/3487016204 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c005730 con 0x7fcc1c005b80 2024-03-23T03:48:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.439+0000 7fcc37e00640 1 --2- 172.21.15.102:0/3487016204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc1c005b80 0x7fcc1c005fa0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800c110 tx=0x7fcc1c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e97b429190b57430 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.439+0000 7fcc37400640 1 -- 172.21.15.102:0/3487016204 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc28005ad0 con 0x7fcc1c005b80 2024-03-23T03:48:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.439+0000 7fcc37400640 1 -- 172.21.15.102:0/3487016204 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc28005c50 con 0x7fcc1c005b80 2024-03-23T03:48:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.439+0000 7fcc37400640 1 -- 172.21.15.102:0/3487016204 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2802c830 con 0x7fcc1c005b80 2024-03-23T03:48:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.439+0000 7fcc3ea00640 1 -- 172.21.15.102:0/3487016204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc1c005b80 msgr2=0x7fcc1c005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.439+0000 7fcc3ea00640 1 --2- 172.21.15.102:0/3487016204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc1c005b80 0x7fcc1c005fa0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800c110 tx=0x7fcc1c007a00 comp rx=0 tx=0).stop 2024-03-23T03:48:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.439+0000 7fcc3ca00640 1 -- 172.21.15.102:0/3487016204 reap_dead start 2024-03-23T03:48:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.440+0000 7fcc3ea00640 1 -- 172.21.15.102:0/3487016204 shutdown_connections 2024-03-23T03:48:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.440+0000 7fcc3ea00640 1 -- 172.21.15.102:0/3487016204 >> 172.21.15.102:0/3487016204 conn(0x7fcc3806a3d0 msgr2=0x7fcc380b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.440+0000 7fcc3ea00640 1 -- 172.21.15.102:0/3487016204 shutdown_connections 2024-03-23T03:48:16.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.440+0000 7fcc3ea00640 1 -- 172.21.15.102:0/3487016204 wait complete. 2024-03-23T03:48:16.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.441+0000 7fcc3ea00640 1 Processor -- start 2024-03-23T03:48:16.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.442+0000 7fcc3ea00640 1 -- start start 2024-03-23T03:48:16.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.442+0000 7fcc3ea00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcc1c005b80 0x7fcc3815bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:16.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.442+0000 7fcc3ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc3813f680 0x7fcc38159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.443+0000 7fcc3ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc3813fb00 0x7fcc3815a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.443+0000 7fcc3ea00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3815b790 con 0x7fcc3813f680 2024-03-23T03:48:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.443+0000 7fcc3ea00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3813c530 con 0x7fcc3813fb00 2024-03-23T03:48:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.443+0000 7fcc3ea00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3815ab10 con 0x7fcc1c005b80 2024-03-23T03:48:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.443+0000 7fcc2f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc3813f680 0x7fcc38159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.443+0000 7fcc2f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc3813f680 0x7fcc38159f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43484/0 (socket says 172.21.15.102:43484) 2024-03-23T03:48:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.443+0000 7fcc2f400640 1 -- 172.21.15.102:0/3191888424 learned_addr learned my addr 172.21.15.102:0/3191888424 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.443+0000 7fcc37e00640 1 --2- 172.21.15.102:0/3191888424 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcc1c005b80 0x7fcc3815bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.443+0000 7fcc2f400640 1 -- 172.21.15.102:0/3191888424 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcc1c005b80 msgr2=0x7fcc3815bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.443+0000 7fcc3ca00640 1 --2- 172.21.15.102:0/3191888424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc3813fb00 0x7fcc3815a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:16.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.443+0000 7fcc2f400640 1 --2- 172.21.15.102:0/3191888424 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcc1c005b80 0x7fcc3815bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.443+0000 7fcc2f400640 1 -- 172.21.15.102:0/3191888424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc3813fb00 msgr2=0x7fcc3815a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.443+0000 7fcc2f400640 1 --2- 172.21.15.102:0/3191888424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc3813fb00 0x7fcc3815a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.443+0000 7fcc2f400640 1 -- 172.21.15.102:0/3191888424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3815adf0 con 0x7fcc3813f680 2024-03-23T03:48:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.444+0000 7fcc2f400640 1 --2- 172.21.15.102:0/3191888424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc3813f680 0x7fcc38159f30 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fcc2000d7d0 tx=0x7fcc2000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.444+0000 7fcc34c00640 1 -- 172.21.15.102:0/3191888424 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2000b6c0 con 0x7fcc3813f680 2024-03-23T03:48:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.444+0000 7fcc34c00640 1 -- 172.21.15.102:0/3191888424 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2000b860 con 0x7fcc3813f680 2024-03-23T03:48:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.444+0000 7fcc34c00640 1 -- 172.21.15.102:0/3191888424 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2000bc10 con 0x7fcc3813f680 2024-03-23T03:48:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.444+0000 7fcc3ea00640 1 -- 172.21.15.102:0/3191888424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3815b0d0 con 0x7fcc3813f680 2024-03-23T03:48:16.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.444+0000 7fcc3ea00640 1 -- 172.21.15.102:0/3191888424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc38158e10 con 0x7fcc3813f680 2024-03-23T03:48:16.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.445+0000 7fcc3ea00640 1 -- 172.21.15.102:0/3191888424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3815b8d0 con 0x7fcc3813f680 2024-03-23T03:48:16.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.446+0000 7fcc34c00640 1 -- 172.21.15.102:0/3191888424 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcc2001e460 con 0x7fcc3813f680 2024-03-23T03:48:16.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.447+0000 7fcc34c00640 1 --2- 172.21.15.102:0/3191888424 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcc1406ac30 0x7fcc1406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:16.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.447+0000 7fcc34c00640 1 -- 172.21.15.102:0/3191888424 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fcc20089610 con 0x7fcc3813f680 2024-03-23T03:48:16.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.447+0000 7fcc37e00640 1 --2- 172.21.15.102:0/3191888424 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcc1406ac30 0x7fcc1406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:16.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.448+0000 7fcc37e00640 1 --2- 172.21.15.102:0/3191888424 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcc1406ac30 0x7fcc1406d0f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcc28034000 tx=0x7fcc28007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:16.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.451+0000 7fcc34c00640 1 -- 172.21.15.102:0/3191888424 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcc3815b8d0 con 0x7fcc3813f680 2024-03-23T03:48:16.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.714+0000 7fcc3ea00640 1 -- 172.21.15.102:0/3191888424 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fcc3815b8d0 con 0x7fcc3813f680 2024-03-23T03:48:16.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.716+0000 7fcc34c00640 1 -- 172.21.15.102:0/3191888424 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fcc3815b8d0 con 0x7fcc3813f680 2024-03-23T03:48:16.716 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:16.716 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:48:16.717 INFO:teuthology.orchestra.run.smithi102.stdout: "auth_dump": [ 2024-03-23T03:48:16.717 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.717 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.a", 2024-03-23T03:48:16.717 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5loBTZFRAA7/tTYyZwrDmwKwiU+dtCEw==", 2024-03-23T03:48:16.717 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.717 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:48:16.717 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:48:16.717 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:48:16.717 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:16.717 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.717 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.717 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.717 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.b", 2024-03-23T03:48:16.717 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5lJKKlNxAAj5+j99K79IhbeEm7/qC+wg==", 2024-03-23T03:48:16.718 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.718 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:48:16.718 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:48:16.718 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:48:16.718 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:16.718 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.718 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.718 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.718 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.c", 2024-03-23T03:48:16.718 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5l0b92JhAA5rvs+mdgmrQX/9lI3GW2rg==", 2024-03-23T03:48:16.718 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.718 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:48:16.718 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:48:16.719 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:48:16.719 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:16.719 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.719 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.719 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.719 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.d", 2024-03-23T03:48:16.719 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5ltjNMDBAAD1h5Nlv2Cp7mJt+DMlBtzw==", 2024-03-23T03:48:16.719 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.719 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:48:16.719 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:48:16.719 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:48:16.719 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:16.719 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.719 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.720 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.720 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.0", 2024-03-23T03:48:16.720 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB2UP5lPW6uORAAejiQReOF95oFliFmk/vdQQ==", 2024-03-23T03:48:16.720 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.720 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:48:16.720 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:48:16.720 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:16.720 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.720 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.720 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.720 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.1", 2024-03-23T03:48:16.720 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB6UP5lFbbUBxAAIl0ZEYtJ+/oi3W6a4OwMhg==", 2024-03-23T03:48:16.720 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.720 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:48:16.720 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:48:16.721 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:16.721 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.721 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.721 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.721 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.2", 2024-03-23T03:48:16.721 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB9UP5l2LaQIBAAKJv7jWJAjlUJrm7+KTXi6Q==", 2024-03-23T03:48:16.721 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.721 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:48:16.721 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:48:16.721 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:16.721 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.721 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.721 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.721 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.3", 2024-03-23T03:48:16.721 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCAUP5lxjWsKhAAzVO8CXUd+4LLvv4f/5xMOg==", 2024-03-23T03:48:16.722 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.722 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:48:16.722 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:48:16.722 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:16.722 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.722 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.722 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.722 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.4", 2024-03-23T03:48:16.722 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCHUP5lHcd2JRAAwID1TPb+Z9ViH51B0Mv/kQ==", 2024-03-23T03:48:16.722 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.722 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:48:16.722 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:48:16.722 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:16.722 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.723 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.723 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.723 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.5", 2024-03-23T03:48:16.723 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCLUP5lApp3AxAADslbuzTe/bf5FML1ckE05w==", 2024-03-23T03:48:16.723 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.723 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:48:16.723 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:48:16.723 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:16.723 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.723 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.723 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.724 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.6", 2024-03-23T03:48:16.724 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCOUP5lsi7VGxAArhWvXQ1VzJ8oSiOXzq8VsQ==", 2024-03-23T03:48:16.724 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.724 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:48:16.724 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:48:16.724 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:16.724 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.724 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.724 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.724 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.7", 2024-03-23T03:48:16.724 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCRUP5lu182JxAA5LaDXfrBkOXEUNAJXFDCJQ==", 2024-03-23T03:48:16.724 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.724 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:48:16.724 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:48:16.724 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:16.725 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.725 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.725 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.725 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.0", 2024-03-23T03:48:16.725 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lxTfoGRAA3WIQ8p1rGSAyJkLZJMk3dA==", 2024-03-23T03:48:16.725 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.725 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:48:16.725 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:48:16.725 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:48:16.725 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:48:16.725 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.725 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.725 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.725 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.1", 2024-03-23T03:48:16.726 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lneSHLhAAwn6ULOajHXrOM650HUcLbw==", 2024-03-23T03:48:16.726 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.726 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:48:16.726 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:48:16.726 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:48:16.726 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:48:16.726 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.726 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.726 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.726 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.2", 2024-03-23T03:48:16.726 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lOIklJBAAhx2d0Xy/j1PG0qsvC3nMqA==", 2024-03-23T03:48:16.726 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.726 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:48:16.726 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:48:16.726 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:48:16.727 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:48:16.727 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.727 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.727 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.727 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.3", 2024-03-23T03:48:16.727 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5l+7BVORAA7PBYS6mkgLA8b4sLCAYgBA==", 2024-03-23T03:48:16.727 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.727 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:48:16.727 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:48:16.727 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:48:16.727 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:48:16.727 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.727 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.727 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.admin", 2024-03-23T03:48:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBwUP5ldlDgBxAAOiqmhC5GulxELun8OiR4lg==", 2024-03-23T03:48:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:48:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:48:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:48:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.bootstrap-mds", 2024-03-23T03:48:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCaUP5ldnOyMhAAGkCxPh2A00NDNNSGtXUw5w==", 2024-03-23T03:48:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile bootstrap-mds" 2024-03-23T03:48:16.728 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.729 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.729 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.729 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.bootstrap-mgr", 2024-03-23T03:48:16.729 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCaUP5l7JSyMhAArwWpWGSlt3rNooLD1bXkbg==", 2024-03-23T03:48:16.729 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.729 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-23T03:48:16.729 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.729 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.729 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.729 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.bootstrap-osd", 2024-03-23T03:48:16.729 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCaUP5lXa2yMhAAnhRAFfOqiAD38BY8hGrqEA==", 2024-03-23T03:48:16.729 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.729 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile bootstrap-osd" 2024-03-23T03:48:16.729 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.729 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.730 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.730 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.bootstrap-rbd", 2024-03-23T03:48:16.730 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCaUP5lucSyMhAArqWmRtwz7KDsdGiGVAc1bA==", 2024-03-23T03:48:16.730 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.730 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-23T03:48:16.730 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.730 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.730 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.730 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-23T03:48:16.730 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCaUP5lLt6yMhAA4cC7qEw+4LcWSSZjFb8gqw==", 2024-03-23T03:48:16.730 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.730 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-23T03:48:16.730 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.730 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.bootstrap-rgw", 2024-03-23T03:48:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCaUP5ld/eyMhAAP0PuFBm17gVCqU0T6bZ78Q==", 2024-03-23T03:48:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-23T03:48:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mgr.x", 2024-03-23T03:48:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5lQXojChAAzepTDh0Pk3fuBZYsFt/+DQ==", 2024-03-23T03:48:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:48:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile mgr", 2024-03-23T03:48:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:16.731 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.732 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:16.732 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:16.732 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mgr.y", 2024-03-23T03:48:16.732 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBwUP5lK59iORAACuBF0JaQ8epyu3Mq3slzjg==", 2024-03-23T03:48:16.732 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:16.732 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:48:16.732 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile mgr", 2024-03-23T03:48:16.732 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:16.732 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.732 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:16.732 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-03-23T03:48:16.732 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:48:16.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.718+0000 7fcc3ea00640 1 -- 172.21.15.102:0/3191888424 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcc1406ac30 msgr2=0x7fcc1406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:16.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.718+0000 7fcc3ea00640 1 --2- 172.21.15.102:0/3191888424 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcc1406ac30 0x7fcc1406d0f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcc28034000 tx=0x7fcc28007000 comp rx=0 tx=0).stop 2024-03-23T03:48:16.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.718+0000 7fcc3ea00640 1 -- 172.21.15.102:0/3191888424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc3813f680 msgr2=0x7fcc38159f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:16.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.718+0000 7fcc3ea00640 1 --2- 172.21.15.102:0/3191888424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc3813f680 0x7fcc38159f30 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fcc2000d7d0 tx=0x7fcc2000dc90 comp rx=0 tx=0).stop 2024-03-23T03:48:16.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.718+0000 7fcc3ea00640 1 -- 172.21.15.102:0/3191888424 shutdown_connections 2024-03-23T03:48:16.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.718+0000 7fcc3ea00640 1 --2- 172.21.15.102:0/3191888424 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcc1406ac30 0x7fcc1406d0f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.718+0000 7fcc3ea00640 1 --2- 172.21.15.102:0/3191888424 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcc3813fb00 0x7fcc3815a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.718+0000 7fcc3ea00640 1 --2- 172.21.15.102:0/3191888424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc3813f680 0x7fcc38159f30 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.718+0000 7fcc3ea00640 1 --2- 172.21.15.102:0/3191888424 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcc1c005b80 0x7fcc3815bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.718+0000 7fcc3ea00640 1 -- 172.21.15.102:0/3191888424 >> 172.21.15.102:0/3191888424 conn(0x7fcc3806a3d0 msgr2=0x7fcc380726a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:16.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.718+0000 7fcc3ea00640 1 -- 172.21.15.102:0/3191888424 shutdown_connections 2024-03-23T03:48:16.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.718+0000 7fcc3ea00640 1 -- 172.21.15.102:0/3191888424 wait complete. 2024-03-23T03:48:16.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-23T03:48:16.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.926+0000 7f1f4b200640 1 Processor -- start 2024-03-23T03:48:16.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.927+0000 7f1f4b200640 1 -- start start 2024-03-23T03:48:16.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.928+0000 7f1f4b200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1f440cb660 0x7f1f440cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:16.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.928+0000 7f1f4b200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4415ae60 con 0x7f1f4406c2b0 2024-03-23T03:48:16.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.928+0000 7f1f4b200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f4415b140 con 0x7f1f440ce930 2024-03-23T03:48:16.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.928+0000 7f1f4b200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f4415bae0 con 0x7f1f440cb660 2024-03-23T03:48:16.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.929+0000 7f1f49200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1f4406c2b0 0x7f1f4406c720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50378/0 (socket says 172.21.15.102:50378) 2024-03-23T03:48:16.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.929+0000 7f1f49200640 1 -- 172.21.15.102:0/125069691 learned_addr learned my addr 172.21.15.102:0/125069691 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:16.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.929+0000 7f1f43400640 1 --2- 172.21.15.102:0/125069691 >> v2:172.21.15.173:3300/0 conn(0x7f1f440cb660 0x7f1f440cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:16.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.929+0000 7f1f42a00640 1 -- 172.21.15.102:0/125069691 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954659976 0 0) 0x7f1f4415ae60 con 0x7f1f4406c2b0 2024-03-23T03:48:16.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.929+0000 7f1f42a00640 1 -- 172.21.15.102:0/125069691 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f28003520 con 0x7f1f4406c2b0 2024-03-23T03:48:16.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.929+0000 7f1f42a00640 1 -- 172.21.15.102:0/125069691 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 25354744 0 0) 0x7f1f28003520 con 0x7f1f4406c2b0 2024-03-23T03:48:16.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.929+0000 7f1f42a00640 1 -- 172.21.15.102:0/125069691 >> v2:172.21.15.173:3300/0 conn(0x7f1f440cb660 msgr2=0x7f1f440cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:16.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.929+0000 7f1f42a00640 1 --2- 172.21.15.102:0/125069691 >> v2:172.21.15.173:3300/0 conn(0x7f1f440cb660 0x7f1f440cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f42a00640 1 -- 172.21.15.102:0/125069691 >> v1:172.21.15.173:6789/0 conn(0x7f1f440ce930 legacy=0x7f1f440ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:16.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f42a00640 1 -- 172.21.15.102:0/125069691 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f4415b420 con 0x7f1f4406c2b0 2024-03-23T03:48:16.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f42a00640 1 -- 172.21.15.102:0/125069691 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f1f3c0019f0 con 0x7f1f4406c2b0 2024-03-23T03:48:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f42a00640 1 -- 172.21.15.102:0/125069691 >> v1:172.21.15.102:6789/0 conn(0x7f1f4406c2b0 legacy=0x7f1f4406c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f42a00640 1 --2- 172.21.15.102:0/125069691 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1f280039c0 0x7f1f28003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f42a00640 1 --2- 172.21.15.102:0/125069691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f280047c0 0x7f1f28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f42a00640 1 --2- 172.21.15.102:0/125069691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f28005180 0x7f1f2800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f42a00640 1 -- 172.21.15.102:0/125069691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f4415bae0 con 0x7f1f28005180 2024-03-23T03:48:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f42a00640 1 -- 172.21.15.102:0/125069691 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2800c080 con 0x7f1f280047c0 2024-03-23T03:48:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f42a00640 1 -- 172.21.15.102:0/125069691 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2800c240 con 0x7f1f280039c0 2024-03-23T03:48:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f49200640 1 --2- 172.21.15.102:0/125069691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f28005180 0x7f1f2800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f43400640 1 --2- 172.21.15.102:0/125069691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f280047c0 0x7f1f28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f43e00640 1 --2- 172.21.15.102:0/125069691 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1f280039c0 0x7f1f28003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f49200640 1 -- 172.21.15.102:0/125069691 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1f280039c0 msgr2=0x7f1f28003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f49200640 1 --2- 172.21.15.102:0/125069691 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1f280039c0 0x7f1f28003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f49200640 1 -- 172.21.15.102:0/125069691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f280047c0 msgr2=0x7f1f28004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:16.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f49200640 1 --2- 172.21.15.102:0/125069691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f280047c0 0x7f1f28004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f49200640 1 -- 172.21.15.102:0/125069691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f28003520 con 0x7f1f28005180 2024-03-23T03:48:16.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f43400640 1 --2- 172.21.15.102:0/125069691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f280047c0 0x7f1f28004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:16.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.930+0000 7f1f43e00640 1 --2- 172.21.15.102:0/125069691 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1f280039c0 0x7f1f28003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:16.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.931+0000 7f1f49200640 1 -- 172.21.15.102:0/125069691 reap_dead start 2024-03-23T03:48:16.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.931+0000 7f1f49200640 1 --2- 172.21.15.102:0/125069691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f28005180 0x7f1f2800b980 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00a760 tx=0x7f1f280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0cf45a3ccc63de server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:16.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.932+0000 7f1f42a00640 1 -- 172.21.15.102:0/125069691 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c034630 con 0x7f1f28005180 2024-03-23T03:48:16.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.932+0000 7f1f42a00640 1 -- 172.21.15.102:0/125069691 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3c02f070 con 0x7f1f28005180 2024-03-23T03:48:16.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.932+0000 7f1f42a00640 1 -- 172.21.15.102:0/125069691 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3c034a10 con 0x7f1f28005180 2024-03-23T03:48:16.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.932+0000 7f1f4b200640 1 -- 172.21.15.102:0/125069691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f28005180 msgr2=0x7f1f2800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:16.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.932+0000 7f1f4b200640 1 --2- 172.21.15.102:0/125069691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f28005180 0x7f1f2800b980 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f1f3c00a760 tx=0x7f1f280057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.932+0000 7f1f4b200640 1 -- 172.21.15.102:0/125069691 shutdown_connections 2024-03-23T03:48:16.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.932+0000 7f1f4b200640 1 --2- 172.21.15.102:0/125069691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f28005180 0x7f1f2800b980 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.932+0000 7f1f4b200640 1 -- 172.21.15.102:0/125069691 >> 172.21.15.102:0/125069691 conn(0x7f1f4406a0b0 msgr2=0x7f1f4406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:16.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.933+0000 7f1f4b200640 1 -- 172.21.15.102:0/125069691 shutdown_connections 2024-03-23T03:48:16.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.933+0000 7f1f4b200640 1 -- 172.21.15.102:0/125069691 wait complete. 2024-03-23T03:48:16.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.934+0000 7f1f4b200640 1 Processor -- start 2024-03-23T03:48:16.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.934+0000 7f1f4b200640 1 -- start start 2024-03-23T03:48:16.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.935+0000 7f1f4b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f28005180 0x7f1f44137200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.935+0000 7f1f4b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f4413a840 0x7f1f440b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.935+0000 7f1f4b200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1f440b2ec0 0x7f1f4414ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.935+0000 7f1f4b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f44148810 con 0x7f1f28005180 2024-03-23T03:48:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.935+0000 7f1f4b200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f44148af0 con 0x7f1f4413a840 2024-03-23T03:48:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.935+0000 7f1f4b200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f44148dd0 con 0x7f1f440b2ec0 2024-03-23T03:48:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.935+0000 7f1f43e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f28005180 0x7f1f44137200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.936+0000 7f1f43e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f28005180 0x7f1f44137200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43500/0 (socket says 172.21.15.102:43500) 2024-03-23T03:48:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.936+0000 7f1f43e00640 1 -- 172.21.15.102:0/3551245620 learned_addr learned my addr 172.21.15.102:0/3551245620 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.936+0000 7f1f43400640 1 --2- 172.21.15.102:0/3551245620 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f4413a840 0x7f1f440b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.936+0000 7f1f43e00640 1 -- 172.21.15.102:0/3551245620 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1f440b2ec0 msgr2=0x7f1f4414ad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.936+0000 7f1f43e00640 1 --2- 172.21.15.102:0/3551245620 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1f440b2ec0 0x7f1f4414ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.936+0000 7f1f43e00640 1 -- 172.21.15.102:0/3551245620 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f4413a840 msgr2=0x7f1f440b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:16.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.936+0000 7f1f43e00640 1 --2- 172.21.15.102:0/3551245620 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f4413a840 0x7f1f440b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:16.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.936+0000 7f1f43e00640 1 -- 172.21.15.102:0/3551245620 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f44155280 con 0x7f1f28005180 2024-03-23T03:48:16.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.936+0000 7f1f43e00640 1 --2- 172.21.15.102:0/3551245620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f28005180 0x7f1f44137200 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400d770 tx=0x7f1f3400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:16.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.936+0000 7f1f3be00640 1 -- 172.21.15.102:0/3551245620 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f34005270 con 0x7f1f28005180 2024-03-23T03:48:16.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.936+0000 7f1f3be00640 1 -- 172.21.15.102:0/3551245620 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f340053d0 con 0x7f1f28005180 2024-03-23T03:48:16.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.936+0000 7f1f3be00640 1 -- 172.21.15.102:0/3551245620 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3400b8f0 con 0x7f1f28005180 2024-03-23T03:48:16.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.936+0000 7f1f4b200640 1 -- 172.21.15.102:0/3551245620 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f441490b0 con 0x7f1f28005180 2024-03-23T03:48:16.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.937+0000 7f1f4b200640 1 -- 172.21.15.102:0/3551245620 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f44137740 con 0x7f1f28005180 2024-03-23T03:48:16.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.938+0000 7f1f4b200640 1 -- 172.21.15.102:0/3551245620 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f44073390 con 0x7f1f28005180 2024-03-23T03:48:16.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.939+0000 7f1f3be00640 1 -- 172.21.15.102:0/3551245620 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1f3400ba50 con 0x7f1f28005180 2024-03-23T03:48:16.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.939+0000 7f1f3be00640 1 --2- 172.21.15.102:0/3551245620 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1f2006ac00 0x7f1f2006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:16.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.940+0000 7f1f3be00640 1 -- 172.21.15.102:0/3551245620 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f1f34089090 con 0x7f1f28005180 2024-03-23T03:48:16.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.940+0000 7f1f43400640 1 --2- 172.21.15.102:0/3551245620 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1f2006ac00 0x7f1f2006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:16.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.940+0000 7f1f43400640 1 --2- 172.21.15.102:0/3551245620 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1f2006ac00 0x7f1f2006d0c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1f441f1640 tx=0x7f1f2c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:16.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:16.944+0000 7f1f3be00640 1 -- 172.21.15.102:0/3551245620 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1f34054470 con 0x7f1f28005180 2024-03-23T03:48:17.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.205+0000 7f1f4b200640 1 -- 172.21.15.102:0/3551245620 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f1f440ce150 con 0x7f1f28005180 2024-03-23T03:48:17.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.213+0000 7f1f3be00640 1 -- 172.21.15.102:0/3551245620 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f1f34019310 con 0x7f1f28005180 2024-03-23T03:48:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.216+0000 7f1f4b200640 1 -- 172.21.15.102:0/3551245620 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1f2006ac00 msgr2=0x7f1f2006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.216+0000 7f1f4b200640 1 --2- 172.21.15.102:0/3551245620 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1f2006ac00 0x7f1f2006d0c0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1f441f1640 tx=0x7f1f2c008040 comp rx=0 tx=0).stop 2024-03-23T03:48:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.216+0000 7f1f4b200640 1 -- 172.21.15.102:0/3551245620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f28005180 msgr2=0x7f1f44137200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.216+0000 7f1f4b200640 1 --2- 172.21.15.102:0/3551245620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f28005180 0x7f1f44137200 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f1f3400d770 tx=0x7f1f3400dc30 comp rx=0 tx=0).stop 2024-03-23T03:48:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.216+0000 7f1f4b200640 1 -- 172.21.15.102:0/3551245620 shutdown_connections 2024-03-23T03:48:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.216+0000 7f1f4b200640 1 --2- 172.21.15.102:0/3551245620 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1f2006ac00 0x7f1f2006d0c0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.216+0000 7f1f4b200640 1 --2- 172.21.15.102:0/3551245620 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1f440b2ec0 0x7f1f4414ad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.216+0000 7f1f4b200640 1 --2- 172.21.15.102:0/3551245620 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1f4413a840 0x7f1f440b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.216+0000 7f1f4b200640 1 --2- 172.21.15.102:0/3551245620 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f28005180 0x7f1f44137200 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.216+0000 7f1f4b200640 1 -- 172.21.15.102:0/3551245620 >> 172.21.15.102:0/3551245620 conn(0x7f1f4406a0b0 msgr2=0x7f1f4415a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:17.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.216+0000 7f1f4b200640 1 -- 172.21.15.102:0/3551245620 shutdown_connections 2024-03-23T03:48:17.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.216+0000 7f1f4b200640 1 -- 172.21.15.102:0/3551245620 wait complete. 2024-03-23T03:48:17.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-23T03:48:17.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.440+0000 7faf81600640 1 Processor -- start 2024-03-23T03:48:17.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.440+0000 7faf81600640 1 -- start start 2024-03-23T03:48:17.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.442+0000 7faf81600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faf7c159690 0x7faf7c0ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:17.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.442+0000 7faf81600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7c15e010 con 0x7faf7c0cc330 2024-03-23T03:48:17.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.442+0000 7faf81600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7c0cc7f0 con 0x7faf7c0cd890 2024-03-23T03:48:17.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.442+0000 7faf81600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7c0ccad0 con 0x7faf7c159690 2024-03-23T03:48:17.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.442+0000 7faf7aa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faf7c159690 0x7faf7c0ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:17.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.442+0000 7faf7a000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7faf7c0cc330 0x7faf7c0b18d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50388/0 (socket says 172.21.15.102:50388) 2024-03-23T03:48:17.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.442+0000 7faf7a000640 1 -- 172.21.15.102:0/2677544606 learned_addr learned my addr 172.21.15.102:0/2677544606 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:17.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.442+0000 7faf7aa00640 1 -- 172.21.15.102:0/2677544606 >> v1:172.21.15.173:6789/0 conn(0x7faf7c0cd890 legacy=0x7faf7c15d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:17.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.443+0000 7faf7aa00640 1 -- 172.21.15.102:0/2677544606 >> v1:172.21.15.102:6789/0 conn(0x7faf7c0cc330 legacy=0x7faf7c0b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:17.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.442+0000 7faf79600640 1 -- 172.21.15.102:0/2677544606 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860805776 0 0) 0x7faf7c15e010 con 0x7faf7c0cc330 2024-03-23T03:48:17.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.443+0000 7faf7aa00640 1 -- 172.21.15.102:0/2677544606 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c0ccdb0 con 0x7faf7c159690 2024-03-23T03:48:17.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.444+0000 7faf7aa00640 1 --2- 172.21.15.102:0/2677544606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c159690 0x7faf7c0ca7b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7faf6c0097a0 tx=0x7faf6c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b373da56a83a25c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:17.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.444+0000 7faf79600640 1 -- 172.21.15.102:0/2677544606 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c031830 con 0x7faf7c159690 2024-03-23T03:48:17.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.444+0000 7faf79600640 1 -- 172.21.15.102:0/2677544606 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6c031990 con 0x7faf7c159690 2024-03-23T03:48:17.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.444+0000 7faf79600640 1 -- 172.21.15.102:0/2677544606 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c031d20 con 0x7faf7c159690 2024-03-23T03:48:17.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.444+0000 7faf81600640 1 -- 172.21.15.102:0/2677544606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c159690 msgr2=0x7faf7c0ca7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:17.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.444+0000 7faf81600640 1 --2- 172.21.15.102:0/2677544606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c159690 0x7faf7c0ca7b0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7faf6c0097a0 tx=0x7faf6c02b8d0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.445+0000 7faf81600640 1 -- 172.21.15.102:0/2677544606 shutdown_connections 2024-03-23T03:48:17.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.445+0000 7faf81600640 1 --2- 172.21.15.102:0/2677544606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c159690 0x7faf7c0ca7b0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.445+0000 7faf81600640 1 -- 172.21.15.102:0/2677544606 >> 172.21.15.102:0/2677544606 conn(0x7faf7c06a0b0 msgr2=0x7faf7c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:17.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.445+0000 7faf81600640 1 -- 172.21.15.102:0/2677544606 shutdown_connections 2024-03-23T03:48:17.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.445+0000 7faf81600640 1 -- 172.21.15.102:0/2677544606 wait complete. 2024-03-23T03:48:17.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.446+0000 7faf81600640 1 Processor -- start 2024-03-23T03:48:17.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.447+0000 7faf81600640 1 -- start start 2024-03-23T03:48:17.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.447+0000 7faf81600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faf7c0cc330 0x7faf7c1557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:17.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.447+0000 7faf81600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c0cd890 0x7faf7c1420b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:17.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.447+0000 7faf81600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf7c159690 0x7faf7c0b0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:17.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.447+0000 7faf81600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c14bac0 con 0x7faf7c0cd890 2024-03-23T03:48:17.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.447+0000 7faf81600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c149a20 con 0x7faf7c159690 2024-03-23T03:48:17.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.447+0000 7faf81600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c155d00 con 0x7faf7c0cc330 2024-03-23T03:48:17.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.448+0000 7faf7a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c0cd890 0x7faf7c1420b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:17.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.448+0000 7faf7a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c0cd890 0x7faf7c1420b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43510/0 (socket says 172.21.15.102:43510) 2024-03-23T03:48:17.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.448+0000 7faf7a000640 1 -- 172.21.15.102:0/423038502 learned_addr learned my addr 172.21.15.102:0/423038502 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:17.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.448+0000 7faf7aa00640 1 --2- 172.21.15.102:0/423038502 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faf7c0cc330 0x7faf7c1557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:17.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.448+0000 7faf7b400640 1 --2- 172.21.15.102:0/423038502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf7c159690 0x7faf7c0b0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:17.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.448+0000 7faf7a000640 1 -- 172.21.15.102:0/423038502 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faf7c0cc330 msgr2=0x7faf7c1557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:17.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.448+0000 7faf7a000640 1 --2- 172.21.15.102:0/423038502 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faf7c0cc330 0x7faf7c1557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.448+0000 7faf7a000640 1 -- 172.21.15.102:0/423038502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf7c159690 msgr2=0x7faf7c0b0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:17.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.448+0000 7faf7a000640 1 --2- 172.21.15.102:0/423038502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf7c159690 0x7faf7c0b0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.448+0000 7faf7a000640 1 -- 172.21.15.102:0/423038502 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6c009400 con 0x7faf7c0cd890 2024-03-23T03:48:17.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.448+0000 7faf7b400640 1 --2- 172.21.15.102:0/423038502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf7c159690 0x7faf7c0b0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:17.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.449+0000 7faf7a000640 1 --2- 172.21.15.102:0/423038502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c0cd890 0x7faf7c1420b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7faf6800c500 tx=0x7faf6800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:17.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.449+0000 7faf72a00640 1 -- 172.21.15.102:0/423038502 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf68007560 con 0x7faf7c0cd890 2024-03-23T03:48:17.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.449+0000 7faf72a00640 1 -- 172.21.15.102:0/423038502 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf680047e0 con 0x7faf7c0cd890 2024-03-23T03:48:17.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.449+0000 7faf72a00640 1 -- 172.21.15.102:0/423038502 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf68004b90 con 0x7faf7c0cd890 2024-03-23T03:48:17.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.449+0000 7faf81600640 1 -- 172.21.15.102:0/423038502 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7c14aeb0 con 0x7faf7c0cd890 2024-03-23T03:48:17.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.449+0000 7faf81600640 1 -- 172.21.15.102:0/423038502 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7c14b470 con 0x7faf7c0cd890 2024-03-23T03:48:17.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.450+0000 7faf81600640 1 -- 172.21.15.102:0/423038502 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7c14bc70 con 0x7faf7c0cd890 2024-03-23T03:48:17.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.451+0000 7faf72a00640 1 -- 172.21.15.102:0/423038502 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7faf68004cf0 con 0x7faf7c0cd890 2024-03-23T03:48:17.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.452+0000 7faf72a00640 1 --2- 172.21.15.102:0/423038502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7faf5c06ac30 0x7faf5c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:17.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.452+0000 7faf72a00640 1 -- 172.21.15.102:0/423038502 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5177+0+0 (secure 0 0 0) 0x7faf68088240 con 0x7faf7c0cd890 2024-03-23T03:48:17.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.452+0000 7faf7aa00640 1 --2- 172.21.15.102:0/423038502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7faf5c06ac30 0x7faf5c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:17.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.453+0000 7faf7aa00640 1 --2- 172.21.15.102:0/423038502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7faf5c06ac30 0x7faf5c06d0f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7faf6c036000 tx=0x7faf6c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:17.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.456+0000 7faf72a00640 1 -- 172.21.15.102:0/423038502 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faf7c14bc70 con 0x7faf7c0cd890 2024-03-23T03:48:17.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.719+0000 7faf81600640 1 -- 172.21.15.102:0/423038502 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7faf7c073390 con 0x7faf7c0cd890 2024-03-23T03:48:17.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.726+0000 7faf72a00640 1 -- 172.21.15.102:0/423038502 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7faf680536c0 con 0x7faf7c0cd890 2024-03-23T03:48:17.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.728+0000 7faf81600640 1 -- 172.21.15.102:0/423038502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7faf5c06ac30 msgr2=0x7faf5c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:17.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.728+0000 7faf81600640 1 --2- 172.21.15.102:0/423038502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7faf5c06ac30 0x7faf5c06d0f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7faf6c036000 tx=0x7faf6c03c040 comp rx=0 tx=0).stop 2024-03-23T03:48:17.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.728+0000 7faf81600640 1 -- 172.21.15.102:0/423038502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c0cd890 msgr2=0x7faf7c1420b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:17.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.728+0000 7faf81600640 1 --2- 172.21.15.102:0/423038502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c0cd890 0x7faf7c1420b0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7faf6800c500 tx=0x7faf6800c9c0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.728+0000 7faf81600640 1 -- 172.21.15.102:0/423038502 shutdown_connections 2024-03-23T03:48:17.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.728+0000 7faf81600640 1 --2- 172.21.15.102:0/423038502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7faf5c06ac30 0x7faf5c06d0f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.728+0000 7faf81600640 1 --2- 172.21.15.102:0/423038502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf7c159690 0x7faf7c0b0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.728+0000 7faf81600640 1 --2- 172.21.15.102:0/423038502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c0cd890 0x7faf7c1420b0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.728+0000 7faf81600640 1 --2- 172.21.15.102:0/423038502 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faf7c0cc330 0x7faf7c1557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.728+0000 7faf81600640 1 -- 172.21.15.102:0/423038502 >> 172.21.15.102:0/423038502 conn(0x7faf7c06a0b0 msgr2=0x7faf7c0b1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:17.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.729+0000 7faf81600640 1 -- 172.21.15.102:0/423038502 shutdown_connections 2024-03-23T03:48:17.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.729+0000 7faf81600640 1 -- 172.21.15.102:0/423038502 wait complete. 2024-03-23T03:48:17.747 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-23T03:48:17.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.947+0000 7f19de600640 1 Processor -- start 2024-03-23T03:48:17.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.947+0000 7f19de600640 1 -- start start 2024-03-23T03:48:17.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.949+0000 7f19de600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f19d8159690 0x7f19d80ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:17.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.949+0000 7f19de600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f19d80cc330 0x7f19d80b18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:17.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.949+0000 7f19de600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d815e010 con 0x7f19d80b1e10 2024-03-23T03:48:17.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.949+0000 7f19de600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d80cc7f0 con 0x7f19d8159690 2024-03-23T03:48:17.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.949+0000 7f19de600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d80ccad0 con 0x7f19d80cc330 2024-03-23T03:48:17.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.949+0000 7f19d7e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f19d80b1e10 0x7f19d80cd890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50404/0 (socket says 172.21.15.102:50404) 2024-03-23T03:48:17.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.949+0000 7f19d7e00640 1 -- 172.21.15.102:0/790548786 learned_addr learned my addr 172.21.15.102:0/790548786 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d6000640 1 -- 172.21.15.102:0/790548786 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298865142 0 0) 0x7f19d815e010 con 0x7f19d80b1e10 2024-03-23T03:48:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d7400640 1 --2- 172.21.15.102:0/790548786 >> v2:172.21.15.173:3300/0 conn(0x7f19d8159690 0x7f19d80ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d6000640 1 -- 172.21.15.102:0/790548786 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19bc005730 con 0x7f19d80b1e10 2024-03-23T03:48:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d6a00640 1 --2- 172.21.15.102:0/790548786 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f19d80cc330 0x7f19d80b18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d6000640 1 -- 172.21.15.102:0/790548786 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2719614406 0 0) 0x7f19bc005730 con 0x7f19d80b1e10 2024-03-23T03:48:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d6000640 1 -- 172.21.15.102:0/790548786 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f19d80cc330 msgr2=0x7f19d80b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d6000640 1 --2- 172.21.15.102:0/790548786 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f19d80cc330 0x7f19d80b18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d6000640 1 -- 172.21.15.102:0/790548786 >> v2:172.21.15.173:3300/0 conn(0x7f19d8159690 msgr2=0x7f19d80ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d6000640 1 --2- 172.21.15.102:0/790548786 >> v2:172.21.15.173:3300/0 conn(0x7f19d8159690 0x7f19d80ca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d6000640 1 -- 172.21.15.102:0/790548786 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d80ccdb0 con 0x7f19d80b1e10 2024-03-23T03:48:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d6000640 1 -- 172.21.15.102:0/790548786 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f19d00019f0 con 0x7f19d80b1e10 2024-03-23T03:48:17.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d6000640 1 -- 172.21.15.102:0/790548786 >> v1:172.21.15.102:6789/0 conn(0x7f19d80b1e10 legacy=0x7f19d80cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d6000640 1 --2- 172.21.15.102:0/790548786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19bc005b80 0x7f19bc005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d6000640 1 --2- 172.21.15.102:0/790548786 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f19d80cc330 0x7f19bc0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d6000640 1 --2- 172.21.15.102:0/790548786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19bc006f10 0x7f19bc007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d6000640 1 -- 172.21.15.102:0/790548786 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d80cc7f0 con 0x7f19bc006f10 2024-03-23T03:48:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d6000640 1 -- 172.21.15.102:0/790548786 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d80ccad0 con 0x7f19bc005b80 2024-03-23T03:48:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d6000640 1 -- 172.21.15.102:0/790548786 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19bc00dc40 con 0x7f19d80cc330 2024-03-23T03:48:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.950+0000 7f19d7e00640 1 --2- 172.21.15.102:0/790548786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19bc006f10 0x7f19bc007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.951+0000 7f19d7e00640 1 -- 172.21.15.102:0/790548786 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f19d80cc330 msgr2=0x7f19bc0069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.951+0000 7f19d7e00640 1 --2- 172.21.15.102:0/790548786 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f19d80cc330 0x7f19bc0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.951+0000 7f19d7e00640 1 -- 172.21.15.102:0/790548786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19bc005b80 msgr2=0x7f19bc005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.951+0000 7f19d7e00640 1 --2- 172.21.15.102:0/790548786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19bc005b80 0x7f19bc005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.951+0000 7f19d7e00640 1 -- 172.21.15.102:0/790548786 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19bc005730 con 0x7f19bc006f10 2024-03-23T03:48:17.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.952+0000 7f19d7e00640 1 --2- 172.21.15.102:0/790548786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19bc006f10 0x7f19bc007390 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f19d000a890 tx=0x7f19bc007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e618de193cdabc62 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:17.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.952+0000 7f19d6000640 1 -- 172.21.15.102:0/790548786 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d0034630 con 0x7f19bc006f10 2024-03-23T03:48:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.952+0000 7f19d6000640 1 -- 172.21.15.102:0/790548786 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d002f070 con 0x7f19bc006f10 2024-03-23T03:48:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.952+0000 7f19d6000640 1 -- 172.21.15.102:0/790548786 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d00349f0 con 0x7f19bc006f10 2024-03-23T03:48:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.952+0000 7f19de600640 1 -- 172.21.15.102:0/790548786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19bc006f10 msgr2=0x7f19bc007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.952+0000 7f19de600640 1 --2- 172.21.15.102:0/790548786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19bc006f10 0x7f19bc007390 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f19d000a890 tx=0x7f19bc007a00 comp rx=0 tx=0).stop 2024-03-23T03:48:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.952+0000 7f19d7e00640 1 -- 172.21.15.102:0/790548786 reap_dead start 2024-03-23T03:48:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.953+0000 7f19de600640 1 -- 172.21.15.102:0/790548786 shutdown_connections 2024-03-23T03:48:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.953+0000 7f19de600640 1 -- 172.21.15.102:0/790548786 >> 172.21.15.102:0/790548786 conn(0x7f19d806a0b0 msgr2=0x7f19d806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.953+0000 7f19de600640 1 -- 172.21.15.102:0/790548786 shutdown_connections 2024-03-23T03:48:17.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.953+0000 7f19de600640 1 -- 172.21.15.102:0/790548786 wait complete. 2024-03-23T03:48:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.954+0000 7f19de600640 1 Processor -- start 2024-03-23T03:48:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.955+0000 7f19de600640 1 -- start start 2024-03-23T03:48:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.955+0000 7f19de600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19bc006f10 0x7f19d8146880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.955+0000 7f19de600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d8149e40 0x7f19d815a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.955+0000 7f19de600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f19d80b0ba0 0x7f19d80b1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.956+0000 7f19de600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d80b1560 con 0x7f19d8149e40 2024-03-23T03:48:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.956+0000 7f19de600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19d815a930 con 0x7f19bc006f10 2024-03-23T03:48:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.956+0000 7f19de600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d815ac10 con 0x7f19d80b0ba0 2024-03-23T03:48:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.956+0000 7f19d6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d8149e40 0x7f19d815a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.956+0000 7f19d6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d8149e40 0x7f19d815a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43538/0 (socket says 172.21.15.102:43538) 2024-03-23T03:48:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.956+0000 7f19d6a00640 1 -- 172.21.15.102:0/1765967163 learned_addr learned my addr 172.21.15.102:0/1765967163 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.956+0000 7f19d7400640 1 --2- 172.21.15.102:0/1765967163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19bc006f10 0x7f19d8146880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:17.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.956+0000 7f19d6a00640 1 -- 172.21.15.102:0/1765967163 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f19d80b0ba0 msgr2=0x7f19d80b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.956+0000 7f19d6a00640 1 --2- 172.21.15.102:0/1765967163 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f19d80b0ba0 0x7f19d80b1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.956+0000 7f19d6a00640 1 -- 172.21.15.102:0/1765967163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19bc006f10 msgr2=0x7f19d8146880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.956+0000 7f19d6a00640 1 --2- 172.21.15.102:0/1765967163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19bc006f10 0x7f19d8146880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.956+0000 7f19d6a00640 1 -- 172.21.15.102:0/1765967163 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d815aef0 con 0x7f19d8149e40 2024-03-23T03:48:17.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.956+0000 7f19d6a00640 1 --2- 172.21.15.102:0/1765967163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d8149e40 0x7f19d815a030 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f19c400d800 tx=0x7f19c400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.957+0000 7f19cf400640 1 -- 172.21.15.102:0/1765967163 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c400b6c0 con 0x7f19d8149e40 2024-03-23T03:48:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.957+0000 7f19cf400640 1 -- 172.21.15.102:0/1765967163 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c400b820 con 0x7f19d8149e40 2024-03-23T03:48:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.957+0000 7f19cf400640 1 -- 172.21.15.102:0/1765967163 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c400bbb0 con 0x7f19d8149e40 2024-03-23T03:48:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.957+0000 7f19de600640 1 -- 172.21.15.102:0/1765967163 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d815b1d0 con 0x7f19d8149e40 2024-03-23T03:48:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.957+0000 7f19de600640 1 -- 172.21.15.102:0/1765967163 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d8146dc0 con 0x7f19d8149e40 2024-03-23T03:48:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.958+0000 7f19de600640 1 -- 172.21.15.102:0/1765967163 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d8073390 con 0x7f19d8149e40 2024-03-23T03:48:17.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.959+0000 7f19cf400640 1 -- 172.21.15.102:0/1765967163 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f19c400bd10 con 0x7f19d8149e40 2024-03-23T03:48:17.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.960+0000 7f19cf400640 1 --2- 172.21.15.102:0/1765967163 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f19b806ac50 0x7f19b806d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:17.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.960+0000 7f19cf400640 1 -- 172.21.15.102:0/1765967163 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f19c4089300 con 0x7f19d8149e40 2024-03-23T03:48:17.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.960+0000 7f19d7400640 1 --2- 172.21.15.102:0/1765967163 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f19b806ac50 0x7f19b806d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:17.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.961+0000 7f19d7400640 1 --2- 172.21.15.102:0/1765967163 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f19b806ac50 0x7f19b806d110 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f19c80047b0 tx=0x7f19c8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:17.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:17.964+0000 7f19cf400640 1 -- 172.21.15.102:0/1765967163 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f19c4054760 con 0x7f19d8149e40 2024-03-23T03:48:18.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.207+0000 7f19de600640 1 -- 172.21.15.102:0/1765967163 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f19d80ab9b0 con 0x7f19d8149e40 2024-03-23T03:48:18.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.214+0000 7f19cf400640 1 -- 172.21.15.102:0/1765967163 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f19c405e030 con 0x7f19d8149e40 2024-03-23T03:48:18.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.216+0000 7f19de600640 1 -- 172.21.15.102:0/1765967163 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f19b806ac50 msgr2=0x7f19b806d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:18.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.216+0000 7f19de600640 1 --2- 172.21.15.102:0/1765967163 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f19b806ac50 0x7f19b806d110 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f19c80047b0 tx=0x7f19c8008040 comp rx=0 tx=0).stop 2024-03-23T03:48:18.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.216+0000 7f19de600640 1 -- 172.21.15.102:0/1765967163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d8149e40 msgr2=0x7f19d815a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:18.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.216+0000 7f19de600640 1 --2- 172.21.15.102:0/1765967163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d8149e40 0x7f19d815a030 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f19c400d800 tx=0x7f19c400dcc0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.217+0000 7f19de600640 1 -- 172.21.15.102:0/1765967163 shutdown_connections 2024-03-23T03:48:18.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.217+0000 7f19de600640 1 --2- 172.21.15.102:0/1765967163 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f19b806ac50 0x7f19b806d110 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.217+0000 7f19de600640 1 --2- 172.21.15.102:0/1765967163 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f19d80b0ba0 0x7f19d80b1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.217+0000 7f19de600640 1 --2- 172.21.15.102:0/1765967163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f19d8149e40 0x7f19d815a030 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.217+0000 7f19de600640 1 --2- 172.21.15.102:0/1765967163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19bc006f10 0x7f19d8146880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.217+0000 7f19de600640 1 -- 172.21.15.102:0/1765967163 >> 172.21.15.102:0/1765967163 conn(0x7f19d806a0b0 msgr2=0x7f19d80b2cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:18.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.217+0000 7f19de600640 1 -- 172.21.15.102:0/1765967163 shutdown_connections 2024-03-23T03:48:18.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.217+0000 7f19de600640 1 -- 172.21.15.102:0/1765967163 wait complete. 2024-03-23T03:48:18.237 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-23T03:48:18.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.437+0000 7f36a4e00640 1 Processor -- start 2024-03-23T03:48:18.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.437+0000 7f36a4e00640 1 -- start start 2024-03-23T03:48:18.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.438+0000 7f36a4e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f36a015a600 0x7f36a006d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:18.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.438+0000 7f36a4e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a006e2b0 con 0x7f36a006dea0 2024-03-23T03:48:18.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.438+0000 7f36a4e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36a00b3490 con 0x7f36a0157180 2024-03-23T03:48:18.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.438+0000 7f36a4e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36a015a1e0 con 0x7f36a015a600 2024-03-23T03:48:18.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.439+0000 7f3697e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f36a015a600 0x7f36a006d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:18.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.439+0000 7f369fe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f36a006dea0 0x7f36a00cc1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50406/0 (socket says 172.21.15.102:50406) 2024-03-23T03:48:18.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.439+0000 7f369fe00640 1 -- 172.21.15.102:0/3517559013 learned_addr learned my addr 172.21.15.102:0/3517559013 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:18.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.439+0000 7f3697e00640 1 -- 172.21.15.102:0/3517559013 >> v1:172.21.15.173:6789/0 conn(0x7f36a0157180 legacy=0x7f36a00b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:18.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.439+0000 7f3697e00640 1 -- 172.21.15.102:0/3517559013 >> v1:172.21.15.102:6789/0 conn(0x7f36a006dea0 legacy=0x7f36a00cc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:18.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.439+0000 7f3697e00640 1 -- 172.21.15.102:0/3517559013 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a00b3770 con 0x7f36a015a600 2024-03-23T03:48:18.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.440+0000 7f3697e00640 1 --2- 172.21.15.102:0/3517559013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36a015a600 0x7f36a006d960 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f36880097a0 tx=0x7f368802da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a92982f9a4084354 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:18.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.440+0000 7f369ea00640 1 -- 172.21.15.102:0/3517559013 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688031870 con 0x7f36a015a600 2024-03-23T03:48:18.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.441+0000 7f369ea00640 1 -- 172.21.15.102:0/3517559013 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36880319d0 con 0x7f36a015a600 2024-03-23T03:48:18.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.441+0000 7f369ea00640 1 -- 172.21.15.102:0/3517559013 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3688031d60 con 0x7f36a015a600 2024-03-23T03:48:18.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.441+0000 7f36a4e00640 1 -- 172.21.15.102:0/3517559013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36a015a600 msgr2=0x7f36a006d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:18.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.441+0000 7f36a4e00640 1 --2- 172.21.15.102:0/3517559013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36a015a600 0x7f36a006d960 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f36880097a0 tx=0x7f368802da50 comp rx=0 tx=0).stop 2024-03-23T03:48:18.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.441+0000 7f36a4e00640 1 -- 172.21.15.102:0/3517559013 shutdown_connections 2024-03-23T03:48:18.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.441+0000 7f36a4e00640 1 --2- 172.21.15.102:0/3517559013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36a015a600 0x7f36a006d960 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.441+0000 7f36a4e00640 1 -- 172.21.15.102:0/3517559013 >> 172.21.15.102:0/3517559013 conn(0x7f36a0065e20 msgr2=0x7f36a0069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:18.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.441+0000 7f36a4e00640 1 -- 172.21.15.102:0/3517559013 shutdown_connections 2024-03-23T03:48:18.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.441+0000 7f36a4e00640 1 -- 172.21.15.102:0/3517559013 wait complete. 2024-03-23T03:48:18.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.443+0000 7f36a4e00640 1 Processor -- start 2024-03-23T03:48:18.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.443+0000 7f36a4e00640 1 -- start start 2024-03-23T03:48:18.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.445+0000 7f36a4e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36a006dea0 0x7f36a015d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:18.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.445+0000 7f36a4e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a0157180 0x7f36a00a74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:18.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.445+0000 7f36a4e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f36a015a600 0x7f36a01e3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:18.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.445+0000 7f36a4e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a01e2f00 con 0x7f36a006dea0 2024-03-23T03:48:18.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.445+0000 7f36a4e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a01e3150 con 0x7f36a0157180 2024-03-23T03:48:18.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.445+0000 7f36a4e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a01e3430 con 0x7f36a015a600 2024-03-23T03:48:18.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.445+0000 7f369f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36a006dea0 0x7f36a015d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:18.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.445+0000 7f369f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36a006dea0 0x7f36a015d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43552/0 (socket says 172.21.15.102:43552) 2024-03-23T03:48:18.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.445+0000 7f369f400640 1 -- 172.21.15.102:0/1673820837 learned_addr learned my addr 172.21.15.102:0/1673820837 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:18.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.445+0000 7f3697e00640 1 --2- 172.21.15.102:0/1673820837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a0157180 0x7f36a00a74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:18.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.445+0000 7f369f400640 1 -- 172.21.15.102:0/1673820837 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f36a015a600 msgr2=0x7f36a01e3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:18.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.445+0000 7f369fe00640 1 --2- 172.21.15.102:0/1673820837 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f36a015a600 0x7f36a01e3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.445+0000 7f369f400640 1 --2- 172.21.15.102:0/1673820837 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f36a015a600 0x7f36a01e3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.445+0000 7f369f400640 1 -- 172.21.15.102:0/1673820837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a0157180 msgr2=0x7f36a00a74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.445+0000 7f369f400640 1 --2- 172.21.15.102:0/1673820837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a0157180 0x7f36a00a74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.445+0000 7f369f400640 1 -- 172.21.15.102:0/1673820837 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3688009400 con 0x7f36a006dea0 2024-03-23T03:48:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.445+0000 7f3697e00640 1 --2- 172.21.15.102:0/1673820837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a0157180 0x7f36a00a74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.446+0000 7f369f400640 1 --2- 172.21.15.102:0/1673820837 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36a006dea0 0x7f36a015d8b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f369000c430 tx=0x7f369000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.446+0000 7f3697400640 1 -- 172.21.15.102:0/1673820837 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3690011790 con 0x7f36a006dea0 2024-03-23T03:48:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.446+0000 7f3697400640 1 -- 172.21.15.102:0/1673820837 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36900118f0 con 0x7f36a006dea0 2024-03-23T03:48:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.446+0000 7f36a4e00640 1 -- 172.21.15.102:0/1673820837 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a00b3260 con 0x7f36a006dea0 2024-03-23T03:48:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.446+0000 7f3697400640 1 -- 172.21.15.102:0/1673820837 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3690011ce0 con 0x7f36a006dea0 2024-03-23T03:48:18.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.446+0000 7f36a4e00640 1 -- 172.21.15.102:0/1673820837 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a01e39f0 con 0x7f36a006dea0 2024-03-23T03:48:18.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.447+0000 7f36a4e00640 1 -- 172.21.15.102:0/1673820837 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36a004e030 con 0x7f36a006dea0 2024-03-23T03:48:18.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.448+0000 7f3697400640 1 -- 172.21.15.102:0/1673820837 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f369001c460 con 0x7f36a006dea0 2024-03-23T03:48:18.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.449+0000 7f3697400640 1 --2- 172.21.15.102:0/1673820837 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f367c06ac30 0x7f367c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:18.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.450+0000 7f3697400640 1 -- 172.21.15.102:0/1673820837 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f3690088fd0 con 0x7f36a006dea0 2024-03-23T03:48:18.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.450+0000 7f3697e00640 1 --2- 172.21.15.102:0/1673820837 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f367c06ac30 0x7f367c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:18.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.450+0000 7f3697e00640 1 --2- 172.21.15.102:0/1673820837 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f367c06ac30 0x7f367c06d0f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3688038000 tx=0x7f368803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:18.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.454+0000 7f3697400640 1 -- 172.21.15.102:0/1673820837 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3690054450 con 0x7f36a006dea0 2024-03-23T03:48:18.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.697+0000 7f36a4e00640 1 -- 172.21.15.102:0/1673820837 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f36a00b1490 con 0x7f36a006dea0 2024-03-23T03:48:18.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.704+0000 7f3697400640 1 -- 172.21.15.102:0/1673820837 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f369005ce60 con 0x7f36a006dea0 2024-03-23T03:48:18.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.706+0000 7f36a4e00640 1 -- 172.21.15.102:0/1673820837 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f367c06ac30 msgr2=0x7f367c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:18.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.706+0000 7f36a4e00640 1 --2- 172.21.15.102:0/1673820837 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f367c06ac30 0x7f367c06d0f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3688038000 tx=0x7f368803b040 comp rx=0 tx=0).stop 2024-03-23T03:48:18.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.706+0000 7f36a4e00640 1 -- 172.21.15.102:0/1673820837 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36a006dea0 msgr2=0x7f36a015d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:18.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.706+0000 7f36a4e00640 1 --2- 172.21.15.102:0/1673820837 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36a006dea0 0x7f36a015d8b0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f369000c430 tx=0x7f369000c8f0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.707+0000 7f36a4e00640 1 -- 172.21.15.102:0/1673820837 shutdown_connections 2024-03-23T03:48:18.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.707+0000 7f36a4e00640 1 --2- 172.21.15.102:0/1673820837 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f367c06ac30 0x7f367c06d0f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.707+0000 7f36a4e00640 1 --2- 172.21.15.102:0/1673820837 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f36a015a600 0x7f36a01e3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.707+0000 7f36a4e00640 1 --2- 172.21.15.102:0/1673820837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36a0157180 0x7f36a00a74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.707+0000 7f36a4e00640 1 --2- 172.21.15.102:0/1673820837 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36a006dea0 0x7f36a015d8b0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.707+0000 7f36a4e00640 1 -- 172.21.15.102:0/1673820837 >> 172.21.15.102:0/1673820837 conn(0x7f36a0065e20 msgr2=0x7f36a0069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:18.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.707+0000 7f36a4e00640 1 -- 172.21.15.102:0/1673820837 shutdown_connections 2024-03-23T03:48:18.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.707+0000 7f36a4e00640 1 -- 172.21.15.102:0/1673820837 wait complete. 2024-03-23T03:48:18.726 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-23T03:48:18.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.932+0000 7fc2cda00640 1 Processor -- start 2024-03-23T03:48:18.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.933+0000 7fc2cda00640 1 -- start start 2024-03-23T03:48:18.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.934+0000 7fc2cda00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2c80cd6b0 0x7fc2c80b8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:18.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.934+0000 7fc2cda00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2c80cda90 0x7fc2c80d1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:18.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.934+0000 7fc2cda00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2c806b840 con 0x7fc2c815cd90 2024-03-23T03:48:18.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.934+0000 7fc2cda00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c806c0e0 con 0x7fc2c80cd6b0 2024-03-23T03:48:18.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.934+0000 7fc2cda00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c806bb20 con 0x7fc2c80cda90 2024-03-23T03:48:18.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.934+0000 7fc2c6a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2c80cd6b0 0x7fc2c80b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:18.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.934+0000 7fc2c6a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2c80cd6b0 0x7fc2c80b8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43560/0 (socket says 172.21.15.102:43560) 2024-03-23T03:48:18.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.934+0000 7fc2c6a00640 1 -- 172.21.15.102:0/2582268026 learned_addr learned my addr 172.21.15.102:0/2582268026 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:18.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.934+0000 7fc2c6a00640 1 -- 172.21.15.102:0/2582268026 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2c80cda90 msgr2=0x7fc2c80d1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:18.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.934+0000 7fc2c6a00640 1 --2- 172.21.15.102:0/2582268026 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2c80cda90 0x7fc2c80d1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.934+0000 7fc2c6a00640 1 -- 172.21.15.102:0/2582268026 >> v1:172.21.15.173:6789/0 conn(0x7fc2c815cd90 legacy=0x7fc2c806b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.934+0000 7fc2c6a00640 1 -- 172.21.15.102:0/2582268026 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c806be00 con 0x7fc2c80cd6b0 2024-03-23T03:48:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.935+0000 7fc2c6a00640 1 --2- 172.21.15.102:0/2582268026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2c80cd6b0 0x7fc2c80b8490 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8009870 tx=0x7fc2b802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3bcebc7015fd9d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.936+0000 7fc2c5600640 1 -- 172.21.15.102:0/2582268026 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8031640 con 0x7fc2c80cd6b0 2024-03-23T03:48:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.936+0000 7fc2c5600640 1 -- 172.21.15.102:0/2582268026 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b80317a0 con 0x7fc2c80cd6b0 2024-03-23T03:48:18.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.936+0000 7fc2c5600640 1 -- 172.21.15.102:0/2582268026 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b8031b30 con 0x7fc2c80cd6b0 2024-03-23T03:48:18.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.936+0000 7fc2cda00640 1 -- 172.21.15.102:0/2582268026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2c80cd6b0 msgr2=0x7fc2c80b8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:18.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.936+0000 7fc2cda00640 1 --2- 172.21.15.102:0/2582268026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2c80cd6b0 0x7fc2c80b8490 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8009870 tx=0x7fc2b802b8f0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.937+0000 7fc2cda00640 1 -- 172.21.15.102:0/2582268026 shutdown_connections 2024-03-23T03:48:18.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.937+0000 7fc2cda00640 1 --2- 172.21.15.102:0/2582268026 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2c80cda90 0x7fc2c80d1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.937+0000 7fc2cda00640 1 --2- 172.21.15.102:0/2582268026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2c80cd6b0 0x7fc2c80b8490 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.937+0000 7fc2cda00640 1 -- 172.21.15.102:0/2582268026 >> 172.21.15.102:0/2582268026 conn(0x7fc2c80cf780 msgr2=0x7fc2c80cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:18.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.937+0000 7fc2cda00640 1 -- 172.21.15.102:0/2582268026 shutdown_connections 2024-03-23T03:48:18.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.937+0000 7fc2cda00640 1 -- 172.21.15.102:0/2582268026 wait complete. 2024-03-23T03:48:18.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.938+0000 7fc2cda00640 1 Processor -- start 2024-03-23T03:48:18.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.938+0000 7fc2cda00640 1 -- start start 2024-03-23T03:48:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.939+0000 7fc2cda00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2c80cd6b0 0x7fc2c80c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.939+0000 7fc2cda00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2c80cda90 0x7fc2c80cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.939+0000 7fc2cda00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2c815cd90 0x7fc2c81557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.939+0000 7fc2cda00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c80cc590 con 0x7fc2c80cda90 2024-03-23T03:48:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.939+0000 7fc2cda00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c80c92d0 con 0x7fc2c815cd90 2024-03-23T03:48:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.939+0000 7fc2cda00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c80cc9b0 con 0x7fc2c80cd6b0 2024-03-23T03:48:18.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.940+0000 7fc2c6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2c80cda90 0x7fc2c80cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.940+0000 7fc2c6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2c80cda90 0x7fc2c80cafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43562/0 (socket says 172.21.15.102:43562) 2024-03-23T03:48:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.940+0000 7fc2c6000640 1 -- 172.21.15.102:0/3174597171 learned_addr learned my addr 172.21.15.102:0/3174597171 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.940+0000 7fc2c7400640 1 --2- 172.21.15.102:0/3174597171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2c815cd90 0x7fc2c81557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.940+0000 7fc2c6a00640 1 --2- 172.21.15.102:0/3174597171 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2c80cd6b0 0x7fc2c80c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.940+0000 7fc2c6000640 1 -- 172.21.15.102:0/3174597171 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2c80cd6b0 msgr2=0x7fc2c80c8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.940+0000 7fc2c6000640 1 --2- 172.21.15.102:0/3174597171 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2c80cd6b0 0x7fc2c80c8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.940+0000 7fc2c6000640 1 -- 172.21.15.102:0/3174597171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2c815cd90 msgr2=0x7fc2c81557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.940+0000 7fc2c6000640 1 --2- 172.21.15.102:0/3174597171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2c815cd90 0x7fc2c81557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.940+0000 7fc2c6000640 1 -- 172.21.15.102:0/3174597171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b80094d0 con 0x7fc2c80cda90 2024-03-23T03:48:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.940+0000 7fc2c7400640 1 --2- 172.21.15.102:0/3174597171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2c815cd90 0x7fc2c81557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.940+0000 7fc2c6000640 1 --2- 172.21.15.102:0/3174597171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2c80cda90 0x7fc2c80cafa0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000b410 tx=0x7fc2b000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.941+0000 7fc2bea00640 1 -- 172.21.15.102:0/3174597171 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0011830 con 0x7fc2c80cda90 2024-03-23T03:48:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.941+0000 7fc2bea00640 1 -- 172.21.15.102:0/3174597171 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b0011990 con 0x7fc2c80cda90 2024-03-23T03:48:18.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.941+0000 7fc2bea00640 1 -- 172.21.15.102:0/3174597171 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b0011d40 con 0x7fc2c80cda90 2024-03-23T03:48:18.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.941+0000 7fc2cda00640 1 -- 172.21.15.102:0/3174597171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c80cb910 con 0x7fc2c80cda90 2024-03-23T03:48:18.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.941+0000 7fc2cda00640 1 -- 172.21.15.102:0/3174597171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c80cbed0 con 0x7fc2c80cda90 2024-03-23T03:48:18.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.942+0000 7fc2cda00640 1 -- 172.21.15.102:0/3174597171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c80cc6d0 con 0x7fc2c80cda90 2024-03-23T03:48:18.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.943+0000 7fc2bea00640 1 -- 172.21.15.102:0/3174597171 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc2b001a400 con 0x7fc2c80cda90 2024-03-23T03:48:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.943+0000 7fc2bea00640 1 --2- 172.21.15.102:0/3174597171 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc2a406ac00 0x7fc2a406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.944+0000 7fc2bea00640 1 -- 172.21.15.102:0/3174597171 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fc2b00892b0 con 0x7fc2c80cda90 2024-03-23T03:48:18.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.944+0000 7fc2c6a00640 1 --2- 172.21.15.102:0/3174597171 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc2a406ac00 0x7fc2a406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:18.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.944+0000 7fc2c6a00640 1 --2- 172.21.15.102:0/3174597171 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc2a406ac00 0x7fc2a406d0c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8036000 tx=0x7fc2b803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:18.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:18.948+0000 7fc2bea00640 1 -- 172.21.15.102:0/3174597171 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc2c80cc6d0 con 0x7fc2c80cda90 2024-03-23T03:48:19.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.208+0000 7fc2cda00640 1 -- 172.21.15.102:0/3174597171 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fc2c80cc6d0 con 0x7fc2c80cda90 2024-03-23T03:48:19.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.215+0000 7fc2bea00640 1 -- 172.21.15.102:0/3174597171 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc2c80cc6d0 con 0x7fc2c80cda90 2024-03-23T03:48:19.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.218+0000 7fc2cda00640 1 -- 172.21.15.102:0/3174597171 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc2a406ac00 msgr2=0x7fc2a406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:19.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.218+0000 7fc2cda00640 1 --2- 172.21.15.102:0/3174597171 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc2a406ac00 0x7fc2a406d0c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8036000 tx=0x7fc2b803c040 comp rx=0 tx=0).stop 2024-03-23T03:48:19.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.218+0000 7fc2cda00640 1 -- 172.21.15.102:0/3174597171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2c80cda90 msgr2=0x7fc2c80cafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:19.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.218+0000 7fc2cda00640 1 --2- 172.21.15.102:0/3174597171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2c80cda90 0x7fc2c80cafa0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc2b000b410 tx=0x7fc2b000b8d0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.218+0000 7fc2cda00640 1 -- 172.21.15.102:0/3174597171 shutdown_connections 2024-03-23T03:48:19.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.218+0000 7fc2cda00640 1 --2- 172.21.15.102:0/3174597171 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc2a406ac00 0x7fc2a406d0c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.218+0000 7fc2cda00640 1 --2- 172.21.15.102:0/3174597171 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2c815cd90 0x7fc2c81557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.218+0000 7fc2cda00640 1 --2- 172.21.15.102:0/3174597171 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2c80cda90 0x7fc2c80cafa0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.218+0000 7fc2cda00640 1 --2- 172.21.15.102:0/3174597171 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2c80cd6b0 0x7fc2c80c8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.218+0000 7fc2cda00640 1 -- 172.21.15.102:0/3174597171 >> 172.21.15.102:0/3174597171 conn(0x7fc2c80cf780 msgr2=0x7fc2c80d1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:19.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.218+0000 7fc2cda00640 1 -- 172.21.15.102:0/3174597171 shutdown_connections 2024-03-23T03:48:19.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.218+0000 7fc2cda00640 1 -- 172.21.15.102:0/3174597171 wait complete. 2024-03-23T03:48:19.237 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-23T03:48:19.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.446+0000 7f3e62000640 1 Processor -- start 2024-03-23T03:48:19.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.446+0000 7f3e62000640 1 -- start start 2024-03-23T03:48:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.447+0000 7f3e62000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e5c159690 0x7f3e5c0ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.448+0000 7f3e62000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f3e5c0cd890 0x7f3e5c0cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.448+0000 7f3e62000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e5c15e010 con 0x7f3e5c0cc330 2024-03-23T03:48:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.448+0000 7f3e62000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e5c0cc7f0 con 0x7f3e5c0cd890 2024-03-23T03:48:19.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.448+0000 7f3e62000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c0ccad0 con 0x7f3e5c159690 2024-03-23T03:48:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.448+0000 7f3e5aa00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3e5c0cc330 0x7f3e5c0b18d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50412/0 (socket says 172.21.15.102:50412) 2024-03-23T03:48:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.448+0000 7f3e5aa00640 1 -- 172.21.15.102:0/3859139428 learned_addr learned my addr 172.21.15.102:0/3859139428 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.448+0000 7f3e5be00640 1 --2- 172.21.15.102:0/3859139428 >> v2:172.21.15.173:3300/0 conn(0x7f3e5c0cd890 0x7f3e5c0cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.448+0000 7f3e5b400640 1 --2- 172.21.15.102:0/3859139428 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e5c159690 0x7f3e5c0ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.449+0000 7f3e5a000640 1 -- 172.21.15.102:0/3859139428 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595711873 0 0) 0x7f3e5c15e010 con 0x7f3e5c0cc330 2024-03-23T03:48:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.449+0000 7f3e5a000640 1 -- 172.21.15.102:0/3859139428 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e40003520 con 0x7f3e5c0cc330 2024-03-23T03:48:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.449+0000 7f3e5a000640 1 -- 172.21.15.102:0/3859139428 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1322036215 0 0) 0x7f3e40003520 con 0x7f3e5c0cc330 2024-03-23T03:48:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.449+0000 7f3e5a000640 1 -- 172.21.15.102:0/3859139428 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e5c159690 msgr2=0x7f3e5c0ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.449+0000 7f3e5a000640 1 --2- 172.21.15.102:0/3859139428 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e5c159690 0x7f3e5c0ca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.449+0000 7f3e5a000640 1 -- 172.21.15.102:0/3859139428 >> v2:172.21.15.173:3300/0 conn(0x7f3e5c0cd890 msgr2=0x7f3e5c0cdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:19.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.449+0000 7f3e5a000640 1 --2- 172.21.15.102:0/3859139428 >> v2:172.21.15.173:3300/0 conn(0x7f3e5c0cd890 0x7f3e5c0cdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.449+0000 7f3e5a000640 1 -- 172.21.15.102:0/3859139428 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5c0ccdb0 con 0x7f3e5c0cc330 2024-03-23T03:48:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.449+0000 7f3e5a000640 1 -- 172.21.15.102:0/3859139428 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f3e48001670 con 0x7f3e5c0cc330 2024-03-23T03:48:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.449+0000 7f3e5a000640 1 -- 172.21.15.102:0/3859139428 >> v1:172.21.15.102:6789/0 conn(0x7f3e5c0cc330 legacy=0x7f3e5c0b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.449+0000 7f3e5a000640 1 --2- 172.21.15.102:0/3859139428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e40003970 0x7f3e40003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.449+0000 7f3e5a000640 1 --2- 172.21.15.102:0/3859139428 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e5c159690 0x7f3e400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.449+0000 7f3e5a000640 1 --2- 172.21.15.102:0/3859139428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e40004d00 0x7f3e40005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.449+0000 7f3e5a000640 1 -- 172.21.15.102:0/3859139428 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c0cc7f0 con 0x7f3e40004d00 2024-03-23T03:48:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.449+0000 7f3e5a000640 1 -- 172.21.15.102:0/3859139428 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c0ccad0 con 0x7f3e40003970 2024-03-23T03:48:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.449+0000 7f3e5a000640 1 -- 172.21.15.102:0/3859139428 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4000b980 con 0x7f3e5c159690 2024-03-23T03:48:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.450+0000 7f3e5be00640 1 --2- 172.21.15.102:0/3859139428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e40004d00 0x7f3e40005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.450+0000 7f3e5b400640 1 --2- 172.21.15.102:0/3859139428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e40003970 0x7f3e40003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.450+0000 7f3e5aa00640 1 --2- 172.21.15.102:0/3859139428 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e5c159690 0x7f3e400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.450+0000 7f3e5be00640 1 -- 172.21.15.102:0/3859139428 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e5c159690 msgr2=0x7f3e400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:19.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.450+0000 7f3e5be00640 1 --2- 172.21.15.102:0/3859139428 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e5c159690 0x7f3e400047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.450+0000 7f3e5be00640 1 -- 172.21.15.102:0/3859139428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e40003970 msgr2=0x7f3e40003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:19.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.450+0000 7f3e5be00640 1 --2- 172.21.15.102:0/3859139428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e40003970 0x7f3e40003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.450+0000 7f3e5be00640 1 -- 172.21.15.102:0/3859139428 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e40003520 con 0x7f3e40004d00 2024-03-23T03:48:19.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.450+0000 7f3e5aa00640 1 --2- 172.21.15.102:0/3859139428 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e5c159690 0x7f3e400047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:19.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.451+0000 7f3e5be00640 1 --2- 172.21.15.102:0/3859139428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e40004d00 0x7f3e40005180 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3e54009fc0 tx=0x7f3e400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=616333736a026461 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:19.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.451+0000 7f3e5a000640 1 -- 172.21.15.102:0/3859139428 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e5402cc60 con 0x7f3e40004d00 2024-03-23T03:48:19.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.451+0000 7f3e5a000640 1 -- 172.21.15.102:0/3859139428 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5400f040 con 0x7f3e40004d00 2024-03-23T03:48:19.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.451+0000 7f3e5a000640 1 -- 172.21.15.102:0/3859139428 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e54034440 con 0x7f3e40004d00 2024-03-23T03:48:19.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.452+0000 7f3e62000640 1 -- 172.21.15.102:0/3859139428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e40004d00 msgr2=0x7f3e40005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:19.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.452+0000 7f3e62000640 1 --2- 172.21.15.102:0/3859139428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e40004d00 0x7f3e40005180 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3e54009fc0 tx=0x7f3e400057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.452+0000 7f3e5be00640 1 -- 172.21.15.102:0/3859139428 reap_dead start 2024-03-23T03:48:19.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.452+0000 7f3e62000640 1 -- 172.21.15.102:0/3859139428 shutdown_connections 2024-03-23T03:48:19.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.452+0000 7f3e62000640 1 -- 172.21.15.102:0/3859139428 >> 172.21.15.102:0/3859139428 conn(0x7f3e5c06a0b0 msgr2=0x7f3e5c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:19.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.452+0000 7f3e62000640 1 -- 172.21.15.102:0/3859139428 shutdown_connections 2024-03-23T03:48:19.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.452+0000 7f3e62000640 1 -- 172.21.15.102:0/3859139428 wait complete. 2024-03-23T03:48:19.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.454+0000 7f3e62000640 1 Processor -- start 2024-03-23T03:48:19.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.454+0000 7f3e62000640 1 -- start start 2024-03-23T03:48:19.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.455+0000 7f3e62000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e40004d00 0x7f3e5c15a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:19.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.455+0000 7f3e62000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e5c142a40 0x7f3e5c1404d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:19.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.455+0000 7f3e62000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e5c14ec40 0x7f3e5c166980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:19.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.455+0000 7f3e62000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c0ae310 con 0x7f3e5c14ec40 2024-03-23T03:48:19.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.455+0000 7f3e62000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c165f30 con 0x7f3e5c142a40 2024-03-23T03:48:19.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.455+0000 7f3e62000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c15aa90 con 0x7f3e40004d00 2024-03-23T03:48:19.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.455+0000 7f3e5be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e5c14ec40 0x7f3e5c166980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:19.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.456+0000 7f3e5be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e5c14ec40 0x7f3e5c166980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43592/0 (socket says 172.21.15.102:43592) 2024-03-23T03:48:19.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.456+0000 7f3e5be00640 1 -- 172.21.15.102:0/1765730575 learned_addr learned my addr 172.21.15.102:0/1765730575 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:19.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.456+0000 7f3e5aa00640 1 --2- 172.21.15.102:0/1765730575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e5c142a40 0x7f3e5c1404d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:19.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.456+0000 7f3e5b400640 1 --2- 172.21.15.102:0/1765730575 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e40004d00 0x7f3e5c15a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:19.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.456+0000 7f3e5be00640 1 -- 172.21.15.102:0/1765730575 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e40004d00 msgr2=0x7f3e5c15a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:19.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.456+0000 7f3e5be00640 1 --2- 172.21.15.102:0/1765730575 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e40004d00 0x7f3e5c15a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.456+0000 7f3e5be00640 1 -- 172.21.15.102:0/1765730575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e5c142a40 msgr2=0x7f3e5c1404d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:19.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.456+0000 7f3e5be00640 1 --2- 172.21.15.102:0/1765730575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e5c142a40 0x7f3e5c1404d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.456+0000 7f3e5be00640 1 -- 172.21.15.102:0/1765730575 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5c15ad70 con 0x7f3e5c14ec40 2024-03-23T03:48:19.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.456+0000 7f3e5be00640 1 --2- 172.21.15.102:0/1765730575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e5c14ec40 0x7f3e5c166980 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3e54037040 tx=0x7f3e54009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:19.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.457+0000 7f3e53400640 1 -- 172.21.15.102:0/1765730575 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e54043070 con 0x7f3e5c14ec40 2024-03-23T03:48:19.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.457+0000 7f3e53400640 1 -- 172.21.15.102:0/1765730575 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e54034b50 con 0x7f3e5c14ec40 2024-03-23T03:48:19.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.457+0000 7f3e53400640 1 -- 172.21.15.102:0/1765730575 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e54003990 con 0x7f3e5c14ec40 2024-03-23T03:48:19.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.457+0000 7f3e62000640 1 -- 172.21.15.102:0/1765730575 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5c15b050 con 0x7f3e5c14ec40 2024-03-23T03:48:19.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.457+0000 7f3e62000640 1 -- 172.21.15.102:0/1765730575 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5c166460 con 0x7f3e5c14ec40 2024-03-23T03:48:19.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.458+0000 7f3e62000640 1 -- 172.21.15.102:0/1765730575 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e5c067080 con 0x7f3e5c14ec40 2024-03-23T03:48:19.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.459+0000 7f3e53400640 1 -- 172.21.15.102:0/1765730575 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3e5403e440 con 0x7f3e5c14ec40 2024-03-23T03:48:19.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.460+0000 7f3e53400640 1 --2- 172.21.15.102:0/1765730575 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e3c06ac00 0x7f3e3c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:19.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.460+0000 7f3e53400640 1 -- 172.21.15.102:0/1765730575 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f3e540aab10 con 0x7f3e5c14ec40 2024-03-23T03:48:19.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.461+0000 7f3e5b400640 1 --2- 172.21.15.102:0/1765730575 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e3c06ac00 0x7f3e3c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:19.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.461+0000 7f3e5b400640 1 --2- 172.21.15.102:0/1765730575 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e3c06ac00 0x7f3e3c06d0c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009950 tx=0x7f3e4c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:19.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.465+0000 7f3e53400640 1 -- 172.21.15.102:0/1765730575 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3e54075f10 con 0x7f3e5c14ec40 2024-03-23T03:48:19.681 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:48:19.688 DEBUG:teuthology.orchestra.run.smithi173:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:48:19.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.711+0000 7f3e62000640 1 -- 172.21.15.102:0/1765730575 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f3e5c073390 con 0x7f3e5c14ec40 2024-03-23T03:48:19.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.715+0000 7f3e53400640 1 -- 172.21.15.102:0/1765730575 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7f3e54039070 con 0x7f3e5c14ec40 2024-03-23T03:48:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.718+0000 7f3e62000640 1 -- 172.21.15.102:0/1765730575 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e3c06ac00 msgr2=0x7f3e3c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.718+0000 7f3e62000640 1 --2- 172.21.15.102:0/1765730575 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e3c06ac00 0x7f3e3c06d0c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009950 tx=0x7f3e4c008040 comp rx=0 tx=0).stop 2024-03-23T03:48:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.718+0000 7f3e62000640 1 -- 172.21.15.102:0/1765730575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e5c14ec40 msgr2=0x7f3e5c166980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.718+0000 7f3e62000640 1 --2- 172.21.15.102:0/1765730575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e5c14ec40 0x7f3e5c166980 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3e54037040 tx=0x7f3e54009750 comp rx=0 tx=0).stop 2024-03-23T03:48:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.718+0000 7f3e62000640 1 -- 172.21.15.102:0/1765730575 shutdown_connections 2024-03-23T03:48:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.718+0000 7f3e62000640 1 --2- 172.21.15.102:0/1765730575 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e3c06ac00 0x7f3e3c06d0c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.718+0000 7f3e62000640 1 --2- 172.21.15.102:0/1765730575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e5c14ec40 0x7f3e5c166980 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.718+0000 7f3e62000640 1 --2- 172.21.15.102:0/1765730575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e5c142a40 0x7f3e5c1404d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.718+0000 7f3e62000640 1 --2- 172.21.15.102:0/1765730575 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e40004d00 0x7f3e5c15a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.718+0000 7f3e62000640 1 -- 172.21.15.102:0/1765730575 >> 172.21.15.102:0/1765730575 conn(0x7f3e5c06a0b0 msgr2=0x7f3e5c0b3c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.718+0000 7f3e62000640 1 -- 172.21.15.102:0/1765730575 shutdown_connections 2024-03-23T03:48:19.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.718+0000 7f3e62000640 1 -- 172.21.15.102:0/1765730575 wait complete. 2024-03-23T03:48:19.736 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:48:19.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.962+0000 7f6cedc00640 1 Processor -- start 2024-03-23T03:48:19.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.962+0000 7f6cedc00640 1 -- start start 2024-03-23T03:48:19.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.963+0000 7f6cedc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6ce815b560 0x7f6ce815b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:19.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.963+0000 7f6cedc00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f6ce815d410 0x7f6ce80b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:19.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.963+0000 7f6cedc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ce806c6f0 con 0x7f6ce80b39b0 2024-03-23T03:48:19.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.963+0000 7f6cedc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce80b4370 con 0x7f6ce815b560 2024-03-23T03:48:19.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.963+0000 7f6cedc00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce806c9d0 con 0x7f6ce815d410 2024-03-23T03:48:19.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.964+0000 7f6ce6a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6ce815b560 0x7f6ce815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:19.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.964+0000 7f6ce6000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6ce80b39b0 0x7f6ce815e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50420/0 (socket says 172.21.15.102:50420) 2024-03-23T03:48:19.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.964+0000 7f6ce6000640 1 -- 172.21.15.102:0/3151698632 learned_addr learned my addr 172.21.15.102:0/3151698632 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:19.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.964+0000 7f6ce5600640 1 -- 172.21.15.102:0/3151698632 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280770664 0 0) 0x7f6ce806c6f0 con 0x7f6ce80b39b0 2024-03-23T03:48:19.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.964+0000 7f6ce7400640 1 --2- 172.21.15.102:0/3151698632 >> v2:172.21.15.173:3300/0 conn(0x7f6ce815d410 0x7f6ce80b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:19.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.964+0000 7f6ce5600640 1 -- 172.21.15.102:0/3151698632 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ccc003520 con 0x7f6ce80b39b0 2024-03-23T03:48:19.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.964+0000 7f6ce6a00640 1 -- 172.21.15.102:0/3151698632 >> v2:172.21.15.173:3300/0 conn(0x7f6ce815d410 msgr2=0x7f6ce80b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:19.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.964+0000 7f6ce6a00640 1 --2- 172.21.15.102:0/3151698632 >> v2:172.21.15.173:3300/0 conn(0x7f6ce815d410 0x7f6ce80b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.964+0000 7f6ce6a00640 1 -- 172.21.15.102:0/3151698632 >> v1:172.21.15.102:6789/0 conn(0x7f6ce80b39b0 legacy=0x7f6ce815e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:19.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.964+0000 7f6ce6a00640 1 -- 172.21.15.102:0/3151698632 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce806ccb0 con 0x7f6ce815b560 2024-03-23T03:48:19.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.964+0000 7f6ce7400640 1 --2- 172.21.15.102:0/3151698632 >> v2:172.21.15.173:3300/0 conn(0x7f6ce815d410 0x7f6ce80b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:19.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.966+0000 7f6ce6a00640 1 --2- 172.21.15.102:0/3151698632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce815b560 0x7f6ce815b940 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8070100 tx=0x7f6cd802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4b8c6b54b5612b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:19.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.966+0000 7f6ce5600640 1 -- 172.21.15.102:0/3151698632 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd802e070 con 0x7f6ce815b560 2024-03-23T03:48:19.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.966+0000 7f6ce5600640 1 -- 172.21.15.102:0/3151698632 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd80315a0 con 0x7f6ce815b560 2024-03-23T03:48:19.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.966+0000 7f6ce5600640 1 -- 172.21.15.102:0/3151698632 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8031910 con 0x7f6ce815b560 2024-03-23T03:48:19.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.966+0000 7f6cedc00640 1 -- 172.21.15.102:0/3151698632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce815b560 msgr2=0x7f6ce815b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:19.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.966+0000 7f6cedc00640 1 --2- 172.21.15.102:0/3151698632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce815b560 0x7f6ce815b940 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8070100 tx=0x7f6cd802b840 comp rx=0 tx=0).stop 2024-03-23T03:48:19.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.966+0000 7f6cedc00640 1 -- 172.21.15.102:0/3151698632 shutdown_connections 2024-03-23T03:48:19.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.966+0000 7f6cedc00640 1 --2- 172.21.15.102:0/3151698632 >> v2:172.21.15.173:3300/0 conn(0x7f6ce815d410 0x7f6ce80b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.966+0000 7f6cedc00640 1 --2- 172.21.15.102:0/3151698632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce815b560 0x7f6ce815b940 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.966+0000 7f6cedc00640 1 -- 172.21.15.102:0/3151698632 >> 172.21.15.102:0/3151698632 conn(0x7f6ce806a3d0 msgr2=0x7f6ce80b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:19.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.967+0000 7f6cedc00640 1 -- 172.21.15.102:0/3151698632 shutdown_connections 2024-03-23T03:48:19.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.967+0000 7f6cedc00640 1 -- 172.21.15.102:0/3151698632 wait complete. 2024-03-23T03:48:19.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.968+0000 7f6cedc00640 1 Processor -- start 2024-03-23T03:48:19.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.968+0000 7f6cedc00640 1 -- start start 2024-03-23T03:48:19.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.969+0000 7f6cedc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ce80b39b0 0x7f6ce81f0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:19.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.969+0000 7f6cedc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce815b560 0x7f6ce81f2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:19.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.969+0000 7f6cedc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ce815d410 0x7f6ce8151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:19.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.969+0000 7f6cedc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce81f18f0 con 0x7f6ce815b560 2024-03-23T03:48:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.969+0000 7f6cedc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce81f1bd0 con 0x7f6ce815d410 2024-03-23T03:48:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.969+0000 7f6cedc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce81f1eb0 con 0x7f6ce80b39b0 2024-03-23T03:48:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.969+0000 7f6ce6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce815b560 0x7f6ce81f2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.969+0000 7f6ce6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce815b560 0x7f6ce81f2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43614/0 (socket says 172.21.15.102:43614) 2024-03-23T03:48:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.969+0000 7f6ce6000640 1 -- 172.21.15.102:0/3964706955 learned_addr learned my addr 172.21.15.102:0/3964706955 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.970+0000 7f6ce6000640 1 -- 172.21.15.102:0/3964706955 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ce80b39b0 msgr2=0x7f6ce81f0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.970+0000 7f6ce6000640 1 --2- 172.21.15.102:0/3964706955 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ce80b39b0 0x7f6ce81f0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.970+0000 7f6ce6000640 1 -- 172.21.15.102:0/3964706955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ce815d410 msgr2=0x7f6ce8151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.970+0000 7f6ce6000640 1 --2- 172.21.15.102:0/3964706955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ce815d410 0x7f6ce8151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.970+0000 7f6ce6000640 1 -- 172.21.15.102:0/3964706955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd8009400 con 0x7f6ce815b560 2024-03-23T03:48:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.970+0000 7f6ce6000640 1 --2- 172.21.15.102:0/3964706955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce815b560 0x7f6ce81f2990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400c430 tx=0x7f6cd400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.970+0000 7f6cdea00640 1 -- 172.21.15.102:0/3964706955 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4005ce0 con 0x7f6ce815b560 2024-03-23T03:48:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.970+0000 7f6cdea00640 1 -- 172.21.15.102:0/3964706955 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd4005e40 con 0x7f6ce815b560 2024-03-23T03:48:19.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.970+0000 7f6cedc00640 1 -- 172.21.15.102:0/3964706955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce81f2190 con 0x7f6ce815b560 2024-03-23T03:48:19.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.970+0000 7f6cdea00640 1 -- 172.21.15.102:0/3964706955 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd40072f0 con 0x7f6ce815b560 2024-03-23T03:48:19.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.970+0000 7f6cedc00640 1 -- 172.21.15.102:0/3964706955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce81f1530 con 0x7f6ce815b560 2024-03-23T03:48:19.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.972+0000 7f6cedc00640 1 -- 172.21.15.102:0/3964706955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce81f26b0 con 0x7f6ce815b560 2024-03-23T03:48:19.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.972+0000 7f6cdea00640 1 -- 172.21.15.102:0/3964706955 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6cd4007450 con 0x7f6ce815b560 2024-03-23T03:48:19.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.973+0000 7f6cdea00640 1 --2- 172.21.15.102:0/3964706955 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6cc806ac00 0x7f6cc806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:19.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.973+0000 7f6cdea00640 1 -- 172.21.15.102:0/3964706955 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f6cd40888e0 con 0x7f6ce815b560 2024-03-23T03:48:19.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.973+0000 7f6ce6a00640 1 --2- 172.21.15.102:0/3964706955 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6cc806ac00 0x7f6cc806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:19.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.974+0000 7f6ce6a00640 1 --2- 172.21.15.102:0/3964706955 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6cc806ac00 0x7f6cc806d0c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8036000 tx=0x7f6cd8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:19.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:19.978+0000 7f6cdea00640 1 -- 172.21.15.102:0/3964706955 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6ce81f26b0 con 0x7f6ce815b560 2024-03-23T03:48:20.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.171+0000 7f6cedc00640 1 -- 172.21.15.102:0/3964706955 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ce81f26b0 con 0x7f6ce815b560 2024-03-23T03:48:20.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.173+0000 7f6cdea00640 1 -- 172.21.15.102:0/3964706955 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+12642 (secure 0 0 0) 0x7f6ce81f26b0 con 0x7f6ce815b560 2024-03-23T03:48:20.173 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:20.174 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":37,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:16.197300+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:48:20.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.175+0000 7f6cedc00640 1 -- 172.21.15.102:0/3964706955 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6cc806ac00 msgr2=0x7f6cc806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:20.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.175+0000 7f6cedc00640 1 --2- 172.21.15.102:0/3964706955 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6cc806ac00 0x7f6cc806d0c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8036000 tx=0x7f6cd8030000 comp rx=0 tx=0).stop 2024-03-23T03:48:20.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.176+0000 7f6cedc00640 1 -- 172.21.15.102:0/3964706955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce815b560 msgr2=0x7f6ce81f2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:20.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.176+0000 7f6cedc00640 1 --2- 172.21.15.102:0/3964706955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce815b560 0x7f6ce81f2990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f6cd400c430 tx=0x7f6cd400c8f0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.176+0000 7f6cedc00640 1 -- 172.21.15.102:0/3964706955 shutdown_connections 2024-03-23T03:48:20.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.176+0000 7f6cedc00640 1 --2- 172.21.15.102:0/3964706955 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6cc806ac00 0x7f6cc806d0c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.176+0000 7f6cedc00640 1 --2- 172.21.15.102:0/3964706955 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ce815d410 0x7f6ce8151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.176+0000 7f6cedc00640 1 --2- 172.21.15.102:0/3964706955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce815b560 0x7f6ce81f2990 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.176+0000 7f6cedc00640 1 --2- 172.21.15.102:0/3964706955 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ce80b39b0 0x7f6ce81f0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.176+0000 7f6cedc00640 1 -- 172.21.15.102:0/3964706955 >> 172.21.15.102:0/3964706955 conn(0x7f6ce806a3d0 msgr2=0x7f6ce8158aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:20.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.176+0000 7f6cedc00640 1 -- 172.21.15.102:0/3964706955 shutdown_connections 2024-03-23T03:48:20.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.176+0000 7f6cedc00640 1 -- 172.21.15.102:0/3964706955 wait complete. 2024-03-23T03:48:20.195 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:48:20.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:48:20.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.398+0000 7f93b9c00640 1 Processor -- start 2024-03-23T03:48:20.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.398+0000 7f93b9c00640 1 -- start start 2024-03-23T03:48:20.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.399+0000 7f93b9c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f93b40cd6b0 0x7f93b406baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:20.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.399+0000 7f93b9c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b406b0a0 con 0x7f93b415cd10 2024-03-23T03:48:20.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.399+0000 7f93b9c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93b40d2330 con 0x7f93b40b1b60 2024-03-23T03:48:20.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.399+0000 7f93b9c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93b40d2610 con 0x7f93b40cd6b0 2024-03-23T03:48:20.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.399+0000 7f93b2000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f93b415cd10 0x7f93b40cda90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50434/0 (socket says 172.21.15.102:50434) 2024-03-23T03:48:20.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.399+0000 7f93b2000640 1 -- 172.21.15.102:0/4002423001 learned_addr learned my addr 172.21.15.102:0/4002423001 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:20.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.400+0000 7f93b1600640 1 -- 172.21.15.102:0/4002423001 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588448473 0 0) 0x7f93b406b0a0 con 0x7f93b415cd10 2024-03-23T03:48:20.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.400+0000 7f93b2a00640 1 --2- 172.21.15.102:0/4002423001 >> v2:172.21.15.173:3300/0 conn(0x7f93b40cd6b0 0x7f93b406baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:20.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.400+0000 7f93b1600640 1 -- 172.21.15.102:0/4002423001 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9398003520 con 0x7f93b415cd10 2024-03-23T03:48:20.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.400+0000 7f93b1600640 1 -- 172.21.15.102:0/4002423001 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3686046236 0 0) 0x7f9398003520 con 0x7f93b415cd10 2024-03-23T03:48:20.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.400+0000 7f93b1600640 1 -- 172.21.15.102:0/4002423001 >> v2:172.21.15.173:3300/0 conn(0x7f93b40cd6b0 msgr2=0x7f93b406baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:20.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.400+0000 7f93b1600640 1 --2- 172.21.15.102:0/4002423001 >> v2:172.21.15.173:3300/0 conn(0x7f93b40cd6b0 0x7f93b406baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.400+0000 7f93b1600640 1 -- 172.21.15.102:0/4002423001 >> v1:172.21.15.173:6789/0 conn(0x7f93b40b1b60 legacy=0x7f93b40d1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:20.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.400+0000 7f93b1600640 1 -- 172.21.15.102:0/4002423001 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b406b2f0 con 0x7f93b415cd10 2024-03-23T03:48:20.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.400+0000 7f93b1600640 1 -- 172.21.15.102:0/4002423001 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f93a0001670 con 0x7f93b415cd10 2024-03-23T03:48:20.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.400+0000 7f93b1600640 1 -- 172.21.15.102:0/4002423001 >> v1:172.21.15.102:6789/0 conn(0x7f93b415cd10 legacy=0x7f93b40cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:20.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.400+0000 7f93b1600640 1 --2- 172.21.15.102:0/4002423001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9398003970 0x7f9398003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:20.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.400+0000 7f93b1600640 1 --2- 172.21.15.102:0/4002423001 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f93980047c0 0x7f9398004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:20.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.400+0000 7f93b1600640 1 --2- 172.21.15.102:0/4002423001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398005180 0x7f939800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:20.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.400+0000 7f93b1600640 1 -- 172.21.15.102:0/4002423001 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b40d2610 con 0x7f9398005180 2024-03-23T03:48:20.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.400+0000 7f93b1600640 1 -- 172.21.15.102:0/4002423001 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939800c0b0 con 0x7f9398003970 2024-03-23T03:48:20.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.400+0000 7f93b1600640 1 -- 172.21.15.102:0/4002423001 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f939800c270 con 0x7f93980047c0 2024-03-23T03:48:20.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.401+0000 7f93b3400640 1 --2- 172.21.15.102:0/4002423001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398005180 0x7f939800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:20.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.401+0000 7f93b2000640 1 --2- 172.21.15.102:0/4002423001 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f93980047c0 0x7f9398004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:20.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.401+0000 7f93b2a00640 1 --2- 172.21.15.102:0/4002423001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9398003970 0x7f9398003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:20.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.401+0000 7f93b3400640 1 -- 172.21.15.102:0/4002423001 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f93980047c0 msgr2=0x7f9398004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:20.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.401+0000 7f93b3400640 1 --2- 172.21.15.102:0/4002423001 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f93980047c0 0x7f9398004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.401+0000 7f93b3400640 1 -- 172.21.15.102:0/4002423001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9398003970 msgr2=0x7f9398003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:20.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.401+0000 7f93b3400640 1 --2- 172.21.15.102:0/4002423001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9398003970 0x7f9398003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.401+0000 7f93b3400640 1 -- 172.21.15.102:0/4002423001 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9398003520 con 0x7f9398005180 2024-03-23T03:48:20.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.401+0000 7f93b2a00640 1 --2- 172.21.15.102:0/4002423001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9398003970 0x7f9398003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:20.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.402+0000 7f93b3400640 1 -- 172.21.15.102:0/4002423001 reap_dead start 2024-03-23T03:48:20.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.402+0000 7f93b3400640 1 --2- 172.21.15.102:0/4002423001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398005180 0x7f939800b980 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f93ac003d80 tx=0x7f93980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51759db53bb4d11f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.402+0000 7f93b1600640 1 -- 172.21.15.102:0/4002423001 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac007220 con 0x7f9398005180 2024-03-23T03:48:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.402+0000 7f93b1600640 1 -- 172.21.15.102:0/4002423001 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93ac02e050 con 0x7f9398005180 2024-03-23T03:48:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.402+0000 7f93b1600640 1 -- 172.21.15.102:0/4002423001 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93ac0075e0 con 0x7f9398005180 2024-03-23T03:48:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.403+0000 7f93b9c00640 1 -- 172.21.15.102:0/4002423001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398005180 msgr2=0x7f939800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.403+0000 7f93b9c00640 1 --2- 172.21.15.102:0/4002423001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398005180 0x7f939800b980 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f93ac003d80 tx=0x7f93980057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.403+0000 7f93b9c00640 1 -- 172.21.15.102:0/4002423001 shutdown_connections 2024-03-23T03:48:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.403+0000 7f93b9c00640 1 --2- 172.21.15.102:0/4002423001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398005180 0x7f939800b980 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.403+0000 7f93b9c00640 1 -- 172.21.15.102:0/4002423001 >> 172.21.15.102:0/4002423001 conn(0x7f93b40cf5d0 msgr2=0x7f93b40cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:20.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.403+0000 7f93b9c00640 1 -- 172.21.15.102:0/4002423001 shutdown_connections 2024-03-23T03:48:20.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.403+0000 7f93b9c00640 1 -- 172.21.15.102:0/4002423001 wait complete. 2024-03-23T03:48:20.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.405+0000 7f93b9c00640 1 Processor -- start 2024-03-23T03:48:20.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.405+0000 7f93b9c00640 1 -- start start 2024-03-23T03:48:20.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.407+0000 7f93b9c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398005180 0x7f93b41558f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:20.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.407+0000 7f93b9c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93b41519b0 0x7f93b40c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:20.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.407+0000 7f93b9c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f93b41587a0 0x7f93b4158c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:20.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.407+0000 7f93b9c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b4159160 con 0x7f9398005180 2024-03-23T03:48:20.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.407+0000 7f93b9c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b40c7d20 con 0x7f93b41519b0 2024-03-23T03:48:20.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.407+0000 7f93b9c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93b40c8000 con 0x7f93b41587a0 2024-03-23T03:48:20.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.407+0000 7f93b2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398005180 0x7f93b41558f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:20.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.407+0000 7f93b2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398005180 0x7f93b41558f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43642/0 (socket says 172.21.15.102:43642) 2024-03-23T03:48:20.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.407+0000 7f93b2a00640 1 -- 172.21.15.102:0/304366392 learned_addr learned my addr 172.21.15.102:0/304366392 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:20.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.407+0000 7f93b2a00640 1 -- 172.21.15.102:0/304366392 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f93b41587a0 msgr2=0x7f93b4158c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:20.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.407+0000 7f93b2a00640 1 --2- 172.21.15.102:0/304366392 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f93b41587a0 0x7f93b4158c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.407+0000 7f93b2a00640 1 -- 172.21.15.102:0/304366392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93b41519b0 msgr2=0x7f93b40c74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:20.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.407+0000 7f93b2a00640 1 --2- 172.21.15.102:0/304366392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93b41519b0 0x7f93b40c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.407+0000 7f93b2a00640 1 -- 172.21.15.102:0/304366392 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b40c82e0 con 0x7f9398005180 2024-03-23T03:48:20.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.408+0000 7f93b2a00640 1 --2- 172.21.15.102:0/304366392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398005180 0x7f93b41558f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f93a4009fd0 tx=0x7f93a400f5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:20.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.408+0000 7f93aaa00640 1 -- 172.21.15.102:0/304366392 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a40145f0 con 0x7f9398005180 2024-03-23T03:48:20.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.408+0000 7f93aaa00640 1 -- 172.21.15.102:0/304366392 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93a4014750 con 0x7f9398005180 2024-03-23T03:48:20.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.408+0000 7f93aaa00640 1 -- 172.21.15.102:0/304366392 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a4014ae0 con 0x7f9398005180 2024-03-23T03:48:20.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.408+0000 7f93b9c00640 1 -- 172.21.15.102:0/304366392 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b40c85c0 con 0x7f9398005180 2024-03-23T03:48:20.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.408+0000 7f93b9c00640 1 -- 172.21.15.102:0/304366392 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b40c8dc0 con 0x7f9398005180 2024-03-23T03:48:20.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.409+0000 7f93b9c00640 1 -- 172.21.15.102:0/304366392 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b4073390 con 0x7f9398005180 2024-03-23T03:48:20.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.410+0000 7f93aaa00640 1 -- 172.21.15.102:0/304366392 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f93a4014c40 con 0x7f9398005180 2024-03-23T03:48:20.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.411+0000 7f93aaa00640 1 --2- 172.21.15.102:0/304366392 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f939406ac00 0x7f939406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:20.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.412+0000 7f93aaa00640 1 -- 172.21.15.102:0/304366392 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f93a4089ec0 con 0x7f9398005180 2024-03-23T03:48:20.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.412+0000 7f93b2000640 1 --2- 172.21.15.102:0/304366392 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f939406ac00 0x7f939406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:20.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.412+0000 7f93b2000640 1 --2- 172.21.15.102:0/304366392 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f939406ac00 0x7f939406d0c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f93b40caa20 tx=0x7f93a0000fb0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:20.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.416+0000 7f93aaa00640 1 -- 172.21.15.102:0/304366392 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f93a4055340 con 0x7f9398005180 2024-03-23T03:48:20.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.608+0000 7f93b9c00640 1 -- 172.21.15.102:0/304366392 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f93b40cdb20 con 0x7f9398005180 2024-03-23T03:48:20.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.609+0000 7f93aaa00640 1 -- 172.21.15.102:0/304366392 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v37) v1 ==== 93+0+10 (secure 0 0 0) 0x7f93a4058960 con 0x7f9398005180 2024-03-23T03:48:20.610 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:48:20.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.612+0000 7f93b9c00640 1 -- 172.21.15.102:0/304366392 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f939406ac00 msgr2=0x7f939406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:20.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.612+0000 7f93b9c00640 1 --2- 172.21.15.102:0/304366392 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f939406ac00 0x7f939406d0c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f93b40caa20 tx=0x7f93a0000fb0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.612+0000 7f93b9c00640 1 -- 172.21.15.102:0/304366392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398005180 msgr2=0x7f93b41558f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:20.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.612+0000 7f93b9c00640 1 --2- 172.21.15.102:0/304366392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398005180 0x7f93b41558f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f93a4009fd0 tx=0x7f93a400f5e0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.612+0000 7f93b9c00640 1 -- 172.21.15.102:0/304366392 shutdown_connections 2024-03-23T03:48:20.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.612+0000 7f93b9c00640 1 --2- 172.21.15.102:0/304366392 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f939406ac00 0x7f939406d0c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.612+0000 7f93b9c00640 1 --2- 172.21.15.102:0/304366392 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f93b41587a0 0x7f93b4158c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.612+0000 7f93b9c00640 1 --2- 172.21.15.102:0/304366392 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93b41519b0 0x7f93b40c74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.612+0000 7f93b9c00640 1 --2- 172.21.15.102:0/304366392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398005180 0x7f93b41558f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.612+0000 7f93b9c00640 1 -- 172.21.15.102:0/304366392 >> 172.21.15.102:0/304366392 conn(0x7f93b40cf5d0 msgr2=0x7f93b40cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:20.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.612+0000 7f93b9c00640 1 -- 172.21.15.102:0/304366392 shutdown_connections 2024-03-23T03:48:20.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.612+0000 7f93b9c00640 1 -- 172.21.15.102:0/304366392 wait complete. 2024-03-23T03:48:20.630 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T03:48:20.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.835+0000 7f24e1200640 1 Processor -- start 2024-03-23T03:48:20.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.835+0000 7f24e1200640 1 -- start start 2024-03-23T03:48:20.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.837+0000 7f24e1200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24dc15cab0 0x7f24dc0b8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:20.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.837+0000 7f24e1200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f24dc0ceeb0 0x7f24dc0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:20.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.837+0000 7f24e1200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24dc0cf850 con 0x7f24dc0b4120 2024-03-23T03:48:20.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.837+0000 7f24e1200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24dc15ce90 con 0x7f24dc15cab0 2024-03-23T03:48:20.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.837+0000 7f24e1200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc0cddf0 con 0x7f24dc0ceeb0 2024-03-23T03:48:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.837+0000 7f24db400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24dc15cab0 0x7f24dc0b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.837+0000 7f24db400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f24dc15cab0 0x7f24dc0b8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43648/0 (socket says 172.21.15.102:43648) 2024-03-23T03:48:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.837+0000 7f24db400640 1 -- 172.21.15.102:0/3192515462 learned_addr learned my addr 172.21.15.102:0/3192515462 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.838+0000 7f24daa00640 1 --2- 172.21.15.102:0/3192515462 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f24dc0ceeb0 0x7f24dc0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.838+0000 7f24db400640 1 -- 172.21.15.102:0/3192515462 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f24dc0ceeb0 msgr2=0x7f24dc0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.838+0000 7f24db400640 1 --2- 172.21.15.102:0/3192515462 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f24dc0ceeb0 0x7f24dc0cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.838+0000 7f24db400640 1 -- 172.21.15.102:0/3192515462 >> v1:172.21.15.173:6789/0 conn(0x7f24dc0b4120 legacy=0x7f24dc0b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:20.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.838+0000 7f24db400640 1 -- 172.21.15.102:0/3192515462 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc0cd570 con 0x7f24dc15cab0 2024-03-23T03:48:20.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.839+0000 7f24db400640 1 --2- 172.21.15.102:0/3192515462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24dc15cab0 0x7f24dc0b8050 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f24cc009870 tx=0x7f24cc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=177574116258db46 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:20.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.839+0000 7f24da000640 1 -- 172.21.15.102:0/3192515462 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24cc031640 con 0x7f24dc15cab0 2024-03-23T03:48:20.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.839+0000 7f24da000640 1 -- 172.21.15.102:0/3192515462 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24cc0317a0 con 0x7f24dc15cab0 2024-03-23T03:48:20.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.839+0000 7f24da000640 1 -- 172.21.15.102:0/3192515462 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24cc031b30 con 0x7f24dc15cab0 2024-03-23T03:48:20.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.840+0000 7f24e1200640 1 -- 172.21.15.102:0/3192515462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24dc15cab0 msgr2=0x7f24dc0b8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:20.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.840+0000 7f24e1200640 1 --2- 172.21.15.102:0/3192515462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24dc15cab0 0x7f24dc0b8050 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f24cc009870 tx=0x7f24cc02b8f0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.840+0000 7f24e1200640 1 -- 172.21.15.102:0/3192515462 shutdown_connections 2024-03-23T03:48:20.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.840+0000 7f24e1200640 1 --2- 172.21.15.102:0/3192515462 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f24dc0ceeb0 0x7f24dc0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.840+0000 7f24e1200640 1 --2- 172.21.15.102:0/3192515462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24dc15cab0 0x7f24dc0b8050 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.840+0000 7f24e1200640 1 -- 172.21.15.102:0/3192515462 >> 172.21.15.102:0/3192515462 conn(0x7f24dc06ae90 msgr2=0x7f24dc06b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:20.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.840+0000 7f24e1200640 1 -- 172.21.15.102:0/3192515462 shutdown_connections 2024-03-23T03:48:20.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.840+0000 7f24e1200640 1 -- 172.21.15.102:0/3192515462 wait complete. 2024-03-23T03:48:20.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.842+0000 7f24e1200640 1 Processor -- start 2024-03-23T03:48:20.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.842+0000 7f24e1200640 1 -- start start 2024-03-23T03:48:20.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.843+0000 7f24e1200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24dc0b4120 0x7f24dc0b6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:20.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.843+0000 7f24e1200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24dc0ceeb0 0x7f24dc1f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:20.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.843+0000 7f24e1200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f24dc15cab0 0x7f24dc15e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:20.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.843+0000 7f24e1200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc1f30a0 con 0x7f24dc0ceeb0 2024-03-23T03:48:20.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.843+0000 7f24e1200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc0c9880 con 0x7f24dc0b4120 2024-03-23T03:48:20.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.843+0000 7f24e1200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc1f5fe0 con 0x7f24dc15cab0 2024-03-23T03:48:20.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.843+0000 7f24daa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24dc0ceeb0 0x7f24dc1f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:20.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.843+0000 7f24daa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24dc0ceeb0 0x7f24dc1f2b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43662/0 (socket says 172.21.15.102:43662) 2024-03-23T03:48:20.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.843+0000 7f24daa00640 1 -- 172.21.15.102:0/4220829323 learned_addr learned my addr 172.21.15.102:0/4220829323 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:20.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.843+0000 7f24db400640 1 --2- 172.21.15.102:0/4220829323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24dc0b4120 0x7f24dc0b6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:20.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.844+0000 7f24dbe00640 1 --2- 172.21.15.102:0/4220829323 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f24dc15cab0 0x7f24dc15e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:20.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.844+0000 7f24daa00640 1 -- 172.21.15.102:0/4220829323 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f24dc15cab0 msgr2=0x7f24dc15e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:20.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.844+0000 7f24daa00640 1 --2- 172.21.15.102:0/4220829323 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f24dc15cab0 0x7f24dc15e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.844+0000 7f24daa00640 1 -- 172.21.15.102:0/4220829323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24dc0b4120 msgr2=0x7f24dc0b6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:20.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.844+0000 7f24daa00640 1 --2- 172.21.15.102:0/4220829323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24dc0b4120 0x7f24dc0b6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:20.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.844+0000 7f24daa00640 1 -- 172.21.15.102:0/4220829323 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24cc0094d0 con 0x7f24dc0ceeb0 2024-03-23T03:48:20.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.844+0000 7f24db400640 1 --2- 172.21.15.102:0/4220829323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24dc0b4120 0x7f24dc0b6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:20.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.844+0000 7f24dbe00640 1 --2- 172.21.15.102:0/4220829323 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f24dc15cab0 0x7f24dc15e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:20.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.844+0000 7f24daa00640 1 --2- 172.21.15.102:0/4220829323 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24dc0ceeb0 0x7f24dc1f2b60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f24c400ce10 tx=0x7f24c400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:20.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.844+0000 7f24d3400640 1 -- 172.21.15.102:0/4220829323 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c4003b90 con 0x7f24dc0ceeb0 2024-03-23T03:48:20.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.844+0000 7f24d3400640 1 -- 172.21.15.102:0/4220829323 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c4003cf0 con 0x7f24dc0ceeb0 2024-03-23T03:48:20.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.844+0000 7f24d3400640 1 -- 172.21.15.102:0/4220829323 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24c4003ea0 con 0x7f24dc0ceeb0 2024-03-23T03:48:20.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.844+0000 7f24e1200640 1 -- 172.21.15.102:0/4220829323 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24dc0b7160 con 0x7f24dc0ceeb0 2024-03-23T03:48:20.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.844+0000 7f24e1200640 1 -- 172.21.15.102:0/4220829323 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24dc1f5990 con 0x7f24dc0ceeb0 2024-03-23T03:48:20.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.846+0000 7f24e1200640 1 -- 172.21.15.102:0/4220829323 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24dc04e030 con 0x7f24dc0ceeb0 2024-03-23T03:48:20.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.847+0000 7f24d3400640 1 -- 172.21.15.102:0/4220829323 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f24c4004240 con 0x7f24dc0ceeb0 2024-03-23T03:48:20.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.847+0000 7f24d3400640 1 --2- 172.21.15.102:0/4220829323 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f24b806ac30 0x7f24b806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:20.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.848+0000 7f24d3400640 1 -- 172.21.15.102:0/4220829323 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f24c4012070 con 0x7f24dc0ceeb0 2024-03-23T03:48:20.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.848+0000 7f24db400640 1 --2- 172.21.15.102:0/4220829323 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f24b806ac30 0x7f24b806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:20.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.848+0000 7f24db400640 1 --2- 172.21.15.102:0/4220829323 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f24b806ac30 0x7f24b806d0f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f24cc036000 tx=0x7f24cc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:20.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:20.852+0000 7f24d3400640 1 -- 172.21.15.102:0/4220829323 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f24c4054f30 con 0x7f24dc0ceeb0 2024-03-23T03:48:21.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.055+0000 7f24e1200640 1 -- 172.21.15.102:0/4220829323 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f24dc0d02d0 con 0x7f24dc0ceeb0 2024-03-23T03:48:21.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.055+0000 7f24d3400640 1 -- 172.21.15.102:0/4220829323 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v37) v1 ==== 84+0+65 (secure 0 0 0) 0x7f24c4058550 con 0x7f24dc0ceeb0 2024-03-23T03:48:21.056 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:21.056 INFO:teuthology.orchestra.run.smithi102.stdout:[ 2024-03-23T03:48:21.056 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:21.056 INFO:teuthology.orchestra.run.smithi102.stdout: "poolnum": 1, 2024-03-23T03:48:21.056 INFO:teuthology.orchestra.run.smithi102.stdout: "poolname": ".mgr" 2024-03-23T03:48:21.056 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:21.056 INFO:teuthology.orchestra.run.smithi102.stdout:] 2024-03-23T03:48:21.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.058+0000 7f24e1200640 1 -- 172.21.15.102:0/4220829323 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f24b806ac30 msgr2=0x7f24b806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:21.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.058+0000 7f24e1200640 1 --2- 172.21.15.102:0/4220829323 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f24b806ac30 0x7f24b806d0f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f24cc036000 tx=0x7f24cc03c040 comp rx=0 tx=0).stop 2024-03-23T03:48:21.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.058+0000 7f24e1200640 1 -- 172.21.15.102:0/4220829323 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24dc0ceeb0 msgr2=0x7f24dc1f2b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:21.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.058+0000 7f24e1200640 1 --2- 172.21.15.102:0/4220829323 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24dc0ceeb0 0x7f24dc1f2b60 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f24c400ce10 tx=0x7f24c400a660 comp rx=0 tx=0).stop 2024-03-23T03:48:21.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.058+0000 7f24e1200640 1 -- 172.21.15.102:0/4220829323 shutdown_connections 2024-03-23T03:48:21.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.058+0000 7f24e1200640 1 --2- 172.21.15.102:0/4220829323 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f24b806ac30 0x7f24b806d0f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:21.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.058+0000 7f24e1200640 1 --2- 172.21.15.102:0/4220829323 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f24dc15cab0 0x7f24dc15e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:21.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.058+0000 7f24e1200640 1 --2- 172.21.15.102:0/4220829323 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24dc0ceeb0 0x7f24dc1f2b60 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:21.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.058+0000 7f24e1200640 1 --2- 172.21.15.102:0/4220829323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f24dc0b4120 0x7f24dc0b6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:21.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.058+0000 7f24e1200640 1 -- 172.21.15.102:0/4220829323 >> 172.21.15.102:0/4220829323 conn(0x7f24dc06ae90 msgr2=0x7f24dc1602c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:21.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.058+0000 7f24e1200640 1 -- 172.21.15.102:0/4220829323 shutdown_connections 2024-03-23T03:48:21.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.058+0000 7f24e1200640 1 -- 172.21.15.102:0/4220829323 wait complete. 2024-03-23T03:48:21.077 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T03:48:21.077 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T03:48:21.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.287+0000 7fa6c4c00640 1 Processor -- start 2024-03-23T03:48:21.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.287+0000 7fa6c4c00640 1 -- start start 2024-03-23T03:48:21.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.288+0000 7fa6c4c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa6c01456a0 0x7fa6c00ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:21.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.288+0000 7fa6c4c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6c0070cd0 0x7fa6c00cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:21.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.288+0000 7fa6c4c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c006d390 con 0x7fa6c0145c20 2024-03-23T03:48:21.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.288+0000 7fa6c4c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c00ad890 con 0x7fa6c01456a0 2024-03-23T03:48:21.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.288+0000 7fa6c4c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c0145a80 con 0x7fa6c0070cd0 2024-03-23T03:48:21.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.289+0000 7fa6bde00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa6c01456a0 0x7fa6c00ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:21.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.289+0000 7fa6bde00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa6c01456a0 0x7fa6c00ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43666/0 (socket says 172.21.15.102:43666) 2024-03-23T03:48:21.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.289+0000 7fa6bde00640 1 -- 172.21.15.102:0/14421154 learned_addr learned my addr 172.21.15.102:0/14421154 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:21.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.289+0000 7fa6b5400640 1 --2- 172.21.15.102:0/14421154 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6c0070cd0 0x7fa6c00cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:21.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.289+0000 7fa6bde00640 1 -- 172.21.15.102:0/14421154 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6c0070cd0 msgr2=0x7fa6c00cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:21.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.289+0000 7fa6bde00640 1 --2- 172.21.15.102:0/14421154 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6c0070cd0 0x7fa6c00cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:21.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.289+0000 7fa6bde00640 1 -- 172.21.15.102:0/14421154 >> v1:172.21.15.102:6789/0 conn(0x7fa6c0145c20 legacy=0x7fa6c00ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:21.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.289+0000 7fa6bde00640 1 -- 172.21.15.102:0/14421154 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c006d670 con 0x7fa6c01456a0 2024-03-23T03:48:21.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.289+0000 7fa6bd400640 1 -- 172.21.15.102:0/14421154 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004351708 0 0) 0x7fa6c006d390 con 0x7fa6c0145c20 2024-03-23T03:48:21.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.290+0000 7fa6bde00640 1 --2- 172.21.15.102:0/14421154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6c01456a0 0x7fa6c00ce930 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009e50 tx=0x7fa6b002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5888fdb5316e9861 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:21.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.291+0000 7fa6bd400640 1 -- 172.21.15.102:0/14421154 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b002cb00 con 0x7fa6c01456a0 2024-03-23T03:48:21.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.291+0000 7fa6bd400640 1 -- 172.21.15.102:0/14421154 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b002cc60 con 0x7fa6c01456a0 2024-03-23T03:48:21.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.291+0000 7fa6bd400640 1 -- 172.21.15.102:0/14421154 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b00343f0 con 0x7fa6c01456a0 2024-03-23T03:48:21.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.291+0000 7fa6c4c00640 1 -- 172.21.15.102:0/14421154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6c01456a0 msgr2=0x7fa6c00ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:21.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.291+0000 7fa6c4c00640 1 --2- 172.21.15.102:0/14421154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6c01456a0 0x7fa6c00ce930 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009e50 tx=0x7fa6b002bf60 comp rx=0 tx=0).stop 2024-03-23T03:48:21.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.292+0000 7fa6c4c00640 1 -- 172.21.15.102:0/14421154 shutdown_connections 2024-03-23T03:48:21.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.292+0000 7fa6c4c00640 1 --2- 172.21.15.102:0/14421154 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6c0070cd0 0x7fa6c00cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:21.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.292+0000 7fa6c4c00640 1 --2- 172.21.15.102:0/14421154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6c01456a0 0x7fa6c00ce930 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:21.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.292+0000 7fa6c4c00640 1 -- 172.21.15.102:0/14421154 >> 172.21.15.102:0/14421154 conn(0x7fa6c006a0b0 msgr2=0x7fa6c006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:21.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.292+0000 7fa6c4c00640 1 -- 172.21.15.102:0/14421154 shutdown_connections 2024-03-23T03:48:21.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.292+0000 7fa6c4c00640 1 -- 172.21.15.102:0/14421154 wait complete. 2024-03-23T03:48:21.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.293+0000 7fa6c4c00640 1 Processor -- start 2024-03-23T03:48:21.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.294+0000 7fa6c4c00640 1 -- start start 2024-03-23T03:48:21.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.294+0000 7fa6c4c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6c0070cd0 0x7fa6c015d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:21.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.294+0000 7fa6c4c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6c01456a0 0x7fa6c015f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:21.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.294+0000 7fa6c4c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6c0145c20 0x7fa6c013a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:21.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.294+0000 7fa6c4c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c015dee0 con 0x7fa6c01456a0 2024-03-23T03:48:21.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.294+0000 7fa6c4c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c015e1c0 con 0x7fa6c0145c20 2024-03-23T03:48:21.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.294+0000 7fa6c4c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c015e4a0 con 0x7fa6c0070cd0 2024-03-23T03:48:21.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.295+0000 7fa6b5400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6c01456a0 0x7fa6c015f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:21.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.295+0000 7fa6b5400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6c01456a0 0x7fa6c015f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43670/0 (socket says 172.21.15.102:43670) 2024-03-23T03:48:21.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.295+0000 7fa6b5400640 1 -- 172.21.15.102:0/388613498 learned_addr learned my addr 172.21.15.102:0/388613498 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:21.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.295+0000 7fa6bde00640 1 --2- 172.21.15.102:0/388613498 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6c0070cd0 0x7fa6c015d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:21.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.295+0000 7fa6be800640 1 --2- 172.21.15.102:0/388613498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6c0145c20 0x7fa6c013a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:21.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.295+0000 7fa6b5400640 1 -- 172.21.15.102:0/388613498 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6c0070cd0 msgr2=0x7fa6c015d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:21.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.295+0000 7fa6b5400640 1 --2- 172.21.15.102:0/388613498 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6c0070cd0 0x7fa6c015d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:21.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.295+0000 7fa6b5400640 1 -- 172.21.15.102:0/388613498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6c0145c20 msgr2=0x7fa6c013a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:21.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.295+0000 7fa6b5400640 1 --2- 172.21.15.102:0/388613498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6c0145c20 0x7fa6c013a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:21.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.295+0000 7fa6b5400640 1 -- 172.21.15.102:0/388613498 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b0009b30 con 0x7fa6c01456a0 2024-03-23T03:48:21.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.295+0000 7fa6b5400640 1 --2- 172.21.15.102:0/388613498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6c01456a0 0x7fa6c015f230 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa6a800d5d0 tx=0x7fa6a800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:21.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.296+0000 7fa6b6a00640 1 -- 172.21.15.102:0/388613498 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8004240 con 0x7fa6c01456a0 2024-03-23T03:48:21.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.296+0000 7fa6b6a00640 1 -- 172.21.15.102:0/388613498 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a80043a0 con 0x7fa6c01456a0 2024-03-23T03:48:21.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.296+0000 7fa6b6a00640 1 -- 172.21.15.102:0/388613498 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a8003830 con 0x7fa6c01456a0 2024-03-23T03:48:21.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.296+0000 7fa6c4c00640 1 -- 172.21.15.102:0/388613498 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c015e780 con 0x7fa6c01456a0 2024-03-23T03:48:21.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.296+0000 7fa6c4c00640 1 -- 172.21.15.102:0/388613498 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c015db40 con 0x7fa6c01456a0 2024-03-23T03:48:21.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.297+0000 7fa6c4c00640 1 -- 172.21.15.102:0/388613498 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c0142390 con 0x7fa6c01456a0 2024-03-23T03:48:21.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.298+0000 7fa6b6a00640 1 -- 172.21.15.102:0/388613498 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa6a8003990 con 0x7fa6c01456a0 2024-03-23T03:48:21.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.299+0000 7fa6b6a00640 1 --2- 172.21.15.102:0/388613498 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa69c06ac00 0x7fa69c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:21.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.299+0000 7fa6b6a00640 1 -- 172.21.15.102:0/388613498 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fa6a8089330 con 0x7fa6c01456a0 2024-03-23T03:48:21.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.299+0000 7fa6bde00640 1 --2- 172.21.15.102:0/388613498 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa69c06ac00 0x7fa69c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:21.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.300+0000 7fa6bde00640 1 --2- 172.21.15.102:0/388613498 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa69c06ac00 0x7fa69c06d0c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0035000 tx=0x7fa6b0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:21.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.303+0000 7fa6b6a00640 1 -- 172.21.15.102:0/388613498 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa6a8054710 con 0x7fa6c01456a0 2024-03-23T03:48:21.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.497+0000 7fa6c4c00640 1 -- 172.21.15.102:0/388613498 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa6c0073390 con 0x7fa6c01456a0 2024-03-23T03:48:21.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.938+0000 7fa6b6a00640 1 -- 172.21.15.102:0/388613498 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v38) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa6a8057d30 con 0x7fa6c01456a0 2024-03-23T03:48:21.939 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' created 2024-03-23T03:48:21.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.942+0000 7fa6c4c00640 1 -- 172.21.15.102:0/388613498 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa69c06ac00 msgr2=0x7fa69c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:21.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.942+0000 7fa6c4c00640 1 --2- 172.21.15.102:0/388613498 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa69c06ac00 0x7fa69c06d0c0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0035000 tx=0x7fa6b0030000 comp rx=0 tx=0).stop 2024-03-23T03:48:21.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.942+0000 7fa6c4c00640 1 -- 172.21.15.102:0/388613498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6c01456a0 msgr2=0x7fa6c015f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:21.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.942+0000 7fa6c4c00640 1 --2- 172.21.15.102:0/388613498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6c01456a0 0x7fa6c015f230 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa6a800d5d0 tx=0x7fa6a800da90 comp rx=0 tx=0).stop 2024-03-23T03:48:21.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.944+0000 7fa6c4c00640 1 -- 172.21.15.102:0/388613498 shutdown_connections 2024-03-23T03:48:21.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.944+0000 7fa6c4c00640 1 --2- 172.21.15.102:0/388613498 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa69c06ac00 0x7fa69c06d0c0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:21.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.944+0000 7fa6c4c00640 1 --2- 172.21.15.102:0/388613498 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa6c0145c20 0x7fa6c013a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:21.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.944+0000 7fa6c4c00640 1 --2- 172.21.15.102:0/388613498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6c01456a0 0x7fa6c015f230 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:21.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.944+0000 7fa6c4c00640 1 --2- 172.21.15.102:0/388613498 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa6c0070cd0 0x7fa6c015d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:21.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.944+0000 7fa6c4c00640 1 -- 172.21.15.102:0/388613498 >> 172.21.15.102:0/388613498 conn(0x7fa6c006a0b0 msgr2=0x7fa6c00ae2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:21.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.944+0000 7fa6c4c00640 1 -- 172.21.15.102:0/388613498 shutdown_connections 2024-03-23T03:48:21.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:21.944+0000 7fa6c4c00640 1 -- 172.21.15.102:0/388613498 wait complete. 2024-03-23T03:48:21.969 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T03:48:22.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.162+0000 7fe6d7200640 1 Processor -- start 2024-03-23T03:48:22.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.162+0000 7fe6d7200640 1 -- start start 2024-03-23T03:48:22.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.163+0000 7fe6d7200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6d815a850 0x7fe6d80cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:22.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.163+0000 7fe6d7200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d815e320 con 0x7fe6d80cacf0 2024-03-23T03:48:22.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.163+0000 7fe6d7200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d80cda20 con 0x7fe6d80b22b0 2024-03-23T03:48:22.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.163+0000 7fe6d7200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d815ac30 con 0x7fe6d815a850 2024-03-23T03:48:22.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.164+0000 7fe6d6800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe6d80cacf0 0x7fe6d806d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50462/0 (socket says 172.21.15.102:50462) 2024-03-23T03:48:22.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.164+0000 7fe6d6800640 1 -- 172.21.15.102:0/3388890555 learned_addr learned my addr 172.21.15.102:0/3388890555 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:22.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.164+0000 7fe6d5e00640 1 --2- 172.21.15.102:0/3388890555 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6d815a850 0x7fe6d80cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:22.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.164+0000 7fe6d4a00640 1 -- 172.21.15.102:0/3388890555 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1470088061 0 0) 0x7fe6d815e320 con 0x7fe6d80cacf0 2024-03-23T03:48:22.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.164+0000 7fe6d4a00640 1 -- 172.21.15.102:0/3388890555 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6bc003520 con 0x7fe6d80cacf0 2024-03-23T03:48:22.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.165+0000 7fe6d4a00640 1 -- 172.21.15.102:0/3388890555 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2475112207 0 0) 0x7fe6bc003520 con 0x7fe6d80cacf0 2024-03-23T03:48:22.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.165+0000 7fe6d4a00640 1 -- 172.21.15.102:0/3388890555 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6d815a850 msgr2=0x7fe6d80cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:22.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.165+0000 7fe6d4a00640 1 --2- 172.21.15.102:0/3388890555 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6d815a850 0x7fe6d80cd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:22.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.165+0000 7fe6d4a00640 1 -- 172.21.15.102:0/3388890555 >> v1:172.21.15.173:6789/0 conn(0x7fe6d80b22b0 legacy=0x7fe6d815eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:22.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.165+0000 7fe6d4a00640 1 -- 172.21.15.102:0/3388890555 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d80cdd00 con 0x7fe6d80cacf0 2024-03-23T03:48:22.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.165+0000 7fe6d4a00640 1 -- 172.21.15.102:0/3388890555 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fe6d00019f0 con 0x7fe6d80cacf0 2024-03-23T03:48:22.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.165+0000 7fe6d4a00640 1 -- 172.21.15.102:0/3388890555 >> v1:172.21.15.102:6789/0 conn(0x7fe6d80cacf0 legacy=0x7fe6d806d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:22.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.165+0000 7fe6d4a00640 1 --2- 172.21.15.102:0/3388890555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6bc0039a0 0x7fe6bc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:22.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.165+0000 7fe6d4a00640 1 --2- 172.21.15.102:0/3388890555 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6d815a850 0x7fe6bc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:22.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.165+0000 7fe6d4a00640 1 --2- 172.21.15.102:0/3388890555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6bc004d00 0x7fe6bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:22.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.165+0000 7fe6d4a00640 1 -- 172.21.15.102:0/3388890555 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d815ac30 con 0x7fe6bc004d00 2024-03-23T03:48:22.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.165+0000 7fe6d4a00640 1 -- 172.21.15.102:0/3388890555 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6bc00ba30 con 0x7fe6bc0039a0 2024-03-23T03:48:22.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.165+0000 7fe6d4a00640 1 -- 172.21.15.102:0/3388890555 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6bc00bbf0 con 0x7fe6d815a850 2024-03-23T03:48:22.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.165+0000 7fe6d6800640 1 --2- 172.21.15.102:0/3388890555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6bc004d00 0x7fe6bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:22.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.166+0000 7fe6d5e00640 1 --2- 172.21.15.102:0/3388890555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6bc0039a0 0x7fe6bc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:22.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.166+0000 7fe6d6800640 1 -- 172.21.15.102:0/3388890555 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6d815a850 msgr2=0x7fe6bc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:22.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.166+0000 7fe6d6800640 1 --2- 172.21.15.102:0/3388890555 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6d815a850 0x7fe6bc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:22.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.166+0000 7fe6d6800640 1 -- 172.21.15.102:0/3388890555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6bc0039a0 msgr2=0x7fe6bc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:22.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.166+0000 7fe6d6800640 1 --2- 172.21.15.102:0/3388890555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6bc0039a0 0x7fe6bc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:22.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.166+0000 7fe6d6800640 1 -- 172.21.15.102:0/3388890555 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6bc003520 con 0x7fe6bc004d00 2024-03-23T03:48:22.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.166+0000 7fe6d5e00640 1 --2- 172.21.15.102:0/3388890555 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6bc0039a0 0x7fe6bc003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:48:22.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.167+0000 7fe6d6800640 1 --2- 172.21.15.102:0/3388890555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6bc004d00 0x7fe6bc005180 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000a7f0 tx=0x7fe6bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f31e6cc4d8820bc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:22.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.167+0000 7fe6d4a00640 1 -- 172.21.15.102:0/3388890555 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d00072d0 con 0x7fe6bc004d00 2024-03-23T03:48:22.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.167+0000 7fe6d4a00640 1 -- 172.21.15.102:0/3388890555 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d002f070 con 0x7fe6bc004d00 2024-03-23T03:48:22.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.167+0000 7fe6d4a00640 1 -- 172.21.15.102:0/3388890555 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0007690 con 0x7fe6bc004d00 2024-03-23T03:48:22.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.168+0000 7fe6d7200640 1 -- 172.21.15.102:0/3388890555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6bc004d00 msgr2=0x7fe6bc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:22.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.168+0000 7fe6d7200640 1 --2- 172.21.15.102:0/3388890555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6bc004d00 0x7fe6bc005180 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000a7f0 tx=0x7fe6bc0057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:22.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.168+0000 7fe6d6800640 1 -- 172.21.15.102:0/3388890555 reap_dead start 2024-03-23T03:48:22.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.168+0000 7fe6d7200640 1 -- 172.21.15.102:0/3388890555 shutdown_connections 2024-03-23T03:48:22.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.168+0000 7fe6d7200640 1 -- 172.21.15.102:0/3388890555 >> 172.21.15.102:0/3388890555 conn(0x7fe6d80b01b0 msgr2=0x7fe6d80b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:22.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.168+0000 7fe6d7200640 1 -- 172.21.15.102:0/3388890555 shutdown_connections 2024-03-23T03:48:22.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.168+0000 7fe6d7200640 1 -- 172.21.15.102:0/3388890555 wait complete. 2024-03-23T03:48:22.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.170+0000 7fe6d7200640 1 Processor -- start 2024-03-23T03:48:22.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.170+0000 7fe6d7200640 1 -- start start 2024-03-23T03:48:22.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.171+0000 7fe6d7200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6bc004d00 0x7fe6d81efbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:22.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.171+0000 7fe6d7200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6d8069c90 0x7fe6d806a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:22.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.171+0000 7fe6d7200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6d81f36d0 0x7fe6d81f3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:22.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.171+0000 7fe6d7200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d806a650 con 0x7fe6bc004d00 2024-03-23T03:48:22.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.171+0000 7fe6d7200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d81f4450 con 0x7fe6d81f36d0 2024-03-23T03:48:22.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.171+0000 7fe6d7200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d8148420 con 0x7fe6d8069c90 2024-03-23T03:48:22.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.171+0000 7fe6d5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6bc004d00 0x7fe6d81efbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:22.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.171+0000 7fe6d5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6bc004d00 0x7fe6d81efbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43698/0 (socket says 172.21.15.102:43698) 2024-03-23T03:48:22.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.171+0000 7fe6d5e00640 1 -- 172.21.15.102:0/3473350823 learned_addr learned my addr 172.21.15.102:0/3473350823 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:22.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.171+0000 7fe6d6800640 1 --2- 172.21.15.102:0/3473350823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6d81f36d0 0x7fe6d81f3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:22.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.171+0000 7fe6d5e00640 1 -- 172.21.15.102:0/3473350823 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6d8069c90 msgr2=0x7fe6d806a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:22.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.171+0000 7fe6d5e00640 1 --2- 172.21.15.102:0/3473350823 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6d8069c90 0x7fe6d806a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:22.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.172+0000 7fe6d5e00640 1 -- 172.21.15.102:0/3473350823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6d81f36d0 msgr2=0x7fe6d81f3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:22.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.172+0000 7fe6d5e00640 1 --2- 172.21.15.102:0/3473350823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6d81f36d0 0x7fe6d81f3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:22.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.172+0000 7fe6d5e00640 1 -- 172.21.15.102:0/3473350823 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d8148700 con 0x7fe6bc004d00 2024-03-23T03:48:22.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.172+0000 7fe6d5e00640 1 --2- 172.21.15.102:0/3473350823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6bc004d00 0x7fe6d81efbb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8011680 tx=0x7fe6c8011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:22.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.172+0000 7fe6ce000640 1 -- 172.21.15.102:0/3473350823 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c800f9c0 con 0x7fe6bc004d00 2024-03-23T03:48:22.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.172+0000 7fe6ce000640 1 -- 172.21.15.102:0/3473350823 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c800fb20 con 0x7fe6bc004d00 2024-03-23T03:48:22.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.172+0000 7fe6ce000640 1 -- 172.21.15.102:0/3473350823 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c800fe90 con 0x7fe6bc004d00 2024-03-23T03:48:22.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.172+0000 7fe6d7200640 1 -- 172.21.15.102:0/3473350823 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d81489e0 con 0x7fe6bc004d00 2024-03-23T03:48:22.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.172+0000 7fe6d7200640 1 -- 172.21.15.102:0/3473350823 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d81f4090 con 0x7fe6bc004d00 2024-03-23T03:48:22.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.173+0000 7fe6d7200640 1 -- 172.21.15.102:0/3473350823 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d804e030 con 0x7fe6bc004d00 2024-03-23T03:48:22.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.174+0000 7fe6ce000640 1 -- 172.21.15.102:0/3473350823 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe6c8015460 con 0x7fe6bc004d00 2024-03-23T03:48:22.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.175+0000 7fe6ce000640 1 --2- 172.21.15.102:0/3473350823 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe6b806ac00 0x7fe6b806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:22.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.175+0000 7fe6ce000640 1 -- 172.21.15.102:0/3473350823 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5547+0+0 (secure 0 0 0) 0x7fe6c8089c90 con 0x7fe6bc004d00 2024-03-23T03:48:22.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.176+0000 7fe6d5400640 1 --2- 172.21.15.102:0/3473350823 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe6b806ac00 0x7fe6b806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:22.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.177+0000 7fe6d5400640 1 --2- 172.21.15.102:0/3473350823 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe6b806ac00 0x7fe6b806d0c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe6d81f35a0 tx=0x7fe6c4007990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:22.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.181+0000 7fe6ce000640 1 -- 172.21.15.102:0/3473350823 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe6c8054e70 con 0x7fe6bc004d00 2024-03-23T03:48:22.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.375+0000 7fe6d7200640 1 -- 172.21.15.102:0/3473350823 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fe6d80ccaa0 con 0x7fe6bc004d00 2024-03-23T03:48:22.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.950+0000 7fe6ce000640 1 -- 172.21.15.102:0/3473350823 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v39) v1 ==== 170+0+0 (secure 0 0 0) 0x7fe6c80220d0 con 0x7fe6bc004d00 2024-03-23T03:48:22.953 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' created 2024-03-23T03:48:22.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.957+0000 7fe6d7200640 1 -- 172.21.15.102:0/3473350823 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe6b806ac00 msgr2=0x7fe6b806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:22.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.957+0000 7fe6d7200640 1 --2- 172.21.15.102:0/3473350823 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe6b806ac00 0x7fe6b806d0c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe6d81f35a0 tx=0x7fe6c4007990 comp rx=0 tx=0).stop 2024-03-23T03:48:22.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.957+0000 7fe6d7200640 1 -- 172.21.15.102:0/3473350823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6bc004d00 msgr2=0x7fe6d81efbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:22.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.957+0000 7fe6d7200640 1 --2- 172.21.15.102:0/3473350823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6bc004d00 0x7fe6d81efbb0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8011680 tx=0x7fe6c8011b40 comp rx=0 tx=0).stop 2024-03-23T03:48:22.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.961+0000 7fe6d7200640 1 -- 172.21.15.102:0/3473350823 shutdown_connections 2024-03-23T03:48:22.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.961+0000 7fe6d7200640 1 --2- 172.21.15.102:0/3473350823 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe6b806ac00 0x7fe6b806d0c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:22.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.961+0000 7fe6d7200640 1 --2- 172.21.15.102:0/3473350823 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe6d81f36d0 0x7fe6d81f3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:22.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.961+0000 7fe6d7200640 1 --2- 172.21.15.102:0/3473350823 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6d8069c90 0x7fe6d806a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:22.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.961+0000 7fe6d7200640 1 --2- 172.21.15.102:0/3473350823 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6bc004d00 0x7fe6d81efbb0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:22.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.961+0000 7fe6d7200640 1 -- 172.21.15.102:0/3473350823 >> 172.21.15.102:0/3473350823 conn(0x7fe6d80b01b0 msgr2=0x7fe6d80b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:22.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.963+0000 7fe6d7200640 1 -- 172.21.15.102:0/3473350823 shutdown_connections 2024-03-23T03:48:22.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:22.964+0000 7fe6d7200640 1 -- 172.21.15.102:0/3473350823 wait complete. 2024-03-23T03:48:22.992 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T03:48:23.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.171+0000 7f0ed8e00640 1 Processor -- start 2024-03-23T03:48:23.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.171+0000 7f0ed8e00640 1 -- start start 2024-03-23T03:48:23.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.173+0000 7f0ed8e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f0ed40ca7b0 0x7f0ed40cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:23.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.173+0000 7f0ed8e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0ed40ce930 0x7f0ed40cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:23.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.173+0000 7f0ed8e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ed4159c10 con 0x7f0ed4159830 2024-03-23T03:48:23.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.173+0000 7f0ed8e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed406de50 con 0x7f0ed40ce930 2024-03-23T03:48:23.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.173+0000 7f0ed8e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ed40cf300 con 0x7f0ed40ca7b0 2024-03-23T03:48:23.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.173+0000 7f0ed2800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0ed40ce930 0x7f0ed40cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:23.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.173+0000 7f0ed2800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0ed40ce930 0x7f0ed40cedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44836/0 (socket says 172.21.15.102:44836) 2024-03-23T03:48:23.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.173+0000 7f0ed2800640 1 -- 172.21.15.102:0/1620693566 learned_addr learned my addr 172.21.15.102:0/1620693566 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:23.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.174+0000 7f0ec9400640 1 --2- 172.21.15.102:0/1620693566 >> v2:172.21.15.173:3300/0 conn(0x7f0ed40ca7b0 0x7f0ed40cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:23.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.174+0000 7f0ed2800640 1 -- 172.21.15.102:0/1620693566 >> v2:172.21.15.173:3300/0 conn(0x7f0ed40ca7b0 msgr2=0x7f0ed40cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:23.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.174+0000 7f0ed2800640 1 --2- 172.21.15.102:0/1620693566 >> v2:172.21.15.173:3300/0 conn(0x7f0ed40ca7b0 0x7f0ed40cdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.174+0000 7f0ed2800640 1 -- 172.21.15.102:0/1620693566 >> v1:172.21.15.173:6789/0 conn(0x7f0ed4159830 legacy=0x7f0ed40cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:23.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.174+0000 7f0ed2800640 1 -- 172.21.15.102:0/1620693566 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed40cd840 con 0x7f0ed40ce930 2024-03-23T03:48:23.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.175+0000 7f0ed2800640 1 --2- 172.21.15.102:0/1620693566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ed40ce930 0x7f0ed40cedc0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc009e80 tx=0x7f0ecc02bfe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff6e944c6bdddbc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:23.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.175+0000 7f0ed1400640 1 -- 172.21.15.102:0/1620693566 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc02cb80 con 0x7f0ed40ce930 2024-03-23T03:48:23.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.175+0000 7f0ed1400640 1 -- 172.21.15.102:0/1620693566 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ecc02cce0 con 0x7f0ed40ce930 2024-03-23T03:48:23.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.175+0000 7f0ed1400640 1 -- 172.21.15.102:0/1620693566 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc034610 con 0x7f0ed40ce930 2024-03-23T03:48:23.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.176+0000 7f0ed8e00640 1 -- 172.21.15.102:0/1620693566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ed40ce930 msgr2=0x7f0ed40cedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:23.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.176+0000 7f0ed8e00640 1 --2- 172.21.15.102:0/1620693566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ed40ce930 0x7f0ed40cedc0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc009e80 tx=0x7f0ecc02bfe0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.176+0000 7f0ed8e00640 1 -- 172.21.15.102:0/1620693566 shutdown_connections 2024-03-23T03:48:23.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.176+0000 7f0ed8e00640 1 --2- 172.21.15.102:0/1620693566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ed40ce930 0x7f0ed40cedc0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.176+0000 7f0ed8e00640 1 --2- 172.21.15.102:0/1620693566 >> v2:172.21.15.173:3300/0 conn(0x7f0ed40ca7b0 0x7f0ed40cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.176+0000 7f0ed8e00640 1 -- 172.21.15.102:0/1620693566 >> 172.21.15.102:0/1620693566 conn(0x7f0ed406a0b0 msgr2=0x7f0ed406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:23.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.176+0000 7f0ed8e00640 1 -- 172.21.15.102:0/1620693566 shutdown_connections 2024-03-23T03:48:23.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.176+0000 7f0ed8e00640 1 -- 172.21.15.102:0/1620693566 wait complete. 2024-03-23T03:48:23.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.177+0000 7f0ed8e00640 1 Processor -- start 2024-03-23T03:48:23.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.178+0000 7f0ed8e00640 1 -- start start 2024-03-23T03:48:23.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.179+0000 7f0ed8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ed40ca7b0 0x7f0ed4153ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:23.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.179+0000 7f0ed8e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0ed40ce930 0x7f0ed40c8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:23.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.179+0000 7f0ed8e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ed4159830 0x7f0ed41575c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:23.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.179+0000 7f0ed8e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed4156520 con 0x7f0ed40ca7b0 2024-03-23T03:48:23.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.179+0000 7f0ed8e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed4156800 con 0x7f0ed4159830 2024-03-23T03:48:23.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.179+0000 7f0ed8e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed4156ae0 con 0x7f0ed40ce930 2024-03-23T03:48:23.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.179+0000 7f0ed1e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ed40ca7b0 0x7f0ed4153ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:23.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.179+0000 7f0ed1e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ed40ca7b0 0x7f0ed4153ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44838/0 (socket says 172.21.15.102:44838) 2024-03-23T03:48:23.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.179+0000 7f0ed1e00640 1 -- 172.21.15.102:0/3448078579 learned_addr learned my addr 172.21.15.102:0/3448078579 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:23.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.179+0000 7f0ed1e00640 1 -- 172.21.15.102:0/3448078579 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0ed40ce930 msgr2=0x7f0ed40c8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:23.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.179+0000 7f0ed1e00640 1 --2- 172.21.15.102:0/3448078579 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0ed40ce930 0x7f0ed40c8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.179+0000 7f0ed1e00640 1 -- 172.21.15.102:0/3448078579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ed4159830 msgr2=0x7f0ed41575c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:23.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.179+0000 7f0ed2800640 1 --2- 172.21.15.102:0/3448078579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ed4159830 0x7f0ed41575c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:23.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.179+0000 7f0ed1e00640 1 --2- 172.21.15.102:0/3448078579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ed4159830 0x7f0ed41575c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.179+0000 7f0ed1e00640 1 -- 172.21.15.102:0/3448078579 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ecc009b30 con 0x7f0ed40ca7b0 2024-03-23T03:48:23.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.180+0000 7f0ed1e00640 1 --2- 172.21.15.102:0/3448078579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ed40ca7b0 0x7f0ed4153ec0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f0ec400c400 tx=0x7f0ec400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:23.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.180+0000 7f0ecaa00640 1 -- 172.21.15.102:0/3448078579 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec40052d0 con 0x7f0ed40ca7b0 2024-03-23T03:48:23.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.180+0000 7f0ecaa00640 1 -- 172.21.15.102:0/3448078579 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec4005430 con 0x7f0ed40ca7b0 2024-03-23T03:48:23.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.180+0000 7f0ecaa00640 1 -- 172.21.15.102:0/3448078579 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4003830 con 0x7f0ed40ca7b0 2024-03-23T03:48:23.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.180+0000 7f0ed8e00640 1 -- 172.21.15.102:0/3448078579 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ed4156dc0 con 0x7f0ed40ca7b0 2024-03-23T03:48:23.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.180+0000 7f0ed8e00640 1 -- 172.21.15.102:0/3448078579 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed4154400 con 0x7f0ed40ca7b0 2024-03-23T03:48:23.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.181+0000 7f0ed8e00640 1 -- 172.21.15.102:0/3448078579 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed41572e0 con 0x7f0ed40ca7b0 2024-03-23T03:48:23.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.183+0000 7f0ecaa00640 1 -- 172.21.15.102:0/3448078579 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0ec4003990 con 0x7f0ed40ca7b0 2024-03-23T03:48:23.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.183+0000 7f0ecaa00640 1 --2- 172.21.15.102:0/3448078579 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0eb006ac30 0x7f0eb006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:23.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.183+0000 7f0ecaa00640 1 -- 172.21.15.102:0/3448078579 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f0ec40889a0 con 0x7f0ed40ca7b0 2024-03-23T03:48:23.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.184+0000 7f0ec9400640 1 --2- 172.21.15.102:0/3448078579 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0eb006ac30 0x7f0eb006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:23.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.184+0000 7f0ec9400640 1 --2- 172.21.15.102:0/3448078579 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0eb006ac30 0x7f0eb006d0f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc006fd0 tx=0x7f0ebc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:23.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.188+0000 7f0ecaa00640 1 -- 172.21.15.102:0/3448078579 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0ed41572e0 con 0x7f0ed40ca7b0 2024-03-23T03:48:23.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.416+0000 7f0ed8e00640 1 -- 172.21.15.102:0/3448078579 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0ed41572e0 con 0x7f0ed40ca7b0 2024-03-23T03:48:23.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.424+0000 7f0ecaa00640 1 -- 172.21.15.102:0/3448078579 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7f0ed41572e0 con 0x7f0ed40ca7b0 2024-03-23T03:48:23.425 INFO:teuthology.orchestra.run.smithi102.stderr:new fs with metadata pool 5 and data pool 6 2024-03-23T03:48:23.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.428+0000 7f0ed8e00640 1 -- 172.21.15.102:0/3448078579 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0eb006ac30 msgr2=0x7f0eb006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:23.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.428+0000 7f0ed8e00640 1 --2- 172.21.15.102:0/3448078579 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0eb006ac30 0x7f0eb006d0f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc006fd0 tx=0x7f0ebc008040 comp rx=0 tx=0).stop 2024-03-23T03:48:23.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.428+0000 7f0ed8e00640 1 -- 172.21.15.102:0/3448078579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ed40ca7b0 msgr2=0x7f0ed4153ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:23.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.428+0000 7f0ed8e00640 1 --2- 172.21.15.102:0/3448078579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ed40ca7b0 0x7f0ed4153ec0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f0ec400c400 tx=0x7f0ec400c8c0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.429+0000 7f0ed8e00640 1 -- 172.21.15.102:0/3448078579 shutdown_connections 2024-03-23T03:48:23.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.429+0000 7f0ed8e00640 1 --2- 172.21.15.102:0/3448078579 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0eb006ac30 0x7f0eb006d0f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.429+0000 7f0ed8e00640 1 --2- 172.21.15.102:0/3448078579 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0ed4159830 0x7f0ed41575c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.429+0000 7f0ed8e00640 1 --2- 172.21.15.102:0/3448078579 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0ed40ce930 0x7f0ed40c8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.429+0000 7f0ed8e00640 1 --2- 172.21.15.102:0/3448078579 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0ed40ca7b0 0x7f0ed4153ec0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.429+0000 7f0ed8e00640 1 -- 172.21.15.102:0/3448078579 >> 172.21.15.102:0/3448078579 conn(0x7f0ed406a0b0 msgr2=0x7f0ed40b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:23.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.429+0000 7f0ed8e00640 1 -- 172.21.15.102:0/3448078579 shutdown_connections 2024-03-23T03:48:23.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.429+0000 7f0ed8e00640 1 -- 172.21.15.102:0/3448078579 wait complete. 2024-03-23T03:48:23.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:48:23.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.632+0000 7f5674400640 1 Processor -- start 2024-03-23T03:48:23.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.632+0000 7f5674400640 1 -- start start 2024-03-23T03:48:23.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.634+0000 7f5674400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f566c0cbab0 0x7f566c0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:23.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.634+0000 7f5674400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566c0b16a0 con 0x7f566c06d390 2024-03-23T03:48:23.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.634+0000 7f5674400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566c0cbfb0 con 0x7f566c0adbc0 2024-03-23T03:48:23.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.634+0000 7f5674400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f566c0cc290 con 0x7f566c0cbab0 2024-03-23T03:48:23.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.634+0000 7f5672400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f566c06d390 0x7f566c0b2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37790/0 (socket says 172.21.15.102:37790) 2024-03-23T03:48:23.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.634+0000 7f5672400640 1 -- 172.21.15.102:0/322095597 learned_addr learned my addr 172.21.15.102:0/322095597 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:23.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.634+0000 7f5671000640 1 --2- 172.21.15.102:0/322095597 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f566c0cbab0 0x7f566c0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:23.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.634+0000 7f566be00640 1 -- 172.21.15.102:0/322095597 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 666258045 0 0) 0x7f566c0b16a0 con 0x7f566c06d390 2024-03-23T03:48:23.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.634+0000 7f566be00640 1 -- 172.21.15.102:0/322095597 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5654003520 con 0x7f566c06d390 2024-03-23T03:48:23.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.635+0000 7f566be00640 1 -- 172.21.15.102:0/322095597 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1066802667 0 0) 0x7f5654003520 con 0x7f566c06d390 2024-03-23T03:48:23.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.635+0000 7f566be00640 1 -- 172.21.15.102:0/322095597 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f566c0cbab0 msgr2=0x7f566c0b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:23.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.635+0000 7f566be00640 1 --2- 172.21.15.102:0/322095597 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f566c0cbab0 0x7f566c0b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.635+0000 7f566be00640 1 -- 172.21.15.102:0/322095597 >> v1:172.21.15.173:6789/0 conn(0x7f566c0adbc0 legacy=0x7f566c0b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:23.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.635+0000 7f566be00640 1 -- 172.21.15.102:0/322095597 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c0b18f0 con 0x7f566c06d390 2024-03-23T03:48:23.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.635+0000 7f566be00640 1 -- 172.21.15.102:0/322095597 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f56640019f0 con 0x7f566c06d390 2024-03-23T03:48:23.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.635+0000 7f566be00640 1 -- 172.21.15.102:0/322095597 >> v1:172.21.15.102:6789/0 conn(0x7f566c06d390 legacy=0x7f566c0b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:23.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.635+0000 7f566be00640 1 --2- 172.21.15.102:0/322095597 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f566c0cbab0 0x7f56540039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:23.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.635+0000 7f566be00640 1 --2- 172.21.15.102:0/322095597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5654003f70 0x7f56540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:23.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.635+0000 7f566be00640 1 --2- 172.21.15.102:0/322095597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5654004d00 0x7f5654005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:23.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.635+0000 7f566be00640 1 -- 172.21.15.102:0/322095597 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c0cc290 con 0x7f5654004d00 2024-03-23T03:48:23.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.635+0000 7f566be00640 1 -- 172.21.15.102:0/322095597 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565400bb70 con 0x7f5654003f70 2024-03-23T03:48:23.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.635+0000 7f566be00640 1 -- 172.21.15.102:0/322095597 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565400bd30 con 0x7f566c0cbab0 2024-03-23T03:48:23.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.635+0000 7f5672400640 1 --2- 172.21.15.102:0/322095597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5654004d00 0x7f5654005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:23.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.636+0000 7f5671a00640 1 --2- 172.21.15.102:0/322095597 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f566c0cbab0 0x7f56540039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:23.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.636+0000 7f5672400640 1 -- 172.21.15.102:0/322095597 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f566c0cbab0 msgr2=0x7f56540039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:23.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.636+0000 7f5672400640 1 --2- 172.21.15.102:0/322095597 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f566c0cbab0 0x7f56540039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.636+0000 7f5672400640 1 -- 172.21.15.102:0/322095597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5654003f70 msgr2=0x7f56540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:23.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.636+0000 7f5671000640 1 --2- 172.21.15.102:0/322095597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5654003f70 0x7f56540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:23.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.636+0000 7f5672400640 1 --2- 172.21.15.102:0/322095597 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5654003f70 0x7f56540047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.636+0000 7f5672400640 1 -- 172.21.15.102:0/322095597 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5654003520 con 0x7f5654004d00 2024-03-23T03:48:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.637+0000 7f5672400640 1 --2- 172.21.15.102:0/322095597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5654004d00 0x7f5654005180 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f566400a760 tx=0x7f56540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e42642e7875841a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.637+0000 7f566be00640 1 -- 172.21.15.102:0/322095597 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5664034630 con 0x7f5654004d00 2024-03-23T03:48:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.637+0000 7f566be00640 1 -- 172.21.15.102:0/322095597 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566402f070 con 0x7f5654004d00 2024-03-23T03:48:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.637+0000 7f566be00640 1 -- 172.21.15.102:0/322095597 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56640349f0 con 0x7f5654004d00 2024-03-23T03:48:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.637+0000 7f5674400640 1 -- 172.21.15.102:0/322095597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5654004d00 msgr2=0x7f5654005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.637+0000 7f5674400640 1 --2- 172.21.15.102:0/322095597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5654004d00 0x7f5654005180 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f566400a760 tx=0x7f56540057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.637+0000 7f5672400640 1 -- 172.21.15.102:0/322095597 reap_dead start 2024-03-23T03:48:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.638+0000 7f5674400640 1 -- 172.21.15.102:0/322095597 shutdown_connections 2024-03-23T03:48:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.638+0000 7f5674400640 1 -- 172.21.15.102:0/322095597 >> 172.21.15.102:0/322095597 conn(0x7f566c06a3d0 msgr2=0x7f566c0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.638+0000 7f5674400640 1 -- 172.21.15.102:0/322095597 shutdown_connections 2024-03-23T03:48:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.638+0000 7f5674400640 1 -- 172.21.15.102:0/322095597 wait complete. 2024-03-23T03:48:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.639+0000 7f5674400640 1 Processor -- start 2024-03-23T03:48:23.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.639+0000 7f5674400640 1 -- start start 2024-03-23T03:48:23.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.640+0000 7f5674400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f566c15c330 0x7f566c0c3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:23.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.640+0000 7f5674400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f566c15c710 0x7f566c131ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:23.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.640+0000 7f5674400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f566c1323e0 0x7f566c1328d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.640+0000 7f5674400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c131a80 con 0x7f566c15c710 2024-03-23T03:48:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.640+0000 7f5674400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c130e00 con 0x7f566c1323e0 2024-03-23T03:48:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.640+0000 7f5674400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f566c1310e0 con 0x7f566c15c330 2024-03-23T03:48:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.640+0000 7f5671000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f566c15c710 0x7f566c131ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.641+0000 7f5671000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f566c15c710 0x7f566c131ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44862/0 (socket says 172.21.15.102:44862) 2024-03-23T03:48:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.641+0000 7f5671000640 1 -- 172.21.15.102:0/1702381920 learned_addr learned my addr 172.21.15.102:0/1702381920 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.641+0000 7f5671a00640 1 --2- 172.21.15.102:0/1702381920 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f566c15c330 0x7f566c0c3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.641+0000 7f5671000640 1 -- 172.21.15.102:0/1702381920 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f566c15c330 msgr2=0x7f566c0c3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.641+0000 7f5672400640 1 --2- 172.21.15.102:0/1702381920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f566c1323e0 0x7f566c1328d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.641+0000 7f5671000640 1 --2- 172.21.15.102:0/1702381920 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f566c15c330 0x7f566c0c3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.641+0000 7f5671000640 1 -- 172.21.15.102:0/1702381920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f566c1323e0 msgr2=0x7f566c1328d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.641+0000 7f5671000640 1 --2- 172.21.15.102:0/1702381920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f566c1323e0 0x7f566c1328d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.641+0000 7f5671000640 1 -- 172.21.15.102:0/1702381920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566c1313c0 con 0x7f566c15c710 2024-03-23T03:48:23.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.641+0000 7f5671a00640 1 --2- 172.21.15.102:0/1702381920 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f566c15c330 0x7f566c0c3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:23.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.641+0000 7f5671000640 1 --2- 172.21.15.102:0/1702381920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f566c15c710 0x7f566c131ea0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5658010860 tx=0x7f5658010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:23.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.641+0000 7f5669600640 1 -- 172.21.15.102:0/1702381920 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5658009610 con 0x7f566c15c710 2024-03-23T03:48:23.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.642+0000 7f5669600640 1 -- 172.21.15.102:0/1702381920 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565800eb60 con 0x7f566c15c710 2024-03-23T03:48:23.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.642+0000 7f5669600640 1 -- 172.21.15.102:0/1702381920 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565801c400 con 0x7f566c15c710 2024-03-23T03:48:23.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.642+0000 7f5674400640 1 -- 172.21.15.102:0/1702381920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566c1316a0 con 0x7f566c15c710 2024-03-23T03:48:23.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.642+0000 7f5674400640 1 -- 172.21.15.102:0/1702381920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f566c130840 con 0x7f566c15c710 2024-03-23T03:48:23.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.643+0000 7f5674400640 1 -- 172.21.15.102:0/1702381920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f566c06ac20 con 0x7f566c15c710 2024-03-23T03:48:23.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.644+0000 7f5669600640 1 -- 172.21.15.102:0/1702381920 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f565801c560 con 0x7f566c15c710 2024-03-23T03:48:23.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.645+0000 7f5669600640 1 --2- 172.21.15.102:0/1702381920 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f564c06ac00 0x7f564c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:23.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.645+0000 7f5669600640 1 -- 172.21.15.102:0/1702381920 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5658089970 con 0x7f566c15c710 2024-03-23T03:48:23.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.645+0000 7f5671a00640 1 --2- 172.21.15.102:0/1702381920 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f564c06ac00 0x7f564c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:23.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.645+0000 7f5671a00640 1 --2- 172.21.15.102:0/1702381920 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f564c06ac00 0x7f564c06d0c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f56600016b0 tx=0x7f5660009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:23.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.649+0000 7f5669600640 1 -- 172.21.15.102:0/1702381920 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5658054b50 con 0x7f566c15c710 2024-03-23T03:48:23.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.835+0000 7f5674400640 1 -- 172.21.15.102:0/1702381920 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f566c073390 con 0x7f566c15c710 2024-03-23T03:48:23.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.838+0000 7f5669600640 1 -- 172.21.15.102:0/1702381920 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+16367 (secure 0 0 0) 0x7f5658058170 con 0x7f566c15c710 2024-03-23T03:48:23.839 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:23.839 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":40,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:23.418161+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:48:21.550281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-23T03:48:22.428475+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:48:23.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.841+0000 7f5674400640 1 -- 172.21.15.102:0/1702381920 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f564c06ac00 msgr2=0x7f564c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:23.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.841+0000 7f5674400640 1 --2- 172.21.15.102:0/1702381920 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f564c06ac00 0x7f564c06d0c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f56600016b0 tx=0x7f5660009040 comp rx=0 tx=0).stop 2024-03-23T03:48:23.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.841+0000 7f5674400640 1 -- 172.21.15.102:0/1702381920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f566c15c710 msgr2=0x7f566c131ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:23.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.841+0000 7f5674400640 1 --2- 172.21.15.102:0/1702381920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f566c15c710 0x7f566c131ea0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5658010860 tx=0x7f5658010d20 comp rx=0 tx=0).stop 2024-03-23T03:48:23.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.841+0000 7f5674400640 1 -- 172.21.15.102:0/1702381920 shutdown_connections 2024-03-23T03:48:23.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.841+0000 7f5674400640 1 --2- 172.21.15.102:0/1702381920 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f564c06ac00 0x7f564c06d0c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.841+0000 7f5674400640 1 --2- 172.21.15.102:0/1702381920 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f566c1323e0 0x7f566c1328d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.841+0000 7f5674400640 1 --2- 172.21.15.102:0/1702381920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f566c15c710 0x7f566c131ea0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.841+0000 7f5674400640 1 --2- 172.21.15.102:0/1702381920 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f566c15c330 0x7f566c0c3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:23.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.841+0000 7f5674400640 1 -- 172.21.15.102:0/1702381920 >> 172.21.15.102:0/1702381920 conn(0x7f566c06a3d0 msgr2=0x7f566c06c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:23.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.841+0000 7f5674400640 1 -- 172.21.15.102:0/1702381920 shutdown_connections 2024-03-23T03:48:23.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:23.841+0000 7f5674400640 1 -- 172.21.15.102:0/1702381920 wait complete. 2024-03-23T03:48:23.859 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:48:24.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.057+0000 7f03bba00640 1 Processor -- start 2024-03-23T03:48:24.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.057+0000 7f03bba00640 1 -- start start 2024-03-23T03:48:24.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.059+0000 7f03bba00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f03b415b560 0x7f03b415b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:24.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.059+0000 7f03bba00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f03b415d410 0x7f03b40b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:24.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.059+0000 7f03bba00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b406c6f0 con 0x7f03b40b39b0 2024-03-23T03:48:24.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.059+0000 7f03bba00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03b40b4370 con 0x7f03b415d410 2024-03-23T03:48:24.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.059+0000 7f03bba00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b406c9d0 con 0x7f03b415b560 2024-03-23T03:48:24.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.059+0000 7f03b9a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f03b415d410 0x7f03b40b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:24.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.059+0000 7f03b9000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f03b415b560 0x7f03b415b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:24.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.060+0000 7f03b9000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f03b415b560 0x7f03b415b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:34048/0 (socket says 172.21.15.102:34048) 2024-03-23T03:48:24.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.060+0000 7f03b9000640 1 -- 172.21.15.102:0/3743675362 learned_addr learned my addr 172.21.15.102:0/3743675362 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:24.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.060+0000 7f03b9000640 1 -- 172.21.15.102:0/3743675362 >> v2:172.21.15.173:3300/0 conn(0x7f03b415d410 msgr2=0x7f03b40b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:24.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.060+0000 7f03b9000640 1 --2- 172.21.15.102:0/3743675362 >> v2:172.21.15.173:3300/0 conn(0x7f03b415d410 0x7f03b40b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:24.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.060+0000 7f03b9000640 1 -- 172.21.15.102:0/3743675362 >> v1:172.21.15.173:6789/0 conn(0x7f03b40b39b0 legacy=0x7f03b415e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:24.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.060+0000 7f03b3400640 1 -- 172.21.15.102:0/3743675362 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219126995 0 0) 0x7f03b406c6f0 con 0x7f03b40b39b0 2024-03-23T03:48:24.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.060+0000 7f03b9000640 1 -- 172.21.15.102:0/3743675362 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b406ccb0 con 0x7f03b415b560 2024-03-23T03:48:24.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.062+0000 7f03b9000640 1 --2- 172.21.15.102:0/3743675362 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f03b415b560 0x7f03b415b940 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009870 tx=0x7f03a402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7077114ff048acbb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:24.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.062+0000 7f03b3400640 1 -- 172.21.15.102:0/3743675362 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4030840 con 0x7f03b415b560 2024-03-23T03:48:24.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.063+0000 7f03b3400640 1 -- 172.21.15.102:0/3743675362 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a40309a0 con 0x7f03b415b560 2024-03-23T03:48:24.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.063+0000 7f03b3400640 1 -- 172.21.15.102:0/3743675362 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4030d30 con 0x7f03b415b560 2024-03-23T03:48:24.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.063+0000 7f03bba00640 1 -- 172.21.15.102:0/3743675362 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f03b415b560 msgr2=0x7f03b415b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:24.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.063+0000 7f03bba00640 1 --2- 172.21.15.102:0/3743675362 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f03b415b560 0x7f03b415b940 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009870 tx=0x7f03a402b980 comp rx=0 tx=0).stop 2024-03-23T03:48:24.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.063+0000 7f03bba00640 1 -- 172.21.15.102:0/3743675362 shutdown_connections 2024-03-23T03:48:24.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.063+0000 7f03bba00640 1 --2- 172.21.15.102:0/3743675362 >> v2:172.21.15.173:3300/0 conn(0x7f03b415d410 0x7f03b40b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:24.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.063+0000 7f03bba00640 1 --2- 172.21.15.102:0/3743675362 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f03b415b560 0x7f03b415b940 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:24.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.063+0000 7f03bba00640 1 -- 172.21.15.102:0/3743675362 >> 172.21.15.102:0/3743675362 conn(0x7f03b406a3d0 msgr2=0x7f03b40b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:24.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.063+0000 7f03bba00640 1 -- 172.21.15.102:0/3743675362 shutdown_connections 2024-03-23T03:48:24.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.063+0000 7f03bba00640 1 -- 172.21.15.102:0/3743675362 wait complete. 2024-03-23T03:48:24.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.065+0000 7f03bba00640 1 Processor -- start 2024-03-23T03:48:24.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.065+0000 7f03bba00640 1 -- start start 2024-03-23T03:48:24.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.066+0000 7f03bba00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03b40b39b0 0x7f03b40a78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:24.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.066+0000 7f03bba00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03b415b560 0x7f03b4159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:24.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.066+0000 7f03bba00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f03b415d410 0x7f03b40aaa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:24.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.066+0000 7f03bba00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b40a8170 con 0x7f03b40b39b0 2024-03-23T03:48:24.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.066+0000 7f03bba00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b40a8450 con 0x7f03b415b560 2024-03-23T03:48:24.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.066+0000 7f03bba00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b40b1840 con 0x7f03b415d410 2024-03-23T03:48:24.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.066+0000 7f03b9000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03b40b39b0 0x7f03b40a78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:24.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.067+0000 7f03b9000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03b40b39b0 0x7f03b40a78b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44872/0 (socket says 172.21.15.102:44872) 2024-03-23T03:48:24.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.067+0000 7f03b9000640 1 -- 172.21.15.102:0/3883467339 learned_addr learned my addr 172.21.15.102:0/3883467339 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:24.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.067+0000 7f03b3e00640 1 --2- 172.21.15.102:0/3883467339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03b415b560 0x7f03b4159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:24.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.067+0000 7f03b9a00640 1 --2- 172.21.15.102:0/3883467339 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f03b415d410 0x7f03b40aaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:24.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.067+0000 7f03b9000640 1 -- 172.21.15.102:0/3883467339 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f03b415d410 msgr2=0x7f03b40aaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:24.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.067+0000 7f03b9000640 1 --2- 172.21.15.102:0/3883467339 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f03b415d410 0x7f03b40aaa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:24.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.067+0000 7f03b9000640 1 -- 172.21.15.102:0/3883467339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03b415b560 msgr2=0x7f03b4159ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:24.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.067+0000 7f03b9000640 1 --2- 172.21.15.102:0/3883467339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03b415b560 0x7f03b4159ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:24.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.067+0000 7f03b9000640 1 -- 172.21.15.102:0/3883467339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a40094d0 con 0x7f03b40b39b0 2024-03-23T03:48:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.067+0000 7f03b9000640 1 --2- 172.21.15.102:0/3883467339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03b40b39b0 0x7f03b40a78b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f03a4000c00 tx=0x7f03a4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.068+0000 7f03b0c00640 1 -- 172.21.15.102:0/3883467339 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4039c90 con 0x7f03b40b39b0 2024-03-23T03:48:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.068+0000 7f03b0c00640 1 -- 172.21.15.102:0/3883467339 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a4039df0 con 0x7f03b40b39b0 2024-03-23T03:48:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.068+0000 7f03bba00640 1 -- 172.21.15.102:0/3883467339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b40b1b20 con 0x7f03b40b39b0 2024-03-23T03:48:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.068+0000 7f03b0c00640 1 -- 172.21.15.102:0/3883467339 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4043640 con 0x7f03b40b39b0 2024-03-23T03:48:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.068+0000 7f03bba00640 1 -- 172.21.15.102:0/3883467339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b40a7df0 con 0x7f03b40b39b0 2024-03-23T03:48:24.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.069+0000 7f03bba00640 1 -- 172.21.15.102:0/3883467339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b40b1d30 con 0x7f03b40b39b0 2024-03-23T03:48:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.070+0000 7f03b0c00640 1 -- 172.21.15.102:0/3883467339 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f03a40437a0 con 0x7f03b40b39b0 2024-03-23T03:48:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.071+0000 7f03b0c00640 1 --2- 172.21.15.102:0/3883467339 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f039406ac00 0x7f039406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.071+0000 7f03b0c00640 1 -- 172.21.15.102:0/3883467339 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f03a403e070 con 0x7f03b40b39b0 2024-03-23T03:48:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.071+0000 7f03b3e00640 1 --2- 172.21.15.102:0/3883467339 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f039406ac00 0x7f039406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:24.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.071+0000 7f03b3e00640 1 --2- 172.21.15.102:0/3883467339 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f039406ac00 0x7f039406d0c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f03b4143f80 tx=0x7f03a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:24.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.075+0000 7f03b0c00640 1 -- 172.21.15.102:0/3883467339 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f03a4077530 con 0x7f03b40b39b0 2024-03-23T03:48:24.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.269+0000 7f03bba00640 1 -- 172.21.15.102:0/3883467339 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03b406ac20 con 0x7f03b40b39b0 2024-03-23T03:48:24.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.272+0000 7f03b0c00640 1 -- 172.21.15.102:0/3883467339 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+16367 (secure 0 0 0) 0x7f03a407ab50 con 0x7f03b40b39b0 2024-03-23T03:48:24.273 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:24.273 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":40,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:23.418161+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:48:21.550281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-23T03:48:22.428475+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:48:24.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.274+0000 7f03bba00640 1 -- 172.21.15.102:0/3883467339 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f039406ac00 msgr2=0x7f039406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:24.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.274+0000 7f03bba00640 1 --2- 172.21.15.102:0/3883467339 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f039406ac00 0x7f039406d0c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f03b4143f80 tx=0x7f03a0009040 comp rx=0 tx=0).stop 2024-03-23T03:48:24.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.274+0000 7f03bba00640 1 -- 172.21.15.102:0/3883467339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03b40b39b0 msgr2=0x7f03b40a78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:24.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.274+0000 7f03bba00640 1 --2- 172.21.15.102:0/3883467339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03b40b39b0 0x7f03b40a78b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f03a4000c00 tx=0x7f03a4004730 comp rx=0 tx=0).stop 2024-03-23T03:48:24.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.275+0000 7f03bba00640 1 -- 172.21.15.102:0/3883467339 shutdown_connections 2024-03-23T03:48:24.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.275+0000 7f03bba00640 1 --2- 172.21.15.102:0/3883467339 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f039406ac00 0x7f039406d0c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:24.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.275+0000 7f03bba00640 1 --2- 172.21.15.102:0/3883467339 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f03b415d410 0x7f03b40aaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:24.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.275+0000 7f03bba00640 1 --2- 172.21.15.102:0/3883467339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03b415b560 0x7f03b4159ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:24.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.275+0000 7f03bba00640 1 --2- 172.21.15.102:0/3883467339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03b40b39b0 0x7f03b40a78b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:24.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.275+0000 7f03bba00640 1 -- 172.21.15.102:0/3883467339 >> 172.21.15.102:0/3883467339 conn(0x7f03b406a3d0 msgr2=0x7f03b40cb460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:24.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.275+0000 7f03bba00640 1 -- 172.21.15.102:0/3883467339 shutdown_connections 2024-03-23T03:48:24.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.275+0000 7f03bba00640 1 -- 172.21.15.102:0/3883467339 wait complete. 2024-03-23T03:48:24.293 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T03:48:24.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.503+0000 7fd9d9c00640 1 Processor -- start 2024-03-23T03:48:24.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.503+0000 7fd9d9c00640 1 -- start start 2024-03-23T03:48:24.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.504+0000 7fd9d9c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd9d4159830 0x7fd9d40cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:24.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.504+0000 7fd9d9c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9d40ca7b0 0x7fd9d40cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:24.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.505+0000 7fd9d9c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d4159c10 con 0x7fd9d40ce930 2024-03-23T03:48:24.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.505+0000 7fd9d9c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d406de50 con 0x7fd9d40ca7b0 2024-03-23T03:48:24.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.505+0000 7fd9d9c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d40cd840 con 0x7fd9d4159830 2024-03-23T03:48:24.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.505+0000 7fd9ca000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9d40ca7b0 0x7fd9d40cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:24.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.505+0000 7fd9ca000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9d40ca7b0 0x7fd9d40cdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44884/0 (socket says 172.21.15.102:44884) 2024-03-23T03:48:24.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.505+0000 7fd9ca000640 1 -- 172.21.15.102:0/1407150690 learned_addr learned my addr 172.21.15.102:0/1407150690 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:24.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.505+0000 7fd9d2a00640 1 --2- 172.21.15.102:0/1407150690 >> v2:172.21.15.173:3300/0 conn(0x7fd9d4159830 0x7fd9d40cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:24.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.505+0000 7fd9ca000640 1 -- 172.21.15.102:0/1407150690 >> v2:172.21.15.173:3300/0 conn(0x7fd9d4159830 msgr2=0x7fd9d40cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:24.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.505+0000 7fd9ca000640 1 --2- 172.21.15.102:0/1407150690 >> v2:172.21.15.173:3300/0 conn(0x7fd9d4159830 0x7fd9d40cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:24.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.505+0000 7fd9ca000640 1 -- 172.21.15.102:0/1407150690 >> v1:172.21.15.173:6789/0 conn(0x7fd9d40ce930 legacy=0x7fd9d40b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:24.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.505+0000 7fd9ca000640 1 -- 172.21.15.102:0/1407150690 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d40ccc30 con 0x7fd9d40ca7b0 2024-03-23T03:48:24.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.506+0000 7fd9ca000640 1 --2- 172.21.15.102:0/1407150690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40ca7b0 0x7fd9d40cdcd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00bb10 tx=0x7fd9bc02e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82d46fb68694ffff server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:24.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.507+0000 7fd9d2000640 1 -- 172.21.15.102:0/1407150690 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc036440 con 0x7fd9d40ca7b0 2024-03-23T03:48:24.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.507+0000 7fd9d2000640 1 -- 172.21.15.102:0/1407150690 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9bc0365a0 con 0x7fd9d40ca7b0 2024-03-23T03:48:24.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.507+0000 7fd9d2000640 1 -- 172.21.15.102:0/1407150690 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc036930 con 0x7fd9d40ca7b0 2024-03-23T03:48:24.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.507+0000 7fd9d9c00640 1 -- 172.21.15.102:0/1407150690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40ca7b0 msgr2=0x7fd9d40cdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:24.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.507+0000 7fd9d9c00640 1 --2- 172.21.15.102:0/1407150690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40ca7b0 0x7fd9d40cdcd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00bb10 tx=0x7fd9bc02e170 comp rx=0 tx=0).stop 2024-03-23T03:48:24.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.507+0000 7fd9d9c00640 1 -- 172.21.15.102:0/1407150690 shutdown_connections 2024-03-23T03:48:24.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.507+0000 7fd9d9c00640 1 --2- 172.21.15.102:0/1407150690 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40ca7b0 0x7fd9d40cdcd0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:24.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.507+0000 7fd9d9c00640 1 --2- 172.21.15.102:0/1407150690 >> v2:172.21.15.173:3300/0 conn(0x7fd9d4159830 0x7fd9d40cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:24.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.507+0000 7fd9d9c00640 1 -- 172.21.15.102:0/1407150690 >> 172.21.15.102:0/1407150690 conn(0x7fd9d406a0b0 msgr2=0x7fd9d406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:24.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.508+0000 7fd9d9c00640 1 -- 172.21.15.102:0/1407150690 shutdown_connections 2024-03-23T03:48:24.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.508+0000 7fd9d9c00640 1 -- 172.21.15.102:0/1407150690 wait complete. 2024-03-23T03:48:24.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.509+0000 7fd9d9c00640 1 Processor -- start 2024-03-23T03:48:24.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.509+0000 7fd9d9c00640 1 -- start start 2024-03-23T03:48:24.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.510+0000 7fd9d9c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40ca7b0 0x7fd9d41f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:24.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.510+0000 7fd9d9c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9d40ce930 0x7fd9d40aeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:24.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.510+0000 7fd9d9c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9d4159830 0x7fd9d4158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.510+0000 7fd9d9c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d41f6550 con 0x7fd9d40ca7b0 2024-03-23T03:48:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.510+0000 7fd9d9c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d41f69e0 con 0x7fd9d4159830 2024-03-23T03:48:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.510+0000 7fd9d9c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d41f5c20 con 0x7fd9d40ce930 2024-03-23T03:48:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.511+0000 7fd9d2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40ca7b0 0x7fd9d41f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.511+0000 7fd9d2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40ca7b0 0x7fd9d41f2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44888/0 (socket says 172.21.15.102:44888) 2024-03-23T03:48:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.511+0000 7fd9d2a00640 1 -- 172.21.15.102:0/2539946551 learned_addr learned my addr 172.21.15.102:0/2539946551 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.511+0000 7fd9ca000640 1 --2- 172.21.15.102:0/2539946551 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9d40ce930 0x7fd9d40aeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.511+0000 7fd9d3400640 1 --2- 172.21.15.102:0/2539946551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9d4159830 0x7fd9d4158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.511+0000 7fd9d2a00640 1 -- 172.21.15.102:0/2539946551 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9d40ce930 msgr2=0x7fd9d40aeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.511+0000 7fd9d2a00640 1 --2- 172.21.15.102:0/2539946551 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9d40ce930 0x7fd9d40aeeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.511+0000 7fd9d2a00640 1 -- 172.21.15.102:0/2539946551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9d4159830 msgr2=0x7fd9d4158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.511+0000 7fd9d2a00640 1 --2- 172.21.15.102:0/2539946551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9d4159830 0x7fd9d4158200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.511+0000 7fd9d2a00640 1 -- 172.21.15.102:0/2539946551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9bc00b770 con 0x7fd9d40ca7b0 2024-03-23T03:48:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.511+0000 7fd9d2a00640 1 --2- 172.21.15.102:0/2539946551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40ca7b0 0x7fd9d41f2c30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400c7f0 tx=0x7fd9c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:24.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.512+0000 7fd9cb400640 1 -- 172.21.15.102:0/2539946551 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4004220 con 0x7fd9d40ca7b0 2024-03-23T03:48:24.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.512+0000 7fd9cb400640 1 -- 172.21.15.102:0/2539946551 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9c40043a0 con 0x7fd9d40ca7b0 2024-03-23T03:48:24.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.512+0000 7fd9d9c00640 1 -- 172.21.15.102:0/2539946551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d41f5f00 con 0x7fd9d40ca7b0 2024-03-23T03:48:24.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.512+0000 7fd9cb400640 1 -- 172.21.15.102:0/2539946551 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4007860 con 0x7fd9d40ca7b0 2024-03-23T03:48:24.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.512+0000 7fd9d9c00640 1 -- 172.21.15.102:0/2539946551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d40af970 con 0x7fd9d40ca7b0 2024-03-23T03:48:24.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.513+0000 7fd9d9c00640 1 -- 172.21.15.102:0/2539946551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d415cc60 con 0x7fd9d40ca7b0 2024-03-23T03:48:24.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.514+0000 7fd9cb400640 1 -- 172.21.15.102:0/2539946551 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd9c4007bd0 con 0x7fd9d40ca7b0 2024-03-23T03:48:24.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.515+0000 7fd9cb400640 1 --2- 172.21.15.102:0/2539946551 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9b006ac30 0x7fd9b006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:24.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.515+0000 7fd9cb400640 1 -- 172.21.15.102:0/2539946551 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd9c4089450 con 0x7fd9d40ca7b0 2024-03-23T03:48:24.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.515+0000 7fd9ca000640 1 --2- 172.21.15.102:0/2539946551 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9b006ac30 0x7fd9b006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:24.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.515+0000 7fd9ca000640 1 --2- 172.21.15.102:0/2539946551 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9b006ac30 0x7fd9b006d0f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc038000 tx=0x7fd9bc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:24.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.519+0000 7fd9cb400640 1 -- 172.21.15.102:0/2539946551 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd9c40544e0 con 0x7fd9d40ca7b0 2024-03-23T03:48:24.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:24.737+0000 7fd9d9c00640 1 -- 172.21.15.102:0/2539946551 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd9d4073390 con 0x7fd9d40ca7b0 2024-03-23T03:48:25.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.661+0000 7fd9cb400640 1 -- 172.21.15.102:0/2539946551 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd9c4057b00 con 0x7fd9d40ca7b0 2024-03-23T03:48:25.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.664+0000 7fd9d9c00640 1 -- 172.21.15.102:0/2539946551 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9b006ac30 msgr2=0x7fd9b006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:25.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.664+0000 7fd9d9c00640 1 --2- 172.21.15.102:0/2539946551 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9b006ac30 0x7fd9b006d0f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc038000 tx=0x7fd9bc008040 comp rx=0 tx=0).stop 2024-03-23T03:48:25.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.664+0000 7fd9d9c00640 1 -- 172.21.15.102:0/2539946551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40ca7b0 msgr2=0x7fd9d41f2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:25.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.664+0000 7fd9d9c00640 1 --2- 172.21.15.102:0/2539946551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40ca7b0 0x7fd9d41f2c30 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400c7f0 tx=0x7fd9c400ccb0 comp rx=0 tx=0).stop 2024-03-23T03:48:25.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.664+0000 7fd9d9c00640 1 -- 172.21.15.102:0/2539946551 shutdown_connections 2024-03-23T03:48:25.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.664+0000 7fd9d9c00640 1 --2- 172.21.15.102:0/2539946551 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9b006ac30 0x7fd9b006d0f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:25.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.664+0000 7fd9d9c00640 1 --2- 172.21.15.102:0/2539946551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9d4159830 0x7fd9d4158200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:25.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.664+0000 7fd9d9c00640 1 --2- 172.21.15.102:0/2539946551 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9d40ce930 0x7fd9d40aeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:25.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.664+0000 7fd9d9c00640 1 --2- 172.21.15.102:0/2539946551 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40ca7b0 0x7fd9d41f2c30 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:25.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.664+0000 7fd9d9c00640 1 -- 172.21.15.102:0/2539946551 >> 172.21.15.102:0/2539946551 conn(0x7fd9d406a0b0 msgr2=0x7fd9d41575e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:25.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.664+0000 7fd9d9c00640 1 -- 172.21.15.102:0/2539946551 shutdown_connections 2024-03-23T03:48:25.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.664+0000 7fd9d9c00640 1 -- 172.21.15.102:0/2539946551 wait complete. 2024-03-23T03:48:25.684 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-23T03:48:25.684 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T03:48:25.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.897+0000 7f6c96400640 1 Processor -- start 2024-03-23T03:48:25.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.898+0000 7f6c96400640 1 -- start start 2024-03-23T03:48:25.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.899+0000 7f6c96400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6c900cf460 0x7f6c90099870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:25.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.899+0000 7f6c96400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c900affd0 con 0x7f6c90098a40 2024-03-23T03:48:25.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.899+0000 7f6c96400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c900cbf50 con 0x7f6c900b1530 2024-03-23T03:48:25.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.899+0000 7f6c96400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c90099db0 con 0x7f6c900cf460 2024-03-23T03:48:25.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.899+0000 7f6c8fe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6c90098a40 0x7f6c900b0740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37798/0 (socket says 172.21.15.102:37798) 2024-03-23T03:48:25.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.899+0000 7f6c8fe00640 1 -- 172.21.15.102:0/1166469943 learned_addr learned my addr 172.21.15.102:0/1166469943 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:25.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.900+0000 7f6c8f400640 1 --2- 172.21.15.102:0/1166469943 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6c900cf460 0x7f6c90099870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:25.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.900+0000 7f6c8e000640 1 -- 172.21.15.102:0/1166469943 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950989113 0 0) 0x7f6c900affd0 con 0x7f6c90098a40 2024-03-23T03:48:25.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.900+0000 7f6c8e000640 1 -- 172.21.15.102:0/1166469943 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c70003520 con 0x7f6c90098a40 2024-03-23T03:48:25.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.900+0000 7f6c8e000640 1 -- 172.21.15.102:0/1166469943 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 218089908 0 0) 0x7f6c70003520 con 0x7f6c90098a40 2024-03-23T03:48:25.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.900+0000 7f6c8e000640 1 -- 172.21.15.102:0/1166469943 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6c900cf460 msgr2=0x7f6c90099870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:25.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.900+0000 7f6c8e000640 1 --2- 172.21.15.102:0/1166469943 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6c900cf460 0x7f6c90099870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:25.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.900+0000 7f6c8e000640 1 -- 172.21.15.102:0/1166469943 >> v1:172.21.15.173:6789/0 conn(0x7f6c900b1530 legacy=0x7f6c900ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:25.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.900+0000 7f6c8e000640 1 -- 172.21.15.102:0/1166469943 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c900cc230 con 0x7f6c90098a40 2024-03-23T03:48:25.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.900+0000 7f6c8e000640 1 -- 172.21.15.102:0/1166469943 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f6c88001670 con 0x7f6c90098a40 2024-03-23T03:48:25.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.900+0000 7f6c8e000640 1 -- 172.21.15.102:0/1166469943 >> v1:172.21.15.102:6789/0 conn(0x7f6c90098a40 legacy=0x7f6c900b0740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:25.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.900+0000 7f6c8e000640 1 --2- 172.21.15.102:0/1166469943 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6c900cf460 0x7f6c700039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:25.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.901+0000 7f6c8e000640 1 --2- 172.21.15.102:0/1166469943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c70003f20 0x7f6c700047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:25.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.901+0000 7f6c8e000640 1 --2- 172.21.15.102:0/1166469943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c70004d00 0x7f6c70005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:25.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.901+0000 7f6c8e000640 1 -- 172.21.15.102:0/1166469943 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c90099db0 con 0x7f6c70004d00 2024-03-23T03:48:25.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.901+0000 7f6c8e000640 1 -- 172.21.15.102:0/1166469943 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7000bb10 con 0x7f6c70003f20 2024-03-23T03:48:25.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.901+0000 7f6c8e000640 1 -- 172.21.15.102:0/1166469943 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7000bcd0 con 0x7f6c900cf460 2024-03-23T03:48:25.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.901+0000 7f6c8fe00640 1 --2- 172.21.15.102:0/1166469943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c70004d00 0x7f6c70005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:25.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.901+0000 7f6c8f400640 1 --2- 172.21.15.102:0/1166469943 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6c900cf460 0x7f6c700039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:25.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.901+0000 7f6c8ea00640 1 --2- 172.21.15.102:0/1166469943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c70003f20 0x7f6c700047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:25.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.901+0000 7f6c8fe00640 1 -- 172.21.15.102:0/1166469943 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6c900cf460 msgr2=0x7f6c700039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:25.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.901+0000 7f6c8fe00640 1 --2- 172.21.15.102:0/1166469943 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6c900cf460 0x7f6c700039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:25.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.901+0000 7f6c8fe00640 1 -- 172.21.15.102:0/1166469943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c70003f20 msgr2=0x7f6c700047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:25.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.901+0000 7f6c8fe00640 1 --2- 172.21.15.102:0/1166469943 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c70003f20 0x7f6c700047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:25.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.901+0000 7f6c8fe00640 1 -- 172.21.15.102:0/1166469943 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c70003520 con 0x7f6c70004d00 2024-03-23T03:48:25.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.901+0000 7f6c8f400640 1 --2- 172.21.15.102:0/1166469943 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6c900cf460 0x7f6c700039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:48:25.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.902+0000 7f6c8fe00640 1 --2- 172.21.15.102:0/1166469943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c70004d00 0x7f6c70005180 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800c240 tx=0x7f6c700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4acf99a15ba16085 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:25.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.903+0000 7f6c8e000640 1 -- 172.21.15.102:0/1166469943 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88005ad0 con 0x7f6c70004d00 2024-03-23T03:48:25.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.903+0000 7f6c8e000640 1 -- 172.21.15.102:0/1166469943 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c88005c30 con 0x7f6c70004d00 2024-03-23T03:48:25.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.903+0000 7f6c8e000640 1 -- 172.21.15.102:0/1166469943 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8802c8b0 con 0x7f6c70004d00 2024-03-23T03:48:25.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.903+0000 7f6c96400640 1 -- 172.21.15.102:0/1166469943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c70004d00 msgr2=0x7f6c70005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:25.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.903+0000 7f6c96400640 1 --2- 172.21.15.102:0/1166469943 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c70004d00 0x7f6c70005180 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800c240 tx=0x7f6c700057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:25.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.903+0000 7f6c8fe00640 1 -- 172.21.15.102:0/1166469943 reap_dead start 2024-03-23T03:48:25.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.903+0000 7f6c96400640 1 -- 172.21.15.102:0/1166469943 shutdown_connections 2024-03-23T03:48:25.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.903+0000 7f6c96400640 1 -- 172.21.15.102:0/1166469943 >> 172.21.15.102:0/1166469943 conn(0x7f6c900b2180 msgr2=0x7f6c900b2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:25.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.904+0000 7f6c96400640 1 -- 172.21.15.102:0/1166469943 shutdown_connections 2024-03-23T03:48:25.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.904+0000 7f6c96400640 1 -- 172.21.15.102:0/1166469943 wait complete. 2024-03-23T03:48:25.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.905+0000 7f6c96400640 1 Processor -- start 2024-03-23T03:48:25.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.905+0000 7f6c96400640 1 -- start start 2024-03-23T03:48:25.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.906+0000 7f6c96400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c900b0ac0 0x7f6c9006b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:25.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.906+0000 7f6c96400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c900b0ea0 0x7f6c9006c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.906+0000 7f6c96400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6c900c54a0 0x7f6c901541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.906+0000 7f6c96400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9006b900 con 0x7f6c900b0ea0 2024-03-23T03:48:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.906+0000 7f6c96400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9006bbe0 con 0x7f6c900b0ac0 2024-03-23T03:48:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.906+0000 7f6c96400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9006bec0 con 0x7f6c900c54a0 2024-03-23T03:48:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.907+0000 7f6c8ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c900b0ea0 0x7f6c9006c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.907+0000 7f6c8ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c900b0ea0 0x7f6c9006c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44910/0 (socket says 172.21.15.102:44910) 2024-03-23T03:48:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.907+0000 7f6c8ea00640 1 -- 172.21.15.102:0/1240612292 learned_addr learned my addr 172.21.15.102:0/1240612292 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.907+0000 7f6c8f400640 1 --2- 172.21.15.102:0/1240612292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c900b0ac0 0x7f6c9006b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.907+0000 7f6c8ea00640 1 -- 172.21.15.102:0/1240612292 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6c900c54a0 msgr2=0x7f6c901541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.907+0000 7f6c8ea00640 1 --2- 172.21.15.102:0/1240612292 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6c900c54a0 0x7f6c901541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.907+0000 7f6c8ea00640 1 -- 172.21.15.102:0/1240612292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c900b0ac0 msgr2=0x7f6c9006b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.907+0000 7f6c8ea00640 1 --2- 172.21.15.102:0/1240612292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c900b0ac0 0x7f6c9006b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.907+0000 7f6c8ea00640 1 -- 172.21.15.102:0/1240612292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9006c1a0 con 0x7f6c900b0ea0 2024-03-23T03:48:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.907+0000 7f6c8ea00640 1 --2- 172.21.15.102:0/1240612292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c900b0ea0 0x7f6c9006c9a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6c7800e8f0 tx=0x7f6c7800edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:25.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.908+0000 7f6c87400640 1 -- 172.21.15.102:0/1240612292 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c78003b80 con 0x7f6c900b0ea0 2024-03-23T03:48:25.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.908+0000 7f6c87400640 1 -- 172.21.15.102:0/1240612292 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c78003ce0 con 0x7f6c900b0ea0 2024-03-23T03:48:25.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.908+0000 7f6c87400640 1 -- 172.21.15.102:0/1240612292 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c78005310 con 0x7f6c900b0ea0 2024-03-23T03:48:25.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.908+0000 7f6c96400640 1 -- 172.21.15.102:0/1240612292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9006c480 con 0x7f6c900b0ea0 2024-03-23T03:48:25.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.908+0000 7f6c96400640 1 -- 172.21.15.102:0/1240612292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c90154730 con 0x7f6c900b0ea0 2024-03-23T03:48:25.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.909+0000 7f6c96400640 1 -- 172.21.15.102:0/1240612292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9006c640 con 0x7f6c900b0ea0 2024-03-23T03:48:25.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.910+0000 7f6c87400640 1 -- 172.21.15.102:0/1240612292 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6c78005470 con 0x7f6c900b0ea0 2024-03-23T03:48:25.911 INFO:tasks.ceph.mds.a.smithi102.stderr:2024-03-23T03:48:25.910+0000 7fc473600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T03:48:25.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.911+0000 7f6c87400640 1 --2- 172.21.15.102:0/1240612292 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6c6c06ac30 0x7f6c6c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:25.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.911+0000 7f6c87400640 1 -- 172.21.15.102:0/1240612292 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6c78089590 con 0x7f6c900b0ea0 2024-03-23T03:48:25.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.911+0000 7f6c8f400640 1 --2- 172.21.15.102:0/1240612292 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6c6c06ac30 0x7f6c6c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:25.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.912+0000 7f6c8f400640 1 --2- 172.21.15.102:0/1240612292 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6c6c06ac30 0x7f6c6c06d0f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6c800099b0 tx=0x7f6c80008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:25.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:25.916+0000 7f6c87400640 1 -- 172.21.15.102:0/1240612292 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6c78054590 con 0x7f6c900b0ea0 2024-03-23T03:48:26.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.148+0000 7f6c96400640 1 -- 172.21.15.102:0/1240612292 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f6c900ccb30 con 0x7f6c900b0ea0 2024-03-23T03:48:26.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.665+0000 7f6c87400640 1 -- 172.21.15.102:0/1240612292 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6c78057bb0 con 0x7f6c900b0ea0 2024-03-23T03:48:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.668+0000 7f6c96400640 1 -- 172.21.15.102:0/1240612292 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6c6c06ac30 msgr2=0x7f6c6c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.668+0000 7f6c96400640 1 --2- 172.21.15.102:0/1240612292 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6c6c06ac30 0x7f6c6c06d0f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6c800099b0 tx=0x7f6c80008040 comp rx=0 tx=0).stop 2024-03-23T03:48:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.668+0000 7f6c96400640 1 -- 172.21.15.102:0/1240612292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c900b0ea0 msgr2=0x7f6c9006c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.668+0000 7f6c96400640 1 --2- 172.21.15.102:0/1240612292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c900b0ea0 0x7f6c9006c9a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6c7800e8f0 tx=0x7f6c7800edb0 comp rx=0 tx=0).stop 2024-03-23T03:48:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.669+0000 7f6c96400640 1 -- 172.21.15.102:0/1240612292 shutdown_connections 2024-03-23T03:48:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.669+0000 7f6c96400640 1 --2- 172.21.15.102:0/1240612292 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6c6c06ac30 0x7f6c6c06d0f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.669+0000 7f6c96400640 1 --2- 172.21.15.102:0/1240612292 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6c900c54a0 0x7f6c901541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.669+0000 7f6c96400640 1 --2- 172.21.15.102:0/1240612292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c900b0ea0 0x7f6c9006c9a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.669+0000 7f6c96400640 1 --2- 172.21.15.102:0/1240612292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6c900b0ac0 0x7f6c9006b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.669+0000 7f6c96400640 1 -- 172.21.15.102:0/1240612292 >> 172.21.15.102:0/1240612292 conn(0x7f6c900b2180 msgr2=0x7f6c9015dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:26.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.669+0000 7f6c96400640 1 -- 172.21.15.102:0/1240612292 shutdown_connections 2024-03-23T03:48:26.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.669+0000 7f6c96400640 1 -- 172.21.15.102:0/1240612292 wait complete. 2024-03-23T03:48:26.689 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T03:48:26.689 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:48:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.894+0000 7f84e3200640 1 Processor -- start 2024-03-23T03:48:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.895+0000 7f84e3200640 1 -- start start 2024-03-23T03:48:26.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.896+0000 7f84e3200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f84e41571d0 0x7f84e40b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:26.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.896+0000 7f84e3200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84e40cdc00 0x7f84e415aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:26.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.896+0000 7f84e3200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f84e415afe0 0x7f84e40cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:26.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.896+0000 7f84e3200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e41575b0 con 0x7f84e41571d0 2024-03-23T03:48:26.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.896+0000 7f84e3200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e4159f20 con 0x7f84e415afe0 2024-03-23T03:48:26.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.896+0000 7f84e3200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e415a200 con 0x7f84e40cdc00 2024-03-23T03:48:26.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.897+0000 7f84e1e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f84e41571d0 0x7f84e40b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:26.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.897+0000 7f84e1e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f84e41571d0 0x7f84e40b35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44916/0 (socket says 172.21.15.102:44916) 2024-03-23T03:48:26.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.897+0000 7f84e1e00640 1 -- 172.21.15.102:0/3878603554 learned_addr learned my addr 172.21.15.102:0/3878603554 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:26.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.897+0000 7f84e2800640 1 --2- 172.21.15.102:0/3878603554 >> v2:172.21.15.173:3300/0 conn(0x7f84e415afe0 0x7f84e40cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:26.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.897+0000 7f84e1e00640 1 -- 172.21.15.102:0/3878603554 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84e40cdc00 msgr2=0x7f84e415aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:26.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.897+0000 7f84d9400640 1 --2- 172.21.15.102:0/3878603554 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84e40cdc00 0x7f84e415aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:26.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.897+0000 7f84e1e00640 1 --2- 172.21.15.102:0/3878603554 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84e40cdc00 0x7f84e415aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:26.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.897+0000 7f84e1e00640 1 -- 172.21.15.102:0/3878603554 >> v2:172.21.15.173:3300/0 conn(0x7f84e415afe0 msgr2=0x7f84e40cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:26.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.897+0000 7f84e1e00640 1 --2- 172.21.15.102:0/3878603554 >> v2:172.21.15.173:3300/0 conn(0x7f84e415afe0 0x7f84e40cae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:26.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.897+0000 7f84e1e00640 1 -- 172.21.15.102:0/3878603554 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e415a4e0 con 0x7f84e41571d0 2024-03-23T03:48:26.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.897+0000 7f84e2800640 1 --2- 172.21.15.102:0/3878603554 >> v2:172.21.15.173:3300/0 conn(0x7f84e415afe0 0x7f84e40cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:26.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.897+0000 7f84d9400640 1 --2- 172.21.15.102:0/3878603554 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84e40cdc00 0x7f84e415aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:26.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.898+0000 7f84e1e00640 1 --2- 172.21.15.102:0/3878603554 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e41571d0 0x7f84e40b35d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f84e40adf30 tx=0x7f84d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe97ad8fb5acb2e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:26.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.898+0000 7f84e1400640 1 -- 172.21.15.102:0/3878603554 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4031870 con 0x7f84e41571d0 2024-03-23T03:48:26.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.899+0000 7f84e1400640 1 -- 172.21.15.102:0/3878603554 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d40319d0 con 0x7f84e41571d0 2024-03-23T03:48:26.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.899+0000 7f84e1400640 1 -- 172.21.15.102:0/3878603554 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d4031d60 con 0x7f84e41571d0 2024-03-23T03:48:26.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.899+0000 7f84e3200640 1 -- 172.21.15.102:0/3878603554 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e41571d0 msgr2=0x7f84e40b35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:26.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.899+0000 7f84e3200640 1 --2- 172.21.15.102:0/3878603554 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e41571d0 0x7f84e40b35d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f84e40adf30 tx=0x7f84d402b910 comp rx=0 tx=0).stop 2024-03-23T03:48:26.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.899+0000 7f84e3200640 1 -- 172.21.15.102:0/3878603554 shutdown_connections 2024-03-23T03:48:26.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.899+0000 7f84e3200640 1 --2- 172.21.15.102:0/3878603554 >> v2:172.21.15.173:3300/0 conn(0x7f84e415afe0 0x7f84e40cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:26.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.899+0000 7f84e3200640 1 --2- 172.21.15.102:0/3878603554 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84e40cdc00 0x7f84e415aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:26.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.899+0000 7f84e3200640 1 --2- 172.21.15.102:0/3878603554 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e41571d0 0x7f84e40b35d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:26.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.899+0000 7f84e3200640 1 -- 172.21.15.102:0/3878603554 >> 172.21.15.102:0/3878603554 conn(0x7f84e4065df0 msgr2=0x7f84e4069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:26.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.899+0000 7f84e3200640 1 -- 172.21.15.102:0/3878603554 shutdown_connections 2024-03-23T03:48:26.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.899+0000 7f84e3200640 1 -- 172.21.15.102:0/3878603554 wait complete. 2024-03-23T03:48:26.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.901+0000 7f84e3200640 1 Processor -- start 2024-03-23T03:48:26.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.901+0000 7f84e3200640 1 -- start start 2024-03-23T03:48:26.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.902+0000 7f84e3200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e40cdc00 0x7f84e4141e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:26.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.902+0000 7f84e3200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84e41571d0 0x7f84e40b0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:26.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.902+0000 7f84e3200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e415afe0 0x7f84e4158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:26.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.902+0000 7f84e3200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e41f2020 con 0x7f84e415afe0 2024-03-23T03:48:26.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.902+0000 7f84e3200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e41f2e40 con 0x7f84e40cdc00 2024-03-23T03:48:26.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.902+0000 7f84e3200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e41f2230 con 0x7f84e41571d0 2024-03-23T03:48:26.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.903+0000 7f84e2800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e415afe0 0x7f84e4158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:26.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.903+0000 7f84e2800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e415afe0 0x7f84e4158c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44932/0 (socket says 172.21.15.102:44932) 2024-03-23T03:48:26.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.903+0000 7f84e2800640 1 -- 172.21.15.102:0/471294077 learned_addr learned my addr 172.21.15.102:0/471294077 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:26.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.903+0000 7f84e2800640 1 -- 172.21.15.102:0/471294077 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84e41571d0 msgr2=0x7f84e40b0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:26.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.903+0000 7f84e2800640 1 --2- 172.21.15.102:0/471294077 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84e41571d0 0x7f84e40b0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:26.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.903+0000 7f84e2800640 1 -- 172.21.15.102:0/471294077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e40cdc00 msgr2=0x7f84e4141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:26.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.903+0000 7f84e2800640 1 --2- 172.21.15.102:0/471294077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e40cdc00 0x7f84e4141e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:26.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.903+0000 7f84e2800640 1 -- 172.21.15.102:0/471294077 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d40094d0 con 0x7f84e415afe0 2024-03-23T03:48:26.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.903+0000 7f84e2800640 1 --2- 172.21.15.102:0/471294077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e415afe0 0x7f84e4158c50 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00cd50 tx=0x7f84dc00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:26.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.904+0000 7f84daa00640 1 -- 172.21.15.102:0/471294077 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc00f680 con 0x7f84e415afe0 2024-03-23T03:48:26.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.904+0000 7f84daa00640 1 -- 172.21.15.102:0/471294077 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84dc00f800 con 0x7f84e415afe0 2024-03-23T03:48:26.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.904+0000 7f84daa00640 1 -- 172.21.15.102:0/471294077 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84dc00fbb0 con 0x7f84e415afe0 2024-03-23T03:48:26.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.904+0000 7f84e3200640 1 -- 172.21.15.102:0/471294077 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e41f2510 con 0x7f84e415afe0 2024-03-23T03:48:26.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.904+0000 7f84e3200640 1 -- 172.21.15.102:0/471294077 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e41f2ad0 con 0x7f84e415afe0 2024-03-23T03:48:26.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.905+0000 7f84e3200640 1 -- 172.21.15.102:0/471294077 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e404e030 con 0x7f84e415afe0 2024-03-23T03:48:26.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.906+0000 7f84daa00640 1 -- 172.21.15.102:0/471294077 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f84dc00b040 con 0x7f84e415afe0 2024-03-23T03:48:26.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.907+0000 7f84daa00640 1 --2- 172.21.15.102:0/471294077 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f84c006ac00 0x7f84c006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:26.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.907+0000 7f84daa00640 1 -- 172.21.15.102:0/471294077 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f84dc089e90 con 0x7f84e415afe0 2024-03-23T03:48:26.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.907+0000 7f84e1e00640 1 --2- 172.21.15.102:0/471294077 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f84c006ac00 0x7f84c006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:26.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.908+0000 7f84e1e00640 1 --2- 172.21.15.102:0/471294077 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f84c006ac00 0x7f84c006d0c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f84d4036000 tx=0x7f84d4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:26.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:26.911+0000 7f84daa00640 1 -- 172.21.15.102:0/471294077 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f84dc054f30 con 0x7f84e415afe0 2024-03-23T03:48:27.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.146+0000 7f84e3200640 1 -- 172.21.15.102:0/471294077 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84e41f2f80 con 0x7f84e415afe0 2024-03-23T03:48:27.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.148+0000 7f84daa00640 1 -- 172.21.15.102:0/471294077 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f84dc01d020 con 0x7f84e415afe0 2024-03-23T03:48:27.148 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:27.148 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6834/1537100581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1537100581},{"type":"v1","addr":"172.21.15.173:6834","nonce":1537100581}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4655,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3206638952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":3206638952},{"type":"v1","addr":"172.21.15.173:6835","nonce":3206638952}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4852,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4005240382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4005240382},{"type":"v1","addr":"172.21.15.102:6837","nonce":4005240382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:48:23.418926+0000","modified":"2024-03-23T03:48:26.657662+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.102:6836/1373501465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":1373501465},{"type":"v1","addr":"172.21.15.102:6836","nonce":1373501465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-23T03:48:27.149 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 16 2024-03-23T03:48:27.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.150+0000 7f84e3200640 1 -- 172.21.15.102:0/471294077 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f84c006ac00 msgr2=0x7f84c006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:27.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.150+0000 7f84e3200640 1 --2- 172.21.15.102:0/471294077 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f84c006ac00 0x7f84c006d0c0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f84d4036000 tx=0x7f84d4030000 comp rx=0 tx=0).stop 2024-03-23T03:48:27.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.150+0000 7f84e3200640 1 -- 172.21.15.102:0/471294077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e415afe0 msgr2=0x7f84e4158c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:27.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.150+0000 7f84e3200640 1 --2- 172.21.15.102:0/471294077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e415afe0 0x7f84e4158c50 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f84dc00cd50 tx=0x7f84dc00a650 comp rx=0 tx=0).stop 2024-03-23T03:48:27.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.150+0000 7f84e3200640 1 -- 172.21.15.102:0/471294077 shutdown_connections 2024-03-23T03:48:27.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.150+0000 7f84e3200640 1 --2- 172.21.15.102:0/471294077 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f84c006ac00 0x7f84c006d0c0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.150+0000 7f84e3200640 1 --2- 172.21.15.102:0/471294077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e415afe0 0x7f84e4158c50 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.150+0000 7f84e3200640 1 --2- 172.21.15.102:0/471294077 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84e41571d0 0x7f84e40b0190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.150+0000 7f84e3200640 1 --2- 172.21.15.102:0/471294077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e40cdc00 0x7f84e4141e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.150+0000 7f84e3200640 1 -- 172.21.15.102:0/471294077 >> 172.21.15.102:0/471294077 conn(0x7f84e4065df0 msgr2=0x7f84e40ab8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:27.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.151+0000 7f84e3200640 1 -- 172.21.15.102:0/471294077 shutdown_connections 2024-03-23T03:48:27.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.151+0000 7f84e3200640 1 -- 172.21.15.102:0/471294077 wait complete. 2024-03-23T03:48:27.168 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:48:27.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.373+0000 7fd681c00640 1 Processor -- start 2024-03-23T03:48:27.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.373+0000 7fd681c00640 1 -- start start 2024-03-23T03:48:27.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.375+0000 7fd681c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd67c0b37f0 0x7fd67c0cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:27.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.375+0000 7fd681c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd67c0ccec0 0x7fd67c06c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:27.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.375+0000 7fd681c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd67c06da70 con 0x7fd67c14aa40 2024-03-23T03:48:27.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.375+0000 7fd681c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd67c06d7d0 con 0x7fd67c0b37f0 2024-03-23T03:48:27.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.375+0000 7fd681c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c0b1d00 con 0x7fd67c0ccec0 2024-03-23T03:48:27.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.375+0000 7fd673e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd67c0b37f0 0x7fd67c0cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:27.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.375+0000 7fd673e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd67c0b37f0 0x7fd67c0cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44946/0 (socket says 172.21.15.102:44946) 2024-03-23T03:48:27.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.375+0000 7fd673e00640 1 -- 172.21.15.102:0/837611382 learned_addr learned my addr 172.21.15.102:0/837611382 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:27.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.375+0000 7fd67aa00640 1 --1- 172.21.15.102:0/837611382 >> v1:172.21.15.102:6789/0 conn(0x7fd67c14aa40 0x7fd67c0cc7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37808/0 (socket says 172.21.15.102:37808) 2024-03-23T03:48:27.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.375+0000 7fd67b400640 1 --2- 172.21.15.102:0/837611382 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd67c0ccec0 0x7fd67c06c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:27.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.376+0000 7fd673e00640 1 -- 172.21.15.102:0/837611382 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd67c0ccec0 msgr2=0x7fd67c06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:27.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.376+0000 7fd673e00640 1 --2- 172.21.15.102:0/837611382 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd67c0ccec0 0x7fd67c06c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.376+0000 7fd673e00640 1 -- 172.21.15.102:0/837611382 >> v1:172.21.15.102:6789/0 conn(0x7fd67c14aa40 legacy=0x7fd67c0cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:27.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.376+0000 7fd673e00640 1 -- 172.21.15.102:0/837611382 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67c0b1fe0 con 0x7fd67c0b37f0 2024-03-23T03:48:27.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.377+0000 7fd673e00640 1 --2- 172.21.15.102:0/837611382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c0b37f0 0x7fd67c0cd7a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd664009870 tx=0x7fd66402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a27ac95bcc9b660 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:27.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.377+0000 7fd67a000640 1 -- 172.21.15.102:0/837611382 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664030840 con 0x7fd67c0b37f0 2024-03-23T03:48:27.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.377+0000 7fd67a000640 1 -- 172.21.15.102:0/837611382 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6640309a0 con 0x7fd67c0b37f0 2024-03-23T03:48:27.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.377+0000 7fd67a000640 1 -- 172.21.15.102:0/837611382 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664030d30 con 0x7fd67c0b37f0 2024-03-23T03:48:27.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.378+0000 7fd681c00640 1 -- 172.21.15.102:0/837611382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c0b37f0 msgr2=0x7fd67c0cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:27.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.378+0000 7fd681c00640 1 --2- 172.21.15.102:0/837611382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c0b37f0 0x7fd67c0cd7a0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fd664009870 tx=0x7fd66402b980 comp rx=0 tx=0).stop 2024-03-23T03:48:27.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.378+0000 7fd681c00640 1 -- 172.21.15.102:0/837611382 shutdown_connections 2024-03-23T03:48:27.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.378+0000 7fd681c00640 1 --2- 172.21.15.102:0/837611382 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd67c0ccec0 0x7fd67c06c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.378+0000 7fd681c00640 1 --2- 172.21.15.102:0/837611382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c0b37f0 0x7fd67c0cd7a0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.378+0000 7fd681c00640 1 -- 172.21.15.102:0/837611382 >> 172.21.15.102:0/837611382 conn(0x7fd67c06a3d0 msgr2=0x7fd67c0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:27.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.378+0000 7fd681c00640 1 -- 172.21.15.102:0/837611382 shutdown_connections 2024-03-23T03:48:27.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.378+0000 7fd681c00640 1 -- 172.21.15.102:0/837611382 wait complete. 2024-03-23T03:48:27.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.380+0000 7fd681c00640 1 Processor -- start 2024-03-23T03:48:27.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.380+0000 7fd681c00640 1 -- start start 2024-03-23T03:48:27.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.381+0000 7fd681c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd67c0b37f0 0x7fd67c0ace50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:27.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.381+0000 7fd681c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd67c0ccec0 0x7fd67c142a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:27.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.381+0000 7fd681c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c14aa40 0x7fd67c0b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:27.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.381+0000 7fd681c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c1f68d0 con 0x7fd67c14aa40 2024-03-23T03:48:27.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.381+0000 7fd681c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c1f6b50 con 0x7fd67c0b37f0 2024-03-23T03:48:27.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.381+0000 7fd681c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c1f6e30 con 0x7fd67c0ccec0 2024-03-23T03:48:27.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.381+0000 7fd67b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c14aa40 0x7fd67c0b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:27.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.381+0000 7fd67b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c14aa40 0x7fd67c0b3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44950/0 (socket says 172.21.15.102:44950) 2024-03-23T03:48:27.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.381+0000 7fd67b400640 1 -- 172.21.15.102:0/2893665214 learned_addr learned my addr 172.21.15.102:0/2893665214 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:27.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.381+0000 7fd67aa00640 1 --2- 172.21.15.102:0/2893665214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd67c0b37f0 0x7fd67c0ace50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:27.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.381+0000 7fd673e00640 1 --2- 172.21.15.102:0/2893665214 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd67c0ccec0 0x7fd67c142a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:27.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.381+0000 7fd67b400640 1 -- 172.21.15.102:0/2893665214 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd67c0ccec0 msgr2=0x7fd67c142a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:27.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.381+0000 7fd67b400640 1 --2- 172.21.15.102:0/2893665214 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd67c0ccec0 0x7fd67c142a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.381+0000 7fd67b400640 1 -- 172.21.15.102:0/2893665214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd67c0b37f0 msgr2=0x7fd67c0ace50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:27.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.381+0000 7fd67b400640 1 --2- 172.21.15.102:0/2893665214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd67c0b37f0 0x7fd67c0ace50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.381+0000 7fd67b400640 1 -- 172.21.15.102:0/2893665214 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6640094d0 con 0x7fd67c14aa40 2024-03-23T03:48:27.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.382+0000 7fd67b400640 1 --2- 172.21.15.102:0/2893665214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c14aa40 0x7fd67c0b3c40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd674007e40 tx=0x7fd67400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:27.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.382+0000 7fd672a00640 1 -- 172.21.15.102:0/2893665214 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd674014620 con 0x7fd67c14aa40 2024-03-23T03:48:27.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.382+0000 7fd672a00640 1 -- 172.21.15.102:0/2893665214 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd67400ee90 con 0x7fd67c14aa40 2024-03-23T03:48:27.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.382+0000 7fd672a00640 1 -- 172.21.15.102:0/2893665214 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6740149e0 con 0x7fd67c14aa40 2024-03-23T03:48:27.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.382+0000 7fd681c00640 1 -- 172.21.15.102:0/2893665214 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd67c1f7110 con 0x7fd67c14aa40 2024-03-23T03:48:27.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.382+0000 7fd681c00640 1 -- 172.21.15.102:0/2893665214 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd67c142fa0 con 0x7fd67c14aa40 2024-03-23T03:48:27.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.383+0000 7fd681c00640 1 -- 172.21.15.102:0/2893665214 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd67c1f7630 con 0x7fd67c14aa40 2024-03-23T03:48:27.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.384+0000 7fd672a00640 1 -- 172.21.15.102:0/2893665214 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd674014b40 con 0x7fd67c14aa40 2024-03-23T03:48:27.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.385+0000 7fd672a00640 1 --2- 172.21.15.102:0/2893665214 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd65806ac00 0x7fd65806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:27.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.385+0000 7fd672a00640 1 -- 172.21.15.102:0/2893665214 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd674089f10 con 0x7fd67c14aa40 2024-03-23T03:48:27.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.385+0000 7fd67aa00640 1 --2- 172.21.15.102:0/2893665214 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd65806ac00 0x7fd65806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:27.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.386+0000 7fd67aa00640 1 --2- 172.21.15.102:0/2893665214 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd65806ac00 0x7fd65806d0c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd66c000f60 tx=0x7fd66c007c80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:27.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.390+0000 7fd672a00640 1 -- 172.21.15.102:0/2893665214 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd67c1f7630 con 0x7fd67c14aa40 2024-03-23T03:48:27.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.583+0000 7fd681c00640 1 -- 172.21.15.102:0/2893665214 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd67c06ac20 con 0x7fd67c14aa40 2024-03-23T03:48:27.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.586+0000 7fd672a00640 1 -- 172.21.15.102:0/2893665214 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+16354 (secure 0 0 0) 0x7fd674054f10 con 0x7fd67c14aa40 2024-03-23T03:48:27.587 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:27.587 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":41,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:24.423308+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:48:21.550281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-23T03:48:22.428475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:48:27.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.589+0000 7fd681c00640 1 -- 172.21.15.102:0/2893665214 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd65806ac00 msgr2=0x7fd65806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:27.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.589+0000 7fd681c00640 1 --2- 172.21.15.102:0/2893665214 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd65806ac00 0x7fd65806d0c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd66c000f60 tx=0x7fd66c007c80 comp rx=0 tx=0).stop 2024-03-23T03:48:27.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.589+0000 7fd681c00640 1 -- 172.21.15.102:0/2893665214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c14aa40 msgr2=0x7fd67c0b3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:27.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.589+0000 7fd681c00640 1 --2- 172.21.15.102:0/2893665214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c14aa40 0x7fd67c0b3c40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fd674007e40 tx=0x7fd67400e4b0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.589+0000 7fd681c00640 1 -- 172.21.15.102:0/2893665214 shutdown_connections 2024-03-23T03:48:27.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.589+0000 7fd681c00640 1 --2- 172.21.15.102:0/2893665214 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd65806ac00 0x7fd65806d0c0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.589+0000 7fd681c00640 1 --2- 172.21.15.102:0/2893665214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd67c14aa40 0x7fd67c0b3c40 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.589+0000 7fd681c00640 1 --2- 172.21.15.102:0/2893665214 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd67c0ccec0 0x7fd67c142a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.589+0000 7fd681c00640 1 --2- 172.21.15.102:0/2893665214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd67c0b37f0 0x7fd67c0ace50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.589+0000 7fd681c00640 1 -- 172.21.15.102:0/2893665214 >> 172.21.15.102:0/2893665214 conn(0x7fd67c06a3d0 msgr2=0x7fd67c0cc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:27.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.589+0000 7fd681c00640 1 -- 172.21.15.102:0/2893665214 shutdown_connections 2024-03-23T03:48:27.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.589+0000 7fd681c00640 1 -- 172.21.15.102:0/2893665214 wait complete. 2024-03-23T03:48:27.608 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T03:48:27.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:48:27.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.804+0000 7f2e96c00640 1 Processor -- start 2024-03-23T03:48:27.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.804+0000 7f2e96c00640 1 -- start start 2024-03-23T03:48:27.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.806+0000 7f2e96c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f2e9006d390 0x7f2e9006d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:27.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.806+0000 7f2e96c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e900b16a0 con 0x7f2e900adbc0 2024-03-23T03:48:27.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.806+0000 7f2e96c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e900cbfb0 con 0x7f2e900cbab0 2024-03-23T03:48:27.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.806+0000 7f2e96c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e9006dd40 con 0x7f2e9006d390 2024-03-23T03:48:27.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.806+0000 7f2e8fe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2e900adbc0 0x7f2e900b41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37812/0 (socket says 172.21.15.102:37812) 2024-03-23T03:48:27.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.806+0000 7f2e8fe00640 1 -- 172.21.15.102:0/2743405515 learned_addr learned my addr 172.21.15.102:0/2743405515 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:27.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.806+0000 7f2e8ea00640 1 -- 172.21.15.102:0/2743405515 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994781498 0 0) 0x7f2e900b16a0 con 0x7f2e900adbc0 2024-03-23T03:48:27.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.806+0000 7f2e94c00640 1 --2- 172.21.15.102:0/2743405515 >> v2:172.21.15.173:3300/0 conn(0x7f2e9006d390 0x7f2e9006d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:27.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.806+0000 7f2e8ea00640 1 -- 172.21.15.102:0/2743405515 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e74003520 con 0x7f2e900adbc0 2024-03-23T03:48:27.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.807+0000 7f2e8ea00640 1 -- 172.21.15.102:0/2743405515 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3422948637 0 0) 0x7f2e74003520 con 0x7f2e900adbc0 2024-03-23T03:48:27.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.807+0000 7f2e8ea00640 1 -- 172.21.15.102:0/2743405515 >> v2:172.21.15.173:3300/0 conn(0x7f2e9006d390 msgr2=0x7f2e9006d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:27.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.807+0000 7f2e8ea00640 1 --2- 172.21.15.102:0/2743405515 >> v2:172.21.15.173:3300/0 conn(0x7f2e9006d390 0x7f2e9006d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.807+0000 7f2e8ea00640 1 -- 172.21.15.102:0/2743405515 >> v1:172.21.15.173:6789/0 conn(0x7f2e900cbab0 legacy=0x7f2e900b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:27.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.807+0000 7f2e8ea00640 1 -- 172.21.15.102:0/2743405515 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e900cc290 con 0x7f2e900adbc0 2024-03-23T03:48:27.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.807+0000 7f2e8ea00640 1 -- 172.21.15.102:0/2743405515 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f2e80001670 con 0x7f2e900adbc0 2024-03-23T03:48:27.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.807+0000 7f2e8ea00640 1 -- 172.21.15.102:0/2743405515 >> v1:172.21.15.102:6789/0 conn(0x7f2e900adbc0 legacy=0x7f2e900b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:27.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.807+0000 7f2e8ea00640 1 --2- 172.21.15.102:0/2743405515 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2e74003970 0x7f2e74003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:27.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.807+0000 7f2e8ea00640 1 --2- 172.21.15.102:0/2743405515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e740047c0 0x7f2e74004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:27.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.807+0000 7f2e8ea00640 1 --2- 172.21.15.102:0/2743405515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e74005180 0x7f2e7400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:27.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.807+0000 7f2e8ea00640 1 -- 172.21.15.102:0/2743405515 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9006dd40 con 0x7f2e740047c0 2024-03-23T03:48:27.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.807+0000 7f2e8ea00640 1 -- 172.21.15.102:0/2743405515 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7400c120 con 0x7f2e74005180 2024-03-23T03:48:27.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.807+0000 7f2e8ea00640 1 -- 172.21.15.102:0/2743405515 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7400c2e0 con 0x7f2e74003970 2024-03-23T03:48:27.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.807+0000 7f2e8f400640 1 --2- 172.21.15.102:0/2743405515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e740047c0 0x7f2e74004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:27.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.808+0000 7f2e94c00640 1 --2- 172.21.15.102:0/2743405515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e74005180 0x7f2e7400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:27.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.808+0000 7f2e8f400640 1 -- 172.21.15.102:0/2743405515 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2e74003970 msgr2=0x7f2e74003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:27.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.808+0000 7f2e8f400640 1 --2- 172.21.15.102:0/2743405515 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2e74003970 0x7f2e74003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.808+0000 7f2e8f400640 1 -- 172.21.15.102:0/2743405515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e74005180 msgr2=0x7f2e7400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:27.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.808+0000 7f2e8f400640 1 --2- 172.21.15.102:0/2743405515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e74005180 0x7f2e7400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.808+0000 7f2e8f400640 1 -- 172.21.15.102:0/2743405515 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e74003520 con 0x7f2e740047c0 2024-03-23T03:48:27.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.808+0000 7f2e94c00640 1 -- 172.21.15.102:0/2743405515 reap_dead start 2024-03-23T03:48:27.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.809+0000 7f2e8f400640 1 --2- 172.21.15.102:0/2743405515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e740047c0 0x7f2e74004c40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2e78001e20 tx=0x7f2e740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d693b6e31d4c7335 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:27.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.809+0000 7f2e8ea00640 1 -- 172.21.15.102:0/2743405515 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e78003b70 con 0x7f2e740047c0 2024-03-23T03:48:27.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.809+0000 7f2e8ea00640 1 -- 172.21.15.102:0/2743405515 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e78003cd0 con 0x7f2e740047c0 2024-03-23T03:48:27.811 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.809+0000 7f2e8ea00640 1 -- 172.21.15.102:0/2743405515 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e78005310 con 0x7f2e740047c0 2024-03-23T03:48:27.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.810+0000 7f2e96c00640 1 -- 172.21.15.102:0/2743405515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e740047c0 msgr2=0x7f2e74004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:27.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.810+0000 7f2e96c00640 1 --2- 172.21.15.102:0/2743405515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e740047c0 0x7f2e74004c40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f2e78001e20 tx=0x7f2e740057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.810+0000 7f2e96c00640 1 -- 172.21.15.102:0/2743405515 shutdown_connections 2024-03-23T03:48:27.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.810+0000 7f2e96c00640 1 --2- 172.21.15.102:0/2743405515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e740047c0 0x7f2e74004c40 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.810+0000 7f2e96c00640 1 -- 172.21.15.102:0/2743405515 >> 172.21.15.102:0/2743405515 conn(0x7f2e9006a3d0 msgr2=0x7f2e900b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:27.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.810+0000 7f2e96c00640 1 -- 172.21.15.102:0/2743405515 shutdown_connections 2024-03-23T03:48:27.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.810+0000 7f2e96c00640 1 -- 172.21.15.102:0/2743405515 wait complete. 2024-03-23T03:48:27.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.812+0000 7f2e96c00640 1 Processor -- start 2024-03-23T03:48:27.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.812+0000 7f2e96c00640 1 -- start start 2024-03-23T03:48:27.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.813+0000 7f2e96c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e740047c0 0x7f2e9013eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:27.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.813+0000 7f2e96c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e9013f000 0x7f2e900af450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:27.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.813+0000 7f2e96c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2e900af990 0x7f2e900ac710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:27.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.813+0000 7f2e96c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e900ac2f0 con 0x7f2e740047c0 2024-03-23T03:48:27.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.813+0000 7f2e96c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9013f480 con 0x7f2e9013f000 2024-03-23T03:48:27.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.813+0000 7f2e96c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e900ab950 con 0x7f2e900af990 2024-03-23T03:48:27.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.813+0000 7f2e8fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e740047c0 0x7f2e9013eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:27.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.813+0000 7f2e8fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e740047c0 0x7f2e9013eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44970/0 (socket says 172.21.15.102:44970) 2024-03-23T03:48:27.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.813+0000 7f2e8fe00640 1 -- 172.21.15.102:0/3734745138 learned_addr learned my addr 172.21.15.102:0/3734745138 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:27.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.813+0000 7f2e94c00640 1 --2- 172.21.15.102:0/3734745138 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2e900af990 0x7f2e900ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:27.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.813+0000 7f2e8f400640 1 --2- 172.21.15.102:0/3734745138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e9013f000 0x7f2e900af450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:27.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.813+0000 7f2e8fe00640 1 -- 172.21.15.102:0/3734745138 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2e900af990 msgr2=0x7f2e900ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:27.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.813+0000 7f2e8fe00640 1 --2- 172.21.15.102:0/3734745138 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2e900af990 0x7f2e900ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.813+0000 7f2e8fe00640 1 -- 172.21.15.102:0/3734745138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e9013f000 msgr2=0x7f2e900af450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:27.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.813+0000 7f2e8fe00640 1 --2- 172.21.15.102:0/3734745138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e9013f000 0x7f2e900af450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:27.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.813+0000 7f2e8fe00640 1 -- 172.21.15.102:0/3734745138 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e900abc30 con 0x7f2e740047c0 2024-03-23T03:48:27.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.813+0000 7f2e94c00640 1 --2- 172.21.15.102:0/3734745138 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2e900af990 0x7f2e900ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:27.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.814+0000 7f2e8fe00640 1 --2- 172.21.15.102:0/3734745138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e740047c0 0x7f2e9013eac0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000c700 tx=0x7f2e8000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:27.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.814+0000 7f2e87e00640 1 -- 172.21.15.102:0/3734745138 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e800074e0 con 0x7f2e740047c0 2024-03-23T03:48:27.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.814+0000 7f2e87e00640 1 -- 172.21.15.102:0/3734745138 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e80007640 con 0x7f2e740047c0 2024-03-23T03:48:27.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.814+0000 7f2e87e00640 1 -- 172.21.15.102:0/3734745138 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e80004a20 con 0x7f2e740047c0 2024-03-23T03:48:27.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.814+0000 7f2e96c00640 1 -- 172.21.15.102:0/3734745138 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e900abf10 con 0x7f2e740047c0 2024-03-23T03:48:27.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.814+0000 7f2e96c00640 1 -- 172.21.15.102:0/3734745138 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e900afe40 con 0x7f2e740047c0 2024-03-23T03:48:27.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.815+0000 7f2e96c00640 1 -- 172.21.15.102:0/3734745138 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e9006ac20 con 0x7f2e740047c0 2024-03-23T03:48:27.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.816+0000 7f2e87e00640 1 -- 172.21.15.102:0/3734745138 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2e80004b80 con 0x7f2e740047c0 2024-03-23T03:48:27.817 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.817+0000 7f2e87e00640 1 --2- 172.21.15.102:0/3734745138 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2e6c06ac00 0x7f2e6c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:27.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.817+0000 7f2e87e00640 1 -- 172.21.15.102:0/3734745138 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2e800889d0 con 0x7f2e740047c0 2024-03-23T03:48:27.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.817+0000 7f2e8f400640 1 --2- 172.21.15.102:0/3734745138 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2e6c06ac00 0x7f2e6c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:27.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.818+0000 7f2e8f400640 1 --2- 172.21.15.102:0/3734745138 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2e6c06ac00 0x7f2e6c06d0c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2e78034000 tx=0x7f2e78009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:27.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:27.822+0000 7f2e87e00640 1 -- 172.21.15.102:0/3734745138 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2e80053a70 con 0x7f2e740047c0 2024-03-23T03:48:28.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.011+0000 7f2e96c00640 1 -- 172.21.15.102:0/3734745138 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2e90073390 con 0x7f2e6c06ac00 2024-03-23T03:48:28.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.035+0000 7f2e87e00640 1 -- 172.21.15.102:0/3734745138 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287260 (secure 0 0 0) 0x7f2e90073390 con 0x7f2e6c06ac00 2024-03-23T03:48:28.039 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:28.042 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:48:28.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.040+0000 7f2e96c00640 1 -- 172.21.15.102:0/3734745138 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2e6c06ac00 msgr2=0x7f2e6c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:28.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.040+0000 7f2e96c00640 1 --2- 172.21.15.102:0/3734745138 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2e6c06ac00 0x7f2e6c06d0c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2e78034000 tx=0x7f2e78009030 comp rx=0 tx=0).stop 2024-03-23T03:48:28.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.040+0000 7f2e96c00640 1 -- 172.21.15.102:0/3734745138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e740047c0 msgr2=0x7f2e9013eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:28.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.040+0000 7f2e96c00640 1 --2- 172.21.15.102:0/3734745138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e740047c0 0x7f2e9013eac0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000c700 tx=0x7f2e8000cbc0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.041+0000 7f2e96c00640 1 -- 172.21.15.102:0/3734745138 shutdown_connections 2024-03-23T03:48:28.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.041+0000 7f2e96c00640 1 --2- 172.21.15.102:0/3734745138 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2e6c06ac00 0x7f2e6c06d0c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.041+0000 7f2e96c00640 1 --2- 172.21.15.102:0/3734745138 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2e900af990 0x7f2e900ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.041+0000 7f2e96c00640 1 --2- 172.21.15.102:0/3734745138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e9013f000 0x7f2e900af450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.041+0000 7f2e96c00640 1 --2- 172.21.15.102:0/3734745138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e740047c0 0x7f2e9013eac0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.041+0000 7f2e96c00640 1 -- 172.21.15.102:0/3734745138 >> 172.21.15.102:0/3734745138 conn(0x7f2e9006a3d0 msgr2=0x7f2e90142bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:28.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.041+0000 7f2e96c00640 1 -- 172.21.15.102:0/3734745138 shutdown_connections 2024-03-23T03:48:28.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.041+0000 7f2e96c00640 1 -- 172.21.15.102:0/3734745138 wait complete. 2024-03-23T03:48:28.059 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-03-23T03:48:27.745941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":189,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221072,"kb_used_data":4904,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753648,"statfs":{"total":773094113280,"available":772867735552,"internally_reserved":0,"allocated":5021696,"data_stored":2800860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":221236794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.810117"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438800+0000","last_change":"2024-03-23T03:48:23.446505+0000","last_active":"2024-03-23T03:48:24.438800+0000","last_peered":"2024-03-23T03:48:24.438800+0000","last_clean":"2024-03-23T03:48:24.438800+0000","last_became_active":"2024-03-23T03:48:23.446196+0000","last_became_peered":"2024-03-23T03:48:23.446196+0000","last_unstale":"2024-03-23T03:48:24.438800+0000","last_undegraded":"2024-03-23T03:48:24.438800+0000","last_fullsized":"2024-03-23T03:48:24.438800+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:50:18.596428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438474+0000","last_change":"2024-03-23T03:48:23.441976+0000","last_active":"2024-03-23T03:48:24.438474+0000","last_peered":"2024-03-23T03:48:24.438474+0000","last_clean":"2024-03-23T03:48:24.438474+0000","last_became_active":"2024-03-23T03:48:23.441710+0000","last_became_peered":"2024-03-23T03:48:23.441710+0000","last_unstale":"2024-03-23T03:48:24.438474+0000","last_undegraded":"2024-03-23T03:48:24.438474+0000","last_fullsized":"2024-03-23T03:48:24.438474+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:54:29.194785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377464+0000","last_change":"2024-03-23T03:48:23.464788+0000","last_active":"2024-03-23T03:48:25.377464+0000","last_peered":"2024-03-23T03:48:25.377464+0000","last_clean":"2024-03-23T03:48:25.377464+0000","last_became_active":"2024-03-23T03:48:23.464505+0000","last_became_peered":"2024-03-23T03:48:23.464505+0000","last_unstale":"2024-03-23T03:48:25.377464+0000","last_undegraded":"2024-03-23T03:48:25.377464+0000","last_fullsized":"2024-03-23T03:48:25.377464+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:31:51.225846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.164253+0000","last_change":"2024-03-23T03:48:23.478086+0000","last_active":"2024-03-23T03:48:25.164253+0000","last_peered":"2024-03-23T03:48:25.164253+0000","last_clean":"2024-03-23T03:48:25.164253+0000","last_became_active":"2024-03-23T03:48:23.477848+0000","last_became_peered":"2024-03-23T03:48:23.477848+0000","last_unstale":"2024-03-23T03:48:25.164253+0000","last_undegraded":"2024-03-23T03:48:25.164253+0000","last_fullsized":"2024-03-23T03:48:25.164253+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:48:47.462755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.163400+0000","last_change":"2024-03-23T03:48:23.474665+0000","last_active":"2024-03-23T03:48:25.163400+0000","last_peered":"2024-03-23T03:48:25.163400+0000","last_clean":"2024-03-23T03:48:25.163400+0000","last_became_active":"2024-03-23T03:48:23.474393+0000","last_became_peered":"2024-03-23T03:48:23.474393+0000","last_unstale":"2024-03-23T03:48:25.163400+0000","last_undegraded":"2024-03-23T03:48:25.163400+0000","last_fullsized":"2024-03-23T03:48:25.163400+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:14:11.643509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376014+0000","last_change":"2024-03-23T03:48:23.451143+0000","last_active":"2024-03-23T03:48:25.376014+0000","last_peered":"2024-03-23T03:48:25.376014+0000","last_clean":"2024-03-23T03:48:25.376014+0000","last_became_active":"2024-03-23T03:48:23.449891+0000","last_became_peered":"2024-03-23T03:48:23.449891+0000","last_unstale":"2024-03-23T03:48:25.376014+0000","last_undegraded":"2024-03-23T03:48:25.376014+0000","last_fullsized":"2024-03-23T03:48:25.376014+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:47.834179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.439436+0000","last_change":"2024-03-23T03:48:23.455088+0000","last_active":"2024-03-23T03:48:24.439436+0000","last_peered":"2024-03-23T03:48:24.439436+0000","last_clean":"2024-03-23T03:48:24.439436+0000","last_became_active":"2024-03-23T03:48:23.454790+0000","last_became_peered":"2024-03-23T03:48:23.454790+0000","last_unstale":"2024-03-23T03:48:24.439436+0000","last_undegraded":"2024-03-23T03:48:24.439436+0000","last_fullsized":"2024-03-23T03:48:24.439436+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:51:59.059621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.437672+0000","last_change":"2024-03-23T03:48:23.443514+0000","last_active":"2024-03-23T03:48:24.437672+0000","last_peered":"2024-03-23T03:48:24.437672+0000","last_clean":"2024-03-23T03:48:24.437672+0000","last_became_active":"2024-03-23T03:48:23.443267+0000","last_became_peered":"2024-03-23T03:48:23.443267+0000","last_unstale":"2024-03-23T03:48:24.437672+0000","last_undegraded":"2024-03-23T03:48:24.437672+0000","last_fullsized":"2024-03-23T03:48:24.437672+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:08:40.284453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377924+0000","last_change":"2024-03-23T03:48:23.464811+0000","last_active":"2024-03-23T03:48:25.377924+0000","last_peered":"2024-03-23T03:48:25.377924+0000","last_clean":"2024-03-23T03:48:25.377924+0000","last_became_active":"2024-03-23T03:48:23.464562+0000","last_became_peered":"2024-03-23T03:48:23.464562+0000","last_unstale":"2024-03-23T03:48:25.377924+0000","last_undegraded":"2024-03-23T03:48:25.377924+0000","last_fullsized":"2024-03-23T03:48:25.377924+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:51:59.059621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440737+0000","last_change":"2024-03-23T03:48:23.467159+0000","last_active":"2024-03-23T03:48:24.440737+0000","last_peered":"2024-03-23T03:48:24.440737+0000","last_clean":"2024-03-23T03:48:24.440737+0000","last_became_active":"2024-03-23T03:48:23.466869+0000","last_became_peered":"2024-03-23T03:48:23.466869+0000","last_unstale":"2024-03-23T03:48:24.440737+0000","last_undegraded":"2024-03-23T03:48:24.440737+0000","last_fullsized":"2024-03-23T03:48:24.440737+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:53.495024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440388+0000","last_change":"2024-03-23T03:48:23.466262+0000","last_active":"2024-03-23T03:48:24.440388+0000","last_peered":"2024-03-23T03:48:24.440388+0000","last_clean":"2024-03-23T03:48:24.440388+0000","last_became_active":"2024-03-23T03:48:23.466000+0000","last_became_peered":"2024-03-23T03:48:23.466000+0000","last_unstale":"2024-03-23T03:48:24.440388+0000","last_undegraded":"2024-03-23T03:48:24.440388+0000","last_fullsized":"2024-03-23T03:48:24.440388+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:38:59.566199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377452+0000","last_change":"2024-03-23T03:48:23.445532+0000","last_active":"2024-03-23T03:48:25.377452+0000","last_peered":"2024-03-23T03:48:25.377452+0000","last_clean":"2024-03-23T03:48:25.377452+0000","last_became_active":"2024-03-23T03:48:23.445352+0000","last_became_peered":"2024-03-23T03:48:23.445352+0000","last_unstale":"2024-03-23T03:48:25.377452+0000","last_undegraded":"2024-03-23T03:48:25.377452+0000","last_fullsized":"2024-03-23T03:48:25.377452+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:38:59.566199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.439862+0000","last_change":"2024-03-23T03:48:23.467235+0000","last_active":"2024-03-23T03:48:24.439862+0000","last_peered":"2024-03-23T03:48:24.439862+0000","last_clean":"2024-03-23T03:48:24.439862+0000","last_became_active":"2024-03-23T03:48:23.466876+0000","last_became_peered":"2024-03-23T03:48:23.466876+0000","last_unstale":"2024-03-23T03:48:24.439862+0000","last_undegraded":"2024-03-23T03:48:24.439862+0000","last_fullsized":"2024-03-23T03:48:24.439862+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:10:35.688861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377850+0000","last_change":"2024-03-23T03:48:23.479234+0000","last_active":"2024-03-23T03:48:25.377850+0000","last_peered":"2024-03-23T03:48:25.377850+0000","last_clean":"2024-03-23T03:48:25.377850+0000","last_became_active":"2024-03-23T03:48:23.477566+0000","last_became_peered":"2024-03-23T03:48:23.477566+0000","last_unstale":"2024-03-23T03:48:25.377850+0000","last_undegraded":"2024-03-23T03:48:25.377850+0000","last_fullsized":"2024-03-23T03:48:25.377850+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:10:24.611081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.439597+0000","last_change":"2024-03-23T03:48:23.445888+0000","last_active":"2024-03-23T03:48:24.439597+0000","last_peered":"2024-03-23T03:48:24.439597+0000","last_clean":"2024-03-23T03:48:24.439597+0000","last_became_active":"2024-03-23T03:48:23.443616+0000","last_became_peered":"2024-03-23T03:48:23.443616+0000","last_unstale":"2024-03-23T03:48:24.439597+0000","last_undegraded":"2024-03-23T03:48:24.439597+0000","last_fullsized":"2024-03-23T03:48:24.439597+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:07:31.748044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.660364+0000","last_change":"2024-03-23T03:48:23.470703+0000","last_active":"2024-03-23T03:48:24.660364+0000","last_peered":"2024-03-23T03:48:24.660364+0000","last_clean":"2024-03-23T03:48:24.660364+0000","last_became_active":"2024-03-23T03:48:23.470405+0000","last_became_peered":"2024-03-23T03:48:23.470405+0000","last_unstale":"2024-03-23T03:48:24.660364+0000","last_undegraded":"2024-03-23T03:48:24.660364+0000","last_fullsized":"2024-03-23T03:48:24.660364+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:06:21.756726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661647+0000","last_change":"2024-03-23T03:48:23.471774+0000","last_active":"2024-03-23T03:48:24.661647+0000","last_peered":"2024-03-23T03:48:24.661647+0000","last_clean":"2024-03-23T03:48:24.661647+0000","last_became_active":"2024-03-23T03:48:23.470743+0000","last_became_peered":"2024-03-23T03:48:23.470743+0000","last_unstale":"2024-03-23T03:48:24.661647+0000","last_undegraded":"2024-03-23T03:48:24.661647+0000","last_fullsized":"2024-03-23T03:48:24.661647+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:20:23.375166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377019+0000","last_change":"2024-03-23T03:48:23.476885+0000","last_active":"2024-03-23T03:48:25.377019+0000","last_peered":"2024-03-23T03:48:25.377019+0000","last_clean":"2024-03-23T03:48:25.377019+0000","last_became_active":"2024-03-23T03:48:23.476467+0000","last_became_peered":"2024-03-23T03:48:23.476467+0000","last_unstale":"2024-03-23T03:48:25.377019+0000","last_undegraded":"2024-03-23T03:48:25.377019+0000","last_fullsized":"2024-03-23T03:48:25.377019+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:01:01.145513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.163915+0000","last_change":"2024-03-23T03:48:23.448656+0000","last_active":"2024-03-23T03:48:25.163915+0000","last_peered":"2024-03-23T03:48:25.163915+0000","last_clean":"2024-03-23T03:48:25.163915+0000","last_became_active":"2024-03-23T03:48:23.448387+0000","last_became_peered":"2024-03-23T03:48:23.448387+0000","last_unstale":"2024-03-23T03:48:25.163915+0000","last_undegraded":"2024-03-23T03:48:25.163915+0000","last_fullsized":"2024-03-23T03:48:25.163915+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:08:00.804604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376605+0000","last_change":"2024-03-23T03:48:23.476883+0000","last_active":"2024-03-23T03:48:25.376605+0000","last_peered":"2024-03-23T03:48:25.376605+0000","last_clean":"2024-03-23T03:48:25.376605+0000","last_became_active":"2024-03-23T03:48:23.476549+0000","last_became_peered":"2024-03-23T03:48:23.476549+0000","last_unstale":"2024-03-23T03:48:25.376605+0000","last_undegraded":"2024-03-23T03:48:25.376605+0000","last_fullsized":"2024-03-23T03:48:25.376605+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:26:46.940664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440215+0000","last_change":"2024-03-23T03:48:23.461082+0000","last_active":"2024-03-23T03:48:24.440215+0000","last_peered":"2024-03-23T03:48:24.440215+0000","last_clean":"2024-03-23T03:48:24.440215+0000","last_became_active":"2024-03-23T03:48:23.459352+0000","last_became_peered":"2024-03-23T03:48:23.459352+0000","last_unstale":"2024-03-23T03:48:24.440215+0000","last_undegraded":"2024-03-23T03:48:24.440215+0000","last_fullsized":"2024-03-23T03:48:24.440215+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:31:51.225846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.378320+0000","last_change":"2024-03-23T03:48:23.463523+0000","last_active":"2024-03-23T03:48:25.378320+0000","last_peered":"2024-03-23T03:48:25.378320+0000","last_clean":"2024-03-23T03:48:25.378320+0000","last_became_active":"2024-03-23T03:48:23.463139+0000","last_became_peered":"2024-03-23T03:48:23.463139+0000","last_unstale":"2024-03-23T03:48:25.378320+0000","last_undegraded":"2024-03-23T03:48:25.378320+0000","last_fullsized":"2024-03-23T03:48:25.378320+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:13.565417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.660949+0000","last_change":"2024-03-23T03:48:23.457871+0000","last_active":"2024-03-23T03:48:24.660949+0000","last_peered":"2024-03-23T03:48:24.660949+0000","last_clean":"2024-03-23T03:48:24.660949+0000","last_became_active":"2024-03-23T03:48:23.457583+0000","last_became_peered":"2024-03-23T03:48:23.457583+0000","last_unstale":"2024-03-23T03:48:24.660949+0000","last_undegraded":"2024-03-23T03:48:24.660949+0000","last_fullsized":"2024-03-23T03:48:24.660949+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:33:05.984877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.163785+0000","last_change":"2024-03-23T03:48:23.459008+0000","last_active":"2024-03-23T03:48:25.163785+0000","last_peered":"2024-03-23T03:48:25.163785+0000","last_clean":"2024-03-23T03:48:25.163785+0000","last_became_active":"2024-03-23T03:48:23.458563+0000","last_became_peered":"2024-03-23T03:48:23.458563+0000","last_unstale":"2024-03-23T03:48:25.163785+0000","last_undegraded":"2024-03-23T03:48:25.163785+0000","last_fullsized":"2024-03-23T03:48:25.163785+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:48:17.163517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376973+0000","last_change":"2024-03-23T03:48:23.464393+0000","last_active":"2024-03-23T03:48:25.376973+0000","last_peered":"2024-03-23T03:48:25.376973+0000","last_clean":"2024-03-23T03:48:25.376973+0000","last_became_active":"2024-03-23T03:48:23.463855+0000","last_became_peered":"2024-03-23T03:48:23.463855+0000","last_unstale":"2024-03-23T03:48:25.376973+0000","last_undegraded":"2024-03-23T03:48:25.376973+0000","last_fullsized":"2024-03-23T03:48:25.376973+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:20:08.939333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376610+0000","last_change":"2024-03-23T03:48:23.470934+0000","last_active":"2024-03-23T03:48:25.376610+0000","last_peered":"2024-03-23T03:48:25.376610+0000","last_clean":"2024-03-23T03:48:25.376610+0000","last_became_active":"2024-03-23T03:48:23.470661+0000","last_became_peered":"2024-03-23T03:48:23.470661+0000","last_unstale":"2024-03-23T03:48:25.376610+0000","last_undegraded":"2024-03-23T03:48:25.376610+0000","last_fullsized":"2024-03-23T03:48:25.376610+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:01:27.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376860+0000","last_change":"2024-03-23T03:48:23.462765+0000","last_active":"2024-03-23T03:48:25.376860+0000","last_peered":"2024-03-23T03:48:25.376860+0000","last_clean":"2024-03-23T03:48:25.376860+0000","last_became_active":"2024-03-23T03:48:23.462428+0000","last_became_peered":"2024-03-23T03:48:23.462428+0000","last_unstale":"2024-03-23T03:48:25.376860+0000","last_undegraded":"2024-03-23T03:48:25.376860+0000","last_fullsized":"2024-03-23T03:48:25.376860+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:24:30.974122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440204+0000","last_change":"2024-03-23T03:48:23.444152+0000","last_active":"2024-03-23T03:48:24.440204+0000","last_peered":"2024-03-23T03:48:24.440204+0000","last_clean":"2024-03-23T03:48:24.440204+0000","last_became_active":"2024-03-23T03:48:23.441122+0000","last_became_peered":"2024-03-23T03:48:23.441122+0000","last_unstale":"2024-03-23T03:48:24.440204+0000","last_undegraded":"2024-03-23T03:48:24.440204+0000","last_fullsized":"2024-03-23T03:48:24.440204+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:01:27.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376520+0000","last_change":"2024-03-23T03:48:23.461956+0000","last_active":"2024-03-23T03:48:25.376520+0000","last_peered":"2024-03-23T03:48:25.376520+0000","last_clean":"2024-03-23T03:48:25.376520+0000","last_became_active":"2024-03-23T03:48:23.460093+0000","last_became_peered":"2024-03-23T03:48:23.460093+0000","last_unstale":"2024-03-23T03:48:25.376520+0000","last_undegraded":"2024-03-23T03:48:25.376520+0000","last_fullsized":"2024-03-23T03:48:25.376520+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:48:47.462755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376542+0000","last_change":"2024-03-23T03:48:23.456528+0000","last_active":"2024-03-23T03:48:25.376542+0000","last_peered":"2024-03-23T03:48:25.376542+0000","last_clean":"2024-03-23T03:48:25.376542+0000","last_became_active":"2024-03-23T03:48:23.456155+0000","last_became_peered":"2024-03-23T03:48:23.456155+0000","last_unstale":"2024-03-23T03:48:25.376542+0000","last_undegraded":"2024-03-23T03:48:25.376542+0000","last_fullsized":"2024-03-23T03:48:25.376542+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:14:11.643509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661437+0000","last_change":"2024-03-23T03:48:23.458360+0000","last_active":"2024-03-23T03:48:24.661437+0000","last_peered":"2024-03-23T03:48:24.661437+0000","last_clean":"2024-03-23T03:48:24.661437+0000","last_became_active":"2024-03-23T03:48:23.458065+0000","last_became_peered":"2024-03-23T03:48:23.458065+0000","last_unstale":"2024-03-23T03:48:24.661437+0000","last_undegraded":"2024-03-23T03:48:24.661437+0000","last_fullsized":"2024-03-23T03:48:24.661437+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:45:45.295605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.375746+0000","last_change":"2024-03-23T03:48:23.443932+0000","last_active":"2024-03-23T03:48:25.375746+0000","last_peered":"2024-03-23T03:48:25.375746+0000","last_clean":"2024-03-23T03:48:25.375746+0000","last_became_active":"2024-03-23T03:48:23.442920+0000","last_became_peered":"2024-03-23T03:48:23.442920+0000","last_unstale":"2024-03-23T03:48:25.375746+0000","last_undegraded":"2024-03-23T03:48:25.375746+0000","last_fullsized":"2024-03-23T03:48:25.375746+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:06:21.756726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661136+0000","last_change":"2024-03-23T03:48:23.470905+0000","last_active":"2024-03-23T03:48:24.661136+0000","last_peered":"2024-03-23T03:48:24.661136+0000","last_clean":"2024-03-23T03:48:24.661136+0000","last_became_active":"2024-03-23T03:48:23.470608+0000","last_became_peered":"2024-03-23T03:48:23.470608+0000","last_unstale":"2024-03-23T03:48:24.661136+0000","last_undegraded":"2024-03-23T03:48:24.661136+0000","last_fullsized":"2024-03-23T03:48:24.661136+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:47.834179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"40'4","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376976+0000","last_change":"2024-03-23T03:48:22.972325+0000","last_active":"2024-03-23T03:48:25.376976+0000","last_peered":"2024-03-23T03:48:25.376976+0000","last_clean":"2024-03-23T03:48:25.376976+0000","last_became_active":"2024-03-23T03:48:22.972050+0000","last_became_peered":"2024-03-23T03:48:22.972050+0000","last_unstale":"2024-03-23T03:48:25.376976+0000","last_undegraded":"2024-03-23T03:48:25.376976+0000","last_fullsized":"2024-03-23T03:48:25.376976+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:27:38.886883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377344+0000","last_change":"2024-03-23T03:48:23.470963+0000","last_active":"2024-03-23T03:48:25.377344+0000","last_peered":"2024-03-23T03:48:25.377344+0000","last_clean":"2024-03-23T03:48:25.377344+0000","last_became_active":"2024-03-23T03:48:23.470717+0000","last_became_peered":"2024-03-23T03:48:23.470717+0000","last_unstale":"2024-03-23T03:48:25.377344+0000","last_undegraded":"2024-03-23T03:48:25.377344+0000","last_fullsized":"2024-03-23T03:48:25.377344+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:52:40.102104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"40'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377732+0000","last_change":"2024-03-23T03:48:22.980772+0000","last_active":"2024-03-23T03:48:25.377732+0000","last_peered":"2024-03-23T03:48:25.377732+0000","last_clean":"2024-03-23T03:48:25.377732+0000","last_became_active":"2024-03-23T03:48:22.980177+0000","last_became_peered":"2024-03-23T03:48:22.980177+0000","last_unstale":"2024-03-23T03:48:25.377732+0000","last_undegraded":"2024-03-23T03:48:25.377732+0000","last_fullsized":"2024-03-23T03:48:25.377732+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:33:04.970397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.378787+0000","last_change":"2024-03-23T03:48:23.475651+0000","last_active":"2024-03-23T03:48:25.378787+0000","last_peered":"2024-03-23T03:48:25.378787+0000","last_clean":"2024-03-23T03:48:25.378787+0000","last_became_active":"2024-03-23T03:48:23.475209+0000","last_became_peered":"2024-03-23T03:48:23.475209+0000","last_unstale":"2024-03-23T03:48:25.378787+0000","last_undegraded":"2024-03-23T03:48:25.378787+0000","last_fullsized":"2024-03-23T03:48:25.378787+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:39:51.342220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"40'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377566+0000","last_change":"2024-03-23T03:48:22.984879+0000","last_active":"2024-03-23T03:48:25.377566+0000","last_peered":"2024-03-23T03:48:25.377566+0000","last_clean":"2024-03-23T03:48:25.377566+0000","last_became_active":"2024-03-23T03:48:22.984651+0000","last_became_peered":"2024-03-23T03:48:22.984651+0000","last_unstale":"2024-03-23T03:48:25.377566+0000","last_undegraded":"2024-03-23T03:48:25.377566+0000","last_fullsized":"2024-03-23T03:48:25.377566+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:40:21.057524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438637+0000","last_change":"2024-03-23T03:48:23.464765+0000","last_active":"2024-03-23T03:48:24.438637+0000","last_peered":"2024-03-23T03:48:24.438637+0000","last_clean":"2024-03-23T03:48:24.438637+0000","last_became_active":"2024-03-23T03:48:23.464335+0000","last_became_peered":"2024-03-23T03:48:23.464335+0000","last_unstale":"2024-03-23T03:48:24.438637+0000","last_undegraded":"2024-03-23T03:48:24.438637+0000","last_fullsized":"2024-03-23T03:48:24.438637+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:36:23.713289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"40'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.378712+0000","last_change":"2024-03-23T03:48:22.980265+0000","last_active":"2024-03-23T03:48:25.378712+0000","last_peered":"2024-03-23T03:48:25.378712+0000","last_clean":"2024-03-23T03:48:25.378712+0000","last_became_active":"2024-03-23T03:48:22.979981+0000","last_became_peered":"2024-03-23T03:48:22.979981+0000","last_unstale":"2024-03-23T03:48:25.378712+0000","last_undegraded":"2024-03-23T03:48:25.378712+0000","last_fullsized":"2024-03-23T03:48:25.378712+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:01:00.131033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376275+0000","last_change":"2024-03-23T03:48:23.447954+0000","last_active":"2024-03-23T03:48:25.376275+0000","last_peered":"2024-03-23T03:48:25.376275+0000","last_clean":"2024-03-23T03:48:25.376275+0000","last_became_active":"2024-03-23T03:48:23.447696+0000","last_became_peered":"2024-03-23T03:48:23.447696+0000","last_unstale":"2024-03-23T03:48:25.376275+0000","last_undegraded":"2024-03-23T03:48:25.376275+0000","last_fullsized":"2024-03-23T03:48:25.376275+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:53.495024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"40'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440460+0000","last_change":"2024-03-23T03:48:22.979758+0000","last_active":"2024-03-23T03:48:24.440460+0000","last_peered":"2024-03-23T03:48:24.440460+0000","last_clean":"2024-03-23T03:48:24.440460+0000","last_became_active":"2024-03-23T03:48:22.979478+0000","last_became_peered":"2024-03-23T03:48:22.979478+0000","last_unstale":"2024-03-23T03:48:24.440460+0000","last_undegraded":"2024-03-23T03:48:24.440460+0000","last_fullsized":"2024-03-23T03:48:24.440460+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:32:24.360007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.439715+0000","last_change":"2024-03-23T03:48:23.467403+0000","last_active":"2024-03-23T03:48:24.439715+0000","last_peered":"2024-03-23T03:48:24.439715+0000","last_clean":"2024-03-23T03:48:24.439715+0000","last_became_active":"2024-03-23T03:48:23.466694+0000","last_became_peered":"2024-03-23T03:48:23.466694+0000","last_unstale":"2024-03-23T03:48:24.439715+0000","last_undegraded":"2024-03-23T03:48:24.439715+0000","last_fullsized":"2024-03-23T03:48:24.439715+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:26:23.251559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"40'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661186+0000","last_change":"2024-03-23T03:48:22.989913+0000","last_active":"2024-03-23T03:48:24.661186+0000","last_peered":"2024-03-23T03:48:24.661186+0000","last_clean":"2024-03-23T03:48:24.661186+0000","last_became_active":"2024-03-23T03:48:22.989594+0000","last_became_peered":"2024-03-23T03:48:22.989594+0000","last_unstale":"2024-03-23T03:48:24.661186+0000","last_undegraded":"2024-03-23T03:48:24.661186+0000","last_fullsized":"2024-03-23T03:48:24.661186+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:35:43.667112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.378481+0000","last_change":"2024-03-23T03:48:23.478954+0000","last_active":"2024-03-23T03:48:25.378481+0000","last_peered":"2024-03-23T03:48:25.378481+0000","last_clean":"2024-03-23T03:48:25.378481+0000","last_became_active":"2024-03-23T03:48:23.476063+0000","last_became_peered":"2024-03-23T03:48:23.476063+0000","last_unstale":"2024-03-23T03:48:25.378481+0000","last_undegraded":"2024-03-23T03:48:25.378481+0000","last_fullsized":"2024-03-23T03:48:25.378481+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:23:17.267833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376082+0000","last_change":"2024-03-23T03:48:22.985851+0000","last_active":"2024-03-23T03:48:25.376082+0000","last_peered":"2024-03-23T03:48:25.376082+0000","last_clean":"2024-03-23T03:48:25.376082+0000","last_became_active":"2024-03-23T03:48:22.985611+0000","last_became_peered":"2024-03-23T03:48:22.985611+0000","last_unstale":"2024-03-23T03:48:25.376082+0000","last_undegraded":"2024-03-23T03:48:25.376082+0000","last_fullsized":"2024-03-23T03:48:25.376082+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:17:17.358011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438071+0000","last_change":"2024-03-23T03:48:23.457581+0000","last_active":"2024-03-23T03:48:24.438071+0000","last_peered":"2024-03-23T03:48:24.438071+0000","last_clean":"2024-03-23T03:48:24.438071+0000","last_became_active":"2024-03-23T03:48:23.457341+0000","last_became_peered":"2024-03-23T03:48:23.457341+0000","last_unstale":"2024-03-23T03:48:24.438071+0000","last_undegraded":"2024-03-23T03:48:24.438071+0000","last_fullsized":"2024-03-23T03:48:24.438071+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:12:31.350656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377837+0000","last_change":"2024-03-23T03:48:22.977569+0000","last_active":"2024-03-23T03:48:25.377837+0000","last_peered":"2024-03-23T03:48:25.377837+0000","last_clean":"2024-03-23T03:48:25.377837+0000","last_became_active":"2024-03-23T03:48:22.977095+0000","last_became_peered":"2024-03-23T03:48:22.977095+0000","last_unstale":"2024-03-23T03:48:25.377837+0000","last_undegraded":"2024-03-23T03:48:25.377837+0000","last_fullsized":"2024-03-23T03:48:25.377837+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:51:29.105968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.164191+0000","last_change":"2024-03-23T03:48:23.478104+0000","last_active":"2024-03-23T03:48:25.164191+0000","last_peered":"2024-03-23T03:48:25.164191+0000","last_clean":"2024-03-23T03:48:25.164191+0000","last_became_active":"2024-03-23T03:48:23.477839+0000","last_became_peered":"2024-03-23T03:48:23.477839+0000","last_unstale":"2024-03-23T03:48:25.164191+0000","last_undegraded":"2024-03-23T03:48:25.164191+0000","last_fullsized":"2024-03-23T03:48:25.164191+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:24:30.974122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438914+0000","last_change":"2024-03-23T03:48:22.972812+0000","last_active":"2024-03-23T03:48:24.438914+0000","last_peered":"2024-03-23T03:48:24.438914+0000","last_clean":"2024-03-23T03:48:24.438914+0000","last_became_active":"2024-03-23T03:48:22.972293+0000","last_became_peered":"2024-03-23T03:48:22.972293+0000","last_unstale":"2024-03-23T03:48:24.438914+0000","last_undegraded":"2024-03-23T03:48:24.438914+0000","last_fullsized":"2024-03-23T03:48:24.438914+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:24:37.457522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661469+0000","last_change":"2024-03-23T03:48:23.474240+0000","last_active":"2024-03-23T03:48:24.661469+0000","last_peered":"2024-03-23T03:48:24.661469+0000","last_clean":"2024-03-23T03:48:24.661469+0000","last_became_active":"2024-03-23T03:48:23.473951+0000","last_became_peered":"2024-03-23T03:48:23.473951+0000","last_unstale":"2024-03-23T03:48:24.661469+0000","last_undegraded":"2024-03-23T03:48:24.661469+0000","last_fullsized":"2024-03-23T03:48:24.661469+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:19:13.754297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440836+0000","last_change":"2024-03-23T03:48:22.982207+0000","last_active":"2024-03-23T03:48:24.440836+0000","last_peered":"2024-03-23T03:48:24.440836+0000","last_clean":"2024-03-23T03:48:24.440836+0000","last_became_active":"2024-03-23T03:48:22.981866+0000","last_became_peered":"2024-03-23T03:48:22.981866+0000","last_unstale":"2024-03-23T03:48:24.440836+0000","last_undegraded":"2024-03-23T03:48:24.440836+0000","last_fullsized":"2024-03-23T03:48:24.440836+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:44:48.293049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377728+0000","last_change":"2024-03-23T03:48:23.470131+0000","last_active":"2024-03-23T03:48:25.377728+0000","last_peered":"2024-03-23T03:48:25.377728+0000","last_clean":"2024-03-23T03:48:25.377728+0000","last_became_active":"2024-03-23T03:48:23.469818+0000","last_became_peered":"2024-03-23T03:48:23.469818+0000","last_unstale":"2024-03-23T03:48:25.377728+0000","last_undegraded":"2024-03-23T03:48:25.377728+0000","last_fullsized":"2024-03-23T03:48:25.377728+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:26:23.251559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"40'3","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440676+0000","last_change":"2024-03-23T03:48:22.986034+0000","last_active":"2024-03-23T03:48:24.440676+0000","last_peered":"2024-03-23T03:48:24.440676+0000","last_clean":"2024-03-23T03:48:24.440676+0000","last_became_active":"2024-03-23T03:48:22.985641+0000","last_became_peered":"2024-03-23T03:48:22.985641+0000","last_unstale":"2024-03-23T03:48:24.440676+0000","last_undegraded":"2024-03-23T03:48:24.440676+0000","last_fullsized":"2024-03-23T03:48:24.440676+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:29.782326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.378929+0000","last_change":"2024-03-23T03:48:23.464888+0000","last_active":"2024-03-23T03:48:25.378929+0000","last_peered":"2024-03-23T03:48:25.378929+0000","last_clean":"2024-03-23T03:48:25.378929+0000","last_became_active":"2024-03-23T03:48:23.464439+0000","last_became_peered":"2024-03-23T03:48:23.464439+0000","last_unstale":"2024-03-23T03:48:25.378929+0000","last_undegraded":"2024-03-23T03:48:25.378929+0000","last_fullsized":"2024-03-23T03:48:25.378929+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:02:16.431086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"40'6","reported_seq":17,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438531+0000","last_change":"2024-03-23T03:48:22.978241+0000","last_active":"2024-03-23T03:48:24.438531+0000","last_peered":"2024-03-23T03:48:24.438531+0000","last_clean":"2024-03-23T03:48:24.438531+0000","last_became_active":"2024-03-23T03:48:22.978012+0000","last_became_peered":"2024-03-23T03:48:22.978012+0000","last_unstale":"2024-03-23T03:48:24.438531+0000","last_undegraded":"2024-03-23T03:48:24.438531+0000","last_fullsized":"2024-03-23T03:48:24.438531+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:12.237431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376527+0000","last_change":"2024-03-23T03:48:23.456812+0000","last_active":"2024-03-23T03:48:25.376527+0000","last_peered":"2024-03-23T03:48:25.376527+0000","last_clean":"2024-03-23T03:48:25.376527+0000","last_became_active":"2024-03-23T03:48:23.456502+0000","last_became_peered":"2024-03-23T03:48:23.456502+0000","last_unstale":"2024-03-23T03:48:25.376527+0000","last_undegraded":"2024-03-23T03:48:25.376527+0000","last_fullsized":"2024-03-23T03:48:25.376527+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:10:35.688861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"40'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438242+0000","last_change":"2024-03-23T03:48:22.969089+0000","last_active":"2024-03-23T03:48:24.438242+0000","last_peered":"2024-03-23T03:48:24.438242+0000","last_clean":"2024-03-23T03:48:24.438242+0000","last_became_active":"2024-03-23T03:48:22.968809+0000","last_became_peered":"2024-03-23T03:48:22.968809+0000","last_unstale":"2024-03-23T03:48:24.438242+0000","last_undegraded":"2024-03-23T03:48:24.438242+0000","last_fullsized":"2024-03-23T03:48:24.438242+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:51:16.799572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.437807+0000","last_change":"2024-03-23T03:48:23.442645+0000","last_active":"2024-03-23T03:48:24.437807+0000","last_peered":"2024-03-23T03:48:24.437807+0000","last_clean":"2024-03-23T03:48:24.437807+0000","last_became_active":"2024-03-23T03:48:23.441584+0000","last_became_peered":"2024-03-23T03:48:23.441584+0000","last_unstale":"2024-03-23T03:48:24.437807+0000","last_undegraded":"2024-03-23T03:48:24.437807+0000","last_fullsized":"2024-03-23T03:48:24.437807+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:49:28.083244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376147+0000","last_change":"2024-03-23T03:48:22.981960+0000","last_active":"2024-03-23T03:48:25.376147+0000","last_peered":"2024-03-23T03:48:25.376147+0000","last_clean":"2024-03-23T03:48:25.376147+0000","last_became_active":"2024-03-23T03:48:22.981671+0000","last_became_peered":"2024-03-23T03:48:22.981671+0000","last_unstale":"2024-03-23T03:48:25.376147+0000","last_undegraded":"2024-03-23T03:48:25.376147+0000","last_fullsized":"2024-03-23T03:48:25.376147+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:32:24.360007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.660955+0000","last_change":"2024-03-23T03:48:23.459475+0000","last_active":"2024-03-23T03:48:24.660955+0000","last_peered":"2024-03-23T03:48:24.660955+0000","last_clean":"2024-03-23T03:48:24.660955+0000","last_became_active":"2024-03-23T03:48:23.458903+0000","last_became_peered":"2024-03-23T03:48:23.458903+0000","last_unstale":"2024-03-23T03:48:24.660955+0000","last_undegraded":"2024-03-23T03:48:24.660955+0000","last_fullsized":"2024-03-23T03:48:24.660955+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:19:04.667783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376087+0000","last_change":"2024-03-23T03:48:22.971422+0000","last_active":"2024-03-23T03:48:25.376087+0000","last_peered":"2024-03-23T03:48:25.376087+0000","last_clean":"2024-03-23T03:48:25.376087+0000","last_became_active":"2024-03-23T03:48:22.971144+0000","last_became_peered":"2024-03-23T03:48:22.971144+0000","last_unstale":"2024-03-23T03:48:25.376087+0000","last_undegraded":"2024-03-23T03:48:25.376087+0000","last_fullsized":"2024-03-23T03:48:25.376087+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:18:10.874509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438143+0000","last_change":"2024-03-23T03:48:23.439626+0000","last_active":"2024-03-23T03:48:24.438143+0000","last_peered":"2024-03-23T03:48:24.438143+0000","last_clean":"2024-03-23T03:48:24.438143+0000","last_became_active":"2024-03-23T03:48:23.439357+0000","last_became_peered":"2024-03-23T03:48:23.439357+0000","last_unstale":"2024-03-23T03:48:24.438143+0000","last_undegraded":"2024-03-23T03:48:24.438143+0000","last_fullsized":"2024-03-23T03:48:24.438143+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:49:31.471021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"40'8","reported_seq":19,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.660555+0000","last_change":"2024-03-23T03:48:22.989453+0000","last_active":"2024-03-23T03:48:24.660555+0000","last_peered":"2024-03-23T03:48:24.660555+0000","last_clean":"2024-03-23T03:48:24.660555+0000","last_became_active":"2024-03-23T03:48:22.989205+0000","last_became_peered":"2024-03-23T03:48:22.989205+0000","last_unstale":"2024-03-23T03:48:24.660555+0000","last_undegraded":"2024-03-23T03:48:24.660555+0000","last_fullsized":"2024-03-23T03:48:24.660555+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:54.181764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438396+0000","last_change":"2024-03-23T03:48:23.462731+0000","last_active":"2024-03-23T03:48:24.438396+0000","last_peered":"2024-03-23T03:48:24.438396+0000","last_clean":"2024-03-23T03:48:24.438396+0000","last_became_active":"2024-03-23T03:48:23.462466+0000","last_became_peered":"2024-03-23T03:48:23.462466+0000","last_unstale":"2024-03-23T03:48:24.438396+0000","last_undegraded":"2024-03-23T03:48:24.438396+0000","last_fullsized":"2024-03-23T03:48:24.438396+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:56:20.636759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"40'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440887+0000","last_change":"2024-03-23T03:48:22.976437+0000","last_active":"2024-03-23T03:48:24.440887+0000","last_peered":"2024-03-23T03:48:24.440887+0000","last_clean":"2024-03-23T03:48:24.440887+0000","last_became_active":"2024-03-23T03:48:22.976178+0000","last_became_peered":"2024-03-23T03:48:22.976178+0000","last_unstale":"2024-03-23T03:48:24.440887+0000","last_undegraded":"2024-03-23T03:48:24.440887+0000","last_fullsized":"2024-03-23T03:48:24.440887+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:17:17.358011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440546+0000","last_change":"2024-03-23T03:48:23.464017+0000","last_active":"2024-03-23T03:48:24.440546+0000","last_peered":"2024-03-23T03:48:24.440546+0000","last_clean":"2024-03-23T03:48:24.440546+0000","last_became_active":"2024-03-23T03:48:23.463538+0000","last_became_peered":"2024-03-23T03:48:23.463538+0000","last_unstale":"2024-03-23T03:48:24.440546+0000","last_undegraded":"2024-03-23T03:48:24.440546+0000","last_fullsized":"2024-03-23T03:48:24.440546+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:57.996508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"40'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.378927+0000","last_change":"2024-03-23T03:48:22.984999+0000","last_active":"2024-03-23T03:48:25.378927+0000","last_peered":"2024-03-23T03:48:25.378927+0000","last_clean":"2024-03-23T03:48:25.378927+0000","last_became_active":"2024-03-23T03:48:22.984693+0000","last_became_peered":"2024-03-23T03:48:22.984693+0000","last_unstale":"2024-03-23T03:48:25.378927+0000","last_undegraded":"2024-03-23T03:48:25.378927+0000","last_fullsized":"2024-03-23T03:48:25.378927+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:54:00.988275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.378654+0000","last_change":"2024-03-23T03:48:23.464489+0000","last_active":"2024-03-23T03:48:25.378654+0000","last_peered":"2024-03-23T03:48:25.378654+0000","last_clean":"2024-03-23T03:48:25.378654+0000","last_became_active":"2024-03-23T03:48:23.463867+0000","last_became_peered":"2024-03-23T03:48:23.463867+0000","last_unstale":"2024-03-23T03:48:25.378654+0000","last_undegraded":"2024-03-23T03:48:25.378654+0000","last_fullsized":"2024-03-23T03:48:25.378654+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:17:34.748041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"40'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.164012+0000","last_change":"2024-03-23T03:48:22.988224+0000","last_active":"2024-03-23T03:48:25.164012+0000","last_peered":"2024-03-23T03:48:25.164012+0000","last_clean":"2024-03-23T03:48:25.164012+0000","last_became_active":"2024-03-23T03:48:22.987976+0000","last_became_peered":"2024-03-23T03:48:22.987976+0000","last_unstale":"2024-03-23T03:48:25.164012+0000","last_undegraded":"2024-03-23T03:48:25.164012+0000","last_fullsized":"2024-03-23T03:48:25.164012+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:42:16.089902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.164002+0000","last_change":"2024-03-23T03:48:23.449959+0000","last_active":"2024-03-23T03:48:25.164002+0000","last_peered":"2024-03-23T03:48:25.164002+0000","last_clean":"2024-03-23T03:48:25.164002+0000","last_became_active":"2024-03-23T03:48:23.449703+0000","last_became_peered":"2024-03-23T03:48:23.449703+0000","last_unstale":"2024-03-23T03:48:25.164002+0000","last_undegraded":"2024-03-23T03:48:25.164002+0000","last_fullsized":"2024-03-23T03:48:25.164002+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:34:53.562904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376514+0000","last_change":"2024-03-23T03:48:22.981892+0000","last_active":"2024-03-23T03:48:25.376514+0000","last_peered":"2024-03-23T03:48:25.376514+0000","last_clean":"2024-03-23T03:48:25.376514+0000","last_became_active":"2024-03-23T03:48:22.981609+0000","last_became_peered":"2024-03-23T03:48:22.981609+0000","last_unstale":"2024-03-23T03:48:25.376514+0000","last_undegraded":"2024-03-23T03:48:25.376514+0000","last_fullsized":"2024-03-23T03:48:25.376514+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:44:48.293049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.660626+0000","last_change":"2024-03-23T03:48:23.458010+0000","last_active":"2024-03-23T03:48:24.660626+0000","last_peered":"2024-03-23T03:48:24.660626+0000","last_clean":"2024-03-23T03:48:24.660626+0000","last_became_active":"2024-03-23T03:48:23.457723+0000","last_became_peered":"2024-03-23T03:48:23.457723+0000","last_unstale":"2024-03-23T03:48:24.660626+0000","last_undegraded":"2024-03-23T03:48:24.660626+0000","last_fullsized":"2024-03-23T03:48:24.660626+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:51:52.504015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"40'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.163485+0000","last_change":"2024-03-23T03:48:22.969409+0000","last_active":"2024-03-23T03:48:25.163485+0000","last_peered":"2024-03-23T03:48:25.163485+0000","last_clean":"2024-03-23T03:48:25.163485+0000","last_became_active":"2024-03-23T03:48:22.969132+0000","last_became_peered":"2024-03-23T03:48:22.969132+0000","last_unstale":"2024-03-23T03:48:25.163485+0000","last_undegraded":"2024-03-23T03:48:25.163485+0000","last_fullsized":"2024-03-23T03:48:25.163485+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:57.542690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376897+0000","last_change":"2024-03-23T03:48:23.460179+0000","last_active":"2024-03-23T03:48:25.376897+0000","last_peered":"2024-03-23T03:48:25.376897+0000","last_clean":"2024-03-23T03:48:25.376897+0000","last_became_active":"2024-03-23T03:48:23.458501+0000","last_became_peered":"2024-03-23T03:48:23.458501+0000","last_unstale":"2024-03-23T03:48:25.376897+0000","last_undegraded":"2024-03-23T03:48:25.376897+0000","last_fullsized":"2024-03-23T03:48:25.376897+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:57.996508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.163834+0000","last_change":"2024-03-23T03:48:22.976947+0000","last_active":"2024-03-23T03:48:25.163834+0000","last_peered":"2024-03-23T03:48:25.163834+0000","last_clean":"2024-03-23T03:48:25.163834+0000","last_became_active":"2024-03-23T03:48:22.976687+0000","last_became_peered":"2024-03-23T03:48:22.976687+0000","last_unstale":"2024-03-23T03:48:25.163834+0000","last_undegraded":"2024-03-23T03:48:25.163834+0000","last_fullsized":"2024-03-23T03:48:25.163834+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:21:21.249313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.439894+0000","last_change":"2024-03-23T03:48:23.470473+0000","last_active":"2024-03-23T03:48:24.439894+0000","last_peered":"2024-03-23T03:48:24.439894+0000","last_clean":"2024-03-23T03:48:24.439894+0000","last_became_active":"2024-03-23T03:48:23.470148+0000","last_became_peered":"2024-03-23T03:48:23.470148+0000","last_unstale":"2024-03-23T03:48:24.439894+0000","last_undegraded":"2024-03-23T03:48:24.439894+0000","last_fullsized":"2024-03-23T03:48:24.439894+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:32:30.609829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661197+0000","last_change":"2024-03-23T03:48:22.988465+0000","last_active":"2024-03-23T03:48:24.661197+0000","last_peered":"2024-03-23T03:48:24.661197+0000","last_clean":"2024-03-23T03:48:24.661197+0000","last_became_active":"2024-03-23T03:48:22.988174+0000","last_became_peered":"2024-03-23T03:48:22.988174+0000","last_unstale":"2024-03-23T03:48:24.661197+0000","last_undegraded":"2024-03-23T03:48:24.661197+0000","last_fullsized":"2024-03-23T03:48:24.661197+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:04:20.350462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438251+0000","last_change":"2024-03-23T03:48:23.460122+0000","last_active":"2024-03-23T03:48:24.438251+0000","last_peered":"2024-03-23T03:48:24.438251+0000","last_clean":"2024-03-23T03:48:24.438251+0000","last_became_active":"2024-03-23T03:48:23.459860+0000","last_became_peered":"2024-03-23T03:48:23.459860+0000","last_unstale":"2024-03-23T03:48:24.438251+0000","last_undegraded":"2024-03-23T03:48:24.438251+0000","last_fullsized":"2024-03-23T03:48:24.438251+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:15:10.844824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.163683+0000","last_change":"2024-03-23T03:48:22.971252+0000","last_active":"2024-03-23T03:48:25.163683+0000","last_peered":"2024-03-23T03:48:25.163683+0000","last_clean":"2024-03-23T03:48:25.163683+0000","last_became_active":"2024-03-23T03:48:22.970961+0000","last_became_peered":"2024-03-23T03:48:22.970961+0000","last_unstale":"2024-03-23T03:48:25.163683+0000","last_undegraded":"2024-03-23T03:48:25.163683+0000","last_fullsized":"2024-03-23T03:48:25.163683+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:02:35.930419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376375+0000","last_change":"2024-03-23T03:48:23.471709+0000","last_active":"2024-03-23T03:48:25.376375+0000","last_peered":"2024-03-23T03:48:25.376375+0000","last_clean":"2024-03-23T03:48:25.376375+0000","last_became_active":"2024-03-23T03:48:23.471437+0000","last_became_peered":"2024-03-23T03:48:23.471437+0000","last_unstale":"2024-03-23T03:48:25.376375+0000","last_undegraded":"2024-03-23T03:48:25.376375+0000","last_fullsized":"2024-03-23T03:48:25.376375+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:46:31.492659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"40'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.441011+0000","last_change":"2024-03-23T03:48:22.986439+0000","last_active":"2024-03-23T03:48:24.441011+0000","last_peered":"2024-03-23T03:48:24.441011+0000","last_clean":"2024-03-23T03:48:24.441011+0000","last_became_active":"2024-03-23T03:48:22.986029+0000","last_became_peered":"2024-03-23T03:48:22.986029+0000","last_unstale":"2024-03-23T03:48:24.441011+0000","last_undegraded":"2024-03-23T03:48:24.441011+0000","last_fullsized":"2024-03-23T03:48:24.441011+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:40:21.057524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661581+0000","last_change":"2024-03-23T03:48:23.474303+0000","last_active":"2024-03-23T03:48:24.661581+0000","last_peered":"2024-03-23T03:48:24.661581+0000","last_clean":"2024-03-23T03:48:24.661581+0000","last_became_active":"2024-03-23T03:48:23.473966+0000","last_became_peered":"2024-03-23T03:48:23.473966+0000","last_unstale":"2024-03-23T03:48:24.661581+0000","last_undegraded":"2024-03-23T03:48:24.661581+0000","last_fullsized":"2024-03-23T03:48:24.661581+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:57:34.228967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"41'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.167496+0000","last_change":"2024-03-23T03:48:22.982756+0000","last_active":"2024-03-23T03:48:25.167496+0000","last_peered":"2024-03-23T03:48:25.167496+0000","last_clean":"2024-03-23T03:48:25.167496+0000","last_became_active":"2024-03-23T03:48:22.982122+0000","last_became_peered":"2024-03-23T03:48:22.982122+0000","last_unstale":"2024-03-23T03:48:25.167496+0000","last_undegraded":"2024-03-23T03:48:25.167496+0000","last_fullsized":"2024-03-23T03:48:25.167496+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:28.805507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438508+0000","last_change":"2024-03-23T03:48:23.475385+0000","last_active":"2024-03-23T03:48:24.438508+0000","last_peered":"2024-03-23T03:48:24.438508+0000","last_clean":"2024-03-23T03:48:24.438508+0000","last_became_active":"2024-03-23T03:48:23.475116+0000","last_became_peered":"2024-03-23T03:48:23.475116+0000","last_unstale":"2024-03-23T03:48:24.438508+0000","last_undegraded":"2024-03-23T03:48:24.438508+0000","last_fullsized":"2024-03-23T03:48:24.438508+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:40:04.714830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377525+0000","last_change":"2024-03-23T03:48:22.970156+0000","last_active":"2024-03-23T03:48:25.377525+0000","last_peered":"2024-03-23T03:48:25.377525+0000","last_clean":"2024-03-23T03:48:25.377525+0000","last_became_active":"2024-03-23T03:48:22.969896+0000","last_became_peered":"2024-03-23T03:48:22.969896+0000","last_unstale":"2024-03-23T03:48:25.377525+0000","last_undegraded":"2024-03-23T03:48:25.377525+0000","last_fullsized":"2024-03-23T03:48:25.377525+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:19:03.653303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376789+0000","last_change":"2024-03-23T03:48:23.450335+0000","last_active":"2024-03-23T03:48:25.376789+0000","last_peered":"2024-03-23T03:48:25.376789+0000","last_clean":"2024-03-23T03:48:25.376789+0000","last_became_active":"2024-03-23T03:48:23.450088+0000","last_became_peered":"2024-03-23T03:48:23.450088+0000","last_unstale":"2024-03-23T03:48:25.376789+0000","last_undegraded":"2024-03-23T03:48:25.376789+0000","last_fullsized":"2024-03-23T03:48:25.376789+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:01:27.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377208+0000","last_change":"2024-03-23T03:48:22.976577+0000","last_active":"2024-03-23T03:48:25.377208+0000","last_peered":"2024-03-23T03:48:25.377208+0000","last_clean":"2024-03-23T03:48:25.377208+0000","last_became_active":"2024-03-23T03:48:22.976313+0000","last_became_peered":"2024-03-23T03:48:22.976313+0000","last_unstale":"2024-03-23T03:48:25.377208+0000","last_undegraded":"2024-03-23T03:48:25.377208+0000","last_fullsized":"2024-03-23T03:48:25.377208+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:57:33.214487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376065+0000","last_change":"2024-03-23T03:48:23.462646+0000","last_active":"2024-03-23T03:48:25.376065+0000","last_peered":"2024-03-23T03:48:25.376065+0000","last_clean":"2024-03-23T03:48:25.376065+0000","last_became_active":"2024-03-23T03:48:23.462371+0000","last_became_peered":"2024-03-23T03:48:23.462371+0000","last_unstale":"2024-03-23T03:48:25.376065+0000","last_undegraded":"2024-03-23T03:48:25.376065+0000","last_fullsized":"2024-03-23T03:48:25.376065+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:20:23.375166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376537+0000","last_change":"2024-03-23T03:48:22.970375+0000","last_active":"2024-03-23T03:48:25.376537+0000","last_peered":"2024-03-23T03:48:25.376537+0000","last_clean":"2024-03-23T03:48:25.376537+0000","last_became_active":"2024-03-23T03:48:22.970132+0000","last_became_peered":"2024-03-23T03:48:22.970132+0000","last_unstale":"2024-03-23T03:48:25.376537+0000","last_undegraded":"2024-03-23T03:48:25.376537+0000","last_fullsized":"2024-03-23T03:48:25.376537+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:31:47.845785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377939+0000","last_change":"2024-03-23T03:48:23.478744+0000","last_active":"2024-03-23T03:48:25.377939+0000","last_peered":"2024-03-23T03:48:25.377939+0000","last_clean":"2024-03-23T03:48:25.377939+0000","last_became_active":"2024-03-23T03:48:23.474090+0000","last_became_peered":"2024-03-23T03:48:23.474090+0000","last_unstale":"2024-03-23T03:48:25.377939+0000","last_undegraded":"2024-03-23T03:48:25.377939+0000","last_fullsized":"2024-03-23T03:48:25.377939+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:44:35.418902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377971+0000","last_change":"2024-03-23T03:48:22.979722+0000","last_active":"2024-03-23T03:48:25.377971+0000","last_peered":"2024-03-23T03:48:25.377971+0000","last_clean":"2024-03-23T03:48:25.377971+0000","last_became_active":"2024-03-23T03:48:22.978953+0000","last_became_peered":"2024-03-23T03:48:22.978953+0000","last_unstale":"2024-03-23T03:48:25.377971+0000","last_undegraded":"2024-03-23T03:48:25.377971+0000","last_fullsized":"2024-03-23T03:48:25.377971+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:20:07.924853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661328+0000","last_change":"2024-03-23T03:48:23.465289+0000","last_active":"2024-03-23T03:48:24.661328+0000","last_peered":"2024-03-23T03:48:24.661328+0000","last_clean":"2024-03-23T03:48:24.661328+0000","last_became_active":"2024-03-23T03:48:23.464931+0000","last_became_peered":"2024-03-23T03:48:23.464931+0000","last_unstale":"2024-03-23T03:48:24.661328+0000","last_undegraded":"2024-03-23T03:48:24.661328+0000","last_fullsized":"2024-03-23T03:48:24.661328+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:32:25.374487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.437736+0000","last_change":"2024-03-23T03:48:22.971532+0000","last_active":"2024-03-23T03:48:24.437736+0000","last_peered":"2024-03-23T03:48:24.437736+0000","last_clean":"2024-03-23T03:48:24.437736+0000","last_became_active":"2024-03-23T03:48:22.971261+0000","last_became_peered":"2024-03-23T03:48:22.971261+0000","last_unstale":"2024-03-23T03:48:24.437736+0000","last_undegraded":"2024-03-23T03:48:24.437736+0000","last_fullsized":"2024-03-23T03:48:24.437736+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:08:54.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440733+0000","last_change":"2024-03-23T03:48:23.458842+0000","last_active":"2024-03-23T03:48:24.440733+0000","last_peered":"2024-03-23T03:48:24.440733+0000","last_clean":"2024-03-23T03:48:24.440733+0000","last_became_active":"2024-03-23T03:48:23.458009+0000","last_became_peered":"2024-03-23T03:48:23.458009+0000","last_unstale":"2024-03-23T03:48:24.440733+0000","last_undegraded":"2024-03-23T03:48:24.440733+0000","last_fullsized":"2024-03-23T03:48:24.440733+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:52:40.102104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661302+0000","last_change":"2024-03-23T03:48:22.990110+0000","last_active":"2024-03-23T03:48:24.661302+0000","last_peered":"2024-03-23T03:48:24.661302+0000","last_clean":"2024-03-23T03:48:24.661302+0000","last_became_active":"2024-03-23T03:48:22.989848+0000","last_became_peered":"2024-03-23T03:48:22.989848+0000","last_unstale":"2024-03-23T03:48:24.661302+0000","last_undegraded":"2024-03-23T03:48:24.661302+0000","last_fullsized":"2024-03-23T03:48:24.661302+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:14:45.272739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":179,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-23T03:48:23.445064+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:48:23.445064+0000","last_peered":"2024-03-23T03:48:23.445064+0000","last_clean":"2024-03-23T03:48:23.445064+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:48:23.445064+0000","last_undegraded":"2024-03-23T03:48:23.445064+0000","last_fullsized":"2024-03-23T03:48:23.445064+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542168,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343764,"statfs":{"total":96636764160,"available":96608014336,"internally_reserved":0,"allocated":1081344,"data_stored":793927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":1.091},{"interface":"front","average":{"1min":1.05,"5min":1.05,"15min":1.05},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.48}]},{"osd":1,"last update":"Sat Mar 23 03:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":4.9109999999999996,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":1.042}]},{"osd":2,"last update":"Sat Mar 23 03:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.165,"15min":1.165},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.9690000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":4.8310000000000004,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.421}]},{"osd":3,"last update":"Sat Mar 23 03:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"max":{"1min":4.8639999999999999,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"min":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"max":{"1min":2.1739999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.536}]},{"osd":4,"last update":"Sat Mar 23 03:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.22},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.5659999999999998,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.6890000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"min":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"max":{"1min":2.645,"5min":2.645,"15min":2.645},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"min":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"max":{"1min":2.5089999999999999,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.661}]},{"osd":6,"last update":"Sat Mar 23 03:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.577,"15min":1.577},"min":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"max":{"1min":2.7730000000000001,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.367},{"interface":"front","average":{"1min":1.536,"5min":1.536,"15min":1.536},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":2.7170000000000001,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":2.0030000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542166,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":475136,"data_stored":202507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:47:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.93899999999999995}]},{"osd":1,"last update":"Sat Mar 23 03:47:47 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":1.222},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.1080000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:47:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":1.05},{"interface":"front","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":1.137}]},{"osd":3,"last update":"Sat Mar 23 03:47:47 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.3580000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:47:47 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.3799999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.272}]},{"osd":5,"last update":"Sat Mar 23 03:47:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":2.4340000000000002,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.6349999999999998,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.4570000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:47:47 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.415,"15min":1.415},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.387},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.5870000000000002,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.494}]}]},{"osd":5,"up_from":14,"seq":60129542166,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27488,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344352,"statfs":{"total":96636764160,"available":96608616448,"internally_reserved":0,"allocated":479232,"data_stored":202037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:47:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.194,"5min":1.194,"15min":1.194},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.71599999999999997}]},{"osd":1,"last update":"Sat Mar 23 03:47:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.244}]},{"osd":2,"last update":"Sat Mar 23 03:47:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.9199999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.3240000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:47:46 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.071},{"interface":"front","average":{"1min":1.482,"5min":1.482,"15min":1.482},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.9969999999999999,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.484}]},{"osd":4,"last update":"Sat Mar 23 03:47:46 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"min":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.367}]},{"osd":6,"last update":"Sat Mar 23 03:47:46 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.4950000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"min":{"1min":1.101,"5min":1.101,"15min":1.101},"max":{"1min":2.6299999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.276}]},{"osd":7,"last update":"Sat Mar 23 03:47:46 2024","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"min":{"1min":1.143,"5min":1.143,"15min":1.143},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.294},{"interface":"front","average":{"1min":1.734,"5min":1.734,"15min":1.734},"min":{"1min":1.236,"5min":1.236,"15min":1.236},"max":{"1min":3.0640000000000001,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.429}]}]},{"osd":4,"up_from":14,"seq":60129542167,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27488,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344352,"statfs":{"total":96636764160,"available":96608616448,"internally_reserved":0,"allocated":479232,"data_stored":202529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.383,"5min":1.383,"15min":1.383},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.4950000000000001}]},{"osd":1,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.262},{"interface":"front","average":{"1min":1.052,"5min":1.052,"15min":1.052},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"last":1.3149999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":3.0859999999999999,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":1.002},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":1.5149999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.129,"15min":1.129},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.399},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":3.1749999999999998,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.1020000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.272,"15min":1.272},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.841,"5min":1.841,"15min":1.841},"last":1.454},{"interface":"front","average":{"1min":1.276,"5min":1.276,"15min":1.276},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.8639999999999999,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.5429999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":3.2570000000000001,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.353},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":1.5700000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":1.587},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.9420000000000002,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.4330000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542167,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27496,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344344,"statfs":{"total":96636764160,"available":96608608256,"internally_reserved":0,"allocated":487424,"data_stored":203523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":1.8149999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":5.8499999999999996}]},{"osd":1,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":3.1850000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":2.2530000000000001},"last":2.7450000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.8999999999999999,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":4.8010000000000002}]},{"osd":4,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":5.9210000000000003},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":4.819,"5min":4.819,"15min":4.819},"last":4.5620000000000003}]},{"osd":5,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.369,"15min":1.369},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":2.1349999999999998,"5min":2.1349999999999998,"15min":2.1349999999999998},"last":30.228000000000002},{"interface":"front","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":5.8920000000000003}]},{"osd":6,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"min":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"max":{"1min":4.7859999999999996,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.9319999999999999},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":4.7610000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":5.9589999999999996},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":4.7400000000000002,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":10.663}]}]},{"osd":2,"up_from":14,"seq":60129542167,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":475136,"data_stored":201995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":0.81000000000000005}]},{"osd":1,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":2.1899999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.1739999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.48199999999999998}]},{"osd":3,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.60799999999999998}]},{"osd":4,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.25,"15min":1.25},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.9119999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.88500000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":1.97,"5min":1.97,"15min":1.97},"last":1},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.1719999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":2.456,"5min":2.456,"15min":2.456},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":1.2390000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.78600000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542166,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":475136,"data_stored":202015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.2749999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":1.2509999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.625,"5min":1.625,"15min":1.625},"last":1.139}]},{"osd":3,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.2410000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.999}]},{"osd":4,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.323,"15min":1.323},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.1739999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.728}]},{"osd":5,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.357,"15min":1.357},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":3.593,"5min":3.593,"15min":3.593},"last":1.577},{"interface":"front","average":{"1min":1.476,"5min":1.476,"15min":1.476},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":1.0189999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.5389999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.387,"15min":1.387},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":3.7429999999999999,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.766}]},{"osd":7,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.47,"15min":1.47},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":3.8839999999999999,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.821},{"interface":"front","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":3.3079999999999998,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.357}]}]},{"osd":0,"up_from":14,"seq":60129542166,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343776,"statfs":{"total":96636764160,"available":96608026624,"internally_reserved":0,"allocated":1069056,"data_stored":792327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":1.244},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.76600000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:47:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.901,"5min":1.901,"15min":1.901},"last":1.3180000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:47:50 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.1659999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:47:50 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.361,"15min":1.361},"min":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":1.9650000000000001,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.294,"15min":1.294},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.087}]},{"osd":5,"last update":"Sat Mar 23 03:47:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.3799999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.29},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"min":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"max":{"1min":2.9449999999999998,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.3480000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:47:50 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.577,"15min":1.577},"min":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"max":{"1min":3.032,"5min":3.032,"15min":3.032},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":2.6339999999999999,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.2789999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:47:50 2024","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.448,"5min":1.448,"15min":1.448},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.335}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:48:28.065 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:48:28.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.312+0000 7ff38b400640 1 Processor -- start 2024-03-23T03:48:28.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.312+0000 7ff38b400640 1 -- start start 2024-03-23T03:48:28.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.314+0000 7ff38b400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ff38415cda0 0x7ff3840cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:28.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.314+0000 7ff38b400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff3840b4a60 0x7ff3840b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:28.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.314+0000 7ff38b400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3840d1830 con 0x7ff3840b5a30 2024-03-23T03:48:28.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.314+0000 7ff38b400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3840b1f20 con 0x7ff38415cda0 2024-03-23T03:48:28.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.314+0000 7ff38b400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3840d0c20 con 0x7ff3840b4a60 2024-03-23T03:48:28.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.314+0000 7ff383e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff3840b5a30 0x7ff3840b4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37820/0 (socket says 172.21.15.102:37820) 2024-03-23T03:48:28.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.314+0000 7ff383e00640 1 -- 172.21.15.102:0/1471047410 learned_addr learned my addr 172.21.15.102:0/1471047410 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:28.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.314+0000 7ff389400640 1 --2- 172.21.15.102:0/1471047410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff3840b4a60 0x7ff3840b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:28.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.315+0000 7ff388a00640 1 --2- 172.21.15.102:0/1471047410 >> v2:172.21.15.173:3300/0 conn(0x7ff38415cda0 0x7ff3840cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:28.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.315+0000 7ff383400640 1 -- 172.21.15.102:0/1471047410 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441791602 0 0) 0x7ff3840d1830 con 0x7ff3840b5a30 2024-03-23T03:48:28.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.315+0000 7ff383400640 1 -- 172.21.15.102:0/1471047410 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff368003520 con 0x7ff3840b5a30 2024-03-23T03:48:28.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.315+0000 7ff383400640 1 -- 172.21.15.102:0/1471047410 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2003413219 0 0) 0x7ff368003520 con 0x7ff3840b5a30 2024-03-23T03:48:28.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.315+0000 7ff383400640 1 -- 172.21.15.102:0/1471047410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff3840b4a60 msgr2=0x7ff3840b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:28.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.315+0000 7ff383400640 1 --2- 172.21.15.102:0/1471047410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff3840b4a60 0x7ff3840b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.315+0000 7ff383400640 1 -- 172.21.15.102:0/1471047410 >> v2:172.21.15.173:3300/0 conn(0x7ff38415cda0 msgr2=0x7ff3840cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:28.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.315+0000 7ff383400640 1 --2- 172.21.15.102:0/1471047410 >> v2:172.21.15.173:3300/0 conn(0x7ff38415cda0 0x7ff3840cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.315+0000 7ff383400640 1 -- 172.21.15.102:0/1471047410 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3840d0f00 con 0x7ff3840b5a30 2024-03-23T03:48:28.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.315+0000 7ff383400640 1 -- 172.21.15.102:0/1471047410 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7ff370001670 con 0x7ff3840b5a30 2024-03-23T03:48:28.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.315+0000 7ff383400640 1 -- 172.21.15.102:0/1471047410 >> v1:172.21.15.102:6789/0 conn(0x7ff3840b5a30 legacy=0x7ff3840b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:28.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.315+0000 7ff383400640 1 --2- 172.21.15.102:0/1471047410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff368003970 0x7ff368003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:28.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.315+0000 7ff383400640 1 --2- 172.21.15.102:0/1471047410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3680047c0 0x7ff368004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:28.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.316+0000 7ff383400640 1 --2- 172.21.15.102:0/1471047410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff3840b4a60 0x7ff368005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:28.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.316+0000 7ff383400640 1 -- 172.21.15.102:0/1471047410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3840b1f20 con 0x7ff368003970 2024-03-23T03:48:28.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.316+0000 7ff383400640 1 -- 172.21.15.102:0/1471047410 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3840d0c20 con 0x7ff3680047c0 2024-03-23T03:48:28.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.316+0000 7ff383400640 1 -- 172.21.15.102:0/1471047410 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff36800b980 con 0x7ff3840b4a60 2024-03-23T03:48:28.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.316+0000 7ff388a00640 1 --2- 172.21.15.102:0/1471047410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff368003970 0x7ff368003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:28.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.316+0000 7ff383e00640 1 --2- 172.21.15.102:0/1471047410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3680047c0 0x7ff368004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:28.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.316+0000 7ff389400640 1 --2- 172.21.15.102:0/1471047410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff3840b4a60 0x7ff368005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:28.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.316+0000 7ff388a00640 1 -- 172.21.15.102:0/1471047410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff3840b4a60 msgr2=0x7ff368005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:28.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.316+0000 7ff388a00640 1 --2- 172.21.15.102:0/1471047410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff3840b4a60 0x7ff368005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.316+0000 7ff388a00640 1 -- 172.21.15.102:0/1471047410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3680047c0 msgr2=0x7ff368004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:28.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.316+0000 7ff388a00640 1 --2- 172.21.15.102:0/1471047410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3680047c0 0x7ff368004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.316+0000 7ff388a00640 1 -- 172.21.15.102:0/1471047410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff368003520 con 0x7ff368003970 2024-03-23T03:48:28.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.318+0000 7ff388a00640 1 --2- 172.21.15.102:0/1471047410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff368003970 0x7ff368003d90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff374009fc0 tx=0x7ff3680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed90fd74dd9fe8a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:28.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.318+0000 7ff383400640 1 -- 172.21.15.102:0/1471047410 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37402cc60 con 0x7ff368003970 2024-03-23T03:48:28.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.318+0000 7ff383400640 1 -- 172.21.15.102:0/1471047410 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37400f040 con 0x7ff368003970 2024-03-23T03:48:28.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.318+0000 7ff383400640 1 -- 172.21.15.102:0/1471047410 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374034440 con 0x7ff368003970 2024-03-23T03:48:28.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.318+0000 7ff38b400640 1 -- 172.21.15.102:0/1471047410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff368003970 msgr2=0x7ff368003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:28.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.318+0000 7ff38b400640 1 --2- 172.21.15.102:0/1471047410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff368003970 0x7ff368003d90 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7ff374009fc0 tx=0x7ff3680057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.318+0000 7ff389400640 1 -- 172.21.15.102:0/1471047410 reap_dead start 2024-03-23T03:48:28.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.319+0000 7ff38b400640 1 -- 172.21.15.102:0/1471047410 shutdown_connections 2024-03-23T03:48:28.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.319+0000 7ff38b400640 1 -- 172.21.15.102:0/1471047410 >> 172.21.15.102:0/1471047410 conn(0x7ff3840cf350 msgr2=0x7ff3840cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:28.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.319+0000 7ff38b400640 1 -- 172.21.15.102:0/1471047410 shutdown_connections 2024-03-23T03:48:28.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.319+0000 7ff38b400640 1 -- 172.21.15.102:0/1471047410 wait complete. 2024-03-23T03:48:28.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.320+0000 7ff38b400640 1 Processor -- start 2024-03-23T03:48:28.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.321+0000 7ff38b400640 1 -- start start 2024-03-23T03:48:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.321+0000 7ff38b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff368003970 0x7ff384158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.321+0000 7ff38b400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff38415f980 0x7ff38415fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.321+0000 7ff38b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3840d2c10 0x7ff3840d3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.321+0000 7ff38b400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3841587c0 con 0x7ff368003970 2024-03-23T03:48:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.321+0000 7ff38b400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3840c7650 con 0x7ff3840d2c10 2024-03-23T03:48:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.321+0000 7ff38b400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3841f6640 con 0x7ff38415f980 2024-03-23T03:48:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.322+0000 7ff388a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff368003970 0x7ff384158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.322+0000 7ff388a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff368003970 0x7ff384158280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45000/0 (socket says 172.21.15.102:45000) 2024-03-23T03:48:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.322+0000 7ff388a00640 1 -- 172.21.15.102:0/2426065806 learned_addr learned my addr 172.21.15.102:0/2426065806 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.322+0000 7ff388a00640 1 -- 172.21.15.102:0/2426065806 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff38415f980 msgr2=0x7ff38415fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:28.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.322+0000 7ff383e00640 1 --2- 172.21.15.102:0/2426065806 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff38415f980 0x7ff38415fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:28.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.322+0000 7ff389400640 1 --2- 172.21.15.102:0/2426065806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3840d2c10 0x7ff3840d3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:28.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.322+0000 7ff388a00640 1 --2- 172.21.15.102:0/2426065806 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff38415f980 0x7ff38415fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.322+0000 7ff388a00640 1 -- 172.21.15.102:0/2426065806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3840d2c10 msgr2=0x7ff3840d3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:28.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.322+0000 7ff388a00640 1 --2- 172.21.15.102:0/2426065806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3840d2c10 0x7ff3840d3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.322+0000 7ff388a00640 1 -- 172.21.15.102:0/2426065806 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3840d35d0 con 0x7ff368003970 2024-03-23T03:48:28.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.323+0000 7ff388a00640 1 --2- 172.21.15.102:0/2426065806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff368003970 0x7ff384158280 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff37402c6d0 tx=0x7ff374009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:28.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.323+0000 7ff380c00640 1 -- 172.21.15.102:0/2426065806 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374034c50 con 0x7ff368003970 2024-03-23T03:48:28.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.323+0000 7ff380c00640 1 -- 172.21.15.102:0/2426065806 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff374034db0 con 0x7ff368003970 2024-03-23T03:48:28.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.323+0000 7ff38b400640 1 -- 172.21.15.102:0/2426065806 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3841f5ac0 con 0x7ff368003970 2024-03-23T03:48:28.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.323+0000 7ff380c00640 1 -- 172.21.15.102:0/2426065806 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374003b90 con 0x7ff368003970 2024-03-23T03:48:28.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.323+0000 7ff38b400640 1 -- 172.21.15.102:0/2426065806 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3841f6080 con 0x7ff368003970 2024-03-23T03:48:28.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.324+0000 7ff38b400640 1 -- 172.21.15.102:0/2426065806 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff384073390 con 0x7ff368003970 2024-03-23T03:48:28.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.325+0000 7ff380c00640 1 -- 172.21.15.102:0/2426065806 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff374040450 con 0x7ff368003970 2024-03-23T03:48:28.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.326+0000 7ff380c00640 1 --2- 172.21.15.102:0/2426065806 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff36406ac00 0x7ff36406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:28.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.326+0000 7ff380c00640 1 -- 172.21.15.102:0/2426065806 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff3740abbf0 con 0x7ff368003970 2024-03-23T03:48:28.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.326+0000 7ff383e00640 1 --2- 172.21.15.102:0/2426065806 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff36406ac00 0x7ff36406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:28.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.327+0000 7ff383e00640 1 --2- 172.21.15.102:0/2426065806 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff36406ac00 0x7ff36406d0c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff370000f60 tx=0x7ff370009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:28.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.330+0000 7ff380c00640 1 -- 172.21.15.102:0/2426065806 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff374076da0 con 0x7ff368003970 2024-03-23T03:48:28.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.523+0000 7ff38b400640 1 -- 172.21.15.102:0/2426065806 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff3840ce080 con 0x7ff36406ac00 2024-03-23T03:48:28.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.540+0000 7ff380c00640 1 -- 172.21.15.102:0/2426065806 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287260 (secure 0 0 0) 0x7ff37000d040 con 0x7ff36406ac00 2024-03-23T03:48:28.545 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:28.546 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:48:28.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.547+0000 7ff38b400640 1 -- 172.21.15.102:0/2426065806 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff36406ac00 msgr2=0x7ff36406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:28.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.547+0000 7ff38b400640 1 --2- 172.21.15.102:0/2426065806 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff36406ac00 0x7ff36406d0c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff370000f60 tx=0x7ff370009040 comp rx=0 tx=0).stop 2024-03-23T03:48:28.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.547+0000 7ff38b400640 1 -- 172.21.15.102:0/2426065806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff368003970 msgr2=0x7ff384158280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:28.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.547+0000 7ff38b400640 1 --2- 172.21.15.102:0/2426065806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff368003970 0x7ff384158280 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ff37402c6d0 tx=0x7ff374009750 comp rx=0 tx=0).stop 2024-03-23T03:48:28.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.547+0000 7ff38b400640 1 -- 172.21.15.102:0/2426065806 shutdown_connections 2024-03-23T03:48:28.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.547+0000 7ff38b400640 1 --2- 172.21.15.102:0/2426065806 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff36406ac00 0x7ff36406d0c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.547+0000 7ff38b400640 1 --2- 172.21.15.102:0/2426065806 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3840d2c10 0x7ff3840d3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.547+0000 7ff38b400640 1 --2- 172.21.15.102:0/2426065806 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff38415f980 0x7ff38415fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.547+0000 7ff38b400640 1 --2- 172.21.15.102:0/2426065806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff368003970 0x7ff384158280 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.547+0000 7ff38b400640 1 -- 172.21.15.102:0/2426065806 >> 172.21.15.102:0/2426065806 conn(0x7ff3840cf350 msgr2=0x7ff3840b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:28.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.547+0000 7ff38b400640 1 -- 172.21.15.102:0/2426065806 shutdown_connections 2024-03-23T03:48:28.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.547+0000 7ff38b400640 1 -- 172.21.15.102:0/2426065806 wait complete. 2024-03-23T03:48:28.567 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2024-03-23T03:48:27.745941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":189,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221072,"kb_used_data":4904,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753648,"statfs":{"total":773094113280,"available":772867735552,"internally_reserved":0,"allocated":5021696,"data_stored":2800860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":221236794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.810117"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438800+0000","last_change":"2024-03-23T03:48:23.446505+0000","last_active":"2024-03-23T03:48:24.438800+0000","last_peered":"2024-03-23T03:48:24.438800+0000","last_clean":"2024-03-23T03:48:24.438800+0000","last_became_active":"2024-03-23T03:48:23.446196+0000","last_became_peered":"2024-03-23T03:48:23.446196+0000","last_unstale":"2024-03-23T03:48:24.438800+0000","last_undegraded":"2024-03-23T03:48:24.438800+0000","last_fullsized":"2024-03-23T03:48:24.438800+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:50:18.596428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438474+0000","last_change":"2024-03-23T03:48:23.441976+0000","last_active":"2024-03-23T03:48:24.438474+0000","last_peered":"2024-03-23T03:48:24.438474+0000","last_clean":"2024-03-23T03:48:24.438474+0000","last_became_active":"2024-03-23T03:48:23.441710+0000","last_became_peered":"2024-03-23T03:48:23.441710+0000","last_unstale":"2024-03-23T03:48:24.438474+0000","last_undegraded":"2024-03-23T03:48:24.438474+0000","last_fullsized":"2024-03-23T03:48:24.438474+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:54:29.194785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377464+0000","last_change":"2024-03-23T03:48:23.464788+0000","last_active":"2024-03-23T03:48:25.377464+0000","last_peered":"2024-03-23T03:48:25.377464+0000","last_clean":"2024-03-23T03:48:25.377464+0000","last_became_active":"2024-03-23T03:48:23.464505+0000","last_became_peered":"2024-03-23T03:48:23.464505+0000","last_unstale":"2024-03-23T03:48:25.377464+0000","last_undegraded":"2024-03-23T03:48:25.377464+0000","last_fullsized":"2024-03-23T03:48:25.377464+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:31:51.225846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.164253+0000","last_change":"2024-03-23T03:48:23.478086+0000","last_active":"2024-03-23T03:48:25.164253+0000","last_peered":"2024-03-23T03:48:25.164253+0000","last_clean":"2024-03-23T03:48:25.164253+0000","last_became_active":"2024-03-23T03:48:23.477848+0000","last_became_peered":"2024-03-23T03:48:23.477848+0000","last_unstale":"2024-03-23T03:48:25.164253+0000","last_undegraded":"2024-03-23T03:48:25.164253+0000","last_fullsized":"2024-03-23T03:48:25.164253+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:48:47.462755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.163400+0000","last_change":"2024-03-23T03:48:23.474665+0000","last_active":"2024-03-23T03:48:25.163400+0000","last_peered":"2024-03-23T03:48:25.163400+0000","last_clean":"2024-03-23T03:48:25.163400+0000","last_became_active":"2024-03-23T03:48:23.474393+0000","last_became_peered":"2024-03-23T03:48:23.474393+0000","last_unstale":"2024-03-23T03:48:25.163400+0000","last_undegraded":"2024-03-23T03:48:25.163400+0000","last_fullsized":"2024-03-23T03:48:25.163400+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:14:11.643509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376014+0000","last_change":"2024-03-23T03:48:23.451143+0000","last_active":"2024-03-23T03:48:25.376014+0000","last_peered":"2024-03-23T03:48:25.376014+0000","last_clean":"2024-03-23T03:48:25.376014+0000","last_became_active":"2024-03-23T03:48:23.449891+0000","last_became_peered":"2024-03-23T03:48:23.449891+0000","last_unstale":"2024-03-23T03:48:25.376014+0000","last_undegraded":"2024-03-23T03:48:25.376014+0000","last_fullsized":"2024-03-23T03:48:25.376014+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:47.834179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.439436+0000","last_change":"2024-03-23T03:48:23.455088+0000","last_active":"2024-03-23T03:48:24.439436+0000","last_peered":"2024-03-23T03:48:24.439436+0000","last_clean":"2024-03-23T03:48:24.439436+0000","last_became_active":"2024-03-23T03:48:23.454790+0000","last_became_peered":"2024-03-23T03:48:23.454790+0000","last_unstale":"2024-03-23T03:48:24.439436+0000","last_undegraded":"2024-03-23T03:48:24.439436+0000","last_fullsized":"2024-03-23T03:48:24.439436+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:51:59.059621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.437672+0000","last_change":"2024-03-23T03:48:23.443514+0000","last_active":"2024-03-23T03:48:24.437672+0000","last_peered":"2024-03-23T03:48:24.437672+0000","last_clean":"2024-03-23T03:48:24.437672+0000","last_became_active":"2024-03-23T03:48:23.443267+0000","last_became_peered":"2024-03-23T03:48:23.443267+0000","last_unstale":"2024-03-23T03:48:24.437672+0000","last_undegraded":"2024-03-23T03:48:24.437672+0000","last_fullsized":"2024-03-23T03:48:24.437672+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:08:40.284453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377924+0000","last_change":"2024-03-23T03:48:23.464811+0000","last_active":"2024-03-23T03:48:25.377924+0000","last_peered":"2024-03-23T03:48:25.377924+0000","last_clean":"2024-03-23T03:48:25.377924+0000","last_became_active":"2024-03-23T03:48:23.464562+0000","last_became_peered":"2024-03-23T03:48:23.464562+0000","last_unstale":"2024-03-23T03:48:25.377924+0000","last_undegraded":"2024-03-23T03:48:25.377924+0000","last_fullsized":"2024-03-23T03:48:25.377924+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:51:59.059621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440737+0000","last_change":"2024-03-23T03:48:23.467159+0000","last_active":"2024-03-23T03:48:24.440737+0000","last_peered":"2024-03-23T03:48:24.440737+0000","last_clean":"2024-03-23T03:48:24.440737+0000","last_became_active":"2024-03-23T03:48:23.466869+0000","last_became_peered":"2024-03-23T03:48:23.466869+0000","last_unstale":"2024-03-23T03:48:24.440737+0000","last_undegraded":"2024-03-23T03:48:24.440737+0000","last_fullsized":"2024-03-23T03:48:24.440737+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:53.495024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440388+0000","last_change":"2024-03-23T03:48:23.466262+0000","last_active":"2024-03-23T03:48:24.440388+0000","last_peered":"2024-03-23T03:48:24.440388+0000","last_clean":"2024-03-23T03:48:24.440388+0000","last_became_active":"2024-03-23T03:48:23.466000+0000","last_became_peered":"2024-03-23T03:48:23.466000+0000","last_unstale":"2024-03-23T03:48:24.440388+0000","last_undegraded":"2024-03-23T03:48:24.440388+0000","last_fullsized":"2024-03-23T03:48:24.440388+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:38:59.566199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377452+0000","last_change":"2024-03-23T03:48:23.445532+0000","last_active":"2024-03-23T03:48:25.377452+0000","last_peered":"2024-03-23T03:48:25.377452+0000","last_clean":"2024-03-23T03:48:25.377452+0000","last_became_active":"2024-03-23T03:48:23.445352+0000","last_became_peered":"2024-03-23T03:48:23.445352+0000","last_unstale":"2024-03-23T03:48:25.377452+0000","last_undegraded":"2024-03-23T03:48:25.377452+0000","last_fullsized":"2024-03-23T03:48:25.377452+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:38:59.566199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.439862+0000","last_change":"2024-03-23T03:48:23.467235+0000","last_active":"2024-03-23T03:48:24.439862+0000","last_peered":"2024-03-23T03:48:24.439862+0000","last_clean":"2024-03-23T03:48:24.439862+0000","last_became_active":"2024-03-23T03:48:23.466876+0000","last_became_peered":"2024-03-23T03:48:23.466876+0000","last_unstale":"2024-03-23T03:48:24.439862+0000","last_undegraded":"2024-03-23T03:48:24.439862+0000","last_fullsized":"2024-03-23T03:48:24.439862+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:10:35.688861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377850+0000","last_change":"2024-03-23T03:48:23.479234+0000","last_active":"2024-03-23T03:48:25.377850+0000","last_peered":"2024-03-23T03:48:25.377850+0000","last_clean":"2024-03-23T03:48:25.377850+0000","last_became_active":"2024-03-23T03:48:23.477566+0000","last_became_peered":"2024-03-23T03:48:23.477566+0000","last_unstale":"2024-03-23T03:48:25.377850+0000","last_undegraded":"2024-03-23T03:48:25.377850+0000","last_fullsized":"2024-03-23T03:48:25.377850+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:10:24.611081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.439597+0000","last_change":"2024-03-23T03:48:23.445888+0000","last_active":"2024-03-23T03:48:24.439597+0000","last_peered":"2024-03-23T03:48:24.439597+0000","last_clean":"2024-03-23T03:48:24.439597+0000","last_became_active":"2024-03-23T03:48:23.443616+0000","last_became_peered":"2024-03-23T03:48:23.443616+0000","last_unstale":"2024-03-23T03:48:24.439597+0000","last_undegraded":"2024-03-23T03:48:24.439597+0000","last_fullsized":"2024-03-23T03:48:24.439597+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:07:31.748044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.660364+0000","last_change":"2024-03-23T03:48:23.470703+0000","last_active":"2024-03-23T03:48:24.660364+0000","last_peered":"2024-03-23T03:48:24.660364+0000","last_clean":"2024-03-23T03:48:24.660364+0000","last_became_active":"2024-03-23T03:48:23.470405+0000","last_became_peered":"2024-03-23T03:48:23.470405+0000","last_unstale":"2024-03-23T03:48:24.660364+0000","last_undegraded":"2024-03-23T03:48:24.660364+0000","last_fullsized":"2024-03-23T03:48:24.660364+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:06:21.756726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661647+0000","last_change":"2024-03-23T03:48:23.471774+0000","last_active":"2024-03-23T03:48:24.661647+0000","last_peered":"2024-03-23T03:48:24.661647+0000","last_clean":"2024-03-23T03:48:24.661647+0000","last_became_active":"2024-03-23T03:48:23.470743+0000","last_became_peered":"2024-03-23T03:48:23.470743+0000","last_unstale":"2024-03-23T03:48:24.661647+0000","last_undegraded":"2024-03-23T03:48:24.661647+0000","last_fullsized":"2024-03-23T03:48:24.661647+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:20:23.375166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377019+0000","last_change":"2024-03-23T03:48:23.476885+0000","last_active":"2024-03-23T03:48:25.377019+0000","last_peered":"2024-03-23T03:48:25.377019+0000","last_clean":"2024-03-23T03:48:25.377019+0000","last_became_active":"2024-03-23T03:48:23.476467+0000","last_became_peered":"2024-03-23T03:48:23.476467+0000","last_unstale":"2024-03-23T03:48:25.377019+0000","last_undegraded":"2024-03-23T03:48:25.377019+0000","last_fullsized":"2024-03-23T03:48:25.377019+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:01:01.145513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.163915+0000","last_change":"2024-03-23T03:48:23.448656+0000","last_active":"2024-03-23T03:48:25.163915+0000","last_peered":"2024-03-23T03:48:25.163915+0000","last_clean":"2024-03-23T03:48:25.163915+0000","last_became_active":"2024-03-23T03:48:23.448387+0000","last_became_peered":"2024-03-23T03:48:23.448387+0000","last_unstale":"2024-03-23T03:48:25.163915+0000","last_undegraded":"2024-03-23T03:48:25.163915+0000","last_fullsized":"2024-03-23T03:48:25.163915+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:08:00.804604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376605+0000","last_change":"2024-03-23T03:48:23.476883+0000","last_active":"2024-03-23T03:48:25.376605+0000","last_peered":"2024-03-23T03:48:25.376605+0000","last_clean":"2024-03-23T03:48:25.376605+0000","last_became_active":"2024-03-23T03:48:23.476549+0000","last_became_peered":"2024-03-23T03:48:23.476549+0000","last_unstale":"2024-03-23T03:48:25.376605+0000","last_undegraded":"2024-03-23T03:48:25.376605+0000","last_fullsized":"2024-03-23T03:48:25.376605+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:26:46.940664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440215+0000","last_change":"2024-03-23T03:48:23.461082+0000","last_active":"2024-03-23T03:48:24.440215+0000","last_peered":"2024-03-23T03:48:24.440215+0000","last_clean":"2024-03-23T03:48:24.440215+0000","last_became_active":"2024-03-23T03:48:23.459352+0000","last_became_peered":"2024-03-23T03:48:23.459352+0000","last_unstale":"2024-03-23T03:48:24.440215+0000","last_undegraded":"2024-03-23T03:48:24.440215+0000","last_fullsized":"2024-03-23T03:48:24.440215+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:31:51.225846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.378320+0000","last_change":"2024-03-23T03:48:23.463523+0000","last_active":"2024-03-23T03:48:25.378320+0000","last_peered":"2024-03-23T03:48:25.378320+0000","last_clean":"2024-03-23T03:48:25.378320+0000","last_became_active":"2024-03-23T03:48:23.463139+0000","last_became_peered":"2024-03-23T03:48:23.463139+0000","last_unstale":"2024-03-23T03:48:25.378320+0000","last_undegraded":"2024-03-23T03:48:25.378320+0000","last_fullsized":"2024-03-23T03:48:25.378320+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:13.565417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.660949+0000","last_change":"2024-03-23T03:48:23.457871+0000","last_active":"2024-03-23T03:48:24.660949+0000","last_peered":"2024-03-23T03:48:24.660949+0000","last_clean":"2024-03-23T03:48:24.660949+0000","last_became_active":"2024-03-23T03:48:23.457583+0000","last_became_peered":"2024-03-23T03:48:23.457583+0000","last_unstale":"2024-03-23T03:48:24.660949+0000","last_undegraded":"2024-03-23T03:48:24.660949+0000","last_fullsized":"2024-03-23T03:48:24.660949+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:33:05.984877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.163785+0000","last_change":"2024-03-23T03:48:23.459008+0000","last_active":"2024-03-23T03:48:25.163785+0000","last_peered":"2024-03-23T03:48:25.163785+0000","last_clean":"2024-03-23T03:48:25.163785+0000","last_became_active":"2024-03-23T03:48:23.458563+0000","last_became_peered":"2024-03-23T03:48:23.458563+0000","last_unstale":"2024-03-23T03:48:25.163785+0000","last_undegraded":"2024-03-23T03:48:25.163785+0000","last_fullsized":"2024-03-23T03:48:25.163785+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:48:17.163517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376973+0000","last_change":"2024-03-23T03:48:23.464393+0000","last_active":"2024-03-23T03:48:25.376973+0000","last_peered":"2024-03-23T03:48:25.376973+0000","last_clean":"2024-03-23T03:48:25.376973+0000","last_became_active":"2024-03-23T03:48:23.463855+0000","last_became_peered":"2024-03-23T03:48:23.463855+0000","last_unstale":"2024-03-23T03:48:25.376973+0000","last_undegraded":"2024-03-23T03:48:25.376973+0000","last_fullsized":"2024-03-23T03:48:25.376973+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:20:08.939333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376610+0000","last_change":"2024-03-23T03:48:23.470934+0000","last_active":"2024-03-23T03:48:25.376610+0000","last_peered":"2024-03-23T03:48:25.376610+0000","last_clean":"2024-03-23T03:48:25.376610+0000","last_became_active":"2024-03-23T03:48:23.470661+0000","last_became_peered":"2024-03-23T03:48:23.470661+0000","last_unstale":"2024-03-23T03:48:25.376610+0000","last_undegraded":"2024-03-23T03:48:25.376610+0000","last_fullsized":"2024-03-23T03:48:25.376610+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:01:27.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376860+0000","last_change":"2024-03-23T03:48:23.462765+0000","last_active":"2024-03-23T03:48:25.376860+0000","last_peered":"2024-03-23T03:48:25.376860+0000","last_clean":"2024-03-23T03:48:25.376860+0000","last_became_active":"2024-03-23T03:48:23.462428+0000","last_became_peered":"2024-03-23T03:48:23.462428+0000","last_unstale":"2024-03-23T03:48:25.376860+0000","last_undegraded":"2024-03-23T03:48:25.376860+0000","last_fullsized":"2024-03-23T03:48:25.376860+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:24:30.974122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440204+0000","last_change":"2024-03-23T03:48:23.444152+0000","last_active":"2024-03-23T03:48:24.440204+0000","last_peered":"2024-03-23T03:48:24.440204+0000","last_clean":"2024-03-23T03:48:24.440204+0000","last_became_active":"2024-03-23T03:48:23.441122+0000","last_became_peered":"2024-03-23T03:48:23.441122+0000","last_unstale":"2024-03-23T03:48:24.440204+0000","last_undegraded":"2024-03-23T03:48:24.440204+0000","last_fullsized":"2024-03-23T03:48:24.440204+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:01:27.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376520+0000","last_change":"2024-03-23T03:48:23.461956+0000","last_active":"2024-03-23T03:48:25.376520+0000","last_peered":"2024-03-23T03:48:25.376520+0000","last_clean":"2024-03-23T03:48:25.376520+0000","last_became_active":"2024-03-23T03:48:23.460093+0000","last_became_peered":"2024-03-23T03:48:23.460093+0000","last_unstale":"2024-03-23T03:48:25.376520+0000","last_undegraded":"2024-03-23T03:48:25.376520+0000","last_fullsized":"2024-03-23T03:48:25.376520+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:48:47.462755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376542+0000","last_change":"2024-03-23T03:48:23.456528+0000","last_active":"2024-03-23T03:48:25.376542+0000","last_peered":"2024-03-23T03:48:25.376542+0000","last_clean":"2024-03-23T03:48:25.376542+0000","last_became_active":"2024-03-23T03:48:23.456155+0000","last_became_peered":"2024-03-23T03:48:23.456155+0000","last_unstale":"2024-03-23T03:48:25.376542+0000","last_undegraded":"2024-03-23T03:48:25.376542+0000","last_fullsized":"2024-03-23T03:48:25.376542+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:14:11.643509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661437+0000","last_change":"2024-03-23T03:48:23.458360+0000","last_active":"2024-03-23T03:48:24.661437+0000","last_peered":"2024-03-23T03:48:24.661437+0000","last_clean":"2024-03-23T03:48:24.661437+0000","last_became_active":"2024-03-23T03:48:23.458065+0000","last_became_peered":"2024-03-23T03:48:23.458065+0000","last_unstale":"2024-03-23T03:48:24.661437+0000","last_undegraded":"2024-03-23T03:48:24.661437+0000","last_fullsized":"2024-03-23T03:48:24.661437+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:45:45.295605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.375746+0000","last_change":"2024-03-23T03:48:23.443932+0000","last_active":"2024-03-23T03:48:25.375746+0000","last_peered":"2024-03-23T03:48:25.375746+0000","last_clean":"2024-03-23T03:48:25.375746+0000","last_became_active":"2024-03-23T03:48:23.442920+0000","last_became_peered":"2024-03-23T03:48:23.442920+0000","last_unstale":"2024-03-23T03:48:25.375746+0000","last_undegraded":"2024-03-23T03:48:25.375746+0000","last_fullsized":"2024-03-23T03:48:25.375746+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:06:21.756726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661136+0000","last_change":"2024-03-23T03:48:23.470905+0000","last_active":"2024-03-23T03:48:24.661136+0000","last_peered":"2024-03-23T03:48:24.661136+0000","last_clean":"2024-03-23T03:48:24.661136+0000","last_became_active":"2024-03-23T03:48:23.470608+0000","last_became_peered":"2024-03-23T03:48:23.470608+0000","last_unstale":"2024-03-23T03:48:24.661136+0000","last_undegraded":"2024-03-23T03:48:24.661136+0000","last_fullsized":"2024-03-23T03:48:24.661136+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:32:47.834179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"40'4","reported_seq":15,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376976+0000","last_change":"2024-03-23T03:48:22.972325+0000","last_active":"2024-03-23T03:48:25.376976+0000","last_peered":"2024-03-23T03:48:25.376976+0000","last_clean":"2024-03-23T03:48:25.376976+0000","last_became_active":"2024-03-23T03:48:22.972050+0000","last_became_peered":"2024-03-23T03:48:22.972050+0000","last_unstale":"2024-03-23T03:48:25.376976+0000","last_undegraded":"2024-03-23T03:48:25.376976+0000","last_fullsized":"2024-03-23T03:48:25.376976+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:27:38.886883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377344+0000","last_change":"2024-03-23T03:48:23.470963+0000","last_active":"2024-03-23T03:48:25.377344+0000","last_peered":"2024-03-23T03:48:25.377344+0000","last_clean":"2024-03-23T03:48:25.377344+0000","last_became_active":"2024-03-23T03:48:23.470717+0000","last_became_peered":"2024-03-23T03:48:23.470717+0000","last_unstale":"2024-03-23T03:48:25.377344+0000","last_undegraded":"2024-03-23T03:48:25.377344+0000","last_fullsized":"2024-03-23T03:48:25.377344+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:52:40.102104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"40'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377732+0000","last_change":"2024-03-23T03:48:22.980772+0000","last_active":"2024-03-23T03:48:25.377732+0000","last_peered":"2024-03-23T03:48:25.377732+0000","last_clean":"2024-03-23T03:48:25.377732+0000","last_became_active":"2024-03-23T03:48:22.980177+0000","last_became_peered":"2024-03-23T03:48:22.980177+0000","last_unstale":"2024-03-23T03:48:25.377732+0000","last_undegraded":"2024-03-23T03:48:25.377732+0000","last_fullsized":"2024-03-23T03:48:25.377732+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:33:04.970397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.378787+0000","last_change":"2024-03-23T03:48:23.475651+0000","last_active":"2024-03-23T03:48:25.378787+0000","last_peered":"2024-03-23T03:48:25.378787+0000","last_clean":"2024-03-23T03:48:25.378787+0000","last_became_active":"2024-03-23T03:48:23.475209+0000","last_became_peered":"2024-03-23T03:48:23.475209+0000","last_unstale":"2024-03-23T03:48:25.378787+0000","last_undegraded":"2024-03-23T03:48:25.378787+0000","last_fullsized":"2024-03-23T03:48:25.378787+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:39:51.342220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"40'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377566+0000","last_change":"2024-03-23T03:48:22.984879+0000","last_active":"2024-03-23T03:48:25.377566+0000","last_peered":"2024-03-23T03:48:25.377566+0000","last_clean":"2024-03-23T03:48:25.377566+0000","last_became_active":"2024-03-23T03:48:22.984651+0000","last_became_peered":"2024-03-23T03:48:22.984651+0000","last_unstale":"2024-03-23T03:48:25.377566+0000","last_undegraded":"2024-03-23T03:48:25.377566+0000","last_fullsized":"2024-03-23T03:48:25.377566+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:40:21.057524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438637+0000","last_change":"2024-03-23T03:48:23.464765+0000","last_active":"2024-03-23T03:48:24.438637+0000","last_peered":"2024-03-23T03:48:24.438637+0000","last_clean":"2024-03-23T03:48:24.438637+0000","last_became_active":"2024-03-23T03:48:23.464335+0000","last_became_peered":"2024-03-23T03:48:23.464335+0000","last_unstale":"2024-03-23T03:48:24.438637+0000","last_undegraded":"2024-03-23T03:48:24.438637+0000","last_fullsized":"2024-03-23T03:48:24.438637+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:36:23.713289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"40'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.378712+0000","last_change":"2024-03-23T03:48:22.980265+0000","last_active":"2024-03-23T03:48:25.378712+0000","last_peered":"2024-03-23T03:48:25.378712+0000","last_clean":"2024-03-23T03:48:25.378712+0000","last_became_active":"2024-03-23T03:48:22.979981+0000","last_became_peered":"2024-03-23T03:48:22.979981+0000","last_unstale":"2024-03-23T03:48:25.378712+0000","last_undegraded":"2024-03-23T03:48:25.378712+0000","last_fullsized":"2024-03-23T03:48:25.378712+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:01:00.131033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376275+0000","last_change":"2024-03-23T03:48:23.447954+0000","last_active":"2024-03-23T03:48:25.376275+0000","last_peered":"2024-03-23T03:48:25.376275+0000","last_clean":"2024-03-23T03:48:25.376275+0000","last_became_active":"2024-03-23T03:48:23.447696+0000","last_became_peered":"2024-03-23T03:48:23.447696+0000","last_unstale":"2024-03-23T03:48:25.376275+0000","last_undegraded":"2024-03-23T03:48:25.376275+0000","last_fullsized":"2024-03-23T03:48:25.376275+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:53.495024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"40'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440460+0000","last_change":"2024-03-23T03:48:22.979758+0000","last_active":"2024-03-23T03:48:24.440460+0000","last_peered":"2024-03-23T03:48:24.440460+0000","last_clean":"2024-03-23T03:48:24.440460+0000","last_became_active":"2024-03-23T03:48:22.979478+0000","last_became_peered":"2024-03-23T03:48:22.979478+0000","last_unstale":"2024-03-23T03:48:24.440460+0000","last_undegraded":"2024-03-23T03:48:24.440460+0000","last_fullsized":"2024-03-23T03:48:24.440460+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:32:24.360007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.439715+0000","last_change":"2024-03-23T03:48:23.467403+0000","last_active":"2024-03-23T03:48:24.439715+0000","last_peered":"2024-03-23T03:48:24.439715+0000","last_clean":"2024-03-23T03:48:24.439715+0000","last_became_active":"2024-03-23T03:48:23.466694+0000","last_became_peered":"2024-03-23T03:48:23.466694+0000","last_unstale":"2024-03-23T03:48:24.439715+0000","last_undegraded":"2024-03-23T03:48:24.439715+0000","last_fullsized":"2024-03-23T03:48:24.439715+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:26:23.251559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"40'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661186+0000","last_change":"2024-03-23T03:48:22.989913+0000","last_active":"2024-03-23T03:48:24.661186+0000","last_peered":"2024-03-23T03:48:24.661186+0000","last_clean":"2024-03-23T03:48:24.661186+0000","last_became_active":"2024-03-23T03:48:22.989594+0000","last_became_peered":"2024-03-23T03:48:22.989594+0000","last_unstale":"2024-03-23T03:48:24.661186+0000","last_undegraded":"2024-03-23T03:48:24.661186+0000","last_fullsized":"2024-03-23T03:48:24.661186+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:35:43.667112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.378481+0000","last_change":"2024-03-23T03:48:23.478954+0000","last_active":"2024-03-23T03:48:25.378481+0000","last_peered":"2024-03-23T03:48:25.378481+0000","last_clean":"2024-03-23T03:48:25.378481+0000","last_became_active":"2024-03-23T03:48:23.476063+0000","last_became_peered":"2024-03-23T03:48:23.476063+0000","last_unstale":"2024-03-23T03:48:25.378481+0000","last_undegraded":"2024-03-23T03:48:25.378481+0000","last_fullsized":"2024-03-23T03:48:25.378481+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:23:17.267833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376082+0000","last_change":"2024-03-23T03:48:22.985851+0000","last_active":"2024-03-23T03:48:25.376082+0000","last_peered":"2024-03-23T03:48:25.376082+0000","last_clean":"2024-03-23T03:48:25.376082+0000","last_became_active":"2024-03-23T03:48:22.985611+0000","last_became_peered":"2024-03-23T03:48:22.985611+0000","last_unstale":"2024-03-23T03:48:25.376082+0000","last_undegraded":"2024-03-23T03:48:25.376082+0000","last_fullsized":"2024-03-23T03:48:25.376082+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:17:17.358011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438071+0000","last_change":"2024-03-23T03:48:23.457581+0000","last_active":"2024-03-23T03:48:24.438071+0000","last_peered":"2024-03-23T03:48:24.438071+0000","last_clean":"2024-03-23T03:48:24.438071+0000","last_became_active":"2024-03-23T03:48:23.457341+0000","last_became_peered":"2024-03-23T03:48:23.457341+0000","last_unstale":"2024-03-23T03:48:24.438071+0000","last_undegraded":"2024-03-23T03:48:24.438071+0000","last_fullsized":"2024-03-23T03:48:24.438071+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:12:31.350656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377837+0000","last_change":"2024-03-23T03:48:22.977569+0000","last_active":"2024-03-23T03:48:25.377837+0000","last_peered":"2024-03-23T03:48:25.377837+0000","last_clean":"2024-03-23T03:48:25.377837+0000","last_became_active":"2024-03-23T03:48:22.977095+0000","last_became_peered":"2024-03-23T03:48:22.977095+0000","last_unstale":"2024-03-23T03:48:25.377837+0000","last_undegraded":"2024-03-23T03:48:25.377837+0000","last_fullsized":"2024-03-23T03:48:25.377837+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:51:29.105968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.164191+0000","last_change":"2024-03-23T03:48:23.478104+0000","last_active":"2024-03-23T03:48:25.164191+0000","last_peered":"2024-03-23T03:48:25.164191+0000","last_clean":"2024-03-23T03:48:25.164191+0000","last_became_active":"2024-03-23T03:48:23.477839+0000","last_became_peered":"2024-03-23T03:48:23.477839+0000","last_unstale":"2024-03-23T03:48:25.164191+0000","last_undegraded":"2024-03-23T03:48:25.164191+0000","last_fullsized":"2024-03-23T03:48:25.164191+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:24:30.974122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438914+0000","last_change":"2024-03-23T03:48:22.972812+0000","last_active":"2024-03-23T03:48:24.438914+0000","last_peered":"2024-03-23T03:48:24.438914+0000","last_clean":"2024-03-23T03:48:24.438914+0000","last_became_active":"2024-03-23T03:48:22.972293+0000","last_became_peered":"2024-03-23T03:48:22.972293+0000","last_unstale":"2024-03-23T03:48:24.438914+0000","last_undegraded":"2024-03-23T03:48:24.438914+0000","last_fullsized":"2024-03-23T03:48:24.438914+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:24:37.457522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661469+0000","last_change":"2024-03-23T03:48:23.474240+0000","last_active":"2024-03-23T03:48:24.661469+0000","last_peered":"2024-03-23T03:48:24.661469+0000","last_clean":"2024-03-23T03:48:24.661469+0000","last_became_active":"2024-03-23T03:48:23.473951+0000","last_became_peered":"2024-03-23T03:48:23.473951+0000","last_unstale":"2024-03-23T03:48:24.661469+0000","last_undegraded":"2024-03-23T03:48:24.661469+0000","last_fullsized":"2024-03-23T03:48:24.661469+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:19:13.754297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440836+0000","last_change":"2024-03-23T03:48:22.982207+0000","last_active":"2024-03-23T03:48:24.440836+0000","last_peered":"2024-03-23T03:48:24.440836+0000","last_clean":"2024-03-23T03:48:24.440836+0000","last_became_active":"2024-03-23T03:48:22.981866+0000","last_became_peered":"2024-03-23T03:48:22.981866+0000","last_unstale":"2024-03-23T03:48:24.440836+0000","last_undegraded":"2024-03-23T03:48:24.440836+0000","last_fullsized":"2024-03-23T03:48:24.440836+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:44:48.293049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377728+0000","last_change":"2024-03-23T03:48:23.470131+0000","last_active":"2024-03-23T03:48:25.377728+0000","last_peered":"2024-03-23T03:48:25.377728+0000","last_clean":"2024-03-23T03:48:25.377728+0000","last_became_active":"2024-03-23T03:48:23.469818+0000","last_became_peered":"2024-03-23T03:48:23.469818+0000","last_unstale":"2024-03-23T03:48:25.377728+0000","last_undegraded":"2024-03-23T03:48:25.377728+0000","last_fullsized":"2024-03-23T03:48:25.377728+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:26:23.251559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"40'3","reported_seq":14,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440676+0000","last_change":"2024-03-23T03:48:22.986034+0000","last_active":"2024-03-23T03:48:24.440676+0000","last_peered":"2024-03-23T03:48:24.440676+0000","last_clean":"2024-03-23T03:48:24.440676+0000","last_became_active":"2024-03-23T03:48:22.985641+0000","last_became_peered":"2024-03-23T03:48:22.985641+0000","last_unstale":"2024-03-23T03:48:24.440676+0000","last_undegraded":"2024-03-23T03:48:24.440676+0000","last_fullsized":"2024-03-23T03:48:24.440676+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:29.782326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.378929+0000","last_change":"2024-03-23T03:48:23.464888+0000","last_active":"2024-03-23T03:48:25.378929+0000","last_peered":"2024-03-23T03:48:25.378929+0000","last_clean":"2024-03-23T03:48:25.378929+0000","last_became_active":"2024-03-23T03:48:23.464439+0000","last_became_peered":"2024-03-23T03:48:23.464439+0000","last_unstale":"2024-03-23T03:48:25.378929+0000","last_undegraded":"2024-03-23T03:48:25.378929+0000","last_fullsized":"2024-03-23T03:48:25.378929+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:02:16.431086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"40'6","reported_seq":17,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438531+0000","last_change":"2024-03-23T03:48:22.978241+0000","last_active":"2024-03-23T03:48:24.438531+0000","last_peered":"2024-03-23T03:48:24.438531+0000","last_clean":"2024-03-23T03:48:24.438531+0000","last_became_active":"2024-03-23T03:48:22.978012+0000","last_became_peered":"2024-03-23T03:48:22.978012+0000","last_unstale":"2024-03-23T03:48:24.438531+0000","last_undegraded":"2024-03-23T03:48:24.438531+0000","last_fullsized":"2024-03-23T03:48:24.438531+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:20:12.237431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376527+0000","last_change":"2024-03-23T03:48:23.456812+0000","last_active":"2024-03-23T03:48:25.376527+0000","last_peered":"2024-03-23T03:48:25.376527+0000","last_clean":"2024-03-23T03:48:25.376527+0000","last_became_active":"2024-03-23T03:48:23.456502+0000","last_became_peered":"2024-03-23T03:48:23.456502+0000","last_unstale":"2024-03-23T03:48:25.376527+0000","last_undegraded":"2024-03-23T03:48:25.376527+0000","last_fullsized":"2024-03-23T03:48:25.376527+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:10:35.688861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"40'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438242+0000","last_change":"2024-03-23T03:48:22.969089+0000","last_active":"2024-03-23T03:48:24.438242+0000","last_peered":"2024-03-23T03:48:24.438242+0000","last_clean":"2024-03-23T03:48:24.438242+0000","last_became_active":"2024-03-23T03:48:22.968809+0000","last_became_peered":"2024-03-23T03:48:22.968809+0000","last_unstale":"2024-03-23T03:48:24.438242+0000","last_undegraded":"2024-03-23T03:48:24.438242+0000","last_fullsized":"2024-03-23T03:48:24.438242+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:51:16.799572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.437807+0000","last_change":"2024-03-23T03:48:23.442645+0000","last_active":"2024-03-23T03:48:24.437807+0000","last_peered":"2024-03-23T03:48:24.437807+0000","last_clean":"2024-03-23T03:48:24.437807+0000","last_became_active":"2024-03-23T03:48:23.441584+0000","last_became_peered":"2024-03-23T03:48:23.441584+0000","last_unstale":"2024-03-23T03:48:24.437807+0000","last_undegraded":"2024-03-23T03:48:24.437807+0000","last_fullsized":"2024-03-23T03:48:24.437807+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:49:28.083244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376147+0000","last_change":"2024-03-23T03:48:22.981960+0000","last_active":"2024-03-23T03:48:25.376147+0000","last_peered":"2024-03-23T03:48:25.376147+0000","last_clean":"2024-03-23T03:48:25.376147+0000","last_became_active":"2024-03-23T03:48:22.981671+0000","last_became_peered":"2024-03-23T03:48:22.981671+0000","last_unstale":"2024-03-23T03:48:25.376147+0000","last_undegraded":"2024-03-23T03:48:25.376147+0000","last_fullsized":"2024-03-23T03:48:25.376147+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:32:24.360007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.660955+0000","last_change":"2024-03-23T03:48:23.459475+0000","last_active":"2024-03-23T03:48:24.660955+0000","last_peered":"2024-03-23T03:48:24.660955+0000","last_clean":"2024-03-23T03:48:24.660955+0000","last_became_active":"2024-03-23T03:48:23.458903+0000","last_became_peered":"2024-03-23T03:48:23.458903+0000","last_unstale":"2024-03-23T03:48:24.660955+0000","last_undegraded":"2024-03-23T03:48:24.660955+0000","last_fullsized":"2024-03-23T03:48:24.660955+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:19:04.667783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376087+0000","last_change":"2024-03-23T03:48:22.971422+0000","last_active":"2024-03-23T03:48:25.376087+0000","last_peered":"2024-03-23T03:48:25.376087+0000","last_clean":"2024-03-23T03:48:25.376087+0000","last_became_active":"2024-03-23T03:48:22.971144+0000","last_became_peered":"2024-03-23T03:48:22.971144+0000","last_unstale":"2024-03-23T03:48:25.376087+0000","last_undegraded":"2024-03-23T03:48:25.376087+0000","last_fullsized":"2024-03-23T03:48:25.376087+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:18:10.874509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438143+0000","last_change":"2024-03-23T03:48:23.439626+0000","last_active":"2024-03-23T03:48:24.438143+0000","last_peered":"2024-03-23T03:48:24.438143+0000","last_clean":"2024-03-23T03:48:24.438143+0000","last_became_active":"2024-03-23T03:48:23.439357+0000","last_became_peered":"2024-03-23T03:48:23.439357+0000","last_unstale":"2024-03-23T03:48:24.438143+0000","last_undegraded":"2024-03-23T03:48:24.438143+0000","last_fullsized":"2024-03-23T03:48:24.438143+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:49:31.471021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"40'8","reported_seq":19,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.660555+0000","last_change":"2024-03-23T03:48:22.989453+0000","last_active":"2024-03-23T03:48:24.660555+0000","last_peered":"2024-03-23T03:48:24.660555+0000","last_clean":"2024-03-23T03:48:24.660555+0000","last_became_active":"2024-03-23T03:48:22.989205+0000","last_became_peered":"2024-03-23T03:48:22.989205+0000","last_unstale":"2024-03-23T03:48:24.660555+0000","last_undegraded":"2024-03-23T03:48:24.660555+0000","last_fullsized":"2024-03-23T03:48:24.660555+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:25:54.181764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438396+0000","last_change":"2024-03-23T03:48:23.462731+0000","last_active":"2024-03-23T03:48:24.438396+0000","last_peered":"2024-03-23T03:48:24.438396+0000","last_clean":"2024-03-23T03:48:24.438396+0000","last_became_active":"2024-03-23T03:48:23.462466+0000","last_became_peered":"2024-03-23T03:48:23.462466+0000","last_unstale":"2024-03-23T03:48:24.438396+0000","last_undegraded":"2024-03-23T03:48:24.438396+0000","last_fullsized":"2024-03-23T03:48:24.438396+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:56:20.636759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"40'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440887+0000","last_change":"2024-03-23T03:48:22.976437+0000","last_active":"2024-03-23T03:48:24.440887+0000","last_peered":"2024-03-23T03:48:24.440887+0000","last_clean":"2024-03-23T03:48:24.440887+0000","last_became_active":"2024-03-23T03:48:22.976178+0000","last_became_peered":"2024-03-23T03:48:22.976178+0000","last_unstale":"2024-03-23T03:48:24.440887+0000","last_undegraded":"2024-03-23T03:48:24.440887+0000","last_fullsized":"2024-03-23T03:48:24.440887+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:17:17.358011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440546+0000","last_change":"2024-03-23T03:48:23.464017+0000","last_active":"2024-03-23T03:48:24.440546+0000","last_peered":"2024-03-23T03:48:24.440546+0000","last_clean":"2024-03-23T03:48:24.440546+0000","last_became_active":"2024-03-23T03:48:23.463538+0000","last_became_peered":"2024-03-23T03:48:23.463538+0000","last_unstale":"2024-03-23T03:48:24.440546+0000","last_undegraded":"2024-03-23T03:48:24.440546+0000","last_fullsized":"2024-03-23T03:48:24.440546+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:57.996508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"40'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.378927+0000","last_change":"2024-03-23T03:48:22.984999+0000","last_active":"2024-03-23T03:48:25.378927+0000","last_peered":"2024-03-23T03:48:25.378927+0000","last_clean":"2024-03-23T03:48:25.378927+0000","last_became_active":"2024-03-23T03:48:22.984693+0000","last_became_peered":"2024-03-23T03:48:22.984693+0000","last_unstale":"2024-03-23T03:48:25.378927+0000","last_undegraded":"2024-03-23T03:48:25.378927+0000","last_fullsized":"2024-03-23T03:48:25.378927+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:54:00.988275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.378654+0000","last_change":"2024-03-23T03:48:23.464489+0000","last_active":"2024-03-23T03:48:25.378654+0000","last_peered":"2024-03-23T03:48:25.378654+0000","last_clean":"2024-03-23T03:48:25.378654+0000","last_became_active":"2024-03-23T03:48:23.463867+0000","last_became_peered":"2024-03-23T03:48:23.463867+0000","last_unstale":"2024-03-23T03:48:25.378654+0000","last_undegraded":"2024-03-23T03:48:25.378654+0000","last_fullsized":"2024-03-23T03:48:25.378654+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:17:34.748041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"40'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.164012+0000","last_change":"2024-03-23T03:48:22.988224+0000","last_active":"2024-03-23T03:48:25.164012+0000","last_peered":"2024-03-23T03:48:25.164012+0000","last_clean":"2024-03-23T03:48:25.164012+0000","last_became_active":"2024-03-23T03:48:22.987976+0000","last_became_peered":"2024-03-23T03:48:22.987976+0000","last_unstale":"2024-03-23T03:48:25.164012+0000","last_undegraded":"2024-03-23T03:48:25.164012+0000","last_fullsized":"2024-03-23T03:48:25.164012+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:42:16.089902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.164002+0000","last_change":"2024-03-23T03:48:23.449959+0000","last_active":"2024-03-23T03:48:25.164002+0000","last_peered":"2024-03-23T03:48:25.164002+0000","last_clean":"2024-03-23T03:48:25.164002+0000","last_became_active":"2024-03-23T03:48:23.449703+0000","last_became_peered":"2024-03-23T03:48:23.449703+0000","last_unstale":"2024-03-23T03:48:25.164002+0000","last_undegraded":"2024-03-23T03:48:25.164002+0000","last_fullsized":"2024-03-23T03:48:25.164002+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:34:53.562904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376514+0000","last_change":"2024-03-23T03:48:22.981892+0000","last_active":"2024-03-23T03:48:25.376514+0000","last_peered":"2024-03-23T03:48:25.376514+0000","last_clean":"2024-03-23T03:48:25.376514+0000","last_became_active":"2024-03-23T03:48:22.981609+0000","last_became_peered":"2024-03-23T03:48:22.981609+0000","last_unstale":"2024-03-23T03:48:25.376514+0000","last_undegraded":"2024-03-23T03:48:25.376514+0000","last_fullsized":"2024-03-23T03:48:25.376514+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:44:48.293049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.660626+0000","last_change":"2024-03-23T03:48:23.458010+0000","last_active":"2024-03-23T03:48:24.660626+0000","last_peered":"2024-03-23T03:48:24.660626+0000","last_clean":"2024-03-23T03:48:24.660626+0000","last_became_active":"2024-03-23T03:48:23.457723+0000","last_became_peered":"2024-03-23T03:48:23.457723+0000","last_unstale":"2024-03-23T03:48:24.660626+0000","last_undegraded":"2024-03-23T03:48:24.660626+0000","last_fullsized":"2024-03-23T03:48:24.660626+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:51:52.504015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"40'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.163485+0000","last_change":"2024-03-23T03:48:22.969409+0000","last_active":"2024-03-23T03:48:25.163485+0000","last_peered":"2024-03-23T03:48:25.163485+0000","last_clean":"2024-03-23T03:48:25.163485+0000","last_became_active":"2024-03-23T03:48:22.969132+0000","last_became_peered":"2024-03-23T03:48:22.969132+0000","last_unstale":"2024-03-23T03:48:25.163485+0000","last_undegraded":"2024-03-23T03:48:25.163485+0000","last_fullsized":"2024-03-23T03:48:25.163485+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:57.542690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376897+0000","last_change":"2024-03-23T03:48:23.460179+0000","last_active":"2024-03-23T03:48:25.376897+0000","last_peered":"2024-03-23T03:48:25.376897+0000","last_clean":"2024-03-23T03:48:25.376897+0000","last_became_active":"2024-03-23T03:48:23.458501+0000","last_became_peered":"2024-03-23T03:48:23.458501+0000","last_unstale":"2024-03-23T03:48:25.376897+0000","last_undegraded":"2024-03-23T03:48:25.376897+0000","last_fullsized":"2024-03-23T03:48:25.376897+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:57.996508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.163834+0000","last_change":"2024-03-23T03:48:22.976947+0000","last_active":"2024-03-23T03:48:25.163834+0000","last_peered":"2024-03-23T03:48:25.163834+0000","last_clean":"2024-03-23T03:48:25.163834+0000","last_became_active":"2024-03-23T03:48:22.976687+0000","last_became_peered":"2024-03-23T03:48:22.976687+0000","last_unstale":"2024-03-23T03:48:25.163834+0000","last_undegraded":"2024-03-23T03:48:25.163834+0000","last_fullsized":"2024-03-23T03:48:25.163834+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:21:21.249313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.439894+0000","last_change":"2024-03-23T03:48:23.470473+0000","last_active":"2024-03-23T03:48:24.439894+0000","last_peered":"2024-03-23T03:48:24.439894+0000","last_clean":"2024-03-23T03:48:24.439894+0000","last_became_active":"2024-03-23T03:48:23.470148+0000","last_became_peered":"2024-03-23T03:48:23.470148+0000","last_unstale":"2024-03-23T03:48:24.439894+0000","last_undegraded":"2024-03-23T03:48:24.439894+0000","last_fullsized":"2024-03-23T03:48:24.439894+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:32:30.609829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661197+0000","last_change":"2024-03-23T03:48:22.988465+0000","last_active":"2024-03-23T03:48:24.661197+0000","last_peered":"2024-03-23T03:48:24.661197+0000","last_clean":"2024-03-23T03:48:24.661197+0000","last_became_active":"2024-03-23T03:48:22.988174+0000","last_became_peered":"2024-03-23T03:48:22.988174+0000","last_unstale":"2024-03-23T03:48:24.661197+0000","last_undegraded":"2024-03-23T03:48:24.661197+0000","last_fullsized":"2024-03-23T03:48:24.661197+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:04:20.350462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438251+0000","last_change":"2024-03-23T03:48:23.460122+0000","last_active":"2024-03-23T03:48:24.438251+0000","last_peered":"2024-03-23T03:48:24.438251+0000","last_clean":"2024-03-23T03:48:24.438251+0000","last_became_active":"2024-03-23T03:48:23.459860+0000","last_became_peered":"2024-03-23T03:48:23.459860+0000","last_unstale":"2024-03-23T03:48:24.438251+0000","last_undegraded":"2024-03-23T03:48:24.438251+0000","last_fullsized":"2024-03-23T03:48:24.438251+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:15:10.844824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.163683+0000","last_change":"2024-03-23T03:48:22.971252+0000","last_active":"2024-03-23T03:48:25.163683+0000","last_peered":"2024-03-23T03:48:25.163683+0000","last_clean":"2024-03-23T03:48:25.163683+0000","last_became_active":"2024-03-23T03:48:22.970961+0000","last_became_peered":"2024-03-23T03:48:22.970961+0000","last_unstale":"2024-03-23T03:48:25.163683+0000","last_undegraded":"2024-03-23T03:48:25.163683+0000","last_fullsized":"2024-03-23T03:48:25.163683+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:02:35.930419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376375+0000","last_change":"2024-03-23T03:48:23.471709+0000","last_active":"2024-03-23T03:48:25.376375+0000","last_peered":"2024-03-23T03:48:25.376375+0000","last_clean":"2024-03-23T03:48:25.376375+0000","last_became_active":"2024-03-23T03:48:23.471437+0000","last_became_peered":"2024-03-23T03:48:23.471437+0000","last_unstale":"2024-03-23T03:48:25.376375+0000","last_undegraded":"2024-03-23T03:48:25.376375+0000","last_fullsized":"2024-03-23T03:48:25.376375+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:46:31.492659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"40'2","reported_seq":13,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.441011+0000","last_change":"2024-03-23T03:48:22.986439+0000","last_active":"2024-03-23T03:48:24.441011+0000","last_peered":"2024-03-23T03:48:24.441011+0000","last_clean":"2024-03-23T03:48:24.441011+0000","last_became_active":"2024-03-23T03:48:22.986029+0000","last_became_peered":"2024-03-23T03:48:22.986029+0000","last_unstale":"2024-03-23T03:48:24.441011+0000","last_undegraded":"2024-03-23T03:48:24.441011+0000","last_fullsized":"2024-03-23T03:48:24.441011+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:40:21.057524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661581+0000","last_change":"2024-03-23T03:48:23.474303+0000","last_active":"2024-03-23T03:48:24.661581+0000","last_peered":"2024-03-23T03:48:24.661581+0000","last_clean":"2024-03-23T03:48:24.661581+0000","last_became_active":"2024-03-23T03:48:23.473966+0000","last_became_peered":"2024-03-23T03:48:23.473966+0000","last_unstale":"2024-03-23T03:48:24.661581+0000","last_undegraded":"2024-03-23T03:48:24.661581+0000","last_fullsized":"2024-03-23T03:48:24.661581+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:57:34.228967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"41'1","reported_seq":12,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.167496+0000","last_change":"2024-03-23T03:48:22.982756+0000","last_active":"2024-03-23T03:48:25.167496+0000","last_peered":"2024-03-23T03:48:25.167496+0000","last_clean":"2024-03-23T03:48:25.167496+0000","last_became_active":"2024-03-23T03:48:22.982122+0000","last_became_peered":"2024-03-23T03:48:22.982122+0000","last_unstale":"2024-03-23T03:48:25.167496+0000","last_undegraded":"2024-03-23T03:48:25.167496+0000","last_fullsized":"2024-03-23T03:48:25.167496+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:28.805507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.438508+0000","last_change":"2024-03-23T03:48:23.475385+0000","last_active":"2024-03-23T03:48:24.438508+0000","last_peered":"2024-03-23T03:48:24.438508+0000","last_clean":"2024-03-23T03:48:24.438508+0000","last_became_active":"2024-03-23T03:48:23.475116+0000","last_became_peered":"2024-03-23T03:48:23.475116+0000","last_unstale":"2024-03-23T03:48:24.438508+0000","last_undegraded":"2024-03-23T03:48:24.438508+0000","last_fullsized":"2024-03-23T03:48:24.438508+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:40:04.714830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377525+0000","last_change":"2024-03-23T03:48:22.970156+0000","last_active":"2024-03-23T03:48:25.377525+0000","last_peered":"2024-03-23T03:48:25.377525+0000","last_clean":"2024-03-23T03:48:25.377525+0000","last_became_active":"2024-03-23T03:48:22.969896+0000","last_became_peered":"2024-03-23T03:48:22.969896+0000","last_unstale":"2024-03-23T03:48:25.377525+0000","last_undegraded":"2024-03-23T03:48:25.377525+0000","last_fullsized":"2024-03-23T03:48:25.377525+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:19:03.653303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376789+0000","last_change":"2024-03-23T03:48:23.450335+0000","last_active":"2024-03-23T03:48:25.376789+0000","last_peered":"2024-03-23T03:48:25.376789+0000","last_clean":"2024-03-23T03:48:25.376789+0000","last_became_active":"2024-03-23T03:48:23.450088+0000","last_became_peered":"2024-03-23T03:48:23.450088+0000","last_unstale":"2024-03-23T03:48:25.376789+0000","last_undegraded":"2024-03-23T03:48:25.376789+0000","last_fullsized":"2024-03-23T03:48:25.376789+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:01:27.304608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377208+0000","last_change":"2024-03-23T03:48:22.976577+0000","last_active":"2024-03-23T03:48:25.377208+0000","last_peered":"2024-03-23T03:48:25.377208+0000","last_clean":"2024-03-23T03:48:25.377208+0000","last_became_active":"2024-03-23T03:48:22.976313+0000","last_became_peered":"2024-03-23T03:48:22.976313+0000","last_unstale":"2024-03-23T03:48:25.377208+0000","last_undegraded":"2024-03-23T03:48:25.377208+0000","last_fullsized":"2024-03-23T03:48:25.377208+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:57:33.214487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376065+0000","last_change":"2024-03-23T03:48:23.462646+0000","last_active":"2024-03-23T03:48:25.376065+0000","last_peered":"2024-03-23T03:48:25.376065+0000","last_clean":"2024-03-23T03:48:25.376065+0000","last_became_active":"2024-03-23T03:48:23.462371+0000","last_became_peered":"2024-03-23T03:48:23.462371+0000","last_unstale":"2024-03-23T03:48:25.376065+0000","last_undegraded":"2024-03-23T03:48:25.376065+0000","last_fullsized":"2024-03-23T03:48:25.376065+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:20:23.375166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.376537+0000","last_change":"2024-03-23T03:48:22.970375+0000","last_active":"2024-03-23T03:48:25.376537+0000","last_peered":"2024-03-23T03:48:25.376537+0000","last_clean":"2024-03-23T03:48:25.376537+0000","last_became_active":"2024-03-23T03:48:22.970132+0000","last_became_peered":"2024-03-23T03:48:22.970132+0000","last_unstale":"2024-03-23T03:48:25.376537+0000","last_undegraded":"2024-03-23T03:48:25.376537+0000","last_fullsized":"2024-03-23T03:48:25.376537+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:31:47.845785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377939+0000","last_change":"2024-03-23T03:48:23.478744+0000","last_active":"2024-03-23T03:48:25.377939+0000","last_peered":"2024-03-23T03:48:25.377939+0000","last_clean":"2024-03-23T03:48:25.377939+0000","last_became_active":"2024-03-23T03:48:23.474090+0000","last_became_peered":"2024-03-23T03:48:23.474090+0000","last_unstale":"2024-03-23T03:48:25.377939+0000","last_undegraded":"2024-03-23T03:48:25.377939+0000","last_fullsized":"2024-03-23T03:48:25.377939+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:44:35.418902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:25.377971+0000","last_change":"2024-03-23T03:48:22.979722+0000","last_active":"2024-03-23T03:48:25.377971+0000","last_peered":"2024-03-23T03:48:25.377971+0000","last_clean":"2024-03-23T03:48:25.377971+0000","last_became_active":"2024-03-23T03:48:22.978953+0000","last_became_peered":"2024-03-23T03:48:22.978953+0000","last_unstale":"2024-03-23T03:48:25.377971+0000","last_undegraded":"2024-03-23T03:48:25.377971+0000","last_fullsized":"2024-03-23T03:48:25.377971+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:20:07.924853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661328+0000","last_change":"2024-03-23T03:48:23.465289+0000","last_active":"2024-03-23T03:48:24.661328+0000","last_peered":"2024-03-23T03:48:24.661328+0000","last_clean":"2024-03-23T03:48:24.661328+0000","last_became_active":"2024-03-23T03:48:23.464931+0000","last_became_peered":"2024-03-23T03:48:23.464931+0000","last_unstale":"2024-03-23T03:48:24.661328+0000","last_undegraded":"2024-03-23T03:48:24.661328+0000","last_fullsized":"2024-03-23T03:48:24.661328+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:32:25.374487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.437736+0000","last_change":"2024-03-23T03:48:22.971532+0000","last_active":"2024-03-23T03:48:24.437736+0000","last_peered":"2024-03-23T03:48:24.437736+0000","last_clean":"2024-03-23T03:48:24.437736+0000","last_became_active":"2024-03-23T03:48:22.971261+0000","last_became_peered":"2024-03-23T03:48:22.971261+0000","last_unstale":"2024-03-23T03:48:24.437736+0000","last_undegraded":"2024-03-23T03:48:24.437736+0000","last_fullsized":"2024-03-23T03:48:24.437736+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:08:54.694550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.440733+0000","last_change":"2024-03-23T03:48:23.458842+0000","last_active":"2024-03-23T03:48:24.440733+0000","last_peered":"2024-03-23T03:48:24.440733+0000","last_clean":"2024-03-23T03:48:24.440733+0000","last_became_active":"2024-03-23T03:48:23.458009+0000","last_became_peered":"2024-03-23T03:48:23.458009+0000","last_unstale":"2024-03-23T03:48:24.440733+0000","last_undegraded":"2024-03-23T03:48:24.440733+0000","last_fullsized":"2024-03-23T03:48:24.440733+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":39,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:22.934633+0000","last_clean_scrub_stamp":"2024-03-23T03:48:22.934633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:52:40.102104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T03:48:24.661302+0000","last_change":"2024-03-23T03:48:22.990110+0000","last_active":"2024-03-23T03:48:24.661302+0000","last_peered":"2024-03-23T03:48:24.661302+0000","last_clean":"2024-03-23T03:48:24.661302+0000","last_became_active":"2024-03-23T03:48:22.989848+0000","last_became_peered":"2024-03-23T03:48:22.989848+0000","last_unstale":"2024-03-23T03:48:24.661302+0000","last_undegraded":"2024-03-23T03:48:24.661302+0000","last_fullsized":"2024-03-23T03:48:24.661302+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:21.920153+0000","last_clean_scrub_stamp":"2024-03-23T03:48:21.920153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:14:45.272739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":179,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-23T03:48:23.445064+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:48:23.445064+0000","last_peered":"2024-03-23T03:48:23.445064+0000","last_clean":"2024-03-23T03:48:23.445064+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:48:23.445064+0000","last_undegraded":"2024-03-23T03:48:23.445064+0000","last_fullsized":"2024-03-23T03:48:23.445064+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542168,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343764,"statfs":{"total":96636764160,"available":96608014336,"internally_reserved":0,"allocated":1081344,"data_stored":793927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.635,"5min":1.635,"15min":1.635},"last":1.091},{"interface":"front","average":{"1min":1.05,"5min":1.05,"15min":1.05},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.48}]},{"osd":1,"last update":"Sat Mar 23 03:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.8959999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":4.9109999999999996,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":1.042}]},{"osd":2,"last update":"Sat Mar 23 03:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.165,"15min":1.165},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.9690000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":4.8310000000000004,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.421}]},{"osd":3,"last update":"Sat Mar 23 03:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"max":{"1min":4.8639999999999999,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"min":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"max":{"1min":2.1739999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.536}]},{"osd":4,"last update":"Sat Mar 23 03:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.268,"15min":1.268},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.22},{"interface":"front","average":{"1min":1.385,"5min":1.385,"15min":1.385},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.5659999999999998,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.6890000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"min":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"max":{"1min":2.645,"5min":2.645,"15min":2.645},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"min":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"max":{"1min":2.5089999999999999,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.661}]},{"osd":6,"last update":"Sat Mar 23 03:47:48 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.577,"15min":1.577},"min":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"max":{"1min":2.7730000000000001,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.367},{"interface":"front","average":{"1min":1.536,"5min":1.536,"15min":1.536},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":2.7170000000000001,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":2.0030000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542166,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":475136,"data_stored":202507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:47:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.93899999999999995}]},{"osd":1,"last update":"Sat Mar 23 03:47:47 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.069,"15min":1.069},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":1.222},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.1080000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:47:47 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"max":{"1min":1.498,"5min":1.498,"15min":1.498},"last":1.05},{"interface":"front","average":{"1min":1.23,"5min":1.23,"15min":1.23},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":1.137}]},{"osd":3,"last update":"Sat Mar 23 03:47:47 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.9139999999999999,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.27,"5min":1.27,"15min":1.27},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.3580000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:47:47 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.389,"15min":1.389},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.3799999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.272}]},{"osd":5,"last update":"Sat Mar 23 03:47:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":2.4340000000000002,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.5589999999999999,"5min":1.5589999999999999,"15min":1.5589999999999999},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":2.6349999999999998,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.4570000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:47:47 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.415,"15min":1.415},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.387},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.5870000000000002,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.494}]}]},{"osd":5,"up_from":14,"seq":60129542166,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27488,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344352,"statfs":{"total":96636764160,"available":96608616448,"internally_reserved":0,"allocated":479232,"data_stored":202037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:47:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.806,"5min":1.806,"15min":1.806},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.194,"5min":1.194,"15min":1.194},"min":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.71599999999999997}]},{"osd":1,"last update":"Sat Mar 23 03:47:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"max":{"1min":1.7669999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.208,"5min":1.208,"15min":1.208},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.8620000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.244}]},{"osd":2,"last update":"Sat Mar 23 03:47:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.9199999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":1.016,"5min":1.016,"15min":1.016},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.3240000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:47:46 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.071},{"interface":"front","average":{"1min":1.482,"5min":1.482,"15min":1.482},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.9969999999999999,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.484}]},{"osd":4,"last update":"Sat Mar 23 03:47:46 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"min":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"max":{"1min":2.1419999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.367}]},{"osd":6,"last update":"Sat Mar 23 03:47:46 2024","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.5429999999999999,"15min":1.5429999999999999},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.4950000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"min":{"1min":1.101,"5min":1.101,"15min":1.101},"max":{"1min":2.6299999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.276}]},{"osd":7,"last update":"Sat Mar 23 03:47:46 2024","interfaces":[{"interface":"back","average":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"min":{"1min":1.143,"5min":1.143,"15min":1.143},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.294},{"interface":"front","average":{"1min":1.734,"5min":1.734,"15min":1.734},"min":{"1min":1.236,"5min":1.236,"15min":1.236},"max":{"1min":3.0640000000000001,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.429}]}]},{"osd":4,"up_from":14,"seq":60129542167,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27488,"kb_used_data":468,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344352,"statfs":{"total":96636764160,"available":96608616448,"internally_reserved":0,"allocated":479232,"data_stored":202529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.383,"5min":1.383,"15min":1.383},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.4950000000000001}]},{"osd":1,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.262},{"interface":"front","average":{"1min":1.052,"5min":1.052,"15min":1.052},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.4159999999999999,"5min":1.4159999999999999,"15min":1.4159999999999999},"last":1.3149999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.341,"15min":1.341},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":3.0859999999999999,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":1.002},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":2.181,"5min":2.181,"15min":2.181},"last":1.5149999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.129,"15min":1.129},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.399},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.2869999999999999,"15min":1.2869999999999999},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":3.1749999999999998,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.1020000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.272,"15min":1.272},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.841,"5min":1.841,"15min":1.841},"last":1.454},{"interface":"front","average":{"1min":1.276,"5min":1.276,"15min":1.276},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.8639999999999999,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.5429999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":3.2570000000000001,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.353},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":1.5700000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.0369999999999999,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":1.587},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.9420000000000002,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.4330000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542167,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27496,"kb_used_data":476,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344344,"statfs":{"total":96636764160,"available":96608608256,"internally_reserved":0,"allocated":487424,"data_stored":203523,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.4199999999999999,"5min":1.4199999999999999,"15min":1.4199999999999999},"last":1.8149999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":5.8499999999999996}]},{"osd":1,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.343,"5min":2.343,"15min":2.343},"last":3.1850000000000001},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":2.2530000000000001},"last":2.7450000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.8999999999999999,"5min":1.8999999999999999,"15min":1.8999999999999999},"last":4.8010000000000002}]},{"osd":4,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.252,"15min":1.252},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.9810000000000001,"5min":1.9810000000000001,"15min":1.9810000000000001},"last":5.9210000000000003},{"interface":"front","average":{"1min":1.468,"5min":1.468,"15min":1.468},"min":{"1min":0.87,"5min":0.87,"15min":0.87},"max":{"1min":4.819,"5min":4.819,"15min":4.819},"last":4.5620000000000003}]},{"osd":5,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.369,"15min":1.369},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":2.1349999999999998,"5min":2.1349999999999998,"15min":2.1349999999999998},"last":30.228000000000002},{"interface":"front","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":5.8920000000000003}]},{"osd":6,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"min":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"max":{"1min":4.7859999999999996,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.9319999999999999},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":4.7610000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.4359999999999999,"15min":1.4359999999999999},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.1629999999999998},"last":5.9589999999999996},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":4.7400000000000002,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":10.663}]}]},{"osd":2,"up_from":14,"seq":60129542167,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27492,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":475136,"data_stored":201995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":2.226,"5min":2.226,"15min":2.226},"last":0.81000000000000005}]},{"osd":1,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":2.1899999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":2.1739999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.48199999999999998}]},{"osd":3,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.60799999999999998}]},{"osd":4,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.25,"15min":1.25},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.9119999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.232,"15min":1.232},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.88500000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"max":{"1min":1.97,"5min":1.97,"15min":1.97},"last":1},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.8500000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.1719999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.3759999999999999,"15min":1.3759999999999999},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":2.456,"5min":2.456,"15min":2.456},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":1.2390000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:47:51 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.78600000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542166,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":475136,"data_stored":202015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.2749999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":1.2509999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.6990000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.78200000000000003},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.625,"5min":1.625,"15min":1.625},"last":1.139}]},{"osd":3,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.032,"5min":1.032,"15min":1.032},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.2410000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.999}]},{"osd":4,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.323,"15min":1.323},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.1739999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.335,"5min":1.335,"15min":1.335},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.3100000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.728}]},{"osd":5,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.357,"15min":1.357},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":3.593,"5min":3.593,"15min":3.593},"last":1.577},{"interface":"front","average":{"1min":1.476,"5min":1.476,"15min":1.476},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":1.0189999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.4319999999999999,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.5389999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.387,"15min":1.387},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":3.7429999999999999,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.766}]},{"osd":7,"last update":"Sat Mar 23 03:47:49 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.47,"15min":1.47},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":3.8839999999999999,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.821},{"interface":"front","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":3.3079999999999998,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.357}]}]},{"osd":0,"up_from":14,"seq":60129542166,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28064,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343776,"statfs":{"total":96636764160,"available":96608026624,"internally_reserved":0,"allocated":1069056,"data_stored":792327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:47:50 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.5009999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":1.244},{"interface":"front","average":{"1min":1,"5min":1,"15min":1},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.76600000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:47:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.901,"5min":1.901,"15min":1.901},"last":1.3180000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:47:50 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.028,"15min":1.028},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"max":{"1min":1.972,"5min":1.972,"15min":1.972},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.1659999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:47:50 2024","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.361,"15min":1.361},"min":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"max":{"1min":1.9650000000000001,"5min":1.9650000000000001,"15min":1.9650000000000001},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.294,"5min":1.294,"15min":1.294},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":1.087}]},{"osd":5,"last update":"Sat Mar 23 03:47:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"min":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.3799999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.29},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"min":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"max":{"1min":2.9449999999999998,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.3480000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:47:50 2024","interfaces":[{"interface":"back","average":{"1min":1.577,"5min":1.577,"15min":1.577},"min":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"max":{"1min":3.032,"5min":3.032,"15min":3.032},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":2.6339999999999999,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.2789999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:47:50 2024","interfaces":[{"interface":"back","average":{"1min":1.54,"5min":1.54,"15min":1.54},"min":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"max":{"1min":2.5249999999999999,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.448,"5min":1.448,"15min":1.448},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.335}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:48:28.571 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T03:48:28.571 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-23T03:48:28.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.828+0000 7fc7cd600640 1 Processor -- start 2024-03-23T03:48:28.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.829+0000 7fc7cd600640 1 -- start start 2024-03-23T03:48:28.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.830+0000 7fc7cd600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7c8160d20 0x7fc7c8164920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:28.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.830+0000 7fc7cd600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c8165830 con 0x7fc7c80cf040 2024-03-23T03:48:28.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.830+0000 7fc7cd600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7c8164f90 con 0x7fc7c80ca7b0 2024-03-23T03:48:28.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.830+0000 7fc7cd600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c8165270 con 0x7fc7c8160d20 2024-03-23T03:48:28.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.831+0000 7fc7c7400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc7c80cf040 0x7fc7c80b1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37830/0 (socket says 172.21.15.102:37830) 2024-03-23T03:48:28.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.831+0000 7fc7c7400640 1 -- 172.21.15.102:0/1723200547 learned_addr learned my addr 172.21.15.102:0/1723200547 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:28.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.831+0000 7fc7c6a00640 1 --2- 172.21.15.102:0/1723200547 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7c8160d20 0x7fc7c8164920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:28.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.831+0000 7fc7c6000640 1 -- 172.21.15.102:0/1723200547 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474990629 0 0) 0x7fc7c8165830 con 0x7fc7c80cf040 2024-03-23T03:48:28.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.831+0000 7fc7c6000640 1 -- 172.21.15.102:0/1723200547 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7a8003520 con 0x7fc7c80cf040 2024-03-23T03:48:28.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.832+0000 7fc7c6000640 1 -- 172.21.15.102:0/1723200547 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2620595869 0 0) 0x7fc7a8003520 con 0x7fc7c80cf040 2024-03-23T03:48:28.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.832+0000 7fc7c6000640 1 -- 172.21.15.102:0/1723200547 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7c8160d20 msgr2=0x7fc7c8164920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:28.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.832+0000 7fc7c6000640 1 --2- 172.21.15.102:0/1723200547 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7c8160d20 0x7fc7c8164920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.832+0000 7fc7c6000640 1 -- 172.21.15.102:0/1723200547 >> v1:172.21.15.173:6789/0 conn(0x7fc7c80ca7b0 legacy=0x7fc7c80ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:28.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.832+0000 7fc7c6000640 1 -- 172.21.15.102:0/1723200547 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c8165550 con 0x7fc7c80cf040 2024-03-23T03:48:28.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.832+0000 7fc7c6000640 1 -- 172.21.15.102:0/1723200547 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fc7c00019f0 con 0x7fc7c80cf040 2024-03-23T03:48:28.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.832+0000 7fc7c6000640 1 -- 172.21.15.102:0/1723200547 >> v1:172.21.15.102:6789/0 conn(0x7fc7c80cf040 legacy=0x7fc7c80b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:28.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.832+0000 7fc7c6000640 1 --2- 172.21.15.102:0/1723200547 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7c8160d20 0x7fc7a80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:28.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.832+0000 7fc7c6000640 1 --2- 172.21.15.102:0/1723200547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7a8003f20 0x7fc7a80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:28.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.832+0000 7fc7c6000640 1 --2- 172.21.15.102:0/1723200547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7a8004d00 0x7fc7a8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:28.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.832+0000 7fc7c6000640 1 -- 172.21.15.102:0/1723200547 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c8165270 con 0x7fc7a8003f20 2024-03-23T03:48:28.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.832+0000 7fc7c6000640 1 -- 172.21.15.102:0/1723200547 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a800bb10 con 0x7fc7a8004d00 2024-03-23T03:48:28.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.832+0000 7fc7c6000640 1 -- 172.21.15.102:0/1723200547 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a800bcd0 con 0x7fc7c8160d20 2024-03-23T03:48:28.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.832+0000 7fc7bfe00640 1 --2- 172.21.15.102:0/1723200547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7a8003f20 0x7fc7a80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:28.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.833+0000 7fc7c6a00640 1 --2- 172.21.15.102:0/1723200547 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7c8160d20 0x7fc7a80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:28.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.833+0000 7fc7bfe00640 1 -- 172.21.15.102:0/1723200547 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7c8160d20 msgr2=0x7fc7a80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:28.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.833+0000 7fc7bfe00640 1 --2- 172.21.15.102:0/1723200547 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7c8160d20 0x7fc7a80039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.833+0000 7fc7bfe00640 1 -- 172.21.15.102:0/1723200547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7a8004d00 msgr2=0x7fc7a8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:28.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.833+0000 7fc7bfe00640 1 --2- 172.21.15.102:0/1723200547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7a8004d00 0x7fc7a8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.833+0000 7fc7bfe00640 1 -- 172.21.15.102:0/1723200547 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a8003520 con 0x7fc7a8003f20 2024-03-23T03:48:28.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.834+0000 7fc7bfe00640 1 --2- 172.21.15.102:0/1723200547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7a8003f20 0x7fc7a80047c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0001750 tx=0x7fc7a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cdc611f9d67d282 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:28.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.834+0000 7fc7c6000640 1 -- 172.21.15.102:0/1723200547 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b00052d0 con 0x7fc7a8003f20 2024-03-23T03:48:28.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.834+0000 7fc7c6000640 1 -- 172.21.15.102:0/1723200547 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b002e050 con 0x7fc7a8003f20 2024-03-23T03:48:28.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.834+0000 7fc7c6000640 1 -- 172.21.15.102:0/1723200547 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b00035f0 con 0x7fc7a8003f20 2024-03-23T03:48:28.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.835+0000 7fc7cd600640 1 -- 172.21.15.102:0/1723200547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7a8003f20 msgr2=0x7fc7a80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:28.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.835+0000 7fc7cd600640 1 --2- 172.21.15.102:0/1723200547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7a8003f20 0x7fc7a80047c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0001750 tx=0x7fc7a80057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.835+0000 7fc7c7400640 1 -- 172.21.15.102:0/1723200547 reap_dead start 2024-03-23T03:48:28.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.835+0000 7fc7cd600640 1 -- 172.21.15.102:0/1723200547 shutdown_connections 2024-03-23T03:48:28.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.835+0000 7fc7cd600640 1 -- 172.21.15.102:0/1723200547 >> 172.21.15.102:0/1723200547 conn(0x7fc7c806a0b0 msgr2=0x7fc7c806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:28.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.835+0000 7fc7cd600640 1 -- 172.21.15.102:0/1723200547 shutdown_connections 2024-03-23T03:48:28.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.835+0000 7fc7cd600640 1 -- 172.21.15.102:0/1723200547 wait complete. 2024-03-23T03:48:28.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.837+0000 7fc7cd600640 1 Processor -- start 2024-03-23T03:48:28.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.837+0000 7fc7cd600640 1 -- start start 2024-03-23T03:48:28.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.838+0000 7fc7cd600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7a8003f20 0x7fc7c80b0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:28.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.838+0000 7fc7cd600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7c80cd110 0x7fc7c80cd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:28.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.838+0000 7fc7cd600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7c80af1e0 0x7fc7c80af660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:28.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.838+0000 7fc7cd600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c80aedc0 con 0x7fc7a8003f20 2024-03-23T03:48:28.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.838+0000 7fc7cd600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c80cde90 con 0x7fc7c80af1e0 2024-03-23T03:48:28.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.838+0000 7fc7cd600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c815e030 con 0x7fc7c80cd110 2024-03-23T03:48:28.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.838+0000 7fc7c6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7a8003f20 0x7fc7c80b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:28.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.838+0000 7fc7c6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7a8003f20 0x7fc7c80b0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45030/0 (socket says 172.21.15.102:45030) 2024-03-23T03:48:28.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.838+0000 7fc7c6a00640 1 -- 172.21.15.102:0/1579958356 learned_addr learned my addr 172.21.15.102:0/1579958356 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:28.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.838+0000 7fc7bfe00640 1 --2- 172.21.15.102:0/1579958356 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7c80cd110 0x7fc7c80cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:28.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.839+0000 7fc7c6a00640 1 -- 172.21.15.102:0/1579958356 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7c80cd110 msgr2=0x7fc7c80cd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:28.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.839+0000 7fc7c6a00640 1 --2- 172.21.15.102:0/1579958356 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7c80cd110 0x7fc7c80cd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.839+0000 7fc7c6a00640 1 -- 172.21.15.102:0/1579958356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7c80af1e0 msgr2=0x7fc7c80af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:28.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.839+0000 7fc7c6a00640 1 --2- 172.21.15.102:0/1579958356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7c80af1e0 0x7fc7c80af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:28.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.839+0000 7fc7c6a00640 1 -- 172.21.15.102:0/1579958356 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c80ae420 con 0x7fc7a8003f20 2024-03-23T03:48:28.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.839+0000 7fc7c6a00640 1 --2- 172.21.15.102:0/1579958356 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7a8003f20 0x7fc7c80b0970 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc7b80108c0 tx=0x7fc7b8010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:28.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.839+0000 7fc7bea00640 1 -- 172.21.15.102:0/1579958356 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b8009610 con 0x7fc7a8003f20 2024-03-23T03:48:28.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.839+0000 7fc7bea00640 1 -- 172.21.15.102:0/1579958356 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7b800eb00 con 0x7fc7a8003f20 2024-03-23T03:48:28.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.839+0000 7fc7bea00640 1 -- 172.21.15.102:0/1579958356 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7b800ee90 con 0x7fc7a8003f20 2024-03-23T03:48:28.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.839+0000 7fc7cd600640 1 -- 172.21.15.102:0/1579958356 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c80ae700 con 0x7fc7a8003f20 2024-03-23T03:48:28.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.840+0000 7fc7cd600640 1 -- 172.21.15.102:0/1579958356 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c80cdab0 con 0x7fc7a8003f20 2024-03-23T03:48:28.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.841+0000 7fc7cd600640 1 -- 172.21.15.102:0/1579958356 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c80af030 con 0x7fc7a8003f20 2024-03-23T03:48:28.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.842+0000 7fc7bea00640 1 -- 172.21.15.102:0/1579958356 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc7b801c400 con 0x7fc7a8003f20 2024-03-23T03:48:28.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.842+0000 7fc7bea00640 1 --2- 172.21.15.102:0/1579958356 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc7a406ac30 0x7fc7a406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:28.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.843+0000 7fc7bea00640 1 -- 172.21.15.102:0/1579958356 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc7b8089960 con 0x7fc7a8003f20 2024-03-23T03:48:28.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.843+0000 7fc7bfe00640 1 --2- 172.21.15.102:0/1579958356 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc7a406ac30 0x7fc7a406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:28.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.844+0000 7fc7bfe00640 1 --2- 172.21.15.102:0/1579958356 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc7a406ac30 0x7fc7a406d0f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0001750 tx=0x7fc7b0009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:28.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:28.847+0000 7fc7bea00640 1 -- 172.21.15.102:0/1579958356 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc7b8054a00 con 0x7fc7a8003f20 2024-03-23T03:48:29.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.107+0000 7fc7cd600640 1 -- 172.21.15.102:0/1579958356 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fc7c8073390 con 0x7fc7a8003f20 2024-03-23T03:48:29.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.113+0000 7fc7bea00640 1 -- 172.21.15.102:0/1579958356 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+0 (secure 0 0 0) 0x7fc7b8058020 con 0x7fc7a8003f20 2024-03-23T03:48:29.113 INFO:teuthology.orchestra.run.smithi102.stderr:updated caps for client.0 2024-03-23T03:48:29.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.115+0000 7fc7cd600640 1 -- 172.21.15.102:0/1579958356 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc7a406ac30 msgr2=0x7fc7a406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:29.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.115+0000 7fc7cd600640 1 --2- 172.21.15.102:0/1579958356 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc7a406ac30 0x7fc7a406d0f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc7b0001750 tx=0x7fc7b0009030 comp rx=0 tx=0).stop 2024-03-23T03:48:29.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.115+0000 7fc7cd600640 1 -- 172.21.15.102:0/1579958356 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7a8003f20 msgr2=0x7fc7c80b0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:29.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.115+0000 7fc7cd600640 1 --2- 172.21.15.102:0/1579958356 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7a8003f20 0x7fc7c80b0970 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc7b80108c0 tx=0x7fc7b8010d80 comp rx=0 tx=0).stop 2024-03-23T03:48:29.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.116+0000 7fc7cd600640 1 -- 172.21.15.102:0/1579958356 shutdown_connections 2024-03-23T03:48:29.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.116+0000 7fc7cd600640 1 --2- 172.21.15.102:0/1579958356 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc7a406ac30 0x7fc7a406d0f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:29.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.116+0000 7fc7cd600640 1 --2- 172.21.15.102:0/1579958356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7c80af1e0 0x7fc7c80af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:29.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.116+0000 7fc7cd600640 1 --2- 172.21.15.102:0/1579958356 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7c80cd110 0x7fc7c80cd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:29.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.116+0000 7fc7cd600640 1 --2- 172.21.15.102:0/1579958356 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7a8003f20 0x7fc7c80b0970 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:29.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.116+0000 7fc7cd600640 1 -- 172.21.15.102:0/1579958356 >> 172.21.15.102:0/1579958356 conn(0x7fc7c806a0b0 msgr2=0x7fc7c80ce900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:29.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.116+0000 7fc7cd600640 1 -- 172.21.15.102:0/1579958356 shutdown_connections 2024-03-23T03:48:29.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.116+0000 7fc7cd600640 1 -- 172.21.15.102:0/1579958356 wait complete. 2024-03-23T03:48:29.134 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:48:29.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.339+0000 7f924ec00640 1 Processor -- start 2024-03-23T03:48:29.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.339+0000 7f924ec00640 1 -- start start 2024-03-23T03:48:29.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.340+0000 7f924ec00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f924815cab0 0x7f92480b8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:29.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.340+0000 7f924ec00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f92480d1c30 0x7f92480cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:29.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.340+0000 7f924ec00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92480cf850 con 0x7f92480ceeb0 2024-03-23T03:48:29.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.340+0000 7f924ec00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f924815ce90 con 0x7f92480d1c30 2024-03-23T03:48:29.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.340+0000 7f924ec00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92480cddf0 con 0x7f924815cab0 2024-03-23T03:48:29.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.341+0000 7f924e200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f92480d1c30 0x7f92480cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:29.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.341+0000 7f924e200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f92480d1c30 0x7f92480cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45036/0 (socket says 172.21.15.102:45036) 2024-03-23T03:48:29.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.341+0000 7f924e200640 1 -- 172.21.15.102:0/844585141 learned_addr learned my addr 172.21.15.102:0/844585141 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:29.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.341+0000 7f924d800640 1 --2- 172.21.15.102:0/844585141 >> v2:172.21.15.173:3300/0 conn(0x7f924815cab0 0x7f92480b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:29.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.341+0000 7f924e200640 1 -- 172.21.15.102:0/844585141 >> v2:172.21.15.173:3300/0 conn(0x7f924815cab0 msgr2=0x7f92480b8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:29.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.341+0000 7f924e200640 1 --2- 172.21.15.102:0/844585141 >> v2:172.21.15.173:3300/0 conn(0x7f924815cab0 0x7f92480b8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:29.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.341+0000 7f924e200640 1 -- 172.21.15.102:0/844585141 >> v1:172.21.15.173:6789/0 conn(0x7f92480ceeb0 legacy=0x7f92480b4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:29.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.341+0000 7f924e200640 1 -- 172.21.15.102:0/844585141 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92480cd570 con 0x7f92480d1c30 2024-03-23T03:48:29.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.342+0000 7f924e200640 1 --2- 172.21.15.102:0/844585141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92480d1c30 0x7f92480cf310 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f924808b100 tx=0x7f924002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea467ef3cb5b0f54 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:29.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.343+0000 7f9247e00640 1 -- 172.21.15.102:0/844585141 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924002c9f0 con 0x7f92480d1c30 2024-03-23T03:48:29.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.343+0000 7f9247e00640 1 -- 172.21.15.102:0/844585141 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924002cb50 con 0x7f92480d1c30 2024-03-23T03:48:29.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.343+0000 7f9247e00640 1 -- 172.21.15.102:0/844585141 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92400343f0 con 0x7f92480d1c30 2024-03-23T03:48:29.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.343+0000 7f924ec00640 1 -- 172.21.15.102:0/844585141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92480d1c30 msgr2=0x7f92480cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:29.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.343+0000 7f924ec00640 1 --2- 172.21.15.102:0/844585141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92480d1c30 0x7f92480cf310 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f924808b100 tx=0x7f924002be50 comp rx=0 tx=0).stop 2024-03-23T03:48:29.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.343+0000 7f924ec00640 1 -- 172.21.15.102:0/844585141 shutdown_connections 2024-03-23T03:48:29.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.343+0000 7f924ec00640 1 --2- 172.21.15.102:0/844585141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92480d1c30 0x7f92480cf310 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:29.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.343+0000 7f924ec00640 1 --2- 172.21.15.102:0/844585141 >> v2:172.21.15.173:3300/0 conn(0x7f924815cab0 0x7f92480b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:29.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.343+0000 7f924ec00640 1 -- 172.21.15.102:0/844585141 >> 172.21.15.102:0/844585141 conn(0x7f924806ae90 msgr2=0x7f924806b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:29.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.343+0000 7f924ec00640 1 -- 172.21.15.102:0/844585141 shutdown_connections 2024-03-23T03:48:29.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.343+0000 7f924ec00640 1 -- 172.21.15.102:0/844585141 wait complete. 2024-03-23T03:48:29.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.345+0000 7f924ec00640 1 Processor -- start 2024-03-23T03:48:29.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.345+0000 7f924ec00640 1 -- start start 2024-03-23T03:48:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.346+0000 7f924ec00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92480ceeb0 0x7f9248150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.346+0000 7f924ec00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92480d1c30 0x7f92480b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.346+0000 7f924ec00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924815cab0 0x7f92480b3520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.346+0000 7f924ec00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92480b3a60 con 0x7f924815cab0 2024-03-23T03:48:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.346+0000 7f924ec00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92480b48a0 con 0x7f92480d1c30 2024-03-23T03:48:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.346+0000 7f924ec00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92480b3c90 con 0x7f92480ceeb0 2024-03-23T03:48:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.346+0000 7f924e200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924815cab0 0x7f92480b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.347+0000 7f924e200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924815cab0 0x7f92480b3520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45052/0 (socket says 172.21.15.102:45052) 2024-03-23T03:48:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.347+0000 7f924e200640 1 -- 172.21.15.102:0/1736060061 learned_addr learned my addr 172.21.15.102:0/1736060061 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.347+0000 7f924d800640 1 --2- 172.21.15.102:0/1736060061 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92480ceeb0 0x7f9248150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:29.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.347+0000 7f924ce00640 1 --2- 172.21.15.102:0/1736060061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92480d1c30 0x7f92480b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.347+0000 7f924e200640 1 -- 172.21.15.102:0/1736060061 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92480ceeb0 msgr2=0x7f9248150480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.347+0000 7f924e200640 1 --2- 172.21.15.102:0/1736060061 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92480ceeb0 0x7f9248150480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.347+0000 7f924e200640 1 -- 172.21.15.102:0/1736060061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92480d1c30 msgr2=0x7f92480b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.347+0000 7f924e200640 1 --2- 172.21.15.102:0/1736060061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92480d1c30 0x7f92480b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.347+0000 7f924e200640 1 -- 172.21.15.102:0/1736060061 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9240009b30 con 0x7f924815cab0 2024-03-23T03:48:29.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.347+0000 7f924e200640 1 --2- 172.21.15.102:0/1736060061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924815cab0 0x7f92480b3520 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9240036040 tx=0x7f9240004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:29.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.348+0000 7f9245600640 1 -- 172.21.15.102:0/1736060061 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9240009510 con 0x7f924815cab0 2024-03-23T03:48:29.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.348+0000 7f9245600640 1 -- 172.21.15.102:0/1736060061 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924003e450 con 0x7f924815cab0 2024-03-23T03:48:29.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.348+0000 7f924ec00640 1 -- 172.21.15.102:0/1736060061 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92480b3f70 con 0x7f924815cab0 2024-03-23T03:48:29.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.348+0000 7f9245600640 1 -- 172.21.15.102:0/1736060061 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924003e7e0 con 0x7f924815cab0 2024-03-23T03:48:29.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.348+0000 7f924ec00640 1 -- 172.21.15.102:0/1736060061 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92480b4530 con 0x7f924815cab0 2024-03-23T03:48:29.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.349+0000 7f924ec00640 1 -- 172.21.15.102:0/1736060061 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92480b4a50 con 0x7f924815cab0 2024-03-23T03:48:29.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.350+0000 7f9245600640 1 -- 172.21.15.102:0/1736060061 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f924003e940 con 0x7f924815cab0 2024-03-23T03:48:29.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.351+0000 7f9245600640 1 --2- 172.21.15.102:0/1736060061 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f922806ac00 0x7f922806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:29.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.351+0000 7f9245600640 1 -- 172.21.15.102:0/1736060061 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f92400ac540 con 0x7f924815cab0 2024-03-23T03:48:29.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.351+0000 7f924d800640 1 --2- 172.21.15.102:0/1736060061 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f922806ac00 0x7f922806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:29.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.352+0000 7f924d800640 1 --2- 172.21.15.102:0/1736060061 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f922806ac00 0x7f922806d0c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f923c006fd0 tx=0x7f923c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:29.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.355+0000 7f9245600640 1 -- 172.21.15.102:0/1736060061 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f92480b4a50 con 0x7f924815cab0 2024-03-23T03:48:29.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.586+0000 7f924ec00640 1 -- 172.21.15.102:0/1736060061 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92480b4a10 con 0x7f924815cab0 2024-03-23T03:48:29.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.587+0000 7f9245600640 1 -- 172.21.15.102:0/1736060061 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f92400775d0 con 0x7f924815cab0 2024-03-23T03:48:29.588 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:29.588 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6834/1537100581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1537100581},{"type":"v1","addr":"172.21.15.173:6834","nonce":1537100581}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4655,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3206638952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":3206638952},{"type":"v1","addr":"172.21.15.173:6835","nonce":3206638952}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4852,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4005240382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4005240382},{"type":"v1","addr":"172.21.15.102:6837","nonce":4005240382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:48:23.418926+0000","modified":"2024-03-23T03:48:26.657662+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.102:6836/1373501465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":1373501465},{"type":"v1","addr":"172.21.15.102:6836","nonce":1373501465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-23T03:48:29.588 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 16 2024-03-23T03:48:29.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.589+0000 7f924ec00640 1 -- 172.21.15.102:0/1736060061 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f922806ac00 msgr2=0x7f922806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:29.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.589+0000 7f924ec00640 1 --2- 172.21.15.102:0/1736060061 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f922806ac00 0x7f922806d0c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f923c006fd0 tx=0x7f923c008040 comp rx=0 tx=0).stop 2024-03-23T03:48:29.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.589+0000 7f924ec00640 1 -- 172.21.15.102:0/1736060061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924815cab0 msgr2=0x7f92480b3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:29.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.589+0000 7f924ec00640 1 --2- 172.21.15.102:0/1736060061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924815cab0 0x7f92480b3520 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9240036040 tx=0x7f9240004a40 comp rx=0 tx=0).stop 2024-03-23T03:48:29.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.590+0000 7f924ec00640 1 -- 172.21.15.102:0/1736060061 shutdown_connections 2024-03-23T03:48:29.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.590+0000 7f924ec00640 1 --2- 172.21.15.102:0/1736060061 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f922806ac00 0x7f922806d0c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:29.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.590+0000 7f924ec00640 1 --2- 172.21.15.102:0/1736060061 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f924815cab0 0x7f92480b3520 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:29.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.590+0000 7f924ec00640 1 --2- 172.21.15.102:0/1736060061 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92480d1c30 0x7f92480b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:29.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.590+0000 7f924ec00640 1 --2- 172.21.15.102:0/1736060061 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92480ceeb0 0x7f9248150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:29.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.590+0000 7f924ec00640 1 -- 172.21.15.102:0/1736060061 >> 172.21.15.102:0/1736060061 conn(0x7f924806ae90 msgr2=0x7f92480b7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:29.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.590+0000 7f924ec00640 1 -- 172.21.15.102:0/1736060061 shutdown_connections 2024-03-23T03:48:29.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:29.590+0000 7f924ec00640 1 -- 172.21.15.102:0/1736060061 wait complete. 2024-03-23T03:48:30.009 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T03:48:23.418926+0000', 'modified': '2024-03-23T03:48:26.657662+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5262}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5262': {'gid': 5262, 'name': 'a', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.102:6836/1373501465', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6834', 'nonce': 1373501465}, {'type': 'v1', 'addr': '172.21.15.102:6836', 'nonce': 1373501465}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T03:48:30.010 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T03:48:30.010 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-23T03:48:30.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.223+0000 7f178d000640 1 Processor -- start 2024-03-23T03:48:30.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.223+0000 7f178d000640 1 -- start start 2024-03-23T03:48:30.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.225+0000 7f178d000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f17880cd6b0 0x7f17880b8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.225+0000 7f178d000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f178815cd90 0x7f178806b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.225+0000 7f178d000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178806b5a0 con 0x7f17880cda90 2024-03-23T03:48:30.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.225+0000 7f178d000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f178806c0e0 con 0x7f17880cd6b0 2024-03-23T03:48:30.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.225+0000 7f178d000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178806b840 con 0x7f178815cd90 2024-03-23T03:48:30.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.225+0000 7f1785e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f17880cd6b0 0x7f17880b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.225+0000 7f1785e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f17880cd6b0 0x7f17880b8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45068/0 (socket says 172.21.15.102:45068) 2024-03-23T03:48:30.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.225+0000 7f1785e00640 1 -- 172.21.15.102:0/467329858 learned_addr learned my addr 172.21.15.102:0/467329858 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:30.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.225+0000 7f1786800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f17880cda90 0x7f17881606c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37832/0 (socket says 172.21.15.102:37832) 2024-03-23T03:48:30.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.226+0000 7f1785400640 1 --2- 172.21.15.102:0/467329858 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f178815cd90 0x7f178806b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.226+0000 7f1785e00640 1 -- 172.21.15.102:0/467329858 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f178815cd90 msgr2=0x7f178806b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:30.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.226+0000 7f1785e00640 1 --2- 172.21.15.102:0/467329858 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f178815cd90 0x7f178806b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.226+0000 7f1785e00640 1 -- 172.21.15.102:0/467329858 >> v1:172.21.15.102:6789/0 conn(0x7f17880cda90 legacy=0x7f17881606c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:30.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.226+0000 7f1785e00640 1 -- 172.21.15.102:0/467329858 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178806bb20 con 0x7f17880cd6b0 2024-03-23T03:48:30.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.226+0000 7f1784a00640 1 -- 172.21.15.102:0/467329858 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625936467 0 0) 0x7f178806b5a0 con 0x7f17880cda90 2024-03-23T03:48:30.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.227+0000 7f1785e00640 1 --2- 172.21.15.102:0/467329858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17880cd6b0 0x7f17880b8490 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1778009e50 tx=0x7f177802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68bd9ddd4e996c90 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:30.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.227+0000 7f1784a00640 1 -- 172.21.15.102:0/467329858 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177802cb00 con 0x7f17880cd6b0 2024-03-23T03:48:30.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.227+0000 7f1784a00640 1 -- 172.21.15.102:0/467329858 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177802cc60 con 0x7f17880cd6b0 2024-03-23T03:48:30.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.227+0000 7f1784a00640 1 -- 172.21.15.102:0/467329858 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17780343f0 con 0x7f17880cd6b0 2024-03-23T03:48:30.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.228+0000 7f178d000640 1 -- 172.21.15.102:0/467329858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17880cd6b0 msgr2=0x7f17880b8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:30.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.228+0000 7f178d000640 1 --2- 172.21.15.102:0/467329858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17880cd6b0 0x7f17880b8490 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1778009e50 tx=0x7f177802bf60 comp rx=0 tx=0).stop 2024-03-23T03:48:30.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.228+0000 7f178d000640 1 -- 172.21.15.102:0/467329858 shutdown_connections 2024-03-23T03:48:30.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.228+0000 7f178d000640 1 --2- 172.21.15.102:0/467329858 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f178815cd90 0x7f178806b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.228+0000 7f178d000640 1 --2- 172.21.15.102:0/467329858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17880cd6b0 0x7f17880b8490 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.228+0000 7f178d000640 1 -- 172.21.15.102:0/467329858 >> 172.21.15.102:0/467329858 conn(0x7f17880cf780 msgr2=0x7f17880cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:30.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.228+0000 7f178d000640 1 -- 172.21.15.102:0/467329858 shutdown_connections 2024-03-23T03:48:30.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.228+0000 7f178d000640 1 -- 172.21.15.102:0/467329858 wait complete. 2024-03-23T03:48:30.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.230+0000 7f178d000640 1 Processor -- start 2024-03-23T03:48:30.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.230+0000 7f178d000640 1 -- start start 2024-03-23T03:48:30.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.231+0000 7f178d000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17880cd6b0 0x7f178815ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.231+0000 7f178d000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17880cda90 0x7f1788157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.231+0000 7f178d000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f178815cd90 0x7f17880cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.231+0000 7f178d000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178815eac0 con 0x7f17880cda90 2024-03-23T03:48:30.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.231+0000 7f178d000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178815dbd0 con 0x7f17880cd6b0 2024-03-23T03:48:30.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.231+0000 7f178d000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178815deb0 con 0x7f178815cd90 2024-03-23T03:48:30.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.231+0000 7f1785400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17880cda90 0x7f1788157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.231+0000 7f1785400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17880cda90 0x7f1788157dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45084/0 (socket says 172.21.15.102:45084) 2024-03-23T03:48:30.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.231+0000 7f1785400640 1 -- 172.21.15.102:0/841421016 learned_addr learned my addr 172.21.15.102:0/841421016 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.232+0000 7f1785400640 1 -- 172.21.15.102:0/841421016 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f178815cd90 msgr2=0x7f17880cb980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.232+0000 7f1785400640 1 --2- 172.21.15.102:0/841421016 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f178815cd90 0x7f17880cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.232+0000 7f1785400640 1 -- 172.21.15.102:0/841421016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17880cd6b0 msgr2=0x7f178815ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:30.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.232+0000 7f1785400640 1 --2- 172.21.15.102:0/841421016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17880cd6b0 0x7f178815ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.232+0000 7f1785400640 1 -- 172.21.15.102:0/841421016 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1778009b30 con 0x7f17880cda90 2024-03-23T03:48:30.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.232+0000 7f1785400640 1 --2- 172.21.15.102:0/841421016 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17880cda90 0x7f1788157dd0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f177000d5d0 tx=0x7f177000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:30.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.232+0000 7f177e000640 1 -- 172.21.15.102:0/841421016 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17700125f0 con 0x7f17880cda90 2024-03-23T03:48:30.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.232+0000 7f177e000640 1 -- 172.21.15.102:0/841421016 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1770012750 con 0x7f17880cda90 2024-03-23T03:48:30.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.232+0000 7f177e000640 1 -- 172.21.15.102:0/841421016 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1770012ae0 con 0x7f17880cda90 2024-03-23T03:48:30.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.232+0000 7f178d000640 1 -- 172.21.15.102:0/841421016 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178815e190 con 0x7f17880cda90 2024-03-23T03:48:30.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.232+0000 7f178d000640 1 -- 172.21.15.102:0/841421016 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f178815e750 con 0x7f17880cda90 2024-03-23T03:48:30.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.235+0000 7f177e000640 1 -- 172.21.15.102:0/841421016 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1770012c40 con 0x7f17880cda90 2024-03-23T03:48:30.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.235+0000 7f177e000640 1 --2- 172.21.15.102:0/841421016 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f176406ac00 0x7f176406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.236+0000 7f177e000640 1 -- 172.21.15.102:0/841421016 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f17700899b0 con 0x7f17880cda90 2024-03-23T03:48:30.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.236+0000 7f1785e00640 1 --2- 172.21.15.102:0/841421016 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f176406ac00 0x7f176406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.236+0000 7f178fc25740 1 Processor -- start 2024-03-23T03:48:30.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.237+0000 7f1785e00640 1 --2- 172.21.15.102:0/841421016 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f176406ac00 0x7f176406d0c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1778035000 tx=0x7f177802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:30.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.237+0000 7f178fc25740 1 -- start start 2024-03-23T03:48:30.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.237+0000 7f178fc25740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd47cd00 0x55bcbd44fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.237+0000 7f178fc25740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bcbd47d0e0 0x55bcbd47fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.237+0000 7f178fc25740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd4501e0 0x55bcbd47ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.238+0000 7f178fc25740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbd42b2c0 con 0x55bcbd4501e0 2024-03-23T03:48:30.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.238+0000 7f178fc25740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbd3c08f0 con 0x55bcbd47cd00 2024-03-23T03:48:30.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.238+0000 7f178fc25740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bcbd3be080 con 0x55bcbd47d0e0 2024-03-23T03:48:30.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.238+0000 7f1786800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd4501e0 0x55bcbd47ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.238+0000 7f1786800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd4501e0 0x55bcbd47ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45100/0 (socket says 172.21.15.102:45100) 2024-03-23T03:48:30.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.238+0000 7f1786800640 1 -- 172.21.15.102:0/3464139035 learned_addr learned my addr 172.21.15.102:0/3464139035 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:30.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.238+0000 7f1786800640 1 -- 172.21.15.102:0/3464139035 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bcbd47d0e0 msgr2=0x55bcbd47fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:30.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.238+0000 7f1786800640 1 --2- 172.21.15.102:0/3464139035 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bcbd47d0e0 0x55bcbd47fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.238+0000 7f1785e00640 1 --2- 172.21.15.102:0/3464139035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd47cd00 0x55bcbd44fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.238+0000 7f1786800640 1 -- 172.21.15.102:0/3464139035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd47cd00 msgr2=0x55bcbd44fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.238+0000 7f1786800640 1 --2- 172.21.15.102:0/3464139035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd47cd00 0x55bcbd44fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.238+0000 7f1786800640 1 -- 172.21.15.102:0/3464139035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcbd384a10 con 0x55bcbd4501e0 2024-03-23T03:48:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.239+0000 7f1786800640 1 --2- 172.21.15.102:0/3464139035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd4501e0 0x55bcbd47ffd0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1780001680 tx=0x7f1780001b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=675af44ceb4fb785 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.239+0000 7f1776a00640 1 -- 172.21.15.102:0/3464139035 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17800074e0 con 0x55bcbd4501e0 2024-03-23T03:48:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.239+0000 7f1776a00640 1 -- 172.21.15.102:0/3464139035 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1780007640 con 0x55bcbd4501e0 2024-03-23T03:48:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.239+0000 7f1776a00640 1 -- 172.21.15.102:0/3464139035 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780004a00 con 0x55bcbd4501e0 2024-03-23T03:48:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.240+0000 7f178fc25740 1 -- 172.21.15.102:0/3464139035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd4501e0 msgr2=0x55bcbd47ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.240+0000 7f178fc25740 1 --2- 172.21.15.102:0/3464139035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd4501e0 0x55bcbd47ffd0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f1780001680 tx=0x7f1780001b40 comp rx=0 tx=0).stop 2024-03-23T03:48:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.240+0000 7f178fc25740 1 -- 172.21.15.102:0/3464139035 shutdown_connections 2024-03-23T03:48:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.240+0000 7f178fc25740 1 --2- 172.21.15.102:0/3464139035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd4501e0 0x55bcbd47ffd0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.240+0000 7f178fc25740 1 --2- 172.21.15.102:0/3464139035 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bcbd47d0e0 0x55bcbd47fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.240+0000 7f178fc25740 1 --2- 172.21.15.102:0/3464139035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd47cd00 0x55bcbd44fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.240+0000 7f178fc25740 1 -- 172.21.15.102:0/3464139035 >> 172.21.15.102:0/3464139035 conn(0x55bcbd451f10 msgr2=0x55bcbd49aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:30.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.240+0000 7f178fc25740 1 -- 172.21.15.102:0/3464139035 shutdown_connections 2024-03-23T03:48:30.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.240+0000 7f178fc25740 1 -- 172.21.15.102:0/3464139035 wait complete. 2024-03-23T03:48:30.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.241+0000 7f178fc25740 1 -- start start 2024-03-23T03:48:30.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.241+0000 7f178fc25740 1 Processor -- start 2024-03-23T03:48:30.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.242+0000 7f178fc25740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd47cd00 0x55bcbd46cf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.242+0000 7f178fc25740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd47d0e0 0x55bcbd483840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.242+0000 7f178fc25740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bcbd4aa5d0 0x55bcbd4aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.242+0000 7f178fc25740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbd3b9970 con 0x55bcbd47d0e0 2024-03-23T03:48:30.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.242+0000 7f178fc25740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbd35b0a0 con 0x55bcbd47cd00 2024-03-23T03:48:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.242+0000 7f178fc25740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bcbd35ad40 con 0x55bcbd4aa5d0 2024-03-23T03:48:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.242+0000 7f178fc25740 10 client.0 _ll_register_callbacks cb 0x55bcbd3b2fe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T03:48:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.242+0000 7f1785400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd47d0e0 0x55bcbd483840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.242+0000 7f1785400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd47d0e0 0x55bcbd483840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45110/0 (socket says 172.21.15.102:45110) 2024-03-23T03:48:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.242+0000 7f1785400640 1 -- 172.21.15.102:0/2916085105 learned_addr learned my addr 172.21.15.102:0/2916085105 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.242+0000 7f1785400640 1 -- 172.21.15.102:0/2916085105 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bcbd4aa5d0 msgr2=0x55bcbd4aaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.242+0000 7f1785e00640 1 --2- 172.21.15.102:0/2916085105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd47cd00 0x55bcbd46cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.242+0000 7f1785400640 1 --2- 172.21.15.102:0/2916085105 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bcbd4aa5d0 0x55bcbd4aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.242+0000 7f1785400640 1 -- 172.21.15.102:0/2916085105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd47cd00 msgr2=0x55bcbd46cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.242+0000 7f1785400640 1 --2- 172.21.15.102:0/2916085105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd47cd00 0x55bcbd46cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.242+0000 7f1785400640 1 -- 172.21.15.102:0/2916085105 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17800012b0 con 0x55bcbd47d0e0 2024-03-23T03:48:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.243+0000 7f1785400640 1 --2- 172.21.15.102:0/2916085105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd47d0e0 0x55bcbd483840 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1770060540 tx=0x7f1770058e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b65d7b068f4620 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.243+0000 7f176b400640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-23T03:48:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.243+0000 7f176b400640 1 -- 172.21.15.102:0/2916085105 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1770060a70 con 0x55bcbd47d0e0 2024-03-23T03:48:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.243+0000 7f176b400640 1 -- 172.21.15.102:0/2916085105 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f177005c170 con 0x55bcbd47d0e0 2024-03-23T03:48:30.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.243+0000 7f178fc25740 1 -- 172.21.15.102:0/2916085105 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bcbd3e3b10 con 0x55bcbd47d0e0 2024-03-23T03:48:30.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.243+0000 7f176b400640 1 -- 172.21.15.102:0/2916085105 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1770054d90 con 0x55bcbd47d0e0 2024-03-23T03:48:30.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.243+0000 7f178fc25740 10 client.5445 fetch_fsmap learned FSMap version 16 2024-03-23T03:48:30.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.243+0000 7f178fc25740 1 -- 172.21.15.102:0/2916085105 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55bcbd3b7ff0 con 0x55bcbd47d0e0 2024-03-23T03:48:30.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.243+0000 7f176b400640 1 -- 172.21.15.102:0/2916085105 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1770085030 con 0x55bcbd47d0e0 2024-03-23T03:48:30.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.243+0000 7f176b400640 1 -- 172.21.15.102:0/2916085105 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 16) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1770057b70 con 0x55bcbd47d0e0 2024-03-23T03:48:30.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.244+0000 7f178fc25740 10 client.5445 fetch_fsmap finished waiting for FSMap version 16 2024-03-23T03:48:30.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.244+0000 7f178fc25740 10 client.5445 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T03:48:30.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.244+0000 7f178fc25740 20 client.5445 populate_metadata read hostname 'smithi102' 2024-03-23T03:48:30.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.244+0000 7f178fc25740 1 --2- 172.21.15.102:0/2916085105 >> [v2:172.21.15.102:6834/1373501465,v1:172.21.15.102:6836/1373501465] conn(0x55bcbd4b5d30 0x55bcbd4b8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.244+0000 7f178fc25740 4 client.5445 mds_command: new command op to 5262 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T03:48:30.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.244+0000 7f178fc25740 1 -- 172.21.15.102:0/2916085105 --> [v2:172.21.15.102:6834/1373501465,v1:172.21.15.102:6836/1373501465] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bcbd0ad3c0 con 0x55bcbd4b5d30 2024-03-23T03:48:30.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.244+0000 7f1785e00640 1 --2- 172.21.15.102:0/2916085105 >> [v2:172.21.15.102:6834/1373501465,v1:172.21.15.102:6836/1373501465] conn(0x55bcbd4b5d30 0x55bcbd4b8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.244+0000 7f1785e00640 1 --2- 172.21.15.102:0/2916085105 >> [v2:172.21.15.102:6834/1373501465,v1:172.21.15.102:6836/1373501465] conn(0x55bcbd4b5d30 0x55bcbd4b8120 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68b5ff60664be70c server_cookie=c9d7aa3e34784f73 in_seq=0 out_seq=0 2024-03-23T03:48:30.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.244+0000 7f176b400640 10 client.5445 ms_handle_connect on v2:172.21.15.102:6834/1373501465 2024-03-23T03:48:30.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.245+0000 7f176b400640 1 -- 172.21.15.102:0/2916085105 <== mds.0 v2:172.21.15.102:6834/1373501465 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55bcbd0ad3c0 con 0x55bcbd4b5d30 2024-03-23T03:48:30.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.245+0000 7f176b400640 10 client.5445 handle_command_reply: tid=0 2024-03-23T03:48:30.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.245+0000 7f178fc25740 1 client.5445 shutdown 2024-03-23T03:48:30.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.245+0000 7f178fc25740 1 -- 172.21.15.102:0/2916085105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd47d0e0 msgr2=0x55bcbd483840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:30.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.245+0000 7f178fc25740 1 --2- 172.21.15.102:0/2916085105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd47d0e0 0x55bcbd483840 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1770060540 tx=0x7f1770058e40 comp rx=0 tx=0).stop 2024-03-23T03:48:30.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.245+0000 7f178fc25740 1 -- 172.21.15.102:0/2916085105 shutdown_connections 2024-03-23T03:48:30.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.245+0000 7f178fc25740 1 --2- 172.21.15.102:0/2916085105 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bcbd4aa5d0 0x55bcbd4aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.245+0000 7f178fc25740 1 --2- 172.21.15.102:0/2916085105 >> [v2:172.21.15.102:6834/1373501465,v1:172.21.15.102:6836/1373501465] conn(0x55bcbd4b5d30 0x55bcbd4b8120 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.245+0000 7f178fc25740 1 --2- 172.21.15.102:0/2916085105 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd47d0e0 0x55bcbd483840 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.245+0000 7f178fc25740 1 --2- 172.21.15.102:0/2916085105 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd47cd00 0x55bcbd46cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.245+0000 7f178fc25740 1 -- 172.21.15.102:0/2916085105 >> 172.21.15.102:0/2916085105 conn(0x55bcbd4501e0 msgr2=0x55bcbd49dfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:30.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.245+0000 7f176b400640 0 client.5445 ms_handle_reset on v2:172.21.15.102:6834/1373501465 2024-03-23T03:48:30.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.246+0000 7f178fc25740 1 -- 172.21.15.102:0/2916085105 shutdown_connections 2024-03-23T03:48:30.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.246+0000 7f178fc25740 1 -- 172.21.15.102:0/2916085105 wait complete. 2024-03-23T03:48:30.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.246+0000 7f178fc25740 20 client.5445 trim_cache size 0 max 16384 2024-03-23T03:48:30.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.262+0000 7f178fc25740 1 Processor -- start 2024-03-23T03:48:30.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.262+0000 7f178fc25740 1 -- start start 2024-03-23T03:48:30.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.263+0000 7f178fc25740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bcbd47cd00 0x55bcbd485050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.263+0000 7f178fc25740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd47d0e0 0x55bcbd49e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.263+0000 7f178fc25740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd4aa5d0 0x55bcbd481060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.263+0000 7f178fc25740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbd120f20 con 0x55bcbd4aa5d0 2024-03-23T03:48:30.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.263+0000 7f178fc25740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbd49ea90 con 0x55bcbd47d0e0 2024-03-23T03:48:30.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.263+0000 7f178fc25740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bcbd49d610 con 0x55bcbd47cd00 2024-03-23T03:48:30.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.263+0000 7f1786800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd4aa5d0 0x55bcbd481060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.263+0000 7f1786800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd4aa5d0 0x55bcbd481060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45112/0 (socket says 172.21.15.102:45112) 2024-03-23T03:48:30.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.263+0000 7f1786800640 1 -- 172.21.15.102:0/1546735294 learned_addr learned my addr 172.21.15.102:0/1546735294 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:30.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.263+0000 7f1786800640 1 -- 172.21.15.102:0/1546735294 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bcbd47cd00 msgr2=0x55bcbd485050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:30.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.263+0000 7f1786800640 1 --2- 172.21.15.102:0/1546735294 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bcbd47cd00 0x55bcbd485050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.263+0000 7f1786800640 1 -- 172.21.15.102:0/1546735294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd47d0e0 msgr2=0x55bcbd49e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:30.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.263+0000 7f1786800640 1 --2- 172.21.15.102:0/1546735294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd47d0e0 0x55bcbd49e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.263+0000 7f1786800640 1 -- 172.21.15.102:0/1546735294 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcbd3ed7c0 con 0x55bcbd4aa5d0 2024-03-23T03:48:30.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.264+0000 7f1786800640 1 --2- 172.21.15.102:0/1546735294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd4aa5d0 0x55bcbd481060 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f178000c4b0 tx=0x7f178000f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7598ff75c8f4fafb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:30.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.264+0000 7f1777400640 1 -- 172.21.15.102:0/1546735294 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17800074e0 con 0x55bcbd4aa5d0 2024-03-23T03:48:30.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.264+0000 7f1777400640 1 -- 172.21.15.102:0/1546735294 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17800047e0 con 0x55bcbd4aa5d0 2024-03-23T03:48:30.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.264+0000 7f1777400640 1 -- 172.21.15.102:0/1546735294 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1780004b90 con 0x55bcbd4aa5d0 2024-03-23T03:48:30.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.264+0000 7f178fc25740 1 -- 172.21.15.102:0/1546735294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd4aa5d0 msgr2=0x55bcbd481060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:30.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.264+0000 7f178fc25740 1 --2- 172.21.15.102:0/1546735294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd4aa5d0 0x55bcbd481060 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f178000c4b0 tx=0x7f178000f000 comp rx=0 tx=0).stop 2024-03-23T03:48:30.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.265+0000 7f178fc25740 1 -- 172.21.15.102:0/1546735294 shutdown_connections 2024-03-23T03:48:30.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.265+0000 7f178fc25740 1 --2- 172.21.15.102:0/1546735294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd4aa5d0 0x55bcbd481060 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.265+0000 7f178fc25740 1 --2- 172.21.15.102:0/1546735294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd47d0e0 0x55bcbd49e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.265+0000 7f178fc25740 1 --2- 172.21.15.102:0/1546735294 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bcbd47cd00 0x55bcbd485050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.265+0000 7f178fc25740 1 -- 172.21.15.102:0/1546735294 >> 172.21.15.102:0/1546735294 conn(0x55bcbd4501e0 msgr2=0x55bcbd455bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:30.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.265+0000 7f178fc25740 1 -- 172.21.15.102:0/1546735294 shutdown_connections 2024-03-23T03:48:30.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.265+0000 7f178fc25740 1 -- 172.21.15.102:0/1546735294 wait complete. 2024-03-23T03:48:30.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.265+0000 7f178fc25740 1 -- start start 2024-03-23T03:48:30.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.266+0000 7f178fc25740 1 Processor -- start 2024-03-23T03:48:30.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.267+0000 7f178fc25740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd47cd00 0x55bcbd509250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.267+0000 7f178fc25740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bcbd47d0e0 0x55bcbd484930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.267+0000 7f178fc25740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd4aa5d0 0x55bcbd484e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.267+0000 7f178fc25740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbd49a4e0 con 0x55bcbd47cd00 2024-03-23T03:48:30.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.267+0000 7f178fc25740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcbd454e70 con 0x55bcbd4aa5d0 2024-03-23T03:48:30.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.267+0000 7f178fc25740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bcbd49a290 con 0x55bcbd47d0e0 2024-03-23T03:48:30.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.267+0000 7f178fc25740 10 client.0 _ll_register_callbacks cb 0x7f1770006e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T03:48:30.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.267+0000 7f1785e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd47cd00 0x55bcbd509250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.267+0000 7f1785e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd47cd00 0x55bcbd509250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45124/0 (socket says 172.21.15.102:45124) 2024-03-23T03:48:30.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.267+0000 7f1785e00640 1 -- 172.21.15.102:0/896991298 learned_addr learned my addr 172.21.15.102:0/896991298 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:30.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.267+0000 7f1785e00640 1 -- 172.21.15.102:0/896991298 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bcbd47d0e0 msgr2=0x55bcbd484930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:30.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.267+0000 7f1785e00640 1 --2- 172.21.15.102:0/896991298 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bcbd47d0e0 0x55bcbd484930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.267+0000 7f1785e00640 1 -- 172.21.15.102:0/896991298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd4aa5d0 msgr2=0x55bcbd484e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:30.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.267+0000 7f1785e00640 1 --2- 172.21.15.102:0/896991298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd4aa5d0 0x55bcbd484e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.267+0000 7f1785e00640 1 -- 172.21.15.102:0/896991298 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bcbd384a10 con 0x55bcbd47cd00 2024-03-23T03:48:30.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.267+0000 7f1785e00640 1 --2- 172.21.15.102:0/896991298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd47cd00 0x55bcbd509250 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1778036040 tx=0x7f1778004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0e93eaffe83f820 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:30.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.267+0000 7f176b400640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-23T03:48:30.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.268+0000 7f176b400640 1 -- 172.21.15.102:0/896991298 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1778045040 con 0x55bcbd47cd00 2024-03-23T03:48:30.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.268+0000 7f176b400640 1 -- 172.21.15.102:0/896991298 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f177804f2c0 con 0x55bcbd47cd00 2024-03-23T03:48:30.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.268+0000 7f178fc25740 1 -- 172.21.15.102:0/896991298 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1780007640 con 0x55bcbd47cd00 2024-03-23T03:48:30.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.268+0000 7f176b400640 1 -- 172.21.15.102:0/896991298 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17780407a0 con 0x55bcbd47cd00 2024-03-23T03:48:30.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.268+0000 7f178fc25740 10 client.5451 fetch_fsmap learned FSMap version 16 2024-03-23T03:48:30.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.268+0000 7f178fc25740 1 -- 172.21.15.102:0/896991298 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x55bcbd35a2b0 con 0x55bcbd47cd00 2024-03-23T03:48:30.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.268+0000 7f176b400640 1 -- 172.21.15.102:0/896991298 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1780007640 con 0x55bcbd47cd00 2024-03-23T03:48:30.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.268+0000 7f176b400640 1 -- 172.21.15.102:0/896991298 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 16) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f177803c070 con 0x55bcbd47cd00 2024-03-23T03:48:30.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.268+0000 7f178fc25740 10 client.5451 fetch_fsmap finished waiting for FSMap version 16 2024-03-23T03:48:30.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.268+0000 7f178fc25740 10 client.5451 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T03:48:30.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.268+0000 7f178fc25740 20 client.5451 populate_metadata read hostname 'smithi102' 2024-03-23T03:48:30.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.268+0000 7f178fc25740 1 --2- 172.21.15.102:0/896991298 >> [v2:172.21.15.102:6834/1373501465,v1:172.21.15.102:6836/1373501465] conn(0x55bcbd44b600 0x55bcbd44ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.268+0000 7f178fc25740 4 client.5451 mds_command: new command op to 5262 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-23T03:48:30.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.268+0000 7f178fc25740 1 -- 172.21.15.102:0/896991298 --> [v2:172.21.15.102:6834/1373501465,v1:172.21.15.102:6836/1373501465] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bcbd0be6c0 con 0x55bcbd44b600 2024-03-23T03:48:30.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.269+0000 7f1785400640 1 --2- 172.21.15.102:0/896991298 >> [v2:172.21.15.102:6834/1373501465,v1:172.21.15.102:6836/1373501465] conn(0x55bcbd44b600 0x55bcbd44ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.269+0000 7f1785400640 1 --2- 172.21.15.102:0/896991298 >> [v2:172.21.15.102:6834/1373501465,v1:172.21.15.102:6836/1373501465] conn(0x55bcbd44b600 0x55bcbd44ba00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=881536deb1380ef2 server_cookie=b89653634822cfcb in_seq=0 out_seq=0 2024-03-23T03:48:30.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.269+0000 7f176b400640 10 client.5451 ms_handle_connect on v2:172.21.15.102:6834/1373501465 2024-03-23T03:48:30.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.269+0000 7f176b400640 1 -- 172.21.15.102:0/896991298 <== mds.0 v2:172.21.15.102:6834/1373501465 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55bcbd0be6c0 con 0x55bcbd44b600 2024-03-23T03:48:30.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.269+0000 7f176b400640 10 client.5451 handle_command_reply: tid=0 2024-03-23T03:48:30.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.269+0000 7f178fc25740 1 client.5451 shutdown 2024-03-23T03:48:30.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.270+0000 7f178fc25740 1 -- 172.21.15.102:0/896991298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd47cd00 msgr2=0x55bcbd509250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:30.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.270+0000 7f178fc25740 1 --2- 172.21.15.102:0/896991298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd47cd00 0x55bcbd509250 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1778036040 tx=0x7f1778004a40 comp rx=0 tx=0).stop 2024-03-23T03:48:30.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.270+0000 7f178fc25740 1 -- 172.21.15.102:0/896991298 shutdown_connections 2024-03-23T03:48:30.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.270+0000 7f178fc25740 1 --2- 172.21.15.102:0/896991298 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55bcbd4aa5d0 0x55bcbd484e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.270+0000 7f178fc25740 1 --2- 172.21.15.102:0/896991298 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55bcbd47d0e0 0x55bcbd484930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.270+0000 7f178fc25740 1 --2- 172.21.15.102:0/896991298 >> [v2:172.21.15.102:6834/1373501465,v1:172.21.15.102:6836/1373501465] conn(0x55bcbd44b600 0x55bcbd44ba00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.270+0000 7f178fc25740 1 --2- 172.21.15.102:0/896991298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bcbd47cd00 0x55bcbd509250 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.270+0000 7f178fc25740 1 -- 172.21.15.102:0/896991298 >> 172.21.15.102:0/896991298 conn(0x55bcbd4501e0 msgr2=0x55bcbd455ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:30.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.270+0000 7f176b400640 0 client.5451 ms_handle_reset on v2:172.21.15.102:6834/1373501465 2024-03-23T03:48:30.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.270+0000 7f178fc25740 1 -- 172.21.15.102:0/896991298 shutdown_connections 2024-03-23T03:48:30.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.270+0000 7f178fc25740 1 -- 172.21.15.102:0/896991298 wait complete. 2024-03-23T03:48:30.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.270+0000 7f178fc25740 20 client.5451 trim_cache size 0 max 16384 2024-03-23T03:48:30.273 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:48:30.274 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "776b2820-05f2-4e50-8d40-18dd76c5f6a0", 2024-03-23T03:48:30.274 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 0, 2024-03-23T03:48:30.274 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 5262, 2024-03-23T03:48:30.274 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-03-23T03:48:30.274 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-03-23T03:48:30.274 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-03-23T03:48:30.274 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 6.8474507899999999, 2024-03-23T03:48:30.274 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 16, 2024-03-23T03:48:30.274 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 41, 2024-03-23T03:48:30.274 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 41, 2024-03-23T03:48:30.274 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 15.117359288999999 2024-03-23T03:48:30.274 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:48:30.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.270+0000 7f178d000640 1 -- 172.21.15.102:0/841421016 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f176406ac00 msgr2=0x7f176406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:30.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.270+0000 7f178d000640 1 --2- 172.21.15.102:0/841421016 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f176406ac00 0x7f176406d0c0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1778035000 tx=0x7f177802e000 comp rx=0 tx=0).stop 2024-03-23T03:48:30.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.270+0000 7f178d000640 1 -- 172.21.15.102:0/841421016 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17880cda90 msgr2=0x7f1788157dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:30.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.270+0000 7f178d000640 1 --2- 172.21.15.102:0/841421016 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17880cda90 0x7f1788157dd0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f177000d5d0 tx=0x7f177000da90 comp rx=0 tx=0).stop 2024-03-23T03:48:30.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.271+0000 7f178d000640 1 -- 172.21.15.102:0/841421016 shutdown_connections 2024-03-23T03:48:30.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.271+0000 7f178d000640 1 --2- 172.21.15.102:0/841421016 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f176406ac00 0x7f176406d0c0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.271+0000 7f178d000640 1 --2- 172.21.15.102:0/841421016 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f178815cd90 0x7f17880cb980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.271+0000 7f178d000640 1 --2- 172.21.15.102:0/841421016 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17880cda90 0x7f1788157dd0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.271+0000 7f178d000640 1 --2- 172.21.15.102:0/841421016 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17880cd6b0 0x7f178815ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.271+0000 7f178d000640 1 -- 172.21.15.102:0/841421016 >> 172.21.15.102:0/841421016 conn(0x7f17880cf780 msgr2=0x7f1788160ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:30.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.271+0000 7f178d000640 1 -- 172.21.15.102:0/841421016 shutdown_connections 2024-03-23T03:48:30.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.271+0000 7f178d000640 1 -- 172.21.15.102:0/841421016 wait complete. 2024-03-23T03:48:30.288 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:48:30.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.487+0000 7f996e200640 1 Processor -- start 2024-03-23T03:48:30.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.487+0000 7f996e200640 1 -- start start 2024-03-23T03:48:30.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.488+0000 7f996e200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f99680adbc0 0x7f99680b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.488+0000 7f996e200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f99680cbab0 0x7f99680b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.488+0000 7f996e200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f996806d390 0x7f996806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.488+0000 7f996e200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f996806dd60 con 0x7f99680cbab0 2024-03-23T03:48:30.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.488+0000 7f996e200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99680b16a0 con 0x7f99680adbc0 2024-03-23T03:48:30.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.488+0000 7f996e200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99680b4700 con 0x7f996806d390 2024-03-23T03:48:30.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.489+0000 7f995ea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f99680cbab0 0x7f99680b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.489+0000 7f995ea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f99680cbab0 0x7f99680b20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45138/0 (socket says 172.21.15.102:45138) 2024-03-23T03:48:30.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.489+0000 7f995ea00640 1 -- 172.21.15.102:0/2871052785 learned_addr learned my addr 172.21.15.102:0/2871052785 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:30.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.489+0000 7f9967400640 1 --2- 172.21.15.102:0/2871052785 >> v2:172.21.15.173:3300/0 conn(0x7f99680adbc0 0x7f99680b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.489+0000 7f995ea00640 1 -- 172.21.15.102:0/2871052785 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f996806d390 msgr2=0x7f996806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:30.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.489+0000 7f995ea00640 1 --2- 172.21.15.102:0/2871052785 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f996806d390 0x7f996806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.489+0000 7f995ea00640 1 -- 172.21.15.102:0/2871052785 >> v2:172.21.15.173:3300/0 conn(0x7f99680adbc0 msgr2=0x7f99680b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:30.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.489+0000 7f995ea00640 1 --2- 172.21.15.102:0/2871052785 >> v2:172.21.15.173:3300/0 conn(0x7f99680adbc0 0x7f99680b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.489+0000 7f995ea00640 1 -- 172.21.15.102:0/2871052785 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99680cbfb0 con 0x7f99680cbab0 2024-03-23T03:48:30.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.490+0000 7f995ea00640 1 --2- 172.21.15.102:0/2871052785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99680cbab0 0x7f99680b20f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9950009870 tx=0x7f995002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253035726b4c02f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:30.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.491+0000 7f9966a00640 1 -- 172.21.15.102:0/2871052785 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99500317c0 con 0x7f99680cbab0 2024-03-23T03:48:30.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.491+0000 7f9966a00640 1 -- 172.21.15.102:0/2871052785 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9950031920 con 0x7f99680cbab0 2024-03-23T03:48:30.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.491+0000 7f9966a00640 1 -- 172.21.15.102:0/2871052785 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9950031cb0 con 0x7f99680cbab0 2024-03-23T03:48:30.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.491+0000 7f996e200640 1 -- 172.21.15.102:0/2871052785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99680cbab0 msgr2=0x7f99680b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:30.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.491+0000 7f996e200640 1 --2- 172.21.15.102:0/2871052785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99680cbab0 0x7f99680b20f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f9950009870 tx=0x7f995002b860 comp rx=0 tx=0).stop 2024-03-23T03:48:30.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.491+0000 7f996e200640 1 -- 172.21.15.102:0/2871052785 shutdown_connections 2024-03-23T03:48:30.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.491+0000 7f996e200640 1 --2- 172.21.15.102:0/2871052785 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f996806d390 0x7f996806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.491+0000 7f996e200640 1 --2- 172.21.15.102:0/2871052785 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99680cbab0 0x7f99680b20f0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.491+0000 7f996e200640 1 --2- 172.21.15.102:0/2871052785 >> v2:172.21.15.173:3300/0 conn(0x7f99680adbc0 0x7f99680b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.491+0000 7f996e200640 1 -- 172.21.15.102:0/2871052785 >> 172.21.15.102:0/2871052785 conn(0x7f996806a3d0 msgr2=0x7f99680b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:30.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.492+0000 7f996e200640 1 -- 172.21.15.102:0/2871052785 shutdown_connections 2024-03-23T03:48:30.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.492+0000 7f996e200640 1 -- 172.21.15.102:0/2871052785 wait complete. 2024-03-23T03:48:30.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.493+0000 7f996e200640 1 Processor -- start 2024-03-23T03:48:30.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.493+0000 7f996e200640 1 -- start start 2024-03-23T03:48:30.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.494+0000 7f996e200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f996806d390 0x7f99681f3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.494+0000 7f996e200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99680adbc0 0x7f9968159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.494+0000 7f996e200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99680cbab0 0x7f99680ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.494+0000 7f996e200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99680ac820 con 0x7f99680adbc0 2024-03-23T03:48:30.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.494+0000 7f996e200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99681f3880 con 0x7f99680cbab0 2024-03-23T03:48:30.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.494+0000 7f996e200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99681f2fe0 con 0x7f996806d390 2024-03-23T03:48:30.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.495+0000 7f995ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99680adbc0 0x7f9968159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.495+0000 7f995ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99680adbc0 0x7f9968159320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45144/0 (socket says 172.21.15.102:45144) 2024-03-23T03:48:30.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.495+0000 7f995ea00640 1 -- 172.21.15.102:0/2967943723 learned_addr learned my addr 172.21.15.102:0/2967943723 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:30.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.495+0000 7f9967e00640 1 --2- 172.21.15.102:0/2967943723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99680cbab0 0x7f99680ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.495+0000 7f9967400640 1 --2- 172.21.15.102:0/2967943723 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f996806d390 0x7f99681f3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.495+0000 7f995ea00640 1 -- 172.21.15.102:0/2967943723 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f996806d390 msgr2=0x7f99681f3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:30.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.495+0000 7f995ea00640 1 --2- 172.21.15.102:0/2967943723 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f996806d390 0x7f99681f3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.495+0000 7f995ea00640 1 -- 172.21.15.102:0/2967943723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99680cbab0 msgr2=0x7f99680ac2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:30.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.495+0000 7f995ea00640 1 --2- 172.21.15.102:0/2967943723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99680cbab0 0x7f99680ac2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.495+0000 7f995ea00640 1 -- 172.21.15.102:0/2967943723 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99500094d0 con 0x7f99680adbc0 2024-03-23T03:48:30.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.495+0000 7f9967400640 1 --2- 172.21.15.102:0/2967943723 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f996806d390 0x7f99681f3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:30.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.496+0000 7f995ea00640 1 --2- 172.21.15.102:0/2967943723 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99680adbc0 0x7f9968159320 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f995002b8a0 tx=0x7f99500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:30.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.496+0000 7f995fe00640 1 -- 172.21.15.102:0/2967943723 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99500329a0 con 0x7f99680adbc0 2024-03-23T03:48:30.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.496+0000 7f995fe00640 1 -- 172.21.15.102:0/2967943723 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9950032b40 con 0x7f99680adbc0 2024-03-23T03:48:30.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.496+0000 7f996e200640 1 -- 172.21.15.102:0/2967943723 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99681f32c0 con 0x7f99680adbc0 2024-03-23T03:48:30.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.496+0000 7f995fe00640 1 -- 172.21.15.102:0/2967943723 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99500453f0 con 0x7f99680adbc0 2024-03-23T03:48:30.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.496+0000 7f996e200640 1 -- 172.21.15.102:0/2967943723 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996815a0d0 con 0x7f99680adbc0 2024-03-23T03:48:30.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.497+0000 7f996e200640 1 -- 172.21.15.102:0/2967943723 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9968159f30 con 0x7f99680adbc0 2024-03-23T03:48:30.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.498+0000 7f995fe00640 1 -- 172.21.15.102:0/2967943723 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f995004e450 con 0x7f99680adbc0 2024-03-23T03:48:30.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.499+0000 7f995fe00640 1 --2- 172.21.15.102:0/2967943723 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f994406ac30 0x7f994406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:30.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.499+0000 7f995fe00640 1 -- 172.21.15.102:0/2967943723 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f99500ab850 con 0x7f99680adbc0 2024-03-23T03:48:30.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.499+0000 7f9967400640 1 --2- 172.21.15.102:0/2967943723 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f994406ac30 0x7f994406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:30.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.500+0000 7f9967400640 1 --2- 172.21.15.102:0/2967943723 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f994406ac30 0x7f994406d0f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f996815e370 tx=0x7f9958008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:30.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.504+0000 7f995fe00640 1 -- 172.21.15.102:0/2967943723 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9968159f30 con 0x7f99680adbc0 2024-03-23T03:48:30.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.739+0000 7f996e200640 1 -- 172.21.15.102:0/2967943723 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9968159f30 con 0x7f99680adbc0 2024-03-23T03:48:30.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.740+0000 7f995fe00640 1 -- 172.21.15.102:0/2967943723 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9968159f30 con 0x7f99680adbc0 2024-03-23T03:48:30.740 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T03:48:30.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.742+0000 7f996e200640 1 -- 172.21.15.102:0/2967943723 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f994406ac30 msgr2=0x7f994406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:30.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.742+0000 7f996e200640 1 --2- 172.21.15.102:0/2967943723 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f994406ac30 0x7f994406d0f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f996815e370 tx=0x7f9958008040 comp rx=0 tx=0).stop 2024-03-23T03:48:30.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.742+0000 7f996e200640 1 -- 172.21.15.102:0/2967943723 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99680adbc0 msgr2=0x7f9968159320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:30.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.742+0000 7f996e200640 1 --2- 172.21.15.102:0/2967943723 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99680adbc0 0x7f9968159320 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f995002b8a0 tx=0x7f99500046c0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.743+0000 7f996e200640 1 -- 172.21.15.102:0/2967943723 shutdown_connections 2024-03-23T03:48:30.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.743+0000 7f996e200640 1 --2- 172.21.15.102:0/2967943723 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f994406ac30 0x7f994406d0f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.743+0000 7f996e200640 1 --2- 172.21.15.102:0/2967943723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99680cbab0 0x7f99680ac2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.743+0000 7f996e200640 1 --2- 172.21.15.102:0/2967943723 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99680adbc0 0x7f9968159320 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.743+0000 7f996e200640 1 --2- 172.21.15.102:0/2967943723 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f996806d390 0x7f99681f3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:30.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.743+0000 7f996e200640 1 -- 172.21.15.102:0/2967943723 >> 172.21.15.102:0/2967943723 conn(0x7f996806a3d0 msgr2=0x7f99680adfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:30.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.743+0000 7f996e200640 1 -- 172.21.15.102:0/2967943723 shutdown_connections 2024-03-23T03:48:30.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:30.743+0000 7f996e200640 1 -- 172.21.15.102:0/2967943723 wait complete. 2024-03-23T03:48:30.761 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-23T03:48:30.761 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:48:30.761 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2024-03-23T03:48:30.761 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T03:48:30.761 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T03:48:30.761 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-23T03:48:30.762 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-23T03:48:30.762 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:48:30.762 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:48:30.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:30.762 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2024-03-23T03:48:30.809 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:48:30.809 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:48:30.809 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:48:30.810 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:48:30.810 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-03-23T03:48:30.810 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:48:30.810 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:48:30.810 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2024-03-23T03:48:30.810 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f0 2024-03-23T03:48:30.810 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:48:30.810 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2024-03-23T03:48:30.810 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f1 2024-03-23T03:48:30.810 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:48:30.810 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:48:30.810 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:48:30.810 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:48:30.810 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:48:30.811 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 42560sec preferred_lft 42560sec 2024-03-23T03:48:30.811 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-03-23T03:48:30.811 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:48:30.811 INFO:teuthology.orchestra.run.smithi102.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:48:30.811 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether aa:17:e3:39:b0:56 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:48:30.811 INFO:teuthology.orchestra.run.smithi102.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T03:48:30.811 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:48:30.811 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::5cce:1fff:fe41:5a05/64 scope link 2024-03-23T03:48:30.811 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:48:30.811 INFO:teuthology.orchestra.run.smithi102.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:48:30.811 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether aa:17:e3:39:b0:56 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T03:48:30.811 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::a817:e3ff:fe39:b056/64 scope link 2024-03-23T03:48:30.811 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:48:30.811 INFO:teuthology.orchestra.run.smithi102.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:48:30.811 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether ce:9b:95:fb:de:c5 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T03:48:30.811 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::cc9b:95ff:fefb:dec5/64 scope link 2024-03-23T03:48:30.812 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:48:30.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:30.812 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:48:30.900 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-23T03:48:30.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:30.904 DEBUG:teuthology.orchestra.run.smithi102:> route 2024-03-23T03:48:30.956 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2024-03-23T03:48:30.956 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:48:30.956 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:48:30.956 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:48:30.956 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:48:30.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:30.958 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:48:30.958 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:48:30.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:48:30.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:48:30.958 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:48:30.958 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:48:31.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:31.142 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-23T03:48:31.191 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T03:48:31.191 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T03:48:31.192 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:48:31.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:31.192 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.0 up 2024-03-23T03:48:31.280 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:48:31.280 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T03:48:31.331 INFO:teuthology.orchestra.run.smithi102.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:48:31.332 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:48:31.332 DEBUG:teuthology.orchestra.run.smithi102:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T03:48:31.381 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-23T03:48:31.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:31.381 DEBUG:teuthology.orchestra.run.smithi102:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:48:31.517 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:48:31.517 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:48:31.517 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:48:31.538 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:48:31.538 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:48:31.538 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:48:31.538 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:48:31.538 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:48:31.564 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:48:31.564 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:48:31.564 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:48:31.564 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:48:31.564 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:48:31.564 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790" 2024-03-23T03:48:31.564 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/ 2024-03-23T03:48:31.565 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=776b2820-05f2-4e50-8d40-18dd76c5f6a0" will pass to kernel 2024-03-23T03:48:31.566 INFO:teuthology.orchestra.run.smithi102.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-23T03:48:31.566 INFO:teuthology.orchestra.run.smithi102.stdout: You just mounted a file system that supports labels which does not 2024-03-23T03:48:31.567 INFO:teuthology.orchestra.run.smithi102.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T03:48:31.567 INFO:teuthology.orchestra.run.smithi102.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T03:48:31.567 INFO:teuthology.orchestra.run.smithi102.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T03:48:31.572 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T03:48:31.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:31.572 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T03:48:31.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:31.660 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:48:31.722 INFO:teuthology.orchestra.run.smithi102.stdout:client4709 2024-03-23T03:48:31.723 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:48:31.723 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:48:31.773 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:48:31.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.976+0000 7f84ed000640 1 Processor -- start 2024-03-23T03:48:31.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.976+0000 7f84ed000640 1 -- start start 2024-03-23T03:48:31.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.977+0000 7f84ed000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f84e80b39b0 0x7f84e80b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:31.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.978+0000 7f84ed000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e806c6f0 con 0x7f84e815b560 2024-03-23T03:48:31.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.978+0000 7f84ed000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e806c9d0 con 0x7f84e815c400 2024-03-23T03:48:31.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.978+0000 7f84ed000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e80b4370 con 0x7f84e80b39b0 2024-03-23T03:48:31.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.978+0000 7f84dd400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f84e80b39b0 0x7f84e80b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:31.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.978+0000 7f84e5e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f84e815b560 0x7f84e815b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37842/0 (socket says 172.21.15.102:37842) 2024-03-23T03:48:31.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.978+0000 7f84e5e00640 1 -- 172.21.15.102:0/2280389323 learned_addr learned my addr 172.21.15.102:0/2280389323 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:31.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.978+0000 7f84dd400640 1 -- 172.21.15.102:0/2280389323 >> v1:172.21.15.173:6789/0 conn(0x7f84e815c400 legacy=0x7f84e815c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:31.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.978+0000 7f84dd400640 1 -- 172.21.15.102:0/2280389323 >> v1:172.21.15.102:6789/0 conn(0x7f84e815b560 legacy=0x7f84e815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:31.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.978+0000 7f84dd400640 1 -- 172.21.15.102:0/2280389323 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e806ccb0 con 0x7f84e80b39b0 2024-03-23T03:48:31.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.978+0000 7f84e5400640 1 -- 172.21.15.102:0/2280389323 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1348856860 0 0) 0x7f84e806c6f0 con 0x7f84e815b560 2024-03-23T03:48:31.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.980+0000 7f84dd400640 1 --2- 172.21.15.102:0/2280389323 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e80b39b0 0x7f84e80b3e30 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f84d00097a0 tx=0x7f84d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8225695cc3e936d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:31.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.980+0000 7f84e5400640 1 -- 172.21.15.102:0/2280389323 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d0031830 con 0x7f84e80b39b0 2024-03-23T03:48:31.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.980+0000 7f84e5400640 1 -- 172.21.15.102:0/2280389323 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d0031990 con 0x7f84e80b39b0 2024-03-23T03:48:31.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.980+0000 7f84e5400640 1 -- 172.21.15.102:0/2280389323 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d0031d20 con 0x7f84e80b39b0 2024-03-23T03:48:31.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.980+0000 7f84ed000640 1 -- 172.21.15.102:0/2280389323 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e80b39b0 msgr2=0x7f84e80b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:31.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.980+0000 7f84ed000640 1 --2- 172.21.15.102:0/2280389323 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e80b39b0 0x7f84e80b3e30 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f84d00097a0 tx=0x7f84d002b8d0 comp rx=0 tx=0).stop 2024-03-23T03:48:31.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.980+0000 7f84ed000640 1 -- 172.21.15.102:0/2280389323 shutdown_connections 2024-03-23T03:48:31.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.981+0000 7f84ed000640 1 --2- 172.21.15.102:0/2280389323 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e80b39b0 0x7f84e80b3e30 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:31.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.981+0000 7f84ed000640 1 -- 172.21.15.102:0/2280389323 >> 172.21.15.102:0/2280389323 conn(0x7f84e806a3d0 msgr2=0x7f84e80b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:31.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.981+0000 7f84ed000640 1 -- 172.21.15.102:0/2280389323 shutdown_connections 2024-03-23T03:48:31.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.981+0000 7f84ed000640 1 -- 172.21.15.102:0/2280389323 wait complete. 2024-03-23T03:48:31.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.982+0000 7f84ed000640 1 Processor -- start 2024-03-23T03:48:31.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.982+0000 7f84ed000640 1 -- start start 2024-03-23T03:48:31.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.983+0000 7f84ed000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84e80b39b0 0x7f84e8154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:31.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.983+0000 7f84ed000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e815b560 0x7f84e813d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:31.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.983+0000 7f84ed000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e815c400 0x7f84e8140010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:31.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.983+0000 7f84ed000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e8154ea0 con 0x7f84e815b560 2024-03-23T03:48:31.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.983+0000 7f84ed000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e8155ae0 con 0x7f84e815c400 2024-03-23T03:48:31.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.983+0000 7f84ed000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e80ae310 con 0x7f84e80b39b0 2024-03-23T03:48:31.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.984+0000 7f84dd400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e815b560 0x7f84e813d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:31.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.984+0000 7f84dd400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e815b560 0x7f84e813d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45164/0 (socket says 172.21.15.102:45164) 2024-03-23T03:48:31.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.984+0000 7f84dd400640 1 -- 172.21.15.102:0/1817397038 learned_addr learned my addr 172.21.15.102:0/1817397038 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:31.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.984+0000 7f84e5e00640 1 --2- 172.21.15.102:0/1817397038 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84e80b39b0 0x7f84e8154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:31.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.984+0000 7f84dd400640 1 -- 172.21.15.102:0/1817397038 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84e80b39b0 msgr2=0x7f84e8154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:31.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.984+0000 7f84dd400640 1 --2- 172.21.15.102:0/1817397038 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84e80b39b0 0x7f84e8154960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:31.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.984+0000 7f84dd400640 1 -- 172.21.15.102:0/1817397038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e815c400 msgr2=0x7f84e8140010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:31.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.984+0000 7f84dd400640 1 --2- 172.21.15.102:0/1817397038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e815c400 0x7f84e8140010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:31.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.984+0000 7f84dd400640 1 -- 172.21.15.102:0/1817397038 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d0009400 con 0x7f84e815b560 2024-03-23T03:48:31.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.984+0000 7f84dd400640 1 --2- 172.21.15.102:0/1817397038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e815b560 0x7f84e813d5e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f84d002bee0 tx=0x7f84d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:31.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.985+0000 7f84dea00640 1 -- 172.21.15.102:0/1817397038 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d003ba70 con 0x7f84e815b560 2024-03-23T03:48:31.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.985+0000 7f84dea00640 1 -- 172.21.15.102:0/1817397038 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84d003bbd0 con 0x7f84e815b560 2024-03-23T03:48:31.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.985+0000 7f84dea00640 1 -- 172.21.15.102:0/1817397038 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d00453f0 con 0x7f84e815b560 2024-03-23T03:48:31.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.985+0000 7f84ed000640 1 -- 172.21.15.102:0/1817397038 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e813e470 con 0x7f84e815b560 2024-03-23T03:48:31.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.985+0000 7f84ed000640 1 -- 172.21.15.102:0/1817397038 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e8155800 con 0x7f84e815b560 2024-03-23T03:48:31.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.986+0000 7f84ed000640 1 -- 172.21.15.102:0/1817397038 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e806ac20 con 0x7f84e815b560 2024-03-23T03:48:31.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.987+0000 7f84dea00640 1 -- 172.21.15.102:0/1817397038 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f84d0045550 con 0x7f84e815b560 2024-03-23T03:48:31.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.988+0000 7f84dea00640 1 --2- 172.21.15.102:0/1817397038 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f84c406ac00 0x7f84c406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:31.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.988+0000 7f84dea00640 1 -- 172.21.15.102:0/1817397038 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f84d00ab7a0 con 0x7f84e815b560 2024-03-23T03:48:31.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.988+0000 7f84e5e00640 1 --2- 172.21.15.102:0/1817397038 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f84c406ac00 0x7f84c406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:31.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.989+0000 7f84e5e00640 1 --2- 172.21.15.102:0/1817397038 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f84c406ac00 0x7f84c406d0c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f84d8000f60 tx=0x7f84d8007b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:31.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:31.992+0000 7f84dea00640 1 -- 172.21.15.102:0/1817397038 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f84d0076820 con 0x7f84e815b560 2024-03-23T03:48:32.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:32.230+0000 7f84ed000640 1 -- 172.21.15.102:0/1817397038 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f84e8073390 con 0x7f84e815b560 2024-03-23T03:48:32.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:32.232+0000 7f84dea00640 1 -- 172.21.15.102:0/1817397038 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7f84d0079e40 con 0x7f84e815b560 2024-03-23T03:48:32.231 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T03:48:32.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:32.233+0000 7f84ed000640 1 -- 172.21.15.102:0/1817397038 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f84c406ac00 msgr2=0x7f84c406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:32.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:32.233+0000 7f84ed000640 1 --2- 172.21.15.102:0/1817397038 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f84c406ac00 0x7f84c406d0c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f84d8000f60 tx=0x7f84d8007b50 comp rx=0 tx=0).stop 2024-03-23T03:48:32.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:32.233+0000 7f84ed000640 1 -- 172.21.15.102:0/1817397038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e815b560 msgr2=0x7f84e813d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:32.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:32.233+0000 7f84ed000640 1 --2- 172.21.15.102:0/1817397038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e815b560 0x7f84e813d5e0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f84d002bee0 tx=0x7f84d00046c0 comp rx=0 tx=0).stop 2024-03-23T03:48:32.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:32.233+0000 7f84ed000640 1 -- 172.21.15.102:0/1817397038 shutdown_connections 2024-03-23T03:48:32.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:32.233+0000 7f84ed000640 1 --2- 172.21.15.102:0/1817397038 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f84c406ac00 0x7f84c406d0c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:32.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:32.233+0000 7f84ed000640 1 --2- 172.21.15.102:0/1817397038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f84e815c400 0x7f84e8140010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:32.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:32.233+0000 7f84ed000640 1 --2- 172.21.15.102:0/1817397038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e815b560 0x7f84e813d5e0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:32.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:32.233+0000 7f84ed000640 1 --2- 172.21.15.102:0/1817397038 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84e80b39b0 0x7f84e8154960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:32.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:32.233+0000 7f84ed000640 1 -- 172.21.15.102:0/1817397038 >> 172.21.15.102:0/1817397038 conn(0x7f84e806a3d0 msgr2=0x7f84e80ad1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:32.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:32.234+0000 7f84ed000640 1 -- 172.21.15.102:0/1817397038 shutdown_connections 2024-03-23T03:48:32.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:32.234+0000 7f84ed000640 1 -- 172.21.15.102:0/1817397038 wait complete. 2024-03-23T03:48:32.251 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-23T03:48:32.252 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:48:32.252 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi173.front.sepia.ceph.com 2024-03-23T03:48:32.252 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-23T03:48:32.252 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-23T03:48:32.252 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-23T03:48:32.252 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-23T03:48:32.252 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:48:32.252 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:48:32.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:32.252 DEBUG:teuthology.orchestra.run.smithi173:> ip addr 2024-03-23T03:48:32.309 INFO:teuthology.orchestra.run.smithi173.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:48:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:48:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:48:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:48:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 ::1/128 scope host 2024-03-23T03:48:32.309 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:48:32.309 INFO:teuthology.orchestra.run.smithi173.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:48:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6a brd ff:ff:ff:ff:ff:ff 2024-03-23T03:48:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f0 2024-03-23T03:48:32.310 INFO:teuthology.orchestra.run.smithi173.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:48:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6b brd ff:ff:ff:ff:ff:ff 2024-03-23T03:48:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f1 2024-03-23T03:48:32.310 INFO:teuthology.orchestra.run.smithi173.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:48:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:58 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:48:32.310 INFO:teuthology.orchestra.run.smithi173.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:48:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:59 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:48:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: inet 172.21.15.173/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:48:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft 42529sec preferred_lft 42529sec 2024-03-23T03:48:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::ec4:7aff:fe88:7259/64 scope link 2024-03-23T03:48:32.310 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:48:32.310 INFO:teuthology.orchestra.run.smithi173.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:48:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 52:69:f2:bf:6d:02 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:48:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T03:48:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:48:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::6894:e9ff:fe5d:1492/64 scope link 2024-03-23T03:48:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:48:32.311 INFO:teuthology.orchestra.run.smithi173.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:48:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether a2:27:ac:42:68:ce brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T03:48:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::a027:acff:fe42:68ce/64 scope link 2024-03-23T03:48:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:48:32.311 INFO:teuthology.orchestra.run.smithi173.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:48:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 52:69:f2:bf:6d:02 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-23T03:48:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::5069:f2ff:febf:6d02/64 scope link 2024-03-23T03:48:32.311 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:48:32.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:32.312 DEBUG:teuthology.orchestra.run.smithi173:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:48:32.401 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-03-23T03:48:32.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:32.406 DEBUG:teuthology.orchestra.run.smithi173:> route 2024-03-23T03:48:32.458 INFO:teuthology.orchestra.run.smithi173.stdout:Kernel IP routing table 2024-03-23T03:48:32.458 INFO:teuthology.orchestra.run.smithi173.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:48:32.458 INFO:teuthology.orchestra.run.smithi173.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:48:32.458 INFO:teuthology.orchestra.run.smithi173.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:48:32.458 INFO:teuthology.orchestra.run.smithi173.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:48:32.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:32.459 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-03-23T03:48:32.459 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-03-23T03:48:32.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:48:32.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:48:32.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:48:32.459 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-03-23T03:48:32.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:32.629 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list 2024-03-23T03:48:32.678 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-23T03:48:32.679 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-23T03:48:32.679 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:48:32.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:32.680 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link set brx.0 up 2024-03-23T03:48:32.773 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:48:32.773 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-23T03:48:32.826 INFO:teuthology.orchestra.run.smithi173.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:48:32.826 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:48:32.826 DEBUG:teuthology.orchestra.run.smithi173:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-23T03:48:32.873 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-23T03:48:32.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:32.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:48:33.057 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:48:33.057 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:48:33.058 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:48:33.078 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:48:33.079 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:48:33.079 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:48:33.079 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:48:33.079 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:48:33.106 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:48:33.107 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:48:33.107 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:48:33.107 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:48:33.107 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:48:33.107 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790" 2024-03-23T03:48:33.107 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/ 2024-03-23T03:48:33.107 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=776b2820-05f2-4e50-8d40-18dd76c5f6a0" will pass to kernel 2024-03-23T03:48:33.108 INFO:teuthology.orchestra.run.smithi173.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-23T03:48:33.109 INFO:teuthology.orchestra.run.smithi173.stdout: You just mounted a file system that supports labels which does not 2024-03-23T03:48:33.109 INFO:teuthology.orchestra.run.smithi173.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T03:48:33.109 INFO:teuthology.orchestra.run.smithi173.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T03:48:33.109 INFO:teuthology.orchestra.run.smithi173.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T03:48:33.112 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T03:48:33.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:33.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-23T03:48:33.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:33.187 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:48:33.238 INFO:teuthology.orchestra.run.smithi173.stdout:client4927 2024-03-23T03:48:33.238 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:48:33.239 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:48:33.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:33.289 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-03-23T03:48:33.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:33.357 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 0 ./subdir) 2024-03-23T03:48:33.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:33.410 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-03-23T03:48:33.459 INFO:teuthology.orchestra.run.smithi102.stderr:./subdir: ceph.quota.max_bytes: No such attribute 2024-03-23T03:48:33.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:48:33.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:33.460 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 1K ./subdir) 2024-03-23T03:48:33.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:33.515 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-03-23T03:48:33.568 INFO:teuthology.orchestra.run.smithi102.stdout:1024 2024-03-23T03:48:33.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:33.569 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 0M ./subdir) 2024-03-23T03:48:33.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:33.621 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-03-23T03:48:33.673 INFO:teuthology.orchestra.run.smithi102.stderr:./subdir: ceph.quota.max_bytes: No such attribute 2024-03-23T03:48:33.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:48:33.674 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T03:48:33.675 DEBUG:teuthology.orchestra.run.smithi102:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:48:33.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:48:33.924 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T03:48:33.924 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:48:33.924 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:48:33.980 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:48:33.980 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:48:34.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:48:34.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.236+0000 7f718d200640 1 Processor -- start 2024-03-23T03:48:34.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.236+0000 7f718d200640 1 -- start start 2024-03-23T03:48:34.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.238+0000 7f718d200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f718815b560 0x7f718815b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:34.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.238+0000 7f718d200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f71880b39b0 0x7f71880b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:34.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.238+0000 7f718d200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f718806c6f0 con 0x7f718815e500 2024-03-23T03:48:34.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.238+0000 7f718d200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71880b4370 con 0x7f71880b39b0 2024-03-23T03:48:34.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.238+0000 7f718d200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f718806c9d0 con 0x7f718815b560 2024-03-23T03:48:34.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.238+0000 7f7185400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f71880b39b0 0x7f71880b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:34.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.238+0000 7f7185400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f71880b39b0 0x7f71880b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55144/0 (socket says 172.21.15.102:55144) 2024-03-23T03:48:34.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.238+0000 7f7185400640 1 -- 172.21.15.102:0/415123798 learned_addr learned my addr 172.21.15.102:0/415123798 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:34.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.238+0000 7f7185400640 1 -- 172.21.15.102:0/415123798 >> v2:172.21.15.173:3300/0 conn(0x7f718815b560 msgr2=0x7f718815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:34.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.238+0000 7f7185e00640 1 --2- 172.21.15.102:0/415123798 >> v2:172.21.15.173:3300/0 conn(0x7f718815b560 0x7f718815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:34.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.238+0000 7f7185400640 1 --2- 172.21.15.102:0/415123798 >> v2:172.21.15.173:3300/0 conn(0x7f718815b560 0x7f718815b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.239+0000 7f7185400640 1 -- 172.21.15.102:0/415123798 >> v1:172.21.15.173:6789/0 conn(0x7f718815e500 legacy=0x7f718815e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:34.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.239+0000 7f7185400640 1 -- 172.21.15.102:0/415123798 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f718806ccb0 con 0x7f71880b39b0 2024-03-23T03:48:34.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.240+0000 7f7185400640 1 --2- 172.21.15.102:0/415123798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71880b39b0 0x7f71880b3e30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f7188071470 tx=0x7f717402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79ed75cfc1f215a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:34.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.240+0000 7f7184a00640 1 -- 172.21.15.102:0/415123798 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7174031870 con 0x7f71880b39b0 2024-03-23T03:48:34.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.240+0000 7f7184a00640 1 -- 172.21.15.102:0/415123798 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71740319d0 con 0x7f71880b39b0 2024-03-23T03:48:34.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.240+0000 7f7184a00640 1 -- 172.21.15.102:0/415123798 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7174031d60 con 0x7f71880b39b0 2024-03-23T03:48:34.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.240+0000 7f718d200640 1 -- 172.21.15.102:0/415123798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71880b39b0 msgr2=0x7f71880b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:34.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.240+0000 7f718d200640 1 --2- 172.21.15.102:0/415123798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71880b39b0 0x7f71880b3e30 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f7188071470 tx=0x7f717402b910 comp rx=0 tx=0).stop 2024-03-23T03:48:34.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.241+0000 7f718d200640 1 -- 172.21.15.102:0/415123798 shutdown_connections 2024-03-23T03:48:34.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.241+0000 7f718d200640 1 --2- 172.21.15.102:0/415123798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71880b39b0 0x7f71880b3e30 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.241+0000 7f718d200640 1 --2- 172.21.15.102:0/415123798 >> v2:172.21.15.173:3300/0 conn(0x7f718815b560 0x7f718815b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.241+0000 7f718d200640 1 -- 172.21.15.102:0/415123798 >> 172.21.15.102:0/415123798 conn(0x7f718806a3d0 msgr2=0x7f71880b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:34.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.241+0000 7f718d200640 1 -- 172.21.15.102:0/415123798 shutdown_connections 2024-03-23T03:48:34.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.241+0000 7f718d200640 1 -- 172.21.15.102:0/415123798 wait complete. 2024-03-23T03:48:34.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.242+0000 7f718d200640 1 Processor -- start 2024-03-23T03:48:34.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.243+0000 7f718d200640 1 -- start start 2024-03-23T03:48:34.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.243+0000 7f718d200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71880b39b0 0x7f718814a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.243+0000 7f718d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f718815b560 0x7f7188153bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.244+0000 7f718d200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f718815e500 0x7f71880cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.244+0000 7f718d200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7188154990 con 0x7f718815e500 2024-03-23T03:48:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.244+0000 7f718d200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71880aeb50 con 0x7f718815b560 2024-03-23T03:48:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.244+0000 7f718d200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71881546b0 con 0x7f71880b39b0 2024-03-23T03:48:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.244+0000 7f7186800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f718815e500 0x7f71880cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.244+0000 7f7186800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f718815e500 0x7f71880cc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55160/0 (socket says 172.21.15.102:55160) 2024-03-23T03:48:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.244+0000 7f7186800640 1 -- 172.21.15.102:0/3494826167 learned_addr learned my addr 172.21.15.102:0/3494826167 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.244+0000 7f7185e00640 1 --2- 172.21.15.102:0/3494826167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71880b39b0 0x7f718814a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.244+0000 7f7185400640 1 --2- 172.21.15.102:0/3494826167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f718815b560 0x7f7188153bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.244+0000 7f7186800640 1 -- 172.21.15.102:0/3494826167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71880b39b0 msgr2=0x7f718814a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:34.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.244+0000 7f7186800640 1 --2- 172.21.15.102:0/3494826167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71880b39b0 0x7f718814a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.244+0000 7f7186800640 1 -- 172.21.15.102:0/3494826167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f718815b560 msgr2=0x7f7188153bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:34.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.244+0000 7f7186800640 1 --2- 172.21.15.102:0/3494826167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f718815b560 0x7f7188153bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.244+0000 7f7186800640 1 -- 172.21.15.102:0/3494826167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71740094d0 con 0x7f718815e500 2024-03-23T03:48:34.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.245+0000 7f7186800640 1 --2- 172.21.15.102:0/3494826167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f718815e500 0x7f71880cc0d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f718000b990 tx=0x7f718000be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:34.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.245+0000 7f717e000640 1 -- 172.21.15.102:0/3494826167 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718000cba0 con 0x7f718815e500 2024-03-23T03:48:34.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.245+0000 7f717e000640 1 -- 172.21.15.102:0/3494826167 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718000cd00 con 0x7f718815e500 2024-03-23T03:48:34.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.245+0000 7f717e000640 1 -- 172.21.15.102:0/3494826167 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7180014680 con 0x7f718815e500 2024-03-23T03:48:34.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.245+0000 7f718d200640 1 -- 172.21.15.102:0/3494826167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71880ae490 con 0x7f718815e500 2024-03-23T03:48:34.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.245+0000 7f718d200640 1 -- 172.21.15.102:0/3494826167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71881538d0 con 0x7f718815e500 2024-03-23T03:48:34.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.246+0000 7f718d200640 1 -- 172.21.15.102:0/3494826167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71880ae2f0 con 0x7f718815e500 2024-03-23T03:48:34.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.247+0000 7f717e000640 1 -- 172.21.15.102:0/3494826167 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f718001d3b0 con 0x7f718815e500 2024-03-23T03:48:34.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.248+0000 7f717e000640 1 --2- 172.21.15.102:0/3494826167 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f716806ac00 0x7f716806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:34.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.248+0000 7f717e000640 1 -- 172.21.15.102:0/3494826167 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7180089510 con 0x7f718815e500 2024-03-23T03:48:34.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.248+0000 7f7185e00640 1 --2- 172.21.15.102:0/3494826167 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f716806ac00 0x7f716806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:34.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.249+0000 7f7185e00640 1 --2- 172.21.15.102:0/3494826167 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f716806ac00 0x7f716806d0c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7178006fd0 tx=0x7f7178008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:34.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.252+0000 7f717e000640 1 -- 172.21.15.102:0/3494826167 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f71880ae2f0 con 0x7f718815e500 2024-03-23T03:48:34.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.446+0000 7f718d200640 1 -- 172.21.15.102:0/3494826167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71880aec90 con 0x7f718815e500 2024-03-23T03:48:34.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.449+0000 7f717e000640 1 -- 172.21.15.102:0/3494826167 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+16354 (secure 0 0 0) 0x7f71880aec90 con 0x7f718815e500 2024-03-23T03:48:34.450 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:34.451 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":41,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:24.423308+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:48:21.550281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-23T03:48:22.428475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:48:34.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.452+0000 7f718d200640 1 -- 172.21.15.102:0/3494826167 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f716806ac00 msgr2=0x7f716806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:34.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.452+0000 7f718d200640 1 --2- 172.21.15.102:0/3494826167 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f716806ac00 0x7f716806d0c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7178006fd0 tx=0x7f7178008040 comp rx=0 tx=0).stop 2024-03-23T03:48:34.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.452+0000 7f718d200640 1 -- 172.21.15.102:0/3494826167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f718815e500 msgr2=0x7f71880cc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:34.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.452+0000 7f718d200640 1 --2- 172.21.15.102:0/3494826167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f718815e500 0x7f71880cc0d0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f718000b990 tx=0x7f718000be50 comp rx=0 tx=0).stop 2024-03-23T03:48:34.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.453+0000 7f718d200640 1 -- 172.21.15.102:0/3494826167 shutdown_connections 2024-03-23T03:48:34.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.453+0000 7f718d200640 1 --2- 172.21.15.102:0/3494826167 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f716806ac00 0x7f716806d0c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.453+0000 7f718d200640 1 --2- 172.21.15.102:0/3494826167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f718815e500 0x7f71880cc0d0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.453+0000 7f718d200640 1 --2- 172.21.15.102:0/3494826167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f718815b560 0x7f7188153bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.453+0000 7f718d200640 1 --2- 172.21.15.102:0/3494826167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71880b39b0 0x7f718814a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.453+0000 7f718d200640 1 -- 172.21.15.102:0/3494826167 >> 172.21.15.102:0/3494826167 conn(0x7f718806a3d0 msgr2=0x7f71880abd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:34.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.453+0000 7f718d200640 1 -- 172.21.15.102:0/3494826167 shutdown_connections 2024-03-23T03:48:34.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.453+0000 7f718d200640 1 -- 172.21.15.102:0/3494826167 wait complete. 2024-03-23T03:48:34.471 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:48:21.550281+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:48:22.428475+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:48:34.472 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:48:34.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.664+0000 7f3822a00640 1 Processor -- start 2024-03-23T03:48:34.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.664+0000 7f3822a00640 1 -- start start 2024-03-23T03:48:34.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.665+0000 7f3822a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f381c15c9e0 0x7f381c15dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:34.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.666+0000 7f3822a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f381c0cb420 0x7f381c160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:34.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.666+0000 7f3822a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381c160530 con 0x7f381c0b12d0 2024-03-23T03:48:34.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.666+0000 7f3822a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f381c160af0 con 0x7f381c15c9e0 2024-03-23T03:48:34.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.666+0000 7f3822a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381c1614c0 con 0x7f381c0cb420 2024-03-23T03:48:34.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.666+0000 7f3821600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f381c15c9e0 0x7f381c15dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:34.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.666+0000 7f3821600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f381c15c9e0 0x7f381c15dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55166/0 (socket says 172.21.15.102:55166) 2024-03-23T03:48:34.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.666+0000 7f3821600640 1 -- 172.21.15.102:0/278476445 learned_addr learned my addr 172.21.15.102:0/278476445 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:34.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.666+0000 7f3821600640 1 -- 172.21.15.102:0/278476445 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f381c0cb420 msgr2=0x7f381c160f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:34.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.666+0000 7f3822000640 1 --2- 172.21.15.102:0/278476445 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f381c0cb420 0x7f381c160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:34.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.666+0000 7f3821600640 1 --2- 172.21.15.102:0/278476445 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f381c0cb420 0x7f381c160f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.666+0000 7f3821600640 1 -- 172.21.15.102:0/278476445 >> v1:172.21.15.173:6789/0 conn(0x7f381c0b12d0 legacy=0x7f381c0cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:34.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.666+0000 7f3821600640 1 -- 172.21.15.102:0/278476445 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381c15cdc0 con 0x7f381c15c9e0 2024-03-23T03:48:34.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.666+0000 7f3822000640 1 --2- 172.21.15.102:0/278476445 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f381c0cb420 0x7f381c160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:34.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.668+0000 7f3821600640 1 --2- 172.21.15.102:0/278476445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f381c15c9e0 0x7f381c15dae0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3810009870 tx=0x7f381002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff3de083f1ca4a1b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:34.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.668+0000 7f381be00640 1 -- 172.21.15.102:0/278476445 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3810031640 con 0x7f381c15c9e0 2024-03-23T03:48:34.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.668+0000 7f381be00640 1 -- 172.21.15.102:0/278476445 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38100317a0 con 0x7f381c15c9e0 2024-03-23T03:48:34.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.668+0000 7f381be00640 1 -- 172.21.15.102:0/278476445 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3810031b30 con 0x7f381c15c9e0 2024-03-23T03:48:34.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.668+0000 7f3822a00640 1 -- 172.21.15.102:0/278476445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f381c15c9e0 msgr2=0x7f381c15dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:34.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.668+0000 7f3822a00640 1 --2- 172.21.15.102:0/278476445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f381c15c9e0 0x7f381c15dae0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f3810009870 tx=0x7f381002b8f0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.669+0000 7f3822a00640 1 -- 172.21.15.102:0/278476445 shutdown_connections 2024-03-23T03:48:34.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.669+0000 7f3822a00640 1 --2- 172.21.15.102:0/278476445 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f381c0cb420 0x7f381c160f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.669+0000 7f3822a00640 1 --2- 172.21.15.102:0/278476445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f381c15c9e0 0x7f381c15dae0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.669+0000 7f3822a00640 1 -- 172.21.15.102:0/278476445 >> 172.21.15.102:0/278476445 conn(0x7f381c0b01b0 msgr2=0x7f381c0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:34.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.669+0000 7f3822a00640 1 -- 172.21.15.102:0/278476445 shutdown_connections 2024-03-23T03:48:34.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.669+0000 7f3822a00640 1 -- 172.21.15.102:0/278476445 wait complete. 2024-03-23T03:48:34.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.670+0000 7f3822a00640 1 Processor -- start 2024-03-23T03:48:34.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.671+0000 7f3822a00640 1 -- start start 2024-03-23T03:48:34.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.671+0000 7f3822a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f381c0b12d0 0x7f381c1557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:34.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.671+0000 7f3822a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f381c0cb420 0x7f381c0aeba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:34.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.671+0000 7f3822a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f381c15c9e0 0x7f381c0ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:34.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.671+0000 7f3822a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c0af3d0 con 0x7f381c0cb420 2024-03-23T03:48:34.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.671+0000 7f3822a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381c155d20 con 0x7f381c15c9e0 2024-03-23T03:48:34.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.671+0000 7f3822a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381c0ab3e0 con 0x7f381c0b12d0 2024-03-23T03:48:34.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.672+0000 7f3820c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f381c0cb420 0x7f381c0aeba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:34.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.672+0000 7f3820c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f381c0cb420 0x7f381c0aeba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55170/0 (socket says 172.21.15.102:55170) 2024-03-23T03:48:34.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.672+0000 7f3820c00640 1 -- 172.21.15.102:0/3513914419 learned_addr learned my addr 172.21.15.102:0/3513914419 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:34.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.672+0000 7f3821600640 1 --2- 172.21.15.102:0/3513914419 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f381c0b12d0 0x7f381c1557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:34.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.672+0000 7f3822000640 1 --2- 172.21.15.102:0/3513914419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f381c15c9e0 0x7f381c0ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:34.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.672+0000 7f3820c00640 1 -- 172.21.15.102:0/3513914419 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f381c0b12d0 msgr2=0x7f381c1557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:34.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.672+0000 7f3820c00640 1 --2- 172.21.15.102:0/3513914419 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f381c0b12d0 0x7f381c1557e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.672+0000 7f3820c00640 1 -- 172.21.15.102:0/3513914419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f381c15c9e0 msgr2=0x7f381c0ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:34.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.672+0000 7f3820c00640 1 --2- 172.21.15.102:0/3513914419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f381c15c9e0 0x7f381c0ce5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.672+0000 7f3820c00640 1 -- 172.21.15.102:0/3513914419 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38100094d0 con 0x7f381c0cb420 2024-03-23T03:48:34.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.673+0000 7f3820c00640 1 --2- 172.21.15.102:0/3513914419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f381c0cb420 0x7f381c0aeba0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f380c00ba50 tx=0x7f380c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:34.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.673+0000 7f3819600640 1 -- 172.21.15.102:0/3513914419 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c00cbe0 con 0x7f381c0cb420 2024-03-23T03:48:34.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.673+0000 7f3819600640 1 -- 172.21.15.102:0/3513914419 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380c00cd80 con 0x7f381c0cb420 2024-03-23T03:48:34.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.673+0000 7f3819600640 1 -- 172.21.15.102:0/3513914419 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380c014680 con 0x7f381c0cb420 2024-03-23T03:48:34.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.673+0000 7f3822a00640 1 -- 172.21.15.102:0/3513914419 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f381c0ab6c0 con 0x7f381c0cb420 2024-03-23T03:48:34.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.673+0000 7f3822a00640 1 -- 172.21.15.102:0/3513914419 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f381c0abc80 con 0x7f381c0cb420 2024-03-23T03:48:34.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.674+0000 7f3822a00640 1 -- 172.21.15.102:0/3513914419 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f381c0abea0 con 0x7f381c0cb420 2024-03-23T03:48:34.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.675+0000 7f3819600640 1 -- 172.21.15.102:0/3513914419 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f380c0147e0 con 0x7f381c0cb420 2024-03-23T03:48:34.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.676+0000 7f3819600640 1 --2- 172.21.15.102:0/3513914419 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f380006ac00 0x7f380006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:34.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.676+0000 7f3819600640 1 -- 172.21.15.102:0/3513914419 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f380c089380 con 0x7f381c0cb420 2024-03-23T03:48:34.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.676+0000 7f3821600640 1 --2- 172.21.15.102:0/3513914419 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f380006ac00 0x7f380006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:34.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.677+0000 7f3821600640 1 --2- 172.21.15.102:0/3513914419 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f380006ac00 0x7f380006d0c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3810036000 tx=0x7f381003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:34.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.681+0000 7f3819600640 1 -- 172.21.15.102:0/3513914419 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f380c0544d0 con 0x7f381c0cb420 2024-03-23T03:48:34.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.867+0000 7f3822a00640 1 -- 172.21.15.102:0/3513914419 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f381c04e030 con 0x7f381c0cb420 2024-03-23T03:48:34.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.868+0000 7f3819600640 1 -- 172.21.15.102:0/3513914419 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f380c057af0 con 0x7f381c0cb420 2024-03-23T03:48:34.869 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:48:34.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.871+0000 7f3822a00640 1 -- 172.21.15.102:0/3513914419 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f380006ac00 msgr2=0x7f380006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:34.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.871+0000 7f3822a00640 1 --2- 172.21.15.102:0/3513914419 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f380006ac00 0x7f380006d0c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3810036000 tx=0x7f381003c040 comp rx=0 tx=0).stop 2024-03-23T03:48:34.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.871+0000 7f3822a00640 1 -- 172.21.15.102:0/3513914419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f381c0cb420 msgr2=0x7f381c0aeba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:34.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.871+0000 7f3822a00640 1 --2- 172.21.15.102:0/3513914419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f381c0cb420 0x7f381c0aeba0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f380c00ba50 tx=0x7f380c00bf10 comp rx=0 tx=0).stop 2024-03-23T03:48:34.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.871+0000 7f3822a00640 1 -- 172.21.15.102:0/3513914419 shutdown_connections 2024-03-23T03:48:34.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.871+0000 7f3822a00640 1 --2- 172.21.15.102:0/3513914419 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f380006ac00 0x7f380006d0c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.871+0000 7f3822a00640 1 --2- 172.21.15.102:0/3513914419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f381c15c9e0 0x7f381c0ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.871+0000 7f3822a00640 1 --2- 172.21.15.102:0/3513914419 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f381c0cb420 0x7f381c0aeba0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.871+0000 7f3822a00640 1 --2- 172.21.15.102:0/3513914419 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f381c0b12d0 0x7f381c1557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:34.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.871+0000 7f3822a00640 1 -- 172.21.15.102:0/3513914419 >> 172.21.15.102:0/3513914419 conn(0x7f381c0b01b0 msgr2=0x7f381c0ade20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:34.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.871+0000 7f3822a00640 1 -- 172.21.15.102:0/3513914419 shutdown_connections 2024-03-23T03:48:34.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:34.871+0000 7f3822a00640 1 -- 172.21.15.102:0/3513914419 wait complete. 2024-03-23T03:48:34.891 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:48:35.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.093+0000 7f25ce800640 1 Processor -- start 2024-03-23T03:48:35.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.093+0000 7f25ce800640 1 -- start start 2024-03-23T03:48:35.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.094+0000 7f25ce800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f25c80ce930 0x7f25c80add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.094+0000 7f25ce800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25c80cd3e0 0x7f25c80ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.094+0000 7f25ce800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f25c806ce90 0x7f25c806d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.094+0000 7f25ce800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c806c760 con 0x7f25c80ce930 2024-03-23T03:48:35.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.095+0000 7f25ce800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c80b42d0 con 0x7f25c806ce90 2024-03-23T03:48:35.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.095+0000 7f25ce800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c80cf250 con 0x7f25c80cd3e0 2024-03-23T03:48:35.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.095+0000 7f25c7400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f25c80ce930 0x7f25c80add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:35.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.095+0000 7f25c7400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f25c80ce930 0x7f25c80add30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55172/0 (socket says 172.21.15.102:55172) 2024-03-23T03:48:35.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.095+0000 7f25c7400640 1 -- 172.21.15.102:0/2967248182 learned_addr learned my addr 172.21.15.102:0/2967248182 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.095+0000 7f25c7400640 1 -- 172.21.15.102:0/2967248182 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25c80cd3e0 msgr2=0x7f25c80ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.095+0000 7f25c7400640 1 --2- 172.21.15.102:0/2967248182 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25c80cd3e0 0x7f25c80ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.095+0000 7f25c7e00640 1 --2- 172.21.15.102:0/2967248182 >> v2:172.21.15.173:3300/0 conn(0x7f25c806ce90 0x7f25c806d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.095+0000 7f25c7400640 1 -- 172.21.15.102:0/2967248182 >> v2:172.21.15.173:3300/0 conn(0x7f25c806ce90 msgr2=0x7f25c806d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.095+0000 7f25c7400640 1 --2- 172.21.15.102:0/2967248182 >> v2:172.21.15.173:3300/0 conn(0x7f25c806ce90 0x7f25c806d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.095+0000 7f25c7400640 1 -- 172.21.15.102:0/2967248182 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c80cdba0 con 0x7f25c80ce930 2024-03-23T03:48:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.096+0000 7f25c7400640 1 --2- 172.21.15.102:0/2967248182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25c80ce930 0x7f25c80add30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f25c80ccc70 tx=0x7f25b802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a36abc80df0dad server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:35.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.097+0000 7f25c6000640 1 -- 172.21.15.102:0/2967248182 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b8031860 con 0x7f25c80ce930 2024-03-23T03:48:35.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.097+0000 7f25c6000640 1 -- 172.21.15.102:0/2967248182 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b80319c0 con 0x7f25c80ce930 2024-03-23T03:48:35.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.097+0000 7f25c6000640 1 -- 172.21.15.102:0/2967248182 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b8031d50 con 0x7f25c80ce930 2024-03-23T03:48:35.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.097+0000 7f25ce800640 1 -- 172.21.15.102:0/2967248182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25c80ce930 msgr2=0x7f25c80add30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:35.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.097+0000 7f25ce800640 1 --2- 172.21.15.102:0/2967248182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25c80ce930 0x7f25c80add30 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f25c80ccc70 tx=0x7f25b802b900 comp rx=0 tx=0).stop 2024-03-23T03:48:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.097+0000 7f25ce800640 1 -- 172.21.15.102:0/2967248182 shutdown_connections 2024-03-23T03:48:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.097+0000 7f25ce800640 1 --2- 172.21.15.102:0/2967248182 >> v2:172.21.15.173:3300/0 conn(0x7f25c806ce90 0x7f25c806d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.097+0000 7f25ce800640 1 --2- 172.21.15.102:0/2967248182 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25c80cd3e0 0x7f25c80ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.097+0000 7f25ce800640 1 --2- 172.21.15.102:0/2967248182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25c80ce930 0x7f25c80add30 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.097+0000 7f25ce800640 1 -- 172.21.15.102:0/2967248182 >> 172.21.15.102:0/2967248182 conn(0x7f25c806a0b0 msgr2=0x7f25c806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.098+0000 7f25ce800640 1 -- 172.21.15.102:0/2967248182 shutdown_connections 2024-03-23T03:48:35.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.098+0000 7f25ce800640 1 -- 172.21.15.102:0/2967248182 wait complete. 2024-03-23T03:48:35.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.099+0000 7f25ce800640 1 Processor -- start 2024-03-23T03:48:35.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.099+0000 7f25ce800640 1 -- start start 2024-03-23T03:48:35.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.100+0000 7f25ce800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25c806ce90 0x7f25c81523e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.100+0000 7f25ce800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25c80cd3e0 0x7f25c813ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.100+0000 7f25ce800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25c80ce930 0x7f25c814d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.100+0000 7f25ce800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c814b670 con 0x7f25c80ce930 2024-03-23T03:48:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.100+0000 7f25ce800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c813cf90 con 0x7f25c806ce90 2024-03-23T03:48:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.100+0000 7f25ce800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c8152a20 con 0x7f25c80cd3e0 2024-03-23T03:48:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.101+0000 7f25c7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25c80ce930 0x7f25c814d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.101+0000 7f25c7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25c80ce930 0x7f25c814d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55180/0 (socket says 172.21.15.102:55180) 2024-03-23T03:48:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.101+0000 7f25c7e00640 1 -- 172.21.15.102:0/2799243518 learned_addr learned my addr 172.21.15.102:0/2799243518 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.101+0000 7f25c7400640 1 --2- 172.21.15.102:0/2799243518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25c806ce90 0x7f25c81523e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.101+0000 7f25c6a00640 1 --2- 172.21.15.102:0/2799243518 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25c80cd3e0 0x7f25c813ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.101+0000 7f25c7e00640 1 -- 172.21.15.102:0/2799243518 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25c80cd3e0 msgr2=0x7f25c813ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.101+0000 7f25c7e00640 1 --2- 172.21.15.102:0/2799243518 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25c80cd3e0 0x7f25c813ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.101+0000 7f25c7e00640 1 -- 172.21.15.102:0/2799243518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25c806ce90 msgr2=0x7f25c81523e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.101+0000 7f25c7e00640 1 --2- 172.21.15.102:0/2799243518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25c806ce90 0x7f25c81523e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.101+0000 7f25c7e00640 1 -- 172.21.15.102:0/2799243518 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b80094d0 con 0x7f25c80ce930 2024-03-23T03:48:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.101+0000 7f25c7e00640 1 --2- 172.21.15.102:0/2799243518 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25c80ce930 0x7f25c814d2c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f25c000cc60 tx=0x7f25c000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:35.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.102+0000 7f25bf400640 1 -- 172.21.15.102:0/2799243518 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c000ce70 con 0x7f25c80ce930 2024-03-23T03:48:35.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.102+0000 7f25bf400640 1 -- 172.21.15.102:0/2799243518 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c0003990 con 0x7f25c80ce930 2024-03-23T03:48:35.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.102+0000 7f25bf400640 1 -- 172.21.15.102:0/2799243518 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c0003d20 con 0x7f25c80ce930 2024-03-23T03:48:35.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.102+0000 7f25ce800640 1 -- 172.21.15.102:0/2799243518 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c813d210 con 0x7f25c80ce930 2024-03-23T03:48:35.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.102+0000 7f25ce800640 1 -- 172.21.15.102:0/2799243518 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c813d7d0 con 0x7f25c80ce930 2024-03-23T03:48:35.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.103+0000 7f25ce800640 1 -- 172.21.15.102:0/2799243518 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25c8073390 con 0x7f25c80ce930 2024-03-23T03:48:35.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.104+0000 7f25bf400640 1 -- 172.21.15.102:0/2799243518 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f25c0003e80 con 0x7f25c80ce930 2024-03-23T03:48:35.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.105+0000 7f25bf400640 1 --2- 172.21.15.102:0/2799243518 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f25a806ac00 0x7f25a806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.105+0000 7f25bf400640 1 -- 172.21.15.102:0/2799243518 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f25c0089880 con 0x7f25c80ce930 2024-03-23T03:48:35.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.105+0000 7f25c7400640 1 --2- 172.21.15.102:0/2799243518 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f25a806ac00 0x7f25a806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:35.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.105+0000 7f25c7400640 1 --2- 172.21.15.102:0/2799243518 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f25a806ac00 0x7f25a806d0c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f25b8036000 tx=0x7f25b803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:35.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.109+0000 7f25bf400640 1 -- 172.21.15.102:0/2799243518 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f25c0054920 con 0x7f25c80ce930 2024-03-23T03:48:35.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.303+0000 7f25ce800640 1 -- 172.21.15.102:0/2799243518 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f25c806c540 con 0x7f25c80ce930 2024-03-23T03:48:35.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.304+0000 7f25bf400640 1 -- 172.21.15.102:0/2799243518 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) v1 ==== 104+0+11 (secure 0 0 0) 0x7f25c001d020 con 0x7f25c80ce930 2024-03-23T03:48:35.304 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:48:35.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.306+0000 7f25ce800640 1 -- 172.21.15.102:0/2799243518 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f25a806ac00 msgr2=0x7f25a806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:35.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.306+0000 7f25ce800640 1 --2- 172.21.15.102:0/2799243518 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f25a806ac00 0x7f25a806d0c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f25b8036000 tx=0x7f25b803c040 comp rx=0 tx=0).stop 2024-03-23T03:48:35.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.306+0000 7f25ce800640 1 -- 172.21.15.102:0/2799243518 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25c80ce930 msgr2=0x7f25c814d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:35.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.306+0000 7f25ce800640 1 --2- 172.21.15.102:0/2799243518 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25c80ce930 0x7f25c814d2c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f25c000cc60 tx=0x7f25c000a5c0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.306+0000 7f25ce800640 1 -- 172.21.15.102:0/2799243518 shutdown_connections 2024-03-23T03:48:35.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.306+0000 7f25ce800640 1 --2- 172.21.15.102:0/2799243518 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f25a806ac00 0x7f25a806d0c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.306+0000 7f25ce800640 1 --2- 172.21.15.102:0/2799243518 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f25c80ce930 0x7f25c814d2c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.306+0000 7f25ce800640 1 --2- 172.21.15.102:0/2799243518 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f25c80cd3e0 0x7f25c813ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.306+0000 7f25ce800640 1 --2- 172.21.15.102:0/2799243518 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f25c806ce90 0x7f25c81523e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.306+0000 7f25ce800640 1 -- 172.21.15.102:0/2799243518 >> 172.21.15.102:0/2799243518 conn(0x7f25c806a0b0 msgr2=0x7f25c806d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:35.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.306+0000 7f25ce800640 1 -- 172.21.15.102:0/2799243518 shutdown_connections 2024-03-23T03:48:35.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.306+0000 7f25ce800640 1 -- 172.21.15.102:0/2799243518 wait complete. 2024-03-23T03:48:35.324 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:48:35.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.536+0000 7f1722c00640 1 Processor -- start 2024-03-23T03:48:35.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.536+0000 7f1722c00640 1 -- start start 2024-03-23T03:48:35.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.538+0000 7f1722c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f171c0add30 0x7f171c0ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.538+0000 7f1722c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f171c06ce90 0x7f171c06d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.538+0000 7f1722c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f171c0cf250 con 0x7f171c0ce930 2024-03-23T03:48:35.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.538+0000 7f1722c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f171c0ae1c0 con 0x7f171c06ce90 2024-03-23T03:48:35.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.538+0000 7f1722c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f171c06c760 con 0x7f171c0add30 2024-03-23T03:48:35.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.538+0000 7f171be00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f171c0ce930 0x7f171c0cd3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53692/0 (socket says 172.21.15.102:53692) 2024-03-23T03:48:35.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.538+0000 7f171be00640 1 -- 172.21.15.102:0/142994606 learned_addr learned my addr 172.21.15.102:0/142994606 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:35.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.538+0000 7f1720c00640 1 --2- 172.21.15.102:0/142994606 >> v2:172.21.15.102:3300/0 conn(0x7f171c06ce90 0x7f171c06d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:35.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.538+0000 7f1713e00640 1 --2- 172.21.15.102:0/142994606 >> v2:172.21.15.173:3300/0 conn(0x7f171c0add30 0x7f171c0ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:35.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.538+0000 7f171b400640 1 -- 172.21.15.102:0/142994606 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942957117 0 0) 0x7f171c0cf250 con 0x7f171c0ce930 2024-03-23T03:48:35.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.539+0000 7f171b400640 1 -- 172.21.15.102:0/142994606 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1700003520 con 0x7f171c0ce930 2024-03-23T03:48:35.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.539+0000 7f1720c00640 1 -- 172.21.15.102:0/142994606 >> v2:172.21.15.173:3300/0 conn(0x7f171c0add30 msgr2=0x7f171c0ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:35.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.539+0000 7f1720c00640 1 --2- 172.21.15.102:0/142994606 >> v2:172.21.15.173:3300/0 conn(0x7f171c0add30 0x7f171c0ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.539+0000 7f1720c00640 1 -- 172.21.15.102:0/142994606 >> v1:172.21.15.102:6789/0 conn(0x7f171c0ce930 legacy=0x7f171c0cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:35.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.539+0000 7f1720c00640 1 -- 172.21.15.102:0/142994606 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f171c0cdba0 con 0x7f171c06ce90 2024-03-23T03:48:35.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.540+0000 7f1720c00640 1 --2- 172.21.15.102:0/142994606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f171c06ce90 0x7f171c06d2e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f171c0cf4f0 tx=0x7f171402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=733c068606b2332e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:35.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.540+0000 7f171b400640 1 -- 172.21.15.102:0/142994606 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f171402cc80 con 0x7f171c06ce90 2024-03-23T03:48:35.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.540+0000 7f171b400640 1 -- 172.21.15.102:0/142994606 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f171402cde0 con 0x7f171c06ce90 2024-03-23T03:48:35.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.540+0000 7f171b400640 1 -- 172.21.15.102:0/142994606 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1714034660 con 0x7f171c06ce90 2024-03-23T03:48:35.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.541+0000 7f1722c00640 1 -- 172.21.15.102:0/142994606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f171c06ce90 msgr2=0x7f171c06d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:35.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.541+0000 7f1722c00640 1 --2- 172.21.15.102:0/142994606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f171c06ce90 0x7f171c06d2e0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f171c0cf4f0 tx=0x7f171402bed0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.541+0000 7f1722c00640 1 -- 172.21.15.102:0/142994606 shutdown_connections 2024-03-23T03:48:35.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.541+0000 7f1722c00640 1 --2- 172.21.15.102:0/142994606 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f171c06ce90 0x7f171c06d2e0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.541+0000 7f1722c00640 1 --2- 172.21.15.102:0/142994606 >> v2:172.21.15.173:3300/0 conn(0x7f171c0add30 0x7f171c0ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.541+0000 7f1722c00640 1 -- 172.21.15.102:0/142994606 >> 172.21.15.102:0/142994606 conn(0x7f171c06a0b0 msgr2=0x7f171c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:35.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.541+0000 7f1722c00640 1 -- 172.21.15.102:0/142994606 shutdown_connections 2024-03-23T03:48:35.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.541+0000 7f1722c00640 1 -- 172.21.15.102:0/142994606 wait complete. 2024-03-23T03:48:35.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.542+0000 7f1722c00640 1 Processor -- start 2024-03-23T03:48:35.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.543+0000 7f1722c00640 1 -- start start 2024-03-23T03:48:35.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.543+0000 7f1722c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f171c06ce90 0x7f171c15a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.543+0000 7f1722c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f171c0add30 0x7f171c157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.543+0000 7f1722c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f171c0ce930 0x7f171c0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.543+0000 7f1722c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c0ab8b0 con 0x7f171c06ce90 2024-03-23T03:48:35.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.543+0000 7f1722c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f171c157880 con 0x7f171c0ce930 2024-03-23T03:48:35.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.543+0000 7f1722c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f171c0b3580 con 0x7f171c0add30 2024-03-23T03:48:35.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.544+0000 7f171be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f171c06ce90 0x7f171c15a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:35.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.544+0000 7f171be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f171c06ce90 0x7f171c15a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55206/0 (socket says 172.21.15.102:55206) 2024-03-23T03:48:35.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.544+0000 7f171be00640 1 -- 172.21.15.102:0/3876529903 learned_addr learned my addr 172.21.15.102:0/3876529903 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:35.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.544+0000 7f1720c00640 1 --2- 172.21.15.102:0/3876529903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f171c0ce930 0x7f171c0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:35.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.544+0000 7f171be00640 1 -- 172.21.15.102:0/3876529903 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f171c0add30 msgr2=0x7f171c157340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:35.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.544+0000 7f171be00640 1 --2- 172.21.15.102:0/3876529903 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f171c0add30 0x7f171c157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.544+0000 7f171be00640 1 -- 172.21.15.102:0/3876529903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f171c0ce930 msgr2=0x7f171c0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:35.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.544+0000 7f171be00640 1 --2- 172.21.15.102:0/3876529903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f171c0ce930 0x7f171c0b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.544+0000 7f171be00640 1 -- 172.21.15.102:0/3876529903 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1714009a70 con 0x7f171c06ce90 2024-03-23T03:48:35.546 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.544+0000 7f1720c00640 1 --2- 172.21.15.102:0/3876529903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f171c0ce930 0x7f171c0b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:35.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.544+0000 7f171be00640 1 --2- 172.21.15.102:0/3876529903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f171c06ce90 0x7f171c15a2e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f170c00c410 tx=0x7f170c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:35.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.545+0000 7f1718c00640 1 -- 172.21.15.102:0/3876529903 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170c005ce0 con 0x7f171c06ce90 2024-03-23T03:48:35.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.545+0000 7f1718c00640 1 -- 172.21.15.102:0/3876529903 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f170c005e40 con 0x7f171c06ce90 2024-03-23T03:48:35.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.545+0000 7f1718c00640 1 -- 172.21.15.102:0/3876529903 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f170c0072f0 con 0x7f171c06ce90 2024-03-23T03:48:35.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.545+0000 7f1722c00640 1 -- 172.21.15.102:0/3876529903 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f171c0abb00 con 0x7f171c06ce90 2024-03-23T03:48:35.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.545+0000 7f1722c00640 1 -- 172.21.15.102:0/3876529903 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f171c0ac0c0 con 0x7f171c06ce90 2024-03-23T03:48:35.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.546+0000 7f1722c00640 1 -- 172.21.15.102:0/3876529903 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f171c073390 con 0x7f171c06ce90 2024-03-23T03:48:35.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.547+0000 7f1718c00640 1 -- 172.21.15.102:0/3876529903 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f170c007510 con 0x7f171c06ce90 2024-03-23T03:48:35.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.548+0000 7f1718c00640 1 --2- 172.21.15.102:0/3876529903 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f16f806ac00 0x7f16f806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.548+0000 7f1718c00640 1 -- 172.21.15.102:0/3876529903 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f170c088bc0 con 0x7f171c06ce90 2024-03-23T03:48:35.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.548+0000 7f1713e00640 1 --2- 172.21.15.102:0/3876529903 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f16f806ac00 0x7f16f806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:35.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.549+0000 7f1713e00640 1 --2- 172.21.15.102:0/3876529903 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f16f806ac00 0x7f16f806d0c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f171c0b13b0 tx=0x7f1704008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:35.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.552+0000 7f1718c00640 1 -- 172.21.15.102:0/3876529903 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f170c053c80 con 0x7f171c06ce90 2024-03-23T03:48:35.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.743+0000 7f1722c00640 1 -- 172.21.15.102:0/3876529903 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f171c06c540 con 0x7f171c06ce90 2024-03-23T03:48:35.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.744+0000 7f1718c00640 1 -- 172.21.15.102:0/3876529903 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) v1 ==== 100+0+11 (secure 0 0 0) 0x7f170c0170a0 con 0x7f171c06ce90 2024-03-23T03:48:35.745 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:48:35.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.747+0000 7f1722c00640 1 -- 172.21.15.102:0/3876529903 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f16f806ac00 msgr2=0x7f16f806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:35.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.747+0000 7f1722c00640 1 --2- 172.21.15.102:0/3876529903 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f16f806ac00 0x7f16f806d0c0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f171c0b13b0 tx=0x7f1704008040 comp rx=0 tx=0).stop 2024-03-23T03:48:35.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.747+0000 7f1722c00640 1 -- 172.21.15.102:0/3876529903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f171c06ce90 msgr2=0x7f171c15a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:35.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.747+0000 7f1722c00640 1 --2- 172.21.15.102:0/3876529903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f171c06ce90 0x7f171c15a2e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f170c00c410 tx=0x7f170c00c8d0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.747+0000 7f1722c00640 1 -- 172.21.15.102:0/3876529903 shutdown_connections 2024-03-23T03:48:35.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.747+0000 7f1722c00640 1 --2- 172.21.15.102:0/3876529903 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f16f806ac00 0x7f16f806d0c0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.747+0000 7f1722c00640 1 --2- 172.21.15.102:0/3876529903 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f171c0ce930 0x7f171c0b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.747+0000 7f1722c00640 1 --2- 172.21.15.102:0/3876529903 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f171c0add30 0x7f171c157340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.747+0000 7f1722c00640 1 --2- 172.21.15.102:0/3876529903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f171c06ce90 0x7f171c15a2e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.747+0000 7f1722c00640 1 -- 172.21.15.102:0/3876529903 >> 172.21.15.102:0/3876529903 conn(0x7f171c06a0b0 msgr2=0x7f171c15c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:35.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.747+0000 7f1722c00640 1 -- 172.21.15.102:0/3876529903 shutdown_connections 2024-03-23T03:48:35.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.747+0000 7f1722c00640 1 -- 172.21.15.102:0/3876529903 wait complete. 2024-03-23T03:48:35.766 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:48:35.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.970+0000 7f0353e00640 1 Processor -- start 2024-03-23T03:48:35.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.970+0000 7f0353e00640 1 -- start start 2024-03-23T03:48:35.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.971+0000 7f0353e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f034c0cb660 0x7f034c0cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.971+0000 7f0353e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f034c06c2b0 0x7f034c06c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.971+0000 7f0353e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f034c15ae60 con 0x7f034c0ce930 2024-03-23T03:48:35.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.971+0000 7f0353e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f034c15bae0 con 0x7f034c06c2b0 2024-03-23T03:48:35.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.971+0000 7f0353e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034c15b140 con 0x7f034c0cb660 2024-03-23T03:48:35.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.972+0000 7f0351e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f034c06c2b0 0x7f034c06c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:35.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.972+0000 7f0350a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f034c0cb660 0x7f034c0cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:35.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.972+0000 7f0351e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f034c06c2b0 0x7f034c06c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:59344/0 (socket says 172.21.15.102:59344) 2024-03-23T03:48:35.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.972+0000 7f0351e00640 1 -- 172.21.15.102:0/2642746010 learned_addr learned my addr 172.21.15.102:0/2642746010 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:35.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.973+0000 7f0351e00640 1 -- 172.21.15.102:0/2642746010 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f034c0cb660 msgr2=0x7f034c0cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:35.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.973+0000 7f0351e00640 1 --2- 172.21.15.102:0/2642746010 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f034c0cb660 0x7f034c0cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.973+0000 7f0351e00640 1 -- 172.21.15.102:0/2642746010 >> v1:172.21.15.173:6789/0 conn(0x7f034c0ce930 legacy=0x7f034c0ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:35.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.973+0000 7f0351e00640 1 -- 172.21.15.102:0/2642746010 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f034c15b420 con 0x7f034c06c2b0 2024-03-23T03:48:35.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.974+0000 7f0351e00640 1 --2- 172.21.15.102:0/2642746010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f034c06c2b0 0x7f034c06c740 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0344009e50 tx=0x7f034402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ec4dabde0d7389b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:35.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.974+0000 7f034be00640 1 -- 172.21.15.102:0/2642746010 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034402cb00 con 0x7f034c06c2b0 2024-03-23T03:48:35.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.975+0000 7f034be00640 1 -- 172.21.15.102:0/2642746010 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034402cc60 con 0x7f034c06c2b0 2024-03-23T03:48:35.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.975+0000 7f034be00640 1 -- 172.21.15.102:0/2642746010 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03440343f0 con 0x7f034c06c2b0 2024-03-23T03:48:35.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.975+0000 7f0353e00640 1 -- 172.21.15.102:0/2642746010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f034c06c2b0 msgr2=0x7f034c06c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:35.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.975+0000 7f0353e00640 1 --2- 172.21.15.102:0/2642746010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f034c06c2b0 0x7f034c06c740 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0344009e50 tx=0x7f034402bf60 comp rx=0 tx=0).stop 2024-03-23T03:48:35.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.975+0000 7f0353e00640 1 -- 172.21.15.102:0/2642746010 shutdown_connections 2024-03-23T03:48:35.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.975+0000 7f0353e00640 1 --2- 172.21.15.102:0/2642746010 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f034c06c2b0 0x7f034c06c740 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.975+0000 7f0353e00640 1 --2- 172.21.15.102:0/2642746010 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f034c0cb660 0x7f034c0cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.975+0000 7f0353e00640 1 -- 172.21.15.102:0/2642746010 >> 172.21.15.102:0/2642746010 conn(0x7f034c06a0b0 msgr2=0x7f034c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:35.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.975+0000 7f0353e00640 1 -- 172.21.15.102:0/2642746010 shutdown_connections 2024-03-23T03:48:35.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.975+0000 7f0353e00640 1 -- 172.21.15.102:0/2642746010 wait complete. 2024-03-23T03:48:35.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.977+0000 7f0353e00640 1 Processor -- start 2024-03-23T03:48:35.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.978+0000 7f0353e00640 1 -- start start 2024-03-23T03:48:35.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.979+0000 7f0353e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f034c06c2b0 0x7f034c0b0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.979+0000 7f0353e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f034c0cb660 0x7f034c1ef6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.979+0000 7f0353e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f034c0ce930 0x7f034c0af120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.979+0000 7f0353e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c14e740 con 0x7f034c0ce930 2024-03-23T03:48:35.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.979+0000 7f0353e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f034c14d100 con 0x7f034c0cb660 2024-03-23T03:48:35.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.979+0000 7f0353e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f034c14d3e0 con 0x7f034c06c2b0 2024-03-23T03:48:35.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.979+0000 7f0351e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f034c0ce930 0x7f034c0af120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:35.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.979+0000 7f0351e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f034c0ce930 0x7f034c0af120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55212/0 (socket says 172.21.15.102:55212) 2024-03-23T03:48:35.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.979+0000 7f0351e00640 1 -- 172.21.15.102:0/116028459 learned_addr learned my addr 172.21.15.102:0/116028459 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:35.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.979+0000 7f0351400640 1 --2- 172.21.15.102:0/116028459 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f034c06c2b0 0x7f034c0b0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:35.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.979+0000 7f0350a00640 1 --2- 172.21.15.102:0/116028459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f034c0cb660 0x7f034c1ef6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:35.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.979+0000 7f0351e00640 1 -- 172.21.15.102:0/116028459 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f034c06c2b0 msgr2=0x7f034c0b0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:35.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.979+0000 7f0351e00640 1 --2- 172.21.15.102:0/116028459 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f034c06c2b0 0x7f034c0b0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.980+0000 7f0351e00640 1 -- 172.21.15.102:0/116028459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f034c0cb660 msgr2=0x7f034c1ef6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:35.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.980+0000 7f0351e00640 1 --2- 172.21.15.102:0/116028459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f034c0cb660 0x7f034c1ef6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:35.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.980+0000 7f0351e00640 1 -- 172.21.15.102:0/116028459 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0344009b30 con 0x7f034c0ce930 2024-03-23T03:48:35.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.980+0000 7f0351e00640 1 --2- 172.21.15.102:0/116028459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f034c0ce930 0x7f034c0af120 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0344036040 tx=0x7f0344004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:35.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.980+0000 7f0349600640 1 -- 172.21.15.102:0/116028459 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0344009510 con 0x7f034c0ce930 2024-03-23T03:48:35.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.980+0000 7f0349600640 1 -- 172.21.15.102:0/116028459 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f034403e450 con 0x7f034c0ce930 2024-03-23T03:48:35.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.980+0000 7f0353e00640 1 -- 172.21.15.102:0/116028459 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f034c155280 con 0x7f034c0ce930 2024-03-23T03:48:35.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.980+0000 7f0349600640 1 -- 172.21.15.102:0/116028459 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034403e820 con 0x7f034c0ce930 2024-03-23T03:48:35.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.980+0000 7f0353e00640 1 -- 172.21.15.102:0/116028459 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f034c14d9a0 con 0x7f034c0ce930 2024-03-23T03:48:35.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.981+0000 7f0353e00640 1 -- 172.21.15.102:0/116028459 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f034c14e8f0 con 0x7f034c0ce930 2024-03-23T03:48:35.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.982+0000 7f0349600640 1 -- 172.21.15.102:0/116028459 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f034403e980 con 0x7f034c0ce930 2024-03-23T03:48:35.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.983+0000 7f0349600640 1 --2- 172.21.15.102:0/116028459 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f032c06ac00 0x7f032c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:35.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.983+0000 7f0349600640 1 -- 172.21.15.102:0/116028459 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f03440ac220 con 0x7f034c0ce930 2024-03-23T03:48:35.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.983+0000 7f0351400640 1 --2- 172.21.15.102:0/116028459 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f032c06ac00 0x7f032c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:35.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.984+0000 7f0351400640 1 --2- 172.21.15.102:0/116028459 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f032c06ac00 0x7f032c06d0c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f03400015a0 tx=0x7f0340009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:35.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:35.988+0000 7f0349600640 1 -- 172.21.15.102:0/116028459 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f034c14e8f0 con 0x7f034c0ce930 2024-03-23T03:48:36.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.192+0000 7f0353e00640 1 -- 172.21.15.102:0/116028459 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f034c073390 con 0x7f034c0ce930 2024-03-23T03:48:36.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.192+0000 7f0349600640 1 -- 172.21.15.102:0/116028459 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v41) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0344077220 con 0x7f034c0ce930 2024-03-23T03:48:36.193 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:48:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.195+0000 7f0353e00640 1 -- 172.21.15.102:0/116028459 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f032c06ac00 msgr2=0x7f032c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.195+0000 7f0353e00640 1 --2- 172.21.15.102:0/116028459 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f032c06ac00 0x7f032c06d0c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f03400015a0 tx=0x7f0340009040 comp rx=0 tx=0).stop 2024-03-23T03:48:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.195+0000 7f0353e00640 1 -- 172.21.15.102:0/116028459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f034c0ce930 msgr2=0x7f034c0af120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.195+0000 7f0353e00640 1 --2- 172.21.15.102:0/116028459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f034c0ce930 0x7f034c0af120 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0344036040 tx=0x7f0344004a40 comp rx=0 tx=0).stop 2024-03-23T03:48:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.195+0000 7f0353e00640 1 -- 172.21.15.102:0/116028459 shutdown_connections 2024-03-23T03:48:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.195+0000 7f0353e00640 1 --2- 172.21.15.102:0/116028459 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f032c06ac00 0x7f032c06d0c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.195+0000 7f0353e00640 1 --2- 172.21.15.102:0/116028459 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f034c0ce930 0x7f034c0af120 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.195+0000 7f0353e00640 1 --2- 172.21.15.102:0/116028459 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f034c0cb660 0x7f034c1ef6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.195+0000 7f0353e00640 1 --2- 172.21.15.102:0/116028459 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f034c06c2b0 0x7f034c0b0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:36.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.195+0000 7f0353e00640 1 -- 172.21.15.102:0/116028459 >> 172.21.15.102:0/116028459 conn(0x7f034c06a0b0 msgr2=0x7f034c159bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:36.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.195+0000 7f0353e00640 1 -- 172.21.15.102:0/116028459 shutdown_connections 2024-03-23T03:48:36.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.195+0000 7f0353e00640 1 -- 172.21.15.102:0/116028459 wait complete. 2024-03-23T03:48:36.214 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-23T03:48:36.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:36.214 DEBUG:teuthology.orchestra.run.smithi102:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-23T03:48:36.317 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:48:36.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:36.318 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T03:48:36.366 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:48:36.366 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:48:36.423 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:48:36.424 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:48:36.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:48:36.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.693+0000 7fab8ac00640 1 Processor -- start 2024-03-23T03:48:36.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.693+0000 7fab8ac00640 1 -- start start 2024-03-23T03:48:36.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.695+0000 7fab8ac00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fab8415c960 0x7fab84160670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:36.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.695+0000 7fab8ac00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fab8415cd40 0x7fab840cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:36.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.695+0000 7fab8ac00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab84160bb0 0x7fab840cf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:36.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.695+0000 7fab8ac00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab840d0980 con 0x7fab8415cd40 2024-03-23T03:48:36.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.695+0000 7fab8ac00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab840d0f40 con 0x7fab8415c960 2024-03-23T03:48:36.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.695+0000 7fab8ac00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab840b87e0 con 0x7fab84160bb0 2024-03-23T03:48:36.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.695+0000 7fab83400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fab8415cd40 0x7fab840cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:36.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.695+0000 7fab83400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fab8415cd40 0x7fab840cf2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55214/0 (socket says 172.21.15.102:55214) 2024-03-23T03:48:36.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.695+0000 7fab83400640 1 -- 172.21.15.102:0/498520991 learned_addr learned my addr 172.21.15.102:0/498520991 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:36.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.696+0000 7fab88c00640 1 --2- 172.21.15.102:0/498520991 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab84160bb0 0x7fab840cf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:36.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.696+0000 7fab83e00640 1 --2- 172.21.15.102:0/498520991 >> v2:172.21.15.173:3300/0 conn(0x7fab8415c960 0x7fab84160670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:36.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.696+0000 7fab83400640 1 -- 172.21.15.102:0/498520991 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab84160bb0 msgr2=0x7fab840cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:36.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.696+0000 7fab83400640 1 --2- 172.21.15.102:0/498520991 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab84160bb0 0x7fab840cf800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:36.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.696+0000 7fab83400640 1 -- 172.21.15.102:0/498520991 >> v2:172.21.15.173:3300/0 conn(0x7fab8415c960 msgr2=0x7fab84160670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:36.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.696+0000 7fab83400640 1 --2- 172.21.15.102:0/498520991 >> v2:172.21.15.173:3300/0 conn(0x7fab8415c960 0x7fab84160670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:36.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.696+0000 7fab83400640 1 -- 172.21.15.102:0/498520991 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab84161130 con 0x7fab8415cd40 2024-03-23T03:48:36.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.697+0000 7fab83400640 1 --2- 172.21.15.102:0/498520991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab8415cd40 0x7fab840cf2c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fab8406c4d0 tx=0x7fab6c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6019c47c76ffeaa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:36.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.697+0000 7fab82a00640 1 -- 172.21.15.102:0/498520991 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c030860 con 0x7fab8415cd40 2024-03-23T03:48:36.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.697+0000 7fab82a00640 1 -- 172.21.15.102:0/498520991 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6c0309c0 con 0x7fab8415cd40 2024-03-23T03:48:36.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.697+0000 7fab82a00640 1 -- 172.21.15.102:0/498520991 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c030d50 con 0x7fab8415cd40 2024-03-23T03:48:36.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.698+0000 7fab8ac00640 1 -- 172.21.15.102:0/498520991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab8415cd40 msgr2=0x7fab840cf2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:36.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.698+0000 7fab8ac00640 1 --2- 172.21.15.102:0/498520991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab8415cd40 0x7fab840cf2c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fab8406c4d0 tx=0x7fab6c02b9a0 comp rx=0 tx=0).stop 2024-03-23T03:48:36.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.698+0000 7fab8ac00640 1 -- 172.21.15.102:0/498520991 shutdown_connections 2024-03-23T03:48:36.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.698+0000 7fab8ac00640 1 --2- 172.21.15.102:0/498520991 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab84160bb0 0x7fab840cf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:36.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.698+0000 7fab8ac00640 1 --2- 172.21.15.102:0/498520991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab8415cd40 0x7fab840cf2c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:36.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.698+0000 7fab8ac00640 1 --2- 172.21.15.102:0/498520991 >> v2:172.21.15.173:3300/0 conn(0x7fab8415c960 0x7fab84160670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:36.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.698+0000 7fab8ac00640 1 -- 172.21.15.102:0/498520991 >> 172.21.15.102:0/498520991 conn(0x7fab840713c0 msgr2=0x7fab840717d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:36.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.698+0000 7fab8ac00640 1 -- 172.21.15.102:0/498520991 shutdown_connections 2024-03-23T03:48:36.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.698+0000 7fab8ac00640 1 -- 172.21.15.102:0/498520991 wait complete. 2024-03-23T03:48:36.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.699+0000 7fab8ac00640 1 Processor -- start 2024-03-23T03:48:36.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.700+0000 7fab8ac00640 1 -- start start 2024-03-23T03:48:36.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.700+0000 7fab8ac00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab8415c960 0x7fab8415d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:36.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.701+0000 7fab8ac00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab8415cd40 0x7fab840b7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:36.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.701+0000 7fab8ac00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab84160bb0 0x7fab840b4b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:36.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.701+0000 7fab8ac00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab840cbd70 con 0x7fab8415cd40 2024-03-23T03:48:36.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.701+0000 7fab8ac00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab840b77b0 con 0x7fab84160bb0 2024-03-23T03:48:36.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.701+0000 7fab8ac00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab840cae80 con 0x7fab8415c960 2024-03-23T03:48:36.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.701+0000 7fab83400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab8415cd40 0x7fab840b7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:36.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.701+0000 7fab83400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab8415cd40 0x7fab840b7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55222/0 (socket says 172.21.15.102:55222) 2024-03-23T03:48:36.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.701+0000 7fab83400640 1 -- 172.21.15.102:0/960070683 learned_addr learned my addr 172.21.15.102:0/960070683 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:36.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.701+0000 7fab83e00640 1 --2- 172.21.15.102:0/960070683 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab8415c960 0x7fab8415d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:36.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.701+0000 7fab83400640 1 -- 172.21.15.102:0/960070683 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab8415c960 msgr2=0x7fab8415d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:36.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.701+0000 7fab83400640 1 --2- 172.21.15.102:0/960070683 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab8415c960 0x7fab8415d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:36.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.701+0000 7fab83400640 1 -- 172.21.15.102:0/960070683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab84160bb0 msgr2=0x7fab840b4b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:36.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.701+0000 7fab83400640 1 --2- 172.21.15.102:0/960070683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab84160bb0 0x7fab840b4b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:36.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.701+0000 7fab83400640 1 -- 172.21.15.102:0/960070683 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6c0094d0 con 0x7fab8415cd40 2024-03-23T03:48:36.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.702+0000 7fab83400640 1 --2- 172.21.15.102:0/960070683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab8415cd40 0x7fab840b7270 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fab6c02b9e0 tx=0x7fab6c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:36.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.702+0000 7fab7be00640 1 -- 172.21.15.102:0/960070683 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c039d60 con 0x7fab8415cd40 2024-03-23T03:48:36.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.702+0000 7fab7be00640 1 -- 172.21.15.102:0/960070683 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6c043400 con 0x7fab8415cd40 2024-03-23T03:48:36.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.702+0000 7fab7be00640 1 -- 172.21.15.102:0/960070683 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c0437b0 con 0x7fab8415cd40 2024-03-23T03:48:36.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.702+0000 7fab8ac00640 1 -- 172.21.15.102:0/960070683 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab840cb160 con 0x7fab8415cd40 2024-03-23T03:48:36.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.702+0000 7fab8ac00640 1 -- 172.21.15.102:0/960070683 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab840cb720 con 0x7fab8415cd40 2024-03-23T03:48:36.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.703+0000 7fab8ac00640 1 -- 172.21.15.102:0/960070683 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab84073390 con 0x7fab8415cd40 2024-03-23T03:48:36.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.704+0000 7fab7be00640 1 -- 172.21.15.102:0/960070683 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fab6c043910 con 0x7fab8415cd40 2024-03-23T03:48:36.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.705+0000 7fab7be00640 1 --2- 172.21.15.102:0/960070683 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab6006ac00 0x7fab6006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:36.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.705+0000 7fab7be00640 1 -- 172.21.15.102:0/960070683 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fab6c03e070 con 0x7fab8415cd40 2024-03-23T03:48:36.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.705+0000 7fab83e00640 1 --2- 172.21.15.102:0/960070683 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab6006ac00 0x7fab6006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:36.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.706+0000 7fab83e00640 1 --2- 172.21.15.102:0/960070683 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab6006ac00 0x7fab6006d0c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fab74006fd0 tx=0x7fab74008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:36.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.709+0000 7fab7be00640 1 -- 172.21.15.102:0/960070683 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fab6c077790 con 0x7fab8415cd40 2024-03-23T03:48:36.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.904+0000 7fab8ac00640 1 -- 172.21.15.102:0/960070683 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab840d0530 con 0x7fab8415cd40 2024-03-23T03:48:36.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.908+0000 7fab7be00640 1 -- 172.21.15.102:0/960070683 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+16354 (secure 0 0 0) 0x7fab6c07adb0 con 0x7fab8415cd40 2024-03-23T03:48:36.909 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:36.909 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":41,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:24.423308+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:48:21.550281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-23T03:48:22.428475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:48:36.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.910+0000 7fab8ac00640 1 -- 172.21.15.102:0/960070683 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab6006ac00 msgr2=0x7fab6006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:36.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.910+0000 7fab8ac00640 1 --2- 172.21.15.102:0/960070683 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab6006ac00 0x7fab6006d0c0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fab74006fd0 tx=0x7fab74008040 comp rx=0 tx=0).stop 2024-03-23T03:48:36.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.910+0000 7fab8ac00640 1 -- 172.21.15.102:0/960070683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab8415cd40 msgr2=0x7fab840b7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:36.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.910+0000 7fab8ac00640 1 --2- 172.21.15.102:0/960070683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab8415cd40 0x7fab840b7270 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fab6c02b9e0 tx=0x7fab6c004730 comp rx=0 tx=0).stop 2024-03-23T03:48:36.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.910+0000 7fab8ac00640 1 -- 172.21.15.102:0/960070683 shutdown_connections 2024-03-23T03:48:36.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.910+0000 7fab8ac00640 1 --2- 172.21.15.102:0/960070683 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab6006ac00 0x7fab6006d0c0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:36.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.910+0000 7fab8ac00640 1 --2- 172.21.15.102:0/960070683 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab84160bb0 0x7fab840b4b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:36.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.910+0000 7fab8ac00640 1 --2- 172.21.15.102:0/960070683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab8415cd40 0x7fab840b7270 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:36.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.910+0000 7fab8ac00640 1 --2- 172.21.15.102:0/960070683 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab8415c960 0x7fab8415d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:36.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.910+0000 7fab8ac00640 1 -- 172.21.15.102:0/960070683 >> 172.21.15.102:0/960070683 conn(0x7fab840713c0 msgr2=0x7fab840cd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:36.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.911+0000 7fab8ac00640 1 -- 172.21.15.102:0/960070683 shutdown_connections 2024-03-23T03:48:36.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:36.911+0000 7fab8ac00640 1 -- 172.21.15.102:0/960070683 wait complete. 2024-03-23T03:48:36.931 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:48:21.550281+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:48:22.428475+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:48:36.931 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:48:37.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.143+0000 7fcfdf200640 1 Processor -- start 2024-03-23T03:48:37.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.143+0000 7fcfdf200640 1 -- start start 2024-03-23T03:48:37.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.145+0000 7fcfdf200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcfd80b5b90 0x7fcfd80b5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:37.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.145+0000 7fcfdf200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fcfd80cf510 0x7fcfd80cf9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:37.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.145+0000 7fcfdf200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd8160ae0 con 0x7fcfd815cc30 2024-03-23T03:48:37.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.145+0000 7fcfdf200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd80cfee0 con 0x7fcfd80cf510 2024-03-23T03:48:37.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.145+0000 7fcfdf200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd80b6510 con 0x7fcfd80b5b90 2024-03-23T03:48:37.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.145+0000 7fcfde800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fcfd80cf510 0x7fcfd80cf9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:37.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.145+0000 7fcfdd400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcfd80b5b90 0x7fcfd80b5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:37.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.145+0000 7fcfdde00640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7fcfd815cc30 0x7fcfd80d16a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:48262/0 (socket says 172.21.15.102:48262) 2024-03-23T03:48:37.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.145+0000 7fcfdd400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcfd80b5b90 0x7fcfd80b5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:51378/0 (socket says 172.21.15.102:51378) 2024-03-23T03:48:37.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.146+0000 7fcfdde00640 1 -- 172.21.15.102:0/841367046 learned_addr learned my addr 172.21.15.102:0/841367046 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:37.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.145+0000 7fcfde800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fcfd80cf510 0x7fcfd80cf9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:59366/0 (socket says 172.21.15.102:59366) 2024-03-23T03:48:37.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.146+0000 7fcfdd400640 1 -- 172.21.15.102:0/841367046 >> v2:172.21.15.173:3300/0 conn(0x7fcfd80cf510 msgr2=0x7fcfd80cf9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:37.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.146+0000 7fcfdd400640 1 --2- 172.21.15.102:0/841367046 >> v2:172.21.15.173:3300/0 conn(0x7fcfd80cf510 0x7fcfd80cf9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.146+0000 7fcfdd400640 1 -- 172.21.15.102:0/841367046 >> v1:172.21.15.173:6789/0 conn(0x7fcfd815cc30 legacy=0x7fcfd80d16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:37.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.146+0000 7fcfdd400640 1 -- 172.21.15.102:0/841367046 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd80d0c00 con 0x7fcfd80b5b90 2024-03-23T03:48:37.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.148+0000 7fcfdd400640 1 --2- 172.21.15.102:0/841367046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcfd80b5b90 0x7fcfd80b5fd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80097a0 tx=0x7fcfc802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=214a452da0deed19 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:37.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.148+0000 7fcfdca00640 1 -- 172.21.15.102:0/841367046 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8031830 con 0x7fcfd80b5b90 2024-03-23T03:48:37.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.148+0000 7fcfdca00640 1 -- 172.21.15.102:0/841367046 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc8031990 con 0x7fcfd80b5b90 2024-03-23T03:48:37.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.148+0000 7fcfdca00640 1 -- 172.21.15.102:0/841367046 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc8031d20 con 0x7fcfd80b5b90 2024-03-23T03:48:37.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.149+0000 7fcfdf200640 1 -- 172.21.15.102:0/841367046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcfd80b5b90 msgr2=0x7fcfd80b5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:37.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.149+0000 7fcfdf200640 1 --2- 172.21.15.102:0/841367046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcfd80b5b90 0x7fcfd80b5fd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80097a0 tx=0x7fcfc802b8d0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.149+0000 7fcfdf200640 1 -- 172.21.15.102:0/841367046 shutdown_connections 2024-03-23T03:48:37.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.149+0000 7fcfdf200640 1 --2- 172.21.15.102:0/841367046 >> v2:172.21.15.173:3300/0 conn(0x7fcfd80cf510 0x7fcfd80cf9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.149+0000 7fcfdf200640 1 --2- 172.21.15.102:0/841367046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcfd80b5b90 0x7fcfd80b5fd0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.149+0000 7fcfdf200640 1 -- 172.21.15.102:0/841367046 >> 172.21.15.102:0/841367046 conn(0x7fcfd806b100 msgr2=0x7fcfd806b510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:37.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.149+0000 7fcfdf200640 1 -- 172.21.15.102:0/841367046 shutdown_connections 2024-03-23T03:48:37.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.149+0000 7fcfdf200640 1 -- 172.21.15.102:0/841367046 wait complete. 2024-03-23T03:48:37.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.151+0000 7fcfdf200640 1 Processor -- start 2024-03-23T03:48:37.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.152+0000 7fcfdf200640 1 -- start start 2024-03-23T03:48:37.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.153+0000 7fcfdf200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcfd80b5b90 0x7fcfd80b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:37.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.153+0000 7fcfdf200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfd80cf510 0x7fcfd81590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:37.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.153+0000 7fcfdf200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfd815cc30 0x7fcfd80c80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:37.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.153+0000 7fcfdf200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd80c8910 con 0x7fcfd80cf510 2024-03-23T03:48:37.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.153+0000 7fcfdf200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd81f2c10 con 0x7fcfd815cc30 2024-03-23T03:48:37.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.153+0000 7fcfdf200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd81f2080 con 0x7fcfd80b5b90 2024-03-23T03:48:37.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.153+0000 7fcfdd400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfd80cf510 0x7fcfd81590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:37.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.153+0000 7fcfdde00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcfd80b5b90 0x7fcfd80b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:37.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.153+0000 7fcfdd400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfd80cf510 0x7fcfd81590a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55238/0 (socket says 172.21.15.102:55238) 2024-03-23T03:48:37.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.153+0000 7fcfdd400640 1 -- 172.21.15.102:0/2788805665 learned_addr learned my addr 172.21.15.102:0/2788805665 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:37.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.154+0000 7fcfde800640 1 --2- 172.21.15.102:0/2788805665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfd815cc30 0x7fcfd80c80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:37.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.154+0000 7fcfdd400640 1 -- 172.21.15.102:0/2788805665 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcfd80b5b90 msgr2=0x7fcfd80b2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:37.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.154+0000 7fcfdd400640 1 --2- 172.21.15.102:0/2788805665 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcfd80b5b90 0x7fcfd80b2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.154+0000 7fcfdd400640 1 -- 172.21.15.102:0/2788805665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfd815cc30 msgr2=0x7fcfd80c80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:37.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.154+0000 7fcfdd400640 1 --2- 172.21.15.102:0/2788805665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfd815cc30 0x7fcfd80c80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.154+0000 7fcfdd400640 1 -- 172.21.15.102:0/2788805665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfc8009400 con 0x7fcfd80cf510 2024-03-23T03:48:37.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.154+0000 7fcfdd400640 1 --2- 172.21.15.102:0/2788805665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfd80cf510 0x7fcfd81590a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80098c0 tx=0x7fcfc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:37.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.155+0000 7fcfd2000640 1 -- 172.21.15.102:0/2788805665 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc803a6f0 con 0x7fcfd80cf510 2024-03-23T03:48:37.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.155+0000 7fcfd2000640 1 -- 172.21.15.102:0/2788805665 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc803a870 con 0x7fcfd80cf510 2024-03-23T03:48:37.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.155+0000 7fcfd2000640 1 -- 172.21.15.102:0/2788805665 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfc803ac20 con 0x7fcfd80cf510 2024-03-23T03:48:37.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.155+0000 7fcfdf200640 1 -- 172.21.15.102:0/2788805665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd81595e0 con 0x7fcfd80cf510 2024-03-23T03:48:37.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.155+0000 7fcfdf200640 1 -- 172.21.15.102:0/2788805665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd81f25c0 con 0x7fcfd80cf510 2024-03-23T03:48:37.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.156+0000 7fcfdf200640 1 -- 172.21.15.102:0/2788805665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd804e030 con 0x7fcfd80cf510 2024-03-23T03:48:37.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.157+0000 7fcfd2000640 1 -- 172.21.15.102:0/2788805665 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcfc8030380 con 0x7fcfd80cf510 2024-03-23T03:48:37.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.158+0000 7fcfd2000640 1 --2- 172.21.15.102:0/2788805665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcfbc06ac00 0x7fcfbc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:37.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.158+0000 7fcfd2000640 1 -- 172.21.15.102:0/2788805665 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fcfc80aad40 con 0x7fcfd80cf510 2024-03-23T03:48:37.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.158+0000 7fcfdde00640 1 --2- 172.21.15.102:0/2788805665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcfbc06ac00 0x7fcfbc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:37.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.159+0000 7fcfdde00640 1 --2- 172.21.15.102:0/2788805665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcfbc06ac00 0x7fcfbc06d0c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc005880 tx=0x7fcfcc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:37.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.162+0000 7fcfd2000640 1 -- 172.21.15.102:0/2788805665 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcfc8075e80 con 0x7fcfd80cf510 2024-03-23T03:48:37.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.354+0000 7fcfdf200640 1 -- 172.21.15.102:0/2788805665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcfd80d0710 con 0x7fcfd80cf510 2024-03-23T03:48:37.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.355+0000 7fcfd2000640 1 -- 172.21.15.102:0/2788805665 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcfc80480a0 con 0x7fcfd80cf510 2024-03-23T03:48:37.356 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:48:37.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.357+0000 7fcfdf200640 1 -- 172.21.15.102:0/2788805665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcfbc06ac00 msgr2=0x7fcfbc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:37.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.357+0000 7fcfdf200640 1 --2- 172.21.15.102:0/2788805665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcfbc06ac00 0x7fcfbc06d0c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc005880 tx=0x7fcfcc009040 comp rx=0 tx=0).stop 2024-03-23T03:48:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.357+0000 7fcfdf200640 1 -- 172.21.15.102:0/2788805665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfd80cf510 msgr2=0x7fcfd81590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.357+0000 7fcfdf200640 1 --2- 172.21.15.102:0/2788805665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfd80cf510 0x7fcfd81590a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fcfc80098c0 tx=0x7fcfc80046c0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.357+0000 7fcfdf200640 1 -- 172.21.15.102:0/2788805665 shutdown_connections 2024-03-23T03:48:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.357+0000 7fcfdf200640 1 --2- 172.21.15.102:0/2788805665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcfbc06ac00 0x7fcfbc06d0c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.358+0000 7fcfdf200640 1 --2- 172.21.15.102:0/2788805665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcfd815cc30 0x7fcfd80c80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.358+0000 7fcfdf200640 1 --2- 172.21.15.102:0/2788805665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfd80cf510 0x7fcfd81590a0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.358+0000 7fcfdf200640 1 --2- 172.21.15.102:0/2788805665 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcfd80b5b90 0x7fcfd80b2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.358+0000 7fcfdf200640 1 -- 172.21.15.102:0/2788805665 >> 172.21.15.102:0/2788805665 conn(0x7fcfd806b100 msgr2=0x7fcfd80c64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.358+0000 7fcfdf200640 1 -- 172.21.15.102:0/2788805665 shutdown_connections 2024-03-23T03:48:37.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.358+0000 7fcfdf200640 1 -- 172.21.15.102:0/2788805665 wait complete. 2024-03-23T03:48:37.378 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:48:37.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.590+0000 7fe8a4600640 1 Processor -- start 2024-03-23T03:48:37.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.590+0000 7fe8a4600640 1 -- start start 2024-03-23T03:48:37.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.592+0000 7fe8a4600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe89c0cd740 0x7fe89c0b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:37.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.592+0000 7fe8a4600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe89c0d0d40 0x7fe89c0cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:37.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.592+0000 7fe8a4600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89c0d20c0 con 0x7fe89c06b0a0 2024-03-23T03:48:37.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.592+0000 7fe8a4600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89c0ce060 con 0x7fe89c0cd740 2024-03-23T03:48:37.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.592+0000 7fe8a4600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0d14b0 con 0x7fe89c0d0d40 2024-03-23T03:48:37.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.592+0000 7fe8a2600640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe89c06b0a0 0x7fe89c06b530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53702/0 (socket says 172.21.15.102:53702) 2024-03-23T03:48:37.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.592+0000 7fe8a2600640 1 -- 172.21.15.102:0/3018923677 learned_addr learned my addr 172.21.15.102:0/3018923677 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:37.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe8a1c00640 1 --2- 172.21.15.102:0/3018923677 >> v2:172.21.15.173:3300/0 conn(0x7fe89c0cd740 0x7fe89c0b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:37.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe8a1200640 1 --2- 172.21.15.102:0/3018923677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe89c0d0d40 0x7fe89c0cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:37.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe89be00640 1 -- 172.21.15.102:0/3018923677 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407676854 0 0) 0x7fe89c0d20c0 con 0x7fe89c06b0a0 2024-03-23T03:48:37.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe89be00640 1 -- 172.21.15.102:0/3018923677 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe880003520 con 0x7fe89c06b0a0 2024-03-23T03:48:37.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe89be00640 1 -- 172.21.15.102:0/3018923677 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 435099473 0 0) 0x7fe880003520 con 0x7fe89c06b0a0 2024-03-23T03:48:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe89be00640 1 -- 172.21.15.102:0/3018923677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe89c0d0d40 msgr2=0x7fe89c0cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe89be00640 1 --2- 172.21.15.102:0/3018923677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe89c0d0d40 0x7fe89c0cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe89be00640 1 -- 172.21.15.102:0/3018923677 >> v2:172.21.15.173:3300/0 conn(0x7fe89c0cd740 msgr2=0x7fe89c0b84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe89be00640 1 --2- 172.21.15.102:0/3018923677 >> v2:172.21.15.173:3300/0 conn(0x7fe89c0cd740 0x7fe89c0b84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe89be00640 1 -- 172.21.15.102:0/3018923677 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c0d1790 con 0x7fe89c06b0a0 2024-03-23T03:48:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe89be00640 1 -- 172.21.15.102:0/3018923677 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fe894001670 con 0x7fe89c06b0a0 2024-03-23T03:48:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe89be00640 1 -- 172.21.15.102:0/3018923677 >> v1:172.21.15.102:6789/0 conn(0x7fe89c06b0a0 legacy=0x7fe89c06b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe89be00640 1 --2- 172.21.15.102:0/3018923677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe880003970 0x7fe880003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe89be00640 1 --2- 172.21.15.102:0/3018923677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8800047c0 0x7fe880004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe89be00640 1 --2- 172.21.15.102:0/3018923677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe89c0d0d40 0x7fe880005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe89be00640 1 -- 172.21.15.102:0/3018923677 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0ce060 con 0x7fe880003970 2024-03-23T03:48:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe89be00640 1 -- 172.21.15.102:0/3018923677 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0d14b0 con 0x7fe8800047c0 2024-03-23T03:48:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe89be00640 1 -- 172.21.15.102:0/3018923677 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe88000b980 con 0x7fe89c0d0d40 2024-03-23T03:48:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.593+0000 7fe8a1c00640 1 --2- 172.21.15.102:0/3018923677 >> v2:172.21.15.173:3300/0 conn(0x7fe89c0cd740 0x7fe89c0b84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:48:37.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.594+0000 7fe8a1c00640 1 --2- 172.21.15.102:0/3018923677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe880003970 0x7fe880003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:37.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.594+0000 7fe8a1200640 1 --2- 172.21.15.102:0/3018923677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8800047c0 0x7fe880004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:37.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.594+0000 7fe8a2600640 1 --2- 172.21.15.102:0/3018923677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe89c0d0d40 0x7fe880005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:37.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.594+0000 7fe8a1c00640 1 -- 172.21.15.102:0/3018923677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe89c0d0d40 msgr2=0x7fe880005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:37.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.594+0000 7fe8a1c00640 1 --2- 172.21.15.102:0/3018923677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe89c0d0d40 0x7fe880005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.594+0000 7fe8a1c00640 1 -- 172.21.15.102:0/3018923677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8800047c0 msgr2=0x7fe880004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:37.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.594+0000 7fe8a1c00640 1 --2- 172.21.15.102:0/3018923677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8800047c0 0x7fe880004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.594+0000 7fe8a1c00640 1 -- 172.21.15.102:0/3018923677 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe880003520 con 0x7fe880003970 2024-03-23T03:48:37.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.595+0000 7fe8a1c00640 1 --2- 172.21.15.102:0/3018923677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe880003970 0x7fe880003d90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe88c0098a0 tx=0x7fe8800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=402ac280ede5430f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:37.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.595+0000 7fe89be00640 1 -- 172.21.15.102:0/3018923677 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c037600 con 0x7fe880003970 2024-03-23T03:48:37.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.596+0000 7fe89be00640 1 -- 172.21.15.102:0/3018923677 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88c037760 con 0x7fe880003970 2024-03-23T03:48:37.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.596+0000 7fe89be00640 1 -- 172.21.15.102:0/3018923677 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c037ad0 con 0x7fe880003970 2024-03-23T03:48:37.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.596+0000 7fe8a4600640 1 -- 172.21.15.102:0/3018923677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe880003970 msgr2=0x7fe880003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:37.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.596+0000 7fe8a4600640 1 --2- 172.21.15.102:0/3018923677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe880003970 0x7fe880003d90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fe88c0098a0 tx=0x7fe8800057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.596+0000 7fe8a2600640 1 -- 172.21.15.102:0/3018923677 reap_dead start 2024-03-23T03:48:37.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.596+0000 7fe8a4600640 1 -- 172.21.15.102:0/3018923677 shutdown_connections 2024-03-23T03:48:37.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.596+0000 7fe8a4600640 1 -- 172.21.15.102:0/3018923677 >> 172.21.15.102:0/3018923677 conn(0x7fe89c0cf5d0 msgr2=0x7fe89c0cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:37.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.596+0000 7fe8a4600640 1 -- 172.21.15.102:0/3018923677 shutdown_connections 2024-03-23T03:48:37.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.596+0000 7fe8a4600640 1 -- 172.21.15.102:0/3018923677 wait complete. 2024-03-23T03:48:37.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.598+0000 7fe8a4600640 1 Processor -- start 2024-03-23T03:48:37.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.598+0000 7fe8a4600640 1 -- start start 2024-03-23T03:48:37.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.599+0000 7fe8a4600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe880003970 0x7fe89c0a9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:37.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.599+0000 7fe8a4600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe89c0a97b0 0x7fe89c0adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:37.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.599+0000 7fe8a4600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe89c14c580 0x7fe89c0ae4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:37.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.599+0000 7fe8a4600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0a9cc0 con 0x7fe880003970 2024-03-23T03:48:37.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.599+0000 7fe8a4600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0a9fa0 con 0x7fe89c0a97b0 2024-03-23T03:48:37.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.599+0000 7fe8a4600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89c0aa280 con 0x7fe89c14c580 2024-03-23T03:48:37.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.599+0000 7fe8a1c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe880003970 0x7fe89c0a9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.599+0000 7fe8a1c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe880003970 0x7fe89c0a9270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55252/0 (socket says 172.21.15.102:55252) 2024-03-23T03:48:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.600+0000 7fe8a1c00640 1 -- 172.21.15.102:0/1780697887 learned_addr learned my addr 172.21.15.102:0/1780697887 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.600+0000 7fe8a1200640 1 --2- 172.21.15.102:0/1780697887 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe89c0a97b0 0x7fe89c0adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.600+0000 7fe8a2600640 1 --2- 172.21.15.102:0/1780697887 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe89c14c580 0x7fe89c0ae4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.600+0000 7fe8a1c00640 1 -- 172.21.15.102:0/1780697887 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe89c14c580 msgr2=0x7fe89c0ae4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.600+0000 7fe8a1c00640 1 --2- 172.21.15.102:0/1780697887 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe89c14c580 0x7fe89c0ae4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.600+0000 7fe8a1c00640 1 -- 172.21.15.102:0/1780697887 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe89c0a97b0 msgr2=0x7fe89c0adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.600+0000 7fe8a1c00640 1 --2- 172.21.15.102:0/1780697887 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe89c0a97b0 0x7fe89c0adf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.600+0000 7fe8a1c00640 1 -- 172.21.15.102:0/1780697887 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe89c0aa560 con 0x7fe880003970 2024-03-23T03:48:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.600+0000 7fe8a2600640 1 --2- 172.21.15.102:0/1780697887 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe89c14c580 0x7fe89c0ae4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.600+0000 7fe8a1c00640 1 --2- 172.21.15.102:0/1780697887 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe880003970 0x7fe89c0a9270 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe88c0099c0 tx=0x7fe88c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.600+0000 7fe899600640 1 -- 172.21.15.102:0/1780697887 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c02f900 con 0x7fe880003970 2024-03-23T03:48:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.601+0000 7fe899600640 1 -- 172.21.15.102:0/1780697887 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe88c02fa60 con 0x7fe880003970 2024-03-23T03:48:37.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.601+0000 7fe899600640 1 -- 172.21.15.102:0/1780697887 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe88c02fdd0 con 0x7fe880003970 2024-03-23T03:48:37.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.601+0000 7fe8a4600640 1 -- 172.21.15.102:0/1780697887 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe89c0aa840 con 0x7fe880003970 2024-03-23T03:48:37.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.601+0000 7fe8a4600640 1 -- 172.21.15.102:0/1780697887 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89c1f5590 con 0x7fe880003970 2024-03-23T03:48:37.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.602+0000 7fe8a4600640 1 -- 172.21.15.102:0/1780697887 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe89c073390 con 0x7fe880003970 2024-03-23T03:48:37.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.603+0000 7fe899600640 1 -- 172.21.15.102:0/1780697887 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe88c04c460 con 0x7fe880003970 2024-03-23T03:48:37.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.603+0000 7fe899600640 1 --2- 172.21.15.102:0/1780697887 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe87c06ac00 0x7fe87c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:37.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.604+0000 7fe899600640 1 -- 172.21.15.102:0/1780697887 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe88c03e040 con 0x7fe880003970 2024-03-23T03:48:37.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.604+0000 7fe8a1200640 1 --2- 172.21.15.102:0/1780697887 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe87c06ac00 0x7fe87c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:37.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.604+0000 7fe8a1200640 1 --2- 172.21.15.102:0/1780697887 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe87c06ac00 0x7fe87c06d0c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0a9bf0 tx=0x7fe88800b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:37.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.608+0000 7fe899600640 1 -- 172.21.15.102:0/1780697887 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe88c076c50 con 0x7fe880003970 2024-03-23T03:48:37.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.800+0000 7fe8a4600640 1 -- 172.21.15.102:0/1780697887 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe89c0abcf0 con 0x7fe880003970 2024-03-23T03:48:37.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.800+0000 7fe899600640 1 -- 172.21.15.102:0/1780697887 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe88c07a270 con 0x7fe880003970 2024-03-23T03:48:37.801 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:48:37.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.803+0000 7fe8a4600640 1 -- 172.21.15.102:0/1780697887 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe87c06ac00 msgr2=0x7fe87c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:37.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.803+0000 7fe8a4600640 1 --2- 172.21.15.102:0/1780697887 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe87c06ac00 0x7fe87c06d0c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe89c0a9bf0 tx=0x7fe88800b040 comp rx=0 tx=0).stop 2024-03-23T03:48:37.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.803+0000 7fe8a4600640 1 -- 172.21.15.102:0/1780697887 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe880003970 msgr2=0x7fe89c0a9270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:37.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.803+0000 7fe8a4600640 1 --2- 172.21.15.102:0/1780697887 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe880003970 0x7fe89c0a9270 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fe88c0099c0 tx=0x7fe88c004730 comp rx=0 tx=0).stop 2024-03-23T03:48:37.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.803+0000 7fe8a4600640 1 -- 172.21.15.102:0/1780697887 shutdown_connections 2024-03-23T03:48:37.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.803+0000 7fe8a4600640 1 --2- 172.21.15.102:0/1780697887 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe87c06ac00 0x7fe87c06d0c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.803+0000 7fe8a4600640 1 --2- 172.21.15.102:0/1780697887 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe89c14c580 0x7fe89c0ae4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.803+0000 7fe8a4600640 1 --2- 172.21.15.102:0/1780697887 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe89c0a97b0 0x7fe89c0adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.803+0000 7fe8a4600640 1 --2- 172.21.15.102:0/1780697887 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe880003970 0x7fe89c0a9270 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:37.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.803+0000 7fe8a4600640 1 -- 172.21.15.102:0/1780697887 >> 172.21.15.102:0/1780697887 conn(0x7fe89c0cf5d0 msgr2=0x7fe89c0b6fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:37.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.803+0000 7fe8a4600640 1 -- 172.21.15.102:0/1780697887 shutdown_connections 2024-03-23T03:48:37.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:37.803+0000 7fe8a4600640 1 -- 172.21.15.102:0/1780697887 wait complete. 2024-03-23T03:48:37.825 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:48:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.035+0000 7f94c9e00640 1 Processor -- start 2024-03-23T03:48:38.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.036+0000 7f94c9e00640 1 -- start start 2024-03-23T03:48:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.037+0000 7f94c9e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f94c40cb560 0x7f94c40b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.037+0000 7f94c9e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c41575b0 con 0x7f94c41571d0 2024-03-23T03:48:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.037+0000 7f94c9e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c4159f20 con 0x7f94c40cdc00 2024-03-23T03:48:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.037+0000 7f94c9e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c415a200 con 0x7f94c40cb560 2024-03-23T03:48:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.038+0000 7f94c8a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f94c41571d0 0x7f94c415aaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53710/0 (socket says 172.21.15.102:53710) 2024-03-23T03:48:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.038+0000 7f94c8a00640 1 -- 172.21.15.102:0/618087611 learned_addr learned my addr 172.21.15.102:0/618087611 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.038+0000 7f94c9400640 1 --2- 172.21.15.102:0/618087611 >> v2:172.21.15.173:3300/0 conn(0x7f94c40cb560 0x7f94c40b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.038+0000 7f94c3400640 1 -- 172.21.15.102:0/618087611 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677301464 0 0) 0x7f94c41575b0 con 0x7f94c41571d0 2024-03-23T03:48:38.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.038+0000 7f94c3400640 1 -- 172.21.15.102:0/618087611 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a8003520 con 0x7f94c41571d0 2024-03-23T03:48:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.039+0000 7f94c3400640 1 -- 172.21.15.102:0/618087611 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 426140282 0 0) 0x7f94a8003520 con 0x7f94c41571d0 2024-03-23T03:48:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.039+0000 7f94c3400640 1 -- 172.21.15.102:0/618087611 >> v2:172.21.15.173:3300/0 conn(0x7f94c40cb560 msgr2=0x7f94c40b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.039+0000 7f94c3400640 1 --2- 172.21.15.102:0/618087611 >> v2:172.21.15.173:3300/0 conn(0x7f94c40cb560 0x7f94c40b35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.039+0000 7f94c3400640 1 -- 172.21.15.102:0/618087611 >> v1:172.21.15.173:6789/0 conn(0x7f94c40cdc00 legacy=0x7f94c40cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.039+0000 7f94c3400640 1 -- 172.21.15.102:0/618087611 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c415a4e0 con 0x7f94c41571d0 2024-03-23T03:48:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.039+0000 7f94c3400640 1 -- 172.21.15.102:0/618087611 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f94b40019f0 con 0x7f94c41571d0 2024-03-23T03:48:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.039+0000 7f94c3400640 1 -- 172.21.15.102:0/618087611 >> v1:172.21.15.102:6789/0 conn(0x7f94c41571d0 legacy=0x7f94c415aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.039+0000 7f94c3400640 1 --2- 172.21.15.102:0/618087611 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94a80039c0 0x7f94a8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.039+0000 7f94c3400640 1 --2- 172.21.15.102:0/618087611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94a80047c0 0x7f94a8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.039+0000 7f94c3400640 1 --2- 172.21.15.102:0/618087611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94a8005180 0x7f94a800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.039+0000 7f94c3400640 1 -- 172.21.15.102:0/618087611 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c415a200 con 0x7f94a8005180 2024-03-23T03:48:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.039+0000 7f94c3400640 1 -- 172.21.15.102:0/618087611 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a800c050 con 0x7f94a80047c0 2024-03-23T03:48:38.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.039+0000 7f94c3400640 1 -- 172.21.15.102:0/618087611 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a800c210 con 0x7f94a80039c0 2024-03-23T03:48:38.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.039+0000 7f94c9400640 1 --2- 172.21.15.102:0/618087611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94a8005180 0x7f94a800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:38.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.040+0000 7f94c9400640 1 -- 172.21.15.102:0/618087611 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94a80039c0 msgr2=0x7f94a8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:38.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.040+0000 7f94c9400640 1 --2- 172.21.15.102:0/618087611 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94a80039c0 0x7f94a8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.040+0000 7f94c9400640 1 -- 172.21.15.102:0/618087611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94a80047c0 msgr2=0x7f94a8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:38.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.040+0000 7f94c9400640 1 --2- 172.21.15.102:0/618087611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94a80047c0 0x7f94a8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.040+0000 7f94c9400640 1 -- 172.21.15.102:0/618087611 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a8003520 con 0x7f94a8005180 2024-03-23T03:48:38.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.040+0000 7f94c9400640 1 -- 172.21.15.102:0/618087611 reap_dead start 2024-03-23T03:48:38.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.041+0000 7f94c9400640 1 --2- 172.21.15.102:0/618087611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94a8005180 0x7f94a800b980 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f94c40cad80 tx=0x7f94a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b0992dff6edd49b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:38.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.041+0000 7f94c3400640 1 -- 172.21.15.102:0/618087611 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc02e890 con 0x7f94a8005180 2024-03-23T03:48:38.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.041+0000 7f94c3400640 1 -- 172.21.15.102:0/618087611 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94bc02ea10 con 0x7f94a8005180 2024-03-23T03:48:38.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.041+0000 7f94c3400640 1 -- 172.21.15.102:0/618087611 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc02edc0 con 0x7f94a8005180 2024-03-23T03:48:38.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.041+0000 7f94c9e00640 1 -- 172.21.15.102:0/618087611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94a8005180 msgr2=0x7f94a800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:38.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.041+0000 7f94c9e00640 1 --2- 172.21.15.102:0/618087611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94a8005180 0x7f94a800b980 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f94c40cad80 tx=0x7f94a80057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.042+0000 7f94c9e00640 1 -- 172.21.15.102:0/618087611 shutdown_connections 2024-03-23T03:48:38.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.042+0000 7f94c9e00640 1 --2- 172.21.15.102:0/618087611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94a8005180 0x7f94a800b980 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.042+0000 7f94c9e00640 1 -- 172.21.15.102:0/618087611 >> 172.21.15.102:0/618087611 conn(0x7f94c4065df0 msgr2=0x7f94c4069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:38.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.042+0000 7f94c9e00640 1 -- 172.21.15.102:0/618087611 shutdown_connections 2024-03-23T03:48:38.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.042+0000 7f94c9e00640 1 -- 172.21.15.102:0/618087611 wait complete. 2024-03-23T03:48:38.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.043+0000 7f94c9e00640 1 Processor -- start 2024-03-23T03:48:38.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.044+0000 7f94c9e00640 1 -- start start 2024-03-23T03:48:38.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94a8005180 0x7f94c40ac710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:38.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c9e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94c40c9de0 0x7f94c40cc860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:38.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94c40ccda0 0x7f94c41f2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:38.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c9e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c40cd250 con 0x7f94c40ccda0 2024-03-23T03:48:38.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c9e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c40ca260 con 0x7f94a8005180 2024-03-23T03:48:38.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c9e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c40ab950 con 0x7f94c40c9de0 2024-03-23T03:48:38.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c9400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94c40ccda0 0x7f94c41f2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:38.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c9400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94c40ccda0 0x7f94c41f2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55262/0 (socket says 172.21.15.102:55262) 2024-03-23T03:48:38.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c9400640 1 -- 172.21.15.102:0/3648162495 learned_addr learned my addr 172.21.15.102:0/3648162495 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:38.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c8a00640 1 --2- 172.21.15.102:0/3648162495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94a8005180 0x7f94c40ac710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:38.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c9400640 1 -- 172.21.15.102:0/3648162495 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94c40c9de0 msgr2=0x7f94c40cc860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:38.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c3e00640 1 --2- 172.21.15.102:0/3648162495 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94c40c9de0 0x7f94c40cc860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c9400640 1 --2- 172.21.15.102:0/3648162495 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94c40c9de0 0x7f94c40cc860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c9400640 1 -- 172.21.15.102:0/3648162495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94a8005180 msgr2=0x7f94c40ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c9400640 1 --2- 172.21.15.102:0/3648162495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94a8005180 0x7f94c40ac710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c9400640 1 -- 172.21.15.102:0/3648162495 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c40abc30 con 0x7f94c40ccda0 2024-03-23T03:48:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c3e00640 1 --2- 172.21.15.102:0/3648162495 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94c40c9de0 0x7f94c40cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.045+0000 7f94c8a00640 1 --2- 172.21.15.102:0/3648162495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94a8005180 0x7f94c40ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.046+0000 7f94c9400640 1 --2- 172.21.15.102:0/3648162495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94c40ccda0 0x7f94c41f2000 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f94bc02e500 tx=0x7f94bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.046+0000 7f94c0c00640 1 -- 172.21.15.102:0/3648162495 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc036ca0 con 0x7f94c40ccda0 2024-03-23T03:48:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.046+0000 7f94c0c00640 1 -- 172.21.15.102:0/3648162495 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94bc036e00 con 0x7f94c40ccda0 2024-03-23T03:48:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.046+0000 7f94c9e00640 1 -- 172.21.15.102:0/3648162495 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c40abf10 con 0x7f94c40ccda0 2024-03-23T03:48:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.046+0000 7f94c0c00640 1 -- 172.21.15.102:0/3648162495 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94bc03f690 con 0x7f94c40ccda0 2024-03-23T03:48:38.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.046+0000 7f94c9e00640 1 -- 172.21.15.102:0/3648162495 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c41f2d70 con 0x7f94c40ccda0 2024-03-23T03:48:38.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.047+0000 7f94c9e00640 1 -- 172.21.15.102:0/3648162495 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94c40ac430 con 0x7f94c40ccda0 2024-03-23T03:48:38.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.048+0000 7f94c0c00640 1 -- 172.21.15.102:0/3648162495 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f94bc0483b0 con 0x7f94c40ccda0 2024-03-23T03:48:38.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.049+0000 7f94c0c00640 1 --2- 172.21.15.102:0/3648162495 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f94a006ac30 0x7f94a006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:38.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.049+0000 7f94c0c00640 1 -- 172.21.15.102:0/3648162495 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f94bc0aa640 con 0x7f94c40ccda0 2024-03-23T03:48:38.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.049+0000 7f94c8a00640 1 --2- 172.21.15.102:0/3648162495 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f94a006ac30 0x7f94a006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:38.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.050+0000 7f94c8a00640 1 --2- 172.21.15.102:0/3648162495 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f94a006ac30 0x7f94a006d0f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f94c40ad350 tx=0x7f94b4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:38.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.054+0000 7f94c0c00640 1 -- 172.21.15.102:0/3648162495 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f94c40ac430 con 0x7f94c40ccda0 2024-03-23T03:48:38.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.254+0000 7f94c9e00640 1 -- 172.21.15.102:0/3648162495 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f94c40ac430 con 0x7f94c40ccda0 2024-03-23T03:48:38.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.255+0000 7f94c0c00640 1 -- 172.21.15.102:0/3648162495 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) v1 ==== 100+0+11 (secure 0 0 0) 0x7f94c40ac430 con 0x7f94c40ccda0 2024-03-23T03:48:38.256 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:48:38.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.258+0000 7f94c9e00640 1 -- 172.21.15.102:0/3648162495 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f94a006ac30 msgr2=0x7f94a006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:38.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.258+0000 7f94c9e00640 1 --2- 172.21.15.102:0/3648162495 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f94a006ac30 0x7f94a006d0f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f94c40ad350 tx=0x7f94b4009040 comp rx=0 tx=0).stop 2024-03-23T03:48:38.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.258+0000 7f94c9e00640 1 -- 172.21.15.102:0/3648162495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94c40ccda0 msgr2=0x7f94c41f2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:38.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.258+0000 7f94c9e00640 1 --2- 172.21.15.102:0/3648162495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94c40ccda0 0x7f94c41f2000 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f94bc02e500 tx=0x7f94bc004730 comp rx=0 tx=0).stop 2024-03-23T03:48:38.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.258+0000 7f94c9e00640 1 -- 172.21.15.102:0/3648162495 shutdown_connections 2024-03-23T03:48:38.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.258+0000 7f94c9e00640 1 --2- 172.21.15.102:0/3648162495 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f94a006ac30 0x7f94a006d0f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.258+0000 7f94c9e00640 1 --2- 172.21.15.102:0/3648162495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94c40ccda0 0x7f94c41f2000 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.258+0000 7f94c9e00640 1 --2- 172.21.15.102:0/3648162495 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94c40c9de0 0x7f94c40cc860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.258+0000 7f94c9e00640 1 --2- 172.21.15.102:0/3648162495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94a8005180 0x7f94c40ac710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.258+0000 7f94c9e00640 1 -- 172.21.15.102:0/3648162495 >> 172.21.15.102:0/3648162495 conn(0x7f94c4065df0 msgr2=0x7f94c4141d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:38.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.258+0000 7f94c9e00640 1 -- 172.21.15.102:0/3648162495 shutdown_connections 2024-03-23T03:48:38.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.258+0000 7f94c9e00640 1 -- 172.21.15.102:0/3648162495 wait complete. 2024-03-23T03:48:38.279 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:48:38.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.484+0000 7ffb49200640 1 Processor -- start 2024-03-23T03:48:38.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.484+0000 7ffb49200640 1 -- start start 2024-03-23T03:48:38.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.485+0000 7ffb49200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ffb440ce930 0x7ffb440ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:38.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.486+0000 7ffb49200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb44070f90 con 0x7ffb440cb660 2024-03-23T03:48:38.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.486+0000 7ffb49200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb440cf250 con 0x7ffb4406d2c0 2024-03-23T03:48:38.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.486+0000 7ffb49200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb4415bae0 con 0x7ffb440ce930 2024-03-23T03:48:38.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.486+0000 7ffb41400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ffb440cb660 0x7ffb44070170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53722/0 (socket says 172.21.15.102:53722) 2024-03-23T03:48:38.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.486+0000 7ffb41400640 1 -- 172.21.15.102:0/2458285342 learned_addr learned my addr 172.21.15.102:0/2458285342 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:38.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.486+0000 7ffb40a00640 1 -- 172.21.15.102:0/2458285342 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3133053802 0 0) 0x7ffb44070f90 con 0x7ffb440cb660 2024-03-23T03:48:38.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.486+0000 7ffb41e00640 1 --2- 172.21.15.102:0/2458285342 >> v2:172.21.15.173:3300/0 conn(0x7ffb440ce930 0x7ffb440ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.486+0000 7ffb40a00640 1 -- 172.21.15.102:0/2458285342 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb28003520 con 0x7ffb440cb660 2024-03-23T03:48:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.487+0000 7ffb40a00640 1 -- 172.21.15.102:0/2458285342 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2985205631 0 0) 0x7ffb28003520 con 0x7ffb440cb660 2024-03-23T03:48:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.487+0000 7ffb40a00640 1 -- 172.21.15.102:0/2458285342 >> v2:172.21.15.173:3300/0 conn(0x7ffb440ce930 msgr2=0x7ffb440ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.487+0000 7ffb40a00640 1 --2- 172.21.15.102:0/2458285342 >> v2:172.21.15.173:3300/0 conn(0x7ffb440ce930 0x7ffb440ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.487+0000 7ffb40a00640 1 -- 172.21.15.102:0/2458285342 >> v1:172.21.15.173:6789/0 conn(0x7ffb4406d2c0 legacy=0x7ffb44070880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.487+0000 7ffb40a00640 1 -- 172.21.15.102:0/2458285342 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4415ae60 con 0x7ffb440cb660 2024-03-23T03:48:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.487+0000 7ffb40a00640 1 -- 172.21.15.102:0/2458285342 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7ffb30001670 con 0x7ffb440cb660 2024-03-23T03:48:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.487+0000 7ffb40a00640 1 -- 172.21.15.102:0/2458285342 >> v1:172.21.15.102:6789/0 conn(0x7ffb440cb660 legacy=0x7ffb44070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.487+0000 7ffb40a00640 1 --2- 172.21.15.102:0/2458285342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb28003970 0x7ffb28003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.487+0000 7ffb40a00640 1 --2- 172.21.15.102:0/2458285342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb280047c0 0x7ffb28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.487+0000 7ffb40a00640 1 --2- 172.21.15.102:0/2458285342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffb28005180 0x7ffb2800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.487+0000 7ffb40a00640 1 -- 172.21.15.102:0/2458285342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4415bae0 con 0x7ffb280047c0 2024-03-23T03:48:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.487+0000 7ffb40a00640 1 -- 172.21.15.102:0/2458285342 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2800c0d0 con 0x7ffb28003970 2024-03-23T03:48:38.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.487+0000 7ffb40a00640 1 -- 172.21.15.102:0/2458285342 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2800c290 con 0x7ffb28005180 2024-03-23T03:48:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.487+0000 7ffb41400640 1 --2- 172.21.15.102:0/2458285342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb280047c0 0x7ffb28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.487+0000 7ffb41e00640 1 --2- 172.21.15.102:0/2458285342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb28003970 0x7ffb28003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.488+0000 7ffb41400640 1 -- 172.21.15.102:0/2458285342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffb28005180 msgr2=0x7ffb2800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.488+0000 7ffb41400640 1 --2- 172.21.15.102:0/2458285342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffb28005180 0x7ffb2800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.488+0000 7ffb41400640 1 -- 172.21.15.102:0/2458285342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb28003970 msgr2=0x7ffb28003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.488+0000 7ffb41400640 1 --2- 172.21.15.102:0/2458285342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb28003970 0x7ffb28003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.488+0000 7ffb41400640 1 -- 172.21.15.102:0/2458285342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb28003520 con 0x7ffb280047c0 2024-03-23T03:48:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.488+0000 7ffb41e00640 1 --2- 172.21.15.102:0/2458285342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb28003970 0x7ffb28003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:38.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.488+0000 7ffb42800640 1 -- 172.21.15.102:0/2458285342 reap_dead start 2024-03-23T03:48:38.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.489+0000 7ffb41400640 1 --2- 172.21.15.102:0/2458285342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb280047c0 0x7ffb28004c40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ffb3000c240 tx=0x7ffb280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee34767633cc6695 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:38.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.489+0000 7ffb40a00640 1 -- 172.21.15.102:0/2458285342 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30005ad0 con 0x7ffb280047c0 2024-03-23T03:48:38.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.489+0000 7ffb40a00640 1 -- 172.21.15.102:0/2458285342 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb30005c50 con 0x7ffb280047c0 2024-03-23T03:48:38.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.489+0000 7ffb40a00640 1 -- 172.21.15.102:0/2458285342 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3002c8b0 con 0x7ffb280047c0 2024-03-23T03:48:38.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.489+0000 7ffb49200640 1 -- 172.21.15.102:0/2458285342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb280047c0 msgr2=0x7ffb28004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:38.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.489+0000 7ffb49200640 1 --2- 172.21.15.102:0/2458285342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb280047c0 0x7ffb28004c40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ffb3000c240 tx=0x7ffb280057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.490+0000 7ffb49200640 1 -- 172.21.15.102:0/2458285342 shutdown_connections 2024-03-23T03:48:38.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.490+0000 7ffb49200640 1 --2- 172.21.15.102:0/2458285342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb280047c0 0x7ffb28004c40 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.490+0000 7ffb49200640 1 -- 172.21.15.102:0/2458285342 >> 172.21.15.102:0/2458285342 conn(0x7ffb4406a0b0 msgr2=0x7ffb4406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:38.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.490+0000 7ffb49200640 1 -- 172.21.15.102:0/2458285342 shutdown_connections 2024-03-23T03:48:38.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.490+0000 7ffb49200640 1 -- 172.21.15.102:0/2458285342 wait complete. 2024-03-23T03:48:38.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.491+0000 7ffb49200640 1 Processor -- start 2024-03-23T03:48:38.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.492+0000 7ffb49200640 1 -- start start 2024-03-23T03:48:38.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.492+0000 7ffb49200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffb280047c0 0x7ffb4414cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:38.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.492+0000 7ffb49200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb440ab8b0 0x7ffb44148110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:38.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.492+0000 7ffb49200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb4415a4a0 0x7ffb44070070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:38.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.492+0000 7ffb49200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4414d7e0 con 0x7ffb4415a4a0 2024-03-23T03:48:38.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.492+0000 7ffb49200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4414dac0 con 0x7ffb440ab8b0 2024-03-23T03:48:38.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.492+0000 7ffb49200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4414dda0 con 0x7ffb280047c0 2024-03-23T03:48:38.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.493+0000 7ffb42800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb4415a4a0 0x7ffb44070070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:38.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.493+0000 7ffb42800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb4415a4a0 0x7ffb44070070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55282/0 (socket says 172.21.15.102:55282) 2024-03-23T03:48:38.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.493+0000 7ffb42800640 1 -- 172.21.15.102:0/215816175 learned_addr learned my addr 172.21.15.102:0/215816175 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:38.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.493+0000 7ffb41400640 1 --2- 172.21.15.102:0/215816175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb440ab8b0 0x7ffb44148110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:38.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.493+0000 7ffb42800640 1 -- 172.21.15.102:0/215816175 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffb280047c0 msgr2=0x7ffb4414cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:38.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.493+0000 7ffb41e00640 1 --2- 172.21.15.102:0/215816175 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffb280047c0 0x7ffb4414cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:38.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.493+0000 7ffb42800640 1 --2- 172.21.15.102:0/215816175 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffb280047c0 0x7ffb4414cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.493+0000 7ffb42800640 1 -- 172.21.15.102:0/215816175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb440ab8b0 msgr2=0x7ffb44148110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:38.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.493+0000 7ffb42800640 1 --2- 172.21.15.102:0/215816175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb440ab8b0 0x7ffb44148110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.493+0000 7ffb42800640 1 -- 172.21.15.102:0/215816175 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4414e080 con 0x7ffb4415a4a0 2024-03-23T03:48:38.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.494+0000 7ffb42800640 1 --2- 172.21.15.102:0/215816175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb4415a4a0 0x7ffb44070070 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ffb4406fdc0 tx=0x7ffb3c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:38.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.494+0000 7ffb3a000640 1 -- 172.21.15.102:0/215816175 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c005d40 con 0x7ffb4415a4a0 2024-03-23T03:48:38.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.494+0000 7ffb3a000640 1 -- 172.21.15.102:0/215816175 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3c00ced0 con 0x7ffb4415a4a0 2024-03-23T03:48:38.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.494+0000 7ffb3a000640 1 -- 172.21.15.102:0/215816175 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c00a850 con 0x7ffb4415a4a0 2024-03-23T03:48:38.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.494+0000 7ffb49200640 1 -- 172.21.15.102:0/215816175 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb44149540 con 0x7ffb4415a4a0 2024-03-23T03:48:38.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.494+0000 7ffb49200640 1 -- 172.21.15.102:0/215816175 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb44148650 con 0x7ffb4415a4a0 2024-03-23T03:48:38.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.495+0000 7ffb49200640 1 -- 172.21.15.102:0/215816175 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb4414e270 con 0x7ffb4415a4a0 2024-03-23T03:48:38.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.496+0000 7ffb3a000640 1 -- 172.21.15.102:0/215816175 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ffb3c00a9b0 con 0x7ffb4415a4a0 2024-03-23T03:48:38.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.497+0000 7ffb3a000640 1 --2- 172.21.15.102:0/215816175 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb2406ac30 0x7ffb2406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:38.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.497+0000 7ffb3a000640 1 -- 172.21.15.102:0/215816175 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ffb3c0883e0 con 0x7ffb4415a4a0 2024-03-23T03:48:38.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.497+0000 7ffb41e00640 1 --2- 172.21.15.102:0/215816175 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb2406ac30 0x7ffb2406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:38.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.498+0000 7ffb41e00640 1 --2- 172.21.15.102:0/215816175 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb2406ac30 0x7ffb2406d0f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffb440caa60 tx=0x7ffb34008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:38.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.501+0000 7ffb3a000640 1 -- 172.21.15.102:0/215816175 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffb3c0533e0 con 0x7ffb4415a4a0 2024-03-23T03:48:38.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.703+0000 7ffb49200640 1 -- 172.21.15.102:0/215816175 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ffb44073390 con 0x7ffb4415a4a0 2024-03-23T03:48:38.704 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.703+0000 7ffb3a000640 1 -- 172.21.15.102:0/215816175 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v41) v1 ==== 80+0+0 (secure 0 0 0) 0x7ffb3c056a00 con 0x7ffb4415a4a0 2024-03-23T03:48:38.704 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:48:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.706+0000 7ffb49200640 1 -- 172.21.15.102:0/215816175 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb2406ac30 msgr2=0x7ffb2406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.706+0000 7ffb49200640 1 --2- 172.21.15.102:0/215816175 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb2406ac30 0x7ffb2406d0f0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ffb440caa60 tx=0x7ffb34008040 comp rx=0 tx=0).stop 2024-03-23T03:48:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.706+0000 7ffb49200640 1 -- 172.21.15.102:0/215816175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb4415a4a0 msgr2=0x7ffb44070070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.706+0000 7ffb49200640 1 --2- 172.21.15.102:0/215816175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb4415a4a0 0x7ffb44070070 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7ffb4406fdc0 tx=0x7ffb3c00cb90 comp rx=0 tx=0).stop 2024-03-23T03:48:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.706+0000 7ffb49200640 1 -- 172.21.15.102:0/215816175 shutdown_connections 2024-03-23T03:48:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.706+0000 7ffb49200640 1 --2- 172.21.15.102:0/215816175 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb2406ac30 0x7ffb2406d0f0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.706+0000 7ffb49200640 1 --2- 172.21.15.102:0/215816175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb4415a4a0 0x7ffb44070070 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.706+0000 7ffb49200640 1 --2- 172.21.15.102:0/215816175 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb440ab8b0 0x7ffb44148110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.706+0000 7ffb49200640 1 --2- 172.21.15.102:0/215816175 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffb280047c0 0x7ffb4414cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.706+0000 7ffb49200640 1 -- 172.21.15.102:0/215816175 >> 172.21.15.102:0/215816175 conn(0x7ffb4406a0b0 msgr2=0x7ffb4415cfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.706+0000 7ffb49200640 1 -- 172.21.15.102:0/215816175 shutdown_connections 2024-03-23T03:48:38.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:38.706+0000 7ffb49200640 1 -- 172.21.15.102:0/215816175 wait complete. 2024-03-23T03:48:38.725 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-23T03:48:38.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:38.725 DEBUG:teuthology.orchestra.run.smithi173:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-03-23T03:48:38.836 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:48:38.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:48:38.837 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T03:48:38.882 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:48:38.882 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:48:38.938 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:48:38.938 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:48:38.938 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:48:38.995 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:48:38.995 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:48:39.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.201+0000 7f856da00640 1 Processor -- start 2024-03-23T03:48:39.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.201+0000 7f856da00640 1 -- start start 2024-03-23T03:48:39.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.203+0000 7f856da00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f856815b560 0x7f856815b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:39.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.203+0000 7f856da00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f85680b39b0 0x7f85680b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:39.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.203+0000 7f856da00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f856806c6f0 con 0x7f856815e500 2024-03-23T03:48:39.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.203+0000 7f856da00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85680b4370 con 0x7f85680b39b0 2024-03-23T03:48:39.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.203+0000 7f856da00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f856806c9d0 con 0x7f856815b560 2024-03-23T03:48:39.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.203+0000 7f8566000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f85680b39b0 0x7f85680b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:39.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.203+0000 7f8567400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f856815e500 0x7f856815e930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53734/0 (socket says 172.21.15.102:53734) 2024-03-23T03:48:39.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.203+0000 7f8567400640 1 -- 172.21.15.102:0/3773406339 learned_addr learned my addr 172.21.15.102:0/3773406339 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:39.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.203+0000 7f8566a00640 1 --2- 172.21.15.102:0/3773406339 >> v2:172.21.15.173:3300/0 conn(0x7f856815b560 0x7f856815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:39.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.203+0000 7f8566000640 1 -- 172.21.15.102:0/3773406339 >> v2:172.21.15.173:3300/0 conn(0x7f856815b560 msgr2=0x7f856815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:39.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.203+0000 7f8565600640 1 -- 172.21.15.102:0/3773406339 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942400838 0 0) 0x7f856806c6f0 con 0x7f856815e500 2024-03-23T03:48:39.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.203+0000 7f8566000640 1 --2- 172.21.15.102:0/3773406339 >> v2:172.21.15.173:3300/0 conn(0x7f856815b560 0x7f856815b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.204+0000 7f8566000640 1 -- 172.21.15.102:0/3773406339 >> v1:172.21.15.102:6789/0 conn(0x7f856815e500 legacy=0x7f856815e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:39.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.204+0000 7f8566000640 1 -- 172.21.15.102:0/3773406339 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856806ccb0 con 0x7f85680b39b0 2024-03-23T03:48:39.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.204+0000 7f8566a00640 1 --2- 172.21.15.102:0/3773406339 >> v2:172.21.15.173:3300/0 conn(0x7f856815b560 0x7f856815b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:39.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.205+0000 7f8566000640 1 --2- 172.21.15.102:0/3773406339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85680b39b0 0x7f85680b3e30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f8568071470 tx=0x7f855402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d14f40540edc9fc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:39.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.205+0000 7f8565600640 1 -- 172.21.15.102:0/3773406339 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8554031830 con 0x7f85680b39b0 2024-03-23T03:48:39.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.205+0000 7f8565600640 1 -- 172.21.15.102:0/3773406339 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8554031990 con 0x7f85680b39b0 2024-03-23T03:48:39.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.205+0000 7f8565600640 1 -- 172.21.15.102:0/3773406339 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8554031d20 con 0x7f85680b39b0 2024-03-23T03:48:39.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.205+0000 7f856da00640 1 -- 172.21.15.102:0/3773406339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85680b39b0 msgr2=0x7f85680b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.205+0000 7f856da00640 1 --2- 172.21.15.102:0/3773406339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85680b39b0 0x7f85680b3e30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f8568071470 tx=0x7f855402b8d0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.205+0000 7f856da00640 1 -- 172.21.15.102:0/3773406339 shutdown_connections 2024-03-23T03:48:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.205+0000 7f856da00640 1 --2- 172.21.15.102:0/3773406339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85680b39b0 0x7f85680b3e30 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.205+0000 7f856da00640 1 --2- 172.21.15.102:0/3773406339 >> v2:172.21.15.173:3300/0 conn(0x7f856815b560 0x7f856815b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.205+0000 7f856da00640 1 -- 172.21.15.102:0/3773406339 >> 172.21.15.102:0/3773406339 conn(0x7f856806a3d0 msgr2=0x7f85680b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.206+0000 7f856da00640 1 -- 172.21.15.102:0/3773406339 shutdown_connections 2024-03-23T03:48:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.206+0000 7f856da00640 1 -- 172.21.15.102:0/3773406339 wait complete. 2024-03-23T03:48:39.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.207+0000 7f856da00640 1 Processor -- start 2024-03-23T03:48:39.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.207+0000 7f856da00640 1 -- start start 2024-03-23T03:48:39.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.208+0000 7f856da00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85680b39b0 0x7f8568156740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:39.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.208+0000 7f856da00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f856815b560 0x7f8568149f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:39.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.208+0000 7f856da00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f856815e500 0x7f85680b11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:39.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.208+0000 7f856da00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8568152540 con 0x7f85680b39b0 2024-03-23T03:48:39.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.208+0000 7f856da00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85680b2330 con 0x7f856815e500 2024-03-23T03:48:39.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.208+0000 7f856da00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85680b3180 con 0x7f856815b560 2024-03-23T03:48:39.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.208+0000 7f8566a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85680b39b0 0x7f8568156740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:39.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.209+0000 7f8566a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85680b39b0 0x7f8568156740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55290/0 (socket says 172.21.15.102:55290) 2024-03-23T03:48:39.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.209+0000 7f8566a00640 1 -- 172.21.15.102:0/2713643058 learned_addr learned my addr 172.21.15.102:0/2713643058 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:39.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.209+0000 7f8566a00640 1 -- 172.21.15.102:0/2713643058 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f856815b560 msgr2=0x7f8568149f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:39.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.209+0000 7f8566a00640 1 --2- 172.21.15.102:0/2713643058 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f856815b560 0x7f8568149f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.209+0000 7f8566a00640 1 -- 172.21.15.102:0/2713643058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f856815e500 msgr2=0x7f85680b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:39.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.209+0000 7f8566a00640 1 --2- 172.21.15.102:0/2713643058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f856815e500 0x7f85680b11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.209+0000 7f8566a00640 1 -- 172.21.15.102:0/2713643058 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8554009400 con 0x7f85680b39b0 2024-03-23T03:48:39.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.209+0000 7f8566a00640 1 --2- 172.21.15.102:0/2713643058 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85680b39b0 0x7f8568156740 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f855800c820 tx=0x7f855800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:39.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.209+0000 7f855ea00640 1 -- 172.21.15.102:0/2713643058 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8558007a30 con 0x7f85680b39b0 2024-03-23T03:48:39.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.209+0000 7f855ea00640 1 -- 172.21.15.102:0/2713643058 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8558007b90 con 0x7f85680b39b0 2024-03-23T03:48:39.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.209+0000 7f856da00640 1 -- 172.21.15.102:0/2713643058 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85680b1d70 con 0x7f85680b39b0 2024-03-23T03:48:39.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.209+0000 7f855ea00640 1 -- 172.21.15.102:0/2713643058 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855800ce70 con 0x7f85680b39b0 2024-03-23T03:48:39.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.210+0000 7f856da00640 1 -- 172.21.15.102:0/2713643058 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8568152160 con 0x7f85680b39b0 2024-03-23T03:48:39.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.211+0000 7f856da00640 1 -- 172.21.15.102:0/2713643058 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f856806ac20 con 0x7f85680b39b0 2024-03-23T03:48:39.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.212+0000 7f855ea00640 1 -- 172.21.15.102:0/2713643058 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f85580113f0 con 0x7f85680b39b0 2024-03-23T03:48:39.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.212+0000 7f855ea00640 1 --2- 172.21.15.102:0/2713643058 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f854806ac00 0x7f854806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:39.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.213+0000 7f855ea00640 1 -- 172.21.15.102:0/2713643058 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f85580895a0 con 0x7f85680b39b0 2024-03-23T03:48:39.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.213+0000 7f8566000640 1 --2- 172.21.15.102:0/2713643058 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f854806ac00 0x7f854806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:39.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.213+0000 7f8566000640 1 --2- 172.21.15.102:0/2713643058 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f854806ac00 0x7f854806d0c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8554036000 tx=0x7f8554030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:39.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.217+0000 7f855ea00640 1 -- 172.21.15.102:0/2713643058 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f85580546d0 con 0x7f85680b39b0 2024-03-23T03:48:39.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.448+0000 7f856da00640 1 -- 172.21.15.102:0/2713643058 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8568073390 con 0x7f85680b39b0 2024-03-23T03:48:39.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.449+0000 7f855ea00640 1 -- 172.21.15.102:0/2713643058 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f8558057cf0 con 0x7f85680b39b0 2024-03-23T03:48:39.450 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:39.450 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6834/1537100581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1537100581},{"type":"v1","addr":"172.21.15.173:6834","nonce":1537100581}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4655,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3206638952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":3206638952},{"type":"v1","addr":"172.21.15.173:6835","nonce":3206638952}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4852,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4005240382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4005240382},{"type":"v1","addr":"172.21.15.102:6837","nonce":4005240382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:48:23.418926+0000","modified":"2024-03-23T03:48:26.657662+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.102:6836/1373501465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":1373501465},{"type":"v1","addr":"172.21.15.102:6836","nonce":1373501465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-23T03:48:39.451 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 16 2024-03-23T03:48:39.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.452+0000 7f856da00640 1 -- 172.21.15.102:0/2713643058 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f854806ac00 msgr2=0x7f854806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:39.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.452+0000 7f856da00640 1 --2- 172.21.15.102:0/2713643058 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f854806ac00 0x7f854806d0c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8554036000 tx=0x7f8554030000 comp rx=0 tx=0).stop 2024-03-23T03:48:39.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.452+0000 7f856da00640 1 -- 172.21.15.102:0/2713643058 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85680b39b0 msgr2=0x7f8568156740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:39.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.452+0000 7f856da00640 1 --2- 172.21.15.102:0/2713643058 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85680b39b0 0x7f8568156740 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f855800c820 tx=0x7f855800cc20 comp rx=0 tx=0).stop 2024-03-23T03:48:39.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.453+0000 7f856da00640 1 -- 172.21.15.102:0/2713643058 shutdown_connections 2024-03-23T03:48:39.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.453+0000 7f856da00640 1 --2- 172.21.15.102:0/2713643058 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f854806ac00 0x7f854806d0c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.453+0000 7f856da00640 1 --2- 172.21.15.102:0/2713643058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f856815e500 0x7f85680b11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.453+0000 7f856da00640 1 --2- 172.21.15.102:0/2713643058 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f856815b560 0x7f8568149f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.453+0000 7f856da00640 1 --2- 172.21.15.102:0/2713643058 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85680b39b0 0x7f8568156740 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.453+0000 7f856da00640 1 -- 172.21.15.102:0/2713643058 >> 172.21.15.102:0/2713643058 conn(0x7f856806a3d0 msgr2=0x7f85680acf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:39.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.453+0000 7f856da00640 1 -- 172.21.15.102:0/2713643058 shutdown_connections 2024-03-23T03:48:39.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.453+0000 7f856da00640 1 -- 172.21.15.102:0/2713643058 wait complete. 2024-03-23T03:48:39.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:48:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.680+0000 7fb479800640 1 Processor -- start 2024-03-23T03:48:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.680+0000 7fb479800640 1 -- start start 2024-03-23T03:48:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.681+0000 7fb479800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb4740b39b0 0x7fb4740b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.681+0000 7fb479800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47406c6f0 con 0x7fb47415c400 2024-03-23T03:48:39.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.682+0000 7fb479800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47406c9d0 con 0x7fb47415b560 2024-03-23T03:48:39.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.682+0000 7fb479800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4740b4370 con 0x7fb4740b39b0 2024-03-23T03:48:39.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.682+0000 7fb473400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb47415c400 0x7fb47415c830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53748/0 (socket says 172.21.15.102:53748) 2024-03-23T03:48:39.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.682+0000 7fb473400640 1 -- 172.21.15.102:0/4223174508 learned_addr learned my addr 172.21.15.102:0/4223174508 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:39.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.682+0000 7fb472000640 1 -- 172.21.15.102:0/4223174508 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357924274 0 0) 0x7fb47406c6f0 con 0x7fb47415c400 2024-03-23T03:48:39.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.682+0000 7fb46a000640 1 --2- 172.21.15.102:0/4223174508 >> v2:172.21.15.173:3300/0 conn(0x7fb4740b39b0 0x7fb4740b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:39.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.682+0000 7fb472000640 1 -- 172.21.15.102:0/4223174508 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb458003520 con 0x7fb47415c400 2024-03-23T03:48:39.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.682+0000 7fb472000640 1 -- 172.21.15.102:0/4223174508 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1032730899 0 0) 0x7fb458003520 con 0x7fb47415c400 2024-03-23T03:48:39.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.683+0000 7fb472000640 1 -- 172.21.15.102:0/4223174508 >> v2:172.21.15.173:3300/0 conn(0x7fb4740b39b0 msgr2=0x7fb4740b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:39.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.683+0000 7fb472000640 1 --2- 172.21.15.102:0/4223174508 >> v2:172.21.15.173:3300/0 conn(0x7fb4740b39b0 0x7fb4740b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.683+0000 7fb472000640 1 -- 172.21.15.102:0/4223174508 >> v1:172.21.15.173:6789/0 conn(0x7fb47415b560 legacy=0x7fb47415b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:39.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.683+0000 7fb472000640 1 -- 172.21.15.102:0/4223174508 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47406ccb0 con 0x7fb47415c400 2024-03-23T03:48:39.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.683+0000 7fb472000640 1 -- 172.21.15.102:0/4223174508 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fb46c0019f0 con 0x7fb47415c400 2024-03-23T03:48:39.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.683+0000 7fb472000640 1 -- 172.21.15.102:0/4223174508 >> v1:172.21.15.102:6789/0 conn(0x7fb47415c400 legacy=0x7fb47415c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:39.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.683+0000 7fb472000640 1 --2- 172.21.15.102:0/4223174508 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb4580039c0 0x7fb458003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:39.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.683+0000 7fb472000640 1 --2- 172.21.15.102:0/4223174508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4580047c0 0x7fb458004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:39.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.683+0000 7fb472000640 1 --2- 172.21.15.102:0/4223174508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb458005180 0x7fb45800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:39.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.683+0000 7fb472000640 1 -- 172.21.15.102:0/4223174508 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4740b4370 con 0x7fb458005180 2024-03-23T03:48:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.683+0000 7fb472000640 1 -- 172.21.15.102:0/4223174508 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45800c050 con 0x7fb4580047c0 2024-03-23T03:48:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.683+0000 7fb472000640 1 -- 172.21.15.102:0/4223174508 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb45800c210 con 0x7fb4580039c0 2024-03-23T03:48:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.683+0000 7fb473400640 1 --2- 172.21.15.102:0/4223174508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb458005180 0x7fb45800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.683+0000 7fb46a000640 1 --2- 172.21.15.102:0/4223174508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4580047c0 0x7fb458004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.683+0000 7fb473400640 1 -- 172.21.15.102:0/4223174508 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb4580039c0 msgr2=0x7fb458003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.683+0000 7fb472a00640 1 --2- 172.21.15.102:0/4223174508 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb4580039c0 0x7fb458003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.683+0000 7fb473400640 1 --2- 172.21.15.102:0/4223174508 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb4580039c0 0x7fb458003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.684+0000 7fb473400640 1 -- 172.21.15.102:0/4223174508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4580047c0 msgr2=0x7fb458004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.684+0000 7fb473400640 1 --2- 172.21.15.102:0/4223174508 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4580047c0 0x7fb458004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.684+0000 7fb473400640 1 -- 172.21.15.102:0/4223174508 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb458003520 con 0x7fb458005180 2024-03-23T03:48:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.684+0000 7fb472a00640 1 --2- 172.21.15.102:0/4223174508 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb4580039c0 0x7fb458003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.684+0000 7fb473400640 1 -- 172.21.15.102:0/4223174508 reap_dead start 2024-03-23T03:48:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.685+0000 7fb473400640 1 --2- 172.21.15.102:0/4223174508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb458005180 0x7fb45800b980 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00a770 tx=0x7fb4580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea2e7eb17c6a8c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.685+0000 7fb472000640 1 -- 172.21.15.102:0/4223174508 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c034630 con 0x7fb458005180 2024-03-23T03:48:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.685+0000 7fb472000640 1 -- 172.21.15.102:0/4223174508 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46c02f070 con 0x7fb458005180 2024-03-23T03:48:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.685+0000 7fb472000640 1 -- 172.21.15.102:0/4223174508 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c034a10 con 0x7fb458005180 2024-03-23T03:48:39.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.685+0000 7fb479800640 1 -- 172.21.15.102:0/4223174508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb458005180 msgr2=0x7fb45800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.685+0000 7fb479800640 1 --2- 172.21.15.102:0/4223174508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb458005180 0x7fb45800b980 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00a770 tx=0x7fb4580057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.685+0000 7fb479800640 1 -- 172.21.15.102:0/4223174508 shutdown_connections 2024-03-23T03:48:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.685+0000 7fb479800640 1 --2- 172.21.15.102:0/4223174508 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb458005180 0x7fb45800b980 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.686+0000 7fb479800640 1 -- 172.21.15.102:0/4223174508 >> 172.21.15.102:0/4223174508 conn(0x7fb47406a3d0 msgr2=0x7fb4740b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.686+0000 7fb479800640 1 -- 172.21.15.102:0/4223174508 shutdown_connections 2024-03-23T03:48:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.686+0000 7fb479800640 1 -- 172.21.15.102:0/4223174508 wait complete. 2024-03-23T03:48:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.687+0000 7fb479800640 1 Processor -- start 2024-03-23T03:48:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.687+0000 7fb479800640 1 -- start start 2024-03-23T03:48:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.688+0000 7fb479800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb458005180 0x7fb4741f0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.688+0000 7fb479800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4741f0660 0x7fb474153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.688+0000 7fb479800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb474151fa0 0x7fb474153a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:39.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.688+0000 7fb479800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb474153fc0 con 0x7fb474151fa0 2024-03-23T03:48:39.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.688+0000 7fb479800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb474157250 con 0x7fb4741f0660 2024-03-23T03:48:39.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.688+0000 7fb479800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4741524a0 con 0x7fb458005180 2024-03-23T03:48:39.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.689+0000 7fb473400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb474151fa0 0x7fb474153a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:39.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.689+0000 7fb473400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb474151fa0 0x7fb474153a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55308/0 (socket says 172.21.15.102:55308) 2024-03-23T03:48:39.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.689+0000 7fb473400640 1 -- 172.21.15.102:0/3203041409 learned_addr learned my addr 172.21.15.102:0/3203041409 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:39.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.689+0000 7fb46a000640 1 --2- 172.21.15.102:0/3203041409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4741f0660 0x7fb474153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:39.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.689+0000 7fb472a00640 1 --2- 172.21.15.102:0/3203041409 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb458005180 0x7fb4741f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:39.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.689+0000 7fb473400640 1 -- 172.21.15.102:0/3203041409 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb458005180 msgr2=0x7fb4741f0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:39.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.689+0000 7fb473400640 1 --2- 172.21.15.102:0/3203041409 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb458005180 0x7fb4741f0120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.689+0000 7fb473400640 1 -- 172.21.15.102:0/3203041409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4741f0660 msgr2=0x7fb474153540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:39.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.689+0000 7fb473400640 1 --2- 172.21.15.102:0/3203041409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4741f0660 0x7fb474153540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.689+0000 7fb473400640 1 -- 172.21.15.102:0/3203041409 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb474152780 con 0x7fb474151fa0 2024-03-23T03:48:39.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.689+0000 7fb473400640 1 --2- 172.21.15.102:0/3203041409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb474151fa0 0x7fb474153a80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb46c02cb70 tx=0x7fb47406d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:39.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.689+0000 7fb46b400640 1 -- 172.21.15.102:0/3203041409 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c004b60 con 0x7fb474151fa0 2024-03-23T03:48:39.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.690+0000 7fb46b400640 1 -- 172.21.15.102:0/3203041409 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46c004cc0 con 0x7fb474151fa0 2024-03-23T03:48:39.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.690+0000 7fb46b400640 1 -- 172.21.15.102:0/3203041409 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c034980 con 0x7fb474151fa0 2024-03-23T03:48:39.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.690+0000 7fb479800640 1 -- 172.21.15.102:0/3203041409 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb474152a60 con 0x7fb474151fa0 2024-03-23T03:48:39.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.690+0000 7fb479800640 1 -- 172.21.15.102:0/3203041409 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb474153020 con 0x7fb474151fa0 2024-03-23T03:48:39.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.691+0000 7fb479800640 1 -- 172.21.15.102:0/3203041409 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb474151e10 con 0x7fb474151fa0 2024-03-23T03:48:39.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.692+0000 7fb46b400640 1 -- 172.21.15.102:0/3203041409 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb46c031030 con 0x7fb474151fa0 2024-03-23T03:48:39.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.692+0000 7fb46b400640 1 --2- 172.21.15.102:0/3203041409 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb45006ac00 0x7fb45006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:39.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.693+0000 7fb46b400640 1 -- 172.21.15.102:0/3203041409 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb46c0aaec0 con 0x7fb474151fa0 2024-03-23T03:48:39.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.693+0000 7fb472a00640 1 --2- 172.21.15.102:0/3203041409 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb45006ac00 0x7fb45006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:39.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.693+0000 7fb472a00640 1 --2- 172.21.15.102:0/3203041409 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb45006ac00 0x7fb45006d0c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb464007fd0 tx=0x7fb464009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:39.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.697+0000 7fb46b400640 1 -- 172.21.15.102:0/3203041409 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb46c075ec0 con 0x7fb474151fa0 2024-03-23T03:48:39.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.882+0000 7fb479800640 1 -- 172.21.15.102:0/3203041409 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb47406ac20 con 0x7fb474151fa0 2024-03-23T03:48:39.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.884+0000 7fb46b400640 1 -- 172.21.15.102:0/3203041409 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+16354 (secure 0 0 0) 0x7fb46c044020 con 0x7fb474151fa0 2024-03-23T03:48:39.885 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:39.886 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":41,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:24.423308+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:48:21.550281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-23T03:48:22.428475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:48:39.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.887+0000 7fb479800640 1 -- 172.21.15.102:0/3203041409 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb45006ac00 msgr2=0x7fb45006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:39.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.887+0000 7fb479800640 1 --2- 172.21.15.102:0/3203041409 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb45006ac00 0x7fb45006d0c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb464007fd0 tx=0x7fb464009040 comp rx=0 tx=0).stop 2024-03-23T03:48:39.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.887+0000 7fb479800640 1 -- 172.21.15.102:0/3203041409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb474151fa0 msgr2=0x7fb474153a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:39.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.887+0000 7fb479800640 1 --2- 172.21.15.102:0/3203041409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb474151fa0 0x7fb474153a80 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fb46c02cb70 tx=0x7fb47406d3b0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.887+0000 7fb479800640 1 -- 172.21.15.102:0/3203041409 shutdown_connections 2024-03-23T03:48:39.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.887+0000 7fb479800640 1 --2- 172.21.15.102:0/3203041409 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb45006ac00 0x7fb45006d0c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.887+0000 7fb479800640 1 --2- 172.21.15.102:0/3203041409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb474151fa0 0x7fb474153a80 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.887+0000 7fb479800640 1 --2- 172.21.15.102:0/3203041409 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb4741f0660 0x7fb474153540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.887+0000 7fb479800640 1 --2- 172.21.15.102:0/3203041409 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb458005180 0x7fb4741f0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:39.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.887+0000 7fb479800640 1 -- 172.21.15.102:0/3203041409 >> 172.21.15.102:0/3203041409 conn(0x7fb47406a3d0 msgr2=0x7fb474158aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:39.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.887+0000 7fb479800640 1 -- 172.21.15.102:0/3203041409 shutdown_connections 2024-03-23T03:48:39.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:39.888+0000 7fb479800640 1 -- 172.21.15.102:0/3203041409 wait complete. 2024-03-23T03:48:39.907 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:48:21.550281+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:48:22.428475+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:48:39.907 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:48:40.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.114+0000 7f4c79400640 1 Processor -- start 2024-03-23T03:48:40.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.115+0000 7f4c79400640 1 -- start start 2024-03-23T03:48:40.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.116+0000 7f4c79400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c740add30 0x7f4c740ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:40.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.116+0000 7f4c79400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4c7406ce90 0x7f4c7406d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:40.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.116+0000 7f4c79400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c740cf250 con 0x7f4c740ce930 2024-03-23T03:48:40.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.116+0000 7f4c79400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7406c760 con 0x7f4c7406ce90 2024-03-23T03:48:40.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.116+0000 7f4c79400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c740b42d0 con 0x7f4c740add30 2024-03-23T03:48:40.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.117+0000 7f4c73200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4c7406ce90 0x7f4c7406d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:40.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.117+0000 7f4c73200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4c7406ce90 0x7f4c7406d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55310/0 (socket says 172.21.15.102:55310) 2024-03-23T03:48:40.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.117+0000 7f4c73200640 1 -- 172.21.15.102:0/921377067 learned_addr learned my addr 172.21.15.102:0/921377067 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:40.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.117+0000 7f4c69e00640 1 --2- 172.21.15.102:0/921377067 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c740add30 0x7f4c740ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:40.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.117+0000 7f4c73200640 1 -- 172.21.15.102:0/921377067 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c740add30 msgr2=0x7f4c740ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:40.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.117+0000 7f4c73200640 1 --2- 172.21.15.102:0/921377067 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c740add30 0x7f4c740ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.117+0000 7f4c73200640 1 -- 172.21.15.102:0/921377067 >> v1:172.21.15.173:6789/0 conn(0x7f4c740ce930 legacy=0x7f4c740cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:40.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.117+0000 7f4c73200640 1 -- 172.21.15.102:0/921377067 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c740cdba0 con 0x7f4c7406ce90 2024-03-23T03:48:40.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.119+0000 7f4c73200640 1 --2- 172.21.15.102:0/921377067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c7406ce90 0x7f4c7406d320 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009e50 tx=0x7f4c6c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e95650629e196574 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:40.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.119+0000 7f4c71e00640 1 -- 172.21.15.102:0/921377067 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c02cb00 con 0x7f4c7406ce90 2024-03-23T03:48:40.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.119+0000 7f4c71e00640 1 -- 172.21.15.102:0/921377067 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6c02cc60 con 0x7f4c7406ce90 2024-03-23T03:48:40.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.119+0000 7f4c71e00640 1 -- 172.21.15.102:0/921377067 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c0343f0 con 0x7f4c7406ce90 2024-03-23T03:48:40.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.120+0000 7f4c79400640 1 -- 172.21.15.102:0/921377067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c7406ce90 msgr2=0x7f4c7406d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:40.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.120+0000 7f4c79400640 1 --2- 172.21.15.102:0/921377067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c7406ce90 0x7f4c7406d320 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009e50 tx=0x7f4c6c02bf60 comp rx=0 tx=0).stop 2024-03-23T03:48:40.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.120+0000 7f4c79400640 1 -- 172.21.15.102:0/921377067 shutdown_connections 2024-03-23T03:48:40.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.120+0000 7f4c79400640 1 --2- 172.21.15.102:0/921377067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c7406ce90 0x7f4c7406d320 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.120+0000 7f4c79400640 1 --2- 172.21.15.102:0/921377067 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c740add30 0x7f4c740ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.120+0000 7f4c79400640 1 -- 172.21.15.102:0/921377067 >> 172.21.15.102:0/921377067 conn(0x7f4c7406a0b0 msgr2=0x7f4c7406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:40.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.120+0000 7f4c79400640 1 -- 172.21.15.102:0/921377067 shutdown_connections 2024-03-23T03:48:40.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.120+0000 7f4c79400640 1 -- 172.21.15.102:0/921377067 wait complete. 2024-03-23T03:48:40.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.122+0000 7f4c79400640 1 Processor -- start 2024-03-23T03:48:40.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.122+0000 7f4c79400640 1 -- start start 2024-03-23T03:48:40.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.123+0000 7f4c79400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c7406ce90 0x7f4c7414bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:40.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.123+0000 7f4c79400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c740add30 0x7f4c741592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:40.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.123+0000 7f4c79400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c740ce930 0x7f4c741f2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:40.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.123+0000 7f4c79400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7415afc0 con 0x7f4c740ce930 2024-03-23T03:48:40.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.123+0000 7f4c79400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7414c2b0 con 0x7f4c7406ce90 2024-03-23T03:48:40.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.123+0000 7f4c79400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c74158000 con 0x7f4c740add30 2024-03-23T03:48:40.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.123+0000 7f4c73200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c740ce930 0x7f4c741f2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:40.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.123+0000 7f4c73200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c740ce930 0x7f4c741f2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55316/0 (socket says 172.21.15.102:55316) 2024-03-23T03:48:40.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.123+0000 7f4c73200640 1 -- 172.21.15.102:0/1870772845 learned_addr learned my addr 172.21.15.102:0/1870772845 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:40.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.123+0000 7f4c69e00640 1 --2- 172.21.15.102:0/1870772845 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c740add30 0x7f4c741592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:40.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.123+0000 7f4c72800640 1 --2- 172.21.15.102:0/1870772845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c7406ce90 0x7f4c7414bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:40.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.123+0000 7f4c73200640 1 -- 172.21.15.102:0/1870772845 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c740add30 msgr2=0x7f4c741592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:40.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.123+0000 7f4c73200640 1 --2- 172.21.15.102:0/1870772845 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c740add30 0x7f4c741592f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.123+0000 7f4c73200640 1 -- 172.21.15.102:0/1870772845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c7406ce90 msgr2=0x7f4c7414bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:40.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.123+0000 7f4c73200640 1 --2- 172.21.15.102:0/1870772845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c7406ce90 0x7f4c7414bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.124+0000 7f4c73200640 1 -- 172.21.15.102:0/1870772845 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c6c009b30 con 0x7f4c740ce930 2024-03-23T03:48:40.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.124+0000 7f4c73200640 1 --2- 172.21.15.102:0/1870772845 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c740ce930 0x7f4c741f2a00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009e50 tx=0x7f4c6c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:40.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.124+0000 7f4c6b400640 1 -- 172.21.15.102:0/1870772845 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c009510 con 0x7f4c740ce930 2024-03-23T03:48:40.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.124+0000 7f4c6b400640 1 -- 172.21.15.102:0/1870772845 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6c03e450 con 0x7f4c740ce930 2024-03-23T03:48:40.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.124+0000 7f4c6b400640 1 -- 172.21.15.102:0/1870772845 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c03e800 con 0x7f4c740ce930 2024-03-23T03:48:40.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.124+0000 7f4c79400640 1 -- 172.21.15.102:0/1870772845 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c74158250 con 0x7f4c740ce930 2024-03-23T03:48:40.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.124+0000 7f4c79400640 1 -- 172.21.15.102:0/1870772845 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c74158810 con 0x7f4c740ce930 2024-03-23T03:48:40.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.126+0000 7f4c79400640 1 -- 172.21.15.102:0/1870772845 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c74066bb0 con 0x7f4c740ce930 2024-03-23T03:48:40.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.127+0000 7f4c6b400640 1 -- 172.21.15.102:0/1870772845 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4c6c03e960 con 0x7f4c740ce930 2024-03-23T03:48:40.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.127+0000 7f4c6b400640 1 --2- 172.21.15.102:0/1870772845 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4c5006ac00 0x7f4c5006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:40.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.128+0000 7f4c6b400640 1 -- 172.21.15.102:0/1870772845 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4c6c0ac1d0 con 0x7f4c740ce930 2024-03-23T03:48:40.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.128+0000 7f4c72800640 1 --2- 172.21.15.102:0/1870772845 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4c5006ac00 0x7f4c5006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:40.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.128+0000 7f4c72800640 1 --2- 172.21.15.102:0/1870772845 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4c5006ac00 0x7f4c5006d0c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4c64000f60 tx=0x7f4c64009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:40.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.132+0000 7f4c6b400640 1 -- 172.21.15.102:0/1870772845 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4c6c077270 con 0x7f4c740ce930 2024-03-23T03:48:40.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.325+0000 7f4c79400640 1 -- 172.21.15.102:0/1870772845 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4c74073390 con 0x7f4c740ce930 2024-03-23T03:48:40.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.326+0000 7f4c6b400640 1 -- 172.21.15.102:0/1870772845 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4c6c07a890 con 0x7f4c740ce930 2024-03-23T03:48:40.327 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:48:40.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.329+0000 7f4c79400640 1 -- 172.21.15.102:0/1870772845 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4c5006ac00 msgr2=0x7f4c5006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:40.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.329+0000 7f4c79400640 1 --2- 172.21.15.102:0/1870772845 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4c5006ac00 0x7f4c5006d0c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4c64000f60 tx=0x7f4c64009040 comp rx=0 tx=0).stop 2024-03-23T03:48:40.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.329+0000 7f4c79400640 1 -- 172.21.15.102:0/1870772845 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c740ce930 msgr2=0x7f4c741f2a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:40.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.329+0000 7f4c79400640 1 --2- 172.21.15.102:0/1870772845 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c740ce930 0x7f4c741f2a00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009e50 tx=0x7f4c6c004a40 comp rx=0 tx=0).stop 2024-03-23T03:48:40.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.329+0000 7f4c79400640 1 -- 172.21.15.102:0/1870772845 shutdown_connections 2024-03-23T03:48:40.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.329+0000 7f4c79400640 1 --2- 172.21.15.102:0/1870772845 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4c5006ac00 0x7f4c5006d0c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.329+0000 7f4c79400640 1 --2- 172.21.15.102:0/1870772845 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c740ce930 0x7f4c741f2a00 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.329+0000 7f4c79400640 1 --2- 172.21.15.102:0/1870772845 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c740add30 0x7f4c741592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.329+0000 7f4c79400640 1 --2- 172.21.15.102:0/1870772845 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c7406ce90 0x7f4c7414bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.329+0000 7f4c79400640 1 -- 172.21.15.102:0/1870772845 >> 172.21.15.102:0/1870772845 conn(0x7f4c7406a0b0 msgr2=0x7f4c7415cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:40.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.329+0000 7f4c79400640 1 -- 172.21.15.102:0/1870772845 shutdown_connections 2024-03-23T03:48:40.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.329+0000 7f4c79400640 1 -- 172.21.15.102:0/1870772845 wait complete. 2024-03-23T03:48:40.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:48:40.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.562+0000 7f1183200640 1 Processor -- start 2024-03-23T03:48:40.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.563+0000 7f1183200640 1 -- start start 2024-03-23T03:48:40.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.564+0000 7f1183200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f117c0cd740 0x7f117c0b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:40.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.564+0000 7f1183200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f117c0d0d40 0x7f117c0cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:40.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.564+0000 7f1183200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117c0d20c0 con 0x7f117c06b0a0 2024-03-23T03:48:40.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.564+0000 7f1183200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f117c0ce060 con 0x7f117c0cd740 2024-03-23T03:48:40.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.564+0000 7f1183200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f117c0d14b0 con 0x7f117c0d0d40 2024-03-23T03:48:40.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.564+0000 7f117be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f117c0cd740 0x7f117c0b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:40.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.564+0000 7f117be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f117c0cd740 0x7f117c0b84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55326/0 (socket says 172.21.15.102:55326) 2024-03-23T03:48:40.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.564+0000 7f117be00640 1 -- 172.21.15.102:0/3282182072 learned_addr learned my addr 172.21.15.102:0/3282182072 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:40.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.565+0000 7f117b400640 1 --2- 172.21.15.102:0/3282182072 >> v2:172.21.15.173:3300/0 conn(0x7f117c0d0d40 0x7f117c0cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:40.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.565+0000 7f117be00640 1 -- 172.21.15.102:0/3282182072 >> v2:172.21.15.173:3300/0 conn(0x7f117c0d0d40 msgr2=0x7f117c0cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:40.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.565+0000 7f117be00640 1 --2- 172.21.15.102:0/3282182072 >> v2:172.21.15.173:3300/0 conn(0x7f117c0d0d40 0x7f117c0cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.565+0000 7f117be00640 1 -- 172.21.15.102:0/3282182072 >> v1:172.21.15.102:6789/0 conn(0x7f117c06b0a0 legacy=0x7f117c06b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:40.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.565+0000 7f117be00640 1 -- 172.21.15.102:0/3282182072 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f117c0d1790 con 0x7f117c0cd740 2024-03-23T03:48:40.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.566+0000 7f117be00640 1 --2- 172.21.15.102:0/3282182072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f117c0cd740 0x7f117c0b84b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f117c0b8050 tx=0x7f116c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93c807c193cca590 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:40.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.566+0000 7f117aa00640 1 -- 172.21.15.102:0/3282182072 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c030860 con 0x7f117c0cd740 2024-03-23T03:48:40.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.566+0000 7f117aa00640 1 -- 172.21.15.102:0/3282182072 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116c0309c0 con 0x7f117c0cd740 2024-03-23T03:48:40.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.566+0000 7f117aa00640 1 -- 172.21.15.102:0/3282182072 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c030d50 con 0x7f117c0cd740 2024-03-23T03:48:40.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.567+0000 7f1183200640 1 -- 172.21.15.102:0/3282182072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f117c0cd740 msgr2=0x7f117c0b84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:40.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.567+0000 7f1183200640 1 --2- 172.21.15.102:0/3282182072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f117c0cd740 0x7f117c0b84b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f117c0b8050 tx=0x7f116c02b9a0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.567+0000 7f1183200640 1 -- 172.21.15.102:0/3282182072 shutdown_connections 2024-03-23T03:48:40.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.567+0000 7f1183200640 1 --2- 172.21.15.102:0/3282182072 >> v2:172.21.15.173:3300/0 conn(0x7f117c0d0d40 0x7f117c0cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.567+0000 7f1183200640 1 --2- 172.21.15.102:0/3282182072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f117c0cd740 0x7f117c0b84b0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.567+0000 7f1183200640 1 -- 172.21.15.102:0/3282182072 >> 172.21.15.102:0/3282182072 conn(0x7f117c0cf5d0 msgr2=0x7f117c0cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:40.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.567+0000 7f1183200640 1 -- 172.21.15.102:0/3282182072 shutdown_connections 2024-03-23T03:48:40.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.567+0000 7f1183200640 1 -- 172.21.15.102:0/3282182072 wait complete. 2024-03-23T03:48:40.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.569+0000 7f1183200640 1 Processor -- start 2024-03-23T03:48:40.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.569+0000 7f1183200640 1 -- start start 2024-03-23T03:48:40.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.570+0000 7f1183200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f117c06b0a0 0x7f117c0ab9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:40.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.570+0000 7f1183200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f117c0cd740 0x7f117c0b4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:40.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.570+0000 7f1183200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f117c0d0d40 0x7f117c0b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:40.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.570+0000 7f1183200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c1f3d20 con 0x7f117c06b0a0 2024-03-23T03:48:40.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.570+0000 7f1183200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117c0aeb70 con 0x7f117c0d0d40 2024-03-23T03:48:40.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.570+0000 7f1183200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117c1f33f0 con 0x7f117c0cd740 2024-03-23T03:48:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.570+0000 7f117be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f117c06b0a0 0x7f117c0ab9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.570+0000 7f117be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f117c06b0a0 0x7f117c0ab9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55340/0 (socket says 172.21.15.102:55340) 2024-03-23T03:48:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.570+0000 7f117be00640 1 -- 172.21.15.102:0/4137420584 learned_addr learned my addr 172.21.15.102:0/4137420584 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.571+0000 7f117b400640 1 --2- 172.21.15.102:0/4137420584 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f117c0cd740 0x7f117c0b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.571+0000 7f117be00640 1 -- 172.21.15.102:0/4137420584 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f117c0cd740 msgr2=0x7f117c0b4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.571+0000 7f117be00640 1 --2- 172.21.15.102:0/4137420584 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f117c0cd740 0x7f117c0b4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.571+0000 7f1181200640 1 --2- 172.21.15.102:0/4137420584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f117c0d0d40 0x7f117c0b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.571+0000 7f117be00640 1 -- 172.21.15.102:0/4137420584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f117c0d0d40 msgr2=0x7f117c0b49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.571+0000 7f117be00640 1 --2- 172.21.15.102:0/4137420584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f117c0d0d40 0x7f117c0b49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.571+0000 7f117be00640 1 -- 172.21.15.102:0/4137420584 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116c0094d0 con 0x7f117c06b0a0 2024-03-23T03:48:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.571+0000 7f117b400640 1 --2- 172.21.15.102:0/4137420584 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f117c0cd740 0x7f117c0b4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:40.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.571+0000 7f117be00640 1 --2- 172.21.15.102:0/4137420584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f117c06b0a0 0x7f117c0ab9c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f116c02b9e0 tx=0x7f116c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:40.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.571+0000 7f1173e00640 1 -- 172.21.15.102:0/4137420584 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c039d60 con 0x7f117c06b0a0 2024-03-23T03:48:40.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.571+0000 7f1173e00640 1 -- 172.21.15.102:0/4137420584 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116c043400 con 0x7f117c06b0a0 2024-03-23T03:48:40.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.571+0000 7f1173e00640 1 -- 172.21.15.102:0/4137420584 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116c0437b0 con 0x7f117c06b0a0 2024-03-23T03:48:40.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.571+0000 7f1183200640 1 -- 172.21.15.102:0/4137420584 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f117c1f36d0 con 0x7f117c06b0a0 2024-03-23T03:48:40.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.572+0000 7f1183200640 1 -- 172.21.15.102:0/4137420584 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f117c1f30f0 con 0x7f117c06b0a0 2024-03-23T03:48:40.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.573+0000 7f1183200640 1 -- 172.21.15.102:0/4137420584 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f117c073390 con 0x7f117c06b0a0 2024-03-23T03:48:40.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.573+0000 7f1173e00640 1 -- 172.21.15.102:0/4137420584 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f116c043910 con 0x7f117c06b0a0 2024-03-23T03:48:40.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.574+0000 7f1173e00640 1 --2- 172.21.15.102:0/4137420584 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f115806ac00 0x7f115806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:40.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.574+0000 7f1173e00640 1 -- 172.21.15.102:0/4137420584 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f116c03e070 con 0x7f117c06b0a0 2024-03-23T03:48:40.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.574+0000 7f117b400640 1 --2- 172.21.15.102:0/4137420584 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f115806ac00 0x7f115806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:40.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.575+0000 7f117b400640 1 --2- 172.21.15.102:0/4137420584 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f115806ac00 0x7f115806d0c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1164006fd0 tx=0x7f1164008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:40.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.579+0000 7f1173e00640 1 -- 172.21.15.102:0/4137420584 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f116c077790 con 0x7f117c06b0a0 2024-03-23T03:48:40.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.771+0000 7f1183200640 1 -- 172.21.15.102:0/4137420584 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f117c0b79f0 con 0x7f117c06b0a0 2024-03-23T03:48:40.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.772+0000 7f1173e00640 1 -- 172.21.15.102:0/4137420584 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v41) v1 ==== 104+0+11 (secure 0 0 0) 0x7f116c07adb0 con 0x7f117c06b0a0 2024-03-23T03:48:40.773 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:48:40.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.775+0000 7f1183200640 1 -- 172.21.15.102:0/4137420584 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f115806ac00 msgr2=0x7f115806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:40.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.775+0000 7f1183200640 1 --2- 172.21.15.102:0/4137420584 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f115806ac00 0x7f115806d0c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f1164006fd0 tx=0x7f1164008040 comp rx=0 tx=0).stop 2024-03-23T03:48:40.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.775+0000 7f1183200640 1 -- 172.21.15.102:0/4137420584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f117c06b0a0 msgr2=0x7f117c0ab9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:40.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.775+0000 7f1183200640 1 --2- 172.21.15.102:0/4137420584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f117c06b0a0 0x7f117c0ab9c0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f116c02b9e0 tx=0x7f116c004730 comp rx=0 tx=0).stop 2024-03-23T03:48:40.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.775+0000 7f1183200640 1 -- 172.21.15.102:0/4137420584 shutdown_connections 2024-03-23T03:48:40.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.775+0000 7f1183200640 1 --2- 172.21.15.102:0/4137420584 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f115806ac00 0x7f115806d0c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.775+0000 7f1183200640 1 --2- 172.21.15.102:0/4137420584 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f117c0d0d40 0x7f117c0b49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.775+0000 7f1183200640 1 --2- 172.21.15.102:0/4137420584 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f117c0cd740 0x7f117c0b4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.775+0000 7f1183200640 1 --2- 172.21.15.102:0/4137420584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f117c06b0a0 0x7f117c0ab9c0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:40.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.775+0000 7f1183200640 1 -- 172.21.15.102:0/4137420584 >> 172.21.15.102:0/4137420584 conn(0x7f117c0cf5d0 msgr2=0x7f117c15f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:40.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.775+0000 7f1183200640 1 -- 172.21.15.102:0/4137420584 shutdown_connections 2024-03-23T03:48:40.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:40.775+0000 7f1183200640 1 -- 172.21.15.102:0/4137420584 wait complete. 2024-03-23T03:48:40.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:48:41.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.005+0000 7f2af5600640 1 Processor -- start 2024-03-23T03:48:41.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.005+0000 7f2af5600640 1 -- start start 2024-03-23T03:48:41.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.007+0000 7f2af5600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f2af0143460 0x7f2af015aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.007+0000 7f2af5600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2af0143930 0x7f2af00cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.007+0000 7f2af5600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2af00cdc50 0x7f2af015e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.007+0000 7f2af5600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af00cc6b0 con 0x7f2af0143930 2024-03-23T03:48:41.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.007+0000 7f2af5600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af00cc950 con 0x7f2af0143460 2024-03-23T03:48:41.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.007+0000 7f2af5600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af00ccc30 con 0x7f2af00cdc50 2024-03-23T03:48:41.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.007+0000 7f2aef400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2af0143930 0x7f2af00cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:41.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.007+0000 7f2aef400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2af0143930 0x7f2af00cd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55344/0 (socket says 172.21.15.102:55344) 2024-03-23T03:48:41.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.007+0000 7f2aef400640 1 -- 172.21.15.102:0/94703594 learned_addr learned my addr 172.21.15.102:0/94703594 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:41.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.007+0000 7f2af4c00640 1 --2- 172.21.15.102:0/94703594 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2af00cdc50 0x7f2af015e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:41.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.007+0000 7f2aefe00640 1 --2- 172.21.15.102:0/94703594 >> v2:172.21.15.173:3300/0 conn(0x7f2af0143460 0x7f2af015aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:41.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.008+0000 7f2aef400640 1 -- 172.21.15.102:0/94703594 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2af00cdc50 msgr2=0x7f2af015e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:41.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.008+0000 7f2aef400640 1 --2- 172.21.15.102:0/94703594 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2af00cdc50 0x7f2af015e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.008+0000 7f2aef400640 1 -- 172.21.15.102:0/94703594 >> v2:172.21.15.173:3300/0 conn(0x7f2af0143460 msgr2=0x7f2af015aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:41.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.008+0000 7f2aef400640 1 --2- 172.21.15.102:0/94703594 >> v2:172.21.15.173:3300/0 conn(0x7f2af0143460 0x7f2af015aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.008+0000 7f2aef400640 1 -- 172.21.15.102:0/94703594 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af00ccf10 con 0x7f2af0143930 2024-03-23T03:48:41.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.009+0000 7f2aef400640 1 --2- 172.21.15.102:0/94703594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2af0143930 0x7f2af00cd710 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2af006e410 tx=0x7f2adc02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9509abdb0ba6fe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:41.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.009+0000 7f2aeea00640 1 -- 172.21.15.102:0/94703594 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc030850 con 0x7f2af0143930 2024-03-23T03:48:41.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.009+0000 7f2aeea00640 1 -- 172.21.15.102:0/94703594 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2adc0309b0 con 0x7f2af0143930 2024-03-23T03:48:41.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.009+0000 7f2aeea00640 1 -- 172.21.15.102:0/94703594 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2adc030d40 con 0x7f2af0143930 2024-03-23T03:48:41.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.010+0000 7f2af5600640 1 -- 172.21.15.102:0/94703594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2af0143930 msgr2=0x7f2af00cd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:41.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.010+0000 7f2af5600640 1 --2- 172.21.15.102:0/94703594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2af0143930 0x7f2af00cd710 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2af006e410 tx=0x7f2adc02b990 comp rx=0 tx=0).stop 2024-03-23T03:48:41.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.010+0000 7f2af5600640 1 -- 172.21.15.102:0/94703594 shutdown_connections 2024-03-23T03:48:41.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.010+0000 7f2af5600640 1 --2- 172.21.15.102:0/94703594 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2af00cdc50 0x7f2af015e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.010+0000 7f2af5600640 1 --2- 172.21.15.102:0/94703594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2af0143930 0x7f2af00cd710 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.010+0000 7f2af5600640 1 --2- 172.21.15.102:0/94703594 >> v2:172.21.15.173:3300/0 conn(0x7f2af0143460 0x7f2af015aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.010+0000 7f2af5600640 1 -- 172.21.15.102:0/94703594 >> 172.21.15.102:0/94703594 conn(0x7f2af0065df0 msgr2=0x7f2af0069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:41.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.010+0000 7f2af5600640 1 -- 172.21.15.102:0/94703594 shutdown_connections 2024-03-23T03:48:41.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.010+0000 7f2af5600640 1 -- 172.21.15.102:0/94703594 wait complete. 2024-03-23T03:48:41.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.012+0000 7f2af5600640 1 Processor -- start 2024-03-23T03:48:41.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.012+0000 7f2af5600640 1 -- start start 2024-03-23T03:48:41.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.013+0000 7f2af5600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2af00cdc50 0x7f2af013bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.013+0000 7f2af5600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2af0143460 0x7f2af00ab960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.013+0000 7f2af5600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2af0143930 0x7f2af00ad300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.013+0000 7f2af5600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af013c230 con 0x7f2af00cdc50 2024-03-23T03:48:41.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.013+0000 7f2af5600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af00ac260 con 0x7f2af0143460 2024-03-23T03:48:41.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.013+0000 7f2af5600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af00ac540 con 0x7f2af0143930 2024-03-23T03:48:41.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.013+0000 7f2aefe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2af00cdc50 0x7f2af013bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:41.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.013+0000 7f2aefe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2af00cdc50 0x7f2af013bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55352/0 (socket says 172.21.15.102:55352) 2024-03-23T03:48:41.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.013+0000 7f2aefe00640 1 -- 172.21.15.102:0/2316969727 learned_addr learned my addr 172.21.15.102:0/2316969727 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:41.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.013+0000 7f2aefe00640 1 -- 172.21.15.102:0/2316969727 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2af0143930 msgr2=0x7f2af00ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:41.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.013+0000 7f2aefe00640 1 --2- 172.21.15.102:0/2316969727 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2af0143930 0x7f2af00ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.013+0000 7f2aefe00640 1 -- 172.21.15.102:0/2316969727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2af0143460 msgr2=0x7f2af00ab960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:41.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.013+0000 7f2aefe00640 1 --2- 172.21.15.102:0/2316969727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2af0143460 0x7f2af00ab960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.013+0000 7f2aefe00640 1 -- 172.21.15.102:0/2316969727 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2adc0094d0 con 0x7f2af00cdc50 2024-03-23T03:48:41.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.014+0000 7f2aefe00640 1 --2- 172.21.15.102:0/2316969727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2af00cdc50 0x7f2af013bcf0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2ae000d800 tx=0x7f2ae000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:41.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.014+0000 7f2ae7e00640 1 -- 172.21.15.102:0/2316969727 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae000b6c0 con 0x7f2af00cdc50 2024-03-23T03:48:41.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.014+0000 7f2ae7e00640 1 -- 172.21.15.102:0/2316969727 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae000b820 con 0x7f2af00cdc50 2024-03-23T03:48:41.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.014+0000 7f2af5600640 1 -- 172.21.15.102:0/2316969727 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af00ac820 con 0x7f2af00cdc50 2024-03-23T03:48:41.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.014+0000 7f2ae7e00640 1 -- 172.21.15.102:0/2316969727 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae000bbb0 con 0x7f2af00cdc50 2024-03-23T03:48:41.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.014+0000 7f2af5600640 1 -- 172.21.15.102:0/2316969727 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af00acde0 con 0x7f2af00cdc50 2024-03-23T03:48:41.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.015+0000 7f2af5600640 1 -- 172.21.15.102:0/2316969727 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af00adf50 con 0x7f2af00cdc50 2024-03-23T03:48:41.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.016+0000 7f2ae7e00640 1 -- 172.21.15.102:0/2316969727 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2ae000bd10 con 0x7f2af00cdc50 2024-03-23T03:48:41.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.017+0000 7f2ae7e00640 1 --2- 172.21.15.102:0/2316969727 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2ad006ac00 0x7f2ad006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.017+0000 7f2ae7e00640 1 -- 172.21.15.102:0/2316969727 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2ae0089ae0 con 0x7f2af00cdc50 2024-03-23T03:48:41.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.018+0000 7f2aef400640 1 --2- 172.21.15.102:0/2316969727 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2ad006ac00 0x7f2ad006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:41.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.018+0000 7f2aef400640 1 --2- 172.21.15.102:0/2316969727 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2ad006ac00 0x7f2ad006d0c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2adc036000 tx=0x7f2adc030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:41.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.022+0000 7f2ae7e00640 1 -- 172.21.15.102:0/2316969727 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2ae0054ae0 con 0x7f2af00cdc50 2024-03-23T03:48:41.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.214+0000 7f2af5600640 1 -- 172.21.15.102:0/2316969727 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2af004e030 con 0x7f2af00cdc50 2024-03-23T03:48:41.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.215+0000 7f2ae7e00640 1 -- 172.21.15.102:0/2316969727 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v41) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2ae0058100 con 0x7f2af00cdc50 2024-03-23T03:48:41.216 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:48:41.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.218+0000 7f2af5600640 1 -- 172.21.15.102:0/2316969727 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2ad006ac00 msgr2=0x7f2ad006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:41.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.218+0000 7f2af5600640 1 --2- 172.21.15.102:0/2316969727 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2ad006ac00 0x7f2ad006d0c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f2adc036000 tx=0x7f2adc030000 comp rx=0 tx=0).stop 2024-03-23T03:48:41.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.218+0000 7f2af5600640 1 -- 172.21.15.102:0/2316969727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2af00cdc50 msgr2=0x7f2af013bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:41.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.218+0000 7f2af5600640 1 --2- 172.21.15.102:0/2316969727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2af00cdc50 0x7f2af013bcf0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2ae000d800 tx=0x7f2ae000dcc0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.218+0000 7f2af5600640 1 -- 172.21.15.102:0/2316969727 shutdown_connections 2024-03-23T03:48:41.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.218+0000 7f2af5600640 1 --2- 172.21.15.102:0/2316969727 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2ad006ac00 0x7f2ad006d0c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.218+0000 7f2af5600640 1 --2- 172.21.15.102:0/2316969727 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2af0143930 0x7f2af00ad300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.218+0000 7f2af5600640 1 --2- 172.21.15.102:0/2316969727 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2af0143460 0x7f2af00ab960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.218+0000 7f2af5600640 1 --2- 172.21.15.102:0/2316969727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2af00cdc50 0x7f2af013bcf0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.218+0000 7f2af5600640 1 -- 172.21.15.102:0/2316969727 >> 172.21.15.102:0/2316969727 conn(0x7f2af0065df0 msgr2=0x7f2af013d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:41.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.218+0000 7f2af5600640 1 -- 172.21.15.102:0/2316969727 shutdown_connections 2024-03-23T03:48:41.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.218+0000 7f2af5600640 1 -- 172.21.15.102:0/2316969727 wait complete. 2024-03-23T03:48:41.237 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:48:41.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.438+0000 7fdee9c00640 1 Processor -- start 2024-03-23T03:48:41.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.438+0000 7fdee9c00640 1 -- start start 2024-03-23T03:48:41.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.440+0000 7fdee9c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fdee415b560 0x7fdee415b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.440+0000 7fdee9c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdee40b39b0 0x7fdee40b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.440+0000 7fdee9c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee406c6f0 con 0x7fdee415e500 2024-03-23T03:48:41.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.440+0000 7fdee9c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee40b4370 con 0x7fdee40b39b0 2024-03-23T03:48:41.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.440+0000 7fdee9c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee406c9d0 con 0x7fdee415b560 2024-03-23T03:48:41.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.440+0000 7fdeda000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdee40b39b0 0x7fdee40b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:41.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.440+0000 7fdeda000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdee40b39b0 0x7fdee40b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55368/0 (socket says 172.21.15.102:55368) 2024-03-23T03:48:41.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.440+0000 7fdeda000640 1 -- 172.21.15.102:0/398520463 learned_addr learned my addr 172.21.15.102:0/398520463 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:41.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.441+0000 7fdee2a00640 1 --2- 172.21.15.102:0/398520463 >> v2:172.21.15.173:3300/0 conn(0x7fdee415b560 0x7fdee415b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:41.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.441+0000 7fdeda000640 1 -- 172.21.15.102:0/398520463 >> v2:172.21.15.173:3300/0 conn(0x7fdee415b560 msgr2=0x7fdee415b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:41.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.441+0000 7fdeda000640 1 --2- 172.21.15.102:0/398520463 >> v2:172.21.15.173:3300/0 conn(0x7fdee415b560 0x7fdee415b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.441+0000 7fdeda000640 1 -- 172.21.15.102:0/398520463 >> v1:172.21.15.173:6789/0 conn(0x7fdee415e500 legacy=0x7fdee415e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:41.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.441+0000 7fdeda000640 1 -- 172.21.15.102:0/398520463 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee406ccb0 con 0x7fdee40b39b0 2024-03-23T03:48:41.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.442+0000 7fdeda000640 1 --2- 172.21.15.102:0/398520463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdee40b39b0 0x7fdee40b3e30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fdee4071470 tx=0x7fdecc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b209d5f27f55b369 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:41.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.443+0000 7fdee2000640 1 -- 172.21.15.102:0/398520463 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdecc031870 con 0x7fdee40b39b0 2024-03-23T03:48:41.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.443+0000 7fdee2000640 1 -- 172.21.15.102:0/398520463 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdecc0319d0 con 0x7fdee40b39b0 2024-03-23T03:48:41.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.443+0000 7fdee2000640 1 -- 172.21.15.102:0/398520463 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdecc031d60 con 0x7fdee40b39b0 2024-03-23T03:48:41.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.443+0000 7fdee9c00640 1 -- 172.21.15.102:0/398520463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdee40b39b0 msgr2=0x7fdee40b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:41.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.443+0000 7fdee9c00640 1 --2- 172.21.15.102:0/398520463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdee40b39b0 0x7fdee40b3e30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fdee4071470 tx=0x7fdecc02b910 comp rx=0 tx=0).stop 2024-03-23T03:48:41.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.443+0000 7fdee9c00640 1 -- 172.21.15.102:0/398520463 shutdown_connections 2024-03-23T03:48:41.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.443+0000 7fdee9c00640 1 --2- 172.21.15.102:0/398520463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdee40b39b0 0x7fdee40b3e30 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.443+0000 7fdee9c00640 1 --2- 172.21.15.102:0/398520463 >> v2:172.21.15.173:3300/0 conn(0x7fdee415b560 0x7fdee415b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.443+0000 7fdee9c00640 1 -- 172.21.15.102:0/398520463 >> 172.21.15.102:0/398520463 conn(0x7fdee406a3d0 msgr2=0x7fdee40b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:41.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.444+0000 7fdee9c00640 1 -- 172.21.15.102:0/398520463 shutdown_connections 2024-03-23T03:48:41.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.444+0000 7fdee9c00640 1 -- 172.21.15.102:0/398520463 wait complete. 2024-03-23T03:48:41.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.445+0000 7fdee9c00640 1 Processor -- start 2024-03-23T03:48:41.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.445+0000 7fdee9c00640 1 -- start start 2024-03-23T03:48:41.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.446+0000 7fdee9c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdee40b39b0 0x7fdee4148d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.446+0000 7fdee9c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdee415b560 0x7fdee4155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.446+0000 7fdee9c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdee415e500 0x7fdee40ace80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.446+0000 7fdee9c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee4155d50 con 0x7fdee415b560 2024-03-23T03:48:41.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.446+0000 7fdee9c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee4156030 con 0x7fdee415e500 2024-03-23T03:48:41.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.446+0000 7fdee9c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee4156310 con 0x7fdee40b39b0 2024-03-23T03:48:41.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.447+0000 7fdeda000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdee415b560 0x7fdee4155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:41.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.447+0000 7fdeda000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdee415b560 0x7fdee4155520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55382/0 (socket says 172.21.15.102:55382) 2024-03-23T03:48:41.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.447+0000 7fdeda000640 1 -- 172.21.15.102:0/50593924 learned_addr learned my addr 172.21.15.102:0/50593924 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:41.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.447+0000 7fdee2a00640 1 --2- 172.21.15.102:0/50593924 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdee40b39b0 0x7fdee4148d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:41.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.447+0000 7fdee3400640 1 --2- 172.21.15.102:0/50593924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdee415e500 0x7fdee40ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:41.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.447+0000 7fdeda000640 1 -- 172.21.15.102:0/50593924 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdee40b39b0 msgr2=0x7fdee4148d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:41.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.447+0000 7fdeda000640 1 --2- 172.21.15.102:0/50593924 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdee40b39b0 0x7fdee4148d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.447+0000 7fdeda000640 1 -- 172.21.15.102:0/50593924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdee415e500 msgr2=0x7fdee40ace80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:41.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.447+0000 7fdeda000640 1 --2- 172.21.15.102:0/50593924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdee415e500 0x7fdee40ace80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.447+0000 7fdeda000640 1 -- 172.21.15.102:0/50593924 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdecc0094d0 con 0x7fdee415b560 2024-03-23T03:48:41.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.447+0000 7fdee3400640 1 --2- 172.21.15.102:0/50593924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdee415e500 0x7fdee40ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:41.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.447+0000 7fdee2a00640 1 --2- 172.21.15.102:0/50593924 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdee40b39b0 0x7fdee4148d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:41.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.447+0000 7fdeda000640 1 --2- 172.21.15.102:0/50593924 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdee415b560 0x7fdee4155520 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fdecc000c00 tx=0x7fdecc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:41.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.448+0000 7fdedb400640 1 -- 172.21.15.102:0/50593924 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdecc03abe0 con 0x7fdee415b560 2024-03-23T03:48:41.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.448+0000 7fdedb400640 1 -- 172.21.15.102:0/50593924 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdecc03ad40 con 0x7fdee415b560 2024-03-23T03:48:41.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.448+0000 7fdee9c00640 1 -- 172.21.15.102:0/50593924 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee41565f0 con 0x7fdee415b560 2024-03-23T03:48:41.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.448+0000 7fdedb400640 1 -- 172.21.15.102:0/50593924 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdecc044610 con 0x7fdee415b560 2024-03-23T03:48:41.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.448+0000 7fdee9c00640 1 -- 172.21.15.102:0/50593924 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee4155a60 con 0x7fdee415b560 2024-03-23T03:48:41.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.449+0000 7fdee9c00640 1 -- 172.21.15.102:0/50593924 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee406ac20 con 0x7fdee415b560 2024-03-23T03:48:41.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.450+0000 7fdedb400640 1 -- 172.21.15.102:0/50593924 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdecc04d450 con 0x7fdee415b560 2024-03-23T03:48:41.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.451+0000 7fdedb400640 1 --2- 172.21.15.102:0/50593924 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdec006ac00 0x7fdec006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.451+0000 7fdedb400640 1 -- 172.21.15.102:0/50593924 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fdecc02d040 con 0x7fdee415b560 2024-03-23T03:48:41.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.452+0000 7fdee2a00640 1 --2- 172.21.15.102:0/50593924 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdec006ac00 0x7fdec006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:41.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.452+0000 7fdee2a00640 1 --2- 172.21.15.102:0/50593924 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdec006ac00 0x7fdec006d0c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fded4006fd0 tx=0x7fded4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:41.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.456+0000 7fdedb400640 1 -- 172.21.15.102:0/50593924 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdecc076b40 con 0x7fdee415b560 2024-03-23T03:48:41.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.694+0000 7fdee9c00640 1 -- 172.21.15.102:0/50593924 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdee4073390 con 0x7fdee415b560 2024-03-23T03:48:41.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.695+0000 7fdedb400640 1 -- 172.21.15.102:0/50593924 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4746 (secure 0 0 0) 0x7fdecc035020 con 0x7fdee415b560 2024-03-23T03:48:41.696 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:41.696 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6834/1537100581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1537100581},{"type":"v1","addr":"172.21.15.173:6834","nonce":1537100581}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4655,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3206638952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":3206638952},{"type":"v1","addr":"172.21.15.173:6835","nonce":3206638952}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4852,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4005240382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4005240382},{"type":"v1","addr":"172.21.15.102:6837","nonce":4005240382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:48:23.418926+0000","modified":"2024-03-23T03:48:26.657662+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.102:6836/1373501465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":1373501465},{"type":"v1","addr":"172.21.15.102:6836","nonce":1373501465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-23T03:48:41.696 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 16 2024-03-23T03:48:41.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.697+0000 7fdee9c00640 1 -- 172.21.15.102:0/50593924 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdec006ac00 msgr2=0x7fdec006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:41.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.697+0000 7fdee9c00640 1 --2- 172.21.15.102:0/50593924 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdec006ac00 0x7fdec006d0c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fded4006fd0 tx=0x7fded4008040 comp rx=0 tx=0).stop 2024-03-23T03:48:41.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.697+0000 7fdee9c00640 1 -- 172.21.15.102:0/50593924 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdee415b560 msgr2=0x7fdee4155520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:41.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.697+0000 7fdee9c00640 1 --2- 172.21.15.102:0/50593924 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdee415b560 0x7fdee4155520 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fdecc000c00 tx=0x7fdecc0046c0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.698+0000 7fdee9c00640 1 -- 172.21.15.102:0/50593924 shutdown_connections 2024-03-23T03:48:41.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.698+0000 7fdee9c00640 1 --2- 172.21.15.102:0/50593924 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdec006ac00 0x7fdec006d0c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.698+0000 7fdee9c00640 1 --2- 172.21.15.102:0/50593924 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdee415e500 0x7fdee40ace80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.698+0000 7fdee9c00640 1 --2- 172.21.15.102:0/50593924 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdee415b560 0x7fdee4155520 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.698+0000 7fdee9c00640 1 --2- 172.21.15.102:0/50593924 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdee40b39b0 0x7fdee4148d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.698+0000 7fdee9c00640 1 -- 172.21.15.102:0/50593924 >> 172.21.15.102:0/50593924 conn(0x7fdee406a3d0 msgr2=0x7fdee40cb160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:41.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.698+0000 7fdee9c00640 1 -- 172.21.15.102:0/50593924 shutdown_connections 2024-03-23T03:48:41.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.698+0000 7fdee9c00640 1 -- 172.21.15.102:0/50593924 wait complete. 2024-03-23T03:48:41.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:48:41.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.922+0000 7fcba2800640 1 Processor -- start 2024-03-23T03:48:41.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.922+0000 7fcba2800640 1 -- start start 2024-03-23T03:48:41.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.924+0000 7fcba2800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcb9c15ccb0 0x7fcb9c0cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.924+0000 7fcba2800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fcb9c0cdc10 0x7fcb9c0d1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.924+0000 7fcba2800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb9c1610c0 con 0x7fcb9c15c8d0 2024-03-23T03:48:41.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.924+0000 7fcba2800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb9c0ce0c0 con 0x7fcb9c15ccb0 2024-03-23T03:48:41.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.924+0000 7fcba2800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb9c0b5a30 con 0x7fcb9c0cdc10 2024-03-23T03:48:41.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.924+0000 7fcb92a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcb9c15ccb0 0x7fcb9c0cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:41.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.924+0000 7fcb92a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcb9c15ccb0 0x7fcb9c0cd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55390/0 (socket says 172.21.15.102:55390) 2024-03-23T03:48:41.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.924+0000 7fcb92a00640 1 -- 172.21.15.102:0/200303403 learned_addr learned my addr 172.21.15.102:0/200303403 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:41.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.925+0000 7fcb9be00640 1 --2- 172.21.15.102:0/200303403 >> v2:172.21.15.173:3300/0 conn(0x7fcb9c0cdc10 0x7fcb9c0d1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:41.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.925+0000 7fcb92a00640 1 -- 172.21.15.102:0/200303403 >> v2:172.21.15.173:3300/0 conn(0x7fcb9c0cdc10 msgr2=0x7fcb9c0d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:41.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.925+0000 7fcb92a00640 1 --2- 172.21.15.102:0/200303403 >> v2:172.21.15.173:3300/0 conn(0x7fcb9c0cdc10 0x7fcb9c0d1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.925+0000 7fcb92a00640 1 -- 172.21.15.102:0/200303403 >> v1:172.21.15.173:6789/0 conn(0x7fcb9c15c8d0 legacy=0x7fcb9c160670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:41.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.925+0000 7fcb92a00640 1 -- 172.21.15.102:0/200303403 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c1613a0 con 0x7fcb9c15ccb0 2024-03-23T03:48:41.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.925+0000 7fcb9be00640 1 --2- 172.21.15.102:0/200303403 >> v2:172.21.15.173:3300/0 conn(0x7fcb9c0cdc10 0x7fcb9c0d1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:41.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.926+0000 7fcb92a00640 1 --2- 172.21.15.102:0/200303403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb9c15ccb0 0x7fcb9c0cd6d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c0ce520 tx=0x7fcb8c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fff4c9dbc011c647 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:41.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.926+0000 7fcb9aa00640 1 -- 172.21.15.102:0/200303403 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c031750 con 0x7fcb9c15ccb0 2024-03-23T03:48:41.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.926+0000 7fcb9aa00640 1 -- 172.21.15.102:0/200303403 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb8c0318b0 con 0x7fcb9c15ccb0 2024-03-23T03:48:41.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.926+0000 7fcb9aa00640 1 -- 172.21.15.102:0/200303403 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb8c031c40 con 0x7fcb9c15ccb0 2024-03-23T03:48:41.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.927+0000 7fcba2800640 1 -- 172.21.15.102:0/200303403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb9c15ccb0 msgr2=0x7fcb9c0cd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:41.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.927+0000 7fcba2800640 1 --2- 172.21.15.102:0/200303403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb9c15ccb0 0x7fcb9c0cd6d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c0ce520 tx=0x7fcb8c02b7f0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.927+0000 7fcba2800640 1 -- 172.21.15.102:0/200303403 shutdown_connections 2024-03-23T03:48:41.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.927+0000 7fcba2800640 1 --2- 172.21.15.102:0/200303403 >> v2:172.21.15.173:3300/0 conn(0x7fcb9c0cdc10 0x7fcb9c0d1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.927+0000 7fcba2800640 1 --2- 172.21.15.102:0/200303403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb9c15ccb0 0x7fcb9c0cd6d0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.927+0000 7fcba2800640 1 -- 172.21.15.102:0/200303403 >> 172.21.15.102:0/200303403 conn(0x7fcb9c06b0e0 msgr2=0x7fcb9c06b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:41.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.927+0000 7fcba2800640 1 -- 172.21.15.102:0/200303403 shutdown_connections 2024-03-23T03:48:41.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.927+0000 7fcba2800640 1 -- 172.21.15.102:0/200303403 wait complete. 2024-03-23T03:48:41.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.929+0000 7fcba2800640 1 Processor -- start 2024-03-23T03:48:41.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.929+0000 7fcba2800640 1 -- start start 2024-03-23T03:48:41.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.930+0000 7fcba2800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcb9c0cdc10 0x7fcb9c14ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.930+0000 7fcba2800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb9c15c8d0 0x7fcb9c0c8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.930+0000 7fcba2800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb9c15ccb0 0x7fcb9c0cab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.930+0000 7fcba2800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c0c6620 con 0x7fcb9c15ccb0 2024-03-23T03:48:41.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.930+0000 7fcba2800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c15d100 con 0x7fcb9c15c8d0 2024-03-23T03:48:41.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.930+0000 7fcba2800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb9c0c68c0 con 0x7fcb9c0cdc10 2024-03-23T03:48:41.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.931+0000 7fcb9be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb9c15ccb0 0x7fcb9c0cab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:41.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.931+0000 7fcb9be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb9c15ccb0 0x7fcb9c0cab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55396/0 (socket says 172.21.15.102:55396) 2024-03-23T03:48:41.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.931+0000 7fcb9be00640 1 -- 172.21.15.102:0/3923333768 learned_addr learned my addr 172.21.15.102:0/3923333768 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:41.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.931+0000 7fcb9b400640 1 --2- 172.21.15.102:0/3923333768 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcb9c0cdc10 0x7fcb9c14ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:41.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.931+0000 7fcb92a00640 1 --2- 172.21.15.102:0/3923333768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb9c15c8d0 0x7fcb9c0c8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:41.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.931+0000 7fcb9be00640 1 -- 172.21.15.102:0/3923333768 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcb9c0cdc10 msgr2=0x7fcb9c14ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:41.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.931+0000 7fcb9be00640 1 --2- 172.21.15.102:0/3923333768 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcb9c0cdc10 0x7fcb9c14ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.931+0000 7fcb9be00640 1 -- 172.21.15.102:0/3923333768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb9c15c8d0 msgr2=0x7fcb9c0c8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:41.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.931+0000 7fcb9be00640 1 --2- 172.21.15.102:0/3923333768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb9c15c8d0 0x7fcb9c0c8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:41.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.931+0000 7fcb9be00640 1 -- 172.21.15.102:0/3923333768 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb8c0094d0 con 0x7fcb9c15ccb0 2024-03-23T03:48:41.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.931+0000 7fcb9b400640 1 --2- 172.21.15.102:0/3923333768 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcb9c0cdc10 0x7fcb9c14ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:48:41.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.932+0000 7fcb9be00640 1 --2- 172.21.15.102:0/3923333768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb9c15ccb0 0x7fcb9c0cab50 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400ce70 tx=0x7fcb9400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:41.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.932+0000 7fcb93e00640 1 -- 172.21.15.102:0/3923333768 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9400d840 con 0x7fcb9c15ccb0 2024-03-23T03:48:41.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.932+0000 7fcb93e00640 1 -- 172.21.15.102:0/3923333768 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9400d9a0 con 0x7fcb9c15ccb0 2024-03-23T03:48:41.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.932+0000 7fcb93e00640 1 -- 172.21.15.102:0/3923333768 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9400dd50 con 0x7fcb9c15ccb0 2024-03-23T03:48:41.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.932+0000 7fcba2800640 1 -- 172.21.15.102:0/3923333768 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb9c0c8e70 con 0x7fcb9c15ccb0 2024-03-23T03:48:41.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.932+0000 7fcba2800640 1 -- 172.21.15.102:0/3923333768 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb9c15d630 con 0x7fcb9c15ccb0 2024-03-23T03:48:41.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.933+0000 7fcba2800640 1 -- 172.21.15.102:0/3923333768 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb9c15d820 con 0x7fcb9c15ccb0 2024-03-23T03:48:41.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.934+0000 7fcb93e00640 1 -- 172.21.15.102:0/3923333768 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcb9401e3b0 con 0x7fcb9c15ccb0 2024-03-23T03:48:41.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.935+0000 7fcb93e00640 1 --2- 172.21.15.102:0/3923333768 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcb7806ac00 0x7fcb7806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:41.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.935+0000 7fcb93e00640 1 -- 172.21.15.102:0/3923333768 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fcb94089f00 con 0x7fcb9c15ccb0 2024-03-23T03:48:41.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.935+0000 7fcb9b400640 1 --2- 172.21.15.102:0/3923333768 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcb7806ac00 0x7fcb7806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:41.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.936+0000 7fcb9b400640 1 --2- 172.21.15.102:0/3923333768 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcb7806ac00 0x7fcb7806d0c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c0d07e0 tx=0x7fcb84009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:41.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:41.939+0000 7fcb93e00640 1 -- 172.21.15.102:0/3923333768 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcb94054f70 con 0x7fcb9c15ccb0 2024-03-23T03:48:42.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.135+0000 7fcba2800640 1 -- 172.21.15.102:0/3923333768 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb9c073390 con 0x7fcb9c15ccb0 2024-03-23T03:48:42.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.138+0000 7fcb93e00640 1 -- 172.21.15.102:0/3923333768 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16403 (secure 0 0 0) 0x7fcb9401a020 con 0x7fcb9c15ccb0 2024-03-23T03:48:42.139 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:42.139 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":42,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:41.662229+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:48:21.550281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-23T03:48:22.428475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:48:42.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.140+0000 7fcba2800640 1 -- 172.21.15.102:0/3923333768 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcb7806ac00 msgr2=0x7fcb7806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:42.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.140+0000 7fcba2800640 1 --2- 172.21.15.102:0/3923333768 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcb7806ac00 0x7fcb7806d0c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c0d07e0 tx=0x7fcb84009040 comp rx=0 tx=0).stop 2024-03-23T03:48:42.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.140+0000 7fcba2800640 1 -- 172.21.15.102:0/3923333768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb9c15ccb0 msgr2=0x7fcb9c0cab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:42.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.140+0000 7fcba2800640 1 --2- 172.21.15.102:0/3923333768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb9c15ccb0 0x7fcb9c0cab50 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400ce70 tx=0x7fcb9400a710 comp rx=0 tx=0).stop 2024-03-23T03:48:42.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.140+0000 7fcba2800640 1 -- 172.21.15.102:0/3923333768 shutdown_connections 2024-03-23T03:48:42.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.140+0000 7fcba2800640 1 --2- 172.21.15.102:0/3923333768 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcb7806ac00 0x7fcb7806d0c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.140+0000 7fcba2800640 1 --2- 172.21.15.102:0/3923333768 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb9c15ccb0 0x7fcb9c0cab50 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.140+0000 7fcba2800640 1 --2- 172.21.15.102:0/3923333768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcb9c15c8d0 0x7fcb9c0c8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.141+0000 7fcba2800640 1 --2- 172.21.15.102:0/3923333768 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcb9c0cdc10 0x7fcb9c14ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.141+0000 7fcba2800640 1 -- 172.21.15.102:0/3923333768 >> 172.21.15.102:0/3923333768 conn(0x7fcb9c06b0e0 msgr2=0x7fcb9c152d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:42.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.141+0000 7fcba2800640 1 -- 172.21.15.102:0/3923333768 shutdown_connections 2024-03-23T03:48:42.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.141+0000 7fcba2800640 1 -- 172.21.15.102:0/3923333768 wait complete. 2024-03-23T03:48:42.160 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T03:48:42.160 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:48:42.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.362+0000 7f4303600640 1 Processor -- start 2024-03-23T03:48:42.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.362+0000 7f4303600640 1 -- start start 2024-03-23T03:48:42.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.364+0000 7f4303600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f42fc0b39b0 0x7f42fc0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:42.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.364+0000 7f4303600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f42fc15c400 0x7f42fc15c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:42.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.364+0000 7f4303600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42fc06c6f0 con 0x7f42fc15b560 2024-03-23T03:48:42.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.364+0000 7f4303600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42fc0b4370 con 0x7f42fc0b39b0 2024-03-23T03:48:42.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.364+0000 7f4303600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42fc06c9d0 con 0x7f42fc15c400 2024-03-23T03:48:42.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.364+0000 7f42f3e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f42fc0b39b0 0x7f42fc0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:42.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.364+0000 7f4300c00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f42fc15b560 0x7f42fc15b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53752/0 (socket says 172.21.15.102:53752) 2024-03-23T03:48:42.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.364+0000 7f4300c00640 1 -- 172.21.15.102:0/476832351 learned_addr learned my addr 172.21.15.102:0/476832351 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:42.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.365+0000 7f42f3e00640 1 -- 172.21.15.102:0/476832351 >> v2:172.21.15.173:3300/0 conn(0x7f42fc15c400 msgr2=0x7f42fc15c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:42.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.365+0000 7f4301600640 1 --2- 172.21.15.102:0/476832351 >> v2:172.21.15.173:3300/0 conn(0x7f42fc15c400 0x7f42fc15c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:42.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.365+0000 7f42fbe00640 1 -- 172.21.15.102:0/476832351 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405322174 0 0) 0x7f42fc06c6f0 con 0x7f42fc15b560 2024-03-23T03:48:42.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.365+0000 7f42f3e00640 1 --2- 172.21.15.102:0/476832351 >> v2:172.21.15.173:3300/0 conn(0x7f42fc15c400 0x7f42fc15c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.365+0000 7f42f3e00640 1 -- 172.21.15.102:0/476832351 >> v1:172.21.15.102:6789/0 conn(0x7f42fc15b560 legacy=0x7f42fc15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:42.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.365+0000 7f42f3e00640 1 -- 172.21.15.102:0/476832351 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42fc06ccb0 con 0x7f42fc0b39b0 2024-03-23T03:48:42.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.365+0000 7f4301600640 1 --2- 172.21.15.102:0/476832351 >> v2:172.21.15.173:3300/0 conn(0x7f42fc15c400 0x7f42fc15c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:42.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.366+0000 7f42f3e00640 1 --2- 172.21.15.102:0/476832351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42fc0b39b0 0x7f42fc0b3e30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f42fc070100 tx=0x7f42e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586f5dfd8d4b4516 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:42.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.366+0000 7f42fbe00640 1 -- 172.21.15.102:0/476832351 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e4031830 con 0x7f42fc0b39b0 2024-03-23T03:48:42.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.366+0000 7f42fbe00640 1 -- 172.21.15.102:0/476832351 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e4031990 con 0x7f42fc0b39b0 2024-03-23T03:48:42.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.366+0000 7f42fbe00640 1 -- 172.21.15.102:0/476832351 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e4031d20 con 0x7f42fc0b39b0 2024-03-23T03:48:42.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.367+0000 7f4303600640 1 -- 172.21.15.102:0/476832351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42fc0b39b0 msgr2=0x7f42fc0b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:42.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.367+0000 7f4303600640 1 --2- 172.21.15.102:0/476832351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42fc0b39b0 0x7f42fc0b3e30 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f42fc070100 tx=0x7f42e402b8d0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.367+0000 7f4303600640 1 -- 172.21.15.102:0/476832351 shutdown_connections 2024-03-23T03:48:42.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.367+0000 7f4303600640 1 --2- 172.21.15.102:0/476832351 >> v2:172.21.15.173:3300/0 conn(0x7f42fc15c400 0x7f42fc15c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.367+0000 7f4303600640 1 --2- 172.21.15.102:0/476832351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42fc0b39b0 0x7f42fc0b3e30 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.367+0000 7f4303600640 1 -- 172.21.15.102:0/476832351 >> 172.21.15.102:0/476832351 conn(0x7f42fc06a3d0 msgr2=0x7f42fc0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:42.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.367+0000 7f4303600640 1 -- 172.21.15.102:0/476832351 shutdown_connections 2024-03-23T03:48:42.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.367+0000 7f4303600640 1 -- 172.21.15.102:0/476832351 wait complete. 2024-03-23T03:48:42.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.369+0000 7f4303600640 1 Processor -- start 2024-03-23T03:48:42.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.369+0000 7f4303600640 1 -- start start 2024-03-23T03:48:42.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.370+0000 7f4303600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f42fc0b39b0 0x7f42fc150d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:42.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.370+0000 7f4303600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42fc15b560 0x7f42fc150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:42.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.370+0000 7f4303600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42fc15c400 0x7f42fc150670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:42.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.370+0000 7f4303600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc150bb0 con 0x7f42fc15b560 2024-03-23T03:48:42.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.370+0000 7f4303600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc0ad650 con 0x7f42fc15c400 2024-03-23T03:48:42.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.370+0000 7f4303600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42fc0ac890 con 0x7f42fc0b39b0 2024-03-23T03:48:42.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.370+0000 7f42f3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42fc15b560 0x7f42fc150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:42.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.370+0000 7f42f3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42fc15b560 0x7f42fc150130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55414/0 (socket says 172.21.15.102:55414) 2024-03-23T03:48:42.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.370+0000 7f42f3e00640 1 -- 172.21.15.102:0/4029302828 learned_addr learned my addr 172.21.15.102:0/4029302828 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:42.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.370+0000 7f4300c00640 1 --2- 172.21.15.102:0/4029302828 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f42fc0b39b0 0x7f42fc150d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:42.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.370+0000 7f4301600640 1 --2- 172.21.15.102:0/4029302828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42fc15c400 0x7f42fc150670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:42.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.370+0000 7f42f3e00640 1 -- 172.21.15.102:0/4029302828 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f42fc0b39b0 msgr2=0x7f42fc150d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:42.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.370+0000 7f42f3e00640 1 --2- 172.21.15.102:0/4029302828 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f42fc0b39b0 0x7f42fc150d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.370+0000 7f42f3e00640 1 -- 172.21.15.102:0/4029302828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42fc15c400 msgr2=0x7f42fc150670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:42.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.370+0000 7f42f3e00640 1 --2- 172.21.15.102:0/4029302828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42fc15c400 0x7f42fc150670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.370+0000 7f42f3e00640 1 -- 172.21.15.102:0/4029302828 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e4009400 con 0x7f42fc15b560 2024-03-23T03:48:42.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.371+0000 7f42f3e00640 1 --2- 172.21.15.102:0/4029302828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42fc15b560 0x7f42fc150130 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f42e4009890 tx=0x7f42e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:42.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.371+0000 7f42f9600640 1 -- 172.21.15.102:0/4029302828 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e403ba70 con 0x7f42fc15b560 2024-03-23T03:48:42.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.371+0000 7f42f9600640 1 -- 172.21.15.102:0/4029302828 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e403bbf0 con 0x7f42fc15b560 2024-03-23T03:48:42.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.371+0000 7f42f9600640 1 -- 172.21.15.102:0/4029302828 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e4045400 con 0x7f42fc15b560 2024-03-23T03:48:42.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.371+0000 7f4303600640 1 -- 172.21.15.102:0/4029302828 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42fc0acb70 con 0x7f42fc15b560 2024-03-23T03:48:42.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.371+0000 7f4303600640 1 -- 172.21.15.102:0/4029302828 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42fc0ad130 con 0x7f42fc15b560 2024-03-23T03:48:42.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.373+0000 7f4303600640 1 -- 172.21.15.102:0/4029302828 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42fc06ac20 con 0x7f42fc15b560 2024-03-23T03:48:42.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.373+0000 7f42f9600640 1 -- 172.21.15.102:0/4029302828 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f42e404e440 con 0x7f42fc15b560 2024-03-23T03:48:42.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.374+0000 7f42f9600640 1 --2- 172.21.15.102:0/4029302828 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f42d806ac00 0x7f42d806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:42.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.374+0000 7f42f9600640 1 -- 172.21.15.102:0/4029302828 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f42e40ab9a0 con 0x7f42fc15b560 2024-03-23T03:48:42.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.374+0000 7f4300c00640 1 --2- 172.21.15.102:0/4029302828 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f42d806ac00 0x7f42d806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:42.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.375+0000 7f4300c00640 1 --2- 172.21.15.102:0/4029302828 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f42d806ac00 0x7f42d806d0c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f42ec000f60 tx=0x7f42ec007b70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:42.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.379+0000 7f42f9600640 1 -- 172.21.15.102:0/4029302828 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f42e4076aa0 con 0x7f42fc15b560 2024-03-23T03:48:42.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.616+0000 7f4303600640 1 -- 172.21.15.102:0/4029302828 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42fc0ad320 con 0x7f42fc15b560 2024-03-23T03:48:42.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.617+0000 7f42f9600640 1 -- 172.21.15.102:0/4029302828 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f42e407a0c0 con 0x7f42fc15b560 2024-03-23T03:48:42.618 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:42.618 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6834/1537100581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1537100581},{"type":"v1","addr":"172.21.15.173:6834","nonce":1537100581}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4655,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3206638952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":3206638952},{"type":"v1","addr":"172.21.15.173:6835","nonce":3206638952}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4852,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4005240382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4005240382},{"type":"v1","addr":"172.21.15.102:6837","nonce":4005240382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:48:23.418926+0000","modified":"2024-03-23T03:48:26.657662+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.102:6836/1373501465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":1373501465},{"type":"v1","addr":"172.21.15.102:6836","nonce":1373501465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-23T03:48:42.618 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 16 2024-03-23T03:48:42.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.620+0000 7f4303600640 1 -- 172.21.15.102:0/4029302828 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f42d806ac00 msgr2=0x7f42d806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:42.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.620+0000 7f4303600640 1 --2- 172.21.15.102:0/4029302828 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f42d806ac00 0x7f42d806d0c0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f42ec000f60 tx=0x7f42ec007b70 comp rx=0 tx=0).stop 2024-03-23T03:48:42.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.620+0000 7f4303600640 1 -- 172.21.15.102:0/4029302828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42fc15b560 msgr2=0x7f42fc150130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:42.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.620+0000 7f4303600640 1 --2- 172.21.15.102:0/4029302828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42fc15b560 0x7f42fc150130 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f42e4009890 tx=0x7f42e40046c0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.620+0000 7f4303600640 1 -- 172.21.15.102:0/4029302828 shutdown_connections 2024-03-23T03:48:42.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.620+0000 7f4303600640 1 --2- 172.21.15.102:0/4029302828 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f42d806ac00 0x7f42d806d0c0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.620+0000 7f4303600640 1 --2- 172.21.15.102:0/4029302828 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42fc15c400 0x7f42fc150670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.620+0000 7f4303600640 1 --2- 172.21.15.102:0/4029302828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42fc15b560 0x7f42fc150130 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.620+0000 7f4303600640 1 --2- 172.21.15.102:0/4029302828 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f42fc0b39b0 0x7f42fc150d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.620+0000 7f4303600640 1 -- 172.21.15.102:0/4029302828 >> 172.21.15.102:0/4029302828 conn(0x7f42fc06a3d0 msgr2=0x7f42fc158aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:42.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.620+0000 7f4303600640 1 -- 172.21.15.102:0/4029302828 shutdown_connections 2024-03-23T03:48:42.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.620+0000 7f4303600640 1 -- 172.21.15.102:0/4029302828 wait complete. 2024-03-23T03:48:42.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:48:42.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.834+0000 7f8fa8400640 1 Processor -- start 2024-03-23T03:48:42.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.834+0000 7f8fa8400640 1 -- start start 2024-03-23T03:48:42.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.835+0000 7f8fa8400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f8fa00adbc0 0x7f8fa00b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:42.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.835+0000 7f8fa8400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8fa00cbab0 0x7f8fa00b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:42.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.835+0000 7f8fa8400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fa00b16a0 con 0x7f8fa006d390 2024-03-23T03:48:42.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.835+0000 7f8fa8400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa00b4700 con 0x7f8fa00cbab0 2024-03-23T03:48:42.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.835+0000 7f8fa8400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fa00cbfb0 con 0x7f8fa00adbc0 2024-03-23T03:48:42.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.836+0000 7f8fa5000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8fa00cbab0 0x7f8fa00b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:42.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.836+0000 7f8fa6400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8fa006d390 0x7f8fa00b2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:53766/0 (socket says 172.21.15.102:53766) 2024-03-23T03:48:42.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.836+0000 7f8fa6400640 1 -- 172.21.15.102:0/2399882193 learned_addr learned my addr 172.21.15.102:0/2399882193 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:42.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.836+0000 7f8fa5a00640 1 --2- 172.21.15.102:0/2399882193 >> v2:172.21.15.173:3300/0 conn(0x7f8fa00adbc0 0x7f8fa00b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:42.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.836+0000 7f8fa5000640 1 -- 172.21.15.102:0/2399882193 >> v2:172.21.15.173:3300/0 conn(0x7f8fa00adbc0 msgr2=0x7f8fa00b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:42.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.836+0000 7f8fa5000640 1 --2- 172.21.15.102:0/2399882193 >> v2:172.21.15.173:3300/0 conn(0x7f8fa00adbc0 0x7f8fa00b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.836+0000 7f8fa5000640 1 -- 172.21.15.102:0/2399882193 >> v1:172.21.15.102:6789/0 conn(0x7f8fa006d390 legacy=0x7f8fa00b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:42.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.836+0000 7f8fa5000640 1 -- 172.21.15.102:0/2399882193 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa00cc290 con 0x7f8fa00cbab0 2024-03-23T03:48:42.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.836+0000 7f8f9fe00640 1 -- 172.21.15.102:0/2399882193 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834924921 0 0) 0x7f8fa00b16a0 con 0x7f8fa006d390 2024-03-23T03:48:42.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.837+0000 7f8fa5000640 1 --2- 172.21.15.102:0/2399882193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa00cbab0 0x7f8fa00b20f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c0097a0 tx=0x7f8f8c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d742280313a7cfb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:42.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.838+0000 7f8f9fe00640 1 -- 172.21.15.102:0/2399882193 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c02e070 con 0x7f8fa00cbab0 2024-03-23T03:48:42.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.838+0000 7f8f9fe00640 1 -- 172.21.15.102:0/2399882193 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8c0315a0 con 0x7f8fa00cbab0 2024-03-23T03:48:42.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.838+0000 7f8f9fe00640 1 -- 172.21.15.102:0/2399882193 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c031930 con 0x7f8fa00cbab0 2024-03-23T03:48:42.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.838+0000 7f8fa8400640 1 -- 172.21.15.102:0/2399882193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa00cbab0 msgr2=0x7f8fa00b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:42.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.838+0000 7f8fa8400640 1 --2- 172.21.15.102:0/2399882193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa00cbab0 0x7f8fa00b20f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c0097a0 tx=0x7f8f8c02b840 comp rx=0 tx=0).stop 2024-03-23T03:48:42.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.838+0000 7f8fa8400640 1 -- 172.21.15.102:0/2399882193 shutdown_connections 2024-03-23T03:48:42.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.838+0000 7f8fa8400640 1 --2- 172.21.15.102:0/2399882193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa00cbab0 0x7f8fa00b20f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.838+0000 7f8fa8400640 1 --2- 172.21.15.102:0/2399882193 >> v2:172.21.15.173:3300/0 conn(0x7f8fa00adbc0 0x7f8fa00b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.838+0000 7f8fa8400640 1 -- 172.21.15.102:0/2399882193 >> 172.21.15.102:0/2399882193 conn(0x7f8fa006a3d0 msgr2=0x7f8fa00b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:42.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.838+0000 7f8fa8400640 1 -- 172.21.15.102:0/2399882193 shutdown_connections 2024-03-23T03:48:42.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.839+0000 7f8fa8400640 1 -- 172.21.15.102:0/2399882193 wait complete. 2024-03-23T03:48:42.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.840+0000 7f8fa8400640 1 Processor -- start 2024-03-23T03:48:42.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.840+0000 7f8fa8400640 1 -- start start 2024-03-23T03:48:42.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.841+0000 7f8fa8400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fa006d390 0x7f8fa01f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:42.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.841+0000 7f8fa8400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8fa00adbc0 0x7f8fa00af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:42.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.841+0000 7f8fa8400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa00cbab0 0x7f8fa01f1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:42.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.841+0000 7f8fa8400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa013e180 con 0x7f8fa00cbab0 2024-03-23T03:48:42.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.841+0000 7f8fa8400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa013e460 con 0x7f8fa006d390 2024-03-23T03:48:42.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.841+0000 7f8fa8400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa013e740 con 0x7f8fa00adbc0 2024-03-23T03:48:42.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.842+0000 7f8fa6400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa00cbab0 0x7f8fa01f1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:42.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.842+0000 7f8fa6400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa00cbab0 0x7f8fa01f1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:55430/0 (socket says 172.21.15.102:55430) 2024-03-23T03:48:42.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.842+0000 7f8fa6400640 1 -- 172.21.15.102:0/2969219692 learned_addr learned my addr 172.21.15.102:0/2969219692 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:42.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.842+0000 7f8fa5a00640 1 --2- 172.21.15.102:0/2969219692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fa006d390 0x7f8fa01f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:42.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.842+0000 7f8fa6400640 1 -- 172.21.15.102:0/2969219692 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8fa00adbc0 msgr2=0x7f8fa00af1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:42.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.842+0000 7f8fa6400640 1 --2- 172.21.15.102:0/2969219692 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8fa00adbc0 0x7f8fa00af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.842+0000 7f8fa6400640 1 -- 172.21.15.102:0/2969219692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fa006d390 msgr2=0x7f8fa01f2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:42.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.842+0000 7f8fa6400640 1 --2- 172.21.15.102:0/2969219692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fa006d390 0x7f8fa01f2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:42.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.842+0000 7f8fa6400640 1 -- 172.21.15.102:0/2969219692 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8c009400 con 0x7f8fa00cbab0 2024-03-23T03:48:42.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.842+0000 7f8fa5a00640 1 --2- 172.21.15.102:0/2969219692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fa006d390 0x7f8fa01f2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:42.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.843+0000 7f8fa6400640 1 --2- 172.21.15.102:0/2969219692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa00cbab0 0x7f8fa01f1800 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800c780 tx=0x7f8f9800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:42.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.844+0000 7f8f9d600640 1 -- 172.21.15.102:0/2969219692 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9800ce90 con 0x7f8fa00cbab0 2024-03-23T03:48:42.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.844+0000 7f8fa8400640 1 -- 172.21.15.102:0/2969219692 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa013ea20 con 0x7f8fa00cbab0 2024-03-23T03:48:42.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.844+0000 7f8f9d600640 1 -- 172.21.15.102:0/2969219692 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f9800a640 con 0x7f8fa00cbab0 2024-03-23T03:48:42.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.844+0000 7f8f9d600640 1 -- 172.21.15.102:0/2969219692 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9800a9d0 con 0x7f8fa00cbab0 2024-03-23T03:48:42.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.844+0000 7f8fa8400640 1 -- 172.21.15.102:0/2969219692 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa013de00 con 0x7f8fa00cbab0 2024-03-23T03:48:42.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.845+0000 7f8fa8400640 1 -- 172.21.15.102:0/2969219692 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa013ef40 con 0x7f8fa00cbab0 2024-03-23T03:48:42.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.846+0000 7f8f9d600640 1 -- 172.21.15.102:0/2969219692 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8f9800ab30 con 0x7f8fa00cbab0 2024-03-23T03:48:42.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.847+0000 7f8f9d600640 1 --2- 172.21.15.102:0/2969219692 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8f8006ac00 0x7f8f8006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:42.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.847+0000 7f8f9d600640 1 -- 172.21.15.102:0/2969219692 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f8f98088cc0 con 0x7f8fa00cbab0 2024-03-23T03:48:42.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.847+0000 7f8fa5a00640 1 --2- 172.21.15.102:0/2969219692 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8f8006ac00 0x7f8f8006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:42.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.847+0000 7f8fa5a00640 1 --2- 172.21.15.102:0/2969219692 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8f8006ac00 0x7f8f8006d0c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8f94006fd0 tx=0x7f8f94008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:42.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:42.853+0000 7f8f9d600640 1 -- 172.21.15.102:0/2969219692 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8fa013ef40 con 0x7f8fa00cbab0 2024-03-23T03:48:43.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.084+0000 7f8fa8400640 1 -- 172.21.15.102:0/2969219692 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fa013ef40 con 0x7f8fa00cbab0 2024-03-23T03:48:43.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.085+0000 7f8f9d600640 1 -- 172.21.15.102:0/2969219692 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f8fa013ef40 con 0x7f8fa00cbab0 2024-03-23T03:48:43.086 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:43.086 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6834/1537100581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1537100581},{"type":"v1","addr":"172.21.15.173:6834","nonce":1537100581}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4655,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3206638952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":3206638952},{"type":"v1","addr":"172.21.15.173:6835","nonce":3206638952}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4852,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4005240382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4005240382},{"type":"v1","addr":"172.21.15.102:6837","nonce":4005240382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:48:23.418926+0000","modified":"2024-03-23T03:48:26.657662+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5262},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5262":{"gid":5262,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.102:6836/1373501465","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":1373501465},{"type":"v1","addr":"172.21.15.102:6836","nonce":1373501465}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-03-23T03:48:43.086 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 16 2024-03-23T03:48:43.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.088+0000 7f8fa8400640 1 -- 172.21.15.102:0/2969219692 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8f8006ac00 msgr2=0x7f8f8006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:43.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.088+0000 7f8fa8400640 1 --2- 172.21.15.102:0/2969219692 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8f8006ac00 0x7f8f8006d0c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f8f94006fd0 tx=0x7f8f94008040 comp rx=0 tx=0).stop 2024-03-23T03:48:43.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.088+0000 7f8fa8400640 1 -- 172.21.15.102:0/2969219692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa00cbab0 msgr2=0x7f8fa01f1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:43.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.088+0000 7f8fa8400640 1 --2- 172.21.15.102:0/2969219692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa00cbab0 0x7f8fa01f1800 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800c780 tx=0x7f8f9800cc40 comp rx=0 tx=0).stop 2024-03-23T03:48:43.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.088+0000 7f8fa8400640 1 -- 172.21.15.102:0/2969219692 shutdown_connections 2024-03-23T03:48:43.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.088+0000 7f8fa8400640 1 --2- 172.21.15.102:0/2969219692 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8f8006ac00 0x7f8f8006d0c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.088+0000 7f8fa8400640 1 --2- 172.21.15.102:0/2969219692 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa00cbab0 0x7f8fa01f1800 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.088+0000 7f8fa8400640 1 --2- 172.21.15.102:0/2969219692 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8fa00adbc0 0x7f8fa00af1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.088+0000 7f8fa8400640 1 --2- 172.21.15.102:0/2969219692 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fa006d390 0x7f8fa01f2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.088+0000 7f8fa8400640 1 -- 172.21.15.102:0/2969219692 >> 172.21.15.102:0/2969219692 conn(0x7f8fa006a3d0 msgr2=0x7f8fa00b2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:43.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.088+0000 7f8fa8400640 1 -- 172.21.15.102:0/2969219692 shutdown_connections 2024-03-23T03:48:43.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.088+0000 7f8fa8400640 1 -- 172.21.15.102:0/2969219692 wait complete. 2024-03-23T03:48:43.107 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:48:43.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.295+0000 7f3450400640 1 Processor -- start 2024-03-23T03:48:43.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.295+0000 7f3450400640 1 -- start start 2024-03-23T03:48:43.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.297+0000 7f3450400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f344815b560 0x7f344815b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:43.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.297+0000 7f3450400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f34480b39b0 0x7f34480b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:43.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.297+0000 7f3450400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f344806c6f0 con 0x7f344815e500 2024-03-23T03:48:43.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.297+0000 7f3450400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34480b4370 con 0x7f34480b39b0 2024-03-23T03:48:43.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.297+0000 7f3450400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344806c9d0 con 0x7f344815b560 2024-03-23T03:48:43.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.297+0000 7f344e400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f344815e500 0x7f344815e990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35888/0 (socket says 172.21.15.102:35888) 2024-03-23T03:48:43.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.297+0000 7f344e400640 1 -- 172.21.15.102:0/1805110549 learned_addr learned my addr 172.21.15.102:0/1805110549 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:43.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.297+0000 7f3447e00640 1 -- 172.21.15.102:0/1805110549 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897198722 0 0) 0x7f344806c6f0 con 0x7f344815e500 2024-03-23T03:48:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.297+0000 7f3447e00640 1 -- 172.21.15.102:0/1805110549 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3430003520 con 0x7f344815e500 2024-03-23T03:48:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.297+0000 7f344da00640 1 --2- 172.21.15.102:0/1805110549 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f344815b560 0x7f344815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.297+0000 7f344d000640 1 --2- 172.21.15.102:0/1805110549 >> v2:172.21.15.173:3300/0 conn(0x7f34480b39b0 0x7f34480b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f3447e00640 1 -- 172.21.15.102:0/1805110549 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1278737435 0 0) 0x7f3430003520 con 0x7f344815e500 2024-03-23T03:48:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f3447e00640 1 -- 172.21.15.102:0/1805110549 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f344815b560 msgr2=0x7f344815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f3447e00640 1 --2- 172.21.15.102:0/1805110549 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f344815b560 0x7f344815b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f3447e00640 1 -- 172.21.15.102:0/1805110549 >> v2:172.21.15.173:3300/0 conn(0x7f34480b39b0 msgr2=0x7f34480b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f3447e00640 1 --2- 172.21.15.102:0/1805110549 >> v2:172.21.15.173:3300/0 conn(0x7f34480b39b0 0x7f34480b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f3447e00640 1 -- 172.21.15.102:0/1805110549 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344806ccb0 con 0x7f344815e500 2024-03-23T03:48:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f3447e00640 1 -- 172.21.15.102:0/1805110549 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f34400019f0 con 0x7f344815e500 2024-03-23T03:48:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f3447e00640 1 -- 172.21.15.102:0/1805110549 >> v1:172.21.15.102:6789/0 conn(0x7f344815e500 legacy=0x7f344815e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f3447e00640 1 --2- 172.21.15.102:0/1805110549 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f344815b560 0x7f3430003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f3447e00640 1 --2- 172.21.15.102:0/1805110549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3430003ff0 0x7f34300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:43.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f3447e00640 1 --2- 172.21.15.102:0/1805110549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3430004d00 0x7f3430005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:43.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f3447e00640 1 -- 172.21.15.102:0/1805110549 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34480b4370 con 0x7f3430003ff0 2024-03-23T03:48:43.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f3447e00640 1 -- 172.21.15.102:0/1805110549 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344806c9d0 con 0x7f3430004d00 2024-03-23T03:48:43.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f3447e00640 1 -- 172.21.15.102:0/1805110549 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343000bae0 con 0x7f344815b560 2024-03-23T03:48:43.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f344d000640 1 --2- 172.21.15.102:0/1805110549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3430003ff0 0x7f34300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:43.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f344d000640 1 -- 172.21.15.102:0/1805110549 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f344815b560 msgr2=0x7f3430003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:43.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f344d000640 1 --2- 172.21.15.102:0/1805110549 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f344815b560 0x7f3430003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f344d000640 1 -- 172.21.15.102:0/1805110549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3430004d00 msgr2=0x7f3430005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:43.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f344d000640 1 --2- 172.21.15.102:0/1805110549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3430004d00 0x7f3430005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.298+0000 7f344d000640 1 -- 172.21.15.102:0/1805110549 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3430003520 con 0x7f3430003ff0 2024-03-23T03:48:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.300+0000 7f344d000640 1 --2- 172.21.15.102:0/1805110549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3430003ff0 0x7f34300047c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f343800ba30 tx=0x7f34300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bfb89362deda025 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.300+0000 7f3447e00640 1 -- 172.21.15.102:0/1805110549 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438032680 con 0x7f3430003ff0 2024-03-23T03:48:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.300+0000 7f3447e00640 1 -- 172.21.15.102:0/1805110549 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34380327e0 con 0x7f3430003ff0 2024-03-23T03:48:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.300+0000 7f3447e00640 1 -- 172.21.15.102:0/1805110549 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438032b70 con 0x7f3430003ff0 2024-03-23T03:48:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.300+0000 7f3450400640 1 -- 172.21.15.102:0/1805110549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3430003ff0 msgr2=0x7f34300047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.300+0000 7f3450400640 1 --2- 172.21.15.102:0/1805110549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3430003ff0 0x7f34300047c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f343800ba30 tx=0x7f34300057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.300+0000 7f344e400640 1 -- 172.21.15.102:0/1805110549 reap_dead start 2024-03-23T03:48:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.300+0000 7f3450400640 1 -- 172.21.15.102:0/1805110549 shutdown_connections 2024-03-23T03:48:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.300+0000 7f3450400640 1 -- 172.21.15.102:0/1805110549 >> 172.21.15.102:0/1805110549 conn(0x7f344806a3d0 msgr2=0x7f34480b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.300+0000 7f3450400640 1 -- 172.21.15.102:0/1805110549 shutdown_connections 2024-03-23T03:48:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.300+0000 7f3450400640 1 -- 172.21.15.102:0/1805110549 wait complete. 2024-03-23T03:48:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.302+0000 7f3450400640 1 Processor -- start 2024-03-23T03:48:43.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.302+0000 7f3450400640 1 -- start start 2024-03-23T03:48:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.303+0000 7f3450400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3430003ff0 0x7f34481f1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.303+0000 7f3450400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34481525d0 0x7f344813f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.303+0000 7f3450400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3448152a50 0x7f344813f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.303+0000 7f3450400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3448141500 con 0x7f3448152a50 2024-03-23T03:48:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.303+0000 7f3450400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34481417a0 con 0x7f34481525d0 2024-03-23T03:48:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.303+0000 7f3450400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3448141a80 con 0x7f3430003ff0 2024-03-23T03:48:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.303+0000 7f344e400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3448152a50 0x7f344813f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:43.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.304+0000 7f344e400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3448152a50 0x7f344813f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46620/0 (socket says 172.21.15.102:46620) 2024-03-23T03:48:43.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.304+0000 7f344e400640 1 -- 172.21.15.102:0/3618345583 learned_addr learned my addr 172.21.15.102:0/3618345583 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:43.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.304+0000 7f344e400640 1 -- 172.21.15.102:0/3618345583 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3430003ff0 msgr2=0x7f34481f1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:43.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.304+0000 7f344e400640 1 --2- 172.21.15.102:0/3618345583 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3430003ff0 0x7f34481f1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.304+0000 7f344e400640 1 -- 172.21.15.102:0/3618345583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34481525d0 msgr2=0x7f344813f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:43.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.304+0000 7f344e400640 1 --2- 172.21.15.102:0/3618345583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34481525d0 0x7f344813f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.304+0000 7f344e400640 1 -- 172.21.15.102:0/3618345583 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3448141d60 con 0x7f3448152a50 2024-03-23T03:48:43.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.304+0000 7f344e400640 1 --2- 172.21.15.102:0/3618345583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3448152a50 0x7f344813f8f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3448070100 tx=0x7f344000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:43.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.304+0000 7f3445600640 1 -- 172.21.15.102:0/3618345583 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344000a610 con 0x7f3448152a50 2024-03-23T03:48:43.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.304+0000 7f3445600640 1 -- 172.21.15.102:0/3618345583 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f344000a790 con 0x7f3448152a50 2024-03-23T03:48:43.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.304+0000 7f3445600640 1 -- 172.21.15.102:0/3618345583 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f344000ab40 con 0x7f3448152a50 2024-03-23T03:48:43.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.304+0000 7f3450400640 1 -- 172.21.15.102:0/3618345583 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3448142040 con 0x7f3448152a50 2024-03-23T03:48:43.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.305+0000 7f3450400640 1 -- 172.21.15.102:0/3618345583 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3448142560 con 0x7f3448152a50 2024-03-23T03:48:43.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.306+0000 7f3450400640 1 -- 172.21.15.102:0/3618345583 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344806ac20 con 0x7f3448152a50 2024-03-23T03:48:43.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.306+0000 7f3445600640 1 -- 172.21.15.102:0/3618345583 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f344000aca0 con 0x7f3448152a50 2024-03-23T03:48:43.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.307+0000 7f3445600640 1 --2- 172.21.15.102:0/3618345583 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f342c06ac30 0x7f342c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:43.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.307+0000 7f3445600640 1 -- 172.21.15.102:0/3618345583 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f3440088e10 con 0x7f3448152a50 2024-03-23T03:48:43.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.307+0000 7f344da00640 1 --2- 172.21.15.102:0/3618345583 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f342c06ac30 0x7f342c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:43.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.308+0000 7f344da00640 1 --2- 172.21.15.102:0/3618345583 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f342c06ac30 0x7f342c06d0f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f343c00b730 tx=0x7f343c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:43.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.312+0000 7f3445600640 1 -- 172.21.15.102:0/3618345583 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f344008c030 con 0x7f3448152a50 2024-03-23T03:48:43.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.491+0000 7f3450400640 1 -- 172.21.15.102:0/3618345583 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3448073390 con 0x7f3448152a50 2024-03-23T03:48:43.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.494+0000 7f3445600640 1 -- 172.21.15.102:0/3618345583 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+16403 (secure 0 0 0) 0x7f344001a070 con 0x7f3448152a50 2024-03-23T03:48:43.495 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:43.495 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":43,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:42.665990+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:48:21.550281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-03-23T03:48:22.428475+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.1d","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:48:43.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.496+0000 7f3450400640 1 -- 172.21.15.102:0/3618345583 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f342c06ac30 msgr2=0x7f342c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:43.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.496+0000 7f3450400640 1 --2- 172.21.15.102:0/3618345583 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f342c06ac30 0x7f342c06d0f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f343c00b730 tx=0x7f343c008040 comp rx=0 tx=0).stop 2024-03-23T03:48:43.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.496+0000 7f3450400640 1 -- 172.21.15.102:0/3618345583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3448152a50 msgr2=0x7f344813f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:43.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.496+0000 7f3450400640 1 --2- 172.21.15.102:0/3618345583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3448152a50 0x7f344813f8f0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3448070100 tx=0x7f344000cf00 comp rx=0 tx=0).stop 2024-03-23T03:48:43.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.496+0000 7f3450400640 1 -- 172.21.15.102:0/3618345583 shutdown_connections 2024-03-23T03:48:43.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.496+0000 7f3450400640 1 --2- 172.21.15.102:0/3618345583 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f342c06ac30 0x7f342c06d0f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.496+0000 7f3450400640 1 --2- 172.21.15.102:0/3618345583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3448152a50 0x7f344813f8f0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.496+0000 7f3450400640 1 --2- 172.21.15.102:0/3618345583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34481525d0 0x7f344813f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.497+0000 7f3450400640 1 --2- 172.21.15.102:0/3618345583 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3430003ff0 0x7f34481f1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.497+0000 7f3450400640 1 -- 172.21.15.102:0/3618345583 >> 172.21.15.102:0/3618345583 conn(0x7f344806a3d0 msgr2=0x7f3448142930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:43.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.497+0000 7f3450400640 1 -- 172.21.15.102:0/3618345583 shutdown_connections 2024-03-23T03:48:43.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.497+0000 7f3450400640 1 -- 172.21.15.102:0/3618345583 wait complete. 2024-03-23T03:48:43.514 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T03:48:43.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.705+0000 7fe5c4200640 1 Processor -- start 2024-03-23T03:48:43.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.705+0000 7fe5c4200640 1 -- start start 2024-03-23T03:48:43.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.706+0000 7fe5c4200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5bc15cd90 0x7fe5bc0b8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:43.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.706+0000 7fe5c4200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5bc06b840 con 0x7fe5bc0cda90 2024-03-23T03:48:43.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.707+0000 7fe5c4200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5bc06bb20 con 0x7fe5bc0cd6b0 2024-03-23T03:48:43.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.707+0000 7fe5c4200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc06c0e0 con 0x7fe5bc15cd90 2024-03-23T03:48:43.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.707+0000 7fe5c2200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe5bc0cda90 0x7fe5bc0d1370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35896/0 (socket says 172.21.15.102:35896) 2024-03-23T03:48:43.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.707+0000 7fe5c2200640 1 -- 172.21.15.102:0/3382392759 learned_addr learned my addr 172.21.15.102:0/3382392759 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:43.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.707+0000 7fe5bbe00640 1 -- 172.21.15.102:0/3382392759 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166397141 0 0) 0x7fe5bc06b840 con 0x7fe5bc0cda90 2024-03-23T03:48:43.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.707+0000 7fe5bbe00640 1 -- 172.21.15.102:0/3382392759 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a4003520 con 0x7fe5bc0cda90 2024-03-23T03:48:43.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.707+0000 7fe5c0e00640 1 --2- 172.21.15.102:0/3382392759 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5bc15cd90 0x7fe5bc0b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:43.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5bbe00640 1 -- 172.21.15.102:0/3382392759 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 454540960 0 0) 0x7fe5a4003520 con 0x7fe5bc0cda90 2024-03-23T03:48:43.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5bbe00640 1 -- 172.21.15.102:0/3382392759 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5bc15cd90 msgr2=0x7fe5bc0b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:43.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5bbe00640 1 --2- 172.21.15.102:0/3382392759 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5bc15cd90 0x7fe5bc0b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5bbe00640 1 -- 172.21.15.102:0/3382392759 >> v1:172.21.15.173:6789/0 conn(0x7fe5bc0cd6b0 legacy=0x7fe5bc06b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:43.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5bbe00640 1 -- 172.21.15.102:0/3382392759 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5bc06be00 con 0x7fe5bc0cda90 2024-03-23T03:48:43.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5bbe00640 1 -- 172.21.15.102:0/3382392759 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fe5b40019f0 con 0x7fe5bc0cda90 2024-03-23T03:48:43.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5bbe00640 1 -- 172.21.15.102:0/3382392759 >> v1:172.21.15.102:6789/0 conn(0x7fe5bc0cda90 legacy=0x7fe5bc0d1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:43.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5bbe00640 1 --2- 172.21.15.102:0/3382392759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5a40039a0 0x7fe5a4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:43.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5bbe00640 1 --2- 172.21.15.102:0/3382392759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5a40047c0 0x7fe5a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:43.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5bbe00640 1 --2- 172.21.15.102:0/3382392759 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5bc15cd90 0x7fe5a40051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:43.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5bbe00640 1 -- 172.21.15.102:0/3382392759 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc06c0e0 con 0x7fe5a40039a0 2024-03-23T03:48:43.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5bbe00640 1 -- 172.21.15.102:0/3382392759 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a400bab0 con 0x7fe5a40047c0 2024-03-23T03:48:43.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5bbe00640 1 -- 172.21.15.102:0/3382392759 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a400bc70 con 0x7fe5bc15cd90 2024-03-23T03:48:43.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5c1800640 1 --2- 172.21.15.102:0/3382392759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5a40039a0 0x7fe5a4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:43.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5c1800640 1 -- 172.21.15.102:0/3382392759 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5bc15cd90 msgr2=0x7fe5a40051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:43.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5c1800640 1 --2- 172.21.15.102:0/3382392759 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5bc15cd90 0x7fe5a40051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5c1800640 1 -- 172.21.15.102:0/3382392759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5a40047c0 msgr2=0x7fe5a4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:43.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5c1800640 1 --2- 172.21.15.102:0/3382392759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5a40047c0 0x7fe5a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.708+0000 7fe5c1800640 1 -- 172.21.15.102:0/3382392759 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a4003520 con 0x7fe5a40039a0 2024-03-23T03:48:43.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.709+0000 7fe5c1800640 1 --2- 172.21.15.102:0/3382392759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5a40039a0 0x7fe5a4003dc0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0001750 tx=0x7fe5a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44566609f731e996 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:43.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.710+0000 7fe5bbe00640 1 -- 172.21.15.102:0/3382392759 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b00052d0 con 0x7fe5a40039a0 2024-03-23T03:48:43.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.710+0000 7fe5bbe00640 1 -- 172.21.15.102:0/3382392759 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b002e050 con 0x7fe5a40039a0 2024-03-23T03:48:43.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.710+0000 7fe5bbe00640 1 -- 172.21.15.102:0/3382392759 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b00035f0 con 0x7fe5a40039a0 2024-03-23T03:48:43.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.710+0000 7fe5c4200640 1 -- 172.21.15.102:0/3382392759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5a40039a0 msgr2=0x7fe5a4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:43.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.710+0000 7fe5c4200640 1 --2- 172.21.15.102:0/3382392759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5a40039a0 0x7fe5a4003dc0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0001750 tx=0x7fe5a40057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.710+0000 7fe5c2200640 1 -- 172.21.15.102:0/3382392759 reap_dead start 2024-03-23T03:48:43.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.710+0000 7fe5c4200640 1 -- 172.21.15.102:0/3382392759 shutdown_connections 2024-03-23T03:48:43.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.710+0000 7fe5c4200640 1 -- 172.21.15.102:0/3382392759 >> 172.21.15.102:0/3382392759 conn(0x7fe5bc0cf780 msgr2=0x7fe5bc0cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:43.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.710+0000 7fe5c4200640 1 -- 172.21.15.102:0/3382392759 shutdown_connections 2024-03-23T03:48:43.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.710+0000 7fe5c4200640 1 -- 172.21.15.102:0/3382392759 wait complete. 2024-03-23T03:48:43.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.712+0000 7fe5c4200640 1 Processor -- start 2024-03-23T03:48:43.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.712+0000 7fe5c4200640 1 -- start start 2024-03-23T03:48:43.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.713+0000 7fe5c4200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5a40039a0 0x7fe5bc156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:43.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.713+0000 7fe5c4200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5bc152450 0x7fe5bc0cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:43.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.713+0000 7fe5c4200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5bc1528d0 0x7fe5bc0cb070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:43.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.713+0000 7fe5c4200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc153090 con 0x7fe5bc1528d0 2024-03-23T03:48:43.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.713+0000 7fe5c4200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc157830 con 0x7fe5bc152450 2024-03-23T03:48:43.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.713+0000 7fe5c4200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5bc157b10 con 0x7fe5a40039a0 2024-03-23T03:48:43.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.713+0000 7fe5c2200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5bc1528d0 0x7fe5bc0cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:43.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.713+0000 7fe5c2200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5bc1528d0 0x7fe5bc0cb070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46634/0 (socket says 172.21.15.102:46634) 2024-03-23T03:48:43.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.713+0000 7fe5c2200640 1 -- 172.21.15.102:0/831244778 learned_addr learned my addr 172.21.15.102:0/831244778 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:43.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.714+0000 7fe5c2200640 1 -- 172.21.15.102:0/831244778 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5a40039a0 msgr2=0x7fe5bc156ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:43.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.714+0000 7fe5c2200640 1 --2- 172.21.15.102:0/831244778 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5a40039a0 0x7fe5bc156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.714+0000 7fe5c2200640 1 -- 172.21.15.102:0/831244778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5bc152450 msgr2=0x7fe5bc0cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:43.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.714+0000 7fe5c2200640 1 --2- 172.21.15.102:0/831244778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5bc152450 0x7fe5bc0cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.714+0000 7fe5c2200640 1 -- 172.21.15.102:0/831244778 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5bc157df0 con 0x7fe5bc1528d0 2024-03-23T03:48:43.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.714+0000 7fe5c2200640 1 --2- 172.21.15.102:0/831244778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5bc1528d0 0x7fe5bc0cb070 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0c9960 tx=0x7fe5b400cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:43.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.714+0000 7fe5b9600640 1 -- 172.21.15.102:0/831244778 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b400a610 con 0x7fe5bc1528d0 2024-03-23T03:48:43.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.714+0000 7fe5b9600640 1 -- 172.21.15.102:0/831244778 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5b400a790 con 0x7fe5bc1528d0 2024-03-23T03:48:43.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.714+0000 7fe5b9600640 1 -- 172.21.15.102:0/831244778 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5b400ab40 con 0x7fe5bc1528d0 2024-03-23T03:48:43.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.714+0000 7fe5c4200640 1 -- 172.21.15.102:0/831244778 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5bc1e7600 con 0x7fe5bc1528d0 2024-03-23T03:48:43.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.714+0000 7fe5c4200640 1 -- 172.21.15.102:0/831244778 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5bc156ff0 con 0x7fe5bc1528d0 2024-03-23T03:48:43.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.715+0000 7fe5c4200640 1 -- 172.21.15.102:0/831244778 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5bc1e7b20 con 0x7fe5bc1528d0 2024-03-23T03:48:43.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.716+0000 7fe5b9600640 1 -- 172.21.15.102:0/831244778 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe5b400aca0 con 0x7fe5bc1528d0 2024-03-23T03:48:43.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.717+0000 7fe5b9600640 1 --2- 172.21.15.102:0/831244778 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe59c06ac00 0x7fe59c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:43.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.717+0000 7fe5b9600640 1 -- 172.21.15.102:0/831244778 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fe5b4088d70 con 0x7fe5bc1528d0 2024-03-23T03:48:43.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.717+0000 7fe5c1800640 1 --2- 172.21.15.102:0/831244778 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe59c06ac00 0x7fe59c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:43.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.718+0000 7fe5c1800640 1 --2- 172.21.15.102:0/831244778 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe59c06ac00 0x7fe59c06d0c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0033000 tx=0x7fe5b002f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:43.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.721+0000 7fe5b9600640 1 -- 172.21.15.102:0/831244778 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe5bc1e7b20 con 0x7fe5bc1528d0 2024-03-23T03:48:43.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.952+0000 7fe5c4200640 1 -- 172.21.15.102:0/831244778 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe5bc1e7b20 con 0x7fe5bc1528d0 2024-03-23T03:48:43.953 INFO:tasks.ceph.mon.a.smithi102.stderr:2024-03-23T03:48:43.953+0000 7f5aab200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T03:48:43.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.971+0000 7fe5b9600640 1 -- 172.21.15.102:0/831244778 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe5bc1e7b20 con 0x7fe5bc1528d0 2024-03-23T03:48:43.972 INFO:teuthology.orchestra.run.smithi102.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T03:48:43.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.974+0000 7fe5c4200640 1 -- 172.21.15.102:0/831244778 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe59c06ac00 msgr2=0x7fe59c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:43.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.974+0000 7fe5c4200640 1 --2- 172.21.15.102:0/831244778 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe59c06ac00 0x7fe59c06d0c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe5b0033000 tx=0x7fe5b002f000 comp rx=0 tx=0).stop 2024-03-23T03:48:43.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.974+0000 7fe5c4200640 1 -- 172.21.15.102:0/831244778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5bc1528d0 msgr2=0x7fe5bc0cb070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:43.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.974+0000 7fe5c4200640 1 --2- 172.21.15.102:0/831244778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5bc1528d0 0x7fe5bc0cb070 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc0c9960 tx=0x7fe5b400cf00 comp rx=0 tx=0).stop 2024-03-23T03:48:43.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.975+0000 7fe5c4200640 1 -- 172.21.15.102:0/831244778 shutdown_connections 2024-03-23T03:48:43.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.975+0000 7fe5c4200640 1 --2- 172.21.15.102:0/831244778 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe59c06ac00 0x7fe59c06d0c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.975+0000 7fe5c4200640 1 --2- 172.21.15.102:0/831244778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5bc1528d0 0x7fe5bc0cb070 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.975+0000 7fe5c4200640 1 --2- 172.21.15.102:0/831244778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5bc152450 0x7fe5bc0cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.975+0000 7fe5c4200640 1 --2- 172.21.15.102:0/831244778 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe5a40039a0 0x7fe5bc156ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:43.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.975+0000 7fe5c4200640 1 -- 172.21.15.102:0/831244778 >> 172.21.15.102:0/831244778 conn(0x7fe5bc0cf780 msgr2=0x7fe5bc0b28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:43.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.975+0000 7fe5c4200640 1 -- 172.21.15.102:0/831244778 shutdown_connections 2024-03-23T03:48:43.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:43.975+0000 7fe5c4200640 1 -- 172.21.15.102:0/831244778 wait complete. 2024-03-23T03:48:43.978 INFO:tasks.ceph.mds.a.smithi102.stderr: -3108> 2024-03-23T03:48:25.910+0000 7fc473600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T03:48:44.000 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T03:48:44.039 INFO:tasks.ceph.mds.a.smithi102.stdout:starting mds.a at 2024-03-23T03:48:44.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.170+0000 7fa36ae00640 1 Processor -- start 2024-03-23T03:48:44.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.170+0000 7fa36ae00640 1 -- start start 2024-03-23T03:48:44.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.172+0000 7fa36ae00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa3640cd740 0x7fa3640b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:44.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.172+0000 7fa36ae00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3640d20c0 con 0x7fa3640d0d40 2024-03-23T03:48:44.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.172+0000 7fa36ae00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3640d14b0 con 0x7fa3640b5aa0 2024-03-23T03:48:44.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.172+0000 7fa36ae00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3640d1790 con 0x7fa3640cd740 2024-03-23T03:48:44.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.172+0000 7fa363400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa3640d0d40 0x7fa3640cdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35906/0 (socket says 172.21.15.102:35906) 2024-03-23T03:48:44.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.172+0000 7fa363400640 1 -- 172.21.15.102:0/3134003932 learned_addr learned my addr 172.21.15.102:0/3134003932 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:44.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.172+0000 7fa363e00640 1 --2- 172.21.15.102:0/3134003932 >> v2:172.21.15.173:3300/0 conn(0x7fa3640cd740 0x7fa3640b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:44.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.172+0000 7fa362a00640 1 -- 172.21.15.102:0/3134003932 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774405536 0 0) 0x7fa3640d20c0 con 0x7fa3640d0d40 2024-03-23T03:48:44.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.172+0000 7fa362a00640 1 -- 172.21.15.102:0/3134003932 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa348003520 con 0x7fa3640d0d40 2024-03-23T03:48:44.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.173+0000 7fa362a00640 1 -- 172.21.15.102:0/3134003932 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3730264050 0 0) 0x7fa348003520 con 0x7fa3640d0d40 2024-03-23T03:48:44.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.173+0000 7fa362a00640 1 -- 172.21.15.102:0/3134003932 >> v2:172.21.15.173:3300/0 conn(0x7fa3640cd740 msgr2=0x7fa3640b84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:44.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.173+0000 7fa362a00640 1 --2- 172.21.15.102:0/3134003932 >> v2:172.21.15.173:3300/0 conn(0x7fa3640cd740 0x7fa3640b84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:44.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.173+0000 7fa362a00640 1 -- 172.21.15.102:0/3134003932 >> v1:172.21.15.173:6789/0 conn(0x7fa3640b5aa0 legacy=0x7fa3640b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:44.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.173+0000 7fa362a00640 1 -- 172.21.15.102:0/3134003932 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3640d1a70 con 0x7fa3640d0d40 2024-03-23T03:48:44.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.173+0000 7fa362a00640 1 -- 172.21.15.102:0/3134003932 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fa34c001670 con 0x7fa3640d0d40 2024-03-23T03:48:44.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.173+0000 7fa362a00640 1 -- 172.21.15.102:0/3134003932 >> v1:172.21.15.102:6789/0 conn(0x7fa3640d0d40 legacy=0x7fa3640cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:44.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.173+0000 7fa362a00640 1 --2- 172.21.15.102:0/3134003932 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa348003970 0x7fa348003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:44.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.173+0000 7fa362a00640 1 --2- 172.21.15.102:0/3134003932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa3480047c0 0x7fa348004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:44.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.173+0000 7fa362a00640 1 --2- 172.21.15.102:0/3134003932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa348005180 0x7fa34800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:44.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.173+0000 7fa362a00640 1 -- 172.21.15.102:0/3134003932 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3640d1790 con 0x7fa348005180 2024-03-23T03:48:44.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.173+0000 7fa362a00640 1 -- 172.21.15.102:0/3134003932 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa34800c0d0 con 0x7fa3480047c0 2024-03-23T03:48:44.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.173+0000 7fa362a00640 1 -- 172.21.15.102:0/3134003932 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34800c290 con 0x7fa348003970 2024-03-23T03:48:44.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.173+0000 7fa363e00640 1 --2- 172.21.15.102:0/3134003932 >> v2:172.21.15.173:3300/0 conn(0x7fa3640cd740 0x7fa3640b84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:48:44.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.173+0000 7fa368e00640 1 --2- 172.21.15.102:0/3134003932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa348005180 0x7fa34800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:44.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.173+0000 7fa368e00640 1 -- 172.21.15.102:0/3134003932 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa348003970 msgr2=0x7fa348003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:44.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.173+0000 7fa368e00640 1 --2- 172.21.15.102:0/3134003932 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa348003970 0x7fa348003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:44.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.174+0000 7fa368e00640 1 -- 172.21.15.102:0/3134003932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa3480047c0 msgr2=0x7fa348004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:44.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.174+0000 7fa368e00640 1 --2- 172.21.15.102:0/3134003932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa3480047c0 0x7fa348004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:44.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.174+0000 7fa368e00640 1 -- 172.21.15.102:0/3134003932 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa348003520 con 0x7fa348005180 2024-03-23T03:48:44.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.174+0000 7fa368e00640 1 -- 172.21.15.102:0/3134003932 reap_dead start 2024-03-23T03:48:44.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.174+0000 7fa368e00640 1 --2- 172.21.15.102:0/3134003932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa348005180 0x7fa34800b980 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa35c001750 tx=0x7fa3480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a273e696c4825ed4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:44.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.175+0000 7fa362a00640 1 -- 172.21.15.102:0/3134003932 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35c0052d0 con 0x7fa348005180 2024-03-23T03:48:44.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.175+0000 7fa362a00640 1 -- 172.21.15.102:0/3134003932 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35c02e050 con 0x7fa348005180 2024-03-23T03:48:44.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.175+0000 7fa362a00640 1 -- 172.21.15.102:0/3134003932 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35c003830 con 0x7fa348005180 2024-03-23T03:48:44.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.175+0000 7fa36ae00640 1 -- 172.21.15.102:0/3134003932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa348005180 msgr2=0x7fa34800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:44.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.175+0000 7fa36ae00640 1 --2- 172.21.15.102:0/3134003932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa348005180 0x7fa34800b980 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa35c001750 tx=0x7fa3480057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:44.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.175+0000 7fa36ae00640 1 -- 172.21.15.102:0/3134003932 shutdown_connections 2024-03-23T03:48:44.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.175+0000 7fa36ae00640 1 --2- 172.21.15.102:0/3134003932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa348005180 0x7fa34800b980 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:44.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.175+0000 7fa36ae00640 1 -- 172.21.15.102:0/3134003932 >> 172.21.15.102:0/3134003932 conn(0x7fa3640cf5d0 msgr2=0x7fa3640cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:44.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.175+0000 7fa36ae00640 1 -- 172.21.15.102:0/3134003932 shutdown_connections 2024-03-23T03:48:44.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.175+0000 7fa36ae00640 1 -- 172.21.15.102:0/3134003932 wait complete. 2024-03-23T03:48:44.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.177+0000 7fa36ae00640 1 Processor -- start 2024-03-23T03:48:44.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.177+0000 7fa36ae00640 1 -- start start 2024-03-23T03:48:44.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.178+0000 7fa36ae00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa348005180 0x7fa364156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:44.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.178+0000 7fa36ae00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa36415b110 0x7fa36415b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:44.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.178+0000 7fa36ae00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa3641f1250 0x7fa3641f16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:44.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.178+0000 7fa36ae00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3641582d0 con 0x7fa348005180 2024-03-23T03:48:44.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.178+0000 7fa36ae00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3641576c0 con 0x7fa36415b110 2024-03-23T03:48:44.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.178+0000 7fa36ae00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3641579a0 con 0x7fa3641f1250 2024-03-23T03:48:44.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.178+0000 7fa363e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa348005180 0x7fa364156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:44.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.178+0000 7fa363e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa348005180 0x7fa364156e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46672/0 (socket says 172.21.15.102:46672) 2024-03-23T03:48:44.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.178+0000 7fa363e00640 1 -- 172.21.15.102:0/2621833787 learned_addr learned my addr 172.21.15.102:0/2621833787 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:44.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.179+0000 7fa363e00640 1 -- 172.21.15.102:0/2621833787 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa3641f1250 msgr2=0x7fa3641f16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:44.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.179+0000 7fa363e00640 1 --2- 172.21.15.102:0/2621833787 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa3641f1250 0x7fa3641f16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:44.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.179+0000 7fa363e00640 1 -- 172.21.15.102:0/2621833787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa36415b110 msgr2=0x7fa36415b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:44.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.179+0000 7fa363e00640 1 --2- 172.21.15.102:0/2621833787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa36415b110 0x7fa36415b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:44.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.179+0000 7fa363e00640 1 -- 172.21.15.102:0/2621833787 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa364157c80 con 0x7fa348005180 2024-03-23T03:48:44.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.179+0000 7fa363e00640 1 --2- 172.21.15.102:0/2621833787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa348005180 0x7fa364156e70 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa354007e70 tx=0x7fa35400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:44.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.179+0000 7fa35be00640 1 -- 172.21.15.102:0/2621833787 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa354003b90 con 0x7fa348005180 2024-03-23T03:48:44.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.179+0000 7fa35be00640 1 -- 172.21.15.102:0/2621833787 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa35400f040 con 0x7fa348005180 2024-03-23T03:48:44.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.179+0000 7fa35be00640 1 -- 172.21.15.102:0/2621833787 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa35400de80 con 0x7fa348005180 2024-03-23T03:48:44.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.179+0000 7fa36ae00640 1 -- 172.21.15.102:0/2621833787 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa364157f60 con 0x7fa348005180 2024-03-23T03:48:44.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.179+0000 7fa36ae00640 1 -- 172.21.15.102:0/2621833787 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3641f1c10 con 0x7fa348005180 2024-03-23T03:48:44.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.180+0000 7fa36ae00640 1 -- 172.21.15.102:0/2621833787 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa364158440 con 0x7fa348005180 2024-03-23T03:48:44.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.181+0000 7fa35be00640 1 -- 172.21.15.102:0/2621833787 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa354004030 con 0x7fa348005180 2024-03-23T03:48:44.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.182+0000 7fa35be00640 1 --2- 172.21.15.102:0/2621833787 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa34006ac30 0x7fa34006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:44.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.182+0000 7fa35be00640 1 -- 172.21.15.102:0/2621833787 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6144+0+0 (secure 0 0 0) 0x7fa354089920 con 0x7fa348005180 2024-03-23T03:48:44.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.183+0000 7fa363400640 1 --2- 172.21.15.102:0/2621833787 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa34006ac30 0x7fa34006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:44.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.183+0000 7fa363400640 1 --2- 172.21.15.102:0/2621833787 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa34006ac30 0x7fa34006d0f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa34c000f60 tx=0x7fa34c0015c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:44.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.187+0000 7fa35be00640 1 -- 172.21.15.102:0/2621833787 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa3540548d0 con 0x7fa348005180 2024-03-23T03:48:44.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.418+0000 7fa36ae00640 1 -- 172.21.15.102:0/2621833787 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa364073390 con 0x7fa348005180 2024-03-23T03:48:44.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.974+0000 7fa35be00640 1 -- 172.21.15.102:0/2621833787 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa354057ef0 con 0x7fa348005180 2024-03-23T03:48:44.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.978+0000 7fa36ae00640 1 -- 172.21.15.102:0/2621833787 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa34006ac30 msgr2=0x7fa34006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:44.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.978+0000 7fa36ae00640 1 --2- 172.21.15.102:0/2621833787 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa34006ac30 0x7fa34006d0f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fa34c000f60 tx=0x7fa34c0015c0 comp rx=0 tx=0).stop 2024-03-23T03:48:44.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.978+0000 7fa36ae00640 1 -- 172.21.15.102:0/2621833787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa348005180 msgr2=0x7fa364156e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:44.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.978+0000 7fa36ae00640 1 --2- 172.21.15.102:0/2621833787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa348005180 0x7fa364156e70 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa354007e70 tx=0x7fa35400d4a0 comp rx=0 tx=0).stop 2024-03-23T03:48:44.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.978+0000 7fa36ae00640 1 -- 172.21.15.102:0/2621833787 shutdown_connections 2024-03-23T03:48:44.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.978+0000 7fa36ae00640 1 --2- 172.21.15.102:0/2621833787 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa34006ac30 0x7fa34006d0f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:44.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.978+0000 7fa36ae00640 1 --2- 172.21.15.102:0/2621833787 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa3641f1250 0x7fa3641f16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:44.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.978+0000 7fa36ae00640 1 --2- 172.21.15.102:0/2621833787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa36415b110 0x7fa36415b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:44.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.978+0000 7fa36ae00640 1 --2- 172.21.15.102:0/2621833787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa348005180 0x7fa364156e70 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:44.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.978+0000 7fa36ae00640 1 -- 172.21.15.102:0/2621833787 >> 172.21.15.102:0/2621833787 conn(0x7fa3640cf5d0 msgr2=0x7fa3640b6090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:44.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.978+0000 7fa36ae00640 1 -- 172.21.15.102:0/2621833787 shutdown_connections 2024-03-23T03:48:44.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:44.978+0000 7fa36ae00640 1 -- 172.21.15.102:0/2621833787 wait complete. 2024-03-23T03:48:44.998 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T03:48:45.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.254+0000 7f426dc00640 1 Processor -- start 2024-03-23T03:48:45.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.255+0000 7f426dc00640 1 -- start start 2024-03-23T03:48:45.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.256+0000 7f426dc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f42680cd740 0x7f42680b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:45.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.256+0000 7f426dc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f42680b5aa0 0x7f42680b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:45.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.256+0000 7f426dc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42680d20c0 con 0x7f42680d0d40 2024-03-23T03:48:45.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.256+0000 7f426dc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42680d14b0 con 0x7f42680b5aa0 2024-03-23T03:48:45.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.256+0000 7f426dc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42680d1790 con 0x7f42680cd740 2024-03-23T03:48:45.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.257+0000 7f4266000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f42680d0d40 0x7f42680cdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35920/0 (socket says 172.21.15.102:35920) 2024-03-23T03:48:45.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.257+0000 7f4266000640 1 -- 172.21.15.102:0/4200599083 learned_addr learned my addr 172.21.15.102:0/4200599083 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:45.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.257+0000 7f4267400640 1 --2- 172.21.15.102:0/4200599083 >> v2:172.21.15.102:3300/0 conn(0x7f42680b5aa0 0x7f42680b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:45.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.257+0000 7f4266a00640 1 --2- 172.21.15.102:0/4200599083 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f42680cd740 0x7f42680b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:45.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.257+0000 7f4267400640 1 -- 172.21.15.102:0/4200599083 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f42680cd740 msgr2=0x7f42680b84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:45.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.257+0000 7f4267400640 1 --2- 172.21.15.102:0/4200599083 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f42680cd740 0x7f42680b84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:45.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.257+0000 7f4267400640 1 -- 172.21.15.102:0/4200599083 >> v1:172.21.15.102:6789/0 conn(0x7f42680d0d40 legacy=0x7f42680cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:45.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.257+0000 7f4267400640 1 -- 172.21.15.102:0/4200599083 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42680d1a70 con 0x7f42680b5aa0 2024-03-23T03:48:45.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.258+0000 7f4267400640 1 --2- 172.21.15.102:0/4200599083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42680b5aa0 0x7f42680b5f30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f42600097a0 tx=0x7f426002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=294f0823e2037414 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:45.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.259+0000 7f4265600640 1 -- 172.21.15.102:0/4200599083 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426002d040 con 0x7f42680b5aa0 2024-03-23T03:48:45.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.259+0000 7f4265600640 1 -- 172.21.15.102:0/4200599083 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4260031810 con 0x7f42680b5aa0 2024-03-23T03:48:45.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.259+0000 7f4265600640 1 -- 172.21.15.102:0/4200599083 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4260031ba0 con 0x7f42680b5aa0 2024-03-23T03:48:45.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.259+0000 7f426dc00640 1 -- 172.21.15.102:0/4200599083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42680b5aa0 msgr2=0x7f42680b5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:45.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.259+0000 7f426dc00640 1 --2- 172.21.15.102:0/4200599083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42680b5aa0 0x7f42680b5f30 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f42600097a0 tx=0x7f426002b8b0 comp rx=0 tx=0).stop 2024-03-23T03:48:45.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.259+0000 7f426dc00640 1 -- 172.21.15.102:0/4200599083 shutdown_connections 2024-03-23T03:48:45.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.259+0000 7f426dc00640 1 --2- 172.21.15.102:0/4200599083 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42680b5aa0 0x7f42680b5f30 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:45.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.260+0000 7f426dc00640 1 --2- 172.21.15.102:0/4200599083 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f42680cd740 0x7f42680b84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:45.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.260+0000 7f426dc00640 1 -- 172.21.15.102:0/4200599083 >> 172.21.15.102:0/4200599083 conn(0x7f42680cf5d0 msgr2=0x7f42680cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:45.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.260+0000 7f426dc00640 1 -- 172.21.15.102:0/4200599083 shutdown_connections 2024-03-23T03:48:45.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.260+0000 7f426dc00640 1 -- 172.21.15.102:0/4200599083 wait complete. 2024-03-23T03:48:45.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.262+0000 7f426dc00640 1 Processor -- start 2024-03-23T03:48:45.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.262+0000 7f426dc00640 1 -- start start 2024-03-23T03:48:45.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.263+0000 7f426dc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f42680b5aa0 0x7f42681586a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:45.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.263+0000 7f426dc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42680cd740 0x7f426814a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.263+0000 7f426dc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42680d0d40 0x7f426814aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.263+0000 7f426dc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4268148610 con 0x7f42680cd740 2024-03-23T03:48:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.263+0000 7f426dc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426815b9f0 con 0x7f42680d0d40 2024-03-23T03:48:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.263+0000 7f426dc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426815ae70 con 0x7f42680b5aa0 2024-03-23T03:48:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.263+0000 7f4266000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42680cd740 0x7f426814a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.263+0000 7f4266000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42680cd740 0x7f426814a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46694/0 (socket says 172.21.15.102:46694) 2024-03-23T03:48:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.263+0000 7f4266000640 1 -- 172.21.15.102:0/548811266 learned_addr learned my addr 172.21.15.102:0/548811266 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.264+0000 7f4267400640 1 --2- 172.21.15.102:0/548811266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42680d0d40 0x7f426814aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.264+0000 7f4266a00640 1 --2- 172.21.15.102:0/548811266 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f42680b5aa0 0x7f42681586a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.264+0000 7f4266000640 1 -- 172.21.15.102:0/548811266 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f42680b5aa0 msgr2=0x7f42681586a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.264+0000 7f4266000640 1 --2- 172.21.15.102:0/548811266 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f42680b5aa0 0x7f42681586a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.264+0000 7f4266000640 1 -- 172.21.15.102:0/548811266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42680d0d40 msgr2=0x7f426814aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.264+0000 7f4266000640 1 --2- 172.21.15.102:0/548811266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42680d0d40 0x7f426814aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:45.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.264+0000 7f4266000640 1 -- 172.21.15.102:0/548811266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4260009400 con 0x7f42680cd740 2024-03-23T03:48:45.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.264+0000 7f4266000640 1 --2- 172.21.15.102:0/548811266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42680cd740 0x7f426814a960 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f425000cbc0 tx=0x7f425000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:45.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.264+0000 7f425ea00640 1 -- 172.21.15.102:0/548811266 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425000aca0 con 0x7f42680cd740 2024-03-23T03:48:45.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.264+0000 7f425ea00640 1 -- 172.21.15.102:0/548811266 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425000ae00 con 0x7f42680cd740 2024-03-23T03:48:45.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.264+0000 7f425ea00640 1 -- 172.21.15.102:0/548811266 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4250012600 con 0x7f42680cd740 2024-03-23T03:48:45.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.265+0000 7f426dc00640 1 -- 172.21.15.102:0/548811266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f426815b150 con 0x7f42680cd740 2024-03-23T03:48:45.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.265+0000 7f426dc00640 1 -- 172.21.15.102:0/548811266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426815b710 con 0x7f42680cd740 2024-03-23T03:48:45.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.266+0000 7f426dc00640 1 -- 172.21.15.102:0/548811266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42680cab20 con 0x7f42680cd740 2024-03-23T03:48:45.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.267+0000 7f425ea00640 1 -- 172.21.15.102:0/548811266 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4250007070 con 0x7f42680cd740 2024-03-23T03:48:45.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.267+0000 7f425ea00640 1 --2- 172.21.15.102:0/548811266 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f424406ac00 0x7f424406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:45.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.267+0000 7f425ea00640 1 -- 172.21.15.102:0/548811266 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f42500892c0 con 0x7f42680cd740 2024-03-23T03:48:45.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.268+0000 7f4266a00640 1 --2- 172.21.15.102:0/548811266 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f424406ac00 0x7f424406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:45.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.268+0000 7f4266a00640 1 --2- 172.21.15.102:0/548811266 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f424406ac00 0x7f424406d0c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4258006fd0 tx=0x7f4258008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:45.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.272+0000 7f425ea00640 1 -- 172.21.15.102:0/548811266 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f42680cab20 con 0x7f42680cd740 2024-03-23T03:48:45.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.465+0000 7f426dc00640 1 -- 172.21.15.102:0/548811266 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f42680cab20 con 0x7f42680cd740 2024-03-23T03:48:45.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.984+0000 7f425ea00640 1 -- 172.21.15.102:0/548811266 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v45) v1 ==== 181+0+0 (secure 0 0 0) 0x7f42680cab20 con 0x7f42680cd740 2024-03-23T03:48:45.985 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' removed 2024-03-23T03:48:45.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.988+0000 7f426dc00640 1 -- 172.21.15.102:0/548811266 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f424406ac00 msgr2=0x7f424406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:45.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.988+0000 7f426dc00640 1 --2- 172.21.15.102:0/548811266 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f424406ac00 0x7f424406d0c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4258006fd0 tx=0x7f4258008040 comp rx=0 tx=0).stop 2024-03-23T03:48:45.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.988+0000 7f426dc00640 1 -- 172.21.15.102:0/548811266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42680cd740 msgr2=0x7f426814a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:45.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.988+0000 7f426dc00640 1 --2- 172.21.15.102:0/548811266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42680cd740 0x7f426814a960 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f425000cbc0 tx=0x7f425000a060 comp rx=0 tx=0).stop 2024-03-23T03:48:45.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.989+0000 7f426dc00640 1 -- 172.21.15.102:0/548811266 shutdown_connections 2024-03-23T03:48:45.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.989+0000 7f426dc00640 1 --2- 172.21.15.102:0/548811266 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f424406ac00 0x7f424406d0c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:45.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.989+0000 7f426dc00640 1 --2- 172.21.15.102:0/548811266 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42680d0d40 0x7f426814aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:45.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.989+0000 7f426dc00640 1 --2- 172.21.15.102:0/548811266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42680cd740 0x7f426814a960 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:45.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.989+0000 7f426dc00640 1 --2- 172.21.15.102:0/548811266 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f42680b5aa0 0x7f42681586a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:45.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.989+0000 7f426dc00640 1 -- 172.21.15.102:0/548811266 >> 172.21.15.102:0/548811266 conn(0x7f42680cf5d0 msgr2=0x7f426815ed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:45.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.989+0000 7f426dc00640 1 -- 172.21.15.102:0/548811266 shutdown_connections 2024-03-23T03:48:45.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:45.989+0000 7f426dc00640 1 -- 172.21.15.102:0/548811266 wait complete. 2024-03-23T03:48:46.014 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T03:48:46.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.226+0000 7f918f600640 1 Processor -- start 2024-03-23T03:48:46.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.226+0000 7f918f600640 1 -- start start 2024-03-23T03:48:46.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.227+0000 7f918f600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9188159830 0x7f91880cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:46.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.227+0000 7f918f600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9188159c10 con 0x7f91880ce930 2024-03-23T03:48:46.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.227+0000 7f918f600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91880cd840 con 0x7f91880ca7b0 2024-03-23T03:48:46.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.227+0000 7f918f600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f918806de50 con 0x7f9188159830 2024-03-23T03:48:46.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.228+0000 7f918d600640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f91880ce930 0x7f91880b18d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35926/0 (socket says 172.21.15.102:35926) 2024-03-23T03:48:46.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.228+0000 7f918d600640 1 -- 172.21.15.102:0/3055484447 learned_addr learned my addr 172.21.15.102:0/3055484447 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:46.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.228+0000 7f918cc00640 1 --2- 172.21.15.102:0/3055484447 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9188159830 0x7f91880cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.228+0000 7f9187e00640 1 -- 172.21.15.102:0/3055484447 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225697243 0 0) 0x7f9188159c10 con 0x7f91880ce930 2024-03-23T03:48:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.228+0000 7f9187e00640 1 -- 172.21.15.102:0/3055484447 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f916c003520 con 0x7f91880ce930 2024-03-23T03:48:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.229+0000 7f9187e00640 1 -- 172.21.15.102:0/3055484447 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1106414832 0 0) 0x7f916c003520 con 0x7f91880ce930 2024-03-23T03:48:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.229+0000 7f9187e00640 1 -- 172.21.15.102:0/3055484447 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9188159830 msgr2=0x7f91880cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.229+0000 7f9187e00640 1 --2- 172.21.15.102:0/3055484447 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9188159830 0x7f91880cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.229+0000 7f9187e00640 1 -- 172.21.15.102:0/3055484447 >> v1:172.21.15.173:6789/0 conn(0x7f91880ca7b0 legacy=0x7f91880cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.229+0000 7f9187e00640 1 -- 172.21.15.102:0/3055484447 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91880ccc30 con 0x7f91880ce930 2024-03-23T03:48:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.229+0000 7f9187e00640 1 -- 172.21.15.102:0/3055484447 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f91800019f0 con 0x7f91880ce930 2024-03-23T03:48:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.229+0000 7f9187e00640 1 -- 172.21.15.102:0/3055484447 >> v1:172.21.15.102:6789/0 conn(0x7f91880ce930 legacy=0x7f91880b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.229+0000 7f9187e00640 1 --2- 172.21.15.102:0/3055484447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f916c0039a0 0x7f916c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.229+0000 7f9187e00640 1 --2- 172.21.15.102:0/3055484447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f916c0047c0 0x7f916c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.229+0000 7f9187e00640 1 --2- 172.21.15.102:0/3055484447 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9188159830 0x7f916c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:46.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.229+0000 7f9187e00640 1 -- 172.21.15.102:0/3055484447 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c00bac0 con 0x7f916c0047c0 2024-03-23T03:48:46.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.229+0000 7f9187e00640 1 -- 172.21.15.102:0/3055484447 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f916c00bc30 con 0x7f916c0039a0 2024-03-23T03:48:46.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.229+0000 7f9187e00640 1 -- 172.21.15.102:0/3055484447 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f916c00e000 con 0x7f9188159830 2024-03-23T03:48:46.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.229+0000 7f917fe00640 1 --2- 172.21.15.102:0/3055484447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f916c0047c0 0x7f916c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:46.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.230+0000 7f918cc00640 1 --2- 172.21.15.102:0/3055484447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f916c0039a0 0x7f916c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:46.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.230+0000 7f917fe00640 1 -- 172.21.15.102:0/3055484447 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9188159830 msgr2=0x7f916c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:46.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.230+0000 7f917fe00640 1 --2- 172.21.15.102:0/3055484447 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9188159830 0x7f916c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:46.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.230+0000 7f917fe00640 1 -- 172.21.15.102:0/3055484447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f916c0039a0 msgr2=0x7f916c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:46.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.230+0000 7f917fe00640 1 --2- 172.21.15.102:0/3055484447 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f916c0039a0 0x7f916c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:46.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.230+0000 7f917fe00640 1 -- 172.21.15.102:0/3055484447 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c003520 con 0x7f916c0047c0 2024-03-23T03:48:46.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.231+0000 7f917fe00640 1 --2- 172.21.15.102:0/3055484447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f916c0047c0 0x7f916c004c40 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f9170001720 tx=0x7f916c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b97d4690baab7c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:46.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.231+0000 7f9187e00640 1 -- 172.21.15.102:0/3055484447 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91700052d0 con 0x7f916c0047c0 2024-03-23T03:48:46.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.231+0000 7f9187e00640 1 -- 172.21.15.102:0/3055484447 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917002e050 con 0x7f916c0047c0 2024-03-23T03:48:46.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.231+0000 7f9187e00640 1 -- 172.21.15.102:0/3055484447 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91700035f0 con 0x7f916c0047c0 2024-03-23T03:48:46.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.232+0000 7f918f600640 1 -- 172.21.15.102:0/3055484447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f916c0047c0 msgr2=0x7f916c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:46.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.232+0000 7f918f600640 1 --2- 172.21.15.102:0/3055484447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f916c0047c0 0x7f916c004c40 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f9170001720 tx=0x7f916c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:46.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.232+0000 7f918d600640 1 -- 172.21.15.102:0/3055484447 reap_dead start 2024-03-23T03:48:46.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.232+0000 7f918f600640 1 -- 172.21.15.102:0/3055484447 shutdown_connections 2024-03-23T03:48:46.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.232+0000 7f918f600640 1 -- 172.21.15.102:0/3055484447 >> 172.21.15.102:0/3055484447 conn(0x7f918806a0b0 msgr2=0x7f918806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:46.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.232+0000 7f918f600640 1 -- 172.21.15.102:0/3055484447 shutdown_connections 2024-03-23T03:48:46.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.232+0000 7f918f600640 1 -- 172.21.15.102:0/3055484447 wait complete. 2024-03-23T03:48:46.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.234+0000 7f918f600640 1 Processor -- start 2024-03-23T03:48:46.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.234+0000 7f918f600640 1 -- start start 2024-03-23T03:48:46.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.235+0000 7f918f600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f916c0047c0 0x7f918814b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:46.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.235+0000 7f918f600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91880ae900 0x7f91880aed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:46.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.235+0000 7f918f600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9188152d60 0x7f9188153210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:46.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.235+0000 7f918f600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91880af2c0 con 0x7f9188152d60 2024-03-23T03:48:46.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.235+0000 7f918f600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f918814b880 con 0x7f91880ae900 2024-03-23T03:48:46.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.235+0000 7f918f600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9188153b10 con 0x7f916c0047c0 2024-03-23T03:48:46.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.235+0000 7f918d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9188152d60 0x7f9188153210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:46.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.235+0000 7f918d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9188152d60 0x7f9188153210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46716/0 (socket says 172.21.15.102:46716) 2024-03-23T03:48:46.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.235+0000 7f918d600640 1 -- 172.21.15.102:0/3754039086 learned_addr learned my addr 172.21.15.102:0/3754039086 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:46.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.236+0000 7f918cc00640 1 --2- 172.21.15.102:0/3754039086 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f916c0047c0 0x7f918814b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:46.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.236+0000 7f917fe00640 1 --2- 172.21.15.102:0/3754039086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91880ae900 0x7f91880aed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:46.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.236+0000 7f918d600640 1 -- 172.21.15.102:0/3754039086 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f916c0047c0 msgr2=0x7f918814b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:46.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.236+0000 7f918d600640 1 --2- 172.21.15.102:0/3754039086 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f916c0047c0 0x7f918814b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:46.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.236+0000 7f918d600640 1 -- 172.21.15.102:0/3754039086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91880ae900 msgr2=0x7f91880aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:46.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.236+0000 7f918d600640 1 --2- 172.21.15.102:0/3754039086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91880ae900 0x7f91880aed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:46.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.236+0000 7f918d600640 1 -- 172.21.15.102:0/3754039086 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9188153df0 con 0x7f9188152d60 2024-03-23T03:48:46.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.236+0000 7f918d600640 1 --2- 172.21.15.102:0/3754039086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9188152d60 0x7f9188153210 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9180001350 tx=0x7f918000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:46.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.236+0000 7f9185600640 1 -- 172.21.15.102:0/3754039086 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9180007c20 con 0x7f9188152d60 2024-03-23T03:48:46.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.236+0000 7f9185600640 1 -- 172.21.15.102:0/3754039086 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918000a610 con 0x7f9188152d60 2024-03-23T03:48:46.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.236+0000 7f9185600640 1 -- 172.21.15.102:0/3754039086 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918000a9c0 con 0x7f9188152d60 2024-03-23T03:48:46.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.236+0000 7f918f600640 1 -- 172.21.15.102:0/3754039086 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91881540d0 con 0x7f9188152d60 2024-03-23T03:48:46.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.236+0000 7f918f600640 1 -- 172.21.15.102:0/3754039086 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9188154690 con 0x7f9188152d60 2024-03-23T03:48:46.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.237+0000 7f918f600640 1 -- 172.21.15.102:0/3754039086 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91880b1f60 con 0x7f9188152d60 2024-03-23T03:48:46.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.238+0000 7f9185600640 1 -- 172.21.15.102:0/3754039086 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f918000ab20 con 0x7f9188152d60 2024-03-23T03:48:46.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.239+0000 7f9185600640 1 --2- 172.21.15.102:0/3754039086 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f916406ac30 0x7f916406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:46.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.239+0000 7f9185600640 1 -- 172.21.15.102:0/3754039086 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5706+0+0 (secure 0 0 0) 0x7f9180088bc0 con 0x7f9188152d60 2024-03-23T03:48:46.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.240+0000 7f918cc00640 1 --2- 172.21.15.102:0/3754039086 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f916406ac30 0x7f916406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:46.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.240+0000 7f918cc00640 1 --2- 172.21.15.102:0/3754039086 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f916406ac30 0x7f916406d0f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9178009950 tx=0x7f9178008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:46.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.244+0000 7f9185600640 1 -- 172.21.15.102:0/3754039086 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9180053d00 con 0x7f9188152d60 2024-03-23T03:48:46.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.442+0000 7f918f600640 1 -- 172.21.15.102:0/3754039086 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9188073390 con 0x7f9188152d60 2024-03-23T03:48:46.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.988+0000 7f9185600640 1 -- 172.21.15.102:0/3754039086 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v46) v1 ==== 169+0+0 (secure 0 0 0) 0x7f918001a070 con 0x7f9188152d60 2024-03-23T03:48:46.989 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' removed 2024-03-23T03:48:46.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.994+0000 7f918f600640 1 -- 172.21.15.102:0/3754039086 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f916406ac30 msgr2=0x7f916406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:46.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.994+0000 7f918f600640 1 --2- 172.21.15.102:0/3754039086 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f916406ac30 0x7f916406d0f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9178009950 tx=0x7f9178008040 comp rx=0 tx=0).stop 2024-03-23T03:48:46.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.994+0000 7f918f600640 1 -- 172.21.15.102:0/3754039086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9188152d60 msgr2=0x7f9188153210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:46.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.994+0000 7f918f600640 1 --2- 172.21.15.102:0/3754039086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9188152d60 0x7f9188153210 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f9180001350 tx=0x7f918000ccd0 comp rx=0 tx=0).stop 2024-03-23T03:48:46.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.994+0000 7f918f600640 1 -- 172.21.15.102:0/3754039086 shutdown_connections 2024-03-23T03:48:46.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.994+0000 7f918f600640 1 --2- 172.21.15.102:0/3754039086 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f916406ac30 0x7f916406d0f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:46.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.994+0000 7f918f600640 1 --2- 172.21.15.102:0/3754039086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9188152d60 0x7f9188153210 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:46.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.994+0000 7f918f600640 1 --2- 172.21.15.102:0/3754039086 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f91880ae900 0x7f91880aed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:46.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.994+0000 7f918f600640 1 --2- 172.21.15.102:0/3754039086 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f916c0047c0 0x7f918814b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:46.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.994+0000 7f918f600640 1 -- 172.21.15.102:0/3754039086 >> 172.21.15.102:0/3754039086 conn(0x7f918806a0b0 msgr2=0x7f9188156130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:46.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.995+0000 7f918f600640 1 -- 172.21.15.102:0/3754039086 shutdown_connections 2024-03-23T03:48:46.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:46.996+0000 7f918f600640 1 -- 172.21.15.102:0/3754039086 wait complete. 2024-03-23T03:48:47.024 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-23T03:48:47.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.226+0000 7f923fe00640 1 Processor -- start 2024-03-23T03:48:47.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.226+0000 7f923fe00640 1 -- start start 2024-03-23T03:48:47.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.227+0000 7f923fe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92400cc200 0x7f92400cab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:47.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.227+0000 7f923fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f92400cb070 0x7f924015a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:47.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.227+0000 7f923fe00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92400c8c90 con 0x7f9240159cd0 2024-03-23T03:48:47.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.227+0000 7f923fe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92400cc6c0 con 0x7f92400cb070 2024-03-23T03:48:47.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.227+0000 7f923fe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92400cc9a0 con 0x7f92400cc200 2024-03-23T03:48:47.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.228+0000 7f923f400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f92400cb070 0x7f924015a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:47.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.228+0000 7f923f400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f92400cb070 0x7f924015a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46730/0 (socket says 172.21.15.102:46730) 2024-03-23T03:48:47.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.228+0000 7f923ea00640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f9240159cd0 0x7f92400ad260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:55496/0 (socket says 172.21.15.102:55496) 2024-03-23T03:48:47.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.228+0000 7f923f400640 1 -- 172.21.15.102:0/1015045042 learned_addr learned my addr 172.21.15.102:0/1015045042 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:47.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.228+0000 7f9236000640 1 --2- 172.21.15.102:0/1015045042 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92400cc200 0x7f92400cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:47.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.228+0000 7f923f400640 1 -- 172.21.15.102:0/1015045042 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92400cc200 msgr2=0x7f92400cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:47.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.228+0000 7f923f400640 1 --2- 172.21.15.102:0/1015045042 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92400cc200 0x7f92400cab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.228+0000 7f923f400640 1 -- 172.21.15.102:0/1015045042 >> v1:172.21.15.173:6789/0 conn(0x7f9240159cd0 legacy=0x7f92400ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:47.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.228+0000 7f923f400640 1 -- 172.21.15.102:0/1015045042 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92400ccc80 con 0x7f92400cb070 2024-03-23T03:48:47.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.228+0000 7f923e000640 1 -- 172.21.15.102:0/1015045042 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2451572013 0 0) 0x7f92400c8c90 con 0x7f9240159cd0 2024-03-23T03:48:47.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.229+0000 7f923f400640 1 --2- 172.21.15.102:0/1015045042 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92400cb070 0x7f924015a0b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9238009870 tx=0x7f923802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dc0f6b258050745 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:47.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.229+0000 7f923e000640 1 -- 172.21.15.102:0/1015045042 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9238030840 con 0x7f92400cb070 2024-03-23T03:48:47.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.229+0000 7f923e000640 1 -- 172.21.15.102:0/1015045042 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92380309a0 con 0x7f92400cb070 2024-03-23T03:48:47.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.230+0000 7f923e000640 1 -- 172.21.15.102:0/1015045042 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9238030d30 con 0x7f92400cb070 2024-03-23T03:48:47.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.230+0000 7f923fe00640 1 -- 172.21.15.102:0/1015045042 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92400cb070 msgr2=0x7f924015a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:47.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.230+0000 7f923fe00640 1 --2- 172.21.15.102:0/1015045042 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92400cb070 0x7f924015a0b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9238009870 tx=0x7f923802b980 comp rx=0 tx=0).stop 2024-03-23T03:48:47.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.230+0000 7f923fe00640 1 -- 172.21.15.102:0/1015045042 shutdown_connections 2024-03-23T03:48:47.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.230+0000 7f923fe00640 1 --2- 172.21.15.102:0/1015045042 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92400cb070 0x7f924015a0b0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.230+0000 7f923fe00640 1 --2- 172.21.15.102:0/1015045042 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92400cc200 0x7f92400cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.230+0000 7f923fe00640 1 -- 172.21.15.102:0/1015045042 >> 172.21.15.102:0/1015045042 conn(0x7f92400b01b0 msgr2=0x7f92400b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:47.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.230+0000 7f923fe00640 1 -- 172.21.15.102:0/1015045042 shutdown_connections 2024-03-23T03:48:47.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.230+0000 7f923fe00640 1 -- 172.21.15.102:0/1015045042 wait complete. 2024-03-23T03:48:47.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.232+0000 7f923fe00640 1 Processor -- start 2024-03-23T03:48:47.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.232+0000 7f923fe00640 1 -- start start 2024-03-23T03:48:47.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.233+0000 7f923fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92400cb070 0x7f9240098dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:47.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.233+0000 7f923fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92400cc200 0x7f92400982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:47.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.233+0000 7f923fe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9240159cd0 0x7f92401f1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:47.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.233+0000 7f923fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92401f23f0 con 0x7f92400cb070 2024-03-23T03:48:47.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.233+0000 7f923fe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92400af720 con 0x7f92400cc200 2024-03-23T03:48:47.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.233+0000 7f923fe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9240158330 con 0x7f9240159cd0 2024-03-23T03:48:47.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.233+0000 7f923ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92400cb070 0x7f9240098dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:47.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.233+0000 7f923ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92400cb070 0x7f9240098dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46744/0 (socket says 172.21.15.102:46744) 2024-03-23T03:48:47.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.233+0000 7f923ea00640 1 -- 172.21.15.102:0/2678504209 learned_addr learned my addr 172.21.15.102:0/2678504209 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:47.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.233+0000 7f9236000640 1 --2- 172.21.15.102:0/2678504209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92400cc200 0x7f92400982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:47.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.234+0000 7f923f400640 1 --2- 172.21.15.102:0/2678504209 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9240159cd0 0x7f92401f1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:47.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.234+0000 7f923ea00640 1 -- 172.21.15.102:0/2678504209 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9240159cd0 msgr2=0x7f92401f1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:47.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.234+0000 7f923ea00640 1 --2- 172.21.15.102:0/2678504209 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9240159cd0 0x7f92401f1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.234+0000 7f923ea00640 1 -- 172.21.15.102:0/2678504209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92400cc200 msgr2=0x7f92400982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:47.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.234+0000 7f923ea00640 1 --2- 172.21.15.102:0/2678504209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92400cc200 0x7f92400982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.234+0000 7f923ea00640 1 -- 172.21.15.102:0/2678504209 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92380094d0 con 0x7f92400cb070 2024-03-23T03:48:47.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.234+0000 7f9236000640 1 --2- 172.21.15.102:0/2678504209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92400cc200 0x7f92400982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:48:47.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.234+0000 7f923ea00640 1 --2- 172.21.15.102:0/2678504209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92400cb070 0x7f9240098dd0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f923000c7c0 tx=0x7f923000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:47.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.234+0000 7f9237400640 1 -- 172.21.15.102:0/2678504209 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923000a640 con 0x7f92400cb070 2024-03-23T03:48:47.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.234+0000 7f9237400640 1 -- 172.21.15.102:0/2678504209 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f923000a7c0 con 0x7f92400cb070 2024-03-23T03:48:47.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.234+0000 7f923fe00640 1 -- 172.21.15.102:0/2678504209 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92400aee30 con 0x7f92400cb070 2024-03-23T03:48:47.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.234+0000 7f9237400640 1 -- 172.21.15.102:0/2678504209 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923000ab70 con 0x7f92400cb070 2024-03-23T03:48:47.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.234+0000 7f923fe00640 1 -- 172.21.15.102:0/2678504209 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92400993b0 con 0x7f92400cb070 2024-03-23T03:48:47.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.236+0000 7f923fe00640 1 -- 172.21.15.102:0/2678504209 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f924004e030 con 0x7f92400cb070 2024-03-23T03:48:47.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.236+0000 7f9237400640 1 -- 172.21.15.102:0/2678504209 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f923000acd0 con 0x7f92400cb070 2024-03-23T03:48:47.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.237+0000 7f9237400640 1 --2- 172.21.15.102:0/2678504209 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f921c06ac50 0x7f921c06d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:47.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.237+0000 7f9237400640 1 -- 172.21.15.102:0/2678504209 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f9230088aa0 con 0x7f92400cb070 2024-03-23T03:48:47.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.237+0000 7f9236000640 1 --2- 172.21.15.102:0/2678504209 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f921c06ac50 0x7f921c06d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:47.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.238+0000 7f9236000640 1 --2- 172.21.15.102:0/2678504209 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f921c06ac50 0x7f921c06d110 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f92400ae130 tx=0x7f9228008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:47.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.242+0000 7f9237400640 1 -- 172.21.15.102:0/2678504209 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9230053eb0 con 0x7f92400cb070 2024-03-23T03:48:47.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.436+0000 7f923fe00640 1 -- 172.21.15.102:0/2678504209 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f92400ae160 con 0x7f92400cb070 2024-03-23T03:48:47.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.445+0000 7f9237400640 1 -- 172.21.15.102:0/2678504209 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9230018020 con 0x7f92400cb070 2024-03-23T03:48:47.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.447+0000 7f923fe00640 1 -- 172.21.15.102:0/2678504209 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f921c06ac50 msgr2=0x7f921c06d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:47.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.448+0000 7f923fe00640 1 --2- 172.21.15.102:0/2678504209 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f921c06ac50 0x7f921c06d110 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f92400ae130 tx=0x7f9228008040 comp rx=0 tx=0).stop 2024-03-23T03:48:47.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.448+0000 7f923fe00640 1 -- 172.21.15.102:0/2678504209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92400cb070 msgr2=0x7f9240098dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:47.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.448+0000 7f923fe00640 1 --2- 172.21.15.102:0/2678504209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92400cb070 0x7f9240098dd0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f923000c7c0 tx=0x7f923000cc80 comp rx=0 tx=0).stop 2024-03-23T03:48:47.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.448+0000 7f923fe00640 1 -- 172.21.15.102:0/2678504209 shutdown_connections 2024-03-23T03:48:47.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.448+0000 7f923fe00640 1 --2- 172.21.15.102:0/2678504209 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f921c06ac50 0x7f921c06d110 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.448+0000 7f923fe00640 1 --2- 172.21.15.102:0/2678504209 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9240159cd0 0x7f92401f1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.448+0000 7f923fe00640 1 --2- 172.21.15.102:0/2678504209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92400cc200 0x7f92400982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.448+0000 7f923fe00640 1 --2- 172.21.15.102:0/2678504209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92400cb070 0x7f9240098dd0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.448+0000 7f923fe00640 1 -- 172.21.15.102:0/2678504209 >> 172.21.15.102:0/2678504209 conn(0x7f92400b01b0 msgr2=0x7f924006dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:47.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.448+0000 7f923fe00640 1 -- 172.21.15.102:0/2678504209 shutdown_connections 2024-03-23T03:48:47.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.448+0000 7f923fe00640 1 -- 172.21.15.102:0/2678504209 wait complete. 2024-03-23T03:48:47.467 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values' 2024-03-23T03:48:47.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.672+0000 7ff42ba00640 1 Processor -- start 2024-03-23T03:48:47.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.673+0000 7ff42ba00640 1 -- start start 2024-03-23T03:48:47.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.674+0000 7ff42ba00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ff4240b3e30 0x7ff42415cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:47.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.674+0000 7ff42ba00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42406c6f0 con 0x7ff42415b560 2024-03-23T03:48:47.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.674+0000 7ff42ba00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42406c9d0 con 0x7ff4240b39b0 2024-03-23T03:48:47.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.674+0000 7ff42ba00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff42406ccb0 con 0x7ff4240b3e30 2024-03-23T03:48:47.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.674+0000 7ff429000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff42415b560 0x7ff42415b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35932/0 (socket says 172.21.15.102:35932) 2024-03-23T03:48:47.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.675+0000 7ff429000640 1 -- 172.21.15.102:0/416314003 learned_addr learned my addr 172.21.15.102:0/416314003 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:47.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.675+0000 7ff429a00640 1 --2- 172.21.15.102:0/416314003 >> v2:172.21.15.173:3300/0 conn(0x7ff4240b3e30 0x7ff42415cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:47.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.675+0000 7ff423e00640 1 -- 172.21.15.102:0/416314003 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360482230 0 0) 0x7ff42406c6f0 con 0x7ff42415b560 2024-03-23T03:48:47.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.675+0000 7ff423e00640 1 -- 172.21.15.102:0/416314003 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff404003520 con 0x7ff42415b560 2024-03-23T03:48:47.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.675+0000 7ff423e00640 1 -- 172.21.15.102:0/416314003 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3291947109 0 0) 0x7ff404003520 con 0x7ff42415b560 2024-03-23T03:48:47.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.675+0000 7ff423e00640 1 -- 172.21.15.102:0/416314003 >> v2:172.21.15.173:3300/0 conn(0x7ff4240b3e30 msgr2=0x7ff42415cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.675+0000 7ff423e00640 1 --2- 172.21.15.102:0/416314003 >> v2:172.21.15.173:3300/0 conn(0x7ff4240b3e30 0x7ff42415cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.675+0000 7ff423e00640 1 -- 172.21.15.102:0/416314003 >> v1:172.21.15.173:6789/0 conn(0x7ff4240b39b0 legacy=0x7ff42415c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.675+0000 7ff423e00640 1 -- 172.21.15.102:0/416314003 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42406cf90 con 0x7ff42415b560 2024-03-23T03:48:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.675+0000 7ff423e00640 1 -- 172.21.15.102:0/416314003 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7ff40c001670 con 0x7ff42415b560 2024-03-23T03:48:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.675+0000 7ff429a00640 1 --2- 172.21.15.102:0/416314003 >> v2:172.21.15.173:3300/0 conn(0x7ff4240b3e30 0x7ff42415cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:48:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.676+0000 7ff423e00640 1 -- 172.21.15.102:0/416314003 >> v1:172.21.15.102:6789/0 conn(0x7ff42415b560 legacy=0x7ff42415b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.676+0000 7ff423e00640 1 --2- 172.21.15.102:0/416314003 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff404003970 0x7ff404003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.676+0000 7ff423e00640 1 --2- 172.21.15.102:0/416314003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4040047c0 0x7ff404004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.676+0000 7ff423e00640 1 --2- 172.21.15.102:0/416314003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff404005180 0x7ff40400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.676+0000 7ff423e00640 1 -- 172.21.15.102:0/416314003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42406ccb0 con 0x7ff4040047c0 2024-03-23T03:48:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.676+0000 7ff423e00640 1 -- 172.21.15.102:0/416314003 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40400c0d0 con 0x7ff404005180 2024-03-23T03:48:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.676+0000 7ff423e00640 1 -- 172.21.15.102:0/416314003 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff40400c290 con 0x7ff404003970 2024-03-23T03:48:47.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.676+0000 7ff41be00640 1 --2- 172.21.15.102:0/416314003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4040047c0 0x7ff404004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:47.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.676+0000 7ff429a00640 1 --2- 172.21.15.102:0/416314003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff404005180 0x7ff40400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:47.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.676+0000 7ff41be00640 1 -- 172.21.15.102:0/416314003 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff404003970 msgr2=0x7ff404003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:47.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.676+0000 7ff41be00640 1 --2- 172.21.15.102:0/416314003 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff404003970 0x7ff404003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.676+0000 7ff41be00640 1 -- 172.21.15.102:0/416314003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff404005180 msgr2=0x7ff40400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:47.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.676+0000 7ff41be00640 1 --2- 172.21.15.102:0/416314003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff404005180 0x7ff40400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.676+0000 7ff41be00640 1 -- 172.21.15.102:0/416314003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff404003520 con 0x7ff4040047c0 2024-03-23T03:48:47.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.676+0000 7ff429a00640 1 --2- 172.21.15.102:0/416314003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff404005180 0x7ff40400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:47.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.676+0000 7ff429a00640 1 -- 172.21.15.102:0/416314003 reap_dead start 2024-03-23T03:48:47.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.677+0000 7ff41be00640 1 --2- 172.21.15.102:0/416314003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4040047c0 0x7ff404004c40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff414001750 tx=0x7ff4040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7439f2e8c00f31d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:47.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.678+0000 7ff423e00640 1 -- 172.21.15.102:0/416314003 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4140052d0 con 0x7ff4040047c0 2024-03-23T03:48:47.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.678+0000 7ff423e00640 1 -- 172.21.15.102:0/416314003 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff41402e050 con 0x7ff4040047c0 2024-03-23T03:48:47.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.678+0000 7ff423e00640 1 -- 172.21.15.102:0/416314003 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414003830 con 0x7ff4040047c0 2024-03-23T03:48:47.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.678+0000 7ff42ba00640 1 -- 172.21.15.102:0/416314003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4040047c0 msgr2=0x7ff404004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:47.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.678+0000 7ff42ba00640 1 --2- 172.21.15.102:0/416314003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4040047c0 0x7ff404004c40 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff414001750 tx=0x7ff4040057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.678+0000 7ff42ba00640 1 -- 172.21.15.102:0/416314003 shutdown_connections 2024-03-23T03:48:47.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.678+0000 7ff42ba00640 1 --2- 172.21.15.102:0/416314003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4040047c0 0x7ff404004c40 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.678+0000 7ff42ba00640 1 -- 172.21.15.102:0/416314003 >> 172.21.15.102:0/416314003 conn(0x7ff42406a3d0 msgr2=0x7ff4240b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:47.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.678+0000 7ff42ba00640 1 -- 172.21.15.102:0/416314003 shutdown_connections 2024-03-23T03:48:47.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.678+0000 7ff42ba00640 1 -- 172.21.15.102:0/416314003 wait complete. 2024-03-23T03:48:47.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.680+0000 7ff42ba00640 1 Processor -- start 2024-03-23T03:48:47.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.680+0000 7ff42ba00640 1 -- start start 2024-03-23T03:48:47.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.681+0000 7ff42ba00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4040047c0 0x7ff424150590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:47.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.681+0000 7ff42ba00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff424157360 0x7ff424153f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:47.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.681+0000 7ff42ba00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff424154460 0x7ff4240cc670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:47.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.681+0000 7ff42ba00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff424156f40 con 0x7ff4040047c0 2024-03-23T03:48:47.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.681+0000 7ff42ba00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff424154970 con 0x7ff424154460 2024-03-23T03:48:47.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.681+0000 7ff42ba00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff424156880 con 0x7ff424157360 2024-03-23T03:48:47.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.681+0000 7ff429000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4040047c0 0x7ff424150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:47.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.682+0000 7ff429000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4040047c0 0x7ff424150590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46758/0 (socket says 172.21.15.102:46758) 2024-03-23T03:48:47.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.682+0000 7ff429000640 1 -- 172.21.15.102:0/3945973507 learned_addr learned my addr 172.21.15.102:0/3945973507 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:47.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.682+0000 7ff41be00640 1 --2- 172.21.15.102:0/3945973507 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff424157360 0x7ff424153f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:47.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.682+0000 7ff429000640 1 -- 172.21.15.102:0/3945973507 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff424157360 msgr2=0x7ff424153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:47.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.682+0000 7ff429000640 1 --2- 172.21.15.102:0/3945973507 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff424157360 0x7ff424153f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.682+0000 7ff429000640 1 -- 172.21.15.102:0/3945973507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff424154460 msgr2=0x7ff4240cc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:47.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.682+0000 7ff429000640 1 --2- 172.21.15.102:0/3945973507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff424154460 0x7ff4240cc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.682+0000 7ff429000640 1 -- 172.21.15.102:0/3945973507 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff424156b60 con 0x7ff4040047c0 2024-03-23T03:48:47.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.682+0000 7ff41be00640 1 --2- 172.21.15.102:0/3945973507 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff424157360 0x7ff424153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:47.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.682+0000 7ff429000640 1 --2- 172.21.15.102:0/3945973507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4040047c0 0x7ff424150590 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff40c00c7a0 tx=0x7ff40c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:47.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.683+0000 7ff421600640 1 -- 172.21.15.102:0/3945973507 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40c0074e0 con 0x7ff4040047c0 2024-03-23T03:48:47.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.683+0000 7ff421600640 1 -- 172.21.15.102:0/3945973507 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40c007640 con 0x7ff4040047c0 2024-03-23T03:48:47.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.683+0000 7ff42ba00640 1 -- 172.21.15.102:0/3945973507 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff424151000 con 0x7ff4040047c0 2024-03-23T03:48:47.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.683+0000 7ff421600640 1 -- 172.21.15.102:0/3945973507 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40c004a00 con 0x7ff4040047c0 2024-03-23T03:48:47.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.683+0000 7ff42ba00640 1 -- 172.21.15.102:0/3945973507 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff424150ad0 con 0x7ff4040047c0 2024-03-23T03:48:47.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.684+0000 7ff42ba00640 1 -- 172.21.15.102:0/3945973507 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42406ac20 con 0x7ff4040047c0 2024-03-23T03:48:47.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.685+0000 7ff421600640 1 -- 172.21.15.102:0/3945973507 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff40c004c20 con 0x7ff4040047c0 2024-03-23T03:48:47.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.686+0000 7ff421600640 1 --2- 172.21.15.102:0/3945973507 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff40006ac00 0x7ff40006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:47.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.686+0000 7ff421600640 1 -- 172.21.15.102:0/3945973507 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5263+0+0 (secure 0 0 0) 0x7ff40c088960 con 0x7ff4040047c0 2024-03-23T03:48:47.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.686+0000 7ff41be00640 1 --2- 172.21.15.102:0/3945973507 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff40006ac00 0x7ff40006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:47.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.687+0000 7ff41be00640 1 --2- 172.21.15.102:0/3945973507 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff40006ac00 0x7ff40006d0c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff414033000 tx=0x7ff414009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:47.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.690+0000 7ff421600640 1 -- 172.21.15.102:0/3945973507 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff40c053cf0 con 0x7ff4040047c0 2024-03-23T03:48:47.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.956+0000 7ff42ba00640 1 -- 172.21.15.102:0/3945973507 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values"]} v 0) v1 -- 0x7ff424156680 con 0x7ff4040047c0 2024-03-23T03:48:47.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.993+0000 7ff421600640 1 -- 172.21.15.102:0/3945973507 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_disable_enable_human_readable_quota_values"]}]=0 v124) v1 ==== 160+0+0 (secure 0 0 0) 0x7ff40c018020 con 0x7ff4040047c0 2024-03-23T03:48:47.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.995+0000 7ff42ba00640 1 -- 172.21.15.102:0/3945973507 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff40006ac00 msgr2=0x7ff40006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:47.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.995+0000 7ff42ba00640 1 --2- 172.21.15.102:0/3945973507 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff40006ac00 0x7ff40006d0c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff414033000 tx=0x7ff414009030 comp rx=0 tx=0).stop 2024-03-23T03:48:47.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.995+0000 7ff42ba00640 1 -- 172.21.15.102:0/3945973507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4040047c0 msgr2=0x7ff424150590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:47.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.995+0000 7ff42ba00640 1 --2- 172.21.15.102:0/3945973507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4040047c0 0x7ff424150590 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7ff40c00c7a0 tx=0x7ff40c00cc60 comp rx=0 tx=0).stop 2024-03-23T03:48:47.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.995+0000 7ff42ba00640 1 -- 172.21.15.102:0/3945973507 shutdown_connections 2024-03-23T03:48:47.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.995+0000 7ff42ba00640 1 --2- 172.21.15.102:0/3945973507 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff40006ac00 0x7ff40006d0c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.995+0000 7ff42ba00640 1 --2- 172.21.15.102:0/3945973507 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff424154460 0x7ff4240cc670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.995+0000 7ff42ba00640 1 --2- 172.21.15.102:0/3945973507 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff424157360 0x7ff424153f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.995+0000 7ff42ba00640 1 --2- 172.21.15.102:0/3945973507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4040047c0 0x7ff424150590 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:47.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.995+0000 7ff42ba00640 1 -- 172.21.15.102:0/3945973507 >> 172.21.15.102:0/3945973507 conn(0x7ff42406a3d0 msgr2=0x7ff424158dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:47.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.996+0000 7ff42ba00640 1 -- 172.21.15.102:0/3945973507 shutdown_connections 2024-03-23T03:48:47.997 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:47.996+0000 7ff42ba00640 1 -- 172.21.15.102:0/3945973507 wait complete. 2024-03-23T03:48:48.016 INFO:tasks.cephfs_test_runner:test: ... ok 2024-03-23T03:48:48.016 INFO:tasks.cephfs_test_runner:Starting test: test_human_readable_quota_invalid_values (tasks.cephfs.test_quota.TestQuota) test invalid values for ceph.quota.max_bytes 2024-03-23T03:48:48.016 INFO:tasks.cephfs_test_runner:test_human_readable_quota_invalid_values (tasks.cephfs.test_quota.TestQuota) 2024-03-23T03:48:48.017 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values' 2024-03-23T03:48:48.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.221+0000 7fcbc1e00640 1 Processor -- start 2024-03-23T03:48:48.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.221+0000 7fcbc1e00640 1 -- start start 2024-03-23T03:48:48.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.223+0000 7fcbc1e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fcbbc13faa0 0x7fcbbc160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:48.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.223+0000 7fcbc1e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcbbc160800 0x7fcbbc0b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:48.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.223+0000 7fcbc1e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbbc0b2030 con 0x7fcbbc0b1be0 2024-03-23T03:48:48.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.223+0000 7fcbc1e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbbc0b38a0 con 0x7fcbbc13faa0 2024-03-23T03:48:48.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.223+0000 7fcbc1e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc0b3b80 con 0x7fcbbc160800 2024-03-23T03:48:48.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.224+0000 7fcbba000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcbbc160800 0x7fcbbc0b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:48.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.224+0000 7fcbbaa00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fcbbc13faa0 0x7fcbbc160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:48.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.224+0000 7fcbbaa00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fcbbc13faa0 0x7fcbbc160f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:58998/0 (socket says 172.21.15.102:58998) 2024-03-23T03:48:48.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.224+0000 7fcbbaa00640 1 -- 172.21.15.102:0/2503318845 learned_addr learned my addr 172.21.15.102:0/2503318845 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:48.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.224+0000 7fcbbb400640 1 --1- 172.21.15.102:0/2503318845 >> v1:172.21.15.173:6789/0 conn(0x7fcbbc0b1be0 0x7fcbbc1422b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:55522/0 (socket says 172.21.15.102:55522) 2024-03-23T03:48:48.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.225+0000 7fcbba000640 1 -- 172.21.15.102:0/2503318845 >> v2:172.21.15.173:3300/0 conn(0x7fcbbc13faa0 msgr2=0x7fcbbc160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:48.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.225+0000 7fcbba000640 1 --2- 172.21.15.102:0/2503318845 >> v2:172.21.15.173:3300/0 conn(0x7fcbbc13faa0 0x7fcbbc160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:48.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.225+0000 7fcbba000640 1 -- 172.21.15.102:0/2503318845 >> v1:172.21.15.173:6789/0 conn(0x7fcbbc0b1be0 legacy=0x7fcbbc1422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:48.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.225+0000 7fcbba000640 1 -- 172.21.15.102:0/2503318845 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbbc0b3e60 con 0x7fcbbc160800 2024-03-23T03:48:48.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.226+0000 7fcbba000640 1 --2- 172.21.15.102:0/2503318845 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcbbc160800 0x7fcbbc0b16a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcba80097a0 tx=0x7fcba802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eedc94d96bf21a85 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:48.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.227+0000 7fcbb9600640 1 -- 172.21.15.102:0/2503318845 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba8031830 con 0x7fcbbc160800 2024-03-23T03:48:48.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.227+0000 7fcbb9600640 1 -- 172.21.15.102:0/2503318845 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba8031990 con 0x7fcbbc160800 2024-03-23T03:48:48.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.227+0000 7fcbb9600640 1 -- 172.21.15.102:0/2503318845 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba8031d20 con 0x7fcbbc160800 2024-03-23T03:48:48.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.228+0000 7fcbc1e00640 1 -- 172.21.15.102:0/2503318845 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcbbc160800 msgr2=0x7fcbbc0b16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:48.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.228+0000 7fcbc1e00640 1 --2- 172.21.15.102:0/2503318845 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcbbc160800 0x7fcbbc0b16a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcba80097a0 tx=0x7fcba802b8d0 comp rx=0 tx=0).stop 2024-03-23T03:48:48.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.228+0000 7fcbc1e00640 1 -- 172.21.15.102:0/2503318845 shutdown_connections 2024-03-23T03:48:48.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.228+0000 7fcbc1e00640 1 --2- 172.21.15.102:0/2503318845 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcbbc160800 0x7fcbbc0b16a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:48.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.228+0000 7fcbc1e00640 1 --2- 172.21.15.102:0/2503318845 >> v2:172.21.15.173:3300/0 conn(0x7fcbbc13faa0 0x7fcbbc160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:48.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.228+0000 7fcbc1e00640 1 -- 172.21.15.102:0/2503318845 >> 172.21.15.102:0/2503318845 conn(0x7fcbbc06a3d0 msgr2=0x7fcbbc0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:48.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.228+0000 7fcbc1e00640 1 -- 172.21.15.102:0/2503318845 shutdown_connections 2024-03-23T03:48:48.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.228+0000 7fcbc1e00640 1 -- 172.21.15.102:0/2503318845 wait complete. 2024-03-23T03:48:48.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.230+0000 7fcbc1e00640 1 Processor -- start 2024-03-23T03:48:48.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.230+0000 7fcbc1e00640 1 -- start start 2024-03-23T03:48:48.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.231+0000 7fcbc1e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcbbc0b1be0 0x7fcbbc1f0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:48.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.231+0000 7fcbc1e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbbc13faa0 0x7fcbbc13cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:48.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.231+0000 7fcbc1e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbbc160800 0x7fcbbc1f3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:48.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.231+0000 7fcbc1e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc1f4ec0 con 0x7fcbbc160800 2024-03-23T03:48:48.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.231+0000 7fcbc1e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc1f42b0 con 0x7fcbbc13faa0 2024-03-23T03:48:48.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.231+0000 7fcbc1e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbbc1f4590 con 0x7fcbbc0b1be0 2024-03-23T03:48:48.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.231+0000 7fcbbb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbbc160800 0x7fcbbc1f3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:48.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.231+0000 7fcbbb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbbc160800 0x7fcbbc1f3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46762/0 (socket says 172.21.15.102:46762) 2024-03-23T03:48:48.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.231+0000 7fcbbb400640 1 -- 172.21.15.102:0/1748417909 learned_addr learned my addr 172.21.15.102:0/1748417909 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:48.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.231+0000 7fcbbaa00640 1 --2- 172.21.15.102:0/1748417909 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcbbc0b1be0 0x7fcbbc1f0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:48.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.231+0000 7fcbba000640 1 --2- 172.21.15.102:0/1748417909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbbc13faa0 0x7fcbbc13cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:48.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.232+0000 7fcbbb400640 1 -- 172.21.15.102:0/1748417909 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcbbc0b1be0 msgr2=0x7fcbbc1f0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:48.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.232+0000 7fcbbb400640 1 --2- 172.21.15.102:0/1748417909 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcbbc0b1be0 0x7fcbbc1f0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:48.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.232+0000 7fcbbb400640 1 -- 172.21.15.102:0/1748417909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbbc13faa0 msgr2=0x7fcbbc13cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:48.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.232+0000 7fcbbb400640 1 --2- 172.21.15.102:0/1748417909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbbc13faa0 0x7fcbbc13cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:48.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.232+0000 7fcbbb400640 1 -- 172.21.15.102:0/1748417909 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba8009400 con 0x7fcbbc160800 2024-03-23T03:48:48.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.232+0000 7fcbbaa00640 1 --2- 172.21.15.102:0/1748417909 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcbbc0b1be0 0x7fcbbc1f0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:48:48.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.232+0000 7fcbbb400640 1 --2- 172.21.15.102:0/1748417909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbbc160800 0x7fcbbc1f3720 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fcbb400c730 tx=0x7fcbb400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:48.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.232+0000 7fcbb3400640 1 -- 172.21.15.102:0/1748417909 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb400ce40 con 0x7fcbbc160800 2024-03-23T03:48:48.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.232+0000 7fcbb3400640 1 -- 172.21.15.102:0/1748417909 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb400a640 con 0x7fcbbc160800 2024-03-23T03:48:48.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.232+0000 7fcbb3400640 1 -- 172.21.15.102:0/1748417909 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb400a9f0 con 0x7fcbbc160800 2024-03-23T03:48:48.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.232+0000 7fcbc1e00640 1 -- 172.21.15.102:0/1748417909 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbbc1f4870 con 0x7fcbbc160800 2024-03-23T03:48:48.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.233+0000 7fcbc1e00640 1 -- 172.21.15.102:0/1748417909 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbbc0ac560 con 0x7fcbbc160800 2024-03-23T03:48:48.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.234+0000 7fcbc1e00640 1 -- 172.21.15.102:0/1748417909 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbbc1f5030 con 0x7fcbbc160800 2024-03-23T03:48:48.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.234+0000 7fcbb3400640 1 -- 172.21.15.102:0/1748417909 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcbb400ab50 con 0x7fcbbc160800 2024-03-23T03:48:48.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.235+0000 7fcbb3400640 1 --2- 172.21.15.102:0/1748417909 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcb9c06ac30 0x7fcb9c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:48.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.235+0000 7fcbb3400640 1 -- 172.21.15.102:0/1748417909 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fcbb40886c0 con 0x7fcbbc160800 2024-03-23T03:48:48.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.236+0000 7fcbbaa00640 1 --2- 172.21.15.102:0/1748417909 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcb9c06ac30 0x7fcb9c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:48.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.236+0000 7fcbbaa00640 1 --2- 172.21.15.102:0/1748417909 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcb9c06ac30 0x7fcb9c06d0f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc157540 tx=0x7fcbac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:48.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.240+0000 7fcbb3400640 1 -- 172.21.15.102:0/1748417909 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcbb4053a70 con 0x7fcbbc160800 2024-03-23T03:48:48.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.513+0000 7fcbc1e00640 1 -- 172.21.15.102:0/1748417909 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values"]} v 0) v1 -- 0x7fcbbc0b2030 con 0x7fcbbc160800 2024-03-23T03:48:48.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.995+0000 7fcbb3400640 1 -- 172.21.15.102:0/1748417909 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values"]}]=0 v125) v1 ==== 156+0+0 (secure 0 0 0) 0x7fcbbc0b2030 con 0x7fcbbc160800 2024-03-23T03:48:48.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.998+0000 7fcbc1e00640 1 -- 172.21.15.102:0/1748417909 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcb9c06ac30 msgr2=0x7fcb9c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:49.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.998+0000 7fcbc1e00640 1 --2- 172.21.15.102:0/1748417909 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcb9c06ac30 0x7fcb9c06d0f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc157540 tx=0x7fcbac008040 comp rx=0 tx=0).stop 2024-03-23T03:48:49.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.998+0000 7fcbc1e00640 1 -- 172.21.15.102:0/1748417909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbbc160800 msgr2=0x7fcbbc1f3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:49.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.998+0000 7fcbc1e00640 1 --2- 172.21.15.102:0/1748417909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbbc160800 0x7fcbbc1f3720 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fcbb400c730 tx=0x7fcbb400cbf0 comp rx=0 tx=0).stop 2024-03-23T03:48:49.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.998+0000 7fcbc1e00640 1 -- 172.21.15.102:0/1748417909 shutdown_connections 2024-03-23T03:48:49.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.999+0000 7fcbc1e00640 1 --2- 172.21.15.102:0/1748417909 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcb9c06ac30 0x7fcb9c06d0f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:49.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.999+0000 7fcbc1e00640 1 --2- 172.21.15.102:0/1748417909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbbc160800 0x7fcbbc1f3720 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:49.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.999+0000 7fcbc1e00640 1 --2- 172.21.15.102:0/1748417909 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcbbc13faa0 0x7fcbbc13cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:49.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.999+0000 7fcbc1e00640 1 --2- 172.21.15.102:0/1748417909 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcbbc0b1be0 0x7fcbbc1f0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:49.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.999+0000 7fcbc1e00640 1 -- 172.21.15.102:0/1748417909 >> 172.21.15.102:0/1748417909 conn(0x7fcbbc06a3d0 msgr2=0x7fcbbc13d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:49.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.999+0000 7fcbc1e00640 1 -- 172.21.15.102:0/1748417909 shutdown_connections 2024-03-23T03:48:49.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:48.999+0000 7fcbc1e00640 1 -- 172.21.15.102:0/1748417909 wait complete. 2024-03-23T03:48:49.019 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-23T03:48:49.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.228+0000 7f55d0200640 1 Processor -- start 2024-03-23T03:48:49.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.228+0000 7f55d0200640 1 -- start start 2024-03-23T03:48:49.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.230+0000 7f55d0200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f55c80b1b60 0x7f55c806baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:49.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.230+0000 7f55d0200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c806b0a0 con 0x7f55c815cd10 2024-03-23T03:48:49.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.230+0000 7f55d0200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55c80d2330 con 0x7f55c80cd6b0 2024-03-23T03:48:49.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.230+0000 7f55d0200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55c80d2610 con 0x7f55c80b1b60 2024-03-23T03:48:49.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.230+0000 7f55cce00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f55c815cd10 0x7f55c80d1ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35948/0 (socket says 172.21.15.102:35948) 2024-03-23T03:48:49.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.230+0000 7f55cce00640 1 -- 172.21.15.102:0/632057616 learned_addr learned my addr 172.21.15.102:0/632057616 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:49.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.230+0000 7f55ce200640 1 --2- 172.21.15.102:0/632057616 >> v2:172.21.15.173:3300/0 conn(0x7f55c80b1b60 0x7f55c806baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:49.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.231+0000 7f55c7e00640 1 -- 172.21.15.102:0/632057616 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191137796 0 0) 0x7f55c806b0a0 con 0x7f55c815cd10 2024-03-23T03:48:49.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.231+0000 7f55c7e00640 1 -- 172.21.15.102:0/632057616 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b0003520 con 0x7f55c815cd10 2024-03-23T03:48:49.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.231+0000 7f55c7e00640 1 -- 172.21.15.102:0/632057616 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1756700318 0 0) 0x7f55b0003520 con 0x7f55c815cd10 2024-03-23T03:48:49.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.231+0000 7f55c7e00640 1 -- 172.21.15.102:0/632057616 >> v2:172.21.15.173:3300/0 conn(0x7f55c80b1b60 msgr2=0x7f55c806baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:49.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.231+0000 7f55c7e00640 1 --2- 172.21.15.102:0/632057616 >> v2:172.21.15.173:3300/0 conn(0x7f55c80b1b60 0x7f55c806baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:49.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.231+0000 7f55c7e00640 1 -- 172.21.15.102:0/632057616 >> v1:172.21.15.173:6789/0 conn(0x7f55c80cd6b0 legacy=0x7f55c80cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:49.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.231+0000 7f55c7e00640 1 -- 172.21.15.102:0/632057616 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c806b2f0 con 0x7f55c815cd10 2024-03-23T03:48:49.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.231+0000 7f55c7e00640 1 -- 172.21.15.102:0/632057616 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f55b8001670 con 0x7f55c815cd10 2024-03-23T03:48:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.231+0000 7f55c7e00640 1 -- 172.21.15.102:0/632057616 >> v1:172.21.15.102:6789/0 conn(0x7f55c815cd10 legacy=0x7f55c80d1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.231+0000 7f55c7e00640 1 --2- 172.21.15.102:0/632057616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b00039c0 0x7f55b0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.231+0000 7f55c7e00640 1 --2- 172.21.15.102:0/632057616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55b00047c0 0x7f55b0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.231+0000 7f55c7e00640 1 --2- 172.21.15.102:0/632057616 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55b0005180 0x7f55b000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.231+0000 7f55c7e00640 1 -- 172.21.15.102:0/632057616 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c80d2610 con 0x7f55b00039c0 2024-03-23T03:48:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.231+0000 7f55c7e00640 1 -- 172.21.15.102:0/632057616 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b000c080 con 0x7f55b00047c0 2024-03-23T03:48:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.231+0000 7f55c7e00640 1 -- 172.21.15.102:0/632057616 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b000c240 con 0x7f55b0005180 2024-03-23T03:48:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.232+0000 7f55cd800640 1 --2- 172.21.15.102:0/632057616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b00039c0 0x7f55b0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.232+0000 7f55cce00640 1 --2- 172.21.15.102:0/632057616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55b00047c0 0x7f55b0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.232+0000 7f55ce200640 1 --2- 172.21.15.102:0/632057616 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55b0005180 0x7f55b000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.232+0000 7f55cd800640 1 -- 172.21.15.102:0/632057616 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55b0005180 msgr2=0x7f55b000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:49.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.232+0000 7f55cd800640 1 --2- 172.21.15.102:0/632057616 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55b0005180 0x7f55b000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:49.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.232+0000 7f55cd800640 1 -- 172.21.15.102:0/632057616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55b00047c0 msgr2=0x7f55b0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:49.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.232+0000 7f55cd800640 1 --2- 172.21.15.102:0/632057616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55b00047c0 0x7f55b0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:49.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.232+0000 7f55cd800640 1 -- 172.21.15.102:0/632057616 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b0003520 con 0x7f55b00039c0 2024-03-23T03:48:49.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.232+0000 7f55ce200640 1 --2- 172.21.15.102:0/632057616 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55b0005180 0x7f55b000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:49.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.232+0000 7f55ce200640 1 -- 172.21.15.102:0/632057616 reap_dead start 2024-03-23T03:48:49.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.234+0000 7f55cd800640 1 --2- 172.21.15.102:0/632057616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b00039c0 0x7f55b0003de0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f55bc001750 tx=0x7f55b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acd64a7b1e113708 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:49.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.234+0000 7f55c7e00640 1 -- 172.21.15.102:0/632057616 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55bc0052d0 con 0x7f55b00039c0 2024-03-23T03:48:49.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.234+0000 7f55c7e00640 1 -- 172.21.15.102:0/632057616 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55bc02e050 con 0x7f55b00039c0 2024-03-23T03:48:49.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.234+0000 7f55c7e00640 1 -- 172.21.15.102:0/632057616 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55bc003830 con 0x7f55b00039c0 2024-03-23T03:48:49.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.234+0000 7f55d0200640 1 -- 172.21.15.102:0/632057616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b00039c0 msgr2=0x7f55b0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:49.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.235+0000 7f55d0200640 1 --2- 172.21.15.102:0/632057616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b00039c0 0x7f55b0003de0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f55bc001750 tx=0x7f55b00057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:49.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.235+0000 7f55d0200640 1 -- 172.21.15.102:0/632057616 shutdown_connections 2024-03-23T03:48:49.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.235+0000 7f55d0200640 1 --2- 172.21.15.102:0/632057616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b00039c0 0x7f55b0003de0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:49.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.235+0000 7f55d0200640 1 -- 172.21.15.102:0/632057616 >> 172.21.15.102:0/632057616 conn(0x7f55c80cf5d0 msgr2=0x7f55c80cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:49.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.235+0000 7f55d0200640 1 -- 172.21.15.102:0/632057616 shutdown_connections 2024-03-23T03:48:49.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.235+0000 7f55d0200640 1 -- 172.21.15.102:0/632057616 wait complete. 2024-03-23T03:48:49.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.237+0000 7f55d0200640 1 Processor -- start 2024-03-23T03:48:49.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.237+0000 7f55d0200640 1 -- start start 2024-03-23T03:48:49.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.238+0000 7f55d0200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b00039c0 0x7f55c815fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:49.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.238+0000 7f55d0200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55c8151d80 0x7f55c8158640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:49.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.238+0000 7f55d0200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55c8158b80 0x7f55c81f5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:49.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.238+0000 7f55d0200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c80caea0 con 0x7f55b00039c0 2024-03-23T03:48:49.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.238+0000 7f55d0200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c8156b30 con 0x7f55c8151d80 2024-03-23T03:48:49.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.238+0000 7f55d0200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55c8156dd0 con 0x7f55c8158b80 2024-03-23T03:48:49.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.238+0000 7f55cd800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b00039c0 0x7f55c815fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:49.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.238+0000 7f55cd800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b00039c0 0x7f55c815fb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46780/0 (socket says 172.21.15.102:46780) 2024-03-23T03:48:49.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.238+0000 7f55cd800640 1 -- 172.21.15.102:0/1642907167 learned_addr learned my addr 172.21.15.102:0/1642907167 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:49.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.239+0000 7f55cce00640 1 --2- 172.21.15.102:0/1642907167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55c8151d80 0x7f55c8158640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:49.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.239+0000 7f55cd800640 1 -- 172.21.15.102:0/1642907167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55c8158b80 msgr2=0x7f55c81f5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:49.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.239+0000 7f55cd800640 1 --2- 172.21.15.102:0/1642907167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55c8158b80 0x7f55c81f5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:49.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.239+0000 7f55cd800640 1 -- 172.21.15.102:0/1642907167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55c8151d80 msgr2=0x7f55c8158640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:49.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.239+0000 7f55cd800640 1 --2- 172.21.15.102:0/1642907167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55c8151d80 0x7f55c8158640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:49.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.239+0000 7f55cd800640 1 -- 172.21.15.102:0/1642907167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c81570b0 con 0x7f55b00039c0 2024-03-23T03:48:49.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.239+0000 7f55cce00640 1 --2- 172.21.15.102:0/1642907167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55c8151d80 0x7f55c8158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:49.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.239+0000 7f55cd800640 1 --2- 172.21.15.102:0/1642907167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b00039c0 0x7f55c815fb20 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f55bc005040 tx=0x7f55bc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:49.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.239+0000 7f55c5600640 1 -- 172.21.15.102:0/1642907167 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55bc007640 con 0x7f55b00039c0 2024-03-23T03:48:49.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.239+0000 7f55c5600640 1 -- 172.21.15.102:0/1642907167 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55bc0069e0 con 0x7f55b00039c0 2024-03-23T03:48:49.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.239+0000 7f55d0200640 1 -- 172.21.15.102:0/1642907167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55c80cab30 con 0x7f55b00039c0 2024-03-23T03:48:49.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.239+0000 7f55c5600640 1 -- 172.21.15.102:0/1642907167 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55bc006d90 con 0x7f55b00039c0 2024-03-23T03:48:49.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.240+0000 7f55d0200640 1 -- 172.21.15.102:0/1642907167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55c815dd90 con 0x7f55b00039c0 2024-03-23T03:48:49.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.241+0000 7f55d0200640 1 -- 172.21.15.102:0/1642907167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55c80cacc0 con 0x7f55b00039c0 2024-03-23T03:48:49.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.242+0000 7f55c5600640 1 -- 172.21.15.102:0/1642907167 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f55bc0035f0 con 0x7f55b00039c0 2024-03-23T03:48:49.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.243+0000 7f55c5600640 1 --2- 172.21.15.102:0/1642907167 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f55ac06ac00 0x7f55ac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:49.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.243+0000 7f55c5600640 1 -- 172.21.15.102:0/1642907167 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f55bc0aae60 con 0x7f55b00039c0 2024-03-23T03:48:49.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.243+0000 7f55cce00640 1 --2- 172.21.15.102:0/1642907167 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f55ac06ac00 0x7f55ac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:49.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.244+0000 7f55cce00640 1 --2- 172.21.15.102:0/1642907167 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f55ac06ac00 0x7f55ac06d0c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f55c80cde50 tx=0x7f55b8000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:49.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.247+0000 7f55c5600640 1 -- 172.21.15.102:0/1642907167 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f55bc076160 con 0x7f55b00039c0 2024-03-23T03:48:49.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.439+0000 7f55d0200640 1 -- 172.21.15.102:0/1642907167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f55c8073390 con 0x7f55b00039c0 2024-03-23T03:48:49.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.446+0000 7f55c5600640 1 -- 172.21.15.102:0/1642907167 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f55bc04b0c0 con 0x7f55b00039c0 2024-03-23T03:48:49.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.448+0000 7f55d0200640 1 -- 172.21.15.102:0/1642907167 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f55ac06ac00 msgr2=0x7f55ac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:49.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.449+0000 7f55d0200640 1 --2- 172.21.15.102:0/1642907167 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f55ac06ac00 0x7f55ac06d0c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f55c80cde50 tx=0x7f55b8000ed0 comp rx=0 tx=0).stop 2024-03-23T03:48:49.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.449+0000 7f55d0200640 1 -- 172.21.15.102:0/1642907167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b00039c0 msgr2=0x7f55c815fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:49.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.449+0000 7f55d0200640 1 --2- 172.21.15.102:0/1642907167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b00039c0 0x7f55c815fb20 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f55bc005040 tx=0x7f55bc001120 comp rx=0 tx=0).stop 2024-03-23T03:48:49.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.449+0000 7f55d0200640 1 -- 172.21.15.102:0/1642907167 shutdown_connections 2024-03-23T03:48:49.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.449+0000 7f55d0200640 1 --2- 172.21.15.102:0/1642907167 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f55ac06ac00 0x7f55ac06d0c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:49.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.449+0000 7f55d0200640 1 --2- 172.21.15.102:0/1642907167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55c8158b80 0x7f55c81f5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:49.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.449+0000 7f55d0200640 1 --2- 172.21.15.102:0/1642907167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55c8151d80 0x7f55c8158640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:49.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.449+0000 7f55d0200640 1 --2- 172.21.15.102:0/1642907167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b00039c0 0x7f55c815fb20 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:49.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.449+0000 7f55d0200640 1 -- 172.21.15.102:0/1642907167 >> 172.21.15.102:0/1642907167 conn(0x7f55c80cf5d0 msgr2=0x7f55c80b6790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:49.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.449+0000 7f55d0200640 1 -- 172.21.15.102:0/1642907167 shutdown_connections 2024-03-23T03:48:49.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:49.449+0000 7f55d0200640 1 -- 172.21.15.102:0/1642907167 wait complete. 2024-03-23T03:48:49.469 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T03:48:49.469 DEBUG:teuthology.orchestra.run.smithi102:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:48:49.580 DEBUG:teuthology.orchestra.run.smithi173:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:48:49.699 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T03:48:49.699 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:48:49.699 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:48:49.756 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:48:49.756 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:48:49.756 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:48:49.813 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:48:49.813 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:48:49.813 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:48:49.821 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:48:49.867 DEBUG:teuthology.orchestra.run.smithi173:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:48:49.870 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:48:49.870 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:48:49.870 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:48:49.964 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:48:49.964 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-23T03:48:49.964 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:48:50.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.173+0000 7f4f7e200640 1 Processor -- start 2024-03-23T03:48:50.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.173+0000 7f4f7e200640 1 -- start start 2024-03-23T03:48:50.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.175+0000 7f4f7e200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f4f780b40c0 0x7f4f7815b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:50.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.175+0000 7f4f7e200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f7806e460 0x7f4f78158e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:50.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.175+0000 7f4f7e200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f7815bea0 con 0x7f4f781442e0 2024-03-23T03:48:50.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.175+0000 7f4f7e200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f7806c280 con 0x7f4f780b40c0 2024-03-23T03:48:50.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.175+0000 7f4f7e200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7806c560 con 0x7f4f7806e460 2024-03-23T03:48:50.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.175+0000 7f4f77400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4f781442e0 0x7f4f780b39b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35958/0 (socket says 172.21.15.102:35958) 2024-03-23T03:48:50.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.175+0000 7f4f77400640 1 -- 172.21.15.102:0/1294841168 learned_addr learned my addr 172.21.15.102:0/1294841168 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:50.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.175+0000 7f4f76000640 1 -- 172.21.15.102:0/1294841168 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610231476 0 0) 0x7f4f68001490 con 0x7f4f781442e0 2024-03-23T03:48:50.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.175+0000 7f4f76000640 1 -- 172.21.15.102:0/1294841168 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f5c003520 con 0x7f4f781442e0 2024-03-23T03:48:50.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.175+0000 7f4f76a00640 1 --2- 172.21.15.102:0/1294841168 >> v2:172.21.15.173:3300/0 conn(0x7f4f780b40c0 0x7f4f7815b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:50.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.175+0000 7f4f77e00640 1 --2- 172.21.15.102:0/1294841168 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f7806e460 0x7f4f78158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:50.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.176+0000 7f4f76000640 1 -- 172.21.15.102:0/1294841168 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3684657079 0 0) 0x7f4f5c003520 con 0x7f4f781442e0 2024-03-23T03:48:50.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.176+0000 7f4f76000640 1 -- 172.21.15.102:0/1294841168 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f7806e460 msgr2=0x7f4f78158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:50.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.176+0000 7f4f76000640 1 --2- 172.21.15.102:0/1294841168 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f7806e460 0x7f4f78158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:50.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.176+0000 7f4f76000640 1 -- 172.21.15.102:0/1294841168 >> v2:172.21.15.173:3300/0 conn(0x7f4f780b40c0 msgr2=0x7f4f7815b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:50.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.176+0000 7f4f76000640 1 --2- 172.21.15.102:0/1294841168 >> v2:172.21.15.173:3300/0 conn(0x7f4f780b40c0 0x7f4f7815b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:50.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.176+0000 7f4f76000640 1 -- 172.21.15.102:0/1294841168 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7806c840 con 0x7f4f781442e0 2024-03-23T03:48:50.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.176+0000 7f4f76000640 1 -- 172.21.15.102:0/1294841168 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f4f68001ba0 con 0x7f4f781442e0 2024-03-23T03:48:50.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.176+0000 7f4f77e00640 1 --2- 172.21.15.102:0/1294841168 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f7806e460 0x7f4f78158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:48:50.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.176+0000 7f4f76000640 1 -- 172.21.15.102:0/1294841168 >> v1:172.21.15.102:6789/0 conn(0x7f4f781442e0 legacy=0x7f4f780b39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:50.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.176+0000 7f4f76000640 1 --2- 172.21.15.102:0/1294841168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f5c003970 0x7f4f5c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:50.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.176+0000 7f4f76000640 1 --2- 172.21.15.102:0/1294841168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f5c0047c0 0x7f4f5c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:50.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.176+0000 7f4f76000640 1 --2- 172.21.15.102:0/1294841168 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f7806e460 0x7f4f5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:50.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.176+0000 7f4f76000640 1 -- 172.21.15.102:0/1294841168 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7806c280 con 0x7f4f5c003970 2024-03-23T03:48:50.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.176+0000 7f4f76000640 1 -- 172.21.15.102:0/1294841168 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7806c560 con 0x7f4f5c0047c0 2024-03-23T03:48:50.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.176+0000 7f4f76000640 1 -- 172.21.15.102:0/1294841168 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f5c00b9b0 con 0x7f4f7806e460 2024-03-23T03:48:50.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.176+0000 7f4f77400640 1 --2- 172.21.15.102:0/1294841168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f5c003970 0x7f4f5c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:50.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.177+0000 7f4f77e00640 1 --2- 172.21.15.102:0/1294841168 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f7806e460 0x7f4f5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:50.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.177+0000 7f4f76a00640 1 --2- 172.21.15.102:0/1294841168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f5c0047c0 0x7f4f5c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:50.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.177+0000 7f4f77400640 1 -- 172.21.15.102:0/1294841168 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f7806e460 msgr2=0x7f4f5c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:50.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.177+0000 7f4f77400640 1 --2- 172.21.15.102:0/1294841168 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f7806e460 0x7f4f5c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:50.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.177+0000 7f4f77400640 1 -- 172.21.15.102:0/1294841168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f5c0047c0 msgr2=0x7f4f5c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:50.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.177+0000 7f4f77400640 1 --2- 172.21.15.102:0/1294841168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f5c0047c0 0x7f4f5c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:50.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.177+0000 7f4f77400640 1 -- 172.21.15.102:0/1294841168 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5c003520 con 0x7f4f5c003970 2024-03-23T03:48:50.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.178+0000 7f4f77400640 1 --2- 172.21.15.102:0/1294841168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f5c003970 0x7f4f5c003d90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800a780 tx=0x7f4f5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad68bc04387f2438 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:50.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.178+0000 7f4f76000640 1 -- 172.21.15.102:0/1294841168 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f680072f0 con 0x7f4f5c003970 2024-03-23T03:48:50.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.178+0000 7f4f76000640 1 -- 172.21.15.102:0/1294841168 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6802f070 con 0x7f4f5c003970 2024-03-23T03:48:50.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.178+0000 7f4f76000640 1 -- 172.21.15.102:0/1294841168 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f68007690 con 0x7f4f5c003970 2024-03-23T03:48:50.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.178+0000 7f4f7e200640 1 -- 172.21.15.102:0/1294841168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f5c003970 msgr2=0x7f4f5c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:50.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.178+0000 7f4f7e200640 1 --2- 172.21.15.102:0/1294841168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f5c003970 0x7f4f5c003d90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4f6800a780 tx=0x7f4f5c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:50.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.178+0000 7f4f77e00640 1 -- 172.21.15.102:0/1294841168 reap_dead start 2024-03-23T03:48:50.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.179+0000 7f4f7e200640 1 -- 172.21.15.102:0/1294841168 shutdown_connections 2024-03-23T03:48:50.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.179+0000 7f4f7e200640 1 -- 172.21.15.102:0/1294841168 >> 172.21.15.102:0/1294841168 conn(0x7f4f780b03a0 msgr2=0x7f4f780b0780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:50.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.179+0000 7f4f7e200640 1 -- 172.21.15.102:0/1294841168 shutdown_connections 2024-03-23T03:48:50.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.179+0000 7f4f7e200640 1 -- 172.21.15.102:0/1294841168 wait complete. 2024-03-23T03:48:50.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.180+0000 7f4f7e200640 1 Processor -- start 2024-03-23T03:48:50.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.181+0000 7f4f7e200640 1 -- start start 2024-03-23T03:48:50.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.181+0000 7f4f7e200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f5c003970 0x7f4f78133850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:50.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.181+0000 7f4f7e200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f78141540 0x7f4f781419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:50.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.181+0000 7f4f7e200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f780ae530 0x7f4f780ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:50.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.182+0000 7f4f7e200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f780aef20 con 0x7f4f780ae530 2024-03-23T03:48:50.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.182+0000 7f4f7e200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f78158990 con 0x7f4f78141540 2024-03-23T03:48:50.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.182+0000 7f4f7e200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f78142000 con 0x7f4f5c003970 2024-03-23T03:48:50.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.182+0000 7f4f77e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f780ae530 0x7f4f780ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:50.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.182+0000 7f4f77e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f780ae530 0x7f4f780ae9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46808/0 (socket says 172.21.15.102:46808) 2024-03-23T03:48:50.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.182+0000 7f4f77e00640 1 -- 172.21.15.102:0/3064078000 learned_addr learned my addr 172.21.15.102:0/3064078000 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:50.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.182+0000 7f4f76a00640 1 --2- 172.21.15.102:0/3064078000 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f78141540 0x7f4f781419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:50.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.182+0000 7f4f77400640 1 --2- 172.21.15.102:0/3064078000 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f5c003970 0x7f4f78133850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:50.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.182+0000 7f4f77e00640 1 -- 172.21.15.102:0/3064078000 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f5c003970 msgr2=0x7f4f78133850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:50.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.182+0000 7f4f77e00640 1 --2- 172.21.15.102:0/3064078000 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f5c003970 0x7f4f78133850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:50.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.182+0000 7f4f77e00640 1 -- 172.21.15.102:0/3064078000 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f78141540 msgr2=0x7f4f781419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:50.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.182+0000 7f4f77e00640 1 --2- 172.21.15.102:0/3064078000 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f78141540 0x7f4f781419a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:50.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.182+0000 7f4f77e00640 1 -- 172.21.15.102:0/3064078000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f781422e0 con 0x7f4f780ae530 2024-03-23T03:48:50.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.183+0000 7f4f77e00640 1 --2- 172.21.15.102:0/3064078000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f780ae530 0x7f4f780ae9e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f4f780cabf0 tx=0x7f4f7000df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:50.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.183+0000 7f4f6f400640 1 -- 172.21.15.102:0/3064078000 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f70004230 con 0x7f4f780ae530 2024-03-23T03:48:50.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.183+0000 7f4f6f400640 1 -- 172.21.15.102:0/3064078000 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7000c070 con 0x7f4f780ae530 2024-03-23T03:48:50.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.183+0000 7f4f6f400640 1 -- 172.21.15.102:0/3064078000 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f70003610 con 0x7f4f780ae530 2024-03-23T03:48:50.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.183+0000 7f4f7e200640 1 -- 172.21.15.102:0/3064078000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f781425c0 con 0x7f4f780ae530 2024-03-23T03:48:50.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.183+0000 7f4f7e200640 1 -- 172.21.15.102:0/3064078000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f78142b80 con 0x7f4f780ae530 2024-03-23T03:48:50.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.184+0000 7f4f7e200640 1 -- 172.21.15.102:0/3064078000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f78073390 con 0x7f4f780ae530 2024-03-23T03:48:50.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.185+0000 7f4f6f400640 1 -- 172.21.15.102:0/3064078000 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4f70003770 con 0x7f4f780ae530 2024-03-23T03:48:50.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.186+0000 7f4f6f400640 1 --2- 172.21.15.102:0/3064078000 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f5806ac30 0x7f4f5806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:50.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.186+0000 7f4f6f400640 1 -- 172.21.15.102:0/3064078000 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f4f70088df0 con 0x7f4f780ae530 2024-03-23T03:48:50.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.187+0000 7f4f77400640 1 --2- 172.21.15.102:0/3064078000 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f5806ac30 0x7f4f5806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:50.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.187+0000 7f4f77400640 1 --2- 172.21.15.102:0/3064078000 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f5806ac30 0x7f4f5806d0f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4f68034000 tx=0x7f4f68009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:50.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.191+0000 7f4f6f400640 1 -- 172.21.15.102:0/3064078000 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4f70054190 con 0x7f4f780ae530 2024-03-23T03:48:50.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.425+0000 7f4f7e200640 1 -- 172.21.15.102:0/3064078000 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f780ccb40 con 0x7f4f780ae530 2024-03-23T03:48:50.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.426+0000 7f4f6f400640 1 -- 172.21.15.102:0/3064078000 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+3541 (secure 0 0 0) 0x7f4f700577b0 con 0x7f4f780ae530 2024-03-23T03:48:50.427 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:50.427 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4652,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6834/1537100581","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1537100581},{"type":"v1","addr":"172.21.15.173:6834","nonce":1537100581}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4655,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3206638952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":3206638952},{"type":"v1","addr":"172.21.15.173:6835","nonce":3206638952}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4852,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4005240382","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4005240382},{"type":"v1","addr":"172.21.15.102:6837","nonce":4005240382}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":5607,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/2107458594","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2107458594},{"type":"v1","addr":"172.21.15.102:6836","nonce":2107458594}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[]} 2024-03-23T03:48:50.427 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 18 2024-03-23T03:48:50.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.428+0000 7f4f7e200640 1 -- 172.21.15.102:0/3064078000 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f5806ac30 msgr2=0x7f4f5806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:50.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.428+0000 7f4f7e200640 1 --2- 172.21.15.102:0/3064078000 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f5806ac30 0x7f4f5806d0f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4f68034000 tx=0x7f4f68009030 comp rx=0 tx=0).stop 2024-03-23T03:48:50.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.428+0000 7f4f7e200640 1 -- 172.21.15.102:0/3064078000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f780ae530 msgr2=0x7f4f780ae9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:50.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.428+0000 7f4f7e200640 1 --2- 172.21.15.102:0/3064078000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f780ae530 0x7f4f780ae9e0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f4f780cabf0 tx=0x7f4f7000df90 comp rx=0 tx=0).stop 2024-03-23T03:48:50.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.429+0000 7f4f7e200640 1 -- 172.21.15.102:0/3064078000 shutdown_connections 2024-03-23T03:48:50.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.429+0000 7f4f7e200640 1 --2- 172.21.15.102:0/3064078000 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f5806ac30 0x7f4f5806d0f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:50.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.429+0000 7f4f7e200640 1 --2- 172.21.15.102:0/3064078000 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f780ae530 0x7f4f780ae9e0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:50.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.429+0000 7f4f7e200640 1 --2- 172.21.15.102:0/3064078000 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f78141540 0x7f4f781419a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:50.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.429+0000 7f4f7e200640 1 --2- 172.21.15.102:0/3064078000 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f5c003970 0x7f4f78133850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:50.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.429+0000 7f4f7e200640 1 -- 172.21.15.102:0/3064078000 >> 172.21.15.102:0/3064078000 conn(0x7f4f780b03a0 msgr2=0x7f4f7815cf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:50.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.429+0000 7f4f7e200640 1 -- 172.21.15.102:0/3064078000 shutdown_connections 2024-03-23T03:48:50.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.429+0000 7f4f7e200640 1 -- 172.21.15.102:0/3064078000 wait complete. 2024-03-23T03:48:50.448 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T03:48:50.448 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-23T03:48:50.448 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T03:48:50.448 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:48:50.448 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T03:48:50.448 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-23T03:48:50.448 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T03:48:50.449 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:48:50.449 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T03:48:50.449 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-23T03:48:50.449 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T03:48:50.449 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:48:50.449 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T03:48:50.449 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-23T03:48:50.449 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T03:48:50.450 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:48:50.466 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T03:48:50.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T03:48:50.472 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T03:48:50.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T03:48:50.477 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T03:48:50.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T03:48:50.478 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T03:48:50.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T03:48:50.518 INFO:tasks.ceph.mds.c:Started 2024-03-23T03:48:50.519 DEBUG:teuthology.parallel:result is None 2024-03-23T03:48:50.524 INFO:tasks.ceph.mds.d:Started 2024-03-23T03:48:50.524 DEBUG:teuthology.parallel:result is None 2024-03-23T03:48:50.525 INFO:tasks.ceph.mds.a:Started 2024-03-23T03:48:50.525 DEBUG:teuthology.parallel:result is None 2024-03-23T03:48:50.530 INFO:tasks.ceph.mds.b:Started 2024-03-23T03:48:50.530 DEBUG:teuthology.parallel:result is None 2024-03-23T03:48:50.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-23T03:48:50.672 INFO:tasks.ceph.mds.c.smithi102.stdout:starting mds.c at 2024-03-23T03:48:50.675 INFO:tasks.ceph.mds.a.smithi102.stdout:starting mds.a at 2024-03-23T03:48:50.693 INFO:tasks.ceph.mds.b.smithi173.stdout:starting mds.b at 2024-03-23T03:48:50.693 INFO:tasks.ceph.mds.d.smithi173.stdout:starting mds.d at 2024-03-23T03:48:50.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.719+0000 7fd0a1600640 1 Processor -- start 2024-03-23T03:48:50.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.720+0000 7fd0a1600640 1 -- start start 2024-03-23T03:48:50.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.721+0000 7fd0a1600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd09c0cd6b0 0x7fd09c0b8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:50.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.721+0000 7fd0a1600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd09c15cd90 0x7fd09c06b060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:50.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.721+0000 7fd0a1600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd09c06b5a0 con 0x7fd09c0cda90 2024-03-23T03:48:50.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.721+0000 7fd0a1600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09c06c0e0 con 0x7fd09c15cd90 2024-03-23T03:48:50.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.721+0000 7fd0a1600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09c06b840 con 0x7fd09c0cd6b0 2024-03-23T03:48:50.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.721+0000 7fd09b400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd09c0cda90 0x7fd09c1606c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35972/0 (socket says 172.21.15.102:35972) 2024-03-23T03:48:50.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.721+0000 7fd09b400640 1 -- 172.21.15.102:0/89457398 learned_addr learned my addr 172.21.15.102:0/89457398 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:50.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.721+0000 7fd092000640 1 --2- 172.21.15.102:0/89457398 >> v2:172.21.15.102:3300/0 conn(0x7fd09c15cd90 0x7fd09c06b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:50.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.721+0000 7fd09aa00640 1 --2- 172.21.15.102:0/89457398 >> v2:172.21.15.173:3300/0 conn(0x7fd09c0cd6b0 0x7fd09c0b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:50.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.722+0000 7fd092000640 1 -- 172.21.15.102:0/89457398 >> v2:172.21.15.173:3300/0 conn(0x7fd09c0cd6b0 msgr2=0x7fd09c0b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:50.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.722+0000 7fd092000640 1 --2- 172.21.15.102:0/89457398 >> v2:172.21.15.173:3300/0 conn(0x7fd09c0cd6b0 0x7fd09c0b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:50.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.722+0000 7fd092000640 1 -- 172.21.15.102:0/89457398 >> v1:172.21.15.102:6789/0 conn(0x7fd09c0cda90 legacy=0x7fd09c1606c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:50.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.722+0000 7fd092000640 1 -- 172.21.15.102:0/89457398 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c06bb20 con 0x7fd09c15cd90 2024-03-23T03:48:50.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.723+0000 7fd092000640 1 --2- 172.21.15.102:0/89457398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd09c15cd90 0x7fd09c06b060 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0b46b0 tx=0x7fd08c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b934420046e420cf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:50.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.723+0000 7fd09a000640 1 -- 172.21.15.102:0/89457398 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c031830 con 0x7fd09c15cd90 2024-03-23T03:48:50.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.723+0000 7fd09a000640 1 -- 172.21.15.102:0/89457398 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08c031990 con 0x7fd09c15cd90 2024-03-23T03:48:50.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.723+0000 7fd09a000640 1 -- 172.21.15.102:0/89457398 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c031d20 con 0x7fd09c15cd90 2024-03-23T03:48:50.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.724+0000 7fd0a1600640 1 -- 172.21.15.102:0/89457398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd09c15cd90 msgr2=0x7fd09c06b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:50.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.724+0000 7fd0a1600640 1 --2- 172.21.15.102:0/89457398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd09c15cd90 0x7fd09c06b060 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0b46b0 tx=0x7fd08c02b8d0 comp rx=0 tx=0).stop 2024-03-23T03:48:50.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.724+0000 7fd0a1600640 1 -- 172.21.15.102:0/89457398 shutdown_connections 2024-03-23T03:48:50.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.724+0000 7fd0a1600640 1 --2- 172.21.15.102:0/89457398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd09c15cd90 0x7fd09c06b060 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:50.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.724+0000 7fd0a1600640 1 --2- 172.21.15.102:0/89457398 >> v2:172.21.15.173:3300/0 conn(0x7fd09c0cd6b0 0x7fd09c0b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:50.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.724+0000 7fd0a1600640 1 -- 172.21.15.102:0/89457398 >> 172.21.15.102:0/89457398 conn(0x7fd09c0cf780 msgr2=0x7fd09c0cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:50.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.724+0000 7fd0a1600640 1 -- 172.21.15.102:0/89457398 shutdown_connections 2024-03-23T03:48:50.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.724+0000 7fd0a1600640 1 -- 172.21.15.102:0/89457398 wait complete. 2024-03-23T03:48:50.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.726+0000 7fd0a1600640 1 Processor -- start 2024-03-23T03:48:50.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.726+0000 7fd0a1600640 1 -- start start 2024-03-23T03:48:50.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.727+0000 7fd0a1600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd09c0cd6b0 0x7fd09c0c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:50.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.727+0000 7fd0a1600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd09c0cda90 0x7fd09c156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:50.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.727+0000 7fd0a1600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd09c15cd90 0x7fd09c1f5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:50.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.727+0000 7fd0a1600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c1586b0 con 0x7fd09c0cda90 2024-03-23T03:48:50.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.727+0000 7fd0a1600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c0c92b0 con 0x7fd09c15cd90 2024-03-23T03:48:50.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.727+0000 7fd0a1600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c1588c0 con 0x7fd09c0cd6b0 2024-03-23T03:48:50.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.727+0000 7fd092000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd09c0cda90 0x7fd09c156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:50.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.727+0000 7fd092000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd09c0cda90 0x7fd09c156a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46852/0 (socket says 172.21.15.102:46852) 2024-03-23T03:48:50.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.727+0000 7fd092000640 1 -- 172.21.15.102:0/1132386769 learned_addr learned my addr 172.21.15.102:0/1132386769 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:50.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.728+0000 7fd09aa00640 1 --2- 172.21.15.102:0/1132386769 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd09c0cd6b0 0x7fd09c0c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:50.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.728+0000 7fd09b400640 1 --2- 172.21.15.102:0/1132386769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd09c15cd90 0x7fd09c1f5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:50.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.728+0000 7fd092000640 1 -- 172.21.15.102:0/1132386769 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd09c0cd6b0 msgr2=0x7fd09c0c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:50.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.728+0000 7fd092000640 1 --2- 172.21.15.102:0/1132386769 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd09c0cd6b0 0x7fd09c0c8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:50.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.728+0000 7fd092000640 1 -- 172.21.15.102:0/1132386769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd09c15cd90 msgr2=0x7fd09c1f5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:50.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.728+0000 7fd092000640 1 --2- 172.21.15.102:0/1132386769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd09c15cd90 0x7fd09c1f5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:50.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.728+0000 7fd092000640 1 -- 172.21.15.102:0/1132386769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08c009400 con 0x7fd09c0cda90 2024-03-23T03:48:50.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.728+0000 7fd092000640 1 --2- 172.21.15.102:0/1132386769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd09c0cda90 0x7fd09c156a60 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd08c000c00 tx=0x7fd08c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:50.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.728+0000 7fd093400640 1 -- 172.21.15.102:0/1132386769 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c03a8f0 con 0x7fd09c0cda90 2024-03-23T03:48:50.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.729+0000 7fd093400640 1 -- 172.21.15.102:0/1132386769 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd08c03aa70 con 0x7fd09c0cda90 2024-03-23T03:48:50.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.729+0000 7fd093400640 1 -- 172.21.15.102:0/1132386769 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd08c03ae20 con 0x7fd09c0cda90 2024-03-23T03:48:50.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.729+0000 7fd0a1600640 1 -- 172.21.15.102:0/1132386769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd09c158ba0 con 0x7fd09c0cda90 2024-03-23T03:48:50.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.729+0000 7fd0a1600640 1 -- 172.21.15.102:0/1132386769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd09c159160 con 0x7fd09c0cda90 2024-03-23T03:48:50.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.730+0000 7fd0a1600640 1 -- 172.21.15.102:0/1132386769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09c15c8d0 con 0x7fd09c0cda90 2024-03-23T03:48:50.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.731+0000 7fd093400640 1 -- 172.21.15.102:0/1132386769 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd08c04d450 con 0x7fd09c0cda90 2024-03-23T03:48:50.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.732+0000 7fd093400640 1 --2- 172.21.15.102:0/1132386769 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd07806ac00 0x7fd07806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:50.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.732+0000 7fd093400640 1 -- 172.21.15.102:0/1132386769 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fd08c0ab170 con 0x7fd09c0cda90 2024-03-23T03:48:50.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.732+0000 7fd09aa00640 1 --2- 172.21.15.102:0/1132386769 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd07806ac00 0x7fd07806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:50.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.733+0000 7fd09aa00640 1 --2- 172.21.15.102:0/1132386769 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd07806ac00 0x7fd07806d0c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd084009b40 tx=0x7fd084009190 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:50.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.737+0000 7fd093400640 1 -- 172.21.15.102:0/1132386769 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd08c076520 con 0x7fd09c0cda90 2024-03-23T03:48:50.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:50.939+0000 7fd0a1600640 1 -- 172.21.15.102:0/1132386769 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd09c073390 con 0x7fd09c0cda90 2024-03-23T03:48:51.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.721+0000 7fd093400640 1 -- 172.21.15.102:0/1132386769 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v51) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd08c079b40 con 0x7fd09c0cda90 2024-03-23T03:48:51.722 INFO:teuthology.orchestra.run.smithi102.stderr: removed all blocklist entries 2024-03-23T03:48:51.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.724+0000 7fd0a1600640 1 -- 172.21.15.102:0/1132386769 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd07806ac00 msgr2=0x7fd07806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:51.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.724+0000 7fd0a1600640 1 --2- 172.21.15.102:0/1132386769 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd07806ac00 0x7fd07806d0c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd084009b40 tx=0x7fd084009190 comp rx=0 tx=0).stop 2024-03-23T03:48:51.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.724+0000 7fd0a1600640 1 -- 172.21.15.102:0/1132386769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd09c0cda90 msgr2=0x7fd09c156a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:51.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.724+0000 7fd0a1600640 1 --2- 172.21.15.102:0/1132386769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd09c0cda90 0x7fd09c156a60 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd08c000c00 tx=0x7fd08c0046c0 comp rx=0 tx=0).stop 2024-03-23T03:48:51.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.725+0000 7fd0a1600640 1 -- 172.21.15.102:0/1132386769 shutdown_connections 2024-03-23T03:48:51.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.725+0000 7fd0a1600640 1 --2- 172.21.15.102:0/1132386769 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd07806ac00 0x7fd07806d0c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:51.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.725+0000 7fd0a1600640 1 --2- 172.21.15.102:0/1132386769 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd09c15cd90 0x7fd09c1f5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:51.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.725+0000 7fd0a1600640 1 --2- 172.21.15.102:0/1132386769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd09c0cda90 0x7fd09c156a60 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:51.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.725+0000 7fd0a1600640 1 --2- 172.21.15.102:0/1132386769 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd09c0cd6b0 0x7fd09c0c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:51.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.725+0000 7fd0a1600640 1 -- 172.21.15.102:0/1132386769 >> 172.21.15.102:0/1132386769 conn(0x7fd09c0cf780 msgr2=0x7fd09c0b5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:51.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.725+0000 7fd0a1600640 1 -- 172.21.15.102:0/1132386769 shutdown_connections 2024-03-23T03:48:51.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.725+0000 7fd0a1600640 1 -- 172.21.15.102:0/1132386769 wait complete. 2024-03-23T03:48:51.745 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-23T03:48:51.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.948+0000 7f4373e00640 1 Processor -- start 2024-03-23T03:48:51.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.948+0000 7f4373e00640 1 -- start start 2024-03-23T03:48:51.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.949+0000 7f4373e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f436c15b560 0x7f436c15b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:51.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.949+0000 7f4373e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f436c15d410 0x7f436c0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:51.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.950+0000 7f4373e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436c06c6f0 con 0x7f436c0b39b0 2024-03-23T03:48:51.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.950+0000 7f4373e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f436c0b4370 con 0x7f436c15d410 2024-03-23T03:48:51.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.950+0000 7f4373e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f436c06c9d0 con 0x7f436c15b560 2024-03-23T03:48:51.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.950+0000 7f4371e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f436c15d410 0x7f436c0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:51.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.950+0000 7f4371e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f436c15d410 0x7f436c0b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46860/0 (socket says 172.21.15.102:46860) 2024-03-23T03:48:51.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.950+0000 7f4371e00640 1 -- 172.21.15.102:0/1232147873 learned_addr learned my addr 172.21.15.102:0/1232147873 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:51.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.950+0000 7f4371400640 1 --2- 172.21.15.102:0/1232147873 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f436c15b560 0x7f436c15b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:51.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.950+0000 7f4371e00640 1 -- 172.21.15.102:0/1232147873 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f436c15b560 msgr2=0x7f436c15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:51.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.950+0000 7f4371e00640 1 --2- 172.21.15.102:0/1232147873 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f436c15b560 0x7f436c15b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:51.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.950+0000 7f4371e00640 1 -- 172.21.15.102:0/1232147873 >> v1:172.21.15.173:6789/0 conn(0x7f436c0b39b0 legacy=0x7f436c15e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:51.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.950+0000 7f4371e00640 1 -- 172.21.15.102:0/1232147873 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c06ccb0 con 0x7f436c15d410 2024-03-23T03:48:51.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.952+0000 7f4371e00640 1 --2- 172.21.15.102:0/1232147873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f436c15d410 0x7f436c0b3e30 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4364009e50 tx=0x7f436402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77432ba158c003c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:51.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.952+0000 7f436be00640 1 -- 172.21.15.102:0/1232147873 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436402ca70 con 0x7f436c15d410 2024-03-23T03:48:51.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.952+0000 7f436be00640 1 -- 172.21.15.102:0/1232147873 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f436402cbd0 con 0x7f436c15d410 2024-03-23T03:48:51.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.952+0000 7f436be00640 1 -- 172.21.15.102:0/1232147873 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43640343f0 con 0x7f436c15d410 2024-03-23T03:48:51.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.953+0000 7f4373e00640 1 -- 172.21.15.102:0/1232147873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f436c15d410 msgr2=0x7f436c0b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:51.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.953+0000 7f4373e00640 1 --2- 172.21.15.102:0/1232147873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f436c15d410 0x7f436c0b3e30 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f4364009e50 tx=0x7f436402bed0 comp rx=0 tx=0).stop 2024-03-23T03:48:51.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.953+0000 7f4373e00640 1 -- 172.21.15.102:0/1232147873 shutdown_connections 2024-03-23T03:48:51.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.953+0000 7f4373e00640 1 --2- 172.21.15.102:0/1232147873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f436c15d410 0x7f436c0b3e30 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:51.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.953+0000 7f4373e00640 1 --2- 172.21.15.102:0/1232147873 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f436c15b560 0x7f436c15b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:51.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.953+0000 7f4373e00640 1 -- 172.21.15.102:0/1232147873 >> 172.21.15.102:0/1232147873 conn(0x7f436c06a3d0 msgr2=0x7f436c0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:51.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.953+0000 7f4373e00640 1 -- 172.21.15.102:0/1232147873 shutdown_connections 2024-03-23T03:48:51.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.953+0000 7f4373e00640 1 -- 172.21.15.102:0/1232147873 wait complete. 2024-03-23T03:48:51.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.955+0000 7f4373e00640 1 Processor -- start 2024-03-23T03:48:51.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.955+0000 7f4373e00640 1 -- start start 2024-03-23T03:48:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.956+0000 7f4373e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f436c0b39b0 0x7f436c159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.956+0000 7f4373e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f436c15b560 0x7f436c143000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.956+0000 7f4373e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f436c15d410 0x7f436c143540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.956+0000 7f4373e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c136e40 con 0x7f436c15d410 2024-03-23T03:48:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.956+0000 7f4373e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f436c136230 con 0x7f436c0b39b0 2024-03-23T03:48:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.956+0000 7f4373e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f436c136510 con 0x7f436c15b560 2024-03-23T03:48:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.956+0000 7f4371e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f436c15d410 0x7f436c143540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.956+0000 7f4371e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f436c15d410 0x7f436c143540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46872/0 (socket says 172.21.15.102:46872) 2024-03-23T03:48:51.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.956+0000 7f4371e00640 1 -- 172.21.15.102:0/3848426572 learned_addr learned my addr 172.21.15.102:0/3848426572 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.956+0000 7f4370a00640 1 --2- 172.21.15.102:0/3848426572 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f436c15b560 0x7f436c143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.957+0000 7f4371e00640 1 -- 172.21.15.102:0/3848426572 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f436c15b560 msgr2=0x7f436c143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.957+0000 7f4371e00640 1 --2- 172.21.15.102:0/3848426572 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f436c15b560 0x7f436c143000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.957+0000 7f4371e00640 1 -- 172.21.15.102:0/3848426572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f436c0b39b0 msgr2=0x7f436c159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.957+0000 7f4371e00640 1 --2- 172.21.15.102:0/3848426572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f436c0b39b0 0x7f436c159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.957+0000 7f4371e00640 1 -- 172.21.15.102:0/3848426572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4364009b30 con 0x7f436c15d410 2024-03-23T03:48:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.957+0000 7f4371e00640 1 --2- 172.21.15.102:0/3848426572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f436c15d410 0x7f436c143540 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4364009e50 tx=0x7f4364004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.957+0000 7f4369600640 1 -- 172.21.15.102:0/3848426572 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4364009510 con 0x7f436c15d410 2024-03-23T03:48:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.957+0000 7f4369600640 1 -- 172.21.15.102:0/3848426572 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f436403e450 con 0x7f436c15d410 2024-03-23T03:48:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.957+0000 7f4373e00640 1 -- 172.21.15.102:0/3848426572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436c1367f0 con 0x7f436c15d410 2024-03-23T03:48:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.957+0000 7f4369600640 1 -- 172.21.15.102:0/3848426572 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f436403e800 con 0x7f436c15d410 2024-03-23T03:48:51.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.957+0000 7f4373e00640 1 -- 172.21.15.102:0/3848426572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f436c159660 con 0x7f436c15d410 2024-03-23T03:48:51.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.958+0000 7f4373e00640 1 -- 172.21.15.102:0/3848426572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f436c06ac20 con 0x7f436c15d410 2024-03-23T03:48:51.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.959+0000 7f4369600640 1 -- 172.21.15.102:0/3848426572 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f436403e960 con 0x7f436c15d410 2024-03-23T03:48:51.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.960+0000 7f4369600640 1 --2- 172.21.15.102:0/3848426572 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f434c06ac00 0x7f434c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:51.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.960+0000 7f4369600640 1 -- 172.21.15.102:0/3848426572 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f43640abf10 con 0x7f436c15d410 2024-03-23T03:48:51.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.961+0000 7f4371400640 1 --2- 172.21.15.102:0/3848426572 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f434c06ac00 0x7f434c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:51.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.961+0000 7f4371400640 1 --2- 172.21.15.102:0/3848426572 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f434c06ac00 0x7f434c06d0c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f43600098f0 tx=0x7f4360008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:51.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:51.965+0000 7f4369600640 1 -- 172.21.15.102:0/3848426572 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f43640772f0 con 0x7f436c15d410 2024-03-23T03:48:52.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.229+0000 7f4373e00640 1 -- 172.21.15.102:0/3848426572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f436c136fb0 con 0x7f436c15d410 2024-03-23T03:48:52.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.230+0000 7f4369600640 1 -- 172.21.15.102:0/3848426572 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f436407a910 con 0x7f436c15d410 2024-03-23T03:48:52.231 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:52.231 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:48:52.231 INFO:teuthology.orchestra.run.smithi102.stdout: "auth_dump": [ 2024-03-23T03:48:52.231 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.232 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.a", 2024-03-23T03:48:52.232 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5loBTZFRAA7/tTYyZwrDmwKwiU+dtCEw==", 2024-03-23T03:48:52.232 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.232 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:48:52.232 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:48:52.232 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:48:52.232 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:52.232 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.232 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.232 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.232 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.b", 2024-03-23T03:48:52.232 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5lJKKlNxAAj5+j99K79IhbeEm7/qC+wg==", 2024-03-23T03:48:52.232 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.232 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:48:52.232 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:48:52.233 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:48:52.233 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:52.233 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.233 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.233 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.233 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.c", 2024-03-23T03:48:52.233 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5l0b92JhAA5rvs+mdgmrQX/9lI3GW2rg==", 2024-03-23T03:48:52.233 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.233 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:48:52.233 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:48:52.233 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:48:52.233 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:52.233 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.233 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.234 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.234 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.d", 2024-03-23T03:48:52.234 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5ltjNMDBAAD1h5Nlv2Cp7mJt+DMlBtzw==", 2024-03-23T03:48:52.234 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.234 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:48:52.234 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:48:52.234 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:48:52.234 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:52.234 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.234 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.234 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.234 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.0", 2024-03-23T03:48:52.234 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB2UP5lPW6uORAAejiQReOF95oFliFmk/vdQQ==", 2024-03-23T03:48:52.234 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.235 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:48:52.235 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:48:52.235 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:52.235 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.235 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.235 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.235 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.1", 2024-03-23T03:48:52.235 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB6UP5lFbbUBxAAIl0ZEYtJ+/oi3W6a4OwMhg==", 2024-03-23T03:48:52.235 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.235 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:48:52.235 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:48:52.235 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:52.235 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.235 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.235 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.236 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.2", 2024-03-23T03:48:52.236 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB9UP5l2LaQIBAAKJv7jWJAjlUJrm7+KTXi6Q==", 2024-03-23T03:48:52.236 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.236 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:48:52.236 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:48:52.236 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:52.236 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.236 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.236 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.236 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.3", 2024-03-23T03:48:52.236 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCAUP5lxjWsKhAAzVO8CXUd+4LLvv4f/5xMOg==", 2024-03-23T03:48:52.236 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.236 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:48:52.236 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:48:52.236 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:52.237 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.237 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.237 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.237 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.4", 2024-03-23T03:48:52.237 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCHUP5lHcd2JRAAwID1TPb+Z9ViH51B0Mv/kQ==", 2024-03-23T03:48:52.237 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.237 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:48:52.237 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:48:52.237 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:52.237 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.237 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.237 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.237 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.5", 2024-03-23T03:48:52.237 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCLUP5lApp3AxAADslbuzTe/bf5FML1ckE05w==", 2024-03-23T03:48:52.238 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.238 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:48:52.238 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:48:52.238 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:52.238 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.238 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.238 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.238 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.6", 2024-03-23T03:48:52.238 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCOUP5lsi7VGxAArhWvXQ1VzJ8oSiOXzq8VsQ==", 2024-03-23T03:48:52.238 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.238 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:48:52.238 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:48:52.238 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:52.238 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.239 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.239 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.239 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.7", 2024-03-23T03:48:52.239 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCRUP5lu182JxAA5LaDXfrBkOXEUNAJXFDCJQ==", 2024-03-23T03:48:52.239 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.239 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:48:52.239 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:48:52.239 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:52.239 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.239 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.239 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.239 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.0", 2024-03-23T03:48:52.239 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lxTfoGRAA3WIQ8p1rGSAyJkLZJMk3dA==", 2024-03-23T03:48:52.239 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.240 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:48:52.240 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow r", 2024-03-23T03:48:52.240 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-23T03:48:52.240 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.240 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.240 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.240 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.1", 2024-03-23T03:48:52.240 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lneSHLhAAwn6ULOajHXrOM650HUcLbw==", 2024-03-23T03:48:52.240 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.240 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:48:52.240 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:48:52.240 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:48:52.240 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:48:52.240 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.240 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.241 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.241 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.2", 2024-03-23T03:48:52.241 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lOIklJBAAhx2d0Xy/j1PG0qsvC3nMqA==", 2024-03-23T03:48:52.241 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.241 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:48:52.241 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:48:52.241 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:48:52.241 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:48:52.241 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.241 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.241 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.241 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.3", 2024-03-23T03:48:52.241 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5l+7BVORAA7PBYS6mkgLA8b4sLCAYgBA==", 2024-03-23T03:48:52.241 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.241 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:48:52.242 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:48:52.242 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:48:52.242 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:48:52.242 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.242 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.242 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.242 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.admin", 2024-03-23T03:48:52.242 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBwUP5ldlDgBxAAOiqmhC5GulxELun8OiR4lg==", 2024-03-23T03:48:52.242 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.242 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:48:52.242 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:48:52.242 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:48:52.242 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:52.242 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.243 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.243 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.243 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mgr.x", 2024-03-23T03:48:52.243 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5lQXojChAAzepTDh0Pk3fuBZYsFt/+DQ==", 2024-03-23T03:48:52.243 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.243 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:48:52.243 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile mgr", 2024-03-23T03:48:52.243 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:52.243 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.243 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:48:52.243 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:52.243 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mgr.y", 2024-03-23T03:48:52.243 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBwUP5lK59iORAACuBF0JaQ8epyu3Mq3slzjg==", 2024-03-23T03:48:52.243 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:48:52.244 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:48:52.244 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile mgr", 2024-03-23T03:48:52.244 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:48:52.244 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.244 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:52.244 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-03-23T03:48:52.244 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:48:52.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.233+0000 7f4373e00640 1 -- 172.21.15.102:0/3848426572 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f434c06ac00 msgr2=0x7f434c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:52.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.233+0000 7f4373e00640 1 --2- 172.21.15.102:0/3848426572 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f434c06ac00 0x7f434c06d0c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f43600098f0 tx=0x7f4360008040 comp rx=0 tx=0).stop 2024-03-23T03:48:52.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.233+0000 7f4373e00640 1 -- 172.21.15.102:0/3848426572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f436c15d410 msgr2=0x7f436c143540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:52.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.233+0000 7f4373e00640 1 --2- 172.21.15.102:0/3848426572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f436c15d410 0x7f436c143540 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f4364009e50 tx=0x7f4364004a40 comp rx=0 tx=0).stop 2024-03-23T03:48:52.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.233+0000 7f4373e00640 1 -- 172.21.15.102:0/3848426572 shutdown_connections 2024-03-23T03:48:52.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.233+0000 7f4373e00640 1 --2- 172.21.15.102:0/3848426572 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f434c06ac00 0x7f434c06d0c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.233+0000 7f4373e00640 1 --2- 172.21.15.102:0/3848426572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f436c15d410 0x7f436c143540 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.233+0000 7f4373e00640 1 --2- 172.21.15.102:0/3848426572 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f436c15b560 0x7f436c143000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.233+0000 7f4373e00640 1 --2- 172.21.15.102:0/3848426572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f436c0b39b0 0x7f436c159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.233+0000 7f4373e00640 1 -- 172.21.15.102:0/3848426572 >> 172.21.15.102:0/3848426572 conn(0x7f436c06a3d0 msgr2=0x7f436c0b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:52.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.234+0000 7f4373e00640 1 -- 172.21.15.102:0/3848426572 shutdown_connections 2024-03-23T03:48:52.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.234+0000 7f4373e00640 1 -- 172.21.15.102:0/3848426572 wait complete. 2024-03-23T03:48:52.253 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:48:52.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.451+0000 7f8b25000640 1 Processor -- start 2024-03-23T03:48:52.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.452+0000 7f8b25000640 1 -- start start 2024-03-23T03:48:52.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.453+0000 7f8b25000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8b200cf5c0 0x7f8b200b8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:52.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.453+0000 7f8b25000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b2015ce90 con 0x7f8b200d0c20 2024-03-23T03:48:52.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.453+0000 7f8b25000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b200cddf0 con 0x7f8b2015cab0 2024-03-23T03:48:52.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.453+0000 7f8b25000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b200cd570 con 0x7f8b200cf5c0 2024-03-23T03:48:52.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.453+0000 7f8b1fe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8b200d0c20 0x7f8b200d10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:35978/0 (socket says 172.21.15.102:35978) 2024-03-23T03:48:52.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.453+0000 7f8b1fe00640 1 -- 172.21.15.102:0/885576107 learned_addr learned my addr 172.21.15.102:0/885576107 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:52.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.454+0000 7f8b16a00640 1 --2- 172.21.15.102:0/885576107 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8b200cf5c0 0x7f8b200b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:52.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.454+0000 7f8b1ea00640 1 -- 172.21.15.102:0/885576107 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813199123 0 0) 0x7f8b2015ce90 con 0x7f8b200d0c20 2024-03-23T03:48:52.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.454+0000 7f8b1ea00640 1 -- 172.21.15.102:0/885576107 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b04003520 con 0x7f8b200d0c20 2024-03-23T03:48:52.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.454+0000 7f8b1ea00640 1 -- 172.21.15.102:0/885576107 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2782409184 0 0) 0x7f8b04003520 con 0x7f8b200d0c20 2024-03-23T03:48:52.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.454+0000 7f8b1ea00640 1 -- 172.21.15.102:0/885576107 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8b200cf5c0 msgr2=0x7f8b200b8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:52.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.454+0000 7f8b1ea00640 1 --2- 172.21.15.102:0/885576107 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8b200cf5c0 0x7f8b200b8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.454+0000 7f8b1ea00640 1 -- 172.21.15.102:0/885576107 >> v1:172.21.15.173:6789/0 conn(0x7f8b2015cab0 legacy=0x7f8b200ceeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:52.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.454+0000 7f8b1ea00640 1 -- 172.21.15.102:0/885576107 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b20161370 con 0x7f8b200d0c20 2024-03-23T03:48:52.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.454+0000 7f8b1ea00640 1 -- 172.21.15.102:0/885576107 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f8b180019f0 con 0x7f8b200d0c20 2024-03-23T03:48:52.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.454+0000 7f8b1ea00640 1 -- 172.21.15.102:0/885576107 >> v1:172.21.15.102:6789/0 conn(0x7f8b200d0c20 legacy=0x7f8b200d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:52.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.454+0000 7f8b1ea00640 1 --2- 172.21.15.102:0/885576107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b040039a0 0x7f8b04003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:52.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.454+0000 7f8b1ea00640 1 --2- 172.21.15.102:0/885576107 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8b200cf5c0 0x7f8b040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:52.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.454+0000 7f8b1ea00640 1 --2- 172.21.15.102:0/885576107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b04004d00 0x7f8b04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:52.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.454+0000 7f8b1ea00640 1 -- 172.21.15.102:0/885576107 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b200cd570 con 0x7f8b040039a0 2024-03-23T03:48:52.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.454+0000 7f8b1ea00640 1 -- 172.21.15.102:0/885576107 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0400ba60 con 0x7f8b04004d00 2024-03-23T03:48:52.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.454+0000 7f8b1ea00640 1 -- 172.21.15.102:0/885576107 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b0400bc20 con 0x7f8b200cf5c0 2024-03-23T03:48:52.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.455+0000 7f8b1f400640 1 --2- 172.21.15.102:0/885576107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b040039a0 0x7f8b04003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:52.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.455+0000 7f8b16a00640 1 --2- 172.21.15.102:0/885576107 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8b200cf5c0 0x7f8b040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:52.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.455+0000 7f8b1fe00640 1 --2- 172.21.15.102:0/885576107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b04004d00 0x7f8b04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:52.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.455+0000 7f8b1f400640 1 -- 172.21.15.102:0/885576107 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8b200cf5c0 msgr2=0x7f8b040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:52.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.455+0000 7f8b1f400640 1 --2- 172.21.15.102:0/885576107 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8b200cf5c0 0x7f8b040047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.455+0000 7f8b1f400640 1 -- 172.21.15.102:0/885576107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b04004d00 msgr2=0x7f8b04005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:52.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.455+0000 7f8b1f400640 1 --2- 172.21.15.102:0/885576107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b04004d00 0x7f8b04005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.455+0000 7f8b1f400640 1 -- 172.21.15.102:0/885576107 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b04003520 con 0x7f8b040039a0 2024-03-23T03:48:52.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.455+0000 7f8b1fe00640 1 --2- 172.21.15.102:0/885576107 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b04004d00 0x7f8b04005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:52.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.456+0000 7f8b1f400640 1 --2- 172.21.15.102:0/885576107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b040039a0 0x7f8b04003dc0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8b10001720 tx=0x7f8b040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=732d77ad0f63dd65 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:52.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.457+0000 7f8b1ea00640 1 -- 172.21.15.102:0/885576107 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b100052d0 con 0x7f8b040039a0 2024-03-23T03:48:52.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.457+0000 7f8b1ea00640 1 -- 172.21.15.102:0/885576107 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b1002e050 con 0x7f8b040039a0 2024-03-23T03:48:52.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.457+0000 7f8b1ea00640 1 -- 172.21.15.102:0/885576107 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b100035f0 con 0x7f8b040039a0 2024-03-23T03:48:52.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.457+0000 7f8b25000640 1 -- 172.21.15.102:0/885576107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b040039a0 msgr2=0x7f8b04003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:52.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.457+0000 7f8b25000640 1 --2- 172.21.15.102:0/885576107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b040039a0 0x7f8b04003dc0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f8b10001720 tx=0x7f8b040057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.457+0000 7f8b1fe00640 1 -- 172.21.15.102:0/885576107 reap_dead start 2024-03-23T03:48:52.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.457+0000 7f8b25000640 1 -- 172.21.15.102:0/885576107 shutdown_connections 2024-03-23T03:48:52.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.457+0000 7f8b25000640 1 -- 172.21.15.102:0/885576107 >> 172.21.15.102:0/885576107 conn(0x7f8b2006ae90 msgr2=0x7f8b2006b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:52.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.457+0000 7f8b25000640 1 -- 172.21.15.102:0/885576107 shutdown_connections 2024-03-23T03:48:52.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.457+0000 7f8b25000640 1 -- 172.21.15.102:0/885576107 wait complete. 2024-03-23T03:48:52.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.459+0000 7f8b25000640 1 Processor -- start 2024-03-23T03:48:52.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.459+0000 7f8b25000640 1 -- start start 2024-03-23T03:48:52.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.460+0000 7f8b25000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8b040039a0 0x7f8b200b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:52.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.460+0000 7f8b25000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b201f21e0 0x7f8b2015f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:52.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.460+0000 7f8b25000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b2015da20 0x7f8b2015dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:52.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.460+0000 7f8b25000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b201f3220 con 0x7f8b2015da20 2024-03-23T03:48:52.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.460+0000 7f8b25000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b200b4870 con 0x7f8b201f21e0 2024-03-23T03:48:52.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.460+0000 7f8b25000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2015d600 con 0x7f8b040039a0 2024-03-23T03:48:52.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.460+0000 7f8b1fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b2015da20 0x7f8b2015dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:52.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.461+0000 7f8b1fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b2015da20 0x7f8b2015dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:46890/0 (socket says 172.21.15.102:46890) 2024-03-23T03:48:52.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.461+0000 7f8b1fe00640 1 -- 172.21.15.102:0/1841489401 learned_addr learned my addr 172.21.15.102:0/1841489401 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:52.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.461+0000 7f8b16a00640 1 --2- 172.21.15.102:0/1841489401 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b201f21e0 0x7f8b2015f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:52.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.461+0000 7f8b1f400640 1 --2- 172.21.15.102:0/1841489401 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8b040039a0 0x7f8b200b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:52.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.461+0000 7f8b1fe00640 1 -- 172.21.15.102:0/1841489401 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8b040039a0 msgr2=0x7f8b200b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:52.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.461+0000 7f8b1fe00640 1 --2- 172.21.15.102:0/1841489401 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8b040039a0 0x7f8b200b4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.461+0000 7f8b1fe00640 1 -- 172.21.15.102:0/1841489401 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b201f21e0 msgr2=0x7f8b2015f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:52.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.461+0000 7f8b1fe00640 1 --2- 172.21.15.102:0/1841489401 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b201f21e0 0x7f8b2015f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.461+0000 7f8b1fe00640 1 -- 172.21.15.102:0/1841489401 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b201f2980 con 0x7f8b2015da20 2024-03-23T03:48:52.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.461+0000 7f8b1fe00640 1 --2- 172.21.15.102:0/1841489401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b2015da20 0x7f8b2015dea0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8b18001580 tx=0x7f8b1800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:52.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.461+0000 7f8b17e00640 1 -- 172.21.15.102:0/1841489401 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b18004d60 con 0x7f8b2015da20 2024-03-23T03:48:52.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.461+0000 7f8b17e00640 1 -- 172.21.15.102:0/1841489401 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b1800bb80 con 0x7f8b2015da20 2024-03-23T03:48:52.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.461+0000 7f8b17e00640 1 -- 172.21.15.102:0/1841489401 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b18007070 con 0x7f8b2015da20 2024-03-23T03:48:52.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.461+0000 7f8b25000640 1 -- 172.21.15.102:0/1841489401 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b201f2c60 con 0x7f8b2015da20 2024-03-23T03:48:52.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.462+0000 7f8b25000640 1 -- 172.21.15.102:0/1841489401 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b201f2660 con 0x7f8b2015da20 2024-03-23T03:48:52.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.463+0000 7f8b25000640 1 -- 172.21.15.102:0/1841489401 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b2015d360 con 0x7f8b2015da20 2024-03-23T03:48:52.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.464+0000 7f8b17e00640 1 -- 172.21.15.102:0/1841489401 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8b180071d0 con 0x7f8b2015da20 2024-03-23T03:48:52.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.465+0000 7f8b17e00640 1 --2- 172.21.15.102:0/1841489401 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8afc06ac00 0x7f8afc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:52.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.465+0000 7f8b17e00640 1 -- 172.21.15.102:0/1841489401 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f8b18088ed0 con 0x7f8b2015da20 2024-03-23T03:48:52.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.465+0000 7f8b1f400640 1 --2- 172.21.15.102:0/1841489401 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8afc06ac00 0x7f8afc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:52.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.466+0000 7f8b1f400640 1 --2- 172.21.15.102:0/1841489401 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8afc06ac00 0x7f8afc06d0c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8b10033000 tx=0x7f8b10009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:52.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.470+0000 7f8b17e00640 1 -- 172.21.15.102:0/1841489401 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8b2015d360 con 0x7f8b2015da20 2024-03-23T03:48:52.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.663+0000 7f8b25000640 1 -- 172.21.15.102:0/1841489401 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b2015d740 con 0x7f8b2015da20 2024-03-23T03:48:52.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.665+0000 7f8b17e00640 1 -- 172.21.15.102:0/1841489401 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+12642 (secure 0 0 0) 0x7f8b2015d740 con 0x7f8b2015da20 2024-03-23T03:48:52.666 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:52.666 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":51,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:51.713322+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:48:52.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.668+0000 7f8b25000640 1 -- 172.21.15.102:0/1841489401 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8afc06ac00 msgr2=0x7f8afc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:52.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.668+0000 7f8b25000640 1 --2- 172.21.15.102:0/1841489401 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8afc06ac00 0x7f8afc06d0c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8b10033000 tx=0x7f8b10009030 comp rx=0 tx=0).stop 2024-03-23T03:48:52.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.668+0000 7f8b25000640 1 -- 172.21.15.102:0/1841489401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b2015da20 msgr2=0x7f8b2015dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:52.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.668+0000 7f8b25000640 1 --2- 172.21.15.102:0/1841489401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b2015da20 0x7f8b2015dea0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f8b18001580 tx=0x7f8b1800b4a0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.668+0000 7f8b25000640 1 -- 172.21.15.102:0/1841489401 shutdown_connections 2024-03-23T03:48:52.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.668+0000 7f8b25000640 1 --2- 172.21.15.102:0/1841489401 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8afc06ac00 0x7f8afc06d0c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.668+0000 7f8b25000640 1 --2- 172.21.15.102:0/1841489401 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b2015da20 0x7f8b2015dea0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.668+0000 7f8b25000640 1 --2- 172.21.15.102:0/1841489401 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b201f21e0 0x7f8b2015f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.668+0000 7f8b25000640 1 --2- 172.21.15.102:0/1841489401 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8b040039a0 0x7f8b200b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.668+0000 7f8b25000640 1 -- 172.21.15.102:0/1841489401 >> 172.21.15.102:0/1841489401 conn(0x7f8b2006ae90 msgr2=0x7f8b200d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:52.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.668+0000 7f8b25000640 1 -- 172.21.15.102:0/1841489401 shutdown_connections 2024-03-23T03:48:52.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.668+0000 7f8b25000640 1 -- 172.21.15.102:0/1841489401 wait complete. 2024-03-23T03:48:52.688 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:48:52.688 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:48:52.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.893+0000 7f458f200640 1 Processor -- start 2024-03-23T03:48:52.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.893+0000 7f458f200640 1 -- start start 2024-03-23T03:48:52.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.895+0000 7f458f200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45881456a0 0x7f45880ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:52.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.895+0000 7f458f200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f4588070cd0 0x7f45880cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:52.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.895+0000 7f458f200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f458806d390 con 0x7f4588145c20 2024-03-23T03:48:52.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.895+0000 7f458f200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45880ad890 con 0x7f4588070cd0 2024-03-23T03:48:52.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.895+0000 7f458f200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4588145a80 con 0x7f45881456a0 2024-03-23T03:48:52.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.896+0000 7f4587e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45881456a0 0x7f45880ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:52.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.896+0000 7f4587e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45881456a0 0x7f45880ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:34366/0 (socket says 172.21.15.102:34366) 2024-03-23T03:48:52.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.896+0000 7f4587e00640 1 -- 172.21.15.102:0/1328262162 learned_addr learned my addr 172.21.15.102:0/1328262162 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:52.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.896+0000 7f4587400640 1 --2- 172.21.15.102:0/1328262162 >> v2:172.21.15.173:3300/0 conn(0x7f4588070cd0 0x7f45880cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:52.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.896+0000 7f4587e00640 1 -- 172.21.15.102:0/1328262162 >> v2:172.21.15.173:3300/0 conn(0x7f4588070cd0 msgr2=0x7f45880cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:52.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.896+0000 7f4587e00640 1 --2- 172.21.15.102:0/1328262162 >> v2:172.21.15.173:3300/0 conn(0x7f4588070cd0 0x7f45880cee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.896+0000 7f4587e00640 1 -- 172.21.15.102:0/1328262162 >> v1:172.21.15.173:6789/0 conn(0x7f4588145c20 legacy=0x7f45880ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:52.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.897+0000 7f4587e00640 1 -- 172.21.15.102:0/1328262162 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458806d670 con 0x7f45881456a0 2024-03-23T03:48:52.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.898+0000 7f4587e00640 1 --2- 172.21.15.102:0/1328262162 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45881456a0 0x7f45880ce930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4578009870 tx=0x7f457802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33f836d009cd00bb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:52.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.899+0000 7f4586a00640 1 -- 172.21.15.102:0/1328262162 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578030840 con 0x7f45881456a0 2024-03-23T03:48:52.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.899+0000 7f4586a00640 1 -- 172.21.15.102:0/1328262162 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45780309a0 con 0x7f45881456a0 2024-03-23T03:48:52.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.899+0000 7f4586a00640 1 -- 172.21.15.102:0/1328262162 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4578030d30 con 0x7f45881456a0 2024-03-23T03:48:52.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.900+0000 7f458f200640 1 -- 172.21.15.102:0/1328262162 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45881456a0 msgr2=0x7f45880ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:52.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.900+0000 7f458f200640 1 --2- 172.21.15.102:0/1328262162 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45881456a0 0x7f45880ce930 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4578009870 tx=0x7f457802b980 comp rx=0 tx=0).stop 2024-03-23T03:48:52.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.900+0000 7f458f200640 1 -- 172.21.15.102:0/1328262162 shutdown_connections 2024-03-23T03:48:52.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.900+0000 7f458f200640 1 --2- 172.21.15.102:0/1328262162 >> v2:172.21.15.173:3300/0 conn(0x7f4588070cd0 0x7f45880cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.900+0000 7f458f200640 1 --2- 172.21.15.102:0/1328262162 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45881456a0 0x7f45880ce930 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.900+0000 7f458f200640 1 -- 172.21.15.102:0/1328262162 >> 172.21.15.102:0/1328262162 conn(0x7f458806a0b0 msgr2=0x7f458806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:52.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.900+0000 7f458f200640 1 -- 172.21.15.102:0/1328262162 shutdown_connections 2024-03-23T03:48:52.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.900+0000 7f458f200640 1 -- 172.21.15.102:0/1328262162 wait complete. 2024-03-23T03:48:52.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.903+0000 7f458f200640 1 Processor -- start 2024-03-23T03:48:52.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.903+0000 7f458f200640 1 -- start start 2024-03-23T03:48:52.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.904+0000 7f458f200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4588070cd0 0x7f4588142b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:52.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.904+0000 7f458f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45881456a0 0x7f4588143040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:52.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.904+0000 7f458f200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4588145c20 0x7f45880c44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:52.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.904+0000 7f458f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45881426e0 con 0x7f45881456a0 2024-03-23T03:48:52.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.904+0000 7f458f200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4588141a60 con 0x7f4588070cd0 2024-03-23T03:48:52.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.904+0000 7f458f200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4588141d40 con 0x7f4588145c20 2024-03-23T03:48:52.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.904+0000 7f4587400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45881456a0 0x7f4588143040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:52.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.904+0000 7f4587400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45881456a0 0x7f4588143040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41702/0 (socket says 172.21.15.102:41702) 2024-03-23T03:48:52.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.904+0000 7f4587400640 1 -- 172.21.15.102:0/1263244735 learned_addr learned my addr 172.21.15.102:0/1263244735 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:52.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.905+0000 7f458d200640 1 --2- 172.21.15.102:0/1263244735 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4588145c20 0x7f45880c44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.905+0000 7f4587e00640 1 --2- 172.21.15.102:0/1263244735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4588070cd0 0x7f4588142b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.905+0000 7f4587400640 1 -- 172.21.15.102:0/1263244735 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4588145c20 msgr2=0x7f45880c44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.905+0000 7f4587400640 1 --2- 172.21.15.102:0/1263244735 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4588145c20 0x7f45880c44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.905+0000 7f4587400640 1 -- 172.21.15.102:0/1263244735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4588070cd0 msgr2=0x7f4588142b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.905+0000 7f4587400640 1 --2- 172.21.15.102:0/1263244735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4588070cd0 0x7f4588142b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.905+0000 7f4587400640 1 -- 172.21.15.102:0/1263244735 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45780094d0 con 0x7f45881456a0 2024-03-23T03:48:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.905+0000 7f458d200640 1 --2- 172.21.15.102:0/1263244735 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4588145c20 0x7f45880c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.905+0000 7f4587400640 1 --2- 172.21.15.102:0/1263244735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45881456a0 0x7f4588143040 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f457400e980 tx=0x7f457400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.905+0000 7f457fe00640 1 -- 172.21.15.102:0/1263244735 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4574009670 con 0x7f45881456a0 2024-03-23T03:48:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.905+0000 7f457fe00640 1 -- 172.21.15.102:0/1263244735 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457400ca00 con 0x7f45881456a0 2024-03-23T03:48:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.905+0000 7f457fe00640 1 -- 172.21.15.102:0/1263244735 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457400cd90 con 0x7f45881456a0 2024-03-23T03:48:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.905+0000 7f458f200640 1 -- 172.21.15.102:0/1263244735 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4588142020 con 0x7f45881456a0 2024-03-23T03:48:52.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.906+0000 7f458f200640 1 -- 172.21.15.102:0/1263244735 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45880c4a00 con 0x7f45881456a0 2024-03-23T03:48:52.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.907+0000 7f458f200640 1 -- 172.21.15.102:0/1263244735 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4588073390 con 0x7f45881456a0 2024-03-23T03:48:52.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.908+0000 7f457fe00640 1 -- 172.21.15.102:0/1263244735 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4574013470 con 0x7f45881456a0 2024-03-23T03:48:52.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.908+0000 7f457fe00640 1 --2- 172.21.15.102:0/1263244735 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f456806ac00 0x7f456806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:52.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.909+0000 7f457fe00640 1 -- 172.21.15.102:0/1263244735 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f4574089370 con 0x7f45881456a0 2024-03-23T03:48:52.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.909+0000 7f4587e00640 1 --2- 172.21.15.102:0/1263244735 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f456806ac00 0x7f456806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:52.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.909+0000 7f4587e00640 1 --2- 172.21.15.102:0/1263244735 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f456806ac00 0x7f456806d0c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4578036000 tx=0x7f457803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:52.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:52.913+0000 7f457fe00640 1 -- 172.21.15.102:0/1263244735 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4574054750 con 0x7f45881456a0 2024-03-23T03:48:53.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.102+0000 7f458f200640 1 -- 172.21.15.102:0/1263244735 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f458806cbc0 con 0x7f45881456a0 2024-03-23T03:48:53.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.103+0000 7f457fe00640 1 -- 172.21.15.102:0/1263244735 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v51) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4574057d70 con 0x7f45881456a0 2024-03-23T03:48:53.104 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:48:53.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.106+0000 7f458f200640 1 -- 172.21.15.102:0/1263244735 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f456806ac00 msgr2=0x7f456806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:53.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.106+0000 7f458f200640 1 --2- 172.21.15.102:0/1263244735 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f456806ac00 0x7f456806d0c0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4578036000 tx=0x7f457803b040 comp rx=0 tx=0).stop 2024-03-23T03:48:53.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.106+0000 7f458f200640 1 -- 172.21.15.102:0/1263244735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45881456a0 msgr2=0x7f4588143040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:53.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.106+0000 7f458f200640 1 --2- 172.21.15.102:0/1263244735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45881456a0 0x7f4588143040 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f457400e980 tx=0x7f457400ee40 comp rx=0 tx=0).stop 2024-03-23T03:48:53.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.106+0000 7f458f200640 1 -- 172.21.15.102:0/1263244735 shutdown_connections 2024-03-23T03:48:53.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.106+0000 7f458f200640 1 --2- 172.21.15.102:0/1263244735 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f456806ac00 0x7f456806d0c0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.106+0000 7f458f200640 1 --2- 172.21.15.102:0/1263244735 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4588145c20 0x7f45880c44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.106+0000 7f458f200640 1 --2- 172.21.15.102:0/1263244735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45881456a0 0x7f4588143040 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.106+0000 7f458f200640 1 --2- 172.21.15.102:0/1263244735 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4588070cd0 0x7f4588142b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.106+0000 7f458f200640 1 -- 172.21.15.102:0/1263244735 >> 172.21.15.102:0/1263244735 conn(0x7f458806a0b0 msgr2=0x7f45880b3fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:53.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.107+0000 7f458f200640 1 -- 172.21.15.102:0/1263244735 shutdown_connections 2024-03-23T03:48:53.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.107+0000 7f458f200640 1 -- 172.21.15.102:0/1263244735 wait complete. 2024-03-23T03:48:53.124 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T03:48:53.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.320+0000 7f55ba200640 1 Processor -- start 2024-03-23T03:48:53.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.321+0000 7f55ba200640 1 -- start start 2024-03-23T03:48:53.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.322+0000 7f55ba200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55b40b5a30 0x7f55b40b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:53.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.322+0000 7f55ba200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f55b40b4350 0x7f55b40b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:53.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.322+0000 7f55ba200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55b40d1830 con 0x7f55b415cda0 2024-03-23T03:48:53.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.322+0000 7f55ba200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55b40b4d20 con 0x7f55b40b4350 2024-03-23T03:48:53.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.322+0000 7f55ba200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b40b1f20 con 0x7f55b40b5a30 2024-03-23T03:48:53.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.322+0000 7f55b3400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f55b415cda0 0x7f55b40cdb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52532/0 (socket says 172.21.15.102:52532) 2024-03-23T03:48:53.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.322+0000 7f55b3400640 1 -- 172.21.15.102:0/3075787040 learned_addr learned my addr 172.21.15.102:0/3075787040 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:53.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.322+0000 7f55b3e00640 1 --2- 172.21.15.102:0/3075787040 >> v2:172.21.15.102:3300/0 conn(0x7f55b40b4350 0x7f55b40b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.323+0000 7f55aaa00640 1 --2- 172.21.15.102:0/3075787040 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55b40b5a30 0x7f55b40b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.323+0000 7f55b2a00640 1 -- 172.21.15.102:0/3075787040 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 381605047 0 0) 0x7f55b40d1830 con 0x7f55b415cda0 2024-03-23T03:48:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.323+0000 7f55b2a00640 1 -- 172.21.15.102:0/3075787040 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5598003520 con 0x7f55b415cda0 2024-03-23T03:48:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.323+0000 7f55b3e00640 1 -- 172.21.15.102:0/3075787040 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55b40b5a30 msgr2=0x7f55b40b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.323+0000 7f55b3e00640 1 --2- 172.21.15.102:0/3075787040 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55b40b5a30 0x7f55b40b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.323+0000 7f55b3e00640 1 -- 172.21.15.102:0/3075787040 >> v1:172.21.15.102:6789/0 conn(0x7f55b415cda0 legacy=0x7f55b40cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:53.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.323+0000 7f55b3e00640 1 -- 172.21.15.102:0/3075787040 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b40d0c20 con 0x7f55b40b4350 2024-03-23T03:48:53.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.324+0000 7f55b3e00640 1 --2- 172.21.15.102:0/3075787040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b40b4350 0x7f55b40b47e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009d90 tx=0x7f55ac02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244ff176e7ac184c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:53.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.324+0000 7f55b2a00640 1 -- 172.21.15.102:0/3075787040 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac02cc60 con 0x7f55b40b4350 2024-03-23T03:48:53.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.325+0000 7f55b2a00640 1 -- 172.21.15.102:0/3075787040 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55ac02cdc0 con 0x7f55b40b4350 2024-03-23T03:48:53.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.325+0000 7f55b2a00640 1 -- 172.21.15.102:0/3075787040 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55ac034660 con 0x7f55b40b4350 2024-03-23T03:48:53.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.325+0000 7f55ba200640 1 -- 172.21.15.102:0/3075787040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b40b4350 msgr2=0x7f55b40b47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:53.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.325+0000 7f55ba200640 1 --2- 172.21.15.102:0/3075787040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b40b4350 0x7f55b40b47e0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f55ac009d90 tx=0x7f55ac02beb0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.325+0000 7f55ba200640 1 -- 172.21.15.102:0/3075787040 shutdown_connections 2024-03-23T03:48:53.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.325+0000 7f55ba200640 1 --2- 172.21.15.102:0/3075787040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b40b4350 0x7f55b40b47e0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.325+0000 7f55ba200640 1 --2- 172.21.15.102:0/3075787040 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55b40b5a30 0x7f55b40b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.325+0000 7f55ba200640 1 -- 172.21.15.102:0/3075787040 >> 172.21.15.102:0/3075787040 conn(0x7f55b40cf350 msgr2=0x7f55b40cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:53.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.325+0000 7f55ba200640 1 -- 172.21.15.102:0/3075787040 shutdown_connections 2024-03-23T03:48:53.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.325+0000 7f55ba200640 1 -- 172.21.15.102:0/3075787040 wait complete. 2024-03-23T03:48:53.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.327+0000 7f55ba200640 1 Processor -- start 2024-03-23T03:48:53.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.327+0000 7f55ba200640 1 -- start start 2024-03-23T03:48:53.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.328+0000 7f55ba200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55b40b4350 0x7f55b41f6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:53.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.328+0000 7f55ba200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b40b5a30 0x7f55b40b2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:53.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.328+0000 7f55ba200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55b415cda0 0x7f55b4159d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:53.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.328+0000 7f55ba200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b41f2140 con 0x7f55b40b5a30 2024-03-23T03:48:53.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.328+0000 7f55ba200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b41f6680 con 0x7f55b40b4350 2024-03-23T03:48:53.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.328+0000 7f55ba200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b41f23c0 con 0x7f55b415cda0 2024-03-23T03:48:53.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.328+0000 7f55aaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b40b5a30 0x7f55b40b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:53.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.328+0000 7f55aaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b40b5a30 0x7f55b40b2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41716/0 (socket says 172.21.15.102:41716) 2024-03-23T03:48:53.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.328+0000 7f55aaa00640 1 -- 172.21.15.102:0/1779996848 learned_addr learned my addr 172.21.15.102:0/1779996848 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:53.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.329+0000 7f55aaa00640 1 -- 172.21.15.102:0/1779996848 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55b415cda0 msgr2=0x7f55b4159d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:53.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.329+0000 7f55aaa00640 1 --2- 172.21.15.102:0/1779996848 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55b415cda0 0x7f55b4159d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.329+0000 7f55aaa00640 1 -- 172.21.15.102:0/1779996848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55b40b4350 msgr2=0x7f55b41f6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:53.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.329+0000 7f55aaa00640 1 --2- 172.21.15.102:0/1779996848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55b40b4350 0x7f55b41f6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.329+0000 7f55aaa00640 1 -- 172.21.15.102:0/1779996848 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ac009a70 con 0x7f55b40b5a30 2024-03-23T03:48:53.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.329+0000 7f55aaa00640 1 --2- 172.21.15.102:0/1779996848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b40b5a30 0x7f55b40b2470 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f559c00d800 tx=0x7f559c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:53.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.329+0000 7f55abe00640 1 -- 172.21.15.102:0/1779996848 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c00b6c0 con 0x7f55b40b5a30 2024-03-23T03:48:53.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.329+0000 7f55abe00640 1 -- 172.21.15.102:0/1779996848 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559c00b820 con 0x7f55b40b5a30 2024-03-23T03:48:53.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.329+0000 7f55abe00640 1 -- 172.21.15.102:0/1779996848 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c00bbb0 con 0x7f55b40b5a30 2024-03-23T03:48:53.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.329+0000 7f55ba200640 1 -- 172.21.15.102:0/1779996848 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b41f26a0 con 0x7f55b40b5a30 2024-03-23T03:48:53.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.329+0000 7f55ba200640 1 -- 172.21.15.102:0/1779996848 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b41f2c60 con 0x7f55b40b5a30 2024-03-23T03:48:53.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.330+0000 7f55ba200640 1 -- 172.21.15.102:0/1779996848 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b4073390 con 0x7f55b40b5a30 2024-03-23T03:48:53.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.331+0000 7f55abe00640 1 -- 172.21.15.102:0/1779996848 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f559c00bd10 con 0x7f55b40b5a30 2024-03-23T03:48:53.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.332+0000 7f55abe00640 1 --2- 172.21.15.102:0/1779996848 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f559006ac30 0x7f559006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:53.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.332+0000 7f55abe00640 1 -- 172.21.15.102:0/1779996848 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f559c089750 con 0x7f55b40b5a30 2024-03-23T03:48:53.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.332+0000 7f55b3400640 1 --2- 172.21.15.102:0/1779996848 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f559006ac30 0x7f559006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:53.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.333+0000 7f55b3400640 1 --2- 172.21.15.102:0/1779996848 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f559006ac30 0x7f559006d0f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f55a4001570 tx=0x7f55a4007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:53.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.336+0000 7f55abe00640 1 -- 172.21.15.102:0/1779996848 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f559c054aa0 con 0x7f55b40b5a30 2024-03-23T03:48:53.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.519+0000 7f55ba200640 1 -- 172.21.15.102:0/1779996848 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f55b40b3ed0 con 0x7f55b40b5a30 2024-03-23T03:48:53.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.519+0000 7f55abe00640 1 -- 172.21.15.102:0/1779996848 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v51) v1 ==== 84+0+65 (secure 0 0 0) 0x7f559c0580c0 con 0x7f55b40b5a30 2024-03-23T03:48:53.520 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:53.520 INFO:teuthology.orchestra.run.smithi102.stdout:[ 2024-03-23T03:48:53.520 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:48:53.520 INFO:teuthology.orchestra.run.smithi102.stdout: "poolnum": 1, 2024-03-23T03:48:53.520 INFO:teuthology.orchestra.run.smithi102.stdout: "poolname": ".mgr" 2024-03-23T03:48:53.520 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:48:53.520 INFO:teuthology.orchestra.run.smithi102.stdout:] 2024-03-23T03:48:53.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.522+0000 7f55ba200640 1 -- 172.21.15.102:0/1779996848 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f559006ac30 msgr2=0x7f559006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:53.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.522+0000 7f55ba200640 1 --2- 172.21.15.102:0/1779996848 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f559006ac30 0x7f559006d0f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f55a4001570 tx=0x7f55a4007930 comp rx=0 tx=0).stop 2024-03-23T03:48:53.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.522+0000 7f55ba200640 1 -- 172.21.15.102:0/1779996848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b40b5a30 msgr2=0x7f55b40b2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:53.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.522+0000 7f55ba200640 1 --2- 172.21.15.102:0/1779996848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b40b5a30 0x7f55b40b2470 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f559c00d800 tx=0x7f559c00dcc0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.522+0000 7f55ba200640 1 -- 172.21.15.102:0/1779996848 shutdown_connections 2024-03-23T03:48:53.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.522+0000 7f55ba200640 1 --2- 172.21.15.102:0/1779996848 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f559006ac30 0x7f559006d0f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.522+0000 7f55ba200640 1 --2- 172.21.15.102:0/1779996848 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f55b415cda0 0x7f55b4159d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.522+0000 7f55ba200640 1 --2- 172.21.15.102:0/1779996848 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b40b5a30 0x7f55b40b2470 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.522+0000 7f55ba200640 1 --2- 172.21.15.102:0/1779996848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55b40b4350 0x7f55b41f6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.522+0000 7f55ba200640 1 -- 172.21.15.102:0/1779996848 >> 172.21.15.102:0/1779996848 conn(0x7f55b40cf350 msgr2=0x7f55b40cb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:53.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.522+0000 7f55ba200640 1 -- 172.21.15.102:0/1779996848 shutdown_connections 2024-03-23T03:48:53.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.522+0000 7f55ba200640 1 -- 172.21.15.102:0/1779996848 wait complete. 2024-03-23T03:48:53.541 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T03:48:53.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T03:48:53.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.739+0000 7f4502a00640 1 Processor -- start 2024-03-23T03:48:53.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.739+0000 7f4502a00640 1 -- start start 2024-03-23T03:48:53.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.740+0000 7f4502a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f44fc15c820 0x7f44fc0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:53.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.741+0000 7f4502a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44fc0ccd90 con 0x7f44fc0cf7c0 2024-03-23T03:48:53.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.741+0000 7f4502a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44fc0b7fc0 con 0x7f44fc160e90 2024-03-23T03:48:53.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.741+0000 7f4502a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc0cdd70 con 0x7f44fc15c820 2024-03-23T03:48:53.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.741+0000 7f4502000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f44fc0cf7c0 0x7f44fc0d0560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52538/0 (socket says 172.21.15.102:52538) 2024-03-23T03:48:53.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.741+0000 7f4502000640 1 -- 172.21.15.102:0/751430553 learned_addr learned my addr 172.21.15.102:0/751430553 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:53.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.741+0000 7f4501600640 1 --2- 172.21.15.102:0/751430553 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f44fc15c820 0x7f44fc0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:53.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.741+0000 7f44fbe00640 1 -- 172.21.15.102:0/751430553 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882134046 0 0) 0x7f44fc0ccd90 con 0x7f44fc0cf7c0 2024-03-23T03:48:53.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.741+0000 7f44fbe00640 1 -- 172.21.15.102:0/751430553 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44e0003520 con 0x7f44fc0cf7c0 2024-03-23T03:48:53.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f44fbe00640 1 -- 172.21.15.102:0/751430553 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 997047920 0 0) 0x7f44e0003520 con 0x7f44fc0cf7c0 2024-03-23T03:48:53.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f44fbe00640 1 -- 172.21.15.102:0/751430553 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f44fc15c820 msgr2=0x7f44fc0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:53.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f44fbe00640 1 --2- 172.21.15.102:0/751430553 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f44fc15c820 0x7f44fc0cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f44fbe00640 1 -- 172.21.15.102:0/751430553 >> v1:172.21.15.173:6789/0 conn(0x7f44fc160e90 legacy=0x7f44fc0cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:53.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f44fbe00640 1 -- 172.21.15.102:0/751430553 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc0b82a0 con 0x7f44fc0cf7c0 2024-03-23T03:48:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f44fbe00640 1 -- 172.21.15.102:0/751430553 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f44f40019f0 con 0x7f44fc0cf7c0 2024-03-23T03:48:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f44fbe00640 1 -- 172.21.15.102:0/751430553 >> v1:172.21.15.102:6789/0 conn(0x7f44fc0cf7c0 legacy=0x7f44fc0d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f44fbe00640 1 --2- 172.21.15.102:0/751430553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44e00039a0 0x7f44e0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f44fbe00640 1 --2- 172.21.15.102:0/751430553 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f44fc15c820 0x7f44e00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f44fbe00640 1 --2- 172.21.15.102:0/751430553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44e0004d00 0x7f44e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f44fbe00640 1 -- 172.21.15.102:0/751430553 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc0cdd70 con 0x7f44e00039a0 2024-03-23T03:48:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f44fbe00640 1 -- 172.21.15.102:0/751430553 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e000bac0 con 0x7f44e0004d00 2024-03-23T03:48:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f44fbe00640 1 -- 172.21.15.102:0/751430553 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e000bc80 con 0x7f44fc15c820 2024-03-23T03:48:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f4501600640 1 --2- 172.21.15.102:0/751430553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44e00039a0 0x7f44e0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f4501600640 1 -- 172.21.15.102:0/751430553 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f44fc15c820 msgr2=0x7f44e00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f4501600640 1 --2- 172.21.15.102:0/751430553 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f44fc15c820 0x7f44e00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f4501600640 1 -- 172.21.15.102:0/751430553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44e0004d00 msgr2=0x7f44e0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:53.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f4501600640 1 --2- 172.21.15.102:0/751430553 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44e0004d00 0x7f44e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.742+0000 7f4501600640 1 -- 172.21.15.102:0/751430553 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e0003520 con 0x7f44e00039a0 2024-03-23T03:48:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.743+0000 7f4501600640 1 --2- 172.21.15.102:0/751430553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44e00039a0 0x7f44e0003dc0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009a90 tx=0x7f44e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=357306d2cfbeac1e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.744+0000 7f44fbe00640 1 -- 172.21.15.102:0/751430553 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec02c6b0 con 0x7f44e00039a0 2024-03-23T03:48:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.744+0000 7f44fbe00640 1 -- 172.21.15.102:0/751430553 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ec02c810 con 0x7f44e00039a0 2024-03-23T03:48:53.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.744+0000 7f44fbe00640 1 -- 172.21.15.102:0/751430553 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ec02cba0 con 0x7f44e00039a0 2024-03-23T03:48:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.744+0000 7f4502a00640 1 -- 172.21.15.102:0/751430553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44e00039a0 msgr2=0x7f44e0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.744+0000 7f4502a00640 1 --2- 172.21.15.102:0/751430553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44e00039a0 0x7f44e0003dc0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f44ec009a90 tx=0x7f44e00057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.744+0000 7f4502000640 1 -- 172.21.15.102:0/751430553 reap_dead start 2024-03-23T03:48:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.744+0000 7f4502a00640 1 -- 172.21.15.102:0/751430553 shutdown_connections 2024-03-23T03:48:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.744+0000 7f4502a00640 1 -- 172.21.15.102:0/751430553 >> 172.21.15.102:0/751430553 conn(0x7f44fc06b050 msgr2=0x7f44fc06b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.744+0000 7f4502a00640 1 -- 172.21.15.102:0/751430553 shutdown_connections 2024-03-23T03:48:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.744+0000 7f4502a00640 1 -- 172.21.15.102:0/751430553 wait complete. 2024-03-23T03:48:53.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.746+0000 7f4502a00640 1 Processor -- start 2024-03-23T03:48:53.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.746+0000 7f4502a00640 1 -- start start 2024-03-23T03:48:53.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.747+0000 7f4502a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f44e00039a0 0x7f44fc1526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:53.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.747+0000 7f4502a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44fc150e50 0x7f44fc146860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:53.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.747+0000 7f4502a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44fc154050 0x7f44fc157190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:53.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.747+0000 7f4502a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc152fb0 con 0x7f44fc150e50 2024-03-23T03:48:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.747+0000 7f4502a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc153290 con 0x7f44fc154050 2024-03-23T03:48:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.747+0000 7f4502a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc153570 con 0x7f44e00039a0 2024-03-23T03:48:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.747+0000 7f4500c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44fc150e50 0x7f44fc146860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.747+0000 7f4500c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44fc150e50 0x7f44fc146860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41734/0 (socket says 172.21.15.102:41734) 2024-03-23T03:48:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.747+0000 7f4500c00640 1 -- 172.21.15.102:0/2901576129 learned_addr learned my addr 172.21.15.102:0/2901576129 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.748+0000 7f4500c00640 1 -- 172.21.15.102:0/2901576129 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f44e00039a0 msgr2=0x7f44fc1526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.748+0000 7f4500c00640 1 --2- 172.21.15.102:0/2901576129 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f44e00039a0 0x7f44fc1526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.748+0000 7f4500c00640 1 -- 172.21.15.102:0/2901576129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44fc154050 msgr2=0x7f44fc157190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.748+0000 7f4500c00640 1 --2- 172.21.15.102:0/2901576129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44fc154050 0x7f44fc157190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.748+0000 7f4500c00640 1 -- 172.21.15.102:0/2901576129 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc153850 con 0x7f44fc150e50 2024-03-23T03:48:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.748+0000 7f4500c00640 1 --2- 172.21.15.102:0/2901576129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44fc150e50 0x7f44fc146860 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f44e800c500 tx=0x7f44e800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.748+0000 7f44f9600640 1 -- 172.21.15.102:0/2901576129 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e80052d0 con 0x7f44fc150e50 2024-03-23T03:48:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.748+0000 7f44f9600640 1 -- 172.21.15.102:0/2901576129 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e8005430 con 0x7f44fc150e50 2024-03-23T03:48:53.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.748+0000 7f44f9600640 1 -- 172.21.15.102:0/2901576129 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e8012870 con 0x7f44fc150e50 2024-03-23T03:48:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.748+0000 7f4502a00640 1 -- 172.21.15.102:0/2901576129 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44fc153b30 con 0x7f44fc150e50 2024-03-23T03:48:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.748+0000 7f4502a00640 1 -- 172.21.15.102:0/2901576129 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44fc146da0 con 0x7f44fc150e50 2024-03-23T03:48:53.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.749+0000 7f4502a00640 1 -- 172.21.15.102:0/2901576129 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44fc04e030 con 0x7f44fc150e50 2024-03-23T03:48:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.750+0000 7f44f9600640 1 -- 172.21.15.102:0/2901576129 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f44e80129d0 con 0x7f44fc150e50 2024-03-23T03:48:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.751+0000 7f44f9600640 1 --2- 172.21.15.102:0/2901576129 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f44dc06ac30 0x7f44dc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.751+0000 7f44f9600640 1 -- 172.21.15.102:0/2901576129 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f44e8088ca0 con 0x7f44fc150e50 2024-03-23T03:48:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.751+0000 7f4501600640 1 --2- 172.21.15.102:0/2901576129 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f44dc06ac30 0x7f44dc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:53.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.752+0000 7f4501600640 1 --2- 172.21.15.102:0/2901576129 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f44dc06ac30 0x7f44dc06d0f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f44ec036000 tx=0x7f44ec02e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:53.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.755+0000 7f44f9600640 1 -- 172.21.15.102:0/2901576129 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f44e8054120 con 0x7f44fc150e50 2024-03-23T03:48:53.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:53.947+0000 7f4502a00640 1 -- 172.21.15.102:0/2901576129 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f44fc0d15a0 con 0x7f44fc150e50 2024-03-23T03:48:54.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.743+0000 7f44f9600640 1 -- 172.21.15.102:0/2901576129 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v52) v1 ==== 138+0+0 (secure 0 0 0) 0x7f44e8057740 con 0x7f44fc150e50 2024-03-23T03:48:54.745 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' created 2024-03-23T03:48:54.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.749+0000 7f4502a00640 1 -- 172.21.15.102:0/2901576129 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f44dc06ac30 msgr2=0x7f44dc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.749+0000 7f4502a00640 1 --2- 172.21.15.102:0/2901576129 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f44dc06ac30 0x7f44dc06d0f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f44ec036000 tx=0x7f44ec02e000 comp rx=0 tx=0).stop 2024-03-23T03:48:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.749+0000 7f4502a00640 1 -- 172.21.15.102:0/2901576129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44fc150e50 msgr2=0x7f44fc146860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.749+0000 7f4502a00640 1 --2- 172.21.15.102:0/2901576129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44fc150e50 0x7f44fc146860 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f44e800c500 tx=0x7f44e800c9c0 comp rx=0 tx=0).stop 2024-03-23T03:48:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.749+0000 7f4502a00640 1 -- 172.21.15.102:0/2901576129 shutdown_connections 2024-03-23T03:48:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.749+0000 7f4502a00640 1 --2- 172.21.15.102:0/2901576129 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f44dc06ac30 0x7f44dc06d0f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.749+0000 7f4502a00640 1 --2- 172.21.15.102:0/2901576129 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44fc154050 0x7f44fc157190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.749+0000 7f4502a00640 1 --2- 172.21.15.102:0/2901576129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44fc150e50 0x7f44fc146860 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:54.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.749+0000 7f4502a00640 1 --2- 172.21.15.102:0/2901576129 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f44e00039a0 0x7f44fc1526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:54.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.749+0000 7f4502a00640 1 -- 172.21.15.102:0/2901576129 >> 172.21.15.102:0/2901576129 conn(0x7f44fc06b050 msgr2=0x7f44fc0cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:54.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.750+0000 7f4502a00640 1 -- 172.21.15.102:0/2901576129 shutdown_connections 2024-03-23T03:48:54.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.750+0000 7f4502a00640 1 -- 172.21.15.102:0/2901576129 wait complete. 2024-03-23T03:48:54.772 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T03:48:54.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.976+0000 7feea7200640 1 Processor -- start 2024-03-23T03:48:54.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.976+0000 7feea7200640 1 -- start start 2024-03-23T03:48:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.977+0000 7feea7200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7feea00ce930 0x7feea00add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.977+0000 7feea7200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feea00cd3e0 0x7feea00ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.977+0000 7feea7200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea00cf250 con 0x7feea006ce90 2024-03-23T03:48:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.977+0000 7feea7200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feea006c760 con 0x7feea00ce930 2024-03-23T03:48:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.977+0000 7feea7200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea00b42d0 con 0x7feea00cd3e0 2024-03-23T03:48:54.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.978+0000 7fee97400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feea00cd3e0 0x7feea00ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:54.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.978+0000 7fee97400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feea00cd3e0 0x7feea00ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:34426/0 (socket says 172.21.15.102:34426) 2024-03-23T03:48:54.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.978+0000 7fee97400640 1 -- 172.21.15.102:0/4226076961 learned_addr learned my addr 172.21.15.102:0/4226076961 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:54.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.978+0000 7feea5200640 1 --1- 172.21.15.102:0/4226076961 >> v1:172.21.15.173:6789/0 conn(0x7feea006ce90 0x7feea015e290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:42026/0 (socket says 172.21.15.102:42026) 2024-03-23T03:48:54.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.979+0000 7fee9fe00640 1 --2- 172.21.15.102:0/4226076961 >> v2:172.21.15.173:3300/0 conn(0x7feea00ce930 0x7feea00add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:54.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.979+0000 7fee97400640 1 -- 172.21.15.102:0/4226076961 >> v2:172.21.15.173:3300/0 conn(0x7feea00ce930 msgr2=0x7feea00add30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:54.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.979+0000 7fee97400640 1 --2- 172.21.15.102:0/4226076961 >> v2:172.21.15.173:3300/0 conn(0x7feea00ce930 0x7feea00add30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:54.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.979+0000 7fee97400640 1 -- 172.21.15.102:0/4226076961 >> v1:172.21.15.173:6789/0 conn(0x7feea006ce90 legacy=0x7feea015e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:54.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.979+0000 7fee97400640 1 -- 172.21.15.102:0/4226076961 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea00cdba0 con 0x7feea00cd3e0 2024-03-23T03:48:54.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.981+0000 7fee97400640 1 --2- 172.21.15.102:0/4226076961 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feea00cd3e0 0x7feea00ced10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fee88009870 tx=0x7fee8802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b35118b4551846cd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:54.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.981+0000 7fee9f400640 1 -- 172.21.15.102:0/4226076961 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee88030840 con 0x7feea00cd3e0 2024-03-23T03:48:54.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.982+0000 7fee9f400640 1 -- 172.21.15.102:0/4226076961 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee880309a0 con 0x7feea00cd3e0 2024-03-23T03:48:54.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.982+0000 7fee9f400640 1 -- 172.21.15.102:0/4226076961 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee88030d30 con 0x7feea00cd3e0 2024-03-23T03:48:54.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.982+0000 7feea7200640 1 -- 172.21.15.102:0/4226076961 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feea00cd3e0 msgr2=0x7feea00ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:54.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.982+0000 7feea7200640 1 --2- 172.21.15.102:0/4226076961 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feea00cd3e0 0x7feea00ced10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fee88009870 tx=0x7fee8802b980 comp rx=0 tx=0).stop 2024-03-23T03:48:54.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.982+0000 7feea7200640 1 -- 172.21.15.102:0/4226076961 shutdown_connections 2024-03-23T03:48:54.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.982+0000 7feea7200640 1 --2- 172.21.15.102:0/4226076961 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feea00cd3e0 0x7feea00ced10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:54.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.982+0000 7feea7200640 1 --2- 172.21.15.102:0/4226076961 >> v2:172.21.15.173:3300/0 conn(0x7feea00ce930 0x7feea00add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:54.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.982+0000 7feea7200640 1 -- 172.21.15.102:0/4226076961 >> 172.21.15.102:0/4226076961 conn(0x7feea006a0b0 msgr2=0x7feea006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:54.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.982+0000 7feea7200640 1 -- 172.21.15.102:0/4226076961 shutdown_connections 2024-03-23T03:48:54.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.982+0000 7feea7200640 1 -- 172.21.15.102:0/4226076961 wait complete. 2024-03-23T03:48:54.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.984+0000 7feea7200640 1 Processor -- start 2024-03-23T03:48:54.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.984+0000 7feea7200640 1 -- start start 2024-03-23T03:48:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.985+0000 7feea7200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feea006ce90 0x7feea0148050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.985+0000 7feea7200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feea00cd3e0 0x7feea00c3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.985+0000 7feea7200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feea00ce930 0x7feea00c4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.985+0000 7feea7200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea00c3880 con 0x7feea006ce90 2024-03-23T03:48:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.985+0000 7feea7200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea0159bf0 con 0x7feea00cd3e0 2024-03-23T03:48:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.985+0000 7feea7200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea0159ed0 con 0x7feea00ce930 2024-03-23T03:48:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.986+0000 7fee9fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feea006ce90 0x7feea0148050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.986+0000 7fee9fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feea006ce90 0x7feea0148050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41738/0 (socket says 172.21.15.102:41738) 2024-03-23T03:48:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.986+0000 7fee9fe00640 1 -- 172.21.15.102:0/3109931310 learned_addr learned my addr 172.21.15.102:0/3109931310 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.986+0000 7fee97400640 1 --2- 172.21.15.102:0/3109931310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feea00cd3e0 0x7feea00c3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.986+0000 7feea5200640 1 --2- 172.21.15.102:0/3109931310 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feea00ce930 0x7feea00c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:54.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.986+0000 7fee9fe00640 1 -- 172.21.15.102:0/3109931310 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feea00ce930 msgr2=0x7feea00c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:54.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.986+0000 7fee9fe00640 1 --2- 172.21.15.102:0/3109931310 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feea00ce930 0x7feea00c4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:54.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.986+0000 7fee9fe00640 1 -- 172.21.15.102:0/3109931310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feea00cd3e0 msgr2=0x7feea00c3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:54.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.986+0000 7fee9fe00640 1 --2- 172.21.15.102:0/3109931310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feea00cd3e0 0x7feea00c3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:54.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.986+0000 7fee9fe00640 1 -- 172.21.15.102:0/3109931310 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee880094d0 con 0x7feea006ce90 2024-03-23T03:48:54.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.986+0000 7feea5200640 1 --2- 172.21.15.102:0/3109931310 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feea00ce930 0x7feea00c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:54.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.986+0000 7fee9fe00640 1 --2- 172.21.15.102:0/3109931310 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feea006ce90 0x7feea0148050 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fee9000e9b0 tx=0x7fee9000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:54.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.987+0000 7fee9cc00640 1 -- 172.21.15.102:0/3109931310 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee90009670 con 0x7feea006ce90 2024-03-23T03:48:54.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.987+0000 7fee9cc00640 1 -- 172.21.15.102:0/3109931310 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9000c9a0 con 0x7feea006ce90 2024-03-23T03:48:54.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.987+0000 7feea7200640 1 -- 172.21.15.102:0/3109931310 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea015a1b0 con 0x7feea006ce90 2024-03-23T03:48:54.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.987+0000 7fee9cc00640 1 -- 172.21.15.102:0/3109931310 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9000cd50 con 0x7feea006ce90 2024-03-23T03:48:54.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.987+0000 7feea7200640 1 -- 172.21.15.102:0/3109931310 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea015a770 con 0x7feea006ce90 2024-03-23T03:48:54.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.988+0000 7feea7200640 1 -- 172.21.15.102:0/3109931310 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea0073390 con 0x7feea006ce90 2024-03-23T03:48:54.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.989+0000 7fee9cc00640 1 -- 172.21.15.102:0/3109931310 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fee90013470 con 0x7feea006ce90 2024-03-23T03:48:54.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.990+0000 7fee9cc00640 1 --2- 172.21.15.102:0/3109931310 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fee7c06ac30 0x7fee7c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:54.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.990+0000 7fee9cc00640 1 -- 172.21.15.102:0/3109931310 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5547+0+0 (secure 0 0 0) 0x7fee90089500 con 0x7feea006ce90 2024-03-23T03:48:54.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.990+0000 7fee97400640 1 --2- 172.21.15.102:0/3109931310 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fee7c06ac30 0x7fee7c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:54.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.991+0000 7fee97400640 1 --2- 172.21.15.102:0/3109931310 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fee7c06ac30 0x7fee7c06d0f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fee88036000 tx=0x7fee8803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:54.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:54.995+0000 7fee9cc00640 1 -- 172.21.15.102:0/3109931310 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fee90054780 con 0x7feea006ce90 2024-03-23T03:48:55.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.189+0000 7feea7200640 1 -- 172.21.15.102:0/3109931310 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7feea00b0940 con 0x7feea006ce90 2024-03-23T03:48:55.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.750+0000 7fee9cc00640 1 -- 172.21.15.102:0/3109931310 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v53) v1 ==== 170+0+0 (secure 0 0 0) 0x7fee90057da0 con 0x7feea006ce90 2024-03-23T03:48:55.751 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' created 2024-03-23T03:48:55.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.754+0000 7feea7200640 1 -- 172.21.15.102:0/3109931310 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fee7c06ac30 msgr2=0x7fee7c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:55.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.755+0000 7feea7200640 1 --2- 172.21.15.102:0/3109931310 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fee7c06ac30 0x7fee7c06d0f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fee88036000 tx=0x7fee8803b040 comp rx=0 tx=0).stop 2024-03-23T03:48:55.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.755+0000 7feea7200640 1 -- 172.21.15.102:0/3109931310 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feea006ce90 msgr2=0x7feea0148050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:55.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.755+0000 7feea7200640 1 --2- 172.21.15.102:0/3109931310 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feea006ce90 0x7feea0148050 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fee9000e9b0 tx=0x7fee9000ee70 comp rx=0 tx=0).stop 2024-03-23T03:48:55.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.756+0000 7feea7200640 1 -- 172.21.15.102:0/3109931310 shutdown_connections 2024-03-23T03:48:55.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.756+0000 7feea7200640 1 --2- 172.21.15.102:0/3109931310 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fee7c06ac30 0x7fee7c06d0f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:55.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.756+0000 7feea7200640 1 --2- 172.21.15.102:0/3109931310 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feea00ce930 0x7feea00c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:55.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.756+0000 7feea7200640 1 --2- 172.21.15.102:0/3109931310 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feea00cd3e0 0x7feea00c3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:55.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.756+0000 7feea7200640 1 --2- 172.21.15.102:0/3109931310 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feea006ce90 0x7feea0148050 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:55.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.756+0000 7feea7200640 1 -- 172.21.15.102:0/3109931310 >> 172.21.15.102:0/3109931310 conn(0x7feea006a0b0 msgr2=0x7feea00ac490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:55.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.760+0000 7feea7200640 1 -- 172.21.15.102:0/3109931310 shutdown_connections 2024-03-23T03:48:55.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.760+0000 7feea7200640 1 -- 172.21.15.102:0/3109931310 wait complete. 2024-03-23T03:48:55.792 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T03:48:55.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.971+0000 7f99b6800640 1 Processor -- start 2024-03-23T03:48:55.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.971+0000 7f99b6800640 1 -- start start 2024-03-23T03:48:55.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.973+0000 7f99b6800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f99b00cd740 0x7f99b00b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:55.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.973+0000 7f99b6800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f99b00b5aa0 0x7f99b00b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:55.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.973+0000 7f99b6800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b00d20c0 con 0x7f99b00d0d40 2024-03-23T03:48:55.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.973+0000 7f99b6800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b00d14b0 con 0x7f99b00cd740 2024-03-23T03:48:55.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.973+0000 7f99b6800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b00d1790 con 0x7f99b00b5aa0 2024-03-23T03:48:55.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.974+0000 7f99afe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f99b00b5aa0 0x7f99b00b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:55.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.974+0000 7f99aea00640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f99b00d0d40 0x7f99b00cdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:42042/0 (socket says 172.21.15.102:42042) 2024-03-23T03:48:55.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.974+0000 7f99aea00640 1 -- 172.21.15.102:0/2677703420 learned_addr learned my addr 172.21.15.102:0/2677703420 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:55.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.974+0000 7f99af400640 1 --2- 172.21.15.102:0/2677703420 >> v2:172.21.15.173:3300/0 conn(0x7f99b00cd740 0x7f99b00b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:55.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.974+0000 7f99afe00640 1 -- 172.21.15.102:0/2677703420 >> v2:172.21.15.173:3300/0 conn(0x7f99b00cd740 msgr2=0x7f99b00b84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:55.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.974+0000 7f99afe00640 1 --2- 172.21.15.102:0/2677703420 >> v2:172.21.15.173:3300/0 conn(0x7f99b00cd740 0x7f99b00b84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:55.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.974+0000 7f99afe00640 1 -- 172.21.15.102:0/2677703420 >> v1:172.21.15.173:6789/0 conn(0x7f99b00d0d40 legacy=0x7f99b00cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:55.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.974+0000 7f99afe00640 1 -- 172.21.15.102:0/2677703420 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b00d1a70 con 0x7f99b00b5aa0 2024-03-23T03:48:55.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.976+0000 7f99afe00640 1 --2- 172.21.15.102:0/2677703420 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f99b00b5aa0 0x7f99b00b5f30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f99a80097a0 tx=0x7f99a802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=421971dda0a08ced server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:55.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.976+0000 7f99ae000640 1 -- 172.21.15.102:0/2677703420 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a802d040 con 0x7f99b00b5aa0 2024-03-23T03:48:55.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.976+0000 7f99ae000640 1 -- 172.21.15.102:0/2677703420 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a8031810 con 0x7f99b00b5aa0 2024-03-23T03:48:55.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.976+0000 7f99ae000640 1 -- 172.21.15.102:0/2677703420 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a8031ba0 con 0x7f99b00b5aa0 2024-03-23T03:48:55.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.977+0000 7f99b6800640 1 -- 172.21.15.102:0/2677703420 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f99b00b5aa0 msgr2=0x7f99b00b5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:55.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.977+0000 7f99b6800640 1 --2- 172.21.15.102:0/2677703420 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f99b00b5aa0 0x7f99b00b5f30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f99a80097a0 tx=0x7f99a802b8b0 comp rx=0 tx=0).stop 2024-03-23T03:48:55.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.977+0000 7f99b6800640 1 -- 172.21.15.102:0/2677703420 shutdown_connections 2024-03-23T03:48:55.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.977+0000 7f99b6800640 1 --2- 172.21.15.102:0/2677703420 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f99b00b5aa0 0x7f99b00b5f30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:55.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.977+0000 7f99b6800640 1 --2- 172.21.15.102:0/2677703420 >> v2:172.21.15.173:3300/0 conn(0x7f99b00cd740 0x7f99b00b84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:55.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.977+0000 7f99b6800640 1 -- 172.21.15.102:0/2677703420 >> 172.21.15.102:0/2677703420 conn(0x7f99b00cf5d0 msgr2=0x7f99b00cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:55.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.977+0000 7f99b6800640 1 -- 172.21.15.102:0/2677703420 shutdown_connections 2024-03-23T03:48:55.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.977+0000 7f99b6800640 1 -- 172.21.15.102:0/2677703420 wait complete. 2024-03-23T03:48:55.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.979+0000 7f99b6800640 1 Processor -- start 2024-03-23T03:48:55.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.979+0000 7f99b6800640 1 -- start start 2024-03-23T03:48:55.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.980+0000 7f99b6800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b00b5aa0 0x7f99b01f1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:55.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.980+0000 7f99b6800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99b00cd740 0x7f99b00b3450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:55.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.980+0000 7f99b6800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f99b00d0d40 0x7f99b00b3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:55.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.980+0000 7f99b6800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b01f6540 con 0x7f99b00b5aa0 2024-03-23T03:48:55.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.980+0000 7f99b6800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b01f6260 con 0x7f99b00cd740 2024-03-23T03:48:55.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.980+0000 7f99b6800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b0155cf0 con 0x7f99b00d0d40 2024-03-23T03:48:55.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.980+0000 7f99af400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b00b5aa0 0x7f99b01f1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:55.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.980+0000 7f99af400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b00b5aa0 0x7f99b01f1780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41744/0 (socket says 172.21.15.102:41744) 2024-03-23T03:48:55.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.980+0000 7f99af400640 1 -- 172.21.15.102:0/3631413990 learned_addr learned my addr 172.21.15.102:0/3631413990 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:55.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.981+0000 7f99aea00640 1 --2- 172.21.15.102:0/3631413990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99b00cd740 0x7f99b00b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:55.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.981+0000 7f99af400640 1 -- 172.21.15.102:0/3631413990 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f99b00d0d40 msgr2=0x7f99b00b3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:55.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.981+0000 7f99afe00640 1 --2- 172.21.15.102:0/3631413990 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f99b00d0d40 0x7f99b00b3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:55.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.981+0000 7f99af400640 1 --2- 172.21.15.102:0/3631413990 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f99b00d0d40 0x7f99b00b3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:55.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.981+0000 7f99af400640 1 -- 172.21.15.102:0/3631413990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99b00cd740 msgr2=0x7f99b00b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:55.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.981+0000 7f99af400640 1 --2- 172.21.15.102:0/3631413990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99b00cd740 0x7f99b00b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:55.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.981+0000 7f99af400640 1 -- 172.21.15.102:0/3631413990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a8009400 con 0x7f99b00b5aa0 2024-03-23T03:48:55.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.981+0000 7f99aea00640 1 --2- 172.21.15.102:0/3631413990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99b00cd740 0x7f99b00b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:55.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.981+0000 7f99afe00640 1 --2- 172.21.15.102:0/3631413990 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f99b00d0d40 0x7f99b00b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:55.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.981+0000 7f99af400640 1 --2- 172.21.15.102:0/3631413990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b00b5aa0 0x7f99b01f1780 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f99a000efc0 tx=0x7f99a000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:55.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.981+0000 7f99a7400640 1 -- 172.21.15.102:0/3631413990 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a0014480 con 0x7f99b00b5aa0 2024-03-23T03:48:55.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.982+0000 7f99a7400640 1 -- 172.21.15.102:0/3631413990 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a000f040 con 0x7f99b00b5aa0 2024-03-23T03:48:55.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.982+0000 7f99b6800640 1 -- 172.21.15.102:0/3631413990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b0155fd0 con 0x7f99b00b5aa0 2024-03-23T03:48:55.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.982+0000 7f99a7400640 1 -- 172.21.15.102:0/3631413990 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a0014800 con 0x7f99b00b5aa0 2024-03-23T03:48:55.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.982+0000 7f99b6800640 1 -- 172.21.15.102:0/3631413990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b01f1cc0 con 0x7f99b00b5aa0 2024-03-23T03:48:55.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.983+0000 7f99b6800640 1 -- 172.21.15.102:0/3631413990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b0073390 con 0x7f99b00b5aa0 2024-03-23T03:48:55.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.984+0000 7f99a7400640 1 -- 172.21.15.102:0/3631413990 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f99a0014960 con 0x7f99b00b5aa0 2024-03-23T03:48:55.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.985+0000 7f99a7400640 1 --2- 172.21.15.102:0/3631413990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f999006ac00 0x7f999006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:55.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.985+0000 7f99a7400640 1 -- 172.21.15.102:0/3631413990 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f99a008a000 con 0x7f99b00b5aa0 2024-03-23T03:48:55.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.985+0000 7f99aea00640 1 --2- 172.21.15.102:0/3631413990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f999006ac00 0x7f999006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:55.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.986+0000 7f99aea00640 1 --2- 172.21.15.102:0/3631413990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f999006ac00 0x7f999006d0c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f99b01f6710 tx=0x7f999c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:55.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:55.992+0000 7f99a7400640 1 -- 172.21.15.102:0/3631413990 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f99a00550f0 con 0x7f99b00b5aa0 2024-03-23T03:48:56.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.219+0000 7f99b6800640 1 -- 172.21.15.102:0/3631413990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f99b00b3230 con 0x7f99b00b5aa0 2024-03-23T03:48:56.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.231+0000 7f99a7400640 1 -- 172.21.15.102:0/3631413990 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v20) v1 ==== 172+0+0 (secure 0 0 0) 0x7f99a0058710 con 0x7f99b00b5aa0 2024-03-23T03:48:56.232 INFO:teuthology.orchestra.run.smithi102.stderr:new fs with metadata pool 7 and data pool 8 2024-03-23T03:48:56.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.235+0000 7f99b6800640 1 -- 172.21.15.102:0/3631413990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f999006ac00 msgr2=0x7f999006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:56.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.235+0000 7f99b6800640 1 --2- 172.21.15.102:0/3631413990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f999006ac00 0x7f999006d0c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f99b01f6710 tx=0x7f999c009040 comp rx=0 tx=0).stop 2024-03-23T03:48:56.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.235+0000 7f99b6800640 1 -- 172.21.15.102:0/3631413990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b00b5aa0 msgr2=0x7f99b01f1780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:56.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.235+0000 7f99b6800640 1 --2- 172.21.15.102:0/3631413990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b00b5aa0 0x7f99b01f1780 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f99a000efc0 tx=0x7f99a000c490 comp rx=0 tx=0).stop 2024-03-23T03:48:56.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.235+0000 7f99b6800640 1 -- 172.21.15.102:0/3631413990 shutdown_connections 2024-03-23T03:48:56.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.235+0000 7f99b6800640 1 --2- 172.21.15.102:0/3631413990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f999006ac00 0x7f999006d0c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.235+0000 7f99b6800640 1 --2- 172.21.15.102:0/3631413990 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f99b00d0d40 0x7f99b00b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.235+0000 7f99b6800640 1 --2- 172.21.15.102:0/3631413990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f99b00cd740 0x7f99b00b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.235+0000 7f99b6800640 1 --2- 172.21.15.102:0/3631413990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b00b5aa0 0x7f99b01f1780 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.235+0000 7f99b6800640 1 -- 172.21.15.102:0/3631413990 >> 172.21.15.102:0/3631413990 conn(0x7f99b00cf5d0 msgr2=0x7f99b00d2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:56.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.237+0000 7f99b6800640 1 -- 172.21.15.102:0/3631413990 shutdown_connections 2024-03-23T03:48:56.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.237+0000 7f99b6800640 1 -- 172.21.15.102:0/3631413990 wait complete. 2024-03-23T03:48:56.267 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:48:56.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.443+0000 7f17b8400640 1 Processor -- start 2024-03-23T03:48:56.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.443+0000 7f17b8400640 1 -- start start 2024-03-23T03:48:56.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.445+0000 7f17b8400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f17b00b4a60 0x7f17b00b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:56.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.445+0000 7f17b8400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b00d1830 con 0x7f17b015cda0 2024-03-23T03:48:56.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.445+0000 7f17b8400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17b00b1f20 con 0x7f17b00b5a30 2024-03-23T03:48:56.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.445+0000 7f17b8400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17b00d0c20 con 0x7f17b00b4a60 2024-03-23T03:48:56.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.445+0000 7f17b5a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f17b015cda0 0x7f17b00cdb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52542/0 (socket says 172.21.15.102:52542) 2024-03-23T03:48:56.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.445+0000 7f17b5a00640 1 -- 172.21.15.102:0/3348402627 learned_addr learned my addr 172.21.15.102:0/3348402627 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:56.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.445+0000 7f17afe00640 1 -- 172.21.15.102:0/3348402627 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939773798 0 0) 0x7f17b00d1830 con 0x7f17b015cda0 2024-03-23T03:48:56.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.445+0000 7f17afe00640 1 -- 172.21.15.102:0/3348402627 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1798003520 con 0x7f17b015cda0 2024-03-23T03:48:56.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.445+0000 7f17b6400640 1 --2- 172.21.15.102:0/3348402627 >> v2:172.21.15.173:3300/0 conn(0x7f17b00b4a60 0x7f17b00b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:56.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.446+0000 7f17afe00640 1 -- 172.21.15.102:0/3348402627 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 962201073 0 0) 0x7f1798003520 con 0x7f17b015cda0 2024-03-23T03:48:56.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.446+0000 7f17afe00640 1 -- 172.21.15.102:0/3348402627 >> v2:172.21.15.173:3300/0 conn(0x7f17b00b4a60 msgr2=0x7f17b00b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:56.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.446+0000 7f17afe00640 1 --2- 172.21.15.102:0/3348402627 >> v2:172.21.15.173:3300/0 conn(0x7f17b00b4a60 0x7f17b00b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.446+0000 7f17afe00640 1 -- 172.21.15.102:0/3348402627 >> v1:172.21.15.173:6789/0 conn(0x7f17b00b5a30 legacy=0x7f17b00b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:56.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.446+0000 7f17afe00640 1 -- 172.21.15.102:0/3348402627 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b00d0f00 con 0x7f17b015cda0 2024-03-23T03:48:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.446+0000 7f17afe00640 1 -- 172.21.15.102:0/3348402627 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f17a80019f0 con 0x7f17b015cda0 2024-03-23T03:48:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.446+0000 7f17afe00640 1 -- 172.21.15.102:0/3348402627 >> v1:172.21.15.102:6789/0 conn(0x7f17b015cda0 legacy=0x7f17b00cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.446+0000 7f17afe00640 1 --2- 172.21.15.102:0/3348402627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1798003970 0x7f1798003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.446+0000 7f17afe00640 1 --2- 172.21.15.102:0/3348402627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f17980047c0 0x7f1798004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.446+0000 7f17afe00640 1 --2- 172.21.15.102:0/3348402627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1798005180 0x7f179800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.446+0000 7f17afe00640 1 -- 172.21.15.102:0/3348402627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b00d0c20 con 0x7f1798005180 2024-03-23T03:48:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.446+0000 7f17afe00640 1 -- 172.21.15.102:0/3348402627 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f179800c020 con 0x7f1798003970 2024-03-23T03:48:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.446+0000 7f17afe00640 1 -- 172.21.15.102:0/3348402627 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f179800c1e0 con 0x7f17980047c0 2024-03-23T03:48:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.446+0000 7f17b6400640 1 --2- 172.21.15.102:0/3348402627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1798005180 0x7f179800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.447+0000 7f17b6400640 1 -- 172.21.15.102:0/3348402627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f17980047c0 msgr2=0x7f1798004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.447+0000 7f17b6400640 1 --2- 172.21.15.102:0/3348402627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f17980047c0 0x7f1798004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.447+0000 7f17b6400640 1 -- 172.21.15.102:0/3348402627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1798003970 msgr2=0x7f1798003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:56.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.447+0000 7f17b6400640 1 --2- 172.21.15.102:0/3348402627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1798003970 0x7f1798003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.447+0000 7f17b6400640 1 -- 172.21.15.102:0/3348402627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1798003520 con 0x7f1798005180 2024-03-23T03:48:56.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.448+0000 7f17b6400640 1 -- 172.21.15.102:0/3348402627 reap_dead start 2024-03-23T03:48:56.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.448+0000 7f17b6400640 1 --2- 172.21.15.102:0/3348402627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1798005180 0x7f179800b980 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f17b008b4f0 tx=0x7f17980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed1e59cb12bc597 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.448+0000 7f17afe00640 1 -- 172.21.15.102:0/3348402627 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a402c680 con 0x7f1798005180 2024-03-23T03:48:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.448+0000 7f17afe00640 1 -- 172.21.15.102:0/3348402627 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a402c800 con 0x7f1798005180 2024-03-23T03:48:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.448+0000 7f17afe00640 1 -- 172.21.15.102:0/3348402627 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a402cbb0 con 0x7f1798005180 2024-03-23T03:48:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.449+0000 7f17b8400640 1 -- 172.21.15.102:0/3348402627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1798005180 msgr2=0x7f179800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.449+0000 7f17b8400640 1 --2- 172.21.15.102:0/3348402627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1798005180 0x7f179800b980 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f17b008b4f0 tx=0x7f17980057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.449+0000 7f17b8400640 1 -- 172.21.15.102:0/3348402627 shutdown_connections 2024-03-23T03:48:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.449+0000 7f17b8400640 1 --2- 172.21.15.102:0/3348402627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1798005180 0x7f179800b980 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.449+0000 7f17b8400640 1 -- 172.21.15.102:0/3348402627 >> 172.21.15.102:0/3348402627 conn(0x7f17b00cf350 msgr2=0x7f17b00cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.449+0000 7f17b8400640 1 -- 172.21.15.102:0/3348402627 shutdown_connections 2024-03-23T03:48:56.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.449+0000 7f17b8400640 1 -- 172.21.15.102:0/3348402627 wait complete. 2024-03-23T03:48:56.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.450+0000 7f17b8400640 1 Processor -- start 2024-03-23T03:48:56.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.451+0000 7f17b8400640 1 -- start start 2024-03-23T03:48:56.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.451+0000 7f17b8400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1798005180 0x7f17b01f6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:56.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.451+0000 7f17b8400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17b00c83b0 0x7f17b015fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:56.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.451+0000 7f17b8400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f17b0142b90 0x7f17b00b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:56.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.451+0000 7f17b8400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b00c8970 con 0x7f1798005180 2024-03-23T03:48:56.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.451+0000 7f17b8400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17b00c8c50 con 0x7f17b00c83b0 2024-03-23T03:48:56.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.451+0000 7f17b8400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17b00c8f30 con 0x7f17b0142b90 2024-03-23T03:48:56.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.452+0000 7f17b5a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1798005180 0x7f17b01f6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:56.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.452+0000 7f17b5a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1798005180 0x7f17b01f6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41752/0 (socket says 172.21.15.102:41752) 2024-03-23T03:48:56.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.452+0000 7f17b5a00640 1 -- 172.21.15.102:0/648153566 learned_addr learned my addr 172.21.15.102:0/648153566 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:56.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.452+0000 7f17b5a00640 1 -- 172.21.15.102:0/648153566 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f17b0142b90 msgr2=0x7f17b00b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:56.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.452+0000 7f17b5000640 1 --2- 172.21.15.102:0/648153566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17b00c83b0 0x7f17b015fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:56.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.452+0000 7f17b5a00640 1 --2- 172.21.15.102:0/648153566 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f17b0142b90 0x7f17b00b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.452+0000 7f17b5a00640 1 -- 172.21.15.102:0/648153566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17b00c83b0 msgr2=0x7f17b015fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:56.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.452+0000 7f17b5a00640 1 --2- 172.21.15.102:0/648153566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17b00c83b0 0x7f17b015fa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.452+0000 7f17b5a00640 1 -- 172.21.15.102:0/648153566 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b0144cc0 con 0x7f1798005180 2024-03-23T03:48:56.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.453+0000 7f17b5a00640 1 --2- 172.21.15.102:0/648153566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1798005180 0x7f17b01f6060 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f17a8002230 tx=0x7f17a800cee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:56.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.453+0000 7f17ad600640 1 -- 172.21.15.102:0/648153566 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a800a7c0 con 0x7f1798005180 2024-03-23T03:48:56.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.453+0000 7f17ad600640 1 -- 172.21.15.102:0/648153566 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a800a920 con 0x7f1798005180 2024-03-23T03:48:56.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.453+0000 7f17b8400640 1 -- 172.21.15.102:0/648153566 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17b00c9210 con 0x7f1798005180 2024-03-23T03:48:56.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.453+0000 7f17ad600640 1 -- 172.21.15.102:0/648153566 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a800acd0 con 0x7f1798005180 2024-03-23T03:48:56.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.453+0000 7f17b8400640 1 -- 172.21.15.102:0/648153566 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17b015ffa0 con 0x7f1798005180 2024-03-23T03:48:56.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.454+0000 7f17b8400640 1 -- 172.21.15.102:0/648153566 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17b0073390 con 0x7f1798005180 2024-03-23T03:48:56.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.455+0000 7f17ad600640 1 -- 172.21.15.102:0/648153566 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f17a80070d0 con 0x7f1798005180 2024-03-23T03:48:56.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.456+0000 7f17ad600640 1 --2- 172.21.15.102:0/648153566 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f179006ac30 0x7f179006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:56.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.456+0000 7f17ad600640 1 -- 172.21.15.102:0/648153566 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f17a8088f90 con 0x7f1798005180 2024-03-23T03:48:56.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.456+0000 7f17b5000640 1 --2- 172.21.15.102:0/648153566 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f179006ac30 0x7f179006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:56.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.457+0000 7f17b5000640 1 --2- 172.21.15.102:0/648153566 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f179006ac30 0x7f179006d0f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f17b01f56b0 tx=0x7f179c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:56.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.460+0000 7f17ad600640 1 -- 172.21.15.102:0/648153566 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f17a8054020 con 0x7f1798005180 2024-03-23T03:48:56.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.653+0000 7f17b8400640 1 -- 172.21.15.102:0/648153566 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17b00b2c50 con 0x7f1798005180 2024-03-23T03:48:56.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.656+0000 7f17ad600640 1 -- 172.21.15.102:0/648153566 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16367 (secure 0 0 0) 0x7f17a8018020 con 0x7f1798005180 2024-03-23T03:48:56.657 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:56.657 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":54,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:56.221520+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:48:53.998390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T03:48:55.242109+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:48:56.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.659+0000 7f17b8400640 1 -- 172.21.15.102:0/648153566 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f179006ac30 msgr2=0x7f179006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:56.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.659+0000 7f17b8400640 1 --2- 172.21.15.102:0/648153566 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f179006ac30 0x7f179006d0f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f17b01f56b0 tx=0x7f179c009040 comp rx=0 tx=0).stop 2024-03-23T03:48:56.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.659+0000 7f17b8400640 1 -- 172.21.15.102:0/648153566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1798005180 msgr2=0x7f17b01f6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:56.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.659+0000 7f17b8400640 1 --2- 172.21.15.102:0/648153566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1798005180 0x7f17b01f6060 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f17a8002230 tx=0x7f17a800cee0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.659+0000 7f17b8400640 1 -- 172.21.15.102:0/648153566 shutdown_connections 2024-03-23T03:48:56.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.659+0000 7f17b8400640 1 --2- 172.21.15.102:0/648153566 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f179006ac30 0x7f179006d0f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.659+0000 7f17b8400640 1 --2- 172.21.15.102:0/648153566 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f17b0142b90 0x7f17b00b3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.659+0000 7f17b8400640 1 --2- 172.21.15.102:0/648153566 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17b00c83b0 0x7f17b015fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.659+0000 7f17b8400640 1 --2- 172.21.15.102:0/648153566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1798005180 0x7f17b01f6060 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.659+0000 7f17b8400640 1 -- 172.21.15.102:0/648153566 >> 172.21.15.102:0/648153566 conn(0x7f17b00cf350 msgr2=0x7f17b015b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:56.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.659+0000 7f17b8400640 1 -- 172.21.15.102:0/648153566 shutdown_connections 2024-03-23T03:48:56.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.659+0000 7f17b8400640 1 -- 172.21.15.102:0/648153566 wait complete. 2024-03-23T03:48:56.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:48:56.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.887+0000 7f89a9000640 1 Processor -- start 2024-03-23T03:48:56.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.887+0000 7f89a9000640 1 -- start start 2024-03-23T03:48:56.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.889+0000 7f89a9000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f89a415b560 0x7f89a415b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:56.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.889+0000 7f89a9000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f89a415d410 0x7f89a40b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:56.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.889+0000 7f89a9000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89a406c6f0 con 0x7f89a40b39b0 2024-03-23T03:48:56.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.889+0000 7f89a9000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89a40b4370 con 0x7f89a415d410 2024-03-23T03:48:56.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.889+0000 7f89a9000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a406c9d0 con 0x7f89a415b560 2024-03-23T03:48:56.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.889+0000 7f89a1400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f89a40b39b0 0x7f89a415e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52550/0 (socket says 172.21.15.102:52550) 2024-03-23T03:48:56.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.889+0000 7f89a1400640 1 -- 172.21.15.102:0/1892320733 learned_addr learned my addr 172.21.15.102:0/1892320733 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:56.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a2800640 1 --2- 172.21.15.102:0/1892320733 >> v2:172.21.15.173:3300/0 conn(0x7f89a415d410 0x7f89a40b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:56.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a1e00640 1 --2- 172.21.15.102:0/1892320733 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f89a415b560 0x7f89a415b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:56.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a0a00640 1 -- 172.21.15.102:0/1892320733 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270088468 0 0) 0x7f89a406c6f0 con 0x7f89a40b39b0 2024-03-23T03:48:56.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a0a00640 1 -- 172.21.15.102:0/1892320733 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8988003520 con 0x7f89a40b39b0 2024-03-23T03:48:56.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a0a00640 1 -- 172.21.15.102:0/1892320733 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1742508678 0 0) 0x7f8988003520 con 0x7f89a40b39b0 2024-03-23T03:48:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a0a00640 1 -- 172.21.15.102:0/1892320733 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f89a415b560 msgr2=0x7f89a415b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a0a00640 1 --2- 172.21.15.102:0/1892320733 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f89a415b560 0x7f89a415b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a0a00640 1 -- 172.21.15.102:0/1892320733 >> v2:172.21.15.173:3300/0 conn(0x7f89a415d410 msgr2=0x7f89a40b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a0a00640 1 --2- 172.21.15.102:0/1892320733 >> v2:172.21.15.173:3300/0 conn(0x7f89a415d410 0x7f89a40b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a0a00640 1 -- 172.21.15.102:0/1892320733 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a406ccb0 con 0x7f89a40b39b0 2024-03-23T03:48:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a0a00640 1 -- 172.21.15.102:0/1892320733 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f8990001670 con 0x7f89a40b39b0 2024-03-23T03:48:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a0a00640 1 -- 172.21.15.102:0/1892320733 >> v1:172.21.15.102:6789/0 conn(0x7f89a40b39b0 legacy=0x7f89a415e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a0a00640 1 --2- 172.21.15.102:0/1892320733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8988003970 0x7f8988003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a0a00640 1 --2- 172.21.15.102:0/1892320733 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f89a415b560 0x7f89880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a0a00640 1 --2- 172.21.15.102:0/1892320733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8988004d00 0x7f8988005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a0a00640 1 -- 172.21.15.102:0/1892320733 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40b4370 con 0x7f8988003970 2024-03-23T03:48:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a0a00640 1 -- 172.21.15.102:0/1892320733 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a406c9d0 con 0x7f8988004d00 2024-03-23T03:48:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.890+0000 7f89a0a00640 1 -- 172.21.15.102:0/1892320733 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f898800b980 con 0x7f89a415b560 2024-03-23T03:48:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.891+0000 7f89a1e00640 1 --2- 172.21.15.102:0/1892320733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8988003970 0x7f8988003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:56.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.891+0000 7f89a1400640 1 --2- 172.21.15.102:0/1892320733 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f89a415b560 0x7f89880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:56.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.891+0000 7f89a2800640 1 --2- 172.21.15.102:0/1892320733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8988004d00 0x7f8988005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:56.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.891+0000 7f89a1e00640 1 -- 172.21.15.102:0/1892320733 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f89a415b560 msgr2=0x7f89880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:56.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.891+0000 7f89a1e00640 1 --2- 172.21.15.102:0/1892320733 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f89a415b560 0x7f89880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.891+0000 7f89a1e00640 1 -- 172.21.15.102:0/1892320733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8988004d00 msgr2=0x7f8988005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:56.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.891+0000 7f89a1e00640 1 --2- 172.21.15.102:0/1892320733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8988004d00 0x7f8988005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.891+0000 7f89a1e00640 1 -- 172.21.15.102:0/1892320733 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8988003520 con 0x7f8988003970 2024-03-23T03:48:56.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.892+0000 7f89a1e00640 1 --2- 172.21.15.102:0/1892320733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8988003970 0x7f8988003d90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8994009ac0 tx=0x7f89880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60152130964de7f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:56.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.892+0000 7f89a0a00640 1 -- 172.21.15.102:0/1892320733 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899402c6b0 con 0x7f8988003970 2024-03-23T03:48:56.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.892+0000 7f89a0a00640 1 -- 172.21.15.102:0/1892320733 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f899402c810 con 0x7f8988003970 2024-03-23T03:48:56.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.892+0000 7f89a0a00640 1 -- 172.21.15.102:0/1892320733 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f899402cba0 con 0x7f8988003970 2024-03-23T03:48:56.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.893+0000 7f89a9000640 1 -- 172.21.15.102:0/1892320733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8988003970 msgr2=0x7f8988003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:56.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.893+0000 7f89a9000640 1 --2- 172.21.15.102:0/1892320733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8988003970 0x7f8988003d90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8994009ac0 tx=0x7f89880057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.893+0000 7f89a2800640 1 -- 172.21.15.102:0/1892320733 reap_dead start 2024-03-23T03:48:56.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.893+0000 7f89a9000640 1 -- 172.21.15.102:0/1892320733 shutdown_connections 2024-03-23T03:48:56.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.893+0000 7f89a9000640 1 -- 172.21.15.102:0/1892320733 >> 172.21.15.102:0/1892320733 conn(0x7f89a406a3d0 msgr2=0x7f89a40b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:56.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.893+0000 7f89a9000640 1 -- 172.21.15.102:0/1892320733 shutdown_connections 2024-03-23T03:48:56.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.893+0000 7f89a9000640 1 -- 172.21.15.102:0/1892320733 wait complete. 2024-03-23T03:48:56.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.895+0000 7f89a9000640 1 Processor -- start 2024-03-23T03:48:56.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.895+0000 7f89a9000640 1 -- start start 2024-03-23T03:48:56.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.896+0000 7f89a9000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8988003970 0x7f89a4154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:56.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.896+0000 7f89a9000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f89a4142ab0 0x7f89a41f6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:56.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.896+0000 7f89a9000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89a41f6590 0x7f89a40abec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:56.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.896+0000 7f89a9000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40aba70 con 0x7f8988003970 2024-03-23T03:48:56.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.896+0000 7f89a9000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89a41f6a10 con 0x7f89a41f6590 2024-03-23T03:48:56.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.896+0000 7f89a9000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89a40a9a80 con 0x7f89a4142ab0 2024-03-23T03:48:56.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.896+0000 7f89a1e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8988003970 0x7f89a4154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:56.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.896+0000 7f89a1e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8988003970 0x7f89a4154770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41758/0 (socket says 172.21.15.102:41758) 2024-03-23T03:48:56.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.896+0000 7f89a1e00640 1 -- 172.21.15.102:0/327806099 learned_addr learned my addr 172.21.15.102:0/327806099 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:56.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.896+0000 7f89a1e00640 1 -- 172.21.15.102:0/327806099 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f89a4142ab0 msgr2=0x7f89a41f6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:56.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.897+0000 7f89a1e00640 1 --2- 172.21.15.102:0/327806099 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f89a4142ab0 0x7f89a41f6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.896+0000 7f89a2800640 1 --2- 172.21.15.102:0/327806099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89a41f6590 0x7f89a40abec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:56.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.897+0000 7f89a1e00640 1 -- 172.21.15.102:0/327806099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89a41f6590 msgr2=0x7f89a40abec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:56.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.897+0000 7f89a1e00640 1 --2- 172.21.15.102:0/327806099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89a41f6590 0x7f89a40abec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:56.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.897+0000 7f89a1e00640 1 -- 172.21.15.102:0/327806099 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89a4154cb0 con 0x7f8988003970 2024-03-23T03:48:56.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.897+0000 7f89a2800640 1 --2- 172.21.15.102:0/327806099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89a41f6590 0x7f89a40abec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:56.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.897+0000 7f89a1e00640 1 --2- 172.21.15.102:0/327806099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8988003970 0x7f89a4154770 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f899402c250 tx=0x7f8994004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:56.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.897+0000 7f899a000640 1 -- 172.21.15.102:0/327806099 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994003810 con 0x7f8988003970 2024-03-23T03:48:56.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.897+0000 7f899a000640 1 -- 172.21.15.102:0/327806099 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8994003970 con 0x7f8988003970 2024-03-23T03:48:56.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.897+0000 7f899a000640 1 -- 172.21.15.102:0/327806099 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8994003ce0 con 0x7f8988003970 2024-03-23T03:48:56.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.897+0000 7f89a9000640 1 -- 172.21.15.102:0/327806099 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89a40a9cd0 con 0x7f8988003970 2024-03-23T03:48:56.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.897+0000 7f89a9000640 1 -- 172.21.15.102:0/327806099 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89a40aa290 con 0x7f8988003970 2024-03-23T03:48:56.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.898+0000 7f89a9000640 1 -- 172.21.15.102:0/327806099 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89a406ac20 con 0x7f8988003970 2024-03-23T03:48:56.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.899+0000 7f899a000640 1 -- 172.21.15.102:0/327806099 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8994034440 con 0x7f8988003970 2024-03-23T03:48:56.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.900+0000 7f899a000640 1 --2- 172.21.15.102:0/327806099 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f898406ac00 0x7f898406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:56.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.900+0000 7f899a000640 1 -- 172.21.15.102:0/327806099 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f89940ab6a0 con 0x7f8988003970 2024-03-23T03:48:56.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.900+0000 7f89a1400640 1 --2- 172.21.15.102:0/327806099 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f898406ac00 0x7f898406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:56.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.901+0000 7f89a1400640 1 --2- 172.21.15.102:0/327806099 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f898406ac00 0x7f898406d0c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8990001eb0 tx=0x7f8990001110 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:56.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:56.905+0000 7f899a000640 1 -- 172.21.15.102:0/327806099 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8994076880 con 0x7f8988003970 2024-03-23T03:48:57.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.096+0000 7f89a9000640 1 -- 172.21.15.102:0/327806099 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89a4073390 con 0x7f8988003970 2024-03-23T03:48:57.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.098+0000 7f899a000640 1 -- 172.21.15.102:0/327806099 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16367 (secure 0 0 0) 0x7f89940044e0 con 0x7f8988003970 2024-03-23T03:48:57.099 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:57.099 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":54,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:56.221520+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:48:53.998390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T03:48:55.242109+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:48:57.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.101+0000 7f89a9000640 1 -- 172.21.15.102:0/327806099 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f898406ac00 msgr2=0x7f898406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:57.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.101+0000 7f89a9000640 1 --2- 172.21.15.102:0/327806099 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f898406ac00 0x7f898406d0c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f8990001eb0 tx=0x7f8990001110 comp rx=0 tx=0).stop 2024-03-23T03:48:57.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.101+0000 7f89a9000640 1 -- 172.21.15.102:0/327806099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8988003970 msgr2=0x7f89a4154770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:57.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.101+0000 7f89a9000640 1 --2- 172.21.15.102:0/327806099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8988003970 0x7f89a4154770 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f899402c250 tx=0x7f8994004730 comp rx=0 tx=0).stop 2024-03-23T03:48:57.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.101+0000 7f89a9000640 1 -- 172.21.15.102:0/327806099 shutdown_connections 2024-03-23T03:48:57.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.101+0000 7f89a9000640 1 --2- 172.21.15.102:0/327806099 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f898406ac00 0x7f898406d0c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:57.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.101+0000 7f89a9000640 1 --2- 172.21.15.102:0/327806099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f89a41f6590 0x7f89a40abec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:57.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.101+0000 7f89a9000640 1 --2- 172.21.15.102:0/327806099 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f89a4142ab0 0x7f89a41f6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:57.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.101+0000 7f89a9000640 1 --2- 172.21.15.102:0/327806099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8988003970 0x7f89a4154770 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:57.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.101+0000 7f89a9000640 1 -- 172.21.15.102:0/327806099 >> 172.21.15.102:0/327806099 conn(0x7f89a406a3d0 msgr2=0x7f89a415d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:57.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.101+0000 7f89a9000640 1 -- 172.21.15.102:0/327806099 shutdown_connections 2024-03-23T03:48:57.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.101+0000 7f89a9000640 1 -- 172.21.15.102:0/327806099 wait complete. 2024-03-23T03:48:57.120 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T03:48:57.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.322+0000 7f5b34600640 1 Processor -- start 2024-03-23T03:48:57.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.322+0000 7f5b34600640 1 -- start start 2024-03-23T03:48:57.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.324+0000 7f5b34600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5b2c0cd740 0x7f5b2c0b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.324+0000 7f5b34600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2c0d20c0 con 0x7f5b2c0d0d40 2024-03-23T03:48:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.324+0000 7f5b34600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2c0d14b0 con 0x7f5b2c0b5aa0 2024-03-23T03:48:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.324+0000 7f5b34600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c0d1790 con 0x7f5b2c0cd740 2024-03-23T03:48:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.324+0000 7f5b31200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5b2c0d0d40 0x7f5b2c0cdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52556/0 (socket says 172.21.15.102:52556) 2024-03-23T03:48:57.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.324+0000 7f5b31200640 1 -- 172.21.15.102:0/2295680273 learned_addr learned my addr 172.21.15.102:0/2295680273 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:57.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.325+0000 7f5b31c00640 1 --2- 172.21.15.102:0/2295680273 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5b2c0cd740 0x7f5b2c0b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:57.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.325+0000 7f5b2be00640 1 -- 172.21.15.102:0/2295680273 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537925219 0 0) 0x7f5b2c0d20c0 con 0x7f5b2c0d0d40 2024-03-23T03:48:57.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.325+0000 7f5b2be00640 1 -- 172.21.15.102:0/2295680273 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b14003520 con 0x7f5b2c0d0d40 2024-03-23T03:48:57.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.325+0000 7f5b2be00640 1 -- 172.21.15.102:0/2295680273 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3300263712 0 0) 0x7f5b14003520 con 0x7f5b2c0d0d40 2024-03-23T03:48:57.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.325+0000 7f5b2be00640 1 -- 172.21.15.102:0/2295680273 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5b2c0cd740 msgr2=0x7f5b2c0b84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:57.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.325+0000 7f5b2be00640 1 --2- 172.21.15.102:0/2295680273 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5b2c0cd740 0x7f5b2c0b84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:57.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.325+0000 7f5b2be00640 1 -- 172.21.15.102:0/2295680273 >> v1:172.21.15.173:6789/0 conn(0x7f5b2c0b5aa0 legacy=0x7f5b2c0b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:57.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.325+0000 7f5b2be00640 1 -- 172.21.15.102:0/2295680273 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2c0d1a70 con 0x7f5b2c0d0d40 2024-03-23T03:48:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.325+0000 7f5b2be00640 1 -- 172.21.15.102:0/2295680273 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f5b1c001670 con 0x7f5b2c0d0d40 2024-03-23T03:48:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.325+0000 7f5b2be00640 1 -- 172.21.15.102:0/2295680273 >> v1:172.21.15.102:6789/0 conn(0x7f5b2c0d0d40 legacy=0x7f5b2c0cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.325+0000 7f5b2be00640 1 --2- 172.21.15.102:0/2295680273 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5b2c0cd740 0x7f5b140039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.325+0000 7f5b2be00640 1 --2- 172.21.15.102:0/2295680273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b14003f70 0x7f5b140047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.325+0000 7f5b2be00640 1 --2- 172.21.15.102:0/2295680273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b14004d00 0x7f5b14005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.325+0000 7f5b2be00640 1 -- 172.21.15.102:0/2295680273 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c0d1790 con 0x7f5b14003f70 2024-03-23T03:48:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.325+0000 7f5b2be00640 1 -- 172.21.15.102:0/2295680273 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1400bac0 con 0x7f5b14004d00 2024-03-23T03:48:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.325+0000 7f5b2be00640 1 -- 172.21.15.102:0/2295680273 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1400bc80 con 0x7f5b2c0cd740 2024-03-23T03:48:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.326+0000 7f5b31200640 1 --2- 172.21.15.102:0/2295680273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b14003f70 0x7f5b140047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.326+0000 7f5b31c00640 1 --2- 172.21.15.102:0/2295680273 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5b2c0cd740 0x7f5b140039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.326+0000 7f5b31200640 1 -- 172.21.15.102:0/2295680273 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5b2c0cd740 msgr2=0x7f5b140039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.326+0000 7f5b31200640 1 --2- 172.21.15.102:0/2295680273 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5b2c0cd740 0x7f5b140039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.326+0000 7f5b31200640 1 -- 172.21.15.102:0/2295680273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b14004d00 msgr2=0x7f5b14005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.326+0000 7f5b31200640 1 --2- 172.21.15.102:0/2295680273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b14004d00 0x7f5b14005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:57.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.326+0000 7f5b31200640 1 -- 172.21.15.102:0/2295680273 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b14003520 con 0x7f5b14003f70 2024-03-23T03:48:57.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.327+0000 7f5b31200640 1 --2- 172.21.15.102:0/2295680273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b14003f70 0x7f5b140047c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00c0e0 tx=0x7f5b140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e68882a87e72757 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:57.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.327+0000 7f5b2be00640 1 -- 172.21.15.102:0/2295680273 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c005ad0 con 0x7f5b14003f70 2024-03-23T03:48:57.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.327+0000 7f5b2be00640 1 -- 172.21.15.102:0/2295680273 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c005c50 con 0x7f5b14003f70 2024-03-23T03:48:57.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.327+0000 7f5b2be00640 1 -- 172.21.15.102:0/2295680273 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c007070 con 0x7f5b14003f70 2024-03-23T03:48:57.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.328+0000 7f5b34600640 1 -- 172.21.15.102:0/2295680273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b14003f70 msgr2=0x7f5b140047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:57.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.328+0000 7f5b34600640 1 --2- 172.21.15.102:0/2295680273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b14003f70 0x7f5b140047c0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c00c0e0 tx=0x7f5b140057f0 comp rx=0 tx=0).stop 2024-03-23T03:48:57.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.328+0000 7f5b32600640 1 -- 172.21.15.102:0/2295680273 reap_dead start 2024-03-23T03:48:57.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.328+0000 7f5b34600640 1 -- 172.21.15.102:0/2295680273 shutdown_connections 2024-03-23T03:48:57.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.328+0000 7f5b34600640 1 -- 172.21.15.102:0/2295680273 >> 172.21.15.102:0/2295680273 conn(0x7f5b2c0cf5d0 msgr2=0x7f5b2c0cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:57.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.328+0000 7f5b34600640 1 -- 172.21.15.102:0/2295680273 shutdown_connections 2024-03-23T03:48:57.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.328+0000 7f5b34600640 1 -- 172.21.15.102:0/2295680273 wait complete. 2024-03-23T03:48:57.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.329+0000 7f5b34600640 1 Processor -- start 2024-03-23T03:48:57.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.329+0000 7f5b34600640 1 -- start start 2024-03-23T03:48:57.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.330+0000 7f5b34600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b14003f70 0x7f5b2c14fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:57.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.330+0000 7f5b34600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5b2c0cbe80 0x7f5b2c14c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:57.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.330+0000 7f5b34600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b2c0cc300 0x7f5b2c14ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:57.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.330+0000 7f5b34600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c1500a0 con 0x7f5b14003f70 2024-03-23T03:48:57.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.330+0000 7f5b34600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c0a95e0 con 0x7f5b2c0cc300 2024-03-23T03:48:57.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.330+0000 7f5b34600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2c0a98c0 con 0x7f5b2c0cbe80 2024-03-23T03:48:57.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.331+0000 7f5b31c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b14003f70 0x7f5b2c14fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:57.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.331+0000 7f5b31c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b14003f70 0x7f5b2c14fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41768/0 (socket says 172.21.15.102:41768) 2024-03-23T03:48:57.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.331+0000 7f5b31c00640 1 -- 172.21.15.102:0/3776290574 learned_addr learned my addr 172.21.15.102:0/3776290574 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.331+0000 7f5b31200640 1 --2- 172.21.15.102:0/3776290574 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5b2c0cbe80 0x7f5b2c14c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.331+0000 7f5b31c00640 1 -- 172.21.15.102:0/3776290574 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5b2c0cbe80 msgr2=0x7f5b2c14c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.331+0000 7f5b31c00640 1 --2- 172.21.15.102:0/3776290574 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5b2c0cbe80 0x7f5b2c14c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.331+0000 7f5b31c00640 1 -- 172.21.15.102:0/3776290574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b2c0cc300 msgr2=0x7f5b2c14ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.331+0000 7f5b31c00640 1 --2- 172.21.15.102:0/3776290574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b2c0cc300 0x7f5b2c14ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.331+0000 7f5b31c00640 1 -- 172.21.15.102:0/3776290574 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b2c0a9ba0 con 0x7f5b14003f70 2024-03-23T03:48:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.331+0000 7f5b31c00640 1 --2- 172.21.15.102:0/3776290574 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b14003f70 0x7f5b2c14fb60 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f5b20010860 tx=0x7f5b20010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.331+0000 7f5b29600640 1 -- 172.21.15.102:0/3776290574 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b20009610 con 0x7f5b14003f70 2024-03-23T03:48:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.331+0000 7f5b29600640 1 -- 172.21.15.102:0/3776290574 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2000eb60 con 0x7f5b14003f70 2024-03-23T03:48:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.331+0000 7f5b29600640 1 -- 172.21.15.102:0/3776290574 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2001c400 con 0x7f5b14003f70 2024-03-23T03:48:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.331+0000 7f5b34600640 1 -- 172.21.15.102:0/3776290574 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b2c0a9e80 con 0x7f5b14003f70 2024-03-23T03:48:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.332+0000 7f5b34600640 1 -- 172.21.15.102:0/3776290574 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b2c0a9270 con 0x7f5b14003f70 2024-03-23T03:48:57.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.333+0000 7f5b34600640 1 -- 172.21.15.102:0/3776290574 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b2c073390 con 0x7f5b14003f70 2024-03-23T03:48:57.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.334+0000 7f5b29600640 1 -- 172.21.15.102:0/3776290574 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5b2001c560 con 0x7f5b14003f70 2024-03-23T03:48:57.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.335+0000 7f5b29600640 1 --2- 172.21.15.102:0/3776290574 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b1006ac30 0x7f5b1006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:57.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.335+0000 7f5b29600640 1 -- 172.21.15.102:0/3776290574 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5b200899f0 con 0x7f5b14003f70 2024-03-23T03:48:57.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.335+0000 7f5b31200640 1 --2- 172.21.15.102:0/3776290574 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b1006ac30 0x7f5b1006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:57.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.336+0000 7f5b31200640 1 --2- 172.21.15.102:0/3776290574 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b1006ac30 0x7f5b1006d0f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c034000 tx=0x7f5b1c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:57.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.340+0000 7f5b29600640 1 -- 172.21.15.102:0/3776290574 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5b20054b40 con 0x7f5b14003f70 2024-03-23T03:48:57.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:57.565+0000 7f5b34600640 1 -- 172.21.15.102:0/3776290574 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5b2c0b20b0 con 0x7f5b14003f70 2024-03-23T03:48:58.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.245+0000 7f5b29600640 1 -- 172.21.15.102:0/3776290574 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v23) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5b20058160 con 0x7f5b14003f70 2024-03-23T03:48:58.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.249+0000 7f5b34600640 1 -- 172.21.15.102:0/3776290574 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b1006ac30 msgr2=0x7f5b1006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:58.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.249+0000 7f5b34600640 1 --2- 172.21.15.102:0/3776290574 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b1006ac30 0x7f5b1006d0f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c034000 tx=0x7f5b1c009030 comp rx=0 tx=0).stop 2024-03-23T03:48:58.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.249+0000 7f5b34600640 1 -- 172.21.15.102:0/3776290574 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b14003f70 msgr2=0x7f5b2c14fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:58.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.249+0000 7f5b34600640 1 --2- 172.21.15.102:0/3776290574 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b14003f70 0x7f5b2c14fb60 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f5b20010860 tx=0x7f5b20010d20 comp rx=0 tx=0).stop 2024-03-23T03:48:58.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.249+0000 7f5b34600640 1 -- 172.21.15.102:0/3776290574 shutdown_connections 2024-03-23T03:48:58.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.249+0000 7f5b34600640 1 --2- 172.21.15.102:0/3776290574 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b1006ac30 0x7f5b1006d0f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:58.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.249+0000 7f5b34600640 1 --2- 172.21.15.102:0/3776290574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b2c0cc300 0x7f5b2c14ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:58.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.249+0000 7f5b34600640 1 --2- 172.21.15.102:0/3776290574 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5b2c0cbe80 0x7f5b2c14c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:58.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.249+0000 7f5b34600640 1 --2- 172.21.15.102:0/3776290574 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b14003f70 0x7f5b2c14fb60 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:58.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.249+0000 7f5b34600640 1 -- 172.21.15.102:0/3776290574 >> 172.21.15.102:0/3776290574 conn(0x7f5b2c0cf5d0 msgr2=0x7f5b2c0b4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:58.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.249+0000 7f5b34600640 1 -- 172.21.15.102:0/3776290574 shutdown_connections 2024-03-23T03:48:58.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.249+0000 7f5b34600640 1 -- 172.21.15.102:0/3776290574 wait complete. 2024-03-23T03:48:58.275 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-23T03:48:58.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T03:48:58.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.488+0000 7fd4ec600640 1 Processor -- start 2024-03-23T03:48:58.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.488+0000 7fd4ec600640 1 -- start start 2024-03-23T03:48:58.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.489+0000 7fd4ec600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd4e40cd740 0x7fd4e40b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:58.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.490+0000 7fd4ec600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd4e40d0d40 0x7fd4e40cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.490+0000 7fd4ec600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4e40d20c0 con 0x7fd4e406b0a0 2024-03-23T03:48:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.490+0000 7fd4ec600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e40ce060 con 0x7fd4e40d0d40 2024-03-23T03:48:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.490+0000 7fd4ec600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4e40d14b0 con 0x7fd4e40cd740 2024-03-23T03:48:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.490+0000 7fd4e9200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd4e40d0d40 0x7fd4e40cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.490+0000 7fd4e9200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd4e40d0d40 0x7fd4e40cdb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41772/0 (socket says 172.21.15.102:41772) 2024-03-23T03:48:58.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.490+0000 7fd4e9200640 1 -- 172.21.15.102:0/2184639128 learned_addr learned my addr 172.21.15.102:0/2184639128 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:58.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.490+0000 7fd4e9200640 1 -- 172.21.15.102:0/2184639128 >> v2:172.21.15.173:3300/0 conn(0x7fd4e40cd740 msgr2=0x7fd4e40b84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:58.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.490+0000 7fd4e9200640 1 --2- 172.21.15.102:0/2184639128 >> v2:172.21.15.173:3300/0 conn(0x7fd4e40cd740 0x7fd4e40b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:58.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.490+0000 7fd4e9200640 1 -- 172.21.15.102:0/2184639128 >> v1:172.21.15.173:6789/0 conn(0x7fd4e406b0a0 legacy=0x7fd4e406b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:58.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.490+0000 7fd4e9200640 1 -- 172.21.15.102:0/2184639128 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e40d1790 con 0x7fd4e40d0d40 2024-03-23T03:48:58.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.491+0000 7fd4e9200640 1 --2- 172.21.15.102:0/2184639128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e40d0d40 0x7fd4e40cdb20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd4e408a100 tx=0x7fd4d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f43f918c21de8324 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:58.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.492+0000 7fd4e3e00640 1 -- 172.21.15.102:0/2184639128 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0031870 con 0x7fd4e40d0d40 2024-03-23T03:48:58.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.492+0000 7fd4e3e00640 1 -- 172.21.15.102:0/2184639128 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d00319d0 con 0x7fd4e40d0d40 2024-03-23T03:48:58.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.492+0000 7fd4e3e00640 1 -- 172.21.15.102:0/2184639128 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d0031d60 con 0x7fd4e40d0d40 2024-03-23T03:48:58.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.492+0000 7fd4ec600640 1 -- 172.21.15.102:0/2184639128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e40d0d40 msgr2=0x7fd4e40cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:58.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.492+0000 7fd4ec600640 1 --2- 172.21.15.102:0/2184639128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e40d0d40 0x7fd4e40cdb20 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fd4e408a100 tx=0x7fd4d002b910 comp rx=0 tx=0).stop 2024-03-23T03:48:58.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.492+0000 7fd4ec600640 1 -- 172.21.15.102:0/2184639128 shutdown_connections 2024-03-23T03:48:58.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.492+0000 7fd4ec600640 1 --2- 172.21.15.102:0/2184639128 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e40d0d40 0x7fd4e40cdb20 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:58.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.492+0000 7fd4ec600640 1 --2- 172.21.15.102:0/2184639128 >> v2:172.21.15.173:3300/0 conn(0x7fd4e40cd740 0x7fd4e40b84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:58.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.492+0000 7fd4ec600640 1 -- 172.21.15.102:0/2184639128 >> 172.21.15.102:0/2184639128 conn(0x7fd4e40cf5d0 msgr2=0x7fd4e40cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:58.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.492+0000 7fd4ec600640 1 -- 172.21.15.102:0/2184639128 shutdown_connections 2024-03-23T03:48:58.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.492+0000 7fd4ec600640 1 -- 172.21.15.102:0/2184639128 wait complete. 2024-03-23T03:48:58.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.494+0000 7fd4ec600640 1 Processor -- start 2024-03-23T03:48:58.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.494+0000 7fd4ec600640 1 -- start start 2024-03-23T03:48:58.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.495+0000 7fd4ec600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4e406b0a0 0x7fd4e414b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:58.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.495+0000 7fd4ec600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd4e40cd740 0x7fd4e40cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:58.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.495+0000 7fd4ec600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e40d0d40 0x7fd4e40ab980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:58.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.495+0000 7fd4ec600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e414be50 con 0x7fd4e40d0d40 2024-03-23T03:48:58.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.495+0000 7fd4ec600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e414c3d0 con 0x7fd4e406b0a0 2024-03-23T03:48:58.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.495+0000 7fd4ec600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e40b1560 con 0x7fd4e40cd740 2024-03-23T03:48:58.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.495+0000 7fd4ea600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e40d0d40 0x7fd4e40ab980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:58.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.496+0000 7fd4ea600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e40d0d40 0x7fd4e40ab980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41784/0 (socket says 172.21.15.102:41784) 2024-03-23T03:48:58.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.496+0000 7fd4ea600640 1 -- 172.21.15.102:0/3610147199 learned_addr learned my addr 172.21.15.102:0/3610147199 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:58.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.496+0000 7fd4e9c00640 1 --2- 172.21.15.102:0/3610147199 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4e406b0a0 0x7fd4e414b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:58.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.496+0000 7fd4e9200640 1 --2- 172.21.15.102:0/3610147199 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd4e40cd740 0x7fd4e40cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:58.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.496+0000 7fd4ea600640 1 -- 172.21.15.102:0/3610147199 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd4e40cd740 msgr2=0x7fd4e40cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:58.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.496+0000 7fd4ea600640 1 --2- 172.21.15.102:0/3610147199 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd4e40cd740 0x7fd4e40cb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:58.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.496+0000 7fd4ea600640 1 -- 172.21.15.102:0/3610147199 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4e406b0a0 msgr2=0x7fd4e414b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:58.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.496+0000 7fd4ea600640 1 --2- 172.21.15.102:0/3610147199 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4e406b0a0 0x7fd4e414b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:58.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.496+0000 7fd4ea600640 1 -- 172.21.15.102:0/3610147199 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d00094d0 con 0x7fd4e40d0d40 2024-03-23T03:48:58.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.496+0000 7fd4e9200640 1 --2- 172.21.15.102:0/3610147199 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd4e40cd740 0x7fd4e40cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:58.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.496+0000 7fd4ea600640 1 --2- 172.21.15.102:0/3610147199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e40d0d40 0x7fd4e40ab980 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00c780 tx=0x7fd4dc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:58.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.496+0000 7fd4e1600640 1 -- 172.21.15.102:0/3610147199 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc003d80 con 0x7fd4e40d0d40 2024-03-23T03:48:58.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.496+0000 7fd4e1600640 1 -- 172.21.15.102:0/3610147199 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4dc00ce90 con 0x7fd4e40d0d40 2024-03-23T03:48:58.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.496+0000 7fd4e1600640 1 -- 172.21.15.102:0/3610147199 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc0052b0 con 0x7fd4e40d0d40 2024-03-23T03:48:58.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.497+0000 7fd4ec600640 1 -- 172.21.15.102:0/3610147199 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e40cbf60 con 0x7fd4e40d0d40 2024-03-23T03:48:58.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.497+0000 7fd4ec600640 1 -- 172.21.15.102:0/3610147199 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e414c060 con 0x7fd4e40d0d40 2024-03-23T03:48:58.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.498+0000 7fd4ec600640 1 -- 172.21.15.102:0/3610147199 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e414c540 con 0x7fd4e40d0d40 2024-03-23T03:48:58.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.499+0000 7fd4e1600640 1 -- 172.21.15.102:0/3610147199 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd4dc005410 con 0x7fd4e40d0d40 2024-03-23T03:48:58.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.499+0000 7fd4e1600640 1 --2- 172.21.15.102:0/3610147199 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd4c406ac00 0x7fd4c406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:58.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.500+0000 7fd4e1600640 1 -- 172.21.15.102:0/3610147199 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd4dc088ad0 con 0x7fd4e40d0d40 2024-03-23T03:48:58.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.500+0000 7fd4e9c00640 1 --2- 172.21.15.102:0/3610147199 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd4c406ac00 0x7fd4c406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:58.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.500+0000 7fd4e9c00640 1 --2- 172.21.15.102:0/3610147199 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd4c406ac00 0x7fd4c406d0c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009890 tx=0x7fd4d4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:58.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.504+0000 7fd4e1600640 1 -- 172.21.15.102:0/3610147199 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd4dc053a40 con 0x7fd4e40d0d40 2024-03-23T03:48:58.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:58.741+0000 7fd4ec600640 1 -- 172.21.15.102:0/3610147199 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd4e4073390 con 0x7fd4e40d0d40 2024-03-23T03:48:59.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.256+0000 7fd4e1600640 1 -- 172.21.15.102:0/3610147199 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd4dc017070 con 0x7fd4e40d0d40 2024-03-23T03:48:59.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.259+0000 7fd4ec600640 1 -- 172.21.15.102:0/3610147199 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd4c406ac00 msgr2=0x7fd4c406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:59.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.259+0000 7fd4ec600640 1 --2- 172.21.15.102:0/3610147199 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd4c406ac00 0x7fd4c406d0c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4009890 tx=0x7fd4d4008040 comp rx=0 tx=0).stop 2024-03-23T03:48:59.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.259+0000 7fd4ec600640 1 -- 172.21.15.102:0/3610147199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e40d0d40 msgr2=0x7fd4e40ab980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:59.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.259+0000 7fd4ec600640 1 --2- 172.21.15.102:0/3610147199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e40d0d40 0x7fd4e40ab980 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc00c780 tx=0x7fd4dc00cc40 comp rx=0 tx=0).stop 2024-03-23T03:48:59.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.260+0000 7fd4ec600640 1 -- 172.21.15.102:0/3610147199 shutdown_connections 2024-03-23T03:48:59.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.260+0000 7fd4ec600640 1 --2- 172.21.15.102:0/3610147199 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd4c406ac00 0x7fd4c406d0c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.260+0000 7fd4ec600640 1 --2- 172.21.15.102:0/3610147199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e40d0d40 0x7fd4e40ab980 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.260+0000 7fd4ec600640 1 --2- 172.21.15.102:0/3610147199 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd4e40cd740 0x7fd4e40cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.260+0000 7fd4ec600640 1 --2- 172.21.15.102:0/3610147199 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4e406b0a0 0x7fd4e414b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.260+0000 7fd4ec600640 1 -- 172.21.15.102:0/3610147199 >> 172.21.15.102:0/3610147199 conn(0x7fd4e40cf5d0 msgr2=0x7fd4e40d3190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:59.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.260+0000 7fd4ec600640 1 -- 172.21.15.102:0/3610147199 shutdown_connections 2024-03-23T03:48:59.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.260+0000 7fd4ec600640 1 -- 172.21.15.102:0/3610147199 wait complete. 2024-03-23T03:48:59.282 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T03:48:59.282 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:48:59.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.489+0000 7fa5c5200640 1 Processor -- start 2024-03-23T03:48:59.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.489+0000 7fa5c5200640 1 -- start start 2024-03-23T03:48:59.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.490+0000 7fa5c5200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa5c00cdc00 0x7fa5c00b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:59.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.490+0000 7fa5c5200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa5c00cae50 0x7fa5c00cb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:59.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.490+0000 7fa5c5200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c01575b0 con 0x7fa5c01571d0 2024-03-23T03:48:59.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.490+0000 7fa5c5200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c00cb820 con 0x7fa5c00cdc00 2024-03-23T03:48:59.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.490+0000 7fa5c5200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c0159f20 con 0x7fa5c00cae50 2024-03-23T03:48:59.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.491+0000 7fa5bf400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa5c01571d0 0x7fa5c015aaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52570/0 (socket says 172.21.15.102:52570) 2024-03-23T03:48:59.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.491+0000 7fa5bf400640 1 -- 172.21.15.102:0/2868034403 learned_addr learned my addr 172.21.15.102:0/2868034403 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:48:59.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.491+0000 7fa5bea00640 1 --2- 172.21.15.102:0/2868034403 >> v2:172.21.15.173:3300/0 conn(0x7fa5c00cdc00 0x7fa5c00b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:59.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.491+0000 7fa5bfe00640 1 --2- 172.21.15.102:0/2868034403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa5c00cae50 0x7fa5c00cb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:59.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.491+0000 7fa5be000640 1 -- 172.21.15.102:0/2868034403 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2695223396 0 0) 0x7fa5c01575b0 con 0x7fa5c01571d0 2024-03-23T03:48:59.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.491+0000 7fa5be000640 1 -- 172.21.15.102:0/2868034403 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5a4005730 con 0x7fa5c01571d0 2024-03-23T03:48:59.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5be000640 1 -- 172.21.15.102:0/2868034403 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1586002661 0 0) 0x7fa5a4005730 con 0x7fa5c01571d0 2024-03-23T03:48:59.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5be000640 1 -- 172.21.15.102:0/2868034403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa5c00cae50 msgr2=0x7fa5c00cb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:59.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5be000640 1 --2- 172.21.15.102:0/2868034403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa5c00cae50 0x7fa5c00cb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5be000640 1 -- 172.21.15.102:0/2868034403 >> v2:172.21.15.173:3300/0 conn(0x7fa5c00cdc00 msgr2=0x7fa5c00b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:59.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5be000640 1 --2- 172.21.15.102:0/2868034403 >> v2:172.21.15.173:3300/0 conn(0x7fa5c00cdc00 0x7fa5c00b35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5be000640 1 -- 172.21.15.102:0/2868034403 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c015a200 con 0x7fa5c01571d0 2024-03-23T03:48:59.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5be000640 1 -- 172.21.15.102:0/2868034403 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fa5b0001670 con 0x7fa5c01571d0 2024-03-23T03:48:59.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5be000640 1 -- 172.21.15.102:0/2868034403 >> v1:172.21.15.102:6789/0 conn(0x7fa5c01571d0 legacy=0x7fa5c015aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:59.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5be000640 1 --2- 172.21.15.102:0/2868034403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa5c00cae50 0x7fa5a4005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:59.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5be000640 1 --2- 172.21.15.102:0/2868034403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5a4006100 0x7fa5a40069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:59.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5be000640 1 --2- 172.21.15.102:0/2868034403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5a4006f10 0x7fa5a4007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:59.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5be000640 1 -- 172.21.15.102:0/2868034403 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c00cb820 con 0x7fa5a4006100 2024-03-23T03:48:59.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5be000640 1 -- 172.21.15.102:0/2868034403 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c0159f20 con 0x7fa5a4006f10 2024-03-23T03:48:59.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5be000640 1 -- 172.21.15.102:0/2868034403 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a400db90 con 0x7fa5c00cae50 2024-03-23T03:48:59.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5bea00640 1 --2- 172.21.15.102:0/2868034403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5a4006100 0x7fa5a40069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:59.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5bf400640 1 --2- 172.21.15.102:0/2868034403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa5c00cae50 0x7fa5a4005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:59.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5bea00640 1 -- 172.21.15.102:0/2868034403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa5c00cae50 msgr2=0x7fa5a4005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:59.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5bea00640 1 --2- 172.21.15.102:0/2868034403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa5c00cae50 0x7fa5a4005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5bea00640 1 -- 172.21.15.102:0/2868034403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5a4006f10 msgr2=0x7fa5a4007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:59.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5bea00640 1 --2- 172.21.15.102:0/2868034403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5a4006f10 0x7fa5a4007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.492+0000 7fa5bea00640 1 -- 172.21.15.102:0/2868034403 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a4005730 con 0x7fa5a4006100 2024-03-23T03:48:59.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.493+0000 7fa5bea00640 1 --2- 172.21.15.102:0/2868034403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5a4006100 0x7fa5a40069d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009ac0 tx=0x7fa5a4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18dd8f509096cd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:59.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.494+0000 7fa5be000640 1 -- 172.21.15.102:0/2868034403 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a802c6d0 con 0x7fa5a4006100 2024-03-23T03:48:59.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.494+0000 7fa5be000640 1 -- 172.21.15.102:0/2868034403 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a802c830 con 0x7fa5a4006100 2024-03-23T03:48:59.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.494+0000 7fa5be000640 1 -- 172.21.15.102:0/2868034403 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a802cbc0 con 0x7fa5a4006100 2024-03-23T03:48:59.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.494+0000 7fa5c5200640 1 -- 172.21.15.102:0/2868034403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5a4006100 msgr2=0x7fa5a40069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:59.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.494+0000 7fa5c5200640 1 --2- 172.21.15.102:0/2868034403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5a4006100 0x7fa5a40069d0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8009ac0 tx=0x7fa5a4007a00 comp rx=0 tx=0).stop 2024-03-23T03:48:59.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.494+0000 7fa5bfe00640 1 -- 172.21.15.102:0/2868034403 reap_dead start 2024-03-23T03:48:59.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.494+0000 7fa5c5200640 1 -- 172.21.15.102:0/2868034403 shutdown_connections 2024-03-23T03:48:59.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.494+0000 7fa5c5200640 1 -- 172.21.15.102:0/2868034403 >> 172.21.15.102:0/2868034403 conn(0x7fa5c0065df0 msgr2=0x7fa5c0069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:59.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.494+0000 7fa5c5200640 1 -- 172.21.15.102:0/2868034403 shutdown_connections 2024-03-23T03:48:59.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.495+0000 7fa5c5200640 1 -- 172.21.15.102:0/2868034403 wait complete. 2024-03-23T03:48:59.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.496+0000 7fa5c5200640 1 Processor -- start 2024-03-23T03:48:59.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.496+0000 7fa5c5200640 1 -- start start 2024-03-23T03:48:59.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.497+0000 7fa5c5200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5a4006100 0x7fa5c01411b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:59.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.497+0000 7fa5c5200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5c00afbc0 0x7fa5c00b0040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:59.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.497+0000 7fa5c5200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa5c006d0d0 0x7fa5c006d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:59.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.497+0000 7fa5c5200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c00b0580 con 0x7fa5a4006100 2024-03-23T03:48:59.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.497+0000 7fa5c5200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c006da90 con 0x7fa5c00afbc0 2024-03-23T03:48:59.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.497+0000 7fa5c5200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c01416f0 con 0x7fa5c006d0d0 2024-03-23T03:48:59.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.498+0000 7fa5bf400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5a4006100 0x7fa5c01411b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:59.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.498+0000 7fa5bf400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5a4006100 0x7fa5c01411b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41814/0 (socket says 172.21.15.102:41814) 2024-03-23T03:48:59.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.498+0000 7fa5bf400640 1 -- 172.21.15.102:0/4209066568 learned_addr learned my addr 172.21.15.102:0/4209066568 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:59.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.498+0000 7fa5bea00640 1 --2- 172.21.15.102:0/4209066568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5c00afbc0 0x7fa5c00b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:59.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.498+0000 7fa5bf400640 1 -- 172.21.15.102:0/4209066568 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa5c006d0d0 msgr2=0x7fa5c006d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:59.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.498+0000 7fa5bf400640 1 --2- 172.21.15.102:0/4209066568 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa5c006d0d0 0x7fa5c006d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.498+0000 7fa5bf400640 1 -- 172.21.15.102:0/4209066568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5c00afbc0 msgr2=0x7fa5c00b0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:59.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.498+0000 7fa5bf400640 1 --2- 172.21.15.102:0/4209066568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5c00afbc0 0x7fa5c00b0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.498+0000 7fa5bf400640 1 -- 172.21.15.102:0/4209066568 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c013f680 con 0x7fa5a4006100 2024-03-23T03:48:59.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.498+0000 7fa5bea00640 1 --2- 172.21.15.102:0/4209066568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5c00afbc0 0x7fa5c00b0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:48:59.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.498+0000 7fa5bf400640 1 --2- 172.21.15.102:0/4209066568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5a4006100 0x7fa5c01411b0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000e820 tx=0x7fa5b000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:59.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.499+0000 7fa5b7400640 1 -- 172.21.15.102:0/4209066568 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b000c7f0 con 0x7fa5a4006100 2024-03-23T03:48:59.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.499+0000 7fa5b7400640 1 -- 172.21.15.102:0/4209066568 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b000c970 con 0x7fa5a4006100 2024-03-23T03:48:59.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.499+0000 7fa5b7400640 1 -- 172.21.15.102:0/4209066568 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b000cd20 con 0x7fa5a4006100 2024-03-23T03:48:59.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.499+0000 7fa5c5200640 1 -- 172.21.15.102:0/4209066568 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c013f960 con 0x7fa5a4006100 2024-03-23T03:48:59.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.499+0000 7fa5c5200640 1 -- 172.21.15.102:0/4209066568 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c013ff20 con 0x7fa5a4006100 2024-03-23T03:48:59.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.500+0000 7fa5c5200640 1 -- 172.21.15.102:0/4209066568 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c00b26c0 con 0x7fa5a4006100 2024-03-23T03:48:59.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.501+0000 7fa5b7400640 1 -- 172.21.15.102:0/4209066568 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa5b001d460 con 0x7fa5a4006100 2024-03-23T03:48:59.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.502+0000 7fa5b7400640 1 --2- 172.21.15.102:0/4209066568 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa59c06ac00 0x7fa59c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:59.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.502+0000 7fa5b7400640 1 -- 172.21.15.102:0/4209066568 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa5b0089570 con 0x7fa5a4006100 2024-03-23T03:48:59.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.502+0000 7fa5bea00640 1 --2- 172.21.15.102:0/4209066568 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa59c06ac00 0x7fa59c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:59.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.503+0000 7fa5bea00640 1 --2- 172.21.15.102:0/4209066568 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa59c06ac00 0x7fa59c06d0c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8036000 tx=0x7fa5a8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:59.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.506+0000 7fa5b7400640 1 -- 172.21.15.102:0/4209066568 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa5b0054610 con 0x7fa5a4006100 2024-03-23T03:48:59.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.737+0000 7fa5c5200640 1 -- 172.21.15.102:0/4209066568 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5c004e030 con 0x7fa5a4006100 2024-03-23T03:48:59.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.738+0000 7fa5b7400640 1 -- 172.21.15.102:0/4209066568 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fa5b0057c30 con 0x7fa5a4006100 2024-03-23T03:48:59.739 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:48:59.739 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5661,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/1848167718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":1848167718},{"type":"v1","addr":"172.21.15.102:6835","nonce":1848167718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/16304019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":16304019},{"type":"v1","addr":"172.21.15.102:6837","nonce":16304019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5673,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6834/2030662139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2030662139},{"type":"v1","addr":"172.21.15.173:6834","nonce":2030662139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:48:56.221998+0000","modified":"2024-03-23T03:48:59.250212+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5676":{"gid":5676,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6835/3269327930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":3269327930},{"type":"v1","addr":"172.21.15.173:6835","nonce":3269327930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-23T03:48:59.739 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 24 2024-03-23T03:48:59.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.742+0000 7fa5c5200640 1 -- 172.21.15.102:0/4209066568 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa59c06ac00 msgr2=0x7fa59c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:59.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.742+0000 7fa5c5200640 1 --2- 172.21.15.102:0/4209066568 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa59c06ac00 0x7fa59c06d0c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fa5a8036000 tx=0x7fa5a8030000 comp rx=0 tx=0).stop 2024-03-23T03:48:59.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.742+0000 7fa5c5200640 1 -- 172.21.15.102:0/4209066568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5a4006100 msgr2=0x7fa5c01411b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:59.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.742+0000 7fa5c5200640 1 --2- 172.21.15.102:0/4209066568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5a4006100 0x7fa5c01411b0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000e820 tx=0x7fa5b000ece0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.742+0000 7fa5c5200640 1 -- 172.21.15.102:0/4209066568 shutdown_connections 2024-03-23T03:48:59.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.742+0000 7fa5c5200640 1 --2- 172.21.15.102:0/4209066568 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa59c06ac00 0x7fa59c06d0c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.742+0000 7fa5c5200640 1 --2- 172.21.15.102:0/4209066568 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa5c006d0d0 0x7fa5c006d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.742+0000 7fa5c5200640 1 --2- 172.21.15.102:0/4209066568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5c00afbc0 0x7fa5c00b0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.742+0000 7fa5c5200640 1 --2- 172.21.15.102:0/4209066568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5a4006100 0x7fa5c01411b0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.742+0000 7fa5c5200640 1 -- 172.21.15.102:0/4209066568 >> 172.21.15.102:0/4209066568 conn(0x7fa5c0065df0 msgr2=0x7fa5c00abf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:59.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.742+0000 7fa5c5200640 1 -- 172.21.15.102:0/4209066568 shutdown_connections 2024-03-23T03:48:59.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.742+0000 7fa5c5200640 1 -- 172.21.15.102:0/4209066568 wait complete. 2024-03-23T03:48:59.761 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:48:59.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.963+0000 7efc7ea00640 1 Processor -- start 2024-03-23T03:48:59.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.963+0000 7efc7ea00640 1 -- start start 2024-03-23T03:48:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.964+0000 7efc7ea00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7efc780adbc0 0x7efc780b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.964+0000 7efc7ea00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc780cbab0 0x7efc780b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.964+0000 7efc7ea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7efc7806d390 0x7efc7806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.964+0000 7efc7ea00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc7806dd60 con 0x7efc7806d390 2024-03-23T03:48:59.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.965+0000 7efc7ea00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc780b16a0 con 0x7efc780adbc0 2024-03-23T03:48:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.965+0000 7efc7ea00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc780b4700 con 0x7efc780cbab0 2024-03-23T03:48:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.965+0000 7efc7ca00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7efc7806d390 0x7efc7806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.965+0000 7efc7ca00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7efc7806d390 0x7efc7806d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41822/0 (socket says 172.21.15.102:41822) 2024-03-23T03:48:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.965+0000 7efc7ca00640 1 -- 172.21.15.102:0/434904627 learned_addr learned my addr 172.21.15.102:0/434904627 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.965+0000 7efc6fe00640 1 --2- 172.21.15.102:0/434904627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc780cbab0 0x7efc780b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:59.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.965+0000 7efc77e00640 1 --2- 172.21.15.102:0/434904627 >> v2:172.21.15.173:3300/0 conn(0x7efc780adbc0 0x7efc780b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.965+0000 7efc7ca00640 1 -- 172.21.15.102:0/434904627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc780cbab0 msgr2=0x7efc780b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.965+0000 7efc7ca00640 1 --2- 172.21.15.102:0/434904627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc780cbab0 0x7efc780b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.965+0000 7efc7ca00640 1 -- 172.21.15.102:0/434904627 >> v2:172.21.15.173:3300/0 conn(0x7efc780adbc0 msgr2=0x7efc780b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:48:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.965+0000 7efc7ca00640 1 --2- 172.21.15.102:0/434904627 >> v2:172.21.15.173:3300/0 conn(0x7efc780adbc0 0x7efc780b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.965+0000 7efc7ca00640 1 -- 172.21.15.102:0/434904627 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc780cbfb0 con 0x7efc7806d390 2024-03-23T03:48:59.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.966+0000 7efc7ca00640 1 --2- 172.21.15.102:0/434904627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc7806d390 0x7efc7806d820 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7efc7806e020 tx=0x7efc7002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0fbc5153ed86e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.967+0000 7efc77400640 1 -- 172.21.15.102:0/434904627 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7002c9e0 con 0x7efc7806d390 2024-03-23T03:48:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.967+0000 7efc77400640 1 -- 172.21.15.102:0/434904627 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc7002cb40 con 0x7efc7806d390 2024-03-23T03:48:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.967+0000 7efc77400640 1 -- 172.21.15.102:0/434904627 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc700343f0 con 0x7efc7806d390 2024-03-23T03:48:59.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.967+0000 7efc7ea00640 1 -- 172.21.15.102:0/434904627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc7806d390 msgr2=0x7efc7806d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:59.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.967+0000 7efc7ea00640 1 --2- 172.21.15.102:0/434904627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc7806d390 0x7efc7806d820 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7efc7806e020 tx=0x7efc7002be40 comp rx=0 tx=0).stop 2024-03-23T03:48:59.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.968+0000 7efc7ea00640 1 -- 172.21.15.102:0/434904627 shutdown_connections 2024-03-23T03:48:59.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.968+0000 7efc7ea00640 1 --2- 172.21.15.102:0/434904627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc7806d390 0x7efc7806d820 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.968+0000 7efc7ea00640 1 --2- 172.21.15.102:0/434904627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc780cbab0 0x7efc780b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.968+0000 7efc7ea00640 1 --2- 172.21.15.102:0/434904627 >> v2:172.21.15.173:3300/0 conn(0x7efc780adbc0 0x7efc780b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.968+0000 7efc7ea00640 1 -- 172.21.15.102:0/434904627 >> 172.21.15.102:0/434904627 conn(0x7efc7806a3d0 msgr2=0x7efc780b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:48:59.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.968+0000 7efc7ea00640 1 -- 172.21.15.102:0/434904627 shutdown_connections 2024-03-23T03:48:59.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.968+0000 7efc7ea00640 1 -- 172.21.15.102:0/434904627 wait complete. 2024-03-23T03:48:59.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.969+0000 7efc7ea00640 1 Processor -- start 2024-03-23T03:48:59.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.970+0000 7efc7ea00640 1 -- start start 2024-03-23T03:48:59.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.970+0000 7efc7ea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc7806d390 0x7efc7814bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:59.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.970+0000 7efc7ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc780adbc0 0x7efc7813a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:59.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.970+0000 7efc7ea00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc780cbab0 0x7efc7814de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:59.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.970+0000 7efc7ea00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7813aa20 con 0x7efc780adbc0 2024-03-23T03:48:59.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.971+0000 7efc7ea00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7813d590 con 0x7efc7806d390 2024-03-23T03:48:59.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.971+0000 7efc7ea00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc781402f0 con 0x7efc780cbab0 2024-03-23T03:48:59.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.971+0000 7efc6fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc780adbc0 0x7efc7813a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:59.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.971+0000 7efc6fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc780adbc0 0x7efc7813a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41838/0 (socket says 172.21.15.102:41838) 2024-03-23T03:48:59.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.971+0000 7efc6fe00640 1 -- 172.21.15.102:0/493859475 learned_addr learned my addr 172.21.15.102:0/493859475 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:48:59.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.971+0000 7efc7ca00640 1 --2- 172.21.15.102:0/493859475 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc780cbab0 0x7efc7814de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:59.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.971+0000 7efc6fe00640 1 -- 172.21.15.102:0/493859475 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc780cbab0 msgr2=0x7efc7814de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:59.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.971+0000 7efc6fe00640 1 --2- 172.21.15.102:0/493859475 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc780cbab0 0x7efc7814de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.971+0000 7efc6fe00640 1 -- 172.21.15.102:0/493859475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc7806d390 msgr2=0x7efc7814bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:48:59.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.971+0000 7efc6fe00640 1 --2- 172.21.15.102:0/493859475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc7806d390 0x7efc7814bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:48:59.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.971+0000 7efc6fe00640 1 -- 172.21.15.102:0/493859475 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc70009b30 con 0x7efc780adbc0 2024-03-23T03:48:59.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.971+0000 7efc6fe00640 1 --2- 172.21.15.102:0/493859475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc780adbc0 0x7efc7813a4e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7efc6000d830 tx=0x7efc6000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:59.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.972+0000 7efc74c00640 1 -- 172.21.15.102:0/493859475 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6000b6c0 con 0x7efc780adbc0 2024-03-23T03:48:59.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.972+0000 7efc74c00640 1 -- 172.21.15.102:0/493859475 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc6000b820 con 0x7efc780adbc0 2024-03-23T03:48:59.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.972+0000 7efc74c00640 1 -- 172.21.15.102:0/493859475 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc6000bbb0 con 0x7efc780adbc0 2024-03-23T03:48:59.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.972+0000 7efc7ea00640 1 -- 172.21.15.102:0/493859475 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7813fa00 con 0x7efc780adbc0 2024-03-23T03:48:59.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.972+0000 7efc7ea00640 1 -- 172.21.15.102:0/493859475 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7813fca0 con 0x7efc780adbc0 2024-03-23T03:48:59.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.973+0000 7efc7ea00640 1 -- 172.21.15.102:0/493859475 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7806ac20 con 0x7efc780adbc0 2024-03-23T03:48:59.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.974+0000 7efc74c00640 1 -- 172.21.15.102:0/493859475 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7efc6001e460 con 0x7efc780adbc0 2024-03-23T03:48:59.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.975+0000 7efc74c00640 1 --2- 172.21.15.102:0/493859475 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efc5406ac00 0x7efc5406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:48:59.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.975+0000 7efc74c00640 1 -- 172.21.15.102:0/493859475 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7efc60089cd0 con 0x7efc780adbc0 2024-03-23T03:48:59.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.975+0000 7efc77e00640 1 --2- 172.21.15.102:0/493859475 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efc5406ac00 0x7efc5406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:48:59.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.976+0000 7efc77e00640 1 --2- 172.21.15.102:0/493859475 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efc5406ac00 0x7efc5406d0c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efc6800b7f0 tx=0x7efc68008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:48:59.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:48:59.980+0000 7efc74c00640 1 -- 172.21.15.102:0/493859475 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efc60054cd0 con 0x7efc780adbc0 2024-03-23T03:49:00.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.173+0000 7efc7ea00640 1 -- 172.21.15.102:0/493859475 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc78073390 con 0x7efc780adbc0 2024-03-23T03:49:00.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.176+0000 7efc74c00640 1 -- 172.21.15.102:0/493859475 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16354 (secure 0 0 0) 0x7efc600582f0 con 0x7efc780adbc0 2024-03-23T03:49:00.177 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:00.177 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":55,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:57.226138+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:48:53.998390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T03:48:55.242109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:49:00.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.181+0000 7efc7ea00640 1 -- 172.21.15.102:0/493859475 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efc5406ac00 msgr2=0x7efc5406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:00.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.181+0000 7efc7ea00640 1 --2- 172.21.15.102:0/493859475 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efc5406ac00 0x7efc5406d0c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7efc6800b7f0 tx=0x7efc68008040 comp rx=0 tx=0).stop 2024-03-23T03:49:00.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.181+0000 7efc7ea00640 1 -- 172.21.15.102:0/493859475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc780adbc0 msgr2=0x7efc7813a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:00.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.181+0000 7efc7ea00640 1 --2- 172.21.15.102:0/493859475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc780adbc0 0x7efc7813a4e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7efc6000d830 tx=0x7efc6000dcf0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.181+0000 7efc7ea00640 1 -- 172.21.15.102:0/493859475 shutdown_connections 2024-03-23T03:49:00.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.181+0000 7efc7ea00640 1 --2- 172.21.15.102:0/493859475 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efc5406ac00 0x7efc5406d0c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.181+0000 7efc7ea00640 1 --2- 172.21.15.102:0/493859475 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efc780cbab0 0x7efc7814de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.181+0000 7efc7ea00640 1 --2- 172.21.15.102:0/493859475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc780adbc0 0x7efc7813a4e0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.181+0000 7efc7ea00640 1 --2- 172.21.15.102:0/493859475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efc7806d390 0x7efc7814bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.181+0000 7efc7ea00640 1 -- 172.21.15.102:0/493859475 >> 172.21.15.102:0/493859475 conn(0x7efc7806a3d0 msgr2=0x7efc780b3670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:00.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.181+0000 7efc7ea00640 1 -- 172.21.15.102:0/493859475 shutdown_connections 2024-03-23T03:49:00.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.181+0000 7efc7ea00640 1 -- 172.21.15.102:0/493859475 wait complete. 2024-03-23T03:49:00.201 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T03:49:00.201 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:49:00.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.405+0000 7fa612e00640 1 Processor -- start 2024-03-23T03:49:00.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.405+0000 7fa612e00640 1 -- start start 2024-03-23T03:49:00.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.407+0000 7fa612e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa60c1571d0 0x7fa60c0b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:00.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.407+0000 7fa612e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa60c0cdc00 0x7fa60c15aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:00.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.407+0000 7fa612e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa60c15afe0 0x7fa60c0cae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:00.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.407+0000 7fa612e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa60c1575b0 con 0x7fa60c0cdc00 2024-03-23T03:49:00.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.407+0000 7fa612e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa60c159f20 con 0x7fa60c1571d0 2024-03-23T03:49:00.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.407+0000 7fa612e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c15a200 con 0x7fa60c15afe0 2024-03-23T03:49:00.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.407+0000 7fa611000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa60c0cdc00 0x7fa60c15aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:00.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.407+0000 7fa611000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa60c0cdc00 0x7fa60c15aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41844/0 (socket says 172.21.15.102:41844) 2024-03-23T03:49:00.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.407+0000 7fa611000640 1 -- 172.21.15.102:0/3167701348 learned_addr learned my addr 172.21.15.102:0/3167701348 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:00.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.407+0000 7fa611000640 1 -- 172.21.15.102:0/3167701348 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa60c15afe0 msgr2=0x7fa60c0cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:00.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.407+0000 7fa611000640 1 --2- 172.21.15.102:0/3167701348 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa60c15afe0 0x7fa60c0cae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.407+0000 7fa611000640 1 -- 172.21.15.102:0/3167701348 >> v2:172.21.15.173:3300/0 conn(0x7fa60c1571d0 msgr2=0x7fa60c0b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:00.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.407+0000 7fa611a00640 1 --2- 172.21.15.102:0/3167701348 >> v2:172.21.15.173:3300/0 conn(0x7fa60c1571d0 0x7fa60c0b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:00.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.408+0000 7fa611000640 1 --2- 172.21.15.102:0/3167701348 >> v2:172.21.15.173:3300/0 conn(0x7fa60c1571d0 0x7fa60c0b35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.408+0000 7fa611000640 1 -- 172.21.15.102:0/3167701348 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60c15a4e0 con 0x7fa60c0cdc00 2024-03-23T03:49:00.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.409+0000 7fa611000640 1 --2- 172.21.15.102:0/3167701348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa60c0cdc00 0x7fa60c15aaa0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa60c0712e0 tx=0x7fa5f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3ba579c2eb4dc3a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:00.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.409+0000 7fa60be00640 1 -- 172.21.15.102:0/3167701348 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8031870 con 0x7fa60c0cdc00 2024-03-23T03:49:00.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.409+0000 7fa60be00640 1 -- 172.21.15.102:0/3167701348 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f80319d0 con 0x7fa60c0cdc00 2024-03-23T03:49:00.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.409+0000 7fa60be00640 1 -- 172.21.15.102:0/3167701348 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8031d60 con 0x7fa60c0cdc00 2024-03-23T03:49:00.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.410+0000 7fa612e00640 1 -- 172.21.15.102:0/3167701348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa60c0cdc00 msgr2=0x7fa60c15aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:00.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.410+0000 7fa612e00640 1 --2- 172.21.15.102:0/3167701348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa60c0cdc00 0x7fa60c15aaa0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa60c0712e0 tx=0x7fa5f802b910 comp rx=0 tx=0).stop 2024-03-23T03:49:00.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.410+0000 7fa612e00640 1 -- 172.21.15.102:0/3167701348 shutdown_connections 2024-03-23T03:49:00.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.410+0000 7fa612e00640 1 --2- 172.21.15.102:0/3167701348 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa60c15afe0 0x7fa60c0cae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.410+0000 7fa612e00640 1 --2- 172.21.15.102:0/3167701348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa60c0cdc00 0x7fa60c15aaa0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.410+0000 7fa612e00640 1 --2- 172.21.15.102:0/3167701348 >> v2:172.21.15.173:3300/0 conn(0x7fa60c1571d0 0x7fa60c0b35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.410+0000 7fa612e00640 1 -- 172.21.15.102:0/3167701348 >> 172.21.15.102:0/3167701348 conn(0x7fa60c065df0 msgr2=0x7fa60c069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:00.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.410+0000 7fa612e00640 1 -- 172.21.15.102:0/3167701348 shutdown_connections 2024-03-23T03:49:00.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.410+0000 7fa612e00640 1 -- 172.21.15.102:0/3167701348 wait complete. 2024-03-23T03:49:00.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.412+0000 7fa612e00640 1 Processor -- start 2024-03-23T03:49:00.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.412+0000 7fa612e00640 1 -- start start 2024-03-23T03:49:00.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.412+0000 7fa612e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa60c0cdc00 0x7fa60c141b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:00.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.413+0000 7fa612e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa60c1571d0 0x7fa60c0b0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:00.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.413+0000 7fa612e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa60c15afe0 0x7fa60c1f1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:00.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.413+0000 7fa612e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c0afab0 con 0x7fa60c15afe0 2024-03-23T03:49:00.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.413+0000 7fa612e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c159970 con 0x7fa60c0cdc00 2024-03-23T03:49:00.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.413+0000 7fa612e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60c142080 con 0x7fa60c1571d0 2024-03-23T03:49:00.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.413+0000 7fa612400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa60c15afe0 0x7fa60c1f1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:00.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.413+0000 7fa612400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa60c15afe0 0x7fa60c1f1d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41856/0 (socket says 172.21.15.102:41856) 2024-03-23T03:49:00.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.413+0000 7fa612400640 1 -- 172.21.15.102:0/2971996730 learned_addr learned my addr 172.21.15.102:0/2971996730 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:00.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.413+0000 7fa611000640 1 --2- 172.21.15.102:0/2971996730 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa60c1571d0 0x7fa60c0b0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:00.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.413+0000 7fa611a00640 1 --2- 172.21.15.102:0/2971996730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa60c0cdc00 0x7fa60c141b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:00.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.413+0000 7fa612400640 1 -- 172.21.15.102:0/2971996730 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa60c1571d0 msgr2=0x7fa60c0b0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:00.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.413+0000 7fa612400640 1 --2- 172.21.15.102:0/2971996730 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa60c1571d0 0x7fa60c0b0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.413+0000 7fa612400640 1 -- 172.21.15.102:0/2971996730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa60c0cdc00 msgr2=0x7fa60c141b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:00.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.413+0000 7fa612400640 1 --2- 172.21.15.102:0/2971996730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa60c0cdc00 0x7fa60c141b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.413+0000 7fa612400640 1 -- 172.21.15.102:0/2971996730 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f80094d0 con 0x7fa60c15afe0 2024-03-23T03:49:00.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.413+0000 7fa611a00640 1 --2- 172.21.15.102:0/2971996730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa60c0cdc00 0x7fa60c141b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:00.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.414+0000 7fa612400640 1 --2- 172.21.15.102:0/2971996730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa60c15afe0 0x7fa60c1f1d70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa60400b970 tx=0x7fa60400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:00.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.414+0000 7fa609600640 1 -- 172.21.15.102:0/2971996730 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60400cae0 con 0x7fa60c15afe0 2024-03-23T03:49:00.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.414+0000 7fa609600640 1 -- 172.21.15.102:0/2971996730 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60400cc60 con 0x7fa60c15afe0 2024-03-23T03:49:00.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.414+0000 7fa612e00640 1 -- 172.21.15.102:0/2971996730 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa60c0afca0 con 0x7fa60c15afe0 2024-03-23T03:49:00.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.414+0000 7fa609600640 1 -- 172.21.15.102:0/2971996730 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa604014440 con 0x7fa60c15afe0 2024-03-23T03:49:00.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.414+0000 7fa612e00640 1 -- 172.21.15.102:0/2971996730 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa60c0b0260 con 0x7fa60c15afe0 2024-03-23T03:49:00.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.415+0000 7fa612e00640 1 -- 172.21.15.102:0/2971996730 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa60c04e030 con 0x7fa60c15afe0 2024-03-23T03:49:00.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.416+0000 7fa609600640 1 -- 172.21.15.102:0/2971996730 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa60401d3b0 con 0x7fa60c15afe0 2024-03-23T03:49:00.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.417+0000 7fa609600640 1 --2- 172.21.15.102:0/2971996730 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa5ec06ac00 0x7fa5ec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:00.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.417+0000 7fa609600640 1 -- 172.21.15.102:0/2971996730 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa6040893c0 con 0x7fa60c15afe0 2024-03-23T03:49:00.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.417+0000 7fa611a00640 1 --2- 172.21.15.102:0/2971996730 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa5ec06ac00 0x7fa5ec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:00.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.418+0000 7fa611a00640 1 --2- 172.21.15.102:0/2971996730 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa5ec06ac00 0x7fa5ec06d0c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa600008890 tx=0x7fa600008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:00.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.421+0000 7fa609600640 1 -- 172.21.15.102:0/2971996730 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa604054450 con 0x7fa60c15afe0 2024-03-23T03:49:00.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.612+0000 7fa612e00640 1 -- 172.21.15.102:0/2971996730 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa60c067680 con 0x7fa5ec06ac00 2024-03-23T03:49:00.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.637+0000 7fa609600640 1 -- 172.21.15.102:0/2971996730 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286793 (secure 0 0 0) 0x7fa60c067680 con 0x7fa5ec06ac00 2024-03-23T03:49:00.639 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:00.643 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:49:00.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.644+0000 7fa612e00640 1 -- 172.21.15.102:0/2971996730 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa5ec06ac00 msgr2=0x7fa5ec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:00.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.644+0000 7fa612e00640 1 --2- 172.21.15.102:0/2971996730 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa5ec06ac00 0x7fa5ec06d0c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa600008890 tx=0x7fa600008040 comp rx=0 tx=0).stop 2024-03-23T03:49:00.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.644+0000 7fa612e00640 1 -- 172.21.15.102:0/2971996730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa60c15afe0 msgr2=0x7fa60c1f1d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:00.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.644+0000 7fa612e00640 1 --2- 172.21.15.102:0/2971996730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa60c15afe0 0x7fa60c1f1d70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa60400b970 tx=0x7fa60400be30 comp rx=0 tx=0).stop 2024-03-23T03:49:00.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.644+0000 7fa612e00640 1 -- 172.21.15.102:0/2971996730 shutdown_connections 2024-03-23T03:49:00.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.644+0000 7fa612e00640 1 --2- 172.21.15.102:0/2971996730 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa5ec06ac00 0x7fa5ec06d0c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.644+0000 7fa612e00640 1 --2- 172.21.15.102:0/2971996730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa60c15afe0 0x7fa60c1f1d70 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.644+0000 7fa612e00640 1 --2- 172.21.15.102:0/2971996730 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa60c1571d0 0x7fa60c0b0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.644+0000 7fa612e00640 1 --2- 172.21.15.102:0/2971996730 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa60c0cdc00 0x7fa60c141b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.644+0000 7fa612e00640 1 -- 172.21.15.102:0/2971996730 >> 172.21.15.102:0/2971996730 conn(0x7fa60c065df0 msgr2=0x7fa60c0cc810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:00.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.645+0000 7fa612e00640 1 -- 172.21.15.102:0/2971996730 shutdown_connections 2024-03-23T03:49:00.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.645+0000 7fa612e00640 1 -- 172.21.15.102:0/2971996730 wait complete. 2024-03-23T03:49:00.664 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-03-23T03:48:59.756278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222336,"kb_used_data":6140,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752384,"statfs":{"total":773094113280,"available":772866441216,"internally_reserved":0,"allocated":6287360,"data_stored":3397934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":221236794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.020087"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.273640+0000","last_change":"2024-03-23T03:48:56.269844+0000","last_active":"2024-03-23T03:48:56.273640+0000","last_peered":"2024-03-23T03:48:56.273640+0000","last_clean":"2024-03-23T03:48:56.273640+0000","last_became_active":"2024-03-23T03:48:56.269647+0000","last_became_peered":"2024-03-23T03:48:56.269647+0000","last_unstale":"2024-03-23T03:48:56.273640+0000","last_undegraded":"2024-03-23T03:48:56.273640+0000","last_fullsized":"2024-03-23T03:48:56.273640+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:15:23.482683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.273181+0000","last_change":"2024-03-23T03:48:56.271920+0000","last_active":"2024-03-23T03:48:56.273181+0000","last_peered":"2024-03-23T03:48:56.273181+0000","last_clean":"2024-03-23T03:48:56.273181+0000","last_became_active":"2024-03-23T03:48:56.271646+0000","last_became_peered":"2024-03-23T03:48:56.271646+0000","last_unstale":"2024-03-23T03:48:56.273181+0000","last_undegraded":"2024-03-23T03:48:56.273181+0000","last_fullsized":"2024-03-23T03:48:56.273181+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:00:24.758423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.267175+0000","last_change":"2024-03-23T03:48:56.264762+0000","last_active":"2024-03-23T03:48:56.267175+0000","last_peered":"2024-03-23T03:48:56.267175+0000","last_clean":"2024-03-23T03:48:56.267175+0000","last_became_active":"2024-03-23T03:48:56.263552+0000","last_became_peered":"2024-03-23T03:48:56.263552+0000","last_unstale":"2024-03-23T03:48:56.267175+0000","last_undegraded":"2024-03-23T03:48:56.267175+0000","last_fullsized":"2024-03-23T03:48:56.267175+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:40.071215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.272268+0000","last_change":"2024-03-23T03:48:56.270521+0000","last_active":"2024-03-23T03:48:56.272268+0000","last_peered":"2024-03-23T03:48:56.272268+0000","last_clean":"2024-03-23T03:48:56.272268+0000","last_became_active":"2024-03-23T03:48:56.270254+0000","last_became_peered":"2024-03-23T03:48:56.270254+0000","last_unstale":"2024-03-23T03:48:56.272268+0000","last_undegraded":"2024-03-23T03:48:56.272268+0000","last_fullsized":"2024-03-23T03:48:56.272268+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:08.127529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.259770+0000","last_change":"2024-03-23T03:48:56.252733+0000","last_active":"2024-03-23T03:48:56.259770+0000","last_peered":"2024-03-23T03:48:56.259770+0000","last_clean":"2024-03-23T03:48:56.259770+0000","last_became_active":"2024-03-23T03:48:56.252442+0000","last_became_peered":"2024-03-23T03:48:56.252442+0000","last_unstale":"2024-03-23T03:48:56.259770+0000","last_undegraded":"2024-03-23T03:48:56.259770+0000","last_fullsized":"2024-03-23T03:48:56.259770+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:44:31.659883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.281716+0000","last_change":"2024-03-23T03:48:56.272907+0000","last_active":"2024-03-23T03:48:56.281716+0000","last_peered":"2024-03-23T03:48:56.281716+0000","last_clean":"2024-03-23T03:48:56.281716+0000","last_became_active":"2024-03-23T03:48:56.272660+0000","last_became_peered":"2024-03-23T03:48:56.272660+0000","last_unstale":"2024-03-23T03:48:56.281716+0000","last_undegraded":"2024-03-23T03:48:56.281716+0000","last_fullsized":"2024-03-23T03:48:56.281716+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:37:28.870018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.279678+0000","last_change":"2024-03-23T03:48:56.277900+0000","last_active":"2024-03-23T03:48:56.279678+0000","last_peered":"2024-03-23T03:48:56.279678+0000","last_clean":"2024-03-23T03:48:56.279678+0000","last_became_active":"2024-03-23T03:48:56.276387+0000","last_became_peered":"2024-03-23T03:48:56.276387+0000","last_unstale":"2024-03-23T03:48:56.279678+0000","last_undegraded":"2024-03-23T03:48:56.279678+0000","last_fullsized":"2024-03-23T03:48:56.279678+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:46:56.378847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.264460+0000","last_change":"2024-03-23T03:48:56.254792+0000","last_active":"2024-03-23T03:48:56.264460+0000","last_peered":"2024-03-23T03:48:56.264460+0000","last_clean":"2024-03-23T03:48:56.264460+0000","last_became_active":"2024-03-23T03:48:56.254483+0000","last_became_peered":"2024-03-23T03:48:56.254483+0000","last_unstale":"2024-03-23T03:48:56.264460+0000","last_undegraded":"2024-03-23T03:48:56.264460+0000","last_fullsized":"2024-03-23T03:48:56.264460+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:35:49.226120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.263377+0000","last_change":"2024-03-23T03:48:56.250567+0000","last_active":"2024-03-23T03:48:56.263377+0000","last_peered":"2024-03-23T03:48:56.263377+0000","last_clean":"2024-03-23T03:48:56.263377+0000","last_became_active":"2024-03-23T03:48:56.250309+0000","last_became_peered":"2024-03-23T03:48:56.250309+0000","last_unstale":"2024-03-23T03:48:56.263377+0000","last_undegraded":"2024-03-23T03:48:56.263377+0000","last_fullsized":"2024-03-23T03:48:56.263377+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:24:18.834311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.271336+0000","last_change":"2024-03-23T03:48:56.265622+0000","last_active":"2024-03-23T03:48:56.271336+0000","last_peered":"2024-03-23T03:48:56.271336+0000","last_clean":"2024-03-23T03:48:56.271336+0000","last_became_active":"2024-03-23T03:48:56.265139+0000","last_became_peered":"2024-03-23T03:48:56.265139+0000","last_unstale":"2024-03-23T03:48:56.271336+0000","last_undegraded":"2024-03-23T03:48:56.271336+0000","last_fullsized":"2024-03-23T03:48:56.271336+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:24:15.405965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.278790+0000","last_change":"2024-03-23T03:48:56.275092+0000","last_active":"2024-03-23T03:48:56.278790+0000","last_peered":"2024-03-23T03:48:56.278790+0000","last_clean":"2024-03-23T03:48:56.278790+0000","last_became_active":"2024-03-23T03:48:56.274799+0000","last_became_peered":"2024-03-23T03:48:56.274799+0000","last_unstale":"2024-03-23T03:48:56.278790+0000","last_undegraded":"2024-03-23T03:48:56.278790+0000","last_fullsized":"2024-03-23T03:48:56.278790+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:04:50.091047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.267385+0000","last_change":"2024-03-23T03:48:56.263890+0000","last_active":"2024-03-23T03:48:56.267385+0000","last_peered":"2024-03-23T03:48:56.267385+0000","last_clean":"2024-03-23T03:48:56.267385+0000","last_became_active":"2024-03-23T03:48:56.263596+0000","last_became_peered":"2024-03-23T03:48:56.263596+0000","last_unstale":"2024-03-23T03:48:56.267385+0000","last_undegraded":"2024-03-23T03:48:56.267385+0000","last_fullsized":"2024-03-23T03:48:56.267385+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:13:51.660301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.272056+0000","last_change":"2024-03-23T03:48:56.266870+0000","last_active":"2024-03-23T03:48:56.272056+0000","last_peered":"2024-03-23T03:48:56.272056+0000","last_clean":"2024-03-23T03:48:56.272056+0000","last_became_active":"2024-03-23T03:48:56.266620+0000","last_became_peered":"2024-03-23T03:48:56.266620+0000","last_unstale":"2024-03-23T03:48:56.272056+0000","last_undegraded":"2024-03-23T03:48:56.272056+0000","last_fullsized":"2024-03-23T03:48:56.272056+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:01.054386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.267069+0000","last_change":"2024-03-23T03:48:56.262773+0000","last_active":"2024-03-23T03:48:56.267069+0000","last_peered":"2024-03-23T03:48:56.267069+0000","last_clean":"2024-03-23T03:48:56.267069+0000","last_became_active":"2024-03-23T03:48:56.262546+0000","last_became_peered":"2024-03-23T03:48:56.262546+0000","last_unstale":"2024-03-23T03:48:56.267069+0000","last_undegraded":"2024-03-23T03:48:56.267069+0000","last_fullsized":"2024-03-23T03:48:56.267069+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:29:25.260317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.271434+0000","last_change":"2024-03-23T03:48:56.263998+0000","last_active":"2024-03-23T03:48:56.271434+0000","last_peered":"2024-03-23T03:48:56.271434+0000","last_clean":"2024-03-23T03:48:56.271434+0000","last_became_active":"2024-03-23T03:48:56.263736+0000","last_became_peered":"2024-03-23T03:48:56.263736+0000","last_unstale":"2024-03-23T03:48:56.271434+0000","last_undegraded":"2024-03-23T03:48:56.271434+0000","last_fullsized":"2024-03-23T03:48:56.271434+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:38:58.872235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.273382+0000","last_change":"2024-03-23T03:48:56.269609+0000","last_active":"2024-03-23T03:48:56.273382+0000","last_peered":"2024-03-23T03:48:56.273382+0000","last_clean":"2024-03-23T03:48:56.273382+0000","last_became_active":"2024-03-23T03:48:56.269281+0000","last_became_peered":"2024-03-23T03:48:56.269281+0000","last_unstale":"2024-03-23T03:48:56.273382+0000","last_undegraded":"2024-03-23T03:48:56.273382+0000","last_fullsized":"2024-03-23T03:48:56.273382+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:59:01.084563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.257340+0000","last_change":"2024-03-23T03:48:56.253595+0000","last_active":"2024-03-23T03:48:56.257340+0000","last_peered":"2024-03-23T03:48:56.257340+0000","last_clean":"2024-03-23T03:48:56.257340+0000","last_became_active":"2024-03-23T03:48:56.253052+0000","last_became_peered":"2024-03-23T03:48:56.253052+0000","last_unstale":"2024-03-23T03:48:56.257340+0000","last_undegraded":"2024-03-23T03:48:56.257340+0000","last_fullsized":"2024-03-23T03:48:56.257340+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:34:33.139812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.273198+0000","last_change":"2024-03-23T03:48:56.271612+0000","last_active":"2024-03-23T03:48:56.273198+0000","last_peered":"2024-03-23T03:48:56.273198+0000","last_clean":"2024-03-23T03:48:56.273198+0000","last_became_active":"2024-03-23T03:48:56.271356+0000","last_became_peered":"2024-03-23T03:48:56.271356+0000","last_unstale":"2024-03-23T03:48:56.273198+0000","last_undegraded":"2024-03-23T03:48:56.273198+0000","last_fullsized":"2024-03-23T03:48:56.273198+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:17:01.745504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.256309+0000","last_change":"2024-03-23T03:48:56.251087+0000","last_active":"2024-03-23T03:48:56.256309+0000","last_peered":"2024-03-23T03:48:56.256309+0000","last_clean":"2024-03-23T03:48:56.256309+0000","last_became_active":"2024-03-23T03:48:56.250791+0000","last_became_peered":"2024-03-23T03:48:56.250791+0000","last_unstale":"2024-03-23T03:48:56.256309+0000","last_undegraded":"2024-03-23T03:48:56.256309+0000","last_fullsized":"2024-03-23T03:48:56.256309+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:03:13.109591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.265656+0000","last_change":"2024-03-23T03:48:56.263858+0000","last_active":"2024-03-23T03:48:56.265656+0000","last_peered":"2024-03-23T03:48:56.265656+0000","last_clean":"2024-03-23T03:48:56.265656+0000","last_became_active":"2024-03-23T03:48:56.263496+0000","last_became_peered":"2024-03-23T03:48:56.263496+0000","last_unstale":"2024-03-23T03:48:56.265656+0000","last_undegraded":"2024-03-23T03:48:56.265656+0000","last_fullsized":"2024-03-23T03:48:56.265656+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:13:20.826378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.263758+0000","last_change":"2024-03-23T03:48:56.262378+0000","last_active":"2024-03-23T03:48:56.263758+0000","last_peered":"2024-03-23T03:48:56.263758+0000","last_clean":"2024-03-23T03:48:56.263758+0000","last_became_active":"2024-03-23T03:48:56.262134+0000","last_became_peered":"2024-03-23T03:48:56.262134+0000","last_unstale":"2024-03-23T03:48:56.263758+0000","last_undegraded":"2024-03-23T03:48:56.263758+0000","last_fullsized":"2024-03-23T03:48:56.263758+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:33:57.629674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.265950+0000","last_change":"2024-03-23T03:48:56.261741+0000","last_active":"2024-03-23T03:48:56.265950+0000","last_peered":"2024-03-23T03:48:56.265950+0000","last_clean":"2024-03-23T03:48:56.265950+0000","last_became_active":"2024-03-23T03:48:56.260465+0000","last_became_peered":"2024-03-23T03:48:56.260465+0000","last_unstale":"2024-03-23T03:48:56.265950+0000","last_undegraded":"2024-03-23T03:48:56.265950+0000","last_fullsized":"2024-03-23T03:48:56.265950+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:48:59.163297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.279559+0000","last_change":"2024-03-23T03:48:56.275890+0000","last_active":"2024-03-23T03:48:56.279559+0000","last_peered":"2024-03-23T03:48:56.279559+0000","last_clean":"2024-03-23T03:48:56.279559+0000","last_became_active":"2024-03-23T03:48:56.274711+0000","last_became_peered":"2024-03-23T03:48:56.274711+0000","last_unstale":"2024-03-23T03:48:56.279559+0000","last_undegraded":"2024-03-23T03:48:56.279559+0000","last_fullsized":"2024-03-23T03:48:56.279559+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:36:16.185523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.266857+0000","last_change":"2024-03-23T03:48:56.262610+0000","last_active":"2024-03-23T03:48:56.266857+0000","last_peered":"2024-03-23T03:48:56.266857+0000","last_clean":"2024-03-23T03:48:56.266857+0000","last_became_active":"2024-03-23T03:48:56.262376+0000","last_became_peered":"2024-03-23T03:48:56.262376+0000","last_unstale":"2024-03-23T03:48:56.266857+0000","last_undegraded":"2024-03-23T03:48:56.266857+0000","last_fullsized":"2024-03-23T03:48:56.266857+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:36:13.491990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.264378+0000","last_change":"2024-03-23T03:48:56.260858+0000","last_active":"2024-03-23T03:48:56.264378+0000","last_peered":"2024-03-23T03:48:56.264378+0000","last_clean":"2024-03-23T03:48:56.264378+0000","last_became_active":"2024-03-23T03:48:56.260599+0000","last_became_peered":"2024-03-23T03:48:56.260599+0000","last_unstale":"2024-03-23T03:48:56.264378+0000","last_undegraded":"2024-03-23T03:48:56.264378+0000","last_fullsized":"2024-03-23T03:48:56.264378+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:01:58.868261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.280090+0000","last_change":"2024-03-23T03:48:56.277162+0000","last_active":"2024-03-23T03:48:56.280090+0000","last_peered":"2024-03-23T03:48:56.280090+0000","last_clean":"2024-03-23T03:48:56.280090+0000","last_became_active":"2024-03-23T03:48:56.276781+0000","last_became_peered":"2024-03-23T03:48:56.276781+0000","last_unstale":"2024-03-23T03:48:56.280090+0000","last_undegraded":"2024-03-23T03:48:56.280090+0000","last_fullsized":"2024-03-23T03:48:56.280090+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:12:22.293248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.278736+0000","last_change":"2024-03-23T03:48:56.272297+0000","last_active":"2024-03-23T03:48:56.278736+0000","last_peered":"2024-03-23T03:48:56.278736+0000","last_clean":"2024-03-23T03:48:56.278736+0000","last_became_active":"2024-03-23T03:48:56.270012+0000","last_became_peered":"2024-03-23T03:48:56.270012+0000","last_unstale":"2024-03-23T03:48:56.278736+0000","last_undegraded":"2024-03-23T03:48:56.278736+0000","last_fullsized":"2024-03-23T03:48:56.278736+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:10:53.300790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.278035+0000","last_change":"2024-03-23T03:48:56.277096+0000","last_active":"2024-03-23T03:48:56.278035+0000","last_peered":"2024-03-23T03:48:56.278035+0000","last_clean":"2024-03-23T03:48:56.278035+0000","last_became_active":"2024-03-23T03:48:56.276824+0000","last_became_peered":"2024-03-23T03:48:56.276824+0000","last_unstale":"2024-03-23T03:48:56.278035+0000","last_undegraded":"2024-03-23T03:48:56.278035+0000","last_fullsized":"2024-03-23T03:48:56.278035+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:25:49.472829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.265074+0000","last_change":"2024-03-23T03:48:56.259672+0000","last_active":"2024-03-23T03:48:56.265074+0000","last_peered":"2024-03-23T03:48:56.265074+0000","last_clean":"2024-03-23T03:48:56.265074+0000","last_became_active":"2024-03-23T03:48:56.259423+0000","last_became_peered":"2024-03-23T03:48:56.259423+0000","last_unstale":"2024-03-23T03:48:56.265074+0000","last_undegraded":"2024-03-23T03:48:56.265074+0000","last_fullsized":"2024-03-23T03:48:56.265074+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:42:11.077571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.275484+0000","last_change":"2024-03-23T03:48:56.274243+0000","last_active":"2024-03-23T03:48:56.275484+0000","last_peered":"2024-03-23T03:48:56.275484+0000","last_clean":"2024-03-23T03:48:56.275484+0000","last_became_active":"2024-03-23T03:48:56.273980+0000","last_became_peered":"2024-03-23T03:48:56.273980+0000","last_unstale":"2024-03-23T03:48:56.275484+0000","last_undegraded":"2024-03-23T03:48:56.275484+0000","last_fullsized":"2024-03-23T03:48:56.275484+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:12.465714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.258079+0000","last_change":"2024-03-23T03:48:56.256380+0000","last_active":"2024-03-23T03:48:56.258079+0000","last_peered":"2024-03-23T03:48:56.258079+0000","last_clean":"2024-03-23T03:48:56.258079+0000","last_became_active":"2024-03-23T03:48:56.256024+0000","last_became_peered":"2024-03-23T03:48:56.256024+0000","last_unstale":"2024-03-23T03:48:56.258079+0000","last_undegraded":"2024-03-23T03:48:56.258079+0000","last_fullsized":"2024-03-23T03:48:56.258079+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:36:45.908256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.242875+0000","last_change":"2024-03-23T03:48:56.241295+0000","last_active":"2024-03-23T03:48:56.242875+0000","last_peered":"2024-03-23T03:48:56.242875+0000","last_clean":"2024-03-23T03:48:56.242875+0000","last_became_active":"2024-03-23T03:48:56.240841+0000","last_became_peered":"2024-03-23T03:48:56.240841+0000","last_unstale":"2024-03-23T03:48:56.242875+0000","last_undegraded":"2024-03-23T03:48:56.242875+0000","last_fullsized":"2024-03-23T03:48:56.242875+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:01.054386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.272207+0000","last_change":"2024-03-23T03:48:56.265433+0000","last_active":"2024-03-23T03:48:56.272207+0000","last_peered":"2024-03-23T03:48:56.272207+0000","last_clean":"2024-03-23T03:48:56.272207+0000","last_became_active":"2024-03-23T03:48:56.264714+0000","last_became_peered":"2024-03-23T03:48:56.264714+0000","last_unstale":"2024-03-23T03:48:56.272207+0000","last_undegraded":"2024-03-23T03:48:56.272207+0000","last_fullsized":"2024-03-23T03:48:56.272207+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:59:06.205562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"54'4","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.264242+0000","last_change":"2024-03-23T03:48:55.776496+0000","last_active":"2024-03-23T03:48:56.264242+0000","last_peered":"2024-03-23T03:48:56.264242+0000","last_clean":"2024-03-23T03:48:56.264242+0000","last_became_active":"2024-03-23T03:48:55.775387+0000","last_became_peered":"2024-03-23T03:48:55.775387+0000","last_unstale":"2024-03-23T03:48:56.264242+0000","last_undegraded":"2024-03-23T03:48:56.264242+0000","last_fullsized":"2024-03-23T03:48:56.264242+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:49:58.504935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.255940+0000","last_change":"2024-03-23T03:48:56.253147+0000","last_active":"2024-03-23T03:48:56.255940+0000","last_peered":"2024-03-23T03:48:56.255940+0000","last_clean":"2024-03-23T03:48:56.255940+0000","last_became_active":"2024-03-23T03:48:56.252867+0000","last_became_peered":"2024-03-23T03:48:56.252867+0000","last_unstale":"2024-03-23T03:48:56.255940+0000","last_undegraded":"2024-03-23T03:48:56.255940+0000","last_fullsized":"2024-03-23T03:48:56.255940+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:43:17.526730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.267930+0000","last_change":"2024-03-23T03:48:55.786072+0000","last_active":"2024-03-23T03:48:56.267930+0000","last_peered":"2024-03-23T03:48:56.267930+0000","last_clean":"2024-03-23T03:48:56.267930+0000","last_became_active":"2024-03-23T03:48:55.785756+0000","last_became_peered":"2024-03-23T03:48:55.785756+0000","last_unstale":"2024-03-23T03:48:56.267930+0000","last_undegraded":"2024-03-23T03:48:56.267930+0000","last_fullsized":"2024-03-23T03:48:56.267930+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:42:06.491607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.267953+0000","last_change":"2024-03-23T03:48:56.265972+0000","last_active":"2024-03-23T03:48:56.267953+0000","last_peered":"2024-03-23T03:48:56.267953+0000","last_clean":"2024-03-23T03:48:56.267953+0000","last_became_active":"2024-03-23T03:48:56.265609+0000","last_became_peered":"2024-03-23T03:48:56.265609+0000","last_unstale":"2024-03-23T03:48:56.267953+0000","last_undegraded":"2024-03-23T03:48:56.267953+0000","last_fullsized":"2024-03-23T03:48:56.267953+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:23:10.621637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.273295+0000","last_change":"2024-03-23T03:48:55.779777+0000","last_active":"2024-03-23T03:48:56.273295+0000","last_peered":"2024-03-23T03:48:56.273295+0000","last_clean":"2024-03-23T03:48:56.273295+0000","last_became_active":"2024-03-23T03:48:55.779535+0000","last_became_peered":"2024-03-23T03:48:55.779535+0000","last_unstale":"2024-03-23T03:48:56.273295+0000","last_undegraded":"2024-03-23T03:48:56.273295+0000","last_fullsized":"2024-03-23T03:48:56.273295+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:59:05.193850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.280110+0000","last_change":"2024-03-23T03:48:56.275704+0000","last_active":"2024-03-23T03:48:56.280110+0000","last_peered":"2024-03-23T03:48:56.280110+0000","last_clean":"2024-03-23T03:48:56.280110+0000","last_became_active":"2024-03-23T03:48:56.275421+0000","last_became_peered":"2024-03-23T03:48:56.275421+0000","last_unstale":"2024-03-23T03:48:56.280110+0000","last_undegraded":"2024-03-23T03:48:56.280110+0000","last_fullsized":"2024-03-23T03:48:56.280110+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:14:25.292802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.261754+0000","last_change":"2024-03-23T03:48:55.775356+0000","last_active":"2024-03-23T03:48:56.261754+0000","last_peered":"2024-03-23T03:48:56.261754+0000","last_clean":"2024-03-23T03:48:56.261754+0000","last_became_active":"2024-03-23T03:48:55.775055+0000","last_became_peered":"2024-03-23T03:48:55.775055+0000","last_unstale":"2024-03-23T03:48:56.261754+0000","last_undegraded":"2024-03-23T03:48:56.261754+0000","last_fullsized":"2024-03-23T03:48:56.261754+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:15:55.585161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.255560+0000","last_change":"2024-03-23T03:48:56.251432+0000","last_active":"2024-03-23T03:48:56.255560+0000","last_peered":"2024-03-23T03:48:56.255560+0000","last_clean":"2024-03-23T03:48:56.255560+0000","last_became_active":"2024-03-23T03:48:56.251131+0000","last_became_peered":"2024-03-23T03:48:56.251131+0000","last_unstale":"2024-03-23T03:48:56.255560+0000","last_undegraded":"2024-03-23T03:48:56.255560+0000","last_fullsized":"2024-03-23T03:48:56.255560+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:48:41.795470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.277624+0000","last_change":"2024-03-23T03:48:55.774218+0000","last_active":"2024-03-23T03:48:56.277624+0000","last_peered":"2024-03-23T03:48:56.277624+0000","last_clean":"2024-03-23T03:48:56.277624+0000","last_became_active":"2024-03-23T03:48:55.772251+0000","last_became_peered":"2024-03-23T03:48:55.772251+0000","last_unstale":"2024-03-23T03:48:56.277624+0000","last_undegraded":"2024-03-23T03:48:56.277624+0000","last_fullsized":"2024-03-23T03:48:56.277624+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:00:23.746711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.276225+0000","last_change":"2024-03-23T03:48:56.274810+0000","last_active":"2024-03-23T03:48:56.276225+0000","last_peered":"2024-03-23T03:48:56.276225+0000","last_clean":"2024-03-23T03:48:56.276225+0000","last_became_active":"2024-03-23T03:48:56.274466+0000","last_became_peered":"2024-03-23T03:48:56.274466+0000","last_unstale":"2024-03-23T03:48:56.276225+0000","last_undegraded":"2024-03-23T03:48:56.276225+0000","last_fullsized":"2024-03-23T03:48:56.276225+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:15:56.596873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.248221+0000","last_change":"2024-03-23T03:48:55.777362+0000","last_active":"2024-03-23T03:48:56.248221+0000","last_peered":"2024-03-23T03:48:56.248221+0000","last_clean":"2024-03-23T03:48:56.248221+0000","last_became_active":"2024-03-23T03:48:55.777020+0000","last_became_peered":"2024-03-23T03:48:55.777020+0000","last_unstale":"2024-03-23T03:48:56.248221+0000","last_undegraded":"2024-03-23T03:48:56.248221+0000","last_fullsized":"2024-03-23T03:48:56.248221+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:25:48.461117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.280071+0000","last_change":"2024-03-23T03:48:56.277008+0000","last_active":"2024-03-23T03:48:56.280071+0000","last_peered":"2024-03-23T03:48:56.280071+0000","last_clean":"2024-03-23T03:48:56.280071+0000","last_became_active":"2024-03-23T03:48:56.276477+0000","last_became_peered":"2024-03-23T03:48:56.276477+0000","last_unstale":"2024-03-23T03:48:56.280071+0000","last_undegraded":"2024-03-23T03:48:56.280071+0000","last_fullsized":"2024-03-23T03:48:56.280071+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:32:35.308769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.247012+0000","last_change":"2024-03-23T03:48:55.782381+0000","last_active":"2024-03-23T03:48:56.247012+0000","last_peered":"2024-03-23T03:48:56.247012+0000","last_clean":"2024-03-23T03:48:56.247012+0000","last_became_active":"2024-03-23T03:48:55.782097+0000","last_became_peered":"2024-03-23T03:48:55.782097+0000","last_unstale":"2024-03-23T03:48:56.247012+0000","last_undegraded":"2024-03-23T03:48:56.247012+0000","last_fullsized":"2024-03-23T03:48:56.247012+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:04:49.079335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.272239+0000","last_change":"2024-03-23T03:48:56.269631+0000","last_active":"2024-03-23T03:48:56.272239+0000","last_peered":"2024-03-23T03:48:56.272239+0000","last_clean":"2024-03-23T03:48:56.272239+0000","last_became_active":"2024-03-23T03:48:56.268238+0000","last_became_peered":"2024-03-23T03:48:56.268238+0000","last_unstale":"2024-03-23T03:48:56.272239+0000","last_undegraded":"2024-03-23T03:48:56.272239+0000","last_fullsized":"2024-03-23T03:48:56.272239+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:40:07.408603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.241625+0000","last_change":"2024-03-23T03:48:55.776679+0000","last_active":"2024-03-23T03:48:56.241625+0000","last_peered":"2024-03-23T03:48:56.241625+0000","last_clean":"2024-03-23T03:48:56.241625+0000","last_became_active":"2024-03-23T03:48:55.776414+0000","last_became_peered":"2024-03-23T03:48:55.776414+0000","last_unstale":"2024-03-23T03:48:56.241625+0000","last_undegraded":"2024-03-23T03:48:56.241625+0000","last_fullsized":"2024-03-23T03:48:56.241625+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:11.454002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.262468+0000","last_change":"2024-03-23T03:48:56.256609+0000","last_active":"2024-03-23T03:48:56.262468+0000","last_peered":"2024-03-23T03:48:56.262468+0000","last_clean":"2024-03-23T03:48:56.262468+0000","last_became_active":"2024-03-23T03:48:56.256313+0000","last_became_peered":"2024-03-23T03:48:56.256313+0000","last_unstale":"2024-03-23T03:48:56.262468+0000","last_undegraded":"2024-03-23T03:48:56.262468+0000","last_fullsized":"2024-03-23T03:48:56.262468+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:10:36.220788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.257226+0000","last_change":"2024-03-23T03:48:55.786598+0000","last_active":"2024-03-23T03:48:56.257226+0000","last_peered":"2024-03-23T03:48:56.257226+0000","last_clean":"2024-03-23T03:48:56.257226+0000","last_became_active":"2024-03-23T03:48:55.786261+0000","last_became_peered":"2024-03-23T03:48:55.786261+0000","last_unstale":"2024-03-23T03:48:56.257226+0000","last_undegraded":"2024-03-23T03:48:56.257226+0000","last_fullsized":"2024-03-23T03:48:56.257226+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:59:57.078934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.253032+0000","last_change":"2024-03-23T03:48:56.245972+0000","last_active":"2024-03-23T03:48:56.253032+0000","last_peered":"2024-03-23T03:48:56.253032+0000","last_clean":"2024-03-23T03:48:56.253032+0000","last_became_active":"2024-03-23T03:48:56.245632+0000","last_became_peered":"2024-03-23T03:48:56.245632+0000","last_unstale":"2024-03-23T03:48:56.253032+0000","last_undegraded":"2024-03-23T03:48:56.253032+0000","last_fullsized":"2024-03-23T03:48:56.253032+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:37:28.870018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.238952+0000","last_change":"2024-03-23T03:48:55.768555+0000","last_active":"2024-03-23T03:48:56.238952+0000","last_peered":"2024-03-23T03:48:56.238952+0000","last_clean":"2024-03-23T03:48:56.238952+0000","last_became_active":"2024-03-23T03:48:55.768015+0000","last_became_peered":"2024-03-23T03:48:55.768015+0000","last_unstale":"2024-03-23T03:48:56.238952+0000","last_undegraded":"2024-03-23T03:48:56.238952+0000","last_fullsized":"2024-03-23T03:48:56.238952+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:33.847006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.257591+0000","last_change":"2024-03-23T03:48:56.254570+0000","last_active":"2024-03-23T03:48:56.257591+0000","last_peered":"2024-03-23T03:48:56.257591+0000","last_clean":"2024-03-23T03:48:56.257591+0000","last_became_active":"2024-03-23T03:48:56.254262+0000","last_became_peered":"2024-03-23T03:48:56.254262+0000","last_unstale":"2024-03-23T03:48:56.257591+0000","last_undegraded":"2024-03-23T03:48:56.257591+0000","last_fullsized":"2024-03-23T03:48:56.257591+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:21:25.502129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"54'3","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.279798+0000","last_change":"2024-03-23T03:48:55.784085+0000","last_active":"2024-03-23T03:48:56.279798+0000","last_peered":"2024-03-23T03:48:56.279798+0000","last_clean":"2024-03-23T03:48:56.279798+0000","last_became_active":"2024-03-23T03:48:55.783801+0000","last_became_peered":"2024-03-23T03:48:55.783801+0000","last_unstale":"2024-03-23T03:48:56.279798+0000","last_undegraded":"2024-03-23T03:48:56.279798+0000","last_fullsized":"2024-03-23T03:48:56.279798+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:56:14.033603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.259752+0000","last_change":"2024-03-23T03:48:56.252633+0000","last_active":"2024-03-23T03:48:56.259752+0000","last_peered":"2024-03-23T03:48:56.259752+0000","last_clean":"2024-03-23T03:48:56.259752+0000","last_became_active":"2024-03-23T03:48:56.252360+0000","last_became_peered":"2024-03-23T03:48:56.252360+0000","last_unstale":"2024-03-23T03:48:56.259752+0000","last_undegraded":"2024-03-23T03:48:56.259752+0000","last_fullsized":"2024-03-23T03:48:56.259752+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:46:32.495476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"54'6","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.283096+0000","last_change":"2024-03-23T03:48:55.780941+0000","last_active":"2024-03-23T03:48:56.283096+0000","last_peered":"2024-03-23T03:48:56.283096+0000","last_clean":"2024-03-23T03:48:56.283096+0000","last_became_active":"2024-03-23T03:48:55.780657+0000","last_became_peered":"2024-03-23T03:48:55.780657+0000","last_unstale":"2024-03-23T03:48:56.283096+0000","last_undegraded":"2024-03-23T03:48:56.283096+0000","last_fullsized":"2024-03-23T03:48:56.283096+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.110218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.273914+0000","last_change":"2024-03-23T03:48:56.271858+0000","last_active":"2024-03-23T03:48:56.273914+0000","last_peered":"2024-03-23T03:48:56.273914+0000","last_clean":"2024-03-23T03:48:56.273914+0000","last_became_active":"2024-03-23T03:48:56.271567+0000","last_became_peered":"2024-03-23T03:48:56.271567+0000","last_unstale":"2024-03-23T03:48:56.273914+0000","last_undegraded":"2024-03-23T03:48:56.273914+0000","last_fullsized":"2024-03-23T03:48:56.273914+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:10:53.300790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.272726+0000","last_change":"2024-03-23T03:48:55.779456+0000","last_active":"2024-03-23T03:48:56.272726+0000","last_peered":"2024-03-23T03:48:56.272726+0000","last_clean":"2024-03-23T03:48:56.272726+0000","last_became_active":"2024-03-23T03:48:55.779164+0000","last_became_peered":"2024-03-23T03:48:55.779164+0000","last_unstale":"2024-03-23T03:48:56.272726+0000","last_undegraded":"2024-03-23T03:48:56.272726+0000","last_fullsized":"2024-03-23T03:48:56.272726+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:17:00.733792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.276218+0000","last_change":"2024-03-23T03:48:56.274098+0000","last_active":"2024-03-23T03:48:56.276218+0000","last_peered":"2024-03-23T03:48:56.276218+0000","last_clean":"2024-03-23T03:48:56.276218+0000","last_became_active":"2024-03-23T03:48:56.273863+0000","last_became_peered":"2024-03-23T03:48:56.273863+0000","last_unstale":"2024-03-23T03:48:56.276218+0000","last_undegraded":"2024-03-23T03:48:56.276218+0000","last_fullsized":"2024-03-23T03:48:56.276218+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:14:05.067038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.247938+0000","last_change":"2024-03-23T03:48:55.784949+0000","last_active":"2024-03-23T03:48:56.247938+0000","last_peered":"2024-03-23T03:48:56.247938+0000","last_clean":"2024-03-23T03:48:56.247938+0000","last_became_active":"2024-03-23T03:48:55.784608+0000","last_became_peered":"2024-03-23T03:48:55.784608+0000","last_unstale":"2024-03-23T03:48:56.247938+0000","last_undegraded":"2024-03-23T03:48:56.247938+0000","last_fullsized":"2024-03-23T03:48:56.247938+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:20:44.840360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.267438+0000","last_change":"2024-03-23T03:48:56.265659+0000","last_active":"2024-03-23T03:48:56.267438+0000","last_peered":"2024-03-23T03:48:56.267438+0000","last_clean":"2024-03-23T03:48:56.267438+0000","last_became_active":"2024-03-23T03:48:56.265413+0000","last_became_peered":"2024-03-23T03:48:56.265413+0000","last_unstale":"2024-03-23T03:48:56.267438+0000","last_undegraded":"2024-03-23T03:48:56.267438+0000","last_fullsized":"2024-03-23T03:48:56.267438+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:06:57.815629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.249840+0000","last_change":"2024-03-23T03:48:55.782219+0000","last_active":"2024-03-23T03:48:56.249840+0000","last_peered":"2024-03-23T03:48:56.249840+0000","last_clean":"2024-03-23T03:48:56.249840+0000","last_became_active":"2024-03-23T03:48:55.781945+0000","last_became_peered":"2024-03-23T03:48:55.781945+0000","last_unstale":"2024-03-23T03:48:56.249840+0000","last_undegraded":"2024-03-23T03:48:56.249840+0000","last_fullsized":"2024-03-23T03:48:56.249840+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:05:17.151576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.257247+0000","last_change":"2024-03-23T03:48:56.251900+0000","last_active":"2024-03-23T03:48:56.257247+0000","last_peered":"2024-03-23T03:48:56.257247+0000","last_clean":"2024-03-23T03:48:56.257247+0000","last_became_active":"2024-03-23T03:48:56.251665+0000","last_became_peered":"2024-03-23T03:48:56.251665+0000","last_unstale":"2024-03-23T03:48:56.257247+0000","last_undegraded":"2024-03-23T03:48:56.257247+0000","last_fullsized":"2024-03-23T03:48:56.257247+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:04.121930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"54'8","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.281044+0000","last_change":"2024-03-23T03:48:55.776122+0000","last_active":"2024-03-23T03:48:56.281044+0000","last_peered":"2024-03-23T03:48:56.281044+0000","last_clean":"2024-03-23T03:48:56.281044+0000","last_became_active":"2024-03-23T03:48:55.775751+0000","last_became_peered":"2024-03-23T03:48:55.775751+0000","last_unstale":"2024-03-23T03:48:56.281044+0000","last_undegraded":"2024-03-23T03:48:56.281044+0000","last_fullsized":"2024-03-23T03:48:56.281044+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:36:12.480278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.261335+0000","last_change":"2024-03-23T03:48:56.246843+0000","last_active":"2024-03-23T03:48:56.261335+0000","last_peered":"2024-03-23T03:48:56.261335+0000","last_clean":"2024-03-23T03:48:56.261335+0000","last_became_active":"2024-03-23T03:48:56.246264+0000","last_became_peered":"2024-03-23T03:48:56.246264+0000","last_unstale":"2024-03-23T03:48:56.261335+0000","last_undegraded":"2024-03-23T03:48:56.261335+0000","last_fullsized":"2024-03-23T03:48:56.261335+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:45:54.545516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.265129+0000","last_change":"2024-03-23T03:48:55.781632+0000","last_active":"2024-03-23T03:48:56.265129+0000","last_peered":"2024-03-23T03:48:56.265129+0000","last_clean":"2024-03-23T03:48:56.265129+0000","last_became_active":"2024-03-23T03:48:55.781376+0000","last_became_peered":"2024-03-23T03:48:55.781376+0000","last_unstale":"2024-03-23T03:48:56.265129+0000","last_undegraded":"2024-03-23T03:48:56.265129+0000","last_fullsized":"2024-03-23T03:48:56.265129+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:40:06.396891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.273556+0000","last_change":"2024-03-23T03:48:56.269228+0000","last_active":"2024-03-23T03:48:56.273556+0000","last_peered":"2024-03-23T03:48:56.273556+0000","last_clean":"2024-03-23T03:48:56.273556+0000","last_became_active":"2024-03-23T03:48:56.268269+0000","last_became_peered":"2024-03-23T03:48:56.268269+0000","last_unstale":"2024-03-23T03:48:56.273556+0000","last_undegraded":"2024-03-23T03:48:56.273556+0000","last_fullsized":"2024-03-23T03:48:56.273556+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:22:29.565107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.276545+0000","last_change":"2024-03-23T03:48:55.763920+0000","last_active":"2024-03-23T03:48:56.276545+0000","last_peered":"2024-03-23T03:48:56.276545+0000","last_clean":"2024-03-23T03:48:56.276545+0000","last_became_active":"2024-03-23T03:48:55.763657+0000","last_became_peered":"2024-03-23T03:48:55.763657+0000","last_unstale":"2024-03-23T03:48:56.276545+0000","last_undegraded":"2024-03-23T03:48:56.276545+0000","last_fullsized":"2024-03-23T03:48:56.276545+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:50:56.772143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.256767+0000","last_change":"2024-03-23T03:48:56.255077+0000","last_active":"2024-03-23T03:48:56.256767+0000","last_peered":"2024-03-23T03:48:56.256767+0000","last_clean":"2024-03-23T03:48:56.256767+0000","last_became_active":"2024-03-23T03:48:56.254811+0000","last_became_peered":"2024-03-23T03:48:56.254811+0000","last_unstale":"2024-03-23T03:48:56.256767+0000","last_undegraded":"2024-03-23T03:48:56.256767+0000","last_fullsized":"2024-03-23T03:48:56.256767+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:43:22.788448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.273935+0000","last_change":"2024-03-23T03:48:55.780236+0000","last_active":"2024-03-23T03:48:56.273935+0000","last_peered":"2024-03-23T03:48:56.273935+0000","last_clean":"2024-03-23T03:48:56.273935+0000","last_became_active":"2024-03-23T03:48:55.779941+0000","last_became_peered":"2024-03-23T03:48:55.779941+0000","last_unstale":"2024-03-23T03:48:56.273935+0000","last_undegraded":"2024-03-23T03:48:56.273935+0000","last_fullsized":"2024-03-23T03:48:56.273935+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:13.367789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.272023+0000","last_change":"2024-03-23T03:48:56.264079+0000","last_active":"2024-03-23T03:48:56.272023+0000","last_peered":"2024-03-23T03:48:56.272023+0000","last_clean":"2024-03-23T03:48:56.272023+0000","last_became_active":"2024-03-23T03:48:56.263787+0000","last_became_peered":"2024-03-23T03:48:56.263787+0000","last_unstale":"2024-03-23T03:48:56.272023+0000","last_undegraded":"2024-03-23T03:48:56.272023+0000","last_fullsized":"2024-03-23T03:48:56.272023+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:33:31.920802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.250955+0000","last_change":"2024-03-23T03:48:55.779509+0000","last_active":"2024-03-23T03:48:56.250955+0000","last_peered":"2024-03-23T03:48:56.250955+0000","last_clean":"2024-03-23T03:48:56.250955+0000","last_became_active":"2024-03-23T03:48:55.779261+0000","last_became_peered":"2024-03-23T03:48:55.779261+0000","last_unstale":"2024-03-23T03:48:56.250955+0000","last_undegraded":"2024-03-23T03:48:56.250955+0000","last_fullsized":"2024-03-23T03:48:56.250955+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:44:44.587651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.245928+0000","last_change":"2024-03-23T03:48:56.243411+0000","last_active":"2024-03-23T03:48:56.245928+0000","last_peered":"2024-03-23T03:48:56.245928+0000","last_clean":"2024-03-23T03:48:56.245928+0000","last_became_active":"2024-03-23T03:48:56.243087+0000","last_became_peered":"2024-03-23T03:48:56.243087+0000","last_unstale":"2024-03-23T03:48:56.245928+0000","last_undegraded":"2024-03-23T03:48:56.245928+0000","last_fullsized":"2024-03-23T03:48:56.245928+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:59:01.084563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.278484+0000","last_change":"2024-03-23T03:48:55.780119+0000","last_active":"2024-03-23T03:48:56.278484+0000","last_peered":"2024-03-23T03:48:56.278484+0000","last_clean":"2024-03-23T03:48:56.278484+0000","last_became_active":"2024-03-23T03:48:55.779873+0000","last_became_peered":"2024-03-23T03:48:55.779873+0000","last_unstale":"2024-03-23T03:48:56.278484+0000","last_undegraded":"2024-03-23T03:48:56.278484+0000","last_fullsized":"2024-03-23T03:48:56.278484+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:27:54.806505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.262951+0000","last_change":"2024-03-23T03:48:56.261119+0000","last_active":"2024-03-23T03:48:56.262951+0000","last_peered":"2024-03-23T03:48:56.262951+0000","last_clean":"2024-03-23T03:48:56.262951+0000","last_became_active":"2024-03-23T03:48:56.260835+0000","last_became_peered":"2024-03-23T03:48:56.260835+0000","last_unstale":"2024-03-23T03:48:56.262951+0000","last_undegraded":"2024-03-23T03:48:56.262951+0000","last_fullsized":"2024-03-23T03:48:56.262951+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:36:00.205847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.231383+0000","last_change":"2024-03-23T03:48:55.781339+0000","last_active":"2024-03-23T03:48:56.231383+0000","last_peered":"2024-03-23T03:48:56.231383+0000","last_clean":"2024-03-23T03:48:56.231383+0000","last_became_active":"2024-03-23T03:48:55.781083+0000","last_became_peered":"2024-03-23T03:48:55.781083+0000","last_unstale":"2024-03-23T03:48:56.231383+0000","last_undegraded":"2024-03-23T03:48:56.231383+0000","last_fullsized":"2024-03-23T03:48:56.231383+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:36:44.896544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.257387+0000","last_change":"2024-03-23T03:48:56.253769+0000","last_active":"2024-03-23T03:48:56.257387+0000","last_peered":"2024-03-23T03:48:56.257387+0000","last_clean":"2024-03-23T03:48:56.257387+0000","last_became_active":"2024-03-23T03:48:56.249295+0000","last_became_peered":"2024-03-23T03:48:56.249295+0000","last_unstale":"2024-03-23T03:48:56.257387+0000","last_undegraded":"2024-03-23T03:48:56.257387+0000","last_fullsized":"2024-03-23T03:48:56.257387+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:22:09.094667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.250717+0000","last_change":"2024-03-23T03:48:55.776213+0000","last_active":"2024-03-23T03:48:56.250717+0000","last_peered":"2024-03-23T03:48:56.250717+0000","last_clean":"2024-03-23T03:48:56.250717+0000","last_became_active":"2024-03-23T03:48:55.775857+0000","last_became_peered":"2024-03-23T03:48:55.775857+0000","last_unstale":"2024-03-23T03:48:56.250717+0000","last_undegraded":"2024-03-23T03:48:56.250717+0000","last_fullsized":"2024-03-23T03:48:56.250717+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:07.115817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.271202+0000","last_change":"2024-03-23T03:48:56.268547+0000","last_active":"2024-03-23T03:48:56.271202+0000","last_peered":"2024-03-23T03:48:56.271202+0000","last_clean":"2024-03-23T03:48:56.271202+0000","last_became_active":"2024-03-23T03:48:56.268240+0000","last_became_peered":"2024-03-23T03:48:56.268240+0000","last_unstale":"2024-03-23T03:48:56.271202+0000","last_undegraded":"2024-03-23T03:48:56.271202+0000","last_fullsized":"2024-03-23T03:48:56.271202+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:31:34.716309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.260003+0000","last_change":"2024-03-23T03:48:55.784710+0000","last_active":"2024-03-23T03:48:56.260003+0000","last_peered":"2024-03-23T03:48:56.260003+0000","last_clean":"2024-03-23T03:48:56.260003+0000","last_became_active":"2024-03-23T03:48:55.784449+0000","last_became_peered":"2024-03-23T03:48:55.784449+0000","last_unstale":"2024-03-23T03:48:56.260003+0000","last_undegraded":"2024-03-23T03:48:56.260003+0000","last_fullsized":"2024-03-23T03:48:56.260003+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:15:32.774243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.242179+0000","last_change":"2024-03-23T03:48:56.241063+0000","last_active":"2024-03-23T03:48:56.242179+0000","last_peered":"2024-03-23T03:48:56.242179+0000","last_clean":"2024-03-23T03:48:56.242179+0000","last_became_active":"2024-03-23T03:48:56.240608+0000","last_became_peered":"2024-03-23T03:48:56.240608+0000","last_unstale":"2024-03-23T03:48:56.242179+0000","last_undegraded":"2024-03-23T03:48:56.242179+0000","last_fullsized":"2024-03-23T03:48:56.242179+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:22:29.565107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.262819+0000","last_change":"2024-03-23T03:48:55.783911+0000","last_active":"2024-03-23T03:48:56.262819+0000","last_peered":"2024-03-23T03:48:56.262819+0000","last_clean":"2024-03-23T03:48:56.262819+0000","last_became_active":"2024-03-23T03:48:55.783403+0000","last_became_peered":"2024-03-23T03:48:55.783403+0000","last_unstale":"2024-03-23T03:48:56.262819+0000","last_undegraded":"2024-03-23T03:48:56.262819+0000","last_fullsized":"2024-03-23T03:48:56.262819+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:33:20.082454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.282760+0000","last_change":"2024-03-23T03:48:56.281595+0000","last_active":"2024-03-23T03:48:56.282760+0000","last_peered":"2024-03-23T03:48:56.282760+0000","last_clean":"2024-03-23T03:48:56.282760+0000","last_became_active":"2024-03-23T03:48:56.280264+0000","last_became_peered":"2024-03-23T03:48:56.280264+0000","last_unstale":"2024-03-23T03:48:56.282760+0000","last_undegraded":"2024-03-23T03:48:56.282760+0000","last_fullsized":"2024-03-23T03:48:56.282760+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:43:17.526730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.256701+0000","last_change":"2024-03-23T03:48:55.785525+0000","last_active":"2024-03-23T03:48:56.256701+0000","last_peered":"2024-03-23T03:48:56.256701+0000","last_clean":"2024-03-23T03:48:56.256701+0000","last_became_active":"2024-03-23T03:48:55.785193+0000","last_became_peered":"2024-03-23T03:48:55.785193+0000","last_unstale":"2024-03-23T03:48:56.256701+0000","last_undegraded":"2024-03-23T03:48:56.256701+0000","last_fullsized":"2024-03-23T03:48:56.256701+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:37:02.476835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.259841+0000","last_change":"2024-03-23T03:48:56.252968+0000","last_active":"2024-03-23T03:48:56.259841+0000","last_peered":"2024-03-23T03:48:56.259841+0000","last_clean":"2024-03-23T03:48:56.259841+0000","last_became_active":"2024-03-23T03:48:56.248398+0000","last_became_peered":"2024-03-23T03:48:56.248398+0000","last_unstale":"2024-03-23T03:48:56.259841+0000","last_undegraded":"2024-03-23T03:48:56.259841+0000","last_fullsized":"2024-03-23T03:48:56.259841+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:21:25.502129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.244106+0000","last_change":"2024-03-23T03:48:55.769507+0000","last_active":"2024-03-23T03:48:56.244106+0000","last_peered":"2024-03-23T03:48:56.244106+0000","last_clean":"2024-03-23T03:48:56.244106+0000","last_became_active":"2024-03-23T03:48:55.769228+0000","last_became_peered":"2024-03-23T03:48:55.769228+0000","last_unstale":"2024-03-23T03:48:56.244106+0000","last_undegraded":"2024-03-23T03:48:56.244106+0000","last_fullsized":"2024-03-23T03:48:56.244106+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:31:15.531840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.265729+0000","last_change":"2024-03-23T03:48:56.261090+0000","last_active":"2024-03-23T03:48:56.265729+0000","last_peered":"2024-03-23T03:48:56.265729+0000","last_clean":"2024-03-23T03:48:56.265729+0000","last_became_active":"2024-03-23T03:48:56.260329+0000","last_became_peered":"2024-03-23T03:48:56.260329+0000","last_unstale":"2024-03-23T03:48:56.265729+0000","last_undegraded":"2024-03-23T03:48:56.265729+0000","last_fullsized":"2024-03-23T03:48:56.265729+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:03:46.310690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.232118+0000","last_change":"2024-03-23T03:48:55.782142+0000","last_active":"2024-03-23T03:48:56.232118+0000","last_peered":"2024-03-23T03:48:56.232118+0000","last_clean":"2024-03-23T03:48:56.232118+0000","last_became_active":"2024-03-23T03:48:55.781873+0000","last_became_peered":"2024-03-23T03:48:55.781873+0000","last_unstale":"2024-03-23T03:48:56.232118+0000","last_undegraded":"2024-03-23T03:48:56.232118+0000","last_fullsized":"2024-03-23T03:48:56.232118+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:34:32.128100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.255916+0000","last_change":"2024-03-23T03:48:56.252518+0000","last_active":"2024-03-23T03:48:56.255916+0000","last_peered":"2024-03-23T03:48:56.255916+0000","last_clean":"2024-03-23T03:48:56.255916+0000","last_became_active":"2024-03-23T03:48:56.252233+0000","last_became_peered":"2024-03-23T03:48:56.252233+0000","last_unstale":"2024-03-23T03:48:56.255916+0000","last_undegraded":"2024-03-23T03:48:56.255916+0000","last_fullsized":"2024-03-23T03:48:56.255916+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:24:15.405965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.231531+0000","last_change":"2024-03-23T03:48:55.782566+0000","last_active":"2024-03-23T03:48:56.231531+0000","last_peered":"2024-03-23T03:48:56.231531+0000","last_clean":"2024-03-23T03:48:56.231531+0000","last_became_active":"2024-03-23T03:48:55.782239+0000","last_became_peered":"2024-03-23T03:48:55.782239+0000","last_unstale":"2024-03-23T03:48:56.231531+0000","last_undegraded":"2024-03-23T03:48:56.231531+0000","last_fullsized":"2024-03-23T03:48:56.231531+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:29:24.248606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.267053+0000","last_change":"2024-03-23T03:48:56.262751+0000","last_active":"2024-03-23T03:48:56.267053+0000","last_peered":"2024-03-23T03:48:56.267053+0000","last_clean":"2024-03-23T03:48:56.267053+0000","last_became_active":"2024-03-23T03:48:56.262484+0000","last_became_peered":"2024-03-23T03:48:56.262484+0000","last_unstale":"2024-03-23T03:48:56.267053+0000","last_undegraded":"2024-03-23T03:48:56.267053+0000","last_fullsized":"2024-03-23T03:48:56.267053+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:19:28.397582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.238924+0000","last_change":"2024-03-23T03:48:55.772104+0000","last_active":"2024-03-23T03:48:56.238924+0000","last_peered":"2024-03-23T03:48:56.238924+0000","last_clean":"2024-03-23T03:48:56.238924+0000","last_became_active":"2024-03-23T03:48:55.771758+0000","last_became_peered":"2024-03-23T03:48:55.771758+0000","last_unstale":"2024-03-23T03:48:56.238924+0000","last_undegraded":"2024-03-23T03:48:56.238924+0000","last_fullsized":"2024-03-23T03:48:56.238924+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:13:19.814666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.281108+0000","last_change":"2024-03-23T03:48:56.275704+0000","last_active":"2024-03-23T03:48:56.281108+0000","last_peered":"2024-03-23T03:48:56.281108+0000","last_clean":"2024-03-23T03:48:56.281108+0000","last_became_active":"2024-03-23T03:48:56.275462+0000","last_became_peered":"2024-03-23T03:48:56.275462+0000","last_unstale":"2024-03-23T03:48:56.281108+0000","last_undegraded":"2024-03-23T03:48:56.281108+0000","last_fullsized":"2024-03-23T03:48:56.281108+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:47:59.122958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.236211+0000","last_change":"2024-03-23T03:48:55.780396+0000","last_active":"2024-03-23T03:48:56.236211+0000","last_peered":"2024-03-23T03:48:56.236211+0000","last_clean":"2024-03-23T03:48:56.236211+0000","last_became_active":"2024-03-23T03:48:55.780039+0000","last_became_peered":"2024-03-23T03:48:55.780039+0000","last_unstale":"2024-03-23T03:48:56.236211+0000","last_undegraded":"2024-03-23T03:48:56.236211+0000","last_fullsized":"2024-03-23T03:48:56.236211+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:11.454002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.263309+0000","last_change":"2024-03-23T03:48:56.259407+0000","last_active":"2024-03-23T03:48:56.263309+0000","last_peered":"2024-03-23T03:48:56.263309+0000","last_clean":"2024-03-23T03:48:56.263309+0000","last_became_active":"2024-03-23T03:48:56.258543+0000","last_became_peered":"2024-03-23T03:48:56.258543+0000","last_unstale":"2024-03-23T03:48:56.263309+0000","last_undegraded":"2024-03-23T03:48:56.263309+0000","last_fullsized":"2024-03-23T03:48:56.263309+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:34.858717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.241911+0000","last_change":"2024-03-23T03:48:55.778988+0000","last_active":"2024-03-23T03:48:56.241911+0000","last_peered":"2024-03-23T03:48:56.241911+0000","last_clean":"2024-03-23T03:48:56.241911+0000","last_became_active":"2024-03-23T03:48:55.778701+0000","last_became_peered":"2024-03-23T03:48:55.778701+0000","last_unstale":"2024-03-23T03:48:56.241911+0000","last_undegraded":"2024-03-23T03:48:56.241911+0000","last_fullsized":"2024-03-23T03:48:56.241911+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:08:42.513969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":190,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-23T03:48:55.775427+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:48:55.775427+0000","last_peered":"2024-03-23T03:48:55.775427+0000","last_clean":"2024-03-23T03:48:55.775427+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:48:55.775427+0000","last_undegraded":"2024-03-23T03:48:55.775427+0000","last_fullsized":"2024-03-23T03:48:55.775427+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542174,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28240,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343600,"statfs":{"total":96636764160,"available":96607846400,"internally_reserved":0,"allocated":1241088,"data_stored":870780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.079,"15min":1.069},"min":{"1min":0.66800000000000004,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.6279999999999999,"5min":1.635,"15min":1.635},"last":0.996},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0569999999999999,"15min":1.052},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.48,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.3280000000000001}]},{"osd":1,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.0589999999999999,"15min":1.0449999999999999},"min":{"1min":0.70099999999999996,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.655,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.2829999999999999,"15min":1.3180000000000001},"min":{"1min":0.72599999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.472,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":1.0649999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.147,"15min":1.159},"min":{"1min":0.79200000000000004,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.4870000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.296},{"interface":"front","average":{"1min":1.198,"5min":1.4059999999999999,"15min":1.4410000000000001},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.522,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.2589999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.4239999999999999,"15min":1.46},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.5900000000000001,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.026},{"interface":"front","average":{"1min":1.319,"5min":1.3580000000000001,"15min":1.3640000000000001},"min":{"1min":1.046,"5min":1.0429999999999999,"15min":1.0429999999999999},"max":{"1min":1.536,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.3109999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.2789999999999999,"15min":1.2709999999999999},"min":{"1min":0.98299999999999998,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.9370000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.367,"5min":1.381,"15min":1.383},"min":{"1min":0.94899999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.9710000000000001,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.145}]},{"osd":5,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4350000000000001,"15min":1.4330000000000001},"min":{"1min":0.97699999999999998,"5min":0.93400000000000005,"15min":0.93400000000000005},"max":{"1min":1.7330000000000001,"5min":2.645,"15min":2.645},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"min":{"1min":1.0229999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"max":{"1min":2.1930000000000001,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.2749999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.5620000000000001,"15min":1.5720000000000001},"min":{"1min":1.2569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"max":{"1min":1.754,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.206},{"interface":"front","average":{"1min":1.444,"5min":1.5169999999999999,"15min":1.5289999999999999},"min":{"1min":1.0489999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":2.0030000000000001,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.242}]}]},{"osd":6,"up_from":14,"seq":60129542172,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":273193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.0369999999999999,"15min":1.042},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0429999999999999,"15min":1.038},"min":{"1min":0.74099999999999999,"5min":0.62,"15min":0.62},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":0.97799999999999998}]},{"osd":1,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0649999999999999,"15min":1.0669999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.3939999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.145,"15min":1.161},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.405,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.0469999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0680000000000001,"15min":1.0660000000000001},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.284},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.212,"15min":1.224},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.419,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":1.55}]},{"osd":3,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.2749999999999999,"15min":1.2889999999999999},"min":{"1min":0.82899999999999996,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.643,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.405},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.2589999999999999,"15min":1.266},"min":{"1min":0.89600000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.4870000000000001,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.256}]},{"osd":4,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.383,"15min":1.387},"min":{"1min":1.0860000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.734,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.175},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.379,"15min":1.389},"min":{"1min":0.871,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.5860000000000001,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.4730000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.431,"15min":1.4410000000000001},"min":{"1min":1.0029999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.7370000000000001,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.681},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.546,"15min":1.554},"min":{"1min":1.2410000000000001,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8220000000000001,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.6060000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.419,"15min":1.4159999999999999},"min":{"1min":1.0269999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.8460000000000001,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.802},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.54,"15min":1.55},"min":{"1min":1.018,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.802,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.6359999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542172,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":638976,"data_stored":280288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.1950000000000001,"15min":1.22},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.5,"5min":1.806,"15min":1.806},"last":1.125},{"interface":"front","average":{"1min":1.129,"5min":1.181,"15min":1.1890000000000001},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.6930000000000001,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.1679999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.671,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.153,"5min":1.1970000000000001,"15min":1.204},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.8180000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.875}]},{"osd":2,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.345,"15min":1.369},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.4950000000000001,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.361,"15min":1.387},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.556,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.036}]},{"osd":3,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.2450000000000001,"15min":1.2589999999999999},"min":{"1min":0.73699999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.4950000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.753},{"interface":"front","average":{"1min":1.298,"5min":1.4450000000000001,"15min":1.4690000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.925,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.2410000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.399,"15min":1.431},"min":{"1min":0.77600000000000002,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.8029999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.671},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.448,"15min":1.456},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":2.0209999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.337}]},{"osd":6,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.5,"15min":1.528},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.7390000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.204},{"interface":"front","average":{"1min":1.444,"5min":1.639,"15min":1.671},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.8049999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.2949999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.599,"15min":1.627},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":2.0059999999999998,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.081},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.6850000000000001,"15min":1.7170000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.0289999999999999,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.4139999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542173,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":273193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.97599999999999998,"15min":0.97199999999999998},"min":{"1min":0.69799999999999995,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.26,"5min":1.383,"15min":1.383},"last":1.7789999999999999},{"interface":"front","average":{"1min":1.012,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.69499999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.4950000000000001,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.879}]},{"osd":1,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.0509999999999999,"15min":1.044},"min":{"1min":0.78400000000000003,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.3759999999999999,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.27},{"interface":"front","average":{"1min":1.139,"5min":1.069,"15min":1.0569999999999999},"min":{"1min":0.86699999999999999,"5min":0.496,"15min":0.496},"max":{"1min":1.387,"5min":1.4159999999999999,"15min":1.4159999999999999},"last":1.6120000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.3109999999999999,"15min":1.331},"min":{"1min":0.88200000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.464,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.343,"5min":1.3140000000000001,"15min":1.3089999999999999},"min":{"1min":0.90400000000000003,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.6000000000000001,"5min":2.181,"15min":2.181},"last":1.6930000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.1220000000000001,"15min":1.1259999999999999},"min":{"1min":0.66800000000000004,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.423,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.212,"5min":1.272,"15min":1.282},"min":{"1min":0.84599999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.548,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.048}]},{"osd":5,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2729999999999999,"15min":1.272},"min":{"1min":0.69499999999999995,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.661,"5min":1.841,"15min":1.841},"last":1.411},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.28,"15min":1.2769999999999999},"min":{"1min":0.72199999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.589,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.1120000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.4059999999999999,"15min":1.421},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.6479999999999999,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.466,"5min":1.4079999999999999,"15min":1.3979999999999999},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.7789999999999999,"5min":1.877,"15min":1.877},"last":1.512}]},{"osd":7,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3660000000000001,"15min":1.3640000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.716,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.4810000000000001,"15min":1.4950000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.651,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.3720000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542173,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":655360,"data_stored":281874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.86399999999999999,"15min":0.87},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.93100000000000005,"15min":0.92700000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":5.8499999999999996,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.39400000000000002}]},{"osd":1,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.97499999999999998,"15min":0.98799999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.056,"15min":1.0800000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.7450000000000001,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.46000000000000002}]},{"osd":2,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":1.04,"15min":1.0640000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.679,"5min":1.823,"15min":1.823},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.127,"5min":1.1559999999999999,"15min":1.161},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.8010000000000002,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":0.42799999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.262,"15min":1.2549999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":5.9210000000000003,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.333,"5min":1.4410000000000001,"15min":1.4590000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":4.5620000000000003,"5min":4.819,"15min":4.819},"last":0.88400000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.6640000000000001,"5min":1.6279999999999999,"15min":1.4550000000000001},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":30.228000000000002,"5min":30.228000000000002,"15min":30.228000000000002},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.373,"5min":1.3680000000000001,"15min":1.367},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":5.8920000000000003,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":0.92600000000000005}]},{"osd":6,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.506,"15min":1.54},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.9319999999999999,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.012},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.407,"15min":1.399},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":4.7610000000000001,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.429,"15min":1.4330000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":5.9589999999999996,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.6399999999999999,"15min":1.6220000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":10.663,"5min":10.663,"15min":10.663},"last":0.98699999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542173,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27660,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344180,"statfs":{"total":96636764160,"available":96608440320,"internally_reserved":0,"allocated":651264,"data_stored":281852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.86899999999999999,"15min":0.88700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":1.4299999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.91300000000000003,"15min":0.95199999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.2729999999999999,"5min":2.226,"15min":2.226},"last":0.56000000000000005}]},{"osd":1,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.97399999999999998,"15min":0.997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.956,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":1.0840000000000001},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.90200000000000002,"15min":0.92400000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.665,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.76900000000000002}]},{"osd":3,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":1.008,"15min":1.034},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.2230000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.077,"15min":1.0920000000000001},"min":{"1min":0.53600000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.4690000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.302}]},{"osd":4,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.2170000000000001,"15min":1.2390000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.605,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.208,"15min":1.224},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.6910000000000001,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.0660000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.357,"15min":1.383},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.9339999999999999,"5min":1.97,"15min":1.97},"last":1.502},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.3280000000000001,"15min":1.357},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.79,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.337}]},{"osd":6,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.321,"15min":1.357},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.498,"5min":2.456,"15min":2.456},"last":1.143},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.47,"15min":1.5},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.0310000000000001,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":1.552}]},{"osd":7,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.3640000000000001,"15min":1.3879999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.7849999999999999,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.3919999999999999,"15min":1.425},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.7769999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.363}]}]},{"osd":1,"up_from":14,"seq":60129542172,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":273193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.85099999999999998,"15min":0.88100000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.278,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.81999999999999995,"15min":0.83099999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.2509999999999999,"5min":1.327,"15min":1.327},"last":1.046}]},{"osd":2,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.95099999999999996,"15min":0.97299999999999998},"min":{"1min":0.49199999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.405,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.234},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.97699999999999998,"15min":0.995},"min":{"1min":0.495,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.4079999999999999,"5min":1.625,"15min":1.625},"last":1.3340000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.1100000000000001,"15min":1.115},"min":{"1min":0.61099999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.6850000000000001,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.4450000000000001},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.002,"15min":1.022},"min":{"1min":0.47099999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.4359999999999999,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.83199999999999996}]},{"osd":4,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.2669999999999999,"15min":1.304},"min":{"1min":0.79400000000000004,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.425,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.3049999999999999,"15min":1.325},"min":{"1min":0.79400000000000004,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.728,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.5209999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.319,"15min":1.3440000000000001},"min":{"1min":0.72599999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.5820000000000001,"5min":3.593,"15min":3.593},"last":1.113},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.4099999999999999,"15min":1.454},"min":{"1min":0.76700000000000002,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.496,"5min":2.359,"15min":2.359},"last":1.3089999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.389,"15min":1.419},"min":{"1min":0.75900000000000001,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.575,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.3500000000000001,"15min":1.3740000000000001},"min":{"1min":0.76900000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.766,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.202}]},{"osd":7,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.4259999999999999,"15min":1.4550000000000001},"min":{"1min":0.81999999999999995,"5min":0.627,"15min":0.627},"max":{"1min":1.821,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.29},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.472,"15min":1.522},"min":{"1min":0.77900000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.5269999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.377}]}]},{"osd":0,"up_from":14,"seq":60129542172,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28216,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343624,"statfs":{"total":96636764160,"available":96607870976,"internally_reserved":0,"allocated":1220608,"data_stored":863561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.94199999999999995,"15min":0.96099999999999997},"min":{"1min":0.46200000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.26,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.62},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.98799999999999999,"15min":0.996},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.60399999999999998}]},{"osd":2,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.98899999999999999,"15min":1.0109999999999999},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.339,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.105,"15min":1.119},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.7849999999999999,"5min":1.901,"15min":1.901},"last":0.77100000000000002}]},{"osd":3,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.0089999999999999,"15min":1.0209999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.2969999999999999,"5min":1.972,"15min":1.972},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.1419999999999999,"15min":1.1719999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3260000000000001,"5min":2.298,"15min":2.298},"last":1.1579999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.345,"15min":1.355},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.4940000000000002,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.27,"15min":1.286},"min":{"1min":0.83099999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.83099999999999996}]},{"osd":5,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.373,"15min":1.391},"min":{"1min":0.89200000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.0070000000000001,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.4379999999999999,"15min":1.458},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.972,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.276}]},{"osd":6,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.514,"15min":1.556},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.8340000000000001,"5min":3.032,"15min":3.032},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.4910000000000001,"15min":1.514},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.9350000000000001,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.079}]},{"osd":7,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.5029999999999999,"15min":1.5269999999999999},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.3809999999999998,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.044},{"interface":"front","average":{"1min":1.298,"5min":1.4179999999999999,"15min":1.4379999999999999},"min":{"1min":0.85599999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.331,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":0.871}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:49:00.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:49:00.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.916+0000 7ffa90200640 1 Processor -- start 2024-03-23T03:49:00.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.916+0000 7ffa90200640 1 -- start start 2024-03-23T03:49:00.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.918+0000 7ffa90200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ffa880cbab0 0x7ffa880b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:00.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.918+0000 7ffa90200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa8806d390 0x7ffa8806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:00.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.918+0000 7ffa90200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa880b16a0 con 0x7ffa880adbc0 2024-03-23T03:49:00.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.918+0000 7ffa90200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa8806dd60 con 0x7ffa880cbab0 2024-03-23T03:49:00.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.918+0000 7ffa90200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa880cbfb0 con 0x7ffa8806d390 2024-03-23T03:49:00.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.918+0000 7ffa8d800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ffa880adbc0 0x7ffa880b41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52576/0 (socket says 172.21.15.102:52576) 2024-03-23T03:49:00.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.918+0000 7ffa8d800640 1 -- 172.21.15.102:0/3095109721 learned_addr learned my addr 172.21.15.102:0/3095109721 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:00.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.918+0000 7ffa8ce00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ffa880cbab0 0x7ffa880b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:00.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.918+0000 7ffa8ce00640 1 -- 172.21.15.102:0/3095109721 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa8806d390 msgr2=0x7ffa8806d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:00.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.918+0000 7ffa8ce00640 1 --2- 172.21.15.102:0/3095109721 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa8806d390 0x7ffa8806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.919+0000 7ffa8ce00640 1 -- 172.21.15.102:0/3095109721 >> v1:172.21.15.102:6789/0 conn(0x7ffa880adbc0 legacy=0x7ffa880b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:00.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.919+0000 7ffa8ce00640 1 -- 172.21.15.102:0/3095109721 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa880cc290 con 0x7ffa880cbab0 2024-03-23T03:49:00.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.919+0000 7ffa87e00640 1 -- 172.21.15.102:0/3095109721 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590570899 0 0) 0x7ffa880b16a0 con 0x7ffa880adbc0 2024-03-23T03:49:00.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.920+0000 7ffa8ce00640 1 --2- 172.21.15.102:0/3095109721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa880cbab0 0x7ffa880b20f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ffa740097a0 tx=0x7ffa7402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c54ae2a2d9a20555 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:00.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.920+0000 7ffa87e00640 1 -- 172.21.15.102:0/3095109721 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa7402e070 con 0x7ffa880cbab0 2024-03-23T03:49:00.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.920+0000 7ffa87e00640 1 -- 172.21.15.102:0/3095109721 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa740315a0 con 0x7ffa880cbab0 2024-03-23T03:49:00.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.920+0000 7ffa87e00640 1 -- 172.21.15.102:0/3095109721 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa74031930 con 0x7ffa880cbab0 2024-03-23T03:49:00.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.920+0000 7ffa90200640 1 -- 172.21.15.102:0/3095109721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa880cbab0 msgr2=0x7ffa880b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:00.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.921+0000 7ffa90200640 1 --2- 172.21.15.102:0/3095109721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa880cbab0 0x7ffa880b20f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7ffa740097a0 tx=0x7ffa7402b840 comp rx=0 tx=0).stop 2024-03-23T03:49:00.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.921+0000 7ffa90200640 1 -- 172.21.15.102:0/3095109721 shutdown_connections 2024-03-23T03:49:00.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.921+0000 7ffa90200640 1 --2- 172.21.15.102:0/3095109721 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa8806d390 0x7ffa8806d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.921+0000 7ffa90200640 1 --2- 172.21.15.102:0/3095109721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa880cbab0 0x7ffa880b20f0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.921+0000 7ffa90200640 1 -- 172.21.15.102:0/3095109721 >> 172.21.15.102:0/3095109721 conn(0x7ffa8806a3d0 msgr2=0x7ffa880b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:00.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.921+0000 7ffa90200640 1 -- 172.21.15.102:0/3095109721 shutdown_connections 2024-03-23T03:49:00.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.921+0000 7ffa90200640 1 -- 172.21.15.102:0/3095109721 wait complete. 2024-03-23T03:49:00.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.922+0000 7ffa90200640 1 Processor -- start 2024-03-23T03:49:00.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.923+0000 7ffa90200640 1 -- start start 2024-03-23T03:49:00.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.923+0000 7ffa90200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa8806d390 0x7ffa88140b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:00.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.924+0000 7ffa90200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa880adbc0 0x7ffa8813e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:00.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.924+0000 7ffa90200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa880cbab0 0x7ffa8813c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:00.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.924+0000 7ffa90200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa881f2670 con 0x7ffa880adbc0 2024-03-23T03:49:00.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.924+0000 7ffa90200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa881f2950 con 0x7ffa880cbab0 2024-03-23T03:49:00.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.924+0000 7ffa90200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa881f2c30 con 0x7ffa8806d390 2024-03-23T03:49:00.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.924+0000 7ffa8ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa880adbc0 0x7ffa8813e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:00.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.924+0000 7ffa8ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa880adbc0 0x7ffa8813e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41868/0 (socket says 172.21.15.102:41868) 2024-03-23T03:49:00.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.924+0000 7ffa8ce00640 1 -- 172.21.15.102:0/2191386899 learned_addr learned my addr 172.21.15.102:0/2191386899 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:00.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.924+0000 7ffa8d800640 1 --2- 172.21.15.102:0/2191386899 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa8806d390 0x7ffa88140b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:00.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.924+0000 7ffa8ce00640 1 -- 172.21.15.102:0/2191386899 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa8806d390 msgr2=0x7ffa88140b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:00.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.924+0000 7ffa8e200640 1 --2- 172.21.15.102:0/2191386899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa880cbab0 0x7ffa8813c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:00.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.924+0000 7ffa8ce00640 1 --2- 172.21.15.102:0/2191386899 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa8806d390 0x7ffa88140b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.924+0000 7ffa8ce00640 1 -- 172.21.15.102:0/2191386899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa880cbab0 msgr2=0x7ffa8813c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:00.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.924+0000 7ffa8ce00640 1 --2- 172.21.15.102:0/2191386899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa880cbab0 0x7ffa8813c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:00.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.924+0000 7ffa8ce00640 1 -- 172.21.15.102:0/2191386899 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa74009400 con 0x7ffa880adbc0 2024-03-23T03:49:00.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.924+0000 7ffa8e200640 1 --2- 172.21.15.102:0/2191386899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa880cbab0 0x7ffa8813c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:00.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.925+0000 7ffa8ce00640 1 --2- 172.21.15.102:0/2191386899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa880adbc0 0x7ffa8813e890 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ffa740097a0 tx=0x7ffa740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:00.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.925+0000 7ffa85600640 1 -- 172.21.15.102:0/2191386899 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa74032640 con 0x7ffa880adbc0 2024-03-23T03:49:00.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.925+0000 7ffa85600640 1 -- 172.21.15.102:0/2191386899 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa740327c0 con 0x7ffa880adbc0 2024-03-23T03:49:00.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.925+0000 7ffa90200640 1 -- 172.21.15.102:0/2191386899 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa881f2f10 con 0x7ffa880adbc0 2024-03-23T03:49:00.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.925+0000 7ffa85600640 1 -- 172.21.15.102:0/2191386899 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa74032b70 con 0x7ffa880adbc0 2024-03-23T03:49:00.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.925+0000 7ffa90200640 1 -- 172.21.15.102:0/2191386899 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa88141040 con 0x7ffa880adbc0 2024-03-23T03:49:00.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.926+0000 7ffa90200640 1 -- 172.21.15.102:0/2191386899 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa881f3430 con 0x7ffa880adbc0 2024-03-23T03:49:00.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.927+0000 7ffa85600640 1 -- 172.21.15.102:0/2191386899 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ffa74032cd0 con 0x7ffa880adbc0 2024-03-23T03:49:00.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.928+0000 7ffa85600640 1 --2- 172.21.15.102:0/2191386899 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa6806ac00 0x7ffa6806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:00.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.928+0000 7ffa85600640 1 -- 172.21.15.102:0/2191386899 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ffa740aacc0 con 0x7ffa880adbc0 2024-03-23T03:49:00.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.928+0000 7ffa8d800640 1 --2- 172.21.15.102:0/2191386899 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa6806ac00 0x7ffa6806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:00.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.929+0000 7ffa8d800640 1 --2- 172.21.15.102:0/2191386899 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa6806ac00 0x7ffa6806d0c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c000f60 tx=0x7ffa7c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:00.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:00.932+0000 7ffa85600640 1 -- 172.21.15.102:0/2191386899 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffa881f3430 con 0x7ffa880adbc0 2024-03-23T03:49:01.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.124+0000 7ffa90200640 1 -- 172.21.15.102:0/2191386899 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffa880b2630 con 0x7ffa6806ac00 2024-03-23T03:49:01.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.147+0000 7ffa85600640 1 -- 172.21.15.102:0/2191386899 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286793 (secure 0 0 0) 0x7ffa7c001c70 con 0x7ffa6806ac00 2024-03-23T03:49:01.152 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:01.153 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:49:01.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.154+0000 7ffa90200640 1 -- 172.21.15.102:0/2191386899 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa6806ac00 msgr2=0x7ffa6806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:01.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.154+0000 7ffa90200640 1 --2- 172.21.15.102:0/2191386899 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa6806ac00 0x7ffa6806d0c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c000f60 tx=0x7ffa7c009040 comp rx=0 tx=0).stop 2024-03-23T03:49:01.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.154+0000 7ffa90200640 1 -- 172.21.15.102:0/2191386899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa880adbc0 msgr2=0x7ffa8813e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:01.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.154+0000 7ffa90200640 1 --2- 172.21.15.102:0/2191386899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa880adbc0 0x7ffa8813e890 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7ffa740097a0 tx=0x7ffa740046c0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.154+0000 7ffa90200640 1 -- 172.21.15.102:0/2191386899 shutdown_connections 2024-03-23T03:49:01.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.154+0000 7ffa90200640 1 --2- 172.21.15.102:0/2191386899 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa6806ac00 0x7ffa6806d0c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.154+0000 7ffa90200640 1 --2- 172.21.15.102:0/2191386899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa880cbab0 0x7ffa8813c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.154+0000 7ffa90200640 1 --2- 172.21.15.102:0/2191386899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa880adbc0 0x7ffa8813e890 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.154+0000 7ffa90200640 1 --2- 172.21.15.102:0/2191386899 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa8806d390 0x7ffa88140b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.154+0000 7ffa90200640 1 -- 172.21.15.102:0/2191386899 >> 172.21.15.102:0/2191386899 conn(0x7ffa8806a3d0 msgr2=0x7ffa88142bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:01.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.154+0000 7ffa90200640 1 -- 172.21.15.102:0/2191386899 shutdown_connections 2024-03-23T03:49:01.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.154+0000 7ffa90200640 1 -- 172.21.15.102:0/2191386899 wait complete. 2024-03-23T03:49:01.173 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-03-23T03:48:59.756278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":57,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":222336,"kb_used_data":6140,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752384,"statfs":{"total":773094113280,"available":772866441216,"internally_reserved":0,"allocated":6287360,"data_stored":3397934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":221236794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.020087"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.273640+0000","last_change":"2024-03-23T03:48:56.269844+0000","last_active":"2024-03-23T03:48:56.273640+0000","last_peered":"2024-03-23T03:48:56.273640+0000","last_clean":"2024-03-23T03:48:56.273640+0000","last_became_active":"2024-03-23T03:48:56.269647+0000","last_became_peered":"2024-03-23T03:48:56.269647+0000","last_unstale":"2024-03-23T03:48:56.273640+0000","last_undegraded":"2024-03-23T03:48:56.273640+0000","last_fullsized":"2024-03-23T03:48:56.273640+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:15:23.482683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.273181+0000","last_change":"2024-03-23T03:48:56.271920+0000","last_active":"2024-03-23T03:48:56.273181+0000","last_peered":"2024-03-23T03:48:56.273181+0000","last_clean":"2024-03-23T03:48:56.273181+0000","last_became_active":"2024-03-23T03:48:56.271646+0000","last_became_peered":"2024-03-23T03:48:56.271646+0000","last_unstale":"2024-03-23T03:48:56.273181+0000","last_undegraded":"2024-03-23T03:48:56.273181+0000","last_fullsized":"2024-03-23T03:48:56.273181+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:00:24.758423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.267175+0000","last_change":"2024-03-23T03:48:56.264762+0000","last_active":"2024-03-23T03:48:56.267175+0000","last_peered":"2024-03-23T03:48:56.267175+0000","last_clean":"2024-03-23T03:48:56.267175+0000","last_became_active":"2024-03-23T03:48:56.263552+0000","last_became_peered":"2024-03-23T03:48:56.263552+0000","last_unstale":"2024-03-23T03:48:56.267175+0000","last_undegraded":"2024-03-23T03:48:56.267175+0000","last_fullsized":"2024-03-23T03:48:56.267175+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:40.071215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.272268+0000","last_change":"2024-03-23T03:48:56.270521+0000","last_active":"2024-03-23T03:48:56.272268+0000","last_peered":"2024-03-23T03:48:56.272268+0000","last_clean":"2024-03-23T03:48:56.272268+0000","last_became_active":"2024-03-23T03:48:56.270254+0000","last_became_peered":"2024-03-23T03:48:56.270254+0000","last_unstale":"2024-03-23T03:48:56.272268+0000","last_undegraded":"2024-03-23T03:48:56.272268+0000","last_fullsized":"2024-03-23T03:48:56.272268+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:08.127529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.259770+0000","last_change":"2024-03-23T03:48:56.252733+0000","last_active":"2024-03-23T03:48:56.259770+0000","last_peered":"2024-03-23T03:48:56.259770+0000","last_clean":"2024-03-23T03:48:56.259770+0000","last_became_active":"2024-03-23T03:48:56.252442+0000","last_became_peered":"2024-03-23T03:48:56.252442+0000","last_unstale":"2024-03-23T03:48:56.259770+0000","last_undegraded":"2024-03-23T03:48:56.259770+0000","last_fullsized":"2024-03-23T03:48:56.259770+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:44:31.659883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.281716+0000","last_change":"2024-03-23T03:48:56.272907+0000","last_active":"2024-03-23T03:48:56.281716+0000","last_peered":"2024-03-23T03:48:56.281716+0000","last_clean":"2024-03-23T03:48:56.281716+0000","last_became_active":"2024-03-23T03:48:56.272660+0000","last_became_peered":"2024-03-23T03:48:56.272660+0000","last_unstale":"2024-03-23T03:48:56.281716+0000","last_undegraded":"2024-03-23T03:48:56.281716+0000","last_fullsized":"2024-03-23T03:48:56.281716+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:37:28.870018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.279678+0000","last_change":"2024-03-23T03:48:56.277900+0000","last_active":"2024-03-23T03:48:56.279678+0000","last_peered":"2024-03-23T03:48:56.279678+0000","last_clean":"2024-03-23T03:48:56.279678+0000","last_became_active":"2024-03-23T03:48:56.276387+0000","last_became_peered":"2024-03-23T03:48:56.276387+0000","last_unstale":"2024-03-23T03:48:56.279678+0000","last_undegraded":"2024-03-23T03:48:56.279678+0000","last_fullsized":"2024-03-23T03:48:56.279678+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:46:56.378847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.264460+0000","last_change":"2024-03-23T03:48:56.254792+0000","last_active":"2024-03-23T03:48:56.264460+0000","last_peered":"2024-03-23T03:48:56.264460+0000","last_clean":"2024-03-23T03:48:56.264460+0000","last_became_active":"2024-03-23T03:48:56.254483+0000","last_became_peered":"2024-03-23T03:48:56.254483+0000","last_unstale":"2024-03-23T03:48:56.264460+0000","last_undegraded":"2024-03-23T03:48:56.264460+0000","last_fullsized":"2024-03-23T03:48:56.264460+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:35:49.226120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.263377+0000","last_change":"2024-03-23T03:48:56.250567+0000","last_active":"2024-03-23T03:48:56.263377+0000","last_peered":"2024-03-23T03:48:56.263377+0000","last_clean":"2024-03-23T03:48:56.263377+0000","last_became_active":"2024-03-23T03:48:56.250309+0000","last_became_peered":"2024-03-23T03:48:56.250309+0000","last_unstale":"2024-03-23T03:48:56.263377+0000","last_undegraded":"2024-03-23T03:48:56.263377+0000","last_fullsized":"2024-03-23T03:48:56.263377+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:24:18.834311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.271336+0000","last_change":"2024-03-23T03:48:56.265622+0000","last_active":"2024-03-23T03:48:56.271336+0000","last_peered":"2024-03-23T03:48:56.271336+0000","last_clean":"2024-03-23T03:48:56.271336+0000","last_became_active":"2024-03-23T03:48:56.265139+0000","last_became_peered":"2024-03-23T03:48:56.265139+0000","last_unstale":"2024-03-23T03:48:56.271336+0000","last_undegraded":"2024-03-23T03:48:56.271336+0000","last_fullsized":"2024-03-23T03:48:56.271336+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:24:15.405965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.278790+0000","last_change":"2024-03-23T03:48:56.275092+0000","last_active":"2024-03-23T03:48:56.278790+0000","last_peered":"2024-03-23T03:48:56.278790+0000","last_clean":"2024-03-23T03:48:56.278790+0000","last_became_active":"2024-03-23T03:48:56.274799+0000","last_became_peered":"2024-03-23T03:48:56.274799+0000","last_unstale":"2024-03-23T03:48:56.278790+0000","last_undegraded":"2024-03-23T03:48:56.278790+0000","last_fullsized":"2024-03-23T03:48:56.278790+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:04:50.091047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.267385+0000","last_change":"2024-03-23T03:48:56.263890+0000","last_active":"2024-03-23T03:48:56.267385+0000","last_peered":"2024-03-23T03:48:56.267385+0000","last_clean":"2024-03-23T03:48:56.267385+0000","last_became_active":"2024-03-23T03:48:56.263596+0000","last_became_peered":"2024-03-23T03:48:56.263596+0000","last_unstale":"2024-03-23T03:48:56.267385+0000","last_undegraded":"2024-03-23T03:48:56.267385+0000","last_fullsized":"2024-03-23T03:48:56.267385+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:13:51.660301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.272056+0000","last_change":"2024-03-23T03:48:56.266870+0000","last_active":"2024-03-23T03:48:56.272056+0000","last_peered":"2024-03-23T03:48:56.272056+0000","last_clean":"2024-03-23T03:48:56.272056+0000","last_became_active":"2024-03-23T03:48:56.266620+0000","last_became_peered":"2024-03-23T03:48:56.266620+0000","last_unstale":"2024-03-23T03:48:56.272056+0000","last_undegraded":"2024-03-23T03:48:56.272056+0000","last_fullsized":"2024-03-23T03:48:56.272056+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:01.054386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.267069+0000","last_change":"2024-03-23T03:48:56.262773+0000","last_active":"2024-03-23T03:48:56.267069+0000","last_peered":"2024-03-23T03:48:56.267069+0000","last_clean":"2024-03-23T03:48:56.267069+0000","last_became_active":"2024-03-23T03:48:56.262546+0000","last_became_peered":"2024-03-23T03:48:56.262546+0000","last_unstale":"2024-03-23T03:48:56.267069+0000","last_undegraded":"2024-03-23T03:48:56.267069+0000","last_fullsized":"2024-03-23T03:48:56.267069+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:29:25.260317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.271434+0000","last_change":"2024-03-23T03:48:56.263998+0000","last_active":"2024-03-23T03:48:56.271434+0000","last_peered":"2024-03-23T03:48:56.271434+0000","last_clean":"2024-03-23T03:48:56.271434+0000","last_became_active":"2024-03-23T03:48:56.263736+0000","last_became_peered":"2024-03-23T03:48:56.263736+0000","last_unstale":"2024-03-23T03:48:56.271434+0000","last_undegraded":"2024-03-23T03:48:56.271434+0000","last_fullsized":"2024-03-23T03:48:56.271434+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:38:58.872235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.273382+0000","last_change":"2024-03-23T03:48:56.269609+0000","last_active":"2024-03-23T03:48:56.273382+0000","last_peered":"2024-03-23T03:48:56.273382+0000","last_clean":"2024-03-23T03:48:56.273382+0000","last_became_active":"2024-03-23T03:48:56.269281+0000","last_became_peered":"2024-03-23T03:48:56.269281+0000","last_unstale":"2024-03-23T03:48:56.273382+0000","last_undegraded":"2024-03-23T03:48:56.273382+0000","last_fullsized":"2024-03-23T03:48:56.273382+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:59:01.084563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.257340+0000","last_change":"2024-03-23T03:48:56.253595+0000","last_active":"2024-03-23T03:48:56.257340+0000","last_peered":"2024-03-23T03:48:56.257340+0000","last_clean":"2024-03-23T03:48:56.257340+0000","last_became_active":"2024-03-23T03:48:56.253052+0000","last_became_peered":"2024-03-23T03:48:56.253052+0000","last_unstale":"2024-03-23T03:48:56.257340+0000","last_undegraded":"2024-03-23T03:48:56.257340+0000","last_fullsized":"2024-03-23T03:48:56.257340+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:34:33.139812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.273198+0000","last_change":"2024-03-23T03:48:56.271612+0000","last_active":"2024-03-23T03:48:56.273198+0000","last_peered":"2024-03-23T03:48:56.273198+0000","last_clean":"2024-03-23T03:48:56.273198+0000","last_became_active":"2024-03-23T03:48:56.271356+0000","last_became_peered":"2024-03-23T03:48:56.271356+0000","last_unstale":"2024-03-23T03:48:56.273198+0000","last_undegraded":"2024-03-23T03:48:56.273198+0000","last_fullsized":"2024-03-23T03:48:56.273198+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:17:01.745504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.256309+0000","last_change":"2024-03-23T03:48:56.251087+0000","last_active":"2024-03-23T03:48:56.256309+0000","last_peered":"2024-03-23T03:48:56.256309+0000","last_clean":"2024-03-23T03:48:56.256309+0000","last_became_active":"2024-03-23T03:48:56.250791+0000","last_became_peered":"2024-03-23T03:48:56.250791+0000","last_unstale":"2024-03-23T03:48:56.256309+0000","last_undegraded":"2024-03-23T03:48:56.256309+0000","last_fullsized":"2024-03-23T03:48:56.256309+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:03:13.109591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.265656+0000","last_change":"2024-03-23T03:48:56.263858+0000","last_active":"2024-03-23T03:48:56.265656+0000","last_peered":"2024-03-23T03:48:56.265656+0000","last_clean":"2024-03-23T03:48:56.265656+0000","last_became_active":"2024-03-23T03:48:56.263496+0000","last_became_peered":"2024-03-23T03:48:56.263496+0000","last_unstale":"2024-03-23T03:48:56.265656+0000","last_undegraded":"2024-03-23T03:48:56.265656+0000","last_fullsized":"2024-03-23T03:48:56.265656+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:13:20.826378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.263758+0000","last_change":"2024-03-23T03:48:56.262378+0000","last_active":"2024-03-23T03:48:56.263758+0000","last_peered":"2024-03-23T03:48:56.263758+0000","last_clean":"2024-03-23T03:48:56.263758+0000","last_became_active":"2024-03-23T03:48:56.262134+0000","last_became_peered":"2024-03-23T03:48:56.262134+0000","last_unstale":"2024-03-23T03:48:56.263758+0000","last_undegraded":"2024-03-23T03:48:56.263758+0000","last_fullsized":"2024-03-23T03:48:56.263758+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:33:57.629674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.265950+0000","last_change":"2024-03-23T03:48:56.261741+0000","last_active":"2024-03-23T03:48:56.265950+0000","last_peered":"2024-03-23T03:48:56.265950+0000","last_clean":"2024-03-23T03:48:56.265950+0000","last_became_active":"2024-03-23T03:48:56.260465+0000","last_became_peered":"2024-03-23T03:48:56.260465+0000","last_unstale":"2024-03-23T03:48:56.265950+0000","last_undegraded":"2024-03-23T03:48:56.265950+0000","last_fullsized":"2024-03-23T03:48:56.265950+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:48:59.163297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.279559+0000","last_change":"2024-03-23T03:48:56.275890+0000","last_active":"2024-03-23T03:48:56.279559+0000","last_peered":"2024-03-23T03:48:56.279559+0000","last_clean":"2024-03-23T03:48:56.279559+0000","last_became_active":"2024-03-23T03:48:56.274711+0000","last_became_peered":"2024-03-23T03:48:56.274711+0000","last_unstale":"2024-03-23T03:48:56.279559+0000","last_undegraded":"2024-03-23T03:48:56.279559+0000","last_fullsized":"2024-03-23T03:48:56.279559+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:36:16.185523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.266857+0000","last_change":"2024-03-23T03:48:56.262610+0000","last_active":"2024-03-23T03:48:56.266857+0000","last_peered":"2024-03-23T03:48:56.266857+0000","last_clean":"2024-03-23T03:48:56.266857+0000","last_became_active":"2024-03-23T03:48:56.262376+0000","last_became_peered":"2024-03-23T03:48:56.262376+0000","last_unstale":"2024-03-23T03:48:56.266857+0000","last_undegraded":"2024-03-23T03:48:56.266857+0000","last_fullsized":"2024-03-23T03:48:56.266857+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:36:13.491990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.264378+0000","last_change":"2024-03-23T03:48:56.260858+0000","last_active":"2024-03-23T03:48:56.264378+0000","last_peered":"2024-03-23T03:48:56.264378+0000","last_clean":"2024-03-23T03:48:56.264378+0000","last_became_active":"2024-03-23T03:48:56.260599+0000","last_became_peered":"2024-03-23T03:48:56.260599+0000","last_unstale":"2024-03-23T03:48:56.264378+0000","last_undegraded":"2024-03-23T03:48:56.264378+0000","last_fullsized":"2024-03-23T03:48:56.264378+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:01:58.868261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.280090+0000","last_change":"2024-03-23T03:48:56.277162+0000","last_active":"2024-03-23T03:48:56.280090+0000","last_peered":"2024-03-23T03:48:56.280090+0000","last_clean":"2024-03-23T03:48:56.280090+0000","last_became_active":"2024-03-23T03:48:56.276781+0000","last_became_peered":"2024-03-23T03:48:56.276781+0000","last_unstale":"2024-03-23T03:48:56.280090+0000","last_undegraded":"2024-03-23T03:48:56.280090+0000","last_fullsized":"2024-03-23T03:48:56.280090+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:12:22.293248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.278736+0000","last_change":"2024-03-23T03:48:56.272297+0000","last_active":"2024-03-23T03:48:56.278736+0000","last_peered":"2024-03-23T03:48:56.278736+0000","last_clean":"2024-03-23T03:48:56.278736+0000","last_became_active":"2024-03-23T03:48:56.270012+0000","last_became_peered":"2024-03-23T03:48:56.270012+0000","last_unstale":"2024-03-23T03:48:56.278736+0000","last_undegraded":"2024-03-23T03:48:56.278736+0000","last_fullsized":"2024-03-23T03:48:56.278736+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:10:53.300790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.278035+0000","last_change":"2024-03-23T03:48:56.277096+0000","last_active":"2024-03-23T03:48:56.278035+0000","last_peered":"2024-03-23T03:48:56.278035+0000","last_clean":"2024-03-23T03:48:56.278035+0000","last_became_active":"2024-03-23T03:48:56.276824+0000","last_became_peered":"2024-03-23T03:48:56.276824+0000","last_unstale":"2024-03-23T03:48:56.278035+0000","last_undegraded":"2024-03-23T03:48:56.278035+0000","last_fullsized":"2024-03-23T03:48:56.278035+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:25:49.472829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.265074+0000","last_change":"2024-03-23T03:48:56.259672+0000","last_active":"2024-03-23T03:48:56.265074+0000","last_peered":"2024-03-23T03:48:56.265074+0000","last_clean":"2024-03-23T03:48:56.265074+0000","last_became_active":"2024-03-23T03:48:56.259423+0000","last_became_peered":"2024-03-23T03:48:56.259423+0000","last_unstale":"2024-03-23T03:48:56.265074+0000","last_undegraded":"2024-03-23T03:48:56.265074+0000","last_fullsized":"2024-03-23T03:48:56.265074+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:42:11.077571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.275484+0000","last_change":"2024-03-23T03:48:56.274243+0000","last_active":"2024-03-23T03:48:56.275484+0000","last_peered":"2024-03-23T03:48:56.275484+0000","last_clean":"2024-03-23T03:48:56.275484+0000","last_became_active":"2024-03-23T03:48:56.273980+0000","last_became_peered":"2024-03-23T03:48:56.273980+0000","last_unstale":"2024-03-23T03:48:56.275484+0000","last_undegraded":"2024-03-23T03:48:56.275484+0000","last_fullsized":"2024-03-23T03:48:56.275484+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:12.465714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.258079+0000","last_change":"2024-03-23T03:48:56.256380+0000","last_active":"2024-03-23T03:48:56.258079+0000","last_peered":"2024-03-23T03:48:56.258079+0000","last_clean":"2024-03-23T03:48:56.258079+0000","last_became_active":"2024-03-23T03:48:56.256024+0000","last_became_peered":"2024-03-23T03:48:56.256024+0000","last_unstale":"2024-03-23T03:48:56.258079+0000","last_undegraded":"2024-03-23T03:48:56.258079+0000","last_fullsized":"2024-03-23T03:48:56.258079+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:36:45.908256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.242875+0000","last_change":"2024-03-23T03:48:56.241295+0000","last_active":"2024-03-23T03:48:56.242875+0000","last_peered":"2024-03-23T03:48:56.242875+0000","last_clean":"2024-03-23T03:48:56.242875+0000","last_became_active":"2024-03-23T03:48:56.240841+0000","last_became_peered":"2024-03-23T03:48:56.240841+0000","last_unstale":"2024-03-23T03:48:56.242875+0000","last_undegraded":"2024-03-23T03:48:56.242875+0000","last_fullsized":"2024-03-23T03:48:56.242875+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:01.054386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.272207+0000","last_change":"2024-03-23T03:48:56.265433+0000","last_active":"2024-03-23T03:48:56.272207+0000","last_peered":"2024-03-23T03:48:56.272207+0000","last_clean":"2024-03-23T03:48:56.272207+0000","last_became_active":"2024-03-23T03:48:56.264714+0000","last_became_peered":"2024-03-23T03:48:56.264714+0000","last_unstale":"2024-03-23T03:48:56.272207+0000","last_undegraded":"2024-03-23T03:48:56.272207+0000","last_fullsized":"2024-03-23T03:48:56.272207+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:59:06.205562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"54'4","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.264242+0000","last_change":"2024-03-23T03:48:55.776496+0000","last_active":"2024-03-23T03:48:56.264242+0000","last_peered":"2024-03-23T03:48:56.264242+0000","last_clean":"2024-03-23T03:48:56.264242+0000","last_became_active":"2024-03-23T03:48:55.775387+0000","last_became_peered":"2024-03-23T03:48:55.775387+0000","last_unstale":"2024-03-23T03:48:56.264242+0000","last_undegraded":"2024-03-23T03:48:56.264242+0000","last_fullsized":"2024-03-23T03:48:56.264242+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:49:58.504935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.255940+0000","last_change":"2024-03-23T03:48:56.253147+0000","last_active":"2024-03-23T03:48:56.255940+0000","last_peered":"2024-03-23T03:48:56.255940+0000","last_clean":"2024-03-23T03:48:56.255940+0000","last_became_active":"2024-03-23T03:48:56.252867+0000","last_became_peered":"2024-03-23T03:48:56.252867+0000","last_unstale":"2024-03-23T03:48:56.255940+0000","last_undegraded":"2024-03-23T03:48:56.255940+0000","last_fullsized":"2024-03-23T03:48:56.255940+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:43:17.526730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.267930+0000","last_change":"2024-03-23T03:48:55.786072+0000","last_active":"2024-03-23T03:48:56.267930+0000","last_peered":"2024-03-23T03:48:56.267930+0000","last_clean":"2024-03-23T03:48:56.267930+0000","last_became_active":"2024-03-23T03:48:55.785756+0000","last_became_peered":"2024-03-23T03:48:55.785756+0000","last_unstale":"2024-03-23T03:48:56.267930+0000","last_undegraded":"2024-03-23T03:48:56.267930+0000","last_fullsized":"2024-03-23T03:48:56.267930+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:42:06.491607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.267953+0000","last_change":"2024-03-23T03:48:56.265972+0000","last_active":"2024-03-23T03:48:56.267953+0000","last_peered":"2024-03-23T03:48:56.267953+0000","last_clean":"2024-03-23T03:48:56.267953+0000","last_became_active":"2024-03-23T03:48:56.265609+0000","last_became_peered":"2024-03-23T03:48:56.265609+0000","last_unstale":"2024-03-23T03:48:56.267953+0000","last_undegraded":"2024-03-23T03:48:56.267953+0000","last_fullsized":"2024-03-23T03:48:56.267953+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:23:10.621637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.273295+0000","last_change":"2024-03-23T03:48:55.779777+0000","last_active":"2024-03-23T03:48:56.273295+0000","last_peered":"2024-03-23T03:48:56.273295+0000","last_clean":"2024-03-23T03:48:56.273295+0000","last_became_active":"2024-03-23T03:48:55.779535+0000","last_became_peered":"2024-03-23T03:48:55.779535+0000","last_unstale":"2024-03-23T03:48:56.273295+0000","last_undegraded":"2024-03-23T03:48:56.273295+0000","last_fullsized":"2024-03-23T03:48:56.273295+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:59:05.193850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.280110+0000","last_change":"2024-03-23T03:48:56.275704+0000","last_active":"2024-03-23T03:48:56.280110+0000","last_peered":"2024-03-23T03:48:56.280110+0000","last_clean":"2024-03-23T03:48:56.280110+0000","last_became_active":"2024-03-23T03:48:56.275421+0000","last_became_peered":"2024-03-23T03:48:56.275421+0000","last_unstale":"2024-03-23T03:48:56.280110+0000","last_undegraded":"2024-03-23T03:48:56.280110+0000","last_fullsized":"2024-03-23T03:48:56.280110+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:14:25.292802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.261754+0000","last_change":"2024-03-23T03:48:55.775356+0000","last_active":"2024-03-23T03:48:56.261754+0000","last_peered":"2024-03-23T03:48:56.261754+0000","last_clean":"2024-03-23T03:48:56.261754+0000","last_became_active":"2024-03-23T03:48:55.775055+0000","last_became_peered":"2024-03-23T03:48:55.775055+0000","last_unstale":"2024-03-23T03:48:56.261754+0000","last_undegraded":"2024-03-23T03:48:56.261754+0000","last_fullsized":"2024-03-23T03:48:56.261754+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:15:55.585161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.255560+0000","last_change":"2024-03-23T03:48:56.251432+0000","last_active":"2024-03-23T03:48:56.255560+0000","last_peered":"2024-03-23T03:48:56.255560+0000","last_clean":"2024-03-23T03:48:56.255560+0000","last_became_active":"2024-03-23T03:48:56.251131+0000","last_became_peered":"2024-03-23T03:48:56.251131+0000","last_unstale":"2024-03-23T03:48:56.255560+0000","last_undegraded":"2024-03-23T03:48:56.255560+0000","last_fullsized":"2024-03-23T03:48:56.255560+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:48:41.795470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.277624+0000","last_change":"2024-03-23T03:48:55.774218+0000","last_active":"2024-03-23T03:48:56.277624+0000","last_peered":"2024-03-23T03:48:56.277624+0000","last_clean":"2024-03-23T03:48:56.277624+0000","last_became_active":"2024-03-23T03:48:55.772251+0000","last_became_peered":"2024-03-23T03:48:55.772251+0000","last_unstale":"2024-03-23T03:48:56.277624+0000","last_undegraded":"2024-03-23T03:48:56.277624+0000","last_fullsized":"2024-03-23T03:48:56.277624+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:00:23.746711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.276225+0000","last_change":"2024-03-23T03:48:56.274810+0000","last_active":"2024-03-23T03:48:56.276225+0000","last_peered":"2024-03-23T03:48:56.276225+0000","last_clean":"2024-03-23T03:48:56.276225+0000","last_became_active":"2024-03-23T03:48:56.274466+0000","last_became_peered":"2024-03-23T03:48:56.274466+0000","last_unstale":"2024-03-23T03:48:56.276225+0000","last_undegraded":"2024-03-23T03:48:56.276225+0000","last_fullsized":"2024-03-23T03:48:56.276225+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:15:56.596873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.248221+0000","last_change":"2024-03-23T03:48:55.777362+0000","last_active":"2024-03-23T03:48:56.248221+0000","last_peered":"2024-03-23T03:48:56.248221+0000","last_clean":"2024-03-23T03:48:56.248221+0000","last_became_active":"2024-03-23T03:48:55.777020+0000","last_became_peered":"2024-03-23T03:48:55.777020+0000","last_unstale":"2024-03-23T03:48:56.248221+0000","last_undegraded":"2024-03-23T03:48:56.248221+0000","last_fullsized":"2024-03-23T03:48:56.248221+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:25:48.461117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.280071+0000","last_change":"2024-03-23T03:48:56.277008+0000","last_active":"2024-03-23T03:48:56.280071+0000","last_peered":"2024-03-23T03:48:56.280071+0000","last_clean":"2024-03-23T03:48:56.280071+0000","last_became_active":"2024-03-23T03:48:56.276477+0000","last_became_peered":"2024-03-23T03:48:56.276477+0000","last_unstale":"2024-03-23T03:48:56.280071+0000","last_undegraded":"2024-03-23T03:48:56.280071+0000","last_fullsized":"2024-03-23T03:48:56.280071+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:32:35.308769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.247012+0000","last_change":"2024-03-23T03:48:55.782381+0000","last_active":"2024-03-23T03:48:56.247012+0000","last_peered":"2024-03-23T03:48:56.247012+0000","last_clean":"2024-03-23T03:48:56.247012+0000","last_became_active":"2024-03-23T03:48:55.782097+0000","last_became_peered":"2024-03-23T03:48:55.782097+0000","last_unstale":"2024-03-23T03:48:56.247012+0000","last_undegraded":"2024-03-23T03:48:56.247012+0000","last_fullsized":"2024-03-23T03:48:56.247012+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:04:49.079335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.272239+0000","last_change":"2024-03-23T03:48:56.269631+0000","last_active":"2024-03-23T03:48:56.272239+0000","last_peered":"2024-03-23T03:48:56.272239+0000","last_clean":"2024-03-23T03:48:56.272239+0000","last_became_active":"2024-03-23T03:48:56.268238+0000","last_became_peered":"2024-03-23T03:48:56.268238+0000","last_unstale":"2024-03-23T03:48:56.272239+0000","last_undegraded":"2024-03-23T03:48:56.272239+0000","last_fullsized":"2024-03-23T03:48:56.272239+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:40:07.408603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.241625+0000","last_change":"2024-03-23T03:48:55.776679+0000","last_active":"2024-03-23T03:48:56.241625+0000","last_peered":"2024-03-23T03:48:56.241625+0000","last_clean":"2024-03-23T03:48:56.241625+0000","last_became_active":"2024-03-23T03:48:55.776414+0000","last_became_peered":"2024-03-23T03:48:55.776414+0000","last_unstale":"2024-03-23T03:48:56.241625+0000","last_undegraded":"2024-03-23T03:48:56.241625+0000","last_fullsized":"2024-03-23T03:48:56.241625+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:11.454002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.262468+0000","last_change":"2024-03-23T03:48:56.256609+0000","last_active":"2024-03-23T03:48:56.262468+0000","last_peered":"2024-03-23T03:48:56.262468+0000","last_clean":"2024-03-23T03:48:56.262468+0000","last_became_active":"2024-03-23T03:48:56.256313+0000","last_became_peered":"2024-03-23T03:48:56.256313+0000","last_unstale":"2024-03-23T03:48:56.262468+0000","last_undegraded":"2024-03-23T03:48:56.262468+0000","last_fullsized":"2024-03-23T03:48:56.262468+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:10:36.220788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.257226+0000","last_change":"2024-03-23T03:48:55.786598+0000","last_active":"2024-03-23T03:48:56.257226+0000","last_peered":"2024-03-23T03:48:56.257226+0000","last_clean":"2024-03-23T03:48:56.257226+0000","last_became_active":"2024-03-23T03:48:55.786261+0000","last_became_peered":"2024-03-23T03:48:55.786261+0000","last_unstale":"2024-03-23T03:48:56.257226+0000","last_undegraded":"2024-03-23T03:48:56.257226+0000","last_fullsized":"2024-03-23T03:48:56.257226+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:59:57.078934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.253032+0000","last_change":"2024-03-23T03:48:56.245972+0000","last_active":"2024-03-23T03:48:56.253032+0000","last_peered":"2024-03-23T03:48:56.253032+0000","last_clean":"2024-03-23T03:48:56.253032+0000","last_became_active":"2024-03-23T03:48:56.245632+0000","last_became_peered":"2024-03-23T03:48:56.245632+0000","last_unstale":"2024-03-23T03:48:56.253032+0000","last_undegraded":"2024-03-23T03:48:56.253032+0000","last_fullsized":"2024-03-23T03:48:56.253032+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:37:28.870018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.238952+0000","last_change":"2024-03-23T03:48:55.768555+0000","last_active":"2024-03-23T03:48:56.238952+0000","last_peered":"2024-03-23T03:48:56.238952+0000","last_clean":"2024-03-23T03:48:56.238952+0000","last_became_active":"2024-03-23T03:48:55.768015+0000","last_became_peered":"2024-03-23T03:48:55.768015+0000","last_unstale":"2024-03-23T03:48:56.238952+0000","last_undegraded":"2024-03-23T03:48:56.238952+0000","last_fullsized":"2024-03-23T03:48:56.238952+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:33.847006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.257591+0000","last_change":"2024-03-23T03:48:56.254570+0000","last_active":"2024-03-23T03:48:56.257591+0000","last_peered":"2024-03-23T03:48:56.257591+0000","last_clean":"2024-03-23T03:48:56.257591+0000","last_became_active":"2024-03-23T03:48:56.254262+0000","last_became_peered":"2024-03-23T03:48:56.254262+0000","last_unstale":"2024-03-23T03:48:56.257591+0000","last_undegraded":"2024-03-23T03:48:56.257591+0000","last_fullsized":"2024-03-23T03:48:56.257591+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:21:25.502129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"54'3","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.279798+0000","last_change":"2024-03-23T03:48:55.784085+0000","last_active":"2024-03-23T03:48:56.279798+0000","last_peered":"2024-03-23T03:48:56.279798+0000","last_clean":"2024-03-23T03:48:56.279798+0000","last_became_active":"2024-03-23T03:48:55.783801+0000","last_became_peered":"2024-03-23T03:48:55.783801+0000","last_unstale":"2024-03-23T03:48:56.279798+0000","last_undegraded":"2024-03-23T03:48:56.279798+0000","last_fullsized":"2024-03-23T03:48:56.279798+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:56:14.033603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.259752+0000","last_change":"2024-03-23T03:48:56.252633+0000","last_active":"2024-03-23T03:48:56.259752+0000","last_peered":"2024-03-23T03:48:56.259752+0000","last_clean":"2024-03-23T03:48:56.259752+0000","last_became_active":"2024-03-23T03:48:56.252360+0000","last_became_peered":"2024-03-23T03:48:56.252360+0000","last_unstale":"2024-03-23T03:48:56.259752+0000","last_undegraded":"2024-03-23T03:48:56.259752+0000","last_fullsized":"2024-03-23T03:48:56.259752+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:46:32.495476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"54'6","reported_seq":16,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.283096+0000","last_change":"2024-03-23T03:48:55.780941+0000","last_active":"2024-03-23T03:48:56.283096+0000","last_peered":"2024-03-23T03:48:56.283096+0000","last_clean":"2024-03-23T03:48:56.283096+0000","last_became_active":"2024-03-23T03:48:55.780657+0000","last_became_peered":"2024-03-23T03:48:55.780657+0000","last_unstale":"2024-03-23T03:48:56.283096+0000","last_undegraded":"2024-03-23T03:48:56.283096+0000","last_fullsized":"2024-03-23T03:48:56.283096+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:03.110218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.273914+0000","last_change":"2024-03-23T03:48:56.271858+0000","last_active":"2024-03-23T03:48:56.273914+0000","last_peered":"2024-03-23T03:48:56.273914+0000","last_clean":"2024-03-23T03:48:56.273914+0000","last_became_active":"2024-03-23T03:48:56.271567+0000","last_became_peered":"2024-03-23T03:48:56.271567+0000","last_unstale":"2024-03-23T03:48:56.273914+0000","last_undegraded":"2024-03-23T03:48:56.273914+0000","last_fullsized":"2024-03-23T03:48:56.273914+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:10:53.300790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.272726+0000","last_change":"2024-03-23T03:48:55.779456+0000","last_active":"2024-03-23T03:48:56.272726+0000","last_peered":"2024-03-23T03:48:56.272726+0000","last_clean":"2024-03-23T03:48:56.272726+0000","last_became_active":"2024-03-23T03:48:55.779164+0000","last_became_peered":"2024-03-23T03:48:55.779164+0000","last_unstale":"2024-03-23T03:48:56.272726+0000","last_undegraded":"2024-03-23T03:48:56.272726+0000","last_fullsized":"2024-03-23T03:48:56.272726+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:17:00.733792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.276218+0000","last_change":"2024-03-23T03:48:56.274098+0000","last_active":"2024-03-23T03:48:56.276218+0000","last_peered":"2024-03-23T03:48:56.276218+0000","last_clean":"2024-03-23T03:48:56.276218+0000","last_became_active":"2024-03-23T03:48:56.273863+0000","last_became_peered":"2024-03-23T03:48:56.273863+0000","last_unstale":"2024-03-23T03:48:56.276218+0000","last_undegraded":"2024-03-23T03:48:56.276218+0000","last_fullsized":"2024-03-23T03:48:56.276218+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:14:05.067038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.247938+0000","last_change":"2024-03-23T03:48:55.784949+0000","last_active":"2024-03-23T03:48:56.247938+0000","last_peered":"2024-03-23T03:48:56.247938+0000","last_clean":"2024-03-23T03:48:56.247938+0000","last_became_active":"2024-03-23T03:48:55.784608+0000","last_became_peered":"2024-03-23T03:48:55.784608+0000","last_unstale":"2024-03-23T03:48:56.247938+0000","last_undegraded":"2024-03-23T03:48:56.247938+0000","last_fullsized":"2024-03-23T03:48:56.247938+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:20:44.840360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.267438+0000","last_change":"2024-03-23T03:48:56.265659+0000","last_active":"2024-03-23T03:48:56.267438+0000","last_peered":"2024-03-23T03:48:56.267438+0000","last_clean":"2024-03-23T03:48:56.267438+0000","last_became_active":"2024-03-23T03:48:56.265413+0000","last_became_peered":"2024-03-23T03:48:56.265413+0000","last_unstale":"2024-03-23T03:48:56.267438+0000","last_undegraded":"2024-03-23T03:48:56.267438+0000","last_fullsized":"2024-03-23T03:48:56.267438+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:06:57.815629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.249840+0000","last_change":"2024-03-23T03:48:55.782219+0000","last_active":"2024-03-23T03:48:56.249840+0000","last_peered":"2024-03-23T03:48:56.249840+0000","last_clean":"2024-03-23T03:48:56.249840+0000","last_became_active":"2024-03-23T03:48:55.781945+0000","last_became_peered":"2024-03-23T03:48:55.781945+0000","last_unstale":"2024-03-23T03:48:56.249840+0000","last_undegraded":"2024-03-23T03:48:56.249840+0000","last_fullsized":"2024-03-23T03:48:56.249840+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:05:17.151576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.257247+0000","last_change":"2024-03-23T03:48:56.251900+0000","last_active":"2024-03-23T03:48:56.257247+0000","last_peered":"2024-03-23T03:48:56.257247+0000","last_clean":"2024-03-23T03:48:56.257247+0000","last_became_active":"2024-03-23T03:48:56.251665+0000","last_became_peered":"2024-03-23T03:48:56.251665+0000","last_unstale":"2024-03-23T03:48:56.257247+0000","last_undegraded":"2024-03-23T03:48:56.257247+0000","last_fullsized":"2024-03-23T03:48:56.257247+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:04.121930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"54'8","reported_seq":18,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.281044+0000","last_change":"2024-03-23T03:48:55.776122+0000","last_active":"2024-03-23T03:48:56.281044+0000","last_peered":"2024-03-23T03:48:56.281044+0000","last_clean":"2024-03-23T03:48:56.281044+0000","last_became_active":"2024-03-23T03:48:55.775751+0000","last_became_peered":"2024-03-23T03:48:55.775751+0000","last_unstale":"2024-03-23T03:48:56.281044+0000","last_undegraded":"2024-03-23T03:48:56.281044+0000","last_fullsized":"2024-03-23T03:48:56.281044+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:36:12.480278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.261335+0000","last_change":"2024-03-23T03:48:56.246843+0000","last_active":"2024-03-23T03:48:56.261335+0000","last_peered":"2024-03-23T03:48:56.261335+0000","last_clean":"2024-03-23T03:48:56.261335+0000","last_became_active":"2024-03-23T03:48:56.246264+0000","last_became_peered":"2024-03-23T03:48:56.246264+0000","last_unstale":"2024-03-23T03:48:56.261335+0000","last_undegraded":"2024-03-23T03:48:56.261335+0000","last_fullsized":"2024-03-23T03:48:56.261335+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:45:54.545516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.265129+0000","last_change":"2024-03-23T03:48:55.781632+0000","last_active":"2024-03-23T03:48:56.265129+0000","last_peered":"2024-03-23T03:48:56.265129+0000","last_clean":"2024-03-23T03:48:56.265129+0000","last_became_active":"2024-03-23T03:48:55.781376+0000","last_became_peered":"2024-03-23T03:48:55.781376+0000","last_unstale":"2024-03-23T03:48:56.265129+0000","last_undegraded":"2024-03-23T03:48:56.265129+0000","last_fullsized":"2024-03-23T03:48:56.265129+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:40:06.396891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.273556+0000","last_change":"2024-03-23T03:48:56.269228+0000","last_active":"2024-03-23T03:48:56.273556+0000","last_peered":"2024-03-23T03:48:56.273556+0000","last_clean":"2024-03-23T03:48:56.273556+0000","last_became_active":"2024-03-23T03:48:56.268269+0000","last_became_peered":"2024-03-23T03:48:56.268269+0000","last_unstale":"2024-03-23T03:48:56.273556+0000","last_undegraded":"2024-03-23T03:48:56.273556+0000","last_fullsized":"2024-03-23T03:48:56.273556+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:22:29.565107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.276545+0000","last_change":"2024-03-23T03:48:55.763920+0000","last_active":"2024-03-23T03:48:56.276545+0000","last_peered":"2024-03-23T03:48:56.276545+0000","last_clean":"2024-03-23T03:48:56.276545+0000","last_became_active":"2024-03-23T03:48:55.763657+0000","last_became_peered":"2024-03-23T03:48:55.763657+0000","last_unstale":"2024-03-23T03:48:56.276545+0000","last_undegraded":"2024-03-23T03:48:56.276545+0000","last_fullsized":"2024-03-23T03:48:56.276545+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:50:56.772143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.256767+0000","last_change":"2024-03-23T03:48:56.255077+0000","last_active":"2024-03-23T03:48:56.256767+0000","last_peered":"2024-03-23T03:48:56.256767+0000","last_clean":"2024-03-23T03:48:56.256767+0000","last_became_active":"2024-03-23T03:48:56.254811+0000","last_became_peered":"2024-03-23T03:48:56.254811+0000","last_unstale":"2024-03-23T03:48:56.256767+0000","last_undegraded":"2024-03-23T03:48:56.256767+0000","last_fullsized":"2024-03-23T03:48:56.256767+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:43:22.788448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"54'1","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.273935+0000","last_change":"2024-03-23T03:48:55.780236+0000","last_active":"2024-03-23T03:48:56.273935+0000","last_peered":"2024-03-23T03:48:56.273935+0000","last_clean":"2024-03-23T03:48:56.273935+0000","last_became_active":"2024-03-23T03:48:55.779941+0000","last_became_peered":"2024-03-23T03:48:55.779941+0000","last_unstale":"2024-03-23T03:48:56.273935+0000","last_undegraded":"2024-03-23T03:48:56.273935+0000","last_fullsized":"2024-03-23T03:48:56.273935+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:53:13.367789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.272023+0000","last_change":"2024-03-23T03:48:56.264079+0000","last_active":"2024-03-23T03:48:56.272023+0000","last_peered":"2024-03-23T03:48:56.272023+0000","last_clean":"2024-03-23T03:48:56.272023+0000","last_became_active":"2024-03-23T03:48:56.263787+0000","last_became_peered":"2024-03-23T03:48:56.263787+0000","last_unstale":"2024-03-23T03:48:56.272023+0000","last_undegraded":"2024-03-23T03:48:56.272023+0000","last_fullsized":"2024-03-23T03:48:56.272023+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:33:31.920802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.250955+0000","last_change":"2024-03-23T03:48:55.779509+0000","last_active":"2024-03-23T03:48:56.250955+0000","last_peered":"2024-03-23T03:48:56.250955+0000","last_clean":"2024-03-23T03:48:56.250955+0000","last_became_active":"2024-03-23T03:48:55.779261+0000","last_became_peered":"2024-03-23T03:48:55.779261+0000","last_unstale":"2024-03-23T03:48:56.250955+0000","last_undegraded":"2024-03-23T03:48:56.250955+0000","last_fullsized":"2024-03-23T03:48:56.250955+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:44:44.587651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.245928+0000","last_change":"2024-03-23T03:48:56.243411+0000","last_active":"2024-03-23T03:48:56.245928+0000","last_peered":"2024-03-23T03:48:56.245928+0000","last_clean":"2024-03-23T03:48:56.245928+0000","last_became_active":"2024-03-23T03:48:56.243087+0000","last_became_peered":"2024-03-23T03:48:56.243087+0000","last_unstale":"2024-03-23T03:48:56.245928+0000","last_undegraded":"2024-03-23T03:48:56.245928+0000","last_fullsized":"2024-03-23T03:48:56.245928+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:59:01.084563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.278484+0000","last_change":"2024-03-23T03:48:55.780119+0000","last_active":"2024-03-23T03:48:56.278484+0000","last_peered":"2024-03-23T03:48:56.278484+0000","last_clean":"2024-03-23T03:48:56.278484+0000","last_became_active":"2024-03-23T03:48:55.779873+0000","last_became_peered":"2024-03-23T03:48:55.779873+0000","last_unstale":"2024-03-23T03:48:56.278484+0000","last_undegraded":"2024-03-23T03:48:56.278484+0000","last_fullsized":"2024-03-23T03:48:56.278484+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:27:54.806505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.262951+0000","last_change":"2024-03-23T03:48:56.261119+0000","last_active":"2024-03-23T03:48:56.262951+0000","last_peered":"2024-03-23T03:48:56.262951+0000","last_clean":"2024-03-23T03:48:56.262951+0000","last_became_active":"2024-03-23T03:48:56.260835+0000","last_became_peered":"2024-03-23T03:48:56.260835+0000","last_unstale":"2024-03-23T03:48:56.262951+0000","last_undegraded":"2024-03-23T03:48:56.262951+0000","last_fullsized":"2024-03-23T03:48:56.262951+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:36:00.205847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.231383+0000","last_change":"2024-03-23T03:48:55.781339+0000","last_active":"2024-03-23T03:48:56.231383+0000","last_peered":"2024-03-23T03:48:56.231383+0000","last_clean":"2024-03-23T03:48:56.231383+0000","last_became_active":"2024-03-23T03:48:55.781083+0000","last_became_peered":"2024-03-23T03:48:55.781083+0000","last_unstale":"2024-03-23T03:48:56.231383+0000","last_undegraded":"2024-03-23T03:48:56.231383+0000","last_fullsized":"2024-03-23T03:48:56.231383+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:36:44.896544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.257387+0000","last_change":"2024-03-23T03:48:56.253769+0000","last_active":"2024-03-23T03:48:56.257387+0000","last_peered":"2024-03-23T03:48:56.257387+0000","last_clean":"2024-03-23T03:48:56.257387+0000","last_became_active":"2024-03-23T03:48:56.249295+0000","last_became_peered":"2024-03-23T03:48:56.249295+0000","last_unstale":"2024-03-23T03:48:56.257387+0000","last_undegraded":"2024-03-23T03:48:56.257387+0000","last_fullsized":"2024-03-23T03:48:56.257387+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:22:09.094667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.250717+0000","last_change":"2024-03-23T03:48:55.776213+0000","last_active":"2024-03-23T03:48:56.250717+0000","last_peered":"2024-03-23T03:48:56.250717+0000","last_clean":"2024-03-23T03:48:56.250717+0000","last_became_active":"2024-03-23T03:48:55.775857+0000","last_became_peered":"2024-03-23T03:48:55.775857+0000","last_unstale":"2024-03-23T03:48:56.250717+0000","last_undegraded":"2024-03-23T03:48:56.250717+0000","last_fullsized":"2024-03-23T03:48:56.250717+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:07.115817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.271202+0000","last_change":"2024-03-23T03:48:56.268547+0000","last_active":"2024-03-23T03:48:56.271202+0000","last_peered":"2024-03-23T03:48:56.271202+0000","last_clean":"2024-03-23T03:48:56.271202+0000","last_became_active":"2024-03-23T03:48:56.268240+0000","last_became_peered":"2024-03-23T03:48:56.268240+0000","last_unstale":"2024-03-23T03:48:56.271202+0000","last_undegraded":"2024-03-23T03:48:56.271202+0000","last_fullsized":"2024-03-23T03:48:56.271202+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:31:34.716309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.260003+0000","last_change":"2024-03-23T03:48:55.784710+0000","last_active":"2024-03-23T03:48:56.260003+0000","last_peered":"2024-03-23T03:48:56.260003+0000","last_clean":"2024-03-23T03:48:56.260003+0000","last_became_active":"2024-03-23T03:48:55.784449+0000","last_became_peered":"2024-03-23T03:48:55.784449+0000","last_unstale":"2024-03-23T03:48:56.260003+0000","last_undegraded":"2024-03-23T03:48:56.260003+0000","last_fullsized":"2024-03-23T03:48:56.260003+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:15:32.774243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.242179+0000","last_change":"2024-03-23T03:48:56.241063+0000","last_active":"2024-03-23T03:48:56.242179+0000","last_peered":"2024-03-23T03:48:56.242179+0000","last_clean":"2024-03-23T03:48:56.242179+0000","last_became_active":"2024-03-23T03:48:56.240608+0000","last_became_peered":"2024-03-23T03:48:56.240608+0000","last_unstale":"2024-03-23T03:48:56.242179+0000","last_undegraded":"2024-03-23T03:48:56.242179+0000","last_fullsized":"2024-03-23T03:48:56.242179+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:22:29.565107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"54'2","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.262819+0000","last_change":"2024-03-23T03:48:55.783911+0000","last_active":"2024-03-23T03:48:56.262819+0000","last_peered":"2024-03-23T03:48:56.262819+0000","last_clean":"2024-03-23T03:48:56.262819+0000","last_became_active":"2024-03-23T03:48:55.783403+0000","last_became_peered":"2024-03-23T03:48:55.783403+0000","last_unstale":"2024-03-23T03:48:56.262819+0000","last_undegraded":"2024-03-23T03:48:56.262819+0000","last_fullsized":"2024-03-23T03:48:56.262819+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:33:20.082454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.282760+0000","last_change":"2024-03-23T03:48:56.281595+0000","last_active":"2024-03-23T03:48:56.282760+0000","last_peered":"2024-03-23T03:48:56.282760+0000","last_clean":"2024-03-23T03:48:56.282760+0000","last_became_active":"2024-03-23T03:48:56.280264+0000","last_became_peered":"2024-03-23T03:48:56.280264+0000","last_unstale":"2024-03-23T03:48:56.282760+0000","last_undegraded":"2024-03-23T03:48:56.282760+0000","last_fullsized":"2024-03-23T03:48:56.282760+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:43:17.526730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.256701+0000","last_change":"2024-03-23T03:48:55.785525+0000","last_active":"2024-03-23T03:48:56.256701+0000","last_peered":"2024-03-23T03:48:56.256701+0000","last_clean":"2024-03-23T03:48:56.256701+0000","last_became_active":"2024-03-23T03:48:55.785193+0000","last_became_peered":"2024-03-23T03:48:55.785193+0000","last_unstale":"2024-03-23T03:48:56.256701+0000","last_undegraded":"2024-03-23T03:48:56.256701+0000","last_fullsized":"2024-03-23T03:48:56.256701+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:37:02.476835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.259841+0000","last_change":"2024-03-23T03:48:56.252968+0000","last_active":"2024-03-23T03:48:56.259841+0000","last_peered":"2024-03-23T03:48:56.259841+0000","last_clean":"2024-03-23T03:48:56.259841+0000","last_became_active":"2024-03-23T03:48:56.248398+0000","last_became_peered":"2024-03-23T03:48:56.248398+0000","last_unstale":"2024-03-23T03:48:56.259841+0000","last_undegraded":"2024-03-23T03:48:56.259841+0000","last_fullsized":"2024-03-23T03:48:56.259841+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:21:25.502129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.244106+0000","last_change":"2024-03-23T03:48:55.769507+0000","last_active":"2024-03-23T03:48:56.244106+0000","last_peered":"2024-03-23T03:48:56.244106+0000","last_clean":"2024-03-23T03:48:56.244106+0000","last_became_active":"2024-03-23T03:48:55.769228+0000","last_became_peered":"2024-03-23T03:48:55.769228+0000","last_unstale":"2024-03-23T03:48:56.244106+0000","last_undegraded":"2024-03-23T03:48:56.244106+0000","last_fullsized":"2024-03-23T03:48:56.244106+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:31:15.531840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.265729+0000","last_change":"2024-03-23T03:48:56.261090+0000","last_active":"2024-03-23T03:48:56.265729+0000","last_peered":"2024-03-23T03:48:56.265729+0000","last_clean":"2024-03-23T03:48:56.265729+0000","last_became_active":"2024-03-23T03:48:56.260329+0000","last_became_peered":"2024-03-23T03:48:56.260329+0000","last_unstale":"2024-03-23T03:48:56.265729+0000","last_undegraded":"2024-03-23T03:48:56.265729+0000","last_fullsized":"2024-03-23T03:48:56.265729+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:03:46.310690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.232118+0000","last_change":"2024-03-23T03:48:55.782142+0000","last_active":"2024-03-23T03:48:56.232118+0000","last_peered":"2024-03-23T03:48:56.232118+0000","last_clean":"2024-03-23T03:48:56.232118+0000","last_became_active":"2024-03-23T03:48:55.781873+0000","last_became_peered":"2024-03-23T03:48:55.781873+0000","last_unstale":"2024-03-23T03:48:56.232118+0000","last_undegraded":"2024-03-23T03:48:56.232118+0000","last_fullsized":"2024-03-23T03:48:56.232118+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:34:32.128100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.255916+0000","last_change":"2024-03-23T03:48:56.252518+0000","last_active":"2024-03-23T03:48:56.255916+0000","last_peered":"2024-03-23T03:48:56.255916+0000","last_clean":"2024-03-23T03:48:56.255916+0000","last_became_active":"2024-03-23T03:48:56.252233+0000","last_became_peered":"2024-03-23T03:48:56.252233+0000","last_unstale":"2024-03-23T03:48:56.255916+0000","last_undegraded":"2024-03-23T03:48:56.255916+0000","last_fullsized":"2024-03-23T03:48:56.255916+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:24:15.405965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.231531+0000","last_change":"2024-03-23T03:48:55.782566+0000","last_active":"2024-03-23T03:48:56.231531+0000","last_peered":"2024-03-23T03:48:56.231531+0000","last_clean":"2024-03-23T03:48:56.231531+0000","last_became_active":"2024-03-23T03:48:55.782239+0000","last_became_peered":"2024-03-23T03:48:55.782239+0000","last_unstale":"2024-03-23T03:48:56.231531+0000","last_undegraded":"2024-03-23T03:48:56.231531+0000","last_fullsized":"2024-03-23T03:48:56.231531+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:29:24.248606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.267053+0000","last_change":"2024-03-23T03:48:56.262751+0000","last_active":"2024-03-23T03:48:56.267053+0000","last_peered":"2024-03-23T03:48:56.267053+0000","last_clean":"2024-03-23T03:48:56.267053+0000","last_became_active":"2024-03-23T03:48:56.262484+0000","last_became_peered":"2024-03-23T03:48:56.262484+0000","last_unstale":"2024-03-23T03:48:56.267053+0000","last_undegraded":"2024-03-23T03:48:56.267053+0000","last_fullsized":"2024-03-23T03:48:56.267053+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:19:28.397582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.238924+0000","last_change":"2024-03-23T03:48:55.772104+0000","last_active":"2024-03-23T03:48:56.238924+0000","last_peered":"2024-03-23T03:48:56.238924+0000","last_clean":"2024-03-23T03:48:56.238924+0000","last_became_active":"2024-03-23T03:48:55.771758+0000","last_became_peered":"2024-03-23T03:48:55.771758+0000","last_unstale":"2024-03-23T03:48:56.238924+0000","last_undegraded":"2024-03-23T03:48:56.238924+0000","last_fullsized":"2024-03-23T03:48:56.238924+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:13:19.814666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.281108+0000","last_change":"2024-03-23T03:48:56.275704+0000","last_active":"2024-03-23T03:48:56.281108+0000","last_peered":"2024-03-23T03:48:56.281108+0000","last_clean":"2024-03-23T03:48:56.281108+0000","last_became_active":"2024-03-23T03:48:56.275462+0000","last_became_peered":"2024-03-23T03:48:56.275462+0000","last_unstale":"2024-03-23T03:48:56.281108+0000","last_undegraded":"2024-03-23T03:48:56.281108+0000","last_fullsized":"2024-03-23T03:48:56.281108+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:47:59.122958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.236211+0000","last_change":"2024-03-23T03:48:55.780396+0000","last_active":"2024-03-23T03:48:56.236211+0000","last_peered":"2024-03-23T03:48:56.236211+0000","last_clean":"2024-03-23T03:48:56.236211+0000","last_became_active":"2024-03-23T03:48:55.780039+0000","last_became_peered":"2024-03-23T03:48:55.780039+0000","last_unstale":"2024-03-23T03:48:56.236211+0000","last_undegraded":"2024-03-23T03:48:56.236211+0000","last_fullsized":"2024-03-23T03:48:56.236211+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:11.454002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":9,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.263309+0000","last_change":"2024-03-23T03:48:56.259407+0000","last_active":"2024-03-23T03:48:56.263309+0000","last_peered":"2024-03-23T03:48:56.263309+0000","last_clean":"2024-03-23T03:48:56.263309+0000","last_became_active":"2024-03-23T03:48:56.258543+0000","last_became_peered":"2024-03-23T03:48:56.258543+0000","last_unstale":"2024-03-23T03:48:56.263309+0000","last_undegraded":"2024-03-23T03:48:56.263309+0000","last_fullsized":"2024-03-23T03:48:56.263309+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:55.734971+0000","last_clean_scrub_stamp":"2024-03-23T03:48:55.734971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:50:34.858717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-03-23T03:48:56.241911+0000","last_change":"2024-03-23T03:48:55.778988+0000","last_active":"2024-03-23T03:48:56.241911+0000","last_peered":"2024-03-23T03:48:56.241911+0000","last_clean":"2024-03-23T03:48:56.241911+0000","last_became_active":"2024-03-23T03:48:55.778701+0000","last_became_peered":"2024-03-23T03:48:55.778701+0000","last_unstale":"2024-03-23T03:48:56.241911+0000","last_undegraded":"2024-03-23T03:48:56.241911+0000","last_fullsized":"2024-03-23T03:48:56.241911+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:48:54.723259+0000","last_clean_scrub_stamp":"2024-03-23T03:48:54.723259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:08:42.513969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":190,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-03-23T03:48:55.775427+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:48:55.775427+0000","last_peered":"2024-03-23T03:48:55.775427+0000","last_clean":"2024-03-23T03:48:55.775427+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:48:55.775427+0000","last_undegraded":"2024-03-23T03:48:55.775427+0000","last_fullsized":"2024-03-23T03:48:55.775427+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542174,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28240,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343600,"statfs":{"total":96636764160,"available":96607846400,"internally_reserved":0,"allocated":1241088,"data_stored":870780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.079,"15min":1.069},"min":{"1min":0.66800000000000004,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.6279999999999999,"5min":1.635,"15min":1.635},"last":0.996},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0569999999999999,"15min":1.052},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.48,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.3280000000000001}]},{"osd":1,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.0589999999999999,"15min":1.0449999999999999},"min":{"1min":0.70099999999999996,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.655,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.2829999999999999,"15min":1.3180000000000001},"min":{"1min":0.72599999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.472,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":1.0649999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.147,"15min":1.159},"min":{"1min":0.79200000000000004,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.4870000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.296},{"interface":"front","average":{"1min":1.198,"5min":1.4059999999999999,"15min":1.4410000000000001},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.522,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.2589999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.4239999999999999,"15min":1.46},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.5900000000000001,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.026},{"interface":"front","average":{"1min":1.319,"5min":1.3580000000000001,"15min":1.3640000000000001},"min":{"1min":1.046,"5min":1.0429999999999999,"15min":1.0429999999999999},"max":{"1min":1.536,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.3109999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.2789999999999999,"15min":1.2709999999999999},"min":{"1min":0.98299999999999998,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.9370000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.367,"5min":1.381,"15min":1.383},"min":{"1min":0.94899999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.9710000000000001,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.145}]},{"osd":5,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4350000000000001,"15min":1.4330000000000001},"min":{"1min":0.97699999999999998,"5min":0.93400000000000005,"15min":0.93400000000000005},"max":{"1min":1.7330000000000001,"5min":2.645,"15min":2.645},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"min":{"1min":1.0229999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"max":{"1min":2.1930000000000001,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.2749999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.5620000000000001,"15min":1.5720000000000001},"min":{"1min":1.2569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"max":{"1min":1.754,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.206},{"interface":"front","average":{"1min":1.444,"5min":1.5169999999999999,"15min":1.5289999999999999},"min":{"1min":1.0489999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":2.0030000000000001,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.242}]}]},{"osd":6,"up_from":14,"seq":60129542172,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":273193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.0369999999999999,"15min":1.042},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0429999999999999,"15min":1.038},"min":{"1min":0.74099999999999999,"5min":0.62,"15min":0.62},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":0.97799999999999998}]},{"osd":1,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0649999999999999,"15min":1.0669999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.3939999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.145,"15min":1.161},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.405,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.0469999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0680000000000001,"15min":1.0660000000000001},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.284},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.212,"15min":1.224},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.419,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":1.55}]},{"osd":3,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.2749999999999999,"15min":1.2889999999999999},"min":{"1min":0.82899999999999996,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.643,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.405},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.2589999999999999,"15min":1.266},"min":{"1min":0.89600000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.4870000000000001,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.256}]},{"osd":4,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.383,"15min":1.387},"min":{"1min":1.0860000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.734,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.175},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.379,"15min":1.389},"min":{"1min":0.871,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.5860000000000001,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.4730000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.431,"15min":1.4410000000000001},"min":{"1min":1.0029999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.7370000000000001,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.681},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.546,"15min":1.554},"min":{"1min":1.2410000000000001,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8220000000000001,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.6060000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.419,"15min":1.4159999999999999},"min":{"1min":1.0269999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.8460000000000001,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.802},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.54,"15min":1.55},"min":{"1min":1.018,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.802,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.6359999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542172,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":638976,"data_stored":280288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.1950000000000001,"15min":1.22},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.5,"5min":1.806,"15min":1.806},"last":1.125},{"interface":"front","average":{"1min":1.129,"5min":1.181,"15min":1.1890000000000001},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.6930000000000001,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.1679999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.671,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.153,"5min":1.1970000000000001,"15min":1.204},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.8180000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.875}]},{"osd":2,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.345,"15min":1.369},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.4950000000000001,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.361,"15min":1.387},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.556,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.036}]},{"osd":3,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.2450000000000001,"15min":1.2589999999999999},"min":{"1min":0.73699999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.4950000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.753},{"interface":"front","average":{"1min":1.298,"5min":1.4450000000000001,"15min":1.4690000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.925,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.2410000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.399,"15min":1.431},"min":{"1min":0.77600000000000002,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.8029999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.671},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.448,"15min":1.456},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":2.0209999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.337}]},{"osd":6,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.5,"15min":1.528},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.7390000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.204},{"interface":"front","average":{"1min":1.444,"5min":1.639,"15min":1.671},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.8049999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.2949999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.599,"15min":1.627},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":2.0059999999999998,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.081},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.6850000000000001,"15min":1.7170000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.0289999999999999,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.4139999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542173,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":626688,"data_stored":273193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.97599999999999998,"15min":0.97199999999999998},"min":{"1min":0.69799999999999995,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.26,"5min":1.383,"15min":1.383},"last":1.7789999999999999},{"interface":"front","average":{"1min":1.012,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.69499999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.4950000000000001,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.879}]},{"osd":1,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.0509999999999999,"15min":1.044},"min":{"1min":0.78400000000000003,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.3759999999999999,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.27},{"interface":"front","average":{"1min":1.139,"5min":1.069,"15min":1.0569999999999999},"min":{"1min":0.86699999999999999,"5min":0.496,"15min":0.496},"max":{"1min":1.387,"5min":1.4159999999999999,"15min":1.4159999999999999},"last":1.6120000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.3109999999999999,"15min":1.331},"min":{"1min":0.88200000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.464,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.343,"5min":1.3140000000000001,"15min":1.3089999999999999},"min":{"1min":0.90400000000000003,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.6000000000000001,"5min":2.181,"15min":2.181},"last":1.6930000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.1220000000000001,"15min":1.1259999999999999},"min":{"1min":0.66800000000000004,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.423,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.212,"5min":1.272,"15min":1.282},"min":{"1min":0.84599999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.548,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.048}]},{"osd":5,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2729999999999999,"15min":1.272},"min":{"1min":0.69499999999999995,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.661,"5min":1.841,"15min":1.841},"last":1.411},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.28,"15min":1.2769999999999999},"min":{"1min":0.72199999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.589,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.1120000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.4059999999999999,"15min":1.421},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.6479999999999999,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.466,"5min":1.4079999999999999,"15min":1.3979999999999999},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.7789999999999999,"5min":1.877,"15min":1.877},"last":1.512}]},{"osd":7,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3660000000000001,"15min":1.3640000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.716,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.397,"5min":1.4810000000000001,"15min":1.4950000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.651,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.3720000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542173,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27668,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344172,"statfs":{"total":96636764160,"available":96608432128,"internally_reserved":0,"allocated":655360,"data_stored":281874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.86399999999999999,"15min":0.87},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.93100000000000005,"15min":0.92700000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":5.8499999999999996,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.39400000000000002}]},{"osd":1,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.97499999999999998,"15min":0.98799999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.056,"15min":1.0800000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.7450000000000001,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.46000000000000002}]},{"osd":2,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":1.04,"15min":1.0640000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.679,"5min":1.823,"15min":1.823},"last":0.47899999999999998},{"interface":"front","average":{"1min":1.127,"5min":1.1559999999999999,"15min":1.161},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.8010000000000002,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":0.42799999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.262,"15min":1.2549999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":5.9210000000000003,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.333,"5min":1.4410000000000001,"15min":1.4590000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":4.5620000000000003,"5min":4.819,"15min":4.819},"last":0.88400000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.6640000000000001,"5min":1.6279999999999999,"15min":1.4550000000000001},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":30.228000000000002,"5min":30.228000000000002,"15min":30.228000000000002},"last":0.74099999999999999},{"interface":"front","average":{"1min":1.373,"5min":1.3680000000000001,"15min":1.367},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":5.8920000000000003,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":0.92600000000000005}]},{"osd":6,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.506,"15min":1.54},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.9319999999999999,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.012},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.407,"15min":1.399},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":4.7610000000000001,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":0.76400000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.429,"15min":1.4330000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":5.9589999999999996,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":0.71199999999999997},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.6399999999999999,"15min":1.6220000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":10.663,"5min":10.663,"15min":10.663},"last":0.98699999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542173,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27660,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344180,"statfs":{"total":96636764160,"available":96608440320,"internally_reserved":0,"allocated":651264,"data_stored":281852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.86899999999999999,"15min":0.88700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":1.4299999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.91300000000000003,"15min":0.95199999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.2729999999999999,"5min":2.226,"15min":2.226},"last":0.56000000000000005}]},{"osd":1,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.97399999999999998,"15min":0.997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.956,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":1.0840000000000001},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.90200000000000002,"15min":0.92400000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.665,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.76900000000000002}]},{"osd":3,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":1.008,"15min":1.034},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.2230000000000001},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.077,"15min":1.0920000000000001},"min":{"1min":0.53600000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.4690000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.302}]},{"osd":4,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.2170000000000001,"15min":1.2390000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.605,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.208,"15min":1.224},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.6910000000000001,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.0660000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.357,"15min":1.383},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.9339999999999999,"5min":1.97,"15min":1.97},"last":1.502},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.3280000000000001,"15min":1.357},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.79,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.337}]},{"osd":6,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.321,"15min":1.357},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.498,"5min":2.456,"15min":2.456},"last":1.143},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.47,"15min":1.5},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.0310000000000001,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":1.552}]},{"osd":7,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.3640000000000001,"15min":1.3879999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.7849999999999999,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.3919999999999999,"15min":1.425},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.7769999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.363}]}]},{"osd":1,"up_from":14,"seq":60129542172,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":626688,"data_stored":273193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.85099999999999998,"15min":0.88100000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.278,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.81999999999999995,"15min":0.83099999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.2509999999999999,"5min":1.327,"15min":1.327},"last":1.046}]},{"osd":2,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.95099999999999996,"15min":0.97299999999999998},"min":{"1min":0.49199999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.405,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.234},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.97699999999999998,"15min":0.995},"min":{"1min":0.495,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.4079999999999999,"5min":1.625,"15min":1.625},"last":1.3340000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.1100000000000001,"15min":1.115},"min":{"1min":0.61099999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.6850000000000001,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.4450000000000001},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.002,"15min":1.022},"min":{"1min":0.47099999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.4359999999999999,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.83199999999999996}]},{"osd":4,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.2669999999999999,"15min":1.304},"min":{"1min":0.79400000000000004,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.425,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.1379999999999999},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.3049999999999999,"15min":1.325},"min":{"1min":0.79400000000000004,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.728,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.5209999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.319,"15min":1.3440000000000001},"min":{"1min":0.72599999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.5820000000000001,"5min":3.593,"15min":3.593},"last":1.113},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.4099999999999999,"15min":1.454},"min":{"1min":0.76700000000000002,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.496,"5min":2.359,"15min":2.359},"last":1.3089999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.389,"15min":1.419},"min":{"1min":0.75900000000000001,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.575,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.3500000000000001,"15min":1.3740000000000001},"min":{"1min":0.76900000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.766,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.202}]},{"osd":7,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.4259999999999999,"15min":1.4550000000000001},"min":{"1min":0.81999999999999995,"5min":0.627,"15min":0.627},"max":{"1min":1.821,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.29},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.472,"15min":1.522},"min":{"1min":0.77900000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.5269999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.377}]}]},{"osd":0,"up_from":14,"seq":60129542172,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28216,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343624,"statfs":{"total":96636764160,"available":96607870976,"internally_reserved":0,"allocated":1220608,"data_stored":863561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.94199999999999995,"15min":0.96099999999999997},"min":{"1min":0.46200000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.26,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.62},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.98799999999999999,"15min":0.996},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":0.60399999999999998}]},{"osd":2,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.98899999999999999,"15min":1.0109999999999999},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.339,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.105,"15min":1.119},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.7849999999999999,"5min":1.901,"15min":1.901},"last":0.77100000000000002}]},{"osd":3,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.0089999999999999,"15min":1.0209999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.2969999999999999,"5min":1.972,"15min":1.972},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.1419999999999999,"15min":1.1719999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3260000000000001,"5min":2.298,"15min":2.298},"last":1.1579999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.345,"15min":1.355},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.4940000000000002,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.27,"15min":1.286},"min":{"1min":0.83099999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.83099999999999996}]},{"osd":5,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.373,"15min":1.391},"min":{"1min":0.89200000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.0070000000000001,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.4379999999999999,"15min":1.458},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.972,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.276}]},{"osd":6,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.514,"15min":1.556},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.8340000000000001,"5min":3.032,"15min":3.032},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.4910000000000001,"15min":1.514},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.9350000000000001,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.079}]},{"osd":7,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.5029999999999999,"15min":1.5269999999999999},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.3809999999999998,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.044},{"interface":"front","average":{"1min":1.298,"5min":1.4179999999999999,"15min":1.4379999999999999},"min":{"1min":0.85599999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.331,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":0.871}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:49:01.178 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T03:49:01.178 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-23T03:49:01.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.430+0000 7f071d200640 1 Processor -- start 2024-03-23T03:49:01.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.430+0000 7f071d200640 1 -- start start 2024-03-23T03:49:01.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.432+0000 7f071d200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f0718160ca0 0x7f0718161ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:01.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.432+0000 7f071d200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f071806d110 0x7f07180b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:01.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.432+0000 7f071d200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07180cda90 con 0x7f071806c080 2024-03-23T03:49:01.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.432+0000 7f071d200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07180ce280 con 0x7f0718160ca0 2024-03-23T03:49:01.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.432+0000 7f071d200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0718162220 con 0x7f071806d110 2024-03-23T03:49:01.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.432+0000 7f0717400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f0718160ca0 0x7f0718161ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:01.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.432+0000 7f0717e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f071806d110 0x7f07180b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:01.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.433+0000 7f0716a00640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7f071806c080 0x7f071806ca00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:42084/0 (socket says 172.21.15.102:42084) 2024-03-23T03:49:01.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.433+0000 7f0716a00640 1 -- 172.21.15.102:0/2880007721 learned_addr learned my addr 172.21.15.102:0/2880007721 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:01.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.433+0000 7f0717400640 1 -- 172.21.15.102:0/2880007721 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f071806d110 msgr2=0x7f07180b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:01.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.433+0000 7f0717400640 1 --2- 172.21.15.102:0/2880007721 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f071806d110 0x7f07180b1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.433+0000 7f0717400640 1 -- 172.21.15.102:0/2880007721 >> v1:172.21.15.173:6789/0 conn(0x7f071806c080 legacy=0x7f071806ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:01.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.433+0000 7f0717400640 1 -- 172.21.15.102:0/2880007721 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07180cc9f0 con 0x7f0718160ca0 2024-03-23T03:49:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.435+0000 7f0717400640 1 --2- 172.21.15.102:0/2880007721 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0718160ca0 0x7f0718161ce0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f07080097a0 tx=0x7f070802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecf8af13417d19b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.435+0000 7f0716000640 1 -- 172.21.15.102:0/2880007721 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0708031780 con 0x7f0718160ca0 2024-03-23T03:49:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.435+0000 7f0716000640 1 -- 172.21.15.102:0/2880007721 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07080318e0 con 0x7f0718160ca0 2024-03-23T03:49:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.435+0000 7f0716000640 1 -- 172.21.15.102:0/2880007721 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0708031c70 con 0x7f0718160ca0 2024-03-23T03:49:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.436+0000 7f071d200640 1 -- 172.21.15.102:0/2880007721 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0718160ca0 msgr2=0x7f0718161ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:01.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.436+0000 7f071d200640 1 --2- 172.21.15.102:0/2880007721 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0718160ca0 0x7f0718161ce0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f07080097a0 tx=0x7f070802b820 comp rx=0 tx=0).stop 2024-03-23T03:49:01.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.436+0000 7f071d200640 1 -- 172.21.15.102:0/2880007721 shutdown_connections 2024-03-23T03:49:01.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.436+0000 7f071d200640 1 --2- 172.21.15.102:0/2880007721 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f071806d110 0x7f07180b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.436+0000 7f071d200640 1 --2- 172.21.15.102:0/2880007721 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0718160ca0 0x7f0718161ce0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.436+0000 7f071d200640 1 -- 172.21.15.102:0/2880007721 >> 172.21.15.102:0/2880007721 conn(0x7f0718065df0 msgr2=0x7f0718069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:01.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.436+0000 7f071d200640 1 -- 172.21.15.102:0/2880007721 shutdown_connections 2024-03-23T03:49:01.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.436+0000 7f071d200640 1 -- 172.21.15.102:0/2880007721 wait complete. 2024-03-23T03:49:01.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.438+0000 7f071d200640 1 Processor -- start 2024-03-23T03:49:01.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.438+0000 7f071d200640 1 -- start start 2024-03-23T03:49:01.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.439+0000 7f071d200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f071806c080 0x7f071815c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:01.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.439+0000 7f071d200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f071806d110 0x7f071815cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:01.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.439+0000 7f071d200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0718160ca0 0x7f07181592c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:01.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.439+0000 7f071d200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071815ac00 con 0x7f0718160ca0 2024-03-23T03:49:01.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.439+0000 7f071d200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0718159ff0 con 0x7f071806d110 2024-03-23T03:49:01.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.439+0000 7f071d200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f071815a2d0 con 0x7f071806c080 2024-03-23T03:49:01.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.439+0000 7f0717e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0718160ca0 0x7f07181592c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:01.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.439+0000 7f0717e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0718160ca0 0x7f07181592c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41884/0 (socket says 172.21.15.102:41884) 2024-03-23T03:49:01.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.439+0000 7f0717e00640 1 -- 172.21.15.102:0/1371661841 learned_addr learned my addr 172.21.15.102:0/1371661841 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:01.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.439+0000 7f0717400640 1 --2- 172.21.15.102:0/1371661841 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f071806c080 0x7f071815c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:01.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.439+0000 7f0716a00640 1 --2- 172.21.15.102:0/1371661841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f071806d110 0x7f071815cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:01.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.440+0000 7f0717e00640 1 -- 172.21.15.102:0/1371661841 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f071806c080 msgr2=0x7f071815c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:01.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.440+0000 7f0717e00640 1 --2- 172.21.15.102:0/1371661841 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f071806c080 0x7f071815c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.440+0000 7f0717e00640 1 -- 172.21.15.102:0/1371661841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f071806d110 msgr2=0x7f071815cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:01.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.440+0000 7f0717e00640 1 --2- 172.21.15.102:0/1371661841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f071806d110 0x7f071815cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.440+0000 7f0717e00640 1 -- 172.21.15.102:0/1371661841 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0708009400 con 0x7f0718160ca0 2024-03-23T03:49:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.440+0000 7f0717e00640 1 --2- 172.21.15.102:0/1371661841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0718160ca0 0x7f07181592c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f071000efc0 tx=0x7f071000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.440+0000 7f070f400640 1 -- 172.21.15.102:0/1371661841 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0710014480 con 0x7f0718160ca0 2024-03-23T03:49:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.440+0000 7f070f400640 1 -- 172.21.15.102:0/1371661841 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071000f040 con 0x7f0718160ca0 2024-03-23T03:49:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.440+0000 7f071d200640 1 -- 172.21.15.102:0/1371661841 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f071815a5b0 con 0x7f0718160ca0 2024-03-23T03:49:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.440+0000 7f070f400640 1 -- 172.21.15.102:0/1371661841 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0710014820 con 0x7f0718160ca0 2024-03-23T03:49:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.440+0000 7f071d200640 1 -- 172.21.15.102:0/1371661841 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07181f5df0 con 0x7f0718160ca0 2024-03-23T03:49:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.442+0000 7f071d200640 1 -- 172.21.15.102:0/1371661841 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07180cbed0 con 0x7f0718160ca0 2024-03-23T03:49:01.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.443+0000 7f070f400640 1 -- 172.21.15.102:0/1371661841 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0710014980 con 0x7f0718160ca0 2024-03-23T03:49:01.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.444+0000 7f070f400640 1 --2- 172.21.15.102:0/1371661841 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06f806ac00 0x7f06f806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:01.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.444+0000 7f070f400640 1 -- 172.21.15.102:0/1371661841 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f071008a380 con 0x7f0718160ca0 2024-03-23T03:49:01.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.444+0000 7f0717400640 1 --2- 172.21.15.102:0/1371661841 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06f806ac00 0x7f06f806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:01.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.445+0000 7f0717400640 1 --2- 172.21.15.102:0/1371661841 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06f806ac00 0x7f06f806d0c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0708036000 tx=0x7f070803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:01.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.449+0000 7f070f400640 1 -- 172.21.15.102:0/1371661841 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0710055420 con 0x7f0718160ca0 2024-03-23T03:49:01.687 INFO:tasks.ceph.mds.d.smithi173.stderr:2024-03-23T03:49:01.687+0000 7f04e9800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T03:49:01.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.717+0000 7f071d200640 1 -- 172.21.15.102:0/1371661841 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f071804e030 con 0x7f0718160ca0 2024-03-23T03:49:01.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.723+0000 7f070f400640 1 -- 172.21.15.102:0/1371661841 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0710058a40 con 0x7f0718160ca0 2024-03-23T03:49:01.724 INFO:teuthology.orchestra.run.smithi102.stderr:updated caps for client.0 2024-03-23T03:49:01.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.726+0000 7f071d200640 1 -- 172.21.15.102:0/1371661841 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06f806ac00 msgr2=0x7f06f806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:01.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.726+0000 7f071d200640 1 --2- 172.21.15.102:0/1371661841 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06f806ac00 0x7f06f806d0c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0708036000 tx=0x7f070803d040 comp rx=0 tx=0).stop 2024-03-23T03:49:01.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.726+0000 7f071d200640 1 -- 172.21.15.102:0/1371661841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0718160ca0 msgr2=0x7f07181592c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:01.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.726+0000 7f071d200640 1 --2- 172.21.15.102:0/1371661841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0718160ca0 0x7f07181592c0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f071000efc0 tx=0x7f071000c490 comp rx=0 tx=0).stop 2024-03-23T03:49:01.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.726+0000 7f071d200640 1 -- 172.21.15.102:0/1371661841 shutdown_connections 2024-03-23T03:49:01.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.726+0000 7f071d200640 1 --2- 172.21.15.102:0/1371661841 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06f806ac00 0x7f06f806d0c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.726+0000 7f071d200640 1 --2- 172.21.15.102:0/1371661841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0718160ca0 0x7f07181592c0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.726+0000 7f071d200640 1 --2- 172.21.15.102:0/1371661841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f071806d110 0x7f071815cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.726+0000 7f071d200640 1 --2- 172.21.15.102:0/1371661841 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f071806c080 0x7f071815c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.727+0000 7f071d200640 1 -- 172.21.15.102:0/1371661841 >> 172.21.15.102:0/1371661841 conn(0x7f0718065df0 msgr2=0x7f07180aef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:01.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.727+0000 7f071d200640 1 -- 172.21.15.102:0/1371661841 shutdown_connections 2024-03-23T03:49:01.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.727+0000 7f071d200640 1 -- 172.21.15.102:0/1371661841 wait complete. 2024-03-23T03:49:01.746 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:49:01.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.951+0000 7ff966600640 1 Processor -- start 2024-03-23T03:49:01.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.951+0000 7ff966600640 1 -- start start 2024-03-23T03:49:01.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.953+0000 7ff966600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ff960160800 0x7ff9600b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:01.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.953+0000 7ff966600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9600b2030 con 0x7ff96013faa0 2024-03-23T03:49:01.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.953+0000 7ff966600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff960161640 con 0x7ff9600b1be0 2024-03-23T03:49:01.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.953+0000 7ff966600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9600b38a0 con 0x7ff960160800 2024-03-23T03:49:01.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.953+0000 7ff95f400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff96013faa0 0x7ff960160f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52584/0 (socket says 172.21.15.102:52584) 2024-03-23T03:49:01.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.953+0000 7ff95f400640 1 -- 172.21.15.102:0/1130132556 learned_addr learned my addr 172.21.15.102:0/1130132556 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:01.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.954+0000 7ff956a00640 1 --2- 172.21.15.102:0/1130132556 >> v2:172.21.15.173:3300/0 conn(0x7ff960160800 0x7ff9600b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:01.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.954+0000 7ff95ea00640 1 -- 172.21.15.102:0/1130132556 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102055659 0 0) 0x7ff950001110 con 0x7ff96013faa0 2024-03-23T03:49:01.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.954+0000 7ff95ea00640 1 -- 172.21.15.102:0/1130132556 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff944003520 con 0x7ff96013faa0 2024-03-23T03:49:01.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.954+0000 7ff95ea00640 1 -- 172.21.15.102:0/1130132556 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3860329598 0 0) 0x7ff944003520 con 0x7ff96013faa0 2024-03-23T03:49:01.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.954+0000 7ff95ea00640 1 -- 172.21.15.102:0/1130132556 >> v2:172.21.15.173:3300/0 conn(0x7ff960160800 msgr2=0x7ff9600b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:01.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.954+0000 7ff95ea00640 1 --2- 172.21.15.102:0/1130132556 >> v2:172.21.15.173:3300/0 conn(0x7ff960160800 0x7ff9600b16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.954+0000 7ff95ea00640 1 -- 172.21.15.102:0/1130132556 >> v1:172.21.15.173:6789/0 conn(0x7ff9600b1be0 legacy=0x7ff96006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:01.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.954+0000 7ff95ea00640 1 -- 172.21.15.102:0/1130132556 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9600b3b80 con 0x7ff96013faa0 2024-03-23T03:49:01.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.954+0000 7ff95ea00640 1 -- 172.21.15.102:0/1130132556 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7ff950001800 con 0x7ff96013faa0 2024-03-23T03:49:01.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.954+0000 7ff95ea00640 1 -- 172.21.15.102:0/1130132556 >> v1:172.21.15.102:6789/0 conn(0x7ff96013faa0 legacy=0x7ff960160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:01.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.954+0000 7ff95ea00640 1 --2- 172.21.15.102:0/1130132556 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff944003970 0x7ff944003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:01.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.954+0000 7ff95ea00640 1 --2- 172.21.15.102:0/1130132556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9440047c0 0x7ff944004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:01.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.955+0000 7ff95ea00640 1 --2- 172.21.15.102:0/1130132556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff944005180 0x7ff94400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:01.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.955+0000 7ff95ea00640 1 -- 172.21.15.102:0/1130132556 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9600b38a0 con 0x7ff9440047c0 2024-03-23T03:49:01.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.955+0000 7ff95ea00640 1 -- 172.21.15.102:0/1130132556 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94400c020 con 0x7ff944005180 2024-03-23T03:49:01.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.955+0000 7ff95ea00640 1 -- 172.21.15.102:0/1130132556 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94400c1e0 con 0x7ff944003970 2024-03-23T03:49:01.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.955+0000 7ff956a00640 1 --2- 172.21.15.102:0/1130132556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9440047c0 0x7ff944004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:01.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.955+0000 7ff956a00640 1 -- 172.21.15.102:0/1130132556 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff944003970 msgr2=0x7ff944003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:01.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.955+0000 7ff956a00640 1 --2- 172.21.15.102:0/1130132556 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff944003970 0x7ff944003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.955+0000 7ff956a00640 1 -- 172.21.15.102:0/1130132556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff944005180 msgr2=0x7ff94400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:01.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.955+0000 7ff95fe00640 1 --2- 172.21.15.102:0/1130132556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff944005180 0x7ff94400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:01.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.955+0000 7ff956a00640 1 --2- 172.21.15.102:0/1130132556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff944005180 0x7ff94400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.955+0000 7ff956a00640 1 -- 172.21.15.102:0/1130132556 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff944003520 con 0x7ff9440047c0 2024-03-23T03:49:01.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.955+0000 7ff95fe00640 1 --2- 172.21.15.102:0/1130132556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff944005180 0x7ff94400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:01.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.955+0000 7ff95fe00640 1 -- 172.21.15.102:0/1130132556 reap_dead start 2024-03-23T03:49:01.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.956+0000 7ff956a00640 1 --2- 172.21.15.102:0/1130132556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9440047c0 0x7ff944004c40 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff948009fc0 tx=0x7ff9440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6969f4ffb104589 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:01.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.957+0000 7ff95ea00640 1 -- 172.21.15.102:0/1130132556 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94802cc20 con 0x7ff9440047c0 2024-03-23T03:49:01.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.957+0000 7ff95ea00640 1 -- 172.21.15.102:0/1130132556 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94800f040 con 0x7ff9440047c0 2024-03-23T03:49:01.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.957+0000 7ff95ea00640 1 -- 172.21.15.102:0/1130132556 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff948034440 con 0x7ff9440047c0 2024-03-23T03:49:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.957+0000 7ff966600640 1 -- 172.21.15.102:0/1130132556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9440047c0 msgr2=0x7ff944004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.957+0000 7ff966600640 1 --2- 172.21.15.102:0/1130132556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9440047c0 0x7ff944004c40 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7ff948009fc0 tx=0x7ff9440057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.957+0000 7ff966600640 1 -- 172.21.15.102:0/1130132556 shutdown_connections 2024-03-23T03:49:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.957+0000 7ff966600640 1 --2- 172.21.15.102:0/1130132556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9440047c0 0x7ff944004c40 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.957+0000 7ff966600640 1 -- 172.21.15.102:0/1130132556 >> 172.21.15.102:0/1130132556 conn(0x7ff96006a3d0 msgr2=0x7ff9600b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.957+0000 7ff966600640 1 -- 172.21.15.102:0/1130132556 shutdown_connections 2024-03-23T03:49:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.957+0000 7ff966600640 1 -- 172.21.15.102:0/1130132556 wait complete. 2024-03-23T03:49:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.959+0000 7ff966600640 1 Processor -- start 2024-03-23T03:49:01.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.959+0000 7ff966600640 1 -- start start 2024-03-23T03:49:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.960+0000 7ff966600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9440047c0 0x7ff9601429d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.960+0000 7ff966600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff96013aed0 0x7ff9600cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.960+0000 7ff966600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9600ca890 0x7ff9600cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.960+0000 7ff966600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9600cb280 con 0x7ff9440047c0 2024-03-23T03:49:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.960+0000 7ff966600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9601400f0 con 0x7ff9600ca890 2024-03-23T03:49:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.960+0000 7ff966600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9601403d0 con 0x7ff96013aed0 2024-03-23T03:49:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.960+0000 7ff95f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9440047c0 0x7ff9601429d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.960+0000 7ff95f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9440047c0 0x7ff9601429d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41892/0 (socket says 172.21.15.102:41892) 2024-03-23T03:49:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.960+0000 7ff95f400640 1 -- 172.21.15.102:0/3410774816 learned_addr learned my addr 172.21.15.102:0/3410774816 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.961+0000 7ff95f400640 1 -- 172.21.15.102:0/3410774816 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff96013aed0 msgr2=0x7ff9600cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.961+0000 7ff95f400640 1 --2- 172.21.15.102:0/3410774816 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff96013aed0 0x7ff9600cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.961+0000 7ff95f400640 1 -- 172.21.15.102:0/3410774816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9600ca890 msgr2=0x7ff9600cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:01.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.961+0000 7ff95fe00640 1 --2- 172.21.15.102:0/3410774816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9600ca890 0x7ff9600cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:01.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.961+0000 7ff95f400640 1 --2- 172.21.15.102:0/3410774816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9600ca890 0x7ff9600cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:01.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.961+0000 7ff95f400640 1 -- 172.21.15.102:0/3410774816 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9601406b0 con 0x7ff9440047c0 2024-03-23T03:49:01.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.961+0000 7ff95fe00640 1 --2- 172.21.15.102:0/3410774816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9600ca890 0x7ff9600cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:01.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.961+0000 7ff95f400640 1 --2- 172.21.15.102:0/3410774816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9440047c0 0x7ff9601429d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ff950001eb0 tx=0x7ff95000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:01.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.961+0000 7ff957e00640 1 -- 172.21.15.102:0/3410774816 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9500074e0 con 0x7ff9440047c0 2024-03-23T03:49:01.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.961+0000 7ff957e00640 1 -- 172.21.15.102:0/3410774816 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff950007640 con 0x7ff9440047c0 2024-03-23T03:49:01.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.961+0000 7ff957e00640 1 -- 172.21.15.102:0/3410774816 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff950004a00 con 0x7ff9440047c0 2024-03-23T03:49:01.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.961+0000 7ff966600640 1 -- 172.21.15.102:0/3410774816 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff960140990 con 0x7ff9440047c0 2024-03-23T03:49:01.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.961+0000 7ff966600640 1 -- 172.21.15.102:0/3410774816 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff960142f10 con 0x7ff9440047c0 2024-03-23T03:49:01.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.963+0000 7ff966600640 1 -- 172.21.15.102:0/3410774816 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff96006ac60 con 0x7ff9440047c0 2024-03-23T03:49:01.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.963+0000 7ff957e00640 1 -- 172.21.15.102:0/3410774816 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff950004b60 con 0x7ff9440047c0 2024-03-23T03:49:01.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.964+0000 7ff957e00640 1 --2- 172.21.15.102:0/3410774816 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff93c06ac30 0x7ff93c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:01.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.964+0000 7ff957e00640 1 -- 172.21.15.102:0/3410774816 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff950088820 con 0x7ff9440047c0 2024-03-23T03:49:01.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.965+0000 7ff956a00640 1 --2- 172.21.15.102:0/3410774816 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff93c06ac30 0x7ff93c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:01.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.965+0000 7ff956a00640 1 --2- 172.21.15.102:0/3410774816 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff93c06ac30 0x7ff93c06d0f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff948036000 tx=0x7ff94802e070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:01.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:01.969+0000 7ff957e00640 1 -- 172.21.15.102:0/3410774816 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff950053820 con 0x7ff9440047c0 2024-03-23T03:49:02.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.199+0000 7ff966600640 1 -- 172.21.15.102:0/3410774816 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff960140eb0 con 0x7ff9440047c0 2024-03-23T03:49:02.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.200+0000 7ff957e00640 1 -- 172.21.15.102:0/3410774816 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4740 (secure 0 0 0) 0x7ff960140eb0 con 0x7ff9440047c0 2024-03-23T03:49:02.200 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:02.200 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5661,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/1848167718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":1848167718},{"type":"v1","addr":"172.21.15.102:6835","nonce":1848167718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/16304019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":16304019},{"type":"v1","addr":"172.21.15.102:6837","nonce":16304019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5673,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6834/2030662139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2030662139},{"type":"v1","addr":"172.21.15.173:6834","nonce":2030662139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:48:56.221998+0000","modified":"2024-03-23T03:48:59.250212+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5676":{"gid":5676,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6835/3269327930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":3269327930},{"type":"v1","addr":"172.21.15.173:6835","nonce":3269327930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-23T03:49:02.201 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 24 2024-03-23T03:49:02.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.202+0000 7ff966600640 1 -- 172.21.15.102:0/3410774816 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff93c06ac30 msgr2=0x7ff93c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:02.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.202+0000 7ff966600640 1 --2- 172.21.15.102:0/3410774816 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff93c06ac30 0x7ff93c06d0f0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7ff948036000 tx=0x7ff94802e070 comp rx=0 tx=0).stop 2024-03-23T03:49:02.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.202+0000 7ff966600640 1 -- 172.21.15.102:0/3410774816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9440047c0 msgr2=0x7ff9601429d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:02.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.202+0000 7ff966600640 1 --2- 172.21.15.102:0/3410774816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9440047c0 0x7ff9601429d0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7ff950001eb0 tx=0x7ff95000ccb0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.202+0000 7ff966600640 1 -- 172.21.15.102:0/3410774816 shutdown_connections 2024-03-23T03:49:02.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.202+0000 7ff966600640 1 --2- 172.21.15.102:0/3410774816 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff93c06ac30 0x7ff93c06d0f0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.202+0000 7ff966600640 1 --2- 172.21.15.102:0/3410774816 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9600ca890 0x7ff9600cad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.202+0000 7ff966600640 1 --2- 172.21.15.102:0/3410774816 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff96013aed0 0x7ff9600cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.202+0000 7ff966600640 1 --2- 172.21.15.102:0/3410774816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9440047c0 0x7ff9601429d0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.202+0000 7ff966600640 1 -- 172.21.15.102:0/3410774816 >> 172.21.15.102:0/3410774816 conn(0x7ff96006a3d0 msgr2=0x7ff9600af910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:02.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.203+0000 7ff966600640 1 -- 172.21.15.102:0/3410774816 shutdown_connections 2024-03-23T03:49:02.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.203+0000 7ff966600640 1 -- 172.21.15.102:0/3410774816 wait complete. 2024-03-23T03:49:02.625 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T03:48:56.221998+0000', 'modified': '2024-03-23T03:48:59.250212+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5676}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5676': {'gid': 5676, 'name': 'd', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.173:6835/3269327930', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6833', 'nonce': 3269327930}, {'type': 'v1', 'addr': '172.21.15.173:6835', 'nonce': 3269327930}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T03:49:02.625 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T03:49:02.625 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-23T03:49:02.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.834+0000 7f4c58a00640 1 Processor -- start 2024-03-23T03:49:02.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.834+0000 7f4c58a00640 1 -- start start 2024-03-23T03:49:02.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.836+0000 7f4c58a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c540cc330 0x7f4c540b18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.836+0000 7f4c58a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c5415e010 con 0x7f4c540b1e10 2024-03-23T03:49:02.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.836+0000 7f4c58a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c540caec0 con 0x7f4c54159690 2024-03-23T03:49:02.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.836+0000 7f4c58a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c540cc7f0 con 0x7f4c540cc330 2024-03-23T03:49:02.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.836+0000 7f4c52800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4c540b1e10 0x7f4c540cd890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:52598/0 (socket says 172.21.15.102:52598) 2024-03-23T03:49:02.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.836+0000 7f4c52800640 1 -- 172.21.15.102:0/2451244202 learned_addr learned my addr 172.21.15.102:0/2451244202 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:02.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.836+0000 7f4c51400640 1 --2- 172.21.15.102:0/2451244202 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c540cc330 0x7f4c540b18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.836+0000 7f4c50a00640 1 -- 172.21.15.102:0/2451244202 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553997835 0 0) 0x7f4c5415e010 con 0x7f4c540b1e10 2024-03-23T03:49:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.836+0000 7f4c50a00640 1 -- 172.21.15.102:0/2451244202 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c38005730 con 0x7f4c540b1e10 2024-03-23T03:49:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.837+0000 7f4c50a00640 1 -- 172.21.15.102:0/2451244202 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2541598398 0 0) 0x7f4c38005730 con 0x7f4c540b1e10 2024-03-23T03:49:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.837+0000 7f4c50a00640 1 -- 172.21.15.102:0/2451244202 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c540cc330 msgr2=0x7f4c540b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.837+0000 7f4c50a00640 1 --2- 172.21.15.102:0/2451244202 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c540cc330 0x7f4c540b18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.837+0000 7f4c50a00640 1 -- 172.21.15.102:0/2451244202 >> v1:172.21.15.173:6789/0 conn(0x7f4c54159690 legacy=0x7f4c540ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.837+0000 7f4c50a00640 1 -- 172.21.15.102:0/2451244202 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c540ccad0 con 0x7f4c540b1e10 2024-03-23T03:49:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.837+0000 7f4c50a00640 1 -- 172.21.15.102:0/2451244202 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f4c4c0019f0 con 0x7f4c540b1e10 2024-03-23T03:49:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.837+0000 7f4c50a00640 1 -- 172.21.15.102:0/2451244202 >> v1:172.21.15.102:6789/0 conn(0x7f4c540b1e10 legacy=0x7f4c540cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.837+0000 7f4c50a00640 1 --2- 172.21.15.102:0/2451244202 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c540cc330 0x7f4c38005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.837+0000 7f4c50a00640 1 --2- 172.21.15.102:0/2451244202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c38006130 0x7f4c380069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.837+0000 7f4c50a00640 1 --2- 172.21.15.102:0/2451244202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c38006f10 0x7f4c38007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.837+0000 7f4c50a00640 1 -- 172.21.15.102:0/2451244202 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c540cc7f0 con 0x7f4c38006130 2024-03-23T03:49:02.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.837+0000 7f4c50a00640 1 -- 172.21.15.102:0/2451244202 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3800dc70 con 0x7f4c38006f10 2024-03-23T03:49:02.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.837+0000 7f4c50a00640 1 -- 172.21.15.102:0/2451244202 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3800de30 con 0x7f4c540cc330 2024-03-23T03:49:02.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.837+0000 7f4c51400640 1 --2- 172.21.15.102:0/2451244202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c38006130 0x7f4c380069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:02.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.838+0000 7f4c51400640 1 -- 172.21.15.102:0/2451244202 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c540cc330 msgr2=0x7f4c38005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:02.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.838+0000 7f4c51400640 1 --2- 172.21.15.102:0/2451244202 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c540cc330 0x7f4c38005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.838+0000 7f4c51400640 1 -- 172.21.15.102:0/2451244202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c38006f10 msgr2=0x7f4c38007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:02.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.838+0000 7f4c51400640 1 --2- 172.21.15.102:0/2451244202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c38006f10 0x7f4c38007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.838+0000 7f4c51400640 1 -- 172.21.15.102:0/2451244202 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c38005730 con 0x7f4c38006130 2024-03-23T03:49:02.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.839+0000 7f4c51400640 1 --2- 172.21.15.102:0/2451244202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c38006130 0x7f4c380069d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f4c40009ac0 tx=0x7f4c38007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51f74f7cbf2b4bcf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:02.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.839+0000 7f4c50a00640 1 -- 172.21.15.102:0/2451244202 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4002c710 con 0x7f4c38006130 2024-03-23T03:49:02.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.839+0000 7f4c50a00640 1 -- 172.21.15.102:0/2451244202 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4002c870 con 0x7f4c38006130 2024-03-23T03:49:02.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.839+0000 7f4c50a00640 1 -- 172.21.15.102:0/2451244202 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4002cc00 con 0x7f4c38006130 2024-03-23T03:49:02.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.840+0000 7f4c58a00640 1 -- 172.21.15.102:0/2451244202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c38006130 msgr2=0x7f4c380069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:02.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.840+0000 7f4c58a00640 1 --2- 172.21.15.102:0/2451244202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c38006130 0x7f4c380069d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f4c40009ac0 tx=0x7f4c38007a00 comp rx=0 tx=0).stop 2024-03-23T03:49:02.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.840+0000 7f4c52800640 1 -- 172.21.15.102:0/2451244202 reap_dead start 2024-03-23T03:49:02.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.840+0000 7f4c58a00640 1 -- 172.21.15.102:0/2451244202 shutdown_connections 2024-03-23T03:49:02.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.840+0000 7f4c58a00640 1 -- 172.21.15.102:0/2451244202 >> 172.21.15.102:0/2451244202 conn(0x7f4c5406a0b0 msgr2=0x7f4c5406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:02.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.840+0000 7f4c58a00640 1 -- 172.21.15.102:0/2451244202 shutdown_connections 2024-03-23T03:49:02.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.840+0000 7f4c58a00640 1 -- 172.21.15.102:0/2451244202 wait complete. 2024-03-23T03:49:02.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.842+0000 7f4c58a00640 1 Processor -- start 2024-03-23T03:49:02.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.842+0000 7f4c58a00640 1 -- start start 2024-03-23T03:49:02.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.843+0000 7f4c58a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c38006130 0x7f4c541f60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.843+0000 7f4c58a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c540b01f0 0x7f4c541f2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.843+0000 7f4c58a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c541f26a0 0x7f4c54147e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.843+0000 7f4c58a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c541f3130 con 0x7f4c38006130 2024-03-23T03:49:02.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.843+0000 7f4c58a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c541f6630 con 0x7f4c541f26a0 2024-03-23T03:49:02.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.843+0000 7f4c58a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c541f2b70 con 0x7f4c540b01f0 2024-03-23T03:49:02.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.843+0000 7f4c51e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c38006130 0x7f4c541f60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:02.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.843+0000 7f4c51e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c38006130 0x7f4c541f60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41916/0 (socket says 172.21.15.102:41916) 2024-03-23T03:49:02.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.843+0000 7f4c51e00640 1 -- 172.21.15.102:0/1019762451 learned_addr learned my addr 172.21.15.102:0/1019762451 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:02.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.843+0000 7f4c51e00640 1 -- 172.21.15.102:0/1019762451 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c540b01f0 msgr2=0x7f4c541f2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:02.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.843+0000 7f4c51400640 1 --2- 172.21.15.102:0/1019762451 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c540b01f0 0x7f4c541f2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:02.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.843+0000 7f4c52800640 1 --2- 172.21.15.102:0/1019762451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c541f26a0 0x7f4c54147e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:02.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.843+0000 7f4c51e00640 1 --2- 172.21.15.102:0/1019762451 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c540b01f0 0x7f4c541f2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.843+0000 7f4c51e00640 1 -- 172.21.15.102:0/1019762451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c541f26a0 msgr2=0x7f4c54147e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:02.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.843+0000 7f4c51e00640 1 --2- 172.21.15.102:0/1019762451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c541f26a0 0x7f4c54147e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.843+0000 7f4c51e00640 1 -- 172.21.15.102:0/1019762451 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c541f2e50 con 0x7f4c38006130 2024-03-23T03:49:02.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.843+0000 7f4c52800640 1 --2- 172.21.15.102:0/1019762451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c541f26a0 0x7f4c54147e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:02.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.844+0000 7f4c51e00640 1 --2- 172.21.15.102:0/1019762451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c38006130 0x7f4c541f60f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400c500 tx=0x7f4c4400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:02.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.844+0000 7f4c4a000640 1 -- 172.21.15.102:0/1019762451 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c440052d0 con 0x7f4c38006130 2024-03-23T03:49:02.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.844+0000 7f4c4a000640 1 -- 172.21.15.102:0/1019762451 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c44005430 con 0x7f4c38006130 2024-03-23T03:49:02.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.844+0000 7f4c4a000640 1 -- 172.21.15.102:0/1019762451 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c44012870 con 0x7f4c38006130 2024-03-23T03:49:02.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.844+0000 7f4c58a00640 1 -- 172.21.15.102:0/1019762451 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c541f58f0 con 0x7f4c38006130 2024-03-23T03:49:02.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.844+0000 7f4c58a00640 1 -- 172.21.15.102:0/1019762451 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c5414ef80 con 0x7f4c38006130 2024-03-23T03:49:02.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.847+0000 7f4c4a000640 1 -- 172.21.15.102:0/1019762451 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4c440129d0 con 0x7f4c38006130 2024-03-23T03:49:02.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.848+0000 7f4c4a000640 1 --2- 172.21.15.102:0/1019762451 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4c3406ac00 0x7f4c3406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.848+0000 7f4c4a000640 1 -- 172.21.15.102:0/1019762451 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4c44089060 con 0x7f4c38006130 2024-03-23T03:49:02.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.849+0000 7f4c51400640 1 --2- 172.21.15.102:0/1019762451 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4c3406ac00 0x7f4c3406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:02.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.849+0000 7f4c5b817740 1 Processor -- start 2024-03-23T03:49:02.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.849+0000 7f4c5b817740 1 -- start start 2024-03-23T03:49:02.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.849+0000 7f4c51400640 1 --2- 172.21.15.102:0/1019762451 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4c3406ac00 0x7f4c3406d0c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4c40036000 tx=0x7f4c40030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:02.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.850+0000 7f4c5b817740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90acd00 0x5619b907fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.850+0000 7f4c5b817740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90ad0e0 0x5619b90afa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.850+0000 7f4c5b817740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90801e0 0x5619b90affd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.850+0000 7f4c5b817740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619b905b2c0 con 0x5619b90801e0 2024-03-23T03:49:02.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.850+0000 7f4c5b817740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619b8ff08f0 con 0x5619b90acd00 2024-03-23T03:49:02.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.850+0000 7f4c5b817740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619b8fee080 con 0x5619b90ad0e0 2024-03-23T03:49:02.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.850+0000 7f4c52800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90801e0 0x5619b90affd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:02.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.850+0000 7f4c52800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90801e0 0x5619b90affd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:41928/0 (socket says 172.21.15.102:41928) 2024-03-23T03:49:02.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.850+0000 7f4c52800640 1 -- 172.21.15.102:0/3740793202 learned_addr learned my addr 172.21.15.102:0/3740793202 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:02.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.851+0000 7f4c51e00640 1 --2- 172.21.15.102:0/3740793202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90acd00 0x5619b907fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:02.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.851+0000 7f4c51400640 1 --2- 172.21.15.102:0/3740793202 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90ad0e0 0x5619b90afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:02.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.851+0000 7f4c52800640 1 -- 172.21.15.102:0/3740793202 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90ad0e0 msgr2=0x5619b90afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:02.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.851+0000 7f4c52800640 1 --2- 172.21.15.102:0/3740793202 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90ad0e0 0x5619b90afa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.851+0000 7f4c52800640 1 -- 172.21.15.102:0/3740793202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90acd00 msgr2=0x5619b907fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:02.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.851+0000 7f4c52800640 1 --2- 172.21.15.102:0/3740793202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90acd00 0x5619b907fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.851+0000 7f4c52800640 1 -- 172.21.15.102:0/3740793202 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619b8fb4a10 con 0x5619b90801e0 2024-03-23T03:49:02.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.851+0000 7f4c52800640 1 --2- 172.21.15.102:0/3740793202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90801e0 0x5619b90affd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f4c540cb5d0 tx=0x7f4c4c00d150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99cdfb98dd4e3a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:02.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.851+0000 7f4c3ea00640 1 -- 172.21.15.102:0/3740793202 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00d920 con 0x5619b90801e0 2024-03-23T03:49:02.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.851+0000 7f4c3ea00640 1 -- 172.21.15.102:0/3740793202 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c00dac0 con 0x5619b90801e0 2024-03-23T03:49:02.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.851+0000 7f4c3ea00640 1 -- 172.21.15.102:0/3740793202 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c00de70 con 0x5619b90801e0 2024-03-23T03:49:02.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.852+0000 7f4c5b817740 1 -- 172.21.15.102:0/3740793202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90801e0 msgr2=0x5619b90affd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.852+0000 7f4c5b817740 1 --2- 172.21.15.102:0/3740793202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90801e0 0x5619b90affd0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f4c540cb5d0 tx=0x7f4c4c00d150 comp rx=0 tx=0).stop 2024-03-23T03:49:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.852+0000 7f4c5b817740 1 -- 172.21.15.102:0/3740793202 shutdown_connections 2024-03-23T03:49:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.852+0000 7f4c5b817740 1 --2- 172.21.15.102:0/3740793202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90801e0 0x5619b90affd0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.852+0000 7f4c5b817740 1 --2- 172.21.15.102:0/3740793202 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90ad0e0 0x5619b90afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.852+0000 7f4c5b817740 1 --2- 172.21.15.102:0/3740793202 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90acd00 0x5619b907fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.852+0000 7f4c5b817740 1 -- 172.21.15.102:0/3740793202 >> 172.21.15.102:0/3740793202 conn(0x5619b9081f10 msgr2=0x5619b90caa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.852+0000 7f4c5b817740 1 -- 172.21.15.102:0/3740793202 shutdown_connections 2024-03-23T03:49:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.852+0000 7f4c5b817740 1 -- 172.21.15.102:0/3740793202 wait complete. 2024-03-23T03:49:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.853+0000 7f4c5b817740 1 -- start start 2024-03-23T03:49:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.853+0000 7f4c5b817740 1 Processor -- start 2024-03-23T03:49:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.854+0000 7f4c5b817740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90acd00 0x5619b90b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.854+0000 7f4c5b817740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90ad0e0 0x5619b909d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.854+0000 7f4c5b817740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90b4470 0x5619b90b2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.854+0000 7f4c5b817740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619b8fe9970 con 0x5619b90acd00 2024-03-23T03:49:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.854+0000 7f4c5b817740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619b8f8b0a0 con 0x5619b90b4470 2024-03-23T03:49:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.854+0000 7f4c5b817740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619b8f8ad40 con 0x5619b90ad0e0 2024-03-23T03:49:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.854+0000 7f4c5b817740 10 client.0 _ll_register_callbacks cb 0x5619b8fe2fe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T03:49:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.854+0000 7f4c51e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90acd00 0x5619b90b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.854+0000 7f4c51e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90acd00 0x5619b90b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53628/0 (socket says 172.21.15.102:53628) 2024-03-23T03:49:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.854+0000 7f4c51e00640 1 -- 172.21.15.102:0/494763123 learned_addr learned my addr 172.21.15.102:0/494763123 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.855+0000 7f4c51e00640 1 -- 172.21.15.102:0/494763123 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90ad0e0 msgr2=0x5619b909d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.855+0000 7f4c51e00640 1 --2- 172.21.15.102:0/494763123 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90ad0e0 0x5619b909d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.855+0000 7f4c51e00640 1 -- 172.21.15.102:0/494763123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90b4470 msgr2=0x5619b90b2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.855+0000 7f4c51e00640 1 --2- 172.21.15.102:0/494763123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90b4470 0x5619b90b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.855+0000 7f4c51e00640 1 -- 172.21.15.102:0/494763123 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c540ccad0 con 0x5619b90acd00 2024-03-23T03:49:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.855+0000 7f4c51e00640 1 --2- 172.21.15.102:0/494763123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90acd00 0x5619b90b0a80 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400cec0 tx=0x7f4c440585f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d2aee93c6ad6c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.855+0000 7f4c33400640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-23T03:49:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.855+0000 7f4c33400640 1 -- 172.21.15.102:0/494763123 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4405fd10 con 0x5619b90acd00 2024-03-23T03:49:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.855+0000 7f4c33400640 1 -- 172.21.15.102:0/494763123 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4c4405fe70 con 0x5619b90acd00 2024-03-23T03:49:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.855+0000 7f4c5b817740 1 -- 172.21.15.102:0/494763123 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5619b9013b10 con 0x5619b90acd00 2024-03-23T03:49:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.855+0000 7f4c33400640 1 -- 172.21.15.102:0/494763123 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c44087080 con 0x5619b90acd00 2024-03-23T03:49:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.855+0000 7f4c5b817740 10 client.5793 fetch_fsmap learned FSMap version 24 2024-03-23T03:49:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.855+0000 7f4c5b817740 1 -- 172.21.15.102:0/494763123 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x5619b8fe7ff0 con 0x5619b90acd00 2024-03-23T03:49:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.856+0000 7f4c33400640 1 -- 172.21.15.102:0/494763123 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c44073070 con 0x5619b90acd00 2024-03-23T03:49:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.856+0000 7f4c33400640 1 -- 172.21.15.102:0/494763123 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 24) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4c44019020 con 0x5619b90acd00 2024-03-23T03:49:02.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.856+0000 7f4c5b817740 10 client.5793 fetch_fsmap finished waiting for FSMap version 24 2024-03-23T03:49:02.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.856+0000 7f4c5b817740 10 client.5793 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-23T03:49:02.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.856+0000 7f4c5b817740 20 client.5793 populate_metadata read hostname 'smithi102' 2024-03-23T03:49:02.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.856+0000 7f4c5b817740 1 --2- 172.21.15.102:0/494763123 >> [v2:172.21.15.173:6833/3269327930,v1:172.21.15.173:6835/3269327930] conn(0x5619b90ada70 0x5619b90ade50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.856+0000 7f4c5b817740 4 client.5793 mds_command: new command op to 5676 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T03:49:02.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.856+0000 7f4c5b817740 1 -- 172.21.15.102:0/494763123 --> [v2:172.21.15.173:6833/3269327930,v1:172.21.15.173:6835/3269327930] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5619b8cdd3c0 con 0x5619b90ada70 2024-03-23T03:49:02.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.857+0000 7f4c51400640 1 --2- 172.21.15.102:0/494763123 >> [v2:172.21.15.173:6833/3269327930,v1:172.21.15.173:6835/3269327930] conn(0x5619b90ada70 0x5619b90ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:02.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.858+0000 7f4c51400640 1 --2- 172.21.15.102:0/494763123 >> [v2:172.21.15.173:6833/3269327930,v1:172.21.15.173:6835/3269327930] conn(0x5619b90ada70 0x5619b90ade50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95c3d084fe50ba9c server_cookie=f59938b466528fbc in_seq=0 out_seq=0 2024-03-23T03:49:02.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.858+0000 7f4c33400640 10 client.5793 ms_handle_connect on v2:172.21.15.173:6833/3269327930 2024-03-23T03:49:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.860+0000 7f4c33400640 1 -- 172.21.15.102:0/494763123 <== mds.0 v2:172.21.15.173:6833/3269327930 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x5619b8cdd3c0 con 0x5619b90ada70 2024-03-23T03:49:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.860+0000 7f4c33400640 10 client.5793 handle_command_reply: tid=0 2024-03-23T03:49:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.860+0000 7f4c5b817740 1 client.5793 shutdown 2024-03-23T03:49:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.860+0000 7f4c5b817740 1 -- 172.21.15.102:0/494763123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90acd00 msgr2=0x5619b90b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.860+0000 7f4c5b817740 1 --2- 172.21.15.102:0/494763123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90acd00 0x5619b90b0a80 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400cec0 tx=0x7f4c440585f0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.860+0000 7f4c5b817740 1 -- 172.21.15.102:0/494763123 shutdown_connections 2024-03-23T03:49:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.860+0000 7f4c5b817740 1 --2- 172.21.15.102:0/494763123 >> [v2:172.21.15.173:6833/3269327930,v1:172.21.15.173:6835/3269327930] conn(0x5619b90ada70 0x5619b90ade50 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.860+0000 7f4c5b817740 1 --2- 172.21.15.102:0/494763123 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90b4470 0x5619b90b2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.860+0000 7f4c5b817740 1 --2- 172.21.15.102:0/494763123 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90ad0e0 0x5619b909d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.860+0000 7f4c5b817740 1 --2- 172.21.15.102:0/494763123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90acd00 0x5619b90b0a80 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.860+0000 7f4c5b817740 1 -- 172.21.15.102:0/494763123 >> 172.21.15.102:0/494763123 conn(0x5619b90801e0 msgr2=0x5619b90cdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.860+0000 7f4c33400640 0 client.5793 ms_handle_reset on v2:172.21.15.173:6833/3269327930 2024-03-23T03:49:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.860+0000 7f4c5b817740 1 -- 172.21.15.102:0/494763123 shutdown_connections 2024-03-23T03:49:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.860+0000 7f4c5b817740 1 -- 172.21.15.102:0/494763123 wait complete. 2024-03-23T03:49:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.861+0000 7f4c5b817740 20 client.5793 trim_cache size 0 max 16384 2024-03-23T03:49:02.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.877+0000 7f4c5b817740 1 Processor -- start 2024-03-23T03:49:02.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.877+0000 7f4c5b817740 1 -- start start 2024-03-23T03:49:02.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.878+0000 7f4c5b817740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90acd00 0x5619b90d9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.878+0000 7f4c5b817740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90ad0e0 0x5619b90d96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.878+0000 7f4c5b817740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90b4470 0x5619b907b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.878+0000 7f4c5b817740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619b8d50f20 con 0x5619b90b4470 2024-03-23T03:49:02.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.878+0000 7f4c5b817740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619b90ece30 con 0x5619b90acd00 2024-03-23T03:49:02.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.878+0000 7f4c5b817740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619b9084130 con 0x5619b90ad0e0 2024-03-23T03:49:02.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.878+0000 7f4c52800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90b4470 0x5619b907b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:02.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.878+0000 7f4c52800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90b4470 0x5619b907b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53632/0 (socket says 172.21.15.102:53632) 2024-03-23T03:49:02.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.878+0000 7f4c52800640 1 -- 172.21.15.102:0/400740305 learned_addr learned my addr 172.21.15.102:0/400740305 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:02.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.878+0000 7f4c52800640 1 -- 172.21.15.102:0/400740305 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90ad0e0 msgr2=0x5619b90d96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:02.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.878+0000 7f4c52800640 1 --2- 172.21.15.102:0/400740305 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90ad0e0 0x5619b90d96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.878+0000 7f4c52800640 1 -- 172.21.15.102:0/400740305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90acd00 msgr2=0x5619b90d9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:02.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.878+0000 7f4c52800640 1 --2- 172.21.15.102:0/400740305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90acd00 0x5619b90d9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.878+0000 7f4c52800640 1 -- 172.21.15.102:0/400740305 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619b901d7c0 con 0x5619b90b4470 2024-03-23T03:49:02.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.879+0000 7f4c52800640 1 --2- 172.21.15.102:0/400740305 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90b4470 0x5619b907b3c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c001cb0 tx=0x7f4c4c006000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16c1e401cbe41ae server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:02.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.879+0000 7f4c3f400640 1 -- 172.21.15.102:0/400740305 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c01b040 con 0x5619b90b4470 2024-03-23T03:49:02.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.879+0000 7f4c3f400640 1 -- 172.21.15.102:0/400740305 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4c00d020 con 0x5619b90b4470 2024-03-23T03:49:02.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.879+0000 7f4c3f400640 1 -- 172.21.15.102:0/400740305 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4c0072b0 con 0x5619b90b4470 2024-03-23T03:49:02.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.879+0000 7f4c5b817740 1 -- 172.21.15.102:0/400740305 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90b4470 msgr2=0x5619b907b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:02.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.879+0000 7f4c5b817740 1 --2- 172.21.15.102:0/400740305 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90b4470 0x5619b907b3c0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c001cb0 tx=0x7f4c4c006000 comp rx=0 tx=0).stop 2024-03-23T03:49:02.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.879+0000 7f4c5b817740 1 -- 172.21.15.102:0/400740305 shutdown_connections 2024-03-23T03:49:02.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.879+0000 7f4c5b817740 1 --2- 172.21.15.102:0/400740305 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90b4470 0x5619b907b3c0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.879+0000 7f4c5b817740 1 --2- 172.21.15.102:0/400740305 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90ad0e0 0x5619b90d96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.879+0000 7f4c5b817740 1 --2- 172.21.15.102:0/400740305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90acd00 0x5619b90d9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.879+0000 7f4c5b817740 1 -- 172.21.15.102:0/400740305 >> 172.21.15.102:0/400740305 conn(0x5619b90801e0 msgr2=0x5619b90aa7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:02.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.879+0000 7f4c5b817740 1 -- 172.21.15.102:0/400740305 shutdown_connections 2024-03-23T03:49:02.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.880+0000 7f4c5b817740 1 -- 172.21.15.102:0/400740305 wait complete. 2024-03-23T03:49:02.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.880+0000 7f4c5b817740 1 -- start start 2024-03-23T03:49:02.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.880+0000 7f4c5b817740 1 Processor -- start 2024-03-23T03:49:02.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.881+0000 7f4c5b817740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90acd00 0x5619b911ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.881+0000 7f4c5b817740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90ad0e0 0x5619b90cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.881+0000 7f4c5b817740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90b4470 0x5619b907b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.881+0000 7f4c5b817740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619b90a6f10 con 0x5619b90acd00 2024-03-23T03:49:02.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.881+0000 7f4c5b817740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619b90a7370 con 0x5619b90ad0e0 2024-03-23T03:49:02.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.881+0000 7f4c5b817740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619b913c2f0 con 0x5619b90b4470 2024-03-23T03:49:02.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.881+0000 7f4c5b817740 10 client.0 _ll_register_callbacks cb 0x7f4c44057b80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T03:49:02.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.882+0000 7f4c51e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90acd00 0x5619b911ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:02.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.882+0000 7f4c51e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90acd00 0x5619b911ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53634/0 (socket says 172.21.15.102:53634) 2024-03-23T03:49:02.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.882+0000 7f4c51e00640 1 -- 172.21.15.102:0/1594354036 learned_addr learned my addr 172.21.15.102:0/1594354036 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:02.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.882+0000 7f4c52800640 1 --2- 172.21.15.102:0/1594354036 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90b4470 0x5619b907b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:02.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.882+0000 7f4c51e00640 1 -- 172.21.15.102:0/1594354036 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90b4470 msgr2=0x5619b907b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:02.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.882+0000 7f4c51e00640 1 --2- 172.21.15.102:0/1594354036 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90b4470 0x5619b907b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.882+0000 7f4c51e00640 1 -- 172.21.15.102:0/1594354036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90ad0e0 msgr2=0x5619b90cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:02.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.882+0000 7f4c51400640 1 --2- 172.21.15.102:0/1594354036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90ad0e0 0x5619b90cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:02.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.882+0000 7f4c51e00640 1 --2- 172.21.15.102:0/1594354036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90ad0e0 0x5619b90cbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.882+0000 7f4c51e00640 1 -- 172.21.15.102:0/1594354036 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5619b8fb4a10 con 0x5619b90acd00 2024-03-23T03:49:02.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.882+0000 7f4c51e00640 1 --2- 172.21.15.102:0/1594354036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90acd00 0x5619b911ec10 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f4c44056100 tx=0x7f4c44001190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c997e4917fe1e5d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:02.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.882+0000 7f4c33400640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-23T03:49:02.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.882+0000 7f4c33400640 1 -- 172.21.15.102:0/1594354036 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4405ff10 con 0x5619b90acd00 2024-03-23T03:49:02.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.883+0000 7f4c33400640 1 -- 172.21.15.102:0/1594354036 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4c440182b0 con 0x5619b90acd00 2024-03-23T03:49:02.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.883+0000 7f4c5b817740 1 -- 172.21.15.102:0/1594354036 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5619b90cd680 con 0x5619b90acd00 2024-03-23T03:49:02.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.883+0000 7f4c33400640 1 -- 172.21.15.102:0/1594354036 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c4405bbe0 con 0x5619b90acd00 2024-03-23T03:49:02.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.883+0000 7f4c5b817740 10 client.5799 fetch_fsmap learned FSMap version 24 2024-03-23T03:49:02.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.883+0000 7f4c5b817740 1 -- 172.21.15.102:0/1594354036 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x5619b8f8a2b0 con 0x5619b90acd00 2024-03-23T03:49:02.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.883+0000 7f4c33400640 1 -- 172.21.15.102:0/1594354036 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5619b90cd680 con 0x5619b90acd00 2024-03-23T03:49:02.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.883+0000 7f4c33400640 1 -- 172.21.15.102:0/1594354036 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 24) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4c4407d050 con 0x5619b90acd00 2024-03-23T03:49:02.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.883+0000 7f4c5b817740 10 client.5799 fetch_fsmap finished waiting for FSMap version 24 2024-03-23T03:49:02.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.883+0000 7f4c5b817740 10 client.5799 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-23T03:49:02.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.883+0000 7f4c5b817740 20 client.5799 populate_metadata read hostname 'smithi102' 2024-03-23T03:49:02.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.883+0000 7f4c5b817740 1 --2- 172.21.15.102:0/1594354036 >> [v2:172.21.15.173:6833/3269327930,v1:172.21.15.173:6835/3269327930] conn(0x5619b907f000 0x5619b907f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:02.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.883+0000 7f4c5b817740 4 client.5799 mds_command: new command op to 5676 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-23T03:49:02.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.883+0000 7f4c5b817740 1 -- 172.21.15.102:0/1594354036 --> [v2:172.21.15.173:6833/3269327930,v1:172.21.15.173:6835/3269327930] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f4c4c00dac0 con 0x5619b907f000 2024-03-23T03:49:02.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.884+0000 7f4c51400640 1 --2- 172.21.15.102:0/1594354036 >> [v2:172.21.15.173:6833/3269327930,v1:172.21.15.173:6835/3269327930] conn(0x5619b907f000 0x5619b907f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:02.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.885+0000 7f4c51400640 1 --2- 172.21.15.102:0/1594354036 >> [v2:172.21.15.173:6833/3269327930,v1:172.21.15.173:6835/3269327930] conn(0x5619b907f000 0x5619b907f3e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35d4be4b6ce05ed7 server_cookie=678764e935c818ca in_seq=0 out_seq=0 2024-03-23T03:49:02.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.885+0000 7f4c33400640 10 client.5799 ms_handle_connect on v2:172.21.15.173:6833/3269327930 2024-03-23T03:49:02.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.885+0000 7f4c33400640 1 -- 172.21.15.102:0/1594354036 <== mds.0 v2:172.21.15.173:6833/3269327930 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x7f4c4c00dac0 con 0x5619b907f000 2024-03-23T03:49:02.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.885+0000 7f4c33400640 10 client.5799 handle_command_reply: tid=0 2024-03-23T03:49:02.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.885+0000 7f4c5b817740 1 client.5799 shutdown 2024-03-23T03:49:02.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.886+0000 7f4c5b817740 1 -- 172.21.15.102:0/1594354036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90acd00 msgr2=0x5619b911ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:02.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.886+0000 7f4c5b817740 1 --2- 172.21.15.102:0/1594354036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90acd00 0x5619b911ec10 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f4c44056100 tx=0x7f4c44001190 comp rx=0 tx=0).stop 2024-03-23T03:49:02.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.886+0000 7f4c5b817740 1 -- 172.21.15.102:0/1594354036 shutdown_connections 2024-03-23T03:49:02.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.886+0000 7f4c5b817740 1 --2- 172.21.15.102:0/1594354036 >> [v2:172.21.15.173:6833/3269327930,v1:172.21.15.173:6835/3269327930] conn(0x5619b907f000 0x5619b907f3e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.886+0000 7f4c5b817740 1 --2- 172.21.15.102:0/1594354036 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x5619b90b4470 0x5619b907b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.886+0000 7f4c5b817740 1 --2- 172.21.15.102:0/1594354036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x5619b90ad0e0 0x5619b90cbea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.886+0000 7f4c33400640 0 client.5799 ms_handle_reset on v2:172.21.15.173:6833/3269327930 2024-03-23T03:49:02.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.886+0000 7f4c5b817740 1 --2- 172.21.15.102:0/1594354036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5619b90acd00 0x5619b911ec10 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.886+0000 7f4c5b817740 1 -- 172.21.15.102:0/1594354036 >> 172.21.15.102:0/1594354036 conn(0x5619b90801e0 msgr2=0x5619b90d98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:02.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.886+0000 7f4c5b817740 1 -- 172.21.15.102:0/1594354036 shutdown_connections 2024-03-23T03:49:02.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.886+0000 7f4c5b817740 1 -- 172.21.15.102:0/1594354036 wait complete. 2024-03-23T03:49:02.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.886+0000 7f4c5b817740 20 client.5799 trim_cache size 0 max 16384 2024-03-23T03:49:02.889 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:49:02.889 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "776b2820-05f2-4e50-8d40-18dd76c5f6a0", 2024-03-23T03:49:02.889 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 0, 2024-03-23T03:49:02.889 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 5676, 2024-03-23T03:49:02.889 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-03-23T03:49:02.889 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-03-23T03:49:02.889 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-03-23T03:49:02.889 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 6.6585984070000004, 2024-03-23T03:49:02.889 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 24, 2024-03-23T03:49:02.889 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 55, 2024-03-23T03:49:02.890 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 55, 2024-03-23T03:49:02.890 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 12.187778173 2024-03-23T03:49:02.890 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:49:02.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.886+0000 7f4c58a00640 1 -- 172.21.15.102:0/1019762451 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4c3406ac00 msgr2=0x7f4c3406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:02.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.886+0000 7f4c58a00640 1 --2- 172.21.15.102:0/1019762451 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4c3406ac00 0x7f4c3406d0c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4c40036000 tx=0x7f4c40030000 comp rx=0 tx=0).stop 2024-03-23T03:49:02.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.887+0000 7f4c58a00640 1 -- 172.21.15.102:0/1019762451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c38006130 msgr2=0x7f4c541f60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:02.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.887+0000 7f4c58a00640 1 --2- 172.21.15.102:0/1019762451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c38006130 0x7f4c541f60f0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f4c4400c500 tx=0x7f4c4400c9c0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.887+0000 7f4c58a00640 1 -- 172.21.15.102:0/1019762451 shutdown_connections 2024-03-23T03:49:02.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.887+0000 7f4c58a00640 1 --2- 172.21.15.102:0/1019762451 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4c3406ac00 0x7f4c3406d0c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.887+0000 7f4c58a00640 1 --2- 172.21.15.102:0/1019762451 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4c541f26a0 0x7f4c54147e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.887+0000 7f4c58a00640 1 --2- 172.21.15.102:0/1019762451 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4c540b01f0 0x7f4c541f2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.887+0000 7f4c58a00640 1 --2- 172.21.15.102:0/1019762451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4c38006130 0x7f4c541f60f0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:02.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.887+0000 7f4c58a00640 1 -- 172.21.15.102:0/1019762451 >> 172.21.15.102:0/1019762451 conn(0x7f4c5406a0b0 msgr2=0x7f4c5406c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:02.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.887+0000 7f4c58a00640 1 -- 172.21.15.102:0/1019762451 shutdown_connections 2024-03-23T03:49:02.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:02.887+0000 7f4c58a00640 1 -- 172.21.15.102:0/1019762451 wait complete. 2024-03-23T03:49:02.904 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:49:03.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.102+0000 7fd5ce400640 1 Processor -- start 2024-03-23T03:49:03.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.102+0000 7fd5ce400640 1 -- start start 2024-03-23T03:49:03.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.103+0000 7fd5ce400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5c80cbab0 0x7fd5c80b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:03.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.103+0000 7fd5ce400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c80b16a0 con 0x7fd5c80adbc0 2024-03-23T03:49:03.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.103+0000 7fd5ce400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c80cbfb0 con 0x7fd5c806d390 2024-03-23T03:49:03.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.103+0000 7fd5ce400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c80cc290 con 0x7fd5c80cbab0 2024-03-23T03:49:03.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.104+0000 7fd5c7400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd5c80adbc0 0x7fd5c80b41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57218/0 (socket says 172.21.15.102:57218) 2024-03-23T03:49:03.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.104+0000 7fd5c7400640 1 -- 172.21.15.102:0/510418295 learned_addr learned my addr 172.21.15.102:0/510418295 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:03.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.104+0000 7fd5bea00640 1 --2- 172.21.15.102:0/510418295 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5c80cbab0 0x7fd5c80b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:03.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.104+0000 7fd5c6a00640 1 -- 172.21.15.102:0/510418295 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3521771178 0 0) 0x7fd5c80b16a0 con 0x7fd5c80adbc0 2024-03-23T03:49:03.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.104+0000 7fd5c6a00640 1 -- 172.21.15.102:0/510418295 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5ac003520 con 0x7fd5c80adbc0 2024-03-23T03:49:03.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.105+0000 7fd5c6a00640 1 -- 172.21.15.102:0/510418295 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3270042247 0 0) 0x7fd5ac003520 con 0x7fd5c80adbc0 2024-03-23T03:49:03.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.105+0000 7fd5c6a00640 1 -- 172.21.15.102:0/510418295 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5c80cbab0 msgr2=0x7fd5c80b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:03.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.105+0000 7fd5c6a00640 1 --2- 172.21.15.102:0/510418295 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5c80cbab0 0x7fd5c80b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:03.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.105+0000 7fd5c6a00640 1 -- 172.21.15.102:0/510418295 >> v1:172.21.15.173:6789/0 conn(0x7fd5c806d390 legacy=0x7fd5c80b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:03.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.105+0000 7fd5c6a00640 1 -- 172.21.15.102:0/510418295 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c80b18f0 con 0x7fd5c80adbc0 2024-03-23T03:49:03.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.105+0000 7fd5c6a00640 1 -- 172.21.15.102:0/510418295 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fd5b8001670 con 0x7fd5c80adbc0 2024-03-23T03:49:03.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.105+0000 7fd5c6a00640 1 -- 172.21.15.102:0/510418295 >> v1:172.21.15.102:6789/0 conn(0x7fd5c80adbc0 legacy=0x7fd5c80b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:03.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.105+0000 7fd5c6a00640 1 --2- 172.21.15.102:0/510418295 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5c80cbab0 0x7fd5ac0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:03.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.105+0000 7fd5c6a00640 1 --2- 172.21.15.102:0/510418295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5ac003f70 0x7fd5ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:03.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.105+0000 7fd5c6a00640 1 --2- 172.21.15.102:0/510418295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5ac004d00 0x7fd5ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:03.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.105+0000 7fd5c6a00640 1 -- 172.21.15.102:0/510418295 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c80cc290 con 0x7fd5ac003f70 2024-03-23T03:49:03.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.105+0000 7fd5c6a00640 1 -- 172.21.15.102:0/510418295 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ac00bb70 con 0x7fd5ac004d00 2024-03-23T03:49:03.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.105+0000 7fd5c6a00640 1 -- 172.21.15.102:0/510418295 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5ac00df40 con 0x7fd5c80cbab0 2024-03-23T03:49:03.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.105+0000 7fd5bea00640 1 --2- 172.21.15.102:0/510418295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5ac003f70 0x7fd5ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:03.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.105+0000 7fd5c7400640 1 --2- 172.21.15.102:0/510418295 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5c80cbab0 0x7fd5ac0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:03.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.106+0000 7fd5bea00640 1 -- 172.21.15.102:0/510418295 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5c80cbab0 msgr2=0x7fd5ac0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:03.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.106+0000 7fd5bea00640 1 --2- 172.21.15.102:0/510418295 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5c80cbab0 0x7fd5ac0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:03.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.106+0000 7fd5c7e00640 1 --2- 172.21.15.102:0/510418295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5ac004d00 0x7fd5ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:03.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.106+0000 7fd5bea00640 1 -- 172.21.15.102:0/510418295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5ac004d00 msgr2=0x7fd5ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:03.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.106+0000 7fd5bea00640 1 --2- 172.21.15.102:0/510418295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5ac004d00 0x7fd5ac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:03.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.106+0000 7fd5bea00640 1 -- 172.21.15.102:0/510418295 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ac003520 con 0x7fd5ac003f70 2024-03-23T03:49:03.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.106+0000 7fd5c7400640 1 --2- 172.21.15.102:0/510418295 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5c80cbab0 0x7fd5ac0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:03.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.106+0000 7fd5c7e00640 1 --2- 172.21.15.102:0/510418295 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5ac004d00 0x7fd5ac005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.107+0000 7fd5bea00640 1 --2- 172.21.15.102:0/510418295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5ac003f70 0x7fd5ac0047c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0009ac0 tx=0x7fd5ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4209a00a03aeab3a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.107+0000 7fd5c6a00640 1 -- 172.21.15.102:0/510418295 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b002c6b0 con 0x7fd5ac003f70 2024-03-23T03:49:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.107+0000 7fd5c6a00640 1 -- 172.21.15.102:0/510418295 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b002c810 con 0x7fd5ac003f70 2024-03-23T03:49:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.107+0000 7fd5c6a00640 1 -- 172.21.15.102:0/510418295 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b002cba0 con 0x7fd5ac003f70 2024-03-23T03:49:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.108+0000 7fd5ce400640 1 -- 172.21.15.102:0/510418295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5ac003f70 msgr2=0x7fd5ac0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.108+0000 7fd5ce400640 1 --2- 172.21.15.102:0/510418295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5ac003f70 0x7fd5ac0047c0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fd5b0009ac0 tx=0x7fd5ac0057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.108+0000 7fd5c7e00640 1 -- 172.21.15.102:0/510418295 reap_dead start 2024-03-23T03:49:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.108+0000 7fd5ce400640 1 -- 172.21.15.102:0/510418295 shutdown_connections 2024-03-23T03:49:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.108+0000 7fd5ce400640 1 -- 172.21.15.102:0/510418295 >> 172.21.15.102:0/510418295 conn(0x7fd5c806a3d0 msgr2=0x7fd5c80b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:03.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.108+0000 7fd5ce400640 1 -- 172.21.15.102:0/510418295 shutdown_connections 2024-03-23T03:49:03.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.108+0000 7fd5ce400640 1 -- 172.21.15.102:0/510418295 wait complete. 2024-03-23T03:49:03.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.110+0000 7fd5ce400640 1 Processor -- start 2024-03-23T03:49:03.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.110+0000 7fd5ce400640 1 -- start start 2024-03-23T03:49:03.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.111+0000 7fd5ce400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5ac003f70 0x7fd5c81f5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.111+0000 7fd5ce400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c80c3610 0x7fd5c81f1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.111+0000 7fd5ce400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5c806c340 0x7fd5c806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.111+0000 7fd5ce400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c80c6b80 con 0x7fd5c80c3610 2024-03-23T03:49:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.111+0000 7fd5ce400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c80c3f60 con 0x7fd5c806c340 2024-03-23T03:49:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.111+0000 7fd5ce400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c806cd00 con 0x7fd5ac003f70 2024-03-23T03:49:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.111+0000 7fd5bea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c80c3610 0x7fd5c81f1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.111+0000 7fd5bea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c80c3610 0x7fd5c81f1b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53650/0 (socket says 172.21.15.102:53650) 2024-03-23T03:49:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.111+0000 7fd5bea00640 1 -- 172.21.15.102:0/1533480675 learned_addr learned my addr 172.21.15.102:0/1533480675 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.111+0000 7fd5bea00640 1 -- 172.21.15.102:0/1533480675 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5ac003f70 msgr2=0x7fd5c81f5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.111+0000 7fd5bea00640 1 --2- 172.21.15.102:0/1533480675 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5ac003f70 0x7fd5c81f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.111+0000 7fd5bea00640 1 -- 172.21.15.102:0/1533480675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5c806c340 msgr2=0x7fd5c806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.111+0000 7fd5bea00640 1 --2- 172.21.15.102:0/1533480675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5c806c340 0x7fd5c806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.111+0000 7fd5bea00640 1 -- 172.21.15.102:0/1533480675 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c80c4200 con 0x7fd5c80c3610 2024-03-23T03:49:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.112+0000 7fd5bea00640 1 --2- 172.21.15.102:0/1533480675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c80c3610 0x7fd5c81f1b70 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fd5b002c250 tx=0x7fd5b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:03.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.112+0000 7fd5bfe00640 1 -- 172.21.15.102:0/1533480675 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b0003810 con 0x7fd5c80c3610 2024-03-23T03:49:03.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.112+0000 7fd5bfe00640 1 -- 172.21.15.102:0/1533480675 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b0003970 con 0x7fd5c80c3610 2024-03-23T03:49:03.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.112+0000 7fd5bfe00640 1 -- 172.21.15.102:0/1533480675 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b0003ce0 con 0x7fd5c80c3610 2024-03-23T03:49:03.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.112+0000 7fd5ce400640 1 -- 172.21.15.102:0/1533480675 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c81f5db0 con 0x7fd5c80c3610 2024-03-23T03:49:03.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.112+0000 7fd5ce400640 1 -- 172.21.15.102:0/1533480675 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c81f6330 con 0x7fd5c80c3610 2024-03-23T03:49:03.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.113+0000 7fd5ce400640 1 -- 172.21.15.102:0/1533480675 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c806ac20 con 0x7fd5c80c3610 2024-03-23T03:49:03.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.114+0000 7fd5bfe00640 1 -- 172.21.15.102:0/1533480675 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd5b0034440 con 0x7fd5c80c3610 2024-03-23T03:49:03.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.115+0000 7fd5bfe00640 1 --2- 172.21.15.102:0/1533480675 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd5a406ac30 0x7fd5a406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:03.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.115+0000 7fd5bfe00640 1 -- 172.21.15.102:0/1533480675 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd5b00ab640 con 0x7fd5c80c3610 2024-03-23T03:49:03.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.115+0000 7fd5c7400640 1 --2- 172.21.15.102:0/1533480675 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd5a406ac30 0x7fd5a406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:03.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.116+0000 7fd5c7400640 1 --2- 172.21.15.102:0/1533480675 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd5a406ac30 0x7fd5a406d0f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8007a10 tx=0x7fd5b8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:03.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.120+0000 7fd5bfe00640 1 -- 172.21.15.102:0/1533480675 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd5b0076940 con 0x7fd5c80c3610 2024-03-23T03:49:03.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.354+0000 7fd5ce400640 1 -- 172.21.15.102:0/1533480675 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd5c8073390 con 0x7fd5c80c3610 2024-03-23T03:49:03.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.355+0000 7fd5bfe00640 1 -- 172.21.15.102:0/1533480675 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd5b00044e0 con 0x7fd5c80c3610 2024-03-23T03:49:03.355 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T03:49:03.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.357+0000 7fd5ce400640 1 -- 172.21.15.102:0/1533480675 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd5a406ac30 msgr2=0x7fd5a406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:03.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.357+0000 7fd5ce400640 1 --2- 172.21.15.102:0/1533480675 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd5a406ac30 0x7fd5a406d0f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd5b8007a10 tx=0x7fd5b8009040 comp rx=0 tx=0).stop 2024-03-23T03:49:03.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.357+0000 7fd5ce400640 1 -- 172.21.15.102:0/1533480675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c80c3610 msgr2=0x7fd5c81f1b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:03.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.357+0000 7fd5ce400640 1 --2- 172.21.15.102:0/1533480675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c80c3610 0x7fd5c81f1b70 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fd5b002c250 tx=0x7fd5b0004730 comp rx=0 tx=0).stop 2024-03-23T03:49:03.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.357+0000 7fd5ce400640 1 -- 172.21.15.102:0/1533480675 shutdown_connections 2024-03-23T03:49:03.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.357+0000 7fd5ce400640 1 --2- 172.21.15.102:0/1533480675 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd5a406ac30 0x7fd5a406d0f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:03.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.357+0000 7fd5ce400640 1 --2- 172.21.15.102:0/1533480675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5c806c340 0x7fd5c806c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:03.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.357+0000 7fd5ce400640 1 --2- 172.21.15.102:0/1533480675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5c80c3610 0x7fd5c81f1b70 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:03.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.357+0000 7fd5ce400640 1 --2- 172.21.15.102:0/1533480675 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5ac003f70 0x7fd5c81f5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:03.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.357+0000 7fd5ce400640 1 -- 172.21.15.102:0/1533480675 >> 172.21.15.102:0/1533480675 conn(0x7fd5c806a3d0 msgr2=0x7fd5c80726a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:03.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.357+0000 7fd5ce400640 1 -- 172.21.15.102:0/1533480675 shutdown_connections 2024-03-23T03:49:03.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:03.358+0000 7fd5ce400640 1 -- 172.21.15.102:0/1533480675 wait complete. 2024-03-23T03:49:03.377 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-23T03:49:03.377 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:49:03.378 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2024-03-23T03:49:03.378 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T03:49:03.378 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T03:49:03.378 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-23T03:49:03.378 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-23T03:49:03.378 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:49:03.378 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:49:03.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:03.378 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2024-03-23T03:49:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:49:03.431 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:49:03.431 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:49:03.431 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:03.431 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-03-23T03:49:03.431 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:49:03.431 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:03.431 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f0 2024-03-23T03:49:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:49:03.431 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:03.431 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f1 2024-03-23T03:49:03.431 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:49:03.432 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:49:03.432 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:03.432 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:49:03.432 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 42527sec preferred_lft 42527sec 2024-03-23T03:49:03.432 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-03-23T03:49:03.432 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:49:03.432 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether aa:17:e3:39:b0:56 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:03.432 INFO:teuthology.orchestra.run.smithi102.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T03:49:03.432 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:03.432 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::5cce:1fff:fe41:5a05/64 scope link 2024-03-23T03:49:03.432 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:03.432 INFO:teuthology.orchestra.run.smithi102.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:49:03.432 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether aa:17:e3:39:b0:56 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T03:49:03.433 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::a817:e3ff:fe39:b056/64 scope link 2024-03-23T03:49:03.433 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:03.433 INFO:teuthology.orchestra.run.smithi102.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:49:03.433 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether ce:9b:95:fb:de:c5 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T03:49:03.433 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::cc9b:95ff:fefb:dec5/64 scope link 2024-03-23T03:49:03.433 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:03.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:03.433 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:49:03.519 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-23T03:49:03.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:03.523 DEBUG:teuthology.orchestra.run.smithi102:> route 2024-03-23T03:49:03.575 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2024-03-23T03:49:03.575 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:49:03.575 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:49:03.575 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:49:03.575 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:49:03.576 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:03.576 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:49:03.576 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:49:03.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:49:03.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:49:03.576 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:49:03.576 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:49:03.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:03.756 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-23T03:49:03.803 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T03:49:03.803 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T03:49:03.803 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:49:03.804 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:03.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.0 up 2024-03-23T03:49:03.888 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:49:03.888 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T03:49:03.939 INFO:teuthology.orchestra.run.smithi102.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:49:03.939 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:49:03.939 DEBUG:teuthology.orchestra.run.smithi102:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T03:49:03.988 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-23T03:49:03.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:03.988 DEBUG:teuthology.orchestra.run.smithi102:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:49:04.126 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:49:04.126 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:49:04.126 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:49:04.148 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:49:04.148 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:49:04.148 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:49:04.148 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:49:04.148 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:49:04.178 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:49:04.178 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:49:04.178 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:49:04.178 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:49:04.178 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:49:04.178 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790" 2024-03-23T03:49:04.178 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/ 2024-03-23T03:49:04.178 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=776b2820-05f2-4e50-8d40-18dd76c5f6a0" will pass to kernel 2024-03-23T03:49:04.181 INFO:teuthology.orchestra.run.smithi102.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-23T03:49:04.181 INFO:teuthology.orchestra.run.smithi102.stdout: You just mounted a file system that supports labels which does not 2024-03-23T03:49:04.181 INFO:teuthology.orchestra.run.smithi102.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T03:49:04.181 INFO:teuthology.orchestra.run.smithi102.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T03:49:04.181 INFO:teuthology.orchestra.run.smithi102.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T03:49:04.186 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T03:49:04.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:04.186 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T03:49:04.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:04.267 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:49:04.346 INFO:teuthology.orchestra.run.smithi102.stdout:client4844 2024-03-23T03:49:04.346 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:49:04.346 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:04.400 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:49:04.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.606+0000 7fb865200640 1 Processor -- start 2024-03-23T03:49:04.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.607+0000 7fb865200640 1 -- start start 2024-03-23T03:49:04.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.608+0000 7fb865200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb86013faa0 0x7fb860160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:04.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.608+0000 7fb865200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8600b38a0 con 0x7fb8600b1db0 2024-03-23T03:49:04.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.608+0000 7fb865200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8600b3b80 con 0x7fb860160800 2024-03-23T03:49:04.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.608+0000 7fb865200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8600b3e60 con 0x7fb86013faa0 2024-03-23T03:49:04.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.608+0000 7fb85de00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb86013faa0 0x7fb860160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:04.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.608+0000 7fb85de00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb86013faa0 0x7fb860160f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53654/0 (socket says 172.21.15.102:53654) 2024-03-23T03:49:04.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.608+0000 7fb85de00640 1 -- 172.21.15.102:0/2017770467 learned_addr learned my addr 172.21.15.102:0/2017770467 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:04.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.609+0000 7fb85de00640 1 -- 172.21.15.102:0/2017770467 >> v1:172.21.15.173:6789/0 conn(0x7fb860160800 legacy=0x7fb8600b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:04.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.609+0000 7fb85de00640 1 -- 172.21.15.102:0/2017770467 >> v1:172.21.15.102:6789/0 conn(0x7fb8600b1db0 legacy=0x7fb86006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:04.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.609+0000 7fb85de00640 1 -- 172.21.15.102:0/2017770467 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8600b4140 con 0x7fb86013faa0 2024-03-23T03:49:04.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.610+0000 7fb85de00640 1 --2- 172.21.15.102:0/2017770467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb86013faa0 0x7fb860160f30 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fb850009870 tx=0x7fb85002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3830f1c34a15d30 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.610+0000 7fb85ca00640 1 -- 172.21.15.102:0/2017770467 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb850030860 con 0x7fb86013faa0 2024-03-23T03:49:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.610+0000 7fb85ca00640 1 -- 172.21.15.102:0/2017770467 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8500309c0 con 0x7fb86013faa0 2024-03-23T03:49:04.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.610+0000 7fb85ca00640 1 -- 172.21.15.102:0/2017770467 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb850030d50 con 0x7fb86013faa0 2024-03-23T03:49:04.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.611+0000 7fb865200640 1 -- 172.21.15.102:0/2017770467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb86013faa0 msgr2=0x7fb860160f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:04.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.611+0000 7fb865200640 1 --2- 172.21.15.102:0/2017770467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb86013faa0 0x7fb860160f30 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fb850009870 tx=0x7fb85002b9a0 comp rx=0 tx=0).stop 2024-03-23T03:49:04.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.611+0000 7fb865200640 1 -- 172.21.15.102:0/2017770467 shutdown_connections 2024-03-23T03:49:04.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.611+0000 7fb865200640 1 --2- 172.21.15.102:0/2017770467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb86013faa0 0x7fb860160f30 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:04.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.611+0000 7fb865200640 1 -- 172.21.15.102:0/2017770467 >> 172.21.15.102:0/2017770467 conn(0x7fb86006a3d0 msgr2=0x7fb8600b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:04.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.611+0000 7fb865200640 1 -- 172.21.15.102:0/2017770467 shutdown_connections 2024-03-23T03:49:04.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.611+0000 7fb865200640 1 -- 172.21.15.102:0/2017770467 wait complete. 2024-03-23T03:49:04.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.613+0000 7fb865200640 1 Processor -- start 2024-03-23T03:49:04.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.613+0000 7fb865200640 1 -- start start 2024-03-23T03:49:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.614+0000 7fb865200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8600b1db0 0x7fb8600ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.614+0000 7fb865200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb86013faa0 0x7fb86015f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.614+0000 7fb865200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb860160800 0x7fb86015fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.614+0000 7fb865200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb860153840 con 0x7fb860160800 2024-03-23T03:49:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.614+0000 7fb865200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb860136960 con 0x7fb8600b1db0 2024-03-23T03:49:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.614+0000 7fb865200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8601f5f00 con 0x7fb86013faa0 2024-03-23T03:49:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.614+0000 7fb85e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb860160800 0x7fb86015fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:04.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.614+0000 7fb85e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb860160800 0x7fb86015fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53658/0 (socket says 172.21.15.102:53658) 2024-03-23T03:49:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.614+0000 7fb85e800640 1 -- 172.21.15.102:0/3267394481 learned_addr learned my addr 172.21.15.102:0/3267394481 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.615+0000 7fb85de00640 1 --2- 172.21.15.102:0/3267394481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8600b1db0 0x7fb8600ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.615+0000 7fb85d400640 1 --2- 172.21.15.102:0/3267394481 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb86013faa0 0x7fb86015f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.615+0000 7fb85e800640 1 -- 172.21.15.102:0/3267394481 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb86013faa0 msgr2=0x7fb86015f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.615+0000 7fb85e800640 1 --2- 172.21.15.102:0/3267394481 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb86013faa0 0x7fb86015f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.615+0000 7fb85e800640 1 -- 172.21.15.102:0/3267394481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8600b1db0 msgr2=0x7fb8600ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.615+0000 7fb85e800640 1 --2- 172.21.15.102:0/3267394481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8600b1db0 0x7fb8600ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.615+0000 7fb85e800640 1 -- 172.21.15.102:0/3267394481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8500094d0 con 0x7fb860160800 2024-03-23T03:49:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.615+0000 7fb85e800640 1 --2- 172.21.15.102:0/3267394481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb860160800 0x7fb86015fae0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb85800c730 tx=0x7fb85800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.615+0000 7fb856000640 1 -- 172.21.15.102:0/3267394481 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb858003b70 con 0x7fb860160800 2024-03-23T03:49:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.615+0000 7fb856000640 1 -- 172.21.15.102:0/3267394481 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb858003cd0 con 0x7fb860160800 2024-03-23T03:49:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.616+0000 7fb856000640 1 -- 172.21.15.102:0/3267394481 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85800aa90 con 0x7fb860160800 2024-03-23T03:49:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.616+0000 7fb865200640 1 -- 172.21.15.102:0/3267394481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8601f61e0 con 0x7fb860160800 2024-03-23T03:49:04.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.616+0000 7fb865200640 1 -- 172.21.15.102:0/3267394481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8601f67a0 con 0x7fb860160800 2024-03-23T03:49:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.617+0000 7fb865200640 1 -- 172.21.15.102:0/3267394481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb860073390 con 0x7fb860160800 2024-03-23T03:49:04.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.618+0000 7fb856000640 1 -- 172.21.15.102:0/3267394481 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb85800abf0 con 0x7fb860160800 2024-03-23T03:49:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.618+0000 7fb856000640 1 --2- 172.21.15.102:0/3267394481 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb83c06ac00 0x7fb83c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.619+0000 7fb856000640 1 -- 172.21.15.102:0/3267394481 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb8580892b0 con 0x7fb860160800 2024-03-23T03:49:04.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.619+0000 7fb85de00640 1 --2- 172.21.15.102:0/3267394481 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb83c06ac00 0x7fb83c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:04.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.619+0000 7fb85de00640 1 --2- 172.21.15.102:0/3267394481 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb83c06ac00 0x7fb83c06d0c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb850036000 tx=0x7fb850008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:04.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.623+0000 7fb856000640 1 -- 172.21.15.102:0/3267394481 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb858054340 con 0x7fb860160800 2024-03-23T03:49:04.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.857+0000 7fb865200640 1 -- 172.21.15.102:0/3267394481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb860136ad0 con 0x7fb860160800 2024-03-23T03:49:04.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.858+0000 7fb856000640 1 -- 172.21.15.102:0/3267394481 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb858057960 con 0x7fb860160800 2024-03-23T03:49:04.859 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T03:49:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.861+0000 7fb865200640 1 -- 172.21.15.102:0/3267394481 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb83c06ac00 msgr2=0x7fb83c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.861+0000 7fb865200640 1 --2- 172.21.15.102:0/3267394481 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb83c06ac00 0x7fb83c06d0c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb850036000 tx=0x7fb850008030 comp rx=0 tx=0).stop 2024-03-23T03:49:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.861+0000 7fb865200640 1 -- 172.21.15.102:0/3267394481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb860160800 msgr2=0x7fb86015fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.861+0000 7fb865200640 1 --2- 172.21.15.102:0/3267394481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb860160800 0x7fb86015fae0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fb85800c730 tx=0x7fb85800cbf0 comp rx=0 tx=0).stop 2024-03-23T03:49:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.861+0000 7fb865200640 1 -- 172.21.15.102:0/3267394481 shutdown_connections 2024-03-23T03:49:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.861+0000 7fb865200640 1 --2- 172.21.15.102:0/3267394481 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb83c06ac00 0x7fb83c06d0c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.861+0000 7fb865200640 1 --2- 172.21.15.102:0/3267394481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb860160800 0x7fb86015fae0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.861+0000 7fb865200640 1 --2- 172.21.15.102:0/3267394481 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb86013faa0 0x7fb86015f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.861+0000 7fb865200640 1 --2- 172.21.15.102:0/3267394481 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8600b1db0 0x7fb8600ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.861+0000 7fb865200640 1 -- 172.21.15.102:0/3267394481 >> 172.21.15.102:0/3267394481 conn(0x7fb86006a3d0 msgr2=0x7fb8601408b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.861+0000 7fb865200640 1 -- 172.21.15.102:0/3267394481 shutdown_connections 2024-03-23T03:49:04.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:04.861+0000 7fb865200640 1 -- 172.21.15.102:0/3267394481 wait complete. 2024-03-23T03:49:04.881 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-23T03:49:04.881 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:49:04.881 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi173.front.sepia.ceph.com 2024-03-23T03:49:04.881 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-23T03:49:04.881 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-23T03:49:04.881 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-23T03:49:04.881 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-23T03:49:04.881 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:49:04.881 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:49:04.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:04.882 DEBUG:teuthology.orchestra.run.smithi173:> ip addr 2024-03-23T03:49:04.941 INFO:teuthology.orchestra.run.smithi173.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:49:04.941 INFO:teuthology.orchestra.run.smithi173.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:49:04.941 INFO:teuthology.orchestra.run.smithi173.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:49:04.941 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:04.942 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 ::1/128 scope host 2024-03-23T03:49:04.942 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:04.942 INFO:teuthology.orchestra.run.smithi173.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:49:04.942 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6a brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:04.942 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f0 2024-03-23T03:49:04.942 INFO:teuthology.orchestra.run.smithi173.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:49:04.942 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6b brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:04.942 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f1 2024-03-23T03:49:04.942 INFO:teuthology.orchestra.run.smithi173.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:49:04.942 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:58 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:04.943 INFO:teuthology.orchestra.run.smithi173.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:49:04.943 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:59 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:04.943 INFO:teuthology.orchestra.run.smithi173.stdout: inet 172.21.15.173/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:49:04.943 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft 42497sec preferred_lft 42497sec 2024-03-23T03:49:04.943 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::ec4:7aff:fe88:7259/64 scope link 2024-03-23T03:49:04.943 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:04.943 INFO:teuthology.orchestra.run.smithi173.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:49:04.943 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 52:69:f2:bf:6d:02 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:04.943 INFO:teuthology.orchestra.run.smithi173.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T03:49:04.943 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:04.943 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::6894:e9ff:fe5d:1492/64 scope link 2024-03-23T03:49:04.943 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:04.943 INFO:teuthology.orchestra.run.smithi173.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:49:04.943 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether a2:27:ac:42:68:ce brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T03:49:04.943 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::a027:acff:fe42:68ce/64 scope link 2024-03-23T03:49:04.943 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:04.944 INFO:teuthology.orchestra.run.smithi173.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:49:04.944 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 52:69:f2:bf:6d:02 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-23T03:49:04.944 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::5069:f2ff:febf:6d02/64 scope link 2024-03-23T03:49:04.944 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:04.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:04.945 DEBUG:teuthology.orchestra.run.smithi173:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:49:05.033 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-03-23T03:49:05.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:05.038 DEBUG:teuthology.orchestra.run.smithi173:> route 2024-03-23T03:49:05.091 INFO:teuthology.orchestra.run.smithi173.stdout:Kernel IP routing table 2024-03-23T03:49:05.091 INFO:teuthology.orchestra.run.smithi173.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:49:05.091 INFO:teuthology.orchestra.run.smithi173.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:49:05.091 INFO:teuthology.orchestra.run.smithi173.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:49:05.091 INFO:teuthology.orchestra.run.smithi173.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:49:05.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:05.091 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-03-23T03:49:05.091 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-03-23T03:49:05.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:49:05.091 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:49:05.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:49:05.092 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-03-23T03:49:05.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:05.277 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list 2024-03-23T03:49:05.326 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-23T03:49:05.326 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-23T03:49:05.327 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:49:05.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:05.327 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link set brx.0 up 2024-03-23T03:49:05.416 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:49:05.417 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-23T03:49:05.469 INFO:teuthology.orchestra.run.smithi173.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:49:05.469 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:49:05.469 DEBUG:teuthology.orchestra.run.smithi173:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-23T03:49:05.519 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-23T03:49:05.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:05.519 DEBUG:teuthology.orchestra.run.smithi173:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:49:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:49:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:49:05.657 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:49:05.679 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:49:05.679 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:49:05.679 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:49:05.679 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:49:05.679 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:49:05.706 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:49:05.707 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:49:05.707 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:49:05.707 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:49:05.707 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:49:05.707 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790" 2024-03-23T03:49:05.707 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/ 2024-03-23T03:49:05.707 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=776b2820-05f2-4e50-8d40-18dd76c5f6a0" will pass to kernel 2024-03-23T03:49:05.709 INFO:teuthology.orchestra.run.smithi173.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-23T03:49:05.709 INFO:teuthology.orchestra.run.smithi173.stdout: You just mounted a file system that supports labels which does not 2024-03-23T03:49:05.709 INFO:teuthology.orchestra.run.smithi173.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T03:49:05.709 INFO:teuthology.orchestra.run.smithi173.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T03:49:05.709 INFO:teuthology.orchestra.run.smithi173.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T03:49:05.713 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T03:49:05.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:05.713 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-23T03:49:05.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:05.796 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:49:05.848 INFO:teuthology.orchestra.run.smithi173.stdout:client5817 2024-03-23T03:49:05.849 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:49:05.849 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:05.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:05.898 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-03-23T03:49:05.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:05.968 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 10A ./subdir) 2024-03-23T03:49:06.023 INFO:teuthology.orchestra.run.smithi102.stderr:setfattr: ./subdir: Invalid argument 2024-03-23T03:49:06.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:49:06.025 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:06.025 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 1y00Ki ./subdir) 2024-03-23T03:49:06.077 INFO:teuthology.orchestra.run.smithi102.stderr:setfattr: ./subdir: Invalid argument 2024-03-23T03:49:06.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:49:06.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:06.078 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v af00 ./subdir) 2024-03-23T03:49:06.130 INFO:teuthology.orchestra.run.smithi102.stderr:setfattr: ./subdir: Invalid argument 2024-03-23T03:49:06.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:49:06.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:06.132 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v G ./subdir) 2024-03-23T03:49:06.182 INFO:teuthology.orchestra.run.smithi102.stderr:setfattr: ./subdir: Invalid argument 2024-03-23T03:49:06.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:49:06.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:06.183 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v '' ./subdir) 2024-03-23T03:49:06.233 INFO:teuthology.orchestra.run.smithi102.stderr:setfattr: ./subdir: Invalid argument 2024-03-23T03:49:06.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:49:06.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:06.234 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v ' ' ./subdir) 2024-03-23T03:49:06.284 INFO:teuthology.orchestra.run.smithi102.stderr:setfattr: ./subdir: Invalid argument 2024-03-23T03:49:06.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:49:06.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:06.285 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v -1t ./subdir) 2024-03-23T03:49:06.335 INFO:teuthology.orchestra.run.smithi102.stderr:setfattr: ./subdir: Invalid argument 2024-03-23T03:49:06.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:49:06.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:06.337 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v -1 ./subdir) 2024-03-23T03:49:06.389 INFO:teuthology.orchestra.run.smithi102.stderr:setfattr: ./subdir: Invalid argument 2024-03-23T03:49:06.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:49:06.390 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T03:49:06.391 DEBUG:teuthology.orchestra.run.smithi102:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:49:06.491 DEBUG:teuthology.orchestra.run.smithi173:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:49:06.617 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T03:49:06.617 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:49:06.618 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:06.675 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:49:06.675 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:06.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:49:06.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.938+0000 7ff53ce00640 1 Processor -- start 2024-03-23T03:49:06.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.938+0000 7ff53ce00640 1 -- start start 2024-03-23T03:49:06.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.939+0000 7ff53ce00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff53813faa0 0x7ff538160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:06.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.940+0000 7ff53ce00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ff538160800 0x7ff5380b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:06.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.940+0000 7ff53ce00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5380b2030 con 0x7ff5380b1be0 2024-03-23T03:49:06.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.940+0000 7ff53ce00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5380b38a0 con 0x7ff538160800 2024-03-23T03:49:06.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.940+0000 7ff53ce00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5380b3b80 con 0x7ff53813faa0 2024-03-23T03:49:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.940+0000 7ff535e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff53813faa0 0x7ff538160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.941+0000 7ff52d400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ff538160800 0x7ff5380b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.941+0000 7ff536800640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7ff5380b1be0 0x7ff5381422b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:58978/0 (socket says 172.21.15.102:58978) 2024-03-23T03:49:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.941+0000 7ff536800640 1 -- 172.21.15.102:0/2486160275 learned_addr learned my addr 172.21.15.102:0/2486160275 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.941+0000 7ff535e00640 1 --2- 172.21.15.102:0/2486160275 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff53813faa0 0x7ff538160f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:50686/0 (socket says 172.21.15.102:50686) 2024-03-23T03:49:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.941+0000 7ff535e00640 1 -- 172.21.15.102:0/2486160275 >> v2:172.21.15.173:3300/0 conn(0x7ff538160800 msgr2=0x7ff5380b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.941+0000 7ff535e00640 1 --2- 172.21.15.102:0/2486160275 >> v2:172.21.15.173:3300/0 conn(0x7ff538160800 0x7ff5380b16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.941+0000 7ff535e00640 1 -- 172.21.15.102:0/2486160275 >> v1:172.21.15.173:6789/0 conn(0x7ff5380b1be0 legacy=0x7ff5381422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:06.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.941+0000 7ff535e00640 1 -- 172.21.15.102:0/2486160275 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5380b3e60 con 0x7ff53813faa0 2024-03-23T03:49:06.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.943+0000 7ff535e00640 1 --2- 172.21.15.102:0/2486160275 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff53813faa0 0x7ff538160f30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff5280097a0 tx=0x7ff52802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95480dcb05e54259 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:06.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.943+0000 7ff535400640 1 -- 172.21.15.102:0/2486160275 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528031830 con 0x7ff53813faa0 2024-03-23T03:49:06.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.943+0000 7ff535400640 1 -- 172.21.15.102:0/2486160275 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff528031990 con 0x7ff53813faa0 2024-03-23T03:49:06.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.943+0000 7ff535400640 1 -- 172.21.15.102:0/2486160275 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff528031d20 con 0x7ff53813faa0 2024-03-23T03:49:06.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.944+0000 7ff53ce00640 1 -- 172.21.15.102:0/2486160275 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff53813faa0 msgr2=0x7ff538160f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:06.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.944+0000 7ff53ce00640 1 --2- 172.21.15.102:0/2486160275 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff53813faa0 0x7ff538160f30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff5280097a0 tx=0x7ff52802b8d0 comp rx=0 tx=0).stop 2024-03-23T03:49:06.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.944+0000 7ff53ce00640 1 -- 172.21.15.102:0/2486160275 shutdown_connections 2024-03-23T03:49:06.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.944+0000 7ff53ce00640 1 --2- 172.21.15.102:0/2486160275 >> v2:172.21.15.173:3300/0 conn(0x7ff538160800 0x7ff5380b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:06.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.944+0000 7ff53ce00640 1 --2- 172.21.15.102:0/2486160275 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff53813faa0 0x7ff538160f30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:06.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.944+0000 7ff53ce00640 1 -- 172.21.15.102:0/2486160275 >> 172.21.15.102:0/2486160275 conn(0x7ff53806a3d0 msgr2=0x7ff5380b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:06.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.944+0000 7ff53ce00640 1 -- 172.21.15.102:0/2486160275 shutdown_connections 2024-03-23T03:49:06.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.944+0000 7ff53ce00640 1 -- 172.21.15.102:0/2486160275 wait complete. 2024-03-23T03:49:06.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.946+0000 7ff53ce00640 1 Processor -- start 2024-03-23T03:49:06.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.946+0000 7ff53ce00640 1 -- start start 2024-03-23T03:49:06.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.947+0000 7ff53ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5380b1be0 0x7ff538152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:06.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.947+0000 7ff53ce00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff53813faa0 0x7ff5380cb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:06.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.947+0000 7ff53ce00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff538160800 0x7ff53806cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.947+0000 7ff53ce00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff538153410 con 0x7ff5380b1be0 2024-03-23T03:49:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.947+0000 7ff53ce00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff538155800 con 0x7ff53813faa0 2024-03-23T03:49:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.947+0000 7ff53ce00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff538155a30 con 0x7ff538160800 2024-03-23T03:49:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.947+0000 7ff535e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5380b1be0 0x7ff538152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.947+0000 7ff535e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5380b1be0 0x7ff538152ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53670/0 (socket says 172.21.15.102:53670) 2024-03-23T03:49:06.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.947+0000 7ff535e00640 1 -- 172.21.15.102:0/185189293 learned_addr learned my addr 172.21.15.102:0/185189293 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.947+0000 7ff535e00640 1 -- 172.21.15.102:0/185189293 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff538160800 msgr2=0x7ff53806cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.947+0000 7ff535e00640 1 --2- 172.21.15.102:0/185189293 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff538160800 0x7ff53806cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.947+0000 7ff535e00640 1 -- 172.21.15.102:0/185189293 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff53813faa0 msgr2=0x7ff5380cb050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.947+0000 7ff535e00640 1 --2- 172.21.15.102:0/185189293 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff53813faa0 0x7ff5380cb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.947+0000 7ff535e00640 1 -- 172.21.15.102:0/185189293 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff528009400 con 0x7ff5380b1be0 2024-03-23T03:49:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.948+0000 7ff535e00640 1 --2- 172.21.15.102:0/185189293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5380b1be0 0x7ff538152ed0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff52802bee0 tx=0x7ff5280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.948+0000 7ff52ea00640 1 -- 172.21.15.102:0/185189293 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff52803b9f0 con 0x7ff5380b1be0 2024-03-23T03:49:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.948+0000 7ff52ea00640 1 -- 172.21.15.102:0/185189293 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff52803bb70 con 0x7ff5380b1be0 2024-03-23T03:49:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.948+0000 7ff53ce00640 1 -- 172.21.15.102:0/185189293 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff538155d10 con 0x7ff5380b1be0 2024-03-23T03:49:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.948+0000 7ff52ea00640 1 -- 172.21.15.102:0/185189293 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5280453f0 con 0x7ff5380b1be0 2024-03-23T03:49:06.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.948+0000 7ff53ce00640 1 -- 172.21.15.102:0/185189293 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5381562d0 con 0x7ff5380b1be0 2024-03-23T03:49:06.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.949+0000 7ff53ce00640 1 -- 172.21.15.102:0/185189293 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5381408a0 con 0x7ff5380b1be0 2024-03-23T03:49:06.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.950+0000 7ff52ea00640 1 -- 172.21.15.102:0/185189293 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff52804e450 con 0x7ff5380b1be0 2024-03-23T03:49:06.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.951+0000 7ff52ea00640 1 --2- 172.21.15.102:0/185189293 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff51406ac00 0x7ff51406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:06.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.952+0000 7ff52ea00640 1 -- 172.21.15.102:0/185189293 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff5280ab730 con 0x7ff5380b1be0 2024-03-23T03:49:06.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.952+0000 7ff52d400640 1 --2- 172.21.15.102:0/185189293 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff51406ac00 0x7ff51406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:06.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.952+0000 7ff52d400640 1 --2- 172.21.15.102:0/185189293 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff51406ac00 0x7ff51406d0c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff520006a90 tx=0x7ff520006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:06.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:06.956+0000 7ff52ea00640 1 -- 172.21.15.102:0/185189293 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff5280766a0 con 0x7ff5380b1be0 2024-03-23T03:49:07.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.154+0000 7ff53ce00640 1 -- 172.21.15.102:0/185189293 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff538073390 con 0x7ff5380b1be0 2024-03-23T03:49:07.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.157+0000 7ff52ea00640 1 -- 172.21.15.102:0/185189293 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16354 (secure 0 0 0) 0x7ff528035090 con 0x7ff5380b1be0 2024-03-23T03:49:07.158 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:07.158 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":55,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:57.226138+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:48:53.998390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T03:48:55.242109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:49:07.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.159+0000 7ff53ce00640 1 -- 172.21.15.102:0/185189293 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff51406ac00 msgr2=0x7ff51406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:07.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.159+0000 7ff53ce00640 1 --2- 172.21.15.102:0/185189293 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff51406ac00 0x7ff51406d0c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff520006a90 tx=0x7ff520006a20 comp rx=0 tx=0).stop 2024-03-23T03:49:07.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.159+0000 7ff53ce00640 1 -- 172.21.15.102:0/185189293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5380b1be0 msgr2=0x7ff538152ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.159+0000 7ff53ce00640 1 --2- 172.21.15.102:0/185189293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5380b1be0 0x7ff538152ed0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff52802bee0 tx=0x7ff5280046c0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.160+0000 7ff53ce00640 1 -- 172.21.15.102:0/185189293 shutdown_connections 2024-03-23T03:49:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.160+0000 7ff53ce00640 1 --2- 172.21.15.102:0/185189293 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff51406ac00 0x7ff51406d0c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.160+0000 7ff53ce00640 1 --2- 172.21.15.102:0/185189293 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff538160800 0x7ff53806cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.160+0000 7ff53ce00640 1 --2- 172.21.15.102:0/185189293 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff53813faa0 0x7ff5380cb050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.160+0000 7ff53ce00640 1 --2- 172.21.15.102:0/185189293 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5380b1be0 0x7ff538152ed0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.160+0000 7ff53ce00640 1 -- 172.21.15.102:0/185189293 >> 172.21.15.102:0/185189293 conn(0x7ff53806a3d0 msgr2=0x7ff5380ae310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.160+0000 7ff53ce00640 1 -- 172.21.15.102:0/185189293 shutdown_connections 2024-03-23T03:49:07.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.160+0000 7ff53ce00640 1 -- 172.21.15.102:0/185189293 wait complete. 2024-03-23T03:49:07.180 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:48:53.998390+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:48:55.242109+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:49:07.180 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:49:07.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.383+0000 7f8f81e00640 1 Processor -- start 2024-03-23T03:49:07.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.383+0000 7f8f81e00640 1 -- start start 2024-03-23T03:49:07.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.385+0000 7f8f81e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f8f7c15ad10 0x7f8f7c0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:07.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.385+0000 7f8f81e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f7c0b2ad0 con 0x7f8f7c0ae830 2024-03-23T03:49:07.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.385+0000 7f8f81e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f7c0b2d00 con 0x7f8f7c0b38b0 2024-03-23T03:49:07.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.385+0000 7f8f81e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f7c15d5b0 con 0x7f8f7c15ad10 2024-03-23T03:49:07.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.385+0000 7f8f7a000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8f7c0ae830 0x7f8f7c15d9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57240/0 (socket says 172.21.15.102:57240) 2024-03-23T03:49:07.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.385+0000 7f8f7a000640 1 -- 172.21.15.102:0/4183511663 learned_addr learned my addr 172.21.15.102:0/4183511663 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:07.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.386+0000 7f8f79600640 1 -- 172.21.15.102:0/4183511663 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128533373 0 0) 0x7f8f7c0b2ad0 con 0x7f8f7c0ae830 2024-03-23T03:49:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.386+0000 7f8f79600640 1 -- 172.21.15.102:0/4183511663 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f60003520 con 0x7f8f7c0ae830 2024-03-23T03:49:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.386+0000 7f8f7aa00640 1 --2- 172.21.15.102:0/4183511663 >> v2:172.21.15.173:3300/0 conn(0x7f8f7c15ad10 0x7f8f7c0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.386+0000 7f8f79600640 1 -- 172.21.15.102:0/4183511663 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 780239102 0 0) 0x7f8f60003520 con 0x7f8f7c0ae830 2024-03-23T03:49:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.386+0000 7f8f79600640 1 -- 172.21.15.102:0/4183511663 >> v2:172.21.15.173:3300/0 conn(0x7f8f7c15ad10 msgr2=0x7f8f7c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.386+0000 7f8f79600640 1 --2- 172.21.15.102:0/4183511663 >> v2:172.21.15.173:3300/0 conn(0x7f8f7c15ad10 0x7f8f7c0b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.386+0000 7f8f79600640 1 -- 172.21.15.102:0/4183511663 >> v1:172.21.15.173:6789/0 conn(0x7f8f7c0b38b0 legacy=0x7f8f7c0b1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.386+0000 7f8f79600640 1 -- 172.21.15.102:0/4183511663 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c0b2fe0 con 0x7f8f7c0ae830 2024-03-23T03:49:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.386+0000 7f8f79600640 1 -- 172.21.15.102:0/4183511663 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f8f68001670 con 0x7f8f7c0ae830 2024-03-23T03:49:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.386+0000 7f8f79600640 1 -- 172.21.15.102:0/4183511663 >> v1:172.21.15.102:6789/0 conn(0x7f8f7c0ae830 legacy=0x7f8f7c15d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.386+0000 7f8f79600640 1 --2- 172.21.15.102:0/4183511663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f600039c0 0x7f8f60003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.386+0000 7f8f79600640 1 --2- 172.21.15.102:0/4183511663 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8f600047c0 0x7f8f60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.386+0000 7f8f79600640 1 --2- 172.21.15.102:0/4183511663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f600051d0 0x7f8f6000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.386+0000 7f8f79600640 1 -- 172.21.15.102:0/4183511663 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c15d5b0 con 0x7f8f600051d0 2024-03-23T03:49:07.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.386+0000 7f8f79600640 1 -- 172.21.15.102:0/4183511663 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6000c100 con 0x7f8f600039c0 2024-03-23T03:49:07.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.386+0000 7f8f79600640 1 -- 172.21.15.102:0/4183511663 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6000c2c0 con 0x7f8f600047c0 2024-03-23T03:49:07.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.386+0000 7f8f7b400640 1 --2- 172.21.15.102:0/4183511663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f600051d0 0x7f8f6000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:07.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.387+0000 7f8f7a000640 1 --2- 172.21.15.102:0/4183511663 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8f600047c0 0x7f8f60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:07.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.387+0000 7f8f7aa00640 1 --2- 172.21.15.102:0/4183511663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f600039c0 0x7f8f60003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:07.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.387+0000 7f8f7b400640 1 -- 172.21.15.102:0/4183511663 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8f600047c0 msgr2=0x7f8f60004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:07.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.387+0000 7f8f7b400640 1 --2- 172.21.15.102:0/4183511663 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8f600047c0 0x7f8f60004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.387+0000 7f8f7b400640 1 -- 172.21.15.102:0/4183511663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f600039c0 msgr2=0x7f8f60003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:07.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.387+0000 7f8f7b400640 1 --2- 172.21.15.102:0/4183511663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f600039c0 0x7f8f60003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.387+0000 7f8f7b400640 1 -- 172.21.15.102:0/4183511663 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f60003520 con 0x7f8f600051d0 2024-03-23T03:49:07.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.387+0000 7f8f7aa00640 1 --2- 172.21.15.102:0/4183511663 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f600039c0 0x7f8f60003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:07.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.387+0000 7f8f7a000640 1 --2- 172.21.15.102:0/4183511663 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8f600047c0 0x7f8f60004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:07.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.388+0000 7f8f7b400640 1 -- 172.21.15.102:0/4183511663 reap_dead start 2024-03-23T03:49:07.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.388+0000 7f8f7b400640 1 --2- 172.21.15.102:0/4183511663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f600051d0 0x7f8f6000b980 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8f74004480 tx=0x7f8f600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a1fe25f3e1065af server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:07.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.388+0000 7f8f79600640 1 -- 172.21.15.102:0/4183511663 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74006b50 con 0x7f8f600051d0 2024-03-23T03:49:07.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.388+0000 7f8f79600640 1 -- 172.21.15.102:0/4183511663 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f74006cd0 con 0x7f8f600051d0 2024-03-23T03:49:07.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.388+0000 7f8f79600640 1 -- 172.21.15.102:0/4183511663 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74007260 con 0x7f8f600051d0 2024-03-23T03:49:07.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.389+0000 7f8f81e00640 1 -- 172.21.15.102:0/4183511663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f600051d0 msgr2=0x7f8f6000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:07.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.389+0000 7f8f81e00640 1 --2- 172.21.15.102:0/4183511663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f600051d0 0x7f8f6000b980 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8f74004480 tx=0x7f8f600057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.389+0000 7f8f81e00640 1 -- 172.21.15.102:0/4183511663 shutdown_connections 2024-03-23T03:49:07.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.389+0000 7f8f81e00640 1 --2- 172.21.15.102:0/4183511663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f600051d0 0x7f8f6000b980 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.389+0000 7f8f81e00640 1 -- 172.21.15.102:0/4183511663 >> 172.21.15.102:0/4183511663 conn(0x7f8f7c0b4c90 msgr2=0x7f8f7c0b50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:07.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.389+0000 7f8f81e00640 1 -- 172.21.15.102:0/4183511663 shutdown_connections 2024-03-23T03:49:07.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.389+0000 7f8f81e00640 1 -- 172.21.15.102:0/4183511663 wait complete. 2024-03-23T03:49:07.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.391+0000 7f8f81e00640 1 Processor -- start 2024-03-23T03:49:07.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.391+0000 7f8f81e00640 1 -- start start 2024-03-23T03:49:07.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.392+0000 7f8f81e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f600051d0 0x7f8f7c155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:07.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.392+0000 7f8f81e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8f7c1e3110 0x7f8f7c1e3590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:07.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.392+0000 7f8f81e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f7c15c630 0x7f8f7c15cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:07.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.392+0000 7f8f81e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c15cff0 con 0x7f8f7c15c630 2024-03-23T03:49:07.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.392+0000 7f8f81e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c15b360 con 0x7f8f600051d0 2024-03-23T03:49:07.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.392+0000 7f8f81e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f7c15b600 con 0x7f8f7c1e3110 2024-03-23T03:49:07.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.392+0000 7f8f7b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f7c15c630 0x7f8f7c15cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:07.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.392+0000 7f8f7b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f7c15c630 0x7f8f7c15cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53700/0 (socket says 172.21.15.102:53700) 2024-03-23T03:49:07.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.392+0000 7f8f7b400640 1 -- 172.21.15.102:0/3685373403 learned_addr learned my addr 172.21.15.102:0/3685373403 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:07.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.392+0000 7f8f7a000640 1 --2- 172.21.15.102:0/3685373403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8f7c1e3110 0x7f8f7c1e3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:07.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.392+0000 7f8f7b400640 1 -- 172.21.15.102:0/3685373403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8f7c1e3110 msgr2=0x7f8f7c1e3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:07.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.392+0000 7f8f7b400640 1 --2- 172.21.15.102:0/3685373403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8f7c1e3110 0x7f8f7c1e3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.392+0000 7f8f7b400640 1 -- 172.21.15.102:0/3685373403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f600051d0 msgr2=0x7f8f7c155f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:07.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.392+0000 7f8f7b400640 1 --2- 172.21.15.102:0/3685373403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f600051d0 0x7f8f7c155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.392+0000 7f8f7b400640 1 -- 172.21.15.102:0/3685373403 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7c15b8e0 con 0x7f8f7c15c630 2024-03-23T03:49:07.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.392+0000 7f8f7a000640 1 --2- 172.21.15.102:0/3685373403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8f7c1e3110 0x7f8f7c1e3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:07.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.393+0000 7f8f7b400640 1 --2- 172.21.15.102:0/3685373403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f7c15c630 0x7f8f7c15cab0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8f74004480 tx=0x7f8f74003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:07.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.393+0000 7f8f73400640 1 -- 172.21.15.102:0/3685373403 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7403d070 con 0x7f8f7c15c630 2024-03-23T03:49:07.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.393+0000 7f8f73400640 1 -- 172.21.15.102:0/3685373403 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f74007df0 con 0x7f8f7c15c630 2024-03-23T03:49:07.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.393+0000 7f8f81e00640 1 -- 172.21.15.102:0/3685373403 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f7c15bbc0 con 0x7f8f7c15c630 2024-03-23T03:49:07.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.393+0000 7f8f73400640 1 -- 172.21.15.102:0/3685373403 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74006e40 con 0x7f8f7c15c630 2024-03-23T03:49:07.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.393+0000 7f8f81e00640 1 -- 172.21.15.102:0/3685373403 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f7c1564d0 con 0x7f8f7c15c630 2024-03-23T03:49:07.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.394+0000 7f8f81e00640 1 -- 172.21.15.102:0/3685373403 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f7c066bb0 con 0x7f8f7c15c630 2024-03-23T03:49:07.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.395+0000 7f8f73400640 1 -- 172.21.15.102:0/3685373403 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8f74041070 con 0x7f8f7c15c630 2024-03-23T03:49:07.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.396+0000 7f8f73400640 1 --2- 172.21.15.102:0/3685373403 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8f5c06ac30 0x7f8f5c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:07.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.396+0000 7f8f73400640 1 -- 172.21.15.102:0/3685373403 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8f740aa620 con 0x7f8f7c15c630 2024-03-23T03:49:07.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.396+0000 7f8f7aa00640 1 --2- 172.21.15.102:0/3685373403 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8f5c06ac30 0x7f8f5c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:07.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.397+0000 7f8f7aa00640 1 --2- 172.21.15.102:0/3685373403 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8f5c06ac30 0x7f8f5c06d0f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c1e76e0 tx=0x7f8f6c009350 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:07.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.401+0000 7f8f73400640 1 -- 172.21.15.102:0/3685373403 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8f74075740 con 0x7f8f7c15c630 2024-03-23T03:49:07.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.591+0000 7f8f81e00640 1 -- 172.21.15.102:0/3685373403 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8f7c071c80 con 0x7f8f7c15c630 2024-03-23T03:49:07.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.592+0000 7f8f73400640 1 -- 172.21.15.102:0/3685373403 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8f7402d020 con 0x7f8f7c15c630 2024-03-23T03:49:07.593 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:49:07.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.595+0000 7f8f81e00640 1 -- 172.21.15.102:0/3685373403 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8f5c06ac30 msgr2=0x7f8f5c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:07.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.595+0000 7f8f81e00640 1 --2- 172.21.15.102:0/3685373403 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8f5c06ac30 0x7f8f5c06d0f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c1e76e0 tx=0x7f8f6c009350 comp rx=0 tx=0).stop 2024-03-23T03:49:07.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.595+0000 7f8f81e00640 1 -- 172.21.15.102:0/3685373403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f7c15c630 msgr2=0x7f8f7c15cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:07.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.595+0000 7f8f81e00640 1 --2- 172.21.15.102:0/3685373403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f7c15c630 0x7f8f7c15cab0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8f74004480 tx=0x7f8f74003a70 comp rx=0 tx=0).stop 2024-03-23T03:49:07.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.595+0000 7f8f81e00640 1 -- 172.21.15.102:0/3685373403 shutdown_connections 2024-03-23T03:49:07.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.595+0000 7f8f81e00640 1 --2- 172.21.15.102:0/3685373403 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8f5c06ac30 0x7f8f5c06d0f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.595+0000 7f8f81e00640 1 --2- 172.21.15.102:0/3685373403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f7c15c630 0x7f8f7c15cab0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.595+0000 7f8f81e00640 1 --2- 172.21.15.102:0/3685373403 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8f7c1e3110 0x7f8f7c1e3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.595+0000 7f8f81e00640 1 --2- 172.21.15.102:0/3685373403 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8f600051d0 0x7f8f7c155f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.595+0000 7f8f81e00640 1 -- 172.21.15.102:0/3685373403 >> 172.21.15.102:0/3685373403 conn(0x7f8f7c0b4c90 msgr2=0x7f8f7c0ade20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:07.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.595+0000 7f8f81e00640 1 -- 172.21.15.102:0/3685373403 shutdown_connections 2024-03-23T03:49:07.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.595+0000 7f8f81e00640 1 -- 172.21.15.102:0/3685373403 wait complete. 2024-03-23T03:49:07.615 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:49:07.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.826+0000 7efdb3600640 1 Processor -- start 2024-03-23T03:49:07.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.826+0000 7efdb3600640 1 -- start start 2024-03-23T03:49:07.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.828+0000 7efdb3600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdac0ce930 0x7efdac0cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:07.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.828+0000 7efdb3600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdac159c10 con 0x7efdac159830 2024-03-23T03:49:07.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.828+0000 7efdb3600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdac0cd840 con 0x7efdac0ca7b0 2024-03-23T03:49:07.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.828+0000 7efdb3600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdac06de50 con 0x7efdac0ce930 2024-03-23T03:49:07.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.828+0000 7efdb0c00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7efdac159830 0x7efdac0cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57250/0 (socket says 172.21.15.102:57250) 2024-03-23T03:49:07.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.828+0000 7efdb0c00640 1 -- 172.21.15.102:0/823522018 learned_addr learned my addr 172.21.15.102:0/823522018 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:07.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.828+0000 7efdab400640 1 -- 172.21.15.102:0/823522018 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685610558 0 0) 0x7efdac159c10 con 0x7efdac159830 2024-03-23T03:49:07.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.828+0000 7efdab400640 1 -- 172.21.15.102:0/823522018 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd90003520 con 0x7efdac159830 2024-03-23T03:49:07.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.828+0000 7efdb1600640 1 --2- 172.21.15.102:0/823522018 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdac0ce930 0x7efdac0cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:07.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.829+0000 7efdab400640 1 -- 172.21.15.102:0/823522018 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3924760799 0 0) 0x7efd90003520 con 0x7efdac159830 2024-03-23T03:49:07.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.829+0000 7efdab400640 1 -- 172.21.15.102:0/823522018 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdac0ce930 msgr2=0x7efdac0cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:07.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.829+0000 7efdab400640 1 --2- 172.21.15.102:0/823522018 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdac0ce930 0x7efdac0cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.829+0000 7efdab400640 1 -- 172.21.15.102:0/823522018 >> v1:172.21.15.173:6789/0 conn(0x7efdac0ca7b0 legacy=0x7efdac0cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:07.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.829+0000 7efdab400640 1 -- 172.21.15.102:0/823522018 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdac0ccc30 con 0x7efdac159830 2024-03-23T03:49:07.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.829+0000 7efdab400640 1 -- 172.21.15.102:0/823522018 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7efd9c0019f0 con 0x7efdac159830 2024-03-23T03:49:07.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.829+0000 7efdab400640 1 -- 172.21.15.102:0/823522018 >> v1:172.21.15.102:6789/0 conn(0x7efdac159830 legacy=0x7efdac0cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:07.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.829+0000 7efdab400640 1 --2- 172.21.15.102:0/823522018 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdac0ce930 0x7efd900039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:07.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.829+0000 7efdab400640 1 --2- 172.21.15.102:0/823522018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd90003f70 0x7efd900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:07.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.829+0000 7efdab400640 1 --2- 172.21.15.102:0/823522018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd90004d00 0x7efd90005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:07.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.829+0000 7efdab400640 1 -- 172.21.15.102:0/823522018 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9000bb20 con 0x7efd90003f70 2024-03-23T03:49:07.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.829+0000 7efdab400640 1 -- 172.21.15.102:0/823522018 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd9000bc90 con 0x7efd90004d00 2024-03-23T03:49:07.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.829+0000 7efdab400640 1 -- 172.21.15.102:0/823522018 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd9000be50 con 0x7efdac0ce930 2024-03-23T03:49:07.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.829+0000 7efdabe00640 1 --2- 172.21.15.102:0/823522018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd90003f70 0x7efd900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:07.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.830+0000 7efdabe00640 1 -- 172.21.15.102:0/823522018 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdac0ce930 msgr2=0x7efd900039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:07.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.830+0000 7efdabe00640 1 --2- 172.21.15.102:0/823522018 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdac0ce930 0x7efd900039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.830+0000 7efdabe00640 1 -- 172.21.15.102:0/823522018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd90004d00 msgr2=0x7efd90005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:07.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.830+0000 7efdb1600640 1 --2- 172.21.15.102:0/823522018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd90004d00 0x7efd90005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:07.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.830+0000 7efdabe00640 1 --2- 172.21.15.102:0/823522018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd90004d00 0x7efd90005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.830+0000 7efdabe00640 1 -- 172.21.15.102:0/823522018 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd90003520 con 0x7efd90003f70 2024-03-23T03:49:07.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.830+0000 7efdb1600640 1 --2- 172.21.15.102:0/823522018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd90004d00 0x7efd90005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:07.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.831+0000 7efdabe00640 1 --2- 172.21.15.102:0/823522018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd90003f70 0x7efd900047c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7efd98001750 tx=0x7efd900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a447445bf562a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:07.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.831+0000 7efdab400640 1 -- 172.21.15.102:0/823522018 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd980052d0 con 0x7efd90003f70 2024-03-23T03:49:07.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.831+0000 7efdab400640 1 -- 172.21.15.102:0/823522018 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9802e050 con 0x7efd90003f70 2024-03-23T03:49:07.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.831+0000 7efdab400640 1 -- 172.21.15.102:0/823522018 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd98003830 con 0x7efd90003f70 2024-03-23T03:49:07.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.832+0000 7efdb3600640 1 -- 172.21.15.102:0/823522018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd90003f70 msgr2=0x7efd900047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:07.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.832+0000 7efdb3600640 1 --2- 172.21.15.102:0/823522018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd90003f70 0x7efd900047c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7efd98001750 tx=0x7efd900057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.832+0000 7efdb1600640 1 -- 172.21.15.102:0/823522018 reap_dead start 2024-03-23T03:49:07.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.832+0000 7efdb3600640 1 -- 172.21.15.102:0/823522018 shutdown_connections 2024-03-23T03:49:07.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.832+0000 7efdb3600640 1 -- 172.21.15.102:0/823522018 >> 172.21.15.102:0/823522018 conn(0x7efdac06a0b0 msgr2=0x7efdac06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:07.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.832+0000 7efdb3600640 1 -- 172.21.15.102:0/823522018 shutdown_connections 2024-03-23T03:49:07.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.832+0000 7efdb3600640 1 -- 172.21.15.102:0/823522018 wait complete. 2024-03-23T03:49:07.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.834+0000 7efdb3600640 1 Processor -- start 2024-03-23T03:49:07.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.834+0000 7efdb3600640 1 -- start start 2024-03-23T03:49:07.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.834+0000 7efdb3600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd90003f70 0x7efdac0aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:07.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.835+0000 7efdb3600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdac0af650 0x7efdac158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:07.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.835+0000 7efdb3600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdac1485e0 0x7efdac148a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:07.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.835+0000 7efdb3600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdac14a220 con 0x7efdac1485e0 2024-03-23T03:49:07.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.835+0000 7efdb3600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdac0af260 con 0x7efd90003f70 2024-03-23T03:49:07.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.835+0000 7efdb3600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdac0afb10 con 0x7efdac0af650 2024-03-23T03:49:07.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.835+0000 7efdb1600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdac1485e0 0x7efdac148a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:07.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.835+0000 7efdb1600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdac1485e0 0x7efdac148a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53714/0 (socket says 172.21.15.102:53714) 2024-03-23T03:49:07.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.835+0000 7efdb1600640 1 -- 172.21.15.102:0/3031203502 learned_addr learned my addr 172.21.15.102:0/3031203502 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:07.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.835+0000 7efdb0c00640 1 --2- 172.21.15.102:0/3031203502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd90003f70 0x7efdac0aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:07.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.835+0000 7efdabe00640 1 --2- 172.21.15.102:0/3031203502 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdac0af650 0x7efdac158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:07.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.836+0000 7efdb1600640 1 -- 172.21.15.102:0/3031203502 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdac0af650 msgr2=0x7efdac158140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:07.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.836+0000 7efdb1600640 1 --2- 172.21.15.102:0/3031203502 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdac0af650 0x7efdac158140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.836+0000 7efdb1600640 1 -- 172.21.15.102:0/3031203502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd90003f70 msgr2=0x7efdac0aed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:07.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.836+0000 7efdb1600640 1 --2- 172.21.15.102:0/3031203502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd90003f70 0x7efdac0aed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:07.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.836+0000 7efdb1600640 1 -- 172.21.15.102:0/3031203502 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdac0afdf0 con 0x7efdac1485e0 2024-03-23T03:49:07.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.836+0000 7efdb1600640 1 --2- 172.21.15.102:0/3031203502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdac1485e0 0x7efdac148a90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7efdac06da60 tx=0x7efda4010a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:07.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.836+0000 7efda8c00640 1 -- 172.21.15.102:0/3031203502 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda4014800 con 0x7efdac1485e0 2024-03-23T03:49:07.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.836+0000 7efda8c00640 1 -- 172.21.15.102:0/3031203502 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda4014960 con 0x7efdac1485e0 2024-03-23T03:49:07.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.836+0000 7efda8c00640 1 -- 172.21.15.102:0/3031203502 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda4014cf0 con 0x7efdac1485e0 2024-03-23T03:49:07.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.836+0000 7efdb3600640 1 -- 172.21.15.102:0/3031203502 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdac0b00d0 con 0x7efdac1485e0 2024-03-23T03:49:07.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.837+0000 7efdb3600640 1 -- 172.21.15.102:0/3031203502 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdac0b0690 con 0x7efdac1485e0 2024-03-23T03:49:07.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.838+0000 7efdb3600640 1 -- 172.21.15.102:0/3031203502 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdac073390 con 0x7efdac1485e0 2024-03-23T03:49:07.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.839+0000 7efda8c00640 1 -- 172.21.15.102:0/3031203502 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7efda4012390 con 0x7efdac1485e0 2024-03-23T03:49:07.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.840+0000 7efda8c00640 1 --2- 172.21.15.102:0/3031203502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efd8c06ac00 0x7efd8c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:07.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.840+0000 7efda8c00640 1 -- 172.21.15.102:0/3031203502 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7efda40892b0 con 0x7efdac1485e0 2024-03-23T03:49:07.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.840+0000 7efdb0c00640 1 --2- 172.21.15.102:0/3031203502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efd8c06ac00 0x7efd8c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:07.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.841+0000 7efdb0c00640 1 --2- 172.21.15.102:0/3031203502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efd8c06ac00 0x7efd8c06d0c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7efd9c001350 tx=0x7efd9c007c10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:07.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:07.844+0000 7efda8c00640 1 -- 172.21.15.102:0/3031203502 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efda40543f0 con 0x7efdac1485e0 2024-03-23T03:49:08.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.041+0000 7efdb3600640 1 -- 172.21.15.102:0/3031203502 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efdac0cb390 con 0x7efdac1485e0 2024-03-23T03:49:08.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.042+0000 7efda8c00640 1 -- 172.21.15.102:0/3031203502 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) v1 ==== 104+0+11 (secure 0 0 0) 0x7efda4057a10 con 0x7efdac1485e0 2024-03-23T03:49:08.042 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:49:08.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.044+0000 7efdb3600640 1 -- 172.21.15.102:0/3031203502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efd8c06ac00 msgr2=0x7efd8c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:08.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.044+0000 7efdb3600640 1 --2- 172.21.15.102:0/3031203502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efd8c06ac00 0x7efd8c06d0c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7efd9c001350 tx=0x7efd9c007c10 comp rx=0 tx=0).stop 2024-03-23T03:49:08.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.044+0000 7efdb3600640 1 -- 172.21.15.102:0/3031203502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdac1485e0 msgr2=0x7efdac148a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:08.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.044+0000 7efdb3600640 1 --2- 172.21.15.102:0/3031203502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdac1485e0 0x7efdac148a90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7efdac06da60 tx=0x7efda4010a10 comp rx=0 tx=0).stop 2024-03-23T03:49:08.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.044+0000 7efdb3600640 1 -- 172.21.15.102:0/3031203502 shutdown_connections 2024-03-23T03:49:08.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.044+0000 7efdb3600640 1 --2- 172.21.15.102:0/3031203502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efd8c06ac00 0x7efd8c06d0c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.044+0000 7efdb3600640 1 --2- 172.21.15.102:0/3031203502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdac1485e0 0x7efdac148a90 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.044+0000 7efdb3600640 1 --2- 172.21.15.102:0/3031203502 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdac0af650 0x7efdac158140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.044+0000 7efdb3600640 1 --2- 172.21.15.102:0/3031203502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efd90003f70 0x7efdac0aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.044+0000 7efdb3600640 1 -- 172.21.15.102:0/3031203502 >> 172.21.15.102:0/3031203502 conn(0x7efdac06a0b0 msgr2=0x7efdac14aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:08.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.044+0000 7efdb3600640 1 -- 172.21.15.102:0/3031203502 shutdown_connections 2024-03-23T03:49:08.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.044+0000 7efdb3600640 1 -- 172.21.15.102:0/3031203502 wait complete. 2024-03-23T03:49:08.063 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:49:08.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.268+0000 7fec8ac00640 1 Processor -- start 2024-03-23T03:49:08.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.269+0000 7fec8ac00640 1 -- start start 2024-03-23T03:49:08.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.270+0000 7fec8ac00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fec840cd6b0 0x7fec8406baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:08.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.270+0000 7fec8ac00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fec840b1b60 0x7fec840d1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:08.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.270+0000 7fec8ac00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8406b0a0 con 0x7fec8415cd10 2024-03-23T03:49:08.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.270+0000 7fec8ac00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec840d2330 con 0x7fec840b1b60 2024-03-23T03:49:08.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.270+0000 7fec8ac00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec840d2610 con 0x7fec840cd6b0 2024-03-23T03:49:08.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.270+0000 7fec88c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fec840b1b60 0x7fec840d1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:08.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.270+0000 7fec88c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fec840b1b60 0x7fec840d1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53728/0 (socket says 172.21.15.102:53728) 2024-03-23T03:49:08.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.270+0000 7fec7b400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fec8415cd10 0x7fec840cda90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57252/0 (socket says 172.21.15.102:57252) 2024-03-23T03:49:08.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.270+0000 7fec88c00640 1 -- 172.21.15.102:0/1911961088 learned_addr learned my addr 172.21.15.102:0/1911961088 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:08.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.271+0000 7fec83e00640 1 --2- 172.21.15.102:0/1911961088 >> v2:172.21.15.173:3300/0 conn(0x7fec840cd6b0 0x7fec8406baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:08.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.271+0000 7fec88c00640 1 -- 172.21.15.102:0/1911961088 >> v2:172.21.15.173:3300/0 conn(0x7fec840cd6b0 msgr2=0x7fec8406baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:08.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.271+0000 7fec88c00640 1 --2- 172.21.15.102:0/1911961088 >> v2:172.21.15.173:3300/0 conn(0x7fec840cd6b0 0x7fec8406baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.271+0000 7fec88c00640 1 -- 172.21.15.102:0/1911961088 >> v1:172.21.15.102:6789/0 conn(0x7fec8415cd10 legacy=0x7fec840cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:08.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.271+0000 7fec88c00640 1 -- 172.21.15.102:0/1911961088 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8406b2f0 con 0x7fec840b1b60 2024-03-23T03:49:08.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.271+0000 7fec83e00640 1 --2- 172.21.15.102:0/1911961088 >> v2:172.21.15.173:3300/0 conn(0x7fec840cd6b0 0x7fec8406baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:08.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.272+0000 7fec88c00640 1 --2- 172.21.15.102:0/1911961088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec840b1b60 0x7fec840d1ae0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fec8408b520 tx=0x7fec7c02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6157f39ba516a98b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:08.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.272+0000 7fec83400640 1 -- 172.21.15.102:0/1911961088 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c02cb10 con 0x7fec840b1b60 2024-03-23T03:49:08.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.272+0000 7fec83400640 1 -- 172.21.15.102:0/1911961088 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7c02cc70 con 0x7fec840b1b60 2024-03-23T03:49:08.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.272+0000 7fec83400640 1 -- 172.21.15.102:0/1911961088 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c0343f0 con 0x7fec840b1b60 2024-03-23T03:49:08.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.273+0000 7fec8ac00640 1 -- 172.21.15.102:0/1911961088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec840b1b60 msgr2=0x7fec840d1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:08.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.273+0000 7fec8ac00640 1 --2- 172.21.15.102:0/1911961088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec840b1b60 0x7fec840d1ae0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fec8408b520 tx=0x7fec7c02bf70 comp rx=0 tx=0).stop 2024-03-23T03:49:08.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.273+0000 7fec8ac00640 1 -- 172.21.15.102:0/1911961088 shutdown_connections 2024-03-23T03:49:08.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.273+0000 7fec8ac00640 1 --2- 172.21.15.102:0/1911961088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec840b1b60 0x7fec840d1ae0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.273+0000 7fec8ac00640 1 --2- 172.21.15.102:0/1911961088 >> v2:172.21.15.173:3300/0 conn(0x7fec840cd6b0 0x7fec8406baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.273+0000 7fec8ac00640 1 -- 172.21.15.102:0/1911961088 >> 172.21.15.102:0/1911961088 conn(0x7fec840cf5d0 msgr2=0x7fec840cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:08.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.273+0000 7fec8ac00640 1 -- 172.21.15.102:0/1911961088 shutdown_connections 2024-03-23T03:49:08.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.273+0000 7fec8ac00640 1 -- 172.21.15.102:0/1911961088 wait complete. 2024-03-23T03:49:08.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.275+0000 7fec8ac00640 1 Processor -- start 2024-03-23T03:49:08.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.275+0000 7fec8ac00640 1 -- start start 2024-03-23T03:49:08.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.276+0000 7fec8ac00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec840b1b60 0x7fec840c8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:08.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.276+0000 7fec8ac00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec840cd6b0 0x7fec841f1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:08.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.276+0000 7fec8ac00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec8415cd10 0x7fec841f2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:08.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.276+0000 7fec8ac00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec841f25d0 con 0x7fec840cd6b0 2024-03-23T03:49:08.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.276+0000 7fec8ac00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec840cb050 con 0x7fec8415cd10 2024-03-23T03:49:08.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.276+0000 7fec8ac00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec840cb330 con 0x7fec840b1b60 2024-03-23T03:49:08.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.276+0000 7fec7b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec840cd6b0 0x7fec841f1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:08.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.276+0000 7fec7b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec840cd6b0 0x7fec841f1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53740/0 (socket says 172.21.15.102:53740) 2024-03-23T03:49:08.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.276+0000 7fec7b400640 1 -- 172.21.15.102:0/3906843470 learned_addr learned my addr 172.21.15.102:0/3906843470 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:08.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.277+0000 7fec88c00640 1 --2- 172.21.15.102:0/3906843470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec8415cd10 0x7fec841f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:08.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.277+0000 7fec83e00640 1 --2- 172.21.15.102:0/3906843470 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec840b1b60 0x7fec840c8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:08.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.277+0000 7fec7b400640 1 -- 172.21.15.102:0/3906843470 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec840b1b60 msgr2=0x7fec840c8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:08.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.277+0000 7fec7b400640 1 --2- 172.21.15.102:0/3906843470 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec840b1b60 0x7fec840c8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.277+0000 7fec7b400640 1 -- 172.21.15.102:0/3906843470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec8415cd10 msgr2=0x7fec841f2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:08.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.277+0000 7fec7b400640 1 --2- 172.21.15.102:0/3906843470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec8415cd10 0x7fec841f2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.277+0000 7fec7b400640 1 -- 172.21.15.102:0/3906843470 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c009b30 con 0x7fec840cd6b0 2024-03-23T03:49:08.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.277+0000 7fec7b400640 1 --2- 172.21.15.102:0/3906843470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec840cd6b0 0x7fec841f1b50 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00c3c0 tx=0x7fec6c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:08.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.277+0000 7fec80c00640 1 -- 172.21.15.102:0/3906843470 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c0074e0 con 0x7fec840cd6b0 2024-03-23T03:49:08.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.277+0000 7fec80c00640 1 -- 172.21.15.102:0/3906843470 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6c007640 con 0x7fec840cd6b0 2024-03-23T03:49:08.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.277+0000 7fec80c00640 1 -- 172.21.15.102:0/3906843470 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6c004a00 con 0x7fec840cd6b0 2024-03-23T03:49:08.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.278+0000 7fec8ac00640 1 -- 172.21.15.102:0/3906843470 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec840cb610 con 0x7fec840cd6b0 2024-03-23T03:49:08.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.278+0000 7fec8ac00640 1 -- 172.21.15.102:0/3906843470 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec840cbbd0 con 0x7fec840cd6b0 2024-03-23T03:49:08.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.279+0000 7fec8ac00640 1 -- 172.21.15.102:0/3906843470 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec840cbdc0 con 0x7fec840cd6b0 2024-03-23T03:49:08.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.280+0000 7fec80c00640 1 -- 172.21.15.102:0/3906843470 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fec6c004b60 con 0x7fec840cd6b0 2024-03-23T03:49:08.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.280+0000 7fec80c00640 1 --2- 172.21.15.102:0/3906843470 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fec6006ac00 0x7fec6006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:08.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.281+0000 7fec80c00640 1 -- 172.21.15.102:0/3906843470 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fec6c088da0 con 0x7fec840cd6b0 2024-03-23T03:49:08.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.281+0000 7fec83e00640 1 --2- 172.21.15.102:0/3906843470 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fec6006ac00 0x7fec6006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:08.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.281+0000 7fec83e00640 1 --2- 172.21.15.102:0/3906843470 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fec6006ac00 0x7fec6006d0c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fec7400a990 tx=0x7fec74008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:08.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.285+0000 7fec80c00640 1 -- 172.21.15.102:0/3906843470 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fec6c053ed0 con 0x7fec840cd6b0 2024-03-23T03:49:08.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.481+0000 7fec8ac00640 1 -- 172.21.15.102:0/3906843470 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fec84073390 con 0x7fec840cd6b0 2024-03-23T03:49:08.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.483+0000 7fec80c00640 1 -- 172.21.15.102:0/3906843470 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) v1 ==== 100+0+11 (secure 0 0 0) 0x7fec6c0574f0 con 0x7fec840cd6b0 2024-03-23T03:49:08.483 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:49:08.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.485+0000 7fec8ac00640 1 -- 172.21.15.102:0/3906843470 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fec6006ac00 msgr2=0x7fec6006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:08.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.485+0000 7fec8ac00640 1 --2- 172.21.15.102:0/3906843470 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fec6006ac00 0x7fec6006d0c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fec7400a990 tx=0x7fec74008040 comp rx=0 tx=0).stop 2024-03-23T03:49:08.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.485+0000 7fec8ac00640 1 -- 172.21.15.102:0/3906843470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec840cd6b0 msgr2=0x7fec841f1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:08.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.485+0000 7fec8ac00640 1 --2- 172.21.15.102:0/3906843470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec840cd6b0 0x7fec841f1b50 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00c3c0 tx=0x7fec6c00c880 comp rx=0 tx=0).stop 2024-03-23T03:49:08.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.485+0000 7fec8ac00640 1 -- 172.21.15.102:0/3906843470 shutdown_connections 2024-03-23T03:49:08.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.485+0000 7fec8ac00640 1 --2- 172.21.15.102:0/3906843470 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fec6006ac00 0x7fec6006d0c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.485+0000 7fec8ac00640 1 --2- 172.21.15.102:0/3906843470 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec8415cd10 0x7fec841f2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.485+0000 7fec8ac00640 1 --2- 172.21.15.102:0/3906843470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec840cd6b0 0x7fec841f1b50 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.485+0000 7fec8ac00640 1 --2- 172.21.15.102:0/3906843470 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec840b1b60 0x7fec840c8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.485+0000 7fec8ac00640 1 -- 172.21.15.102:0/3906843470 >> 172.21.15.102:0/3906843470 conn(0x7fec840cf5d0 msgr2=0x7fec8415adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:08.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.485+0000 7fec8ac00640 1 -- 172.21.15.102:0/3906843470 shutdown_connections 2024-03-23T03:49:08.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.485+0000 7fec8ac00640 1 -- 172.21.15.102:0/3906843470 wait complete. 2024-03-23T03:49:08.505 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:49:08.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.707+0000 7f0f2e200640 1 Processor -- start 2024-03-23T03:49:08.708 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.707+0000 7f0f2e200640 1 -- start start 2024-03-23T03:49:08.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.709+0000 7f0f2e200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f0f280ce930 0x7f0f280ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:08.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.709+0000 7f0f2e200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0f2806d2c0 0x7f0f280cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:08.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.709+0000 7f0f2e200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f280cf250 con 0x7f0f280cb660 2024-03-23T03:49:08.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.709+0000 7f0f2e200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f2815bae0 con 0x7f0f2806d2c0 2024-03-23T03:49:08.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.709+0000 7f0f2e200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f2815ae60 con 0x7f0f280ce930 2024-03-23T03:49:08.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.709+0000 7f0f27e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0f2806d2c0 0x7f0f280cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:08.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.709+0000 7f0f27e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0f2806d2c0 0x7f0f280cbae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53750/0 (socket says 172.21.15.102:53750) 2024-03-23T03:49:08.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.709+0000 7f0f27e00640 1 -- 172.21.15.102:0/1742866965 learned_addr learned my addr 172.21.15.102:0/1742866965 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:08.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.709+0000 7f0f27400640 1 --2- 172.21.15.102:0/1742866965 >> v2:172.21.15.173:3300/0 conn(0x7f0f280ce930 0x7f0f280ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:08.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.710+0000 7f0f27e00640 1 -- 172.21.15.102:0/1742866965 >> v2:172.21.15.173:3300/0 conn(0x7f0f280ce930 msgr2=0x7f0f280ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:08.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.710+0000 7f0f27e00640 1 --2- 172.21.15.102:0/1742866965 >> v2:172.21.15.173:3300/0 conn(0x7f0f280ce930 0x7f0f280ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.710+0000 7f0f27e00640 1 -- 172.21.15.102:0/1742866965 >> v1:172.21.15.173:6789/0 conn(0x7f0f280cb660 legacy=0x7f0f28070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:08.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.710+0000 7f0f27e00640 1 -- 172.21.15.102:0/1742866965 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2815b140 con 0x7f0f2806d2c0 2024-03-23T03:49:08.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.711+0000 7f0f27e00640 1 --2- 172.21.15.102:0/1742866965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f2806d2c0 0x7f0f280cbae0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0f280cf540 tx=0x7f0f2002b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76f921bf604a8223 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:08.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.711+0000 7f0f26a00640 1 -- 172.21.15.102:0/1742866965 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f20031890 con 0x7f0f2806d2c0 2024-03-23T03:49:08.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.712+0000 7f0f26a00640 1 -- 172.21.15.102:0/1742866965 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f200319f0 con 0x7f0f2806d2c0 2024-03-23T03:49:08.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.712+0000 7f0f26a00640 1 -- 172.21.15.102:0/1742866965 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f20031d80 con 0x7f0f2806d2c0 2024-03-23T03:49:08.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.712+0000 7f0f2e200640 1 -- 172.21.15.102:0/1742866965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f2806d2c0 msgr2=0x7f0f280cbae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.712+0000 7f0f2e200640 1 --2- 172.21.15.102:0/1742866965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f2806d2c0 0x7f0f280cbae0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0f280cf540 tx=0x7f0f2002b930 comp rx=0 tx=0).stop 2024-03-23T03:49:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.712+0000 7f0f2e200640 1 -- 172.21.15.102:0/1742866965 shutdown_connections 2024-03-23T03:49:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.712+0000 7f0f2e200640 1 --2- 172.21.15.102:0/1742866965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f2806d2c0 0x7f0f280cbae0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.712+0000 7f0f2e200640 1 --2- 172.21.15.102:0/1742866965 >> v2:172.21.15.173:3300/0 conn(0x7f0f280ce930 0x7f0f280ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.712+0000 7f0f2e200640 1 -- 172.21.15.102:0/1742866965 >> 172.21.15.102:0/1742866965 conn(0x7f0f2806a0b0 msgr2=0x7f0f2806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.713+0000 7f0f2e200640 1 -- 172.21.15.102:0/1742866965 shutdown_connections 2024-03-23T03:49:08.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.713+0000 7f0f2e200640 1 -- 172.21.15.102:0/1742866965 wait complete. 2024-03-23T03:49:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.714+0000 7f0f2e200640 1 Processor -- start 2024-03-23T03:49:08.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.714+0000 7f0f2e200640 1 -- start start 2024-03-23T03:49:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.715+0000 7f0f2e200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f2806d2c0 0x7f0f2815a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.715+0000 7f0f2e200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f280cb660 0x7f0f2814f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.715+0000 7f0f2e200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0f280ce930 0x7f0f280ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.715+0000 7f0f2e200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f28156950 con 0x7f0f280cb660 2024-03-23T03:49:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.715+0000 7f0f2e200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f280adc40 con 0x7f0f2806d2c0 2024-03-23T03:49:08.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.715+0000 7f0f2e200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f280b2a10 con 0x7f0f280ce930 2024-03-23T03:49:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.716+0000 7f0f1ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f280cb660 0x7f0f2814f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.716+0000 7f0f1ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f280cb660 0x7f0f2814f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53754/0 (socket says 172.21.15.102:53754) 2024-03-23T03:49:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.716+0000 7f0f1ea00640 1 -- 172.21.15.102:0/23375497 learned_addr learned my addr 172.21.15.102:0/23375497 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.716+0000 7f0f27e00640 1 --2- 172.21.15.102:0/23375497 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0f280ce930 0x7f0f280ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.716+0000 7f0f27400640 1 --2- 172.21.15.102:0/23375497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f2806d2c0 0x7f0f2815a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.716+0000 7f0f1ea00640 1 -- 172.21.15.102:0/23375497 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0f280ce930 msgr2=0x7f0f280ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.716+0000 7f0f1ea00640 1 --2- 172.21.15.102:0/23375497 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0f280ce930 0x7f0f280ab8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.716+0000 7f0f1ea00640 1 -- 172.21.15.102:0/23375497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f2806d2c0 msgr2=0x7f0f2815a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.716+0000 7f0f1ea00640 1 --2- 172.21.15.102:0/23375497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f2806d2c0 0x7f0f2815a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.716+0000 7f0f1ea00640 1 -- 172.21.15.102:0/23375497 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f200094d0 con 0x7f0f280cb660 2024-03-23T03:49:08.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.717+0000 7f0f1ea00640 1 --2- 172.21.15.102:0/23375497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f280cb660 0x7f0f2814f0b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f0f1000c460 tx=0x7f0f1000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:08.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.717+0000 7f0f1fe00640 1 -- 172.21.15.102:0/23375497 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f10012830 con 0x7f0f280cb660 2024-03-23T03:49:08.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.717+0000 7f0f1fe00640 1 -- 172.21.15.102:0/23375497 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f100129d0 con 0x7f0f280cb660 2024-03-23T03:49:08.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.717+0000 7f0f1fe00640 1 -- 172.21.15.102:0/23375497 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f10012d80 con 0x7f0f280cb660 2024-03-23T03:49:08.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.717+0000 7f0f2e200640 1 -- 172.21.15.102:0/23375497 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f28155280 con 0x7f0f280cb660 2024-03-23T03:49:08.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.717+0000 7f0f2e200640 1 -- 172.21.15.102:0/23375497 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f280b2730 con 0x7f0f280cb660 2024-03-23T03:49:08.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.718+0000 7f0f2e200640 1 -- 172.21.15.102:0/23375497 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f28156a90 con 0x7f0f280cb660 2024-03-23T03:49:08.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.719+0000 7f0f1fe00640 1 -- 172.21.15.102:0/23375497 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0f1001a3b0 con 0x7f0f280cb660 2024-03-23T03:49:08.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.720+0000 7f0f1fe00640 1 --2- 172.21.15.102:0/23375497 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0f0406ac00 0x7f0f0406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:08.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.720+0000 7f0f1fe00640 1 -- 172.21.15.102:0/23375497 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0f10088fe0 con 0x7f0f280cb660 2024-03-23T03:49:08.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.720+0000 7f0f27400640 1 --2- 172.21.15.102:0/23375497 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0f0406ac00 0x7f0f0406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:08.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.721+0000 7f0f27400640 1 --2- 172.21.15.102:0/23375497 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0f0406ac00 0x7f0f0406d0c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0f18003aa0 tx=0x7f0f18009830 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:08.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.724+0000 7f0f1fe00640 1 -- 172.21.15.102:0/23375497 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0f28156a90 con 0x7f0f280cb660 2024-03-23T03:49:08.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.925+0000 7f0f2e200640 1 -- 172.21.15.102:0/23375497 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0f28156a90 con 0x7f0f280cb660 2024-03-23T03:49:08.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.926+0000 7f0f1fe00640 1 -- 172.21.15.102:0/23375497 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v55) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0f28156a90 con 0x7f0f280cb660 2024-03-23T03:49:08.926 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:49:08.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.928+0000 7f0f2e200640 1 -- 172.21.15.102:0/23375497 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0f0406ac00 msgr2=0x7f0f0406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:08.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.928+0000 7f0f2e200640 1 --2- 172.21.15.102:0/23375497 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0f0406ac00 0x7f0f0406d0c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0f18003aa0 tx=0x7f0f18009830 comp rx=0 tx=0).stop 2024-03-23T03:49:08.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.928+0000 7f0f2e200640 1 -- 172.21.15.102:0/23375497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f280cb660 msgr2=0x7f0f2814f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:08.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.928+0000 7f0f2e200640 1 --2- 172.21.15.102:0/23375497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f280cb660 0x7f0f2814f0b0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f0f1000c460 tx=0x7f0f1000c920 comp rx=0 tx=0).stop 2024-03-23T03:49:08.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.928+0000 7f0f2e200640 1 -- 172.21.15.102:0/23375497 shutdown_connections 2024-03-23T03:49:08.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.928+0000 7f0f2e200640 1 --2- 172.21.15.102:0/23375497 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0f0406ac00 0x7f0f0406d0c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.928+0000 7f0f2e200640 1 --2- 172.21.15.102:0/23375497 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0f280ce930 0x7f0f280ab8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.928+0000 7f0f2e200640 1 --2- 172.21.15.102:0/23375497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f280cb660 0x7f0f2814f0b0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.928+0000 7f0f2e200640 1 --2- 172.21.15.102:0/23375497 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f2806d2c0 0x7f0f2815a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:08.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.928+0000 7f0f2e200640 1 -- 172.21.15.102:0/23375497 >> 172.21.15.102:0/23375497 conn(0x7f0f2806a0b0 msgr2=0x7f0f281523e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:08.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.928+0000 7f0f2e200640 1 -- 172.21.15.102:0/23375497 shutdown_connections 2024-03-23T03:49:08.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:08.928+0000 7f0f2e200640 1 -- 172.21.15.102:0/23375497 wait complete. 2024-03-23T03:49:08.945 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-23T03:49:08.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:08.946 DEBUG:teuthology.orchestra.run.smithi102:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-23T03:49:09.044 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:49:09.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:09.044 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T03:49:09.090 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:49:09.091 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:09.147 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:49:09.147 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:09.197 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:49:09.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.408+0000 7f8eef200640 1 Processor -- start 2024-03-23T03:49:09.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.408+0000 7f8eef200640 1 -- start start 2024-03-23T03:49:09.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.409+0000 7f8eef200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f8ee80cbab0 0x7f8ee80b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:09.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.410+0000 7f8eef200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee80b16a0 con 0x7f8ee80adbc0 2024-03-23T03:49:09.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.410+0000 7f8eef200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ee80cbfb0 con 0x7f8ee806d390 2024-03-23T03:49:09.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.410+0000 7f8eef200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ee80cc290 con 0x7f8ee80cbab0 2024-03-23T03:49:09.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.410+0000 7f8ee7e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8ee80adbc0 0x7f8ee80b41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57260/0 (socket says 172.21.15.102:57260) 2024-03-23T03:49:09.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.410+0000 7f8ee7e00640 1 -- 172.21.15.102:0/963994997 learned_addr learned my addr 172.21.15.102:0/963994997 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:09.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.410+0000 7f8ee6a00640 1 -- 172.21.15.102:0/963994997 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264089526 0 0) 0x7f8ee80b16a0 con 0x7f8ee80adbc0 2024-03-23T03:49:09.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.410+0000 7f8ee6a00640 1 -- 172.21.15.102:0/963994997 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ecc003520 con 0x7f8ee80adbc0 2024-03-23T03:49:09.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.410+0000 7f8ee7400640 1 --2- 172.21.15.102:0/963994997 >> v2:172.21.15.173:3300/0 conn(0x7f8ee80cbab0 0x7f8ee80b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:09.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.410+0000 7f8ee6a00640 1 -- 172.21.15.102:0/963994997 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1266414959 0 0) 0x7f8ecc003520 con 0x7f8ee80adbc0 2024-03-23T03:49:09.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.410+0000 7f8ee6a00640 1 -- 172.21.15.102:0/963994997 >> v2:172.21.15.173:3300/0 conn(0x7f8ee80cbab0 msgr2=0x7f8ee80b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:09.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee6a00640 1 --2- 172.21.15.102:0/963994997 >> v2:172.21.15.173:3300/0 conn(0x7f8ee80cbab0 0x7f8ee80b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee6a00640 1 -- 172.21.15.102:0/963994997 >> v1:172.21.15.173:6789/0 conn(0x7f8ee806d390 legacy=0x7f8ee80b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:09.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee6a00640 1 -- 172.21.15.102:0/963994997 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee80b18f0 con 0x7f8ee80adbc0 2024-03-23T03:49:09.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee6a00640 1 -- 172.21.15.102:0/963994997 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f8ed8001670 con 0x7f8ee80adbc0 2024-03-23T03:49:09.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee7400640 1 --2- 172.21.15.102:0/963994997 >> v2:172.21.15.173:3300/0 conn(0x7f8ee80cbab0 0x7f8ee80b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:09.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee6a00640 1 -- 172.21.15.102:0/963994997 >> v1:172.21.15.102:6789/0 conn(0x7f8ee80adbc0 legacy=0x7f8ee80b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:09.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee6a00640 1 --2- 172.21.15.102:0/963994997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ecc003970 0x7f8ecc003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:09.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee6a00640 1 --2- 172.21.15.102:0/963994997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ecc0047c0 0x7f8ecc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:09.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee6a00640 1 --2- 172.21.15.102:0/963994997 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8ecc005380 0x7f8ecc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:09.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee6a00640 1 -- 172.21.15.102:0/963994997 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee80cc290 con 0x7f8ecc003970 2024-03-23T03:49:09.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee6a00640 1 -- 172.21.15.102:0/963994997 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc00c220 con 0x7f8ecc0047c0 2024-03-23T03:49:09.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee6a00640 1 -- 172.21.15.102:0/963994997 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc00e5f0 con 0x7f8ecc005380 2024-03-23T03:49:09.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee7e00640 1 --2- 172.21.15.102:0/963994997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ecc003970 0x7f8ecc003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:09.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee7400640 1 --2- 172.21.15.102:0/963994997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ecc0047c0 0x7f8ecc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:09.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee7e00640 1 -- 172.21.15.102:0/963994997 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8ecc005380 msgr2=0x7f8ecc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:09.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee7e00640 1 --2- 172.21.15.102:0/963994997 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8ecc005380 0x7f8ecc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee7e00640 1 -- 172.21.15.102:0/963994997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ecc0047c0 msgr2=0x7f8ecc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:09.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee7e00640 1 --2- 172.21.15.102:0/963994997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ecc0047c0 0x7f8ecc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.411+0000 7f8ee7e00640 1 -- 172.21.15.102:0/963994997 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc003520 con 0x7f8ecc003970 2024-03-23T03:49:09.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.412+0000 7f8eed200640 1 -- 172.21.15.102:0/963994997 reap_dead start 2024-03-23T03:49:09.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.412+0000 7f8ee7e00640 1 --2- 172.21.15.102:0/963994997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ecc003970 0x7f8ecc003d70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f8ed800c210 tx=0x7f8ecc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3078940c369da47c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:09.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.413+0000 7f8ee6a00640 1 -- 172.21.15.102:0/963994997 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed802cae0 con 0x7f8ecc003970 2024-03-23T03:49:09.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.413+0000 7f8ee6a00640 1 -- 172.21.15.102:0/963994997 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed802cc60 con 0x7f8ecc003970 2024-03-23T03:49:09.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.413+0000 7f8ee6a00640 1 -- 172.21.15.102:0/963994997 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8005b10 con 0x7f8ecc003970 2024-03-23T03:49:09.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.413+0000 7f8eef200640 1 -- 172.21.15.102:0/963994997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ecc003970 msgr2=0x7f8ecc003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:09.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.413+0000 7f8eef200640 1 --2- 172.21.15.102:0/963994997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ecc003970 0x7f8ecc003d70 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f8ed800c210 tx=0x7f8ecc0057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.413+0000 7f8eef200640 1 -- 172.21.15.102:0/963994997 shutdown_connections 2024-03-23T03:49:09.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.413+0000 7f8eef200640 1 --2- 172.21.15.102:0/963994997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ecc003970 0x7f8ecc003d70 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.413+0000 7f8eef200640 1 -- 172.21.15.102:0/963994997 >> 172.21.15.102:0/963994997 conn(0x7f8ee806a3d0 msgr2=0x7f8ee80b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:09.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.413+0000 7f8eef200640 1 -- 172.21.15.102:0/963994997 shutdown_connections 2024-03-23T03:49:09.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.413+0000 7f8eef200640 1 -- 172.21.15.102:0/963994997 wait complete. 2024-03-23T03:49:09.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.415+0000 7f8eef200640 1 Processor -- start 2024-03-23T03:49:09.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.415+0000 7f8eef200640 1 -- start start 2024-03-23T03:49:09.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.416+0000 7f8eef200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ecc003970 0x7f8ee81591d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:09.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.416+0000 7f8eef200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ee813f800 0x7f8ee815b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:09.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.416+0000 7f8eef200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8ee815be70 0x7f8ee81f1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:09.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.416+0000 7f8eef200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee815c2f0 con 0x7f8ee813f800 2024-03-23T03:49:09.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.416+0000 7f8eef200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee815a490 con 0x7f8ecc003970 2024-03-23T03:49:09.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.416+0000 7f8eef200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ee80acfd0 con 0x7f8ee815be70 2024-03-23T03:49:09.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.417+0000 7f8ee7400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ee813f800 0x7f8ee815b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:09.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.417+0000 7f8ee7400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ee813f800 0x7f8ee815b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53786/0 (socket says 172.21.15.102:53786) 2024-03-23T03:49:09.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.417+0000 7f8ee7400640 1 -- 172.21.15.102:0/3177665714 learned_addr learned my addr 172.21.15.102:0/3177665714 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:09.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.417+0000 7f8eed200640 1 --2- 172.21.15.102:0/3177665714 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8ee815be70 0x7f8ee81f1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:09.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.417+0000 7f8ee7e00640 1 --2- 172.21.15.102:0/3177665714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ecc003970 0x7f8ee81591d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:09.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.417+0000 7f8ee7400640 1 -- 172.21.15.102:0/3177665714 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8ee815be70 msgr2=0x7f8ee81f1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:09.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.417+0000 7f8ee7400640 1 --2- 172.21.15.102:0/3177665714 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8ee815be70 0x7f8ee81f1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.417+0000 7f8ee7400640 1 -- 172.21.15.102:0/3177665714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ecc003970 msgr2=0x7f8ee81591d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:09.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.417+0000 7f8ee7400640 1 --2- 172.21.15.102:0/3177665714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ecc003970 0x7f8ee81591d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.417+0000 7f8ee7400640 1 -- 172.21.15.102:0/3177665714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee813fc80 con 0x7f8ee813f800 2024-03-23T03:49:09.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.418+0000 7f8ee7400640 1 --2- 172.21.15.102:0/3177665714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ee813f800 0x7f8ee815b930 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400e870 tx=0x7f8ed400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:09.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.418+0000 7f8edfe00640 1 -- 172.21.15.102:0/3177665714 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed400a8c0 con 0x7f8ee813f800 2024-03-23T03:49:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.418+0000 7f8edfe00640 1 -- 172.21.15.102:0/3177665714 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed400aa20 con 0x7f8ee813f800 2024-03-23T03:49:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.418+0000 7f8edfe00640 1 -- 172.21.15.102:0/3177665714 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed400adb0 con 0x7f8ee813f800 2024-03-23T03:49:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.418+0000 7f8eef200640 1 -- 172.21.15.102:0/3177665714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee81f2060 con 0x7f8ee813f800 2024-03-23T03:49:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.418+0000 7f8eef200640 1 -- 172.21.15.102:0/3177665714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ee815a120 con 0x7f8ee813f800 2024-03-23T03:49:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.419+0000 7f8eef200640 1 -- 172.21.15.102:0/3177665714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee806ac20 con 0x7f8ee813f800 2024-03-23T03:49:09.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.420+0000 7f8edfe00640 1 -- 172.21.15.102:0/3177665714 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8ed4015480 con 0x7f8ee813f800 2024-03-23T03:49:09.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.421+0000 7f8edfe00640 1 --2- 172.21.15.102:0/3177665714 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8ec806ac00 0x7f8ec806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:09.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.421+0000 7f8edfe00640 1 -- 172.21.15.102:0/3177665714 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8ed408a140 con 0x7f8ee813f800 2024-03-23T03:49:09.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.421+0000 7f8ee7e00640 1 --2- 172.21.15.102:0/3177665714 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8ec806ac00 0x7f8ec806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:09.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.422+0000 7f8ee7e00640 1 --2- 172.21.15.102:0/3177665714 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8ec806ac00 0x7f8ec806d0c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8034000 tx=0x7f8ed8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:09.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.425+0000 7f8edfe00640 1 -- 172.21.15.102:0/3177665714 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8ed40551e0 con 0x7f8ee813f800 2024-03-23T03:49:09.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.619+0000 7f8eef200640 1 -- 172.21.15.102:0/3177665714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ee8073390 con 0x7f8ee813f800 2024-03-23T03:49:09.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.622+0000 7f8edfe00640 1 -- 172.21.15.102:0/3177665714 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16354 (secure 0 0 0) 0x7f8ed4058800 con 0x7f8ee813f800 2024-03-23T03:49:09.623 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:09.623 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":55,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:57.226138+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:48:53.998390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T03:48:55.242109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:49:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.625+0000 7f8eef200640 1 -- 172.21.15.102:0/3177665714 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8ec806ac00 msgr2=0x7f8ec806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.625+0000 7f8eef200640 1 --2- 172.21.15.102:0/3177665714 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8ec806ac00 0x7f8ec806d0c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8034000 tx=0x7f8ed8009030 comp rx=0 tx=0).stop 2024-03-23T03:49:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.625+0000 7f8eef200640 1 -- 172.21.15.102:0/3177665714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ee813f800 msgr2=0x7f8ee815b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.625+0000 7f8eef200640 1 --2- 172.21.15.102:0/3177665714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ee813f800 0x7f8ee815b930 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f8ed400e870 tx=0x7f8ed400ed30 comp rx=0 tx=0).stop 2024-03-23T03:49:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.625+0000 7f8eef200640 1 -- 172.21.15.102:0/3177665714 shutdown_connections 2024-03-23T03:49:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.625+0000 7f8eef200640 1 --2- 172.21.15.102:0/3177665714 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8ec806ac00 0x7f8ec806d0c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.625+0000 7f8eef200640 1 --2- 172.21.15.102:0/3177665714 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8ee815be70 0x7f8ee81f1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.625+0000 7f8eef200640 1 --2- 172.21.15.102:0/3177665714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ee813f800 0x7f8ee815b930 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.625+0000 7f8eef200640 1 --2- 172.21.15.102:0/3177665714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ecc003970 0x7f8ee81591d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.625+0000 7f8eef200640 1 -- 172.21.15.102:0/3177665714 >> 172.21.15.102:0/3177665714 conn(0x7f8ee806a3d0 msgr2=0x7f8ee8141e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.625+0000 7f8eef200640 1 -- 172.21.15.102:0/3177665714 shutdown_connections 2024-03-23T03:49:09.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.625+0000 7f8eef200640 1 -- 172.21.15.102:0/3177665714 wait complete. 2024-03-23T03:49:09.644 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:48:53.998390+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:48:55.242109+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:49:09.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:49:09.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.856+0000 7fd9bfe00640 1 Processor -- start 2024-03-23T03:49:09.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.856+0000 7fd9bfe00640 1 -- start start 2024-03-23T03:49:09.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.857+0000 7fd9bfe00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd9c015a850 0x7fd9c00cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:09.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.857+0000 7fd9bfe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9c00b22b0 0x7fd9c015eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:09.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.857+0000 7fd9bfe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c015f020 0x7fd9c00cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:09.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.857+0000 7fd9bfe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9c015ac30 con 0x7fd9c00b22b0 2024-03-23T03:49:09.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.857+0000 7fd9bfe00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9c015e320 con 0x7fd9c015a850 2024-03-23T03:49:09.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.857+0000 7fd9bfe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c00cda20 con 0x7fd9c015f020 2024-03-23T03:49:09.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.857+0000 7fd9b6000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9c00b22b0 0x7fd9c015eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:09.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.857+0000 7fd9b6000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9c00b22b0 0x7fd9c015eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53794/0 (socket says 172.21.15.102:53794) 2024-03-23T03:49:09.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.857+0000 7fd9b6000640 1 -- 172.21.15.102:0/2976377103 learned_addr learned my addr 172.21.15.102:0/2976377103 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:09.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.858+0000 7fd9bea00640 1 --2- 172.21.15.102:0/2976377103 >> v2:172.21.15.173:3300/0 conn(0x7fd9c015a850 0x7fd9c00cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:09.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.858+0000 7fd9bf400640 1 --2- 172.21.15.102:0/2976377103 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c015f020 0x7fd9c00cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:09.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.858+0000 7fd9b6000640 1 -- 172.21.15.102:0/2976377103 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c015f020 msgr2=0x7fd9c00cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:09.859 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.858+0000 7fd9b6000640 1 --2- 172.21.15.102:0/2976377103 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c015f020 0x7fd9c00cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.858+0000 7fd9b6000640 1 -- 172.21.15.102:0/2976377103 >> v2:172.21.15.173:3300/0 conn(0x7fd9c015a850 msgr2=0x7fd9c00cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:09.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.858+0000 7fd9b6000640 1 --2- 172.21.15.102:0/2976377103 >> v2:172.21.15.173:3300/0 conn(0x7fd9c015a850 0x7fd9c00cd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.858+0000 7fd9b6000640 1 -- 172.21.15.102:0/2976377103 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c00cdd00 con 0x7fd9c00b22b0 2024-03-23T03:49:09.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.858+0000 7fd9bea00640 1 --2- 172.21.15.102:0/2976377103 >> v2:172.21.15.173:3300/0 conn(0x7fd9c015a850 0x7fd9c00cd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:09.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.858+0000 7fd9bf400640 1 --2- 172.21.15.102:0/2976377103 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c015f020 0x7fd9c00cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:09.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.859+0000 7fd9b6000640 1 --2- 172.21.15.102:0/2976377103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c00b22b0 0x7fd9c015eae0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd9c006adb0 tx=0x7fd9a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e6c5d8cbf54ef9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:09.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.859+0000 7fd9be000640 1 -- 172.21.15.102:0/2976377103 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8031870 con 0x7fd9c00b22b0 2024-03-23T03:49:09.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.859+0000 7fd9be000640 1 -- 172.21.15.102:0/2976377103 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a80319d0 con 0x7fd9c00b22b0 2024-03-23T03:49:09.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.859+0000 7fd9be000640 1 -- 172.21.15.102:0/2976377103 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8031d60 con 0x7fd9c00b22b0 2024-03-23T03:49:09.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.860+0000 7fd9bfe00640 1 -- 172.21.15.102:0/2976377103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c00b22b0 msgr2=0x7fd9c015eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:09.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.860+0000 7fd9bfe00640 1 --2- 172.21.15.102:0/2976377103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c00b22b0 0x7fd9c015eae0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd9c006adb0 tx=0x7fd9a802b910 comp rx=0 tx=0).stop 2024-03-23T03:49:09.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.860+0000 7fd9bfe00640 1 -- 172.21.15.102:0/2976377103 shutdown_connections 2024-03-23T03:49:09.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.860+0000 7fd9bfe00640 1 --2- 172.21.15.102:0/2976377103 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c015f020 0x7fd9c00cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.860+0000 7fd9bfe00640 1 --2- 172.21.15.102:0/2976377103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c00b22b0 0x7fd9c015eae0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.860+0000 7fd9bfe00640 1 --2- 172.21.15.102:0/2976377103 >> v2:172.21.15.173:3300/0 conn(0x7fd9c015a850 0x7fd9c00cd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.860+0000 7fd9bfe00640 1 -- 172.21.15.102:0/2976377103 >> 172.21.15.102:0/2976377103 conn(0x7fd9c00b01b0 msgr2=0x7fd9c00b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:09.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.860+0000 7fd9bfe00640 1 -- 172.21.15.102:0/2976377103 shutdown_connections 2024-03-23T03:49:09.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.860+0000 7fd9bfe00640 1 -- 172.21.15.102:0/2976377103 wait complete. 2024-03-23T03:49:09.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.862+0000 7fd9bfe00640 1 Processor -- start 2024-03-23T03:49:09.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.862+0000 7fd9bfe00640 1 -- start start 2024-03-23T03:49:09.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.863+0000 7fd9bfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c00b22b0 0x7fd9c01539c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:09.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.863+0000 7fd9bfe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9c015a850 0x7fd9c00ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:09.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.863+0000 7fd9bfe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c015f020 0x7fd9c00c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:09.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.863+0000 7fd9bfe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c00c8c20 con 0x7fd9c00b22b0 2024-03-23T03:49:09.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.863+0000 7fd9bfe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c01f5470 con 0x7fd9c015a850 2024-03-23T03:49:09.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.863+0000 7fd9bfe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9c01f56c0 con 0x7fd9c015f020 2024-03-23T03:49:09.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.863+0000 7fd9bea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c00b22b0 0x7fd9c01539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:09.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.863+0000 7fd9bea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c00b22b0 0x7fd9c01539c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53810/0 (socket says 172.21.15.102:53810) 2024-03-23T03:49:09.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.863+0000 7fd9bea00640 1 -- 172.21.15.102:0/2856542760 learned_addr learned my addr 172.21.15.102:0/2856542760 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:09.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.864+0000 7fd9bea00640 1 -- 172.21.15.102:0/2856542760 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c015f020 msgr2=0x7fd9c00c7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:09.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.864+0000 7fd9bf400640 1 --2- 172.21.15.102:0/2856542760 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c015f020 0x7fd9c00c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:09.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.864+0000 7fd9b6000640 1 --2- 172.21.15.102:0/2856542760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9c015a850 0x7fd9c00ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:09.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.864+0000 7fd9bea00640 1 --2- 172.21.15.102:0/2856542760 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c015f020 0x7fd9c00c7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.864+0000 7fd9bea00640 1 -- 172.21.15.102:0/2856542760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9c015a850 msgr2=0x7fd9c00ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:09.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.864+0000 7fd9bea00640 1 --2- 172.21.15.102:0/2856542760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9c015a850 0x7fd9c00ca160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:09.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.864+0000 7fd9bea00640 1 -- 172.21.15.102:0/2856542760 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a80094d0 con 0x7fd9c00b22b0 2024-03-23T03:49:09.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.864+0000 7fd9bea00640 1 --2- 172.21.15.102:0/2856542760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c00b22b0 0x7fd9c01539c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000cee0 tx=0x7fd9b000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:09.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.864+0000 7fd9b7e00640 1 -- 172.21.15.102:0/2856542760 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b000d840 con 0x7fd9c00b22b0 2024-03-23T03:49:09.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.864+0000 7fd9b7e00640 1 -- 172.21.15.102:0/2856542760 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b000d9a0 con 0x7fd9c00b22b0 2024-03-23T03:49:09.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.864+0000 7fd9b7e00640 1 -- 172.21.15.102:0/2856542760 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b000dd30 con 0x7fd9c00b22b0 2024-03-23T03:49:09.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.864+0000 7fd9bfe00640 1 -- 172.21.15.102:0/2856542760 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9c01f59a0 con 0x7fd9c00b22b0 2024-03-23T03:49:09.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.864+0000 7fd9bfe00640 1 -- 172.21.15.102:0/2856542760 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9c01f5f60 con 0x7fd9c00b22b0 2024-03-23T03:49:09.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.866+0000 7fd9bfe00640 1 -- 172.21.15.102:0/2856542760 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9c004e030 con 0x7fd9c00b22b0 2024-03-23T03:49:09.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.867+0000 7fd9b7e00640 1 -- 172.21.15.102:0/2856542760 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd9b0015440 con 0x7fd9c00b22b0 2024-03-23T03:49:09.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.867+0000 7fd9b7e00640 1 --2- 172.21.15.102:0/2856542760 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd99c06ac30 0x7fd99c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:09.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.867+0000 7fd9b7e00640 1 -- 172.21.15.102:0/2856542760 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd9b0089e10 con 0x7fd9c00b22b0 2024-03-23T03:49:09.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.868+0000 7fd9b6000640 1 --2- 172.21.15.102:0/2856542760 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd99c06ac30 0x7fd99c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:09.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.868+0000 7fd9b6000640 1 --2- 172.21.15.102:0/2856542760 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd99c06ac30 0x7fd99c06d0f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8036000 tx=0x7fd9a803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:09.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:09.872+0000 7fd9b7e00640 1 -- 172.21.15.102:0/2856542760 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd9b008d030 con 0x7fd9c00b22b0 2024-03-23T03:49:10.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.063+0000 7fd9bfe00640 1 -- 172.21.15.102:0/2856542760 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd9c00ccaa0 con 0x7fd9c00b22b0 2024-03-23T03:49:10.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.064+0000 7fd9b7e00640 1 -- 172.21.15.102:0/2856542760 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd9b001a020 con 0x7fd9c00b22b0 2024-03-23T03:49:10.065 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:49:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.066+0000 7fd9bfe00640 1 -- 172.21.15.102:0/2856542760 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd99c06ac30 msgr2=0x7fd99c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.066+0000 7fd9bfe00640 1 --2- 172.21.15.102:0/2856542760 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd99c06ac30 0x7fd99c06d0f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8036000 tx=0x7fd9a803c040 comp rx=0 tx=0).stop 2024-03-23T03:49:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.067+0000 7fd9bfe00640 1 -- 172.21.15.102:0/2856542760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c00b22b0 msgr2=0x7fd9c01539c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.067+0000 7fd9bfe00640 1 --2- 172.21.15.102:0/2856542760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c00b22b0 0x7fd9c01539c0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000cee0 tx=0x7fd9b000a710 comp rx=0 tx=0).stop 2024-03-23T03:49:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.067+0000 7fd9bfe00640 1 -- 172.21.15.102:0/2856542760 shutdown_connections 2024-03-23T03:49:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.067+0000 7fd9bfe00640 1 --2- 172.21.15.102:0/2856542760 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd99c06ac30 0x7fd99c06d0f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.067+0000 7fd9bfe00640 1 --2- 172.21.15.102:0/2856542760 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9c015f020 0x7fd9c00c7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.067+0000 7fd9bfe00640 1 --2- 172.21.15.102:0/2856542760 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9c015a850 0x7fd9c00ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.067+0000 7fd9bfe00640 1 --2- 172.21.15.102:0/2856542760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9c00b22b0 0x7fd9c01539c0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.067+0000 7fd9bfe00640 1 -- 172.21.15.102:0/2856542760 >> 172.21.15.102:0/2856542760 conn(0x7fd9c00b01b0 msgr2=0x7fd9c00b0590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.067+0000 7fd9bfe00640 1 -- 172.21.15.102:0/2856542760 shutdown_connections 2024-03-23T03:49:10.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.067+0000 7fd9bfe00640 1 -- 172.21.15.102:0/2856542760 wait complete. 2024-03-23T03:49:10.085 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:49:10.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.292+0000 7f766ee00640 1 Processor -- start 2024-03-23T03:49:10.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.292+0000 7f766ee00640 1 -- start start 2024-03-23T03:49:10.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.293+0000 7f766ee00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76680cd740 0x7f76680b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:10.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.293+0000 7f766ee00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f76680d0d40 0x7f76680cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:10.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.293+0000 7f766ee00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76680d20c0 con 0x7f766806b0a0 2024-03-23T03:49:10.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.293+0000 7f766ee00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76680ce060 con 0x7f76680d0d40 2024-03-23T03:49:10.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.293+0000 7f766ee00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76680d14b0 con 0x7f76680cd740 2024-03-23T03:49:10.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.294+0000 7f766ce00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f766806b0a0 0x7f766806b530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57274/0 (socket says 172.21.15.102:57274) 2024-03-23T03:49:10.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.294+0000 7f766ce00640 1 -- 172.21.15.102:0/1934464946 learned_addr learned my addr 172.21.15.102:0/1934464946 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:10.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.294+0000 7f7667e00640 1 --2- 172.21.15.102:0/1934464946 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76680cd740 0x7f76680b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:10.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.294+0000 7f7667400640 1 --2- 172.21.15.102:0/1934464946 >> v2:172.21.15.173:3300/0 conn(0x7f76680d0d40 0x7f76680cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:10.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.294+0000 7f7666a00640 1 -- 172.21.15.102:0/1934464946 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095977935 0 0) 0x7f76680d20c0 con 0x7f766806b0a0 2024-03-23T03:49:10.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.294+0000 7f7666a00640 1 -- 172.21.15.102:0/1934464946 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f764c003520 con 0x7f766806b0a0 2024-03-23T03:49:10.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.295+0000 7f7666a00640 1 -- 172.21.15.102:0/1934464946 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2623490135 0 0) 0x7f764c003520 con 0x7f766806b0a0 2024-03-23T03:49:10.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.295+0000 7f7667e00640 1 -- 172.21.15.102:0/1934464946 >> v2:172.21.15.173:3300/0 conn(0x7f76680d0d40 msgr2=0x7f76680cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:10.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.295+0000 7f7667e00640 1 --2- 172.21.15.102:0/1934464946 >> v2:172.21.15.173:3300/0 conn(0x7f76680d0d40 0x7f76680cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.295+0000 7f7667e00640 1 -- 172.21.15.102:0/1934464946 >> v1:172.21.15.102:6789/0 conn(0x7f766806b0a0 legacy=0x7f766806b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:10.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.295+0000 7f7667e00640 1 -- 172.21.15.102:0/1934464946 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76680d1790 con 0x7f76680cd740 2024-03-23T03:49:10.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.296+0000 7f7667e00640 1 --2- 172.21.15.102:0/1934464946 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76680cd740 0x7f76680b84b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f76580097a0 tx=0x7f765802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73d53cf699d781d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:10.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.297+0000 7f7666a00640 1 -- 172.21.15.102:0/1934464946 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76600019f0 con 0x7f76680cd740 2024-03-23T03:49:10.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.297+0000 7f7666a00640 1 -- 172.21.15.102:0/1934464946 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f765802e050 con 0x7f76680cd740 2024-03-23T03:49:10.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.297+0000 7f7666a00640 1 -- 172.21.15.102:0/1934464946 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658030710 con 0x7f76680cd740 2024-03-23T03:49:10.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.297+0000 7f766ee00640 1 -- 172.21.15.102:0/1934464946 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76680cd740 msgr2=0x7f76680b84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:10.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.297+0000 7f766ee00640 1 --2- 172.21.15.102:0/1934464946 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76680cd740 0x7f76680b84b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f76580097a0 tx=0x7f765802b790 comp rx=0 tx=0).stop 2024-03-23T03:49:10.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.297+0000 7f766ee00640 1 -- 172.21.15.102:0/1934464946 shutdown_connections 2024-03-23T03:49:10.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.297+0000 7f766ee00640 1 --2- 172.21.15.102:0/1934464946 >> v2:172.21.15.173:3300/0 conn(0x7f76680d0d40 0x7f76680cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.297+0000 7f766ee00640 1 --2- 172.21.15.102:0/1934464946 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76680cd740 0x7f76680b84b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.297+0000 7f766ee00640 1 -- 172.21.15.102:0/1934464946 >> 172.21.15.102:0/1934464946 conn(0x7f76680cf5d0 msgr2=0x7f76680cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:10.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.297+0000 7f766ee00640 1 -- 172.21.15.102:0/1934464946 shutdown_connections 2024-03-23T03:49:10.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.297+0000 7f766ee00640 1 -- 172.21.15.102:0/1934464946 wait complete. 2024-03-23T03:49:10.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.299+0000 7f766ee00640 1 Processor -- start 2024-03-23T03:49:10.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.299+0000 7f766ee00640 1 -- start start 2024-03-23T03:49:10.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.300+0000 7f766ee00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f766806b0a0 0x7f76680cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:10.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.300+0000 7f766ee00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76680cd740 0x7f76680ca450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:10.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.300+0000 7f766ee00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76680d0d40 0x7f7668154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:10.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.300+0000 7f766ee00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7668155130 con 0x7f766806b0a0 2024-03-23T03:49:10.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.300+0000 7f766ee00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76680ca990 con 0x7f76680cd740 2024-03-23T03:49:10.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.300+0000 7f766ee00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76680cac30 con 0x7f76680d0d40 2024-03-23T03:49:10.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.301+0000 7f7667e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f766806b0a0 0x7f76680cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:10.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.301+0000 7f7667e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f766806b0a0 0x7f76680cc4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53814/0 (socket says 172.21.15.102:53814) 2024-03-23T03:49:10.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.301+0000 7f7667e00640 1 -- 172.21.15.102:0/3915783405 learned_addr learned my addr 172.21.15.102:0/3915783405 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:10.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.301+0000 7f766ce00640 1 --2- 172.21.15.102:0/3915783405 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76680d0d40 0x7f7668154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:10.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.301+0000 7f7667400640 1 --2- 172.21.15.102:0/3915783405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76680cd740 0x7f76680ca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:10.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.301+0000 7f7667e00640 1 -- 172.21.15.102:0/3915783405 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76680d0d40 msgr2=0x7f7668154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:10.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.301+0000 7f7667e00640 1 --2- 172.21.15.102:0/3915783405 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76680d0d40 0x7f7668154300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.301+0000 7f7667e00640 1 -- 172.21.15.102:0/3915783405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76680cd740 msgr2=0x7f76680ca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:10.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.301+0000 7f7667e00640 1 --2- 172.21.15.102:0/3915783405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76680cd740 0x7f76680ca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.301+0000 7f7667e00640 1 -- 172.21.15.102:0/3915783405 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7658009400 con 0x7f766806b0a0 2024-03-23T03:49:10.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.301+0000 7f7667e00640 1 --2- 172.21.15.102:0/3915783405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f766806b0a0 0x7f76680cc4c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7658000c00 tx=0x7f76580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:10.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.302+0000 7f765fe00640 1 -- 172.21.15.102:0/3915783405 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658003ea0 con 0x7f766806b0a0 2024-03-23T03:49:10.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.302+0000 7f765fe00640 1 -- 172.21.15.102:0/3915783405 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76580044c0 con 0x7f766806b0a0 2024-03-23T03:49:10.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.302+0000 7f765fe00640 1 -- 172.21.15.102:0/3915783405 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658030750 con 0x7f766806b0a0 2024-03-23T03:49:10.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.302+0000 7f766ee00640 1 -- 172.21.15.102:0/3915783405 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76680caf10 con 0x7f766806b0a0 2024-03-23T03:49:10.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.302+0000 7f766ee00640 1 -- 172.21.15.102:0/3915783405 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76680cb4d0 con 0x7f766806b0a0 2024-03-23T03:49:10.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.303+0000 7f766ee00640 1 -- 172.21.15.102:0/3915783405 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7668073390 con 0x7f766806b0a0 2024-03-23T03:49:10.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.304+0000 7f765fe00640 1 -- 172.21.15.102:0/3915783405 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7658039070 con 0x7f766806b0a0 2024-03-23T03:49:10.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.305+0000 7f765fe00640 1 --2- 172.21.15.102:0/3915783405 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f764406ac30 0x7f764406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:10.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.305+0000 7f765fe00640 1 -- 172.21.15.102:0/3915783405 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f76580aab90 con 0x7f766806b0a0 2024-03-23T03:49:10.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.305+0000 7f7667400640 1 --2- 172.21.15.102:0/3915783405 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f764406ac30 0x7f764406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:10.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.306+0000 7f7667400640 1 --2- 172.21.15.102:0/3915783405 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f764406ac30 0x7f764406d0f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f76680cdb40 tx=0x7f7650008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:10.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.310+0000 7f765fe00640 1 -- 172.21.15.102:0/3915783405 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7658075c10 con 0x7f766806b0a0 2024-03-23T03:49:10.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.503+0000 7f766ee00640 1 -- 172.21.15.102:0/3915783405 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7668155270 con 0x7f766806b0a0 2024-03-23T03:49:10.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.505+0000 7f765fe00640 1 -- 172.21.15.102:0/3915783405 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7658040020 con 0x7f766806b0a0 2024-03-23T03:49:10.505 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:49:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.507+0000 7f766ee00640 1 -- 172.21.15.102:0/3915783405 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f764406ac30 msgr2=0x7f764406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.507+0000 7f766ee00640 1 --2- 172.21.15.102:0/3915783405 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f764406ac30 0x7f764406d0f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f76680cdb40 tx=0x7f7650008040 comp rx=0 tx=0).stop 2024-03-23T03:49:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.507+0000 7f766ee00640 1 -- 172.21.15.102:0/3915783405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f766806b0a0 msgr2=0x7f76680cc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.507+0000 7f766ee00640 1 --2- 172.21.15.102:0/3915783405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f766806b0a0 0x7f76680cc4c0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f7658000c00 tx=0x7f76580046c0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.507+0000 7f766ee00640 1 -- 172.21.15.102:0/3915783405 shutdown_connections 2024-03-23T03:49:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.507+0000 7f766ee00640 1 --2- 172.21.15.102:0/3915783405 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f764406ac30 0x7f764406d0f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.507+0000 7f766ee00640 1 --2- 172.21.15.102:0/3915783405 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76680d0d40 0x7f7668154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.507+0000 7f766ee00640 1 --2- 172.21.15.102:0/3915783405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76680cd740 0x7f76680ca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.507+0000 7f766ee00640 1 --2- 172.21.15.102:0/3915783405 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f766806b0a0 0x7f76680cc4c0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.507+0000 7f766ee00640 1 -- 172.21.15.102:0/3915783405 >> 172.21.15.102:0/3915783405 conn(0x7f76680cf5d0 msgr2=0x7f76680c86d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:10.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.507+0000 7f766ee00640 1 -- 172.21.15.102:0/3915783405 shutdown_connections 2024-03-23T03:49:10.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.508+0000 7f766ee00640 1 -- 172.21.15.102:0/3915783405 wait complete. 2024-03-23T03:49:10.526 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:49:10.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.740+0000 7fb964c00640 1 Processor -- start 2024-03-23T03:49:10.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.741+0000 7fb964c00640 1 -- start start 2024-03-23T03:49:10.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.742+0000 7fb964c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb9600cd740 0x7fb9600b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:10.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.742+0000 7fb964c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb9600b5aa0 0x7fb9600b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:10.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.742+0000 7fb964c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9600d20c0 con 0x7fb9600d0d40 2024-03-23T03:49:10.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.742+0000 7fb964c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9600d14b0 con 0x7fb9600b5aa0 2024-03-23T03:49:10.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.742+0000 7fb964c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9600d1790 con 0x7fb9600cd740 2024-03-23T03:49:10.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.742+0000 7fb95e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb9600b5aa0 0x7fb9600b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:10.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.742+0000 7fb95e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb9600b5aa0 0x7fb9600b5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53826/0 (socket says 172.21.15.102:53826) 2024-03-23T03:49:10.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.742+0000 7fb95e800640 1 -- 172.21.15.102:0/2246966805 learned_addr learned my addr 172.21.15.102:0/2246966805 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:10.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.743+0000 7fb95de00640 1 --2- 172.21.15.102:0/2246966805 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb9600cd740 0x7fb9600b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:10.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.743+0000 7fb95e800640 1 -- 172.21.15.102:0/2246966805 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb9600cd740 msgr2=0x7fb9600b84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:10.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.743+0000 7fb95e800640 1 --2- 172.21.15.102:0/2246966805 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb9600cd740 0x7fb9600b84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.743+0000 7fb95e800640 1 -- 172.21.15.102:0/2246966805 >> v1:172.21.15.173:6789/0 conn(0x7fb9600d0d40 legacy=0x7fb9600cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:10.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.743+0000 7fb95e800640 1 -- 172.21.15.102:0/2246966805 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9600d1a70 con 0x7fb9600b5aa0 2024-03-23T03:49:10.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.744+0000 7fb95e800640 1 --2- 172.21.15.102:0/2246966805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9600b5aa0 0x7fb9600b5f30 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb958009e50 tx=0x7fb95802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab9ac95fcff7215 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:10.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.744+0000 7fb95d400640 1 -- 172.21.15.102:0/2246966805 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95802ca70 con 0x7fb9600b5aa0 2024-03-23T03:49:10.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.744+0000 7fb95d400640 1 -- 172.21.15.102:0/2246966805 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb95802cbd0 con 0x7fb9600b5aa0 2024-03-23T03:49:10.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.744+0000 7fb95d400640 1 -- 172.21.15.102:0/2246966805 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9580343f0 con 0x7fb9600b5aa0 2024-03-23T03:49:10.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.745+0000 7fb964c00640 1 -- 172.21.15.102:0/2246966805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9600b5aa0 msgr2=0x7fb9600b5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:10.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.745+0000 7fb964c00640 1 --2- 172.21.15.102:0/2246966805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9600b5aa0 0x7fb9600b5f30 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fb958009e50 tx=0x7fb95802bed0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.745+0000 7fb964c00640 1 -- 172.21.15.102:0/2246966805 shutdown_connections 2024-03-23T03:49:10.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.745+0000 7fb964c00640 1 --2- 172.21.15.102:0/2246966805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9600b5aa0 0x7fb9600b5f30 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.745+0000 7fb964c00640 1 --2- 172.21.15.102:0/2246966805 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb9600cd740 0x7fb9600b84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.745+0000 7fb964c00640 1 -- 172.21.15.102:0/2246966805 >> 172.21.15.102:0/2246966805 conn(0x7fb9600cf5d0 msgr2=0x7fb9600cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:10.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.745+0000 7fb964c00640 1 -- 172.21.15.102:0/2246966805 shutdown_connections 2024-03-23T03:49:10.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.745+0000 7fb964c00640 1 -- 172.21.15.102:0/2246966805 wait complete. 2024-03-23T03:49:10.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.747+0000 7fb964c00640 1 Processor -- start 2024-03-23T03:49:10.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.747+0000 7fb964c00640 1 -- start start 2024-03-23T03:49:10.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.748+0000 7fb964c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9600b5aa0 0x7fb9600c6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:10.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.748+0000 7fb964c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb9600cd740 0x7fb9600c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:10.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.748+0000 7fb964c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9600d0d40 0x7fb9600c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:10.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.748+0000 7fb964c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9600ca890 con 0x7fb9600b5aa0 2024-03-23T03:49:10.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.748+0000 7fb964c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9600caae0 con 0x7fb9600d0d40 2024-03-23T03:49:10.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.748+0000 7fb964c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9600cadc0 con 0x7fb9600cd740 2024-03-23T03:49:10.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.748+0000 7fb95de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9600b5aa0 0x7fb9600c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:10.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.748+0000 7fb95de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9600b5aa0 0x7fb9600c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53832/0 (socket says 172.21.15.102:53832) 2024-03-23T03:49:10.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.748+0000 7fb95de00640 1 -- 172.21.15.102:0/686098625 learned_addr learned my addr 172.21.15.102:0/686098625 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:10.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.749+0000 7fb95e800640 1 --2- 172.21.15.102:0/686098625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9600d0d40 0x7fb9600c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:10.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.749+0000 7fb955400640 1 --2- 172.21.15.102:0/686098625 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb9600cd740 0x7fb9600c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:10.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.749+0000 7fb95de00640 1 -- 172.21.15.102:0/686098625 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb9600cd740 msgr2=0x7fb9600c69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:10.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.749+0000 7fb95de00640 1 --2- 172.21.15.102:0/686098625 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb9600cd740 0x7fb9600c69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.749+0000 7fb95de00640 1 -- 172.21.15.102:0/686098625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9600d0d40 msgr2=0x7fb9600c6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:10.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.749+0000 7fb95de00640 1 --2- 172.21.15.102:0/686098625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9600d0d40 0x7fb9600c6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.749+0000 7fb95de00640 1 -- 172.21.15.102:0/686098625 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb958009b30 con 0x7fb9600b5aa0 2024-03-23T03:49:10.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.749+0000 7fb95e800640 1 --2- 172.21.15.102:0/686098625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9600d0d40 0x7fb9600c6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:10.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.749+0000 7fb955400640 1 --2- 172.21.15.102:0/686098625 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb9600cd740 0x7fb9600c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:10.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.749+0000 7fb95de00640 1 --2- 172.21.15.102:0/686098625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9600b5aa0 0x7fb9600c6460 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb95000c7f0 tx=0x7fb95000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:10.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.749+0000 7fb956a00640 1 -- 172.21.15.102:0/686098625 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb950004220 con 0x7fb9600b5aa0 2024-03-23T03:49:10.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.749+0000 7fb956a00640 1 -- 172.21.15.102:0/686098625 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9500043a0 con 0x7fb9600b5aa0 2024-03-23T03:49:10.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.749+0000 7fb964c00640 1 -- 172.21.15.102:0/686098625 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9600cb0a0 con 0x7fb9600b5aa0 2024-03-23T03:49:10.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.749+0000 7fb956a00640 1 -- 172.21.15.102:0/686098625 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb950007860 con 0x7fb9600b5aa0 2024-03-23T03:49:10.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.749+0000 7fb964c00640 1 -- 172.21.15.102:0/686098625 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9600cb8a0 con 0x7fb9600b5aa0 2024-03-23T03:49:10.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.750+0000 7fb964c00640 1 -- 172.21.15.102:0/686098625 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9600d1180 con 0x7fb9600b5aa0 2024-03-23T03:49:10.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.752+0000 7fb956a00640 1 -- 172.21.15.102:0/686098625 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb950007bd0 con 0x7fb9600b5aa0 2024-03-23T03:49:10.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.753+0000 7fb956a00640 1 --2- 172.21.15.102:0/686098625 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb93c06ac00 0x7fb93c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:10.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.753+0000 7fb956a00640 1 -- 172.21.15.102:0/686098625 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb950089520 con 0x7fb9600b5aa0 2024-03-23T03:49:10.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.753+0000 7fb955400640 1 --2- 172.21.15.102:0/686098625 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb93c06ac00 0x7fb93c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:10.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.753+0000 7fb955400640 1 --2- 172.21.15.102:0/686098625 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb93c06ac00 0x7fb93c06d0c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb9600cff40 tx=0x7fb948008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:10.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.757+0000 7fb956a00640 1 -- 172.21.15.102:0/686098625 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb9600d1180 con 0x7fb9600b5aa0 2024-03-23T03:49:10.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.940+0000 7fb964c00640 1 -- 172.21.15.102:0/686098625 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb9600d1180 con 0x7fb9600b5aa0 2024-03-23T03:49:10.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.941+0000 7fb956a00640 1 -- 172.21.15.102:0/686098625 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb9600d1180 con 0x7fb9600b5aa0 2024-03-23T03:49:10.942 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:49:10.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.943+0000 7fb964c00640 1 -- 172.21.15.102:0/686098625 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb93c06ac00 msgr2=0x7fb93c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:10.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.943+0000 7fb964c00640 1 --2- 172.21.15.102:0/686098625 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb93c06ac00 0x7fb93c06d0c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fb9600cff40 tx=0x7fb948008040 comp rx=0 tx=0).stop 2024-03-23T03:49:10.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.943+0000 7fb964c00640 1 -- 172.21.15.102:0/686098625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9600b5aa0 msgr2=0x7fb9600c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:10.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.943+0000 7fb964c00640 1 --2- 172.21.15.102:0/686098625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9600b5aa0 0x7fb9600c6460 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fb95000c7f0 tx=0x7fb95000ccb0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.944+0000 7fb964c00640 1 -- 172.21.15.102:0/686098625 shutdown_connections 2024-03-23T03:49:10.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.944+0000 7fb964c00640 1 --2- 172.21.15.102:0/686098625 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb93c06ac00 0x7fb93c06d0c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.944+0000 7fb964c00640 1 --2- 172.21.15.102:0/686098625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9600d0d40 0x7fb9600c6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.944+0000 7fb964c00640 1 --2- 172.21.15.102:0/686098625 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb9600cd740 0x7fb9600c69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.944+0000 7fb964c00640 1 --2- 172.21.15.102:0/686098625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9600b5aa0 0x7fb9600c6460 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.944+0000 7fb964c00640 1 -- 172.21.15.102:0/686098625 >> 172.21.15.102:0/686098625 conn(0x7fb9600cf5d0 msgr2=0x7fb96015e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.944+0000 7fb964c00640 1 -- 172.21.15.102:0/686098625 shutdown_connections 2024-03-23T03:49:10.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:10.944+0000 7fb964c00640 1 -- 172.21.15.102:0/686098625 wait complete. 2024-03-23T03:49:10.962 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:49:11.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.156+0000 7f1393c00640 1 Processor -- start 2024-03-23T03:49:11.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.157+0000 7f1393c00640 1 -- start start 2024-03-23T03:49:11.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.158+0000 7f1393c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f138c155080 0x7f138c0ce7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:11.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.158+0000 7f1393c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f138c15ed80 con 0x7f138c15f210 2024-03-23T03:49:11.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.158+0000 7f1393c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f138c155460 con 0x7f138c15b6b0 2024-03-23T03:49:11.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.158+0000 7f1393c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f138c15c000 con 0x7f138c155080 2024-03-23T03:49:11.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.159+0000 7f1383e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f138c15f210 0x7f138c15afa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57286/0 (socket says 172.21.15.102:57286) 2024-03-23T03:49:11.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.159+0000 7f1383e00640 1 -- 172.21.15.102:0/2802457342 learned_addr learned my addr 172.21.15.102:0/2802457342 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:11.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.159+0000 7f1391200640 1 --2- 172.21.15.102:0/2802457342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f138c155080 0x7f138c0ce7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.159+0000 7f138be00640 1 -- 172.21.15.102:0/2802457342 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648042629 0 0) 0x7f138c15ed80 con 0x7f138c15f210 2024-03-23T03:49:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.159+0000 7f138be00640 1 -- 172.21.15.102:0/2802457342 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1370003520 con 0x7f138c15f210 2024-03-23T03:49:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.160+0000 7f138be00640 1 -- 172.21.15.102:0/2802457342 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 124711832 0 0) 0x7f1370003520 con 0x7f138c15f210 2024-03-23T03:49:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.160+0000 7f138be00640 1 -- 172.21.15.102:0/2802457342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f138c155080 msgr2=0x7f138c0ce7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.160+0000 7f138be00640 1 --2- 172.21.15.102:0/2802457342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f138c155080 0x7f138c0ce7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.160+0000 7f138be00640 1 -- 172.21.15.102:0/2802457342 >> v1:172.21.15.173:6789/0 conn(0x7f138c15b6b0 legacy=0x7f138c06ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.160+0000 7f138be00640 1 -- 172.21.15.102:0/2802457342 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c15bd20 con 0x7f138c15f210 2024-03-23T03:49:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.160+0000 7f138be00640 1 -- 172.21.15.102:0/2802457342 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f1374001670 con 0x7f138c15f210 2024-03-23T03:49:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.160+0000 7f138be00640 1 -- 172.21.15.102:0/2802457342 >> v1:172.21.15.102:6789/0 conn(0x7f138c15f210 legacy=0x7f138c15afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.160+0000 7f138be00640 1 --2- 172.21.15.102:0/2802457342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f138c155080 0x7f13700039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.160+0000 7f138be00640 1 --2- 172.21.15.102:0/2802457342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1370003f20 0x7f13700047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.160+0000 7f138be00640 1 --2- 172.21.15.102:0/2802457342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1370004d00 0x7f1370005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:11.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.160+0000 7f138be00640 1 -- 172.21.15.102:0/2802457342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c15c000 con 0x7f1370003f20 2024-03-23T03:49:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.160+0000 7f138be00640 1 -- 172.21.15.102:0/2802457342 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137000ba60 con 0x7f1370004d00 2024-03-23T03:49:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.160+0000 7f138be00640 1 -- 172.21.15.102:0/2802457342 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137000bc20 con 0x7f138c155080 2024-03-23T03:49:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.160+0000 7f1383e00640 1 --2- 172.21.15.102:0/2802457342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1370003f20 0x7f13700047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.160+0000 7f1391200640 1 --2- 172.21.15.102:0/2802457342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f138c155080 0x7f13700039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.160+0000 7f1391c00640 1 --2- 172.21.15.102:0/2802457342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1370004d00 0x7f1370005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.161+0000 7f1383e00640 1 -- 172.21.15.102:0/2802457342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f138c155080 msgr2=0x7f13700039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.161+0000 7f1383e00640 1 --2- 172.21.15.102:0/2802457342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f138c155080 0x7f13700039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.161+0000 7f1383e00640 1 -- 172.21.15.102:0/2802457342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1370004d00 msgr2=0x7f1370005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.161+0000 7f1383e00640 1 --2- 172.21.15.102:0/2802457342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1370004d00 0x7f1370005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.161+0000 7f1383e00640 1 -- 172.21.15.102:0/2802457342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1370003520 con 0x7f1370003f20 2024-03-23T03:49:11.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.161+0000 7f1391200640 1 --2- 172.21.15.102:0/2802457342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f138c155080 0x7f13700039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:49:11.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.162+0000 7f1383e00640 1 --2- 172.21.15.102:0/2802457342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1370003f20 0x7f13700047c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f137400c240 tx=0x7f13700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e1b83946849b905 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:11.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.162+0000 7f138be00640 1 -- 172.21.15.102:0/2802457342 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1374005ad0 con 0x7f1370003f20 2024-03-23T03:49:11.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.162+0000 7f138be00640 1 -- 172.21.15.102:0/2802457342 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1374005c50 con 0x7f1370003f20 2024-03-23T03:49:11.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.162+0000 7f138be00640 1 -- 172.21.15.102:0/2802457342 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137402c8b0 con 0x7f1370003f20 2024-03-23T03:49:11.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.163+0000 7f1393c00640 1 -- 172.21.15.102:0/2802457342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1370003f20 msgr2=0x7f13700047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.163+0000 7f1393c00640 1 --2- 172.21.15.102:0/2802457342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1370003f20 0x7f13700047c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f137400c240 tx=0x7f13700057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.163+0000 7f1391c00640 1 -- 172.21.15.102:0/2802457342 reap_dead start 2024-03-23T03:49:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.163+0000 7f1393c00640 1 -- 172.21.15.102:0/2802457342 shutdown_connections 2024-03-23T03:49:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.163+0000 7f1393c00640 1 -- 172.21.15.102:0/2802457342 >> 172.21.15.102:0/2802457342 conn(0x7f138c06d3a0 msgr2=0x7f138c06d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.163+0000 7f1393c00640 1 -- 172.21.15.102:0/2802457342 shutdown_connections 2024-03-23T03:49:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.163+0000 7f1393c00640 1 -- 172.21.15.102:0/2802457342 wait complete. 2024-03-23T03:49:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.164+0000 7f1393c00640 1 Processor -- start 2024-03-23T03:49:11.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.165+0000 7f1393c00640 1 -- start start 2024-03-23T03:49:11.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.165+0000 7f1393c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1370003f20 0x7f138c0adf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.166+0000 7f1393c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f138c1e4e00 0x7f138c1e5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.166+0000 7f1393c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f138c1f66d0 0x7f138c1f6b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.166+0000 7f1393c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c15bb40 con 0x7f138c1f66d0 2024-03-23T03:49:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.166+0000 7f1393c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f138c1e7230 con 0x7f138c1e4e00 2024-03-23T03:49:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.166+0000 7f1393c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f138c0ae460 con 0x7f1370003f20 2024-03-23T03:49:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.166+0000 7f1391c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f138c1f66d0 0x7f138c1f6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.166+0000 7f1391c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f138c1f66d0 0x7f138c1f6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53850/0 (socket says 172.21.15.102:53850) 2024-03-23T03:49:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.166+0000 7f1391c00640 1 -- 172.21.15.102:0/1230358788 learned_addr learned my addr 172.21.15.102:0/1230358788 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.166+0000 7f1391200640 1 --2- 172.21.15.102:0/1230358788 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1370003f20 0x7f138c0adf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.166+0000 7f1391c00640 1 -- 172.21.15.102:0/1230358788 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1370003f20 msgr2=0x7f138c0adf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.166+0000 7f1383e00640 1 --2- 172.21.15.102:0/1230358788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f138c1e4e00 0x7f138c1e5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.166+0000 7f1391c00640 1 --2- 172.21.15.102:0/1230358788 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1370003f20 0x7f138c0adf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.166+0000 7f1391c00640 1 -- 172.21.15.102:0/1230358788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f138c1e4e00 msgr2=0x7f138c1e5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:11.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.166+0000 7f1391c00640 1 --2- 172.21.15.102:0/1230358788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f138c1e4e00 0x7f138c1e5280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.166+0000 7f1391c00640 1 -- 172.21.15.102:0/1230358788 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c1e7f60 con 0x7f138c1f66d0 2024-03-23T03:49:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.167+0000 7f1391c00640 1 --2- 172.21.15.102:0/1230358788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f138c1f66d0 0x7f138c1f6b50 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f138c0cdf80 tx=0x7f138400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.167+0000 7f1389600640 1 -- 172.21.15.102:0/1230358788 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138400eed0 con 0x7f138c1f66d0 2024-03-23T03:49:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.167+0000 7f1389600640 1 -- 172.21.15.102:0/1230358788 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1384003970 con 0x7f138c1f66d0 2024-03-23T03:49:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.167+0000 7f1393c00640 1 -- 172.21.15.102:0/1230358788 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f138c1e5d60 con 0x7f138c1f66d0 2024-03-23T03:49:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.167+0000 7f1389600640 1 -- 172.21.15.102:0/1230358788 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1384003d20 con 0x7f138c1f66d0 2024-03-23T03:49:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.167+0000 7f1393c00640 1 -- 172.21.15.102:0/1230358788 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f138c1e5a80 con 0x7f138c1f66d0 2024-03-23T03:49:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.168+0000 7f1393c00640 1 -- 172.21.15.102:0/1230358788 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f138c1e80f0 con 0x7f138c1f66d0 2024-03-23T03:49:11.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.169+0000 7f1389600640 1 -- 172.21.15.102:0/1230358788 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f13840050d0 con 0x7f138c1f66d0 2024-03-23T03:49:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.170+0000 7f1389600640 1 --2- 172.21.15.102:0/1230358788 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f136806ac30 0x7f136806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.170+0000 7f1389600640 1 -- 172.21.15.102:0/1230358788 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1384089240 con 0x7f138c1f66d0 2024-03-23T03:49:11.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.170+0000 7f1391200640 1 --2- 172.21.15.102:0/1230358788 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f136806ac30 0x7f136806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:11.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.171+0000 7f1391200640 1 --2- 172.21.15.102:0/1230358788 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f136806ac30 0x7f136806d0f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f137c0099b0 tx=0x7f137c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:11.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.174+0000 7f1389600640 1 -- 172.21.15.102:0/1230358788 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1384054240 con 0x7f138c1f66d0 2024-03-23T03:49:11.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.380+0000 7f1393c00640 1 -- 172.21.15.102:0/1230358788 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f138c066bb0 con 0x7f138c1f66d0 2024-03-23T03:49:11.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.381+0000 7f1389600640 1 -- 172.21.15.102:0/1230358788 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v55) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1384057860 con 0x7f138c1f66d0 2024-03-23T03:49:11.382 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:49:11.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.384+0000 7f1393c00640 1 -- 172.21.15.102:0/1230358788 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f136806ac30 msgr2=0x7f136806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:11.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.384+0000 7f1393c00640 1 --2- 172.21.15.102:0/1230358788 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f136806ac30 0x7f136806d0f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f137c0099b0 tx=0x7f137c008040 comp rx=0 tx=0).stop 2024-03-23T03:49:11.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.384+0000 7f1393c00640 1 -- 172.21.15.102:0/1230358788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f138c1f66d0 msgr2=0x7f138c1f6b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:11.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.384+0000 7f1393c00640 1 --2- 172.21.15.102:0/1230358788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f138c1f66d0 0x7f138c1f6b50 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f138c0cdf80 tx=0x7f138400eb90 comp rx=0 tx=0).stop 2024-03-23T03:49:11.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.384+0000 7f1393c00640 1 -- 172.21.15.102:0/1230358788 shutdown_connections 2024-03-23T03:49:11.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.384+0000 7f1393c00640 1 --2- 172.21.15.102:0/1230358788 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f136806ac30 0x7f136806d0f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:11.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.384+0000 7f1393c00640 1 --2- 172.21.15.102:0/1230358788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f138c1f66d0 0x7f138c1f6b50 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:11.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.384+0000 7f1393c00640 1 --2- 172.21.15.102:0/1230358788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f138c1e4e00 0x7f138c1e5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:11.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.384+0000 7f1393c00640 1 --2- 172.21.15.102:0/1230358788 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1370003f20 0x7f138c0adf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:11.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.384+0000 7f1393c00640 1 -- 172.21.15.102:0/1230358788 >> 172.21.15.102:0/1230358788 conn(0x7f138c06d3a0 msgr2=0x7f138c0ce790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:11.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.384+0000 7f1393c00640 1 -- 172.21.15.102:0/1230358788 shutdown_connections 2024-03-23T03:49:11.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.384+0000 7f1393c00640 1 -- 172.21.15.102:0/1230358788 wait complete. 2024-03-23T03:49:11.403 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-23T03:49:11.403 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:11.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-03-23T03:49:11.512 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:49:11.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:11.512 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T03:49:11.563 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:49:11.563 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:11.618 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:49:11.619 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:49:11.619 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:11.675 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:49:11.675 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:49:11.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.890+0000 7fb9af800640 1 Processor -- start 2024-03-23T03:49:11.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.890+0000 7fb9af800640 1 -- start start 2024-03-23T03:49:11.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.892+0000 7fb9af800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb9a815cd40 0x7fb9a80cf2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:11.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.892+0000 7fb9af800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a80d0980 con 0x7fb9a815c960 2024-03-23T03:49:11.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.892+0000 7fb9af800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a80d0f40 con 0x7fb9a80cf800 2024-03-23T03:49:11.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.892+0000 7fb9af800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9a80cfcb0 con 0x7fb9a815cd40 2024-03-23T03:49:11.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.892+0000 7fb9ace00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb9a815c960 0x7fb9a8160670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57288/0 (socket says 172.21.15.102:57288) 2024-03-23T03:49:11.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.892+0000 7fb9ace00640 1 -- 172.21.15.102:0/774335589 learned_addr learned my addr 172.21.15.102:0/774335589 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:11.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.892+0000 7fb9a7400640 1 -- 172.21.15.102:0/774335589 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665332283 0 0) 0x7fb9a80d0980 con 0x7fb9a815c960 2024-03-23T03:49:11.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.892+0000 7fb9a7400640 1 -- 172.21.15.102:0/774335589 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb98c003520 con 0x7fb9a815c960 2024-03-23T03:49:11.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.893+0000 7fb9a7e00640 1 --2- 172.21.15.102:0/774335589 >> v2:172.21.15.173:3300/0 conn(0x7fb9a815cd40 0x7fb9a80cf2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.893+0000 7fb9a7400640 1 -- 172.21.15.102:0/774335589 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2654526780 0 0) 0x7fb98c003520 con 0x7fb9a815c960 2024-03-23T03:49:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.893+0000 7fb9a7400640 1 -- 172.21.15.102:0/774335589 >> v2:172.21.15.173:3300/0 conn(0x7fb9a815cd40 msgr2=0x7fb9a80cf2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.893+0000 7fb9a7400640 1 --2- 172.21.15.102:0/774335589 >> v2:172.21.15.173:3300/0 conn(0x7fb9a815cd40 0x7fb9a80cf2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.893+0000 7fb9a7400640 1 -- 172.21.15.102:0/774335589 >> v1:172.21.15.173:6789/0 conn(0x7fb9a80cf800 legacy=0x7fb9a80d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.893+0000 7fb9a7400640 1 -- 172.21.15.102:0/774335589 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a80b87e0 con 0x7fb9a815c960 2024-03-23T03:49:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.893+0000 7fb9a7400640 1 -- 172.21.15.102:0/774335589 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fb998001670 con 0x7fb9a815c960 2024-03-23T03:49:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.893+0000 7fb9a7400640 1 -- 172.21.15.102:0/774335589 >> v1:172.21.15.102:6789/0 conn(0x7fb9a815c960 legacy=0x7fb9a8160670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.893+0000 7fb9a7400640 1 --2- 172.21.15.102:0/774335589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb98c0039c0 0x7fb98c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.893+0000 7fb9a7400640 1 --2- 172.21.15.102:0/774335589 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb98c0047c0 0x7fb98c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.893+0000 7fb9a7400640 1 --2- 172.21.15.102:0/774335589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb98c0052a0 0x7fb98c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.893+0000 7fb9a7400640 1 -- 172.21.15.102:0/774335589 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a80cfcb0 con 0x7fb98c0052a0 2024-03-23T03:49:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.893+0000 7fb9a7400640 1 -- 172.21.15.102:0/774335589 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb98c00c1e0 con 0x7fb98c0039c0 2024-03-23T03:49:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.893+0000 7fb9a7400640 1 -- 172.21.15.102:0/774335589 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb98c00c3a0 con 0x7fb98c0047c0 2024-03-23T03:49:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.893+0000 7fb9a7e00640 1 --2- 172.21.15.102:0/774335589 >> v2:172.21.15.173:3300/0 conn(0x7fb9a815cd40 0x7fb9a80cf2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:11.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.893+0000 7fb9ad800640 1 --2- 172.21.15.102:0/774335589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb98c0052a0 0x7fb98c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.893+0000 7fb9ace00640 1 --2- 172.21.15.102:0/774335589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb98c0039c0 0x7fb98c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.894+0000 7fb9a7e00640 1 --2- 172.21.15.102:0/774335589 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb98c0047c0 0x7fb98c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.894+0000 7fb9ad800640 1 -- 172.21.15.102:0/774335589 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb98c0047c0 msgr2=0x7fb98c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.894+0000 7fb9ad800640 1 --2- 172.21.15.102:0/774335589 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb98c0047c0 0x7fb98c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.894+0000 7fb9ad800640 1 -- 172.21.15.102:0/774335589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb98c0039c0 msgr2=0x7fb98c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.894+0000 7fb9ad800640 1 --2- 172.21.15.102:0/774335589 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb98c0039c0 0x7fb98c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.894+0000 7fb9ad800640 1 -- 172.21.15.102:0/774335589 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98c003520 con 0x7fb98c0052a0 2024-03-23T03:49:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.894+0000 7fb9ad800640 1 -- 172.21.15.102:0/774335589 reap_dead start 2024-03-23T03:49:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.895+0000 7fb9ad800640 1 --2- 172.21.15.102:0/774335589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb98c0052a0 0x7fb98c00b980 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0004480 tx=0x7fb98c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56a5d18d662b5ae1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.895+0000 7fb9a7400640 1 -- 172.21.15.102:0/774335589 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a0006b50 con 0x7fb98c0052a0 2024-03-23T03:49:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.895+0000 7fb9a7400640 1 -- 172.21.15.102:0/774335589 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a0006cd0 con 0x7fb98c0052a0 2024-03-23T03:49:11.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.895+0000 7fb9a7400640 1 -- 172.21.15.102:0/774335589 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a0007260 con 0x7fb98c0052a0 2024-03-23T03:49:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.895+0000 7fb9af800640 1 -- 172.21.15.102:0/774335589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb98c0052a0 msgr2=0x7fb98c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.895+0000 7fb9af800640 1 --2- 172.21.15.102:0/774335589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb98c0052a0 0x7fb98c00b980 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0004480 tx=0x7fb98c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.895+0000 7fb9af800640 1 -- 172.21.15.102:0/774335589 shutdown_connections 2024-03-23T03:49:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.895+0000 7fb9af800640 1 --2- 172.21.15.102:0/774335589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb98c0052a0 0x7fb98c00b980 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.895+0000 7fb9af800640 1 -- 172.21.15.102:0/774335589 >> 172.21.15.102:0/774335589 conn(0x7fb9a80713c0 msgr2=0x7fb9a80717d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.896+0000 7fb9af800640 1 -- 172.21.15.102:0/774335589 shutdown_connections 2024-03-23T03:49:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.896+0000 7fb9af800640 1 -- 172.21.15.102:0/774335589 wait complete. 2024-03-23T03:49:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.897+0000 7fb9af800640 1 Processor -- start 2024-03-23T03:49:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.897+0000 7fb9af800640 1 -- start start 2024-03-23T03:49:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.898+0000 7fb9af800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb98c0052a0 0x7fb9a80d2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.898+0000 7fb9af800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9a815f9d0 0x7fb9a80d1e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:11.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.898+0000 7fb9af800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9a8151080 0x7fb9a8152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.898+0000 7fb9af800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a80b52e0 con 0x7fb9a8151080 2024-03-23T03:49:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.898+0000 7fb9af800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a80d13d0 con 0x7fb9a815f9d0 2024-03-23T03:49:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.898+0000 7fb9af800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a80d2360 con 0x7fb98c0052a0 2024-03-23T03:49:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.899+0000 7fb9ad800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9a8151080 0x7fb9a8152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.899+0000 7fb9ad800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9a8151080 0x7fb9a8152940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53870/0 (socket says 172.21.15.102:53870) 2024-03-23T03:49:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.899+0000 7fb9ad800640 1 -- 172.21.15.102:0/1363547574 learned_addr learned my addr 172.21.15.102:0/1363547574 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.899+0000 7fb9ace00640 1 --2- 172.21.15.102:0/1363547574 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb98c0052a0 0x7fb9a80d2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.899+0000 7fb9ad800640 1 -- 172.21.15.102:0/1363547574 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb98c0052a0 msgr2=0x7fb9a80d2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.899+0000 7fb9ad800640 1 --2- 172.21.15.102:0/1363547574 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb98c0052a0 0x7fb9a80d2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.899+0000 7fb9ad800640 1 -- 172.21.15.102:0/1363547574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9a815f9d0 msgr2=0x7fb9a80d1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.899+0000 7fb9ad800640 1 --2- 172.21.15.102:0/1363547574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9a815f9d0 0x7fb9a80d1e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.899+0000 7fb9ad800640 1 -- 172.21.15.102:0/1363547574 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a80d3320 con 0x7fb9a8151080 2024-03-23T03:49:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.899+0000 7fb9ace00640 1 --2- 172.21.15.102:0/1363547574 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb98c0052a0 0x7fb9a80d2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:11.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.899+0000 7fb9ad800640 1 --2- 172.21.15.102:0/1363547574 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9a8151080 0x7fb9a8152940 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb9a00045a0 tx=0x7fb9a0003a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:11.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.900+0000 7fb9a4c00640 1 -- 172.21.15.102:0/1363547574 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a003d070 con 0x7fb9a8151080 2024-03-23T03:49:11.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.900+0000 7fb9a4c00640 1 -- 172.21.15.102:0/1363547574 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a0007a80 con 0x7fb9a8151080 2024-03-23T03:49:11.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.900+0000 7fb9af800640 1 -- 172.21.15.102:0/1363547574 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a80d3600 con 0x7fb9a8151080 2024-03-23T03:49:11.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.900+0000 7fb9a4c00640 1 -- 172.21.15.102:0/1363547574 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a0043660 con 0x7fb9a8151080 2024-03-23T03:49:11.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.900+0000 7fb9af800640 1 -- 172.21.15.102:0/1363547574 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a80d1900 con 0x7fb9a8151080 2024-03-23T03:49:11.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.901+0000 7fb9af800640 1 -- 172.21.15.102:0/1363547574 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a80d3840 con 0x7fb9a8151080 2024-03-23T03:49:11.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.902+0000 7fb9a4c00640 1 -- 172.21.15.102:0/1363547574 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb9a0041070 con 0x7fb9a8151080 2024-03-23T03:49:11.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.902+0000 7fb9a4c00640 1 --2- 172.21.15.102:0/1363547574 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb98406ac30 0x7fb98406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:11.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.903+0000 7fb9a4c00640 1 -- 172.21.15.102:0/1363547574 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb9a00aa690 con 0x7fb9a8151080 2024-03-23T03:49:11.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.903+0000 7fb9ace00640 1 --2- 172.21.15.102:0/1363547574 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb98406ac30 0x7fb98406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:11.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.903+0000 7fb9ace00640 1 --2- 172.21.15.102:0/1363547574 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb98406ac30 0x7fb98406d0f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb9a80d2c20 tx=0x7fb998000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:11.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:11.907+0000 7fb9a4c00640 1 -- 172.21.15.102:0/1363547574 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb9a00757b0 con 0x7fb9a8151080 2024-03-23T03:49:12.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.141+0000 7fb9af800640 1 -- 172.21.15.102:0/1363547574 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9a80d0320 con 0x7fb9a8151080 2024-03-23T03:49:12.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.142+0000 7fb9a4c00640 1 -- 172.21.15.102:0/1363547574 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fb9a0007df0 con 0x7fb9a8151080 2024-03-23T03:49:12.143 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:12.144 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5661,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/1848167718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":1848167718},{"type":"v1","addr":"172.21.15.102:6835","nonce":1848167718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/16304019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":16304019},{"type":"v1","addr":"172.21.15.102:6837","nonce":16304019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5673,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6834/2030662139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2030662139},{"type":"v1","addr":"172.21.15.173:6834","nonce":2030662139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:48:56.221998+0000","modified":"2024-03-23T03:48:59.250212+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5676":{"gid":5676,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6835/3269327930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":3269327930},{"type":"v1","addr":"172.21.15.173:6835","nonce":3269327930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-23T03:49:12.144 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 24 2024-03-23T03:49:12.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.146+0000 7fb9af800640 1 -- 172.21.15.102:0/1363547574 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb98406ac30 msgr2=0x7fb98406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:12.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.146+0000 7fb9af800640 1 --2- 172.21.15.102:0/1363547574 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb98406ac30 0x7fb98406d0f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb9a80d2c20 tx=0x7fb998000ed0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.146+0000 7fb9af800640 1 -- 172.21.15.102:0/1363547574 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9a8151080 msgr2=0x7fb9a8152940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:12.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.146+0000 7fb9af800640 1 --2- 172.21.15.102:0/1363547574 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9a8151080 0x7fb9a8152940 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fb9a00045a0 tx=0x7fb9a0003a70 comp rx=0 tx=0).stop 2024-03-23T03:49:12.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.146+0000 7fb9af800640 1 -- 172.21.15.102:0/1363547574 shutdown_connections 2024-03-23T03:49:12.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.146+0000 7fb9af800640 1 --2- 172.21.15.102:0/1363547574 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb98406ac30 0x7fb98406d0f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.146+0000 7fb9af800640 1 --2- 172.21.15.102:0/1363547574 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9a8151080 0x7fb9a8152940 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.146+0000 7fb9af800640 1 --2- 172.21.15.102:0/1363547574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9a815f9d0 0x7fb9a80d1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.146+0000 7fb9af800640 1 --2- 172.21.15.102:0/1363547574 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb98c0052a0 0x7fb9a80d2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.146+0000 7fb9af800640 1 -- 172.21.15.102:0/1363547574 >> 172.21.15.102:0/1363547574 conn(0x7fb9a80713c0 msgr2=0x7fb9a815bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:12.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.146+0000 7fb9af800640 1 -- 172.21.15.102:0/1363547574 shutdown_connections 2024-03-23T03:49:12.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.146+0000 7fb9af800640 1 -- 172.21.15.102:0/1363547574 wait complete. 2024-03-23T03:49:12.167 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:49:12.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.378+0000 7fec39e00640 1 Processor -- start 2024-03-23T03:49:12.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.378+0000 7fec39e00640 1 -- start start 2024-03-23T03:49:12.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.379+0000 7fec39e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec3413faa0 0x7fec34160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:12.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.379+0000 7fec39e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fec34160800 0x7fec340b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:12.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.379+0000 7fec39e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec340b2030 con 0x7fec340b1be0 2024-03-23T03:49:12.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.380+0000 7fec39e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec340b38a0 con 0x7fec34160800 2024-03-23T03:49:12.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.380+0000 7fec39e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec340b3b80 con 0x7fec3413faa0 2024-03-23T03:49:12.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.380+0000 7fec2be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fec34160800 0x7fec340b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:12.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.380+0000 7fec33400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fec340b1be0 0x7fec341422b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57302/0 (socket says 172.21.15.102:57302) 2024-03-23T03:49:12.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.380+0000 7fec33400640 1 -- 172.21.15.102:0/1783237302 learned_addr learned my addr 172.21.15.102:0/1783237302 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:12.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.380+0000 7fec2be00640 1 -- 172.21.15.102:0/1783237302 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec3413faa0 msgr2=0x7fec34160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:12.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.380+0000 7fec32a00640 1 --2- 172.21.15.102:0/1783237302 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec3413faa0 0x7fec34160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:12.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.380+0000 7fec32000640 1 -- 172.21.15.102:0/1783237302 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004490434 0 0) 0x7fec2c001490 con 0x7fec340b1be0 2024-03-23T03:49:12.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.380+0000 7fec2be00640 1 --2- 172.21.15.102:0/1783237302 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec3413faa0 0x7fec34160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.380+0000 7fec2be00640 1 -- 172.21.15.102:0/1783237302 >> v1:172.21.15.102:6789/0 conn(0x7fec340b1be0 legacy=0x7fec341422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:12.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.380+0000 7fec2be00640 1 -- 172.21.15.102:0/1783237302 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec340b3e60 con 0x7fec34160800 2024-03-23T03:49:12.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.380+0000 7fec32a00640 1 --2- 172.21.15.102:0/1783237302 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec3413faa0 0x7fec34160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:12.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.381+0000 7fec2be00640 1 --2- 172.21.15.102:0/1783237302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec34160800 0x7fec340b16a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0097a0 tx=0x7fec1c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1baef5ba8854660 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:12.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.382+0000 7fec32000640 1 -- 172.21.15.102:0/1783237302 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c031830 con 0x7fec34160800 2024-03-23T03:49:12.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.382+0000 7fec32000640 1 -- 172.21.15.102:0/1783237302 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1c031990 con 0x7fec34160800 2024-03-23T03:49:12.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.382+0000 7fec32000640 1 -- 172.21.15.102:0/1783237302 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c031d20 con 0x7fec34160800 2024-03-23T03:49:12.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.382+0000 7fec39e00640 1 -- 172.21.15.102:0/1783237302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec34160800 msgr2=0x7fec340b16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:12.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.382+0000 7fec39e00640 1 --2- 172.21.15.102:0/1783237302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec34160800 0x7fec340b16a0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0097a0 tx=0x7fec1c02b8d0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.382+0000 7fec39e00640 1 -- 172.21.15.102:0/1783237302 shutdown_connections 2024-03-23T03:49:12.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.382+0000 7fec39e00640 1 --2- 172.21.15.102:0/1783237302 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec34160800 0x7fec340b16a0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.382+0000 7fec39e00640 1 --2- 172.21.15.102:0/1783237302 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec3413faa0 0x7fec34160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.382+0000 7fec39e00640 1 -- 172.21.15.102:0/1783237302 >> 172.21.15.102:0/1783237302 conn(0x7fec3406a3d0 msgr2=0x7fec340b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:12.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.383+0000 7fec39e00640 1 -- 172.21.15.102:0/1783237302 shutdown_connections 2024-03-23T03:49:12.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.383+0000 7fec39e00640 1 -- 172.21.15.102:0/1783237302 wait complete. 2024-03-23T03:49:12.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.384+0000 7fec39e00640 1 Processor -- start 2024-03-23T03:49:12.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.385+0000 7fec39e00640 1 -- start start 2024-03-23T03:49:12.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.386+0000 7fec39e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec340b1be0 0x7fec341f27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:12.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.386+0000 7fec39e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec3413faa0 0x7fec34154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:12.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.386+0000 7fec39e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec34160800 0x7fec34156150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:12.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.386+0000 7fec39e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec34154a60 con 0x7fec3413faa0 2024-03-23T03:49:12.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.386+0000 7fec39e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec34154d40 con 0x7fec34160800 2024-03-23T03:49:12.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.386+0000 7fec39e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec34155020 con 0x7fec340b1be0 2024-03-23T03:49:12.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.386+0000 7fec2be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec3413faa0 0x7fec34154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:12.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.386+0000 7fec2be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec3413faa0 0x7fec34154210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53884/0 (socket says 172.21.15.102:53884) 2024-03-23T03:49:12.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.386+0000 7fec2be00640 1 -- 172.21.15.102:0/1276206241 learned_addr learned my addr 172.21.15.102:0/1276206241 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:12.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.386+0000 7fec32a00640 1 --2- 172.21.15.102:0/1276206241 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec340b1be0 0x7fec341f27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:12.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.387+0000 7fec33400640 1 --2- 172.21.15.102:0/1276206241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec34160800 0x7fec34156150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:12.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.387+0000 7fec2be00640 1 -- 172.21.15.102:0/1276206241 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec340b1be0 msgr2=0x7fec341f27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:12.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.387+0000 7fec2be00640 1 --2- 172.21.15.102:0/1276206241 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec340b1be0 0x7fec341f27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.387+0000 7fec2be00640 1 -- 172.21.15.102:0/1276206241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec34160800 msgr2=0x7fec34156150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:12.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.387+0000 7fec2be00640 1 --2- 172.21.15.102:0/1276206241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec34160800 0x7fec34156150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.387+0000 7fec2be00640 1 -- 172.21.15.102:0/1276206241 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1c009400 con 0x7fec3413faa0 2024-03-23T03:49:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.387+0000 7fec33400640 1 --2- 172.21.15.102:0/1276206241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec34160800 0x7fec34156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.387+0000 7fec2be00640 1 --2- 172.21.15.102:0/1276206241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec3413faa0 0x7fec34154210 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0098c0 tx=0x7fec1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.387+0000 7fec2b400640 1 -- 172.21.15.102:0/1276206241 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c03ba70 con 0x7fec3413faa0 2024-03-23T03:49:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.387+0000 7fec2b400640 1 -- 172.21.15.102:0/1276206241 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1c03bbd0 con 0x7fec3413faa0 2024-03-23T03:49:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.387+0000 7fec2b400640 1 -- 172.21.15.102:0/1276206241 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c0453f0 con 0x7fec3413faa0 2024-03-23T03:49:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.387+0000 7fec39e00640 1 -- 172.21.15.102:0/1276206241 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec34155300 con 0x7fec3413faa0 2024-03-23T03:49:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.388+0000 7fec39e00640 1 -- 172.21.15.102:0/1276206241 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec34154750 con 0x7fec3413faa0 2024-03-23T03:49:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.389+0000 7fec39e00640 1 -- 172.21.15.102:0/1276206241 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec34155820 con 0x7fec3413faa0 2024-03-23T03:49:12.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.390+0000 7fec2b400640 1 -- 172.21.15.102:0/1276206241 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fec1c04e450 con 0x7fec3413faa0 2024-03-23T03:49:12.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.390+0000 7fec2b400640 1 --2- 172.21.15.102:0/1276206241 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fec1006ac00 0x7fec1006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:12.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.391+0000 7fec2b400640 1 -- 172.21.15.102:0/1276206241 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fec1c0ab770 con 0x7fec3413faa0 2024-03-23T03:49:12.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.391+0000 7fec32a00640 1 --2- 172.21.15.102:0/1276206241 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fec1006ac00 0x7fec1006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:12.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.391+0000 7fec32a00640 1 --2- 172.21.15.102:0/1276206241 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fec1006ac00 0x7fec1006d0c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fec340b48d0 tx=0x7fec24008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:12.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.396+0000 7fec2b400640 1 -- 172.21.15.102:0/1276206241 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fec34155820 con 0x7fec3413faa0 2024-03-23T03:49:12.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.589+0000 7fec39e00640 1 -- 172.21.15.102:0/1276206241 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec34155820 con 0x7fec3413faa0 2024-03-23T03:49:12.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.593+0000 7fec2b400640 1 -- 172.21.15.102:0/1276206241 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16354 (secure 0 0 0) 0x7fec34155820 con 0x7fec3413faa0 2024-03-23T03:49:12.594 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:12.594 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":55,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:57.226138+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:48:53.998390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T03:48:55.242109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:49:12.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.595+0000 7fec39e00640 1 -- 172.21.15.102:0/1276206241 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fec1006ac00 msgr2=0x7fec1006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:12.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.595+0000 7fec39e00640 1 --2- 172.21.15.102:0/1276206241 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fec1006ac00 0x7fec1006d0c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fec340b48d0 tx=0x7fec24008040 comp rx=0 tx=0).stop 2024-03-23T03:49:12.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.595+0000 7fec39e00640 1 -- 172.21.15.102:0/1276206241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec3413faa0 msgr2=0x7fec34154210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:12.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.595+0000 7fec39e00640 1 --2- 172.21.15.102:0/1276206241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec3413faa0 0x7fec34154210 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fec1c0098c0 tx=0x7fec1c0046c0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.596+0000 7fec39e00640 1 -- 172.21.15.102:0/1276206241 shutdown_connections 2024-03-23T03:49:12.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.596+0000 7fec39e00640 1 --2- 172.21.15.102:0/1276206241 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fec1006ac00 0x7fec1006d0c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.596+0000 7fec39e00640 1 --2- 172.21.15.102:0/1276206241 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec34160800 0x7fec34156150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.596+0000 7fec39e00640 1 --2- 172.21.15.102:0/1276206241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec3413faa0 0x7fec34154210 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.596+0000 7fec39e00640 1 --2- 172.21.15.102:0/1276206241 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec340b1be0 0x7fec341f27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.596+0000 7fec39e00640 1 -- 172.21.15.102:0/1276206241 >> 172.21.15.102:0/1276206241 conn(0x7fec3406a3d0 msgr2=0x7fec340ad540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:12.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.596+0000 7fec39e00640 1 -- 172.21.15.102:0/1276206241 shutdown_connections 2024-03-23T03:49:12.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.596+0000 7fec39e00640 1 -- 172.21.15.102:0/1276206241 wait complete. 2024-03-23T03:49:12.614 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:48:53.998390+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:48:55.242109+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:49:12.615 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:49:12.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.822+0000 7f551b400640 1 Processor -- start 2024-03-23T03:49:12.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.822+0000 7f551b400640 1 -- start start 2024-03-23T03:49:12.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.824+0000 7f551b400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5514160d20 0x7f5514164920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:12.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.824+0000 7f551b400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f55140cf040 0x7f55140b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:12.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.824+0000 7f551b400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5514165830 con 0x7f55140ca7b0 2024-03-23T03:49:12.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.824+0000 7f551b400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5514164f90 con 0x7f55140cf040 2024-03-23T03:49:12.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.824+0000 7f551b400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5514165270 con 0x7f5514160d20 2024-03-23T03:49:12.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.824+0000 7f5513e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f55140ca7b0 0x7f55140ce930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:57316/0 (socket says 172.21.15.102:57316) 2024-03-23T03:49:12.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.824+0000 7f5513e00640 1 -- 172.21.15.102:0/3480038173 learned_addr learned my addr 172.21.15.102:0/3480038173 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:12.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.824+0000 7f5518a00640 1 --2- 172.21.15.102:0/3480038173 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5514160d20 0x7f5514164920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:12.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5519400640 1 --2- 172.21.15.102:0/3480038173 >> v2:172.21.15.173:3300/0 conn(0x7f55140cf040 0x7f55140b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:12.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5513400640 1 -- 172.21.15.102:0/3480038173 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581822755 0 0) 0x7f5514165830 con 0x7f55140ca7b0 2024-03-23T03:49:12.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5513400640 1 -- 172.21.15.102:0/3480038173 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54fc003520 con 0x7f55140ca7b0 2024-03-23T03:49:12.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5513400640 1 -- 172.21.15.102:0/3480038173 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4130029712 0 0) 0x7f54fc003520 con 0x7f55140ca7b0 2024-03-23T03:49:12.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5513400640 1 -- 172.21.15.102:0/3480038173 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5514160d20 msgr2=0x7f5514164920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:12.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5513400640 1 --2- 172.21.15.102:0/3480038173 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5514160d20 0x7f5514164920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5513400640 1 -- 172.21.15.102:0/3480038173 >> v2:172.21.15.173:3300/0 conn(0x7f55140cf040 msgr2=0x7f55140b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:12.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5513400640 1 --2- 172.21.15.102:0/3480038173 >> v2:172.21.15.173:3300/0 conn(0x7f55140cf040 0x7f55140b1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5513400640 1 -- 172.21.15.102:0/3480038173 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5514165550 con 0x7f55140ca7b0 2024-03-23T03:49:12.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5513400640 1 -- 172.21.15.102:0/3480038173 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f5500001670 con 0x7f55140ca7b0 2024-03-23T03:49:12.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5513400640 1 -- 172.21.15.102:0/3480038173 >> v1:172.21.15.102:6789/0 conn(0x7f55140ca7b0 legacy=0x7f55140ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:12.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5513400640 1 --2- 172.21.15.102:0/3480038173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54fc003970 0x7f54fc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:12.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5513400640 1 --2- 172.21.15.102:0/3480038173 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5514160d20 0x7f54fc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:12.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5513400640 1 --2- 172.21.15.102:0/3480038173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54fc004d00 0x7f54fc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:12.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5513400640 1 -- 172.21.15.102:0/3480038173 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5514164f90 con 0x7f54fc004d00 2024-03-23T03:49:12.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5513400640 1 -- 172.21.15.102:0/3480038173 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5514165270 con 0x7f54fc003970 2024-03-23T03:49:12.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5513400640 1 -- 172.21.15.102:0/3480038173 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc00b9b0 con 0x7f5514160d20 2024-03-23T03:49:12.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.825+0000 7f5519400640 1 --2- 172.21.15.102:0/3480038173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54fc004d00 0x7f54fc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:12.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.826+0000 7f5513e00640 1 --2- 172.21.15.102:0/3480038173 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5514160d20 0x7f54fc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:12.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.826+0000 7f5519400640 1 -- 172.21.15.102:0/3480038173 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5514160d20 msgr2=0x7f54fc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:12.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.826+0000 7f5518a00640 1 --2- 172.21.15.102:0/3480038173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54fc003970 0x7f54fc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:12.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.826+0000 7f5519400640 1 --2- 172.21.15.102:0/3480038173 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5514160d20 0x7f54fc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.826+0000 7f5519400640 1 -- 172.21.15.102:0/3480038173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54fc003970 msgr2=0x7f54fc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:12.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.826+0000 7f5519400640 1 --2- 172.21.15.102:0/3480038173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f54fc003970 0x7f54fc003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.826+0000 7f5519400640 1 -- 172.21.15.102:0/3480038173 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc003520 con 0x7f54fc004d00 2024-03-23T03:49:12.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.826+0000 7f5513e00640 1 --2- 172.21.15.102:0/3480038173 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5514160d20 0x7f54fc0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:12.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.827+0000 7f5519400640 1 --2- 172.21.15.102:0/3480038173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54fc004d00 0x7f54fc005180 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f550c009f90 tx=0x7f54fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2bffbcc895bc6de server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:12.829 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.827+0000 7f5513400640 1 -- 172.21.15.102:0/3480038173 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c02cc20 con 0x7f54fc004d00 2024-03-23T03:49:12.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.827+0000 7f5513400640 1 -- 172.21.15.102:0/3480038173 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550c00f040 con 0x7f54fc004d00 2024-03-23T03:49:12.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.827+0000 7f5513400640 1 -- 172.21.15.102:0/3480038173 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c034440 con 0x7f54fc004d00 2024-03-23T03:49:12.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.828+0000 7f551b400640 1 -- 172.21.15.102:0/3480038173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54fc004d00 msgr2=0x7f54fc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:12.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.828+0000 7f551b400640 1 --2- 172.21.15.102:0/3480038173 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f54fc004d00 0x7f54fc005180 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f550c009f90 tx=0x7f54fc0057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.828+0000 7f5519400640 1 -- 172.21.15.102:0/3480038173 reap_dead start 2024-03-23T03:49:12.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.828+0000 7f551b400640 1 -- 172.21.15.102:0/3480038173 shutdown_connections 2024-03-23T03:49:12.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.828+0000 7f551b400640 1 -- 172.21.15.102:0/3480038173 >> 172.21.15.102:0/3480038173 conn(0x7f551406a0b0 msgr2=0x7f551406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:12.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.828+0000 7f551b400640 1 -- 172.21.15.102:0/3480038173 shutdown_connections 2024-03-23T03:49:12.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.828+0000 7f551b400640 1 -- 172.21.15.102:0/3480038173 wait complete. 2024-03-23T03:49:12.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.830+0000 7f551b400640 1 Processor -- start 2024-03-23T03:49:12.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.830+0000 7f551b400640 1 -- start start 2024-03-23T03:49:12.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.831+0000 7f551b400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f54fc004d00 0x7f5514145a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:12.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.831+0000 7f551b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55140ccb90 0x7f55140cbc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:12.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.831+0000 7f551b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55140b0490 0x7f55140b0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:12.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.831+0000 7f551b400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55140b0e50 con 0x7f55140ccb90 2024-03-23T03:49:12.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.831+0000 7f551b400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551415ea70 con 0x7f55140b0490 2024-03-23T03:49:12.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.831+0000 7f551b400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5514146490 con 0x7f54fc004d00 2024-03-23T03:49:12.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.831+0000 7f5513e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55140ccb90 0x7f55140cbc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:12.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.831+0000 7f5513e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55140ccb90 0x7f55140cbc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53910/0 (socket says 172.21.15.102:53910) 2024-03-23T03:49:12.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.831+0000 7f5513e00640 1 -- 172.21.15.102:0/1918596078 learned_addr learned my addr 172.21.15.102:0/1918596078 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:12.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.832+0000 7f5519400640 1 --2- 172.21.15.102:0/1918596078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55140b0490 0x7f55140b0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:12.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.832+0000 7f5513e00640 1 -- 172.21.15.102:0/1918596078 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f54fc004d00 msgr2=0x7f5514145a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:12.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.832+0000 7f5513e00640 1 --2- 172.21.15.102:0/1918596078 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f54fc004d00 0x7f5514145a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.832+0000 7f5513e00640 1 -- 172.21.15.102:0/1918596078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55140b0490 msgr2=0x7f55140b0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:12.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.832+0000 7f5513e00640 1 --2- 172.21.15.102:0/1918596078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55140b0490 0x7f55140b0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:12.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.832+0000 7f5513e00640 1 -- 172.21.15.102:0/1918596078 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5514145f90 con 0x7f55140ccb90 2024-03-23T03:49:12.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.832+0000 7f5513e00640 1 --2- 172.21.15.102:0/1918596078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55140ccb90 0x7f55140cbc00 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f550000d8e0 tx=0x7f550000dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:12.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.832+0000 7f5510c00640 1 -- 172.21.15.102:0/1918596078 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55000074e0 con 0x7f55140ccb90 2024-03-23T03:49:12.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.832+0000 7f5510c00640 1 -- 172.21.15.102:0/1918596078 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5500007640 con 0x7f55140ccb90 2024-03-23T03:49:12.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.832+0000 7f5510c00640 1 -- 172.21.15.102:0/1918596078 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55000049e0 con 0x7f55140ccb90 2024-03-23T03:49:12.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.832+0000 7f551b400640 1 -- 172.21.15.102:0/1918596078 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55140cdc50 con 0x7f55140ccb90 2024-03-23T03:49:12.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.832+0000 7f551b400640 1 -- 172.21.15.102:0/1918596078 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55140cd040 con 0x7f55140ccb90 2024-03-23T03:49:12.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.834+0000 7f551b400640 1 -- 172.21.15.102:0/1918596078 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5514073390 con 0x7f55140ccb90 2024-03-23T03:49:12.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.835+0000 7f5510c00640 1 -- 172.21.15.102:0/1918596078 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5500004b40 con 0x7f55140ccb90 2024-03-23T03:49:12.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.836+0000 7f5510c00640 1 --2- 172.21.15.102:0/1918596078 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f54f406ac00 0x7f54f406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:12.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.836+0000 7f5510c00640 1 -- 172.21.15.102:0/1918596078 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5500088f30 con 0x7f55140ccb90 2024-03-23T03:49:12.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.836+0000 7f5518a00640 1 --2- 172.21.15.102:0/1918596078 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f54f406ac00 0x7f54f406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:12.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.837+0000 7f5518a00640 1 --2- 172.21.15.102:0/1918596078 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f54f406ac00 0x7f54f406d0c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5508009950 tx=0x7f5508008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:12.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:12.841+0000 7f5510c00640 1 -- 172.21.15.102:0/1918596078 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5500053f30 con 0x7f55140ccb90 2024-03-23T03:49:13.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.038+0000 7f551b400640 1 -- 172.21.15.102:0/1918596078 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f55140cdde0 con 0x7f55140ccb90 2024-03-23T03:49:13.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.038+0000 7f5510c00640 1 -- 172.21.15.102:0/1918596078 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5500019020 con 0x7f55140ccb90 2024-03-23T03:49:13.039 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:49:13.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.041+0000 7f551b400640 1 -- 172.21.15.102:0/1918596078 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f54f406ac00 msgr2=0x7f54f406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:13.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.041+0000 7f551b400640 1 --2- 172.21.15.102:0/1918596078 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f54f406ac00 0x7f54f406d0c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f5508009950 tx=0x7f5508008040 comp rx=0 tx=0).stop 2024-03-23T03:49:13.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.041+0000 7f551b400640 1 -- 172.21.15.102:0/1918596078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55140ccb90 msgr2=0x7f55140cbc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:13.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.041+0000 7f551b400640 1 --2- 172.21.15.102:0/1918596078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55140ccb90 0x7f55140cbc00 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f550000d8e0 tx=0x7f550000dda0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.041+0000 7f551b400640 1 -- 172.21.15.102:0/1918596078 shutdown_connections 2024-03-23T03:49:13.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.041+0000 7f551b400640 1 --2- 172.21.15.102:0/1918596078 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f54f406ac00 0x7f54f406d0c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.041+0000 7f551b400640 1 --2- 172.21.15.102:0/1918596078 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f55140b0490 0x7f55140b0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.041+0000 7f551b400640 1 --2- 172.21.15.102:0/1918596078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55140ccb90 0x7f55140cbc00 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.041+0000 7f551b400640 1 --2- 172.21.15.102:0/1918596078 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f54fc004d00 0x7f5514145a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.041+0000 7f551b400640 1 -- 172.21.15.102:0/1918596078 >> 172.21.15.102:0/1918596078 conn(0x7f551406a0b0 msgr2=0x7f5514151ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:13.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.041+0000 7f551b400640 1 -- 172.21.15.102:0/1918596078 shutdown_connections 2024-03-23T03:49:13.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.041+0000 7f551b400640 1 -- 172.21.15.102:0/1918596078 wait complete. 2024-03-23T03:49:13.062 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:49:13.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.277+0000 7ff057200640 1 Processor -- start 2024-03-23T03:49:13.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.277+0000 7ff057200640 1 -- start start 2024-03-23T03:49:13.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.279+0000 7ff057200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff05815c820 0x7ff0580cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:13.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.279+0000 7ff057200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff0580cf7c0 0x7ff0580d0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:13.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.279+0000 7ff057200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0580ccd90 con 0x7ff058160e90 2024-03-23T03:49:13.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.279+0000 7ff057200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0580cdd70 con 0x7ff0580cf7c0 2024-03-23T03:49:13.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.279+0000 7ff057200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0580b7fc0 con 0x7ff05815c820 2024-03-23T03:49:13.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.279+0000 7ff056800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff0580cf7c0 0x7ff0580d0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:13.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.279+0000 7ff04d400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff058160e90 0x7ff0580cf0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45132/0 (socket says 172.21.15.102:45132) 2024-03-23T03:49:13.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.279+0000 7ff04d400640 1 -- 172.21.15.102:0/3184559271 learned_addr learned my addr 172.21.15.102:0/3184559271 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:13.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.280+0000 7ff055e00640 1 --2- 172.21.15.102:0/3184559271 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff05815c820 0x7ff0580cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:13.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.280+0000 7ff055400640 1 -- 172.21.15.102:0/3184559271 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931473589 0 0) 0x7ff0580ccd90 con 0x7ff058160e90 2024-03-23T03:49:13.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.280+0000 7ff055400640 1 -- 172.21.15.102:0/3184559271 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff03c003520 con 0x7ff058160e90 2024-03-23T03:49:13.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.280+0000 7ff056800640 1 -- 172.21.15.102:0/3184559271 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff05815c820 msgr2=0x7ff0580cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:13.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.280+0000 7ff056800640 1 --2- 172.21.15.102:0/3184559271 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff05815c820 0x7ff0580cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.280+0000 7ff056800640 1 -- 172.21.15.102:0/3184559271 >> v1:172.21.15.102:6789/0 conn(0x7ff058160e90 legacy=0x7ff0580cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:13.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.280+0000 7ff056800640 1 -- 172.21.15.102:0/3184559271 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0580b82a0 con 0x7ff0580cf7c0 2024-03-23T03:49:13.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.281+0000 7ff056800640 1 --2- 172.21.15.102:0/3184559271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0580cf7c0 0x7ff0580d0560 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff050009d90 tx=0x7ff05002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac2d89e112d2d6d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:13.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.281+0000 7ff055400640 1 -- 172.21.15.102:0/3184559271 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05002cce0 con 0x7ff0580cf7c0 2024-03-23T03:49:13.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.281+0000 7ff055400640 1 -- 172.21.15.102:0/3184559271 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05002ce40 con 0x7ff0580cf7c0 2024-03-23T03:49:13.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.281+0000 7ff055400640 1 -- 172.21.15.102:0/3184559271 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff050034660 con 0x7ff0580cf7c0 2024-03-23T03:49:13.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.282+0000 7ff057200640 1 -- 172.21.15.102:0/3184559271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0580cf7c0 msgr2=0x7ff0580d0560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:13.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.282+0000 7ff057200640 1 --2- 172.21.15.102:0/3184559271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0580cf7c0 0x7ff0580d0560 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7ff050009d90 tx=0x7ff05002bf30 comp rx=0 tx=0).stop 2024-03-23T03:49:13.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.282+0000 7ff057200640 1 -- 172.21.15.102:0/3184559271 shutdown_connections 2024-03-23T03:49:13.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.282+0000 7ff057200640 1 --2- 172.21.15.102:0/3184559271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0580cf7c0 0x7ff0580d0560 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.282+0000 7ff057200640 1 --2- 172.21.15.102:0/3184559271 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff05815c820 0x7ff0580cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.282+0000 7ff057200640 1 -- 172.21.15.102:0/3184559271 >> 172.21.15.102:0/3184559271 conn(0x7ff05806b050 msgr2=0x7ff05806b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:13.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.282+0000 7ff057200640 1 -- 172.21.15.102:0/3184559271 shutdown_connections 2024-03-23T03:49:13.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.282+0000 7ff057200640 1 -- 172.21.15.102:0/3184559271 wait complete. 2024-03-23T03:49:13.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.284+0000 7ff057200640 1 Processor -- start 2024-03-23T03:49:13.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.284+0000 7ff057200640 1 -- start start 2024-03-23T03:49:13.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.285+0000 7ff057200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff0580cf7c0 0x7ff058158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:13.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.285+0000 7ff057200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff05815c820 0x7ff0580b68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:13.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.285+0000 7ff057200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff058160e90 0x7ff0581f1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:13.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.285+0000 7ff057200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0581594a0 con 0x7ff05815c820 2024-03-23T03:49:13.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.285+0000 7ff057200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff058158f40 con 0x7ff058160e90 2024-03-23T03:49:13.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.285+0000 7ff057200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0580b2000 con 0x7ff0580cf7c0 2024-03-23T03:49:13.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.285+0000 7ff04d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff05815c820 0x7ff0580b68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:13.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.285+0000 7ff04d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff05815c820 0x7ff0580b68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47282/0 (socket says 172.21.15.102:47282) 2024-03-23T03:49:13.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.285+0000 7ff04d400640 1 -- 172.21.15.102:0/3750191862 learned_addr learned my addr 172.21.15.102:0/3750191862 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:13.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.285+0000 7ff055e00640 1 --2- 172.21.15.102:0/3750191862 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff0580cf7c0 0x7ff058158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:13.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.285+0000 7ff04d400640 1 -- 172.21.15.102:0/3750191862 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff0580cf7c0 msgr2=0x7ff058158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:13.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.285+0000 7ff04d400640 1 --2- 172.21.15.102:0/3750191862 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff0580cf7c0 0x7ff058158a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.285+0000 7ff04d400640 1 -- 172.21.15.102:0/3750191862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff058160e90 msgr2=0x7ff0581f1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:13.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.285+0000 7ff04d400640 1 --2- 172.21.15.102:0/3750191862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff058160e90 0x7ff0581f1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.286+0000 7ff04d400640 1 -- 172.21.15.102:0/3750191862 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff050009a70 con 0x7ff05815c820 2024-03-23T03:49:13.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.286+0000 7ff055e00640 1 --2- 172.21.15.102:0/3750191862 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff0580cf7c0 0x7ff058158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:13.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.286+0000 7ff04d400640 1 --2- 172.21.15.102:0/3750191862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff05815c820 0x7ff0580b68e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff04000c410 tx=0x7ff04000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:13.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.286+0000 7ff04ea00640 1 -- 172.21.15.102:0/3750191862 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff040005ce0 con 0x7ff05815c820 2024-03-23T03:49:13.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.286+0000 7ff04ea00640 1 -- 172.21.15.102:0/3750191862 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff040005e40 con 0x7ff05815c820 2024-03-23T03:49:13.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.286+0000 7ff057200640 1 -- 172.21.15.102:0/3750191862 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0580b22e0 con 0x7ff05815c820 2024-03-23T03:49:13.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.286+0000 7ff04ea00640 1 -- 172.21.15.102:0/3750191862 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0400072f0 con 0x7ff05815c820 2024-03-23T03:49:13.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.286+0000 7ff057200640 1 -- 172.21.15.102:0/3750191862 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0580b28a0 con 0x7ff05815c820 2024-03-23T03:49:13.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.288+0000 7ff057200640 1 -- 172.21.15.102:0/3750191862 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0580d2480 con 0x7ff05815c820 2024-03-23T03:49:13.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.288+0000 7ff04ea00640 1 -- 172.21.15.102:0/3750191862 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff040007510 con 0x7ff05815c820 2024-03-23T03:49:13.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.289+0000 7ff04ea00640 1 --2- 172.21.15.102:0/3750191862 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff03406ac00 0x7ff03406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:13.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.289+0000 7ff04ea00640 1 -- 172.21.15.102:0/3750191862 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff040088b00 con 0x7ff05815c820 2024-03-23T03:49:13.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.290+0000 7ff055e00640 1 --2- 172.21.15.102:0/3750191862 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff03406ac00 0x7ff03406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:13.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.290+0000 7ff055e00640 1 --2- 172.21.15.102:0/3750191862 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff03406ac00 0x7ff03406d0c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff048006fd0 tx=0x7ff048008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:13.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.294+0000 7ff04ea00640 1 -- 172.21.15.102:0/3750191862 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff040053c60 con 0x7ff05815c820 2024-03-23T03:49:13.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.486+0000 7ff057200640 1 -- 172.21.15.102:0/3750191862 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff05804e030 con 0x7ff05815c820 2024-03-23T03:49:13.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.486+0000 7ff04ea00640 1 -- 172.21.15.102:0/3750191862 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff040017020 con 0x7ff05815c820 2024-03-23T03:49:13.487 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:49:13.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.489+0000 7ff057200640 1 -- 172.21.15.102:0/3750191862 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff03406ac00 msgr2=0x7ff03406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:13.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.489+0000 7ff057200640 1 --2- 172.21.15.102:0/3750191862 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff03406ac00 0x7ff03406d0c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff048006fd0 tx=0x7ff048008040 comp rx=0 tx=0).stop 2024-03-23T03:49:13.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.489+0000 7ff057200640 1 -- 172.21.15.102:0/3750191862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff05815c820 msgr2=0x7ff0580b68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:13.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.489+0000 7ff057200640 1 --2- 172.21.15.102:0/3750191862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff05815c820 0x7ff0580b68e0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff04000c410 tx=0x7ff04000c8d0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.489+0000 7ff057200640 1 -- 172.21.15.102:0/3750191862 shutdown_connections 2024-03-23T03:49:13.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.489+0000 7ff057200640 1 --2- 172.21.15.102:0/3750191862 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff03406ac00 0x7ff03406d0c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.489+0000 7ff057200640 1 --2- 172.21.15.102:0/3750191862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff058160e90 0x7ff0581f1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.489+0000 7ff057200640 1 --2- 172.21.15.102:0/3750191862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff05815c820 0x7ff0580b68e0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.489+0000 7ff057200640 1 --2- 172.21.15.102:0/3750191862 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff0580cf7c0 0x7ff058158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.489+0000 7ff057200640 1 -- 172.21.15.102:0/3750191862 >> 172.21.15.102:0/3750191862 conn(0x7ff05806b050 msgr2=0x7ff0580cf0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:13.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.489+0000 7ff057200640 1 -- 172.21.15.102:0/3750191862 shutdown_connections 2024-03-23T03:49:13.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.489+0000 7ff057200640 1 -- 172.21.15.102:0/3750191862 wait complete. 2024-03-23T03:49:13.508 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:49:13.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.721+0000 7f6944800640 1 Processor -- start 2024-03-23T03:49:13.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.721+0000 7f6944800640 1 -- start start 2024-03-23T03:49:13.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.723+0000 7f6944800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f693c0b8330 0x7f693c0cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:13.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.723+0000 7f6944800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693c0cdf90 con 0x7f693c0cdb40 2024-03-23T03:49:13.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.723+0000 7f6944800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f693c0d0e30 con 0x7f693c15ca20 2024-03-23T03:49:13.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.723+0000 7f6944800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f693c0d0b30 con 0x7f693c0b8330 2024-03-23T03:49:13.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.723+0000 7f6941400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f693c0b8330 0x7f693c0cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:13.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.723+0000 7f6942800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f693c0cdb40 0x7f693c0d12c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45142/0 (socket says 172.21.15.102:45142) 2024-03-23T03:49:13.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.723+0000 7f6942800640 1 -- 172.21.15.102:0/402322650 learned_addr learned my addr 172.21.15.102:0/402322650 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:13.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.724+0000 7f6941400640 1 -- 172.21.15.102:0/402322650 >> v1:172.21.15.173:6789/0 conn(0x7f693c15ca20 legacy=0x7f693c1605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:13.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.724+0000 7f6941400640 1 -- 172.21.15.102:0/402322650 >> v1:172.21.15.102:6789/0 conn(0x7f693c0cdb40 legacy=0x7f693c0d12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:13.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.724+0000 7f6941400640 1 -- 172.21.15.102:0/402322650 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693c160eb0 con 0x7f693c0b8330 2024-03-23T03:49:13.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.724+0000 7f6940a00640 1 -- 172.21.15.102:0/402322650 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131564867 0 0) 0x7f6938001490 con 0x7f693c0cdb40 2024-03-23T03:49:13.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.725+0000 7f6941400640 1 --2- 172.21.15.102:0/402322650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f693c0b8330 0x7f693c0cd600 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f69280097d0 tx=0x7f692802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2140ebe1d5a305d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:13.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.725+0000 7f6940a00640 1 -- 172.21.15.102:0/402322650 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6928031830 con 0x7f693c0b8330 2024-03-23T03:49:13.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.725+0000 7f6940a00640 1 -- 172.21.15.102:0/402322650 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6928031990 con 0x7f693c0b8330 2024-03-23T03:49:13.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.725+0000 7f6940a00640 1 -- 172.21.15.102:0/402322650 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6928031d20 con 0x7f693c0b8330 2024-03-23T03:49:13.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.726+0000 7f6944800640 1 -- 172.21.15.102:0/402322650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f693c0b8330 msgr2=0x7f693c0cd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:13.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.726+0000 7f6944800640 1 --2- 172.21.15.102:0/402322650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f693c0b8330 0x7f693c0cd600 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f69280097d0 tx=0x7f692802b8d0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.726+0000 7f6944800640 1 -- 172.21.15.102:0/402322650 shutdown_connections 2024-03-23T03:49:13.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.726+0000 7f6944800640 1 --2- 172.21.15.102:0/402322650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f693c0b8330 0x7f693c0cd600 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.726+0000 7f6944800640 1 -- 172.21.15.102:0/402322650 >> 172.21.15.102:0/402322650 conn(0x7f693c0cf7d0 msgr2=0x7f693c0cfbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:13.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.726+0000 7f6944800640 1 -- 172.21.15.102:0/402322650 shutdown_connections 2024-03-23T03:49:13.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.726+0000 7f6944800640 1 -- 172.21.15.102:0/402322650 wait complete. 2024-03-23T03:49:13.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.728+0000 7f6944800640 1 Processor -- start 2024-03-23T03:49:13.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.728+0000 7f6944800640 1 -- start start 2024-03-23T03:49:13.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.729+0000 7f6944800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f693c0b8330 0x7f693c150e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:13.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.729+0000 7f6944800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f693c0cdb40 0x7f693c0c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:13.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.729+0000 7f6944800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f693c15ca20 0x7f693c15a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:13.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.729+0000 7f6944800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693c0c9030 con 0x7f693c0cdb40 2024-03-23T03:49:13.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.729+0000 7f6944800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693c0c8420 con 0x7f693c0b8330 2024-03-23T03:49:13.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.729+0000 7f6944800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f693c0c8700 con 0x7f693c15ca20 2024-03-23T03:49:13.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.729+0000 7f6941400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f693c0cdb40 0x7f693c0c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:13.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.730+0000 7f6941400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f693c0cdb40 0x7f693c0c7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47304/0 (socket says 172.21.15.102:47304) 2024-03-23T03:49:13.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.730+0000 7f6941400640 1 -- 172.21.15.102:0/1625087627 learned_addr learned my addr 172.21.15.102:0/1625087627 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:13.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.730+0000 7f6942800640 1 --2- 172.21.15.102:0/1625087627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f693c15ca20 0x7f693c15a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:13.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.730+0000 7f6941e00640 1 --2- 172.21.15.102:0/1625087627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f693c0b8330 0x7f693c150e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:13.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.730+0000 7f6941400640 1 -- 172.21.15.102:0/1625087627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f693c15ca20 msgr2=0x7f693c15a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:13.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.730+0000 7f6941400640 1 --2- 172.21.15.102:0/1625087627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f693c15ca20 0x7f693c15a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.730+0000 7f6941400640 1 -- 172.21.15.102:0/1625087627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f693c0b8330 msgr2=0x7f693c150e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:13.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.730+0000 7f6941400640 1 --2- 172.21.15.102:0/1625087627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f693c0b8330 0x7f693c150e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.730+0000 7f6941400640 1 -- 172.21.15.102:0/1625087627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6928009400 con 0x7f693c0cdb40 2024-03-23T03:49:13.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.730+0000 7f6942800640 1 --2- 172.21.15.102:0/1625087627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f693c15ca20 0x7f693c15a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:13.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.730+0000 7f6941400640 1 --2- 172.21.15.102:0/1625087627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f693c0cdb40 0x7f693c0c7ab0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f69280097d0 tx=0x7f69280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:13.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.731+0000 7f6936000640 1 -- 172.21.15.102:0/1625087627 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692803ba70 con 0x7f693c0cdb40 2024-03-23T03:49:13.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.731+0000 7f6936000640 1 -- 172.21.15.102:0/1625087627 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f692803bbd0 con 0x7f693c0cdb40 2024-03-23T03:49:13.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.731+0000 7f6944800640 1 -- 172.21.15.102:0/1625087627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f693c0c89e0 con 0x7f693c0cdb40 2024-03-23T03:49:13.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.731+0000 7f6936000640 1 -- 172.21.15.102:0/1625087627 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69280453f0 con 0x7f693c0cdb40 2024-03-23T03:49:13.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.731+0000 7f6944800640 1 -- 172.21.15.102:0/1625087627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693c151360 con 0x7f693c0cdb40 2024-03-23T03:49:13.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.732+0000 7f6944800640 1 -- 172.21.15.102:0/1625087627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693c0c91a0 con 0x7f693c0cdb40 2024-03-23T03:49:13.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.733+0000 7f6936000640 1 -- 172.21.15.102:0/1625087627 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f692804e450 con 0x7f693c0cdb40 2024-03-23T03:49:13.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.734+0000 7f6936000640 1 --2- 172.21.15.102:0/1625087627 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f691c06ac00 0x7f691c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:13.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.734+0000 7f6936000640 1 -- 172.21.15.102:0/1625087627 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f69280ab770 con 0x7f693c0cdb40 2024-03-23T03:49:13.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.735+0000 7f6941e00640 1 --2- 172.21.15.102:0/1625087627 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f691c06ac00 0x7f691c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:13.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.735+0000 7f6941e00640 1 --2- 172.21.15.102:0/1625087627 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f691c06ac00 0x7f691c06d0c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f693000a750 tx=0x7f6930008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:13.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.739+0000 7f6936000640 1 -- 172.21.15.102:0/1625087627 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6928076800 con 0x7f693c0cdb40 2024-03-23T03:49:13.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.933+0000 7f6944800640 1 -- 172.21.15.102:0/1625087627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f693c15ce00 con 0x7f693c0cdb40 2024-03-23T03:49:13.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.934+0000 7f6936000640 1 -- 172.21.15.102:0/1625087627 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6928079e20 con 0x7f693c0cdb40 2024-03-23T03:49:13.935 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:49:13.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.937+0000 7f6944800640 1 -- 172.21.15.102:0/1625087627 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f691c06ac00 msgr2=0x7f691c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:13.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.937+0000 7f6944800640 1 --2- 172.21.15.102:0/1625087627 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f691c06ac00 0x7f691c06d0c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f693000a750 tx=0x7f6930008040 comp rx=0 tx=0).stop 2024-03-23T03:49:13.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.937+0000 7f6944800640 1 -- 172.21.15.102:0/1625087627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f693c0cdb40 msgr2=0x7f693c0c7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:13.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.937+0000 7f6944800640 1 --2- 172.21.15.102:0/1625087627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f693c0cdb40 0x7f693c0c7ab0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f69280097d0 tx=0x7f69280046c0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.937+0000 7f6944800640 1 -- 172.21.15.102:0/1625087627 shutdown_connections 2024-03-23T03:49:13.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.937+0000 7f6944800640 1 --2- 172.21.15.102:0/1625087627 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f691c06ac00 0x7f691c06d0c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.937+0000 7f6944800640 1 --2- 172.21.15.102:0/1625087627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f693c15ca20 0x7f693c15a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.937+0000 7f6944800640 1 --2- 172.21.15.102:0/1625087627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f693c0cdb40 0x7f693c0c7ab0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.937+0000 7f6944800640 1 --2- 172.21.15.102:0/1625087627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f693c0b8330 0x7f693c150e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:13.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.937+0000 7f6944800640 1 -- 172.21.15.102:0/1625087627 >> 172.21.15.102:0/1625087627 conn(0x7f693c0cf7d0 msgr2=0x7f693c0b39d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:13.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.937+0000 7f6944800640 1 -- 172.21.15.102:0/1625087627 shutdown_connections 2024-03-23T03:49:13.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:13.938+0000 7f6944800640 1 -- 172.21.15.102:0/1625087627 wait complete. 2024-03-23T03:49:13.957 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:49:14.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.152+0000 7f40b9600640 1 Processor -- start 2024-03-23T03:49:14.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.152+0000 7f40b9600640 1 -- start start 2024-03-23T03:49:14.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.153+0000 7f40b9600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f40b415cab0 0x7f40b40b8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:14.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.153+0000 7f40b9600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f40b40ceeb0 0x7f40b40cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:14.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.153+0000 7f40b9600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b40cf850 con 0x7f40b40b4120 2024-03-23T03:49:14.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.154+0000 7f40b9600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40b415ce90 con 0x7f40b40ceeb0 2024-03-23T03:49:14.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.154+0000 7f40b9600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b40cddf0 con 0x7f40b415cab0 2024-03-23T03:49:14.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.154+0000 7f40b3400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f40b40ceeb0 0x7f40b40cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:14.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.154+0000 7f40b3400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f40b40ceeb0 0x7f40b40cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47310/0 (socket says 172.21.15.102:47310) 2024-03-23T03:49:14.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.154+0000 7f40b3400640 1 -- 172.21.15.102:0/630416093 learned_addr learned my addr 172.21.15.102:0/630416093 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:14.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.154+0000 7f40b3e00640 1 --2- 172.21.15.102:0/630416093 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f40b415cab0 0x7f40b40b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:14.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.154+0000 7f40b3400640 1 -- 172.21.15.102:0/630416093 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f40b415cab0 msgr2=0x7f40b40b8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:14.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.154+0000 7f40b3400640 1 --2- 172.21.15.102:0/630416093 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f40b415cab0 0x7f40b40b8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.154+0000 7f40b3400640 1 -- 172.21.15.102:0/630416093 >> v1:172.21.15.173:6789/0 conn(0x7f40b40b4120 legacy=0x7f40b40b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:14.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.155+0000 7f40b3400640 1 -- 172.21.15.102:0/630416093 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b40cd570 con 0x7f40b40ceeb0 2024-03-23T03:49:14.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.156+0000 7f40b3400640 1 --2- 172.21.15.102:0/630416093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40b40ceeb0 0x7f40b40cf310 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f40a0009870 tx=0x7f40a002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=320b59b5c630c73e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:14.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.156+0000 7f40b2a00640 1 -- 172.21.15.102:0/630416093 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a0031640 con 0x7f40b40ceeb0 2024-03-23T03:49:14.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.156+0000 7f40b2a00640 1 -- 172.21.15.102:0/630416093 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a00317a0 con 0x7f40b40ceeb0 2024-03-23T03:49:14.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.156+0000 7f40b2a00640 1 -- 172.21.15.102:0/630416093 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a0031b30 con 0x7f40b40ceeb0 2024-03-23T03:49:14.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.157+0000 7f40b9600640 1 -- 172.21.15.102:0/630416093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40b40ceeb0 msgr2=0x7f40b40cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:14.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.157+0000 7f40b9600640 1 --2- 172.21.15.102:0/630416093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40b40ceeb0 0x7f40b40cf310 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f40a0009870 tx=0x7f40a002b8f0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.157+0000 7f40b9600640 1 -- 172.21.15.102:0/630416093 shutdown_connections 2024-03-23T03:49:14.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.157+0000 7f40b9600640 1 --2- 172.21.15.102:0/630416093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40b40ceeb0 0x7f40b40cf310 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.157+0000 7f40b9600640 1 --2- 172.21.15.102:0/630416093 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f40b415cab0 0x7f40b40b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.157+0000 7f40b9600640 1 -- 172.21.15.102:0/630416093 >> 172.21.15.102:0/630416093 conn(0x7f40b406ae90 msgr2=0x7f40b406b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:14.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.157+0000 7f40b9600640 1 -- 172.21.15.102:0/630416093 shutdown_connections 2024-03-23T03:49:14.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.157+0000 7f40b9600640 1 -- 172.21.15.102:0/630416093 wait complete. 2024-03-23T03:49:14.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.159+0000 7f40b9600640 1 Processor -- start 2024-03-23T03:49:14.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.159+0000 7f40b9600640 1 -- start start 2024-03-23T03:49:14.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.160+0000 7f40b9600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40b40b4120 0x7f40b40d1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:14.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.160+0000 7f40b9600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f40b40ceeb0 0x7f40b40d1c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:14.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.160+0000 7f40b9600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40b415cab0 0x7f40b415e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:14.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.160+0000 7f40b9600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b41f36c0 con 0x7f40b40b4120 2024-03-23T03:49:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.160+0000 7f40b9600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b40b6910 con 0x7f40b415cab0 2024-03-23T03:49:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.160+0000 7f40b9600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b40b5ab0 con 0x7f40b40ceeb0 2024-03-23T03:49:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.160+0000 7f40b3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40b40b4120 0x7f40b40d1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.160+0000 7f40b3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40b40b4120 0x7f40b40d1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47326/0 (socket says 172.21.15.102:47326) 2024-03-23T03:49:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.160+0000 7f40b3e00640 1 -- 172.21.15.102:0/1099824765 learned_addr learned my addr 172.21.15.102:0/1099824765 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.160+0000 7f40b3e00640 1 -- 172.21.15.102:0/1099824765 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f40b40ceeb0 msgr2=0x7f40b40d1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.160+0000 7f40b3e00640 1 --2- 172.21.15.102:0/1099824765 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f40b40ceeb0 0x7f40b40d1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.160+0000 7f40b3e00640 1 -- 172.21.15.102:0/1099824765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40b415cab0 msgr2=0x7f40b415e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.160+0000 7f40b3e00640 1 --2- 172.21.15.102:0/1099824765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40b415cab0 0x7f40b415e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.160+0000 7f40b3e00640 1 -- 172.21.15.102:0/1099824765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a00094d0 con 0x7f40b40b4120 2024-03-23T03:49:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.161+0000 7f40b3e00640 1 --2- 172.21.15.102:0/1099824765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40b40b4120 0x7f40b40d1750 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f40a4007e40 tx=0x7f40a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.161+0000 7f40abe00640 1 -- 172.21.15.102:0/1099824765 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a4014620 con 0x7f40b40b4120 2024-03-23T03:49:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.161+0000 7f40abe00640 1 -- 172.21.15.102:0/1099824765 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a400ee90 con 0x7f40b40b4120 2024-03-23T03:49:14.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.161+0000 7f40b9600640 1 -- 172.21.15.102:0/1099824765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b40b5d00 con 0x7f40b40b4120 2024-03-23T03:49:14.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.161+0000 7f40abe00640 1 -- 172.21.15.102:0/1099824765 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a40149c0 con 0x7f40b40b4120 2024-03-23T03:49:14.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.161+0000 7f40b9600640 1 -- 172.21.15.102:0/1099824765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b40b62c0 con 0x7f40b40b4120 2024-03-23T03:49:14.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.162+0000 7f40b9600640 1 -- 172.21.15.102:0/1099824765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b40735a0 con 0x7f40b40b4120 2024-03-23T03:49:14.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.163+0000 7f40abe00640 1 -- 172.21.15.102:0/1099824765 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f40a4014b20 con 0x7f40b40b4120 2024-03-23T03:49:14.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.164+0000 7f40abe00640 1 --2- 172.21.15.102:0/1099824765 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f409406ac00 0x7f409406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:14.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.164+0000 7f40abe00640 1 -- 172.21.15.102:0/1099824765 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f40a4089f00 con 0x7f40b40b4120 2024-03-23T03:49:14.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.164+0000 7f40b3400640 1 --2- 172.21.15.102:0/1099824765 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f409406ac00 0x7f409406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:14.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.165+0000 7f40b3400640 1 --2- 172.21.15.102:0/1099824765 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f409406ac00 0x7f409406d0c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f40a0036000 tx=0x7f40a0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:14.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.169+0000 7f40abe00640 1 -- 172.21.15.102:0/1099824765 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f40a4054f00 con 0x7f40b40b4120 2024-03-23T03:49:14.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.382+0000 7f40b9600640 1 -- 172.21.15.102:0/1099824765 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40b404e030 con 0x7f40b40b4120 2024-03-23T03:49:14.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.382+0000 7f40abe00640 1 -- 172.21.15.102:0/1099824765 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4740 (secure 0 0 0) 0x7f40a4058520 con 0x7f40b40b4120 2024-03-23T03:49:14.383 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:14.383 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5661,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/1848167718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":1848167718},{"type":"v1","addr":"172.21.15.102:6835","nonce":1848167718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/16304019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":16304019},{"type":"v1","addr":"172.21.15.102:6837","nonce":16304019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5673,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6834/2030662139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2030662139},{"type":"v1","addr":"172.21.15.173:6834","nonce":2030662139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:48:56.221998+0000","modified":"2024-03-23T03:48:59.250212+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5676":{"gid":5676,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6835/3269327930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":3269327930},{"type":"v1","addr":"172.21.15.173:6835","nonce":3269327930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-23T03:49:14.383 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 24 2024-03-23T03:49:14.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.385+0000 7f40b9600640 1 -- 172.21.15.102:0/1099824765 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f409406ac00 msgr2=0x7f409406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:14.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.385+0000 7f40b9600640 1 --2- 172.21.15.102:0/1099824765 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f409406ac00 0x7f409406d0c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f40a0036000 tx=0x7f40a0030000 comp rx=0 tx=0).stop 2024-03-23T03:49:14.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.385+0000 7f40b9600640 1 -- 172.21.15.102:0/1099824765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40b40b4120 msgr2=0x7f40b40d1750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:14.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.385+0000 7f40b9600640 1 --2- 172.21.15.102:0/1099824765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40b40b4120 0x7f40b40d1750 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f40a4007e40 tx=0x7f40a400e4b0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.385+0000 7f40b9600640 1 -- 172.21.15.102:0/1099824765 shutdown_connections 2024-03-23T03:49:14.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.385+0000 7f40b9600640 1 --2- 172.21.15.102:0/1099824765 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f409406ac00 0x7f409406d0c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.385+0000 7f40b9600640 1 --2- 172.21.15.102:0/1099824765 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f40b415cab0 0x7f40b415e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.385+0000 7f40b9600640 1 --2- 172.21.15.102:0/1099824765 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f40b40ceeb0 0x7f40b40d1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.385+0000 7f40b9600640 1 --2- 172.21.15.102:0/1099824765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40b40b4120 0x7f40b40d1750 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.385+0000 7f40b9600640 1 -- 172.21.15.102:0/1099824765 >> 172.21.15.102:0/1099824765 conn(0x7f40b406ae90 msgr2=0x7f40b406b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:14.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.385+0000 7f40b9600640 1 -- 172.21.15.102:0/1099824765 shutdown_connections 2024-03-23T03:49:14.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.385+0000 7f40b9600640 1 -- 172.21.15.102:0/1099824765 wait complete. 2024-03-23T03:49:14.404 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:49:14.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.600+0000 7f4b91400640 1 Processor -- start 2024-03-23T03:49:14.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.600+0000 7f4b91400640 1 -- start start 2024-03-23T03:49:14.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.601+0000 7f4b91400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4b8c0adbc0 0x7f4b8c0b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:14.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.601+0000 7f4b91400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f4b8c06d390 0x7f4b8c06d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:14.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.601+0000 7f4b91400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b8c0b16a0 con 0x7f4b8c0cbab0 2024-03-23T03:49:14.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.601+0000 7f4b91400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b8c06dd40 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.601+0000 7f4b91400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b8c0b4700 con 0x7f4b8c06d390 2024-03-23T03:49:14.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.602+0000 7f4b8a800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4b8c0adbc0 0x7f4b8c0b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:14.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.602+0000 7f4b89e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4b8c0cbab0 0x7f4b8c0b20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45156/0 (socket says 172.21.15.102:45156) 2024-03-23T03:49:14.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.602+0000 7f4b89e00640 1 -- 172.21.15.102:0/1481904842 learned_addr learned my addr 172.21.15.102:0/1481904842 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:14.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.602+0000 7f4b8b200640 1 --2- 172.21.15.102:0/1481904842 >> v2:172.21.15.173:3300/0 conn(0x7f4b8c06d390 0x7f4b8c06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:14.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.602+0000 7f4b8a800640 1 -- 172.21.15.102:0/1481904842 >> v2:172.21.15.173:3300/0 conn(0x7f4b8c06d390 msgr2=0x7f4b8c06d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:14.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.602+0000 7f4b8a800640 1 --2- 172.21.15.102:0/1481904842 >> v2:172.21.15.173:3300/0 conn(0x7f4b8c06d390 0x7f4b8c06d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.602+0000 7f4b8a800640 1 -- 172.21.15.102:0/1481904842 >> v1:172.21.15.102:6789/0 conn(0x7f4b8c0cbab0 legacy=0x7f4b8c0b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:14.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.602+0000 7f4b8a800640 1 -- 172.21.15.102:0/1481904842 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b8c0cbfb0 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.603+0000 7f4b8a800640 1 --2- 172.21.15.102:0/1481904842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b8c0adbc0 0x7f4b8c0b41c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c0ccf20 tx=0x7f4b7802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244afbfe5ca079f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:14.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.604+0000 7f4b89400640 1 -- 172.21.15.102:0/1481904842 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7802e070 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.604+0000 7f4b89400640 1 -- 172.21.15.102:0/1481904842 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b780315a0 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.604+0000 7f4b89400640 1 -- 172.21.15.102:0/1481904842 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b78031930 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.604+0000 7f4b91400640 1 -- 172.21.15.102:0/1481904842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b8c0adbc0 msgr2=0x7f4b8c0b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:14.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.604+0000 7f4b91400640 1 --2- 172.21.15.102:0/1481904842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b8c0adbc0 0x7f4b8c0b41c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f4b8c0ccf20 tx=0x7f4b7802b840 comp rx=0 tx=0).stop 2024-03-23T03:49:14.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.604+0000 7f4b91400640 1 -- 172.21.15.102:0/1481904842 shutdown_connections 2024-03-23T03:49:14.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.604+0000 7f4b91400640 1 --2- 172.21.15.102:0/1481904842 >> v2:172.21.15.173:3300/0 conn(0x7f4b8c06d390 0x7f4b8c06d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.604+0000 7f4b91400640 1 --2- 172.21.15.102:0/1481904842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b8c0adbc0 0x7f4b8c0b41c0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.605+0000 7f4b91400640 1 -- 172.21.15.102:0/1481904842 >> 172.21.15.102:0/1481904842 conn(0x7f4b8c06a3d0 msgr2=0x7f4b8c0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:14.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.605+0000 7f4b91400640 1 -- 172.21.15.102:0/1481904842 shutdown_connections 2024-03-23T03:49:14.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.605+0000 7f4b91400640 1 -- 172.21.15.102:0/1481904842 wait complete. 2024-03-23T03:49:14.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.606+0000 7f4b91400640 1 Processor -- start 2024-03-23T03:49:14.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.606+0000 7f4b91400640 1 -- start start 2024-03-23T03:49:14.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.607+0000 7f4b91400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b8c06d390 0x7f4b8c1f3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:14.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.607+0000 7f4b91400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b8c0adbc0 0x7f4b8c139a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:14.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.607+0000 7f4b91400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b8c0cbab0 0x7f4b8c0c69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:14.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.607+0000 7f4b91400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c0c4310 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.607+0000 7f4b91400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c1f3ea0 con 0x7f4b8c0cbab0 2024-03-23T03:49:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.607+0000 7f4b91400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8c136550 con 0x7f4b8c06d390 2024-03-23T03:49:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.608+0000 7f4b89e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b8c0adbc0 0x7f4b8c139a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.608+0000 7f4b89e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b8c0adbc0 0x7f4b8c139a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47340/0 (socket says 172.21.15.102:47340) 2024-03-23T03:49:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.608+0000 7f4b89e00640 1 -- 172.21.15.102:0/2770535805 learned_addr learned my addr 172.21.15.102:0/2770535805 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.608+0000 7f4b8a800640 1 --2- 172.21.15.102:0/2770535805 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b8c06d390 0x7f4b8c1f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.608+0000 7f4b89e00640 1 -- 172.21.15.102:0/2770535805 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b8c06d390 msgr2=0x7f4b8c1f3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.608+0000 7f4b89e00640 1 --2- 172.21.15.102:0/2770535805 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b8c06d390 0x7f4b8c1f3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.608+0000 7f4b89e00640 1 -- 172.21.15.102:0/2770535805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b8c0cbab0 msgr2=0x7f4b8c0c69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.608+0000 7f4b89e00640 1 --2- 172.21.15.102:0/2770535805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b8c0cbab0 0x7f4b8c0c69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.608+0000 7f4b89e00640 1 -- 172.21.15.102:0/2770535805 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b78009400 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.608+0000 7f4b89e00640 1 --2- 172.21.15.102:0/2770535805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b8c0adbc0 0x7f4b8c139a40 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00c500 tx=0x7f4b7c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.608+0000 7f4b82a00640 1 -- 172.21.15.102:0/2770535805 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c005ce0 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.609+0000 7f4b82a00640 1 -- 172.21.15.102:0/2770535805 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b7c005e40 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.609+0000 7f4b82a00640 1 -- 172.21.15.102:0/2770535805 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0072f0 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.609+0000 7f4b91400640 1 -- 172.21.15.102:0/2770535805 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8c136830 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.609+0000 7f4b91400640 1 -- 172.21.15.102:0/2770535805 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8c136bc0 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.610+0000 7f4b91400640 1 -- 172.21.15.102:0/2770535805 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b8c137030 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.611+0000 7f4b82a00640 1 -- 172.21.15.102:0/2770535805 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4b7c007510 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.612+0000 7f4b82a00640 1 --2- 172.21.15.102:0/2770535805 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4b6c06ac00 0x7f4b6c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:14.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.612+0000 7f4b82a00640 1 -- 172.21.15.102:0/2770535805 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4b7c0889b0 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.612+0000 7f4b8a800640 1 --2- 172.21.15.102:0/2770535805 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4b6c06ac00 0x7f4b6c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:14.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.613+0000 7f4b8a800640 1 --2- 172.21.15.102:0/2770535805 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4b6c06ac00 0x7f4b6c06d0c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4b78036000 tx=0x7f4b7803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:14.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.616+0000 7f4b82a00640 1 -- 172.21.15.102:0/2770535805 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4b7c053af0 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.813+0000 7f4b91400640 1 -- 172.21.15.102:0/2770535805 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b8c0b0dc0 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.816 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.815+0000 7f4b82a00640 1 -- 172.21.15.102:0/2770535805 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16354 (secure 0 0 0) 0x7f4b7c017020 con 0x7f4b8c0adbc0 2024-03-23T03:49:14.816 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:14.816 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":55,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:57.226138+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:48:53.998390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T03:48:55.242109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:49:14.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.818+0000 7f4b91400640 1 -- 172.21.15.102:0/2770535805 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4b6c06ac00 msgr2=0x7f4b6c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:14.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.818+0000 7f4b91400640 1 --2- 172.21.15.102:0/2770535805 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4b6c06ac00 0x7f4b6c06d0c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4b78036000 tx=0x7f4b7803d040 comp rx=0 tx=0).stop 2024-03-23T03:49:14.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.818+0000 7f4b91400640 1 -- 172.21.15.102:0/2770535805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b8c0adbc0 msgr2=0x7f4b8c139a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:14.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.818+0000 7f4b91400640 1 --2- 172.21.15.102:0/2770535805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b8c0adbc0 0x7f4b8c139a40 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00c500 tx=0x7f4b7c00c9c0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.818+0000 7f4b91400640 1 -- 172.21.15.102:0/2770535805 shutdown_connections 2024-03-23T03:49:14.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.818+0000 7f4b91400640 1 --2- 172.21.15.102:0/2770535805 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4b6c06ac00 0x7f4b6c06d0c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.819 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.818+0000 7f4b91400640 1 --2- 172.21.15.102:0/2770535805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b8c0cbab0 0x7f4b8c0c69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.818+0000 7f4b91400640 1 --2- 172.21.15.102:0/2770535805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b8c0adbc0 0x7f4b8c139a40 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.818+0000 7f4b91400640 1 --2- 172.21.15.102:0/2770535805 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b8c06d390 0x7f4b8c1f3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:14.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.818+0000 7f4b91400640 1 -- 172.21.15.102:0/2770535805 >> 172.21.15.102:0/2770535805 conn(0x7f4b8c06a3d0 msgr2=0x7f4b8c0726a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:14.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.819+0000 7f4b91400640 1 -- 172.21.15.102:0/2770535805 shutdown_connections 2024-03-23T03:49:14.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:14.819+0000 7f4b91400640 1 -- 172.21.15.102:0/2770535805 wait complete. 2024-03-23T03:49:14.838 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T03:49:14.838 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:49:15.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.037+0000 7faa1d800640 1 Processor -- start 2024-03-23T03:49:15.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.037+0000 7faa1d800640 1 -- start start 2024-03-23T03:49:15.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.038+0000 7faa1d800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faa180adbc0 0x7faa180b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.038+0000 7faa1d800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faa1806d390 0x7faa1806d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.038+0000 7faa1d800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa180b16a0 con 0x7faa180cbab0 2024-03-23T03:49:15.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.038+0000 7faa1d800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa1806dd60 con 0x7faa1806d390 2024-03-23T03:49:15.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.038+0000 7faa1d800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa180b4700 con 0x7faa180adbc0 2024-03-23T03:49:15.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.039+0000 7faa17400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faa1806d390 0x7faa1806d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:15.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.039+0000 7faa17400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faa1806d390 0x7faa1806d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47348/0 (socket says 172.21.15.102:47348) 2024-03-23T03:49:15.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.039+0000 7faa17400640 1 -- 172.21.15.102:0/90845474 learned_addr learned my addr 172.21.15.102:0/90845474 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:15.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.039+0000 7faa16a00640 1 --2- 172.21.15.102:0/90845474 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faa180adbc0 0x7faa180b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:15.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.039+0000 7faa17400640 1 -- 172.21.15.102:0/90845474 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faa180adbc0 msgr2=0x7faa180b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:15.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.039+0000 7faa17400640 1 --2- 172.21.15.102:0/90845474 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faa180adbc0 0x7faa180b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.039+0000 7faa17400640 1 -- 172.21.15.102:0/90845474 >> v1:172.21.15.173:6789/0 conn(0x7faa180cbab0 legacy=0x7faa180b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:15.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.039+0000 7faa17400640 1 -- 172.21.15.102:0/90845474 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa180cbfb0 con 0x7faa1806d390 2024-03-23T03:49:15.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.040+0000 7faa17400640 1 --2- 172.21.15.102:0/90845474 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa1806d390 0x7faa1806d820 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7faa10009e50 tx=0x7faa1002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dabd98383fb8a99 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:15.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.041+0000 7faa16000640 1 -- 172.21.15.102:0/90845474 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1002c9e0 con 0x7faa1806d390 2024-03-23T03:49:15.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.041+0000 7faa16000640 1 -- 172.21.15.102:0/90845474 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1002cb40 con 0x7faa1806d390 2024-03-23T03:49:15.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.041+0000 7faa16000640 1 -- 172.21.15.102:0/90845474 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa100343f0 con 0x7faa1806d390 2024-03-23T03:49:15.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.041+0000 7faa1d800640 1 -- 172.21.15.102:0/90845474 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa1806d390 msgr2=0x7faa1806d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:15.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.041+0000 7faa1d800640 1 --2- 172.21.15.102:0/90845474 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa1806d390 0x7faa1806d820 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7faa10009e50 tx=0x7faa1002be40 comp rx=0 tx=0).stop 2024-03-23T03:49:15.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.041+0000 7faa1d800640 1 -- 172.21.15.102:0/90845474 shutdown_connections 2024-03-23T03:49:15.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.041+0000 7faa1d800640 1 --2- 172.21.15.102:0/90845474 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa1806d390 0x7faa1806d820 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.041+0000 7faa1d800640 1 --2- 172.21.15.102:0/90845474 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faa180adbc0 0x7faa180b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.041+0000 7faa1d800640 1 -- 172.21.15.102:0/90845474 >> 172.21.15.102:0/90845474 conn(0x7faa1806a3d0 msgr2=0x7faa180b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:15.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.041+0000 7faa1d800640 1 -- 172.21.15.102:0/90845474 shutdown_connections 2024-03-23T03:49:15.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.042+0000 7faa1d800640 1 -- 172.21.15.102:0/90845474 wait complete. 2024-03-23T03:49:15.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.043+0000 7faa1d800640 1 Processor -- start 2024-03-23T03:49:15.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.043+0000 7faa1d800640 1 -- start start 2024-03-23T03:49:15.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.044+0000 7faa1d800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faa1806d390 0x7faa1813b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.044+0000 7faa1d800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa180adbc0 0x7faa1813e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.044+0000 7faa1d800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa180cbab0 0x7faa181562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.044+0000 7faa1d800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1813ba40 con 0x7faa180cbab0 2024-03-23T03:49:15.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.044+0000 7faa1d800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1813e950 con 0x7faa180adbc0 2024-03-23T03:49:15.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.044+0000 7faa1d800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1813e070 con 0x7faa1806d390 2024-03-23T03:49:15.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.044+0000 7faa17400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa180cbab0 0x7faa181562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:15.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.044+0000 7faa17400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa180cbab0 0x7faa181562d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47360/0 (socket says 172.21.15.102:47360) 2024-03-23T03:49:15.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.044+0000 7faa17400640 1 -- 172.21.15.102:0/3927449797 learned_addr learned my addr 172.21.15.102:0/3927449797 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:15.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.045+0000 7faa16a00640 1 --2- 172.21.15.102:0/3927449797 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faa1806d390 0x7faa1813b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:15.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.045+0000 7faa17400640 1 -- 172.21.15.102:0/3927449797 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faa1806d390 msgr2=0x7faa1813b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:15.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.045+0000 7faa17400640 1 --2- 172.21.15.102:0/3927449797 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faa1806d390 0x7faa1813b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.045+0000 7faa17400640 1 -- 172.21.15.102:0/3927449797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa180adbc0 msgr2=0x7faa1813e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:15.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.045+0000 7faa17400640 1 --2- 172.21.15.102:0/3927449797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa180adbc0 0x7faa1813e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.045+0000 7faa17400640 1 -- 172.21.15.102:0/3927449797 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa10009b30 con 0x7faa180cbab0 2024-03-23T03:49:15.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.045+0000 7faa17400640 1 --2- 172.21.15.102:0/3927449797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa180cbab0 0x7faa181562d0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7faa10036040 tx=0x7faa10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:15.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.045+0000 7faa0f400640 1 -- 172.21.15.102:0/3927449797 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa10009510 con 0x7faa180cbab0 2024-03-23T03:49:15.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.045+0000 7faa0f400640 1 -- 172.21.15.102:0/3927449797 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1003e450 con 0x7faa180cbab0 2024-03-23T03:49:15.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.045+0000 7faa0f400640 1 -- 172.21.15.102:0/3927449797 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1003e820 con 0x7faa180cbab0 2024-03-23T03:49:15.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.045+0000 7faa1d800640 1 -- 172.21.15.102:0/3927449797 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa18140fb0 con 0x7faa180cbab0 2024-03-23T03:49:15.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.046+0000 7faa1d800640 1 -- 172.21.15.102:0/3927449797 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1813dd90 con 0x7faa180cbab0 2024-03-23T03:49:15.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.047+0000 7faa1d800640 1 -- 172.21.15.102:0/3927449797 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1806ac20 con 0x7faa180cbab0 2024-03-23T03:49:15.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.048+0000 7faa0f400640 1 -- 172.21.15.102:0/3927449797 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7faa1003e980 con 0x7faa180cbab0 2024-03-23T03:49:15.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.048+0000 7faa0f400640 1 --2- 172.21.15.102:0/3927449797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa9f406ac00 0x7fa9f406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.049+0000 7faa0f400640 1 -- 172.21.15.102:0/3927449797 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7faa100ac5e0 con 0x7faa180cbab0 2024-03-23T03:49:15.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.049+0000 7faa16a00640 1 --2- 172.21.15.102:0/3927449797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa9f406ac00 0x7fa9f406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:15.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.049+0000 7faa16a00640 1 --2- 172.21.15.102:0/3927449797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa9f406ac00 0x7fa9f406d0c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7faa08006fd0 tx=0x7faa08008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:15.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.054+0000 7faa0f400640 1 -- 172.21.15.102:0/3927449797 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faa100775e0 con 0x7faa180cbab0 2024-03-23T03:49:15.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.279+0000 7faa1d800640 1 -- 172.21.15.102:0/3927449797 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa18073390 con 0x7faa180cbab0 2024-03-23T03:49:15.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.281+0000 7faa0f400640 1 -- 172.21.15.102:0/3927449797 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4740 (secure 0 0 0) 0x7faa10033e80 con 0x7faa180cbab0 2024-03-23T03:49:15.281 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:15.281 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5661,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/1848167718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":1848167718},{"type":"v1","addr":"172.21.15.102:6835","nonce":1848167718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/16304019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":16304019},{"type":"v1","addr":"172.21.15.102:6837","nonce":16304019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5673,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6834/2030662139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2030662139},{"type":"v1","addr":"172.21.15.173:6834","nonce":2030662139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:48:56.221998+0000","modified":"2024-03-23T03:48:59.250212+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5676":{"gid":5676,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6835/3269327930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":3269327930},{"type":"v1","addr":"172.21.15.173:6835","nonce":3269327930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-23T03:49:15.282 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 24 2024-03-23T03:49:15.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.283+0000 7faa1d800640 1 -- 172.21.15.102:0/3927449797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa9f406ac00 msgr2=0x7fa9f406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:15.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.283+0000 7faa1d800640 1 --2- 172.21.15.102:0/3927449797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa9f406ac00 0x7fa9f406d0c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7faa08006fd0 tx=0x7faa08008040 comp rx=0 tx=0).stop 2024-03-23T03:49:15.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.283+0000 7faa1d800640 1 -- 172.21.15.102:0/3927449797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa180cbab0 msgr2=0x7faa181562d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:15.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.283+0000 7faa1d800640 1 --2- 172.21.15.102:0/3927449797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa180cbab0 0x7faa181562d0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7faa10036040 tx=0x7faa10004a40 comp rx=0 tx=0).stop 2024-03-23T03:49:15.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.283+0000 7faa1d800640 1 -- 172.21.15.102:0/3927449797 shutdown_connections 2024-03-23T03:49:15.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.283+0000 7faa1d800640 1 --2- 172.21.15.102:0/3927449797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa9f406ac00 0x7fa9f406d0c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.283+0000 7faa1d800640 1 --2- 172.21.15.102:0/3927449797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa180cbab0 0x7faa181562d0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.283+0000 7faa1d800640 1 --2- 172.21.15.102:0/3927449797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa180adbc0 0x7faa1813e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.283+0000 7faa1d800640 1 --2- 172.21.15.102:0/3927449797 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faa1806d390 0x7faa1813b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.283+0000 7faa1d800640 1 -- 172.21.15.102:0/3927449797 >> 172.21.15.102:0/3927449797 conn(0x7faa1806a3d0 msgr2=0x7faa18142bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:15.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.283+0000 7faa1d800640 1 -- 172.21.15.102:0/3927449797 shutdown_connections 2024-03-23T03:49:15.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.283+0000 7faa1d800640 1 -- 172.21.15.102:0/3927449797 wait complete. 2024-03-23T03:49:15.302 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:49:15.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.494+0000 7fcd45200640 1 Processor -- start 2024-03-23T03:49:15.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.494+0000 7fcd45200640 1 -- start start 2024-03-23T03:49:15.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.496+0000 7fcd45200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcd400cbab0 0x7fcd400b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.496+0000 7fcd45200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcd4006d390 0x7fcd4006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.496+0000 7fcd45200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd400b16a0 con 0x7fcd400adbc0 2024-03-23T03:49:15.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.496+0000 7fcd45200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd4006dd60 con 0x7fcd4006d390 2024-03-23T03:49:15.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.496+0000 7fcd45200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd400cbfb0 con 0x7fcd400cbab0 2024-03-23T03:49:15.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.496+0000 7fcd3de00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fcd400adbc0 0x7fcd400b41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45170/0 (socket says 172.21.15.102:45170) 2024-03-23T03:49:15.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.496+0000 7fcd3de00640 1 -- 172.21.15.102:0/358405375 learned_addr learned my addr 172.21.15.102:0/358405375 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:15.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.496+0000 7fcd3e800640 1 --2- 172.21.15.102:0/358405375 >> v2:172.21.15.102:3300/0 conn(0x7fcd4006d390 0x7fcd4006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:15.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.497+0000 7fcd3e800640 1 -- 172.21.15.102:0/358405375 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcd400cbab0 msgr2=0x7fcd400b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:15.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.497+0000 7fcd35400640 1 --2- 172.21.15.102:0/358405375 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcd400cbab0 0x7fcd400b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:15.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.497+0000 7fcd3d400640 1 -- 172.21.15.102:0/358405375 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450834942 0 0) 0x7fcd400b16a0 con 0x7fcd400adbc0 2024-03-23T03:49:15.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.497+0000 7fcd3e800640 1 --2- 172.21.15.102:0/358405375 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcd400cbab0 0x7fcd400b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.497+0000 7fcd3e800640 1 -- 172.21.15.102:0/358405375 >> v1:172.21.15.102:6789/0 conn(0x7fcd400adbc0 legacy=0x7fcd400b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:15.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.497+0000 7fcd3e800640 1 -- 172.21.15.102:0/358405375 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd400cc290 con 0x7fcd4006d390 2024-03-23T03:49:15.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.497+0000 7fcd35400640 1 --2- 172.21.15.102:0/358405375 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcd400cbab0 0x7fcd400b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:15.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.498+0000 7fcd3e800640 1 --2- 172.21.15.102:0/358405375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd4006d390 0x7fcd4006d820 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009d90 tx=0x7fcd3802e020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bfdaa8a45993e6f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:15.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.498+0000 7fcd3d400640 1 -- 172.21.15.102:0/358405375 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3802edd0 con 0x7fcd4006d390 2024-03-23T03:49:15.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.498+0000 7fcd3d400640 1 -- 172.21.15.102:0/358405375 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd38036440 con 0x7fcd4006d390 2024-03-23T03:49:15.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.498+0000 7fcd3d400640 1 -- 172.21.15.102:0/358405375 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd380367d0 con 0x7fcd4006d390 2024-03-23T03:49:15.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.499+0000 7fcd45200640 1 -- 172.21.15.102:0/358405375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd4006d390 msgr2=0x7fcd4006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:15.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.499+0000 7fcd45200640 1 --2- 172.21.15.102:0/358405375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd4006d390 0x7fcd4006d820 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fcd38009d90 tx=0x7fcd3802e020 comp rx=0 tx=0).stop 2024-03-23T03:49:15.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.499+0000 7fcd45200640 1 -- 172.21.15.102:0/358405375 shutdown_connections 2024-03-23T03:49:15.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.499+0000 7fcd45200640 1 --2- 172.21.15.102:0/358405375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd4006d390 0x7fcd4006d820 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.499+0000 7fcd45200640 1 --2- 172.21.15.102:0/358405375 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcd400cbab0 0x7fcd400b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.499+0000 7fcd45200640 1 -- 172.21.15.102:0/358405375 >> 172.21.15.102:0/358405375 conn(0x7fcd4006a3d0 msgr2=0x7fcd400b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:15.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.499+0000 7fcd45200640 1 -- 172.21.15.102:0/358405375 shutdown_connections 2024-03-23T03:49:15.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.499+0000 7fcd45200640 1 -- 172.21.15.102:0/358405375 wait complete. 2024-03-23T03:49:15.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.500+0000 7fcd45200640 1 Processor -- start 2024-03-23T03:49:15.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.501+0000 7fcd45200640 1 -- start start 2024-03-23T03:49:15.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.501+0000 7fcd45200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcd4006d390 0x7fcd4015c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.501+0000 7fcd45200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd400adbc0 0x7fcd40140dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.502+0000 7fcd45200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd400cbab0 0x7fcd4013bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.502+0000 7fcd45200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd40140930 con 0x7fcd400adbc0 2024-03-23T03:49:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.502+0000 7fcd45200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4013fa80 con 0x7fcd400cbab0 2024-03-23T03:49:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.502+0000 7fcd45200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd401f2070 con 0x7fcd4006d390 2024-03-23T03:49:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.502+0000 7fcd35400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd400adbc0 0x7fcd40140dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.502+0000 7fcd35400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd400adbc0 0x7fcd40140dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47380/0 (socket says 172.21.15.102:47380) 2024-03-23T03:49:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.502+0000 7fcd35400640 1 -- 172.21.15.102:0/1115031223 learned_addr learned my addr 172.21.15.102:0/1115031223 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.502+0000 7fcd3de00640 1 --2- 172.21.15.102:0/1115031223 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcd4006d390 0x7fcd4015c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.502+0000 7fcd35400640 1 -- 172.21.15.102:0/1115031223 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcd4006d390 msgr2=0x7fcd4015c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.502+0000 7fcd35400640 1 --2- 172.21.15.102:0/1115031223 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcd4006d390 0x7fcd4015c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.502+0000 7fcd35400640 1 -- 172.21.15.102:0/1115031223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd400cbab0 msgr2=0x7fcd4013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.502+0000 7fcd35400640 1 --2- 172.21.15.102:0/1115031223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd400cbab0 0x7fcd4013bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.502+0000 7fcd35400640 1 -- 172.21.15.102:0/1115031223 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd38009a70 con 0x7fcd400adbc0 2024-03-23T03:49:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.502+0000 7fcd35400640 1 --2- 172.21.15.102:0/1115031223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd400adbc0 0x7fcd40140dc0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800c7f0 tx=0x7fcd2800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.503+0000 7fcd36a00640 1 -- 172.21.15.102:0/1115031223 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd28007a30 con 0x7fcd400adbc0 2024-03-23T03:49:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.503+0000 7fcd36a00640 1 -- 172.21.15.102:0/1115031223 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd28007b90 con 0x7fcd400adbc0 2024-03-23T03:49:15.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.503+0000 7fcd36a00640 1 -- 172.21.15.102:0/1115031223 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2800ce40 con 0x7fcd400adbc0 2024-03-23T03:49:15.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.503+0000 7fcd45200640 1 -- 172.21.15.102:0/1115031223 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd4013fd20 con 0x7fcd400adbc0 2024-03-23T03:49:15.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.503+0000 7fcd45200640 1 -- 172.21.15.102:0/1115031223 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd401402e0 con 0x7fcd400adbc0 2024-03-23T03:49:15.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.504+0000 7fcd45200640 1 -- 172.21.15.102:0/1115031223 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd4006ac20 con 0x7fcd400adbc0 2024-03-23T03:49:15.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.505+0000 7fcd36a00640 1 -- 172.21.15.102:0/1115031223 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcd280113f0 con 0x7fcd400adbc0 2024-03-23T03:49:15.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.506+0000 7fcd36a00640 1 --2- 172.21.15.102:0/1115031223 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcd1c06ac00 0x7fcd1c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.506+0000 7fcd36a00640 1 -- 172.21.15.102:0/1115031223 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fcd28089650 con 0x7fcd400adbc0 2024-03-23T03:49:15.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.506+0000 7fcd3de00640 1 --2- 172.21.15.102:0/1115031223 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcd1c06ac00 0x7fcd1c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:15.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.507+0000 7fcd3de00640 1 --2- 172.21.15.102:0/1115031223 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcd1c06ac00 0x7fcd1c06d0c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcd30007fd0 tx=0x7fcd30009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:15.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.510+0000 7fcd36a00640 1 -- 172.21.15.102:0/1115031223 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcd280546f0 con 0x7fcd400adbc0 2024-03-23T03:49:15.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.726+0000 7fcd45200640 1 -- 172.21.15.102:0/1115031223 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd40073390 con 0x7fcd400adbc0 2024-03-23T03:49:15.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.727+0000 7fcd36a00640 1 -- 172.21.15.102:0/1115031223 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4740 (secure 0 0 0) 0x7fcd28087e60 con 0x7fcd400adbc0 2024-03-23T03:49:15.728 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:15.728 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5661,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/1848167718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":1848167718},{"type":"v1","addr":"172.21.15.102:6835","nonce":1848167718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/16304019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":16304019},{"type":"v1","addr":"172.21.15.102:6837","nonce":16304019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5673,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6834/2030662139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2030662139},{"type":"v1","addr":"172.21.15.173:6834","nonce":2030662139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:48:56.221998+0000","modified":"2024-03-23T03:48:59.250212+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5676},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5676":{"gid":5676,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.173:6835/3269327930","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":3269327930},{"type":"v1","addr":"172.21.15.173:6835","nonce":3269327930}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-03-23T03:49:15.729 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 24 2024-03-23T03:49:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.730+0000 7fcd45200640 1 -- 172.21.15.102:0/1115031223 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcd1c06ac00 msgr2=0x7fcd1c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.730+0000 7fcd45200640 1 --2- 172.21.15.102:0/1115031223 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcd1c06ac00 0x7fcd1c06d0c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fcd30007fd0 tx=0x7fcd30009040 comp rx=0 tx=0).stop 2024-03-23T03:49:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.730+0000 7fcd45200640 1 -- 172.21.15.102:0/1115031223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd400adbc0 msgr2=0x7fcd40140dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.730+0000 7fcd45200640 1 --2- 172.21.15.102:0/1115031223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd400adbc0 0x7fcd40140dc0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fcd2800c7f0 tx=0x7fcd2800cbf0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.730+0000 7fcd45200640 1 -- 172.21.15.102:0/1115031223 shutdown_connections 2024-03-23T03:49:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.730+0000 7fcd45200640 1 --2- 172.21.15.102:0/1115031223 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcd1c06ac00 0x7fcd1c06d0c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.730+0000 7fcd45200640 1 --2- 172.21.15.102:0/1115031223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd400cbab0 0x7fcd4013bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.730+0000 7fcd45200640 1 --2- 172.21.15.102:0/1115031223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd400adbc0 0x7fcd40140dc0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.730+0000 7fcd45200640 1 --2- 172.21.15.102:0/1115031223 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcd4006d390 0x7fcd4015c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.730+0000 7fcd45200640 1 -- 172.21.15.102:0/1115031223 >> 172.21.15.102:0/1115031223 conn(0x7fcd4006a3d0 msgr2=0x7fcd4015d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.730+0000 7fcd45200640 1 -- 172.21.15.102:0/1115031223 shutdown_connections 2024-03-23T03:49:15.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.730+0000 7fcd45200640 1 -- 172.21.15.102:0/1115031223 wait complete. 2024-03-23T03:49:15.750 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:49:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.956+0000 7f841b200640 1 Processor -- start 2024-03-23T03:49:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.956+0000 7f841b200640 1 -- start start 2024-03-23T03:49:15.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.958+0000 7f841b200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f841413faa0 0x7f8414160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.958+0000 7f841b200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8414160800 0x7f84140b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.958+0000 7f841b200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f84140b1be0 0x7f84141422b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.958+0000 7f841b200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84140b2030 con 0x7f841413faa0 2024-03-23T03:49:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.958+0000 7f841b200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84140b38a0 con 0x7f84140b1be0 2024-03-23T03:49:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.958+0000 7f841b200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84140b3b80 con 0x7f8414160800 2024-03-23T03:49:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.958+0000 7f8413e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f841413faa0 0x7f8414160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.958+0000 7f8413e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f841413faa0 0x7f8414160f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47382/0 (socket says 172.21.15.102:47382) 2024-03-23T03:49:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.958+0000 7f8413e00640 1 -- 172.21.15.102:0/2048900269 learned_addr learned my addr 172.21.15.102:0/2048900269 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.959+0000 7f840b400640 1 --2- 172.21.15.102:0/2048900269 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8414160800 0x7f84140b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.959+0000 7f8419200640 1 --2- 172.21.15.102:0/2048900269 >> v2:172.21.15.173:3300/0 conn(0x7f84140b1be0 0x7f84141422b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.959+0000 7f8413e00640 1 -- 172.21.15.102:0/2048900269 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8414160800 msgr2=0x7f84140b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.959+0000 7f8413e00640 1 --2- 172.21.15.102:0/2048900269 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8414160800 0x7f84140b16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.959+0000 7f8413e00640 1 -- 172.21.15.102:0/2048900269 >> v2:172.21.15.173:3300/0 conn(0x7f84140b1be0 msgr2=0x7f84141422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.959+0000 7f8413e00640 1 --2- 172.21.15.102:0/2048900269 >> v2:172.21.15.173:3300/0 conn(0x7f84140b1be0 0x7f84141422b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.959+0000 7f8413e00640 1 -- 172.21.15.102:0/2048900269 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84140b3e60 con 0x7f841413faa0 2024-03-23T03:49:15.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.960+0000 7f8413e00640 1 --2- 172.21.15.102:0/2048900269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f841413faa0 0x7f8414160f30 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8414143350 tx=0x7f840402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937c651f49b6bc6f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:15.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.961+0000 7f8413400640 1 -- 172.21.15.102:0/2048900269 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404031860 con 0x7f841413faa0 2024-03-23T03:49:15.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.961+0000 7f8413400640 1 -- 172.21.15.102:0/2048900269 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84040319c0 con 0x7f841413faa0 2024-03-23T03:49:15.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.961+0000 7f8413400640 1 -- 172.21.15.102:0/2048900269 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404031d50 con 0x7f841413faa0 2024-03-23T03:49:15.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.961+0000 7f841b200640 1 -- 172.21.15.102:0/2048900269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f841413faa0 msgr2=0x7f8414160f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:15.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.961+0000 7f841b200640 1 --2- 172.21.15.102:0/2048900269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f841413faa0 0x7f8414160f30 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8414143350 tx=0x7f840402b900 comp rx=0 tx=0).stop 2024-03-23T03:49:15.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.962+0000 7f841b200640 1 -- 172.21.15.102:0/2048900269 shutdown_connections 2024-03-23T03:49:15.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.962+0000 7f841b200640 1 --2- 172.21.15.102:0/2048900269 >> v2:172.21.15.173:3300/0 conn(0x7f84140b1be0 0x7f84141422b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.962+0000 7f841b200640 1 --2- 172.21.15.102:0/2048900269 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8414160800 0x7f84140b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.962+0000 7f841b200640 1 --2- 172.21.15.102:0/2048900269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f841413faa0 0x7f8414160f30 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.962+0000 7f841b200640 1 -- 172.21.15.102:0/2048900269 >> 172.21.15.102:0/2048900269 conn(0x7f841406a3d0 msgr2=0x7f84140b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:15.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.962+0000 7f841b200640 1 -- 172.21.15.102:0/2048900269 shutdown_connections 2024-03-23T03:49:15.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.962+0000 7f841b200640 1 -- 172.21.15.102:0/2048900269 wait complete. 2024-03-23T03:49:15.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.964+0000 7f841b200640 1 Processor -- start 2024-03-23T03:49:15.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.964+0000 7f841b200640 1 -- start start 2024-03-23T03:49:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.965+0000 7f841b200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84140b1be0 0x7f841415e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.965+0000 7f841b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f841413faa0 0x7f841415e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.965+0000 7f841b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8414160800 0x7f841415ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.965+0000 7f841b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84140cb2a0 con 0x7f841413faa0 2024-03-23T03:49:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.965+0000 7f841b200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84140ca970 con 0x7f8414160800 2024-03-23T03:49:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.965+0000 7f841b200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84140cac50 con 0x7f84140b1be0 2024-03-23T03:49:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.965+0000 7f840b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f841413faa0 0x7f841415e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.965+0000 7f840b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f841413faa0 0x7f841415e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47394/0 (socket says 172.21.15.102:47394) 2024-03-23T03:49:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.965+0000 7f840b400640 1 -- 172.21.15.102:0/403170236 learned_addr learned my addr 172.21.15.102:0/403170236 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.966+0000 7f8413e00640 1 --2- 172.21.15.102:0/403170236 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84140b1be0 0x7f841415e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.966+0000 7f840b400640 1 -- 172.21.15.102:0/403170236 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84140b1be0 msgr2=0x7f841415e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.966+0000 7f840b400640 1 --2- 172.21.15.102:0/403170236 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84140b1be0 0x7f841415e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.966+0000 7f840b400640 1 -- 172.21.15.102:0/403170236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8414160800 msgr2=0x7f841415ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:15.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.966+0000 7f840b400640 1 --2- 172.21.15.102:0/403170236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8414160800 0x7f841415ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:15.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.966+0000 7f840b400640 1 -- 172.21.15.102:0/403170236 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84040094d0 con 0x7f841413faa0 2024-03-23T03:49:15.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.966+0000 7f8413e00640 1 --2- 172.21.15.102:0/403170236 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84140b1be0 0x7f841415e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:15.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.966+0000 7f840b400640 1 --2- 172.21.15.102:0/403170236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f841413faa0 0x7f841415e6d0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00d5c0 tx=0x7f83fc00da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:15.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.966+0000 7f8410c00640 1 -- 172.21.15.102:0/403170236 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83fc0125f0 con 0x7f841413faa0 2024-03-23T03:49:15.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.966+0000 7f8410c00640 1 -- 172.21.15.102:0/403170236 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83fc012750 con 0x7f841413faa0 2024-03-23T03:49:15.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.966+0000 7f8410c00640 1 -- 172.21.15.102:0/403170236 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83fc012b00 con 0x7f841413faa0 2024-03-23T03:49:15.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.966+0000 7f841b200640 1 -- 172.21.15.102:0/403170236 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84140caf30 con 0x7f841413faa0 2024-03-23T03:49:15.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.966+0000 7f841b200640 1 -- 172.21.15.102:0/403170236 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84140cd3f0 con 0x7f841413faa0 2024-03-23T03:49:15.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.967+0000 7f841b200640 1 -- 172.21.15.102:0/403170236 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8414073390 con 0x7f841413faa0 2024-03-23T03:49:15.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.969+0000 7f8410c00640 1 -- 172.21.15.102:0/403170236 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f83fc012c60 con 0x7f841413faa0 2024-03-23T03:49:15.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.969+0000 7f8410c00640 1 --2- 172.21.15.102:0/403170236 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f83f006ac00 0x7f83f006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:15.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.970+0000 7f8410c00640 1 -- 172.21.15.102:0/403170236 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f83fc0898e0 con 0x7f841413faa0 2024-03-23T03:49:15.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.970+0000 7f8413e00640 1 --2- 172.21.15.102:0/403170236 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f83f006ac00 0x7f83f006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:15.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.971+0000 7f8413e00640 1 --2- 172.21.15.102:0/403170236 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f83f006ac00 0x7f83f006d0c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8404036000 tx=0x7f840403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:15.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:15.974+0000 7f8410c00640 1 -- 172.21.15.102:0/403170236 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f83fc054980 con 0x7f841413faa0 2024-03-23T03:49:16.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.170+0000 7f841b200640 1 -- 172.21.15.102:0/403170236 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f841406cdc0 con 0x7f841413faa0 2024-03-23T03:49:16.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.172+0000 7f8410c00640 1 -- 172.21.15.102:0/403170236 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v55) v1 ==== 74+0+16354 (secure 0 0 0) 0x7f83fc057fa0 con 0x7f841413faa0 2024-03-23T03:49:16.173 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:16.173 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":55,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:48:57.226138+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:48:53.998390+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T03:48:55.242109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:49:16.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.174+0000 7f841b200640 1 -- 172.21.15.102:0/403170236 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f83f006ac00 msgr2=0x7f83f006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:16.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.174+0000 7f841b200640 1 --2- 172.21.15.102:0/403170236 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f83f006ac00 0x7f83f006d0c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8404036000 tx=0x7f840403c040 comp rx=0 tx=0).stop 2024-03-23T03:49:16.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.174+0000 7f841b200640 1 -- 172.21.15.102:0/403170236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f841413faa0 msgr2=0x7f841415e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:16.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.174+0000 7f841b200640 1 --2- 172.21.15.102:0/403170236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f841413faa0 0x7f841415e6d0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00d5c0 tx=0x7f83fc00da80 comp rx=0 tx=0).stop 2024-03-23T03:49:16.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.175+0000 7f841b200640 1 -- 172.21.15.102:0/403170236 shutdown_connections 2024-03-23T03:49:16.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.175+0000 7f841b200640 1 --2- 172.21.15.102:0/403170236 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f83f006ac00 0x7f83f006d0c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.175+0000 7f841b200640 1 --2- 172.21.15.102:0/403170236 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8414160800 0x7f841415ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.175+0000 7f841b200640 1 --2- 172.21.15.102:0/403170236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f841413faa0 0x7f841415e6d0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.175+0000 7f841b200640 1 --2- 172.21.15.102:0/403170236 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f84140b1be0 0x7f841415e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.175+0000 7f841b200640 1 -- 172.21.15.102:0/403170236 >> 172.21.15.102:0/403170236 conn(0x7f841406a3d0 msgr2=0x7f84140b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:16.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.175+0000 7f841b200640 1 -- 172.21.15.102:0/403170236 shutdown_connections 2024-03-23T03:49:16.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.175+0000 7f841b200640 1 -- 172.21.15.102:0/403170236 wait complete. 2024-03-23T03:49:16.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T03:49:16.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.407+0000 7f6114000640 1 Processor -- start 2024-03-23T03:49:16.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.407+0000 7f6114000640 1 -- start start 2024-03-23T03:49:16.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.409+0000 7f6114000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f610c159690 0x7f610c0ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:16.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.409+0000 7f6114000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f610c0cc330 0x7f610c0b18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:16.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.409+0000 7f6114000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f610c15e010 con 0x7f610c0b1e10 2024-03-23T03:49:16.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.409+0000 7f6114000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f610c0cc7f0 con 0x7f610c0cc330 2024-03-23T03:49:16.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.409+0000 7f6114000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f610c0ccad0 con 0x7f610c159690 2024-03-23T03:49:16.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.409+0000 7f6111600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f610c159690 0x7f610c0ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:16.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.409+0000 7f6111600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f610c159690 0x7f610c0ca7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:51576/0 (socket says 172.21.15.102:51576) 2024-03-23T03:49:16.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.409+0000 7f6111600640 1 -- 172.21.15.102:0/605931929 learned_addr learned my addr 172.21.15.102:0/605931929 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:16.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.410+0000 7f6110c00640 1 --2- 172.21.15.102:0/605931929 >> v2:172.21.15.173:3300/0 conn(0x7f610c0cc330 0x7f610c0b18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:16.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.410+0000 7f6111600640 1 -- 172.21.15.102:0/605931929 >> v2:172.21.15.173:3300/0 conn(0x7f610c0cc330 msgr2=0x7f610c0b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:16.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.410+0000 7f6111600640 1 --2- 172.21.15.102:0/605931929 >> v2:172.21.15.173:3300/0 conn(0x7f610c0cc330 0x7f610c0b18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.410+0000 7f6111600640 1 -- 172.21.15.102:0/605931929 >> v1:172.21.15.173:6789/0 conn(0x7f610c0b1e10 legacy=0x7f610c0cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:16.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.410+0000 7f6111600640 1 -- 172.21.15.102:0/605931929 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610c0ccdb0 con 0x7f610c159690 2024-03-23T03:49:16.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.412+0000 7f6111600640 1 --2- 172.21.15.102:0/605931929 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f610c159690 0x7f610c0ca7b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6100009870 tx=0x7f610002db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d624036c54314b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:16.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.412+0000 7f610be00640 1 -- 172.21.15.102:0/605931929 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610002e940 con 0x7f610c159690 2024-03-23T03:49:16.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.412+0000 7f610be00640 1 -- 172.21.15.102:0/605931929 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f610002eaa0 con 0x7f610c159690 2024-03-23T03:49:16.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.412+0000 7f610be00640 1 -- 172.21.15.102:0/605931929 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f610002ee30 con 0x7f610c159690 2024-03-23T03:49:16.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.413+0000 7f6114000640 1 -- 172.21.15.102:0/605931929 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f610c159690 msgr2=0x7f610c0ca7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:16.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.413+0000 7f6114000640 1 --2- 172.21.15.102:0/605931929 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f610c159690 0x7f610c0ca7b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6100009870 tx=0x7f610002db90 comp rx=0 tx=0).stop 2024-03-23T03:49:16.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.413+0000 7f6114000640 1 -- 172.21.15.102:0/605931929 shutdown_connections 2024-03-23T03:49:16.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.413+0000 7f6114000640 1 --2- 172.21.15.102:0/605931929 >> v2:172.21.15.173:3300/0 conn(0x7f610c0cc330 0x7f610c0b18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.413+0000 7f6114000640 1 --2- 172.21.15.102:0/605931929 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f610c159690 0x7f610c0ca7b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.413+0000 7f6114000640 1 -- 172.21.15.102:0/605931929 >> 172.21.15.102:0/605931929 conn(0x7f610c06a0b0 msgr2=0x7f610c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:16.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.413+0000 7f6114000640 1 -- 172.21.15.102:0/605931929 shutdown_connections 2024-03-23T03:49:16.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.413+0000 7f6114000640 1 -- 172.21.15.102:0/605931929 wait complete. 2024-03-23T03:49:16.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.414+0000 7f6114000640 1 Processor -- start 2024-03-23T03:49:16.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.415+0000 7f6114000640 1 -- start start 2024-03-23T03:49:16.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.416+0000 7f6114000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f610c0b1e10 0x7f610c154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:16.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.416+0000 7f6114000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f610c0cc330 0x7f610c1568b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:16.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.416+0000 7f6114000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f610c159690 0x7f610c0b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:16.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.416+0000 7f6114000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610c0b0f50 con 0x7f610c0b1e10 2024-03-23T03:49:16.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.416+0000 7f6114000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610c0ae020 con 0x7f610c0cc330 2024-03-23T03:49:16.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.416+0000 7f6114000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f610c0ae300 con 0x7f610c159690 2024-03-23T03:49:16.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.416+0000 7f6111600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f610c0b1e10 0x7f610c154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:16.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.416+0000 7f6111600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f610c0b1e10 0x7f610c154800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47396/0 (socket says 172.21.15.102:47396) 2024-03-23T03:49:16.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.416+0000 7f6111600640 1 -- 172.21.15.102:0/2471894872 learned_addr learned my addr 172.21.15.102:0/2471894872 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:16.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.416+0000 7f6111600640 1 -- 172.21.15.102:0/2471894872 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f610c159690 msgr2=0x7f610c0b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:16.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.416+0000 7f6111600640 1 --2- 172.21.15.102:0/2471894872 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f610c159690 0x7f610c0b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.416+0000 7f6111600640 1 -- 172.21.15.102:0/2471894872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f610c0cc330 msgr2=0x7f610c1568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:16.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.416+0000 7f6111600640 1 --2- 172.21.15.102:0/2471894872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f610c0cc330 0x7f610c1568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.416+0000 7f6111600640 1 -- 172.21.15.102:0/2471894872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61000094d0 con 0x7f610c0b1e10 2024-03-23T03:49:16.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.417+0000 7f6111600640 1 --2- 172.21.15.102:0/2471894872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f610c0b1e10 0x7f610c154800 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6100000c00 tx=0x7f61000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:16.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.417+0000 7f6109600640 1 -- 172.21.15.102:0/2471894872 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6100040600 con 0x7f610c0b1e10 2024-03-23T03:49:16.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.417+0000 7f6109600640 1 -- 172.21.15.102:0/2471894872 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6100040760 con 0x7f610c0b1e10 2024-03-23T03:49:16.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.417+0000 7f6114000640 1 -- 172.21.15.102:0/2471894872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f610c0ae5e0 con 0x7f610c0b1e10 2024-03-23T03:49:16.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.417+0000 7f6109600640 1 -- 172.21.15.102:0/2471894872 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6100040b10 con 0x7f610c0b1e10 2024-03-23T03:49:16.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.417+0000 7f6114000640 1 -- 172.21.15.102:0/2471894872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f610c1f6360 con 0x7f610c0b1e10 2024-03-23T03:49:16.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.418+0000 7f6114000640 1 -- 172.21.15.102:0/2471894872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f610c0ae440 con 0x7f610c0b1e10 2024-03-23T03:49:16.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.419+0000 7f6109600640 1 -- 172.21.15.102:0/2471894872 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6100040c70 con 0x7f610c0b1e10 2024-03-23T03:49:16.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.420+0000 7f6109600640 1 --2- 172.21.15.102:0/2471894872 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f60ec06ac00 0x7f60ec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:16.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.420+0000 7f6109600640 1 -- 172.21.15.102:0/2471894872 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f61000abe50 con 0x7f610c0b1e10 2024-03-23T03:49:16.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.420+0000 7f6110c00640 1 --2- 172.21.15.102:0/2471894872 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f60ec06ac00 0x7f60ec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:16.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.421+0000 7f6110c00640 1 --2- 172.21.15.102:0/2471894872 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f60ec06ac00 0x7f60ec06d0c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f60f8006a90 tx=0x7f60f8006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:16.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.424+0000 7f6109600640 1 -- 172.21.15.102:0/2471894872 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f610c0ae440 con 0x7f610c0b1e10 2024-03-23T03:49:16.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.655+0000 7f6114000640 1 -- 172.21.15.102:0/2471894872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f610c0ae440 con 0x7f610c0b1e10 2024-03-23T03:49:16.657 INFO:tasks.ceph.mon.a.smithi102.stderr:2024-03-23T03:49:16.657+0000 7f5aab200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T03:49:16.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.670+0000 7f6109600640 1 -- 172.21.15.102:0/2471894872 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) v1 ==== 161+0+0 (secure 0 0 0) 0x7f610c0ae440 con 0x7f610c0b1e10 2024-03-23T03:49:16.670 INFO:teuthology.orchestra.run.smithi102.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T03:49:16.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.672+0000 7f6114000640 1 -- 172.21.15.102:0/2471894872 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f60ec06ac00 msgr2=0x7f60ec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:16.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.672+0000 7f6114000640 1 --2- 172.21.15.102:0/2471894872 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f60ec06ac00 0x7f60ec06d0c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f60f8006a90 tx=0x7f60f8006a20 comp rx=0 tx=0).stop 2024-03-23T03:49:16.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.672+0000 7f6114000640 1 -- 172.21.15.102:0/2471894872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f610c0b1e10 msgr2=0x7f610c154800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:16.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.672+0000 7f6114000640 1 --2- 172.21.15.102:0/2471894872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f610c0b1e10 0x7f610c154800 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f6100000c00 tx=0x7f61000046c0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.672+0000 7f6114000640 1 -- 172.21.15.102:0/2471894872 shutdown_connections 2024-03-23T03:49:16.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.672+0000 7f6114000640 1 --2- 172.21.15.102:0/2471894872 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f60ec06ac00 0x7f60ec06d0c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.672+0000 7f6114000640 1 --2- 172.21.15.102:0/2471894872 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f610c159690 0x7f610c0b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.672+0000 7f6114000640 1 --2- 172.21.15.102:0/2471894872 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f610c0cc330 0x7f610c1568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.672+0000 7f6114000640 1 --2- 172.21.15.102:0/2471894872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f610c0b1e10 0x7f610c154800 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.672+0000 7f6114000640 1 -- 172.21.15.102:0/2471894872 >> 172.21.15.102:0/2471894872 conn(0x7f610c06a0b0 msgr2=0x7f610c0b1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:16.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.672+0000 7f6114000640 1 -- 172.21.15.102:0/2471894872 shutdown_connections 2024-03-23T03:49:16.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.672+0000 7f6114000640 1 -- 172.21.15.102:0/2471894872 wait complete. 2024-03-23T03:49:16.691 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T03:49:16.694 INFO:tasks.ceph.mds.d.smithi173.stderr: -2654> 2024-03-23T03:49:01.687+0000 7f04e9800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T03:49:16.769 INFO:tasks.ceph.mds.d.smithi173.stdout:starting mds.d at 2024-03-23T03:49:16.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.888+0000 7efcefc00640 1 Processor -- start 2024-03-23T03:49:16.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.888+0000 7efcefc00640 1 -- start start 2024-03-23T03:49:16.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.890+0000 7efcefc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efce80ce930 0x7efce80add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:16.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.890+0000 7efcefc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7efce806ce90 0x7efce806d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:16.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.890+0000 7efcefc00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce80cdba0 con 0x7efce80cd3e0 2024-03-23T03:49:16.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.890+0000 7efcefc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efce806c760 con 0x7efce806ce90 2024-03-23T03:49:16.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.890+0000 7efcefc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce80b42d0 con 0x7efce80ce930 2024-03-23T03:49:16.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.890+0000 7efcedc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7efce806ce90 0x7efce806d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:16.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.890+0000 7efcedc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7efce806ce90 0x7efce806d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47402/0 (socket says 172.21.15.102:47402) 2024-03-23T03:49:16.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.890+0000 7efcedc00640 1 -- 172.21.15.102:0/470414270 learned_addr learned my addr 172.21.15.102:0/470414270 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:16.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.891+0000 7efced200640 1 --2- 172.21.15.102:0/470414270 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efce80ce930 0x7efce80add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:16.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.891+0000 7efcedc00640 1 -- 172.21.15.102:0/470414270 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efce80ce930 msgr2=0x7efce80add30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:16.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.891+0000 7efcedc00640 1 --2- 172.21.15.102:0/470414270 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efce80ce930 0x7efce80add30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.891+0000 7efcedc00640 1 -- 172.21.15.102:0/470414270 >> v1:172.21.15.173:6789/0 conn(0x7efce80cd3e0 legacy=0x7efce80ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:16.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.891+0000 7efcedc00640 1 -- 172.21.15.102:0/470414270 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce80cde80 con 0x7efce806ce90 2024-03-23T03:49:16.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.891+0000 7efced200640 1 --2- 172.21.15.102:0/470414270 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efce80ce930 0x7efce80add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:16.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.892+0000 7efcedc00640 1 --2- 172.21.15.102:0/470414270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efce806ce90 0x7efce806d320 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7efce0009e50 tx=0x7efce002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d48845672aeb2f44 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:16.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.892+0000 7efce7400640 1 -- 172.21.15.102:0/470414270 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce002cb00 con 0x7efce806ce90 2024-03-23T03:49:16.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.892+0000 7efce7400640 1 -- 172.21.15.102:0/470414270 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce002cc60 con 0x7efce806ce90 2024-03-23T03:49:16.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.892+0000 7efce7400640 1 -- 172.21.15.102:0/470414270 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce00343f0 con 0x7efce806ce90 2024-03-23T03:49:16.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.893+0000 7efcefc00640 1 -- 172.21.15.102:0/470414270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efce806ce90 msgr2=0x7efce806d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:16.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.893+0000 7efcefc00640 1 --2- 172.21.15.102:0/470414270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efce806ce90 0x7efce806d320 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7efce0009e50 tx=0x7efce002bf60 comp rx=0 tx=0).stop 2024-03-23T03:49:16.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.893+0000 7efcefc00640 1 -- 172.21.15.102:0/470414270 shutdown_connections 2024-03-23T03:49:16.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.893+0000 7efcefc00640 1 --2- 172.21.15.102:0/470414270 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efce806ce90 0x7efce806d320 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.893+0000 7efcefc00640 1 --2- 172.21.15.102:0/470414270 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efce80ce930 0x7efce80add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.893+0000 7efcefc00640 1 -- 172.21.15.102:0/470414270 >> 172.21.15.102:0/470414270 conn(0x7efce806a0b0 msgr2=0x7efce806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:16.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.893+0000 7efcefc00640 1 -- 172.21.15.102:0/470414270 shutdown_connections 2024-03-23T03:49:16.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.893+0000 7efcefc00640 1 -- 172.21.15.102:0/470414270 wait complete. 2024-03-23T03:49:16.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.895+0000 7efcefc00640 1 Processor -- start 2024-03-23T03:49:16.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.895+0000 7efcefc00640 1 -- start start 2024-03-23T03:49:16.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.896+0000 7efcefc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efce806ce90 0x7efce81555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:16.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.896+0000 7efcefc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efce80cd3e0 0x7efce814a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:16.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.896+0000 7efcefc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efce80ce930 0x7efce81564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:16.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.896+0000 7efcefc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce8155130 con 0x7efce80cd3e0 2024-03-23T03:49:16.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.896+0000 7efcefc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce8154b20 con 0x7efce806ce90 2024-03-23T03:49:16.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.896+0000 7efcefc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce8156e20 con 0x7efce80ce930 2024-03-23T03:49:16.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.896+0000 7efce7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efce80cd3e0 0x7efce814a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:16.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.896+0000 7efce7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efce80cd3e0 0x7efce814a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47418/0 (socket says 172.21.15.102:47418) 2024-03-23T03:49:16.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.896+0000 7efce7e00640 1 -- 172.21.15.102:0/3013619832 learned_addr learned my addr 172.21.15.102:0/3013619832 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:16.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.896+0000 7efce7e00640 1 -- 172.21.15.102:0/3013619832 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efce80ce930 msgr2=0x7efce81564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:16.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.896+0000 7efce7e00640 1 --2- 172.21.15.102:0/3013619832 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efce80ce930 0x7efce81564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.896+0000 7efce7e00640 1 -- 172.21.15.102:0/3013619832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efce806ce90 msgr2=0x7efce81555c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:16.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.896+0000 7efce7e00640 1 --2- 172.21.15.102:0/3013619832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efce806ce90 0x7efce81555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:16.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.896+0000 7efce7e00640 1 -- 172.21.15.102:0/3013619832 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce0009b30 con 0x7efce80cd3e0 2024-03-23T03:49:16.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.897+0000 7efce7e00640 1 --2- 172.21.15.102:0/3013619832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efce80cd3e0 0x7efce814a530 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7efcd800c450 tx=0x7efcd800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:16.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.897+0000 7efce4c00640 1 -- 172.21.15.102:0/3013619832 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd80052d0 con 0x7efce80cd3e0 2024-03-23T03:49:16.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.897+0000 7efce4c00640 1 -- 172.21.15.102:0/3013619832 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd8005430 con 0x7efce80cd3e0 2024-03-23T03:49:16.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.897+0000 7efce4c00640 1 -- 172.21.15.102:0/3013619832 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd8012870 con 0x7efce80cd3e0 2024-03-23T03:49:16.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.897+0000 7efcefc00640 1 -- 172.21.15.102:0/3013619832 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce8154dc0 con 0x7efce80cd3e0 2024-03-23T03:49:16.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.897+0000 7efcefc00640 1 -- 172.21.15.102:0/3013619832 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce8157100 con 0x7efce80cd3e0 2024-03-23T03:49:16.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.898+0000 7efcefc00640 1 -- 172.21.15.102:0/3013619832 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce8073390 con 0x7efce80cd3e0 2024-03-23T03:49:16.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.899+0000 7efce4c00640 1 -- 172.21.15.102:0/3013619832 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7efcd80129d0 con 0x7efce80cd3e0 2024-03-23T03:49:16.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.900+0000 7efce4c00640 1 --2- 172.21.15.102:0/3013619832 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efcc406ac00 0x7efcc406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:16.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.900+0000 7efce4c00640 1 -- 172.21.15.102:0/3013619832 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6115+0+0 (secure 0 0 0) 0x7efcd8089010 con 0x7efce80cd3e0 2024-03-23T03:49:16.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.900+0000 7efced200640 1 --2- 172.21.15.102:0/3013619832 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efcc406ac00 0x7efcc406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:16.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.901+0000 7efced200640 1 --2- 172.21.15.102:0/3013619832 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efcc406ac00 0x7efcc406d0c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7efcd0006fd0 tx=0x7efcd0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:16.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:16.905+0000 7efce4c00640 1 -- 172.21.15.102:0/3013619832 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efcd8054050 con 0x7efce80cd3e0 2024-03-23T03:49:17.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.133+0000 7efcefc00640 1 -- 172.21.15.102:0/3013619832 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7efce8155270 con 0x7efce80cd3e0 2024-03-23T03:49:17.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.677+0000 7efce4c00640 1 -- 172.21.15.102:0/3013619832 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7efce8155270 con 0x7efce80cd3e0 2024-03-23T03:49:17.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.680+0000 7efcefc00640 1 -- 172.21.15.102:0/3013619832 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efcc406ac00 msgr2=0x7efcc406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:17.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.680+0000 7efcefc00640 1 --2- 172.21.15.102:0/3013619832 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efcc406ac00 0x7efcc406d0c0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7efcd0006fd0 tx=0x7efcd0008040 comp rx=0 tx=0).stop 2024-03-23T03:49:17.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.680+0000 7efcefc00640 1 -- 172.21.15.102:0/3013619832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efce80cd3e0 msgr2=0x7efce814a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:17.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.680+0000 7efcefc00640 1 --2- 172.21.15.102:0/3013619832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efce80cd3e0 0x7efce814a530 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7efcd800c450 tx=0x7efcd800c910 comp rx=0 tx=0).stop 2024-03-23T03:49:17.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.681+0000 7efcefc00640 1 -- 172.21.15.102:0/3013619832 shutdown_connections 2024-03-23T03:49:17.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.681+0000 7efcefc00640 1 --2- 172.21.15.102:0/3013619832 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efcc406ac00 0x7efcc406d0c0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:17.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.681+0000 7efcefc00640 1 --2- 172.21.15.102:0/3013619832 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efce80ce930 0x7efce81564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:17.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.681+0000 7efcefc00640 1 --2- 172.21.15.102:0/3013619832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efce80cd3e0 0x7efce814a530 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:17.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.681+0000 7efcefc00640 1 --2- 172.21.15.102:0/3013619832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efce806ce90 0x7efce81555c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:17.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.681+0000 7efcefc00640 1 -- 172.21.15.102:0/3013619832 >> 172.21.15.102:0/3013619832 conn(0x7efce806a0b0 msgr2=0x7efce80ab8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:17.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.681+0000 7efcefc00640 1 -- 172.21.15.102:0/3013619832 shutdown_connections 2024-03-23T03:49:17.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.681+0000 7efcefc00640 1 -- 172.21.15.102:0/3013619832 wait complete. 2024-03-23T03:49:17.701 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T03:49:17.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.916+0000 7fc2b0400640 1 Processor -- start 2024-03-23T03:49:17.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.916+0000 7fc2b0400640 1 -- start start 2024-03-23T03:49:17.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.918+0000 7fc2b0400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2a815a850 0x7fc2a806aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:17.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.918+0000 7fc2b0400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc2a806d7b0 0x7fc2a806b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:17.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.918+0000 7fc2b0400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a806f430 con 0x7fc2a80659d0 2024-03-23T03:49:17.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.918+0000 7fc2b0400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a815be60 con 0x7fc2a815a850 2024-03-23T03:49:17.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.918+0000 7fc2b0400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a815b0a0 con 0x7fc2a806d7b0 2024-03-23T03:49:17.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.918+0000 7fc2ada00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2a815a850 0x7fc2a806aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:17.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.918+0000 7fc2ada00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc2a815a850 0x7fc2a806aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47420/0 (socket says 172.21.15.102:47420) 2024-03-23T03:49:17.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.918+0000 7fc2ada00640 1 -- 172.21.15.102:0/226908334 learned_addr learned my addr 172.21.15.102:0/226908334 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:17.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.918+0000 7fc2ada00640 1 -- 172.21.15.102:0/226908334 >> v2:172.21.15.173:3300/0 conn(0x7fc2a806d7b0 msgr2=0x7fc2a806b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:17.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.918+0000 7fc2ada00640 1 --2- 172.21.15.102:0/226908334 >> v2:172.21.15.173:3300/0 conn(0x7fc2a806d7b0 0x7fc2a806b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:17.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.918+0000 7fc2ada00640 1 -- 172.21.15.102:0/226908334 >> v1:172.21.15.173:6789/0 conn(0x7fc2a80659d0 legacy=0x7fc2a815e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:17.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.919+0000 7fc2ada00640 1 -- 172.21.15.102:0/226908334 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a815b380 con 0x7fc2a815a850 2024-03-23T03:49:17.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.920+0000 7fc2ada00640 1 --2- 172.21.15.102:0/226908334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a815a850 0x7fc2a806aca0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc2a806c3b0 tx=0x7fc29802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a37044c3b1a69fd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:17.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.920+0000 7fc2a7e00640 1 -- 172.21.15.102:0/226908334 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298031870 con 0x7fc2a815a850 2024-03-23T03:49:17.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.920+0000 7fc2a7e00640 1 -- 172.21.15.102:0/226908334 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2980319d0 con 0x7fc2a815a850 2024-03-23T03:49:17.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.920+0000 7fc2a7e00640 1 -- 172.21.15.102:0/226908334 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298031d60 con 0x7fc2a815a850 2024-03-23T03:49:17.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.920+0000 7fc2b0400640 1 -- 172.21.15.102:0/226908334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a815a850 msgr2=0x7fc2a806aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:17.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.920+0000 7fc2b0400640 1 --2- 172.21.15.102:0/226908334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a815a850 0x7fc2a806aca0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fc2a806c3b0 tx=0x7fc29802b910 comp rx=0 tx=0).stop 2024-03-23T03:49:17.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.921+0000 7fc2b0400640 1 -- 172.21.15.102:0/226908334 shutdown_connections 2024-03-23T03:49:17.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.921+0000 7fc2b0400640 1 --2- 172.21.15.102:0/226908334 >> v2:172.21.15.173:3300/0 conn(0x7fc2a806d7b0 0x7fc2a806b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:17.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.921+0000 7fc2b0400640 1 --2- 172.21.15.102:0/226908334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a815a850 0x7fc2a806aca0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:17.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.921+0000 7fc2b0400640 1 -- 172.21.15.102:0/226908334 >> 172.21.15.102:0/226908334 conn(0x7fc2a8070a20 msgr2=0x7fc2a8070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:17.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.921+0000 7fc2b0400640 1 -- 172.21.15.102:0/226908334 shutdown_connections 2024-03-23T03:49:17.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.921+0000 7fc2b0400640 1 -- 172.21.15.102:0/226908334 wait complete. 2024-03-23T03:49:17.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.922+0000 7fc2b0400640 1 Processor -- start 2024-03-23T03:49:17.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.922+0000 7fc2b0400640 1 -- start start 2024-03-23T03:49:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.923+0000 7fc2b0400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a80659d0 0x7fc2a8099f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.923+0000 7fc2b0400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2a806d7b0 0x7fc2a813adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.923+0000 7fc2b0400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2a815a850 0x7fc2a81f3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.923+0000 7fc2b0400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a813c220 con 0x7fc2a80659d0 2024-03-23T03:49:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.923+0000 7fc2b0400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a813b300 con 0x7fc2a806d7b0 2024-03-23T03:49:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.923+0000 7fc2b0400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a813b5a0 con 0x7fc2a815a850 2024-03-23T03:49:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.923+0000 7fc2ada00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a80659d0 0x7fc2a8099f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.924+0000 7fc2ada00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a80659d0 0x7fc2a8099f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47424/0 (socket says 172.21.15.102:47424) 2024-03-23T03:49:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.924+0000 7fc2ada00640 1 -- 172.21.15.102:0/43723448 learned_addr learned my addr 172.21.15.102:0/43723448 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.924+0000 7fc2ada00640 1 -- 172.21.15.102:0/43723448 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2a815a850 msgr2=0x7fc2a81f3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.924+0000 7fc2ae400640 1 --2- 172.21.15.102:0/43723448 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2a815a850 0x7fc2a81f3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:17.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.924+0000 7fc2ada00640 1 --2- 172.21.15.102:0/43723448 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2a815a850 0x7fc2a81f3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.924+0000 7fc2ada00640 1 -- 172.21.15.102:0/43723448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2a806d7b0 msgr2=0x7fc2a813adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.924+0000 7fc2ada00640 1 --2- 172.21.15.102:0/43723448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2a806d7b0 0x7fc2a813adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.924+0000 7fc2ada00640 1 -- 172.21.15.102:0/43723448 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2980094d0 con 0x7fc2a80659d0 2024-03-23T03:49:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.924+0000 7fc2ada00640 1 --2- 172.21.15.102:0/43723448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a80659d0 0x7fc2a8099f80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc298000c00 tx=0x7fc2980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:17.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.925+0000 7fc2a5600640 1 -- 172.21.15.102:0/43723448 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29803ad10 con 0x7fc2a80659d0 2024-03-23T03:49:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.925+0000 7fc2a5600640 1 -- 172.21.15.102:0/43723448 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29803ae90 con 0x7fc2a80659d0 2024-03-23T03:49:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.925+0000 7fc2b0400640 1 -- 172.21.15.102:0/43723448 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a813b880 con 0x7fc2a80659d0 2024-03-23T03:49:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.925+0000 7fc2a5600640 1 -- 172.21.15.102:0/43723448 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc298044630 con 0x7fc2a80659d0 2024-03-23T03:49:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.925+0000 7fc2b0400640 1 -- 172.21.15.102:0/43723448 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a813be40 con 0x7fc2a80659d0 2024-03-23T03:49:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.926+0000 7fc2b0400640 1 -- 172.21.15.102:0/43723448 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a8066700 con 0x7fc2a80659d0 2024-03-23T03:49:17.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.927+0000 7fc2a5600640 1 -- 172.21.15.102:0/43723448 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc29804d450 con 0x7fc2a80659d0 2024-03-23T03:49:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.928+0000 7fc2a5600640 1 --2- 172.21.15.102:0/43723448 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc28806ac00 0x7fc28806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.928+0000 7fc2a5600640 1 -- 172.21.15.102:0/43723448 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fc29802d040 con 0x7fc2a80659d0 2024-03-23T03:49:17.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.928+0000 7fc2ad000640 1 --2- 172.21.15.102:0/43723448 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc28806ac00 0x7fc28806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:17.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.928+0000 7fc2ad000640 1 --2- 172.21.15.102:0/43723448 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc28806ac00 0x7fc28806d0c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8150650 tx=0x7fc294006c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:17.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:17.933+0000 7fc2a5600640 1 -- 172.21.15.102:0/43723448 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc298076ca0 con 0x7fc2a80659d0 2024-03-23T03:49:18.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:18.127+0000 7fc2b0400640 1 -- 172.21.15.102:0/43723448 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc2a8066bb0 con 0x7fc2a80659d0 2024-03-23T03:49:18.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:18.833+0000 7fc2a5600640 1 -- 172.21.15.102:0/43723448 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v57) v1 ==== 181+0+0 (secure 0 0 0) 0x7fc29807a2c0 con 0x7fc2a80659d0 2024-03-23T03:49:18.834 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' removed 2024-03-23T03:49:18.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:18.837+0000 7fc2b0400640 1 -- 172.21.15.102:0/43723448 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc28806ac00 msgr2=0x7fc28806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:18.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:18.837+0000 7fc2b0400640 1 --2- 172.21.15.102:0/43723448 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc28806ac00 0x7fc28806d0c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fc2a8150650 tx=0x7fc294006c90 comp rx=0 tx=0).stop 2024-03-23T03:49:18.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:18.837+0000 7fc2b0400640 1 -- 172.21.15.102:0/43723448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a80659d0 msgr2=0x7fc2a8099f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:18.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:18.837+0000 7fc2b0400640 1 --2- 172.21.15.102:0/43723448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a80659d0 0x7fc2a8099f80 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fc298000c00 tx=0x7fc2980046c0 comp rx=0 tx=0).stop 2024-03-23T03:49:18.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:18.837+0000 7fc2b0400640 1 -- 172.21.15.102:0/43723448 shutdown_connections 2024-03-23T03:49:18.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:18.837+0000 7fc2b0400640 1 --2- 172.21.15.102:0/43723448 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc28806ac00 0x7fc28806d0c0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:18.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:18.837+0000 7fc2b0400640 1 --2- 172.21.15.102:0/43723448 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc2a815a850 0x7fc2a81f3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:18.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:18.837+0000 7fc2b0400640 1 --2- 172.21.15.102:0/43723448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2a806d7b0 0x7fc2a813adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:18.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:18.837+0000 7fc2b0400640 1 --2- 172.21.15.102:0/43723448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc2a80659d0 0x7fc2a8099f80 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:18.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:18.837+0000 7fc2b0400640 1 -- 172.21.15.102:0/43723448 >> 172.21.15.102:0/43723448 conn(0x7fc2a8070a20 msgr2=0x7fc2a80b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:18.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:18.838+0000 7fc2b0400640 1 -- 172.21.15.102:0/43723448 shutdown_connections 2024-03-23T03:49:18.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:18.838+0000 7fc2b0400640 1 -- 172.21.15.102:0/43723448 wait complete. 2024-03-23T03:49:18.861 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T03:49:19.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.058+0000 7f6547600640 1 Processor -- start 2024-03-23T03:49:19.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.058+0000 7f6547600640 1 -- start start 2024-03-23T03:49:19.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.060+0000 7f6547600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f65400ce930 0x7f65400add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:19.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.060+0000 7f6547600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f654006ce90 0x7f654006d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:19.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.060+0000 7f6547600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65400cdba0 con 0x7f65400cd3e0 2024-03-23T03:49:19.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.060+0000 7f6547600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f654006c760 con 0x7f654006ce90 2024-03-23T03:49:19.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.060+0000 7f6547600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65400b42d0 con 0x7f65400ce930 2024-03-23T03:49:19.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.060+0000 7f6537e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f65400cd3e0 0x7f65400ced10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45180/0 (socket says 172.21.15.102:45180) 2024-03-23T03:49:19.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.060+0000 7f6537e00640 1 -- 172.21.15.102:0/3469965109 learned_addr learned my addr 172.21.15.102:0/3469965109 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:19.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f6544c00640 1 --2- 172.21.15.102:0/3469965109 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f65400ce930 0x7f65400add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:19.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f653fe00640 1 -- 172.21.15.102:0/3469965109 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829236239 0 0) 0x7f65400cdba0 con 0x7f65400cd3e0 2024-03-23T03:49:19.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f653fe00640 1 -- 172.21.15.102:0/3469965109 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6520003520 con 0x7f65400cd3e0 2024-03-23T03:49:19.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f6545600640 1 --2- 172.21.15.102:0/3469965109 >> v2:172.21.15.173:3300/0 conn(0x7f654006ce90 0x7f654006d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:19.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f653fe00640 1 -- 172.21.15.102:0/3469965109 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1195989817 0 0) 0x7f6520003520 con 0x7f65400cd3e0 2024-03-23T03:49:19.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f653fe00640 1 -- 172.21.15.102:0/3469965109 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f65400ce930 msgr2=0x7f65400add30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:19.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f653fe00640 1 --2- 172.21.15.102:0/3469965109 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f65400ce930 0x7f65400add30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:19.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f653fe00640 1 -- 172.21.15.102:0/3469965109 >> v2:172.21.15.173:3300/0 conn(0x7f654006ce90 msgr2=0x7f654006d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:19.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f653fe00640 1 --2- 172.21.15.102:0/3469965109 >> v2:172.21.15.173:3300/0 conn(0x7f654006ce90 0x7f654006d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:19.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f653fe00640 1 -- 172.21.15.102:0/3469965109 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65400cde80 con 0x7f65400cd3e0 2024-03-23T03:49:19.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f653fe00640 1 -- 172.21.15.102:0/3469965109 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f6530001670 con 0x7f65400cd3e0 2024-03-23T03:49:19.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f653fe00640 1 -- 172.21.15.102:0/3469965109 >> v1:172.21.15.102:6789/0 conn(0x7f65400cd3e0 legacy=0x7f65400ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:19.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f653fe00640 1 --2- 172.21.15.102:0/3469965109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6520003970 0x7f6520003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:19.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f653fe00640 1 --2- 172.21.15.102:0/3469965109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65200047c0 0x7f6520004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:19.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f653fe00640 1 --2- 172.21.15.102:0/3469965109 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f65400ce930 0x7f6520005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:19.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f653fe00640 1 -- 172.21.15.102:0/3469965109 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654006c760 con 0x7f65200047c0 2024-03-23T03:49:19.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f653fe00640 1 -- 172.21.15.102:0/3469965109 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65400b42d0 con 0x7f6520003970 2024-03-23T03:49:19.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f653fe00640 1 -- 172.21.15.102:0/3469965109 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f652000b980 con 0x7f65400ce930 2024-03-23T03:49:19.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f6545600640 1 --2- 172.21.15.102:0/3469965109 >> v2:172.21.15.173:3300/0 conn(0x7f654006ce90 0x7f654006d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:49:19.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.061+0000 7f6537e00640 1 --2- 172.21.15.102:0/3469965109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65200047c0 0x7f6520004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:19.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.062+0000 7f6537e00640 1 -- 172.21.15.102:0/3469965109 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f65400ce930 msgr2=0x7f6520005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:19.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.062+0000 7f6537e00640 1 --2- 172.21.15.102:0/3469965109 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f65400ce930 0x7f6520005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:19.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.062+0000 7f6537e00640 1 -- 172.21.15.102:0/3469965109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6520003970 msgr2=0x7f6520003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:19.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.062+0000 7f6537e00640 1 --2- 172.21.15.102:0/3469965109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6520003970 0x7f6520003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:19.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.062+0000 7f6537e00640 1 -- 172.21.15.102:0/3469965109 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6520003520 con 0x7f65200047c0 2024-03-23T03:49:19.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.063+0000 7f6537e00640 1 --2- 172.21.15.102:0/3469965109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65200047c0 0x7f6520004c40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f653000c240 tx=0x7f65200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b583d2666cc80c82 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:19.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.063+0000 7f653fe00640 1 -- 172.21.15.102:0/3469965109 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653002cb00 con 0x7f65200047c0 2024-03-23T03:49:19.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.063+0000 7f653fe00640 1 -- 172.21.15.102:0/3469965109 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653002cc60 con 0x7f65200047c0 2024-03-23T03:49:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.063+0000 7f653fe00640 1 -- 172.21.15.102:0/3469965109 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6530005af0 con 0x7f65200047c0 2024-03-23T03:49:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.064+0000 7f6547600640 1 -- 172.21.15.102:0/3469965109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65200047c0 msgr2=0x7f6520004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.064+0000 7f6547600640 1 --2- 172.21.15.102:0/3469965109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65200047c0 0x7f6520004c40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f653000c240 tx=0x7f65200057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.064+0000 7f6545600640 1 -- 172.21.15.102:0/3469965109 reap_dead start 2024-03-23T03:49:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.064+0000 7f6547600640 1 -- 172.21.15.102:0/3469965109 shutdown_connections 2024-03-23T03:49:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.064+0000 7f6547600640 1 -- 172.21.15.102:0/3469965109 >> 172.21.15.102:0/3469965109 conn(0x7f654006a0b0 msgr2=0x7f654006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.064+0000 7f6547600640 1 -- 172.21.15.102:0/3469965109 shutdown_connections 2024-03-23T03:49:19.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.064+0000 7f6547600640 1 -- 172.21.15.102:0/3469965109 wait complete. 2024-03-23T03:49:19.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.066+0000 7f6547600640 1 Processor -- start 2024-03-23T03:49:19.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.066+0000 7f6547600640 1 -- start start 2024-03-23T03:49:19.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.067+0000 7f6547600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65200047c0 0x7f65400b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:19.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.067+0000 7f6547600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65400acaa0 0x7f65400ae760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:19.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.067+0000 7f6547600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f65400acf00 0x7f65400aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:19.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.067+0000 7f6547600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65400b3630 con 0x7f65200047c0 2024-03-23T03:49:19.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.067+0000 7f6547600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65400ac2c0 con 0x7f65400acaa0 2024-03-23T03:49:19.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.067+0000 7f6547600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6540151ea0 con 0x7f65400acf00 2024-03-23T03:49:19.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.067+0000 7f6544c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65200047c0 0x7f65400b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:19.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.067+0000 7f6544c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65200047c0 0x7f65400b2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47442/0 (socket says 172.21.15.102:47442) 2024-03-23T03:49:19.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.067+0000 7f6544c00640 1 -- 172.21.15.102:0/3092902452 learned_addr learned my addr 172.21.15.102:0/3092902452 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:19.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.067+0000 7f6544c00640 1 -- 172.21.15.102:0/3092902452 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f65400acf00 msgr2=0x7f65400aeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:19.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.067+0000 7f6544c00640 1 --2- 172.21.15.102:0/3092902452 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f65400acf00 0x7f65400aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:19.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.067+0000 7f6544c00640 1 -- 172.21.15.102:0/3092902452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65400acaa0 msgr2=0x7f65400ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:19.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.067+0000 7f6544c00640 1 --2- 172.21.15.102:0/3092902452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65400acaa0 0x7f65400ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:19.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.067+0000 7f6544c00640 1 -- 172.21.15.102:0/3092902452 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6540152180 con 0x7f65200047c0 2024-03-23T03:49:19.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.068+0000 7f6544c00640 1 --2- 172.21.15.102:0/3092902452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65200047c0 0x7f65400b2c90 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f652800e7a0 tx=0x7f652800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:19.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.068+0000 7f653d600640 1 -- 172.21.15.102:0/3092902452 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652800eeb0 con 0x7f65200047c0 2024-03-23T03:49:19.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.068+0000 7f653d600640 1 -- 172.21.15.102:0/3092902452 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f652800c940 con 0x7f65200047c0 2024-03-23T03:49:19.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.068+0000 7f653d600640 1 -- 172.21.15.102:0/3092902452 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f652800cd10 con 0x7f65200047c0 2024-03-23T03:49:19.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.068+0000 7f6547600640 1 -- 172.21.15.102:0/3092902452 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65400ac580 con 0x7f65200047c0 2024-03-23T03:49:19.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.068+0000 7f6547600640 1 -- 172.21.15.102:0/3092902452 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f654014e2e0 con 0x7f65200047c0 2024-03-23T03:49:19.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.069+0000 7f6547600640 1 -- 172.21.15.102:0/3092902452 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65400b0e30 con 0x7f65200047c0 2024-03-23T03:49:19.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.070+0000 7f653d600640 1 -- 172.21.15.102:0/3092902452 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6528013470 con 0x7f65200047c0 2024-03-23T03:49:19.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.071+0000 7f653d600640 1 --2- 172.21.15.102:0/3092902452 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f651c06ac00 0x7f651c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:19.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.071+0000 7f653d600640 1 -- 172.21.15.102:0/3092902452 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5677+0+0 (secure 0 0 0) 0x7f6528089330 con 0x7f65200047c0 2024-03-23T03:49:19.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.071+0000 7f6537e00640 1 --2- 172.21.15.102:0/3092902452 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f651c06ac00 0x7f651c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:19.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.072+0000 7f6537e00640 1 --2- 172.21.15.102:0/3092902452 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f651c06ac00 0x7f651c06d0c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6530034000 tx=0x7f653002e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:19.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.075+0000 7f653d600640 1 -- 172.21.15.102:0/3092902452 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6528054520 con 0x7f65200047c0 2024-03-23T03:49:19.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.269+0000 7f6547600640 1 -- 172.21.15.102:0/3092902452 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6540073390 con 0x7f65200047c0 2024-03-23T03:49:19.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.837+0000 7f653d600640 1 -- 172.21.15.102:0/3092902452 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v58) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6528057b40 con 0x7f65200047c0 2024-03-23T03:49:19.838 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' removed 2024-03-23T03:49:19.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.843+0000 7f6547600640 1 -- 172.21.15.102:0/3092902452 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f651c06ac00 msgr2=0x7f651c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:19.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.843+0000 7f6547600640 1 --2- 172.21.15.102:0/3092902452 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f651c06ac00 0x7f651c06d0c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6530034000 tx=0x7f653002e000 comp rx=0 tx=0).stop 2024-03-23T03:49:19.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.843+0000 7f6547600640 1 -- 172.21.15.102:0/3092902452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65200047c0 msgr2=0x7f65400b2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:19.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.843+0000 7f6547600640 1 --2- 172.21.15.102:0/3092902452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65200047c0 0x7f65400b2c90 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f652800e7a0 tx=0x7f652800ec60 comp rx=0 tx=0).stop 2024-03-23T03:49:19.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.845+0000 7f6547600640 1 -- 172.21.15.102:0/3092902452 shutdown_connections 2024-03-23T03:49:19.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.845+0000 7f6547600640 1 --2- 172.21.15.102:0/3092902452 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f651c06ac00 0x7f651c06d0c0 secure :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6530034000 tx=0x7f653002e000 comp rx=0 tx=0).stop 2024-03-23T03:49:19.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.845+0000 7f6547600640 1 --2- 172.21.15.102:0/3092902452 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f65400acf00 0x7f65400aeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:19.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.845+0000 7f6547600640 1 --2- 172.21.15.102:0/3092902452 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f65400acaa0 0x7f65400ae760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:19.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.845+0000 7f6547600640 1 --2- 172.21.15.102:0/3092902452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65200047c0 0x7f65400b2c90 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:19.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.845+0000 7f6547600640 1 -- 172.21.15.102:0/3092902452 >> 172.21.15.102:0/3092902452 conn(0x7f654006a0b0 msgr2=0x7f65400ced10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:19.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.848+0000 7f6547600640 1 -- 172.21.15.102:0/3092902452 shutdown_connections 2024-03-23T03:49:19.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:19.849+0000 7f6547600640 1 -- 172.21.15.102:0/3092902452 wait complete. 2024-03-23T03:49:19.873 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-23T03:49:19.994 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:49:20.001 DEBUG:teuthology.orchestra.run.smithi173:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:49:20.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.060+0000 7f29a0c00640 1 Processor -- start 2024-03-23T03:49:20.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.060+0000 7f29a0c00640 1 -- start start 2024-03-23T03:49:20.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.062+0000 7f29a0c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f299c06b3b0 0x7f299c1654b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:20.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.062+0000 7f29a0c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f299c06d830 con 0x7f299c15aa10 2024-03-23T03:49:20.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.062+0000 7f29a0c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f299c0b3aa0 con 0x7f299c06d1c0 2024-03-23T03:49:20.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.062+0000 7f29a0c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f299c15adf0 con 0x7f299c06b3b0 2024-03-23T03:49:20.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.062+0000 7f2999e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f299c15aa10 0x7f299c06ee60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45184/0 (socket says 172.21.15.102:45184) 2024-03-23T03:49:20.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.062+0000 7f2999e00640 1 -- 172.21.15.102:0/3934532700 learned_addr learned my addr 172.21.15.102:0/3934532700 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:20.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.062+0000 7f299a800640 1 --2- 172.21.15.102:0/3934532700 >> v2:172.21.15.173:3300/0 conn(0x7f299c06b3b0 0x7f299c1654b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:20.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.063+0000 7f2998a00640 1 -- 172.21.15.102:0/3934532700 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820828427 0 0) 0x7f299c06d830 con 0x7f299c15aa10 2024-03-23T03:49:20.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.063+0000 7f2998a00640 1 -- 172.21.15.102:0/3934532700 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2980003520 con 0x7f299c15aa10 2024-03-23T03:49:20.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.063+0000 7f2998a00640 1 -- 172.21.15.102:0/3934532700 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097866743 0 0) 0x7f299c0b3aa0 con 0x7f299c06d1c0 2024-03-23T03:49:20.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.063+0000 7f299a800640 1 -- 172.21.15.102:0/3934532700 >> v1:172.21.15.173:6789/0 conn(0x7f299c06d1c0 legacy=0x7f299c06aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:20.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.063+0000 7f299a800640 1 -- 172.21.15.102:0/3934532700 >> v1:172.21.15.102:6789/0 conn(0x7f299c15aa10 legacy=0x7f299c06ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:20.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.063+0000 7f299a800640 1 -- 172.21.15.102:0/3934532700 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c0b3d80 con 0x7f299c06b3b0 2024-03-23T03:49:20.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.064+0000 7f299a800640 1 --2- 172.21.15.102:0/3934532700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f299c06b3b0 0x7f299c1654b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2994009d90 tx=0x7f299402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49b2b401bfc61f20 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:20.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.065+0000 7f2998a00640 1 -- 172.21.15.102:0/3934532700 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299402cc60 con 0x7f299c06b3b0 2024-03-23T03:49:20.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.065+0000 7f2998a00640 1 -- 172.21.15.102:0/3934532700 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299402cdc0 con 0x7f299c06b3b0 2024-03-23T03:49:20.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.065+0000 7f2998a00640 1 -- 172.21.15.102:0/3934532700 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2994034640 con 0x7f299c06b3b0 2024-03-23T03:49:20.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.065+0000 7f29a0c00640 1 -- 172.21.15.102:0/3934532700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f299c06b3b0 msgr2=0x7f299c1654b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:20.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.065+0000 7f29a0c00640 1 --2- 172.21.15.102:0/3934532700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f299c06b3b0 0x7f299c1654b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2994009d90 tx=0x7f299402beb0 comp rx=0 tx=0).stop 2024-03-23T03:49:20.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.066+0000 7f29a0c00640 1 -- 172.21.15.102:0/3934532700 shutdown_connections 2024-03-23T03:49:20.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.066+0000 7f29a0c00640 1 --2- 172.21.15.102:0/3934532700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f299c06b3b0 0x7f299c1654b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:20.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.066+0000 7f29a0c00640 1 -- 172.21.15.102:0/3934532700 >> 172.21.15.102:0/3934532700 conn(0x7f299c070a80 msgr2=0x7f299c070e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:20.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.066+0000 7f29a0c00640 1 -- 172.21.15.102:0/3934532700 shutdown_connections 2024-03-23T03:49:20.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.066+0000 7f29a0c00640 1 -- 172.21.15.102:0/3934532700 wait complete. 2024-03-23T03:49:20.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.067+0000 7f29a0c00640 1 Processor -- start 2024-03-23T03:49:20.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.068+0000 7f29a0c00640 1 -- start start 2024-03-23T03:49:20.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.068+0000 7f29a0c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f299c06b3b0 0x7f299c09a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:20.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.068+0000 7f29a0c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f299c06d1c0 0x7f299c14f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:20.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.068+0000 7f29a0c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f299c15aa10 0x7f299c0c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:20.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.069+0000 7f29a0c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c09a560 con 0x7f299c06d1c0 2024-03-23T03:49:20.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.069+0000 7f29a0c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c0998e0 con 0x7f299c06b3b0 2024-03-23T03:49:20.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.069+0000 7f29a0c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f299c099bc0 con 0x7f299c15aa10 2024-03-23T03:49:20.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.069+0000 7f2999400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f299c06d1c0 0x7f299c14f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:20.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.069+0000 7f2999400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f299c06d1c0 0x7f299c14f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47446/0 (socket says 172.21.15.102:47446) 2024-03-23T03:49:20.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.069+0000 7f2999400640 1 -- 172.21.15.102:0/962014095 learned_addr learned my addr 172.21.15.102:0/962014095 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:20.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.069+0000 7f2999e00640 1 --2- 172.21.15.102:0/962014095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f299c06b3b0 0x7f299c09a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:20.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.069+0000 7f299a800640 1 --2- 172.21.15.102:0/962014095 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f299c15aa10 0x7f299c0c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:20.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.069+0000 7f2999400640 1 -- 172.21.15.102:0/962014095 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f299c15aa10 msgr2=0x7f299c0c88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:20.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.069+0000 7f2999400640 1 --2- 172.21.15.102:0/962014095 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f299c15aa10 0x7f299c0c88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:20.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.069+0000 7f2999400640 1 -- 172.21.15.102:0/962014095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f299c06b3b0 msgr2=0x7f299c09a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:20.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.069+0000 7f2999400640 1 --2- 172.21.15.102:0/962014095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f299c06b3b0 0x7f299c09a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:20.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.069+0000 7f2999400640 1 -- 172.21.15.102:0/962014095 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2994009a70 con 0x7f299c06d1c0 2024-03-23T03:49:20.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.070+0000 7f2999400640 1 --2- 172.21.15.102:0/962014095 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f299c06d1c0 0x7f299c14f780 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f298800c460 tx=0x7f298800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:20.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.070+0000 7f2992000640 1 -- 172.21.15.102:0/962014095 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2988005480 con 0x7f299c06d1c0 2024-03-23T03:49:20.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.070+0000 7f2992000640 1 -- 172.21.15.102:0/962014095 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29880035f0 con 0x7f299c06d1c0 2024-03-23T03:49:20.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.070+0000 7f2992000640 1 -- 172.21.15.102:0/962014095 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29880039a0 con 0x7f299c06d1c0 2024-03-23T03:49:20.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.070+0000 7f29a0c00640 1 -- 172.21.15.102:0/962014095 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f299c099ea0 con 0x7f299c06d1c0 2024-03-23T03:49:20.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.070+0000 7f29a0c00640 1 -- 172.21.15.102:0/962014095 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f299c09c730 con 0x7f299c06d1c0 2024-03-23T03:49:20.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.071+0000 7f29a0c00640 1 -- 172.21.15.102:0/962014095 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f299c067750 con 0x7f299c06d1c0 2024-03-23T03:49:20.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.072+0000 7f2992000640 1 -- 172.21.15.102:0/962014095 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2988003b00 con 0x7f299c06d1c0 2024-03-23T03:49:20.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.073+0000 7f2992000640 1 --2- 172.21.15.102:0/962014095 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f297c06ac00 0x7f297c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:20.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.073+0000 7f2992000640 1 -- 172.21.15.102:0/962014095 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f2988088b60 con 0x7f299c06d1c0 2024-03-23T03:49:20.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.073+0000 7f2999e00640 1 --2- 172.21.15.102:0/962014095 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f297c06ac00 0x7f297c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:20.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.074+0000 7f2999e00640 1 --2- 172.21.15.102:0/962014095 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f297c06ac00 0x7f297c06d0c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f298c005880 tx=0x7f298c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:20.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.078+0000 7f2992000640 1 -- 172.21.15.102:0/962014095 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2988053fa0 con 0x7f299c06d1c0 2024-03-23T03:49:20.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.275+0000 7f29a0c00640 1 -- 172.21.15.102:0/962014095 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f299c066bb0 con 0x7f299c06d1c0 2024-03-23T03:49:20.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.284+0000 7f2992000640 1 -- 172.21.15.102:0/962014095 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f29880575c0 con 0x7f299c06d1c0 2024-03-23T03:49:20.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.287+0000 7f29a0c00640 1 -- 172.21.15.102:0/962014095 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f297c06ac00 msgr2=0x7f297c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:20.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.287+0000 7f29a0c00640 1 --2- 172.21.15.102:0/962014095 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f297c06ac00 0x7f297c06d0c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f298c005880 tx=0x7f298c009040 comp rx=0 tx=0).stop 2024-03-23T03:49:20.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.287+0000 7f29a0c00640 1 -- 172.21.15.102:0/962014095 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f299c06d1c0 msgr2=0x7f299c14f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:20.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.287+0000 7f29a0c00640 1 --2- 172.21.15.102:0/962014095 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f299c06d1c0 0x7f299c14f780 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f298800c460 tx=0x7f298800c920 comp rx=0 tx=0).stop 2024-03-23T03:49:20.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.287+0000 7f29a0c00640 1 -- 172.21.15.102:0/962014095 shutdown_connections 2024-03-23T03:49:20.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.287+0000 7f29a0c00640 1 --2- 172.21.15.102:0/962014095 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f297c06ac00 0x7f297c06d0c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:20.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.287+0000 7f29a0c00640 1 --2- 172.21.15.102:0/962014095 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f299c15aa10 0x7f299c0c88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:20.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.287+0000 7f29a0c00640 1 --2- 172.21.15.102:0/962014095 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f299c06d1c0 0x7f299c14f780 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:20.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.287+0000 7f29a0c00640 1 --2- 172.21.15.102:0/962014095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f299c06b3b0 0x7f299c09a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:20.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.287+0000 7f29a0c00640 1 -- 172.21.15.102:0/962014095 >> 172.21.15.102:0/962014095 conn(0x7f299c070a80 msgr2=0x7f299c070e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:20.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.287+0000 7f29a0c00640 1 -- 172.21.15.102:0/962014095 shutdown_connections 2024-03-23T03:49:20.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.287+0000 7f29a0c00640 1 -- 172.21.15.102:0/962014095 wait complete. 2024-03-23T03:49:20.306 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values' 2024-03-23T03:49:20.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.505+0000 7f5b5cc00640 1 Processor -- start 2024-03-23T03:49:20.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.505+0000 7f5b5cc00640 1 -- start start 2024-03-23T03:49:20.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.507+0000 7f5b5cc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5b580b3e30 0x7f5b5815cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:20.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.507+0000 7f5b5cc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5806c6f0 con 0x7f5b5815b560 2024-03-23T03:49:20.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.507+0000 7f5b5cc00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b5806c9d0 con 0x7f5b580b39b0 2024-03-23T03:49:20.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.507+0000 7f5b5cc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b5806ccb0 con 0x7f5b580b3e30 2024-03-23T03:49:20.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.507+0000 7f5b55e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5b5815b560 0x7f5b5815b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45188/0 (socket says 172.21.15.102:45188) 2024-03-23T03:49:20.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.507+0000 7f5b55e00640 1 -- 172.21.15.102:0/1123365915 learned_addr learned my addr 172.21.15.102:0/1123365915 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:20.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.507+0000 7f5b56800640 1 --2- 172.21.15.102:0/1123365915 >> v2:172.21.15.102:3300/0 conn(0x7f5b580b3e30 0x7f5b5815cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:20.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.508+0000 7f5b56800640 1 -- 172.21.15.102:0/1123365915 >> v1:172.21.15.173:6789/0 conn(0x7f5b580b39b0 legacy=0x7f5b5815c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:20.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.508+0000 7f5b56800640 1 -- 172.21.15.102:0/1123365915 >> v1:172.21.15.102:6789/0 conn(0x7f5b5815b560 legacy=0x7f5b5815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:20.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.508+0000 7f5b56800640 1 -- 172.21.15.102:0/1123365915 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b5806cf90 con 0x7f5b580b3e30 2024-03-23T03:49:20.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.509+0000 7f5b56800640 1 --2- 172.21.15.102:0/1123365915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b580b3e30 0x7f5b5815cb10 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5b500097a0 tx=0x7f5b5002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0cdd22fee10146c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:20.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.509+0000 7f5b54a00640 1 -- 172.21.15.102:0/1123365915 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b50031830 con 0x7f5b580b3e30 2024-03-23T03:49:20.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.509+0000 7f5b54a00640 1 -- 172.21.15.102:0/1123365915 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b50031990 con 0x7f5b580b3e30 2024-03-23T03:49:20.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.509+0000 7f5b54a00640 1 -- 172.21.15.102:0/1123365915 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b50031d20 con 0x7f5b580b3e30 2024-03-23T03:49:20.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.510+0000 7f5b5cc00640 1 -- 172.21.15.102:0/1123365915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b580b3e30 msgr2=0x7f5b5815cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:20.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.510+0000 7f5b5cc00640 1 --2- 172.21.15.102:0/1123365915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b580b3e30 0x7f5b5815cb10 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5b500097a0 tx=0x7f5b5002b8d0 comp rx=0 tx=0).stop 2024-03-23T03:49:20.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.510+0000 7f5b5cc00640 1 -- 172.21.15.102:0/1123365915 shutdown_connections 2024-03-23T03:49:20.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.510+0000 7f5b5cc00640 1 --2- 172.21.15.102:0/1123365915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b580b3e30 0x7f5b5815cb10 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:20.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.510+0000 7f5b5cc00640 1 -- 172.21.15.102:0/1123365915 >> 172.21.15.102:0/1123365915 conn(0x7f5b5806a3d0 msgr2=0x7f5b580b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:20.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.510+0000 7f5b5cc00640 1 -- 172.21.15.102:0/1123365915 shutdown_connections 2024-03-23T03:49:20.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.510+0000 7f5b5cc00640 1 -- 172.21.15.102:0/1123365915 wait complete. 2024-03-23T03:49:20.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.512+0000 7f5b5cc00640 1 Processor -- start 2024-03-23T03:49:20.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.512+0000 7f5b5cc00640 1 -- start start 2024-03-23T03:49:20.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.513+0000 7f5b5cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b580b39b0 0x7f5b58153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:20.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.513+0000 7f5b5cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b580b3e30 0x7f5b581524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:20.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.513+0000 7f5b5cc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5b5815b560 0x7f5b580af360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:20.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.513+0000 7f5b5cc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b58154e10 con 0x7f5b580b3e30 2024-03-23T03:49:20.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.513+0000 7f5b5cc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b58154850 con 0x7f5b580b39b0 2024-03-23T03:49:20.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.513+0000 7f5b5cc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b58154b30 con 0x7f5b5815b560 2024-03-23T03:49:20.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.513+0000 7f5b55400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b580b3e30 0x7f5b581524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:20.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.513+0000 7f5b55400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b580b3e30 0x7f5b581524d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47462/0 (socket says 172.21.15.102:47462) 2024-03-23T03:49:20.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.513+0000 7f5b55400640 1 -- 172.21.15.102:0/1840943369 learned_addr learned my addr 172.21.15.102:0/1840943369 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:20.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.513+0000 7f5b55400640 1 -- 172.21.15.102:0/1840943369 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5b5815b560 msgr2=0x7f5b580af360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:20.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.513+0000 7f5b55400640 1 --2- 172.21.15.102:0/1840943369 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5b5815b560 0x7f5b580af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:20.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.514+0000 7f5b55400640 1 -- 172.21.15.102:0/1840943369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b580b39b0 msgr2=0x7f5b58153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:20.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.514+0000 7f5b55400640 1 --2- 172.21.15.102:0/1840943369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b580b39b0 0x7f5b58153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:20.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.514+0000 7f5b55400640 1 -- 172.21.15.102:0/1840943369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b50009400 con 0x7f5b580b3e30 2024-03-23T03:49:20.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.514+0000 7f5b55400640 1 --2- 172.21.15.102:0/1840943369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b580b3e30 0x7f5b581524d0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f5b4000c7c0 tx=0x7f5b4000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:20.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.514+0000 7f5b4e000640 1 -- 172.21.15.102:0/1840943369 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4000aa50 con 0x7f5b580b3e30 2024-03-23T03:49:20.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.514+0000 7f5b4e000640 1 -- 172.21.15.102:0/1840943369 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b4000abb0 con 0x7f5b580b3e30 2024-03-23T03:49:20.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.514+0000 7f5b5cc00640 1 -- 172.21.15.102:0/1840943369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b580b18d0 con 0x7f5b580b3e30 2024-03-23T03:49:20.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.514+0000 7f5b4e000640 1 -- 172.21.15.102:0/1840943369 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b40012420 con 0x7f5b580b3e30 2024-03-23T03:49:20.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.514+0000 7f5b5cc00640 1 -- 172.21.15.102:0/1840943369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b581544d0 con 0x7f5b580b3e30 2024-03-23T03:49:20.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.515+0000 7f5b5cc00640 1 -- 172.21.15.102:0/1840943369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b580b1df0 con 0x7f5b580b3e30 2024-03-23T03:49:20.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.516+0000 7f5b4e000640 1 -- 172.21.15.102:0/1840943369 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5b40012580 con 0x7f5b580b3e30 2024-03-23T03:49:20.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.518+0000 7f5b4e000640 1 --2- 172.21.15.102:0/1840943369 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b3406ac00 0x7f5b3406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:20.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.518+0000 7f5b4e000640 1 -- 172.21.15.102:0/1840943369 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f5b40089120 con 0x7f5b580b3e30 2024-03-23T03:49:20.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.518+0000 7f5b55e00640 1 --2- 172.21.15.102:0/1840943369 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b3406ac00 0x7f5b3406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:20.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.519+0000 7f5b55e00640 1 --2- 172.21.15.102:0/1840943369 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b3406ac00 0x7f5b3406d0c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5b48001550 tx=0x7f5b48007a40 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:20.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.522+0000 7f5b4e000640 1 -- 172.21.15.102:0/1840943369 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5b580b1df0 con 0x7f5b580b3e30 2024-03-23T03:49:20.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.799+0000 7f5b5cc00640 1 -- 172.21.15.102:0/1840943369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values"]} v 0) v1 -- 0x7f5b580b1df0 con 0x7f5b580b3e30 2024-03-23T03:49:20.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.861+0000 7f5b4e000640 1 -- 172.21.15.102:0/1840943369 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_invalid_values"]}]=0 v155) v1 ==== 153+0+0 (secure 0 0 0) 0x7f5b580b1df0 con 0x7f5b580b3e30 2024-03-23T03:49:20.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.865+0000 7f5b5cc00640 1 -- 172.21.15.102:0/1840943369 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b3406ac00 msgr2=0x7f5b3406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:20.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.865+0000 7f5b5cc00640 1 --2- 172.21.15.102:0/1840943369 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b3406ac00 0x7f5b3406d0c0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f5b48001550 tx=0x7f5b48007a40 comp rx=0 tx=0).stop 2024-03-23T03:49:20.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.865+0000 7f5b5cc00640 1 -- 172.21.15.102:0/1840943369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b580b3e30 msgr2=0x7f5b581524d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:20.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.865+0000 7f5b5cc00640 1 --2- 172.21.15.102:0/1840943369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b580b3e30 0x7f5b581524d0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f5b4000c7c0 tx=0x7f5b4000cc80 comp rx=0 tx=0).stop 2024-03-23T03:49:20.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.865+0000 7f5b5cc00640 1 -- 172.21.15.102:0/1840943369 shutdown_connections 2024-03-23T03:49:20.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.865+0000 7f5b5cc00640 1 --2- 172.21.15.102:0/1840943369 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5b3406ac00 0x7f5b3406d0c0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:20.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.865+0000 7f5b5cc00640 1 --2- 172.21.15.102:0/1840943369 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5b5815b560 0x7f5b580af360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:20.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.865+0000 7f5b5cc00640 1 --2- 172.21.15.102:0/1840943369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5b580b3e30 0x7f5b581524d0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:20.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.865+0000 7f5b5cc00640 1 --2- 172.21.15.102:0/1840943369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5b580b39b0 0x7f5b58153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:20.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.865+0000 7f5b5cc00640 1 -- 172.21.15.102:0/1840943369 >> 172.21.15.102:0/1840943369 conn(0x7f5b5806a3d0 msgr2=0x7f5b5815a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:20.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.865+0000 7f5b5cc00640 1 -- 172.21.15.102:0/1840943369 shutdown_connections 2024-03-23T03:49:20.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:20.865+0000 7f5b5cc00640 1 -- 172.21.15.102:0/1840943369 wait complete. 2024-03-23T03:49:20.885 INFO:tasks.cephfs_test_runner:test invalid values for ceph.quota.max_bytes ... ok 2024-03-23T03:49:20.885 INFO:tasks.cephfs_test_runner:Starting test: test_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) test human-readable values for setting ceph.quota.max_bytes 2024-03-23T03:49:20.885 INFO:tasks.cephfs_test_runner:test_human_readable_quota_values (tasks.cephfs.test_quota.TestQuota) 2024-03-23T03:49:20.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values' 2024-03-23T03:49:21.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.086+0000 7f8b93000640 1 Processor -- start 2024-03-23T03:49:21.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.087+0000 7f8b93000640 1 -- start start 2024-03-23T03:49:21.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.087+0000 7f8b93000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f8b8c0cb280 0x7f8b8c0ce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:21.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.087+0000 7f8b93000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8b8c06fac0 0x7f8b8c0cec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:21.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.087+0000 7f8b93000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8c06e620 con 0x7f8b8c0ce780 2024-03-23T03:49:21.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.087+0000 7f8b93000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b8c099e20 con 0x7f8b8c06fac0 2024-03-23T03:49:21.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.087+0000 7f8b93000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b8c099580 con 0x7f8b8c0cb280 2024-03-23T03:49:21.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.087+0000 7f8b8b400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8b8c0ce780 0x7f8b8c0cb660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:45198/0 (socket says 172.21.15.102:45198) 2024-03-23T03:49:21.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.087+0000 7f8b8b400640 1 -- 172.21.15.102:0/200382314 learned_addr learned my addr 172.21.15.102:0/200382314 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:21.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.087+0000 7f8b91000640 1 --2- 172.21.15.102:0/200382314 >> v2:172.21.15.102:3300/0 conn(0x7f8b8c06fac0 0x7f8b8c0cec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:21.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.088+0000 7f8b8be00640 1 --2- 172.21.15.102:0/200382314 >> v2:172.21.15.173:3300/0 conn(0x7f8b8c0cb280 0x7f8b8c0ce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:21.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.088+0000 7f8b8aa00640 1 -- 172.21.15.102:0/200382314 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246944303 0 0) 0x7f8b8c06e620 con 0x7f8b8c0ce780 2024-03-23T03:49:21.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.088+0000 7f8b91000640 1 -- 172.21.15.102:0/200382314 >> v2:172.21.15.173:3300/0 conn(0x7f8b8c0cb280 msgr2=0x7f8b8c0ce240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:21.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.088+0000 7f8b91000640 1 --2- 172.21.15.102:0/200382314 >> v2:172.21.15.173:3300/0 conn(0x7f8b8c0cb280 0x7f8b8c0ce240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:21.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.088+0000 7f8b91000640 1 -- 172.21.15.102:0/200382314 >> v1:172.21.15.102:6789/0 conn(0x7f8b8c0ce780 legacy=0x7f8b8c0cb660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:21.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.088+0000 7f8b91000640 1 -- 172.21.15.102:0/200382314 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8c099860 con 0x7f8b8c06fac0 2024-03-23T03:49:21.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.089+0000 7f8b91000640 1 --2- 172.21.15.102:0/200382314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b8c06fac0 0x7f8b8c0cec30 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c0706b0 tx=0x7f8b8402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b5e4efb8bd6590 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:21.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.089+0000 7f8b8aa00640 1 -- 172.21.15.102:0/200382314 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8402cb60 con 0x7f8b8c06fac0 2024-03-23T03:49:21.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.089+0000 7f8b8aa00640 1 -- 172.21.15.102:0/200382314 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8402ccc0 con 0x7f8b8c06fac0 2024-03-23T03:49:21.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.089+0000 7f8b8aa00640 1 -- 172.21.15.102:0/200382314 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b84034660 con 0x7f8b8c06fac0 2024-03-23T03:49:21.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.090+0000 7f8b93000640 1 -- 172.21.15.102:0/200382314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b8c06fac0 msgr2=0x7f8b8c0cec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:21.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.090+0000 7f8b93000640 1 --2- 172.21.15.102:0/200382314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b8c06fac0 0x7f8b8c0cec30 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c0706b0 tx=0x7f8b8402bdb0 comp rx=0 tx=0).stop 2024-03-23T03:49:21.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.090+0000 7f8b93000640 1 -- 172.21.15.102:0/200382314 shutdown_connections 2024-03-23T03:49:21.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.090+0000 7f8b93000640 1 --2- 172.21.15.102:0/200382314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b8c06fac0 0x7f8b8c0cec30 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:21.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.090+0000 7f8b93000640 1 --2- 172.21.15.102:0/200382314 >> v2:172.21.15.173:3300/0 conn(0x7f8b8c0cb280 0x7f8b8c0ce240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:21.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.090+0000 7f8b93000640 1 -- 172.21.15.102:0/200382314 >> 172.21.15.102:0/200382314 conn(0x7f8b8c0b2160 msgr2=0x7f8b8c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:21.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.090+0000 7f8b93000640 1 -- 172.21.15.102:0/200382314 shutdown_connections 2024-03-23T03:49:21.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.090+0000 7f8b93000640 1 -- 172.21.15.102:0/200382314 wait complete. 2024-03-23T03:49:21.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.092+0000 7f8b93000640 1 Processor -- start 2024-03-23T03:49:21.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.092+0000 7f8b93000640 1 -- start start 2024-03-23T03:49:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.093+0000 7f8b93000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b8c06fac0 0x7f8b8c158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.093+0000 7f8b93000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b8c0cb280 0x7f8b8c0ccf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.093+0000 7f8b93000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8b8c0ce780 0x7f8b8c0cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.093+0000 7f8b93000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c0c78c0 con 0x7f8b8c0cb280 2024-03-23T03:49:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.093+0000 7f8b93000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c0c7ba0 con 0x7f8b8c06fac0 2024-03-23T03:49:21.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.093+0000 7f8b93000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c0c7e80 con 0x7f8b8c0ce780 2024-03-23T03:49:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.094+0000 7f8b8b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b8c0cb280 0x7f8b8c0ccf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.094+0000 7f8b8b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b8c0cb280 0x7f8b8c0ccf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47488/0 (socket says 172.21.15.102:47488) 2024-03-23T03:49:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.094+0000 7f8b8b400640 1 -- 172.21.15.102:0/855824608 learned_addr learned my addr 172.21.15.102:0/855824608 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.094+0000 7f8b8b400640 1 -- 172.21.15.102:0/855824608 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8b8c0ce780 msgr2=0x7f8b8c0cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.094+0000 7f8b8b400640 1 --2- 172.21.15.102:0/855824608 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8b8c0ce780 0x7f8b8c0cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.094+0000 7f8b8b400640 1 -- 172.21.15.102:0/855824608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b8c06fac0 msgr2=0x7f8b8c158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.094+0000 7f8b8b400640 1 --2- 172.21.15.102:0/855824608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b8c06fac0 0x7f8b8c158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.094+0000 7f8b8b400640 1 -- 172.21.15.102:0/855824608 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b84009a70 con 0x7f8b8c0cb280 2024-03-23T03:49:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.094+0000 7f8b8b400640 1 --2- 172.21.15.102:0/855824608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b8c0cb280 0x7f8b8c0ccf50 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8b7400c430 tx=0x7f8b7400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.094+0000 7f8b83e00640 1 -- 172.21.15.102:0/855824608 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b740074e0 con 0x7f8b8c0cb280 2024-03-23T03:49:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.095+0000 7f8b83e00640 1 -- 172.21.15.102:0/855824608 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b74007660 con 0x7f8b8c0cb280 2024-03-23T03:49:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.095+0000 7f8b83e00640 1 -- 172.21.15.102:0/855824608 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b74004a20 con 0x7f8b8c0cb280 2024-03-23T03:49:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.095+0000 7f8b93000640 1 -- 172.21.15.102:0/855824608 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b8c15adc0 con 0x7f8b8c0cb280 2024-03-23T03:49:21.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.095+0000 7f8b93000640 1 -- 172.21.15.102:0/855824608 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b8c15c8f0 con 0x7f8b8c0cb280 2024-03-23T03:49:21.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.096+0000 7f8b93000640 1 -- 172.21.15.102:0/855824608 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b8c15d2d0 con 0x7f8b8c0cb280 2024-03-23T03:49:21.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.097+0000 7f8b83e00640 1 -- 172.21.15.102:0/855824608 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8b74004b80 con 0x7f8b8c0cb280 2024-03-23T03:49:21.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.098+0000 7f8b83e00640 1 --2- 172.21.15.102:0/855824608 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8b6806ac00 0x7f8b6806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:21.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.098+0000 7f8b83e00640 1 -- 172.21.15.102:0/855824608 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f8b74088fd0 con 0x7f8b8c0cb280 2024-03-23T03:49:21.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.098+0000 7f8b8be00640 1 --2- 172.21.15.102:0/855824608 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8b6806ac00 0x7f8b6806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:21.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.099+0000 7f8b8be00640 1 --2- 172.21.15.102:0/855824608 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8b6806ac00 0x7f8b6806d0c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c006fd0 tx=0x7f8b7c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:21.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.102+0000 7f8b83e00640 1 -- 172.21.15.102:0/855824608 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8b74054370 con 0x7f8b8c0cb280 2024-03-23T03:49:21.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.370+0000 7f8b93000640 1 -- 172.21.15.102:0/855824608 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values"]} v 0) v1 -- 0x7f8b8c066bb0 con 0x7f8b8c0cb280 2024-03-23T03:49:21.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.872+0000 7f8b83e00640 1 -- 172.21.15.102:0/855824608 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values"]}]=0 v156) v1 ==== 148+0+0 (secure 0 0 0) 0x7f8b74057990 con 0x7f8b8c0cb280 2024-03-23T03:49:21.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.875+0000 7f8b93000640 1 -- 172.21.15.102:0/855824608 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8b6806ac00 msgr2=0x7f8b6806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:21.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.875+0000 7f8b93000640 1 --2- 172.21.15.102:0/855824608 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8b6806ac00 0x7f8b6806d0c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c006fd0 tx=0x7f8b7c008040 comp rx=0 tx=0).stop 2024-03-23T03:49:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.875+0000 7f8b93000640 1 -- 172.21.15.102:0/855824608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b8c0cb280 msgr2=0x7f8b8c0ccf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.875+0000 7f8b93000640 1 --2- 172.21.15.102:0/855824608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b8c0cb280 0x7f8b8c0ccf50 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8b7400c430 tx=0x7f8b7400c8f0 comp rx=0 tx=0).stop 2024-03-23T03:49:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.875+0000 7f8b93000640 1 -- 172.21.15.102:0/855824608 shutdown_connections 2024-03-23T03:49:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.875+0000 7f8b93000640 1 --2- 172.21.15.102:0/855824608 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8b6806ac00 0x7f8b6806d0c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.875+0000 7f8b93000640 1 --2- 172.21.15.102:0/855824608 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8b8c0ce780 0x7f8b8c0cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.875+0000 7f8b93000640 1 --2- 172.21.15.102:0/855824608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b8c0cb280 0x7f8b8c0ccf50 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.875+0000 7f8b93000640 1 --2- 172.21.15.102:0/855824608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b8c06fac0 0x7f8b8c158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.875+0000 7f8b93000640 1 -- 172.21.15.102:0/855824608 >> 172.21.15.102:0/855824608 conn(0x7f8b8c0b2160 msgr2=0x7f8b8c0b1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:21.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.876+0000 7f8b93000640 1 -- 172.21.15.102:0/855824608 shutdown_connections 2024-03-23T03:49:21.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:21.876+0000 7f8b93000640 1 -- 172.21.15.102:0/855824608 wait complete. 2024-03-23T03:49:21.897 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-23T03:49:22.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.104+0000 7fd7dd000640 1 Processor -- start 2024-03-23T03:49:22.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.105+0000 7fd7dd000640 1 -- start start 2024-03-23T03:49:22.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.106+0000 7fd7dd000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd7d80cd740 0x7fd7d80b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.106+0000 7fd7dd000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7d80d0d40 0x7fd7d80cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.106+0000 7fd7dd000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7d806b0a0 0x7fd7d806b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.106+0000 7fd7dd000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7d80d20c0 con 0x7fd7d80d0d40 2024-03-23T03:49:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.106+0000 7fd7dd000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7d80ce060 con 0x7fd7d80cd740 2024-03-23T03:49:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.106+0000 7fd7dd000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d80d14b0 con 0x7fd7d806b0a0 2024-03-23T03:49:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.106+0000 7fd7d5400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7d80d0d40 0x7fd7d80cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.107+0000 7fd7d5400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd7d80d0d40 0x7fd7d80cdb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47494/0 (socket says 172.21.15.102:47494) 2024-03-23T03:49:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.107+0000 7fd7d5400640 1 -- 172.21.15.102:0/731690410 learned_addr learned my addr 172.21.15.102:0/731690410 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.107+0000 7fd7d6800640 1 --2- 172.21.15.102:0/731690410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7d806b0a0 0x7fd7d806b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.107+0000 7fd7d5e00640 1 --2- 172.21.15.102:0/731690410 >> v2:172.21.15.173:3300/0 conn(0x7fd7d80cd740 0x7fd7d80b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.107+0000 7fd7d5400640 1 -- 172.21.15.102:0/731690410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7d806b0a0 msgr2=0x7fd7d806b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.107+0000 7fd7d5400640 1 --2- 172.21.15.102:0/731690410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7d806b0a0 0x7fd7d806b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:22.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.107+0000 7fd7d5400640 1 -- 172.21.15.102:0/731690410 >> v2:172.21.15.173:3300/0 conn(0x7fd7d80cd740 msgr2=0x7fd7d80b84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.107+0000 7fd7d5400640 1 --2- 172.21.15.102:0/731690410 >> v2:172.21.15.173:3300/0 conn(0x7fd7d80cd740 0x7fd7d80b84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.107+0000 7fd7d5400640 1 -- 172.21.15.102:0/731690410 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d80d1790 con 0x7fd7d80d0d40 2024-03-23T03:49:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.107+0000 7fd7d5e00640 1 --2- 172.21.15.102:0/731690410 >> v2:172.21.15.173:3300/0 conn(0x7fd7d80cd740 0x7fd7d80b84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.108+0000 7fd7d5400640 1 --2- 172.21.15.102:0/731690410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d80d0d40 0x7fd7d80cdb20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd7d808a100 tx=0x7fd7c802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe23b6cfbcf5d75 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.109+0000 7fd7d4a00640 1 -- 172.21.15.102:0/731690410 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c802ca70 con 0x7fd7d80d0d40 2024-03-23T03:49:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.109+0000 7fd7d4a00640 1 -- 172.21.15.102:0/731690410 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c802cbd0 con 0x7fd7d80d0d40 2024-03-23T03:49:22.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.109+0000 7fd7d4a00640 1 -- 172.21.15.102:0/731690410 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c80343f0 con 0x7fd7d80d0d40 2024-03-23T03:49:22.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.109+0000 7fd7dd000640 1 -- 172.21.15.102:0/731690410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d80d0d40 msgr2=0x7fd7d80cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:22.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.109+0000 7fd7dd000640 1 --2- 172.21.15.102:0/731690410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d80d0d40 0x7fd7d80cdb20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd7d808a100 tx=0x7fd7c802bed0 comp rx=0 tx=0).stop 2024-03-23T03:49:22.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.110+0000 7fd7dd000640 1 -- 172.21.15.102:0/731690410 shutdown_connections 2024-03-23T03:49:22.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.110+0000 7fd7dd000640 1 --2- 172.21.15.102:0/731690410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7d806b0a0 0x7fd7d806b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:22.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.110+0000 7fd7dd000640 1 --2- 172.21.15.102:0/731690410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d80d0d40 0x7fd7d80cdb20 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:22.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.110+0000 7fd7dd000640 1 --2- 172.21.15.102:0/731690410 >> v2:172.21.15.173:3300/0 conn(0x7fd7d80cd740 0x7fd7d80b84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:22.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.110+0000 7fd7dd000640 1 -- 172.21.15.102:0/731690410 >> 172.21.15.102:0/731690410 conn(0x7fd7d80cf5d0 msgr2=0x7fd7d80cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:22.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.110+0000 7fd7dd000640 1 -- 172.21.15.102:0/731690410 shutdown_connections 2024-03-23T03:49:22.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.110+0000 7fd7dd000640 1 -- 172.21.15.102:0/731690410 wait complete. 2024-03-23T03:49:22.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.112+0000 7fd7dd000640 1 Processor -- start 2024-03-23T03:49:22.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.112+0000 7fd7dd000640 1 -- start start 2024-03-23T03:49:22.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.113+0000 7fd7dd000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d806b0a0 0x7fd7d81f1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:22.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.113+0000 7fd7dd000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7d80cd740 0x7fd7d80cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:22.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.114+0000 7fd7dd000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7d80d0d40 0x7fd7d81593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:22.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.114+0000 7fd7dd000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d8158730 con 0x7fd7d806b0a0 2024-03-23T03:49:22.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.114+0000 7fd7dd000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d8158a10 con 0x7fd7d80d0d40 2024-03-23T03:49:22.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.114+0000 7fd7dd000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d8158cf0 con 0x7fd7d80cd740 2024-03-23T03:49:22.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.114+0000 7fd7d5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d806b0a0 0x7fd7d81f1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:22.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.114+0000 7fd7d5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d806b0a0 0x7fd7d81f1cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:47500/0 (socket says 172.21.15.102:47500) 2024-03-23T03:49:22.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.114+0000 7fd7d5e00640 1 -- 172.21.15.102:0/1419456917 learned_addr learned my addr 172.21.15.102:0/1419456917 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:22.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.114+0000 7fd7d5400640 1 --2- 172.21.15.102:0/1419456917 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7d80cd740 0x7fd7d80cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:22.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.114+0000 7fd7d5e00640 1 -- 172.21.15.102:0/1419456917 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7d80cd740 msgr2=0x7fd7d80cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:22.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.114+0000 7fd7d5e00640 1 --2- 172.21.15.102:0/1419456917 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7d80cd740 0x7fd7d80cc1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:22.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.114+0000 7fd7d5e00640 1 -- 172.21.15.102:0/1419456917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7d80d0d40 msgr2=0x7fd7d81593a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:22.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.114+0000 7fd7d6800640 1 --2- 172.21.15.102:0/1419456917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7d80d0d40 0x7fd7d81593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:22.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.114+0000 7fd7d5e00640 1 --2- 172.21.15.102:0/1419456917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7d80d0d40 0x7fd7d81593a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:22.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.114+0000 7fd7d5e00640 1 -- 172.21.15.102:0/1419456917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c8009b30 con 0x7fd7d806b0a0 2024-03-23T03:49:22.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.115+0000 7fd7d5e00640 1 --2- 172.21.15.102:0/1419456917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d806b0a0 0x7fd7d81f1cf0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd7c000c7f0 tx=0x7fd7c000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:22.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.115+0000 7fd7ce000640 1 -- 172.21.15.102:0/1419456917 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c0007a30 con 0x7fd7d806b0a0 2024-03-23T03:49:22.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.115+0000 7fd7ce000640 1 -- 172.21.15.102:0/1419456917 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c0007b90 con 0x7fd7d806b0a0 2024-03-23T03:49:22.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.115+0000 7fd7ce000640 1 -- 172.21.15.102:0/1419456917 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c000ce40 con 0x7fd7d806b0a0 2024-03-23T03:49:22.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.115+0000 7fd7dd000640 1 -- 172.21.15.102:0/1419456917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d8158fd0 con 0x7fd7d806b0a0 2024-03-23T03:49:22.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.115+0000 7fd7dd000640 1 -- 172.21.15.102:0/1419456917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d80cc6e0 con 0x7fd7d806b0a0 2024-03-23T03:49:22.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.116+0000 7fd7dd000640 1 -- 172.21.15.102:0/1419456917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7d80c9d20 con 0x7fd7d806b0a0 2024-03-23T03:49:22.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.117+0000 7fd7ce000640 1 -- 172.21.15.102:0/1419456917 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd7c00113f0 con 0x7fd7d806b0a0 2024-03-23T03:49:22.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.118+0000 7fd7ce000640 1 --2- 172.21.15.102:0/1419456917 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd7b406ac00 0x7fd7b406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:22.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.118+0000 7fd7ce000640 1 -- 172.21.15.102:0/1419456917 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fd7c0089390 con 0x7fd7d806b0a0 2024-03-23T03:49:22.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.118+0000 7fd7d5400640 1 --2- 172.21.15.102:0/1419456917 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd7b406ac00 0x7fd7b406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:22.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.119+0000 7fd7d5400640 1 --2- 172.21.15.102:0/1419456917 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd7b406ac00 0x7fd7b406d0c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8035000 tx=0x7fd7c8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:22.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.122+0000 7fd7ce000640 1 -- 172.21.15.102:0/1419456917 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd7c0054720 con 0x7fd7d806b0a0 2024-03-23T03:49:22.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.319+0000 7fd7dd000640 1 -- 172.21.15.102:0/1419456917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd7d8073390 con 0x7fd7d806b0a0 2024-03-23T03:49:22.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.325+0000 7fd7ce000640 1 -- 172.21.15.102:0/1419456917 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd7c0057d40 con 0x7fd7d806b0a0 2024-03-23T03:49:22.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.327+0000 7fd7dd000640 1 -- 172.21.15.102:0/1419456917 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd7b406ac00 msgr2=0x7fd7b406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:22.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.327+0000 7fd7dd000640 1 --2- 172.21.15.102:0/1419456917 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd7b406ac00 0x7fd7b406d0c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8035000 tx=0x7fd7c8030000 comp rx=0 tx=0).stop 2024-03-23T03:49:22.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.328+0000 7fd7dd000640 1 -- 172.21.15.102:0/1419456917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d806b0a0 msgr2=0x7fd7d81f1cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:22.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.328+0000 7fd7dd000640 1 --2- 172.21.15.102:0/1419456917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d806b0a0 0x7fd7d81f1cf0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd7c000c7f0 tx=0x7fd7c000cbf0 comp rx=0 tx=0).stop 2024-03-23T03:49:22.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.328+0000 7fd7dd000640 1 -- 172.21.15.102:0/1419456917 shutdown_connections 2024-03-23T03:49:22.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.328+0000 7fd7dd000640 1 --2- 172.21.15.102:0/1419456917 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd7b406ac00 0x7fd7b406d0c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:22.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.328+0000 7fd7dd000640 1 --2- 172.21.15.102:0/1419456917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7d80d0d40 0x7fd7d81593a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:22.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.328+0000 7fd7dd000640 1 --2- 172.21.15.102:0/1419456917 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd7d80cd740 0x7fd7d80cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:22.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.328+0000 7fd7dd000640 1 --2- 172.21.15.102:0/1419456917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7d806b0a0 0x7fd7d81f1cf0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:22.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.328+0000 7fd7dd000640 1 -- 172.21.15.102:0/1419456917 >> 172.21.15.102:0/1419456917 conn(0x7fd7d80cf5d0 msgr2=0x7fd7d815f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:22.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.328+0000 7fd7dd000640 1 -- 172.21.15.102:0/1419456917 shutdown_connections 2024-03-23T03:49:22.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:22.328+0000 7fd7dd000640 1 -- 172.21.15.102:0/1419456917 wait complete. 2024-03-23T03:49:22.345 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T03:49:22.346 DEBUG:teuthology.orchestra.run.smithi102:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:49:22.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:49:22.576 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T03:49:22.576 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:49:22.576 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:22.632 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:49:22.632 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:49:22.632 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:22.687 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:49:22.687 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:49:22.687 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:22.741 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:49:22.741 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:49:22.742 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:22.797 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:49:22.797 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-23T03:49:22.797 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:49:23.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.014+0000 7f2ff5200640 1 Processor -- start 2024-03-23T03:49:23.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.014+0000 7f2ff5200640 1 -- start start 2024-03-23T03:49:23.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.015+0000 7f2ff5200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff013faa0 0x7f2ff0160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:23.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.016+0000 7f2ff5200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f2ff0160800 0x7f2ff00b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:23.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.016+0000 7f2ff5200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ff00b2030 con 0x7f2ff00b1be0 2024-03-23T03:49:23.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.016+0000 7f2ff5200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ff00b38a0 con 0x7f2ff0160800 2024-03-23T03:49:23.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.016+0000 7f2ff5200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff00b3b80 con 0x7f2ff013faa0 2024-03-23T03:49:23.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.016+0000 7f2fede00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff013faa0 0x7f2ff0160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:23.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.016+0000 7f2fede00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff013faa0 0x7f2ff0160f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:33990/0 (socket says 172.21.15.102:33990) 2024-03-23T03:49:23.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.016+0000 7f2fede00640 1 -- 172.21.15.102:0/1864480009 learned_addr learned my addr 172.21.15.102:0/1864480009 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:23.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.017+0000 7f2fe5400640 1 --2- 172.21.15.102:0/1864480009 >> v2:172.21.15.173:3300/0 conn(0x7f2ff0160800 0x7f2ff00b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:23.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.017+0000 7f2fede00640 1 -- 172.21.15.102:0/1864480009 >> v2:172.21.15.173:3300/0 conn(0x7f2ff0160800 msgr2=0x7f2ff00b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:23.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.017+0000 7f2fede00640 1 --2- 172.21.15.102:0/1864480009 >> v2:172.21.15.173:3300/0 conn(0x7f2ff0160800 0x7f2ff00b16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:23.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.017+0000 7f2fede00640 1 -- 172.21.15.102:0/1864480009 >> v1:172.21.15.173:6789/0 conn(0x7f2ff00b1be0 legacy=0x7f2ff01422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:23.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.017+0000 7f2fede00640 1 -- 172.21.15.102:0/1864480009 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff00b3e60 con 0x7f2ff013faa0 2024-03-23T03:49:23.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.019+0000 7f2fede00640 1 --2- 172.21.15.102:0/1864480009 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff013faa0 0x7f2ff0160f30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009e50 tx=0x7f2fe002bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f225f9596dacb8b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:23.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.019+0000 7f2fed400640 1 -- 172.21.15.102:0/1864480009 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe002cb00 con 0x7f2ff013faa0 2024-03-23T03:49:23.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.019+0000 7f2fed400640 1 -- 172.21.15.102:0/1864480009 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe002cc60 con 0x7f2ff013faa0 2024-03-23T03:49:23.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.019+0000 7f2fed400640 1 -- 172.21.15.102:0/1864480009 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe00343f0 con 0x7f2ff013faa0 2024-03-23T03:49:23.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.020+0000 7f2ff5200640 1 -- 172.21.15.102:0/1864480009 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff013faa0 msgr2=0x7f2ff0160f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:23.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.020+0000 7f2ff5200640 1 --2- 172.21.15.102:0/1864480009 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff013faa0 0x7f2ff0160f30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0009e50 tx=0x7f2fe002bf60 comp rx=0 tx=0).stop 2024-03-23T03:49:23.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.020+0000 7f2ff5200640 1 -- 172.21.15.102:0/1864480009 shutdown_connections 2024-03-23T03:49:23.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.020+0000 7f2ff5200640 1 --2- 172.21.15.102:0/1864480009 >> v2:172.21.15.173:3300/0 conn(0x7f2ff0160800 0x7f2ff00b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:23.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.020+0000 7f2ff5200640 1 --2- 172.21.15.102:0/1864480009 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff013faa0 0x7f2ff0160f30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:23.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.020+0000 7f2ff5200640 1 -- 172.21.15.102:0/1864480009 >> 172.21.15.102:0/1864480009 conn(0x7f2ff006a3d0 msgr2=0x7f2ff00b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:23.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.020+0000 7f2ff5200640 1 -- 172.21.15.102:0/1864480009 shutdown_connections 2024-03-23T03:49:23.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.020+0000 7f2ff5200640 1 -- 172.21.15.102:0/1864480009 wait complete. 2024-03-23T03:49:23.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.022+0000 7f2ff5200640 1 Processor -- start 2024-03-23T03:49:23.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.022+0000 7f2ff5200640 1 -- start start 2024-03-23T03:49:23.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.023+0000 7f2ff5200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00b1be0 0x7f2ff01f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:23.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.023+0000 7f2ff5200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff013faa0 0x7f2ff01397f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:23.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.023+0000 7f2ff5200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ff0160800 0x7f2ff01417c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:23.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.023+0000 7f2ff5200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff00ca890 con 0x7f2ff00b1be0 2024-03-23T03:49:23.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.023+0000 7f2ff5200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff00cb130 con 0x7f2ff0160800 2024-03-23T03:49:23.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.023+0000 7f2ff5200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff01f6ac0 con 0x7f2ff013faa0 2024-03-23T03:49:23.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.024+0000 7f2fede00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00b1be0 0x7f2ff01f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:23.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.024+0000 7f2fede00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00b1be0 0x7f2ff01f2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52070/0 (socket says 172.21.15.102:52070) 2024-03-23T03:49:23.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.024+0000 7f2fede00640 1 -- 172.21.15.102:0/3516303035 learned_addr learned my addr 172.21.15.102:0/3516303035 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:23.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.024+0000 7f2fee800640 1 --2- 172.21.15.102:0/3516303035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ff0160800 0x7f2ff01417c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:23.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.024+0000 7f2fe5400640 1 --2- 172.21.15.102:0/3516303035 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff013faa0 0x7f2ff01397f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:23.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.024+0000 7f2fede00640 1 -- 172.21.15.102:0/3516303035 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff013faa0 msgr2=0x7f2ff01397f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:23.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.024+0000 7f2fede00640 1 --2- 172.21.15.102:0/3516303035 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff013faa0 0x7f2ff01397f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:23.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.024+0000 7f2fede00640 1 -- 172.21.15.102:0/3516303035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ff0160800 msgr2=0x7f2ff01417c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:23.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.024+0000 7f2fede00640 1 --2- 172.21.15.102:0/3516303035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ff0160800 0x7f2ff01417c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:23.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.024+0000 7f2fede00640 1 -- 172.21.15.102:0/3516303035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe0009b30 con 0x7f2ff00b1be0 2024-03-23T03:49:23.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.024+0000 7f2fede00640 1 --2- 172.21.15.102:0/3516303035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00b1be0 0x7f2ff01f2770 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0036040 tx=0x7f2fe0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:23.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.025+0000 7f2fe6a00640 1 -- 172.21.15.102:0/3516303035 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0009510 con 0x7f2ff00b1be0 2024-03-23T03:49:23.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.025+0000 7f2fe6a00640 1 -- 172.21.15.102:0/3516303035 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe003e450 con 0x7f2ff00b1be0 2024-03-23T03:49:23.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.025+0000 7f2fe6a00640 1 -- 172.21.15.102:0/3516303035 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe003e800 con 0x7f2ff00b1be0 2024-03-23T03:49:23.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.025+0000 7f2ff5200640 1 -- 172.21.15.102:0/3516303035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ff00caae0 con 0x7f2ff00b1be0 2024-03-23T03:49:23.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.025+0000 7f2ff5200640 1 -- 172.21.15.102:0/3516303035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ff01f2cb0 con 0x7f2ff00b1be0 2024-03-23T03:49:23.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.026+0000 7f2ff5200640 1 -- 172.21.15.102:0/3516303035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff0073390 con 0x7f2ff00b1be0 2024-03-23T03:49:23.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.027+0000 7f2fe6a00640 1 -- 172.21.15.102:0/3516303035 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2fe003e960 con 0x7f2ff00b1be0 2024-03-23T03:49:23.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.028+0000 7f2fe6a00640 1 --2- 172.21.15.102:0/3516303035 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fcc06ac00 0x7f2fcc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:23.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.028+0000 7f2fe6a00640 1 -- 172.21.15.102:0/3516303035 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f2fe00abef0 con 0x7f2ff00b1be0 2024-03-23T03:49:23.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.028+0000 7f2fe5400640 1 --2- 172.21.15.102:0/3516303035 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fcc06ac00 0x7f2fcc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:23.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.029+0000 7f2fe5400640 1 --2- 172.21.15.102:0/3516303035 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fcc06ac00 0x7f2fcc06d0c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00c3730 tx=0x7f2fd8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:23.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.033+0000 7f2fe6a00640 1 -- 172.21.15.102:0/3516303035 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2fe00772a0 con 0x7f2ff00b1be0 2024-03-23T03:49:23.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.261+0000 7f2ff5200640 1 -- 172.21.15.102:0/3516303035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ff00ae310 con 0x7f2ff00b1be0 2024-03-23T03:49:23.263 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.262+0000 7f2fe6a00640 1 -- 172.21.15.102:0/3516303035 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f2fe0043de0 con 0x7f2ff00b1be0 2024-03-23T03:49:23.263 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:23.264 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":26,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4901,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2569463916","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6833","nonce":2569463916},{"type":"v1","addr":"172.21.15.173:6835","nonce":2569463916}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26},{"gid":5661,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/1848167718","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":1848167718},{"type":"v1","addr":"172.21.15.102:6835","nonce":1848167718}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5664,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/16304019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":16304019},{"type":"v1","addr":"172.21.15.102:6837","nonce":16304019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5673,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6834/2030662139","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2030662139},{"type":"v1","addr":"172.21.15.173:6834","nonce":2030662139}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[]} 2024-03-23T03:49:23.264 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 26 2024-03-23T03:49:23.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.265+0000 7f2ff5200640 1 -- 172.21.15.102:0/3516303035 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fcc06ac00 msgr2=0x7f2fcc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:23.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.265+0000 7f2ff5200640 1 --2- 172.21.15.102:0/3516303035 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fcc06ac00 0x7f2fcc06d0c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00c3730 tx=0x7f2fd8008040 comp rx=0 tx=0).stop 2024-03-23T03:49:23.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.265+0000 7f2ff5200640 1 -- 172.21.15.102:0/3516303035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00b1be0 msgr2=0x7f2ff01f2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:23.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.265+0000 7f2ff5200640 1 --2- 172.21.15.102:0/3516303035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00b1be0 0x7f2ff01f2770 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f2fe0036040 tx=0x7f2fe0004a40 comp rx=0 tx=0).stop 2024-03-23T03:49:23.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.265+0000 7f2ff5200640 1 -- 172.21.15.102:0/3516303035 shutdown_connections 2024-03-23T03:49:23.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.265+0000 7f2ff5200640 1 --2- 172.21.15.102:0/3516303035 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2fcc06ac00 0x7f2fcc06d0c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:23.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.265+0000 7f2ff5200640 1 --2- 172.21.15.102:0/3516303035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2ff0160800 0x7f2ff01417c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:23.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.265+0000 7f2ff5200640 1 --2- 172.21.15.102:0/3516303035 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2ff013faa0 0x7f2ff01397f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:23.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.265+0000 7f2ff5200640 1 --2- 172.21.15.102:0/3516303035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ff00b1be0 0x7f2ff01f2770 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:23.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.265+0000 7f2ff5200640 1 -- 172.21.15.102:0/3516303035 >> 172.21.15.102:0/3516303035 conn(0x7f2ff006a3d0 msgr2=0x7f2ff00ae8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:23.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.266+0000 7f2ff5200640 1 -- 172.21.15.102:0/3516303035 shutdown_connections 2024-03-23T03:49:23.267 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:23.266+0000 7f2ff5200640 1 -- 172.21.15.102:0/3516303035 wait complete. 2024-03-23T03:49:23.284 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T03:49:23.284 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-23T03:49:23.284 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T03:49:23.284 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:49:23.284 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T03:49:23.284 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-23T03:49:23.285 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T03:49:23.285 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:49:23.285 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T03:49:23.285 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-23T03:49:23.285 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T03:49:23.285 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:49:23.285 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T03:49:23.286 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-23T03:49:23.286 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T03:49:23.286 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:49:23.303 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T03:49:23.304 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T03:49:23.304 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T03:49:23.304 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T03:49:23.308 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T03:49:23.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T03:49:23.350 INFO:tasks.ceph.mds.c:Started 2024-03-23T03:49:23.351 DEBUG:teuthology.parallel:result is None 2024-03-23T03:49:23.355 INFO:tasks.ceph.mds.d:Started 2024-03-23T03:49:23.355 DEBUG:teuthology.parallel:result is None 2024-03-23T03:49:23.356 INFO:tasks.ceph.mds.a:Started 2024-03-23T03:49:23.356 DEBUG:teuthology.parallel:result is None 2024-03-23T03:49:23.501 INFO:tasks.ceph.mds.a.smithi102.stdout:starting mds.a at 2024-03-23T03:49:23.503 INFO:tasks.ceph.mds.c.smithi102.stdout:starting mds.c at 2024-03-23T03:49:23.534 INFO:tasks.ceph.mds.d.smithi173.stdout:starting mds.d at 2024-03-23T03:49:25.885 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-03-23T03:49:29.308 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T03:49:29.308 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T03:49:29.316 INFO:tasks.ceph.mds.b:Started 2024-03-23T03:49:29.317 DEBUG:teuthology.parallel:result is None 2024-03-23T03:49:29.317 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-23T03:49:29.499 INFO:tasks.ceph.mds.b.smithi173.stdout:starting mds.b at 2024-03-23T03:49:29.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.536+0000 7f3b41600640 1 Processor -- start 2024-03-23T03:49:29.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.536+0000 7f3b41600640 1 -- start start 2024-03-23T03:49:29.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.538+0000 7f3b41600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f3b3c0b4b20 0x7f3b3c0d0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:29.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.538+0000 7f3b41600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b3c0b8470 0x7f3b3c0b5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:29.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.538+0000 7f3b41600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3c0d1460 con 0x7f3b3c15cba0 2024-03-23T03:49:29.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.538+0000 7f3b41600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b3c15cf80 con 0x7f3b3c0b4b20 2024-03-23T03:49:29.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.538+0000 7f3b41600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c0d1740 con 0x7f3b3c0b8470 2024-03-23T03:49:29.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.538+0000 7f3b3aa00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3b3c15cba0 0x7f3b3c0b4410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33636/0 (socket says 172.21.15.102:33636) 2024-03-23T03:49:29.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.538+0000 7f3b3aa00640 1 -- 172.21.15.102:0/3705934099 learned_addr learned my addr 172.21.15.102:0/3705934099 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:29.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.538+0000 7f3b3a000640 1 --2- 172.21.15.102:0/3705934099 >> v2:172.21.15.173:3300/0 conn(0x7f3b3c0b4b20 0x7f3b3c0d0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:29.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.538+0000 7f3b3b400640 1 --2- 172.21.15.102:0/3705934099 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b3c0b8470 0x7f3b3c0b5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:29.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.538+0000 7f3b39600640 1 -- 172.21.15.102:0/3705934099 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650904297 0 0) 0x7f3b3c0d1460 con 0x7f3b3c15cba0 2024-03-23T03:49:29.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.538+0000 7f3b39600640 1 -- 172.21.15.102:0/3705934099 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b20003520 con 0x7f3b3c15cba0 2024-03-23T03:49:29.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.539+0000 7f3b39600640 1 -- 172.21.15.102:0/3705934099 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2600467020 0 0) 0x7f3b20003520 con 0x7f3b3c15cba0 2024-03-23T03:49:29.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.539+0000 7f3b39600640 1 -- 172.21.15.102:0/3705934099 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b3c0b8470 msgr2=0x7f3b3c0b5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:29.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.539+0000 7f3b39600640 1 --2- 172.21.15.102:0/3705934099 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b3c0b8470 0x7f3b3c0b5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:29.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.539+0000 7f3b39600640 1 -- 172.21.15.102:0/3705934099 >> v2:172.21.15.173:3300/0 conn(0x7f3b3c0b4b20 msgr2=0x7f3b3c0d0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:29.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.539+0000 7f3b39600640 1 --2- 172.21.15.102:0/3705934099 >> v2:172.21.15.173:3300/0 conn(0x7f3b3c0b4b20 0x7f3b3c0d0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:29.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.539+0000 7f3b39600640 1 -- 172.21.15.102:0/3705934099 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3c0d1a20 con 0x7f3b3c15cba0 2024-03-23T03:49:29.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.539+0000 7f3b39600640 1 -- 172.21.15.102:0/3705934099 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f3b2c001690 con 0x7f3b3c15cba0 2024-03-23T03:49:29.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.539+0000 7f3b39600640 1 -- 172.21.15.102:0/3705934099 >> v1:172.21.15.102:6789/0 conn(0x7f3b3c15cba0 legacy=0x7f3b3c0b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:29.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.539+0000 7f3b39600640 1 --2- 172.21.15.102:0/3705934099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b20003970 0x7f3b20003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:29.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.539+0000 7f3b39600640 1 --2- 172.21.15.102:0/3705934099 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b3c0b8470 0x7f3b200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:29.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.539+0000 7f3b39600640 1 --2- 172.21.15.102:0/3705934099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20004d00 0x7f3b20005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:29.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.539+0000 7f3b39600640 1 -- 172.21.15.102:0/3705934099 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c15cf80 con 0x7f3b20004d00 2024-03-23T03:49:29.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.539+0000 7f3b39600640 1 -- 172.21.15.102:0/3705934099 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c0d1740 con 0x7f3b20003970 2024-03-23T03:49:29.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.539+0000 7f3b39600640 1 -- 172.21.15.102:0/3705934099 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2000b980 con 0x7f3b3c0b8470 2024-03-23T03:49:29.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.539+0000 7f3b3b400640 1 --2- 172.21.15.102:0/3705934099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20004d00 0x7f3b20005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:29.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.539+0000 7f3b3aa00640 1 --2- 172.21.15.102:0/3705934099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b20003970 0x7f3b20003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:29.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.539+0000 7f3b3a000640 1 --2- 172.21.15.102:0/3705934099 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b3c0b8470 0x7f3b200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:29.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.540+0000 7f3b3b400640 1 -- 172.21.15.102:0/3705934099 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b3c0b8470 msgr2=0x7f3b200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:29.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.540+0000 7f3b3b400640 1 --2- 172.21.15.102:0/3705934099 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b3c0b8470 0x7f3b200047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:29.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.540+0000 7f3b3b400640 1 -- 172.21.15.102:0/3705934099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b20003970 msgr2=0x7f3b20003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:29.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.540+0000 7f3b3b400640 1 --2- 172.21.15.102:0/3705934099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b20003970 0x7f3b20003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:29.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.540+0000 7f3b3b400640 1 -- 172.21.15.102:0/3705934099 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b20003520 con 0x7f3b20004d00 2024-03-23T03:49:29.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.541+0000 7f3b3b400640 1 --2- 172.21.15.102:0/3705934099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20004d00 0x7f3b20005180 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400e170 tx=0x7f3b200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91d1c737ff979f4e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:29.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.541+0000 7f3b39600640 1 -- 172.21.15.102:0/3705934099 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3402cbf0 con 0x7f3b20004d00 2024-03-23T03:49:29.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.541+0000 7f3b39600640 1 -- 172.21.15.102:0/3705934099 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3400f040 con 0x7f3b20004d00 2024-03-23T03:49:29.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.541+0000 7f3b39600640 1 -- 172.21.15.102:0/3705934099 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b34034440 con 0x7f3b20004d00 2024-03-23T03:49:29.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.541+0000 7f3b41600640 1 -- 172.21.15.102:0/3705934099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20004d00 msgr2=0x7f3b20005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:29.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.542+0000 7f3b41600640 1 --2- 172.21.15.102:0/3705934099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b20004d00 0x7f3b20005180 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f3b3400e170 tx=0x7f3b200057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:29.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.542+0000 7f3b3b400640 1 -- 172.21.15.102:0/3705934099 reap_dead start 2024-03-23T03:49:29.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.542+0000 7f3b41600640 1 -- 172.21.15.102:0/3705934099 shutdown_connections 2024-03-23T03:49:29.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.542+0000 7f3b41600640 1 -- 172.21.15.102:0/3705934099 >> 172.21.15.102:0/3705934099 conn(0x7f3b3c0cf5f0 msgr2=0x7f3b3c0cfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:29.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.542+0000 7f3b41600640 1 -- 172.21.15.102:0/3705934099 shutdown_connections 2024-03-23T03:49:29.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.542+0000 7f3b41600640 1 -- 172.21.15.102:0/3705934099 wait complete. 2024-03-23T03:49:29.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.544+0000 7f3b41600640 1 Processor -- start 2024-03-23T03:49:29.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.544+0000 7f3b41600640 1 -- start start 2024-03-23T03:49:29.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.545+0000 7f3b41600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b20004d00 0x7f3b3c1e5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:29.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.545+0000 7f3b41600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b3c1e7460 0x7f3b3c1e1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:29.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.545+0000 7f3b41600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b3c1e2310 0x7f3b3c1f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:29.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.545+0000 7f3b41600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c1e27c0 con 0x7f3b3c1e7460 2024-03-23T03:49:29.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.545+0000 7f3b41600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c0cad10 con 0x7f3b3c1e2310 2024-03-23T03:49:29.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.545+0000 7f3b41600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c0cb570 con 0x7f3b20004d00 2024-03-23T03:49:29.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.545+0000 7f3b3a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b3c1e7460 0x7f3b3c1e1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:29.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.545+0000 7f3b3a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b3c1e7460 0x7f3b3c1e1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52130/0 (socket says 172.21.15.102:52130) 2024-03-23T03:49:29.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.545+0000 7f3b3a000640 1 -- 172.21.15.102:0/804926763 learned_addr learned my addr 172.21.15.102:0/804926763 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:29.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.545+0000 7f3b3aa00640 1 --2- 172.21.15.102:0/804926763 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b20004d00 0x7f3b3c1e5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:29.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.545+0000 7f3b3b400640 1 --2- 172.21.15.102:0/804926763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b3c1e2310 0x7f3b3c1f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:29.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.545+0000 7f3b3a000640 1 -- 172.21.15.102:0/804926763 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b20004d00 msgr2=0x7f3b3c1e5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:29.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.545+0000 7f3b3a000640 1 --2- 172.21.15.102:0/804926763 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b20004d00 0x7f3b3c1e5380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:29.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.545+0000 7f3b3a000640 1 -- 172.21.15.102:0/804926763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b3c1e2310 msgr2=0x7f3b3c1f5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:29.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.545+0000 7f3b3a000640 1 --2- 172.21.15.102:0/804926763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b3c1e2310 0x7f3b3c1f5bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:29.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.545+0000 7f3b3a000640 1 -- 172.21.15.102:0/804926763 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3c1f68b0 con 0x7f3b3c1e7460 2024-03-23T03:49:29.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.545+0000 7f3b3aa00640 1 --2- 172.21.15.102:0/804926763 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b20004d00 0x7f3b3c1e5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:29.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.546+0000 7f3b3a000640 1 --2- 172.21.15.102:0/804926763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b3c1e7460 0x7f3b3c1e1dd0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f3b28011aa0 tx=0x7f3b28011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:29.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.546+0000 7f3b32a00640 1 -- 172.21.15.102:0/804926763 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28009680 con 0x7f3b3c1e7460 2024-03-23T03:49:29.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.546+0000 7f3b32a00640 1 -- 172.21.15.102:0/804926763 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2800fb80 con 0x7f3b3c1e7460 2024-03-23T03:49:29.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.546+0000 7f3b41600640 1 -- 172.21.15.102:0/804926763 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b3c0caf20 con 0x7f3b3c1e7460 2024-03-23T03:49:29.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.546+0000 7f3b32a00640 1 -- 172.21.15.102:0/804926763 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28014440 con 0x7f3b3c1e7460 2024-03-23T03:49:29.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.546+0000 7f3b41600640 1 -- 172.21.15.102:0/804926763 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3c1e58c0 con 0x7f3b3c1e7460 2024-03-23T03:49:29.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.547+0000 7f3b41600640 1 -- 172.21.15.102:0/804926763 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3c073390 con 0x7f3b3c1e7460 2024-03-23T03:49:29.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.548+0000 7f3b32a00640 1 -- 172.21.15.102:0/804926763 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3b280145a0 con 0x7f3b3c1e7460 2024-03-23T03:49:29.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.549+0000 7f3b32a00640 1 --2- 172.21.15.102:0/804926763 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b1c06ac30 0x7f3b1c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:29.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.550+0000 7f3b32a00640 1 -- 172.21.15.102:0/804926763 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f3b28089b00 con 0x7f3b3c1e7460 2024-03-23T03:49:29.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.550+0000 7f3b3aa00640 1 --2- 172.21.15.102:0/804926763 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b1c06ac30 0x7f3b1c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:29.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.550+0000 7f3b3aa00640 1 --2- 172.21.15.102:0/804926763 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b1c06ac30 0x7f3b1c06d0f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0017c0 tx=0x7f3b2c000ed0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:29.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.554+0000 7f3b32a00640 1 -- 172.21.15.102:0/804926763 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3b28054e00 con 0x7f3b3c1e7460 2024-03-23T03:49:29.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:29.765+0000 7f3b41600640 1 -- 172.21.15.102:0/804926763 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3b3c0cb6b0 con 0x7f3b3c1e7460 2024-03-23T03:49:30.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.692+0000 7f3b32a00640 1 -- 172.21.15.102:0/804926763 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v63) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3b28058420 con 0x7f3b3c1e7460 2024-03-23T03:49:30.694 INFO:teuthology.orchestra.run.smithi102.stderr: removed all blocklist entries 2024-03-23T03:49:30.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.696+0000 7f3b41600640 1 -- 172.21.15.102:0/804926763 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b1c06ac30 msgr2=0x7f3b1c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:30.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.696+0000 7f3b41600640 1 --2- 172.21.15.102:0/804926763 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b1c06ac30 0x7f3b1c06d0f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3b2c0017c0 tx=0x7f3b2c000ed0 comp rx=0 tx=0).stop 2024-03-23T03:49:30.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.696+0000 7f3b41600640 1 -- 172.21.15.102:0/804926763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b3c1e7460 msgr2=0x7f3b3c1e1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:30.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.696+0000 7f3b41600640 1 --2- 172.21.15.102:0/804926763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b3c1e7460 0x7f3b3c1e1dd0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f3b28011aa0 tx=0x7f3b28011f60 comp rx=0 tx=0).stop 2024-03-23T03:49:30.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.697+0000 7f3b41600640 1 -- 172.21.15.102:0/804926763 shutdown_connections 2024-03-23T03:49:30.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.697+0000 7f3b41600640 1 --2- 172.21.15.102:0/804926763 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b1c06ac30 0x7f3b1c06d0f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:30.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.697+0000 7f3b41600640 1 --2- 172.21.15.102:0/804926763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b3c1e2310 0x7f3b3c1f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:30.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.697+0000 7f3b41600640 1 --2- 172.21.15.102:0/804926763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b3c1e7460 0x7f3b3c1e1dd0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:30.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.697+0000 7f3b41600640 1 --2- 172.21.15.102:0/804926763 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b20004d00 0x7f3b3c1e5380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:30.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.697+0000 7f3b41600640 1 -- 172.21.15.102:0/804926763 >> 172.21.15.102:0/804926763 conn(0x7f3b3c0cf5f0 msgr2=0x7f3b3c0b2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:30.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.697+0000 7f3b41600640 1 -- 172.21.15.102:0/804926763 shutdown_connections 2024-03-23T03:49:30.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.697+0000 7f3b41600640 1 -- 172.21.15.102:0/804926763 wait complete. 2024-03-23T03:49:30.718 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-23T03:49:30.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.929+0000 7f1305200640 1 Processor -- start 2024-03-23T03:49:30.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.930+0000 7f1305200640 1 -- start start 2024-03-23T03:49:30.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.931+0000 7f1305200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f130015cda0 0x7f13000cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:30.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.931+0000 7f1305200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f13000b4a60 0x7f13000b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:30.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.931+0000 7f1305200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13000d1830 con 0x7f13000b5a30 2024-03-23T03:49:30.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.931+0000 7f1305200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13000b1f20 con 0x7f13000b4a60 2024-03-23T03:49:30.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.931+0000 7f1305200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13000d0c20 con 0x7f130015cda0 2024-03-23T03:49:30.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.932+0000 7f12fde00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f130015cda0 0x7f13000cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:30.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.932+0000 7f12fe800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f13000b4a60 0x7f13000b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:30.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.932+0000 7f12fde00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f130015cda0 0x7f13000cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:34104/0 (socket says 172.21.15.102:34104) 2024-03-23T03:49:30.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.932+0000 7f12fe800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f13000b4a60 0x7f13000b5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:47754/0 (socket says 172.21.15.102:47754) 2024-03-23T03:49:30.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.932+0000 7f12fde00640 1 -- 172.21.15.102:0/4145137311 learned_addr learned my addr 172.21.15.102:0/4145137311 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.933+0000 7f12fe800640 1 -- 172.21.15.102:0/4145137311 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f130015cda0 msgr2=0x7f13000cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.933+0000 7f12fe800640 1 --2- 172.21.15.102:0/4145137311 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f130015cda0 0x7f13000cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.933+0000 7f12fe800640 1 -- 172.21.15.102:0/4145137311 >> v1:172.21.15.173:6789/0 conn(0x7f13000b5a30 legacy=0x7f13000b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.933+0000 7f12fe800640 1 -- 172.21.15.102:0/4145137311 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13000d0f00 con 0x7f13000b4a60 2024-03-23T03:49:30.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.933+0000 7f12fde00640 1 --2- 172.21.15.102:0/4145137311 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f130015cda0 0x7f13000cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:49:30.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.934+0000 7f12fe800640 1 --2- 172.21.15.102:0/4145137311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13000b4a60 0x7f13000b5e90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f12f8009d90 tx=0x7f12f802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=349c29619e5193f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:30.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.934+0000 7f12fd400640 1 -- 172.21.15.102:0/4145137311 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f802cc50 con 0x7f13000b4a60 2024-03-23T03:49:30.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.934+0000 7f12fd400640 1 -- 172.21.15.102:0/4145137311 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f802cdb0 con 0x7f13000b4a60 2024-03-23T03:49:30.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.934+0000 7f12fd400640 1 -- 172.21.15.102:0/4145137311 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f8034660 con 0x7f13000b4a60 2024-03-23T03:49:30.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.935+0000 7f1305200640 1 -- 172.21.15.102:0/4145137311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13000b4a60 msgr2=0x7f13000b5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:30.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.935+0000 7f1305200640 1 --2- 172.21.15.102:0/4145137311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13000b4a60 0x7f13000b5e90 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f12f8009d90 tx=0x7f12f802bea0 comp rx=0 tx=0).stop 2024-03-23T03:49:30.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.935+0000 7f1305200640 1 -- 172.21.15.102:0/4145137311 shutdown_connections 2024-03-23T03:49:30.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.935+0000 7f1305200640 1 --2- 172.21.15.102:0/4145137311 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13000b4a60 0x7f13000b5e90 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:30.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.935+0000 7f1305200640 1 --2- 172.21.15.102:0/4145137311 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f130015cda0 0x7f13000cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:30.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.935+0000 7f1305200640 1 -- 172.21.15.102:0/4145137311 >> 172.21.15.102:0/4145137311 conn(0x7f13000cf350 msgr2=0x7f13000cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:30.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.935+0000 7f1305200640 1 -- 172.21.15.102:0/4145137311 shutdown_connections 2024-03-23T03:49:30.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.935+0000 7f1305200640 1 -- 172.21.15.102:0/4145137311 wait complete. 2024-03-23T03:49:30.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.937+0000 7f1305200640 1 Processor -- start 2024-03-23T03:49:30.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.937+0000 7f1305200640 1 -- start start 2024-03-23T03:49:30.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.938+0000 7f1305200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13000b4a60 0x7f13000c6b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:30.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.938+0000 7f1305200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13000b5a30 0x7f13000ca9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:30.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.938+0000 7f1305200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f130015cda0 0x7f13000caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:30.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.938+0000 7f1305200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130014a570 con 0x7f130015cda0 2024-03-23T03:49:30.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.938+0000 7f1305200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130014a850 con 0x7f13000b5a30 2024-03-23T03:49:30.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.938+0000 7f1305200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130014ab30 con 0x7f13000b4a60 2024-03-23T03:49:30.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.938+0000 7f12fe800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f130015cda0 0x7f13000caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:30.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.938+0000 7f12fe800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f130015cda0 0x7f13000caf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52136/0 (socket says 172.21.15.102:52136) 2024-03-23T03:49:30.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.938+0000 7f12fe800640 1 -- 172.21.15.102:0/2906715300 learned_addr learned my addr 172.21.15.102:0/2906715300 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:30.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.939+0000 7f12fde00640 1 --2- 172.21.15.102:0/2906715300 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13000b4a60 0x7f13000c6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:30.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.939+0000 7f12f5400640 1 --2- 172.21.15.102:0/2906715300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13000b5a30 0x7f13000ca9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:30.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.939+0000 7f12fe800640 1 -- 172.21.15.102:0/2906715300 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13000b4a60 msgr2=0x7f13000c6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:30.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.939+0000 7f12fe800640 1 --2- 172.21.15.102:0/2906715300 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13000b4a60 0x7f13000c6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:30.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.939+0000 7f12fe800640 1 -- 172.21.15.102:0/2906715300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13000b5a30 msgr2=0x7f13000ca9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:30.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.939+0000 7f12fe800640 1 --2- 172.21.15.102:0/2906715300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13000b5a30 0x7f13000ca9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:30.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.939+0000 7f12fe800640 1 -- 172.21.15.102:0/2906715300 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f00094d0 con 0x7f130015cda0 2024-03-23T03:49:30.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.939+0000 7f12fe800640 1 --2- 172.21.15.102:0/2906715300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f130015cda0 0x7f13000caf30 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f12f8009eb0 tx=0x7f12f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:30.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.939+0000 7f12f6a00640 1 -- 172.21.15.102:0/2906715300 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f803e610 con 0x7f130015cda0 2024-03-23T03:49:30.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.939+0000 7f12f6a00640 1 -- 172.21.15.102:0/2906715300 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f803e770 con 0x7f130015cda0 2024-03-23T03:49:30.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.939+0000 7f1305200640 1 -- 172.21.15.102:0/2906715300 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f8009a70 con 0x7f130015cda0 2024-03-23T03:49:30.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.939+0000 7f12f6a00640 1 -- 172.21.15.102:0/2906715300 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f803eb60 con 0x7f130015cda0 2024-03-23T03:49:30.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.940+0000 7f1305200640 1 -- 172.21.15.102:0/2906715300 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130014b0f0 con 0x7f130015cda0 2024-03-23T03:49:30.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.941+0000 7f1305200640 1 -- 172.21.15.102:0/2906715300 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1300073390 con 0x7f130015cda0 2024-03-23T03:49:30.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.942+0000 7f12f6a00640 1 -- 172.21.15.102:0/2906715300 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f12f803b050 con 0x7f130015cda0 2024-03-23T03:49:30.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.942+0000 7f12f6a00640 1 --2- 172.21.15.102:0/2906715300 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f12dc06ac00 0x7f12dc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:30.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.943+0000 7f12f6a00640 1 -- 172.21.15.102:0/2906715300 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f12f80ab390 con 0x7f130015cda0 2024-03-23T03:49:30.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.943+0000 7f12fde00640 1 --2- 172.21.15.102:0/2906715300 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f12dc06ac00 0x7f12dc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:30.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.944+0000 7f12fde00640 1 --2- 172.21.15.102:0/2906715300 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f12dc06ac00 0x7f12dc06d0c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f12f0009c50 tx=0x7f12f0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:30.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:30.948+0000 7f12f6a00640 1 -- 172.21.15.102:0/2906715300 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f12f80768a0 con 0x7f130015cda0 2024-03-23T03:49:31.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.211+0000 7f1305200640 1 -- 172.21.15.102:0/2906715300 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f130014b2b0 con 0x7f130015cda0 2024-03-23T03:49:31.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.213+0000 7f12f6a00640 1 -- 172.21.15.102:0/2906715300 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f12f8009450 con 0x7f130015cda0 2024-03-23T03:49:31.214 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:31.214 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:49:31.214 INFO:teuthology.orchestra.run.smithi102.stdout: "auth_dump": [ 2024-03-23T03:49:31.214 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.214 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.a", 2024-03-23T03:49:31.214 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5loBTZFRAA7/tTYyZwrDmwKwiU+dtCEw==", 2024-03-23T03:49:31.214 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.214 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:49:31.214 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:49:31.214 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:49:31.214 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:49:31.214 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.215 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.215 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.215 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.b", 2024-03-23T03:49:31.215 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5lJKKlNxAAj5+j99K79IhbeEm7/qC+wg==", 2024-03-23T03:49:31.215 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.215 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:49:31.215 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:49:31.215 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:49:31.215 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:49:31.215 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.215 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.215 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.215 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.c", 2024-03-23T03:49:31.215 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5l0b92JhAA5rvs+mdgmrQX/9lI3GW2rg==", 2024-03-23T03:49:31.215 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.215 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:49:31.216 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:49:31.216 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:49:31.216 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:49:31.216 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.216 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.216 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.216 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.d", 2024-03-23T03:49:31.216 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5ltjNMDBAAD1h5Nlv2Cp7mJt+DMlBtzw==", 2024-03-23T03:49:31.216 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.216 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:49:31.216 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:49:31.216 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:49:31.216 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:49:31.217 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.217 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.217 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.217 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.0", 2024-03-23T03:49:31.217 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB2UP5lPW6uORAAejiQReOF95oFliFmk/vdQQ==", 2024-03-23T03:49:31.217 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.217 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:49:31.217 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:49:31.217 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:49:31.217 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.217 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.217 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.217 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.1", 2024-03-23T03:49:31.217 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB6UP5lFbbUBxAAIl0ZEYtJ+/oi3W6a4OwMhg==", 2024-03-23T03:49:31.217 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.217 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:49:31.218 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:49:31.218 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:49:31.218 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.218 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.218 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.218 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.2", 2024-03-23T03:49:31.218 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB9UP5l2LaQIBAAKJv7jWJAjlUJrm7+KTXi6Q==", 2024-03-23T03:49:31.218 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.218 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:49:31.218 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:49:31.218 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:49:31.218 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.218 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.218 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.219 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.3", 2024-03-23T03:49:31.219 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCAUP5lxjWsKhAAzVO8CXUd+4LLvv4f/5xMOg==", 2024-03-23T03:49:31.219 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.219 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:49:31.219 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:49:31.219 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:49:31.219 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.219 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.219 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.219 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.4", 2024-03-23T03:49:31.219 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCHUP5lHcd2JRAAwID1TPb+Z9ViH51B0Mv/kQ==", 2024-03-23T03:49:31.219 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.219 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:49:31.219 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:49:31.219 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:49:31.219 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.220 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.220 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.220 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.5", 2024-03-23T03:49:31.220 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCLUP5lApp3AxAADslbuzTe/bf5FML1ckE05w==", 2024-03-23T03:49:31.220 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.220 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:49:31.220 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:49:31.220 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:49:31.220 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.220 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.220 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.220 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.6", 2024-03-23T03:49:31.220 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCOUP5lsi7VGxAArhWvXQ1VzJ8oSiOXzq8VsQ==", 2024-03-23T03:49:31.220 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.220 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:49:31.221 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:49:31.221 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:49:31.221 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.221 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.221 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.221 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.7", 2024-03-23T03:49:31.221 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCRUP5lu182JxAA5LaDXfrBkOXEUNAJXFDCJQ==", 2024-03-23T03:49:31.221 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.221 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:49:31.221 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:49:31.221 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:49:31.221 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.221 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.221 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.221 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.0", 2024-03-23T03:49:31.221 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lxTfoGRAA3WIQ8p1rGSAyJkLZJMk3dA==", 2024-03-23T03:49:31.222 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.222 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:49:31.222 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow r", 2024-03-23T03:49:31.222 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-23T03:49:31.222 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.222 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.222 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.222 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.1", 2024-03-23T03:49:31.222 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lneSHLhAAwn6ULOajHXrOM650HUcLbw==", 2024-03-23T03:49:31.222 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.222 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:49:31.222 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:49:31.222 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:49:31.222 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:49:31.222 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.223 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.223 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.223 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.2", 2024-03-23T03:49:31.223 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lOIklJBAAhx2d0Xy/j1PG0qsvC3nMqA==", 2024-03-23T03:49:31.223 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.223 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:49:31.223 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:49:31.223 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:49:31.223 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:49:31.223 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.223 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.223 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.223 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.3", 2024-03-23T03:49:31.223 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5l+7BVORAA7PBYS6mkgLA8b4sLCAYgBA==", 2024-03-23T03:49:31.223 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.223 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:49:31.224 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:49:31.224 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:49:31.224 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:49:31.224 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.224 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.224 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.224 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.admin", 2024-03-23T03:49:31.224 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBwUP5ldlDgBxAAOiqmhC5GulxELun8OiR4lg==", 2024-03-23T03:49:31.224 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.224 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:49:31.224 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:49:31.224 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:49:31.224 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:49:31.224 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.225 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.225 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.225 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mgr.x", 2024-03-23T03:49:31.225 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5lQXojChAAzepTDh0Pk3fuBZYsFt/+DQ==", 2024-03-23T03:49:31.225 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.225 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:49:31.225 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile mgr", 2024-03-23T03:49:31.225 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:49:31.225 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.225 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:49:31.225 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:31.225 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mgr.y", 2024-03-23T03:49:31.225 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBwUP5lK59iORAACuBF0JaQ8epyu3Mq3slzjg==", 2024-03-23T03:49:31.225 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:49:31.225 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:49:31.225 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile mgr", 2024-03-23T03:49:31.225 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:49:31.226 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.226 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:31.226 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-03-23T03:49:31.226 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:49:31.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.215+0000 7f1305200640 1 -- 172.21.15.102:0/2906715300 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f12dc06ac00 msgr2=0x7f12dc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:31.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.215+0000 7f1305200640 1 --2- 172.21.15.102:0/2906715300 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f12dc06ac00 0x7f12dc06d0c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f12f0009c50 tx=0x7f12f0008040 comp rx=0 tx=0).stop 2024-03-23T03:49:31.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.215+0000 7f1305200640 1 -- 172.21.15.102:0/2906715300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f130015cda0 msgr2=0x7f13000caf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:31.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.216+0000 7f1305200640 1 --2- 172.21.15.102:0/2906715300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f130015cda0 0x7f13000caf30 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f12f8009eb0 tx=0x7f12f8004a40 comp rx=0 tx=0).stop 2024-03-23T03:49:31.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.216+0000 7f1305200640 1 -- 172.21.15.102:0/2906715300 shutdown_connections 2024-03-23T03:49:31.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.216+0000 7f1305200640 1 --2- 172.21.15.102:0/2906715300 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f12dc06ac00 0x7f12dc06d0c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.216+0000 7f1305200640 1 --2- 172.21.15.102:0/2906715300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f130015cda0 0x7f13000caf30 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.216+0000 7f1305200640 1 --2- 172.21.15.102:0/2906715300 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13000b5a30 0x7f13000ca9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.216+0000 7f1305200640 1 --2- 172.21.15.102:0/2906715300 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f13000b4a60 0x7f13000c6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.216+0000 7f1305200640 1 -- 172.21.15.102:0/2906715300 >> 172.21.15.102:0/2906715300 conn(0x7f13000cf350 msgr2=0x7f13000b3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:31.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.216+0000 7f1305200640 1 -- 172.21.15.102:0/2906715300 shutdown_connections 2024-03-23T03:49:31.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.216+0000 7f1305200640 1 -- 172.21.15.102:0/2906715300 wait complete. 2024-03-23T03:49:31.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:49:31.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.441+0000 7fdd9a000640 1 Processor -- start 2024-03-23T03:49:31.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.441+0000 7fdd9a000640 1 -- start start 2024-03-23T03:49:31.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.443+0000 7fdd9a000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd940b4a60 0x7fdd940b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:31.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.443+0000 7fdd9a000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd940d1830 con 0x7fdd9415cda0 2024-03-23T03:49:31.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.443+0000 7fdd9a000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd940b1f20 con 0x7fdd940b5a30 2024-03-23T03:49:31.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.443+0000 7fdd9a000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd940d0c20 con 0x7fdd940b4a60 2024-03-23T03:49:31.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.443+0000 7fdd93400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdd9415cda0 0x7fdd940cdb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33638/0 (socket says 172.21.15.102:33638) 2024-03-23T03:49:31.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.443+0000 7fdd93400640 1 -- 172.21.15.102:0/288587967 learned_addr learned my addr 172.21.15.102:0/288587967 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:31.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.444+0000 7fdd92000640 1 -- 172.21.15.102:0/288587967 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2524334929 0 0) 0x7fdd940d1830 con 0x7fdd9415cda0 2024-03-23T03:49:31.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.444+0000 7fdd92000640 1 -- 172.21.15.102:0/288587967 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd74003520 con 0x7fdd9415cda0 2024-03-23T03:49:31.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.444+0000 7fdd93e00640 1 --2- 172.21.15.102:0/288587967 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd940b4a60 0x7fdd940b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:31.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.444+0000 7fdd92000640 1 -- 172.21.15.102:0/288587967 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1858135539 0 0) 0x7fdd74003520 con 0x7fdd9415cda0 2024-03-23T03:49:31.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.444+0000 7fdd92000640 1 -- 172.21.15.102:0/288587967 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd940b4a60 msgr2=0x7fdd940b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:31.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.444+0000 7fdd92000640 1 --2- 172.21.15.102:0/288587967 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd940b4a60 0x7fdd940b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.444+0000 7fdd92000640 1 -- 172.21.15.102:0/288587967 >> v1:172.21.15.173:6789/0 conn(0x7fdd940b5a30 legacy=0x7fdd940b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:31.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.444+0000 7fdd92000640 1 -- 172.21.15.102:0/288587967 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd940d0f00 con 0x7fdd9415cda0 2024-03-23T03:49:31.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.444+0000 7fdd92000640 1 -- 172.21.15.102:0/288587967 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fdd84001670 con 0x7fdd9415cda0 2024-03-23T03:49:31.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.444+0000 7fdd92000640 1 -- 172.21.15.102:0/288587967 >> v1:172.21.15.102:6789/0 conn(0x7fdd9415cda0 legacy=0x7fdd940cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:31.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.444+0000 7fdd92000640 1 --2- 172.21.15.102:0/288587967 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd740039a0 0x7fdd74003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:31.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.444+0000 7fdd92000640 1 --2- 172.21.15.102:0/288587967 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd740047c0 0x7fdd74004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:31.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.444+0000 7fdd92000640 1 --2- 172.21.15.102:0/288587967 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd940b4a60 0x7fdd74005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:31.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.444+0000 7fdd92000640 1 -- 172.21.15.102:0/288587967 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd940d0c20 con 0x7fdd740047c0 2024-03-23T03:49:31.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.444+0000 7fdd92000640 1 -- 172.21.15.102:0/288587967 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7400bb40 con 0x7fdd740039a0 2024-03-23T03:49:31.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.444+0000 7fdd92000640 1 -- 172.21.15.102:0/288587967 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7400bd00 con 0x7fdd940b4a60 2024-03-23T03:49:31.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.445+0000 7fdd92a00640 1 --2- 172.21.15.102:0/288587967 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd740047c0 0x7fdd74004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:31.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.445+0000 7fdd93e00640 1 --2- 172.21.15.102:0/288587967 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd940b4a60 0x7fdd74005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:31.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.445+0000 7fdd92a00640 1 -- 172.21.15.102:0/288587967 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd940b4a60 msgr2=0x7fdd74005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:31.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.445+0000 7fdd92a00640 1 --2- 172.21.15.102:0/288587967 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd940b4a60 0x7fdd74005210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.445+0000 7fdd92a00640 1 -- 172.21.15.102:0/288587967 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd740039a0 msgr2=0x7fdd74003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:31.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.445+0000 7fdd92a00640 1 --2- 172.21.15.102:0/288587967 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd740039a0 0x7fdd74003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.445+0000 7fdd92a00640 1 -- 172.21.15.102:0/288587967 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd74003520 con 0x7fdd740047c0 2024-03-23T03:49:31.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.446+0000 7fdd92a00640 1 --2- 172.21.15.102:0/288587967 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd740047c0 0x7fdd74004c40 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c001e50 tx=0x7fdd740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79cb3cac34cd948a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:31.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.446+0000 7fdd92000640 1 -- 172.21.15.102:0/288587967 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c003b70 con 0x7fdd740047c0 2024-03-23T03:49:31.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.446+0000 7fdd92000640 1 -- 172.21.15.102:0/288587967 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7c003cd0 con 0x7fdd740047c0 2024-03-23T03:49:31.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.446+0000 7fdd92000640 1 -- 172.21.15.102:0/288587967 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7c005310 con 0x7fdd740047c0 2024-03-23T03:49:31.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.447+0000 7fdd9a000640 1 -- 172.21.15.102:0/288587967 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd740047c0 msgr2=0x7fdd74004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:31.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.447+0000 7fdd9a000640 1 --2- 172.21.15.102:0/288587967 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd740047c0 0x7fdd74004c40 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c001e50 tx=0x7fdd740057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.447+0000 7fdd93e00640 1 -- 172.21.15.102:0/288587967 reap_dead start 2024-03-23T03:49:31.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.447+0000 7fdd9a000640 1 -- 172.21.15.102:0/288587967 shutdown_connections 2024-03-23T03:49:31.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.447+0000 7fdd9a000640 1 -- 172.21.15.102:0/288587967 >> 172.21.15.102:0/288587967 conn(0x7fdd940cf350 msgr2=0x7fdd940cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:31.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.447+0000 7fdd9a000640 1 -- 172.21.15.102:0/288587967 shutdown_connections 2024-03-23T03:49:31.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.447+0000 7fdd9a000640 1 -- 172.21.15.102:0/288587967 wait complete. 2024-03-23T03:49:31.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.449+0000 7fdd9a000640 1 Processor -- start 2024-03-23T03:49:31.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.449+0000 7fdd9a000640 1 -- start start 2024-03-23T03:49:31.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.450+0000 7fdd9a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd740047c0 0x7fdd9415ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:31.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.450+0000 7fdd9a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd940c8f40 0x7fdd940cb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:31.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.450+0000 7fdd9a000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd941589a0 0x7fdd94158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:31.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.450+0000 7fdd9a000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd940c93c0 con 0x7fdd740047c0 2024-03-23T03:49:31.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.450+0000 7fdd9a000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd94159360 con 0x7fdd940c8f40 2024-03-23T03:49:31.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.450+0000 7fdd9a000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9415f140 con 0x7fdd941589a0 2024-03-23T03:49:31.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.450+0000 7fdd93400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd740047c0 0x7fdd9415ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:31.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.450+0000 7fdd93400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd740047c0 0x7fdd9415ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52146/0 (socket says 172.21.15.102:52146) 2024-03-23T03:49:31.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.450+0000 7fdd93400640 1 -- 172.21.15.102:0/2214140035 learned_addr learned my addr 172.21.15.102:0/2214140035 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:31.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.451+0000 7fdd93400640 1 -- 172.21.15.102:0/2214140035 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd941589a0 msgr2=0x7fdd94158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:31.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.451+0000 7fdd93400640 1 --2- 172.21.15.102:0/2214140035 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd941589a0 0x7fdd94158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.451+0000 7fdd93400640 1 -- 172.21.15.102:0/2214140035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd940c8f40 msgr2=0x7fdd940cb7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:31.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.451+0000 7fdd92a00640 1 --2- 172.21.15.102:0/2214140035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd940c8f40 0x7fdd940cb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:31.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.451+0000 7fdd93400640 1 --2- 172.21.15.102:0/2214140035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd940c8f40 0x7fdd940cb7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.451+0000 7fdd93400640 1 -- 172.21.15.102:0/2214140035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9415f420 con 0x7fdd740047c0 2024-03-23T03:49:31.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.451+0000 7fdd93400640 1 --2- 172.21.15.102:0/2214140035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd740047c0 0x7fdd9415ea60 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fdd8400c700 tx=0x7fdd8400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:31.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.451+0000 7fdd8b400640 1 -- 172.21.15.102:0/2214140035 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd840072d0 con 0x7fdd740047c0 2024-03-23T03:49:31.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.451+0000 7fdd8b400640 1 -- 172.21.15.102:0/2214140035 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd84007430 con 0x7fdd740047c0 2024-03-23T03:49:31.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.451+0000 7fdd9a000640 1 -- 172.21.15.102:0/2214140035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9415f700 con 0x7fdd740047c0 2024-03-23T03:49:31.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.451+0000 7fdd8b400640 1 -- 172.21.15.102:0/2214140035 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd840047e0 con 0x7fdd740047c0 2024-03-23T03:49:31.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.451+0000 7fdd9a000640 1 -- 172.21.15.102:0/2214140035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9415fcc0 con 0x7fdd740047c0 2024-03-23T03:49:31.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.453+0000 7fdd9a000640 1 -- 172.21.15.102:0/2214140035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9415fee0 con 0x7fdd740047c0 2024-03-23T03:49:31.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.453+0000 7fdd8b400640 1 -- 172.21.15.102:0/2214140035 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdd84004940 con 0x7fdd740047c0 2024-03-23T03:49:31.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.454+0000 7fdd8b400640 1 --2- 172.21.15.102:0/2214140035 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdd7006ac00 0x7fdd7006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:31.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.454+0000 7fdd8b400640 1 -- 172.21.15.102:0/2214140035 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fdd84088900 con 0x7fdd740047c0 2024-03-23T03:49:31.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.455+0000 7fdd92a00640 1 --2- 172.21.15.102:0/2214140035 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdd7006ac00 0x7fdd7006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:31.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.455+0000 7fdd92a00640 1 --2- 172.21.15.102:0/2214140035 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdd7006ac00 0x7fdd7006d0c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c034000 tx=0x7fdd7c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:31.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.459+0000 7fdd8b400640 1 -- 172.21.15.102:0/2214140035 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdd84053c50 con 0x7fdd740047c0 2024-03-23T03:49:31.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.654+0000 7fdd9a000640 1 -- 172.21.15.102:0/2214140035 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd94073390 con 0x7fdd740047c0 2024-03-23T03:49:31.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.655+0000 7fdd8b400640 1 -- 172.21.15.102:0/2214140035 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+12642 (secure 0 0 0) 0x7fdd84018020 con 0x7fdd740047c0 2024-03-23T03:49:31.656 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:31.656 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":63,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:49:30.679801+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:49:31.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.657+0000 7fdd9a000640 1 -- 172.21.15.102:0/2214140035 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdd7006ac00 msgr2=0x7fdd7006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:31.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.657+0000 7fdd9a000640 1 --2- 172.21.15.102:0/2214140035 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdd7006ac00 0x7fdd7006d0c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fdd7c034000 tx=0x7fdd7c009030 comp rx=0 tx=0).stop 2024-03-23T03:49:31.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.657+0000 7fdd9a000640 1 -- 172.21.15.102:0/2214140035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd740047c0 msgr2=0x7fdd9415ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:31.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.657+0000 7fdd9a000640 1 --2- 172.21.15.102:0/2214140035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd740047c0 0x7fdd9415ea60 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fdd8400c700 tx=0x7fdd8400cbc0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.657+0000 7fdd9a000640 1 -- 172.21.15.102:0/2214140035 shutdown_connections 2024-03-23T03:49:31.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.657+0000 7fdd9a000640 1 --2- 172.21.15.102:0/2214140035 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdd7006ac00 0x7fdd7006d0c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.657+0000 7fdd9a000640 1 --2- 172.21.15.102:0/2214140035 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd941589a0 0x7fdd94158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.657+0000 7fdd9a000640 1 --2- 172.21.15.102:0/2214140035 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd940c8f40 0x7fdd940cb7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.657+0000 7fdd9a000640 1 --2- 172.21.15.102:0/2214140035 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd740047c0 0x7fdd9415ea60 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.657+0000 7fdd9a000640 1 -- 172.21.15.102:0/2214140035 >> 172.21.15.102:0/2214140035 conn(0x7fdd940cf350 msgr2=0x7fdd940b2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:31.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.658+0000 7fdd9a000640 1 -- 172.21.15.102:0/2214140035 shutdown_connections 2024-03-23T03:49:31.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.658+0000 7fdd9a000640 1 -- 172.21.15.102:0/2214140035 wait complete. 2024-03-23T03:49:31.677 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:49:31.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:49:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.882+0000 7f1c5a400640 1 Processor -- start 2024-03-23T03:49:31.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.882+0000 7f1c5a400640 1 -- start start 2024-03-23T03:49:31.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.884+0000 7f1c5a400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1c540cf7c0 0x7f1c540cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:31.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.884+0000 7f1c5a400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1c54160e90 0x7f1c540d0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:31.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.884+0000 7f1c5a400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c540ccd90 con 0x7f1c5415c820 2024-03-23T03:49:31.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.884+0000 7f1c5a400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c540cdd70 con 0x7f1c540cf7c0 2024-03-23T03:49:31.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.884+0000 7f1c5a400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c540b7fc0 con 0x7f1c54160e90 2024-03-23T03:49:31.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.885+0000 7f1c53e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1c540cf7c0 0x7f1c540cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.885+0000 7f1c59a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1c54160e90 0x7f1c540d0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.885+0000 7f1c53e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1c540cf7c0 0x7f1c540cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:47796/0 (socket says 172.21.15.102:47796) 2024-03-23T03:49:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.885+0000 7f1c53e00640 1 -- 172.21.15.102:0/1362631041 learned_addr learned my addr 172.21.15.102:0/1362631041 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.885+0000 7f1c59a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1c54160e90 0x7f1c540d0560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:34130/0 (socket says 172.21.15.102:34130) 2024-03-23T03:49:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.885+0000 7f1c59000640 1 --1- 172.21.15.102:0/1362631041 >> v1:172.21.15.173:6789/0 conn(0x7f1c5415c820 0x7f1c540cf0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:49422/0 (socket says 172.21.15.102:49422) 2024-03-23T03:49:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.885+0000 7f1c53e00640 1 -- 172.21.15.102:0/1362631041 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1c54160e90 msgr2=0x7f1c540d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.885+0000 7f1c53e00640 1 --2- 172.21.15.102:0/1362631041 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1c54160e90 0x7f1c540d0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.885+0000 7f1c53e00640 1 -- 172.21.15.102:0/1362631041 >> v1:172.21.15.173:6789/0 conn(0x7f1c5415c820 legacy=0x7f1c540cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:31.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.885+0000 7f1c53e00640 1 -- 172.21.15.102:0/1362631041 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c540b82a0 con 0x7f1c540cf7c0 2024-03-23T03:49:31.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.886+0000 7f1c59a00640 1 --2- 172.21.15.102:0/1362631041 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1c54160e90 0x7f1c540d0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:49:31.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.887+0000 7f1c53e00640 1 --2- 172.21.15.102:0/1362631041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c540cf7c0 0x7f1c540cd830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009870 tx=0x7f1c4002ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1d54d270e270c78 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:31.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.887+0000 7f1c53400640 1 -- 172.21.15.102:0/1362631041 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4002f830 con 0x7f1c540cf7c0 2024-03-23T03:49:31.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.887+0000 7f1c53400640 1 -- 172.21.15.102:0/1362631041 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4002f990 con 0x7f1c540cf7c0 2024-03-23T03:49:31.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.887+0000 7f1c53400640 1 -- 172.21.15.102:0/1362631041 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c4002fd20 con 0x7f1c540cf7c0 2024-03-23T03:49:31.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.888+0000 7f1c5a400640 1 -- 172.21.15.102:0/1362631041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c540cf7c0 msgr2=0x7f1c540cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:31.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.888+0000 7f1c5a400640 1 --2- 172.21.15.102:0/1362631041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c540cf7c0 0x7f1c540cd830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1c40009870 tx=0x7f1c4002ba10 comp rx=0 tx=0).stop 2024-03-23T03:49:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.888+0000 7f1c5a400640 1 -- 172.21.15.102:0/1362631041 shutdown_connections 2024-03-23T03:49:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.888+0000 7f1c5a400640 1 --2- 172.21.15.102:0/1362631041 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1c54160e90 0x7f1c540d0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.888+0000 7f1c5a400640 1 --2- 172.21.15.102:0/1362631041 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c540cf7c0 0x7f1c540cd830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.888+0000 7f1c5a400640 1 -- 172.21.15.102:0/1362631041 >> 172.21.15.102:0/1362631041 conn(0x7f1c5406b050 msgr2=0x7f1c5406b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.888+0000 7f1c5a400640 1 -- 172.21.15.102:0/1362631041 shutdown_connections 2024-03-23T03:49:31.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.888+0000 7f1c5a400640 1 -- 172.21.15.102:0/1362631041 wait complete. 2024-03-23T03:49:31.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.891+0000 7f1c5a400640 1 Processor -- start 2024-03-23T03:49:31.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.891+0000 7f1c5a400640 1 -- start start 2024-03-23T03:49:31.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.892+0000 7f1c5a400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1c540cf7c0 0x7f1c54157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:31.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.892+0000 7f1c5a400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c5415c820 0x7f1c5414dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:31.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.892+0000 7f1c5a400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c54160e90 0x7f1c540b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:31.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.892+0000 7f1c5a400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c540b3f50 con 0x7f1c5415c820 2024-03-23T03:49:31.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.892+0000 7f1c5a400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c540c79c0 con 0x7f1c54160e90 2024-03-23T03:49:31.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.892+0000 7f1c5a400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c540c76e0 con 0x7f1c540cf7c0 2024-03-23T03:49:31.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.892+0000 7f1c53e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c5415c820 0x7f1c5414dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:31.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.893+0000 7f1c53e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c5415c820 0x7f1c5414dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52156/0 (socket says 172.21.15.102:52156) 2024-03-23T03:49:31.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.893+0000 7f1c53e00640 1 -- 172.21.15.102:0/558568705 learned_addr learned my addr 172.21.15.102:0/558568705 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:31.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.893+0000 7f1c59000640 1 --2- 172.21.15.102:0/558568705 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1c540cf7c0 0x7f1c54157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:31.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.893+0000 7f1c59a00640 1 --2- 172.21.15.102:0/558568705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c54160e90 0x7f1c540b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:31.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.893+0000 7f1c53e00640 1 -- 172.21.15.102:0/558568705 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1c540cf7c0 msgr2=0x7f1c54157a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:31.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.893+0000 7f1c53e00640 1 --2- 172.21.15.102:0/558568705 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1c540cf7c0 0x7f1c54157a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.893+0000 7f1c53e00640 1 -- 172.21.15.102:0/558568705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c54160e90 msgr2=0x7f1c540b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:31.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.893+0000 7f1c53e00640 1 --2- 172.21.15.102:0/558568705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c54160e90 0x7f1c540b43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:31.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.893+0000 7f1c53e00640 1 -- 172.21.15.102:0/558568705 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4c009a70 con 0x7f1c5415c820 2024-03-23T03:49:31.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.893+0000 7f1c53e00640 1 --2- 172.21.15.102:0/558568705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c5415c820 0x7f1c5414dc80 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f1c40000c00 tx=0x7f1c40004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:31.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.894+0000 7f1c50c00640 1 -- 172.21.15.102:0/558568705 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40037c40 con 0x7f1c5415c820 2024-03-23T03:49:31.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.894+0000 7f1c50c00640 1 -- 172.21.15.102:0/558568705 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c40037da0 con 0x7f1c5415c820 2024-03-23T03:49:31.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.894+0000 7f1c50c00640 1 -- 172.21.15.102:0/558568705 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c40041640 con 0x7f1c5415c820 2024-03-23T03:49:31.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.894+0000 7f1c5a400640 1 -- 172.21.15.102:0/558568705 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c400094d0 con 0x7f1c5415c820 2024-03-23T03:49:31.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.894+0000 7f1c5a400640 1 -- 172.21.15.102:0/558568705 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c540b3be0 con 0x7f1c5415c820 2024-03-23T03:49:31.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.895+0000 7f1c5a400640 1 -- 172.21.15.102:0/558568705 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c540b4100 con 0x7f1c5415c820 2024-03-23T03:49:31.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.896+0000 7f1c50c00640 1 -- 172.21.15.102:0/558568705 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1c400417a0 con 0x7f1c5415c820 2024-03-23T03:49:31.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.897+0000 7f1c50c00640 1 --2- 172.21.15.102:0/558568705 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1c3406ac00 0x7f1c3406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:31.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.897+0000 7f1c50c00640 1 -- 172.21.15.102:0/558568705 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f1c4003c070 con 0x7f1c5415c820 2024-03-23T03:49:31.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.897+0000 7f1c59000640 1 --2- 172.21.15.102:0/558568705 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1c3406ac00 0x7f1c3406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:31.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.898+0000 7f1c59000640 1 --2- 172.21.15.102:0/558568705 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1c3406ac00 0x7f1c3406d0c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1c44000f60 tx=0x7f1c44007ca0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:31.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:31.903+0000 7f1c50c00640 1 -- 172.21.15.102:0/558568705 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1c540b4100 con 0x7f1c5415c820 2024-03-23T03:49:32.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.097+0000 7f1c5a400640 1 -- 172.21.15.102:0/558568705 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1c540d2480 con 0x7f1c5415c820 2024-03-23T03:49:32.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.098+0000 7f1c50c00640 1 -- 172.21.15.102:0/558568705 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v63) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1c40077100 con 0x7f1c5415c820 2024-03-23T03:49:32.099 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:49:32.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.101+0000 7f1c5a400640 1 -- 172.21.15.102:0/558568705 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1c3406ac00 msgr2=0x7f1c3406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:32.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.101+0000 7f1c5a400640 1 --2- 172.21.15.102:0/558568705 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1c3406ac00 0x7f1c3406d0c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1c44000f60 tx=0x7f1c44007ca0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.101+0000 7f1c5a400640 1 -- 172.21.15.102:0/558568705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c5415c820 msgr2=0x7f1c5414dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:32.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.101+0000 7f1c5a400640 1 --2- 172.21.15.102:0/558568705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c5415c820 0x7f1c5414dc80 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f1c40000c00 tx=0x7f1c40004730 comp rx=0 tx=0).stop 2024-03-23T03:49:32.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.101+0000 7f1c5a400640 1 -- 172.21.15.102:0/558568705 shutdown_connections 2024-03-23T03:49:32.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.101+0000 7f1c5a400640 1 --2- 172.21.15.102:0/558568705 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1c3406ac00 0x7f1c3406d0c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.101+0000 7f1c5a400640 1 --2- 172.21.15.102:0/558568705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c54160e90 0x7f1c540b43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.101+0000 7f1c5a400640 1 --2- 172.21.15.102:0/558568705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1c5415c820 0x7f1c5414dc80 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.101+0000 7f1c5a400640 1 --2- 172.21.15.102:0/558568705 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1c540cf7c0 0x7f1c54157a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.101+0000 7f1c5a400640 1 -- 172.21.15.102:0/558568705 >> 172.21.15.102:0/558568705 conn(0x7f1c5406b050 msgr2=0x7f1c5406b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:32.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.101+0000 7f1c5a400640 1 -- 172.21.15.102:0/558568705 shutdown_connections 2024-03-23T03:49:32.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.101+0000 7f1c5a400640 1 -- 172.21.15.102:0/558568705 wait complete. 2024-03-23T03:49:32.120 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T03:49:32.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.320+0000 7f0441400640 1 Processor -- start 2024-03-23T03:49:32.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.320+0000 7f0441400640 1 -- start start 2024-03-23T03:49:32.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.322+0000 7f0441400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f043c0b39b0 0x7f043c0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:32.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.322+0000 7f0441400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f043c15c400 0x7f043c15c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:32.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.322+0000 7f0441400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f043c06c6f0 con 0x7f043c15b560 2024-03-23T03:49:32.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.322+0000 7f0441400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f043c0b4370 con 0x7f043c0b39b0 2024-03-23T03:49:32.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.322+0000 7f0441400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f043c06c9d0 con 0x7f043c15c400 2024-03-23T03:49:32.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.322+0000 7f0439e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f043c0b39b0 0x7f043c0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:32.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.322+0000 7f043a800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f043c15b560 0x7f043c15b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33646/0 (socket says 172.21.15.102:33646) 2024-03-23T03:49:32.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.322+0000 7f0439e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f043c0b39b0 0x7f043c0b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52162/0 (socket says 172.21.15.102:52162) 2024-03-23T03:49:32.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.322+0000 7f043a800640 1 -- 172.21.15.102:0/3757386527 learned_addr learned my addr 172.21.15.102:0/3757386527 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:32.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.322+0000 7f043b200640 1 --2- 172.21.15.102:0/3757386527 >> v2:172.21.15.173:3300/0 conn(0x7f043c15c400 0x7f043c15c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:32.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.322+0000 7f0439e00640 1 -- 172.21.15.102:0/3757386527 >> v2:172.21.15.173:3300/0 conn(0x7f043c15c400 msgr2=0x7f043c15c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:32.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.322+0000 7f0439e00640 1 --2- 172.21.15.102:0/3757386527 >> v2:172.21.15.173:3300/0 conn(0x7f043c15c400 0x7f043c15c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.322+0000 7f0439e00640 1 -- 172.21.15.102:0/3757386527 >> v1:172.21.15.102:6789/0 conn(0x7f043c15b560 legacy=0x7f043c15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:32.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.322+0000 7f0439e00640 1 -- 172.21.15.102:0/3757386527 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c06ccb0 con 0x7f043c0b39b0 2024-03-23T03:49:32.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.322+0000 7f043b200640 1 --2- 172.21.15.102:0/3757386527 >> v2:172.21.15.173:3300/0 conn(0x7f043c15c400 0x7f043c15c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:32.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.322+0000 7f0439400640 1 -- 172.21.15.102:0/3757386527 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244915673 0 0) 0x7f043c06c6f0 con 0x7f043c15b560 2024-03-23T03:49:32.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.324+0000 7f0439e00640 1 --2- 172.21.15.102:0/3757386527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f043c0b39b0 0x7f043c0b3e30 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f043c070100 tx=0x7f042c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66f4a9ed75af5901 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:32.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.324+0000 7f0439400640 1 -- 172.21.15.102:0/3757386527 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c031830 con 0x7f043c0b39b0 2024-03-23T03:49:32.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.324+0000 7f0439400640 1 -- 172.21.15.102:0/3757386527 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042c031990 con 0x7f043c0b39b0 2024-03-23T03:49:32.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.324+0000 7f0439400640 1 -- 172.21.15.102:0/3757386527 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c031d20 con 0x7f043c0b39b0 2024-03-23T03:49:32.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.324+0000 7f0441400640 1 -- 172.21.15.102:0/3757386527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f043c0b39b0 msgr2=0x7f043c0b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:32.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.324+0000 7f0441400640 1 --2- 172.21.15.102:0/3757386527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f043c0b39b0 0x7f043c0b3e30 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f043c070100 tx=0x7f042c02b8d0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.325+0000 7f0441400640 1 -- 172.21.15.102:0/3757386527 shutdown_connections 2024-03-23T03:49:32.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.325+0000 7f0441400640 1 --2- 172.21.15.102:0/3757386527 >> v2:172.21.15.173:3300/0 conn(0x7f043c15c400 0x7f043c15c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.325+0000 7f0441400640 1 --2- 172.21.15.102:0/3757386527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f043c0b39b0 0x7f043c0b3e30 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.325+0000 7f0441400640 1 -- 172.21.15.102:0/3757386527 >> 172.21.15.102:0/3757386527 conn(0x7f043c06a3d0 msgr2=0x7f043c0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:32.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.325+0000 7f0441400640 1 -- 172.21.15.102:0/3757386527 shutdown_connections 2024-03-23T03:49:32.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.325+0000 7f0441400640 1 -- 172.21.15.102:0/3757386527 wait complete. 2024-03-23T03:49:32.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.326+0000 7f0441400640 1 Processor -- start 2024-03-23T03:49:32.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.326+0000 7f0441400640 1 -- start start 2024-03-23T03:49:32.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.327+0000 7f0441400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f043c0b39b0 0x7f043c0cb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:32.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.327+0000 7f0441400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f043c15b560 0x7f043c1f5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:32.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.327+0000 7f0441400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f043c15c400 0x7f043c14d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:32.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.327+0000 7f0441400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c1f6450 con 0x7f043c15b560 2024-03-23T03:49:32.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.327+0000 7f0441400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f043c14d590 con 0x7f043c0b39b0 2024-03-23T03:49:32.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.327+0000 7f0441400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f043c0cb6f0 con 0x7f043c15c400 2024-03-23T03:49:32.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.328+0000 7f0439e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f043c15b560 0x7f043c1f5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:32.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.328+0000 7f0439e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f043c15b560 0x7f043c1f5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52174/0 (socket says 172.21.15.102:52174) 2024-03-23T03:49:32.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.328+0000 7f0439e00640 1 -- 172.21.15.102:0/2792307144 learned_addr learned my addr 172.21.15.102:0/2792307144 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:32.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.328+0000 7f043b200640 1 --2- 172.21.15.102:0/2792307144 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f043c15c400 0x7f043c14d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:32.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.328+0000 7f043a800640 1 --2- 172.21.15.102:0/2792307144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f043c0b39b0 0x7f043c0cb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:32.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.328+0000 7f0439e00640 1 -- 172.21.15.102:0/2792307144 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f043c15c400 msgr2=0x7f043c14d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:32.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.328+0000 7f0439e00640 1 --2- 172.21.15.102:0/2792307144 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f043c15c400 0x7f043c14d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.328+0000 7f0439e00640 1 -- 172.21.15.102:0/2792307144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f043c0b39b0 msgr2=0x7f043c0cb080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:32.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.328+0000 7f0439e00640 1 --2- 172.21.15.102:0/2792307144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f043c0b39b0 0x7f043c0cb080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.328+0000 7f0439e00640 1 -- 172.21.15.102:0/2792307144 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042c009400 con 0x7f043c15b560 2024-03-23T03:49:32.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.328+0000 7f043b200640 1 --2- 172.21.15.102:0/2792307144 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f043c15c400 0x7f043c14d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:32.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.328+0000 7f043a800640 1 --2- 172.21.15.102:0/2792307144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f043c0b39b0 0x7f043c0cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:32.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.328+0000 7f0439e00640 1 --2- 172.21.15.102:0/2792307144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f043c15b560 0x7f043c1f5f10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f042c009890 tx=0x7f042c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:32.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.329+0000 7f0432a00640 1 -- 172.21.15.102:0/2792307144 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c004230 con 0x7f043c15b560 2024-03-23T03:49:32.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.329+0000 7f0432a00640 1 -- 172.21.15.102:0/2792307144 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f042c004390 con 0x7f043c15b560 2024-03-23T03:49:32.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.329+0000 7f0441400640 1 -- 172.21.15.102:0/2792307144 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f043c0cb9d0 con 0x7f043c15b560 2024-03-23T03:49:32.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.329+0000 7f0432a00640 1 -- 172.21.15.102:0/2792307144 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042c03a920 con 0x7f043c15b560 2024-03-23T03:49:32.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.329+0000 7f0441400640 1 -- 172.21.15.102:0/2792307144 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f043c0cbf90 con 0x7f043c15b560 2024-03-23T03:49:32.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.330+0000 7f0441400640 1 -- 172.21.15.102:0/2792307144 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f043c06ac20 con 0x7f043c15b560 2024-03-23T03:49:32.331 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.331+0000 7f0432a00640 1 -- 172.21.15.102:0/2792307144 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f042c03aa80 con 0x7f043c15b560 2024-03-23T03:49:32.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.332+0000 7f0432a00640 1 --2- 172.21.15.102:0/2792307144 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f041806ac30 0x7f041806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:32.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.332+0000 7f0432a00640 1 -- 172.21.15.102:0/2792307144 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f042c0ab7a0 con 0x7f043c15b560 2024-03-23T03:49:32.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.332+0000 7f043a800640 1 --2- 172.21.15.102:0/2792307144 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f041806ac30 0x7f041806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:32.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.333+0000 7f043a800640 1 --2- 172.21.15.102:0/2792307144 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f041806ac30 0x7f041806d0f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0424000f60 tx=0x7f0424009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:32.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.336+0000 7f0432a00640 1 -- 172.21.15.102:0/2792307144 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f042c076d50 con 0x7f043c15b560 2024-03-23T03:49:32.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.518+0000 7f0441400640 1 -- 172.21.15.102:0/2792307144 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f043c073390 con 0x7f043c15b560 2024-03-23T03:49:32.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.519+0000 7f0432a00640 1 -- 172.21.15.102:0/2792307144 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v63) v1 ==== 84+0+65 (secure 0 0 0) 0x7f042c035090 con 0x7f043c15b560 2024-03-23T03:49:32.519 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:32.519 INFO:teuthology.orchestra.run.smithi102.stdout:[ 2024-03-23T03:49:32.519 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:49:32.520 INFO:teuthology.orchestra.run.smithi102.stdout: "poolnum": 1, 2024-03-23T03:49:32.520 INFO:teuthology.orchestra.run.smithi102.stdout: "poolname": ".mgr" 2024-03-23T03:49:32.520 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:49:32.520 INFO:teuthology.orchestra.run.smithi102.stdout:] 2024-03-23T03:49:32.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.521+0000 7f0441400640 1 -- 172.21.15.102:0/2792307144 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f041806ac30 msgr2=0x7f041806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:32.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.521+0000 7f0441400640 1 --2- 172.21.15.102:0/2792307144 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f041806ac30 0x7f041806d0f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0424000f60 tx=0x7f0424009040 comp rx=0 tx=0).stop 2024-03-23T03:49:32.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.521+0000 7f0441400640 1 -- 172.21.15.102:0/2792307144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f043c15b560 msgr2=0x7f043c1f5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:32.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.521+0000 7f0441400640 1 --2- 172.21.15.102:0/2792307144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f043c15b560 0x7f043c1f5f10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f042c009890 tx=0x7f042c0046c0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.521+0000 7f0441400640 1 -- 172.21.15.102:0/2792307144 shutdown_connections 2024-03-23T03:49:32.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.521+0000 7f0441400640 1 --2- 172.21.15.102:0/2792307144 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f041806ac30 0x7f041806d0f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.521+0000 7f0441400640 1 --2- 172.21.15.102:0/2792307144 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f043c15c400 0x7f043c14d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.521+0000 7f0441400640 1 --2- 172.21.15.102:0/2792307144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f043c15b560 0x7f043c1f5f10 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.521+0000 7f0441400640 1 --2- 172.21.15.102:0/2792307144 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f043c0b39b0 0x7f043c0cb080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.521+0000 7f0441400640 1 -- 172.21.15.102:0/2792307144 >> 172.21.15.102:0/2792307144 conn(0x7f043c06a3d0 msgr2=0x7f043c0abd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:32.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.521+0000 7f0441400640 1 -- 172.21.15.102:0/2792307144 shutdown_connections 2024-03-23T03:49:32.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.522+0000 7f0441400640 1 -- 172.21.15.102:0/2792307144 wait complete. 2024-03-23T03:49:32.539 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T03:49:32.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T03:49:32.596 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-03-23T03:49:32.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.742+0000 7f7b97a00640 1 Processor -- start 2024-03-23T03:49:32.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.742+0000 7f7b97a00640 1 -- start start 2024-03-23T03:49:32.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.743+0000 7f7b97a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b9015d540 0x7f7b9015d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:32.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.743+0000 7f7b97a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b90161380 con 0x7f7b90160500 2024-03-23T03:49:32.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.743+0000 7f7b97a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b90160ae0 con 0x7f7b900aec40 2024-03-23T03:49:32.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.743+0000 7f7b97a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b90160dc0 con 0x7f7b9015d540 2024-03-23T03:49:32.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.744+0000 7f7b8fe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7b90160500 0x7f7b900712e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33656/0 (socket says 172.21.15.102:33656) 2024-03-23T03:49:32.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.744+0000 7f7b8fe00640 1 -- 172.21.15.102:0/1307719001 learned_addr learned my addr 172.21.15.102:0/1307719001 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:32.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.744+0000 7f7b95a00640 1 --2- 172.21.15.102:0/1307719001 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b9015d540 0x7f7b9015d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:32.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.744+0000 7f7b8f400640 1 -- 172.21.15.102:0/1307719001 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900800020 0 0) 0x7f7b90161380 con 0x7f7b90160500 2024-03-23T03:49:32.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.744+0000 7f7b8f400640 1 -- 172.21.15.102:0/1307719001 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b78003520 con 0x7f7b90160500 2024-03-23T03:49:32.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8f400640 1 -- 172.21.15.102:0/1307719001 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1900868852 0 0) 0x7f7b78003520 con 0x7f7b90160500 2024-03-23T03:49:32.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8f400640 1 -- 172.21.15.102:0/1307719001 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b9015d540 msgr2=0x7f7b9015d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:32.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8f400640 1 --2- 172.21.15.102:0/1307719001 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b9015d540 0x7f7b9015d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8f400640 1 -- 172.21.15.102:0/1307719001 >> v1:172.21.15.173:6789/0 conn(0x7f7b900aec40 legacy=0x7f7b900b3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:32.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8f400640 1 -- 172.21.15.102:0/1307719001 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b901610a0 con 0x7f7b90160500 2024-03-23T03:49:32.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8f400640 1 -- 172.21.15.102:0/1307719001 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f7b7c001670 con 0x7f7b90160500 2024-03-23T03:49:32.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8f400640 1 -- 172.21.15.102:0/1307719001 >> v1:172.21.15.102:6789/0 conn(0x7f7b90160500 legacy=0x7f7b900712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:32.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8f400640 1 --2- 172.21.15.102:0/1307719001 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b9015d540 0x7f7b780039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:32.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8f400640 1 --2- 172.21.15.102:0/1307719001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b78003f20 0x7f7b780047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:32.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8f400640 1 --2- 172.21.15.102:0/1307719001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b78004d00 0x7f7b78005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:32.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8f400640 1 -- 172.21.15.102:0/1307719001 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b90160dc0 con 0x7f7b78003f20 2024-03-23T03:49:32.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8f400640 1 -- 172.21.15.102:0/1307719001 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7800ba60 con 0x7f7b78004d00 2024-03-23T03:49:32.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8f400640 1 -- 172.21.15.102:0/1307719001 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7800bc20 con 0x7f7b9015d540 2024-03-23T03:49:32.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8fe00640 1 --2- 172.21.15.102:0/1307719001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b78003f20 0x7f7b780047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:32.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b95000640 1 --2- 172.21.15.102:0/1307719001 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b9015d540 0x7f7b780039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:32.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b95a00640 1 --2- 172.21.15.102:0/1307719001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b78004d00 0x7f7b78005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:32.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8fe00640 1 -- 172.21.15.102:0/1307719001 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b9015d540 msgr2=0x7f7b780039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:32.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8fe00640 1 --2- 172.21.15.102:0/1307719001 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b9015d540 0x7f7b780039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8fe00640 1 -- 172.21.15.102:0/1307719001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b78004d00 msgr2=0x7f7b78005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:32.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.745+0000 7f7b8fe00640 1 --2- 172.21.15.102:0/1307719001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b78004d00 0x7f7b78005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.746+0000 7f7b8fe00640 1 -- 172.21.15.102:0/1307719001 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b78003520 con 0x7f7b78003f20 2024-03-23T03:49:32.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.747+0000 7f7b8fe00640 1 --2- 172.21.15.102:0/1307719001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b78003f20 0x7f7b780047c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00c240 tx=0x7f7b780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1cd953018c87a0a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:32.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.747+0000 7f7b8f400640 1 -- 172.21.15.102:0/1307719001 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c005ad0 con 0x7f7b78003f20 2024-03-23T03:49:32.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.747+0000 7f7b8f400640 1 -- 172.21.15.102:0/1307719001 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b7c005c30 con 0x7f7b78003f20 2024-03-23T03:49:32.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.747+0000 7f7b8f400640 1 -- 172.21.15.102:0/1307719001 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c02caf0 con 0x7f7b78003f20 2024-03-23T03:49:32.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.747+0000 7f7b97a00640 1 -- 172.21.15.102:0/1307719001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b78003f20 msgr2=0x7f7b780047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:32.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.747+0000 7f7b97a00640 1 --2- 172.21.15.102:0/1307719001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b78003f20 0x7f7b780047c0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c00c240 tx=0x7f7b780057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.747+0000 7f7b95a00640 1 -- 172.21.15.102:0/1307719001 reap_dead start 2024-03-23T03:49:32.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.748+0000 7f7b97a00640 1 -- 172.21.15.102:0/1307719001 shutdown_connections 2024-03-23T03:49:32.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.748+0000 7f7b97a00640 1 -- 172.21.15.102:0/1307719001 >> 172.21.15.102:0/1307719001 conn(0x7f7b9006d300 msgr2=0x7f7b9006d710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:32.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.748+0000 7f7b97a00640 1 -- 172.21.15.102:0/1307719001 shutdown_connections 2024-03-23T03:49:32.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.748+0000 7f7b97a00640 1 -- 172.21.15.102:0/1307719001 wait complete. 2024-03-23T03:49:32.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.749+0000 7f7b97a00640 1 Processor -- start 2024-03-23T03:49:32.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.749+0000 7f7b97a00640 1 -- start start 2024-03-23T03:49:32.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.750+0000 7f7b97a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b78003f20 0x7f7b901520a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:32.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.750+0000 7f7b97a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b900ca680 0x7f7b900cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:32.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.750+0000 7f7b97a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9015f2f0 0x7f7b9015f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:32.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.750+0000 7f7b97a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b900c9680 con 0x7f7b9015f2f0 2024-03-23T03:49:32.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.750+0000 7f7b97a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9015fce0 con 0x7f7b900ca680 2024-03-23T03:49:32.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.750+0000 7f7b97a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b90155df0 con 0x7f7b78003f20 2024-03-23T03:49:32.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.751+0000 7f7b95a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9015f2f0 0x7f7b9015f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:32.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.751+0000 7f7b95a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9015f2f0 0x7f7b9015f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52182/0 (socket says 172.21.15.102:52182) 2024-03-23T03:49:32.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.751+0000 7f7b95a00640 1 -- 172.21.15.102:0/314605042 learned_addr learned my addr 172.21.15.102:0/314605042 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:32.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.751+0000 7f7b8fe00640 1 --2- 172.21.15.102:0/314605042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b900ca680 0x7f7b900cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:32.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.751+0000 7f7b95a00640 1 -- 172.21.15.102:0/314605042 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b78003f20 msgr2=0x7f7b901520a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:32.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.751+0000 7f7b95000640 1 --2- 172.21.15.102:0/314605042 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b78003f20 0x7f7b901520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:32.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.751+0000 7f7b95a00640 1 --2- 172.21.15.102:0/314605042 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b78003f20 0x7f7b901520a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.751+0000 7f7b95a00640 1 -- 172.21.15.102:0/314605042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b900ca680 msgr2=0x7f7b900cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:32.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.751+0000 7f7b95a00640 1 --2- 172.21.15.102:0/314605042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b900ca680 0x7f7b900cab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:32.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.751+0000 7f7b95a00640 1 -- 172.21.15.102:0/314605042 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b90071b70 con 0x7f7b9015f2f0 2024-03-23T03:49:32.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.751+0000 7f7b95a00640 1 --2- 172.21.15.102:0/314605042 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9015f2f0 0x7f7b9015f7a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f7b900d05d0 tx=0x7f7b880115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:32.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.751+0000 7f7b8cc00640 1 -- 172.21.15.102:0/314605042 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8800e070 con 0x7f7b9015f2f0 2024-03-23T03:49:32.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.752+0000 7f7b8cc00640 1 -- 172.21.15.102:0/314605042 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b88011e10 con 0x7f7b9015f2f0 2024-03-23T03:49:32.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.752+0000 7f7b97a00640 1 -- 172.21.15.102:0/314605042 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b900cb140 con 0x7f7b9015f2f0 2024-03-23T03:49:32.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.752+0000 7f7b8cc00640 1 -- 172.21.15.102:0/314605042 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88003990 con 0x7f7b9015f2f0 2024-03-23T03:49:32.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.752+0000 7f7b97a00640 1 -- 172.21.15.102:0/314605042 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b90156040 con 0x7f7b9015f2f0 2024-03-23T03:49:32.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.753+0000 7f7b97a00640 1 -- 172.21.15.102:0/314605042 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b90073390 con 0x7f7b9015f2f0 2024-03-23T03:49:32.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.754+0000 7f7b8cc00640 1 -- 172.21.15.102:0/314605042 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7b88003af0 con 0x7f7b9015f2f0 2024-03-23T03:49:32.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.754+0000 7f7b8cc00640 1 --2- 172.21.15.102:0/314605042 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7b7006ac30 0x7f7b7006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:32.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.755+0000 7f7b8cc00640 1 -- 172.21.15.102:0/314605042 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f7b880897f0 con 0x7f7b9015f2f0 2024-03-23T03:49:32.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.755+0000 7f7b95000640 1 --2- 172.21.15.102:0/314605042 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7b7006ac30 0x7f7b7006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:32.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.755+0000 7f7b95000640 1 --2- 172.21.15.102:0/314605042 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7b7006ac30 0x7f7b7006d0f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7b84007fd0 tx=0x7f7b84009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:32.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.759+0000 7f7b8cc00640 1 -- 172.21.15.102:0/314605042 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7b88054bd0 con 0x7f7b9015f2f0 2024-03-23T03:49:32.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:32.954+0000 7f7b97a00640 1 -- 172.21.15.102:0/314605042 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f7b900b01a0 con 0x7f7b9015f2f0 2024-03-23T03:49:33.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:33.945+0000 7f7b8cc00640 1 -- 172.21.15.102:0/314605042 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v64) v1 ==== 138+0+0 (secure 0 0 0) 0x7f7b880581f0 con 0x7f7b9015f2f0 2024-03-23T03:49:33.946 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' created 2024-03-23T03:49:33.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:33.949+0000 7f7b97a00640 1 -- 172.21.15.102:0/314605042 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7b7006ac30 msgr2=0x7f7b7006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:33.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:33.949+0000 7f7b97a00640 1 --2- 172.21.15.102:0/314605042 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7b7006ac30 0x7f7b7006d0f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7b84007fd0 tx=0x7f7b84009040 comp rx=0 tx=0).stop 2024-03-23T03:49:33.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:33.949+0000 7f7b97a00640 1 -- 172.21.15.102:0/314605042 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9015f2f0 msgr2=0x7f7b9015f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:33.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:33.949+0000 7f7b97a00640 1 --2- 172.21.15.102:0/314605042 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9015f2f0 0x7f7b9015f7a0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f7b900d05d0 tx=0x7f7b880115a0 comp rx=0 tx=0).stop 2024-03-23T03:49:33.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:33.949+0000 7f7b97a00640 1 -- 172.21.15.102:0/314605042 shutdown_connections 2024-03-23T03:49:33.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:33.949+0000 7f7b97a00640 1 --2- 172.21.15.102:0/314605042 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7b7006ac30 0x7f7b7006d0f0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:33.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:33.949+0000 7f7b97a00640 1 --2- 172.21.15.102:0/314605042 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9015f2f0 0x7f7b9015f7a0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:33.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:33.949+0000 7f7b97a00640 1 --2- 172.21.15.102:0/314605042 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b900ca680 0x7f7b900cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:33.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:33.949+0000 7f7b97a00640 1 --2- 172.21.15.102:0/314605042 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b78003f20 0x7f7b901520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:33.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:33.949+0000 7f7b97a00640 1 -- 172.21.15.102:0/314605042 >> 172.21.15.102:0/314605042 conn(0x7f7b9006d300 msgr2=0x7f7b900b3550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:33.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:33.950+0000 7f7b97a00640 1 -- 172.21.15.102:0/314605042 shutdown_connections 2024-03-23T03:49:33.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:33.950+0000 7f7b97a00640 1 -- 172.21.15.102:0/314605042 wait complete. 2024-03-23T03:49:33.977 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T03:49:34.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.182+0000 7fd16a600640 1 Processor -- start 2024-03-23T03:49:34.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.183+0000 7fd16a600640 1 -- start start 2024-03-23T03:49:34.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.184+0000 7fd16a600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd1640ce930 0x7fd1640cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:34.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.184+0000 7fd16a600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd164159c10 con 0x7fd1640ca7b0 2024-03-23T03:49:34.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.184+0000 7fd16a600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1640cd840 con 0x7fd164159830 2024-03-23T03:49:34.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.184+0000 7fd16a600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd16406de50 con 0x7fd1640ce930 2024-03-23T03:49:34.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.185+0000 7fd163e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd1640ce930 0x7fd1640cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:34.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.185+0000 7fd163e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd1640ce930 0x7fd1640cedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49744/0 (socket says 172.21.15.102:49744) 2024-03-23T03:49:34.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.185+0000 7fd163e00640 1 -- 172.21.15.102:0/4292590523 learned_addr learned my addr 172.21.15.102:0/4292590523 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:34.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.185+0000 7fd163e00640 1 -- 172.21.15.102:0/4292590523 >> v1:172.21.15.173:6789/0 conn(0x7fd164159830 legacy=0x7fd1640cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:34.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.185+0000 7fd163e00640 1 -- 172.21.15.102:0/4292590523 >> v1:172.21.15.102:6789/0 conn(0x7fd1640ca7b0 legacy=0x7fd1640cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:34.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.185+0000 7fd163e00640 1 -- 172.21.15.102:0/4292590523 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1640ccc30 con 0x7fd1640ce930 2024-03-23T03:49:34.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.186+0000 7fd163e00640 1 --2- 172.21.15.102:0/4292590523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1640ce930 0x7fd1640cedc0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009e80 tx=0x7fd15c02c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2c5f128f7b56f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:34.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.187+0000 7fd162a00640 1 -- 172.21.15.102:0/4292590523 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c02cc10 con 0x7fd1640ce930 2024-03-23T03:49:34.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.187+0000 7fd162a00640 1 -- 172.21.15.102:0/4292590523 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15c02cd70 con 0x7fd1640ce930 2024-03-23T03:49:34.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.187+0000 7fd162a00640 1 -- 172.21.15.102:0/4292590523 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c034610 con 0x7fd1640ce930 2024-03-23T03:49:34.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.187+0000 7fd16a600640 1 -- 172.21.15.102:0/4292590523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1640ce930 msgr2=0x7fd1640cedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:34.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.187+0000 7fd16a600640 1 --2- 172.21.15.102:0/4292590523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1640ce930 0x7fd1640cedc0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009e80 tx=0x7fd15c02c070 comp rx=0 tx=0).stop 2024-03-23T03:49:34.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.188+0000 7fd16a600640 1 -- 172.21.15.102:0/4292590523 shutdown_connections 2024-03-23T03:49:34.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.188+0000 7fd16a600640 1 --2- 172.21.15.102:0/4292590523 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1640ce930 0x7fd1640cedc0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:34.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.188+0000 7fd16a600640 1 -- 172.21.15.102:0/4292590523 >> 172.21.15.102:0/4292590523 conn(0x7fd16406a0b0 msgr2=0x7fd16406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:34.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.188+0000 7fd16a600640 1 -- 172.21.15.102:0/4292590523 shutdown_connections 2024-03-23T03:49:34.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.188+0000 7fd16a600640 1 -- 172.21.15.102:0/4292590523 wait complete. 2024-03-23T03:49:34.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.189+0000 7fd16a600640 1 Processor -- start 2024-03-23T03:49:34.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.190+0000 7fd16a600640 1 -- start start 2024-03-23T03:49:34.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.190+0000 7fd16a600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1640ca7b0 0x7fd164148620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:34.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.190+0000 7fd16a600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1640ce930 0x7fd164150240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:34.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.190+0000 7fd16a600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd164159830 0x7fd1640acca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:34.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.190+0000 7fd16a600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd164150e70 con 0x7fd1640ca7b0 2024-03-23T03:49:34.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.190+0000 7fd16a600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd164150b90 con 0x7fd1640ce930 2024-03-23T03:49:34.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.190+0000 7fd16a600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd164147b40 con 0x7fd164159830 2024-03-23T03:49:34.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.191+0000 7fd163400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1640ca7b0 0x7fd164148620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:34.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.191+0000 7fd163400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1640ca7b0 0x7fd164148620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49758/0 (socket says 172.21.15.102:49758) 2024-03-23T03:49:34.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.191+0000 7fd163400640 1 -- 172.21.15.102:0/3391609899 learned_addr learned my addr 172.21.15.102:0/3391609899 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:34.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.191+0000 7fd163400640 1 -- 172.21.15.102:0/3391609899 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd164159830 msgr2=0x7fd1640acca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:34.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.191+0000 7fd163e00640 1 --2- 172.21.15.102:0/3391609899 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd164159830 0x7fd1640acca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:34.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.191+0000 7fd15aa00640 1 --2- 172.21.15.102:0/3391609899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1640ce930 0x7fd164150240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:34.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.191+0000 7fd163400640 1 --2- 172.21.15.102:0/3391609899 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd164159830 0x7fd1640acca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:34.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.191+0000 7fd163400640 1 -- 172.21.15.102:0/3391609899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1640ce930 msgr2=0x7fd164150240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:34.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.191+0000 7fd163400640 1 --2- 172.21.15.102:0/3391609899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1640ce930 0x7fd164150240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:34.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.191+0000 7fd163400640 1 -- 172.21.15.102:0/3391609899 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15c009b30 con 0x7fd1640ca7b0 2024-03-23T03:49:34.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.192+0000 7fd163400640 1 --2- 172.21.15.102:0/3391609899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1640ca7b0 0x7fd164148620 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd15400c400 tx=0x7fd15400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:34.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.192+0000 7fd15be00640 1 -- 172.21.15.102:0/3391609899 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1540052d0 con 0x7fd1640ca7b0 2024-03-23T03:49:34.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.192+0000 7fd15be00640 1 -- 172.21.15.102:0/3391609899 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd154005430 con 0x7fd1640ca7b0 2024-03-23T03:49:34.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.192+0000 7fd15be00640 1 -- 172.21.15.102:0/3391609899 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154003830 con 0x7fd1640ca7b0 2024-03-23T03:49:34.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.192+0000 7fd16a600640 1 -- 172.21.15.102:0/3391609899 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd164147e20 con 0x7fd1640ca7b0 2024-03-23T03:49:34.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.192+0000 7fd16a600640 1 -- 172.21.15.102:0/3391609899 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd164148b60 con 0x7fd1640ca7b0 2024-03-23T03:49:34.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.193+0000 7fd16a600640 1 -- 172.21.15.102:0/3391609899 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd164073390 con 0x7fd1640ca7b0 2024-03-23T03:49:34.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.195+0000 7fd15be00640 1 -- 172.21.15.102:0/3391609899 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd154003990 con 0x7fd1640ca7b0 2024-03-23T03:49:34.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.195+0000 7fd15be00640 1 --2- 172.21.15.102:0/3391609899 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd14006ac30 0x7fd14006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:34.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.196+0000 7fd15be00640 1 -- 172.21.15.102:0/3391609899 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5547+0+0 (secure 0 0 0) 0x7fd154088bf0 con 0x7fd1640ca7b0 2024-03-23T03:49:34.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.196+0000 7fd15aa00640 1 --2- 172.21.15.102:0/3391609899 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd14006ac30 0x7fd14006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:34.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.196+0000 7fd15aa00640 1 --2- 172.21.15.102:0/3391609899 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd14006ac30 0x7fd14006d0f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd164072980 tx=0x7fd14c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:34.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.200+0000 7fd15be00640 1 -- 172.21.15.102:0/3391609899 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd154053ee0 con 0x7fd1640ca7b0 2024-03-23T03:49:34.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.396+0000 7fd16a600640 1 -- 172.21.15.102:0/3391609899 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd164154de0 con 0x7fd1640ca7b0 2024-03-23T03:49:34.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.949+0000 7fd15be00640 1 -- 172.21.15.102:0/3391609899 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v65) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd154057500 con 0x7fd1640ca7b0 2024-03-23T03:49:34.951 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' created 2024-03-23T03:49:34.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.954+0000 7fd16a600640 1 -- 172.21.15.102:0/3391609899 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd14006ac30 msgr2=0x7fd14006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:34.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.954+0000 7fd16a600640 1 --2- 172.21.15.102:0/3391609899 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd14006ac30 0x7fd14006d0f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd164072980 tx=0x7fd14c009040 comp rx=0 tx=0).stop 2024-03-23T03:49:34.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.954+0000 7fd16a600640 1 -- 172.21.15.102:0/3391609899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1640ca7b0 msgr2=0x7fd164148620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:34.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.954+0000 7fd16a600640 1 --2- 172.21.15.102:0/3391609899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1640ca7b0 0x7fd164148620 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fd15400c400 tx=0x7fd15400c8c0 comp rx=0 tx=0).stop 2024-03-23T03:49:34.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.955+0000 7fd16a600640 1 -- 172.21.15.102:0/3391609899 shutdown_connections 2024-03-23T03:49:34.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.955+0000 7fd16a600640 1 --2- 172.21.15.102:0/3391609899 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd14006ac30 0x7fd14006d0f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:34.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.955+0000 7fd16a600640 1 --2- 172.21.15.102:0/3391609899 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd164159830 0x7fd1640acca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:34.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.955+0000 7fd16a600640 1 --2- 172.21.15.102:0/3391609899 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1640ce930 0x7fd164150240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:34.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.955+0000 7fd16a600640 1 --2- 172.21.15.102:0/3391609899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1640ca7b0 0x7fd164148620 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:34.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.955+0000 7fd16a600640 1 -- 172.21.15.102:0/3391609899 >> 172.21.15.102:0/3391609899 conn(0x7fd16406a0b0 msgr2=0x7fd1640b24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:34.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.955+0000 7fd16a600640 1 -- 172.21.15.102:0/3391609899 shutdown_connections 2024-03-23T03:49:34.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:34.955+0000 7fd16a600640 1 -- 172.21.15.102:0/3391609899 wait complete. 2024-03-23T03:49:34.988 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T03:49:35.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.168+0000 7fe130800640 1 Processor -- start 2024-03-23T03:49:35.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.168+0000 7fe130800640 1 -- start start 2024-03-23T03:49:35.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.169+0000 7fe130800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe1280add30 0x7fe1280ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:35.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.169+0000 7fe130800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe12806ce90 0x7fe12806d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:35.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.169+0000 7fe130800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1280cf250 con 0x7fe1280ce930 2024-03-23T03:49:35.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.169+0000 7fe130800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe12806c760 con 0x7fe12806ce90 2024-03-23T03:49:35.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.169+0000 7fe130800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1280b42d0 con 0x7fe1280add30 2024-03-23T03:49:35.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.170+0000 7fe12e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe12806ce90 0x7fe12806d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:35.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.170+0000 7fe12e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe12806ce90 0x7fe12806d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49760/0 (socket says 172.21.15.102:49760) 2024-03-23T03:49:35.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.170+0000 7fe12e800640 1 -- 172.21.15.102:0/227624101 learned_addr learned my addr 172.21.15.102:0/227624101 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:35.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.170+0000 7fe12d400640 1 --2- 172.21.15.102:0/227624101 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe1280add30 0x7fe1280ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:35.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.170+0000 7fe12e800640 1 -- 172.21.15.102:0/227624101 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe1280add30 msgr2=0x7fe1280ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:35.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.170+0000 7fe12e800640 1 --2- 172.21.15.102:0/227624101 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe1280add30 0x7fe1280ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.170+0000 7fe12e800640 1 -- 172.21.15.102:0/227624101 >> v1:172.21.15.173:6789/0 conn(0x7fe1280ce930 legacy=0x7fe1280cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:35.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.170+0000 7fe12e800640 1 -- 172.21.15.102:0/227624101 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1280cdba0 con 0x7fe12806ce90 2024-03-23T03:49:35.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.172+0000 7fe12e800640 1 --2- 172.21.15.102:0/227624101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe12806ce90 0x7fe12806d320 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe124009e50 tx=0x7fe12402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2471b283b7378d31 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:35.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.172+0000 7fe12ca00640 1 -- 172.21.15.102:0/227624101 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12402cb00 con 0x7fe12806ce90 2024-03-23T03:49:35.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.172+0000 7fe12ca00640 1 -- 172.21.15.102:0/227624101 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12402cc60 con 0x7fe12806ce90 2024-03-23T03:49:35.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.172+0000 7fe12ca00640 1 -- 172.21.15.102:0/227624101 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1240343f0 con 0x7fe12806ce90 2024-03-23T03:49:35.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.173+0000 7fe130800640 1 -- 172.21.15.102:0/227624101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe12806ce90 msgr2=0x7fe12806d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:35.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.173+0000 7fe130800640 1 --2- 172.21.15.102:0/227624101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe12806ce90 0x7fe12806d320 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fe124009e50 tx=0x7fe12402bf60 comp rx=0 tx=0).stop 2024-03-23T03:49:35.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.173+0000 7fe130800640 1 -- 172.21.15.102:0/227624101 shutdown_connections 2024-03-23T03:49:35.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.173+0000 7fe130800640 1 --2- 172.21.15.102:0/227624101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe12806ce90 0x7fe12806d320 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.173+0000 7fe130800640 1 --2- 172.21.15.102:0/227624101 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe1280add30 0x7fe1280ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.173+0000 7fe130800640 1 -- 172.21.15.102:0/227624101 >> 172.21.15.102:0/227624101 conn(0x7fe12806a0b0 msgr2=0x7fe12806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:35.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.173+0000 7fe130800640 1 -- 172.21.15.102:0/227624101 shutdown_connections 2024-03-23T03:49:35.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.173+0000 7fe130800640 1 -- 172.21.15.102:0/227624101 wait complete. 2024-03-23T03:49:35.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.175+0000 7fe130800640 1 Processor -- start 2024-03-23T03:49:35.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.175+0000 7fe130800640 1 -- start start 2024-03-23T03:49:35.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.176+0000 7fe130800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe12806ce90 0x7fe1280b0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:35.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.176+0000 7fe130800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe1280add30 0x7fe128156ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:35.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.176+0000 7fe130800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1280ce930 0x7fe1280b2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:35.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.176+0000 7fe130800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1280aff20 con 0x7fe1280ce930 2024-03-23T03:49:35.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.176+0000 7fe130800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe128154ed0 con 0x7fe12806ce90 2024-03-23T03:49:35.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.176+0000 7fe130800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1281551b0 con 0x7fe1280add30 2024-03-23T03:49:35.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.176+0000 7fe12e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1280ce930 0x7fe1280b2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:35.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.176+0000 7fe12e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1280ce930 0x7fe1280b2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49776/0 (socket says 172.21.15.102:49776) 2024-03-23T03:49:35.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.176+0000 7fe12e800640 1 -- 172.21.15.102:0/853481244 learned_addr learned my addr 172.21.15.102:0/853481244 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:35.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.177+0000 7fe12d400640 1 --2- 172.21.15.102:0/853481244 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe1280add30 0x7fe128156ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:35.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.177+0000 7fe12de00640 1 --2- 172.21.15.102:0/853481244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe12806ce90 0x7fe1280b0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:35.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.177+0000 7fe12e800640 1 -- 172.21.15.102:0/853481244 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe1280add30 msgr2=0x7fe128156ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:35.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.177+0000 7fe12e800640 1 --2- 172.21.15.102:0/853481244 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe1280add30 0x7fe128156ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.177+0000 7fe12e800640 1 -- 172.21.15.102:0/853481244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe12806ce90 msgr2=0x7fe1280b0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:35.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.177+0000 7fe12e800640 1 --2- 172.21.15.102:0/853481244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe12806ce90 0x7fe1280b0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.177+0000 7fe12e800640 1 -- 172.21.15.102:0/853481244 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe124009b30 con 0x7fe1280ce930 2024-03-23T03:49:35.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.177+0000 7fe12e800640 1 --2- 172.21.15.102:0/853481244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1280ce930 0x7fe1280b2710 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe124009e50 tx=0x7fe124004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:35.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.177+0000 7fe122000640 1 -- 172.21.15.102:0/853481244 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe124009510 con 0x7fe1280ce930 2024-03-23T03:49:35.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.178+0000 7fe122000640 1 -- 172.21.15.102:0/853481244 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12403e450 con 0x7fe1280ce930 2024-03-23T03:49:35.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.178+0000 7fe130800640 1 -- 172.21.15.102:0/853481244 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe128155490 con 0x7fe1280ce930 2024-03-23T03:49:35.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.178+0000 7fe122000640 1 -- 172.21.15.102:0/853481244 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe12403e800 con 0x7fe1280ce930 2024-03-23T03:49:35.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.178+0000 7fe130800640 1 -- 172.21.15.102:0/853481244 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe128155a50 con 0x7fe1280ce930 2024-03-23T03:49:35.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.179+0000 7fe130800640 1 -- 172.21.15.102:0/853481244 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe12815df90 con 0x7fe1280ce930 2024-03-23T03:49:35.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.180+0000 7fe122000640 1 -- 172.21.15.102:0/853481244 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe12403e960 con 0x7fe1280ce930 2024-03-23T03:49:35.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.181+0000 7fe122000640 1 --2- 172.21.15.102:0/853481244 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe10c06ac00 0x7fe10c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:35.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.181+0000 7fe122000640 1 -- 172.21.15.102:0/853481244 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5929+0+0 (secure 0 0 0) 0x7fe1240ac5a0 con 0x7fe1280ce930 2024-03-23T03:49:35.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.181+0000 7fe12de00640 1 --2- 172.21.15.102:0/853481244 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe10c06ac00 0x7fe10c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:35.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.182+0000 7fe12de00640 1 --2- 172.21.15.102:0/853481244 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe10c06ac00 0x7fe10c06d0c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe11c000f60 tx=0x7fe11c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:35.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.185+0000 7fe122000640 1 -- 172.21.15.102:0/853481244 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe1240777a0 con 0x7fe1280ce930 2024-03-23T03:49:35.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.413+0000 7fe130800640 1 -- 172.21.15.102:0/853481244 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe128073390 con 0x7fe1280ce930 2024-03-23T03:49:35.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.421+0000 7fe122000640 1 -- 172.21.15.102:0/853481244 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v29) v1 ==== 173+0+0 (secure 0 0 0) 0x7fe12407adc0 con 0x7fe1280ce930 2024-03-23T03:49:35.421 INFO:teuthology.orchestra.run.smithi102.stderr:new fs with metadata pool 9 and data pool 10 2024-03-23T03:49:35.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.424+0000 7fe130800640 1 -- 172.21.15.102:0/853481244 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe10c06ac00 msgr2=0x7fe10c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:35.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.424+0000 7fe130800640 1 --2- 172.21.15.102:0/853481244 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe10c06ac00 0x7fe10c06d0c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe11c000f60 tx=0x7fe11c009040 comp rx=0 tx=0).stop 2024-03-23T03:49:35.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.424+0000 7fe130800640 1 -- 172.21.15.102:0/853481244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1280ce930 msgr2=0x7fe1280b2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:35.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.424+0000 7fe130800640 1 --2- 172.21.15.102:0/853481244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1280ce930 0x7fe1280b2710 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fe124009e50 tx=0x7fe124004a40 comp rx=0 tx=0).stop 2024-03-23T03:49:35.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.424+0000 7fe130800640 1 -- 172.21.15.102:0/853481244 shutdown_connections 2024-03-23T03:49:35.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.424+0000 7fe130800640 1 --2- 172.21.15.102:0/853481244 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe10c06ac00 0x7fe10c06d0c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.424+0000 7fe130800640 1 --2- 172.21.15.102:0/853481244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1280ce930 0x7fe1280b2710 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.424+0000 7fe130800640 1 --2- 172.21.15.102:0/853481244 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe1280add30 0x7fe128156ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.424+0000 7fe130800640 1 --2- 172.21.15.102:0/853481244 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe12806ce90 0x7fe1280b0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.424+0000 7fe130800640 1 -- 172.21.15.102:0/853481244 >> 172.21.15.102:0/853481244 conn(0x7fe12806a0b0 msgr2=0x7fe12815a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:35.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.425+0000 7fe130800640 1 -- 172.21.15.102:0/853481244 shutdown_connections 2024-03-23T03:49:35.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.425+0000 7fe130800640 1 -- 172.21.15.102:0/853481244 wait complete. 2024-03-23T03:49:35.453 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:49:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.632+0000 7f67b7200640 1 Processor -- start 2024-03-23T03:49:35.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.632+0000 7f67b7200640 1 -- start start 2024-03-23T03:49:35.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.633+0000 7f67b7200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f67b8070290 0x7f67b80b33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:35.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.633+0000 7f67b7200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f67b815d1f0 0x7f67b813eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:35.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.633+0000 7f67b7200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b80b2f50 con 0x7f67b8070670 2024-03-23T03:49:35.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.633+0000 7f67b7200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67b815d680 con 0x7f67b8070290 2024-03-23T03:49:35.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.633+0000 7f67b7200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b80b3920 con 0x7f67b815d1f0 2024-03-23T03:49:35.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.634+0000 7f67b6800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f67b815d1f0 0x7f67b813eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:35.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.634+0000 7f67b5e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f67b8070290 0x7f67b80b33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:35.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.634+0000 7f67b5e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f67b8070290 0x7f67b80b33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:35808/0 (socket says 172.21.15.102:35808) 2024-03-23T03:49:35.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.634+0000 7f67b5e00640 1 -- 172.21.15.102:0/2708041634 learned_addr learned my addr 172.21.15.102:0/2708041634 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.635+0000 7f67b5e00640 1 -- 172.21.15.102:0/2708041634 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f67b815d1f0 msgr2=0x7f67b813eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.635+0000 7f67b5e00640 1 --2- 172.21.15.102:0/2708041634 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f67b815d1f0 0x7f67b813eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.635+0000 7f67b5e00640 1 -- 172.21.15.102:0/2708041634 >> v1:172.21.15.173:6789/0 conn(0x7f67b8070670 legacy=0x7f67b815cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.635+0000 7f67b5e00640 1 -- 172.21.15.102:0/2708041634 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b80b2620 con 0x7f67b8070290 2024-03-23T03:49:35.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.635+0000 7f67b6800640 1 --2- 172.21.15.102:0/2708041634 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f67b815d1f0 0x7f67b813eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:49:35.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.636+0000 7f67b5e00640 1 --2- 172.21.15.102:0/2708041634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67b8070290 0x7f67b80b33e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f67a8009870 tx=0x7f67a802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ef559f1dff53423 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:35.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.637+0000 7f67b4a00640 1 -- 172.21.15.102:0/2708041634 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a8030840 con 0x7f67b8070290 2024-03-23T03:49:35.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.637+0000 7f67b4a00640 1 -- 172.21.15.102:0/2708041634 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a80309a0 con 0x7f67b8070290 2024-03-23T03:49:35.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.637+0000 7f67b4a00640 1 -- 172.21.15.102:0/2708041634 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a8030d30 con 0x7f67b8070290 2024-03-23T03:49:35.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.637+0000 7f67b7200640 1 -- 172.21.15.102:0/2708041634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67b8070290 msgr2=0x7f67b80b33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:35.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.637+0000 7f67b7200640 1 --2- 172.21.15.102:0/2708041634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67b8070290 0x7f67b80b33e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f67a8009870 tx=0x7f67a802b980 comp rx=0 tx=0).stop 2024-03-23T03:49:35.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.638+0000 7f67b7200640 1 -- 172.21.15.102:0/2708041634 shutdown_connections 2024-03-23T03:49:35.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.638+0000 7f67b7200640 1 --2- 172.21.15.102:0/2708041634 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f67b815d1f0 0x7f67b813eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.638+0000 7f67b7200640 1 --2- 172.21.15.102:0/2708041634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67b8070290 0x7f67b80b33e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.638+0000 7f67b7200640 1 -- 172.21.15.102:0/2708041634 >> 172.21.15.102:0/2708041634 conn(0x7f67b8065df0 msgr2=0x7f67b8069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:35.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.638+0000 7f67b7200640 1 -- 172.21.15.102:0/2708041634 shutdown_connections 2024-03-23T03:49:35.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.638+0000 7f67b7200640 1 -- 172.21.15.102:0/2708041634 wait complete. 2024-03-23T03:49:35.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.639+0000 7f67b7200640 1 Processor -- start 2024-03-23T03:49:35.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.639+0000 7f67b7200640 1 -- start start 2024-03-23T03:49:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.640+0000 7f67b7200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67b8070290 0x7f67b8135eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.640+0000 7f67b7200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67b8070670 0x7f67b8136640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.640+0000 7f67b7200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f67b815d1f0 0x7f67b80ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.640+0000 7f67b7200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b815c090 con 0x7f67b8070290 2024-03-23T03:49:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.640+0000 7f67b7200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b814a330 con 0x7f67b8070670 2024-03-23T03:49:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.640+0000 7f67b7200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b81363f0 con 0x7f67b815d1f0 2024-03-23T03:49:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.640+0000 7f67b5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67b8070290 0x7f67b8135eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:35.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.640+0000 7f67b5e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67b8070290 0x7f67b8135eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49790/0 (socket says 172.21.15.102:49790) 2024-03-23T03:49:35.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.641+0000 7f67b5e00640 1 -- 172.21.15.102:0/2762628399 learned_addr learned my addr 172.21.15.102:0/2762628399 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.641+0000 7f67b6800640 1 --2- 172.21.15.102:0/2762628399 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f67b815d1f0 0x7f67b80ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.641+0000 7f67b5400640 1 --2- 172.21.15.102:0/2762628399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67b8070670 0x7f67b8136640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.641+0000 7f67b5e00640 1 -- 172.21.15.102:0/2762628399 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f67b815d1f0 msgr2=0x7f67b80ccb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.641+0000 7f67b5e00640 1 --2- 172.21.15.102:0/2762628399 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f67b815d1f0 0x7f67b80ccb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.641+0000 7f67b5e00640 1 -- 172.21.15.102:0/2762628399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67b8070670 msgr2=0x7f67b8136640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.641+0000 7f67b5e00640 1 --2- 172.21.15.102:0/2762628399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67b8070670 0x7f67b8136640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.641+0000 7f67b5e00640 1 -- 172.21.15.102:0/2762628399 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b0009a70 con 0x7f67b8070290 2024-03-23T03:49:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.641+0000 7f67b6800640 1 --2- 172.21.15.102:0/2762628399 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f67b815d1f0 0x7f67b80ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.641+0000 7f67b5e00640 1 --2- 172.21.15.102:0/2762628399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67b8070290 0x7f67b8135eb0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f67a8000c00 tx=0x7f67a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.642+0000 7f67ae000640 1 -- 172.21.15.102:0/2762628399 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a8039a90 con 0x7f67b8070290 2024-03-23T03:49:35.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.642+0000 7f67ae000640 1 -- 172.21.15.102:0/2762628399 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a8039bf0 con 0x7f67b8070290 2024-03-23T03:49:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.642+0000 7f67ae000640 1 -- 172.21.15.102:0/2762628399 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a8043410 con 0x7f67b8070290 2024-03-23T03:49:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.642+0000 7f67b7200640 1 -- 172.21.15.102:0/2762628399 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a80094d0 con 0x7f67b8070290 2024-03-23T03:49:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.642+0000 7f67b7200640 1 -- 172.21.15.102:0/2762628399 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b814a050 con 0x7f67b8070290 2024-03-23T03:49:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.643+0000 7f67b7200640 1 -- 172.21.15.102:0/2762628399 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67b804e030 con 0x7f67b8070290 2024-03-23T03:49:35.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.644+0000 7f67ae000640 1 -- 172.21.15.102:0/2762628399 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f67a802f390 con 0x7f67b8070290 2024-03-23T03:49:35.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.645+0000 7f67ae000640 1 --2- 172.21.15.102:0/2762628399 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f679806ac00 0x7f679806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:35.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.645+0000 7f67ae000640 1 -- 172.21.15.102:0/2762628399 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f67a80ab500 con 0x7f67b8070290 2024-03-23T03:49:35.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.645+0000 7f67b5400640 1 --2- 172.21.15.102:0/2762628399 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f679806ac00 0x7f679806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:35.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.646+0000 7f67b5400640 1 --2- 172.21.15.102:0/2762628399 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f679806ac00 0x7f679806d0c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f67b813d7b0 tx=0x7f67a4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:35.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.649+0000 7f67ae000640 1 -- 172.21.15.102:0/2762628399 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f67a8076750 con 0x7f67b8070290 2024-03-23T03:49:35.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.836+0000 7f67b7200640 1 -- 172.21.15.102:0/2762628399 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67b80cbbe0 con 0x7f67b8070290 2024-03-23T03:49:35.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.839+0000 7f67ae000640 1 -- 172.21.15.102:0/2762628399 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16309 (secure 0 0 0) 0x7f67a8079d70 con 0x7f67b8070290 2024-03-23T03:49:35.840 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:35.840 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":66,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:49:35.415060+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:49:33.007757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-23T03:49:34.449411+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:49:35.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.842+0000 7f67b7200640 1 -- 172.21.15.102:0/2762628399 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f679806ac00 msgr2=0x7f679806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:35.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.842+0000 7f67b7200640 1 --2- 172.21.15.102:0/2762628399 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f679806ac00 0x7f679806d0c0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f67b813d7b0 tx=0x7f67a4009040 comp rx=0 tx=0).stop 2024-03-23T03:49:35.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.842+0000 7f67b7200640 1 -- 172.21.15.102:0/2762628399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67b8070290 msgr2=0x7f67b8135eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:35.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.842+0000 7f67b7200640 1 --2- 172.21.15.102:0/2762628399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67b8070290 0x7f67b8135eb0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f67a8000c00 tx=0x7f67a8004730 comp rx=0 tx=0).stop 2024-03-23T03:49:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.842+0000 7f67b7200640 1 -- 172.21.15.102:0/2762628399 shutdown_connections 2024-03-23T03:49:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.842+0000 7f67b7200640 1 --2- 172.21.15.102:0/2762628399 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f679806ac00 0x7f679806d0c0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.842+0000 7f67b7200640 1 --2- 172.21.15.102:0/2762628399 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f67b815d1f0 0x7f67b80ccb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.842+0000 7f67b7200640 1 --2- 172.21.15.102:0/2762628399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f67b8070670 0x7f67b8136640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.842+0000 7f67b7200640 1 --2- 172.21.15.102:0/2762628399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67b8070290 0x7f67b8135eb0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.842+0000 7f67b7200640 1 -- 172.21.15.102:0/2762628399 >> 172.21.15.102:0/2762628399 conn(0x7f67b8065df0 msgr2=0x7f67b80ae0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:35.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.842+0000 7f67b7200640 1 -- 172.21.15.102:0/2762628399 shutdown_connections 2024-03-23T03:49:35.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:35.842+0000 7f67b7200640 1 -- 172.21.15.102:0/2762628399 wait complete. 2024-03-23T03:49:35.860 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:49:36.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.044+0000 7f6178e00640 1 Processor -- start 2024-03-23T03:49:36.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.044+0000 7f6178e00640 1 -- start start 2024-03-23T03:49:36.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.046+0000 7f6178e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f617415b560 0x7f617415b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:36.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.046+0000 7f6178e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f617415d410 0x7f61740b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:36.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.046+0000 7f6178e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617406c6f0 con 0x7f61740b39b0 2024-03-23T03:49:36.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.046+0000 7f6178e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61740b4370 con 0x7f617415b560 2024-03-23T03:49:36.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.046+0000 7f6178e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617406c9d0 con 0x7f617415d410 2024-03-23T03:49:36.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.046+0000 7f6171e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f617415b560 0x7f617415b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:36.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.046+0000 7f6171e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f617415b560 0x7f617415b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49792/0 (socket says 172.21.15.102:49792) 2024-03-23T03:49:36.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.046+0000 7f6169400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f61740b39b0 0x7f617415e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50614/0 (socket says 172.21.15.102:50614) 2024-03-23T03:49:36.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.046+0000 7f6171e00640 1 -- 172.21.15.102:0/689189722 learned_addr learned my addr 172.21.15.102:0/689189722 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:36.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.046+0000 7f6172800640 1 --2- 172.21.15.102:0/689189722 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f617415d410 0x7f61740b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:36.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.046+0000 7f6171e00640 1 -- 172.21.15.102:0/689189722 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f617415d410 msgr2=0x7f61740b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:36.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.046+0000 7f6171e00640 1 --2- 172.21.15.102:0/689189722 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f617415d410 0x7f61740b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:36.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.046+0000 7f6171e00640 1 -- 172.21.15.102:0/689189722 >> v1:172.21.15.102:6789/0 conn(0x7f61740b39b0 legacy=0x7f617415e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:36.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.046+0000 7f6171e00640 1 -- 172.21.15.102:0/689189722 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617406ccb0 con 0x7f617415b560 2024-03-23T03:49:36.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.048+0000 7f6171e00640 1 --2- 172.21.15.102:0/689189722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f617415b560 0x7f617415b940 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6164009870 tx=0x7f616402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83dc0e0938ccfc66 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:36.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.048+0000 7f6171400640 1 -- 172.21.15.102:0/689189722 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6164030840 con 0x7f617415b560 2024-03-23T03:49:36.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.048+0000 7f6171400640 1 -- 172.21.15.102:0/689189722 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61640309a0 con 0x7f617415b560 2024-03-23T03:49:36.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.048+0000 7f6171400640 1 -- 172.21.15.102:0/689189722 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6164030d30 con 0x7f617415b560 2024-03-23T03:49:36.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.049+0000 7f6178e00640 1 -- 172.21.15.102:0/689189722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f617415b560 msgr2=0x7f617415b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:36.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.049+0000 7f6178e00640 1 --2- 172.21.15.102:0/689189722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f617415b560 0x7f617415b940 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f6164009870 tx=0x7f616402b980 comp rx=0 tx=0).stop 2024-03-23T03:49:36.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.049+0000 7f6178e00640 1 -- 172.21.15.102:0/689189722 shutdown_connections 2024-03-23T03:49:36.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.049+0000 7f6178e00640 1 --2- 172.21.15.102:0/689189722 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f617415d410 0x7f61740b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:36.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.049+0000 7f6178e00640 1 --2- 172.21.15.102:0/689189722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f617415b560 0x7f617415b940 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:36.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.049+0000 7f6178e00640 1 -- 172.21.15.102:0/689189722 >> 172.21.15.102:0/689189722 conn(0x7f617406a3d0 msgr2=0x7f61740b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:36.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.049+0000 7f6178e00640 1 -- 172.21.15.102:0/689189722 shutdown_connections 2024-03-23T03:49:36.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.049+0000 7f6178e00640 1 -- 172.21.15.102:0/689189722 wait complete. 2024-03-23T03:49:36.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.050+0000 7f6178e00640 1 Processor -- start 2024-03-23T03:49:36.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.051+0000 7f6178e00640 1 -- start start 2024-03-23T03:49:36.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.051+0000 7f6178e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f61740b39b0 0x7f61740b2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:36.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.051+0000 7f6178e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f617415b560 0x7f6174141c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:36.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.051+0000 7f6178e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f617415d410 0x7f6174142150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:36.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.051+0000 7f6178e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61741563c0 con 0x7f617415b560 2024-03-23T03:49:36.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.051+0000 7f6178e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61741f5ab0 con 0x7f617415d410 2024-03-23T03:49:36.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.051+0000 7f6178e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61741f5d90 con 0x7f61740b39b0 2024-03-23T03:49:36.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.052+0000 7f6169400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f617415b560 0x7f6174141c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:36.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.052+0000 7f6169400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f617415b560 0x7f6174141c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49802/0 (socket says 172.21.15.102:49802) 2024-03-23T03:49:36.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.052+0000 7f6169400640 1 -- 172.21.15.102:0/2983417780 learned_addr learned my addr 172.21.15.102:0/2983417780 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:36.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.052+0000 7f6169400640 1 -- 172.21.15.102:0/2983417780 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f61740b39b0 msgr2=0x7f61740b2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:36.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.052+0000 7f6169400640 1 --2- 172.21.15.102:0/2983417780 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f61740b39b0 0x7f61740b2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:36.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.052+0000 7f6169400640 1 -- 172.21.15.102:0/2983417780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f617415d410 msgr2=0x7f6174142150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:36.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.052+0000 7f6172800640 1 --2- 172.21.15.102:0/2983417780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f617415d410 0x7f6174142150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:36.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.052+0000 7f6169400640 1 --2- 172.21.15.102:0/2983417780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f617415d410 0x7f6174142150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:36.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.052+0000 7f6169400640 1 -- 172.21.15.102:0/2983417780 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61640094d0 con 0x7f617415b560 2024-03-23T03:49:36.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.052+0000 7f6169400640 1 --2- 172.21.15.102:0/2983417780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f617415b560 0x7f6174141c10 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f615c00c4d0 tx=0x7f615c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:36.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.053+0000 7f616aa00640 1 -- 172.21.15.102:0/2983417780 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c007480 con 0x7f617415b560 2024-03-23T03:49:36.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.053+0000 7f616aa00640 1 -- 172.21.15.102:0/2983417780 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f615c0075e0 con 0x7f617415b560 2024-03-23T03:49:36.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.053+0000 7f616aa00640 1 -- 172.21.15.102:0/2983417780 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c0049e0 con 0x7f617415b560 2024-03-23T03:49:36.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.053+0000 7f6178e00640 1 -- 172.21.15.102:0/2983417780 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61741f6070 con 0x7f617415b560 2024-03-23T03:49:36.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.053+0000 7f6178e00640 1 -- 172.21.15.102:0/2983417780 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61741f6630 con 0x7f617415b560 2024-03-23T03:49:36.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.054+0000 7f6178e00640 1 -- 172.21.15.102:0/2983417780 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617406ac20 con 0x7f617415b560 2024-03-23T03:49:36.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.055+0000 7f616aa00640 1 -- 172.21.15.102:0/2983417780 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f615c004b40 con 0x7f617415b560 2024-03-23T03:49:36.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.056+0000 7f616aa00640 1 --2- 172.21.15.102:0/2983417780 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f615006ac00 0x7f615006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:36.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.056+0000 7f616aa00640 1 -- 172.21.15.102:0/2983417780 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f615c088440 con 0x7f617415b560 2024-03-23T03:49:36.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.056+0000 7f6171e00640 1 --2- 172.21.15.102:0/2983417780 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f615006ac00 0x7f615006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:36.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.057+0000 7f6171e00640 1 --2- 172.21.15.102:0/2983417780 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f615006ac00 0x7f615006d0c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6164036000 tx=0x7f616403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:36.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.060+0000 7f616aa00640 1 -- 172.21.15.102:0/2983417780 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f615c053620 con 0x7f617415b560 2024-03-23T03:49:36.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.250+0000 7f6178e00640 1 -- 172.21.15.102:0/2983417780 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6174073390 con 0x7f617415b560 2024-03-23T03:49:36.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.254+0000 7f616aa00640 1 -- 172.21.15.102:0/2983417780 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+16309 (secure 0 0 0) 0x7f615c056c40 con 0x7f617415b560 2024-03-23T03:49:36.254 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:36.255 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":66,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:49:35.415060+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:49:33.007757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-23T03:49:34.449411+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:49:36.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.256+0000 7f6178e00640 1 -- 172.21.15.102:0/2983417780 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f615006ac00 msgr2=0x7f615006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:36.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.256+0000 7f6178e00640 1 --2- 172.21.15.102:0/2983417780 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f615006ac00 0x7f615006d0c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f6164036000 tx=0x7f616403b040 comp rx=0 tx=0).stop 2024-03-23T03:49:36.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.256+0000 7f6178e00640 1 -- 172.21.15.102:0/2983417780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f617415b560 msgr2=0x7f6174141c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:36.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.256+0000 7f6178e00640 1 --2- 172.21.15.102:0/2983417780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f617415b560 0x7f6174141c10 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f615c00c4d0 tx=0x7f615c00c990 comp rx=0 tx=0).stop 2024-03-23T03:49:36.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.256+0000 7f6178e00640 1 -- 172.21.15.102:0/2983417780 shutdown_connections 2024-03-23T03:49:36.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.256+0000 7f6178e00640 1 --2- 172.21.15.102:0/2983417780 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f615006ac00 0x7f615006d0c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:36.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.257+0000 7f6178e00640 1 --2- 172.21.15.102:0/2983417780 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f617415d410 0x7f6174142150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:36.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.257+0000 7f6178e00640 1 --2- 172.21.15.102:0/2983417780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f617415b560 0x7f6174141c10 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:36.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.257+0000 7f6178e00640 1 --2- 172.21.15.102:0/2983417780 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f61740b39b0 0x7f61740b2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:36.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.257+0000 7f6178e00640 1 -- 172.21.15.102:0/2983417780 >> 172.21.15.102:0/2983417780 conn(0x7f617406a3d0 msgr2=0x7f6174143610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:36.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.257+0000 7f6178e00640 1 -- 172.21.15.102:0/2983417780 shutdown_connections 2024-03-23T03:49:36.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.257+0000 7f6178e00640 1 -- 172.21.15.102:0/2983417780 wait complete. 2024-03-23T03:49:36.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T03:49:36.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.475+0000 7fab41800640 1 Processor -- start 2024-03-23T03:49:36.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.475+0000 7fab41800640 1 -- start start 2024-03-23T03:49:36.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.477+0000 7fab41800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3c1456a0 0x7fab3c0ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:36.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.477+0000 7fab41800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fab3c145c20 0x7fab3c0cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:36.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.477+0000 7fab41800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab3c06d390 con 0x7fab3c070cd0 2024-03-23T03:49:36.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.477+0000 7fab41800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab3c0ad890 con 0x7fab3c145c20 2024-03-23T03:49:36.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.477+0000 7fab41800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c145a80 con 0x7fab3c1456a0 2024-03-23T03:49:36.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.477+0000 7fab3b400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fab3c145c20 0x7fab3c0cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:36.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.477+0000 7fab3aa00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3c1456a0 0x7fab3c0ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:36.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.478+0000 7fab3aa00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3c1456a0 0x7fab3c0ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:47344/0 (socket says 172.21.15.102:47344) 2024-03-23T03:49:36.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.478+0000 7fab3b400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fab3c145c20 0x7fab3c0cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:35844/0 (socket says 172.21.15.102:35844) 2024-03-23T03:49:36.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.478+0000 7fab3aa00640 1 -- 172.21.15.102:0/2629027058 learned_addr learned my addr 172.21.15.102:0/2629027058 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:36.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.478+0000 7fab3b400640 1 -- 172.21.15.102:0/2629027058 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3c1456a0 msgr2=0x7fab3c0ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:36.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.478+0000 7fab3b400640 1 --2- 172.21.15.102:0/2629027058 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3c1456a0 0x7fab3c0ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:36.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.478+0000 7fab3b400640 1 -- 172.21.15.102:0/2629027058 >> v1:172.21.15.173:6789/0 conn(0x7fab3c070cd0 legacy=0x7fab3c0ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:36.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.478+0000 7fab3b400640 1 -- 172.21.15.102:0/2629027058 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3c06d670 con 0x7fab3c145c20 2024-03-23T03:49:36.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.478+0000 7fab3aa00640 1 --2- 172.21.15.102:0/2629027058 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3c1456a0 0x7fab3c0ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:49:36.480 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.480+0000 7fab3b400640 1 --2- 172.21.15.102:0/2629027058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab3c145c20 0x7fab3c0cee70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fab34009dc0 tx=0x7fab3402be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8a21174edcb491c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:36.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.480+0000 7fab3a000640 1 -- 172.21.15.102:0/2629027058 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab3402cc40 con 0x7fab3c145c20 2024-03-23T03:49:36.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.480+0000 7fab3a000640 1 -- 172.21.15.102:0/2629027058 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3402cda0 con 0x7fab3c145c20 2024-03-23T03:49:36.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.480+0000 7fab3a000640 1 -- 172.21.15.102:0/2629027058 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34034660 con 0x7fab3c145c20 2024-03-23T03:49:36.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.481+0000 7fab41800640 1 -- 172.21.15.102:0/2629027058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab3c145c20 msgr2=0x7fab3c0cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:36.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.481+0000 7fab41800640 1 --2- 172.21.15.102:0/2629027058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab3c145c20 0x7fab3c0cee70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fab34009dc0 tx=0x7fab3402be90 comp rx=0 tx=0).stop 2024-03-23T03:49:36.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.481+0000 7fab41800640 1 -- 172.21.15.102:0/2629027058 shutdown_connections 2024-03-23T03:49:36.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.481+0000 7fab41800640 1 --2- 172.21.15.102:0/2629027058 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab3c145c20 0x7fab3c0cee70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:36.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.481+0000 7fab41800640 1 --2- 172.21.15.102:0/2629027058 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3c1456a0 0x7fab3c0ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:36.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.481+0000 7fab41800640 1 -- 172.21.15.102:0/2629027058 >> 172.21.15.102:0/2629027058 conn(0x7fab3c06a0b0 msgr2=0x7fab3c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:36.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.481+0000 7fab41800640 1 -- 172.21.15.102:0/2629027058 shutdown_connections 2024-03-23T03:49:36.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.481+0000 7fab41800640 1 -- 172.21.15.102:0/2629027058 wait complete. 2024-03-23T03:49:36.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.483+0000 7fab41800640 1 Processor -- start 2024-03-23T03:49:36.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.483+0000 7fab41800640 1 -- start start 2024-03-23T03:49:36.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.484+0000 7fab41800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3c070cd0 0x7fab3c13c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:36.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.484+0000 7fab41800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1456a0 0x7fab3c15b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:36.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.484+0000 7fab41800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab3c145c20 0x7fab3c15bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:36.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.484+0000 7fab41800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c0afa30 con 0x7fab3c1456a0 2024-03-23T03:49:36.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.484+0000 7fab41800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c0b2df0 con 0x7fab3c145c20 2024-03-23T03:49:36.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.484+0000 7fab41800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c0afc80 con 0x7fab3c070cd0 2024-03-23T03:49:36.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.484+0000 7fab33e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1456a0 0x7fab3c15b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:36.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.484+0000 7fab33e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1456a0 0x7fab3c15b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49806/0 (socket says 172.21.15.102:49806) 2024-03-23T03:49:36.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.484+0000 7fab33e00640 1 -- 172.21.15.102:0/1472998502 learned_addr learned my addr 172.21.15.102:0/1472998502 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:36.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.484+0000 7fab3aa00640 1 --2- 172.21.15.102:0/1472998502 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3c070cd0 0x7fab3c13c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:36.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.484+0000 7fab33e00640 1 -- 172.21.15.102:0/1472998502 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3c070cd0 msgr2=0x7fab3c13c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:36.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.484+0000 7fab33e00640 1 --2- 172.21.15.102:0/1472998502 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3c070cd0 0x7fab3c13c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:36.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.484+0000 7fab33e00640 1 -- 172.21.15.102:0/1472998502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab3c145c20 msgr2=0x7fab3c15bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:36.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.484+0000 7fab33e00640 1 --2- 172.21.15.102:0/1472998502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab3c145c20 0x7fab3c15bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:36.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.484+0000 7fab33e00640 1 -- 172.21.15.102:0/1472998502 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2c0094d0 con 0x7fab3c1456a0 2024-03-23T03:49:36.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.485+0000 7fab33e00640 1 --2- 172.21.15.102:0/1472998502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1456a0 0x7fab3c15b690 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fab2400c400 tx=0x7fab2400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:36.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.485+0000 7fab32a00640 1 -- 172.21.15.102:0/1472998502 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab240052d0 con 0x7fab3c1456a0 2024-03-23T03:49:36.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.485+0000 7fab32a00640 1 -- 172.21.15.102:0/1472998502 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab24005430 con 0x7fab3c1456a0 2024-03-23T03:49:36.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.485+0000 7fab32a00640 1 -- 172.21.15.102:0/1472998502 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab24003830 con 0x7fab3c1456a0 2024-03-23T03:49:36.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.485+0000 7fab41800640 1 -- 172.21.15.102:0/1472998502 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab34009a70 con 0x7fab3c1456a0 2024-03-23T03:49:36.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.485+0000 7fab41800640 1 -- 172.21.15.102:0/1472998502 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3c0b0240 con 0x7fab3c1456a0 2024-03-23T03:49:36.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.486+0000 7fab41800640 1 -- 172.21.15.102:0/1472998502 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3c073390 con 0x7fab3c1456a0 2024-03-23T03:49:36.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.487+0000 7fab32a00640 1 -- 172.21.15.102:0/1472998502 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fab24003990 con 0x7fab3c1456a0 2024-03-23T03:49:36.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.488+0000 7fab32a00640 1 --2- 172.21.15.102:0/1472998502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab1806ac30 0x7fab1806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:36.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.488+0000 7fab32a00640 1 -- 172.21.15.102:0/1472998502 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fab24088a40 con 0x7fab3c1456a0 2024-03-23T03:49:36.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.488+0000 7fab3aa00640 1 --2- 172.21.15.102:0/1472998502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab1806ac30 0x7fab1806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:36.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.489+0000 7fab3aa00640 1 --2- 172.21.15.102:0/1472998502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab1806ac30 0x7fab1806d0f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fab3c0b4820 tx=0x7fab2c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:36.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.493+0000 7fab32a00640 1 -- 172.21.15.102:0/1472998502 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fab2408ae60 con 0x7fab3c1456a0 2024-03-23T03:49:36.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:36.730+0000 7fab41800640 1 -- 172.21.15.102:0/1472998502 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fab3c0cdee0 con 0x7fab3c1456a0 2024-03-23T03:49:37.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.437+0000 7fab32a00640 1 -- 172.21.15.102:0/1472998502 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v32) v1 ==== 118+0+0 (secure 0 0 0) 0x7fab24053ae0 con 0x7fab3c1456a0 2024-03-23T03:49:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.440+0000 7fab41800640 1 -- 172.21.15.102:0/1472998502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab1806ac30 msgr2=0x7fab1806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.440+0000 7fab41800640 1 --2- 172.21.15.102:0/1472998502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab1806ac30 0x7fab1806d0f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fab3c0b4820 tx=0x7fab2c008040 comp rx=0 tx=0).stop 2024-03-23T03:49:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.440+0000 7fab41800640 1 -- 172.21.15.102:0/1472998502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1456a0 msgr2=0x7fab3c15b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:37.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.440+0000 7fab41800640 1 --2- 172.21.15.102:0/1472998502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1456a0 0x7fab3c15b690 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fab2400c400 tx=0x7fab2400c8c0 comp rx=0 tx=0).stop 2024-03-23T03:49:37.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.440+0000 7fab41800640 1 -- 172.21.15.102:0/1472998502 shutdown_connections 2024-03-23T03:49:37.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.440+0000 7fab41800640 1 --2- 172.21.15.102:0/1472998502 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab1806ac30 0x7fab1806d0f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:37.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.440+0000 7fab41800640 1 --2- 172.21.15.102:0/1472998502 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fab3c145c20 0x7fab3c15bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:37.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.440+0000 7fab41800640 1 --2- 172.21.15.102:0/1472998502 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab3c1456a0 0x7fab3c15b690 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:37.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.440+0000 7fab41800640 1 --2- 172.21.15.102:0/1472998502 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fab3c070cd0 0x7fab3c13c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:37.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.440+0000 7fab41800640 1 -- 172.21.15.102:0/1472998502 >> 172.21.15.102:0/1472998502 conn(0x7fab3c06a0b0 msgr2=0x7fab3c0acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:37.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.441+0000 7fab41800640 1 -- 172.21.15.102:0/1472998502 shutdown_connections 2024-03-23T03:49:37.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.441+0000 7fab41800640 1 -- 172.21.15.102:0/1472998502 wait complete. 2024-03-23T03:49:37.461 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-23T03:49:37.461 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T03:49:37.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.672+0000 7fc189000640 1 Processor -- start 2024-03-23T03:49:37.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.672+0000 7fc189000640 1 -- start start 2024-03-23T03:49:37.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.674+0000 7fc189000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc1840cf130 0x7fc1840cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:37.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.674+0000 7fc189000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc1840cdb30 0x7fc1840d1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:37.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.674+0000 7fc189000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1840cf730 con 0x7fc18415c980 2024-03-23T03:49:37.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.674+0000 7fc189000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1840cdfa0 con 0x7fc1840cdb30 2024-03-23T03:49:37.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.674+0000 7fc189000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1840cfa10 con 0x7fc1840cf130 2024-03-23T03:49:37.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.674+0000 7fc182800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc1840cdb30 0x7fc1840d1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:37.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.674+0000 7fc182800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc1840cdb30 0x7fc1840d1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49820/0 (socket says 172.21.15.102:49820) 2024-03-23T03:49:37.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.674+0000 7fc182800640 1 -- 172.21.15.102:0/231768627 learned_addr learned my addr 172.21.15.102:0/231768627 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:37.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.675+0000 7fc181400640 1 --2- 172.21.15.102:0/231768627 >> v2:172.21.15.173:3300/0 conn(0x7fc1840cf130 0x7fc1840cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:37.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.675+0000 7fc182800640 1 -- 172.21.15.102:0/231768627 >> v2:172.21.15.173:3300/0 conn(0x7fc1840cf130 msgr2=0x7fc1840cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:37.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.675+0000 7fc182800640 1 --2- 172.21.15.102:0/231768627 >> v2:172.21.15.173:3300/0 conn(0x7fc1840cf130 0x7fc1840cd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:37.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.675+0000 7fc182800640 1 -- 172.21.15.102:0/231768627 >> v1:172.21.15.173:6789/0 conn(0x7fc18415c980 legacy=0x7fc184161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:37.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.675+0000 7fc182800640 1 -- 172.21.15.102:0/231768627 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1840cfcf0 con 0x7fc1840cdb30 2024-03-23T03:49:37.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.676+0000 7fc182800640 1 --2- 172.21.15.102:0/231768627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1840cdb30 0x7fc1840d1240 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc18408ad90 tx=0x7fc17c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1499d332ae3f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:37.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.676+0000 7fc180a00640 1 -- 172.21.15.102:0/231768627 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c031870 con 0x7fc1840cdb30 2024-03-23T03:49:37.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.676+0000 7fc180a00640 1 -- 172.21.15.102:0/231768627 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17c0319d0 con 0x7fc1840cdb30 2024-03-23T03:49:37.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.676+0000 7fc180a00640 1 -- 172.21.15.102:0/231768627 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c031d60 con 0x7fc1840cdb30 2024-03-23T03:49:37.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.677+0000 7fc189000640 1 -- 172.21.15.102:0/231768627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1840cdb30 msgr2=0x7fc1840d1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:37.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.677+0000 7fc189000640 1 --2- 172.21.15.102:0/231768627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1840cdb30 0x7fc1840d1240 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc18408ad90 tx=0x7fc17c02b910 comp rx=0 tx=0).stop 2024-03-23T03:49:37.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.677+0000 7fc189000640 1 -- 172.21.15.102:0/231768627 shutdown_connections 2024-03-23T03:49:37.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.677+0000 7fc189000640 1 --2- 172.21.15.102:0/231768627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1840cdb30 0x7fc1840d1240 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:37.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.677+0000 7fc189000640 1 --2- 172.21.15.102:0/231768627 >> v2:172.21.15.173:3300/0 conn(0x7fc1840cf130 0x7fc1840cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:37.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.677+0000 7fc189000640 1 -- 172.21.15.102:0/231768627 >> 172.21.15.102:0/231768627 conn(0x7fc1840713c0 msgr2=0x7fc1840717d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:37.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.677+0000 7fc189000640 1 -- 172.21.15.102:0/231768627 shutdown_connections 2024-03-23T03:49:37.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.677+0000 7fc189000640 1 -- 172.21.15.102:0/231768627 wait complete. 2024-03-23T03:49:37.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.679+0000 7fc189000640 1 Processor -- start 2024-03-23T03:49:37.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.679+0000 7fc189000640 1 -- start start 2024-03-23T03:49:37.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.680+0000 7fc189000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1840cdb30 0x7fc18414cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:37.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.680+0000 7fc189000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1840cf130 0x7fc18415d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:37.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.680+0000 7fc189000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc18415c980 0x7fc18415f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:37.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.680+0000 7fc189000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18415d940 con 0x7fc1840cdb30 2024-03-23T03:49:37.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.680+0000 7fc189000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18415dc20 con 0x7fc1840cf130 2024-03-23T03:49:37.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.680+0000 7fc189000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc18415df00 con 0x7fc18415c980 2024-03-23T03:49:37.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.680+0000 7fc181e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1840cdb30 0x7fc18414cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:37.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.680+0000 7fc181e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1840cdb30 0x7fc18414cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49828/0 (socket says 172.21.15.102:49828) 2024-03-23T03:49:37.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.680+0000 7fc181e00640 1 -- 172.21.15.102:0/694560971 learned_addr learned my addr 172.21.15.102:0/694560971 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:37.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.680+0000 7fc181e00640 1 -- 172.21.15.102:0/694560971 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc18415c980 msgr2=0x7fc18415f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:37.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.680+0000 7fc181e00640 1 --2- 172.21.15.102:0/694560971 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc18415c980 0x7fc18415f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:37.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.680+0000 7fc181e00640 1 -- 172.21.15.102:0/694560971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1840cf130 msgr2=0x7fc18415d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:37.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.680+0000 7fc181400640 1 --2- 172.21.15.102:0/694560971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1840cf130 0x7fc18415d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:37.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.680+0000 7fc181e00640 1 --2- 172.21.15.102:0/694560971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1840cf130 0x7fc18415d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:37.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.680+0000 7fc181e00640 1 -- 172.21.15.102:0/694560971 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c0094d0 con 0x7fc1840cdb30 2024-03-23T03:49:37.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.681+0000 7fc181e00640 1 --2- 172.21.15.102:0/694560971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1840cdb30 0x7fc18414cba0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc17400ba50 tx=0x7fc17400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:37.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.681+0000 7fc17a000640 1 -- 172.21.15.102:0/694560971 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17400cbe0 con 0x7fc1840cdb30 2024-03-23T03:49:37.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.681+0000 7fc17a000640 1 -- 172.21.15.102:0/694560971 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc17400cd80 con 0x7fc1840cdb30 2024-03-23T03:49:37.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.681+0000 7fc17a000640 1 -- 172.21.15.102:0/694560971 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc174014680 con 0x7fc1840cdb30 2024-03-23T03:49:37.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.681+0000 7fc189000640 1 -- 172.21.15.102:0/694560971 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc18415e1e0 con 0x7fc1840cdb30 2024-03-23T03:49:37.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.681+0000 7fc189000640 1 -- 172.21.15.102:0/694560971 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc18415e9e0 con 0x7fc1840cdb30 2024-03-23T03:49:37.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.682+0000 7fc189000640 1 -- 172.21.15.102:0/694560971 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc18415e700 con 0x7fc1840cdb30 2024-03-23T03:49:37.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.683+0000 7fc17a000640 1 -- 172.21.15.102:0/694560971 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc1740147e0 con 0x7fc1840cdb30 2024-03-23T03:49:37.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.684+0000 7fc17a000640 1 --2- 172.21.15.102:0/694560971 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc16406ac00 0x7fc16406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:37.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.684+0000 7fc17a000640 1 -- 172.21.15.102:0/694560971 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc174089420 con 0x7fc1840cdb30 2024-03-23T03:49:37.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.684+0000 7fc181400640 1 --2- 172.21.15.102:0/694560971 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc16406ac00 0x7fc16406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:37.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.685+0000 7fc181400640 1 --2- 172.21.15.102:0/694560971 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc16406ac00 0x7fc16406d0c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc1840b81c0 tx=0x7fc170008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:37.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.688+0000 7fc17a000640 1 -- 172.21.15.102:0/694560971 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc18415e700 con 0x7fc1840cdb30 2024-03-23T03:49:37.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:37.920+0000 7fc189000640 1 -- 172.21.15.102:0/694560971 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc18415e700 con 0x7fc1840cdb30 2024-03-23T03:49:38.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.447+0000 7fc17a000640 1 -- 172.21.15.102:0/694560971 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v33) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc18415e700 con 0x7fc1840cdb30 2024-03-23T03:49:38.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.451+0000 7fc189000640 1 -- 172.21.15.102:0/694560971 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc16406ac00 msgr2=0x7fc16406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:38.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.451+0000 7fc189000640 1 --2- 172.21.15.102:0/694560971 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc16406ac00 0x7fc16406d0c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fc1840b81c0 tx=0x7fc170008040 comp rx=0 tx=0).stop 2024-03-23T03:49:38.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.451+0000 7fc189000640 1 -- 172.21.15.102:0/694560971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1840cdb30 msgr2=0x7fc18414cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:38.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.451+0000 7fc189000640 1 --2- 172.21.15.102:0/694560971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1840cdb30 0x7fc18414cba0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc17400ba50 tx=0x7fc17400bf10 comp rx=0 tx=0).stop 2024-03-23T03:49:38.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.451+0000 7fc189000640 1 -- 172.21.15.102:0/694560971 shutdown_connections 2024-03-23T03:49:38.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.451+0000 7fc189000640 1 --2- 172.21.15.102:0/694560971 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc16406ac00 0x7fc16406d0c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:38.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.451+0000 7fc189000640 1 --2- 172.21.15.102:0/694560971 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc18415c980 0x7fc18415f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:38.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.451+0000 7fc189000640 1 --2- 172.21.15.102:0/694560971 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1840cf130 0x7fc18415d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:38.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.451+0000 7fc189000640 1 --2- 172.21.15.102:0/694560971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1840cdb30 0x7fc18414cba0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:38.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.451+0000 7fc189000640 1 -- 172.21.15.102:0/694560971 >> 172.21.15.102:0/694560971 conn(0x7fc1840713c0 msgr2=0x7fc184158100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:38.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.451+0000 7fc189000640 1 -- 172.21.15.102:0/694560971 shutdown_connections 2024-03-23T03:49:38.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.452+0000 7fc189000640 1 -- 172.21.15.102:0/694560971 wait complete. 2024-03-23T03:49:38.476 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T03:49:38.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:49:38.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.680+0000 7fb6ea400640 1 Processor -- start 2024-03-23T03:49:38.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.680+0000 7fb6ea400640 1 -- start start 2024-03-23T03:49:38.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.681+0000 7fb6ea400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6e415b560 0x7fb6e415b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:38.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.681+0000 7fb6ea400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb6e415d410 0x7fb6e40b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:38.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.681+0000 7fb6ea400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6e406c6f0 con 0x7fb6e40b39b0 2024-03-23T03:49:38.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.681+0000 7fb6ea400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6e40b4370 con 0x7fb6e415d410 2024-03-23T03:49:38.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.681+0000 7fb6ea400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e406c9d0 con 0x7fb6e415b560 2024-03-23T03:49:38.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.682+0000 7fb6e3e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb6e415d410 0x7fb6e40b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:38.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.682+0000 7fb6e3e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb6e415d410 0x7fb6e40b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49840/0 (socket says 172.21.15.102:49840) 2024-03-23T03:49:38.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.682+0000 7fb6e3e00640 1 -- 172.21.15.102:0/2924410018 learned_addr learned my addr 172.21.15.102:0/2924410018 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:38.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.682+0000 7fb6e3400640 1 --2- 172.21.15.102:0/2924410018 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6e415b560 0x7fb6e415b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:38.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.682+0000 7fb6e3e00640 1 -- 172.21.15.102:0/2924410018 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6e415b560 msgr2=0x7fb6e415b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:38.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.682+0000 7fb6e3e00640 1 --2- 172.21.15.102:0/2924410018 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6e415b560 0x7fb6e415b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:38.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.682+0000 7fb6e3e00640 1 -- 172.21.15.102:0/2924410018 >> v1:172.21.15.173:6789/0 conn(0x7fb6e40b39b0 legacy=0x7fb6e415e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:38.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.682+0000 7fb6e3e00640 1 -- 172.21.15.102:0/2924410018 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e406ccb0 con 0x7fb6e415d410 2024-03-23T03:49:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.683+0000 7fb6e3e00640 1 --2- 172.21.15.102:0/2924410018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6e415d410 0x7fb6e40b3e30 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009e50 tx=0x7fb6dc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91dbd4d3a7b3292 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.683+0000 7fb6e2a00640 1 -- 172.21.15.102:0/2924410018 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc02ca70 con 0x7fb6e415d410 2024-03-23T03:49:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.684+0000 7fb6e2a00640 1 -- 172.21.15.102:0/2924410018 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6dc02cbd0 con 0x7fb6e415d410 2024-03-23T03:49:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.684+0000 7fb6e2a00640 1 -- 172.21.15.102:0/2924410018 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc0343f0 con 0x7fb6e415d410 2024-03-23T03:49:38.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.684+0000 7fb6ea400640 1 -- 172.21.15.102:0/2924410018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6e415d410 msgr2=0x7fb6e40b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:38.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.684+0000 7fb6ea400640 1 --2- 172.21.15.102:0/2924410018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6e415d410 0x7fb6e40b3e30 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc009e50 tx=0x7fb6dc02bed0 comp rx=0 tx=0).stop 2024-03-23T03:49:38.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.684+0000 7fb6ea400640 1 -- 172.21.15.102:0/2924410018 shutdown_connections 2024-03-23T03:49:38.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.684+0000 7fb6ea400640 1 --2- 172.21.15.102:0/2924410018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6e415d410 0x7fb6e40b3e30 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:38.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.684+0000 7fb6ea400640 1 --2- 172.21.15.102:0/2924410018 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6e415b560 0x7fb6e415b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:38.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.684+0000 7fb6ea400640 1 -- 172.21.15.102:0/2924410018 >> 172.21.15.102:0/2924410018 conn(0x7fb6e406a3d0 msgr2=0x7fb6e40b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:38.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.684+0000 7fb6ea400640 1 -- 172.21.15.102:0/2924410018 shutdown_connections 2024-03-23T03:49:38.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.684+0000 7fb6ea400640 1 -- 172.21.15.102:0/2924410018 wait complete. 2024-03-23T03:49:38.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.686+0000 7fb6ea400640 1 Processor -- start 2024-03-23T03:49:38.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.687+0000 7fb6ea400640 1 -- start start 2024-03-23T03:49:38.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.688+0000 7fb6ea400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6e40b39b0 0x7fb6e4159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:38.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.688+0000 7fb6ea400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6e415b560 0x7fb6e4143000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:38.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.688+0000 7fb6ea400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6e415d410 0x7fb6e4143540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:38.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.688+0000 7fb6ea400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e4136e40 con 0x7fb6e40b39b0 2024-03-23T03:49:38.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.688+0000 7fb6ea400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e4136230 con 0x7fb6e415b560 2024-03-23T03:49:38.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.688+0000 7fb6ea400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e4136510 con 0x7fb6e415d410 2024-03-23T03:49:38.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.688+0000 7fb6e3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6e40b39b0 0x7fb6e4159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:38.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.688+0000 7fb6e3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6e40b39b0 0x7fb6e4159120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49850/0 (socket says 172.21.15.102:49850) 2024-03-23T03:49:38.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.688+0000 7fb6e3400640 1 -- 172.21.15.102:0/3847405442 learned_addr learned my addr 172.21.15.102:0/3847405442 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:38.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.688+0000 7fb6daa00640 1 --2- 172.21.15.102:0/3847405442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6e415b560 0x7fb6e4143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:38.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.689+0000 7fb6e3400640 1 -- 172.21.15.102:0/3847405442 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6e415d410 msgr2=0x7fb6e4143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:38.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.689+0000 7fb6e3e00640 1 --2- 172.21.15.102:0/3847405442 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6e415d410 0x7fb6e4143540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:38.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.689+0000 7fb6e3400640 1 --2- 172.21.15.102:0/3847405442 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6e415d410 0x7fb6e4143540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:38.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.689+0000 7fb6e3400640 1 -- 172.21.15.102:0/3847405442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6e415b560 msgr2=0x7fb6e4143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:38.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.689+0000 7fb6e3400640 1 --2- 172.21.15.102:0/3847405442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6e415b560 0x7fb6e4143000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:38.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.689+0000 7fb6e3400640 1 -- 172.21.15.102:0/3847405442 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc009b30 con 0x7fb6e40b39b0 2024-03-23T03:49:38.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.689+0000 7fb6e3e00640 1 --2- 172.21.15.102:0/3847405442 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6e415d410 0x7fb6e4143540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:38.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.689+0000 7fb6e3400640 1 --2- 172.21.15.102:0/3847405442 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6e40b39b0 0x7fb6e4159120 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400d800 tx=0x7fb6d400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:38.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.689+0000 7fb6dbe00640 1 -- 172.21.15.102:0/3847405442 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d400b6c0 con 0x7fb6e40b39b0 2024-03-23T03:49:38.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.689+0000 7fb6dbe00640 1 -- 172.21.15.102:0/3847405442 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6d400b820 con 0x7fb6e40b39b0 2024-03-23T03:49:38.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.689+0000 7fb6dbe00640 1 -- 172.21.15.102:0/3847405442 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6d400bbb0 con 0x7fb6e40b39b0 2024-03-23T03:49:38.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.689+0000 7fb6ea400640 1 -- 172.21.15.102:0/3847405442 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e41367f0 con 0x7fb6e40b39b0 2024-03-23T03:49:38.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.690+0000 7fb6ea400640 1 -- 172.21.15.102:0/3847405442 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e4159660 con 0x7fb6e40b39b0 2024-03-23T03:49:38.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.691+0000 7fb6ea400640 1 -- 172.21.15.102:0/3847405442 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e406ac20 con 0x7fb6e40b39b0 2024-03-23T03:49:38.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.692+0000 7fb6dbe00640 1 -- 172.21.15.102:0/3847405442 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb6d400bd10 con 0x7fb6e40b39b0 2024-03-23T03:49:38.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.692+0000 7fb6dbe00640 1 --2- 172.21.15.102:0/3847405442 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6c006ac00 0x7fb6c006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:38.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.693+0000 7fb6dbe00640 1 -- 172.21.15.102:0/3847405442 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb6d4089a50 con 0x7fb6e40b39b0 2024-03-23T03:49:38.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.693+0000 7fb6daa00640 1 --2- 172.21.15.102:0/3847405442 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6c006ac00 0x7fb6c006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:38.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.693+0000 7fb6daa00640 1 --2- 172.21.15.102:0/3847405442 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6c006ac00 0x7fb6c006d0c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc006fd0 tx=0x7fb6cc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:38.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.697+0000 7fb6dbe00640 1 -- 172.21.15.102:0/3847405442 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb6d4054ae0 con 0x7fb6e40b39b0 2024-03-23T03:49:38.893 INFO:tasks.ceph.mds.d.smithi173.stderr:2024-03-23T03:49:38.892+0000 7fe7dbe00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T03:49:38.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.928+0000 7fb6ea400640 1 -- 172.21.15.102:0/3847405442 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb6e4073390 con 0x7fb6e40b39b0 2024-03-23T03:49:38.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.929+0000 7fb6dbe00640 1 -- 172.21.15.102:0/3847405442 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4744 (secure 0 0 0) 0x7fb6d4058100 con 0x7fb6e40b39b0 2024-03-23T03:49:38.930 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:38.930 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5230,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3155961019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3155961019},{"type":"v1","addr":"172.21.15.173:6835","nonce":3155961019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6021,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/251450026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":251450026},{"type":"v1","addr":"172.21.15.102:6836","nonce":251450026}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6024,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3299889122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":3299889122},{"type":"v1","addr":"172.21.15.102:6837","nonce":3299889122}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:49:35.415835+0000","modified":"2024-03-23T03:49:38.434294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6030":{"gid":6030,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":4,"addr":"172.21.15.173:6833/3695568749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3695568749},{"type":"v1","addr":"172.21.15.173:6833","nonce":3695568749}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-23T03:49:38.931 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 33 2024-03-23T03:49:38.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.932+0000 7fb6ea400640 1 -- 172.21.15.102:0/3847405442 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6c006ac00 msgr2=0x7fb6c006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:38.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.932+0000 7fb6ea400640 1 --2- 172.21.15.102:0/3847405442 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6c006ac00 0x7fb6c006d0c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc006fd0 tx=0x7fb6cc008040 comp rx=0 tx=0).stop 2024-03-23T03:49:38.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.932+0000 7fb6ea400640 1 -- 172.21.15.102:0/3847405442 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6e40b39b0 msgr2=0x7fb6e4159120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:38.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.932+0000 7fb6ea400640 1 --2- 172.21.15.102:0/3847405442 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6e40b39b0 0x7fb6e4159120 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400d800 tx=0x7fb6d400dcc0 comp rx=0 tx=0).stop 2024-03-23T03:49:38.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.932+0000 7fb6ea400640 1 -- 172.21.15.102:0/3847405442 shutdown_connections 2024-03-23T03:49:38.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.932+0000 7fb6ea400640 1 --2- 172.21.15.102:0/3847405442 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6c006ac00 0x7fb6c006d0c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:38.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.932+0000 7fb6ea400640 1 --2- 172.21.15.102:0/3847405442 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb6e415d410 0x7fb6e4143540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:38.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.932+0000 7fb6ea400640 1 --2- 172.21.15.102:0/3847405442 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6e415b560 0x7fb6e4143000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:38.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.932+0000 7fb6ea400640 1 --2- 172.21.15.102:0/3847405442 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb6e40b39b0 0x7fb6e4159120 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:38.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.932+0000 7fb6ea400640 1 -- 172.21.15.102:0/3847405442 >> 172.21.15.102:0/3847405442 conn(0x7fb6e406a3d0 msgr2=0x7fb6e40b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:38.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.932+0000 7fb6ea400640 1 -- 172.21.15.102:0/3847405442 shutdown_connections 2024-03-23T03:49:38.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:38.932+0000 7fb6ea400640 1 -- 172.21.15.102:0/3847405442 wait complete. 2024-03-23T03:49:38.951 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:49:39.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.153+0000 7f2159c00640 1 Processor -- start 2024-03-23T03:49:39.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.154+0000 7f2159c00640 1 -- start start 2024-03-23T03:49:39.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.155+0000 7f2159c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21540cbab0 0x7f21540b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:39.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.155+0000 7f2159c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f215406d390 0x7f215406d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:39.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.155+0000 7f2159c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21540b16a0 con 0x7f21540adbc0 2024-03-23T03:49:39.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.155+0000 7f2159c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f215406dd60 con 0x7f215406d390 2024-03-23T03:49:39.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.155+0000 7f2159c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21540cbfb0 con 0x7f21540cbab0 2024-03-23T03:49:39.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.156+0000 7f2152a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f21540adbc0 0x7f21540b41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50630/0 (socket says 172.21.15.102:50630) 2024-03-23T03:49:39.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.156+0000 7f2152a00640 1 -- 172.21.15.102:0/4154808669 learned_addr learned my addr 172.21.15.102:0/4154808669 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:39.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.156+0000 7f2153400640 1 --2- 172.21.15.102:0/4154808669 >> v2:172.21.15.173:3300/0 conn(0x7f215406d390 0x7f215406d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:39.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.156+0000 7f214a000640 1 --2- 172.21.15.102:0/4154808669 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21540cbab0 0x7f21540b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:39.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.156+0000 7f2152000640 1 -- 172.21.15.102:0/4154808669 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867473681 0 0) 0x7f21540b16a0 con 0x7f21540adbc0 2024-03-23T03:49:39.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.156+0000 7f2152000640 1 -- 172.21.15.102:0/4154808669 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2138003520 con 0x7f21540adbc0 2024-03-23T03:49:39.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.156+0000 7f2152000640 1 -- 172.21.15.102:0/4154808669 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1215798445 0 0) 0x7f2138003520 con 0x7f21540adbc0 2024-03-23T03:49:39.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.156+0000 7f2152000640 1 -- 172.21.15.102:0/4154808669 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21540cbab0 msgr2=0x7f21540b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:39.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.156+0000 7f2152000640 1 --2- 172.21.15.102:0/4154808669 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21540cbab0 0x7f21540b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.156+0000 7f2152000640 1 -- 172.21.15.102:0/4154808669 >> v2:172.21.15.173:3300/0 conn(0x7f215406d390 msgr2=0x7f215406d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:39.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.156+0000 7f2152000640 1 --2- 172.21.15.102:0/4154808669 >> v2:172.21.15.173:3300/0 conn(0x7f215406d390 0x7f215406d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.156+0000 7f2152000640 1 -- 172.21.15.102:0/4154808669 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21540cc290 con 0x7f21540adbc0 2024-03-23T03:49:39.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.157+0000 7f2152000640 1 -- 172.21.15.102:0/4154808669 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f21440019f0 con 0x7f21540adbc0 2024-03-23T03:49:39.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.157+0000 7f2152000640 1 -- 172.21.15.102:0/4154808669 >> v1:172.21.15.102:6789/0 conn(0x7f21540adbc0 legacy=0x7f21540b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:39.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.157+0000 7f2152000640 1 --2- 172.21.15.102:0/4154808669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2138003970 0x7f2138003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:39.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.157+0000 7f2152000640 1 --2- 172.21.15.102:0/4154808669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21380047c0 0x7f2138004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:39.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.157+0000 7f2152000640 1 --2- 172.21.15.102:0/4154808669 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21540cbab0 0x7f2138005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:39.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.157+0000 7f2152000640 1 -- 172.21.15.102:0/4154808669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f215406dd60 con 0x7f21380047c0 2024-03-23T03:49:39.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.157+0000 7f2152000640 1 -- 172.21.15.102:0/4154808669 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21540cbfb0 con 0x7f2138003970 2024-03-23T03:49:39.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.157+0000 7f2152000640 1 -- 172.21.15.102:0/4154808669 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213800b9d0 con 0x7f21540cbab0 2024-03-23T03:49:39.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.157+0000 7f214a000640 1 --2- 172.21.15.102:0/4154808669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21380047c0 0x7f2138004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:39.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.157+0000 7f2152a00640 1 --2- 172.21.15.102:0/4154808669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2138003970 0x7f2138003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:39.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.157+0000 7f214a000640 1 -- 172.21.15.102:0/4154808669 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21540cbab0 msgr2=0x7f2138005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:39.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.157+0000 7f214a000640 1 --2- 172.21.15.102:0/4154808669 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f21540cbab0 0x7f2138005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.157+0000 7f214a000640 1 -- 172.21.15.102:0/4154808669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2138003970 msgr2=0x7f2138003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:39.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.157+0000 7f214a000640 1 --2- 172.21.15.102:0/4154808669 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2138003970 0x7f2138003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.157+0000 7f214a000640 1 -- 172.21.15.102:0/4154808669 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2138003520 con 0x7f21380047c0 2024-03-23T03:49:39.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.158+0000 7f214a000640 1 --2- 172.21.15.102:0/4154808669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21380047c0 0x7f2138004c40 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f213c009ac0 tx=0x7f21380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82f632c8572957b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:39.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.159+0000 7f2152000640 1 -- 172.21.15.102:0/4154808669 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c02c710 con 0x7f21380047c0 2024-03-23T03:49:39.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.159+0000 7f2152000640 1 -- 172.21.15.102:0/4154808669 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213c02c870 con 0x7f21380047c0 2024-03-23T03:49:39.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.159+0000 7f2152000640 1 -- 172.21.15.102:0/4154808669 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c02cc00 con 0x7f21380047c0 2024-03-23T03:49:39.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.159+0000 7f2159c00640 1 -- 172.21.15.102:0/4154808669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21380047c0 msgr2=0x7f2138004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:39.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.159+0000 7f2159c00640 1 --2- 172.21.15.102:0/4154808669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21380047c0 0x7f2138004c40 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f213c009ac0 tx=0x7f21380057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.159+0000 7f2153400640 1 -- 172.21.15.102:0/4154808669 reap_dead start 2024-03-23T03:49:39.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.159+0000 7f2159c00640 1 -- 172.21.15.102:0/4154808669 shutdown_connections 2024-03-23T03:49:39.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.159+0000 7f2159c00640 1 -- 172.21.15.102:0/4154808669 >> 172.21.15.102:0/4154808669 conn(0x7f215406a3d0 msgr2=0x7f21540b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:39.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.159+0000 7f2159c00640 1 -- 172.21.15.102:0/4154808669 shutdown_connections 2024-03-23T03:49:39.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.159+0000 7f2159c00640 1 -- 172.21.15.102:0/4154808669 wait complete. 2024-03-23T03:49:39.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.161+0000 7f2159c00640 1 Processor -- start 2024-03-23T03:49:39.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.161+0000 7f2159c00640 1 -- start start 2024-03-23T03:49:39.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.162+0000 7f2159c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21380047c0 0x7f2154159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:39.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.162+0000 7f2159c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21541f29f0 0x7f215413b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:39.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.162+0000 7f2159c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f215415c5e0 0x7f215415a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:39.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.162+0000 7f2159c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21540abdd0 con 0x7f21380047c0 2024-03-23T03:49:39.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.162+0000 7f2159c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21540ac910 con 0x7f21541f29f0 2024-03-23T03:49:39.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.162+0000 7f2159c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21540abfe0 con 0x7f215415c5e0 2024-03-23T03:49:39.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.162+0000 7f2152a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21380047c0 0x7f2154159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:39.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.162+0000 7f2152a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21380047c0 0x7f2154159250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49878/0 (socket says 172.21.15.102:49878) 2024-03-23T03:49:39.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.162+0000 7f2152a00640 1 -- 172.21.15.102:0/2785552387 learned_addr learned my addr 172.21.15.102:0/2785552387 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:39.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.163+0000 7f2153400640 1 --2- 172.21.15.102:0/2785552387 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f215415c5e0 0x7f215415a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:39.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.163+0000 7f214a000640 1 --2- 172.21.15.102:0/2785552387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21541f29f0 0x7f215413b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:39.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.163+0000 7f2152a00640 1 -- 172.21.15.102:0/2785552387 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f215415c5e0 msgr2=0x7f215415a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:39.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.163+0000 7f2152a00640 1 --2- 172.21.15.102:0/2785552387 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f215415c5e0 0x7f215415a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.163+0000 7f2152a00640 1 -- 172.21.15.102:0/2785552387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21541f29f0 msgr2=0x7f215413b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:39.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.163+0000 7f2152a00640 1 --2- 172.21.15.102:0/2785552387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21541f29f0 0x7f215413b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.163+0000 7f2152a00640 1 -- 172.21.15.102:0/2785552387 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21540ac2c0 con 0x7f21380047c0 2024-03-23T03:49:39.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.163+0000 7f2153400640 1 --2- 172.21.15.102:0/2785552387 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f215415c5e0 0x7f215415a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:39.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.163+0000 7f2152a00640 1 --2- 172.21.15.102:0/2785552387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21380047c0 0x7f2154159250 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f2144002150 tx=0x7f214400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:39.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.163+0000 7f214b400640 1 -- 172.21.15.102:0/2785552387 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2144004d60 con 0x7f21380047c0 2024-03-23T03:49:39.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.163+0000 7f214b400640 1 -- 172.21.15.102:0/2785552387 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21440070d0 con 0x7f21380047c0 2024-03-23T03:49:39.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.163+0000 7f214b400640 1 -- 172.21.15.102:0/2785552387 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2144007460 con 0x7f21380047c0 2024-03-23T03:49:39.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.163+0000 7f2159c00640 1 -- 172.21.15.102:0/2785552387 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21540ac5a0 con 0x7f21380047c0 2024-03-23T03:49:39.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.164+0000 7f2159c00640 1 -- 172.21.15.102:0/2785552387 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f215413b790 con 0x7f21380047c0 2024-03-23T03:49:39.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.165+0000 7f2159c00640 1 -- 172.21.15.102:0/2785552387 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f215406ac20 con 0x7f21380047c0 2024-03-23T03:49:39.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.166+0000 7f214b400640 1 -- 172.21.15.102:0/2785552387 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f21440075c0 con 0x7f21380047c0 2024-03-23T03:49:39.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.166+0000 7f214b400640 1 --2- 172.21.15.102:0/2785552387 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f213006ab90 0x7f213006d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:39.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.167+0000 7f214b400640 1 -- 172.21.15.102:0/2785552387 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2144089630 con 0x7f21380047c0 2024-03-23T03:49:39.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.167+0000 7f214a000640 1 --2- 172.21.15.102:0/2785552387 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f213006ab90 0x7f213006d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:39.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.167+0000 7f214a000640 1 --2- 172.21.15.102:0/2785552387 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f213006ab90 0x7f213006d050 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f213c036000 tx=0x7f213c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:39.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.171+0000 7f214b400640 1 -- 172.21.15.102:0/2785552387 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f21440546c0 con 0x7f21380047c0 2024-03-23T03:49:39.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.364+0000 7f2159c00640 1 -- 172.21.15.102:0/2785552387 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2154073390 con 0x7f21380047c0 2024-03-23T03:49:39.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.367+0000 7f214b400640 1 -- 172.21.15.102:0/2785552387 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16296 (secure 0 0 0) 0x7f2144057ce0 con 0x7f21380047c0 2024-03-23T03:49:39.368 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:39.368 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":67,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:49:36.420326+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:49:33.007757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-23T03:49:34.449411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:49:39.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.369+0000 7f2159c00640 1 -- 172.21.15.102:0/2785552387 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f213006ab90 msgr2=0x7f213006d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:39.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.369+0000 7f2159c00640 1 --2- 172.21.15.102:0/2785552387 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f213006ab90 0x7f213006d050 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f213c036000 tx=0x7f213c030000 comp rx=0 tx=0).stop 2024-03-23T03:49:39.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.369+0000 7f2159c00640 1 -- 172.21.15.102:0/2785552387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21380047c0 msgr2=0x7f2154159250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:39.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.369+0000 7f2159c00640 1 --2- 172.21.15.102:0/2785552387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21380047c0 0x7f2154159250 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f2144002150 tx=0x7f214400b4a0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.370+0000 7f2159c00640 1 -- 172.21.15.102:0/2785552387 shutdown_connections 2024-03-23T03:49:39.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.370+0000 7f2159c00640 1 --2- 172.21.15.102:0/2785552387 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f213006ab90 0x7f213006d050 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.370+0000 7f2159c00640 1 --2- 172.21.15.102:0/2785552387 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f215415c5e0 0x7f215415a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.370+0000 7f2159c00640 1 --2- 172.21.15.102:0/2785552387 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f21541f29f0 0x7f215413b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.370+0000 7f2159c00640 1 --2- 172.21.15.102:0/2785552387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f21380047c0 0x7f2154159250 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.370+0000 7f2159c00640 1 -- 172.21.15.102:0/2785552387 >> 172.21.15.102:0/2785552387 conn(0x7f215406a3d0 msgr2=0x7f21540b3780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:39.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.370+0000 7f2159c00640 1 -- 172.21.15.102:0/2785552387 shutdown_connections 2024-03-23T03:49:39.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.370+0000 7f2159c00640 1 -- 172.21.15.102:0/2785552387 wait complete. 2024-03-23T03:49:39.388 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T03:49:39.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:49:39.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.595+0000 7f1659000640 1 Processor -- start 2024-03-23T03:49:39.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.596+0000 7f1659000640 1 -- start start 2024-03-23T03:49:39.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.597+0000 7f1659000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f165415cda0 0x7f16540cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:39.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.597+0000 7f1659000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16540d1830 con 0x7f16540b5a30 2024-03-23T03:49:39.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.597+0000 7f1659000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16540b1f20 con 0x7f16540b4a60 2024-03-23T03:49:39.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.597+0000 7f1659000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16540d0c20 con 0x7f165415cda0 2024-03-23T03:49:39.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.598+0000 7f1651400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f16540b5a30 0x7f16540b4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50644/0 (socket says 172.21.15.102:50644) 2024-03-23T03:49:39.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.598+0000 7f1651400640 1 -- 172.21.15.102:0/116100748 learned_addr learned my addr 172.21.15.102:0/116100748 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:39.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.598+0000 7f1651e00640 1 --2- 172.21.15.102:0/116100748 >> v2:172.21.15.173:3300/0 conn(0x7f165415cda0 0x7f16540cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:39.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.598+0000 7f1650a00640 1 -- 172.21.15.102:0/116100748 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572886495 0 0) 0x7f16540d1830 con 0x7f16540b5a30 2024-03-23T03:49:39.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.598+0000 7f1650a00640 1 -- 172.21.15.102:0/116100748 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1638003520 con 0x7f16540b5a30 2024-03-23T03:49:39.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.598+0000 7f1650a00640 1 -- 172.21.15.102:0/116100748 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3505122805 0 0) 0x7f1638003520 con 0x7f16540b5a30 2024-03-23T03:49:39.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.598+0000 7f1650a00640 1 -- 172.21.15.102:0/116100748 >> v2:172.21.15.173:3300/0 conn(0x7f165415cda0 msgr2=0x7f16540cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:39.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.598+0000 7f1650a00640 1 --2- 172.21.15.102:0/116100748 >> v2:172.21.15.173:3300/0 conn(0x7f165415cda0 0x7f16540cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.598+0000 7f1650a00640 1 -- 172.21.15.102:0/116100748 >> v1:172.21.15.173:6789/0 conn(0x7f16540b4a60 legacy=0x7f16540d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:39.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.598+0000 7f1650a00640 1 -- 172.21.15.102:0/116100748 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16540d0f00 con 0x7f16540b5a30 2024-03-23T03:49:39.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.598+0000 7f1650a00640 1 -- 172.21.15.102:0/116100748 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f16440019f0 con 0x7f16540b5a30 2024-03-23T03:49:39.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.599+0000 7f1650a00640 1 -- 172.21.15.102:0/116100748 >> v1:172.21.15.102:6789/0 conn(0x7f16540b5a30 legacy=0x7f16540b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:39.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.599+0000 7f1650a00640 1 --2- 172.21.15.102:0/116100748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16380039c0 0x7f1638003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:39.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.599+0000 7f1650a00640 1 --2- 172.21.15.102:0/116100748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16380047c0 0x7f1638004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:39.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.599+0000 7f1650a00640 1 --2- 172.21.15.102:0/116100748 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1638005180 0x7f163800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:39.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.599+0000 7f1650a00640 1 -- 172.21.15.102:0/116100748 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16540d0c20 con 0x7f16380039c0 2024-03-23T03:49:39.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.599+0000 7f1650a00640 1 -- 172.21.15.102:0/116100748 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f163800c080 con 0x7f16380047c0 2024-03-23T03:49:39.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.599+0000 7f1650a00640 1 -- 172.21.15.102:0/116100748 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f163800c240 con 0x7f1638005180 2024-03-23T03:49:39.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.599+0000 7f1651e00640 1 --2- 172.21.15.102:0/116100748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16380039c0 0x7f1638003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:39.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.599+0000 7f1651e00640 1 -- 172.21.15.102:0/116100748 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1638005180 msgr2=0x7f163800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:39.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.599+0000 7f1651e00640 1 --2- 172.21.15.102:0/116100748 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1638005180 0x7f163800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.599+0000 7f1651e00640 1 -- 172.21.15.102:0/116100748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16380047c0 msgr2=0x7f1638004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:39.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.599+0000 7f1651e00640 1 --2- 172.21.15.102:0/116100748 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16380047c0 0x7f1638004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.599+0000 7f1651e00640 1 -- 172.21.15.102:0/116100748 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1638003520 con 0x7f16380039c0 2024-03-23T03:49:39.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.599+0000 7f1652800640 1 -- 172.21.15.102:0/116100748 reap_dead start 2024-03-23T03:49:39.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.600+0000 7f1651e00640 1 --2- 172.21.15.102:0/116100748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16380039c0 0x7f1638003de0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f163c009ac0 tx=0x7f16380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51a6d35c7862c897 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:39.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.601+0000 7f1650a00640 1 -- 172.21.15.102:0/116100748 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c02c6d0 con 0x7f16380039c0 2024-03-23T03:49:39.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.601+0000 7f1650a00640 1 -- 172.21.15.102:0/116100748 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f163c02c830 con 0x7f16380039c0 2024-03-23T03:49:39.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.601+0000 7f1650a00640 1 -- 172.21.15.102:0/116100748 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f163c02cbc0 con 0x7f16380039c0 2024-03-23T03:49:39.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.601+0000 7f1659000640 1 -- 172.21.15.102:0/116100748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16380039c0 msgr2=0x7f1638003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:39.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.601+0000 7f1659000640 1 --2- 172.21.15.102:0/116100748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16380039c0 0x7f1638003de0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f163c009ac0 tx=0x7f16380057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.601+0000 7f1659000640 1 -- 172.21.15.102:0/116100748 shutdown_connections 2024-03-23T03:49:39.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.601+0000 7f1659000640 1 --2- 172.21.15.102:0/116100748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16380039c0 0x7f1638003de0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.601+0000 7f1659000640 1 -- 172.21.15.102:0/116100748 >> 172.21.15.102:0/116100748 conn(0x7f16540cf350 msgr2=0x7f16540cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:39.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.601+0000 7f1659000640 1 -- 172.21.15.102:0/116100748 shutdown_connections 2024-03-23T03:49:39.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.602+0000 7f1659000640 1 -- 172.21.15.102:0/116100748 wait complete. 2024-03-23T03:49:39.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.603+0000 7f1659000640 1 Processor -- start 2024-03-23T03:49:39.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.603+0000 7f1659000640 1 -- start start 2024-03-23T03:49:39.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.604+0000 7f1659000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f16380039c0 0x7f1654156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:39.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.604+0000 7f1659000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16540cb270 0x7f16540c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:39.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.604+0000 7f1659000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16541f2fa0 0x7f16540cb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:39.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.604+0000 7f1659000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1654157460 con 0x7f16540cb270 2024-03-23T03:49:39.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.604+0000 7f1659000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16540ca1d0 con 0x7f16541f2fa0 2024-03-23T03:49:39.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.604+0000 7f1659000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16540ca4b0 con 0x7f16380039c0 2024-03-23T03:49:39.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.605+0000 7f1651400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16540cb270 0x7f16540c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:39.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.605+0000 7f1651400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16540cb270 0x7f16540c99a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49906/0 (socket says 172.21.15.102:49906) 2024-03-23T03:49:39.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.605+0000 7f1651400640 1 -- 172.21.15.102:0/235386207 learned_addr learned my addr 172.21.15.102:0/235386207 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:39.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.605+0000 7f1651e00640 1 --2- 172.21.15.102:0/235386207 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f16380039c0 0x7f1654156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:39.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.605+0000 7f1652800640 1 --2- 172.21.15.102:0/235386207 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16541f2fa0 0x7f16540cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:39.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.605+0000 7f1651400640 1 -- 172.21.15.102:0/235386207 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f16380039c0 msgr2=0x7f1654156f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:39.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.605+0000 7f1651400640 1 --2- 172.21.15.102:0/235386207 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f16380039c0 0x7f1654156f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.605+0000 7f1651400640 1 -- 172.21.15.102:0/235386207 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16541f2fa0 msgr2=0x7f16540cb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:39.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.605+0000 7f1651400640 1 --2- 172.21.15.102:0/235386207 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16541f2fa0 0x7f16540cb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.605+0000 7f1651400640 1 -- 172.21.15.102:0/235386207 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16540ca790 con 0x7f16540cb270 2024-03-23T03:49:39.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.605+0000 7f1651e00640 1 --2- 172.21.15.102:0/235386207 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f16380039c0 0x7f1654156f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:39.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.605+0000 7f1651400640 1 --2- 172.21.15.102:0/235386207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16540cb270 0x7f16540c99a0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f164400caa0 tx=0x7f164400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:39.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.606+0000 7f164a000640 1 -- 172.21.15.102:0/235386207 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164400a790 con 0x7f16540cb270 2024-03-23T03:49:39.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.606+0000 7f164a000640 1 -- 172.21.15.102:0/235386207 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164400a8f0 con 0x7f16540cb270 2024-03-23T03:49:39.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.606+0000 7f164a000640 1 -- 172.21.15.102:0/235386207 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164400aca0 con 0x7f16540cb270 2024-03-23T03:49:39.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.606+0000 7f1659000640 1 -- 172.21.15.102:0/235386207 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16540caa70 con 0x7f16540cb270 2024-03-23T03:49:39.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.606+0000 7f1659000640 1 -- 172.21.15.102:0/235386207 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16540c9ee0 con 0x7f16540cb270 2024-03-23T03:49:39.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.607+0000 7f1659000640 1 -- 172.21.15.102:0/235386207 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16540caf90 con 0x7f16540cb270 2024-03-23T03:49:39.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.608+0000 7f164a000640 1 -- 172.21.15.102:0/235386207 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1644007150 con 0x7f16540cb270 2024-03-23T03:49:39.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.609+0000 7f164a000640 1 --2- 172.21.15.102:0/235386207 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f163006ac30 0x7f163006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:39.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.609+0000 7f164a000640 1 -- 172.21.15.102:0/235386207 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1644088bd0 con 0x7f16540cb270 2024-03-23T03:49:39.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.609+0000 7f1651e00640 1 --2- 172.21.15.102:0/235386207 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f163006ac30 0x7f163006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:39.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.610+0000 7f1651e00640 1 --2- 172.21.15.102:0/235386207 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f163006ac30 0x7f163006d0f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f163c036000 tx=0x7f163c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:39.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.613+0000 7f164a000640 1 -- 172.21.15.102:0/235386207 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f16540caf90 con 0x7f16540cb270 2024-03-23T03:49:39.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.808+0000 7f1659000640 1 -- 172.21.15.102:0/235386207 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16540d14c0 con 0x7f163006ac30 2024-03-23T03:49:39.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.831+0000 7f164a000640 1 -- 172.21.15.102:0/235386207 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286896 (secure 0 0 0) 0x7f163c004140 con 0x7f163006ac30 2024-03-23T03:49:39.836 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:39.837 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:49:39.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.837+0000 7f1659000640 1 -- 172.21.15.102:0/235386207 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f163006ac30 msgr2=0x7f163006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:39.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.837+0000 7f1659000640 1 --2- 172.21.15.102:0/235386207 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f163006ac30 0x7f163006d0f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f163c036000 tx=0x7f163c030000 comp rx=0 tx=0).stop 2024-03-23T03:49:39.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.837+0000 7f1659000640 1 -- 172.21.15.102:0/235386207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16540cb270 msgr2=0x7f16540c99a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:39.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.837+0000 7f1659000640 1 --2- 172.21.15.102:0/235386207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16540cb270 0x7f16540c99a0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f164400caa0 tx=0x7f164400cf60 comp rx=0 tx=0).stop 2024-03-23T03:49:39.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.838+0000 7f1659000640 1 -- 172.21.15.102:0/235386207 shutdown_connections 2024-03-23T03:49:39.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.838+0000 7f1659000640 1 --2- 172.21.15.102:0/235386207 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f163006ac30 0x7f163006d0f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.838+0000 7f1659000640 1 --2- 172.21.15.102:0/235386207 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16541f2fa0 0x7f16540cb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.838+0000 7f1659000640 1 --2- 172.21.15.102:0/235386207 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16540cb270 0x7f16540c99a0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.838+0000 7f1659000640 1 --2- 172.21.15.102:0/235386207 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f16380039c0 0x7f1654156f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:39.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.838+0000 7f1659000640 1 -- 172.21.15.102:0/235386207 >> 172.21.15.102:0/235386207 conn(0x7f16540cf350 msgr2=0x7f165415c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:39.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.838+0000 7f1659000640 1 -- 172.21.15.102:0/235386207 shutdown_connections 2024-03-23T03:49:39.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:39.838+0000 7f1659000640 1 -- 172.21.15.102:0/235386207 wait complete. 2024-03-23T03:49:39.856 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-03-23T03:49:39.768461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":67,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223648,"kb_used_data":7444,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751072,"statfs":{"total":773094113280,"available":772865097728,"internally_reserved":0,"allocated":7622656,"data_stored":4013160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":221236794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.827805"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436646+0000","last_change":"2024-03-23T03:49:35.437508+0000","last_active":"2024-03-23T03:49:36.436646+0000","last_peered":"2024-03-23T03:49:36.436646+0000","last_clean":"2024-03-23T03:49:36.436646+0000","last_became_active":"2024-03-23T03:49:35.437196+0000","last_became_peered":"2024-03-23T03:49:35.437196+0000","last_unstale":"2024-03-23T03:49:36.436646+0000","last_undegraded":"2024-03-23T03:49:36.436646+0000","last_fullsized":"2024-03-23T03:49:36.436646+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:50:23.362975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436468+0000","last_change":"2024-03-23T03:49:35.453041+0000","last_active":"2024-03-23T03:49:36.436468+0000","last_peered":"2024-03-23T03:49:36.436468+0000","last_clean":"2024-03-23T03:49:36.436468+0000","last_became_active":"2024-03-23T03:49:35.446517+0000","last_became_peered":"2024-03-23T03:49:35.446517+0000","last_unstale":"2024-03-23T03:49:36.436468+0000","last_undegraded":"2024-03-23T03:49:36.436468+0000","last_fullsized":"2024-03-23T03:49:36.436468+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:55:05.527463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.458021+0000","last_change":"2024-03-23T03:49:35.454914+0000","last_active":"2024-03-23T03:49:35.458021+0000","last_peered":"2024-03-23T03:49:35.458021+0000","last_clean":"2024-03-23T03:49:35.458021+0000","last_became_active":"2024-03-23T03:49:35.454627+0000","last_became_peered":"2024-03-23T03:49:35.454627+0000","last_unstale":"2024-03-23T03:49:35.458021+0000","last_undegraded":"2024-03-23T03:49:35.458021+0000","last_fullsized":"2024-03-23T03:49:35.458021+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:26:04.416714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.461936+0000","last_change":"2024-03-23T03:49:35.460833+0000","last_active":"2024-03-23T03:49:35.461936+0000","last_peered":"2024-03-23T03:49:35.461936+0000","last_clean":"2024-03-23T03:49:35.461936+0000","last_became_active":"2024-03-23T03:49:35.460282+0000","last_became_peered":"2024-03-23T03:49:35.460282+0000","last_unstale":"2024-03-23T03:49:35.461936+0000","last_undegraded":"2024-03-23T03:49:35.461936+0000","last_fullsized":"2024-03-23T03:49:35.461936+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:10.974686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.460801+0000","last_change":"2024-03-23T03:49:35.457923+0000","last_active":"2024-03-23T03:49:35.460801+0000","last_peered":"2024-03-23T03:49:35.460801+0000","last_clean":"2024-03-23T03:49:35.460801+0000","last_became_active":"2024-03-23T03:49:35.449801+0000","last_became_peered":"2024-03-23T03:49:35.449801+0000","last_unstale":"2024-03-23T03:49:35.460801+0000","last_undegraded":"2024-03-23T03:49:35.460801+0000","last_fullsized":"2024-03-23T03:49:35.460801+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:19:24.808232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650800+0000","last_change":"2024-03-23T03:49:35.447252+0000","last_active":"2024-03-23T03:49:36.650800+0000","last_peered":"2024-03-23T03:49:36.650800+0000","last_clean":"2024-03-23T03:49:36.650800+0000","last_became_active":"2024-03-23T03:49:35.446694+0000","last_became_peered":"2024-03-23T03:49:35.446694+0000","last_unstale":"2024-03-23T03:49:36.650800+0000","last_undegraded":"2024-03-23T03:49:36.650800+0000","last_fullsized":"2024-03-23T03:49:36.650800+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:49:49.765798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.458990+0000","last_change":"2024-03-23T03:49:35.451510+0000","last_active":"2024-03-23T03:49:35.458990+0000","last_peered":"2024-03-23T03:49:35.458990+0000","last_clean":"2024-03-23T03:49:35.458990+0000","last_became_active":"2024-03-23T03:49:35.451304+0000","last_became_peered":"2024-03-23T03:49:35.451304+0000","last_unstale":"2024-03-23T03:49:35.458990+0000","last_undegraded":"2024-03-23T03:49:35.458990+0000","last_fullsized":"2024-03-23T03:49:35.458990+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:58.894188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.435985+0000","last_change":"2024-03-23T03:49:35.443995+0000","last_active":"2024-03-23T03:49:36.435985+0000","last_peered":"2024-03-23T03:49:36.435985+0000","last_clean":"2024-03-23T03:49:36.435985+0000","last_became_active":"2024-03-23T03:49:35.443707+0000","last_became_peered":"2024-03-23T03:49:35.443707+0000","last_unstale":"2024-03-23T03:49:36.435985+0000","last_undegraded":"2024-03-23T03:49:36.435985+0000","last_fullsized":"2024-03-23T03:49:36.435985+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:44.372289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.467360+0000","last_change":"2024-03-23T03:49:35.464954+0000","last_active":"2024-03-23T03:49:35.467360+0000","last_peered":"2024-03-23T03:49:35.467360+0000","last_clean":"2024-03-23T03:49:35.467360+0000","last_became_active":"2024-03-23T03:49:35.464712+0000","last_became_peered":"2024-03-23T03:49:35.464712+0000","last_unstale":"2024-03-23T03:49:35.467360+0000","last_undegraded":"2024-03-23T03:49:35.467360+0000","last_fullsized":"2024-03-23T03:49:35.467360+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:19:55.183686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.457930+0000","last_change":"2024-03-23T03:49:35.453993+0000","last_active":"2024-03-23T03:49:35.457930+0000","last_peered":"2024-03-23T03:49:35.457930+0000","last_clean":"2024-03-23T03:49:35.457930+0000","last_became_active":"2024-03-23T03:49:35.453784+0000","last_became_peered":"2024-03-23T03:49:35.453784+0000","last_unstale":"2024-03-23T03:49:35.457930+0000","last_undegraded":"2024-03-23T03:49:35.457930+0000","last_fullsized":"2024-03-23T03:49:35.457930+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:58:17.359323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.462075+0000","last_change":"2024-03-23T03:49:35.459222+0000","last_active":"2024-03-23T03:49:35.462075+0000","last_peered":"2024-03-23T03:49:35.462075+0000","last_clean":"2024-03-23T03:49:35.462075+0000","last_became_active":"2024-03-23T03:49:35.457229+0000","last_became_peered":"2024-03-23T03:49:35.457229+0000","last_unstale":"2024-03-23T03:49:35.462075+0000","last_undegraded":"2024-03-23T03:49:35.462075+0000","last_fullsized":"2024-03-23T03:49:35.462075+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:08.096604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436139+0000","last_change":"2024-03-23T03:49:35.444731+0000","last_active":"2024-03-23T03:49:36.436139+0000","last_peered":"2024-03-23T03:49:36.436139+0000","last_clean":"2024-03-23T03:49:36.436139+0000","last_became_active":"2024-03-23T03:49:35.444423+0000","last_became_peered":"2024-03-23T03:49:35.444423+0000","last_unstale":"2024-03-23T03:49:36.436139+0000","last_undegraded":"2024-03-23T03:49:36.436139+0000","last_fullsized":"2024-03-23T03:49:36.436139+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:14:59.255644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436223+0000","last_change":"2024-03-23T03:49:35.446054+0000","last_active":"2024-03-23T03:49:36.436223+0000","last_peered":"2024-03-23T03:49:36.436223+0000","last_clean":"2024-03-23T03:49:36.436223+0000","last_became_active":"2024-03-23T03:49:35.445824+0000","last_became_peered":"2024-03-23T03:49:35.445824+0000","last_unstale":"2024-03-23T03:49:36.436223+0000","last_undegraded":"2024-03-23T03:49:36.436223+0000","last_fullsized":"2024-03-23T03:49:36.436223+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:10:50.530595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.456440+0000","last_change":"2024-03-23T03:49:35.450244+0000","last_active":"2024-03-23T03:49:35.456440+0000","last_peered":"2024-03-23T03:49:35.456440+0000","last_clean":"2024-03-23T03:49:35.456440+0000","last_became_active":"2024-03-23T03:49:35.449990+0000","last_became_peered":"2024-03-23T03:49:35.449990+0000","last_unstale":"2024-03-23T03:49:35.456440+0000","last_undegraded":"2024-03-23T03:49:35.456440+0000","last_fullsized":"2024-03-23T03:49:35.456440+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:55:40.868490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.457973+0000","last_change":"2024-03-23T03:49:35.454183+0000","last_active":"2024-03-23T03:49:35.457973+0000","last_peered":"2024-03-23T03:49:35.457973+0000","last_clean":"2024-03-23T03:49:35.457973+0000","last_became_active":"2024-03-23T03:49:35.453933+0000","last_became_peered":"2024-03-23T03:49:35.453933+0000","last_unstale":"2024-03-23T03:49:35.457973+0000","last_undegraded":"2024-03-23T03:49:35.457973+0000","last_fullsized":"2024-03-23T03:49:35.457973+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:50:27.972663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.459013+0000","last_change":"2024-03-23T03:49:35.452441+0000","last_active":"2024-03-23T03:49:35.459013+0000","last_peered":"2024-03-23T03:49:35.459013+0000","last_clean":"2024-03-23T03:49:35.459013+0000","last_became_active":"2024-03-23T03:49:35.452214+0000","last_became_peered":"2024-03-23T03:49:35.452214+0000","last_unstale":"2024-03-23T03:49:35.459013+0000","last_undegraded":"2024-03-23T03:49:35.459013+0000","last_fullsized":"2024-03-23T03:49:35.459013+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:02:00.932577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436802+0000","last_change":"2024-03-23T03:49:35.446531+0000","last_active":"2024-03-23T03:49:36.436802+0000","last_peered":"2024-03-23T03:49:36.436802+0000","last_clean":"2024-03-23T03:49:36.436802+0000","last_became_active":"2024-03-23T03:49:35.446139+0000","last_became_peered":"2024-03-23T03:49:35.446139+0000","last_unstale":"2024-03-23T03:49:36.436802+0000","last_undegraded":"2024-03-23T03:49:36.436802+0000","last_fullsized":"2024-03-23T03:49:36.436802+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:10:50.530595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.462584+0000","last_change":"2024-03-23T03:49:35.461897+0000","last_active":"2024-03-23T03:49:35.462584+0000","last_peered":"2024-03-23T03:49:35.462584+0000","last_clean":"2024-03-23T03:49:35.462584+0000","last_became_active":"2024-03-23T03:49:35.454272+0000","last_became_peered":"2024-03-23T03:49:35.454272+0000","last_unstale":"2024-03-23T03:49:35.462584+0000","last_undegraded":"2024-03-23T03:49:35.462584+0000","last_fullsized":"2024-03-23T03:49:35.462584+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:43:44.069662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.462828+0000","last_change":"2024-03-23T03:49:35.461117+0000","last_active":"2024-03-23T03:49:35.462828+0000","last_peered":"2024-03-23T03:49:35.462828+0000","last_clean":"2024-03-23T03:49:35.462828+0000","last_became_active":"2024-03-23T03:49:35.460441+0000","last_became_peered":"2024-03-23T03:49:35.460441+0000","last_unstale":"2024-03-23T03:49:35.462828+0000","last_undegraded":"2024-03-23T03:49:35.462828+0000","last_fullsized":"2024-03-23T03:49:35.462828+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:42:51.688200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.452071+0000","last_change":"2024-03-23T03:49:35.450256+0000","last_active":"2024-03-23T03:49:35.452071+0000","last_peered":"2024-03-23T03:49:35.452071+0000","last_clean":"2024-03-23T03:49:35.452071+0000","last_became_active":"2024-03-23T03:49:35.449940+0000","last_became_peered":"2024-03-23T03:49:35.449940+0000","last_unstale":"2024-03-23T03:49:35.452071+0000","last_undegraded":"2024-03-23T03:49:35.452071+0000","last_fullsized":"2024-03-23T03:49:35.452071+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:47:23.751438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436431+0000","last_change":"2024-03-23T03:49:35.439856+0000","last_active":"2024-03-23T03:49:36.436431+0000","last_peered":"2024-03-23T03:49:36.436431+0000","last_clean":"2024-03-23T03:49:36.436431+0000","last_became_active":"2024-03-23T03:49:35.439575+0000","last_became_peered":"2024-03-23T03:49:35.439575+0000","last_unstale":"2024-03-23T03:49:36.436431+0000","last_undegraded":"2024-03-23T03:49:36.436431+0000","last_fullsized":"2024-03-23T03:49:36.436431+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:48:11.607155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.451289+0000","last_change":"2024-03-23T03:49:35.445764+0000","last_active":"2024-03-23T03:49:35.451289+0000","last_peered":"2024-03-23T03:49:35.451289+0000","last_clean":"2024-03-23T03:49:35.451289+0000","last_became_active":"2024-03-23T03:49:35.445473+0000","last_became_peered":"2024-03-23T03:49:35.445473+0000","last_unstale":"2024-03-23T03:49:35.451289+0000","last_undegraded":"2024-03-23T03:49:35.451289+0000","last_fullsized":"2024-03-23T03:49:35.451289+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:45:25.906413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.449054+0000","last_change":"2024-03-23T03:49:35.443789+0000","last_active":"2024-03-23T03:49:35.449054+0000","last_peered":"2024-03-23T03:49:35.449054+0000","last_clean":"2024-03-23T03:49:35.449054+0000","last_became_active":"2024-03-23T03:49:35.442935+0000","last_became_peered":"2024-03-23T03:49:35.442935+0000","last_unstale":"2024-03-23T03:49:35.449054+0000","last_undegraded":"2024-03-23T03:49:35.449054+0000","last_fullsized":"2024-03-23T03:49:35.449054+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:17:26.874645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.456781+0000","last_change":"2024-03-23T03:49:35.452298+0000","last_active":"2024-03-23T03:49:35.456781+0000","last_peered":"2024-03-23T03:49:35.456781+0000","last_clean":"2024-03-23T03:49:35.456781+0000","last_became_active":"2024-03-23T03:49:35.449400+0000","last_became_peered":"2024-03-23T03:49:35.449400+0000","last_unstale":"2024-03-23T03:49:35.456781+0000","last_undegraded":"2024-03-23T03:49:35.456781+0000","last_fullsized":"2024-03-23T03:49:35.456781+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:55:05.527463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436491+0000","last_change":"2024-03-23T03:49:35.457522+0000","last_active":"2024-03-23T03:49:36.436491+0000","last_peered":"2024-03-23T03:49:36.436491+0000","last_clean":"2024-03-23T03:49:36.436491+0000","last_became_active":"2024-03-23T03:49:35.457014+0000","last_became_peered":"2024-03-23T03:49:35.457014+0000","last_unstale":"2024-03-23T03:49:36.436491+0000","last_undegraded":"2024-03-23T03:49:36.436491+0000","last_fullsized":"2024-03-23T03:49:36.436491+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:59.256052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436213+0000","last_change":"2024-03-23T03:49:35.457586+0000","last_active":"2024-03-23T03:49:36.436213+0000","last_peered":"2024-03-23T03:49:36.436213+0000","last_clean":"2024-03-23T03:49:36.436213+0000","last_became_active":"2024-03-23T03:49:35.457285+0000","last_became_peered":"2024-03-23T03:49:35.457285+0000","last_unstale":"2024-03-23T03:49:36.436213+0000","last_undegraded":"2024-03-23T03:49:36.436213+0000","last_fullsized":"2024-03-23T03:49:36.436213+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:19:31.816124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.462091+0000","last_change":"2024-03-23T03:49:35.459386+0000","last_active":"2024-03-23T03:49:35.462091+0000","last_peered":"2024-03-23T03:49:35.462091+0000","last_clean":"2024-03-23T03:49:35.462091+0000","last_became_active":"2024-03-23T03:49:35.457607+0000","last_became_peered":"2024-03-23T03:49:35.457607+0000","last_unstale":"2024-03-23T03:49:35.462091+0000","last_undegraded":"2024-03-23T03:49:35.462091+0000","last_fullsized":"2024-03-23T03:49:35.462091+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:19:24.808232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436337+0000","last_change":"2024-03-23T03:49:35.453262+0000","last_active":"2024-03-23T03:49:36.436337+0000","last_peered":"2024-03-23T03:49:36.436337+0000","last_clean":"2024-03-23T03:49:36.436337+0000","last_became_active":"2024-03-23T03:49:35.452899+0000","last_became_peered":"2024-03-23T03:49:35.452899+0000","last_unstale":"2024-03-23T03:49:36.436337+0000","last_undegraded":"2024-03-23T03:49:36.436337+0000","last_fullsized":"2024-03-23T03:49:36.436337+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:19:31.816124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.467421+0000","last_change":"2024-03-23T03:49:35.465195+0000","last_active":"2024-03-23T03:49:35.467421+0000","last_peered":"2024-03-23T03:49:35.467421+0000","last_clean":"2024-03-23T03:49:35.467421+0000","last_became_active":"2024-03-23T03:49:35.463733+0000","last_became_peered":"2024-03-23T03:49:35.463733+0000","last_unstale":"2024-03-23T03:49:35.467421+0000","last_undegraded":"2024-03-23T03:49:35.467421+0000","last_fullsized":"2024-03-23T03:49:35.467421+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:15:27.492278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.435912+0000","last_change":"2024-03-23T03:49:35.444622+0000","last_active":"2024-03-23T03:49:36.435912+0000","last_peered":"2024-03-23T03:49:36.435912+0000","last_clean":"2024-03-23T03:49:36.435912+0000","last_became_active":"2024-03-23T03:49:35.444343+0000","last_became_peered":"2024-03-23T03:49:35.444343+0000","last_unstale":"2024-03-23T03:49:36.435912+0000","last_undegraded":"2024-03-23T03:49:36.435912+0000","last_fullsized":"2024-03-23T03:49:36.435912+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:24:33.040816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.461301+0000","last_change":"2024-03-23T03:49:35.456971+0000","last_active":"2024-03-23T03:49:35.461301+0000","last_peered":"2024-03-23T03:49:35.461301+0000","last_clean":"2024-03-23T03:49:35.461301+0000","last_became_active":"2024-03-23T03:49:35.456670+0000","last_became_peered":"2024-03-23T03:49:35.456670+0000","last_unstale":"2024-03-23T03:49:35.461301+0000","last_undegraded":"2024-03-23T03:49:35.461301+0000","last_fullsized":"2024-03-23T03:49:35.461301+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:32:51.943312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650598+0000","last_change":"2024-03-23T03:49:35.457747+0000","last_active":"2024-03-23T03:49:36.650598+0000","last_peered":"2024-03-23T03:49:36.650598+0000","last_clean":"2024-03-23T03:49:36.650598+0000","last_became_active":"2024-03-23T03:49:35.457503+0000","last_became_peered":"2024-03-23T03:49:35.457503+0000","last_unstale":"2024-03-23T03:49:36.650598+0000","last_undegraded":"2024-03-23T03:49:36.650598+0000","last_fullsized":"2024-03-23T03:49:36.650598+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:23:14.905130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.456472+0000","last_change":"2024-03-23T03:49:35.444306+0000","last_active":"2024-03-23T03:49:35.456472+0000","last_peered":"2024-03-23T03:49:35.456472+0000","last_clean":"2024-03-23T03:49:35.456472+0000","last_became_active":"2024-03-23T03:49:35.444035+0000","last_became_peered":"2024-03-23T03:49:35.444035+0000","last_unstale":"2024-03-23T03:49:35.456472+0000","last_undegraded":"2024-03-23T03:49:35.456472+0000","last_fullsized":"2024-03-23T03:49:35.456472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:27:28.185090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"66'4","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436845+0000","last_change":"2024-03-23T03:49:34.982234+0000","last_active":"2024-03-23T03:49:36.436845+0000","last_peered":"2024-03-23T03:49:36.436845+0000","last_clean":"2024-03-23T03:49:36.436845+0000","last_became_active":"2024-03-23T03:49:34.981843+0000","last_became_peered":"2024-03-23T03:49:34.981843+0000","last_unstale":"2024-03-23T03:49:36.436845+0000","last_undegraded":"2024-03-23T03:49:36.436845+0000","last_fullsized":"2024-03-23T03:49:36.436845+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:32:24.880206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.468212+0000","last_change":"2024-03-23T03:49:35.464085+0000","last_active":"2024-03-23T03:49:35.468212+0000","last_peered":"2024-03-23T03:49:35.468212+0000","last_clean":"2024-03-23T03:49:35.468212+0000","last_became_active":"2024-03-23T03:49:35.463812+0000","last_became_peered":"2024-03-23T03:49:35.463812+0000","last_unstale":"2024-03-23T03:49:35.468212+0000","last_undegraded":"2024-03-23T03:49:35.468212+0000","last_fullsized":"2024-03-23T03:49:35.468212+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:23:44.334673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"66'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650966+0000","last_change":"2024-03-23T03:49:34.983531+0000","last_active":"2024-03-23T03:49:36.650966+0000","last_peered":"2024-03-23T03:49:36.650966+0000","last_clean":"2024-03-23T03:49:36.650966+0000","last_became_active":"2024-03-23T03:49:34.983191+0000","last_became_peered":"2024-03-23T03:49:34.983191+0000","last_unstale":"2024-03-23T03:49:36.650966+0000","last_undegraded":"2024-03-23T03:49:36.650966+0000","last_fullsized":"2024-03-23T03:49:36.650966+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:38:05.708048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650785+0000","last_change":"2024-03-23T03:49:35.457863+0000","last_active":"2024-03-23T03:49:36.650785+0000","last_peered":"2024-03-23T03:49:36.650785+0000","last_clean":"2024-03-23T03:49:36.650785+0000","last_became_active":"2024-03-23T03:49:35.456778+0000","last_became_peered":"2024-03-23T03:49:35.456778+0000","last_unstale":"2024-03-23T03:49:36.650785+0000","last_undegraded":"2024-03-23T03:49:36.650785+0000","last_fullsized":"2024-03-23T03:49:36.650785+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:41:07.345825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650762+0000","last_change":"2024-03-23T03:49:34.982798+0000","last_active":"2024-03-23T03:49:36.650762+0000","last_peered":"2024-03-23T03:49:36.650762+0000","last_clean":"2024-03-23T03:49:36.650762+0000","last_became_active":"2024-03-23T03:49:34.982539+0000","last_became_peered":"2024-03-23T03:49:34.982539+0000","last_unstale":"2024-03-23T03:49:36.650762+0000","last_undegraded":"2024-03-23T03:49:36.650762+0000","last_fullsized":"2024-03-23T03:49:36.650762+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:36:05.087323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.458067+0000","last_change":"2024-03-23T03:49:35.455447+0000","last_active":"2024-03-23T03:49:35.458067+0000","last_peered":"2024-03-23T03:49:35.458067+0000","last_clean":"2024-03-23T03:49:35.458067+0000","last_became_active":"2024-03-23T03:49:35.455169+0000","last_became_peered":"2024-03-23T03:49:35.455169+0000","last_unstale":"2024-03-23T03:49:35.458067+0000","last_undegraded":"2024-03-23T03:49:35.458067+0000","last_fullsized":"2024-03-23T03:49:35.458067+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:57:56.162095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650963+0000","last_change":"2024-03-23T03:49:34.974516+0000","last_active":"2024-03-23T03:49:36.650963+0000","last_peered":"2024-03-23T03:49:36.650963+0000","last_clean":"2024-03-23T03:49:36.650963+0000","last_became_active":"2024-03-23T03:49:34.974222+0000","last_became_peered":"2024-03-23T03:49:34.974222+0000","last_unstale":"2024-03-23T03:49:36.650963+0000","last_undegraded":"2024-03-23T03:49:36.650963+0000","last_fullsized":"2024-03-23T03:49:36.650963+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:13:03.702919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.455353+0000","last_change":"2024-03-23T03:49:35.448493+0000","last_active":"2024-03-23T03:49:35.455353+0000","last_peered":"2024-03-23T03:49:35.455353+0000","last_clean":"2024-03-23T03:49:35.455353+0000","last_became_active":"2024-03-23T03:49:35.446419+0000","last_became_peered":"2024-03-23T03:49:35.446419+0000","last_unstale":"2024-03-23T03:49:35.455353+0000","last_undegraded":"2024-03-23T03:49:35.455353+0000","last_fullsized":"2024-03-23T03:49:35.455353+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:14.484998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"66'1","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.445371+0000","last_change":"2024-03-23T03:49:34.973855+0000","last_active":"2024-03-23T03:49:35.445371+0000","last_peered":"2024-03-23T03:49:35.445371+0000","last_clean":"2024-03-23T03:49:35.445371+0000","last_became_active":"2024-03-23T03:49:34.973552+0000","last_became_peered":"2024-03-23T03:49:34.973552+0000","last_unstale":"2024-03-23T03:49:35.445371+0000","last_undegraded":"2024-03-23T03:49:35.445371+0000","last_fullsized":"2024-03-23T03:49:35.445371+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:23:13.893311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.435553+0000","last_change":"2024-03-23T03:49:35.458601+0000","last_active":"2024-03-23T03:49:36.435553+0000","last_peered":"2024-03-23T03:49:36.435553+0000","last_clean":"2024-03-23T03:49:36.435553+0000","last_became_active":"2024-03-23T03:49:35.456971+0000","last_became_peered":"2024-03-23T03:49:35.456971+0000","last_unstale":"2024-03-23T03:49:36.435553+0000","last_undegraded":"2024-03-23T03:49:36.435553+0000","last_fullsized":"2024-03-23T03:49:36.435553+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:18:35.380354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"66'1","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.454992+0000","last_change":"2024-03-23T03:49:34.970357+0000","last_active":"2024-03-23T03:49:35.454992+0000","last_peered":"2024-03-23T03:49:35.454992+0000","last_clean":"2024-03-23T03:49:35.454992+0000","last_became_active":"2024-03-23T03:49:34.968925+0000","last_became_peered":"2024-03-23T03:49:34.968925+0000","last_unstale":"2024-03-23T03:49:35.454992+0000","last_undegraded":"2024-03-23T03:49:35.454992+0000","last_fullsized":"2024-03-23T03:49:35.454992+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:36:41.972438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.463066+0000","last_change":"2024-03-23T03:49:35.461068+0000","last_active":"2024-03-23T03:49:35.463066+0000","last_peered":"2024-03-23T03:49:35.463066+0000","last_clean":"2024-03-23T03:49:35.463066+0000","last_became_active":"2024-03-23T03:49:35.459712+0000","last_became_peered":"2024-03-23T03:49:35.459712+0000","last_unstale":"2024-03-23T03:49:35.463066+0000","last_undegraded":"2024-03-23T03:49:35.463066+0000","last_fullsized":"2024-03-23T03:49:35.463066+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:02:27.532316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.433277+0000","last_change":"2024-03-23T03:49:34.988264+0000","last_active":"2024-03-23T03:49:35.433277+0000","last_peered":"2024-03-23T03:49:35.433277+0000","last_clean":"2024-03-23T03:49:35.433277+0000","last_became_active":"2024-03-23T03:49:34.987918+0000","last_became_peered":"2024-03-23T03:49:34.987918+0000","last_unstale":"2024-03-23T03:49:35.433277+0000","last_undegraded":"2024-03-23T03:49:35.433277+0000","last_fullsized":"2024-03-23T03:49:35.433277+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:52:53.648955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.464566+0000","last_change":"2024-03-23T03:49:35.463629+0000","last_active":"2024-03-23T03:49:35.464566+0000","last_peered":"2024-03-23T03:49:35.464566+0000","last_clean":"2024-03-23T03:49:35.464566+0000","last_became_active":"2024-03-23T03:49:35.463417+0000","last_became_peered":"2024-03-23T03:49:35.463417+0000","last_unstale":"2024-03-23T03:49:35.464566+0000","last_undegraded":"2024-03-23T03:49:35.464566+0000","last_fullsized":"2024-03-23T03:49:35.464566+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:37:55.139651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.431787+0000","last_change":"2024-03-23T03:49:34.990891+0000","last_active":"2024-03-23T03:49:35.431787+0000","last_peered":"2024-03-23T03:49:35.431787+0000","last_clean":"2024-03-23T03:49:35.431787+0000","last_became_active":"2024-03-23T03:49:34.990610+0000","last_became_peered":"2024-03-23T03:49:34.990610+0000","last_unstale":"2024-03-23T03:49:35.431787+0000","last_undegraded":"2024-03-23T03:49:35.431787+0000","last_fullsized":"2024-03-23T03:49:35.431787+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:27:38.167471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436413+0000","last_change":"2024-03-23T03:49:35.456262+0000","last_active":"2024-03-23T03:49:36.436413+0000","last_peered":"2024-03-23T03:49:36.436413+0000","last_clean":"2024-03-23T03:49:36.436413+0000","last_became_active":"2024-03-23T03:49:35.456043+0000","last_became_peered":"2024-03-23T03:49:35.456043+0000","last_unstale":"2024-03-23T03:49:36.436413+0000","last_undegraded":"2024-03-23T03:49:36.436413+0000","last_fullsized":"2024-03-23T03:49:36.436413+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:48:11.607155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.436947+0000","last_change":"2024-03-23T03:49:34.992625+0000","last_active":"2024-03-23T03:49:35.436947+0000","last_peered":"2024-03-23T03:49:35.436947+0000","last_clean":"2024-03-23T03:49:35.436947+0000","last_became_active":"2024-03-23T03:49:34.992007+0000","last_became_peered":"2024-03-23T03:49:34.992007+0000","last_unstale":"2024-03-23T03:49:35.436947+0000","last_undegraded":"2024-03-23T03:49:35.436947+0000","last_fullsized":"2024-03-23T03:49:35.436947+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:12:14.699545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.458195+0000","last_change":"2024-03-23T03:49:35.457436+0000","last_active":"2024-03-23T03:49:35.458195+0000","last_peered":"2024-03-23T03:49:35.458195+0000","last_clean":"2024-03-23T03:49:35.458195+0000","last_became_active":"2024-03-23T03:49:35.457171+0000","last_became_peered":"2024-03-23T03:49:35.457171+0000","last_unstale":"2024-03-23T03:49:35.458195+0000","last_undegraded":"2024-03-23T03:49:35.458195+0000","last_fullsized":"2024-03-23T03:49:35.458195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:14:01.589671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.437429+0000","last_change":"2024-03-23T03:49:34.981367+0000","last_active":"2024-03-23T03:49:35.437429+0000","last_peered":"2024-03-23T03:49:35.437429+0000","last_clean":"2024-03-23T03:49:35.437429+0000","last_became_active":"2024-03-23T03:49:34.981078+0000","last_became_peered":"2024-03-23T03:49:34.981078+0000","last_unstale":"2024-03-23T03:49:35.437429+0000","last_undegraded":"2024-03-23T03:49:35.437429+0000","last_fullsized":"2024-03-23T03:49:35.437429+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:39.665461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.454580+0000","last_change":"2024-03-23T03:49:35.447863+0000","last_active":"2024-03-23T03:49:35.454580+0000","last_peered":"2024-03-23T03:49:35.454580+0000","last_clean":"2024-03-23T03:49:35.454580+0000","last_became_active":"2024-03-23T03:49:35.447368+0000","last_became_peered":"2024-03-23T03:49:35.447368+0000","last_unstale":"2024-03-23T03:49:35.454580+0000","last_undegraded":"2024-03-23T03:49:35.454580+0000","last_fullsized":"2024-03-23T03:49:35.454580+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:08.096604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"66'3","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.461741+0000","last_change":"2024-03-23T03:49:34.992577+0000","last_active":"2024-03-23T03:49:35.461741+0000","last_peered":"2024-03-23T03:49:35.461741+0000","last_clean":"2024-03-23T03:49:35.461741+0000","last_became_active":"2024-03-23T03:49:34.991853+0000","last_became_peered":"2024-03-23T03:49:34.991853+0000","last_unstale":"2024-03-23T03:49:35.461741+0000","last_undegraded":"2024-03-23T03:49:35.461741+0000","last_fullsized":"2024-03-23T03:49:35.461741+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:45:28.059241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.462822+0000","last_change":"2024-03-23T03:49:35.447025+0000","last_active":"2024-03-23T03:49:35.462822+0000","last_peered":"2024-03-23T03:49:35.462822+0000","last_clean":"2024-03-23T03:49:35.462822+0000","last_became_active":"2024-03-23T03:49:35.446231+0000","last_became_peered":"2024-03-23T03:49:35.446231+0000","last_unstale":"2024-03-23T03:49:35.462822+0000","last_undegraded":"2024-03-23T03:49:35.462822+0000","last_fullsized":"2024-03-23T03:49:35.462822+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:16:15.365194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"66'6","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436596+0000","last_change":"2024-03-23T03:49:34.982535+0000","last_active":"2024-03-23T03:49:36.436596+0000","last_peered":"2024-03-23T03:49:36.436596+0000","last_clean":"2024-03-23T03:49:36.436596+0000","last_became_active":"2024-03-23T03:49:34.982293+0000","last_became_peered":"2024-03-23T03:49:34.982293+0000","last_unstale":"2024-03-23T03:49:36.436596+0000","last_undegraded":"2024-03-23T03:49:36.436596+0000","last_fullsized":"2024-03-23T03:49:36.436596+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:59:28.803065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.465237+0000","last_change":"2024-03-23T03:49:35.451899+0000","last_active":"2024-03-23T03:49:35.465237+0000","last_peered":"2024-03-23T03:49:35.465237+0000","last_clean":"2024-03-23T03:49:35.465237+0000","last_became_active":"2024-03-23T03:49:35.450542+0000","last_became_peered":"2024-03-23T03:49:35.450542+0000","last_unstale":"2024-03-23T03:49:35.465237+0000","last_undegraded":"2024-03-23T03:49:35.465237+0000","last_fullsized":"2024-03-23T03:49:35.465237+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:40:59.178993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"66'2","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.462261+0000","last_change":"2024-03-23T03:49:34.992490+0000","last_active":"2024-03-23T03:49:35.462261+0000","last_peered":"2024-03-23T03:49:35.462261+0000","last_clean":"2024-03-23T03:49:35.462261+0000","last_became_active":"2024-03-23T03:49:34.992098+0000","last_became_peered":"2024-03-23T03:49:34.992098+0000","last_unstale":"2024-03-23T03:49:35.462261+0000","last_undegraded":"2024-03-23T03:49:35.462261+0000","last_fullsized":"2024-03-23T03:49:35.462261+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:00:19.043728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.435960+0000","last_change":"2024-03-23T03:49:35.441503+0000","last_active":"2024-03-23T03:49:36.435960+0000","last_peered":"2024-03-23T03:49:36.435960+0000","last_clean":"2024-03-23T03:49:36.435960+0000","last_became_active":"2024-03-23T03:49:35.441247+0000","last_became_peered":"2024-03-23T03:49:35.441247+0000","last_unstale":"2024-03-23T03:49:36.435960+0000","last_undegraded":"2024-03-23T03:49:36.435960+0000","last_fullsized":"2024-03-23T03:49:36.435960+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:55:05.527463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.435974+0000","last_change":"2024-03-23T03:49:34.981217+0000","last_active":"2024-03-23T03:49:36.435974+0000","last_peered":"2024-03-23T03:49:36.435974+0000","last_clean":"2024-03-23T03:49:36.435974+0000","last_became_active":"2024-03-23T03:49:34.976302+0000","last_became_peered":"2024-03-23T03:49:34.976302+0000","last_unstale":"2024-03-23T03:49:36.435974+0000","last_undegraded":"2024-03-23T03:49:36.435974+0000","last_fullsized":"2024-03-23T03:49:36.435974+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:12:56.894905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.460257+0000","last_change":"2024-03-23T03:49:35.448205+0000","last_active":"2024-03-23T03:49:35.460257+0000","last_peered":"2024-03-23T03:49:35.460257+0000","last_clean":"2024-03-23T03:49:35.460257+0000","last_became_active":"2024-03-23T03:49:35.447452+0000","last_became_peered":"2024-03-23T03:49:35.447452+0000","last_unstale":"2024-03-23T03:49:35.460257+0000","last_undegraded":"2024-03-23T03:49:35.460257+0000","last_fullsized":"2024-03-23T03:49:35.460257+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:16:47.272585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.437354+0000","last_change":"2024-03-23T03:49:34.985827+0000","last_active":"2024-03-23T03:49:35.437354+0000","last_peered":"2024-03-23T03:49:35.437354+0000","last_clean":"2024-03-23T03:49:35.437354+0000","last_became_active":"2024-03-23T03:49:34.985545+0000","last_became_peered":"2024-03-23T03:49:34.985545+0000","last_unstale":"2024-03-23T03:49:35.437354+0000","last_undegraded":"2024-03-23T03:49:35.437354+0000","last_fullsized":"2024-03-23T03:49:35.437354+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:16:19.571022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436196+0000","last_change":"2024-03-23T03:49:35.453213+0000","last_active":"2024-03-23T03:49:36.436196+0000","last_peered":"2024-03-23T03:49:36.436196+0000","last_clean":"2024-03-23T03:49:36.436196+0000","last_became_active":"2024-03-23T03:49:35.446364+0000","last_became_peered":"2024-03-23T03:49:35.446364+0000","last_unstale":"2024-03-23T03:49:36.436196+0000","last_undegraded":"2024-03-23T03:49:36.436196+0000","last_fullsized":"2024-03-23T03:49:36.436196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:59.256052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"66'8","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.467523+0000","last_change":"2024-03-23T03:49:34.988658+0000","last_active":"2024-03-23T03:49:35.467523+0000","last_peered":"2024-03-23T03:49:35.467523+0000","last_clean":"2024-03-23T03:49:35.467523+0000","last_became_active":"2024-03-23T03:49:34.988406+0000","last_became_peered":"2024-03-23T03:49:34.988406+0000","last_unstale":"2024-03-23T03:49:35.467523+0000","last_undegraded":"2024-03-23T03:49:35.467523+0000","last_fullsized":"2024-03-23T03:49:35.467523+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:03:13.452675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.467398+0000","last_change":"2024-03-23T03:49:35.465262+0000","last_active":"2024-03-23T03:49:35.467398+0000","last_peered":"2024-03-23T03:49:35.467398+0000","last_clean":"2024-03-23T03:49:35.467398+0000","last_became_active":"2024-03-23T03:49:35.464461+0000","last_became_peered":"2024-03-23T03:49:35.464461+0000","last_unstale":"2024-03-23T03:49:35.467398+0000","last_undegraded":"2024-03-23T03:49:35.467398+0000","last_fullsized":"2024-03-23T03:49:35.467398+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:48:11.607155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"66'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436519+0000","last_change":"2024-03-23T03:49:34.974809+0000","last_active":"2024-03-23T03:49:36.436519+0000","last_peered":"2024-03-23T03:49:36.436519+0000","last_clean":"2024-03-23T03:49:36.436519+0000","last_became_active":"2024-03-23T03:49:34.974425+0000","last_became_peered":"2024-03-23T03:49:34.974425+0000","last_unstale":"2024-03-23T03:49:36.436519+0000","last_undegraded":"2024-03-23T03:49:36.436519+0000","last_fullsized":"2024-03-23T03:49:36.436519+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:59:28.803065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.465694+0000","last_change":"2024-03-23T03:49:35.463979+0000","last_active":"2024-03-23T03:49:35.465694+0000","last_peered":"2024-03-23T03:49:35.465694+0000","last_clean":"2024-03-23T03:49:35.465694+0000","last_became_active":"2024-03-23T03:49:35.463719+0000","last_became_peered":"2024-03-23T03:49:35.463719+0000","last_unstale":"2024-03-23T03:49:35.465694+0000","last_undegraded":"2024-03-23T03:49:35.465694+0000","last_fullsized":"2024-03-23T03:49:35.465694+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:44:43.035657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"66'2","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.463585+0000","last_change":"2024-03-23T03:49:34.992119+0000","last_active":"2024-03-23T03:49:35.463585+0000","last_peered":"2024-03-23T03:49:35.463585+0000","last_clean":"2024-03-23T03:49:35.463585+0000","last_became_active":"2024-03-23T03:49:34.991843+0000","last_became_peered":"2024-03-23T03:49:34.991843+0000","last_unstale":"2024-03-23T03:49:35.463585+0000","last_undegraded":"2024-03-23T03:49:35.463585+0000","last_fullsized":"2024-03-23T03:49:35.463585+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:03:16.437712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.468239+0000","last_change":"2024-03-23T03:49:35.464324+0000","last_active":"2024-03-23T03:49:35.468239+0000","last_peered":"2024-03-23T03:49:35.468239+0000","last_clean":"2024-03-23T03:49:35.468239+0000","last_became_active":"2024-03-23T03:49:35.464019+0000","last_became_peered":"2024-03-23T03:49:35.464019+0000","last_unstale":"2024-03-23T03:49:35.468239+0000","last_undegraded":"2024-03-23T03:49:35.468239+0000","last_fullsized":"2024-03-23T03:49:35.468239+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:58:39.422775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"66'1","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.461619+0000","last_change":"2024-03-23T03:49:34.992779+0000","last_active":"2024-03-23T03:49:35.461619+0000","last_peered":"2024-03-23T03:49:35.461619+0000","last_clean":"2024-03-23T03:49:35.461619+0000","last_became_active":"2024-03-23T03:49:34.992252+0000","last_became_peered":"2024-03-23T03:49:34.992252+0000","last_unstale":"2024-03-23T03:49:35.461619+0000","last_undegraded":"2024-03-23T03:49:35.461619+0000","last_fullsized":"2024-03-23T03:49:35.461619+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:10:01.095131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.455634+0000","last_change":"2024-03-23T03:49:35.448284+0000","last_active":"2024-03-23T03:49:35.455634+0000","last_peered":"2024-03-23T03:49:35.455634+0000","last_clean":"2024-03-23T03:49:35.455634+0000","last_became_active":"2024-03-23T03:49:35.447986+0000","last_became_peered":"2024-03-23T03:49:35.447986+0000","last_unstale":"2024-03-23T03:49:35.455634+0000","last_undegraded":"2024-03-23T03:49:35.455634+0000","last_fullsized":"2024-03-23T03:49:35.455634+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:32:51.943312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.435546+0000","last_change":"2024-03-23T03:49:34.987012+0000","last_active":"2024-03-23T03:49:35.435546+0000","last_peered":"2024-03-23T03:49:35.435546+0000","last_clean":"2024-03-23T03:49:35.435546+0000","last_became_active":"2024-03-23T03:49:34.986690+0000","last_became_peered":"2024-03-23T03:49:34.986690+0000","last_unstale":"2024-03-23T03:49:35.435546+0000","last_undegraded":"2024-03-23T03:49:35.435546+0000","last_fullsized":"2024-03-23T03:49:35.435546+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:35:00.404847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650831+0000","last_change":"2024-03-23T03:49:35.457675+0000","last_active":"2024-03-23T03:49:36.650831+0000","last_peered":"2024-03-23T03:49:36.650831+0000","last_clean":"2024-03-23T03:49:36.650831+0000","last_became_active":"2024-03-23T03:49:35.457421+0000","last_became_peered":"2024-03-23T03:49:35.457421+0000","last_unstale":"2024-03-23T03:49:36.650831+0000","last_undegraded":"2024-03-23T03:49:36.650831+0000","last_fullsized":"2024-03-23T03:49:36.650831+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:36:42.984257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650294+0000","last_change":"2024-03-23T03:49:34.988857+0000","last_active":"2024-03-23T03:49:36.650294+0000","last_peered":"2024-03-23T03:49:36.650294+0000","last_clean":"2024-03-23T03:49:36.650294+0000","last_became_active":"2024-03-23T03:49:34.988325+0000","last_became_peered":"2024-03-23T03:49:34.988325+0000","last_unstale":"2024-03-23T03:49:36.650294+0000","last_undegraded":"2024-03-23T03:49:36.650294+0000","last_fullsized":"2024-03-23T03:49:36.650294+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:51:17.672508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.435911+0000","last_change":"2024-03-23T03:49:35.438748+0000","last_active":"2024-03-23T03:49:36.435911+0000","last_peered":"2024-03-23T03:49:36.435911+0000","last_clean":"2024-03-23T03:49:36.435911+0000","last_became_active":"2024-03-23T03:49:35.438469+0000","last_became_peered":"2024-03-23T03:49:35.438469+0000","last_unstale":"2024-03-23T03:49:36.435911+0000","last_undegraded":"2024-03-23T03:49:36.435911+0000","last_fullsized":"2024-03-23T03:49:36.435911+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:50:23.362975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436060+0000","last_change":"2024-03-23T03:49:34.976528+0000","last_active":"2024-03-23T03:49:36.436060+0000","last_peered":"2024-03-23T03:49:36.436060+0000","last_clean":"2024-03-23T03:49:36.436060+0000","last_became_active":"2024-03-23T03:49:34.976051+0000","last_became_peered":"2024-03-23T03:49:34.976051+0000","last_unstale":"2024-03-23T03:49:36.436060+0000","last_undegraded":"2024-03-23T03:49:36.436060+0000","last_fullsized":"2024-03-23T03:49:36.436060+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:12:35.311936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.460279+0000","last_change":"2024-03-23T03:49:35.453603+0000","last_active":"2024-03-23T03:49:35.460279+0000","last_peered":"2024-03-23T03:49:35.460279+0000","last_clean":"2024-03-23T03:49:35.460279+0000","last_became_active":"2024-03-23T03:49:35.449568+0000","last_became_peered":"2024-03-23T03:49:35.449568+0000","last_unstale":"2024-03-23T03:49:35.460279+0000","last_undegraded":"2024-03-23T03:49:35.460279+0000","last_fullsized":"2024-03-23T03:49:35.460279+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:44:03.577479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650560+0000","last_change":"2024-03-23T03:49:34.987803+0000","last_active":"2024-03-23T03:49:36.650560+0000","last_peered":"2024-03-23T03:49:36.650560+0000","last_clean":"2024-03-23T03:49:36.650560+0000","last_became_active":"2024-03-23T03:49:34.987511+0000","last_became_peered":"2024-03-23T03:49:34.987511+0000","last_unstale":"2024-03-23T03:49:36.650560+0000","last_undegraded":"2024-03-23T03:49:36.650560+0000","last_fullsized":"2024-03-23T03:49:36.650560+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:45:30.133076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650773+0000","last_change":"2024-03-23T03:49:35.464179+0000","last_active":"2024-03-23T03:49:36.650773+0000","last_peered":"2024-03-23T03:49:36.650773+0000","last_clean":"2024-03-23T03:49:36.650773+0000","last_became_active":"2024-03-23T03:49:35.463860+0000","last_became_peered":"2024-03-23T03:49:35.463860+0000","last_unstale":"2024-03-23T03:49:36.650773+0000","last_undegraded":"2024-03-23T03:49:36.650773+0000","last_fullsized":"2024-03-23T03:49:36.650773+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:02:35.537184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436667+0000","last_change":"2024-03-23T03:49:34.986922+0000","last_active":"2024-03-23T03:49:36.436667+0000","last_peered":"2024-03-23T03:49:36.436667+0000","last_clean":"2024-03-23T03:49:36.436667+0000","last_became_active":"2024-03-23T03:49:34.986632+0000","last_became_peered":"2024-03-23T03:49:34.986632+0000","last_unstale":"2024-03-23T03:49:36.436667+0000","last_undegraded":"2024-03-23T03:49:36.436667+0000","last_fullsized":"2024-03-23T03:49:36.436667+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:57:01.780866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.459687+0000","last_change":"2024-03-23T03:49:35.454834+0000","last_active":"2024-03-23T03:49:35.459687+0000","last_peered":"2024-03-23T03:49:35.459687+0000","last_clean":"2024-03-23T03:49:35.459687+0000","last_became_active":"2024-03-23T03:49:35.454564+0000","last_became_peered":"2024-03-23T03:49:35.454564+0000","last_unstale":"2024-03-23T03:49:35.459687+0000","last_undegraded":"2024-03-23T03:49:35.459687+0000","last_fullsized":"2024-03-23T03:49:35.459687+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:30:24.052315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436410+0000","last_change":"2024-03-23T03:49:34.974283+0000","last_active":"2024-03-23T03:49:36.436410+0000","last_peered":"2024-03-23T03:49:36.436410+0000","last_clean":"2024-03-23T03:49:36.436410+0000","last_became_active":"2024-03-23T03:49:34.973333+0000","last_became_peered":"2024-03-23T03:49:34.973333+0000","last_unstale":"2024-03-23T03:49:36.436410+0000","last_undegraded":"2024-03-23T03:49:36.436410+0000","last_fullsized":"2024-03-23T03:49:36.436410+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:47:27.337094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436244+0000","last_change":"2024-03-23T03:49:35.445810+0000","last_active":"2024-03-23T03:49:36.436244+0000","last_peered":"2024-03-23T03:49:36.436244+0000","last_clean":"2024-03-23T03:49:36.436244+0000","last_became_active":"2024-03-23T03:49:35.445512+0000","last_became_peered":"2024-03-23T03:49:35.445512+0000","last_unstale":"2024-03-23T03:49:36.436244+0000","last_undegraded":"2024-03-23T03:49:36.436244+0000","last_fullsized":"2024-03-23T03:49:36.436244+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:42:12.588157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650894+0000","last_change":"2024-03-23T03:49:34.984506+0000","last_active":"2024-03-23T03:49:36.650894+0000","last_peered":"2024-03-23T03:49:36.650894+0000","last_clean":"2024-03-23T03:49:36.650894+0000","last_became_active":"2024-03-23T03:49:34.983529+0000","last_became_peered":"2024-03-23T03:49:34.983529+0000","last_unstale":"2024-03-23T03:49:36.650894+0000","last_undegraded":"2024-03-23T03:49:36.650894+0000","last_fullsized":"2024-03-23T03:49:36.650894+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:49:11.082091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.454744+0000","last_change":"2024-03-23T03:49:35.447908+0000","last_active":"2024-03-23T03:49:35.454744+0000","last_peered":"2024-03-23T03:49:35.454744+0000","last_clean":"2024-03-23T03:49:35.454744+0000","last_became_active":"2024-03-23T03:49:35.443188+0000","last_became_peered":"2024-03-23T03:49:35.443188+0000","last_unstale":"2024-03-23T03:49:35.454744+0000","last_undegraded":"2024-03-23T03:49:35.454744+0000","last_fullsized":"2024-03-23T03:49:35.454744+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:55:49.182877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436584+0000","last_change":"2024-03-23T03:49:34.980078+0000","last_active":"2024-03-23T03:49:36.436584+0000","last_peered":"2024-03-23T03:49:36.436584+0000","last_clean":"2024-03-23T03:49:36.436584+0000","last_became_active":"2024-03-23T03:49:34.979815+0000","last_became_peered":"2024-03-23T03:49:34.979815+0000","last_unstale":"2024-03-23T03:49:36.436584+0000","last_undegraded":"2024-03-23T03:49:36.436584+0000","last_fullsized":"2024-03-23T03:49:36.436584+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:12:56.894905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.458135+0000","last_change":"2024-03-23T03:49:35.456042+0000","last_active":"2024-03-23T03:49:35.458135+0000","last_peered":"2024-03-23T03:49:35.458135+0000","last_clean":"2024-03-23T03:49:35.458135+0000","last_became_active":"2024-03-23T03:49:35.450458+0000","last_became_peered":"2024-03-23T03:49:35.450458+0000","last_unstale":"2024-03-23T03:49:35.458135+0000","last_undegraded":"2024-03-23T03:49:35.458135+0000","last_fullsized":"2024-03-23T03:49:35.458135+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:14:12.558627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.436609+0000","last_change":"2024-03-23T03:49:34.987542+0000","last_active":"2024-03-23T03:49:35.436609+0000","last_peered":"2024-03-23T03:49:35.436609+0000","last_clean":"2024-03-23T03:49:35.436609+0000","last_became_active":"2024-03-23T03:49:34.986389+0000","last_became_peered":"2024-03-23T03:49:34.986389+0000","last_unstale":"2024-03-23T03:49:35.436609+0000","last_undegraded":"2024-03-23T03:49:35.436609+0000","last_fullsized":"2024-03-23T03:49:35.436609+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:32:48.810886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.468200+0000","last_change":"2024-03-23T03:49:35.463865+0000","last_active":"2024-03-23T03:49:35.468200+0000","last_peered":"2024-03-23T03:49:35.468200+0000","last_clean":"2024-03-23T03:49:35.468200+0000","last_became_active":"2024-03-23T03:49:35.463091+0000","last_became_peered":"2024-03-23T03:49:35.463091+0000","last_unstale":"2024-03-23T03:49:35.468200+0000","last_undegraded":"2024-03-23T03:49:35.468200+0000","last_fullsized":"2024-03-23T03:49:35.468200+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:48:18.905476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.433420+0000","last_change":"2024-03-23T03:49:34.967418+0000","last_active":"2024-03-23T03:49:35.433420+0000","last_peered":"2024-03-23T03:49:35.433420+0000","last_clean":"2024-03-23T03:49:35.433420+0000","last_became_active":"2024-03-23T03:49:34.967079+0000","last_became_peered":"2024-03-23T03:49:34.967079+0000","last_unstale":"2024-03-23T03:49:35.433420+0000","last_undegraded":"2024-03-23T03:49:35.433420+0000","last_fullsized":"2024-03-23T03:49:35.433420+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:45:59.962877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.435897+0000","last_change":"2024-03-23T03:49:35.446298+0000","last_active":"2024-03-23T03:49:36.435897+0000","last_peered":"2024-03-23T03:49:36.435897+0000","last_clean":"2024-03-23T03:49:36.435897+0000","last_became_active":"2024-03-23T03:49:35.446060+0000","last_became_peered":"2024-03-23T03:49:35.446060+0000","last_unstale":"2024-03-23T03:49:36.435897+0000","last_undegraded":"2024-03-23T03:49:36.435897+0000","last_fullsized":"2024-03-23T03:49:36.435897+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:24:31.514455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.433328+0000","last_change":"2024-03-23T03:49:34.985626+0000","last_active":"2024-03-23T03:49:35.433328+0000","last_peered":"2024-03-23T03:49:35.433328+0000","last_clean":"2024-03-23T03:49:35.433328+0000","last_became_active":"2024-03-23T03:49:34.985354+0000","last_became_peered":"2024-03-23T03:49:34.985354+0000","last_unstale":"2024-03-23T03:49:35.433328+0000","last_undegraded":"2024-03-23T03:49:35.433328+0000","last_fullsized":"2024-03-23T03:49:35.433328+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:35:48.939105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.454591+0000","last_change":"2024-03-23T03:49:35.448917+0000","last_active":"2024-03-23T03:49:35.454591+0000","last_peered":"2024-03-23T03:49:35.454591+0000","last_clean":"2024-03-23T03:49:35.454591+0000","last_became_active":"2024-03-23T03:49:35.448608+0000","last_became_peered":"2024-03-23T03:49:35.448608+0000","last_unstale":"2024-03-23T03:49:35.454591+0000","last_undegraded":"2024-03-23T03:49:35.454591+0000","last_fullsized":"2024-03-23T03:49:35.454591+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:52:01.988987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.435953+0000","last_change":"2024-03-23T03:49:34.988815+0000","last_active":"2024-03-23T03:49:36.435953+0000","last_peered":"2024-03-23T03:49:36.435953+0000","last_clean":"2024-03-23T03:49:36.435953+0000","last_became_active":"2024-03-23T03:49:34.988496+0000","last_became_peered":"2024-03-23T03:49:34.988496+0000","last_unstale":"2024-03-23T03:49:36.435953+0000","last_undegraded":"2024-03-23T03:49:36.435953+0000","last_fullsized":"2024-03-23T03:49:36.435953+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:57:01.780866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.462761+0000","last_change":"2024-03-23T03:49:35.460666+0000","last_active":"2024-03-23T03:49:35.462761+0000","last_peered":"2024-03-23T03:49:35.462761+0000","last_clean":"2024-03-23T03:49:35.462761+0000","last_became_active":"2024-03-23T03:49:35.459729+0000","last_became_peered":"2024-03-23T03:49:35.459729+0000","last_unstale":"2024-03-23T03:49:35.462761+0000","last_undegraded":"2024-03-23T03:49:35.462761+0000","last_fullsized":"2024-03-23T03:49:35.462761+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:40.677281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.435008+0000","last_change":"2024-03-23T03:49:34.972259+0000","last_active":"2024-03-23T03:49:35.435008+0000","last_peered":"2024-03-23T03:49:35.435008+0000","last_clean":"2024-03-23T03:49:35.435008+0000","last_became_active":"2024-03-23T03:49:34.972003+0000","last_became_peered":"2024-03-23T03:49:34.972003+0000","last_unstale":"2024-03-23T03:49:35.435008+0000","last_undegraded":"2024-03-23T03:49:35.435008+0000","last_fullsized":"2024-03-23T03:49:35.435008+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:41:06.334006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":201,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-23T03:49:34.972946+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:49:34.972946+0000","last_peered":"2024-03-23T03:49:34.972946+0000","last_clean":"2024-03-23T03:49:34.972946+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:49:34.972946+0000","last_undegraded":"2024-03-23T03:49:34.972946+0000","last_fullsized":"2024-03-23T03:49:34.972946+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542182,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28400,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343440,"statfs":{"total":96636764160,"available":96607682560,"internally_reserved":0,"allocated":1404928,"data_stored":943779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.079,"15min":1.069},"min":{"1min":0.66800000000000004,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.6279999999999999,"5min":1.635,"15min":1.635},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0569999999999999,"15min":1.052},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.48,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.67100000000000004}]},{"osd":1,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.0589999999999999,"15min":1.0449999999999999},"min":{"1min":0.70099999999999996,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.655,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.2829999999999999,"15min":1.3180000000000001},"min":{"1min":0.72599999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.472,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":1.0960000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.147,"15min":1.159},"min":{"1min":0.79200000000000004,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.4870000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.198,"5min":1.4059999999999999,"15min":1.4410000000000001},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.522,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.3500000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.4239999999999999,"15min":1.46},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.5900000000000001,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.044},{"interface":"front","average":{"1min":1.319,"5min":1.3580000000000001,"15min":1.3640000000000001},"min":{"1min":1.046,"5min":1.0429999999999999,"15min":1.0429999999999999},"max":{"1min":1.536,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.22}]},{"osd":4,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.2789999999999999,"15min":1.2709999999999999},"min":{"1min":0.98299999999999998,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.9370000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.381},{"interface":"front","average":{"1min":1.367,"5min":1.381,"15min":1.383},"min":{"1min":0.94899999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.9710000000000001,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.1339999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4350000000000001,"15min":1.4330000000000001},"min":{"1min":0.97699999999999998,"5min":0.93400000000000005,"15min":0.93400000000000005},"max":{"1min":1.7330000000000001,"5min":2.645,"15min":2.645},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"min":{"1min":1.0229999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"max":{"1min":2.1930000000000001,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.4670000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.5620000000000001,"15min":1.5720000000000001},"min":{"1min":1.2569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"max":{"1min":1.754,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.444,"5min":1.5169999999999999,"15min":1.5289999999999999},"min":{"1min":1.0489999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":2.0030000000000001,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.5549999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542180,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":798720,"data_stored":352669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.0369999999999999,"15min":1.042},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0429999999999999,"15min":1.038},"min":{"1min":0.74099999999999999,"5min":0.62,"15min":0.62},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":1.3180000000000001}]},{"osd":1,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0649999999999999,"15min":1.0669999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.3939999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.145,"15min":1.161},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.405,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.2729999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0680000000000001,"15min":1.0660000000000001},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.044},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.212,"15min":1.224},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.419,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":1.135}]},{"osd":3,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.2749999999999999,"15min":1.2889999999999999},"min":{"1min":0.82899999999999996,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.643,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.2589999999999999,"15min":1.266},"min":{"1min":0.89600000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.4870000000000001,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.5029999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.383,"15min":1.387},"min":{"1min":1.0860000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.734,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.363},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.379,"15min":1.389},"min":{"1min":0.871,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.5860000000000001,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.4099999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.431,"15min":1.4410000000000001},"min":{"1min":1.0029999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.7370000000000001,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.3},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.546,"15min":1.554},"min":{"1min":1.2410000000000001,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8220000000000001,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.4379999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.419,"15min":1.4159999999999999},"min":{"1min":1.0269999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.8460000000000001,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.163},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.54,"15min":1.55},"min":{"1min":1.018,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.802,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.343}]}]},{"osd":5,"up_from":14,"seq":60129542180,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":798720,"data_stored":352657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.1950000000000001,"15min":1.22},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.5,"5min":1.806,"15min":1.806},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.129,"5min":1.181,"15min":1.1890000000000001},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.6930000000000001,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.3049999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.671,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.153,"5min":1.1970000000000001,"15min":1.204},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.8180000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.036}]},{"osd":2,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.345,"15min":1.369},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.4950000000000001,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.361,"15min":1.387},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.556,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.84099999999999997}]},{"osd":3,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.2450000000000001,"15min":1.2589999999999999},"min":{"1min":0.73699999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.4950000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.298,"5min":1.4450000000000001,"15min":1.4690000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.925,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.385}]},{"osd":4,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.399,"15min":1.431},"min":{"1min":0.77600000000000002,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.8029999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.121},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.448,"15min":1.456},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":2.0209999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.3360000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.5,"15min":1.528},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.7390000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.444,"5min":1.639,"15min":1.671},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.8049999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.417}]},{"osd":7,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.599,"15min":1.627},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":2.0059999999999998,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.6850000000000001,"15min":1.7170000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.0289999999999999,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.446}]}]},{"osd":4,"up_from":14,"seq":60129542181,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":806912,"data_stored":354199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.97599999999999998,"15min":0.97199999999999998},"min":{"1min":0.69799999999999995,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.26,"5min":1.383,"15min":1.383},"last":1.038},{"interface":"front","average":{"1min":1.012,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.69499999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.4950000000000001,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.117}]},{"osd":1,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.0509999999999999,"15min":1.044},"min":{"1min":0.78400000000000003,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.3759999999999999,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.139,"5min":1.069,"15min":1.0569999999999999},"min":{"1min":0.86699999999999999,"5min":0.496,"15min":0.496},"max":{"1min":1.387,"5min":1.4159999999999999,"15min":1.4159999999999999},"last":1.214}]},{"osd":2,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.3109999999999999,"15min":1.331},"min":{"1min":0.88200000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.464,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":1.472},{"interface":"front","average":{"1min":1.343,"5min":1.3140000000000001,"15min":1.3089999999999999},"min":{"1min":0.90400000000000003,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.6000000000000001,"5min":2.181,"15min":2.181},"last":1.4410000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.1220000000000001,"15min":1.1259999999999999},"min":{"1min":0.66800000000000004,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.423,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.272},{"interface":"front","average":{"1min":1.212,"5min":1.272,"15min":1.282},"min":{"1min":0.84599999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.548,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.173}]},{"osd":5,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2729999999999999,"15min":1.272},"min":{"1min":0.69499999999999995,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.661,"5min":1.841,"15min":1.841},"last":1.363},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.28,"15min":1.2769999999999999},"min":{"1min":0.72199999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.589,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.399}]},{"osd":6,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.4059999999999999,"15min":1.421},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.6479999999999999,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.466,"5min":1.4079999999999999,"15min":1.3979999999999999},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.7789999999999999,"5min":1.877,"15min":1.877},"last":1.542}]},{"osd":7,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3660000000000001,"15min":1.3640000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.716,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":1.498},{"interface":"front","average":{"1min":1.397,"5min":1.4810000000000001,"15min":1.4950000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.651,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.5660000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542181,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":794624,"data_stored":352635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.86399999999999999,"15min":0.87},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.37},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.93100000000000005,"15min":0.92700000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":5.8499999999999996,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.33800000000000002}]},{"osd":1,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.97499999999999998,"15min":0.98799999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.056,"15min":1.0800000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.7450000000000001,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.44900000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":1.04,"15min":1.0640000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.679,"5min":1.823,"15min":1.823},"last":0.435},{"interface":"front","average":{"1min":1.127,"5min":1.1559999999999999,"15min":1.161},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.8010000000000002,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":0.48199999999999998}]},{"osd":4,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.262,"15min":1.2549999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":5.9210000000000003,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.333,"5min":1.4410000000000001,"15min":1.4590000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":4.5620000000000003,"5min":4.819,"15min":4.819},"last":0.77100000000000002}]},{"osd":5,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.6640000000000001,"5min":1.6279999999999999,"15min":1.4550000000000001},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":30.228000000000002,"5min":30.228000000000002,"15min":30.228000000000002},"last":0.75},{"interface":"front","average":{"1min":1.373,"5min":1.3680000000000001,"15min":1.367},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":5.8920000000000003,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":0.91400000000000003}]},{"osd":6,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.506,"15min":1.54},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.9319999999999999,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.407,"15min":1.399},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":4.7610000000000001,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":0.96799999999999997}]},{"osd":7,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.429,"15min":1.4330000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":5.9589999999999996,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.6399999999999999,"15min":1.6220000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":10.663,"5min":10.663,"15min":10.663},"last":0.73299999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542181,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":794624,"data_stored":352635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.86899999999999999,"15min":0.88700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.91300000000000003,"15min":0.95199999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.2729999999999999,"5min":2.226,"15min":2.226},"last":0.56799999999999995}]},{"osd":1,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.97399999999999998,"15min":0.997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.956,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.90200000000000002,"15min":0.92400000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.665,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.60499999999999998}]},{"osd":3,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":1.008,"15min":1.034},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.077,"15min":1.0920000000000001},"min":{"1min":0.53600000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.4690000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.97299999999999998}]},{"osd":4,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.2170000000000001,"15min":1.2390000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.605,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.208,"15min":1.224},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.6910000000000001,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.86099999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.357,"15min":1.383},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.9339999999999999,"5min":1.97,"15min":1.97},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.3280000000000001,"15min":1.357},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.79,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.92900000000000005}]},{"osd":6,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.321,"15min":1.357},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.498,"5min":2.456,"15min":2.456},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.47,"15min":1.5},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.0310000000000001,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":1.284}]},{"osd":7,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.3640000000000001,"15min":1.3879999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.7849999999999999,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.069},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.3919999999999999,"15min":1.425},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.7769999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.78200000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":823296,"data_stored":354963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.85099999999999998,"15min":0.88100000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.278,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.753,"5min":0.81999999999999995,"15min":0.83099999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.2509999999999999,"5min":1.327,"15min":1.327},"last":0.95399999999999996}]},{"osd":2,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.95099999999999996,"15min":0.97299999999999998},"min":{"1min":0.49199999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.405,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.879},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.97699999999999998,"15min":0.995},"min":{"1min":0.495,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.4079999999999999,"5min":1.625,"15min":1.625},"last":0.74199999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.1100000000000001,"15min":1.115},"min":{"1min":0.61099999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.6850000000000001,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.077},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.002,"15min":1.022},"min":{"1min":0.47099999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.4359999999999999,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.57399999999999995}]},{"osd":4,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.2669999999999999,"15min":1.304},"min":{"1min":0.79400000000000004,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.425,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.5},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.3049999999999999,"15min":1.325},"min":{"1min":0.79400000000000004,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.728,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.4750000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.319,"15min":1.3440000000000001},"min":{"1min":0.72599999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.5820000000000001,"5min":3.593,"15min":3.593},"last":1.167},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.4099999999999999,"15min":1.454},"min":{"1min":0.76700000000000002,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.496,"5min":2.359,"15min":2.359},"last":1.425}]},{"osd":6,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.389,"15min":1.419},"min":{"1min":0.75900000000000001,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.575,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.345},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.3500000000000001,"15min":1.3740000000000001},"min":{"1min":0.76900000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.766,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.2070000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.4259999999999999,"15min":1.4550000000000001},"min":{"1min":0.81999999999999995,"5min":0.627,"15min":0.627},"max":{"1min":1.821,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.329},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.472,"15min":1.522},"min":{"1min":0.77900000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.5269999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.556}]}]},{"osd":0,"up_from":14,"seq":60129542180,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28388,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343452,"statfs":{"total":96636764160,"available":96607694848,"internally_reserved":0,"allocated":1400832,"data_stored":949623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.94199999999999995,"15min":0.96099999999999997},"min":{"1min":0.46200000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.26,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.98799999999999999,"15min":0.996},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":1.2050000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.98899999999999999,"15min":1.0109999999999999},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.339,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.105,"15min":1.119},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.7849999999999999,"5min":1.901,"15min":1.901},"last":0.96299999999999997}]},{"osd":3,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.0089999999999999,"15min":1.0209999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.2969999999999999,"5min":1.972,"15min":1.972},"last":1.5189999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.1419999999999999,"15min":1.1719999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3260000000000001,"5min":2.298,"15min":2.298},"last":0.61799999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.345,"15min":1.355},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.4940000000000002,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":1.294},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.27,"15min":1.286},"min":{"1min":0.83099999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.425}]},{"osd":5,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.373,"15min":1.391},"min":{"1min":0.89200000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.0070000000000001,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.339},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.4379999999999999,"15min":1.458},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.972,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.462}]},{"osd":6,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.514,"15min":1.556},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.8340000000000001,"5min":3.032,"15min":3.032},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.4910000000000001,"15min":1.514},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.9350000000000001,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.476}]},{"osd":7,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.5029999999999999,"15min":1.5269999999999999},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.3809999999999998,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.605},{"interface":"front","average":{"1min":1.298,"5min":1.4179999999999999,"15min":1.4379999999999999},"min":{"1min":0.85599999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.331,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.2490000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:49:39.861 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:49:40.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.107+0000 7f5922e00640 1 Processor -- start 2024-03-23T03:49:40.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.107+0000 7f5922e00640 1 -- start start 2024-03-23T03:49:40.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.108+0000 7f5922e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f591c0cb560 0x7f591c0b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:40.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.108+0000 7f5922e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f591c1575b0 con 0x7f591c0cdc00 2024-03-23T03:49:40.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.109+0000 7f5922e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f591c159f20 con 0x7f591c1571d0 2024-03-23T03:49:40.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.109+0000 7f5922e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591c15a200 con 0x7f591c0cb560 2024-03-23T03:49:40.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.109+0000 7f5921000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f591c0cdc00 0x7f591c0cae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50648/0 (socket says 172.21.15.102:50648) 2024-03-23T03:49:40.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.109+0000 7f5921000640 1 -- 172.21.15.102:0/625615885 learned_addr learned my addr 172.21.15.102:0/625615885 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:40.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.109+0000 7f5922400640 1 --2- 172.21.15.102:0/625615885 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f591c0cb560 0x7f591c0b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:40.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.109+0000 7f591be00640 1 -- 172.21.15.102:0/625615885 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997085078 0 0) 0x7f591c1575b0 con 0x7f591c0cdc00 2024-03-23T03:49:40.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.109+0000 7f591be00640 1 -- 172.21.15.102:0/625615885 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5904003520 con 0x7f591c0cdc00 2024-03-23T03:49:40.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.110+0000 7f591be00640 1 -- 172.21.15.102:0/625615885 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1226749556 0 0) 0x7f5904003520 con 0x7f591c0cdc00 2024-03-23T03:49:40.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.110+0000 7f591be00640 1 -- 172.21.15.102:0/625615885 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f591c0cb560 msgr2=0x7f591c0b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:40.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.110+0000 7f591be00640 1 --2- 172.21.15.102:0/625615885 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f591c0cb560 0x7f591c0b35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.110+0000 7f591be00640 1 -- 172.21.15.102:0/625615885 >> v1:172.21.15.173:6789/0 conn(0x7f591c1571d0 legacy=0x7f591c15aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:40.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.110+0000 7f591be00640 1 -- 172.21.15.102:0/625615885 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591c15a4e0 con 0x7f591c0cdc00 2024-03-23T03:49:40.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.110+0000 7f591be00640 1 -- 172.21.15.102:0/625615885 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f5908001670 con 0x7f591c0cdc00 2024-03-23T03:49:40.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.110+0000 7f591be00640 1 -- 172.21.15.102:0/625615885 >> v1:172.21.15.102:6789/0 conn(0x7f591c0cdc00 legacy=0x7f591c0cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:40.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.110+0000 7f591be00640 1 --2- 172.21.15.102:0/625615885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59040039a0 0x7f5904003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:40.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.110+0000 7f591be00640 1 --2- 172.21.15.102:0/625615885 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f591c0cb560 0x7f59040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:40.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.110+0000 7f591be00640 1 --2- 172.21.15.102:0/625615885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5904004d00 0x7f5904005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:40.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.110+0000 7f591be00640 1 -- 172.21.15.102:0/625615885 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c15a200 con 0x7f5904004d00 2024-03-23T03:49:40.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.110+0000 7f591be00640 1 -- 172.21.15.102:0/625615885 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590400ba90 con 0x7f59040039a0 2024-03-23T03:49:40.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.110+0000 7f591be00640 1 -- 172.21.15.102:0/625615885 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590400bc50 con 0x7f591c0cb560 2024-03-23T03:49:40.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.110+0000 7f5922400640 1 --2- 172.21.15.102:0/625615885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5904004d00 0x7f5904005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:40.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.110+0000 7f5921000640 1 --2- 172.21.15.102:0/625615885 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f591c0cb560 0x7f59040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:40.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.110+0000 7f5921a00640 1 --2- 172.21.15.102:0/625615885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59040039a0 0x7f5904003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:40.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.110+0000 7f5922400640 1 -- 172.21.15.102:0/625615885 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f591c0cb560 msgr2=0x7f59040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:40.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.111+0000 7f5922400640 1 --2- 172.21.15.102:0/625615885 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f591c0cb560 0x7f59040047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.111+0000 7f5922400640 1 -- 172.21.15.102:0/625615885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59040039a0 msgr2=0x7f5904003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:40.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.111+0000 7f5922400640 1 --2- 172.21.15.102:0/625615885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59040039a0 0x7f5904003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.111+0000 7f5922400640 1 -- 172.21.15.102:0/625615885 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5904003520 con 0x7f5904004d00 2024-03-23T03:49:40.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.111+0000 7f5921000640 1 --2- 172.21.15.102:0/625615885 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f591c0cb560 0x7f59040047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:40.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.111+0000 7f5921a00640 1 --2- 172.21.15.102:0/625615885 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59040039a0 0x7f5904003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:40.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.112+0000 7f5922400640 1 --2- 172.21.15.102:0/625615885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5904004d00 0x7f5904005180 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f591c06ab30 tx=0x7f59040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf8e732fc421ad8e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:40.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.112+0000 7f591be00640 1 -- 172.21.15.102:0/625615885 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591402eda0 con 0x7f5904004d00 2024-03-23T03:49:40.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.112+0000 7f591be00640 1 -- 172.21.15.102:0/625615885 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5914011040 con 0x7f5904004d00 2024-03-23T03:49:40.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.112+0000 7f591be00640 1 -- 172.21.15.102:0/625615885 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591400b310 con 0x7f5904004d00 2024-03-23T03:49:40.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.112+0000 7f5922e00640 1 -- 172.21.15.102:0/625615885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5904004d00 msgr2=0x7f5904005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:40.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.112+0000 7f5922e00640 1 --2- 172.21.15.102:0/625615885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5904004d00 0x7f5904005180 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f591c06ab30 tx=0x7f59040057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.112+0000 7f5922400640 1 -- 172.21.15.102:0/625615885 reap_dead start 2024-03-23T03:49:40.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.113+0000 7f5922e00640 1 -- 172.21.15.102:0/625615885 shutdown_connections 2024-03-23T03:49:40.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.113+0000 7f5922e00640 1 -- 172.21.15.102:0/625615885 >> 172.21.15.102:0/625615885 conn(0x7f591c065df0 msgr2=0x7f591c069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:40.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.113+0000 7f5922e00640 1 -- 172.21.15.102:0/625615885 shutdown_connections 2024-03-23T03:49:40.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.113+0000 7f5922e00640 1 -- 172.21.15.102:0/625615885 wait complete. 2024-03-23T03:49:40.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.114+0000 7f5922e00640 1 Processor -- start 2024-03-23T03:49:40.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.115+0000 7f5922e00640 1 -- start start 2024-03-23T03:49:40.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.115+0000 7f5922e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5904004d00 0x7f591c140ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:40.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.115+0000 7f5922e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f591c0ae0c0 0x7f591c0ae540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:40.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.115+0000 7f5922e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f591c0aea80 0x7f591c1f5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:40.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.115+0000 7f5922e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c141530 con 0x7f591c0ae0c0 2024-03-23T03:49:40.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.115+0000 7f5922e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c0b06d0 con 0x7f5904004d00 2024-03-23T03:49:40.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.115+0000 7f5922e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591c0aef00 con 0x7f591c0aea80 2024-03-23T03:49:40.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.116+0000 7f5921000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f591c0ae0c0 0x7f591c0ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:40.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.116+0000 7f5921000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f591c0ae0c0 0x7f591c0ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49916/0 (socket says 172.21.15.102:49916) 2024-03-23T03:49:40.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.116+0000 7f5921000640 1 -- 172.21.15.102:0/1669198990 learned_addr learned my addr 172.21.15.102:0/1669198990 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:40.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.116+0000 7f5921a00640 1 --2- 172.21.15.102:0/1669198990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5904004d00 0x7f591c140ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:40.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.116+0000 7f5922400640 1 --2- 172.21.15.102:0/1669198990 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f591c0aea80 0x7f591c1f5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:40.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.116+0000 7f5921000640 1 -- 172.21.15.102:0/1669198990 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f591c0aea80 msgr2=0x7f591c1f5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:40.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.116+0000 7f5921000640 1 --2- 172.21.15.102:0/1669198990 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f591c0aea80 0x7f591c1f5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.116+0000 7f5921000640 1 -- 172.21.15.102:0/1669198990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5904004d00 msgr2=0x7f591c140ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:40.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.116+0000 7f5921000640 1 --2- 172.21.15.102:0/1669198990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5904004d00 0x7f591c140ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.116+0000 7f5921000640 1 -- 172.21.15.102:0/1669198990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591c0c3580 con 0x7f591c0ae0c0 2024-03-23T03:49:40.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.117+0000 7f5921000640 1 --2- 172.21.15.102:0/1669198990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f591c0ae0c0 0x7f591c0ae540 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f590800d810 tx=0x7f590800dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:40.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.117+0000 7f5919600640 1 -- 172.21.15.102:0/1669198990 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59080074e0 con 0x7f591c0ae0c0 2024-03-23T03:49:40.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.117+0000 7f5919600640 1 -- 172.21.15.102:0/1669198990 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5908007640 con 0x7f591c0ae0c0 2024-03-23T03:49:40.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.117+0000 7f5922e00640 1 -- 172.21.15.102:0/1669198990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591c0c3860 con 0x7f591c0ae0c0 2024-03-23T03:49:40.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.117+0000 7f5919600640 1 -- 172.21.15.102:0/1669198990 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59080049e0 con 0x7f591c0ae0c0 2024-03-23T03:49:40.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.117+0000 7f5922e00640 1 -- 172.21.15.102:0/1669198990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f591c0c3e20 con 0x7f591c0ae0c0 2024-03-23T03:49:40.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.118+0000 7f5922e00640 1 -- 172.21.15.102:0/1669198990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f591c04e030 con 0x7f591c0ae0c0 2024-03-23T03:49:40.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.119+0000 7f5919600640 1 -- 172.21.15.102:0/1669198990 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5908004b40 con 0x7f591c0ae0c0 2024-03-23T03:49:40.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.120+0000 7f5919600640 1 --2- 172.21.15.102:0/1669198990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f58fc06ac00 0x7f58fc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:40.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.120+0000 7f5919600640 1 -- 172.21.15.102:0/1669198990 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5908088f50 con 0x7f591c0ae0c0 2024-03-23T03:49:40.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.120+0000 7f5921a00640 1 --2- 172.21.15.102:0/1669198990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f58fc06ac00 0x7f58fc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:40.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.121+0000 7f5921a00640 1 --2- 172.21.15.102:0/1669198990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f58fc06ac00 0x7f58fc06d0c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5910001730 tx=0x7f5910009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:40.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.125+0000 7f5919600640 1 -- 172.21.15.102:0/1669198990 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5908053f50 con 0x7f591c0ae0c0 2024-03-23T03:49:40.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.315+0000 7f5922e00640 1 -- 172.21.15.102:0/1669198990 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f591c0b35d0 con 0x7f58fc06ac00 2024-03-23T03:49:40.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.324+0000 7f5919600640 1 -- 172.21.15.102:0/1669198990 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286896 (secure 0 0 0) 0x7f591000d050 con 0x7f58fc06ac00 2024-03-23T03:49:40.327 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:40.332 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:49:40.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.334+0000 7f5922e00640 1 -- 172.21.15.102:0/1669198990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f58fc06ac00 msgr2=0x7f58fc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:40.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.334+0000 7f5922e00640 1 --2- 172.21.15.102:0/1669198990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f58fc06ac00 0x7f58fc06d0c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5910001730 tx=0x7f5910009040 comp rx=0 tx=0).stop 2024-03-23T03:49:40.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.334+0000 7f5922e00640 1 -- 172.21.15.102:0/1669198990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f591c0ae0c0 msgr2=0x7f591c0ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:40.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.334+0000 7f5922e00640 1 --2- 172.21.15.102:0/1669198990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f591c0ae0c0 0x7f591c0ae540 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f590800d810 tx=0x7f590800dcd0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.334+0000 7f5922e00640 1 -- 172.21.15.102:0/1669198990 shutdown_connections 2024-03-23T03:49:40.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.334+0000 7f5922e00640 1 --2- 172.21.15.102:0/1669198990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f58fc06ac00 0x7f58fc06d0c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.334+0000 7f5922e00640 1 --2- 172.21.15.102:0/1669198990 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f591c0aea80 0x7f591c1f5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.334+0000 7f5922e00640 1 --2- 172.21.15.102:0/1669198990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f591c0ae0c0 0x7f591c0ae540 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.334+0000 7f5922e00640 1 --2- 172.21.15.102:0/1669198990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5904004d00 0x7f591c140ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.334+0000 7f5922e00640 1 -- 172.21.15.102:0/1669198990 >> 172.21.15.102:0/1669198990 conn(0x7f591c065df0 msgr2=0x7f591c06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:40.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.334+0000 7f5922e00640 1 -- 172.21.15.102:0/1669198990 shutdown_connections 2024-03-23T03:49:40.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.334+0000 7f5922e00640 1 -- 172.21.15.102:0/1669198990 wait complete. 2024-03-23T03:49:40.353 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-03-23T03:49:39.768461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":67,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223648,"kb_used_data":7444,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754751072,"statfs":{"total":773094113280,"available":772865097728,"internally_reserved":0,"allocated":7622656,"data_stored":4013160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":221236794},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.827805"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436646+0000","last_change":"2024-03-23T03:49:35.437508+0000","last_active":"2024-03-23T03:49:36.436646+0000","last_peered":"2024-03-23T03:49:36.436646+0000","last_clean":"2024-03-23T03:49:36.436646+0000","last_became_active":"2024-03-23T03:49:35.437196+0000","last_became_peered":"2024-03-23T03:49:35.437196+0000","last_unstale":"2024-03-23T03:49:36.436646+0000","last_undegraded":"2024-03-23T03:49:36.436646+0000","last_fullsized":"2024-03-23T03:49:36.436646+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:50:23.362975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436468+0000","last_change":"2024-03-23T03:49:35.453041+0000","last_active":"2024-03-23T03:49:36.436468+0000","last_peered":"2024-03-23T03:49:36.436468+0000","last_clean":"2024-03-23T03:49:36.436468+0000","last_became_active":"2024-03-23T03:49:35.446517+0000","last_became_peered":"2024-03-23T03:49:35.446517+0000","last_unstale":"2024-03-23T03:49:36.436468+0000","last_undegraded":"2024-03-23T03:49:36.436468+0000","last_fullsized":"2024-03-23T03:49:36.436468+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:55:05.527463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.458021+0000","last_change":"2024-03-23T03:49:35.454914+0000","last_active":"2024-03-23T03:49:35.458021+0000","last_peered":"2024-03-23T03:49:35.458021+0000","last_clean":"2024-03-23T03:49:35.458021+0000","last_became_active":"2024-03-23T03:49:35.454627+0000","last_became_peered":"2024-03-23T03:49:35.454627+0000","last_unstale":"2024-03-23T03:49:35.458021+0000","last_undegraded":"2024-03-23T03:49:35.458021+0000","last_fullsized":"2024-03-23T03:49:35.458021+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:26:04.416714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.461936+0000","last_change":"2024-03-23T03:49:35.460833+0000","last_active":"2024-03-23T03:49:35.461936+0000","last_peered":"2024-03-23T03:49:35.461936+0000","last_clean":"2024-03-23T03:49:35.461936+0000","last_became_active":"2024-03-23T03:49:35.460282+0000","last_became_peered":"2024-03-23T03:49:35.460282+0000","last_unstale":"2024-03-23T03:49:35.461936+0000","last_undegraded":"2024-03-23T03:49:35.461936+0000","last_fullsized":"2024-03-23T03:49:35.461936+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:10.974686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.460801+0000","last_change":"2024-03-23T03:49:35.457923+0000","last_active":"2024-03-23T03:49:35.460801+0000","last_peered":"2024-03-23T03:49:35.460801+0000","last_clean":"2024-03-23T03:49:35.460801+0000","last_became_active":"2024-03-23T03:49:35.449801+0000","last_became_peered":"2024-03-23T03:49:35.449801+0000","last_unstale":"2024-03-23T03:49:35.460801+0000","last_undegraded":"2024-03-23T03:49:35.460801+0000","last_fullsized":"2024-03-23T03:49:35.460801+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:19:24.808232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650800+0000","last_change":"2024-03-23T03:49:35.447252+0000","last_active":"2024-03-23T03:49:36.650800+0000","last_peered":"2024-03-23T03:49:36.650800+0000","last_clean":"2024-03-23T03:49:36.650800+0000","last_became_active":"2024-03-23T03:49:35.446694+0000","last_became_peered":"2024-03-23T03:49:35.446694+0000","last_unstale":"2024-03-23T03:49:36.650800+0000","last_undegraded":"2024-03-23T03:49:36.650800+0000","last_fullsized":"2024-03-23T03:49:36.650800+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:49:49.765798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.458990+0000","last_change":"2024-03-23T03:49:35.451510+0000","last_active":"2024-03-23T03:49:35.458990+0000","last_peered":"2024-03-23T03:49:35.458990+0000","last_clean":"2024-03-23T03:49:35.458990+0000","last_became_active":"2024-03-23T03:49:35.451304+0000","last_became_peered":"2024-03-23T03:49:35.451304+0000","last_unstale":"2024-03-23T03:49:35.458990+0000","last_undegraded":"2024-03-23T03:49:35.458990+0000","last_fullsized":"2024-03-23T03:49:35.458990+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:07:58.894188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.435985+0000","last_change":"2024-03-23T03:49:35.443995+0000","last_active":"2024-03-23T03:49:36.435985+0000","last_peered":"2024-03-23T03:49:36.435985+0000","last_clean":"2024-03-23T03:49:36.435985+0000","last_became_active":"2024-03-23T03:49:35.443707+0000","last_became_peered":"2024-03-23T03:49:35.443707+0000","last_unstale":"2024-03-23T03:49:36.435985+0000","last_undegraded":"2024-03-23T03:49:36.435985+0000","last_fullsized":"2024-03-23T03:49:36.435985+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:44.372289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.467360+0000","last_change":"2024-03-23T03:49:35.464954+0000","last_active":"2024-03-23T03:49:35.467360+0000","last_peered":"2024-03-23T03:49:35.467360+0000","last_clean":"2024-03-23T03:49:35.467360+0000","last_became_active":"2024-03-23T03:49:35.464712+0000","last_became_peered":"2024-03-23T03:49:35.464712+0000","last_unstale":"2024-03-23T03:49:35.467360+0000","last_undegraded":"2024-03-23T03:49:35.467360+0000","last_fullsized":"2024-03-23T03:49:35.467360+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:19:55.183686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.457930+0000","last_change":"2024-03-23T03:49:35.453993+0000","last_active":"2024-03-23T03:49:35.457930+0000","last_peered":"2024-03-23T03:49:35.457930+0000","last_clean":"2024-03-23T03:49:35.457930+0000","last_became_active":"2024-03-23T03:49:35.453784+0000","last_became_peered":"2024-03-23T03:49:35.453784+0000","last_unstale":"2024-03-23T03:49:35.457930+0000","last_undegraded":"2024-03-23T03:49:35.457930+0000","last_fullsized":"2024-03-23T03:49:35.457930+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:58:17.359323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.462075+0000","last_change":"2024-03-23T03:49:35.459222+0000","last_active":"2024-03-23T03:49:35.462075+0000","last_peered":"2024-03-23T03:49:35.462075+0000","last_clean":"2024-03-23T03:49:35.462075+0000","last_became_active":"2024-03-23T03:49:35.457229+0000","last_became_peered":"2024-03-23T03:49:35.457229+0000","last_unstale":"2024-03-23T03:49:35.462075+0000","last_undegraded":"2024-03-23T03:49:35.462075+0000","last_fullsized":"2024-03-23T03:49:35.462075+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:08.096604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436139+0000","last_change":"2024-03-23T03:49:35.444731+0000","last_active":"2024-03-23T03:49:36.436139+0000","last_peered":"2024-03-23T03:49:36.436139+0000","last_clean":"2024-03-23T03:49:36.436139+0000","last_became_active":"2024-03-23T03:49:35.444423+0000","last_became_peered":"2024-03-23T03:49:35.444423+0000","last_unstale":"2024-03-23T03:49:36.436139+0000","last_undegraded":"2024-03-23T03:49:36.436139+0000","last_fullsized":"2024-03-23T03:49:36.436139+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:14:59.255644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436223+0000","last_change":"2024-03-23T03:49:35.446054+0000","last_active":"2024-03-23T03:49:36.436223+0000","last_peered":"2024-03-23T03:49:36.436223+0000","last_clean":"2024-03-23T03:49:36.436223+0000","last_became_active":"2024-03-23T03:49:35.445824+0000","last_became_peered":"2024-03-23T03:49:35.445824+0000","last_unstale":"2024-03-23T03:49:36.436223+0000","last_undegraded":"2024-03-23T03:49:36.436223+0000","last_fullsized":"2024-03-23T03:49:36.436223+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:10:50.530595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.456440+0000","last_change":"2024-03-23T03:49:35.450244+0000","last_active":"2024-03-23T03:49:35.456440+0000","last_peered":"2024-03-23T03:49:35.456440+0000","last_clean":"2024-03-23T03:49:35.456440+0000","last_became_active":"2024-03-23T03:49:35.449990+0000","last_became_peered":"2024-03-23T03:49:35.449990+0000","last_unstale":"2024-03-23T03:49:35.456440+0000","last_undegraded":"2024-03-23T03:49:35.456440+0000","last_fullsized":"2024-03-23T03:49:35.456440+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:55:40.868490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.457973+0000","last_change":"2024-03-23T03:49:35.454183+0000","last_active":"2024-03-23T03:49:35.457973+0000","last_peered":"2024-03-23T03:49:35.457973+0000","last_clean":"2024-03-23T03:49:35.457973+0000","last_became_active":"2024-03-23T03:49:35.453933+0000","last_became_peered":"2024-03-23T03:49:35.453933+0000","last_unstale":"2024-03-23T03:49:35.457973+0000","last_undegraded":"2024-03-23T03:49:35.457973+0000","last_fullsized":"2024-03-23T03:49:35.457973+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:50:27.972663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.459013+0000","last_change":"2024-03-23T03:49:35.452441+0000","last_active":"2024-03-23T03:49:35.459013+0000","last_peered":"2024-03-23T03:49:35.459013+0000","last_clean":"2024-03-23T03:49:35.459013+0000","last_became_active":"2024-03-23T03:49:35.452214+0000","last_became_peered":"2024-03-23T03:49:35.452214+0000","last_unstale":"2024-03-23T03:49:35.459013+0000","last_undegraded":"2024-03-23T03:49:35.459013+0000","last_fullsized":"2024-03-23T03:49:35.459013+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:02:00.932577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436802+0000","last_change":"2024-03-23T03:49:35.446531+0000","last_active":"2024-03-23T03:49:36.436802+0000","last_peered":"2024-03-23T03:49:36.436802+0000","last_clean":"2024-03-23T03:49:36.436802+0000","last_became_active":"2024-03-23T03:49:35.446139+0000","last_became_peered":"2024-03-23T03:49:35.446139+0000","last_unstale":"2024-03-23T03:49:36.436802+0000","last_undegraded":"2024-03-23T03:49:36.436802+0000","last_fullsized":"2024-03-23T03:49:36.436802+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:10:50.530595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.462584+0000","last_change":"2024-03-23T03:49:35.461897+0000","last_active":"2024-03-23T03:49:35.462584+0000","last_peered":"2024-03-23T03:49:35.462584+0000","last_clean":"2024-03-23T03:49:35.462584+0000","last_became_active":"2024-03-23T03:49:35.454272+0000","last_became_peered":"2024-03-23T03:49:35.454272+0000","last_unstale":"2024-03-23T03:49:35.462584+0000","last_undegraded":"2024-03-23T03:49:35.462584+0000","last_fullsized":"2024-03-23T03:49:35.462584+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:43:44.069662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.462828+0000","last_change":"2024-03-23T03:49:35.461117+0000","last_active":"2024-03-23T03:49:35.462828+0000","last_peered":"2024-03-23T03:49:35.462828+0000","last_clean":"2024-03-23T03:49:35.462828+0000","last_became_active":"2024-03-23T03:49:35.460441+0000","last_became_peered":"2024-03-23T03:49:35.460441+0000","last_unstale":"2024-03-23T03:49:35.462828+0000","last_undegraded":"2024-03-23T03:49:35.462828+0000","last_fullsized":"2024-03-23T03:49:35.462828+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:42:51.688200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.452071+0000","last_change":"2024-03-23T03:49:35.450256+0000","last_active":"2024-03-23T03:49:35.452071+0000","last_peered":"2024-03-23T03:49:35.452071+0000","last_clean":"2024-03-23T03:49:35.452071+0000","last_became_active":"2024-03-23T03:49:35.449940+0000","last_became_peered":"2024-03-23T03:49:35.449940+0000","last_unstale":"2024-03-23T03:49:35.452071+0000","last_undegraded":"2024-03-23T03:49:35.452071+0000","last_fullsized":"2024-03-23T03:49:35.452071+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:47:23.751438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436431+0000","last_change":"2024-03-23T03:49:35.439856+0000","last_active":"2024-03-23T03:49:36.436431+0000","last_peered":"2024-03-23T03:49:36.436431+0000","last_clean":"2024-03-23T03:49:36.436431+0000","last_became_active":"2024-03-23T03:49:35.439575+0000","last_became_peered":"2024-03-23T03:49:35.439575+0000","last_unstale":"2024-03-23T03:49:36.436431+0000","last_undegraded":"2024-03-23T03:49:36.436431+0000","last_fullsized":"2024-03-23T03:49:36.436431+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:48:11.607155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.451289+0000","last_change":"2024-03-23T03:49:35.445764+0000","last_active":"2024-03-23T03:49:35.451289+0000","last_peered":"2024-03-23T03:49:35.451289+0000","last_clean":"2024-03-23T03:49:35.451289+0000","last_became_active":"2024-03-23T03:49:35.445473+0000","last_became_peered":"2024-03-23T03:49:35.445473+0000","last_unstale":"2024-03-23T03:49:35.451289+0000","last_undegraded":"2024-03-23T03:49:35.451289+0000","last_fullsized":"2024-03-23T03:49:35.451289+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:45:25.906413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.449054+0000","last_change":"2024-03-23T03:49:35.443789+0000","last_active":"2024-03-23T03:49:35.449054+0000","last_peered":"2024-03-23T03:49:35.449054+0000","last_clean":"2024-03-23T03:49:35.449054+0000","last_became_active":"2024-03-23T03:49:35.442935+0000","last_became_peered":"2024-03-23T03:49:35.442935+0000","last_unstale":"2024-03-23T03:49:35.449054+0000","last_undegraded":"2024-03-23T03:49:35.449054+0000","last_fullsized":"2024-03-23T03:49:35.449054+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:17:26.874645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.456781+0000","last_change":"2024-03-23T03:49:35.452298+0000","last_active":"2024-03-23T03:49:35.456781+0000","last_peered":"2024-03-23T03:49:35.456781+0000","last_clean":"2024-03-23T03:49:35.456781+0000","last_became_active":"2024-03-23T03:49:35.449400+0000","last_became_peered":"2024-03-23T03:49:35.449400+0000","last_unstale":"2024-03-23T03:49:35.456781+0000","last_undegraded":"2024-03-23T03:49:35.456781+0000","last_fullsized":"2024-03-23T03:49:35.456781+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:55:05.527463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436491+0000","last_change":"2024-03-23T03:49:35.457522+0000","last_active":"2024-03-23T03:49:36.436491+0000","last_peered":"2024-03-23T03:49:36.436491+0000","last_clean":"2024-03-23T03:49:36.436491+0000","last_became_active":"2024-03-23T03:49:35.457014+0000","last_became_peered":"2024-03-23T03:49:35.457014+0000","last_unstale":"2024-03-23T03:49:36.436491+0000","last_undegraded":"2024-03-23T03:49:36.436491+0000","last_fullsized":"2024-03-23T03:49:36.436491+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:59.256052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436213+0000","last_change":"2024-03-23T03:49:35.457586+0000","last_active":"2024-03-23T03:49:36.436213+0000","last_peered":"2024-03-23T03:49:36.436213+0000","last_clean":"2024-03-23T03:49:36.436213+0000","last_became_active":"2024-03-23T03:49:35.457285+0000","last_became_peered":"2024-03-23T03:49:35.457285+0000","last_unstale":"2024-03-23T03:49:36.436213+0000","last_undegraded":"2024-03-23T03:49:36.436213+0000","last_fullsized":"2024-03-23T03:49:36.436213+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:19:31.816124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.462091+0000","last_change":"2024-03-23T03:49:35.459386+0000","last_active":"2024-03-23T03:49:35.462091+0000","last_peered":"2024-03-23T03:49:35.462091+0000","last_clean":"2024-03-23T03:49:35.462091+0000","last_became_active":"2024-03-23T03:49:35.457607+0000","last_became_peered":"2024-03-23T03:49:35.457607+0000","last_unstale":"2024-03-23T03:49:35.462091+0000","last_undegraded":"2024-03-23T03:49:35.462091+0000","last_fullsized":"2024-03-23T03:49:35.462091+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:19:24.808232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436337+0000","last_change":"2024-03-23T03:49:35.453262+0000","last_active":"2024-03-23T03:49:36.436337+0000","last_peered":"2024-03-23T03:49:36.436337+0000","last_clean":"2024-03-23T03:49:36.436337+0000","last_became_active":"2024-03-23T03:49:35.452899+0000","last_became_peered":"2024-03-23T03:49:35.452899+0000","last_unstale":"2024-03-23T03:49:36.436337+0000","last_undegraded":"2024-03-23T03:49:36.436337+0000","last_fullsized":"2024-03-23T03:49:36.436337+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:19:31.816124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.467421+0000","last_change":"2024-03-23T03:49:35.465195+0000","last_active":"2024-03-23T03:49:35.467421+0000","last_peered":"2024-03-23T03:49:35.467421+0000","last_clean":"2024-03-23T03:49:35.467421+0000","last_became_active":"2024-03-23T03:49:35.463733+0000","last_became_peered":"2024-03-23T03:49:35.463733+0000","last_unstale":"2024-03-23T03:49:35.467421+0000","last_undegraded":"2024-03-23T03:49:35.467421+0000","last_fullsized":"2024-03-23T03:49:35.467421+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:15:27.492278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.435912+0000","last_change":"2024-03-23T03:49:35.444622+0000","last_active":"2024-03-23T03:49:36.435912+0000","last_peered":"2024-03-23T03:49:36.435912+0000","last_clean":"2024-03-23T03:49:36.435912+0000","last_became_active":"2024-03-23T03:49:35.444343+0000","last_became_peered":"2024-03-23T03:49:35.444343+0000","last_unstale":"2024-03-23T03:49:36.435912+0000","last_undegraded":"2024-03-23T03:49:36.435912+0000","last_fullsized":"2024-03-23T03:49:36.435912+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:24:33.040816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.461301+0000","last_change":"2024-03-23T03:49:35.456971+0000","last_active":"2024-03-23T03:49:35.461301+0000","last_peered":"2024-03-23T03:49:35.461301+0000","last_clean":"2024-03-23T03:49:35.461301+0000","last_became_active":"2024-03-23T03:49:35.456670+0000","last_became_peered":"2024-03-23T03:49:35.456670+0000","last_unstale":"2024-03-23T03:49:35.461301+0000","last_undegraded":"2024-03-23T03:49:35.461301+0000","last_fullsized":"2024-03-23T03:49:35.461301+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:32:51.943312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650598+0000","last_change":"2024-03-23T03:49:35.457747+0000","last_active":"2024-03-23T03:49:36.650598+0000","last_peered":"2024-03-23T03:49:36.650598+0000","last_clean":"2024-03-23T03:49:36.650598+0000","last_became_active":"2024-03-23T03:49:35.457503+0000","last_became_peered":"2024-03-23T03:49:35.457503+0000","last_unstale":"2024-03-23T03:49:36.650598+0000","last_undegraded":"2024-03-23T03:49:36.650598+0000","last_fullsized":"2024-03-23T03:49:36.650598+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:23:14.905130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.456472+0000","last_change":"2024-03-23T03:49:35.444306+0000","last_active":"2024-03-23T03:49:35.456472+0000","last_peered":"2024-03-23T03:49:35.456472+0000","last_clean":"2024-03-23T03:49:35.456472+0000","last_became_active":"2024-03-23T03:49:35.444035+0000","last_became_peered":"2024-03-23T03:49:35.444035+0000","last_unstale":"2024-03-23T03:49:35.456472+0000","last_undegraded":"2024-03-23T03:49:35.456472+0000","last_fullsized":"2024-03-23T03:49:35.456472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:27:28.185090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"66'4","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436845+0000","last_change":"2024-03-23T03:49:34.982234+0000","last_active":"2024-03-23T03:49:36.436845+0000","last_peered":"2024-03-23T03:49:36.436845+0000","last_clean":"2024-03-23T03:49:36.436845+0000","last_became_active":"2024-03-23T03:49:34.981843+0000","last_became_peered":"2024-03-23T03:49:34.981843+0000","last_unstale":"2024-03-23T03:49:36.436845+0000","last_undegraded":"2024-03-23T03:49:36.436845+0000","last_fullsized":"2024-03-23T03:49:36.436845+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:32:24.880206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.468212+0000","last_change":"2024-03-23T03:49:35.464085+0000","last_active":"2024-03-23T03:49:35.468212+0000","last_peered":"2024-03-23T03:49:35.468212+0000","last_clean":"2024-03-23T03:49:35.468212+0000","last_became_active":"2024-03-23T03:49:35.463812+0000","last_became_peered":"2024-03-23T03:49:35.463812+0000","last_unstale":"2024-03-23T03:49:35.468212+0000","last_undegraded":"2024-03-23T03:49:35.468212+0000","last_fullsized":"2024-03-23T03:49:35.468212+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:23:44.334673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"66'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650966+0000","last_change":"2024-03-23T03:49:34.983531+0000","last_active":"2024-03-23T03:49:36.650966+0000","last_peered":"2024-03-23T03:49:36.650966+0000","last_clean":"2024-03-23T03:49:36.650966+0000","last_became_active":"2024-03-23T03:49:34.983191+0000","last_became_peered":"2024-03-23T03:49:34.983191+0000","last_unstale":"2024-03-23T03:49:36.650966+0000","last_undegraded":"2024-03-23T03:49:36.650966+0000","last_fullsized":"2024-03-23T03:49:36.650966+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:38:05.708048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650785+0000","last_change":"2024-03-23T03:49:35.457863+0000","last_active":"2024-03-23T03:49:36.650785+0000","last_peered":"2024-03-23T03:49:36.650785+0000","last_clean":"2024-03-23T03:49:36.650785+0000","last_became_active":"2024-03-23T03:49:35.456778+0000","last_became_peered":"2024-03-23T03:49:35.456778+0000","last_unstale":"2024-03-23T03:49:36.650785+0000","last_undegraded":"2024-03-23T03:49:36.650785+0000","last_fullsized":"2024-03-23T03:49:36.650785+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:41:07.345825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650762+0000","last_change":"2024-03-23T03:49:34.982798+0000","last_active":"2024-03-23T03:49:36.650762+0000","last_peered":"2024-03-23T03:49:36.650762+0000","last_clean":"2024-03-23T03:49:36.650762+0000","last_became_active":"2024-03-23T03:49:34.982539+0000","last_became_peered":"2024-03-23T03:49:34.982539+0000","last_unstale":"2024-03-23T03:49:36.650762+0000","last_undegraded":"2024-03-23T03:49:36.650762+0000","last_fullsized":"2024-03-23T03:49:36.650762+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:36:05.087323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.458067+0000","last_change":"2024-03-23T03:49:35.455447+0000","last_active":"2024-03-23T03:49:35.458067+0000","last_peered":"2024-03-23T03:49:35.458067+0000","last_clean":"2024-03-23T03:49:35.458067+0000","last_became_active":"2024-03-23T03:49:35.455169+0000","last_became_peered":"2024-03-23T03:49:35.455169+0000","last_unstale":"2024-03-23T03:49:35.458067+0000","last_undegraded":"2024-03-23T03:49:35.458067+0000","last_fullsized":"2024-03-23T03:49:35.458067+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:57:56.162095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650963+0000","last_change":"2024-03-23T03:49:34.974516+0000","last_active":"2024-03-23T03:49:36.650963+0000","last_peered":"2024-03-23T03:49:36.650963+0000","last_clean":"2024-03-23T03:49:36.650963+0000","last_became_active":"2024-03-23T03:49:34.974222+0000","last_became_peered":"2024-03-23T03:49:34.974222+0000","last_unstale":"2024-03-23T03:49:36.650963+0000","last_undegraded":"2024-03-23T03:49:36.650963+0000","last_fullsized":"2024-03-23T03:49:36.650963+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:13:03.702919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.455353+0000","last_change":"2024-03-23T03:49:35.448493+0000","last_active":"2024-03-23T03:49:35.455353+0000","last_peered":"2024-03-23T03:49:35.455353+0000","last_clean":"2024-03-23T03:49:35.455353+0000","last_became_active":"2024-03-23T03:49:35.446419+0000","last_became_peered":"2024-03-23T03:49:35.446419+0000","last_unstale":"2024-03-23T03:49:35.455353+0000","last_undegraded":"2024-03-23T03:49:35.455353+0000","last_fullsized":"2024-03-23T03:49:35.455353+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:10:14.484998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"66'1","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.445371+0000","last_change":"2024-03-23T03:49:34.973855+0000","last_active":"2024-03-23T03:49:35.445371+0000","last_peered":"2024-03-23T03:49:35.445371+0000","last_clean":"2024-03-23T03:49:35.445371+0000","last_became_active":"2024-03-23T03:49:34.973552+0000","last_became_peered":"2024-03-23T03:49:34.973552+0000","last_unstale":"2024-03-23T03:49:35.445371+0000","last_undegraded":"2024-03-23T03:49:35.445371+0000","last_fullsized":"2024-03-23T03:49:35.445371+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:23:13.893311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.435553+0000","last_change":"2024-03-23T03:49:35.458601+0000","last_active":"2024-03-23T03:49:36.435553+0000","last_peered":"2024-03-23T03:49:36.435553+0000","last_clean":"2024-03-23T03:49:36.435553+0000","last_became_active":"2024-03-23T03:49:35.456971+0000","last_became_peered":"2024-03-23T03:49:35.456971+0000","last_unstale":"2024-03-23T03:49:36.435553+0000","last_undegraded":"2024-03-23T03:49:36.435553+0000","last_fullsized":"2024-03-23T03:49:36.435553+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:18:35.380354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"66'1","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.454992+0000","last_change":"2024-03-23T03:49:34.970357+0000","last_active":"2024-03-23T03:49:35.454992+0000","last_peered":"2024-03-23T03:49:35.454992+0000","last_clean":"2024-03-23T03:49:35.454992+0000","last_became_active":"2024-03-23T03:49:34.968925+0000","last_became_peered":"2024-03-23T03:49:34.968925+0000","last_unstale":"2024-03-23T03:49:35.454992+0000","last_undegraded":"2024-03-23T03:49:35.454992+0000","last_fullsized":"2024-03-23T03:49:35.454992+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:36:41.972438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.463066+0000","last_change":"2024-03-23T03:49:35.461068+0000","last_active":"2024-03-23T03:49:35.463066+0000","last_peered":"2024-03-23T03:49:35.463066+0000","last_clean":"2024-03-23T03:49:35.463066+0000","last_became_active":"2024-03-23T03:49:35.459712+0000","last_became_peered":"2024-03-23T03:49:35.459712+0000","last_unstale":"2024-03-23T03:49:35.463066+0000","last_undegraded":"2024-03-23T03:49:35.463066+0000","last_fullsized":"2024-03-23T03:49:35.463066+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:02:27.532316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.433277+0000","last_change":"2024-03-23T03:49:34.988264+0000","last_active":"2024-03-23T03:49:35.433277+0000","last_peered":"2024-03-23T03:49:35.433277+0000","last_clean":"2024-03-23T03:49:35.433277+0000","last_became_active":"2024-03-23T03:49:34.987918+0000","last_became_peered":"2024-03-23T03:49:34.987918+0000","last_unstale":"2024-03-23T03:49:35.433277+0000","last_undegraded":"2024-03-23T03:49:35.433277+0000","last_fullsized":"2024-03-23T03:49:35.433277+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:52:53.648955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.464566+0000","last_change":"2024-03-23T03:49:35.463629+0000","last_active":"2024-03-23T03:49:35.464566+0000","last_peered":"2024-03-23T03:49:35.464566+0000","last_clean":"2024-03-23T03:49:35.464566+0000","last_became_active":"2024-03-23T03:49:35.463417+0000","last_became_peered":"2024-03-23T03:49:35.463417+0000","last_unstale":"2024-03-23T03:49:35.464566+0000","last_undegraded":"2024-03-23T03:49:35.464566+0000","last_fullsized":"2024-03-23T03:49:35.464566+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:37:55.139651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.431787+0000","last_change":"2024-03-23T03:49:34.990891+0000","last_active":"2024-03-23T03:49:35.431787+0000","last_peered":"2024-03-23T03:49:35.431787+0000","last_clean":"2024-03-23T03:49:35.431787+0000","last_became_active":"2024-03-23T03:49:34.990610+0000","last_became_peered":"2024-03-23T03:49:34.990610+0000","last_unstale":"2024-03-23T03:49:35.431787+0000","last_undegraded":"2024-03-23T03:49:35.431787+0000","last_fullsized":"2024-03-23T03:49:35.431787+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:27:38.167471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436413+0000","last_change":"2024-03-23T03:49:35.456262+0000","last_active":"2024-03-23T03:49:36.436413+0000","last_peered":"2024-03-23T03:49:36.436413+0000","last_clean":"2024-03-23T03:49:36.436413+0000","last_became_active":"2024-03-23T03:49:35.456043+0000","last_became_peered":"2024-03-23T03:49:35.456043+0000","last_unstale":"2024-03-23T03:49:36.436413+0000","last_undegraded":"2024-03-23T03:49:36.436413+0000","last_fullsized":"2024-03-23T03:49:36.436413+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:48:11.607155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.436947+0000","last_change":"2024-03-23T03:49:34.992625+0000","last_active":"2024-03-23T03:49:35.436947+0000","last_peered":"2024-03-23T03:49:35.436947+0000","last_clean":"2024-03-23T03:49:35.436947+0000","last_became_active":"2024-03-23T03:49:34.992007+0000","last_became_peered":"2024-03-23T03:49:34.992007+0000","last_unstale":"2024-03-23T03:49:35.436947+0000","last_undegraded":"2024-03-23T03:49:35.436947+0000","last_fullsized":"2024-03-23T03:49:35.436947+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:12:14.699545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.458195+0000","last_change":"2024-03-23T03:49:35.457436+0000","last_active":"2024-03-23T03:49:35.458195+0000","last_peered":"2024-03-23T03:49:35.458195+0000","last_clean":"2024-03-23T03:49:35.458195+0000","last_became_active":"2024-03-23T03:49:35.457171+0000","last_became_peered":"2024-03-23T03:49:35.457171+0000","last_unstale":"2024-03-23T03:49:35.458195+0000","last_undegraded":"2024-03-23T03:49:35.458195+0000","last_fullsized":"2024-03-23T03:49:35.458195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:14:01.589671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.437429+0000","last_change":"2024-03-23T03:49:34.981367+0000","last_active":"2024-03-23T03:49:35.437429+0000","last_peered":"2024-03-23T03:49:35.437429+0000","last_clean":"2024-03-23T03:49:35.437429+0000","last_became_active":"2024-03-23T03:49:34.981078+0000","last_became_peered":"2024-03-23T03:49:34.981078+0000","last_unstale":"2024-03-23T03:49:35.437429+0000","last_undegraded":"2024-03-23T03:49:35.437429+0000","last_fullsized":"2024-03-23T03:49:35.437429+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:39.665461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.454580+0000","last_change":"2024-03-23T03:49:35.447863+0000","last_active":"2024-03-23T03:49:35.454580+0000","last_peered":"2024-03-23T03:49:35.454580+0000","last_clean":"2024-03-23T03:49:35.454580+0000","last_became_active":"2024-03-23T03:49:35.447368+0000","last_became_peered":"2024-03-23T03:49:35.447368+0000","last_unstale":"2024-03-23T03:49:35.454580+0000","last_undegraded":"2024-03-23T03:49:35.454580+0000","last_fullsized":"2024-03-23T03:49:35.454580+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:09:08.096604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"66'3","reported_seq":13,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.461741+0000","last_change":"2024-03-23T03:49:34.992577+0000","last_active":"2024-03-23T03:49:35.461741+0000","last_peered":"2024-03-23T03:49:35.461741+0000","last_clean":"2024-03-23T03:49:35.461741+0000","last_became_active":"2024-03-23T03:49:34.991853+0000","last_became_peered":"2024-03-23T03:49:34.991853+0000","last_unstale":"2024-03-23T03:49:35.461741+0000","last_undegraded":"2024-03-23T03:49:35.461741+0000","last_fullsized":"2024-03-23T03:49:35.461741+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:45:28.059241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.462822+0000","last_change":"2024-03-23T03:49:35.447025+0000","last_active":"2024-03-23T03:49:35.462822+0000","last_peered":"2024-03-23T03:49:35.462822+0000","last_clean":"2024-03-23T03:49:35.462822+0000","last_became_active":"2024-03-23T03:49:35.446231+0000","last_became_peered":"2024-03-23T03:49:35.446231+0000","last_unstale":"2024-03-23T03:49:35.462822+0000","last_undegraded":"2024-03-23T03:49:35.462822+0000","last_fullsized":"2024-03-23T03:49:35.462822+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:16:15.365194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"66'6","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436596+0000","last_change":"2024-03-23T03:49:34.982535+0000","last_active":"2024-03-23T03:49:36.436596+0000","last_peered":"2024-03-23T03:49:36.436596+0000","last_clean":"2024-03-23T03:49:36.436596+0000","last_became_active":"2024-03-23T03:49:34.982293+0000","last_became_peered":"2024-03-23T03:49:34.982293+0000","last_unstale":"2024-03-23T03:49:36.436596+0000","last_undegraded":"2024-03-23T03:49:36.436596+0000","last_fullsized":"2024-03-23T03:49:36.436596+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:59:28.803065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.465237+0000","last_change":"2024-03-23T03:49:35.451899+0000","last_active":"2024-03-23T03:49:35.465237+0000","last_peered":"2024-03-23T03:49:35.465237+0000","last_clean":"2024-03-23T03:49:35.465237+0000","last_became_active":"2024-03-23T03:49:35.450542+0000","last_became_peered":"2024-03-23T03:49:35.450542+0000","last_unstale":"2024-03-23T03:49:35.465237+0000","last_undegraded":"2024-03-23T03:49:35.465237+0000","last_fullsized":"2024-03-23T03:49:35.465237+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:40:59.178993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"66'2","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.462261+0000","last_change":"2024-03-23T03:49:34.992490+0000","last_active":"2024-03-23T03:49:35.462261+0000","last_peered":"2024-03-23T03:49:35.462261+0000","last_clean":"2024-03-23T03:49:35.462261+0000","last_became_active":"2024-03-23T03:49:34.992098+0000","last_became_peered":"2024-03-23T03:49:34.992098+0000","last_unstale":"2024-03-23T03:49:35.462261+0000","last_undegraded":"2024-03-23T03:49:35.462261+0000","last_fullsized":"2024-03-23T03:49:35.462261+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:00:19.043728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.435960+0000","last_change":"2024-03-23T03:49:35.441503+0000","last_active":"2024-03-23T03:49:36.435960+0000","last_peered":"2024-03-23T03:49:36.435960+0000","last_clean":"2024-03-23T03:49:36.435960+0000","last_became_active":"2024-03-23T03:49:35.441247+0000","last_became_peered":"2024-03-23T03:49:35.441247+0000","last_unstale":"2024-03-23T03:49:36.435960+0000","last_undegraded":"2024-03-23T03:49:36.435960+0000","last_fullsized":"2024-03-23T03:49:36.435960+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:55:05.527463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.435974+0000","last_change":"2024-03-23T03:49:34.981217+0000","last_active":"2024-03-23T03:49:36.435974+0000","last_peered":"2024-03-23T03:49:36.435974+0000","last_clean":"2024-03-23T03:49:36.435974+0000","last_became_active":"2024-03-23T03:49:34.976302+0000","last_became_peered":"2024-03-23T03:49:34.976302+0000","last_unstale":"2024-03-23T03:49:36.435974+0000","last_undegraded":"2024-03-23T03:49:36.435974+0000","last_fullsized":"2024-03-23T03:49:36.435974+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:12:56.894905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.460257+0000","last_change":"2024-03-23T03:49:35.448205+0000","last_active":"2024-03-23T03:49:35.460257+0000","last_peered":"2024-03-23T03:49:35.460257+0000","last_clean":"2024-03-23T03:49:35.460257+0000","last_became_active":"2024-03-23T03:49:35.447452+0000","last_became_peered":"2024-03-23T03:49:35.447452+0000","last_unstale":"2024-03-23T03:49:35.460257+0000","last_undegraded":"2024-03-23T03:49:35.460257+0000","last_fullsized":"2024-03-23T03:49:35.460257+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:16:47.272585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.437354+0000","last_change":"2024-03-23T03:49:34.985827+0000","last_active":"2024-03-23T03:49:35.437354+0000","last_peered":"2024-03-23T03:49:35.437354+0000","last_clean":"2024-03-23T03:49:35.437354+0000","last_became_active":"2024-03-23T03:49:34.985545+0000","last_became_peered":"2024-03-23T03:49:34.985545+0000","last_unstale":"2024-03-23T03:49:35.437354+0000","last_undegraded":"2024-03-23T03:49:35.437354+0000","last_fullsized":"2024-03-23T03:49:35.437354+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:16:19.571022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436196+0000","last_change":"2024-03-23T03:49:35.453213+0000","last_active":"2024-03-23T03:49:36.436196+0000","last_peered":"2024-03-23T03:49:36.436196+0000","last_clean":"2024-03-23T03:49:36.436196+0000","last_became_active":"2024-03-23T03:49:35.446364+0000","last_became_peered":"2024-03-23T03:49:35.446364+0000","last_unstale":"2024-03-23T03:49:36.436196+0000","last_undegraded":"2024-03-23T03:49:36.436196+0000","last_fullsized":"2024-03-23T03:49:36.436196+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:59.256052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"66'8","reported_seq":18,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.467523+0000","last_change":"2024-03-23T03:49:34.988658+0000","last_active":"2024-03-23T03:49:35.467523+0000","last_peered":"2024-03-23T03:49:35.467523+0000","last_clean":"2024-03-23T03:49:35.467523+0000","last_became_active":"2024-03-23T03:49:34.988406+0000","last_became_peered":"2024-03-23T03:49:34.988406+0000","last_unstale":"2024-03-23T03:49:35.467523+0000","last_undegraded":"2024-03-23T03:49:35.467523+0000","last_fullsized":"2024-03-23T03:49:35.467523+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:03:13.452675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.467398+0000","last_change":"2024-03-23T03:49:35.465262+0000","last_active":"2024-03-23T03:49:35.467398+0000","last_peered":"2024-03-23T03:49:35.467398+0000","last_clean":"2024-03-23T03:49:35.467398+0000","last_became_active":"2024-03-23T03:49:35.464461+0000","last_became_peered":"2024-03-23T03:49:35.464461+0000","last_unstale":"2024-03-23T03:49:35.467398+0000","last_undegraded":"2024-03-23T03:49:35.467398+0000","last_fullsized":"2024-03-23T03:49:35.467398+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:48:11.607155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"66'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436519+0000","last_change":"2024-03-23T03:49:34.974809+0000","last_active":"2024-03-23T03:49:36.436519+0000","last_peered":"2024-03-23T03:49:36.436519+0000","last_clean":"2024-03-23T03:49:36.436519+0000","last_became_active":"2024-03-23T03:49:34.974425+0000","last_became_peered":"2024-03-23T03:49:34.974425+0000","last_unstale":"2024-03-23T03:49:36.436519+0000","last_undegraded":"2024-03-23T03:49:36.436519+0000","last_fullsized":"2024-03-23T03:49:36.436519+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:59:28.803065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.465694+0000","last_change":"2024-03-23T03:49:35.463979+0000","last_active":"2024-03-23T03:49:35.465694+0000","last_peered":"2024-03-23T03:49:35.465694+0000","last_clean":"2024-03-23T03:49:35.465694+0000","last_became_active":"2024-03-23T03:49:35.463719+0000","last_became_peered":"2024-03-23T03:49:35.463719+0000","last_unstale":"2024-03-23T03:49:35.465694+0000","last_undegraded":"2024-03-23T03:49:35.465694+0000","last_fullsized":"2024-03-23T03:49:35.465694+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:44:43.035657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"66'2","reported_seq":12,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.463585+0000","last_change":"2024-03-23T03:49:34.992119+0000","last_active":"2024-03-23T03:49:35.463585+0000","last_peered":"2024-03-23T03:49:35.463585+0000","last_clean":"2024-03-23T03:49:35.463585+0000","last_became_active":"2024-03-23T03:49:34.991843+0000","last_became_peered":"2024-03-23T03:49:34.991843+0000","last_unstale":"2024-03-23T03:49:35.463585+0000","last_undegraded":"2024-03-23T03:49:35.463585+0000","last_fullsized":"2024-03-23T03:49:35.463585+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:03:16.437712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.468239+0000","last_change":"2024-03-23T03:49:35.464324+0000","last_active":"2024-03-23T03:49:35.468239+0000","last_peered":"2024-03-23T03:49:35.468239+0000","last_clean":"2024-03-23T03:49:35.468239+0000","last_became_active":"2024-03-23T03:49:35.464019+0000","last_became_peered":"2024-03-23T03:49:35.464019+0000","last_unstale":"2024-03-23T03:49:35.468239+0000","last_undegraded":"2024-03-23T03:49:35.468239+0000","last_fullsized":"2024-03-23T03:49:35.468239+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:58:39.422775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"66'1","reported_seq":11,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.461619+0000","last_change":"2024-03-23T03:49:34.992779+0000","last_active":"2024-03-23T03:49:35.461619+0000","last_peered":"2024-03-23T03:49:35.461619+0000","last_clean":"2024-03-23T03:49:35.461619+0000","last_became_active":"2024-03-23T03:49:34.992252+0000","last_became_peered":"2024-03-23T03:49:34.992252+0000","last_unstale":"2024-03-23T03:49:35.461619+0000","last_undegraded":"2024-03-23T03:49:35.461619+0000","last_fullsized":"2024-03-23T03:49:35.461619+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:10:01.095131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.455634+0000","last_change":"2024-03-23T03:49:35.448284+0000","last_active":"2024-03-23T03:49:35.455634+0000","last_peered":"2024-03-23T03:49:35.455634+0000","last_clean":"2024-03-23T03:49:35.455634+0000","last_became_active":"2024-03-23T03:49:35.447986+0000","last_became_peered":"2024-03-23T03:49:35.447986+0000","last_unstale":"2024-03-23T03:49:35.455634+0000","last_undegraded":"2024-03-23T03:49:35.455634+0000","last_fullsized":"2024-03-23T03:49:35.455634+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:32:51.943312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.435546+0000","last_change":"2024-03-23T03:49:34.987012+0000","last_active":"2024-03-23T03:49:35.435546+0000","last_peered":"2024-03-23T03:49:35.435546+0000","last_clean":"2024-03-23T03:49:35.435546+0000","last_became_active":"2024-03-23T03:49:34.986690+0000","last_became_peered":"2024-03-23T03:49:34.986690+0000","last_unstale":"2024-03-23T03:49:35.435546+0000","last_undegraded":"2024-03-23T03:49:35.435546+0000","last_fullsized":"2024-03-23T03:49:35.435546+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:35:00.404847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650831+0000","last_change":"2024-03-23T03:49:35.457675+0000","last_active":"2024-03-23T03:49:36.650831+0000","last_peered":"2024-03-23T03:49:36.650831+0000","last_clean":"2024-03-23T03:49:36.650831+0000","last_became_active":"2024-03-23T03:49:35.457421+0000","last_became_peered":"2024-03-23T03:49:35.457421+0000","last_unstale":"2024-03-23T03:49:36.650831+0000","last_undegraded":"2024-03-23T03:49:36.650831+0000","last_fullsized":"2024-03-23T03:49:36.650831+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:36:42.984257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650294+0000","last_change":"2024-03-23T03:49:34.988857+0000","last_active":"2024-03-23T03:49:36.650294+0000","last_peered":"2024-03-23T03:49:36.650294+0000","last_clean":"2024-03-23T03:49:36.650294+0000","last_became_active":"2024-03-23T03:49:34.988325+0000","last_became_peered":"2024-03-23T03:49:34.988325+0000","last_unstale":"2024-03-23T03:49:36.650294+0000","last_undegraded":"2024-03-23T03:49:36.650294+0000","last_fullsized":"2024-03-23T03:49:36.650294+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:51:17.672508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.435911+0000","last_change":"2024-03-23T03:49:35.438748+0000","last_active":"2024-03-23T03:49:36.435911+0000","last_peered":"2024-03-23T03:49:36.435911+0000","last_clean":"2024-03-23T03:49:36.435911+0000","last_became_active":"2024-03-23T03:49:35.438469+0000","last_became_peered":"2024-03-23T03:49:35.438469+0000","last_unstale":"2024-03-23T03:49:36.435911+0000","last_undegraded":"2024-03-23T03:49:36.435911+0000","last_fullsized":"2024-03-23T03:49:36.435911+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:50:23.362975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436060+0000","last_change":"2024-03-23T03:49:34.976528+0000","last_active":"2024-03-23T03:49:36.436060+0000","last_peered":"2024-03-23T03:49:36.436060+0000","last_clean":"2024-03-23T03:49:36.436060+0000","last_became_active":"2024-03-23T03:49:34.976051+0000","last_became_peered":"2024-03-23T03:49:34.976051+0000","last_unstale":"2024-03-23T03:49:36.436060+0000","last_undegraded":"2024-03-23T03:49:36.436060+0000","last_fullsized":"2024-03-23T03:49:36.436060+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:12:35.311936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.460279+0000","last_change":"2024-03-23T03:49:35.453603+0000","last_active":"2024-03-23T03:49:35.460279+0000","last_peered":"2024-03-23T03:49:35.460279+0000","last_clean":"2024-03-23T03:49:35.460279+0000","last_became_active":"2024-03-23T03:49:35.449568+0000","last_became_peered":"2024-03-23T03:49:35.449568+0000","last_unstale":"2024-03-23T03:49:35.460279+0000","last_undegraded":"2024-03-23T03:49:35.460279+0000","last_fullsized":"2024-03-23T03:49:35.460279+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:44:03.577479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650560+0000","last_change":"2024-03-23T03:49:34.987803+0000","last_active":"2024-03-23T03:49:36.650560+0000","last_peered":"2024-03-23T03:49:36.650560+0000","last_clean":"2024-03-23T03:49:36.650560+0000","last_became_active":"2024-03-23T03:49:34.987511+0000","last_became_peered":"2024-03-23T03:49:34.987511+0000","last_unstale":"2024-03-23T03:49:36.650560+0000","last_undegraded":"2024-03-23T03:49:36.650560+0000","last_fullsized":"2024-03-23T03:49:36.650560+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:45:30.133076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650773+0000","last_change":"2024-03-23T03:49:35.464179+0000","last_active":"2024-03-23T03:49:36.650773+0000","last_peered":"2024-03-23T03:49:36.650773+0000","last_clean":"2024-03-23T03:49:36.650773+0000","last_became_active":"2024-03-23T03:49:35.463860+0000","last_became_peered":"2024-03-23T03:49:35.463860+0000","last_unstale":"2024-03-23T03:49:36.650773+0000","last_undegraded":"2024-03-23T03:49:36.650773+0000","last_fullsized":"2024-03-23T03:49:36.650773+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:02:35.537184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"66'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436667+0000","last_change":"2024-03-23T03:49:34.986922+0000","last_active":"2024-03-23T03:49:36.436667+0000","last_peered":"2024-03-23T03:49:36.436667+0000","last_clean":"2024-03-23T03:49:36.436667+0000","last_became_active":"2024-03-23T03:49:34.986632+0000","last_became_peered":"2024-03-23T03:49:34.986632+0000","last_unstale":"2024-03-23T03:49:36.436667+0000","last_undegraded":"2024-03-23T03:49:36.436667+0000","last_fullsized":"2024-03-23T03:49:36.436667+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:57:01.780866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.459687+0000","last_change":"2024-03-23T03:49:35.454834+0000","last_active":"2024-03-23T03:49:35.459687+0000","last_peered":"2024-03-23T03:49:35.459687+0000","last_clean":"2024-03-23T03:49:35.459687+0000","last_became_active":"2024-03-23T03:49:35.454564+0000","last_became_peered":"2024-03-23T03:49:35.454564+0000","last_unstale":"2024-03-23T03:49:35.459687+0000","last_undegraded":"2024-03-23T03:49:35.459687+0000","last_fullsized":"2024-03-23T03:49:35.459687+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:30:24.052315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436410+0000","last_change":"2024-03-23T03:49:34.974283+0000","last_active":"2024-03-23T03:49:36.436410+0000","last_peered":"2024-03-23T03:49:36.436410+0000","last_clean":"2024-03-23T03:49:36.436410+0000","last_became_active":"2024-03-23T03:49:34.973333+0000","last_became_peered":"2024-03-23T03:49:34.973333+0000","last_unstale":"2024-03-23T03:49:36.436410+0000","last_undegraded":"2024-03-23T03:49:36.436410+0000","last_fullsized":"2024-03-23T03:49:36.436410+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:47:27.337094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436244+0000","last_change":"2024-03-23T03:49:35.445810+0000","last_active":"2024-03-23T03:49:36.436244+0000","last_peered":"2024-03-23T03:49:36.436244+0000","last_clean":"2024-03-23T03:49:36.436244+0000","last_became_active":"2024-03-23T03:49:35.445512+0000","last_became_peered":"2024-03-23T03:49:35.445512+0000","last_unstale":"2024-03-23T03:49:36.436244+0000","last_undegraded":"2024-03-23T03:49:36.436244+0000","last_fullsized":"2024-03-23T03:49:36.436244+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:42:12.588157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.650894+0000","last_change":"2024-03-23T03:49:34.984506+0000","last_active":"2024-03-23T03:49:36.650894+0000","last_peered":"2024-03-23T03:49:36.650894+0000","last_clean":"2024-03-23T03:49:36.650894+0000","last_became_active":"2024-03-23T03:49:34.983529+0000","last_became_peered":"2024-03-23T03:49:34.983529+0000","last_unstale":"2024-03-23T03:49:36.650894+0000","last_undegraded":"2024-03-23T03:49:36.650894+0000","last_fullsized":"2024-03-23T03:49:36.650894+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:49:11.082091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.454744+0000","last_change":"2024-03-23T03:49:35.447908+0000","last_active":"2024-03-23T03:49:35.454744+0000","last_peered":"2024-03-23T03:49:35.454744+0000","last_clean":"2024-03-23T03:49:35.454744+0000","last_became_active":"2024-03-23T03:49:35.443188+0000","last_became_peered":"2024-03-23T03:49:35.443188+0000","last_unstale":"2024-03-23T03:49:35.454744+0000","last_undegraded":"2024-03-23T03:49:35.454744+0000","last_fullsized":"2024-03-23T03:49:35.454744+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:55:49.182877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.436584+0000","last_change":"2024-03-23T03:49:34.980078+0000","last_active":"2024-03-23T03:49:36.436584+0000","last_peered":"2024-03-23T03:49:36.436584+0000","last_clean":"2024-03-23T03:49:36.436584+0000","last_became_active":"2024-03-23T03:49:34.979815+0000","last_became_peered":"2024-03-23T03:49:34.979815+0000","last_unstale":"2024-03-23T03:49:36.436584+0000","last_undegraded":"2024-03-23T03:49:36.436584+0000","last_fullsized":"2024-03-23T03:49:36.436584+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:12:56.894905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.458135+0000","last_change":"2024-03-23T03:49:35.456042+0000","last_active":"2024-03-23T03:49:35.458135+0000","last_peered":"2024-03-23T03:49:35.458135+0000","last_clean":"2024-03-23T03:49:35.458135+0000","last_became_active":"2024-03-23T03:49:35.450458+0000","last_became_peered":"2024-03-23T03:49:35.450458+0000","last_unstale":"2024-03-23T03:49:35.458135+0000","last_undegraded":"2024-03-23T03:49:35.458135+0000","last_fullsized":"2024-03-23T03:49:35.458135+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:14:12.558627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.436609+0000","last_change":"2024-03-23T03:49:34.987542+0000","last_active":"2024-03-23T03:49:35.436609+0000","last_peered":"2024-03-23T03:49:35.436609+0000","last_clean":"2024-03-23T03:49:35.436609+0000","last_became_active":"2024-03-23T03:49:34.986389+0000","last_became_peered":"2024-03-23T03:49:34.986389+0000","last_unstale":"2024-03-23T03:49:35.436609+0000","last_undegraded":"2024-03-23T03:49:35.436609+0000","last_fullsized":"2024-03-23T03:49:35.436609+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:32:48.810886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.468200+0000","last_change":"2024-03-23T03:49:35.463865+0000","last_active":"2024-03-23T03:49:35.468200+0000","last_peered":"2024-03-23T03:49:35.468200+0000","last_clean":"2024-03-23T03:49:35.468200+0000","last_became_active":"2024-03-23T03:49:35.463091+0000","last_became_peered":"2024-03-23T03:49:35.463091+0000","last_unstale":"2024-03-23T03:49:35.468200+0000","last_undegraded":"2024-03-23T03:49:35.468200+0000","last_fullsized":"2024-03-23T03:49:35.468200+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:48:18.905476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.433420+0000","last_change":"2024-03-23T03:49:34.967418+0000","last_active":"2024-03-23T03:49:35.433420+0000","last_peered":"2024-03-23T03:49:35.433420+0000","last_clean":"2024-03-23T03:49:35.433420+0000","last_became_active":"2024-03-23T03:49:34.967079+0000","last_became_peered":"2024-03-23T03:49:34.967079+0000","last_unstale":"2024-03-23T03:49:35.433420+0000","last_undegraded":"2024-03-23T03:49:35.433420+0000","last_fullsized":"2024-03-23T03:49:35.433420+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:45:59.962877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.435897+0000","last_change":"2024-03-23T03:49:35.446298+0000","last_active":"2024-03-23T03:49:36.435897+0000","last_peered":"2024-03-23T03:49:36.435897+0000","last_clean":"2024-03-23T03:49:36.435897+0000","last_became_active":"2024-03-23T03:49:35.446060+0000","last_became_peered":"2024-03-23T03:49:35.446060+0000","last_unstale":"2024-03-23T03:49:36.435897+0000","last_undegraded":"2024-03-23T03:49:36.435897+0000","last_fullsized":"2024-03-23T03:49:36.435897+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:24:31.514455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.433328+0000","last_change":"2024-03-23T03:49:34.985626+0000","last_active":"2024-03-23T03:49:35.433328+0000","last_peered":"2024-03-23T03:49:35.433328+0000","last_clean":"2024-03-23T03:49:35.433328+0000","last_became_active":"2024-03-23T03:49:34.985354+0000","last_became_peered":"2024-03-23T03:49:34.985354+0000","last_unstale":"2024-03-23T03:49:35.433328+0000","last_undegraded":"2024-03-23T03:49:35.433328+0000","last_fullsized":"2024-03-23T03:49:35.433328+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:35:48.939105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.454591+0000","last_change":"2024-03-23T03:49:35.448917+0000","last_active":"2024-03-23T03:49:35.454591+0000","last_peered":"2024-03-23T03:49:35.454591+0000","last_clean":"2024-03-23T03:49:35.454591+0000","last_became_active":"2024-03-23T03:49:35.448608+0000","last_became_peered":"2024-03-23T03:49:35.448608+0000","last_unstale":"2024-03-23T03:49:35.454591+0000","last_undegraded":"2024-03-23T03:49:35.454591+0000","last_fullsized":"2024-03-23T03:49:35.454591+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:52:01.988987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-03-23T03:49:36.435953+0000","last_change":"2024-03-23T03:49:34.988815+0000","last_active":"2024-03-23T03:49:36.435953+0000","last_peered":"2024-03-23T03:49:36.435953+0000","last_clean":"2024-03-23T03:49:36.435953+0000","last_became_active":"2024-03-23T03:49:34.988496+0000","last_became_peered":"2024-03-23T03:49:34.988496+0000","last_unstale":"2024-03-23T03:49:36.435953+0000","last_undegraded":"2024-03-23T03:49:36.435953+0000","last_fullsized":"2024-03-23T03:49:36.435953+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:57:01.780866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":9,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.462761+0000","last_change":"2024-03-23T03:49:35.460666+0000","last_active":"2024-03-23T03:49:35.462761+0000","last_peered":"2024-03-23T03:49:35.462761+0000","last_clean":"2024-03-23T03:49:35.462761+0000","last_became_active":"2024-03-23T03:49:35.459729+0000","last_became_peered":"2024-03-23T03:49:35.459729+0000","last_unstale":"2024-03-23T03:49:35.462761+0000","last_undegraded":"2024-03-23T03:49:35.462761+0000","last_fullsized":"2024-03-23T03:49:35.462761+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:34.939331+0000","last_clean_scrub_stamp":"2024-03-23T03:49:34.939331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:40.677281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":10,"reported_epoch":66,"state":"active+clean","last_fresh":"2024-03-23T03:49:35.435008+0000","last_change":"2024-03-23T03:49:34.972259+0000","last_active":"2024-03-23T03:49:35.435008+0000","last_peered":"2024-03-23T03:49:35.435008+0000","last_clean":"2024-03-23T03:49:35.435008+0000","last_became_active":"2024-03-23T03:49:34.972003+0000","last_became_peered":"2024-03-23T03:49:34.972003+0000","last_unstale":"2024-03-23T03:49:35.435008+0000","last_undegraded":"2024-03-23T03:49:35.435008+0000","last_fullsized":"2024-03-23T03:49:35.435008+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:49:33.927511+0000","last_clean_scrub_stamp":"2024-03-23T03:49:33.927511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:41:06.334006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":201,"reported_epoch":65,"state":"active+clean","last_fresh":"2024-03-23T03:49:34.972946+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:49:34.972946+0000","last_peered":"2024-03-23T03:49:34.972946+0000","last_clean":"2024-03-23T03:49:34.972946+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:49:34.972946+0000","last_undegraded":"2024-03-23T03:49:34.972946+0000","last_fullsized":"2024-03-23T03:49:34.972946+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542182,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28400,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343440,"statfs":{"total":96636764160,"available":96607682560,"internally_reserved":0,"allocated":1404928,"data_stored":943779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.079,"15min":1.069},"min":{"1min":0.66800000000000004,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.6279999999999999,"5min":1.635,"15min":1.635},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.0860000000000001,"5min":1.0569999999999999,"15min":1.052},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.48,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.67100000000000004}]},{"osd":1,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.139,"5min":1.0589999999999999,"15min":1.0449999999999999},"min":{"1min":0.70099999999999996,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.655,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.2829999999999999,"15min":1.3180000000000001},"min":{"1min":0.72599999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.472,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":1.0960000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.147,"15min":1.159},"min":{"1min":0.79200000000000004,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.4870000000000001,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.198,"5min":1.4059999999999999,"15min":1.4410000000000001},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"max":{"1min":1.522,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.3500000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.4239999999999999,"15min":1.46},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.5900000000000001,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.044},{"interface":"front","average":{"1min":1.319,"5min":1.3580000000000001,"15min":1.3640000000000001},"min":{"1min":1.046,"5min":1.0429999999999999,"15min":1.0429999999999999},"max":{"1min":1.536,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.22}]},{"osd":4,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.2789999999999999,"15min":1.2709999999999999},"min":{"1min":0.98299999999999998,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.9370000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.381},{"interface":"front","average":{"1min":1.367,"5min":1.381,"15min":1.383},"min":{"1min":0.94899999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.9710000000000001,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.1339999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4350000000000001,"15min":1.4330000000000001},"min":{"1min":0.97699999999999998,"5min":0.93400000000000005,"15min":0.93400000000000005},"max":{"1min":1.7330000000000001,"5min":2.645,"15min":2.645},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4610000000000001,"15min":1.4610000000000001},"min":{"1min":1.0229999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"max":{"1min":2.1930000000000001,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.4670000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.5620000000000001,"15min":1.5720000000000001},"min":{"1min":1.2569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"max":{"1min":1.754,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.444,"5min":1.5169999999999999,"15min":1.5289999999999999},"min":{"1min":1.0489999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":2.0030000000000001,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.5549999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542180,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":798720,"data_stored":352669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.0369999999999999,"15min":1.042},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0429999999999999,"15min":1.038},"min":{"1min":0.74099999999999999,"5min":0.62,"15min":0.62},"max":{"1min":1.946,"5min":1.946,"15min":1.946},"last":1.3180000000000001}]},{"osd":1,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.052,"5min":1.0649999999999999,"15min":1.0669999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.3939999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.145,"15min":1.161},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.405,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.2729999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0680000000000001,"15min":1.0660000000000001},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":2.044,"5min":2.044,"15min":2.044},"last":1.044},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.212,"15min":1.224},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.419,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":1.135}]},{"osd":3,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.2749999999999999,"15min":1.2889999999999999},"min":{"1min":0.82899999999999996,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.643,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.2589999999999999,"15min":1.266},"min":{"1min":0.89600000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"max":{"1min":1.4870000000000001,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.5029999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.383,"15min":1.387},"min":{"1min":1.0860000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.734,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.363},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.379,"15min":1.389},"min":{"1min":0.871,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.5860000000000001,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.4099999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.431,"15min":1.4410000000000001},"min":{"1min":1.0029999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.7370000000000001,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.3},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.546,"15min":1.554},"min":{"1min":1.2410000000000001,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8220000000000001,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.4379999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:48:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4359999999999999,"5min":1.419,"15min":1.4159999999999999},"min":{"1min":1.0269999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.8460000000000001,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.163},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.54,"15min":1.55},"min":{"1min":1.018,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.802,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.343}]}]},{"osd":5,"up_from":14,"seq":60129542180,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":798720,"data_stored":352657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.1950000000000001,"15min":1.22},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.5,"5min":1.806,"15min":1.806},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.129,"5min":1.181,"15min":1.1890000000000001},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.6930000000000001,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.3049999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.671,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.153,"5min":1.1970000000000001,"15min":1.204},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":1.8180000000000001,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.036}]},{"osd":2,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.345,"15min":1.369},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.4950000000000001,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.361,"15min":1.387},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.556,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.84099999999999997}]},{"osd":3,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.2450000000000001,"15min":1.2589999999999999},"min":{"1min":0.73699999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.4950000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.298,"5min":1.4450000000000001,"15min":1.4690000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":1.925,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.385}]},{"osd":4,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.399,"15min":1.431},"min":{"1min":0.77600000000000002,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.8029999999999999,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.121},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.448,"15min":1.456},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":2.0209999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.3360000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.5,"15min":1.528},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.7390000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.444,"5min":1.639,"15min":1.671},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.8049999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.417}]},{"osd":7,"last update":"Sat Mar 23 03:48:48 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.599,"15min":1.627},"min":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"max":{"1min":2.0059999999999998,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.6850000000000001,"15min":1.7170000000000001},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2.0289999999999999,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.446}]}]},{"osd":4,"up_from":14,"seq":60129542181,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":806912,"data_stored":354199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.97599999999999998,"15min":0.97199999999999998},"min":{"1min":0.69799999999999995,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.26,"5min":1.383,"15min":1.383},"last":1.038},{"interface":"front","average":{"1min":1.012,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.69499999999999995,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.4950000000000001,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.117}]},{"osd":1,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.0509999999999999,"15min":1.044},"min":{"1min":0.78400000000000003,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.3759999999999999,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.139,"5min":1.069,"15min":1.0569999999999999},"min":{"1min":0.86699999999999999,"5min":0.496,"15min":0.496},"max":{"1min":1.387,"5min":1.4159999999999999,"15min":1.4159999999999999},"last":1.214}]},{"osd":2,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.3109999999999999,"15min":1.331},"min":{"1min":0.88200000000000001,"5min":0.625,"15min":0.625},"max":{"1min":1.464,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":1.472},{"interface":"front","average":{"1min":1.343,"5min":1.3140000000000001,"15min":1.3089999999999999},"min":{"1min":0.90400000000000003,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.6000000000000001,"5min":2.181,"15min":2.181},"last":1.4410000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.1220000000000001,"15min":1.1259999999999999},"min":{"1min":0.66800000000000004,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.423,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":1.272},{"interface":"front","average":{"1min":1.212,"5min":1.272,"15min":1.282},"min":{"1min":0.84599999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.548,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.173}]},{"osd":5,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.2729999999999999,"15min":1.272},"min":{"1min":0.69499999999999995,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.661,"5min":1.841,"15min":1.841},"last":1.363},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.28,"15min":1.2769999999999999},"min":{"1min":0.72199999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.589,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.399}]},{"osd":6,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.4059999999999999,"15min":1.421},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.6479999999999999,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.466,"5min":1.4079999999999999,"15min":1.3979999999999999},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.7789999999999999,"5min":1.877,"15min":1.877},"last":1.542}]},{"osd":7,"last update":"Sat Mar 23 03:48:49 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3660000000000001,"15min":1.3640000000000001},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.716,"5min":2.0369999999999999,"15min":2.0369999999999999},"last":1.498},{"interface":"front","average":{"1min":1.397,"5min":1.4810000000000001,"15min":1.4950000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.651,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.5660000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542181,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":794624,"data_stored":352635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.86399999999999999,"15min":0.87},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.37},{"interface":"front","average":{"1min":0.95199999999999996,"5min":0.93100000000000005,"15min":0.92700000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":5.8499999999999996,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.33800000000000002}]},{"osd":1,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.97499999999999998,"15min":0.98799999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":3.1850000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.91500000000000004,"5min":1.056,"15min":1.0800000000000001},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":2.7450000000000001,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.44900000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":1.04,"15min":1.0640000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.679,"5min":1.823,"15min":1.823},"last":0.435},{"interface":"front","average":{"1min":1.127,"5min":1.1559999999999999,"15min":1.161},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":4.8010000000000002,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":0.48199999999999998}]},{"osd":4,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.262,"15min":1.2549999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":5.9210000000000003,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.333,"5min":1.4410000000000001,"15min":1.4590000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":4.5620000000000003,"5min":4.819,"15min":4.819},"last":0.77100000000000002}]},{"osd":5,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":2.6640000000000001,"5min":1.6279999999999999,"15min":1.4550000000000001},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":30.228000000000002,"5min":30.228000000000002,"15min":30.228000000000002},"last":0.75},{"interface":"front","average":{"1min":1.373,"5min":1.3680000000000001,"15min":1.367},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":5.8920000000000003,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":0.91400000000000003}]},{"osd":6,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.506,"15min":1.54},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.9319999999999999,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.407,"15min":1.399},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":4.7610000000000001,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":0.96799999999999997}]},{"osd":7,"last update":"Sat Mar 23 03:48:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.429,"15min":1.4330000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":5.9589999999999996,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.6399999999999999,"15min":1.6220000000000001},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":10.663,"5min":10.663,"15min":10.663},"last":0.73299999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542181,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27800,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344040,"statfs":{"total":96636764160,"available":96608296960,"internally_reserved":0,"allocated":794624,"data_stored":352635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.86899999999999999,"15min":0.88700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.4299999999999999,"5min":1.4299999999999999,"15min":1.4299999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.91300000000000003,"15min":0.95199999999999996},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.2729999999999999,"5min":2.226,"15min":2.226},"last":0.56799999999999995}]},{"osd":1,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.97399999999999998,"15min":0.997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.956,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.90200000000000002,"15min":0.92400000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.665,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.60499999999999998}]},{"osd":3,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":1.008,"15min":1.034},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.6419999999999999,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.077,"15min":1.0920000000000001},"min":{"1min":0.53600000000000003,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.4690000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.97299999999999998}]},{"osd":4,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.2170000000000001,"15min":1.2390000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.605,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.1140000000000001,"5min":1.208,"15min":1.224},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.6910000000000001,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.86099999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.357,"15min":1.383},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.9339999999999999,"5min":1.97,"15min":1.97},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.3280000000000001,"15min":1.357},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.79,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.92900000000000005}]},{"osd":6,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.321,"15min":1.357},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":1.498,"5min":2.456,"15min":2.456},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.47,"15min":1.5},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.0310000000000001,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":1.284}]},{"osd":7,"last update":"Sat Mar 23 03:48:55 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.3640000000000001,"15min":1.3879999999999999},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.7849999999999999,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.069},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.3919999999999999,"15min":1.425},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.7769999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.78200000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542180,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27832,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344008,"statfs":{"total":96636764160,"available":96608264192,"internally_reserved":0,"allocated":823296,"data_stored":354963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.85099999999999998,"15min":0.88100000000000001},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.278,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.753,"5min":0.81999999999999995,"15min":0.83099999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.2509999999999999,"5min":1.327,"15min":1.327},"last":0.95399999999999996}]},{"osd":2,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.95099999999999996,"15min":0.97299999999999998},"min":{"1min":0.49199999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.405,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.879},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.97699999999999998,"15min":0.995},"min":{"1min":0.495,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.4079999999999999,"5min":1.625,"15min":1.625},"last":0.74199999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.1100000000000001,"15min":1.115},"min":{"1min":0.61099999999999999,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.6850000000000001,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.077},{"interface":"front","average":{"1min":0.88600000000000001,"5min":1.002,"15min":1.022},"min":{"1min":0.47099999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.4359999999999999,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.57399999999999995}]},{"osd":4,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.2669999999999999,"15min":1.304},"min":{"1min":0.79400000000000004,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.425,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.5},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.3049999999999999,"15min":1.325},"min":{"1min":0.79400000000000004,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.728,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.4750000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.319,"15min":1.3440000000000001},"min":{"1min":0.72599999999999998,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.5820000000000001,"5min":3.593,"15min":3.593},"last":1.167},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.4099999999999999,"15min":1.454},"min":{"1min":0.76700000000000002,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.496,"5min":2.359,"15min":2.359},"last":1.425}]},{"osd":6,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.389,"15min":1.419},"min":{"1min":0.75900000000000001,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.575,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.345},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.3500000000000001,"15min":1.3740000000000001},"min":{"1min":0.76900000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.766,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.2070000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:48:52 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.4259999999999999,"15min":1.4550000000000001},"min":{"1min":0.81999999999999995,"5min":0.627,"15min":0.627},"max":{"1min":1.821,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.329},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.472,"15min":1.522},"min":{"1min":0.77900000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.5269999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.556}]}]},{"osd":0,"up_from":14,"seq":60129542180,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28388,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343452,"statfs":{"total":96636764160,"available":96607694848,"internally_reserved":0,"allocated":1400832,"data_stored":949623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.94199999999999995,"15min":0.96099999999999997},"min":{"1min":0.46200000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.26,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.98799999999999999,"15min":0.996},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.506,"5min":1.506,"15min":1.506},"last":1.2050000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.98899999999999999,"15min":1.0109999999999999},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.339,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.105,"15min":1.119},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.7849999999999999,"5min":1.901,"15min":1.901},"last":0.96299999999999997}]},{"osd":3,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.0089999999999999,"15min":1.0209999999999999},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.2969999999999999,"5min":1.972,"15min":1.972},"last":1.5189999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.1419999999999999,"15min":1.1719999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3260000000000001,"5min":2.298,"15min":2.298},"last":0.61799999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.345,"15min":1.355},"min":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"max":{"1min":2.4940000000000002,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":1.294},{"interface":"front","average":{"1min":1.1779999999999999,"5min":1.27,"15min":1.286},"min":{"1min":0.83099999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":2.2400000000000002,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.425}]},{"osd":5,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.373,"15min":1.391},"min":{"1min":0.89200000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.0070000000000001,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.339},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.4379999999999999,"15min":1.458},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"max":{"1min":1.972,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.462}]},{"osd":6,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2649999999999999,"5min":1.514,"15min":1.556},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.8340000000000001,"5min":3.032,"15min":3.032},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.4910000000000001,"15min":1.514},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.9350000000000001,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.476}]},{"osd":7,"last update":"Sat Mar 23 03:48:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.5029999999999999,"15min":1.5269999999999999},"min":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"max":{"1min":2.3809999999999998,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.605},{"interface":"front","average":{"1min":1.298,"5min":1.4179999999999999,"15min":1.4379999999999999},"min":{"1min":0.85599999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.331,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.2490000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:49:40.358 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T03:49:40.358 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-23T03:49:40.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.607+0000 7f8d48c00640 1 Processor -- start 2024-03-23T03:49:40.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.608+0000 7f8d48c00640 1 -- start start 2024-03-23T03:49:40.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.609+0000 7f8d48c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f8d44159830 0x7f8d440cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:40.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.609+0000 7f8d48c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8d440ca7b0 0x7f8d440cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:40.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.609+0000 7f8d48c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d44159c10 con 0x7f8d440ce930 2024-03-23T03:49:40.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.609+0000 7f8d48c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d4406de50 con 0x7f8d440ca7b0 2024-03-23T03:49:40.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.609+0000 7f8d48c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d440cd840 con 0x7f8d44159830 2024-03-23T03:49:40.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.610+0000 7f8d3be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8d440ca7b0 0x7f8d440cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:40.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.610+0000 7f8d3be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8d440ca7b0 0x7f8d440cdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49930/0 (socket says 172.21.15.102:49930) 2024-03-23T03:49:40.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.610+0000 7f8d3be00640 1 -- 172.21.15.102:0/988968328 learned_addr learned my addr 172.21.15.102:0/988968328 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:40.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.610+0000 7f8d41e00640 1 --2- 172.21.15.102:0/988968328 >> v2:172.21.15.173:3300/0 conn(0x7f8d44159830 0x7f8d440cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:40.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.610+0000 7f8d3be00640 1 -- 172.21.15.102:0/988968328 >> v2:172.21.15.173:3300/0 conn(0x7f8d44159830 msgr2=0x7f8d440cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:40.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.610+0000 7f8d3be00640 1 --2- 172.21.15.102:0/988968328 >> v2:172.21.15.173:3300/0 conn(0x7f8d44159830 0x7f8d440cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.610+0000 7f8d3be00640 1 -- 172.21.15.102:0/988968328 >> v1:172.21.15.102:6789/0 conn(0x7f8d440ce930 legacy=0x7f8d440b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:40.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.610+0000 7f8d3be00640 1 -- 172.21.15.102:0/988968328 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d440ccc30 con 0x7f8d440ca7b0 2024-03-23T03:49:40.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.611+0000 7f8d3be00640 1 --2- 172.21.15.102:0/988968328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d440ca7b0 0x7f8d440cdcd0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c009870 tx=0x7f8d2c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=583490996531b9f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:40.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.612+0000 7f8d41400640 1 -- 172.21.15.102:0/988968328 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c02f840 con 0x7f8d440ca7b0 2024-03-23T03:49:40.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.612+0000 7f8d41400640 1 -- 172.21.15.102:0/988968328 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2c02f9a0 con 0x7f8d440ca7b0 2024-03-23T03:49:40.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.612+0000 7f8d41400640 1 -- 172.21.15.102:0/988968328 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c02fd30 con 0x7f8d440ca7b0 2024-03-23T03:49:40.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.612+0000 7f8d48c00640 1 -- 172.21.15.102:0/988968328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d440ca7b0 msgr2=0x7f8d440cdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:40.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.612+0000 7f8d48c00640 1 --2- 172.21.15.102:0/988968328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d440ca7b0 0x7f8d440cdcd0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c009870 tx=0x7f8d2c02ba20 comp rx=0 tx=0).stop 2024-03-23T03:49:40.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.612+0000 7f8d48c00640 1 -- 172.21.15.102:0/988968328 shutdown_connections 2024-03-23T03:49:40.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.612+0000 7f8d48c00640 1 --2- 172.21.15.102:0/988968328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d440ca7b0 0x7f8d440cdcd0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.612+0000 7f8d48c00640 1 --2- 172.21.15.102:0/988968328 >> v2:172.21.15.173:3300/0 conn(0x7f8d44159830 0x7f8d440cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.612+0000 7f8d48c00640 1 -- 172.21.15.102:0/988968328 >> 172.21.15.102:0/988968328 conn(0x7f8d4406a0b0 msgr2=0x7f8d4406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:40.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.612+0000 7f8d48c00640 1 -- 172.21.15.102:0/988968328 shutdown_connections 2024-03-23T03:49:40.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.612+0000 7f8d48c00640 1 -- 172.21.15.102:0/988968328 wait complete. 2024-03-23T03:49:40.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.614+0000 7f8d48c00640 1 Processor -- start 2024-03-23T03:49:40.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.614+0000 7f8d48c00640 1 -- start start 2024-03-23T03:49:40.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.615+0000 7f8d48c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d440ca7b0 0x7f8d4414b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:40.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.615+0000 7f8d48c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d440ce930 0x7f8d440b0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:40.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.615+0000 7f8d48c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d44159830 0x7f8d440b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:40.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.615+0000 7f8d48c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d440b0f80 con 0x7f8d440ce930 2024-03-23T03:49:40.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.615+0000 7f8d48c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d440ad230 con 0x7f8d440ca7b0 2024-03-23T03:49:40.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.615+0000 7f8d48c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d44159e60 con 0x7f8d44159830 2024-03-23T03:49:40.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.615+0000 7f8d3be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d440ce930 0x7f8d440b0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:40.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.615+0000 7f8d3be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d440ce930 0x7f8d440b0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49936/0 (socket says 172.21.15.102:49936) 2024-03-23T03:49:40.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.615+0000 7f8d3be00640 1 -- 172.21.15.102:0/732499578 learned_addr learned my addr 172.21.15.102:0/732499578 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:40.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.616+0000 7f8d41e00640 1 --2- 172.21.15.102:0/732499578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d440ca7b0 0x7f8d4414b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:40.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.616+0000 7f8d3be00640 1 -- 172.21.15.102:0/732499578 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d44159830 msgr2=0x7f8d440b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:40.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.616+0000 7f8d3be00640 1 --2- 172.21.15.102:0/732499578 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d44159830 0x7f8d440b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.616+0000 7f8d3be00640 1 -- 172.21.15.102:0/732499578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d440ca7b0 msgr2=0x7f8d4414b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:40.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.616+0000 7f8d3be00640 1 --2- 172.21.15.102:0/732499578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d440ca7b0 0x7f8d4414b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.616+0000 7f8d3be00640 1 -- 172.21.15.102:0/732499578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c0094d0 con 0x7f8d440ce930 2024-03-23T03:49:40.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.616+0000 7f8d3be00640 1 --2- 172.21.15.102:0/732499578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d440ce930 0x7f8d440b0500 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c000c00 tx=0x7f8d2c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:40.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.616+0000 7f8d3a000640 1 -- 172.21.15.102:0/732499578 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c037c40 con 0x7f8d440ce930 2024-03-23T03:49:40.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.616+0000 7f8d3a000640 1 -- 172.21.15.102:0/732499578 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d2c037da0 con 0x7f8d440ce930 2024-03-23T03:49:40.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.616+0000 7f8d3a000640 1 -- 172.21.15.102:0/732499578 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c041640 con 0x7f8d440ce930 2024-03-23T03:49:40.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.616+0000 7f8d48c00640 1 -- 172.21.15.102:0/732499578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d4415a140 con 0x7f8d440ce930 2024-03-23T03:49:40.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.617+0000 7f8d48c00640 1 -- 172.21.15.102:0/732499578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d4415a700 con 0x7f8d440ce930 2024-03-23T03:49:40.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.618+0000 7f8d48c00640 1 -- 172.21.15.102:0/732499578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d44073390 con 0x7f8d440ce930 2024-03-23T03:49:40.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.619+0000 7f8d3a000640 1 -- 172.21.15.102:0/732499578 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8d2c0417a0 con 0x7f8d440ce930 2024-03-23T03:49:40.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.620+0000 7f8d3a000640 1 --2- 172.21.15.102:0/732499578 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d2006ac00 0x7f8d2006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:40.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.620+0000 7f8d3a000640 1 -- 172.21.15.102:0/732499578 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8d2c03c070 con 0x7f8d440ce930 2024-03-23T03:49:40.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.620+0000 7f8d41e00640 1 --2- 172.21.15.102:0/732499578 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d2006ac00 0x7f8d2006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:40.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.621+0000 7f8d41e00640 1 --2- 172.21.15.102:0/732499578 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d2006ac00 0x7f8d2006d0c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8d34006fd0 tx=0x7f8d34008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:40.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.625+0000 7f8d3a000640 1 -- 172.21.15.102:0/732499578 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8d2c0df580 con 0x7f8d440ce930 2024-03-23T03:49:40.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.883+0000 7f8d48c00640 1 -- 172.21.15.102:0/732499578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f8d4415a8c0 con 0x7f8d440ce930 2024-03-23T03:49:40.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.890+0000 7f8d3a000640 1 -- 172.21.15.102:0/732499578 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+0 (secure 0 0 0) 0x7f8d2c07ac80 con 0x7f8d440ce930 2024-03-23T03:49:40.890 INFO:teuthology.orchestra.run.smithi102.stderr:updated caps for client.0 2024-03-23T03:49:40.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.892+0000 7f8d48c00640 1 -- 172.21.15.102:0/732499578 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d2006ac00 msgr2=0x7f8d2006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:40.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.892+0000 7f8d48c00640 1 --2- 172.21.15.102:0/732499578 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d2006ac00 0x7f8d2006d0c0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8d34006fd0 tx=0x7f8d34008040 comp rx=0 tx=0).stop 2024-03-23T03:49:40.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.892+0000 7f8d48c00640 1 -- 172.21.15.102:0/732499578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d440ce930 msgr2=0x7f8d440b0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:40.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.892+0000 7f8d48c00640 1 --2- 172.21.15.102:0/732499578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d440ce930 0x7f8d440b0500 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c000c00 tx=0x7f8d2c004730 comp rx=0 tx=0).stop 2024-03-23T03:49:40.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.892+0000 7f8d48c00640 1 -- 172.21.15.102:0/732499578 shutdown_connections 2024-03-23T03:49:40.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.892+0000 7f8d48c00640 1 --2- 172.21.15.102:0/732499578 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8d2006ac00 0x7f8d2006d0c0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.892+0000 7f8d48c00640 1 --2- 172.21.15.102:0/732499578 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d44159830 0x7f8d440b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.892+0000 7f8d48c00640 1 --2- 172.21.15.102:0/732499578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d440ce930 0x7f8d440b0500 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.892+0000 7f8d48c00640 1 --2- 172.21.15.102:0/732499578 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d440ca7b0 0x7f8d4414b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:40.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.892+0000 7f8d48c00640 1 -- 172.21.15.102:0/732499578 >> 172.21.15.102:0/732499578 conn(0x7f8d4406a0b0 msgr2=0x7f8d440b37b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:40.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.892+0000 7f8d48c00640 1 -- 172.21.15.102:0/732499578 shutdown_connections 2024-03-23T03:49:40.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:40.892+0000 7f8d48c00640 1 -- 172.21.15.102:0/732499578 wait complete. 2024-03-23T03:49:40.911 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:49:41.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.113+0000 7fc1ab200640 1 Processor -- start 2024-03-23T03:49:41.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.113+0000 7fc1ab200640 1 -- start start 2024-03-23T03:49:41.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.114+0000 7fc1ab200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc1ac1571d0 0x7fc1ac0b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:41.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.114+0000 7fc1ab200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc1ac0cdc00 0x7fc1ac15aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:41.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.114+0000 7fc1ab200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1ac1575b0 con 0x7fc1ac15afe0 2024-03-23T03:49:41.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.114+0000 7fc1ab200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1ac159f20 con 0x7fc1ac0cdc00 2024-03-23T03:49:41.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.114+0000 7fc1ab200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac15a200 con 0x7fc1ac1571d0 2024-03-23T03:49:41.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.115+0000 7fc1a9400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc1ac0cdc00 0x7fc1ac15aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:41.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.115+0000 7fc1a9e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc1ac1571d0 0x7fc1ac0b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:41.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.115+0000 7fc1a9400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc1ac0cdc00 0x7fc1ac15aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:36000/0 (socket says 172.21.15.102:36000) 2024-03-23T03:49:41.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.115+0000 7fc1a9400640 1 -- 172.21.15.102:0/1765925631 learned_addr learned my addr 172.21.15.102:0/1765925631 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:41.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.116+0000 7fc1a9400640 1 -- 172.21.15.102:0/1765925631 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc1ac1571d0 msgr2=0x7fc1ac0b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:41.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.116+0000 7fc1a9400640 1 --2- 172.21.15.102:0/1765925631 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc1ac1571d0 0x7fc1ac0b35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:41.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.116+0000 7fc1a9400640 1 -- 172.21.15.102:0/1765925631 >> v1:172.21.15.173:6789/0 conn(0x7fc1ac15afe0 legacy=0x7fc1ac0cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:41.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.116+0000 7fc1a9400640 1 -- 172.21.15.102:0/1765925631 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ac15a4e0 con 0x7fc1ac0cdc00 2024-03-23T03:49:41.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.117+0000 7fc1a9400640 1 --2- 172.21.15.102:0/1765925631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1ac0cdc00 0x7fc1ac15aaa0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc198009870 tx=0x7fc19802db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1018c34ad31826d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:41.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.118+0000 7fc1a8a00640 1 -- 172.21.15.102:0/1765925631 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19802e940 con 0x7fc1ac0cdc00 2024-03-23T03:49:41.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.118+0000 7fc1a8a00640 1 -- 172.21.15.102:0/1765925631 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc19802eaa0 con 0x7fc1ac0cdc00 2024-03-23T03:49:41.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.118+0000 7fc1a8a00640 1 -- 172.21.15.102:0/1765925631 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19802ee30 con 0x7fc1ac0cdc00 2024-03-23T03:49:41.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.118+0000 7fc1ab200640 1 -- 172.21.15.102:0/1765925631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1ac0cdc00 msgr2=0x7fc1ac15aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:41.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.118+0000 7fc1ab200640 1 --2- 172.21.15.102:0/1765925631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1ac0cdc00 0x7fc1ac15aaa0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fc198009870 tx=0x7fc19802db90 comp rx=0 tx=0).stop 2024-03-23T03:49:41.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.118+0000 7fc1ab200640 1 -- 172.21.15.102:0/1765925631 shutdown_connections 2024-03-23T03:49:41.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.118+0000 7fc1ab200640 1 --2- 172.21.15.102:0/1765925631 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1ac0cdc00 0x7fc1ac15aaa0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:41.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.118+0000 7fc1ab200640 1 --2- 172.21.15.102:0/1765925631 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc1ac1571d0 0x7fc1ac0b35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:41.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.118+0000 7fc1ab200640 1 -- 172.21.15.102:0/1765925631 >> 172.21.15.102:0/1765925631 conn(0x7fc1ac065df0 msgr2=0x7fc1ac069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:41.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.118+0000 7fc1ab200640 1 -- 172.21.15.102:0/1765925631 shutdown_connections 2024-03-23T03:49:41.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.118+0000 7fc1ab200640 1 -- 172.21.15.102:0/1765925631 wait complete. 2024-03-23T03:49:41.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.120+0000 7fc1ab200640 1 Processor -- start 2024-03-23T03:49:41.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.120+0000 7fc1ab200640 1 -- start start 2024-03-23T03:49:41.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.121+0000 7fc1ab200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc1ac0cdc00 0x7fc1ac0afc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:41.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.121+0000 7fc1ab200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1ac1571d0 0x7fc1ac141190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:41.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.121+0000 7fc1ab200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1ac15afe0 0x7fc1ac0b0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:41.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.121+0000 7fc1ab200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac0af7e0 con 0x7fc1ac15afe0 2024-03-23T03:49:41.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.121+0000 7fc1ab200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac0b0e00 con 0x7fc1ac1571d0 2024-03-23T03:49:41.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.121+0000 7fc1ab200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac0af190 con 0x7fc1ac0cdc00 2024-03-23T03:49:41.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.122+0000 7fc1aa800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1ac15afe0 0x7fc1ac0b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:41.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.122+0000 7fc1aa800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1ac15afe0 0x7fc1ac0b0290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49946/0 (socket says 172.21.15.102:49946) 2024-03-23T03:49:41.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.122+0000 7fc1aa800640 1 -- 172.21.15.102:0/1826629031 learned_addr learned my addr 172.21.15.102:0/1826629031 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:41.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.122+0000 7fc1a9e00640 1 --2- 172.21.15.102:0/1826629031 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc1ac0cdc00 0x7fc1ac0afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:41.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.122+0000 7fc1a9400640 1 --2- 172.21.15.102:0/1826629031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1ac1571d0 0x7fc1ac141190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:41.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.122+0000 7fc1aa800640 1 -- 172.21.15.102:0/1826629031 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc1ac0cdc00 msgr2=0x7fc1ac0afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:41.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.122+0000 7fc1aa800640 1 --2- 172.21.15.102:0/1826629031 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc1ac0cdc00 0x7fc1ac0afc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:41.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.122+0000 7fc1aa800640 1 -- 172.21.15.102:0/1826629031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1ac1571d0 msgr2=0x7fc1ac141190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:41.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.122+0000 7fc1aa800640 1 --2- 172.21.15.102:0/1826629031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1ac1571d0 0x7fc1ac141190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:41.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.122+0000 7fc1aa800640 1 -- 172.21.15.102:0/1826629031 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1980094d0 con 0x7fc1ac15afe0 2024-03-23T03:49:41.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.123+0000 7fc1aa800640 1 --2- 172.21.15.102:0/1826629031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1ac15afe0 0x7fc1ac0b0290 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc1a400c6b0 tx=0x7fc1a400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:41.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.123+0000 7fc1a2000640 1 -- 172.21.15.102:0/1826629031 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a4003b70 con 0x7fc1ac15afe0 2024-03-23T03:49:41.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.123+0000 7fc1a2000640 1 -- 172.21.15.102:0/1826629031 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a4003cf0 con 0x7fc1ac15afe0 2024-03-23T03:49:41.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.123+0000 7fc1a2000640 1 -- 172.21.15.102:0/1826629031 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a4003ea0 con 0x7fc1ac15afe0 2024-03-23T03:49:41.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.123+0000 7fc1ab200640 1 -- 172.21.15.102:0/1826629031 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ac0af470 con 0x7fc1ac15afe0 2024-03-23T03:49:41.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.123+0000 7fc1ab200640 1 -- 172.21.15.102:0/1826629031 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ac0b10e0 con 0x7fc1ac15afe0 2024-03-23T03:49:41.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.124+0000 7fc1ab200640 1 -- 172.21.15.102:0/1826629031 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1ac06e330 con 0x7fc1ac15afe0 2024-03-23T03:49:41.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.125+0000 7fc1a2000640 1 -- 172.21.15.102:0/1826629031 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc1a4005310 con 0x7fc1ac15afe0 2024-03-23T03:49:41.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.126+0000 7fc1a2000640 1 --2- 172.21.15.102:0/1826629031 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc18c06ac00 0x7fc18c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:41.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.126+0000 7fc1a2000640 1 -- 172.21.15.102:0/1826629031 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc1a4089280 con 0x7fc1ac15afe0 2024-03-23T03:49:41.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.126+0000 7fc1a9e00640 1 --2- 172.21.15.102:0/1826629031 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc18c06ac00 0x7fc18c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:41.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.127+0000 7fc1a9e00640 1 --2- 172.21.15.102:0/1826629031 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc18c06ac00 0x7fc18c06d0c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc19c009ad0 tx=0x7fc19c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:41.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.130+0000 7fc1a2000640 1 -- 172.21.15.102:0/1826629031 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc1a4054300 con 0x7fc1ac15afe0 2024-03-23T03:49:41.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.363+0000 7fc1ab200640 1 -- 172.21.15.102:0/1826629031 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1ac04e030 con 0x7fc1ac15afe0 2024-03-23T03:49:41.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.364+0000 7fc1a2000640 1 -- 172.21.15.102:0/1826629031 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4744 (secure 0 0 0) 0x7fc1a4057920 con 0x7fc1ac15afe0 2024-03-23T03:49:41.366 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:41.366 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5230,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3155961019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3155961019},{"type":"v1","addr":"172.21.15.173:6835","nonce":3155961019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6021,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/251450026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":251450026},{"type":"v1","addr":"172.21.15.102:6836","nonce":251450026}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6024,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3299889122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":3299889122},{"type":"v1","addr":"172.21.15.102:6837","nonce":3299889122}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:49:35.415835+0000","modified":"2024-03-23T03:49:38.434294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6030":{"gid":6030,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":4,"addr":"172.21.15.173:6833/3695568749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3695568749},{"type":"v1","addr":"172.21.15.173:6833","nonce":3695568749}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-23T03:49:41.367 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 33 2024-03-23T03:49:41.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.367+0000 7fc1ab200640 1 -- 172.21.15.102:0/1826629031 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc18c06ac00 msgr2=0x7fc18c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:41.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.367+0000 7fc1ab200640 1 --2- 172.21.15.102:0/1826629031 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc18c06ac00 0x7fc18c06d0c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fc19c009ad0 tx=0x7fc19c008040 comp rx=0 tx=0).stop 2024-03-23T03:49:41.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.367+0000 7fc1ab200640 1 -- 172.21.15.102:0/1826629031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1ac15afe0 msgr2=0x7fc1ac0b0290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:41.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.367+0000 7fc1ab200640 1 --2- 172.21.15.102:0/1826629031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1ac15afe0 0x7fc1ac0b0290 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fc1a400c6b0 tx=0x7fc1a400cb70 comp rx=0 tx=0).stop 2024-03-23T03:49:41.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.367+0000 7fc1ab200640 1 -- 172.21.15.102:0/1826629031 shutdown_connections 2024-03-23T03:49:41.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.367+0000 7fc1ab200640 1 --2- 172.21.15.102:0/1826629031 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc18c06ac00 0x7fc18c06d0c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:41.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.367+0000 7fc1ab200640 1 --2- 172.21.15.102:0/1826629031 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc1ac15afe0 0x7fc1ac0b0290 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:41.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.367+0000 7fc1ab200640 1 --2- 172.21.15.102:0/1826629031 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc1ac1571d0 0x7fc1ac141190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:41.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.367+0000 7fc1ab200640 1 --2- 172.21.15.102:0/1826629031 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc1ac0cdc00 0x7fc1ac0afc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:41.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.367+0000 7fc1ab200640 1 -- 172.21.15.102:0/1826629031 >> 172.21.15.102:0/1826629031 conn(0x7fc1ac065df0 msgr2=0x7fc1ac15c820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:41.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.367+0000 7fc1ab200640 1 -- 172.21.15.102:0/1826629031 shutdown_connections 2024-03-23T03:49:41.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:41.367+0000 7fc1ab200640 1 -- 172.21.15.102:0/1826629031 wait complete. 2024-03-23T03:49:41.789 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 33, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T03:49:35.415835+0000', 'modified': '2024-03-23T03:49:38.434294+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6030}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6030': {'gid': 6030, 'name': 'd', 'rank': 0, 'incarnation': 29, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.173:6833/3695568749', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.173:6832', 'nonce': 3695568749}, {'type': 'v1', 'addr': '172.21.15.173:6833', 'nonce': 3695568749}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T03:49:41.789 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T03:49:41.789 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-23T03:49:42.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.007+0000 7f1860c00640 1 Processor -- start 2024-03-23T03:49:42.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.008+0000 7f1860c00640 1 -- start start 2024-03-23T03:49:42.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.009+0000 7f1860c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f185c15c820 0x7f185c0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.009+0000 7f1860c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f185c160e90 0x7f185c0cf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.009+0000 7f1860c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185c0cfa30 con 0x7f185c0cf5f0 2024-03-23T03:49:42.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.009+0000 7f1860c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f185c0cdd70 con 0x7f185c160e90 2024-03-23T03:49:42.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.009+0000 7f1860c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185c0ccd90 con 0x7f185c15c820 2024-03-23T03:49:42.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.009+0000 7f185aa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f185c160e90 0x7f185c0cf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.009+0000 7f185aa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f185c160e90 0x7f185c0cf0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49952/0 (socket says 172.21.15.102:49952) 2024-03-23T03:49:42.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.009+0000 7f185be00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f185c0cf5f0 0x7f185c0d0560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50654/0 (socket says 172.21.15.102:50654) 2024-03-23T03:49:42.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.009+0000 7f185aa00640 1 -- 172.21.15.102:0/2461950375 learned_addr learned my addr 172.21.15.102:0/2461950375 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:42.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.010+0000 7f185b400640 1 --2- 172.21.15.102:0/2461950375 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f185c15c820 0x7f185c0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.010+0000 7f185aa00640 1 -- 172.21.15.102:0/2461950375 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f185c15c820 msgr2=0x7f185c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:42.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.010+0000 7f185aa00640 1 --2- 172.21.15.102:0/2461950375 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f185c15c820 0x7f185c0cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.010+0000 7f185aa00640 1 -- 172.21.15.102:0/2461950375 >> v1:172.21.15.102:6789/0 conn(0x7f185c0cf5f0 legacy=0x7f185c0d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:42.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.010+0000 7f185aa00640 1 -- 172.21.15.102:0/2461950375 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c0b7fc0 con 0x7f185c160e90 2024-03-23T03:49:42.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.011+0000 7f185aa00640 1 --2- 172.21.15.102:0/2461950375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185c160e90 0x7f185c0cf0b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f184c009e50 tx=0x7f184c02bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec2f3dff72656ec2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:42.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.011+0000 7f185a000640 1 -- 172.21.15.102:0/2461950375 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c02cb90 con 0x7f185c160e90 2024-03-23T03:49:42.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.011+0000 7f185a000640 1 -- 172.21.15.102:0/2461950375 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184c02ccf0 con 0x7f185c160e90 2024-03-23T03:49:42.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.011+0000 7f185a000640 1 -- 172.21.15.102:0/2461950375 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c034610 con 0x7f185c160e90 2024-03-23T03:49:42.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.012+0000 7f1860c00640 1 -- 172.21.15.102:0/2461950375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185c160e90 msgr2=0x7f185c0cf0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:42.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.012+0000 7f1860c00640 1 --2- 172.21.15.102:0/2461950375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185c160e90 0x7f185c0cf0b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f184c009e50 tx=0x7f184c02bff0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.012+0000 7f1860c00640 1 -- 172.21.15.102:0/2461950375 shutdown_connections 2024-03-23T03:49:42.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.012+0000 7f1860c00640 1 --2- 172.21.15.102:0/2461950375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185c160e90 0x7f185c0cf0b0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.012+0000 7f1860c00640 1 --2- 172.21.15.102:0/2461950375 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f185c15c820 0x7f185c0cd830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.012+0000 7f1860c00640 1 -- 172.21.15.102:0/2461950375 >> 172.21.15.102:0/2461950375 conn(0x7f185c06b070 msgr2=0x7f185c06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:42.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.012+0000 7f1860c00640 1 -- 172.21.15.102:0/2461950375 shutdown_connections 2024-03-23T03:49:42.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.012+0000 7f1860c00640 1 -- 172.21.15.102:0/2461950375 wait complete. 2024-03-23T03:49:42.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.014+0000 7f1860c00640 1 Processor -- start 2024-03-23T03:49:42.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.014+0000 7f1860c00640 1 -- start start 2024-03-23T03:49:42.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.015+0000 7f1860c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f185c0cf5f0 0x7f185c145ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.015+0000 7f1860c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f185c15c820 0x7f185c0b4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.015+0000 7f1860c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185c160e90 0x7f185c15cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.015+0000 7f1860c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c146390 con 0x7f185c160e90 2024-03-23T03:49:42.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.015+0000 7f1860c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f185c0b3180 con 0x7f185c15c820 2024-03-23T03:49:42.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.015+0000 7f1860c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f185c0b3460 con 0x7f185c0cf5f0 2024-03-23T03:49:42.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.015+0000 7f185be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185c160e90 0x7f185c15cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.015+0000 7f185be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185c160e90 0x7f185c15cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49962/0 (socket says 172.21.15.102:49962) 2024-03-23T03:49:42.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.015+0000 7f185be00640 1 -- 172.21.15.102:0/2808128195 learned_addr learned my addr 172.21.15.102:0/2808128195 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:42.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.016+0000 7f185aa00640 1 --2- 172.21.15.102:0/2808128195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f185c15c820 0x7f185c0b4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.016+0000 7f185be00640 1 -- 172.21.15.102:0/2808128195 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f185c0cf5f0 msgr2=0x7f185c145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:42.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.016+0000 7f185be00640 1 --2- 172.21.15.102:0/2808128195 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f185c0cf5f0 0x7f185c145ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.016+0000 7f185be00640 1 -- 172.21.15.102:0/2808128195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f185c15c820 msgr2=0x7f185c0b4220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:42.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.016+0000 7f185be00640 1 --2- 172.21.15.102:0/2808128195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f185c15c820 0x7f185c0b4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.016+0000 7f185be00640 1 -- 172.21.15.102:0/2808128195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c009b30 con 0x7f185c160e90 2024-03-23T03:49:42.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.016+0000 7f185be00640 1 --2- 172.21.15.102:0/2808128195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185c160e90 0x7f185c15cc00 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f1854001eb0 tx=0x7f185400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:42.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.016+0000 7f1853400640 1 -- 172.21.15.102:0/2808128195 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18540074e0 con 0x7f185c160e90 2024-03-23T03:49:42.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.016+0000 7f1853400640 1 -- 172.21.15.102:0/2808128195 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1854007640 con 0x7f185c160e90 2024-03-23T03:49:42.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.016+0000 7f1853400640 1 -- 172.21.15.102:0/2808128195 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18540049e0 con 0x7f185c160e90 2024-03-23T03:49:42.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.016+0000 7f1860c00640 1 -- 172.21.15.102:0/2808128195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185c0b3740 con 0x7f185c160e90 2024-03-23T03:49:42.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.016+0000 7f1860c00640 1 -- 172.21.15.102:0/2808128195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185c0b3d00 con 0x7f185c160e90 2024-03-23T03:49:42.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.018+0000 7f1853400640 1 -- 172.21.15.102:0/2808128195 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1854004b40 con 0x7f185c160e90 2024-03-23T03:49:42.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.019+0000 7f1853400640 1 --2- 172.21.15.102:0/2808128195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f183806ac00 0x7f183806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.019+0000 7f1853400640 1 -- 172.21.15.102:0/2808128195 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6174+0+0 (secure 0 0 0) 0x7f1854088ed0 con 0x7f185c160e90 2024-03-23T03:49:42.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.019+0000 7f185b400640 1 --2- 172.21.15.102:0/2808128195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f183806ac00 0x7f183806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.020+0000 7f1864e3a740 1 Processor -- start 2024-03-23T03:49:42.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.020+0000 7f185b400640 1 --2- 172.21.15.102:0/2808128195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f183806ac00 0x7f183806d0c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f18440098f0 tx=0x7f1844008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:42.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.020+0000 7f1864e3a740 1 -- start start 2024-03-23T03:49:42.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.021+0000 7f1864e3a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e879eb0 0x56400e84ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.021+0000 7f1864e3a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e87a290 0x56400e87cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.021+0000 7f1864e3a740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e84d390 0x56400e87d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.021+0000 7f1864e3a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x56400e8282c0 con 0x56400e87a290 2024-03-23T03:49:42.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.021+0000 7f1864e3a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56400e7bd8f0 con 0x56400e879eb0 2024-03-23T03:49:42.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.021+0000 7f1864e3a740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x56400e7bb080 con 0x56400e84d390 2024-03-23T03:49:42.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.021+0000 7f185aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e87a290 0x56400e87cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.021+0000 7f185aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e87a290 0x56400e87cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49968/0 (socket says 172.21.15.102:49968) 2024-03-23T03:49:42.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.021+0000 7f185aa00640 1 -- 172.21.15.102:0/555029108 learned_addr learned my addr 172.21.15.102:0/555029108 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:42.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.022+0000 7f185be00640 1 --2- 172.21.15.102:0/555029108 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e84d390 0x56400e87d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.022+0000 7f185b400640 1 --2- 172.21.15.102:0/555029108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e879eb0 0x56400e84ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.022+0000 7f185aa00640 1 -- 172.21.15.102:0/555029108 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e84d390 msgr2=0x56400e87d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:42.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.022+0000 7f185aa00640 1 --2- 172.21.15.102:0/555029108 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e84d390 0x56400e87d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.022+0000 7f185aa00640 1 -- 172.21.15.102:0/555029108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e879eb0 msgr2=0x56400e84ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:42.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.022+0000 7f185aa00640 1 --2- 172.21.15.102:0/555029108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e879eb0 0x56400e84ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.022+0000 7f185aa00640 1 -- 172.21.15.102:0/555029108 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56400e781a10 con 0x56400e87a290 2024-03-23T03:49:42.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.022+0000 7f185b400640 1 --2- 172.21.15.102:0/555029108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e879eb0 0x56400e84ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:42.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.022+0000 7f185aa00640 1 --2- 172.21.15.102:0/555029108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e87a290 0x56400e87cc40 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f184c009e50 tx=0x7f184c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38db15c68ba5f9b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:42.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.022+0000 7f184be00640 1 -- 172.21.15.102:0/555029108 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c02cb90 con 0x56400e87a290 2024-03-23T03:49:42.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.022+0000 7f184be00640 1 -- 172.21.15.102:0/555029108 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184c03e430 con 0x56400e87a290 2024-03-23T03:49:42.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.022+0000 7f184be00640 1 -- 172.21.15.102:0/555029108 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c03e7c0 con 0x56400e87a290 2024-03-23T03:49:42.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.023+0000 7f1864e3a740 1 -- 172.21.15.102:0/555029108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e87a290 msgr2=0x56400e87cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:42.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.023+0000 7f1864e3a740 1 --2- 172.21.15.102:0/555029108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e87a290 0x56400e87cc40 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f184c009e50 tx=0x7f184c02e000 comp rx=0 tx=0).stop 2024-03-23T03:49:42.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.023+0000 7f1864e3a740 1 -- 172.21.15.102:0/555029108 shutdown_connections 2024-03-23T03:49:42.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.023+0000 7f1864e3a740 1 --2- 172.21.15.102:0/555029108 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e84d390 0x56400e87d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.023+0000 7f1864e3a740 1 --2- 172.21.15.102:0/555029108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e87a290 0x56400e87cc40 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.023+0000 7f1864e3a740 1 --2- 172.21.15.102:0/555029108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e879eb0 0x56400e84ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.023+0000 7f1864e3a740 1 -- 172.21.15.102:0/555029108 >> 172.21.15.102:0/555029108 conn(0x56400e84f0c0 msgr2=0x56400e897c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:42.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.023+0000 7f1864e3a740 1 -- 172.21.15.102:0/555029108 shutdown_connections 2024-03-23T03:49:42.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.023+0000 7f1864e3a740 1 -- 172.21.15.102:0/555029108 wait complete. 2024-03-23T03:49:42.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.024+0000 7f1864e3a740 1 -- start start 2024-03-23T03:49:42.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.024+0000 7f1864e3a740 1 Processor -- start 2024-03-23T03:49:42.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.025+0000 7f1864e3a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e879eb0 0x56400e8691e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.025+0000 7f1864e3a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e87a290 0x56400e87ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.025+0000 7f1864e3a740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e877a20 0x56400e87bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.025+0000 7f1864e3a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x56400e7b6970 con 0x56400e879eb0 2024-03-23T03:49:42.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.025+0000 7f1864e3a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56400e7580a0 con 0x56400e87a290 2024-03-23T03:49:42.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.025+0000 7f1864e3a740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x56400e757d40 con 0x56400e877a20 2024-03-23T03:49:42.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.025+0000 7f1864e3a740 10 client.0 _ll_register_callbacks cb 0x56400e7affe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T03:49:42.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.025+0000 7f185b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e879eb0 0x56400e8691e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.025+0000 7f185b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e879eb0 0x56400e8691e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49984/0 (socket says 172.21.15.102:49984) 2024-03-23T03:49:42.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.025+0000 7f185b400640 1 -- 172.21.15.102:0/1002340687 learned_addr learned my addr 172.21.15.102:0/1002340687 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:42.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.026+0000 7f185be00640 1 --2- 172.21.15.102:0/1002340687 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e877a20 0x56400e87bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.026+0000 7f185b400640 1 -- 172.21.15.102:0/1002340687 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e877a20 msgr2=0x56400e87bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:42.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.026+0000 7f185b400640 1 --2- 172.21.15.102:0/1002340687 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e877a20 0x56400e87bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.026+0000 7f185aa00640 1 --2- 172.21.15.102:0/1002340687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e87a290 0x56400e87ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.026+0000 7f185b400640 1 -- 172.21.15.102:0/1002340687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e87a290 msgr2=0x56400e87ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:42.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.026+0000 7f185b400640 1 --2- 172.21.15.102:0/1002340687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e87a290 0x56400e87ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.026+0000 7f185b400640 1 -- 172.21.15.102:0/1002340687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f185c0b7fc0 con 0x56400e879eb0 2024-03-23T03:49:42.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.026+0000 7f185b400640 1 --2- 172.21.15.102:0/1002340687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e879eb0 0x56400e8691e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f184400def0 tx=0x7f1844006e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b419de0d4dd83a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:42.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.026+0000 7f1848c00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-23T03:49:42.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.026+0000 7f1848c00640 1 -- 172.21.15.102:0/1002340687 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844014e40 con 0x56400e879eb0 2024-03-23T03:49:42.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.026+0000 7f1848c00640 1 -- 172.21.15.102:0/1002340687 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6174+0+0 (secure 0 0 0) 0x7f18440125f0 con 0x56400e879eb0 2024-03-23T03:49:42.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.026+0000 7f1864e3a740 1 -- 172.21.15.102:0/1002340687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56400e7e0b10 con 0x56400e879eb0 2024-03-23T03:49:42.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.027+0000 7f1848c00640 1 -- 172.21.15.102:0/1002340687 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1844005990 con 0x56400e879eb0 2024-03-23T03:49:42.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.027+0000 7f1864e3a740 10 client.6150 fetch_fsmap learned FSMap version 33 2024-03-23T03:49:42.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.027+0000 7f1864e3a740 1 -- 172.21.15.102:0/1002340687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x56400e7b4ff0 con 0x56400e879eb0 2024-03-23T03:49:42.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.027+0000 7f1848c00640 1 -- 172.21.15.102:0/1002340687 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56400e7e0b10 con 0x56400e879eb0 2024-03-23T03:49:42.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.027+0000 7f1848c00640 1 -- 172.21.15.102:0/1002340687 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1844005af0 con 0x56400e879eb0 2024-03-23T03:49:42.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.027+0000 7f1864e3a740 10 client.6150 fetch_fsmap finished waiting for FSMap version 33 2024-03-23T03:49:42.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.027+0000 7f1864e3a740 10 client.6150 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-23T03:49:42.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.027+0000 7f1864e3a740 20 client.6150 populate_metadata read hostname 'smithi102' 2024-03-23T03:49:42.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.027+0000 7f1864e3a740 1 --2- 172.21.15.102:0/1002340687 >> [v2:172.21.15.173:6832/3695568749,v1:172.21.15.173:6833/3695568749] conn(0x56400e8b2bf0 0x56400e8b4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.027+0000 7f1864e3a740 4 client.6150 mds_command: new command op to 6030 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T03:49:42.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.027+0000 7f1864e3a740 1 -- 172.21.15.102:0/1002340687 --> [v2:172.21.15.173:6832/3695568749,v1:172.21.15.173:6833/3695568749] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56400e4aa3c0 con 0x56400e8b2bf0 2024-03-23T03:49:42.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.028+0000 7f185aa00640 1 --2- 172.21.15.102:0/1002340687 >> [v2:172.21.15.173:6832/3695568749,v1:172.21.15.173:6833/3695568749] conn(0x56400e8b2bf0 0x56400e8b4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.029+0000 7f185aa00640 1 --2- 172.21.15.102:0/1002340687 >> [v2:172.21.15.173:6832/3695568749,v1:172.21.15.173:6833/3695568749] conn(0x56400e8b2bf0 0x56400e8b4fe0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29ef9cdf3c8a23c6 server_cookie=8b6175d8a97c3498 in_seq=0 out_seq=0 2024-03-23T03:49:42.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.029+0000 7f1848c00640 10 client.6150 ms_handle_connect on v2:172.21.15.173:6832/3695568749 2024-03-23T03:49:42.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.031+0000 7f1848c00640 1 -- 172.21.15.102:0/1002340687 <== mds.0 v2:172.21.15.173:6832/3695568749 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x56400e4aa3c0 con 0x56400e8b2bf0 2024-03-23T03:49:42.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.031+0000 7f1848c00640 10 client.6150 handle_command_reply: tid=0 2024-03-23T03:49:42.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.031+0000 7f1864e3a740 1 client.6150 shutdown 2024-03-23T03:49:42.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.031+0000 7f1864e3a740 1 -- 172.21.15.102:0/1002340687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e879eb0 msgr2=0x56400e8691e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:42.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.031+0000 7f1864e3a740 1 --2- 172.21.15.102:0/1002340687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e879eb0 0x56400e8691e0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f184400def0 tx=0x7f1844006e10 comp rx=0 tx=0).stop 2024-03-23T03:49:42.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.031+0000 7f1864e3a740 1 -- 172.21.15.102:0/1002340687 shutdown_connections 2024-03-23T03:49:42.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.031+0000 7f1864e3a740 1 --2- 172.21.15.102:0/1002340687 >> [v2:172.21.15.173:6832/3695568749,v1:172.21.15.173:6833/3695568749] conn(0x56400e8b2bf0 0x56400e8b4fe0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.031+0000 7f1864e3a740 1 --2- 172.21.15.102:0/1002340687 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e877a20 0x56400e87bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.031+0000 7f1864e3a740 1 --2- 172.21.15.102:0/1002340687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e87a290 0x56400e87ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.031+0000 7f1848c00640 0 client.6150 ms_handle_reset on v2:172.21.15.173:6832/3695568749 2024-03-23T03:49:42.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.031+0000 7f1864e3a740 1 --2- 172.21.15.102:0/1002340687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e879eb0 0x56400e8691e0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.031+0000 7f1864e3a740 1 -- 172.21.15.102:0/1002340687 >> 172.21.15.102:0/1002340687 conn(0x56400e84d390 msgr2=0x56400e89b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:42.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.031+0000 7f1864e3a740 1 -- 172.21.15.102:0/1002340687 shutdown_connections 2024-03-23T03:49:42.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.031+0000 7f1864e3a740 1 -- 172.21.15.102:0/1002340687 wait complete. 2024-03-23T03:49:42.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.031+0000 7f1864e3a740 20 client.6150 trim_cache size 0 max 16384 2024-03-23T03:49:42.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.047+0000 7f1864e3a740 1 Processor -- start 2024-03-23T03:49:42.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.047+0000 7f1864e3a740 1 -- start start 2024-03-23T03:49:42.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.048+0000 7f1864e3a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e877a20 0x56400e89b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.048+0000 7f1864e3a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e879eb0 0x56400e89b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.048+0000 7f1864e3a740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e87a290 0x56400e8762b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.048+0000 7f1864e3a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x56400e51df20 con 0x56400e879eb0 2024-03-23T03:49:42.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.048+0000 7f1864e3a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56400e880480 con 0x56400e877a20 2024-03-23T03:49:42.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.048+0000 7f1864e3a740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x56400e8775e0 con 0x56400e87a290 2024-03-23T03:49:42.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.048+0000 7f185aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e879eb0 0x56400e89b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.049+0000 7f185aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e879eb0 0x56400e89b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49992/0 (socket says 172.21.15.102:49992) 2024-03-23T03:49:42.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.049+0000 7f185aa00640 1 -- 172.21.15.102:0/3088441008 learned_addr learned my addr 172.21.15.102:0/3088441008 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:42.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.049+0000 7f185b400640 1 --2- 172.21.15.102:0/3088441008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e877a20 0x56400e89b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.049+0000 7f185aa00640 1 -- 172.21.15.102:0/3088441008 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e87a290 msgr2=0x56400e8762b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:42.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.049+0000 7f185aa00640 1 --2- 172.21.15.102:0/3088441008 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e87a290 0x56400e8762b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.049+0000 7f185aa00640 1 -- 172.21.15.102:0/3088441008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e877a20 msgr2=0x56400e89b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:42.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.049+0000 7f185aa00640 1 --2- 172.21.15.102:0/3088441008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e877a20 0x56400e89b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.049+0000 7f185aa00640 1 -- 172.21.15.102:0/3088441008 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56400e7ea7c0 con 0x56400e879eb0 2024-03-23T03:49:42.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.049+0000 7f185b400640 1 --2- 172.21.15.102:0/3088441008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e877a20 0x56400e89b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:42.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.049+0000 7f185aa00640 1 --2- 172.21.15.102:0/3088441008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e879eb0 0x56400e89b690 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f184c02c600 tx=0x7f184c009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c537f833a633742c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:42.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.049+0000 7f1850c00640 1 -- 172.21.15.102:0/3088441008 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c003990 con 0x56400e879eb0 2024-03-23T03:49:42.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.049+0000 7f1850c00640 1 -- 172.21.15.102:0/3088441008 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184c003af0 con 0x56400e879eb0 2024-03-23T03:49:42.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.049+0000 7f1850c00640 1 -- 172.21.15.102:0/3088441008 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184c003e60 con 0x56400e879eb0 2024-03-23T03:49:42.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.050+0000 7f1864e3a740 1 -- 172.21.15.102:0/3088441008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e879eb0 msgr2=0x56400e89b690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:42.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.050+0000 7f1864e3a740 1 --2- 172.21.15.102:0/3088441008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e879eb0 0x56400e89b690 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f184c02c600 tx=0x7f184c009810 comp rx=0 tx=0).stop 2024-03-23T03:49:42.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.050+0000 7f1864e3a740 1 -- 172.21.15.102:0/3088441008 shutdown_connections 2024-03-23T03:49:42.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.050+0000 7f1864e3a740 1 --2- 172.21.15.102:0/3088441008 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e87a290 0x56400e8762b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.050+0000 7f1864e3a740 1 --2- 172.21.15.102:0/3088441008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e879eb0 0x56400e89b690 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.050+0000 7f1864e3a740 1 --2- 172.21.15.102:0/3088441008 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e877a20 0x56400e89b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.050+0000 7f1864e3a740 1 -- 172.21.15.102:0/3088441008 >> 172.21.15.102:0/3088441008 conn(0x56400e84d390 msgr2=0x56400e87b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:42.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.050+0000 7f1864e3a740 1 -- 172.21.15.102:0/3088441008 shutdown_connections 2024-03-23T03:49:42.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.050+0000 7f1864e3a740 1 -- 172.21.15.102:0/3088441008 wait complete. 2024-03-23T03:49:42.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.051+0000 7f1864e3a740 1 -- start start 2024-03-23T03:49:42.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.051+0000 7f1864e3a740 1 Processor -- start 2024-03-23T03:49:42.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.052+0000 7f1864e3a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e877a20 0x56400e8e4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.052+0000 7f1864e3a740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e879eb0 0x56400e87b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.052+0000 7f1864e3a740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e87a290 0x56400e87bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.052+0000 7f1864e3a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x56400e8fdf60 con 0x56400e877a20 2024-03-23T03:49:42.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.052+0000 7f1864e3a740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x56400e865f00 con 0x56400e879eb0 2024-03-23T03:49:42.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.052+0000 7f1864e3a740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x56400e84ec00 con 0x56400e87a290 2024-03-23T03:49:42.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.052+0000 7f1864e3a740 10 client.0 _ll_register_callbacks cb 0x56400e892170 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T03:49:42.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.052+0000 7f185b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e877a20 0x56400e8e4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.052+0000 7f185b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e877a20 0x56400e8e4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50004/0 (socket says 172.21.15.102:50004) 2024-03-23T03:49:42.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.052+0000 7f185b400640 1 -- 172.21.15.102:0/2616363997 learned_addr learned my addr 172.21.15.102:0/2616363997 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:42.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.053+0000 7f185b400640 1 -- 172.21.15.102:0/2616363997 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e87a290 msgr2=0x56400e87bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:42.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.053+0000 7f185b400640 1 --2- 172.21.15.102:0/2616363997 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e87a290 0x56400e87bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.053+0000 7f185aa00640 1 --2- 172.21.15.102:0/2616363997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e879eb0 0x56400e87b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.053+0000 7f185b400640 1 -- 172.21.15.102:0/2616363997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e879eb0 msgr2=0x56400e87b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:42.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.053+0000 7f185b400640 1 --2- 172.21.15.102:0/2616363997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e879eb0 0x56400e87b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.053+0000 7f185b400640 1 -- 172.21.15.102:0/2616363997 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56400e781a10 con 0x56400e877a20 2024-03-23T03:49:42.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.053+0000 7f185b400640 1 --2- 172.21.15.102:0/2616363997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e877a20 0x56400e8e4cf0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1844012850 tx=0x7f1844012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae6e40c857081392 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:42.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.053+0000 7f1848c00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-23T03:49:42.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.053+0000 7f1848c00640 1 -- 172.21.15.102:0/2616363997 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1844012ce0 con 0x56400e877a20 2024-03-23T03:49:42.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.053+0000 7f1848c00640 1 -- 172.21.15.102:0/2616363997 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6174+0+0 (secure 0 0 0) 0x7f1844012e40 con 0x56400e877a20 2024-03-23T03:49:42.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.053+0000 7f1864e3a740 1 -- 172.21.15.102:0/2616363997 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f184c03e430 con 0x56400e877a20 2024-03-23T03:49:42.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.053+0000 7f1848c00640 1 -- 172.21.15.102:0/2616363997 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1844005fb0 con 0x56400e877a20 2024-03-23T03:49:42.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.054+0000 7f1864e3a740 10 client.6156 fetch_fsmap learned FSMap version 33 2024-03-23T03:49:42.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.054+0000 7f1864e3a740 1 -- 172.21.15.102:0/2616363997 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x56400e7572b0 con 0x56400e877a20 2024-03-23T03:49:42.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.054+0000 7f1848c00640 1 -- 172.21.15.102:0/2616363997 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184c03e430 con 0x56400e877a20 2024-03-23T03:49:42.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.054+0000 7f1848c00640 1 -- 172.21.15.102:0/2616363997 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1844006110 con 0x56400e877a20 2024-03-23T03:49:42.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.054+0000 7f1864e3a740 10 client.6156 fetch_fsmap finished waiting for FSMap version 33 2024-03-23T03:49:42.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.054+0000 7f1864e3a740 10 client.6156 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-23T03:49:42.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.054+0000 7f1864e3a740 20 client.6156 populate_metadata read hostname 'smithi102' 2024-03-23T03:49:42.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.054+0000 7f1864e3a740 1 --2- 172.21.15.102:0/2616363997 >> [v2:172.21.15.173:6832/3695568749,v1:172.21.15.173:6833/3695568749] conn(0x56400e84c360 0x56400e84c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.054+0000 7f1864e3a740 4 client.6156 mds_command: new command op to 6030 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-23T03:49:42.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.054+0000 7f1864e3a740 1 -- 172.21.15.102:0/2616363997 --> [v2:172.21.15.173:6832/3695568749,v1:172.21.15.173:6833/3695568749] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56400e4bb6c0 con 0x56400e84c360 2024-03-23T03:49:42.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.055+0000 7f185aa00640 1 --2- 172.21.15.102:0/2616363997 >> [v2:172.21.15.173:6832/3695568749,v1:172.21.15.173:6833/3695568749] conn(0x56400e84c360 0x56400e84c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.055+0000 7f185aa00640 1 --2- 172.21.15.102:0/2616363997 >> [v2:172.21.15.173:6832/3695568749,v1:172.21.15.173:6833/3695568749] conn(0x56400e84c360 0x56400e84c740 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96bf4636743a956d server_cookie=8a61d16b0e72a36b in_seq=0 out_seq=0 2024-03-23T03:49:42.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.055+0000 7f1848c00640 10 client.6156 ms_handle_connect on v2:172.21.15.173:6832/3695568749 2024-03-23T03:49:42.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.056+0000 7f1848c00640 1 -- 172.21.15.102:0/2616363997 <== mds.0 v2:172.21.15.173:6832/3695568749 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x56400e4bb6c0 con 0x56400e84c360 2024-03-23T03:49:42.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.056+0000 7f1848c00640 10 client.6156 handle_command_reply: tid=0 2024-03-23T03:49:42.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.056+0000 7f1864e3a740 1 client.6156 shutdown 2024-03-23T03:49:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.056+0000 7f1864e3a740 1 -- 172.21.15.102:0/2616363997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e877a20 msgr2=0x56400e8e4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.056+0000 7f1864e3a740 1 --2- 172.21.15.102:0/2616363997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e877a20 0x56400e8e4cf0 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1844012850 tx=0x7f1844012000 comp rx=0 tx=0).stop 2024-03-23T03:49:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.056+0000 7f1864e3a740 1 -- 172.21.15.102:0/2616363997 shutdown_connections 2024-03-23T03:49:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.056+0000 7f1864e3a740 1 --2- 172.21.15.102:0/2616363997 >> [v2:172.21.15.173:6832/3695568749,v1:172.21.15.173:6833/3695568749] conn(0x56400e84c360 0x56400e84c740 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.056+0000 7f1864e3a740 1 --2- 172.21.15.102:0/2616363997 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x56400e87a290 0x56400e87bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.056+0000 7f1864e3a740 1 --2- 172.21.15.102:0/2616363997 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x56400e879eb0 0x56400e87b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.056+0000 7f1848c00640 0 client.6156 ms_handle_reset on v2:172.21.15.173:6832/3695568749 2024-03-23T03:49:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.056+0000 7f1864e3a740 1 --2- 172.21.15.102:0/2616363997 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56400e877a20 0x56400e8e4cf0 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.056+0000 7f1864e3a740 1 -- 172.21.15.102:0/2616363997 >> 172.21.15.102:0/2616363997 conn(0x56400e84d390 msgr2=0x56400e891400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.056+0000 7f1864e3a740 1 -- 172.21.15.102:0/2616363997 shutdown_connections 2024-03-23T03:49:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.057+0000 7f1864e3a740 1 -- 172.21.15.102:0/2616363997 wait complete. 2024-03-23T03:49:42.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.057+0000 7f1864e3a740 20 client.6156 trim_cache size 0 max 16384 2024-03-23T03:49:42.060 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:49:42.060 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "776b2820-05f2-4e50-8d40-18dd76c5f6a0", 2024-03-23T03:49:42.060 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 0, 2024-03-23T03:49:42.060 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 6030, 2024-03-23T03:49:42.060 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-03-23T03:49:42.060 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-03-23T03:49:42.060 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-03-23T03:49:42.060 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 6.6358988050000001, 2024-03-23T03:49:42.060 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 33, 2024-03-23T03:49:42.060 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 68, 2024-03-23T03:49:42.060 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 67, 2024-03-23T03:49:42.060 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 18.518855726000002 2024-03-23T03:49:42.060 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:49:42.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.057+0000 7f1860c00640 1 -- 172.21.15.102:0/2808128195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f183806ac00 msgr2=0x7f183806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:42.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.057+0000 7f1860c00640 1 --2- 172.21.15.102:0/2808128195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f183806ac00 0x7f183806d0c0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f18440098f0 tx=0x7f1844008040 comp rx=0 tx=0).stop 2024-03-23T03:49:42.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.057+0000 7f1860c00640 1 -- 172.21.15.102:0/2808128195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185c160e90 msgr2=0x7f185c15cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:42.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.057+0000 7f1860c00640 1 --2- 172.21.15.102:0/2808128195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185c160e90 0x7f185c15cc00 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f1854001eb0 tx=0x7f185400c820 comp rx=0 tx=0).stop 2024-03-23T03:49:42.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.057+0000 7f1860c00640 1 -- 172.21.15.102:0/2808128195 shutdown_connections 2024-03-23T03:49:42.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.057+0000 7f1860c00640 1 --2- 172.21.15.102:0/2808128195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f183806ac00 0x7f183806d0c0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.057+0000 7f1860c00640 1 --2- 172.21.15.102:0/2808128195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f185c160e90 0x7f185c15cc00 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.057+0000 7f1860c00640 1 --2- 172.21.15.102:0/2808128195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f185c15c820 0x7f185c0b4220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.057+0000 7f1860c00640 1 --2- 172.21.15.102:0/2808128195 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f185c0cf5f0 0x7f185c145ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.057+0000 7f1860c00640 1 -- 172.21.15.102:0/2808128195 >> 172.21.15.102:0/2808128195 conn(0x7f185c06b070 msgr2=0x7f185c0d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:42.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.057+0000 7f1860c00640 1 -- 172.21.15.102:0/2808128195 shutdown_connections 2024-03-23T03:49:42.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.058+0000 7f1860c00640 1 -- 172.21.15.102:0/2808128195 wait complete. 2024-03-23T03:49:42.074 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:49:42.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.277+0000 7fe3a2000640 1 Processor -- start 2024-03-23T03:49:42.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.278+0000 7fe3a2000640 1 -- start start 2024-03-23T03:49:42.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.279+0000 7fe3a2000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe39c15ccd0 0x7fe39c06be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.279+0000 7fe3a2000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39c06c380 con 0x7fe39c098a40 2024-03-23T03:49:42.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.279+0000 7fe3a2000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39c06ada0 con 0x7fe39c0cf3c0 2024-03-23T03:49:42.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.279+0000 7fe3a2000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe39c06b080 con 0x7fe39c15ccd0 2024-03-23T03:49:42.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.279+0000 7fe39b400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe39c098a40 0x7fe39c098e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50656/0 (socket says 172.21.15.102:50656) 2024-03-23T03:49:42.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.279+0000 7fe39b400640 1 -- 172.21.15.102:0/1702742350 learned_addr learned my addr 172.21.15.102:0/1702742350 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:42.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.279+0000 7fe39aa00640 1 --2- 172.21.15.102:0/1702742350 >> v2:172.21.15.102:3300/0 conn(0x7fe39c15ccd0 0x7fe39c06be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.280+0000 7fe39a000640 1 -- 172.21.15.102:0/1702742350 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216828925 0 0) 0x7fe38c001110 con 0x7fe39c098a40 2024-03-23T03:49:42.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.280+0000 7fe39a000640 1 -- 172.21.15.102:0/1702742350 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe380003520 con 0x7fe39c098a40 2024-03-23T03:49:42.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.280+0000 7fe39aa00640 1 -- 172.21.15.102:0/1702742350 >> v1:172.21.15.173:6789/0 conn(0x7fe39c0cf3c0 legacy=0x7fe39c06e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:42.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.280+0000 7fe39aa00640 1 -- 172.21.15.102:0/1702742350 >> v1:172.21.15.102:6789/0 conn(0x7fe39c098a40 legacy=0x7fe39c098e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:42.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.280+0000 7fe39aa00640 1 -- 172.21.15.102:0/1702742350 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39c06b360 con 0x7fe39c15ccd0 2024-03-23T03:49:42.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.281+0000 7fe39aa00640 1 --2- 172.21.15.102:0/1702742350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe39c15ccd0 0x7fe39c06be40 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fe3880097a0 tx=0x7fe38802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39810ab9a7f4912 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:42.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.281+0000 7fe39a000640 1 -- 172.21.15.102:0/1702742350 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38802e070 con 0x7fe39c15ccd0 2024-03-23T03:49:42.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.281+0000 7fe39a000640 1 -- 172.21.15.102:0/1702742350 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3880315a0 con 0x7fe39c15ccd0 2024-03-23T03:49:42.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.281+0000 7fe39a000640 1 -- 172.21.15.102:0/1702742350 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe388031930 con 0x7fe39c15ccd0 2024-03-23T03:49:42.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.282+0000 7fe3a2000640 1 -- 172.21.15.102:0/1702742350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe39c15ccd0 msgr2=0x7fe39c06be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:42.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.282+0000 7fe3a2000640 1 --2- 172.21.15.102:0/1702742350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe39c15ccd0 0x7fe39c06be40 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fe3880097a0 tx=0x7fe38802b840 comp rx=0 tx=0).stop 2024-03-23T03:49:42.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.282+0000 7fe3a2000640 1 -- 172.21.15.102:0/1702742350 shutdown_connections 2024-03-23T03:49:42.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.282+0000 7fe3a2000640 1 --2- 172.21.15.102:0/1702742350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe39c15ccd0 0x7fe39c06be40 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.282+0000 7fe3a2000640 1 -- 172.21.15.102:0/1702742350 >> 172.21.15.102:0/1702742350 conn(0x7fe39c0b2160 msgr2=0x7fe39c0b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:42.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.282+0000 7fe3a2000640 1 -- 172.21.15.102:0/1702742350 shutdown_connections 2024-03-23T03:49:42.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.282+0000 7fe3a2000640 1 -- 172.21.15.102:0/1702742350 wait complete. 2024-03-23T03:49:42.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.284+0000 7fe3a2000640 1 Processor -- start 2024-03-23T03:49:42.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.284+0000 7fe3a2000640 1 -- start start 2024-03-23T03:49:42.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.285+0000 7fe3a2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe39c098a40 0x7fe39c0b5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.285+0000 7fe3a2000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe39c0cf3c0 0x7fe39c1f2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.285+0000 7fe3a2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe39c15ccd0 0x7fe39c15b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.285+0000 7fe3a2000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c1f35d0 con 0x7fe39c15ccd0 2024-03-23T03:49:42.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.285+0000 7fe3a2000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c1f3010 con 0x7fe39c098a40 2024-03-23T03:49:42.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.285+0000 7fe3a2000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c1f32f0 con 0x7fe39c0cf3c0 2024-03-23T03:49:42.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.285+0000 7fe39be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe39c15ccd0 0x7fe39c15b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.285+0000 7fe39be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe39c15ccd0 0x7fe39c15b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50024/0 (socket says 172.21.15.102:50024) 2024-03-23T03:49:42.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.285+0000 7fe39be00640 1 -- 172.21.15.102:0/2644138020 learned_addr learned my addr 172.21.15.102:0/2644138020 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:42.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.285+0000 7fe39be00640 1 -- 172.21.15.102:0/2644138020 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe39c0cf3c0 msgr2=0x7fe39c1f2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:42.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.285+0000 7fe39be00640 1 --2- 172.21.15.102:0/2644138020 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe39c0cf3c0 0x7fe39c1f2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.285+0000 7fe39be00640 1 -- 172.21.15.102:0/2644138020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe39c098a40 msgr2=0x7fe39c0b5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:42.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.285+0000 7fe39be00640 1 --2- 172.21.15.102:0/2644138020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe39c098a40 0x7fe39c0b5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.285+0000 7fe39be00640 1 -- 172.21.15.102:0/2644138020 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe388009400 con 0x7fe39c15ccd0 2024-03-23T03:49:42.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.286+0000 7fe39be00640 1 --2- 172.21.15.102:0/2644138020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe39c15ccd0 0x7fe39c15b590 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fe39400c780 tx=0x7fe39400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:42.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.286+0000 7fe393400640 1 -- 172.21.15.102:0/2644138020 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe394004b60 con 0x7fe39c15ccd0 2024-03-23T03:49:42.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.286+0000 7fe393400640 1 -- 172.21.15.102:0/2644138020 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39400ce90 con 0x7fe39c15ccd0 2024-03-23T03:49:42.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.286+0000 7fe3a2000640 1 -- 172.21.15.102:0/2644138020 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe39c0c5320 con 0x7fe39c15ccd0 2024-03-23T03:49:42.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.286+0000 7fe393400640 1 -- 172.21.15.102:0/2644138020 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39400a850 con 0x7fe39c15ccd0 2024-03-23T03:49:42.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.286+0000 7fe3a2000640 1 -- 172.21.15.102:0/2644138020 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39c1f37f0 con 0x7fe39c15ccd0 2024-03-23T03:49:42.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.287+0000 7fe3a2000640 1 -- 172.21.15.102:0/2644138020 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe39c0c5840 con 0x7fe39c15ccd0 2024-03-23T03:49:42.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.288+0000 7fe393400640 1 -- 172.21.15.102:0/2644138020 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe39400a9b0 con 0x7fe39c15ccd0 2024-03-23T03:49:42.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.289+0000 7fe393400640 1 --2- 172.21.15.102:0/2644138020 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe37c06ac00 0x7fe37c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:42.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.289+0000 7fe393400640 1 -- 172.21.15.102:0/2644138020 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6174+0+0 (secure 0 0 0) 0x7fe394089700 con 0x7fe39c15ccd0 2024-03-23T03:49:42.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.289+0000 7fe39b400640 1 --2- 172.21.15.102:0/2644138020 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe37c06ac00 0x7fe37c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:42.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.290+0000 7fe39b400640 1 --2- 172.21.15.102:0/2644138020 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe37c06ac00 0x7fe37c06d0c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe38c005880 tx=0x7fe38c001150 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:42.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.294+0000 7fe393400640 1 -- 172.21.15.102:0/2644138020 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe39c0c5840 con 0x7fe39c15ccd0 2024-03-23T03:49:42.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.532+0000 7fe3a2000640 1 -- 172.21.15.102:0/2644138020 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe39c0c5840 con 0x7fe39c15ccd0 2024-03-23T03:49:42.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.533+0000 7fe393400640 1 -- 172.21.15.102:0/2644138020 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v33) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe39c0c5840 con 0x7fe39c15ccd0 2024-03-23T03:49:42.534 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T03:49:42.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.536+0000 7fe3a2000640 1 -- 172.21.15.102:0/2644138020 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe37c06ac00 msgr2=0x7fe37c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:42.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.536+0000 7fe3a2000640 1 --2- 172.21.15.102:0/2644138020 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe37c06ac00 0x7fe37c06d0c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe38c005880 tx=0x7fe38c001150 comp rx=0 tx=0).stop 2024-03-23T03:49:42.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.536+0000 7fe3a2000640 1 -- 172.21.15.102:0/2644138020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe39c15ccd0 msgr2=0x7fe39c15b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:42.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.536+0000 7fe3a2000640 1 --2- 172.21.15.102:0/2644138020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe39c15ccd0 0x7fe39c15b590 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7fe39400c780 tx=0x7fe39400cc40 comp rx=0 tx=0).stop 2024-03-23T03:49:42.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.536+0000 7fe3a2000640 1 -- 172.21.15.102:0/2644138020 shutdown_connections 2024-03-23T03:49:42.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.536+0000 7fe3a2000640 1 --2- 172.21.15.102:0/2644138020 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe37c06ac00 0x7fe37c06d0c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.536+0000 7fe3a2000640 1 --2- 172.21.15.102:0/2644138020 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe39c15ccd0 0x7fe39c15b590 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.536+0000 7fe3a2000640 1 --2- 172.21.15.102:0/2644138020 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe39c0cf3c0 0x7fe39c1f2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.536+0000 7fe3a2000640 1 --2- 172.21.15.102:0/2644138020 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe39c098a40 0x7fe39c0b5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:42.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.536+0000 7fe3a2000640 1 -- 172.21.15.102:0/2644138020 >> 172.21.15.102:0/2644138020 conn(0x7fe39c0b2160 msgr2=0x7fe39c0b2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:42.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.536+0000 7fe3a2000640 1 -- 172.21.15.102:0/2644138020 shutdown_connections 2024-03-23T03:49:42.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:42.536+0000 7fe3a2000640 1 -- 172.21.15.102:0/2644138020 wait complete. 2024-03-23T03:49:42.555 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-23T03:49:42.556 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:49:42.556 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2024-03-23T03:49:42.556 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T03:49:42.556 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T03:49:42.556 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-23T03:49:42.556 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-23T03:49:42.556 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:49:42.556 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:49:42.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:42.556 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2024-03-23T03:49:42.607 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:49:42.608 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:49:42.608 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:49:42.608 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:42.608 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-03-23T03:49:42.608 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:42.608 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:49:42.608 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:42.608 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f0 2024-03-23T03:49:42.608 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:49:42.608 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:42.608 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f1 2024-03-23T03:49:42.608 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:49:42.609 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:42.609 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:49:42.609 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:42.609 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:49:42.609 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 42488sec preferred_lft 42488sec 2024-03-23T03:49:42.609 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-03-23T03:49:42.609 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:42.609 INFO:teuthology.orchestra.run.smithi102.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:49:42.609 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether aa:17:e3:39:b0:56 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:42.609 INFO:teuthology.orchestra.run.smithi102.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T03:49:42.609 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:42.609 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::5cce:1fff:fe41:5a05/64 scope link 2024-03-23T03:49:42.609 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:42.609 INFO:teuthology.orchestra.run.smithi102.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:49:42.609 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether aa:17:e3:39:b0:56 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T03:49:42.610 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::a817:e3ff:fe39:b056/64 scope link 2024-03-23T03:49:42.610 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:42.610 INFO:teuthology.orchestra.run.smithi102.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:49:42.610 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether ce:9b:95:fb:de:c5 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T03:49:42.610 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::cc9b:95ff:fefb:dec5/64 scope link 2024-03-23T03:49:42.610 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:42.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:42.611 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:49:42.702 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-23T03:49:42.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:42.706 DEBUG:teuthology.orchestra.run.smithi102:> route 2024-03-23T03:49:42.754 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2024-03-23T03:49:42.754 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:49:42.754 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:49:42.754 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:49:42.754 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:49:42.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:42.755 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:49:42.755 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:49:42.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:49:42.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:49:42.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:49:42.755 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:49:42.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:42.930 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-23T03:49:42.979 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T03:49:42.979 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T03:49:42.980 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:49:42.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:42.980 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.0 up 2024-03-23T03:49:43.062 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:49:43.063 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T03:49:43.113 INFO:teuthology.orchestra.run.smithi102.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:49:43.113 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:49:43.114 DEBUG:teuthology.orchestra.run.smithi102:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T03:49:43.162 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-23T03:49:43.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:43.163 DEBUG:teuthology.orchestra.run.smithi102:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:49:43.296 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:49:43.296 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:49:43.296 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:49:43.319 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:49:43.319 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:49:43.319 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:49:43.319 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:49:43.319 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:49:43.347 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:49:43.347 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:49:43.347 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:49:43.347 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:49:43.348 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:49:43.348 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790" 2024-03-23T03:49:43.348 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/ 2024-03-23T03:49:43.348 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=776b2820-05f2-4e50-8d40-18dd76c5f6a0" will pass to kernel 2024-03-23T03:49:43.350 INFO:teuthology.orchestra.run.smithi102.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-23T03:49:43.350 INFO:teuthology.orchestra.run.smithi102.stdout: You just mounted a file system that supports labels which does not 2024-03-23T03:49:43.351 INFO:teuthology.orchestra.run.smithi102.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T03:49:43.351 INFO:teuthology.orchestra.run.smithi102.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T03:49:43.351 INFO:teuthology.orchestra.run.smithi102.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T03:49:43.354 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T03:49:43.355 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:43.355 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T03:49:43.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:43.429 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:49:43.499 INFO:teuthology.orchestra.run.smithi102.stdout:client6168 2024-03-23T03:49:43.500 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:49:43.500 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:43.552 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:49:43.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.756+0000 7ffa4c200640 1 Processor -- start 2024-03-23T03:49:43.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.756+0000 7ffa4c200640 1 -- start start 2024-03-23T03:49:43.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.757+0000 7ffa4c200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ffa4415b560 0x7ffa4415b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:43.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.757+0000 7ffa4c200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ffa440b39b0 0x7ffa440b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:43.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.757+0000 7ffa4c200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa4415e500 0x7ffa4415e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:43.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.757+0000 7ffa4c200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa440b4370 con 0x7ffa4415b560 2024-03-23T03:49:43.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.757+0000 7ffa4c200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa4406c6f0 con 0x7ffa440b39b0 2024-03-23T03:49:43.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.757+0000 7ffa4c200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa4406c9d0 con 0x7ffa4415e500 2024-03-23T03:49:43.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.758+0000 7ffa49800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ffa4415b560 0x7ffa4415b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:43.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.758+0000 7ffa49800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ffa4415b560 0x7ffa4415b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54264/0 (socket says 172.21.15.102:54264) 2024-03-23T03:49:43.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.758+0000 7ffa49800640 1 -- 172.21.15.102:0/2391816539 learned_addr learned my addr 172.21.15.102:0/2391816539 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:43.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.758+0000 7ffa4a200640 1 --2- 172.21.15.102:0/2391816539 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa4415e500 0x7ffa4415e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:43.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.758+0000 7ffa48e00640 1 --2- 172.21.15.102:0/2391816539 >> v2:172.21.15.173:3300/0 conn(0x7ffa440b39b0 0x7ffa440b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:43.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.758+0000 7ffa49800640 1 -- 172.21.15.102:0/2391816539 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa4415e500 msgr2=0x7ffa4415e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:43.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.758+0000 7ffa49800640 1 --2- 172.21.15.102:0/2391816539 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa4415e500 0x7ffa4415e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:43.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.758+0000 7ffa49800640 1 -- 172.21.15.102:0/2391816539 >> v2:172.21.15.173:3300/0 conn(0x7ffa440b39b0 msgr2=0x7ffa440b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:43.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.758+0000 7ffa49800640 1 --2- 172.21.15.102:0/2391816539 >> v2:172.21.15.173:3300/0 conn(0x7ffa440b39b0 0x7ffa440b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:43.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.758+0000 7ffa49800640 1 -- 172.21.15.102:0/2391816539 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa4406ccb0 con 0x7ffa4415b560 2024-03-23T03:49:43.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.758+0000 7ffa48e00640 1 --2- 172.21.15.102:0/2391816539 >> v2:172.21.15.173:3300/0 conn(0x7ffa440b39b0 0x7ffa440b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:43.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.760+0000 7ffa49800640 1 --2- 172.21.15.102:0/2391816539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa4415b560 0x7ffa4415b940 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ffa440ce180 tx=0x7ffa3802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82876fa0094775af server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:43.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.760+0000 7ffa43e00640 1 -- 172.21.15.102:0/2391816539 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa38031860 con 0x7ffa4415b560 2024-03-23T03:49:43.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.760+0000 7ffa43e00640 1 -- 172.21.15.102:0/2391816539 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa380319c0 con 0x7ffa4415b560 2024-03-23T03:49:43.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.760+0000 7ffa43e00640 1 -- 172.21.15.102:0/2391816539 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa38031d50 con 0x7ffa4415b560 2024-03-23T03:49:43.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.761+0000 7ffa4c200640 1 -- 172.21.15.102:0/2391816539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa4415b560 msgr2=0x7ffa4415b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:43.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.761+0000 7ffa4c200640 1 --2- 172.21.15.102:0/2391816539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa4415b560 0x7ffa4415b940 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ffa440ce180 tx=0x7ffa3802b900 comp rx=0 tx=0).stop 2024-03-23T03:49:43.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.761+0000 7ffa4c200640 1 -- 172.21.15.102:0/2391816539 shutdown_connections 2024-03-23T03:49:43.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.761+0000 7ffa4c200640 1 --2- 172.21.15.102:0/2391816539 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa4415e500 0x7ffa4415e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:43.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.761+0000 7ffa4c200640 1 --2- 172.21.15.102:0/2391816539 >> v2:172.21.15.173:3300/0 conn(0x7ffa440b39b0 0x7ffa440b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:43.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.761+0000 7ffa4c200640 1 --2- 172.21.15.102:0/2391816539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa4415b560 0x7ffa4415b940 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:43.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.761+0000 7ffa4c200640 1 -- 172.21.15.102:0/2391816539 >> 172.21.15.102:0/2391816539 conn(0x7ffa4406a3d0 msgr2=0x7ffa440b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:43.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.761+0000 7ffa4c200640 1 -- 172.21.15.102:0/2391816539 shutdown_connections 2024-03-23T03:49:43.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.761+0000 7ffa4c200640 1 -- 172.21.15.102:0/2391816539 wait complete. 2024-03-23T03:49:43.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.763+0000 7ffa4c200640 1 Processor -- start 2024-03-23T03:49:43.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.763+0000 7ffa4c200640 1 -- start start 2024-03-23T03:49:43.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.764+0000 7ffa4c200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa440b39b0 0x7ffa441548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:43.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.764+0000 7ffa4c200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa4415b560 0x7ffa441512d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:43.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.764+0000 7ffa4c200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa4415e500 0x7ffa441580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:43.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.764+0000 7ffa4c200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa44156e10 con 0x7ffa4415b560 2024-03-23T03:49:43.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.764+0000 7ffa4c200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa44154de0 con 0x7ffa440b39b0 2024-03-23T03:49:43.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.764+0000 7ffa4c200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa44157020 con 0x7ffa4415e500 2024-03-23T03:49:43.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.764+0000 7ffa48e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa4415b560 0x7ffa441512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:43.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.764+0000 7ffa48e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa4415b560 0x7ffa441512d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54278/0 (socket says 172.21.15.102:54278) 2024-03-23T03:49:43.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.764+0000 7ffa48e00640 1 -- 172.21.15.102:0/2156137475 learned_addr learned my addr 172.21.15.102:0/2156137475 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:43.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.764+0000 7ffa49800640 1 --2- 172.21.15.102:0/2156137475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa440b39b0 0x7ffa441548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:43.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.764+0000 7ffa4a200640 1 --2- 172.21.15.102:0/2156137475 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa4415e500 0x7ffa441580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:43.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.764+0000 7ffa48e00640 1 -- 172.21.15.102:0/2156137475 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa4415e500 msgr2=0x7ffa441580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:43.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.764+0000 7ffa48e00640 1 --2- 172.21.15.102:0/2156137475 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa4415e500 0x7ffa441580c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:43.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.765+0000 7ffa48e00640 1 -- 172.21.15.102:0/2156137475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa440b39b0 msgr2=0x7ffa441548a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:43.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.765+0000 7ffa48e00640 1 --2- 172.21.15.102:0/2156137475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa440b39b0 0x7ffa441548a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:43.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.765+0000 7ffa48e00640 1 -- 172.21.15.102:0/2156137475 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa380094d0 con 0x7ffa4415b560 2024-03-23T03:49:43.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.765+0000 7ffa49800640 1 --2- 172.21.15.102:0/2156137475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa440b39b0 0x7ffa441548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:43.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.765+0000 7ffa48e00640 1 --2- 172.21.15.102:0/2156137475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa4415b560 0x7ffa441512d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400c7f0 tx=0x7ffa3400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:43.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.765+0000 7ffa41600640 1 -- 172.21.15.102:0/2156137475 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa34007820 con 0x7ffa4415b560 2024-03-23T03:49:43.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.765+0000 7ffa41600640 1 -- 172.21.15.102:0/2156137475 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa34007980 con 0x7ffa4415b560 2024-03-23T03:49:43.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.765+0000 7ffa41600640 1 -- 172.21.15.102:0/2156137475 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa34007d30 con 0x7ffa4415b560 2024-03-23T03:49:43.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.765+0000 7ffa4c200640 1 -- 172.21.15.102:0/2156137475 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa44157300 con 0x7ffa4415b560 2024-03-23T03:49:43.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.765+0000 7ffa4c200640 1 -- 172.21.15.102:0/2156137475 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa441578c0 con 0x7ffa4415b560 2024-03-23T03:49:43.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.766+0000 7ffa4c200640 1 -- 172.21.15.102:0/2156137475 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa4406ac20 con 0x7ffa4415b560 2024-03-23T03:49:43.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.767+0000 7ffa41600640 1 -- 172.21.15.102:0/2156137475 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ffa340113f0 con 0x7ffa4415b560 2024-03-23T03:49:43.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.769+0000 7ffa41600640 1 --2- 172.21.15.102:0/2156137475 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa2806ac00 0x7ffa2806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:43.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.769+0000 7ffa41600640 1 -- 172.21.15.102:0/2156137475 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6174+0+0 (secure 0 0 0) 0x7ffa340898e0 con 0x7ffa4415b560 2024-03-23T03:49:43.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.769+0000 7ffa49800640 1 --2- 172.21.15.102:0/2156137475 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa2806ac00 0x7ffa2806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:43.770 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.770+0000 7ffa49800640 1 --2- 172.21.15.102:0/2156137475 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa2806ac00 0x7ffa2806d0c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ffa38036000 tx=0x7ffa3803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:43.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:43.773+0000 7ffa41600640 1 -- 172.21.15.102:0/2156137475 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffa340547d0 con 0x7ffa4415b560 2024-03-23T03:49:44.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:44.018+0000 7ffa4c200640 1 -- 172.21.15.102:0/2156137475 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ffa44073390 con 0x7ffa4415b560 2024-03-23T03:49:44.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:44.019+0000 7ffa41600640 1 -- 172.21.15.102:0/2156137475 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v33) v1 ==== 53+0+73 (secure 0 0 0) 0x7ffa34057df0 con 0x7ffa4415b560 2024-03-23T03:49:44.020 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T03:49:44.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:44.022+0000 7ffa4c200640 1 -- 172.21.15.102:0/2156137475 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa2806ac00 msgr2=0x7ffa2806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:44.022+0000 7ffa4c200640 1 --2- 172.21.15.102:0/2156137475 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa2806ac00 0x7ffa2806d0c0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ffa38036000 tx=0x7ffa3803c040 comp rx=0 tx=0).stop 2024-03-23T03:49:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:44.022+0000 7ffa4c200640 1 -- 172.21.15.102:0/2156137475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa4415b560 msgr2=0x7ffa441512d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:44.022+0000 7ffa4c200640 1 --2- 172.21.15.102:0/2156137475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa4415b560 0x7ffa441512d0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400c7f0 tx=0x7ffa3400ccb0 comp rx=0 tx=0).stop 2024-03-23T03:49:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:44.022+0000 7ffa4c200640 1 -- 172.21.15.102:0/2156137475 shutdown_connections 2024-03-23T03:49:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:44.022+0000 7ffa4c200640 1 --2- 172.21.15.102:0/2156137475 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa2806ac00 0x7ffa2806d0c0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:44.022+0000 7ffa4c200640 1 --2- 172.21.15.102:0/2156137475 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa4415e500 0x7ffa441580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:44.022+0000 7ffa4c200640 1 --2- 172.21.15.102:0/2156137475 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa4415b560 0x7ffa441512d0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:44.022+0000 7ffa4c200640 1 --2- 172.21.15.102:0/2156137475 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa440b39b0 0x7ffa441548a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:44.022+0000 7ffa4c200640 1 -- 172.21.15.102:0/2156137475 >> 172.21.15.102:0/2156137475 conn(0x7ffa4406a3d0 msgr2=0x7ffa440ae310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:44.022+0000 7ffa4c200640 1 -- 172.21.15.102:0/2156137475 shutdown_connections 2024-03-23T03:49:44.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:44.022+0000 7ffa4c200640 1 -- 172.21.15.102:0/2156137475 wait complete. 2024-03-23T03:49:44.041 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-23T03:49:44.041 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:49:44.041 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi173.front.sepia.ceph.com 2024-03-23T03:49:44.042 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-23T03:49:44.042 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-23T03:49:44.042 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-23T03:49:44.042 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-23T03:49:44.042 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:49:44.042 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:49:44.042 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:44.042 DEBUG:teuthology.orchestra.run.smithi173:> ip addr 2024-03-23T03:49:44.096 INFO:teuthology.orchestra.run.smithi173.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:49:44.096 INFO:teuthology.orchestra.run.smithi173.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:49:44.096 INFO:teuthology.orchestra.run.smithi173.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:49:44.096 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:44.096 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 ::1/128 scope host 2024-03-23T03:49:44.096 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:44.096 INFO:teuthology.orchestra.run.smithi173.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:49:44.096 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6a brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:44.096 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f0 2024-03-23T03:49:44.096 INFO:teuthology.orchestra.run.smithi173.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:49:44.096 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6b brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:44.096 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f1 2024-03-23T03:49:44.096 INFO:teuthology.orchestra.run.smithi173.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:49:44.096 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:58 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:44.097 INFO:teuthology.orchestra.run.smithi173.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:49:44.097 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:59 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:44.097 INFO:teuthology.orchestra.run.smithi173.stdout: inet 172.21.15.173/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:49:44.097 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft 42457sec preferred_lft 42457sec 2024-03-23T03:49:44.097 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::ec4:7aff:fe88:7259/64 scope link 2024-03-23T03:49:44.097 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:44.097 INFO:teuthology.orchestra.run.smithi173.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:49:44.097 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 52:69:f2:bf:6d:02 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:49:44.097 INFO:teuthology.orchestra.run.smithi173.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T03:49:44.097 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:44.097 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::6894:e9ff:fe5d:1492/64 scope link 2024-03-23T03:49:44.097 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:44.097 INFO:teuthology.orchestra.run.smithi173.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:49:44.097 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether a2:27:ac:42:68:ce brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T03:49:44.097 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::a027:acff:fe42:68ce/64 scope link 2024-03-23T03:49:44.097 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:44.098 INFO:teuthology.orchestra.run.smithi173.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:49:44.098 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 52:69:f2:bf:6d:02 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-23T03:49:44.098 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::5069:f2ff:febf:6d02/64 scope link 2024-03-23T03:49:44.098 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:49:44.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:44.099 DEBUG:teuthology.orchestra.run.smithi173:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:49:44.178 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-03-23T03:49:44.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:44.182 DEBUG:teuthology.orchestra.run.smithi173:> route 2024-03-23T03:49:44.235 INFO:teuthology.orchestra.run.smithi173.stdout:Kernel IP routing table 2024-03-23T03:49:44.235 INFO:teuthology.orchestra.run.smithi173.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:49:44.235 INFO:teuthology.orchestra.run.smithi173.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:49:44.235 INFO:teuthology.orchestra.run.smithi173.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:49:44.235 INFO:teuthology.orchestra.run.smithi173.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:49:44.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:44.236 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-03-23T03:49:44.236 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-03-23T03:49:44.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:49:44.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:49:44.236 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:49:44.237 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-03-23T03:49:44.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:44.419 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list 2024-03-23T03:49:44.468 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-23T03:49:44.468 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-23T03:49:44.469 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:49:44.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:44.469 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link set brx.0 up 2024-03-23T03:49:44.563 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:49:44.563 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-23T03:49:44.619 INFO:teuthology.orchestra.run.smithi173.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:49:44.620 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:49:44.620 DEBUG:teuthology.orchestra.run.smithi173:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-23T03:49:44.673 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-23T03:49:44.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:44.673 DEBUG:teuthology.orchestra.run.smithi173:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:49:44.807 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:49:44.807 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:49:44.807 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:49:44.828 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:49:44.828 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:49:44.828 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:49:44.828 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:49:44.828 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:49:44.854 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:49:44.854 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:49:44.854 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:49:44.854 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:49:44.854 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:49:44.854 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790" 2024-03-23T03:49:44.854 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/ 2024-03-23T03:49:44.854 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=776b2820-05f2-4e50-8d40-18dd76c5f6a0" will pass to kernel 2024-03-23T03:49:44.856 INFO:teuthology.orchestra.run.smithi173.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-23T03:49:44.857 INFO:teuthology.orchestra.run.smithi173.stdout: You just mounted a file system that supports labels which does not 2024-03-23T03:49:44.857 INFO:teuthology.orchestra.run.smithi173.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T03:49:44.857 INFO:teuthology.orchestra.run.smithi173.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T03:49:44.857 INFO:teuthology.orchestra.run.smithi173.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T03:49:44.860 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T03:49:44.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:44.861 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-23T03:49:44.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:44.944 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:49:44.995 INFO:teuthology.orchestra.run.smithi173.stdout:client4991 2024-03-23T03:49:44.996 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:49:44.996 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:45.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:45.045 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-03-23T03:49:45.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:45.114 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-03-23T03:49:45.168 INFO:teuthology.orchestra.run.smithi102.stderr:./subdir: ceph.quota.max_bytes: No such attribute 2024-03-23T03:49:45.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:49:45.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:45.169 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 10K ./subdir) 2024-03-23T03:49:45.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:45.224 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-03-23T03:49:45.275 INFO:teuthology.orchestra.run.smithi102.stdout:10240 2024-03-23T03:49:45.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:45.276 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 100Ki ./subdir) 2024-03-23T03:49:45.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:45.332 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-03-23T03:49:45.387 INFO:teuthology.orchestra.run.smithi102.stdout:102400 2024-03-23T03:49:45.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:45.387 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 10M ./subdir) 2024-03-23T03:49:45.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:45.446 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-03-23T03:49:45.497 INFO:teuthology.orchestra.run.smithi102.stdout:10485760 2024-03-23T03:49:45.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:45.498 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 100Mi ./subdir) 2024-03-23T03:49:45.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:45.554 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-03-23T03:49:45.607 INFO:teuthology.orchestra.run.smithi102.stdout:104857600 2024-03-23T03:49:45.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:45.608 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 2G ./subdir) 2024-03-23T03:49:45.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:45.668 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-03-23T03:49:45.722 INFO:teuthology.orchestra.run.smithi102.stdout:2147483648 2024-03-23T03:49:45.723 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:45.723 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 4Gi ./subdir) 2024-03-23T03:49:45.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:45.784 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-03-23T03:49:45.840 INFO:teuthology.orchestra.run.smithi102.stdout:4294967296 2024-03-23T03:49:45.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:45.841 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 1T ./subdir) 2024-03-23T03:49:45.898 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:45.899 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-03-23T03:49:45.955 INFO:teuthology.orchestra.run.smithi102.stdout:1099511627776 2024-03-23T03:49:45.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:45.955 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 2Ti ./subdir) 2024-03-23T03:49:46.013 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:46.013 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_bytes ./subdir) 2024-03-23T03:49:46.071 INFO:teuthology.orchestra.run.smithi102.stdout:2199023255552 2024-03-23T03:49:46.071 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T03:49:46.072 DEBUG:teuthology.orchestra.run.smithi102:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:49:46.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:49:46.319 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T03:49:46.320 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:49:46.320 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:46.377 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:49:46.377 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:46.432 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:49:46.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.643+0000 7fa0a6000640 1 Processor -- start 2024-03-23T03:49:46.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.643+0000 7fa0a6000640 1 -- start start 2024-03-23T03:49:46.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.645+0000 7fa0a6000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa0a00b39b0 0x7fa0a00b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:46.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.645+0000 7fa0a6000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa0a015c400 0x7fa0a015c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:46.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.645+0000 7fa0a6000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a006c6f0 con 0x7fa0a015b560 2024-03-23T03:49:46.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.645+0000 7fa0a6000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a00b4370 con 0x7fa0a015c400 2024-03-23T03:49:46.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.645+0000 7fa0a6000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a006c9d0 con 0x7fa0a00b39b0 2024-03-23T03:49:46.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.645+0000 7fa09f400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa0a015b560 0x7fa0a015b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42258/0 (socket says 172.21.15.102:42258) 2024-03-23T03:49:46.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.645+0000 7fa09f400640 1 -- 172.21.15.102:0/1171001496 learned_addr learned my addr 172.21.15.102:0/1171001496 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:46.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.645+0000 7fa09fe00640 1 --2- 172.21.15.102:0/1171001496 >> v2:172.21.15.102:3300/0 conn(0x7fa0a015c400 0x7fa0a015c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:46.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.645+0000 7fa09ea00640 1 --2- 172.21.15.102:0/1171001496 >> v2:172.21.15.173:3300/0 conn(0x7fa0a00b39b0 0x7fa0a00b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:46.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.646+0000 7fa09fe00640 1 -- 172.21.15.102:0/1171001496 >> v2:172.21.15.173:3300/0 conn(0x7fa0a00b39b0 msgr2=0x7fa0a00b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:46.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.646+0000 7fa09fe00640 1 --2- 172.21.15.102:0/1171001496 >> v2:172.21.15.173:3300/0 conn(0x7fa0a00b39b0 0x7fa0a00b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:46.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.646+0000 7fa09fe00640 1 -- 172.21.15.102:0/1171001496 >> v1:172.21.15.102:6789/0 conn(0x7fa0a015b560 legacy=0x7fa0a015b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:46.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.646+0000 7fa09e000640 1 -- 172.21.15.102:0/1171001496 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056173415 0 0) 0x7fa0a006c6f0 con 0x7fa0a015b560 2024-03-23T03:49:46.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.646+0000 7fa09fe00640 1 -- 172.21.15.102:0/1171001496 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a006ccb0 con 0x7fa0a015c400 2024-03-23T03:49:46.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.647+0000 7fa09fe00640 1 --2- 172.21.15.102:0/1171001496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a015c400 0x7fa0a015c830 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fa0a00ce180 tx=0x7fa09802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7455b9ff7f32c06 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:46.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.647+0000 7fa09e000640 1 -- 172.21.15.102:0/1171001496 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09802cc80 con 0x7fa0a015c400 2024-03-23T03:49:46.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.647+0000 7fa09e000640 1 -- 172.21.15.102:0/1171001496 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09802cde0 con 0x7fa0a015c400 2024-03-23T03:49:46.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.647+0000 7fa09e000640 1 -- 172.21.15.102:0/1171001496 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098034660 con 0x7fa0a015c400 2024-03-23T03:49:46.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.648+0000 7fa0a6000640 1 -- 172.21.15.102:0/1171001496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a015c400 msgr2=0x7fa0a015c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:46.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.648+0000 7fa0a6000640 1 --2- 172.21.15.102:0/1171001496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a015c400 0x7fa0a015c830 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fa0a00ce180 tx=0x7fa09802bed0 comp rx=0 tx=0).stop 2024-03-23T03:49:46.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.648+0000 7fa0a6000640 1 -- 172.21.15.102:0/1171001496 shutdown_connections 2024-03-23T03:49:46.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.648+0000 7fa0a6000640 1 --2- 172.21.15.102:0/1171001496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a015c400 0x7fa0a015c830 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:46.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.648+0000 7fa0a6000640 1 --2- 172.21.15.102:0/1171001496 >> v2:172.21.15.173:3300/0 conn(0x7fa0a00b39b0 0x7fa0a00b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:46.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.648+0000 7fa0a6000640 1 -- 172.21.15.102:0/1171001496 >> 172.21.15.102:0/1171001496 conn(0x7fa0a006a3d0 msgr2=0x7fa0a00b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:46.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.648+0000 7fa0a6000640 1 -- 172.21.15.102:0/1171001496 shutdown_connections 2024-03-23T03:49:46.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.648+0000 7fa0a6000640 1 -- 172.21.15.102:0/1171001496 wait complete. 2024-03-23T03:49:46.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.650+0000 7fa0a6000640 1 Processor -- start 2024-03-23T03:49:46.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.650+0000 7fa0a6000640 1 -- start start 2024-03-23T03:49:46.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.651+0000 7fa0a6000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a00b39b0 0x7fa0a00abd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:46.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.651+0000 7fa0a6000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa0a015b560 0x7fa0a00cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:46.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.651+0000 7fa0a6000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0a015c400 0x7fa0a00cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:46.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.651+0000 7fa0a6000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a01f20b0 con 0x7fa0a00b39b0 2024-03-23T03:49:46.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.651+0000 7fa0a6000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a00ac5c0 con 0x7fa0a015c400 2024-03-23T03:49:46.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.651+0000 7fa0a6000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a00ac8a0 con 0x7fa0a015b560 2024-03-23T03:49:46.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.651+0000 7fa09f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a00b39b0 0x7fa0a00abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:46.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.651+0000 7fa09f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a00b39b0 0x7fa0a00abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54304/0 (socket says 172.21.15.102:54304) 2024-03-23T03:49:46.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.651+0000 7fa09f400640 1 -- 172.21.15.102:0/4075750680 learned_addr learned my addr 172.21.15.102:0/4075750680 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.651+0000 7fa09fe00640 1 --2- 172.21.15.102:0/4075750680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0a015c400 0x7fa0a00cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.651+0000 7fa09f400640 1 -- 172.21.15.102:0/4075750680 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa0a015b560 msgr2=0x7fa0a00cb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.651+0000 7fa09f400640 1 --2- 172.21.15.102:0/4075750680 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa0a015b560 0x7fa0a00cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.651+0000 7fa09f400640 1 -- 172.21.15.102:0/4075750680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0a015c400 msgr2=0x7fa0a00cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.651+0000 7fa09f400640 1 --2- 172.21.15.102:0/4075750680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0a015c400 0x7fa0a00cbd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.651+0000 7fa09f400640 1 -- 172.21.15.102:0/4075750680 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa098009a70 con 0x7fa0a00b39b0 2024-03-23T03:49:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.652+0000 7fa09f400640 1 --2- 172.21.15.102:0/4075750680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a00b39b0 0x7fa0a00abd00 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa09000c4d0 tx=0x7fa09000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.652+0000 7fa097400640 1 -- 172.21.15.102:0/4075750680 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0900074e0 con 0x7fa0a00b39b0 2024-03-23T03:49:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.652+0000 7fa097400640 1 -- 172.21.15.102:0/4075750680 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa090007660 con 0x7fa0a00b39b0 2024-03-23T03:49:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.652+0000 7fa097400640 1 -- 172.21.15.102:0/4075750680 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090004a20 con 0x7fa0a00b39b0 2024-03-23T03:49:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.652+0000 7fa0a6000640 1 -- 172.21.15.102:0/4075750680 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a00acb80 con 0x7fa0a00b39b0 2024-03-23T03:49:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.652+0000 7fa0a6000640 1 -- 172.21.15.102:0/4075750680 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a00ac240 con 0x7fa0a00b39b0 2024-03-23T03:49:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.653+0000 7fa0a6000640 1 -- 172.21.15.102:0/4075750680 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a006ac20 con 0x7fa0a00b39b0 2024-03-23T03:49:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.654+0000 7fa097400640 1 -- 172.21.15.102:0/4075750680 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa090004b80 con 0x7fa0a00b39b0 2024-03-23T03:49:46.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.655+0000 7fa097400640 1 --2- 172.21.15.102:0/4075750680 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa08006ac00 0x7fa08006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:46.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.655+0000 7fa097400640 1 -- 172.21.15.102:0/4075750680 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7fa0900886f0 con 0x7fa0a00b39b0 2024-03-23T03:49:46.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.655+0000 7fa09ea00640 1 --2- 172.21.15.102:0/4075750680 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa08006ac00 0x7fa08006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:46.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.656+0000 7fa09ea00640 1 --2- 172.21.15.102:0/4075750680 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa08006ac00 0x7fa08006d0c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa0a00cc3d0 tx=0x7fa08c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:46.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.660+0000 7fa097400640 1 -- 172.21.15.102:0/4075750680 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa0900536e0 con 0x7fa0a00b39b0 2024-03-23T03:49:46.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.856+0000 7fa0a6000640 1 -- 172.21.15.102:0/4075750680 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0a0073390 con 0x7fa0a00b39b0 2024-03-23T03:49:46.860 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.860+0000 7fa097400640 1 -- 172.21.15.102:0/4075750680 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16536 (secure 0 0 0) 0x7fa090056d00 con 0x7fa0a00b39b0 2024-03-23T03:49:46.861 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:46.861 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":69,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:49:43.895428+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:49:33.007757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-23T03:49:34.449411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.e","mappings":[{"from":7,"to":2}]},{"pgid":"10.23","mappings":[{"from":7,"to":3}]},{"pgid":"10.2e","mappings":[{"from":4,"to":3}]},{"pgid":"10.2f","mappings":[{"from":4,"to":5}]},{"pgid":"10.3c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:49:46.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.863+0000 7fa0a6000640 1 -- 172.21.15.102:0/4075750680 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa08006ac00 msgr2=0x7fa08006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:46.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.863+0000 7fa0a6000640 1 --2- 172.21.15.102:0/4075750680 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa08006ac00 0x7fa08006d0c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa0a00cc3d0 tx=0x7fa08c008040 comp rx=0 tx=0).stop 2024-03-23T03:49:46.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.863+0000 7fa0a6000640 1 -- 172.21.15.102:0/4075750680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a00b39b0 msgr2=0x7fa0a00abd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:46.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.863+0000 7fa0a6000640 1 --2- 172.21.15.102:0/4075750680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a00b39b0 0x7fa0a00abd00 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fa09000c4d0 tx=0x7fa09000c990 comp rx=0 tx=0).stop 2024-03-23T03:49:46.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.863+0000 7fa0a6000640 1 -- 172.21.15.102:0/4075750680 shutdown_connections 2024-03-23T03:49:46.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.863+0000 7fa0a6000640 1 --2- 172.21.15.102:0/4075750680 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa08006ac00 0x7fa08006d0c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:46.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.863+0000 7fa0a6000640 1 --2- 172.21.15.102:0/4075750680 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa0a015c400 0x7fa0a00cbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:46.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.863+0000 7fa0a6000640 1 --2- 172.21.15.102:0/4075750680 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa0a015b560 0x7fa0a00cb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:46.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.863+0000 7fa0a6000640 1 --2- 172.21.15.102:0/4075750680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0a00b39b0 0x7fa0a00abd00 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:46.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.863+0000 7fa0a6000640 1 -- 172.21.15.102:0/4075750680 >> 172.21.15.102:0/4075750680 conn(0x7fa0a006a3d0 msgr2=0x7fa0a01591a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:46.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.863+0000 7fa0a6000640 1 -- 172.21.15.102:0/4075750680 shutdown_connections 2024-03-23T03:49:46.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:46.863+0000 7fa0a6000640 1 -- 172.21.15.102:0/4075750680 wait complete. 2024-03-23T03:49:46.881 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:49:33.007757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:49:34.449411+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:49:46.882 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:49:47.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.088+0000 7f1946000640 1 Processor -- start 2024-03-23T03:49:47.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.089+0000 7f1946000640 1 -- start start 2024-03-23T03:49:47.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.090+0000 7f1946000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f19400cd740 0x7f19400b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.090+0000 7f1946000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19400d20c0 con 0x7f19400d0d40 2024-03-23T03:49:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.090+0000 7f1946000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19400d14b0 con 0x7f19400b5aa0 2024-03-23T03:49:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.090+0000 7f1946000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19400d1790 con 0x7f19400cd740 2024-03-23T03:49:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.090+0000 7f193ea00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f19400d0d40 0x7f19400cdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42260/0 (socket says 172.21.15.102:42260) 2024-03-23T03:49:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.090+0000 7f193ea00640 1 -- 172.21.15.102:0/2138873675 learned_addr learned my addr 172.21.15.102:0/2138873675 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.091+0000 7f193f400640 1 --2- 172.21.15.102:0/2138873675 >> v2:172.21.15.173:3300/0 conn(0x7f19400cd740 0x7f19400b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.091+0000 7f193e000640 1 -- 172.21.15.102:0/2138873675 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113563775 0 0) 0x7f19400d20c0 con 0x7f19400d0d40 2024-03-23T03:49:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.091+0000 7f193e000640 1 -- 172.21.15.102:0/2138873675 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1924003520 con 0x7f19400d0d40 2024-03-23T03:49:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.091+0000 7f193e000640 1 -- 172.21.15.102:0/2138873675 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2414497849 0 0) 0x7f1924003520 con 0x7f19400d0d40 2024-03-23T03:49:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.091+0000 7f193e000640 1 -- 172.21.15.102:0/2138873675 >> v2:172.21.15.173:3300/0 conn(0x7f19400cd740 msgr2=0x7f19400b84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.091+0000 7f193e000640 1 --2- 172.21.15.102:0/2138873675 >> v2:172.21.15.173:3300/0 conn(0x7f19400cd740 0x7f19400b84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.091+0000 7f193e000640 1 -- 172.21.15.102:0/2138873675 >> v1:172.21.15.173:6789/0 conn(0x7f19400b5aa0 legacy=0x7f19400b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.091+0000 7f193e000640 1 -- 172.21.15.102:0/2138873675 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19400d1a70 con 0x7f19400d0d40 2024-03-23T03:49:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.091+0000 7f193e000640 1 -- 172.21.15.102:0/2138873675 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f19300019f0 con 0x7f19400d0d40 2024-03-23T03:49:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.092+0000 7f193e000640 1 -- 172.21.15.102:0/2138873675 >> v1:172.21.15.102:6789/0 conn(0x7f19400d0d40 legacy=0x7f19400cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.092+0000 7f193e000640 1 --2- 172.21.15.102:0/2138873675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1924003970 0x7f1924003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.092+0000 7f193e000640 1 --2- 172.21.15.102:0/2138873675 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f19240047c0 0x7f1924004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.092+0000 7f193e000640 1 --2- 172.21.15.102:0/2138873675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1924005180 0x7f192400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.092+0000 7f193e000640 1 -- 172.21.15.102:0/2138873675 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19400d1790 con 0x7f1924005180 2024-03-23T03:49:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.092+0000 7f193e000640 1 -- 172.21.15.102:0/2138873675 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f192400c020 con 0x7f1924003970 2024-03-23T03:49:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.092+0000 7f193e000640 1 -- 172.21.15.102:0/2138873675 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f192400c1e0 con 0x7f19240047c0 2024-03-23T03:49:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.092+0000 7f193fe00640 1 --2- 172.21.15.102:0/2138873675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1924005180 0x7f192400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.092+0000 7f193f400640 1 --2- 172.21.15.102:0/2138873675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1924003970 0x7f1924003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.092+0000 7f193ea00640 1 --2- 172.21.15.102:0/2138873675 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f19240047c0 0x7f1924004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.092+0000 7f193fe00640 1 -- 172.21.15.102:0/2138873675 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f19240047c0 msgr2=0x7f1924004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.092+0000 7f193fe00640 1 --2- 172.21.15.102:0/2138873675 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f19240047c0 0x7f1924004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.092+0000 7f193fe00640 1 -- 172.21.15.102:0/2138873675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1924003970 msgr2=0x7f1924003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.092+0000 7f193fe00640 1 --2- 172.21.15.102:0/2138873675 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1924003970 0x7f1924003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.092+0000 7f193fe00640 1 -- 172.21.15.102:0/2138873675 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1924003520 con 0x7f1924005180 2024-03-23T03:49:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.093+0000 7f193fe00640 1 -- 172.21.15.102:0/2138873675 reap_dead start 2024-03-23T03:49:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.094+0000 7f193fe00640 1 --2- 172.21.15.102:0/2138873675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1924005180 0x7f192400b980 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f1938001750 tx=0x7f19240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c678b5a326804d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.094+0000 7f193e000640 1 -- 172.21.15.102:0/2138873675 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19380052d0 con 0x7f1924005180 2024-03-23T03:49:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.094+0000 7f193e000640 1 -- 172.21.15.102:0/2138873675 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193802e050 con 0x7f1924005180 2024-03-23T03:49:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.094+0000 7f193e000640 1 -- 172.21.15.102:0/2138873675 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1938003830 con 0x7f1924005180 2024-03-23T03:49:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.094+0000 7f1946000640 1 -- 172.21.15.102:0/2138873675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1924005180 msgr2=0x7f192400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.094+0000 7f1946000640 1 --2- 172.21.15.102:0/2138873675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1924005180 0x7f192400b980 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f1938001750 tx=0x7f19240057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.094+0000 7f1946000640 1 -- 172.21.15.102:0/2138873675 shutdown_connections 2024-03-23T03:49:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.094+0000 7f1946000640 1 --2- 172.21.15.102:0/2138873675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1924005180 0x7f192400b980 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.094+0000 7f1946000640 1 -- 172.21.15.102:0/2138873675 >> 172.21.15.102:0/2138873675 conn(0x7f19400cf5d0 msgr2=0x7f19400cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:47.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.095+0000 7f1946000640 1 -- 172.21.15.102:0/2138873675 shutdown_connections 2024-03-23T03:49:47.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.095+0000 7f1946000640 1 -- 172.21.15.102:0/2138873675 wait complete. 2024-03-23T03:49:47.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.096+0000 7f1946000640 1 Processor -- start 2024-03-23T03:49:47.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.096+0000 7f1946000640 1 -- start start 2024-03-23T03:49:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.097+0000 7f1946000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1924005180 0x7f1940157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.097+0000 7f1946000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19400caa90 0x7f1940155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.097+0000 7f1946000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f194014ea10 0x7f194014ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.097+0000 7f1946000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19400cb1e0 con 0x7f194014ea10 2024-03-23T03:49:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.097+0000 7f1946000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19400cb4c0 con 0x7f19400caa90 2024-03-23T03:49:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.097+0000 7f1946000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19400cb7a0 con 0x7f1924005180 2024-03-23T03:49:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.098+0000 7f193fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f194014ea10 0x7f194014ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.098+0000 7f193fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f194014ea10 0x7f194014ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54324/0 (socket says 172.21.15.102:54324) 2024-03-23T03:49:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.098+0000 7f193fe00640 1 -- 172.21.15.102:0/3598520093 learned_addr learned my addr 172.21.15.102:0/3598520093 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.098+0000 7f193f400640 1 --2- 172.21.15.102:0/3598520093 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1924005180 0x7f1940157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.098+0000 7f193ea00640 1 --2- 172.21.15.102:0/3598520093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19400caa90 0x7f1940155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.098+0000 7f193fe00640 1 -- 172.21.15.102:0/3598520093 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1924005180 msgr2=0x7f1940157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.098+0000 7f193fe00640 1 --2- 172.21.15.102:0/3598520093 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1924005180 0x7f1940157fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.098+0000 7f193fe00640 1 -- 172.21.15.102:0/3598520093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19400caa90 msgr2=0x7f1940155430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.098+0000 7f193fe00640 1 --2- 172.21.15.102:0/3598520093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19400caa90 0x7f1940155430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.098+0000 7f193fe00640 1 -- 172.21.15.102:0/3598520093 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19400cba80 con 0x7f194014ea10 2024-03-23T03:49:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.098+0000 7f193fe00640 1 --2- 172.21.15.102:0/3598520093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f194014ea10 0x7f194014ee90 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f1938001870 tx=0x7f1938001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.099+0000 7f1937400640 1 -- 172.21.15.102:0/3598520093 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1938039070 con 0x7f194014ea10 2024-03-23T03:49:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.099+0000 7f1937400640 1 -- 172.21.15.102:0/3598520093 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1938000ed0 con 0x7f194014ea10 2024-03-23T03:49:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.099+0000 7f1937400640 1 -- 172.21.15.102:0/3598520093 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1938007030 con 0x7f194014ea10 2024-03-23T03:49:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.099+0000 7f1946000640 1 -- 172.21.15.102:0/3598520093 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19400cbd60 con 0x7f194014ea10 2024-03-23T03:49:47.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.099+0000 7f1946000640 1 -- 172.21.15.102:0/3598520093 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f194014f3d0 con 0x7f194014ea10 2024-03-23T03:49:47.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.100+0000 7f1946000640 1 -- 172.21.15.102:0/3598520093 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19400cbf50 con 0x7f194014ea10 2024-03-23T03:49:47.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.101+0000 7f1937400640 1 -- 172.21.15.102:0/3598520093 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f193803c020 con 0x7f194014ea10 2024-03-23T03:49:47.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.102+0000 7f1937400640 1 --2- 172.21.15.102:0/3598520093 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f191c06ac00 0x7f191c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.102+0000 7f1937400640 1 -- 172.21.15.102:0/3598520093 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7f19380a98d0 con 0x7f194014ea10 2024-03-23T03:49:47.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.102+0000 7f193f400640 1 --2- 172.21.15.102:0/3598520093 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f191c06ac00 0x7f191c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:47.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.103+0000 7f193f400640 1 --2- 172.21.15.102:0/3598520093 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f191c06ac00 0x7f191c06d0c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f19280098d0 tx=0x7f1928008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:47.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.106+0000 7f1937400640 1 -- 172.21.15.102:0/3598520093 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1938074900 con 0x7f194014ea10 2024-03-23T03:49:47.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.297+0000 7f1946000640 1 -- 172.21.15.102:0/3598520093 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1940073390 con 0x7f194014ea10 2024-03-23T03:49:47.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.297+0000 7f1937400640 1 -- 172.21.15.102:0/3598520093 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1938077f20 con 0x7f194014ea10 2024-03-23T03:49:47.298 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:49:47.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.300+0000 7f1946000640 1 -- 172.21.15.102:0/3598520093 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f191c06ac00 msgr2=0x7f191c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:47.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.300+0000 7f1946000640 1 --2- 172.21.15.102:0/3598520093 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f191c06ac00 0x7f191c06d0c0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f19280098d0 tx=0x7f1928008040 comp rx=0 tx=0).stop 2024-03-23T03:49:47.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.300+0000 7f1946000640 1 -- 172.21.15.102:0/3598520093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f194014ea10 msgr2=0x7f194014ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:47.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.300+0000 7f1946000640 1 --2- 172.21.15.102:0/3598520093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f194014ea10 0x7f194014ee90 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f1938001870 tx=0x7f1938001120 comp rx=0 tx=0).stop 2024-03-23T03:49:47.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.300+0000 7f1946000640 1 -- 172.21.15.102:0/3598520093 shutdown_connections 2024-03-23T03:49:47.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.300+0000 7f1946000640 1 --2- 172.21.15.102:0/3598520093 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f191c06ac00 0x7f191c06d0c0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.300+0000 7f1946000640 1 --2- 172.21.15.102:0/3598520093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f194014ea10 0x7f194014ee90 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.300+0000 7f1946000640 1 --2- 172.21.15.102:0/3598520093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19400caa90 0x7f1940155430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.300+0000 7f1946000640 1 --2- 172.21.15.102:0/3598520093 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1924005180 0x7f1940157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.300+0000 7f1946000640 1 -- 172.21.15.102:0/3598520093 >> 172.21.15.102:0/3598520093 conn(0x7f19400cf5d0 msgr2=0x7f19400b7090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:47.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.300+0000 7f1946000640 1 -- 172.21.15.102:0/3598520093 shutdown_connections 2024-03-23T03:49:47.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.300+0000 7f1946000640 1 -- 172.21.15.102:0/3598520093 wait complete. 2024-03-23T03:49:47.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:49:47.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.524+0000 7f8673000640 1 Processor -- start 2024-03-23T03:49:47.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.525+0000 7f8673000640 1 -- start start 2024-03-23T03:49:47.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.526+0000 7f8673000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f866c0cd740 0x7f866c0b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.526+0000 7f8673000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f866c0d0d40 0x7f866c0cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.526+0000 7f8673000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866c0d20c0 con 0x7f866c06b0a0 2024-03-23T03:49:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.526+0000 7f8673000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f866c0ce060 con 0x7f866c0cd740 2024-03-23T03:49:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.526+0000 7f8673000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f866c0d14b0 con 0x7f866c0d0d40 2024-03-23T03:49:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.526+0000 7f866be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f866c0cd740 0x7f866c0b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.527+0000 7f866be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f866c0cd740 0x7f866c0b84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54340/0 (socket says 172.21.15.102:54340) 2024-03-23T03:49:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.527+0000 7f866be00640 1 -- 172.21.15.102:0/1690243605 learned_addr learned my addr 172.21.15.102:0/1690243605 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.527+0000 7f866b400640 1 --2- 172.21.15.102:0/1690243605 >> v2:172.21.15.173:3300/0 conn(0x7f866c0d0d40 0x7f866c0cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.527+0000 7f866be00640 1 -- 172.21.15.102:0/1690243605 >> v2:172.21.15.173:3300/0 conn(0x7f866c0d0d40 msgr2=0x7f866c0cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.527+0000 7f866be00640 1 --2- 172.21.15.102:0/1690243605 >> v2:172.21.15.173:3300/0 conn(0x7f866c0d0d40 0x7f866c0cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.527+0000 7f866be00640 1 -- 172.21.15.102:0/1690243605 >> v1:172.21.15.173:6789/0 conn(0x7f866c06b0a0 legacy=0x7f866c06b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:47.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.527+0000 7f866be00640 1 -- 172.21.15.102:0/1690243605 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866c0d1790 con 0x7f866c0cd740 2024-03-23T03:49:47.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.528+0000 7f866be00640 1 --2- 172.21.15.102:0/1690243605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f866c0cd740 0x7f866c0b84b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f866c0b8050 tx=0x7f865c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef0b985308504854 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:47.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.529+0000 7f866aa00640 1 -- 172.21.15.102:0/1690243605 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c031870 con 0x7f866c0cd740 2024-03-23T03:49:47.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.529+0000 7f866aa00640 1 -- 172.21.15.102:0/1690243605 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f865c0319d0 con 0x7f866c0cd740 2024-03-23T03:49:47.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.529+0000 7f866aa00640 1 -- 172.21.15.102:0/1690243605 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f865c031d60 con 0x7f866c0cd740 2024-03-23T03:49:47.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.529+0000 7f8673000640 1 -- 172.21.15.102:0/1690243605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f866c0cd740 msgr2=0x7f866c0b84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:47.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.529+0000 7f8673000640 1 --2- 172.21.15.102:0/1690243605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f866c0cd740 0x7f866c0b84b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f866c0b8050 tx=0x7f865c02b910 comp rx=0 tx=0).stop 2024-03-23T03:49:47.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.530+0000 7f8673000640 1 -- 172.21.15.102:0/1690243605 shutdown_connections 2024-03-23T03:49:47.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.530+0000 7f8673000640 1 --2- 172.21.15.102:0/1690243605 >> v2:172.21.15.173:3300/0 conn(0x7f866c0d0d40 0x7f866c0cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.530+0000 7f8673000640 1 --2- 172.21.15.102:0/1690243605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f866c0cd740 0x7f866c0b84b0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.530+0000 7f8673000640 1 -- 172.21.15.102:0/1690243605 >> 172.21.15.102:0/1690243605 conn(0x7f866c0cf5d0 msgr2=0x7f866c0cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:47.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.530+0000 7f8673000640 1 -- 172.21.15.102:0/1690243605 shutdown_connections 2024-03-23T03:49:47.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.530+0000 7f8673000640 1 -- 172.21.15.102:0/1690243605 wait complete. 2024-03-23T03:49:47.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.531+0000 7f8673000640 1 Processor -- start 2024-03-23T03:49:47.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.532+0000 7f8673000640 1 -- start start 2024-03-23T03:49:47.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.532+0000 7f8673000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f866c06b0a0 0x7f866c0cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.532+0000 7f8673000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f866c0cd740 0x7f866c14c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.532+0000 7f8673000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f866c0d0d40 0x7f866c14cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.532+0000 7f8673000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866c0ae700 con 0x7f866c0cd740 2024-03-23T03:49:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.532+0000 7f8673000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866c1f67b0 con 0x7f866c06b0a0 2024-03-23T03:49:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.532+0000 7f8673000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866c14d170 con 0x7f866c0d0d40 2024-03-23T03:49:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.533+0000 7f866b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f866c0cd740 0x7f866c14c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.533+0000 7f866b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f866c0cd740 0x7f866c14c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54346/0 (socket says 172.21.15.102:54346) 2024-03-23T03:49:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.533+0000 7f866b400640 1 -- 172.21.15.102:0/558485583 learned_addr learned my addr 172.21.15.102:0/558485583 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.533+0000 7f866b400640 1 -- 172.21.15.102:0/558485583 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f866c0d0d40 msgr2=0x7f866c14cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.533+0000 7f866b400640 1 --2- 172.21.15.102:0/558485583 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f866c0d0d40 0x7f866c14cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.533+0000 7f866b400640 1 -- 172.21.15.102:0/558485583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f866c06b0a0 msgr2=0x7f866c0cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.533+0000 7f866b400640 1 --2- 172.21.15.102:0/558485583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f866c06b0a0 0x7f866c0cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.533+0000 7f866b400640 1 -- 172.21.15.102:0/558485583 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f865c0094d0 con 0x7f866c0cd740 2024-03-23T03:49:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.533+0000 7f866b400640 1 --2- 172.21.15.102:0/558485583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f866c0cd740 0x7f866c14c6f0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f8654007ea0 tx=0x7f8654007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.534+0000 7f8663e00640 1 -- 172.21.15.102:0/558485583 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8654014620 con 0x7f866c0cd740 2024-03-23T03:49:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.534+0000 7f8663e00640 1 -- 172.21.15.102:0/558485583 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8654014780 con 0x7f866c0cd740 2024-03-23T03:49:47.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.534+0000 7f8673000640 1 -- 172.21.15.102:0/558485583 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866c1f64d0 con 0x7f866c0cd740 2024-03-23T03:49:47.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.534+0000 7f8663e00640 1 -- 172.21.15.102:0/558485583 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8654014b30 con 0x7f866c0cd740 2024-03-23T03:49:47.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.534+0000 7f8673000640 1 -- 172.21.15.102:0/558485583 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f866c14f950 con 0x7f866c0cd740 2024-03-23T03:49:47.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.535+0000 7f8673000640 1 -- 172.21.15.102:0/558485583 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f866c073390 con 0x7f866c0cd740 2024-03-23T03:49:47.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.536+0000 7f8663e00640 1 -- 172.21.15.102:0/558485583 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8654014c90 con 0x7f866c0cd740 2024-03-23T03:49:47.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.537+0000 7f8663e00640 1 --2- 172.21.15.102:0/558485583 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f864806ac00 0x7f864806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.537+0000 7f8663e00640 1 -- 172.21.15.102:0/558485583 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7f8654089f40 con 0x7f866c0cd740 2024-03-23T03:49:47.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.537+0000 7f866be00640 1 --2- 172.21.15.102:0/558485583 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f864806ac00 0x7f864806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:47.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.538+0000 7f866be00640 1 --2- 172.21.15.102:0/558485583 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f864806ac00 0x7f864806d0c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f865c036000 tx=0x7f865c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:47.542 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.542+0000 7f8663e00640 1 -- 172.21.15.102:0/558485583 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8654054eb0 con 0x7f866c0cd740 2024-03-23T03:49:47.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.734+0000 7f8673000640 1 -- 172.21.15.102:0/558485583 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f866c14f4d0 con 0x7f866c0cd740 2024-03-23T03:49:47.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.735+0000 7f8663e00640 1 -- 172.21.15.102:0/558485583 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7f866c14f4d0 con 0x7f866c0cd740 2024-03-23T03:49:47.736 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:49:47.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.737+0000 7f8673000640 1 -- 172.21.15.102:0/558485583 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f864806ac00 msgr2=0x7f864806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:47.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.737+0000 7f8673000640 1 --2- 172.21.15.102:0/558485583 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f864806ac00 0x7f864806d0c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f865c036000 tx=0x7f865c030000 comp rx=0 tx=0).stop 2024-03-23T03:49:47.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.738+0000 7f8673000640 1 -- 172.21.15.102:0/558485583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f866c0cd740 msgr2=0x7f866c14c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:47.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.738+0000 7f8673000640 1 --2- 172.21.15.102:0/558485583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f866c0cd740 0x7f866c14c6f0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f8654007ea0 tx=0x7f8654007f90 comp rx=0 tx=0).stop 2024-03-23T03:49:47.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.738+0000 7f8673000640 1 -- 172.21.15.102:0/558485583 shutdown_connections 2024-03-23T03:49:47.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.738+0000 7f8673000640 1 --2- 172.21.15.102:0/558485583 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f864806ac00 0x7f864806d0c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.738+0000 7f8673000640 1 --2- 172.21.15.102:0/558485583 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f866c0d0d40 0x7f866c14cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.738+0000 7f8673000640 1 --2- 172.21.15.102:0/558485583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f866c0cd740 0x7f866c14c6f0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.738+0000 7f8673000640 1 --2- 172.21.15.102:0/558485583 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f866c06b0a0 0x7f866c0cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.738+0000 7f8673000640 1 -- 172.21.15.102:0/558485583 >> 172.21.15.102:0/558485583 conn(0x7f866c0cf5d0 msgr2=0x7f866c0c8740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:47.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.738+0000 7f8673000640 1 -- 172.21.15.102:0/558485583 shutdown_connections 2024-03-23T03:49:47.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.738+0000 7f8673000640 1 -- 172.21.15.102:0/558485583 wait complete. 2024-03-23T03:49:47.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:49:47.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.972+0000 7f2cdb200640 1 Processor -- start 2024-03-23T03:49:47.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.973+0000 7f2cdb200640 1 -- start start 2024-03-23T03:49:47.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.974+0000 7f2cdb200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f2cdc1571d0 0x7f2cdc0b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.974+0000 7f2cdb200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2cdc0cdc00 0x7f2cdc15aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.974+0000 7f2cdb200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cdc1575b0 con 0x7f2cdc15afe0 2024-03-23T03:49:47.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.974+0000 7f2cdb200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cdc159f20 con 0x7f2cdc0cdc00 2024-03-23T03:49:47.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.974+0000 7f2cdb200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cdc15a200 con 0x7f2cdc1571d0 2024-03-23T03:49:47.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.975+0000 7f2cd1400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2cdc0cdc00 0x7f2cdc15aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:47.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.975+0000 7f2cd1400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2cdc0cdc00 0x7f2cdc15aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54354/0 (socket says 172.21.15.102:54354) 2024-03-23T03:49:47.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.975+0000 7f2cd1400640 1 -- 172.21.15.102:0/2210223695 learned_addr learned my addr 172.21.15.102:0/2210223695 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:47.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.975+0000 7f2cd9e00640 1 --2- 172.21.15.102:0/2210223695 >> v2:172.21.15.173:3300/0 conn(0x7f2cdc1571d0 0x7f2cdc0b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:47.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.975+0000 7f2cd1400640 1 -- 172.21.15.102:0/2210223695 >> v2:172.21.15.173:3300/0 conn(0x7f2cdc1571d0 msgr2=0x7f2cdc0b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:47.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.975+0000 7f2cd1400640 1 --2- 172.21.15.102:0/2210223695 >> v2:172.21.15.173:3300/0 conn(0x7f2cdc1571d0 0x7f2cdc0b35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.975+0000 7f2cd1400640 1 -- 172.21.15.102:0/2210223695 >> v1:172.21.15.173:6789/0 conn(0x7f2cdc15afe0 legacy=0x7f2cdc0cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:47.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.975+0000 7f2cd1400640 1 -- 172.21.15.102:0/2210223695 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc15a4e0 con 0x7f2cdc0cdc00 2024-03-23T03:49:47.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.976+0000 7f2cd1400640 1 --2- 172.21.15.102:0/2210223695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cdc0cdc00 0x7f2cdc15aaa0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc0712e0 tx=0x7f2cc402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bad4d6e4eb9627d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:47.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.977+0000 7f2cd9400640 1 -- 172.21.15.102:0/2210223695 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc402ee90 con 0x7f2cdc0cdc00 2024-03-23T03:49:47.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.977+0000 7f2cd9400640 1 -- 172.21.15.102:0/2210223695 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc4036440 con 0x7f2cdc0cdc00 2024-03-23T03:49:47.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.977+0000 7f2cd9400640 1 -- 172.21.15.102:0/2210223695 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc40367d0 con 0x7f2cdc0cdc00 2024-03-23T03:49:47.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.977+0000 7f2cdb200640 1 -- 172.21.15.102:0/2210223695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cdc0cdc00 msgr2=0x7f2cdc15aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:47.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.977+0000 7f2cdb200640 1 --2- 172.21.15.102:0/2210223695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cdc0cdc00 0x7f2cdc15aaa0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2cdc0712e0 tx=0x7f2cc402e0e0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.977+0000 7f2cdb200640 1 -- 172.21.15.102:0/2210223695 shutdown_connections 2024-03-23T03:49:47.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.977+0000 7f2cdb200640 1 --2- 172.21.15.102:0/2210223695 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cdc0cdc00 0x7f2cdc15aaa0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.977+0000 7f2cdb200640 1 --2- 172.21.15.102:0/2210223695 >> v2:172.21.15.173:3300/0 conn(0x7f2cdc1571d0 0x7f2cdc0b35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.977+0000 7f2cdb200640 1 -- 172.21.15.102:0/2210223695 >> 172.21.15.102:0/2210223695 conn(0x7f2cdc065df0 msgr2=0x7f2cdc069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:47.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.977+0000 7f2cdb200640 1 -- 172.21.15.102:0/2210223695 shutdown_connections 2024-03-23T03:49:47.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.977+0000 7f2cdb200640 1 -- 172.21.15.102:0/2210223695 wait complete. 2024-03-23T03:49:47.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.979+0000 7f2cdb200640 1 Processor -- start 2024-03-23T03:49:47.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.979+0000 7f2cdb200640 1 -- start start 2024-03-23T03:49:47.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.980+0000 7f2cdb200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cdc0cdc00 0x7f2cdc0ac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.980+0000 7f2cdb200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cdc1571d0 0x7f2cdc13be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.980+0000 7f2cdb200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2cdc15afe0 0x7f2cdc06cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.980+0000 7f2cdb200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc0ac940 con 0x7f2cdc1571d0 2024-03-23T03:49:47.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.980+0000 7f2cdb200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc0acc20 con 0x7f2cdc0cdc00 2024-03-23T03:49:47.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.980+0000 7f2cdb200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc0acf00 con 0x7f2cdc15afe0 2024-03-23T03:49:47.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.981+0000 7f2cd1400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cdc1571d0 0x7f2cdc13be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:47.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.981+0000 7f2cd1400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cdc1571d0 0x7f2cdc13be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54366/0 (socket says 172.21.15.102:54366) 2024-03-23T03:49:47.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.981+0000 7f2cd1400640 1 -- 172.21.15.102:0/3971753870 learned_addr learned my addr 172.21.15.102:0/3971753870 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:47.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.981+0000 7f2cd9e00640 1 --2- 172.21.15.102:0/3971753870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cdc0cdc00 0x7f2cdc0ac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:47.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.981+0000 7f2cd1400640 1 -- 172.21.15.102:0/3971753870 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2cdc15afe0 msgr2=0x7f2cdc06cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:47.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.981+0000 7f2cd1400640 1 --2- 172.21.15.102:0/3971753870 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2cdc15afe0 0x7f2cdc06cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.981+0000 7f2cd1400640 1 -- 172.21.15.102:0/3971753870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cdc0cdc00 msgr2=0x7f2cdc0ac0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:47.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.981+0000 7f2cd1400640 1 --2- 172.21.15.102:0/3971753870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cdc0cdc00 0x7f2cdc0ac0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:47.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.981+0000 7f2cd1400640 1 -- 172.21.15.102:0/3971753870 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc400b770 con 0x7f2cdc1571d0 2024-03-23T03:49:47.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.981+0000 7f2cd9e00640 1 --2- 172.21.15.102:0/3971753870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cdc0cdc00 0x7f2cdc0ac0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:47.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.981+0000 7f2cd1400640 1 --2- 172.21.15.102:0/3971753870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cdc1571d0 0x7f2cdc13be10 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2cc402e120 tx=0x7f2cc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:47.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.982+0000 7f2cd2a00640 1 -- 172.21.15.102:0/3971753870 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc40348d0 con 0x7f2cdc1571d0 2024-03-23T03:49:47.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.982+0000 7f2cd2a00640 1 -- 172.21.15.102:0/3971753870 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc4034a30 con 0x7f2cdc1571d0 2024-03-23T03:49:47.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.982+0000 7f2cdb200640 1 -- 172.21.15.102:0/3971753870 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cdc0ad1e0 con 0x7f2cdc1571d0 2024-03-23T03:49:47.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.982+0000 7f2cd2a00640 1 -- 172.21.15.102:0/3971753870 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc4034de0 con 0x7f2cdc1571d0 2024-03-23T03:49:47.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.982+0000 7f2cdb200640 1 -- 172.21.15.102:0/3971753870 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cdc0ae8b0 con 0x7f2cdc1571d0 2024-03-23T03:49:47.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.983+0000 7f2cdb200640 1 -- 172.21.15.102:0/3971753870 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cdc04e030 con 0x7f2cdc1571d0 2024-03-23T03:49:47.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.984+0000 7f2cd2a00640 1 -- 172.21.15.102:0/3971753870 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2cc40403e0 con 0x7f2cdc1571d0 2024-03-23T03:49:47.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.985+0000 7f2cd2a00640 1 --2- 172.21.15.102:0/3971753870 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2cb806ac00 0x7f2cb806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:47.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.985+0000 7f2cd2a00640 1 -- 172.21.15.102:0/3971753870 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7f2cc40ab7e0 con 0x7f2cdc1571d0 2024-03-23T03:49:47.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.986+0000 7f2cd9e00640 1 --2- 172.21.15.102:0/3971753870 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2cb806ac00 0x7f2cb806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:47.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.986+0000 7f2cd9e00640 1 --2- 172.21.15.102:0/3971753870 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2cb806ac00 0x7f2cb806d0c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc006fd0 tx=0x7f2ccc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:47.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:47.990+0000 7f2cd2a00640 1 -- 172.21.15.102:0/3971753870 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2cc40767f0 con 0x7f2cdc1571d0 2024-03-23T03:49:48.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.180+0000 7f2cdb200640 1 -- 172.21.15.102:0/3971753870 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2cdc15aa00 con 0x7f2cdc1571d0 2024-03-23T03:49:48.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.180+0000 7f2cd2a00640 1 -- 172.21.15.102:0/3971753870 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2cc403b0a0 con 0x7f2cdc1571d0 2024-03-23T03:49:48.181 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:49:48.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.183+0000 7f2cdb200640 1 -- 172.21.15.102:0/3971753870 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2cb806ac00 msgr2=0x7f2cb806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:48.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.183+0000 7f2cdb200640 1 --2- 172.21.15.102:0/3971753870 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2cb806ac00 0x7f2cb806d0c0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc006fd0 tx=0x7f2ccc008040 comp rx=0 tx=0).stop 2024-03-23T03:49:48.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.183+0000 7f2cdb200640 1 -- 172.21.15.102:0/3971753870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cdc1571d0 msgr2=0x7f2cdc13be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:48.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.183+0000 7f2cdb200640 1 --2- 172.21.15.102:0/3971753870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cdc1571d0 0x7f2cdc13be10 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f2cc402e120 tx=0x7f2cc40046c0 comp rx=0 tx=0).stop 2024-03-23T03:49:48.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.183+0000 7f2cdb200640 1 -- 172.21.15.102:0/3971753870 shutdown_connections 2024-03-23T03:49:48.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.183+0000 7f2cdb200640 1 --2- 172.21.15.102:0/3971753870 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2cb806ac00 0x7f2cb806d0c0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:48.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.183+0000 7f2cdb200640 1 --2- 172.21.15.102:0/3971753870 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2cdc15afe0 0x7f2cdc06cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:48.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.183+0000 7f2cdb200640 1 --2- 172.21.15.102:0/3971753870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cdc1571d0 0x7f2cdc13be10 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:48.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.183+0000 7f2cdb200640 1 --2- 172.21.15.102:0/3971753870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2cdc0cdc00 0x7f2cdc0ac0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:48.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.183+0000 7f2cdb200640 1 -- 172.21.15.102:0/3971753870 >> 172.21.15.102:0/3971753870 conn(0x7f2cdc065df0 msgr2=0x7f2cdc15bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:48.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.183+0000 7f2cdb200640 1 -- 172.21.15.102:0/3971753870 shutdown_connections 2024-03-23T03:49:48.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.183+0000 7f2cdb200640 1 -- 172.21.15.102:0/3971753870 wait complete. 2024-03-23T03:49:48.202 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:49:48.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.407+0000 7fdc19e00640 1 Processor -- start 2024-03-23T03:49:48.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.407+0000 7fdc19e00640 1 -- start start 2024-03-23T03:49:48.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.409+0000 7fdc19e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fdc140cf040 0x7fdc141422d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:48.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.409+0000 7fdc19e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1406cdc0 con 0x7fdc1413fa80 2024-03-23T03:49:48.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.409+0000 7fdc19e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1406d060 con 0x7fdc14160dd0 2024-03-23T03:49:48.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.409+0000 7fdc19e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc1406d340 con 0x7fdc140cf040 2024-03-23T03:49:48.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.409+0000 7fdc12a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdc1413fa80 0x7fdc1413ff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42262/0 (socket says 172.21.15.102:42262) 2024-03-23T03:49:48.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.409+0000 7fdc12a00640 1 -- 172.21.15.102:0/698290804 learned_addr learned my addr 172.21.15.102:0/698290804 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:48.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.410+0000 7fdc13400640 1 --2- 172.21.15.102:0/698290804 >> v2:172.21.15.173:3300/0 conn(0x7fdc140cf040 0x7fdc141422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:48.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.410+0000 7fdc12000640 1 -- 172.21.15.102:0/698290804 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1329547535 0 0) 0x7fdc1406cdc0 con 0x7fdc1413fa80 2024-03-23T03:49:48.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.410+0000 7fdc12000640 1 -- 172.21.15.102:0/698290804 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf8003520 con 0x7fdc1413fa80 2024-03-23T03:49:48.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.410+0000 7fdc12000640 1 -- 172.21.15.102:0/698290804 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 717395509 0 0) 0x7fdbf8003520 con 0x7fdc1413fa80 2024-03-23T03:49:48.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.410+0000 7fdc12000640 1 -- 172.21.15.102:0/698290804 >> v2:172.21.15.173:3300/0 conn(0x7fdc140cf040 msgr2=0x7fdc141422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:48.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.410+0000 7fdc12000640 1 --2- 172.21.15.102:0/698290804 >> v2:172.21.15.173:3300/0 conn(0x7fdc140cf040 0x7fdc141422d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:48.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.410+0000 7fdc12000640 1 -- 172.21.15.102:0/698290804 >> v1:172.21.15.173:6789/0 conn(0x7fdc14160dd0 legacy=0x7fdc140ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.410+0000 7fdc12000640 1 -- 172.21.15.102:0/698290804 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1406d620 con 0x7fdc1413fa80 2024-03-23T03:49:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.410+0000 7fdc12000640 1 -- 172.21.15.102:0/698290804 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fdc04001670 con 0x7fdc1413fa80 2024-03-23T03:49:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.410+0000 7fdc12000640 1 -- 172.21.15.102:0/698290804 >> v1:172.21.15.102:6789/0 conn(0x7fdc1413fa80 legacy=0x7fdc1413ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.410+0000 7fdc12000640 1 --2- 172.21.15.102:0/698290804 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdbf8003970 0x7fdbf8003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.410+0000 7fdc12000640 1 --2- 172.21.15.102:0/698290804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbf80047c0 0x7fdbf8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.410+0000 7fdc12000640 1 --2- 172.21.15.102:0/698290804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbf8005180 0x7fdbf800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.410+0000 7fdc12000640 1 -- 172.21.15.102:0/698290804 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1406d340 con 0x7fdbf80047c0 2024-03-23T03:49:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.410+0000 7fdc12000640 1 -- 172.21.15.102:0/698290804 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf800c0b0 con 0x7fdbf8005180 2024-03-23T03:49:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.410+0000 7fdc12000640 1 -- 172.21.15.102:0/698290804 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf800c270 con 0x7fdbf8003970 2024-03-23T03:49:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.410+0000 7fdc0a000640 1 --2- 172.21.15.102:0/698290804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbf80047c0 0x7fdbf8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.411+0000 7fdc12a00640 1 --2- 172.21.15.102:0/698290804 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdbf8003970 0x7fdbf8003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.411+0000 7fdc13400640 1 --2- 172.21.15.102:0/698290804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbf8005180 0x7fdbf800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.411+0000 7fdc0a000640 1 -- 172.21.15.102:0/698290804 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdbf8003970 msgr2=0x7fdbf8003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.411+0000 7fdc0a000640 1 --2- 172.21.15.102:0/698290804 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdbf8003970 0x7fdbf8003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:48.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.411+0000 7fdc0a000640 1 -- 172.21.15.102:0/698290804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbf8005180 msgr2=0x7fdbf800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:48.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.411+0000 7fdc0a000640 1 --2- 172.21.15.102:0/698290804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbf8005180 0x7fdbf800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:48.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.411+0000 7fdc0a000640 1 -- 172.21.15.102:0/698290804 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf8003520 con 0x7fdbf80047c0 2024-03-23T03:49:48.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.411+0000 7fdc12a00640 1 --2- 172.21.15.102:0/698290804 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdbf8003970 0x7fdbf8003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:48.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.411+0000 7fdc13400640 1 --2- 172.21.15.102:0/698290804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdbf8005180 0x7fdbf800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:48.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.411+0000 7fdc13400640 1 -- 172.21.15.102:0/698290804 reap_dead start 2024-03-23T03:49:48.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.412+0000 7fdc0a000640 1 --2- 172.21.15.102:0/698290804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbf80047c0 0x7fdbf8004c40 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc003d80 tx=0x7fdbf80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d1059cc24afb535 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:48.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.412+0000 7fdc12000640 1 -- 172.21.15.102:0/698290804 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc032600 con 0x7fdbf80047c0 2024-03-23T03:49:48.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.412+0000 7fdc12000640 1 -- 172.21.15.102:0/698290804 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbfc02e050 con 0x7fdbf80047c0 2024-03-23T03:49:48.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.412+0000 7fdc12000640 1 -- 172.21.15.102:0/698290804 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc0329e0 con 0x7fdbf80047c0 2024-03-23T03:49:48.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.413+0000 7fdc19e00640 1 -- 172.21.15.102:0/698290804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbf80047c0 msgr2=0x7fdbf8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:48.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.413+0000 7fdc19e00640 1 --2- 172.21.15.102:0/698290804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbf80047c0 0x7fdbf8004c40 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc003d80 tx=0x7fdbf80057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:48.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.413+0000 7fdc19e00640 1 -- 172.21.15.102:0/698290804 shutdown_connections 2024-03-23T03:49:48.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.413+0000 7fdc19e00640 1 --2- 172.21.15.102:0/698290804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbf80047c0 0x7fdbf8004c40 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:48.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.413+0000 7fdc19e00640 1 -- 172.21.15.102:0/698290804 >> 172.21.15.102:0/698290804 conn(0x7fdc1406a0b0 msgr2=0x7fdc1406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:48.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.413+0000 7fdc19e00640 1 -- 172.21.15.102:0/698290804 shutdown_connections 2024-03-23T03:49:48.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.413+0000 7fdc19e00640 1 -- 172.21.15.102:0/698290804 wait complete. 2024-03-23T03:49:48.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.415+0000 7fdc19e00640 1 Processor -- start 2024-03-23T03:49:48.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.415+0000 7fdc19e00640 1 -- start start 2024-03-23T03:49:48.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.416+0000 7fdc19e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbf80047c0 0x7fdc141e7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:48.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.416+0000 7fdc19e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc140af830 0x7fdc14156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:48.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.416+0000 7fdc19e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdc140b0470 0x7fdc140b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:48.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.416+0000 7fdc19e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc141e8b60 con 0x7fdbf80047c0 2024-03-23T03:49:48.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.416+0000 7fdc19e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc141f2820 con 0x7fdc140af830 2024-03-23T03:49:48.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.416+0000 7fdc19e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc141f2510 con 0x7fdc140b0470 2024-03-23T03:49:48.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.416+0000 7fdc12a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbf80047c0 0x7fdc141e7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:48.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.416+0000 7fdc12a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbf80047c0 0x7fdc141e7bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54386/0 (socket says 172.21.15.102:54386) 2024-03-23T03:49:48.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.416+0000 7fdc12a00640 1 -- 172.21.15.102:0/3024981687 learned_addr learned my addr 172.21.15.102:0/3024981687 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:48.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.416+0000 7fdc0a000640 1 --2- 172.21.15.102:0/3024981687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc140af830 0x7fdc14156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:48.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.417+0000 7fdc13400640 1 --2- 172.21.15.102:0/3024981687 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdc140b0470 0x7fdc140b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:48.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.417+0000 7fdc12a00640 1 -- 172.21.15.102:0/3024981687 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdc140b0470 msgr2=0x7fdc140b08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:48.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.417+0000 7fdc12a00640 1 --2- 172.21.15.102:0/3024981687 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdc140b0470 0x7fdc140b08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:48.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.417+0000 7fdc12a00640 1 -- 172.21.15.102:0/3024981687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc140af830 msgr2=0x7fdc14156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:48.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.417+0000 7fdc12a00640 1 --2- 172.21.15.102:0/3024981687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc140af830 0x7fdc14156c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:48.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.417+0000 7fdc12a00640 1 -- 172.21.15.102:0/3024981687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc140afcb0 con 0x7fdbf80047c0 2024-03-23T03:49:48.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.417+0000 7fdc13400640 1 --2- 172.21.15.102:0/3024981687 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdc140b0470 0x7fdc140b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:48.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.417+0000 7fdc0a000640 1 --2- 172.21.15.102:0/3024981687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc140af830 0x7fdc14156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:48.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.417+0000 7fdc12a00640 1 --2- 172.21.15.102:0/3024981687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbf80047c0 0x7fdc141e7bc0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400d760 tx=0x7fdc0400dc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:48.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.417+0000 7fdc0be00640 1 -- 172.21.15.102:0/3024981687 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc040074e0 con 0x7fdbf80047c0 2024-03-23T03:49:48.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.417+0000 7fdc0be00640 1 -- 172.21.15.102:0/3024981687 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc04007640 con 0x7fdbf80047c0 2024-03-23T03:49:48.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.417+0000 7fdc19e00640 1 -- 172.21.15.102:0/3024981687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc141e82c0 con 0x7fdbf80047c0 2024-03-23T03:49:48.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.417+0000 7fdc0be00640 1 -- 172.21.15.102:0/3024981687 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc04004a00 con 0x7fdbf80047c0 2024-03-23T03:49:48.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.417+0000 7fdc19e00640 1 -- 172.21.15.102:0/3024981687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc141e8880 con 0x7fdbf80047c0 2024-03-23T03:49:48.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.419+0000 7fdc19e00640 1 -- 172.21.15.102:0/3024981687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc140670d0 con 0x7fdbf80047c0 2024-03-23T03:49:48.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.419+0000 7fdc0be00640 1 -- 172.21.15.102:0/3024981687 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdc04004b60 con 0x7fdbf80047c0 2024-03-23T03:49:48.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.421+0000 7fdc0be00640 1 --2- 172.21.15.102:0/3024981687 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdbf006ac00 0x7fdbf006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:48.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.421+0000 7fdc0be00640 1 -- 172.21.15.102:0/3024981687 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7fdc04089180 con 0x7fdbf80047c0 2024-03-23T03:49:48.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.421+0000 7fdc0a000640 1 --2- 172.21.15.102:0/3024981687 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdbf006ac00 0x7fdbf006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:48.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.422+0000 7fdc0a000640 1 --2- 172.21.15.102:0/3024981687 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdbf006ac00 0x7fdbf006d0c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc034000 tx=0x7fdbfc009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:48.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.425+0000 7fdc0be00640 1 -- 172.21.15.102:0/3024981687 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdc040540f0 con 0x7fdbf80047c0 2024-03-23T03:49:48.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.618+0000 7fdc19e00640 1 -- 172.21.15.102:0/3024981687 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fdc14073390 con 0x7fdbf80047c0 2024-03-23T03:49:48.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.619+0000 7fdc0be00640 1 -- 172.21.15.102:0/3024981687 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v69) v1 ==== 80+0+0 (secure 0 0 0) 0x7fdc04057710 con 0x7fdbf80047c0 2024-03-23T03:49:48.619 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:49:48.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.621+0000 7fdc19e00640 1 -- 172.21.15.102:0/3024981687 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdbf006ac00 msgr2=0x7fdbf006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:48.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.621+0000 7fdc19e00640 1 --2- 172.21.15.102:0/3024981687 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdbf006ac00 0x7fdbf006d0c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc034000 tx=0x7fdbfc009030 comp rx=0 tx=0).stop 2024-03-23T03:49:48.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.621+0000 7fdc19e00640 1 -- 172.21.15.102:0/3024981687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbf80047c0 msgr2=0x7fdc141e7bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:48.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.621+0000 7fdc19e00640 1 --2- 172.21.15.102:0/3024981687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbf80047c0 0x7fdc141e7bc0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400d760 tx=0x7fdc0400dc20 comp rx=0 tx=0).stop 2024-03-23T03:49:48.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.621+0000 7fdc19e00640 1 -- 172.21.15.102:0/3024981687 shutdown_connections 2024-03-23T03:49:48.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.621+0000 7fdc19e00640 1 --2- 172.21.15.102:0/3024981687 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdbf006ac00 0x7fdbf006d0c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:48.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.621+0000 7fdc19e00640 1 --2- 172.21.15.102:0/3024981687 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdc140b0470 0x7fdc140b08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:48.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.621+0000 7fdc19e00640 1 --2- 172.21.15.102:0/3024981687 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdc140af830 0x7fdc14156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:48.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.621+0000 7fdc19e00640 1 --2- 172.21.15.102:0/3024981687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdbf80047c0 0x7fdc141e7bc0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:48.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.621+0000 7fdc19e00640 1 -- 172.21.15.102:0/3024981687 >> 172.21.15.102:0/3024981687 conn(0x7fdc1406a0b0 msgr2=0x7fdc140ab8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:48.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.622+0000 7fdc19e00640 1 -- 172.21.15.102:0/3024981687 shutdown_connections 2024-03-23T03:49:48.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:48.622+0000 7fdc19e00640 1 -- 172.21.15.102:0/3024981687 wait complete. 2024-03-23T03:49:48.639 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-23T03:49:48.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:48.639 DEBUG:teuthology.orchestra.run.smithi102:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-23T03:49:48.740 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:49:48.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:48.740 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T03:49:48.792 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:49:48.792 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:48.848 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:49:48.848 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:48.900 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:49:49.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.113+0000 7f41e1600640 1 Processor -- start 2024-03-23T03:49:49.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.113+0000 7f41e1600640 1 -- start start 2024-03-23T03:49:49.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.114+0000 7f41e1600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f41dc0adbc0 0x7f41dc0b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:49.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.114+0000 7f41e1600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f41dc06d390 0x7f41dc06d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:49.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.115+0000 7f41e1600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41dc0b16a0 con 0x7f41dc0cbab0 2024-03-23T03:49:49.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.115+0000 7f41e1600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41dc06dd40 con 0x7f41dc0adbc0 2024-03-23T03:49:49.115 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.115+0000 7f41e1600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc0b4700 con 0x7f41dc06d390 2024-03-23T03:49:49.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.115+0000 7f41daa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f41dc0adbc0 0x7f41dc0b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:49.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.115+0000 7f41daa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f41dc0adbc0 0x7f41dc0b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54402/0 (socket says 172.21.15.102:54402) 2024-03-23T03:49:49.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.115+0000 7f41daa00640 1 -- 172.21.15.102:0/3368947628 learned_addr learned my addr 172.21.15.102:0/3368947628 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:49.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.115+0000 7f41db400640 1 --2- 172.21.15.102:0/3368947628 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f41dc06d390 0x7f41dc06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:49.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.115+0000 7f41daa00640 1 -- 172.21.15.102:0/3368947628 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f41dc06d390 msgr2=0x7f41dc06d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:49.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.115+0000 7f41daa00640 1 --2- 172.21.15.102:0/3368947628 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f41dc06d390 0x7f41dc06d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.115+0000 7f41daa00640 1 -- 172.21.15.102:0/3368947628 >> v1:172.21.15.173:6789/0 conn(0x7f41dc0cbab0 legacy=0x7f41dc0b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:49.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.115+0000 7f41daa00640 1 -- 172.21.15.102:0/3368947628 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41dc0cbfb0 con 0x7f41dc0adbc0 2024-03-23T03:49:49.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.115+0000 7f41db400640 1 --2- 172.21.15.102:0/3368947628 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f41dc06d390 0x7f41dc06d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:49.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.116+0000 7f41daa00640 1 --2- 172.21.15.102:0/3368947628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41dc0adbc0 0x7f41dc0b41c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00bb10 tx=0x7f41cc02e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8bc1b318859bd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:49.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.117+0000 7f41da000640 1 -- 172.21.15.102:0/3368947628 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc02ede0 con 0x7f41dc0adbc0 2024-03-23T03:49:49.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.117+0000 7f41da000640 1 -- 172.21.15.102:0/3368947628 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc036440 con 0x7f41dc0adbc0 2024-03-23T03:49:49.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.117+0000 7f41da000640 1 -- 172.21.15.102:0/3368947628 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc0367d0 con 0x7f41dc0adbc0 2024-03-23T03:49:49.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.117+0000 7f41e1600640 1 -- 172.21.15.102:0/3368947628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41dc0adbc0 msgr2=0x7f41dc0b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:49.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.117+0000 7f41e1600640 1 --2- 172.21.15.102:0/3368947628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41dc0adbc0 0x7f41dc0b41c0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00bb10 tx=0x7f41cc02e030 comp rx=0 tx=0).stop 2024-03-23T03:49:49.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.117+0000 7f41e1600640 1 -- 172.21.15.102:0/3368947628 shutdown_connections 2024-03-23T03:49:49.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.117+0000 7f41e1600640 1 --2- 172.21.15.102:0/3368947628 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f41dc06d390 0x7f41dc06d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.117+0000 7f41e1600640 1 --2- 172.21.15.102:0/3368947628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41dc0adbc0 0x7f41dc0b41c0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.117+0000 7f41e1600640 1 -- 172.21.15.102:0/3368947628 >> 172.21.15.102:0/3368947628 conn(0x7f41dc06a3d0 msgr2=0x7f41dc0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:49.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.117+0000 7f41e1600640 1 -- 172.21.15.102:0/3368947628 shutdown_connections 2024-03-23T03:49:49.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.117+0000 7f41e1600640 1 -- 172.21.15.102:0/3368947628 wait complete. 2024-03-23T03:49:49.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.119+0000 7f41e1600640 1 Processor -- start 2024-03-23T03:49:49.119 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.119+0000 7f41e1600640 1 -- start start 2024-03-23T03:49:49.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.120+0000 7f41e1600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41dc06d390 0x7f41dc1eda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:49.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.120+0000 7f41e1600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41dc0adbc0 0x7f41dc15dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:49.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.120+0000 7f41e1600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f41dc0cbab0 0x7f41dc0affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:49.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.120+0000 7f41e1600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc1ef050 con 0x7f41dc06d390 2024-03-23T03:49:49.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.120+0000 7f41e1600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc15cc20 con 0x7f41dc0adbc0 2024-03-23T03:49:49.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.120+0000 7f41e1600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc1ee3d0 con 0x7f41dc0cbab0 2024-03-23T03:49:49.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.120+0000 7f41daa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41dc06d390 0x7f41dc1eda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:49.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.120+0000 7f41daa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41dc06d390 0x7f41dc1eda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54406/0 (socket says 172.21.15.102:54406) 2024-03-23T03:49:49.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.120+0000 7f41daa00640 1 -- 172.21.15.102:0/1339805802 learned_addr learned my addr 172.21.15.102:0/1339805802 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:49.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.121+0000 7f41daa00640 1 -- 172.21.15.102:0/1339805802 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f41dc0cbab0 msgr2=0x7f41dc0affe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:49.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.121+0000 7f41daa00640 1 --2- 172.21.15.102:0/1339805802 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f41dc0cbab0 0x7f41dc0affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.121+0000 7f41daa00640 1 -- 172.21.15.102:0/1339805802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41dc0adbc0 msgr2=0x7f41dc15dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:49.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.121+0000 7f41daa00640 1 --2- 172.21.15.102:0/1339805802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41dc0adbc0 0x7f41dc15dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.121+0000 7f41daa00640 1 -- 172.21.15.102:0/1339805802 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41cc00b770 con 0x7f41dc06d390 2024-03-23T03:49:49.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.121+0000 7f41daa00640 1 --2- 172.21.15.102:0/1339805802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41dc06d390 0x7f41dc1eda90 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00b2c0 tx=0x7f41cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:49.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.121+0000 7f41d3400640 1 -- 172.21.15.102:0/1339805802 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc03f840 con 0x7f41dc06d390 2024-03-23T03:49:49.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.121+0000 7f41d3400640 1 -- 172.21.15.102:0/1339805802 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc03f9a0 con 0x7f41dc06d390 2024-03-23T03:49:49.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.121+0000 7f41d3400640 1 -- 172.21.15.102:0/1339805802 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc03fd50 con 0x7f41dc06d390 2024-03-23T03:49:49.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.121+0000 7f41e1600640 1 -- 172.21.15.102:0/1339805802 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41dc1ee6b0 con 0x7f41dc06d390 2024-03-23T03:49:49.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.121+0000 7f41e1600640 1 -- 172.21.15.102:0/1339805802 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41dc1eec70 con 0x7f41dc06d390 2024-03-23T03:49:49.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.123+0000 7f41e1600640 1 -- 172.21.15.102:0/1339805802 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41dc06ac20 con 0x7f41dc06d390 2024-03-23T03:49:49.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.123+0000 7f41d3400640 1 -- 172.21.15.102:0/1339805802 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f41cc03d040 con 0x7f41dc06d390 2024-03-23T03:49:49.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.124+0000 7f41d3400640 1 --2- 172.21.15.102:0/1339805802 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f41b806ac00 0x7f41b806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:49.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.124+0000 7f41d3400640 1 -- 172.21.15.102:0/1339805802 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7f41cc0ae060 con 0x7f41dc06d390 2024-03-23T03:49:49.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.124+0000 7f41d2000640 1 --2- 172.21.15.102:0/1339805802 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f41b806ac00 0x7f41b806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:49.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.125+0000 7f41d2000640 1 --2- 172.21.15.102:0/1339805802 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f41b806ac00 0x7f41b806d0c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0cb1a0 tx=0x7f41c40079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:49.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.129+0000 7f41d3400640 1 -- 172.21.15.102:0/1339805802 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f41cc0e1580 con 0x7f41dc06d390 2024-03-23T03:49:49.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.321+0000 7f41e1600640 1 -- 172.21.15.102:0/1339805802 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41dc073390 con 0x7f41dc06d390 2024-03-23T03:49:49.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.324+0000 7f41d3400640 1 -- 172.21.15.102:0/1339805802 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16536 (secure 0 0 0) 0x7f41cc078fd0 con 0x7f41dc06d390 2024-03-23T03:49:49.326 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:49.326 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":69,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:49:43.895428+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:49:33.007757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-23T03:49:34.449411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.e","mappings":[{"from":7,"to":2}]},{"pgid":"10.23","mappings":[{"from":7,"to":3}]},{"pgid":"10.2e","mappings":[{"from":4,"to":3}]},{"pgid":"10.2f","mappings":[{"from":4,"to":5}]},{"pgid":"10.3c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:49:49.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.327+0000 7f41e1600640 1 -- 172.21.15.102:0/1339805802 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f41b806ac00 msgr2=0x7f41b806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:49.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.327+0000 7f41e1600640 1 --2- 172.21.15.102:0/1339805802 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f41b806ac00 0x7f41b806d0c0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f41dc0cb1a0 tx=0x7f41c40079b0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.327+0000 7f41e1600640 1 -- 172.21.15.102:0/1339805802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41dc06d390 msgr2=0x7f41dc1eda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:49.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.327+0000 7f41e1600640 1 --2- 172.21.15.102:0/1339805802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41dc06d390 0x7f41dc1eda90 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f41cc00b2c0 tx=0x7f41cc0046c0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.327+0000 7f41e1600640 1 -- 172.21.15.102:0/1339805802 shutdown_connections 2024-03-23T03:49:49.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.327+0000 7f41e1600640 1 --2- 172.21.15.102:0/1339805802 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f41b806ac00 0x7f41b806d0c0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.327+0000 7f41e1600640 1 --2- 172.21.15.102:0/1339805802 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f41dc0cbab0 0x7f41dc0affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.327+0000 7f41e1600640 1 --2- 172.21.15.102:0/1339805802 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f41dc0adbc0 0x7f41dc15dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.327+0000 7f41e1600640 1 --2- 172.21.15.102:0/1339805802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41dc06d390 0x7f41dc1eda90 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.327+0000 7f41e1600640 1 -- 172.21.15.102:0/1339805802 >> 172.21.15.102:0/1339805802 conn(0x7f41dc06a3d0 msgr2=0x7f41dc140c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:49.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.328+0000 7f41e1600640 1 -- 172.21.15.102:0/1339805802 shutdown_connections 2024-03-23T03:49:49.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.328+0000 7f41e1600640 1 -- 172.21.15.102:0/1339805802 wait complete. 2024-03-23T03:49:49.346 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:49:33.007757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:49:34.449411+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:49:49.347 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:49:49.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.554+0000 7f60e7e00640 1 Processor -- start 2024-03-23T03:49:49.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.554+0000 7f60e7e00640 1 -- start start 2024-03-23T03:49:49.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.556+0000 7f60e7e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f60e815c9e0 0x7f60e815dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:49.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.556+0000 7f60e7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f60e80b12d0 0x7f60e8160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:49.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.556+0000 7f60e7e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e8160530 con 0x7f60e80cad10 2024-03-23T03:49:49.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.556+0000 7f60e7e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e8160af0 con 0x7f60e80b12d0 2024-03-23T03:49:49.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.556+0000 7f60e7e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e81614c0 con 0x7f60e815c9e0 2024-03-23T03:49:49.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.556+0000 7f60e6000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f60e80b12d0 0x7f60e8160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:49.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.556+0000 7f60e6000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f60e80b12d0 0x7f60e8160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54410/0 (socket says 172.21.15.102:54410) 2024-03-23T03:49:49.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.556+0000 7f60e6000640 1 -- 172.21.15.102:0/2521408433 learned_addr learned my addr 172.21.15.102:0/2521408433 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:49.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.556+0000 7f60e6a00640 1 --2- 172.21.15.102:0/2521408433 >> v2:172.21.15.173:3300/0 conn(0x7f60e815c9e0 0x7f60e815dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:49.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.556+0000 7f60e6000640 1 -- 172.21.15.102:0/2521408433 >> v2:172.21.15.173:3300/0 conn(0x7f60e815c9e0 msgr2=0x7f60e815dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:49.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.557+0000 7f60e6000640 1 --2- 172.21.15.102:0/2521408433 >> v2:172.21.15.173:3300/0 conn(0x7f60e815c9e0 0x7f60e815dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.557+0000 7f60e6000640 1 -- 172.21.15.102:0/2521408433 >> v1:172.21.15.173:6789/0 conn(0x7f60e80cad10 legacy=0x7f60e80c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:49.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.557+0000 7f60e6000640 1 -- 172.21.15.102:0/2521408433 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e815cdc0 con 0x7f60e80b12d0 2024-03-23T03:49:49.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.558+0000 7f60e6000640 1 --2- 172.21.15.102:0/2521408433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60e80b12d0 0x7f60e8160f80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f60d4009870 tx=0x7f60d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07807978e980e35 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:49.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.558+0000 7f60e5600640 1 -- 172.21.15.102:0/2521408433 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d4031870 con 0x7f60e80b12d0 2024-03-23T03:49:49.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.558+0000 7f60e5600640 1 -- 172.21.15.102:0/2521408433 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d40319d0 con 0x7f60e80b12d0 2024-03-23T03:49:49.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.558+0000 7f60e5600640 1 -- 172.21.15.102:0/2521408433 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d4031d60 con 0x7f60e80b12d0 2024-03-23T03:49:49.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.558+0000 7f60e7e00640 1 -- 172.21.15.102:0/2521408433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60e80b12d0 msgr2=0x7f60e8160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:49.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.558+0000 7f60e7e00640 1 --2- 172.21.15.102:0/2521408433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60e80b12d0 0x7f60e8160f80 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f60d4009870 tx=0x7f60d402b910 comp rx=0 tx=0).stop 2024-03-23T03:49:49.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.559+0000 7f60e7e00640 1 -- 172.21.15.102:0/2521408433 shutdown_connections 2024-03-23T03:49:49.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.559+0000 7f60e7e00640 1 --2- 172.21.15.102:0/2521408433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60e80b12d0 0x7f60e8160f80 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.559+0000 7f60e7e00640 1 --2- 172.21.15.102:0/2521408433 >> v2:172.21.15.173:3300/0 conn(0x7f60e815c9e0 0x7f60e815dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.559+0000 7f60e7e00640 1 -- 172.21.15.102:0/2521408433 >> 172.21.15.102:0/2521408433 conn(0x7f60e80b01b0 msgr2=0x7f60e80b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:49.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.559+0000 7f60e7e00640 1 -- 172.21.15.102:0/2521408433 shutdown_connections 2024-03-23T03:49:49.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.559+0000 7f60e7e00640 1 -- 172.21.15.102:0/2521408433 wait complete. 2024-03-23T03:49:49.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.560+0000 7f60e7e00640 1 Processor -- start 2024-03-23T03:49:49.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.561+0000 7f60e7e00640 1 -- start start 2024-03-23T03:49:49.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.561+0000 7f60e7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60e80b12d0 0x7f60e80cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:49.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.561+0000 7f60e7e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f60e80cad10 0x7f60e80cdf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:49.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.561+0000 7f60e7e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60e815c9e0 0x7f60e80ce450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:49.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.561+0000 7f60e7e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e81523c0 con 0x7f60e80b12d0 2024-03-23T03:49:49.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.561+0000 7f60e7e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e80cdb70 con 0x7f60e815c9e0 2024-03-23T03:49:49.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.561+0000 7f60e7e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e81585a0 con 0x7f60e80cad10 2024-03-23T03:49:49.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.562+0000 7f60e6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60e80b12d0 0x7f60e80cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:49.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.562+0000 7f60e6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60e80b12d0 0x7f60e80cd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54412/0 (socket says 172.21.15.102:54412) 2024-03-23T03:49:49.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.562+0000 7f60e6a00640 1 -- 172.21.15.102:0/4060952150 learned_addr learned my addr 172.21.15.102:0/4060952150 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:49.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.562+0000 7f60e6000640 1 --2- 172.21.15.102:0/4060952150 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f60e80cad10 0x7f60e80cdf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:49.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.562+0000 7f60e7400640 1 --2- 172.21.15.102:0/4060952150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60e815c9e0 0x7f60e80ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:49.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.562+0000 7f60e6a00640 1 -- 172.21.15.102:0/4060952150 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f60e80cad10 msgr2=0x7f60e80cdf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:49.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.562+0000 7f60e6a00640 1 --2- 172.21.15.102:0/4060952150 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f60e80cad10 0x7f60e80cdf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.562+0000 7f60e6a00640 1 -- 172.21.15.102:0/4060952150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60e815c9e0 msgr2=0x7f60e80ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:49.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.562+0000 7f60e6a00640 1 --2- 172.21.15.102:0/4060952150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60e815c9e0 0x7f60e80ce450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.562+0000 7f60e6a00640 1 -- 172.21.15.102:0/4060952150 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d40094d0 con 0x7f60e80b12d0 2024-03-23T03:49:49.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.562+0000 7f60e6a00640 1 --2- 172.21.15.102:0/4060952150 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60e80b12d0 0x7f60e80cd0f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f60d800ccd0 tx=0x7f60d800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:49.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.563+0000 7f60dea00640 1 -- 172.21.15.102:0/4060952150 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d8003990 con 0x7f60e80b12d0 2024-03-23T03:49:49.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.563+0000 7f60dea00640 1 -- 172.21.15.102:0/4060952150 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d8003af0 con 0x7f60e80b12d0 2024-03-23T03:49:49.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.563+0000 7f60dea00640 1 -- 172.21.15.102:0/4060952150 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d8004020 con 0x7f60e80b12d0 2024-03-23T03:49:49.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.563+0000 7f60e7e00640 1 -- 172.21.15.102:0/4060952150 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e8158880 con 0x7f60e80b12d0 2024-03-23T03:49:49.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.563+0000 7f60e7e00640 1 -- 172.21.15.102:0/4060952150 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e8158e40 con 0x7f60e80b12d0 2024-03-23T03:49:49.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.564+0000 7f60e7e00640 1 -- 172.21.15.102:0/4060952150 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e804e030 con 0x7f60e80b12d0 2024-03-23T03:49:49.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.565+0000 7f60dea00640 1 -- 172.21.15.102:0/4060952150 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f60d8004180 con 0x7f60e80b12d0 2024-03-23T03:49:49.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.566+0000 7f60dea00640 1 --2- 172.21.15.102:0/4060952150 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f60c806ac00 0x7f60c806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:49.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.566+0000 7f60dea00640 1 -- 172.21.15.102:0/4060952150 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7f60d8089a40 con 0x7f60e80b12d0 2024-03-23T03:49:49.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.566+0000 7f60e6000640 1 --2- 172.21.15.102:0/4060952150 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f60c806ac00 0x7f60c806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:49.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.567+0000 7f60e6000640 1 --2- 172.21.15.102:0/4060952150 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f60c806ac00 0x7f60c806d0c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f60d4036000 tx=0x7f60d403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:49.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.570+0000 7f60dea00640 1 -- 172.21.15.102:0/4060952150 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f60d80549b0 con 0x7f60e80b12d0 2024-03-23T03:49:49.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.766+0000 7f60e7e00640 1 -- 172.21.15.102:0/4060952150 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f60e80caaf0 con 0x7f60e80b12d0 2024-03-23T03:49:49.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.767+0000 7f60dea00640 1 -- 172.21.15.102:0/4060952150 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f60d801d070 con 0x7f60e80b12d0 2024-03-23T03:49:49.768 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:49:49.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.770+0000 7f60e7e00640 1 -- 172.21.15.102:0/4060952150 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f60c806ac00 msgr2=0x7f60c806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:49.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.770+0000 7f60e7e00640 1 --2- 172.21.15.102:0/4060952150 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f60c806ac00 0x7f60c806d0c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f60d4036000 tx=0x7f60d403c040 comp rx=0 tx=0).stop 2024-03-23T03:49:49.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.770+0000 7f60e7e00640 1 -- 172.21.15.102:0/4060952150 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60e80b12d0 msgr2=0x7f60e80cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:49.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.770+0000 7f60e7e00640 1 --2- 172.21.15.102:0/4060952150 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60e80b12d0 0x7f60e80cd0f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f60d800ccd0 tx=0x7f60d800a5c0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.770+0000 7f60e7e00640 1 -- 172.21.15.102:0/4060952150 shutdown_connections 2024-03-23T03:49:49.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.770+0000 7f60e7e00640 1 --2- 172.21.15.102:0/4060952150 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f60c806ac00 0x7f60c806d0c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.770+0000 7f60e7e00640 1 --2- 172.21.15.102:0/4060952150 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60e815c9e0 0x7f60e80ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.770+0000 7f60e7e00640 1 --2- 172.21.15.102:0/4060952150 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f60e80cad10 0x7f60e80cdf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.770+0000 7f60e7e00640 1 --2- 172.21.15.102:0/4060952150 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60e80b12d0 0x7f60e80cd0f0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:49.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.770+0000 7f60e7e00640 1 -- 172.21.15.102:0/4060952150 >> 172.21.15.102:0/4060952150 conn(0x7f60e80b01b0 msgr2=0x7f60e80c91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:49.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.771+0000 7f60e7e00640 1 -- 172.21.15.102:0/4060952150 shutdown_connections 2024-03-23T03:49:49.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:49.771+0000 7f60e7e00640 1 -- 172.21.15.102:0/4060952150 wait complete. 2024-03-23T03:49:49.789 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:49:50.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.002+0000 7fbb19e00640 1 Processor -- start 2024-03-23T03:49:50.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.002+0000 7fbb19e00640 1 -- start start 2024-03-23T03:49:50.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.004+0000 7fbb19e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbb140ca7b0 0x7fbb140cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.004+0000 7fbb19e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fbb140ce930 0x7fbb140cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.004+0000 7fbb19e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb14159c10 con 0x7fbb14159830 2024-03-23T03:49:50.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.004+0000 7fbb19e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb1406de50 con 0x7fbb140ca7b0 2024-03-23T03:49:50.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.004+0000 7fbb19e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb140cf300 con 0x7fbb140ce930 2024-03-23T03:49:50.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.004+0000 7fbb12a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbb14159830 0x7fbb140cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42266/0 (socket says 172.21.15.102:42266) 2024-03-23T03:49:50.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.004+0000 7fbb12a00640 1 -- 172.21.15.102:0/4152347676 learned_addr learned my addr 172.21.15.102:0/4152347676 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:50.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.004+0000 7fbb0a000640 1 --2- 172.21.15.102:0/4152347676 >> v2:172.21.15.102:3300/0 conn(0x7fbb140ca7b0 0x7fbb140cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:50.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.004+0000 7fbb12000640 1 -- 172.21.15.102:0/4152347676 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067284385 0 0) 0x7fbb14159c10 con 0x7fbb14159830 2024-03-23T03:49:50.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.004+0000 7fbb12000640 1 -- 172.21.15.102:0/4152347676 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaf8005730 con 0x7fbb14159830 2024-03-23T03:49:50.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.005+0000 7fbb0a000640 1 -- 172.21.15.102:0/4152347676 >> v2:172.21.15.173:3300/0 conn(0x7fbb140ce930 msgr2=0x7fbb140cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:50.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.005+0000 7fbb13400640 1 --2- 172.21.15.102:0/4152347676 >> v2:172.21.15.173:3300/0 conn(0x7fbb140ce930 0x7fbb140cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:50.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.005+0000 7fbb0a000640 1 --2- 172.21.15.102:0/4152347676 >> v2:172.21.15.173:3300/0 conn(0x7fbb140ce930 0x7fbb140cedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.005+0000 7fbb0a000640 1 -- 172.21.15.102:0/4152347676 >> v1:172.21.15.102:6789/0 conn(0x7fbb14159830 legacy=0x7fbb140cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:50.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.005+0000 7fbb0a000640 1 -- 172.21.15.102:0/4152347676 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb140cd840 con 0x7fbb140ca7b0 2024-03-23T03:49:50.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.006+0000 7fbb0a000640 1 --2- 172.21.15.102:0/4152347676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb140ca7b0 0x7fbb140cdcd0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0097d0 tx=0x7fbafc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fed4a3927934386 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:50.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.006+0000 7fbb12000640 1 -- 172.21.15.102:0/4152347676 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc030610 con 0x7fbb140ca7b0 2024-03-23T03:49:50.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.006+0000 7fbb12000640 1 -- 172.21.15.102:0/4152347676 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbafc030770 con 0x7fbb140ca7b0 2024-03-23T03:49:50.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.006+0000 7fbb12000640 1 -- 172.21.15.102:0/4152347676 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc030b00 con 0x7fbb140ca7b0 2024-03-23T03:49:50.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.006+0000 7fbb19e00640 1 -- 172.21.15.102:0/4152347676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb140ca7b0 msgr2=0x7fbb140cdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:50.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.006+0000 7fbb19e00640 1 --2- 172.21.15.102:0/4152347676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb140ca7b0 0x7fbb140cdcd0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0097d0 tx=0x7fbafc02b960 comp rx=0 tx=0).stop 2024-03-23T03:49:50.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.007+0000 7fbb19e00640 1 -- 172.21.15.102:0/4152347676 shutdown_connections 2024-03-23T03:49:50.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.007+0000 7fbb19e00640 1 --2- 172.21.15.102:0/4152347676 >> v2:172.21.15.173:3300/0 conn(0x7fbb140ce930 0x7fbb140cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.007+0000 7fbb19e00640 1 --2- 172.21.15.102:0/4152347676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb140ca7b0 0x7fbb140cdcd0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.007+0000 7fbb19e00640 1 -- 172.21.15.102:0/4152347676 >> 172.21.15.102:0/4152347676 conn(0x7fbb1406a0b0 msgr2=0x7fbb1406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:50.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.007+0000 7fbb19e00640 1 -- 172.21.15.102:0/4152347676 shutdown_connections 2024-03-23T03:49:50.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.007+0000 7fbb19e00640 1 -- 172.21.15.102:0/4152347676 wait complete. 2024-03-23T03:49:50.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.008+0000 7fbb19e00640 1 Processor -- start 2024-03-23T03:49:50.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.009+0000 7fbb19e00640 1 -- start start 2024-03-23T03:49:50.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.009+0000 7fbb19e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb140ca7b0 0x7fbb14154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.009+0000 7fbb19e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb140ce930 0x7fbb14152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.010+0000 7fbb19e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb14159830 0x7fbb14070050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.010+0000 7fbb19e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb141535a0 con 0x7fbb140ce930 2024-03-23T03:49:50.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.010+0000 7fbb19e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb14153880 con 0x7fbb140ca7b0 2024-03-23T03:49:50.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.010+0000 7fbb19e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb14153b60 con 0x7fbb14159830 2024-03-23T03:49:50.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.010+0000 7fbb0a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb140ce930 0x7fbb14152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:50.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.010+0000 7fbb0a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb140ce930 0x7fbb14152bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54434/0 (socket says 172.21.15.102:54434) 2024-03-23T03:49:50.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.010+0000 7fbb0a000640 1 -- 172.21.15.102:0/4265529960 learned_addr learned my addr 172.21.15.102:0/4265529960 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.010+0000 7fbb0a000640 1 -- 172.21.15.102:0/4265529960 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb14159830 msgr2=0x7fbb14070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.010+0000 7fbb0a000640 1 --2- 172.21.15.102:0/4265529960 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb14159830 0x7fbb14070050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.010+0000 7fbb0a000640 1 -- 172.21.15.102:0/4265529960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb140ca7b0 msgr2=0x7fbb14154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.010+0000 7fbb0a000640 1 --2- 172.21.15.102:0/4265529960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb140ca7b0 0x7fbb14154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.010+0000 7fbb0a000640 1 -- 172.21.15.102:0/4265529960 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbafc009400 con 0x7fbb140ce930 2024-03-23T03:49:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.010+0000 7fbb0a000640 1 --2- 172.21.15.102:0/4265529960 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb140ce930 0x7fbb14152bc0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0098f0 tx=0x7fbafc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.011+0000 7fbb0be00640 1 -- 172.21.15.102:0/4265529960 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc03e070 con 0x7fbb140ce930 2024-03-23T03:49:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.011+0000 7fbb0be00640 1 -- 172.21.15.102:0/4265529960 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbafc030e10 con 0x7fbb140ce930 2024-03-23T03:49:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.011+0000 7fbb0be00640 1 -- 172.21.15.102:0/4265529960 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbafc039c50 con 0x7fbb140ce930 2024-03-23T03:49:50.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.011+0000 7fbb19e00640 1 -- 172.21.15.102:0/4265529960 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb14153e40 con 0x7fbb140ce930 2024-03-23T03:49:50.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.011+0000 7fbb19e00640 1 -- 172.21.15.102:0/4265529960 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1414b570 con 0x7fbb140ce930 2024-03-23T03:49:50.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.012+0000 7fbb19e00640 1 -- 172.21.15.102:0/4265529960 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb14154360 con 0x7fbb140ce930 2024-03-23T03:49:50.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.013+0000 7fbb0be00640 1 -- 172.21.15.102:0/4265529960 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbafc0423f0 con 0x7fbb140ce930 2024-03-23T03:49:50.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.014+0000 7fbb0be00640 1 --2- 172.21.15.102:0/4265529960 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbaf006ac00 0x7fbaf006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.014+0000 7fbb0be00640 1 -- 172.21.15.102:0/4265529960 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7fbafc0aaf10 con 0x7fbb140ce930 2024-03-23T03:49:50.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.014+0000 7fbb12a00640 1 --2- 172.21.15.102:0/4265529960 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbaf006ac00 0x7fbaf006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:50.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.015+0000 7fbb12a00640 1 --2- 172.21.15.102:0/4265529960 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbaf006ac00 0x7fbaf006d0c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbb04001610 tx=0x7fbb04007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:50.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.018+0000 7fbb0be00640 1 -- 172.21.15.102:0/4265529960 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbb14154360 con 0x7fbb140ce930 2024-03-23T03:49:50.134 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:49:50.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:49:50.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.213+0000 7fbb19e00640 1 -- 172.21.15.102:0/4265529960 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbb14154360 con 0x7fbb140ce930 2024-03-23T03:49:50.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.214+0000 7fbb0be00640 1 -- 172.21.15.102:0/4265529960 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbafc048020 con 0x7fbb140ce930 2024-03-23T03:49:50.214 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:49:50.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.216+0000 7fbb19e00640 1 -- 172.21.15.102:0/4265529960 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbaf006ac00 msgr2=0x7fbaf006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:50.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.216+0000 7fbb19e00640 1 --2- 172.21.15.102:0/4265529960 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbaf006ac00 0x7fbaf006d0c0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fbb04001610 tx=0x7fbb04007930 comp rx=0 tx=0).stop 2024-03-23T03:49:50.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.216+0000 7fbb19e00640 1 -- 172.21.15.102:0/4265529960 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb140ce930 msgr2=0x7fbb14152bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:50.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.216+0000 7fbb19e00640 1 --2- 172.21.15.102:0/4265529960 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb140ce930 0x7fbb14152bc0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0098f0 tx=0x7fbafc0046c0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.216+0000 7fbb19e00640 1 -- 172.21.15.102:0/4265529960 shutdown_connections 2024-03-23T03:49:50.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.216+0000 7fbb19e00640 1 --2- 172.21.15.102:0/4265529960 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbaf006ac00 0x7fbaf006d0c0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.216+0000 7fbb19e00640 1 --2- 172.21.15.102:0/4265529960 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbb14159830 0x7fbb14070050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.216+0000 7fbb19e00640 1 --2- 172.21.15.102:0/4265529960 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb140ce930 0x7fbb14152bc0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.216+0000 7fbb19e00640 1 --2- 172.21.15.102:0/4265529960 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbb140ca7b0 0x7fbb14154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.216+0000 7fbb19e00640 1 -- 172.21.15.102:0/4265529960 >> 172.21.15.102:0/4265529960 conn(0x7fbb1406a0b0 msgr2=0x7fbb1406a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:50.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.217+0000 7fbb19e00640 1 -- 172.21.15.102:0/4265529960 shutdown_connections 2024-03-23T03:49:50.218 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.217+0000 7fbb19e00640 1 -- 172.21.15.102:0/4265529960 wait complete. 2024-03-23T03:49:50.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:49:50.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.436+0000 7f613fe00640 1 Processor -- start 2024-03-23T03:49:50.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.436+0000 7f613fe00640 1 -- start start 2024-03-23T03:49:50.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.437+0000 7f613fe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6140143460 0x7f614015aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.438+0000 7f613fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6140143930 0x7f61400cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.438+0000 7f613fe00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f61400cdc50 0x7f614015e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.438+0000 7f613fe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61400cc6b0 con 0x7f6140143930 2024-03-23T03:49:50.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.438+0000 7f613fe00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61400cc950 con 0x7f61400cdc50 2024-03-23T03:49:50.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.438+0000 7f613fe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61400ccc30 con 0x7f6140143460 2024-03-23T03:49:50.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.438+0000 7f6136000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6140143930 0x7f61400cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:50.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.438+0000 7f6136000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6140143930 0x7f61400cd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54438/0 (socket says 172.21.15.102:54438) 2024-03-23T03:49:50.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.438+0000 7f6136000640 1 -- 172.21.15.102:0/3827015879 learned_addr learned my addr 172.21.15.102:0/3827015879 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:50.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.438+0000 7f613ea00640 1 --2- 172.21.15.102:0/3827015879 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6140143460 0x7f614015aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:50.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.438+0000 7f613f400640 1 --2- 172.21.15.102:0/3827015879 >> v2:172.21.15.173:3300/0 conn(0x7f61400cdc50 0x7f614015e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:50.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.439+0000 7f6136000640 1 -- 172.21.15.102:0/3827015879 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6140143460 msgr2=0x7f614015aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:50.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.439+0000 7f6136000640 1 --2- 172.21.15.102:0/3827015879 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6140143460 0x7f614015aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.439+0000 7f6136000640 1 -- 172.21.15.102:0/3827015879 >> v2:172.21.15.173:3300/0 conn(0x7f61400cdc50 msgr2=0x7f614015e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:50.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.439+0000 7f6136000640 1 --2- 172.21.15.102:0/3827015879 >> v2:172.21.15.173:3300/0 conn(0x7f61400cdc50 0x7f614015e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.439+0000 7f6136000640 1 -- 172.21.15.102:0/3827015879 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61400ccf10 con 0x7f6140143930 2024-03-23T03:49:50.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.440+0000 7f6136000640 1 --2- 172.21.15.102:0/3827015879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6140143930 0x7f61400cd710 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f61400cc190 tx=0x7f612802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6f543606463aad9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:50.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.440+0000 7f613e000640 1 -- 172.21.15.102:0/3827015879 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6128030860 con 0x7f6140143930 2024-03-23T03:49:50.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.440+0000 7f613e000640 1 -- 172.21.15.102:0/3827015879 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61280309c0 con 0x7f6140143930 2024-03-23T03:49:50.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.440+0000 7f613e000640 1 -- 172.21.15.102:0/3827015879 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6128030d50 con 0x7f6140143930 2024-03-23T03:49:50.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.441+0000 7f613fe00640 1 -- 172.21.15.102:0/3827015879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6140143930 msgr2=0x7f61400cd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:50.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.441+0000 7f613fe00640 1 --2- 172.21.15.102:0/3827015879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6140143930 0x7f61400cd710 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f61400cc190 tx=0x7f612802b9a0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.441+0000 7f613fe00640 1 -- 172.21.15.102:0/3827015879 shutdown_connections 2024-03-23T03:49:50.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.441+0000 7f613fe00640 1 --2- 172.21.15.102:0/3827015879 >> v2:172.21.15.173:3300/0 conn(0x7f61400cdc50 0x7f614015e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.441+0000 7f613fe00640 1 --2- 172.21.15.102:0/3827015879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6140143930 0x7f61400cd710 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.441+0000 7f613fe00640 1 --2- 172.21.15.102:0/3827015879 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6140143460 0x7f614015aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.441+0000 7f613fe00640 1 -- 172.21.15.102:0/3827015879 >> 172.21.15.102:0/3827015879 conn(0x7f6140065df0 msgr2=0x7f6140069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:50.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.441+0000 7f613fe00640 1 -- 172.21.15.102:0/3827015879 shutdown_connections 2024-03-23T03:49:50.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.441+0000 7f613fe00640 1 -- 172.21.15.102:0/3827015879 wait complete. 2024-03-23T03:49:50.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.443+0000 7f613fe00640 1 Processor -- start 2024-03-23T03:49:50.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.443+0000 7f613fe00640 1 -- start start 2024-03-23T03:49:50.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.444+0000 7f613fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61400cdc50 0x7f614006cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.444+0000 7f613fe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6140143460 0x7f6140138b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.444+0000 7f613fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6140143930 0x7f6140139090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.444+0000 7f613fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61401f2850 con 0x7f6140143930 2024-03-23T03:49:50.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.444+0000 7f613fe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6140139760 con 0x7f61400cdc50 2024-03-23T03:49:50.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.444+0000 7f613fe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6140139a40 con 0x7f6140143460 2024-03-23T03:49:50.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.444+0000 7f613f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6140143930 0x7f6140139090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:50.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.444+0000 7f613f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6140143930 0x7f6140139090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54450/0 (socket says 172.21.15.102:54450) 2024-03-23T03:49:50.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.444+0000 7f613f400640 1 -- 172.21.15.102:0/590304616 learned_addr learned my addr 172.21.15.102:0/590304616 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:50.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.444+0000 7f613ea00640 1 --2- 172.21.15.102:0/590304616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61400cdc50 0x7f614006cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:50.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.444+0000 7f6136000640 1 --2- 172.21.15.102:0/590304616 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6140143460 0x7f6140138b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:50.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.445+0000 7f613f400640 1 -- 172.21.15.102:0/590304616 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6140143460 msgr2=0x7f6140138b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:50.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.445+0000 7f613f400640 1 --2- 172.21.15.102:0/590304616 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6140143460 0x7f6140138b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.445+0000 7f613f400640 1 -- 172.21.15.102:0/590304616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61400cdc50 msgr2=0x7f614006cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:50.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.445+0000 7f613f400640 1 --2- 172.21.15.102:0/590304616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61400cdc50 0x7f614006cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.445+0000 7f613f400640 1 -- 172.21.15.102:0/590304616 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61280094d0 con 0x7f6140143930 2024-03-23T03:49:50.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.445+0000 7f613f400640 1 --2- 172.21.15.102:0/590304616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6140143930 0x7f6140139090 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f613800d800 tx=0x7f613800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:50.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.445+0000 7f6137e00640 1 -- 172.21.15.102:0/590304616 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61380096c0 con 0x7f6140143930 2024-03-23T03:49:50.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.445+0000 7f6137e00640 1 -- 172.21.15.102:0/590304616 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6138009840 con 0x7f6140143930 2024-03-23T03:49:50.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.445+0000 7f613fe00640 1 -- 172.21.15.102:0/590304616 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6140139d20 con 0x7f6140143930 2024-03-23T03:49:50.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.445+0000 7f6137e00640 1 -- 172.21.15.102:0/590304616 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138009bf0 con 0x7f6140143930 2024-03-23T03:49:50.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.445+0000 7f613fe00640 1 -- 172.21.15.102:0/590304616 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f614013a2e0 con 0x7f6140143930 2024-03-23T03:49:50.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.446+0000 7f613fe00640 1 -- 172.21.15.102:0/590304616 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f614013a470 con 0x7f6140143930 2024-03-23T03:49:50.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.447+0000 7f6137e00640 1 -- 172.21.15.102:0/590304616 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6138009d50 con 0x7f6140143930 2024-03-23T03:49:50.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.448+0000 7f6137e00640 1 --2- 172.21.15.102:0/590304616 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f611c06ac00 0x7f611c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.448+0000 7f6137e00640 1 -- 172.21.15.102:0/590304616 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7f6138089be0 con 0x7f6140143930 2024-03-23T03:49:50.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.449+0000 7f613ea00640 1 --2- 172.21.15.102:0/590304616 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f611c06ac00 0x7f611c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:50.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.449+0000 7f613ea00640 1 --2- 172.21.15.102:0/590304616 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f611c06ac00 0x7f611c06d0c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6130009a60 tx=0x7f6130003990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:50.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.453+0000 7f6137e00640 1 -- 172.21.15.102:0/590304616 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6138054c80 con 0x7f6140143930 2024-03-23T03:49:50.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.645+0000 7f613fe00640 1 -- 172.21.15.102:0/590304616 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f614004e030 con 0x7f6140143930 2024-03-23T03:49:50.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.646+0000 7f6137e00640 1 -- 172.21.15.102:0/590304616 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7f61380180a0 con 0x7f6140143930 2024-03-23T03:49:50.646 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:49:50.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.648+0000 7f613fe00640 1 -- 172.21.15.102:0/590304616 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f611c06ac00 msgr2=0x7f611c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:50.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.648+0000 7f613fe00640 1 --2- 172.21.15.102:0/590304616 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f611c06ac00 0x7f611c06d0c0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f6130009a60 tx=0x7f6130003990 comp rx=0 tx=0).stop 2024-03-23T03:49:50.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.648+0000 7f613fe00640 1 -- 172.21.15.102:0/590304616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6140143930 msgr2=0x7f6140139090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:50.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.648+0000 7f613fe00640 1 --2- 172.21.15.102:0/590304616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6140143930 0x7f6140139090 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f613800d800 tx=0x7f613800dcc0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.648+0000 7f613fe00640 1 -- 172.21.15.102:0/590304616 shutdown_connections 2024-03-23T03:49:50.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.648+0000 7f613fe00640 1 --2- 172.21.15.102:0/590304616 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f611c06ac00 0x7f611c06d0c0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.648+0000 7f613fe00640 1 --2- 172.21.15.102:0/590304616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6140143930 0x7f6140139090 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.648+0000 7f613fe00640 1 --2- 172.21.15.102:0/590304616 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6140143460 0x7f6140138b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.648+0000 7f613fe00640 1 --2- 172.21.15.102:0/590304616 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f61400cdc50 0x7f614006cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.648+0000 7f613fe00640 1 -- 172.21.15.102:0/590304616 >> 172.21.15.102:0/590304616 conn(0x7f6140065df0 msgr2=0x7f614013f150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:50.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.648+0000 7f613fe00640 1 -- 172.21.15.102:0/590304616 shutdown_connections 2024-03-23T03:49:50.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.648+0000 7f613fe00640 1 -- 172.21.15.102:0/590304616 wait complete. 2024-03-23T03:49:50.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:49:50.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.873+0000 7f0578200640 1 Processor -- start 2024-03-23T03:49:50.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.873+0000 7f0578200640 1 -- start start 2024-03-23T03:49:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.875+0000 7f0578200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05700cb660 0x7f05700cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.875+0000 7f0578200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f057006c2b0 0x7f057006c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.875+0000 7f0578200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057015ae60 con 0x7f05700ce930 2024-03-23T03:49:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.875+0000 7f0578200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f057015bae0 con 0x7f057006c2b0 2024-03-23T03:49:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.875+0000 7f0578200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057015b140 con 0x7f05700cb660 2024-03-23T03:49:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.875+0000 7f0576200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f057006c2b0 0x7f057006c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.875+0000 7f0575800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f05700ce930 0x7f05700ced10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42272/0 (socket says 172.21.15.102:42272) 2024-03-23T03:49:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.875+0000 7f0575800640 1 -- 172.21.15.102:0/535985134 learned_addr learned my addr 172.21.15.102:0/535985134 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:50.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.875+0000 7f0574e00640 1 --2- 172.21.15.102:0/535985134 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05700cb660 0x7f05700cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:50.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.876+0000 7f0576200640 1 -- 172.21.15.102:0/535985134 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05700cb660 msgr2=0x7f05700cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:50.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.876+0000 7f0576200640 1 --2- 172.21.15.102:0/535985134 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05700cb660 0x7f05700cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.876+0000 7f0576200640 1 -- 172.21.15.102:0/535985134 >> v1:172.21.15.102:6789/0 conn(0x7f05700ce930 legacy=0x7f05700ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:50.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.876+0000 7f0576200640 1 -- 172.21.15.102:0/535985134 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057015b420 con 0x7f057006c2b0 2024-03-23T03:49:50.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.876+0000 7f056fe00640 1 -- 172.21.15.102:0/535985134 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722798256 0 0) 0x7f057015ae60 con 0x7f05700ce930 2024-03-23T03:49:50.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.877+0000 7f0576200640 1 --2- 172.21.15.102:0/535985134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f057006c2b0 0x7f057006c740 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f0568009d90 tx=0x7f056802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f758835acad15928 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:50.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.877+0000 7f056fe00640 1 -- 172.21.15.102:0/535985134 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056802cc50 con 0x7f057006c2b0 2024-03-23T03:49:50.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.877+0000 7f056fe00640 1 -- 172.21.15.102:0/535985134 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056802cdb0 con 0x7f057006c2b0 2024-03-23T03:49:50.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.877+0000 7f056fe00640 1 -- 172.21.15.102:0/535985134 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0568034660 con 0x7f057006c2b0 2024-03-23T03:49:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.878+0000 7f0578200640 1 -- 172.21.15.102:0/535985134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f057006c2b0 msgr2=0x7f057006c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.878+0000 7f0578200640 1 --2- 172.21.15.102:0/535985134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f057006c2b0 0x7f057006c740 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f0568009d90 tx=0x7f056802bea0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.878+0000 7f0578200640 1 -- 172.21.15.102:0/535985134 shutdown_connections 2024-03-23T03:49:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.878+0000 7f0578200640 1 --2- 172.21.15.102:0/535985134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f057006c2b0 0x7f057006c740 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.878+0000 7f0578200640 1 --2- 172.21.15.102:0/535985134 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05700cb660 0x7f05700cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.878+0000 7f0578200640 1 -- 172.21.15.102:0/535985134 >> 172.21.15.102:0/535985134 conn(0x7f057006a0b0 msgr2=0x7f057006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.878+0000 7f0578200640 1 -- 172.21.15.102:0/535985134 shutdown_connections 2024-03-23T03:49:50.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.878+0000 7f0578200640 1 -- 172.21.15.102:0/535985134 wait complete. 2024-03-23T03:49:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.880+0000 7f0578200640 1 Processor -- start 2024-03-23T03:49:50.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.880+0000 7f0578200640 1 -- start start 2024-03-23T03:49:50.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.881+0000 7f0578200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f057006c2b0 0x7f057013e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.881+0000 7f0578200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05700cb660 0x7f0570144ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.881+0000 7f0578200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05700ce930 0x7f0570159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.881+0000 7f0578200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057013e8d0 con 0x7f057006c2b0 2024-03-23T03:49:50.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.881+0000 7f0578200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05701f6530 con 0x7f05700ce930 2024-03-23T03:49:50.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.881+0000 7f0578200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057014e6f0 con 0x7f05700cb660 2024-03-23T03:49:50.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.881+0000 7f0575800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f057006c2b0 0x7f057013e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:50.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.881+0000 7f0575800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f057006c2b0 0x7f057013e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54474/0 (socket says 172.21.15.102:54474) 2024-03-23T03:49:50.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.881+0000 7f0575800640 1 -- 172.21.15.102:0/2340309318 learned_addr learned my addr 172.21.15.102:0/2340309318 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:50.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.881+0000 7f0575800640 1 -- 172.21.15.102:0/2340309318 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05700cb660 msgr2=0x7f0570144ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:50.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.881+0000 7f0575800640 1 --2- 172.21.15.102:0/2340309318 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05700cb660 0x7f0570144ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.881+0000 7f0575800640 1 -- 172.21.15.102:0/2340309318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05700ce930 msgr2=0x7f0570159bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:50.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.881+0000 7f0575800640 1 --2- 172.21.15.102:0/2340309318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05700ce930 0x7f0570159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:50.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.881+0000 7f0575800640 1 -- 172.21.15.102:0/2340309318 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0568009a70 con 0x7f057006c2b0 2024-03-23T03:49:50.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.882+0000 7f0575800640 1 --2- 172.21.15.102:0/2340309318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f057006c2b0 0x7f057013e390 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f056400c500 tx=0x7f056400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:50.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.882+0000 7f056d600640 1 -- 172.21.15.102:0/2340309318 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564007560 con 0x7f057006c2b0 2024-03-23T03:49:50.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.882+0000 7f056d600640 1 -- 172.21.15.102:0/2340309318 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05640047e0 con 0x7f057006c2b0 2024-03-23T03:49:50.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.882+0000 7f0578200640 1 -- 172.21.15.102:0/2340309318 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0570155280 con 0x7f057006c2b0 2024-03-23T03:49:50.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.882+0000 7f056d600640 1 -- 172.21.15.102:0/2340309318 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564004b90 con 0x7f057006c2b0 2024-03-23T03:49:50.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.882+0000 7f0578200640 1 -- 172.21.15.102:0/2340309318 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05701f5c00 con 0x7f057006c2b0 2024-03-23T03:49:50.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.883+0000 7f0578200640 1 -- 172.21.15.102:0/2340309318 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0570073390 con 0x7f057006c2b0 2024-03-23T03:49:50.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.884+0000 7f056d600640 1 -- 172.21.15.102:0/2340309318 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0564004cf0 con 0x7f057006c2b0 2024-03-23T03:49:50.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.885+0000 7f056d600640 1 --2- 172.21.15.102:0/2340309318 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f055006ac00 0x7f055006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:50.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.885+0000 7f056d600640 1 -- 172.21.15.102:0/2340309318 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7f05640886c0 con 0x7f057006c2b0 2024-03-23T03:49:50.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.886+0000 7f0574e00640 1 --2- 172.21.15.102:0/2340309318 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f055006ac00 0x7f055006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:50.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.886+0000 7f0574e00640 1 --2- 172.21.15.102:0/2340309318 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f055006ac00 0x7f055006d0c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0570150130 tx=0x7f055c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:50.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:50.889+0000 7f056d600640 1 -- 172.21.15.102:0/2340309318 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f05640536c0 con 0x7f057006c2b0 2024-03-23T03:49:51.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.101+0000 7f0578200640 1 -- 172.21.15.102:0/2340309318 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f05700cdf40 con 0x7f057006c2b0 2024-03-23T03:49:51.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.102+0000 7f056d600640 1 -- 172.21.15.102:0/2340309318 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v69) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0564056ce0 con 0x7f057006c2b0 2024-03-23T03:49:51.103 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:49:51.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.105+0000 7f0578200640 1 -- 172.21.15.102:0/2340309318 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f055006ac00 msgr2=0x7f055006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:51.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.105+0000 7f0578200640 1 --2- 172.21.15.102:0/2340309318 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f055006ac00 0x7f055006d0c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f0570150130 tx=0x7f055c008040 comp rx=0 tx=0).stop 2024-03-23T03:49:51.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.105+0000 7f0578200640 1 -- 172.21.15.102:0/2340309318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f057006c2b0 msgr2=0x7f057013e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:51.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.105+0000 7f0578200640 1 --2- 172.21.15.102:0/2340309318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f057006c2b0 0x7f057013e390 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f056400c500 tx=0x7f056400c9c0 comp rx=0 tx=0).stop 2024-03-23T03:49:51.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.105+0000 7f0578200640 1 -- 172.21.15.102:0/2340309318 shutdown_connections 2024-03-23T03:49:51.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.105+0000 7f0578200640 1 --2- 172.21.15.102:0/2340309318 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f055006ac00 0x7f055006d0c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:51.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.105+0000 7f0578200640 1 --2- 172.21.15.102:0/2340309318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05700ce930 0x7f0570159bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:51.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.105+0000 7f0578200640 1 --2- 172.21.15.102:0/2340309318 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05700cb660 0x7f0570144ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:51.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.105+0000 7f0578200640 1 --2- 172.21.15.102:0/2340309318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f057006c2b0 0x7f057013e390 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:51.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.105+0000 7f0578200640 1 -- 172.21.15.102:0/2340309318 >> 172.21.15.102:0/2340309318 conn(0x7f057006a0b0 msgr2=0x7f0570152030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:51.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.105+0000 7f0578200640 1 -- 172.21.15.102:0/2340309318 shutdown_connections 2024-03-23T03:49:51.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.105+0000 7f0578200640 1 -- 172.21.15.102:0/2340309318 wait complete. 2024-03-23T03:49:51.125 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-23T03:49:51.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:51.125 DEBUG:teuthology.orchestra.run.smithi173:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-03-23T03:49:51.232 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:49:51.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:49:51.232 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T03:49:51.285 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:49:51.285 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:51.341 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:49:51.341 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:49:51.341 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:49:51.397 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:49:51.397 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:49:51.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.614+0000 7f22c3800640 1 Processor -- start 2024-03-23T03:49:51.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.615+0000 7f22c3800640 1 -- start start 2024-03-23T03:49:51.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.616+0000 7f22c3800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f22bc0b39b0 0x7f22bc0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:51.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.616+0000 7f22c3800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22bc15c400 0x7f22bc15c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:51.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.616+0000 7f22c3800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22bc06c6f0 con 0x7f22bc15b560 2024-03-23T03:49:51.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.616+0000 7f22c3800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22bc0b4370 con 0x7f22bc0b39b0 2024-03-23T03:49:51.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.616+0000 7f22c3800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc06c9d0 con 0x7f22bc15c400 2024-03-23T03:49:51.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.617+0000 7f22b3e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f22bc0b39b0 0x7f22bc0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:51.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.617+0000 7f22c1800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22bc15c400 0x7f22bc15c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:51.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.617+0000 7f22b3e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f22bc0b39b0 0x7f22bc0b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:54990/0 (socket says 172.21.15.102:54990) 2024-03-23T03:49:51.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.617+0000 7f22b3e00640 1 -- 172.21.15.102:0/4187808342 learned_addr learned my addr 172.21.15.102:0/4187808342 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:51.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.618+0000 7f22b3e00640 1 -- 172.21.15.102:0/4187808342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22bc15c400 msgr2=0x7f22bc15c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:51.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.618+0000 7f22b3e00640 1 --2- 172.21.15.102:0/4187808342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22bc15c400 0x7f22bc15c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:51.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.618+0000 7f22b3e00640 1 -- 172.21.15.102:0/4187808342 >> v1:172.21.15.173:6789/0 conn(0x7f22bc15b560 legacy=0x7f22bc15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:51.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.618+0000 7f22b3e00640 1 -- 172.21.15.102:0/4187808342 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc06ccb0 con 0x7f22bc0b39b0 2024-03-23T03:49:51.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.619+0000 7f22b3e00640 1 --2- 172.21.15.102:0/4187808342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22bc0b39b0 0x7f22bc0b3e30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f22ac009870 tx=0x7f22ac02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d56f8faf48c9c2a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:51.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.620+0000 7f22bbe00640 1 -- 172.21.15.102:0/4187808342 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ac030840 con 0x7f22bc0b39b0 2024-03-23T03:49:51.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.620+0000 7f22bbe00640 1 -- 172.21.15.102:0/4187808342 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22ac0309a0 con 0x7f22bc0b39b0 2024-03-23T03:49:51.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.620+0000 7f22bbe00640 1 -- 172.21.15.102:0/4187808342 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ac030d30 con 0x7f22bc0b39b0 2024-03-23T03:49:51.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.620+0000 7f22c3800640 1 -- 172.21.15.102:0/4187808342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22bc0b39b0 msgr2=0x7f22bc0b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:51.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.620+0000 7f22c3800640 1 --2- 172.21.15.102:0/4187808342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22bc0b39b0 0x7f22bc0b3e30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f22ac009870 tx=0x7f22ac02b980 comp rx=0 tx=0).stop 2024-03-23T03:49:51.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.620+0000 7f22c3800640 1 -- 172.21.15.102:0/4187808342 shutdown_connections 2024-03-23T03:49:51.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.620+0000 7f22c3800640 1 --2- 172.21.15.102:0/4187808342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22bc15c400 0x7f22bc15c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:51.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.620+0000 7f22c3800640 1 --2- 172.21.15.102:0/4187808342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22bc0b39b0 0x7f22bc0b3e30 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:51.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.620+0000 7f22c3800640 1 -- 172.21.15.102:0/4187808342 >> 172.21.15.102:0/4187808342 conn(0x7f22bc06a3d0 msgr2=0x7f22bc0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:51.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.620+0000 7f22c3800640 1 -- 172.21.15.102:0/4187808342 shutdown_connections 2024-03-23T03:49:51.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.620+0000 7f22c3800640 1 -- 172.21.15.102:0/4187808342 wait complete. 2024-03-23T03:49:51.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.623+0000 7f22c3800640 1 Processor -- start 2024-03-23T03:49:51.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.623+0000 7f22c3800640 1 -- start start 2024-03-23T03:49:51.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.624+0000 7f22c3800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22bc0b39b0 0x7f22bc14b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:51.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.624+0000 7f22c3800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22bc15b560 0x7f22bc1536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:51.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.624+0000 7f22c3800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22bc15c400 0x7f22bc0cc310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:51.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.624+0000 7f22c3800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc154d50 con 0x7f22bc15b560 2024-03-23T03:49:51.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.624+0000 7f22c3800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc154140 con 0x7f22bc0b39b0 2024-03-23T03:49:51.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.624+0000 7f22c3800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc154420 con 0x7f22bc15c400 2024-03-23T03:49:51.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.624+0000 7f22b3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22bc15b560 0x7f22bc1536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:51.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.624+0000 7f22b3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22bc15b560 0x7f22bc1536a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54488/0 (socket says 172.21.15.102:54488) 2024-03-23T03:49:51.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.624+0000 7f22b3e00640 1 -- 172.21.15.102:0/459299665 learned_addr learned my addr 172.21.15.102:0/459299665 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:51.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.625+0000 7f22c1800640 1 --2- 172.21.15.102:0/459299665 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22bc15c400 0x7f22bc0cc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:51.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.625+0000 7f22c0e00640 1 --2- 172.21.15.102:0/459299665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22bc0b39b0 0x7f22bc14b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:51.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.625+0000 7f22b3e00640 1 -- 172.21.15.102:0/459299665 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22bc15c400 msgr2=0x7f22bc0cc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:51.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.625+0000 7f22b3e00640 1 --2- 172.21.15.102:0/459299665 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22bc15c400 0x7f22bc0cc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:51.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.625+0000 7f22b3e00640 1 -- 172.21.15.102:0/459299665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22bc0b39b0 msgr2=0x7f22bc14b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:51.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.625+0000 7f22b3e00640 1 --2- 172.21.15.102:0/459299665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22bc0b39b0 0x7f22bc14b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:51.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.625+0000 7f22b3e00640 1 -- 172.21.15.102:0/459299665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ac0094d0 con 0x7f22bc15b560 2024-03-23T03:49:51.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.625+0000 7f22c0e00640 1 --2- 172.21.15.102:0/459299665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22bc0b39b0 0x7f22bc14b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:51.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.625+0000 7f22b3e00640 1 --2- 172.21.15.102:0/459299665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22bc15b560 0x7f22bc1536a0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f22ac000c00 tx=0x7f22ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:51.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.625+0000 7f22b9600640 1 -- 172.21.15.102:0/459299665 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ac039c90 con 0x7f22bc15b560 2024-03-23T03:49:51.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.626+0000 7f22b9600640 1 -- 172.21.15.102:0/459299665 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22ac039df0 con 0x7f22bc15b560 2024-03-23T03:49:51.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.626+0000 7f22b9600640 1 -- 172.21.15.102:0/459299665 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ac043640 con 0x7f22bc15b560 2024-03-23T03:49:51.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.626+0000 7f22c3800640 1 -- 172.21.15.102:0/459299665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22bc154700 con 0x7f22bc15b560 2024-03-23T03:49:51.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.626+0000 7f22c3800640 1 -- 172.21.15.102:0/459299665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22bc14b840 con 0x7f22bc15b560 2024-03-23T03:49:51.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.627+0000 7f22c3800640 1 -- 172.21.15.102:0/459299665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22bc14ba60 con 0x7f22bc15b560 2024-03-23T03:49:51.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.628+0000 7f22b9600640 1 -- 172.21.15.102:0/459299665 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f22ac0437a0 con 0x7f22bc15b560 2024-03-23T03:49:51.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.629+0000 7f22b9600640 1 --2- 172.21.15.102:0/459299665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f229806ac30 0x7f229806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:51.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.629+0000 7f22b9600640 1 -- 172.21.15.102:0/459299665 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7f22ac03e070 con 0x7f22bc15b560 2024-03-23T03:49:51.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.629+0000 7f22c0e00640 1 --2- 172.21.15.102:0/459299665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f229806ac30 0x7f229806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:51.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.630+0000 7f22c0e00640 1 --2- 172.21.15.102:0/459299665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f229806ac30 0x7f229806d0f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f22bc1f55e0 tx=0x7f22a4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:51.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.633+0000 7f22b9600640 1 -- 172.21.15.102:0/459299665 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f22ac0774d0 con 0x7f22bc15b560 2024-03-23T03:49:51.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.861+0000 7f22c3800640 1 -- 172.21.15.102:0/459299665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22bc0abd00 con 0x7f22bc15b560 2024-03-23T03:49:51.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.863+0000 7f22b9600640 1 -- 172.21.15.102:0/459299665 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4744 (secure 0 0 0) 0x7f22ac07aaf0 con 0x7f22bc15b560 2024-03-23T03:49:51.863 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:51.864 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5230,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3155961019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3155961019},{"type":"v1","addr":"172.21.15.173:6835","nonce":3155961019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6021,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/251450026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":251450026},{"type":"v1","addr":"172.21.15.102:6836","nonce":251450026}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6024,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3299889122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":3299889122},{"type":"v1","addr":"172.21.15.102:6837","nonce":3299889122}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:49:35.415835+0000","modified":"2024-03-23T03:49:38.434294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6030":{"gid":6030,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":4,"addr":"172.21.15.173:6833/3695568749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3695568749},{"type":"v1","addr":"172.21.15.173:6833","nonce":3695568749}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-23T03:49:51.864 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 33 2024-03-23T03:49:51.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.865+0000 7f22c3800640 1 -- 172.21.15.102:0/459299665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f229806ac30 msgr2=0x7f229806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:51.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.865+0000 7f22c3800640 1 --2- 172.21.15.102:0/459299665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f229806ac30 0x7f229806d0f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f22bc1f55e0 tx=0x7f22a4009040 comp rx=0 tx=0).stop 2024-03-23T03:49:51.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.865+0000 7f22c3800640 1 -- 172.21.15.102:0/459299665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22bc15b560 msgr2=0x7f22bc1536a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:51.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.865+0000 7f22c3800640 1 --2- 172.21.15.102:0/459299665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22bc15b560 0x7f22bc1536a0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f22ac000c00 tx=0x7f22ac004730 comp rx=0 tx=0).stop 2024-03-23T03:49:51.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.865+0000 7f22c3800640 1 -- 172.21.15.102:0/459299665 shutdown_connections 2024-03-23T03:49:51.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.865+0000 7f22c3800640 1 --2- 172.21.15.102:0/459299665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f229806ac30 0x7f229806d0f0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:51.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.865+0000 7f22c3800640 1 --2- 172.21.15.102:0/459299665 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22bc15c400 0x7f22bc0cc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:51.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.865+0000 7f22c3800640 1 --2- 172.21.15.102:0/459299665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22bc15b560 0x7f22bc1536a0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:51.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.865+0000 7f22c3800640 1 --2- 172.21.15.102:0/459299665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22bc0b39b0 0x7f22bc14b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:51.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.865+0000 7f22c3800640 1 -- 172.21.15.102:0/459299665 >> 172.21.15.102:0/459299665 conn(0x7f22bc06a3d0 msgr2=0x7f22bc0cb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:51.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.865+0000 7f22c3800640 1 -- 172.21.15.102:0/459299665 shutdown_connections 2024-03-23T03:49:51.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:51.865+0000 7f22c3800640 1 -- 172.21.15.102:0/459299665 wait complete. 2024-03-23T03:49:51.885 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:49:52.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.080+0000 7f6bbee00640 1 Processor -- start 2024-03-23T03:49:52.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.080+0000 7f6bbee00640 1 -- start start 2024-03-23T03:49:52.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.082+0000 7f6bbee00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6bb815cda0 0x7f6bb80cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.082+0000 7f6bbee00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f6bb80b4a60 0x7f6bb80b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.082+0000 7f6bbee00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb80d1830 con 0x7f6bb80b5a30 2024-03-23T03:49:52.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.082+0000 7f6bbee00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb80b1f20 con 0x7f6bb815cda0 2024-03-23T03:49:52.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.082+0000 7f6bbee00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb80d0c20 con 0x7f6bb80b4a60 2024-03-23T03:49:52.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.082+0000 7f6bb7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6bb815cda0 0x7f6bb80cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.082+0000 7f6bb7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6bb815cda0 0x7f6bb80cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54490/0 (socket says 172.21.15.102:54490) 2024-03-23T03:49:52.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.082+0000 7f6bb7e00640 1 -- 172.21.15.102:0/2327555120 learned_addr learned my addr 172.21.15.102:0/2327555120 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:52.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.082+0000 7f6bb7400640 1 --1- 172.21.15.102:0/2327555120 >> v1:172.21.15.102:6789/0 conn(0x7f6bb80b5a30 0x7f6bb80b4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42280/0 (socket says 172.21.15.102:42280) 2024-03-23T03:49:52.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.083+0000 7f6bbce00640 1 --2- 172.21.15.102:0/2327555120 >> v2:172.21.15.173:3300/0 conn(0x7f6bb80b4a60 0x7f6bb80b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.083+0000 7f6bb7e00640 1 -- 172.21.15.102:0/2327555120 >> v2:172.21.15.173:3300/0 conn(0x7f6bb80b4a60 msgr2=0x7f6bb80b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:52.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.083+0000 7f6bb7e00640 1 --2- 172.21.15.102:0/2327555120 >> v2:172.21.15.173:3300/0 conn(0x7f6bb80b4a60 0x7f6bb80b5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.083+0000 7f6bb7e00640 1 -- 172.21.15.102:0/2327555120 >> v1:172.21.15.102:6789/0 conn(0x7f6bb80b5a30 legacy=0x7f6bb80b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:52.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.083+0000 7f6bb7e00640 1 -- 172.21.15.102:0/2327555120 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb80d0f00 con 0x7f6bb815cda0 2024-03-23T03:49:52.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.083+0000 7f6bb6a00640 1 -- 172.21.15.102:0/2327555120 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959999107 0 0) 0x7f6bb80d1830 con 0x7f6bb80b5a30 2024-03-23T03:49:52.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.084+0000 7f6bb7e00640 1 --2- 172.21.15.102:0/2327555120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bb815cda0 0x7f6bb80cdb40 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6bb80b8460 tx=0x7f6ba802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dcff5c388e0b6e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:52.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.084+0000 7f6bb6a00640 1 -- 172.21.15.102:0/2327555120 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba802cb20 con 0x7f6bb815cda0 2024-03-23T03:49:52.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.084+0000 7f6bb6a00640 1 -- 172.21.15.102:0/2327555120 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba802cc80 con 0x7f6bb815cda0 2024-03-23T03:49:52.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.084+0000 7f6bb6a00640 1 -- 172.21.15.102:0/2327555120 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba80343f0 con 0x7f6bb815cda0 2024-03-23T03:49:52.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.085+0000 7f6bbee00640 1 -- 172.21.15.102:0/2327555120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bb815cda0 msgr2=0x7f6bb80cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:52.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.085+0000 7f6bbee00640 1 --2- 172.21.15.102:0/2327555120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bb815cda0 0x7f6bb80cdb40 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f6bb80b8460 tx=0x7f6ba802bf80 comp rx=0 tx=0).stop 2024-03-23T03:49:52.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.085+0000 7f6bbee00640 1 -- 172.21.15.102:0/2327555120 shutdown_connections 2024-03-23T03:49:52.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.085+0000 7f6bbee00640 1 --2- 172.21.15.102:0/2327555120 >> v2:172.21.15.173:3300/0 conn(0x7f6bb80b4a60 0x7f6bb80b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.085+0000 7f6bbee00640 1 --2- 172.21.15.102:0/2327555120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bb815cda0 0x7f6bb80cdb40 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.085+0000 7f6bbee00640 1 -- 172.21.15.102:0/2327555120 >> 172.21.15.102:0/2327555120 conn(0x7f6bb80cf350 msgr2=0x7f6bb80cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:52.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.085+0000 7f6bbee00640 1 -- 172.21.15.102:0/2327555120 shutdown_connections 2024-03-23T03:49:52.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.085+0000 7f6bbee00640 1 -- 172.21.15.102:0/2327555120 wait complete. 2024-03-23T03:49:52.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.087+0000 7f6bbee00640 1 Processor -- start 2024-03-23T03:49:52.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.087+0000 7f6bbee00640 1 -- start start 2024-03-23T03:49:52.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.088+0000 7f6bbee00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bb80b4a60 0x7f6bb81f5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.088+0000 7f6bbee00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6bb80b5a30 0x7f6bb8156b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.088+0000 7f6bbee00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bb815cda0 0x7f6bb80b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.088+0000 7f6bbee00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb815d240 con 0x7f6bb815cda0 2024-03-23T03:49:52.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.088+0000 7f6bbee00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb80cb6d0 con 0x7f6bb80b4a60 2024-03-23T03:49:52.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.088+0000 7f6bbee00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb815d420 con 0x7f6bb80b5a30 2024-03-23T03:49:52.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.088+0000 7f6bbce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bb815cda0 0x7f6bb80b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.089+0000 7f6bbce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bb815cda0 0x7f6bb80b3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54496/0 (socket says 172.21.15.102:54496) 2024-03-23T03:49:52.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.089+0000 7f6bbce00640 1 -- 172.21.15.102:0/230818545 learned_addr learned my addr 172.21.15.102:0/230818545 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:52.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.089+0000 7f6bb7e00640 1 --2- 172.21.15.102:0/230818545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bb80b4a60 0x7f6bb81f5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.089+0000 7f6bb7400640 1 --2- 172.21.15.102:0/230818545 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6bb80b5a30 0x7f6bb8156b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.089+0000 7f6bbce00640 1 -- 172.21.15.102:0/230818545 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6bb80b5a30 msgr2=0x7f6bb8156b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:52.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.089+0000 7f6bbce00640 1 --2- 172.21.15.102:0/230818545 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6bb80b5a30 0x7f6bb8156b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.089+0000 7f6bbce00640 1 -- 172.21.15.102:0/230818545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bb80b4a60 msgr2=0x7f6bb81f5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:52.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.089+0000 7f6bbce00640 1 --2- 172.21.15.102:0/230818545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bb80b4a60 0x7f6bb81f5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.089+0000 7f6bbce00640 1 -- 172.21.15.102:0/230818545 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba8009b30 con 0x7f6bb815cda0 2024-03-23T03:49:52.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.089+0000 7f6bbce00640 1 --2- 172.21.15.102:0/230818545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bb815cda0 0x7f6bb80b3b20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000c5c0 tx=0x7f6bb000ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:52.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.090+0000 7f6bafe00640 1 -- 172.21.15.102:0/230818545 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0004220 con 0x7f6bb815cda0 2024-03-23T03:49:52.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.090+0000 7f6bafe00640 1 -- 172.21.15.102:0/230818545 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb00043a0 con 0x7f6bb815cda0 2024-03-23T03:49:52.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.090+0000 7f6bafe00640 1 -- 172.21.15.102:0/230818545 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bb0003850 con 0x7f6bb815cda0 2024-03-23T03:49:52.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.090+0000 7f6bbee00640 1 -- 172.21.15.102:0/230818545 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb815d700 con 0x7f6bb815cda0 2024-03-23T03:49:52.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.090+0000 7f6bbee00640 1 -- 172.21.15.102:0/230818545 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb81570a0 con 0x7f6bb815cda0 2024-03-23T03:49:52.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.091+0000 7f6bbee00640 1 -- 172.21.15.102:0/230818545 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb815dc20 con 0x7f6bb815cda0 2024-03-23T03:49:52.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.092+0000 7f6bafe00640 1 -- 172.21.15.102:0/230818545 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6bb00039b0 con 0x7f6bb815cda0 2024-03-23T03:49:52.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.093+0000 7f6bafe00640 1 --2- 172.21.15.102:0/230818545 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6b9406ac30 0x7f6b9406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.093+0000 7f6bafe00640 1 -- 172.21.15.102:0/230818545 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7f6bb00892f0 con 0x7f6bb815cda0 2024-03-23T03:49:52.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.093+0000 7f6bb7e00640 1 --2- 172.21.15.102:0/230818545 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6b9406ac30 0x7f6b9406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.094+0000 7f6bb7e00640 1 --2- 172.21.15.102:0/230818545 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6b9406ac30 0x7f6b9406d0f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8035000 tx=0x7f6ba8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:52.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.097+0000 7f6bafe00640 1 -- 172.21.15.102:0/230818545 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6bb815dc20 con 0x7f6bb815cda0 2024-03-23T03:49:52.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.289+0000 7f6bbee00640 1 -- 172.21.15.102:0/230818545 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bb815dc20 con 0x7f6bb815cda0 2024-03-23T03:49:52.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.292+0000 7f6bafe00640 1 -- 172.21.15.102:0/230818545 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16536 (secure 0 0 0) 0x7f6bb815dc20 con 0x7f6bb815cda0 2024-03-23T03:49:52.293 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:52.293 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":69,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:49:43.895428+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:49:33.007757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-23T03:49:34.449411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.e","mappings":[{"from":7,"to":2}]},{"pgid":"10.23","mappings":[{"from":7,"to":3}]},{"pgid":"10.2e","mappings":[{"from":4,"to":3}]},{"pgid":"10.2f","mappings":[{"from":4,"to":5}]},{"pgid":"10.3c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:49:52.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.294+0000 7f6bbee00640 1 -- 172.21.15.102:0/230818545 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6b9406ac30 msgr2=0x7f6b9406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:52.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.294+0000 7f6bbee00640 1 --2- 172.21.15.102:0/230818545 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6b9406ac30 0x7f6b9406d0f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f6ba8035000 tx=0x7f6ba8030000 comp rx=0 tx=0).stop 2024-03-23T03:49:52.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.294+0000 7f6bbee00640 1 -- 172.21.15.102:0/230818545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bb815cda0 msgr2=0x7f6bb80b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:52.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.294+0000 7f6bbee00640 1 --2- 172.21.15.102:0/230818545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bb815cda0 0x7f6bb80b3b20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6bb000c5c0 tx=0x7f6bb000ca80 comp rx=0 tx=0).stop 2024-03-23T03:49:52.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.295+0000 7f6bbee00640 1 -- 172.21.15.102:0/230818545 shutdown_connections 2024-03-23T03:49:52.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.295+0000 7f6bbee00640 1 --2- 172.21.15.102:0/230818545 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6b9406ac30 0x7f6b9406d0f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.295+0000 7f6bbee00640 1 --2- 172.21.15.102:0/230818545 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6bb815cda0 0x7f6bb80b3b20 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.295+0000 7f6bbee00640 1 --2- 172.21.15.102:0/230818545 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6bb80b5a30 0x7f6bb8156b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.295+0000 7f6bbee00640 1 --2- 172.21.15.102:0/230818545 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6bb80b4a60 0x7f6bb81f5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.295+0000 7f6bbee00640 1 -- 172.21.15.102:0/230818545 >> 172.21.15.102:0/230818545 conn(0x7f6bb80cf350 msgr2=0x7f6bb80cf730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:52.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.295+0000 7f6bbee00640 1 -- 172.21.15.102:0/230818545 shutdown_connections 2024-03-23T03:49:52.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.295+0000 7f6bbee00640 1 -- 172.21.15.102:0/230818545 wait complete. 2024-03-23T03:49:52.313 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:49:33.007757+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:49:34.449411+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:49:52.313 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:49:52.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.513+0000 7f5832000640 1 Processor -- start 2024-03-23T03:49:52.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.513+0000 7f5832000640 1 -- start start 2024-03-23T03:49:52.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.514+0000 7f5832000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f582c159830 0x7f582c0cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.514+0000 7f5832000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f582c0ca7b0 0x7f582c0cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.514+0000 7f5832000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f582c159c10 con 0x7f582c0ce930 2024-03-23T03:49:52.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.514+0000 7f5832000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f582c06de50 con 0x7f582c159830 2024-03-23T03:49:52.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.514+0000 7f5832000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582c0cd840 con 0x7f582c0ca7b0 2024-03-23T03:49:52.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.514+0000 7f582b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f582c159830 0x7f582c0cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.515+0000 7f582b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f582c159830 0x7f582c0cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54506/0 (socket says 172.21.15.102:54506) 2024-03-23T03:49:52.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.515+0000 7f582b400640 1 -- 172.21.15.102:0/2699379958 learned_addr learned my addr 172.21.15.102:0/2699379958 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:52.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.515+0000 7f5822a00640 1 --2- 172.21.15.102:0/2699379958 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f582c0ca7b0 0x7f582c0cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.515+0000 7f582b400640 1 -- 172.21.15.102:0/2699379958 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f582c0ca7b0 msgr2=0x7f582c0cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:52.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.515+0000 7f582b400640 1 --2- 172.21.15.102:0/2699379958 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f582c0ca7b0 0x7f582c0cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.515+0000 7f582b400640 1 -- 172.21.15.102:0/2699379958 >> v1:172.21.15.173:6789/0 conn(0x7f582c0ce930 legacy=0x7f582c0b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:52.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.515+0000 7f582b400640 1 -- 172.21.15.102:0/2699379958 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c0ccc30 con 0x7f582c159830 2024-03-23T03:49:52.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.516+0000 7f582b400640 1 --2- 172.21.15.102:0/2699379958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f582c159830 0x7f582c0cc3c0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f581c009870 tx=0x7f581c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55fa467d6c6bf791 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:52.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.517+0000 7f582aa00640 1 -- 172.21.15.102:0/2699379958 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c030840 con 0x7f582c159830 2024-03-23T03:49:52.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.517+0000 7f582aa00640 1 -- 172.21.15.102:0/2699379958 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c0309a0 con 0x7f582c159830 2024-03-23T03:49:52.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.517+0000 7f582aa00640 1 -- 172.21.15.102:0/2699379958 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c030d30 con 0x7f582c159830 2024-03-23T03:49:52.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.517+0000 7f5832000640 1 -- 172.21.15.102:0/2699379958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f582c159830 msgr2=0x7f582c0cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:52.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.517+0000 7f5832000640 1 --2- 172.21.15.102:0/2699379958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f582c159830 0x7f582c0cc3c0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f581c009870 tx=0x7f581c02b980 comp rx=0 tx=0).stop 2024-03-23T03:49:52.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.517+0000 7f5832000640 1 -- 172.21.15.102:0/2699379958 shutdown_connections 2024-03-23T03:49:52.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.517+0000 7f5832000640 1 --2- 172.21.15.102:0/2699379958 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f582c0ca7b0 0x7f582c0cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.517+0000 7f5832000640 1 --2- 172.21.15.102:0/2699379958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f582c159830 0x7f582c0cc3c0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.517+0000 7f5832000640 1 -- 172.21.15.102:0/2699379958 >> 172.21.15.102:0/2699379958 conn(0x7f582c06a0b0 msgr2=0x7f582c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:52.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.517+0000 7f5832000640 1 -- 172.21.15.102:0/2699379958 shutdown_connections 2024-03-23T03:49:52.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.518+0000 7f5832000640 1 -- 172.21.15.102:0/2699379958 wait complete. 2024-03-23T03:49:52.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.519+0000 7f5832000640 1 Processor -- start 2024-03-23T03:49:52.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.519+0000 7f5832000640 1 -- start start 2024-03-23T03:49:52.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.520+0000 7f5832000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f582c0ca7b0 0x7f582c14e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.520+0000 7f5832000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f582c0ce930 0x7f582c1431a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.520+0000 7f5832000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f582c159830 0x7f582c0b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.520+0000 7f5832000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c1436e0 con 0x7f582c0ca7b0 2024-03-23T03:49:52.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.520+0000 7f5832000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582c14d160 con 0x7f582c0ce930 2024-03-23T03:49:52.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.520+0000 7f5832000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f582c14d440 con 0x7f582c159830 2024-03-23T03:49:52.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.520+0000 7f582b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f582c0ca7b0 0x7f582c14e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.520+0000 7f582b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f582c0ca7b0 0x7f582c14e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:54520/0 (socket says 172.21.15.102:54520) 2024-03-23T03:49:52.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.520+0000 7f582b400640 1 -- 172.21.15.102:0/1293145951 learned_addr learned my addr 172.21.15.102:0/1293145951 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:52.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.521+0000 7f582b400640 1 -- 172.21.15.102:0/1293145951 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f582c159830 msgr2=0x7f582c0b23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:52.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.521+0000 7f582be00640 1 --2- 172.21.15.102:0/1293145951 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f582c159830 0x7f582c0b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.521+0000 7f5822a00640 1 --2- 172.21.15.102:0/1293145951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f582c0ce930 0x7f582c1431a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.521+0000 7f582b400640 1 --2- 172.21.15.102:0/1293145951 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f582c159830 0x7f582c0b23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.521+0000 7f582b400640 1 -- 172.21.15.102:0/1293145951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f582c0ce930 msgr2=0x7f582c1431a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:52.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.521+0000 7f582b400640 1 --2- 172.21.15.102:0/1293145951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f582c0ce930 0x7f582c1431a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.521+0000 7f582b400640 1 -- 172.21.15.102:0/1293145951 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581c0094d0 con 0x7f582c0ca7b0 2024-03-23T03:49:52.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.521+0000 7f582be00640 1 --2- 172.21.15.102:0/1293145951 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f582c159830 0x7f582c0b23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:52.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.521+0000 7f582b400640 1 --2- 172.21.15.102:0/1293145951 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f582c0ca7b0 0x7f582c14e200 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f581c009870 tx=0x7f581c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:52.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.521+0000 7f5823e00640 1 -- 172.21.15.102:0/1293145951 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c039c90 con 0x7f582c0ca7b0 2024-03-23T03:49:52.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.521+0000 7f5823e00640 1 -- 172.21.15.102:0/1293145951 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c039df0 con 0x7f582c0ca7b0 2024-03-23T03:49:52.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.521+0000 7f5823e00640 1 -- 172.21.15.102:0/1293145951 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c043620 con 0x7f582c0ca7b0 2024-03-23T03:49:52.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.521+0000 7f5832000640 1 -- 172.21.15.102:0/1293145951 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f582c14d720 con 0x7f582c0ca7b0 2024-03-23T03:49:52.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.522+0000 7f5832000640 1 -- 172.21.15.102:0/1293145951 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f582c14dce0 con 0x7f582c0ca7b0 2024-03-23T03:49:52.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.523+0000 7f5832000640 1 -- 172.21.15.102:0/1293145951 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f582c14ded0 con 0x7f582c0ca7b0 2024-03-23T03:49:52.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.523+0000 7f5823e00640 1 -- 172.21.15.102:0/1293145951 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f581c043780 con 0x7f582c0ca7b0 2024-03-23T03:49:52.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.524+0000 7f5823e00640 1 --2- 172.21.15.102:0/1293145951 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f580806ac00 0x7f580806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.524+0000 7f5823e00640 1 -- 172.21.15.102:0/1293145951 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7f581c03e070 con 0x7f582c0ca7b0 2024-03-23T03:49:52.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.525+0000 7f5822a00640 1 --2- 172.21.15.102:0/1293145951 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f580806ac00 0x7f580806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.525+0000 7f5822a00640 1 --2- 172.21.15.102:0/1293145951 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f580806ac00 0x7f580806d0c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f582c0b34f0 tx=0x7f581400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:52.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.529+0000 7f5823e00640 1 -- 172.21.15.102:0/1293145951 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f581c077630 con 0x7f582c0ca7b0 2024-03-23T03:49:52.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.724+0000 7f5832000640 1 -- 172.21.15.102:0/1293145951 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f582c073390 con 0x7f582c0ca7b0 2024-03-23T03:49:52.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.724+0000 7f5823e00640 1 -- 172.21.15.102:0/1293145951 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v69) v1 ==== 93+0+10 (secure 0 0 0) 0x7f581c07ac50 con 0x7f582c0ca7b0 2024-03-23T03:49:52.725 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:49:52.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.727+0000 7f5832000640 1 -- 172.21.15.102:0/1293145951 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f580806ac00 msgr2=0x7f580806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:52.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.727+0000 7f5832000640 1 --2- 172.21.15.102:0/1293145951 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f580806ac00 0x7f580806d0c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f582c0b34f0 tx=0x7f581400b040 comp rx=0 tx=0).stop 2024-03-23T03:49:52.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.727+0000 7f5832000640 1 -- 172.21.15.102:0/1293145951 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f582c0ca7b0 msgr2=0x7f582c14e200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:52.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.727+0000 7f5832000640 1 --2- 172.21.15.102:0/1293145951 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f582c0ca7b0 0x7f582c14e200 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f581c009870 tx=0x7f581c004730 comp rx=0 tx=0).stop 2024-03-23T03:49:52.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.727+0000 7f5832000640 1 -- 172.21.15.102:0/1293145951 shutdown_connections 2024-03-23T03:49:52.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.727+0000 7f5832000640 1 --2- 172.21.15.102:0/1293145951 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f580806ac00 0x7f580806d0c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.727+0000 7f5832000640 1 --2- 172.21.15.102:0/1293145951 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f582c159830 0x7f582c0b23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.727+0000 7f5832000640 1 --2- 172.21.15.102:0/1293145951 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f582c0ce930 0x7f582c1431a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.727+0000 7f5832000640 1 --2- 172.21.15.102:0/1293145951 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f582c0ca7b0 0x7f582c14e200 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.727+0000 7f5832000640 1 -- 172.21.15.102:0/1293145951 >> 172.21.15.102:0/1293145951 conn(0x7f582c06a0b0 msgr2=0x7f582c158180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:52.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.727+0000 7f5832000640 1 -- 172.21.15.102:0/1293145951 shutdown_connections 2024-03-23T03:49:52.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.727+0000 7f5832000640 1 -- 172.21.15.102:0/1293145951 wait complete. 2024-03-23T03:49:52.745 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:49:52.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.949+0000 7f3c7e600640 1 Processor -- start 2024-03-23T03:49:52.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.950+0000 7f3c7e600640 1 -- start start 2024-03-23T03:49:52.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.951+0000 7f3c7e600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3c78160fc0 0x7f3c7806b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.951+0000 7f3c7e600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f3c780c3d70 0x7f3c780ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.951+0000 7f3c7e600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c78099bc0 con 0x7f3c78099400 2024-03-23T03:49:52.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.951+0000 7f3c7e600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c780ce400 con 0x7f3c78160fc0 2024-03-23T03:49:52.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.951+0000 7f3c7e600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c780ce6e0 con 0x7f3c780c3d70 2024-03-23T03:49:52.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.952+0000 7f3c77400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3c78160fc0 0x7f3c7806b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.952+0000 7f3c77400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3c78160fc0 0x7f3c7806b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40100/0 (socket says 172.21.15.102:40100) 2024-03-23T03:49:52.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.952+0000 7f3c77400640 1 -- 172.21.15.102:0/2939847414 learned_addr learned my addr 172.21.15.102:0/2939847414 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:52.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.952+0000 7f3c76a00640 1 --1- 172.21.15.102:0/2939847414 >> v1:172.21.15.102:6789/0 conn(0x7f3c78099400 0x7f3c780c3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47454/0 (socket says 172.21.15.102:47454) 2024-03-23T03:49:52.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.952+0000 7f3c77400640 1 -- 172.21.15.102:0/2939847414 >> v2:172.21.15.173:3300/0 conn(0x7f3c780c3d70 msgr2=0x7f3c780ceee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:52.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.952+0000 7f3c77e00640 1 --2- 172.21.15.102:0/2939847414 >> v2:172.21.15.173:3300/0 conn(0x7f3c780c3d70 0x7f3c780ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.952+0000 7f3c77400640 1 --2- 172.21.15.102:0/2939847414 >> v2:172.21.15.173:3300/0 conn(0x7f3c780c3d70 0x7f3c780ceee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.952+0000 7f3c77400640 1 -- 172.21.15.102:0/2939847414 >> v1:172.21.15.102:6789/0 conn(0x7f3c78099400 legacy=0x7f3c780c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:52.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.952+0000 7f3c77400640 1 -- 172.21.15.102:0/2939847414 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c780ce9c0 con 0x7f3c78160fc0 2024-03-23T03:49:52.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.952+0000 7f3c77e00640 1 --2- 172.21.15.102:0/2939847414 >> v2:172.21.15.173:3300/0 conn(0x7f3c780c3d70 0x7f3c780ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:52.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.953+0000 7f3c77400640 1 --2- 172.21.15.102:0/2939847414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c78160fc0 0x7f3c7806b260 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f3c780b2d30 tx=0x7f3c6802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595dd8605adb4fa9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:52.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.954+0000 7f3c76000640 1 -- 172.21.15.102:0/2939847414 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68030860 con 0x7f3c78160fc0 2024-03-23T03:49:52.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.954+0000 7f3c76000640 1 -- 172.21.15.102:0/2939847414 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c680309c0 con 0x7f3c78160fc0 2024-03-23T03:49:52.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.954+0000 7f3c76000640 1 -- 172.21.15.102:0/2939847414 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68030d50 con 0x7f3c78160fc0 2024-03-23T03:49:52.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.954+0000 7f3c7e600640 1 -- 172.21.15.102:0/2939847414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c78160fc0 msgr2=0x7f3c7806b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:52.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.954+0000 7f3c7e600640 1 --2- 172.21.15.102:0/2939847414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c78160fc0 0x7f3c7806b260 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f3c780b2d30 tx=0x7f3c6802b9a0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.954+0000 7f3c7e600640 1 -- 172.21.15.102:0/2939847414 shutdown_connections 2024-03-23T03:49:52.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.954+0000 7f3c7e600640 1 --2- 172.21.15.102:0/2939847414 >> v2:172.21.15.173:3300/0 conn(0x7f3c780c3d70 0x7f3c780ceee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.954+0000 7f3c7e600640 1 --2- 172.21.15.102:0/2939847414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c78160fc0 0x7f3c7806b260 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.954+0000 7f3c7e600640 1 -- 172.21.15.102:0/2939847414 >> 172.21.15.102:0/2939847414 conn(0x7f3c780b2180 msgr2=0x7f3c780b2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:52.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.954+0000 7f3c7e600640 1 -- 172.21.15.102:0/2939847414 shutdown_connections 2024-03-23T03:49:52.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.954+0000 7f3c7e600640 1 -- 172.21.15.102:0/2939847414 wait complete. 2024-03-23T03:49:52.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.956+0000 7f3c7e600640 1 Processor -- start 2024-03-23T03:49:52.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.956+0000 7f3c7e600640 1 -- start start 2024-03-23T03:49:52.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.957+0000 7f3c7e600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3c78099400 0x7f3c780c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.957+0000 7f3c7e600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c780c3d70 0x7f3c7815c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.957+0000 7f3c7e600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c78160fc0 0x7f3c780c9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.957+0000 7f3c7e600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c780c7920 con 0x7f3c780c3d70 2024-03-23T03:49:52.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.957+0000 7f3c7e600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c780ca5d0 con 0x7f3c78160fc0 2024-03-23T03:49:52.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.957+0000 7f3c7e600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c780c5a00 con 0x7f3c78099400 2024-03-23T03:49:52.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.957+0000 7f3c76a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c780c3d70 0x7f3c7815c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.958+0000 7f3c76a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c780c3d70 0x7f3c7815c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40106/0 (socket says 172.21.15.102:40106) 2024-03-23T03:49:52.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.958+0000 7f3c76a00640 1 -- 172.21.15.102:0/4184360284 learned_addr learned my addr 172.21.15.102:0/4184360284 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:52.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.958+0000 7f3c77400640 1 --2- 172.21.15.102:0/4184360284 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3c78099400 0x7f3c780c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.958+0000 7f3c77e00640 1 --2- 172.21.15.102:0/4184360284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c78160fc0 0x7f3c780c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.958+0000 7f3c76a00640 1 -- 172.21.15.102:0/4184360284 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3c78099400 msgr2=0x7f3c780c54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:52.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.958+0000 7f3c76a00640 1 --2- 172.21.15.102:0/4184360284 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3c78099400 0x7f3c780c54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.958+0000 7f3c76a00640 1 -- 172.21.15.102:0/4184360284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c78160fc0 msgr2=0x7f3c780c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:52.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.958+0000 7f3c76a00640 1 --2- 172.21.15.102:0/4184360284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c78160fc0 0x7f3c780c9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:52.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.958+0000 7f3c76a00640 1 -- 172.21.15.102:0/4184360284 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c680094d0 con 0x7f3c780c3d70 2024-03-23T03:49:52.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.958+0000 7f3c76a00640 1 --2- 172.21.15.102:0/4184360284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c780c3d70 0x7f3c7815c780 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f3c6000c800 tx=0x7f3c6000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:52.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.958+0000 7f3c6f400640 1 -- 172.21.15.102:0/4184360284 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6000a640 con 0x7f3c780c3d70 2024-03-23T03:49:52.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.959+0000 7f3c6f400640 1 -- 172.21.15.102:0/4184360284 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6000a7a0 con 0x7f3c780c3d70 2024-03-23T03:49:52.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.959+0000 7f3c6f400640 1 -- 172.21.15.102:0/4184360284 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6000ab50 con 0x7f3c780c3d70 2024-03-23T03:49:52.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.959+0000 7f3c7e600640 1 -- 172.21.15.102:0/4184360284 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c780c9d30 con 0x7f3c780c3d70 2024-03-23T03:49:52.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.959+0000 7f3c7e600640 1 -- 172.21.15.102:0/4184360284 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c780ca2f0 con 0x7f3c780c3d70 2024-03-23T03:49:52.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.960+0000 7f3c7e600640 1 -- 172.21.15.102:0/4184360284 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c780c7a90 con 0x7f3c780c3d70 2024-03-23T03:49:52.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.961+0000 7f3c6f400640 1 -- 172.21.15.102:0/4184360284 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3c6000acb0 con 0x7f3c780c3d70 2024-03-23T03:49:52.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.962+0000 7f3c6f400640 1 --2- 172.21.15.102:0/4184360284 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3c5406ac00 0x7f3c5406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:52.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.962+0000 7f3c6f400640 1 -- 172.21.15.102:0/4184360284 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7f3c60088cc0 con 0x7f3c780c3d70 2024-03-23T03:49:52.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.962+0000 7f3c77400640 1 --2- 172.21.15.102:0/4184360284 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3c5406ac00 0x7f3c5406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:52.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.963+0000 7f3c77400640 1 --2- 172.21.15.102:0/4184360284 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3c5406ac00 0x7f3c5406d0c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3c68036000 tx=0x7f3c6803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:52.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:52.966+0000 7f3c6f400640 1 -- 172.21.15.102:0/4184360284 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3c6008f080 con 0x7f3c780c3d70 2024-03-23T03:49:53.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.160+0000 7f3c7e600640 1 -- 172.21.15.102:0/4184360284 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3c78066bb0 con 0x7f3c780c3d70 2024-03-23T03:49:53.161 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.161+0000 7f3c6f400640 1 -- 172.21.15.102:0/4184360284 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3c600170a0 con 0x7f3c780c3d70 2024-03-23T03:49:53.161 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:49:53.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.163+0000 7f3c7e600640 1 -- 172.21.15.102:0/4184360284 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3c5406ac00 msgr2=0x7f3c5406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:53.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.163+0000 7f3c7e600640 1 --2- 172.21.15.102:0/4184360284 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3c5406ac00 0x7f3c5406d0c0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f3c68036000 tx=0x7f3c6803b040 comp rx=0 tx=0).stop 2024-03-23T03:49:53.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.163+0000 7f3c7e600640 1 -- 172.21.15.102:0/4184360284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c780c3d70 msgr2=0x7f3c7815c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:53.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.163+0000 7f3c7e600640 1 --2- 172.21.15.102:0/4184360284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c780c3d70 0x7f3c7815c780 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f3c6000c800 tx=0x7f3c6000ccc0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.163+0000 7f3c7e600640 1 -- 172.21.15.102:0/4184360284 shutdown_connections 2024-03-23T03:49:53.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.163+0000 7f3c7e600640 1 --2- 172.21.15.102:0/4184360284 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3c5406ac00 0x7f3c5406d0c0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.163+0000 7f3c7e600640 1 --2- 172.21.15.102:0/4184360284 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c78160fc0 0x7f3c780c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.163+0000 7f3c7e600640 1 --2- 172.21.15.102:0/4184360284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c780c3d70 0x7f3c7815c780 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.163+0000 7f3c7e600640 1 --2- 172.21.15.102:0/4184360284 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3c78099400 0x7f3c780c54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.163+0000 7f3c7e600640 1 -- 172.21.15.102:0/4184360284 >> 172.21.15.102:0/4184360284 conn(0x7f3c780b2180 msgr2=0x7f3c780cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:53.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.164+0000 7f3c7e600640 1 -- 172.21.15.102:0/4184360284 shutdown_connections 2024-03-23T03:49:53.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.164+0000 7f3c7e600640 1 -- 172.21.15.102:0/4184360284 wait complete. 2024-03-23T03:49:53.184 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:49:53.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.393+0000 7fadf8c00640 1 Processor -- start 2024-03-23T03:49:53.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.394+0000 7fadf8c00640 1 -- start start 2024-03-23T03:49:53.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.395+0000 7fadf8c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fadf40cd740 0x7fadf40b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:53.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.395+0000 7fadf8c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fadf40b5aa0 0x7fadf40b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:53.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.395+0000 7fadf8c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf40d20c0 con 0x7fadf40d0d40 2024-03-23T03:49:53.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.395+0000 7fadf8c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf40d14b0 con 0x7fadf40b5aa0 2024-03-23T03:49:53.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.395+0000 7fadf8c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf40d1790 con 0x7fadf40cd740 2024-03-23T03:49:53.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.395+0000 7fadf2800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fadf40b5aa0 0x7fadf40b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:53.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.395+0000 7fadf1400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fadf40d0d40 0x7fadf40cdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47460/0 (socket says 172.21.15.102:47460) 2024-03-23T03:49:53.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.395+0000 7fadf1400640 1 -- 172.21.15.102:0/551764575 learned_addr learned my addr 172.21.15.102:0/551764575 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:53.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.396+0000 7fadf1e00640 1 --2- 172.21.15.102:0/551764575 >> v2:172.21.15.173:3300/0 conn(0x7fadf40cd740 0x7fadf40b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:53.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.396+0000 7fadf2800640 1 -- 172.21.15.102:0/551764575 >> v2:172.21.15.173:3300/0 conn(0x7fadf40cd740 msgr2=0x7fadf40b84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:53.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.396+0000 7fadf2800640 1 --2- 172.21.15.102:0/551764575 >> v2:172.21.15.173:3300/0 conn(0x7fadf40cd740 0x7fadf40b84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.396+0000 7fadf2800640 1 -- 172.21.15.102:0/551764575 >> v1:172.21.15.102:6789/0 conn(0x7fadf40d0d40 legacy=0x7fadf40cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:53.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.396+0000 7fadf2800640 1 -- 172.21.15.102:0/551764575 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf40d1a70 con 0x7fadf40b5aa0 2024-03-23T03:49:53.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.396+0000 7fadf0a00640 1 -- 172.21.15.102:0/551764575 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216766621 0 0) 0x7fadf40d20c0 con 0x7fadf40d0d40 2024-03-23T03:49:53.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.397+0000 7fadf2800640 1 --2- 172.21.15.102:0/551764575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadf40b5aa0 0x7fadf40b5f30 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fadf408a100 tx=0x7fadec02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21194ed184a6a24f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:53.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.397+0000 7fadf0a00640 1 -- 172.21.15.102:0/551764575 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec02cc80 con 0x7fadf40b5aa0 2024-03-23T03:49:53.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.397+0000 7fadf0a00640 1 -- 172.21.15.102:0/551764575 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadec02cde0 con 0x7fadf40b5aa0 2024-03-23T03:49:53.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.397+0000 7fadf0a00640 1 -- 172.21.15.102:0/551764575 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadec034660 con 0x7fadf40b5aa0 2024-03-23T03:49:53.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.398+0000 7fadf8c00640 1 -- 172.21.15.102:0/551764575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadf40b5aa0 msgr2=0x7fadf40b5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:53.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.398+0000 7fadf8c00640 1 --2- 172.21.15.102:0/551764575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadf40b5aa0 0x7fadf40b5f30 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fadf408a100 tx=0x7fadec02bed0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.398+0000 7fadf8c00640 1 -- 172.21.15.102:0/551764575 shutdown_connections 2024-03-23T03:49:53.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.398+0000 7fadf8c00640 1 --2- 172.21.15.102:0/551764575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadf40b5aa0 0x7fadf40b5f30 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.398+0000 7fadf8c00640 1 --2- 172.21.15.102:0/551764575 >> v2:172.21.15.173:3300/0 conn(0x7fadf40cd740 0x7fadf40b84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.398+0000 7fadf8c00640 1 -- 172.21.15.102:0/551764575 >> 172.21.15.102:0/551764575 conn(0x7fadf40cf5d0 msgr2=0x7fadf40cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:53.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.398+0000 7fadf8c00640 1 -- 172.21.15.102:0/551764575 shutdown_connections 2024-03-23T03:49:53.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.398+0000 7fadf8c00640 1 -- 172.21.15.102:0/551764575 wait complete. 2024-03-23T03:49:53.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.400+0000 7fadf8c00640 1 Processor -- start 2024-03-23T03:49:53.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.400+0000 7fadf8c00640 1 -- start start 2024-03-23T03:49:53.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.401+0000 7fadf8c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadf40b5aa0 0x7fadf40c8040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:53.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.401+0000 7fadf8c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadf40cd740 0x7fadf415ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:53.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.401+0000 7fadf8c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fadf40d0d40 0x7fadf415ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:53.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.401+0000 7fadf8c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf40cac30 con 0x7fadf40b5aa0 2024-03-23T03:49:53.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.401+0000 7fadf8c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf40caf10 con 0x7fadf40cd740 2024-03-23T03:49:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.401+0000 7fadf8c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf40cb1f0 con 0x7fadf40d0d40 2024-03-23T03:49:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.401+0000 7fadf1e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadf40b5aa0 0x7fadf40c8040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.401+0000 7fadf1e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadf40b5aa0 0x7fadf40c8040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40114/0 (socket says 172.21.15.102:40114) 2024-03-23T03:49:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.401+0000 7fadf1e00640 1 -- 172.21.15.102:0/1859523984 learned_addr learned my addr 172.21.15.102:0/1859523984 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.401+0000 7fadf1e00640 1 -- 172.21.15.102:0/1859523984 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fadf40d0d40 msgr2=0x7fadf415ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.402+0000 7fadf1400640 1 --2- 172.21.15.102:0/1859523984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadf40cd740 0x7fadf415ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.402+0000 7fadf1e00640 1 --2- 172.21.15.102:0/1859523984 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fadf40d0d40 0x7fadf415ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.402+0000 7fadf1e00640 1 -- 172.21.15.102:0/1859523984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadf40cd740 msgr2=0x7fadf415ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.402+0000 7fadf1e00640 1 --2- 172.21.15.102:0/1859523984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadf40cd740 0x7fadf415ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.402+0000 7fadf1e00640 1 -- 172.21.15.102:0/1859523984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadec009a70 con 0x7fadf40b5aa0 2024-03-23T03:49:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.402+0000 7fadf1400640 1 --2- 172.21.15.102:0/1859523984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadf40cd740 0x7fadf415ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:53.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.402+0000 7fadf1e00640 1 --2- 172.21.15.102:0/1859523984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadf40b5aa0 0x7fadf40c8040 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fade400d800 tx=0x7fade400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:53.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.402+0000 7fadea000640 1 -- 172.21.15.102:0/1859523984 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade400b6c0 con 0x7fadf40b5aa0 2024-03-23T03:49:53.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.402+0000 7fadea000640 1 -- 172.21.15.102:0/1859523984 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade400b820 con 0x7fadf40b5aa0 2024-03-23T03:49:53.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.402+0000 7fadea000640 1 -- 172.21.15.102:0/1859523984 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade400bbb0 con 0x7fadf40b5aa0 2024-03-23T03:49:53.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.402+0000 7fadf8c00640 1 -- 172.21.15.102:0/1859523984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadf40cb4d0 con 0x7fadf40b5aa0 2024-03-23T03:49:53.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.402+0000 7fadf8c00640 1 -- 172.21.15.102:0/1859523984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf40c8580 con 0x7fadf40b5aa0 2024-03-23T03:49:53.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.403+0000 7fadf8c00640 1 -- 172.21.15.102:0/1859523984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf40b8170 con 0x7fadf40b5aa0 2024-03-23T03:49:53.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.404+0000 7fadea000640 1 -- 172.21.15.102:0/1859523984 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fade400bd10 con 0x7fadf40b5aa0 2024-03-23T03:49:53.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.405+0000 7fadea000640 1 --2- 172.21.15.102:0/1859523984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fadd006ac00 0x7fadd006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:53.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.405+0000 7fadea000640 1 -- 172.21.15.102:0/1859523984 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7fade4089b70 con 0x7fadf40b5aa0 2024-03-23T03:49:53.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.406+0000 7fadf1400640 1 --2- 172.21.15.102:0/1859523984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fadd006ac00 0x7fadd006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:53.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.406+0000 7fadf1400640 1 --2- 172.21.15.102:0/1859523984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fadd006ac00 0x7fadd006d0c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fadf415bca0 tx=0x7faddc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:53.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.410+0000 7fadea000640 1 -- 172.21.15.102:0/1859523984 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fade4054ae0 con 0x7fadf40b5aa0 2024-03-23T03:49:53.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.602+0000 7fadf8c00640 1 -- 172.21.15.102:0/1859523984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fadf4073390 con 0x7fadf40b5aa0 2024-03-23T03:49:53.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.603+0000 7fadea000640 1 -- 172.21.15.102:0/1859523984 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7fade4058100 con 0x7fadf40b5aa0 2024-03-23T03:49:53.604 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:49:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.606+0000 7fadf8c00640 1 -- 172.21.15.102:0/1859523984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fadd006ac00 msgr2=0x7fadd006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.606+0000 7fadf8c00640 1 --2- 172.21.15.102:0/1859523984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fadd006ac00 0x7fadd006d0c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fadf415bca0 tx=0x7faddc009040 comp rx=0 tx=0).stop 2024-03-23T03:49:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.606+0000 7fadf8c00640 1 -- 172.21.15.102:0/1859523984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadf40b5aa0 msgr2=0x7fadf40c8040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.606+0000 7fadf8c00640 1 --2- 172.21.15.102:0/1859523984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadf40b5aa0 0x7fadf40c8040 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fade400d800 tx=0x7fade400dcc0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.606+0000 7fadf8c00640 1 -- 172.21.15.102:0/1859523984 shutdown_connections 2024-03-23T03:49:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.606+0000 7fadf8c00640 1 --2- 172.21.15.102:0/1859523984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fadd006ac00 0x7fadd006d0c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.606+0000 7fadf8c00640 1 --2- 172.21.15.102:0/1859523984 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fadf40d0d40 0x7fadf415ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.606+0000 7fadf8c00640 1 --2- 172.21.15.102:0/1859523984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fadf40cd740 0x7fadf415ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.606+0000 7fadf8c00640 1 --2- 172.21.15.102:0/1859523984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fadf40b5aa0 0x7fadf40c8040 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.606+0000 7fadf8c00640 1 -- 172.21.15.102:0/1859523984 >> 172.21.15.102:0/1859523984 conn(0x7fadf40cf5d0 msgr2=0x7fadf40b42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.606+0000 7fadf8c00640 1 -- 172.21.15.102:0/1859523984 shutdown_connections 2024-03-23T03:49:53.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.606+0000 7fadf8c00640 1 -- 172.21.15.102:0/1859523984 wait complete. 2024-03-23T03:49:53.626 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:49:53.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.836+0000 7f5f6fc00640 1 Processor -- start 2024-03-23T03:49:53.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.837+0000 7f5f6fc00640 1 -- start start 2024-03-23T03:49:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.838+0000 7f5f6fc00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f5f6815cda0 0x7f5f680cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.838+0000 7f5f6fc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5f680b5a30 0x7f5f680b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.838+0000 7f5f6fc00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f680d1830 con 0x7f5f680b4350 2024-03-23T03:49:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.838+0000 7f5f6fc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f680b1f20 con 0x7f5f680b5a30 2024-03-23T03:49:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.838+0000 7f5f6fc00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f680d0c20 con 0x7f5f6815cda0 2024-03-23T03:49:53.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.838+0000 7f5f67e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5f680b5a30 0x7f5f680b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:53.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.838+0000 7f5f67e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5f680b5a30 0x7f5f680b5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40118/0 (socket says 172.21.15.102:40118) 2024-03-23T03:49:53.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.838+0000 7f5f67e00640 1 -- 172.21.15.102:0/752358321 learned_addr learned my addr 172.21.15.102:0/752358321 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:53.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.839+0000 7f5f6d200640 1 --2- 172.21.15.102:0/752358321 >> v2:172.21.15.173:3300/0 conn(0x7f5f6815cda0 0x7f5f680cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:53.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.839+0000 7f5f67e00640 1 -- 172.21.15.102:0/752358321 >> v2:172.21.15.173:3300/0 conn(0x7f5f6815cda0 msgr2=0x7f5f680cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:53.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.839+0000 7f5f67e00640 1 --2- 172.21.15.102:0/752358321 >> v2:172.21.15.173:3300/0 conn(0x7f5f6815cda0 0x7f5f680cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.839+0000 7f5f67e00640 1 -- 172.21.15.102:0/752358321 >> v1:172.21.15.173:6789/0 conn(0x7f5f680b4350 legacy=0x7f5f680d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:53.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.839+0000 7f5f67e00640 1 -- 172.21.15.102:0/752358321 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f680d0f00 con 0x7f5f680b5a30 2024-03-23T03:49:53.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.840+0000 7f5f67e00640 1 --2- 172.21.15.102:0/752358321 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f680b5a30 0x7f5f680b5e90 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f5f6806ad20 tx=0x7f5f5002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47cdb14d0330f782 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:53.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.840+0000 7f5f67400640 1 -- 172.21.15.102:0/752358321 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f50031870 con 0x7f5f680b5a30 2024-03-23T03:49:53.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.840+0000 7f5f67400640 1 -- 172.21.15.102:0/752358321 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f500319d0 con 0x7f5f680b5a30 2024-03-23T03:49:53.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.840+0000 7f5f67400640 1 -- 172.21.15.102:0/752358321 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f50031d60 con 0x7f5f680b5a30 2024-03-23T03:49:53.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.841+0000 7f5f6fc00640 1 -- 172.21.15.102:0/752358321 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f680b5a30 msgr2=0x7f5f680b5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:53.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.841+0000 7f5f6fc00640 1 --2- 172.21.15.102:0/752358321 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f680b5a30 0x7f5f680b5e90 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f5f6806ad20 tx=0x7f5f5002b910 comp rx=0 tx=0).stop 2024-03-23T03:49:53.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.841+0000 7f5f6fc00640 1 -- 172.21.15.102:0/752358321 shutdown_connections 2024-03-23T03:49:53.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.841+0000 7f5f6fc00640 1 --2- 172.21.15.102:0/752358321 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f680b5a30 0x7f5f680b5e90 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.841+0000 7f5f6fc00640 1 --2- 172.21.15.102:0/752358321 >> v2:172.21.15.173:3300/0 conn(0x7f5f6815cda0 0x7f5f680cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.841+0000 7f5f6fc00640 1 -- 172.21.15.102:0/752358321 >> 172.21.15.102:0/752358321 conn(0x7f5f680cf350 msgr2=0x7f5f680cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:53.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.841+0000 7f5f6fc00640 1 -- 172.21.15.102:0/752358321 shutdown_connections 2024-03-23T03:49:53.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.841+0000 7f5f6fc00640 1 -- 172.21.15.102:0/752358321 wait complete. 2024-03-23T03:49:53.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.842+0000 7f5f6fc00640 1 Processor -- start 2024-03-23T03:49:53.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.843+0000 7f5f6fc00640 1 -- start start 2024-03-23T03:49:53.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.843+0000 7f5f6fc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5f680b4350 0x7f5f6815f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:53.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.843+0000 7f5f6fc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f680b5a30 0x7f5f6814a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:53.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.843+0000 7f5f6fc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f6815cda0 0x7f5f6814ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:53.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.843+0000 7f5f6fc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6815db50 con 0x7f5f6815cda0 2024-03-23T03:49:53.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.843+0000 7f5f6fc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6815d240 con 0x7f5f680b5a30 2024-03-23T03:49:53.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.843+0000 7f5f6fc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f680c8ee0 con 0x7f5f680b4350 2024-03-23T03:49:53.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.844+0000 7f5f6dc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f6815cda0 0x7f5f6814ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:53.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.844+0000 7f5f6dc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f6815cda0 0x7f5f6814ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40120/0 (socket says 172.21.15.102:40120) 2024-03-23T03:49:53.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.844+0000 7f5f6dc00640 1 -- 172.21.15.102:0/2076283246 learned_addr learned my addr 172.21.15.102:0/2076283246 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.844+0000 7f5f6dc00640 1 -- 172.21.15.102:0/2076283246 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5f680b4350 msgr2=0x7f5f6815f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.844+0000 7f5f6dc00640 1 --2- 172.21.15.102:0/2076283246 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5f680b4350 0x7f5f6815f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.844+0000 7f5f6dc00640 1 -- 172.21.15.102:0/2076283246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f680b5a30 msgr2=0x7f5f6814a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.844+0000 7f5f67e00640 1 --2- 172.21.15.102:0/2076283246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f680b5a30 0x7f5f6814a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.844+0000 7f5f6dc00640 1 --2- 172.21.15.102:0/2076283246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f680b5a30 0x7f5f6814a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.844+0000 7f5f6dc00640 1 -- 172.21.15.102:0/2076283246 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f500094d0 con 0x7f5f6815cda0 2024-03-23T03:49:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.845+0000 7f5f6dc00640 1 --2- 172.21.15.102:0/2076283246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f6815cda0 0x7f5f6814ae00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5f6000c780 tx=0x7f5f6000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:53.846 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.845+0000 7f5f64c00640 1 -- 172.21.15.102:0/2076283246 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f60003d80 con 0x7f5f6815cda0 2024-03-23T03:49:53.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.845+0000 7f5f64c00640 1 -- 172.21.15.102:0/2076283246 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6000ce90 con 0x7f5f6815cda0 2024-03-23T03:49:53.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.845+0000 7f5f64c00640 1 -- 172.21.15.102:0/2076283246 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f600052b0 con 0x7f5f6815cda0 2024-03-23T03:49:53.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.845+0000 7f5f6fc00640 1 -- 172.21.15.102:0/2076283246 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6815d490 con 0x7f5f6815cda0 2024-03-23T03:49:53.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.845+0000 7f5f6fc00640 1 -- 172.21.15.102:0/2076283246 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6815fd70 con 0x7f5f6815cda0 2024-03-23T03:49:53.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.846+0000 7f5f6fc00640 1 -- 172.21.15.102:0/2076283246 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f6815ff60 con 0x7f5f6815cda0 2024-03-23T03:49:53.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.847+0000 7f5f64c00640 1 -- 172.21.15.102:0/2076283246 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5f60005410 con 0x7f5f6815cda0 2024-03-23T03:49:53.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.848+0000 7f5f64c00640 1 --2- 172.21.15.102:0/2076283246 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5f4406ac00 0x7f5f4406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:53.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.848+0000 7f5f64c00640 1 -- 172.21.15.102:0/2076283246 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7f5f60088a40 con 0x7f5f6815cda0 2024-03-23T03:49:53.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.848+0000 7f5f6d200640 1 --2- 172.21.15.102:0/2076283246 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5f4406ac00 0x7f5f4406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:53.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.849+0000 7f5f6d200640 1 --2- 172.21.15.102:0/2076283246 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5f4406ac00 0x7f5f4406d0c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5f68160210 tx=0x7f5f58008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:53.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:53.853+0000 7f5f64c00640 1 -- 172.21.15.102:0/2076283246 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5f60053a40 con 0x7f5f6815cda0 2024-03-23T03:49:54.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.081+0000 7f5f6fc00640 1 -- 172.21.15.102:0/2076283246 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f6815dc90 con 0x7f5f6815cda0 2024-03-23T03:49:54.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.082+0000 7f5f64c00640 1 -- 172.21.15.102:0/2076283246 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4744 (secure 0 0 0) 0x7f5f6815dc90 con 0x7f5f6815cda0 2024-03-23T03:49:54.085 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:54.086 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5230,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3155961019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3155961019},{"type":"v1","addr":"172.21.15.173:6835","nonce":3155961019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6021,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/251450026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":251450026},{"type":"v1","addr":"172.21.15.102:6836","nonce":251450026}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6024,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3299889122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":3299889122},{"type":"v1","addr":"172.21.15.102:6837","nonce":3299889122}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:49:35.415835+0000","modified":"2024-03-23T03:49:38.434294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6030":{"gid":6030,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":4,"addr":"172.21.15.173:6833/3695568749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3695568749},{"type":"v1","addr":"172.21.15.173:6833","nonce":3695568749}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-23T03:49:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 33 2024-03-23T03:49:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.085+0000 7f5f6fc00640 1 -- 172.21.15.102:0/2076283246 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5f4406ac00 msgr2=0x7f5f4406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.085+0000 7f5f6fc00640 1 --2- 172.21.15.102:0/2076283246 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5f4406ac00 0x7f5f4406d0c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f5f68160210 tx=0x7f5f58008040 comp rx=0 tx=0).stop 2024-03-23T03:49:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.085+0000 7f5f6fc00640 1 -- 172.21.15.102:0/2076283246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f6815cda0 msgr2=0x7f5f6814ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.085+0000 7f5f6fc00640 1 --2- 172.21.15.102:0/2076283246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f6815cda0 0x7f5f6814ae00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f5f6000c780 tx=0x7f5f6000cc40 comp rx=0 tx=0).stop 2024-03-23T03:49:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.085+0000 7f5f6fc00640 1 -- 172.21.15.102:0/2076283246 shutdown_connections 2024-03-23T03:49:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.085+0000 7f5f6fc00640 1 --2- 172.21.15.102:0/2076283246 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5f4406ac00 0x7f5f4406d0c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.085+0000 7f5f6fc00640 1 --2- 172.21.15.102:0/2076283246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f6815cda0 0x7f5f6814ae00 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.085+0000 7f5f6fc00640 1 --2- 172.21.15.102:0/2076283246 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5f680b5a30 0x7f5f6814a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.085+0000 7f5f6fc00640 1 --2- 172.21.15.102:0/2076283246 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5f680b4350 0x7f5f6815f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.085+0000 7f5f6fc00640 1 -- 172.21.15.102:0/2076283246 >> 172.21.15.102:0/2076283246 conn(0x7f5f680cf350 msgr2=0x7f5f680cb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.085+0000 7f5f6fc00640 1 -- 172.21.15.102:0/2076283246 shutdown_connections 2024-03-23T03:49:54.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.085+0000 7f5f6fc00640 1 -- 172.21.15.102:0/2076283246 wait complete. 2024-03-23T03:49:54.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:49:54.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.306+0000 7fd0e5600640 1 Processor -- start 2024-03-23T03:49:54.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.306+0000 7fd0e5600640 1 -- start start 2024-03-23T03:49:54.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.308+0000 7fd0e5600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd0e015b560 0x7fd0e015b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:54.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.308+0000 7fd0e5600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e015f320 con 0x7fd0e00b39b0 2024-03-23T03:49:54.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.308+0000 7fd0e5600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0e006c6f0 con 0x7fd0e015d410 2024-03-23T03:49:54.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.308+0000 7fd0e5600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0e006c9d0 con 0x7fd0e015b560 2024-03-23T03:49:54.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.308+0000 7fd0dea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd0e015b560 0x7fd0e015b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:54.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.308+0000 7fd0dea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd0e015b560 0x7fd0e015b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40122/0 (socket says 172.21.15.102:40122) 2024-03-23T03:49:54.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.308+0000 7fd0dea00640 1 -- 172.21.15.102:0/3116110519 learned_addr learned my addr 172.21.15.102:0/3116110519 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:54.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.309+0000 7fd0dea00640 1 -- 172.21.15.102:0/3116110519 >> v1:172.21.15.173:6789/0 conn(0x7fd0e015d410 legacy=0x7fd0e015ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:54.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.309+0000 7fd0dea00640 1 -- 172.21.15.102:0/3116110519 >> v1:172.21.15.102:6789/0 conn(0x7fd0e00b39b0 legacy=0x7fd0e015e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:54.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.309+0000 7fd0dea00640 1 -- 172.21.15.102:0/3116110519 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e006ccb0 con 0x7fd0e015b560 2024-03-23T03:49:54.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.310+0000 7fd0dea00640 1 --2- 172.21.15.102:0/3116110519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0e015b560 0x7fd0e015b940 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0009870 tx=0x7fd0d002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383cf3d07262e8c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.310+0000 7fd0de000640 1 -- 172.21.15.102:0/3116110519 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0031870 con 0x7fd0e015b560 2024-03-23T03:49:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.310+0000 7fd0de000640 1 -- 172.21.15.102:0/3116110519 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d00319d0 con 0x7fd0e015b560 2024-03-23T03:49:54.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.310+0000 7fd0de000640 1 -- 172.21.15.102:0/3116110519 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0031d60 con 0x7fd0e015b560 2024-03-23T03:49:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.311+0000 7fd0e5600640 1 -- 172.21.15.102:0/3116110519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0e015b560 msgr2=0x7fd0e015b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.311+0000 7fd0e5600640 1 --2- 172.21.15.102:0/3116110519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0e015b560 0x7fd0e015b940 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0009870 tx=0x7fd0d002b910 comp rx=0 tx=0).stop 2024-03-23T03:49:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.311+0000 7fd0e5600640 1 -- 172.21.15.102:0/3116110519 shutdown_connections 2024-03-23T03:49:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.311+0000 7fd0e5600640 1 --2- 172.21.15.102:0/3116110519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0e015b560 0x7fd0e015b940 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.311+0000 7fd0e5600640 1 -- 172.21.15.102:0/3116110519 >> 172.21.15.102:0/3116110519 conn(0x7fd0e006a3d0 msgr2=0x7fd0e00b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.311+0000 7fd0e5600640 1 -- 172.21.15.102:0/3116110519 shutdown_connections 2024-03-23T03:49:54.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.311+0000 7fd0e5600640 1 -- 172.21.15.102:0/3116110519 wait complete. 2024-03-23T03:49:54.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.313+0000 7fd0e5600640 1 Processor -- start 2024-03-23T03:49:54.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.314+0000 7fd0e5600640 1 -- start start 2024-03-23T03:49:54.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.315+0000 7fd0e5600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0e00b39b0 0x7fd0e01f60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.315+0000 7fd0e5600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0e015b560 0x7fd0e00abd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.315+0000 7fd0e5600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0e015d410 0x7fd0e0152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.315+0000 7fd0e5600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e01f65f0 con 0x7fd0e015d410 2024-03-23T03:49:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.315+0000 7fd0e5600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e013a1b0 con 0x7fd0e00b39b0 2024-03-23T03:49:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.315+0000 7fd0e5600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e00cbdb0 con 0x7fd0e015b560 2024-03-23T03:49:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.315+0000 7fd0df400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0e015d410 0x7fd0e0152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.315+0000 7fd0df400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0e015d410 0x7fd0e0152ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40138/0 (socket says 172.21.15.102:40138) 2024-03-23T03:49:54.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.315+0000 7fd0df400640 1 -- 172.21.15.102:0/1998850725 learned_addr learned my addr 172.21.15.102:0/1998850725 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.315+0000 7fd0dea00640 1 --2- 172.21.15.102:0/1998850725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0e00b39b0 0x7fd0e01f60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.315+0000 7fd0d6000640 1 --2- 172.21.15.102:0/1998850725 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0e015b560 0x7fd0e00abd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.316+0000 7fd0df400640 1 -- 172.21.15.102:0/1998850725 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0e015b560 msgr2=0x7fd0e00abd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:54.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.316+0000 7fd0df400640 1 --2- 172.21.15.102:0/1998850725 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0e015b560 0x7fd0e00abd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.316+0000 7fd0df400640 1 -- 172.21.15.102:0/1998850725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0e00b39b0 msgr2=0x7fd0e01f60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.316+0000 7fd0df400640 1 --2- 172.21.15.102:0/1998850725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0e00b39b0 0x7fd0e01f60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.316+0000 7fd0df400640 1 -- 172.21.15.102:0/1998850725 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d00094d0 con 0x7fd0e015d410 2024-03-23T03:49:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.316+0000 7fd0df400640 1 --2- 172.21.15.102:0/1998850725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0e015d410 0x7fd0e0152ce0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800b990 tx=0x7fd0d800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.316+0000 7fd0d7400640 1 -- 172.21.15.102:0/1998850725 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d800cba0 con 0x7fd0e015d410 2024-03-23T03:49:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.316+0000 7fd0d7400640 1 -- 172.21.15.102:0/1998850725 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0d800cd00 con 0x7fd0e015d410 2024-03-23T03:49:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.316+0000 7fd0e5600640 1 -- 172.21.15.102:0/1998850725 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e00cc090 con 0x7fd0e015d410 2024-03-23T03:49:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.316+0000 7fd0d7400640 1 -- 172.21.15.102:0/1998850725 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8014680 con 0x7fd0e015d410 2024-03-23T03:49:54.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.316+0000 7fd0e5600640 1 -- 172.21.15.102:0/1998850725 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e013a490 con 0x7fd0e015d410 2024-03-23T03:49:54.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.317+0000 7fd0e5600640 1 -- 172.21.15.102:0/1998850725 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e0152a00 con 0x7fd0e015d410 2024-03-23T03:49:54.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.318+0000 7fd0d7400640 1 -- 172.21.15.102:0/1998850725 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd0d801d3b0 con 0x7fd0e015d410 2024-03-23T03:49:54.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.319+0000 7fd0d7400640 1 --2- 172.21.15.102:0/1998850725 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd0bc06ac30 0x7fd0bc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:54.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.319+0000 7fd0d7400640 1 -- 172.21.15.102:0/1998850725 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7fd0d80894a0 con 0x7fd0e015d410 2024-03-23T03:49:54.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.320+0000 7fd0dea00640 1 --2- 172.21.15.102:0/1998850725 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd0bc06ac30 0x7fd0bc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:54.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.320+0000 7fd0dea00640 1 --2- 172.21.15.102:0/1998850725 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd0bc06ac30 0x7fd0bc06d0f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0036000 tx=0x7fd0d003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:54.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.324+0000 7fd0d7400640 1 -- 172.21.15.102:0/1998850725 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd0d80544a0 con 0x7fd0e015d410 2024-03-23T03:49:54.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.518+0000 7fd0e5600640 1 -- 172.21.15.102:0/1998850725 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0e006ac20 con 0x7fd0e015d410 2024-03-23T03:49:54.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.520+0000 7fd0d7400640 1 -- 172.21.15.102:0/1998850725 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16536 (secure 0 0 0) 0x7fd0d8057ac0 con 0x7fd0e015d410 2024-03-23T03:49:54.521 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:54.521 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":69,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:49:43.895428+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:49:33.007757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-23T03:49:34.449411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.e","mappings":[{"from":7,"to":2}]},{"pgid":"10.23","mappings":[{"from":7,"to":3}]},{"pgid":"10.2e","mappings":[{"from":4,"to":3}]},{"pgid":"10.2f","mappings":[{"from":4,"to":5}]},{"pgid":"10.3c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:49:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.523+0000 7fd0e5600640 1 -- 172.21.15.102:0/1998850725 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd0bc06ac30 msgr2=0x7fd0bc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:54.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.523+0000 7fd0e5600640 1 --2- 172.21.15.102:0/1998850725 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd0bc06ac30 0x7fd0bc06d0f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd0d0036000 tx=0x7fd0d003c040 comp rx=0 tx=0).stop 2024-03-23T03:49:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.523+0000 7fd0e5600640 1 -- 172.21.15.102:0/1998850725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0e015d410 msgr2=0x7fd0e0152ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.523+0000 7fd0e5600640 1 --2- 172.21.15.102:0/1998850725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0e015d410 0x7fd0e0152ce0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800b990 tx=0x7fd0d800be50 comp rx=0 tx=0).stop 2024-03-23T03:49:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.523+0000 7fd0e5600640 1 -- 172.21.15.102:0/1998850725 shutdown_connections 2024-03-23T03:49:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.523+0000 7fd0e5600640 1 --2- 172.21.15.102:0/1998850725 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd0bc06ac30 0x7fd0bc06d0f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.523+0000 7fd0e5600640 1 --2- 172.21.15.102:0/1998850725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0e015d410 0x7fd0e0152ce0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.523+0000 7fd0e5600640 1 --2- 172.21.15.102:0/1998850725 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0e015b560 0x7fd0e00abd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.523+0000 7fd0e5600640 1 --2- 172.21.15.102:0/1998850725 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0e00b39b0 0x7fd0e01f60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.523+0000 7fd0e5600640 1 -- 172.21.15.102:0/1998850725 >> 172.21.15.102:0/1998850725 conn(0x7fd0e006a3d0 msgr2=0x7fd0e015c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.523+0000 7fd0e5600640 1 -- 172.21.15.102:0/1998850725 shutdown_connections 2024-03-23T03:49:54.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.523+0000 7fd0e5600640 1 -- 172.21.15.102:0/1998850725 wait complete. 2024-03-23T03:49:54.543 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T03:49:54.543 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:49:54.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.739+0000 7fe842e00640 1 Processor -- start 2024-03-23T03:49:54.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.739+0000 7fe842e00640 1 -- start start 2024-03-23T03:49:54.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.740+0000 7fe842e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe83c0cde20 0x7fe83c165360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:54.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.740+0000 7fe842e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe83c15b2b0 con 0x7fe83c143930 2024-03-23T03:49:54.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.740+0000 7fe842e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe83c0cc6b0 con 0x7fe83c143460 2024-03-23T03:49:54.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.740+0000 7fe842e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c0cc950 con 0x7fe83c0cde20 2024-03-23T03:49:54.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.741+0000 7fe841000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe83c143930 0x7fe83c0cd710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47466/0 (socket says 172.21.15.102:47466) 2024-03-23T03:49:54.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.741+0000 7fe841000640 1 -- 172.21.15.102:0/582160753 learned_addr learned my addr 172.21.15.102:0/582160753 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.741+0000 7fe83be00640 1 -- 172.21.15.102:0/582160753 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 765886987 0 0) 0x7fe83c15b2b0 con 0x7fe83c143930 2024-03-23T03:49:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.741+0000 7fe842400640 1 --2- 172.21.15.102:0/582160753 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe83c0cde20 0x7fe83c165360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.741+0000 7fe83be00640 1 -- 172.21.15.102:0/582160753 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe824003520 con 0x7fe83c143930 2024-03-23T03:49:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.741+0000 7fe83be00640 1 -- 172.21.15.102:0/582160753 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 391391305 0 0) 0x7fe824003520 con 0x7fe83c143930 2024-03-23T03:49:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.741+0000 7fe83be00640 1 -- 172.21.15.102:0/582160753 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe83c0cde20 msgr2=0x7fe83c165360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.741+0000 7fe83be00640 1 --2- 172.21.15.102:0/582160753 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe83c0cde20 0x7fe83c165360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.741+0000 7fe83be00640 1 -- 172.21.15.102:0/582160753 >> v1:172.21.15.173:6789/0 conn(0x7fe83c143460 legacy=0x7fe83c15aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.741+0000 7fe83be00640 1 -- 172.21.15.102:0/582160753 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83c0ccc30 con 0x7fe83c143930 2024-03-23T03:49:54.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.741+0000 7fe83be00640 1 -- 172.21.15.102:0/582160753 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fe828001670 con 0x7fe83c143930 2024-03-23T03:49:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.742+0000 7fe83be00640 1 -- 172.21.15.102:0/582160753 >> v1:172.21.15.102:6789/0 conn(0x7fe83c143930 legacy=0x7fe83c0cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.742+0000 7fe83be00640 1 --2- 172.21.15.102:0/582160753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8240039a0 0x7fe824003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.742+0000 7fe83be00640 1 --2- 172.21.15.102:0/582160753 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe83c0cde20 0x7fe8240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.742+0000 7fe83be00640 1 --2- 172.21.15.102:0/582160753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe824004d00 0x7fe824005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.742+0000 7fe83be00640 1 -- 172.21.15.102:0/582160753 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c0cc950 con 0x7fe8240039a0 2024-03-23T03:49:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.742+0000 7fe83be00640 1 -- 172.21.15.102:0/582160753 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82400bba0 con 0x7fe824004d00 2024-03-23T03:49:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.742+0000 7fe83be00640 1 -- 172.21.15.102:0/582160753 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe82400bd30 con 0x7fe83c0cde20 2024-03-23T03:49:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.742+0000 7fe841a00640 1 --2- 172.21.15.102:0/582160753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8240039a0 0x7fe824003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.742+0000 7fe841a00640 1 -- 172.21.15.102:0/582160753 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe83c0cde20 msgr2=0x7fe8240047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.742+0000 7fe841a00640 1 --2- 172.21.15.102:0/582160753 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe83c0cde20 0x7fe8240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.742+0000 7fe841a00640 1 -- 172.21.15.102:0/582160753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe824004d00 msgr2=0x7fe824005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.742+0000 7fe841a00640 1 --2- 172.21.15.102:0/582160753 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe824004d00 0x7fe824005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.742+0000 7fe841a00640 1 -- 172.21.15.102:0/582160753 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe824003520 con 0x7fe8240039a0 2024-03-23T03:49:54.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.743+0000 7fe841a00640 1 --2- 172.21.15.102:0/582160753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8240039a0 0x7fe824003dc0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe830001780 tx=0x7fe8240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d0557d93fb4b33 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.744+0000 7fe83be00640 1 -- 172.21.15.102:0/582160753 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8300052d0 con 0x7fe8240039a0 2024-03-23T03:49:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.744+0000 7fe83be00640 1 -- 172.21.15.102:0/582160753 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe83002e050 con 0x7fe8240039a0 2024-03-23T03:49:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.744+0000 7fe83be00640 1 -- 172.21.15.102:0/582160753 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8300035f0 con 0x7fe8240039a0 2024-03-23T03:49:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.744+0000 7fe842e00640 1 -- 172.21.15.102:0/582160753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8240039a0 msgr2=0x7fe824003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.744+0000 7fe842e00640 1 --2- 172.21.15.102:0/582160753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8240039a0 0x7fe824003dc0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe830001780 tx=0x7fe8240057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.744+0000 7fe842400640 1 -- 172.21.15.102:0/582160753 reap_dead start 2024-03-23T03:49:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.744+0000 7fe842e00640 1 -- 172.21.15.102:0/582160753 shutdown_connections 2024-03-23T03:49:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.744+0000 7fe842e00640 1 -- 172.21.15.102:0/582160753 >> 172.21.15.102:0/582160753 conn(0x7fe83c065df0 msgr2=0x7fe83c069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.744+0000 7fe842e00640 1 -- 172.21.15.102:0/582160753 shutdown_connections 2024-03-23T03:49:54.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.744+0000 7fe842e00640 1 -- 172.21.15.102:0/582160753 wait complete. 2024-03-23T03:49:54.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.746+0000 7fe842e00640 1 Processor -- start 2024-03-23T03:49:54.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.746+0000 7fe842e00640 1 -- start start 2024-03-23T03:49:54.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.747+0000 7fe842e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe8240039a0 0x7fe83c1f18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:54.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.747+0000 7fe842e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe83c1f5480 0x7fe83c1f3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:54.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.747+0000 7fe842e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe83c156930 0x7fe83c142240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:54.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.747+0000 7fe842e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c1f5a40 con 0x7fe83c156930 2024-03-23T03:49:54.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.747+0000 7fe842e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c1f5d20 con 0x7fe83c1f5480 2024-03-23T03:49:54.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.747+0000 7fe842e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe83c1f6000 con 0x7fe8240039a0 2024-03-23T03:49:54.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.747+0000 7fe842400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe83c156930 0x7fe83c142240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:54.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.747+0000 7fe842400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe83c156930 0x7fe83c142240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40162/0 (socket says 172.21.15.102:40162) 2024-03-23T03:49:54.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.747+0000 7fe842400640 1 -- 172.21.15.102:0/1206545538 learned_addr learned my addr 172.21.15.102:0/1206545538 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:54.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.748+0000 7fe842400640 1 -- 172.21.15.102:0/1206545538 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe8240039a0 msgr2=0x7fe83c1f18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:54.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.748+0000 7fe842400640 1 --2- 172.21.15.102:0/1206545538 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe8240039a0 0x7fe83c1f18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.748+0000 7fe842400640 1 -- 172.21.15.102:0/1206545538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe83c1f5480 msgr2=0x7fe83c1f3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:54.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.748+0000 7fe842400640 1 --2- 172.21.15.102:0/1206545538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe83c1f5480 0x7fe83c1f3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.748+0000 7fe842400640 1 -- 172.21.15.102:0/1206545538 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83c1f1fa0 con 0x7fe83c156930 2024-03-23T03:49:54.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.748+0000 7fe842400640 1 --2- 172.21.15.102:0/1206545538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe83c156930 0x7fe83c142240 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0cb9c0 tx=0x7fe83400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.748+0000 7fe839600640 1 -- 172.21.15.102:0/1206545538 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83400ed90 con 0x7fe83c156930 2024-03-23T03:49:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.748+0000 7fe839600640 1 -- 172.21.15.102:0/1206545538 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe834014420 con 0x7fe83c156930 2024-03-23T03:49:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.748+0000 7fe839600640 1 -- 172.21.15.102:0/1206545538 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8340147d0 con 0x7fe83c156930 2024-03-23T03:49:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.748+0000 7fe842e00640 1 -- 172.21.15.102:0/1206545538 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe83c1f62e0 con 0x7fe83c156930 2024-03-23T03:49:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.748+0000 7fe842e00640 1 -- 172.21.15.102:0/1206545538 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe83c156f90 con 0x7fe83c156930 2024-03-23T03:49:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.749+0000 7fe842e00640 1 -- 172.21.15.102:0/1206545538 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe83c04e030 con 0x7fe83c156930 2024-03-23T03:49:54.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.750+0000 7fe839600640 1 -- 172.21.15.102:0/1206545538 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe834014930 con 0x7fe83c156930 2024-03-23T03:49:54.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.751+0000 7fe839600640 1 --2- 172.21.15.102:0/1206545538 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe81c06ac00 0x7fe81c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:54.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.751+0000 7fe839600640 1 -- 172.21.15.102:0/1206545538 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7fe834089a40 con 0x7fe83c156930 2024-03-23T03:49:54.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.751+0000 7fe841a00640 1 --2- 172.21.15.102:0/1206545538 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe81c06ac00 0x7fe81c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:54.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.752+0000 7fe841a00640 1 --2- 172.21.15.102:0/1206545538 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe81c06ac00 0x7fe81c06d0c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe830033000 tx=0x7fe83002f000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:54.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.756+0000 7fe839600640 1 -- 172.21.15.102:0/1206545538 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe8340549b0 con 0x7fe83c156930 2024-03-23T03:49:54.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.986+0000 7fe842e00640 1 -- 172.21.15.102:0/1206545538 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe83c163320 con 0x7fe83c156930 2024-03-23T03:49:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.986+0000 7fe839600640 1 -- 172.21.15.102:0/1206545538 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4744 (secure 0 0 0) 0x7fe834057fd0 con 0x7fe83c156930 2024-03-23T03:49:54.987 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:54.987 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5230,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3155961019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3155961019},{"type":"v1","addr":"172.21.15.173:6835","nonce":3155961019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6021,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/251450026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":251450026},{"type":"v1","addr":"172.21.15.102:6836","nonce":251450026}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6024,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3299889122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":3299889122},{"type":"v1","addr":"172.21.15.102:6837","nonce":3299889122}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:49:35.415835+0000","modified":"2024-03-23T03:49:38.434294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6030":{"gid":6030,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":4,"addr":"172.21.15.173:6833/3695568749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3695568749},{"type":"v1","addr":"172.21.15.173:6833","nonce":3695568749}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-23T03:49:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 33 2024-03-23T03:49:54.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.989+0000 7fe842e00640 1 -- 172.21.15.102:0/1206545538 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe81c06ac00 msgr2=0x7fe81c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:54.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.989+0000 7fe842e00640 1 --2- 172.21.15.102:0/1206545538 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe81c06ac00 0x7fe81c06d0c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fe830033000 tx=0x7fe83002f000 comp rx=0 tx=0).stop 2024-03-23T03:49:54.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.989+0000 7fe842e00640 1 -- 172.21.15.102:0/1206545538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe83c156930 msgr2=0x7fe83c142240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:54.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.989+0000 7fe842e00640 1 --2- 172.21.15.102:0/1206545538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe83c156930 0x7fe83c142240 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe83c0cb9c0 tx=0x7fe83400e4b0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.989+0000 7fe842e00640 1 -- 172.21.15.102:0/1206545538 shutdown_connections 2024-03-23T03:49:54.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.989+0000 7fe842e00640 1 --2- 172.21.15.102:0/1206545538 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe81c06ac00 0x7fe81c06d0c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.989+0000 7fe842e00640 1 --2- 172.21.15.102:0/1206545538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe83c156930 0x7fe83c142240 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.989+0000 7fe842e00640 1 --2- 172.21.15.102:0/1206545538 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe83c1f5480 0x7fe83c1f3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.989+0000 7fe842e00640 1 --2- 172.21.15.102:0/1206545538 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe8240039a0 0x7fe83c1f18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:54.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.989+0000 7fe842e00640 1 -- 172.21.15.102:0/1206545538 >> 172.21.15.102:0/1206545538 conn(0x7fe83c065df0 msgr2=0x7fe83c15d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:54.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.989+0000 7fe842e00640 1 -- 172.21.15.102:0/1206545538 shutdown_connections 2024-03-23T03:49:54.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:54.990+0000 7fe842e00640 1 -- 172.21.15.102:0/1206545538 wait complete. 2024-03-23T03:49:55.008 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:49:55.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.210+0000 7fbda1200640 1 Processor -- start 2024-03-23T03:49:55.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.210+0000 7fbda1200640 1 -- start start 2024-03-23T03:49:55.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.211+0000 7fbda1200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fbd9c15b560 0x7fbd9c15b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:55.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.211+0000 7fbda1200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd9c0b39b0 0x7fbd9c0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:55.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.211+0000 7fbda1200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbd9c15e500 0x7fbd9c15e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:55.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.211+0000 7fbda1200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd9c0b4370 con 0x7fbd9c15e500 2024-03-23T03:49:55.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.211+0000 7fbda1200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd9c06c6f0 con 0x7fbd9c15b560 2024-03-23T03:49:55.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.211+0000 7fbda1200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c06c9d0 con 0x7fbd9c0b39b0 2024-03-23T03:49:55.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.212+0000 7fbd9a800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbd9c15e500 0x7fbd9c15e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:55.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.212+0000 7fbd9a800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbd9c15e500 0x7fbd9c15e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40168/0 (socket says 172.21.15.102:40168) 2024-03-23T03:49:55.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.212+0000 7fbd9a800640 1 -- 172.21.15.102:0/2335928292 learned_addr learned my addr 172.21.15.102:0/2335928292 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:55.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.212+0000 7fbd99e00640 1 --2- 172.21.15.102:0/2335928292 >> v2:172.21.15.173:3300/0 conn(0x7fbd9c15b560 0x7fbd9c15b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:55.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.212+0000 7fbd91400640 1 --2- 172.21.15.102:0/2335928292 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd9c0b39b0 0x7fbd9c0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:55.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.212+0000 7fbd9a800640 1 -- 172.21.15.102:0/2335928292 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd9c0b39b0 msgr2=0x7fbd9c0b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:55.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.212+0000 7fbd9a800640 1 --2- 172.21.15.102:0/2335928292 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd9c0b39b0 0x7fbd9c0b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.212+0000 7fbd9a800640 1 -- 172.21.15.102:0/2335928292 >> v2:172.21.15.173:3300/0 conn(0x7fbd9c15b560 msgr2=0x7fbd9c15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:55.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.212+0000 7fbd9a800640 1 --2- 172.21.15.102:0/2335928292 >> v2:172.21.15.173:3300/0 conn(0x7fbd9c15b560 0x7fbd9c15b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.212+0000 7fbd9a800640 1 -- 172.21.15.102:0/2335928292 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c06ccb0 con 0x7fbd9c15e500 2024-03-23T03:49:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.213+0000 7fbd9a800640 1 --2- 172.21.15.102:0/2335928292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd9c15e500 0x7fbd9c15e990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c071470 tx=0x7fbd9402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9191138a50f1e665 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.213+0000 7fbd99400640 1 -- 172.21.15.102:0/2335928292 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9402ca80 con 0x7fbd9c15e500 2024-03-23T03:49:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.214+0000 7fbd99400640 1 -- 172.21.15.102:0/2335928292 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd9402cbe0 con 0x7fbd9c15e500 2024-03-23T03:49:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.214+0000 7fbd99400640 1 -- 172.21.15.102:0/2335928292 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd940343f0 con 0x7fbd9c15e500 2024-03-23T03:49:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.214+0000 7fbda1200640 1 -- 172.21.15.102:0/2335928292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd9c15e500 msgr2=0x7fbd9c15e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.214+0000 7fbda1200640 1 --2- 172.21.15.102:0/2335928292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd9c15e500 0x7fbd9c15e990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fbd9c071470 tx=0x7fbd9402bee0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.214+0000 7fbda1200640 1 -- 172.21.15.102:0/2335928292 shutdown_connections 2024-03-23T03:49:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.214+0000 7fbda1200640 1 --2- 172.21.15.102:0/2335928292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd9c15e500 0x7fbd9c15e990 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.214+0000 7fbda1200640 1 --2- 172.21.15.102:0/2335928292 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd9c0b39b0 0x7fbd9c0b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.214+0000 7fbda1200640 1 --2- 172.21.15.102:0/2335928292 >> v2:172.21.15.173:3300/0 conn(0x7fbd9c15b560 0x7fbd9c15b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.214+0000 7fbda1200640 1 -- 172.21.15.102:0/2335928292 >> 172.21.15.102:0/2335928292 conn(0x7fbd9c06a3d0 msgr2=0x7fbd9c0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:55.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.214+0000 7fbda1200640 1 -- 172.21.15.102:0/2335928292 shutdown_connections 2024-03-23T03:49:55.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.214+0000 7fbda1200640 1 -- 172.21.15.102:0/2335928292 wait complete. 2024-03-23T03:49:55.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.216+0000 7fbda1200640 1 Processor -- start 2024-03-23T03:49:55.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.216+0000 7fbda1200640 1 -- start start 2024-03-23T03:49:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.217+0000 7fbda1200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd9c0b39b0 0x7fbd9c0af5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.217+0000 7fbda1200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd9c15b560 0x7fbd9c0b2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.217+0000 7fbda1200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd9c15e500 0x7fbd9c0b3430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.217+0000 7fbda1200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c0abe80 con 0x7fbd9c15b560 2024-03-23T03:49:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.217+0000 7fbda1200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c158320 con 0x7fbd9c15e500 2024-03-23T03:49:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.217+0000 7fbda1200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c151fa0 con 0x7fbd9c0b39b0 2024-03-23T03:49:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.217+0000 7fbd91400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd9c15b560 0x7fbd9c0b2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.217+0000 7fbd91400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd9c15b560 0x7fbd9c0b2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40172/0 (socket says 172.21.15.102:40172) 2024-03-23T03:49:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.217+0000 7fbd91400640 1 -- 172.21.15.102:0/884583751 learned_addr learned my addr 172.21.15.102:0/884583751 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:55.219 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.218+0000 7fbd99e00640 1 --2- 172.21.15.102:0/884583751 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd9c0b39b0 0x7fbd9c0af5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:55.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.218+0000 7fbd91400640 1 -- 172.21.15.102:0/884583751 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd9c0b39b0 msgr2=0x7fbd9c0af5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:55.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.218+0000 7fbd9a800640 1 --2- 172.21.15.102:0/884583751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd9c15e500 0x7fbd9c0b3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:55.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.218+0000 7fbd91400640 1 --2- 172.21.15.102:0/884583751 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd9c0b39b0 0x7fbd9c0af5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.218+0000 7fbd91400640 1 -- 172.21.15.102:0/884583751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd9c15e500 msgr2=0x7fbd9c0b3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:55.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.218+0000 7fbd91400640 1 --2- 172.21.15.102:0/884583751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd9c15e500 0x7fbd9c0b3430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.218+0000 7fbd91400640 1 -- 172.21.15.102:0/884583751 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd94009b30 con 0x7fbd9c15b560 2024-03-23T03:49:55.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.218+0000 7fbd99e00640 1 --2- 172.21.15.102:0/884583751 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd9c0b39b0 0x7fbd9c0af5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:55.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.218+0000 7fbd91400640 1 --2- 172.21.15.102:0/884583751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd9c15b560 0x7fbd9c0b2ef0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fbd8400d800 tx=0x7fbd8400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:55.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.218+0000 7fbd92a00640 1 -- 172.21.15.102:0/884583751 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd840096a0 con 0x7fbd9c15b560 2024-03-23T03:49:55.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.218+0000 7fbd92a00640 1 -- 172.21.15.102:0/884583751 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd84009800 con 0x7fbd9c15b560 2024-03-23T03:49:55.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.218+0000 7fbda1200640 1 -- 172.21.15.102:0/884583751 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9c1577a0 con 0x7fbd9c15b560 2024-03-23T03:49:55.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.218+0000 7fbd92a00640 1 -- 172.21.15.102:0/884583751 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd84009b90 con 0x7fbd9c15b560 2024-03-23T03:49:55.220 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.218+0000 7fbda1200640 1 -- 172.21.15.102:0/884583751 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9c157d60 con 0x7fbd9c15b560 2024-03-23T03:49:55.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.219+0000 7fbda1200640 1 -- 172.21.15.102:0/884583751 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9c06ac20 con 0x7fbd9c15b560 2024-03-23T03:49:55.221 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.220+0000 7fbd92a00640 1 -- 172.21.15.102:0/884583751 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbd84009cf0 con 0x7fbd9c15b560 2024-03-23T03:49:55.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.221+0000 7fbd92a00640 1 --2- 172.21.15.102:0/884583751 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd7806ac30 0x7fbd7806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:55.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.221+0000 7fbd92a00640 1 -- 172.21.15.102:0/884583751 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7fbd840898b0 con 0x7fbd9c15b560 2024-03-23T03:49:55.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.221+0000 7fbd99e00640 1 --2- 172.21.15.102:0/884583751 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd7806ac30 0x7fbd7806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:55.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.222+0000 7fbd99e00640 1 --2- 172.21.15.102:0/884583751 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd7806ac30 0x7fbd7806d0f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c006fd0 tx=0x7fbd8c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:55.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.225+0000 7fbd92a00640 1 -- 172.21.15.102:0/884583751 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbd84054950 con 0x7fbd9c15b560 2024-03-23T03:49:55.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.453+0000 7fbda1200640 1 -- 172.21.15.102:0/884583751 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd9c073390 con 0x7fbd9c15b560 2024-03-23T03:49:55.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.454+0000 7fbd92a00640 1 -- 172.21.15.102:0/884583751 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4744 (secure 0 0 0) 0x7fbd8400b440 con 0x7fbd9c15b560 2024-03-23T03:49:55.455 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:55.455 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5230,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3155961019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3155961019},{"type":"v1","addr":"172.21.15.173:6835","nonce":3155961019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":6021,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/251450026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":251450026},{"type":"v1","addr":"172.21.15.102:6836","nonce":251450026}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6024,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3299889122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":3299889122},{"type":"v1","addr":"172.21.15.102:6837","nonce":3299889122}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:49:35.415835+0000","modified":"2024-03-23T03:49:38.434294+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6030},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6030":{"gid":6030,"name":"d","rank":0,"incarnation":29,"state":"up:active","state_seq":4,"addr":"172.21.15.173:6833/3695568749","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3695568749},{"type":"v1","addr":"172.21.15.173:6833","nonce":3695568749}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-03-23T03:49:55.456 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 33 2024-03-23T03:49:55.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.457+0000 7fbda1200640 1 -- 172.21.15.102:0/884583751 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd7806ac30 msgr2=0x7fbd7806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:55.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.457+0000 7fbda1200640 1 --2- 172.21.15.102:0/884583751 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd7806ac30 0x7fbd7806d0f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c006fd0 tx=0x7fbd8c008040 comp rx=0 tx=0).stop 2024-03-23T03:49:55.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.457+0000 7fbda1200640 1 -- 172.21.15.102:0/884583751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd9c15b560 msgr2=0x7fbd9c0b2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:55.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.457+0000 7fbda1200640 1 --2- 172.21.15.102:0/884583751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd9c15b560 0x7fbd9c0b2ef0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fbd8400d800 tx=0x7fbd8400dcc0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.457+0000 7fbda1200640 1 -- 172.21.15.102:0/884583751 shutdown_connections 2024-03-23T03:49:55.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.457+0000 7fbda1200640 1 --2- 172.21.15.102:0/884583751 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd7806ac30 0x7fbd7806d0f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.457+0000 7fbda1200640 1 --2- 172.21.15.102:0/884583751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd9c15e500 0x7fbd9c0b3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.457+0000 7fbda1200640 1 --2- 172.21.15.102:0/884583751 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd9c15b560 0x7fbd9c0b2ef0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.457+0000 7fbda1200640 1 --2- 172.21.15.102:0/884583751 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd9c0b39b0 0x7fbd9c0af5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.457+0000 7fbda1200640 1 -- 172.21.15.102:0/884583751 >> 172.21.15.102:0/884583751 conn(0x7fbd9c06a3d0 msgr2=0x7fbd9c159d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:55.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.457+0000 7fbda1200640 1 -- 172.21.15.102:0/884583751 shutdown_connections 2024-03-23T03:49:55.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.457+0000 7fbda1200640 1 -- 172.21.15.102:0/884583751 wait complete. 2024-03-23T03:49:55.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:49:55.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.683+0000 7eff4e800640 1 Processor -- start 2024-03-23T03:49:55.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.683+0000 7eff4e800640 1 -- start start 2024-03-23T03:49:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.684+0000 7eff4e800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff4813f200 0x7eff480ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.684+0000 7eff4e800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7eff4815d350 0x7eff4806c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.684+0000 7eff4e800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff480ce440 con 0x7eff480cd520 2024-03-23T03:49:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.684+0000 7eff4e800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff480cd980 con 0x7eff4815d350 2024-03-23T03:49:55.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.684+0000 7eff4e800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff480cdba0 con 0x7eff4813f200 2024-03-23T03:49:55.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.685+0000 7eff47400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff4813f200 0x7eff480ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:55.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.685+0000 7eff47400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff4813f200 0x7eff480ccfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:38778/0 (socket says 172.21.15.102:38778) 2024-03-23T03:49:55.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.685+0000 7eff47e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7eff4815d350 0x7eff4806c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:55.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.685+0000 7eff47400640 1 -- 172.21.15.102:0/2733890751 learned_addr learned my addr 172.21.15.102:0/2733890751 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:55.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.686+0000 7eff47400640 1 -- 172.21.15.102:0/2733890751 >> v2:172.21.15.173:3300/0 conn(0x7eff4815d350 msgr2=0x7eff4806c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:55.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.686+0000 7eff47400640 1 --2- 172.21.15.102:0/2733890751 >> v2:172.21.15.173:3300/0 conn(0x7eff4815d350 0x7eff4806c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.686+0000 7eff47400640 1 -- 172.21.15.102:0/2733890751 >> v1:172.21.15.173:6789/0 conn(0x7eff480cd520 legacy=0x7eff4815cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:55.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.686+0000 7eff47400640 1 -- 172.21.15.102:0/2733890751 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff480cde80 con 0x7eff4813f200 2024-03-23T03:49:55.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.687+0000 7eff47400640 1 --2- 172.21.15.102:0/2733890751 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff4813f200 0x7eff480ccfe0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7eff38009870 tx=0x7eff3802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9731b65c24e8d06 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:55.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.688+0000 7eff46a00640 1 -- 172.21.15.102:0/2733890751 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff38031640 con 0x7eff4813f200 2024-03-23T03:49:55.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.688+0000 7eff46a00640 1 -- 172.21.15.102:0/2733890751 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff380317a0 con 0x7eff4813f200 2024-03-23T03:49:55.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.688+0000 7eff46a00640 1 -- 172.21.15.102:0/2733890751 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff38031b30 con 0x7eff4813f200 2024-03-23T03:49:55.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.689+0000 7eff4e800640 1 -- 172.21.15.102:0/2733890751 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff4813f200 msgr2=0x7eff480ccfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:55.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.689+0000 7eff4e800640 1 --2- 172.21.15.102:0/2733890751 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff4813f200 0x7eff480ccfe0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7eff38009870 tx=0x7eff3802b8f0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.689+0000 7eff4e800640 1 -- 172.21.15.102:0/2733890751 shutdown_connections 2024-03-23T03:49:55.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.689+0000 7eff4e800640 1 --2- 172.21.15.102:0/2733890751 >> v2:172.21.15.173:3300/0 conn(0x7eff4815d350 0x7eff4806c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.689+0000 7eff4e800640 1 --2- 172.21.15.102:0/2733890751 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff4813f200 0x7eff480ccfe0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.689+0000 7eff4e800640 1 -- 172.21.15.102:0/2733890751 >> 172.21.15.102:0/2733890751 conn(0x7eff4806a3d0 msgr2=0x7eff480b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:55.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.689+0000 7eff4e800640 1 -- 172.21.15.102:0/2733890751 shutdown_connections 2024-03-23T03:49:55.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.689+0000 7eff4e800640 1 -- 172.21.15.102:0/2733890751 wait complete. 2024-03-23T03:49:55.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.691+0000 7eff4e800640 1 Processor -- start 2024-03-23T03:49:55.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.691+0000 7eff4e800640 1 -- start start 2024-03-23T03:49:55.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.692+0000 7eff4e800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff480cd520 0x7eff4814fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:55.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.692+0000 7eff4e800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff4813f200 0x7eff480c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:55.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.692+0000 7eff4e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff4815d350 0x7eff4815a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:55.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.692+0000 7eff4e800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff480c4bc0 con 0x7eff4815d350 2024-03-23T03:49:55.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.692+0000 7eff4e800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff481f5e10 con 0x7eff480cd520 2024-03-23T03:49:55.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.692+0000 7eff4e800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff481f60f0 con 0x7eff4813f200 2024-03-23T03:49:55.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.692+0000 7eff47e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff4815d350 0x7eff4815a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:55.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.692+0000 7eff47e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff4815d350 0x7eff4815a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40188/0 (socket says 172.21.15.102:40188) 2024-03-23T03:49:55.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.692+0000 7eff47e00640 1 -- 172.21.15.102:0/379893834 learned_addr learned my addr 172.21.15.102:0/379893834 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:55.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.692+0000 7eff3ea00640 1 --2- 172.21.15.102:0/379893834 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff4813f200 0x7eff480c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:55.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.692+0000 7eff47400640 1 --2- 172.21.15.102:0/379893834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff480cd520 0x7eff4814fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:55.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.692+0000 7eff47e00640 1 -- 172.21.15.102:0/379893834 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff4813f200 msgr2=0x7eff480c41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:55.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.692+0000 7eff47e00640 1 --2- 172.21.15.102:0/379893834 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff4813f200 0x7eff480c41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.692+0000 7eff47e00640 1 -- 172.21.15.102:0/379893834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff480cd520 msgr2=0x7eff4814fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:55.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.692+0000 7eff47e00640 1 --2- 172.21.15.102:0/379893834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff480cd520 0x7eff4814fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.692+0000 7eff47e00640 1 -- 172.21.15.102:0/379893834 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff380094d0 con 0x7eff4815d350 2024-03-23T03:49:55.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.692+0000 7eff47400640 1 --2- 172.21.15.102:0/379893834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff480cd520 0x7eff4814fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:55.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.693+0000 7eff47e00640 1 --2- 172.21.15.102:0/379893834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff4815d350 0x7eff4815a6d0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7eff4000efc0 tx=0x7eff4000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:55.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.693+0000 7eff3fe00640 1 -- 172.21.15.102:0/379893834 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff40009ba0 con 0x7eff4815d350 2024-03-23T03:49:55.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.693+0000 7eff3fe00640 1 -- 172.21.15.102:0/379893834 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff4000f040 con 0x7eff4815d350 2024-03-23T03:49:55.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.693+0000 7eff3fe00640 1 -- 172.21.15.102:0/379893834 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff400146c0 con 0x7eff4815d350 2024-03-23T03:49:55.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.693+0000 7eff4e800640 1 -- 172.21.15.102:0/379893834 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff481f63d0 con 0x7eff4815d350 2024-03-23T03:49:55.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.693+0000 7eff4e800640 1 -- 172.21.15.102:0/379893834 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff48150240 con 0x7eff4815d350 2024-03-23T03:49:55.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.694+0000 7eff4e800640 1 -- 172.21.15.102:0/379893834 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff481f68f0 con 0x7eff4815d350 2024-03-23T03:49:55.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.695+0000 7eff3fe00640 1 -- 172.21.15.102:0/379893834 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7eff40014820 con 0x7eff4815d350 2024-03-23T03:49:55.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.696+0000 7eff3fe00640 1 --2- 172.21.15.102:0/379893834 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7eff2406ac00 0x7eff2406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:55.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.696+0000 7eff3fe00640 1 -- 172.21.15.102:0/379893834 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7eff4008a0d0 con 0x7eff4815d350 2024-03-23T03:49:55.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.696+0000 7eff47400640 1 --2- 172.21.15.102:0/379893834 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7eff2406ac00 0x7eff2406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:55.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.697+0000 7eff47400640 1 --2- 172.21.15.102:0/379893834 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7eff2406ac00 0x7eff2406d0c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7eff38036000 tx=0x7eff3803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:55.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.701+0000 7eff3fe00640 1 -- 172.21.15.102:0/379893834 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7eff481f68f0 con 0x7eff4815d350 2024-03-23T03:49:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.896+0000 7eff4e800640 1 -- 172.21.15.102:0/379893834 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff481f68f0 con 0x7eff4815d350 2024-03-23T03:49:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.897+0000 7eff3fe00640 1 -- 172.21.15.102:0/379893834 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+16536 (secure 0 0 0) 0x7eff481f68f0 con 0x7eff4815d350 2024-03-23T03:49:55.898 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:49:55.898 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":69,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:49:43.895428+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:49:33.007757+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-03-23T03:49:34.449411+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.e","mappings":[{"from":7,"to":2}]},{"pgid":"10.23","mappings":[{"from":7,"to":3}]},{"pgid":"10.2e","mappings":[{"from":4,"to":3}]},{"pgid":"10.2f","mappings":[{"from":4,"to":5}]},{"pgid":"10.3c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:49:55.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.900+0000 7eff4e800640 1 -- 172.21.15.102:0/379893834 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7eff2406ac00 msgr2=0x7eff2406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:55.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.900+0000 7eff4e800640 1 --2- 172.21.15.102:0/379893834 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7eff2406ac00 0x7eff2406d0c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7eff38036000 tx=0x7eff3803c040 comp rx=0 tx=0).stop 2024-03-23T03:49:55.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.900+0000 7eff4e800640 1 -- 172.21.15.102:0/379893834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff4815d350 msgr2=0x7eff4815a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:55.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.900+0000 7eff4e800640 1 --2- 172.21.15.102:0/379893834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff4815d350 0x7eff4815a6d0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7eff4000efc0 tx=0x7eff4000c490 comp rx=0 tx=0).stop 2024-03-23T03:49:55.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.900+0000 7eff4e800640 1 -- 172.21.15.102:0/379893834 shutdown_connections 2024-03-23T03:49:55.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.900+0000 7eff4e800640 1 --2- 172.21.15.102:0/379893834 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7eff2406ac00 0x7eff2406d0c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.900+0000 7eff4e800640 1 --2- 172.21.15.102:0/379893834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff4815d350 0x7eff4815a6d0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.900+0000 7eff4e800640 1 --2- 172.21.15.102:0/379893834 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff4813f200 0x7eff480c41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.900+0000 7eff4e800640 1 --2- 172.21.15.102:0/379893834 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff480cd520 0x7eff4814fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.900+0000 7eff4e800640 1 -- 172.21.15.102:0/379893834 >> 172.21.15.102:0/379893834 conn(0x7eff4806a3d0 msgr2=0x7eff4815bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.901+0000 7eff4e800640 1 -- 172.21.15.102:0/379893834 shutdown_connections 2024-03-23T03:49:55.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:55.901+0000 7eff4e800640 1 -- 172.21.15.102:0/379893834 wait complete. 2024-03-23T03:49:55.919 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T03:49:56.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.115+0000 7f78ea200640 1 Processor -- start 2024-03-23T03:49:56.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.116+0000 7f78ea200640 1 -- start start 2024-03-23T03:49:56.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.117+0000 7f78ea200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f78e413fa80 0x7f78e413ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:56.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.117+0000 7f78ea200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78e41404a0 con 0x7f78e40cf040 2024-03-23T03:49:56.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.117+0000 7f78ea200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78e406cdc0 con 0x7f78e4160dd0 2024-03-23T03:49:56.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.117+0000 7f78ea200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78e406d060 con 0x7f78e413fa80 2024-03-23T03:49:56.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.118+0000 7f78e3e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f78e40cf040 0x7f78e41422d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47470/0 (socket says 172.21.15.102:47470) 2024-03-23T03:49:56.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.118+0000 7f78e3e00640 1 -- 172.21.15.102:0/2331422527 learned_addr learned my addr 172.21.15.102:0/2331422527 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:56.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.118+0000 7f78e3400640 1 --2- 172.21.15.102:0/2331422527 >> v2:172.21.15.173:3300/0 conn(0x7f78e413fa80 0x7f78e413ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:56.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.118+0000 7f78e2a00640 1 -- 172.21.15.102:0/2331422527 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254243362 0 0) 0x7f78e41404a0 con 0x7f78e40cf040 2024-03-23T03:49:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.118+0000 7f78e2a00640 1 -- 172.21.15.102:0/2331422527 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78c8003520 con 0x7f78e40cf040 2024-03-23T03:49:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.118+0000 7f78e2a00640 1 -- 172.21.15.102:0/2331422527 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3443300381 0 0) 0x7f78c8003520 con 0x7f78e40cf040 2024-03-23T03:49:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.118+0000 7f78e2a00640 1 -- 172.21.15.102:0/2331422527 >> v2:172.21.15.173:3300/0 conn(0x7f78e413fa80 msgr2=0x7f78e413ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.118+0000 7f78e2a00640 1 --2- 172.21.15.102:0/2331422527 >> v2:172.21.15.173:3300/0 conn(0x7f78e413fa80 0x7f78e413ff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.118+0000 7f78e2a00640 1 -- 172.21.15.102:0/2331422527 >> v1:172.21.15.173:6789/0 conn(0x7f78e4160dd0 legacy=0x7f78e40ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.119+0000 7f78e2a00640 1 -- 172.21.15.102:0/2331422527 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e406d340 con 0x7f78e40cf040 2024-03-23T03:49:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.119+0000 7f78e2a00640 1 -- 172.21.15.102:0/2331422527 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f78dc0019f0 con 0x7f78e40cf040 2024-03-23T03:49:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.119+0000 7f78e2a00640 1 -- 172.21.15.102:0/2331422527 >> v1:172.21.15.102:6789/0 conn(0x7f78e40cf040 legacy=0x7f78e41422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.119+0000 7f78e2a00640 1 --2- 172.21.15.102:0/2331422527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78c80039c0 0x7f78c8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.119+0000 7f78e2a00640 1 --2- 172.21.15.102:0/2331422527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78c80047c0 0x7f78c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:56.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.119+0000 7f78e2a00640 1 --2- 172.21.15.102:0/2331422527 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78c8005180 0x7f78c800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:56.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.119+0000 7f78e2a00640 1 -- 172.21.15.102:0/2331422527 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e406d060 con 0x7f78c80039c0 2024-03-23T03:49:56.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.119+0000 7f78e2a00640 1 -- 172.21.15.102:0/2331422527 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c800c080 con 0x7f78c80047c0 2024-03-23T03:49:56.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.119+0000 7f78e2a00640 1 -- 172.21.15.102:0/2331422527 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c800c240 con 0x7f78c8005180 2024-03-23T03:49:56.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.119+0000 7f78e3400640 1 --2- 172.21.15.102:0/2331422527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78c80039c0 0x7f78c8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:56.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.119+0000 7f78e3400640 1 -- 172.21.15.102:0/2331422527 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78c8005180 msgr2=0x7f78c800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:56.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.119+0000 7f78e3400640 1 --2- 172.21.15.102:0/2331422527 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78c8005180 0x7f78c800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:56.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.119+0000 7f78e3400640 1 -- 172.21.15.102:0/2331422527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78c80047c0 msgr2=0x7f78c8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:56.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.119+0000 7f78e3400640 1 --2- 172.21.15.102:0/2331422527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78c80047c0 0x7f78c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:56.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.119+0000 7f78e3400640 1 -- 172.21.15.102:0/2331422527 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c8003520 con 0x7f78c80039c0 2024-03-23T03:49:56.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.119+0000 7f78e3e00640 1 -- 172.21.15.102:0/2331422527 reap_dead start 2024-03-23T03:49:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.120+0000 7f78e3400640 1 --2- 172.21.15.102:0/2331422527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78c80039c0 0x7f78c8003de0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f78d4009ac0 tx=0x7f78c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78215d5fcc98adf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.121+0000 7f78e2a00640 1 -- 172.21.15.102:0/2331422527 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d402c6b0 con 0x7f78c80039c0 2024-03-23T03:49:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.121+0000 7f78e2a00640 1 -- 172.21.15.102:0/2331422527 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d402c810 con 0x7f78c80039c0 2024-03-23T03:49:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.121+0000 7f78e2a00640 1 -- 172.21.15.102:0/2331422527 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d402cba0 con 0x7f78c80039c0 2024-03-23T03:49:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.121+0000 7f78ea200640 1 -- 172.21.15.102:0/2331422527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78c80039c0 msgr2=0x7f78c8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.121+0000 7f78ea200640 1 --2- 172.21.15.102:0/2331422527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78c80039c0 0x7f78c8003de0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f78d4009ac0 tx=0x7f78c80057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.121+0000 7f78ea200640 1 -- 172.21.15.102:0/2331422527 shutdown_connections 2024-03-23T03:49:56.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.121+0000 7f78ea200640 1 --2- 172.21.15.102:0/2331422527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78c80039c0 0x7f78c8003de0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:56.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.121+0000 7f78ea200640 1 -- 172.21.15.102:0/2331422527 >> 172.21.15.102:0/2331422527 conn(0x7f78e406a0b0 msgr2=0x7f78e406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:56.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.121+0000 7f78ea200640 1 -- 172.21.15.102:0/2331422527 shutdown_connections 2024-03-23T03:49:56.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.122+0000 7f78ea200640 1 -- 172.21.15.102:0/2331422527 wait complete. 2024-03-23T03:49:56.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.123+0000 7f78ea200640 1 Processor -- start 2024-03-23T03:49:56.125 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.123+0000 7f78ea200640 1 -- start start 2024-03-23T03:49:56.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.124+0000 7f78ea200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78c80039c0 0x7f78e41e6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:56.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.124+0000 7f78ea200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78e413efd0 0x7f78e4151cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:56.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.124+0000 7f78ea200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78e40b30e0 0x7f78e40b3560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.124+0000 7f78ea200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e41521f0 con 0x7f78c80039c0 2024-03-23T03:49:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.124+0000 7f78ea200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78e40b3aa0 con 0x7f78e40b30e0 2024-03-23T03:49:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.124+0000 7f78ea200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78e413cf90 con 0x7f78e413efd0 2024-03-23T03:49:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.125+0000 7f78e3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78c80039c0 0x7f78e41e6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.125+0000 7f78e3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78c80039c0 0x7f78e41e6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40210/0 (socket says 172.21.15.102:40210) 2024-03-23T03:49:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.125+0000 7f78e3400640 1 -- 172.21.15.102:0/805166815 learned_addr learned my addr 172.21.15.102:0/805166815 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.125+0000 7f78daa00640 1 --2- 172.21.15.102:0/805166815 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78e413efd0 0x7f78e4151cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.125+0000 7f78e3e00640 1 --2- 172.21.15.102:0/805166815 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78e40b30e0 0x7f78e40b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.125+0000 7f78e3400640 1 -- 172.21.15.102:0/805166815 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78e413efd0 msgr2=0x7f78e4151cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.125+0000 7f78e3400640 1 --2- 172.21.15.102:0/805166815 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78e413efd0 0x7f78e4151cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.125+0000 7f78e3400640 1 -- 172.21.15.102:0/805166815 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78e40b30e0 msgr2=0x7f78e40b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.125+0000 7f78e3400640 1 --2- 172.21.15.102:0/805166815 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78e40b30e0 0x7f78e40b3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:56.128 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.125+0000 7f78e3400640 1 -- 172.21.15.102:0/805166815 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e41e7820 con 0x7f78c80039c0 2024-03-23T03:49:56.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.126+0000 7f78e3400640 1 --2- 172.21.15.102:0/805166815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78c80039c0 0x7f78e41e6b70 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f78d402c250 tx=0x7f78d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:56.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.126+0000 7f78dbe00640 1 -- 172.21.15.102:0/805166815 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d4003810 con 0x7f78c80039c0 2024-03-23T03:49:56.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.126+0000 7f78dbe00640 1 -- 172.21.15.102:0/805166815 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d4003970 con 0x7f78c80039c0 2024-03-23T03:49:56.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.126+0000 7f78dbe00640 1 -- 172.21.15.102:0/805166815 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d4003d00 con 0x7f78c80039c0 2024-03-23T03:49:56.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.126+0000 7f78ea200640 1 -- 172.21.15.102:0/805166815 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78e41e7b00 con 0x7f78c80039c0 2024-03-23T03:49:56.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.126+0000 7f78ea200640 1 -- 172.21.15.102:0/805166815 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78e40aedd0 con 0x7f78c80039c0 2024-03-23T03:49:56.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.127+0000 7f78ea200640 1 -- 172.21.15.102:0/805166815 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78e413ca60 con 0x7f78c80039c0 2024-03-23T03:49:56.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.128+0000 7f78dbe00640 1 -- 172.21.15.102:0/805166815 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f78d4034440 con 0x7f78c80039c0 2024-03-23T03:49:56.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.129+0000 7f78dbe00640 1 --2- 172.21.15.102:0/805166815 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78c006ac00 0x7f78c006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:56.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.129+0000 7f78dbe00640 1 -- 172.21.15.102:0/805166815 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6174+0+0 (secure 0 0 0) 0x7f78d40ab4a0 con 0x7f78c80039c0 2024-03-23T03:49:56.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.129+0000 7f78daa00640 1 --2- 172.21.15.102:0/805166815 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78c006ac00 0x7f78c006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:56.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.130+0000 7f78daa00640 1 --2- 172.21.15.102:0/805166815 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78c006ac00 0x7f78c006d0c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f78e4159060 tx=0x7f78cc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:56.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.135+0000 7f78dbe00640 1 -- 172.21.15.102:0/805166815 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f78d4076380 con 0x7f78c80039c0 2024-03-23T03:49:56.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.351+0000 7f78ea200640 1 -- 172.21.15.102:0/805166815 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f78e4160600 con 0x7f78c80039c0 2024-03-23T03:49:56.353 INFO:tasks.ceph.mon.a.smithi102.stderr:2024-03-23T03:49:56.352+0000 7f5aab200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T03:49:56.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.363+0000 7f78dbe00640 1 -- 172.21.15.102:0/805166815 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v34) v1 ==== 161+0+0 (secure 0 0 0) 0x7f78d4003e60 con 0x7f78c80039c0 2024-03-23T03:49:56.364 INFO:teuthology.orchestra.run.smithi102.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T03:49:56.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.366+0000 7f78ea200640 1 -- 172.21.15.102:0/805166815 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78c006ac00 msgr2=0x7f78c006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:56.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.366+0000 7f78ea200640 1 --2- 172.21.15.102:0/805166815 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78c006ac00 0x7f78c006d0c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f78e4159060 tx=0x7f78cc009040 comp rx=0 tx=0).stop 2024-03-23T03:49:56.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.366+0000 7f78ea200640 1 -- 172.21.15.102:0/805166815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78c80039c0 msgr2=0x7f78e41e6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:56.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.366+0000 7f78ea200640 1 --2- 172.21.15.102:0/805166815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78c80039c0 0x7f78e41e6b70 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f78d402c250 tx=0x7f78d4004730 comp rx=0 tx=0).stop 2024-03-23T03:49:56.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.366+0000 7f78ea200640 1 -- 172.21.15.102:0/805166815 shutdown_connections 2024-03-23T03:49:56.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.366+0000 7f78ea200640 1 --2- 172.21.15.102:0/805166815 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78c006ac00 0x7f78c006d0c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:56.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.366+0000 7f78ea200640 1 --2- 172.21.15.102:0/805166815 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78e40b30e0 0x7f78e40b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:56.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.366+0000 7f78ea200640 1 --2- 172.21.15.102:0/805166815 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78e413efd0 0x7f78e4151cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:56.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.366+0000 7f78ea200640 1 --2- 172.21.15.102:0/805166815 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78c80039c0 0x7f78e41e6b70 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:56.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.366+0000 7f78ea200640 1 -- 172.21.15.102:0/805166815 >> 172.21.15.102:0/805166815 conn(0x7f78e406a0b0 msgr2=0x7f78e415e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:56.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.366+0000 7f78ea200640 1 -- 172.21.15.102:0/805166815 shutdown_connections 2024-03-23T03:49:56.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.366+0000 7f78ea200640 1 -- 172.21.15.102:0/805166815 wait complete. 2024-03-23T03:49:56.384 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T03:49:56.392 INFO:tasks.ceph.mds.d.smithi173.stderr: -4393> 2024-03-23T03:49:38.892+0000 7fe7dbe00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T03:49:56.465 INFO:tasks.ceph.mds.d.smithi173.stdout:starting mds.d at 2024-03-23T03:49:56.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.589+0000 7f2faea00640 1 Processor -- start 2024-03-23T03:49:56.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.589+0000 7f2faea00640 1 -- start start 2024-03-23T03:49:56.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.591+0000 7f2faea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2fa80ce930 0x7f2fa80add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:56.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.591+0000 7f2faea00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f2fa806ce90 0x7f2fa806d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:56.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.591+0000 7f2faea00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa80cdba0 con 0x7f2fa80cd3e0 2024-03-23T03:49:56.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.591+0000 7f2faea00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa806c760 con 0x7f2fa80ce930 2024-03-23T03:49:56.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.591+0000 7f2faea00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa80b42d0 con 0x7f2fa806ce90 2024-03-23T03:49:56.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.591+0000 7f2fa7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2fa80ce930 0x7f2fa80add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:56.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.591+0000 7f2fa7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2fa80ce930 0x7f2fa80add30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40218/0 (socket says 172.21.15.102:40218) 2024-03-23T03:49:56.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.591+0000 7f2fa7e00640 1 -- 172.21.15.102:0/4008581581 learned_addr learned my addr 172.21.15.102:0/4008581581 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:56.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.592+0000 7f2faca00640 1 --2- 172.21.15.102:0/4008581581 >> v2:172.21.15.173:3300/0 conn(0x7f2fa806ce90 0x7f2fa806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:56.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.592+0000 7f2fa7e00640 1 -- 172.21.15.102:0/4008581581 >> v2:172.21.15.173:3300/0 conn(0x7f2fa806ce90 msgr2=0x7f2fa806d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:56.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.592+0000 7f2fa7e00640 1 --2- 172.21.15.102:0/4008581581 >> v2:172.21.15.173:3300/0 conn(0x7f2fa806ce90 0x7f2fa806d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:56.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.592+0000 7f2fa7e00640 1 -- 172.21.15.102:0/4008581581 >> v1:172.21.15.173:6789/0 conn(0x7f2fa80cd3e0 legacy=0x7f2fa80ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:56.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.592+0000 7f2fa7e00640 1 -- 172.21.15.102:0/4008581581 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa80cde80 con 0x7f2fa80ce930 2024-03-23T03:49:56.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.593+0000 7f2fa7e00640 1 --2- 172.21.15.102:0/4008581581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa80ce930 0x7f2fa80add30 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f2fa806c4b0 tx=0x7f2f9802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a6f45d1b50f204 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:56.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.594+0000 7f2fa7400640 1 -- 172.21.15.102:0/4008581581 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9802ca80 con 0x7f2fa80ce930 2024-03-23T03:49:56.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.594+0000 7f2fa7400640 1 -- 172.21.15.102:0/4008581581 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9802cbe0 con 0x7f2fa80ce930 2024-03-23T03:49:56.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.594+0000 7f2fa7400640 1 -- 172.21.15.102:0/4008581581 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f980343f0 con 0x7f2fa80ce930 2024-03-23T03:49:56.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.594+0000 7f2faea00640 1 -- 172.21.15.102:0/4008581581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa80ce930 msgr2=0x7f2fa80add30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:56.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.594+0000 7f2faea00640 1 --2- 172.21.15.102:0/4008581581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa80ce930 0x7f2fa80add30 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f2fa806c4b0 tx=0x7f2f9802bee0 comp rx=0 tx=0).stop 2024-03-23T03:49:56.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.594+0000 7f2faea00640 1 -- 172.21.15.102:0/4008581581 shutdown_connections 2024-03-23T03:49:56.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.594+0000 7f2faea00640 1 --2- 172.21.15.102:0/4008581581 >> v2:172.21.15.173:3300/0 conn(0x7f2fa806ce90 0x7f2fa806d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:56.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.594+0000 7f2faea00640 1 --2- 172.21.15.102:0/4008581581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa80ce930 0x7f2fa80add30 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:56.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.594+0000 7f2faea00640 1 -- 172.21.15.102:0/4008581581 >> 172.21.15.102:0/4008581581 conn(0x7f2fa806a0b0 msgr2=0x7f2fa806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:56.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.595+0000 7f2faea00640 1 -- 172.21.15.102:0/4008581581 shutdown_connections 2024-03-23T03:49:56.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.595+0000 7f2faea00640 1 -- 172.21.15.102:0/4008581581 wait complete. 2024-03-23T03:49:56.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.596+0000 7f2faea00640 1 Processor -- start 2024-03-23T03:49:56.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.597+0000 7f2faea00640 1 -- start start 2024-03-23T03:49:56.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.597+0000 7f2faea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa806ce90 0x7f2fa814c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:56.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.597+0000 7f2faea00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fa80cd3e0 0x7f2fa81effc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.597+0000 7f2faea00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fa80ce930 0x7f2fa80ae290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.597+0000 7f2faea00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa8159ce0 con 0x7f2fa806ce90 2024-03-23T03:49:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.597+0000 7f2faea00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa814fc50 con 0x7f2fa80ce930 2024-03-23T03:49:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.597+0000 7f2faea00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa8159f30 con 0x7f2fa80cd3e0 2024-03-23T03:49:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.598+0000 7f2fa7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa806ce90 0x7f2fa814c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.598+0000 7f2fa7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa806ce90 0x7f2fa814c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40224/0 (socket says 172.21.15.102:40224) 2024-03-23T03:49:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.598+0000 7f2fa7e00640 1 -- 172.21.15.102:0/1647857474 learned_addr learned my addr 172.21.15.102:0/1647857474 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.598+0000 7f2fa7e00640 1 -- 172.21.15.102:0/1647857474 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fa80cd3e0 msgr2=0x7f2fa81effc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.598+0000 7f2fa7e00640 1 --2- 172.21.15.102:0/1647857474 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fa80cd3e0 0x7f2fa81effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.598+0000 7f2fa7e00640 1 -- 172.21.15.102:0/1647857474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fa80ce930 msgr2=0x7f2fa80ae290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.598+0000 7f2fa7e00640 1 --2- 172.21.15.102:0/1647857474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fa80ce930 0x7f2fa80ae290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.598+0000 7f2fa7e00640 1 -- 172.21.15.102:0/1647857474 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f98009b30 con 0x7f2fa806ce90 2024-03-23T03:49:56.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.598+0000 7f2fa7e00640 1 --2- 172.21.15.102:0/1647857474 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa806ce90 0x7f2fa814c3e0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f2f98000f80 tx=0x7f2f98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:56.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.599+0000 7f2fa4c00640 1 -- 172.21.15.102:0/1647857474 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f98009510 con 0x7f2fa806ce90 2024-03-23T03:49:56.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.599+0000 7f2fa4c00640 1 -- 172.21.15.102:0/1647857474 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9803e450 con 0x7f2fa806ce90 2024-03-23T03:49:56.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.599+0000 7f2fa4c00640 1 -- 172.21.15.102:0/1647857474 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9803e800 con 0x7f2fa806ce90 2024-03-23T03:49:56.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.599+0000 7f2faea00640 1 -- 172.21.15.102:0/1647857474 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa815a210 con 0x7f2fa806ce90 2024-03-23T03:49:56.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.599+0000 7f2faea00640 1 -- 172.21.15.102:0/1647857474 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa814c920 con 0x7f2fa806ce90 2024-03-23T03:49:56.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.600+0000 7f2faea00640 1 -- 172.21.15.102:0/1647857474 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa80cd850 con 0x7f2fa806ce90 2024-03-23T03:49:56.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.601+0000 7f2fa4c00640 1 -- 172.21.15.102:0/1647857474 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2f9802cd50 con 0x7f2fa806ce90 2024-03-23T03:49:56.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.602+0000 7f2fa4c00640 1 --2- 172.21.15.102:0/1647857474 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2f8406ac00 0x7f2f8406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:56.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.602+0000 7f2fa4c00640 1 -- 172.21.15.102:0/1647857474 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6260+0+0 (secure 0 0 0) 0x7f2f980ac060 con 0x7f2fa806ce90 2024-03-23T03:49:56.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.602+0000 7f2f9f400640 1 --2- 172.21.15.102:0/1647857474 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2f8406ac00 0x7f2f8406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:56.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.603+0000 7f2f9f400640 1 --2- 172.21.15.102:0/1647857474 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2f8406ac00 0x7f2f8406d0c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80cf2a0 tx=0x7f2f900079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:56.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.607+0000 7f2fa4c00640 1 -- 172.21.15.102:0/1647857474 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2fa80cd850 con 0x7f2fa806ce90 2024-03-23T03:49:56.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:56.824+0000 7f2faea00640 1 -- 172.21.15.102:0/1647857474 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2fa80cd850 con 0x7f2fa806ce90 2024-03-23T03:49:57.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.369+0000 7f2fa4c00640 1 -- 172.21.15.102:0/1647857474 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2fa80cd850 con 0x7f2fa806ce90 2024-03-23T03:49:57.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.373+0000 7f2faea00640 1 -- 172.21.15.102:0/1647857474 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2f8406ac00 msgr2=0x7f2f8406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:57.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.373+0000 7f2faea00640 1 --2- 172.21.15.102:0/1647857474 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2f8406ac00 0x7f2f8406d0c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2fa80cf2a0 tx=0x7f2f900079b0 comp rx=0 tx=0).stop 2024-03-23T03:49:57.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.373+0000 7f2faea00640 1 -- 172.21.15.102:0/1647857474 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa806ce90 msgr2=0x7f2fa814c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:57.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.373+0000 7f2faea00640 1 --2- 172.21.15.102:0/1647857474 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa806ce90 0x7f2fa814c3e0 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f2f98000f80 tx=0x7f2f98004a40 comp rx=0 tx=0).stop 2024-03-23T03:49:57.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.373+0000 7f2faea00640 1 -- 172.21.15.102:0/1647857474 shutdown_connections 2024-03-23T03:49:57.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.373+0000 7f2faea00640 1 --2- 172.21.15.102:0/1647857474 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2f8406ac00 0x7f2f8406d0c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:57.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.373+0000 7f2faea00640 1 --2- 172.21.15.102:0/1647857474 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2fa80ce930 0x7f2fa80ae290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:57.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.373+0000 7f2faea00640 1 --2- 172.21.15.102:0/1647857474 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2fa80cd3e0 0x7f2fa81effc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:57.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.373+0000 7f2faea00640 1 --2- 172.21.15.102:0/1647857474 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2fa806ce90 0x7f2fa814c3e0 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:57.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.373+0000 7f2faea00640 1 -- 172.21.15.102:0/1647857474 >> 172.21.15.102:0/1647857474 conn(0x7f2fa806a0b0 msgr2=0x7f2fa80afe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:57.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.373+0000 7f2faea00640 1 -- 172.21.15.102:0/1647857474 shutdown_connections 2024-03-23T03:49:57.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.373+0000 7f2faea00640 1 -- 172.21.15.102:0/1647857474 wait complete. 2024-03-23T03:49:57.395 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T03:49:57.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.601+0000 7fb386400640 1 Processor -- start 2024-03-23T03:49:57.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.602+0000 7fb386400640 1 -- start start 2024-03-23T03:49:57.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.603+0000 7fb386400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb38006ce90 0x7fb38006d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:57.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.603+0000 7fb386400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3800cdba0 con 0x7fb3800add30 2024-03-23T03:49:57.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.603+0000 7fb386400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3800cde80 con 0x7fb3800ce930 2024-03-23T03:49:57.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.603+0000 7fb386400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3800ae1c0 con 0x7fb38006ce90 2024-03-23T03:49:57.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.604+0000 7fb377e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb3800add30 0x7fb3800ced10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47482/0 (socket says 172.21.15.102:47482) 2024-03-23T03:49:57.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.604+0000 7fb377e00640 1 -- 172.21.15.102:0/672440921 learned_addr learned my addr 172.21.15.102:0/672440921 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.604+0000 7fb37fe00640 1 --2- 172.21.15.102:0/672440921 >> v2:172.21.15.173:3300/0 conn(0x7fb38006ce90 0x7fb38006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.604+0000 7fb37ea00640 1 -- 172.21.15.102:0/672440921 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989932340 0 0) 0x7fb3800cdba0 con 0x7fb3800add30 2024-03-23T03:49:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.604+0000 7fb37ea00640 1 -- 172.21.15.102:0/672440921 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb364003520 con 0x7fb3800add30 2024-03-23T03:49:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.604+0000 7fb37ea00640 1 -- 172.21.15.102:0/672440921 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1225780167 0 0) 0x7fb364003520 con 0x7fb3800add30 2024-03-23T03:49:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.604+0000 7fb37ea00640 1 -- 172.21.15.102:0/672440921 >> v2:172.21.15.173:3300/0 conn(0x7fb38006ce90 msgr2=0x7fb38006d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.604+0000 7fb37ea00640 1 --2- 172.21.15.102:0/672440921 >> v2:172.21.15.173:3300/0 conn(0x7fb38006ce90 0x7fb38006d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.604+0000 7fb37ea00640 1 -- 172.21.15.102:0/672440921 >> v1:172.21.15.173:6789/0 conn(0x7fb3800ce930 legacy=0x7fb3800cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.604+0000 7fb37ea00640 1 -- 172.21.15.102:0/672440921 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3800ce160 con 0x7fb3800add30 2024-03-23T03:49:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.604+0000 7fb37ea00640 1 -- 172.21.15.102:0/672440921 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fb368001670 con 0x7fb3800add30 2024-03-23T03:49:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.604+0000 7fb37ea00640 1 -- 172.21.15.102:0/672440921 >> v1:172.21.15.102:6789/0 conn(0x7fb3800add30 legacy=0x7fb3800ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:57.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.605+0000 7fb37ea00640 1 --2- 172.21.15.102:0/672440921 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb364003970 0x7fb364003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.605+0000 7fb37ea00640 1 --2- 172.21.15.102:0/672440921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3640047c0 0x7fb364004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.605+0000 7fb37ea00640 1 --2- 172.21.15.102:0/672440921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb364005180 0x7fb36400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.605+0000 7fb37ea00640 1 -- 172.21.15.102:0/672440921 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36400bff0 con 0x7fb364005180 2024-03-23T03:49:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.605+0000 7fb37ea00640 1 -- 172.21.15.102:0/672440921 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36400c160 con 0x7fb3640047c0 2024-03-23T03:49:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.605+0000 7fb37ea00640 1 -- 172.21.15.102:0/672440921 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb36400c320 con 0x7fb364003970 2024-03-23T03:49:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.605+0000 7fb37fe00640 1 --2- 172.21.15.102:0/672440921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb364005180 0x7fb36400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.605+0000 7fb37f400640 1 --2- 172.21.15.102:0/672440921 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb364003970 0x7fb364003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.605+0000 7fb377e00640 1 --2- 172.21.15.102:0/672440921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3640047c0 0x7fb364004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.605+0000 7fb37fe00640 1 -- 172.21.15.102:0/672440921 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb364003970 msgr2=0x7fb364003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.605+0000 7fb37fe00640 1 --2- 172.21.15.102:0/672440921 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb364003970 0x7fb364003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.605+0000 7fb37fe00640 1 -- 172.21.15.102:0/672440921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3640047c0 msgr2=0x7fb364004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.605+0000 7fb37fe00640 1 --2- 172.21.15.102:0/672440921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3640047c0 0x7fb364004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.605+0000 7fb37fe00640 1 -- 172.21.15.102:0/672440921 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb364003520 con 0x7fb364005180 2024-03-23T03:49:57.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.605+0000 7fb377e00640 1 --2- 172.21.15.102:0/672440921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3640047c0 0x7fb364004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:57.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.606+0000 7fb37fe00640 1 -- 172.21.15.102:0/672440921 reap_dead start 2024-03-23T03:49:57.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.606+0000 7fb37fe00640 1 --2- 172.21.15.102:0/672440921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb364005180 0x7fb36400b980 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fb378009fc0 tx=0x7fb3640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=397e00559898977c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:57.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.607+0000 7fb37ea00640 1 -- 172.21.15.102:0/672440921 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37802cc60 con 0x7fb364005180 2024-03-23T03:49:57.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.607+0000 7fb37ea00640 1 -- 172.21.15.102:0/672440921 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37800f040 con 0x7fb364005180 2024-03-23T03:49:57.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.607+0000 7fb37ea00640 1 -- 172.21.15.102:0/672440921 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb378003990 con 0x7fb364005180 2024-03-23T03:49:57.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.607+0000 7fb386400640 1 -- 172.21.15.102:0/672440921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb364005180 msgr2=0x7fb36400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:57.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.607+0000 7fb386400640 1 --2- 172.21.15.102:0/672440921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb364005180 0x7fb36400b980 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fb378009fc0 tx=0x7fb3640057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:57.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.607+0000 7fb386400640 1 -- 172.21.15.102:0/672440921 shutdown_connections 2024-03-23T03:49:57.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.607+0000 7fb386400640 1 --2- 172.21.15.102:0/672440921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb364005180 0x7fb36400b980 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:57.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.607+0000 7fb386400640 1 -- 172.21.15.102:0/672440921 >> 172.21.15.102:0/672440921 conn(0x7fb38006a0b0 msgr2=0x7fb38006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:57.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.607+0000 7fb386400640 1 -- 172.21.15.102:0/672440921 shutdown_connections 2024-03-23T03:49:57.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.607+0000 7fb386400640 1 -- 172.21.15.102:0/672440921 wait complete. 2024-03-23T03:49:57.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.609+0000 7fb386400640 1 Processor -- start 2024-03-23T03:49:57.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.609+0000 7fb386400640 1 -- start start 2024-03-23T03:49:57.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.610+0000 7fb386400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb364005180 0x7fb38015c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:57.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.610+0000 7fb386400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb380145af0 0x7fb3801f17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:57.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.610+0000 7fb386400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb38015e110 0x7fb38015e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:57.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.610+0000 7fb386400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38015ead0 con 0x7fb380145af0 2024-03-23T03:49:57.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.610+0000 7fb386400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb38015c9a0 con 0x7fb364005180 2024-03-23T03:49:57.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.610+0000 7fb386400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38015cc80 con 0x7fb38015e110 2024-03-23T03:49:57.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.611+0000 7fb377e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb380145af0 0x7fb3801f17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:57.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.611+0000 7fb377e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb380145af0 0x7fb3801f17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40232/0 (socket says 172.21.15.102:40232) 2024-03-23T03:49:57.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.611+0000 7fb377e00640 1 -- 172.21.15.102:0/1023044793 learned_addr learned my addr 172.21.15.102:0/1023044793 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:57.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.611+0000 7fb377e00640 1 -- 172.21.15.102:0/1023044793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb38015e110 msgr2=0x7fb38015e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:57.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.611+0000 7fb377e00640 1 --2- 172.21.15.102:0/1023044793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb38015e110 0x7fb38015e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:57.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.611+0000 7fb377e00640 1 -- 172.21.15.102:0/1023044793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb364005180 msgr2=0x7fb38015c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:57.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.611+0000 7fb377e00640 1 --2- 172.21.15.102:0/1023044793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb364005180 0x7fb38015c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:57.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.611+0000 7fb377e00640 1 -- 172.21.15.102:0/1023044793 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38015cf60 con 0x7fb380145af0 2024-03-23T03:49:57.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.611+0000 7fb377e00640 1 --2- 172.21.15.102:0/1023044793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb380145af0 0x7fb3801f17f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb368000ff0 tx=0x7fb36800dc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:57.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.611+0000 7fb377400640 1 -- 172.21.15.102:0/1023044793 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3680074e0 con 0x7fb380145af0 2024-03-23T03:49:57.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.612+0000 7fb377400640 1 -- 172.21.15.102:0/1023044793 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb368007640 con 0x7fb380145af0 2024-03-23T03:49:57.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.612+0000 7fb377400640 1 -- 172.21.15.102:0/1023044793 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3680049e0 con 0x7fb380145af0 2024-03-23T03:49:57.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.612+0000 7fb386400640 1 -- 172.21.15.102:0/1023044793 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb38015d240 con 0x7fb380145af0 2024-03-23T03:49:57.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.612+0000 7fb386400640 1 -- 172.21.15.102:0/1023044793 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb380145f30 con 0x7fb380145af0 2024-03-23T03:49:57.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.613+0000 7fb386400640 1 -- 172.21.15.102:0/1023044793 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb380073390 con 0x7fb380145af0 2024-03-23T03:49:57.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.614+0000 7fb377400640 1 -- 172.21.15.102:0/1023044793 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb368004b40 con 0x7fb380145af0 2024-03-23T03:49:57.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.614+0000 7fb377400640 1 --2- 172.21.15.102:0/1023044793 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb35c06ac30 0x7fb35c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:57.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.615+0000 7fb377400640 1 -- 172.21.15.102:0/1023044793 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6260+0+0 (secure 0 0 0) 0x7fb368088f60 con 0x7fb380145af0 2024-03-23T03:49:57.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.615+0000 7fb37f400640 1 --2- 172.21.15.102:0/1023044793 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb35c06ac30 0x7fb35c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:57.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.615+0000 7fb37f400640 1 --2- 172.21.15.102:0/1023044793 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb35c06ac30 0x7fb35c06d0f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb3700015d0 tx=0x7fb370009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:57.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.619+0000 7fb377400640 1 -- 172.21.15.102:0/1023044793 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb368053e60 con 0x7fb380145af0 2024-03-23T03:49:57.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:57.814+0000 7fb386400640 1 -- 172.21.15.102:0/1023044793 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb38006c540 con 0x7fb380145af0 2024-03-23T03:49:58.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.385+0000 7fb377400640 1 -- 172.21.15.102:0/1023044793 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v71) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb368019070 con 0x7fb380145af0 2024-03-23T03:49:58.387 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' removed 2024-03-23T03:49:58.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.389+0000 7fb386400640 1 -- 172.21.15.102:0/1023044793 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb35c06ac30 msgr2=0x7fb35c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:58.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.389+0000 7fb386400640 1 --2- 172.21.15.102:0/1023044793 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb35c06ac30 0x7fb35c06d0f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb3700015d0 tx=0x7fb370009040 comp rx=0 tx=0).stop 2024-03-23T03:49:58.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.389+0000 7fb386400640 1 -- 172.21.15.102:0/1023044793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb380145af0 msgr2=0x7fb3801f17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:58.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.389+0000 7fb386400640 1 --2- 172.21.15.102:0/1023044793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb380145af0 0x7fb3801f17f0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb368000ff0 tx=0x7fb36800dc70 comp rx=0 tx=0).stop 2024-03-23T03:49:58.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.390+0000 7fb386400640 1 -- 172.21.15.102:0/1023044793 shutdown_connections 2024-03-23T03:49:58.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.390+0000 7fb386400640 1 --2- 172.21.15.102:0/1023044793 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb35c06ac30 0x7fb35c06d0f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:58.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.390+0000 7fb386400640 1 --2- 172.21.15.102:0/1023044793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb38015e110 0x7fb38015e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:58.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.390+0000 7fb386400640 1 --2- 172.21.15.102:0/1023044793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb380145af0 0x7fb3801f17f0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:58.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.390+0000 7fb386400640 1 --2- 172.21.15.102:0/1023044793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb364005180 0x7fb38015c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:58.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.390+0000 7fb386400640 1 -- 172.21.15.102:0/1023044793 >> 172.21.15.102:0/1023044793 conn(0x7fb38006a0b0 msgr2=0x7fb3800af800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:58.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.390+0000 7fb386400640 1 -- 172.21.15.102:0/1023044793 shutdown_connections 2024-03-23T03:49:58.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.390+0000 7fb386400640 1 -- 172.21.15.102:0/1023044793 wait complete. 2024-03-23T03:49:58.412 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T03:49:58.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.615+0000 7fe44a200640 1 Processor -- start 2024-03-23T03:49:58.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.616+0000 7fe44a200640 1 -- start start 2024-03-23T03:49:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.617+0000 7fe44a200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe44406a3a0 0x7fe4440b23c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.617+0000 7fe44a200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44406aa30 con 0x7fe4440ada70 2024-03-23T03:49:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.617+0000 7fe44a200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4440b3f40 con 0x7fe4440b08c0 2024-03-23T03:49:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.617+0000 7fe44a200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe44415cee0 con 0x7fe44406a3a0 2024-03-23T03:49:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.617+0000 7fe448e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe4440ada70 0x7fe44415d5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47494/0 (socket says 172.21.15.102:47494) 2024-03-23T03:49:58.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.617+0000 7fe448e00640 1 -- 172.21.15.102:0/479373142 learned_addr learned my addr 172.21.15.102:0/479373142 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:49:58.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.618+0000 7fe443400640 1 -- 172.21.15.102:0/479373142 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103670596 0 0) 0x7fe44406aa30 con 0x7fe4440ada70 2024-03-23T03:49:58.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.618+0000 7fe449800640 1 --2- 172.21.15.102:0/479373142 >> v2:172.21.15.173:3300/0 conn(0x7fe44406a3a0 0x7fe4440b23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:58.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.618+0000 7fe443400640 1 -- 172.21.15.102:0/479373142 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe428003520 con 0x7fe4440ada70 2024-03-23T03:49:58.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.618+0000 7fe443400640 1 -- 172.21.15.102:0/479373142 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1088721294 0 0) 0x7fe428003520 con 0x7fe4440ada70 2024-03-23T03:49:58.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.618+0000 7fe443400640 1 -- 172.21.15.102:0/479373142 >> v2:172.21.15.173:3300/0 conn(0x7fe44406a3a0 msgr2=0x7fe4440b23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:58.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.618+0000 7fe443400640 1 --2- 172.21.15.102:0/479373142 >> v2:172.21.15.173:3300/0 conn(0x7fe44406a3a0 0x7fe4440b23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:58.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.618+0000 7fe443400640 1 -- 172.21.15.102:0/479373142 >> v1:172.21.15.173:6789/0 conn(0x7fe4440b08c0 legacy=0x7fe444069c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.618+0000 7fe443400640 1 -- 172.21.15.102:0/479373142 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4440b41c0 con 0x7fe4440ada70 2024-03-23T03:49:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.618+0000 7fe443400640 1 -- 172.21.15.102:0/479373142 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fe4340019f0 con 0x7fe4440ada70 2024-03-23T03:49:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.619+0000 7fe443400640 1 -- 172.21.15.102:0/479373142 >> v1:172.21.15.102:6789/0 conn(0x7fe4440ada70 legacy=0x7fe44415d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.619+0000 7fe443400640 1 --2- 172.21.15.102:0/479373142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe428003970 0x7fe428003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.619+0000 7fe443400640 1 --2- 172.21.15.102:0/479373142 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe4280047c0 0x7fe428004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.619+0000 7fe443400640 1 --2- 172.21.15.102:0/479373142 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe428005180 0x7fe42800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.619+0000 7fe443400640 1 -- 172.21.15.102:0/479373142 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44415cee0 con 0x7fe428003970 2024-03-23T03:49:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.619+0000 7fe443400640 1 -- 172.21.15.102:0/479373142 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe42800c020 con 0x7fe428005180 2024-03-23T03:49:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.619+0000 7fe443400640 1 -- 172.21.15.102:0/479373142 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe42800c1e0 con 0x7fe4280047c0 2024-03-23T03:49:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.619+0000 7fe448e00640 1 --2- 172.21.15.102:0/479373142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe428003970 0x7fe428003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.619+0000 7fe448e00640 1 -- 172.21.15.102:0/479373142 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe4280047c0 msgr2=0x7fe428004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.619+0000 7fe448e00640 1 --2- 172.21.15.102:0/479373142 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe4280047c0 0x7fe428004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:58.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.619+0000 7fe448e00640 1 -- 172.21.15.102:0/479373142 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe428005180 msgr2=0x7fe42800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:58.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.619+0000 7fe448e00640 1 --2- 172.21.15.102:0/479373142 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe428005180 0x7fe42800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:58.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.619+0000 7fe448e00640 1 -- 172.21.15.102:0/479373142 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe428003520 con 0x7fe428003970 2024-03-23T03:49:58.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.619+0000 7fe449800640 1 -- 172.21.15.102:0/479373142 reap_dead start 2024-03-23T03:49:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.620+0000 7fe448e00640 1 --2- 172.21.15.102:0/479373142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe428003970 0x7fe428003d90 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fe43400a820 tx=0x7fe4280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2bc1dd74dccc36 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.621+0000 7fe443400640 1 -- 172.21.15.102:0/479373142 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4340072d0 con 0x7fe428003970 2024-03-23T03:49:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.621+0000 7fe443400640 1 -- 172.21.15.102:0/479373142 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe43402f070 con 0x7fe428003970 2024-03-23T03:49:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.621+0000 7fe443400640 1 -- 172.21.15.102:0/479373142 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe434007730 con 0x7fe428003970 2024-03-23T03:49:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.621+0000 7fe44a200640 1 -- 172.21.15.102:0/479373142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe428003970 msgr2=0x7fe428003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.621+0000 7fe44a200640 1 --2- 172.21.15.102:0/479373142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe428003970 0x7fe428003d90 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fe43400a820 tx=0x7fe4280057f0 comp rx=0 tx=0).stop 2024-03-23T03:49:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.621+0000 7fe44a200640 1 -- 172.21.15.102:0/479373142 shutdown_connections 2024-03-23T03:49:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.621+0000 7fe44a200640 1 --2- 172.21.15.102:0/479373142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe428003970 0x7fe428003d90 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.621+0000 7fe44a200640 1 -- 172.21.15.102:0/479373142 >> 172.21.15.102:0/479373142 conn(0x7fe4440b48e0 msgr2=0x7fe4440b4cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:58.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.622+0000 7fe44a200640 1 -- 172.21.15.102:0/479373142 shutdown_connections 2024-03-23T03:49:58.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.622+0000 7fe44a200640 1 -- 172.21.15.102:0/479373142 wait complete. 2024-03-23T03:49:58.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.623+0000 7fe44a200640 1 Processor -- start 2024-03-23T03:49:58.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.623+0000 7fe44a200640 1 -- start start 2024-03-23T03:49:58.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.624+0000 7fe44a200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe428003970 0x7fe444144150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:58.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.624+0000 7fe44a200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe44406d800 0x7fe4441567d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:58.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.624+0000 7fe44a200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe44406dc60 0x7fe444156d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:58.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.624+0000 7fe44a200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe444144690 con 0x7fe44406d800 2024-03-23T03:49:58.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.624+0000 7fe44a200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4441f59c0 con 0x7fe428003970 2024-03-23T03:49:58.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.624+0000 7fe44a200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4441f5ca0 con 0x7fe44406dc60 2024-03-23T03:49:58.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.625+0000 7fe443e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe44406d800 0x7fe4441567d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:58.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.625+0000 7fe443e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe44406d800 0x7fe4441567d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40250/0 (socket says 172.21.15.102:40250) 2024-03-23T03:49:58.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.625+0000 7fe443e00640 1 -- 172.21.15.102:0/3413825190 learned_addr learned my addr 172.21.15.102:0/3413825190 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:58.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.625+0000 7fe448e00640 1 --2- 172.21.15.102:0/3413825190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe428003970 0x7fe444144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:58.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.625+0000 7fe449800640 1 --2- 172.21.15.102:0/3413825190 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe44406dc60 0x7fe444156d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:58.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.625+0000 7fe443e00640 1 -- 172.21.15.102:0/3413825190 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe44406dc60 msgr2=0x7fe444156d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:58.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.625+0000 7fe443e00640 1 --2- 172.21.15.102:0/3413825190 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe44406dc60 0x7fe444156d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:58.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.625+0000 7fe443e00640 1 -- 172.21.15.102:0/3413825190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe428003970 msgr2=0x7fe444144150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:58.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.625+0000 7fe443e00640 1 --2- 172.21.15.102:0/3413825190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe428003970 0x7fe444144150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:58.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.625+0000 7fe443e00640 1 -- 172.21.15.102:0/3413825190 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4441f5f80 con 0x7fe44406d800 2024-03-23T03:49:58.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.625+0000 7fe443e00640 1 --2- 172.21.15.102:0/3413825190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe44406d800 0x7fe4441567d0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fe42c00c3c0 tx=0x7fe42c001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:58.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.626+0000 7fe440c00640 1 -- 172.21.15.102:0/3413825190 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42c0052d0 con 0x7fe44406d800 2024-03-23T03:49:58.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.626+0000 7fe440c00640 1 -- 172.21.15.102:0/3413825190 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42c005450 con 0x7fe44406d800 2024-03-23T03:49:58.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.626+0000 7fe440c00640 1 -- 172.21.15.102:0/3413825190 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42c003830 con 0x7fe44406d800 2024-03-23T03:49:58.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.626+0000 7fe44a200640 1 -- 172.21.15.102:0/3413825190 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4441f6260 con 0x7fe44406d800 2024-03-23T03:49:58.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.626+0000 7fe44a200640 1 -- 172.21.15.102:0/3413825190 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe444146a30 con 0x7fe44406d800 2024-03-23T03:49:58.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.627+0000 7fe44a200640 1 -- 172.21.15.102:0/3413825190 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4441f68b0 con 0x7fe44406d800 2024-03-23T03:49:58.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.628+0000 7fe440c00640 1 -- 172.21.15.102:0/3413825190 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe42c003990 con 0x7fe44406d800 2024-03-23T03:49:58.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.629+0000 7fe440c00640 1 --2- 172.21.15.102:0/3413825190 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe42006ac30 0x7fe42006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:58.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.629+0000 7fe440c00640 1 -- 172.21.15.102:0/3413825190 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5793+0+0 (secure 0 0 0) 0x7fe42c0888b0 con 0x7fe44406d800 2024-03-23T03:49:58.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.629+0000 7fe448e00640 1 --2- 172.21.15.102:0/3413825190 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe42006ac30 0x7fe42006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:58.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.630+0000 7fe448e00640 1 --2- 172.21.15.102:0/3413825190 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe42006ac30 0x7fe42006d0f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe434034000 tx=0x7fe434007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:58.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.634+0000 7fe440c00640 1 -- 172.21.15.102:0/3413825190 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe4441f68b0 con 0x7fe44406d800 2024-03-23T03:49:58.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:58.831+0000 7fe44a200640 1 -- 172.21.15.102:0/3413825190 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe4441f68b0 con 0x7fe44406d800 2024-03-23T03:49:59.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.385+0000 7fe440c00640 1 -- 172.21.15.102:0/3413825190 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v72) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe4441f68b0 con 0x7fe44406d800 2024-03-23T03:49:59.386 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' removed 2024-03-23T03:49:59.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.390+0000 7fe44a200640 1 -- 172.21.15.102:0/3413825190 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe42006ac30 msgr2=0x7fe42006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:59.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.390+0000 7fe44a200640 1 --2- 172.21.15.102:0/3413825190 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe42006ac30 0x7fe42006d0f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe434034000 tx=0x7fe434007000 comp rx=0 tx=0).stop 2024-03-23T03:49:59.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.390+0000 7fe44a200640 1 -- 172.21.15.102:0/3413825190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe44406d800 msgr2=0x7fe4441567d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:59.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.390+0000 7fe44a200640 1 --2- 172.21.15.102:0/3413825190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe44406d800 0x7fe4441567d0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fe42c00c3c0 tx=0x7fe42c001e70 comp rx=0 tx=0).stop 2024-03-23T03:49:59.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.390+0000 7fe44a200640 1 -- 172.21.15.102:0/3413825190 shutdown_connections 2024-03-23T03:49:59.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.390+0000 7fe44a200640 1 --2- 172.21.15.102:0/3413825190 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe42006ac30 0x7fe42006d0f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:59.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.390+0000 7fe44a200640 1 --2- 172.21.15.102:0/3413825190 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe44406dc60 0x7fe444156d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:59.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.390+0000 7fe44a200640 1 --2- 172.21.15.102:0/3413825190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe44406d800 0x7fe4441567d0 secure :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fe42c00c3c0 tx=0x7fe42c001e70 comp rx=0 tx=0).stop 2024-03-23T03:49:59.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.390+0000 7fe44a200640 1 --2- 172.21.15.102:0/3413825190 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe428003970 0x7fe444144150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:59.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.390+0000 7fe44a200640 1 -- 172.21.15.102:0/3413825190 >> 172.21.15.102:0/3413825190 conn(0x7fe4440b48e0 msgr2=0x7fe4440af3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:59.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.391+0000 7fe44a200640 1 -- 172.21.15.102:0/3413825190 shutdown_connections 2024-03-23T03:49:59.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.391+0000 7fe44a200640 1 -- 172.21.15.102:0/3413825190 wait complete. 2024-03-23T03:49:59.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-23T03:49:59.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.609+0000 7f37fb800640 1 Processor -- start 2024-03-23T03:49:59.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.609+0000 7f37fb800640 1 -- start start 2024-03-23T03:49:59.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.610+0000 7f37fb800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37f40ce930 0x7f37f40add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:59.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.610+0000 7f37fb800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f37f40cd3e0 0x7f37f40ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:59.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.610+0000 7f37fb800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f37f406ce90 0x7f37f406d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:59.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.610+0000 7f37fb800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f406c760 con 0x7f37f40ce930 2024-03-23T03:49:59.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.610+0000 7f37fb800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f40b42d0 con 0x7f37f40cd3e0 2024-03-23T03:49:59.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.610+0000 7f37fb800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f40cf250 con 0x7f37f406ce90 2024-03-23T03:49:59.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.611+0000 7f37f8e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37f40ce930 0x7f37f40add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:59.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.611+0000 7f37f8e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f37f40ce930 0x7f37f40add30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40266/0 (socket says 172.21.15.102:40266) 2024-03-23T03:49:59.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.611+0000 7f37f8e00640 1 -- 172.21.15.102:0/2017535408 learned_addr learned my addr 172.21.15.102:0/2017535408 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:59.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.611+0000 7f37f8e00640 1 -- 172.21.15.102:0/2017535408 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f37f406ce90 msgr2=0x7f37f406d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:59.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.611+0000 7f37f8e00640 1 --2- 172.21.15.102:0/2017535408 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f37f406ce90 0x7f37f406d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:59.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.611+0000 7f37f8e00640 1 -- 172.21.15.102:0/2017535408 >> v2:172.21.15.173:3300/0 conn(0x7f37f40cd3e0 msgr2=0x7f37f40ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:49:59.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.611+0000 7f37ebe00640 1 --2- 172.21.15.102:0/2017535408 >> v2:172.21.15.173:3300/0 conn(0x7f37f40cd3e0 0x7f37f40ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:59.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.611+0000 7f37f8e00640 1 --2- 172.21.15.102:0/2017535408 >> v2:172.21.15.173:3300/0 conn(0x7f37f40cd3e0 0x7f37f40ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:59.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.611+0000 7f37f8e00640 1 -- 172.21.15.102:0/2017535408 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f40cdba0 con 0x7f37f40ce930 2024-03-23T03:49:59.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.611+0000 7f37ebe00640 1 --2- 172.21.15.102:0/2017535408 >> v2:172.21.15.173:3300/0 conn(0x7f37f40cd3e0 0x7f37f40ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:49:59.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.613+0000 7f37f8e00640 1 --2- 172.21.15.102:0/2017535408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37f40ce930 0x7f37f40add30 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f37f40cf4f0 tx=0x7f37e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7d1961f026a9f6a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:59.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.613+0000 7f37f3e00640 1 -- 172.21.15.102:0/2017535408 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e4031870 con 0x7f37f40ce930 2024-03-23T03:49:59.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.613+0000 7f37f3e00640 1 -- 172.21.15.102:0/2017535408 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e40319d0 con 0x7f37f40ce930 2024-03-23T03:49:59.614 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.613+0000 7f37f3e00640 1 -- 172.21.15.102:0/2017535408 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e4031d60 con 0x7f37f40ce930 2024-03-23T03:49:59.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.614+0000 7f37fb800640 1 -- 172.21.15.102:0/2017535408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37f40ce930 msgr2=0x7f37f40add30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:59.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.614+0000 7f37fb800640 1 --2- 172.21.15.102:0/2017535408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37f40ce930 0x7f37f40add30 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f37f40cf4f0 tx=0x7f37e402b910 comp rx=0 tx=0).stop 2024-03-23T03:49:59.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.614+0000 7f37fb800640 1 -- 172.21.15.102:0/2017535408 shutdown_connections 2024-03-23T03:49:59.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.614+0000 7f37fb800640 1 --2- 172.21.15.102:0/2017535408 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f37f406ce90 0x7f37f406d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:59.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.614+0000 7f37fb800640 1 --2- 172.21.15.102:0/2017535408 >> v2:172.21.15.173:3300/0 conn(0x7f37f40cd3e0 0x7f37f40ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:59.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.614+0000 7f37fb800640 1 --2- 172.21.15.102:0/2017535408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37f40ce930 0x7f37f40add30 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:59.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.614+0000 7f37fb800640 1 -- 172.21.15.102:0/2017535408 >> 172.21.15.102:0/2017535408 conn(0x7f37f406a0b0 msgr2=0x7f37f406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:59.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.614+0000 7f37fb800640 1 -- 172.21.15.102:0/2017535408 shutdown_connections 2024-03-23T03:49:59.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.615+0000 7f37fb800640 1 -- 172.21.15.102:0/2017535408 wait complete. 2024-03-23T03:49:59.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.618+0000 7f37fb800640 1 Processor -- start 2024-03-23T03:49:59.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.618+0000 7f37fb800640 1 -- start start 2024-03-23T03:49:59.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.619+0000 7f37fb800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37f406ce90 0x7f37f4148aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:59.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.619+0000 7f37fb800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f37f40cd3e0 0x7f37f415a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.619+0000 7f37fb800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37f40ce930 0x7f37f414e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.619+0000 7f37fb800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f415d130 con 0x7f37f406ce90 2024-03-23T03:49:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.619+0000 7f37fb800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f415c090 con 0x7f37f40ce930 2024-03-23T03:49:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.619+0000 7f37fb800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f415c370 con 0x7f37f40cd3e0 2024-03-23T03:49:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.619+0000 7f37f8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37f406ce90 0x7f37f4148aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.620+0000 7f37f8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37f406ce90 0x7f37f4148aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40276/0 (socket says 172.21.15.102:40276) 2024-03-23T03:49:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.620+0000 7f37f8e00640 1 -- 172.21.15.102:0/1433763131 learned_addr learned my addr 172.21.15.102:0/1433763131 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:49:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.620+0000 7f37ebe00640 1 --2- 172.21.15.102:0/1433763131 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f37f40cd3e0 0x7f37f415a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.620+0000 7f37f9800640 1 --2- 172.21.15.102:0/1433763131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37f40ce930 0x7f37f414e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.620+0000 7f37f8e00640 1 -- 172.21.15.102:0/1433763131 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f37f40cd3e0 msgr2=0x7f37f415a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.620+0000 7f37f8e00640 1 --2- 172.21.15.102:0/1433763131 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f37f40cd3e0 0x7f37f415a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.620+0000 7f37f8e00640 1 -- 172.21.15.102:0/1433763131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37f40ce930 msgr2=0x7f37f414e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.620+0000 7f37f8e00640 1 --2- 172.21.15.102:0/1433763131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37f40ce930 0x7f37f414e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.620+0000 7f37f8e00640 1 -- 172.21.15.102:0/1433763131 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e40094d0 con 0x7f37f406ce90 2024-03-23T03:49:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.620+0000 7f37f9800640 1 --2- 172.21.15.102:0/1433763131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37f40ce930 0x7f37f414e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:49:59.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.621+0000 7f37f8e00640 1 --2- 172.21.15.102:0/1433763131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37f406ce90 0x7f37f4148aa0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f37e4000c00 tx=0x7f37e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:59.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.621+0000 7f37f1600640 1 -- 172.21.15.102:0/1433763131 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e4031840 con 0x7f37f406ce90 2024-03-23T03:49:59.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.621+0000 7f37f1600640 1 -- 172.21.15.102:0/1433763131 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e40443f0 con 0x7f37f406ce90 2024-03-23T03:49:59.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.621+0000 7f37f1600640 1 -- 172.21.15.102:0/1433763131 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e40447a0 con 0x7f37f406ce90 2024-03-23T03:49:59.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.621+0000 7f37fb800640 1 -- 172.21.15.102:0/1433763131 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f415c650 con 0x7f37f406ce90 2024-03-23T03:49:59.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.622+0000 7f37fb800640 1 -- 172.21.15.102:0/1433763131 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f415cc10 con 0x7f37f406ce90 2024-03-23T03:49:59.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.623+0000 7f37fb800640 1 -- 172.21.15.102:0/1433763131 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f4073390 con 0x7f37f406ce90 2024-03-23T03:49:59.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.624+0000 7f37f1600640 1 -- 172.21.15.102:0/1433763131 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f37e404d450 con 0x7f37f406ce90 2024-03-23T03:49:59.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.626+0000 7f37f1600640 1 --2- 172.21.15.102:0/1433763131 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f37d006ac00 0x7f37d006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:49:59.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.626+0000 7f37f1600640 1 -- 172.21.15.102:0/1433763131 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f37e402d040 con 0x7f37f406ce90 2024-03-23T03:49:59.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.626+0000 7f37ebe00640 1 --2- 172.21.15.102:0/1433763131 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f37d006ac00 0x7f37d006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:49:59.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.627+0000 7f37ebe00640 1 --2- 172.21.15.102:0/1433763131 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f37d006ac00 0x7f37d006d0c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f37f414c190 tx=0x7f37dc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:49:59.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.630+0000 7f37f1600640 1 -- 172.21.15.102:0/1433763131 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f37e4076d40 con 0x7f37f406ce90 2024-03-23T03:49:59.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.819+0000 7f37fb800640 1 -- 172.21.15.102:0/1433763131 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f37f406c540 con 0x7f37f406ce90 2024-03-23T03:49:59.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.827+0000 7f37f1600640 1 -- 172.21.15.102:0/1433763131 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f37e407a360 con 0x7f37f406ce90 2024-03-23T03:49:59.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.830+0000 7f37fb800640 1 -- 172.21.15.102:0/1433763131 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f37d006ac00 msgr2=0x7f37d006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:59.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.830+0000 7f37fb800640 1 --2- 172.21.15.102:0/1433763131 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f37d006ac00 0x7f37d006d0c0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f37f414c190 tx=0x7f37dc008040 comp rx=0 tx=0).stop 2024-03-23T03:49:59.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.830+0000 7f37fb800640 1 -- 172.21.15.102:0/1433763131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37f406ce90 msgr2=0x7f37f4148aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:49:59.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.830+0000 7f37fb800640 1 --2- 172.21.15.102:0/1433763131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37f406ce90 0x7f37f4148aa0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f37e4000c00 tx=0x7f37e40046c0 comp rx=0 tx=0).stop 2024-03-23T03:49:59.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.830+0000 7f37fb800640 1 -- 172.21.15.102:0/1433763131 shutdown_connections 2024-03-23T03:49:59.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.830+0000 7f37fb800640 1 --2- 172.21.15.102:0/1433763131 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f37d006ac00 0x7f37d006d0c0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:59.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.830+0000 7f37fb800640 1 --2- 172.21.15.102:0/1433763131 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f37f40ce930 0x7f37f414e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:59.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.830+0000 7f37fb800640 1 --2- 172.21.15.102:0/1433763131 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f37f40cd3e0 0x7f37f415a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:59.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.830+0000 7f37fb800640 1 --2- 172.21.15.102:0/1433763131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37f406ce90 0x7f37f4148aa0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:49:59.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.830+0000 7f37fb800640 1 -- 172.21.15.102:0/1433763131 >> 172.21.15.102:0/1433763131 conn(0x7f37f406a0b0 msgr2=0x7f37f406a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:49:59.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.830+0000 7f37fb800640 1 -- 172.21.15.102:0/1433763131 shutdown_connections 2024-03-23T03:49:59.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:49:59.830+0000 7f37fb800640 1 -- 172.21.15.102:0/1433763131 wait complete. 2024-03-23T03:49:59.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values' 2024-03-23T03:50:00.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.047+0000 7f06c5600640 1 Processor -- start 2024-03-23T03:50:00.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.047+0000 7f06c5600640 1 -- start start 2024-03-23T03:50:00.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.048+0000 7f06c5600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f06c00b39b0 0x7f06c00b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:00.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.049+0000 7f06c5600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f06c015c400 0x7f06c015c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:00.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.049+0000 7f06c5600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06c006c6f0 con 0x7f06c015b560 2024-03-23T03:50:00.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.049+0000 7f06c5600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c00b4370 con 0x7f06c015c400 2024-03-23T03:50:00.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.049+0000 7f06c5600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06c006c9d0 con 0x7f06c00b39b0 2024-03-23T03:50:00.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.049+0000 7f06bea00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f06c015b560 0x7f06c015b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47510/0 (socket says 172.21.15.102:47510) 2024-03-23T03:50:00.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.049+0000 7f06bea00640 1 -- 172.21.15.102:0/3302049724 learned_addr learned my addr 172.21.15.102:0/3302049724 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:00.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.049+0000 7f06bf400640 1 --2- 172.21.15.102:0/3302049724 >> v2:172.21.15.102:3300/0 conn(0x7f06c015c400 0x7f06c015c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:00.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.049+0000 7f06be000640 1 --2- 172.21.15.102:0/3302049724 >> v2:172.21.15.173:3300/0 conn(0x7f06c00b39b0 0x7f06c00b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:00.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.050+0000 7f06bd600640 1 -- 172.21.15.102:0/3302049724 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535971056 0 0) 0x7f06c006c6f0 con 0x7f06c015b560 2024-03-23T03:50:00.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.050+0000 7f06bd600640 1 -- 172.21.15.102:0/3302049724 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06a4003520 con 0x7f06c015b560 2024-03-23T03:50:00.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.050+0000 7f06bf400640 1 -- 172.21.15.102:0/3302049724 >> v2:172.21.15.173:3300/0 conn(0x7f06c00b39b0 msgr2=0x7f06c00b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:00.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.050+0000 7f06bf400640 1 --2- 172.21.15.102:0/3302049724 >> v2:172.21.15.173:3300/0 conn(0x7f06c00b39b0 0x7f06c00b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:00.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.050+0000 7f06bf400640 1 -- 172.21.15.102:0/3302049724 >> v1:172.21.15.102:6789/0 conn(0x7f06c015b560 legacy=0x7f06c015b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:00.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.050+0000 7f06bf400640 1 -- 172.21.15.102:0/3302049724 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06c006ccb0 con 0x7f06c015c400 2024-03-23T03:50:00.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.051+0000 7f06bf400640 1 --2- 172.21.15.102:0/3302049724 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06c015c400 0x7f06c015c830 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f06c00ce180 tx=0x7f06b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8fac9b0cdb6ab19 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:00.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.051+0000 7f06bd600640 1 -- 172.21.15.102:0/3302049724 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b802cc80 con 0x7f06c015c400 2024-03-23T03:50:00.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.051+0000 7f06bd600640 1 -- 172.21.15.102:0/3302049724 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06b802cde0 con 0x7f06c015c400 2024-03-23T03:50:00.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.051+0000 7f06bd600640 1 -- 172.21.15.102:0/3302049724 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06b8034640 con 0x7f06c015c400 2024-03-23T03:50:00.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.052+0000 7f06c5600640 1 -- 172.21.15.102:0/3302049724 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06c015c400 msgr2=0x7f06c015c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:00.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.052+0000 7f06c5600640 1 --2- 172.21.15.102:0/3302049724 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06c015c400 0x7f06c015c830 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f06c00ce180 tx=0x7f06b802bed0 comp rx=0 tx=0).stop 2024-03-23T03:50:00.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.052+0000 7f06c5600640 1 -- 172.21.15.102:0/3302049724 shutdown_connections 2024-03-23T03:50:00.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.052+0000 7f06c5600640 1 --2- 172.21.15.102:0/3302049724 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06c015c400 0x7f06c015c830 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:00.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.052+0000 7f06c5600640 1 --2- 172.21.15.102:0/3302049724 >> v2:172.21.15.173:3300/0 conn(0x7f06c00b39b0 0x7f06c00b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:00.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.052+0000 7f06c5600640 1 -- 172.21.15.102:0/3302049724 >> 172.21.15.102:0/3302049724 conn(0x7f06c006a3d0 msgr2=0x7f06c00b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:00.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.052+0000 7f06c5600640 1 -- 172.21.15.102:0/3302049724 shutdown_connections 2024-03-23T03:50:00.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.052+0000 7f06c5600640 1 -- 172.21.15.102:0/3302049724 wait complete. 2024-03-23T03:50:00.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.053+0000 7f06c5600640 1 Processor -- start 2024-03-23T03:50:00.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.054+0000 7f06c5600640 1 -- start start 2024-03-23T03:50:00.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.054+0000 7f06c5600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f06c00b39b0 0x7f06c0146200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:00.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.054+0000 7f06c5600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06c015b560 0x7f06c0155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:00.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.054+0000 7f06c5600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06c015c400 0x7f06c0154140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:00.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.054+0000 7f06c5600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c0146740 con 0x7f06c015b560 2024-03-23T03:50:00.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.054+0000 7f06c5600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06c01f2520 con 0x7f06c015c400 2024-03-23T03:50:00.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.054+0000 7f06c5600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06c01f1910 con 0x7f06c00b39b0 2024-03-23T03:50:00.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.055+0000 7f06be000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06c015b560 0x7f06c0155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:00.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.055+0000 7f06be000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06c015b560 0x7f06c0155e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40300/0 (socket says 172.21.15.102:40300) 2024-03-23T03:50:00.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.055+0000 7f06be000640 1 -- 172.21.15.102:0/2829961015 learned_addr learned my addr 172.21.15.102:0/2829961015 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:00.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.055+0000 7f06be000640 1 -- 172.21.15.102:0/2829961015 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f06c00b39b0 msgr2=0x7f06c0146200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:00.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.055+0000 7f06be000640 1 --2- 172.21.15.102:0/2829961015 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f06c00b39b0 0x7f06c0146200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:00.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.055+0000 7f06be000640 1 -- 172.21.15.102:0/2829961015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06c015c400 msgr2=0x7f06c0154140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:00.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.055+0000 7f06be000640 1 --2- 172.21.15.102:0/2829961015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06c015c400 0x7f06c0154140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:00.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.055+0000 7f06be000640 1 -- 172.21.15.102:0/2829961015 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b8009a70 con 0x7f06c015b560 2024-03-23T03:50:00.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.055+0000 7f06be000640 1 --2- 172.21.15.102:0/2829961015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06c015b560 0x7f06c0155e20 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f06ac00e9b0 tx=0x7f06ac00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:00.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.056+0000 7f06b6a00640 1 -- 172.21.15.102:0/2829961015 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ac009670 con 0x7f06c015b560 2024-03-23T03:50:00.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.056+0000 7f06b6a00640 1 -- 172.21.15.102:0/2829961015 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ac00c9a0 con 0x7f06c015b560 2024-03-23T03:50:00.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.056+0000 7f06c5600640 1 -- 172.21.15.102:0/2829961015 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06c01f1bf0 con 0x7f06c015b560 2024-03-23T03:50:00.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.056+0000 7f06b6a00640 1 -- 172.21.15.102:0/2829961015 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ac00cd50 con 0x7f06c015b560 2024-03-23T03:50:00.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.056+0000 7f06c5600640 1 -- 172.21.15.102:0/2829961015 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06c01f21b0 con 0x7f06c015b560 2024-03-23T03:50:00.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.057+0000 7f06c5600640 1 -- 172.21.15.102:0/2829961015 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06c006ac20 con 0x7f06c015b560 2024-03-23T03:50:00.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.058+0000 7f06b6a00640 1 -- 172.21.15.102:0/2829961015 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f06ac013470 con 0x7f06c015b560 2024-03-23T03:50:00.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.059+0000 7f06b6a00640 1 --2- 172.21.15.102:0/2829961015 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06a006ac00 0x7f06a006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:00.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.059+0000 7f06b6a00640 1 -- 172.21.15.102:0/2829961015 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f06ac089490 con 0x7f06c015b560 2024-03-23T03:50:00.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.059+0000 7f06bea00640 1 --2- 172.21.15.102:0/2829961015 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06a006ac00 0x7f06a006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:00.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.060+0000 7f06bea00640 1 --2- 172.21.15.102:0/2829961015 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06a006ac00 0x7f06a006d0c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f06b0000f60 tx=0x7f06b00078f0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:00.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.064+0000 7f06b6a00640 1 -- 172.21.15.102:0/2829961015 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f06ac054830 con 0x7f06c015b560 2024-03-23T03:50:00.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.324+0000 7f06c5600640 1 -- 172.21.15.102:0/2829961015 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values"]} v 0) v1 -- 0x7f06c0073390 con 0x7f06c015b560 2024-03-23T03:50:00.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.398+0000 7f06b6a00640 1 -- 172.21.15.102:0/2829961015 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_human_readable_quota_values"]}]=0 v189) v1 ==== 145+0+0 (secure 0 0 0) 0x7f06ac057e50 con 0x7f06c015b560 2024-03-23T03:50:00.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.401+0000 7f06c5600640 1 -- 172.21.15.102:0/2829961015 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06a006ac00 msgr2=0x7f06a006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.401+0000 7f06c5600640 1 --2- 172.21.15.102:0/2829961015 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06a006ac00 0x7f06a006d0c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f06b0000f60 tx=0x7f06b00078f0 comp rx=0 tx=0).stop 2024-03-23T03:50:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.401+0000 7f06c5600640 1 -- 172.21.15.102:0/2829961015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06c015b560 msgr2=0x7f06c0155e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.401+0000 7f06c5600640 1 --2- 172.21.15.102:0/2829961015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06c015b560 0x7f06c0155e20 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f06ac00e9b0 tx=0x7f06ac00ee70 comp rx=0 tx=0).stop 2024-03-23T03:50:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.402+0000 7f06c5600640 1 -- 172.21.15.102:0/2829961015 shutdown_connections 2024-03-23T03:50:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.402+0000 7f06c5600640 1 --2- 172.21.15.102:0/2829961015 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06a006ac00 0x7f06a006d0c0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.402+0000 7f06c5600640 1 --2- 172.21.15.102:0/2829961015 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06c015c400 0x7f06c0154140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.402+0000 7f06c5600640 1 --2- 172.21.15.102:0/2829961015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06c015b560 0x7f06c0155e20 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.402+0000 7f06c5600640 1 --2- 172.21.15.102:0/2829961015 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f06c00b39b0 0x7f06c0146200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.402+0000 7f06c5600640 1 -- 172.21.15.102:0/2829961015 >> 172.21.15.102:0/2829961015 conn(0x7f06c006a3d0 msgr2=0x7f06c015d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.402+0000 7f06c5600640 1 -- 172.21.15.102:0/2829961015 shutdown_connections 2024-03-23T03:50:00.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.402+0000 7f06c5600640 1 -- 172.21.15.102:0/2829961015 wait complete. 2024-03-23T03:50:00.420 INFO:tasks.cephfs_test_runner:test human-readable values for setting ceph.quota.max_bytes ... ok 2024-03-23T03:50:00.420 INFO:tasks.cephfs_test_runner:Starting test: test_remote_update_df (tasks.cephfs.test_quota.TestQuota) That when a client modifies the quota on a directory used 2024-03-23T03:50:00.420 INFO:tasks.cephfs_test_runner:test_remote_update_df (tasks.cephfs.test_quota.TestQuota) 2024-03-23T03:50:00.420 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_df' 2024-03-23T03:50:00.632 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.632+0000 7f49ebc00640 1 Processor -- start 2024-03-23T03:50:00.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.632+0000 7f49ebc00640 1 -- start start 2024-03-23T03:50:00.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.633+0000 7f49ebc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f49e40b5a30 0x7f49e40b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:00.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.633+0000 7f49ebc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e40d1830 con 0x7f49e415cda0 2024-03-23T03:50:00.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.633+0000 7f49ebc00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e40b1f20 con 0x7f49e40b4350 2024-03-23T03:50:00.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.633+0000 7f49ebc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e40d0c20 con 0x7f49e40b5a30 2024-03-23T03:50:00.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.634+0000 7f49e3e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f49e40b5a30 0x7f49e40b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:00.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.634+0000 7f49e9200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f49e415cda0 0x7f49e40cdb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47522/0 (socket says 172.21.15.102:47522) 2024-03-23T03:50:00.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.634+0000 7f49e9200640 1 -- 172.21.15.102:0/3313163652 learned_addr learned my addr 172.21.15.102:0/3313163652 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:00.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.634+0000 7f49e3e00640 1 -- 172.21.15.102:0/3313163652 >> v1:172.21.15.173:6789/0 conn(0x7f49e40b4350 legacy=0x7f49e40d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:00.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.634+0000 7f49e3e00640 1 -- 172.21.15.102:0/3313163652 >> v1:172.21.15.102:6789/0 conn(0x7f49e415cda0 legacy=0x7f49e40cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:00.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.634+0000 7f49e3e00640 1 -- 172.21.15.102:0/3313163652 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e40d0f00 con 0x7f49e40b5a30 2024-03-23T03:50:00.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.634+0000 7f49e3400640 1 -- 172.21.15.102:0/3313163652 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4188737356 0 0) 0x7f49e40d1830 con 0x7f49e415cda0 2024-03-23T03:50:00.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.635+0000 7f49e3e00640 1 --2- 172.21.15.102:0/3313163652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e40b5a30 0x7f49e40b5e90 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f49d00097a0 tx=0x7f49d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bba54ee055287c10 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:00.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.636+0000 7f49e3400640 1 -- 172.21.15.102:0/3313163652 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0031830 con 0x7f49e40b5a30 2024-03-23T03:50:00.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.636+0000 7f49e3400640 1 -- 172.21.15.102:0/3313163652 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d0031990 con 0x7f49e40b5a30 2024-03-23T03:50:00.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.636+0000 7f49e3400640 1 -- 172.21.15.102:0/3313163652 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d0031d20 con 0x7f49e40b5a30 2024-03-23T03:50:00.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.636+0000 7f49ebc00640 1 -- 172.21.15.102:0/3313163652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e40b5a30 msgr2=0x7f49e40b5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:00.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.636+0000 7f49ebc00640 1 --2- 172.21.15.102:0/3313163652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e40b5a30 0x7f49e40b5e90 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f49d00097a0 tx=0x7f49d002b8d0 comp rx=0 tx=0).stop 2024-03-23T03:50:00.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.636+0000 7f49ebc00640 1 -- 172.21.15.102:0/3313163652 shutdown_connections 2024-03-23T03:50:00.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.636+0000 7f49ebc00640 1 --2- 172.21.15.102:0/3313163652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e40b5a30 0x7f49e40b5e90 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:00.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.636+0000 7f49ebc00640 1 -- 172.21.15.102:0/3313163652 >> 172.21.15.102:0/3313163652 conn(0x7f49e40cf350 msgr2=0x7f49e40cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:00.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.636+0000 7f49ebc00640 1 -- 172.21.15.102:0/3313163652 shutdown_connections 2024-03-23T03:50:00.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.636+0000 7f49ebc00640 1 -- 172.21.15.102:0/3313163652 wait complete. 2024-03-23T03:50:00.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.638+0000 7f49ebc00640 1 Processor -- start 2024-03-23T03:50:00.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.638+0000 7f49ebc00640 1 -- start start 2024-03-23T03:50:00.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.639+0000 7f49ebc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f49e40b4350 0x7f49e40c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:00.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.639+0000 7f49ebc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49e40b5a30 0x7f49e40c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:00.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.639+0000 7f49ebc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e415cda0 0x7f49e41f5820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:00.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.639+0000 7f49ebc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e415f610 con 0x7f49e415cda0 2024-03-23T03:50:00.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.639+0000 7f49ebc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e40c6b10 con 0x7f49e40b5a30 2024-03-23T03:50:00.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.639+0000 7f49ebc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e40c95f0 con 0x7f49e40b4350 2024-03-23T03:50:00.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.639+0000 7f49e9c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e415cda0 0x7f49e41f5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:00.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.639+0000 7f49e9c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e415cda0 0x7f49e41f5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40312/0 (socket says 172.21.15.102:40312) 2024-03-23T03:50:00.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.639+0000 7f49e9c00640 1 -- 172.21.15.102:0/3194418385 learned_addr learned my addr 172.21.15.102:0/3194418385 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:00.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.640+0000 7f49e9200640 1 --2- 172.21.15.102:0/3194418385 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f49e40b4350 0x7f49e40c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:00.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.640+0000 7f49e3e00640 1 --2- 172.21.15.102:0/3194418385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49e40b5a30 0x7f49e40c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:00.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.640+0000 7f49e9c00640 1 -- 172.21.15.102:0/3194418385 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f49e40b4350 msgr2=0x7f49e40c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:00.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.640+0000 7f49e9c00640 1 --2- 172.21.15.102:0/3194418385 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f49e40b4350 0x7f49e40c8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:00.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.640+0000 7f49e9c00640 1 -- 172.21.15.102:0/3194418385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49e40b5a30 msgr2=0x7f49e40c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:00.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.640+0000 7f49e9c00640 1 --2- 172.21.15.102:0/3194418385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49e40b5a30 0x7f49e40c65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:00.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.640+0000 7f49e9c00640 1 -- 172.21.15.102:0/3194418385 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d0009400 con 0x7f49e415cda0 2024-03-23T03:50:00.641 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.640+0000 7f49e9c00640 1 --2- 172.21.15.102:0/3194418385 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e415cda0 0x7f49e41f5820 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00c780 tx=0x7f49dc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:00.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.640+0000 7f49e0c00640 1 -- 172.21.15.102:0/3194418385 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49dc004b60 con 0x7f49e415cda0 2024-03-23T03:50:00.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.640+0000 7f49e0c00640 1 -- 172.21.15.102:0/3194418385 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49dc00ce90 con 0x7f49e415cda0 2024-03-23T03:50:00.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.640+0000 7f49e0c00640 1 -- 172.21.15.102:0/3194418385 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49dc00a850 con 0x7f49e415cda0 2024-03-23T03:50:00.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.640+0000 7f49ebc00640 1 -- 172.21.15.102:0/3194418385 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e415ef50 con 0x7f49e415cda0 2024-03-23T03:50:00.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.640+0000 7f49ebc00640 1 -- 172.21.15.102:0/3194418385 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e415ec40 con 0x7f49e415cda0 2024-03-23T03:50:00.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.642+0000 7f49ebc00640 1 -- 172.21.15.102:0/3194418385 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e415f750 con 0x7f49e415cda0 2024-03-23T03:50:00.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.642+0000 7f49e0c00640 1 -- 172.21.15.102:0/3194418385 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f49dc00a9b0 con 0x7f49e415cda0 2024-03-23T03:50:00.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.643+0000 7f49e0c00640 1 --2- 172.21.15.102:0/3194418385 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f49c406ac00 0x7f49c406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:00.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.643+0000 7f49e0c00640 1 -- 172.21.15.102:0/3194418385 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f49dc089350 con 0x7f49e415cda0 2024-03-23T03:50:00.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.643+0000 7f49e9200640 1 --2- 172.21.15.102:0/3194418385 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f49c406ac00 0x7f49c406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:00.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.644+0000 7f49e9200640 1 --2- 172.21.15.102:0/3194418385 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f49c406ac00 0x7f49c406d0c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f49d8000f60 tx=0x7f49d8007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:00.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.647+0000 7f49e0c00640 1 -- 172.21.15.102:0/3194418385 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f49e415f750 con 0x7f49e415cda0 2024-03-23T03:50:00.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:00.914+0000 7f49ebc00640 1 -- 172.21.15.102:0/3194418385 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_df"]} v 0) v1 -- 0x7f49e415f750 con 0x7f49e415cda0 2024-03-23T03:50:01.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.402+0000 7f49e0c00640 1 -- 172.21.15.102:0/3194418385 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_df"]}]=0 v190) v1 ==== 137+0+0 (secure 0 0 0) 0x7f49e415f750 con 0x7f49e415cda0 2024-03-23T03:50:01.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.406+0000 7f49ebc00640 1 -- 172.21.15.102:0/3194418385 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f49c406ac00 msgr2=0x7f49c406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:01.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.406+0000 7f49ebc00640 1 --2- 172.21.15.102:0/3194418385 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f49c406ac00 0x7f49c406d0c0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f49d8000f60 tx=0x7f49d8007c00 comp rx=0 tx=0).stop 2024-03-23T03:50:01.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.406+0000 7f49ebc00640 1 -- 172.21.15.102:0/3194418385 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e415cda0 msgr2=0x7f49e41f5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:01.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.406+0000 7f49ebc00640 1 --2- 172.21.15.102:0/3194418385 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e415cda0 0x7f49e41f5820 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f49dc00c780 tx=0x7f49dc00cc40 comp rx=0 tx=0).stop 2024-03-23T03:50:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.406+0000 7f49ebc00640 1 -- 172.21.15.102:0/3194418385 shutdown_connections 2024-03-23T03:50:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.406+0000 7f49ebc00640 1 --2- 172.21.15.102:0/3194418385 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f49c406ac00 0x7f49c406d0c0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.406+0000 7f49ebc00640 1 --2- 172.21.15.102:0/3194418385 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49e415cda0 0x7f49e41f5820 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.406+0000 7f49ebc00640 1 --2- 172.21.15.102:0/3194418385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49e40b5a30 0x7f49e40c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.406+0000 7f49ebc00640 1 --2- 172.21.15.102:0/3194418385 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f49e40b4350 0x7f49e40c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.406+0000 7f49ebc00640 1 -- 172.21.15.102:0/3194418385 >> 172.21.15.102:0/3194418385 conn(0x7f49e40cf350 msgr2=0x7f49e40b2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.406+0000 7f49ebc00640 1 -- 172.21.15.102:0/3194418385 shutdown_connections 2024-03-23T03:50:01.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.406+0000 7f49ebc00640 1 -- 172.21.15.102:0/3194418385 wait complete. 2024-03-23T03:50:01.428 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-23T03:50:01.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.627+0000 7f678b400640 1 Processor -- start 2024-03-23T03:50:01.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.627+0000 7f678b400640 1 -- start start 2024-03-23T03:50:01.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.629+0000 7f678b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6784159840 0x7f678406aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:01.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.629+0000 7f678b400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f67840cb690 0x7f678406b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:01.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.629+0000 7f678b400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6784159c20 con 0x7f6784065ab0 2024-03-23T03:50:01.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.629+0000 7f678b400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67840b3300 con 0x7f6784159840 2024-03-23T03:50:01.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.629+0000 7f678b400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67840b35e0 con 0x7f67840cb690 2024-03-23T03:50:01.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.629+0000 7f6783e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6784065ab0 0x7f67840abd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47538/0 (socket says 172.21.15.102:47538) 2024-03-23T03:50:01.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.629+0000 7f6783e00640 1 -- 172.21.15.102:0/779145299 learned_addr learned my addr 172.21.15.102:0/779145299 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:01.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.629+0000 7f6788a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6784159840 0x7f678406aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:01.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.630+0000 7f6788a00640 1 -- 172.21.15.102:0/779145299 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f67840cb690 msgr2=0x7f678406b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:01.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.630+0000 7f6788a00640 1 --2- 172.21.15.102:0/779145299 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f67840cb690 0x7f678406b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:01.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.630+0000 7f6788a00640 1 -- 172.21.15.102:0/779145299 >> v1:172.21.15.102:6789/0 conn(0x7f6784065ab0 legacy=0x7f67840abd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:01.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.630+0000 7f6788a00640 1 -- 172.21.15.102:0/779145299 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67840b38c0 con 0x7f6784159840 2024-03-23T03:50:01.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.631+0000 7f6788a00640 1 --2- 172.21.15.102:0/779145299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6784159840 0x7f678406aca0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f67740097a0 tx=0x7f677402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bfa7c4143fdcdc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:01.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.631+0000 7f6783400640 1 -- 172.21.15.102:0/779145299 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677402d040 con 0x7f6784159840 2024-03-23T03:50:01.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.631+0000 7f6783400640 1 -- 172.21.15.102:0/779145299 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6774031810 con 0x7f6784159840 2024-03-23T03:50:01.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.631+0000 7f6783400640 1 -- 172.21.15.102:0/779145299 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6774031ba0 con 0x7f6784159840 2024-03-23T03:50:01.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.632+0000 7f678b400640 1 -- 172.21.15.102:0/779145299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6784159840 msgr2=0x7f678406aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:01.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.632+0000 7f678b400640 1 --2- 172.21.15.102:0/779145299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6784159840 0x7f678406aca0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f67740097a0 tx=0x7f677402b8b0 comp rx=0 tx=0).stop 2024-03-23T03:50:01.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.632+0000 7f678b400640 1 -- 172.21.15.102:0/779145299 shutdown_connections 2024-03-23T03:50:01.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.632+0000 7f678b400640 1 --2- 172.21.15.102:0/779145299 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f67840cb690 0x7f678406b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:01.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.632+0000 7f678b400640 1 --2- 172.21.15.102:0/779145299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6784159840 0x7f678406aca0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:01.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.632+0000 7f678b400640 1 -- 172.21.15.102:0/779145299 >> 172.21.15.102:0/779145299 conn(0x7f6784070a20 msgr2=0x7f6784070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:01.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.632+0000 7f678b400640 1 -- 172.21.15.102:0/779145299 shutdown_connections 2024-03-23T03:50:01.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.632+0000 7f678b400640 1 -- 172.21.15.102:0/779145299 wait complete. 2024-03-23T03:50:01.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.633+0000 7f678b400640 1 Processor -- start 2024-03-23T03:50:01.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.634+0000 7f678b400640 1 -- start start 2024-03-23T03:50:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.634+0000 7f678b400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6784065ab0 0x7f678414fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.634+0000 7f678b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67840cb690 0x7f678413f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.634+0000 7f678b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6784159840 0x7f6784152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.634+0000 7f678b400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6784151e70 con 0x7f67840cb690 2024-03-23T03:50:01.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.635+0000 7f678b400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67841511f0 con 0x7f6784159840 2024-03-23T03:50:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.635+0000 7f678b400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67841514d0 con 0x7f6784065ab0 2024-03-23T03:50:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.635+0000 7f6783e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67840cb690 0x7f678413f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.635+0000 7f6783e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67840cb690 0x7f678413f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40334/0 (socket says 172.21.15.102:40334) 2024-03-23T03:50:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.635+0000 7f6783e00640 1 -- 172.21.15.102:0/2381915835 learned_addr learned my addr 172.21.15.102:0/2381915835 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.635+0000 7f6788a00640 1 --2- 172.21.15.102:0/2381915835 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6784065ab0 0x7f678414fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.635+0000 7f6783e00640 1 -- 172.21.15.102:0/2381915835 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6784065ab0 msgr2=0x7f678414fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.635+0000 7f6783e00640 1 --2- 172.21.15.102:0/2381915835 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6784065ab0 0x7f678414fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.635+0000 7f6783e00640 1 -- 172.21.15.102:0/2381915835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6784159840 msgr2=0x7f6784152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.635+0000 7f6789400640 1 --2- 172.21.15.102:0/2381915835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6784159840 0x7f6784152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.635+0000 7f6783e00640 1 --2- 172.21.15.102:0/2381915835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6784159840 0x7f6784152290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.635+0000 7f6783e00640 1 -- 172.21.15.102:0/2381915835 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6774009400 con 0x7f67840cb690 2024-03-23T03:50:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.635+0000 7f6783e00640 1 --2- 172.21.15.102:0/2381915835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67840cb690 0x7f678413f600 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f676c00cba0 tx=0x7f676c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:01.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.636+0000 7f6780c00640 1 -- 172.21.15.102:0/2381915835 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c00ad40 con 0x7f67840cb690 2024-03-23T03:50:01.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.636+0000 7f6780c00640 1 -- 172.21.15.102:0/2381915835 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676c012400 con 0x7f67840cb690 2024-03-23T03:50:01.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.636+0000 7f6780c00640 1 -- 172.21.15.102:0/2381915835 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676c00aea0 con 0x7f67840cb690 2024-03-23T03:50:01.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.636+0000 7f678b400640 1 -- 172.21.15.102:0/2381915835 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67841517b0 con 0x7f67840cb690 2024-03-23T03:50:01.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.636+0000 7f678b400640 1 -- 172.21.15.102:0/2381915835 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678413fb40 con 0x7f67840cb690 2024-03-23T03:50:01.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.637+0000 7f678b400640 1 -- 172.21.15.102:0/2381915835 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6784151fb0 con 0x7f67840cb690 2024-03-23T03:50:01.638 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.638+0000 7f6780c00640 1 -- 172.21.15.102:0/2381915835 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f676c007070 con 0x7f67840cb690 2024-03-23T03:50:01.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.638+0000 7f6780c00640 1 --2- 172.21.15.102:0/2381915835 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f676006ac00 0x7f676006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:01.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.639+0000 7f6780c00640 1 -- 172.21.15.102:0/2381915835 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f676c089180 con 0x7f67840cb690 2024-03-23T03:50:01.639 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.639+0000 7f6788a00640 1 --2- 172.21.15.102:0/2381915835 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f676006ac00 0x7f676006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:01.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.639+0000 7f6788a00640 1 --2- 172.21.15.102:0/2381915835 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f676006ac00 0x7f676006d0c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6774036000 tx=0x7f677403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:01.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.643+0000 7f6780c00640 1 -- 172.21.15.102:0/2381915835 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6784151fb0 con 0x7f67840cb690 2024-03-23T03:50:01.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.839+0000 7f678b400640 1 -- 172.21.15.102:0/2381915835 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f6784151fb0 con 0x7f67840cb690 2024-03-23T03:50:01.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.846+0000 7f6780c00640 1 -- 172.21.15.102:0/2381915835 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6784151fb0 con 0x7f67840cb690 2024-03-23T03:50:01.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.849+0000 7f678b400640 1 -- 172.21.15.102:0/2381915835 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f676006ac00 msgr2=0x7f676006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:01.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.849+0000 7f678b400640 1 --2- 172.21.15.102:0/2381915835 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f676006ac00 0x7f676006d0c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6774036000 tx=0x7f677403c040 comp rx=0 tx=0).stop 2024-03-23T03:50:01.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.849+0000 7f678b400640 1 -- 172.21.15.102:0/2381915835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67840cb690 msgr2=0x7f678413f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:01.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.849+0000 7f678b400640 1 --2- 172.21.15.102:0/2381915835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67840cb690 0x7f678413f600 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f676c00cba0 tx=0x7f676c00a430 comp rx=0 tx=0).stop 2024-03-23T03:50:01.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.849+0000 7f678b400640 1 -- 172.21.15.102:0/2381915835 shutdown_connections 2024-03-23T03:50:01.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.849+0000 7f678b400640 1 --2- 172.21.15.102:0/2381915835 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f676006ac00 0x7f676006d0c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:01.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.849+0000 7f678b400640 1 --2- 172.21.15.102:0/2381915835 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6784159840 0x7f6784152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:01.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.849+0000 7f678b400640 1 --2- 172.21.15.102:0/2381915835 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67840cb690 0x7f678413f600 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:01.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.849+0000 7f678b400640 1 --2- 172.21.15.102:0/2381915835 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6784065ab0 0x7f678414fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:01.850 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.849+0000 7f678b400640 1 -- 172.21.15.102:0/2381915835 >> 172.21.15.102:0/2381915835 conn(0x7f6784070a20 msgr2=0x7f67840af050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:01.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.850+0000 7f678b400640 1 -- 172.21.15.102:0/2381915835 shutdown_connections 2024-03-23T03:50:01.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:01.850+0000 7f678b400640 1 -- 172.21.15.102:0/2381915835 wait complete. 2024-03-23T03:50:01.867 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T03:50:01.868 DEBUG:teuthology.orchestra.run.smithi102:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:50:01.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:50:02.089 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T03:50:02.090 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:50:02.090 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:02.147 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:50:02.148 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:50:02.148 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:02.195 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:50:02.195 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:50:02.195 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:02.248 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:50:02.249 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:50:02.249 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:02.289 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:50:02.289 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-23T03:50:02.289 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:50:02.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.502+0000 7f2bb4400640 1 Processor -- start 2024-03-23T03:50:02.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.503+0000 7f2bb4400640 1 -- start start 2024-03-23T03:50:02.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.504+0000 7f2bb4400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f2bac13f200 0x7f2bac0ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:02.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.504+0000 7f2bb4400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bac0ce440 con 0x7f2bac15d350 2024-03-23T03:50:02.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.504+0000 7f2bb4400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bac0cdba0 con 0x7f2bac0cd520 2024-03-23T03:50:02.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.504+0000 7f2bb4400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bac0cd9b0 con 0x7f2bac13f200 2024-03-23T03:50:02.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.505+0000 7f2bb2400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2bac15d350 0x7f2bac06c8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:47540/0 (socket says 172.21.15.102:47540) 2024-03-23T03:50:02.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.505+0000 7f2bb2400640 1 -- 172.21.15.102:0/3437391700 learned_addr learned my addr 172.21.15.102:0/3437391700 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:02.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.505+0000 7f2bb1a00640 1 --2- 172.21.15.102:0/3437391700 >> v2:172.21.15.173:3300/0 conn(0x7f2bac13f200 0x7f2bac0ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:02.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.505+0000 7f2babe00640 1 -- 172.21.15.102:0/3437391700 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786751933 0 0) 0x7f2bac0ce440 con 0x7f2bac15d350 2024-03-23T03:50:02.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.505+0000 7f2babe00640 1 -- 172.21.15.102:0/3437391700 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b94003520 con 0x7f2bac15d350 2024-03-23T03:50:02.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.505+0000 7f2babe00640 1 -- 172.21.15.102:0/3437391700 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3226221262 0 0) 0x7f2b94003520 con 0x7f2bac15d350 2024-03-23T03:50:02.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.505+0000 7f2babe00640 1 -- 172.21.15.102:0/3437391700 >> v2:172.21.15.173:3300/0 conn(0x7f2bac13f200 msgr2=0x7f2bac0ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:02.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.505+0000 7f2babe00640 1 --2- 172.21.15.102:0/3437391700 >> v2:172.21.15.173:3300/0 conn(0x7f2bac13f200 0x7f2bac0ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:02.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.505+0000 7f2babe00640 1 -- 172.21.15.102:0/3437391700 >> v1:172.21.15.173:6789/0 conn(0x7f2bac0cd520 legacy=0x7f2bac15cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:02.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.505+0000 7f2babe00640 1 -- 172.21.15.102:0/3437391700 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bac0cde80 con 0x7f2bac15d350 2024-03-23T03:50:02.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.505+0000 7f2babe00640 1 -- 172.21.15.102:0/3437391700 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f2ba40019f0 con 0x7f2bac15d350 2024-03-23T03:50:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.505+0000 7f2babe00640 1 -- 172.21.15.102:0/3437391700 >> v1:172.21.15.102:6789/0 conn(0x7f2bac15d350 legacy=0x7f2bac06c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.505+0000 7f2babe00640 1 --2- 172.21.15.102:0/3437391700 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2b94003970 0x7f2b94003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.506+0000 7f2babe00640 1 --2- 172.21.15.102:0/3437391700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b940047c0 0x7f2b94004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.506+0000 7f2babe00640 1 --2- 172.21.15.102:0/3437391700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b94005180 0x7f2b9400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.506+0000 7f2babe00640 1 -- 172.21.15.102:0/3437391700 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9400e300 con 0x7f2b940047c0 2024-03-23T03:50:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.506+0000 7f2babe00640 1 -- 172.21.15.102:0/3437391700 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9400e470 con 0x7f2b94005180 2024-03-23T03:50:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.506+0000 7f2babe00640 1 -- 172.21.15.102:0/3437391700 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9400e630 con 0x7f2b94003970 2024-03-23T03:50:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.506+0000 7f2bb1000640 1 --2- 172.21.15.102:0/3437391700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b940047c0 0x7f2b94004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.506+0000 7f2bb1a00640 1 --2- 172.21.15.102:0/3437391700 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2b94003970 0x7f2b94003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.506+0000 7f2bb1000640 1 -- 172.21.15.102:0/3437391700 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2b94003970 msgr2=0x7f2b94003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.506+0000 7f2bb1000640 1 --2- 172.21.15.102:0/3437391700 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2b94003970 0x7f2b94003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.506+0000 7f2bb1000640 1 -- 172.21.15.102:0/3437391700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b94005180 msgr2=0x7f2b9400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.506+0000 7f2bb1000640 1 --2- 172.21.15.102:0/3437391700 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b94005180 0x7f2b9400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:02.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.506+0000 7f2bb1000640 1 -- 172.21.15.102:0/3437391700 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b94003520 con 0x7f2b940047c0 2024-03-23T03:50:02.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.506+0000 7f2bb2400640 1 -- 172.21.15.102:0/3437391700 reap_dead start 2024-03-23T03:50:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.507+0000 7f2bb1000640 1 --2- 172.21.15.102:0/3437391700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b940047c0 0x7f2b94004c40 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c001750 tx=0x7f2b940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df133ff7d54fdaff server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.507+0000 7f2babe00640 1 -- 172.21.15.102:0/3437391700 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c0052d0 con 0x7f2b940047c0 2024-03-23T03:50:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.507+0000 7f2babe00640 1 -- 172.21.15.102:0/3437391700 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9c02e050 con 0x7f2b940047c0 2024-03-23T03:50:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.507+0000 7f2babe00640 1 -- 172.21.15.102:0/3437391700 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c003830 con 0x7f2b940047c0 2024-03-23T03:50:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.508+0000 7f2bb4400640 1 -- 172.21.15.102:0/3437391700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b940047c0 msgr2=0x7f2b94004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.508+0000 7f2bb4400640 1 --2- 172.21.15.102:0/3437391700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b940047c0 0x7f2b94004c40 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c001750 tx=0x7f2b940057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.508+0000 7f2bb4400640 1 -- 172.21.15.102:0/3437391700 shutdown_connections 2024-03-23T03:50:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.508+0000 7f2bb4400640 1 --2- 172.21.15.102:0/3437391700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2b940047c0 0x7f2b94004c40 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.508+0000 7f2bb4400640 1 -- 172.21.15.102:0/3437391700 >> 172.21.15.102:0/3437391700 conn(0x7f2bac06a3d0 msgr2=0x7f2bac0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:02.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.508+0000 7f2bb4400640 1 -- 172.21.15.102:0/3437391700 shutdown_connections 2024-03-23T03:50:02.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.508+0000 7f2bb4400640 1 -- 172.21.15.102:0/3437391700 wait complete. 2024-03-23T03:50:02.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.510+0000 7f2bb4400640 1 Processor -- start 2024-03-23T03:50:02.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.510+0000 7f2bb4400640 1 -- start start 2024-03-23T03:50:02.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.511+0000 7f2bb4400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b940047c0 0x7f2bac14f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:02.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.511+0000 7f2bb4400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2bac1526d0 0x7f2bac15b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:02.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.511+0000 7f2bb4400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bac15b840 0x7f2bac1508d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:02.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.511+0000 7f2bb4400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac1522b0 con 0x7f2bac15b840 2024-03-23T03:50:02.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.511+0000 7f2bb4400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac15bcf0 con 0x7f2b940047c0 2024-03-23T03:50:02.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.511+0000 7f2bb4400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac14f810 con 0x7f2bac1526d0 2024-03-23T03:50:02.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.511+0000 7f2bb2400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bac15b840 0x7f2bac1508d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:02.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.511+0000 7f2bb2400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bac15b840 0x7f2bac1508d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:40356/0 (socket says 172.21.15.102:40356) 2024-03-23T03:50:02.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.511+0000 7f2bb2400640 1 -- 172.21.15.102:0/977966338 learned_addr learned my addr 172.21.15.102:0/977966338 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:02.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.512+0000 7f2bb1a00640 1 --2- 172.21.15.102:0/977966338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b940047c0 0x7f2bac14f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.512+0000 7f2bb2400640 1 -- 172.21.15.102:0/977966338 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2bac1526d0 msgr2=0x7f2bac15b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.512+0000 7f2bb1000640 1 --2- 172.21.15.102:0/977966338 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2bac1526d0 0x7f2bac15b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.512+0000 7f2bb2400640 1 --2- 172.21.15.102:0/977966338 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2bac1526d0 0x7f2bac15b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.512+0000 7f2bb2400640 1 -- 172.21.15.102:0/977966338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b940047c0 msgr2=0x7f2bac14f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.512+0000 7f2bb2400640 1 --2- 172.21.15.102:0/977966338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b940047c0 0x7f2bac14f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.512+0000 7f2bb2400640 1 -- 172.21.15.102:0/977966338 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bac151630 con 0x7f2bac15b840 2024-03-23T03:50:02.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.512+0000 7f2bb1a00640 1 --2- 172.21.15.102:0/977966338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b940047c0 0x7f2bac14f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:02.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.512+0000 7f2bb1000640 1 --2- 172.21.15.102:0/977966338 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2bac1526d0 0x7f2bac15b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:02.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.512+0000 7f2bb2400640 1 --2- 172.21.15.102:0/977966338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bac15b840 0x7f2bac1508d0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4001560 tx=0x7f2ba400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:02.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.512+0000 7f2ba9600640 1 -- 172.21.15.102:0/977966338 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba400cee0 con 0x7f2bac15b840 2024-03-23T03:50:02.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.512+0000 7f2ba9600640 1 -- 172.21.15.102:0/977966338 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba400a5b0 con 0x7f2bac15b840 2024-03-23T03:50:02.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.512+0000 7f2ba9600640 1 -- 172.21.15.102:0/977966338 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba400a960 con 0x7f2bac15b840 2024-03-23T03:50:02.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.512+0000 7f2bb4400640 1 -- 172.21.15.102:0/977966338 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bac151910 con 0x7f2bac15b840 2024-03-23T03:50:02.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.512+0000 7f2bb4400640 1 -- 172.21.15.102:0/977966338 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bac151ed0 con 0x7f2bac15b840 2024-03-23T03:50:02.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.514+0000 7f2bb4400640 1 -- 172.21.15.102:0/977966338 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bac1523f0 con 0x7f2bac15b840 2024-03-23T03:50:02.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.514+0000 7f2ba9600640 1 -- 172.21.15.102:0/977966338 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2ba400aac0 con 0x7f2bac15b840 2024-03-23T03:50:02.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.515+0000 7f2ba9600640 1 --2- 172.21.15.102:0/977966338 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2b9006ac00 0x7f2b9006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:02.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.515+0000 7f2ba9600640 1 -- 172.21.15.102:0/977966338 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f2ba4088910 con 0x7f2bac15b840 2024-03-23T03:50:02.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.516+0000 7f2bb1a00640 1 --2- 172.21.15.102:0/977966338 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2b9006ac00 0x7f2b9006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:02.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.516+0000 7f2bb1a00640 1 --2- 172.21.15.102:0/977966338 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2b9006ac00 0x7f2b9006d0c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2ba000aae0 tx=0x7f2ba0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:02.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.520+0000 7f2ba9600640 1 -- 172.21.15.102:0/977966338 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2ba4053ca0 con 0x7f2bac15b840 2024-03-23T03:50:02.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.751+0000 7f2bb4400640 1 -- 172.21.15.102:0/977966338 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bac06abf0 con 0x7f2bac15b840 2024-03-23T03:50:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.752+0000 7f2ba9600640 1 -- 172.21.15.102:0/977966338 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+3538 (secure 0 0 0) 0x7f2ba401a070 con 0x7f2bac15b840 2024-03-23T03:50:02.753 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:02.753 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":35,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5230,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3155961019","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3155961019},{"type":"v1","addr":"172.21.15.173:6835","nonce":3155961019}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":28},{"gid":5356,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/3704444435","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":3704444435},{"type":"v1","addr":"172.21.15.173:6833","nonce":3704444435}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":35},{"gid":6021,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/251450026","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":251450026},{"type":"v1","addr":"172.21.15.102:6836","nonce":251450026}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6024,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3299889122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":3299889122},{"type":"v1","addr":"172.21.15.102:6837","nonce":3299889122}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[]} 2024-03-23T03:50:02.753 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 35 2024-03-23T03:50:02.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.755+0000 7f2bb4400640 1 -- 172.21.15.102:0/977966338 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2b9006ac00 msgr2=0x7f2b9006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:02.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.755+0000 7f2bb4400640 1 --2- 172.21.15.102:0/977966338 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2b9006ac00 0x7f2b9006d0c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f2ba000aae0 tx=0x7f2ba0008040 comp rx=0 tx=0).stop 2024-03-23T03:50:02.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.755+0000 7f2bb4400640 1 -- 172.21.15.102:0/977966338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bac15b840 msgr2=0x7f2bac1508d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:02.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.755+0000 7f2bb4400640 1 --2- 172.21.15.102:0/977966338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bac15b840 0x7f2bac1508d0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4001560 tx=0x7f2ba400cc30 comp rx=0 tx=0).stop 2024-03-23T03:50:02.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.755+0000 7f2bb4400640 1 -- 172.21.15.102:0/977966338 shutdown_connections 2024-03-23T03:50:02.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.755+0000 7f2bb4400640 1 --2- 172.21.15.102:0/977966338 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2b9006ac00 0x7f2b9006d0c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:02.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.755+0000 7f2bb4400640 1 --2- 172.21.15.102:0/977966338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bac15b840 0x7f2bac1508d0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:02.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.755+0000 7f2bb4400640 1 --2- 172.21.15.102:0/977966338 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2bac1526d0 0x7f2bac15b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:02.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.755+0000 7f2bb4400640 1 --2- 172.21.15.102:0/977966338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2b940047c0 0x7f2bac14f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:02.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.755+0000 7f2bb4400640 1 -- 172.21.15.102:0/977966338 >> 172.21.15.102:0/977966338 conn(0x7f2bac06a3d0 msgr2=0x7f2bac15c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:02.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.755+0000 7f2bb4400640 1 -- 172.21.15.102:0/977966338 shutdown_connections 2024-03-23T03:50:02.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:02.755+0000 7f2bb4400640 1 -- 172.21.15.102:0/977966338 wait complete. 2024-03-23T03:50:02.775 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T03:50:02.775 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-23T03:50:02.775 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T03:50:02.775 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:50:02.776 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T03:50:02.776 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-23T03:50:02.776 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T03:50:02.776 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:50:02.776 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T03:50:02.776 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-23T03:50:02.776 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T03:50:02.776 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:50:02.777 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T03:50:02.777 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-23T03:50:02.777 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T03:50:02.777 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:50:02.794 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T03:50:02.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T03:50:02.798 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T03:50:02.798 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T03:50:02.802 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T03:50:02.802 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T03:50:02.803 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T03:50:02.804 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T03:50:02.845 INFO:tasks.ceph.mds.c:Started 2024-03-23T03:50:02.845 DEBUG:teuthology.parallel:result is None 2024-03-23T03:50:02.851 INFO:tasks.ceph.mds.b:Started 2024-03-23T03:50:02.851 DEBUG:teuthology.parallel:result is None 2024-03-23T03:50:02.852 INFO:tasks.ceph.mds.a:Started 2024-03-23T03:50:02.852 DEBUG:teuthology.parallel:result is None 2024-03-23T03:50:02.857 INFO:tasks.ceph.mds.d:Started 2024-03-23T03:50:02.857 DEBUG:teuthology.parallel:result is None 2024-03-23T03:50:02.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-23T03:50:02.994 INFO:tasks.ceph.mds.c.smithi102.stdout:starting mds.c at 2024-03-23T03:50:03.000 INFO:tasks.ceph.mds.a.smithi102.stdout:starting mds.a at 2024-03-23T03:50:03.011 INFO:tasks.ceph.mds.b.smithi173.stdout:starting mds.b at 2024-03-23T03:50:03.013 INFO:tasks.ceph.mds.d.smithi173.stdout:starting mds.d at 2024-03-23T03:50:03.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.045+0000 7fa98b600640 1 Processor -- start 2024-03-23T03:50:03.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.045+0000 7fa98b600640 1 -- start start 2024-03-23T03:50:03.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.046+0000 7fa98b600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa9840cb660 0x7fa9840cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:03.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.046+0000 7fa98b600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa98415ae60 con 0x7fa98406c2b0 2024-03-23T03:50:03.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.046+0000 7fa98b600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa98415b140 con 0x7fa9840ce930 2024-03-23T03:50:03.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.046+0000 7fa98b600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98415bae0 con 0x7fa9840cb660 2024-03-23T03:50:03.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.047+0000 7fa989600640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa98406c2b0 0x7fa98406c720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38616/0 (socket says 172.21.15.102:38616) 2024-03-23T03:50:03.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.047+0000 7fa989600640 1 -- 172.21.15.102:0/2201293934 learned_addr learned my addr 172.21.15.102:0/2201293934 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:03.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.047+0000 7fa983e00640 1 --2- 172.21.15.102:0/2201293934 >> v2:172.21.15.173:3300/0 conn(0x7fa9840cb660 0x7fa9840cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:03.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.047+0000 7fa983400640 1 -- 172.21.15.102:0/2201293934 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404105839 0 0) 0x7fa98415ae60 con 0x7fa98406c2b0 2024-03-23T03:50:03.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.047+0000 7fa983400640 1 -- 172.21.15.102:0/2201293934 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa96c003520 con 0x7fa98406c2b0 2024-03-23T03:50:03.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.047+0000 7fa983400640 1 -- 172.21.15.102:0/2201293934 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2517306555 0 0) 0x7fa98415b140 con 0x7fa9840ce930 2024-03-23T03:50:03.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.047+0000 7fa983400640 1 -- 172.21.15.102:0/2201293934 --> v1:172.21.15.173:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa98415ae60 con 0x7fa9840ce930 2024-03-23T03:50:03.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.047+0000 7fa983e00640 1 -- 172.21.15.102:0/2201293934 >> v1:172.21.15.173:6789/0 conn(0x7fa9840ce930 legacy=0x7fa9840ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:03.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.047+0000 7fa983e00640 1 -- 172.21.15.102:0/2201293934 >> v1:172.21.15.102:6789/0 conn(0x7fa98406c2b0 legacy=0x7fa98406c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:03.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.047+0000 7fa983e00640 1 -- 172.21.15.102:0/2201293934 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa98415b420 con 0x7fa9840cb660 2024-03-23T03:50:03.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.048+0000 7fa983e00640 1 --2- 172.21.15.102:0/2201293934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9840cb660 0x7fa9840cbae0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa9700097a0 tx=0x7fa97002b740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cba84785fb5ac1b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:03.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.049+0000 7fa983400640 1 -- 172.21.15.102:0/2201293934 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97c0019f0 con 0x7fa9840cb660 2024-03-23T03:50:03.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.049+0000 7fa983400640 1 -- 172.21.15.102:0/2201293934 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97002e050 con 0x7fa9840cb660 2024-03-23T03:50:03.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.049+0000 7fa983400640 1 -- 172.21.15.102:0/2201293934 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa970030710 con 0x7fa9840cb660 2024-03-23T03:50:03.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.049+0000 7fa98b600640 1 -- 172.21.15.102:0/2201293934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9840cb660 msgr2=0x7fa9840cbae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:03.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.049+0000 7fa98b600640 1 --2- 172.21.15.102:0/2201293934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9840cb660 0x7fa9840cbae0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa9700097a0 tx=0x7fa97002b740 comp rx=0 tx=0).stop 2024-03-23T03:50:03.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.049+0000 7fa98b600640 1 -- 172.21.15.102:0/2201293934 shutdown_connections 2024-03-23T03:50:03.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.049+0000 7fa98b600640 1 --2- 172.21.15.102:0/2201293934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9840cb660 0x7fa9840cbae0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:03.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.049+0000 7fa98b600640 1 -- 172.21.15.102:0/2201293934 >> 172.21.15.102:0/2201293934 conn(0x7fa98406a0b0 msgr2=0x7fa98406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:03.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.049+0000 7fa98b600640 1 -- 172.21.15.102:0/2201293934 shutdown_connections 2024-03-23T03:50:03.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.050+0000 7fa98b600640 1 -- 172.21.15.102:0/2201293934 wait complete. 2024-03-23T03:50:03.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.051+0000 7fa98b600640 1 Processor -- start 2024-03-23T03:50:03.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.051+0000 7fa98b600640 1 -- start start 2024-03-23T03:50:03.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.052+0000 7fa98b600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa98406c2b0 0x7fa984149380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:03.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.052+0000 7fa98b600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9840cb660 0x7fa9841487d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:03.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.052+0000 7fa98b600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9840ce930 0x7fa9841595f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:03.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.052+0000 7fa98b600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9840aca30 con 0x7fa98406c2b0 2024-03-23T03:50:03.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.052+0000 7fa98b600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa984148d10 con 0x7fa9840cb660 2024-03-23T03:50:03.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.052+0000 7fa98b600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9841f1d60 con 0x7fa9840ce930 2024-03-23T03:50:03.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.052+0000 7fa988c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa98406c2b0 0x7fa984149380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:03.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.052+0000 7fa988c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa98406c2b0 0x7fa984149380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59720/0 (socket says 172.21.15.102:59720) 2024-03-23T03:50:03.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.052+0000 7fa988c00640 1 -- 172.21.15.102:0/2054722999 learned_addr learned my addr 172.21.15.102:0/2054722999 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:03.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.052+0000 7fa983e00640 1 --2- 172.21.15.102:0/2054722999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9840cb660 0x7fa9841487d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:03.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.052+0000 7fa989600640 1 --2- 172.21.15.102:0/2054722999 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9840ce930 0x7fa9841595f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:03.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.053+0000 7fa988c00640 1 -- 172.21.15.102:0/2054722999 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9840ce930 msgr2=0x7fa9841595f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:03.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.053+0000 7fa988c00640 1 --2- 172.21.15.102:0/2054722999 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9840ce930 0x7fa9841595f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:03.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.053+0000 7fa988c00640 1 -- 172.21.15.102:0/2054722999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9840cb660 msgr2=0x7fa9841487d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:03.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.053+0000 7fa988c00640 1 --2- 172.21.15.102:0/2054722999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9840cb660 0x7fa9841487d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:03.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.053+0000 7fa988c00640 1 -- 172.21.15.102:0/2054722999 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa970009400 con 0x7fa98406c2b0 2024-03-23T03:50:03.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.053+0000 7fa989600640 1 --2- 172.21.15.102:0/2054722999 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9840ce930 0x7fa9841595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:50:03.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.053+0000 7fa988c00640 1 --2- 172.21.15.102:0/2054722999 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa98406c2b0 0x7fa984149380 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa97800c430 tx=0x7fa97800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:03.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.053+0000 7fa980c00640 1 -- 172.21.15.102:0/2054722999 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9780052d0 con 0x7fa98406c2b0 2024-03-23T03:50:03.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.053+0000 7fa980c00640 1 -- 172.21.15.102:0/2054722999 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa978005450 con 0x7fa98406c2b0 2024-03-23T03:50:03.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.053+0000 7fa980c00640 1 -- 172.21.15.102:0/2054722999 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa978003830 con 0x7fa98406c2b0 2024-03-23T03:50:03.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.053+0000 7fa98b600640 1 -- 172.21.15.102:0/2054722999 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa984155280 con 0x7fa98406c2b0 2024-03-23T03:50:03.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.053+0000 7fa98b600640 1 -- 172.21.15.102:0/2054722999 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9840ad280 con 0x7fa98406c2b0 2024-03-23T03:50:03.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.054+0000 7fa98b600640 1 -- 172.21.15.102:0/2054722999 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa984073390 con 0x7fa98406c2b0 2024-03-23T03:50:03.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.055+0000 7fa980c00640 1 -- 172.21.15.102:0/2054722999 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa978003990 con 0x7fa98406c2b0 2024-03-23T03:50:03.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.057+0000 7fa980c00640 1 --2- 172.21.15.102:0/2054722999 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa96406ac00 0x7fa96406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:03.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.057+0000 7fa980c00640 1 -- 172.21.15.102:0/2054722999 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fa978088750 con 0x7fa98406c2b0 2024-03-23T03:50:03.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.057+0000 7fa983e00640 1 --2- 172.21.15.102:0/2054722999 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa96406ac00 0x7fa96406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:03.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.058+0000 7fa983e00640 1 --2- 172.21.15.102:0/2054722999 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa96406ac00 0x7fa96406d0c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa970034000 tx=0x7fa970008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:03.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.061+0000 7fa980c00640 1 -- 172.21.15.102:0/2054722999 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa978053b00 con 0x7fa98406c2b0 2024-03-23T03:50:03.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:03.267+0000 7fa98b600640 1 -- 172.21.15.102:0/2054722999 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa9840cc430 con 0x7fa98406c2b0 2024-03-23T03:50:04.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.052+0000 7fa980c00640 1 -- 172.21.15.102:0/2054722999 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v77) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa978057120 con 0x7fa98406c2b0 2024-03-23T03:50:04.053 INFO:teuthology.orchestra.run.smithi102.stderr: removed all blocklist entries 2024-03-23T03:50:04.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.057+0000 7fa98b600640 1 -- 172.21.15.102:0/2054722999 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa96406ac00 msgr2=0x7fa96406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:04.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.057+0000 7fa98b600640 1 --2- 172.21.15.102:0/2054722999 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa96406ac00 0x7fa96406d0c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa970034000 tx=0x7fa970008030 comp rx=0 tx=0).stop 2024-03-23T03:50:04.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.057+0000 7fa98b600640 1 -- 172.21.15.102:0/2054722999 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa98406c2b0 msgr2=0x7fa984149380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:04.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.057+0000 7fa98b600640 1 --2- 172.21.15.102:0/2054722999 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa98406c2b0 0x7fa984149380 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fa97800c430 tx=0x7fa97800c8f0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.057+0000 7fa98b600640 1 -- 172.21.15.102:0/2054722999 shutdown_connections 2024-03-23T03:50:04.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.057+0000 7fa98b600640 1 --2- 172.21.15.102:0/2054722999 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa96406ac00 0x7fa96406d0c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.057+0000 7fa98b600640 1 --2- 172.21.15.102:0/2054722999 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9840ce930 0x7fa9841595f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.057+0000 7fa98b600640 1 --2- 172.21.15.102:0/2054722999 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9840cb660 0x7fa9841487d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.057+0000 7fa98b600640 1 --2- 172.21.15.102:0/2054722999 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa98406c2b0 0x7fa984149380 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.057+0000 7fa98b600640 1 -- 172.21.15.102:0/2054722999 >> 172.21.15.102:0/2054722999 conn(0x7fa98406a0b0 msgr2=0x7fa984158a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:04.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.057+0000 7fa98b600640 1 -- 172.21.15.102:0/2054722999 shutdown_connections 2024-03-23T03:50:04.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.057+0000 7fa98b600640 1 -- 172.21.15.102:0/2054722999 wait complete. 2024-03-23T03:50:04.076 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-23T03:50:04.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.281+0000 7fac8d400640 1 Processor -- start 2024-03-23T03:50:04.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.281+0000 7fac8d400640 1 -- start start 2024-03-23T03:50:04.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.282+0000 7fac8d400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fac880b3e30 0x7fac8815cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:04.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.282+0000 7fac8d400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac8806c6f0 con 0x7fac880b39b0 2024-03-23T03:50:04.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.282+0000 7fac8d400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac8806c9d0 con 0x7fac8815b560 2024-03-23T03:50:04.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.282+0000 7fac8d400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac8806ccb0 con 0x7fac880b3e30 2024-03-23T03:50:04.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.283+0000 7fac85e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fac880b39b0 0x7fac8815c400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38626/0 (socket says 172.21.15.102:38626) 2024-03-23T03:50:04.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.283+0000 7fac85e00640 1 -- 172.21.15.102:0/1863716556 learned_addr learned my addr 172.21.15.102:0/1863716556 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:04.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.283+0000 7fac85400640 1 -- 172.21.15.102:0/1863716556 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2113918499 0 0) 0x7fac8806c6f0 con 0x7fac880b39b0 2024-03-23T03:50:04.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.283+0000 7fac85400640 1 -- 172.21.15.102:0/1863716556 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac6c003520 con 0x7fac880b39b0 2024-03-23T03:50:04.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.283+0000 7fac87200640 1 --2- 172.21.15.102:0/1863716556 >> v2:172.21.15.173:3300/0 conn(0x7fac880b3e30 0x7fac8815cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:04.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.283+0000 7fac85400640 1 -- 172.21.15.102:0/1863716556 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1457110185 0 0) 0x7fac6c003520 con 0x7fac880b39b0 2024-03-23T03:50:04.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.284+0000 7fac85400640 1 -- 172.21.15.102:0/1863716556 >> v2:172.21.15.173:3300/0 conn(0x7fac880b3e30 msgr2=0x7fac8815cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:04.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.284+0000 7fac85400640 1 --2- 172.21.15.102:0/1863716556 >> v2:172.21.15.173:3300/0 conn(0x7fac880b3e30 0x7fac8815cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.284+0000 7fac85400640 1 -- 172.21.15.102:0/1863716556 >> v1:172.21.15.173:6789/0 conn(0x7fac8815b560 legacy=0x7fac8815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:04.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.284+0000 7fac85400640 1 -- 172.21.15.102:0/1863716556 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac8806cf90 con 0x7fac880b39b0 2024-03-23T03:50:04.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.284+0000 7fac85400640 1 -- 172.21.15.102:0/1863716556 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fac74001670 con 0x7fac880b39b0 2024-03-23T03:50:04.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.284+0000 7fac85400640 1 -- 172.21.15.102:0/1863716556 >> v1:172.21.15.102:6789/0 conn(0x7fac880b39b0 legacy=0x7fac8815c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:04.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.284+0000 7fac85400640 1 --2- 172.21.15.102:0/1863716556 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fac6c003970 0x7fac6c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:04.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.284+0000 7fac85400640 1 --2- 172.21.15.102:0/1863716556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac6c0047c0 0x7fac6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:04.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.284+0000 7fac85400640 1 --2- 172.21.15.102:0/1863716556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6c005180 0x7fac6c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:04.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.284+0000 7fac85400640 1 -- 172.21.15.102:0/1863716556 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8806ccb0 con 0x7fac6c005180 2024-03-23T03:50:04.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.284+0000 7fac85400640 1 -- 172.21.15.102:0/1863716556 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c00c0d0 con 0x7fac6c0047c0 2024-03-23T03:50:04.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.284+0000 7fac85400640 1 -- 172.21.15.102:0/1863716556 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c00c290 con 0x7fac6c003970 2024-03-23T03:50:04.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.284+0000 7fac87200640 1 --2- 172.21.15.102:0/1863716556 >> v2:172.21.15.173:3300/0 conn(0x7fac880b3e30 0x7fac8815cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:50:04.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.284+0000 7fac87200640 1 --2- 172.21.15.102:0/1863716556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6c005180 0x7fac6c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:04.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.284+0000 7fac85e00640 1 --2- 172.21.15.102:0/1863716556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac6c0047c0 0x7fac6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:04.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.284+0000 7fac86800640 1 --2- 172.21.15.102:0/1863716556 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fac6c003970 0x7fac6c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:04.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.285+0000 7fac87200640 1 -- 172.21.15.102:0/1863716556 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fac6c003970 msgr2=0x7fac6c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:04.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.285+0000 7fac87200640 1 --2- 172.21.15.102:0/1863716556 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fac6c003970 0x7fac6c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.285+0000 7fac87200640 1 -- 172.21.15.102:0/1863716556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac6c0047c0 msgr2=0x7fac6c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:04.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.285+0000 7fac87200640 1 --2- 172.21.15.102:0/1863716556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac6c0047c0 0x7fac6c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.285+0000 7fac87200640 1 -- 172.21.15.102:0/1863716556 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c003520 con 0x7fac6c005180 2024-03-23T03:50:04.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.285+0000 7fac87200640 1 -- 172.21.15.102:0/1863716556 reap_dead start 2024-03-23T03:50:04.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.286+0000 7fac87200640 1 --2- 172.21.15.102:0/1863716556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6c005180 0x7fac6c00b980 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fac800073e0 tx=0x7fac6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b82d185dd083ef48 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:04.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.286+0000 7fac85400640 1 -- 172.21.15.102:0/1863716556 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac8002cdc0 con 0x7fac6c005180 2024-03-23T03:50:04.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.286+0000 7fac85400640 1 -- 172.21.15.102:0/1863716556 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac80034440 con 0x7fac6c005180 2024-03-23T03:50:04.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.286+0000 7fac85400640 1 -- 172.21.15.102:0/1863716556 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac800347f0 con 0x7fac6c005180 2024-03-23T03:50:04.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.286+0000 7fac8d400640 1 -- 172.21.15.102:0/1863716556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6c005180 msgr2=0x7fac6c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:04.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.286+0000 7fac8d400640 1 --2- 172.21.15.102:0/1863716556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6c005180 0x7fac6c00b980 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fac800073e0 tx=0x7fac6c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.287+0000 7fac8d400640 1 -- 172.21.15.102:0/1863716556 shutdown_connections 2024-03-23T03:50:04.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.287+0000 7fac8d400640 1 --2- 172.21.15.102:0/1863716556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6c005180 0x7fac6c00b980 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.287+0000 7fac8d400640 1 -- 172.21.15.102:0/1863716556 >> 172.21.15.102:0/1863716556 conn(0x7fac8806a3d0 msgr2=0x7fac880b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:04.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.287+0000 7fac8d400640 1 -- 172.21.15.102:0/1863716556 shutdown_connections 2024-03-23T03:50:04.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.287+0000 7fac8d400640 1 -- 172.21.15.102:0/1863716556 wait complete. 2024-03-23T03:50:04.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.288+0000 7fac8d400640 1 Processor -- start 2024-03-23T03:50:04.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.288+0000 7fac8d400640 1 -- start start 2024-03-23T03:50:04.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.289+0000 7fac8d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6c005180 0x7fac88156840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:04.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.289+0000 7fac8d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac88154590 0x7fac881488c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.289+0000 7fac8d400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fac88154a10 0x7fac88148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.289+0000 7fac8d400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac881f2270 con 0x7fac6c005180 2024-03-23T03:50:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.289+0000 7fac8d400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac88153650 con 0x7fac88154590 2024-03-23T03:50:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.289+0000 7fac8d400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac880ae310 con 0x7fac88154a10 2024-03-23T03:50:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.289+0000 7fac86800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6c005180 0x7fac88156840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.290+0000 7fac86800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6c005180 0x7fac88156840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59736/0 (socket says 172.21.15.102:59736) 2024-03-23T03:50:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.290+0000 7fac86800640 1 -- 172.21.15.102:0/521764984 learned_addr learned my addr 172.21.15.102:0/521764984 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.290+0000 7fac85e00640 1 --2- 172.21.15.102:0/521764984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac88154590 0x7fac881488c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.290+0000 7fac86800640 1 -- 172.21.15.102:0/521764984 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fac88154a10 msgr2=0x7fac88148e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.290+0000 7fac86800640 1 --2- 172.21.15.102:0/521764984 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fac88154a10 0x7fac88148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.290+0000 7fac86800640 1 -- 172.21.15.102:0/521764984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac88154590 msgr2=0x7fac881488c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.290+0000 7fac86800640 1 --2- 172.21.15.102:0/521764984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac88154590 0x7fac881488c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.290+0000 7fac86800640 1 -- 172.21.15.102:0/521764984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac880ae5f0 con 0x7fac6c005180 2024-03-23T03:50:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.290+0000 7fac86800640 1 --2- 172.21.15.102:0/521764984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6c005180 0x7fac88156840 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fac7800e4d0 tx=0x7fac7800e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.290+0000 7fac7ea00640 1 -- 172.21.15.102:0/521764984 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac780052e0 con 0x7fac6c005180 2024-03-23T03:50:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.290+0000 7fac7ea00640 1 -- 172.21.15.102:0/521764984 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac78005440 con 0x7fac6c005180 2024-03-23T03:50:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.290+0000 7fac7ea00640 1 -- 172.21.15.102:0/521764984 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac78003810 con 0x7fac6c005180 2024-03-23T03:50:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.290+0000 7fac8d400640 1 -- 172.21.15.102:0/521764984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac880ae8d0 con 0x7fac6c005180 2024-03-23T03:50:04.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.291+0000 7fac8d400640 1 -- 172.21.15.102:0/521764984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac88156d80 con 0x7fac6c005180 2024-03-23T03:50:04.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.292+0000 7fac8d400640 1 -- 172.21.15.102:0/521764984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac8806ac20 con 0x7fac6c005180 2024-03-23T03:50:04.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.293+0000 7fac7ea00640 1 -- 172.21.15.102:0/521764984 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fac78003970 con 0x7fac6c005180 2024-03-23T03:50:04.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.294+0000 7fac7ea00640 1 --2- 172.21.15.102:0/521764984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fac6806ac00 0x7fac6806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:04.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.294+0000 7fac7ea00640 1 -- 172.21.15.102:0/521764984 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fac78088f40 con 0x7fac6c005180 2024-03-23T03:50:04.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.294+0000 7fac85e00640 1 --2- 172.21.15.102:0/521764984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fac6806ac00 0x7fac6806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:04.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.295+0000 7fac85e00640 1 --2- 172.21.15.102:0/521764984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fac6806ac00 0x7fac6806d0c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fac74001eb0 tx=0x7fac74001150 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:04.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.299+0000 7fac7ea00640 1 -- 172.21.15.102:0/521764984 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fac78054290 con 0x7fac6c005180 2024-03-23T03:50:04.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.564+0000 7fac8d400640 1 -- 172.21.15.102:0/521764984 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fac88073390 con 0x7fac6c005180 2024-03-23T03:50:04.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.565+0000 7fac7ea00640 1 -- 172.21.15.102:0/521764984 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fac780578b0 con 0x7fac6c005180 2024-03-23T03:50:04.566 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:04.566 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:50:04.566 INFO:teuthology.orchestra.run.smithi102.stdout: "auth_dump": [ 2024-03-23T03:50:04.566 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.567 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.a", 2024-03-23T03:50:04.567 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5loBTZFRAA7/tTYyZwrDmwKwiU+dtCEw==", 2024-03-23T03:50:04.567 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.567 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:50:04.567 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:50:04.567 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:50:04.567 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:04.567 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.567 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.567 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.567 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.b", 2024-03-23T03:50:04.567 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5lJKKlNxAAj5+j99K79IhbeEm7/qC+wg==", 2024-03-23T03:50:04.567 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.567 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:50:04.567 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:50:04.568 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:50:04.568 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:04.568 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.568 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.568 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.568 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.c", 2024-03-23T03:50:04.568 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5l0b92JhAA5rvs+mdgmrQX/9lI3GW2rg==", 2024-03-23T03:50:04.568 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.568 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:50:04.568 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:50:04.568 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:50:04.569 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:04.569 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.569 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.569 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.569 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.d", 2024-03-23T03:50:04.569 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5ltjNMDBAAD1h5Nlv2Cp7mJt+DMlBtzw==", 2024-03-23T03:50:04.569 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.569 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:50:04.569 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:50:04.569 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:50:04.569 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:04.569 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.569 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.569 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.570 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.0", 2024-03-23T03:50:04.570 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB2UP5lPW6uORAAejiQReOF95oFliFmk/vdQQ==", 2024-03-23T03:50:04.570 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.570 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:50:04.570 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:50:04.570 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:04.570 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.570 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.570 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.570 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.1", 2024-03-23T03:50:04.570 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB6UP5lFbbUBxAAIl0ZEYtJ+/oi3W6a4OwMhg==", 2024-03-23T03:50:04.570 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.570 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:50:04.570 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:50:04.571 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:04.571 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.571 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.571 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.571 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.2", 2024-03-23T03:50:04.571 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB9UP5l2LaQIBAAKJv7jWJAjlUJrm7+KTXi6Q==", 2024-03-23T03:50:04.571 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.571 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:50:04.571 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:50:04.571 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:04.571 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.571 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.571 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.571 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.3", 2024-03-23T03:50:04.571 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCAUP5lxjWsKhAAzVO8CXUd+4LLvv4f/5xMOg==", 2024-03-23T03:50:04.572 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.572 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:50:04.572 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:50:04.572 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:04.572 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.572 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.572 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.572 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.4", 2024-03-23T03:50:04.572 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCHUP5lHcd2JRAAwID1TPb+Z9ViH51B0Mv/kQ==", 2024-03-23T03:50:04.572 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.572 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:50:04.572 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:50:04.572 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:04.572 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.572 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.573 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.573 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.5", 2024-03-23T03:50:04.573 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCLUP5lApp3AxAADslbuzTe/bf5FML1ckE05w==", 2024-03-23T03:50:04.573 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.573 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:50:04.573 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:50:04.573 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:04.573 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.573 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.573 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.573 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.6", 2024-03-23T03:50:04.573 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCOUP5lsi7VGxAArhWvXQ1VzJ8oSiOXzq8VsQ==", 2024-03-23T03:50:04.573 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.573 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:50:04.573 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:50:04.574 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:04.574 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.574 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.574 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.574 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.7", 2024-03-23T03:50:04.574 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCRUP5lu182JxAA5LaDXfrBkOXEUNAJXFDCJQ==", 2024-03-23T03:50:04.574 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.574 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:50:04.574 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:50:04.574 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:04.574 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.574 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.574 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.574 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.0", 2024-03-23T03:50:04.574 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lxTfoGRAA3WIQ8p1rGSAyJkLZJMk3dA==", 2024-03-23T03:50:04.575 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.575 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:50:04.575 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow r", 2024-03-23T03:50:04.575 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-23T03:50:04.575 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.575 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.575 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.575 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.1", 2024-03-23T03:50:04.575 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lneSHLhAAwn6ULOajHXrOM650HUcLbw==", 2024-03-23T03:50:04.575 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.575 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:50:04.575 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:50:04.575 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:50:04.575 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:50:04.575 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.576 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.576 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.576 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.2", 2024-03-23T03:50:04.576 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lOIklJBAAhx2d0Xy/j1PG0qsvC3nMqA==", 2024-03-23T03:50:04.576 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.576 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:50:04.576 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:50:04.576 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:50:04.576 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:50:04.576 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.576 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.576 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.576 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.3", 2024-03-23T03:50:04.576 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5l+7BVORAA7PBYS6mkgLA8b4sLCAYgBA==", 2024-03-23T03:50:04.576 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.577 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:50:04.577 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:50:04.577 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:50:04.577 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:50:04.577 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.577 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.577 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.577 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.admin", 2024-03-23T03:50:04.577 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBwUP5ldlDgBxAAOiqmhC5GulxELun8OiR4lg==", 2024-03-23T03:50:04.577 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.577 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:50:04.577 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:50:04.577 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:50:04.577 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:04.577 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.577 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.578 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.578 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mgr.x", 2024-03-23T03:50:04.578 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5lQXojChAAzepTDh0Pk3fuBZYsFt/+DQ==", 2024-03-23T03:50:04.578 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.578 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:50:04.578 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile mgr", 2024-03-23T03:50:04.578 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:04.578 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.578 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:04.578 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:04.578 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mgr.y", 2024-03-23T03:50:04.578 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBwUP5lK59iORAACuBF0JaQ8epyu3Mq3slzjg==", 2024-03-23T03:50:04.578 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:04.578 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:50:04.579 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile mgr", 2024-03-23T03:50:04.579 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:04.579 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.579 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:04.579 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-03-23T03:50:04.579 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:50:04.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.568+0000 7fac8d400640 1 -- 172.21.15.102:0/521764984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fac6806ac00 msgr2=0x7fac6806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:04.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.568+0000 7fac8d400640 1 --2- 172.21.15.102:0/521764984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fac6806ac00 0x7fac6806d0c0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fac74001eb0 tx=0x7fac74001150 comp rx=0 tx=0).stop 2024-03-23T03:50:04.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.568+0000 7fac8d400640 1 -- 172.21.15.102:0/521764984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6c005180 msgr2=0x7fac88156840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:04.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.568+0000 7fac8d400640 1 --2- 172.21.15.102:0/521764984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6c005180 0x7fac88156840 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fac7800e4d0 tx=0x7fac7800e990 comp rx=0 tx=0).stop 2024-03-23T03:50:04.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.568+0000 7fac8d400640 1 -- 172.21.15.102:0/521764984 shutdown_connections 2024-03-23T03:50:04.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.568+0000 7fac8d400640 1 --2- 172.21.15.102:0/521764984 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fac6806ac00 0x7fac6806d0c0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.568+0000 7fac8d400640 1 --2- 172.21.15.102:0/521764984 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fac88154a10 0x7fac88148e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.568+0000 7fac8d400640 1 --2- 172.21.15.102:0/521764984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fac88154590 0x7fac881488c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.568+0000 7fac8d400640 1 --2- 172.21.15.102:0/521764984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac6c005180 0x7fac88156840 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.568+0000 7fac8d400640 1 -- 172.21.15.102:0/521764984 >> 172.21.15.102:0/521764984 conn(0x7fac8806a3d0 msgr2=0x7fac880b1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:04.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.568+0000 7fac8d400640 1 -- 172.21.15.102:0/521764984 shutdown_connections 2024-03-23T03:50:04.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.568+0000 7fac8d400640 1 -- 172.21.15.102:0/521764984 wait complete. 2024-03-23T03:50:04.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:50:04.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.789+0000 7f337c800640 1 Processor -- start 2024-03-23T03:50:04.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.790+0000 7f337c800640 1 -- start start 2024-03-23T03:50:04.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.791+0000 7f337c800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f337415cda0 0x7f33740cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:04.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.791+0000 7f337c800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f33740b5a30 0x7f33740b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:04.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.791+0000 7f337c800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33740d1830 con 0x7f33740b4350 2024-03-23T03:50:04.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.791+0000 7f337c800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33740b1f20 con 0x7f33740b5a30 2024-03-23T03:50:04.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.791+0000 7f337c800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33740d0c20 con 0x7f337415cda0 2024-03-23T03:50:04.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.791+0000 7f3379400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f33740b5a30 0x7f33740b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:04.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.791+0000 7f3379400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f33740b5a30 0x7f33740b5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59752/0 (socket says 172.21.15.102:59752) 2024-03-23T03:50:04.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.791+0000 7f3379400640 1 -- 172.21.15.102:0/4213042528 learned_addr learned my addr 172.21.15.102:0/4213042528 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:04.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.791+0000 7f3379400640 1 -- 172.21.15.102:0/4213042528 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f337415cda0 msgr2=0x7f33740cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:04.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.791+0000 7f3379400640 1 --2- 172.21.15.102:0/4213042528 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f337415cda0 0x7f33740cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.791+0000 7f3379400640 1 -- 172.21.15.102:0/4213042528 >> v1:172.21.15.173:6789/0 conn(0x7f33740b4350 legacy=0x7f33740d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:04.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.791+0000 7f3379400640 1 -- 172.21.15.102:0/4213042528 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33740d0f00 con 0x7f33740b5a30 2024-03-23T03:50:04.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.792+0000 7f3379400640 1 --2- 172.21.15.102:0/4213042528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33740b5a30 0x7f33740b5e90 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3360009870 tx=0x7f336002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1018f9396ef100 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:04.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.793+0000 7f3378a00640 1 -- 172.21.15.102:0/4213042528 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360031640 con 0x7f33740b5a30 2024-03-23T03:50:04.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.793+0000 7f3378a00640 1 -- 172.21.15.102:0/4213042528 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33600317a0 con 0x7f33740b5a30 2024-03-23T03:50:04.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.793+0000 7f3378a00640 1 -- 172.21.15.102:0/4213042528 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360031b30 con 0x7f33740b5a30 2024-03-23T03:50:04.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.793+0000 7f337c800640 1 -- 172.21.15.102:0/4213042528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33740b5a30 msgr2=0x7f33740b5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:04.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.793+0000 7f337c800640 1 --2- 172.21.15.102:0/4213042528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33740b5a30 0x7f33740b5e90 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f3360009870 tx=0x7f336002b8f0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.793+0000 7f337c800640 1 -- 172.21.15.102:0/4213042528 shutdown_connections 2024-03-23T03:50:04.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.793+0000 7f337c800640 1 --2- 172.21.15.102:0/4213042528 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33740b5a30 0x7f33740b5e90 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.793+0000 7f337c800640 1 --2- 172.21.15.102:0/4213042528 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f337415cda0 0x7f33740cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.793+0000 7f337c800640 1 -- 172.21.15.102:0/4213042528 >> 172.21.15.102:0/4213042528 conn(0x7f33740cf350 msgr2=0x7f33740cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:04.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.793+0000 7f337c800640 1 -- 172.21.15.102:0/4213042528 shutdown_connections 2024-03-23T03:50:04.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.793+0000 7f337c800640 1 -- 172.21.15.102:0/4213042528 wait complete. 2024-03-23T03:50:04.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.795+0000 7f337c800640 1 Processor -- start 2024-03-23T03:50:04.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.795+0000 7f337c800640 1 -- start start 2024-03-23T03:50:04.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.796+0000 7f337c800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33740b4350 0x7f3374158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:04.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.796+0000 7f337c800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33740b5a30 0x7f337415d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:04.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.796+0000 7f337c800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f337415cda0 0x7f337415d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:04.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.796+0000 7f337c800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33740d2600 con 0x7f33740b4350 2024-03-23T03:50:04.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.796+0000 7f337c800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3374159480 con 0x7f33740b5a30 2024-03-23T03:50:04.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.796+0000 7f337c800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33740c6580 con 0x7f337415cda0 2024-03-23T03:50:04.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.796+0000 7f3379e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33740b4350 0x7f3374158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:04.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.796+0000 7f3379e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33740b4350 0x7f3374158f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59764/0 (socket says 172.21.15.102:59764) 2024-03-23T03:50:04.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.796+0000 7f3379e00640 1 -- 172.21.15.102:0/3128319895 learned_addr learned my addr 172.21.15.102:0/3128319895 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:04.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.797+0000 7f3379e00640 1 -- 172.21.15.102:0/3128319895 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f337415cda0 msgr2=0x7f337415d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:04.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.797+0000 7f3379e00640 1 --2- 172.21.15.102:0/3128319895 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f337415cda0 0x7f337415d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.797+0000 7f3379e00640 1 -- 172.21.15.102:0/3128319895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33740b5a30 msgr2=0x7f337415d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:04.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.797+0000 7f3379e00640 1 --2- 172.21.15.102:0/3128319895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33740b5a30 0x7f337415d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:04.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.797+0000 7f3379e00640 1 -- 172.21.15.102:0/3128319895 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33600094d0 con 0x7f33740b4350 2024-03-23T03:50:04.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.797+0000 7f3379e00640 1 --2- 172.21.15.102:0/3128319895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33740b4350 0x7f3374158f40 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f336800b410 tx=0x7f336800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:04.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.797+0000 7f336e000640 1 -- 172.21.15.102:0/3128319895 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3368011830 con 0x7f33740b4350 2024-03-23T03:50:04.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.797+0000 7f336e000640 1 -- 172.21.15.102:0/3128319895 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33680119b0 con 0x7f33740b4350 2024-03-23T03:50:04.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.797+0000 7f336e000640 1 -- 172.21.15.102:0/3128319895 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3368011d60 con 0x7f33740b4350 2024-03-23T03:50:04.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.797+0000 7f337c800640 1 -- 172.21.15.102:0/3128319895 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3374160000 con 0x7f33740b4350 2024-03-23T03:50:04.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.798+0000 7f337c800640 1 -- 172.21.15.102:0/3128319895 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33740d2260 con 0x7f33740b4350 2024-03-23T03:50:04.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.799+0000 7f337c800640 1 -- 172.21.15.102:0/3128319895 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33740c6a80 con 0x7f33740b4350 2024-03-23T03:50:04.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.799+0000 7f336e000640 1 -- 172.21.15.102:0/3128319895 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f336801a400 con 0x7f33740b4350 2024-03-23T03:50:04.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.800+0000 7f336e000640 1 --2- 172.21.15.102:0/3128319895 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f335406ac00 0x7f335406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:04.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.800+0000 7f336e000640 1 -- 172.21.15.102:0/3128319895 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f3368089370 con 0x7f33740b4350 2024-03-23T03:50:04.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.800+0000 7f3379400640 1 --2- 172.21.15.102:0/3128319895 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f335406ac00 0x7f335406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:04.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.801+0000 7f3379400640 1 --2- 172.21.15.102:0/3128319895 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f335406ac00 0x7f335406d0c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3360036000 tx=0x7f3360030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:04.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:04.805+0000 7f336e000640 1 -- 172.21.15.102:0/3128319895 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f33740c6a80 con 0x7f33740b4350 2024-03-23T03:50:05.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.003+0000 7f337c800640 1 -- 172.21.15.102:0/3128319895 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33740c6a80 con 0x7f33740b4350 2024-03-23T03:50:05.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.005+0000 7f336e000640 1 -- 172.21.15.102:0/3128319895 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+12643 (secure 0 0 0) 0x7f33740c6a80 con 0x7f33740b4350 2024-03-23T03:50:05.005 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:05.005 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":77,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:04.044680+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:50:05.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.007+0000 7f337c800640 1 -- 172.21.15.102:0/3128319895 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f335406ac00 msgr2=0x7f335406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:05.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.007+0000 7f337c800640 1 --2- 172.21.15.102:0/3128319895 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f335406ac00 0x7f335406d0c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f3360036000 tx=0x7f3360030000 comp rx=0 tx=0).stop 2024-03-23T03:50:05.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.007+0000 7f337c800640 1 -- 172.21.15.102:0/3128319895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33740b4350 msgr2=0x7f3374158f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:05.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.007+0000 7f337c800640 1 --2- 172.21.15.102:0/3128319895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33740b4350 0x7f3374158f40 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f336800b410 tx=0x7f336800b8d0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.007+0000 7f337c800640 1 -- 172.21.15.102:0/3128319895 shutdown_connections 2024-03-23T03:50:05.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.007+0000 7f337c800640 1 --2- 172.21.15.102:0/3128319895 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f335406ac00 0x7f335406d0c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.007+0000 7f337c800640 1 --2- 172.21.15.102:0/3128319895 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f337415cda0 0x7f337415d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.007+0000 7f337c800640 1 --2- 172.21.15.102:0/3128319895 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33740b5a30 0x7f337415d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.007+0000 7f337c800640 1 --2- 172.21.15.102:0/3128319895 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33740b4350 0x7f3374158f40 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.007+0000 7f337c800640 1 -- 172.21.15.102:0/3128319895 >> 172.21.15.102:0/3128319895 conn(0x7f33740cf350 msgr2=0x7f33740cc2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:05.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.007+0000 7f337c800640 1 -- 172.21.15.102:0/3128319895 shutdown_connections 2024-03-23T03:50:05.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.008+0000 7f337c800640 1 -- 172.21.15.102:0/3128319895 wait complete. 2024-03-23T03:50:05.027 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:50:05.028 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:50:05.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.227+0000 7fb5c8400640 1 Processor -- start 2024-03-23T03:50:05.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.227+0000 7fb5c8400640 1 -- start start 2024-03-23T03:50:05.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.229+0000 7fb5c8400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb5c00ce930 0x7fb5c00add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:05.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.229+0000 7fb5c8400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb5c00cd3e0 0x7fb5c00ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:05.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.229+0000 7fb5c8400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb5c006ce90 0x7fb5c006d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:05.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.229+0000 7fb5c8400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c006c760 con 0x7fb5c00cd3e0 2024-03-23T03:50:05.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.229+0000 7fb5c8400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c00b42d0 con 0x7fb5c006ce90 2024-03-23T03:50:05.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.229+0000 7fb5c8400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c00cf250 con 0x7fb5c00ce930 2024-03-23T03:50:05.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.230+0000 7fb5c5000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb5c00cd3e0 0x7fb5c00ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:05.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.230+0000 7fb5c5000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb5c00cd3e0 0x7fb5c00ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59768/0 (socket says 172.21.15.102:59768) 2024-03-23T03:50:05.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.230+0000 7fb5c5000640 1 -- 172.21.15.102:0/3377570777 learned_addr learned my addr 172.21.15.102:0/3377570777 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:05.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.230+0000 7fb5c5a00640 1 --2- 172.21.15.102:0/3377570777 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb5c00ce930 0x7fb5c00add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:05.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.230+0000 7fb5c6400640 1 --2- 172.21.15.102:0/3377570777 >> v2:172.21.15.173:3300/0 conn(0x7fb5c006ce90 0x7fb5c006d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:05.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.230+0000 7fb5c5000640 1 -- 172.21.15.102:0/3377570777 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb5c00ce930 msgr2=0x7fb5c00add30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:05.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.230+0000 7fb5c5000640 1 --2- 172.21.15.102:0/3377570777 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb5c00ce930 0x7fb5c00add30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.230+0000 7fb5c5000640 1 -- 172.21.15.102:0/3377570777 >> v2:172.21.15.173:3300/0 conn(0x7fb5c006ce90 msgr2=0x7fb5c006d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:05.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.230+0000 7fb5c5000640 1 --2- 172.21.15.102:0/3377570777 >> v2:172.21.15.173:3300/0 conn(0x7fb5c006ce90 0x7fb5c006d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.230+0000 7fb5c5000640 1 -- 172.21.15.102:0/3377570777 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c00cdba0 con 0x7fb5c00cd3e0 2024-03-23T03:50:05.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.232+0000 7fb5c5000640 1 --2- 172.21.15.102:0/3377570777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5c00cd3e0 0x7fb5c00ced10 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fb5c00ccc70 tx=0x7fb5ac02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6014c362e5fd8cc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:05.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.232+0000 7fb5bfe00640 1 -- 172.21.15.102:0/3377570777 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac031860 con 0x7fb5c00cd3e0 2024-03-23T03:50:05.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.232+0000 7fb5bfe00640 1 -- 172.21.15.102:0/3377570777 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5ac0319c0 con 0x7fb5c00cd3e0 2024-03-23T03:50:05.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.232+0000 7fb5bfe00640 1 -- 172.21.15.102:0/3377570777 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac031d50 con 0x7fb5c00cd3e0 2024-03-23T03:50:05.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.232+0000 7fb5c8400640 1 -- 172.21.15.102:0/3377570777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5c00cd3e0 msgr2=0x7fb5c00ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:05.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.232+0000 7fb5c8400640 1 --2- 172.21.15.102:0/3377570777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5c00cd3e0 0x7fb5c00ced10 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fb5c00ccc70 tx=0x7fb5ac02b900 comp rx=0 tx=0).stop 2024-03-23T03:50:05.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.233+0000 7fb5c8400640 1 -- 172.21.15.102:0/3377570777 shutdown_connections 2024-03-23T03:50:05.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.233+0000 7fb5c8400640 1 --2- 172.21.15.102:0/3377570777 >> v2:172.21.15.173:3300/0 conn(0x7fb5c006ce90 0x7fb5c006d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.233+0000 7fb5c8400640 1 --2- 172.21.15.102:0/3377570777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5c00cd3e0 0x7fb5c00ced10 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.233+0000 7fb5c8400640 1 --2- 172.21.15.102:0/3377570777 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb5c00ce930 0x7fb5c00add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.233+0000 7fb5c8400640 1 -- 172.21.15.102:0/3377570777 >> 172.21.15.102:0/3377570777 conn(0x7fb5c006a0b0 msgr2=0x7fb5c006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:05.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.233+0000 7fb5c8400640 1 -- 172.21.15.102:0/3377570777 shutdown_connections 2024-03-23T03:50:05.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.233+0000 7fb5c8400640 1 -- 172.21.15.102:0/3377570777 wait complete. 2024-03-23T03:50:05.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.234+0000 7fb5c8400640 1 Processor -- start 2024-03-23T03:50:05.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.235+0000 7fb5c8400640 1 -- start start 2024-03-23T03:50:05.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.236+0000 7fb5c8400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb5c006ce90 0x7fb5c01533c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:05.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.236+0000 7fb5c8400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c00cd3e0 0x7fb5c015b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:05.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.236+0000 7fb5c8400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5c00ce930 0x7fb5c014fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:05.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.236+0000 7fb5c8400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c015a860 con 0x7fb5c00ce930 2024-03-23T03:50:05.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.236+0000 7fb5c8400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c015b3a0 con 0x7fb5c00cd3e0 2024-03-23T03:50:05.238 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.236+0000 7fb5c8400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c015bd70 con 0x7fb5c006ce90 2024-03-23T03:50:05.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.236+0000 7fb5c6400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5c00ce930 0x7fb5c014fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:05.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.236+0000 7fb5c6400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5c00ce930 0x7fb5c014fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59772/0 (socket says 172.21.15.102:59772) 2024-03-23T03:50:05.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.236+0000 7fb5c6400640 1 -- 172.21.15.102:0/1461154486 learned_addr learned my addr 172.21.15.102:0/1461154486 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:05.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.236+0000 7fb5c5a00640 1 --2- 172.21.15.102:0/1461154486 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb5c006ce90 0x7fb5c01533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:05.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.236+0000 7fb5c5000640 1 --2- 172.21.15.102:0/1461154486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c00cd3e0 0x7fb5c015b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:05.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.236+0000 7fb5c6400640 1 -- 172.21.15.102:0/1461154486 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb5c006ce90 msgr2=0x7fb5c01533c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:05.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.236+0000 7fb5c6400640 1 --2- 172.21.15.102:0/1461154486 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb5c006ce90 0x7fb5c01533c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.236+0000 7fb5c6400640 1 -- 172.21.15.102:0/1461154486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c00cd3e0 msgr2=0x7fb5c015b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:05.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.236+0000 7fb5c6400640 1 --2- 172.21.15.102:0/1461154486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c00cd3e0 0x7fb5c015b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.236+0000 7fb5c6400640 1 -- 172.21.15.102:0/1461154486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac0094d0 con 0x7fb5c00ce930 2024-03-23T03:50:05.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.237+0000 7fb5c5000640 1 --2- 172.21.15.102:0/1461154486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c00cd3e0 0x7fb5c015b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:05.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.237+0000 7fb5c6400640 1 --2- 172.21.15.102:0/1461154486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5c00ce930 0x7fb5c014fb40 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8007e90 tx=0x7fb5b800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:05.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.237+0000 7fb5bd600640 1 -- 172.21.15.102:0/1461154486 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8014420 con 0x7fb5c00ce930 2024-03-23T03:50:05.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.237+0000 7fb5bd600640 1 -- 172.21.15.102:0/1461154486 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b8014580 con 0x7fb5c00ce930 2024-03-23T03:50:05.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.237+0000 7fb5bd600640 1 -- 172.21.15.102:0/1461154486 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8014930 con 0x7fb5c00ce930 2024-03-23T03:50:05.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.237+0000 7fb5c8400640 1 -- 172.21.15.102:0/1461154486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c015aa70 con 0x7fb5c00ce930 2024-03-23T03:50:05.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.237+0000 7fb5c8400640 1 -- 172.21.15.102:0/1461154486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c015b030 con 0x7fb5c00ce930 2024-03-23T03:50:05.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.239+0000 7fb5c8400640 1 -- 172.21.15.102:0/1461154486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c015b530 con 0x7fb5c00ce930 2024-03-23T03:50:05.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.240+0000 7fb5bd600640 1 -- 172.21.15.102:0/1461154486 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb5b8014a90 con 0x7fb5c00ce930 2024-03-23T03:50:05.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.241+0000 7fb5bd600640 1 --2- 172.21.15.102:0/1461154486 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb5a006ac00 0x7fb5a006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:05.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.241+0000 7fb5bd600640 1 -- 172.21.15.102:0/1461154486 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb5b8089a20 con 0x7fb5c00ce930 2024-03-23T03:50:05.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.241+0000 7fb5c5a00640 1 --2- 172.21.15.102:0/1461154486 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb5a006ac00 0x7fb5a006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:05.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.242+0000 7fb5c5a00640 1 --2- 172.21.15.102:0/1461154486 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb5a006ac00 0x7fb5a006d0c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4009a50 tx=0x7fb5b4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:05.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.245+0000 7fb5bd600640 1 -- 172.21.15.102:0/1461154486 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb5b8054ea0 con 0x7fb5c00ce930 2024-03-23T03:50:05.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.441+0000 7fb5c8400640 1 -- 172.21.15.102:0/1461154486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb5c0073390 con 0x7fb5c00ce930 2024-03-23T03:50:05.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.442+0000 7fb5bd600640 1 -- 172.21.15.102:0/1461154486 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb5b80584c0 con 0x7fb5c00ce930 2024-03-23T03:50:05.443 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:50:05.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.445+0000 7fb5c8400640 1 -- 172.21.15.102:0/1461154486 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb5a006ac00 msgr2=0x7fb5a006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:05.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.445+0000 7fb5c8400640 1 --2- 172.21.15.102:0/1461154486 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb5a006ac00 0x7fb5a006d0c0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4009a50 tx=0x7fb5b4008040 comp rx=0 tx=0).stop 2024-03-23T03:50:05.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.445+0000 7fb5c8400640 1 -- 172.21.15.102:0/1461154486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5c00ce930 msgr2=0x7fb5c014fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:05.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.445+0000 7fb5c8400640 1 --2- 172.21.15.102:0/1461154486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5c00ce930 0x7fb5c014fb40 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8007e90 tx=0x7fb5b800e4b0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.445+0000 7fb5c8400640 1 -- 172.21.15.102:0/1461154486 shutdown_connections 2024-03-23T03:50:05.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.445+0000 7fb5c8400640 1 --2- 172.21.15.102:0/1461154486 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb5a006ac00 0x7fb5a006d0c0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.445+0000 7fb5c8400640 1 --2- 172.21.15.102:0/1461154486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5c00ce930 0x7fb5c014fb40 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.445+0000 7fb5c8400640 1 --2- 172.21.15.102:0/1461154486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb5c00cd3e0 0x7fb5c015b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.445+0000 7fb5c8400640 1 --2- 172.21.15.102:0/1461154486 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb5c006ce90 0x7fb5c01533c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.445+0000 7fb5c8400640 1 -- 172.21.15.102:0/1461154486 >> 172.21.15.102:0/1461154486 conn(0x7fb5c006a0b0 msgr2=0x7fb5c00b0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:05.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.445+0000 7fb5c8400640 1 -- 172.21.15.102:0/1461154486 shutdown_connections 2024-03-23T03:50:05.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.445+0000 7fb5c8400640 1 -- 172.21.15.102:0/1461154486 wait complete. 2024-03-23T03:50:05.463 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T03:50:05.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.658+0000 7f10d2800640 1 Processor -- start 2024-03-23T03:50:05.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.658+0000 7f10d2800640 1 -- start start 2024-03-23T03:50:05.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.660+0000 7f10d2800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f10cc13f200 0x7f10cc0ccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:05.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.660+0000 7f10d2800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc0ce440 con 0x7f10cc0cd520 2024-03-23T03:50:05.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.660+0000 7f10d2800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10cc0cdba0 con 0x7f10cc15d350 2024-03-23T03:50:05.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.660+0000 7f10d2800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10cc0cd9b0 con 0x7f10cc13f200 2024-03-23T03:50:05.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.660+0000 7f10caa00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f10cc0cd520 0x7f10cc15cc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38630/0 (socket says 172.21.15.102:38630) 2024-03-23T03:50:05.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.660+0000 7f10caa00640 1 -- 172.21.15.102:0/4127207572 learned_addr learned my addr 172.21.15.102:0/4127207572 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:05.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.660+0000 7f10cb400640 1 --2- 172.21.15.102:0/4127207572 >> v2:172.21.15.173:3300/0 conn(0x7f10cc13f200 0x7f10cc0ccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:05.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.660+0000 7f10ca000640 1 -- 172.21.15.102:0/4127207572 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248326403 0 0) 0x7f10cc0ce440 con 0x7f10cc0cd520 2024-03-23T03:50:05.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.660+0000 7f10ca000640 1 -- 172.21.15.102:0/4127207572 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b0003520 con 0x7f10cc0cd520 2024-03-23T03:50:05.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.661+0000 7f10ca000640 1 -- 172.21.15.102:0/4127207572 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 486378144 0 0) 0x7f10b0003520 con 0x7f10cc0cd520 2024-03-23T03:50:05.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.661+0000 7f10ca000640 1 -- 172.21.15.102:0/4127207572 >> v2:172.21.15.173:3300/0 conn(0x7f10cc13f200 msgr2=0x7f10cc0ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:05.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.661+0000 7f10ca000640 1 --2- 172.21.15.102:0/4127207572 >> v2:172.21.15.173:3300/0 conn(0x7f10cc13f200 0x7f10cc0ccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.661+0000 7f10ca000640 1 -- 172.21.15.102:0/4127207572 >> v1:172.21.15.173:6789/0 conn(0x7f10cc15d350 legacy=0x7f10cc06c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:05.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.661+0000 7f10ca000640 1 -- 172.21.15.102:0/4127207572 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc0cde80 con 0x7f10cc0cd520 2024-03-23T03:50:05.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.661+0000 7f10ca000640 1 -- 172.21.15.102:0/4127207572 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f10b8001670 con 0x7f10cc0cd520 2024-03-23T03:50:05.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.661+0000 7f10ca000640 1 -- 172.21.15.102:0/4127207572 >> v1:172.21.15.102:6789/0 conn(0x7f10cc0cd520 legacy=0x7f10cc15cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:05.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.661+0000 7f10ca000640 1 --2- 172.21.15.102:0/4127207572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b0003970 0x7f10b0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.661+0000 7f10ca000640 1 --2- 172.21.15.102:0/4127207572 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10b00047c0 0x7f10b0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.661+0000 7f10ca000640 1 --2- 172.21.15.102:0/4127207572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10b0005180 0x7f10b000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.661+0000 7f10ca000640 1 -- 172.21.15.102:0/4127207572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b000c0f0 con 0x7f10b0003970 2024-03-23T03:50:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.661+0000 7f10ca000640 1 -- 172.21.15.102:0/4127207572 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b000c260 con 0x7f10b0005180 2024-03-23T03:50:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.661+0000 7f10ca000640 1 -- 172.21.15.102:0/4127207572 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b000e630 con 0x7f10b00047c0 2024-03-23T03:50:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.661+0000 7f10cb400640 1 --2- 172.21.15.102:0/4127207572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b0003970 0x7f10b0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.662+0000 7f10caa00640 1 --2- 172.21.15.102:0/4127207572 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10b00047c0 0x7f10b0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.662+0000 7f10cbe00640 1 --2- 172.21.15.102:0/4127207572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10b0005180 0x7f10b000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.662+0000 7f10cb400640 1 -- 172.21.15.102:0/4127207572 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10b00047c0 msgr2=0x7f10b0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.662+0000 7f10cb400640 1 --2- 172.21.15.102:0/4127207572 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10b00047c0 0x7f10b0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.662+0000 7f10cb400640 1 -- 172.21.15.102:0/4127207572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10b0005180 msgr2=0x7f10b000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.662+0000 7f10cb400640 1 --2- 172.21.15.102:0/4127207572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10b0005180 0x7f10b000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.662+0000 7f10cb400640 1 -- 172.21.15.102:0/4127207572 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b0003520 con 0x7f10b0003970 2024-03-23T03:50:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.662+0000 7f10cbe00640 1 --2- 172.21.15.102:0/4127207572 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10b0005180 0x7f10b000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:05.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.662+0000 7f10cbe00640 1 -- 172.21.15.102:0/4127207572 reap_dead start 2024-03-23T03:50:05.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.662+0000 7f10caa00640 1 --2- 172.21.15.102:0/4127207572 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10b00047c0 0x7f10b0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:05.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.663+0000 7f10cb400640 1 --2- 172.21.15.102:0/4127207572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b0003970 0x7f10b0003d90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f10bc009ac0 tx=0x7f10b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77bdbf18b5a4cb4c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:05.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.663+0000 7f10ca000640 1 -- 172.21.15.102:0/4127207572 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10bc02c6d0 con 0x7f10b0003970 2024-03-23T03:50:05.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.663+0000 7f10ca000640 1 -- 172.21.15.102:0/4127207572 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10bc02c830 con 0x7f10b0003970 2024-03-23T03:50:05.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.663+0000 7f10ca000640 1 -- 172.21.15.102:0/4127207572 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10bc02cbc0 con 0x7f10b0003970 2024-03-23T03:50:05.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.664+0000 7f10d2800640 1 -- 172.21.15.102:0/4127207572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b0003970 msgr2=0x7f10b0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:05.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.664+0000 7f10d2800640 1 --2- 172.21.15.102:0/4127207572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b0003970 0x7f10b0003d90 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f10bc009ac0 tx=0x7f10b00057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.664+0000 7f10d2800640 1 -- 172.21.15.102:0/4127207572 shutdown_connections 2024-03-23T03:50:05.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.664+0000 7f10d2800640 1 --2- 172.21.15.102:0/4127207572 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b0003970 0x7f10b0003d90 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.664+0000 7f10d2800640 1 -- 172.21.15.102:0/4127207572 >> 172.21.15.102:0/4127207572 conn(0x7f10cc06a3d0 msgr2=0x7f10cc0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:05.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.664+0000 7f10d2800640 1 -- 172.21.15.102:0/4127207572 shutdown_connections 2024-03-23T03:50:05.666 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.664+0000 7f10d2800640 1 -- 172.21.15.102:0/4127207572 wait complete. 2024-03-23T03:50:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.666+0000 7f10d2800640 1 Processor -- start 2024-03-23T03:50:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.666+0000 7f10d2800640 1 -- start start 2024-03-23T03:50:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.667+0000 7f10d2800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b0003970 0x7f10cc153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.667+0000 7f10d2800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10cc158650 0x7f10cc155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.667+0000 7f10d2800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10cc155a70 0x7f10cc13c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.667+0000 7f10d2800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc155ef0 con 0x7f10b0003970 2024-03-23T03:50:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.667+0000 7f10d2800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc1581c0 con 0x7f10cc155a70 2024-03-23T03:50:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.667+0000 7f10d2800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10cc157890 con 0x7f10cc158650 2024-03-23T03:50:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.667+0000 7f10cb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b0003970 0x7f10cc153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.667+0000 7f10cb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b0003970 0x7f10cc153200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59802/0 (socket says 172.21.15.102:59802) 2024-03-23T03:50:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.667+0000 7f10cb400640 1 -- 172.21.15.102:0/4069872901 learned_addr learned my addr 172.21.15.102:0/4069872901 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.667+0000 7f10cb400640 1 -- 172.21.15.102:0/4069872901 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10cc158650 msgr2=0x7f10cc155530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:05.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.667+0000 7f10caa00640 1 --2- 172.21.15.102:0/4069872901 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10cc158650 0x7f10cc155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.667+0000 7f10cb400640 1 --2- 172.21.15.102:0/4069872901 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10cc158650 0x7f10cc155530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.667+0000 7f10cbe00640 1 --2- 172.21.15.102:0/4069872901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10cc155a70 0x7f10cc13c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.667+0000 7f10cb400640 1 -- 172.21.15.102:0/4069872901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10cc155a70 msgr2=0x7f10cc13c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.667+0000 7f10cb400640 1 --2- 172.21.15.102:0/4069872901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10cc155a70 0x7f10cc13c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.667+0000 7f10cb400640 1 -- 172.21.15.102:0/4069872901 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10cc157b70 con 0x7f10b0003970 2024-03-23T03:50:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.668+0000 7f10cb400640 1 --2- 172.21.15.102:0/4069872901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b0003970 0x7f10cc153200 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f10bc02c270 tx=0x7f10bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.668+0000 7f10c3400640 1 -- 172.21.15.102:0/4069872901 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10bc003810 con 0x7f10b0003970 2024-03-23T03:50:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.668+0000 7f10c3400640 1 -- 172.21.15.102:0/4069872901 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10bc003970 con 0x7f10b0003970 2024-03-23T03:50:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.668+0000 7f10c3400640 1 -- 172.21.15.102:0/4069872901 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10bc003d00 con 0x7f10b0003970 2024-03-23T03:50:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.668+0000 7f10d2800640 1 -- 172.21.15.102:0/4069872901 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10cc157e50 con 0x7f10b0003970 2024-03-23T03:50:05.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.668+0000 7f10d2800640 1 -- 172.21.15.102:0/4069872901 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10cc157540 con 0x7f10b0003970 2024-03-23T03:50:05.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.669+0000 7f10d2800640 1 -- 172.21.15.102:0/4069872901 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10cc06ac20 con 0x7f10b0003970 2024-03-23T03:50:05.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.670+0000 7f10c3400640 1 -- 172.21.15.102:0/4069872901 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f10bc034440 con 0x7f10b0003970 2024-03-23T03:50:05.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.671+0000 7f10c3400640 1 --2- 172.21.15.102:0/4069872901 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f10ac06ac30 0x7f10ac06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:05.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.671+0000 7f10c3400640 1 -- 172.21.15.102:0/4069872901 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f10bc0ab310 con 0x7f10b0003970 2024-03-23T03:50:05.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.672+0000 7f10caa00640 1 --2- 172.21.15.102:0/4069872901 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f10ac06ac30 0x7f10ac06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:05.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.672+0000 7f10caa00640 1 --2- 172.21.15.102:0/4069872901 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f10ac06ac30 0x7f10ac06d0f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f10b8001eb0 tx=0x7f10b8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:05.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.676+0000 7f10c3400640 1 -- 172.21.15.102:0/4069872901 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f10bc0768c0 con 0x7f10b0003970 2024-03-23T03:50:05.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.861+0000 7f10d2800640 1 -- 172.21.15.102:0/4069872901 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f10cc073390 con 0x7f10b0003970 2024-03-23T03:50:05.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.861+0000 7f10c3400640 1 -- 172.21.15.102:0/4069872901 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) v1 ==== 84+0+65 (secure 0 0 0) 0x7f10bc003e60 con 0x7f10b0003970 2024-03-23T03:50:05.862 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:05.862 INFO:teuthology.orchestra.run.smithi102.stdout:[ 2024-03-23T03:50:05.862 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:05.862 INFO:teuthology.orchestra.run.smithi102.stdout: "poolnum": 1, 2024-03-23T03:50:05.862 INFO:teuthology.orchestra.run.smithi102.stdout: "poolname": ".mgr" 2024-03-23T03:50:05.863 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:05.863 INFO:teuthology.orchestra.run.smithi102.stdout:] 2024-03-23T03:50:05.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.864+0000 7f10d2800640 1 -- 172.21.15.102:0/4069872901 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f10ac06ac30 msgr2=0x7f10ac06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:05.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.864+0000 7f10d2800640 1 --2- 172.21.15.102:0/4069872901 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f10ac06ac30 0x7f10ac06d0f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f10b8001eb0 tx=0x7f10b8009040 comp rx=0 tx=0).stop 2024-03-23T03:50:05.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.864+0000 7f10d2800640 1 -- 172.21.15.102:0/4069872901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b0003970 msgr2=0x7f10cc153200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:05.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.864+0000 7f10d2800640 1 --2- 172.21.15.102:0/4069872901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b0003970 0x7f10cc153200 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f10bc02c270 tx=0x7f10bc004730 comp rx=0 tx=0).stop 2024-03-23T03:50:05.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.864+0000 7f10d2800640 1 -- 172.21.15.102:0/4069872901 shutdown_connections 2024-03-23T03:50:05.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.864+0000 7f10d2800640 1 --2- 172.21.15.102:0/4069872901 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f10ac06ac30 0x7f10ac06d0f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.864+0000 7f10d2800640 1 --2- 172.21.15.102:0/4069872901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10cc155a70 0x7f10cc13c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.864+0000 7f10d2800640 1 --2- 172.21.15.102:0/4069872901 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10cc158650 0x7f10cc155530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.864+0000 7f10d2800640 1 --2- 172.21.15.102:0/4069872901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b0003970 0x7f10cc153200 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:05.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.864+0000 7f10d2800640 1 -- 172.21.15.102:0/4069872901 >> 172.21.15.102:0/4069872901 conn(0x7f10cc06a3d0 msgr2=0x7f10cc070070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:05.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.864+0000 7f10d2800640 1 -- 172.21.15.102:0/4069872901 shutdown_connections 2024-03-23T03:50:05.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:05.864+0000 7f10d2800640 1 -- 172.21.15.102:0/4069872901 wait complete. 2024-03-23T03:50:05.883 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T03:50:05.883 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T03:50:06.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.096+0000 7feb41a00640 1 Processor -- start 2024-03-23T03:50:06.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.096+0000 7feb41a00640 1 -- start start 2024-03-23T03:50:06.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.098+0000 7feb41a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feb3c0add30 0x7feb3c0ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:06.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.098+0000 7feb41a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7feb3c06ce90 0x7feb3c06d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:06.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.098+0000 7feb41a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb3c0cf250 con 0x7feb3c0ce930 2024-03-23T03:50:06.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.098+0000 7feb41a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb3c0ae1c0 con 0x7feb3c0add30 2024-03-23T03:50:06.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.098+0000 7feb41a00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb3c06c760 con 0x7feb3c06ce90 2024-03-23T03:50:06.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.099+0000 7feb32000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feb3c0add30 0x7feb3c0ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:06.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.099+0000 7feb3b400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7feb3c06ce90 0x7feb3c06d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:06.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.099+0000 7feb32000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7feb3c0add30 0x7feb3c0ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59810/0 (socket says 172.21.15.102:59810) 2024-03-23T03:50:06.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.099+0000 7feb32000640 1 -- 172.21.15.102:0/2836123532 learned_addr learned my addr 172.21.15.102:0/2836123532 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:06.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.099+0000 7feb32000640 1 -- 172.21.15.102:0/2836123532 >> v2:172.21.15.173:3300/0 conn(0x7feb3c06ce90 msgr2=0x7feb3c06d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:06.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.099+0000 7feb32000640 1 --2- 172.21.15.102:0/2836123532 >> v2:172.21.15.173:3300/0 conn(0x7feb3c06ce90 0x7feb3c06d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:06.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.099+0000 7feb32000640 1 -- 172.21.15.102:0/2836123532 >> v1:172.21.15.173:6789/0 conn(0x7feb3c0ce930 legacy=0x7feb3c0cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:06.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.099+0000 7feb32000640 1 -- 172.21.15.102:0/2836123532 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb3c0cdba0 con 0x7feb3c0add30 2024-03-23T03:50:06.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.100+0000 7feb32000640 1 --2- 172.21.15.102:0/2836123532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3c0add30 0x7feb3c0ced10 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7feb3c15f370 tx=0x7feb2402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845e484e82436583 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:06.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.100+0000 7feb3a000640 1 -- 172.21.15.102:0/2836123532 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24031870 con 0x7feb3c0add30 2024-03-23T03:50:06.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.101+0000 7feb3a000640 1 -- 172.21.15.102:0/2836123532 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb240319d0 con 0x7feb3c0add30 2024-03-23T03:50:06.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.101+0000 7feb3a000640 1 -- 172.21.15.102:0/2836123532 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb24031d60 con 0x7feb3c0add30 2024-03-23T03:50:06.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.101+0000 7feb41a00640 1 -- 172.21.15.102:0/2836123532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3c0add30 msgr2=0x7feb3c0ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:06.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.101+0000 7feb41a00640 1 --2- 172.21.15.102:0/2836123532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3c0add30 0x7feb3c0ced10 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7feb3c15f370 tx=0x7feb2402b910 comp rx=0 tx=0).stop 2024-03-23T03:50:06.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.101+0000 7feb41a00640 1 -- 172.21.15.102:0/2836123532 shutdown_connections 2024-03-23T03:50:06.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.101+0000 7feb41a00640 1 --2- 172.21.15.102:0/2836123532 >> v2:172.21.15.173:3300/0 conn(0x7feb3c06ce90 0x7feb3c06d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:06.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.101+0000 7feb41a00640 1 --2- 172.21.15.102:0/2836123532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3c0add30 0x7feb3c0ced10 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:06.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.101+0000 7feb41a00640 1 -- 172.21.15.102:0/2836123532 >> 172.21.15.102:0/2836123532 conn(0x7feb3c06a0b0 msgr2=0x7feb3c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:06.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.101+0000 7feb41a00640 1 -- 172.21.15.102:0/2836123532 shutdown_connections 2024-03-23T03:50:06.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.101+0000 7feb41a00640 1 -- 172.21.15.102:0/2836123532 wait complete. 2024-03-23T03:50:06.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.103+0000 7feb41a00640 1 Processor -- start 2024-03-23T03:50:06.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.104+0000 7feb41a00640 1 -- start start 2024-03-23T03:50:06.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.105+0000 7feb41a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb3c06ce90 0x7feb3c1531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:06.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.105+0000 7feb41a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feb3c0add30 0x7feb3c158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:06.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.105+0000 7feb41a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3c0ce930 0x7feb3c15df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:06.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.105+0000 7feb41a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c159930 con 0x7feb3c0ce930 2024-03-23T03:50:06.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.105+0000 7feb41a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c159ef0 con 0x7feb3c06ce90 2024-03-23T03:50:06.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.105+0000 7feb41a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb3c15a9d0 con 0x7feb3c0add30 2024-03-23T03:50:06.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.105+0000 7feb3b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3c0ce930 0x7feb3c15df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:06.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.105+0000 7feb3b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3c0ce930 0x7feb3c15df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59812/0 (socket says 172.21.15.102:59812) 2024-03-23T03:50:06.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.105+0000 7feb3b400640 1 -- 172.21.15.102:0/2393395799 learned_addr learned my addr 172.21.15.102:0/2393395799 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:06.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.105+0000 7feb32000640 1 --2- 172.21.15.102:0/2393395799 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feb3c0add30 0x7feb3c158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:06.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.105+0000 7feb3aa00640 1 --2- 172.21.15.102:0/2393395799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb3c06ce90 0x7feb3c1531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:06.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.105+0000 7feb3b400640 1 -- 172.21.15.102:0/2393395799 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feb3c0add30 msgr2=0x7feb3c158a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.105+0000 7feb3b400640 1 --2- 172.21.15.102:0/2393395799 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feb3c0add30 0x7feb3c158a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.105+0000 7feb3b400640 1 -- 172.21.15.102:0/2393395799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb3c06ce90 msgr2=0x7feb3c1531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.105+0000 7feb3b400640 1 --2- 172.21.15.102:0/2393395799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb3c06ce90 0x7feb3c1531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.105+0000 7feb3b400640 1 -- 172.21.15.102:0/2393395799 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb240094d0 con 0x7feb3c0ce930 2024-03-23T03:50:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.106+0000 7feb3b400640 1 --2- 172.21.15.102:0/2393395799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3c0ce930 0x7feb3c15df90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7feb3400e930 tx=0x7feb3400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.106+0000 7feb33400640 1 -- 172.21.15.102:0/2393395799 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb34009690 con 0x7feb3c0ce930 2024-03-23T03:50:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.106+0000 7feb33400640 1 -- 172.21.15.102:0/2393395799 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb3400c980 con 0x7feb3c0ce930 2024-03-23T03:50:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.106+0000 7feb33400640 1 -- 172.21.15.102:0/2393395799 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3400cd30 con 0x7feb3c0ce930 2024-03-23T03:50:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.106+0000 7feb41a00640 1 -- 172.21.15.102:0/2393395799 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb3c15acb0 con 0x7feb3c0ce930 2024-03-23T03:50:06.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.106+0000 7feb41a00640 1 -- 172.21.15.102:0/2393395799 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb3c158f50 con 0x7feb3c0ce930 2024-03-23T03:50:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.107+0000 7feb41a00640 1 -- 172.21.15.102:0/2393395799 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb3c15a030 con 0x7feb3c0ce930 2024-03-23T03:50:06.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.108+0000 7feb33400640 1 -- 172.21.15.102:0/2393395799 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7feb34013470 con 0x7feb3c0ce930 2024-03-23T03:50:06.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.109+0000 7feb33400640 1 --2- 172.21.15.102:0/2393395799 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7feb1806ac00 0x7feb1806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:06.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.109+0000 7feb33400640 1 -- 172.21.15.102:0/2393395799 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5177+0+0 (secure 0 0 0) 0x7feb34089470 con 0x7feb3c0ce930 2024-03-23T03:50:06.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.109+0000 7feb3aa00640 1 --2- 172.21.15.102:0/2393395799 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7feb1806ac00 0x7feb1806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:06.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.110+0000 7feb3aa00640 1 --2- 172.21.15.102:0/2393395799 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7feb1806ac00 0x7feb1806d0c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7feb2c009f20 tx=0x7feb2c0094d0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:06.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.114+0000 7feb33400640 1 -- 172.21.15.102:0/2393395799 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7feb3c15a030 con 0x7feb3c0ce930 2024-03-23T03:50:06.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.294+0000 7feb41a00640 1 -- 172.21.15.102:0/2393395799 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7feb3c15a030 con 0x7feb3c0ce930 2024-03-23T03:50:06.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.447+0000 7feb33400640 1 -- 172.21.15.102:0/2393395799 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) v1 ==== 138+0+0 (secure 0 0 0) 0x7feb3c15a030 con 0x7feb3c0ce930 2024-03-23T03:50:06.448 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' created 2024-03-23T03:50:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.451+0000 7feb41a00640 1 -- 172.21.15.102:0/2393395799 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7feb1806ac00 msgr2=0x7feb1806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.451+0000 7feb41a00640 1 --2- 172.21.15.102:0/2393395799 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7feb1806ac00 0x7feb1806d0c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7feb2c009f20 tx=0x7feb2c0094d0 comp rx=0 tx=0).stop 2024-03-23T03:50:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.451+0000 7feb41a00640 1 -- 172.21.15.102:0/2393395799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3c0ce930 msgr2=0x7feb3c15df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.451+0000 7feb41a00640 1 --2- 172.21.15.102:0/2393395799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3c0ce930 0x7feb3c15df90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7feb3400e930 tx=0x7feb3400edf0 comp rx=0 tx=0).stop 2024-03-23T03:50:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.452+0000 7feb41a00640 1 -- 172.21.15.102:0/2393395799 shutdown_connections 2024-03-23T03:50:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.452+0000 7feb41a00640 1 --2- 172.21.15.102:0/2393395799 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7feb1806ac00 0x7feb1806d0c0 secure :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7feb2c009f20 tx=0x7feb2c0094d0 comp rx=0 tx=0).stop 2024-03-23T03:50:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.452+0000 7feb41a00640 1 --2- 172.21.15.102:0/2393395799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb3c0ce930 0x7feb3c15df90 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.452+0000 7feb41a00640 1 --2- 172.21.15.102:0/2393395799 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7feb3c0add30 0x7feb3c158a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.452+0000 7feb41a00640 1 --2- 172.21.15.102:0/2393395799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feb3c06ce90 0x7feb3c1531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.452+0000 7feb41a00640 1 -- 172.21.15.102:0/2393395799 >> 172.21.15.102:0/2393395799 conn(0x7feb3c06a0b0 msgr2=0x7feb3c15cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.453+0000 7feb41a00640 1 -- 172.21.15.102:0/2393395799 shutdown_connections 2024-03-23T03:50:06.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.453+0000 7feb41a00640 1 -- 172.21.15.102:0/2393395799 wait complete. 2024-03-23T03:50:06.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T03:50:06.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.673+0000 7f1b89800640 1 Processor -- start 2024-03-23T03:50:06.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.673+0000 7f1b89800640 1 -- start start 2024-03-23T03:50:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.675+0000 7f1b89800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1b840cd740 0x7f1b840b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.675+0000 7f1b89800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b840d0d40 0x7f1b840cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.675+0000 7f1b89800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b840d20c0 con 0x7f1b8406b0a0 2024-03-23T03:50:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.675+0000 7f1b89800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b840ce060 con 0x7f1b840cd740 2024-03-23T03:50:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.675+0000 7f1b89800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840d14b0 con 0x7f1b840d0d40 2024-03-23T03:50:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.675+0000 7f1b82a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1b840cd740 0x7f1b840b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.675+0000 7f1b82a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1b840cd740 0x7f1b840b84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59820/0 (socket says 172.21.15.102:59820) 2024-03-23T03:50:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.675+0000 7f1b82a00640 1 -- 172.21.15.102:0/3446421106 learned_addr learned my addr 172.21.15.102:0/3446421106 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.675+0000 7f1b82a00640 1 -- 172.21.15.102:0/3446421106 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b840d0d40 msgr2=0x7f1b840cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.675+0000 7f1b82000640 1 --2- 172.21.15.102:0/3446421106 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b840d0d40 0x7f1b840cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.675+0000 7f1b82a00640 1 --2- 172.21.15.102:0/3446421106 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b840d0d40 0x7f1b840cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.675+0000 7f1b82a00640 1 -- 172.21.15.102:0/3446421106 >> v1:172.21.15.102:6789/0 conn(0x7f1b8406b0a0 legacy=0x7f1b8406b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:06.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.675+0000 7f1b81600640 1 -- 172.21.15.102:0/3446421106 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584249855 0 0) 0x7f1b840d20c0 con 0x7f1b8406b0a0 2024-03-23T03:50:06.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.676+0000 7f1b82a00640 1 -- 172.21.15.102:0/3446421106 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b840d1790 con 0x7f1b840cd740 2024-03-23T03:50:06.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.676+0000 7f1b82000640 1 --2- 172.21.15.102:0/3446421106 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b840d0d40 0x7f1b840cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:06.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.677+0000 7f1b82a00640 1 --2- 172.21.15.102:0/3446421106 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b840cd740 0x7f1b840b84b0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009870 tx=0x7f1b7402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b446287682ff1bdc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:06.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.677+0000 7f1b81600640 1 -- 172.21.15.102:0/3446421106 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b74030840 con 0x7f1b840cd740 2024-03-23T03:50:06.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.677+0000 7f1b81600640 1 -- 172.21.15.102:0/3446421106 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b740309a0 con 0x7f1b840cd740 2024-03-23T03:50:06.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.677+0000 7f1b81600640 1 -- 172.21.15.102:0/3446421106 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b74030d30 con 0x7f1b840cd740 2024-03-23T03:50:06.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.677+0000 7f1b89800640 1 -- 172.21.15.102:0/3446421106 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b840cd740 msgr2=0x7f1b840b84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:06.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.677+0000 7f1b89800640 1 --2- 172.21.15.102:0/3446421106 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b840cd740 0x7f1b840b84b0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f1b74009870 tx=0x7f1b7402b980 comp rx=0 tx=0).stop 2024-03-23T03:50:06.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.677+0000 7f1b89800640 1 -- 172.21.15.102:0/3446421106 shutdown_connections 2024-03-23T03:50:06.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.677+0000 7f1b89800640 1 --2- 172.21.15.102:0/3446421106 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b840d0d40 0x7f1b840cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:06.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.677+0000 7f1b89800640 1 --2- 172.21.15.102:0/3446421106 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b840cd740 0x7f1b840b84b0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:06.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.678+0000 7f1b89800640 1 -- 172.21.15.102:0/3446421106 >> 172.21.15.102:0/3446421106 conn(0x7f1b840cf5d0 msgr2=0x7f1b840cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:06.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.678+0000 7f1b89800640 1 -- 172.21.15.102:0/3446421106 shutdown_connections 2024-03-23T03:50:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.678+0000 7f1b89800640 1 -- 172.21.15.102:0/3446421106 wait complete. 2024-03-23T03:50:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.679+0000 7f1b89800640 1 Processor -- start 2024-03-23T03:50:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.679+0000 7f1b89800640 1 -- start start 2024-03-23T03:50:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.680+0000 7f1b89800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b8406b0a0 0x7f1b840ae6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.680+0000 7f1b89800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b840cd740 0x7f1b8414da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.680+0000 7f1b89800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b840d0d40 0x7f1b840cba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.680+0000 7f1b89800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840b0980 con 0x7f1b840cd740 2024-03-23T03:50:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.680+0000 7f1b89800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840aafb0 con 0x7f1b840d0d40 2024-03-23T03:50:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.680+0000 7f1b89800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b840ab290 con 0x7f1b8406b0a0 2024-03-23T03:50:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.680+0000 7f1b82000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b840cd740 0x7f1b8414da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.680+0000 7f1b82000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b840cd740 0x7f1b8414da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59822/0 (socket says 172.21.15.102:59822) 2024-03-23T03:50:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.681+0000 7f1b82000640 1 -- 172.21.15.102:0/1996420006 learned_addr learned my addr 172.21.15.102:0/1996420006 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.681+0000 7f1b82000640 1 -- 172.21.15.102:0/1996420006 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b8406b0a0 msgr2=0x7f1b840ae6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.681+0000 7f1b82000640 1 --2- 172.21.15.102:0/1996420006 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b8406b0a0 0x7f1b840ae6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.681+0000 7f1b82000640 1 -- 172.21.15.102:0/1996420006 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b840d0d40 msgr2=0x7f1b840cba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.681+0000 7f1b82000640 1 --2- 172.21.15.102:0/1996420006 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b840d0d40 0x7f1b840cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.681+0000 7f1b82000640 1 -- 172.21.15.102:0/1996420006 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b740094d0 con 0x7f1b840cd740 2024-03-23T03:50:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.681+0000 7f1b82000640 1 --2- 172.21.15.102:0/1996420006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b840cd740 0x7f1b8414da40 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000c7f0 tx=0x7f1b7000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.681+0000 7f1b7aa00640 1 -- 172.21.15.102:0/1996420006 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b70007a30 con 0x7f1b840cd740 2024-03-23T03:50:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.681+0000 7f1b7aa00640 1 -- 172.21.15.102:0/1996420006 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b70007b90 con 0x7f1b840cd740 2024-03-23T03:50:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.681+0000 7f1b7aa00640 1 -- 172.21.15.102:0/1996420006 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7000ce40 con 0x7f1b840cd740 2024-03-23T03:50:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.681+0000 7f1b89800640 1 -- 172.21.15.102:0/1996420006 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b840b02c0 con 0x7f1b840cd740 2024-03-23T03:50:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.682+0000 7f1b89800640 1 -- 172.21.15.102:0/1996420006 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b840aec10 con 0x7f1b840cd740 2024-03-23T03:50:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.682+0000 7f1b89800640 1 -- 172.21.15.102:0/1996420006 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b840b0120 con 0x7f1b840cd740 2024-03-23T03:50:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.684+0000 7f1b7aa00640 1 -- 172.21.15.102:0/1996420006 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1b70024480 con 0x7f1b840cd740 2024-03-23T03:50:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.685+0000 7f1b7aa00640 1 --2- 172.21.15.102:0/1996420006 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b6406ac00 0x7f1b6406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.685+0000 7f1b7aa00640 1 -- 172.21.15.102:0/1996420006 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f1b70089740 con 0x7f1b840cd740 2024-03-23T03:50:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.685+0000 7f1b82a00640 1 --2- 172.21.15.102:0/1996420006 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b6406ac00 0x7f1b6406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:06.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.686+0000 7f1b82a00640 1 --2- 172.21.15.102:0/1996420006 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b6406ac00 0x7f1b6406d0c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1b74036000 tx=0x7f1b7403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:06.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.689+0000 7f1b7aa00640 1 -- 172.21.15.102:0/1996420006 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1b700549a0 con 0x7f1b840cd740 2024-03-23T03:50:06.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:06.870+0000 7f1b89800640 1 -- 172.21.15.102:0/1996420006 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1b840b0120 con 0x7f1b840cd740 2024-03-23T03:50:07.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.460+0000 7f1b7aa00640 1 -- 172.21.15.102:0/1996420006 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v79) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1b840b0120 con 0x7f1b840cd740 2024-03-23T03:50:07.462 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' created 2024-03-23T03:50:07.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.466+0000 7f1b89800640 1 -- 172.21.15.102:0/1996420006 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b6406ac00 msgr2=0x7f1b6406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:07.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.466+0000 7f1b89800640 1 --2- 172.21.15.102:0/1996420006 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b6406ac00 0x7f1b6406d0c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1b74036000 tx=0x7f1b7403b040 comp rx=0 tx=0).stop 2024-03-23T03:50:07.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.466+0000 7f1b89800640 1 -- 172.21.15.102:0/1996420006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b840cd740 msgr2=0x7f1b8414da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:07.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.466+0000 7f1b89800640 1 --2- 172.21.15.102:0/1996420006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b840cd740 0x7f1b8414da40 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f1b7000c7f0 tx=0x7f1b7000cbf0 comp rx=0 tx=0).stop 2024-03-23T03:50:07.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.467+0000 7f1b89800640 1 -- 172.21.15.102:0/1996420006 shutdown_connections 2024-03-23T03:50:07.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.467+0000 7f1b89800640 1 --2- 172.21.15.102:0/1996420006 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b6406ac00 0x7f1b6406d0c0 secure :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1b74036000 tx=0x7f1b7403b040 comp rx=0 tx=0).stop 2024-03-23T03:50:07.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.467+0000 7f1b89800640 1 --2- 172.21.15.102:0/1996420006 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b840d0d40 0x7f1b840cba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:07.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.467+0000 7f1b89800640 1 --2- 172.21.15.102:0/1996420006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b840cd740 0x7f1b8414da40 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:07.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.467+0000 7f1b89800640 1 --2- 172.21.15.102:0/1996420006 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b8406b0a0 0x7f1b840ae6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:07.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.467+0000 7f1b89800640 1 -- 172.21.15.102:0/1996420006 >> 172.21.15.102:0/1996420006 conn(0x7f1b840cf5d0 msgr2=0x7f1b8406b530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:07.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.468+0000 7f1b89800640 1 -- 172.21.15.102:0/1996420006 shutdown_connections 2024-03-23T03:50:07.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.469+0000 7f1b89800640 1 -- 172.21.15.102:0/1996420006 wait complete. 2024-03-23T03:50:07.501 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T03:50:07.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.685+0000 7f6e66c00640 1 Processor -- start 2024-03-23T03:50:07.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.685+0000 7f6e66c00640 1 -- start start 2024-03-23T03:50:07.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.687+0000 7f6e66c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e600cb420 0x7f6e6015dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:07.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.687+0000 7f6e66c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6e600b12d0 0x7f6e60160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:07.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.687+0000 7f6e66c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e60160530 con 0x7f6e6015c9e0 2024-03-23T03:50:07.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.687+0000 7f6e66c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e60160af0 con 0x7f6e600b12d0 2024-03-23T03:50:07.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.687+0000 7f6e66c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e601614c0 con 0x7f6e600cb420 2024-03-23T03:50:07.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.687+0000 7f6e66200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6e600b12d0 0x7f6e60160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:07.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.687+0000 7f6e66200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6e600b12d0 0x7f6e60160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59834/0 (socket says 172.21.15.102:59834) 2024-03-23T03:50:07.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.687+0000 7f6e66200640 1 -- 172.21.15.102:0/1783537379 learned_addr learned my addr 172.21.15.102:0/1783537379 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:07.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.688+0000 7f6e64e00640 1 --2- 172.21.15.102:0/1783537379 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e600cb420 0x7f6e6015dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:07.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.688+0000 7f6e66200640 1 -- 172.21.15.102:0/1783537379 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e600cb420 msgr2=0x7f6e6015dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:07.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.688+0000 7f6e66200640 1 --2- 172.21.15.102:0/1783537379 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e600cb420 0x7f6e6015dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:07.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.688+0000 7f6e66200640 1 -- 172.21.15.102:0/1783537379 >> v1:172.21.15.173:6789/0 conn(0x7f6e6015c9e0 legacy=0x7f6e600cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:07.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.688+0000 7f6e66200640 1 -- 172.21.15.102:0/1783537379 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e6015cdc0 con 0x7f6e600b12d0 2024-03-23T03:50:07.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.689+0000 7f6e66200640 1 --2- 172.21.15.102:0/1783537379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e600b12d0 0x7f6e60160f80 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6e58009e50 tx=0x7f6e5802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de31911b9798374 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:07.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.689+0000 7f6e5fe00640 1 -- 172.21.15.102:0/1783537379 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5802cb00 con 0x7f6e600b12d0 2024-03-23T03:50:07.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.690+0000 7f6e5fe00640 1 -- 172.21.15.102:0/1783537379 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5802cc60 con 0x7f6e600b12d0 2024-03-23T03:50:07.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.690+0000 7f6e5fe00640 1 -- 172.21.15.102:0/1783537379 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e580343f0 con 0x7f6e600b12d0 2024-03-23T03:50:07.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.690+0000 7f6e66c00640 1 -- 172.21.15.102:0/1783537379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e600b12d0 msgr2=0x7f6e60160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:07.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.690+0000 7f6e66c00640 1 --2- 172.21.15.102:0/1783537379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e600b12d0 0x7f6e60160f80 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f6e58009e50 tx=0x7f6e5802bf60 comp rx=0 tx=0).stop 2024-03-23T03:50:07.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.690+0000 7f6e66c00640 1 -- 172.21.15.102:0/1783537379 shutdown_connections 2024-03-23T03:50:07.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.690+0000 7f6e66c00640 1 --2- 172.21.15.102:0/1783537379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e600b12d0 0x7f6e60160f80 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:07.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.690+0000 7f6e66c00640 1 --2- 172.21.15.102:0/1783537379 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e600cb420 0x7f6e6015dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:07.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.690+0000 7f6e66c00640 1 -- 172.21.15.102:0/1783537379 >> 172.21.15.102:0/1783537379 conn(0x7f6e600b01b0 msgr2=0x7f6e600b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:07.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.690+0000 7f6e66c00640 1 -- 172.21.15.102:0/1783537379 shutdown_connections 2024-03-23T03:50:07.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.690+0000 7f6e66c00640 1 -- 172.21.15.102:0/1783537379 wait complete. 2024-03-23T03:50:07.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.692+0000 7f6e66c00640 1 Processor -- start 2024-03-23T03:50:07.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.692+0000 7f6e66c00640 1 -- start start 2024-03-23T03:50:07.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.693+0000 7f6e66c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e600b12d0 0x7f6e6006d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:07.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.693+0000 7f6e66c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e600cb420 0x7f6e60097580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:07.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.693+0000 7f6e66c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e6015c9e0 0x7f6e60097ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:07.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.693+0000 7f6e66c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6006fee0 con 0x7f6e600b12d0 2024-03-23T03:50:07.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.693+0000 7f6e66c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e601578d0 con 0x7f6e600cb420 2024-03-23T03:50:07.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.693+0000 7f6e66c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e60157030 con 0x7f6e6015c9e0 2024-03-23T03:50:07.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.693+0000 7f6e65800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e600b12d0 0x7f6e6006d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:07.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.694+0000 7f6e66200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e6015c9e0 0x7f6e60097ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:07.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.694+0000 7f6e65800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e600b12d0 0x7f6e6006d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59842/0 (socket says 172.21.15.102:59842) 2024-03-23T03:50:07.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.694+0000 7f6e65800640 1 -- 172.21.15.102:0/1228575950 learned_addr learned my addr 172.21.15.102:0/1228575950 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:07.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.694+0000 7f6e64e00640 1 --2- 172.21.15.102:0/1228575950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e600cb420 0x7f6e60097580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:07.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.694+0000 7f6e65800640 1 -- 172.21.15.102:0/1228575950 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e6015c9e0 msgr2=0x7f6e60097ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:07.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.694+0000 7f6e65800640 1 --2- 172.21.15.102:0/1228575950 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e6015c9e0 0x7f6e60097ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:07.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.694+0000 7f6e65800640 1 -- 172.21.15.102:0/1228575950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e600cb420 msgr2=0x7f6e60097580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:07.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.694+0000 7f6e65800640 1 --2- 172.21.15.102:0/1228575950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e600cb420 0x7f6e60097580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:07.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.694+0000 7f6e65800640 1 -- 172.21.15.102:0/1228575950 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e58009b30 con 0x7f6e600b12d0 2024-03-23T03:50:07.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.694+0000 7f6e65800640 1 --2- 172.21.15.102:0/1228575950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e600b12d0 0x7f6e6006d430 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400c450 tx=0x7f6e5400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:07.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.695+0000 7f6e5d600640 1 -- 172.21.15.102:0/1228575950 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e540052d0 con 0x7f6e600b12d0 2024-03-23T03:50:07.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.695+0000 7f6e5d600640 1 -- 172.21.15.102:0/1228575950 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e54005430 con 0x7f6e600b12d0 2024-03-23T03:50:07.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.695+0000 7f6e66c00640 1 -- 172.21.15.102:0/1228575950 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e60157310 con 0x7f6e600b12d0 2024-03-23T03:50:07.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.695+0000 7f6e5d600640 1 -- 172.21.15.102:0/1228575950 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54012870 con 0x7f6e600b12d0 2024-03-23T03:50:07.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.695+0000 7f6e66c00640 1 -- 172.21.15.102:0/1228575950 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e600ac140 con 0x7f6e600b12d0 2024-03-23T03:50:07.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.696+0000 7f6e66c00640 1 -- 172.21.15.102:0/1228575950 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e600abfa0 con 0x7f6e600b12d0 2024-03-23T03:50:07.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.698+0000 7f6e5d600640 1 -- 172.21.15.102:0/1228575950 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6e540129d0 con 0x7f6e600b12d0 2024-03-23T03:50:07.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.699+0000 7f6e5d600640 1 --2- 172.21.15.102:0/1228575950 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6e4406ac00 0x7f6e4406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:07.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.699+0000 7f6e5d600640 1 -- 172.21.15.102:0/1228575950 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f6e540897e0 con 0x7f6e600b12d0 2024-03-23T03:50:07.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.699+0000 7f6e64e00640 1 --2- 172.21.15.102:0/1228575950 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6e4406ac00 0x7f6e4406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:07.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.700+0000 7f6e64e00640 1 --2- 172.21.15.102:0/1228575950 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6e4406ac00 0x7f6e4406d0c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6e60156290 tx=0x7f6e50008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:07.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.702+0000 7f6e5d600640 1 -- 172.21.15.102:0/1228575950 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6e54054950 con 0x7f6e600b12d0 2024-03-23T03:50:07.935 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.935+0000 7f6e66c00640 1 -- 172.21.15.102:0/1228575950 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f6e600abfa0 con 0x7f6e600b12d0 2024-03-23T03:50:07.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.944+0000 7f6e5d600640 1 -- 172.21.15.102:0/1228575950 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v37) v1 ==== 174+0+0 (secure 0 0 0) 0x7f6e600abfa0 con 0x7f6e600b12d0 2024-03-23T03:50:07.946 INFO:teuthology.orchestra.run.smithi102.stderr:new fs with metadata pool 11 and data pool 12 2024-03-23T03:50:07.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.949+0000 7f6e66c00640 1 -- 172.21.15.102:0/1228575950 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6e4406ac00 msgr2=0x7f6e4406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:07.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.949+0000 7f6e66c00640 1 --2- 172.21.15.102:0/1228575950 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6e4406ac00 0x7f6e4406d0c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6e60156290 tx=0x7f6e50008040 comp rx=0 tx=0).stop 2024-03-23T03:50:07.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.949+0000 7f6e66c00640 1 -- 172.21.15.102:0/1228575950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e600b12d0 msgr2=0x7f6e6006d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:07.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.949+0000 7f6e66c00640 1 --2- 172.21.15.102:0/1228575950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e600b12d0 0x7f6e6006d430 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400c450 tx=0x7f6e5400c910 comp rx=0 tx=0).stop 2024-03-23T03:50:07.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.949+0000 7f6e66c00640 1 -- 172.21.15.102:0/1228575950 shutdown_connections 2024-03-23T03:50:07.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.949+0000 7f6e66c00640 1 --2- 172.21.15.102:0/1228575950 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6e4406ac00 0x7f6e4406d0c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:07.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.949+0000 7f6e66c00640 1 --2- 172.21.15.102:0/1228575950 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e6015c9e0 0x7f6e60097ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:07.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.949+0000 7f6e66c00640 1 --2- 172.21.15.102:0/1228575950 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e600cb420 0x7f6e60097580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:07.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.949+0000 7f6e66c00640 1 --2- 172.21.15.102:0/1228575950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e600b12d0 0x7f6e6006d430 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:07.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.949+0000 7f6e66c00640 1 -- 172.21.15.102:0/1228575950 >> 172.21.15.102:0/1228575950 conn(0x7f6e600b01b0 msgr2=0x7f6e6015b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:07.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.949+0000 7f6e66c00640 1 -- 172.21.15.102:0/1228575950 shutdown_connections 2024-03-23T03:50:07.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:07.949+0000 7f6e66c00640 1 -- 172.21.15.102:0/1228575950 wait complete. 2024-03-23T03:50:07.990 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:50:08.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.164+0000 7f94f2e00640 1 Processor -- start 2024-03-23T03:50:08.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.164+0000 7f94f2e00640 1 -- start start 2024-03-23T03:50:08.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.166+0000 7f94f2e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f94ec0b37f0 0x7f94ec0cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:08.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.166+0000 7f94f2e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec06da70 con 0x7f94ec0ccec0 2024-03-23T03:50:08.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.166+0000 7f94f2e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94ec0b1d00 con 0x7f94ec14aa40 2024-03-23T03:50:08.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.166+0000 7f94f2e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94ec0b1fe0 con 0x7f94ec0b37f0 2024-03-23T03:50:08.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.166+0000 7f94f0e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f94ec0ccec0 0x7f94ec06c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38642/0 (socket says 172.21.15.102:38642) 2024-03-23T03:50:08.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.166+0000 7f94f0e00640 1 -- 172.21.15.102:0/3031956776 learned_addr learned my addr 172.21.15.102:0/3031956776 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:08.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.167+0000 7f94e3400640 1 --2- 172.21.15.102:0/3031956776 >> v2:172.21.15.173:3300/0 conn(0x7f94ec0b37f0 0x7f94ec0cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.167+0000 7f94eb400640 1 -- 172.21.15.102:0/3031956776 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986769679 0 0) 0x7f94ec06da70 con 0x7f94ec0ccec0 2024-03-23T03:50:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.167+0000 7f94eb400640 1 -- 172.21.15.102:0/3031956776 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94d0003520 con 0x7f94ec0ccec0 2024-03-23T03:50:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.167+0000 7f94eb400640 1 -- 172.21.15.102:0/3031956776 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1314428918 0 0) 0x7f94d0003520 con 0x7f94ec0ccec0 2024-03-23T03:50:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.167+0000 7f94eb400640 1 -- 172.21.15.102:0/3031956776 >> v2:172.21.15.173:3300/0 conn(0x7f94ec0b37f0 msgr2=0x7f94ec0cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.167+0000 7f94eb400640 1 --2- 172.21.15.102:0/3031956776 >> v2:172.21.15.173:3300/0 conn(0x7f94ec0b37f0 0x7f94ec0cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.167+0000 7f94eb400640 1 -- 172.21.15.102:0/3031956776 >> v1:172.21.15.173:6789/0 conn(0x7f94ec14aa40 legacy=0x7f94ec0cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.167+0000 7f94eb400640 1 -- 172.21.15.102:0/3031956776 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ec0b22c0 con 0x7f94ec0ccec0 2024-03-23T03:50:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.167+0000 7f94eb400640 1 -- 172.21.15.102:0/3031956776 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f94e4001a10 con 0x7f94ec0ccec0 2024-03-23T03:50:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.167+0000 7f94eb400640 1 -- 172.21.15.102:0/3031956776 >> v1:172.21.15.102:6789/0 conn(0x7f94ec0ccec0 legacy=0x7f94ec06c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.168+0000 7f94eb400640 1 --2- 172.21.15.102:0/3031956776 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94d0003970 0x7f94d0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.168+0000 7f94eb400640 1 --2- 172.21.15.102:0/3031956776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94d00047c0 0x7f94d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.168+0000 7f94eb400640 1 --2- 172.21.15.102:0/3031956776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94d0005180 0x7f94d000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.168+0000 7f94eb400640 1 -- 172.21.15.102:0/3031956776 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec0b1fe0 con 0x7f94d00047c0 2024-03-23T03:50:08.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.168+0000 7f94eb400640 1 -- 172.21.15.102:0/3031956776 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d000c020 con 0x7f94d0005180 2024-03-23T03:50:08.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.168+0000 7f94eb400640 1 -- 172.21.15.102:0/3031956776 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d000c1e0 con 0x7f94d0003970 2024-03-23T03:50:08.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.168+0000 7f94e3400640 1 --2- 172.21.15.102:0/3031956776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94d00047c0 0x7f94d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.168+0000 7f94e3400640 1 -- 172.21.15.102:0/3031956776 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94d0003970 msgr2=0x7f94d0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.168+0000 7f94ebe00640 1 --2- 172.21.15.102:0/3031956776 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94d0003970 0x7f94d0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.168+0000 7f94f0e00640 1 --2- 172.21.15.102:0/3031956776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94d0005180 0x7f94d000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:08.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.168+0000 7f94e3400640 1 --2- 172.21.15.102:0/3031956776 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94d0003970 0x7f94d0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.168+0000 7f94e3400640 1 -- 172.21.15.102:0/3031956776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94d0005180 msgr2=0x7f94d000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.168+0000 7f94e3400640 1 --2- 172.21.15.102:0/3031956776 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94d0005180 0x7f94d000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.168+0000 7f94e3400640 1 -- 172.21.15.102:0/3031956776 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d0003520 con 0x7f94d00047c0 2024-03-23T03:50:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.168+0000 7f94f0e00640 1 -- 172.21.15.102:0/3031956776 reap_dead start 2024-03-23T03:50:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.168+0000 7f94ebe00640 1 --2- 172.21.15.102:0/3031956776 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94d0003970 0x7f94d0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.169+0000 7f94e3400640 1 --2- 172.21.15.102:0/3031956776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94d00047c0 0x7f94d0004c40 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f94d4009a90 tx=0x7f94d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3736718c51d8740c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.170+0000 7f94eb400640 1 -- 172.21.15.102:0/3031956776 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d402c680 con 0x7f94d00047c0 2024-03-23T03:50:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.170+0000 7f94eb400640 1 -- 172.21.15.102:0/3031956776 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d402c7e0 con 0x7f94d00047c0 2024-03-23T03:50:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.170+0000 7f94eb400640 1 -- 172.21.15.102:0/3031956776 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d402cb70 con 0x7f94d00047c0 2024-03-23T03:50:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.170+0000 7f94f2e00640 1 -- 172.21.15.102:0/3031956776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94d00047c0 msgr2=0x7f94d0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.170+0000 7f94f2e00640 1 --2- 172.21.15.102:0/3031956776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94d00047c0 0x7f94d0004c40 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f94d4009a90 tx=0x7f94d00057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.170+0000 7f94f2e00640 1 -- 172.21.15.102:0/3031956776 shutdown_connections 2024-03-23T03:50:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.170+0000 7f94f2e00640 1 --2- 172.21.15.102:0/3031956776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94d00047c0 0x7f94d0004c40 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.170+0000 7f94f2e00640 1 -- 172.21.15.102:0/3031956776 >> 172.21.15.102:0/3031956776 conn(0x7f94ec06a3d0 msgr2=0x7f94ec0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:08.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.170+0000 7f94f2e00640 1 -- 172.21.15.102:0/3031956776 shutdown_connections 2024-03-23T03:50:08.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.170+0000 7f94f2e00640 1 -- 172.21.15.102:0/3031956776 wait complete. 2024-03-23T03:50:08.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.172+0000 7f94f2e00640 1 Processor -- start 2024-03-23T03:50:08.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.172+0000 7f94f2e00640 1 -- start start 2024-03-23T03:50:08.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.173+0000 7f94f2e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94d00047c0 0x7f94ec160730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:08.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.173+0000 7f94f2e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94ec163ae0 0x7f94ec142670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:08.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.173+0000 7f94f2e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94ec142bb0 0x7f94ec140380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:08.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.173+0000 7f94f2e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec1fac70 con 0x7f94d00047c0 2024-03-23T03:50:08.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.173+0000 7f94f2e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec161cd0 con 0x7f94ec142bb0 2024-03-23T03:50:08.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.173+0000 7f94f2e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94ec1613a0 con 0x7f94ec163ae0 2024-03-23T03:50:08.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.173+0000 7f94ebe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94d00047c0 0x7f94ec160730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:08.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.174+0000 7f94ebe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94d00047c0 0x7f94ec160730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59858/0 (socket says 172.21.15.102:59858) 2024-03-23T03:50:08.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.174+0000 7f94ebe00640 1 -- 172.21.15.102:0/3363070228 learned_addr learned my addr 172.21.15.102:0/3363070228 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:08.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.174+0000 7f94f0e00640 1 --2- 172.21.15.102:0/3363070228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94ec142bb0 0x7f94ec140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:08.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.174+0000 7f94e3400640 1 --2- 172.21.15.102:0/3363070228 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94ec163ae0 0x7f94ec142670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:08.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.174+0000 7f94ebe00640 1 -- 172.21.15.102:0/3363070228 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94ec163ae0 msgr2=0x7f94ec142670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:08.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.174+0000 7f94ebe00640 1 --2- 172.21.15.102:0/3363070228 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94ec163ae0 0x7f94ec142670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.174+0000 7f94ebe00640 1 -- 172.21.15.102:0/3363070228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94ec142bb0 msgr2=0x7f94ec140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:08.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.174+0000 7f94ebe00640 1 --2- 172.21.15.102:0/3363070228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94ec142bb0 0x7f94ec140380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.174+0000 7f94ebe00640 1 -- 172.21.15.102:0/3363070228 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94ec161680 con 0x7f94d00047c0 2024-03-23T03:50:08.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.174+0000 7f94ebe00640 1 --2- 172.21.15.102:0/3363070228 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94d00047c0 0x7f94ec160730 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f94dc00d4a0 tx=0x7f94dc00d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:08.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.174+0000 7f94e8c00640 1 -- 172.21.15.102:0/3363070228 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc005480 con 0x7f94d00047c0 2024-03-23T03:50:08.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.175+0000 7f94e8c00640 1 -- 172.21.15.102:0/3363070228 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94dc007030 con 0x7f94d00047c0 2024-03-23T03:50:08.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.175+0000 7f94f2e00640 1 -- 172.21.15.102:0/3363070228 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94ec161960 con 0x7f94d00047c0 2024-03-23T03:50:08.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.175+0000 7f94e8c00640 1 -- 172.21.15.102:0/3363070228 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94dc0073c0 con 0x7f94d00047c0 2024-03-23T03:50:08.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.175+0000 7f94f2e00640 1 -- 172.21.15.102:0/3363070228 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94ec160c70 con 0x7f94d00047c0 2024-03-23T03:50:08.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.176+0000 7f94f2e00640 1 -- 172.21.15.102:0/3363070228 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94ec0cc770 con 0x7f94d00047c0 2024-03-23T03:50:08.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.177+0000 7f94e8c00640 1 -- 172.21.15.102:0/3363070228 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f94dc007520 con 0x7f94d00047c0 2024-03-23T03:50:08.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.178+0000 7f94e8c00640 1 --2- 172.21.15.102:0/3363070228 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f94c806ac30 0x7f94c806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:08.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.178+0000 7f94e8c00640 1 -- 172.21.15.102:0/3363070228 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f94dc088980 con 0x7f94d00047c0 2024-03-23T03:50:08.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.178+0000 7f94e3400640 1 --2- 172.21.15.102:0/3363070228 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f94c806ac30 0x7f94c806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:08.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.179+0000 7f94e3400640 1 --2- 172.21.15.102:0/3363070228 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f94c806ac30 0x7f94c806d0f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f94d4036000 tx=0x7f94d4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:08.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.183+0000 7f94e8c00640 1 -- 172.21.15.102:0/3363070228 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f94dc053a10 con 0x7f94d00047c0 2024-03-23T03:50:08.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.364+0000 7f94f2e00640 1 -- 172.21.15.102:0/3363070228 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94ec06ac20 con 0x7f94d00047c0 2024-03-23T03:50:08.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.367+0000 7f94e8c00640 1 -- 172.21.15.102:0/3363070228 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16370 (secure 0 0 0) 0x7f94dc019090 con 0x7f94d00047c0 2024-03-23T03:50:08.368 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:08.369 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":80,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:07.936938+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:50:06.347556+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-23T03:50:06.924370+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:50:08.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.370+0000 7f94f2e00640 1 -- 172.21.15.102:0/3363070228 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f94c806ac30 msgr2=0x7f94c806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:08.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.370+0000 7f94f2e00640 1 --2- 172.21.15.102:0/3363070228 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f94c806ac30 0x7f94c806d0f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f94d4036000 tx=0x7f94d4030000 comp rx=0 tx=0).stop 2024-03-23T03:50:08.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.370+0000 7f94f2e00640 1 -- 172.21.15.102:0/3363070228 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94d00047c0 msgr2=0x7f94ec160730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:08.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.370+0000 7f94f2e00640 1 --2- 172.21.15.102:0/3363070228 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94d00047c0 0x7f94ec160730 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f94dc00d4a0 tx=0x7f94dc00d960 comp rx=0 tx=0).stop 2024-03-23T03:50:08.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.370+0000 7f94f2e00640 1 -- 172.21.15.102:0/3363070228 shutdown_connections 2024-03-23T03:50:08.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.370+0000 7f94f2e00640 1 --2- 172.21.15.102:0/3363070228 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f94c806ac30 0x7f94c806d0f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.370+0000 7f94f2e00640 1 --2- 172.21.15.102:0/3363070228 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94ec142bb0 0x7f94ec140380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.370+0000 7f94f2e00640 1 --2- 172.21.15.102:0/3363070228 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94ec163ae0 0x7f94ec142670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.370+0000 7f94f2e00640 1 --2- 172.21.15.102:0/3363070228 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94d00047c0 0x7f94ec160730 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.370+0000 7f94f2e00640 1 -- 172.21.15.102:0/3363070228 >> 172.21.15.102:0/3363070228 conn(0x7f94ec06a3d0 msgr2=0x7f94ec070070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:08.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.370+0000 7f94f2e00640 1 -- 172.21.15.102:0/3363070228 shutdown_connections 2024-03-23T03:50:08.372 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.370+0000 7f94f2e00640 1 -- 172.21.15.102:0/3363070228 wait complete. 2024-03-23T03:50:08.388 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:50:08.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.586+0000 7f6370000640 1 Processor -- start 2024-03-23T03:50:08.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.586+0000 7f6370000640 1 -- start start 2024-03-23T03:50:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.587+0000 7f6370000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f636815b560 0x7f636815b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.587+0000 7f6370000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f636815d410 0x7f63680b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.587+0000 7f6370000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f636806c6f0 con 0x7f63680b39b0 2024-03-23T03:50:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.587+0000 7f6370000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63680b4370 con 0x7f636815d410 2024-03-23T03:50:08.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.588+0000 7f6370000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f636806c9d0 con 0x7f636815b560 2024-03-23T03:50:08.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.588+0000 7f636cc00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f63680b39b0 0x7f636815e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38658/0 (socket says 172.21.15.102:38658) 2024-03-23T03:50:08.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.588+0000 7f636cc00640 1 -- 172.21.15.102:0/650993883 learned_addr learned my addr 172.21.15.102:0/650993883 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:08.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.588+0000 7f636e000640 1 --2- 172.21.15.102:0/650993883 >> v2:172.21.15.102:3300/0 conn(0x7f636815d410 0x7f63680b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:08.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.588+0000 7f636d600640 1 --2- 172.21.15.102:0/650993883 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f636815b560 0x7f636815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:08.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.589+0000 7f6367e00640 1 -- 172.21.15.102:0/650993883 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935903483 0 0) 0x7f636806c6f0 con 0x7f63680b39b0 2024-03-23T03:50:08.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.589+0000 7f6367e00640 1 -- 172.21.15.102:0/650993883 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6350003520 con 0x7f63680b39b0 2024-03-23T03:50:08.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.589+0000 7f636e000640 1 -- 172.21.15.102:0/650993883 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f636815b560 msgr2=0x7f636815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:08.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.589+0000 7f636e000640 1 --2- 172.21.15.102:0/650993883 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f636815b560 0x7f636815b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.589+0000 7f636e000640 1 -- 172.21.15.102:0/650993883 >> v1:172.21.15.102:6789/0 conn(0x7f63680b39b0 legacy=0x7f636815e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:08.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.589+0000 7f636e000640 1 -- 172.21.15.102:0/650993883 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636806ccb0 con 0x7f636815d410 2024-03-23T03:50:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.590+0000 7f636e000640 1 --2- 172.21.15.102:0/650993883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f636815d410 0x7f63680b3e30 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f6360009d90 tx=0x7f636002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c756ab6cf8de356 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.590+0000 7f6367e00640 1 -- 172.21.15.102:0/650993883 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636002cc50 con 0x7f636815d410 2024-03-23T03:50:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.590+0000 7f6367e00640 1 -- 172.21.15.102:0/650993883 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f636002cdb0 con 0x7f636815d410 2024-03-23T03:50:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.590+0000 7f6367e00640 1 -- 172.21.15.102:0/650993883 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6360034660 con 0x7f636815d410 2024-03-23T03:50:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.590+0000 7f6370000640 1 -- 172.21.15.102:0/650993883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f636815d410 msgr2=0x7f63680b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:08.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.590+0000 7f6370000640 1 --2- 172.21.15.102:0/650993883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f636815d410 0x7f63680b3e30 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f6360009d90 tx=0x7f636002bea0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.591+0000 7f6370000640 1 -- 172.21.15.102:0/650993883 shutdown_connections 2024-03-23T03:50:08.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.591+0000 7f6370000640 1 --2- 172.21.15.102:0/650993883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f636815d410 0x7f63680b3e30 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.591+0000 7f6370000640 1 --2- 172.21.15.102:0/650993883 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f636815b560 0x7f636815b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.591+0000 7f6370000640 1 -- 172.21.15.102:0/650993883 >> 172.21.15.102:0/650993883 conn(0x7f636806a3d0 msgr2=0x7f63680b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:08.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.591+0000 7f6370000640 1 -- 172.21.15.102:0/650993883 shutdown_connections 2024-03-23T03:50:08.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.591+0000 7f6370000640 1 -- 172.21.15.102:0/650993883 wait complete. 2024-03-23T03:50:08.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.592+0000 7f6370000640 1 Processor -- start 2024-03-23T03:50:08.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.592+0000 7f6370000640 1 -- start start 2024-03-23T03:50:08.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.593+0000 7f6370000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63680b39b0 0x7f63681549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:08.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.593+0000 7f6370000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f636815b560 0x7f6368151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:08.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.593+0000 7f6370000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f636815d410 0x7f63680af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:08.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.593+0000 7f6370000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6368154540 con 0x7f63680b39b0 2024-03-23T03:50:08.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.593+0000 7f6370000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6368151970 con 0x7f636815d410 2024-03-23T03:50:08.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.593+0000 7f6370000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6368153930 con 0x7f636815b560 2024-03-23T03:50:08.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.593+0000 7f636d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63680b39b0 0x7f63681549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:08.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.594+0000 7f636d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63680b39b0 0x7f63681549d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59878/0 (socket says 172.21.15.102:59878) 2024-03-23T03:50:08.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.594+0000 7f636d600640 1 -- 172.21.15.102:0/429496992 learned_addr learned my addr 172.21.15.102:0/429496992 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:08.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.594+0000 7f636cc00640 1 --2- 172.21.15.102:0/429496992 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f636815b560 0x7f6368151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:08.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.594+0000 7f636e000640 1 --2- 172.21.15.102:0/429496992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f636815d410 0x7f63680af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:08.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.594+0000 7f636d600640 1 -- 172.21.15.102:0/429496992 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f636815b560 msgr2=0x7f6368151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:08.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.594+0000 7f636d600640 1 --2- 172.21.15.102:0/429496992 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f636815b560 0x7f6368151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.594+0000 7f636d600640 1 -- 172.21.15.102:0/429496992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f636815d410 msgr2=0x7f63680af490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:08.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.594+0000 7f636d600640 1 --2- 172.21.15.102:0/429496992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f636815d410 0x7f63680af490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.594+0000 7f636d600640 1 -- 172.21.15.102:0/429496992 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6360009a70 con 0x7f63680b39b0 2024-03-23T03:50:08.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.594+0000 7f636d600640 1 --2- 172.21.15.102:0/429496992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63680b39b0 0x7f63681549d0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f635c00d830 tx=0x7f635c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:08.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.594+0000 7f6365600640 1 -- 172.21.15.102:0/429496992 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c00b6c0 con 0x7f63680b39b0 2024-03-23T03:50:08.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.595+0000 7f6365600640 1 -- 172.21.15.102:0/429496992 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f635c00b820 con 0x7f63680b39b0 2024-03-23T03:50:08.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.595+0000 7f6370000640 1 -- 172.21.15.102:0/429496992 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6368153c10 con 0x7f63680b39b0 2024-03-23T03:50:08.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.595+0000 7f6365600640 1 -- 172.21.15.102:0/429496992 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f635c00bbb0 con 0x7f63680b39b0 2024-03-23T03:50:08.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.595+0000 7f6370000640 1 -- 172.21.15.102:0/429496992 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63681541d0 con 0x7f63680b39b0 2024-03-23T03:50:08.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.596+0000 7f6370000640 1 -- 172.21.15.102:0/429496992 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f636806ac20 con 0x7f63680b39b0 2024-03-23T03:50:08.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.597+0000 7f6365600640 1 -- 172.21.15.102:0/429496992 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f635c01e460 con 0x7f63680b39b0 2024-03-23T03:50:08.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.598+0000 7f6365600640 1 --2- 172.21.15.102:0/429496992 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f634806ac00 0x7f634806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:08.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.598+0000 7f6365600640 1 -- 172.21.15.102:0/429496992 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f635c089ba0 con 0x7f63680b39b0 2024-03-23T03:50:08.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.598+0000 7f636cc00640 1 --2- 172.21.15.102:0/429496992 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f634806ac00 0x7f634806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:08.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.598+0000 7f636cc00640 1 --2- 172.21.15.102:0/429496992 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f634806ac00 0x7f634806d0c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f636812b6c0 tx=0x7f6354009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:08.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.602+0000 7f6365600640 1 -- 172.21.15.102:0/429496992 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f635c054cd0 con 0x7f63680b39b0 2024-03-23T03:50:08.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.780+0000 7f6370000640 1 -- 172.21.15.102:0/429496992 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6368073390 con 0x7f63680b39b0 2024-03-23T03:50:08.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.782+0000 7f6365600640 1 -- 172.21.15.102:0/429496992 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16370 (secure 0 0 0) 0x7f635c0582f0 con 0x7f63680b39b0 2024-03-23T03:50:08.783 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:08.783 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":80,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:07.936938+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:50:06.347556+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-23T03:50:06.924370+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:50:08.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.785+0000 7f6370000640 1 -- 172.21.15.102:0/429496992 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f634806ac00 msgr2=0x7f634806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:08.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.786+0000 7f6370000640 1 --2- 172.21.15.102:0/429496992 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f634806ac00 0x7f634806d0c0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f636812b6c0 tx=0x7f6354009040 comp rx=0 tx=0).stop 2024-03-23T03:50:08.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.786+0000 7f6370000640 1 -- 172.21.15.102:0/429496992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63680b39b0 msgr2=0x7f63681549d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:08.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.786+0000 7f6370000640 1 --2- 172.21.15.102:0/429496992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63680b39b0 0x7f63681549d0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f635c00d830 tx=0x7f635c00dcf0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.786+0000 7f6370000640 1 -- 172.21.15.102:0/429496992 shutdown_connections 2024-03-23T03:50:08.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.786+0000 7f6370000640 1 --2- 172.21.15.102:0/429496992 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f634806ac00 0x7f634806d0c0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.786+0000 7f6370000640 1 --2- 172.21.15.102:0/429496992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f636815d410 0x7f63680af490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.786+0000 7f6370000640 1 --2- 172.21.15.102:0/429496992 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f636815b560 0x7f6368151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.786+0000 7f6370000640 1 --2- 172.21.15.102:0/429496992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f63680b39b0 0x7f63681549d0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:08.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.786+0000 7f6370000640 1 -- 172.21.15.102:0/429496992 >> 172.21.15.102:0/429496992 conn(0x7f636806a3d0 msgr2=0x7f63680cc490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:08.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.786+0000 7f6370000640 1 -- 172.21.15.102:0/429496992 shutdown_connections 2024-03-23T03:50:08.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:08.786+0000 7f6370000640 1 -- 172.21.15.102:0/429496992 wait complete. 2024-03-23T03:50:08.805 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T03:50:09.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.012+0000 7f09e5200640 1 Processor -- start 2024-03-23T03:50:09.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.012+0000 7f09e5200640 1 -- start start 2024-03-23T03:50:09.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.014+0000 7f09e5200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09e0069c90 0x7f09e006a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:09.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.014+0000 7f09e5200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f09e00b26e0 0x7f09e00b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:09.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.014+0000 7f09e5200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09e006a630 con 0x7f09e015ca00 2024-03-23T03:50:09.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.014+0000 7f09e5200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09e006fc80 con 0x7f09e00b26e0 2024-03-23T03:50:09.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.014+0000 7f09e5200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e00cf310 con 0x7f09e0069c90 2024-03-23T03:50:09.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.014+0000 7f09dde00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f09e015ca00 0x7f09e006f570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38670/0 (socket says 172.21.15.102:38670) 2024-03-23T03:50:09.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.014+0000 7f09dde00640 1 -- 172.21.15.102:0/908096019 learned_addr learned my addr 172.21.15.102:0/908096019 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:09.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.014+0000 7f09de800640 1 --2- 172.21.15.102:0/908096019 >> v2:172.21.15.102:3300/0 conn(0x7f09e00b26e0 0x7f09e00b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:09.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.014+0000 7f09dd400640 1 --2- 172.21.15.102:0/908096019 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09e0069c90 0x7f09e006a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:09.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.014+0000 7f09dca00640 1 -- 172.21.15.102:0/908096019 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404388602 0 0) 0x7f09e006a630 con 0x7f09e015ca00 2024-03-23T03:50:09.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.015+0000 7f09dca00640 1 -- 172.21.15.102:0/908096019 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09c4003520 con 0x7f09e015ca00 2024-03-23T03:50:09.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.015+0000 7f09de800640 1 -- 172.21.15.102:0/908096019 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09e0069c90 msgr2=0x7f09e006a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:09.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.015+0000 7f09de800640 1 --2- 172.21.15.102:0/908096019 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09e0069c90 0x7f09e006a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:09.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.015+0000 7f09de800640 1 -- 172.21.15.102:0/908096019 >> v1:172.21.15.102:6789/0 conn(0x7f09e015ca00 legacy=0x7f09e006f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:09.016 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.015+0000 7f09de800640 1 -- 172.21.15.102:0/908096019 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e00cf5f0 con 0x7f09e00b26e0 2024-03-23T03:50:09.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.017+0000 7f09de800640 1 --2- 172.21.15.102:0/908096019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09e00b26e0 0x7f09e00b2b70 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f09d8009d90 tx=0x7f09d802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=133caa2d1a9e8d64 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:09.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.017+0000 7f09dca00640 1 -- 172.21.15.102:0/908096019 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d802cc50 con 0x7f09e00b26e0 2024-03-23T03:50:09.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.017+0000 7f09dca00640 1 -- 172.21.15.102:0/908096019 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d802cdb0 con 0x7f09e00b26e0 2024-03-23T03:50:09.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.017+0000 7f09dca00640 1 -- 172.21.15.102:0/908096019 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d8034660 con 0x7f09e00b26e0 2024-03-23T03:50:09.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.017+0000 7f09e5200640 1 -- 172.21.15.102:0/908096019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09e00b26e0 msgr2=0x7f09e00b2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:09.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.017+0000 7f09e5200640 1 --2- 172.21.15.102:0/908096019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09e00b26e0 0x7f09e00b2b70 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f09d8009d90 tx=0x7f09d802bea0 comp rx=0 tx=0).stop 2024-03-23T03:50:09.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.018+0000 7f09e5200640 1 -- 172.21.15.102:0/908096019 shutdown_connections 2024-03-23T03:50:09.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.018+0000 7f09e5200640 1 --2- 172.21.15.102:0/908096019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09e00b26e0 0x7f09e00b2b70 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:09.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.018+0000 7f09e5200640 1 --2- 172.21.15.102:0/908096019 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09e0069c90 0x7f09e006a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:09.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.018+0000 7f09e5200640 1 -- 172.21.15.102:0/908096019 >> 172.21.15.102:0/908096019 conn(0x7f09e006d440 msgr2=0x7f09e006d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:09.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.018+0000 7f09e5200640 1 -- 172.21.15.102:0/908096019 shutdown_connections 2024-03-23T03:50:09.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.018+0000 7f09e5200640 1 -- 172.21.15.102:0/908096019 wait complete. 2024-03-23T03:50:09.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.019+0000 7f09e5200640 1 Processor -- start 2024-03-23T03:50:09.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.019+0000 7f09e5200640 1 -- start start 2024-03-23T03:50:09.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.020+0000 7f09e5200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09e0069c90 0x7f09e0153390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:09.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.020+0000 7f09e5200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09e00b26e0 0x7f09e0154bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:09.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.020+0000 7f09e5200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09e015ca00 0x7f09e00cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:09.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.020+0000 7f09e5200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e00cd530 con 0x7f09e015ca00 2024-03-23T03:50:09.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.020+0000 7f09e5200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e00cc8b0 con 0x7f09e0069c90 2024-03-23T03:50:09.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.020+0000 7f09e5200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09e00ccb90 con 0x7f09e00b26e0 2024-03-23T03:50:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.021+0000 7f09de800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09e015ca00 0x7f09e00cd950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.021+0000 7f09de800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09e015ca00 0x7f09e00cd950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59906/0 (socket says 172.21.15.102:59906) 2024-03-23T03:50:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.021+0000 7f09de800640 1 -- 172.21.15.102:0/472605569 learned_addr learned my addr 172.21.15.102:0/472605569 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.021+0000 7f09dde00640 1 --2- 172.21.15.102:0/472605569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09e0069c90 0x7f09e0153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.021+0000 7f09dd400640 1 --2- 172.21.15.102:0/472605569 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09e00b26e0 0x7f09e0154bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.021+0000 7f09de800640 1 -- 172.21.15.102:0/472605569 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09e00b26e0 msgr2=0x7f09e0154bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.021+0000 7f09de800640 1 --2- 172.21.15.102:0/472605569 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09e00b26e0 0x7f09e0154bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.021+0000 7f09de800640 1 -- 172.21.15.102:0/472605569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09e0069c90 msgr2=0x7f09e0153390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.021+0000 7f09de800640 1 --2- 172.21.15.102:0/472605569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09e0069c90 0x7f09e0153390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.021+0000 7f09de800640 1 -- 172.21.15.102:0/472605569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d8009a70 con 0x7f09e015ca00 2024-03-23T03:50:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.021+0000 7f09dde00640 1 --2- 172.21.15.102:0/472605569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09e0069c90 0x7f09e0153390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:50:09.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.022+0000 7f09de800640 1 --2- 172.21.15.102:0/472605569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09e015ca00 0x7f09e00cd950 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f09d8009d90 tx=0x7f09d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.022+0000 7f09d6000640 1 -- 172.21.15.102:0/472605569 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d8040650 con 0x7f09e015ca00 2024-03-23T03:50:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.022+0000 7f09d6000640 1 -- 172.21.15.102:0/472605569 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d80407b0 con 0x7f09e015ca00 2024-03-23T03:50:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.022+0000 7f09e5200640 1 -- 172.21.15.102:0/472605569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e00cce70 con 0x7f09e015ca00 2024-03-23T03:50:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.022+0000 7f09d6000640 1 -- 172.21.15.102:0/472605569 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d8040b60 con 0x7f09e015ca00 2024-03-23T03:50:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.022+0000 7f09e5200640 1 -- 172.21.15.102:0/472605569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e01538d0 con 0x7f09e015ca00 2024-03-23T03:50:09.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.023+0000 7f09e5200640 1 -- 172.21.15.102:0/472605569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e00cd670 con 0x7f09e015ca00 2024-03-23T03:50:09.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.025+0000 7f09d6000640 1 -- 172.21.15.102:0/472605569 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f09d8040cc0 con 0x7f09e015ca00 2024-03-23T03:50:09.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.026+0000 7f09d6000640 1 --2- 172.21.15.102:0/472605569 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f09c006ac00 0x7f09c006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:09.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.026+0000 7f09d6000640 1 -- 172.21.15.102:0/472605569 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f09d80ac270 con 0x7f09e015ca00 2024-03-23T03:50:09.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.026+0000 7f09dde00640 1 --2- 172.21.15.102:0/472605569 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f09c006ac00 0x7f09c006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:09.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.027+0000 7f09dde00640 1 --2- 172.21.15.102:0/472605569 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f09c006ac00 0x7f09c006d0c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f09d00011e0 tx=0x7f09d0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:09.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.031+0000 7f09d6000640 1 -- 172.21.15.102:0/472605569 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f09e00cd670 con 0x7f09e015ca00 2024-03-23T03:50:09.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.245+0000 7f09e5200640 1 -- 172.21.15.102:0/472605569 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f09e00cd670 con 0x7f09e015ca00 2024-03-23T03:50:09.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.962+0000 7f09d6000640 1 -- 172.21.15.102:0/472605569 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v40) v1 ==== 118+0+0 (secure 0 0 0) 0x7f09e00cd670 con 0x7f09e015ca00 2024-03-23T03:50:09.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.964+0000 7f09e5200640 1 -- 172.21.15.102:0/472605569 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f09c006ac00 msgr2=0x7f09c006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:09.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.964+0000 7f09e5200640 1 --2- 172.21.15.102:0/472605569 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f09c006ac00 0x7f09c006d0c0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f09d00011e0 tx=0x7f09d0009040 comp rx=0 tx=0).stop 2024-03-23T03:50:09.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.964+0000 7f09e5200640 1 -- 172.21.15.102:0/472605569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09e015ca00 msgr2=0x7f09e00cd950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.964+0000 7f09e5200640 1 --2- 172.21.15.102:0/472605569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09e015ca00 0x7f09e00cd950 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f09d8009d90 tx=0x7f09d8004a40 comp rx=0 tx=0).stop 2024-03-23T03:50:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.965+0000 7f09e5200640 1 -- 172.21.15.102:0/472605569 shutdown_connections 2024-03-23T03:50:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.965+0000 7f09e5200640 1 --2- 172.21.15.102:0/472605569 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f09c006ac00 0x7f09c006d0c0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.965+0000 7f09e5200640 1 --2- 172.21.15.102:0/472605569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09e015ca00 0x7f09e00cd950 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.965+0000 7f09e5200640 1 --2- 172.21.15.102:0/472605569 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09e00b26e0 0x7f09e0154bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.965+0000 7f09e5200640 1 --2- 172.21.15.102:0/472605569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f09e0069c90 0x7f09e0153390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.965+0000 7f09e5200640 1 -- 172.21.15.102:0/472605569 >> 172.21.15.102:0/472605569 conn(0x7f09e006d440 msgr2=0x7f09e0159090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.965+0000 7f09e5200640 1 -- 172.21.15.102:0/472605569 shutdown_connections 2024-03-23T03:50:09.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:09.965+0000 7f09e5200640 1 -- 172.21.15.102:0/472605569 wait complete. 2024-03-23T03:50:09.986 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-23T03:50:09.986 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T03:50:10.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.198+0000 7f66e2000640 1 Processor -- start 2024-03-23T03:50:10.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.198+0000 7f66e2000640 1 -- start start 2024-03-23T03:50:10.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.200+0000 7f66e2000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f66dc0cd6b0 0x7f66dc06baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:10.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.200+0000 7f66e2000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66dc06b0a0 con 0x7f66dc15cd10 2024-03-23T03:50:10.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.200+0000 7f66e2000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66dc0d2330 con 0x7f66dc0b1b60 2024-03-23T03:50:10.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.200+0000 7f66e2000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66dc0d2610 con 0x7f66dc0cd6b0 2024-03-23T03:50:10.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.200+0000 7f66daa00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f66dc15cd10 0x7f66dc0cda90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38672/0 (socket says 172.21.15.102:38672) 2024-03-23T03:50:10.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.200+0000 7f66daa00640 1 -- 172.21.15.102:0/175280632 learned_addr learned my addr 172.21.15.102:0/175280632 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:10.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.200+0000 7f66da000640 1 -- 172.21.15.102:0/175280632 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197503310 0 0) 0x7f66dc06b0a0 con 0x7f66dc15cd10 2024-03-23T03:50:10.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.201+0000 7f66da000640 1 -- 172.21.15.102:0/175280632 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66c0003520 con 0x7f66dc15cd10 2024-03-23T03:50:10.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.200+0000 7f66db400640 1 --2- 172.21.15.102:0/175280632 >> v2:172.21.15.173:3300/0 conn(0x7f66dc0cd6b0 0x7f66dc06baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:10.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.201+0000 7f66da000640 1 -- 172.21.15.102:0/175280632 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4264285350 0 0) 0x7f66c0003520 con 0x7f66dc15cd10 2024-03-23T03:50:10.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.201+0000 7f66da000640 1 -- 172.21.15.102:0/175280632 >> v2:172.21.15.173:3300/0 conn(0x7f66dc0cd6b0 msgr2=0x7f66dc06baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:10.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.201+0000 7f66da000640 1 --2- 172.21.15.102:0/175280632 >> v2:172.21.15.173:3300/0 conn(0x7f66dc0cd6b0 0x7f66dc06baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:10.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.201+0000 7f66da000640 1 -- 172.21.15.102:0/175280632 >> v1:172.21.15.173:6789/0 conn(0x7f66dc0b1b60 legacy=0x7f66dc0d1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:10.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.201+0000 7f66da000640 1 -- 172.21.15.102:0/175280632 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc06b2f0 con 0x7f66dc15cd10 2024-03-23T03:50:10.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.201+0000 7f66da000640 1 -- 172.21.15.102:0/175280632 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f66c80019f0 con 0x7f66dc15cd10 2024-03-23T03:50:10.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.201+0000 7f66da000640 1 -- 172.21.15.102:0/175280632 >> v1:172.21.15.102:6789/0 conn(0x7f66dc15cd10 legacy=0x7f66dc0cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:10.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.201+0000 7f66da000640 1 --2- 172.21.15.102:0/175280632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66c00039c0 0x7f66c0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:10.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.201+0000 7f66da000640 1 --2- 172.21.15.102:0/175280632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c00047c0 0x7f66c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:10.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.201+0000 7f66da000640 1 --2- 172.21.15.102:0/175280632 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f66c00051d0 0x7f66c000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:10.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.201+0000 7f66da000640 1 -- 172.21.15.102:0/175280632 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc0d2610 con 0x7f66c00047c0 2024-03-23T03:50:10.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.201+0000 7f66da000640 1 -- 172.21.15.102:0/175280632 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66c000c100 con 0x7f66c00039c0 2024-03-23T03:50:10.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.201+0000 7f66da000640 1 -- 172.21.15.102:0/175280632 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66c000c2c0 con 0x7f66c00051d0 2024-03-23T03:50:10.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.201+0000 7f66daa00640 1 --2- 172.21.15.102:0/175280632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c00047c0 0x7f66c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:10.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.202+0000 7f66db400640 1 --2- 172.21.15.102:0/175280632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66c00039c0 0x7f66c0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:10.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.202+0000 7f66daa00640 1 -- 172.21.15.102:0/175280632 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f66c00051d0 msgr2=0x7f66c000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:10.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.202+0000 7f66daa00640 1 --2- 172.21.15.102:0/175280632 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f66c00051d0 0x7f66c000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:10.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.202+0000 7f66daa00640 1 -- 172.21.15.102:0/175280632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66c00039c0 msgr2=0x7f66c0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:10.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.202+0000 7f66daa00640 1 --2- 172.21.15.102:0/175280632 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66c00039c0 0x7f66c0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:10.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.202+0000 7f66daa00640 1 -- 172.21.15.102:0/175280632 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c0003520 con 0x7f66c00047c0 2024-03-23T03:50:10.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.202+0000 7f66dbe00640 1 -- 172.21.15.102:0/175280632 reap_dead start 2024-03-23T03:50:10.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.203+0000 7f66daa00640 1 --2- 172.21.15.102:0/175280632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c00047c0 0x7f66c0004c40 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f66dc15d110 tx=0x7f66c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da6ae373d54d08d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:10.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.203+0000 7f66da000640 1 -- 172.21.15.102:0/175280632 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c8034630 con 0x7f66c00047c0 2024-03-23T03:50:10.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.203+0000 7f66da000640 1 -- 172.21.15.102:0/175280632 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c802f070 con 0x7f66c00047c0 2024-03-23T03:50:10.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.203+0000 7f66da000640 1 -- 172.21.15.102:0/175280632 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c8034a10 con 0x7f66c00047c0 2024-03-23T03:50:10.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.203+0000 7f66e2000640 1 -- 172.21.15.102:0/175280632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c00047c0 msgr2=0x7f66c0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:10.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.203+0000 7f66e2000640 1 --2- 172.21.15.102:0/175280632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c00047c0 0x7f66c0004c40 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f66dc15d110 tx=0x7f66c00057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:10.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.204+0000 7f66e2000640 1 -- 172.21.15.102:0/175280632 shutdown_connections 2024-03-23T03:50:10.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.204+0000 7f66e2000640 1 --2- 172.21.15.102:0/175280632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c00047c0 0x7f66c0004c40 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:10.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.204+0000 7f66e2000640 1 -- 172.21.15.102:0/175280632 >> 172.21.15.102:0/175280632 conn(0x7f66dc0cf5d0 msgr2=0x7f66dc0cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:10.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.204+0000 7f66e2000640 1 -- 172.21.15.102:0/175280632 shutdown_connections 2024-03-23T03:50:10.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.204+0000 7f66e2000640 1 -- 172.21.15.102:0/175280632 wait complete. 2024-03-23T03:50:10.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.205+0000 7f66e2000640 1 Processor -- start 2024-03-23T03:50:10.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.206+0000 7f66e2000640 1 -- start start 2024-03-23T03:50:10.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.206+0000 7f66e2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c00047c0 0x7f66dc0c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:10.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.206+0000 7f66e2000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f66dc1f26d0 0x7f66dc1f1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:10.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.206+0000 7f66e2000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66dc1f2b50 0x7f66dc1f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:10.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.207+0000 7f66e2000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc1f22d0 con 0x7f66c00047c0 2024-03-23T03:50:10.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.207+0000 7f66e2000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc0c7a10 con 0x7f66dc1f2b50 2024-03-23T03:50:10.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.207+0000 7f66e2000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66dc1541c0 con 0x7f66dc1f26d0 2024-03-23T03:50:10.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.207+0000 7f66db400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c00047c0 0x7f66dc0c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:10.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.207+0000 7f66db400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c00047c0 0x7f66dc0c74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59926/0 (socket says 172.21.15.102:59926) 2024-03-23T03:50:10.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.207+0000 7f66db400640 1 -- 172.21.15.102:0/1848673826 learned_addr learned my addr 172.21.15.102:0/1848673826 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:10.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.207+0000 7f66db400640 1 -- 172.21.15.102:0/1848673826 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f66dc1f26d0 msgr2=0x7f66dc1f1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:10.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.207+0000 7f66db400640 1 --2- 172.21.15.102:0/1848673826 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f66dc1f26d0 0x7f66dc1f1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:10.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.207+0000 7f66db400640 1 -- 172.21.15.102:0/1848673826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66dc1f2b50 msgr2=0x7f66dc1f1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:10.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.207+0000 7f66db400640 1 --2- 172.21.15.102:0/1848673826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66dc1f2b50 0x7f66dc1f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:10.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.207+0000 7f66db400640 1 -- 172.21.15.102:0/1848673826 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66dc1f66d0 con 0x7f66c00047c0 2024-03-23T03:50:10.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.208+0000 7f66db400640 1 --2- 172.21.15.102:0/1848673826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c00047c0 0x7f66dc0c74d0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f66d400f530 tx=0x7f66d400f9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:10.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.208+0000 7f66d3400640 1 -- 172.21.15.102:0/1848673826 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d4014870 con 0x7f66c00047c0 2024-03-23T03:50:10.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.208+0000 7f66d3400640 1 -- 172.21.15.102:0/1848673826 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d40149d0 con 0x7f66c00047c0 2024-03-23T03:50:10.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.208+0000 7f66d3400640 1 -- 172.21.15.102:0/1848673826 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d4014d80 con 0x7f66c00047c0 2024-03-23T03:50:10.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.208+0000 7f66e2000640 1 -- 172.21.15.102:0/1848673826 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66dc154480 con 0x7f66c00047c0 2024-03-23T03:50:10.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.208+0000 7f66e2000640 1 -- 172.21.15.102:0/1848673826 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66dc154a40 con 0x7f66c00047c0 2024-03-23T03:50:10.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.209+0000 7f66e2000640 1 -- 172.21.15.102:0/1848673826 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66dc073390 con 0x7f66c00047c0 2024-03-23T03:50:10.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.210+0000 7f66d3400640 1 -- 172.21.15.102:0/1848673826 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f66d4015410 con 0x7f66c00047c0 2024-03-23T03:50:10.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.211+0000 7f66d3400640 1 --2- 172.21.15.102:0/1848673826 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f66bc06ac30 0x7f66bc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:10.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.211+0000 7f66d3400640 1 -- 172.21.15.102:0/1848673826 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f66d4089d80 con 0x7f66c00047c0 2024-03-23T03:50:10.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.211+0000 7f66daa00640 1 --2- 172.21.15.102:0/1848673826 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f66bc06ac30 0x7f66bc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:10.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.212+0000 7f66daa00640 1 --2- 172.21.15.102:0/1848673826 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f66bc06ac30 0x7f66bc06d0f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f66c8034000 tx=0x7f66c8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:10.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.216+0000 7f66d3400640 1 -- 172.21.15.102:0/1848673826 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f66d4054e10 con 0x7f66c00047c0 2024-03-23T03:50:10.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.450+0000 7f66e2000640 1 -- 172.21.15.102:0/1848673826 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f66dc0b20b0 con 0x7f66c00047c0 2024-03-23T03:50:10.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.965+0000 7f66d3400640 1 -- 172.21.15.102:0/1848673826 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v41) v1 ==== 122+0+0 (secure 0 0 0) 0x7f66d4058430 con 0x7f66c00047c0 2024-03-23T03:50:10.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.968+0000 7f66e2000640 1 -- 172.21.15.102:0/1848673826 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f66bc06ac30 msgr2=0x7f66bc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:10.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.968+0000 7f66e2000640 1 --2- 172.21.15.102:0/1848673826 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f66bc06ac30 0x7f66bc06d0f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f66c8034000 tx=0x7f66c8009030 comp rx=0 tx=0).stop 2024-03-23T03:50:10.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.968+0000 7f66e2000640 1 -- 172.21.15.102:0/1848673826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c00047c0 msgr2=0x7f66dc0c74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:10.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.968+0000 7f66e2000640 1 --2- 172.21.15.102:0/1848673826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c00047c0 0x7f66dc0c74d0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f66d400f530 tx=0x7f66d400f9f0 comp rx=0 tx=0).stop 2024-03-23T03:50:10.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.968+0000 7f66e2000640 1 -- 172.21.15.102:0/1848673826 shutdown_connections 2024-03-23T03:50:10.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.969+0000 7f66e2000640 1 --2- 172.21.15.102:0/1848673826 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f66bc06ac30 0x7f66bc06d0f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:10.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.969+0000 7f66e2000640 1 --2- 172.21.15.102:0/1848673826 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66dc1f2b50 0x7f66dc1f1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:10.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.969+0000 7f66e2000640 1 --2- 172.21.15.102:0/1848673826 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f66dc1f26d0 0x7f66dc1f1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:10.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.969+0000 7f66e2000640 1 --2- 172.21.15.102:0/1848673826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66c00047c0 0x7f66dc0c74d0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:10.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.969+0000 7f66e2000640 1 -- 172.21.15.102:0/1848673826 >> 172.21.15.102:0/1848673826 conn(0x7f66dc0cf5d0 msgr2=0x7f66dc0b7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:10.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.969+0000 7f66e2000640 1 -- 172.21.15.102:0/1848673826 shutdown_connections 2024-03-23T03:50:10.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:10.969+0000 7f66e2000640 1 -- 172.21.15.102:0/1848673826 wait complete. 2024-03-23T03:50:10.989 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T03:50:10.989 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:50:11.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.185+0000 7f0c27200640 1 Processor -- start 2024-03-23T03:50:11.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.185+0000 7f0c27200640 1 -- start start 2024-03-23T03:50:11.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.186+0000 7f0c27200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f0c28070290 0x7f0c280b33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:11.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.186+0000 7f0c27200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0c28070670 0x7f0c2815cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:11.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.186+0000 7f0c27200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0c2815d020 0x7f0c2806c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:11.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.186+0000 7f0c27200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c280b2f50 con 0x7f0c2815d020 2024-03-23T03:50:11.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.186+0000 7f0c27200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c280b3920 con 0x7f0c28070290 2024-03-23T03:50:11.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.186+0000 7f0c27200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c280b2620 con 0x7f0c28070670 2024-03-23T03:50:11.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.187+0000 7f0c26800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0c2815d020 0x7f0c2806c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:11.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.187+0000 7f0c26800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0c2815d020 0x7f0c2806c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59938/0 (socket says 172.21.15.102:59938) 2024-03-23T03:50:11.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.187+0000 7f0c26800640 1 -- 172.21.15.102:0/2758693462 learned_addr learned my addr 172.21.15.102:0/2758693462 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:11.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.187+0000 7f0c25e00640 1 --2- 172.21.15.102:0/2758693462 >> v2:172.21.15.173:3300/0 conn(0x7f0c28070290 0x7f0c280b33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:11.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.187+0000 7f0c1d400640 1 --2- 172.21.15.102:0/2758693462 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0c28070670 0x7f0c2815cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:11.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.187+0000 7f0c26800640 1 -- 172.21.15.102:0/2758693462 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0c28070670 msgr2=0x7f0c2815cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:11.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.187+0000 7f0c26800640 1 --2- 172.21.15.102:0/2758693462 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0c28070670 0x7f0c2815cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.187+0000 7f0c26800640 1 -- 172.21.15.102:0/2758693462 >> v2:172.21.15.173:3300/0 conn(0x7f0c28070290 msgr2=0x7f0c280b33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:11.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.187+0000 7f0c26800640 1 --2- 172.21.15.102:0/2758693462 >> v2:172.21.15.173:3300/0 conn(0x7f0c28070290 0x7f0c280b33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.187+0000 7f0c26800640 1 -- 172.21.15.102:0/2758693462 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c280b2900 con 0x7f0c2815d020 2024-03-23T03:50:11.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.188+0000 7f0c26800640 1 --2- 172.21.15.102:0/2758693462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c2815d020 0x7f0c2806c080 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f0c2806ab30 tx=0x7f0c2002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ce34ac748f823dd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:11.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.189+0000 7f0c25400640 1 -- 172.21.15.102:0/2758693462 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2002ca70 con 0x7f0c2815d020 2024-03-23T03:50:11.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.189+0000 7f0c25400640 1 -- 172.21.15.102:0/2758693462 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2002cbd0 con 0x7f0c2815d020 2024-03-23T03:50:11.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.189+0000 7f0c25400640 1 -- 172.21.15.102:0/2758693462 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c200343f0 con 0x7f0c2815d020 2024-03-23T03:50:11.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.189+0000 7f0c27200640 1 -- 172.21.15.102:0/2758693462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c2815d020 msgr2=0x7f0c2806c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:11.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.189+0000 7f0c27200640 1 --2- 172.21.15.102:0/2758693462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c2815d020 0x7f0c2806c080 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f0c2806ab30 tx=0x7f0c2002bed0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.189+0000 7f0c27200640 1 -- 172.21.15.102:0/2758693462 shutdown_connections 2024-03-23T03:50:11.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.189+0000 7f0c27200640 1 --2- 172.21.15.102:0/2758693462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c2815d020 0x7f0c2806c080 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.189+0000 7f0c27200640 1 --2- 172.21.15.102:0/2758693462 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0c28070670 0x7f0c2815cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.189+0000 7f0c27200640 1 --2- 172.21.15.102:0/2758693462 >> v2:172.21.15.173:3300/0 conn(0x7f0c28070290 0x7f0c280b33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.189+0000 7f0c27200640 1 -- 172.21.15.102:0/2758693462 >> 172.21.15.102:0/2758693462 conn(0x7f0c28065df0 msgr2=0x7f0c28069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:11.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.189+0000 7f0c27200640 1 -- 172.21.15.102:0/2758693462 shutdown_connections 2024-03-23T03:50:11.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.190+0000 7f0c27200640 1 -- 172.21.15.102:0/2758693462 wait complete. 2024-03-23T03:50:11.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.191+0000 7f0c27200640 1 Processor -- start 2024-03-23T03:50:11.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.191+0000 7f0c27200640 1 -- start start 2024-03-23T03:50:11.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.192+0000 7f0c27200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0c28070290 0x7f0c281524f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:11.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.192+0000 7f0c27200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c28070670 0x7f0c28135470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:11.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.192+0000 7f0c27200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c2815d020 0x7f0c281326c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:11.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.192+0000 7f0c27200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c28159230 con 0x7f0c28070670 2024-03-23T03:50:11.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.192+0000 7f0c27200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c28159460 con 0x7f0c2815d020 2024-03-23T03:50:11.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.192+0000 7f0c27200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c28159740 con 0x7f0c28070290 2024-03-23T03:50:11.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.193+0000 7f0c1d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c28070670 0x7f0c28135470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:11.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.193+0000 7f0c1d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c28070670 0x7f0c28135470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59942/0 (socket says 172.21.15.102:59942) 2024-03-23T03:50:11.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.193+0000 7f0c1d400640 1 -- 172.21.15.102:0/3464679565 learned_addr learned my addr 172.21.15.102:0/3464679565 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:11.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.193+0000 7f0c1d400640 1 -- 172.21.15.102:0/3464679565 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0c28070290 msgr2=0x7f0c281524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:11.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.193+0000 7f0c1d400640 1 --2- 172.21.15.102:0/3464679565 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0c28070290 0x7f0c281524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.193+0000 7f0c1d400640 1 -- 172.21.15.102:0/3464679565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c2815d020 msgr2=0x7f0c281326c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:11.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.193+0000 7f0c1d400640 1 --2- 172.21.15.102:0/3464679565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c2815d020 0x7f0c281326c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.193+0000 7f0c1d400640 1 -- 172.21.15.102:0/3464679565 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c20009b30 con 0x7f0c28070670 2024-03-23T03:50:11.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.193+0000 7f0c1d400640 1 --2- 172.21.15.102:0/3464679565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c28070670 0x7f0c28135470 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f0c1000d800 tx=0x7f0c1000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:11.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.194+0000 7f0c1ea00640 1 -- 172.21.15.102:0/3464679565 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1000b6c0 con 0x7f0c28070670 2024-03-23T03:50:11.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.194+0000 7f0c1ea00640 1 -- 172.21.15.102:0/3464679565 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c1000b820 con 0x7f0c28070670 2024-03-23T03:50:11.194 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.194+0000 7f0c1ea00640 1 -- 172.21.15.102:0/3464679565 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1000bbb0 con 0x7f0c28070670 2024-03-23T03:50:11.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.194+0000 7f0c27200640 1 -- 172.21.15.102:0/3464679565 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c28159a20 con 0x7f0c28070670 2024-03-23T03:50:11.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.194+0000 7f0c27200640 1 -- 172.21.15.102:0/3464679565 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c28159fe0 con 0x7f0c28070670 2024-03-23T03:50:11.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.195+0000 7f0c27200640 1 -- 172.21.15.102:0/3464679565 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c2804e030 con 0x7f0c28070670 2024-03-23T03:50:11.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.196+0000 7f0c1ea00640 1 -- 172.21.15.102:0/3464679565 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0c1000bd10 con 0x7f0c28070670 2024-03-23T03:50:11.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.197+0000 7f0c1ea00640 1 --2- 172.21.15.102:0/3464679565 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0c0406ac00 0x7f0c0406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:11.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.197+0000 7f0c1ea00640 1 -- 172.21.15.102:0/3464679565 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0c10089a40 con 0x7f0c28070670 2024-03-23T03:50:11.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.197+0000 7f0c25e00640 1 --2- 172.21.15.102:0/3464679565 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0c0406ac00 0x7f0c0406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:11.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.198+0000 7f0c25e00640 1 --2- 172.21.15.102:0/3464679565 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0c0406ac00 0x7f0c0406d0c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0c28138fb0 tx=0x7f0c18008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:11.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.201+0000 7f0c1ea00640 1 -- 172.21.15.102:0/3464679565 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0c1008be60 con 0x7f0c28070670 2024-03-23T03:50:11.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.425+0000 7f0c27200640 1 -- 172.21.15.102:0/3464679565 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c28158c70 con 0x7f0c28070670 2024-03-23T03:50:11.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.426+0000 7f0c1ea00640 1 -- 172.21.15.102:0/3464679565 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f0c10054a50 con 0x7f0c28070670 2024-03-23T03:50:11.426 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:11.426 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2079098809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2079098809},{"type":"v1","addr":"172.21.15.173:6833","nonce":2079098809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":5413,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/459102175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":459102175},{"type":"v1","addr":"172.21.15.173:6835","nonce":459102175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2962195583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2962195583},{"type":"v1","addr":"172.21.15.102:6835","nonce":2962195583}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:50:07.937613+0000","modified":"2024-03-23T03:50:10.958097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6384":{"gid":6384,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/3091739472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3091739472},{"type":"v1","addr":"172.21.15.102:6837","nonce":3091739472}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T03:50:11.426 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 41 2024-03-23T03:50:11.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.428+0000 7f0c27200640 1 -- 172.21.15.102:0/3464679565 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0c0406ac00 msgr2=0x7f0c0406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:11.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.429+0000 7f0c27200640 1 --2- 172.21.15.102:0/3464679565 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0c0406ac00 0x7f0c0406d0c0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f0c28138fb0 tx=0x7f0c18008040 comp rx=0 tx=0).stop 2024-03-23T03:50:11.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.429+0000 7f0c27200640 1 -- 172.21.15.102:0/3464679565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c28070670 msgr2=0x7f0c28135470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:11.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.429+0000 7f0c27200640 1 --2- 172.21.15.102:0/3464679565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c28070670 0x7f0c28135470 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f0c1000d800 tx=0x7f0c1000dcc0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.429+0000 7f0c27200640 1 -- 172.21.15.102:0/3464679565 shutdown_connections 2024-03-23T03:50:11.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.429+0000 7f0c27200640 1 --2- 172.21.15.102:0/3464679565 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0c0406ac00 0x7f0c0406d0c0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.429+0000 7f0c27200640 1 --2- 172.21.15.102:0/3464679565 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0c2815d020 0x7f0c281326c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.429+0000 7f0c27200640 1 --2- 172.21.15.102:0/3464679565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c28070670 0x7f0c28135470 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.429+0000 7f0c27200640 1 --2- 172.21.15.102:0/3464679565 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0c28070290 0x7f0c281524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.429+0000 7f0c27200640 1 -- 172.21.15.102:0/3464679565 >> 172.21.15.102:0/3464679565 conn(0x7f0c28065df0 msgr2=0x7f0c2813cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:11.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.429+0000 7f0c27200640 1 -- 172.21.15.102:0/3464679565 shutdown_connections 2024-03-23T03:50:11.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.429+0000 7f0c27200640 1 -- 172.21.15.102:0/3464679565 wait complete. 2024-03-23T03:50:11.446 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:50:11.642 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.642+0000 7f9952400640 1 Processor -- start 2024-03-23T03:50:11.643 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.642+0000 7f9952400640 1 -- start start 2024-03-23T03:50:11.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.643+0000 7f9952400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f994c15cab0 0x7f994c0b8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:11.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.643+0000 7f9952400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f994c0ceeb0 0x7f994c0cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:11.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.643+0000 7f9952400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f994c0cf850 con 0x7f994c0b4120 2024-03-23T03:50:11.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.643+0000 7f9952400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c15ce90 con 0x7f994c0ceeb0 2024-03-23T03:50:11.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.643+0000 7f9952400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c0cddf0 con 0x7f994c15cab0 2024-03-23T03:50:11.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.644+0000 7f9951000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f994c15cab0 0x7f994c0b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:11.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.644+0000 7f994be00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f994c0ceeb0 0x7f994c0cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:11.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.644+0000 7f994be00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f994c0ceeb0 0x7f994c0cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:35506/0 (socket says 172.21.15.102:35506) 2024-03-23T03:50:11.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.644+0000 7f994be00640 1 -- 172.21.15.102:0/3076646411 learned_addr learned my addr 172.21.15.102:0/3076646411 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:11.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.645+0000 7f9951000640 1 -- 172.21.15.102:0/3076646411 >> v2:172.21.15.173:3300/0 conn(0x7f994c0ceeb0 msgr2=0x7f994c0cf310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:11.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.645+0000 7f9951000640 1 --2- 172.21.15.102:0/3076646411 >> v2:172.21.15.173:3300/0 conn(0x7f994c0ceeb0 0x7f994c0cf310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.645+0000 7f9951000640 1 -- 172.21.15.102:0/3076646411 >> v1:172.21.15.173:6789/0 conn(0x7f994c0b4120 legacy=0x7f994c0b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:11.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.645+0000 7f9951000640 1 -- 172.21.15.102:0/3076646411 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0cd570 con 0x7f994c15cab0 2024-03-23T03:50:11.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.645+0000 7f994be00640 1 --2- 172.21.15.102:0/3076646411 >> v2:172.21.15.173:3300/0 conn(0x7f994c0ceeb0 0x7f994c0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:50:11.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.646+0000 7f9951000640 1 --2- 172.21.15.102:0/3076646411 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f994c15cab0 0x7f994c0b8050 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f993c009d90 tx=0x7f993c02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cce5c086cbf1dd0a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:11.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.647+0000 7f994b400640 1 -- 172.21.15.102:0/3076646411 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c02cc60 con 0x7f994c15cab0 2024-03-23T03:50:11.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.647+0000 7f994b400640 1 -- 172.21.15.102:0/3076646411 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c02cdc0 con 0x7f994c15cab0 2024-03-23T03:50:11.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.647+0000 7f994b400640 1 -- 172.21.15.102:0/3076646411 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993c034660 con 0x7f994c15cab0 2024-03-23T03:50:11.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.648+0000 7f9952400640 1 -- 172.21.15.102:0/3076646411 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f994c15cab0 msgr2=0x7f994c0b8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:11.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.648+0000 7f9952400640 1 --2- 172.21.15.102:0/3076646411 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f994c15cab0 0x7f994c0b8050 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f993c009d90 tx=0x7f993c02beb0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.648+0000 7f9952400640 1 -- 172.21.15.102:0/3076646411 shutdown_connections 2024-03-23T03:50:11.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.648+0000 7f9952400640 1 --2- 172.21.15.102:0/3076646411 >> v2:172.21.15.173:3300/0 conn(0x7f994c0ceeb0 0x7f994c0cf310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.648+0000 7f9952400640 1 --2- 172.21.15.102:0/3076646411 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f994c15cab0 0x7f994c0b8050 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.648+0000 7f9952400640 1 -- 172.21.15.102:0/3076646411 >> 172.21.15.102:0/3076646411 conn(0x7f994c06ae90 msgr2=0x7f994c06b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:11.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.648+0000 7f9952400640 1 -- 172.21.15.102:0/3076646411 shutdown_connections 2024-03-23T03:50:11.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.648+0000 7f9952400640 1 -- 172.21.15.102:0/3076646411 wait complete. 2024-03-23T03:50:11.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.650+0000 7f9952400640 1 Processor -- start 2024-03-23T03:50:11.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.650+0000 7f9952400640 1 -- start start 2024-03-23T03:50:11.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.652+0000 7f9952400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f994c0b4120 0x7f994c15f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:11.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.652+0000 7f9952400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f994c0ceeb0 0x7f994c14c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:11.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.652+0000 7f9952400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f994c15cab0 0x7f994c15dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:11.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.652+0000 7f9952400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c15ff30 con 0x7f994c15cab0 2024-03-23T03:50:11.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.652+0000 7f9952400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c0c6c20 con 0x7f994c0b4120 2024-03-23T03:50:11.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.652+0000 7f9952400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c15e950 con 0x7f994c0ceeb0 2024-03-23T03:50:11.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.652+0000 7f9951a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f994c15cab0 0x7f994c15dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:11.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.652+0000 7f9951a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f994c15cab0 0x7f994c15dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59950/0 (socket says 172.21.15.102:59950) 2024-03-23T03:50:11.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.652+0000 7f9951a00640 1 -- 172.21.15.102:0/2951953294 learned_addr learned my addr 172.21.15.102:0/2951953294 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:11.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.652+0000 7f9951000640 1 --2- 172.21.15.102:0/2951953294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f994c0b4120 0x7f994c15f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:11.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.652+0000 7f994be00640 1 --2- 172.21.15.102:0/2951953294 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f994c0ceeb0 0x7f994c14c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:11.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.653+0000 7f9951a00640 1 -- 172.21.15.102:0/2951953294 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f994c0ceeb0 msgr2=0x7f994c14c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:11.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.653+0000 7f9951a00640 1 --2- 172.21.15.102:0/2951953294 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f994c0ceeb0 0x7f994c14c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.653+0000 7f9951a00640 1 -- 172.21.15.102:0/2951953294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f994c0b4120 msgr2=0x7f994c15f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:11.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.653+0000 7f9951a00640 1 --2- 172.21.15.102:0/2951953294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f994c0b4120 0x7f994c15f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.653+0000 7f9951a00640 1 -- 172.21.15.102:0/2951953294 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99340094d0 con 0x7f994c15cab0 2024-03-23T03:50:11.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.653+0000 7f9951a00640 1 --2- 172.21.15.102:0/2951953294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f994c15cab0 0x7f994c15dd90 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f9944001eb0 tx=0x7f994400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:11.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.653+0000 7f9948c00640 1 -- 172.21.15.102:0/2951953294 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99440052d0 con 0x7f994c15cab0 2024-03-23T03:50:11.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.653+0000 7f9948c00640 1 -- 172.21.15.102:0/2951953294 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9944005430 con 0x7f994c15cab0 2024-03-23T03:50:11.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.653+0000 7f9948c00640 1 -- 172.21.15.102:0/2951953294 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9944003810 con 0x7f994c15cab0 2024-03-23T03:50:11.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.653+0000 7f9952400640 1 -- 172.21.15.102:0/2951953294 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f993c009a70 con 0x7f994c15cab0 2024-03-23T03:50:11.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.653+0000 7f9952400640 1 -- 172.21.15.102:0/2951953294 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994c15ef10 con 0x7f994c15cab0 2024-03-23T03:50:11.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.654+0000 7f9952400640 1 -- 172.21.15.102:0/2951953294 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994c04e030 con 0x7f994c15cab0 2024-03-23T03:50:11.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.655+0000 7f9948c00640 1 -- 172.21.15.102:0/2951953294 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9944003970 con 0x7f994c15cab0 2024-03-23T03:50:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.656+0000 7f9948c00640 1 --2- 172.21.15.102:0/2951953294 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f992806ac30 0x7f992806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.656+0000 7f9948c00640 1 -- 172.21.15.102:0/2951953294 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9944088a00 con 0x7f994c15cab0 2024-03-23T03:50:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.656+0000 7f9951000640 1 --2- 172.21.15.102:0/2951953294 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f992806ac30 0x7f992806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:11.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.657+0000 7f9951000640 1 --2- 172.21.15.102:0/2951953294 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f992806ac30 0x7f992806d0f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f993c036000 tx=0x7f993c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:11.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.661+0000 7f9948c00640 1 -- 172.21.15.102:0/2951953294 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9944053aa0 con 0x7f994c15cab0 2024-03-23T03:50:11.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.856+0000 7f9952400640 1 -- 172.21.15.102:0/2951953294 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f994c158100 con 0x7f994c15cab0 2024-03-23T03:50:11.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.858+0000 7f9948c00640 1 -- 172.21.15.102:0/2951953294 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16357 (secure 0 0 0) 0x7f9944017090 con 0x7f994c15cab0 2024-03-23T03:50:11.859 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:11.859 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":81,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:08.942399+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:50:06.347556+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-23T03:50:06.924370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:50:11.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.860+0000 7f9952400640 1 -- 172.21.15.102:0/2951953294 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f992806ac30 msgr2=0x7f992806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:11.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.860+0000 7f9952400640 1 --2- 172.21.15.102:0/2951953294 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f992806ac30 0x7f992806d0f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f993c036000 tx=0x7f993c030000 comp rx=0 tx=0).stop 2024-03-23T03:50:11.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.860+0000 7f9952400640 1 -- 172.21.15.102:0/2951953294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f994c15cab0 msgr2=0x7f994c15dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:11.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.860+0000 7f9952400640 1 --2- 172.21.15.102:0/2951953294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f994c15cab0 0x7f994c15dd90 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f9944001eb0 tx=0x7f994400c890 comp rx=0 tx=0).stop 2024-03-23T03:50:11.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.861+0000 7f9952400640 1 -- 172.21.15.102:0/2951953294 shutdown_connections 2024-03-23T03:50:11.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.861+0000 7f9952400640 1 --2- 172.21.15.102:0/2951953294 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f992806ac30 0x7f992806d0f0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.861+0000 7f9952400640 1 --2- 172.21.15.102:0/2951953294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f994c15cab0 0x7f994c15dd90 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.861+0000 7f9952400640 1 --2- 172.21.15.102:0/2951953294 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f994c0ceeb0 0x7f994c14c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.861+0000 7f9952400640 1 --2- 172.21.15.102:0/2951953294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f994c0b4120 0x7f994c15f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:11.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.861+0000 7f9952400640 1 -- 172.21.15.102:0/2951953294 >> 172.21.15.102:0/2951953294 conn(0x7f994c06ae90 msgr2=0x7f994c0b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:11.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.861+0000 7f9952400640 1 -- 172.21.15.102:0/2951953294 shutdown_connections 2024-03-23T03:50:11.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:11.861+0000 7f9952400640 1 -- 172.21.15.102:0/2951953294 wait complete. 2024-03-23T03:50:11.881 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T03:50:11.881 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:50:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.086+0000 7f5a26a00640 1 Processor -- start 2024-03-23T03:50:12.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.086+0000 7f5a26a00640 1 -- start start 2024-03-23T03:50:12.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.088+0000 7f5a26a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5a200adbc0 0x7f5a200b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:12.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.088+0000 7f5a26a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f5a200cbab0 0x7f5a200b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:12.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.088+0000 7f5a26a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a200b16a0 con 0x7f5a2006d390 2024-03-23T03:50:12.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.088+0000 7f5a26a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a200b4700 con 0x7f5a200adbc0 2024-03-23T03:50:12.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.088+0000 7f5a26a00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a200cbfb0 con 0x7f5a200cbab0 2024-03-23T03:50:12.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.088+0000 7f5a24a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5a2006d390 0x7f5a200b2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38688/0 (socket says 172.21.15.102:38688) 2024-03-23T03:50:12.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.088+0000 7f5a24a00640 1 -- 172.21.15.102:0/3433992470 learned_addr learned my addr 172.21.15.102:0/3433992470 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:12.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.088+0000 7f5a1fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5a200adbc0 0x7f5a200b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.089+0000 7f5a1f400640 1 --2- 172.21.15.102:0/3433992470 >> v2:172.21.15.173:3300/0 conn(0x7f5a200cbab0 0x7f5a200b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.089+0000 7f5a1fe00640 1 -- 172.21.15.102:0/3433992470 >> v2:172.21.15.173:3300/0 conn(0x7f5a200cbab0 msgr2=0x7f5a200b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.089+0000 7f5a1fe00640 1 --2- 172.21.15.102:0/3433992470 >> v2:172.21.15.173:3300/0 conn(0x7f5a200cbab0 0x7f5a200b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.089+0000 7f5a1fe00640 1 -- 172.21.15.102:0/3433992470 >> v1:172.21.15.102:6789/0 conn(0x7f5a2006d390 legacy=0x7f5a200b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.089+0000 7f5a1fe00640 1 -- 172.21.15.102:0/3433992470 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a200cc290 con 0x7f5a200adbc0 2024-03-23T03:50:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.089+0000 7f5a1f400640 1 --2- 172.21.15.102:0/3433992470 >> v2:172.21.15.173:3300/0 conn(0x7f5a200cbab0 0x7f5a200b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:50:12.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.090+0000 7f5a1fe00640 1 --2- 172.21.15.102:0/3433992470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a200adbc0 0x7f5a200b41c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f5a100097a0 tx=0x7f5a1002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ac91d2cc810e38c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.090+0000 7f5a1ea00640 1 -- 172.21.15.102:0/3433992470 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a1002e070 con 0x7f5a200adbc0 2024-03-23T03:50:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.090+0000 7f5a1ea00640 1 -- 172.21.15.102:0/3433992470 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a100315a0 con 0x7f5a200adbc0 2024-03-23T03:50:12.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.090+0000 7f5a1ea00640 1 -- 172.21.15.102:0/3433992470 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10031930 con 0x7f5a200adbc0 2024-03-23T03:50:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.091+0000 7f5a26a00640 1 -- 172.21.15.102:0/3433992470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a200adbc0 msgr2=0x7f5a200b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.091+0000 7f5a26a00640 1 --2- 172.21.15.102:0/3433992470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a200adbc0 0x7f5a200b41c0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f5a100097a0 tx=0x7f5a1002b840 comp rx=0 tx=0).stop 2024-03-23T03:50:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.091+0000 7f5a26a00640 1 -- 172.21.15.102:0/3433992470 shutdown_connections 2024-03-23T03:50:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.091+0000 7f5a26a00640 1 --2- 172.21.15.102:0/3433992470 >> v2:172.21.15.173:3300/0 conn(0x7f5a200cbab0 0x7f5a200b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.091+0000 7f5a26a00640 1 --2- 172.21.15.102:0/3433992470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a200adbc0 0x7f5a200b41c0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.091+0000 7f5a26a00640 1 -- 172.21.15.102:0/3433992470 >> 172.21.15.102:0/3433992470 conn(0x7f5a2006a3d0 msgr2=0x7f5a200b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.091+0000 7f5a26a00640 1 -- 172.21.15.102:0/3433992470 shutdown_connections 2024-03-23T03:50:12.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.091+0000 7f5a26a00640 1 -- 172.21.15.102:0/3433992470 wait complete. 2024-03-23T03:50:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.093+0000 7f5a26a00640 1 Processor -- start 2024-03-23T03:50:12.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.093+0000 7f5a26a00640 1 -- start start 2024-03-23T03:50:12.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.094+0000 7f5a26a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2006d390 0x7f5a201589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.094+0000 7f5a26a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a200adbc0 0x7f5a201398a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.094+0000 7f5a26a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5a200cbab0 0x7f5a2013bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.094+0000 7f5a26a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2015afe0 con 0x7f5a2006d390 2024-03-23T03:50:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.094+0000 7f5a26a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2015c9d0 con 0x7f5a200adbc0 2024-03-23T03:50:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.094+0000 7f5a26a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a2015ccb0 con 0x7f5a200cbab0 2024-03-23T03:50:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.094+0000 7f5a1fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2006d390 0x7f5a201589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.094+0000 7f5a1fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2006d390 0x7f5a201589d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59966/0 (socket says 172.21.15.102:59966) 2024-03-23T03:50:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.094+0000 7f5a1fe00640 1 -- 172.21.15.102:0/3002202404 learned_addr learned my addr 172.21.15.102:0/3002202404 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.094+0000 7f5a1fe00640 1 -- 172.21.15.102:0/3002202404 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5a200cbab0 msgr2=0x7f5a2013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.094+0000 7f5a1fe00640 1 --2- 172.21.15.102:0/3002202404 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5a200cbab0 0x7f5a2013bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.094+0000 7f5a1fe00640 1 -- 172.21.15.102:0/3002202404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a200adbc0 msgr2=0x7f5a201398a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.094+0000 7f5a1fe00640 1 --2- 172.21.15.102:0/3002202404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a200adbc0 0x7f5a201398a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.094+0000 7f5a1fe00640 1 -- 172.21.15.102:0/3002202404 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a10009400 con 0x7f5a2006d390 2024-03-23T03:50:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.094+0000 7f5a1fe00640 1 --2- 172.21.15.102:0/3002202404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2006d390 0x7f5a201589d0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f5a10034040 tx=0x7f5a100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:12.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.095+0000 7f5a17e00640 1 -- 172.21.15.102:0/3002202404 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10032840 con 0x7f5a2006d390 2024-03-23T03:50:12.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.095+0000 7f5a17e00640 1 -- 172.21.15.102:0/3002202404 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a100329a0 con 0x7f5a2006d390 2024-03-23T03:50:12.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.095+0000 7f5a17e00640 1 -- 172.21.15.102:0/3002202404 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a10032d50 con 0x7f5a2006d390 2024-03-23T03:50:12.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.095+0000 7f5a26a00640 1 -- 172.21.15.102:0/3002202404 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a2015ac70 con 0x7f5a2006d390 2024-03-23T03:50:12.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.095+0000 7f5a26a00640 1 -- 172.21.15.102:0/3002202404 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2015a1d0 con 0x7f5a2006d390 2024-03-23T03:50:12.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.096+0000 7f5a26a00640 1 -- 172.21.15.102:0/3002202404 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a2006ac20 con 0x7f5a2006d390 2024-03-23T03:50:12.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.097+0000 7f5a17e00640 1 -- 172.21.15.102:0/3002202404 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5a1004e430 con 0x7f5a2006d390 2024-03-23T03:50:12.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.098+0000 7f5a17e00640 1 --2- 172.21.15.102:0/3002202404 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5a0006ac30 0x7f5a0006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:12.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.098+0000 7f5a17e00640 1 -- 172.21.15.102:0/3002202404 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5a100ab4d0 con 0x7f5a2006d390 2024-03-23T03:50:12.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.099+0000 7f5a1f400640 1 --2- 172.21.15.102:0/3002202404 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5a0006ac30 0x7f5a0006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:12.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.099+0000 7f5a1f400640 1 --2- 172.21.15.102:0/3002202404 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5a0006ac30 0x7f5a0006d0f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c006fd0 tx=0x7f5a0c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:12.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.103+0000 7f5a17e00640 1 -- 172.21.15.102:0/3002202404 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5a10076560 con 0x7f5a2006d390 2024-03-23T03:50:12.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.296+0000 7f5a26a00640 1 -- 172.21.15.102:0/3002202404 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5a200b2630 con 0x7f5a0006ac30 2024-03-23T03:50:12.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.305+0000 7f5a17e00640 1 -- 172.21.15.102:0/3002202404 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+276359 (secure 0 0 0) 0x7f5a0c009d80 con 0x7f5a0006ac30 2024-03-23T03:50:12.310 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:12.311 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:50:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.311+0000 7f5a26a00640 1 -- 172.21.15.102:0/3002202404 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5a0006ac30 msgr2=0x7f5a0006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.311+0000 7f5a26a00640 1 --2- 172.21.15.102:0/3002202404 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5a0006ac30 0x7f5a0006d0f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c006fd0 tx=0x7f5a0c008040 comp rx=0 tx=0).stop 2024-03-23T03:50:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.311+0000 7f5a26a00640 1 -- 172.21.15.102:0/3002202404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2006d390 msgr2=0x7f5a201589d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.311+0000 7f5a26a00640 1 --2- 172.21.15.102:0/3002202404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2006d390 0x7f5a201589d0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f5a10034040 tx=0x7f5a100046c0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.312+0000 7f5a26a00640 1 -- 172.21.15.102:0/3002202404 shutdown_connections 2024-03-23T03:50:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.312+0000 7f5a26a00640 1 --2- 172.21.15.102:0/3002202404 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5a0006ac30 0x7f5a0006d0f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.312+0000 7f5a26a00640 1 --2- 172.21.15.102:0/3002202404 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5a200cbab0 0x7f5a2013bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.312+0000 7f5a26a00640 1 --2- 172.21.15.102:0/3002202404 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5a200adbc0 0x7f5a201398a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.312+0000 7f5a26a00640 1 --2- 172.21.15.102:0/3002202404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a2006d390 0x7f5a201589d0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.312+0000 7f5a26a00640 1 -- 172.21.15.102:0/3002202404 >> 172.21.15.102:0/3002202404 conn(0x7f5a2006a3d0 msgr2=0x7f5a20138f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:12.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.312+0000 7f5a26a00640 1 -- 172.21.15.102:0/3002202404 shutdown_connections 2024-03-23T03:50:12.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.312+0000 7f5a26a00640 1 -- 172.21.15.102:0/3002202404 wait complete. 2024-03-23T03:50:12.331 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-03-23T03:50:11.778281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":228756,"kb_used_data":8500,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754745964,"statfs":{"total":773094113280,"available":772859867136,"internally_reserved":0,"allocated":8704000,"data_stored":4534281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":225431098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.333625"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450165+0000","last_change":"2024-03-23T03:50:06.450165+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450165+0000","last_undegraded":"2024-03-23T03:50:06.450165+0000","last_fullsized":"2024-03-23T03:50:06.450165+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:32:26.890806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.461707+0000","last_change":"2024-03-23T03:50:06.461707+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.461707+0000","last_undegraded":"2024-03-23T03:50:06.461707+0000","last_fullsized":"2024-03-23T03:50:06.461707+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:14:39.947258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.461900+0000","last_change":"2024-03-23T03:50:06.461900+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.461900+0000","last_undegraded":"2024-03-23T03:50:06.461900+0000","last_fullsized":"2024-03-23T03:50:06.461900+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:03:17.838071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.463303+0000","last_change":"2024-03-23T03:50:06.463303+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.463303+0000","last_undegraded":"2024-03-23T03:50:06.463303+0000","last_fullsized":"2024-03-23T03:50:06.463303+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:33:34.230046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.455179+0000","last_change":"2024-03-23T03:50:06.455179+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.455179+0000","last_undegraded":"2024-03-23T03:50:06.455179+0000","last_fullsized":"2024-03-23T03:50:06.455179+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:33:34.230046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.469417+0000","last_change":"2024-03-23T03:50:06.469417+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.469417+0000","last_undegraded":"2024-03-23T03:50:06.469417+0000","last_fullsized":"2024-03-23T03:50:06.469417+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:36:00.228119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.467620+0000","last_change":"2024-03-23T03:50:06.467620+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.467620+0000","last_undegraded":"2024-03-23T03:50:06.467620+0000","last_fullsized":"2024-03-23T03:50:06.467620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:06:01.399011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450165+0000","last_change":"2024-03-23T03:50:06.450165+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450165+0000","last_undegraded":"2024-03-23T03:50:06.450165+0000","last_fullsized":"2024-03-23T03:50:06.450165+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:59:56.498434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.462227+0000","last_change":"2024-03-23T03:50:06.462227+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.462227+0000","last_undegraded":"2024-03-23T03:50:06.462227+0000","last_fullsized":"2024-03-23T03:50:06.462227+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:44:27.821644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.469099+0000","last_change":"2024-03-23T03:50:06.469099+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.469099+0000","last_undegraded":"2024-03-23T03:50:06.469099+0000","last_fullsized":"2024-03-23T03:50:06.469099+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:36:57.803871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450817+0000","last_change":"2024-03-23T03:50:06.450817+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450817+0000","last_undegraded":"2024-03-23T03:50:06.450817+0000","last_fullsized":"2024-03-23T03:50:06.450817+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:18:47.365643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.462222+0000","last_change":"2024-03-23T03:50:06.462222+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.462222+0000","last_undegraded":"2024-03-23T03:50:06.462222+0000","last_fullsized":"2024-03-23T03:50:06.462222+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:48:07.848353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.470833+0000","last_change":"2024-03-23T03:50:06.470833+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.470833+0000","last_undegraded":"2024-03-23T03:50:06.470833+0000","last_fullsized":"2024-03-23T03:50:06.470833+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:36:15.054606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.460079+0000","last_change":"2024-03-23T03:50:06.460079+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.460079+0000","last_undegraded":"2024-03-23T03:50:06.460079+0000","last_fullsized":"2024-03-23T03:50:06.460079+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:28:24.935685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450491+0000","last_change":"2024-03-23T03:50:06.450491+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450491+0000","last_undegraded":"2024-03-23T03:50:06.450491+0000","last_fullsized":"2024-03-23T03:50:06.450491+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:24:41.782032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.467435+0000","last_change":"2024-03-23T03:50:06.467435+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.467435+0000","last_undegraded":"2024-03-23T03:50:06.467435+0000","last_fullsized":"2024-03-23T03:50:06.467435+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:49:00.654026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.460903+0000","last_change":"2024-03-23T03:50:06.460903+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.460903+0000","last_undegraded":"2024-03-23T03:50:06.460903+0000","last_fullsized":"2024-03-23T03:50:06.460903+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:59:01.344309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.455371+0000","last_change":"2024-03-23T03:50:06.455371+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.455371+0000","last_undegraded":"2024-03-23T03:50:06.455371+0000","last_fullsized":"2024-03-23T03:50:06.455371+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:19:21.548516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450380+0000","last_change":"2024-03-23T03:50:06.450380+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450380+0000","last_undegraded":"2024-03-23T03:50:06.450380+0000","last_fullsized":"2024-03-23T03:50:06.450380+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:54:59.757010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.451352+0000","last_change":"2024-03-23T03:50:06.451352+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.451352+0000","last_undegraded":"2024-03-23T03:50:06.451352+0000","last_fullsized":"2024-03-23T03:50:06.451352+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:03:17.838071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.455710+0000","last_change":"2024-03-23T03:50:06.455710+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.455710+0000","last_undegraded":"2024-03-23T03:50:06.455710+0000","last_fullsized":"2024-03-23T03:50:06.455710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:41:32.913577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.456982+0000","last_change":"2024-03-23T03:50:06.456982+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.456982+0000","last_undegraded":"2024-03-23T03:50:06.456982+0000","last_fullsized":"2024-03-23T03:50:06.456982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:46:17.453220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.455224+0000","last_change":"2024-03-23T03:50:06.455224+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.455224+0000","last_undegraded":"2024-03-23T03:50:06.455224+0000","last_fullsized":"2024-03-23T03:50:06.455224+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:54:23.987953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.469854+0000","last_change":"2024-03-23T03:50:06.469854+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.469854+0000","last_undegraded":"2024-03-23T03:50:06.469854+0000","last_fullsized":"2024-03-23T03:50:06.469854+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:14:48.590038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.467198+0000","last_change":"2024-03-23T03:50:06.467198+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.467198+0000","last_undegraded":"2024-03-23T03:50:06.467198+0000","last_fullsized":"2024-03-23T03:50:06.467198+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:51:13.757234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.460543+0000","last_change":"2024-03-23T03:50:06.460543+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.460543+0000","last_undegraded":"2024-03-23T03:50:06.460543+0000","last_fullsized":"2024-03-23T03:50:06.460543+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:01:47.992165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.460249+0000","last_change":"2024-03-23T03:50:06.460249+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.460249+0000","last_undegraded":"2024-03-23T03:50:06.460249+0000","last_fullsized":"2024-03-23T03:50:06.460249+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:58.116627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.461287+0000","last_change":"2024-03-23T03:50:06.461287+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.461287+0000","last_undegraded":"2024-03-23T03:50:06.461287+0000","last_fullsized":"2024-03-23T03:50:06.461287+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:19:21.548516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450250+0000","last_change":"2024-03-23T03:50:06.450250+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450250+0000","last_undegraded":"2024-03-23T03:50:06.450250+0000","last_fullsized":"2024-03-23T03:50:06.450250+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:36:27.444641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.460712+0000","last_change":"2024-03-23T03:50:06.460712+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.460712+0000","last_undegraded":"2024-03-23T03:50:06.460712+0000","last_fullsized":"2024-03-23T03:50:06.460712+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:13.215153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450710+0000","last_change":"2024-03-23T03:50:06.450710+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450710+0000","last_undegraded":"2024-03-23T03:50:06.450710+0000","last_fullsized":"2024-03-23T03:50:06.450710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:01:47.992165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.455254+0000","last_change":"2024-03-23T03:50:06.455254+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.455254+0000","last_undegraded":"2024-03-23T03:50:06.455254+0000","last_fullsized":"2024-03-23T03:50:06.455254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:11:43.166129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":210,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-23T03:50:04.652817+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:50:04.652817+0000","last_peered":"2024-03-23T03:50:04.652817+0000","last_clean":"2024-03-23T03:50:04.652817+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:50:04.652817+0000","last_undegraded":"2024-03-23T03:50:04.652817+0000","last_fullsized":"2024-03-23T03:50:04.652817+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542188,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28516,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343324,"statfs":{"total":96636764160,"available":96607563776,"internally_reserved":0,"allocated":1531904,"data_stored":1008794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.079,"15min":1.069},"min":{"1min":0.68200000000000005,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.573,"5min":1.635,"15min":1.635},"last":1.075},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0449999999999999,"15min":1.048},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.417,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.353}]},{"osd":1,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0549999999999999,"15min":1.044},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.544,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.466},{"interface":"front","average":{"1min":1.085,"5min":1.2330000000000001,"15min":1.3009999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.3999999999999999,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":0.90600000000000003}]},{"osd":2,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.139,"15min":1.1559999999999999},"min":{"1min":0.76400000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.7909999999999999,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.24},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.345,"15min":1.421},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.4630000000000001,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":0.57799999999999996}]},{"osd":3,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.355,"15min":1.4370000000000001},"min":{"1min":0.873,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.397,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.3380000000000001,"15min":1.3580000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":1.5840000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.6359999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.266,"15min":1.2669999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.6299999999999999,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.8220000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.357,"15min":1.375},"min":{"1min":0.77900000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0619999999999998,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.5189999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.3999999999999999,"15min":1.4219999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.6539999999999999,"5min":2.645,"15min":2.645},"last":1.278},{"interface":"front","average":{"1min":1.339,"5min":1.4370000000000001,"15min":1.4530000000000001},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":1.7050000000000001,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.696}]},{"osd":6,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.5229999999999999,"15min":1.5589999999999999},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":1.8200000000000001,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.9279999999999999},{"interface":"front","average":{"1min":1.415,"5min":1.4930000000000001,"15min":1.5209999999999999},"min":{"1min":1.1319999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":2.1509999999999998,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.734}]}]},{"osd":6,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.0309999999999999,"15min":1.04},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.3979999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0429999999999999,"15min":1.038},"min":{"1min":0.72099999999999997,"5min":0.62,"15min":0.62},"max":{"1min":1.6120000000000001,"5min":1.946,"15min":1.946},"last":0.66100000000000003}]},{"osd":1,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.874},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.129,"15min":1.1559999999999999},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.4810000000000001,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.80500000000000005}]},{"osd":2,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.075,"15min":1.069},"min":{"1min":0.73699999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.5449999999999999,"5min":2.044,"15min":2.044},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1950000000000001,"15min":1.218},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.96899999999999997}]},{"osd":3,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2629999999999999,"15min":1.2849999999999999},"min":{"1min":0.76800000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.4910000000000001,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2549999999999999,"15min":1.2649999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.6319999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.94199999999999995}]},{"osd":4,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3759999999999999,"15min":1.3839999999999999},"min":{"1min":0.67700000000000005,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.73,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.363,"15min":1.383},"min":{"1min":0.63400000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.762,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.448}]},{"osd":5,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.415,"15min":1.4359999999999999},"min":{"1min":0.71099999999999997,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.6919999999999999,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.456,"5min":1.5249999999999999,"15min":1.5469999999999999},"min":{"1min":0.79700000000000004,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8149999999999999,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.091}]},{"osd":7,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.4059999999999999,"15min":1.4119999999999999},"min":{"1min":0.77600000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.802,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.5129999999999999,"15min":1.5409999999999999},"min":{"1min":0.84099999999999997,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.74,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.254}]}]},{"osd":5,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.1759999999999999,"15min":1.214},"min":{"1min":0.67900000000000005,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.6779999999999999,"5min":1.806,"15min":1.806},"last":1.2},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.157,"15min":1.181},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.554,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.90800000000000003}]},{"osd":1,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.81699999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.5489999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.026,"5min":1.1599999999999999,"15min":1.1919999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4219999999999999,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.0489999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.298,"15min":1.3540000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5429999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3340000000000001,"15min":1.3779999999999999},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.772,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.0840000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.21,"15min":1.2470000000000001},"min":{"1min":0.70599999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.653,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.393,"15min":1.452},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.6779999999999999,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.137}]},{"osd":4,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.3420000000000001,"15min":1.4119999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.671,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.282,"5min":1.4119999999999999,"15min":1.444},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.77,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.2270000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.458,"15min":1.514},"min":{"1min":0.76700000000000002,"5min":0.63,"15min":0.63},"max":{"1min":1.873,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.5940000000000001,"15min":1.6559999999999999},"min":{"1min":0.86199999999999999,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.956,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.397}]},{"osd":7,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.538,"15min":1.607},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.673,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.411,"5min":1.6200000000000001,"15min":1.696},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.855,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.6100000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.002,"15min":0.97999999999999998},"min":{"1min":0.67800000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.0149999999999999,"15min":1.012},"min":{"1min":0.81200000000000006,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3069999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.71399999999999997}]},{"osd":1,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.0700000000000001,"15min":1.05},"min":{"1min":0.74299999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.704,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.147,"5min":1.0880000000000001,"15min":1.0640000000000001},"min":{"1min":0.50600000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.181}]},{"osd":2,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2949999999999999,"15min":1.325},"min":{"1min":0.81599999999999995,"5min":0.625,"15min":0.625},"max":{"1min":1.569,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.29,"5min":1.3100000000000001,"15min":1.3080000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.8680000000000001,"5min":2.181,"15min":2.181},"last":1.2869999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1339999999999999,"15min":1.1299999999999999},"min":{"1min":0.73199999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.26,"15min":1.278},"min":{"1min":0.82899999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.5720000000000001,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":0.91600000000000004}]},{"osd":5,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.278,"15min":1.274},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.298,"15min":1.2829999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.2349999999999999,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.228}]},{"osd":6,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.415,"15min":1.4239999999999999},"min":{"1min":1.143,"5min":0.748,"15min":0.748},"max":{"1min":1.833,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.425,"15min":1.4039999999999999},"min":{"1min":0.745,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":1.377}]},{"osd":7,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.385,"15min":1.371},"min":{"1min":1.0880000000000001,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":2.3479999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.4970000000000001,"15min":1.5009999999999999},"min":{"1min":1.056,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.974,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.25}]}]},{"osd":3,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.83899999999999997,"15min":0.86099999999999999},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.256,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.446,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":1.4610000000000001}]},{"osd":1,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.92800000000000005,"15min":0.97199999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.997,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.034,"15min":1.0720000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.712,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.90400000000000003}]},{"osd":2,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.006,"15min":1.052},"min":{"1min":0.435,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.613,"5min":1.823,"15min":1.823},"last":1.109},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.1319999999999999,"15min":1.153},"min":{"1min":0.48199999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.6020000000000001,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":1.3700000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.2310000000000001,"15min":1.2450000000000001},"min":{"1min":0.67600000000000005,"5min":0.625,"15min":0.625},"max":{"1min":1.3859999999999999,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":1.054},{"interface":"front","average":{"1min":1.151,"5min":1.377,"15min":1.4370000000000001},"min":{"1min":0.77100000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.4770000000000001,"5min":4.819,"15min":4.819},"last":1.4850000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.603,"15min":1.4470000000000001},"min":{"1min":0.75,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.0299999999999998,"5min":30.228000000000002,"15min":30.228000000000002},"last":1.165},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.3320000000000001,"15min":1.355},"min":{"1min":0.91400000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.637,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":1.2709999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.448,"15min":1.52},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.5860000000000001,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.206},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3999999999999999,"15min":1.397},"min":{"1min":0.90100000000000002,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.1429999999999998,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":1.5}]},{"osd":7,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.395,"15min":1.4219999999999999},"min":{"1min":0.81200000000000006,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.71,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.577,"15min":1.601},"min":{"1min":0.73299999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.629,"5min":10.663,"15min":10.663},"last":1.258}]}]},{"osd":2,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.84999999999999998,"15min":0.88100000000000001},"min":{"1min":0.318,"5min":0.307,"15min":0.307},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.1930000000000001},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.88700000000000001,"15min":0.94299999999999995},"min":{"1min":0.53500000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.23,"5min":2.226,"15min":2.226},"last":0.872}]},{"osd":1,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.96899999999999997,"15min":0.995},"min":{"1min":0.621,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.5109999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":1.02},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.88,"15min":0.91700000000000004},"min":{"1min":0.53700000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.282,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.70399999999999996}]},{"osd":3,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.98499999999999999,"15min":1.026},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.4750000000000001,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.403},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0549999999999999,"15min":1.085},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3380000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.3540000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.2090000000000001,"15min":1.236},"min":{"1min":0.96399999999999997,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.488,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.179,"15min":1.214},"min":{"1min":0.66300000000000003,"5min":0.627,"15min":0.627},"max":{"1min":1.47,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.2390000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.3360000000000001,"15min":1.3759999999999999},"min":{"1min":0.76200000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.304,"15min":1.349},"min":{"1min":0.75,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.5880000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.819}]},{"osd":6,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.268,"15min":1.3400000000000001},"min":{"1min":0.82199999999999995,"5min":0.753,"15min":0.753},"max":{"1min":1.5109999999999999,"5min":2.456,"15min":2.456},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.359,"5min":1.4390000000000001,"15min":1.4890000000000001},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.5089999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.335,"15min":1.379},"min":{"1min":0.82699999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.6870000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.8660000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.3460000000000001,"15min":1.4099999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5900000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.119}]}]},{"osd":1,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.81599999999999995,"15min":0.87},"min":{"1min":0.40200000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.1819999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.82999999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.64000000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.92800000000000005,"15min":0.96599999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.4570000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.94899999999999995,"15min":0.98599999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.4750000000000001,"5min":1.625,"15min":1.625},"last":0.83599999999999997}]},{"osd":3,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.117,"15min":1.117},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.7929999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.99399999999999999,"15min":1.0189999999999999},"min":{"1min":0.57399999999999995,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.5600000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.68200000000000005}]},{"osd":4,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.232,"15min":1.292},"min":{"1min":0.69199999999999995,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.752,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.2749999999999999,"15min":1.3149999999999999},"min":{"1min":0.82599999999999996,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.575,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.01}]},{"osd":5,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.2729999999999999,"15min":1.329},"min":{"1min":0.82399999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.7470000000000001,"5min":3.593,"15min":3.593},"last":1.105},{"interface":"front","average":{"1min":1.276,"5min":1.3700000000000001,"15min":1.4399999999999999},"min":{"1min":0.76200000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6719999999999999,"5min":2.359,"15min":2.359},"last":1.671}]},{"osd":6,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.349,"15min":1.405},"min":{"1min":0.79300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.1709999999999998,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.484},{"interface":"front","average":{"1min":1.147,"5min":1.302,"15min":1.3580000000000001},"min":{"1min":0.77800000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.696,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.26}]},{"osd":7,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.3759999999999999,"15min":1.4379999999999999},"min":{"1min":0.81100000000000005,"5min":0.627,"15min":0.627},"max":{"1min":1.7649999999999999,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.7},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.4219999999999999,"15min":1.506},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.7569999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.5640000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28528,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343312,"statfs":{"total":96636764160,"available":96607551488,"internally_reserved":0,"allocated":1544192,"data_stored":1014931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.90000000000000002,"15min":0.94699999999999995},"min":{"1min":0.435,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.1599999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.98899999999999999},"min":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.3640000000000001,"5min":1.506,"15min":1.506},"last":0.90900000000000003}]},{"osd":2,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.95799999999999996,"15min":1.0009999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1279999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.91900000000000004,"5min":1.0640000000000001,"15min":1.105},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.4350000000000001,"5min":1.901,"15min":1.901},"last":1.2749999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.97499999999999998,"15min":1.01},"min":{"1min":0.48999999999999999,"5min":0.433,"15min":0.433},"max":{"1min":1.5189999999999999,"5min":1.972,"15min":1.972},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.103,"15min":1.159},"min":{"1min":0.51400000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3180000000000001,"5min":2.298,"15min":2.298},"last":1.018}]},{"osd":4,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.304,"15min":1.3420000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.599,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":1.05},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.252,"15min":1.28},"min":{"1min":0.83099999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.4990000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.23}]},{"osd":5,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.3340000000000001,"15min":1.3779999999999999},"min":{"1min":0.878,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.4450000000000001,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.169},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.403,"15min":1.4470000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.49,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.2030000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.46,"15min":1.538},"min":{"1min":0.85799999999999998,"5min":0.747,"15min":0.747},"max":{"1min":1.5940000000000001,"5min":3.032,"15min":3.032},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.28,"5min":1.4419999999999999,"15min":1.498},"min":{"1min":0.98199999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.5740000000000001,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.5069999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.46,"15min":1.5129999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.7110000000000001,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.3939999999999999,"15min":1.4299999999999999},"min":{"1min":0.871,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.008,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.423}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:50:12.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:50:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.580+0000 7f16c4c00640 1 Processor -- start 2024-03-23T03:50:12.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.580+0000 7f16c4c00640 1 -- start start 2024-03-23T03:50:12.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.582+0000 7f16c4c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f16c014aa40 0x7f16c00cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:12.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.582+0000 7f16c4c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f16c00cc7b0 0x7f16c00ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:12.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.582+0000 7f16c4c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16c006da70 con 0x7f16c00b27e0 2024-03-23T03:50:12.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.582+0000 7f16c4c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16c00b1d00 con 0x7f16c00cc7b0 2024-03-23T03:50:12.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.582+0000 7f16c4c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c00b1fe0 con 0x7f16c014aa40 2024-03-23T03:50:12.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.582+0000 7f16bd400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f16c00cc7b0 0x7f16c00ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:12.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.582+0000 7f16bde00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f16c014aa40 0x7f16c00cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:12.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.583+0000 7f16bd400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f16c00cc7b0 0x7f16c00ccc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:35530/0 (socket says 172.21.15.102:35530) 2024-03-23T03:50:12.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.583+0000 7f16bd400640 1 -- 172.21.15.102:0/2233481415 learned_addr learned my addr 172.21.15.102:0/2233481415 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:12.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.583+0000 7f16bd400640 1 -- 172.21.15.102:0/2233481415 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f16c014aa40 msgr2=0x7f16c00cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:12.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.583+0000 7f16bd400640 1 --2- 172.21.15.102:0/2233481415 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f16c014aa40 0x7f16c00cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.583+0000 7f16bd400640 1 -- 172.21.15.102:0/2233481415 >> v1:172.21.15.173:6789/0 conn(0x7f16c00b27e0 legacy=0x7f16c00b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:12.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.583+0000 7f16bd400640 1 -- 172.21.15.102:0/2233481415 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16c00b22c0 con 0x7f16c00cc7b0 2024-03-23T03:50:12.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.585+0000 7f16bd400640 1 --2- 172.21.15.102:0/2233481415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16c00cc7b0 0x7f16c00ccc30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f16ac009870 tx=0x7f16ac02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65cb506a76ed6ef2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:12.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.585+0000 7f16bca00640 1 -- 172.21.15.102:0/2233481415 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac030840 con 0x7f16c00cc7b0 2024-03-23T03:50:12.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.585+0000 7f16bca00640 1 -- 172.21.15.102:0/2233481415 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16ac0309a0 con 0x7f16c00cc7b0 2024-03-23T03:50:12.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.585+0000 7f16bca00640 1 -- 172.21.15.102:0/2233481415 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16ac030d30 con 0x7f16c00cc7b0 2024-03-23T03:50:12.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.586+0000 7f16c4c00640 1 -- 172.21.15.102:0/2233481415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16c00cc7b0 msgr2=0x7f16c00ccc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:12.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.586+0000 7f16c4c00640 1 --2- 172.21.15.102:0/2233481415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16c00cc7b0 0x7f16c00ccc30 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f16ac009870 tx=0x7f16ac02b980 comp rx=0 tx=0).stop 2024-03-23T03:50:12.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.586+0000 7f16c4c00640 1 -- 172.21.15.102:0/2233481415 shutdown_connections 2024-03-23T03:50:12.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.586+0000 7f16c4c00640 1 --2- 172.21.15.102:0/2233481415 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16c00cc7b0 0x7f16c00ccc30 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.586+0000 7f16c4c00640 1 --2- 172.21.15.102:0/2233481415 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f16c014aa40 0x7f16c00cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.586+0000 7f16c4c00640 1 -- 172.21.15.102:0/2233481415 >> 172.21.15.102:0/2233481415 conn(0x7f16c006a3d0 msgr2=0x7f16c00b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:12.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.586+0000 7f16c4c00640 1 -- 172.21.15.102:0/2233481415 shutdown_connections 2024-03-23T03:50:12.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.586+0000 7f16c4c00640 1 -- 172.21.15.102:0/2233481415 wait complete. 2024-03-23T03:50:12.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.588+0000 7f16c4c00640 1 Processor -- start 2024-03-23T03:50:12.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.588+0000 7f16c4c00640 1 -- start start 2024-03-23T03:50:12.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.589+0000 7f16c4c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f16c00b27e0 0x7f16c0160410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:12.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.589+0000 7f16c4c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16c00cc7b0 0x7f16c00ac780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:12.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.589+0000 7f16c4c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16c014aa40 0x7f16c00accc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:12.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.590+0000 7f16c4c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c00ad200 con 0x7f16c014aa40 2024-03-23T03:50:12.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.590+0000 7f16c4c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16c0163450 con 0x7f16c00cc7b0 2024-03-23T03:50:12.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.590+0000 7f16c4c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16c0163730 con 0x7f16c00b27e0 2024-03-23T03:50:12.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.590+0000 7f16be800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16c014aa40 0x7f16c00accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:12.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.590+0000 7f16be800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16c014aa40 0x7f16c00accc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:59976/0 (socket says 172.21.15.102:59976) 2024-03-23T03:50:12.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.590+0000 7f16be800640 1 -- 172.21.15.102:0/4050229709 learned_addr learned my addr 172.21.15.102:0/4050229709 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:12.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.590+0000 7f16bde00640 1 --2- 172.21.15.102:0/4050229709 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f16c00b27e0 0x7f16c0160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:12.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.590+0000 7f16bd400640 1 --2- 172.21.15.102:0/4050229709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16c00cc7b0 0x7f16c00ac780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:12.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.590+0000 7f16be800640 1 -- 172.21.15.102:0/4050229709 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f16c00b27e0 msgr2=0x7f16c0160410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:12.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.590+0000 7f16be800640 1 --2- 172.21.15.102:0/4050229709 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f16c00b27e0 0x7f16c0160410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.590+0000 7f16be800640 1 -- 172.21.15.102:0/4050229709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16c00cc7b0 msgr2=0x7f16c00ac780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:12.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.590+0000 7f16be800640 1 --2- 172.21.15.102:0/4050229709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16c00cc7b0 0x7f16c00ac780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.590+0000 7f16be800640 1 -- 172.21.15.102:0/4050229709 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac0094d0 con 0x7f16c014aa40 2024-03-23T03:50:12.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.591+0000 7f16be800640 1 --2- 172.21.15.102:0/4050229709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16c014aa40 0x7f16c00accc0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f16b800c6b0 tx=0x7f16b800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:12.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.591+0000 7f16b6000640 1 -- 172.21.15.102:0/4050229709 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8003b70 con 0x7f16c014aa40 2024-03-23T03:50:12.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.591+0000 7f16b6000640 1 -- 172.21.15.102:0/4050229709 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b8003cf0 con 0x7f16c014aa40 2024-03-23T03:50:12.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.591+0000 7f16b6000640 1 -- 172.21.15.102:0/4050229709 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8003ea0 con 0x7f16c014aa40 2024-03-23T03:50:12.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.591+0000 7f16c4c00640 1 -- 172.21.15.102:0/4050229709 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16c0163a10 con 0x7f16c014aa40 2024-03-23T03:50:12.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.591+0000 7f16c4c00640 1 -- 172.21.15.102:0/4050229709 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16c0160950 con 0x7f16c014aa40 2024-03-23T03:50:12.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.592+0000 7f16c4c00640 1 -- 172.21.15.102:0/4050229709 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16c01f78d0 con 0x7f16c014aa40 2024-03-23T03:50:12.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.593+0000 7f16b6000640 1 -- 172.21.15.102:0/4050229709 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f16b8005310 con 0x7f16c014aa40 2024-03-23T03:50:12.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.594+0000 7f16b6000640 1 --2- 172.21.15.102:0/4050229709 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f16a006ac00 0x7f16a006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:12.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.594+0000 7f16b6000640 1 -- 172.21.15.102:0/4050229709 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f16b8089270 con 0x7f16c014aa40 2024-03-23T03:50:12.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.594+0000 7f16bde00640 1 --2- 172.21.15.102:0/4050229709 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f16a006ac00 0x7f16a006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:12.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.595+0000 7f16bde00640 1 --2- 172.21.15.102:0/4050229709 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f16a006ac00 0x7f16a006d0c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009ad0 tx=0x7f16b0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:12.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.600+0000 7f16b6000640 1 -- 172.21.15.102:0/4050229709 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f16b8054300 con 0x7f16c014aa40 2024-03-23T03:50:12.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.783+0000 7f16c4c00640 1 -- 172.21.15.102:0/4050229709 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f16c0073390 con 0x7f16a006ac00 2024-03-23T03:50:12.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.791+0000 7f16b6000640 1 -- 172.21.15.102:0/4050229709 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+276359 (secure 0 0 0) 0x7f16c0073390 con 0x7f16a006ac00 2024-03-23T03:50:12.794 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:12.797 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:50:12.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.797+0000 7f16c4c00640 1 -- 172.21.15.102:0/4050229709 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f16a006ac00 msgr2=0x7f16a006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:12.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.797+0000 7f16c4c00640 1 --2- 172.21.15.102:0/4050229709 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f16a006ac00 0x7f16a006d0c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009ad0 tx=0x7f16b0008040 comp rx=0 tx=0).stop 2024-03-23T03:50:12.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.797+0000 7f16c4c00640 1 -- 172.21.15.102:0/4050229709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16c014aa40 msgr2=0x7f16c00accc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:12.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.797+0000 7f16c4c00640 1 --2- 172.21.15.102:0/4050229709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16c014aa40 0x7f16c00accc0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f16b800c6b0 tx=0x7f16b800cb70 comp rx=0 tx=0).stop 2024-03-23T03:50:12.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.797+0000 7f16c4c00640 1 -- 172.21.15.102:0/4050229709 shutdown_connections 2024-03-23T03:50:12.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.797+0000 7f16c4c00640 1 --2- 172.21.15.102:0/4050229709 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f16a006ac00 0x7f16a006d0c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.797+0000 7f16c4c00640 1 --2- 172.21.15.102:0/4050229709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16c014aa40 0x7f16c00accc0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.797+0000 7f16c4c00640 1 --2- 172.21.15.102:0/4050229709 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16c00cc7b0 0x7f16c00ac780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.797+0000 7f16c4c00640 1 --2- 172.21.15.102:0/4050229709 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f16c00b27e0 0x7f16c0160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:12.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.797+0000 7f16c4c00640 1 -- 172.21.15.102:0/4050229709 >> 172.21.15.102:0/4050229709 conn(0x7f16c006a3d0 msgr2=0x7f16c00b2c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:12.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.798+0000 7f16c4c00640 1 -- 172.21.15.102:0/4050229709 shutdown_connections 2024-03-23T03:50:12.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:12.798+0000 7f16c4c00640 1 -- 172.21.15.102:0/4050229709 wait complete. 2024-03-23T03:50:12.815 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-03-23T03:50:11.778281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":228756,"kb_used_data":8500,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754745964,"statfs":{"total":773094113280,"available":772859867136,"internally_reserved":0,"allocated":8704000,"data_stored":4534281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":225431098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.333625"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450165+0000","last_change":"2024-03-23T03:50:06.450165+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450165+0000","last_undegraded":"2024-03-23T03:50:06.450165+0000","last_fullsized":"2024-03-23T03:50:06.450165+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:32:26.890806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.461707+0000","last_change":"2024-03-23T03:50:06.461707+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.461707+0000","last_undegraded":"2024-03-23T03:50:06.461707+0000","last_fullsized":"2024-03-23T03:50:06.461707+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:14:39.947258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.461900+0000","last_change":"2024-03-23T03:50:06.461900+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.461900+0000","last_undegraded":"2024-03-23T03:50:06.461900+0000","last_fullsized":"2024-03-23T03:50:06.461900+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:03:17.838071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.463303+0000","last_change":"2024-03-23T03:50:06.463303+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.463303+0000","last_undegraded":"2024-03-23T03:50:06.463303+0000","last_fullsized":"2024-03-23T03:50:06.463303+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:33:34.230046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.455179+0000","last_change":"2024-03-23T03:50:06.455179+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.455179+0000","last_undegraded":"2024-03-23T03:50:06.455179+0000","last_fullsized":"2024-03-23T03:50:06.455179+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:33:34.230046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.469417+0000","last_change":"2024-03-23T03:50:06.469417+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.469417+0000","last_undegraded":"2024-03-23T03:50:06.469417+0000","last_fullsized":"2024-03-23T03:50:06.469417+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:36:00.228119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.467620+0000","last_change":"2024-03-23T03:50:06.467620+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.467620+0000","last_undegraded":"2024-03-23T03:50:06.467620+0000","last_fullsized":"2024-03-23T03:50:06.467620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:06:01.399011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450165+0000","last_change":"2024-03-23T03:50:06.450165+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450165+0000","last_undegraded":"2024-03-23T03:50:06.450165+0000","last_fullsized":"2024-03-23T03:50:06.450165+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:59:56.498434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.462227+0000","last_change":"2024-03-23T03:50:06.462227+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.462227+0000","last_undegraded":"2024-03-23T03:50:06.462227+0000","last_fullsized":"2024-03-23T03:50:06.462227+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:44:27.821644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.469099+0000","last_change":"2024-03-23T03:50:06.469099+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.469099+0000","last_undegraded":"2024-03-23T03:50:06.469099+0000","last_fullsized":"2024-03-23T03:50:06.469099+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:36:57.803871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450817+0000","last_change":"2024-03-23T03:50:06.450817+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450817+0000","last_undegraded":"2024-03-23T03:50:06.450817+0000","last_fullsized":"2024-03-23T03:50:06.450817+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:18:47.365643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.462222+0000","last_change":"2024-03-23T03:50:06.462222+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.462222+0000","last_undegraded":"2024-03-23T03:50:06.462222+0000","last_fullsized":"2024-03-23T03:50:06.462222+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:48:07.848353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.470833+0000","last_change":"2024-03-23T03:50:06.470833+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.470833+0000","last_undegraded":"2024-03-23T03:50:06.470833+0000","last_fullsized":"2024-03-23T03:50:06.470833+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:36:15.054606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.460079+0000","last_change":"2024-03-23T03:50:06.460079+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.460079+0000","last_undegraded":"2024-03-23T03:50:06.460079+0000","last_fullsized":"2024-03-23T03:50:06.460079+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:28:24.935685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450491+0000","last_change":"2024-03-23T03:50:06.450491+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450491+0000","last_undegraded":"2024-03-23T03:50:06.450491+0000","last_fullsized":"2024-03-23T03:50:06.450491+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:24:41.782032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.467435+0000","last_change":"2024-03-23T03:50:06.467435+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.467435+0000","last_undegraded":"2024-03-23T03:50:06.467435+0000","last_fullsized":"2024-03-23T03:50:06.467435+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:49:00.654026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.460903+0000","last_change":"2024-03-23T03:50:06.460903+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.460903+0000","last_undegraded":"2024-03-23T03:50:06.460903+0000","last_fullsized":"2024-03-23T03:50:06.460903+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:59:01.344309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.455371+0000","last_change":"2024-03-23T03:50:06.455371+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.455371+0000","last_undegraded":"2024-03-23T03:50:06.455371+0000","last_fullsized":"2024-03-23T03:50:06.455371+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:19:21.548516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450380+0000","last_change":"2024-03-23T03:50:06.450380+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450380+0000","last_undegraded":"2024-03-23T03:50:06.450380+0000","last_fullsized":"2024-03-23T03:50:06.450380+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:54:59.757010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.451352+0000","last_change":"2024-03-23T03:50:06.451352+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.451352+0000","last_undegraded":"2024-03-23T03:50:06.451352+0000","last_fullsized":"2024-03-23T03:50:06.451352+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:03:17.838071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.455710+0000","last_change":"2024-03-23T03:50:06.455710+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.455710+0000","last_undegraded":"2024-03-23T03:50:06.455710+0000","last_fullsized":"2024-03-23T03:50:06.455710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:41:32.913577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.456982+0000","last_change":"2024-03-23T03:50:06.456982+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.456982+0000","last_undegraded":"2024-03-23T03:50:06.456982+0000","last_fullsized":"2024-03-23T03:50:06.456982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:46:17.453220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.455224+0000","last_change":"2024-03-23T03:50:06.455224+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.455224+0000","last_undegraded":"2024-03-23T03:50:06.455224+0000","last_fullsized":"2024-03-23T03:50:06.455224+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:54:23.987953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.469854+0000","last_change":"2024-03-23T03:50:06.469854+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.469854+0000","last_undegraded":"2024-03-23T03:50:06.469854+0000","last_fullsized":"2024-03-23T03:50:06.469854+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:14:48.590038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.467198+0000","last_change":"2024-03-23T03:50:06.467198+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.467198+0000","last_undegraded":"2024-03-23T03:50:06.467198+0000","last_fullsized":"2024-03-23T03:50:06.467198+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:51:13.757234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.460543+0000","last_change":"2024-03-23T03:50:06.460543+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.460543+0000","last_undegraded":"2024-03-23T03:50:06.460543+0000","last_fullsized":"2024-03-23T03:50:06.460543+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:01:47.992165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.460249+0000","last_change":"2024-03-23T03:50:06.460249+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.460249+0000","last_undegraded":"2024-03-23T03:50:06.460249+0000","last_fullsized":"2024-03-23T03:50:06.460249+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:58.116627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.461287+0000","last_change":"2024-03-23T03:50:06.461287+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.461287+0000","last_undegraded":"2024-03-23T03:50:06.461287+0000","last_fullsized":"2024-03-23T03:50:06.461287+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:19:21.548516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450250+0000","last_change":"2024-03-23T03:50:06.450250+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450250+0000","last_undegraded":"2024-03-23T03:50:06.450250+0000","last_fullsized":"2024-03-23T03:50:06.450250+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:36:27.444641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.460712+0000","last_change":"2024-03-23T03:50:06.460712+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.460712+0000","last_undegraded":"2024-03-23T03:50:06.460712+0000","last_fullsized":"2024-03-23T03:50:06.460712+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:13.215153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450710+0000","last_change":"2024-03-23T03:50:06.450710+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450710+0000","last_undegraded":"2024-03-23T03:50:06.450710+0000","last_fullsized":"2024-03-23T03:50:06.450710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:01:47.992165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.455254+0000","last_change":"2024-03-23T03:50:06.455254+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.455254+0000","last_undegraded":"2024-03-23T03:50:06.455254+0000","last_fullsized":"2024-03-23T03:50:06.455254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:11:43.166129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":210,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-23T03:50:04.652817+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:50:04.652817+0000","last_peered":"2024-03-23T03:50:04.652817+0000","last_clean":"2024-03-23T03:50:04.652817+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:50:04.652817+0000","last_undegraded":"2024-03-23T03:50:04.652817+0000","last_fullsized":"2024-03-23T03:50:04.652817+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542188,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28516,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343324,"statfs":{"total":96636764160,"available":96607563776,"internally_reserved":0,"allocated":1531904,"data_stored":1008794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.079,"15min":1.069},"min":{"1min":0.68200000000000005,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.573,"5min":1.635,"15min":1.635},"last":1.075},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0449999999999999,"15min":1.048},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.417,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.353}]},{"osd":1,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0549999999999999,"15min":1.044},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.544,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.466},{"interface":"front","average":{"1min":1.085,"5min":1.2330000000000001,"15min":1.3009999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.3999999999999999,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":0.90600000000000003}]},{"osd":2,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.139,"15min":1.1559999999999999},"min":{"1min":0.76400000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.7909999999999999,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.24},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.345,"15min":1.421},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.4630000000000001,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":0.57799999999999996}]},{"osd":3,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.355,"15min":1.4370000000000001},"min":{"1min":0.873,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.397,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.3380000000000001,"15min":1.3580000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":1.5840000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.6359999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.266,"15min":1.2669999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.6299999999999999,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.8220000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.357,"15min":1.375},"min":{"1min":0.77900000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0619999999999998,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.5189999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.3999999999999999,"15min":1.4219999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.6539999999999999,"5min":2.645,"15min":2.645},"last":1.278},{"interface":"front","average":{"1min":1.339,"5min":1.4370000000000001,"15min":1.4530000000000001},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":1.7050000000000001,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.696}]},{"osd":6,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.5229999999999999,"15min":1.5589999999999999},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":1.8200000000000001,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.9279999999999999},{"interface":"front","average":{"1min":1.415,"5min":1.4930000000000001,"15min":1.5209999999999999},"min":{"1min":1.1319999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":2.1509999999999998,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.734}]}]},{"osd":6,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.0309999999999999,"15min":1.04},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.3979999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0429999999999999,"15min":1.038},"min":{"1min":0.72099999999999997,"5min":0.62,"15min":0.62},"max":{"1min":1.6120000000000001,"5min":1.946,"15min":1.946},"last":0.66100000000000003}]},{"osd":1,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.874},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.129,"15min":1.1559999999999999},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.4810000000000001,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.80500000000000005}]},{"osd":2,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.075,"15min":1.069},"min":{"1min":0.73699999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.5449999999999999,"5min":2.044,"15min":2.044},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1950000000000001,"15min":1.218},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.96899999999999997}]},{"osd":3,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2629999999999999,"15min":1.2849999999999999},"min":{"1min":0.76800000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.4910000000000001,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2549999999999999,"15min":1.2649999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.6319999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.94199999999999995}]},{"osd":4,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3759999999999999,"15min":1.3839999999999999},"min":{"1min":0.67700000000000005,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.73,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.363,"15min":1.383},"min":{"1min":0.63400000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.762,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.448}]},{"osd":5,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.415,"15min":1.4359999999999999},"min":{"1min":0.71099999999999997,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.6919999999999999,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.456,"5min":1.5249999999999999,"15min":1.5469999999999999},"min":{"1min":0.79700000000000004,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8149999999999999,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.091}]},{"osd":7,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.4059999999999999,"15min":1.4119999999999999},"min":{"1min":0.77600000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.802,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.5129999999999999,"15min":1.5409999999999999},"min":{"1min":0.84099999999999997,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.74,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.254}]}]},{"osd":5,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.1759999999999999,"15min":1.214},"min":{"1min":0.67900000000000005,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.6779999999999999,"5min":1.806,"15min":1.806},"last":1.2},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.157,"15min":1.181},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.554,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.90800000000000003}]},{"osd":1,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.81699999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.5489999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.026,"5min":1.1599999999999999,"15min":1.1919999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4219999999999999,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.0489999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.298,"15min":1.3540000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5429999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3340000000000001,"15min":1.3779999999999999},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.772,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.0840000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.21,"15min":1.2470000000000001},"min":{"1min":0.70599999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.653,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.393,"15min":1.452},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.6779999999999999,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.137}]},{"osd":4,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.3420000000000001,"15min":1.4119999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.671,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.282,"5min":1.4119999999999999,"15min":1.444},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.77,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.2270000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.458,"15min":1.514},"min":{"1min":0.76700000000000002,"5min":0.63,"15min":0.63},"max":{"1min":1.873,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.5940000000000001,"15min":1.6559999999999999},"min":{"1min":0.86199999999999999,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.956,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.397}]},{"osd":7,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.538,"15min":1.607},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.673,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.411,"5min":1.6200000000000001,"15min":1.696},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.855,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.6100000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.002,"15min":0.97999999999999998},"min":{"1min":0.67800000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.0149999999999999,"15min":1.012},"min":{"1min":0.81200000000000006,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3069999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.71399999999999997}]},{"osd":1,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.0700000000000001,"15min":1.05},"min":{"1min":0.74299999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.704,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.147,"5min":1.0880000000000001,"15min":1.0640000000000001},"min":{"1min":0.50600000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.181}]},{"osd":2,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2949999999999999,"15min":1.325},"min":{"1min":0.81599999999999995,"5min":0.625,"15min":0.625},"max":{"1min":1.569,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.29,"5min":1.3100000000000001,"15min":1.3080000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.8680000000000001,"5min":2.181,"15min":2.181},"last":1.2869999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1339999999999999,"15min":1.1299999999999999},"min":{"1min":0.73199999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.26,"15min":1.278},"min":{"1min":0.82899999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.5720000000000001,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":0.91600000000000004}]},{"osd":5,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.278,"15min":1.274},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.298,"15min":1.2829999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.2349999999999999,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.228}]},{"osd":6,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.415,"15min":1.4239999999999999},"min":{"1min":1.143,"5min":0.748,"15min":0.748},"max":{"1min":1.833,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.425,"15min":1.4039999999999999},"min":{"1min":0.745,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":1.377}]},{"osd":7,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.385,"15min":1.371},"min":{"1min":1.0880000000000001,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":2.3479999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.4970000000000001,"15min":1.5009999999999999},"min":{"1min":1.056,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.974,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.25}]}]},{"osd":3,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.83899999999999997,"15min":0.86099999999999999},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.256,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.446,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":1.4610000000000001}]},{"osd":1,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.92800000000000005,"15min":0.97199999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.997,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.034,"15min":1.0720000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.712,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.90400000000000003}]},{"osd":2,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.006,"15min":1.052},"min":{"1min":0.435,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.613,"5min":1.823,"15min":1.823},"last":1.109},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.1319999999999999,"15min":1.153},"min":{"1min":0.48199999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.6020000000000001,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":1.3700000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.2310000000000001,"15min":1.2450000000000001},"min":{"1min":0.67600000000000005,"5min":0.625,"15min":0.625},"max":{"1min":1.3859999999999999,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":1.054},{"interface":"front","average":{"1min":1.151,"5min":1.377,"15min":1.4370000000000001},"min":{"1min":0.77100000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.4770000000000001,"5min":4.819,"15min":4.819},"last":1.4850000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.603,"15min":1.4470000000000001},"min":{"1min":0.75,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.0299999999999998,"5min":30.228000000000002,"15min":30.228000000000002},"last":1.165},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.3320000000000001,"15min":1.355},"min":{"1min":0.91400000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.637,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":1.2709999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.448,"15min":1.52},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.5860000000000001,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.206},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3999999999999999,"15min":1.397},"min":{"1min":0.90100000000000002,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.1429999999999998,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":1.5}]},{"osd":7,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.395,"15min":1.4219999999999999},"min":{"1min":0.81200000000000006,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.71,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.577,"15min":1.601},"min":{"1min":0.73299999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.629,"5min":10.663,"15min":10.663},"last":1.258}]}]},{"osd":2,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.84999999999999998,"15min":0.88100000000000001},"min":{"1min":0.318,"5min":0.307,"15min":0.307},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.1930000000000001},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.88700000000000001,"15min":0.94299999999999995},"min":{"1min":0.53500000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.23,"5min":2.226,"15min":2.226},"last":0.872}]},{"osd":1,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.96899999999999997,"15min":0.995},"min":{"1min":0.621,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.5109999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":1.02},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.88,"15min":0.91700000000000004},"min":{"1min":0.53700000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.282,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.70399999999999996}]},{"osd":3,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.98499999999999999,"15min":1.026},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.4750000000000001,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.403},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0549999999999999,"15min":1.085},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3380000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.3540000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.2090000000000001,"15min":1.236},"min":{"1min":0.96399999999999997,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.488,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.179,"15min":1.214},"min":{"1min":0.66300000000000003,"5min":0.627,"15min":0.627},"max":{"1min":1.47,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.2390000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.3360000000000001,"15min":1.3759999999999999},"min":{"1min":0.76200000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.304,"15min":1.349},"min":{"1min":0.75,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.5880000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.819}]},{"osd":6,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.268,"15min":1.3400000000000001},"min":{"1min":0.82199999999999995,"5min":0.753,"15min":0.753},"max":{"1min":1.5109999999999999,"5min":2.456,"15min":2.456},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.359,"5min":1.4390000000000001,"15min":1.4890000000000001},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.5089999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.335,"15min":1.379},"min":{"1min":0.82699999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.6870000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.8660000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.3460000000000001,"15min":1.4099999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5900000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.119}]}]},{"osd":1,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.81599999999999995,"15min":0.87},"min":{"1min":0.40200000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.1819999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.82999999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.64000000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.92800000000000005,"15min":0.96599999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.4570000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.94899999999999995,"15min":0.98599999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.4750000000000001,"5min":1.625,"15min":1.625},"last":0.83599999999999997}]},{"osd":3,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.117,"15min":1.117},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.7929999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.99399999999999999,"15min":1.0189999999999999},"min":{"1min":0.57399999999999995,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.5600000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.68200000000000005}]},{"osd":4,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.232,"15min":1.292},"min":{"1min":0.69199999999999995,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.752,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.2749999999999999,"15min":1.3149999999999999},"min":{"1min":0.82599999999999996,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.575,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.01}]},{"osd":5,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.2729999999999999,"15min":1.329},"min":{"1min":0.82399999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.7470000000000001,"5min":3.593,"15min":3.593},"last":1.105},{"interface":"front","average":{"1min":1.276,"5min":1.3700000000000001,"15min":1.4399999999999999},"min":{"1min":0.76200000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6719999999999999,"5min":2.359,"15min":2.359},"last":1.671}]},{"osd":6,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.349,"15min":1.405},"min":{"1min":0.79300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.1709999999999998,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.484},{"interface":"front","average":{"1min":1.147,"5min":1.302,"15min":1.3580000000000001},"min":{"1min":0.77800000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.696,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.26}]},{"osd":7,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.3759999999999999,"15min":1.4379999999999999},"min":{"1min":0.81100000000000005,"5min":0.627,"15min":0.627},"max":{"1min":1.7649999999999999,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.7},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.4219999999999999,"15min":1.506},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.7569999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.5640000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28528,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343312,"statfs":{"total":96636764160,"available":96607551488,"internally_reserved":0,"allocated":1544192,"data_stored":1014931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.90000000000000002,"15min":0.94699999999999995},"min":{"1min":0.435,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.1599999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.98899999999999999},"min":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.3640000000000001,"5min":1.506,"15min":1.506},"last":0.90900000000000003}]},{"osd":2,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.95799999999999996,"15min":1.0009999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1279999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.91900000000000004,"5min":1.0640000000000001,"15min":1.105},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.4350000000000001,"5min":1.901,"15min":1.901},"last":1.2749999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.97499999999999998,"15min":1.01},"min":{"1min":0.48999999999999999,"5min":0.433,"15min":0.433},"max":{"1min":1.5189999999999999,"5min":1.972,"15min":1.972},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.103,"15min":1.159},"min":{"1min":0.51400000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3180000000000001,"5min":2.298,"15min":2.298},"last":1.018}]},{"osd":4,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.304,"15min":1.3420000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.599,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":1.05},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.252,"15min":1.28},"min":{"1min":0.83099999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.4990000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.23}]},{"osd":5,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.3340000000000001,"15min":1.3779999999999999},"min":{"1min":0.878,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.4450000000000001,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.169},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.403,"15min":1.4470000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.49,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.2030000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.46,"15min":1.538},"min":{"1min":0.85799999999999998,"5min":0.747,"15min":0.747},"max":{"1min":1.5940000000000001,"5min":3.032,"15min":3.032},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.28,"5min":1.4419999999999999,"15min":1.498},"min":{"1min":0.98199999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.5740000000000001,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.5069999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.46,"15min":1.5129999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.7110000000000001,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.3939999999999999,"15min":1.4299999999999999},"min":{"1min":0.871,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.008,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.423}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:50:12.820 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:50:13.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.067+0000 7fc839000640 1 Processor -- start 2024-03-23T03:50:13.068 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.067+0000 7fc839000640 1 -- start start 2024-03-23T03:50:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.068+0000 7fc839000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc83415cc00 0x7fc8340d03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.068+0000 7fc839000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc83415cfe0 0x7fc8340d0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.068+0000 7fc839000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc83406ba00 con 0x7fc8340caf40 2024-03-23T03:50:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.068+0000 7fc839000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8340d0e40 con 0x7fc83415cfe0 2024-03-23T03:50:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.068+0000 7fc839000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc83406b3d0 con 0x7fc83415cc00 2024-03-23T03:50:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.069+0000 7fc831400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc83415cfe0 0x7fc8340d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.069+0000 7fc831400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc83415cfe0 0x7fc8340d0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38698/0 (socket says 172.21.15.102:38698) 2024-03-23T03:50:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.069+0000 7fc831400640 1 -- 172.21.15.102:0/3704319232 learned_addr learned my addr 172.21.15.102:0/3704319232 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.069+0000 7fc831e00640 1 --2- 172.21.15.102:0/3704319232 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc83415cc00 0x7fc8340d03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.069+0000 7fc831400640 1 -- 172.21.15.102:0/3704319232 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc83415cc00 msgr2=0x7fc8340d03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.069+0000 7fc831400640 1 --2- 172.21.15.102:0/3704319232 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc83415cc00 0x7fc8340d03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.069+0000 7fc831400640 1 -- 172.21.15.102:0/3704319232 >> v1:172.21.15.173:6789/0 conn(0x7fc8340caf40 legacy=0x7fc8340cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:13.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.069+0000 7fc831400640 1 -- 172.21.15.102:0/3704319232 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8340cdbc0 con 0x7fc83415cfe0 2024-03-23T03:50:13.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.070+0000 7fc831400640 1 --2- 172.21.15.102:0/3704319232 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc83415cfe0 0x7fc8340d0900 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009e50 tx=0x7fc81c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c31f92680f0ee2b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:13.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.070+0000 7fc830a00640 1 -- 172.21.15.102:0/3704319232 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81c02ca80 con 0x7fc83415cfe0 2024-03-23T03:50:13.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.071+0000 7fc830a00640 1 -- 172.21.15.102:0/3704319232 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81c02cbe0 con 0x7fc83415cfe0 2024-03-23T03:50:13.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.071+0000 7fc830a00640 1 -- 172.21.15.102:0/3704319232 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81c0343f0 con 0x7fc83415cfe0 2024-03-23T03:50:13.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.071+0000 7fc839000640 1 -- 172.21.15.102:0/3704319232 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc83415cfe0 msgr2=0x7fc8340d0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:13.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.071+0000 7fc839000640 1 --2- 172.21.15.102:0/3704319232 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc83415cfe0 0x7fc8340d0900 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fc81c009e50 tx=0x7fc81c02bee0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.071+0000 7fc839000640 1 -- 172.21.15.102:0/3704319232 shutdown_connections 2024-03-23T03:50:13.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.071+0000 7fc839000640 1 --2- 172.21.15.102:0/3704319232 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc83415cfe0 0x7fc8340d0900 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.071+0000 7fc839000640 1 --2- 172.21.15.102:0/3704319232 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc83415cc00 0x7fc8340d03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.071+0000 7fc839000640 1 -- 172.21.15.102:0/3704319232 >> 172.21.15.102:0/3704319232 conn(0x7fc83406e580 msgr2=0x7fc83406e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:13.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.071+0000 7fc839000640 1 -- 172.21.15.102:0/3704319232 shutdown_connections 2024-03-23T03:50:13.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.071+0000 7fc839000640 1 -- 172.21.15.102:0/3704319232 wait complete. 2024-03-23T03:50:13.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.073+0000 7fc839000640 1 Processor -- start 2024-03-23T03:50:13.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.073+0000 7fc839000640 1 -- start start 2024-03-23T03:50:13.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.074+0000 7fc839000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8340caf40 0x7fc83414ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:13.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.074+0000 7fc839000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc83415cc00 0x7fc8340c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:13.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.074+0000 7fc839000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc83414b370 0x7fc8340c4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:13.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.074+0000 7fc839000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8340c8b00 con 0x7fc8340caf40 2024-03-23T03:50:13.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.074+0000 7fc839000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8340c93a0 con 0x7fc83415cc00 2024-03-23T03:50:13.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.074+0000 7fc839000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8340c4810 con 0x7fc83414b370 2024-03-23T03:50:13.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.074+0000 7fc831e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8340caf40 0x7fc83414ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:13.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.074+0000 7fc831e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8340caf40 0x7fc83414ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38706/0 (socket says 172.21.15.102:38706) 2024-03-23T03:50:13.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.074+0000 7fc831e00640 1 -- 172.21.15.102:0/2639306941 learned_addr learned my addr 172.21.15.102:0/2639306941 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:13.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.075+0000 7fc831400640 1 --2- 172.21.15.102:0/2639306941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc83415cc00 0x7fc8340c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:13.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.075+0000 7fc832800640 1 --2- 172.21.15.102:0/2639306941 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc83414b370 0x7fc8340c4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:13.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.075+0000 7fc831e00640 1 -- 172.21.15.102:0/2639306941 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc83414b370 msgr2=0x7fc8340c4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:13.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.075+0000 7fc831e00640 1 --2- 172.21.15.102:0/2639306941 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc83414b370 0x7fc8340c4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.075+0000 7fc831e00640 1 -- 172.21.15.102:0/2639306941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc83415cc00 msgr2=0x7fc8340c42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:13.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.075+0000 7fc831e00640 1 --2- 172.21.15.102:0/2639306941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc83415cc00 0x7fc8340c42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.075+0000 7fc831e00640 1 -- 172.21.15.102:0/2639306941 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81c009b30 con 0x7fc8340caf40 2024-03-23T03:50:13.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.075+0000 7fc831400640 1 --2- 172.21.15.102:0/2639306941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc83415cc00 0x7fc8340c42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:13.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.075+0000 7fc832800640 1 --2- 172.21.15.102:0/2639306941 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc83414b370 0x7fc8340c4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:13.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.075+0000 7fc831e00640 1 --2- 172.21.15.102:0/2639306941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8340caf40 0x7fc83414ac30 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fc82400c850 tx=0x7fc82400cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:13.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.075+0000 7fc82a000640 1 -- 172.21.15.102:0/2639306941 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc824004020 con 0x7fc8340caf40 2024-03-23T03:50:13.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.075+0000 7fc82a000640 1 -- 172.21.15.102:0/2639306941 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8240041a0 con 0x7fc8340caf40 2024-03-23T03:50:13.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.075+0000 7fc839000640 1 -- 172.21.15.102:0/2639306941 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8340c73e0 con 0x7fc8340caf40 2024-03-23T03:50:13.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.075+0000 7fc82a000640 1 -- 172.21.15.102:0/2639306941 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8240077d0 con 0x7fc8340caf40 2024-03-23T03:50:13.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.076+0000 7fc839000640 1 -- 172.21.15.102:0/2639306941 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8340c9030 con 0x7fc8340caf40 2024-03-23T03:50:13.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.077+0000 7fc839000640 1 -- 172.21.15.102:0/2639306941 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc834067750 con 0x7fc8340caf40 2024-03-23T03:50:13.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.077+0000 7fc82a000640 1 -- 172.21.15.102:0/2639306941 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc824007930 con 0x7fc8340caf40 2024-03-23T03:50:13.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.078+0000 7fc82a000640 1 --2- 172.21.15.102:0/2639306941 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc81006ac30 0x7fc81006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:13.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.078+0000 7fc82a000640 1 -- 172.21.15.102:0/2639306941 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc8240892f0 con 0x7fc8340caf40 2024-03-23T03:50:13.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.078+0000 7fc831400640 1 --2- 172.21.15.102:0/2639306941 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc81006ac30 0x7fc81006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:13.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.079+0000 7fc831400640 1 --2- 172.21.15.102:0/2639306941 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc81006ac30 0x7fc81006d0f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc81c035000 tx=0x7fc81c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:13.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.082+0000 7fc82a000640 1 -- 172.21.15.102:0/2639306941 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc8240542f0 con 0x7fc8340caf40 2024-03-23T03:50:13.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.272+0000 7fc839000640 1 -- 172.21.15.102:0/2639306941 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc83406b690 con 0x7fc81006ac30 2024-03-23T03:50:13.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.293+0000 7fc82a000640 1 -- 172.21.15.102:0/2639306941 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+276359 (secure 0 0 0) 0x7fc81c0343f0 con 0x7fc81006ac30 2024-03-23T03:50:13.295 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:13.299 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:50:13.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.300+0000 7fc839000640 1 -- 172.21.15.102:0/2639306941 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc81006ac30 msgr2=0x7fc81006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:13.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.300+0000 7fc839000640 1 --2- 172.21.15.102:0/2639306941 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc81006ac30 0x7fc81006d0f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc81c035000 tx=0x7fc81c030000 comp rx=0 tx=0).stop 2024-03-23T03:50:13.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.300+0000 7fc839000640 1 -- 172.21.15.102:0/2639306941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8340caf40 msgr2=0x7fc83414ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:13.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.300+0000 7fc839000640 1 --2- 172.21.15.102:0/2639306941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8340caf40 0x7fc83414ac30 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fc82400c850 tx=0x7fc82400cd10 comp rx=0 tx=0).stop 2024-03-23T03:50:13.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.300+0000 7fc839000640 1 -- 172.21.15.102:0/2639306941 shutdown_connections 2024-03-23T03:50:13.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.300+0000 7fc839000640 1 --2- 172.21.15.102:0/2639306941 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc81006ac30 0x7fc81006d0f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.300+0000 7fc839000640 1 --2- 172.21.15.102:0/2639306941 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc83414b370 0x7fc8340c4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.300+0000 7fc839000640 1 --2- 172.21.15.102:0/2639306941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc83415cc00 0x7fc8340c42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.300+0000 7fc839000640 1 --2- 172.21.15.102:0/2639306941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8340caf40 0x7fc83414ac30 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.300+0000 7fc839000640 1 -- 172.21.15.102:0/2639306941 >> 172.21.15.102:0/2639306941 conn(0x7fc83406e580 msgr2=0x7fc8340cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:13.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.301+0000 7fc839000640 1 -- 172.21.15.102:0/2639306941 shutdown_connections 2024-03-23T03:50:13.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.301+0000 7fc839000640 1 -- 172.21.15.102:0/2639306941 wait complete. 2024-03-23T03:50:13.322 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":177,"stamp":"2024-03-23T03:50:11.778281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":228756,"kb_used_data":8500,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754745964,"statfs":{"total":773094113280,"available":772859867136,"internally_reserved":0,"allocated":8704000,"data_stored":4534281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":225431098},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.333625"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450165+0000","last_change":"2024-03-23T03:50:06.450165+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450165+0000","last_undegraded":"2024-03-23T03:50:06.450165+0000","last_fullsized":"2024-03-23T03:50:06.450165+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:32:26.890806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.461707+0000","last_change":"2024-03-23T03:50:06.461707+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.461707+0000","last_undegraded":"2024-03-23T03:50:06.461707+0000","last_fullsized":"2024-03-23T03:50:06.461707+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:14:39.947258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.461900+0000","last_change":"2024-03-23T03:50:06.461900+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.461900+0000","last_undegraded":"2024-03-23T03:50:06.461900+0000","last_fullsized":"2024-03-23T03:50:06.461900+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:03:17.838071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.463303+0000","last_change":"2024-03-23T03:50:06.463303+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.463303+0000","last_undegraded":"2024-03-23T03:50:06.463303+0000","last_fullsized":"2024-03-23T03:50:06.463303+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:33:34.230046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.455179+0000","last_change":"2024-03-23T03:50:06.455179+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.455179+0000","last_undegraded":"2024-03-23T03:50:06.455179+0000","last_fullsized":"2024-03-23T03:50:06.455179+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:33:34.230046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.469417+0000","last_change":"2024-03-23T03:50:06.469417+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.469417+0000","last_undegraded":"2024-03-23T03:50:06.469417+0000","last_fullsized":"2024-03-23T03:50:06.469417+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:36:00.228119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.467620+0000","last_change":"2024-03-23T03:50:06.467620+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.467620+0000","last_undegraded":"2024-03-23T03:50:06.467620+0000","last_fullsized":"2024-03-23T03:50:06.467620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:06:01.399011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450165+0000","last_change":"2024-03-23T03:50:06.450165+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450165+0000","last_undegraded":"2024-03-23T03:50:06.450165+0000","last_fullsized":"2024-03-23T03:50:06.450165+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:59:56.498434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.462227+0000","last_change":"2024-03-23T03:50:06.462227+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.462227+0000","last_undegraded":"2024-03-23T03:50:06.462227+0000","last_fullsized":"2024-03-23T03:50:06.462227+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:44:27.821644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.469099+0000","last_change":"2024-03-23T03:50:06.469099+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.469099+0000","last_undegraded":"2024-03-23T03:50:06.469099+0000","last_fullsized":"2024-03-23T03:50:06.469099+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:36:57.803871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450817+0000","last_change":"2024-03-23T03:50:06.450817+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450817+0000","last_undegraded":"2024-03-23T03:50:06.450817+0000","last_fullsized":"2024-03-23T03:50:06.450817+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:18:47.365643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.462222+0000","last_change":"2024-03-23T03:50:06.462222+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.462222+0000","last_undegraded":"2024-03-23T03:50:06.462222+0000","last_fullsized":"2024-03-23T03:50:06.462222+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:48:07.848353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.470833+0000","last_change":"2024-03-23T03:50:06.470833+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.470833+0000","last_undegraded":"2024-03-23T03:50:06.470833+0000","last_fullsized":"2024-03-23T03:50:06.470833+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:36:15.054606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.460079+0000","last_change":"2024-03-23T03:50:06.460079+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.460079+0000","last_undegraded":"2024-03-23T03:50:06.460079+0000","last_fullsized":"2024-03-23T03:50:06.460079+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:28:24.935685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450491+0000","last_change":"2024-03-23T03:50:06.450491+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450491+0000","last_undegraded":"2024-03-23T03:50:06.450491+0000","last_fullsized":"2024-03-23T03:50:06.450491+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:24:41.782032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.467435+0000","last_change":"2024-03-23T03:50:06.467435+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.467435+0000","last_undegraded":"2024-03-23T03:50:06.467435+0000","last_fullsized":"2024-03-23T03:50:06.467435+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:49:00.654026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.460903+0000","last_change":"2024-03-23T03:50:06.460903+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.460903+0000","last_undegraded":"2024-03-23T03:50:06.460903+0000","last_fullsized":"2024-03-23T03:50:06.460903+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:59:01.344309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.455371+0000","last_change":"2024-03-23T03:50:06.455371+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.455371+0000","last_undegraded":"2024-03-23T03:50:06.455371+0000","last_fullsized":"2024-03-23T03:50:06.455371+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:19:21.548516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450380+0000","last_change":"2024-03-23T03:50:06.450380+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450380+0000","last_undegraded":"2024-03-23T03:50:06.450380+0000","last_fullsized":"2024-03-23T03:50:06.450380+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:54:59.757010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.451352+0000","last_change":"2024-03-23T03:50:06.451352+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.451352+0000","last_undegraded":"2024-03-23T03:50:06.451352+0000","last_fullsized":"2024-03-23T03:50:06.451352+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:03:17.838071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.455710+0000","last_change":"2024-03-23T03:50:06.455710+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.455710+0000","last_undegraded":"2024-03-23T03:50:06.455710+0000","last_fullsized":"2024-03-23T03:50:06.455710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:41:32.913577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.456982+0000","last_change":"2024-03-23T03:50:06.456982+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.456982+0000","last_undegraded":"2024-03-23T03:50:06.456982+0000","last_fullsized":"2024-03-23T03:50:06.456982+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:46:17.453220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.455224+0000","last_change":"2024-03-23T03:50:06.455224+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.455224+0000","last_undegraded":"2024-03-23T03:50:06.455224+0000","last_fullsized":"2024-03-23T03:50:06.455224+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:54:23.987953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.469854+0000","last_change":"2024-03-23T03:50:06.469854+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.469854+0000","last_undegraded":"2024-03-23T03:50:06.469854+0000","last_fullsized":"2024-03-23T03:50:06.469854+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:14:48.590038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.467198+0000","last_change":"2024-03-23T03:50:06.467198+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.467198+0000","last_undegraded":"2024-03-23T03:50:06.467198+0000","last_fullsized":"2024-03-23T03:50:06.467198+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:51:13.757234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.460543+0000","last_change":"2024-03-23T03:50:06.460543+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.460543+0000","last_undegraded":"2024-03-23T03:50:06.460543+0000","last_fullsized":"2024-03-23T03:50:06.460543+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:01:47.992165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.460249+0000","last_change":"2024-03-23T03:50:06.460249+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.460249+0000","last_undegraded":"2024-03-23T03:50:06.460249+0000","last_fullsized":"2024-03-23T03:50:06.460249+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:58.116627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.461287+0000","last_change":"2024-03-23T03:50:06.461287+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.461287+0000","last_undegraded":"2024-03-23T03:50:06.461287+0000","last_fullsized":"2024-03-23T03:50:06.461287+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:19:21.548516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450250+0000","last_change":"2024-03-23T03:50:06.450250+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450250+0000","last_undegraded":"2024-03-23T03:50:06.450250+0000","last_fullsized":"2024-03-23T03:50:06.450250+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:36:27.444641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.460712+0000","last_change":"2024-03-23T03:50:06.460712+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.460712+0000","last_undegraded":"2024-03-23T03:50:06.460712+0000","last_fullsized":"2024-03-23T03:50:06.460712+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:13.215153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.450710+0000","last_change":"2024-03-23T03:50:06.450710+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.450710+0000","last_undegraded":"2024-03-23T03:50:06.450710+0000","last_fullsized":"2024-03-23T03:50:06.450710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:01:47.992165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:07.443611+0000","last_change":"2024-03-23T03:50:07.443611+0000","last_active":"2024-03-23T03:50:07.443611+0000","last_peered":"2024-03-23T03:50:07.443611+0000","last_clean":"2024-03-23T03:50:07.443611+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:07.443611+0000","last_undegraded":"2024-03-23T03:50:07.443611+0000","last_fullsized":"2024-03-23T03:50:07.443611+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":1,"reported_epoch":78,"state":"creating+peering","last_fresh":"2024-03-23T03:50:06.455254+0000","last_change":"2024-03-23T03:50:06.455254+0000","last_active":"2024-03-23T03:50:06.429749+0000","last_peered":"2024-03-23T03:50:06.429749+0000","last_clean":"2024-03-23T03:50:06.429749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:06.455254+0000","last_undegraded":"2024-03-23T03:50:06.455254+0000","last_fullsized":"2024-03-23T03:50:06.455254+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:11:43.166129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":210,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-03-23T03:50:04.652817+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:50:04.652817+0000","last_peered":"2024-03-23T03:50:04.652817+0000","last_clean":"2024-03-23T03:50:04.652817+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:50:04.652817+0000","last_undegraded":"2024-03-23T03:50:04.652817+0000","last_fullsized":"2024-03-23T03:50:04.652817+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542188,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28516,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343324,"statfs":{"total":96636764160,"available":96607563776,"internally_reserved":0,"allocated":1531904,"data_stored":1008794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.079,"15min":1.069},"min":{"1min":0.68200000000000005,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.573,"5min":1.635,"15min":1.635},"last":1.075},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0449999999999999,"15min":1.048},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.417,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.353}]},{"osd":1,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0549999999999999,"15min":1.044},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.544,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.466},{"interface":"front","average":{"1min":1.085,"5min":1.2330000000000001,"15min":1.3009999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.3999999999999999,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":0.90600000000000003}]},{"osd":2,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.139,"15min":1.1559999999999999},"min":{"1min":0.76400000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.7909999999999999,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.24},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.345,"15min":1.421},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.4630000000000001,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":0.57799999999999996}]},{"osd":3,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.355,"15min":1.4370000000000001},"min":{"1min":0.873,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.397,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.3380000000000001,"15min":1.3580000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":1.5840000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.6359999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.266,"15min":1.2669999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.6299999999999999,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.8220000000000001},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.357,"15min":1.375},"min":{"1min":0.77900000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0619999999999998,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.5189999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.3999999999999999,"15min":1.4219999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.6539999999999999,"5min":2.645,"15min":2.645},"last":1.278},{"interface":"front","average":{"1min":1.339,"5min":1.4370000000000001,"15min":1.4530000000000001},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":1.7050000000000001,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.696}]},{"osd":6,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.5229999999999999,"15min":1.5589999999999999},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":1.8200000000000001,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.9279999999999999},{"interface":"front","average":{"1min":1.415,"5min":1.4930000000000001,"15min":1.5209999999999999},"min":{"1min":1.1319999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":2.1509999999999998,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.734}]}]},{"osd":6,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.0309999999999999,"15min":1.04},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.3979999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0429999999999999,"15min":1.038},"min":{"1min":0.72099999999999997,"5min":0.62,"15min":0.62},"max":{"1min":1.6120000000000001,"5min":1.946,"15min":1.946},"last":0.66100000000000003}]},{"osd":1,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":0.874},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.129,"15min":1.1559999999999999},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.4810000000000001,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.80500000000000005}]},{"osd":2,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.075,"15min":1.069},"min":{"1min":0.73699999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.5449999999999999,"5min":2.044,"15min":2.044},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1950000000000001,"15min":1.218},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":0.96899999999999997}]},{"osd":3,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2629999999999999,"15min":1.2849999999999999},"min":{"1min":0.76800000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.4910000000000001,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2549999999999999,"15min":1.2649999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.6319999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.94199999999999995}]},{"osd":4,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3759999999999999,"15min":1.3839999999999999},"min":{"1min":0.67700000000000005,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.73,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.363,"15min":1.383},"min":{"1min":0.63400000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.762,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.448}]},{"osd":5,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.415,"15min":1.4359999999999999},"min":{"1min":0.71099999999999997,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.6919999999999999,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.456,"5min":1.5249999999999999,"15min":1.5469999999999999},"min":{"1min":0.79700000000000004,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8149999999999999,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.091}]},{"osd":7,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.4059999999999999,"15min":1.4119999999999999},"min":{"1min":0.77600000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.802,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.3740000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.5129999999999999,"15min":1.5409999999999999},"min":{"1min":0.84099999999999997,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.74,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.254}]}]},{"osd":5,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.1759999999999999,"15min":1.214},"min":{"1min":0.67900000000000005,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.6779999999999999,"5min":1.806,"15min":1.806},"last":1.2},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.157,"15min":1.181},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.554,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.90800000000000003}]},{"osd":1,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.81699999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.5489999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.026,"5min":1.1599999999999999,"15min":1.1919999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4219999999999999,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.0489999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.298,"15min":1.3540000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5429999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3340000000000001,"15min":1.3779999999999999},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.772,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.0840000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.21,"15min":1.2470000000000001},"min":{"1min":0.70599999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.653,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.393,"15min":1.452},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.6779999999999999,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.137}]},{"osd":4,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.3420000000000001,"15min":1.4119999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.671,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.282,"5min":1.4119999999999999,"15min":1.444},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.77,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.2270000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.458,"15min":1.514},"min":{"1min":0.76700000000000002,"5min":0.63,"15min":0.63},"max":{"1min":1.873,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.5940000000000001,"15min":1.6559999999999999},"min":{"1min":0.86199999999999999,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.956,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.397}]},{"osd":7,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.538,"15min":1.607},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.673,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.411,"5min":1.6200000000000001,"15min":1.696},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.855,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.6100000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32032,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339808,"statfs":{"total":96636764160,"available":96603963392,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.002,"15min":0.97999999999999998},"min":{"1min":0.67800000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.0149999999999999,"15min":1.012},"min":{"1min":0.81200000000000006,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3069999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.71399999999999997}]},{"osd":1,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.0700000000000001,"15min":1.05},"min":{"1min":0.74299999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.704,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.147,"5min":1.0880000000000001,"15min":1.0640000000000001},"min":{"1min":0.50600000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.181}]},{"osd":2,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2949999999999999,"15min":1.325},"min":{"1min":0.81599999999999995,"5min":0.625,"15min":0.625},"max":{"1min":1.569,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.29,"5min":1.3100000000000001,"15min":1.3080000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.8680000000000001,"5min":2.181,"15min":2.181},"last":1.2869999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1339999999999999,"15min":1.1299999999999999},"min":{"1min":0.73199999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.26,"15min":1.278},"min":{"1min":0.82899999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.5720000000000001,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":0.91600000000000004}]},{"osd":5,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.278,"15min":1.274},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.298,"15min":1.2829999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.2349999999999999,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.228}]},{"osd":6,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.415,"15min":1.4239999999999999},"min":{"1min":1.143,"5min":0.748,"15min":0.748},"max":{"1min":1.833,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.425,"15min":1.4039999999999999},"min":{"1min":0.745,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":1.377}]},{"osd":7,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.385,"15min":1.371},"min":{"1min":1.0880000000000001,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":2.3479999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.4970000000000001,"15min":1.5009999999999999},"min":{"1min":1.056,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.974,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.25}]}]},{"osd":3,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.83899999999999997,"15min":0.86099999999999999},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.256,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.446,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":1.4610000000000001}]},{"osd":1,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.92800000000000005,"15min":0.97199999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.997,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.98199999999999998},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.034,"15min":1.0720000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.712,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.90400000000000003}]},{"osd":2,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.006,"15min":1.052},"min":{"1min":0.435,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.613,"5min":1.823,"15min":1.823},"last":1.109},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.1319999999999999,"15min":1.153},"min":{"1min":0.48199999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.6020000000000001,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":1.3700000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.2310000000000001,"15min":1.2450000000000001},"min":{"1min":0.67600000000000005,"5min":0.625,"15min":0.625},"max":{"1min":1.3859999999999999,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":1.054},{"interface":"front","average":{"1min":1.151,"5min":1.377,"15min":1.4370000000000001},"min":{"1min":0.77100000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.4770000000000001,"5min":4.819,"15min":4.819},"last":1.4850000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.603,"15min":1.4470000000000001},"min":{"1min":0.75,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.0299999999999998,"5min":30.228000000000002,"15min":30.228000000000002},"last":1.165},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.3320000000000001,"15min":1.355},"min":{"1min":0.91400000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.637,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":1.2709999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.448,"15min":1.52},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.5860000000000001,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.206},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3999999999999999,"15min":1.397},"min":{"1min":0.90100000000000002,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.1429999999999998,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":1.5}]},{"osd":7,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.395,"15min":1.4219999999999999},"min":{"1min":0.81200000000000006,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.71,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.577,"15min":1.601},"min":{"1min":0.73299999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.629,"5min":10.663,"15min":10.663},"last":1.258}]}]},{"osd":2,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.84999999999999998,"15min":0.88100000000000001},"min":{"1min":0.318,"5min":0.307,"15min":0.307},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.1930000000000001},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.88700000000000001,"15min":0.94299999999999995},"min":{"1min":0.53500000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.23,"5min":2.226,"15min":2.226},"last":0.872}]},{"osd":1,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.96899999999999997,"15min":0.995},"min":{"1min":0.621,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.5109999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":1.02},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.88,"15min":0.91700000000000004},"min":{"1min":0.53700000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.282,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.70399999999999996}]},{"osd":3,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.98499999999999999,"15min":1.026},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.4750000000000001,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":1.403},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0549999999999999,"15min":1.085},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3380000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.3540000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.2090000000000001,"15min":1.236},"min":{"1min":0.96399999999999997,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.488,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.179,"15min":1.214},"min":{"1min":0.66300000000000003,"5min":0.627,"15min":0.627},"max":{"1min":1.47,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.2390000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.3360000000000001,"15min":1.3759999999999999},"min":{"1min":0.76200000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.304,"15min":1.349},"min":{"1min":0.75,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.5880000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.819}]},{"osd":6,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.268,"15min":1.3400000000000001},"min":{"1min":0.82199999999999995,"5min":0.753,"15min":0.753},"max":{"1min":1.5109999999999999,"5min":2.456,"15min":2.456},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.359,"5min":1.4390000000000001,"15min":1.4890000000000001},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.5089999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.335,"15min":1.379},"min":{"1min":0.82699999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.6870000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.8660000000000001},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.3460000000000001,"15min":1.4099999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5900000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.119}]}]},{"osd":1,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27936,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343904,"statfs":{"total":96636764160,"available":96608157696,"internally_reserved":0,"allocated":937984,"data_stored":418426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.81599999999999995,"15min":0.87},"min":{"1min":0.40200000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.1819999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.82999999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.64000000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.92800000000000005,"15min":0.96599999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.4570000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.94899999999999995,"15min":0.98599999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.4750000000000001,"5min":1.625,"15min":1.625},"last":0.83599999999999997}]},{"osd":3,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.117,"15min":1.117},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.7929999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.99399999999999999,"15min":1.0189999999999999},"min":{"1min":0.57399999999999995,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.5600000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.68200000000000005}]},{"osd":4,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.232,"15min":1.292},"min":{"1min":0.69199999999999995,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.752,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.2749999999999999,"15min":1.3149999999999999},"min":{"1min":0.82599999999999996,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.575,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.01}]},{"osd":5,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.2729999999999999,"15min":1.329},"min":{"1min":0.82399999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.7470000000000001,"5min":3.593,"15min":3.593},"last":1.105},{"interface":"front","average":{"1min":1.276,"5min":1.3700000000000001,"15min":1.4399999999999999},"min":{"1min":0.76200000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6719999999999999,"5min":2.359,"15min":2.359},"last":1.671}]},{"osd":6,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.349,"15min":1.405},"min":{"1min":0.79300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.1709999999999998,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.484},{"interface":"front","average":{"1min":1.147,"5min":1.302,"15min":1.3580000000000001},"min":{"1min":0.77800000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.696,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.26}]},{"osd":7,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.3759999999999999,"15min":1.4379999999999999},"min":{"1min":0.81100000000000005,"5min":0.627,"15min":0.627},"max":{"1min":1.7649999999999999,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.7},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.4219999999999999,"15min":1.506},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.7569999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.5640000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28528,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343312,"statfs":{"total":96636764160,"available":96607551488,"internally_reserved":0,"allocated":1544192,"data_stored":1014931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.90000000000000002,"15min":0.94699999999999995},"min":{"1min":0.435,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.1599999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.98899999999999999},"min":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.3640000000000001,"5min":1.506,"15min":1.506},"last":0.90900000000000003}]},{"osd":2,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.95799999999999996,"15min":1.0009999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1279999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.91900000000000004,"5min":1.0640000000000001,"15min":1.105},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.4350000000000001,"5min":1.901,"15min":1.901},"last":1.2749999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.97499999999999998,"15min":1.01},"min":{"1min":0.48999999999999999,"5min":0.433,"15min":0.433},"max":{"1min":1.5189999999999999,"5min":1.972,"15min":1.972},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.103,"15min":1.159},"min":{"1min":0.51400000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3180000000000001,"5min":2.298,"15min":2.298},"last":1.018}]},{"osd":4,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.304,"15min":1.3420000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.599,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":1.05},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.252,"15min":1.28},"min":{"1min":0.83099999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.4990000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.23}]},{"osd":5,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.3340000000000001,"15min":1.3779999999999999},"min":{"1min":0.878,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.4450000000000001,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.169},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.403,"15min":1.4470000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.49,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.2030000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.46,"15min":1.538},"min":{"1min":0.85799999999999998,"5min":0.747,"15min":0.747},"max":{"1min":1.5940000000000001,"5min":3.032,"15min":3.032},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.28,"5min":1.4419999999999999,"15min":1.498},"min":{"1min":0.98199999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.5740000000000001,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.5069999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.46,"15min":1.5129999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.7110000000000001,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.3939999999999999,"15min":1.4299999999999999},"min":{"1min":0.871,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.008,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.423}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:50:13.327 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3f is not active+clean 2024-03-23T03:50:13.327 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.327 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3e is not active+clean 2024-03-23T03:50:13.327 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.327 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3d is not active+clean 2024-03-23T03:50:13.327 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.328 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3c is not active+clean 2024-03-23T03:50:13.328 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.328 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3b is not active+clean 2024-03-23T03:50:13.328 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.328 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3a is not active+clean 2024-03-23T03:50:13.328 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.328 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.39 is not active+clean 2024-03-23T03:50:13.328 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.328 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.38 is not active+clean 2024-03-23T03:50:13.328 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.328 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.37 is not active+clean 2024-03-23T03:50:13.329 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.329 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.36 is not active+clean 2024-03-23T03:50:13.329 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.329 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.35 is not active+clean 2024-03-23T03:50:13.329 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.329 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.34 is not active+clean 2024-03-23T03:50:13.329 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.329 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.33 is not active+clean 2024-03-23T03:50:13.329 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.329 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.32 is not active+clean 2024-03-23T03:50:13.329 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.329 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.31 is not active+clean 2024-03-23T03:50:13.330 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.330 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.30 is not active+clean 2024-03-23T03:50:13.330 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.330 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2f is not active+clean 2024-03-23T03:50:13.330 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.330 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2e is not active+clean 2024-03-23T03:50:13.330 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.330 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2d is not active+clean 2024-03-23T03:50:13.330 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.330 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2c is not active+clean 2024-03-23T03:50:13.330 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.331 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2b is not active+clean 2024-03-23T03:50:13.331 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.331 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2a is not active+clean 2024-03-23T03:50:13.331 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.331 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.29 is not active+clean 2024-03-23T03:50:13.331 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.331 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.28 is not active+clean 2024-03-23T03:50:13.331 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.331 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.27 is not active+clean 2024-03-23T03:50:13.331 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.331 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.26 is not active+clean 2024-03-23T03:50:13.331 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.332 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.25 is not active+clean 2024-03-23T03:50:13.332 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.332 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.24 is not active+clean 2024-03-23T03:50:13.332 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.332 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.23 is not active+clean 2024-03-23T03:50:13.332 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.332 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.22 is not active+clean 2024-03-23T03:50:13.332 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.332 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.21 is not active+clean 2024-03-23T03:50:13.332 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.332 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.20 is not active+clean 2024-03-23T03:50:13.332 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.333 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.18 is not active+clean 2024-03-23T03:50:13.333 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.333 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1f is not active+clean 2024-03-23T03:50:13.333 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.450165+0000', 'last_change': '2024-03-23T03:50:06.450165+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.450165+0000', 'last_undegraded': '2024-03-23T03:50:06.450165+0000', 'last_fullsized': '2024-03-23T03:50:06.450165+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T09:32:26.890806+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-23T03:50:13.333 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.19 is not active+clean 2024-03-23T03:50:13.333 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.333 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1e is not active+clean 2024-03-23T03:50:13.333 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.461707+0000', 'last_change': '2024-03-23T03:50:06.461707+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.461707+0000', 'last_undegraded': '2024-03-23T03:50:06.461707+0000', 'last_fullsized': '2024-03-23T03:50:06.461707+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T08:14:39.947258+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-23T03:50:13.333 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1a is not active+clean 2024-03-23T03:50:13.333 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.333 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1d is not active+clean 2024-03-23T03:50:13.334 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.461900+0000', 'last_change': '2024-03-23T03:50:06.461900+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.461900+0000', 'last_undegraded': '2024-03-23T03:50:06.461900+0000', 'last_fullsized': '2024-03-23T03:50:06.461900+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T09:03:17.838071+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 0], 'acting': [7, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-23T03:50:13.334 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1b is not active+clean 2024-03-23T03:50:13.334 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.334 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1c is not active+clean 2024-03-23T03:50:13.334 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.463303+0000', 'last_change': '2024-03-23T03:50:06.463303+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.463303+0000', 'last_undegraded': '2024-03-23T03:50:06.463303+0000', 'last_fullsized': '2024-03-23T03:50:06.463303+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T06:33:34.230046+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-23T03:50:13.334 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1c is not active+clean 2024-03-23T03:50:13.334 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.334 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1b is not active+clean 2024-03-23T03:50:13.334 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.455179+0000', 'last_change': '2024-03-23T03:50:06.455179+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.455179+0000', 'last_undegraded': '2024-03-23T03:50:06.455179+0000', 'last_fullsized': '2024-03-23T03:50:06.455179+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T06:33:34.230046+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-23T03:50:13.334 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1d is not active+clean 2024-03-23T03:50:13.334 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1a is not active+clean 2024-03-23T03:50:13.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.469417+0000', 'last_change': '2024-03-23T03:50:06.469417+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.469417+0000', 'last_undegraded': '2024-03-23T03:50:06.469417+0000', 'last_fullsized': '2024-03-23T03:50:06.469417+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T05:36:00.228119+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-23T03:50:13.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1e is not active+clean 2024-03-23T03:50:13.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.19 is not active+clean 2024-03-23T03:50:13.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.467620+0000', 'last_change': '2024-03-23T03:50:06.467620+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.467620+0000', 'last_undegraded': '2024-03-23T03:50:06.467620+0000', 'last_fullsized': '2024-03-23T03:50:06.467620+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T10:06:01.399011+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-23T03:50:13.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1f is not active+clean 2024-03-23T03:50:13.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.18 is not active+clean 2024-03-23T03:50:13.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.450165+0000', 'last_change': '2024-03-23T03:50:06.450165+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.450165+0000', 'last_undegraded': '2024-03-23T03:50:06.450165+0000', 'last_fullsized': '2024-03-23T03:50:06.450165+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T04:59:56.498434+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-23T03:50:13.335 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.10 is not active+clean 2024-03-23T03:50:13.335 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.17 is not active+clean 2024-03-23T03:50:13.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.462227+0000', 'last_change': '2024-03-23T03:50:06.462227+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.462227+0000', 'last_undegraded': '2024-03-23T03:50:06.462227+0000', 'last_fullsized': '2024-03-23T03:50:06.462227+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T05:44:27.821644+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-23T03:50:13.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.11 is not active+clean 2024-03-23T03:50:13.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.16 is not active+clean 2024-03-23T03:50:13.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.469099+0000', 'last_change': '2024-03-23T03:50:06.469099+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.469099+0000', 'last_undegraded': '2024-03-23T03:50:06.469099+0000', 'last_fullsized': '2024-03-23T03:50:06.469099+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T15:36:57.803871+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-23T03:50:13.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.12 is not active+clean 2024-03-23T03:50:13.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.15 is not active+clean 2024-03-23T03:50:13.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.450817+0000', 'last_change': '2024-03-23T03:50:06.450817+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.450817+0000', 'last_undegraded': '2024-03-23T03:50:06.450817+0000', 'last_fullsized': '2024-03-23T03:50:06.450817+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T07:18:47.365643+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-23T03:50:13.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.13 is not active+clean 2024-03-23T03:50:13.336 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.336 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.14 is not active+clean 2024-03-23T03:50:13.337 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.462222+0000', 'last_change': '2024-03-23T03:50:06.462222+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.462222+0000', 'last_undegraded': '2024-03-23T03:50:06.462222+0000', 'last_fullsized': '2024-03-23T03:50:06.462222+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T05:48:07.848353+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-23T03:50:13.337 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.14 is not active+clean 2024-03-23T03:50:13.337 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.337 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.13 is not active+clean 2024-03-23T03:50:13.337 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.470833+0000', 'last_change': '2024-03-23T03:50:06.470833+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.470833+0000', 'last_undegraded': '2024-03-23T03:50:06.470833+0000', 'last_fullsized': '2024-03-23T03:50:06.470833+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T09:36:15.054606+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-23T03:50:13.337 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.15 is not active+clean 2024-03-23T03:50:13.337 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.337 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.12 is not active+clean 2024-03-23T03:50:13.337 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.460079+0000', 'last_change': '2024-03-23T03:50:06.460079+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.460079+0000', 'last_undegraded': '2024-03-23T03:50:06.460079+0000', 'last_fullsized': '2024-03-23T03:50:06.460079+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T13:28:24.935685+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-23T03:50:13.337 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.16 is not active+clean 2024-03-23T03:50:13.337 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.337 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.11 is not active+clean 2024-03-23T03:50:13.338 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.450491+0000', 'last_change': '2024-03-23T03:50:06.450491+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.450491+0000', 'last_undegraded': '2024-03-23T03:50:06.450491+0000', 'last_fullsized': '2024-03-23T03:50:06.450491+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T10:24:41.782032+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-23T03:50:13.338 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.17 is not active+clean 2024-03-23T03:50:13.338 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.338 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.10 is not active+clean 2024-03-23T03:50:13.338 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.467435+0000', 'last_change': '2024-03-23T03:50:06.467435+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.467435+0000', 'last_undegraded': '2024-03-23T03:50:06.467435+0000', 'last_fullsized': '2024-03-23T03:50:06.467435+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T08:49:00.654026+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-23T03:50:13.338 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.8 is not active+clean 2024-03-23T03:50:13.338 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.338 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.f is not active+clean 2024-03-23T03:50:13.338 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.460903+0000', 'last_change': '2024-03-23T03:50:06.460903+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.460903+0000', 'last_undegraded': '2024-03-23T03:50:06.460903+0000', 'last_fullsized': '2024-03-23T03:50:06.460903+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T03:59:01.344309+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-23T03:50:13.338 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.9 is not active+clean 2024-03-23T03:50:13.339 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.339 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.e is not active+clean 2024-03-23T03:50:13.339 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.455371+0000', 'last_change': '2024-03-23T03:50:06.455371+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.455371+0000', 'last_undegraded': '2024-03-23T03:50:06.455371+0000', 'last_fullsized': '2024-03-23T03:50:06.455371+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T09:19:21.548516+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-23T03:50:13.339 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.a is not active+clean 2024-03-23T03:50:13.339 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.339 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.d is not active+clean 2024-03-23T03:50:13.339 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.450380+0000', 'last_change': '2024-03-23T03:50:06.450380+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.450380+0000', 'last_undegraded': '2024-03-23T03:50:06.450380+0000', 'last_fullsized': '2024-03-23T03:50:06.450380+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T06:54:59.757010+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-23T03:50:13.339 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.b is not active+clean 2024-03-23T03:50:13.339 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.339 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.c is not active+clean 2024-03-23T03:50:13.339 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.451352+0000', 'last_change': '2024-03-23T03:50:06.451352+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.451352+0000', 'last_undegraded': '2024-03-23T03:50:06.451352+0000', 'last_fullsized': '2024-03-23T03:50:06.451352+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T09:03:17.838071+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-23T03:50:13.339 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.c is not active+clean 2024-03-23T03:50:13.340 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.340 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.b is not active+clean 2024-03-23T03:50:13.340 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.455710+0000', 'last_change': '2024-03-23T03:50:06.455710+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.455710+0000', 'last_undegraded': '2024-03-23T03:50:06.455710+0000', 'last_fullsized': '2024-03-23T03:50:06.455710+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T07:41:32.913577+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-23T03:50:13.340 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.e is not active+clean 2024-03-23T03:50:13.340 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.340 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.9 is not active+clean 2024-03-23T03:50:13.340 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.456982+0000', 'last_change': '2024-03-23T03:50:06.456982+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.456982+0000', 'last_undegraded': '2024-03-23T03:50:06.456982+0000', 'last_fullsized': '2024-03-23T03:50:06.456982+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T15:46:17.453220+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-23T03:50:13.340 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.f is not active+clean 2024-03-23T03:50:13.340 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.340 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.8 is not active+clean 2024-03-23T03:50:13.340 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.455224+0000', 'last_change': '2024-03-23T03:50:06.455224+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.455224+0000', 'last_undegraded': '2024-03-23T03:50:06.455224+0000', 'last_fullsized': '2024-03-23T03:50:06.455224+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T12:54:23.987953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-23T03:50:13.341 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.0 is not active+clean 2024-03-23T03:50:13.341 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.341 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.7 is not active+clean 2024-03-23T03:50:13.341 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.469854+0000', 'last_change': '2024-03-23T03:50:06.469854+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.469854+0000', 'last_undegraded': '2024-03-23T03:50:06.469854+0000', 'last_fullsized': '2024-03-23T03:50:06.469854+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T15:14:48.590038+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-23T03:50:13.341 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.1 is not active+clean 2024-03-23T03:50:13.341 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.341 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.6 is not active+clean 2024-03-23T03:50:13.341 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.467198+0000', 'last_change': '2024-03-23T03:50:06.467198+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.467198+0000', 'last_undegraded': '2024-03-23T03:50:06.467198+0000', 'last_fullsized': '2024-03-23T03:50:06.467198+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T04:51:13.757234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 2], 'acting': [0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-23T03:50:13.341 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.2 is not active+clean 2024-03-23T03:50:13.341 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.341 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.5 is not active+clean 2024-03-23T03:50:13.342 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.460543+0000', 'last_change': '2024-03-23T03:50:06.460543+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.460543+0000', 'last_undegraded': '2024-03-23T03:50:06.460543+0000', 'last_fullsized': '2024-03-23T03:50:06.460543+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T06:01:47.992165+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 5], 'acting': [7, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-23T03:50:13.342 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.3 is not active+clean 2024-03-23T03:50:13.342 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.342 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.4 is not active+clean 2024-03-23T03:50:13.342 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.460249+0000', 'last_change': '2024-03-23T03:50:06.460249+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.460249+0000', 'last_undegraded': '2024-03-23T03:50:06.460249+0000', 'last_fullsized': '2024-03-23T03:50:06.460249+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T06:59:58.116627+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-03-23T03:50:13.342 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.4 is not active+clean 2024-03-23T03:50:13.342 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.342 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.3 is not active+clean 2024-03-23T03:50:13.342 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.461287+0000', 'last_change': '2024-03-23T03:50:06.461287+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.461287+0000', 'last_undegraded': '2024-03-23T03:50:06.461287+0000', 'last_fullsized': '2024-03-23T03:50:06.461287+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T09:19:21.548516+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-23T03:50:13.342 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.5 is not active+clean 2024-03-23T03:50:13.342 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.343 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.2 is not active+clean 2024-03-23T03:50:13.343 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.450250+0000', 'last_change': '2024-03-23T03:50:06.450250+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.450250+0000', 'last_undegraded': '2024-03-23T03:50:06.450250+0000', 'last_fullsized': '2024-03-23T03:50:06.450250+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T09:36:27.444641+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-23T03:50:13.343 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.6 is not active+clean 2024-03-23T03:50:13.343 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.343 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.1 is not active+clean 2024-03-23T03:50:13.343 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.460712+0000', 'last_change': '2024-03-23T03:50:06.460712+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.460712+0000', 'last_undegraded': '2024-03-23T03:50:06.460712+0000', 'last_fullsized': '2024-03-23T03:50:06.460712+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T12:47:13.215153+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-23T03:50:13.343 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.7 is not active+clean 2024-03-23T03:50:13.343 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.343 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.0 is not active+clean 2024-03-23T03:50:13.343 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.450710+0000', 'last_change': '2024-03-23T03:50:06.450710+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.450710+0000', 'last_undegraded': '2024-03-23T03:50:06.450710+0000', 'last_fullsized': '2024-03-23T03:50:06.450710+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T06:01:47.992165+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-23T03:50:13.343 INFO:tasks.cephfs.filesystem.ceph_manager:PG 12.d is not active+clean 2024-03-23T03:50:13.343 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '12.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T03:50:07.443611+0000', 'last_change': '2024-03-23T03:50:07.443611+0000', 'last_active': '2024-03-23T03:50:07.443611+0000', 'last_peered': '2024-03-23T03:50:07.443611+0000', 'last_clean': '2024-03-23T03:50:07.443611+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:07.443611+0000', 'last_undegraded': '2024-03-23T03:50:07.443611+0000', 'last_fullsized': '2024-03-23T03:50:07.443611+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:07.443611+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T03:50:13.344 INFO:tasks.cephfs.filesystem.ceph_manager:PG 11.a is not active+clean 2024-03-23T03:50:13.344 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '11.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 78, 'state': 'creating+peering', 'last_fresh': '2024-03-23T03:50:06.455254+0000', 'last_change': '2024-03-23T03:50:06.455254+0000', 'last_active': '2024-03-23T03:50:06.429749+0000', 'last_peered': '2024-03-23T03:50:06.429749+0000', 'last_clean': '2024-03-23T03:50:06.429749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T03:50:06.455254+0000', 'last_undegraded': '2024-03-23T03:50:06.455254+0000', 'last_fullsized': '2024-03-23T03:50:06.455254+0000', 'mapping_epoch': 78, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 78, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'last_clean_scrub_stamp': '2024-03-23T03:50:06.429749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-03-24T09:11:43.166129+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-23T03:50:13.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-23T03:50:13.419 INFO:tasks.ceph.mds.a.smithi102.stderr:2024-03-23T03:50:13.418+0000 7f5d41800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T03:50:13.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.571+0000 7f7dc0a00640 1 Processor -- start 2024-03-23T03:50:13.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.571+0000 7f7dc0a00640 1 -- start start 2024-03-23T03:50:13.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.573+0000 7f7dc0a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dbc15b560 0x7f7dbc15b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:13.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.573+0000 7f7dc0a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7dbc15d410 0x7f7dbc0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:13.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.573+0000 7f7dc0a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dbc06c6f0 con 0x7f7dbc0b39b0 2024-03-23T03:50:13.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.573+0000 7f7dc0a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dbc0b4370 con 0x7f7dbc15d410 2024-03-23T03:50:13.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.573+0000 7f7dc0a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc06c9d0 con 0x7f7dbc15b560 2024-03-23T03:50:13.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.573+0000 7f7dba800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7dbc15d410 0x7f7dbc0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:13.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.573+0000 7f7dba800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7dbc15d410 0x7f7dbc0b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38710/0 (socket says 172.21.15.102:38710) 2024-03-23T03:50:13.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.573+0000 7f7dba800640 1 -- 172.21.15.102:0/944807929 learned_addr learned my addr 172.21.15.102:0/944807929 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:13.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.573+0000 7f7db9e00640 1 --2- 172.21.15.102:0/944807929 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dbc15b560 0x7f7dbc15b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:13.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.574+0000 7f7dba800640 1 -- 172.21.15.102:0/944807929 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dbc15b560 msgr2=0x7f7dbc15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:13.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.574+0000 7f7dba800640 1 --2- 172.21.15.102:0/944807929 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dbc15b560 0x7f7dbc15b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.574+0000 7f7dba800640 1 -- 172.21.15.102:0/944807929 >> v1:172.21.15.173:6789/0 conn(0x7f7dbc0b39b0 legacy=0x7f7dbc15e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:13.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.574+0000 7f7dba800640 1 -- 172.21.15.102:0/944807929 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc06ccb0 con 0x7f7dbc15d410 2024-03-23T03:50:13.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.575+0000 7f7dba800640 1 --2- 172.21.15.102:0/944807929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dbc15d410 0x7f7dbc0b3e30 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f7db4009e50 tx=0x7f7db402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f85248fde32f5ab server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:13.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.575+0000 7f7db8a00640 1 -- 172.21.15.102:0/944807929 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db402ca70 con 0x7f7dbc15d410 2024-03-23T03:50:13.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.575+0000 7f7db8a00640 1 -- 172.21.15.102:0/944807929 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db402cbd0 con 0x7f7dbc15d410 2024-03-23T03:50:13.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.575+0000 7f7db8a00640 1 -- 172.21.15.102:0/944807929 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db40343f0 con 0x7f7dbc15d410 2024-03-23T03:50:13.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.576+0000 7f7dc0a00640 1 -- 172.21.15.102:0/944807929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dbc15d410 msgr2=0x7f7dbc0b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:13.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.576+0000 7f7dc0a00640 1 --2- 172.21.15.102:0/944807929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dbc15d410 0x7f7dbc0b3e30 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f7db4009e50 tx=0x7f7db402bed0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.576+0000 7f7dc0a00640 1 -- 172.21.15.102:0/944807929 shutdown_connections 2024-03-23T03:50:13.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.576+0000 7f7dc0a00640 1 --2- 172.21.15.102:0/944807929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dbc15d410 0x7f7dbc0b3e30 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.576+0000 7f7dc0a00640 1 --2- 172.21.15.102:0/944807929 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dbc15b560 0x7f7dbc15b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.576+0000 7f7dc0a00640 1 -- 172.21.15.102:0/944807929 >> 172.21.15.102:0/944807929 conn(0x7f7dbc06a3d0 msgr2=0x7f7dbc0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:13.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.576+0000 7f7dc0a00640 1 -- 172.21.15.102:0/944807929 shutdown_connections 2024-03-23T03:50:13.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.576+0000 7f7dc0a00640 1 -- 172.21.15.102:0/944807929 wait complete. 2024-03-23T03:50:13.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.577+0000 7f7dc0a00640 1 Processor -- start 2024-03-23T03:50:13.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.578+0000 7f7dc0a00640 1 -- start start 2024-03-23T03:50:13.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.578+0000 7f7dc0a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7dbc0b39b0 0x7f7dbc13f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:13.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.578+0000 7f7dc0a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dbc15b560 0x7f7dbc141810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:13.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.578+0000 7f7dc0a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dbc15d410 0x7f7dbc141d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:13.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.578+0000 7f7dc0a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc152b00 con 0x7f7dbc15d410 2024-03-23T03:50:13.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.578+0000 7f7dc0a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc140080 con 0x7f7dbc0b39b0 2024-03-23T03:50:13.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.578+0000 7f7dc0a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dbc1540e0 con 0x7f7dbc15b560 2024-03-23T03:50:13.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.579+0000 7f7dba800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dbc15d410 0x7f7dbc141d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:13.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.579+0000 7f7dba800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dbc15d410 0x7f7dbc141d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38716/0 (socket says 172.21.15.102:38716) 2024-03-23T03:50:13.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.579+0000 7f7dba800640 1 -- 172.21.15.102:0/628801547 learned_addr learned my addr 172.21.15.102:0/628801547 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:13.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.579+0000 7f7db9e00640 1 --2- 172.21.15.102:0/628801547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7dbc0b39b0 0x7f7dbc13f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:13.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.579+0000 7f7db9400640 1 --2- 172.21.15.102:0/628801547 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dbc15b560 0x7f7dbc141810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:13.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.579+0000 7f7dba800640 1 -- 172.21.15.102:0/628801547 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dbc15b560 msgr2=0x7f7dbc141810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:13.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.579+0000 7f7dba800640 1 --2- 172.21.15.102:0/628801547 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dbc15b560 0x7f7dbc141810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.579+0000 7f7dba800640 1 -- 172.21.15.102:0/628801547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7dbc0b39b0 msgr2=0x7f7dbc13f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:13.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.579+0000 7f7dba800640 1 --2- 172.21.15.102:0/628801547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7dbc0b39b0 0x7f7dbc13f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.579+0000 7f7dba800640 1 -- 172.21.15.102:0/628801547 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db4009b30 con 0x7f7dbc15d410 2024-03-23T03:50:13.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.579+0000 7f7db9400640 1 --2- 172.21.15.102:0/628801547 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dbc15b560 0x7f7dbc141810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:13.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.579+0000 7f7db9e00640 1 --2- 172.21.15.102:0/628801547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7dbc0b39b0 0x7f7dbc13f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:13.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.579+0000 7f7dba800640 1 --2- 172.21.15.102:0/628801547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dbc15d410 0x7f7dbc141d50 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f7db4009e50 tx=0x7f7db4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:13.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.580+0000 7f7db2000640 1 -- 172.21.15.102:0/628801547 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db4009510 con 0x7f7dbc15d410 2024-03-23T03:50:13.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.580+0000 7f7db2000640 1 -- 172.21.15.102:0/628801547 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db403e450 con 0x7f7dbc15d410 2024-03-23T03:50:13.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.580+0000 7f7db2000640 1 -- 172.21.15.102:0/628801547 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db403e800 con 0x7f7dbc15d410 2024-03-23T03:50:13.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.580+0000 7f7dc0a00640 1 -- 172.21.15.102:0/628801547 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dbc152160 con 0x7f7dbc15d410 2024-03-23T03:50:13.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.580+0000 7f7dc0a00640 1 -- 172.21.15.102:0/628801547 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dbc152720 con 0x7f7dbc15d410 2024-03-23T03:50:13.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.581+0000 7f7dc0a00640 1 -- 172.21.15.102:0/628801547 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dbc152c40 con 0x7f7dbc15d410 2024-03-23T03:50:13.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.582+0000 7f7db2000640 1 -- 172.21.15.102:0/628801547 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7db403e960 con 0x7f7dbc15d410 2024-03-23T03:50:13.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.583+0000 7f7db2000640 1 --2- 172.21.15.102:0/628801547 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d9c06ac00 0x7f7d9c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:13.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.583+0000 7f7db2000640 1 -- 172.21.15.102:0/628801547 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7db40ac2c0 con 0x7f7dbc15d410 2024-03-23T03:50:13.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.583+0000 7f7db9e00640 1 --2- 172.21.15.102:0/628801547 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d9c06ac00 0x7f7d9c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:13.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.584+0000 7f7db9e00640 1 --2- 172.21.15.102:0/628801547 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d9c06ac00 0x7f7d9c06d0c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7dac006fd0 tx=0x7f7dac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:13.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.587+0000 7f7db2000640 1 -- 172.21.15.102:0/628801547 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7dbc152c40 con 0x7f7dbc15d410 2024-03-23T03:50:13.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.856+0000 7f7dc0a00640 1 -- 172.21.15.102:0/628801547 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f7dbc152c40 con 0x7f7dbc15d410 2024-03-23T03:50:13.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.857+0000 7f7db2000640 1 -- 172.21.15.102:0/628801547 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1108 (secure 0 0 0) 0x7f7dbc152c40 con 0x7f7dbc15d410 2024-03-23T03:50:13.857 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:13.858 INFO:teuthology.orchestra.run.smithi102.stdout:{"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":219,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":81,"num_osds":8,"num_up_osds":8,"osd_up_since":1711165606,"num_in_osds":8,"osd_in_since":1711165598,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":234246144,"bytes_avail":772859867136,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":41,"id":5,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":5,"rank":0,"name":"a","status":"up:active","gid":6384}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-03-23T03:47:59.736985+0000","services":{}},"progress_events":{}} 2024-03-23T03:50:13.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.859+0000 7f7dc0a00640 1 -- 172.21.15.102:0/628801547 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d9c06ac00 msgr2=0x7f7d9c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:13.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.859+0000 7f7dc0a00640 1 --2- 172.21.15.102:0/628801547 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d9c06ac00 0x7f7d9c06d0c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7dac006fd0 tx=0x7f7dac008040 comp rx=0 tx=0).stop 2024-03-23T03:50:13.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.859+0000 7f7dc0a00640 1 -- 172.21.15.102:0/628801547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dbc15d410 msgr2=0x7f7dbc141d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:13.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.859+0000 7f7dc0a00640 1 --2- 172.21.15.102:0/628801547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dbc15d410 0x7f7dbc141d50 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f7db4009e50 tx=0x7f7db4004a40 comp rx=0 tx=0).stop 2024-03-23T03:50:13.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.860+0000 7f7dc0a00640 1 -- 172.21.15.102:0/628801547 shutdown_connections 2024-03-23T03:50:13.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.860+0000 7f7dc0a00640 1 --2- 172.21.15.102:0/628801547 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d9c06ac00 0x7f7d9c06d0c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.860+0000 7f7dc0a00640 1 --2- 172.21.15.102:0/628801547 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dbc15d410 0x7f7dbc141d50 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.860+0000 7f7dc0a00640 1 --2- 172.21.15.102:0/628801547 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dbc15b560 0x7f7dbc141810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.860+0000 7f7dc0a00640 1 --2- 172.21.15.102:0/628801547 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7dbc0b39b0 0x7f7dbc13f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:13.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.860+0000 7f7dc0a00640 1 -- 172.21.15.102:0/628801547 >> 172.21.15.102:0/628801547 conn(0x7f7dbc06a3d0 msgr2=0x7f7dbc159930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:13.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.860+0000 7f7dc0a00640 1 -- 172.21.15.102:0/628801547 shutdown_connections 2024-03-23T03:50:13.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:13.860+0000 7f7dc0a00640 1 -- 172.21.15.102:0/628801547 wait complete. 2024-03-23T03:50:13.880 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-23T03:50:13.880 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:50:14.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.085+0000 7fdd7e800640 1 Processor -- start 2024-03-23T03:50:14.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.085+0000 7fdd7e800640 1 -- start start 2024-03-23T03:50:14.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.086+0000 7fdd7e800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd7815b560 0x7fdd7815b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:14.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.087+0000 7fdd7e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdd7815d410 0x7fdd780b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:14.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.087+0000 7fdd7e800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7806c6f0 con 0x7fdd780b39b0 2024-03-23T03:50:14.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.087+0000 7fdd7e800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd780b4370 con 0x7fdd7815d410 2024-03-23T03:50:14.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.087+0000 7fdd7e800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7806c9d0 con 0x7fdd7815b560 2024-03-23T03:50:14.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.087+0000 7fdd77e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fdd7815d410 0x7fdd780b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:14.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.087+0000 7fdd6fe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fdd780b39b0 0x7fdd7815e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44296/0 (socket says 172.21.15.102:44296) 2024-03-23T03:50:14.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.087+0000 7fdd6fe00640 1 -- 172.21.15.102:0/3597130133 learned_addr learned my addr 172.21.15.102:0/3597130133 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:14.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.087+0000 7fdd77400640 1 --2- 172.21.15.102:0/3597130133 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd7815b560 0x7fdd7815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:14.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.087+0000 7fdd77e00640 1 -- 172.21.15.102:0/3597130133 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd7815b560 msgr2=0x7fdd7815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:14.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.087+0000 7fdd77e00640 1 --2- 172.21.15.102:0/3597130133 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd7815b560 0x7fdd7815b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:14.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.087+0000 7fdd77e00640 1 -- 172.21.15.102:0/3597130133 >> v1:172.21.15.102:6789/0 conn(0x7fdd780b39b0 legacy=0x7fdd7815e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:14.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.087+0000 7fdd77e00640 1 -- 172.21.15.102:0/3597130133 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7806ccb0 con 0x7fdd7815d410 2024-03-23T03:50:14.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.087+0000 7fdd77400640 1 --2- 172.21.15.102:0/3597130133 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd7815b560 0x7fdd7815b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:14.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.089+0000 7fdd77e00640 1 --2- 172.21.15.102:0/3597130133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd7815d410 0x7fdd780b3e30 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fdd70009d90 tx=0x7fdd7002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b64aee9709e76ef server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:14.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.089+0000 7fdd76a00640 1 -- 172.21.15.102:0/3597130133 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd7002cc50 con 0x7fdd7815d410 2024-03-23T03:50:14.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.089+0000 7fdd76a00640 1 -- 172.21.15.102:0/3597130133 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd7002cdb0 con 0x7fdd7815d410 2024-03-23T03:50:14.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.089+0000 7fdd76a00640 1 -- 172.21.15.102:0/3597130133 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd70034660 con 0x7fdd7815d410 2024-03-23T03:50:14.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.089+0000 7fdd7e800640 1 -- 172.21.15.102:0/3597130133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd7815d410 msgr2=0x7fdd780b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:14.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.089+0000 7fdd7e800640 1 --2- 172.21.15.102:0/3597130133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd7815d410 0x7fdd780b3e30 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fdd70009d90 tx=0x7fdd7002bea0 comp rx=0 tx=0).stop 2024-03-23T03:50:14.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.090+0000 7fdd7e800640 1 -- 172.21.15.102:0/3597130133 shutdown_connections 2024-03-23T03:50:14.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.090+0000 7fdd7e800640 1 --2- 172.21.15.102:0/3597130133 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd7815d410 0x7fdd780b3e30 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:14.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.090+0000 7fdd7e800640 1 --2- 172.21.15.102:0/3597130133 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd7815b560 0x7fdd7815b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:14.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.090+0000 7fdd7e800640 1 -- 172.21.15.102:0/3597130133 >> 172.21.15.102:0/3597130133 conn(0x7fdd7806a3d0 msgr2=0x7fdd780b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:14.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.090+0000 7fdd7e800640 1 -- 172.21.15.102:0/3597130133 shutdown_connections 2024-03-23T03:50:14.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.090+0000 7fdd7e800640 1 -- 172.21.15.102:0/3597130133 wait complete. 2024-03-23T03:50:14.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.091+0000 7fdd7e800640 1 Processor -- start 2024-03-23T03:50:14.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.092+0000 7fdd7e800640 1 -- start start 2024-03-23T03:50:14.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.092+0000 7fdd7e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd780b39b0 0x7fdd781370f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:14.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.092+0000 7fdd7e800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd7815b560 0x7fdd78159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:14.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.092+0000 7fdd7e800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd7815d410 0x7fdd78159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:14.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.092+0000 7fdd7e800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd78141960 con 0x7fdd780b39b0 2024-03-23T03:50:14.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.092+0000 7fdd7e800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd78141c40 con 0x7fdd7815b560 2024-03-23T03:50:14.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.092+0000 7fdd7e800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd780b1bf0 con 0x7fdd7815d410 2024-03-23T03:50:14.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.093+0000 7fdd77400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd780b39b0 0x7fdd781370f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:14.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.093+0000 7fdd77400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd780b39b0 0x7fdd781370f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38722/0 (socket says 172.21.15.102:38722) 2024-03-23T03:50:14.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.093+0000 7fdd77400640 1 -- 172.21.15.102:0/521422934 learned_addr learned my addr 172.21.15.102:0/521422934 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:14.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.093+0000 7fdd77e00640 1 --2- 172.21.15.102:0/521422934 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd7815d410 0x7fdd78159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:14.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.093+0000 7fdd77400640 1 -- 172.21.15.102:0/521422934 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd7815d410 msgr2=0x7fdd78159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:14.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.093+0000 7fdd77400640 1 --2- 172.21.15.102:0/521422934 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd7815d410 0x7fdd78159e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:14.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.093+0000 7fdd77400640 1 -- 172.21.15.102:0/521422934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd7815b560 msgr2=0x7fdd78159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:14.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.093+0000 7fdd6fe00640 1 --2- 172.21.15.102:0/521422934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd7815b560 0x7fdd78159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:14.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.093+0000 7fdd77400640 1 --2- 172.21.15.102:0/521422934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd7815b560 0x7fdd78159930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:14.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.093+0000 7fdd77400640 1 -- 172.21.15.102:0/521422934 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd70009a70 con 0x7fdd780b39b0 2024-03-23T03:50:14.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.093+0000 7fdd77e00640 1 --2- 172.21.15.102:0/521422934 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd7815d410 0x7fdd78159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:14.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.094+0000 7fdd77400640 1 --2- 172.21.15.102:0/521422934 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd780b39b0 0x7fdd781370f0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fdd6800c7f0 tx=0x7fdd6800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:14.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.094+0000 7fdd6f400640 1 -- 172.21.15.102:0/521422934 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd68007a30 con 0x7fdd780b39b0 2024-03-23T03:50:14.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.094+0000 7fdd6f400640 1 -- 172.21.15.102:0/521422934 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd68007b90 con 0x7fdd780b39b0 2024-03-23T03:50:14.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.094+0000 7fdd7e800640 1 -- 172.21.15.102:0/521422934 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd780b1ed0 con 0x7fdd780b39b0 2024-03-23T03:50:14.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.094+0000 7fdd6f400640 1 -- 172.21.15.102:0/521422934 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6800ce40 con 0x7fdd780b39b0 2024-03-23T03:50:14.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.094+0000 7fdd7e800640 1 -- 172.21.15.102:0/521422934 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd781f2470 con 0x7fdd780b39b0 2024-03-23T03:50:14.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.095+0000 7fdd7e800640 1 -- 172.21.15.102:0/521422934 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd780b23f0 con 0x7fdd780b39b0 2024-03-23T03:50:14.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.096+0000 7fdd6f400640 1 -- 172.21.15.102:0/521422934 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fdd680113f0 con 0x7fdd780b39b0 2024-03-23T03:50:14.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.097+0000 7fdd6f400640 1 --2- 172.21.15.102:0/521422934 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdd5406ac00 0x7fdd5406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:14.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.097+0000 7fdd6f400640 1 -- 172.21.15.102:0/521422934 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fdd68089650 con 0x7fdd780b39b0 2024-03-23T03:50:14.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.097+0000 7fdd6fe00640 1 --2- 172.21.15.102:0/521422934 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdd5406ac00 0x7fdd5406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:14.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.098+0000 7fdd6fe00640 1 --2- 172.21.15.102:0/521422934 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdd5406ac00 0x7fdd5406d0c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdd781f5960 tx=0x7fdd60009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:14.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.102+0000 7fdd6f400640 1 -- 172.21.15.102:0/521422934 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fdd680546f0 con 0x7fdd780b39b0 2024-03-23T03:50:14.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.293+0000 7fdd7e800640 1 -- 172.21.15.102:0/521422934 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdd7815be80 con 0x7fdd5406ac00 2024-03-23T03:50:14.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.302+0000 7fdd6f400640 1 -- 172.21.15.102:0/521422934 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287058 (secure 0 0 0) 0x7fdd600013f0 con 0x7fdd5406ac00 2024-03-23T03:50:14.307 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:14.308 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:50:14.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.309+0000 7fdd7e800640 1 -- 172.21.15.102:0/521422934 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdd5406ac00 msgr2=0x7fdd5406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:14.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.309+0000 7fdd7e800640 1 --2- 172.21.15.102:0/521422934 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdd5406ac00 0x7fdd5406d0c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fdd781f5960 tx=0x7fdd60009040 comp rx=0 tx=0).stop 2024-03-23T03:50:14.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.309+0000 7fdd7e800640 1 -- 172.21.15.102:0/521422934 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd780b39b0 msgr2=0x7fdd781370f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:14.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.309+0000 7fdd7e800640 1 --2- 172.21.15.102:0/521422934 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd780b39b0 0x7fdd781370f0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fdd6800c7f0 tx=0x7fdd6800cbf0 comp rx=0 tx=0).stop 2024-03-23T03:50:14.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.309+0000 7fdd7e800640 1 -- 172.21.15.102:0/521422934 shutdown_connections 2024-03-23T03:50:14.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.309+0000 7fdd7e800640 1 --2- 172.21.15.102:0/521422934 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fdd5406ac00 0x7fdd5406d0c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:14.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.309+0000 7fdd7e800640 1 --2- 172.21.15.102:0/521422934 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fdd7815d410 0x7fdd78159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:14.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.309+0000 7fdd7e800640 1 --2- 172.21.15.102:0/521422934 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd7815b560 0x7fdd78159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:14.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.309+0000 7fdd7e800640 1 --2- 172.21.15.102:0/521422934 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd780b39b0 0x7fdd781370f0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:14.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.309+0000 7fdd7e800640 1 -- 172.21.15.102:0/521422934 >> 172.21.15.102:0/521422934 conn(0x7fdd7806a3d0 msgr2=0x7fdd780cc200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:14.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.309+0000 7fdd7e800640 1 -- 172.21.15.102:0/521422934 shutdown_connections 2024-03-23T03:50:14.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:14.309+0000 7fdd7e800640 1 -- 172.21.15.102:0/521422934 wait complete. 2024-03-23T03:50:14.327 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-03-23T03:50:13.778751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233328,"kb_used_data":8936,"kb_used_omap":12,"kb_used_meta":224243,"kb_avail":754741392,"statfs":{"total":773094113280,"available":772855185408,"internally_reserved":0,"allocated":9150464,"data_stored":4744444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":229625402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.327491"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.083897+0000","last_change":"2024-03-23T03:50:07.985162+0000","last_active":"2024-03-23T03:50:09.083897+0000","last_peered":"2024-03-23T03:50:09.083897+0000","last_clean":"2024-03-23T03:50:09.083897+0000","last_became_active":"2024-03-23T03:50:07.984847+0000","last_became_peered":"2024-03-23T03:50:07.984847+0000","last_unstale":"2024-03-23T03:50:09.083897+0000","last_undegraded":"2024-03-23T03:50:09.083897+0000","last_fullsized":"2024-03-23T03:50:09.083897+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:04:19.679117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360851+0000","last_change":"2024-03-23T03:50:07.981152+0000","last_active":"2024-03-23T03:50:09.360851+0000","last_peered":"2024-03-23T03:50:09.360851+0000","last_clean":"2024-03-23T03:50:09.360851+0000","last_became_active":"2024-03-23T03:50:07.980487+0000","last_became_peered":"2024-03-23T03:50:07.980487+0000","last_unstale":"2024-03-23T03:50:09.360851+0000","last_undegraded":"2024-03-23T03:50:09.360851+0000","last_fullsized":"2024-03-23T03:50:09.360851+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:37:14.327084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.362062+0000","last_change":"2024-03-23T03:50:07.975950+0000","last_active":"2024-03-23T03:50:09.362062+0000","last_peered":"2024-03-23T03:50:09.362062+0000","last_clean":"2024-03-23T03:50:09.362062+0000","last_became_active":"2024-03-23T03:50:07.975645+0000","last_became_peered":"2024-03-23T03:50:07.975645+0000","last_unstale":"2024-03-23T03:50:09.362062+0000","last_undegraded":"2024-03-23T03:50:09.362062+0000","last_fullsized":"2024-03-23T03:50:09.362062+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:51.983253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.364317+0000","last_change":"2024-03-23T03:50:07.987162+0000","last_active":"2024-03-23T03:50:09.364317+0000","last_peered":"2024-03-23T03:50:09.364317+0000","last_clean":"2024-03-23T03:50:09.364317+0000","last_became_active":"2024-03-23T03:50:07.985348+0000","last_became_peered":"2024-03-23T03:50:07.985348+0000","last_unstale":"2024-03-23T03:50:09.364317+0000","last_undegraded":"2024-03-23T03:50:09.364317+0000","last_fullsized":"2024-03-23T03:50:09.364317+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:51:14.771096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.963479+0000","last_change":"2024-03-23T03:50:07.975736+0000","last_active":"2024-03-23T03:50:08.963479+0000","last_peered":"2024-03-23T03:50:08.963479+0000","last_clean":"2024-03-23T03:50:08.963479+0000","last_became_active":"2024-03-23T03:50:07.975452+0000","last_became_peered":"2024-03-23T03:50:07.975452+0000","last_unstale":"2024-03-23T03:50:08.963479+0000","last_undegraded":"2024-03-23T03:50:08.963479+0000","last_fullsized":"2024-03-23T03:50:08.963479+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:04:43.377739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.961718+0000","last_change":"2024-03-23T03:50:07.982576+0000","last_active":"2024-03-23T03:50:08.961718+0000","last_peered":"2024-03-23T03:50:08.961718+0000","last_clean":"2024-03-23T03:50:08.961718+0000","last_became_active":"2024-03-23T03:50:07.982157+0000","last_became_peered":"2024-03-23T03:50:07.982157+0000","last_unstale":"2024-03-23T03:50:08.961718+0000","last_undegraded":"2024-03-23T03:50:08.961718+0000","last_fullsized":"2024-03-23T03:50:08.961718+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:22:58.863186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.362824+0000","last_change":"2024-03-23T03:50:07.990242+0000","last_active":"2024-03-23T03:50:09.362824+0000","last_peered":"2024-03-23T03:50:09.362824+0000","last_clean":"2024-03-23T03:50:09.362824+0000","last_became_active":"2024-03-23T03:50:07.989955+0000","last_became_peered":"2024-03-23T03:50:07.989955+0000","last_unstale":"2024-03-23T03:50:09.362824+0000","last_undegraded":"2024-03-23T03:50:09.362824+0000","last_fullsized":"2024-03-23T03:50:09.362824+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:52:23.829948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.963160+0000","last_change":"2024-03-23T03:50:07.976336+0000","last_active":"2024-03-23T03:50:08.963160+0000","last_peered":"2024-03-23T03:50:08.963160+0000","last_clean":"2024-03-23T03:50:08.963160+0000","last_became_active":"2024-03-23T03:50:07.976096+0000","last_became_peered":"2024-03-23T03:50:07.976096+0000","last_unstale":"2024-03-23T03:50:08.963160+0000","last_undegraded":"2024-03-23T03:50:08.963160+0000","last_fullsized":"2024-03-23T03:50:08.963160+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:00.120966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.362040+0000","last_change":"2024-03-23T03:50:07.977112+0000","last_active":"2024-03-23T03:50:09.362040+0000","last_peered":"2024-03-23T03:50:09.362040+0000","last_clean":"2024-03-23T03:50:09.362040+0000","last_became_active":"2024-03-23T03:50:07.976817+0000","last_became_peered":"2024-03-23T03:50:07.976817+0000","last_unstale":"2024-03-23T03:50:09.362040+0000","last_undegraded":"2024-03-23T03:50:09.362040+0000","last_fullsized":"2024-03-23T03:50:09.362040+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:43:48.835469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.363663+0000","last_change":"2024-03-23T03:50:07.982107+0000","last_active":"2024-03-23T03:50:09.363663+0000","last_peered":"2024-03-23T03:50:09.363663+0000","last_clean":"2024-03-23T03:50:09.363663+0000","last_became_active":"2024-03-23T03:50:07.981845+0000","last_became_peered":"2024-03-23T03:50:07.981845+0000","last_unstale":"2024-03-23T03:50:09.363663+0000","last_undegraded":"2024-03-23T03:50:09.363663+0000","last_fullsized":"2024-03-23T03:50:09.363663+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:49:01.667888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360664+0000","last_change":"2024-03-23T03:50:07.969991+0000","last_active":"2024-03-23T03:50:09.360664+0000","last_peered":"2024-03-23T03:50:09.360664+0000","last_clean":"2024-03-23T03:50:09.360664+0000","last_became_active":"2024-03-23T03:50:07.968616+0000","last_became_peered":"2024-03-23T03:50:07.968616+0000","last_unstale":"2024-03-23T03:50:09.360664+0000","last_undegraded":"2024-03-23T03:50:09.360664+0000","last_fullsized":"2024-03-23T03:50:09.360664+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:54:25.001815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.364561+0000","last_change":"2024-03-23T03:50:07.989625+0000","last_active":"2024-03-23T03:50:09.364561+0000","last_peered":"2024-03-23T03:50:09.364561+0000","last_clean":"2024-03-23T03:50:09.364561+0000","last_became_active":"2024-03-23T03:50:07.989278+0000","last_became_peered":"2024-03-23T03:50:07.989278+0000","last_unstale":"2024-03-23T03:50:09.364561+0000","last_undegraded":"2024-03-23T03:50:09.364561+0000","last_fullsized":"2024-03-23T03:50:09.364561+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:24:37.296539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.963424+0000","last_change":"2024-03-23T03:50:07.962160+0000","last_active":"2024-03-23T03:50:08.963424+0000","last_peered":"2024-03-23T03:50:08.963424+0000","last_clean":"2024-03-23T03:50:08.963424+0000","last_became_active":"2024-03-23T03:50:07.961925+0000","last_became_peered":"2024-03-23T03:50:07.961925+0000","last_unstale":"2024-03-23T03:50:08.963424+0000","last_undegraded":"2024-03-23T03:50:08.963424+0000","last_fullsized":"2024-03-23T03:50:08.963424+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:30:33.982397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.961802+0000","last_change":"2024-03-23T03:50:07.973918+0000","last_active":"2024-03-23T03:50:08.961802+0000","last_peered":"2024-03-23T03:50:08.961802+0000","last_clean":"2024-03-23T03:50:08.961802+0000","last_became_active":"2024-03-23T03:50:07.973640+0000","last_became_peered":"2024-03-23T03:50:07.973640+0000","last_unstale":"2024-03-23T03:50:08.961802+0000","last_undegraded":"2024-03-23T03:50:08.961802+0000","last_fullsized":"2024-03-23T03:50:08.961802+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:44:44.884910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.359708+0000","last_change":"2024-03-23T03:50:07.970523+0000","last_active":"2024-03-23T03:50:09.359708+0000","last_peered":"2024-03-23T03:50:09.359708+0000","last_clean":"2024-03-23T03:50:09.359708+0000","last_became_active":"2024-03-23T03:50:07.970290+0000","last_became_peered":"2024-03-23T03:50:07.970290+0000","last_unstale":"2024-03-23T03:50:09.359708+0000","last_undegraded":"2024-03-23T03:50:09.359708+0000","last_fullsized":"2024-03-23T03:50:09.359708+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:56:32.828204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.083319+0000","last_change":"2024-03-23T03:50:07.979150+0000","last_active":"2024-03-23T03:50:09.083319+0000","last_peered":"2024-03-23T03:50:09.083319+0000","last_clean":"2024-03-23T03:50:09.083319+0000","last_became_active":"2024-03-23T03:50:07.978572+0000","last_became_peered":"2024-03-23T03:50:07.978572+0000","last_unstale":"2024-03-23T03:50:09.083319+0000","last_undegraded":"2024-03-23T03:50:09.083319+0000","last_fullsized":"2024-03-23T03:50:09.083319+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:00:48.310161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.964090+0000","last_change":"2024-03-23T03:50:07.982920+0000","last_active":"2024-03-23T03:50:08.964090+0000","last_peered":"2024-03-23T03:50:08.964090+0000","last_clean":"2024-03-23T03:50:08.964090+0000","last_became_active":"2024-03-23T03:50:07.982614+0000","last_became_peered":"2024-03-23T03:50:07.982614+0000","last_unstale":"2024-03-23T03:50:08.964090+0000","last_undegraded":"2024-03-23T03:50:08.964090+0000","last_fullsized":"2024-03-23T03:50:08.964090+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:11.127549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361723+0000","last_change":"2024-03-23T03:50:07.984552+0000","last_active":"2024-03-23T03:50:09.361723+0000","last_peered":"2024-03-23T03:50:09.361723+0000","last_clean":"2024-03-23T03:50:09.361723+0000","last_became_active":"2024-03-23T03:50:07.984324+0000","last_became_peered":"2024-03-23T03:50:07.984324+0000","last_unstale":"2024-03-23T03:50:09.361723+0000","last_undegraded":"2024-03-23T03:50:09.361723+0000","last_fullsized":"2024-03-23T03:50:09.361723+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:04:27.774838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.962689+0000","last_change":"2024-03-23T03:50:07.970516+0000","last_active":"2024-03-23T03:50:08.962689+0000","last_peered":"2024-03-23T03:50:08.962689+0000","last_clean":"2024-03-23T03:50:08.962689+0000","last_became_active":"2024-03-23T03:50:07.970206+0000","last_became_peered":"2024-03-23T03:50:07.970206+0000","last_unstale":"2024-03-23T03:50:08.962689+0000","last_undegraded":"2024-03-23T03:50:08.962689+0000","last_fullsized":"2024-03-23T03:50:08.962689+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:54:37.302704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.963554+0000","last_change":"2024-03-23T03:50:07.968797+0000","last_active":"2024-03-23T03:50:08.963554+0000","last_peered":"2024-03-23T03:50:08.963554+0000","last_clean":"2024-03-23T03:50:08.963554+0000","last_became_active":"2024-03-23T03:50:07.967001+0000","last_became_peered":"2024-03-23T03:50:07.967001+0000","last_unstale":"2024-03-23T03:50:08.963554+0000","last_undegraded":"2024-03-23T03:50:08.963554+0000","last_fullsized":"2024-03-23T03:50:08.963554+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:53:38.187915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361416+0000","last_change":"2024-03-23T03:50:07.972275+0000","last_active":"2024-03-23T03:50:09.361416+0000","last_peered":"2024-03-23T03:50:09.361416+0000","last_clean":"2024-03-23T03:50:09.361416+0000","last_became_active":"2024-03-23T03:50:07.970671+0000","last_became_peered":"2024-03-23T03:50:07.970671+0000","last_unstale":"2024-03-23T03:50:09.361416+0000","last_undegraded":"2024-03-23T03:50:09.361416+0000","last_fullsized":"2024-03-23T03:50:09.361416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:24:15.374234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.963363+0000","last_change":"2024-03-23T03:50:07.981881+0000","last_active":"2024-03-23T03:50:08.963363+0000","last_peered":"2024-03-23T03:50:08.963363+0000","last_clean":"2024-03-23T03:50:08.963363+0000","last_became_active":"2024-03-23T03:50:07.981257+0000","last_became_peered":"2024-03-23T03:50:07.981257+0000","last_unstale":"2024-03-23T03:50:08.963363+0000","last_undegraded":"2024-03-23T03:50:08.963363+0000","last_fullsized":"2024-03-23T03:50:08.963363+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:38:50.990655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360935+0000","last_change":"2024-03-23T03:50:07.977132+0000","last_active":"2024-03-23T03:50:09.360935+0000","last_peered":"2024-03-23T03:50:09.360935+0000","last_clean":"2024-03-23T03:50:09.360935+0000","last_became_active":"2024-03-23T03:50:07.976415+0000","last_became_peered":"2024-03-23T03:50:07.976415+0000","last_unstale":"2024-03-23T03:50:09.360935+0000","last_undegraded":"2024-03-23T03:50:09.360935+0000","last_fullsized":"2024-03-23T03:50:09.360935+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:52:23.829948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.359597+0000","last_change":"2024-03-23T03:50:07.971606+0000","last_active":"2024-03-23T03:50:09.359597+0000","last_peered":"2024-03-23T03:50:09.359597+0000","last_clean":"2024-03-23T03:50:09.359597+0000","last_became_active":"2024-03-23T03:50:07.971262+0000","last_became_peered":"2024-03-23T03:50:07.971262+0000","last_unstale":"2024-03-23T03:50:09.359597+0000","last_undegraded":"2024-03-23T03:50:09.359597+0000","last_fullsized":"2024-03-23T03:50:09.359597+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:25:09.588118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360889+0000","last_change":"2024-03-23T03:50:07.976509+0000","last_active":"2024-03-23T03:50:09.360889+0000","last_peered":"2024-03-23T03:50:09.360889+0000","last_clean":"2024-03-23T03:50:09.360889+0000","last_became_active":"2024-03-23T03:50:07.976257+0000","last_became_peered":"2024-03-23T03:50:07.976257+0000","last_unstale":"2024-03-23T03:50:09.360889+0000","last_undegraded":"2024-03-23T03:50:09.360889+0000","last_fullsized":"2024-03-23T03:50:09.360889+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:11:44.179992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.363950+0000","last_change":"2024-03-23T03:50:07.984536+0000","last_active":"2024-03-23T03:50:09.363950+0000","last_peered":"2024-03-23T03:50:09.363950+0000","last_clean":"2024-03-23T03:50:09.363950+0000","last_became_active":"2024-03-23T03:50:07.984283+0000","last_became_peered":"2024-03-23T03:50:07.984283+0000","last_unstale":"2024-03-23T03:50:09.363950+0000","last_undegraded":"2024-03-23T03:50:09.363950+0000","last_fullsized":"2024-03-23T03:50:09.363950+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:00:48.310161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.083967+0000","last_change":"2024-03-23T03:50:07.979561+0000","last_active":"2024-03-23T03:50:09.083967+0000","last_peered":"2024-03-23T03:50:09.083967+0000","last_clean":"2024-03-23T03:50:09.083967+0000","last_became_active":"2024-03-23T03:50:07.979298+0000","last_became_peered":"2024-03-23T03:50:07.979298+0000","last_unstale":"2024-03-23T03:50:09.083967+0000","last_undegraded":"2024-03-23T03:50:09.083967+0000","last_fullsized":"2024-03-23T03:50:09.083967+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:51.983253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.963624+0000","last_change":"2024-03-23T03:50:07.977833+0000","last_active":"2024-03-23T03:50:08.963624+0000","last_peered":"2024-03-23T03:50:08.963624+0000","last_clean":"2024-03-23T03:50:08.963624+0000","last_became_active":"2024-03-23T03:50:07.977573+0000","last_became_peered":"2024-03-23T03:50:07.977573+0000","last_unstale":"2024-03-23T03:50:08.963624+0000","last_undegraded":"2024-03-23T03:50:08.963624+0000","last_fullsized":"2024-03-23T03:50:08.963624+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:26:57.947062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360493+0000","last_change":"2024-03-23T03:50:07.986075+0000","last_active":"2024-03-23T03:50:09.360493+0000","last_peered":"2024-03-23T03:50:09.360493+0000","last_clean":"2024-03-23T03:50:09.360493+0000","last_became_active":"2024-03-23T03:50:07.985787+0000","last_became_peered":"2024-03-23T03:50:07.985787+0000","last_unstale":"2024-03-23T03:50:09.360493+0000","last_undegraded":"2024-03-23T03:50:09.360493+0000","last_fullsized":"2024-03-23T03:50:09.360493+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:46:18.467082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.083282+0000","last_change":"2024-03-23T03:50:07.988386+0000","last_active":"2024-03-23T03:50:09.083282+0000","last_peered":"2024-03-23T03:50:09.083282+0000","last_clean":"2024-03-23T03:50:09.083282+0000","last_became_active":"2024-03-23T03:50:07.988110+0000","last_became_peered":"2024-03-23T03:50:07.988110+0000","last_unstale":"2024-03-23T03:50:09.083282+0000","last_undegraded":"2024-03-23T03:50:09.083282+0000","last_fullsized":"2024-03-23T03:50:09.083282+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:59.498074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361311+0000","last_change":"2024-03-23T03:50:07.986125+0000","last_active":"2024-03-23T03:50:09.361311+0000","last_peered":"2024-03-23T03:50:09.361311+0000","last_clean":"2024-03-23T03:50:09.361311+0000","last_became_active":"2024-03-23T03:50:07.985868+0000","last_became_peered":"2024-03-23T03:50:07.985868+0000","last_unstale":"2024-03-23T03:50:09.361311+0000","last_undegraded":"2024-03-23T03:50:09.361311+0000","last_fullsized":"2024-03-23T03:50:09.361311+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:28:46.329477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.083991+0000","last_change":"2024-03-23T03:50:07.985084+0000","last_active":"2024-03-23T03:50:09.083991+0000","last_peered":"2024-03-23T03:50:09.083991+0000","last_clean":"2024-03-23T03:50:09.083991+0000","last_became_active":"2024-03-23T03:50:07.984799+0000","last_became_peered":"2024-03-23T03:50:07.984799+0000","last_unstale":"2024-03-23T03:50:09.083991+0000","last_undegraded":"2024-03-23T03:50:09.083991+0000","last_fullsized":"2024-03-23T03:50:09.083991+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:28:46.329477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360693+0000","last_change":"2024-03-23T03:50:07.976108+0000","last_active":"2024-03-23T03:50:09.360693+0000","last_peered":"2024-03-23T03:50:09.360693+0000","last_clean":"2024-03-23T03:50:09.360693+0000","last_became_active":"2024-03-23T03:50:07.975827+0000","last_became_peered":"2024-03-23T03:50:07.975827+0000","last_unstale":"2024-03-23T03:50:09.360693+0000","last_undegraded":"2024-03-23T03:50:09.360693+0000","last_fullsized":"2024-03-23T03:50:09.360693+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:00:48.310161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"80'4","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.975450+0000","last_change":"2024-03-23T03:50:07.485073+0000","last_active":"2024-03-23T03:50:07.975450+0000","last_peered":"2024-03-23T03:50:07.975450+0000","last_clean":"2024-03-23T03:50:07.975450+0000","last_became_active":"2024-03-23T03:50:07.484772+0000","last_became_peered":"2024-03-23T03:50:07.484772+0000","last_unstale":"2024-03-23T03:50:07.975450+0000","last_undegraded":"2024-03-23T03:50:07.975450+0000","last_fullsized":"2024-03-23T03:50:07.975450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:32:26.890806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360280+0000","last_change":"2024-03-23T03:50:07.978522+0000","last_active":"2024-03-23T03:50:09.360280+0000","last_peered":"2024-03-23T03:50:09.360280+0000","last_clean":"2024-03-23T03:50:09.360280+0000","last_became_active":"2024-03-23T03:50:07.978228+0000","last_became_peered":"2024-03-23T03:50:07.978228+0000","last_unstale":"2024-03-23T03:50:09.360280+0000","last_undegraded":"2024-03-23T03:50:09.360280+0000","last_fullsized":"2024-03-23T03:50:09.360280+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:56:17.025451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"80'1","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.977221+0000","last_change":"2024-03-23T03:50:07.495284+0000","last_active":"2024-03-23T03:50:07.977221+0000","last_peered":"2024-03-23T03:50:07.977221+0000","last_clean":"2024-03-23T03:50:07.977221+0000","last_became_active":"2024-03-23T03:50:07.494905+0000","last_became_peered":"2024-03-23T03:50:07.494905+0000","last_unstale":"2024-03-23T03:50:07.977221+0000","last_undegraded":"2024-03-23T03:50:07.977221+0000","last_fullsized":"2024-03-23T03:50:07.977221+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:14:39.947258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361184+0000","last_change":"2024-03-23T03:50:07.983834+0000","last_active":"2024-03-23T03:50:09.361184+0000","last_peered":"2024-03-23T03:50:09.361184+0000","last_clean":"2024-03-23T03:50:09.361184+0000","last_became_active":"2024-03-23T03:50:07.982738+0000","last_became_peered":"2024-03-23T03:50:07.982738+0000","last_unstale":"2024-03-23T03:50:09.361184+0000","last_undegraded":"2024-03-23T03:50:09.361184+0000","last_fullsized":"2024-03-23T03:50:09.361184+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:03:40.282904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"80'2","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.989655+0000","last_change":"2024-03-23T03:50:07.493968+0000","last_active":"2024-03-23T03:50:07.989655+0000","last_peered":"2024-03-23T03:50:07.989655+0000","last_clean":"2024-03-23T03:50:07.989655+0000","last_became_active":"2024-03-23T03:50:07.492951+0000","last_became_peered":"2024-03-23T03:50:07.492951+0000","last_unstale":"2024-03-23T03:50:07.989655+0000","last_undegraded":"2024-03-23T03:50:07.989655+0000","last_fullsized":"2024-03-23T03:50:07.989655+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:03:17.838071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361813+0000","last_change":"2024-03-23T03:50:07.972504+0000","last_active":"2024-03-23T03:50:09.361813+0000","last_peered":"2024-03-23T03:50:09.361813+0000","last_clean":"2024-03-23T03:50:09.361813+0000","last_became_active":"2024-03-23T03:50:07.971522+0000","last_became_peered":"2024-03-23T03:50:07.971522+0000","last_unstale":"2024-03-23T03:50:09.361813+0000","last_undegraded":"2024-03-23T03:50:09.361813+0000","last_fullsized":"2024-03-23T03:50:09.361813+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:58:13.062345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"80'2","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.990621+0000","last_change":"2024-03-23T03:50:07.489667+0000","last_active":"2024-03-23T03:50:07.990621+0000","last_peered":"2024-03-23T03:50:07.990621+0000","last_clean":"2024-03-23T03:50:07.990621+0000","last_became_active":"2024-03-23T03:50:07.489392+0000","last_became_peered":"2024-03-23T03:50:07.489392+0000","last_unstale":"2024-03-23T03:50:07.990621+0000","last_undegraded":"2024-03-23T03:50:07.990621+0000","last_fullsized":"2024-03-23T03:50:07.990621+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:33:34.230046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360484+0000","last_change":"2024-03-23T03:50:07.979064+0000","last_active":"2024-03-23T03:50:09.360484+0000","last_peered":"2024-03-23T03:50:09.360484+0000","last_clean":"2024-03-23T03:50:09.360484+0000","last_became_active":"2024-03-23T03:50:07.978603+0000","last_became_peered":"2024-03-23T03:50:07.978603+0000","last_unstale":"2024-03-23T03:50:09.360484+0000","last_undegraded":"2024-03-23T03:50:09.360484+0000","last_fullsized":"2024-03-23T03:50:09.360484+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:05:05.868365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"80'1","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.973889+0000","last_change":"2024-03-23T03:50:07.487673+0000","last_active":"2024-03-23T03:50:07.973889+0000","last_peered":"2024-03-23T03:50:07.973889+0000","last_clean":"2024-03-23T03:50:07.973889+0000","last_became_active":"2024-03-23T03:50:07.487404+0000","last_became_peered":"2024-03-23T03:50:07.487404+0000","last_unstale":"2024-03-23T03:50:07.973889+0000","last_undegraded":"2024-03-23T03:50:07.973889+0000","last_fullsized":"2024-03-23T03:50:07.973889+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:33:34.230046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361097+0000","last_change":"2024-03-23T03:50:07.977636+0000","last_active":"2024-03-23T03:50:09.361097+0000","last_peered":"2024-03-23T03:50:09.361097+0000","last_clean":"2024-03-23T03:50:09.361097+0000","last_became_active":"2024-03-23T03:50:07.976207+0000","last_became_peered":"2024-03-23T03:50:07.976207+0000","last_unstale":"2024-03-23T03:50:09.361097+0000","last_undegraded":"2024-03-23T03:50:09.361097+0000","last_fullsized":"2024-03-23T03:50:09.361097+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:41:33.927440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"80'1","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.982962+0000","last_change":"2024-03-23T03:50:07.485797+0000","last_active":"2024-03-23T03:50:07.982962+0000","last_peered":"2024-03-23T03:50:07.982962+0000","last_clean":"2024-03-23T03:50:07.982962+0000","last_became_active":"2024-03-23T03:50:07.485509+0000","last_became_peered":"2024-03-23T03:50:07.485509+0000","last_unstale":"2024-03-23T03:50:07.982962+0000","last_undegraded":"2024-03-23T03:50:07.982962+0000","last_fullsized":"2024-03-23T03:50:07.982962+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:36:00.228119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361816+0000","last_change":"2024-03-23T03:50:07.979985+0000","last_active":"2024-03-23T03:50:09.361816+0000","last_peered":"2024-03-23T03:50:09.361816+0000","last_clean":"2024-03-23T03:50:09.361816+0000","last_became_active":"2024-03-23T03:50:07.979664+0000","last_became_peered":"2024-03-23T03:50:07.979664+0000","last_unstale":"2024-03-23T03:50:09.361816+0000","last_undegraded":"2024-03-23T03:50:09.361816+0000","last_fullsized":"2024-03-23T03:50:09.361816+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:28:01.892197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.959506+0000","last_change":"2024-03-23T03:50:07.486711+0000","last_active":"2024-03-23T03:50:07.959506+0000","last_peered":"2024-03-23T03:50:07.959506+0000","last_clean":"2024-03-23T03:50:07.959506+0000","last_became_active":"2024-03-23T03:50:07.485208+0000","last_became_peered":"2024-03-23T03:50:07.485208+0000","last_unstale":"2024-03-23T03:50:07.959506+0000","last_undegraded":"2024-03-23T03:50:07.959506+0000","last_fullsized":"2024-03-23T03:50:07.959506+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:06:01.399011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.363797+0000","last_change":"2024-03-23T03:50:07.991022+0000","last_active":"2024-03-23T03:50:09.363797+0000","last_peered":"2024-03-23T03:50:09.363797+0000","last_clean":"2024-03-23T03:50:09.363797+0000","last_became_active":"2024-03-23T03:50:07.990720+0000","last_became_peered":"2024-03-23T03:50:07.990720+0000","last_unstale":"2024-03-23T03:50:09.363797+0000","last_undegraded":"2024-03-23T03:50:09.363797+0000","last_fullsized":"2024-03-23T03:50:09.363797+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:35:16.507303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.948890+0000","last_change":"2024-03-23T03:50:07.489100+0000","last_active":"2024-03-23T03:50:07.948890+0000","last_peered":"2024-03-23T03:50:07.948890+0000","last_clean":"2024-03-23T03:50:07.948890+0000","last_became_active":"2024-03-23T03:50:07.488834+0000","last_became_peered":"2024-03-23T03:50:07.488834+0000","last_unstale":"2024-03-23T03:50:07.948890+0000","last_undegraded":"2024-03-23T03:50:07.948890+0000","last_fullsized":"2024-03-23T03:50:07.948890+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:59:56.498434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.359650+0000","last_change":"2024-03-23T03:50:07.978838+0000","last_active":"2024-03-23T03:50:09.359650+0000","last_peered":"2024-03-23T03:50:09.359650+0000","last_clean":"2024-03-23T03:50:09.359650+0000","last_became_active":"2024-03-23T03:50:07.978543+0000","last_became_peered":"2024-03-23T03:50:07.978543+0000","last_unstale":"2024-03-23T03:50:09.359650+0000","last_undegraded":"2024-03-23T03:50:09.359650+0000","last_fullsized":"2024-03-23T03:50:09.359650+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:00:38.798289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.971436+0000","last_change":"2024-03-23T03:50:07.494652+0000","last_active":"2024-03-23T03:50:07.971436+0000","last_peered":"2024-03-23T03:50:07.971436+0000","last_clean":"2024-03-23T03:50:07.971436+0000","last_became_active":"2024-03-23T03:50:07.494262+0000","last_became_peered":"2024-03-23T03:50:07.494262+0000","last_unstale":"2024-03-23T03:50:07.971436+0000","last_undegraded":"2024-03-23T03:50:07.971436+0000","last_fullsized":"2024-03-23T03:50:07.971436+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:44:27.821644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.364290+0000","last_change":"2024-03-23T03:50:07.986168+0000","last_active":"2024-03-23T03:50:09.364290+0000","last_peered":"2024-03-23T03:50:09.364290+0000","last_clean":"2024-03-23T03:50:09.364290+0000","last_became_active":"2024-03-23T03:50:07.985372+0000","last_became_peered":"2024-03-23T03:50:07.985372+0000","last_unstale":"2024-03-23T03:50:09.364290+0000","last_undegraded":"2024-03-23T03:50:09.364290+0000","last_fullsized":"2024-03-23T03:50:09.364290+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:37.696559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.962088+0000","last_change":"2024-03-23T03:50:07.490510+0000","last_active":"2024-03-23T03:50:07.962088+0000","last_peered":"2024-03-23T03:50:07.962088+0000","last_clean":"2024-03-23T03:50:07.962088+0000","last_became_active":"2024-03-23T03:50:07.489708+0000","last_became_peered":"2024-03-23T03:50:07.489708+0000","last_unstale":"2024-03-23T03:50:07.962088+0000","last_undegraded":"2024-03-23T03:50:07.962088+0000","last_fullsized":"2024-03-23T03:50:07.962088+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:36:57.803871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.961937+0000","last_change":"2024-03-23T03:50:07.966381+0000","last_active":"2024-03-23T03:50:08.961937+0000","last_peered":"2024-03-23T03:50:08.961937+0000","last_clean":"2024-03-23T03:50:08.961937+0000","last_became_active":"2024-03-23T03:50:07.965394+0000","last_became_peered":"2024-03-23T03:50:07.965394+0000","last_unstale":"2024-03-23T03:50:08.961937+0000","last_undegraded":"2024-03-23T03:50:08.961937+0000","last_fullsized":"2024-03-23T03:50:08.961937+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:15:51.539800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"80'3","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.973690+0000","last_change":"2024-03-23T03:50:07.486375+0000","last_active":"2024-03-23T03:50:07.973690+0000","last_peered":"2024-03-23T03:50:07.973690+0000","last_clean":"2024-03-23T03:50:07.973690+0000","last_became_active":"2024-03-23T03:50:07.485742+0000","last_became_peered":"2024-03-23T03:50:07.485742+0000","last_unstale":"2024-03-23T03:50:07.973690+0000","last_undegraded":"2024-03-23T03:50:07.973690+0000","last_fullsized":"2024-03-23T03:50:07.973690+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:18:47.365643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361652+0000","last_change":"2024-03-23T03:50:07.974510+0000","last_active":"2024-03-23T03:50:09.361652+0000","last_peered":"2024-03-23T03:50:09.361652+0000","last_clean":"2024-03-23T03:50:09.361652+0000","last_became_active":"2024-03-23T03:50:07.974244+0000","last_became_peered":"2024-03-23T03:50:07.974244+0000","last_unstale":"2024-03-23T03:50:09.361652+0000","last_undegraded":"2024-03-23T03:50:09.361652+0000","last_fullsized":"2024-03-23T03:50:09.361652+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:24:37.296539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"80'6","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.999807+0000","last_change":"2024-03-23T03:50:07.490882+0000","last_active":"2024-03-23T03:50:07.999807+0000","last_peered":"2024-03-23T03:50:07.999807+0000","last_clean":"2024-03-23T03:50:07.999807+0000","last_became_active":"2024-03-23T03:50:07.490570+0000","last_became_peered":"2024-03-23T03:50:07.490570+0000","last_unstale":"2024-03-23T03:50:07.999807+0000","last_undegraded":"2024-03-23T03:50:07.999807+0000","last_fullsized":"2024-03-23T03:50:07.999807+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:48:07.848353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.364668+0000","last_change":"2024-03-23T03:50:07.977207+0000","last_active":"2024-03-23T03:50:09.364668+0000","last_peered":"2024-03-23T03:50:09.364668+0000","last_clean":"2024-03-23T03:50:09.364668+0000","last_became_active":"2024-03-23T03:50:07.976942+0000","last_became_peered":"2024-03-23T03:50:07.976942+0000","last_unstale":"2024-03-23T03:50:09.364668+0000","last_undegraded":"2024-03-23T03:50:09.364668+0000","last_fullsized":"2024-03-23T03:50:09.364668+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:04:27.774838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"80'2","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.986274+0000","last_change":"2024-03-23T03:50:07.489295+0000","last_active":"2024-03-23T03:50:07.986274+0000","last_peered":"2024-03-23T03:50:07.986274+0000","last_clean":"2024-03-23T03:50:07.986274+0000","last_became_active":"2024-03-23T03:50:07.488973+0000","last_became_peered":"2024-03-23T03:50:07.488973+0000","last_unstale":"2024-03-23T03:50:07.986274+0000","last_undegraded":"2024-03-23T03:50:07.986274+0000","last_fullsized":"2024-03-23T03:50:07.986274+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:36:15.054606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.362795+0000","last_change":"2024-03-23T03:50:07.985931+0000","last_active":"2024-03-23T03:50:09.362795+0000","last_peered":"2024-03-23T03:50:09.362795+0000","last_clean":"2024-03-23T03:50:09.362795+0000","last_became_active":"2024-03-23T03:50:07.984873+0000","last_became_peered":"2024-03-23T03:50:07.984873+0000","last_unstale":"2024-03-23T03:50:09.362795+0000","last_undegraded":"2024-03-23T03:50:09.362795+0000","last_fullsized":"2024-03-23T03:50:09.362795+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:06:58.331998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.962430+0000","last_change":"2024-03-23T03:50:07.495545+0000","last_active":"2024-03-23T03:50:07.962430+0000","last_peered":"2024-03-23T03:50:07.962430+0000","last_clean":"2024-03-23T03:50:07.962430+0000","last_became_active":"2024-03-23T03:50:07.494844+0000","last_became_peered":"2024-03-23T03:50:07.494844+0000","last_unstale":"2024-03-23T03:50:07.962430+0000","last_undegraded":"2024-03-23T03:50:07.962430+0000","last_fullsized":"2024-03-23T03:50:07.962430+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:28:24.935685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.364537+0000","last_change":"2024-03-23T03:50:07.990806+0000","last_active":"2024-03-23T03:50:09.364537+0000","last_peered":"2024-03-23T03:50:09.364537+0000","last_clean":"2024-03-23T03:50:09.364537+0000","last_became_active":"2024-03-23T03:50:07.990503+0000","last_became_peered":"2024-03-23T03:50:07.990503+0000","last_unstale":"2024-03-23T03:50:09.364537+0000","last_undegraded":"2024-03-23T03:50:09.364537+0000","last_fullsized":"2024-03-23T03:50:09.364537+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:03:40.282904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.948311+0000","last_change":"2024-03-23T03:50:07.485542+0000","last_active":"2024-03-23T03:50:07.948311+0000","last_peered":"2024-03-23T03:50:07.948311+0000","last_clean":"2024-03-23T03:50:07.948311+0000","last_became_active":"2024-03-23T03:50:07.485234+0000","last_became_peered":"2024-03-23T03:50:07.485234+0000","last_unstale":"2024-03-23T03:50:07.948311+0000","last_undegraded":"2024-03-23T03:50:07.948311+0000","last_fullsized":"2024-03-23T03:50:07.948311+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:24:41.782032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.364022+0000","last_change":"2024-03-23T03:50:07.993058+0000","last_active":"2024-03-23T03:50:09.364022+0000","last_peered":"2024-03-23T03:50:09.364022+0000","last_clean":"2024-03-23T03:50:09.364022+0000","last_became_active":"2024-03-23T03:50:07.992770+0000","last_became_peered":"2024-03-23T03:50:07.992770+0000","last_unstale":"2024-03-23T03:50:09.364022+0000","last_undegraded":"2024-03-23T03:50:09.364022+0000","last_fullsized":"2024-03-23T03:50:09.364022+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:28.715917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"80'8","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.987497+0000","last_change":"2024-03-23T03:50:07.477358+0000","last_active":"2024-03-23T03:50:07.987497+0000","last_peered":"2024-03-23T03:50:07.987497+0000","last_clean":"2024-03-23T03:50:07.987497+0000","last_became_active":"2024-03-23T03:50:07.476820+0000","last_became_peered":"2024-03-23T03:50:07.476820+0000","last_unstale":"2024-03-23T03:50:07.987497+0000","last_undegraded":"2024-03-23T03:50:07.987497+0000","last_fullsized":"2024-03-23T03:50:07.987497+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:49:00.654026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.960903+0000","last_change":"2024-03-23T03:50:07.967082+0000","last_active":"2024-03-23T03:50:08.960903+0000","last_peered":"2024-03-23T03:50:08.960903+0000","last_clean":"2024-03-23T03:50:08.960903+0000","last_became_active":"2024-03-23T03:50:07.966795+0000","last_became_peered":"2024-03-23T03:50:07.966795+0000","last_unstale":"2024-03-23T03:50:08.960903+0000","last_undegraded":"2024-03-23T03:50:08.960903+0000","last_fullsized":"2024-03-23T03:50:08.960903+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:09.402202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"80'1","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.986814+0000","last_change":"2024-03-23T03:50:07.496080+0000","last_active":"2024-03-23T03:50:07.986814+0000","last_peered":"2024-03-23T03:50:07.986814+0000","last_clean":"2024-03-23T03:50:07.986814+0000","last_became_active":"2024-03-23T03:50:07.494410+0000","last_became_peered":"2024-03-23T03:50:07.494410+0000","last_unstale":"2024-03-23T03:50:07.986814+0000","last_undegraded":"2024-03-23T03:50:07.986814+0000","last_fullsized":"2024-03-23T03:50:07.986814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:59:01.344309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.963138+0000","last_change":"2024-03-23T03:50:07.986511+0000","last_active":"2024-03-23T03:50:08.963138+0000","last_peered":"2024-03-23T03:50:08.963138+0000","last_clean":"2024-03-23T03:50:08.963138+0000","last_became_active":"2024-03-23T03:50:07.986224+0000","last_became_peered":"2024-03-23T03:50:07.986224+0000","last_unstale":"2024-03-23T03:50:08.963138+0000","last_undegraded":"2024-03-23T03:50:08.963138+0000","last_fullsized":"2024-03-23T03:50:08.963138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:29:23.097203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"80'2","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.979122+0000","last_change":"2024-03-23T03:50:07.489105+0000","last_active":"2024-03-23T03:50:07.979122+0000","last_peered":"2024-03-23T03:50:07.979122+0000","last_clean":"2024-03-23T03:50:07.979122+0000","last_became_active":"2024-03-23T03:50:07.488633+0000","last_became_peered":"2024-03-23T03:50:07.488633+0000","last_unstale":"2024-03-23T03:50:07.979122+0000","last_undegraded":"2024-03-23T03:50:07.979122+0000","last_fullsized":"2024-03-23T03:50:07.979122+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:19:21.548516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.963270+0000","last_change":"2024-03-23T03:50:07.972568+0000","last_active":"2024-03-23T03:50:08.963270+0000","last_peered":"2024-03-23T03:50:08.963270+0000","last_clean":"2024-03-23T03:50:08.963270+0000","last_became_active":"2024-03-23T03:50:07.972316+0000","last_became_peered":"2024-03-23T03:50:07.972316+0000","last_unstale":"2024-03-23T03:50:08.963270+0000","last_undegraded":"2024-03-23T03:50:08.963270+0000","last_fullsized":"2024-03-23T03:50:08.963270+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:20:18.613759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"80'1","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.984267+0000","last_change":"2024-03-23T03:50:07.490256+0000","last_active":"2024-03-23T03:50:07.984267+0000","last_peered":"2024-03-23T03:50:07.984267+0000","last_clean":"2024-03-23T03:50:07.984267+0000","last_became_active":"2024-03-23T03:50:07.489709+0000","last_became_peered":"2024-03-23T03:50:07.489709+0000","last_unstale":"2024-03-23T03:50:07.984267+0000","last_undegraded":"2024-03-23T03:50:07.984267+0000","last_fullsized":"2024-03-23T03:50:07.984267+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:54:59.757010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.962255+0000","last_change":"2024-03-23T03:50:07.970717+0000","last_active":"2024-03-23T03:50:08.962255+0000","last_peered":"2024-03-23T03:50:08.962255+0000","last_clean":"2024-03-23T03:50:08.962255+0000","last_became_active":"2024-03-23T03:50:07.970435+0000","last_became_peered":"2024-03-23T03:50:07.970435+0000","last_unstale":"2024-03-23T03:50:08.962255+0000","last_undegraded":"2024-03-23T03:50:08.962255+0000","last_fullsized":"2024-03-23T03:50:08.962255+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:13.975703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.962584+0000","last_change":"2024-03-23T03:50:07.478328+0000","last_active":"2024-03-23T03:50:07.962584+0000","last_peered":"2024-03-23T03:50:07.962584+0000","last_clean":"2024-03-23T03:50:07.962584+0000","last_became_active":"2024-03-23T03:50:07.478059+0000","last_became_peered":"2024-03-23T03:50:07.478059+0000","last_unstale":"2024-03-23T03:50:07.962584+0000","last_undegraded":"2024-03-23T03:50:07.962584+0000","last_fullsized":"2024-03-23T03:50:07.962584+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:03:17.838071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360585+0000","last_change":"2024-03-23T03:50:07.978890+0000","last_active":"2024-03-23T03:50:09.360585+0000","last_peered":"2024-03-23T03:50:09.360585+0000","last_clean":"2024-03-23T03:50:09.360585+0000","last_became_active":"2024-03-23T03:50:07.978236+0000","last_became_peered":"2024-03-23T03:50:07.978236+0000","last_unstale":"2024-03-23T03:50:09.360585+0000","last_undegraded":"2024-03-23T03:50:09.360585+0000","last_fullsized":"2024-03-23T03:50:09.360585+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:41:32.947352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"80'2","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.974863+0000","last_change":"2024-03-23T03:50:07.487422+0000","last_active":"2024-03-23T03:50:07.974863+0000","last_peered":"2024-03-23T03:50:07.974863+0000","last_clean":"2024-03-23T03:50:07.974863+0000","last_became_active":"2024-03-23T03:50:07.487176+0000","last_became_peered":"2024-03-23T03:50:07.487176+0000","last_unstale":"2024-03-23T03:50:07.974863+0000","last_undegraded":"2024-03-23T03:50:07.974863+0000","last_fullsized":"2024-03-23T03:50:07.974863+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:41:32.913577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360906+0000","last_change":"2024-03-23T03:50:07.980838+0000","last_active":"2024-03-23T03:50:09.360906+0000","last_peered":"2024-03-23T03:50:09.360906+0000","last_clean":"2024-03-23T03:50:09.360906+0000","last_became_active":"2024-03-23T03:50:07.980465+0000","last_became_peered":"2024-03-23T03:50:07.980465+0000","last_unstale":"2024-03-23T03:50:09.360906+0000","last_undegraded":"2024-03-23T03:50:09.360906+0000","last_fullsized":"2024-03-23T03:50:09.360906+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:44:23.205733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.957056+0000","last_change":"2024-03-23T03:50:07.490733+0000","last_active":"2024-03-23T03:50:07.957056+0000","last_peered":"2024-03-23T03:50:07.957056+0000","last_clean":"2024-03-23T03:50:07.957056+0000","last_became_active":"2024-03-23T03:50:07.490461+0000","last_became_peered":"2024-03-23T03:50:07.490461+0000","last_unstale":"2024-03-23T03:50:07.957056+0000","last_undegraded":"2024-03-23T03:50:07.957056+0000","last_fullsized":"2024-03-23T03:50:07.957056+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:46:17.453220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.364245+0000","last_change":"2024-03-23T03:50:07.982642+0000","last_active":"2024-03-23T03:50:09.364245+0000","last_peered":"2024-03-23T03:50:09.364245+0000","last_clean":"2024-03-23T03:50:09.364245+0000","last_became_active":"2024-03-23T03:50:07.978117+0000","last_became_peered":"2024-03-23T03:50:07.978117+0000","last_unstale":"2024-03-23T03:50:09.364245+0000","last_undegraded":"2024-03-23T03:50:09.364245+0000","last_fullsized":"2024-03-23T03:50:09.364245+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:28:01.892197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.957788+0000","last_change":"2024-03-23T03:50:07.494059+0000","last_active":"2024-03-23T03:50:07.957788+0000","last_peered":"2024-03-23T03:50:07.957788+0000","last_clean":"2024-03-23T03:50:07.957788+0000","last_became_active":"2024-03-23T03:50:07.493829+0000","last_became_peered":"2024-03-23T03:50:07.493829+0000","last_unstale":"2024-03-23T03:50:07.957788+0000","last_undegraded":"2024-03-23T03:50:07.957788+0000","last_fullsized":"2024-03-23T03:50:07.957788+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:54:23.987953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360990+0000","last_change":"2024-03-23T03:50:07.982204+0000","last_active":"2024-03-23T03:50:09.360990+0000","last_peered":"2024-03-23T03:50:09.360990+0000","last_clean":"2024-03-23T03:50:09.360990+0000","last_became_active":"2024-03-23T03:50:07.981108+0000","last_became_peered":"2024-03-23T03:50:07.981108+0000","last_unstale":"2024-03-23T03:50:09.360990+0000","last_undegraded":"2024-03-23T03:50:09.360990+0000","last_fullsized":"2024-03-23T03:50:09.360990+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:00:18.878974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"80'2","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.974784+0000","last_change":"2024-03-23T03:50:07.485213+0000","last_active":"2024-03-23T03:50:07.974784+0000","last_peered":"2024-03-23T03:50:07.974784+0000","last_clean":"2024-03-23T03:50:07.974784+0000","last_became_active":"2024-03-23T03:50:07.484905+0000","last_became_peered":"2024-03-23T03:50:07.484905+0000","last_unstale":"2024-03-23T03:50:07.974784+0000","last_undegraded":"2024-03-23T03:50:07.974784+0000","last_fullsized":"2024-03-23T03:50:07.974784+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:14:48.590038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.362905+0000","last_change":"2024-03-23T03:50:07.986303+0000","last_active":"2024-03-23T03:50:09.362905+0000","last_peered":"2024-03-23T03:50:09.362905+0000","last_clean":"2024-03-23T03:50:09.362905+0000","last_became_active":"2024-03-23T03:50:07.985696+0000","last_became_peered":"2024-03-23T03:50:07.985696+0000","last_unstale":"2024-03-23T03:50:09.362905+0000","last_undegraded":"2024-03-23T03:50:09.362905+0000","last_fullsized":"2024-03-23T03:50:09.362905+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:58:13.062345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.965824+0000","last_change":"2024-03-23T03:50:07.482606+0000","last_active":"2024-03-23T03:50:07.965824+0000","last_peered":"2024-03-23T03:50:07.965824+0000","last_clean":"2024-03-23T03:50:07.965824+0000","last_became_active":"2024-03-23T03:50:07.482285+0000","last_became_peered":"2024-03-23T03:50:07.482285+0000","last_unstale":"2024-03-23T03:50:07.965824+0000","last_undegraded":"2024-03-23T03:50:07.965824+0000","last_fullsized":"2024-03-23T03:50:07.965824+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:51:13.757234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361429+0000","last_change":"2024-03-23T03:50:07.986046+0000","last_active":"2024-03-23T03:50:09.361429+0000","last_peered":"2024-03-23T03:50:09.361429+0000","last_clean":"2024-03-23T03:50:09.361429+0000","last_became_active":"2024-03-23T03:50:07.982397+0000","last_became_peered":"2024-03-23T03:50:07.982397+0000","last_unstale":"2024-03-23T03:50:09.361429+0000","last_undegraded":"2024-03-23T03:50:09.361429+0000","last_fullsized":"2024-03-23T03:50:09.361429+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:06:58.331998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.971106+0000","last_change":"2024-03-23T03:50:07.494355+0000","last_active":"2024-03-23T03:50:07.971106+0000","last_peered":"2024-03-23T03:50:07.971106+0000","last_clean":"2024-03-23T03:50:07.971106+0000","last_became_active":"2024-03-23T03:50:07.494076+0000","last_became_peered":"2024-03-23T03:50:07.494076+0000","last_unstale":"2024-03-23T03:50:07.971106+0000","last_undegraded":"2024-03-23T03:50:07.971106+0000","last_fullsized":"2024-03-23T03:50:07.971106+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:01:47.992165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360157+0000","last_change":"2024-03-23T03:50:07.981581+0000","last_active":"2024-03-23T03:50:09.360157+0000","last_peered":"2024-03-23T03:50:09.360157+0000","last_clean":"2024-03-23T03:50:09.360157+0000","last_became_active":"2024-03-23T03:50:07.978579+0000","last_became_peered":"2024-03-23T03:50:07.978579+0000","last_unstale":"2024-03-23T03:50:09.360157+0000","last_undegraded":"2024-03-23T03:50:09.360157+0000","last_fullsized":"2024-03-23T03:50:09.360157+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:58:35.441218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.963492+0000","last_change":"2024-03-23T03:50:07.493956+0000","last_active":"2024-03-23T03:50:07.963492+0000","last_peered":"2024-03-23T03:50:07.963492+0000","last_clean":"2024-03-23T03:50:07.963492+0000","last_became_active":"2024-03-23T03:50:07.493656+0000","last_became_peered":"2024-03-23T03:50:07.493656+0000","last_unstale":"2024-03-23T03:50:07.963492+0000","last_undegraded":"2024-03-23T03:50:07.963492+0000","last_fullsized":"2024-03-23T03:50:07.963492+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:58.116627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.363387+0000","last_change":"2024-03-23T03:50:07.992710+0000","last_active":"2024-03-23T03:50:09.363387+0000","last_peered":"2024-03-23T03:50:09.363387+0000","last_clean":"2024-03-23T03:50:09.363387+0000","last_became_active":"2024-03-23T03:50:07.990216+0000","last_became_peered":"2024-03-23T03:50:07.990216+0000","last_unstale":"2024-03-23T03:50:09.363387+0000","last_undegraded":"2024-03-23T03:50:09.363387+0000","last_fullsized":"2024-03-23T03:50:09.363387+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:24:15.374234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.975346+0000","last_change":"2024-03-23T03:50:07.487283+0000","last_active":"2024-03-23T03:50:07.975346+0000","last_peered":"2024-03-23T03:50:07.975346+0000","last_clean":"2024-03-23T03:50:07.975346+0000","last_became_active":"2024-03-23T03:50:07.486961+0000","last_became_peered":"2024-03-23T03:50:07.486961+0000","last_unstale":"2024-03-23T03:50:07.975346+0000","last_undegraded":"2024-03-23T03:50:07.975346+0000","last_fullsized":"2024-03-23T03:50:07.975346+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:19:21.548516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.363077+0000","last_change":"2024-03-23T03:50:07.990718+0000","last_active":"2024-03-23T03:50:09.363077+0000","last_peered":"2024-03-23T03:50:09.363077+0000","last_clean":"2024-03-23T03:50:09.363077+0000","last_became_active":"2024-03-23T03:50:07.989661+0000","last_became_peered":"2024-03-23T03:50:07.989661+0000","last_unstale":"2024-03-23T03:50:09.363077+0000","last_undegraded":"2024-03-23T03:50:09.363077+0000","last_fullsized":"2024-03-23T03:50:09.363077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:43:48.835469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.950493+0000","last_change":"2024-03-23T03:50:07.490922+0000","last_active":"2024-03-23T03:50:07.950493+0000","last_peered":"2024-03-23T03:50:07.950493+0000","last_clean":"2024-03-23T03:50:07.950493+0000","last_became_active":"2024-03-23T03:50:07.490610+0000","last_became_peered":"2024-03-23T03:50:07.490610+0000","last_unstale":"2024-03-23T03:50:07.950493+0000","last_undegraded":"2024-03-23T03:50:07.950493+0000","last_fullsized":"2024-03-23T03:50:07.950493+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:36:27.444641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361918+0000","last_change":"2024-03-23T03:50:07.974413+0000","last_active":"2024-03-23T03:50:09.361918+0000","last_peered":"2024-03-23T03:50:09.361918+0000","last_clean":"2024-03-23T03:50:09.361918+0000","last_became_active":"2024-03-23T03:50:07.973059+0000","last_became_peered":"2024-03-23T03:50:07.973059+0000","last_unstale":"2024-03-23T03:50:09.361918+0000","last_undegraded":"2024-03-23T03:50:09.361918+0000","last_fullsized":"2024-03-23T03:50:09.361918+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:59.498074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.975256+0000","last_change":"2024-03-23T03:50:07.486181+0000","last_active":"2024-03-23T03:50:07.975256+0000","last_peered":"2024-03-23T03:50:07.975256+0000","last_clean":"2024-03-23T03:50:07.975256+0000","last_became_active":"2024-03-23T03:50:07.485883+0000","last_became_peered":"2024-03-23T03:50:07.485883+0000","last_unstale":"2024-03-23T03:50:07.975256+0000","last_undegraded":"2024-03-23T03:50:07.975256+0000","last_fullsized":"2024-03-23T03:50:07.975256+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:13.215153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.362020+0000","last_change":"2024-03-23T03:50:07.974134+0000","last_active":"2024-03-23T03:50:09.362020+0000","last_peered":"2024-03-23T03:50:09.362020+0000","last_clean":"2024-03-23T03:50:09.362020+0000","last_became_active":"2024-03-23T03:50:07.972868+0000","last_became_peered":"2024-03-23T03:50:07.972868+0000","last_unstale":"2024-03-23T03:50:09.362020+0000","last_undegraded":"2024-03-23T03:50:09.362020+0000","last_fullsized":"2024-03-23T03:50:09.362020+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:37.696559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.961901+0000","last_change":"2024-03-23T03:50:07.483623+0000","last_active":"2024-03-23T03:50:07.961901+0000","last_peered":"2024-03-23T03:50:07.961901+0000","last_clean":"2024-03-23T03:50:07.961901+0000","last_became_active":"2024-03-23T03:50:07.483354+0000","last_became_peered":"2024-03-23T03:50:07.483354+0000","last_unstale":"2024-03-23T03:50:07.961901+0000","last_undegraded":"2024-03-23T03:50:07.961901+0000","last_fullsized":"2024-03-23T03:50:07.961901+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:01:47.992165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.083860+0000","last_change":"2024-03-23T03:50:07.984423+0000","last_active":"2024-03-23T03:50:09.083860+0000","last_peered":"2024-03-23T03:50:09.083860+0000","last_clean":"2024-03-23T03:50:09.083860+0000","last_became_active":"2024-03-23T03:50:07.983811+0000","last_became_peered":"2024-03-23T03:50:07.983811+0000","last_unstale":"2024-03-23T03:50:09.083860+0000","last_undegraded":"2024-03-23T03:50:09.083860+0000","last_fullsized":"2024-03-23T03:50:09.083860+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:24:37.296539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.963515+0000","last_change":"2024-03-23T03:50:07.493185+0000","last_active":"2024-03-23T03:50:07.963515+0000","last_peered":"2024-03-23T03:50:07.963515+0000","last_clean":"2024-03-23T03:50:07.963515+0000","last_became_active":"2024-03-23T03:50:07.492771+0000","last_became_peered":"2024-03-23T03:50:07.492771+0000","last_unstale":"2024-03-23T03:50:07.963515+0000","last_undegraded":"2024-03-23T03:50:07.963515+0000","last_fullsized":"2024-03-23T03:50:07.963515+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:11:43.166129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":213,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.967011+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:50:07.967011+0000","last_peered":"2024-03-23T03:50:07.967011+0000","last_clean":"2024-03-23T03:50:07.967011+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:50:07.967011+0000","last_undegraded":"2024-03-23T03:50:07.967011+0000","last_fullsized":"2024-03-23T03:50:07.967011+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542189,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28580,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343260,"statfs":{"total":96636764160,"available":96607498240,"internally_reserved":0,"allocated":1585152,"data_stored":1035263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.079,"15min":1.069},"min":{"1min":0.68200000000000005,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.573,"5min":1.635,"15min":1.635},"last":1.5209999999999999},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0449999999999999,"15min":1.048},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.417,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.91400000000000003}]},{"osd":1,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0549999999999999,"15min":1.044},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.544,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.399},{"interface":"front","average":{"1min":1.085,"5min":1.2330000000000001,"15min":1.3009999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.3999999999999999,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":0.81000000000000005}]},{"osd":2,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.139,"15min":1.1559999999999999},"min":{"1min":0.76400000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.7909999999999999,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.345,"15min":1.421},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.4630000000000001,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.464}]},{"osd":3,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.355,"15min":1.4370000000000001},"min":{"1min":0.873,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.397,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.27,"5min":1.3380000000000001,"15min":1.3580000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":1.5840000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.1539999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.266,"15min":1.2669999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.6299999999999999,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.022},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.357,"15min":1.375},"min":{"1min":0.77900000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0619999999999998,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.351}]},{"osd":5,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.3999999999999999,"15min":1.4219999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.6539999999999999,"5min":2.645,"15min":2.645},"last":1.272},{"interface":"front","average":{"1min":1.339,"5min":1.4370000000000001,"15min":1.4530000000000001},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":1.7050000000000001,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.1160000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.5229999999999999,"15min":1.5589999999999999},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":1.8200000000000001,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.304},{"interface":"front","average":{"1min":1.415,"5min":1.4930000000000001,"15min":1.5209999999999999},"min":{"1min":1.1319999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":2.1509999999999998,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.1930000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542187,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":999424,"data_stored":446437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.0309999999999999,"15min":1.04},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.3979999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0429999999999999,"15min":1.038},"min":{"1min":0.72099999999999997,"5min":0.62,"15min":0.62},"max":{"1min":1.6120000000000001,"5min":1.946,"15min":1.946},"last":1.516}]},{"osd":1,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.01},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.129,"15min":1.1559999999999999},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.4810000000000001,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.5369999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.075,"15min":1.069},"min":{"1min":0.73699999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.5449999999999999,"5min":2.044,"15min":2.044},"last":1.411},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1950000000000001,"15min":1.218},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":1.1180000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2629999999999999,"15min":1.2849999999999999},"min":{"1min":0.76800000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.4910000000000001,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2549999999999999,"15min":1.2649999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.6319999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.2130000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3759999999999999,"15min":1.3839999999999999},"min":{"1min":0.67700000000000005,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.73,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.363,"15min":1.383},"min":{"1min":0.63400000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.762,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.6299999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.415,"15min":1.4359999999999999},"min":{"1min":0.71099999999999997,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.6919999999999999,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.478},{"interface":"front","average":{"1min":1.456,"5min":1.5249999999999999,"15min":1.5469999999999999},"min":{"1min":0.79700000000000004,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8149999999999999,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.615}]},{"osd":7,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.4059999999999999,"15min":1.4119999999999999},"min":{"1min":0.77600000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.802,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.573},{"interface":"front","average":{"1min":1.423,"5min":1.5129999999999999,"15min":1.5409999999999999},"min":{"1min":0.84099999999999997,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.74,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.262}]}]},{"osd":5,"up_from":14,"seq":60129542187,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":991232,"data_stored":444907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.1759999999999999,"15min":1.214},"min":{"1min":0.67900000000000005,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.6779999999999999,"5min":1.806,"15min":1.806},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.157,"15min":1.181},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.554,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.048}]},{"osd":1,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.81699999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.5489999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.083},{"interface":"front","average":{"1min":1.026,"5min":1.1599999999999999,"15min":1.1919999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4219999999999999,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.98999999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.298,"15min":1.3540000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5429999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3340000000000001,"15min":1.3779999999999999},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.772,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.321}]},{"osd":3,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.21,"15min":1.2470000000000001},"min":{"1min":0.70599999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.653,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.236},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.393,"15min":1.452},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.6779999999999999,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.1699999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.3420000000000001,"15min":1.4119999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.671,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.282,"5min":1.4119999999999999,"15min":1.444},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.77,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.294}]},{"osd":6,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.458,"15min":1.514},"min":{"1min":0.76700000000000002,"5min":0.63,"15min":0.63},"max":{"1min":1.873,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.208},{"interface":"front","average":{"1min":1.464,"5min":1.5940000000000001,"15min":1.6559999999999999},"min":{"1min":0.86199999999999999,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.956,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.389}]},{"osd":7,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.538,"15min":1.607},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.673,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.133},{"interface":"front","average":{"1min":1.411,"5min":1.6200000000000001,"15min":1.696},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.855,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.357}]}]},{"osd":4,"up_from":14,"seq":60129542188,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":1007616,"data_stored":446541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.002,"15min":0.97999999999999998},"min":{"1min":0.67800000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.03,"5min":1.0149999999999999,"15min":1.012},"min":{"1min":0.81200000000000006,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3069999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.56100000000000005}]},{"osd":1,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.0700000000000001,"15min":1.05},"min":{"1min":0.74299999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.704,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.147,"5min":1.0880000000000001,"15min":1.0640000000000001},"min":{"1min":0.50600000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.47099999999999997}]},{"osd":2,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2949999999999999,"15min":1.325},"min":{"1min":0.81599999999999995,"5min":0.625,"15min":0.625},"max":{"1min":1.569,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.29,"5min":1.3100000000000001,"15min":1.3080000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.8680000000000001,"5min":2.181,"15min":2.181},"last":0.59899999999999998}]},{"osd":3,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1339999999999999,"15min":1.1299999999999999},"min":{"1min":0.73199999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.26,"15min":1.278},"min":{"1min":0.82899999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.5720000000000001,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":0.73799999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.278,"15min":1.274},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.298,"15min":1.2829999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.2349999999999999,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":0.48899999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.415,"15min":1.4239999999999999},"min":{"1min":1.143,"5min":0.748,"15min":0.748},"max":{"1min":1.833,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.478,"5min":1.425,"15min":1.4039999999999999},"min":{"1min":0.745,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":0.54800000000000004}]},{"osd":7,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.385,"15min":1.371},"min":{"1min":1.0880000000000001,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":2.3479999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.4970000000000001,"15min":1.5009999999999999},"min":{"1min":1.056,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.974,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":0.80000000000000004}]}]},{"osd":3,"up_from":14,"seq":60129542188,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":991232,"data_stored":445455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.83899999999999997,"15min":0.86099999999999999},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.256,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.446,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.76400000000000001}]},{"osd":1,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.92800000000000005,"15min":0.97199999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.997,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":1.0960000000000001},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.034,"15min":1.0720000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.712,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":1.327}]},{"osd":2,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.006,"15min":1.052},"min":{"1min":0.435,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.613,"5min":1.823,"15min":1.823},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.1319999999999999,"15min":1.153},"min":{"1min":0.48199999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.6020000000000001,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":1.4119999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.2310000000000001,"15min":1.2450000000000001},"min":{"1min":0.67600000000000005,"5min":0.625,"15min":0.625},"max":{"1min":1.3859999999999999,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":1.226},{"interface":"front","average":{"1min":1.151,"5min":1.377,"15min":1.4370000000000001},"min":{"1min":0.77100000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.4770000000000001,"5min":4.819,"15min":4.819},"last":1.2729999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.603,"15min":1.4470000000000001},"min":{"1min":0.75,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.0299999999999998,"5min":30.228000000000002,"15min":30.228000000000002},"last":1.474},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.3320000000000001,"15min":1.355},"min":{"1min":0.91400000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.637,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":1.387}]},{"osd":6,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.448,"15min":1.52},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.5860000000000001,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.53},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3999999999999999,"15min":1.397},"min":{"1min":0.90100000000000002,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.1429999999999998,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":1.6240000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.395,"15min":1.4219999999999999},"min":{"1min":0.81200000000000006,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.71,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.577,"15min":1.601},"min":{"1min":0.73299999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.629,"5min":10.663,"15min":10.663},"last":1.5840000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542188,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":999424,"data_stored":445615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.84999999999999998,"15min":0.88100000000000001},"min":{"1min":0.318,"5min":0.307,"15min":0.307},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.1120000000000001},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.88700000000000001,"15min":0.94299999999999995},"min":{"1min":0.53500000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.23,"5min":2.226,"15min":2.226},"last":0.56999999999999995}]},{"osd":1,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.96899999999999997,"15min":0.995},"min":{"1min":0.621,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.5109999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.88,"15min":0.91700000000000004},"min":{"1min":0.53700000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.282,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.73599999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.98499999999999999,"15min":1.026},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.4750000000000001,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0549999999999999,"15min":1.085},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3380000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.222}]},{"osd":4,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.2090000000000001,"15min":1.236},"min":{"1min":0.96399999999999997,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.488,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":0.876},{"interface":"front","average":{"1min":1.087,"5min":1.179,"15min":1.214},"min":{"1min":0.66300000000000003,"5min":0.627,"15min":0.627},"max":{"1min":1.47,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.93100000000000005}]},{"osd":5,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.3360000000000001,"15min":1.3759999999999999},"min":{"1min":0.76200000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.304},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.304,"15min":1.349},"min":{"1min":0.75,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.5880000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.081}]},{"osd":6,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.268,"15min":1.3400000000000001},"min":{"1min":0.82199999999999995,"5min":0.753,"15min":0.753},"max":{"1min":1.5109999999999999,"5min":2.456,"15min":2.456},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.359,"5min":1.4390000000000001,"15min":1.4890000000000001},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.3959999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.335,"15min":1.379},"min":{"1min":0.82699999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.6870000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.3460000000000001,"15min":1.4099999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5900000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.022}]}]},{"osd":1,"up_from":14,"seq":60129542187,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":444985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.81599999999999995,"15min":0.87},"min":{"1min":0.40200000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.1819999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.82999999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.92000000000000004}]},{"osd":2,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.92800000000000005,"15min":0.96599999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.4570000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.3100000000000001},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.94899999999999995,"15min":0.98599999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.4750000000000001,"5min":1.625,"15min":1.625},"last":0.81000000000000005}]},{"osd":3,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.117,"15min":1.117},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.7929999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.873},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.99399999999999999,"15min":1.0189999999999999},"min":{"1min":0.57399999999999995,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.5600000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.208}]},{"osd":4,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.232,"15min":1.292},"min":{"1min":0.69199999999999995,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.752,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.2749999999999999,"15min":1.3149999999999999},"min":{"1min":0.82599999999999996,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.575,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.05}]},{"osd":5,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.2729999999999999,"15min":1.329},"min":{"1min":0.82399999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.7470000000000001,"5min":3.593,"15min":3.593},"last":1.369},{"interface":"front","average":{"1min":1.276,"5min":1.3700000000000001,"15min":1.4399999999999999},"min":{"1min":0.76200000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6719999999999999,"5min":2.359,"15min":2.359},"last":1.109}]},{"osd":6,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.349,"15min":1.405},"min":{"1min":0.79300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.1709999999999998,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.147,"5min":1.302,"15min":1.3580000000000001},"min":{"1min":0.77800000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.696,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.4530000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.3759999999999999,"15min":1.4379999999999999},"min":{"1min":0.81100000000000005,"5min":0.627,"15min":0.627},"max":{"1min":1.7649999999999999,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.4219999999999999,"15min":1.506},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.7569999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.4159999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542187,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28564,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343276,"statfs":{"total":96636764160,"available":96607514624,"internally_reserved":0,"allocated":1581056,"data_stored":1035241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.90000000000000002,"15min":0.94699999999999995},"min":{"1min":0.435,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.1599999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.98899999999999999},"min":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.3640000000000001,"5min":1.506,"15min":1.506},"last":0.497}]},{"osd":2,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.95799999999999996,"15min":1.0009999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1279999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.91900000000000004,"5min":1.0640000000000001,"15min":1.105},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.4350000000000001,"5min":1.901,"15min":1.901},"last":0.45300000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.97499999999999998,"15min":1.01},"min":{"1min":0.48999999999999999,"5min":0.433,"15min":0.433},"max":{"1min":1.5189999999999999,"5min":1.972,"15min":1.972},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.103,"15min":1.159},"min":{"1min":0.51400000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3180000000000001,"5min":2.298,"15min":2.298},"last":0.72399999999999998}]},{"osd":4,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.304,"15min":1.3420000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.599,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.252,"15min":1.28},"min":{"1min":0.83099999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.4990000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.81799999999999995}]},{"osd":5,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.3340000000000001,"15min":1.3779999999999999},"min":{"1min":0.878,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.4450000000000001,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.171},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.403,"15min":1.4470000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.49,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.208}]},{"osd":6,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.46,"15min":1.538},"min":{"1min":0.85799999999999998,"5min":0.747,"15min":0.747},"max":{"1min":1.5940000000000001,"5min":3.032,"15min":3.032},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.4419999999999999,"15min":1.498},"min":{"1min":0.98199999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.5740000000000001,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":0.874}]},{"osd":7,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.46,"15min":1.5129999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.7110000000000001,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.329,"5min":1.3939999999999999,"15min":1.4299999999999999},"min":{"1min":0.871,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.008,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.044}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:50:17.332 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:50:17.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.550+0000 7fbd64e00640 1 Processor -- start 2024-03-23T03:50:17.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.550+0000 7fbd64e00640 1 -- start start 2024-03-23T03:50:17.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.552+0000 7fbd64e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fbd6015b560 0x7fbd6015b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:17.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.552+0000 7fbd64e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbd6015d410 0x7fbd600b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:17.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.552+0000 7fbd64e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd6006c6f0 con 0x7fbd600b39b0 2024-03-23T03:50:17.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.552+0000 7fbd64e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd600b4370 con 0x7fbd6015d410 2024-03-23T03:50:17.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.552+0000 7fbd64e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd6006c9d0 con 0x7fbd6015b560 2024-03-23T03:50:17.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.552+0000 7fbd5e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbd6015d410 0x7fbd600b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:17.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.552+0000 7fbd5e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbd6015d410 0x7fbd600b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38738/0 (socket says 172.21.15.102:38738) 2024-03-23T03:50:17.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.552+0000 7fbd5e800640 1 -- 172.21.15.102:0/2544790259 learned_addr learned my addr 172.21.15.102:0/2544790259 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:17.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.552+0000 7fbd5e800640 1 -- 172.21.15.102:0/2544790259 >> v2:172.21.15.173:3300/0 conn(0x7fbd6015b560 msgr2=0x7fbd6015b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:17.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.552+0000 7fbd5de00640 1 --2- 172.21.15.102:0/2544790259 >> v2:172.21.15.173:3300/0 conn(0x7fbd6015b560 0x7fbd6015b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:17.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.553+0000 7fbd5e800640 1 --2- 172.21.15.102:0/2544790259 >> v2:172.21.15.173:3300/0 conn(0x7fbd6015b560 0x7fbd6015b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:17.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.553+0000 7fbd5e800640 1 -- 172.21.15.102:0/2544790259 >> v1:172.21.15.173:6789/0 conn(0x7fbd600b39b0 legacy=0x7fbd6015e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:17.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.553+0000 7fbd5e800640 1 -- 172.21.15.102:0/2544790259 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6006ccb0 con 0x7fbd6015d410 2024-03-23T03:50:17.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.553+0000 7fbd5de00640 1 --2- 172.21.15.102:0/2544790259 >> v2:172.21.15.173:3300/0 conn(0x7fbd6015b560 0x7fbd6015b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:17.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.554+0000 7fbd5e800640 1 --2- 172.21.15.102:0/2544790259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd6015d410 0x7fbd600b3e30 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fbd60071470 tx=0x7fbd5802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1411bbb6e32482 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:17.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.554+0000 7fbd5d400640 1 -- 172.21.15.102:0/2544790259 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5802c9f0 con 0x7fbd6015d410 2024-03-23T03:50:17.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.554+0000 7fbd5d400640 1 -- 172.21.15.102:0/2544790259 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5802cb50 con 0x7fbd6015d410 2024-03-23T03:50:17.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.554+0000 7fbd5d400640 1 -- 172.21.15.102:0/2544790259 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd580343f0 con 0x7fbd6015d410 2024-03-23T03:50:17.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.555+0000 7fbd64e00640 1 -- 172.21.15.102:0/2544790259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd6015d410 msgr2=0x7fbd600b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:17.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.555+0000 7fbd64e00640 1 --2- 172.21.15.102:0/2544790259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd6015d410 0x7fbd600b3e30 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fbd60071470 tx=0x7fbd5802be50 comp rx=0 tx=0).stop 2024-03-23T03:50:17.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.555+0000 7fbd64e00640 1 -- 172.21.15.102:0/2544790259 shutdown_connections 2024-03-23T03:50:17.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.555+0000 7fbd64e00640 1 --2- 172.21.15.102:0/2544790259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd6015d410 0x7fbd600b3e30 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:17.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.555+0000 7fbd64e00640 1 --2- 172.21.15.102:0/2544790259 >> v2:172.21.15.173:3300/0 conn(0x7fbd6015b560 0x7fbd6015b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:17.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.555+0000 7fbd64e00640 1 -- 172.21.15.102:0/2544790259 >> 172.21.15.102:0/2544790259 conn(0x7fbd6006a3d0 msgr2=0x7fbd600b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:17.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.555+0000 7fbd64e00640 1 -- 172.21.15.102:0/2544790259 shutdown_connections 2024-03-23T03:50:17.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.555+0000 7fbd64e00640 1 -- 172.21.15.102:0/2544790259 wait complete. 2024-03-23T03:50:17.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.557+0000 7fbd64e00640 1 Processor -- start 2024-03-23T03:50:17.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.557+0000 7fbd64e00640 1 -- start start 2024-03-23T03:50:17.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.558+0000 7fbd64e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd600b39b0 0x7fbd60153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:17.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.558+0000 7fbd64e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd6015b560 0x7fbd600cba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:17.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.558+0000 7fbd64e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd6015d410 0x7fbd601e5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:17.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.558+0000 7fbd64e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd60153720 con 0x7fbd6015d410 2024-03-23T03:50:17.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.558+0000 7fbd64e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd60152ae0 con 0x7fbd600b39b0 2024-03-23T03:50:17.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.558+0000 7fbd64e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd60152d80 con 0x7fbd6015b560 2024-03-23T03:50:17.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.558+0000 7fbd5e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd6015d410 0x7fbd601e5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:17.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.559+0000 7fbd5e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd6015d410 0x7fbd601e5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38740/0 (socket says 172.21.15.102:38740) 2024-03-23T03:50:17.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.559+0000 7fbd5e800640 1 -- 172.21.15.102:0/603957881 learned_addr learned my addr 172.21.15.102:0/603957881 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:17.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.559+0000 7fbd5de00640 1 --2- 172.21.15.102:0/603957881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd600b39b0 0x7fbd60153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:17.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.559+0000 7fbd57e00640 1 --2- 172.21.15.102:0/603957881 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd6015b560 0x7fbd600cba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:17.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.559+0000 7fbd5e800640 1 -- 172.21.15.102:0/603957881 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd6015b560 msgr2=0x7fbd600cba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:17.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.559+0000 7fbd5e800640 1 --2- 172.21.15.102:0/603957881 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd6015b560 0x7fbd600cba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:17.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.559+0000 7fbd5e800640 1 -- 172.21.15.102:0/603957881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd600b39b0 msgr2=0x7fbd60153b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:17.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.559+0000 7fbd5e800640 1 --2- 172.21.15.102:0/603957881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd600b39b0 0x7fbd60153b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:17.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.559+0000 7fbd5e800640 1 -- 172.21.15.102:0/603957881 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd58009b30 con 0x7fbd6015d410 2024-03-23T03:50:17.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.559+0000 7fbd5de00640 1 --2- 172.21.15.102:0/603957881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd600b39b0 0x7fbd60153b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:17.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.559+0000 7fbd57e00640 1 --2- 172.21.15.102:0/603957881 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd6015b560 0x7fbd600cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:17.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.559+0000 7fbd5e800640 1 --2- 172.21.15.102:0/603957881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd6015d410 0x7fbd601e5fc0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fbd58036040 tx=0x7fbd58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:17.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.560+0000 7fbd56000640 1 -- 172.21.15.102:0/603957881 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5803e450 con 0x7fbd6015d410 2024-03-23T03:50:17.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.560+0000 7fbd56000640 1 -- 172.21.15.102:0/603957881 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5803e5d0 con 0x7fbd6015d410 2024-03-23T03:50:17.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.560+0000 7fbd56000640 1 -- 172.21.15.102:0/603957881 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5803e980 con 0x7fbd6015d410 2024-03-23T03:50:17.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.560+0000 7fbd64e00640 1 -- 172.21.15.102:0/603957881 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd60153060 con 0x7fbd6015d410 2024-03-23T03:50:17.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.560+0000 7fbd64e00640 1 -- 172.21.15.102:0/603957881 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd601424f0 con 0x7fbd6015d410 2024-03-23T03:50:17.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.561+0000 7fbd64e00640 1 -- 172.21.15.102:0/603957881 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd60153860 con 0x7fbd6015d410 2024-03-23T03:50:17.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.562+0000 7fbd56000640 1 -- 172.21.15.102:0/603957881 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbd5803eae0 con 0x7fbd6015d410 2024-03-23T03:50:17.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.563+0000 7fbd56000640 1 --2- 172.21.15.102:0/603957881 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd3c06ac00 0x7fbd3c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:17.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.563+0000 7fbd56000640 1 -- 172.21.15.102:0/603957881 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbd580ac340 con 0x7fbd6015d410 2024-03-23T03:50:17.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.563+0000 7fbd5de00640 1 --2- 172.21.15.102:0/603957881 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd3c06ac00 0x7fbd3c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:17.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.564+0000 7fbd5de00640 1 --2- 172.21.15.102:0/603957881 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd3c06ac00 0x7fbd3c06d0c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbd60142920 tx=0x7fbd50008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:17.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.569+0000 7fbd56000640 1 -- 172.21.15.102:0/603957881 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbd60153860 con 0x7fbd6015d410 2024-03-23T03:50:17.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.762+0000 7fbd64e00640 1 -- 172.21.15.102:0/603957881 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbd6015be80 con 0x7fbd3c06ac00 2024-03-23T03:50:17.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.782+0000 7fbd56000640 1 -- 172.21.15.102:0/603957881 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287058 (secure 0 0 0) 0x7fbd5000c040 con 0x7fbd3c06ac00 2024-03-23T03:50:17.787 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:17.789 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:50:17.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.788+0000 7fbd64e00640 1 -- 172.21.15.102:0/603957881 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd3c06ac00 msgr2=0x7fbd3c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:17.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.788+0000 7fbd64e00640 1 --2- 172.21.15.102:0/603957881 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd3c06ac00 0x7fbd3c06d0c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbd60142920 tx=0x7fbd50008040 comp rx=0 tx=0).stop 2024-03-23T03:50:17.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.788+0000 7fbd64e00640 1 -- 172.21.15.102:0/603957881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd6015d410 msgr2=0x7fbd601e5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:17.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.788+0000 7fbd64e00640 1 --2- 172.21.15.102:0/603957881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd6015d410 0x7fbd601e5fc0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fbd58036040 tx=0x7fbd58004a40 comp rx=0 tx=0).stop 2024-03-23T03:50:17.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.788+0000 7fbd64e00640 1 -- 172.21.15.102:0/603957881 shutdown_connections 2024-03-23T03:50:17.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.788+0000 7fbd64e00640 1 --2- 172.21.15.102:0/603957881 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd3c06ac00 0x7fbd3c06d0c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:17.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.788+0000 7fbd64e00640 1 --2- 172.21.15.102:0/603957881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd6015d410 0x7fbd601e5fc0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:17.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.788+0000 7fbd64e00640 1 --2- 172.21.15.102:0/603957881 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd6015b560 0x7fbd600cba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:17.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.788+0000 7fbd64e00640 1 --2- 172.21.15.102:0/603957881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd600b39b0 0x7fbd60153b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:17.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.788+0000 7fbd64e00640 1 -- 172.21.15.102:0/603957881 >> 172.21.15.102:0/603957881 conn(0x7fbd6006a3d0 msgr2=0x7fbd600b0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:17.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.788+0000 7fbd64e00640 1 -- 172.21.15.102:0/603957881 shutdown_connections 2024-03-23T03:50:17.791 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:17.788+0000 7fbd64e00640 1 -- 172.21.15.102:0/603957881 wait complete. 2024-03-23T03:50:17.809 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":179,"stamp":"2024-03-23T03:50:15.779671+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":188,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":233328,"kb_used_data":8936,"kb_used_omap":12,"kb_used_meta":224243,"kb_avail":754741392,"statfs":{"total":773094113280,"available":772855185408,"internally_reserved":0,"allocated":9150464,"data_stored":4744444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":229625402},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":52,"apply_latency_ms":52,"commit_latency_ns":52000000,"apply_latency_ns":52000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002664"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.083897+0000","last_change":"2024-03-23T03:50:07.985162+0000","last_active":"2024-03-23T03:50:09.083897+0000","last_peered":"2024-03-23T03:50:09.083897+0000","last_clean":"2024-03-23T03:50:09.083897+0000","last_became_active":"2024-03-23T03:50:07.984847+0000","last_became_peered":"2024-03-23T03:50:07.984847+0000","last_unstale":"2024-03-23T03:50:09.083897+0000","last_undegraded":"2024-03-23T03:50:09.083897+0000","last_fullsized":"2024-03-23T03:50:09.083897+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:04:19.679117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360851+0000","last_change":"2024-03-23T03:50:07.981152+0000","last_active":"2024-03-23T03:50:09.360851+0000","last_peered":"2024-03-23T03:50:09.360851+0000","last_clean":"2024-03-23T03:50:09.360851+0000","last_became_active":"2024-03-23T03:50:07.980487+0000","last_became_peered":"2024-03-23T03:50:07.980487+0000","last_unstale":"2024-03-23T03:50:09.360851+0000","last_undegraded":"2024-03-23T03:50:09.360851+0000","last_fullsized":"2024-03-23T03:50:09.360851+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:37:14.327084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.362062+0000","last_change":"2024-03-23T03:50:07.975950+0000","last_active":"2024-03-23T03:50:09.362062+0000","last_peered":"2024-03-23T03:50:09.362062+0000","last_clean":"2024-03-23T03:50:09.362062+0000","last_became_active":"2024-03-23T03:50:07.975645+0000","last_became_peered":"2024-03-23T03:50:07.975645+0000","last_unstale":"2024-03-23T03:50:09.362062+0000","last_undegraded":"2024-03-23T03:50:09.362062+0000","last_fullsized":"2024-03-23T03:50:09.362062+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:51.983253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.364317+0000","last_change":"2024-03-23T03:50:07.987162+0000","last_active":"2024-03-23T03:50:09.364317+0000","last_peered":"2024-03-23T03:50:09.364317+0000","last_clean":"2024-03-23T03:50:09.364317+0000","last_became_active":"2024-03-23T03:50:07.985348+0000","last_became_peered":"2024-03-23T03:50:07.985348+0000","last_unstale":"2024-03-23T03:50:09.364317+0000","last_undegraded":"2024-03-23T03:50:09.364317+0000","last_fullsized":"2024-03-23T03:50:09.364317+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:51:14.771096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.963479+0000","last_change":"2024-03-23T03:50:07.975736+0000","last_active":"2024-03-23T03:50:08.963479+0000","last_peered":"2024-03-23T03:50:08.963479+0000","last_clean":"2024-03-23T03:50:08.963479+0000","last_became_active":"2024-03-23T03:50:07.975452+0000","last_became_peered":"2024-03-23T03:50:07.975452+0000","last_unstale":"2024-03-23T03:50:08.963479+0000","last_undegraded":"2024-03-23T03:50:08.963479+0000","last_fullsized":"2024-03-23T03:50:08.963479+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:04:43.377739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.961718+0000","last_change":"2024-03-23T03:50:07.982576+0000","last_active":"2024-03-23T03:50:08.961718+0000","last_peered":"2024-03-23T03:50:08.961718+0000","last_clean":"2024-03-23T03:50:08.961718+0000","last_became_active":"2024-03-23T03:50:07.982157+0000","last_became_peered":"2024-03-23T03:50:07.982157+0000","last_unstale":"2024-03-23T03:50:08.961718+0000","last_undegraded":"2024-03-23T03:50:08.961718+0000","last_fullsized":"2024-03-23T03:50:08.961718+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:22:58.863186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.362824+0000","last_change":"2024-03-23T03:50:07.990242+0000","last_active":"2024-03-23T03:50:09.362824+0000","last_peered":"2024-03-23T03:50:09.362824+0000","last_clean":"2024-03-23T03:50:09.362824+0000","last_became_active":"2024-03-23T03:50:07.989955+0000","last_became_peered":"2024-03-23T03:50:07.989955+0000","last_unstale":"2024-03-23T03:50:09.362824+0000","last_undegraded":"2024-03-23T03:50:09.362824+0000","last_fullsized":"2024-03-23T03:50:09.362824+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:52:23.829948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.963160+0000","last_change":"2024-03-23T03:50:07.976336+0000","last_active":"2024-03-23T03:50:08.963160+0000","last_peered":"2024-03-23T03:50:08.963160+0000","last_clean":"2024-03-23T03:50:08.963160+0000","last_became_active":"2024-03-23T03:50:07.976096+0000","last_became_peered":"2024-03-23T03:50:07.976096+0000","last_unstale":"2024-03-23T03:50:08.963160+0000","last_undegraded":"2024-03-23T03:50:08.963160+0000","last_fullsized":"2024-03-23T03:50:08.963160+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:00.120966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.362040+0000","last_change":"2024-03-23T03:50:07.977112+0000","last_active":"2024-03-23T03:50:09.362040+0000","last_peered":"2024-03-23T03:50:09.362040+0000","last_clean":"2024-03-23T03:50:09.362040+0000","last_became_active":"2024-03-23T03:50:07.976817+0000","last_became_peered":"2024-03-23T03:50:07.976817+0000","last_unstale":"2024-03-23T03:50:09.362040+0000","last_undegraded":"2024-03-23T03:50:09.362040+0000","last_fullsized":"2024-03-23T03:50:09.362040+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:43:48.835469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.363663+0000","last_change":"2024-03-23T03:50:07.982107+0000","last_active":"2024-03-23T03:50:09.363663+0000","last_peered":"2024-03-23T03:50:09.363663+0000","last_clean":"2024-03-23T03:50:09.363663+0000","last_became_active":"2024-03-23T03:50:07.981845+0000","last_became_peered":"2024-03-23T03:50:07.981845+0000","last_unstale":"2024-03-23T03:50:09.363663+0000","last_undegraded":"2024-03-23T03:50:09.363663+0000","last_fullsized":"2024-03-23T03:50:09.363663+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:49:01.667888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360664+0000","last_change":"2024-03-23T03:50:07.969991+0000","last_active":"2024-03-23T03:50:09.360664+0000","last_peered":"2024-03-23T03:50:09.360664+0000","last_clean":"2024-03-23T03:50:09.360664+0000","last_became_active":"2024-03-23T03:50:07.968616+0000","last_became_peered":"2024-03-23T03:50:07.968616+0000","last_unstale":"2024-03-23T03:50:09.360664+0000","last_undegraded":"2024-03-23T03:50:09.360664+0000","last_fullsized":"2024-03-23T03:50:09.360664+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:54:25.001815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.364561+0000","last_change":"2024-03-23T03:50:07.989625+0000","last_active":"2024-03-23T03:50:09.364561+0000","last_peered":"2024-03-23T03:50:09.364561+0000","last_clean":"2024-03-23T03:50:09.364561+0000","last_became_active":"2024-03-23T03:50:07.989278+0000","last_became_peered":"2024-03-23T03:50:07.989278+0000","last_unstale":"2024-03-23T03:50:09.364561+0000","last_undegraded":"2024-03-23T03:50:09.364561+0000","last_fullsized":"2024-03-23T03:50:09.364561+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:24:37.296539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.963424+0000","last_change":"2024-03-23T03:50:07.962160+0000","last_active":"2024-03-23T03:50:08.963424+0000","last_peered":"2024-03-23T03:50:08.963424+0000","last_clean":"2024-03-23T03:50:08.963424+0000","last_became_active":"2024-03-23T03:50:07.961925+0000","last_became_peered":"2024-03-23T03:50:07.961925+0000","last_unstale":"2024-03-23T03:50:08.963424+0000","last_undegraded":"2024-03-23T03:50:08.963424+0000","last_fullsized":"2024-03-23T03:50:08.963424+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:30:33.982397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.961802+0000","last_change":"2024-03-23T03:50:07.973918+0000","last_active":"2024-03-23T03:50:08.961802+0000","last_peered":"2024-03-23T03:50:08.961802+0000","last_clean":"2024-03-23T03:50:08.961802+0000","last_became_active":"2024-03-23T03:50:07.973640+0000","last_became_peered":"2024-03-23T03:50:07.973640+0000","last_unstale":"2024-03-23T03:50:08.961802+0000","last_undegraded":"2024-03-23T03:50:08.961802+0000","last_fullsized":"2024-03-23T03:50:08.961802+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:44:44.884910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.359708+0000","last_change":"2024-03-23T03:50:07.970523+0000","last_active":"2024-03-23T03:50:09.359708+0000","last_peered":"2024-03-23T03:50:09.359708+0000","last_clean":"2024-03-23T03:50:09.359708+0000","last_became_active":"2024-03-23T03:50:07.970290+0000","last_became_peered":"2024-03-23T03:50:07.970290+0000","last_unstale":"2024-03-23T03:50:09.359708+0000","last_undegraded":"2024-03-23T03:50:09.359708+0000","last_fullsized":"2024-03-23T03:50:09.359708+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:56:32.828204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.083319+0000","last_change":"2024-03-23T03:50:07.979150+0000","last_active":"2024-03-23T03:50:09.083319+0000","last_peered":"2024-03-23T03:50:09.083319+0000","last_clean":"2024-03-23T03:50:09.083319+0000","last_became_active":"2024-03-23T03:50:07.978572+0000","last_became_peered":"2024-03-23T03:50:07.978572+0000","last_unstale":"2024-03-23T03:50:09.083319+0000","last_undegraded":"2024-03-23T03:50:09.083319+0000","last_fullsized":"2024-03-23T03:50:09.083319+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:00:48.310161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.964090+0000","last_change":"2024-03-23T03:50:07.982920+0000","last_active":"2024-03-23T03:50:08.964090+0000","last_peered":"2024-03-23T03:50:08.964090+0000","last_clean":"2024-03-23T03:50:08.964090+0000","last_became_active":"2024-03-23T03:50:07.982614+0000","last_became_peered":"2024-03-23T03:50:07.982614+0000","last_unstale":"2024-03-23T03:50:08.964090+0000","last_undegraded":"2024-03-23T03:50:08.964090+0000","last_fullsized":"2024-03-23T03:50:08.964090+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:11.127549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361723+0000","last_change":"2024-03-23T03:50:07.984552+0000","last_active":"2024-03-23T03:50:09.361723+0000","last_peered":"2024-03-23T03:50:09.361723+0000","last_clean":"2024-03-23T03:50:09.361723+0000","last_became_active":"2024-03-23T03:50:07.984324+0000","last_became_peered":"2024-03-23T03:50:07.984324+0000","last_unstale":"2024-03-23T03:50:09.361723+0000","last_undegraded":"2024-03-23T03:50:09.361723+0000","last_fullsized":"2024-03-23T03:50:09.361723+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:04:27.774838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.962689+0000","last_change":"2024-03-23T03:50:07.970516+0000","last_active":"2024-03-23T03:50:08.962689+0000","last_peered":"2024-03-23T03:50:08.962689+0000","last_clean":"2024-03-23T03:50:08.962689+0000","last_became_active":"2024-03-23T03:50:07.970206+0000","last_became_peered":"2024-03-23T03:50:07.970206+0000","last_unstale":"2024-03-23T03:50:08.962689+0000","last_undegraded":"2024-03-23T03:50:08.962689+0000","last_fullsized":"2024-03-23T03:50:08.962689+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:54:37.302704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.963554+0000","last_change":"2024-03-23T03:50:07.968797+0000","last_active":"2024-03-23T03:50:08.963554+0000","last_peered":"2024-03-23T03:50:08.963554+0000","last_clean":"2024-03-23T03:50:08.963554+0000","last_became_active":"2024-03-23T03:50:07.967001+0000","last_became_peered":"2024-03-23T03:50:07.967001+0000","last_unstale":"2024-03-23T03:50:08.963554+0000","last_undegraded":"2024-03-23T03:50:08.963554+0000","last_fullsized":"2024-03-23T03:50:08.963554+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:53:38.187915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361416+0000","last_change":"2024-03-23T03:50:07.972275+0000","last_active":"2024-03-23T03:50:09.361416+0000","last_peered":"2024-03-23T03:50:09.361416+0000","last_clean":"2024-03-23T03:50:09.361416+0000","last_became_active":"2024-03-23T03:50:07.970671+0000","last_became_peered":"2024-03-23T03:50:07.970671+0000","last_unstale":"2024-03-23T03:50:09.361416+0000","last_undegraded":"2024-03-23T03:50:09.361416+0000","last_fullsized":"2024-03-23T03:50:09.361416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:24:15.374234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.963363+0000","last_change":"2024-03-23T03:50:07.981881+0000","last_active":"2024-03-23T03:50:08.963363+0000","last_peered":"2024-03-23T03:50:08.963363+0000","last_clean":"2024-03-23T03:50:08.963363+0000","last_became_active":"2024-03-23T03:50:07.981257+0000","last_became_peered":"2024-03-23T03:50:07.981257+0000","last_unstale":"2024-03-23T03:50:08.963363+0000","last_undegraded":"2024-03-23T03:50:08.963363+0000","last_fullsized":"2024-03-23T03:50:08.963363+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:38:50.990655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360935+0000","last_change":"2024-03-23T03:50:07.977132+0000","last_active":"2024-03-23T03:50:09.360935+0000","last_peered":"2024-03-23T03:50:09.360935+0000","last_clean":"2024-03-23T03:50:09.360935+0000","last_became_active":"2024-03-23T03:50:07.976415+0000","last_became_peered":"2024-03-23T03:50:07.976415+0000","last_unstale":"2024-03-23T03:50:09.360935+0000","last_undegraded":"2024-03-23T03:50:09.360935+0000","last_fullsized":"2024-03-23T03:50:09.360935+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:52:23.829948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.359597+0000","last_change":"2024-03-23T03:50:07.971606+0000","last_active":"2024-03-23T03:50:09.359597+0000","last_peered":"2024-03-23T03:50:09.359597+0000","last_clean":"2024-03-23T03:50:09.359597+0000","last_became_active":"2024-03-23T03:50:07.971262+0000","last_became_peered":"2024-03-23T03:50:07.971262+0000","last_unstale":"2024-03-23T03:50:09.359597+0000","last_undegraded":"2024-03-23T03:50:09.359597+0000","last_fullsized":"2024-03-23T03:50:09.359597+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:25:09.588118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360889+0000","last_change":"2024-03-23T03:50:07.976509+0000","last_active":"2024-03-23T03:50:09.360889+0000","last_peered":"2024-03-23T03:50:09.360889+0000","last_clean":"2024-03-23T03:50:09.360889+0000","last_became_active":"2024-03-23T03:50:07.976257+0000","last_became_peered":"2024-03-23T03:50:07.976257+0000","last_unstale":"2024-03-23T03:50:09.360889+0000","last_undegraded":"2024-03-23T03:50:09.360889+0000","last_fullsized":"2024-03-23T03:50:09.360889+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:11:44.179992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.363950+0000","last_change":"2024-03-23T03:50:07.984536+0000","last_active":"2024-03-23T03:50:09.363950+0000","last_peered":"2024-03-23T03:50:09.363950+0000","last_clean":"2024-03-23T03:50:09.363950+0000","last_became_active":"2024-03-23T03:50:07.984283+0000","last_became_peered":"2024-03-23T03:50:07.984283+0000","last_unstale":"2024-03-23T03:50:09.363950+0000","last_undegraded":"2024-03-23T03:50:09.363950+0000","last_fullsized":"2024-03-23T03:50:09.363950+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:00:48.310161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.083967+0000","last_change":"2024-03-23T03:50:07.979561+0000","last_active":"2024-03-23T03:50:09.083967+0000","last_peered":"2024-03-23T03:50:09.083967+0000","last_clean":"2024-03-23T03:50:09.083967+0000","last_became_active":"2024-03-23T03:50:07.979298+0000","last_became_peered":"2024-03-23T03:50:07.979298+0000","last_unstale":"2024-03-23T03:50:09.083967+0000","last_undegraded":"2024-03-23T03:50:09.083967+0000","last_fullsized":"2024-03-23T03:50:09.083967+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:54:51.983253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.963624+0000","last_change":"2024-03-23T03:50:07.977833+0000","last_active":"2024-03-23T03:50:08.963624+0000","last_peered":"2024-03-23T03:50:08.963624+0000","last_clean":"2024-03-23T03:50:08.963624+0000","last_became_active":"2024-03-23T03:50:07.977573+0000","last_became_peered":"2024-03-23T03:50:07.977573+0000","last_unstale":"2024-03-23T03:50:08.963624+0000","last_undegraded":"2024-03-23T03:50:08.963624+0000","last_fullsized":"2024-03-23T03:50:08.963624+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:26:57.947062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360493+0000","last_change":"2024-03-23T03:50:07.986075+0000","last_active":"2024-03-23T03:50:09.360493+0000","last_peered":"2024-03-23T03:50:09.360493+0000","last_clean":"2024-03-23T03:50:09.360493+0000","last_became_active":"2024-03-23T03:50:07.985787+0000","last_became_peered":"2024-03-23T03:50:07.985787+0000","last_unstale":"2024-03-23T03:50:09.360493+0000","last_undegraded":"2024-03-23T03:50:09.360493+0000","last_fullsized":"2024-03-23T03:50:09.360493+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:46:18.467082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.083282+0000","last_change":"2024-03-23T03:50:07.988386+0000","last_active":"2024-03-23T03:50:09.083282+0000","last_peered":"2024-03-23T03:50:09.083282+0000","last_clean":"2024-03-23T03:50:09.083282+0000","last_became_active":"2024-03-23T03:50:07.988110+0000","last_became_peered":"2024-03-23T03:50:07.988110+0000","last_unstale":"2024-03-23T03:50:09.083282+0000","last_undegraded":"2024-03-23T03:50:09.083282+0000","last_fullsized":"2024-03-23T03:50:09.083282+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:59.498074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361311+0000","last_change":"2024-03-23T03:50:07.986125+0000","last_active":"2024-03-23T03:50:09.361311+0000","last_peered":"2024-03-23T03:50:09.361311+0000","last_clean":"2024-03-23T03:50:09.361311+0000","last_became_active":"2024-03-23T03:50:07.985868+0000","last_became_peered":"2024-03-23T03:50:07.985868+0000","last_unstale":"2024-03-23T03:50:09.361311+0000","last_undegraded":"2024-03-23T03:50:09.361311+0000","last_fullsized":"2024-03-23T03:50:09.361311+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:28:46.329477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.083991+0000","last_change":"2024-03-23T03:50:07.985084+0000","last_active":"2024-03-23T03:50:09.083991+0000","last_peered":"2024-03-23T03:50:09.083991+0000","last_clean":"2024-03-23T03:50:09.083991+0000","last_became_active":"2024-03-23T03:50:07.984799+0000","last_became_peered":"2024-03-23T03:50:07.984799+0000","last_unstale":"2024-03-23T03:50:09.083991+0000","last_undegraded":"2024-03-23T03:50:09.083991+0000","last_fullsized":"2024-03-23T03:50:09.083991+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:28:46.329477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360693+0000","last_change":"2024-03-23T03:50:07.976108+0000","last_active":"2024-03-23T03:50:09.360693+0000","last_peered":"2024-03-23T03:50:09.360693+0000","last_clean":"2024-03-23T03:50:09.360693+0000","last_became_active":"2024-03-23T03:50:07.975827+0000","last_became_peered":"2024-03-23T03:50:07.975827+0000","last_unstale":"2024-03-23T03:50:09.360693+0000","last_undegraded":"2024-03-23T03:50:09.360693+0000","last_fullsized":"2024-03-23T03:50:09.360693+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:00:48.310161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"80'4","reported_seq":15,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.975450+0000","last_change":"2024-03-23T03:50:07.485073+0000","last_active":"2024-03-23T03:50:07.975450+0000","last_peered":"2024-03-23T03:50:07.975450+0000","last_clean":"2024-03-23T03:50:07.975450+0000","last_became_active":"2024-03-23T03:50:07.484772+0000","last_became_peered":"2024-03-23T03:50:07.484772+0000","last_unstale":"2024-03-23T03:50:07.975450+0000","last_undegraded":"2024-03-23T03:50:07.975450+0000","last_fullsized":"2024-03-23T03:50:07.975450+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:32:26.890806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360280+0000","last_change":"2024-03-23T03:50:07.978522+0000","last_active":"2024-03-23T03:50:09.360280+0000","last_peered":"2024-03-23T03:50:09.360280+0000","last_clean":"2024-03-23T03:50:09.360280+0000","last_became_active":"2024-03-23T03:50:07.978228+0000","last_became_peered":"2024-03-23T03:50:07.978228+0000","last_unstale":"2024-03-23T03:50:09.360280+0000","last_undegraded":"2024-03-23T03:50:09.360280+0000","last_fullsized":"2024-03-23T03:50:09.360280+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:56:17.025451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"80'1","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.977221+0000","last_change":"2024-03-23T03:50:07.495284+0000","last_active":"2024-03-23T03:50:07.977221+0000","last_peered":"2024-03-23T03:50:07.977221+0000","last_clean":"2024-03-23T03:50:07.977221+0000","last_became_active":"2024-03-23T03:50:07.494905+0000","last_became_peered":"2024-03-23T03:50:07.494905+0000","last_unstale":"2024-03-23T03:50:07.977221+0000","last_undegraded":"2024-03-23T03:50:07.977221+0000","last_fullsized":"2024-03-23T03:50:07.977221+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:14:39.947258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361184+0000","last_change":"2024-03-23T03:50:07.983834+0000","last_active":"2024-03-23T03:50:09.361184+0000","last_peered":"2024-03-23T03:50:09.361184+0000","last_clean":"2024-03-23T03:50:09.361184+0000","last_became_active":"2024-03-23T03:50:07.982738+0000","last_became_peered":"2024-03-23T03:50:07.982738+0000","last_unstale":"2024-03-23T03:50:09.361184+0000","last_undegraded":"2024-03-23T03:50:09.361184+0000","last_fullsized":"2024-03-23T03:50:09.361184+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:03:40.282904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"80'2","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.989655+0000","last_change":"2024-03-23T03:50:07.493968+0000","last_active":"2024-03-23T03:50:07.989655+0000","last_peered":"2024-03-23T03:50:07.989655+0000","last_clean":"2024-03-23T03:50:07.989655+0000","last_became_active":"2024-03-23T03:50:07.492951+0000","last_became_peered":"2024-03-23T03:50:07.492951+0000","last_unstale":"2024-03-23T03:50:07.989655+0000","last_undegraded":"2024-03-23T03:50:07.989655+0000","last_fullsized":"2024-03-23T03:50:07.989655+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:03:17.838071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361813+0000","last_change":"2024-03-23T03:50:07.972504+0000","last_active":"2024-03-23T03:50:09.361813+0000","last_peered":"2024-03-23T03:50:09.361813+0000","last_clean":"2024-03-23T03:50:09.361813+0000","last_became_active":"2024-03-23T03:50:07.971522+0000","last_became_peered":"2024-03-23T03:50:07.971522+0000","last_unstale":"2024-03-23T03:50:09.361813+0000","last_undegraded":"2024-03-23T03:50:09.361813+0000","last_fullsized":"2024-03-23T03:50:09.361813+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:58:13.062345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"80'2","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.990621+0000","last_change":"2024-03-23T03:50:07.489667+0000","last_active":"2024-03-23T03:50:07.990621+0000","last_peered":"2024-03-23T03:50:07.990621+0000","last_clean":"2024-03-23T03:50:07.990621+0000","last_became_active":"2024-03-23T03:50:07.489392+0000","last_became_peered":"2024-03-23T03:50:07.489392+0000","last_unstale":"2024-03-23T03:50:07.990621+0000","last_undegraded":"2024-03-23T03:50:07.990621+0000","last_fullsized":"2024-03-23T03:50:07.990621+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:33:34.230046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360484+0000","last_change":"2024-03-23T03:50:07.979064+0000","last_active":"2024-03-23T03:50:09.360484+0000","last_peered":"2024-03-23T03:50:09.360484+0000","last_clean":"2024-03-23T03:50:09.360484+0000","last_became_active":"2024-03-23T03:50:07.978603+0000","last_became_peered":"2024-03-23T03:50:07.978603+0000","last_unstale":"2024-03-23T03:50:09.360484+0000","last_undegraded":"2024-03-23T03:50:09.360484+0000","last_fullsized":"2024-03-23T03:50:09.360484+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:05:05.868365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"80'1","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.973889+0000","last_change":"2024-03-23T03:50:07.487673+0000","last_active":"2024-03-23T03:50:07.973889+0000","last_peered":"2024-03-23T03:50:07.973889+0000","last_clean":"2024-03-23T03:50:07.973889+0000","last_became_active":"2024-03-23T03:50:07.487404+0000","last_became_peered":"2024-03-23T03:50:07.487404+0000","last_unstale":"2024-03-23T03:50:07.973889+0000","last_undegraded":"2024-03-23T03:50:07.973889+0000","last_fullsized":"2024-03-23T03:50:07.973889+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:33:34.230046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361097+0000","last_change":"2024-03-23T03:50:07.977636+0000","last_active":"2024-03-23T03:50:09.361097+0000","last_peered":"2024-03-23T03:50:09.361097+0000","last_clean":"2024-03-23T03:50:09.361097+0000","last_became_active":"2024-03-23T03:50:07.976207+0000","last_became_peered":"2024-03-23T03:50:07.976207+0000","last_unstale":"2024-03-23T03:50:09.361097+0000","last_undegraded":"2024-03-23T03:50:09.361097+0000","last_fullsized":"2024-03-23T03:50:09.361097+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:41:33.927440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"80'1","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.982962+0000","last_change":"2024-03-23T03:50:07.485797+0000","last_active":"2024-03-23T03:50:07.982962+0000","last_peered":"2024-03-23T03:50:07.982962+0000","last_clean":"2024-03-23T03:50:07.982962+0000","last_became_active":"2024-03-23T03:50:07.485509+0000","last_became_peered":"2024-03-23T03:50:07.485509+0000","last_unstale":"2024-03-23T03:50:07.982962+0000","last_undegraded":"2024-03-23T03:50:07.982962+0000","last_fullsized":"2024-03-23T03:50:07.982962+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:36:00.228119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361816+0000","last_change":"2024-03-23T03:50:07.979985+0000","last_active":"2024-03-23T03:50:09.361816+0000","last_peered":"2024-03-23T03:50:09.361816+0000","last_clean":"2024-03-23T03:50:09.361816+0000","last_became_active":"2024-03-23T03:50:07.979664+0000","last_became_peered":"2024-03-23T03:50:07.979664+0000","last_unstale":"2024-03-23T03:50:09.361816+0000","last_undegraded":"2024-03-23T03:50:09.361816+0000","last_fullsized":"2024-03-23T03:50:09.361816+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:28:01.892197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.959506+0000","last_change":"2024-03-23T03:50:07.486711+0000","last_active":"2024-03-23T03:50:07.959506+0000","last_peered":"2024-03-23T03:50:07.959506+0000","last_clean":"2024-03-23T03:50:07.959506+0000","last_became_active":"2024-03-23T03:50:07.485208+0000","last_became_peered":"2024-03-23T03:50:07.485208+0000","last_unstale":"2024-03-23T03:50:07.959506+0000","last_undegraded":"2024-03-23T03:50:07.959506+0000","last_fullsized":"2024-03-23T03:50:07.959506+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:06:01.399011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.363797+0000","last_change":"2024-03-23T03:50:07.991022+0000","last_active":"2024-03-23T03:50:09.363797+0000","last_peered":"2024-03-23T03:50:09.363797+0000","last_clean":"2024-03-23T03:50:09.363797+0000","last_became_active":"2024-03-23T03:50:07.990720+0000","last_became_peered":"2024-03-23T03:50:07.990720+0000","last_unstale":"2024-03-23T03:50:09.363797+0000","last_undegraded":"2024-03-23T03:50:09.363797+0000","last_fullsized":"2024-03-23T03:50:09.363797+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:35:16.507303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.948890+0000","last_change":"2024-03-23T03:50:07.489100+0000","last_active":"2024-03-23T03:50:07.948890+0000","last_peered":"2024-03-23T03:50:07.948890+0000","last_clean":"2024-03-23T03:50:07.948890+0000","last_became_active":"2024-03-23T03:50:07.488834+0000","last_became_peered":"2024-03-23T03:50:07.488834+0000","last_unstale":"2024-03-23T03:50:07.948890+0000","last_undegraded":"2024-03-23T03:50:07.948890+0000","last_fullsized":"2024-03-23T03:50:07.948890+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:59:56.498434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.359650+0000","last_change":"2024-03-23T03:50:07.978838+0000","last_active":"2024-03-23T03:50:09.359650+0000","last_peered":"2024-03-23T03:50:09.359650+0000","last_clean":"2024-03-23T03:50:09.359650+0000","last_became_active":"2024-03-23T03:50:07.978543+0000","last_became_peered":"2024-03-23T03:50:07.978543+0000","last_unstale":"2024-03-23T03:50:09.359650+0000","last_undegraded":"2024-03-23T03:50:09.359650+0000","last_fullsized":"2024-03-23T03:50:09.359650+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:00:38.798289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.971436+0000","last_change":"2024-03-23T03:50:07.494652+0000","last_active":"2024-03-23T03:50:07.971436+0000","last_peered":"2024-03-23T03:50:07.971436+0000","last_clean":"2024-03-23T03:50:07.971436+0000","last_became_active":"2024-03-23T03:50:07.494262+0000","last_became_peered":"2024-03-23T03:50:07.494262+0000","last_unstale":"2024-03-23T03:50:07.971436+0000","last_undegraded":"2024-03-23T03:50:07.971436+0000","last_fullsized":"2024-03-23T03:50:07.971436+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:44:27.821644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.364290+0000","last_change":"2024-03-23T03:50:07.986168+0000","last_active":"2024-03-23T03:50:09.364290+0000","last_peered":"2024-03-23T03:50:09.364290+0000","last_clean":"2024-03-23T03:50:09.364290+0000","last_became_active":"2024-03-23T03:50:07.985372+0000","last_became_peered":"2024-03-23T03:50:07.985372+0000","last_unstale":"2024-03-23T03:50:09.364290+0000","last_undegraded":"2024-03-23T03:50:09.364290+0000","last_fullsized":"2024-03-23T03:50:09.364290+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:37.696559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.962088+0000","last_change":"2024-03-23T03:50:07.490510+0000","last_active":"2024-03-23T03:50:07.962088+0000","last_peered":"2024-03-23T03:50:07.962088+0000","last_clean":"2024-03-23T03:50:07.962088+0000","last_became_active":"2024-03-23T03:50:07.489708+0000","last_became_peered":"2024-03-23T03:50:07.489708+0000","last_unstale":"2024-03-23T03:50:07.962088+0000","last_undegraded":"2024-03-23T03:50:07.962088+0000","last_fullsized":"2024-03-23T03:50:07.962088+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:36:57.803871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.961937+0000","last_change":"2024-03-23T03:50:07.966381+0000","last_active":"2024-03-23T03:50:08.961937+0000","last_peered":"2024-03-23T03:50:08.961937+0000","last_clean":"2024-03-23T03:50:08.961937+0000","last_became_active":"2024-03-23T03:50:07.965394+0000","last_became_peered":"2024-03-23T03:50:07.965394+0000","last_unstale":"2024-03-23T03:50:08.961937+0000","last_undegraded":"2024-03-23T03:50:08.961937+0000","last_fullsized":"2024-03-23T03:50:08.961937+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:15:51.539800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"80'3","reported_seq":14,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.973690+0000","last_change":"2024-03-23T03:50:07.486375+0000","last_active":"2024-03-23T03:50:07.973690+0000","last_peered":"2024-03-23T03:50:07.973690+0000","last_clean":"2024-03-23T03:50:07.973690+0000","last_became_active":"2024-03-23T03:50:07.485742+0000","last_became_peered":"2024-03-23T03:50:07.485742+0000","last_unstale":"2024-03-23T03:50:07.973690+0000","last_undegraded":"2024-03-23T03:50:07.973690+0000","last_fullsized":"2024-03-23T03:50:07.973690+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:18:47.365643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361652+0000","last_change":"2024-03-23T03:50:07.974510+0000","last_active":"2024-03-23T03:50:09.361652+0000","last_peered":"2024-03-23T03:50:09.361652+0000","last_clean":"2024-03-23T03:50:09.361652+0000","last_became_active":"2024-03-23T03:50:07.974244+0000","last_became_peered":"2024-03-23T03:50:07.974244+0000","last_unstale":"2024-03-23T03:50:09.361652+0000","last_undegraded":"2024-03-23T03:50:09.361652+0000","last_fullsized":"2024-03-23T03:50:09.361652+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:24:37.296539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"80'6","reported_seq":17,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.999807+0000","last_change":"2024-03-23T03:50:07.490882+0000","last_active":"2024-03-23T03:50:07.999807+0000","last_peered":"2024-03-23T03:50:07.999807+0000","last_clean":"2024-03-23T03:50:07.999807+0000","last_became_active":"2024-03-23T03:50:07.490570+0000","last_became_peered":"2024-03-23T03:50:07.490570+0000","last_unstale":"2024-03-23T03:50:07.999807+0000","last_undegraded":"2024-03-23T03:50:07.999807+0000","last_fullsized":"2024-03-23T03:50:07.999807+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:48:07.848353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.364668+0000","last_change":"2024-03-23T03:50:07.977207+0000","last_active":"2024-03-23T03:50:09.364668+0000","last_peered":"2024-03-23T03:50:09.364668+0000","last_clean":"2024-03-23T03:50:09.364668+0000","last_became_active":"2024-03-23T03:50:07.976942+0000","last_became_peered":"2024-03-23T03:50:07.976942+0000","last_unstale":"2024-03-23T03:50:09.364668+0000","last_undegraded":"2024-03-23T03:50:09.364668+0000","last_fullsized":"2024-03-23T03:50:09.364668+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:04:27.774838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"80'2","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.986274+0000","last_change":"2024-03-23T03:50:07.489295+0000","last_active":"2024-03-23T03:50:07.986274+0000","last_peered":"2024-03-23T03:50:07.986274+0000","last_clean":"2024-03-23T03:50:07.986274+0000","last_became_active":"2024-03-23T03:50:07.488973+0000","last_became_peered":"2024-03-23T03:50:07.488973+0000","last_unstale":"2024-03-23T03:50:07.986274+0000","last_undegraded":"2024-03-23T03:50:07.986274+0000","last_fullsized":"2024-03-23T03:50:07.986274+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:36:15.054606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.362795+0000","last_change":"2024-03-23T03:50:07.985931+0000","last_active":"2024-03-23T03:50:09.362795+0000","last_peered":"2024-03-23T03:50:09.362795+0000","last_clean":"2024-03-23T03:50:09.362795+0000","last_became_active":"2024-03-23T03:50:07.984873+0000","last_became_peered":"2024-03-23T03:50:07.984873+0000","last_unstale":"2024-03-23T03:50:09.362795+0000","last_undegraded":"2024-03-23T03:50:09.362795+0000","last_fullsized":"2024-03-23T03:50:09.362795+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:06:58.331998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.962430+0000","last_change":"2024-03-23T03:50:07.495545+0000","last_active":"2024-03-23T03:50:07.962430+0000","last_peered":"2024-03-23T03:50:07.962430+0000","last_clean":"2024-03-23T03:50:07.962430+0000","last_became_active":"2024-03-23T03:50:07.494844+0000","last_became_peered":"2024-03-23T03:50:07.494844+0000","last_unstale":"2024-03-23T03:50:07.962430+0000","last_undegraded":"2024-03-23T03:50:07.962430+0000","last_fullsized":"2024-03-23T03:50:07.962430+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:28:24.935685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.364537+0000","last_change":"2024-03-23T03:50:07.990806+0000","last_active":"2024-03-23T03:50:09.364537+0000","last_peered":"2024-03-23T03:50:09.364537+0000","last_clean":"2024-03-23T03:50:09.364537+0000","last_became_active":"2024-03-23T03:50:07.990503+0000","last_became_peered":"2024-03-23T03:50:07.990503+0000","last_unstale":"2024-03-23T03:50:09.364537+0000","last_undegraded":"2024-03-23T03:50:09.364537+0000","last_fullsized":"2024-03-23T03:50:09.364537+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:03:40.282904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.948311+0000","last_change":"2024-03-23T03:50:07.485542+0000","last_active":"2024-03-23T03:50:07.948311+0000","last_peered":"2024-03-23T03:50:07.948311+0000","last_clean":"2024-03-23T03:50:07.948311+0000","last_became_active":"2024-03-23T03:50:07.485234+0000","last_became_peered":"2024-03-23T03:50:07.485234+0000","last_unstale":"2024-03-23T03:50:07.948311+0000","last_undegraded":"2024-03-23T03:50:07.948311+0000","last_fullsized":"2024-03-23T03:50:07.948311+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:24:41.782032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.364022+0000","last_change":"2024-03-23T03:50:07.993058+0000","last_active":"2024-03-23T03:50:09.364022+0000","last_peered":"2024-03-23T03:50:09.364022+0000","last_clean":"2024-03-23T03:50:09.364022+0000","last_became_active":"2024-03-23T03:50:07.992770+0000","last_became_peered":"2024-03-23T03:50:07.992770+0000","last_unstale":"2024-03-23T03:50:09.364022+0000","last_undegraded":"2024-03-23T03:50:09.364022+0000","last_fullsized":"2024-03-23T03:50:09.364022+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:28.715917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"80'8","reported_seq":19,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.987497+0000","last_change":"2024-03-23T03:50:07.477358+0000","last_active":"2024-03-23T03:50:07.987497+0000","last_peered":"2024-03-23T03:50:07.987497+0000","last_clean":"2024-03-23T03:50:07.987497+0000","last_became_active":"2024-03-23T03:50:07.476820+0000","last_became_peered":"2024-03-23T03:50:07.476820+0000","last_unstale":"2024-03-23T03:50:07.987497+0000","last_undegraded":"2024-03-23T03:50:07.987497+0000","last_fullsized":"2024-03-23T03:50:07.987497+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:49:00.654026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.960903+0000","last_change":"2024-03-23T03:50:07.967082+0000","last_active":"2024-03-23T03:50:08.960903+0000","last_peered":"2024-03-23T03:50:08.960903+0000","last_clean":"2024-03-23T03:50:08.960903+0000","last_became_active":"2024-03-23T03:50:07.966795+0000","last_became_peered":"2024-03-23T03:50:07.966795+0000","last_unstale":"2024-03-23T03:50:08.960903+0000","last_undegraded":"2024-03-23T03:50:08.960903+0000","last_fullsized":"2024-03-23T03:50:08.960903+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:40:09.402202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"80'1","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.986814+0000","last_change":"2024-03-23T03:50:07.496080+0000","last_active":"2024-03-23T03:50:07.986814+0000","last_peered":"2024-03-23T03:50:07.986814+0000","last_clean":"2024-03-23T03:50:07.986814+0000","last_became_active":"2024-03-23T03:50:07.494410+0000","last_became_peered":"2024-03-23T03:50:07.494410+0000","last_unstale":"2024-03-23T03:50:07.986814+0000","last_undegraded":"2024-03-23T03:50:07.986814+0000","last_fullsized":"2024-03-23T03:50:07.986814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:59:01.344309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.963138+0000","last_change":"2024-03-23T03:50:07.986511+0000","last_active":"2024-03-23T03:50:08.963138+0000","last_peered":"2024-03-23T03:50:08.963138+0000","last_clean":"2024-03-23T03:50:08.963138+0000","last_became_active":"2024-03-23T03:50:07.986224+0000","last_became_peered":"2024-03-23T03:50:07.986224+0000","last_unstale":"2024-03-23T03:50:08.963138+0000","last_undegraded":"2024-03-23T03:50:08.963138+0000","last_fullsized":"2024-03-23T03:50:08.963138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:29:23.097203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"80'2","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.979122+0000","last_change":"2024-03-23T03:50:07.489105+0000","last_active":"2024-03-23T03:50:07.979122+0000","last_peered":"2024-03-23T03:50:07.979122+0000","last_clean":"2024-03-23T03:50:07.979122+0000","last_became_active":"2024-03-23T03:50:07.488633+0000","last_became_peered":"2024-03-23T03:50:07.488633+0000","last_unstale":"2024-03-23T03:50:07.979122+0000","last_undegraded":"2024-03-23T03:50:07.979122+0000","last_fullsized":"2024-03-23T03:50:07.979122+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:19:21.548516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.963270+0000","last_change":"2024-03-23T03:50:07.972568+0000","last_active":"2024-03-23T03:50:08.963270+0000","last_peered":"2024-03-23T03:50:08.963270+0000","last_clean":"2024-03-23T03:50:08.963270+0000","last_became_active":"2024-03-23T03:50:07.972316+0000","last_became_peered":"2024-03-23T03:50:07.972316+0000","last_unstale":"2024-03-23T03:50:08.963270+0000","last_undegraded":"2024-03-23T03:50:08.963270+0000","last_fullsized":"2024-03-23T03:50:08.963270+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:20:18.613759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"80'1","reported_seq":12,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.984267+0000","last_change":"2024-03-23T03:50:07.490256+0000","last_active":"2024-03-23T03:50:07.984267+0000","last_peered":"2024-03-23T03:50:07.984267+0000","last_clean":"2024-03-23T03:50:07.984267+0000","last_became_active":"2024-03-23T03:50:07.489709+0000","last_became_peered":"2024-03-23T03:50:07.489709+0000","last_unstale":"2024-03-23T03:50:07.984267+0000","last_undegraded":"2024-03-23T03:50:07.984267+0000","last_fullsized":"2024-03-23T03:50:07.984267+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:54:59.757010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:08.962255+0000","last_change":"2024-03-23T03:50:07.970717+0000","last_active":"2024-03-23T03:50:08.962255+0000","last_peered":"2024-03-23T03:50:08.962255+0000","last_clean":"2024-03-23T03:50:08.962255+0000","last_became_active":"2024-03-23T03:50:07.970435+0000","last_became_peered":"2024-03-23T03:50:07.970435+0000","last_unstale":"2024-03-23T03:50:08.962255+0000","last_undegraded":"2024-03-23T03:50:08.962255+0000","last_fullsized":"2024-03-23T03:50:08.962255+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:19:13.975703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.962584+0000","last_change":"2024-03-23T03:50:07.478328+0000","last_active":"2024-03-23T03:50:07.962584+0000","last_peered":"2024-03-23T03:50:07.962584+0000","last_clean":"2024-03-23T03:50:07.962584+0000","last_became_active":"2024-03-23T03:50:07.478059+0000","last_became_peered":"2024-03-23T03:50:07.478059+0000","last_unstale":"2024-03-23T03:50:07.962584+0000","last_undegraded":"2024-03-23T03:50:07.962584+0000","last_fullsized":"2024-03-23T03:50:07.962584+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:03:17.838071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360585+0000","last_change":"2024-03-23T03:50:07.978890+0000","last_active":"2024-03-23T03:50:09.360585+0000","last_peered":"2024-03-23T03:50:09.360585+0000","last_clean":"2024-03-23T03:50:09.360585+0000","last_became_active":"2024-03-23T03:50:07.978236+0000","last_became_peered":"2024-03-23T03:50:07.978236+0000","last_unstale":"2024-03-23T03:50:09.360585+0000","last_undegraded":"2024-03-23T03:50:09.360585+0000","last_fullsized":"2024-03-23T03:50:09.360585+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:41:32.947352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"80'2","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.974863+0000","last_change":"2024-03-23T03:50:07.487422+0000","last_active":"2024-03-23T03:50:07.974863+0000","last_peered":"2024-03-23T03:50:07.974863+0000","last_clean":"2024-03-23T03:50:07.974863+0000","last_became_active":"2024-03-23T03:50:07.487176+0000","last_became_peered":"2024-03-23T03:50:07.487176+0000","last_unstale":"2024-03-23T03:50:07.974863+0000","last_undegraded":"2024-03-23T03:50:07.974863+0000","last_fullsized":"2024-03-23T03:50:07.974863+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:41:32.913577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360906+0000","last_change":"2024-03-23T03:50:07.980838+0000","last_active":"2024-03-23T03:50:09.360906+0000","last_peered":"2024-03-23T03:50:09.360906+0000","last_clean":"2024-03-23T03:50:09.360906+0000","last_became_active":"2024-03-23T03:50:07.980465+0000","last_became_peered":"2024-03-23T03:50:07.980465+0000","last_unstale":"2024-03-23T03:50:09.360906+0000","last_undegraded":"2024-03-23T03:50:09.360906+0000","last_fullsized":"2024-03-23T03:50:09.360906+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:44:23.205733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.957056+0000","last_change":"2024-03-23T03:50:07.490733+0000","last_active":"2024-03-23T03:50:07.957056+0000","last_peered":"2024-03-23T03:50:07.957056+0000","last_clean":"2024-03-23T03:50:07.957056+0000","last_became_active":"2024-03-23T03:50:07.490461+0000","last_became_peered":"2024-03-23T03:50:07.490461+0000","last_unstale":"2024-03-23T03:50:07.957056+0000","last_undegraded":"2024-03-23T03:50:07.957056+0000","last_fullsized":"2024-03-23T03:50:07.957056+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:46:17.453220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.364245+0000","last_change":"2024-03-23T03:50:07.982642+0000","last_active":"2024-03-23T03:50:09.364245+0000","last_peered":"2024-03-23T03:50:09.364245+0000","last_clean":"2024-03-23T03:50:09.364245+0000","last_became_active":"2024-03-23T03:50:07.978117+0000","last_became_peered":"2024-03-23T03:50:07.978117+0000","last_unstale":"2024-03-23T03:50:09.364245+0000","last_undegraded":"2024-03-23T03:50:09.364245+0000","last_fullsized":"2024-03-23T03:50:09.364245+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:28:01.892197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.957788+0000","last_change":"2024-03-23T03:50:07.494059+0000","last_active":"2024-03-23T03:50:07.957788+0000","last_peered":"2024-03-23T03:50:07.957788+0000","last_clean":"2024-03-23T03:50:07.957788+0000","last_became_active":"2024-03-23T03:50:07.493829+0000","last_became_peered":"2024-03-23T03:50:07.493829+0000","last_unstale":"2024-03-23T03:50:07.957788+0000","last_undegraded":"2024-03-23T03:50:07.957788+0000","last_fullsized":"2024-03-23T03:50:07.957788+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:54:23.987953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360990+0000","last_change":"2024-03-23T03:50:07.982204+0000","last_active":"2024-03-23T03:50:09.360990+0000","last_peered":"2024-03-23T03:50:09.360990+0000","last_clean":"2024-03-23T03:50:09.360990+0000","last_became_active":"2024-03-23T03:50:07.981108+0000","last_became_peered":"2024-03-23T03:50:07.981108+0000","last_unstale":"2024-03-23T03:50:09.360990+0000","last_undegraded":"2024-03-23T03:50:09.360990+0000","last_fullsized":"2024-03-23T03:50:09.360990+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:00:18.878974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"80'2","reported_seq":13,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.974784+0000","last_change":"2024-03-23T03:50:07.485213+0000","last_active":"2024-03-23T03:50:07.974784+0000","last_peered":"2024-03-23T03:50:07.974784+0000","last_clean":"2024-03-23T03:50:07.974784+0000","last_became_active":"2024-03-23T03:50:07.484905+0000","last_became_peered":"2024-03-23T03:50:07.484905+0000","last_unstale":"2024-03-23T03:50:07.974784+0000","last_undegraded":"2024-03-23T03:50:07.974784+0000","last_fullsized":"2024-03-23T03:50:07.974784+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:14:48.590038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.362905+0000","last_change":"2024-03-23T03:50:07.986303+0000","last_active":"2024-03-23T03:50:09.362905+0000","last_peered":"2024-03-23T03:50:09.362905+0000","last_clean":"2024-03-23T03:50:09.362905+0000","last_became_active":"2024-03-23T03:50:07.985696+0000","last_became_peered":"2024-03-23T03:50:07.985696+0000","last_unstale":"2024-03-23T03:50:09.362905+0000","last_undegraded":"2024-03-23T03:50:09.362905+0000","last_fullsized":"2024-03-23T03:50:09.362905+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:58:13.062345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.965824+0000","last_change":"2024-03-23T03:50:07.482606+0000","last_active":"2024-03-23T03:50:07.965824+0000","last_peered":"2024-03-23T03:50:07.965824+0000","last_clean":"2024-03-23T03:50:07.965824+0000","last_became_active":"2024-03-23T03:50:07.482285+0000","last_became_peered":"2024-03-23T03:50:07.482285+0000","last_unstale":"2024-03-23T03:50:07.965824+0000","last_undegraded":"2024-03-23T03:50:07.965824+0000","last_fullsized":"2024-03-23T03:50:07.965824+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:51:13.757234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361429+0000","last_change":"2024-03-23T03:50:07.986046+0000","last_active":"2024-03-23T03:50:09.361429+0000","last_peered":"2024-03-23T03:50:09.361429+0000","last_clean":"2024-03-23T03:50:09.361429+0000","last_became_active":"2024-03-23T03:50:07.982397+0000","last_became_peered":"2024-03-23T03:50:07.982397+0000","last_unstale":"2024-03-23T03:50:09.361429+0000","last_undegraded":"2024-03-23T03:50:09.361429+0000","last_fullsized":"2024-03-23T03:50:09.361429+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:06:58.331998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.971106+0000","last_change":"2024-03-23T03:50:07.494355+0000","last_active":"2024-03-23T03:50:07.971106+0000","last_peered":"2024-03-23T03:50:07.971106+0000","last_clean":"2024-03-23T03:50:07.971106+0000","last_became_active":"2024-03-23T03:50:07.494076+0000","last_became_peered":"2024-03-23T03:50:07.494076+0000","last_unstale":"2024-03-23T03:50:07.971106+0000","last_undegraded":"2024-03-23T03:50:07.971106+0000","last_fullsized":"2024-03-23T03:50:07.971106+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:01:47.992165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.360157+0000","last_change":"2024-03-23T03:50:07.981581+0000","last_active":"2024-03-23T03:50:09.360157+0000","last_peered":"2024-03-23T03:50:09.360157+0000","last_clean":"2024-03-23T03:50:09.360157+0000","last_became_active":"2024-03-23T03:50:07.978579+0000","last_became_peered":"2024-03-23T03:50:07.978579+0000","last_unstale":"2024-03-23T03:50:09.360157+0000","last_undegraded":"2024-03-23T03:50:09.360157+0000","last_fullsized":"2024-03-23T03:50:09.360157+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:58:35.441218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.963492+0000","last_change":"2024-03-23T03:50:07.493956+0000","last_active":"2024-03-23T03:50:07.963492+0000","last_peered":"2024-03-23T03:50:07.963492+0000","last_clean":"2024-03-23T03:50:07.963492+0000","last_became_active":"2024-03-23T03:50:07.493656+0000","last_became_peered":"2024-03-23T03:50:07.493656+0000","last_unstale":"2024-03-23T03:50:07.963492+0000","last_undegraded":"2024-03-23T03:50:07.963492+0000","last_fullsized":"2024-03-23T03:50:07.963492+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:59:58.116627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.363387+0000","last_change":"2024-03-23T03:50:07.992710+0000","last_active":"2024-03-23T03:50:09.363387+0000","last_peered":"2024-03-23T03:50:09.363387+0000","last_clean":"2024-03-23T03:50:09.363387+0000","last_became_active":"2024-03-23T03:50:07.990216+0000","last_became_peered":"2024-03-23T03:50:07.990216+0000","last_unstale":"2024-03-23T03:50:09.363387+0000","last_undegraded":"2024-03-23T03:50:09.363387+0000","last_fullsized":"2024-03-23T03:50:09.363387+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:24:15.374234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.975346+0000","last_change":"2024-03-23T03:50:07.487283+0000","last_active":"2024-03-23T03:50:07.975346+0000","last_peered":"2024-03-23T03:50:07.975346+0000","last_clean":"2024-03-23T03:50:07.975346+0000","last_became_active":"2024-03-23T03:50:07.486961+0000","last_became_peered":"2024-03-23T03:50:07.486961+0000","last_unstale":"2024-03-23T03:50:07.975346+0000","last_undegraded":"2024-03-23T03:50:07.975346+0000","last_fullsized":"2024-03-23T03:50:07.975346+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:19:21.548516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.363077+0000","last_change":"2024-03-23T03:50:07.990718+0000","last_active":"2024-03-23T03:50:09.363077+0000","last_peered":"2024-03-23T03:50:09.363077+0000","last_clean":"2024-03-23T03:50:09.363077+0000","last_became_active":"2024-03-23T03:50:07.989661+0000","last_became_peered":"2024-03-23T03:50:07.989661+0000","last_unstale":"2024-03-23T03:50:09.363077+0000","last_undegraded":"2024-03-23T03:50:09.363077+0000","last_fullsized":"2024-03-23T03:50:09.363077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:43:48.835469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.950493+0000","last_change":"2024-03-23T03:50:07.490922+0000","last_active":"2024-03-23T03:50:07.950493+0000","last_peered":"2024-03-23T03:50:07.950493+0000","last_clean":"2024-03-23T03:50:07.950493+0000","last_became_active":"2024-03-23T03:50:07.490610+0000","last_became_peered":"2024-03-23T03:50:07.490610+0000","last_unstale":"2024-03-23T03:50:07.950493+0000","last_undegraded":"2024-03-23T03:50:07.950493+0000","last_fullsized":"2024-03-23T03:50:07.950493+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:36:27.444641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.361918+0000","last_change":"2024-03-23T03:50:07.974413+0000","last_active":"2024-03-23T03:50:09.361918+0000","last_peered":"2024-03-23T03:50:09.361918+0000","last_clean":"2024-03-23T03:50:09.361918+0000","last_became_active":"2024-03-23T03:50:07.973059+0000","last_became_peered":"2024-03-23T03:50:07.973059+0000","last_unstale":"2024-03-23T03:50:09.361918+0000","last_undegraded":"2024-03-23T03:50:09.361918+0000","last_fullsized":"2024-03-23T03:50:09.361918+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:59.498074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.975256+0000","last_change":"2024-03-23T03:50:07.486181+0000","last_active":"2024-03-23T03:50:07.975256+0000","last_peered":"2024-03-23T03:50:07.975256+0000","last_clean":"2024-03-23T03:50:07.975256+0000","last_became_active":"2024-03-23T03:50:07.485883+0000","last_became_peered":"2024-03-23T03:50:07.485883+0000","last_unstale":"2024-03-23T03:50:07.975256+0000","last_undegraded":"2024-03-23T03:50:07.975256+0000","last_fullsized":"2024-03-23T03:50:07.975256+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:13.215153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.362020+0000","last_change":"2024-03-23T03:50:07.974134+0000","last_active":"2024-03-23T03:50:09.362020+0000","last_peered":"2024-03-23T03:50:09.362020+0000","last_clean":"2024-03-23T03:50:09.362020+0000","last_became_active":"2024-03-23T03:50:07.972868+0000","last_became_peered":"2024-03-23T03:50:07.972868+0000","last_unstale":"2024-03-23T03:50:09.362020+0000","last_undegraded":"2024-03-23T03:50:09.362020+0000","last_fullsized":"2024-03-23T03:50:09.362020+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:02:37.696559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.961901+0000","last_change":"2024-03-23T03:50:07.483623+0000","last_active":"2024-03-23T03:50:07.961901+0000","last_peered":"2024-03-23T03:50:07.961901+0000","last_clean":"2024-03-23T03:50:07.961901+0000","last_became_active":"2024-03-23T03:50:07.483354+0000","last_became_peered":"2024-03-23T03:50:07.483354+0000","last_unstale":"2024-03-23T03:50:07.961901+0000","last_undegraded":"2024-03-23T03:50:07.961901+0000","last_fullsized":"2024-03-23T03:50:07.961901+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:01:47.992165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-03-23T03:50:09.083860+0000","last_change":"2024-03-23T03:50:07.984423+0000","last_active":"2024-03-23T03:50:09.083860+0000","last_peered":"2024-03-23T03:50:09.083860+0000","last_clean":"2024-03-23T03:50:09.083860+0000","last_became_active":"2024-03-23T03:50:07.983811+0000","last_became_peered":"2024-03-23T03:50:07.983811+0000","last_unstale":"2024-03-23T03:50:09.083860+0000","last_undegraded":"2024-03-23T03:50:09.083860+0000","last_fullsized":"2024-03-23T03:50:09.083860+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:07.443611+0000","last_clean_scrub_stamp":"2024-03-23T03:50:07.443611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:24:37.296539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.963515+0000","last_change":"2024-03-23T03:50:07.493185+0000","last_active":"2024-03-23T03:50:07.963515+0000","last_peered":"2024-03-23T03:50:07.963515+0000","last_clean":"2024-03-23T03:50:07.963515+0000","last_became_active":"2024-03-23T03:50:07.492771+0000","last_became_peered":"2024-03-23T03:50:07.492771+0000","last_unstale":"2024-03-23T03:50:07.963515+0000","last_undegraded":"2024-03-23T03:50:07.963515+0000","last_fullsized":"2024-03-23T03:50:07.963515+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:06.429749+0000","last_clean_scrub_stamp":"2024-03-23T03:50:06.429749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:11:43.166129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":213,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-03-23T03:50:07.967011+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:50:07.967011+0000","last_peered":"2024-03-23T03:50:07.967011+0000","last_clean":"2024-03-23T03:50:07.967011+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:50:07.967011+0000","last_undegraded":"2024-03-23T03:50:07.967011+0000","last_fullsized":"2024-03-23T03:50:07.967011+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542189,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28580,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343260,"statfs":{"total":96636764160,"available":96607498240,"internally_reserved":0,"allocated":1585152,"data_stored":1035263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.079,"15min":1.069},"min":{"1min":0.68200000000000005,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.573,"5min":1.635,"15min":1.635},"last":1.5209999999999999},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0449999999999999,"15min":1.048},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.417,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.91400000000000003}]},{"osd":1,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0549999999999999,"15min":1.044},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.544,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.399},{"interface":"front","average":{"1min":1.085,"5min":1.2330000000000001,"15min":1.3009999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.3999999999999999,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":0.81000000000000005}]},{"osd":2,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.139,"15min":1.1559999999999999},"min":{"1min":0.76400000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.7909999999999999,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.345,"15min":1.421},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.4630000000000001,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.464}]},{"osd":3,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.355,"15min":1.4370000000000001},"min":{"1min":0.873,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.397,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.27,"5min":1.3380000000000001,"15min":1.3580000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":1.5840000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.1539999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.266,"15min":1.2669999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.6299999999999999,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.022},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.357,"15min":1.375},"min":{"1min":0.77900000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0619999999999998,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.351}]},{"osd":5,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.3999999999999999,"15min":1.4219999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.6539999999999999,"5min":2.645,"15min":2.645},"last":1.272},{"interface":"front","average":{"1min":1.339,"5min":1.4370000000000001,"15min":1.4530000000000001},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":1.7050000000000001,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.1160000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.5229999999999999,"15min":1.5589999999999999},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":1.8200000000000001,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.304},{"interface":"front","average":{"1min":1.415,"5min":1.4930000000000001,"15min":1.5209999999999999},"min":{"1min":1.1319999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":2.1509999999999998,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.1930000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542187,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":999424,"data_stored":446437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.0309999999999999,"15min":1.04},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.3979999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0429999999999999,"15min":1.038},"min":{"1min":0.72099999999999997,"5min":0.62,"15min":0.62},"max":{"1min":1.6120000000000001,"5min":1.946,"15min":1.946},"last":1.516}]},{"osd":1,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.01},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.129,"15min":1.1559999999999999},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.4810000000000001,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.5369999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.075,"15min":1.069},"min":{"1min":0.73699999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.5449999999999999,"5min":2.044,"15min":2.044},"last":1.411},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1950000000000001,"15min":1.218},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":1.1180000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2629999999999999,"15min":1.2849999999999999},"min":{"1min":0.76800000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.4910000000000001,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2549999999999999,"15min":1.2649999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.6319999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.2130000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3759999999999999,"15min":1.3839999999999999},"min":{"1min":0.67700000000000005,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.73,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.363,"15min":1.383},"min":{"1min":0.63400000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.762,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.6299999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.415,"15min":1.4359999999999999},"min":{"1min":0.71099999999999997,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.6919999999999999,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.478},{"interface":"front","average":{"1min":1.456,"5min":1.5249999999999999,"15min":1.5469999999999999},"min":{"1min":0.79700000000000004,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8149999999999999,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.615}]},{"osd":7,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.4059999999999999,"15min":1.4119999999999999},"min":{"1min":0.77600000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.802,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.573},{"interface":"front","average":{"1min":1.423,"5min":1.5129999999999999,"15min":1.5409999999999999},"min":{"1min":0.84099999999999997,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.74,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.262}]}]},{"osd":5,"up_from":14,"seq":60129542187,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":991232,"data_stored":444907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.1759999999999999,"15min":1.214},"min":{"1min":0.67900000000000005,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.6779999999999999,"5min":1.806,"15min":1.806},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.157,"15min":1.181},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.554,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.048}]},{"osd":1,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.81699999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.5489999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.083},{"interface":"front","average":{"1min":1.026,"5min":1.1599999999999999,"15min":1.1919999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4219999999999999,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.98999999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.298,"15min":1.3540000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5429999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3340000000000001,"15min":1.3779999999999999},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.772,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.321}]},{"osd":3,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.21,"15min":1.2470000000000001},"min":{"1min":0.70599999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.653,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.236},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.393,"15min":1.452},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.6779999999999999,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.1699999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.3420000000000001,"15min":1.4119999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.671,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.282,"5min":1.4119999999999999,"15min":1.444},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.77,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.294}]},{"osd":6,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.458,"15min":1.514},"min":{"1min":0.76700000000000002,"5min":0.63,"15min":0.63},"max":{"1min":1.873,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.208},{"interface":"front","average":{"1min":1.464,"5min":1.5940000000000001,"15min":1.6559999999999999},"min":{"1min":0.86199999999999999,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.956,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.389}]},{"osd":7,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.538,"15min":1.607},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.673,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.133},{"interface":"front","average":{"1min":1.411,"5min":1.6200000000000001,"15min":1.696},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.855,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.357}]}]},{"osd":4,"up_from":14,"seq":60129542188,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32104,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339736,"statfs":{"total":96636764160,"available":96603889664,"internally_reserved":0,"allocated":1007616,"data_stored":446541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.002,"15min":0.97999999999999998},"min":{"1min":0.67800000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":1.03,"5min":1.0149999999999999,"15min":1.012},"min":{"1min":0.81200000000000006,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3069999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.56100000000000005}]},{"osd":1,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.0700000000000001,"15min":1.05},"min":{"1min":0.74299999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.704,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.147,"5min":1.0880000000000001,"15min":1.0640000000000001},"min":{"1min":0.50600000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.47099999999999997}]},{"osd":2,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2949999999999999,"15min":1.325},"min":{"1min":0.81599999999999995,"5min":0.625,"15min":0.625},"max":{"1min":1.569,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":1.29,"5min":1.3100000000000001,"15min":1.3080000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.8680000000000001,"5min":2.181,"15min":2.181},"last":0.59899999999999998}]},{"osd":3,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1339999999999999,"15min":1.1299999999999999},"min":{"1min":0.73199999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.26,"15min":1.278},"min":{"1min":0.82899999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.5720000000000001,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":0.73799999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.278,"15min":1.274},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.298,"15min":1.2829999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.2349999999999999,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":0.48899999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.415,"15min":1.4239999999999999},"min":{"1min":1.143,"5min":0.748,"15min":0.748},"max":{"1min":1.833,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.478,"5min":1.425,"15min":1.4039999999999999},"min":{"1min":0.745,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":0.54800000000000004}]},{"osd":7,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.385,"15min":1.371},"min":{"1min":1.0880000000000001,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":2.3479999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.4970000000000001,"15min":1.5009999999999999},"min":{"1min":1.056,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.974,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":0.80000000000000004}]}]},{"osd":3,"up_from":14,"seq":60129542188,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":991232,"data_stored":445455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.83899999999999997,"15min":0.86099999999999999},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.256,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.446,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.76400000000000001}]},{"osd":1,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.92800000000000005,"15min":0.97199999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.997,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":1.0960000000000001},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.034,"15min":1.0720000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.712,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":1.327}]},{"osd":2,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.006,"15min":1.052},"min":{"1min":0.435,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.613,"5min":1.823,"15min":1.823},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.1319999999999999,"15min":1.153},"min":{"1min":0.48199999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.6020000000000001,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":1.4119999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.2310000000000001,"15min":1.2450000000000001},"min":{"1min":0.67600000000000005,"5min":0.625,"15min":0.625},"max":{"1min":1.3859999999999999,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":1.226},{"interface":"front","average":{"1min":1.151,"5min":1.377,"15min":1.4370000000000001},"min":{"1min":0.77100000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.4770000000000001,"5min":4.819,"15min":4.819},"last":1.2729999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.603,"15min":1.4470000000000001},"min":{"1min":0.75,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.0299999999999998,"5min":30.228000000000002,"15min":30.228000000000002},"last":1.474},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.3320000000000001,"15min":1.355},"min":{"1min":0.91400000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.637,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":1.387}]},{"osd":6,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.448,"15min":1.52},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.5860000000000001,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.53},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3999999999999999,"15min":1.397},"min":{"1min":0.90100000000000002,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.1429999999999998,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":1.6240000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.395,"15min":1.4219999999999999},"min":{"1min":0.81200000000000006,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.71,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.298,"5min":1.577,"15min":1.601},"min":{"1min":0.73299999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.629,"5min":10.663,"15min":10.663},"last":1.5840000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542188,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":999424,"data_stored":445615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.84999999999999998,"15min":0.88100000000000001},"min":{"1min":0.318,"5min":0.307,"15min":0.307},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":1.1120000000000001},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.88700000000000001,"15min":0.94299999999999995},"min":{"1min":0.53500000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.23,"5min":2.226,"15min":2.226},"last":0.56999999999999995}]},{"osd":1,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.96899999999999997,"15min":0.995},"min":{"1min":0.621,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.5109999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.88,"15min":0.91700000000000004},"min":{"1min":0.53700000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.282,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.73599999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.98499999999999999,"15min":1.026},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.4750000000000001,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0549999999999999,"15min":1.085},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3380000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.222}]},{"osd":4,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.2090000000000001,"15min":1.236},"min":{"1min":0.96399999999999997,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.488,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":0.876},{"interface":"front","average":{"1min":1.087,"5min":1.179,"15min":1.214},"min":{"1min":0.66300000000000003,"5min":0.627,"15min":0.627},"max":{"1min":1.47,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.93100000000000005}]},{"osd":5,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.3360000000000001,"15min":1.3759999999999999},"min":{"1min":0.76200000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.304},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.304,"15min":1.349},"min":{"1min":0.75,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.5880000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.081}]},{"osd":6,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.268,"15min":1.3400000000000001},"min":{"1min":0.82199999999999995,"5min":0.753,"15min":0.753},"max":{"1min":1.5109999999999999,"5min":2.456,"15min":2.456},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.359,"5min":1.4390000000000001,"15min":1.4890000000000001},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.3959999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.335,"15min":1.379},"min":{"1min":0.82699999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.6870000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":0.85399999999999998},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.3460000000000001,"15min":1.4099999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5900000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.022}]}]},{"osd":1,"up_from":14,"seq":60129542187,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343848,"statfs":{"total":96636764160,"available":96608100352,"internally_reserved":0,"allocated":995328,"data_stored":444985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.81599999999999995,"15min":0.87},"min":{"1min":0.40200000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.1819999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.82999999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.92000000000000004}]},{"osd":2,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.92800000000000005,"15min":0.96599999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.4570000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.3100000000000001},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.94899999999999995,"15min":0.98599999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.4750000000000001,"5min":1.625,"15min":1.625},"last":0.81000000000000005}]},{"osd":3,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.117,"15min":1.117},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.7929999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.873},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.99399999999999999,"15min":1.0189999999999999},"min":{"1min":0.57399999999999995,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.5600000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.208}]},{"osd":4,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.232,"15min":1.292},"min":{"1min":0.69199999999999995,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.752,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.2749999999999999,"15min":1.3149999999999999},"min":{"1min":0.82599999999999996,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.575,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.05}]},{"osd":5,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.2729999999999999,"15min":1.329},"min":{"1min":0.82399999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.7470000000000001,"5min":3.593,"15min":3.593},"last":1.369},{"interface":"front","average":{"1min":1.276,"5min":1.3700000000000001,"15min":1.4399999999999999},"min":{"1min":0.76200000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6719999999999999,"5min":2.359,"15min":2.359},"last":1.109}]},{"osd":6,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.349,"15min":1.405},"min":{"1min":0.79300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.1709999999999998,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.147,"5min":1.302,"15min":1.3580000000000001},"min":{"1min":0.77800000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.696,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.4530000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.3759999999999999,"15min":1.4379999999999999},"min":{"1min":0.81100000000000005,"5min":0.627,"15min":0.627},"max":{"1min":1.7649999999999999,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.4219999999999999,"15min":1.506},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.7569999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.4159999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542187,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28564,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343276,"statfs":{"total":96636764160,"available":96607514624,"internally_reserved":0,"allocated":1581056,"data_stored":1035241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.90000000000000002,"15min":0.94699999999999995},"min":{"1min":0.435,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.1599999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.98899999999999999},"min":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.3640000000000001,"5min":1.506,"15min":1.506},"last":0.497}]},{"osd":2,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.95799999999999996,"15min":1.0009999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1279999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.91900000000000004,"5min":1.0640000000000001,"15min":1.105},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.4350000000000001,"5min":1.901,"15min":1.901},"last":0.45300000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.97499999999999998,"15min":1.01},"min":{"1min":0.48999999999999999,"5min":0.433,"15min":0.433},"max":{"1min":1.5189999999999999,"5min":1.972,"15min":1.972},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.103,"15min":1.159},"min":{"1min":0.51400000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3180000000000001,"5min":2.298,"15min":2.298},"last":0.72399999999999998}]},{"osd":4,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.304,"15min":1.3420000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.599,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":0.92900000000000005},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.252,"15min":1.28},"min":{"1min":0.83099999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.4990000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.81799999999999995}]},{"osd":5,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.3340000000000001,"15min":1.3779999999999999},"min":{"1min":0.878,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.4450000000000001,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.171},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.403,"15min":1.4470000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.49,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.208}]},{"osd":6,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.46,"15min":1.538},"min":{"1min":0.85799999999999998,"5min":0.747,"15min":0.747},"max":{"1min":1.5940000000000001,"5min":3.032,"15min":3.032},"last":1.0269999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.4419999999999999,"15min":1.498},"min":{"1min":0.98199999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.5740000000000001,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":0.874}]},{"osd":7,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.46,"15min":1.5129999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.7110000000000001,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":1.329,"5min":1.3939999999999999,"15min":1.4299999999999999},"min":{"1min":0.871,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.008,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.044}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:50:17.814 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T03:50:17.814 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-23T03:50:18.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.069+0000 7f70d3a00640 1 Processor -- start 2024-03-23T03:50:18.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.069+0000 7f70d3a00640 1 -- start start 2024-03-23T03:50:18.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.071+0000 7f70d3a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f70cc0cd740 0x7f70cc0b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:18.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.071+0000 7f70d3a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f70cc0b5aa0 0x7f70cc0b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:18.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.071+0000 7f70d3a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70cc0d20c0 con 0x7f70cc0d0d40 2024-03-23T03:50:18.071 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.071+0000 7f70d3a00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70cc0d14b0 con 0x7f70cc0cd740 2024-03-23T03:50:18.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.071+0000 7f70d3a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc0d1790 con 0x7f70cc0b5aa0 2024-03-23T03:50:18.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.071+0000 7f70d1a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f70cc0b5aa0 0x7f70cc0b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:18.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.071+0000 7f70d1000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f70cc0cd740 0x7f70cc0b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:18.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.072+0000 7f70d1a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f70cc0b5aa0 0x7f70cc0b5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:37382/0 (socket says 172.21.15.102:37382) 2024-03-23T03:50:18.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.072+0000 7f70d1a00640 1 -- 172.21.15.102:0/568779346 learned_addr learned my addr 172.21.15.102:0/568779346 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:18.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.072+0000 7f70d1000640 1 -- 172.21.15.102:0/568779346 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f70cc0b5aa0 msgr2=0x7f70cc0b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:18.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.072+0000 7f70d1000640 1 --2- 172.21.15.102:0/568779346 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f70cc0b5aa0 0x7f70cc0b5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.072+0000 7f70d1000640 1 -- 172.21.15.102:0/568779346 >> v1:172.21.15.173:6789/0 conn(0x7f70cc0d0d40 legacy=0x7f70cc0cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:18.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.072+0000 7f70d1000640 1 -- 172.21.15.102:0/568779346 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70cc0d1a70 con 0x7f70cc0cd740 2024-03-23T03:50:18.073 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.072+0000 7f70d1a00640 1 --2- 172.21.15.102:0/568779346 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f70cc0b5aa0 0x7f70cc0b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:50:18.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.074+0000 7f70d1000640 1 --2- 172.21.15.102:0/568779346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70cc0cd740 0x7f70cc0b84b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f70bc0097a0 tx=0x7f70bc02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b767f963b8af0c3a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:18.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.074+0000 7f70cbe00640 1 -- 172.21.15.102:0/568779346 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc02d040 con 0x7f70cc0cd740 2024-03-23T03:50:18.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.074+0000 7f70cbe00640 1 -- 172.21.15.102:0/568779346 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70bc031810 con 0x7f70cc0cd740 2024-03-23T03:50:18.075 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.074+0000 7f70cbe00640 1 -- 172.21.15.102:0/568779346 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70bc031ba0 con 0x7f70cc0cd740 2024-03-23T03:50:18.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.075+0000 7f70d3a00640 1 -- 172.21.15.102:0/568779346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70cc0cd740 msgr2=0x7f70cc0b84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:18.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.075+0000 7f70d3a00640 1 --2- 172.21.15.102:0/568779346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70cc0cd740 0x7f70cc0b84b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f70bc0097a0 tx=0x7f70bc02b8b0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.075+0000 7f70d3a00640 1 -- 172.21.15.102:0/568779346 shutdown_connections 2024-03-23T03:50:18.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.075+0000 7f70d3a00640 1 --2- 172.21.15.102:0/568779346 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f70cc0b5aa0 0x7f70cc0b5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.075+0000 7f70d3a00640 1 --2- 172.21.15.102:0/568779346 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70cc0cd740 0x7f70cc0b84b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.075+0000 7f70d3a00640 1 -- 172.21.15.102:0/568779346 >> 172.21.15.102:0/568779346 conn(0x7f70cc0cf5d0 msgr2=0x7f70cc0cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:18.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.075+0000 7f70d3a00640 1 -- 172.21.15.102:0/568779346 shutdown_connections 2024-03-23T03:50:18.076 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.075+0000 7f70d3a00640 1 -- 172.21.15.102:0/568779346 wait complete. 2024-03-23T03:50:18.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.077+0000 7f70d3a00640 1 Processor -- start 2024-03-23T03:50:18.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.078+0000 7f70d3a00640 1 -- start start 2024-03-23T03:50:18.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.079+0000 7f70d3a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70cc0b5aa0 0x7f70cc0cbf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:18.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.079+0000 7f70d3a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f70cc0cd740 0x7f70cc15b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:18.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.079+0000 7f70d3a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70cc0d0d40 0x7f70cc0ab940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:18.080 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.079+0000 7f70d3a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc0cc7e0 con 0x7f70cc0d0d40 2024-03-23T03:50:18.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.079+0000 7f70d3a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc0a93a0 con 0x7f70cc0b5aa0 2024-03-23T03:50:18.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.079+0000 7f70d3a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70cc0a9680 con 0x7f70cc0cd740 2024-03-23T03:50:18.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.079+0000 7f70d1a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70cc0d0d40 0x7f70cc0ab940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:18.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.079+0000 7f70d1a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70cc0d0d40 0x7f70cc0ab940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38748/0 (socket says 172.21.15.102:38748) 2024-03-23T03:50:18.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.079+0000 7f70d1a00640 1 -- 172.21.15.102:0/936640160 learned_addr learned my addr 172.21.15.102:0/936640160 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:18.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.079+0000 7f70c3e00640 1 --2- 172.21.15.102:0/936640160 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f70cc0cd740 0x7f70cc15b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:18.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.079+0000 7f70d1000640 1 --2- 172.21.15.102:0/936640160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70cc0b5aa0 0x7f70cc0cbf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:18.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.080+0000 7f70d1a00640 1 -- 172.21.15.102:0/936640160 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f70cc0cd740 msgr2=0x7f70cc15b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:18.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.080+0000 7f70d1a00640 1 --2- 172.21.15.102:0/936640160 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f70cc0cd740 0x7f70cc15b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.080+0000 7f70d1a00640 1 -- 172.21.15.102:0/936640160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70cc0b5aa0 msgr2=0x7f70cc0cbf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:18.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.080+0000 7f70d1a00640 1 --2- 172.21.15.102:0/936640160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70cc0b5aa0 0x7f70cc0cbf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.080+0000 7f70d1a00640 1 -- 172.21.15.102:0/936640160 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c4009b30 con 0x7f70cc0d0d40 2024-03-23T03:50:18.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.080+0000 7f70d1000640 1 --2- 172.21.15.102:0/936640160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70cc0b5aa0 0x7f70cc0cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:18.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.080+0000 7f70d1a00640 1 --2- 172.21.15.102:0/936640160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70cc0d0d40 0x7f70cc0ab940 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f70c400efc0 tx=0x7f70c400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:18.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.080+0000 7f70c9600640 1 -- 172.21.15.102:0/936640160 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c4018070 con 0x7f70cc0d0d40 2024-03-23T03:50:18.082 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.080+0000 7f70c9600640 1 -- 172.21.15.102:0/936640160 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c400f040 con 0x7f70cc0d0d40 2024-03-23T03:50:18.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.080+0000 7f70c9600640 1 -- 172.21.15.102:0/936640160 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c400ce00 con 0x7f70cc0d0d40 2024-03-23T03:50:18.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.080+0000 7f70d3a00640 1 -- 172.21.15.102:0/936640160 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70bc009400 con 0x7f70cc0d0d40 2024-03-23T03:50:18.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.080+0000 7f70d3a00640 1 -- 172.21.15.102:0/936640160 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70cc0a9c40 con 0x7f70cc0d0d40 2024-03-23T03:50:18.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.081+0000 7f70d3a00640 1 -- 172.21.15.102:0/936640160 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70cc0d2530 con 0x7f70cc0d0d40 2024-03-23T03:50:18.083 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.082+0000 7f70c9600640 1 -- 172.21.15.102:0/936640160 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f70c4014480 con 0x7f70cc0d0d40 2024-03-23T03:50:18.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.083+0000 7f70c9600640 1 --2- 172.21.15.102:0/936640160 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f70a806ac00 0x7f70a806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:18.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.083+0000 7f70c9600640 1 -- 172.21.15.102:0/936640160 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f70c4089760 con 0x7f70cc0d0d40 2024-03-23T03:50:18.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.084+0000 7f70d1000640 1 --2- 172.21.15.102:0/936640160 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f70a806ac00 0x7f70a806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:18.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.084+0000 7f70d1000640 1 --2- 172.21.15.102:0/936640160 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f70a806ac00 0x7f70a806d0c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f70bc036000 tx=0x7f70bc03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:18.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.088+0000 7f70c9600640 1 -- 172.21.15.102:0/936640160 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f70cc0d2530 con 0x7f70cc0d0d40 2024-03-23T03:50:18.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.354+0000 7f70d3a00640 1 -- 172.21.15.102:0/936640160 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f70cc0d2530 con 0x7f70cc0d0d40 2024-03-23T03:50:18.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.361+0000 7f70c9600640 1 -- 172.21.15.102:0/936640160 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+0 (secure 0 0 0) 0x7f70cc0d2530 con 0x7f70cc0d0d40 2024-03-23T03:50:18.362 INFO:teuthology.orchestra.run.smithi102.stderr:updated caps for client.0 2024-03-23T03:50:18.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.364+0000 7f70d3a00640 1 -- 172.21.15.102:0/936640160 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f70a806ac00 msgr2=0x7f70a806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:18.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.364+0000 7f70d3a00640 1 --2- 172.21.15.102:0/936640160 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f70a806ac00 0x7f70a806d0c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f70bc036000 tx=0x7f70bc03c040 comp rx=0 tx=0).stop 2024-03-23T03:50:18.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.364+0000 7f70d3a00640 1 -- 172.21.15.102:0/936640160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70cc0d0d40 msgr2=0x7f70cc0ab940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:18.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.364+0000 7f70d3a00640 1 --2- 172.21.15.102:0/936640160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70cc0d0d40 0x7f70cc0ab940 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f70c400efc0 tx=0x7f70c400c490 comp rx=0 tx=0).stop 2024-03-23T03:50:18.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.364+0000 7f70d3a00640 1 -- 172.21.15.102:0/936640160 shutdown_connections 2024-03-23T03:50:18.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.364+0000 7f70d3a00640 1 --2- 172.21.15.102:0/936640160 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f70a806ac00 0x7f70a806d0c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.364+0000 7f70d3a00640 1 --2- 172.21.15.102:0/936640160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70cc0d0d40 0x7f70cc0ab940 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.364+0000 7f70d3a00640 1 --2- 172.21.15.102:0/936640160 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f70cc0cd740 0x7f70cc15b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.364+0000 7f70d3a00640 1 --2- 172.21.15.102:0/936640160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f70cc0b5aa0 0x7f70cc0cbf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.364+0000 7f70d3a00640 1 -- 172.21.15.102:0/936640160 >> 172.21.15.102:0/936640160 conn(0x7f70cc0cf5d0 msgr2=0x7f70cc15efe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:18.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.364+0000 7f70d3a00640 1 -- 172.21.15.102:0/936640160 shutdown_connections 2024-03-23T03:50:18.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.365+0000 7f70d3a00640 1 -- 172.21.15.102:0/936640160 wait complete. 2024-03-23T03:50:18.385 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:50:18.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.589+0000 7fc585800640 1 Processor -- start 2024-03-23T03:50:18.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.590+0000 7fc585800640 1 -- start start 2024-03-23T03:50:18.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.591+0000 7fc585800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc58015b560 0x7fc58015b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:18.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.591+0000 7fc585800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc5800b39b0 0x7fc5800b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:18.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.591+0000 7fc585800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58006c6f0 con 0x7fc58015e500 2024-03-23T03:50:18.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.591+0000 7fc585800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5800b4370 con 0x7fc58015b560 2024-03-23T03:50:18.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.591+0000 7fc585800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc58006c9d0 con 0x7fc5800b39b0 2024-03-23T03:50:18.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.591+0000 7fc57ea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc58015b560 0x7fc58015b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:18.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.591+0000 7fc57ea00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc58015b560 0x7fc58015b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38754/0 (socket says 172.21.15.102:38754) 2024-03-23T03:50:18.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.591+0000 7fc57ea00640 1 -- 172.21.15.102:0/1599642077 learned_addr learned my addr 172.21.15.102:0/1599642077 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:18.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.592+0000 7fc57ea00640 1 -- 172.21.15.102:0/1599642077 >> v2:172.21.15.173:3300/0 conn(0x7fc5800b39b0 msgr2=0x7fc5800b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:18.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.592+0000 7fc57e000640 1 --2- 172.21.15.102:0/1599642077 >> v2:172.21.15.173:3300/0 conn(0x7fc5800b39b0 0x7fc5800b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:18.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.592+0000 7fc57ea00640 1 --2- 172.21.15.102:0/1599642077 >> v2:172.21.15.173:3300/0 conn(0x7fc5800b39b0 0x7fc5800b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.592+0000 7fc57ea00640 1 -- 172.21.15.102:0/1599642077 >> v1:172.21.15.173:6789/0 conn(0x7fc58015e500 legacy=0x7fc58015e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:18.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.592+0000 7fc57ea00640 1 -- 172.21.15.102:0/1599642077 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58006ccb0 con 0x7fc58015b560 2024-03-23T03:50:18.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.592+0000 7fc57e000640 1 --2- 172.21.15.102:0/1599642077 >> v2:172.21.15.173:3300/0 conn(0x7fc5800b39b0 0x7fc5800b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:18.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.593+0000 7fc57ea00640 1 --2- 172.21.15.102:0/1599642077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc58015b560 0x7fc58015b940 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc5800ce180 tx=0x7fc57002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b1134a4a0ea041d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:18.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.593+0000 7fc57d600640 1 -- 172.21.15.102:0/1599642077 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570031870 con 0x7fc58015b560 2024-03-23T03:50:18.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.593+0000 7fc57d600640 1 -- 172.21.15.102:0/1599642077 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5700319d0 con 0x7fc58015b560 2024-03-23T03:50:18.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.593+0000 7fc57d600640 1 -- 172.21.15.102:0/1599642077 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570031d60 con 0x7fc58015b560 2024-03-23T03:50:18.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.594+0000 7fc585800640 1 -- 172.21.15.102:0/1599642077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc58015b560 msgr2=0x7fc58015b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:18.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.594+0000 7fc585800640 1 --2- 172.21.15.102:0/1599642077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc58015b560 0x7fc58015b940 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc5800ce180 tx=0x7fc57002b910 comp rx=0 tx=0).stop 2024-03-23T03:50:18.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.594+0000 7fc585800640 1 -- 172.21.15.102:0/1599642077 shutdown_connections 2024-03-23T03:50:18.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.594+0000 7fc585800640 1 --2- 172.21.15.102:0/1599642077 >> v2:172.21.15.173:3300/0 conn(0x7fc5800b39b0 0x7fc5800b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.594+0000 7fc585800640 1 --2- 172.21.15.102:0/1599642077 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc58015b560 0x7fc58015b940 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.594+0000 7fc585800640 1 -- 172.21.15.102:0/1599642077 >> 172.21.15.102:0/1599642077 conn(0x7fc58006a3d0 msgr2=0x7fc5800b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:18.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.594+0000 7fc585800640 1 -- 172.21.15.102:0/1599642077 shutdown_connections 2024-03-23T03:50:18.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.594+0000 7fc585800640 1 -- 172.21.15.102:0/1599642077 wait complete. 2024-03-23T03:50:18.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.596+0000 7fc585800640 1 Processor -- start 2024-03-23T03:50:18.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.596+0000 7fc585800640 1 -- start start 2024-03-23T03:50:18.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.597+0000 7fc585800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5800b39b0 0x7fc580139f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:18.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.597+0000 7fc585800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc58015b560 0x7fc5800cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:18.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.597+0000 7fc585800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc58015e500 0x7fc5800cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:18.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.597+0000 7fc585800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc580151fa0 con 0x7fc5800b39b0 2024-03-23T03:50:18.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.597+0000 7fc585800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc580152b20 con 0x7fc58015e500 2024-03-23T03:50:18.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.597+0000 7fc585800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5801521f0 con 0x7fc58015b560 2024-03-23T03:50:18.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.597+0000 7fc57ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5800b39b0 0x7fc580139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:18.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.597+0000 7fc57ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5800b39b0 0x7fc580139f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38768/0 (socket says 172.21.15.102:38768) 2024-03-23T03:50:18.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.597+0000 7fc57ea00640 1 -- 172.21.15.102:0/928279720 learned_addr learned my addr 172.21.15.102:0/928279720 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:18.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.598+0000 7fc57ea00640 1 -- 172.21.15.102:0/928279720 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc58015b560 msgr2=0x7fc5800cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:18.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.598+0000 7fc57ea00640 1 --2- 172.21.15.102:0/928279720 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc58015b560 0x7fc5800cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.598+0000 7fc57f400640 1 --2- 172.21.15.102:0/928279720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc58015e500 0x7fc5800cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:18.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.598+0000 7fc57ea00640 1 -- 172.21.15.102:0/928279720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc58015e500 msgr2=0x7fc5800cca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:18.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.598+0000 7fc57ea00640 1 --2- 172.21.15.102:0/928279720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc58015e500 0x7fc5800cca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.598+0000 7fc57ea00640 1 -- 172.21.15.102:0/928279720 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5700094d0 con 0x7fc5800b39b0 2024-03-23T03:50:18.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.598+0000 7fc57f400640 1 --2- 172.21.15.102:0/928279720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc58015e500 0x7fc5800cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:18.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.598+0000 7fc57ea00640 1 --2- 172.21.15.102:0/928279720 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5800b39b0 0x7fc580139f70 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc570000c00 tx=0x7fc5700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:18.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.598+0000 7fc576a00640 1 -- 172.21.15.102:0/928279720 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc570031840 con 0x7fc5800b39b0 2024-03-23T03:50:18.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.598+0000 7fc576a00640 1 -- 172.21.15.102:0/928279720 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5700443f0 con 0x7fc5800b39b0 2024-03-23T03:50:18.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.598+0000 7fc576a00640 1 -- 172.21.15.102:0/928279720 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5700447c0 con 0x7fc5800b39b0 2024-03-23T03:50:18.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.598+0000 7fc585800640 1 -- 172.21.15.102:0/928279720 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5801524d0 con 0x7fc5800b39b0 2024-03-23T03:50:18.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.599+0000 7fc585800640 1 -- 172.21.15.102:0/928279720 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc580141fb0 con 0x7fc5800b39b0 2024-03-23T03:50:18.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.600+0000 7fc585800640 1 -- 172.21.15.102:0/928279720 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc580142170 con 0x7fc5800b39b0 2024-03-23T03:50:18.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.601+0000 7fc576a00640 1 -- 172.21.15.102:0/928279720 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc57004d450 con 0x7fc5800b39b0 2024-03-23T03:50:18.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.602+0000 7fc576a00640 1 --2- 172.21.15.102:0/928279720 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc56006ac00 0x7fc56006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:18.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.602+0000 7fc576a00640 1 -- 172.21.15.102:0/928279720 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc57002d040 con 0x7fc5800b39b0 2024-03-23T03:50:18.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.602+0000 7fc57e000640 1 --2- 172.21.15.102:0/928279720 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc56006ac00 0x7fc56006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:18.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.603+0000 7fc57e000640 1 --2- 172.21.15.102:0/928279720 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc56006ac00 0x7fc56006d0c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc56c006fd0 tx=0x7fc56c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:18.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.607+0000 7fc576a00640 1 -- 172.21.15.102:0/928279720 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc570076d40 con 0x7fc5800b39b0 2024-03-23T03:50:18.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.839+0000 7fc585800640 1 -- 172.21.15.102:0/928279720 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc580067100 con 0x7fc5800b39b0 2024-03-23T03:50:18.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.844+0000 7fc576a00640 1 -- 172.21.15.102:0/928279720 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4745 (secure 0 0 0) 0x7fc570035070 con 0x7fc5800b39b0 2024-03-23T03:50:18.845 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:18.845 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2079098809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2079098809},{"type":"v1","addr":"172.21.15.173:6833","nonce":2079098809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":5413,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/459102175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":459102175},{"type":"v1","addr":"172.21.15.173:6835","nonce":459102175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2962195583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2962195583},{"type":"v1","addr":"172.21.15.102:6835","nonce":2962195583}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:50:07.937613+0000","modified":"2024-03-23T03:50:10.958097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6384":{"gid":6384,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/3091739472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3091739472},{"type":"v1","addr":"172.21.15.102:6837","nonce":3091739472}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T03:50:18.845 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 41 2024-03-23T03:50:18.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.847+0000 7fc585800640 1 -- 172.21.15.102:0/928279720 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc56006ac00 msgr2=0x7fc56006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:18.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.847+0000 7fc585800640 1 --2- 172.21.15.102:0/928279720 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc56006ac00 0x7fc56006d0c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fc56c006fd0 tx=0x7fc56c008040 comp rx=0 tx=0).stop 2024-03-23T03:50:18.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.847+0000 7fc585800640 1 -- 172.21.15.102:0/928279720 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5800b39b0 msgr2=0x7fc580139f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:18.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.847+0000 7fc585800640 1 --2- 172.21.15.102:0/928279720 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5800b39b0 0x7fc580139f70 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc570000c00 tx=0x7fc5700046c0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.847 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.847+0000 7fc585800640 1 -- 172.21.15.102:0/928279720 shutdown_connections 2024-03-23T03:50:18.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.847+0000 7fc585800640 1 --2- 172.21.15.102:0/928279720 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc56006ac00 0x7fc56006d0c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.847+0000 7fc585800640 1 --2- 172.21.15.102:0/928279720 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc58015e500 0x7fc5800cca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.847+0000 7fc585800640 1 --2- 172.21.15.102:0/928279720 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc58015b560 0x7fc5800cc4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.847+0000 7fc585800640 1 --2- 172.21.15.102:0/928279720 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5800b39b0 0x7fc580139f70 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:18.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.847+0000 7fc585800640 1 -- 172.21.15.102:0/928279720 >> 172.21.15.102:0/928279720 conn(0x7fc58006a3d0 msgr2=0x7fc58015e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:18.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.847+0000 7fc585800640 1 -- 172.21.15.102:0/928279720 shutdown_connections 2024-03-23T03:50:18.848 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:18.847+0000 7fc585800640 1 -- 172.21.15.102:0/928279720 wait complete. 2024-03-23T03:50:19.270 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 41, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T03:50:07.937613+0000', 'modified': '2024-03-23T03:50:10.958097+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6384}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6384': {'gid': 6384, 'name': 'a', 'rank': 0, 'incarnation': 37, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.102:6837/3091739472', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 3091739472}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 3091739472}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T03:50:19.270 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T03:50:19.271 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-23T03:50:19.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.483+0000 7f073ca00640 1 Processor -- start 2024-03-23T03:50:19.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.483+0000 7f073ca00640 1 -- start start 2024-03-23T03:50:19.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.485+0000 7f073ca00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f07380af910 0x7f07380ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.485+0000 7f073ca00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f07380cefe0 0x7f07380aff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.485+0000 7f073ca00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0738098a40 con 0x7f073815a7e0 2024-03-23T03:50:19.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.485+0000 7f073ca00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0738099a50 con 0x7f07380cefe0 2024-03-23T03:50:19.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.485+0000 7f073ca00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0738098c90 con 0x7f07380af910 2024-03-23T03:50:19.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.485+0000 7f0737e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f07380cefe0 0x7f07380aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.485+0000 7f0737e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f07380cefe0 0x7f07380aff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38776/0 (socket says 172.21.15.102:38776) 2024-03-23T03:50:19.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.485+0000 7f0737e00640 1 -- 172.21.15.102:0/517652417 learned_addr learned my addr 172.21.15.102:0/517652417 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:19.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.485+0000 7f0736a00640 1 --2- 172.21.15.102:0/517652417 >> v2:172.21.15.173:3300/0 conn(0x7f07380af910 0x7f07380ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.485+0000 7f0737e00640 1 -- 172.21.15.102:0/517652417 >> v2:172.21.15.173:3300/0 conn(0x7f07380af910 msgr2=0x7f07380ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:19.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.485+0000 7f0737e00640 1 --2- 172.21.15.102:0/517652417 >> v2:172.21.15.173:3300/0 conn(0x7f07380af910 0x7f07380ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.485+0000 7f0737e00640 1 -- 172.21.15.102:0/517652417 >> v1:172.21.15.173:6789/0 conn(0x7f073815a7e0 legacy=0x7f07380ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:19.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.486+0000 7f0737e00640 1 -- 172.21.15.102:0/517652417 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0738098f70 con 0x7f07380cefe0 2024-03-23T03:50:19.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.486+0000 7f0736a00640 1 --2- 172.21.15.102:0/517652417 >> v2:172.21.15.173:3300/0 conn(0x7f07380af910 0x7f07380ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:19.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.487+0000 7f0737e00640 1 --2- 172.21.15.102:0/517652417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07380cefe0 0x7f07380aff10 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f07380b1b60 tx=0x7f073002b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0d152d6b9e97ea8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:19.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.487+0000 7f0736000640 1 -- 172.21.15.102:0/517652417 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0730031890 con 0x7f07380cefe0 2024-03-23T03:50:19.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.487+0000 7f0736000640 1 -- 172.21.15.102:0/517652417 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07300319f0 con 0x7f07380cefe0 2024-03-23T03:50:19.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.487+0000 7f0736000640 1 -- 172.21.15.102:0/517652417 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0730031d80 con 0x7f07380cefe0 2024-03-23T03:50:19.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.487+0000 7f073ca00640 1 -- 172.21.15.102:0/517652417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07380cefe0 msgr2=0x7f07380aff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:19.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.487+0000 7f073ca00640 1 --2- 172.21.15.102:0/517652417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07380cefe0 0x7f07380aff10 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f07380b1b60 tx=0x7f073002b930 comp rx=0 tx=0).stop 2024-03-23T03:50:19.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.487+0000 7f073ca00640 1 -- 172.21.15.102:0/517652417 shutdown_connections 2024-03-23T03:50:19.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.487+0000 7f073ca00640 1 --2- 172.21.15.102:0/517652417 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07380cefe0 0x7f07380aff10 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.487+0000 7f073ca00640 1 --2- 172.21.15.102:0/517652417 >> v2:172.21.15.173:3300/0 conn(0x7f07380af910 0x7f07380ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.487+0000 7f073ca00640 1 -- 172.21.15.102:0/517652417 >> 172.21.15.102:0/517652417 conn(0x7f07380d05f0 msgr2=0x7f07380d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:19.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.488+0000 7f073ca00640 1 -- 172.21.15.102:0/517652417 shutdown_connections 2024-03-23T03:50:19.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.488+0000 7f073ca00640 1 -- 172.21.15.102:0/517652417 wait complete. 2024-03-23T03:50:19.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.489+0000 7f073ca00640 1 Processor -- start 2024-03-23T03:50:19.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.489+0000 7f073ca00640 1 -- start start 2024-03-23T03:50:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.490+0000 7f073ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07380af910 0x7f07380c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.490+0000 7f073ca00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f07380cefe0 0x7f073815d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.490+0000 7f073ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f073815a7e0 0x7f0738157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.490+0000 7f073ca00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073815d680 con 0x7f07380af910 2024-03-23T03:50:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.490+0000 7f073ca00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07380b5ca0 con 0x7f073815a7e0 2024-03-23T03:50:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.490+0000 7f073ca00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f073815ccb0 con 0x7f07380cefe0 2024-03-23T03:50:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.491+0000 7f0737400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07380af910 0x7f07380c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.491+0000 7f0737400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07380af910 0x7f07380c6a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38778/0 (socket says 172.21.15.102:38778) 2024-03-23T03:50:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.491+0000 7f0737400640 1 -- 172.21.15.102:0/1420429004 learned_addr learned my addr 172.21.15.102:0/1420429004 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.491+0000 7f0736a00640 1 --2- 172.21.15.102:0/1420429004 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f07380cefe0 0x7f073815d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.491+0000 7f0737e00640 1 --2- 172.21.15.102:0/1420429004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f073815a7e0 0x7f0738157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.491+0000 7f0737400640 1 -- 172.21.15.102:0/1420429004 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f07380cefe0 msgr2=0x7f073815d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:19.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.491+0000 7f0737400640 1 --2- 172.21.15.102:0/1420429004 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f07380cefe0 0x7f073815d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.491+0000 7f0737400640 1 -- 172.21.15.102:0/1420429004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f073815a7e0 msgr2=0x7f0738157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:19.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.491+0000 7f0737400640 1 --2- 172.21.15.102:0/1420429004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f073815a7e0 0x7f0738157910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.491+0000 7f0737400640 1 -- 172.21.15.102:0/1420429004 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07300094d0 con 0x7f07380af910 2024-03-23T03:50:19.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.491+0000 7f0737e00640 1 --2- 172.21.15.102:0/1420429004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f073815a7e0 0x7f0738157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:19.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.491+0000 7f0737400640 1 --2- 172.21.15.102:0/1420429004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07380af910 0x7f07380c6a90 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f072800c7c0 tx=0x7f072800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:19.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.492+0000 7f072f400640 1 -- 172.21.15.102:0/1420429004 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072800aa50 con 0x7f07380af910 2024-03-23T03:50:19.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.492+0000 7f072f400640 1 -- 172.21.15.102:0/1420429004 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f072800abb0 con 0x7f07380af910 2024-03-23T03:50:19.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.492+0000 7f072f400640 1 -- 172.21.15.102:0/1420429004 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0728012420 con 0x7f07380af910 2024-03-23T03:50:19.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.492+0000 7f073ca00640 1 -- 172.21.15.102:0/1420429004 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07380c4f30 con 0x7f07380af910 2024-03-23T03:50:19.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.492+0000 7f073ca00640 1 -- 172.21.15.102:0/1420429004 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0738157e50 con 0x7f07380af910 2024-03-23T03:50:19.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.495+0000 7f072f400640 1 -- 172.21.15.102:0/1420429004 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0728012580 con 0x7f07380af910 2024-03-23T03:50:19.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.496+0000 7f0740de4740 1 Processor -- start 2024-03-23T03:50:19.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.496+0000 7f0740de4740 1 -- start start 2024-03-23T03:50:19.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.496+0000 7f072f400640 1 --2- 172.21.15.102:0/1420429004 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f071806ac00 0x7f071806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.496+0000 7f072f400640 1 -- 172.21.15.102:0/1420429004 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0728089780 con 0x7f07380af910 2024-03-23T03:50:19.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.496+0000 7f0737e00640 1 --2- 172.21.15.102:0/1420429004 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f071806ac00 0x7f071806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.497+0000 7f0740de4740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa422c0 0x55efafa15260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.497+0000 7f0740de4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa426a0 0x55efafa45050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.497+0000 7f0740de4740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa157a0 0x55efafa45590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.497+0000 7f0740de4740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efaf9ec2c0 con 0x55efafa422c0 2024-03-23T03:50:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.497+0000 7f0740de4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efaf9818f0 con 0x55efafa426a0 2024-03-23T03:50:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.497+0000 7f0740de4740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x55efaf97f080 con 0x55efafa157a0 2024-03-23T03:50:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.497+0000 7f0737400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa422c0 0x55efafa15260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.497+0000 7f0737e00640 1 --2- 172.21.15.102:0/1420429004 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f071806ac00 0x7f071806d0c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0730036000 tx=0x7f073003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.497+0000 7f0737400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa422c0 0x55efafa15260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38790/0 (socket says 172.21.15.102:38790) 2024-03-23T03:50:19.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.497+0000 7f0737400640 1 -- 172.21.15.102:0/1879810162 learned_addr learned my addr 172.21.15.102:0/1879810162 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.498+0000 7f0737400640 1 -- 172.21.15.102:0/1879810162 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa157a0 msgr2=0x55efafa45590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.498+0000 7f0737400640 1 --2- 172.21.15.102:0/1879810162 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa157a0 0x55efafa45590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.498+0000 7f0737400640 1 -- 172.21.15.102:0/1879810162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa426a0 msgr2=0x55efafa45050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.498+0000 7f0736a00640 1 --2- 172.21.15.102:0/1879810162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa426a0 0x55efafa45050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.498+0000 7f0737400640 1 --2- 172.21.15.102:0/1879810162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa426a0 0x55efafa45050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.498+0000 7f0737400640 1 -- 172.21.15.102:0/1879810162 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efaf945a10 con 0x55efafa422c0 2024-03-23T03:50:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.498+0000 7f0737400640 1 --2- 172.21.15.102:0/1879810162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa422c0 0x55efafa15260 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f072800afd0 tx=0x7f0728058c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a441dc93aeebf2d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:19.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.498+0000 7f0723e00640 1 -- 172.21.15.102:0/1879810162 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072805c120 con 0x55efafa422c0 2024-03-23T03:50:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.498+0000 7f0723e00640 1 -- 172.21.15.102:0/1879810162 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07280608c0 con 0x55efafa422c0 2024-03-23T03:50:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.498+0000 7f0723e00640 1 -- 172.21.15.102:0/1879810162 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0728054be0 con 0x55efafa422c0 2024-03-23T03:50:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.499+0000 7f0740de4740 1 -- 172.21.15.102:0/1879810162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa422c0 msgr2=0x55efafa15260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.499+0000 7f0740de4740 1 --2- 172.21.15.102:0/1879810162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa422c0 0x55efafa15260 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f072800afd0 tx=0x7f0728058c90 comp rx=0 tx=0).stop 2024-03-23T03:50:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.499+0000 7f0740de4740 1 -- 172.21.15.102:0/1879810162 shutdown_connections 2024-03-23T03:50:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.499+0000 7f0740de4740 1 --2- 172.21.15.102:0/1879810162 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa157a0 0x55efafa45590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.499+0000 7f0740de4740 1 --2- 172.21.15.102:0/1879810162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa426a0 0x55efafa45050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.499+0000 7f0740de4740 1 --2- 172.21.15.102:0/1879810162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa422c0 0x55efafa15260 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.499+0000 7f0740de4740 1 -- 172.21.15.102:0/1879810162 >> 172.21.15.102:0/1879810162 conn(0x55efafa174d0 msgr2=0x55efafa60050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.499+0000 7f0740de4740 1 -- 172.21.15.102:0/1879810162 shutdown_connections 2024-03-23T03:50:19.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.499+0000 7f0740de4740 1 -- 172.21.15.102:0/1879810162 wait complete. 2024-03-23T03:50:19.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.500+0000 7f0740de4740 1 -- start start 2024-03-23T03:50:19.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.500+0000 7f0740de4740 1 Processor -- start 2024-03-23T03:50:19.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.501+0000 7f0740de4740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa422c0 0x55efafa2f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.501+0000 7f0740de4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa426a0 0x55efafa30910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.501+0000 7f0740de4740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa30e50 0x55efafa63560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.501+0000 7f0740de4740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efaf97a970 con 0x55efafa422c0 2024-03-23T03:50:19.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.501+0000 7f0740de4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efaf91c0a0 con 0x55efafa426a0 2024-03-23T03:50:19.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.501+0000 7f0740de4740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x55efaf91bd40 con 0x55efafa30e50 2024-03-23T03:50:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.501+0000 7f0740de4740 10 client.0 _ll_register_callbacks cb 0x55efaf973fe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T03:50:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.502+0000 7f0737400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa422c0 0x55efafa2f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.502+0000 7f0737400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa422c0 0x55efafa2f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38796/0 (socket says 172.21.15.102:38796) 2024-03-23T03:50:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.502+0000 7f0737400640 1 -- 172.21.15.102:0/1266748368 learned_addr learned my addr 172.21.15.102:0/1266748368 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.502+0000 7f0736a00640 1 --2- 172.21.15.102:0/1266748368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa426a0 0x55efafa30910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.502+0000 7f0737400640 1 -- 172.21.15.102:0/1266748368 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa30e50 msgr2=0x55efafa63560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.502+0000 7f0737400640 1 --2- 172.21.15.102:0/1266748368 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa30e50 0x55efafa63560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.502+0000 7f0737400640 1 -- 172.21.15.102:0/1266748368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa426a0 msgr2=0x55efafa30910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.502+0000 7f0737400640 1 --2- 172.21.15.102:0/1266748368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa426a0 0x55efafa30910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.502+0000 7f0737400640 1 -- 172.21.15.102:0/1266748368 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0738157e50 con 0x55efafa422c0 2024-03-23T03:50:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.502+0000 7f0737400640 1 --2- 172.21.15.102:0/1266748368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa422c0 0x55efafa2f740 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f0728058690 tx=0x7f072800af60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad18d73dbf58c48 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.502+0000 7f0720c00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-23T03:50:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.503+0000 7f0720c00640 1 -- 172.21.15.102:0/1266748368 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07280579c0 con 0x55efafa422c0 2024-03-23T03:50:19.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.503+0000 7f0720c00640 1 -- 172.21.15.102:0/1266748368 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0728074030 con 0x55efafa422c0 2024-03-23T03:50:19.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.503+0000 7f0740de4740 1 -- 172.21.15.102:0/1266748368 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55efaf9a4b10 con 0x55efafa422c0 2024-03-23T03:50:19.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.503+0000 7f0720c00640 1 -- 172.21.15.102:0/1266748368 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0728057b20 con 0x55efafa422c0 2024-03-23T03:50:19.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.503+0000 7f0740de4740 10 client.6540 fetch_fsmap learned FSMap version 41 2024-03-23T03:50:19.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.503+0000 7f0740de4740 1 -- 172.21.15.102:0/1266748368 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=2+}) v3 -- 0x55efaf978ff0 con 0x55efafa422c0 2024-03-23T03:50:19.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.503+0000 7f0720c00640 1 -- 172.21.15.102:0/1266748368 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55efaf9a4b10 con 0x55efafa422c0 2024-03-23T03:50:19.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.503+0000 7f0720c00640 1 -- 172.21.15.102:0/1266748368 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 41) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0728022020 con 0x55efafa422c0 2024-03-23T03:50:19.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.503+0000 7f0740de4740 10 client.6540 fetch_fsmap finished waiting for FSMap version 41 2024-03-23T03:50:19.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.503+0000 7f0740de4740 10 client.6540 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T03:50:19.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.503+0000 7f0740de4740 20 client.6540 populate_metadata read hostname 'smithi102' 2024-03-23T03:50:19.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.503+0000 7f0740de4740 1 --2- 172.21.15.102:0/1266748368 >> [v2:172.21.15.102:6836/3091739472,v1:172.21.15.102:6837/3091739472] conn(0x55efafa32b40 0x55efafa47090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.503+0000 7f0740de4740 4 client.6540 mds_command: new command op to 6384 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T03:50:19.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.503+0000 7f0740de4740 1 -- 172.21.15.102:0/1266748368 --> [v2:172.21.15.102:6836/3091739472,v1:172.21.15.102:6837/3091739472] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55efaf66e3c0 con 0x55efafa32b40 2024-03-23T03:50:19.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.504+0000 7f0736a00640 1 --2- 172.21.15.102:0/1266748368 >> [v2:172.21.15.102:6836/3091739472,v1:172.21.15.102:6837/3091739472] conn(0x55efafa32b40 0x55efafa47090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.504+0000 7f0736a00640 1 --2- 172.21.15.102:0/1266748368 >> [v2:172.21.15.102:6836/3091739472,v1:172.21.15.102:6837/3091739472] conn(0x55efafa32b40 0x55efafa47090 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb9d4a9498a5503e server_cookie=1dd6b75effe6feb8 in_seq=0 out_seq=0 2024-03-23T03:50:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.504+0000 7f0720c00640 10 client.6540 ms_handle_connect on v2:172.21.15.102:6836/3091739472 2024-03-23T03:50:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.505+0000 7f0720c00640 1 -- 172.21.15.102:0/1266748368 <== mds.0 v2:172.21.15.102:6836/3091739472 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55efaf66e3c0 con 0x55efafa32b40 2024-03-23T03:50:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.505+0000 7f0720c00640 10 client.6540 handle_command_reply: tid=0 2024-03-23T03:50:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.505+0000 7f0740de4740 1 client.6540 shutdown 2024-03-23T03:50:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.505+0000 7f0740de4740 1 -- 172.21.15.102:0/1266748368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa422c0 msgr2=0x55efafa2f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.505+0000 7f0740de4740 1 --2- 172.21.15.102:0/1266748368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa422c0 0x55efafa2f740 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f0728058690 tx=0x7f072800af60 comp rx=0 tx=0).stop 2024-03-23T03:50:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.506+0000 7f0740de4740 1 -- 172.21.15.102:0/1266748368 shutdown_connections 2024-03-23T03:50:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.506+0000 7f0740de4740 1 --2- 172.21.15.102:0/1266748368 >> [v2:172.21.15.102:6836/3091739472,v1:172.21.15.102:6837/3091739472] conn(0x55efafa32b40 0x55efafa47090 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.506+0000 7f0740de4740 1 --2- 172.21.15.102:0/1266748368 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa30e50 0x55efafa63560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.506+0000 7f0740de4740 1 --2- 172.21.15.102:0/1266748368 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa426a0 0x55efafa30910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.506+0000 7f0740de4740 1 --2- 172.21.15.102:0/1266748368 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa422c0 0x55efafa2f740 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.506+0000 7f0720c00640 0 client.6540 ms_handle_reset on v2:172.21.15.102:6836/3091739472 2024-03-23T03:50:19.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.506+0000 7f0740de4740 1 -- 172.21.15.102:0/1266748368 >> 172.21.15.102:0/1266748368 conn(0x55efafa157a0 msgr2=0x55efafa10530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:19.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.506+0000 7f0740de4740 1 -- 172.21.15.102:0/1266748368 shutdown_connections 2024-03-23T03:50:19.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.506+0000 7f0740de4740 1 -- 172.21.15.102:0/1266748368 wait complete. 2024-03-23T03:50:19.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.506+0000 7f0740de4740 20 client.6540 trim_cache size 0 max 16384 2024-03-23T03:50:19.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.522+0000 7f0740de4740 1 Processor -- start 2024-03-23T03:50:19.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.522+0000 7f0740de4740 1 -- start start 2024-03-23T03:50:19.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.523+0000 7f0740de4740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa30e50 0x55efafac4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.523+0000 7f0740de4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa422c0 0x55efafa45230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.523+0000 7f0740de4740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa426a0 0x55efafa45770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.523+0000 7f0740de4740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efaf6e1f20 con 0x55efafa30e50 2024-03-23T03:50:19.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.523+0000 7f0740de4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efafa16070 con 0x55efafa422c0 2024-03-23T03:50:19.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.523+0000 7f0740de4740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x55efafa59ff0 con 0x55efafa426a0 2024-03-23T03:50:19.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.523+0000 7f0737400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa30e50 0x55efafac4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.523+0000 7f0737400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa30e50 0x55efafac4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38810/0 (socket says 172.21.15.102:38810) 2024-03-23T03:50:19.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.523+0000 7f0737400640 1 -- 172.21.15.102:0/448911251 learned_addr learned my addr 172.21.15.102:0/448911251 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:19.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.524+0000 7f0737e00640 1 --2- 172.21.15.102:0/448911251 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa426a0 0x55efafa45770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.524+0000 7f0737400640 1 -- 172.21.15.102:0/448911251 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa426a0 msgr2=0x55efafa45770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:19.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.524+0000 7f0736a00640 1 --2- 172.21.15.102:0/448911251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa422c0 0x55efafa45230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.524+0000 7f0737400640 1 --2- 172.21.15.102:0/448911251 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa426a0 0x55efafa45770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.524+0000 7f0737400640 1 -- 172.21.15.102:0/448911251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa422c0 msgr2=0x55efafa45230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:19.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.524+0000 7f0737400640 1 --2- 172.21.15.102:0/448911251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa422c0 0x55efafa45230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.524+0000 7f0737400640 1 -- 172.21.15.102:0/448911251 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55efaf9ae7c0 con 0x55efafa30e50 2024-03-23T03:50:19.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.524+0000 7f0737e00640 1 --2- 172.21.15.102:0/448911251 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa426a0 0x55efafa45770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.524+0000 7f0737400640 1 --2- 172.21.15.102:0/448911251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa30e50 0x55efafac4160 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f0728054ef0 tx=0x7f0728087000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=774523425b08fb4f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.524+0000 7f072cc00640 1 -- 172.21.15.102:0/448911251 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072805c120 con 0x55efafa30e50 2024-03-23T03:50:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.525+0000 7f072cc00640 1 -- 172.21.15.102:0/448911251 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07280579c0 con 0x55efafa30e50 2024-03-23T03:50:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.525+0000 7f072cc00640 1 -- 172.21.15.102:0/448911251 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072805e3d0 con 0x55efafa30e50 2024-03-23T03:50:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.525+0000 7f0740de4740 1 -- 172.21.15.102:0/448911251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa30e50 msgr2=0x55efafac4160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.525+0000 7f0740de4740 1 --2- 172.21.15.102:0/448911251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa30e50 0x55efafac4160 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f0728054ef0 tx=0x7f0728087000 comp rx=0 tx=0).stop 2024-03-23T03:50:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.525+0000 7f0740de4740 1 -- 172.21.15.102:0/448911251 shutdown_connections 2024-03-23T03:50:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.525+0000 7f0740de4740 1 --2- 172.21.15.102:0/448911251 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa426a0 0x55efafa45770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.525+0000 7f0740de4740 1 --2- 172.21.15.102:0/448911251 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa422c0 0x55efafa45230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.525+0000 7f0740de4740 1 --2- 172.21.15.102:0/448911251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafa30e50 0x55efafac4160 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.525+0000 7f0740de4740 1 -- 172.21.15.102:0/448911251 >> 172.21.15.102:0/448911251 conn(0x55efafa157a0 msgr2=0x55efafa1c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.525+0000 7f0740de4740 1 -- 172.21.15.102:0/448911251 shutdown_connections 2024-03-23T03:50:19.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.526+0000 7f0740de4740 1 -- 172.21.15.102:0/448911251 wait complete. 2024-03-23T03:50:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.526+0000 7f0740de4740 1 -- start start 2024-03-23T03:50:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.526+0000 7f0740de4740 1 Processor -- start 2024-03-23T03:50:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.527+0000 7f0740de4740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa422c0 0x55efafa45230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.527+0000 7f0740de4740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa426a0 0x55efafa45770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.527+0000 7f0740de4740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafac3560 0x55efafa2fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.527+0000 7f0740de4740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efafa100b0 con 0x55efafac3560 2024-03-23T03:50:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.527+0000 7f0740de4740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55efafa59ab0 con 0x55efafa422c0 2024-03-23T03:50:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.527+0000 7f0740de4740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x55efafa48ff0 con 0x55efafa426a0 2024-03-23T03:50:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.527+0000 7f0740de4740 10 client.0 _ll_register_callbacks cb 0x7f0728057ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T03:50:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.528+0000 7f0737e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafac3560 0x55efafa2fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.528+0000 7f0737e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafac3560 0x55efafa2fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38814/0 (socket says 172.21.15.102:38814) 2024-03-23T03:50:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.528+0000 7f0737e00640 1 -- 172.21.15.102:0/1344241544 learned_addr learned my addr 172.21.15.102:0/1344241544 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.528+0000 7f0736a00640 1 --2- 172.21.15.102:0/1344241544 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa426a0 0x55efafa45770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.528+0000 7f0737400640 1 --2- 172.21.15.102:0/1344241544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa422c0 0x55efafa45230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.528+0000 7f0737e00640 1 -- 172.21.15.102:0/1344241544 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa426a0 msgr2=0x55efafa45770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.528+0000 7f0737e00640 1 --2- 172.21.15.102:0/1344241544 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa426a0 0x55efafa45770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.528+0000 7f0737e00640 1 -- 172.21.15.102:0/1344241544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa422c0 msgr2=0x55efafa45230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.528+0000 7f0737e00640 1 --2- 172.21.15.102:0/1344241544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa422c0 0x55efafa45230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.528+0000 7f0737e00640 1 -- 172.21.15.102:0/1344241544 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55efaf978ff0 con 0x55efafac3560 2024-03-23T03:50:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.529+0000 7f0737e00640 1 --2- 172.21.15.102:0/1344241544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafac3560 0x55efafa2fc40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f0730006b10 tx=0x7f073003ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5e17f3a774c7e2a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.529+0000 7f0720c00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-23T03:50:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.529+0000 7f0720c00640 1 -- 172.21.15.102:0/1344241544 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f073004a070 con 0x55efafac3560 2024-03-23T03:50:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.529+0000 7f0720c00640 1 -- 172.21.15.102:0/1344241544 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0730045850 con 0x55efafac3560 2024-03-23T03:50:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.529+0000 7f0740de4740 1 -- 172.21.15.102:0/1344241544 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f07280608c0 con 0x55efafac3560 2024-03-23T03:50:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.529+0000 7f0720c00640 1 -- 172.21.15.102:0/1344241544 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0730055ab0 con 0x55efafac3560 2024-03-23T03:50:19.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.529+0000 7f0740de4740 10 client.6546 fetch_fsmap learned FSMap version 41 2024-03-23T03:50:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.529+0000 7f0740de4740 1 -- 172.21.15.102:0/1344241544 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=2+}) v3 -- 0x55efaf91b2b0 con 0x55efafac3560 2024-03-23T03:50:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.529+0000 7f0720c00640 1 -- 172.21.15.102:0/1344241544 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07280608c0 con 0x55efafac3560 2024-03-23T03:50:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.529+0000 7f0720c00640 1 -- 172.21.15.102:0/1344241544 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 41) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0730045a10 con 0x55efafac3560 2024-03-23T03:50:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.529+0000 7f0740de4740 10 client.6546 fetch_fsmap finished waiting for FSMap version 41 2024-03-23T03:50:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.529+0000 7f0740de4740 10 client.6546 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T03:50:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.529+0000 7f0740de4740 20 client.6546 populate_metadata read hostname 'smithi102' 2024-03-23T03:50:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.529+0000 7f0740de4740 1 --2- 172.21.15.102:0/1344241544 >> [v2:172.21.15.102:6836/3091739472,v1:172.21.15.102:6837/3091739472] conn(0x55efafa30580 0x55efafac99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.529+0000 7f0740de4740 4 client.6546 mds_command: new command op to 6384 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-23T03:50:19.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.529+0000 7f0740de4740 1 -- 172.21.15.102:0/1344241544 --> [v2:172.21.15.102:6836/3091739472,v1:172.21.15.102:6837/3091739472] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55efaf67f6c0 con 0x55efafa30580 2024-03-23T03:50:19.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.530+0000 7f0737400640 1 --2- 172.21.15.102:0/1344241544 >> [v2:172.21.15.102:6836/3091739472,v1:172.21.15.102:6837/3091739472] conn(0x55efafa30580 0x55efafac99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.530+0000 7f0737400640 1 --2- 172.21.15.102:0/1344241544 >> [v2:172.21.15.102:6836/3091739472,v1:172.21.15.102:6837/3091739472] conn(0x55efafa30580 0x55efafac99f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dbf41d537fb1172 server_cookie=a05f4bff1f74b6f in_seq=0 out_seq=0 2024-03-23T03:50:19.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.530+0000 7f0720c00640 10 client.6546 ms_handle_connect on v2:172.21.15.102:6836/3091739472 2024-03-23T03:50:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.530+0000 7f0720c00640 1 -- 172.21.15.102:0/1344241544 <== mds.0 v2:172.21.15.102:6836/3091739472 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x55efaf67f6c0 con 0x55efafa30580 2024-03-23T03:50:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.531+0000 7f0720c00640 10 client.6546 handle_command_reply: tid=0 2024-03-23T03:50:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.531+0000 7f0740de4740 1 client.6546 shutdown 2024-03-23T03:50:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.531+0000 7f0740de4740 1 -- 172.21.15.102:0/1344241544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafac3560 msgr2=0x55efafa2fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.531+0000 7f0740de4740 1 --2- 172.21.15.102:0/1344241544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafac3560 0x55efafa2fc40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f0730006b10 tx=0x7f073003ab80 comp rx=0 tx=0).stop 2024-03-23T03:50:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.531+0000 7f0740de4740 1 -- 172.21.15.102:0/1344241544 shutdown_connections 2024-03-23T03:50:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.531+0000 7f0740de4740 1 --2- 172.21.15.102:0/1344241544 >> [v2:172.21.15.102:6836/3091739472,v1:172.21.15.102:6837/3091739472] conn(0x55efafa30580 0x55efafac99f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.531+0000 7f0740de4740 1 --2- 172.21.15.102:0/1344241544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55efafac3560 0x55efafa2fc40 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.531+0000 7f0740de4740 1 --2- 172.21.15.102:0/1344241544 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55efafa426a0 0x55efafa45770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.531+0000 7f0740de4740 1 --2- 172.21.15.102:0/1344241544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55efafa422c0 0x55efafa45230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.531+0000 7f0720c00640 0 client.6546 ms_handle_reset on v2:172.21.15.102:6836/3091739472 2024-03-23T03:50:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.531+0000 7f0740de4740 1 -- 172.21.15.102:0/1344241544 >> 172.21.15.102:0/1344241544 conn(0x55efafa157a0 msgr2=0x55efafa48600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.531+0000 7f0740de4740 1 -- 172.21.15.102:0/1344241544 shutdown_connections 2024-03-23T03:50:19.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.531+0000 7f0740de4740 1 -- 172.21.15.102:0/1344241544 wait complete. 2024-03-23T03:50:19.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.531+0000 7f0740de4740 20 client.6546 trim_cache size 0 max 16384 2024-03-23T03:50:19.535 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:50:19.535 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "776b2820-05f2-4e50-8d40-18dd76c5f6a0", 2024-03-23T03:50:19.535 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 0, 2024-03-23T03:50:19.535 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 6384, 2024-03-23T03:50:19.535 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-03-23T03:50:19.535 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-03-23T03:50:19.535 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-03-23T03:50:19.535 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 11.589156385000001, 2024-03-23T03:50:19.535 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 41, 2024-03-23T03:50:19.535 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 81, 2024-03-23T03:50:19.536 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 81, 2024-03-23T03:50:19.536 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 16.526571334 2024-03-23T03:50:19.536 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:50:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.532+0000 7f073ca00640 1 -- 172.21.15.102:0/1420429004 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f071806ac00 msgr2=0x7f071806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.532+0000 7f073ca00640 1 --2- 172.21.15.102:0/1420429004 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f071806ac00 0x7f071806d0c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0730036000 tx=0x7f073003c040 comp rx=0 tx=0).stop 2024-03-23T03:50:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.532+0000 7f073ca00640 1 -- 172.21.15.102:0/1420429004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07380af910 msgr2=0x7f07380c6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.532+0000 7f073ca00640 1 --2- 172.21.15.102:0/1420429004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07380af910 0x7f07380c6a90 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f072800c7c0 tx=0x7f072800cc80 comp rx=0 tx=0).stop 2024-03-23T03:50:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.532+0000 7f073ca00640 1 -- 172.21.15.102:0/1420429004 shutdown_connections 2024-03-23T03:50:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.532+0000 7f073ca00640 1 --2- 172.21.15.102:0/1420429004 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f071806ac00 0x7f071806d0c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.532+0000 7f073ca00640 1 --2- 172.21.15.102:0/1420429004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f073815a7e0 0x7f0738157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.532+0000 7f073ca00640 1 --2- 172.21.15.102:0/1420429004 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f07380cefe0 0x7f073815d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.532+0000 7f073ca00640 1 --2- 172.21.15.102:0/1420429004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07380af910 0x7f07380c6a90 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.532+0000 7f073ca00640 1 -- 172.21.15.102:0/1420429004 >> 172.21.15.102:0/1420429004 conn(0x7f07380d05f0 msgr2=0x7f073806e350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.532+0000 7f073ca00640 1 -- 172.21.15.102:0/1420429004 shutdown_connections 2024-03-23T03:50:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.532+0000 7f073ca00640 1 -- 172.21.15.102:0/1420429004 wait complete. 2024-03-23T03:50:19.551 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:50:19.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.751+0000 7f6944e00640 1 Processor -- start 2024-03-23T03:50:19.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.751+0000 7f6944e00640 1 -- start start 2024-03-23T03:50:19.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.752+0000 7f6944e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f694015a8b0 0x7f694006d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.752+0000 7f6944e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f694015ac90 con 0x7f69400659d0 2024-03-23T03:50:19.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.752+0000 7f6944e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6940065f70 con 0x7f694006b3b0 2024-03-23T03:50:19.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.752+0000 7f6944e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69400b1ca0 con 0x7f694015a8b0 2024-03-23T03:50:19.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.753+0000 7f693d400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f69400659d0 0x7f694006aca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44308/0 (socket says 172.21.15.102:44308) 2024-03-23T03:50:19.753 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.753+0000 7f693d400640 1 -- 172.21.15.102:0/837136627 learned_addr learned my addr 172.21.15.102:0/837136627 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.753+0000 7f693de00640 1 --2- 172.21.15.102:0/837136627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f694015a8b0 0x7f694006d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.753+0000 7f693ca00640 1 -- 172.21.15.102:0/837136627 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396086719 0 0) 0x7f694015ac90 con 0x7f69400659d0 2024-03-23T03:50:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.753+0000 7f693ca00640 1 -- 172.21.15.102:0/837136627 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6924003520 con 0x7f69400659d0 2024-03-23T03:50:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.753+0000 7f693ca00640 1 -- 172.21.15.102:0/837136627 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1073326964 0 0) 0x7f6924003520 con 0x7f69400659d0 2024-03-23T03:50:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.753+0000 7f693ca00640 1 -- 172.21.15.102:0/837136627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f694015a8b0 msgr2=0x7f694006d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.753+0000 7f693ca00640 1 --2- 172.21.15.102:0/837136627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f694015a8b0 0x7f694006d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.753+0000 7f693ca00640 1 -- 172.21.15.102:0/837136627 >> v1:172.21.15.173:6789/0 conn(0x7f694006b3b0 legacy=0x7f6940164380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.754+0000 7f693ca00640 1 -- 172.21.15.102:0/837136627 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69400b1f80 con 0x7f69400659d0 2024-03-23T03:50:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.754+0000 7f693ca00640 1 -- 172.21.15.102:0/837136627 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f6930001670 con 0x7f69400659d0 2024-03-23T03:50:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.754+0000 7f693ca00640 1 -- 172.21.15.102:0/837136627 >> v1:172.21.15.102:6789/0 conn(0x7f69400659d0 legacy=0x7f694006aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.754+0000 7f693ca00640 1 --2- 172.21.15.102:0/837136627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69240039a0 0x7f6924003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.754+0000 7f693ca00640 1 --2- 172.21.15.102:0/837136627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69240047c0 0x7f6924004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.754+0000 7f693ca00640 1 --2- 172.21.15.102:0/837136627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f694015a8b0 0x7f6924005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.754+0000 7f693ca00640 1 -- 172.21.15.102:0/837136627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69400b1ca0 con 0x7f69240047c0 2024-03-23T03:50:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.754+0000 7f693ca00640 1 -- 172.21.15.102:0/837136627 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f692400bb10 con 0x7f69240039a0 2024-03-23T03:50:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.754+0000 7f693ca00640 1 -- 172.21.15.102:0/837136627 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f692400bcd0 con 0x7f694015a8b0 2024-03-23T03:50:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.754+0000 7f693d400640 1 --2- 172.21.15.102:0/837136627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69240047c0 0x7f6924004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.754+0000 7f693de00640 1 --2- 172.21.15.102:0/837136627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69240039a0 0x7f6924003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.754+0000 7f693e800640 1 --2- 172.21.15.102:0/837136627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f694015a8b0 0x7f6924005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.755+0000 7f693d400640 1 -- 172.21.15.102:0/837136627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f694015a8b0 msgr2=0x7f6924005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.755+0000 7f693d400640 1 --2- 172.21.15.102:0/837136627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f694015a8b0 0x7f6924005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.756 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.755+0000 7f693d400640 1 -- 172.21.15.102:0/837136627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69240039a0 msgr2=0x7f6924003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:19.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.755+0000 7f693d400640 1 --2- 172.21.15.102:0/837136627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69240039a0 0x7f6924003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.755+0000 7f693d400640 1 -- 172.21.15.102:0/837136627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6924003520 con 0x7f69240047c0 2024-03-23T03:50:19.757 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.756+0000 7f693d400640 1 --2- 172.21.15.102:0/837136627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69240047c0 0x7f6924004c40 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f693000c0e0 tx=0x7f69240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c07b8082d69015e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.756+0000 7f693ca00640 1 -- 172.21.15.102:0/837136627 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6930005ad0 con 0x7f69240047c0 2024-03-23T03:50:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.756+0000 7f693ca00640 1 -- 172.21.15.102:0/837136627 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6930005c30 con 0x7f69240047c0 2024-03-23T03:50:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.756+0000 7f693ca00640 1 -- 172.21.15.102:0/837136627 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693002c810 con 0x7f69240047c0 2024-03-23T03:50:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.757+0000 7f6944e00640 1 -- 172.21.15.102:0/837136627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69240047c0 msgr2=0x7f6924004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.757+0000 7f6944e00640 1 --2- 172.21.15.102:0/837136627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69240047c0 0x7f6924004c40 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f693000c0e0 tx=0x7f69240057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.757+0000 7f693e800640 1 -- 172.21.15.102:0/837136627 reap_dead start 2024-03-23T03:50:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.757+0000 7f6944e00640 1 -- 172.21.15.102:0/837136627 shutdown_connections 2024-03-23T03:50:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.757+0000 7f6944e00640 1 -- 172.21.15.102:0/837136627 >> 172.21.15.102:0/837136627 conn(0x7f6940070a20 msgr2=0x7f6940070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.757+0000 7f6944e00640 1 -- 172.21.15.102:0/837136627 shutdown_connections 2024-03-23T03:50:19.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.757+0000 7f6944e00640 1 -- 172.21.15.102:0/837136627 wait complete. 2024-03-23T03:50:19.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.759+0000 7f6944e00640 1 Processor -- start 2024-03-23T03:50:19.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.759+0000 7f6944e00640 1 -- start start 2024-03-23T03:50:19.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.759+0000 7f6944e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69240047c0 0x7f694015d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.760+0000 7f6944e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6940155180 0x7f6940151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.760+0000 7f6944e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f694014d4b0 0x7f694014d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.760+0000 7f6944e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694014f250 con 0x7f694014d4b0 2024-03-23T03:50:19.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.760+0000 7f6944e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694015db50 con 0x7f69240047c0 2024-03-23T03:50:19.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.760+0000 7f6944e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f694014de70 con 0x7f6940155180 2024-03-23T03:50:19.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.760+0000 7f693e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f694014d4b0 0x7f694014d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.760+0000 7f693e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f694014d4b0 0x7f694014d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38826/0 (socket says 172.21.15.102:38826) 2024-03-23T03:50:19.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.760+0000 7f693e800640 1 -- 172.21.15.102:0/21459881 learned_addr learned my addr 172.21.15.102:0/21459881 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:19.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.760+0000 7f693de00640 1 --2- 172.21.15.102:0/21459881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69240047c0 0x7f694015d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.760+0000 7f693d400640 1 --2- 172.21.15.102:0/21459881 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6940155180 0x7f6940151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.760+0000 7f693e800640 1 -- 172.21.15.102:0/21459881 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6940155180 msgr2=0x7f6940151480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:19.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.760+0000 7f693e800640 1 --2- 172.21.15.102:0/21459881 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6940155180 0x7f6940151480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.760+0000 7f693e800640 1 -- 172.21.15.102:0/21459881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69240047c0 msgr2=0x7f694015d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:19.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.760+0000 7f693e800640 1 --2- 172.21.15.102:0/21459881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69240047c0 0x7f694015d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:19.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.760+0000 7f693e800640 1 -- 172.21.15.102:0/21459881 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69401543b0 con 0x7f694014d4b0 2024-03-23T03:50:19.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.760+0000 7f693d400640 1 --2- 172.21.15.102:0/21459881 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6940155180 0x7f6940151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:19.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.760+0000 7f693de00640 1 --2- 172.21.15.102:0/21459881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69240047c0 0x7f694015d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:19.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.761+0000 7f693e800640 1 --2- 172.21.15.102:0/21459881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f694014d4b0 0x7f694014d930 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f69400706d0 tx=0x7f693800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:19.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.761+0000 7f6936000640 1 -- 172.21.15.102:0/21459881 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6938003970 con 0x7f694014d4b0 2024-03-23T03:50:19.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.761+0000 7f6936000640 1 -- 172.21.15.102:0/21459881 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6938003ad0 con 0x7f694014d4b0 2024-03-23T03:50:19.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.761+0000 7f6936000640 1 -- 172.21.15.102:0/21459881 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6938003e80 con 0x7f694014d4b0 2024-03-23T03:50:19.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.761+0000 7f6944e00640 1 -- 172.21.15.102:0/21459881 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6940154600 con 0x7f694014d4b0 2024-03-23T03:50:19.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.761+0000 7f6944e00640 1 -- 172.21.15.102:0/21459881 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6940154bc0 con 0x7f694014d4b0 2024-03-23T03:50:19.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.763+0000 7f6944e00640 1 -- 172.21.15.102:0/21459881 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f694014f3c0 con 0x7f694014d4b0 2024-03-23T03:50:19.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.763+0000 7f6936000640 1 -- 172.21.15.102:0/21459881 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f69380050d0 con 0x7f694014d4b0 2024-03-23T03:50:19.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.764+0000 7f6936000640 1 --2- 172.21.15.102:0/21459881 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f692006ac00 0x7f692006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:19.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.764+0000 7f6936000640 1 -- 172.21.15.102:0/21459881 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f69380892c0 con 0x7f694014d4b0 2024-03-23T03:50:19.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.764+0000 7f693de00640 1 --2- 172.21.15.102:0/21459881 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f692006ac00 0x7f692006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:19.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.765+0000 7f693de00640 1 --2- 172.21.15.102:0/21459881 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f692006ac00 0x7f692006d0c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f692c006fd0 tx=0x7f692c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:19.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:19.769+0000 7f6936000640 1 -- 172.21.15.102:0/21459881 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f694014f3c0 con 0x7f694014d4b0 2024-03-23T03:50:20.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:20.006+0000 7f6944e00640 1 -- 172.21.15.102:0/21459881 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f69400661b0 con 0x7f694014d4b0 2024-03-23T03:50:20.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:20.007+0000 7f6936000640 1 -- 172.21.15.102:0/21459881 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v41) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6938054350 con 0x7f694014d4b0 2024-03-23T03:50:20.007 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T03:50:20.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:20.009+0000 7f6944e00640 1 -- 172.21.15.102:0/21459881 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f692006ac00 msgr2=0x7f692006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:20.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:20.009+0000 7f6944e00640 1 --2- 172.21.15.102:0/21459881 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f692006ac00 0x7f692006d0c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f692c006fd0 tx=0x7f692c008040 comp rx=0 tx=0).stop 2024-03-23T03:50:20.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:20.009+0000 7f6944e00640 1 -- 172.21.15.102:0/21459881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f694014d4b0 msgr2=0x7f694014d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:20.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:20.009+0000 7f6944e00640 1 --2- 172.21.15.102:0/21459881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f694014d4b0 0x7f694014d930 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f69400706d0 tx=0x7f693800ec10 comp rx=0 tx=0).stop 2024-03-23T03:50:20.010 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:20.009+0000 7f6944e00640 1 -- 172.21.15.102:0/21459881 shutdown_connections 2024-03-23T03:50:20.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:20.009+0000 7f6944e00640 1 --2- 172.21.15.102:0/21459881 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f692006ac00 0x7f692006d0c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:20.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:20.009+0000 7f6944e00640 1 --2- 172.21.15.102:0/21459881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f694014d4b0 0x7f694014d930 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:20.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:20.009+0000 7f6944e00640 1 --2- 172.21.15.102:0/21459881 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6940155180 0x7f6940151480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:20.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:20.009+0000 7f6944e00640 1 --2- 172.21.15.102:0/21459881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69240047c0 0x7f694015d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:20.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:20.009+0000 7f6944e00640 1 -- 172.21.15.102:0/21459881 >> 172.21.15.102:0/21459881 conn(0x7f6940070a20 msgr2=0x7f69400abd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:20.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:20.010+0000 7f6944e00640 1 -- 172.21.15.102:0/21459881 shutdown_connections 2024-03-23T03:50:20.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:20.010+0000 7f6944e00640 1 -- 172.21.15.102:0/21459881 wait complete. 2024-03-23T03:50:20.027 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-23T03:50:20.027 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:50:20.027 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2024-03-23T03:50:20.027 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T03:50:20.027 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T03:50:20.027 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-23T03:50:20.027 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-23T03:50:20.027 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:50:20.028 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:50:20.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:20.028 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2024-03-23T03:50:20.069 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:50:20.069 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:50:20.069 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:50:20.069 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:20.069 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-03-23T03:50:20.069 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:20.070 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:50:20.070 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:20.070 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f0 2024-03-23T03:50:20.070 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:50:20.070 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:20.070 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f1 2024-03-23T03:50:20.070 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:50:20.070 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:20.070 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:50:20.070 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:20.070 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:50:20.070 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 42450sec preferred_lft 42450sec 2024-03-23T03:50:20.070 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-03-23T03:50:20.071 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:20.071 INFO:teuthology.orchestra.run.smithi102.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:50:20.071 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether aa:17:e3:39:b0:56 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:20.071 INFO:teuthology.orchestra.run.smithi102.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T03:50:20.071 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:20.071 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::5cce:1fff:fe41:5a05/64 scope link 2024-03-23T03:50:20.071 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:20.071 INFO:teuthology.orchestra.run.smithi102.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:50:20.071 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether aa:17:e3:39:b0:56 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T03:50:20.071 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::a817:e3ff:fe39:b056/64 scope link 2024-03-23T03:50:20.071 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:20.071 INFO:teuthology.orchestra.run.smithi102.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:50:20.071 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether ce:9b:95:fb:de:c5 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T03:50:20.071 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::cc9b:95ff:fefb:dec5/64 scope link 2024-03-23T03:50:20.072 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:20.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:20.072 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:50:20.195 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-23T03:50:20.200 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:20.200 DEBUG:teuthology.orchestra.run.smithi102:> route 2024-03-23T03:50:20.245 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2024-03-23T03:50:20.245 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:50:20.245 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:50:20.245 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:50:20.245 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:50:20.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:20.246 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:50:20.246 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:50:20.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:50:20.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:50:20.246 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:50:20.246 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:50:20.290 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:50:20.337 DEBUG:teuthology.orchestra.run.smithi173:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:50:20.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:20.441 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-23T03:50:20.526 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T03:50:20.526 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T03:50:20.527 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:50:20.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:20.527 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.0 up 2024-03-23T03:50:20.662 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:50:20.663 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T03:50:20.714 INFO:teuthology.orchestra.run.smithi102.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:50:20.714 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:50:20.714 DEBUG:teuthology.orchestra.run.smithi102:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T03:50:20.761 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-23T03:50:20.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:20.761 DEBUG:teuthology.orchestra.run.smithi102:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:50:20.878 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:50:20.878 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:50:20.878 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:50:20.896 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:50:20.896 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:50:20.896 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:50:20.896 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:50:20.896 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:50:20.921 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:50:20.921 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:50:20.922 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:50:20.922 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:50:20.922 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:50:20.922 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790" 2024-03-23T03:50:20.922 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/ 2024-03-23T03:50:20.922 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=776b2820-05f2-4e50-8d40-18dd76c5f6a0" will pass to kernel 2024-03-23T03:50:20.923 INFO:teuthology.orchestra.run.smithi102.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-23T03:50:20.923 INFO:teuthology.orchestra.run.smithi102.stdout: You just mounted a file system that supports labels which does not 2024-03-23T03:50:20.923 INFO:teuthology.orchestra.run.smithi102.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T03:50:20.923 INFO:teuthology.orchestra.run.smithi102.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T03:50:20.923 INFO:teuthology.orchestra.run.smithi102.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T03:50:20.927 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T03:50:20.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:20.927 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T03:50:21.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:21.001 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:50:21.053 INFO:teuthology.orchestra.run.smithi102.stdout:client5120 2024-03-23T03:50:21.053 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:50:21.053 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:21.096 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:50:21.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.343+0000 7f459d000640 1 Processor -- start 2024-03-23T03:50:21.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.343+0000 7f459d000640 1 -- start start 2024-03-23T03:50:21.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.345+0000 7f459d000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f45980adbc0 0x7f45980b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:21.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.345+0000 7f459d000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45980b16a0 con 0x7f459806d390 2024-03-23T03:50:21.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.345+0000 7f459d000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45980b4700 con 0x7f45980cbab0 2024-03-23T03:50:21.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.345+0000 7f459d000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45980cbfb0 con 0x7f45980adbc0 2024-03-23T03:50:21.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.345+0000 7f4596800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f459806d390 0x7f45980b2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:44312/0 (socket says 172.21.15.102:44312) 2024-03-23T03:50:21.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.345+0000 7f4596800640 1 -- 172.21.15.102:0/2785959301 learned_addr learned my addr 172.21.15.102:0/2785959301 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:21.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.345+0000 7f4595e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f45980adbc0 0x7f45980b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:21.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.345+0000 7f4595e00640 1 -- 172.21.15.102:0/2785959301 >> v1:172.21.15.173:6789/0 conn(0x7f45980cbab0 legacy=0x7f45980b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:21.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.346+0000 7f4595e00640 1 -- 172.21.15.102:0/2785959301 >> v1:172.21.15.102:6789/0 conn(0x7f459806d390 legacy=0x7f45980b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:21.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.346+0000 7f4595e00640 1 -- 172.21.15.102:0/2785959301 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45980cc290 con 0x7f45980adbc0 2024-03-23T03:50:21.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.346+0000 7f4594a00640 1 -- 172.21.15.102:0/2785959301 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558611721 0 0) 0x7f45980b16a0 con 0x7f459806d390 2024-03-23T03:50:21.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.347+0000 7f4595e00640 1 --2- 172.21.15.102:0/2785959301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980adbc0 0x7f45980b41c0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f45880097a0 tx=0x7f458802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f172dc7b9563be server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:21.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.347+0000 7f4594a00640 1 -- 172.21.15.102:0/2785959301 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f458802e070 con 0x7f45980adbc0 2024-03-23T03:50:21.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.347+0000 7f4594a00640 1 -- 172.21.15.102:0/2785959301 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45880315a0 con 0x7f45980adbc0 2024-03-23T03:50:21.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.347+0000 7f4594a00640 1 -- 172.21.15.102:0/2785959301 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4588031930 con 0x7f45980adbc0 2024-03-23T03:50:21.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.347+0000 7f459d000640 1 -- 172.21.15.102:0/2785959301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980adbc0 msgr2=0x7f45980b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:21.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.347+0000 7f459d000640 1 --2- 172.21.15.102:0/2785959301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980adbc0 0x7f45980b41c0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f45880097a0 tx=0x7f458802b840 comp rx=0 tx=0).stop 2024-03-23T03:50:21.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.348+0000 7f459d000640 1 -- 172.21.15.102:0/2785959301 shutdown_connections 2024-03-23T03:50:21.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.348+0000 7f459d000640 1 --2- 172.21.15.102:0/2785959301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980adbc0 0x7f45980b41c0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:21.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.348+0000 7f459d000640 1 -- 172.21.15.102:0/2785959301 >> 172.21.15.102:0/2785959301 conn(0x7f459806a3d0 msgr2=0x7f45980b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:21.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.348+0000 7f459d000640 1 -- 172.21.15.102:0/2785959301 shutdown_connections 2024-03-23T03:50:21.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.348+0000 7f459d000640 1 -- 172.21.15.102:0/2785959301 wait complete. 2024-03-23T03:50:21.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.349+0000 7f459d000640 1 Processor -- start 2024-03-23T03:50:21.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.349+0000 7f459d000640 1 -- start start 2024-03-23T03:50:21.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.350+0000 7f459d000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f459806d390 0x7f45980b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:21.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.350+0000 7f459d000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980adbc0 0x7f459815aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:21.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.350+0000 7f459d000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45980cbab0 0x7f459815af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:21.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.350+0000 7f459d000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45981f6bf0 con 0x7f45980adbc0 2024-03-23T03:50:21.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.350+0000 7f459d000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45981f3640 con 0x7f459806d390 2024-03-23T03:50:21.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.350+0000 7f459d000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45981f5e30 con 0x7f45980cbab0 2024-03-23T03:50:21.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.351+0000 7f4595400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980adbc0 0x7f459815aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:21.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.351+0000 7f4595400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980adbc0 0x7f459815aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:38834/0 (socket says 172.21.15.102:38834) 2024-03-23T03:50:21.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.351+0000 7f4595400640 1 -- 172.21.15.102:0/1022349108 learned_addr learned my addr 172.21.15.102:0/1022349108 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:21.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.351+0000 7f4596800640 1 --2- 172.21.15.102:0/1022349108 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45980cbab0 0x7f459815af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:21.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.351+0000 7f4595e00640 1 --2- 172.21.15.102:0/1022349108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f459806d390 0x7f45980b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:21.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.351+0000 7f4595400640 1 -- 172.21.15.102:0/1022349108 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45980cbab0 msgr2=0x7f459815af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:21.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.351+0000 7f4595400640 1 --2- 172.21.15.102:0/1022349108 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45980cbab0 0x7f459815af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:21.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.351+0000 7f4595400640 1 -- 172.21.15.102:0/1022349108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f459806d390 msgr2=0x7f45980b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:21.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.351+0000 7f4595400640 1 --2- 172.21.15.102:0/1022349108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f459806d390 0x7f45980b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:21.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.351+0000 7f4595400640 1 -- 172.21.15.102:0/1022349108 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4588009400 con 0x7f45980adbc0 2024-03-23T03:50:21.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.351+0000 7f4595400640 1 --2- 172.21.15.102:0/1022349108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980adbc0 0x7f459815aa50 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f458400c400 tx=0x7f458400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:21.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.352+0000 7f458e000640 1 -- 172.21.15.102:0/1022349108 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4584011790 con 0x7f45980adbc0 2024-03-23T03:50:21.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.352+0000 7f458e000640 1 -- 172.21.15.102:0/1022349108 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4584011930 con 0x7f45980adbc0 2024-03-23T03:50:21.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.352+0000 7f459d000640 1 -- 172.21.15.102:0/1022349108 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45981f6110 con 0x7f45980adbc0 2024-03-23T03:50:21.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.352+0000 7f458e000640 1 -- 172.21.15.102:0/1022349108 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4584011ce0 con 0x7f45980adbc0 2024-03-23T03:50:21.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.352+0000 7f459d000640 1 -- 172.21.15.102:0/1022349108 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45981f66d0 con 0x7f45980adbc0 2024-03-23T03:50:21.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.353+0000 7f459d000640 1 -- 172.21.15.102:0/1022349108 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f459806ac20 con 0x7f45980adbc0 2024-03-23T03:50:21.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.354+0000 7f458e000640 1 -- 172.21.15.102:0/1022349108 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f458401c460 con 0x7f45980adbc0 2024-03-23T03:50:21.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.355+0000 7f458e000640 1 --2- 172.21.15.102:0/1022349108 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f457806ac30 0x7f457806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:21.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.355+0000 7f458e000640 1 -- 172.21.15.102:0/1022349108 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f45840891e0 con 0x7f45980adbc0 2024-03-23T03:50:21.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.355+0000 7f4595e00640 1 --2- 172.21.15.102:0/1022349108 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f457806ac30 0x7f457806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:21.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.356+0000 7f4595e00640 1 --2- 172.21.15.102:0/1022349108 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f457806ac30 0x7f457806d0f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4588036000 tx=0x7f458803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:21.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.359+0000 7f458e000640 1 -- 172.21.15.102:0/1022349108 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4584054320 con 0x7f45980adbc0 2024-03-23T03:50:21.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.594+0000 7f459d000640 1 -- 172.21.15.102:0/1022349108 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4598073390 con 0x7f45980adbc0 2024-03-23T03:50:21.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.594+0000 7f458e000640 1 -- 172.21.15.102:0/1022349108 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v41) v1 ==== 53+0+73 (secure 0 0 0) 0x7f4584057940 con 0x7f45980adbc0 2024-03-23T03:50:21.595 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T03:50:21.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.597+0000 7f459d000640 1 -- 172.21.15.102:0/1022349108 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f457806ac30 msgr2=0x7f457806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:21.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.597+0000 7f459d000640 1 --2- 172.21.15.102:0/1022349108 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f457806ac30 0x7f457806d0f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4588036000 tx=0x7f458803d040 comp rx=0 tx=0).stop 2024-03-23T03:50:21.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.597+0000 7f459d000640 1 -- 172.21.15.102:0/1022349108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980adbc0 msgr2=0x7f459815aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:21.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.597+0000 7f459d000640 1 --2- 172.21.15.102:0/1022349108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980adbc0 0x7f459815aa50 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f458400c400 tx=0x7f458400c8c0 comp rx=0 tx=0).stop 2024-03-23T03:50:21.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.597+0000 7f459d000640 1 -- 172.21.15.102:0/1022349108 shutdown_connections 2024-03-23T03:50:21.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.597+0000 7f459d000640 1 --2- 172.21.15.102:0/1022349108 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f457806ac30 0x7f457806d0f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:21.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.597+0000 7f459d000640 1 --2- 172.21.15.102:0/1022349108 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45980cbab0 0x7f459815af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:21.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.597+0000 7f459d000640 1 --2- 172.21.15.102:0/1022349108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45980adbc0 0x7f459815aa50 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:21.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.597+0000 7f459d000640 1 --2- 172.21.15.102:0/1022349108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f459806d390 0x7f45980b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:21.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.597+0000 7f459d000640 1 -- 172.21.15.102:0/1022349108 >> 172.21.15.102:0/1022349108 conn(0x7f459806a3d0 msgr2=0x7f45980cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:21.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.597+0000 7f459d000640 1 -- 172.21.15.102:0/1022349108 shutdown_connections 2024-03-23T03:50:21.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:21.597+0000 7f459d000640 1 -- 172.21.15.102:0/1022349108 wait complete. 2024-03-23T03:50:21.614 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-23T03:50:21.614 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:50:21.614 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi173.front.sepia.ceph.com 2024-03-23T03:50:21.614 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-23T03:50:21.615 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-23T03:50:21.615 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-23T03:50:21.615 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-23T03:50:21.615 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:50:21.615 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:50:21.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:21.615 DEBUG:teuthology.orchestra.run.smithi173:> ip addr 2024-03-23T03:50:21.666 INFO:teuthology.orchestra.run.smithi173.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:50:21.667 INFO:teuthology.orchestra.run.smithi173.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:50:21.667 INFO:teuthology.orchestra.run.smithi173.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:50:21.667 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:21.667 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 ::1/128 scope host 2024-03-23T03:50:21.667 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:21.667 INFO:teuthology.orchestra.run.smithi173.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:50:21.667 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6a brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:21.667 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f0 2024-03-23T03:50:21.667 INFO:teuthology.orchestra.run.smithi173.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:50:21.667 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6b brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:21.667 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f1 2024-03-23T03:50:21.667 INFO:teuthology.orchestra.run.smithi173.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:50:21.667 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:58 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:21.667 INFO:teuthology.orchestra.run.smithi173.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:50:21.668 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:59 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:21.668 INFO:teuthology.orchestra.run.smithi173.stdout: inet 172.21.15.173/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:50:21.668 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft 42420sec preferred_lft 42420sec 2024-03-23T03:50:21.668 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::ec4:7aff:fe88:7259/64 scope link 2024-03-23T03:50:21.668 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:21.668 INFO:teuthology.orchestra.run.smithi173.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:50:21.668 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 52:69:f2:bf:6d:02 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:21.668 INFO:teuthology.orchestra.run.smithi173.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T03:50:21.668 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:21.668 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::6894:e9ff:fe5d:1492/64 scope link 2024-03-23T03:50:21.668 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:21.668 INFO:teuthology.orchestra.run.smithi173.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:50:21.668 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether a2:27:ac:42:68:ce brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T03:50:21.668 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::a027:acff:fe42:68ce/64 scope link 2024-03-23T03:50:21.669 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:21.669 INFO:teuthology.orchestra.run.smithi173.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:50:21.669 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 52:69:f2:bf:6d:02 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-23T03:50:21.669 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::5069:f2ff:febf:6d02/64 scope link 2024-03-23T03:50:21.669 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:21.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:21.669 DEBUG:teuthology.orchestra.run.smithi173:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:50:21.793 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-03-23T03:50:21.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:21.798 DEBUG:teuthology.orchestra.run.smithi173:> route 2024-03-23T03:50:21.845 INFO:teuthology.orchestra.run.smithi173.stdout:Kernel IP routing table 2024-03-23T03:50:21.846 INFO:teuthology.orchestra.run.smithi173.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:50:21.846 INFO:teuthology.orchestra.run.smithi173.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:50:21.846 INFO:teuthology.orchestra.run.smithi173.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:50:21.846 INFO:teuthology.orchestra.run.smithi173.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:50:21.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:21.846 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-03-23T03:50:21.846 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-03-23T03:50:21.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:50:21.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:50:21.847 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:50:21.847 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-03-23T03:50:22.046 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:22.046 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list 2024-03-23T03:50:22.091 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-23T03:50:22.091 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-23T03:50:22.091 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:50:22.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:22.092 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link set brx.0 up 2024-03-23T03:50:22.220 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:50:22.220 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-23T03:50:22.268 INFO:teuthology.orchestra.run.smithi173.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:50:22.268 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:50:22.269 DEBUG:teuthology.orchestra.run.smithi173:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-23T03:50:22.357 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-23T03:50:22.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:22.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:50:22.514 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:50:22.514 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:50:22.514 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:50:22.533 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:50:22.533 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:50:22.533 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:50:22.533 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:50:22.533 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:50:22.559 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:50:22.559 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:50:22.559 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:50:22.559 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:50:22.560 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:50:22.560 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790" 2024-03-23T03:50:22.560 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/ 2024-03-23T03:50:22.560 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=776b2820-05f2-4e50-8d40-18dd76c5f6a0" will pass to kernel 2024-03-23T03:50:22.561 INFO:teuthology.orchestra.run.smithi173.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-23T03:50:22.561 INFO:teuthology.orchestra.run.smithi173.stdout: You just mounted a file system that supports labels which does not 2024-03-23T03:50:22.561 INFO:teuthology.orchestra.run.smithi173.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T03:50:22.561 INFO:teuthology.orchestra.run.smithi173.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T03:50:22.561 INFO:teuthology.orchestra.run.smithi173.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T03:50:22.565 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T03:50:22.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:22.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-23T03:50:22.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:22.641 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:50:22.688 INFO:teuthology.orchestra.run.smithi173.stdout:client6564 2024-03-23T03:50:22.688 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:50:22.688 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:22.733 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:50:22.733 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:22.826 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:50:22.826 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:22.876 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:50:23.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.091+0000 7faf81800640 1 Processor -- start 2024-03-23T03:50:23.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.091+0000 7faf81800640 1 -- start start 2024-03-23T03:50:23.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.093+0000 7faf81800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faf7c15b560 0x7faf7c15b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.093+0000 7faf81800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faf7c15d410 0x7faf7c0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.093+0000 7faf81800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7c06c6f0 con 0x7faf7c0b39b0 2024-03-23T03:50:23.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.093+0000 7faf81800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7c0b4370 con 0x7faf7c15d410 2024-03-23T03:50:23.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.093+0000 7faf81800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c06c9d0 con 0x7faf7c15b560 2024-03-23T03:50:23.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.093+0000 7faf7b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faf7c15d410 0x7faf7c0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:23.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.093+0000 7faf7b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faf7c15d410 0x7faf7c0b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42818/0 (socket says 172.21.15.102:42818) 2024-03-23T03:50:23.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.093+0000 7faf7b400640 1 -- 172.21.15.102:0/841128812 learned_addr learned my addr 172.21.15.102:0/841128812 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:23.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.093+0000 7faf73e00640 1 --1- 172.21.15.102:0/841128812 >> v1:172.21.15.102:6789/0 conn(0x7faf7c0b39b0 0x7faf7c15e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42342/0 (socket says 172.21.15.102:42342) 2024-03-23T03:50:23.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.094+0000 7faf7aa00640 1 --2- 172.21.15.102:0/841128812 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faf7c15b560 0x7faf7c15b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:23.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.094+0000 7faf7b400640 1 -- 172.21.15.102:0/841128812 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faf7c15b560 msgr2=0x7faf7c15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:23.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.094+0000 7faf7b400640 1 --2- 172.21.15.102:0/841128812 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faf7c15b560 0x7faf7c15b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.094+0000 7faf7b400640 1 -- 172.21.15.102:0/841128812 >> v1:172.21.15.102:6789/0 conn(0x7faf7c0b39b0 legacy=0x7faf7c15e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:23.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.094+0000 7faf7b400640 1 -- 172.21.15.102:0/841128812 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c06ccb0 con 0x7faf7c15d410 2024-03-23T03:50:23.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.094+0000 7faf7a000640 1 -- 172.21.15.102:0/841128812 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000498191 0 0) 0x7faf7c06c6f0 con 0x7faf7c0b39b0 2024-03-23T03:50:23.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.095+0000 7faf7b400640 1 --2- 172.21.15.102:0/841128812 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c15d410 0x7faf7c0b3e30 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7faf74009e50 tx=0x7faf7402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdb83e091f04a1a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:23.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.095+0000 7faf7a000640 1 -- 172.21.15.102:0/841128812 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf7402cb00 con 0x7faf7c15d410 2024-03-23T03:50:23.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.095+0000 7faf7a000640 1 -- 172.21.15.102:0/841128812 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf7402cc60 con 0x7faf7c15d410 2024-03-23T03:50:23.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.095+0000 7faf7a000640 1 -- 172.21.15.102:0/841128812 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf740343f0 con 0x7faf7c15d410 2024-03-23T03:50:23.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.096+0000 7faf81800640 1 -- 172.21.15.102:0/841128812 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c15d410 msgr2=0x7faf7c0b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:23.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.096+0000 7faf81800640 1 --2- 172.21.15.102:0/841128812 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c15d410 0x7faf7c0b3e30 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7faf74009e50 tx=0x7faf7402bf60 comp rx=0 tx=0).stop 2024-03-23T03:50:23.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.096+0000 7faf81800640 1 -- 172.21.15.102:0/841128812 shutdown_connections 2024-03-23T03:50:23.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.096+0000 7faf81800640 1 --2- 172.21.15.102:0/841128812 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c15d410 0x7faf7c0b3e30 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.096+0000 7faf81800640 1 --2- 172.21.15.102:0/841128812 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faf7c15b560 0x7faf7c15b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.096+0000 7faf81800640 1 -- 172.21.15.102:0/841128812 >> 172.21.15.102:0/841128812 conn(0x7faf7c06a3d0 msgr2=0x7faf7c0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:23.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.096+0000 7faf81800640 1 -- 172.21.15.102:0/841128812 shutdown_connections 2024-03-23T03:50:23.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.096+0000 7faf81800640 1 -- 172.21.15.102:0/841128812 wait complete. 2024-03-23T03:50:23.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.098+0000 7faf81800640 1 Processor -- start 2024-03-23T03:50:23.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.098+0000 7faf81800640 1 -- start start 2024-03-23T03:50:23.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.099+0000 7faf81800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faf7c0b39b0 0x7faf7c1432b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.099+0000 7faf81800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c15b560 0x7faf7c0b0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.099+0000 7faf81800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf7c15d410 0x7faf7c1e72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.099+0000 7faf81800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c1437f0 con 0x7faf7c15b560 2024-03-23T03:50:23.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.099+0000 7faf81800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c153220 con 0x7faf7c15d410 2024-03-23T03:50:23.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.099+0000 7faf81800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c1e5e30 con 0x7faf7c0b39b0 2024-03-23T03:50:23.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.099+0000 7faf73e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c15b560 0x7faf7c0b0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:23.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.099+0000 7faf73e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c15b560 0x7faf7c0b0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42828/0 (socket says 172.21.15.102:42828) 2024-03-23T03:50:23.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.099+0000 7faf73e00640 1 -- 172.21.15.102:0/2621559819 learned_addr learned my addr 172.21.15.102:0/2621559819 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:23.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.100+0000 7faf73e00640 1 -- 172.21.15.102:0/2621559819 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faf7c0b39b0 msgr2=0x7faf7c1432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:23.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.100+0000 7faf73e00640 1 --2- 172.21.15.102:0/2621559819 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faf7c0b39b0 0x7faf7c1432b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.100+0000 7faf73e00640 1 -- 172.21.15.102:0/2621559819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf7c15d410 msgr2=0x7faf7c1e72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:23.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.100+0000 7faf73e00640 1 --2- 172.21.15.102:0/2621559819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf7c15d410 0x7faf7c1e72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.100+0000 7faf73e00640 1 -- 172.21.15.102:0/2621559819 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf74009b30 con 0x7faf7c15b560 2024-03-23T03:50:23.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.100+0000 7faf73e00640 1 --2- 172.21.15.102:0/2621559819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c15b560 0x7faf7c0b0320 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7faf6400c430 tx=0x7faf6400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:23.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.100+0000 7faf72a00640 1 -- 172.21.15.102:0/2621559819 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf640074e0 con 0x7faf7c15b560 2024-03-23T03:50:23.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.100+0000 7faf72a00640 1 -- 172.21.15.102:0/2621559819 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf64007660 con 0x7faf7c15b560 2024-03-23T03:50:23.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.100+0000 7faf81800640 1 -- 172.21.15.102:0/2621559819 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7c1e52b0 con 0x7faf7c15b560 2024-03-23T03:50:23.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.100+0000 7faf72a00640 1 -- 172.21.15.102:0/2621559819 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf64004a20 con 0x7faf7c15b560 2024-03-23T03:50:23.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.100+0000 7faf81800640 1 -- 172.21.15.102:0/2621559819 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7c1e5870 con 0x7faf7c15b560 2024-03-23T03:50:23.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.101+0000 7faf81800640 1 -- 172.21.15.102:0/2621559819 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7c152ff0 con 0x7faf7c15b560 2024-03-23T03:50:23.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.102+0000 7faf72a00640 1 -- 172.21.15.102:0/2621559819 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7faf64004b80 con 0x7faf7c15b560 2024-03-23T03:50:23.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.103+0000 7faf72a00640 1 --2- 172.21.15.102:0/2621559819 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7faf5806ac00 0x7faf5806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.103+0000 7faf72a00640 1 -- 172.21.15.102:0/2621559819 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7faf64089350 con 0x7faf7c15b560 2024-03-23T03:50:23.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.104+0000 7faf7aa00640 1 --2- 172.21.15.102:0/2621559819 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7faf5806ac00 0x7faf5806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:23.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.104+0000 7faf7aa00640 1 --2- 172.21.15.102:0/2621559819 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7faf5806ac00 0x7faf5806d0c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7faf7c154fc0 tx=0x7faf6c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:23.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.108+0000 7faf72a00640 1 -- 172.21.15.102:0/2621559819 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7faf7c152ff0 con 0x7faf7c15b560 2024-03-23T03:50:23.297 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.296+0000 7faf81800640 1 -- 172.21.15.102:0/2621559819 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf7c152ff0 con 0x7faf7c15b560 2024-03-23T03:50:23.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.299+0000 7faf72a00640 1 -- 172.21.15.102:0/2621559819 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16357 (secure 0 0 0) 0x7faf7c152ff0 con 0x7faf7c15b560 2024-03-23T03:50:23.300 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:23.301 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":81,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:08.942399+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:50:06.347556+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-23T03:50:06.924370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:50:23.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.302+0000 7faf81800640 1 -- 172.21.15.102:0/2621559819 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7faf5806ac00 msgr2=0x7faf5806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:23.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.302+0000 7faf81800640 1 --2- 172.21.15.102:0/2621559819 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7faf5806ac00 0x7faf5806d0c0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7faf7c154fc0 tx=0x7faf6c008040 comp rx=0 tx=0).stop 2024-03-23T03:50:23.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.302+0000 7faf81800640 1 -- 172.21.15.102:0/2621559819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c15b560 msgr2=0x7faf7c0b0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:23.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.302+0000 7faf81800640 1 --2- 172.21.15.102:0/2621559819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c15b560 0x7faf7c0b0320 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7faf6400c430 tx=0x7faf6400c8f0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.302+0000 7faf81800640 1 -- 172.21.15.102:0/2621559819 shutdown_connections 2024-03-23T03:50:23.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.302+0000 7faf81800640 1 --2- 172.21.15.102:0/2621559819 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7faf5806ac00 0x7faf5806d0c0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.302+0000 7faf81800640 1 --2- 172.21.15.102:0/2621559819 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faf7c15d410 0x7faf7c1e72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.302+0000 7faf81800640 1 --2- 172.21.15.102:0/2621559819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf7c15b560 0x7faf7c0b0320 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.302+0000 7faf81800640 1 --2- 172.21.15.102:0/2621559819 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faf7c0b39b0 0x7faf7c1432b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.302+0000 7faf81800640 1 -- 172.21.15.102:0/2621559819 >> 172.21.15.102:0/2621559819 conn(0x7faf7c06a3d0 msgr2=0x7faf7c15c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:23.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.302+0000 7faf81800640 1 -- 172.21.15.102:0/2621559819 shutdown_connections 2024-03-23T03:50:23.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.302+0000 7faf81800640 1 -- 172.21.15.102:0/2621559819 wait complete. 2024-03-23T03:50:23.319 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:50:06.347556+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:50:06.924370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:50:23.319 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:50:23.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.514+0000 7ffbac200640 1 Processor -- start 2024-03-23T03:50:23.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.514+0000 7ffbac200640 1 -- start start 2024-03-23T03:50:23.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.515+0000 7ffbac200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ffba4159830 0x7ffba40cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.516+0000 7ffbac200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffba40ce930 0x7ffba40cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.516+0000 7ffbac200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba4159c10 con 0x7ffba40ca7b0 2024-03-23T03:50:23.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.516+0000 7ffbac200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffba406de50 con 0x7ffba4159830 2024-03-23T03:50:23.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.516+0000 7ffbac200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba40cf300 con 0x7ffba40ce930 2024-03-23T03:50:23.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.516+0000 7ffba9800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ffba4159830 0x7ffba40cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:23.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.516+0000 7ffba8e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ffba40ca7b0 0x7ffba40cdcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42354/0 (socket says 172.21.15.102:42354) 2024-03-23T03:50:23.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.516+0000 7ffba9800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ffba4159830 0x7ffba40cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42838/0 (socket says 172.21.15.102:42838) 2024-03-23T03:50:23.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.516+0000 7ffba8e00640 1 -- 172.21.15.102:0/1246859498 learned_addr learned my addr 172.21.15.102:0/1246859498 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:23.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.516+0000 7ffba9800640 1 -- 172.21.15.102:0/1246859498 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffba40ce930 msgr2=0x7ffba40cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:23.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.516+0000 7ffbaa200640 1 --2- 172.21.15.102:0/1246859498 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffba40ce930 0x7ffba40cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:23.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.516+0000 7ffba3e00640 1 -- 172.21.15.102:0/1246859498 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940906219 0 0) 0x7ffba4159c10 con 0x7ffba40ca7b0 2024-03-23T03:50:23.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.516+0000 7ffba9800640 1 --2- 172.21.15.102:0/1246859498 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffba40ce930 0x7ffba40cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.517+0000 7ffba9800640 1 -- 172.21.15.102:0/1246859498 >> v1:172.21.15.102:6789/0 conn(0x7ffba40ca7b0 legacy=0x7ffba40cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:23.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.517+0000 7ffba9800640 1 -- 172.21.15.102:0/1246859498 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba40cd840 con 0x7ffba4159830 2024-03-23T03:50:23.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.517+0000 7ffbaa200640 1 --2- 172.21.15.102:0/1246859498 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffba40ce930 0x7ffba40cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:23.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.518+0000 7ffba9800640 1 --2- 172.21.15.102:0/1246859498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffba4159830 0x7ffba40cc3c0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ffb9800ba40 tx=0x7ffb9802e110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5284ed62dfe7a2dc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:23.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.518+0000 7ffba3e00640 1 -- 172.21.15.102:0/1246859498 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb98036440 con 0x7ffba4159830 2024-03-23T03:50:23.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.518+0000 7ffba3e00640 1 -- 172.21.15.102:0/1246859498 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb980365a0 con 0x7ffba4159830 2024-03-23T03:50:23.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.518+0000 7ffba3e00640 1 -- 172.21.15.102:0/1246859498 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb98036930 con 0x7ffba4159830 2024-03-23T03:50:23.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.519+0000 7ffbac200640 1 -- 172.21.15.102:0/1246859498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffba4159830 msgr2=0x7ffba40cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:23.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.519+0000 7ffbac200640 1 --2- 172.21.15.102:0/1246859498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffba4159830 0x7ffba40cc3c0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ffb9800ba40 tx=0x7ffb9802e110 comp rx=0 tx=0).stop 2024-03-23T03:50:23.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.519+0000 7ffbac200640 1 -- 172.21.15.102:0/1246859498 shutdown_connections 2024-03-23T03:50:23.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.519+0000 7ffbac200640 1 --2- 172.21.15.102:0/1246859498 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffba40ce930 0x7ffba40cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.519+0000 7ffbac200640 1 --2- 172.21.15.102:0/1246859498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffba4159830 0x7ffba40cc3c0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.519+0000 7ffbac200640 1 -- 172.21.15.102:0/1246859498 >> 172.21.15.102:0/1246859498 conn(0x7ffba406a0b0 msgr2=0x7ffba406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:23.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.519+0000 7ffbac200640 1 -- 172.21.15.102:0/1246859498 shutdown_connections 2024-03-23T03:50:23.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.519+0000 7ffbac200640 1 -- 172.21.15.102:0/1246859498 wait complete. 2024-03-23T03:50:23.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.521+0000 7ffbac200640 1 Processor -- start 2024-03-23T03:50:23.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.521+0000 7ffbac200640 1 -- start start 2024-03-23T03:50:23.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.522+0000 7ffbac200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffba40ca7b0 0x7ffba4149320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.522+0000 7ffbac200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffba40ce930 0x7ffba40b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.522+0000 7ffbac200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffba4159830 0x7ffba40b3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.522+0000 7ffbac200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba4159dc0 con 0x7ffba40ce930 2024-03-23T03:50:23.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.522+0000 7ffbac200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba415a010 con 0x7ffba4159830 2024-03-23T03:50:23.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.522+0000 7ffbac200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffba415a2f0 con 0x7ffba40ca7b0 2024-03-23T03:50:23.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.522+0000 7ffba8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffba40ce930 0x7ffba40b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:23.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.522+0000 7ffba8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffba40ce930 0x7ffba40b37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42854/0 (socket says 172.21.15.102:42854) 2024-03-23T03:50:23.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.522+0000 7ffba8e00640 1 -- 172.21.15.102:0/3020583605 learned_addr learned my addr 172.21.15.102:0/3020583605 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:23.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.522+0000 7ffba8e00640 1 -- 172.21.15.102:0/3020583605 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffba40ca7b0 msgr2=0x7ffba4149320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:23.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.522+0000 7ffba8e00640 1 --2- 172.21.15.102:0/3020583605 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffba40ca7b0 0x7ffba4149320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.522+0000 7ffba8e00640 1 -- 172.21.15.102:0/3020583605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffba4159830 msgr2=0x7ffba40b3cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:23.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.522+0000 7ffba8e00640 1 --2- 172.21.15.102:0/3020583605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffba4159830 0x7ffba40b3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.522+0000 7ffba8e00640 1 -- 172.21.15.102:0/3020583605 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9800b6a0 con 0x7ffba40ce930 2024-03-23T03:50:23.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.523+0000 7ffba8e00640 1 --2- 172.21.15.102:0/3020583605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffba40ce930 0x7ffba40b37b0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7ffb9000c430 tx=0x7ffb9000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:23.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.523+0000 7ffba1600640 1 -- 172.21.15.102:0/3020583605 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb900074e0 con 0x7ffba40ce930 2024-03-23T03:50:23.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.523+0000 7ffba1600640 1 -- 172.21.15.102:0/3020583605 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb90007660 con 0x7ffba40ce930 2024-03-23T03:50:23.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.523+0000 7ffba1600640 1 -- 172.21.15.102:0/3020583605 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb90004a20 con 0x7ffba40ce930 2024-03-23T03:50:23.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.523+0000 7ffbac200640 1 -- 172.21.15.102:0/3020583605 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba415a5d0 con 0x7ffba40ce930 2024-03-23T03:50:23.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.523+0000 7ffbac200640 1 -- 172.21.15.102:0/3020583605 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba415ab90 con 0x7ffba40ce930 2024-03-23T03:50:23.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.524+0000 7ffbac200640 1 -- 172.21.15.102:0/3020583605 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba415adb0 con 0x7ffba40ce930 2024-03-23T03:50:23.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.525+0000 7ffba1600640 1 -- 172.21.15.102:0/3020583605 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ffb90004b80 con 0x7ffba40ce930 2024-03-23T03:50:23.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.527+0000 7ffba1600640 1 --2- 172.21.15.102:0/3020583605 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb8406ac00 0x7ffb8406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.527+0000 7ffba1600640 1 -- 172.21.15.102:0/3020583605 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ffb900892f0 con 0x7ffba40ce930 2024-03-23T03:50:23.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.527+0000 7ffba9800640 1 --2- 172.21.15.102:0/3020583605 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb8406ac00 0x7ffb8406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:23.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.527+0000 7ffba9800640 1 --2- 172.21.15.102:0/3020583605 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb8406ac00 0x7ffb8406d0c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ffb98038000 tx=0x7ffb98030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:23.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.532+0000 7ffba1600640 1 -- 172.21.15.102:0/3020583605 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffb900543c0 con 0x7ffba40ce930 2024-03-23T03:50:23.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.724+0000 7ffbac200640 1 -- 172.21.15.102:0/3020583605 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ffba4073390 con 0x7ffba40ce930 2024-03-23T03:50:23.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.725+0000 7ffba1600640 1 -- 172.21.15.102:0/3020583605 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7ffb900579e0 con 0x7ffba40ce930 2024-03-23T03:50:23.725 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:50:23.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.728+0000 7ffbac200640 1 -- 172.21.15.102:0/3020583605 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb8406ac00 msgr2=0x7ffb8406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:23.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.728+0000 7ffbac200640 1 --2- 172.21.15.102:0/3020583605 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb8406ac00 0x7ffb8406d0c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ffb98038000 tx=0x7ffb98030000 comp rx=0 tx=0).stop 2024-03-23T03:50:23.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.728+0000 7ffbac200640 1 -- 172.21.15.102:0/3020583605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffba40ce930 msgr2=0x7ffba40b37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:23.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.728+0000 7ffbac200640 1 --2- 172.21.15.102:0/3020583605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffba40ce930 0x7ffba40b37b0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7ffb9000c430 tx=0x7ffb9000c8f0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.728+0000 7ffbac200640 1 -- 172.21.15.102:0/3020583605 shutdown_connections 2024-03-23T03:50:23.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.728+0000 7ffbac200640 1 --2- 172.21.15.102:0/3020583605 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb8406ac00 0x7ffb8406d0c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.728+0000 7ffbac200640 1 --2- 172.21.15.102:0/3020583605 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffba4159830 0x7ffba40b3cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.728+0000 7ffbac200640 1 --2- 172.21.15.102:0/3020583605 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffba40ce930 0x7ffba40b37b0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.728+0000 7ffbac200640 1 --2- 172.21.15.102:0/3020583605 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffba40ca7b0 0x7ffba4149320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.728+0000 7ffbac200640 1 -- 172.21.15.102:0/3020583605 >> 172.21.15.102:0/3020583605 conn(0x7ffba406a0b0 msgr2=0x7ffba40ab8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:23.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.728+0000 7ffbac200640 1 -- 172.21.15.102:0/3020583605 shutdown_connections 2024-03-23T03:50:23.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.728+0000 7ffbac200640 1 -- 172.21.15.102:0/3020583605 wait complete. 2024-03-23T03:50:23.747 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:50:23.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.961+0000 7f960b200640 1 Processor -- start 2024-03-23T03:50:23.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.961+0000 7f960b200640 1 -- start start 2024-03-23T03:50:23.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.962+0000 7f960b200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f960c160bd0 0x7f960c0cad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.962+0000 7f960b200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f960c0cce70 0x7f960c0cb250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.962+0000 7f960b200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f960c165890 con 0x7f960c0cd650 2024-03-23T03:50:23.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.962+0000 7f960b200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f960c0c9a90 con 0x7f960c0cce70 2024-03-23T03:50:23.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.962+0000 7f960b200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f960c1621d0 con 0x7f960c160bd0 2024-03-23T03:50:23.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.963+0000 7f9609400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f960c0cce70 0x7f960c0cb250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:23.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.963+0000 7f9609400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f960c0cce70 0x7f960c0cb250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42870/0 (socket says 172.21.15.102:42870) 2024-03-23T03:50:23.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.963+0000 7f9609400640 1 -- 172.21.15.102:0/1308064603 learned_addr learned my addr 172.21.15.102:0/1308064603 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:23.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.963+0000 7f9609e00640 1 --2- 172.21.15.102:0/1308064603 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f960c160bd0 0x7f960c0cad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:23.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.963+0000 7f9609400640 1 -- 172.21.15.102:0/1308064603 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f960c160bd0 msgr2=0x7f960c0cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.963+0000 7f9609400640 1 --2- 172.21.15.102:0/1308064603 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f960c160bd0 0x7f960c0cad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.963+0000 7f9609400640 1 -- 172.21.15.102:0/1308064603 >> v1:172.21.15.173:6789/0 conn(0x7f960c0cd650 legacy=0x7f960c069c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.963+0000 7f9609400640 1 -- 172.21.15.102:0/1308064603 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960c0cd370 con 0x7f960c0cce70 2024-03-23T03:50:23.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.964+0000 7f9609400640 1 --2- 172.21.15.102:0/1308064603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c0cce70 0x7f960c0cb250 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009870 tx=0x7f95f402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a9f0cba55919bd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:23.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.965+0000 7f9608a00640 1 -- 172.21.15.102:0/1308064603 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f4031640 con 0x7f960c0cce70 2024-03-23T03:50:23.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.965+0000 7f9608a00640 1 -- 172.21.15.102:0/1308064603 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f40317a0 con 0x7f960c0cce70 2024-03-23T03:50:23.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.965+0000 7f9608a00640 1 -- 172.21.15.102:0/1308064603 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f4031b30 con 0x7f960c0cce70 2024-03-23T03:50:23.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.965+0000 7f960b200640 1 -- 172.21.15.102:0/1308064603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c0cce70 msgr2=0x7f960c0cb250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:23.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.965+0000 7f960b200640 1 --2- 172.21.15.102:0/1308064603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c0cce70 0x7f960c0cb250 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009870 tx=0x7f95f402b8f0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.965+0000 7f960b200640 1 -- 172.21.15.102:0/1308064603 shutdown_connections 2024-03-23T03:50:23.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.965+0000 7f960b200640 1 --2- 172.21.15.102:0/1308064603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c0cce70 0x7f960c0cb250 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.965+0000 7f960b200640 1 --2- 172.21.15.102:0/1308064603 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f960c160bd0 0x7f960c0cad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.965+0000 7f960b200640 1 -- 172.21.15.102:0/1308064603 >> 172.21.15.102:0/1308064603 conn(0x7f960c0b01b0 msgr2=0x7f960c0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:23.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.965+0000 7f960b200640 1 -- 172.21.15.102:0/1308064603 shutdown_connections 2024-03-23T03:50:23.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.965+0000 7f960b200640 1 -- 172.21.15.102:0/1308064603 wait complete. 2024-03-23T03:50:23.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.967+0000 7f960b200640 1 Processor -- start 2024-03-23T03:50:23.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.967+0000 7f960b200640 1 -- start start 2024-03-23T03:50:23.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.968+0000 7f960b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f960c0cce70 0x7f960c0af5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.968+0000 7f960b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c0cd650 0x7f960c0ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.968+0000 7f960b200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f960c160bd0 0x7f960c0ac0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.968+0000 7f960b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960c0ae510 con 0x7f960c0cd650 2024-03-23T03:50:23.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.968+0000 7f960b200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960c0ae7f0 con 0x7f960c0cce70 2024-03-23T03:50:23.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.968+0000 7f960b200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f960c0aead0 con 0x7f960c160bd0 2024-03-23T03:50:23.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.968+0000 7f9609400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c0cd650 0x7f960c0ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:23.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.968+0000 7f9609400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c0cd650 0x7f960c0ade10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42880/0 (socket says 172.21.15.102:42880) 2024-03-23T03:50:23.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.968+0000 7f9609400640 1 -- 172.21.15.102:0/1004278027 learned_addr learned my addr 172.21.15.102:0/1004278027 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:23.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.969+0000 7f9609e00640 1 --2- 172.21.15.102:0/1004278027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f960c0cce70 0x7f960c0af5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:23.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.969+0000 7f960a800640 1 --2- 172.21.15.102:0/1004278027 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f960c160bd0 0x7f960c0ac0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:23.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.969+0000 7f9609400640 1 -- 172.21.15.102:0/1004278027 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f960c160bd0 msgr2=0x7f960c0ac0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:23.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.969+0000 7f9609400640 1 --2- 172.21.15.102:0/1004278027 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f960c160bd0 0x7f960c0ac0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.969+0000 7f9609400640 1 -- 172.21.15.102:0/1004278027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f960c0cce70 msgr2=0x7f960c0af5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:23.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.969+0000 7f9609400640 1 --2- 172.21.15.102:0/1004278027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f960c0cce70 0x7f960c0af5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:23.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.969+0000 7f9609400640 1 -- 172.21.15.102:0/1004278027 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f40094d0 con 0x7f960c0cd650 2024-03-23T03:50:23.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.969+0000 7f9609e00640 1 --2- 172.21.15.102:0/1004278027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f960c0cce70 0x7f960c0af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:23.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.969+0000 7f9609400640 1 --2- 172.21.15.102:0/1004278027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c0cd650 0x7f960c0ade10 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009870 tx=0x7f95f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:23.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.970+0000 7f9602000640 1 -- 172.21.15.102:0/1004278027 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f403ac60 con 0x7f960c0cd650 2024-03-23T03:50:23.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.970+0000 7f9602000640 1 -- 172.21.15.102:0/1004278027 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f403ade0 con 0x7f960c0cd650 2024-03-23T03:50:23.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.970+0000 7f9602000640 1 -- 172.21.15.102:0/1004278027 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f4044630 con 0x7f960c0cd650 2024-03-23T03:50:23.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.970+0000 7f960b200640 1 -- 172.21.15.102:0/1004278027 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f960c0aedb0 con 0x7f960c0cd650 2024-03-23T03:50:23.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.970+0000 7f960b200640 1 -- 172.21.15.102:0/1004278027 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f960c0ac5e0 con 0x7f960c0cd650 2024-03-23T03:50:23.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.971+0000 7f960b200640 1 -- 172.21.15.102:0/1004278027 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f960c0af2d0 con 0x7f960c0cd650 2024-03-23T03:50:23.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.972+0000 7f9602000640 1 -- 172.21.15.102:0/1004278027 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f95f402f050 con 0x7f960c0cd650 2024-03-23T03:50:23.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.973+0000 7f9602000640 1 --2- 172.21.15.102:0/1004278027 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f95e806ac00 0x7f95e806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:23.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.974+0000 7f9602000640 1 -- 172.21.15.102:0/1004278027 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f95f40ab7c0 con 0x7f960c0cd650 2024-03-23T03:50:23.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.974+0000 7f9609e00640 1 --2- 172.21.15.102:0/1004278027 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f95e806ac00 0x7f95e806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:23.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.974+0000 7f9609e00640 1 --2- 172.21.15.102:0/1004278027 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f95e806ac00 0x7f95e806d0c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f95fc00b820 tx=0x7f95fc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:23.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:23.978+0000 7f9602000640 1 -- 172.21.15.102:0/1004278027 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f960c0af2d0 con 0x7f960c0cd650 2024-03-23T03:50:24.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.159+0000 7f960b200640 1 -- 172.21.15.102:0/1004278027 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f960c0af2d0 con 0x7f960c0cd650 2024-03-23T03:50:24.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.160+0000 7f9602000640 1 -- 172.21.15.102:0/1004278027 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7f960c0af2d0 con 0x7f960c0cd650 2024-03-23T03:50:24.160 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:50:24.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.162+0000 7f960b200640 1 -- 172.21.15.102:0/1004278027 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f95e806ac00 msgr2=0x7f95e806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:24.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.162+0000 7f960b200640 1 --2- 172.21.15.102:0/1004278027 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f95e806ac00 0x7f95e806d0c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f95fc00b820 tx=0x7f95fc008040 comp rx=0 tx=0).stop 2024-03-23T03:50:24.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.162+0000 7f960b200640 1 -- 172.21.15.102:0/1004278027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c0cd650 msgr2=0x7f960c0ade10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:24.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.162+0000 7f960b200640 1 --2- 172.21.15.102:0/1004278027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c0cd650 0x7f960c0ade10 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009870 tx=0x7f95f40046c0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.162+0000 7f960b200640 1 -- 172.21.15.102:0/1004278027 shutdown_connections 2024-03-23T03:50:24.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.162+0000 7f960b200640 1 --2- 172.21.15.102:0/1004278027 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f95e806ac00 0x7f95e806d0c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.162+0000 7f960b200640 1 --2- 172.21.15.102:0/1004278027 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f960c160bd0 0x7f960c0ac0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.162+0000 7f960b200640 1 --2- 172.21.15.102:0/1004278027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f960c0cd650 0x7f960c0ade10 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.162+0000 7f960b200640 1 --2- 172.21.15.102:0/1004278027 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f960c0cce70 0x7f960c0af5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.162+0000 7f960b200640 1 -- 172.21.15.102:0/1004278027 >> 172.21.15.102:0/1004278027 conn(0x7f960c0b01b0 msgr2=0x7f960c0c9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:24.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.162+0000 7f960b200640 1 -- 172.21.15.102:0/1004278027 shutdown_connections 2024-03-23T03:50:24.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.162+0000 7f960b200640 1 -- 172.21.15.102:0/1004278027 wait complete. 2024-03-23T03:50:24.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:50:24.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.394+0000 7fdddae00640 1 Processor -- start 2024-03-23T03:50:24.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.394+0000 7fdddae00640 1 -- start start 2024-03-23T03:50:24.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.396+0000 7fdddae00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fddd40ce930 0x7fddd40add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:24.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.396+0000 7fdddae00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fddd406ce90 0x7fddd406d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:24.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.396+0000 7fdddae00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd40cdba0 con 0x7fddd40cd3e0 2024-03-23T03:50:24.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.396+0000 7fdddae00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd406c760 con 0x7fddd40ce930 2024-03-23T03:50:24.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.396+0000 7fdddae00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd40b42d0 con 0x7fddd406ce90 2024-03-23T03:50:24.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.396+0000 7fddd3e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fddd40ce930 0x7fddd40add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:24.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.396+0000 7fddcb400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fddd40cd3e0 0x7fddd40ced10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42366/0 (socket says 172.21.15.102:42366) 2024-03-23T03:50:24.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.396+0000 7fddcb400640 1 -- 172.21.15.102:0/1464790788 learned_addr learned my addr 172.21.15.102:0/1464790788 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:24.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.396+0000 7fddd8e00640 1 --2- 172.21.15.102:0/1464790788 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fddd406ce90 0x7fddd406d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:24.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.396+0000 7fddd3e00640 1 -- 172.21.15.102:0/1464790788 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fddd406ce90 msgr2=0x7fddd406d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:24.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.396+0000 7fddd3e00640 1 --2- 172.21.15.102:0/1464790788 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fddd406ce90 0x7fddd406d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.396+0000 7fddd3e00640 1 -- 172.21.15.102:0/1464790788 >> v1:172.21.15.102:6789/0 conn(0x7fddd40cd3e0 legacy=0x7fddd40ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:24.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.397+0000 7fddd3e00640 1 -- 172.21.15.102:0/1464790788 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd40cde80 con 0x7fddd40ce930 2024-03-23T03:50:24.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.397+0000 7fddd8e00640 1 --2- 172.21.15.102:0/1464790788 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fddd406ce90 0x7fddd406d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:24.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.396+0000 7fddd3400640 1 -- 172.21.15.102:0/1464790788 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833548759 0 0) 0x7fddd40cdba0 con 0x7fddd40cd3e0 2024-03-23T03:50:24.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.398+0000 7fddd3e00640 1 --2- 172.21.15.102:0/1464790788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd40ce930 0x7fddd40add30 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fddc40097a0 tx=0x7fddc402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ad3b72a77f00bbb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:24.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.398+0000 7fddd3400640 1 -- 172.21.15.102:0/1464790788 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc402d040 con 0x7fddd40ce930 2024-03-23T03:50:24.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.398+0000 7fddd3400640 1 -- 172.21.15.102:0/1464790788 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc4031810 con 0x7fddd40ce930 2024-03-23T03:50:24.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.398+0000 7fddd3400640 1 -- 172.21.15.102:0/1464790788 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc4031ba0 con 0x7fddd40ce930 2024-03-23T03:50:24.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.398+0000 7fdddae00640 1 -- 172.21.15.102:0/1464790788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd40ce930 msgr2=0x7fddd40add30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:24.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.398+0000 7fdddae00640 1 --2- 172.21.15.102:0/1464790788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd40ce930 0x7fddd40add30 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fddc40097a0 tx=0x7fddc402b8b0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.399+0000 7fdddae00640 1 -- 172.21.15.102:0/1464790788 shutdown_connections 2024-03-23T03:50:24.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.399+0000 7fdddae00640 1 --2- 172.21.15.102:0/1464790788 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fddd406ce90 0x7fddd406d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.399+0000 7fdddae00640 1 --2- 172.21.15.102:0/1464790788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd40ce930 0x7fddd40add30 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.399+0000 7fdddae00640 1 -- 172.21.15.102:0/1464790788 >> 172.21.15.102:0/1464790788 conn(0x7fddd406a0b0 msgr2=0x7fddd406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:24.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.399+0000 7fdddae00640 1 -- 172.21.15.102:0/1464790788 shutdown_connections 2024-03-23T03:50:24.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.399+0000 7fdddae00640 1 -- 172.21.15.102:0/1464790788 wait complete. 2024-03-23T03:50:24.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.400+0000 7fdddae00640 1 Processor -- start 2024-03-23T03:50:24.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.401+0000 7fdddae00640 1 -- start start 2024-03-23T03:50:24.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.401+0000 7fdddae00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd406ce90 0x7fddd4150750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:24.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.402+0000 7fdddae00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fddd40cd3e0 0x7fddd40a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:24.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.402+0000 7fdddae00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fddd40ce930 0x7fddd40a99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:24.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.402+0000 7fdddae00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd4152380 con 0x7fddd406ce90 2024-03-23T03:50:24.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.402+0000 7fdddae00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd4151ef0 con 0x7fddd40ce930 2024-03-23T03:50:24.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.402+0000 7fdddae00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd4151000 con 0x7fddd40cd3e0 2024-03-23T03:50:24.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.402+0000 7fddd3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd406ce90 0x7fddd4150750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:24.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.402+0000 7fddd3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd406ce90 0x7fddd4150750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42886/0 (socket says 172.21.15.102:42886) 2024-03-23T03:50:24.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.402+0000 7fddd3e00640 1 -- 172.21.15.102:0/1153263153 learned_addr learned my addr 172.21.15.102:0/1153263153 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:24.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.402+0000 7fddcb400640 1 --2- 172.21.15.102:0/1153263153 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fddd40cd3e0 0x7fddd40a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:24.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.402+0000 7fddd3e00640 1 -- 172.21.15.102:0/1153263153 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fddd40cd3e0 msgr2=0x7fddd40a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:24.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.402+0000 7fddd3e00640 1 --2- 172.21.15.102:0/1153263153 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fddd40cd3e0 0x7fddd40a9470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.402+0000 7fddd3e00640 1 -- 172.21.15.102:0/1153263153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fddd40ce930 msgr2=0x7fddd40a99b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:24.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.402+0000 7fddd3e00640 1 --2- 172.21.15.102:0/1153263153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fddd40ce930 0x7fddd40a99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.402+0000 7fddd3e00640 1 -- 172.21.15.102:0/1153263153 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc4009400 con 0x7fddd406ce90 2024-03-23T03:50:24.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.403+0000 7fddd3e00640 1 --2- 172.21.15.102:0/1153263153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd406ce90 0x7fddd4150750 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fddc4000c00 tx=0x7fddc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:24.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.403+0000 7fddd0c00640 1 -- 172.21.15.102:0/1153263153 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc403bae0 con 0x7fddd406ce90 2024-03-23T03:50:24.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.403+0000 7fddd0c00640 1 -- 172.21.15.102:0/1153263153 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc403bc40 con 0x7fddd406ce90 2024-03-23T03:50:24.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.403+0000 7fddd0c00640 1 -- 172.21.15.102:0/1153263153 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc4004030 con 0x7fddd406ce90 2024-03-23T03:50:24.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.403+0000 7fdddae00640 1 -- 172.21.15.102:0/1153263153 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd41512e0 con 0x7fddd406ce90 2024-03-23T03:50:24.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.403+0000 7fdddae00640 1 -- 172.21.15.102:0/1153263153 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd41518a0 con 0x7fddd406ce90 2024-03-23T03:50:24.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.404+0000 7fdddae00640 1 -- 172.21.15.102:0/1153263153 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd4073390 con 0x7fddd406ce90 2024-03-23T03:50:24.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.405+0000 7fddd0c00640 1 -- 172.21.15.102:0/1153263153 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fddc4043070 con 0x7fddd406ce90 2024-03-23T03:50:24.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.406+0000 7fddd0c00640 1 --2- 172.21.15.102:0/1153263153 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fddb006ac00 0x7fddb006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:24.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.406+0000 7fddd0c00640 1 -- 172.21.15.102:0/1153263153 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fddc40ab4b0 con 0x7fddd406ce90 2024-03-23T03:50:24.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.406+0000 7fddcb400640 1 --2- 172.21.15.102:0/1153263153 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fddb006ac00 0x7fddb006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:24.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.407+0000 7fddcb400640 1 --2- 172.21.15.102:0/1153263153 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fddb006ac00 0x7fddb006d0c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fddbc007fd0 tx=0x7fddbc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:24.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.411+0000 7fddd0c00640 1 -- 172.21.15.102:0/1153263153 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fddc40764b0 con 0x7fddd406ce90 2024-03-23T03:50:24.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.602+0000 7fdddae00640 1 -- 172.21.15.102:0/1153263153 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fddd4152bc0 con 0x7fddd406ce90 2024-03-23T03:50:24.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.603+0000 7fddd0c00640 1 -- 172.21.15.102:0/1153263153 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7fddc4079ad0 con 0x7fddd406ce90 2024-03-23T03:50:24.604 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:50:24.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.606+0000 7fdddae00640 1 -- 172.21.15.102:0/1153263153 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fddb006ac00 msgr2=0x7fddb006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:24.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.606+0000 7fdddae00640 1 --2- 172.21.15.102:0/1153263153 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fddb006ac00 0x7fddb006d0c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fddbc007fd0 tx=0x7fddbc009040 comp rx=0 tx=0).stop 2024-03-23T03:50:24.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.606+0000 7fdddae00640 1 -- 172.21.15.102:0/1153263153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd406ce90 msgr2=0x7fddd4150750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:24.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.606+0000 7fdddae00640 1 --2- 172.21.15.102:0/1153263153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd406ce90 0x7fddd4150750 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fddc4000c00 tx=0x7fddc40046c0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.606+0000 7fdddae00640 1 -- 172.21.15.102:0/1153263153 shutdown_connections 2024-03-23T03:50:24.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.606+0000 7fdddae00640 1 --2- 172.21.15.102:0/1153263153 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fddb006ac00 0x7fddb006d0c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.606+0000 7fdddae00640 1 --2- 172.21.15.102:0/1153263153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fddd40ce930 0x7fddd40a99b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.606+0000 7fdddae00640 1 --2- 172.21.15.102:0/1153263153 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fddd40cd3e0 0x7fddd40a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.606+0000 7fdddae00640 1 --2- 172.21.15.102:0/1153263153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fddd406ce90 0x7fddd4150750 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.606+0000 7fdddae00640 1 -- 172.21.15.102:0/1153263153 >> 172.21.15.102:0/1153263153 conn(0x7fddd406a0b0 msgr2=0x7fddd40ced10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:24.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.606+0000 7fdddae00640 1 -- 172.21.15.102:0/1153263153 shutdown_connections 2024-03-23T03:50:24.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.606+0000 7fdddae00640 1 -- 172.21.15.102:0/1153263153 wait complete. 2024-03-23T03:50:24.625 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:50:24.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.832+0000 7f2218a00640 1 Processor -- start 2024-03-23T03:50:24.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.832+0000 7f2218a00640 1 -- start start 2024-03-23T03:50:24.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.833+0000 7f2218a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f221415cc70 0x7f22140cdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:24.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.833+0000 7f2218a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22140b4e60 0x7f22140d0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:24.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.833+0000 7f2218a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22141607d0 con 0x7f22140b1f20 2024-03-23T03:50:24.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.833+0000 7f2218a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2214160a70 con 0x7f221415cc70 2024-03-23T03:50:24.835 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.833+0000 7f2218a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2214160d50 con 0x7f22140b4e60 2024-03-23T03:50:24.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.833+0000 7f2211e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f221415cc70 0x7f22140cdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:24.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.833+0000 7f2211400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f22140b1f20 0x7f22140b4750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42372/0 (socket says 172.21.15.102:42372) 2024-03-23T03:50:24.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.833+0000 7f2211400640 1 -- 172.21.15.102:0/1258917691 learned_addr learned my addr 172.21.15.102:0/1258917691 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:24.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.834+0000 7f2212800640 1 --2- 172.21.15.102:0/1258917691 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22140b4e60 0x7f22140d0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:24.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.834+0000 7f2211e00640 1 -- 172.21.15.102:0/1258917691 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22140b4e60 msgr2=0x7f22140d0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:24.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.834+0000 7f2211e00640 1 --2- 172.21.15.102:0/1258917691 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22140b4e60 0x7f22140d0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.834+0000 7f2211e00640 1 -- 172.21.15.102:0/1258917691 >> v1:172.21.15.102:6789/0 conn(0x7f22140b1f20 legacy=0x7f22140b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:24.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.834+0000 7f2211e00640 1 -- 172.21.15.102:0/1258917691 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2214161030 con 0x7f221415cc70 2024-03-23T03:50:24.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.834+0000 7f2212800640 1 --2- 172.21.15.102:0/1258917691 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22140b4e60 0x7f22140d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:24.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.834+0000 7f2210a00640 1 -- 172.21.15.102:0/1258917691 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155563772 0 0) 0x7f22141607d0 con 0x7f22140b1f20 2024-03-23T03:50:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.835+0000 7f2211e00640 1 --2- 172.21.15.102:0/1258917691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f221415cc70 0x7f22140cdba0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f22040097a0 tx=0x7f220402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41fcdeac790beda6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.835+0000 7f2210a00640 1 -- 172.21.15.102:0/1258917691 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2204031830 con 0x7f221415cc70 2024-03-23T03:50:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.836+0000 7f2210a00640 1 -- 172.21.15.102:0/1258917691 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2204031990 con 0x7f221415cc70 2024-03-23T03:50:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.836+0000 7f2210a00640 1 -- 172.21.15.102:0/1258917691 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2204031d20 con 0x7f221415cc70 2024-03-23T03:50:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.836+0000 7f2218a00640 1 -- 172.21.15.102:0/1258917691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f221415cc70 msgr2=0x7f22140cdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.836+0000 7f2218a00640 1 --2- 172.21.15.102:0/1258917691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f221415cc70 0x7f22140cdba0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f22040097a0 tx=0x7f220402b8d0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.836+0000 7f2218a00640 1 -- 172.21.15.102:0/1258917691 shutdown_connections 2024-03-23T03:50:24.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.836+0000 7f2218a00640 1 --2- 172.21.15.102:0/1258917691 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22140b4e60 0x7f22140d0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.836+0000 7f2218a00640 1 --2- 172.21.15.102:0/1258917691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f221415cc70 0x7f22140cdba0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.836+0000 7f2218a00640 1 -- 172.21.15.102:0/1258917691 >> 172.21.15.102:0/1258917691 conn(0x7f22140cf790 msgr2=0x7f22140cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:24.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.836+0000 7f2218a00640 1 -- 172.21.15.102:0/1258917691 shutdown_connections 2024-03-23T03:50:24.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.836+0000 7f2218a00640 1 -- 172.21.15.102:0/1258917691 wait complete. 2024-03-23T03:50:24.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.838+0000 7f2218a00640 1 Processor -- start 2024-03-23T03:50:24.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.838+0000 7f2218a00640 1 -- start start 2024-03-23T03:50:24.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.839+0000 7f2218a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22140b1f20 0x7f221414cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:24.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.839+0000 7f2218a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22140b4e60 0x7f22140caf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:24.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.839+0000 7f2218a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f221415cc70 0x7f2214158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:24.841 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.839+0000 7f2218a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221414d2c0 con 0x7f22140b4e60 2024-03-23T03:50:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.839+0000 7f2218a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22140cb910 con 0x7f221415cc70 2024-03-23T03:50:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.839+0000 7f2218a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22140cbbf0 con 0x7f22140b1f20 2024-03-23T03:50:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.839+0000 7f2211400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22140b4e60 0x7f22140caf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.839+0000 7f2211400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22140b4e60 0x7f22140caf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42902/0 (socket says 172.21.15.102:42902) 2024-03-23T03:50:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.839+0000 7f2211400640 1 -- 172.21.15.102:0/1556140489 learned_addr learned my addr 172.21.15.102:0/1556140489 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.840+0000 7f2211e00640 1 --2- 172.21.15.102:0/1556140489 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22140b1f20 0x7f221414cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.840+0000 7f2212800640 1 --2- 172.21.15.102:0/1556140489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f221415cc70 0x7f2214158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.840+0000 7f2211400640 1 -- 172.21.15.102:0/1556140489 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22140b1f20 msgr2=0x7f221414cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.840+0000 7f2211400640 1 --2- 172.21.15.102:0/1556140489 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22140b1f20 0x7f221414cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.840+0000 7f2211400640 1 -- 172.21.15.102:0/1556140489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f221415cc70 msgr2=0x7f2214158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:24.842 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.840+0000 7f2211400640 1 --2- 172.21.15.102:0/1556140489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f221415cc70 0x7f2214158a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:24.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.840+0000 7f2211400640 1 -- 172.21.15.102:0/1556140489 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2204009400 con 0x7f22140b4e60 2024-03-23T03:50:24.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.840+0000 7f2211400640 1 --2- 172.21.15.102:0/1556140489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22140b4e60 0x7f22140caf30 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f21fc00c410 tx=0x7f21fc00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:24.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.840+0000 7f220a000640 1 -- 172.21.15.102:0/1556140489 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc0074e0 con 0x7f22140b4e60 2024-03-23T03:50:24.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.840+0000 7f220a000640 1 -- 172.21.15.102:0/1556140489 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21fc007660 con 0x7f22140b4e60 2024-03-23T03:50:24.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.840+0000 7f220a000640 1 -- 172.21.15.102:0/1556140489 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21fc004a20 con 0x7f22140b4e60 2024-03-23T03:50:24.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.840+0000 7f2218a00640 1 -- 172.21.15.102:0/1556140489 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22140cbed0 con 0x7f22140b4e60 2024-03-23T03:50:24.843 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.841+0000 7f2218a00640 1 -- 172.21.15.102:0/1556140489 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22140cc490 con 0x7f22140b4e60 2024-03-23T03:50:24.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.842+0000 7f2218a00640 1 -- 172.21.15.102:0/1556140489 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2214066bb0 con 0x7f22140b4e60 2024-03-23T03:50:24.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.843+0000 7f220a000640 1 -- 172.21.15.102:0/1556140489 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f21fc004b80 con 0x7f22140b4e60 2024-03-23T03:50:24.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.843+0000 7f220a000640 1 --2- 172.21.15.102:0/1556140489 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f21f006ac00 0x7f21f006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:24.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.844+0000 7f220a000640 1 -- 172.21.15.102:0/1556140489 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f21fc0892f0 con 0x7f22140b4e60 2024-03-23T03:50:24.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.844+0000 7f2211e00640 1 --2- 172.21.15.102:0/1556140489 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f21f006ac00 0x7f21f006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:24.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.845+0000 7f2211e00640 1 --2- 172.21.15.102:0/1556140489 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f21f006ac00 0x7f21f006d0c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2204036000 tx=0x7f2204008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:24.849 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:24.849+0000 7f220a000640 1 -- 172.21.15.102:0/1556140489 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f21fc057e10 con 0x7f22140b4e60 2024-03-23T03:50:25.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.057+0000 7f2218a00640 1 -- 172.21.15.102:0/1556140489 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f22140cd8d0 con 0x7f22140b4e60 2024-03-23T03:50:25.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.058+0000 7f220a000640 1 -- 172.21.15.102:0/1556140489 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v81) v1 ==== 80+0+0 (secure 0 0 0) 0x7f21fc0579a0 con 0x7f22140b4e60 2024-03-23T03:50:25.058 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:50:25.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.060+0000 7f2218a00640 1 -- 172.21.15.102:0/1556140489 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f21f006ac00 msgr2=0x7f21f006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:25.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.061+0000 7f2218a00640 1 --2- 172.21.15.102:0/1556140489 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f21f006ac00 0x7f21f006d0c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2204036000 tx=0x7f2204008030 comp rx=0 tx=0).stop 2024-03-23T03:50:25.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.061+0000 7f2218a00640 1 -- 172.21.15.102:0/1556140489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22140b4e60 msgr2=0x7f22140caf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:25.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.061+0000 7f2218a00640 1 --2- 172.21.15.102:0/1556140489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22140b4e60 0x7f22140caf30 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f21fc00c410 tx=0x7f21fc00c8d0 comp rx=0 tx=0).stop 2024-03-23T03:50:25.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.061+0000 7f2218a00640 1 -- 172.21.15.102:0/1556140489 shutdown_connections 2024-03-23T03:50:25.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.061+0000 7f2218a00640 1 --2- 172.21.15.102:0/1556140489 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f21f006ac00 0x7f21f006d0c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:25.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.061+0000 7f2218a00640 1 --2- 172.21.15.102:0/1556140489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f221415cc70 0x7f2214158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:25.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.061+0000 7f2218a00640 1 --2- 172.21.15.102:0/1556140489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22140b4e60 0x7f22140caf30 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:25.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.061+0000 7f2218a00640 1 --2- 172.21.15.102:0/1556140489 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f22140b1f20 0x7f221414cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:25.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.061+0000 7f2218a00640 1 -- 172.21.15.102:0/1556140489 >> 172.21.15.102:0/1556140489 conn(0x7f22140cf790 msgr2=0x7f221406b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:25.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.061+0000 7f2218a00640 1 -- 172.21.15.102:0/1556140489 shutdown_connections 2024-03-23T03:50:25.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.061+0000 7f2218a00640 1 -- 172.21.15.102:0/1556140489 wait complete. 2024-03-23T03:50:25.080 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-23T03:50:25.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:25.081 DEBUG:teuthology.orchestra.run.smithi173:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-03-23T03:50:25.191 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:50:25.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:25.192 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T03:50:25.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:25.245 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-03-23T03:50:25.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:25.311 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 134217728 ./subdir) 2024-03-23T03:50:25.371 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:50:25.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.582+0000 7f8d21200640 1 Processor -- start 2024-03-23T03:50:25.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.582+0000 7f8d21200640 1 -- start start 2024-03-23T03:50:25.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.583+0000 7f8d21200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d1c143460 0x7f8d1c15aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:25.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.584+0000 7f8d21200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1c0cc6b0 con 0x7f8d1c0cde20 2024-03-23T03:50:25.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.584+0000 7f8d21200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d1c0cc950 con 0x7f8d1c143930 2024-03-23T03:50:25.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.584+0000 7f8d21200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c0ccc30 con 0x7f8d1c143460 2024-03-23T03:50:25.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.584+0000 7f8d1be00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8d1c0cde20 0x7f8d1c070020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42386/0 (socket says 172.21.15.102:42386) 2024-03-23T03:50:25.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.584+0000 7f8d1be00640 1 -- 172.21.15.102:0/2737139989 learned_addr learned my addr 172.21.15.102:0/2737139989 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:25.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.584+0000 7f8d1b400640 1 --2- 172.21.15.102:0/2737139989 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d1c143460 0x7f8d1c15aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:25.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.584+0000 7f8d1a000640 1 -- 172.21.15.102:0/2737139989 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178369899 0 0) 0x7f8d1c0cc6b0 con 0x7f8d1c0cde20 2024-03-23T03:50:25.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.584+0000 7f8d1a000640 1 -- 172.21.15.102:0/2737139989 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d00003520 con 0x7f8d1c0cde20 2024-03-23T03:50:25.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.585+0000 7f8d1a000640 1 -- 172.21.15.102:0/2737139989 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3797672952 0 0) 0x7f8d00003520 con 0x7f8d1c0cde20 2024-03-23T03:50:25.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.585+0000 7f8d1a000640 1 -- 172.21.15.102:0/2737139989 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d1c143460 msgr2=0x7f8d1c15aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:25.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.585+0000 7f8d1a000640 1 --2- 172.21.15.102:0/2737139989 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d1c143460 0x7f8d1c15aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:25.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.585+0000 7f8d1a000640 1 -- 172.21.15.102:0/2737139989 >> v1:172.21.15.173:6789/0 conn(0x7f8d1c143930 legacy=0x7f8d1c0cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:25.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.585+0000 7f8d1a000640 1 -- 172.21.15.102:0/2737139989 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1c0ccf10 con 0x7f8d1c0cde20 2024-03-23T03:50:25.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.585+0000 7f8d1a000640 1 -- 172.21.15.102:0/2737139989 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f8d140019f0 con 0x7f8d1c0cde20 2024-03-23T03:50:25.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.585+0000 7f8d1a000640 1 -- 172.21.15.102:0/2737139989 >> v1:172.21.15.102:6789/0 conn(0x7f8d1c0cde20 legacy=0x7f8d1c070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:25.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.585+0000 7f8d1a000640 1 --2- 172.21.15.102:0/2737139989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d000039a0 0x7f8d00003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:25.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.585+0000 7f8d1a000640 1 --2- 172.21.15.102:0/2737139989 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d1c143460 0x7f8d000047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:25.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.585+0000 7f8d1a000640 1 --2- 172.21.15.102:0/2737139989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d00004d00 0x7f8d00005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:25.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.585+0000 7f8d1a000640 1 -- 172.21.15.102:0/2737139989 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c0ccc30 con 0x7f8d00004d00 2024-03-23T03:50:25.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.585+0000 7f8d1a000640 1 -- 172.21.15.102:0/2737139989 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0000bac0 con 0x7f8d000039a0 2024-03-23T03:50:25.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.585+0000 7f8d1a000640 1 -- 172.21.15.102:0/2737139989 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0000bc80 con 0x7f8d1c143460 2024-03-23T03:50:25.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.585+0000 7f8d1be00640 1 --2- 172.21.15.102:0/2737139989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d00004d00 0x7f8d00005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:25.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.586+0000 7f8d1b400640 1 --2- 172.21.15.102:0/2737139989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d000039a0 0x7f8d00003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:25.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.586+0000 7f8d1aa00640 1 --2- 172.21.15.102:0/2737139989 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d1c143460 0x7f8d000047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:25.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.586+0000 7f8d1be00640 1 -- 172.21.15.102:0/2737139989 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d1c143460 msgr2=0x7f8d000047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:25.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.586+0000 7f8d1be00640 1 --2- 172.21.15.102:0/2737139989 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d1c143460 0x7f8d000047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:25.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.586+0000 7f8d1be00640 1 -- 172.21.15.102:0/2737139989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d000039a0 msgr2=0x7f8d00003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:25.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.586+0000 7f8d1be00640 1 --2- 172.21.15.102:0/2737139989 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d000039a0 0x7f8d00003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:25.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.586+0000 7f8d1be00640 1 -- 172.21.15.102:0/2737139989 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d00003520 con 0x7f8d00004d00 2024-03-23T03:50:25.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.586+0000 7f8d1aa00640 1 --2- 172.21.15.102:0/2737139989 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d1c143460 0x7f8d000047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:25.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.587+0000 7f8d1be00640 1 --2- 172.21.15.102:0/2737139989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d00004d00 0x7f8d00005180 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400a7f0 tx=0x7f8d000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2fc82e0d3f96ac4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:25.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.587+0000 7f8d1a000640 1 -- 172.21.15.102:0/2737139989 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d140072d0 con 0x7f8d00004d00 2024-03-23T03:50:25.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.587+0000 7f8d1a000640 1 -- 172.21.15.102:0/2737139989 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1402f070 con 0x7f8d00004d00 2024-03-23T03:50:25.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.587+0000 7f8d1a000640 1 -- 172.21.15.102:0/2737139989 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d140076b0 con 0x7f8d00004d00 2024-03-23T03:50:25.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.588+0000 7f8d21200640 1 -- 172.21.15.102:0/2737139989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d00004d00 msgr2=0x7f8d00005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:25.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.588+0000 7f8d21200640 1 --2- 172.21.15.102:0/2737139989 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d00004d00 0x7f8d00005180 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400a7f0 tx=0x7f8d000057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:25.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.588+0000 7f8d1be00640 1 -- 172.21.15.102:0/2737139989 reap_dead start 2024-03-23T03:50:25.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.588+0000 7f8d21200640 1 -- 172.21.15.102:0/2737139989 shutdown_connections 2024-03-23T03:50:25.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.588+0000 7f8d21200640 1 -- 172.21.15.102:0/2737139989 >> 172.21.15.102:0/2737139989 conn(0x7f8d1c065df0 msgr2=0x7f8d1c069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:25.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.588+0000 7f8d21200640 1 -- 172.21.15.102:0/2737139989 shutdown_connections 2024-03-23T03:50:25.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.588+0000 7f8d21200640 1 -- 172.21.15.102:0/2737139989 wait complete. 2024-03-23T03:50:25.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.589+0000 7f8d21200640 1 Processor -- start 2024-03-23T03:50:25.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.590+0000 7f8d21200640 1 -- start start 2024-03-23T03:50:25.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.591+0000 7f8d21200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d00004d00 0x7f8d1c0b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:25.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.591+0000 7f8d21200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d1c0c34c0 0x7f8d1c0cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:25.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.591+0000 7f8d21200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d1c13aa70 0x7f8d1c13af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:25.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.591+0000 7f8d21200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c15b150 con 0x7f8d1c13aa70 2024-03-23T03:50:25.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.591+0000 7f8d21200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c15b3f0 con 0x7f8d1c0c34c0 2024-03-23T03:50:25.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.591+0000 7f8d21200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1c15b6d0 con 0x7f8d00004d00 2024-03-23T03:50:25.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.591+0000 7f8d1be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d1c13aa70 0x7f8d1c13af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:25.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.591+0000 7f8d1be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d1c13aa70 0x7f8d1c13af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42918/0 (socket says 172.21.15.102:42918) 2024-03-23T03:50:25.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.591+0000 7f8d1be00640 1 -- 172.21.15.102:0/1614056314 learned_addr learned my addr 172.21.15.102:0/1614056314 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:25.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.591+0000 7f8d1b400640 1 --2- 172.21.15.102:0/1614056314 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d00004d00 0x7f8d1c0b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:25.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.591+0000 7f8d1be00640 1 -- 172.21.15.102:0/1614056314 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d00004d00 msgr2=0x7f8d1c0b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:25.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.591+0000 7f8d1be00640 1 --2- 172.21.15.102:0/1614056314 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d00004d00 0x7f8d1c0b1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:25.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.591+0000 7f8d1be00640 1 -- 172.21.15.102:0/1614056314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d1c0c34c0 msgr2=0x7f8d1c0cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:25.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.591+0000 7f8d1aa00640 1 --2- 172.21.15.102:0/1614056314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d1c0c34c0 0x7f8d1c0cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:25.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.591+0000 7f8d1be00640 1 --2- 172.21.15.102:0/1614056314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d1c0c34c0 0x7f8d1c0cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:25.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.591+0000 7f8d1be00640 1 -- 172.21.15.102:0/1614056314 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d1c15b9b0 con 0x7f8d1c13aa70 2024-03-23T03:50:25.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.591+0000 7f8d1aa00640 1 --2- 172.21.15.102:0/1614056314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d1c0c34c0 0x7f8d1c0cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:25.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.592+0000 7f8d1be00640 1 --2- 172.21.15.102:0/1614056314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d1c13aa70 0x7f8d1c13af00 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f8d14001cb0 tx=0x7f8d1c0a74a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:25.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.592+0000 7f8d13400640 1 -- 172.21.15.102:0/1614056314 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d14004d60 con 0x7f8d1c13aa70 2024-03-23T03:50:25.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.592+0000 7f8d13400640 1 -- 172.21.15.102:0/1614056314 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d14001250 con 0x7f8d1c13aa70 2024-03-23T03:50:25.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.592+0000 7f8d21200640 1 -- 172.21.15.102:0/1614056314 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d1c15bc90 con 0x7f8d1c13aa70 2024-03-23T03:50:25.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.592+0000 7f8d13400640 1 -- 172.21.15.102:0/1614056314 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d140072d0 con 0x7f8d1c13aa70 2024-03-23T03:50:25.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.592+0000 7f8d21200640 1 -- 172.21.15.102:0/1614056314 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d1c0b2440 con 0x7f8d1c13aa70 2024-03-23T03:50:25.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.593+0000 7f8d21200640 1 -- 172.21.15.102:0/1614056314 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d1c0b2440 con 0x7f8d1c13aa70 2024-03-23T03:50:25.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.594+0000 7f8d13400640 1 -- 172.21.15.102:0/1614056314 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8d14031030 con 0x7f8d1c13aa70 2024-03-23T03:50:25.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.595+0000 7f8d13400640 1 --2- 172.21.15.102:0/1614056314 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8cfc06ac00 0x7f8cfc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:25.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.595+0000 7f8d13400640 1 -- 172.21.15.102:0/1614056314 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8d140ab0e0 con 0x7f8d1c13aa70 2024-03-23T03:50:25.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.596+0000 7f8d1b400640 1 --2- 172.21.15.102:0/1614056314 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8cfc06ac00 0x7f8cfc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:25.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.596+0000 7f8d1b400640 1 --2- 172.21.15.102:0/1614056314 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8cfc06ac00 0x7f8cfc06d0c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c009950 tx=0x7f8d0c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:25.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.600+0000 7f8d13400640 1 -- 172.21.15.102:0/1614056314 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8d140760e0 con 0x7f8d1c13aa70 2024-03-23T03:50:25.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.833+0000 7f8d21200640 1 -- 172.21.15.102:0/1614056314 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8d1c0b2440 con 0x7f8d1c13aa70 2024-03-23T03:50:25.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.834+0000 7f8d13400640 1 -- 172.21.15.102:0/1614056314 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v41) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8d14079700 con 0x7f8d1c13aa70 2024-03-23T03:50:25.835 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T03:50:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.837+0000 7f8d21200640 1 -- 172.21.15.102:0/1614056314 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8cfc06ac00 msgr2=0x7f8cfc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.837+0000 7f8d21200640 1 --2- 172.21.15.102:0/1614056314 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8cfc06ac00 0x7f8cfc06d0c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c009950 tx=0x7f8d0c008040 comp rx=0 tx=0).stop 2024-03-23T03:50:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.837+0000 7f8d21200640 1 -- 172.21.15.102:0/1614056314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d1c13aa70 msgr2=0x7f8d1c13af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.837+0000 7f8d21200640 1 --2- 172.21.15.102:0/1614056314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d1c13aa70 0x7f8d1c13af00 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f8d14001cb0 tx=0x7f8d1c0a74a0 comp rx=0 tx=0).stop 2024-03-23T03:50:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.837+0000 7f8d21200640 1 -- 172.21.15.102:0/1614056314 shutdown_connections 2024-03-23T03:50:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.837+0000 7f8d21200640 1 --2- 172.21.15.102:0/1614056314 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8cfc06ac00 0x7f8cfc06d0c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.837+0000 7f8d21200640 1 --2- 172.21.15.102:0/1614056314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d1c13aa70 0x7f8d1c13af00 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.837+0000 7f8d21200640 1 --2- 172.21.15.102:0/1614056314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8d1c0c34c0 0x7f8d1c0cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.837+0000 7f8d21200640 1 --2- 172.21.15.102:0/1614056314 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8d00004d00 0x7f8d1c0b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.837+0000 7f8d21200640 1 -- 172.21.15.102:0/1614056314 >> 172.21.15.102:0/1614056314 conn(0x7f8d1c065df0 msgr2=0x7f8d1c13bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.837+0000 7f8d21200640 1 -- 172.21.15.102:0/1614056314 shutdown_connections 2024-03-23T03:50:25.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:25.837+0000 7f8d21200640 1 -- 172.21.15.102:0/1614056314 wait complete. 2024-03-23T03:50:25.855 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-23T03:50:25.855 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:50:25.855 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi173.front.sepia.ceph.com 2024-03-23T03:50:25.855 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-23T03:50:25.855 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-23T03:50:25.855 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-23T03:50:25.855 INFO:tasks.cephfs.mount:self.cephfs_mntpt = /subdir 2024-03-23T03:50:25.855 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:50:25.855 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:50:25.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:25.856 DEBUG:teuthology.orchestra.run.smithi173:> ip addr 2024-03-23T03:50:25.910 INFO:teuthology.orchestra.run.smithi173.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:50:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:50:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:50:25.910 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 ::1/128 scope host 2024-03-23T03:50:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:25.911 INFO:teuthology.orchestra.run.smithi173.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:50:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6a brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f0 2024-03-23T03:50:25.911 INFO:teuthology.orchestra.run.smithi173.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:50:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6b brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f1 2024-03-23T03:50:25.911 INFO:teuthology.orchestra.run.smithi173.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:50:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:58 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:25.911 INFO:teuthology.orchestra.run.smithi173.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:50:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:59 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: inet 172.21.15.173/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:50:25.911 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft 42416sec preferred_lft 42416sec 2024-03-23T03:50:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::ec4:7aff:fe88:7259/64 scope link 2024-03-23T03:50:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:25.912 INFO:teuthology.orchestra.run.smithi173.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:50:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 52:69:f2:bf:6d:02 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T03:50:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::6894:e9ff:fe5d:1492/64 scope link 2024-03-23T03:50:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:25.912 INFO:teuthology.orchestra.run.smithi173.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:50:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether a2:27:ac:42:68:ce brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T03:50:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::a027:acff:fe42:68ce/64 scope link 2024-03-23T03:50:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:25.912 INFO:teuthology.orchestra.run.smithi173.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:50:25.912 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 52:69:f2:bf:6d:02 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-23T03:50:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::5069:f2ff:febf:6d02/64 scope link 2024-03-23T03:50:25.913 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:25.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:25.913 DEBUG:teuthology.orchestra.run.smithi173:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:50:25.995 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-03-23T03:50:26.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:26.000 DEBUG:teuthology.orchestra.run.smithi173:> route 2024-03-23T03:50:26.053 INFO:teuthology.orchestra.run.smithi173.stdout:Kernel IP routing table 2024-03-23T03:50:26.053 INFO:teuthology.orchestra.run.smithi173.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:50:26.053 INFO:teuthology.orchestra.run.smithi173.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:50:26.054 INFO:teuthology.orchestra.run.smithi173.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:50:26.054 INFO:teuthology.orchestra.run.smithi173.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:50:26.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:26.054 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-03-23T03:50:26.054 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-03-23T03:50:26.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:50:26.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:50:26.054 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:50:26.055 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-03-23T03:50:26.236 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:26.236 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list 2024-03-23T03:50:26.285 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-23T03:50:26.285 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-23T03:50:26.286 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:50:26.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:26.287 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link set brx.0 up 2024-03-23T03:50:26.377 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:50:26.377 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-23T03:50:26.430 INFO:teuthology.orchestra.run.smithi173.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:50:26.430 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:50:26.430 DEBUG:teuthology.orchestra.run.smithi173:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-23T03:50:26.477 INFO:tasks.cephfs.kernel_mount:mounting using device: :/subdir 2024-03-23T03:50:26.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:26.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/subdir /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:50:26.663 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:50:26.663 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:50:26.663 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:50:26.683 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:50:26.683 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:50:26.683 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:50:26.683 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:50:26.684 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:50:26.711 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:50:26.711 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:50:26.711 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:50:26.711 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:50:26.711 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:50:26.711 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790" 2024-03-23T03:50:26.711 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/subdir 2024-03-23T03:50:26.711 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=776b2820-05f2-4e50-8d40-18dd76c5f6a0" will pass to kernel 2024-03-23T03:50:26.716 INFO:teuthology.orchestra.run.smithi173.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-23T03:50:26.716 INFO:teuthology.orchestra.run.smithi173.stdout: You just mounted a file system that supports labels which does not 2024-03-23T03:50:26.716 INFO:teuthology.orchestra.run.smithi173.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T03:50:26.716 INFO:teuthology.orchestra.run.smithi173.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T03:50:26.716 INFO:teuthology.orchestra.run.smithi173.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T03:50:26.720 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T03:50:26.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:26.720 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-23T03:50:26.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:26.812 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:50:26.893 INFO:teuthology.orchestra.run.smithi173.stdout:client5500 2024-03-23T03:50:26.893 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:50:26.893 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:26.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:26.946 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec df -B1 .) 2024-03-23T03:50:27.010 INFO:teuthology.orchestra.run.smithi173.stdout:Filesystem 1B-blocks Used Available Use% Mounted on 2024-03-23T03:50:27.010 INFO:teuthology.orchestra.run.smithi173.stdout:172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/subdir 134217728 0 134217728 0% /home/ubuntu/cephtest/mnt.1 2024-03-23T03:50:27.011 WARNING:tasks.cephfs.mount:['Filesystem 1B-blocks Used Available Use% Mounted on', '172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/subdir 134217728 0 134217728 0% /home/ubuntu/cephtest/mnt.1'] 2024-03-23T03:50:27.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:27.012 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 268435456 ./subdir) 2024-03-23T03:50:27.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:27.080 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec df -B1 .) 2024-03-23T03:50:27.140 INFO:teuthology.orchestra.run.smithi173.stdout:Filesystem 1B-blocks Used Available Use% Mounted on 2024-03-23T03:50:27.140 INFO:teuthology.orchestra.run.smithi173.stdout:172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/subdir 268435456 0 268435456 0% /home/ubuntu/cephtest/mnt.1 2024-03-23T03:50:27.141 WARNING:tasks.cephfs.mount:['Filesystem 1B-blocks Used Available Use% Mounted on', '172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/subdir 268435456 0 268435456 0% /home/ubuntu/cephtest/mnt.1'] 2024-03-23T03:50:27.142 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T03:50:27.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:50:27.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:50:27.376 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T03:50:27.377 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:50:27.377 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:27.435 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:50:27.435 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:27.485 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:50:27.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.699+0000 7f0d89600640 1 Processor -- start 2024-03-23T03:50:27.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.699+0000 7f0d89600640 1 -- start start 2024-03-23T03:50:27.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.700+0000 7f0d89600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0d84143930 0x7f0d840cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:27.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.700+0000 7f0d89600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d8415b2b0 con 0x7f0d840cdc50 2024-03-23T03:50:27.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.701+0000 7f0d89600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d840cc6b0 con 0x7f0d84143460 2024-03-23T03:50:27.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.701+0000 7f0d89600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d840cc950 con 0x7f0d84143930 2024-03-23T03:50:27.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.701+0000 7f0d7b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f0d84143930 0x7f0d840cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:27.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.701+0000 7f0d88c00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0d840cdc50 0x7f0d84070020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42400/0 (socket says 172.21.15.102:42400) 2024-03-23T03:50:27.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.701+0000 7f0d88c00640 1 -- 172.21.15.102:0/3805228397 learned_addr learned my addr 172.21.15.102:0/3805228397 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:27.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.701+0000 7f0d7b400640 1 -- 172.21.15.102:0/3805228397 >> v1:172.21.15.173:6789/0 conn(0x7f0d84143460 legacy=0x7f0d8415aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:27.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.701+0000 7f0d7b400640 1 -- 172.21.15.102:0/3805228397 >> v1:172.21.15.102:6789/0 conn(0x7f0d840cdc50 legacy=0x7f0d84070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:27.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.701+0000 7f0d7b400640 1 -- 172.21.15.102:0/3805228397 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d840ccc30 con 0x7f0d84143930 2024-03-23T03:50:27.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.701+0000 7f0d83400640 1 -- 172.21.15.102:0/3805228397 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116254919 0 0) 0x7f0d8415b2b0 con 0x7f0d840cdc50 2024-03-23T03:50:27.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.702+0000 7f0d7b400640 1 --2- 172.21.15.102:0/3805228397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d84143930 0x7f0d840cd710 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c0097a0 tx=0x7f0d6c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c1c1665ff961090 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:27.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.703+0000 7f0d83400640 1 -- 172.21.15.102:0/3805228397 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c030610 con 0x7f0d84143930 2024-03-23T03:50:27.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.703+0000 7f0d83400640 1 -- 172.21.15.102:0/3805228397 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6c030770 con 0x7f0d84143930 2024-03-23T03:50:27.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.703+0000 7f0d83400640 1 -- 172.21.15.102:0/3805228397 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c030b00 con 0x7f0d84143930 2024-03-23T03:50:27.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.703+0000 7f0d89600640 1 -- 172.21.15.102:0/3805228397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d84143930 msgr2=0x7f0d840cd710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:27.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.703+0000 7f0d89600640 1 --2- 172.21.15.102:0/3805228397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d84143930 0x7f0d840cd710 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c0097a0 tx=0x7f0d6c02b960 comp rx=0 tx=0).stop 2024-03-23T03:50:27.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.703+0000 7f0d89600640 1 -- 172.21.15.102:0/3805228397 shutdown_connections 2024-03-23T03:50:27.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.703+0000 7f0d89600640 1 --2- 172.21.15.102:0/3805228397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d84143930 0x7f0d840cd710 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:27.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.703+0000 7f0d89600640 1 -- 172.21.15.102:0/3805228397 >> 172.21.15.102:0/3805228397 conn(0x7f0d84065df0 msgr2=0x7f0d84069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:27.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.703+0000 7f0d89600640 1 -- 172.21.15.102:0/3805228397 shutdown_connections 2024-03-23T03:50:27.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.704+0000 7f0d89600640 1 -- 172.21.15.102:0/3805228397 wait complete. 2024-03-23T03:50:27.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.705+0000 7f0d89600640 1 Processor -- start 2024-03-23T03:50:27.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.705+0000 7f0d89600640 1 -- start start 2024-03-23T03:50:27.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.706+0000 7f0d89600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0d840cdc50 0x7f0d84159ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:27.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.706+0000 7f0d89600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d84143460 0x7f0d8406d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:27.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.706+0000 7f0d89600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d84143930 0x7f0d8413e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:27.709 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.706+0000 7f0d89600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d840ab920 con 0x7f0d84143460 2024-03-23T03:50:27.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.706+0000 7f0d89600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d8406e0b0 con 0x7f0d84143930 2024-03-23T03:50:27.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.706+0000 7f0d89600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d841f63c0 con 0x7f0d840cdc50 2024-03-23T03:50:27.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.706+0000 7f0d7b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d84143460 0x7f0d8406d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:27.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.707+0000 7f0d7b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d84143460 0x7f0d8406d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42934/0 (socket says 172.21.15.102:42934) 2024-03-23T03:50:27.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.707+0000 7f0d7b400640 1 -- 172.21.15.102:0/2925732362 learned_addr learned my addr 172.21.15.102:0/2925732362 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:27.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.707+0000 7f0d83e00640 1 --2- 172.21.15.102:0/2925732362 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0d840cdc50 0x7f0d84159ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:27.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.707+0000 7f0d88c00640 1 --2- 172.21.15.102:0/2925732362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d84143930 0x7f0d8413e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:27.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.707+0000 7f0d7b400640 1 -- 172.21.15.102:0/2925732362 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0d840cdc50 msgr2=0x7f0d84159ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:27.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.707+0000 7f0d7b400640 1 --2- 172.21.15.102:0/2925732362 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0d840cdc50 0x7f0d84159ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:27.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.707+0000 7f0d7b400640 1 -- 172.21.15.102:0/2925732362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d84143930 msgr2=0x7f0d8413e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:27.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.707+0000 7f0d7b400640 1 --2- 172.21.15.102:0/2925732362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d84143930 0x7f0d8413e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:27.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.707+0000 7f0d7b400640 1 -- 172.21.15.102:0/2925732362 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c009400 con 0x7f0d84143460 2024-03-23T03:50:27.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.707+0000 7f0d7b400640 1 --2- 172.21.15.102:0/2925732362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d84143460 0x7f0d8406d660 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c000c00 tx=0x7f0d6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:27.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.708+0000 7f0d80c00640 1 -- 172.21.15.102:0/2925732362 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c03f070 con 0x7f0d84143460 2024-03-23T03:50:27.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.708+0000 7f0d80c00640 1 -- 172.21.15.102:0/2925732362 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d6c03a810 con 0x7f0d84143460 2024-03-23T03:50:27.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.708+0000 7f0d89600640 1 -- 172.21.15.102:0/2925732362 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d841f5600 con 0x7f0d84143460 2024-03-23T03:50:27.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.708+0000 7f0d80c00640 1 -- 172.21.15.102:0/2925732362 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d6c03aba0 con 0x7f0d84143460 2024-03-23T03:50:27.711 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.708+0000 7f0d89600640 1 -- 172.21.15.102:0/2925732362 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d841f5bc0 con 0x7f0d84143460 2024-03-23T03:50:27.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.709+0000 7f0d89600640 1 -- 172.21.15.102:0/2925732362 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d8404e030 con 0x7f0d84143460 2024-03-23T03:50:27.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.710+0000 7f0d80c00640 1 -- 172.21.15.102:0/2925732362 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0d6c043070 con 0x7f0d84143460 2024-03-23T03:50:27.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.711+0000 7f0d80c00640 1 --2- 172.21.15.102:0/2925732362 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0d6006ac00 0x7f0d6006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:27.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.711+0000 7f0d80c00640 1 -- 172.21.15.102:0/2925732362 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0d6c0ab2d0 con 0x7f0d84143460 2024-03-23T03:50:27.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.711+0000 7f0d83e00640 1 --2- 172.21.15.102:0/2925732362 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0d6006ac00 0x7f0d6006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:27.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.712+0000 7f0d83e00640 1 --2- 172.21.15.102:0/2925732362 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0d6006ac00 0x7f0d6006d0c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0d8406d5b0 tx=0x7f0d74009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:27.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.715+0000 7f0d80c00640 1 -- 172.21.15.102:0/2925732362 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0d6c076480 con 0x7f0d84143460 2024-03-23T03:50:27.918 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.918+0000 7f0d89600640 1 -- 172.21.15.102:0/2925732362 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d8415bba0 con 0x7f0d84143460 2024-03-23T03:50:27.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.921+0000 7f0d80c00640 1 -- 172.21.15.102:0/2925732362 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16357 (secure 0 0 0) 0x7f0d6c079aa0 con 0x7f0d84143460 2024-03-23T03:50:27.922 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:27.922 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":81,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:08.942399+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:50:06.347556+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-23T03:50:06.924370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:50:27.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.924+0000 7f0d89600640 1 -- 172.21.15.102:0/2925732362 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0d6006ac00 msgr2=0x7f0d6006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:27.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.924+0000 7f0d89600640 1 --2- 172.21.15.102:0/2925732362 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0d6006ac00 0x7f0d6006d0c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0d8406d5b0 tx=0x7f0d74009040 comp rx=0 tx=0).stop 2024-03-23T03:50:27.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.924+0000 7f0d89600640 1 -- 172.21.15.102:0/2925732362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d84143460 msgr2=0x7f0d8406d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:27.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.924+0000 7f0d89600640 1 --2- 172.21.15.102:0/2925732362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d84143460 0x7f0d8406d660 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f0d6c000c00 tx=0x7f0d6c0046c0 comp rx=0 tx=0).stop 2024-03-23T03:50:27.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.924+0000 7f0d89600640 1 -- 172.21.15.102:0/2925732362 shutdown_connections 2024-03-23T03:50:27.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.924+0000 7f0d89600640 1 --2- 172.21.15.102:0/2925732362 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0d6006ac00 0x7f0d6006d0c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:27.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.924+0000 7f0d89600640 1 --2- 172.21.15.102:0/2925732362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d84143930 0x7f0d8413e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:27.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.924+0000 7f0d89600640 1 --2- 172.21.15.102:0/2925732362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d84143460 0x7f0d8406d660 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:27.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.924+0000 7f0d89600640 1 --2- 172.21.15.102:0/2925732362 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0d840cdc50 0x7f0d84159ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:27.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.924+0000 7f0d89600640 1 -- 172.21.15.102:0/2925732362 >> 172.21.15.102:0/2925732362 conn(0x7f0d84065df0 msgr2=0x7f0d8415ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:27.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.924+0000 7f0d89600640 1 -- 172.21.15.102:0/2925732362 shutdown_connections 2024-03-23T03:50:27.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:27.924+0000 7f0d89600640 1 -- 172.21.15.102:0/2925732362 wait complete. 2024-03-23T03:50:27.945 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:50:06.347556+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:50:06.924370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:50:27.946 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:50:28.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.153+0000 7f72aca00640 1 Processor -- start 2024-03-23T03:50:28.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.153+0000 7f72aca00640 1 -- start start 2024-03-23T03:50:28.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.154+0000 7f72aca00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72a815c9e0 0x7f72a815dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:28.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.154+0000 7f72aca00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f72a80cb420 0x7f72a8160f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:28.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.155+0000 7f72aca00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a8160530 con 0x7f72a80b12d0 2024-03-23T03:50:28.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.155+0000 7f72aca00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a8160af0 con 0x7f72a80cb420 2024-03-23T03:50:28.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.155+0000 7f72aca00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a81614c0 con 0x7f72a815c9e0 2024-03-23T03:50:28.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.155+0000 7f72a7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f72a80cb420 0x7f72a8160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:28.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.155+0000 7f72a7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f72a80cb420 0x7f72a8160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42936/0 (socket says 172.21.15.102:42936) 2024-03-23T03:50:28.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.155+0000 7f72a7e00640 1 -- 172.21.15.102:0/4013411114 learned_addr learned my addr 172.21.15.102:0/4013411114 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:28.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.155+0000 7f72a7400640 1 --2- 172.21.15.102:0/4013411114 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72a815c9e0 0x7f72a815dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:28.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.156+0000 7f72a7e00640 1 -- 172.21.15.102:0/4013411114 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72a815c9e0 msgr2=0x7f72a815dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:28.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.156+0000 7f72a7e00640 1 --2- 172.21.15.102:0/4013411114 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72a815c9e0 0x7f72a815dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.156+0000 7f72a7e00640 1 -- 172.21.15.102:0/4013411114 >> v1:172.21.15.173:6789/0 conn(0x7f72a80b12d0 legacy=0x7f72a80cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:28.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.156+0000 7f72a7e00640 1 -- 172.21.15.102:0/4013411114 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a815cdc0 con 0x7f72a80cb420 2024-03-23T03:50:28.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.157+0000 7f72a7e00640 1 --2- 172.21.15.102:0/4013411114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72a80cb420 0x7f72a8160f80 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f72a0009e50 tx=0x7f72a002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4e025c83c4c332 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:28.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.157+0000 7f72a6000640 1 -- 172.21.15.102:0/4013411114 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a002cb00 con 0x7f72a80cb420 2024-03-23T03:50:28.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.157+0000 7f72a6000640 1 -- 172.21.15.102:0/4013411114 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a002cc60 con 0x7f72a80cb420 2024-03-23T03:50:28.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.157+0000 7f72a6000640 1 -- 172.21.15.102:0/4013411114 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72a00343f0 con 0x7f72a80cb420 2024-03-23T03:50:28.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.158+0000 7f72aca00640 1 -- 172.21.15.102:0/4013411114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72a80cb420 msgr2=0x7f72a8160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:28.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.158+0000 7f72aca00640 1 --2- 172.21.15.102:0/4013411114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72a80cb420 0x7f72a8160f80 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f72a0009e50 tx=0x7f72a002bf60 comp rx=0 tx=0).stop 2024-03-23T03:50:28.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.158+0000 7f72aca00640 1 -- 172.21.15.102:0/4013411114 shutdown_connections 2024-03-23T03:50:28.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.158+0000 7f72aca00640 1 --2- 172.21.15.102:0/4013411114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72a80cb420 0x7f72a8160f80 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.158+0000 7f72aca00640 1 --2- 172.21.15.102:0/4013411114 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72a815c9e0 0x7f72a815dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.158+0000 7f72aca00640 1 -- 172.21.15.102:0/4013411114 >> 172.21.15.102:0/4013411114 conn(0x7f72a80b01b0 msgr2=0x7f72a80b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:28.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.158+0000 7f72aca00640 1 -- 172.21.15.102:0/4013411114 shutdown_connections 2024-03-23T03:50:28.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.158+0000 7f72aca00640 1 -- 172.21.15.102:0/4013411114 wait complete. 2024-03-23T03:50:28.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.160+0000 7f72aca00640 1 Processor -- start 2024-03-23T03:50:28.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.160+0000 7f72aca00640 1 -- start start 2024-03-23T03:50:28.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.161+0000 7f72aca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a80b12d0 0x7f72a8145240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:28.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.161+0000 7f72aca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72a80cb420 0x7f72a8071370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:28.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.161+0000 7f72aca00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72a815c9e0 0x7f72a80ce540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:28.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.161+0000 7f72aca00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a8152120 con 0x7f72a80cb420 2024-03-23T03:50:28.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.161+0000 7f72aca00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a8155e70 con 0x7f72a80b12d0 2024-03-23T03:50:28.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.161+0000 7f72aca00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a8156150 con 0x7f72a815c9e0 2024-03-23T03:50:28.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.161+0000 7f72a6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72a80cb420 0x7f72a8071370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:28.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.161+0000 7f72a6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72a80cb420 0x7f72a8071370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42938/0 (socket says 172.21.15.102:42938) 2024-03-23T03:50:28.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.161+0000 7f72a6a00640 1 -- 172.21.15.102:0/3925242273 learned_addr learned my addr 172.21.15.102:0/3925242273 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.162+0000 7f72a7400640 1 --2- 172.21.15.102:0/3925242273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a80b12d0 0x7f72a8145240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.162+0000 7f72a7e00640 1 --2- 172.21.15.102:0/3925242273 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72a815c9e0 0x7f72a80ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.162+0000 7f72a6a00640 1 -- 172.21.15.102:0/3925242273 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72a815c9e0 msgr2=0x7f72a80ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.162+0000 7f72a6a00640 1 --2- 172.21.15.102:0/3925242273 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72a815c9e0 0x7f72a80ce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.162+0000 7f72a6a00640 1 -- 172.21.15.102:0/3925242273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a80b12d0 msgr2=0x7f72a8145240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.162+0000 7f72a6a00640 1 --2- 172.21.15.102:0/3925242273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a80b12d0 0x7f72a8145240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.162+0000 7f72a6a00640 1 -- 172.21.15.102:0/3925242273 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a0009b30 con 0x7f72a80cb420 2024-03-23T03:50:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.162+0000 7f72a7400640 1 --2- 172.21.15.102:0/3925242273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a80b12d0 0x7f72a8145240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.162+0000 7f72a6a00640 1 --2- 172.21.15.102:0/3925242273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72a80cb420 0x7f72a8071370 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f729400c400 tx=0x7f729400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.163+0000 7f729f400640 1 -- 172.21.15.102:0/3925242273 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72940052d0 con 0x7f72a80cb420 2024-03-23T03:50:28.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.163+0000 7f729f400640 1 -- 172.21.15.102:0/3925242273 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7294005430 con 0x7f72a80cb420 2024-03-23T03:50:28.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.163+0000 7f729f400640 1 -- 172.21.15.102:0/3925242273 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7294003830 con 0x7f72a80cb420 2024-03-23T03:50:28.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.163+0000 7f72aca00640 1 -- 172.21.15.102:0/3925242273 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a8156430 con 0x7f72a80cb420 2024-03-23T03:50:28.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.163+0000 7f72aca00640 1 -- 172.21.15.102:0/3925242273 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a81569f0 con 0x7f72a80cb420 2024-03-23T03:50:28.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.164+0000 7f72aca00640 1 -- 172.21.15.102:0/3925242273 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a804e030 con 0x7f72a80cb420 2024-03-23T03:50:28.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.165+0000 7f729f400640 1 -- 172.21.15.102:0/3925242273 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7294003990 con 0x7f72a80cb420 2024-03-23T03:50:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.166+0000 7f729f400640 1 --2- 172.21.15.102:0/3925242273 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f728806ac00 0x7f728806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.166+0000 7f729f400640 1 -- 172.21.15.102:0/3925242273 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7294088b40 con 0x7f72a80cb420 2024-03-23T03:50:28.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.166+0000 7f72a7400640 1 --2- 172.21.15.102:0/3925242273 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f728806ac00 0x7f728806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:28.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.167+0000 7f72a7400640 1 --2- 172.21.15.102:0/3925242273 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f728806ac00 0x7f728806d0c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7298009ac0 tx=0x7f7298008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:28.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.170+0000 7f729f400640 1 -- 172.21.15.102:0/3925242273 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7294053ab0 con 0x7f72a80cb420 2024-03-23T03:50:28.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.364+0000 7f72aca00640 1 -- 172.21.15.102:0/3925242273 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f72a80caaf0 con 0x7f72a80cb420 2024-03-23T03:50:28.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.365+0000 7f729f400640 1 -- 172.21.15.102:0/3925242273 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7294088e60 con 0x7f72a80cb420 2024-03-23T03:50:28.366 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:50:28.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.368+0000 7f72aca00640 1 -- 172.21.15.102:0/3925242273 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f728806ac00 msgr2=0x7f728806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:28.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.368+0000 7f72aca00640 1 --2- 172.21.15.102:0/3925242273 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f728806ac00 0x7f728806d0c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7298009ac0 tx=0x7f7298008040 comp rx=0 tx=0).stop 2024-03-23T03:50:28.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.368+0000 7f72aca00640 1 -- 172.21.15.102:0/3925242273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72a80cb420 msgr2=0x7f72a8071370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:28.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.368+0000 7f72aca00640 1 --2- 172.21.15.102:0/3925242273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72a80cb420 0x7f72a8071370 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f729400c400 tx=0x7f729400c8c0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.368+0000 7f72aca00640 1 -- 172.21.15.102:0/3925242273 shutdown_connections 2024-03-23T03:50:28.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.368+0000 7f72aca00640 1 --2- 172.21.15.102:0/3925242273 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f728806ac00 0x7f728806d0c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.368+0000 7f72aca00640 1 --2- 172.21.15.102:0/3925242273 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f72a815c9e0 0x7f72a80ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.368+0000 7f72aca00640 1 --2- 172.21.15.102:0/3925242273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72a80cb420 0x7f72a8071370 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.368+0000 7f72aca00640 1 --2- 172.21.15.102:0/3925242273 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72a80b12d0 0x7f72a8145240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.368+0000 7f72aca00640 1 -- 172.21.15.102:0/3925242273 >> 172.21.15.102:0/3925242273 conn(0x7f72a80b01b0 msgr2=0x7f72a815e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:28.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.368+0000 7f72aca00640 1 -- 172.21.15.102:0/3925242273 shutdown_connections 2024-03-23T03:50:28.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.368+0000 7f72aca00640 1 -- 172.21.15.102:0/3925242273 wait complete. 2024-03-23T03:50:28.387 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:50:28.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.591+0000 7f09a5800640 1 Processor -- start 2024-03-23T03:50:28.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.591+0000 7f09a5800640 1 -- start start 2024-03-23T03:50:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.593+0000 7f09a5800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09a00b84b0 0x7f09a00d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.593+0000 7f09a5800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f09a00b5aa0 0x7f09a00b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.593+0000 7f09a5800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a00d1280 con 0x7f09a00cd740 2024-03-23T03:50:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.593+0000 7f09a5800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a00d20c0 con 0x7f09a00b5aa0 2024-03-23T03:50:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.593+0000 7f09a5800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a00b8910 con 0x7f09a00b84b0 2024-03-23T03:50:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.593+0000 7f099ea00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f09a00cd740 0x7f09a00cdb20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42412/0 (socket says 172.21.15.102:42412) 2024-03-23T03:50:28.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.593+0000 7f099ea00640 1 -- 172.21.15.102:0/515904167 learned_addr learned my addr 172.21.15.102:0/515904167 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:28.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099e000640 1 --2- 172.21.15.102:0/515904167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09a00b84b0 0x7f09a00d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099f400640 1 --2- 172.21.15.102:0/515904167 >> v2:172.21.15.173:3300/0 conn(0x7f09a00b5aa0 0x7f09a00b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099d600640 1 -- 172.21.15.102:0/515904167 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572234011 0 0) 0x7f09a00d1280 con 0x7f09a00cd740 2024-03-23T03:50:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099d600640 1 -- 172.21.15.102:0/515904167 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0984003520 con 0x7f09a00cd740 2024-03-23T03:50:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099d600640 1 -- 172.21.15.102:0/515904167 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 924112506 0 0) 0x7f0984003520 con 0x7f09a00cd740 2024-03-23T03:50:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099d600640 1 -- 172.21.15.102:0/515904167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09a00b84b0 msgr2=0x7f09a00d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099d600640 1 --2- 172.21.15.102:0/515904167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09a00b84b0 0x7f09a00d0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099d600640 1 -- 172.21.15.102:0/515904167 >> v2:172.21.15.173:3300/0 conn(0x7f09a00b5aa0 msgr2=0x7f09a00b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099d600640 1 --2- 172.21.15.102:0/515904167 >> v2:172.21.15.173:3300/0 conn(0x7f09a00b5aa0 0x7f09a00b5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099d600640 1 -- 172.21.15.102:0/515904167 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a00d14b0 con 0x7f09a00cd740 2024-03-23T03:50:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099d600640 1 -- 172.21.15.102:0/515904167 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f0988001670 con 0x7f09a00cd740 2024-03-23T03:50:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099d600640 1 -- 172.21.15.102:0/515904167 >> v1:172.21.15.102:6789/0 conn(0x7f09a00cd740 legacy=0x7f09a00cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:28.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099d600640 1 --2- 172.21.15.102:0/515904167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09a00b84b0 0x7f0984003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:28.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099d600640 1 --2- 172.21.15.102:0/515904167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0984003ef0 0x7f09840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:28.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099d600640 1 --2- 172.21.15.102:0/515904167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0984004d00 0x7f0984005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:28.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099d600640 1 -- 172.21.15.102:0/515904167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a00d20c0 con 0x7f0984004d00 2024-03-23T03:50:28.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099d600640 1 -- 172.21.15.102:0/515904167 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a00b8910 con 0x7f0984003ef0 2024-03-23T03:50:28.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.594+0000 7f099d600640 1 -- 172.21.15.102:0/515904167 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098400b980 con 0x7f09a00b84b0 2024-03-23T03:50:28.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.595+0000 7f099f400640 1 --2- 172.21.15.102:0/515904167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0984004d00 0x7f0984005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:28.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.595+0000 7f099e000640 1 --2- 172.21.15.102:0/515904167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0984003ef0 0x7f09840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:28.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.595+0000 7f099ea00640 1 --2- 172.21.15.102:0/515904167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09a00b84b0 0x7f0984003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:28.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.595+0000 7f099f400640 1 -- 172.21.15.102:0/515904167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09a00b84b0 msgr2=0x7f0984003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:28.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.595+0000 7f099f400640 1 --2- 172.21.15.102:0/515904167 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09a00b84b0 0x7f0984003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.595+0000 7f099f400640 1 -- 172.21.15.102:0/515904167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0984003ef0 msgr2=0x7f09840047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.595+0000 7f099f400640 1 --2- 172.21.15.102:0/515904167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0984003ef0 0x7f09840047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.595+0000 7f099f400640 1 -- 172.21.15.102:0/515904167 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0984003520 con 0x7f0984004d00 2024-03-23T03:50:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.595+0000 7f099e000640 1 --2- 172.21.15.102:0/515904167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0984003ef0 0x7f09840047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.596+0000 7f099f400640 1 --2- 172.21.15.102:0/515904167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0984004d00 0x7f0984005180 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f0998009ac0 tx=0x7f09840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6f8b0bac676f6b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.596+0000 7f099d600640 1 -- 172.21.15.102:0/515904167 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099802c6d0 con 0x7f0984004d00 2024-03-23T03:50:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.596+0000 7f099d600640 1 -- 172.21.15.102:0/515904167 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099802c830 con 0x7f0984004d00 2024-03-23T03:50:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.596+0000 7f099d600640 1 -- 172.21.15.102:0/515904167 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099802cbc0 con 0x7f0984004d00 2024-03-23T03:50:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.597+0000 7f09a5800640 1 -- 172.21.15.102:0/515904167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0984004d00 msgr2=0x7f0984005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.597+0000 7f09a5800640 1 --2- 172.21.15.102:0/515904167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0984004d00 0x7f0984005180 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f0998009ac0 tx=0x7f09840057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.597+0000 7f099f400640 1 -- 172.21.15.102:0/515904167 reap_dead start 2024-03-23T03:50:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.597+0000 7f09a5800640 1 -- 172.21.15.102:0/515904167 shutdown_connections 2024-03-23T03:50:28.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.597+0000 7f09a5800640 1 -- 172.21.15.102:0/515904167 >> 172.21.15.102:0/515904167 conn(0x7f09a00cf5d0 msgr2=0x7f09a00cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:28.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.597+0000 7f09a5800640 1 -- 172.21.15.102:0/515904167 shutdown_connections 2024-03-23T03:50:28.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.597+0000 7f09a5800640 1 -- 172.21.15.102:0/515904167 wait complete. 2024-03-23T03:50:28.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.599+0000 7f09a5800640 1 Processor -- start 2024-03-23T03:50:28.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.599+0000 7f09a5800640 1 -- start start 2024-03-23T03:50:28.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.600+0000 7f09a5800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0984004d00 0x7f09a01f17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:28.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.600+0000 7f09a5800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09a00caa90 0x7f09a00ac2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:28.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.600+0000 7f09a5800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09a00caf10 0x7f09a00ac810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.600+0000 7f09a5800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a00acd50 con 0x7f09a00caf10 2024-03-23T03:50:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.600+0000 7f09a5800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a00cb470 con 0x7f0984004d00 2024-03-23T03:50:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.600+0000 7f09a5800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a00cb750 con 0x7f09a00caa90 2024-03-23T03:50:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.600+0000 7f099f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09a00caf10 0x7f09a00ac810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.600+0000 7f099f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09a00caf10 0x7f09a00ac810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42956/0 (socket says 172.21.15.102:42956) 2024-03-23T03:50:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.600+0000 7f099f400640 1 -- 172.21.15.102:0/3231804064 learned_addr learned my addr 172.21.15.102:0/3231804064 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.600+0000 7f099ea00640 1 --2- 172.21.15.102:0/3231804064 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0984004d00 0x7f09a01f17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.601+0000 7f099e000640 1 --2- 172.21.15.102:0/3231804064 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09a00caa90 0x7f09a00ac2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.601+0000 7f099f400640 1 -- 172.21.15.102:0/3231804064 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09a00caa90 msgr2=0x7f09a00ac2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.601+0000 7f099f400640 1 --2- 172.21.15.102:0/3231804064 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09a00caa90 0x7f09a00ac2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.601+0000 7f099f400640 1 -- 172.21.15.102:0/3231804064 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0984004d00 msgr2=0x7f09a01f17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.601+0000 7f099f400640 1 --2- 172.21.15.102:0/3231804064 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0984004d00 0x7f09a01f17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.601+0000 7f099f400640 1 -- 172.21.15.102:0/3231804064 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a00cba30 con 0x7f09a00caf10 2024-03-23T03:50:28.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.601+0000 7f099ea00640 1 --2- 172.21.15.102:0/3231804064 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0984004d00 0x7f09a01f17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:28.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.601+0000 7f099f400640 1 --2- 172.21.15.102:0/3231804064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09a00caf10 0x7f09a00ac810 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f0998009ac0 tx=0x7f0998004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:28.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.601+0000 7f0996a00640 1 -- 172.21.15.102:0/3231804064 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099802cdb0 con 0x7f09a00caf10 2024-03-23T03:50:28.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.601+0000 7f0996a00640 1 -- 172.21.15.102:0/3231804064 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0998003610 con 0x7f09a00caf10 2024-03-23T03:50:28.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.601+0000 7f0996a00640 1 -- 172.21.15.102:0/3231804064 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09980039c0 con 0x7f09a00caf10 2024-03-23T03:50:28.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.601+0000 7f09a5800640 1 -- 172.21.15.102:0/3231804064 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a00cbd10 con 0x7f09a00caf10 2024-03-23T03:50:28.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.601+0000 7f09a5800640 1 -- 172.21.15.102:0/3231804064 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a01f1d20 con 0x7f09a00caf10 2024-03-23T03:50:28.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.603+0000 7f09a5800640 1 -- 172.21.15.102:0/3231804064 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a00cc230 con 0x7f09a00caf10 2024-03-23T03:50:28.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.603+0000 7f0996a00640 1 -- 172.21.15.102:0/3231804064 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0998003b20 con 0x7f09a00caf10 2024-03-23T03:50:28.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.604+0000 7f0996a00640 1 --2- 172.21.15.102:0/3231804064 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f097c06ac30 0x7f097c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:28.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.604+0000 7f0996a00640 1 -- 172.21.15.102:0/3231804064 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f09980aa3b0 con 0x7f09a00caf10 2024-03-23T03:50:28.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.605+0000 7f099ea00640 1 --2- 172.21.15.102:0/3231804064 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f097c06ac30 0x7f097c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:28.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.605+0000 7f099ea00640 1 --2- 172.21.15.102:0/3231804064 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f097c06ac30 0x7f097c06d0f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f09880017a0 tx=0x7f0988009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:28.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.609+0000 7f0996a00640 1 -- 172.21.15.102:0/3231804064 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f09a00cc230 con 0x7f09a00caf10 2024-03-23T03:50:28.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.800+0000 7f09a5800640 1 -- 172.21.15.102:0/3231804064 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f09a015e0e0 con 0x7f09a00caf10 2024-03-23T03:50:28.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.801+0000 7f0996a00640 1 -- 172.21.15.102:0/3231804064 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7f099807de60 con 0x7f09a00caf10 2024-03-23T03:50:28.802 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:50:28.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.803+0000 7f09a5800640 1 -- 172.21.15.102:0/3231804064 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f097c06ac30 msgr2=0x7f097c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:28.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.803+0000 7f09a5800640 1 --2- 172.21.15.102:0/3231804064 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f097c06ac30 0x7f097c06d0f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f09880017a0 tx=0x7f0988009040 comp rx=0 tx=0).stop 2024-03-23T03:50:28.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.803+0000 7f09a5800640 1 -- 172.21.15.102:0/3231804064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09a00caf10 msgr2=0x7f09a00ac810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:28.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.803+0000 7f09a5800640 1 --2- 172.21.15.102:0/3231804064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09a00caf10 0x7f09a00ac810 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f0998009ac0 tx=0x7f0998004730 comp rx=0 tx=0).stop 2024-03-23T03:50:28.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.803+0000 7f09a5800640 1 -- 172.21.15.102:0/3231804064 shutdown_connections 2024-03-23T03:50:28.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.803+0000 7f09a5800640 1 --2- 172.21.15.102:0/3231804064 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f097c06ac30 0x7f097c06d0f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.803+0000 7f09a5800640 1 --2- 172.21.15.102:0/3231804064 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09a00caf10 0x7f09a00ac810 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.803+0000 7f09a5800640 1 --2- 172.21.15.102:0/3231804064 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f09a00caa90 0x7f09a00ac2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.803+0000 7f09a5800640 1 --2- 172.21.15.102:0/3231804064 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0984004d00 0x7f09a01f17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:28.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.803+0000 7f09a5800640 1 -- 172.21.15.102:0/3231804064 >> 172.21.15.102:0/3231804064 conn(0x7f09a00cf5d0 msgr2=0x7f09a00cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:28.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.804+0000 7f09a5800640 1 -- 172.21.15.102:0/3231804064 shutdown_connections 2024-03-23T03:50:28.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:28.804+0000 7f09a5800640 1 -- 172.21.15.102:0/3231804064 wait complete. 2024-03-23T03:50:28.821 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:50:29.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.025+0000 7f1e6a200640 1 Processor -- start 2024-03-23T03:50:29.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.025+0000 7f1e6a200640 1 -- start start 2024-03-23T03:50:29.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.027+0000 7f1e6a200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1e640b84b0 0x7f1e640d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:29.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.027+0000 7f1e6a200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e640d1280 con 0x7f1e640b5aa0 2024-03-23T03:50:29.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.027+0000 7f1e6a200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e640d20c0 con 0x7f1e640cd740 2024-03-23T03:50:29.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.027+0000 7f1e6a200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e640b8940 con 0x7f1e640b84b0 2024-03-23T03:50:29.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.027+0000 7f1e63e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1e640b5aa0 0x7f1e640b5f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42424/0 (socket says 172.21.15.102:42424) 2024-03-23T03:50:29.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.027+0000 7f1e63e00640 1 -- 172.21.15.102:0/187149336 learned_addr learned my addr 172.21.15.102:0/187149336 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:29.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.027+0000 7f1e5aa00640 1 --2- 172.21.15.102:0/187149336 >> v2:172.21.15.173:3300/0 conn(0x7f1e640b84b0 0x7f1e640d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:29.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.028+0000 7f1e62a00640 1 -- 172.21.15.102:0/187149336 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2466648797 0 0) 0x7f1e640d1280 con 0x7f1e640b5aa0 2024-03-23T03:50:29.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.028+0000 7f1e62a00640 1 -- 172.21.15.102:0/187149336 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e48003520 con 0x7f1e640b5aa0 2024-03-23T03:50:29.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.028+0000 7f1e62a00640 1 -- 172.21.15.102:0/187149336 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4275505217 0 0) 0x7f1e48003520 con 0x7f1e640b5aa0 2024-03-23T03:50:29.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.028+0000 7f1e62a00640 1 -- 172.21.15.102:0/187149336 >> v2:172.21.15.173:3300/0 conn(0x7f1e640b84b0 msgr2=0x7f1e640d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:29.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.028+0000 7f1e62a00640 1 --2- 172.21.15.102:0/187149336 >> v2:172.21.15.173:3300/0 conn(0x7f1e640b84b0 0x7f1e640d0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.028+0000 7f1e62a00640 1 -- 172.21.15.102:0/187149336 >> v1:172.21.15.173:6789/0 conn(0x7f1e640cd740 legacy=0x7f1e640cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:29.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.028+0000 7f1e62a00640 1 -- 172.21.15.102:0/187149336 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e640d14b0 con 0x7f1e640b5aa0 2024-03-23T03:50:29.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.028+0000 7f1e62a00640 1 -- 172.21.15.102:0/187149336 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f1e5c001670 con 0x7f1e640b5aa0 2024-03-23T03:50:29.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.028+0000 7f1e62a00640 1 -- 172.21.15.102:0/187149336 >> v1:172.21.15.102:6789/0 conn(0x7f1e640b5aa0 legacy=0x7f1e640b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:29.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.028+0000 7f1e62a00640 1 --2- 172.21.15.102:0/187149336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e480039c0 0x7f1e48003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:29.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.028+0000 7f1e62a00640 1 --2- 172.21.15.102:0/187149336 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e480047c0 0x7f1e48004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:29.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.028+0000 7f1e62a00640 1 --2- 172.21.15.102:0/187149336 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1e48005180 0x7f1e4800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:29.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.028+0000 7f1e62a00640 1 -- 172.21.15.102:0/187149336 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e640b8940 con 0x7f1e480039c0 2024-03-23T03:50:29.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.028+0000 7f1e62a00640 1 -- 172.21.15.102:0/187149336 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4800c080 con 0x7f1e480047c0 2024-03-23T03:50:29.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.028+0000 7f1e62a00640 1 -- 172.21.15.102:0/187149336 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4800c240 con 0x7f1e48005180 2024-03-23T03:50:29.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.028+0000 7f1e63400640 1 --2- 172.21.15.102:0/187149336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e480039c0 0x7f1e48003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:29.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.029+0000 7f1e5aa00640 1 --2- 172.21.15.102:0/187149336 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e480047c0 0x7f1e48004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:29.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.029+0000 7f1e63400640 1 -- 172.21.15.102:0/187149336 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1e48005180 msgr2=0x7f1e4800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:29.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.029+0000 7f1e63400640 1 --2- 172.21.15.102:0/187149336 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1e48005180 0x7f1e4800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.029+0000 7f1e63400640 1 -- 172.21.15.102:0/187149336 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e480047c0 msgr2=0x7f1e48004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:29.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.029+0000 7f1e63400640 1 --2- 172.21.15.102:0/187149336 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e480047c0 0x7f1e48004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.029+0000 7f1e63e00640 1 -- 172.21.15.102:0/187149336 reap_dead start 2024-03-23T03:50:29.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.029+0000 7f1e63400640 1 -- 172.21.15.102:0/187149336 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e48003520 con 0x7f1e480039c0 2024-03-23T03:50:29.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.029+0000 7f1e5aa00640 1 --2- 172.21.15.102:0/187149336 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e480047c0 0x7f1e48004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:29.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.030+0000 7f1e63400640 1 --2- 172.21.15.102:0/187149336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e480039c0 0x7f1e48003de0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f1e54001e50 tx=0x7f1e480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e316e7b65dba73 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:29.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.030+0000 7f1e62a00640 1 -- 172.21.15.102:0/187149336 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e54003b70 con 0x7f1e480039c0 2024-03-23T03:50:29.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.030+0000 7f1e62a00640 1 -- 172.21.15.102:0/187149336 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e54003cd0 con 0x7f1e480039c0 2024-03-23T03:50:29.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.030+0000 7f1e62a00640 1 -- 172.21.15.102:0/187149336 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e54005310 con 0x7f1e480039c0 2024-03-23T03:50:29.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.030+0000 7f1e6a200640 1 -- 172.21.15.102:0/187149336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e480039c0 msgr2=0x7f1e48003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:29.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.030+0000 7f1e6a200640 1 --2- 172.21.15.102:0/187149336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e480039c0 0x7f1e48003de0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f1e54001e50 tx=0x7f1e480057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.031+0000 7f1e6a200640 1 -- 172.21.15.102:0/187149336 shutdown_connections 2024-03-23T03:50:29.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.031+0000 7f1e6a200640 1 --2- 172.21.15.102:0/187149336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e480039c0 0x7f1e48003de0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.031+0000 7f1e6a200640 1 -- 172.21.15.102:0/187149336 >> 172.21.15.102:0/187149336 conn(0x7f1e640cf5d0 msgr2=0x7f1e640cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:29.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.031+0000 7f1e6a200640 1 -- 172.21.15.102:0/187149336 shutdown_connections 2024-03-23T03:50:29.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.031+0000 7f1e6a200640 1 -- 172.21.15.102:0/187149336 wait complete. 2024-03-23T03:50:29.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.032+0000 7f1e6a200640 1 Processor -- start 2024-03-23T03:50:29.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.032+0000 7f1e6a200640 1 -- start start 2024-03-23T03:50:29.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.033+0000 7f1e6a200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e480039c0 0x7f1e640cc130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:29.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.033+0000 7f1e6a200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1e640cb680 0x7f1e641f6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:29.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.033+0000 7f1e6a200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e640aaf10 0x7f1e640ab390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:29.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.033+0000 7f1e6a200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e640ab8d0 con 0x7f1e640aaf10 2024-03-23T03:50:29.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.033+0000 7f1e6a200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e640aa840 con 0x7f1e480039c0 2024-03-23T03:50:29.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.033+0000 7f1e6a200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e640a97e0 con 0x7f1e640cb680 2024-03-23T03:50:29.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.034+0000 7f1e63e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e640aaf10 0x7f1e640ab390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:29.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.034+0000 7f1e63e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e640aaf10 0x7f1e640ab390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42960/0 (socket says 172.21.15.102:42960) 2024-03-23T03:50:29.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.034+0000 7f1e63e00640 1 -- 172.21.15.102:0/80808267 learned_addr learned my addr 172.21.15.102:0/80808267 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:29.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.034+0000 7f1e5aa00640 1 --2- 172.21.15.102:0/80808267 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1e640cb680 0x7f1e641f6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:29.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.034+0000 7f1e63e00640 1 -- 172.21.15.102:0/80808267 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1e640cb680 msgr2=0x7f1e641f6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:29.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.034+0000 7f1e63e00640 1 --2- 172.21.15.102:0/80808267 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1e640cb680 0x7f1e641f6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.034+0000 7f1e63e00640 1 -- 172.21.15.102:0/80808267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e480039c0 msgr2=0x7f1e640cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:29.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.034+0000 7f1e63e00640 1 --2- 172.21.15.102:0/80808267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e480039c0 0x7f1e640cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.034+0000 7f1e63e00640 1 -- 172.21.15.102:0/80808267 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e641f6690 con 0x7f1e640aaf10 2024-03-23T03:50:29.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.034+0000 7f1e5aa00640 1 --2- 172.21.15.102:0/80808267 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1e640cb680 0x7f1e641f6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:29.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.034+0000 7f1e63e00640 1 --2- 172.21.15.102:0/80808267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e640aaf10 0x7f1e640ab390 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c000ff0 tx=0x7f1e5c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:29.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.035+0000 7f1e5be00640 1 -- 172.21.15.102:0/80808267 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c007440 con 0x7f1e640aaf10 2024-03-23T03:50:29.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.035+0000 7f1e5be00640 1 -- 172.21.15.102:0/80808267 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e5c0075a0 con 0x7f1e640aaf10 2024-03-23T03:50:29.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.035+0000 7f1e6a200640 1 -- 172.21.15.102:0/80808267 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e640a9a80 con 0x7f1e640aaf10 2024-03-23T03:50:29.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.035+0000 7f1e5be00640 1 -- 172.21.15.102:0/80808267 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c004a20 con 0x7f1e640aaf10 2024-03-23T03:50:29.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.035+0000 7f1e6a200640 1 -- 172.21.15.102:0/80808267 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e640aa040 con 0x7f1e640aaf10 2024-03-23T03:50:29.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.036+0000 7f1e6a200640 1 -- 172.21.15.102:0/80808267 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e64073390 con 0x7f1e640aaf10 2024-03-23T03:50:29.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.037+0000 7f1e5be00640 1 -- 172.21.15.102:0/80808267 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1e5c004b80 con 0x7f1e640aaf10 2024-03-23T03:50:29.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.038+0000 7f1e5be00640 1 --2- 172.21.15.102:0/80808267 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1e4006ac00 0x7f1e4006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:29.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.038+0000 7f1e5be00640 1 -- 172.21.15.102:0/80808267 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1e5c088e80 con 0x7f1e640aaf10 2024-03-23T03:50:29.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.038+0000 7f1e63400640 1 --2- 172.21.15.102:0/80808267 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1e4006ac00 0x7f1e4006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:29.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.039+0000 7f1e63400640 1 --2- 172.21.15.102:0/80808267 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1e4006ac00 0x7f1e4006d0c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1e54034000 tx=0x7f1e54030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:29.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.042+0000 7f1e5be00640 1 -- 172.21.15.102:0/80808267 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1e5c053f20 con 0x7f1e640aaf10 2024-03-23T03:50:29.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.223+0000 7f1e6a200640 1 -- 172.21.15.102:0/80808267 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1e640a9520 con 0x7f1e640aaf10 2024-03-23T03:50:29.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.224+0000 7f1e5be00640 1 -- 172.21.15.102:0/80808267 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1e5c057540 con 0x7f1e640aaf10 2024-03-23T03:50:29.224 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:50:29.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.226+0000 7f1e6a200640 1 -- 172.21.15.102:0/80808267 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1e4006ac00 msgr2=0x7f1e4006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:29.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.226+0000 7f1e6a200640 1 --2- 172.21.15.102:0/80808267 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1e4006ac00 0x7f1e4006d0c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1e54034000 tx=0x7f1e54030000 comp rx=0 tx=0).stop 2024-03-23T03:50:29.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.226+0000 7f1e6a200640 1 -- 172.21.15.102:0/80808267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e640aaf10 msgr2=0x7f1e640ab390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:29.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.226+0000 7f1e6a200640 1 --2- 172.21.15.102:0/80808267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e640aaf10 0x7f1e640ab390 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c000ff0 tx=0x7f1e5c00c890 comp rx=0 tx=0).stop 2024-03-23T03:50:29.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.227+0000 7f1e6a200640 1 -- 172.21.15.102:0/80808267 shutdown_connections 2024-03-23T03:50:29.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.227+0000 7f1e6a200640 1 --2- 172.21.15.102:0/80808267 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1e4006ac00 0x7f1e4006d0c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.227+0000 7f1e6a200640 1 --2- 172.21.15.102:0/80808267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e640aaf10 0x7f1e640ab390 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.227+0000 7f1e6a200640 1 --2- 172.21.15.102:0/80808267 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1e640cb680 0x7f1e641f6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.227+0000 7f1e6a200640 1 --2- 172.21.15.102:0/80808267 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e480039c0 0x7f1e640cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.227+0000 7f1e6a200640 1 -- 172.21.15.102:0/80808267 >> 172.21.15.102:0/80808267 conn(0x7f1e640cf5d0 msgr2=0x7f1e640b4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:29.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.227+0000 7f1e6a200640 1 -- 172.21.15.102:0/80808267 shutdown_connections 2024-03-23T03:50:29.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.227+0000 7f1e6a200640 1 -- 172.21.15.102:0/80808267 wait complete. 2024-03-23T03:50:29.244 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:50:29.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.435+0000 7fe6b9600640 1 Processor -- start 2024-03-23T03:50:29.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.436+0000 7fe6b9600640 1 -- start start 2024-03-23T03:50:29.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.437+0000 7fe6b9600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe6b40ce930 0x7fe6b40ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:29.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.437+0000 7fe6b9600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6b40cb660 0x7fe6b40cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:29.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.437+0000 7fe6b9600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b40cf250 con 0x7fe6b4070170 2024-03-23T03:50:29.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.437+0000 7fe6b9600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b415bae0 con 0x7fe6b40ce930 2024-03-23T03:50:29.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.437+0000 7fe6b9600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b415ae60 con 0x7fe6b40cb660 2024-03-23T03:50:29.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.437+0000 7fe6b3400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe6b4070170 0x7fe6b4070600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42432/0 (socket says 172.21.15.102:42432) 2024-03-23T03:50:29.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b3400640 1 -- 172.21.15.102:0/2421369047 learned_addr learned my addr 172.21.15.102:0/2421369047 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:29.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b2000640 1 --2- 172.21.15.102:0/2421369047 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6b40cb660 0x7fe6b40cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:29.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b2a00640 1 --2- 172.21.15.102:0/2421369047 >> v2:172.21.15.173:3300/0 conn(0x7fe6b40ce930 0x7fe6b40ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:29.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b1600640 1 -- 172.21.15.102:0/2421369047 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4285446280 0 0) 0x7fe6b40cf250 con 0x7fe6b4070170 2024-03-23T03:50:29.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b1600640 1 -- 172.21.15.102:0/2421369047 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe698003520 con 0x7fe6b4070170 2024-03-23T03:50:29.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b1600640 1 -- 172.21.15.102:0/2421369047 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1689012970 0 0) 0x7fe698003520 con 0x7fe6b4070170 2024-03-23T03:50:29.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b1600640 1 -- 172.21.15.102:0/2421369047 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6b40cb660 msgr2=0x7fe6b40cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:29.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b1600640 1 --2- 172.21.15.102:0/2421369047 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6b40cb660 0x7fe6b40cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b1600640 1 -- 172.21.15.102:0/2421369047 >> v2:172.21.15.173:3300/0 conn(0x7fe6b40ce930 msgr2=0x7fe6b40ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:29.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b1600640 1 --2- 172.21.15.102:0/2421369047 >> v2:172.21.15.173:3300/0 conn(0x7fe6b40ce930 0x7fe6b40ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b1600640 1 -- 172.21.15.102:0/2421369047 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b415b140 con 0x7fe6b4070170 2024-03-23T03:50:29.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b1600640 1 -- 172.21.15.102:0/2421369047 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fe6ac0019f0 con 0x7fe6b4070170 2024-03-23T03:50:29.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b1600640 1 -- 172.21.15.102:0/2421369047 >> v1:172.21.15.102:6789/0 conn(0x7fe6b4070170 legacy=0x7fe6b4070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:29.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b1600640 1 --2- 172.21.15.102:0/2421369047 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6b40cb660 0x7fe698003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:29.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b1600640 1 --2- 172.21.15.102:0/2421369047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe698003ef0 0x7fe6980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:29.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b1600640 1 --2- 172.21.15.102:0/2421369047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe698004d00 0x7fe698005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:29.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b1600640 1 -- 172.21.15.102:0/2421369047 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b415bae0 con 0x7fe698003ef0 2024-03-23T03:50:29.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b1600640 1 -- 172.21.15.102:0/2421369047 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b415ae60 con 0x7fe698004d00 2024-03-23T03:50:29.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.438+0000 7fe6b1600640 1 -- 172.21.15.102:0/2421369047 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe69800b980 con 0x7fe6b40cb660 2024-03-23T03:50:29.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.439+0000 7fe6b2000640 1 --2- 172.21.15.102:0/2421369047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe698003ef0 0x7fe6980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:29.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.439+0000 7fe6b2000640 1 -- 172.21.15.102:0/2421369047 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6b40cb660 msgr2=0x7fe698003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:29.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.439+0000 7fe6b2000640 1 --2- 172.21.15.102:0/2421369047 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6b40cb660 0x7fe698003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.439+0000 7fe6b2000640 1 -- 172.21.15.102:0/2421369047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe698004d00 msgr2=0x7fe698005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:29.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.439+0000 7fe6b2000640 1 --2- 172.21.15.102:0/2421369047 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe698004d00 0x7fe698005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.439+0000 7fe6b2000640 1 -- 172.21.15.102:0/2421369047 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe698003520 con 0x7fe698003ef0 2024-03-23T03:50:29.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.440+0000 7fe6b2000640 1 --2- 172.21.15.102:0/2421369047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe698003ef0 0x7fe6980047c0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0009ac0 tx=0x7fe6980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd98b8ad34a99e72 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:29.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.440+0000 7fe6b1600640 1 -- 172.21.15.102:0/2421369047 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a002c6d0 con 0x7fe698003ef0 2024-03-23T03:50:29.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.440+0000 7fe6b1600640 1 -- 172.21.15.102:0/2421369047 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a002c830 con 0x7fe698003ef0 2024-03-23T03:50:29.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.440+0000 7fe6b1600640 1 -- 172.21.15.102:0/2421369047 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6a002cbc0 con 0x7fe698003ef0 2024-03-23T03:50:29.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.441+0000 7fe6b9600640 1 -- 172.21.15.102:0/2421369047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe698003ef0 msgr2=0x7fe6980047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:29.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.441+0000 7fe6b9600640 1 --2- 172.21.15.102:0/2421369047 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe698003ef0 0x7fe6980047c0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fe6a0009ac0 tx=0x7fe6980057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.441+0000 7fe6b3400640 1 -- 172.21.15.102:0/2421369047 reap_dead start 2024-03-23T03:50:29.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.441+0000 7fe6b9600640 1 -- 172.21.15.102:0/2421369047 shutdown_connections 2024-03-23T03:50:29.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.441+0000 7fe6b9600640 1 -- 172.21.15.102:0/2421369047 >> 172.21.15.102:0/2421369047 conn(0x7fe6b406a0b0 msgr2=0x7fe6b406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:29.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.441+0000 7fe6b9600640 1 -- 172.21.15.102:0/2421369047 shutdown_connections 2024-03-23T03:50:29.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.441+0000 7fe6b9600640 1 -- 172.21.15.102:0/2421369047 wait complete. 2024-03-23T03:50:29.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.443+0000 7fe6b9600640 1 Processor -- start 2024-03-23T03:50:29.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.443+0000 7fe6b9600640 1 -- start start 2024-03-23T03:50:29.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.444+0000 7fe6b9600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe698003ef0 0x7fe6b40aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:29.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.444+0000 7fe6b9600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6b40af200 0x7fe6b4145370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:29.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.444+0000 7fe6b9600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6b41458b0 0x7fe6b4145d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:29.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.444+0000 7fe6b9600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b4159580 con 0x7fe6b41458b0 2024-03-23T03:50:29.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.444+0000 7fe6b9600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b40af680 con 0x7fe698003ef0 2024-03-23T03:50:29.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.444+0000 7fe6b9600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b4146ab0 con 0x7fe6b40af200 2024-03-23T03:50:29.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.444+0000 7fe6b3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6b41458b0 0x7fe6b4145d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:29.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.444+0000 7fe6b3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6b41458b0 0x7fe6b4145d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42968/0 (socket says 172.21.15.102:42968) 2024-03-23T03:50:29.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.444+0000 7fe6b3400640 1 -- 172.21.15.102:0/2850194471 learned_addr learned my addr 172.21.15.102:0/2850194471 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:29.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.444+0000 7fe6b3400640 1 -- 172.21.15.102:0/2850194471 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6b40af200 msgr2=0x7fe6b4145370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:29.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.444+0000 7fe6b3400640 1 --2- 172.21.15.102:0/2850194471 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6b40af200 0x7fe6b4145370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.444+0000 7fe6b3400640 1 -- 172.21.15.102:0/2850194471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe698003ef0 msgr2=0x7fe6b40aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:29.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.444+0000 7fe6b3400640 1 --2- 172.21.15.102:0/2850194471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe698003ef0 0x7fe6b40aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.444+0000 7fe6b3400640 1 -- 172.21.15.102:0/2850194471 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b4155280 con 0x7fe6b41458b0 2024-03-23T03:50:29.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.444+0000 7fe6b3400640 1 --2- 172.21.15.102:0/2850194471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6b41458b0 0x7fe6b4145d30 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40ae160 tx=0x7fe6ac00cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:29.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.445+0000 7fe6aaa00640 1 -- 172.21.15.102:0/2850194471 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac00a6d0 con 0x7fe6b41458b0 2024-03-23T03:50:29.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.445+0000 7fe6aaa00640 1 -- 172.21.15.102:0/2850194471 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ac00a830 con 0x7fe6b41458b0 2024-03-23T03:50:29.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.445+0000 7fe6aaa00640 1 -- 172.21.15.102:0/2850194471 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ac00aba0 con 0x7fe6b41458b0 2024-03-23T03:50:29.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.445+0000 7fe6b9600640 1 -- 172.21.15.102:0/2850194471 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b4146270 con 0x7fe6b41458b0 2024-03-23T03:50:29.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.445+0000 7fe6b9600640 1 -- 172.21.15.102:0/2850194471 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b41467d0 con 0x7fe6b41458b0 2024-03-23T03:50:29.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.446+0000 7fe6b9600640 1 -- 172.21.15.102:0/2850194471 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b41596c0 con 0x7fe6b41458b0 2024-03-23T03:50:29.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.447+0000 7fe6aaa00640 1 -- 172.21.15.102:0/2850194471 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe6ac00ad00 con 0x7fe6b41458b0 2024-03-23T03:50:29.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.448+0000 7fe6aaa00640 1 --2- 172.21.15.102:0/2850194471 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe69406ac00 0x7fe69406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:29.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.448+0000 7fe6aaa00640 1 -- 172.21.15.102:0/2850194471 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe6ac0889e0 con 0x7fe6b41458b0 2024-03-23T03:50:29.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.448+0000 7fe6b2a00640 1 --2- 172.21.15.102:0/2850194471 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe69406ac00 0x7fe69406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:29.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.449+0000 7fe6b2a00640 1 --2- 172.21.15.102:0/2850194471 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe69406ac00 0x7fe69406d0c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4006a90 tx=0x7fe6a4006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:29.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.453+0000 7fe6aaa00640 1 -- 172.21.15.102:0/2850194471 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe6b41596c0 con 0x7fe6b41458b0 2024-03-23T03:50:29.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.659+0000 7fe6b9600640 1 -- 172.21.15.102:0/2850194471 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe6b41596c0 con 0x7fe6b41458b0 2024-03-23T03:50:29.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.660+0000 7fe6aaa00640 1 -- 172.21.15.102:0/2850194471 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v81) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe6b41596c0 con 0x7fe6b41458b0 2024-03-23T03:50:29.660 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:50:29.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.662+0000 7fe6b9600640 1 -- 172.21.15.102:0/2850194471 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe69406ac00 msgr2=0x7fe69406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.662+0000 7fe6b9600640 1 --2- 172.21.15.102:0/2850194471 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe69406ac00 0x7fe69406d0c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe6a4006a90 tx=0x7fe6a4006a20 comp rx=0 tx=0).stop 2024-03-23T03:50:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.662+0000 7fe6b9600640 1 -- 172.21.15.102:0/2850194471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6b41458b0 msgr2=0x7fe6b4145d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.662+0000 7fe6b9600640 1 --2- 172.21.15.102:0/2850194471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6b41458b0 0x7fe6b4145d30 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fe6b40ae160 tx=0x7fe6ac00cf00 comp rx=0 tx=0).stop 2024-03-23T03:50:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.663+0000 7fe6b9600640 1 -- 172.21.15.102:0/2850194471 shutdown_connections 2024-03-23T03:50:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.663+0000 7fe6b9600640 1 --2- 172.21.15.102:0/2850194471 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe69406ac00 0x7fe69406d0c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.663+0000 7fe6b9600640 1 --2- 172.21.15.102:0/2850194471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6b41458b0 0x7fe6b4145d30 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.663+0000 7fe6b9600640 1 --2- 172.21.15.102:0/2850194471 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe6b40af200 0x7fe6b4145370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.663+0000 7fe6b9600640 1 --2- 172.21.15.102:0/2850194471 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe698003ef0 0x7fe6b40aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.663+0000 7fe6b9600640 1 -- 172.21.15.102:0/2850194471 >> 172.21.15.102:0/2850194471 conn(0x7fe6b406a0b0 msgr2=0x7fe6b415bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.663+0000 7fe6b9600640 1 -- 172.21.15.102:0/2850194471 shutdown_connections 2024-03-23T03:50:29.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:29.663+0000 7fe6b9600640 1 -- 172.21.15.102:0/2850194471 wait complete. 2024-03-23T03:50:29.680 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-23T03:50:29.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:29.680 DEBUG:teuthology.orchestra.run.smithi102:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-23T03:50:29.781 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:50:29.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:29.782 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T03:50:29.828 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:50:29.828 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:29.882 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:50:29.883 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:29.932 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:50:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.150+0000 7fa949000640 1 Processor -- start 2024-03-23T03:50:30.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.150+0000 7fa949000640 1 -- start start 2024-03-23T03:50:30.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.151+0000 7fa949000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9440ccec0 0x7fa9440cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:30.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.152+0000 7fa949000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa94406da70 con 0x7fa9440b37f0 2024-03-23T03:50:30.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.152+0000 7fa949000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9440b1d00 con 0x7fa94414aa40 2024-03-23T03:50:30.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.152+0000 7fa949000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9440b1fe0 con 0x7fa9440ccec0 2024-03-23T03:50:30.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.152+0000 7fa941400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa9440b37f0 0x7fa94406c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42440/0 (socket says 172.21.15.102:42440) 2024-03-23T03:50:30.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.152+0000 7fa941400640 1 -- 172.21.15.102:0/859605977 learned_addr learned my addr 172.21.15.102:0/859605977 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:30.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.152+0000 7fa940a00640 1 -- 172.21.15.102:0/859605977 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137288759 0 0) 0x7fa94406da70 con 0x7fa9440b37f0 2024-03-23T03:50:30.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.152+0000 7fa942800640 1 --2- 172.21.15.102:0/859605977 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9440ccec0 0x7fa9440cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:30.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.152+0000 7fa940a00640 1 -- 172.21.15.102:0/859605977 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa928003520 con 0x7fa9440b37f0 2024-03-23T03:50:30.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.153+0000 7fa940a00640 1 -- 172.21.15.102:0/859605977 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 456000088 0 0) 0x7fa928003520 con 0x7fa9440b37f0 2024-03-23T03:50:30.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.153+0000 7fa940a00640 1 -- 172.21.15.102:0/859605977 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9440ccec0 msgr2=0x7fa9440cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:30.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.153+0000 7fa940a00640 1 --2- 172.21.15.102:0/859605977 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9440ccec0 0x7fa9440cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.153+0000 7fa940a00640 1 -- 172.21.15.102:0/859605977 >> v1:172.21.15.173:6789/0 conn(0x7fa94414aa40 legacy=0x7fa9440cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:30.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.153+0000 7fa940a00640 1 -- 172.21.15.102:0/859605977 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9440b22c0 con 0x7fa9440b37f0 2024-03-23T03:50:30.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.153+0000 7fa940a00640 1 -- 172.21.15.102:0/859605977 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fa930001670 con 0x7fa9440b37f0 2024-03-23T03:50:30.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.153+0000 7fa940a00640 1 -- 172.21.15.102:0/859605977 >> v1:172.21.15.102:6789/0 conn(0x7fa9440b37f0 legacy=0x7fa94406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:30.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.153+0000 7fa940a00640 1 --2- 172.21.15.102:0/859605977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9280039a0 0x7fa928003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:30.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.153+0000 7fa940a00640 1 --2- 172.21.15.102:0/859605977 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9440ccec0 0x7fa9280047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:30.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.153+0000 7fa940a00640 1 --2- 172.21.15.102:0/859605977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa928004d00 0x7fa928005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:30.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.153+0000 7fa940a00640 1 -- 172.21.15.102:0/859605977 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9440b1fe0 con 0x7fa9280039a0 2024-03-23T03:50:30.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.153+0000 7fa940a00640 1 -- 172.21.15.102:0/859605977 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92800bb10 con 0x7fa928004d00 2024-03-23T03:50:30.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.153+0000 7fa940a00640 1 -- 172.21.15.102:0/859605977 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92800bca0 con 0x7fa9440ccec0 2024-03-23T03:50:30.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.153+0000 7fa941e00640 1 --2- 172.21.15.102:0/859605977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9280039a0 0x7fa928003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:30.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.153+0000 7fa941400640 1 --2- 172.21.15.102:0/859605977 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9440ccec0 0x7fa9280047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:30.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.154+0000 7fa942800640 1 --2- 172.21.15.102:0/859605977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa928004d00 0x7fa928005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:30.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.154+0000 7fa941e00640 1 -- 172.21.15.102:0/859605977 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9440ccec0 msgr2=0x7fa9280047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:30.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.154+0000 7fa941e00640 1 --2- 172.21.15.102:0/859605977 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9440ccec0 0x7fa9280047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.154+0000 7fa941e00640 1 -- 172.21.15.102:0/859605977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa928004d00 msgr2=0x7fa928005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:30.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.154+0000 7fa941e00640 1 --2- 172.21.15.102:0/859605977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa928004d00 0x7fa928005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.154+0000 7fa941e00640 1 -- 172.21.15.102:0/859605977 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa928003520 con 0x7fa9280039a0 2024-03-23T03:50:30.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.154+0000 7fa942800640 1 --2- 172.21.15.102:0/859605977 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa928004d00 0x7fa928005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:30.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.155+0000 7fa941e00640 1 --2- 172.21.15.102:0/859605977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9280039a0 0x7fa928003dc0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fa934001780 tx=0x7fa9280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3990ec41cccacf6d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:30.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.155+0000 7fa940a00640 1 -- 172.21.15.102:0/859605977 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9340052d0 con 0x7fa9280039a0 2024-03-23T03:50:30.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.155+0000 7fa940a00640 1 -- 172.21.15.102:0/859605977 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93402e050 con 0x7fa9280039a0 2024-03-23T03:50:30.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.155+0000 7fa940a00640 1 -- 172.21.15.102:0/859605977 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9340035f0 con 0x7fa9280039a0 2024-03-23T03:50:30.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.156+0000 7fa949000640 1 -- 172.21.15.102:0/859605977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9280039a0 msgr2=0x7fa928003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:30.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.156+0000 7fa949000640 1 --2- 172.21.15.102:0/859605977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9280039a0 0x7fa928003dc0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fa934001780 tx=0x7fa9280057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.156+0000 7fa942800640 1 -- 172.21.15.102:0/859605977 reap_dead start 2024-03-23T03:50:30.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.156+0000 7fa949000640 1 -- 172.21.15.102:0/859605977 shutdown_connections 2024-03-23T03:50:30.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.156+0000 7fa949000640 1 -- 172.21.15.102:0/859605977 >> 172.21.15.102:0/859605977 conn(0x7fa94406a3d0 msgr2=0x7fa9440b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:30.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.156+0000 7fa949000640 1 -- 172.21.15.102:0/859605977 shutdown_connections 2024-03-23T03:50:30.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.156+0000 7fa949000640 1 -- 172.21.15.102:0/859605977 wait complete. 2024-03-23T03:50:30.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.157+0000 7fa949000640 1 Processor -- start 2024-03-23T03:50:30.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.158+0000 7fa949000640 1 -- start start 2024-03-23T03:50:30.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.158+0000 7fa949000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9280039a0 0x7fa944143700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:30.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.158+0000 7fa949000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa944144d80 0x7fa944160510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:30.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.158+0000 7fa949000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9440af9e0 0x7fa94415f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:30.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.159+0000 7fa949000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa944143ce0 con 0x7fa9440af9e0 2024-03-23T03:50:30.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.159+0000 7fa949000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa944143fc0 con 0x7fa944144d80 2024-03-23T03:50:30.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.159+0000 7fa949000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9441442a0 con 0x7fa9280039a0 2024-03-23T03:50:30.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.159+0000 7fa942800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9440af9e0 0x7fa94415f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.159+0000 7fa942800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9440af9e0 0x7fa94415f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42974/0 (socket says 172.21.15.102:42974) 2024-03-23T03:50:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.159+0000 7fa942800640 1 -- 172.21.15.102:0/3030987249 learned_addr learned my addr 172.21.15.102:0/3030987249 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.159+0000 7fa941400640 1 --2- 172.21.15.102:0/3030987249 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa944144d80 0x7fa944160510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.159+0000 7fa941e00640 1 --2- 172.21.15.102:0/3030987249 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9280039a0 0x7fa944143700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.159+0000 7fa942800640 1 -- 172.21.15.102:0/3030987249 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9280039a0 msgr2=0x7fa944143700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.159+0000 7fa942800640 1 --2- 172.21.15.102:0/3030987249 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9280039a0 0x7fa944143700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.159+0000 7fa942800640 1 -- 172.21.15.102:0/3030987249 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa944144d80 msgr2=0x7fa944160510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.159+0000 7fa942800640 1 --2- 172.21.15.102:0/3030987249 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa944144d80 0x7fa944160510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.159+0000 7fa942800640 1 -- 172.21.15.102:0/3030987249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa944144580 con 0x7fa9440af9e0 2024-03-23T03:50:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.159+0000 7fa941e00640 1 --2- 172.21.15.102:0/3030987249 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9280039a0 0x7fa944143700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.159+0000 7fa941400640 1 --2- 172.21.15.102:0/3030987249 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa944144d80 0x7fa944160510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.160+0000 7fa942800640 1 --2- 172.21.15.102:0/3030987249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9440af9e0 0x7fa94415f990 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fa9440b1120 tx=0x7fa93c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:30.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.160+0000 7fa93a000640 1 -- 172.21.15.102:0/3030987249 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c007cf0 con 0x7fa9440af9e0 2024-03-23T03:50:30.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.160+0000 7fa93a000640 1 -- 172.21.15.102:0/3030987249 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93c007e70 con 0x7fa9440af9e0 2024-03-23T03:50:30.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.160+0000 7fa93a000640 1 -- 172.21.15.102:0/3030987249 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa93c014620 con 0x7fa9440af9e0 2024-03-23T03:50:30.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.160+0000 7fa949000640 1 -- 172.21.15.102:0/3030987249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa944144860 con 0x7fa9440af9e0 2024-03-23T03:50:30.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.160+0000 7fa949000640 1 -- 172.21.15.102:0/3030987249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9440afe70 con 0x7fa9440af9e0 2024-03-23T03:50:30.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.161+0000 7fa949000640 1 -- 172.21.15.102:0/3030987249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9440b00f0 con 0x7fa9440af9e0 2024-03-23T03:50:30.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.162+0000 7fa93a000640 1 -- 172.21.15.102:0/3030987249 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa93c01d3b0 con 0x7fa9440af9e0 2024-03-23T03:50:30.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.163+0000 7fa93a000640 1 --2- 172.21.15.102:0/3030987249 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa92406ac00 0x7fa92406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:30.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.163+0000 7fa93a000640 1 -- 172.21.15.102:0/3030987249 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa93c089270 con 0x7fa9440af9e0 2024-03-23T03:50:30.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.163+0000 7fa941e00640 1 --2- 172.21.15.102:0/3030987249 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa92406ac00 0x7fa92406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:30.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.164+0000 7fa941e00640 1 --2- 172.21.15.102:0/3030987249 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa92406ac00 0x7fa92406d0c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa934033000 tx=0x7fa934009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:30.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.168+0000 7fa93a000640 1 -- 172.21.15.102:0/3030987249 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa93c054300 con 0x7fa9440af9e0 2024-03-23T03:50:30.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.365+0000 7fa949000640 1 -- 172.21.15.102:0/3030987249 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa94406ac20 con 0x7fa9440af9e0 2024-03-23T03:50:30.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.366+0000 7fa93a000640 1 -- 172.21.15.102:0/3030987249 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16357 (secure 0 0 0) 0x7fa93c057920 con 0x7fa9440af9e0 2024-03-23T03:50:30.367 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:30.367 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":81,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:08.942399+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:50:06.347556+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-23T03:50:06.924370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:50:30.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.369+0000 7fa949000640 1 -- 172.21.15.102:0/3030987249 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa92406ac00 msgr2=0x7fa92406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:30.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.369+0000 7fa949000640 1 --2- 172.21.15.102:0/3030987249 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa92406ac00 0x7fa92406d0c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fa934033000 tx=0x7fa934009030 comp rx=0 tx=0).stop 2024-03-23T03:50:30.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.369+0000 7fa949000640 1 -- 172.21.15.102:0/3030987249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9440af9e0 msgr2=0x7fa94415f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:30.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.369+0000 7fa949000640 1 --2- 172.21.15.102:0/3030987249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9440af9e0 0x7fa94415f990 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fa9440b1120 tx=0x7fa93c009da0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.369+0000 7fa949000640 1 -- 172.21.15.102:0/3030987249 shutdown_connections 2024-03-23T03:50:30.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.369+0000 7fa949000640 1 --2- 172.21.15.102:0/3030987249 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa92406ac00 0x7fa92406d0c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.369+0000 7fa949000640 1 --2- 172.21.15.102:0/3030987249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9440af9e0 0x7fa94415f990 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.369+0000 7fa949000640 1 --2- 172.21.15.102:0/3030987249 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa944144d80 0x7fa944160510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.369+0000 7fa949000640 1 --2- 172.21.15.102:0/3030987249 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa9280039a0 0x7fa944143700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.369+0000 7fa949000640 1 -- 172.21.15.102:0/3030987249 >> 172.21.15.102:0/3030987249 conn(0x7fa94406a3d0 msgr2=0x7fa944149850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:30.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.369+0000 7fa949000640 1 -- 172.21.15.102:0/3030987249 shutdown_connections 2024-03-23T03:50:30.371 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.369+0000 7fa949000640 1 -- 172.21.15.102:0/3030987249 wait complete. 2024-03-23T03:50:30.390 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:50:06.347556+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:50:06.924370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:50:30.390 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:50:30.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.592+0000 7f3f87c00640 1 Processor -- start 2024-03-23T03:50:30.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.592+0000 7f3f87c00640 1 -- start start 2024-03-23T03:50:30.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.594+0000 7f3f87c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3f800add30 0x7f3f800ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:30.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.594+0000 7f3f87c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3f8006ce90 0x7f3f8006d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:30.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.594+0000 7f3f87c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f800cf250 con 0x7f3f800ce930 2024-03-23T03:50:30.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.594+0000 7f3f87c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f800ae1c0 con 0x7f3f800add30 2024-03-23T03:50:30.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.594+0000 7f3f87c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8006c760 con 0x7f3f8006ce90 2024-03-23T03:50:30.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.594+0000 7f3f7fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3f800add30 0x7f3f800ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:30.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.594+0000 7f3f7fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3f800add30 0x7f3f800ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42980/0 (socket says 172.21.15.102:42980) 2024-03-23T03:50:30.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.594+0000 7f3f7fe00640 1 -- 172.21.15.102:0/1404964575 learned_addr learned my addr 172.21.15.102:0/1404964575 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:30.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.594+0000 7f3f85c00640 1 --2- 172.21.15.102:0/1404964575 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3f8006ce90 0x7f3f8006d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:30.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.595+0000 7f3f7fe00640 1 -- 172.21.15.102:0/1404964575 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3f8006ce90 msgr2=0x7f3f8006d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:30.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.595+0000 7f3f7fe00640 1 --2- 172.21.15.102:0/1404964575 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3f8006ce90 0x7f3f8006d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.595+0000 7f3f7fe00640 1 -- 172.21.15.102:0/1404964575 >> v1:172.21.15.173:6789/0 conn(0x7f3f800ce930 legacy=0x7f3f800cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:30.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.595+0000 7f3f7fe00640 1 -- 172.21.15.102:0/1404964575 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f800cdba0 con 0x7f3f800add30 2024-03-23T03:50:30.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.595+0000 7f3f85c00640 1 --2- 172.21.15.102:0/1404964575 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3f8006ce90 0x7f3f8006d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:30.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.596+0000 7f3f7fe00640 1 --2- 172.21.15.102:0/1404964575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f800add30 0x7f3f800ced10 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c009870 tx=0x7f3f6c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7171c70203f478d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:30.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.596+0000 7f3f7f400640 1 -- 172.21.15.102:0/1404964575 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c031640 con 0x7f3f800add30 2024-03-23T03:50:30.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.597+0000 7f3f7f400640 1 -- 172.21.15.102:0/1404964575 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6c0317a0 con 0x7f3f800add30 2024-03-23T03:50:30.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.597+0000 7f3f7f400640 1 -- 172.21.15.102:0/1404964575 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c031b30 con 0x7f3f800add30 2024-03-23T03:50:30.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.597+0000 7f3f87c00640 1 -- 172.21.15.102:0/1404964575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f800add30 msgr2=0x7f3f800ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:30.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.597+0000 7f3f87c00640 1 --2- 172.21.15.102:0/1404964575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f800add30 0x7f3f800ced10 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c009870 tx=0x7f3f6c02b8f0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.597+0000 7f3f87c00640 1 -- 172.21.15.102:0/1404964575 shutdown_connections 2024-03-23T03:50:30.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.597+0000 7f3f87c00640 1 --2- 172.21.15.102:0/1404964575 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3f8006ce90 0x7f3f8006d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.597+0000 7f3f87c00640 1 --2- 172.21.15.102:0/1404964575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f800add30 0x7f3f800ced10 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.597+0000 7f3f87c00640 1 -- 172.21.15.102:0/1404964575 >> 172.21.15.102:0/1404964575 conn(0x7f3f8006a0b0 msgr2=0x7f3f8006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:30.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.597+0000 7f3f87c00640 1 -- 172.21.15.102:0/1404964575 shutdown_connections 2024-03-23T03:50:30.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.598+0000 7f3f87c00640 1 -- 172.21.15.102:0/1404964575 wait complete. 2024-03-23T03:50:30.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.599+0000 7f3f87c00640 1 Processor -- start 2024-03-23T03:50:30.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.599+0000 7f3f87c00640 1 -- start start 2024-03-23T03:50:30.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.600+0000 7f3f87c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f8006ce90 0x7f3f8015a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:30.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.600+0000 7f3f87c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f800add30 0x7f3f80159050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:30.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.600+0000 7f3f87c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3f800ce930 0x7f3f80154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:30.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.600+0000 7f3f87c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8014db10 con 0x7f3f800add30 2024-03-23T03:50:30.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.600+0000 7f3f87c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f80155c00 con 0x7f3f8006ce90 2024-03-23T03:50:30.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.600+0000 7f3f87c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8006de60 con 0x7f3f800ce930 2024-03-23T03:50:30.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.600+0000 7f3f7fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f800add30 0x7f3f80159050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:30.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.601+0000 7f3f7fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f800add30 0x7f3f80159050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42996/0 (socket says 172.21.15.102:42996) 2024-03-23T03:50:30.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.601+0000 7f3f7fe00640 1 -- 172.21.15.102:0/67437629 learned_addr learned my addr 172.21.15.102:0/67437629 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:30.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.601+0000 7f3f85c00640 1 --2- 172.21.15.102:0/67437629 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3f800ce930 0x7f3f80154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:30.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.601+0000 7f3f7fe00640 1 -- 172.21.15.102:0/67437629 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3f800ce930 msgr2=0x7f3f80154a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:30.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.601+0000 7f3f7fe00640 1 --2- 172.21.15.102:0/67437629 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3f800ce930 0x7f3f80154a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.601+0000 7f3f85200640 1 --2- 172.21.15.102:0/67437629 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f8006ce90 0x7f3f8015a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:30.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.601+0000 7f3f7fe00640 1 -- 172.21.15.102:0/67437629 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f8006ce90 msgr2=0x7f3f8015a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:30.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.601+0000 7f3f7fe00640 1 --2- 172.21.15.102:0/67437629 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f8006ce90 0x7f3f8015a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.601+0000 7f3f7fe00640 1 -- 172.21.15.102:0/67437629 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6c0094d0 con 0x7f3f800add30 2024-03-23T03:50:30.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.601+0000 7f3f7fe00640 1 --2- 172.21.15.102:0/67437629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f800add30 0x7f3f80159050 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c000c00 tx=0x7f3f6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:30.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.601+0000 7f3f7cc00640 1 -- 172.21.15.102:0/67437629 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c03ac60 con 0x7f3f800add30 2024-03-23T03:50:30.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.602+0000 7f3f7cc00640 1 -- 172.21.15.102:0/67437629 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f6c03ade0 con 0x7f3f800add30 2024-03-23T03:50:30.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.602+0000 7f3f7cc00640 1 -- 172.21.15.102:0/67437629 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c044630 con 0x7f3f800add30 2024-03-23T03:50:30.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.602+0000 7f3f87c00640 1 -- 172.21.15.102:0/67437629 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f801535f0 con 0x7f3f800add30 2024-03-23T03:50:30.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.602+0000 7f3f87c00640 1 -- 172.21.15.102:0/67437629 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f80155920 con 0x7f3f800add30 2024-03-23T03:50:30.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.603+0000 7f3f87c00640 1 -- 172.21.15.102:0/67437629 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f80153430 con 0x7f3f800add30 2024-03-23T03:50:30.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.604+0000 7f3f7cc00640 1 -- 172.21.15.102:0/67437629 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3f6c02f050 con 0x7f3f800add30 2024-03-23T03:50:30.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.604+0000 7f3f7cc00640 1 --2- 172.21.15.102:0/67437629 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3f6006ac30 0x7f3f6006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:30.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.605+0000 7f3f7cc00640 1 -- 172.21.15.102:0/67437629 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3f6c0ab5f0 con 0x7f3f800add30 2024-03-23T03:50:30.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.605+0000 7f3f85200640 1 --2- 172.21.15.102:0/67437629 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3f6006ac30 0x7f3f6006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:30.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.605+0000 7f3f85200640 1 --2- 172.21.15.102:0/67437629 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3f6006ac30 0x7f3f6006d0f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f3f7400a750 tx=0x7f3f74008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:30.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.609+0000 7f3f7cc00640 1 -- 172.21.15.102:0/67437629 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3f6c076680 con 0x7f3f800add30 2024-03-23T03:50:30.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.804+0000 7f3f87c00640 1 -- 172.21.15.102:0/67437629 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3f80073390 con 0x7f3f800add30 2024-03-23T03:50:30.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.805+0000 7f3f7cc00640 1 -- 172.21.15.102:0/67437629 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3f6c079ca0 con 0x7f3f800add30 2024-03-23T03:50:30.806 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:50:30.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.808+0000 7f3f87c00640 1 -- 172.21.15.102:0/67437629 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3f6006ac30 msgr2=0x7f3f6006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:30.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.808+0000 7f3f87c00640 1 --2- 172.21.15.102:0/67437629 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3f6006ac30 0x7f3f6006d0f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f3f7400a750 tx=0x7f3f74008040 comp rx=0 tx=0).stop 2024-03-23T03:50:30.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.808+0000 7f3f87c00640 1 -- 172.21.15.102:0/67437629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f800add30 msgr2=0x7f3f80159050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:30.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.808+0000 7f3f87c00640 1 --2- 172.21.15.102:0/67437629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f800add30 0x7f3f80159050 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c000c00 tx=0x7f3f6c0046c0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.808+0000 7f3f87c00640 1 -- 172.21.15.102:0/67437629 shutdown_connections 2024-03-23T03:50:30.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.808+0000 7f3f87c00640 1 --2- 172.21.15.102:0/67437629 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3f6006ac30 0x7f3f6006d0f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.808+0000 7f3f87c00640 1 --2- 172.21.15.102:0/67437629 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3f800ce930 0x7f3f80154a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.808+0000 7f3f87c00640 1 --2- 172.21.15.102:0/67437629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f800add30 0x7f3f80159050 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.808+0000 7f3f87c00640 1 --2- 172.21.15.102:0/67437629 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f8006ce90 0x7f3f8015a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:30.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.808+0000 7f3f87c00640 1 -- 172.21.15.102:0/67437629 >> 172.21.15.102:0/67437629 conn(0x7f3f8006a0b0 msgr2=0x7f3f800cbc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:30.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.808+0000 7f3f87c00640 1 -- 172.21.15.102:0/67437629 shutdown_connections 2024-03-23T03:50:30.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:30.808+0000 7f3f87c00640 1 -- 172.21.15.102:0/67437629 wait complete. 2024-03-23T03:50:30.828 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:50:31.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.037+0000 7fd5a6400640 1 Processor -- start 2024-03-23T03:50:31.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.037+0000 7fd5a6400640 1 -- start start 2024-03-23T03:50:31.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.039+0000 7fd5a6400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd5a00cf040 0x7fd5a0164920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.039+0000 7fd5a6400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a0165830 con 0x7fd5a0160d20 2024-03-23T03:50:31.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.039+0000 7fd5a6400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5a0164f90 con 0x7fd5a00ca7b0 2024-03-23T03:50:31.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.039+0000 7fd5a6400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5a0165270 con 0x7fd5a00cf040 2024-03-23T03:50:31.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.039+0000 7fd59f400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd5a0160d20 0x7fd5a00ce930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42448/0 (socket says 172.21.15.102:42448) 2024-03-23T03:50:31.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.039+0000 7fd59f400640 1 -- 172.21.15.102:0/3582378188 learned_addr learned my addr 172.21.15.102:0/3582378188 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:31.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.039+0000 7fd59ea00640 1 --2- 172.21.15.102:0/3582378188 >> v2:172.21.15.102:3300/0 conn(0x7fd5a00cf040 0x7fd5a0164920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.039+0000 7fd59e000640 1 -- 172.21.15.102:0/3582378188 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477799441 0 0) 0x7fd5a0165830 con 0x7fd5a0160d20 2024-03-23T03:50:31.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.040+0000 7fd59e000640 1 -- 172.21.15.102:0/3582378188 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd584003520 con 0x7fd5a0160d20 2024-03-23T03:50:31.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.040+0000 7fd59ea00640 1 -- 172.21.15.102:0/3582378188 >> v1:172.21.15.173:6789/0 conn(0x7fd5a00ca7b0 legacy=0x7fd5a00b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:31.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.040+0000 7fd59ea00640 1 -- 172.21.15.102:0/3582378188 >> v1:172.21.15.102:6789/0 conn(0x7fd5a0160d20 legacy=0x7fd5a00ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:31.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.040+0000 7fd59ea00640 1 -- 172.21.15.102:0/3582378188 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a0165550 con 0x7fd5a00cf040 2024-03-23T03:50:31.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.041+0000 7fd59ea00640 1 --2- 172.21.15.102:0/3582378188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a00cf040 0x7fd5a0164920 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0097a0 tx=0x7fd58c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2abfb65cf5bcfd08 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:31.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.041+0000 7fd59e000640 1 -- 172.21.15.102:0/3582378188 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c02e070 con 0x7fd5a00cf040 2024-03-23T03:50:31.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.041+0000 7fd59e000640 1 -- 172.21.15.102:0/3582378188 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58c0315a0 con 0x7fd5a00cf040 2024-03-23T03:50:31.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.041+0000 7fd59e000640 1 -- 172.21.15.102:0/3582378188 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58c031910 con 0x7fd5a00cf040 2024-03-23T03:50:31.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.042+0000 7fd5a6400640 1 -- 172.21.15.102:0/3582378188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a00cf040 msgr2=0x7fd5a0164920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:31.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.042+0000 7fd5a6400640 1 --2- 172.21.15.102:0/3582378188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a00cf040 0x7fd5a0164920 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0097a0 tx=0x7fd58c02b840 comp rx=0 tx=0).stop 2024-03-23T03:50:31.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.042+0000 7fd5a6400640 1 -- 172.21.15.102:0/3582378188 shutdown_connections 2024-03-23T03:50:31.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.042+0000 7fd5a6400640 1 --2- 172.21.15.102:0/3582378188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a00cf040 0x7fd5a0164920 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.042+0000 7fd5a6400640 1 -- 172.21.15.102:0/3582378188 >> 172.21.15.102:0/3582378188 conn(0x7fd5a006a0b0 msgr2=0x7fd5a006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:31.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.042+0000 7fd5a6400640 1 -- 172.21.15.102:0/3582378188 shutdown_connections 2024-03-23T03:50:31.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.042+0000 7fd5a6400640 1 -- 172.21.15.102:0/3582378188 wait complete. 2024-03-23T03:50:31.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.044+0000 7fd5a6400640 1 Processor -- start 2024-03-23T03:50:31.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.044+0000 7fd5a6400640 1 -- start start 2024-03-23T03:50:31.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.045+0000 7fd5a6400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a00ca7b0 0x7fd5a015d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.045+0000 7fd5a6400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5a00cf040 0x7fd5a014e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.045+0000 7fd5a6400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5a0160d20 0x7fd5a015f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.045+0000 7fd5a6400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a00b0540 con 0x7fd5a00ca7b0 2024-03-23T03:50:31.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.045+0000 7fd5a6400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a00af8c0 con 0x7fd5a0160d20 2024-03-23T03:50:31.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.045+0000 7fd5a6400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5a00afba0 con 0x7fd5a00cf040 2024-03-23T03:50:31.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.045+0000 7fd59f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a00ca7b0 0x7fd5a015d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.045+0000 7fd59f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a00ca7b0 0x7fd5a015d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43012/0 (socket says 172.21.15.102:43012) 2024-03-23T03:50:31.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.045+0000 7fd59f400640 1 -- 172.21.15.102:0/1361761220 learned_addr learned my addr 172.21.15.102:0/1361761220 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:31.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.045+0000 7fd59f400640 1 -- 172.21.15.102:0/1361761220 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5a00cf040 msgr2=0x7fd5a014e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:31.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.045+0000 7fd59f400640 1 --2- 172.21.15.102:0/1361761220 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5a00cf040 0x7fd5a014e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.045+0000 7fd59f400640 1 -- 172.21.15.102:0/1361761220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5a0160d20 msgr2=0x7fd5a015f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:31.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.045+0000 7fd59fe00640 1 --2- 172.21.15.102:0/1361761220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5a0160d20 0x7fd5a015f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.045+0000 7fd59f400640 1 --2- 172.21.15.102:0/1361761220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5a0160d20 0x7fd5a015f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.045+0000 7fd59f400640 1 -- 172.21.15.102:0/1361761220 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c009400 con 0x7fd5a00ca7b0 2024-03-23T03:50:31.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.046+0000 7fd59f400640 1 --2- 172.21.15.102:0/1361761220 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a00ca7b0 0x7fd5a015d6d0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd59000c390 tx=0x7fd59000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:31.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.046+0000 7fd597400640 1 -- 172.21.15.102:0/1361761220 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd590005ce0 con 0x7fd5a00ca7b0 2024-03-23T03:50:31.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.046+0000 7fd597400640 1 -- 172.21.15.102:0/1361761220 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd590005e40 con 0x7fd5a00ca7b0 2024-03-23T03:50:31.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.046+0000 7fd597400640 1 -- 172.21.15.102:0/1361761220 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5900072f0 con 0x7fd5a00ca7b0 2024-03-23T03:50:31.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.046+0000 7fd5a6400640 1 -- 172.21.15.102:0/1361761220 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5a00afe80 con 0x7fd5a00ca7b0 2024-03-23T03:50:31.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.046+0000 7fd5a6400640 1 -- 172.21.15.102:0/1361761220 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a015dc10 con 0x7fd5a00ca7b0 2024-03-23T03:50:31.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.048+0000 7fd5a6400640 1 -- 172.21.15.102:0/1361761220 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5a006def0 con 0x7fd5a00ca7b0 2024-03-23T03:50:31.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.049+0000 7fd597400640 1 -- 172.21.15.102:0/1361761220 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd590007450 con 0x7fd5a00ca7b0 2024-03-23T03:50:31.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.049+0000 7fd597400640 1 --2- 172.21.15.102:0/1361761220 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd58006ac00 0x7fd58006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.050+0000 7fd597400640 1 -- 172.21.15.102:0/1361761220 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd590088ab0 con 0x7fd5a00ca7b0 2024-03-23T03:50:31.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.050+0000 7fd59ea00640 1 --2- 172.21.15.102:0/1361761220 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd58006ac00 0x7fd58006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.050+0000 7fd59ea00640 1 --2- 172.21.15.102:0/1361761220 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd58006ac00 0x7fd58006d0c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd58c036000 tx=0x7fd58c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:31.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.054+0000 7fd597400640 1 -- 172.21.15.102:0/1361761220 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd590053b40 con 0x7fd5a00ca7b0 2024-03-23T03:50:31.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.248+0000 7fd5a6400640 1 -- 172.21.15.102:0/1361761220 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd5a0073390 con 0x7fd5a00ca7b0 2024-03-23T03:50:31.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.249+0000 7fd597400640 1 -- 172.21.15.102:0/1361761220 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd5900170a0 con 0x7fd5a00ca7b0 2024-03-23T03:50:31.250 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:50:31.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.251+0000 7fd5a6400640 1 -- 172.21.15.102:0/1361761220 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd58006ac00 msgr2=0x7fd58006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:31.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.251+0000 7fd5a6400640 1 --2- 172.21.15.102:0/1361761220 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd58006ac00 0x7fd58006d0c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fd58c036000 tx=0x7fd58c030000 comp rx=0 tx=0).stop 2024-03-23T03:50:31.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.251+0000 7fd5a6400640 1 -- 172.21.15.102:0/1361761220 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a00ca7b0 msgr2=0x7fd5a015d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:31.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.251+0000 7fd5a6400640 1 --2- 172.21.15.102:0/1361761220 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a00ca7b0 0x7fd5a015d6d0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7fd59000c390 tx=0x7fd59000c850 comp rx=0 tx=0).stop 2024-03-23T03:50:31.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.251+0000 7fd5a6400640 1 -- 172.21.15.102:0/1361761220 shutdown_connections 2024-03-23T03:50:31.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.251+0000 7fd5a6400640 1 --2- 172.21.15.102:0/1361761220 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd58006ac00 0x7fd58006d0c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.251+0000 7fd5a6400640 1 --2- 172.21.15.102:0/1361761220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd5a0160d20 0x7fd5a015f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.251+0000 7fd5a6400640 1 --2- 172.21.15.102:0/1361761220 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd5a00cf040 0x7fd5a014e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.251+0000 7fd5a6400640 1 --2- 172.21.15.102:0/1361761220 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5a00ca7b0 0x7fd5a015d6d0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.251+0000 7fd5a6400640 1 -- 172.21.15.102:0/1361761220 >> 172.21.15.102:0/1361761220 conn(0x7fd5a006a0b0 msgr2=0x7fd5a0153a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:31.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.251+0000 7fd5a6400640 1 -- 172.21.15.102:0/1361761220 shutdown_connections 2024-03-23T03:50:31.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.252+0000 7fd5a6400640 1 -- 172.21.15.102:0/1361761220 wait complete. 2024-03-23T03:50:31.270 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:50:31.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.482+0000 7f4b7e400640 1 Processor -- start 2024-03-23T03:50:31.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.482+0000 7f4b7e400640 1 -- start start 2024-03-23T03:50:31.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.483+0000 7f4b7e400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b78160800 0x7f4b780b5b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.483+0000 7f4b7e400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b780cd850 con 0x7f4b780b4710 2024-03-23T03:50:31.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.483+0000 7f4b7e400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b780cdab0 con 0x7f4b7815ccf0 2024-03-23T03:50:31.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.483+0000 7f4b7e400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7815d0d0 con 0x7f4b78160800 2024-03-23T03:50:31.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.483+0000 7f4b77e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4b780b4710 0x7f4b780d12e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42464/0 (socket says 172.21.15.102:42464) 2024-03-23T03:50:31.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.483+0000 7f4b77e00640 1 -- 172.21.15.102:0/1295001062 learned_addr learned my addr 172.21.15.102:0/1295001062 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:31.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.484+0000 7f4b76a00640 1 --2- 172.21.15.102:0/1295001062 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b78160800 0x7f4b780b5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.484+0000 7f4b76000640 1 -- 172.21.15.102:0/1295001062 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042936430 0 0) 0x7f4b780cd850 con 0x7f4b780b4710 2024-03-23T03:50:31.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.484+0000 7f4b76000640 1 -- 172.21.15.102:0/1295001062 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b5c003520 con 0x7f4b780b4710 2024-03-23T03:50:31.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.484+0000 7f4b76000640 1 -- 172.21.15.102:0/1295001062 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1881754037 0 0) 0x7f4b5c003520 con 0x7f4b780b4710 2024-03-23T03:50:31.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.484+0000 7f4b76000640 1 -- 172.21.15.102:0/1295001062 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b78160800 msgr2=0x7f4b780b5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:31.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.484+0000 7f4b76000640 1 --2- 172.21.15.102:0/1295001062 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b78160800 0x7f4b780b5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.484+0000 7f4b76000640 1 -- 172.21.15.102:0/1295001062 >> v1:172.21.15.173:6789/0 conn(0x7f4b7815ccf0 legacy=0x7f4b780cd140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:31.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.484+0000 7f4b76000640 1 -- 172.21.15.102:0/1295001062 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b780cdd90 con 0x7f4b780b4710 2024-03-23T03:50:31.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.484+0000 7f4b76000640 1 -- 172.21.15.102:0/1295001062 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f4b700019f0 con 0x7f4b780b4710 2024-03-23T03:50:31.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.485+0000 7f4b76000640 1 -- 172.21.15.102:0/1295001062 >> v1:172.21.15.102:6789/0 conn(0x7f4b780b4710 legacy=0x7f4b780d12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:31.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.485+0000 7f4b76000640 1 --2- 172.21.15.102:0/1295001062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b5c0039a0 0x7f4b5c003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.485+0000 7f4b76000640 1 --2- 172.21.15.102:0/1295001062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b5c0047c0 0x7f4b5c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.485+0000 7f4b76000640 1 --2- 172.21.15.102:0/1295001062 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b78160800 0x7f4b5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.485+0000 7f4b76000640 1 -- 172.21.15.102:0/1295001062 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7815d0d0 con 0x7f4b5c0039a0 2024-03-23T03:50:31.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.485+0000 7f4b76000640 1 -- 172.21.15.102:0/1295001062 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c00ba30 con 0x7f4b5c0047c0 2024-03-23T03:50:31.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.485+0000 7f4b76000640 1 -- 172.21.15.102:0/1295001062 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b5c00bba0 con 0x7f4b78160800 2024-03-23T03:50:31.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.485+0000 7f4b77400640 1 --2- 172.21.15.102:0/1295001062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b5c0039a0 0x7f4b5c003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.486+0000 7f4b76a00640 1 --2- 172.21.15.102:0/1295001062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b5c0047c0 0x7f4b5c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.486+0000 7f4b77400640 1 -- 172.21.15.102:0/1295001062 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b78160800 msgr2=0x7f4b5c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:31.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.486+0000 7f4b77e00640 1 --2- 172.21.15.102:0/1295001062 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b78160800 0x7f4b5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.486+0000 7f4b77400640 1 --2- 172.21.15.102:0/1295001062 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b78160800 0x7f4b5c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.486+0000 7f4b77400640 1 -- 172.21.15.102:0/1295001062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b5c0047c0 msgr2=0x7f4b5c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:31.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.486+0000 7f4b77400640 1 --2- 172.21.15.102:0/1295001062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b5c0047c0 0x7f4b5c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.486+0000 7f4b77400640 1 -- 172.21.15.102:0/1295001062 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c003520 con 0x7f4b5c0039a0 2024-03-23T03:50:31.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.486+0000 7f4b76a00640 1 --2- 172.21.15.102:0/1295001062 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b5c0047c0 0x7f4b5c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:31.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.487+0000 7f4b77400640 1 --2- 172.21.15.102:0/1295001062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b5c0039a0 0x7f4b5c003d80 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f4b68001720 tx=0x7f4b5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a383cae7b508b24 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:31.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.487+0000 7f4b76000640 1 -- 172.21.15.102:0/1295001062 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b680052d0 con 0x7f4b5c0039a0 2024-03-23T03:50:31.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.487+0000 7f4b76000640 1 -- 172.21.15.102:0/1295001062 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6802e050 con 0x7f4b5c0039a0 2024-03-23T03:50:31.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.487+0000 7f4b76000640 1 -- 172.21.15.102:0/1295001062 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b680035f0 con 0x7f4b5c0039a0 2024-03-23T03:50:31.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.488+0000 7f4b7e400640 1 -- 172.21.15.102:0/1295001062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b5c0039a0 msgr2=0x7f4b5c003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:31.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.488+0000 7f4b7e400640 1 --2- 172.21.15.102:0/1295001062 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b5c0039a0 0x7f4b5c003d80 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f4b68001720 tx=0x7f4b5c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.488+0000 7f4b77e00640 1 -- 172.21.15.102:0/1295001062 reap_dead start 2024-03-23T03:50:31.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.488+0000 7f4b7e400640 1 -- 172.21.15.102:0/1295001062 shutdown_connections 2024-03-23T03:50:31.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.488+0000 7f4b7e400640 1 -- 172.21.15.102:0/1295001062 >> 172.21.15.102:0/1295001062 conn(0x7f4b780cf780 msgr2=0x7f4b780cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:31.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.488+0000 7f4b7e400640 1 -- 172.21.15.102:0/1295001062 shutdown_connections 2024-03-23T03:50:31.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.488+0000 7f4b7e400640 1 -- 172.21.15.102:0/1295001062 wait complete. 2024-03-23T03:50:31.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.490+0000 7f4b7e400640 1 Processor -- start 2024-03-23T03:50:31.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.490+0000 7f4b7e400640 1 -- start start 2024-03-23T03:50:31.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.491+0000 7f4b7e400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b5c0039a0 0x7f4b7815b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.491+0000 7f4b7e400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b780c9060 0x7f4b7815af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.491+0000 7f4b7e400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b781f2a70 0x7f4b780cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.491+0000 7f4b7e400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b780cbe90 con 0x7f4b781f2a70 2024-03-23T03:50:31.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.491+0000 7f4b7e400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b781e6350 con 0x7f4b5c0039a0 2024-03-23T03:50:31.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.491+0000 7f4b7e400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b781e6630 con 0x7f4b780c9060 2024-03-23T03:50:31.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.491+0000 7f4b77e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b781f2a70 0x7f4b780cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.491+0000 7f4b77e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b781f2a70 0x7f4b780cb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43030/0 (socket says 172.21.15.102:43030) 2024-03-23T03:50:31.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.491+0000 7f4b77e00640 1 -- 172.21.15.102:0/3119563888 learned_addr learned my addr 172.21.15.102:0/3119563888 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:31.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.491+0000 7f4b77400640 1 --2- 172.21.15.102:0/3119563888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b5c0039a0 0x7f4b7815b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.492+0000 7f4b76a00640 1 --2- 172.21.15.102:0/3119563888 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b780c9060 0x7f4b7815af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.492+0000 7f4b77e00640 1 -- 172.21.15.102:0/3119563888 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b780c9060 msgr2=0x7f4b7815af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:31.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.492+0000 7f4b77e00640 1 --2- 172.21.15.102:0/3119563888 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b780c9060 0x7f4b7815af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.492+0000 7f4b77e00640 1 -- 172.21.15.102:0/3119563888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b5c0039a0 msgr2=0x7f4b7815b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:31.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.492+0000 7f4b77e00640 1 --2- 172.21.15.102:0/3119563888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b5c0039a0 0x7f4b7815b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.492+0000 7f4b77e00640 1 -- 172.21.15.102:0/3119563888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b781e6910 con 0x7f4b781f2a70 2024-03-23T03:50:31.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.492+0000 7f4b77e00640 1 --2- 172.21.15.102:0/3119563888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b781f2a70 0x7f4b780cb1b0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f4b70001330 tx=0x7f4b7000ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.492+0000 7f4b6f400640 1 -- 172.21.15.102:0/3119563888 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b70008940 con 0x7f4b781f2a70 2024-03-23T03:50:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.492+0000 7f4b6f400640 1 -- 172.21.15.102:0/3119563888 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b70008aa0 con 0x7f4b781f2a70 2024-03-23T03:50:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.492+0000 7f4b6f400640 1 -- 172.21.15.102:0/3119563888 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b70015670 con 0x7f4b781f2a70 2024-03-23T03:50:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.492+0000 7f4b7e400640 1 -- 172.21.15.102:0/3119563888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b781e6bf0 con 0x7f4b781f2a70 2024-03-23T03:50:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.493+0000 7f4b7e400640 1 -- 172.21.15.102:0/3119563888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b781e8f00 con 0x7f4b781f2a70 2024-03-23T03:50:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.494+0000 7f4b7e400640 1 -- 172.21.15.102:0/3119563888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b781e9120 con 0x7f4b781f2a70 2024-03-23T03:50:31.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.495+0000 7f4b6f400640 1 -- 172.21.15.102:0/3119563888 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4b700157d0 con 0x7f4b781f2a70 2024-03-23T03:50:31.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.496+0000 7f4b6f400640 1 --2- 172.21.15.102:0/3119563888 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4b5806ac30 0x7f4b5806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.496+0000 7f4b6f400640 1 -- 172.21.15.102:0/3119563888 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4b700896b0 con 0x7f4b781f2a70 2024-03-23T03:50:31.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.496+0000 7f4b77400640 1 --2- 172.21.15.102:0/3119563888 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4b5806ac30 0x7f4b5806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.497+0000 7f4b77400640 1 --2- 172.21.15.102:0/3119563888 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4b5806ac30 0x7f4b5806d0f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4b68033000 tx=0x7f4b68009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:31.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.501+0000 7f4b6f400640 1 -- 172.21.15.102:0/3119563888 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4b70054750 con 0x7f4b781f2a70 2024-03-23T03:50:31.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.679+0000 7f4b7e400640 1 -- 172.21.15.102:0/3119563888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4b781e6de0 con 0x7f4b781f2a70 2024-03-23T03:50:31.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.680+0000 7f4b6f400640 1 -- 172.21.15.102:0/3119563888 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4b70057d70 con 0x7f4b781f2a70 2024-03-23T03:50:31.681 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:50:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.683+0000 7f4b7e400640 1 -- 172.21.15.102:0/3119563888 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4b5806ac30 msgr2=0x7f4b5806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.683+0000 7f4b7e400640 1 --2- 172.21.15.102:0/3119563888 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4b5806ac30 0x7f4b5806d0f0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f4b68033000 tx=0x7f4b68009030 comp rx=0 tx=0).stop 2024-03-23T03:50:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.683+0000 7f4b7e400640 1 -- 172.21.15.102:0/3119563888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b781f2a70 msgr2=0x7f4b780cb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.683+0000 7f4b7e400640 1 --2- 172.21.15.102:0/3119563888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b781f2a70 0x7f4b780cb1b0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f4b70001330 tx=0x7f4b7000ef70 comp rx=0 tx=0).stop 2024-03-23T03:50:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.683+0000 7f4b7e400640 1 -- 172.21.15.102:0/3119563888 shutdown_connections 2024-03-23T03:50:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.683+0000 7f4b7e400640 1 --2- 172.21.15.102:0/3119563888 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4b5806ac30 0x7f4b5806d0f0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.683+0000 7f4b7e400640 1 --2- 172.21.15.102:0/3119563888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4b781f2a70 0x7f4b780cb1b0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.683+0000 7f4b7e400640 1 --2- 172.21.15.102:0/3119563888 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4b780c9060 0x7f4b7815af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.683+0000 7f4b7e400640 1 --2- 172.21.15.102:0/3119563888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4b5c0039a0 0x7f4b7815b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.683+0000 7f4b7e400640 1 -- 172.21.15.102:0/3119563888 >> 172.21.15.102:0/3119563888 conn(0x7f4b780cf780 msgr2=0x7f4b780b7690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:31.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.683+0000 7f4b7e400640 1 -- 172.21.15.102:0/3119563888 shutdown_connections 2024-03-23T03:50:31.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.683+0000 7f4b7e400640 1 -- 172.21.15.102:0/3119563888 wait complete. 2024-03-23T03:50:31.702 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:50:31.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.896+0000 7f07c7000640 1 Processor -- start 2024-03-23T03:50:31.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.896+0000 7f07c7000640 1 -- start start 2024-03-23T03:50:31.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.897+0000 7f07c7000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f07c00cd6b0 0x7f07c00b8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.897+0000 7f07c7000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f07c00cda90 0x7f07c00d1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.897+0000 7f07c7000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c006b840 con 0x7f07c015cd90 2024-03-23T03:50:31.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.897+0000 7f07c7000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c006c0e0 con 0x7f07c00cd6b0 2024-03-23T03:50:31.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.897+0000 7f07c7000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c006bb20 con 0x7f07c00cda90 2024-03-23T03:50:31.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.898+0000 7f07bfe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f07c00cd6b0 0x7f07c00b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.898+0000 7f07bf400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f07c015cd90 0x7f07c006b060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:42476/0 (socket says 172.21.15.102:42476) 2024-03-23T03:50:31.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.898+0000 7f07bf400640 1 -- 172.21.15.102:0/1998636880 learned_addr learned my addr 172.21.15.102:0/1998636880 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:31.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.898+0000 7f07c5000640 1 --2- 172.21.15.102:0/1998636880 >> v2:172.21.15.173:3300/0 conn(0x7f07c00cda90 0x7f07c00d1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.898+0000 7f07bfe00640 1 -- 172.21.15.102:0/1998636880 >> v2:172.21.15.173:3300/0 conn(0x7f07c00cda90 msgr2=0x7f07c00d1370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:31.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.898+0000 7f07bfe00640 1 --2- 172.21.15.102:0/1998636880 >> v2:172.21.15.173:3300/0 conn(0x7f07c00cda90 0x7f07c00d1370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.898+0000 7f07bfe00640 1 -- 172.21.15.102:0/1998636880 >> v1:172.21.15.102:6789/0 conn(0x7f07c015cd90 legacy=0x7f07c006b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:31.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.898+0000 7f07bfe00640 1 -- 172.21.15.102:0/1998636880 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c006be00 con 0x7f07c00cd6b0 2024-03-23T03:50:31.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.899+0000 7f07bfe00640 1 --2- 172.21.15.102:0/1998636880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07c00cd6b0 0x7f07c00b8490 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f07c008b730 tx=0x7f07b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b784a69bed9cc687 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:31.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.900+0000 7f07bea00640 1 -- 172.21.15.102:0/1998636880 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b0031830 con 0x7f07c00cd6b0 2024-03-23T03:50:31.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.900+0000 7f07bea00640 1 -- 172.21.15.102:0/1998636880 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b0031990 con 0x7f07c00cd6b0 2024-03-23T03:50:31.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.900+0000 7f07bea00640 1 -- 172.21.15.102:0/1998636880 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b0031d20 con 0x7f07c00cd6b0 2024-03-23T03:50:31.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.900+0000 7f07c7000640 1 -- 172.21.15.102:0/1998636880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07c00cd6b0 msgr2=0x7f07c00b8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:31.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.900+0000 7f07c7000640 1 --2- 172.21.15.102:0/1998636880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07c00cd6b0 0x7f07c00b8490 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f07c008b730 tx=0x7f07b002b8d0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.900+0000 7f07c7000640 1 -- 172.21.15.102:0/1998636880 shutdown_connections 2024-03-23T03:50:31.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.900+0000 7f07c7000640 1 --2- 172.21.15.102:0/1998636880 >> v2:172.21.15.173:3300/0 conn(0x7f07c00cda90 0x7f07c00d1370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.900+0000 7f07c7000640 1 --2- 172.21.15.102:0/1998636880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07c00cd6b0 0x7f07c00b8490 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.900+0000 7f07c7000640 1 -- 172.21.15.102:0/1998636880 >> 172.21.15.102:0/1998636880 conn(0x7f07c00cf780 msgr2=0x7f07c00cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:31.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.900+0000 7f07c7000640 1 -- 172.21.15.102:0/1998636880 shutdown_connections 2024-03-23T03:50:31.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.900+0000 7f07c7000640 1 -- 172.21.15.102:0/1998636880 wait complete. 2024-03-23T03:50:31.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.902+0000 7f07c7000640 1 Processor -- start 2024-03-23T03:50:31.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.902+0000 7f07c7000640 1 -- start start 2024-03-23T03:50:31.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.903+0000 7f07c7000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f07c00cd6b0 0x7f07c015b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.903+0000 7f07c7000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07c00cda90 0x7f07c0154170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.903+0000 7f07c7000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07c015cd90 0x7f07c01546b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.903+0000 7f07c7000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c00ca370 con 0x7f07c015cd90 2024-03-23T03:50:31.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.903+0000 7f07c7000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c00cc1e0 con 0x7f07c00cda90 2024-03-23T03:50:31.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.903+0000 7f07c7000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c00ca5f0 con 0x7f07c00cd6b0 2024-03-23T03:50:31.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.903+0000 7f07c5000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07c015cd90 0x7f07c01546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.904+0000 7f07c5000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07c015cd90 0x7f07c01546b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43046/0 (socket says 172.21.15.102:43046) 2024-03-23T03:50:31.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.904+0000 7f07c5000640 1 -- 172.21.15.102:0/706943314 learned_addr learned my addr 172.21.15.102:0/706943314 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:31.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.904+0000 7f07bfe00640 1 --2- 172.21.15.102:0/706943314 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f07c00cd6b0 0x7f07c015b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.904+0000 7f07bf400640 1 --2- 172.21.15.102:0/706943314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07c00cda90 0x7f07c0154170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.904+0000 7f07c5000640 1 -- 172.21.15.102:0/706943314 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f07c00cd6b0 msgr2=0x7f07c015b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:31.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.904+0000 7f07c5000640 1 --2- 172.21.15.102:0/706943314 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f07c00cd6b0 0x7f07c015b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.904+0000 7f07c5000640 1 -- 172.21.15.102:0/706943314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07c00cda90 msgr2=0x7f07c0154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:31.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.904+0000 7f07c5000640 1 --2- 172.21.15.102:0/706943314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07c00cda90 0x7f07c0154170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:31.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.904+0000 7f07c5000640 1 -- 172.21.15.102:0/706943314 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b0009400 con 0x7f07c015cd90 2024-03-23T03:50:31.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.904+0000 7f07c5000640 1 --2- 172.21.15.102:0/706943314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07c015cd90 0x7f07c01546b0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f07b800efc0 tx=0x7f07b800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:31.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.905+0000 7f07b7e00640 1 -- 172.21.15.102:0/706943314 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b8014480 con 0x7f07c015cd90 2024-03-23T03:50:31.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.905+0000 7f07b7e00640 1 -- 172.21.15.102:0/706943314 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b800f040 con 0x7f07c015cd90 2024-03-23T03:50:31.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.905+0000 7f07b7e00640 1 -- 172.21.15.102:0/706943314 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b8014820 con 0x7f07c015cd90 2024-03-23T03:50:31.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.905+0000 7f07c7000640 1 -- 172.21.15.102:0/706943314 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c00cc430 con 0x7f07c015cd90 2024-03-23T03:50:31.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.905+0000 7f07c7000640 1 -- 172.21.15.102:0/706943314 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c01f2640 con 0x7f07c015cd90 2024-03-23T03:50:31.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.906+0000 7f07c7000640 1 -- 172.21.15.102:0/706943314 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c00d26d0 con 0x7f07c015cd90 2024-03-23T03:50:31.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.907+0000 7f07b7e00640 1 -- 172.21.15.102:0/706943314 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f07b8014980 con 0x7f07c015cd90 2024-03-23T03:50:31.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.907+0000 7f07b7e00640 1 --2- 172.21.15.102:0/706943314 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f07a006ac00 0x7f07a006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:31.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.908+0000 7f07b7e00640 1 -- 172.21.15.102:0/706943314 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f07b808a070 con 0x7f07c015cd90 2024-03-23T03:50:31.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.908+0000 7f07bfe00640 1 --2- 172.21.15.102:0/706943314 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f07a006ac00 0x7f07a006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:31.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.908+0000 7f07bfe00640 1 --2- 172.21.15.102:0/706943314 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f07a006ac00 0x7f07a006d0c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f07b0036000 tx=0x7f07b003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:31.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:31.912+0000 7f07b7e00640 1 -- 172.21.15.102:0/706943314 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f07b80bd580 con 0x7f07c015cd90 2024-03-23T03:50:32.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.111+0000 7f07c7000640 1 -- 172.21.15.102:0/706943314 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f07c00cc950 con 0x7f07c015cd90 2024-03-23T03:50:32.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.112+0000 7f07b7e00640 1 -- 172.21.15.102:0/706943314 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v81) v1 ==== 80+0+0 (secure 0 0 0) 0x7f07b8018080 con 0x7f07c015cd90 2024-03-23T03:50:32.113 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:50:32.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.115+0000 7f07c7000640 1 -- 172.21.15.102:0/706943314 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f07a006ac00 msgr2=0x7f07a006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:32.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.115+0000 7f07c7000640 1 --2- 172.21.15.102:0/706943314 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f07a006ac00 0x7f07a006d0c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f07b0036000 tx=0x7f07b003c040 comp rx=0 tx=0).stop 2024-03-23T03:50:32.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.115+0000 7f07c7000640 1 -- 172.21.15.102:0/706943314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07c015cd90 msgr2=0x7f07c01546b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:32.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.115+0000 7f07c7000640 1 --2- 172.21.15.102:0/706943314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07c015cd90 0x7f07c01546b0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f07b800efc0 tx=0x7f07b800c490 comp rx=0 tx=0).stop 2024-03-23T03:50:32.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.115+0000 7f07c7000640 1 -- 172.21.15.102:0/706943314 shutdown_connections 2024-03-23T03:50:32.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.115+0000 7f07c7000640 1 --2- 172.21.15.102:0/706943314 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f07a006ac00 0x7f07a006d0c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:32.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.115+0000 7f07c7000640 1 --2- 172.21.15.102:0/706943314 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f07c015cd90 0x7f07c01546b0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:32.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.115+0000 7f07c7000640 1 --2- 172.21.15.102:0/706943314 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07c00cda90 0x7f07c0154170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:32.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.115+0000 7f07c7000640 1 --2- 172.21.15.102:0/706943314 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f07c00cd6b0 0x7f07c015b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:32.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.115+0000 7f07c7000640 1 -- 172.21.15.102:0/706943314 >> 172.21.15.102:0/706943314 conn(0x7f07c00cf780 msgr2=0x7f07c006b060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:32.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.115+0000 7f07c7000640 1 -- 172.21.15.102:0/706943314 shutdown_connections 2024-03-23T03:50:32.116 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.115+0000 7f07c7000640 1 -- 172.21.15.102:0/706943314 wait complete. 2024-03-23T03:50:32.133 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-23T03:50:32.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:32.133 DEBUG:teuthology.orchestra.run.smithi173:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-03-23T03:50:32.245 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:50:32.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:32.245 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T03:50:32.297 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:50:32.297 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:32.351 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:50:32.351 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:50:32.352 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:32.406 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:50:32.407 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:50:32.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.617+0000 7f3f1bc00640 1 Processor -- start 2024-03-23T03:50:32.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.617+0000 7f3f1bc00640 1 -- start start 2024-03-23T03:50:32.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.618+0000 7f3f1bc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3f1415b560 0x7f3f1415b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:32.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.618+0000 7f3f1bc00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f3f1415d410 0x7f3f140b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:32.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.619+0000 7f3f1bc00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f1406c6f0 con 0x7f3f140b39b0 2024-03-23T03:50:32.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.619+0000 7f3f1bc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f140b4370 con 0x7f3f1415b560 2024-03-23T03:50:32.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.619+0000 7f3f1bc00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f1406c9d0 con 0x7f3f1415d410 2024-03-23T03:50:32.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.619+0000 7f3f19200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3f1415b560 0x7f3f1415b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:32.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.619+0000 7f3f19200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3f1415b560 0x7f3f1415b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43054/0 (socket says 172.21.15.102:43054) 2024-03-23T03:50:32.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.619+0000 7f3f19200640 1 -- 172.21.15.102:0/1741897331 learned_addr learned my addr 172.21.15.102:0/1741897331 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:32.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.619+0000 7f3f19200640 1 -- 172.21.15.102:0/1741897331 >> v2:172.21.15.173:3300/0 conn(0x7f3f1415d410 msgr2=0x7f3f140b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:32.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.619+0000 7f3f19200640 1 --2- 172.21.15.102:0/1741897331 >> v2:172.21.15.173:3300/0 conn(0x7f3f1415d410 0x7f3f140b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:32.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.619+0000 7f3f19200640 1 -- 172.21.15.102:0/1741897331 >> v1:172.21.15.173:6789/0 conn(0x7f3f140b39b0 legacy=0x7f3f1415e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:32.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.619+0000 7f3f19200640 1 -- 172.21.15.102:0/1741897331 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f1406ccb0 con 0x7f3f1415b560 2024-03-23T03:50:32.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.620+0000 7f3f19200640 1 --2- 172.21.15.102:0/1741897331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f1415b560 0x7f3f1415b940 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f3f14070100 tx=0x7f3f0402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e3116c0590b91a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:32.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.621+0000 7f3f13e00640 1 -- 172.21.15.102:0/1741897331 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f040317e0 con 0x7f3f1415b560 2024-03-23T03:50:32.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.621+0000 7f3f13e00640 1 -- 172.21.15.102:0/1741897331 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f04031940 con 0x7f3f1415b560 2024-03-23T03:50:32.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.621+0000 7f3f13e00640 1 -- 172.21.15.102:0/1741897331 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f04031cd0 con 0x7f3f1415b560 2024-03-23T03:50:32.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.621+0000 7f3f1bc00640 1 -- 172.21.15.102:0/1741897331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f1415b560 msgr2=0x7f3f1415b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:32.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.621+0000 7f3f1bc00640 1 --2- 172.21.15.102:0/1741897331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f1415b560 0x7f3f1415b940 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f3f14070100 tx=0x7f3f0402b880 comp rx=0 tx=0).stop 2024-03-23T03:50:32.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.621+0000 7f3f1bc00640 1 -- 172.21.15.102:0/1741897331 shutdown_connections 2024-03-23T03:50:32.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.621+0000 7f3f1bc00640 1 --2- 172.21.15.102:0/1741897331 >> v2:172.21.15.173:3300/0 conn(0x7f3f1415d410 0x7f3f140b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:32.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.621+0000 7f3f1bc00640 1 --2- 172.21.15.102:0/1741897331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f1415b560 0x7f3f1415b940 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:32.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.621+0000 7f3f1bc00640 1 -- 172.21.15.102:0/1741897331 >> 172.21.15.102:0/1741897331 conn(0x7f3f1406a3d0 msgr2=0x7f3f140b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:32.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.621+0000 7f3f1bc00640 1 -- 172.21.15.102:0/1741897331 shutdown_connections 2024-03-23T03:50:32.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.622+0000 7f3f1bc00640 1 -- 172.21.15.102:0/1741897331 wait complete. 2024-03-23T03:50:32.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.623+0000 7f3f1bc00640 1 Processor -- start 2024-03-23T03:50:32.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.623+0000 7f3f1bc00640 1 -- start start 2024-03-23T03:50:32.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.624+0000 7f3f1bc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f140b39b0 0x7f3f1415a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:32.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.624+0000 7f3f1bc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3f1415b560 0x7f3f140b1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:32.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.624+0000 7f3f1bc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f1415d410 0x7f3f14141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:32.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.624+0000 7f3f1bc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f140b27e0 con 0x7f3f1415d410 2024-03-23T03:50:32.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.624+0000 7f3f1bc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f140b2ac0 con 0x7f3f140b39b0 2024-03-23T03:50:32.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.624+0000 7f3f1bc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f140b2da0 con 0x7f3f1415b560 2024-03-23T03:50:32.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.625+0000 7f3f19c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f1415d410 0x7f3f14141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:32.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.625+0000 7f3f19c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f1415d410 0x7f3f14141ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43070/0 (socket says 172.21.15.102:43070) 2024-03-23T03:50:32.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.625+0000 7f3f19c00640 1 -- 172.21.15.102:0/3097635490 learned_addr learned my addr 172.21.15.102:0/3097635490 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:32.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.625+0000 7f3f19200640 1 --2- 172.21.15.102:0/3097635490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f140b39b0 0x7f3f1415a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:32.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.625+0000 7f3f0be00640 1 --2- 172.21.15.102:0/3097635490 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3f1415b560 0x7f3f140b1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:32.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.625+0000 7f3f19c00640 1 -- 172.21.15.102:0/3097635490 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3f1415b560 msgr2=0x7f3f140b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:32.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.625+0000 7f3f19c00640 1 --2- 172.21.15.102:0/3097635490 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3f1415b560 0x7f3f140b1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:32.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.625+0000 7f3f19c00640 1 -- 172.21.15.102:0/3097635490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f140b39b0 msgr2=0x7f3f1415a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:32.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.625+0000 7f3f19c00640 1 --2- 172.21.15.102:0/3097635490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f140b39b0 0x7f3f1415a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:32.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.625+0000 7f3f19c00640 1 -- 172.21.15.102:0/3097635490 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f040094d0 con 0x7f3f1415d410 2024-03-23T03:50:32.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.625+0000 7f3f19200640 1 --2- 172.21.15.102:0/3097635490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f140b39b0 0x7f3f1415a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:50:32.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.625+0000 7f3f0be00640 1 --2- 172.21.15.102:0/3097635490 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3f1415b560 0x7f3f140b1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:32.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.625+0000 7f3f19c00640 1 --2- 172.21.15.102:0/3097635490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f1415d410 0x7f3f14141ae0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00b970 tx=0x7f3f0c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:32.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.626+0000 7f3f11600640 1 -- 172.21.15.102:0/3097635490 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c00cb40 con 0x7f3f1415d410 2024-03-23T03:50:32.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.626+0000 7f3f11600640 1 -- 172.21.15.102:0/3097635490 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f0c00ccc0 con 0x7f3f1415d410 2024-03-23T03:50:32.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.626+0000 7f3f11600640 1 -- 172.21.15.102:0/3097635490 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0c014680 con 0x7f3f1415d410 2024-03-23T03:50:32.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.626+0000 7f3f1bc00640 1 -- 172.21.15.102:0/3097635490 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f140b3080 con 0x7f3f1415d410 2024-03-23T03:50:32.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.626+0000 7f3f1bc00640 1 -- 172.21.15.102:0/3097635490 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f1415a920 con 0x7f3f1415d410 2024-03-23T03:50:32.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.627+0000 7f3f1bc00640 1 -- 172.21.15.102:0/3097635490 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f140b35a0 con 0x7f3f1415d410 2024-03-23T03:50:32.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.628+0000 7f3f11600640 1 -- 172.21.15.102:0/3097635490 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3f0c01d3b0 con 0x7f3f1415d410 2024-03-23T03:50:32.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.629+0000 7f3f11600640 1 --2- 172.21.15.102:0/3097635490 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3ef006ac00 0x7f3ef006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:32.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.629+0000 7f3f11600640 1 -- 172.21.15.102:0/3097635490 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3f0c089430 con 0x7f3f1415d410 2024-03-23T03:50:32.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.629+0000 7f3f19200640 1 --2- 172.21.15.102:0/3097635490 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3ef006ac00 0x7f3ef006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:32.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.630+0000 7f3f19200640 1 --2- 172.21.15.102:0/3097635490 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3ef006ac00 0x7f3ef006d0c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3f04036000 tx=0x7f3f0403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:32.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.634+0000 7f3f11600640 1 -- 172.21.15.102:0/3097635490 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3f140b35a0 con 0x7f3f1415d410 2024-03-23T03:50:32.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.867+0000 7f3f1bc00640 1 -- 172.21.15.102:0/3097635490 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f140b35a0 con 0x7f3f1415d410 2024-03-23T03:50:32.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.868+0000 7f3f11600640 1 -- 172.21.15.102:0/3097635490 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f3f140b35a0 con 0x7f3f1415d410 2024-03-23T03:50:32.869 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:32.869 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2079098809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2079098809},{"type":"v1","addr":"172.21.15.173:6833","nonce":2079098809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":5413,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/459102175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":459102175},{"type":"v1","addr":"172.21.15.173:6835","nonce":459102175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2962195583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2962195583},{"type":"v1","addr":"172.21.15.102:6835","nonce":2962195583}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:50:07.937613+0000","modified":"2024-03-23T03:50:10.958097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6384":{"gid":6384,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/3091739472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3091739472},{"type":"v1","addr":"172.21.15.102:6837","nonce":3091739472}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T03:50:32.869 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 41 2024-03-23T03:50:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.871+0000 7f3f1bc00640 1 -- 172.21.15.102:0/3097635490 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3ef006ac00 msgr2=0x7f3ef006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.871+0000 7f3f1bc00640 1 --2- 172.21.15.102:0/3097635490 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3ef006ac00 0x7f3ef006d0c0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3f04036000 tx=0x7f3f0403d040 comp rx=0 tx=0).stop 2024-03-23T03:50:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.871+0000 7f3f1bc00640 1 -- 172.21.15.102:0/3097635490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f1415d410 msgr2=0x7f3f14141ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.871+0000 7f3f1bc00640 1 --2- 172.21.15.102:0/3097635490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f1415d410 0x7f3f14141ae0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c00b970 tx=0x7f3f0c00be30 comp rx=0 tx=0).stop 2024-03-23T03:50:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.871+0000 7f3f1bc00640 1 -- 172.21.15.102:0/3097635490 shutdown_connections 2024-03-23T03:50:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.871+0000 7f3f1bc00640 1 --2- 172.21.15.102:0/3097635490 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3ef006ac00 0x7f3ef006d0c0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:32.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.871+0000 7f3f1bc00640 1 --2- 172.21.15.102:0/3097635490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f1415d410 0x7f3f14141ae0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:32.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.871+0000 7f3f1bc00640 1 --2- 172.21.15.102:0/3097635490 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3f1415b560 0x7f3f140b1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:32.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.871+0000 7f3f1bc00640 1 --2- 172.21.15.102:0/3097635490 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3f140b39b0 0x7f3f1415a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:32.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.871+0000 7f3f1bc00640 1 -- 172.21.15.102:0/3097635490 >> 172.21.15.102:0/3097635490 conn(0x7f3f1406a3d0 msgr2=0x7f3f140ae3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:32.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.871+0000 7f3f1bc00640 1 -- 172.21.15.102:0/3097635490 shutdown_connections 2024-03-23T03:50:32.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:32.871+0000 7f3f1bc00640 1 -- 172.21.15.102:0/3097635490 wait complete. 2024-03-23T03:50:32.891 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:50:33.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.097+0000 7f8c4a000640 1 Processor -- start 2024-03-23T03:50:33.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.097+0000 7f8c4a000640 1 -- start start 2024-03-23T03:50:33.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.099+0000 7f8c4a000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8c440b37f0 0x7f8c440cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:33.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.099+0000 7f8c4a000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f8c440ccec0 0x7f8c4406c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:33.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.099+0000 7f8c4a000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c4406da70 con 0x7f8c4414aa40 2024-03-23T03:50:33.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.099+0000 7f8c4a000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c4406d7d0 con 0x7f8c440b37f0 2024-03-23T03:50:33.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.099+0000 7f8c4a000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c440b1d00 con 0x7f8c440ccec0 2024-03-23T03:50:33.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.099+0000 7f8c42a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8c440b37f0 0x7f8c440cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:33.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.099+0000 7f8c42a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8c440b37f0 0x7f8c440cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50614/0 (socket says 172.21.15.102:50614) 2024-03-23T03:50:33.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.099+0000 7f8c42a00640 1 -- 172.21.15.102:0/4012966397 learned_addr learned my addr 172.21.15.102:0/4012966397 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:33.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.100+0000 7f8c43e00640 1 --2- 172.21.15.102:0/4012966397 >> v2:172.21.15.173:3300/0 conn(0x7f8c440ccec0 0x7f8c4406c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:33.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.100+0000 7f8c42a00640 1 -- 172.21.15.102:0/4012966397 >> v2:172.21.15.173:3300/0 conn(0x7f8c440ccec0 msgr2=0x7f8c4406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:33.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.100+0000 7f8c42a00640 1 --2- 172.21.15.102:0/4012966397 >> v2:172.21.15.173:3300/0 conn(0x7f8c440ccec0 0x7f8c4406c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.100+0000 7f8c42a00640 1 -- 172.21.15.102:0/4012966397 >> v1:172.21.15.173:6789/0 conn(0x7f8c4414aa40 legacy=0x7f8c440cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:33.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.100+0000 7f8c42a00640 1 -- 172.21.15.102:0/4012966397 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c440b1fe0 con 0x7f8c440b37f0 2024-03-23T03:50:33.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.100+0000 7f8c43e00640 1 --2- 172.21.15.102:0/4012966397 >> v2:172.21.15.173:3300/0 conn(0x7f8c440ccec0 0x7f8c4406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:33.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.101+0000 7f8c42a00640 1 --2- 172.21.15.102:0/4012966397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c440b37f0 0x7f8c440cd7a0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f8c440adae0 tx=0x7f8c3002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=552627c223612204 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:33.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.101+0000 7f8c42000640 1 -- 172.21.15.102:0/4012966397 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c30031870 con 0x7f8c440b37f0 2024-03-23T03:50:33.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.101+0000 7f8c42000640 1 -- 172.21.15.102:0/4012966397 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c300319d0 con 0x7f8c440b37f0 2024-03-23T03:50:33.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.101+0000 7f8c42000640 1 -- 172.21.15.102:0/4012966397 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c30031d60 con 0x7f8c440b37f0 2024-03-23T03:50:33.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.102+0000 7f8c4a000640 1 -- 172.21.15.102:0/4012966397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c440b37f0 msgr2=0x7f8c440cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:33.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.102+0000 7f8c4a000640 1 --2- 172.21.15.102:0/4012966397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c440b37f0 0x7f8c440cd7a0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f8c440adae0 tx=0x7f8c3002b910 comp rx=0 tx=0).stop 2024-03-23T03:50:33.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.102+0000 7f8c4a000640 1 -- 172.21.15.102:0/4012966397 shutdown_connections 2024-03-23T03:50:33.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.102+0000 7f8c4a000640 1 --2- 172.21.15.102:0/4012966397 >> v2:172.21.15.173:3300/0 conn(0x7f8c440ccec0 0x7f8c4406c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.102+0000 7f8c4a000640 1 --2- 172.21.15.102:0/4012966397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c440b37f0 0x7f8c440cd7a0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.102+0000 7f8c4a000640 1 -- 172.21.15.102:0/4012966397 >> 172.21.15.102:0/4012966397 conn(0x7f8c4406a3d0 msgr2=0x7f8c440b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:33.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.102+0000 7f8c4a000640 1 -- 172.21.15.102:0/4012966397 shutdown_connections 2024-03-23T03:50:33.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.102+0000 7f8c4a000640 1 -- 172.21.15.102:0/4012966397 wait complete. 2024-03-23T03:50:33.104 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.104+0000 7f8c4a000640 1 Processor -- start 2024-03-23T03:50:33.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.104+0000 7f8c4a000640 1 -- start start 2024-03-23T03:50:33.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.105+0000 7f8c4a000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8c440b37f0 0x7f8c440a9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:33.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.105+0000 7f8c4a000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c440ccec0 0x7f8c44161e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:33.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.105+0000 7f8c4a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c4414aa40 0x7f8c44162380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:33.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.105+0000 7f8c4a000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c44144880 con 0x7f8c4414aa40 2024-03-23T03:50:33.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.105+0000 7f8c4a000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c440abc70 con 0x7f8c440ccec0 2024-03-23T03:50:33.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.105+0000 7f8c4a000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c44162a20 con 0x7f8c440b37f0 2024-03-23T03:50:33.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.105+0000 7f8c43e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c4414aa40 0x7f8c44162380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:33.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.105+0000 7f8c43e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c4414aa40 0x7f8c44162380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50620/0 (socket says 172.21.15.102:50620) 2024-03-23T03:50:33.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.105+0000 7f8c43e00640 1 -- 172.21.15.102:0/2305945113 learned_addr learned my addr 172.21.15.102:0/2305945113 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:33.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.105+0000 7f8c42a00640 1 --2- 172.21.15.102:0/2305945113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c440ccec0 0x7f8c44161e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:33.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.105+0000 7f8c43e00640 1 -- 172.21.15.102:0/2305945113 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8c440b37f0 msgr2=0x7f8c440a9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:33.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.105+0000 7f8c43400640 1 --2- 172.21.15.102:0/2305945113 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8c440b37f0 0x7f8c440a9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:33.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.106+0000 7f8c43e00640 1 --2- 172.21.15.102:0/2305945113 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8c440b37f0 0x7f8c440a9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.106+0000 7f8c43e00640 1 -- 172.21.15.102:0/2305945113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c440ccec0 msgr2=0x7f8c44161e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:33.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.106+0000 7f8c43e00640 1 --2- 172.21.15.102:0/2305945113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c440ccec0 0x7f8c44161e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.106+0000 7f8c43e00640 1 -- 172.21.15.102:0/2305945113 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c300094d0 con 0x7f8c4414aa40 2024-03-23T03:50:33.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.106+0000 7f8c42a00640 1 --2- 172.21.15.102:0/2305945113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c440ccec0 0x7f8c44161e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:33.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.106+0000 7f8c43400640 1 --2- 172.21.15.102:0/2305945113 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8c440b37f0 0x7f8c440a9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:33.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.106+0000 7f8c43e00640 1 --2- 172.21.15.102:0/2305945113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c4414aa40 0x7f8c44162380 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00c7f0 tx=0x7f8c3c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:33.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.106+0000 7f8c3b400640 1 -- 172.21.15.102:0/2305945113 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c007820 con 0x7f8c4414aa40 2024-03-23T03:50:33.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.106+0000 7f8c3b400640 1 -- 172.21.15.102:0/2305945113 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c3c007980 con 0x7f8c4414aa40 2024-03-23T03:50:33.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.106+0000 7f8c4a000640 1 -- 172.21.15.102:0/2305945113 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c44162d00 con 0x7f8c4414aa40 2024-03-23T03:50:33.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.106+0000 7f8c3b400640 1 -- 172.21.15.102:0/2305945113 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c3c007d70 con 0x7f8c4414aa40 2024-03-23T03:50:33.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.106+0000 7f8c4a000640 1 -- 172.21.15.102:0/2305945113 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c440ac1d0 con 0x7f8c4414aa40 2024-03-23T03:50:33.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.108+0000 7f8c4a000640 1 -- 172.21.15.102:0/2305945113 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c44162f40 con 0x7f8c4414aa40 2024-03-23T03:50:33.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.109+0000 7f8c3b400640 1 -- 172.21.15.102:0/2305945113 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8c3c0113f0 con 0x7f8c4414aa40 2024-03-23T03:50:33.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.109+0000 7f8c3b400640 1 --2- 172.21.15.102:0/2305945113 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8c2406ac00 0x7f8c2406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:33.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.109+0000 7f8c3b400640 1 -- 172.21.15.102:0/2305945113 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8c3c0895d0 con 0x7f8c4414aa40 2024-03-23T03:50:33.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.110+0000 7f8c43400640 1 --2- 172.21.15.102:0/2305945113 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8c2406ac00 0x7f8c2406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:33.111 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.110+0000 7f8c43400640 1 --2- 172.21.15.102:0/2305945113 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8c2406ac00 0x7f8c2406d0c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8c34007eb0 tx=0x7f8c340073c0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:33.114 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.114+0000 7f8c3b400640 1 -- 172.21.15.102:0/2305945113 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8c3c054700 con 0x7f8c4414aa40 2024-03-23T03:50:33.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.309+0000 7f8c4a000640 1 -- 172.21.15.102:0/2305945113 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c4406ac20 con 0x7f8c4414aa40 2024-03-23T03:50:33.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.312+0000 7f8c3b400640 1 -- 172.21.15.102:0/2305945113 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16357 (secure 0 0 0) 0x7f8c3c057d20 con 0x7f8c4414aa40 2024-03-23T03:50:33.313 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:33.313 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":81,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:08.942399+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:50:06.347556+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-23T03:50:06.924370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:50:33.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.315+0000 7f8c4a000640 1 -- 172.21.15.102:0/2305945113 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8c2406ac00 msgr2=0x7f8c2406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:33.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.315+0000 7f8c4a000640 1 --2- 172.21.15.102:0/2305945113 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8c2406ac00 0x7f8c2406d0c0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f8c34007eb0 tx=0x7f8c340073c0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.315+0000 7f8c4a000640 1 -- 172.21.15.102:0/2305945113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c4414aa40 msgr2=0x7f8c44162380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:33.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.315+0000 7f8c4a000640 1 --2- 172.21.15.102:0/2305945113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c4414aa40 0x7f8c44162380 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c00c7f0 tx=0x7f8c3c00ccb0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.315+0000 7f8c4a000640 1 -- 172.21.15.102:0/2305945113 shutdown_connections 2024-03-23T03:50:33.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.315+0000 7f8c4a000640 1 --2- 172.21.15.102:0/2305945113 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8c2406ac00 0x7f8c2406d0c0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.315+0000 7f8c4a000640 1 --2- 172.21.15.102:0/2305945113 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8c4414aa40 0x7f8c44162380 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.315+0000 7f8c4a000640 1 --2- 172.21.15.102:0/2305945113 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8c440ccec0 0x7f8c44161e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.315+0000 7f8c4a000640 1 --2- 172.21.15.102:0/2305945113 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8c440b37f0 0x7f8c440a9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.315+0000 7f8c4a000640 1 -- 172.21.15.102:0/2305945113 >> 172.21.15.102:0/2305945113 conn(0x7f8c4406a3d0 msgr2=0x7f8c440cc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:33.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.315+0000 7f8c4a000640 1 -- 172.21.15.102:0/2305945113 shutdown_connections 2024-03-23T03:50:33.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.315+0000 7f8c4a000640 1 -- 172.21.15.102:0/2305945113 wait complete. 2024-03-23T03:50:33.335 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:50:06.347556+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:50:06.924370+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:50:33.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:50:33.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.546+0000 7fe804600640 1 Processor -- start 2024-03-23T03:50:33.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.546+0000 7fe804600640 1 -- start start 2024-03-23T03:50:33.548 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.548+0000 7fe804600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7fc06ce90 0x7fe7fc06d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:33.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.548+0000 7fe804600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc0cdba0 con 0x7fe7fc0ce930 2024-03-23T03:50:33.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.548+0000 7fe804600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc0cde80 con 0x7fe7fc0add30 2024-03-23T03:50:33.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.548+0000 7fe804600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0ae1c0 con 0x7fe7fc06ce90 2024-03-23T03:50:33.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.548+0000 7fe801c00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe7fc0ce930 0x7fe7fc0cd3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60918/0 (socket says 172.21.15.102:60918) 2024-03-23T03:50:33.549 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.548+0000 7fe801c00640 1 -- 172.21.15.102:0/385947677 learned_addr learned my addr 172.21.15.102:0/385947677 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:33.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.549+0000 7fe802600640 1 --2- 172.21.15.102:0/385947677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7fc06ce90 0x7fe7fc06d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:33.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.549+0000 7fe7fbe00640 1 -- 172.21.15.102:0/385947677 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706222387 0 0) 0x7fe7fc0cdba0 con 0x7fe7fc0ce930 2024-03-23T03:50:33.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.549+0000 7fe7fbe00640 1 -- 172.21.15.102:0/385947677 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e4003520 con 0x7fe7fc0ce930 2024-03-23T03:50:33.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.549+0000 7fe7fbe00640 1 -- 172.21.15.102:0/385947677 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3902005936 0 0) 0x7fe7e4003520 con 0x7fe7fc0ce930 2024-03-23T03:50:33.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.549+0000 7fe7fbe00640 1 -- 172.21.15.102:0/385947677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7fc06ce90 msgr2=0x7fe7fc06d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:33.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.549+0000 7fe7fbe00640 1 --2- 172.21.15.102:0/385947677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7fc06ce90 0x7fe7fc06d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.549+0000 7fe7fbe00640 1 -- 172.21.15.102:0/385947677 >> v1:172.21.15.173:6789/0 conn(0x7fe7fc0add30 legacy=0x7fe7fc0ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.549+0000 7fe7fbe00640 1 -- 172.21.15.102:0/385947677 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc0ce160 con 0x7fe7fc0ce930 2024-03-23T03:50:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.549+0000 7fe7fbe00640 1 -- 172.21.15.102:0/385947677 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fe7f0001670 con 0x7fe7fc0ce930 2024-03-23T03:50:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.549+0000 7fe7fbe00640 1 -- 172.21.15.102:0/385947677 >> v1:172.21.15.102:6789/0 conn(0x7fe7fc0ce930 legacy=0x7fe7fc0cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.549+0000 7fe7fbe00640 1 --2- 172.21.15.102:0/385947677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7e40039a0 0x7fe7e4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.549+0000 7fe7fbe00640 1 --2- 172.21.15.102:0/385947677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7e40047c0 0x7fe7e4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.549+0000 7fe7fbe00640 1 --2- 172.21.15.102:0/385947677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7fc06ce90 0x7fe7e4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.549+0000 7fe7fbe00640 1 -- 172.21.15.102:0/385947677 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e400ba30 con 0x7fe7e40047c0 2024-03-23T03:50:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.549+0000 7fe7fbe00640 1 -- 172.21.15.102:0/385947677 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e400bb70 con 0x7fe7e40039a0 2024-03-23T03:50:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.549+0000 7fe7fbe00640 1 -- 172.21.15.102:0/385947677 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e400bd30 con 0x7fe7fc06ce90 2024-03-23T03:50:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.550+0000 7fe801200640 1 --2- 172.21.15.102:0/385947677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7e40047c0 0x7fe7e4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.550+0000 7fe801c00640 1 --2- 172.21.15.102:0/385947677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7e40039a0 0x7fe7e4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:33.552 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.550+0000 7fe802600640 1 --2- 172.21.15.102:0/385947677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7fc06ce90 0x7fe7e4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:33.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.550+0000 7fe801200640 1 -- 172.21.15.102:0/385947677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7fc06ce90 msgr2=0x7fe7e4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:33.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.550+0000 7fe801200640 1 --2- 172.21.15.102:0/385947677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7fc06ce90 0x7fe7e4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.550+0000 7fe801200640 1 -- 172.21.15.102:0/385947677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7e40039a0 msgr2=0x7fe7e4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:33.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.550+0000 7fe801200640 1 --2- 172.21.15.102:0/385947677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7e40039a0 0x7fe7e4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.550+0000 7fe801200640 1 -- 172.21.15.102:0/385947677 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e4003520 con 0x7fe7e40047c0 2024-03-23T03:50:33.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.550+0000 7fe802600640 1 --2- 172.21.15.102:0/385947677 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7fc06ce90 0x7fe7e4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:33.554 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.551+0000 7fe801200640 1 --2- 172.21.15.102:0/385947677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7e40047c0 0x7fe7e4004c40 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8001720 tx=0x7fe7e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d7611d78f4320b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.551+0000 7fe7fbe00640 1 -- 172.21.15.102:0/385947677 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e80052d0 con 0x7fe7e40047c0 2024-03-23T03:50:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.551+0000 7fe7fbe00640 1 -- 172.21.15.102:0/385947677 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e802e050 con 0x7fe7e40047c0 2024-03-23T03:50:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.551+0000 7fe7fbe00640 1 -- 172.21.15.102:0/385947677 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e80035f0 con 0x7fe7e40047c0 2024-03-23T03:50:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.552+0000 7fe804600640 1 -- 172.21.15.102:0/385947677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7e40047c0 msgr2=0x7fe7e4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.552+0000 7fe804600640 1 --2- 172.21.15.102:0/385947677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7e40047c0 0x7fe7e4004c40 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8001720 tx=0x7fe7e40057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.552+0000 7fe802600640 1 -- 172.21.15.102:0/385947677 reap_dead start 2024-03-23T03:50:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.552+0000 7fe804600640 1 -- 172.21.15.102:0/385947677 shutdown_connections 2024-03-23T03:50:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.552+0000 7fe804600640 1 -- 172.21.15.102:0/385947677 >> 172.21.15.102:0/385947677 conn(0x7fe7fc06a0b0 msgr2=0x7fe7fc06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.552+0000 7fe804600640 1 -- 172.21.15.102:0/385947677 shutdown_connections 2024-03-23T03:50:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.552+0000 7fe804600640 1 -- 172.21.15.102:0/385947677 wait complete. 2024-03-23T03:50:33.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.554+0000 7fe804600640 1 Processor -- start 2024-03-23T03:50:33.556 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.554+0000 7fe804600640 1 -- start start 2024-03-23T03:50:33.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.555+0000 7fe804600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7e40047c0 0x7fe7fc159250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:33.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.555+0000 7fe804600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7fc0b16c0 0x7fe7fc14d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:33.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.555+0000 7fe804600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7fc15d430 0x7fe7fc0cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:33.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.555+0000 7fe804600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc1f64a0 con 0x7fe7fc15d430 2024-03-23T03:50:33.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.555+0000 7fe804600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc159790 con 0x7fe7e40047c0 2024-03-23T03:50:33.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.555+0000 7fe804600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc1f5820 con 0x7fe7fc0b16c0 2024-03-23T03:50:33.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.555+0000 7fe802600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7fc15d430 0x7fe7fc0cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:33.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.555+0000 7fe802600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7fc15d430 0x7fe7fc0cbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50632/0 (socket says 172.21.15.102:50632) 2024-03-23T03:50:33.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.555+0000 7fe802600640 1 -- 172.21.15.102:0/2735800527 learned_addr learned my addr 172.21.15.102:0/2735800527 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:33.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.556+0000 7fe801c00640 1 --2- 172.21.15.102:0/2735800527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7e40047c0 0x7fe7fc159250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:33.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.556+0000 7fe801200640 1 --2- 172.21.15.102:0/2735800527 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7fc0b16c0 0x7fe7fc14d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:33.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.556+0000 7fe802600640 1 -- 172.21.15.102:0/2735800527 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7fc0b16c0 msgr2=0x7fe7fc14d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:33.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.556+0000 7fe802600640 1 --2- 172.21.15.102:0/2735800527 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7fc0b16c0 0x7fe7fc14d360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.556+0000 7fe802600640 1 -- 172.21.15.102:0/2735800527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7e40047c0 msgr2=0x7fe7fc159250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:33.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.556+0000 7fe802600640 1 --2- 172.21.15.102:0/2735800527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7e40047c0 0x7fe7fc159250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.556+0000 7fe802600640 1 -- 172.21.15.102:0/2735800527 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc1f5b00 con 0x7fe7fc15d430 2024-03-23T03:50:33.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.556+0000 7fe802600640 1 --2- 172.21.15.102:0/2735800527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7fc15d430 0x7fe7fc0cbc50 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc0ccb90 tx=0x7fe7f400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:33.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.556+0000 7fe7f9600640 1 -- 172.21.15.102:0/2735800527 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4017400 con 0x7fe7fc15d430 2024-03-23T03:50:33.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.556+0000 7fe7f9600640 1 -- 172.21.15.102:0/2735800527 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f4011040 con 0x7fe7fc15d430 2024-03-23T03:50:33.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.556+0000 7fe7f9600640 1 -- 172.21.15.102:0/2735800527 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4017780 con 0x7fe7fc15d430 2024-03-23T03:50:33.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.556+0000 7fe804600640 1 -- 172.21.15.102:0/2735800527 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7fc1f5de0 con 0x7fe7fc15d430 2024-03-23T03:50:33.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.557+0000 7fe804600640 1 -- 172.21.15.102:0/2735800527 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc14d8a0 con 0x7fe7fc15d430 2024-03-23T03:50:33.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.558+0000 7fe804600640 1 -- 172.21.15.102:0/2735800527 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc1f65e0 con 0x7fe7fc15d430 2024-03-23T03:50:33.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.558+0000 7fe7f9600640 1 -- 172.21.15.102:0/2735800527 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe7f40203b0 con 0x7fe7fc15d430 2024-03-23T03:50:33.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.559+0000 7fe7f9600640 1 --2- 172.21.15.102:0/2735800527 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe7dc06ac30 0x7fe7dc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:33.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.559+0000 7fe7f9600640 1 -- 172.21.15.102:0/2735800527 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe7f4089740 con 0x7fe7fc15d430 2024-03-23T03:50:33.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.560+0000 7fe801c00640 1 --2- 172.21.15.102:0/2735800527 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe7dc06ac30 0x7fe7dc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:33.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.560+0000 7fe801c00640 1 --2- 172.21.15.102:0/2735800527 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe7dc06ac30 0x7fe7dc06d0f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0001050 tx=0x7fe7f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:33.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.564+0000 7fe7f9600640 1 -- 172.21.15.102:0/2735800527 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe7f4054740 con 0x7fe7fc15d430 2024-03-23T03:50:33.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.759+0000 7fe804600640 1 -- 172.21.15.102:0/2735800527 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe7fc073390 con 0x7fe7fc15d430 2024-03-23T03:50:33.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.760+0000 7fe7f9600640 1 -- 172.21.15.102:0/2735800527 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe7f4057d60 con 0x7fe7fc15d430 2024-03-23T03:50:33.761 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:50:33.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.762+0000 7fe804600640 1 -- 172.21.15.102:0/2735800527 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe7dc06ac30 msgr2=0x7fe7dc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:33.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.762+0000 7fe804600640 1 --2- 172.21.15.102:0/2735800527 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe7dc06ac30 0x7fe7dc06d0f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0001050 tx=0x7fe7f0009040 comp rx=0 tx=0).stop 2024-03-23T03:50:33.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.762+0000 7fe804600640 1 -- 172.21.15.102:0/2735800527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7fc15d430 msgr2=0x7fe7fc0cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:33.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.762+0000 7fe804600640 1 --2- 172.21.15.102:0/2735800527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7fc15d430 0x7fe7fc0cbc50 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc0ccb90 tx=0x7fe7f400f4a0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.763+0000 7fe804600640 1 -- 172.21.15.102:0/2735800527 shutdown_connections 2024-03-23T03:50:33.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.763+0000 7fe804600640 1 --2- 172.21.15.102:0/2735800527 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe7dc06ac30 0x7fe7dc06d0f0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.763+0000 7fe804600640 1 --2- 172.21.15.102:0/2735800527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7fc15d430 0x7fe7fc0cbc50 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.763+0000 7fe804600640 1 --2- 172.21.15.102:0/2735800527 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7fc0b16c0 0x7fe7fc14d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.763+0000 7fe804600640 1 --2- 172.21.15.102:0/2735800527 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7e40047c0 0x7fe7fc159250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:33.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.763+0000 7fe804600640 1 -- 172.21.15.102:0/2735800527 >> 172.21.15.102:0/2735800527 conn(0x7fe7fc06a0b0 msgr2=0x7fe7fc15c260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:33.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.763+0000 7fe804600640 1 -- 172.21.15.102:0/2735800527 shutdown_connections 2024-03-23T03:50:33.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:33.763+0000 7fe804600640 1 -- 172.21.15.102:0/2735800527 wait complete. 2024-03-23T03:50:33.782 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:50:34.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.000+0000 7f10ba600640 1 Processor -- start 2024-03-23T03:50:34.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.000+0000 7f10ba600640 1 -- start start 2024-03-23T03:50:34.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.001+0000 7f10ba600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f10b40ca7b0 0x7f10b40cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.002+0000 7f10ba600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b4159c10 con 0x7f10b4159830 2024-03-23T03:50:34.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.002+0000 7f10ba600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10b40cd840 con 0x7f10b40ce930 2024-03-23T03:50:34.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.002+0000 7f10ba600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10b406de50 con 0x7f10b40ca7b0 2024-03-23T03:50:34.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.002+0000 7f10b3400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f10b4159830 0x7f10b40cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60934/0 (socket says 172.21.15.102:60934) 2024-03-23T03:50:34.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.002+0000 7f10b3400640 1 -- 172.21.15.102:0/3587115758 learned_addr learned my addr 172.21.15.102:0/3587115758 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:34.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.002+0000 7f10aaa00640 1 --2- 172.21.15.102:0/3587115758 >> v2:172.21.15.173:3300/0 conn(0x7f10b40ca7b0 0x7f10b40cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.002+0000 7f10b2a00640 1 -- 172.21.15.102:0/3587115758 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311559294 0 0) 0x7f10b4159c10 con 0x7f10b4159830 2024-03-23T03:50:34.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.002+0000 7f10b2a00640 1 -- 172.21.15.102:0/3587115758 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1098003520 con 0x7f10b4159830 2024-03-23T03:50:34.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.003+0000 7f10b2a00640 1 -- 172.21.15.102:0/3587115758 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 47997090 0 0) 0x7f1098003520 con 0x7f10b4159830 2024-03-23T03:50:34.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.003+0000 7f10b2a00640 1 -- 172.21.15.102:0/3587115758 >> v2:172.21.15.173:3300/0 conn(0x7f10b40ca7b0 msgr2=0x7f10b40cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:34.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.003+0000 7f10b2a00640 1 --2- 172.21.15.102:0/3587115758 >> v2:172.21.15.173:3300/0 conn(0x7f10b40ca7b0 0x7f10b40cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.003+0000 7f10b2a00640 1 -- 172.21.15.102:0/3587115758 >> v1:172.21.15.173:6789/0 conn(0x7f10b40ce930 legacy=0x7f10b40b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.003+0000 7f10b2a00640 1 -- 172.21.15.102:0/3587115758 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b40ccc30 con 0x7f10b4159830 2024-03-23T03:50:34.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.003+0000 7f10b2a00640 1 -- 172.21.15.102:0/3587115758 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f10a4001670 con 0x7f10b4159830 2024-03-23T03:50:34.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.003+0000 7f10b2a00640 1 -- 172.21.15.102:0/3587115758 >> v1:172.21.15.102:6789/0 conn(0x7f10b4159830 legacy=0x7f10b40cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.003+0000 7f10b2a00640 1 --2- 172.21.15.102:0/3587115758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10980039c0 0x7f1098003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.003+0000 7f10b2a00640 1 --2- 172.21.15.102:0/3587115758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10980047c0 0x7f1098004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.003+0000 7f10b2a00640 1 --2- 172.21.15.102:0/3587115758 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10980051d0 0x7f109800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.003+0000 7f10b2a00640 1 -- 172.21.15.102:0/3587115758 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109800c0e0 con 0x7f10980039c0 2024-03-23T03:50:34.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.003+0000 7f10b2a00640 1 -- 172.21.15.102:0/3587115758 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109800c250 con 0x7f10980047c0 2024-03-23T03:50:34.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.003+0000 7f10b2a00640 1 -- 172.21.15.102:0/3587115758 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f109800c410 con 0x7f10980051d0 2024-03-23T03:50:34.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.003+0000 7f10b3400640 1 --2- 172.21.15.102:0/3587115758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10980039c0 0x7f1098003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.004+0000 7f10b3e00640 1 --2- 172.21.15.102:0/3587115758 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10980051d0 0x7f109800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.004+0000 7f10b3400640 1 -- 172.21.15.102:0/3587115758 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10980051d0 msgr2=0x7f109800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:34.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.004+0000 7f10aaa00640 1 --2- 172.21.15.102:0/3587115758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10980047c0 0x7f1098004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.004+0000 7f10b3400640 1 --2- 172.21.15.102:0/3587115758 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10980051d0 0x7f109800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.004+0000 7f10b3400640 1 -- 172.21.15.102:0/3587115758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10980047c0 msgr2=0x7f1098004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:34.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.004+0000 7f10b3400640 1 --2- 172.21.15.102:0/3587115758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10980047c0 0x7f1098004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.004+0000 7f10b3400640 1 -- 172.21.15.102:0/3587115758 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1098003520 con 0x7f10980039c0 2024-03-23T03:50:34.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.004+0000 7f10b3e00640 1 -- 172.21.15.102:0/3587115758 reap_dead start 2024-03-23T03:50:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.005+0000 7f10b3400640 1 --2- 172.21.15.102:0/3587115758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10980039c0 0x7f1098003de0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f10a400c0e0 tx=0x7f10980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fda3939605e600a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.005+0000 7f10b2a00640 1 -- 172.21.15.102:0/3587115758 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a4005ad0 con 0x7f10980039c0 2024-03-23T03:50:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.005+0000 7f10b2a00640 1 -- 172.21.15.102:0/3587115758 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10a4005c30 con 0x7f10980039c0 2024-03-23T03:50:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.005+0000 7f10b2a00640 1 -- 172.21.15.102:0/3587115758 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10a402ca50 con 0x7f10980039c0 2024-03-23T03:50:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.006+0000 7f10ba600640 1 -- 172.21.15.102:0/3587115758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10980039c0 msgr2=0x7f1098003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.006+0000 7f10ba600640 1 --2- 172.21.15.102:0/3587115758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10980039c0 0x7f1098003de0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f10a400c0e0 tx=0x7f10980057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.006+0000 7f10ba600640 1 -- 172.21.15.102:0/3587115758 shutdown_connections 2024-03-23T03:50:34.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.006+0000 7f10ba600640 1 --2- 172.21.15.102:0/3587115758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10980039c0 0x7f1098003de0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.006+0000 7f10ba600640 1 -- 172.21.15.102:0/3587115758 >> 172.21.15.102:0/3587115758 conn(0x7f10b406a0b0 msgr2=0x7f10b406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:34.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.006+0000 7f10ba600640 1 -- 172.21.15.102:0/3587115758 shutdown_connections 2024-03-23T03:50:34.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.006+0000 7f10ba600640 1 -- 172.21.15.102:0/3587115758 wait complete. 2024-03-23T03:50:34.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.008+0000 7f10ba600640 1 Processor -- start 2024-03-23T03:50:34.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.008+0000 7f10ba600640 1 -- start start 2024-03-23T03:50:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.009+0000 7f10ba600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10980039c0 0x7f10b4070b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.009+0000 7f10ba600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b41f2e60 0x7f10b41f4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.009+0000 7f10ba600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10b41561e0 0x7f10b4156690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.009+0000 7f10ba600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b4156bd0 con 0x7f10b41f2e60 2024-03-23T03:50:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.009+0000 7f10ba600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10b415b970 con 0x7f10b41561e0 2024-03-23T03:50:34.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.009+0000 7f10ba600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10b4153380 con 0x7f10980039c0 2024-03-23T03:50:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.009+0000 7f10aaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b41f2e60 0x7f10b41f4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.009+0000 7f10aaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b41f2e60 0x7f10b41f4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50638/0 (socket says 172.21.15.102:50638) 2024-03-23T03:50:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.009+0000 7f10aaa00640 1 -- 172.21.15.102:0/3510179530 learned_addr learned my addr 172.21.15.102:0/3510179530 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.009+0000 7f10b3400640 1 --2- 172.21.15.102:0/3510179530 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10980039c0 0x7f10b4070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.009+0000 7f10aaa00640 1 -- 172.21.15.102:0/3510179530 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10980039c0 msgr2=0x7f10b4070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.009+0000 7f10b3e00640 1 --2- 172.21.15.102:0/3510179530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10b41561e0 0x7f10b4156690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.009+0000 7f10aaa00640 1 --2- 172.21.15.102:0/3510179530 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10980039c0 0x7f10b4070b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.009+0000 7f10aaa00640 1 -- 172.21.15.102:0/3510179530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10b41561e0 msgr2=0x7f10b4156690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.009+0000 7f10aaa00640 1 --2- 172.21.15.102:0/3510179530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10b41561e0 0x7f10b4156690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.009+0000 7f10aaa00640 1 -- 172.21.15.102:0/3510179530 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b4153660 con 0x7f10b41f2e60 2024-03-23T03:50:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.010+0000 7f10aaa00640 1 --2- 172.21.15.102:0/3510179530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b41f2e60 0x7f10b41f4bb0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f109c00ed20 tx=0x7f109c00c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.010+0000 7f10abe00640 1 -- 172.21.15.102:0/3510179530 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c019480 con 0x7f10b41f2e60 2024-03-23T03:50:34.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.010+0000 7f10abe00640 1 -- 172.21.15.102:0/3510179530 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109c0195e0 con 0x7f10b41f2e60 2024-03-23T03:50:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.010+0000 7f10abe00640 1 -- 172.21.15.102:0/3510179530 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109c019950 con 0x7f10b41f2e60 2024-03-23T03:50:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.010+0000 7f10ba600640 1 -- 172.21.15.102:0/3510179530 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10b4153940 con 0x7f10b41f2e60 2024-03-23T03:50:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.010+0000 7f10ba600640 1 -- 172.21.15.102:0/3510179530 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10b4159f80 con 0x7f10b41f2e60 2024-03-23T03:50:34.013 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.011+0000 7f10ba600640 1 -- 172.21.15.102:0/3510179530 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10b4153e60 con 0x7f10b41f2e60 2024-03-23T03:50:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.012+0000 7f10abe00640 1 -- 172.21.15.102:0/3510179530 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f109c019ab0 con 0x7f10b41f2e60 2024-03-23T03:50:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.013+0000 7f10abe00640 1 --2- 172.21.15.102:0/3510179530 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f109006ac30 0x7f109006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.013+0000 7f10abe00640 1 -- 172.21.15.102:0/3510179530 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f109c014070 con 0x7f10b41f2e60 2024-03-23T03:50:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.013+0000 7f10b3400640 1 --2- 172.21.15.102:0/3510179530 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f109006ac30 0x7f109006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.014+0000 7f10b3400640 1 --2- 172.21.15.102:0/3510179530 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f109006ac30 0x7f109006d0f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f10a4034000 tx=0x7f10a4009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:34.017 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.017+0000 7f10abe00640 1 -- 172.21.15.102:0/3510179530 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f109c054e60 con 0x7f10b41f2e60 2024-03-23T03:50:34.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.212+0000 7f10ba600640 1 -- 172.21.15.102:0/3510179530 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f10b4073390 con 0x7f10b41f2e60 2024-03-23T03:50:34.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.213+0000 7f10abe00640 1 -- 172.21.15.102:0/3510179530 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+11 (secure 0 0 0) 0x7f109c01e020 con 0x7f10b41f2e60 2024-03-23T03:50:34.213 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:50:34.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.215+0000 7f10ba600640 1 -- 172.21.15.102:0/3510179530 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f109006ac30 msgr2=0x7f109006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.215+0000 7f10ba600640 1 --2- 172.21.15.102:0/3510179530 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f109006ac30 0x7f109006d0f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f10a4034000 tx=0x7f10a4009030 comp rx=0 tx=0).stop 2024-03-23T03:50:34.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.215+0000 7f10ba600640 1 -- 172.21.15.102:0/3510179530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b41f2e60 msgr2=0x7f10b41f4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.215+0000 7f10ba600640 1 --2- 172.21.15.102:0/3510179530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b41f2e60 0x7f10b41f4bb0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f109c00ed20 tx=0x7f109c00c6b0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.215+0000 7f10ba600640 1 -- 172.21.15.102:0/3510179530 shutdown_connections 2024-03-23T03:50:34.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.215+0000 7f10ba600640 1 --2- 172.21.15.102:0/3510179530 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f109006ac30 0x7f109006d0f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.215+0000 7f10ba600640 1 --2- 172.21.15.102:0/3510179530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f10b41561e0 0x7f10b4156690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.215+0000 7f10ba600640 1 --2- 172.21.15.102:0/3510179530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10b41f2e60 0x7f10b41f4bb0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.215+0000 7f10ba600640 1 --2- 172.21.15.102:0/3510179530 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f10980039c0 0x7f10b4070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.216+0000 7f10ba600640 1 -- 172.21.15.102:0/3510179530 >> 172.21.15.102:0/3510179530 conn(0x7f10b406a0b0 msgr2=0x7f10b40b2a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:34.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.216+0000 7f10ba600640 1 -- 172.21.15.102:0/3510179530 shutdown_connections 2024-03-23T03:50:34.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.216+0000 7f10ba600640 1 -- 172.21.15.102:0/3510179530 wait complete. 2024-03-23T03:50:34.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:50:34.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.449+0000 7f468b200640 1 Processor -- start 2024-03-23T03:50:34.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.449+0000 7f468b200640 1 -- start start 2024-03-23T03:50:34.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.451+0000 7f468b200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46840c3620 0x7f46840c3a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.451+0000 7f468b200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46840afd20 con 0x7f4684098a40 2024-03-23T03:50:34.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.451+0000 7f468b200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46840c3f40 con 0x7f46840ae000 2024-03-23T03:50:34.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.451+0000 7f468b200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4684099770 con 0x7f46840c3620 2024-03-23T03:50:34.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.451+0000 7f4683400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4684098a40 0x7f4684098ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60950/0 (socket says 172.21.15.102:60950) 2024-03-23T03:50:34.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.451+0000 7f4683400640 1 -- 172.21.15.102:0/3652508003 learned_addr learned my addr 172.21.15.102:0/3652508003 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:34.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.452+0000 7f4683e00640 1 --2- 172.21.15.102:0/3652508003 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46840c3620 0x7f46840c3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.452+0000 7f4682a00640 1 -- 172.21.15.102:0/3652508003 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138181685 0 0) 0x7f46840afd20 con 0x7f4684098a40 2024-03-23T03:50:34.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.452+0000 7f4682a00640 1 -- 172.21.15.102:0/3652508003 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4668003520 con 0x7f4684098a40 2024-03-23T03:50:34.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.452+0000 7f4682a00640 1 -- 172.21.15.102:0/3652508003 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 47605410 0 0) 0x7f4668003520 con 0x7f4684098a40 2024-03-23T03:50:34.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.452+0000 7f4682a00640 1 -- 172.21.15.102:0/3652508003 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46840c3620 msgr2=0x7f46840c3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:34.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.452+0000 7f4682a00640 1 --2- 172.21.15.102:0/3652508003 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46840c3620 0x7f46840c3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.452+0000 7f4682a00640 1 -- 172.21.15.102:0/3652508003 >> v1:172.21.15.173:6789/0 conn(0x7f46840ae000 legacy=0x7f46840ae460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.452+0000 7f4682a00640 1 -- 172.21.15.102:0/3652508003 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46840ad520 con 0x7f4684098a40 2024-03-23T03:50:34.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.452+0000 7f4682a00640 1 -- 172.21.15.102:0/3652508003 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f466c001670 con 0x7f4684098a40 2024-03-23T03:50:34.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.453+0000 7f4682a00640 1 -- 172.21.15.102:0/3652508003 >> v1:172.21.15.102:6789/0 conn(0x7f4684098a40 legacy=0x7f4684098ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.453+0000 7f4682a00640 1 --2- 172.21.15.102:0/3652508003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46680039a0 0x7f4668003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.453+0000 7f4682a00640 1 --2- 172.21.15.102:0/3652508003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4668004350 0x7f4668004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.453+0000 7f4682a00640 1 --2- 172.21.15.102:0/3652508003 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46840c3620 0x7f4668005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.453+0000 7f4682a00640 1 -- 172.21.15.102:0/3652508003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4684099770 con 0x7f46680039a0 2024-03-23T03:50:34.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.453+0000 7f4682a00640 1 -- 172.21.15.102:0/3652508003 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466800b980 con 0x7f4668004350 2024-03-23T03:50:34.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.453+0000 7f4682a00640 1 -- 172.21.15.102:0/3652508003 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466800bb40 con 0x7f46840c3620 2024-03-23T03:50:34.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.453+0000 7f4683e00640 1 --2- 172.21.15.102:0/3652508003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46680039a0 0x7f4668003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.453+0000 7f4683e00640 1 -- 172.21.15.102:0/3652508003 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46840c3620 msgr2=0x7f4668005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:34.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.453+0000 7f4683e00640 1 --2- 172.21.15.102:0/3652508003 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46840c3620 0x7f4668005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.453+0000 7f4683e00640 1 -- 172.21.15.102:0/3652508003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4668004350 msgr2=0x7f4668004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:34.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.453+0000 7f4683e00640 1 --2- 172.21.15.102:0/3652508003 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4668004350 0x7f4668004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.453+0000 7f4683e00640 1 -- 172.21.15.102:0/3652508003 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4668003520 con 0x7f46680039a0 2024-03-23T03:50:34.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.454+0000 7f4683e00640 1 --2- 172.21.15.102:0/3652508003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46680039a0 0x7f4668003d80 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f4674009fc0 tx=0x7f46680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592536d4cfd23a9c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:34.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.455+0000 7f4682a00640 1 -- 172.21.15.102:0/3652508003 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467402cc60 con 0x7f46680039a0 2024-03-23T03:50:34.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.455+0000 7f4682a00640 1 -- 172.21.15.102:0/3652508003 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467400f040 con 0x7f46680039a0 2024-03-23T03:50:34.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.455+0000 7f4682a00640 1 -- 172.21.15.102:0/3652508003 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4674034440 con 0x7f46680039a0 2024-03-23T03:50:34.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.455+0000 7f468b200640 1 -- 172.21.15.102:0/3652508003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46680039a0 msgr2=0x7f4668003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.455+0000 7f468b200640 1 --2- 172.21.15.102:0/3652508003 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46680039a0 0x7f4668003d80 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f4674009fc0 tx=0x7f46680057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.455+0000 7f4689200640 1 -- 172.21.15.102:0/3652508003 reap_dead start 2024-03-23T03:50:34.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.455+0000 7f468b200640 1 -- 172.21.15.102:0/3652508003 shutdown_connections 2024-03-23T03:50:34.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.455+0000 7f468b200640 1 -- 172.21.15.102:0/3652508003 >> 172.21.15.102:0/3652508003 conn(0x7f46840b2140 msgr2=0x7f46840b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:34.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.456+0000 7f468b200640 1 -- 172.21.15.102:0/3652508003 shutdown_connections 2024-03-23T03:50:34.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.456+0000 7f468b200640 1 -- 172.21.15.102:0/3652508003 wait complete. 2024-03-23T03:50:34.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.457+0000 7f468b200640 1 Processor -- start 2024-03-23T03:50:34.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.457+0000 7f468b200640 1 -- start start 2024-03-23T03:50:34.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.458+0000 7f468b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46680039a0 0x7f46840b5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.458+0000 7f468b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f468415db50 0x7f46841f9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.458+0000 7f468b200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46841f9f40 0x7f46841f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.458+0000 7f468b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46840b59b0 con 0x7f468415db50 2024-03-23T03:50:34.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.458+0000 7f468b200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46841fa3c0 con 0x7f46680039a0 2024-03-23T03:50:34.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.458+0000 7f468b200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46840b49b0 con 0x7f46841f9f40 2024-03-23T03:50:34.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.459+0000 7f4683400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f468415db50 0x7f46841f9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.459+0000 7f4683400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f468415db50 0x7f46841f9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50652/0 (socket says 172.21.15.102:50652) 2024-03-23T03:50:34.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.459+0000 7f4683400640 1 -- 172.21.15.102:0/2490944515 learned_addr learned my addr 172.21.15.102:0/2490944515 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:34.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.459+0000 7f4683e00640 1 --2- 172.21.15.102:0/2490944515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46680039a0 0x7f46840b5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.459+0000 7f4689200640 1 --2- 172.21.15.102:0/2490944515 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46841f9f40 0x7f46841f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.459+0000 7f4683400640 1 -- 172.21.15.102:0/2490944515 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46841f9f40 msgr2=0x7f46841f6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.459+0000 7f4683400640 1 --2- 172.21.15.102:0/2490944515 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46841f9f40 0x7f46841f6480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.459+0000 7f4683400640 1 -- 172.21.15.102:0/2490944515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46680039a0 msgr2=0x7f46840b5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.459+0000 7f4683400640 1 --2- 172.21.15.102:0/2490944515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46680039a0 0x7f46840b5ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.459+0000 7f4683400640 1 -- 172.21.15.102:0/2490944515 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46841fa830 con 0x7f468415db50 2024-03-23T03:50:34.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.459+0000 7f4683400640 1 --2- 172.21.15.102:0/2490944515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f468415db50 0x7f46841f9a00 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f466c000f60 tx=0x7f466c00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:34.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.460+0000 7f467be00640 1 -- 172.21.15.102:0/2490944515 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c0074e0 con 0x7f468415db50 2024-03-23T03:50:34.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.460+0000 7f467be00640 1 -- 172.21.15.102:0/2490944515 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466c007640 con 0x7f468415db50 2024-03-23T03:50:34.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.460+0000 7f467be00640 1 -- 172.21.15.102:0/2490944515 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466c004a00 con 0x7f468415db50 2024-03-23T03:50:34.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.460+0000 7f468b200640 1 -- 172.21.15.102:0/2490944515 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46840b46d0 con 0x7f468415db50 2024-03-23T03:50:34.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.460+0000 7f468b200640 1 -- 172.21.15.102:0/2490944515 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46841f7950 con 0x7f468415db50 2024-03-23T03:50:34.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.461+0000 7f468b200640 1 -- 172.21.15.102:0/2490944515 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4684066bb0 con 0x7f468415db50 2024-03-23T03:50:34.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.462+0000 7f467be00640 1 -- 172.21.15.102:0/2490944515 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f466c004b60 con 0x7f468415db50 2024-03-23T03:50:34.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.463+0000 7f467be00640 1 --2- 172.21.15.102:0/2490944515 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f466006ac30 0x7f466006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.463+0000 7f467be00640 1 -- 172.21.15.102:0/2490944515 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f466c088a00 con 0x7f468415db50 2024-03-23T03:50:34.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.463+0000 7f4683e00640 1 --2- 172.21.15.102:0/2490944515 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f466006ac30 0x7f466006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.464+0000 7f4683e00640 1 --2- 172.21.15.102:0/2490944515 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f466006ac30 0x7f466006d0f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f467402c600 tx=0x7f467402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:34.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.467+0000 7f467be00640 1 -- 172.21.15.102:0/2490944515 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f466c053a00 con 0x7f468415db50 2024-03-23T03:50:34.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.668+0000 7f468b200640 1 -- 172.21.15.102:0/2490944515 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f46840b2fa0 con 0x7f468415db50 2024-03-23T03:50:34.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.669+0000 7f467be00640 1 -- 172.21.15.102:0/2490944515 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+11 (secure 0 0 0) 0x7f466c018020 con 0x7f468415db50 2024-03-23T03:50:34.670 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:50:34.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.672+0000 7f468b200640 1 -- 172.21.15.102:0/2490944515 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f466006ac30 msgr2=0x7f466006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.672+0000 7f468b200640 1 --2- 172.21.15.102:0/2490944515 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f466006ac30 0x7f466006d0f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f467402c600 tx=0x7f467402e000 comp rx=0 tx=0).stop 2024-03-23T03:50:34.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.672+0000 7f468b200640 1 -- 172.21.15.102:0/2490944515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f468415db50 msgr2=0x7f46841f9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.672+0000 7f468b200640 1 --2- 172.21.15.102:0/2490944515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f468415db50 0x7f46841f9a00 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f466c000f60 tx=0x7f466c00ccd0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.672+0000 7f468b200640 1 -- 172.21.15.102:0/2490944515 shutdown_connections 2024-03-23T03:50:34.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.672+0000 7f468b200640 1 --2- 172.21.15.102:0/2490944515 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f466006ac30 0x7f466006d0f0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.672+0000 7f468b200640 1 --2- 172.21.15.102:0/2490944515 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46841f9f40 0x7f46841f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.672+0000 7f468b200640 1 --2- 172.21.15.102:0/2490944515 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f468415db50 0x7f46841f9a00 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.672+0000 7f468b200640 1 --2- 172.21.15.102:0/2490944515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46680039a0 0x7f46840b5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.672+0000 7f468b200640 1 -- 172.21.15.102:0/2490944515 >> 172.21.15.102:0/2490944515 conn(0x7f46840b2140 msgr2=0x7f46840cf0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:34.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.672+0000 7f468b200640 1 -- 172.21.15.102:0/2490944515 shutdown_connections 2024-03-23T03:50:34.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.672+0000 7f468b200640 1 -- 172.21.15.102:0/2490944515 wait complete. 2024-03-23T03:50:34.693 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:50:34.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.903+0000 7fe47bc00640 1 Processor -- start 2024-03-23T03:50:34.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.904+0000 7fe47bc00640 1 -- start start 2024-03-23T03:50:34.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.905+0000 7fe47bc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe4740b39b0 0x7fe4740b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.905+0000 7fe47bc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe47415c400 0x7fe47415c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.905+0000 7fe47bc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe47406c6f0 con 0x7fe47415b560 2024-03-23T03:50:34.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.905+0000 7fe47bc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4740b4370 con 0x7fe4740b39b0 2024-03-23T03:50:34.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.905+0000 7fe47bc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47406c9d0 con 0x7fe47415c400 2024-03-23T03:50:34.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.905+0000 7fe473e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe4740b39b0 0x7fe4740b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.905+0000 7fe479200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fe47415b560 0x7fe47415b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60954/0 (socket says 172.21.15.102:60954) 2024-03-23T03:50:34.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.905+0000 7fe479200640 1 -- 172.21.15.102:0/1353872386 learned_addr learned my addr 172.21.15.102:0/1353872386 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.906+0000 7fe479c00640 1 --2- 172.21.15.102:0/1353872386 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe47415c400 0x7fe47415c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.906+0000 7fe473e00640 1 -- 172.21.15.102:0/1353872386 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe47415c400 msgr2=0x7fe47415c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.906+0000 7fe473e00640 1 --2- 172.21.15.102:0/1353872386 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe47415c400 0x7fe47415c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.906+0000 7fe473e00640 1 -- 172.21.15.102:0/1353872386 >> v1:172.21.15.102:6789/0 conn(0x7fe47415b560 legacy=0x7fe47415b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.906+0000 7fe473e00640 1 -- 172.21.15.102:0/1353872386 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47406ccb0 con 0x7fe4740b39b0 2024-03-23T03:50:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.906+0000 7fe473400640 1 -- 172.21.15.102:0/1353872386 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287501684 0 0) 0x7fe47406c6f0 con 0x7fe47415b560 2024-03-23T03:50:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.906+0000 7fe479c00640 1 --2- 172.21.15.102:0/1353872386 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe47415c400 0x7fe47415c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.907+0000 7fe473e00640 1 --2- 172.21.15.102:0/1353872386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4740b39b0 0x7fe4740b3e30 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fe4600097a0 tx=0x7fe46002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54af9765d53c52f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.907+0000 7fe473400640 1 -- 172.21.15.102:0/1353872386 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46002d040 con 0x7fe4740b39b0 2024-03-23T03:50:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.907+0000 7fe473400640 1 -- 172.21.15.102:0/1353872386 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe460031810 con 0x7fe4740b39b0 2024-03-23T03:50:34.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.907+0000 7fe473400640 1 -- 172.21.15.102:0/1353872386 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe460031ba0 con 0x7fe4740b39b0 2024-03-23T03:50:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.908+0000 7fe47bc00640 1 -- 172.21.15.102:0/1353872386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4740b39b0 msgr2=0x7fe4740b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.908+0000 7fe47bc00640 1 --2- 172.21.15.102:0/1353872386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4740b39b0 0x7fe4740b3e30 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fe4600097a0 tx=0x7fe46002b8b0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.908+0000 7fe47bc00640 1 -- 172.21.15.102:0/1353872386 shutdown_connections 2024-03-23T03:50:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.908+0000 7fe47bc00640 1 --2- 172.21.15.102:0/1353872386 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe47415c400 0x7fe47415c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.908+0000 7fe47bc00640 1 --2- 172.21.15.102:0/1353872386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4740b39b0 0x7fe4740b3e30 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.908+0000 7fe47bc00640 1 -- 172.21.15.102:0/1353872386 >> 172.21.15.102:0/1353872386 conn(0x7fe47406a3d0 msgr2=0x7fe4740b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:34.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.908+0000 7fe47bc00640 1 -- 172.21.15.102:0/1353872386 shutdown_connections 2024-03-23T03:50:34.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.908+0000 7fe47bc00640 1 -- 172.21.15.102:0/1353872386 wait complete. 2024-03-23T03:50:34.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.910+0000 7fe47bc00640 1 Processor -- start 2024-03-23T03:50:34.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.910+0000 7fe47bc00640 1 -- start start 2024-03-23T03:50:34.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.911+0000 7fe47bc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4740b39b0 0x7fe474149d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.911+0000 7fe47bc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe47415b560 0x7fe4741f6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.911+0000 7fe47bc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe47415c400 0x7fe4740af360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.911+0000 7fe47bc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4741f6680 con 0x7fe4740b39b0 2024-03-23T03:50:34.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.911+0000 7fe47bc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4740b1940 con 0x7fe47415c400 2024-03-23T03:50:34.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.911+0000 7fe47bc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4740b1c20 con 0x7fe47415b560 2024-03-23T03:50:34.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.911+0000 7fe479200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4740b39b0 0x7fe474149d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.911+0000 7fe479200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4740b39b0 0x7fe474149d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50664/0 (socket says 172.21.15.102:50664) 2024-03-23T03:50:34.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.911+0000 7fe479200640 1 -- 172.21.15.102:0/1982723880 learned_addr learned my addr 172.21.15.102:0/1982723880 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:34.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.912+0000 7fe473e00640 1 --2- 172.21.15.102:0/1982723880 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe47415b560 0x7fe4741f6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.912+0000 7fe479c00640 1 --2- 172.21.15.102:0/1982723880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe47415c400 0x7fe4740af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.912+0000 7fe479200640 1 -- 172.21.15.102:0/1982723880 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe47415b560 msgr2=0x7fe4741f6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.912+0000 7fe479200640 1 --2- 172.21.15.102:0/1982723880 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe47415b560 0x7fe4741f6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.912+0000 7fe479200640 1 -- 172.21.15.102:0/1982723880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe47415c400 msgr2=0x7fe4740af360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:34.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.912+0000 7fe479200640 1 --2- 172.21.15.102:0/1982723880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe47415c400 0x7fe4740af360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:34.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.912+0000 7fe479200640 1 -- 172.21.15.102:0/1982723880 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe460009400 con 0x7fe4740b39b0 2024-03-23T03:50:34.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.912+0000 7fe479200640 1 --2- 172.21.15.102:0/1982723880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4740b39b0 0x7fe474149d60 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fe46800c4d0 tx=0x7fe46800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:34.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.912+0000 7fe470c00640 1 -- 172.21.15.102:0/1982723880 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4680074e0 con 0x7fe4740b39b0 2024-03-23T03:50:34.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.912+0000 7fe470c00640 1 -- 172.21.15.102:0/1982723880 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe468007660 con 0x7fe4740b39b0 2024-03-23T03:50:34.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.912+0000 7fe47bc00640 1 -- 172.21.15.102:0/1982723880 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4740b1f00 con 0x7fe4740b39b0 2024-03-23T03:50:34.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.912+0000 7fe470c00640 1 -- 172.21.15.102:0/1982723880 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468004a20 con 0x7fe4740b39b0 2024-03-23T03:50:34.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.912+0000 7fe47bc00640 1 -- 172.21.15.102:0/1982723880 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4740b24c0 con 0x7fe4740b39b0 2024-03-23T03:50:34.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.914+0000 7fe47bc00640 1 -- 172.21.15.102:0/1982723880 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe47406ac20 con 0x7fe4740b39b0 2024-03-23T03:50:34.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.915+0000 7fe470c00640 1 -- 172.21.15.102:0/1982723880 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe468004b80 con 0x7fe4740b39b0 2024-03-23T03:50:34.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.916+0000 7fe470c00640 1 --2- 172.21.15.102:0/1982723880 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe45406ac00 0x7fe45406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:34.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.916+0000 7fe470c00640 1 -- 172.21.15.102:0/1982723880 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe4680886a0 con 0x7fe4740b39b0 2024-03-23T03:50:34.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.916+0000 7fe473e00640 1 --2- 172.21.15.102:0/1982723880 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe45406ac00 0x7fe45406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:34.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.917+0000 7fe473e00640 1 --2- 172.21.15.102:0/1982723880 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe45406ac00 0x7fe45406d0c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe460036000 tx=0x7fe46003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:34.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:34.921+0000 7fe470c00640 1 -- 172.21.15.102:0/1982723880 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe468053720 con 0x7fe4740b39b0 2024-03-23T03:50:35.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.149+0000 7fe47bc00640 1 -- 172.21.15.102:0/1982723880 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe474073390 con 0x7fe4740b39b0 2024-03-23T03:50:35.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.150+0000 7fe470c00640 1 -- 172.21.15.102:0/1982723880 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4745 (secure 0 0 0) 0x7fe468056d40 con 0x7fe4740b39b0 2024-03-23T03:50:35.151 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:35.151 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2079098809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2079098809},{"type":"v1","addr":"172.21.15.173:6833","nonce":2079098809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":5413,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/459102175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":459102175},{"type":"v1","addr":"172.21.15.173:6835","nonce":459102175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2962195583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2962195583},{"type":"v1","addr":"172.21.15.102:6835","nonce":2962195583}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:50:07.937613+0000","modified":"2024-03-23T03:50:10.958097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6384":{"gid":6384,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/3091739472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3091739472},{"type":"v1","addr":"172.21.15.102:6837","nonce":3091739472}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T03:50:35.151 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 41 2024-03-23T03:50:35.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.153+0000 7fe47bc00640 1 -- 172.21.15.102:0/1982723880 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe45406ac00 msgr2=0x7fe45406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:35.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.153+0000 7fe47bc00640 1 --2- 172.21.15.102:0/1982723880 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe45406ac00 0x7fe45406d0c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe460036000 tx=0x7fe46003c040 comp rx=0 tx=0).stop 2024-03-23T03:50:35.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.153+0000 7fe47bc00640 1 -- 172.21.15.102:0/1982723880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4740b39b0 msgr2=0x7fe474149d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:35.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.153+0000 7fe47bc00640 1 --2- 172.21.15.102:0/1982723880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4740b39b0 0x7fe474149d60 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fe46800c4d0 tx=0x7fe46800c990 comp rx=0 tx=0).stop 2024-03-23T03:50:35.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.153+0000 7fe47bc00640 1 -- 172.21.15.102:0/1982723880 shutdown_connections 2024-03-23T03:50:35.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.153+0000 7fe47bc00640 1 --2- 172.21.15.102:0/1982723880 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe45406ac00 0x7fe45406d0c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:35.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.153+0000 7fe47bc00640 1 --2- 172.21.15.102:0/1982723880 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe47415c400 0x7fe4740af360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:35.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.153+0000 7fe47bc00640 1 --2- 172.21.15.102:0/1982723880 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe47415b560 0x7fe4741f6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:35.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.153+0000 7fe47bc00640 1 --2- 172.21.15.102:0/1982723880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4740b39b0 0x7fe474149d60 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:35.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.153+0000 7fe47bc00640 1 -- 172.21.15.102:0/1982723880 >> 172.21.15.102:0/1982723880 conn(0x7fe47406a3d0 msgr2=0x7fe47415e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:35.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.153+0000 7fe47bc00640 1 -- 172.21.15.102:0/1982723880 shutdown_connections 2024-03-23T03:50:35.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.153+0000 7fe47bc00640 1 -- 172.21.15.102:0/1982723880 wait complete. 2024-03-23T03:50:35.170 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:50:35.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.373+0000 7f3311a00640 1 Processor -- start 2024-03-23T03:50:35.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.373+0000 7f3311a00640 1 -- start start 2024-03-23T03:50:35.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.375+0000 7f3311a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f330c160800 0x7f330c0b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:35.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.375+0000 7f3311a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f330c0b1be0 0x7f330c06c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:35.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.375+0000 7f3311a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f330c0b2030 con 0x7f330c13faa0 2024-03-23T03:50:35.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.375+0000 7f3311a00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f330c161640 con 0x7f330c0b1be0 2024-03-23T03:50:35.375 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.375+0000 7f3311a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330c0b38a0 con 0x7f330c160800 2024-03-23T03:50:35.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.375+0000 7f330a000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f330c160800 0x7f330c0b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:35.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.375+0000 7f330b400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f330c0b1be0 0x7f330c06c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:35.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.376+0000 7f330a000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f330c160800 0x7f330c0b16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:59980/0 (socket says 172.21.15.102:59980) 2024-03-23T03:50:35.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.376+0000 7f330a000640 1 -- 172.21.15.102:0/2955638873 learned_addr learned my addr 172.21.15.102:0/2955638873 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:35.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.376+0000 7f330b400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f330c0b1be0 0x7f330c06c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:38734/0 (socket says 172.21.15.102:38734) 2024-03-23T03:50:35.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.376+0000 7f330b400640 1 -- 172.21.15.102:0/2955638873 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f330c160800 msgr2=0x7f330c0b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:35.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.376+0000 7f330b400640 1 --2- 172.21.15.102:0/2955638873 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f330c160800 0x7f330c0b16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:35.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.376+0000 7f330b400640 1 -- 172.21.15.102:0/2955638873 >> v1:172.21.15.173:6789/0 conn(0x7f330c13faa0 legacy=0x7f330c160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:35.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.376+0000 7f330b400640 1 -- 172.21.15.102:0/2955638873 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c0b3b80 con 0x7f330c0b1be0 2024-03-23T03:50:35.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.376+0000 7f330a000640 1 --2- 172.21.15.102:0/2955638873 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f330c160800 0x7f330c0b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:50:35.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.378+0000 7f330b400640 1 --2- 172.21.15.102:0/2955638873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f330c0b1be0 0x7f330c06c280 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f33040097a0 tx=0x7f330402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b4cfea66a9285f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:35.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.379+0000 7f3309600640 1 -- 172.21.15.102:0/2955638873 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330402d040 con 0x7f330c0b1be0 2024-03-23T03:50:35.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.379+0000 7f3309600640 1 -- 172.21.15.102:0/2955638873 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3304031810 con 0x7f330c0b1be0 2024-03-23T03:50:35.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.379+0000 7f3309600640 1 -- 172.21.15.102:0/2955638873 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3304031ba0 con 0x7f330c0b1be0 2024-03-23T03:50:35.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.380+0000 7f3311a00640 1 -- 172.21.15.102:0/2955638873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f330c0b1be0 msgr2=0x7f330c06c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:35.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.380+0000 7f3311a00640 1 --2- 172.21.15.102:0/2955638873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f330c0b1be0 0x7f330c06c280 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f33040097a0 tx=0x7f330402b8b0 comp rx=0 tx=0).stop 2024-03-23T03:50:35.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.380+0000 7f3311a00640 1 -- 172.21.15.102:0/2955638873 shutdown_connections 2024-03-23T03:50:35.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.380+0000 7f3311a00640 1 --2- 172.21.15.102:0/2955638873 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f330c0b1be0 0x7f330c06c280 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:35.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.380+0000 7f3311a00640 1 --2- 172.21.15.102:0/2955638873 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f330c160800 0x7f330c0b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:35.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.380+0000 7f3311a00640 1 -- 172.21.15.102:0/2955638873 >> 172.21.15.102:0/2955638873 conn(0x7f330c06a3d0 msgr2=0x7f330c0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:35.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.380+0000 7f3311a00640 1 -- 172.21.15.102:0/2955638873 shutdown_connections 2024-03-23T03:50:35.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.380+0000 7f3311a00640 1 -- 172.21.15.102:0/2955638873 wait complete. 2024-03-23T03:50:35.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.382+0000 7f3311a00640 1 Processor -- start 2024-03-23T03:50:35.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.382+0000 7f3311a00640 1 -- start start 2024-03-23T03:50:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.383+0000 7f3311a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f330c0b1be0 0x7f330c0aff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.383+0000 7f3311a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f330c13faa0 0x7f330c13afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.383+0000 7f3311a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f330c160800 0x7f330c0cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.383+0000 7f3311a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c155650 con 0x7f330c13faa0 2024-03-23T03:50:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.383+0000 7f3311a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f330c0af920 con 0x7f330c160800 2024-03-23T03:50:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.383+0000 7f3311a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f330c0aeda0 con 0x7f330c0b1be0 2024-03-23T03:50:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.384+0000 7f330a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f330c13faa0 0x7f330c13afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.384+0000 7f330a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f330c13faa0 0x7f330c13afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50676/0 (socket says 172.21.15.102:50676) 2024-03-23T03:50:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.384+0000 7f330a000640 1 -- 172.21.15.102:0/1938981479 learned_addr learned my addr 172.21.15.102:0/1938981479 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.384+0000 7f330aa00640 1 --2- 172.21.15.102:0/1938981479 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f330c0b1be0 0x7f330c0aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.384+0000 7f330b400640 1 --2- 172.21.15.102:0/1938981479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f330c160800 0x7f330c0cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:35.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.384+0000 7f330a000640 1 -- 172.21.15.102:0/1938981479 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f330c0b1be0 msgr2=0x7f330c0aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:35.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.384+0000 7f330a000640 1 --2- 172.21.15.102:0/1938981479 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f330c0b1be0 0x7f330c0aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:35.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.384+0000 7f330a000640 1 -- 172.21.15.102:0/1938981479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f330c160800 msgr2=0x7f330c0cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:35.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.384+0000 7f330a000640 1 --2- 172.21.15.102:0/1938981479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f330c160800 0x7f330c0cacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:35.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.384+0000 7f330a000640 1 -- 172.21.15.102:0/1938981479 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f80094d0 con 0x7f330c13faa0 2024-03-23T03:50:35.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.384+0000 7f330aa00640 1 --2- 172.21.15.102:0/1938981479 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f330c0b1be0 0x7f330c0aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:50:35.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.384+0000 7f330a000640 1 --2- 172.21.15.102:0/1938981479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f330c13faa0 0x7f330c13afa0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f32f800e980 tx=0x7f32f800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:35.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.385+0000 7f3302a00640 1 -- 172.21.15.102:0/1938981479 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f800ca00 con 0x7f330c13faa0 2024-03-23T03:50:35.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.385+0000 7f3302a00640 1 -- 172.21.15.102:0/1938981479 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f800cb60 con 0x7f330c13faa0 2024-03-23T03:50:35.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.385+0000 7f3311a00640 1 -- 172.21.15.102:0/1938981479 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3304009400 con 0x7f330c13faa0 2024-03-23T03:50:35.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.385+0000 7f3302a00640 1 -- 172.21.15.102:0/1938981479 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f8013470 con 0x7f330c13faa0 2024-03-23T03:50:35.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.385+0000 7f3311a00640 1 -- 172.21.15.102:0/1938981479 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f330c0af360 con 0x7f330c13faa0 2024-03-23T03:50:35.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.386+0000 7f3311a00640 1 -- 172.21.15.102:0/1938981479 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f330c073390 con 0x7f330c13faa0 2024-03-23T03:50:35.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.387+0000 7f3302a00640 1 -- 172.21.15.102:0/1938981479 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f32f80135d0 con 0x7f330c13faa0 2024-03-23T03:50:35.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.388+0000 7f3302a00640 1 --2- 172.21.15.102:0/1938981479 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f32ec06ac00 0x7f32ec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:35.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.388+0000 7f3302a00640 1 -- 172.21.15.102:0/1938981479 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f32f80897e0 con 0x7f330c13faa0 2024-03-23T03:50:35.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.388+0000 7f330aa00640 1 --2- 172.21.15.102:0/1938981479 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f32ec06ac00 0x7f32ec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:35.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.389+0000 7f330aa00640 1 --2- 172.21.15.102:0/1938981479 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f32ec06ac00 0x7f32ec06d0c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f32fc0012e0 tx=0x7f32fc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:35.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.392+0000 7f3302a00640 1 -- 172.21.15.102:0/1938981479 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f32f8054870 con 0x7f330c13faa0 2024-03-23T03:50:35.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.572+0000 7f3311a00640 1 -- 172.21.15.102:0/1938981479 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f330c06b250 con 0x7f330c13faa0 2024-03-23T03:50:35.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.574+0000 7f3302a00640 1 -- 172.21.15.102:0/1938981479 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16357 (secure 0 0 0) 0x7f32f8057e90 con 0x7f330c13faa0 2024-03-23T03:50:35.575 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:35.575 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":81,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:08.942399+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:50:06.347556+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-23T03:50:06.924370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:50:35.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.577+0000 7f3311a00640 1 -- 172.21.15.102:0/1938981479 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f32ec06ac00 msgr2=0x7f32ec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:35.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.577+0000 7f3311a00640 1 --2- 172.21.15.102:0/1938981479 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f32ec06ac00 0x7f32ec06d0c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f32fc0012e0 tx=0x7f32fc009040 comp rx=0 tx=0).stop 2024-03-23T03:50:35.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.577+0000 7f3311a00640 1 -- 172.21.15.102:0/1938981479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f330c13faa0 msgr2=0x7f330c13afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:35.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.577+0000 7f3311a00640 1 --2- 172.21.15.102:0/1938981479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f330c13faa0 0x7f330c13afa0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f32f800e980 tx=0x7f32f800ee40 comp rx=0 tx=0).stop 2024-03-23T03:50:35.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.577+0000 7f3311a00640 1 -- 172.21.15.102:0/1938981479 shutdown_connections 2024-03-23T03:50:35.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.577+0000 7f3311a00640 1 --2- 172.21.15.102:0/1938981479 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f32ec06ac00 0x7f32ec06d0c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:35.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.578+0000 7f3311a00640 1 --2- 172.21.15.102:0/1938981479 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f330c160800 0x7f330c0cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:35.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.578+0000 7f3311a00640 1 --2- 172.21.15.102:0/1938981479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f330c13faa0 0x7f330c13afa0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:35.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.578+0000 7f3311a00640 1 --2- 172.21.15.102:0/1938981479 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f330c0b1be0 0x7f330c0aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:35.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.578+0000 7f3311a00640 1 -- 172.21.15.102:0/1938981479 >> 172.21.15.102:0/1938981479 conn(0x7f330c06a3d0 msgr2=0x7f330c070900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:35.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.578+0000 7f3311a00640 1 -- 172.21.15.102:0/1938981479 shutdown_connections 2024-03-23T03:50:35.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.578+0000 7f3311a00640 1 -- 172.21.15.102:0/1938981479 wait complete. 2024-03-23T03:50:35.595 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T03:50:35.595 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:50:35.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.795+0000 7f3c49c00640 1 Processor -- start 2024-03-23T03:50:35.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.795+0000 7f3c49c00640 1 -- start start 2024-03-23T03:50:35.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.797+0000 7f3c49c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3c440b3e30 0x7f3c4415cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:35.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.797+0000 7f3c49c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c4406c6f0 con 0x7f3c440b39b0 2024-03-23T03:50:35.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.797+0000 7f3c49c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c4406c9d0 con 0x7f3c4415b560 2024-03-23T03:50:35.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.797+0000 7f3c49c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c4406ccb0 con 0x7f3c440b3e30 2024-03-23T03:50:35.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.797+0000 7f3c42000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3c440b39b0 0x7f3c4415c400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60962/0 (socket says 172.21.15.102:60962) 2024-03-23T03:50:35.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.797+0000 7f3c42000640 1 -- 172.21.15.102:0/37429259 learned_addr learned my addr 172.21.15.102:0/37429259 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:35.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.797+0000 7f3c43400640 1 --2- 172.21.15.102:0/37429259 >> v2:172.21.15.102:3300/0 conn(0x7f3c440b3e30 0x7f3c4415cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.797+0000 7f3c41600640 1 -- 172.21.15.102:0/37429259 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970889529 0 0) 0x7f3c4406c6f0 con 0x7f3c440b39b0 2024-03-23T03:50:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.798+0000 7f3c41600640 1 -- 172.21.15.102:0/37429259 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c28003520 con 0x7f3c440b39b0 2024-03-23T03:50:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.798+0000 7f3c43400640 1 -- 172.21.15.102:0/37429259 >> v1:172.21.15.173:6789/0 conn(0x7f3c4415b560 legacy=0x7f3c4415b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.798+0000 7f3c43400640 1 -- 172.21.15.102:0/37429259 >> v1:172.21.15.102:6789/0 conn(0x7f3c440b39b0 legacy=0x7f3c4415c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.798+0000 7f3c43400640 1 -- 172.21.15.102:0/37429259 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4406cf90 con 0x7f3c440b3e30 2024-03-23T03:50:35.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.799+0000 7f3c43400640 1 --2- 172.21.15.102:0/37429259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c440b3e30 0x7f3c4415cb10 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009d90 tx=0x7f3c3c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa07d03b45d4764e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:35.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.799+0000 7f3c41600640 1 -- 172.21.15.102:0/37429259 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c02cd00 con 0x7f3c440b3e30 2024-03-23T03:50:35.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.799+0000 7f3c41600640 1 -- 172.21.15.102:0/37429259 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3c02ce60 con 0x7f3c440b3e30 2024-03-23T03:50:35.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.799+0000 7f3c41600640 1 -- 172.21.15.102:0/37429259 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c034660 con 0x7f3c440b3e30 2024-03-23T03:50:35.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.799+0000 7f3c49c00640 1 -- 172.21.15.102:0/37429259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c440b3e30 msgr2=0x7f3c4415cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:35.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.799+0000 7f3c49c00640 1 --2- 172.21.15.102:0/37429259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c440b3e30 0x7f3c4415cb10 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009d90 tx=0x7f3c3c02bf50 comp rx=0 tx=0).stop 2024-03-23T03:50:35.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.800+0000 7f3c49c00640 1 -- 172.21.15.102:0/37429259 shutdown_connections 2024-03-23T03:50:35.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.800+0000 7f3c49c00640 1 --2- 172.21.15.102:0/37429259 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c440b3e30 0x7f3c4415cb10 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:35.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.800+0000 7f3c49c00640 1 -- 172.21.15.102:0/37429259 >> 172.21.15.102:0/37429259 conn(0x7f3c4406a3d0 msgr2=0x7f3c440b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:35.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.800+0000 7f3c49c00640 1 -- 172.21.15.102:0/37429259 shutdown_connections 2024-03-23T03:50:35.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.800+0000 7f3c49c00640 1 -- 172.21.15.102:0/37429259 wait complete. 2024-03-23T03:50:35.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.801+0000 7f3c49c00640 1 Processor -- start 2024-03-23T03:50:35.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.801+0000 7f3c49c00640 1 -- start start 2024-03-23T03:50:35.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.802+0000 7f3c49c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3c440b39b0 0x7f3c441f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:35.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.802+0000 7f3c49c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c440b3e30 0x7f3c44157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:35.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.802+0000 7f3c49c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c4415b560 0x7f3c440af600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:35.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.802+0000 7f3c49c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c44156c90 con 0x7f3c4415b560 2024-03-23T03:50:35.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.802+0000 7f3c49c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c441f3050 con 0x7f3c440b3e30 2024-03-23T03:50:35.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.802+0000 7f3c49c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c440b0610 con 0x7f3c440b39b0 2024-03-23T03:50:35.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.803+0000 7f3c43400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c4415b560 0x7f3c440af600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:35.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.803+0000 7f3c43400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c4415b560 0x7f3c440af600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50706/0 (socket says 172.21.15.102:50706) 2024-03-23T03:50:35.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.803+0000 7f3c43400640 1 -- 172.21.15.102:0/1562813707 learned_addr learned my addr 172.21.15.102:0/1562813707 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:35.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.803+0000 7f3c43400640 1 -- 172.21.15.102:0/1562813707 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3c440b39b0 msgr2=0x7f3c441f2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:35.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.803+0000 7f3c42a00640 1 --2- 172.21.15.102:0/1562813707 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3c440b39b0 0x7f3c441f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:35.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.803+0000 7f3c42000640 1 --2- 172.21.15.102:0/1562813707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c440b3e30 0x7f3c44157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:35.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.803+0000 7f3c43400640 1 --2- 172.21.15.102:0/1562813707 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3c440b39b0 0x7f3c441f2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:35.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.803+0000 7f3c43400640 1 -- 172.21.15.102:0/1562813707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c440b3e30 msgr2=0x7f3c44157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:35.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.803+0000 7f3c43400640 1 --2- 172.21.15.102:0/1562813707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c440b3e30 0x7f3c44157120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:35.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.803+0000 7f3c43400640 1 -- 172.21.15.102:0/1562813707 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3c009a70 con 0x7f3c4415b560 2024-03-23T03:50:35.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.803+0000 7f3c42a00640 1 --2- 172.21.15.102:0/1562813707 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3c440b39b0 0x7f3c441f2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:35.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.803+0000 7f3c43400640 1 --2- 172.21.15.102:0/1562813707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c4415b560 0x7f3c440af600 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009eb0 tx=0x7f3c3c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:35.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.804+0000 7f3c3aa00640 1 -- 172.21.15.102:0/1562813707 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c03d600 con 0x7f3c4415b560 2024-03-23T03:50:35.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.804+0000 7f3c3aa00640 1 -- 172.21.15.102:0/1562813707 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c3c03d780 con 0x7f3c4415b560 2024-03-23T03:50:35.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.804+0000 7f3c3aa00640 1 -- 172.21.15.102:0/1562813707 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c03db30 con 0x7f3c4415b560 2024-03-23T03:50:35.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.804+0000 7f3c49c00640 1 -- 172.21.15.102:0/1562813707 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c44156640 con 0x7f3c4415b560 2024-03-23T03:50:35.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.804+0000 7f3c49c00640 1 -- 172.21.15.102:0/1562813707 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c44157660 con 0x7f3c4415b560 2024-03-23T03:50:35.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.805+0000 7f3c49c00640 1 -- 172.21.15.102:0/1562813707 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c4406ac20 con 0x7f3c4415b560 2024-03-23T03:50:35.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.806+0000 7f3c3aa00640 1 -- 172.21.15.102:0/1562813707 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3c3c03b040 con 0x7f3c4415b560 2024-03-23T03:50:35.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.807+0000 7f3c3aa00640 1 --2- 172.21.15.102:0/1562813707 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3c2406ac00 0x7f3c2406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:35.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.807+0000 7f3c3aa00640 1 -- 172.21.15.102:0/1562813707 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3c3c0abae0 con 0x7f3c4415b560 2024-03-23T03:50:35.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.807+0000 7f3c42a00640 1 --2- 172.21.15.102:0/1562813707 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3c2406ac00 0x7f3c2406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:35.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.808+0000 7f3c42a00640 1 --2- 172.21.15.102:0/1562813707 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3c2406ac00 0x7f3c2406d0c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3c44151a00 tx=0x7f3c34009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:35.812 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:35.811+0000 7f3c3aa00640 1 -- 172.21.15.102:0/1562813707 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3c3c076c00 con 0x7f3c4415b560 2024-03-23T03:50:36.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.043+0000 7f3c49c00640 1 -- 172.21.15.102:0/1562813707 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c44073390 con 0x7f3c4415b560 2024-03-23T03:50:36.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.044+0000 7f3c3aa00640 1 -- 172.21.15.102:0/1562813707 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f3c3c039020 con 0x7f3c4415b560 2024-03-23T03:50:36.045 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:36.045 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2079098809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2079098809},{"type":"v1","addr":"172.21.15.173:6833","nonce":2079098809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":5413,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/459102175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":459102175},{"type":"v1","addr":"172.21.15.173:6835","nonce":459102175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2962195583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2962195583},{"type":"v1","addr":"172.21.15.102:6835","nonce":2962195583}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:50:07.937613+0000","modified":"2024-03-23T03:50:10.958097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6384":{"gid":6384,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/3091739472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3091739472},{"type":"v1","addr":"172.21.15.102:6837","nonce":3091739472}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T03:50:36.045 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 41 2024-03-23T03:50:36.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.047+0000 7f3c49c00640 1 -- 172.21.15.102:0/1562813707 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3c2406ac00 msgr2=0x7f3c2406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:36.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.047+0000 7f3c49c00640 1 --2- 172.21.15.102:0/1562813707 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3c2406ac00 0x7f3c2406d0c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f3c44151a00 tx=0x7f3c34009040 comp rx=0 tx=0).stop 2024-03-23T03:50:36.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.047+0000 7f3c49c00640 1 -- 172.21.15.102:0/1562813707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c4415b560 msgr2=0x7f3c440af600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:36.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.047+0000 7f3c49c00640 1 --2- 172.21.15.102:0/1562813707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c4415b560 0x7f3c440af600 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c009eb0 tx=0x7f3c3c004a40 comp rx=0 tx=0).stop 2024-03-23T03:50:36.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.047+0000 7f3c49c00640 1 -- 172.21.15.102:0/1562813707 shutdown_connections 2024-03-23T03:50:36.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.047+0000 7f3c49c00640 1 --2- 172.21.15.102:0/1562813707 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3c2406ac00 0x7f3c2406d0c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.047+0000 7f3c49c00640 1 --2- 172.21.15.102:0/1562813707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c4415b560 0x7f3c440af600 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.047+0000 7f3c49c00640 1 --2- 172.21.15.102:0/1562813707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c440b3e30 0x7f3c44157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.047+0000 7f3c49c00640 1 --2- 172.21.15.102:0/1562813707 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3c440b39b0 0x7f3c441f2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.047+0000 7f3c49c00640 1 -- 172.21.15.102:0/1562813707 >> 172.21.15.102:0/1562813707 conn(0x7f3c4406a3d0 msgr2=0x7f3c4415cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:36.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.047+0000 7f3c49c00640 1 -- 172.21.15.102:0/1562813707 shutdown_connections 2024-03-23T03:50:36.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.047+0000 7f3c49c00640 1 -- 172.21.15.102:0/1562813707 wait complete. 2024-03-23T03:50:36.066 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:50:36.270 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.270+0000 7f3e6e400640 1 Processor -- start 2024-03-23T03:50:36.271 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.270+0000 7f3e6e400640 1 -- start start 2024-03-23T03:50:36.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.272+0000 7f3e6e400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f3e680b3e30 0x7f3e6815cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:36.272 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.272+0000 7f3e6e400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6806c6f0 con 0x7f3e6815b560 2024-03-23T03:50:36.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.272+0000 7f3e6e400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e6806c9d0 con 0x7f3e680b39b0 2024-03-23T03:50:36.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.272+0000 7f3e6e400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e6806ccb0 con 0x7f3e680b3e30 2024-03-23T03:50:36.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.272+0000 7f3e67400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3e6815b560 0x7f3e6815b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60978/0 (socket says 172.21.15.102:60978) 2024-03-23T03:50:36.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.272+0000 7f3e67400640 1 -- 172.21.15.102:0/2400108942 learned_addr learned my addr 172.21.15.102:0/2400108942 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:36.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.272+0000 7f3e66a00640 1 -- 172.21.15.102:0/2400108942 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392385822 0 0) 0x7f3e6806c6f0 con 0x7f3e6815b560 2024-03-23T03:50:36.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.272+0000 7f3e66a00640 1 -- 172.21.15.102:0/2400108942 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e4c003520 con 0x7f3e6815b560 2024-03-23T03:50:36.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.272+0000 7f3e67e00640 1 --2- 172.21.15.102:0/2400108942 >> v2:172.21.15.173:3300/0 conn(0x7f3e680b3e30 0x7f3e6815cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:36.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.273+0000 7f3e66a00640 1 -- 172.21.15.102:0/2400108942 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1297622183 0 0) 0x7f3e4c003520 con 0x7f3e6815b560 2024-03-23T03:50:36.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.273+0000 7f3e66a00640 1 -- 172.21.15.102:0/2400108942 >> v2:172.21.15.173:3300/0 conn(0x7f3e680b3e30 msgr2=0x7f3e6815cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:36.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.273+0000 7f3e66a00640 1 --2- 172.21.15.102:0/2400108942 >> v2:172.21.15.173:3300/0 conn(0x7f3e680b3e30 0x7f3e6815cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.273+0000 7f3e66a00640 1 -- 172.21.15.102:0/2400108942 >> v1:172.21.15.173:6789/0 conn(0x7f3e680b39b0 legacy=0x7f3e6815c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:36.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.273+0000 7f3e66a00640 1 -- 172.21.15.102:0/2400108942 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6806cf90 con 0x7f3e6815b560 2024-03-23T03:50:36.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.273+0000 7f3e66a00640 1 -- 172.21.15.102:0/2400108942 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f3e58001670 con 0x7f3e6815b560 2024-03-23T03:50:36.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.273+0000 7f3e66a00640 1 -- 172.21.15.102:0/2400108942 >> v1:172.21.15.102:6789/0 conn(0x7f3e6815b560 legacy=0x7f3e6815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:36.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.273+0000 7f3e66a00640 1 --2- 172.21.15.102:0/2400108942 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e4c003970 0x7f3e4c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:36.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.273+0000 7f3e66a00640 1 --2- 172.21.15.102:0/2400108942 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e4c0047c0 0x7f3e4c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:36.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.273+0000 7f3e66a00640 1 --2- 172.21.15.102:0/2400108942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e4c005180 0x7f3e4c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:36.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.273+0000 7f3e66a00640 1 -- 172.21.15.102:0/2400108942 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6806ccb0 con 0x7f3e4c005180 2024-03-23T03:50:36.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.273+0000 7f3e66a00640 1 -- 172.21.15.102:0/2400108942 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c00c0b0 con 0x7f3e4c003970 2024-03-23T03:50:36.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.273+0000 7f3e66a00640 1 -- 172.21.15.102:0/2400108942 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c00c270 con 0x7f3e4c0047c0 2024-03-23T03:50:36.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.273+0000 7f3e67e00640 1 --2- 172.21.15.102:0/2400108942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e4c005180 0x7f3e4c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:36.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.273+0000 7f3e67e00640 1 -- 172.21.15.102:0/2400108942 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e4c0047c0 msgr2=0x7f3e4c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:36.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.273+0000 7f3e5ea00640 1 --2- 172.21.15.102:0/2400108942 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e4c0047c0 0x7f3e4c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:36.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.273+0000 7f3e67400640 1 --2- 172.21.15.102:0/2400108942 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e4c003970 0x7f3e4c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:36.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.274+0000 7f3e67e00640 1 --2- 172.21.15.102:0/2400108942 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e4c0047c0 0x7f3e4c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.274+0000 7f3e67e00640 1 -- 172.21.15.102:0/2400108942 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e4c003970 msgr2=0x7f3e4c003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:36.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.274+0000 7f3e67e00640 1 --2- 172.21.15.102:0/2400108942 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e4c003970 0x7f3e4c003d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.274+0000 7f3e67e00640 1 -- 172.21.15.102:0/2400108942 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4c003520 con 0x7f3e4c005180 2024-03-23T03:50:36.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.274+0000 7f3e5ea00640 1 --2- 172.21.15.102:0/2400108942 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e4c0047c0 0x7f3e4c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:36.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.274+0000 7f3e67e00640 1 -- 172.21.15.102:0/2400108942 reap_dead start 2024-03-23T03:50:36.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.275+0000 7f3e67e00640 1 --2- 172.21.15.102:0/2400108942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e4c005180 0x7f3e4c00b980 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f3e60007330 tx=0x7f3e4c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8200c1d9f8d221 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:36.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.275+0000 7f3e66a00640 1 -- 172.21.15.102:0/2400108942 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6002ccd0 con 0x7f3e4c005180 2024-03-23T03:50:36.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.275+0000 7f3e66a00640 1 -- 172.21.15.102:0/2400108942 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e6002ce30 con 0x7f3e4c005180 2024-03-23T03:50:36.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.275+0000 7f3e66a00640 1 -- 172.21.15.102:0/2400108942 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60003bd0 con 0x7f3e4c005180 2024-03-23T03:50:36.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.275+0000 7f3e6e400640 1 -- 172.21.15.102:0/2400108942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e4c005180 msgr2=0x7f3e4c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:36.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.275+0000 7f3e6e400640 1 --2- 172.21.15.102:0/2400108942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e4c005180 0x7f3e4c00b980 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f3e60007330 tx=0x7f3e4c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.275+0000 7f3e6e400640 1 -- 172.21.15.102:0/2400108942 shutdown_connections 2024-03-23T03:50:36.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.275+0000 7f3e6e400640 1 --2- 172.21.15.102:0/2400108942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e4c005180 0x7f3e4c00b980 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.275+0000 7f3e6e400640 1 -- 172.21.15.102:0/2400108942 >> 172.21.15.102:0/2400108942 conn(0x7f3e6806a3d0 msgr2=0x7f3e680b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:36.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.276+0000 7f3e6e400640 1 -- 172.21.15.102:0/2400108942 shutdown_connections 2024-03-23T03:50:36.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.276+0000 7f3e6e400640 1 -- 172.21.15.102:0/2400108942 wait complete. 2024-03-23T03:50:36.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.277+0000 7f3e6e400640 1 Processor -- start 2024-03-23T03:50:36.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.277+0000 7f3e6e400640 1 -- start start 2024-03-23T03:50:36.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.278+0000 7f3e6e400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e4c005180 0x7f3e68148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:36.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.278+0000 7f3e6e400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e68155eb0 0x7f3e68149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:36.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.278+0000 7f3e6e400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e68149f50 0x7f3e68156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:36.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.278+0000 7f3e6e400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e681495f0 con 0x7f3e68149f50 2024-03-23T03:50:36.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.278+0000 7f3e6e400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e680cc410 con 0x7f3e4c005180 2024-03-23T03:50:36.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.278+0000 7f3e6e400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e68148f30 con 0x7f3e68155eb0 2024-03-23T03:50:36.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.279+0000 7f3e67e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e68149f50 0x7f3e68156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:36.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.279+0000 7f3e67e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e68149f50 0x7f3e68156b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50728/0 (socket says 172.21.15.102:50728) 2024-03-23T03:50:36.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.279+0000 7f3e67e00640 1 -- 172.21.15.102:0/2434930917 learned_addr learned my addr 172.21.15.102:0/2434930917 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:36.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.279+0000 7f3e67e00640 1 -- 172.21.15.102:0/2434930917 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e68155eb0 msgr2=0x7f3e68149a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:36.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.279+0000 7f3e5ea00640 1 --2- 172.21.15.102:0/2434930917 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e68155eb0 0x7f3e68149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:36.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.279+0000 7f3e67e00640 1 --2- 172.21.15.102:0/2434930917 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e68155eb0 0x7f3e68149a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.279+0000 7f3e67e00640 1 -- 172.21.15.102:0/2434930917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e4c005180 msgr2=0x7f3e68148690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:36.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.279+0000 7f3e67e00640 1 --2- 172.21.15.102:0/2434930917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e4c005180 0x7f3e68148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.279+0000 7f3e67e00640 1 -- 172.21.15.102:0/2434930917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e68149210 con 0x7f3e68149f50 2024-03-23T03:50:36.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.279+0000 7f3e67e00640 1 --2- 172.21.15.102:0/2434930917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e68149f50 0x7f3e68156b90 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3e60007450 tx=0x7f3e60004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:36.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.280+0000 7f3e5fe00640 1 -- 172.21.15.102:0/2434930917 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60004860 con 0x7f3e68149f50 2024-03-23T03:50:36.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.280+0000 7f3e5fe00640 1 -- 172.21.15.102:0/2434930917 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e60003e10 con 0x7f3e68149f50 2024-03-23T03:50:36.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.280+0000 7f3e6e400640 1 -- 172.21.15.102:0/2434930917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e680cc6f0 con 0x7f3e68149f50 2024-03-23T03:50:36.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.280+0000 7f3e5fe00640 1 -- 172.21.15.102:0/2434930917 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60040680 con 0x7f3e68149f50 2024-03-23T03:50:36.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.280+0000 7f3e6e400640 1 -- 172.21.15.102:0/2434930917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e680cbfc0 con 0x7f3e68149f50 2024-03-23T03:50:36.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.281+0000 7f3e6e400640 1 -- 172.21.15.102:0/2434930917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e680cc1b0 con 0x7f3e68149f50 2024-03-23T03:50:36.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.282+0000 7f3e5fe00640 1 -- 172.21.15.102:0/2434930917 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3e600407e0 con 0x7f3e68149f50 2024-03-23T03:50:36.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.283+0000 7f3e5fe00640 1 --2- 172.21.15.102:0/2434930917 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e4406ac00 0x7f3e4406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:36.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.283+0000 7f3e5fe00640 1 -- 172.21.15.102:0/2434930917 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3e600aba20 con 0x7f3e68149f50 2024-03-23T03:50:36.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.283+0000 7f3e67400640 1 --2- 172.21.15.102:0/2434930917 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e4406ac00 0x7f3e4406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:36.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.284+0000 7f3e67400640 1 --2- 172.21.15.102:0/2434930917 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e4406ac00 0x7f3e4406d0c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3e58000ff0 tx=0x7f3e58009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:36.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.287+0000 7f3e5fe00640 1 -- 172.21.15.102:0/2434930917 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3e60076ab0 con 0x7f3e68149f50 2024-03-23T03:50:36.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.517+0000 7f3e6e400640 1 -- 172.21.15.102:0/2434930917 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e6806ac20 con 0x7f3e68149f50 2024-03-23T03:50:36.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.519+0000 7f3e5fe00640 1 -- 172.21.15.102:0/2434930917 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f3e6007a0d0 con 0x7f3e68149f50 2024-03-23T03:50:36.519 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:36.520 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":41,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2079098809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2079098809},{"type":"v1","addr":"172.21.15.173:6833","nonce":2079098809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":5413,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/459102175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":459102175},{"type":"v1","addr":"172.21.15.173:6835","nonce":459102175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2962195583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2962195583},{"type":"v1","addr":"172.21.15.102:6835","nonce":2962195583}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:50:07.937613+0000","modified":"2024-03-23T03:50:10.958097+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6384},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6384":{"gid":6384,"name":"a","rank":0,"incarnation":37,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/3091739472","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3091739472},{"type":"v1","addr":"172.21.15.102:6837","nonce":3091739472}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-03-23T03:50:36.520 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 41 2024-03-23T03:50:36.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.521+0000 7f3e6e400640 1 -- 172.21.15.102:0/2434930917 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e4406ac00 msgr2=0x7f3e4406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:36.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.521+0000 7f3e6e400640 1 --2- 172.21.15.102:0/2434930917 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e4406ac00 0x7f3e4406d0c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f3e58000ff0 tx=0x7f3e58009040 comp rx=0 tx=0).stop 2024-03-23T03:50:36.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.521+0000 7f3e6e400640 1 -- 172.21.15.102:0/2434930917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e68149f50 msgr2=0x7f3e68156b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:36.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.521+0000 7f3e6e400640 1 --2- 172.21.15.102:0/2434930917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e68149f50 0x7f3e68156b90 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f3e60007450 tx=0x7f3e60004ab0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.521+0000 7f3e6e400640 1 -- 172.21.15.102:0/2434930917 shutdown_connections 2024-03-23T03:50:36.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.521+0000 7f3e6e400640 1 --2- 172.21.15.102:0/2434930917 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3e4406ac00 0x7f3e4406d0c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.521+0000 7f3e6e400640 1 --2- 172.21.15.102:0/2434930917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e68149f50 0x7f3e68156b90 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.521+0000 7f3e6e400640 1 --2- 172.21.15.102:0/2434930917 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e68155eb0 0x7f3e68149a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.521+0000 7f3e6e400640 1 --2- 172.21.15.102:0/2434930917 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e4c005180 0x7f3e68148690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.521+0000 7f3e6e400640 1 -- 172.21.15.102:0/2434930917 >> 172.21.15.102:0/2434930917 conn(0x7f3e6806a3d0 msgr2=0x7f3e6815e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:36.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.521+0000 7f3e6e400640 1 -- 172.21.15.102:0/2434930917 shutdown_connections 2024-03-23T03:50:36.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.521+0000 7f3e6e400640 1 -- 172.21.15.102:0/2434930917 wait complete. 2024-03-23T03:50:36.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:50:36.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.739+0000 7f3928c00640 1 Processor -- start 2024-03-23T03:50:36.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.739+0000 7f3928c00640 1 -- start start 2024-03-23T03:50:36.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.740+0000 7f3928c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f392415cda0 0x7f39240cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:36.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.740+0000 7f3928c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f39240b5a30 0x7f39240b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:36.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.741+0000 7f3928c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39240d1830 con 0x7f39240b4350 2024-03-23T03:50:36.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.741+0000 7f3928c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39240b1f20 con 0x7f392415cda0 2024-03-23T03:50:36.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.741+0000 7f3928c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39240d0c20 con 0x7f39240b5a30 2024-03-23T03:50:36.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.741+0000 7f3921e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f392415cda0 0x7f39240cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:36.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.741+0000 7f3922800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f39240b4350 0x7f39240d1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60982/0 (socket says 172.21.15.102:60982) 2024-03-23T03:50:36.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.741+0000 7f3922800640 1 -- 172.21.15.102:0/486325808 learned_addr learned my addr 172.21.15.102:0/486325808 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:36.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.741+0000 7f3921400640 1 --2- 172.21.15.102:0/486325808 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f39240b5a30 0x7f39240b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:36.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.742+0000 7f3921e00640 1 -- 172.21.15.102:0/486325808 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f39240b5a30 msgr2=0x7f39240b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:36.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.742+0000 7f3921e00640 1 --2- 172.21.15.102:0/486325808 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f39240b5a30 0x7f39240b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.742+0000 7f3921e00640 1 -- 172.21.15.102:0/486325808 >> v1:172.21.15.102:6789/0 conn(0x7f39240b4350 legacy=0x7f39240d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:36.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.742+0000 7f3921e00640 1 -- 172.21.15.102:0/486325808 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39240d0f00 con 0x7f392415cda0 2024-03-23T03:50:36.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.742+0000 7f3920a00640 1 -- 172.21.15.102:0/486325808 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68402679 0 0) 0x7f39240d1830 con 0x7f39240b4350 2024-03-23T03:50:36.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.743+0000 7f3921e00640 1 --2- 172.21.15.102:0/486325808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f392415cda0 0x7f39240cdb40 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f3914009d90 tx=0x7f391402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d36234e05b87458 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:36.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.743+0000 7f3920a00640 1 -- 172.21.15.102:0/486325808 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391402cc50 con 0x7f392415cda0 2024-03-23T03:50:36.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.743+0000 7f3920a00640 1 -- 172.21.15.102:0/486325808 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391402cdb0 con 0x7f392415cda0 2024-03-23T03:50:36.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.743+0000 7f3920a00640 1 -- 172.21.15.102:0/486325808 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3914034660 con 0x7f392415cda0 2024-03-23T03:50:36.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.744+0000 7f3928c00640 1 -- 172.21.15.102:0/486325808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f392415cda0 msgr2=0x7f39240cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:36.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.744+0000 7f3928c00640 1 --2- 172.21.15.102:0/486325808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f392415cda0 0x7f39240cdb40 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f3914009d90 tx=0x7f391402bea0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.744+0000 7f3928c00640 1 -- 172.21.15.102:0/486325808 shutdown_connections 2024-03-23T03:50:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.744+0000 7f3928c00640 1 --2- 172.21.15.102:0/486325808 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f39240b5a30 0x7f39240b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.744+0000 7f3928c00640 1 --2- 172.21.15.102:0/486325808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f392415cda0 0x7f39240cdb40 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.744+0000 7f3928c00640 1 -- 172.21.15.102:0/486325808 >> 172.21.15.102:0/486325808 conn(0x7f39240cf350 msgr2=0x7f39240cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.744+0000 7f3928c00640 1 -- 172.21.15.102:0/486325808 shutdown_connections 2024-03-23T03:50:36.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.744+0000 7f3928c00640 1 -- 172.21.15.102:0/486325808 wait complete. 2024-03-23T03:50:36.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.746+0000 7f3928c00640 1 Processor -- start 2024-03-23T03:50:36.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.746+0000 7f3928c00640 1 -- start start 2024-03-23T03:50:36.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.747+0000 7f3928c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39240b4350 0x7f39241f1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:36.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.747+0000 7f3928c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f39240b5a30 0x7f39240c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:36.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.747+0000 7f3928c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f392415cda0 0x7f39240c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:36.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.747+0000 7f3928c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392415d240 con 0x7f392415cda0 2024-03-23T03:50:36.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.747+0000 7f3928c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392415d520 con 0x7f39240b4350 2024-03-23T03:50:36.748 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.747+0000 7f3928c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f392415d800 con 0x7f39240b5a30 2024-03-23T03:50:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.747+0000 7f3922800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f392415cda0 0x7f39240c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.747+0000 7f3922800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f392415cda0 0x7f39240c72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50754/0 (socket says 172.21.15.102:50754) 2024-03-23T03:50:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.747+0000 7f3922800640 1 -- 172.21.15.102:0/669946040 learned_addr learned my addr 172.21.15.102:0/669946040 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.747+0000 7f3922800640 1 -- 172.21.15.102:0/669946040 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f39240b5a30 msgr2=0x7f39240c6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.747+0000 7f3922800640 1 --2- 172.21.15.102:0/669946040 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f39240b5a30 0x7f39240c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.747+0000 7f3922800640 1 -- 172.21.15.102:0/669946040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39240b4350 msgr2=0x7f39241f1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.747+0000 7f3922800640 1 --2- 172.21.15.102:0/669946040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39240b4350 0x7f39241f1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.747+0000 7f3922800640 1 -- 172.21.15.102:0/669946040 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3914009a70 con 0x7f392415cda0 2024-03-23T03:50:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.748+0000 7f3922800640 1 --2- 172.21.15.102:0/669946040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f392415cda0 0x7f39240c72a0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f391c00c450 tx=0x7f391c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.748+0000 7f391a000640 1 -- 172.21.15.102:0/669946040 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c0074e0 con 0x7f392415cda0 2024-03-23T03:50:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.748+0000 7f391a000640 1 -- 172.21.15.102:0/669946040 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391c007660 con 0x7f392415cda0 2024-03-23T03:50:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.748+0000 7f391a000640 1 -- 172.21.15.102:0/669946040 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c004a20 con 0x7f392415cda0 2024-03-23T03:50:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.748+0000 7f3928c00640 1 -- 172.21.15.102:0/669946040 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f392415dae0 con 0x7f392415cda0 2024-03-23T03:50:36.749 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.748+0000 7f3928c00640 1 -- 172.21.15.102:0/669946040 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39240c66a0 con 0x7f392415cda0 2024-03-23T03:50:36.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.749+0000 7f3928c00640 1 -- 172.21.15.102:0/669946040 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f392415e000 con 0x7f392415cda0 2024-03-23T03:50:36.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.750+0000 7f391a000640 1 -- 172.21.15.102:0/669946040 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f391c004b80 con 0x7f392415cda0 2024-03-23T03:50:36.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.751+0000 7f391a000640 1 --2- 172.21.15.102:0/669946040 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f390006ac30 0x7f390006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:36.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.751+0000 7f391a000640 1 -- 172.21.15.102:0/669946040 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f391c088650 con 0x7f392415cda0 2024-03-23T03:50:36.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.751+0000 7f3921e00640 1 --2- 172.21.15.102:0/669946040 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f390006ac30 0x7f390006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:36.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.752+0000 7f3921e00640 1 --2- 172.21.15.102:0/669946040 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f390006ac30 0x7f390006d0f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3914036000 tx=0x7f391402e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:36.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.755+0000 7f391a000640 1 -- 172.21.15.102:0/669946040 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f392415e000 con 0x7f392415cda0 2024-03-23T03:50:36.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.935+0000 7f3928c00640 1 -- 172.21.15.102:0/669946040 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f392415e000 con 0x7f392415cda0 2024-03-23T03:50:36.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.937+0000 7f391a000640 1 -- 172.21.15.102:0/669946040 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16357 (secure 0 0 0) 0x7f392415e000 con 0x7f392415cda0 2024-03-23T03:50:36.938 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:36.938 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":81,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:08.942399+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:50:06.347556+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-03-23T03:50:06.924370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:50:36.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.939+0000 7f3928c00640 1 -- 172.21.15.102:0/669946040 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f390006ac30 msgr2=0x7f390006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.939+0000 7f3928c00640 1 --2- 172.21.15.102:0/669946040 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f390006ac30 0x7f390006d0f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f3914036000 tx=0x7f391402e000 comp rx=0 tx=0).stop 2024-03-23T03:50:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.940+0000 7f3928c00640 1 -- 172.21.15.102:0/669946040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f392415cda0 msgr2=0x7f39240c72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.940+0000 7f3928c00640 1 --2- 172.21.15.102:0/669946040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f392415cda0 0x7f39240c72a0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f391c00c450 tx=0x7f391c00c910 comp rx=0 tx=0).stop 2024-03-23T03:50:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.940+0000 7f3928c00640 1 -- 172.21.15.102:0/669946040 shutdown_connections 2024-03-23T03:50:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.940+0000 7f3928c00640 1 --2- 172.21.15.102:0/669946040 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f390006ac30 0x7f390006d0f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.940+0000 7f3928c00640 1 --2- 172.21.15.102:0/669946040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f392415cda0 0x7f39240c72a0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.940+0000 7f3928c00640 1 --2- 172.21.15.102:0/669946040 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f39240b5a30 0x7f39240c6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.940+0000 7f3928c00640 1 --2- 172.21.15.102:0/669946040 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39240b4350 0x7f39241f1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.940+0000 7f3928c00640 1 -- 172.21.15.102:0/669946040 >> 172.21.15.102:0/669946040 conn(0x7f39240cf350 msgr2=0x7f392415bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.940+0000 7f3928c00640 1 -- 172.21.15.102:0/669946040 shutdown_connections 2024-03-23T03:50:36.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:36.940+0000 7f3928c00640 1 -- 172.21.15.102:0/669946040 wait complete. 2024-03-23T03:50:36.960 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T03:50:37.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.164+0000 7f7ebbe00640 1 Processor -- start 2024-03-23T03:50:37.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.164+0000 7f7ebbe00640 1 -- start start 2024-03-23T03:50:37.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.166+0000 7f7ebbe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7ebc140cf0 0x7f7ebc0b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:37.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.166+0000 7f7ebbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7ebc0cb480 0x7f7ebc141410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:37.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.166+0000 7f7ebbe00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ebc1410d0 con 0x7f7ebc0b23e0 2024-03-23T03:50:37.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.166+0000 7f7ebbe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ebc0cd1b0 con 0x7f7ebc0cb480 2024-03-23T03:50:37.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.166+0000 7f7ebbe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc0cd490 con 0x7f7ebc140cf0 2024-03-23T03:50:37.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.166+0000 7f7ebb400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7ebc0cb480 0x7f7ebc141410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:37.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.166+0000 7f7ebb400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7ebc0cb480 0x7f7ebc141410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50762/0 (socket says 172.21.15.102:50762) 2024-03-23T03:50:37.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.166+0000 7f7ebb400640 1 -- 172.21.15.102:0/3462425096 learned_addr learned my addr 172.21.15.102:0/3462425096 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:37.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.167+0000 7f7ebaa00640 1 --2- 172.21.15.102:0/3462425096 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7ebc140cf0 0x7f7ebc0b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:37.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.167+0000 7f7ebb400640 1 -- 172.21.15.102:0/3462425096 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7ebc140cf0 msgr2=0x7f7ebc0b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:37.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.167+0000 7f7ebb400640 1 --2- 172.21.15.102:0/3462425096 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7ebc140cf0 0x7f7ebc0b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:37.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.167+0000 7f7ebb400640 1 -- 172.21.15.102:0/3462425096 >> v1:172.21.15.173:6789/0 conn(0x7f7ebc0b23e0 legacy=0x7f7ebc0cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:37.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.167+0000 7f7ebb400640 1 -- 172.21.15.102:0/3462425096 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc0cd770 con 0x7f7ebc0cb480 2024-03-23T03:50:37.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.167+0000 7f7ebaa00640 1 --2- 172.21.15.102:0/3462425096 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7ebc140cf0 0x7f7ebc0b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:37.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.168+0000 7f7ebb400640 1 --2- 172.21.15.102:0/3462425096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ebc0cb480 0x7f7ebc141410 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4009e50 tx=0x7f7eb402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e1af5d3bf974e49 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:37.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.168+0000 7f7eba000640 1 -- 172.21.15.102:0/3462425096 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb402ca70 con 0x7f7ebc0cb480 2024-03-23T03:50:37.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.168+0000 7f7eba000640 1 -- 172.21.15.102:0/3462425096 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb402cbd0 con 0x7f7ebc0cb480 2024-03-23T03:50:37.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.168+0000 7f7eba000640 1 -- 172.21.15.102:0/3462425096 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb40343f0 con 0x7f7ebc0cb480 2024-03-23T03:50:37.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.169+0000 7f7ebbe00640 1 -- 172.21.15.102:0/3462425096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ebc0cb480 msgr2=0x7f7ebc141410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:37.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.169+0000 7f7ebbe00640 1 --2- 172.21.15.102:0/3462425096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ebc0cb480 0x7f7ebc141410 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4009e50 tx=0x7f7eb402bed0 comp rx=0 tx=0).stop 2024-03-23T03:50:37.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.169+0000 7f7ebbe00640 1 -- 172.21.15.102:0/3462425096 shutdown_connections 2024-03-23T03:50:37.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.169+0000 7f7ebbe00640 1 --2- 172.21.15.102:0/3462425096 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ebc0cb480 0x7f7ebc141410 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:37.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.169+0000 7f7ebbe00640 1 --2- 172.21.15.102:0/3462425096 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7ebc140cf0 0x7f7ebc0b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:37.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.169+0000 7f7ebbe00640 1 -- 172.21.15.102:0/3462425096 >> 172.21.15.102:0/3462425096 conn(0x7f7ebc065e20 msgr2=0x7f7ebc069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:37.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.169+0000 7f7ebbe00640 1 -- 172.21.15.102:0/3462425096 shutdown_connections 2024-03-23T03:50:37.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.169+0000 7f7ebbe00640 1 -- 172.21.15.102:0/3462425096 wait complete. 2024-03-23T03:50:37.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.171+0000 7f7ebbe00640 1 Processor -- start 2024-03-23T03:50:37.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.171+0000 7f7ebbe00640 1 -- start start 2024-03-23T03:50:37.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.172+0000 7f7ebbe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7ebc0b23e0 0x7f7ebc15a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:37.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.172+0000 7f7ebbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ebc0cb480 0x7f7ebc15c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:37.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.172+0000 7f7ebbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ebc140cf0 0x7f7ebc13a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:37.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.172+0000 7f7ebbe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc1394b0 con 0x7f7ebc140cf0 2024-03-23T03:50:37.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.172+0000 7f7ebbe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc13ac00 con 0x7f7ebc0cb480 2024-03-23T03:50:37.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.172+0000 7f7ebbe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc15b7e0 con 0x7f7ebc0b23e0 2024-03-23T03:50:37.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.172+0000 7f7ebb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ebc140cf0 0x7f7ebc13a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:37.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.172+0000 7f7ebb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ebc140cf0 0x7f7ebc13a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50764/0 (socket says 172.21.15.102:50764) 2024-03-23T03:50:37.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.172+0000 7f7ebb400640 1 -- 172.21.15.102:0/2516818599 learned_addr learned my addr 172.21.15.102:0/2516818599 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:37.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.173+0000 7f7ebaa00640 1 --2- 172.21.15.102:0/2516818599 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7ebc0b23e0 0x7f7ebc15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:37.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.173+0000 7f7ebb400640 1 -- 172.21.15.102:0/2516818599 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7ebc0b23e0 msgr2=0x7f7ebc15a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:37.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.173+0000 7f7ebb400640 1 --2- 172.21.15.102:0/2516818599 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7ebc0b23e0 0x7f7ebc15a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:37.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.173+0000 7f7eb2000640 1 --2- 172.21.15.102:0/2516818599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ebc0cb480 0x7f7ebc15c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:37.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.173+0000 7f7ebb400640 1 -- 172.21.15.102:0/2516818599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ebc0cb480 msgr2=0x7f7ebc15c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:37.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.173+0000 7f7ebb400640 1 --2- 172.21.15.102:0/2516818599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ebc0cb480 0x7f7ebc15c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:37.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.173+0000 7f7ebb400640 1 -- 172.21.15.102:0/2516818599 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb4009b30 con 0x7f7ebc140cf0 2024-03-23T03:50:37.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.173+0000 7f7eb2000640 1 --2- 172.21.15.102:0/2516818599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ebc0cb480 0x7f7ebc15c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:37.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.173+0000 7f7ebb400640 1 --2- 172.21.15.102:0/2516818599 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ebc140cf0 0x7f7ebc13a6c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4036040 tx=0x7f7eb4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:37.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.173+0000 7f7eb3400640 1 -- 172.21.15.102:0/2516818599 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb403e450 con 0x7f7ebc140cf0 2024-03-23T03:50:37.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.173+0000 7f7eb3400640 1 -- 172.21.15.102:0/2516818599 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb403e5d0 con 0x7f7ebc140cf0 2024-03-23T03:50:37.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.173+0000 7f7eb3400640 1 -- 172.21.15.102:0/2516818599 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eb403e980 con 0x7f7ebc140cf0 2024-03-23T03:50:37.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.173+0000 7f7ebbe00640 1 -- 172.21.15.102:0/2516818599 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ebc15bac0 con 0x7f7ebc140cf0 2024-03-23T03:50:37.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.174+0000 7f7ebbe00640 1 -- 172.21.15.102:0/2516818599 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ebc15c080 con 0x7f7ebc140cf0 2024-03-23T03:50:37.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.175+0000 7f7ebbe00640 1 -- 172.21.15.102:0/2516818599 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ebc15cdc0 con 0x7f7ebc140cf0 2024-03-23T03:50:37.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.175+0000 7f7eb3400640 1 -- 172.21.15.102:0/2516818599 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7eb403eae0 con 0x7f7ebc140cf0 2024-03-23T03:50:37.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.176+0000 7f7eb3400640 1 --2- 172.21.15.102:0/2516818599 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7e9806ac00 0x7f7e9806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:37.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.176+0000 7f7eb3400640 1 -- 172.21.15.102:0/2516818599 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7eb40ac310 con 0x7f7ebc140cf0 2024-03-23T03:50:37.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.177+0000 7f7ebaa00640 1 --2- 172.21.15.102:0/2516818599 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7e9806ac00 0x7f7e9806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:37.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.177+0000 7f7ebaa00640 1 --2- 172.21.15.102:0/2516818599 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7e9806ac00 0x7f7e9806d0c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7eac00b840 tx=0x7f7eac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:37.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.181+0000 7f7eb3400640 1 -- 172.21.15.102:0/2516818599 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7eb4077450 con 0x7f7ebc140cf0 2024-03-23T03:50:37.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.413+0000 7f7ebbe00640 1 -- 172.21.15.102:0/2516818599 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7ebc04e030 con 0x7f7ebc140cf0 2024-03-23T03:50:37.424 INFO:tasks.ceph.mon.a.smithi102.stderr:2024-03-23T03:50:37.415+0000 7f5aab200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T03:50:37.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.433+0000 7f7eb3400640 1 -- 172.21.15.102:0/2516818599 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v42) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7eb407aa70 con 0x7f7ebc140cf0 2024-03-23T03:50:37.433 INFO:teuthology.orchestra.run.smithi102.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T03:50:37.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.435+0000 7f7ebbe00640 1 -- 172.21.15.102:0/2516818599 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7e9806ac00 msgr2=0x7f7e9806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:37.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.435+0000 7f7ebbe00640 1 --2- 172.21.15.102:0/2516818599 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7e9806ac00 0x7f7e9806d0c0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f7eac00b840 tx=0x7f7eac008040 comp rx=0 tx=0).stop 2024-03-23T03:50:37.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.435+0000 7f7ebbe00640 1 -- 172.21.15.102:0/2516818599 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ebc140cf0 msgr2=0x7f7ebc13a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:37.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.435+0000 7f7ebbe00640 1 --2- 172.21.15.102:0/2516818599 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ebc140cf0 0x7f7ebc13a6c0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f7eb4036040 tx=0x7f7eb4004a40 comp rx=0 tx=0).stop 2024-03-23T03:50:37.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.436+0000 7f7ebbe00640 1 -- 172.21.15.102:0/2516818599 shutdown_connections 2024-03-23T03:50:37.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.436+0000 7f7ebbe00640 1 --2- 172.21.15.102:0/2516818599 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7e9806ac00 0x7f7e9806d0c0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:37.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.436+0000 7f7ebbe00640 1 --2- 172.21.15.102:0/2516818599 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7ebc140cf0 0x7f7ebc13a6c0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:37.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.436+0000 7f7ebbe00640 1 --2- 172.21.15.102:0/2516818599 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7ebc0cb480 0x7f7ebc15c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:37.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.436+0000 7f7ebbe00640 1 --2- 172.21.15.102:0/2516818599 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7ebc0b23e0 0x7f7ebc15a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:37.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.436+0000 7f7ebbe00640 1 -- 172.21.15.102:0/2516818599 >> 172.21.15.102:0/2516818599 conn(0x7f7ebc065e20 msgr2=0x7f7ebc06c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:37.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.436+0000 7f7ebbe00640 1 -- 172.21.15.102:0/2516818599 shutdown_connections 2024-03-23T03:50:37.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.436+0000 7f7ebbe00640 1 -- 172.21.15.102:0/2516818599 wait complete. 2024-03-23T03:50:37.446 INFO:tasks.ceph.mds.a.smithi102.stderr: -5225> 2024-03-23T03:50:13.418+0000 7f5d41800640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T03:50:37.456 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T03:50:37.507 INFO:tasks.ceph.mds.a.smithi102.stdout:starting mds.a at 2024-03-23T03:50:37.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.646+0000 7f7b39e00640 1 Processor -- start 2024-03-23T03:50:37.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.646+0000 7f7b39e00640 1 -- start start 2024-03-23T03:50:37.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.648+0000 7f7b39e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f7b340b49f0 0x7f7b340d0d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:37.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.648+0000 7f7b39e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b3415d0f0 con 0x7f7b3415cd10 2024-03-23T03:50:37.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.648+0000 7f7b39e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b340d1a80 con 0x7f7b340d1290 2024-03-23T03:50:37.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.648+0000 7f7b39e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b340d1d60 con 0x7f7b340b49f0 2024-03-23T03:50:37.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.648+0000 7f7b32a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7b3415cd10 0x7f7b340b42e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60988/0 (socket says 172.21.15.102:60988) 2024-03-23T03:50:37.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.648+0000 7f7b32a00640 1 -- 172.21.15.102:0/3423112588 learned_addr learned my addr 172.21.15.102:0/3423112588 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.648+0000 7f7b32000640 1 --2- 172.21.15.102:0/3423112588 >> v2:172.21.15.173:3300/0 conn(0x7f7b340b49f0 0x7f7b340d0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.648+0000 7f7b31600640 1 -- 172.21.15.102:0/3423112588 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2872553015 0 0) 0x7f7b3415d0f0 con 0x7f7b3415cd10 2024-03-23T03:50:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.648+0000 7f7b31600640 1 -- 172.21.15.102:0/3423112588 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b14003520 con 0x7f7b3415cd10 2024-03-23T03:50:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.649+0000 7f7b31600640 1 -- 172.21.15.102:0/3423112588 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3638135676 0 0) 0x7f7b14003520 con 0x7f7b3415cd10 2024-03-23T03:50:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.649+0000 7f7b31600640 1 -- 172.21.15.102:0/3423112588 >> v2:172.21.15.173:3300/0 conn(0x7f7b340b49f0 msgr2=0x7f7b340d0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.649+0000 7f7b31600640 1 --2- 172.21.15.102:0/3423112588 >> v2:172.21.15.173:3300/0 conn(0x7f7b340b49f0 0x7f7b340d0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.649+0000 7f7b31600640 1 -- 172.21.15.102:0/3423112588 >> v1:172.21.15.173:6789/0 conn(0x7f7b340d1290 legacy=0x7f7b340b5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.649+0000 7f7b31600640 1 -- 172.21.15.102:0/3423112588 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b340d2040 con 0x7f7b3415cd10 2024-03-23T03:50:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.649+0000 7f7b31600640 1 -- 172.21.15.102:0/3423112588 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f7b24001670 con 0x7f7b3415cd10 2024-03-23T03:50:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.649+0000 7f7b31600640 1 -- 172.21.15.102:0/3423112588 >> v1:172.21.15.102:6789/0 conn(0x7f7b3415cd10 legacy=0x7f7b340b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.649+0000 7f7b31600640 1 --2- 172.21.15.102:0/3423112588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b140039c0 0x7f7b14003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:37.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.649+0000 7f7b31600640 1 --2- 172.21.15.102:0/3423112588 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b140047c0 0x7f7b14004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.649+0000 7f7b31600640 1 --2- 172.21.15.102:0/3423112588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b14005180 0x7f7b1400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.649+0000 7f7b31600640 1 -- 172.21.15.102:0/3423112588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b340d1d60 con 0x7f7b14005180 2024-03-23T03:50:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.649+0000 7f7b31600640 1 -- 172.21.15.102:0/3423112588 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1400c060 con 0x7f7b140039c0 2024-03-23T03:50:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.649+0000 7f7b31600640 1 -- 172.21.15.102:0/3423112588 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b1400c220 con 0x7f7b140047c0 2024-03-23T03:50:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.649+0000 7f7b33400640 1 --2- 172.21.15.102:0/3423112588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b14005180 0x7f7b1400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.649+0000 7f7b32a00640 1 --2- 172.21.15.102:0/3423112588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b140039c0 0x7f7b14003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.649+0000 7f7b33400640 1 -- 172.21.15.102:0/3423112588 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b140047c0 msgr2=0x7f7b14004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.650+0000 7f7b33400640 1 --2- 172.21.15.102:0/3423112588 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b140047c0 0x7f7b14004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.650+0000 7f7b33400640 1 -- 172.21.15.102:0/3423112588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b140039c0 msgr2=0x7f7b14003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.650+0000 7f7b33400640 1 --2- 172.21.15.102:0/3423112588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b140039c0 0x7f7b14003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.650+0000 7f7b33400640 1 -- 172.21.15.102:0/3423112588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b14003520 con 0x7f7b14005180 2024-03-23T03:50:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.650+0000 7f7b33400640 1 -- 172.21.15.102:0/3423112588 reap_dead start 2024-03-23T03:50:37.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.651+0000 7f7b33400640 1 --2- 172.21.15.102:0/3423112588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b14005180 0x7f7b1400b980 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c001750 tx=0x7f7b140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d2e4b83c006d027 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:37.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.651+0000 7f7b31600640 1 -- 172.21.15.102:0/3423112588 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c0052d0 con 0x7f7b14005180 2024-03-23T03:50:37.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.651+0000 7f7b31600640 1 -- 172.21.15.102:0/3423112588 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2c02e050 con 0x7f7b14005180 2024-03-23T03:50:37.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.651+0000 7f7b31600640 1 -- 172.21.15.102:0/3423112588 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c003830 con 0x7f7b14005180 2024-03-23T03:50:37.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.651+0000 7f7b39e00640 1 -- 172.21.15.102:0/3423112588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b14005180 msgr2=0x7f7b1400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:37.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.651+0000 7f7b39e00640 1 --2- 172.21.15.102:0/3423112588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b14005180 0x7f7b1400b980 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c001750 tx=0x7f7b140057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:37.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.651+0000 7f7b39e00640 1 -- 172.21.15.102:0/3423112588 shutdown_connections 2024-03-23T03:50:37.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.651+0000 7f7b39e00640 1 --2- 172.21.15.102:0/3423112588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b14005180 0x7f7b1400b980 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:37.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.651+0000 7f7b39e00640 1 -- 172.21.15.102:0/3423112588 >> 172.21.15.102:0/3423112588 conn(0x7f7b340cf2e0 msgr2=0x7f7b340cf6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:37.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.651+0000 7f7b39e00640 1 -- 172.21.15.102:0/3423112588 shutdown_connections 2024-03-23T03:50:37.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.652+0000 7f7b39e00640 1 -- 172.21.15.102:0/3423112588 wait complete. 2024-03-23T03:50:37.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.653+0000 7f7b39e00640 1 Processor -- start 2024-03-23T03:50:37.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.653+0000 7f7b39e00640 1 -- start start 2024-03-23T03:50:37.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.654+0000 7f7b39e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b14005180 0x7f7b341f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.654+0000 7f7b39e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b341597a0 0x7f7b34159c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.654+0000 7f7b39e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b341f3490 0x7f7b341f3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.654+0000 7f7b39e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b3415a160 con 0x7f7b341f3490 2024-03-23T03:50:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.654+0000 7f7b39e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b341f41f0 con 0x7f7b341597a0 2024-03-23T03:50:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.654+0000 7f7b39e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b341f44d0 con 0x7f7b14005180 2024-03-23T03:50:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.655+0000 7f7b33400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b341f3490 0x7f7b341f3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.655+0000 7f7b33400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b341f3490 0x7f7b341f3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50802/0 (socket says 172.21.15.102:50802) 2024-03-23T03:50:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.655+0000 7f7b33400640 1 -- 172.21.15.102:0/1881042952 learned_addr learned my addr 172.21.15.102:0/1881042952 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.655+0000 7f7b32a00640 1 --2- 172.21.15.102:0/1881042952 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b14005180 0x7f7b341f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.655+0000 7f7b32000640 1 --2- 172.21.15.102:0/1881042952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b341597a0 0x7f7b34159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.655+0000 7f7b33400640 1 -- 172.21.15.102:0/1881042952 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b14005180 msgr2=0x7f7b341f2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.655+0000 7f7b33400640 1 --2- 172.21.15.102:0/1881042952 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b14005180 0x7f7b341f2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.655+0000 7f7b33400640 1 -- 172.21.15.102:0/1881042952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b341597a0 msgr2=0x7f7b34159c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:37.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.655+0000 7f7b33400640 1 --2- 172.21.15.102:0/1881042952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b341597a0 0x7f7b34159c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.655+0000 7f7b33400640 1 -- 172.21.15.102:0/1881042952 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b341f47b0 con 0x7f7b341f3490 2024-03-23T03:50:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.655+0000 7f7b32a00640 1 --2- 172.21.15.102:0/1881042952 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b14005180 0x7f7b341f2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.655+0000 7f7b32000640 1 --2- 172.21.15.102:0/1881042952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b341597a0 0x7f7b34159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.655+0000 7f7b33400640 1 --2- 172.21.15.102:0/1881042952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b341f3490 0x7f7b341f3940 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c001870 tx=0x7f7b2c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.656+0000 7f7b2b400640 1 -- 172.21.15.102:0/1881042952 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c039070 con 0x7f7b341f3490 2024-03-23T03:50:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.656+0000 7f7b2b400640 1 -- 172.21.15.102:0/1881042952 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2c0052e0 con 0x7f7b341f3490 2024-03-23T03:50:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.656+0000 7f7b2b400640 1 -- 172.21.15.102:0/1881042952 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c007030 con 0x7f7b341f3490 2024-03-23T03:50:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.656+0000 7f7b39e00640 1 -- 172.21.15.102:0/1881042952 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b341f4a90 con 0x7f7b341f3490 2024-03-23T03:50:37.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.656+0000 7f7b39e00640 1 -- 172.21.15.102:0/1881042952 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b341f2fc0 con 0x7f7b341f3490 2024-03-23T03:50:37.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.657+0000 7f7b39e00640 1 -- 172.21.15.102:0/1881042952 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b341f4fb0 con 0x7f7b341f3490 2024-03-23T03:50:37.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.658+0000 7f7b2b400640 1 -- 172.21.15.102:0/1881042952 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7b2c03c020 con 0x7f7b341f3490 2024-03-23T03:50:37.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.659+0000 7f7b2b400640 1 --2- 172.21.15.102:0/1881042952 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7b1006ac00 0x7f7b1006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:37.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.659+0000 7f7b2b400640 1 -- 172.21.15.102:0/1881042952 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f7b2c0a9920 con 0x7f7b341f3490 2024-03-23T03:50:37.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.659+0000 7f7b32a00640 1 --2- 172.21.15.102:0/1881042952 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7b1006ac00 0x7f7b1006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:37.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.660+0000 7f7b32a00640 1 --2- 172.21.15.102:0/1881042952 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7b1006ac00 0x7f7b1006d0c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7b24000c00 tx=0x7f7b24009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:37.664 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.663+0000 7f7b2b400640 1 -- 172.21.15.102:0/1881042952 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7b341f4fb0 con 0x7f7b341f3490 2024-03-23T03:50:37.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:37.898+0000 7f7b39e00640 1 -- 172.21.15.102:0/1881042952 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7b341f4fb0 con 0x7f7b341f3490 2024-03-23T03:50:38.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.436+0000 7f7b2b400640 1 -- 172.21.15.102:0/1881042952 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v43) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7b2c074950 con 0x7f7b341f3490 2024-03-23T03:50:38.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.441+0000 7f7b39e00640 1 -- 172.21.15.102:0/1881042952 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7b1006ac00 msgr2=0x7f7b1006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:38.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.441+0000 7f7b39e00640 1 --2- 172.21.15.102:0/1881042952 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7b1006ac00 0x7f7b1006d0c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7b24000c00 tx=0x7f7b24009040 comp rx=0 tx=0).stop 2024-03-23T03:50:38.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.441+0000 7f7b39e00640 1 -- 172.21.15.102:0/1881042952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b341f3490 msgr2=0x7f7b341f3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:38.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.441+0000 7f7b39e00640 1 --2- 172.21.15.102:0/1881042952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b341f3490 0x7f7b341f3940 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c001870 tx=0x7f7b2c001120 comp rx=0 tx=0).stop 2024-03-23T03:50:38.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.442+0000 7f7b39e00640 1 -- 172.21.15.102:0/1881042952 shutdown_connections 2024-03-23T03:50:38.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.442+0000 7f7b39e00640 1 --2- 172.21.15.102:0/1881042952 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7b1006ac00 0x7f7b1006d0c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:38.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.442+0000 7f7b39e00640 1 --2- 172.21.15.102:0/1881042952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b341f3490 0x7f7b341f3940 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:38.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.442+0000 7f7b39e00640 1 --2- 172.21.15.102:0/1881042952 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b341597a0 0x7f7b34159c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:38.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.442+0000 7f7b39e00640 1 --2- 172.21.15.102:0/1881042952 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7b14005180 0x7f7b341f2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:38.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.442+0000 7f7b39e00640 1 -- 172.21.15.102:0/1881042952 >> 172.21.15.102:0/1881042952 conn(0x7f7b340cf2e0 msgr2=0x7f7b340cd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:38.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.442+0000 7f7b39e00640 1 -- 172.21.15.102:0/1881042952 shutdown_connections 2024-03-23T03:50:38.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.442+0000 7f7b39e00640 1 -- 172.21.15.102:0/1881042952 wait complete. 2024-03-23T03:50:38.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T03:50:38.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.673+0000 7f0cd3e00640 1 Processor -- start 2024-03-23T03:50:38.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.673+0000 7f0cd3e00640 1 -- start start 2024-03-23T03:50:38.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.674+0000 7f0cd3e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f0cd40b1c50 0x7f0cd40ca300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:38.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.674+0000 7f0cd3e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cd406a3a0 0x7f0cd406ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:38.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.674+0000 7f0cd3e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd40b33b0 con 0x7f0cd40b2030 2024-03-23T03:50:38.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.674+0000 7f0cd3e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd40b2f90 con 0x7f0cd40b1c50 2024-03-23T03:50:38.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.674+0000 7f0cd3e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd4159410 con 0x7f0cd406a3a0 2024-03-23T03:50:38.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.675+0000 7f0cca000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0cd40b2030 0x7f0cd4069c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32770/0 (socket says 172.21.15.102:32770) 2024-03-23T03:50:38.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.675+0000 7f0cca000640 1 -- 172.21.15.102:0/2808781849 learned_addr learned my addr 172.21.15.102:0/2808781849 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:38.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.675+0000 7f0cd2a00640 1 --2- 172.21.15.102:0/2808781849 >> v2:172.21.15.173:3300/0 conn(0x7f0cd40b1c50 0x7f0cd40ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:38.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.675+0000 7f0cd3400640 1 --2- 172.21.15.102:0/2808781849 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cd406a3a0 0x7f0cd406ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:38.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.675+0000 7f0cd2000640 1 -- 172.21.15.102:0/2808781849 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006673817 0 0) 0x7f0cd40b33b0 con 0x7f0cd40b2030 2024-03-23T03:50:38.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.675+0000 7f0cd2000640 1 -- 172.21.15.102:0/2808781849 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cb4003520 con 0x7f0cd40b2030 2024-03-23T03:50:38.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.675+0000 7f0cd2000640 1 -- 172.21.15.102:0/2808781849 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4225028168 0 0) 0x7f0cb4003520 con 0x7f0cd40b2030 2024-03-23T03:50:38.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.675+0000 7f0cd2000640 1 -- 172.21.15.102:0/2808781849 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cd406a3a0 msgr2=0x7f0cd406ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:38.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.675+0000 7f0cd2000640 1 --2- 172.21.15.102:0/2808781849 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cd406a3a0 0x7f0cd406ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:38.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.675+0000 7f0cd2000640 1 -- 172.21.15.102:0/2808781849 >> v2:172.21.15.173:3300/0 conn(0x7f0cd40b1c50 msgr2=0x7f0cd40ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:38.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.675+0000 7f0cd2000640 1 --2- 172.21.15.102:0/2808781849 >> v2:172.21.15.173:3300/0 conn(0x7f0cd40b1c50 0x7f0cd40ca300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:38.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.675+0000 7f0cd2000640 1 -- 172.21.15.102:0/2808781849 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd40b2bd0 con 0x7f0cd40b2030 2024-03-23T03:50:38.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.676+0000 7f0cd2000640 1 -- 172.21.15.102:0/2808781849 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f0cbc001670 con 0x7f0cd40b2030 2024-03-23T03:50:38.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.676+0000 7f0cd2000640 1 -- 172.21.15.102:0/2808781849 >> v1:172.21.15.102:6789/0 conn(0x7f0cd40b2030 legacy=0x7f0cd4069c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:38.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.676+0000 7f0cd2000640 1 --2- 172.21.15.102:0/2808781849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cb4003970 0x7f0cb4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:38.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.676+0000 7f0cd2000640 1 --2- 172.21.15.102:0/2808781849 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cd406a3a0 0x7f0cb40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:38.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.676+0000 7f0cd2000640 1 --2- 172.21.15.102:0/2808781849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cb4004d00 0x7f0cb4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:38.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.676+0000 7f0cd2000640 1 -- 172.21.15.102:0/2808781849 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd40b2f90 con 0x7f0cb4004d00 2024-03-23T03:50:38.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.676+0000 7f0cd2000640 1 -- 172.21.15.102:0/2808781849 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd4159410 con 0x7f0cb4003970 2024-03-23T03:50:38.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.676+0000 7f0cd2000640 1 -- 172.21.15.102:0/2808781849 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb400ba00 con 0x7f0cd406a3a0 2024-03-23T03:50:38.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.676+0000 7f0cd3400640 1 --2- 172.21.15.102:0/2808781849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cb4004d00 0x7f0cb4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:38.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.676+0000 7f0cd3400640 1 -- 172.21.15.102:0/2808781849 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cd406a3a0 msgr2=0x7f0cb40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:38.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.676+0000 7f0cd3400640 1 --2- 172.21.15.102:0/2808781849 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cd406a3a0 0x7f0cb40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:38.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.676+0000 7f0cd3400640 1 -- 172.21.15.102:0/2808781849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cb4003970 msgr2=0x7f0cb4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:38.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.676+0000 7f0cd2a00640 1 --2- 172.21.15.102:0/2808781849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cb4003970 0x7f0cb4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:38.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.676+0000 7f0cd3400640 1 --2- 172.21.15.102:0/2808781849 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cb4003970 0x7f0cb4003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:38.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.676+0000 7f0cd3400640 1 -- 172.21.15.102:0/2808781849 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb4003520 con 0x7f0cb4004d00 2024-03-23T03:50:38.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.677+0000 7f0cd3400640 1 --2- 172.21.15.102:0/2808781849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cb4004d00 0x7f0cb4005180 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc009fc0 tx=0x7f0cb40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbfc0e8cd252305e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:38.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.678+0000 7f0cd2000640 1 -- 172.21.15.102:0/2808781849 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc02cc60 con 0x7f0cb4004d00 2024-03-23T03:50:38.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.678+0000 7f0cd2000640 1 -- 172.21.15.102:0/2808781849 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ccc00f040 con 0x7f0cb4004d00 2024-03-23T03:50:38.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.678+0000 7f0cd2000640 1 -- 172.21.15.102:0/2808781849 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc034440 con 0x7f0cb4004d00 2024-03-23T03:50:38.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.678+0000 7f0cd3e00640 1 -- 172.21.15.102:0/2808781849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cb4004d00 msgr2=0x7f0cb4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:38.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.678+0000 7f0cd3e00640 1 --2- 172.21.15.102:0/2808781849 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cb4004d00 0x7f0cb4005180 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc009fc0 tx=0x7f0cb40057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:38.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.678+0000 7f0cd3400640 1 -- 172.21.15.102:0/2808781849 reap_dead start 2024-03-23T03:50:38.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.678+0000 7f0cd3e00640 1 -- 172.21.15.102:0/2808781849 shutdown_connections 2024-03-23T03:50:38.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.678+0000 7f0cd3e00640 1 -- 172.21.15.102:0/2808781849 >> 172.21.15.102:0/2808781849 conn(0x7f0cd40b01b0 msgr2=0x7f0cd40b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:38.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.678+0000 7f0cd3e00640 1 -- 172.21.15.102:0/2808781849 shutdown_connections 2024-03-23T03:50:38.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.678+0000 7f0cd3e00640 1 -- 172.21.15.102:0/2808781849 wait complete. 2024-03-23T03:50:38.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.680+0000 7f0cd3e00640 1 Processor -- start 2024-03-23T03:50:38.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.680+0000 7f0cd3e00640 1 -- start start 2024-03-23T03:50:38.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.681+0000 7f0cd3e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cb4004d00 0x7f0cd40c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:38.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.681+0000 7f0cd3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cd41595e0 0x7f0cd4159a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:38.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.681+0000 7f0cd3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cd414a2e0 0x7f0cd414a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:38.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.681+0000 7f0cd3e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd41473b0 con 0x7f0cd41595e0 2024-03-23T03:50:38.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.681+0000 7f0cd3e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd4159f60 con 0x7f0cd414a2e0 2024-03-23T03:50:38.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.681+0000 7f0cd3e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd4094eb0 con 0x7f0cb4004d00 2024-03-23T03:50:38.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.681+0000 7f0cca000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cd41595e0 0x7f0cd4159a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:38.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.682+0000 7f0cca000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cd41595e0 0x7f0cd4159a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50820/0 (socket says 172.21.15.102:50820) 2024-03-23T03:50:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.682+0000 7f0cca000640 1 -- 172.21.15.102:0/3988955740 learned_addr learned my addr 172.21.15.102:0/3988955740 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.682+0000 7f0cd2a00640 1 --2- 172.21.15.102:0/3988955740 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cb4004d00 0x7f0cd40c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.682+0000 7f0cd3400640 1 --2- 172.21.15.102:0/3988955740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cd414a2e0 0x7f0cd414a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.682+0000 7f0cca000640 1 -- 172.21.15.102:0/3988955740 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cb4004d00 msgr2=0x7f0cd40c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.682+0000 7f0cca000640 1 --2- 172.21.15.102:0/3988955740 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cb4004d00 0x7f0cd40c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.682+0000 7f0cca000640 1 -- 172.21.15.102:0/3988955740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cd414a2e0 msgr2=0x7f0cd414a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.682+0000 7f0cca000640 1 --2- 172.21.15.102:0/3988955740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cd414a2e0 0x7f0cd414a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.682+0000 7f0cca000640 1 -- 172.21.15.102:0/3988955740 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd4095190 con 0x7f0cd41595e0 2024-03-23T03:50:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.682+0000 7f0cd3400640 1 --2- 172.21.15.102:0/3988955740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cd414a2e0 0x7f0cd414a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.682+0000 7f0cca000640 1 --2- 172.21.15.102:0/3988955740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cd41595e0 0x7f0cd4159a20 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00c730 tx=0x7f0cbc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.682+0000 7f0ccb400640 1 -- 172.21.15.102:0/3988955740 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc0072d0 con 0x7f0cd41595e0 2024-03-23T03:50:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.682+0000 7f0ccb400640 1 -- 172.21.15.102:0/3988955740 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cbc007430 con 0x7f0cd41595e0 2024-03-23T03:50:38.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.682+0000 7f0ccb400640 1 -- 172.21.15.102:0/3988955740 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc0047e0 con 0x7f0cd41595e0 2024-03-23T03:50:38.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.682+0000 7f0cd3e00640 1 -- 172.21.15.102:0/3988955740 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd4095470 con 0x7f0cd41595e0 2024-03-23T03:50:38.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.683+0000 7f0cd3e00640 1 -- 172.21.15.102:0/3988955740 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd4095a30 con 0x7f0cd41595e0 2024-03-23T03:50:38.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.684+0000 7f0cd3e00640 1 -- 172.21.15.102:0/3988955740 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd4095c50 con 0x7f0cd41595e0 2024-03-23T03:50:38.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.685+0000 7f0ccb400640 1 -- 172.21.15.102:0/3988955740 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0cbc004b50 con 0x7f0cd41595e0 2024-03-23T03:50:38.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.685+0000 7f0ccb400640 1 --2- 172.21.15.102:0/3988955740 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0cb006ac00 0x7f0cb006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:38.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.686+0000 7f0ccb400640 1 -- 172.21.15.102:0/3988955740 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f0cbc088ba0 con 0x7f0cd41595e0 2024-03-23T03:50:38.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.686+0000 7f0cd2a00640 1 --2- 172.21.15.102:0/3988955740 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0cb006ac00 0x7f0cb006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:38.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.686+0000 7f0cd2a00640 1 --2- 172.21.15.102:0/3988955740 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0cb006ac00 0x7f0cb006d0c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4009a90 tx=0x7f0cc4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:38.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.690+0000 7f0ccb400640 1 -- 172.21.15.102:0/3988955740 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0cbc053bd0 con 0x7f0cd41595e0 2024-03-23T03:50:38.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:38.881+0000 7f0cd3e00640 1 -- 172.21.15.102:0/3988955740 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0cd404e030 con 0x7f0cd41595e0 2024-03-23T03:50:39.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.447+0000 7f0ccb400640 1 -- 172.21.15.102:0/3988955740 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v83) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0cbc0571f0 con 0x7f0cd41595e0 2024-03-23T03:50:39.449 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' removed 2024-03-23T03:50:39.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.451+0000 7f0cd3e00640 1 -- 172.21.15.102:0/3988955740 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0cb006ac00 msgr2=0x7f0cb006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:39.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.451+0000 7f0cd3e00640 1 --2- 172.21.15.102:0/3988955740 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0cb006ac00 0x7f0cb006d0c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f0cc4009a90 tx=0x7f0cc4008040 comp rx=0 tx=0).stop 2024-03-23T03:50:39.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.451+0000 7f0cd3e00640 1 -- 172.21.15.102:0/3988955740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cd41595e0 msgr2=0x7f0cd4159a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:39.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.451+0000 7f0cd3e00640 1 --2- 172.21.15.102:0/3988955740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cd41595e0 0x7f0cd4159a20 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00c730 tx=0x7f0cbc00cbf0 comp rx=0 tx=0).stop 2024-03-23T03:50:39.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.451+0000 7f0cd3e00640 1 -- 172.21.15.102:0/3988955740 shutdown_connections 2024-03-23T03:50:39.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.451+0000 7f0cd3e00640 1 --2- 172.21.15.102:0/3988955740 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0cb006ac00 0x7f0cb006d0c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:39.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.451+0000 7f0cd3e00640 1 --2- 172.21.15.102:0/3988955740 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cd414a2e0 0x7f0cd414a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:39.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.451+0000 7f0cd3e00640 1 --2- 172.21.15.102:0/3988955740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cd41595e0 0x7f0cd4159a20 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:39.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.451+0000 7f0cd3e00640 1 --2- 172.21.15.102:0/3988955740 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cb4004d00 0x7f0cd40c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:39.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.451+0000 7f0cd3e00640 1 -- 172.21.15.102:0/3988955740 >> 172.21.15.102:0/3988955740 conn(0x7f0cd40b01b0 msgr2=0x7f0cd40b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:39.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.451+0000 7f0cd3e00640 1 -- 172.21.15.102:0/3988955740 shutdown_connections 2024-03-23T03:50:39.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.451+0000 7f0cd3e00640 1 -- 172.21.15.102:0/3988955740 wait complete. 2024-03-23T03:50:39.471 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T03:50:39.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.676+0000 7f4f8c000640 1 Processor -- start 2024-03-23T03:50:39.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.676+0000 7f4f8c000640 1 -- start start 2024-03-23T03:50:39.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.678+0000 7f4f8c000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f4f840cf040 0x7f4f84164920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:39.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.678+0000 7f4f8c000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f84165830 con 0x7f4f840ca7b0 2024-03-23T03:50:39.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.678+0000 7f4f8c000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f84164f90 con 0x7f4f84160d20 2024-03-23T03:50:39.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.678+0000 7f4f8c000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f84165270 con 0x7f4f840cf040 2024-03-23T03:50:39.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.678+0000 7f4f8a000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4f840ca7b0 0x7f4f840b1b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32784/0 (socket says 172.21.15.102:32784) 2024-03-23T03:50:39.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.678+0000 7f4f8a000640 1 -- 172.21.15.102:0/2953500398 learned_addr learned my addr 172.21.15.102:0/2953500398 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:39.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.679+0000 7f4f88c00640 1 --2- 172.21.15.102:0/2953500398 >> v2:172.21.15.173:3300/0 conn(0x7f4f840cf040 0x7f4f84164920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:39.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.679+0000 7f4f83e00640 1 -- 172.21.15.102:0/2953500398 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1400140338 0 0) 0x7f4f84165830 con 0x7f4f840ca7b0 2024-03-23T03:50:39.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.679+0000 7f4f83e00640 1 -- 172.21.15.102:0/2953500398 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f6c003520 con 0x7f4f840ca7b0 2024-03-23T03:50:39.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.679+0000 7f4f83e00640 1 -- 172.21.15.102:0/2953500398 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3388729365 0 0) 0x7f4f6c003520 con 0x7f4f840ca7b0 2024-03-23T03:50:39.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.679+0000 7f4f83e00640 1 -- 172.21.15.102:0/2953500398 >> v2:172.21.15.173:3300/0 conn(0x7f4f840cf040 msgr2=0x7f4f84164920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:39.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.679+0000 7f4f83e00640 1 --2- 172.21.15.102:0/2953500398 >> v2:172.21.15.173:3300/0 conn(0x7f4f840cf040 0x7f4f84164920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:39.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.679+0000 7f4f83e00640 1 -- 172.21.15.102:0/2953500398 >> v1:172.21.15.173:6789/0 conn(0x7f4f84160d20 legacy=0x7f4f840ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:39.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.679+0000 7f4f83e00640 1 -- 172.21.15.102:0/2953500398 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f84165550 con 0x7f4f840ca7b0 2024-03-23T03:50:39.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.679+0000 7f4f83e00640 1 -- 172.21.15.102:0/2953500398 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f4f7c001670 con 0x7f4f840ca7b0 2024-03-23T03:50:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.679+0000 7f4f83e00640 1 -- 172.21.15.102:0/2953500398 >> v1:172.21.15.102:6789/0 conn(0x7f4f840ca7b0 legacy=0x7f4f840b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.679+0000 7f4f83e00640 1 --2- 172.21.15.102:0/2953500398 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f6c003970 0x7f4f6c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.679+0000 7f4f83e00640 1 --2- 172.21.15.102:0/2953500398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f6c0047c0 0x7f4f6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.679+0000 7f4f83e00640 1 --2- 172.21.15.102:0/2953500398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f6c005180 0x7f4f6c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.680+0000 7f4f83e00640 1 -- 172.21.15.102:0/2953500398 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f84165270 con 0x7f4f6c005180 2024-03-23T03:50:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.680+0000 7f4f83e00640 1 -- 172.21.15.102:0/2953500398 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c00c120 con 0x7f4f6c0047c0 2024-03-23T03:50:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.680+0000 7f4f83e00640 1 -- 172.21.15.102:0/2953500398 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c00c2e0 con 0x7f4f6c003970 2024-03-23T03:50:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.680+0000 7f4f8a000640 1 --2- 172.21.15.102:0/2953500398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f6c005180 0x7f4f6c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.680+0000 7f4f89600640 1 --2- 172.21.15.102:0/2953500398 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f6c003970 0x7f4f6c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.680+0000 7f4f8a000640 1 -- 172.21.15.102:0/2953500398 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f6c003970 msgr2=0x7f4f6c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:39.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.680+0000 7f4f8a000640 1 --2- 172.21.15.102:0/2953500398 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f6c003970 0x7f4f6c003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:39.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.680+0000 7f4f8a000640 1 -- 172.21.15.102:0/2953500398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f6c0047c0 msgr2=0x7f4f6c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:39.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.680+0000 7f4f8a000640 1 --2- 172.21.15.102:0/2953500398 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f6c0047c0 0x7f4f6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:39.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.680+0000 7f4f8a000640 1 -- 172.21.15.102:0/2953500398 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6c003520 con 0x7f4f6c005180 2024-03-23T03:50:39.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.681+0000 7f4f8a000640 1 -- 172.21.15.102:0/2953500398 reap_dead start 2024-03-23T03:50:39.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.681+0000 7f4f8a000640 1 --2- 172.21.15.102:0/2953500398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f6c005180 0x7f4f6c00b980 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c00c240 tx=0x7f4f6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6a5958ea76882f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:39.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.681+0000 7f4f83e00640 1 -- 172.21.15.102:0/2953500398 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c005ad0 con 0x7f4f6c005180 2024-03-23T03:50:39.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.682+0000 7f4f83e00640 1 -- 172.21.15.102:0/2953500398 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7c005c70 con 0x7f4f6c005180 2024-03-23T03:50:39.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.682+0000 7f4f83e00640 1 -- 172.21.15.102:0/2953500398 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c02c8b0 con 0x7f4f6c005180 2024-03-23T03:50:39.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.682+0000 7f4f8c000640 1 -- 172.21.15.102:0/2953500398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f6c005180 msgr2=0x7f4f6c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:39.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.682+0000 7f4f8c000640 1 --2- 172.21.15.102:0/2953500398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f6c005180 0x7f4f6c00b980 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c00c240 tx=0x7f4f6c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:39.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.682+0000 7f4f8c000640 1 -- 172.21.15.102:0/2953500398 shutdown_connections 2024-03-23T03:50:39.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.682+0000 7f4f8c000640 1 --2- 172.21.15.102:0/2953500398 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f6c005180 0x7f4f6c00b980 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:39.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.682+0000 7f4f8c000640 1 -- 172.21.15.102:0/2953500398 >> 172.21.15.102:0/2953500398 conn(0x7f4f8406a0b0 msgr2=0x7f4f8406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:39.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.682+0000 7f4f8c000640 1 -- 172.21.15.102:0/2953500398 shutdown_connections 2024-03-23T03:50:39.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.682+0000 7f4f8c000640 1 -- 172.21.15.102:0/2953500398 wait complete. 2024-03-23T03:50:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.684+0000 7f4f8c000640 1 Processor -- start 2024-03-23T03:50:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.684+0000 7f4f8c000640 1 -- start start 2024-03-23T03:50:39.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.685+0000 7f4f8c000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f6c005180 0x7f4f841415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.685+0000 7f4f8c000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f8415ee20 0x7f4f8415f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.685+0000 7f4f8c000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f8406d2a0 0x7f4f8406d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.685+0000 7f4f8c000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f841604f0 con 0x7f4f8406d2a0 2024-03-23T03:50:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.685+0000 7f4f8c000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8406dc60 con 0x7f4f8415ee20 2024-03-23T03:50:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.685+0000 7f4f8c000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8415f970 con 0x7f4f6c005180 2024-03-23T03:50:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.685+0000 7f4f8a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f8406d2a0 0x7f4f8406d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.685+0000 7f4f8a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f8406d2a0 0x7f4f8406d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50826/0 (socket says 172.21.15.102:50826) 2024-03-23T03:50:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.685+0000 7f4f8a000640 1 -- 172.21.15.102:0/1073460588 learned_addr learned my addr 172.21.15.102:0/1073460588 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.686+0000 7f4f88c00640 1 --2- 172.21.15.102:0/1073460588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f8415ee20 0x7f4f8415f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.686+0000 7f4f89600640 1 --2- 172.21.15.102:0/1073460588 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f6c005180 0x7f4f841415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.686+0000 7f4f8a000640 1 -- 172.21.15.102:0/1073460588 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f6c005180 msgr2=0x7f4f841415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.686+0000 7f4f8a000640 1 --2- 172.21.15.102:0/1073460588 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f6c005180 0x7f4f841415c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.686+0000 7f4f8a000640 1 -- 172.21.15.102:0/1073460588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f8415ee20 msgr2=0x7f4f8415f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:39.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.686+0000 7f4f8a000640 1 --2- 172.21.15.102:0/1073460588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f8415ee20 0x7f4f8415f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:39.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.686+0000 7f4f8a000640 1 -- 172.21.15.102:0/1073460588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8415fc50 con 0x7f4f8406d2a0 2024-03-23T03:50:39.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.686+0000 7f4f88c00640 1 --2- 172.21.15.102:0/1073460588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f8415ee20 0x7f4f8415f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:39.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.686+0000 7f4f8a000640 1 --2- 172.21.15.102:0/1073460588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f8406d2a0 0x7f4f8406d720 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c00c360 tx=0x7f4f7c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:39.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.686+0000 7f4f81600640 1 -- 172.21.15.102:0/1073460588 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c0049e0 con 0x7f4f8406d2a0 2024-03-23T03:50:39.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.686+0000 7f4f81600640 1 -- 172.21.15.102:0/1073460588 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7c004b40 con 0x7f4f8406d2a0 2024-03-23T03:50:39.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.686+0000 7f4f81600640 1 -- 172.21.15.102:0/1073460588 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c001640 con 0x7f4f8406d2a0 2024-03-23T03:50:39.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.686+0000 7f4f8c000640 1 -- 172.21.15.102:0/1073460588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f8415ff30 con 0x7f4f8406d2a0 2024-03-23T03:50:39.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.686+0000 7f4f8c000640 1 -- 172.21.15.102:0/1073460588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f84162790 con 0x7f4f8406d2a0 2024-03-23T03:50:39.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.687+0000 7f4f8c000640 1 -- 172.21.15.102:0/1073460588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f84162920 con 0x7f4f8406d2a0 2024-03-23T03:50:39.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.689+0000 7f4f81600640 1 -- 172.21.15.102:0/1073460588 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4f7c032070 con 0x7f4f8406d2a0 2024-03-23T03:50:39.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.689+0000 7f4f81600640 1 --2- 172.21.15.102:0/1073460588 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f6806ac00 0x7f4f6806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:39.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.689+0000 7f4f81600640 1 -- 172.21.15.102:0/1073460588 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5677+0+0 (secure 0 0 0) 0x7f4f7c03c070 con 0x7f4f8406d2a0 2024-03-23T03:50:39.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.690+0000 7f4f89600640 1 --2- 172.21.15.102:0/1073460588 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f6806ac00 0x7f4f6806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:39.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.690+0000 7f4f89600640 1 --2- 172.21.15.102:0/1073460588 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f6806ac00 0x7f4f6806d0c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4f78000c00 tx=0x7f4f78009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:39.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.694+0000 7f4f81600640 1 -- 172.21.15.102:0/1073460588 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4f7c075840 con 0x7f4f8406d2a0 2024-03-23T03:50:39.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:39.901+0000 7f4f8c000640 1 -- 172.21.15.102:0/1073460588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4f84073390 con 0x7f4f8406d2a0 2024-03-23T03:50:40.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.712+0000 7f4f81600640 1 -- 172.21.15.102:0/1073460588 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v84) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4f7c078e60 con 0x7f4f8406d2a0 2024-03-23T03:50:40.714 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' removed 2024-03-23T03:50:40.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.718+0000 7f4f8c000640 1 -- 172.21.15.102:0/1073460588 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f6806ac00 msgr2=0x7f4f6806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:40.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.718+0000 7f4f8c000640 1 --2- 172.21.15.102:0/1073460588 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f6806ac00 0x7f4f6806d0c0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4f78000c00 tx=0x7f4f78009040 comp rx=0 tx=0).stop 2024-03-23T03:50:40.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.718+0000 7f4f8c000640 1 -- 172.21.15.102:0/1073460588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f8406d2a0 msgr2=0x7f4f8406d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:40.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.718+0000 7f4f8c000640 1 --2- 172.21.15.102:0/1073460588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f8406d2a0 0x7f4f8406d720 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c00c360 tx=0x7f4f7c001a00 comp rx=0 tx=0).stop 2024-03-23T03:50:40.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.718+0000 7f4f8c000640 1 -- 172.21.15.102:0/1073460588 shutdown_connections 2024-03-23T03:50:40.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.718+0000 7f4f8c000640 1 --2- 172.21.15.102:0/1073460588 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f6806ac00 0x7f4f6806d0c0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:40.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.718+0000 7f4f8c000640 1 --2- 172.21.15.102:0/1073460588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f8406d2a0 0x7f4f8406d720 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:40.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.718+0000 7f4f8c000640 1 --2- 172.21.15.102:0/1073460588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f8415ee20 0x7f4f8415f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:40.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.718+0000 7f4f8c000640 1 --2- 172.21.15.102:0/1073460588 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f6c005180 0x7f4f841415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:40.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.718+0000 7f4f8c000640 1 -- 172.21.15.102:0/1073460588 >> 172.21.15.102:0/1073460588 conn(0x7f4f8406a0b0 msgr2=0x7f4f840b0a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:40.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.719+0000 7f4f8c000640 1 -- 172.21.15.102:0/1073460588 shutdown_connections 2024-03-23T03:50:40.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.719+0000 7f4f8c000640 1 -- 172.21.15.102:0/1073460588 wait complete. 2024-03-23T03:50:40.745 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-23T03:50:40.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.951+0000 7f3e11800640 1 Processor -- start 2024-03-23T03:50:40.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.951+0000 7f3e11800640 1 -- start start 2024-03-23T03:50:40.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.952+0000 7f3e11800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e0c0b22b0 0x7f3e0c15eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:40.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.952+0000 7f3e11800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e0c15e320 con 0x7f3e0c15a850 2024-03-23T03:50:40.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.952+0000 7f3e11800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e0c0cda20 con 0x7f3e0c15f020 2024-03-23T03:50:40.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.952+0000 7f3e11800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c15ac30 con 0x7f3e0c0b22b0 2024-03-23T03:50:40.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.952+0000 7f3e0be00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f3e0c15a850 0x7f3e0c0cd1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32792/0 (socket says 172.21.15.102:32792) 2024-03-23T03:50:40.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.952+0000 7f3e0be00640 1 -- 172.21.15.102:0/2910892326 learned_addr learned my addr 172.21.15.102:0/2910892326 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:40.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.953+0000 7f3e0aa00640 1 -- 172.21.15.102:0/2910892326 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612251596 0 0) 0x7f3e0c15e320 con 0x7f3e0c15a850 2024-03-23T03:50:40.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.953+0000 7f3e0aa00640 1 -- 172.21.15.102:0/2910892326 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3df0003520 con 0x7f3e0c15a850 2024-03-23T03:50:40.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.953+0000 7f3e0b400640 1 --2- 172.21.15.102:0/2910892326 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e0c0b22b0 0x7f3e0c15eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:40.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.953+0000 7f3e0aa00640 1 -- 172.21.15.102:0/2910892326 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3578932045 0 0) 0x7f3df0003520 con 0x7f3e0c15a850 2024-03-23T03:50:40.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.953+0000 7f3e0aa00640 1 -- 172.21.15.102:0/2910892326 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e0c0b22b0 msgr2=0x7f3e0c15eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:40.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.953+0000 7f3e0aa00640 1 --2- 172.21.15.102:0/2910892326 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e0c0b22b0 0x7f3e0c15eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:40.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.953+0000 7f3e0aa00640 1 -- 172.21.15.102:0/2910892326 >> v1:172.21.15.173:6789/0 conn(0x7f3e0c15f020 legacy=0x7f3e0c0cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:40.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.953+0000 7f3e0aa00640 1 -- 172.21.15.102:0/2910892326 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0c0cdd00 con 0x7f3e0c15a850 2024-03-23T03:50:40.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.953+0000 7f3e0aa00640 1 -- 172.21.15.102:0/2910892326 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f3e040019f0 con 0x7f3e0c15a850 2024-03-23T03:50:40.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.953+0000 7f3e0aa00640 1 -- 172.21.15.102:0/2910892326 >> v1:172.21.15.102:6789/0 conn(0x7f3e0c15a850 legacy=0x7f3e0c0cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:40.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.953+0000 7f3e0aa00640 1 --2- 172.21.15.102:0/2910892326 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e0c0b22b0 0x7f3df00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:40.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.953+0000 7f3e0aa00640 1 --2- 172.21.15.102:0/2910892326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3df0004000 0x7f3df00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:40.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.953+0000 7f3e0aa00640 1 --2- 172.21.15.102:0/2910892326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3df0004d00 0x7f3df0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:40.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.953+0000 7f3e0aa00640 1 -- 172.21.15.102:0/2910892326 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c15ac30 con 0x7f3df0004000 2024-03-23T03:50:40.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.953+0000 7f3e0aa00640 1 -- 172.21.15.102:0/2910892326 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3df000bb70 con 0x7f3df0004d00 2024-03-23T03:50:40.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.953+0000 7f3e0aa00640 1 -- 172.21.15.102:0/2910892326 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3df000bd30 con 0x7f3e0c0b22b0 2024-03-23T03:50:40.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.954+0000 7f3e0b400640 1 --2- 172.21.15.102:0/2910892326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3df0004000 0x7f3df00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:40.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.954+0000 7f3e0be00640 1 --2- 172.21.15.102:0/2910892326 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e0c0b22b0 0x7f3df00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:40.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.954+0000 7f3e0b400640 1 -- 172.21.15.102:0/2910892326 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e0c0b22b0 msgr2=0x7f3df00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:40.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.954+0000 7f3e0b400640 1 --2- 172.21.15.102:0/2910892326 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3e0c0b22b0 0x7f3df00039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:40.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.954+0000 7f3e0b400640 1 -- 172.21.15.102:0/2910892326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3df0004d00 msgr2=0x7f3df0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:40.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.954+0000 7f3e0b400640 1 --2- 172.21.15.102:0/2910892326 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3df0004d00 0x7f3df0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:40.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.954+0000 7f3e0b400640 1 -- 172.21.15.102:0/2910892326 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3df0003520 con 0x7f3df0004000 2024-03-23T03:50:40.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.955+0000 7f3e0b400640 1 --2- 172.21.15.102:0/2910892326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3df0004000 0x7f3df00047c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f3df400ba30 tx=0x7f3df00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a21d8c3dfb75e4d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:40.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.955+0000 7f3e0aa00640 1 -- 172.21.15.102:0/2910892326 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4032640 con 0x7f3df0004000 2024-03-23T03:50:40.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.955+0000 7f3e0aa00640 1 -- 172.21.15.102:0/2910892326 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3df40327a0 con 0x7f3df0004000 2024-03-23T03:50:40.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.955+0000 7f3e0aa00640 1 -- 172.21.15.102:0/2910892326 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3df4032b30 con 0x7f3df0004000 2024-03-23T03:50:40.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.956+0000 7f3e11800640 1 -- 172.21.15.102:0/2910892326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3df0004000 msgr2=0x7f3df00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:40.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.956+0000 7f3e11800640 1 --2- 172.21.15.102:0/2910892326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3df0004000 0x7f3df00047c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f3df400ba30 tx=0x7f3df00057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:40.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.956+0000 7f3e10e00640 1 -- 172.21.15.102:0/2910892326 reap_dead start 2024-03-23T03:50:40.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.956+0000 7f3e11800640 1 -- 172.21.15.102:0/2910892326 shutdown_connections 2024-03-23T03:50:40.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.956+0000 7f3e11800640 1 -- 172.21.15.102:0/2910892326 >> 172.21.15.102:0/2910892326 conn(0x7f3e0c0b01b0 msgr2=0x7f3e0c0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:40.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.956+0000 7f3e11800640 1 -- 172.21.15.102:0/2910892326 shutdown_connections 2024-03-23T03:50:40.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.956+0000 7f3e11800640 1 -- 172.21.15.102:0/2910892326 wait complete. 2024-03-23T03:50:40.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.958+0000 7f3e11800640 1 Processor -- start 2024-03-23T03:50:40.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.958+0000 7f3e11800640 1 -- start start 2024-03-23T03:50:40.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.959+0000 7f3e11800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3df0004000 0x7f3e0c0709a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:40.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.959+0000 7f3e11800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e0c1511b0 0x7f3e0c0ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:40.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.959+0000 7f3e11800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c0711a0 0x7f3e0c0ae810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:40.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.959+0000 7f3e11800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c071a20 con 0x7f3e0c0711a0 2024-03-23T03:50:40.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.959+0000 7f3e11800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c098560 con 0x7f3e0c1511b0 2024-03-23T03:50:40.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.959+0000 7f3e11800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c071740 con 0x7f3df0004000 2024-03-23T03:50:40.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.959+0000 7f3e10e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c0711a0 0x7f3e0c0ae810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:40.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.959+0000 7f3e10e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c0711a0 0x7f3e0c0ae810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50840/0 (socket says 172.21.15.102:50840) 2024-03-23T03:50:40.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.959+0000 7f3e10e00640 1 -- 172.21.15.102:0/425057506 learned_addr learned my addr 172.21.15.102:0/425057506 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:40.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.959+0000 7f3e0be00640 1 --2- 172.21.15.102:0/425057506 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3df0004000 0x7f3e0c0709a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:40.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.959+0000 7f3e0b400640 1 --2- 172.21.15.102:0/425057506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e0c1511b0 0x7f3e0c0ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:40.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.959+0000 7f3e10e00640 1 -- 172.21.15.102:0/425057506 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3df0004000 msgr2=0x7f3e0c0709a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:40.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.959+0000 7f3e10e00640 1 --2- 172.21.15.102:0/425057506 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3df0004000 0x7f3e0c0709a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:40.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.959+0000 7f3e10e00640 1 -- 172.21.15.102:0/425057506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e0c1511b0 msgr2=0x7f3e0c0ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:40.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.959+0000 7f3e10e00640 1 --2- 172.21.15.102:0/425057506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e0c1511b0 0x7f3e0c0ae2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:40.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.959+0000 7f3e10e00640 1 -- 172.21.15.102:0/425057506 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0c15af40 con 0x7f3e0c0711a0 2024-03-23T03:50:40.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.960+0000 7f3e10e00640 1 --2- 172.21.15.102:0/425057506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c0711a0 0x7f3e0c0ae810 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c0c8900 tx=0x7f3dfc00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:40.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.960+0000 7f3e03e00640 1 -- 172.21.15.102:0/425057506 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc0074e0 con 0x7f3e0c0711a0 2024-03-23T03:50:40.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.960+0000 7f3e03e00640 1 -- 172.21.15.102:0/425057506 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dfc007640 con 0x7f3e0c0711a0 2024-03-23T03:50:40.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.960+0000 7f3e03e00640 1 -- 172.21.15.102:0/425057506 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dfc005a00 con 0x7f3e0c0711a0 2024-03-23T03:50:40.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.960+0000 7f3e11800640 1 -- 172.21.15.102:0/425057506 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0c15b220 con 0x7f3e0c0711a0 2024-03-23T03:50:40.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.960+0000 7f3e11800640 1 -- 172.21.15.102:0/425057506 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e0c070ee0 con 0x7f3e0c0711a0 2024-03-23T03:50:40.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.961+0000 7f3e11800640 1 -- 172.21.15.102:0/425057506 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e0c15ada0 con 0x7f3e0c0711a0 2024-03-23T03:50:40.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.962+0000 7f3e03e00640 1 -- 172.21.15.102:0/425057506 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3dfc005b60 con 0x7f3e0c0711a0 2024-03-23T03:50:40.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.963+0000 7f3e03e00640 1 --2- 172.21.15.102:0/425057506 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3de806ac30 0x7f3de806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:40.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.963+0000 7f3e03e00640 1 -- 172.21.15.102:0/425057506 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f3dfc0884f0 con 0x7f3e0c0711a0 2024-03-23T03:50:40.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.963+0000 7f3e0be00640 1 --2- 172.21.15.102:0/425057506 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3de806ac30 0x7f3de806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:40.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.964+0000 7f3e0be00640 1 --2- 172.21.15.102:0/425057506 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3de806ac30 0x7f3de806d0f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400e1c0 tx=0x7f3e04009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:40.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:40.968+0000 7f3e03e00640 1 -- 172.21.15.102:0/425057506 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3e0c15ada0 con 0x7f3e0c0711a0 2024-03-23T03:50:41.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.161+0000 7f3e11800640 1 -- 172.21.15.102:0/425057506 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3e0c15b740 con 0x7f3e0c0711a0 2024-03-23T03:50:41.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.167+0000 7f3e03e00640 1 -- 172.21.15.102:0/425057506 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3e0c15b740 con 0x7f3e0c0711a0 2024-03-23T03:50:41.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.169+0000 7f3e11800640 1 -- 172.21.15.102:0/425057506 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3de806ac30 msgr2=0x7f3de806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:41.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.169+0000 7f3e11800640 1 --2- 172.21.15.102:0/425057506 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3de806ac30 0x7f3de806d0f0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400e1c0 tx=0x7f3e04009040 comp rx=0 tx=0).stop 2024-03-23T03:50:41.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.169+0000 7f3e11800640 1 -- 172.21.15.102:0/425057506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c0711a0 msgr2=0x7f3e0c0ae810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:41.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.169+0000 7f3e11800640 1 --2- 172.21.15.102:0/425057506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c0711a0 0x7f3e0c0ae810 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c0c8900 tx=0x7f3dfc00c820 comp rx=0 tx=0).stop 2024-03-23T03:50:41.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.170+0000 7f3e11800640 1 -- 172.21.15.102:0/425057506 shutdown_connections 2024-03-23T03:50:41.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.170+0000 7f3e11800640 1 --2- 172.21.15.102:0/425057506 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3de806ac30 0x7f3de806d0f0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.170+0000 7f3e11800640 1 --2- 172.21.15.102:0/425057506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c0711a0 0x7f3e0c0ae810 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.170+0000 7f3e11800640 1 --2- 172.21.15.102:0/425057506 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3e0c1511b0 0x7f3e0c0ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.170+0000 7f3e11800640 1 --2- 172.21.15.102:0/425057506 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3df0004000 0x7f3e0c0709a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.170+0000 7f3e11800640 1 -- 172.21.15.102:0/425057506 >> 172.21.15.102:0/425057506 conn(0x7f3e0c0b01b0 msgr2=0x7f3e0c0af840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:41.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.170+0000 7f3e11800640 1 -- 172.21.15.102:0/425057506 shutdown_connections 2024-03-23T03:50:41.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.170+0000 7f3e11800640 1 -- 172.21.15.102:0/425057506 wait complete. 2024-03-23T03:50:41.189 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_df' 2024-03-23T03:50:41.400 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.400+0000 7fb8b4c00640 1 Processor -- start 2024-03-23T03:50:41.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.400+0000 7fb8b4c00640 1 -- start start 2024-03-23T03:50:41.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.401+0000 7fb8b4c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb8b006d390 0x7fb8b006d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:41.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.401+0000 7fb8b4c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b00b16a0 con 0x7fb8b00cbab0 2024-03-23T03:50:41.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.401+0000 7fb8b4c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b00cbfb0 con 0x7fb8b00adbc0 2024-03-23T03:50:41.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.401+0000 7fb8b4c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b006dd40 con 0x7fb8b006d390 2024-03-23T03:50:41.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.402+0000 7fb8a7e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fb8b00cbab0 0x7fb8b00b20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32794/0 (socket says 172.21.15.102:32794) 2024-03-23T03:50:41.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.402+0000 7fb8a7e00640 1 -- 172.21.15.102:0/1696920993 learned_addr learned my addr 172.21.15.102:0/1696920993 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:41.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.402+0000 7fb8ae800640 1 --2- 172.21.15.102:0/1696920993 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb8b006d390 0x7fb8b006d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:41.404 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.402+0000 7fb8ad400640 1 -- 172.21.15.102:0/1696920993 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167987111 0 0) 0x7fb8b00b16a0 con 0x7fb8b00cbab0 2024-03-23T03:50:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.402+0000 7fb8ad400640 1 -- 172.21.15.102:0/1696920993 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb894005730 con 0x7fb8b00cbab0 2024-03-23T03:50:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8ad400640 1 -- 172.21.15.102:0/1696920993 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2777534011 0 0) 0x7fb894005730 con 0x7fb8b00cbab0 2024-03-23T03:50:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8ad400640 1 -- 172.21.15.102:0/1696920993 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb8b006d390 msgr2=0x7fb8b006d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8ad400640 1 --2- 172.21.15.102:0/1696920993 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb8b006d390 0x7fb8b006d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8ad400640 1 -- 172.21.15.102:0/1696920993 >> v1:172.21.15.173:6789/0 conn(0x7fb8b00adbc0 legacy=0x7fb8b00b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8ad400640 1 -- 172.21.15.102:0/1696920993 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b00cc290 con 0x7fb8b00cbab0 2024-03-23T03:50:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8ad400640 1 -- 172.21.15.102:0/1696920993 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fb898001670 con 0x7fb8b00cbab0 2024-03-23T03:50:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8ad400640 1 -- 172.21.15.102:0/1696920993 >> v1:172.21.15.102:6789/0 conn(0x7fb8b00cbab0 legacy=0x7fb8b00b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8ad400640 1 --2- 172.21.15.102:0/1696920993 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb8b006d390 0x7fb894005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8ad400640 1 --2- 172.21.15.102:0/1696920993 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb894006130 0x7fb8940069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8ad400640 1 --2- 172.21.15.102:0/1696920993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb894006f10 0x7fb894007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:41.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8ad400640 1 -- 172.21.15.102:0/1696920993 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b006dd40 con 0x7fb894006130 2024-03-23T03:50:41.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8ad400640 1 -- 172.21.15.102:0/1696920993 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89400dd20 con 0x7fb894006f10 2024-03-23T03:50:41.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8ad400640 1 -- 172.21.15.102:0/1696920993 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89400dee0 con 0x7fb8b006d390 2024-03-23T03:50:41.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8a7e00640 1 --2- 172.21.15.102:0/1696920993 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb894006130 0x7fb8940069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:41.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8ae800640 1 --2- 172.21.15.102:0/1696920993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb894006f10 0x7fb894007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:41.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8ade00640 1 --2- 172.21.15.102:0/1696920993 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb8b006d390 0x7fb894005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:41.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8a7e00640 1 -- 172.21.15.102:0/1696920993 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb8b006d390 msgr2=0x7fb894005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:41.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8a7e00640 1 --2- 172.21.15.102:0/1696920993 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb8b006d390 0x7fb894005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8a7e00640 1 -- 172.21.15.102:0/1696920993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb894006f10 msgr2=0x7fb894007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:41.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8a7e00640 1 --2- 172.21.15.102:0/1696920993 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb894006f10 0x7fb894007390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.403+0000 7fb8a7e00640 1 -- 172.21.15.102:0/1696920993 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb894005730 con 0x7fb894006130 2024-03-23T03:50:41.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.404+0000 7fb8ade00640 1 --2- 172.21.15.102:0/1696920993 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb8b006d390 0x7fb894005bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:41.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.405+0000 7fb8a7e00640 1 --2- 172.21.15.102:0/1696920993 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb894006130 0x7fb8940069d0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fb89800c210 tx=0x7fb894007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c71d79c5a3d8a052 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.405+0000 7fb8ad400640 1 -- 172.21.15.102:0/1696920993 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898005ad0 con 0x7fb894006130 2024-03-23T03:50:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.405+0000 7fb8ad400640 1 -- 172.21.15.102:0/1696920993 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb898005c30 con 0x7fb894006130 2024-03-23T03:50:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.405+0000 7fb8ad400640 1 -- 172.21.15.102:0/1696920993 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89802c890 con 0x7fb894006130 2024-03-23T03:50:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.405+0000 7fb8b4c00640 1 -- 172.21.15.102:0/1696920993 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb894006130 msgr2=0x7fb8940069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.405+0000 7fb8b4c00640 1 --2- 172.21.15.102:0/1696920993 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb894006130 0x7fb8940069d0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fb89800c210 tx=0x7fb894007a00 comp rx=0 tx=0).stop 2024-03-23T03:50:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.405+0000 7fb8ae800640 1 -- 172.21.15.102:0/1696920993 reap_dead start 2024-03-23T03:50:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.406+0000 7fb8b4c00640 1 -- 172.21.15.102:0/1696920993 shutdown_connections 2024-03-23T03:50:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.406+0000 7fb8b4c00640 1 -- 172.21.15.102:0/1696920993 >> 172.21.15.102:0/1696920993 conn(0x7fb8b006a3d0 msgr2=0x7fb8b00b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.406+0000 7fb8b4c00640 1 -- 172.21.15.102:0/1696920993 shutdown_connections 2024-03-23T03:50:41.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.406+0000 7fb8b4c00640 1 -- 172.21.15.102:0/1696920993 wait complete. 2024-03-23T03:50:41.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.407+0000 7fb8b4c00640 1 Processor -- start 2024-03-23T03:50:41.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.408+0000 7fb8b4c00640 1 -- start start 2024-03-23T03:50:41.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.409+0000 7fb8b4c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb894006130 0x7fb8b01f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:41.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.409+0000 7fb8b4c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b01593d0 0x7fb8b015be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:41.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.409+0000 7fb8b4c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8b015c370 0x7fb8b013e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:41.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.409+0000 7fb8b4c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b01f3280 con 0x7fb8b01593d0 2024-03-23T03:50:41.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.409+0000 7fb8b4c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b015c9d0 con 0x7fb8b015c370 2024-03-23T03:50:41.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.409+0000 7fb8b4c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b015ccb0 con 0x7fb894006130 2024-03-23T03:50:41.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.409+0000 7fb8a7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b01593d0 0x7fb8b015be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:41.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.409+0000 7fb8a7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b01593d0 0x7fb8b015be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50856/0 (socket says 172.21.15.102:50856) 2024-03-23T03:50:41.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.409+0000 7fb8a7e00640 1 -- 172.21.15.102:0/261238797 learned_addr learned my addr 172.21.15.102:0/261238797 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:41.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.409+0000 7fb8ade00640 1 --2- 172.21.15.102:0/261238797 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb894006130 0x7fb8b01f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:41.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.409+0000 7fb8ae800640 1 --2- 172.21.15.102:0/261238797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8b015c370 0x7fb8b013e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:41.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.409+0000 7fb8a7e00640 1 -- 172.21.15.102:0/261238797 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb894006130 msgr2=0x7fb8b01f2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:41.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.409+0000 7fb8a7e00640 1 --2- 172.21.15.102:0/261238797 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb894006130 0x7fb8b01f2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.409+0000 7fb8a7e00640 1 -- 172.21.15.102:0/261238797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8b015c370 msgr2=0x7fb8b013e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:41.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.409+0000 7fb8a7e00640 1 --2- 172.21.15.102:0/261238797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8b015c370 0x7fb8b013e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.409+0000 7fb8a7e00640 1 -- 172.21.15.102:0/261238797 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b01f2540 con 0x7fb8b01593d0 2024-03-23T03:50:41.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.410+0000 7fb8a7e00640 1 --2- 172.21.15.102:0/261238797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b01593d0 0x7fb8b015be30 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fb89802c430 tx=0x7fb898001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:41.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.410+0000 7fb8a6000640 1 -- 172.21.15.102:0/261238797 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb898039070 con 0x7fb8b01593d0 2024-03-23T03:50:41.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.410+0000 7fb8a6000640 1 -- 172.21.15.102:0/261238797 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb89802ccc0 con 0x7fb8b01593d0 2024-03-23T03:50:41.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.410+0000 7fb8b4c00640 1 -- 172.21.15.102:0/261238797 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b01f2820 con 0x7fb8b01593d0 2024-03-23T03:50:41.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.410+0000 7fb8a6000640 1 -- 172.21.15.102:0/261238797 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb89803c040 con 0x7fb8b01593d0 2024-03-23T03:50:41.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.410+0000 7fb8b4c00640 1 -- 172.21.15.102:0/261238797 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b013f680 con 0x7fb8b01593d0 2024-03-23T03:50:41.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.411+0000 7fb8b4c00640 1 -- 172.21.15.102:0/261238797 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b015c780 con 0x7fb8b01593d0 2024-03-23T03:50:41.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.412+0000 7fb8a6000640 1 -- 172.21.15.102:0/261238797 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb898032070 con 0x7fb8b01593d0 2024-03-23T03:50:41.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.413+0000 7fb8a6000640 1 --2- 172.21.15.102:0/261238797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb88c06ac00 0x7fb88c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:41.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.413+0000 7fb8a6000640 1 -- 172.21.15.102:0/261238797 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fb8980aa3b0 con 0x7fb8b01593d0 2024-03-23T03:50:41.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.413+0000 7fb8ade00640 1 --2- 172.21.15.102:0/261238797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb88c06ac00 0x7fb88c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:41.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.414+0000 7fb8ade00640 1 --2- 172.21.15.102:0/261238797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb88c06ac00 0x7fb88c06d0c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0007fd0 tx=0x7fb8a0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:41.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.418+0000 7fb8a6000640 1 -- 172.21.15.102:0/261238797 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb8980757e0 con 0x7fb8b01593d0 2024-03-23T03:50:41.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.690+0000 7fb8b4c00640 1 -- 172.21.15.102:0/261238797 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_df"]} v 0) v1 -- 0x7fb8b01f29b0 con 0x7fb8b01593d0 2024-03-23T03:50:41.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.716+0000 7fb8a6000640 1 -- 172.21.15.102:0/261238797 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_df"]}]=0 v227) v1 ==== 134+0+0 (secure 0 0 0) 0x7fb898078e00 con 0x7fb8b01593d0 2024-03-23T03:50:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.718+0000 7fb8b4c00640 1 -- 172.21.15.102:0/261238797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb88c06ac00 msgr2=0x7fb88c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.718+0000 7fb8b4c00640 1 --2- 172.21.15.102:0/261238797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb88c06ac00 0x7fb88c06d0c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0007fd0 tx=0x7fb8a0009040 comp rx=0 tx=0).stop 2024-03-23T03:50:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.718+0000 7fb8b4c00640 1 -- 172.21.15.102:0/261238797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b01593d0 msgr2=0x7fb8b015be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.718+0000 7fb8b4c00640 1 --2- 172.21.15.102:0/261238797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b01593d0 0x7fb8b015be30 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fb89802c430 tx=0x7fb898001a00 comp rx=0 tx=0).stop 2024-03-23T03:50:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.718+0000 7fb8b4c00640 1 -- 172.21.15.102:0/261238797 shutdown_connections 2024-03-23T03:50:41.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.718+0000 7fb8b4c00640 1 --2- 172.21.15.102:0/261238797 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb88c06ac00 0x7fb88c06d0c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.718+0000 7fb8b4c00640 1 --2- 172.21.15.102:0/261238797 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8b015c370 0x7fb8b013e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.718+0000 7fb8b4c00640 1 --2- 172.21.15.102:0/261238797 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8b01593d0 0x7fb8b015be30 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.718+0000 7fb8b4c00640 1 --2- 172.21.15.102:0/261238797 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb894006130 0x7fb8b01f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.718+0000 7fb8b4c00640 1 -- 172.21.15.102:0/261238797 >> 172.21.15.102:0/261238797 conn(0x7fb8b006a3d0 msgr2=0x7fb8b00b1ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:41.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.718+0000 7fb8b4c00640 1 -- 172.21.15.102:0/261238797 shutdown_connections 2024-03-23T03:50:41.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.719+0000 7fb8b4c00640 1 -- 172.21.15.102:0/261238797 wait complete. 2024-03-23T03:50:41.738 INFO:tasks.cephfs_test_runner:That when a client modifies the quota on a directory used ... ok 2024-03-23T03:50:41.738 INFO:tasks.cephfs_test_runner:Starting test: test_remote_update_getfattr (tasks.cephfs.test_quota.TestQuota) That quota changes made from one client are visible to another 2024-03-23T03:50:41.738 INFO:tasks.cephfs_test_runner:test_remote_update_getfattr (tasks.cephfs.test_quota.TestQuota) 2024-03-23T03:50:41.739 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr' 2024-03-23T03:50:41.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.937+0000 7fbcc0e00640 1 Processor -- start 2024-03-23T03:50:41.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.937+0000 7fbcc0e00640 1 -- start start 2024-03-23T03:50:41.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.939+0000 7fbcc0e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbcbc0adbc0 0x7fbcbc0b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:41.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.939+0000 7fbcc0e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcbc0cbab0 0x7fbcbc0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:41.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.939+0000 7fbcc0e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcbc0b16a0 con 0x7fbcbc06d390 2024-03-23T03:50:41.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.939+0000 7fbcc0e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcbc0b4700 con 0x7fbcbc0adbc0 2024-03-23T03:50:41.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.939+0000 7fbcc0e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc0cbfb0 con 0x7fbcbc0cbab0 2024-03-23T03:50:41.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.939+0000 7fbcb9e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbcbc0adbc0 0x7fbcbc0b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:41.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.939+0000 7fbcba800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbcbc06d390 0x7fbcbc0b2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:32798/0 (socket says 172.21.15.102:32798) 2024-03-23T03:50:41.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.939+0000 7fbcba800640 1 -- 172.21.15.102:0/1138867683 learned_addr learned my addr 172.21.15.102:0/1138867683 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:41.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.940+0000 7fbcb1400640 1 --2- 172.21.15.102:0/1138867683 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcbc0cbab0 0x7fbcbc0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:41.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.940+0000 7fbcb9e00640 1 -- 172.21.15.102:0/1138867683 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcbc0cbab0 msgr2=0x7fbcbc0b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:41.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.940+0000 7fbcb9e00640 1 --2- 172.21.15.102:0/1138867683 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcbc0cbab0 0x7fbcbc0b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.940+0000 7fbcb9e00640 1 -- 172.21.15.102:0/1138867683 >> v1:172.21.15.102:6789/0 conn(0x7fbcbc06d390 legacy=0x7fbcbc0b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:41.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.940+0000 7fbcb9e00640 1 -- 172.21.15.102:0/1138867683 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcbc0cc290 con 0x7fbcbc0adbc0 2024-03-23T03:50:41.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.940+0000 7fbcb9400640 1 -- 172.21.15.102:0/1138867683 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865809191 0 0) 0x7fbcbc0b16a0 con 0x7fbcbc06d390 2024-03-23T03:50:41.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.941+0000 7fbcb9e00640 1 --2- 172.21.15.102:0/1138867683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcbc0adbc0 0x7fbcbc0b41c0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fbcac0097a0 tx=0x7fbcac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b4dba42aaca8b37 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:41.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.941+0000 7fbcb9400640 1 -- 172.21.15.102:0/1138867683 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac02e070 con 0x7fbcbc0adbc0 2024-03-23T03:50:41.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.941+0000 7fbcb9400640 1 -- 172.21.15.102:0/1138867683 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcac0315a0 con 0x7fbcbc0adbc0 2024-03-23T03:50:41.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.941+0000 7fbcb9400640 1 -- 172.21.15.102:0/1138867683 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac031930 con 0x7fbcbc0adbc0 2024-03-23T03:50:41.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.942+0000 7fbcc0e00640 1 -- 172.21.15.102:0/1138867683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcbc0adbc0 msgr2=0x7fbcbc0b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:41.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.942+0000 7fbcc0e00640 1 --2- 172.21.15.102:0/1138867683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcbc0adbc0 0x7fbcbc0b41c0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fbcac0097a0 tx=0x7fbcac02b840 comp rx=0 tx=0).stop 2024-03-23T03:50:41.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.942+0000 7fbcc0e00640 1 -- 172.21.15.102:0/1138867683 shutdown_connections 2024-03-23T03:50:41.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.942+0000 7fbcc0e00640 1 --2- 172.21.15.102:0/1138867683 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcbc0cbab0 0x7fbcbc0b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.942+0000 7fbcc0e00640 1 --2- 172.21.15.102:0/1138867683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcbc0adbc0 0x7fbcbc0b41c0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.942+0000 7fbcc0e00640 1 -- 172.21.15.102:0/1138867683 >> 172.21.15.102:0/1138867683 conn(0x7fbcbc06a3d0 msgr2=0x7fbcbc0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:41.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.942+0000 7fbcc0e00640 1 -- 172.21.15.102:0/1138867683 shutdown_connections 2024-03-23T03:50:41.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.942+0000 7fbcc0e00640 1 -- 172.21.15.102:0/1138867683 wait complete. 2024-03-23T03:50:41.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.944+0000 7fbcc0e00640 1 Processor -- start 2024-03-23T03:50:41.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.944+0000 7fbcc0e00640 1 -- start start 2024-03-23T03:50:41.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.945+0000 7fbcc0e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcbc06d390 0x7fbcbc15bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:41.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.945+0000 7fbcc0e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcbc0adbc0 0x7fbcbc15d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:41.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.945+0000 7fbcc0e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcbc0cbab0 0x7fbcbc15d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:41.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.945+0000 7fbcc0e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc15abc0 con 0x7fbcbc06d390 2024-03-23T03:50:41.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.945+0000 7fbcc0e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc15cea0 con 0x7fbcbc0cbab0 2024-03-23T03:50:41.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.945+0000 7fbcc0e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcbc15c8e0 con 0x7fbcbc0adbc0 2024-03-23T03:50:41.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.945+0000 7fbcb9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcbc06d390 0x7fbcbc15bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:41.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.945+0000 7fbcb9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcbc06d390 0x7fbcbc15bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:50880/0 (socket says 172.21.15.102:50880) 2024-03-23T03:50:41.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.945+0000 7fbcb9e00640 1 -- 172.21.15.102:0/3546797673 learned_addr learned my addr 172.21.15.102:0/3546797673 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:41.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.946+0000 7fbcb9e00640 1 -- 172.21.15.102:0/3546797673 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcbc0adbc0 msgr2=0x7fbcbc15d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:41.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.946+0000 7fbcb9e00640 1 --2- 172.21.15.102:0/3546797673 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcbc0adbc0 0x7fbcbc15d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.946+0000 7fbcb9e00640 1 -- 172.21.15.102:0/3546797673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcbc0cbab0 msgr2=0x7fbcbc15d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:41.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.946+0000 7fbcb9e00640 1 --2- 172.21.15.102:0/3546797673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcbc0cbab0 0x7fbcbc15d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:41.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.946+0000 7fbcb9e00640 1 -- 172.21.15.102:0/3546797673 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcac009400 con 0x7fbcbc06d390 2024-03-23T03:50:41.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.946+0000 7fbcb9e00640 1 --2- 172.21.15.102:0/3546797673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcbc06d390 0x7fbcbc15bf90 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fbcac034040 tx=0x7fbcac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.946+0000 7fbcb2a00640 1 -- 172.21.15.102:0/3546797673 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac032840 con 0x7fbcbc06d390 2024-03-23T03:50:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.946+0000 7fbcb2a00640 1 -- 172.21.15.102:0/3546797673 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcac0329a0 con 0x7fbcbc06d390 2024-03-23T03:50:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.946+0000 7fbcb2a00640 1 -- 172.21.15.102:0/3546797673 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcac032d50 con 0x7fbcbc06d390 2024-03-23T03:50:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.946+0000 7fbcc0e00640 1 -- 172.21.15.102:0/3546797673 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcbc15cbc0 con 0x7fbcbc06d390 2024-03-23T03:50:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.946+0000 7fbcc0e00640 1 -- 172.21.15.102:0/3546797673 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcbc13f230 con 0x7fbcbc06d390 2024-03-23T03:50:41.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.947+0000 7fbcc0e00640 1 -- 172.21.15.102:0/3546797673 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcbc13edb0 con 0x7fbcbc06d390 2024-03-23T03:50:41.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.948+0000 7fbcb2a00640 1 -- 172.21.15.102:0/3546797673 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbcac04e430 con 0x7fbcbc06d390 2024-03-23T03:50:41.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.949+0000 7fbcb2a00640 1 --2- 172.21.15.102:0/3546797673 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbc9806ac30 0x7fbc9806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:41.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.950+0000 7fbcb2a00640 1 -- 172.21.15.102:0/3546797673 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fbcac0ab1c0 con 0x7fbcbc06d390 2024-03-23T03:50:41.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.950+0000 7fbcb1400640 1 --2- 172.21.15.102:0/3546797673 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbc9806ac30 0x7fbc9806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:41.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.950+0000 7fbcb1400640 1 --2- 172.21.15.102:0/3546797673 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbc9806ac30 0x7fbc9806d0f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc13dbc0 tx=0x7fbca4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:41.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:41.954+0000 7fbcb2a00640 1 -- 172.21.15.102:0/3546797673 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbcbc13edb0 con 0x7fbcbc06d390 2024-03-23T03:50:42.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.222+0000 7fbcc0e00640 1 -- 172.21.15.102:0/3546797673 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr"]} v 0) v1 -- 0x7fbcbc13edb0 con 0x7fbcbc06d390 2024-03-23T03:50:42.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.730+0000 7fbcb2a00640 1 -- 172.21.15.102:0/3546797673 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr"]}]=0 v228) v1 ==== 143+0+0 (secure 0 0 0) 0x7fbcbc13edb0 con 0x7fbcbc06d390 2024-03-23T03:50:42.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.733+0000 7fbcc0e00640 1 -- 172.21.15.102:0/3546797673 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbc9806ac30 msgr2=0x7fbc9806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:42.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.733+0000 7fbcc0e00640 1 --2- 172.21.15.102:0/3546797673 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbc9806ac30 0x7fbc9806d0f0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc13dbc0 tx=0x7fbca4008040 comp rx=0 tx=0).stop 2024-03-23T03:50:42.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.733+0000 7fbcc0e00640 1 -- 172.21.15.102:0/3546797673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcbc06d390 msgr2=0x7fbcbc15bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:42.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.733+0000 7fbcc0e00640 1 --2- 172.21.15.102:0/3546797673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcbc06d390 0x7fbcbc15bf90 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fbcac034040 tx=0x7fbcac0046c0 comp rx=0 tx=0).stop 2024-03-23T03:50:42.734 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.733+0000 7fbcc0e00640 1 -- 172.21.15.102:0/3546797673 shutdown_connections 2024-03-23T03:50:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.733+0000 7fbcc0e00640 1 --2- 172.21.15.102:0/3546797673 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbc9806ac30 0x7fbc9806d0f0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.733+0000 7fbcc0e00640 1 --2- 172.21.15.102:0/3546797673 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcbc0cbab0 0x7fbcbc15d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.733+0000 7fbcc0e00640 1 --2- 172.21.15.102:0/3546797673 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcbc0adbc0 0x7fbcbc15d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.733+0000 7fbcc0e00640 1 --2- 172.21.15.102:0/3546797673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcbc06d390 0x7fbcbc15bf90 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.733+0000 7fbcc0e00640 1 -- 172.21.15.102:0/3546797673 >> 172.21.15.102:0/3546797673 conn(0x7fbcbc06a3d0 msgr2=0x7fbcbc13af00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.733+0000 7fbcc0e00640 1 -- 172.21.15.102:0/3546797673 shutdown_connections 2024-03-23T03:50:42.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.734+0000 7fbcc0e00640 1 -- 172.21.15.102:0/3546797673 wait complete. 2024-03-23T03:50:42.753 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-23T03:50:42.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.960+0000 7f1789a00640 1 Processor -- start 2024-03-23T03:50:42.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.960+0000 7f1789a00640 1 -- start start 2024-03-23T03:50:42.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.961+0000 7f1789a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f17841581a0 0x7f1784070030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:42.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.961+0000 7f1789a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f17840cc990 0x7f178406ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:42.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.961+0000 7f1789a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1784070570 con 0x7f178406ca10 2024-03-23T03:50:42.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.961+0000 7f1789a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17840707f0 con 0x7f17840cc990 2024-03-23T03:50:42.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.961+0000 7f1789a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1784070ad0 con 0x7f17841581a0 2024-03-23T03:50:42.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.962+0000 7f177b400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f178406ca10 0x7f17840cc280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56750/0 (socket says 172.21.15.102:56750) 2024-03-23T03:50:42.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.962+0000 7f177b400640 1 -- 172.21.15.102:0/1349913456 learned_addr learned my addr 172.21.15.102:0/1349913456 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:42.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.962+0000 7f1789000640 1 --2- 172.21.15.102:0/1349913456 >> v2:172.21.15.102:3300/0 conn(0x7f17840cc990 0x7f178406ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:42.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.962+0000 7f1783e00640 1 --2- 172.21.15.102:0/1349913456 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f17841581a0 0x7f1784070030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:42.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.962+0000 7f1783400640 1 -- 172.21.15.102:0/1349913456 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416969997 0 0) 0x7f1784070570 con 0x7f178406ca10 2024-03-23T03:50:42.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.962+0000 7f1783400640 1 -- 172.21.15.102:0/1349913456 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1768003520 con 0x7f178406ca10 2024-03-23T03:50:42.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.962+0000 7f1789000640 1 -- 172.21.15.102:0/1349913456 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f17841581a0 msgr2=0x7f1784070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:42.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.962+0000 7f1789000640 1 --2- 172.21.15.102:0/1349913456 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f17841581a0 0x7f1784070030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:42.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.962+0000 7f1789000640 1 -- 172.21.15.102:0/1349913456 >> v1:172.21.15.102:6789/0 conn(0x7f178406ca10 legacy=0x7f17840cc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:42.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.962+0000 7f1789000640 1 -- 172.21.15.102:0/1349913456 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178406c210 con 0x7f17840cc990 2024-03-23T03:50:42.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.962+0000 7f1783e00640 1 --2- 172.21.15.102:0/1349913456 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f17841581a0 0x7f1784070030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:42.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.963+0000 7f1789000640 1 --2- 172.21.15.102:0/1349913456 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17840cc990 0x7f178406ce70 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f177c009d90 tx=0x7f177c02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e980bfa861703b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:42.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.964+0000 7f1783400640 1 -- 172.21.15.102:0/1349913456 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c02cce0 con 0x7f17840cc990 2024-03-23T03:50:42.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.964+0000 7f1783400640 1 -- 172.21.15.102:0/1349913456 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177c02ce40 con 0x7f17840cc990 2024-03-23T03:50:42.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.964+0000 7f1783400640 1 -- 172.21.15.102:0/1349913456 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c034660 con 0x7f17840cc990 2024-03-23T03:50:42.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.964+0000 7f1789a00640 1 -- 172.21.15.102:0/1349913456 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17840cc990 msgr2=0x7f178406ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:42.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.964+0000 7f1789a00640 1 --2- 172.21.15.102:0/1349913456 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17840cc990 0x7f178406ce70 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f177c009d90 tx=0x7f177c02bf30 comp rx=0 tx=0).stop 2024-03-23T03:50:42.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.964+0000 7f1789a00640 1 -- 172.21.15.102:0/1349913456 shutdown_connections 2024-03-23T03:50:42.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.964+0000 7f1789a00640 1 --2- 172.21.15.102:0/1349913456 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17840cc990 0x7f178406ce70 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:42.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.964+0000 7f1789a00640 1 --2- 172.21.15.102:0/1349913456 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f17841581a0 0x7f1784070030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:42.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.964+0000 7f1789a00640 1 -- 172.21.15.102:0/1349913456 >> 172.21.15.102:0/1349913456 conn(0x7f1784066000 msgr2=0x7f1784069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:42.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.965+0000 7f1789a00640 1 -- 172.21.15.102:0/1349913456 shutdown_connections 2024-03-23T03:50:42.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.965+0000 7f1789a00640 1 -- 172.21.15.102:0/1349913456 wait complete. 2024-03-23T03:50:42.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.966+0000 7f1789a00640 1 Processor -- start 2024-03-23T03:50:42.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.966+0000 7f1789a00640 1 -- start start 2024-03-23T03:50:42.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.967+0000 7f1789a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f178406ca10 0x7f178413b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:42.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.967+0000 7f1789a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17840cc990 0x7f17841ef6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:42.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.967+0000 7f1789a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17841581a0 0x7f1784142570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:42.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.967+0000 7f1789a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178413bdd0 con 0x7f17841581a0 2024-03-23T03:50:42.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.967+0000 7f1789a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17841f4180 con 0x7f17840cc990 2024-03-23T03:50:42.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.967+0000 7f1789a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178413adb0 con 0x7f178406ca10 2024-03-23T03:50:42.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.967+0000 7f1789000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17841581a0 0x7f1784142570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:42.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.967+0000 7f1789000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17841581a0 0x7f1784142570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44650/0 (socket says 172.21.15.102:44650) 2024-03-23T03:50:42.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.967+0000 7f1789000640 1 -- 172.21.15.102:0/596605961 learned_addr learned my addr 172.21.15.102:0/596605961 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.968+0000 7f1783e00640 1 --2- 172.21.15.102:0/596605961 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f178406ca10 0x7f178413b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.968+0000 7f1789000640 1 -- 172.21.15.102:0/596605961 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f178406ca10 msgr2=0x7f178413b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.968+0000 7f1789000640 1 --2- 172.21.15.102:0/596605961 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f178406ca10 0x7f178413b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.968+0000 7f1789000640 1 -- 172.21.15.102:0/596605961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17840cc990 msgr2=0x7f17841ef6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.968+0000 7f1789000640 1 --2- 172.21.15.102:0/596605961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17840cc990 0x7f17841ef6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.968+0000 7f1789000640 1 -- 172.21.15.102:0/596605961 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f177c009a70 con 0x7f17841581a0 2024-03-23T03:50:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.968+0000 7f1789000640 1 --2- 172.21.15.102:0/596605961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17841581a0 0x7f1784142570 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f177c009d90 tx=0x7f177c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.968+0000 7f1780c00640 1 -- 172.21.15.102:0/596605961 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c03d600 con 0x7f17841581a0 2024-03-23T03:50:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.969+0000 7f1780c00640 1 -- 172.21.15.102:0/596605961 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177c03d780 con 0x7f17841581a0 2024-03-23T03:50:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.969+0000 7f1789a00640 1 -- 172.21.15.102:0/596605961 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178413b090 con 0x7f17841581a0 2024-03-23T03:50:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.969+0000 7f1780c00640 1 -- 172.21.15.102:0/596605961 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c03db30 con 0x7f17841581a0 2024-03-23T03:50:42.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.969+0000 7f1789a00640 1 -- 172.21.15.102:0/596605961 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17841efc30 con 0x7f17841581a0 2024-03-23T03:50:42.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.970+0000 7f1789a00640 1 -- 172.21.15.102:0/596605961 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f178413b5b0 con 0x7f17841581a0 2024-03-23T03:50:42.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.971+0000 7f1780c00640 1 -- 172.21.15.102:0/596605961 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f177c03b040 con 0x7f17841581a0 2024-03-23T03:50:42.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.972+0000 7f1780c00640 1 --2- 172.21.15.102:0/596605961 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f176006ac00 0x7f176006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:42.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.972+0000 7f1780c00640 1 -- 172.21.15.102:0/596605961 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f177c0ab6e0 con 0x7f17841581a0 2024-03-23T03:50:42.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.972+0000 7f1783e00640 1 --2- 172.21.15.102:0/596605961 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f176006ac00 0x7f176006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:42.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.973+0000 7f1783e00640 1 --2- 172.21.15.102:0/596605961 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f176006ac00 0x7f176006d0c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f177400b840 tx=0x7f1774008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:42.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:42.976+0000 7f1780c00640 1 -- 172.21.15.102:0/596605961 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f178413b5b0 con 0x7f17841581a0 2024-03-23T03:50:43.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.180+0000 7f1789a00640 1 -- 172.21.15.102:0/596605961 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f178413b5b0 con 0x7f17841581a0 2024-03-23T03:50:43.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.186+0000 7f1780c00640 1 -- 172.21.15.102:0/596605961 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7f178413b5b0 con 0x7f17841581a0 2024-03-23T03:50:43.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.189+0000 7f1789a00640 1 -- 172.21.15.102:0/596605961 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f176006ac00 msgr2=0x7f176006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:43.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.189+0000 7f1789a00640 1 --2- 172.21.15.102:0/596605961 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f176006ac00 0x7f176006d0c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f177400b840 tx=0x7f1774008040 comp rx=0 tx=0).stop 2024-03-23T03:50:43.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.189+0000 7f1789a00640 1 -- 172.21.15.102:0/596605961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17841581a0 msgr2=0x7f1784142570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:43.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.189+0000 7f1789a00640 1 --2- 172.21.15.102:0/596605961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17841581a0 0x7f1784142570 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f177c009d90 tx=0x7f177c004a40 comp rx=0 tx=0).stop 2024-03-23T03:50:43.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.189+0000 7f1789a00640 1 -- 172.21.15.102:0/596605961 shutdown_connections 2024-03-23T03:50:43.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.189+0000 7f1789a00640 1 --2- 172.21.15.102:0/596605961 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f176006ac00 0x7f176006d0c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:43.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.189+0000 7f1789a00640 1 --2- 172.21.15.102:0/596605961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17841581a0 0x7f1784142570 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:43.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.189+0000 7f1789a00640 1 --2- 172.21.15.102:0/596605961 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f17840cc990 0x7f17841ef6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:43.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.189+0000 7f1789a00640 1 --2- 172.21.15.102:0/596605961 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f178406ca10 0x7f178413b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:43.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.189+0000 7f1789a00640 1 -- 172.21.15.102:0/596605961 >> 172.21.15.102:0/596605961 conn(0x7f1784066000 msgr2=0x7f17840cc280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:43.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.189+0000 7f1789a00640 1 -- 172.21.15.102:0/596605961 shutdown_connections 2024-03-23T03:50:43.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.189+0000 7f1789a00640 1 -- 172.21.15.102:0/596605961 wait complete. 2024-03-23T03:50:43.208 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T03:50:43.208 DEBUG:teuthology.orchestra.run.smithi102:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:50:43.309 DEBUG:teuthology.orchestra.run.smithi173:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:50:43.434 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T03:50:43.434 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:50:43.434 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:43.493 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:50:43.493 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:50:43.493 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:43.546 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:50:43.547 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:50:43.547 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:43.603 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:50:43.603 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:50:43.603 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:43.661 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:50:43.661 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-23T03:50:43.662 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:50:43.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.879+0000 7ff62da00640 1 Processor -- start 2024-03-23T03:50:43.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.879+0000 7ff62da00640 1 -- start start 2024-03-23T03:50:43.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.880+0000 7ff62da00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff6280cbab0 0x7ff6280b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:43.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.881+0000 7ff62da00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6280b16a0 con 0x7ff62806d390 2024-03-23T03:50:43.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.881+0000 7ff62da00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6280cbfb0 con 0x7ff6280adbc0 2024-03-23T03:50:43.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.881+0000 7ff62da00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6280cc290 con 0x7ff6280cbab0 2024-03-23T03:50:43.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.881+0000 7ff627400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff62806d390 0x7ff6280b2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56756/0 (socket says 172.21.15.102:56756) 2024-03-23T03:50:43.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.881+0000 7ff627400640 1 -- 172.21.15.102:0/512959763 learned_addr learned my addr 172.21.15.102:0/512959763 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:43.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.881+0000 7ff626000640 1 --2- 172.21.15.102:0/512959763 >> v2:172.21.15.102:3300/0 conn(0x7ff6280cbab0 0x7ff6280b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:43.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.881+0000 7ff626000640 1 -- 172.21.15.102:0/512959763 >> v1:172.21.15.173:6789/0 conn(0x7ff6280adbc0 legacy=0x7ff6280b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:43.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.881+0000 7ff626000640 1 -- 172.21.15.102:0/512959763 >> v1:172.21.15.102:6789/0 conn(0x7ff62806d390 legacy=0x7ff6280b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:43.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.882+0000 7ff626000640 1 -- 172.21.15.102:0/512959763 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6280b18f0 con 0x7ff6280cbab0 2024-03-23T03:50:43.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.882+0000 7ff625600640 1 -- 172.21.15.102:0/512959763 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119178790 0 0) 0x7ff6280b16a0 con 0x7ff62806d390 2024-03-23T03:50:43.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.883+0000 7ff626000640 1 --2- 172.21.15.102:0/512959763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6280cbab0 0x7ff6280b20f0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7ff6140097a0 tx=0x7ff61402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7806bc7da468b9c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:43.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.883+0000 7ff625600640 1 -- 172.21.15.102:0/512959763 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61402e070 con 0x7ff6280cbab0 2024-03-23T03:50:43.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.883+0000 7ff625600640 1 -- 172.21.15.102:0/512959763 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6140315a0 con 0x7ff6280cbab0 2024-03-23T03:50:43.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.883+0000 7ff625600640 1 -- 172.21.15.102:0/512959763 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff614031930 con 0x7ff6280cbab0 2024-03-23T03:50:43.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.883+0000 7ff62da00640 1 -- 172.21.15.102:0/512959763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6280cbab0 msgr2=0x7ff6280b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:43.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.883+0000 7ff62da00640 1 --2- 172.21.15.102:0/512959763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6280cbab0 0x7ff6280b20f0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7ff6140097a0 tx=0x7ff61402b840 comp rx=0 tx=0).stop 2024-03-23T03:50:43.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.884+0000 7ff62da00640 1 -- 172.21.15.102:0/512959763 shutdown_connections 2024-03-23T03:50:43.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.884+0000 7ff62da00640 1 --2- 172.21.15.102:0/512959763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6280cbab0 0x7ff6280b20f0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:43.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.884+0000 7ff62da00640 1 -- 172.21.15.102:0/512959763 >> 172.21.15.102:0/512959763 conn(0x7ff62806a3d0 msgr2=0x7ff6280b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:43.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.884+0000 7ff62da00640 1 -- 172.21.15.102:0/512959763 shutdown_connections 2024-03-23T03:50:43.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.884+0000 7ff62da00640 1 -- 172.21.15.102:0/512959763 wait complete. 2024-03-23T03:50:43.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.885+0000 7ff62da00640 1 Processor -- start 2024-03-23T03:50:43.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.885+0000 7ff62da00640 1 -- start start 2024-03-23T03:50:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.886+0000 7ff62da00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff62806d390 0x7ff6280c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.886+0000 7ff62da00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff6280adbc0 0x7ff6280c8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.886+0000 7ff62da00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6280cbab0 0x7ff628140d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.886+0000 7ff62da00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6280c3d80 con 0x7ff62806d390 2024-03-23T03:50:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.886+0000 7ff62da00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6280c8bf0 con 0x7ff6280cbab0 2024-03-23T03:50:43.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.886+0000 7ff62da00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6280c8ed0 con 0x7ff6280adbc0 2024-03-23T03:50:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.887+0000 7ff626a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff62806d390 0x7ff6280c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.887+0000 7ff626a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff62806d390 0x7ff6280c9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44668/0 (socket says 172.21.15.102:44668) 2024-03-23T03:50:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.887+0000 7ff626a00640 1 -- 172.21.15.102:0/513458109 learned_addr learned my addr 172.21.15.102:0/513458109 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.887+0000 7ff626a00640 1 -- 172.21.15.102:0/513458109 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff6280adbc0 msgr2=0x7ff6280c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.887+0000 7ff626000640 1 --2- 172.21.15.102:0/513458109 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff6280adbc0 0x7ff6280c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.887+0000 7ff627400640 1 --2- 172.21.15.102:0/513458109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6280cbab0 0x7ff628140d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.887+0000 7ff626a00640 1 --2- 172.21.15.102:0/513458109 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff6280adbc0 0x7ff6280c8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.887+0000 7ff626a00640 1 -- 172.21.15.102:0/513458109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6280cbab0 msgr2=0x7ff628140d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.887+0000 7ff626a00640 1 --2- 172.21.15.102:0/513458109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6280cbab0 0x7ff628140d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.887+0000 7ff626a00640 1 -- 172.21.15.102:0/513458109 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff614009400 con 0x7ff62806d390 2024-03-23T03:50:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.887+0000 7ff626a00640 1 --2- 172.21.15.102:0/513458109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff62806d390 0x7ff6280c9ae0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7ff61800c430 tx=0x7ff61800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.888+0000 7ff61ea00640 1 -- 172.21.15.102:0/513458109 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff618011790 con 0x7ff62806d390 2024-03-23T03:50:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.888+0000 7ff61ea00640 1 -- 172.21.15.102:0/513458109 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6180118f0 con 0x7ff62806d390 2024-03-23T03:50:43.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.888+0000 7ff61ea00640 1 -- 172.21.15.102:0/513458109 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff618011ce0 con 0x7ff62806d390 2024-03-23T03:50:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.888+0000 7ff62da00640 1 -- 172.21.15.102:0/513458109 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6280c91b0 con 0x7ff62806d390 2024-03-23T03:50:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.888+0000 7ff62da00640 1 -- 172.21.15.102:0/513458109 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6280ae2a0 con 0x7ff62806d390 2024-03-23T03:50:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.889+0000 7ff62da00640 1 -- 172.21.15.102:0/513458109 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6280c96d0 con 0x7ff62806d390 2024-03-23T03:50:43.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.890+0000 7ff61ea00640 1 -- 172.21.15.102:0/513458109 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff61801c460 con 0x7ff62806d390 2024-03-23T03:50:43.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.890+0000 7ff61ea00640 1 --2- 172.21.15.102:0/513458109 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff60806ac00 0x7ff60806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:43.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.891+0000 7ff61ea00640 1 -- 172.21.15.102:0/513458109 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5263+0+0 (secure 0 0 0) 0x7ff618089100 con 0x7ff62806d390 2024-03-23T03:50:43.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.891+0000 7ff626000640 1 --2- 172.21.15.102:0/513458109 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff60806ac00 0x7ff60806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:43.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.891+0000 7ff626000640 1 --2- 172.21.15.102:0/513458109 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff60806ac00 0x7ff60806d0c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff614036000 tx=0x7ff61403d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:43.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:43.895+0000 7ff61ea00640 1 -- 172.21.15.102:0/513458109 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff6280c96d0 con 0x7ff62806d390 2024-03-23T03:50:44.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.128+0000 7ff62da00640 1 -- 172.21.15.102:0/513458109 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6280c3ec0 con 0x7ff62806d390 2024-03-23T03:50:44.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.130+0000 7ff61ea00640 1 -- 172.21.15.102:0/513458109 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+3538 (secure 0 0 0) 0x7ff6280c3ec0 con 0x7ff62806d390 2024-03-23T03:50:44.130 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:44.131 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":43,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5051,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/2079098809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":2079098809},{"type":"v1","addr":"172.21.15.173:6833","nonce":2079098809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":5413,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/459102175","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":459102175},{"type":"v1","addr":"172.21.15.173:6835","nonce":459102175}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/2962195583","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":2962195583},{"type":"v1","addr":"172.21.15.102:6835","nonce":2962195583}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36},{"gid":6756,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3693753809","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3693753809},{"type":"v1","addr":"172.21.15.102:6837","nonce":3693753809}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":43}],"filesystems":[]} 2024-03-23T03:50:44.131 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 43 2024-03-23T03:50:44.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.132+0000 7ff62da00640 1 -- 172.21.15.102:0/513458109 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff60806ac00 msgr2=0x7ff60806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:44.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.132+0000 7ff62da00640 1 --2- 172.21.15.102:0/513458109 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff60806ac00 0x7ff60806d0c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff614036000 tx=0x7ff61403d040 comp rx=0 tx=0).stop 2024-03-23T03:50:44.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.132+0000 7ff62da00640 1 -- 172.21.15.102:0/513458109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff62806d390 msgr2=0x7ff6280c9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:44.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.132+0000 7ff62da00640 1 --2- 172.21.15.102:0/513458109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff62806d390 0x7ff6280c9ae0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7ff61800c430 tx=0x7ff61800c8f0 comp rx=0 tx=0).stop 2024-03-23T03:50:44.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.132+0000 7ff62da00640 1 -- 172.21.15.102:0/513458109 shutdown_connections 2024-03-23T03:50:44.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.132+0000 7ff62da00640 1 --2- 172.21.15.102:0/513458109 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff60806ac00 0x7ff60806d0c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:44.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.132+0000 7ff62da00640 1 --2- 172.21.15.102:0/513458109 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff6280cbab0 0x7ff628140d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:44.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.132+0000 7ff62da00640 1 --2- 172.21.15.102:0/513458109 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff6280adbc0 0x7ff6280c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:44.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.132+0000 7ff62da00640 1 --2- 172.21.15.102:0/513458109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff62806d390 0x7ff6280c9ae0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:44.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.132+0000 7ff62da00640 1 -- 172.21.15.102:0/513458109 >> 172.21.15.102:0/513458109 conn(0x7ff62806a3d0 msgr2=0x7ff6280b3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:44.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.132+0000 7ff62da00640 1 -- 172.21.15.102:0/513458109 shutdown_connections 2024-03-23T03:50:44.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.132+0000 7ff62da00640 1 -- 172.21.15.102:0/513458109 wait complete. 2024-03-23T03:50:44.152 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T03:50:44.153 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-23T03:50:44.153 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T03:50:44.153 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:50:44.153 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T03:50:44.153 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-23T03:50:44.153 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T03:50:44.153 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:50:44.154 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T03:50:44.154 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-23T03:50:44.154 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T03:50:44.154 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:50:44.154 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T03:50:44.154 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-23T03:50:44.154 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T03:50:44.155 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:50:44.172 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T03:50:44.172 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T03:50:44.173 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T03:50:44.173 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T03:50:44.175 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T03:50:44.175 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T03:50:44.181 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T03:50:44.181 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T03:50:44.220 INFO:tasks.ceph.mds.c:Started 2024-03-23T03:50:44.220 DEBUG:teuthology.parallel:result is None 2024-03-23T03:50:44.225 INFO:tasks.ceph.mds.a:Started 2024-03-23T03:50:44.225 DEBUG:teuthology.parallel:result is None 2024-03-23T03:50:44.227 INFO:tasks.ceph.mds.d:Started 2024-03-23T03:50:44.228 DEBUG:teuthology.parallel:result is None 2024-03-23T03:50:44.234 INFO:tasks.ceph.mds.b:Started 2024-03-23T03:50:44.234 DEBUG:teuthology.parallel:result is None 2024-03-23T03:50:44.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-23T03:50:44.377 INFO:tasks.ceph.mds.c.smithi102.stdout:starting mds.c at 2024-03-23T03:50:44.377 INFO:tasks.ceph.mds.a.smithi102.stdout:starting mds.a at 2024-03-23T03:50:44.386 INFO:tasks.ceph.mds.d.smithi173.stdout:starting mds.d at 2024-03-23T03:50:44.395 INFO:tasks.ceph.mds.b.smithi173.stdout:starting mds.b at 2024-03-23T03:50:44.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.419+0000 7f1dcba00640 1 Processor -- start 2024-03-23T03:50:44.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.419+0000 7f1dcba00640 1 -- start start 2024-03-23T03:50:44.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.420+0000 7f1dcba00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1dc4159690 0x7f1dc40ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:44.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.420+0000 7f1dcba00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1dc40cd890 0x7f1dc40cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:44.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.420+0000 7f1dcba00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dc415e010 con 0x7f1dc40cc330 2024-03-23T03:50:44.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.420+0000 7f1dcba00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc40ce1f0 con 0x7f1dc4159690 2024-03-23T03:50:44.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.420+0000 7f1dcba00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dc40cc7f0 con 0x7f1dc40cd890 2024-03-23T03:50:44.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.420+0000 7f1dc9000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1dc4159690 0x7f1dc40ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:44.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.420+0000 7f1dc9a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1dc40cd890 0x7f1dc40cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:44.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.420+0000 7f1dc9000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f1dc4159690 0x7f1dc40ca7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44694/0 (socket says 172.21.15.102:44694) 2024-03-23T03:50:44.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.421+0000 7f1dc9000640 1 -- 172.21.15.102:0/4132901448 learned_addr learned my addr 172.21.15.102:0/4132901448 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:44.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.421+0000 7f1dc9000640 1 -- 172.21.15.102:0/4132901448 >> v2:172.21.15.173:3300/0 conn(0x7f1dc40cd890 msgr2=0x7f1dc40cdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:44.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.421+0000 7f1dc9000640 1 --2- 172.21.15.102:0/4132901448 >> v2:172.21.15.173:3300/0 conn(0x7f1dc40cd890 0x7f1dc40cdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:44.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.421+0000 7f1dc9000640 1 -- 172.21.15.102:0/4132901448 >> v1:172.21.15.173:6789/0 conn(0x7f1dc40cc330 legacy=0x7f1dc40b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:44.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.421+0000 7f1dc9000640 1 -- 172.21.15.102:0/4132901448 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc40ccad0 con 0x7f1dc4159690 2024-03-23T03:50:44.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.421+0000 7f1dc9a00640 1 --2- 172.21.15.102:0/4132901448 >> v2:172.21.15.173:3300/0 conn(0x7f1dc40cd890 0x7f1dc40cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:50:44.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.422+0000 7f1dc9000640 1 --2- 172.21.15.102:0/4132901448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dc4159690 0x7f1dc40ca7b0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40addb0 tx=0x7f1db802e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=122aec8d37751db1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:44.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.424+0000 7f1dc3400640 1 -- 172.21.15.102:0/4132901448 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db8036440 con 0x7f1dc4159690 2024-03-23T03:50:44.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.424+0000 7f1dc3400640 1 -- 172.21.15.102:0/4132901448 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db80365a0 con 0x7f1dc4159690 2024-03-23T03:50:44.425 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.424+0000 7f1dc3400640 1 -- 172.21.15.102:0/4132901448 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db8036930 con 0x7f1dc4159690 2024-03-23T03:50:44.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.425+0000 7f1dcba00640 1 -- 172.21.15.102:0/4132901448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dc4159690 msgr2=0x7f1dc40ca7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:44.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.425+0000 7f1dcba00640 1 --2- 172.21.15.102:0/4132901448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dc4159690 0x7f1dc40ca7b0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f1dc40addb0 tx=0x7f1db802e170 comp rx=0 tx=0).stop 2024-03-23T03:50:44.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.425+0000 7f1dcba00640 1 -- 172.21.15.102:0/4132901448 shutdown_connections 2024-03-23T03:50:44.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.425+0000 7f1dcba00640 1 --2- 172.21.15.102:0/4132901448 >> v2:172.21.15.173:3300/0 conn(0x7f1dc40cd890 0x7f1dc40cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:44.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.425+0000 7f1dcba00640 1 --2- 172.21.15.102:0/4132901448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dc4159690 0x7f1dc40ca7b0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:44.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.425+0000 7f1dcba00640 1 -- 172.21.15.102:0/4132901448 >> 172.21.15.102:0/4132901448 conn(0x7f1dc406a0b0 msgr2=0x7f1dc406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:44.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.425+0000 7f1dcba00640 1 -- 172.21.15.102:0/4132901448 shutdown_connections 2024-03-23T03:50:44.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.425+0000 7f1dcba00640 1 -- 172.21.15.102:0/4132901448 wait complete. 2024-03-23T03:50:44.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.427+0000 7f1dcba00640 1 Processor -- start 2024-03-23T03:50:44.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.427+0000 7f1dcba00640 1 -- start start 2024-03-23T03:50:44.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.428+0000 7f1dcba00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1dc40cc330 0x7f1dc414a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:44.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.428+0000 7f1dcba00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dc40cd890 0x7f1dc4148410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:44.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.428+0000 7f1dcba00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1dc4159690 0x7f1dc4148b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:44.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.428+0000 7f1dcba00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc414b910 con 0x7f1dc40cd890 2024-03-23T03:50:44.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.428+0000 7f1dcba00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc414ad00 con 0x7f1dc4159690 2024-03-23T03:50:44.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.428+0000 7f1dcba00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dc414afe0 con 0x7f1dc40cc330 2024-03-23T03:50:44.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.429+0000 7f1dc3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dc40cd890 0x7f1dc4148410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:44.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.429+0000 7f1dc3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dc40cd890 0x7f1dc4148410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44704/0 (socket says 172.21.15.102:44704) 2024-03-23T03:50:44.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.429+0000 7f1dc3e00640 1 -- 172.21.15.102:0/4186612575 learned_addr learned my addr 172.21.15.102:0/4186612575 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:44.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.429+0000 7f1dc9000640 1 --2- 172.21.15.102:0/4186612575 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1dc40cc330 0x7f1dc414a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:44.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.429+0000 7f1dc9a00640 1 --2- 172.21.15.102:0/4186612575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1dc4159690 0x7f1dc4148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:44.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.429+0000 7f1dc3e00640 1 -- 172.21.15.102:0/4186612575 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1dc40cc330 msgr2=0x7f1dc414a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:44.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.429+0000 7f1dc3e00640 1 --2- 172.21.15.102:0/4186612575 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1dc40cc330 0x7f1dc414a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:44.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.429+0000 7f1dc3e00640 1 -- 172.21.15.102:0/4186612575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1dc4159690 msgr2=0x7f1dc4148b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:44.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.429+0000 7f1dc3e00640 1 --2- 172.21.15.102:0/4186612575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1dc4159690 0x7f1dc4148b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:44.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.429+0000 7f1dc3e00640 1 -- 172.21.15.102:0/4186612575 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dbc009a70 con 0x7f1dc40cd890 2024-03-23T03:50:44.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.430+0000 7f1dc3e00640 1 --2- 172.21.15.102:0/4186612575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dc40cd890 0x7f1dc4148410 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f1db000c450 tx=0x7f1db000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:44.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.430+0000 7f1dc0c00640 1 -- 172.21.15.102:0/4186612575 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db00052d0 con 0x7f1dc40cd890 2024-03-23T03:50:44.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.430+0000 7f1dc0c00640 1 -- 172.21.15.102:0/4186612575 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db0005430 con 0x7f1dc40cd890 2024-03-23T03:50:44.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.430+0000 7f1dc0c00640 1 -- 172.21.15.102:0/4186612575 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db0003830 con 0x7f1dc40cd890 2024-03-23T03:50:44.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.430+0000 7f1dcba00640 1 -- 172.21.15.102:0/4186612575 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db800b770 con 0x7f1dc40cd890 2024-03-23T03:50:44.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.430+0000 7f1dcba00640 1 -- 172.21.15.102:0/4186612575 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dc414b5a0 con 0x7f1dc40cd890 2024-03-23T03:50:44.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.431+0000 7f1dcba00640 1 -- 172.21.15.102:0/4186612575 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dc4073390 con 0x7f1dc40cd890 2024-03-23T03:50:44.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.432+0000 7f1dc0c00640 1 -- 172.21.15.102:0/4186612575 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1db0003990 con 0x7f1dc40cd890 2024-03-23T03:50:44.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.433+0000 7f1dc0c00640 1 --2- 172.21.15.102:0/4186612575 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1da406ac00 0x7f1da406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:44.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.433+0000 7f1dc0c00640 1 -- 172.21.15.102:0/4186612575 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f1db0088730 con 0x7f1dc40cd890 2024-03-23T03:50:44.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.433+0000 7f1dc9000640 1 --2- 172.21.15.102:0/4186612575 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1da406ac00 0x7f1da406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:44.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.434+0000 7f1dc9000640 1 --2- 172.21.15.102:0/4186612575 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1da406ac00 0x7f1da406d0c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1db8038000 tx=0x7f1db8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:44.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.437+0000 7f1dc0c00640 1 -- 172.21.15.102:0/4186612575 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1db0053ae0 con 0x7f1dc40cd890 2024-03-23T03:50:44.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:44.645+0000 7f1dcba00640 1 -- 172.21.15.102:0/4186612575 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1dc40cf230 con 0x7f1dc40cd890 2024-03-23T03:50:45.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.435+0000 7f1dc0c00640 1 -- 172.21.15.102:0/4186612575 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v89) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1db0057100 con 0x7f1dc40cd890 2024-03-23T03:50:45.436 INFO:teuthology.orchestra.run.smithi102.stderr: removed all blocklist entries 2024-03-23T03:50:45.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.439+0000 7f1dcba00640 1 -- 172.21.15.102:0/4186612575 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1da406ac00 msgr2=0x7f1da406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:45.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.439+0000 7f1dcba00640 1 --2- 172.21.15.102:0/4186612575 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1da406ac00 0x7f1da406d0c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1db8038000 tx=0x7f1db8008040 comp rx=0 tx=0).stop 2024-03-23T03:50:45.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.439+0000 7f1dcba00640 1 -- 172.21.15.102:0/4186612575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dc40cd890 msgr2=0x7f1dc4148410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:45.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.439+0000 7f1dcba00640 1 --2- 172.21.15.102:0/4186612575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dc40cd890 0x7f1dc4148410 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f1db000c450 tx=0x7f1db000c850 comp rx=0 tx=0).stop 2024-03-23T03:50:45.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.439+0000 7f1dcba00640 1 -- 172.21.15.102:0/4186612575 shutdown_connections 2024-03-23T03:50:45.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.439+0000 7f1dcba00640 1 --2- 172.21.15.102:0/4186612575 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1da406ac00 0x7f1da406d0c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:45.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.439+0000 7f1dcba00640 1 --2- 172.21.15.102:0/4186612575 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1dc4159690 0x7f1dc4148b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:45.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.439+0000 7f1dcba00640 1 --2- 172.21.15.102:0/4186612575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1dc40cd890 0x7f1dc4148410 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:45.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.439+0000 7f1dcba00640 1 --2- 172.21.15.102:0/4186612575 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1dc40cc330 0x7f1dc414a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:45.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.440+0000 7f1dcba00640 1 -- 172.21.15.102:0/4186612575 >> 172.21.15.102:0/4186612575 conn(0x7f1dc406a0b0 msgr2=0x7f1dc406a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:45.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.440+0000 7f1dcba00640 1 -- 172.21.15.102:0/4186612575 shutdown_connections 2024-03-23T03:50:45.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.440+0000 7f1dcba00640 1 -- 172.21.15.102:0/4186612575 wait complete. 2024-03-23T03:50:45.461 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-23T03:50:45.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.671+0000 7fa8b7400640 1 Processor -- start 2024-03-23T03:50:45.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.672+0000 7fa8b7400640 1 -- start start 2024-03-23T03:50:45.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.673+0000 7fa8b7400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa8b013faa0 0x7fa8b0160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:45.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.673+0000 7fa8b7400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b00b38a0 con 0x7fa8b00b1db0 2024-03-23T03:50:45.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.673+0000 7fa8b7400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b00b3b80 con 0x7fa8b0160800 2024-03-23T03:50:45.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.673+0000 7fa8b7400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b00b3e60 con 0x7fa8b013faa0 2024-03-23T03:50:45.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.673+0000 7fa8b4a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa8b013faa0 0x7fa8b0160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:45.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.674+0000 7fa8b4a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fa8b013faa0 0x7fa8b0160f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44720/0 (socket says 172.21.15.102:44720) 2024-03-23T03:50:45.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.674+0000 7fa8b4a00640 1 -- 172.21.15.102:0/3820069260 learned_addr learned my addr 172.21.15.102:0/3820069260 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:45.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.674+0000 7fa8b4a00640 1 -- 172.21.15.102:0/3820069260 >> v1:172.21.15.173:6789/0 conn(0x7fa8b0160800 legacy=0x7fa8b00b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:45.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.674+0000 7fa8b4a00640 1 -- 172.21.15.102:0/3820069260 >> v1:172.21.15.102:6789/0 conn(0x7fa8b00b1db0 legacy=0x7fa8b006c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:45.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.674+0000 7fa8b4a00640 1 -- 172.21.15.102:0/3820069260 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b00b4140 con 0x7fa8b013faa0 2024-03-23T03:50:45.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.675+0000 7fa8b4a00640 1 --2- 172.21.15.102:0/3820069260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8b013faa0 0x7fa8b0160f30 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fa898009870 tx=0x7fa89802b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24e0b4e9a5bc7fd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:45.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.675+0000 7fa8afe00640 1 -- 172.21.15.102:0/3820069260 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa898030850 con 0x7fa8b013faa0 2024-03-23T03:50:45.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.675+0000 7fa8afe00640 1 -- 172.21.15.102:0/3820069260 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8980309b0 con 0x7fa8b013faa0 2024-03-23T03:50:45.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.675+0000 7fa8afe00640 1 -- 172.21.15.102:0/3820069260 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa898030d40 con 0x7fa8b013faa0 2024-03-23T03:50:45.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.676+0000 7fa8b7400640 1 -- 172.21.15.102:0/3820069260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8b013faa0 msgr2=0x7fa8b0160f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:45.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.676+0000 7fa8b7400640 1 --2- 172.21.15.102:0/3820069260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8b013faa0 0x7fa8b0160f30 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fa898009870 tx=0x7fa89802b990 comp rx=0 tx=0).stop 2024-03-23T03:50:45.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.676+0000 7fa8b7400640 1 -- 172.21.15.102:0/3820069260 shutdown_connections 2024-03-23T03:50:45.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.676+0000 7fa8b7400640 1 --2- 172.21.15.102:0/3820069260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8b013faa0 0x7fa8b0160f30 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:45.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.676+0000 7fa8b7400640 1 -- 172.21.15.102:0/3820069260 >> 172.21.15.102:0/3820069260 conn(0x7fa8b006a3d0 msgr2=0x7fa8b00b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:45.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.676+0000 7fa8b7400640 1 -- 172.21.15.102:0/3820069260 shutdown_connections 2024-03-23T03:50:45.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.676+0000 7fa8b7400640 1 -- 172.21.15.102:0/3820069260 wait complete. 2024-03-23T03:50:45.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.678+0000 7fa8b7400640 1 Processor -- start 2024-03-23T03:50:45.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.678+0000 7fa8b7400640 1 -- start start 2024-03-23T03:50:45.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.679+0000 7fa8b7400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8b00b1db0 0x7fa8b0155230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:45.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.679+0000 7fa8b7400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa8b013faa0 0x7fa8b01efcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:45.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.679+0000 7fa8b7400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8b0160800 0x7fa8b00ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:45.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.679+0000 7fa8b7400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b0138950 con 0x7fa8b0160800 2024-03-23T03:50:45.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.679+0000 7fa8b7400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b01f0200 con 0x7fa8b00b1db0 2024-03-23T03:50:45.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.679+0000 7fa8b7400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b01f2bc0 con 0x7fa8b013faa0 2024-03-23T03:50:45.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.679+0000 7fa8b5400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8b0160800 0x7fa8b00ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:45.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.679+0000 7fa8b5400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8b0160800 0x7fa8b00ccd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44724/0 (socket says 172.21.15.102:44724) 2024-03-23T03:50:45.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.679+0000 7fa8b5400640 1 -- 172.21.15.102:0/4123076269 learned_addr learned my addr 172.21.15.102:0/4123076269 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:45.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.680+0000 7fa8b4a00640 1 --2- 172.21.15.102:0/4123076269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8b00b1db0 0x7fa8b0155230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:45.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.680+0000 7fa8a7e00640 1 --2- 172.21.15.102:0/4123076269 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa8b013faa0 0x7fa8b01efcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:45.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.680+0000 7fa8b5400640 1 -- 172.21.15.102:0/4123076269 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa8b013faa0 msgr2=0x7fa8b01efcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:45.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.680+0000 7fa8b5400640 1 --2- 172.21.15.102:0/4123076269 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa8b013faa0 0x7fa8b01efcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:45.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.680+0000 7fa8b5400640 1 -- 172.21.15.102:0/4123076269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8b00b1db0 msgr2=0x7fa8b0155230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:45.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.680+0000 7fa8b5400640 1 --2- 172.21.15.102:0/4123076269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8b00b1db0 0x7fa8b0155230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:45.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.680+0000 7fa8b5400640 1 -- 172.21.15.102:0/4123076269 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8980094d0 con 0x7fa8b0160800 2024-03-23T03:50:45.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.680+0000 7fa8b4a00640 1 --2- 172.21.15.102:0/4123076269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8b00b1db0 0x7fa8b0155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:45.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.680+0000 7fa8b5400640 1 --2- 172.21.15.102:0/4123076269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8b0160800 0x7fa8b00ccd90 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fa8a800c730 tx=0x7fa8a800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:45.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.680+0000 7fa8ad600640 1 -- 172.21.15.102:0/4123076269 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a8003b70 con 0x7fa8b0160800 2024-03-23T03:50:45.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.680+0000 7fa8ad600640 1 -- 172.21.15.102:0/4123076269 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a8003cd0 con 0x7fa8b0160800 2024-03-23T03:50:45.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.680+0000 7fa8b7400640 1 -- 172.21.15.102:0/4123076269 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b01f2ea0 con 0x7fa8b0160800 2024-03-23T03:50:45.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.680+0000 7fa8ad600640 1 -- 172.21.15.102:0/4123076269 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a800aa90 con 0x7fa8b0160800 2024-03-23T03:50:45.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.681+0000 7fa8b7400640 1 -- 172.21.15.102:0/4123076269 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b01f3460 con 0x7fa8b0160800 2024-03-23T03:50:45.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.682+0000 7fa8b7400640 1 -- 172.21.15.102:0/4123076269 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b0073390 con 0x7fa8b0160800 2024-03-23T03:50:45.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.682+0000 7fa8ad600640 1 -- 172.21.15.102:0/4123076269 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa8a800abf0 con 0x7fa8b0160800 2024-03-23T03:50:45.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.683+0000 7fa8ad600640 1 --2- 172.21.15.102:0/4123076269 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa88c06ac00 0x7fa88c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:45.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.683+0000 7fa8ad600640 1 -- 172.21.15.102:0/4123076269 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fa8a8088f80 con 0x7fa8b0160800 2024-03-23T03:50:45.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.683+0000 7fa8b4a00640 1 --2- 172.21.15.102:0/4123076269 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa88c06ac00 0x7fa88c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:45.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.684+0000 7fa8b4a00640 1 --2- 172.21.15.102:0/4123076269 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa88c06ac00 0x7fa88c06d0c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa898036000 tx=0x7fa89803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:45.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.688+0000 7fa8ad600640 1 -- 172.21.15.102:0/4123076269 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa8a80bc580 con 0x7fa8b0160800 2024-03-23T03:50:45.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.955+0000 7fa8b7400640 1 -- 172.21.15.102:0/4123076269 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa8b0138ae0 con 0x7fa8b0160800 2024-03-23T03:50:45.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.956+0000 7fa8ad600640 1 -- 172.21.15.102:0/4123076269 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fa8a8054370 con 0x7fa8b0160800 2024-03-23T03:50:45.957 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:45.957 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:50:45.957 INFO:teuthology.orchestra.run.smithi102.stdout: "auth_dump": [ 2024-03-23T03:50:45.957 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.957 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.a", 2024-03-23T03:50:45.957 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5loBTZFRAA7/tTYyZwrDmwKwiU+dtCEw==", 2024-03-23T03:50:45.957 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.957 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:50:45.957 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:50:45.957 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:50:45.957 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:45.957 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.958 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.958 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.958 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.b", 2024-03-23T03:50:45.958 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5lJKKlNxAAj5+j99K79IhbeEm7/qC+wg==", 2024-03-23T03:50:45.958 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.958 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:50:45.958 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:50:45.958 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:50:45.958 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:45.958 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.958 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.958 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.959 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.c", 2024-03-23T03:50:45.959 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5l0b92JhAA5rvs+mdgmrQX/9lI3GW2rg==", 2024-03-23T03:50:45.959 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.959 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:50:45.959 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:50:45.959 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:50:45.959 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:45.959 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.959 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.959 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.959 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.d", 2024-03-23T03:50:45.960 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5ltjNMDBAAD1h5Nlv2Cp7mJt+DMlBtzw==", 2024-03-23T03:50:45.960 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.960 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:50:45.960 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:50:45.960 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:50:45.960 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:45.960 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.960 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.960 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.960 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.0", 2024-03-23T03:50:45.960 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB2UP5lPW6uORAAejiQReOF95oFliFmk/vdQQ==", 2024-03-23T03:50:45.960 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.960 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:50:45.960 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:50:45.961 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:45.961 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.961 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.961 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.961 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.1", 2024-03-23T03:50:45.961 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB6UP5lFbbUBxAAIl0ZEYtJ+/oi3W6a4OwMhg==", 2024-03-23T03:50:45.961 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.961 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:50:45.961 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:50:45.961 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:45.961 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.961 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.962 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.962 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.2", 2024-03-23T03:50:45.962 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB9UP5l2LaQIBAAKJv7jWJAjlUJrm7+KTXi6Q==", 2024-03-23T03:50:45.962 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.962 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:50:45.962 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:50:45.962 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:45.962 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.962 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.962 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.962 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.3", 2024-03-23T03:50:45.962 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCAUP5lxjWsKhAAzVO8CXUd+4LLvv4f/5xMOg==", 2024-03-23T03:50:45.962 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.962 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:50:45.963 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:50:45.963 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:45.963 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.963 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.963 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.963 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.4", 2024-03-23T03:50:45.963 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCHUP5lHcd2JRAAwID1TPb+Z9ViH51B0Mv/kQ==", 2024-03-23T03:50:45.963 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.963 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:50:45.963 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:50:45.963 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:45.963 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.964 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.964 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.964 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.5", 2024-03-23T03:50:45.964 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCLUP5lApp3AxAADslbuzTe/bf5FML1ckE05w==", 2024-03-23T03:50:45.964 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.964 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:50:45.964 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:50:45.964 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:45.964 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.964 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.964 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.964 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.6", 2024-03-23T03:50:45.964 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCOUP5lsi7VGxAArhWvXQ1VzJ8oSiOXzq8VsQ==", 2024-03-23T03:50:45.964 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.965 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:50:45.965 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:50:45.965 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:45.965 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.965 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.965 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.965 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.7", 2024-03-23T03:50:45.965 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCRUP5lu182JxAA5LaDXfrBkOXEUNAJXFDCJQ==", 2024-03-23T03:50:45.965 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.965 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:50:45.965 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:50:45.965 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:45.965 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.965 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.966 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.966 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.0", 2024-03-23T03:50:45.966 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lxTfoGRAA3WIQ8p1rGSAyJkLZJMk3dA==", 2024-03-23T03:50:45.966 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:50:45.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow r", 2024-03-23T03:50:45.966 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-23T03:50:45.966 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.966 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.966 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.966 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.1", 2024-03-23T03:50:45.966 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lneSHLhAAwn6ULOajHXrOM650HUcLbw==", 2024-03-23T03:50:45.966 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.966 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:50:45.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:50:45.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:50:45.967 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:50:45.967 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.967 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.967 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.967 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.2", 2024-03-23T03:50:45.967 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lOIklJBAAhx2d0Xy/j1PG0qsvC3nMqA==", 2024-03-23T03:50:45.967 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:50:45.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:50:45.967 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:50:45.967 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:50:45.967 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.968 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.968 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.968 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.3", 2024-03-23T03:50:45.968 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5l+7BVORAA7PBYS6mkgLA8b4sLCAYgBA==", 2024-03-23T03:50:45.968 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.968 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:50:45.968 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:50:45.968 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:50:45.968 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:50:45.968 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.968 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.968 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.968 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.admin", 2024-03-23T03:50:45.968 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBwUP5ldlDgBxAAOiqmhC5GulxELun8OiR4lg==", 2024-03-23T03:50:45.968 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:50:45.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:50:45.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:50:45.969 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:45.969 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.969 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.969 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.969 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mgr.x", 2024-03-23T03:50:45.969 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5lQXojChAAzepTDh0Pk3fuBZYsFt/+DQ==", 2024-03-23T03:50:45.969 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:50:45.969 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile mgr", 2024-03-23T03:50:45.969 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:45.969 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.970 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:50:45.970 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:45.970 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mgr.y", 2024-03-23T03:50:45.970 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBwUP5lK59iORAACuBF0JaQ8epyu3Mq3slzjg==", 2024-03-23T03:50:45.970 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:50:45.970 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:50:45.970 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile mgr", 2024-03-23T03:50:45.970 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:50:45.970 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.970 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:45.970 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-03-23T03:50:45.970 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:50:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.959+0000 7fa8b7400640 1 -- 172.21.15.102:0/4123076269 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa88c06ac00 msgr2=0x7fa88c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.959+0000 7fa8b7400640 1 --2- 172.21.15.102:0/4123076269 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa88c06ac00 0x7fa88c06d0c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa898036000 tx=0x7fa89803b040 comp rx=0 tx=0).stop 2024-03-23T03:50:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.959+0000 7fa8b7400640 1 -- 172.21.15.102:0/4123076269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8b0160800 msgr2=0x7fa8b00ccd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.959+0000 7fa8b7400640 1 --2- 172.21.15.102:0/4123076269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8b0160800 0x7fa8b00ccd90 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fa8a800c730 tx=0x7fa8a800cbf0 comp rx=0 tx=0).stop 2024-03-23T03:50:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.959+0000 7fa8b7400640 1 -- 172.21.15.102:0/4123076269 shutdown_connections 2024-03-23T03:50:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.959+0000 7fa8b7400640 1 --2- 172.21.15.102:0/4123076269 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa88c06ac00 0x7fa88c06d0c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.959+0000 7fa8b7400640 1 --2- 172.21.15.102:0/4123076269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8b0160800 0x7fa8b00ccd90 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.959+0000 7fa8b7400640 1 --2- 172.21.15.102:0/4123076269 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa8b013faa0 0x7fa8b01efcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.959+0000 7fa8b7400640 1 --2- 172.21.15.102:0/4123076269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8b00b1db0 0x7fa8b0155230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:45.972 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.959+0000 7fa8b7400640 1 -- 172.21.15.102:0/4123076269 >> 172.21.15.102:0/4123076269 conn(0x7fa8b006a3d0 msgr2=0x7fa8b0070900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:45.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.959+0000 7fa8b7400640 1 -- 172.21.15.102:0/4123076269 shutdown_connections 2024-03-23T03:50:45.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:45.959+0000 7fa8b7400640 1 -- 172.21.15.102:0/4123076269 wait complete. 2024-03-23T03:50:45.979 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:50:46.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.185+0000 7f73e9e00640 1 Processor -- start 2024-03-23T03:50:46.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.185+0000 7f73e9e00640 1 -- start start 2024-03-23T03:50:46.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.187+0000 7f73e9e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f73e40adbc0 0x7f73e40b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:46.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.187+0000 7f73e9e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f73e406d390 0x7f73e406d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:46.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.187+0000 7f73e9e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73e40b16a0 con 0x7f73e40cbab0 2024-03-23T03:50:46.188 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.187+0000 7f73e9e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e406dd40 con 0x7f73e406d390 2024-03-23T03:50:46.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.187+0000 7f73e9e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e40b4700 con 0x7f73e40adbc0 2024-03-23T03:50:46.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.187+0000 7f73e3400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f73e406d390 0x7f73e406d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:46.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.187+0000 7f73e3400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f73e406d390 0x7f73e406d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44740/0 (socket says 172.21.15.102:44740) 2024-03-23T03:50:46.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.187+0000 7f73e3400640 1 -- 172.21.15.102:0/2897433994 learned_addr learned my addr 172.21.15.102:0/2897433994 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:46.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.187+0000 7f73e2a00640 1 --2- 172.21.15.102:0/2897433994 >> v2:172.21.15.173:3300/0 conn(0x7f73e40adbc0 0x7f73e40b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:46.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.188+0000 7f73e3400640 1 -- 172.21.15.102:0/2897433994 >> v2:172.21.15.173:3300/0 conn(0x7f73e40adbc0 msgr2=0x7f73e40b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:46.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.188+0000 7f73e3400640 1 --2- 172.21.15.102:0/2897433994 >> v2:172.21.15.173:3300/0 conn(0x7f73e40adbc0 0x7f73e40b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.188+0000 7f73e3400640 1 -- 172.21.15.102:0/2897433994 >> v1:172.21.15.173:6789/0 conn(0x7f73e40cbab0 legacy=0x7f73e40b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:46.189 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.188+0000 7f73e3400640 1 -- 172.21.15.102:0/2897433994 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e40cbfb0 con 0x7f73e406d390 2024-03-23T03:50:46.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.189+0000 7f73e3400640 1 --2- 172.21.15.102:0/2897433994 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73e406d390 0x7f73e406d800 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f73e406e020 tx=0x7f73dc02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf06393987185bf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:46.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.189+0000 7f73e1600640 1 -- 172.21.15.102:0/2897433994 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc02c9f0 con 0x7f73e406d390 2024-03-23T03:50:46.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.189+0000 7f73e1600640 1 -- 172.21.15.102:0/2897433994 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73dc02cb50 con 0x7f73e406d390 2024-03-23T03:50:46.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.189+0000 7f73e1600640 1 -- 172.21.15.102:0/2897433994 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73dc0343f0 con 0x7f73e406d390 2024-03-23T03:50:46.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.190+0000 7f73e9e00640 1 -- 172.21.15.102:0/2897433994 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73e406d390 msgr2=0x7f73e406d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:46.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.190+0000 7f73e9e00640 1 --2- 172.21.15.102:0/2897433994 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73e406d390 0x7f73e406d800 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f73e406e020 tx=0x7f73dc02be50 comp rx=0 tx=0).stop 2024-03-23T03:50:46.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.190+0000 7f73e9e00640 1 -- 172.21.15.102:0/2897433994 shutdown_connections 2024-03-23T03:50:46.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.190+0000 7f73e9e00640 1 --2- 172.21.15.102:0/2897433994 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73e406d390 0x7f73e406d800 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.190+0000 7f73e9e00640 1 --2- 172.21.15.102:0/2897433994 >> v2:172.21.15.173:3300/0 conn(0x7f73e40adbc0 0x7f73e40b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.190+0000 7f73e9e00640 1 -- 172.21.15.102:0/2897433994 >> 172.21.15.102:0/2897433994 conn(0x7f73e406a3d0 msgr2=0x7f73e40b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:46.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.190+0000 7f73e9e00640 1 -- 172.21.15.102:0/2897433994 shutdown_connections 2024-03-23T03:50:46.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.190+0000 7f73e9e00640 1 -- 172.21.15.102:0/2897433994 wait complete. 2024-03-23T03:50:46.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.192+0000 7f73e9e00640 1 Processor -- start 2024-03-23T03:50:46.193 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.192+0000 7f73e9e00640 1 -- start start 2024-03-23T03:50:46.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.193+0000 7f73e9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73e406d390 0x7f73e41e65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:46.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.193+0000 7f73e9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73e40adbc0 0x7f73e4159040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:46.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.194+0000 7f73e9e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f73e40cbab0 0x7f73e4159580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:46.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.194+0000 7f73e9e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e40afe00 con 0x7f73e406d390 2024-03-23T03:50:46.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.194+0000 7f73e9e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e40af1f0 con 0x7f73e40adbc0 2024-03-23T03:50:46.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.194+0000 7f73e9e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e40af4d0 con 0x7f73e40cbab0 2024-03-23T03:50:46.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.194+0000 7f73e2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73e406d390 0x7f73e41e65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:46.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.194+0000 7f73e2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73e406d390 0x7f73e41e65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44746/0 (socket says 172.21.15.102:44746) 2024-03-23T03:50:46.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.194+0000 7f73e2a00640 1 -- 172.21.15.102:0/2713460335 learned_addr learned my addr 172.21.15.102:0/2713460335 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:46.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.194+0000 7f73e2a00640 1 -- 172.21.15.102:0/2713460335 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f73e40cbab0 msgr2=0x7f73e4159580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:46.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.194+0000 7f73e2a00640 1 --2- 172.21.15.102:0/2713460335 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f73e40cbab0 0x7f73e4159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.194+0000 7f73e2a00640 1 -- 172.21.15.102:0/2713460335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73e40adbc0 msgr2=0x7f73e4159040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:46.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.194+0000 7f73e2a00640 1 --2- 172.21.15.102:0/2713460335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73e40adbc0 0x7f73e4159040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.194+0000 7f73e2a00640 1 -- 172.21.15.102:0/2713460335 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73dc009b30 con 0x7f73e406d390 2024-03-23T03:50:46.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.194+0000 7f73e2a00640 1 --2- 172.21.15.102:0/2713460335 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73e406d390 0x7f73e41e65a0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f73d400d800 tx=0x7f73d400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:46.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.195+0000 7f73db400640 1 -- 172.21.15.102:0/2713460335 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d400b6c0 con 0x7f73e406d390 2024-03-23T03:50:46.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.195+0000 7f73db400640 1 -- 172.21.15.102:0/2713460335 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d400b820 con 0x7f73e406d390 2024-03-23T03:50:46.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.195+0000 7f73db400640 1 -- 172.21.15.102:0/2713460335 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d400bbb0 con 0x7f73e406d390 2024-03-23T03:50:46.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.195+0000 7f73e9e00640 1 -- 172.21.15.102:0/2713460335 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e40af7b0 con 0x7f73e406d390 2024-03-23T03:50:46.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.195+0000 7f73e9e00640 1 -- 172.21.15.102:0/2713460335 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e40b0290 con 0x7f73e406d390 2024-03-23T03:50:46.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.196+0000 7f73e9e00640 1 -- 172.21.15.102:0/2713460335 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e406ac20 con 0x7f73e406d390 2024-03-23T03:50:46.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.197+0000 7f73db400640 1 -- 172.21.15.102:0/2713460335 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f73d400bd10 con 0x7f73e406d390 2024-03-23T03:50:46.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.198+0000 7f73db400640 1 --2- 172.21.15.102:0/2713460335 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f73c406ac00 0x7f73c406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:46.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.198+0000 7f73db400640 1 -- 172.21.15.102:0/2713460335 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f73d40897c0 con 0x7f73e406d390 2024-03-23T03:50:46.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.198+0000 7f73e2000640 1 --2- 172.21.15.102:0/2713460335 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f73c406ac00 0x7f73c406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:46.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.199+0000 7f73e2000640 1 --2- 172.21.15.102:0/2713460335 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f73c406ac00 0x7f73c406d0c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f73e40ae4a0 tx=0x7f73d00079b0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:46.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.204+0000 7f73db400640 1 -- 172.21.15.102:0/2713460335 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f73d4054a80 con 0x7f73e406d390 2024-03-23T03:50:46.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.400+0000 7f73e9e00640 1 -- 172.21.15.102:0/2713460335 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73e4073390 con 0x7f73e406d390 2024-03-23T03:50:46.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.402+0000 7f73db400640 1 -- 172.21.15.102:0/2713460335 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+12643 (secure 0 0 0) 0x7f73d40580a0 con 0x7f73e406d390 2024-03-23T03:50:46.403 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:46.403 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":89,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:45.428445+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:50:46.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.404+0000 7f73e9e00640 1 -- 172.21.15.102:0/2713460335 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f73c406ac00 msgr2=0x7f73c406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:46.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.404+0000 7f73e9e00640 1 --2- 172.21.15.102:0/2713460335 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f73c406ac00 0x7f73c406d0c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f73e40ae4a0 tx=0x7f73d00079b0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.404+0000 7f73e9e00640 1 -- 172.21.15.102:0/2713460335 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73e406d390 msgr2=0x7f73e41e65a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:46.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.404+0000 7f73e9e00640 1 --2- 172.21.15.102:0/2713460335 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73e406d390 0x7f73e41e65a0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f73d400d800 tx=0x7f73d400dcc0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.404+0000 7f73e9e00640 1 -- 172.21.15.102:0/2713460335 shutdown_connections 2024-03-23T03:50:46.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.404+0000 7f73e9e00640 1 --2- 172.21.15.102:0/2713460335 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f73c406ac00 0x7f73c406d0c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.404+0000 7f73e9e00640 1 --2- 172.21.15.102:0/2713460335 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f73e40cbab0 0x7f73e4159580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.404+0000 7f73e9e00640 1 --2- 172.21.15.102:0/2713460335 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73e40adbc0 0x7f73e4159040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.404+0000 7f73e9e00640 1 --2- 172.21.15.102:0/2713460335 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f73e406d390 0x7f73e41e65a0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.404+0000 7f73e9e00640 1 -- 172.21.15.102:0/2713460335 >> 172.21.15.102:0/2713460335 conn(0x7f73e406a3d0 msgr2=0x7f73e4141440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:46.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.405+0000 7f73e9e00640 1 -- 172.21.15.102:0/2713460335 shutdown_connections 2024-03-23T03:50:46.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.405+0000 7f73e9e00640 1 -- 172.21.15.102:0/2713460335 wait complete. 2024-03-23T03:50:46.423 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:50:46.423 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:50:46.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.622+0000 7eff71800640 1 Processor -- start 2024-03-23T03:50:46.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.622+0000 7eff71800640 1 -- start start 2024-03-23T03:50:46.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.624+0000 7eff71800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7eff6c0cf040 0x7eff6c1604c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:46.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.624+0000 7eff71800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff6c06d390 con 0x7eff6c1456a0 2024-03-23T03:50:46.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.624+0000 7eff71800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff6c06d670 con 0x7eff6c070cd0 2024-03-23T03:50:46.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.624+0000 7eff71800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff6c0ad890 con 0x7eff6c0cf040 2024-03-23T03:50:46.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.624+0000 7eff6aa00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7eff6c1456a0 0x7eff6c0ce930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56790/0 (socket says 172.21.15.102:56790) 2024-03-23T03:50:46.625 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.624+0000 7eff6aa00640 1 -- 172.21.15.102:0/2086770581 learned_addr learned my addr 172.21.15.102:0/2086770581 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:46.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.625+0000 7eff6a000640 1 -- 172.21.15.102:0/2086770581 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 447488429 0 0) 0x7eff6c06d390 con 0x7eff6c1456a0 2024-03-23T03:50:46.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.625+0000 7eff6a000640 1 -- 172.21.15.102:0/2086770581 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff50003520 con 0x7eff6c1456a0 2024-03-23T03:50:46.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.625+0000 7eff62000640 1 --2- 172.21.15.102:0/2086770581 >> v2:172.21.15.173:3300/0 conn(0x7eff6c0cf040 0x7eff6c1604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:46.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.625+0000 7eff6a000640 1 -- 172.21.15.102:0/2086770581 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3137693532 0 0) 0x7eff50003520 con 0x7eff6c1456a0 2024-03-23T03:50:46.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.625+0000 7eff6a000640 1 -- 172.21.15.102:0/2086770581 >> v2:172.21.15.173:3300/0 conn(0x7eff6c0cf040 msgr2=0x7eff6c1604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:46.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.625+0000 7eff6a000640 1 --2- 172.21.15.102:0/2086770581 >> v2:172.21.15.173:3300/0 conn(0x7eff6c0cf040 0x7eff6c1604c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.625+0000 7eff6a000640 1 -- 172.21.15.102:0/2086770581 >> v1:172.21.15.173:6789/0 conn(0x7eff6c070cd0 legacy=0x7eff6c160a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:46.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.625+0000 7eff6a000640 1 -- 172.21.15.102:0/2086770581 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff6c0701f0 con 0x7eff6c1456a0 2024-03-23T03:50:46.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.625+0000 7eff6a000640 1 -- 172.21.15.102:0/2086770581 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7eff5c001670 con 0x7eff6c1456a0 2024-03-23T03:50:46.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.625+0000 7eff6a000640 1 -- 172.21.15.102:0/2086770581 >> v1:172.21.15.102:6789/0 conn(0x7eff6c1456a0 legacy=0x7eff6c0ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:46.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.625+0000 7eff6a000640 1 --2- 172.21.15.102:0/2086770581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff50003970 0x7eff50003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:46.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.625+0000 7eff6a000640 1 --2- 172.21.15.102:0/2086770581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff500047c0 0x7eff50004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:46.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.625+0000 7eff6a000640 1 --2- 172.21.15.102:0/2086770581 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff50005180 0x7eff5000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:46.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.625+0000 7eff6a000640 1 -- 172.21.15.102:0/2086770581 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c0ad890 con 0x7eff50003970 2024-03-23T03:50:46.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.625+0000 7eff6a000640 1 -- 172.21.15.102:0/2086770581 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff5000c0b0 con 0x7eff500047c0 2024-03-23T03:50:46.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.625+0000 7eff6a000640 1 -- 172.21.15.102:0/2086770581 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff5000c270 con 0x7eff50005180 2024-03-23T03:50:46.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.626+0000 7eff6aa00640 1 --2- 172.21.15.102:0/2086770581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff50003970 0x7eff50003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:46.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.626+0000 7eff6b400640 1 --2- 172.21.15.102:0/2086770581 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff50005180 0x7eff5000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:46.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.626+0000 7eff62000640 1 --2- 172.21.15.102:0/2086770581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff500047c0 0x7eff50004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:46.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.626+0000 7eff6aa00640 1 -- 172.21.15.102:0/2086770581 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff50005180 msgr2=0x7eff5000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:46.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.626+0000 7eff6aa00640 1 --2- 172.21.15.102:0/2086770581 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff50005180 0x7eff5000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.626+0000 7eff6aa00640 1 -- 172.21.15.102:0/2086770581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff500047c0 msgr2=0x7eff50004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:46.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.626+0000 7eff6aa00640 1 --2- 172.21.15.102:0/2086770581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff500047c0 0x7eff50004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.626+0000 7eff6aa00640 1 -- 172.21.15.102:0/2086770581 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff50003520 con 0x7eff50003970 2024-03-23T03:50:46.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.626+0000 7eff6b400640 1 -- 172.21.15.102:0/2086770581 reap_dead start 2024-03-23T03:50:46.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.627+0000 7eff6aa00640 1 --2- 172.21.15.102:0/2086770581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff50003970 0x7eff50003d70 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7eff5c001eb0 tx=0x7eff500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8133f9e6f2138e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:46.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.628+0000 7eff6a000640 1 -- 172.21.15.102:0/2086770581 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c005ad0 con 0x7eff50003970 2024-03-23T03:50:46.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.628+0000 7eff6a000640 1 -- 172.21.15.102:0/2086770581 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff5c005c50 con 0x7eff50003970 2024-03-23T03:50:46.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.628+0000 7eff6a000640 1 -- 172.21.15.102:0/2086770581 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff5c02c890 con 0x7eff50003970 2024-03-23T03:50:46.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.628+0000 7eff71800640 1 -- 172.21.15.102:0/2086770581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff50003970 msgr2=0x7eff50003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:46.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.628+0000 7eff71800640 1 --2- 172.21.15.102:0/2086770581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff50003970 0x7eff50003d70 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7eff5c001eb0 tx=0x7eff500057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.628+0000 7eff71800640 1 -- 172.21.15.102:0/2086770581 shutdown_connections 2024-03-23T03:50:46.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.628+0000 7eff71800640 1 --2- 172.21.15.102:0/2086770581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff50003970 0x7eff50003d70 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.628+0000 7eff71800640 1 -- 172.21.15.102:0/2086770581 >> 172.21.15.102:0/2086770581 conn(0x7eff6c06a0b0 msgr2=0x7eff6c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:46.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.628+0000 7eff71800640 1 -- 172.21.15.102:0/2086770581 shutdown_connections 2024-03-23T03:50:46.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.628+0000 7eff71800640 1 -- 172.21.15.102:0/2086770581 wait complete. 2024-03-23T03:50:46.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.630+0000 7eff71800640 1 Processor -- start 2024-03-23T03:50:46.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.630+0000 7eff71800640 1 -- start start 2024-03-23T03:50:46.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.631+0000 7eff71800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff50003970 0x7eff6c1f1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:46.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.631+0000 7eff71800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff6c1429e0 0x7eff6c13eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:46.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.631+0000 7eff71800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff6c142e40 0x7eff6c13f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:46.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.631+0000 7eff71800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c0c5010 con 0x7eff6c1429e0 2024-03-23T03:50:46.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.631+0000 7eff71800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c1f20e0 con 0x7eff50003970 2024-03-23T03:50:46.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.631+0000 7eff71800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff6c143750 con 0x7eff6c142e40 2024-03-23T03:50:46.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.631+0000 7eff62000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff6c1429e0 0x7eff6c13eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:46.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.631+0000 7eff62000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff6c1429e0 0x7eff6c13eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44762/0 (socket says 172.21.15.102:44762) 2024-03-23T03:50:46.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.631+0000 7eff62000640 1 -- 172.21.15.102:0/1767350350 learned_addr learned my addr 172.21.15.102:0/1767350350 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:46.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.632+0000 7eff6aa00640 1 --2- 172.21.15.102:0/1767350350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff50003970 0x7eff6c1f1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:46.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.632+0000 7eff6b400640 1 --2- 172.21.15.102:0/1767350350 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff6c142e40 0x7eff6c13f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:46.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.632+0000 7eff62000640 1 -- 172.21.15.102:0/1767350350 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff6c142e40 msgr2=0x7eff6c13f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:46.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.632+0000 7eff62000640 1 --2- 172.21.15.102:0/1767350350 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff6c142e40 0x7eff6c13f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.632+0000 7eff62000640 1 -- 172.21.15.102:0/1767350350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff50003970 msgr2=0x7eff6c1f1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:46.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.632+0000 7eff62000640 1 --2- 172.21.15.102:0/1767350350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff50003970 0x7eff6c1f1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.632+0000 7eff62000640 1 -- 172.21.15.102:0/1767350350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff6c143a30 con 0x7eff6c1429e0 2024-03-23T03:50:46.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.632+0000 7eff62000640 1 --2- 172.21.15.102:0/1767350350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff6c1429e0 0x7eff6c13eb20 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7eff5400f600 tx=0x7eff5400fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:46.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.632+0000 7eff63400640 1 -- 172.21.15.102:0/1767350350 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff54014870 con 0x7eff6c1429e0 2024-03-23T03:50:46.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.632+0000 7eff63400640 1 -- 172.21.15.102:0/1767350350 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff540149d0 con 0x7eff6c1429e0 2024-03-23T03:50:46.634 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.632+0000 7eff63400640 1 -- 172.21.15.102:0/1767350350 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff54014d40 con 0x7eff6c1429e0 2024-03-23T03:50:46.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.632+0000 7eff71800640 1 -- 172.21.15.102:0/1767350350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff6c143d10 con 0x7eff6c1429e0 2024-03-23T03:50:46.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.633+0000 7eff71800640 1 -- 172.21.15.102:0/1767350350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff6c1442d0 con 0x7eff6c1429e0 2024-03-23T03:50:46.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.634+0000 7eff71800640 1 -- 172.21.15.102:0/1767350350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff6c0709f0 con 0x7eff6c1429e0 2024-03-23T03:50:46.635 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.635+0000 7eff63400640 1 -- 172.21.15.102:0/1767350350 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7eff5400d490 con 0x7eff6c1429e0 2024-03-23T03:50:46.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.635+0000 7eff63400640 1 --2- 172.21.15.102:0/1767350350 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7eff4806ac30 0x7eff4806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:46.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.635+0000 7eff63400640 1 -- 172.21.15.102:0/1767350350 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5177+0+0 (secure 0 0 0) 0x7eff54089ab0 con 0x7eff6c1429e0 2024-03-23T03:50:46.636 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.636+0000 7eff6aa00640 1 --2- 172.21.15.102:0/1767350350 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7eff4806ac30 0x7eff4806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:46.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.636+0000 7eff6aa00640 1 --2- 172.21.15.102:0/1767350350 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7eff4806ac30 0x7eff4806d0f0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7eff5c034000 tx=0x7eff5c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:46.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.640+0000 7eff63400640 1 -- 172.21.15.102:0/1767350350 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7eff54054e90 con 0x7eff6c1429e0 2024-03-23T03:50:46.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.820+0000 7eff71800640 1 -- 172.21.15.102:0/1767350350 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7eff6c073390 con 0x7eff6c1429e0 2024-03-23T03:50:46.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.821+0000 7eff63400640 1 -- 172.21.15.102:0/1767350350 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7eff540584b0 con 0x7eff6c1429e0 2024-03-23T03:50:46.822 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:50:46.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.824+0000 7eff71800640 1 -- 172.21.15.102:0/1767350350 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7eff4806ac30 msgr2=0x7eff4806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:46.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.824+0000 7eff71800640 1 --2- 172.21.15.102:0/1767350350 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7eff4806ac30 0x7eff4806d0f0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7eff5c034000 tx=0x7eff5c009030 comp rx=0 tx=0).stop 2024-03-23T03:50:46.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.824+0000 7eff71800640 1 -- 172.21.15.102:0/1767350350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff6c1429e0 msgr2=0x7eff6c13eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:46.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.824+0000 7eff71800640 1 --2- 172.21.15.102:0/1767350350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff6c1429e0 0x7eff6c13eb20 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7eff5400f600 tx=0x7eff5400fac0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.824+0000 7eff71800640 1 -- 172.21.15.102:0/1767350350 shutdown_connections 2024-03-23T03:50:46.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.824+0000 7eff71800640 1 --2- 172.21.15.102:0/1767350350 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7eff4806ac30 0x7eff4806d0f0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.824+0000 7eff71800640 1 --2- 172.21.15.102:0/1767350350 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7eff6c142e40 0x7eff6c13f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.824+0000 7eff71800640 1 --2- 172.21.15.102:0/1767350350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff6c1429e0 0x7eff6c13eb20 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.824+0000 7eff71800640 1 --2- 172.21.15.102:0/1767350350 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7eff50003970 0x7eff6c1f1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:46.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.824+0000 7eff71800640 1 -- 172.21.15.102:0/1767350350 >> 172.21.15.102:0/1767350350 conn(0x7eff6c06a0b0 msgr2=0x7eff6c1445a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:46.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.824+0000 7eff71800640 1 -- 172.21.15.102:0/1767350350 shutdown_connections 2024-03-23T03:50:46.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:46.824+0000 7eff71800640 1 -- 172.21.15.102:0/1767350350 wait complete. 2024-03-23T03:50:46.843 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T03:50:47.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.040+0000 7f055e800640 1 Processor -- start 2024-03-23T03:50:47.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.040+0000 7f055e800640 1 -- start start 2024-03-23T03:50:47.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.041+0000 7f055e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f055815c690 0x7f05580d0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:47.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.041+0000 7f055e800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f055815ca70 0x7f05580cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:47.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.042+0000 7f055e800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05580d1760 con 0x7f05580d2e10 2024-03-23T03:50:47.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.042+0000 7f055e800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05580d0b50 con 0x7f055815c690 2024-03-23T03:50:47.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.042+0000 7f055e800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05580d0e30 con 0x7f055815ca70 2024-03-23T03:50:47.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.042+0000 7f055d400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f055815c690 0x7f05580d0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:47.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.042+0000 7f055d400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f055815c690 0x7f05580d0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44772/0 (socket says 172.21.15.102:44772) 2024-03-23T03:50:47.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.042+0000 7f055d400640 1 -- 172.21.15.102:0/1941707593 learned_addr learned my addr 172.21.15.102:0/1941707593 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:47.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.042+0000 7f055de00640 1 --1- 172.21.15.102:0/1941707593 >> v1:172.21.15.102:6789/0 conn(0x7f05580d2e10 0x7f05580b41b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56792/0 (socket says 172.21.15.102:56792) 2024-03-23T03:50:47.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.042+0000 7f055ca00640 1 --2- 172.21.15.102:0/1941707593 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f055815ca70 0x7f05580cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:47.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.042+0000 7f055d400640 1 -- 172.21.15.102:0/1941707593 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f055815ca70 msgr2=0x7f05580cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:47.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.042+0000 7f055d400640 1 --2- 172.21.15.102:0/1941707593 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f055815ca70 0x7f05580cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.042+0000 7f055d400640 1 -- 172.21.15.102:0/1941707593 >> v1:172.21.15.102:6789/0 conn(0x7f05580d2e10 legacy=0x7f05580b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:47.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.042+0000 7f055d400640 1 -- 172.21.15.102:0/1941707593 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05580d1110 con 0x7f055815c690 2024-03-23T03:50:47.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.042+0000 7f055ca00640 1 --2- 172.21.15.102:0/1941707593 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f055815ca70 0x7f05580cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:47.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.044+0000 7f055d400640 1 --2- 172.21.15.102:0/1941707593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f055815c690 0x7f05580d0300 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f054c009870 tx=0x7f054c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29fe8b5564e1b597 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:47.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.044+0000 7f0557e00640 1 -- 172.21.15.102:0/1941707593 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c030840 con 0x7f055815c690 2024-03-23T03:50:47.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.044+0000 7f0557e00640 1 -- 172.21.15.102:0/1941707593 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054c0309a0 con 0x7f055815c690 2024-03-23T03:50:47.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.044+0000 7f0557e00640 1 -- 172.21.15.102:0/1941707593 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c030d30 con 0x7f055815c690 2024-03-23T03:50:47.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.044+0000 7f055e800640 1 -- 172.21.15.102:0/1941707593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f055815c690 msgr2=0x7f05580d0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:47.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.044+0000 7f055e800640 1 --2- 172.21.15.102:0/1941707593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f055815c690 0x7f05580d0300 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f054c009870 tx=0x7f054c02b980 comp rx=0 tx=0).stop 2024-03-23T03:50:47.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.045+0000 7f055e800640 1 -- 172.21.15.102:0/1941707593 shutdown_connections 2024-03-23T03:50:47.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.045+0000 7f055e800640 1 --2- 172.21.15.102:0/1941707593 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f055815ca70 0x7f05580cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.045+0000 7f055e800640 1 --2- 172.21.15.102:0/1941707593 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f055815c690 0x7f05580d0300 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.045+0000 7f055e800640 1 -- 172.21.15.102:0/1941707593 >> 172.21.15.102:0/1941707593 conn(0x7f05580717b0 msgr2=0x7f0558071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:47.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.045+0000 7f055e800640 1 -- 172.21.15.102:0/1941707593 shutdown_connections 2024-03-23T03:50:47.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.045+0000 7f055e800640 1 -- 172.21.15.102:0/1941707593 wait complete. 2024-03-23T03:50:47.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.046+0000 7f055e800640 1 Processor -- start 2024-03-23T03:50:47.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.047+0000 7f055e800640 1 -- start start 2024-03-23T03:50:47.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.047+0000 7f055e800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05580d2e10 0x7f055814e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:47.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.047+0000 7f055e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f055815c690 0x7f05580b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.047+0000 7f055e800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f055815ca70 0x7f055815d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.047+0000 7f055e800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05581536d0 con 0x7f055815c690 2024-03-23T03:50:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.047+0000 7f055e800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05580b2ae0 con 0x7f055815ca70 2024-03-23T03:50:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.047+0000 7f055e800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05580b2dc0 con 0x7f05580d2e10 2024-03-23T03:50:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.048+0000 7f055ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f055815c690 0x7f05580b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.048+0000 7f055ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f055815c690 0x7f05580b2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44782/0 (socket says 172.21.15.102:44782) 2024-03-23T03:50:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.048+0000 7f055ca00640 1 -- 172.21.15.102:0/3997013017 learned_addr learned my addr 172.21.15.102:0/3997013017 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.048+0000 7f055d400640 1 --2- 172.21.15.102:0/3997013017 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05580d2e10 0x7f055814e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.048+0000 7f055ca00640 1 -- 172.21.15.102:0/3997013017 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05580d2e10 msgr2=0x7f055814e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.048+0000 7f055ca00640 1 --2- 172.21.15.102:0/3997013017 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05580d2e10 0x7f055814e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.048+0000 7f055ca00640 1 -- 172.21.15.102:0/3997013017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f055815ca70 msgr2=0x7f055815d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.048+0000 7f055ca00640 1 --2- 172.21.15.102:0/3997013017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f055815ca70 0x7f055815d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.048+0000 7f055ca00640 1 -- 172.21.15.102:0/3997013017 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c0094d0 con 0x7f055815c690 2024-03-23T03:50:47.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.049+0000 7f055ca00640 1 --2- 172.21.15.102:0/3997013017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f055815c690 0x7f05580b2190 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f054400c7f0 tx=0x7f054400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:47.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.049+0000 7f0555600640 1 -- 172.21.15.102:0/3997013017 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0544007a30 con 0x7f055815c690 2024-03-23T03:50:47.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.049+0000 7f0555600640 1 -- 172.21.15.102:0/3997013017 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0544007b90 con 0x7f055815c690 2024-03-23T03:50:47.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.049+0000 7f055e800640 1 -- 172.21.15.102:0/3997013017 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05580b30a0 con 0x7f055815c690 2024-03-23T03:50:47.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.049+0000 7f0555600640 1 -- 172.21.15.102:0/3997013017 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054400ce40 con 0x7f055815c690 2024-03-23T03:50:47.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.049+0000 7f055e800640 1 -- 172.21.15.102:0/3997013017 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05580b3660 con 0x7f055815c690 2024-03-23T03:50:47.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.050+0000 7f055e800640 1 -- 172.21.15.102:0/3997013017 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f055804e030 con 0x7f055815c690 2024-03-23T03:50:47.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.051+0000 7f0555600640 1 -- 172.21.15.102:0/3997013017 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f05440113f0 con 0x7f055815c690 2024-03-23T03:50:47.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.052+0000 7f0555600640 1 --2- 172.21.15.102:0/3997013017 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f053806ac00 0x7f053806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:47.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.052+0000 7f0555600640 1 -- 172.21.15.102:0/3997013017 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f05440893a0 con 0x7f055815c690 2024-03-23T03:50:47.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.052+0000 7f055d400640 1 --2- 172.21.15.102:0/3997013017 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f053806ac00 0x7f053806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:47.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.053+0000 7f055d400640 1 --2- 172.21.15.102:0/3997013017 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f053806ac00 0x7f053806d0c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f054c036000 tx=0x7f054c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:47.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.056+0000 7f0555600640 1 -- 172.21.15.102:0/3997013017 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f05440546f0 con 0x7f055815c690 2024-03-23T03:50:47.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.252+0000 7f055e800640 1 -- 172.21.15.102:0/3997013017 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f055815a2a0 con 0x7f055815c690 2024-03-23T03:50:47.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.253+0000 7f0555600640 1 -- 172.21.15.102:0/3997013017 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v89) v1 ==== 84+0+65 (secure 0 0 0) 0x7f054401f060 con 0x7f055815c690 2024-03-23T03:50:47.253 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:47.254 INFO:teuthology.orchestra.run.smithi102.stdout:[ 2024-03-23T03:50:47.254 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:50:47.254 INFO:teuthology.orchestra.run.smithi102.stdout: "poolnum": 1, 2024-03-23T03:50:47.254 INFO:teuthology.orchestra.run.smithi102.stdout: "poolname": ".mgr" 2024-03-23T03:50:47.254 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:50:47.254 INFO:teuthology.orchestra.run.smithi102.stdout:] 2024-03-23T03:50:47.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.255+0000 7f055e800640 1 -- 172.21.15.102:0/3997013017 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f053806ac00 msgr2=0x7f053806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:47.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.255+0000 7f055e800640 1 --2- 172.21.15.102:0/3997013017 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f053806ac00 0x7f053806d0c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f054c036000 tx=0x7f054c008030 comp rx=0 tx=0).stop 2024-03-23T03:50:47.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.256+0000 7f055e800640 1 -- 172.21.15.102:0/3997013017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f055815c690 msgr2=0x7f05580b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:47.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.256+0000 7f055e800640 1 --2- 172.21.15.102:0/3997013017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f055815c690 0x7f05580b2190 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f054400c7f0 tx=0x7f054400cbf0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.256+0000 7f055e800640 1 -- 172.21.15.102:0/3997013017 shutdown_connections 2024-03-23T03:50:47.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.256+0000 7f055e800640 1 --2- 172.21.15.102:0/3997013017 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f053806ac00 0x7f053806d0c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.256+0000 7f055e800640 1 --2- 172.21.15.102:0/3997013017 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f055815ca70 0x7f055815d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.256+0000 7f055e800640 1 --2- 172.21.15.102:0/3997013017 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f055815c690 0x7f05580b2190 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.256+0000 7f055e800640 1 --2- 172.21.15.102:0/3997013017 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05580d2e10 0x7f055814e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.256+0000 7f055e800640 1 -- 172.21.15.102:0/3997013017 >> 172.21.15.102:0/3997013017 conn(0x7f05580717b0 msgr2=0x7f05580c6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:47.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.256+0000 7f055e800640 1 -- 172.21.15.102:0/3997013017 shutdown_connections 2024-03-23T03:50:47.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.256+0000 7f055e800640 1 -- 172.21.15.102:0/3997013017 wait complete. 2024-03-23T03:50:47.275 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T03:50:47.275 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T03:50:47.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.489+0000 7ffbc3000640 1 Processor -- start 2024-03-23T03:50:47.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.489+0000 7ffbc3000640 1 -- start start 2024-03-23T03:50:47.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.491+0000 7ffbc3000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ffbbc15a850 0x7ffbbc06aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:47.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.491+0000 7ffbc3000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ffbbc06d7b0 0x7ffbbc06b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:47.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.491+0000 7ffbc3000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbbc06f430 con 0x7ffbbc0659d0 2024-03-23T03:50:47.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.491+0000 7ffbc3000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbbc15be60 con 0x7ffbbc06d7b0 2024-03-23T03:50:47.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.491+0000 7ffbc3000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbbc15b0a0 con 0x7ffbbc15a850 2024-03-23T03:50:47.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.491+0000 7ffbc1000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ffbbc0659d0 0x7ffbbc15e510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56796/0 (socket says 172.21.15.102:56796) 2024-03-23T03:50:47.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.491+0000 7ffbc1000640 1 -- 172.21.15.102:0/431920242 learned_addr learned my addr 172.21.15.102:0/431920242 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:47.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.492+0000 7ffbbbe00640 1 --2- 172.21.15.102:0/431920242 >> v2:172.21.15.173:3300/0 conn(0x7ffbbc15a850 0x7ffbbc06aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:47.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.492+0000 7ffbbb400640 1 --2- 172.21.15.102:0/431920242 >> v2:172.21.15.102:3300/0 conn(0x7ffbbc06d7b0 0x7ffbbc06b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:47.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.492+0000 7ffbbb400640 1 -- 172.21.15.102:0/431920242 >> v2:172.21.15.173:3300/0 conn(0x7ffbbc15a850 msgr2=0x7ffbbc06aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:47.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.492+0000 7ffbbb400640 1 --2- 172.21.15.102:0/431920242 >> v2:172.21.15.173:3300/0 conn(0x7ffbbc15a850 0x7ffbbc06aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.492+0000 7ffbbb400640 1 -- 172.21.15.102:0/431920242 >> v1:172.21.15.102:6789/0 conn(0x7ffbbc0659d0 legacy=0x7ffbbc15e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:47.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.492+0000 7ffbbb400640 1 -- 172.21.15.102:0/431920242 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbbc15b380 con 0x7ffbbc06d7b0 2024-03-23T03:50:47.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.493+0000 7ffbbb400640 1 --2- 172.21.15.102:0/431920242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbbc06d7b0 0x7ffbbc06b1e0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc0ae240 tx=0x7ffba802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55ee9acbf067e5d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:47.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.494+0000 7ffbbaa00640 1 -- 172.21.15.102:0/431920242 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba8031830 con 0x7ffbbc06d7b0 2024-03-23T03:50:47.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.494+0000 7ffbbaa00640 1 -- 172.21.15.102:0/431920242 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba8031990 con 0x7ffbbc06d7b0 2024-03-23T03:50:47.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.494+0000 7ffbbaa00640 1 -- 172.21.15.102:0/431920242 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba8031d20 con 0x7ffbbc06d7b0 2024-03-23T03:50:47.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.494+0000 7ffbc3000640 1 -- 172.21.15.102:0/431920242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbbc06d7b0 msgr2=0x7ffbbc06b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:47.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.494+0000 7ffbc3000640 1 --2- 172.21.15.102:0/431920242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbbc06d7b0 0x7ffbbc06b1e0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ffbbc0ae240 tx=0x7ffba802b8d0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.494+0000 7ffbc3000640 1 -- 172.21.15.102:0/431920242 shutdown_connections 2024-03-23T03:50:47.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.494+0000 7ffbc3000640 1 --2- 172.21.15.102:0/431920242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbbc06d7b0 0x7ffbbc06b1e0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.494+0000 7ffbc3000640 1 --2- 172.21.15.102:0/431920242 >> v2:172.21.15.173:3300/0 conn(0x7ffbbc15a850 0x7ffbbc06aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.494+0000 7ffbc3000640 1 -- 172.21.15.102:0/431920242 >> 172.21.15.102:0/431920242 conn(0x7ffbbc070a20 msgr2=0x7ffbbc070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:47.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.494+0000 7ffbc3000640 1 -- 172.21.15.102:0/431920242 shutdown_connections 2024-03-23T03:50:47.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.494+0000 7ffbc3000640 1 -- 172.21.15.102:0/431920242 wait complete. 2024-03-23T03:50:47.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.496+0000 7ffbc3000640 1 Processor -- start 2024-03-23T03:50:47.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.496+0000 7ffbc3000640 1 -- start start 2024-03-23T03:50:47.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.497+0000 7ffbc3000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbbc0659d0 0x7ffbbc0cdf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:47.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.497+0000 7ffbc3000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbbc06d7b0 0x7ffbbc0cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:47.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.497+0000 7ffbc3000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffbbc15a850 0x7ffbbc151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:47.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.497+0000 7ffbc3000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc154960 con 0x7ffbbc0659d0 2024-03-23T03:50:47.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.497+0000 7ffbc3000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc154c40 con 0x7ffbbc06d7b0 2024-03-23T03:50:47.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.497+0000 7ffbc3000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc154f20 con 0x7ffbbc15a850 2024-03-23T03:50:47.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.497+0000 7ffbbbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbbc0659d0 0x7ffbbc0cdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:47.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.497+0000 7ffbbbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbbc0659d0 0x7ffbbc0cdf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44802/0 (socket says 172.21.15.102:44802) 2024-03-23T03:50:47.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.497+0000 7ffbbbe00640 1 -- 172.21.15.102:0/1685910411 learned_addr learned my addr 172.21.15.102:0/1685910411 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:47.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.498+0000 7ffbbb400640 1 --2- 172.21.15.102:0/1685910411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbbc06d7b0 0x7ffbbc0cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:47.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.498+0000 7ffbbbe00640 1 -- 172.21.15.102:0/1685910411 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffbbc15a850 msgr2=0x7ffbbc151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:47.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.498+0000 7ffbbbe00640 1 --2- 172.21.15.102:0/1685910411 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffbbc15a850 0x7ffbbc151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.498+0000 7ffbbbe00640 1 -- 172.21.15.102:0/1685910411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbbc06d7b0 msgr2=0x7ffbbc0cb720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:47.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.498+0000 7ffbbbe00640 1 --2- 172.21.15.102:0/1685910411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbbc06d7b0 0x7ffbbc0cb720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.498+0000 7ffbbbe00640 1 -- 172.21.15.102:0/1685910411 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba8009400 con 0x7ffbbc0659d0 2024-03-23T03:50:47.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.498+0000 7ffbbb400640 1 --2- 172.21.15.102:0/1685910411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbbc06d7b0 0x7ffbbc0cb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:47.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.498+0000 7ffbbbe00640 1 --2- 172.21.15.102:0/1685910411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbbc0659d0 0x7ffbbc0cdf70 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ffbac00e9b0 tx=0x7ffbac00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:47.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.498+0000 7ffbb3e00640 1 -- 172.21.15.102:0/1685910411 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac009670 con 0x7ffbbc0659d0 2024-03-23T03:50:47.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.498+0000 7ffbb3e00640 1 -- 172.21.15.102:0/1685910411 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbac00c9a0 con 0x7ffbbc0659d0 2024-03-23T03:50:47.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.498+0000 7ffbc3000640 1 -- 172.21.15.102:0/1685910411 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbbc155200 con 0x7ffbbc0659d0 2024-03-23T03:50:47.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.498+0000 7ffbb3e00640 1 -- 172.21.15.102:0/1685910411 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbac00cd50 con 0x7ffbbc0659d0 2024-03-23T03:50:47.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.498+0000 7ffbc3000640 1 -- 172.21.15.102:0/1685910411 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbbc1f0480 con 0x7ffbbc0659d0 2024-03-23T03:50:47.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.499+0000 7ffbc3000640 1 -- 172.21.15.102:0/1685910411 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbbc066700 con 0x7ffbbc0659d0 2024-03-23T03:50:47.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.501+0000 7ffbb3e00640 1 -- 172.21.15.102:0/1685910411 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ffbac013470 con 0x7ffbbc0659d0 2024-03-23T03:50:47.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.501+0000 7ffbb3e00640 1 --2- 172.21.15.102:0/1685910411 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb9c06ac00 0x7ffb9c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:47.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.502+0000 7ffbb3e00640 1 -- 172.21.15.102:0/1685910411 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5177+0+0 (secure 0 0 0) 0x7ffbac0893e0 con 0x7ffbbc0659d0 2024-03-23T03:50:47.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.502+0000 7ffbbb400640 1 --2- 172.21.15.102:0/1685910411 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb9c06ac00 0x7ffb9c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:47.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.502+0000 7ffbbb400640 1 --2- 172.21.15.102:0/1685910411 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb9c06ac00 0x7ffb9c06d0c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ffba8036000 tx=0x7ffba8008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:47.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.506+0000 7ffbb3e00640 1 -- 172.21.15.102:0/1685910411 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffbac0547d0 con 0x7ffbbc0659d0 2024-03-23T03:50:47.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.697+0000 7ffbc3000640 1 -- 172.21.15.102:0/1685910411 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7ffbbc066bb0 con 0x7ffbbc0659d0 2024-03-23T03:50:47.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.809+0000 7ffbb3e00640 1 -- 172.21.15.102:0/1685910411 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v90) v1 ==== 138+0+0 (secure 0 0 0) 0x7ffbac057df0 con 0x7ffbbc0659d0 2024-03-23T03:50:47.810 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' created 2024-03-23T03:50:47.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.813+0000 7ffbc3000640 1 -- 172.21.15.102:0/1685910411 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb9c06ac00 msgr2=0x7ffb9c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:47.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.813+0000 7ffbc3000640 1 --2- 172.21.15.102:0/1685910411 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb9c06ac00 0x7ffb9c06d0c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ffba8036000 tx=0x7ffba8008030 comp rx=0 tx=0).stop 2024-03-23T03:50:47.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.813+0000 7ffbc3000640 1 -- 172.21.15.102:0/1685910411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbbc0659d0 msgr2=0x7ffbbc0cdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:47.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.813+0000 7ffbc3000640 1 --2- 172.21.15.102:0/1685910411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbbc0659d0 0x7ffbbc0cdf70 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ffbac00e9b0 tx=0x7ffbac00ee70 comp rx=0 tx=0).stop 2024-03-23T03:50:47.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.813+0000 7ffbc3000640 1 -- 172.21.15.102:0/1685910411 shutdown_connections 2024-03-23T03:50:47.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.813+0000 7ffbc3000640 1 --2- 172.21.15.102:0/1685910411 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffb9c06ac00 0x7ffb9c06d0c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.813+0000 7ffbc3000640 1 --2- 172.21.15.102:0/1685910411 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffbbc15a850 0x7ffbbc151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.814 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.813+0000 7ffbc3000640 1 --2- 172.21.15.102:0/1685910411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffbbc06d7b0 0x7ffbbc0cb720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.813+0000 7ffbc3000640 1 --2- 172.21.15.102:0/1685910411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbbc0659d0 0x7ffbbc0cdf70 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:47.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.813+0000 7ffbc3000640 1 -- 172.21.15.102:0/1685910411 >> 172.21.15.102:0/1685910411 conn(0x7ffbbc070a20 msgr2=0x7ffbbc152120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:47.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.813+0000 7ffbc3000640 1 -- 172.21.15.102:0/1685910411 shutdown_connections 2024-03-23T03:50:47.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:47.814+0000 7ffbc3000640 1 -- 172.21.15.102:0/1685910411 wait complete. 2024-03-23T03:50:47.836 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T03:50:48.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.042+0000 7f0cfd800640 1 Processor -- start 2024-03-23T03:50:48.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.043+0000 7f0cfd800640 1 -- start start 2024-03-23T03:50:48.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.044+0000 7f0cfd800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f0cf80ca7b0 0x7f0cf80cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:48.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.044+0000 7f0cfd800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf8159c10 con 0x7f0cf80ce930 2024-03-23T03:50:48.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.044+0000 7f0cfd800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf80cd840 con 0x7f0cf8159830 2024-03-23T03:50:48.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.044+0000 7f0cfd800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf806de50 con 0x7f0cf80ca7b0 2024-03-23T03:50:48.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.045+0000 7f0cf7400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0cf80ce930 0x7f0cf80b18d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56812/0 (socket says 172.21.15.102:56812) 2024-03-23T03:50:48.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.045+0000 7f0cf7400640 1 -- 172.21.15.102:0/1065321423 learned_addr learned my addr 172.21.15.102:0/1065321423 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:48.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.045+0000 7f0cf6000640 1 --2- 172.21.15.102:0/1065321423 >> v2:172.21.15.173:3300/0 conn(0x7f0cf80ca7b0 0x7f0cf80cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:48.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.045+0000 7f0cf5600640 1 -- 172.21.15.102:0/1065321423 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655416384 0 0) 0x7f0cf8159c10 con 0x7f0cf80ce930 2024-03-23T03:50:48.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.045+0000 7f0cf5600640 1 -- 172.21.15.102:0/1065321423 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cdc003520 con 0x7f0cf80ce930 2024-03-23T03:50:48.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.045+0000 7f0cf5600640 1 -- 172.21.15.102:0/1065321423 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4059106970 0 0) 0x7f0cdc003520 con 0x7f0cf80ce930 2024-03-23T03:50:48.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.045+0000 7f0cf5600640 1 -- 172.21.15.102:0/1065321423 >> v2:172.21.15.173:3300/0 conn(0x7f0cf80ca7b0 msgr2=0x7f0cf80cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:48.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.045+0000 7f0cf5600640 1 --2- 172.21.15.102:0/1065321423 >> v2:172.21.15.173:3300/0 conn(0x7f0cf80ca7b0 0x7f0cf80cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:48.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.045+0000 7f0cf5600640 1 -- 172.21.15.102:0/1065321423 >> v1:172.21.15.173:6789/0 conn(0x7f0cf8159830 legacy=0x7f0cf80cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:48.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.045+0000 7f0cf5600640 1 -- 172.21.15.102:0/1065321423 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf80ccc30 con 0x7f0cf80ce930 2024-03-23T03:50:48.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.045+0000 7f0cf5600640 1 -- 172.21.15.102:0/1065321423 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f0cf00019f0 con 0x7f0cf80ce930 2024-03-23T03:50:48.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.045+0000 7f0cf5600640 1 -- 172.21.15.102:0/1065321423 >> v1:172.21.15.102:6789/0 conn(0x7f0cf80ce930 legacy=0x7f0cf80b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:48.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.046+0000 7f0cf5600640 1 --2- 172.21.15.102:0/1065321423 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cdc003970 0x7f0cdc003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:48.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.046+0000 7f0cf5600640 1 --2- 172.21.15.102:0/1065321423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc0047c0 0x7f0cdc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:48.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.046+0000 7f0cf5600640 1 --2- 172.21.15.102:0/1065321423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cdc005180 0x7f0cdc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:48.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.046+0000 7f0cf5600640 1 -- 172.21.15.102:0/1065321423 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc00c060 con 0x7f0cdc0047c0 2024-03-23T03:50:48.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.046+0000 7f0cf5600640 1 -- 172.21.15.102:0/1065321423 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc00c1d0 con 0x7f0cdc005180 2024-03-23T03:50:48.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.046+0000 7f0cf5600640 1 -- 172.21.15.102:0/1065321423 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc00c390 con 0x7f0cdc003970 2024-03-23T03:50:48.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.046+0000 7f0cf6000640 1 --2- 172.21.15.102:0/1065321423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc0047c0 0x7f0cdc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:48.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.046+0000 7f0cf6000640 1 -- 172.21.15.102:0/1065321423 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cdc003970 msgr2=0x7f0cdc003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:48.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.046+0000 7f0cf6a00640 1 --2- 172.21.15.102:0/1065321423 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cdc003970 0x7f0cdc003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:48.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.046+0000 7f0cf7400640 1 --2- 172.21.15.102:0/1065321423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cdc005180 0x7f0cdc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:48.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.046+0000 7f0cf6000640 1 --2- 172.21.15.102:0/1065321423 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cdc003970 0x7f0cdc003d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:48.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.046+0000 7f0cf6000640 1 -- 172.21.15.102:0/1065321423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cdc005180 msgr2=0x7f0cdc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:48.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.046+0000 7f0cf6000640 1 --2- 172.21.15.102:0/1065321423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cdc005180 0x7f0cdc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:48.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.046+0000 7f0cf6000640 1 -- 172.21.15.102:0/1065321423 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc003520 con 0x7f0cdc0047c0 2024-03-23T03:50:48.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.046+0000 7f0cf7400640 1 --2- 172.21.15.102:0/1065321423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cdc005180 0x7f0cdc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:48.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.046+0000 7f0cf7400640 1 -- 172.21.15.102:0/1065321423 reap_dead start 2024-03-23T03:50:48.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.047+0000 7f0cf6000640 1 --2- 172.21.15.102:0/1065321423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc0047c0 0x7f0cdc004c40 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f0ce400ba30 tx=0x7f0cdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54d98bcc596d6c3f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:48.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.048+0000 7f0cf5600640 1 -- 172.21.15.102:0/1065321423 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4032640 con 0x7f0cdc0047c0 2024-03-23T03:50:48.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.048+0000 7f0cf5600640 1 -- 172.21.15.102:0/1065321423 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce40327a0 con 0x7f0cdc0047c0 2024-03-23T03:50:48.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.048+0000 7f0cf5600640 1 -- 172.21.15.102:0/1065321423 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4032b30 con 0x7f0cdc0047c0 2024-03-23T03:50:48.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.048+0000 7f0cfd800640 1 -- 172.21.15.102:0/1065321423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc0047c0 msgr2=0x7f0cdc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:48.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.048+0000 7f0cfd800640 1 --2- 172.21.15.102:0/1065321423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc0047c0 0x7f0cdc004c40 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f0ce400ba30 tx=0x7f0cdc0057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:48.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.048+0000 7f0cfd800640 1 -- 172.21.15.102:0/1065321423 shutdown_connections 2024-03-23T03:50:48.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.048+0000 7f0cfd800640 1 --2- 172.21.15.102:0/1065321423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc0047c0 0x7f0cdc004c40 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:48.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.048+0000 7f0cfd800640 1 -- 172.21.15.102:0/1065321423 >> 172.21.15.102:0/1065321423 conn(0x7f0cf806a0b0 msgr2=0x7f0cf806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:48.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.048+0000 7f0cfd800640 1 -- 172.21.15.102:0/1065321423 shutdown_connections 2024-03-23T03:50:48.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.048+0000 7f0cfd800640 1 -- 172.21.15.102:0/1065321423 wait complete. 2024-03-23T03:50:48.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.050+0000 7f0cfd800640 1 Processor -- start 2024-03-23T03:50:48.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.050+0000 7f0cfd800640 1 -- start start 2024-03-23T03:50:48.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.051+0000 7f0cfd800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cdc0047c0 0x7f0cf81f19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.051+0000 7f0cfd800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cf81e4c40 0x7f0cf81e50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.051+0000 7f0cfd800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cf81e5ac0 0x7f0cf81e5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.051+0000 7f0cfd800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf81e6480 con 0x7f0cf81e5ac0 2024-03-23T03:50:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.051+0000 7f0cfd800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf81e5700 con 0x7f0cf81e4c40 2024-03-23T03:50:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.051+0000 7f0cfd800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf8156bd0 con 0x7f0cdc0047c0 2024-03-23T03:50:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.051+0000 7f0cf7400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cf81e5ac0 0x7f0cf81e5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.051+0000 7f0cf7400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cf81e5ac0 0x7f0cf81e5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44820/0 (socket says 172.21.15.102:44820) 2024-03-23T03:50:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.051+0000 7f0cf7400640 1 -- 172.21.15.102:0/3866265825 learned_addr learned my addr 172.21.15.102:0/3866265825 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.052+0000 7f0cf6000640 1 --2- 172.21.15.102:0/3866265825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cf81e4c40 0x7f0cf81e50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.052+0000 7f0cf6a00640 1 --2- 172.21.15.102:0/3866265825 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cdc0047c0 0x7f0cf81f19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.052+0000 7f0cf7400640 1 -- 172.21.15.102:0/3866265825 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cdc0047c0 msgr2=0x7f0cf81f19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:48.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.052+0000 7f0cf7400640 1 --2- 172.21.15.102:0/3866265825 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cdc0047c0 0x7f0cf81f19d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:48.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.052+0000 7f0cf7400640 1 -- 172.21.15.102:0/3866265825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cf81e4c40 msgr2=0x7f0cf81e50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:48.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.052+0000 7f0cf7400640 1 --2- 172.21.15.102:0/3866265825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cf81e4c40 0x7f0cf81e50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:48.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.052+0000 7f0cf7400640 1 -- 172.21.15.102:0/3866265825 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf8156eb0 con 0x7f0cf81e5ac0 2024-03-23T03:50:48.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.052+0000 7f0cf7400640 1 --2- 172.21.15.102:0/3866265825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cf81e5ac0 0x7f0cf81e5f40 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0000f80 tx=0x7f0cf000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:48.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.052+0000 7f0ceea00640 1 -- 172.21.15.102:0/3866265825 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0004d60 con 0x7f0cf81e5ac0 2024-03-23T03:50:48.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.052+0000 7f0ceea00640 1 -- 172.21.15.102:0/3866265825 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf000bb80 con 0x7f0cf81e5ac0 2024-03-23T03:50:48.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.052+0000 7f0ceea00640 1 -- 172.21.15.102:0/3866265825 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0007070 con 0x7f0cf81e5ac0 2024-03-23T03:50:48.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.052+0000 7f0cfd800640 1 -- 172.21.15.102:0/3866265825 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf8157190 con 0x7f0cf81e5ac0 2024-03-23T03:50:48.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.053+0000 7f0cfd800640 1 -- 172.21.15.102:0/3866265825 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf8149a80 con 0x7f0cf81e5ac0 2024-03-23T03:50:48.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.054+0000 7f0cfd800640 1 -- 172.21.15.102:0/3866265825 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf8073390 con 0x7f0cf81e5ac0 2024-03-23T03:50:48.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.055+0000 7f0ceea00640 1 -- 172.21.15.102:0/3866265825 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0cf00073e0 con 0x7f0cf81e5ac0 2024-03-23T03:50:48.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.055+0000 7f0ceea00640 1 --2- 172.21.15.102:0/3866265825 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0cd806ac00 0x7f0cd806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:48.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.055+0000 7f0ceea00640 1 -- 172.21.15.102:0/3866265825 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f0cf0089940 con 0x7f0cf81e5ac0 2024-03-23T03:50:48.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.056+0000 7f0cf6a00640 1 --2- 172.21.15.102:0/3866265825 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0cd806ac00 0x7f0cd806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:48.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.056+0000 7f0cf6a00640 1 --2- 172.21.15.102:0/3866265825 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0cd806ac00 0x7f0cd806d0c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8003c90 tx=0x7f0ce8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:48.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.060+0000 7f0ceea00640 1 -- 172.21.15.102:0/3866265825 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0cf0054b20 con 0x7f0cf81e5ac0 2024-03-23T03:50:48.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.254+0000 7f0cfd800640 1 -- 172.21.15.102:0/3866265825 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0cf8070f40 con 0x7f0cf81e5ac0 2024-03-23T03:50:48.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.817+0000 7f0ceea00640 1 -- 172.21.15.102:0/3866265825 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v91) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0cf0058140 con 0x7f0cf81e5ac0 2024-03-23T03:50:48.818 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' created 2024-03-23T03:50:48.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.823+0000 7f0cfd800640 1 -- 172.21.15.102:0/3866265825 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0cd806ac00 msgr2=0x7f0cd806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:48.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.823+0000 7f0cfd800640 1 --2- 172.21.15.102:0/3866265825 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0cd806ac00 0x7f0cd806d0c0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f0ce8003c90 tx=0x7f0ce8009040 comp rx=0 tx=0).stop 2024-03-23T03:50:48.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.823+0000 7f0cfd800640 1 -- 172.21.15.102:0/3866265825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cf81e5ac0 msgr2=0x7f0cf81e5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:48.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.823+0000 7f0cfd800640 1 --2- 172.21.15.102:0/3866265825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cf81e5ac0 0x7f0cf81e5f40 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0000f80 tx=0x7f0cf000b4a0 comp rx=0 tx=0).stop 2024-03-23T03:50:48.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.823+0000 7f0cfd800640 1 -- 172.21.15.102:0/3866265825 shutdown_connections 2024-03-23T03:50:48.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.823+0000 7f0cfd800640 1 --2- 172.21.15.102:0/3866265825 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0cd806ac00 0x7f0cd806d0c0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:48.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.823+0000 7f0cfd800640 1 --2- 172.21.15.102:0/3866265825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cf81e5ac0 0x7f0cf81e5f40 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:48.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.823+0000 7f0cfd800640 1 --2- 172.21.15.102:0/3866265825 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0cf81e4c40 0x7f0cf81e50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:48.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.823+0000 7f0cfd800640 1 --2- 172.21.15.102:0/3866265825 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0cdc0047c0 0x7f0cf81f19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:48.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.823+0000 7f0cfd800640 1 -- 172.21.15.102:0/3866265825 >> 172.21.15.102:0/3866265825 conn(0x7f0cf806a0b0 msgr2=0x7f0cf8155fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:48.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.824+0000 7f0cfd800640 1 -- 172.21.15.102:0/3866265825 shutdown_connections 2024-03-23T03:50:48.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:48.824+0000 7f0cfd800640 1 -- 172.21.15.102:0/3866265825 wait complete. 2024-03-23T03:50:48.860 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T03:50:49.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.045+0000 7f965be00640 1 Processor -- start 2024-03-23T03:50:49.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.046+0000 7f965be00640 1 -- start start 2024-03-23T03:50:49.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.048+0000 7f965be00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f96540b84b0 0x7f96540d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.048+0000 7f965be00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96540d1280 con 0x7f96540b5aa0 2024-03-23T03:50:49.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.048+0000 7f965be00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96540d20c0 con 0x7f96540cd740 2024-03-23T03:50:49.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.048+0000 7f965be00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96540b8910 con 0x7f96540b84b0 2024-03-23T03:50:49.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.048+0000 7f9659e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f96540b5aa0 0x7f96540b5f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56826/0 (socket says 172.21.15.102:56826) 2024-03-23T03:50:49.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.048+0000 7f9659e00640 1 -- 172.21.15.102:0/4171243410 learned_addr learned my addr 172.21.15.102:0/4171243410 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:49.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.048+0000 7f9658a00640 1 --2- 172.21.15.102:0/4171243410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f96540b84b0 0x7f96540d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.048+0000 7f9653e00640 1 -- 172.21.15.102:0/4171243410 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510204756 0 0) 0x7f96540d1280 con 0x7f96540b5aa0 2024-03-23T03:50:49.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.048+0000 7f9653e00640 1 -- 172.21.15.102:0/4171243410 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f963c003520 con 0x7f96540b5aa0 2024-03-23T03:50:49.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.049+0000 7f9653e00640 1 -- 172.21.15.102:0/4171243410 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 628398581 0 0) 0x7f963c003520 con 0x7f96540b5aa0 2024-03-23T03:50:49.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.049+0000 7f9653e00640 1 -- 172.21.15.102:0/4171243410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f96540b84b0 msgr2=0x7f96540d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:49.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.049+0000 7f9653e00640 1 --2- 172.21.15.102:0/4171243410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f96540b84b0 0x7f96540d0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.049+0000 7f9653e00640 1 -- 172.21.15.102:0/4171243410 >> v1:172.21.15.173:6789/0 conn(0x7f96540cd740 legacy=0x7f96540cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.049+0000 7f9653e00640 1 -- 172.21.15.102:0/4171243410 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96540d14b0 con 0x7f96540b5aa0 2024-03-23T03:50:49.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.049+0000 7f9653e00640 1 -- 172.21.15.102:0/4171243410 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f964c0019f0 con 0x7f96540b5aa0 2024-03-23T03:50:49.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.049+0000 7f9653e00640 1 -- 172.21.15.102:0/4171243410 >> v1:172.21.15.102:6789/0 conn(0x7f96540b5aa0 legacy=0x7f96540b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.049+0000 7f9653e00640 1 --2- 172.21.15.102:0/4171243410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f963c0039a0 0x7f963c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.049+0000 7f9653e00640 1 --2- 172.21.15.102:0/4171243410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f96540b84b0 0x7f963c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.049+0000 7f9653e00640 1 --2- 172.21.15.102:0/4171243410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963c004d00 0x7f963c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.049+0000 7f9653e00640 1 -- 172.21.15.102:0/4171243410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96540b8910 con 0x7f963c004d00 2024-03-23T03:50:49.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.049+0000 7f9653e00640 1 -- 172.21.15.102:0/4171243410 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963c00ba30 con 0x7f963c0039a0 2024-03-23T03:50:49.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.049+0000 7f9653e00640 1 -- 172.21.15.102:0/4171243410 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963c00bbf0 con 0x7f96540b84b0 2024-03-23T03:50:49.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.049+0000 7f9659e00640 1 --2- 172.21.15.102:0/4171243410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963c004d00 0x7f963c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.050+0000 7f9659400640 1 --2- 172.21.15.102:0/4171243410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f963c0039a0 0x7f963c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.050+0000 7f9658a00640 1 --2- 172.21.15.102:0/4171243410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f96540b84b0 0x7f963c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.050+0000 7f9659e00640 1 -- 172.21.15.102:0/4171243410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f96540b84b0 msgr2=0x7f963c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:49.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.050+0000 7f9659e00640 1 --2- 172.21.15.102:0/4171243410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f96540b84b0 0x7f963c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.050+0000 7f9659e00640 1 -- 172.21.15.102:0/4171243410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f963c0039a0 msgr2=0x7f963c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:49.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.050+0000 7f9659e00640 1 --2- 172.21.15.102:0/4171243410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f963c0039a0 0x7f963c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.050+0000 7f9659e00640 1 -- 172.21.15.102:0/4171243410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c003520 con 0x7f963c004d00 2024-03-23T03:50:49.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.051+0000 7f9659e00640 1 --2- 172.21.15.102:0/4171243410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963c004d00 0x7f963c005180 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f964c00a800 tx=0x7f963c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19194a57325a9f31 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:49.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.051+0000 7f9653e00640 1 -- 172.21.15.102:0/4171243410 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c0072d0 con 0x7f963c004d00 2024-03-23T03:50:49.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.051+0000 7f9653e00640 1 -- 172.21.15.102:0/4171243410 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f964c02f070 con 0x7f963c004d00 2024-03-23T03:50:49.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.051+0000 7f9653e00640 1 -- 172.21.15.102:0/4171243410 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964c0076b0 con 0x7f963c004d00 2024-03-23T03:50:49.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.052+0000 7f965be00640 1 -- 172.21.15.102:0/4171243410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963c004d00 msgr2=0x7f963c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.052+0000 7f965be00640 1 --2- 172.21.15.102:0/4171243410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963c004d00 0x7f963c005180 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f964c00a800 tx=0x7f963c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.052+0000 7f9659e00640 1 -- 172.21.15.102:0/4171243410 reap_dead start 2024-03-23T03:50:49.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.052+0000 7f965be00640 1 -- 172.21.15.102:0/4171243410 shutdown_connections 2024-03-23T03:50:49.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.052+0000 7f965be00640 1 -- 172.21.15.102:0/4171243410 >> 172.21.15.102:0/4171243410 conn(0x7f96540cf5d0 msgr2=0x7f96540cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:49.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.052+0000 7f965be00640 1 -- 172.21.15.102:0/4171243410 shutdown_connections 2024-03-23T03:50:49.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.052+0000 7f965be00640 1 -- 172.21.15.102:0/4171243410 wait complete. 2024-03-23T03:50:49.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.054+0000 7f965be00640 1 Processor -- start 2024-03-23T03:50:49.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.054+0000 7f965be00640 1 -- start start 2024-03-23T03:50:49.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.055+0000 7f965be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963c004d00 0x7f96541582c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.055+0000 7f965be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96541534d0 0x7f96541599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.055+0000 7f965be00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f96540c9c60 0x7f96540b3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.055+0000 7f965be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9654159f10 con 0x7f963c004d00 2024-03-23T03:50:49.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.055+0000 7f965be00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9654153950 con 0x7f96541534d0 2024-03-23T03:50:49.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.055+0000 7f965be00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9654158930 con 0x7f96540c9c60 2024-03-23T03:50:49.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.055+0000 7f9659400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963c004d00 0x7f96541582c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.055+0000 7f9659400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963c004d00 0x7f96541582c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44832/0 (socket says 172.21.15.102:44832) 2024-03-23T03:50:49.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.055+0000 7f9659400640 1 -- 172.21.15.102:0/1502584052 learned_addr learned my addr 172.21.15.102:0/1502584052 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:49.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.055+0000 7f9659400640 1 -- 172.21.15.102:0/1502584052 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f96540c9c60 msgr2=0x7f96540b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.055+0000 7f9659400640 1 --2- 172.21.15.102:0/1502584052 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f96540c9c60 0x7f96540b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.055+0000 7f9659400640 1 -- 172.21.15.102:0/1502584052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96541534d0 msgr2=0x7f96541599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.055+0000 7f9659400640 1 --2- 172.21.15.102:0/1502584052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96541534d0 0x7f96541599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.055+0000 7f9659400640 1 -- 172.21.15.102:0/1502584052 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9654158c10 con 0x7f963c004d00 2024-03-23T03:50:49.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.056+0000 7f9659400640 1 --2- 172.21.15.102:0/1502584052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963c004d00 0x7f96541582c0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f964800e550 tx=0x7f964800ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:49.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.056+0000 7f9651600640 1 -- 172.21.15.102:0/1502584052 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96480052e0 con 0x7f963c004d00 2024-03-23T03:50:49.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.056+0000 7f9651600640 1 -- 172.21.15.102:0/1502584052 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9648005440 con 0x7f963c004d00 2024-03-23T03:50:49.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.056+0000 7f965be00640 1 -- 172.21.15.102:0/1502584052 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9654158ef0 con 0x7f963c004d00 2024-03-23T03:50:49.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.056+0000 7f9651600640 1 -- 172.21.15.102:0/1502584052 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9648003810 con 0x7f963c004d00 2024-03-23T03:50:49.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.056+0000 7f965be00640 1 -- 172.21.15.102:0/1502584052 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96541594b0 con 0x7f963c004d00 2024-03-23T03:50:49.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.057+0000 7f965be00640 1 -- 172.21.15.102:0/1502584052 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9654159640 con 0x7f963c004d00 2024-03-23T03:50:49.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.058+0000 7f9651600640 1 -- 172.21.15.102:0/1502584052 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9648003970 con 0x7f963c004d00 2024-03-23T03:50:49.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.059+0000 7f9651600640 1 --2- 172.21.15.102:0/1502584052 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f963406ac00 0x7f963406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.059+0000 7f9651600640 1 -- 172.21.15.102:0/1502584052 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f9648089150 con 0x7f963c004d00 2024-03-23T03:50:49.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.060+0000 7f9658a00640 1 --2- 172.21.15.102:0/1502584052 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f963406ac00 0x7f963406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.060+0000 7f9658a00640 1 --2- 172.21.15.102:0/1502584052 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f963406ac00 0x7f963406d0c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f96540b8220 tx=0x7f9640008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:49.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.065+0000 7f9651600640 1 -- 172.21.15.102:0/1502584052 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9648054240 con 0x7f963c004d00 2024-03-23T03:50:49.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.293+0000 7f965be00640 1 -- 172.21.15.102:0/1502584052 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f9654073390 con 0x7f963c004d00 2024-03-23T03:50:49.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.303+0000 7f9651600640 1 -- 172.21.15.102:0/1502584052 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v45) v1 ==== 174+0+0 (secure 0 0 0) 0x7f9648057860 con 0x7f963c004d00 2024-03-23T03:50:49.305 INFO:teuthology.orchestra.run.smithi102.stderr:new fs with metadata pool 13 and data pool 14 2024-03-23T03:50:49.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.307+0000 7f965be00640 1 -- 172.21.15.102:0/1502584052 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f963406ac00 msgr2=0x7f963406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.307+0000 7f965be00640 1 --2- 172.21.15.102:0/1502584052 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f963406ac00 0x7f963406d0c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f96540b8220 tx=0x7f9640008040 comp rx=0 tx=0).stop 2024-03-23T03:50:49.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.307+0000 7f965be00640 1 -- 172.21.15.102:0/1502584052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963c004d00 msgr2=0x7f96541582c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.307+0000 7f965be00640 1 --2- 172.21.15.102:0/1502584052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963c004d00 0x7f96541582c0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f964800e550 tx=0x7f964800ea10 comp rx=0 tx=0).stop 2024-03-23T03:50:49.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.307+0000 7f965be00640 1 -- 172.21.15.102:0/1502584052 shutdown_connections 2024-03-23T03:50:49.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.307+0000 7f965be00640 1 --2- 172.21.15.102:0/1502584052 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f963406ac00 0x7f963406d0c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.307+0000 7f965be00640 1 --2- 172.21.15.102:0/1502584052 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f96540c9c60 0x7f96540b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.307+0000 7f965be00640 1 --2- 172.21.15.102:0/1502584052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96541534d0 0x7f96541599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.307+0000 7f965be00640 1 --2- 172.21.15.102:0/1502584052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f963c004d00 0x7f96541582c0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.308 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.307+0000 7f965be00640 1 -- 172.21.15.102:0/1502584052 >> 172.21.15.102:0/1502584052 conn(0x7f96540cf5d0 msgr2=0x7f96540b6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:49.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.308+0000 7f965be00640 1 -- 172.21.15.102:0/1502584052 shutdown_connections 2024-03-23T03:50:49.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.308+0000 7f965be00640 1 -- 172.21.15.102:0/1502584052 wait complete. 2024-03-23T03:50:49.339 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:50:49.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.515+0000 7f5cfb200640 1 Processor -- start 2024-03-23T03:50:49.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.515+0000 7f5cfb200640 1 -- start start 2024-03-23T03:50:49.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.517+0000 7f5cfb200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5cf4143460 0x7f5cf415aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.517+0000 7f5cfb200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f5cf4143930 0x7f5cf40cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.517 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.517+0000 7f5cfb200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cf40cc6b0 con 0x7f5cf40cdc50 2024-03-23T03:50:49.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.517+0000 7f5cfb200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5cf40cc950 con 0x7f5cf4143930 2024-03-23T03:50:49.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.517+0000 7f5cfb200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf40ccc30 con 0x7f5cf4143460 2024-03-23T03:50:49.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.517+0000 7f5cfa800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5cf40cdc50 0x7f5cf415e1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56834/0 (socket says 172.21.15.102:56834) 2024-03-23T03:50:49.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.517+0000 7f5cfa800640 1 -- 172.21.15.102:0/1585808762 learned_addr learned my addr 172.21.15.102:0/1585808762 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.517+0000 7f5cf9400640 1 --2- 172.21.15.102:0/1585808762 >> v2:172.21.15.173:3300/0 conn(0x7f5cf4143930 0x7f5cf40cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.518+0000 7f5cf9e00640 1 --2- 172.21.15.102:0/1585808762 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5cf4143460 0x7f5cf415aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.518+0000 7f5cf8a00640 1 -- 172.21.15.102:0/1585808762 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592806335 0 0) 0x7f5cf40cc6b0 con 0x7f5cf40cdc50 2024-03-23T03:50:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.518+0000 7f5cf8a00640 1 -- 172.21.15.102:0/1585808762 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cdc003520 con 0x7f5cf40cdc50 2024-03-23T03:50:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.518+0000 7f5cf8a00640 1 -- 172.21.15.102:0/1585808762 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1959168998 0 0) 0x7f5cdc003520 con 0x7f5cf40cdc50 2024-03-23T03:50:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.518+0000 7f5cf8a00640 1 -- 172.21.15.102:0/1585808762 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5cf4143460 msgr2=0x7f5cf415aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.518+0000 7f5cf8a00640 1 --2- 172.21.15.102:0/1585808762 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5cf4143460 0x7f5cf415aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.518+0000 7f5cf8a00640 1 -- 172.21.15.102:0/1585808762 >> v2:172.21.15.173:3300/0 conn(0x7f5cf4143930 msgr2=0x7f5cf40cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.518+0000 7f5cf8a00640 1 --2- 172.21.15.102:0/1585808762 >> v2:172.21.15.173:3300/0 conn(0x7f5cf4143930 0x7f5cf40cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.518+0000 7f5cf8a00640 1 -- 172.21.15.102:0/1585808762 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf40ccf10 con 0x7f5cf40cdc50 2024-03-23T03:50:49.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.518+0000 7f5cf8a00640 1 -- 172.21.15.102:0/1585808762 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f5cf00019f0 con 0x7f5cf40cdc50 2024-03-23T03:50:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.518+0000 7f5cf8a00640 1 -- 172.21.15.102:0/1585808762 >> v1:172.21.15.102:6789/0 conn(0x7f5cf40cdc50 legacy=0x7f5cf415e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.518+0000 7f5cf8a00640 1 --2- 172.21.15.102:0/1585808762 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5cdc003970 0x7f5cdc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.518+0000 7f5cf8a00640 1 --2- 172.21.15.102:0/1585808762 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5cf4143460 0x7f5cdc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.518+0000 7f5cf8a00640 1 --2- 172.21.15.102:0/1585808762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc004d00 0x7f5cdc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.518+0000 7f5cf8a00640 1 -- 172.21.15.102:0/1585808762 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf40cc950 con 0x7f5cdc004d00 2024-03-23T03:50:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.518+0000 7f5cf8a00640 1 -- 172.21.15.102:0/1585808762 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf40ccc30 con 0x7f5cdc003970 2024-03-23T03:50:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.518+0000 7f5cf8a00640 1 -- 172.21.15.102:0/1585808762 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cdc00ba40 con 0x7f5cf4143460 2024-03-23T03:50:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.519+0000 7f5cfa800640 1 --2- 172.21.15.102:0/1585808762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc004d00 0x7f5cdc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.519+0000 7f5cf9400640 1 --2- 172.21.15.102:0/1585808762 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5cf4143460 0x7f5cdc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.519+0000 7f5cf9e00640 1 --2- 172.21.15.102:0/1585808762 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5cdc003970 0x7f5cdc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.519+0000 7f5cfa800640 1 -- 172.21.15.102:0/1585808762 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5cf4143460 msgr2=0x7f5cdc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.519+0000 7f5cfa800640 1 --2- 172.21.15.102:0/1585808762 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5cf4143460 0x7f5cdc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.519+0000 7f5cfa800640 1 -- 172.21.15.102:0/1585808762 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5cdc003970 msgr2=0x7f5cdc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:49.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.519+0000 7f5cfa800640 1 --2- 172.21.15.102:0/1585808762 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5cdc003970 0x7f5cdc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.519+0000 7f5cfa800640 1 -- 172.21.15.102:0/1585808762 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cdc003520 con 0x7f5cdc004d00 2024-03-23T03:50:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.520+0000 7f5cfa800640 1 --2- 172.21.15.102:0/1585808762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc004d00 0x7f5cdc005180 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000a830 tx=0x7f5cdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e69c7f22be6ad83 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.520+0000 7f5cf8a00640 1 -- 172.21.15.102:0/1585808762 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0034630 con 0x7f5cdc004d00 2024-03-23T03:50:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.520+0000 7f5cf8a00640 1 -- 172.21.15.102:0/1585808762 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf002f070 con 0x7f5cdc004d00 2024-03-23T03:50:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.520+0000 7f5cf8a00640 1 -- 172.21.15.102:0/1585808762 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cf00349f0 con 0x7f5cdc004d00 2024-03-23T03:50:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.521+0000 7f5cfb200640 1 -- 172.21.15.102:0/1585808762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc004d00 msgr2=0x7f5cdc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.521+0000 7f5cfb200640 1 --2- 172.21.15.102:0/1585808762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cdc004d00 0x7f5cdc005180 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000a830 tx=0x7f5cdc0057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.521+0000 7f5cfa800640 1 -- 172.21.15.102:0/1585808762 reap_dead start 2024-03-23T03:50:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.521+0000 7f5cfb200640 1 -- 172.21.15.102:0/1585808762 shutdown_connections 2024-03-23T03:50:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.521+0000 7f5cfb200640 1 -- 172.21.15.102:0/1585808762 >> 172.21.15.102:0/1585808762 conn(0x7f5cf4065df0 msgr2=0x7f5cf4069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:49.523 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.521+0000 7f5cfb200640 1 -- 172.21.15.102:0/1585808762 shutdown_connections 2024-03-23T03:50:49.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.521+0000 7f5cfb200640 1 -- 172.21.15.102:0/1585808762 wait complete. 2024-03-23T03:50:49.524 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.523+0000 7f5cfb200640 1 Processor -- start 2024-03-23T03:50:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.523+0000 7f5cfb200640 1 -- start start 2024-03-23T03:50:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.524+0000 7f5cfb200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5cdc004d00 0x7f5cf40cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.524+0000 7f5cfb200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cf4149910 0x7f5cf413b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.524+0000 7f5cfb200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5cf4149d70 0x7f5cf413bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.524+0000 7f5cfb200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf413a610 con 0x7f5cf4149910 2024-03-23T03:50:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.524+0000 7f5cfb200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf40cb340 con 0x7f5cdc004d00 2024-03-23T03:50:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.524+0000 7f5cfb200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cf413a820 con 0x7f5cf4149d70 2024-03-23T03:50:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.524+0000 7f5cf9400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cf4149910 0x7f5cf413b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.524+0000 7f5cf9400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cf4149910 0x7f5cf413b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44842/0 (socket says 172.21.15.102:44842) 2024-03-23T03:50:49.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.524+0000 7f5cf9400640 1 -- 172.21.15.102:0/4235357991 learned_addr learned my addr 172.21.15.102:0/4235357991 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:49.526 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.525+0000 7f5cf9e00640 1 --2- 172.21.15.102:0/4235357991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5cdc004d00 0x7f5cf40cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.525+0000 7f5cfa800640 1 --2- 172.21.15.102:0/4235357991 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5cf4149d70 0x7f5cf413bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.525+0000 7f5cf9400640 1 -- 172.21.15.102:0/4235357991 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5cf4149d70 msgr2=0x7f5cf413bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.525+0000 7f5cf9400640 1 --2- 172.21.15.102:0/4235357991 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5cf4149d70 0x7f5cf413bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.525+0000 7f5cf9400640 1 -- 172.21.15.102:0/4235357991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5cdc004d00 msgr2=0x7f5cf40cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.525+0000 7f5cf9400640 1 --2- 172.21.15.102:0/4235357991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5cdc004d00 0x7f5cf40cae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.525+0000 7f5cf9400640 1 -- 172.21.15.102:0/4235357991 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cf413ab00 con 0x7f5cf4149910 2024-03-23T03:50:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.525+0000 7f5cfa800640 1 --2- 172.21.15.102:0/4235357991 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5cf4149d70 0x7f5cf413bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.525+0000 7f5cf9400640 1 --2- 172.21.15.102:0/4235357991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cf4149910 0x7f5cf413b5e0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40116d0 tx=0x7f5ce4011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.525+0000 7f5cee000640 1 -- 172.21.15.102:0/4235357991 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce400f960 con 0x7f5cf4149910 2024-03-23T03:50:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.525+0000 7f5cee000640 1 -- 172.21.15.102:0/4235357991 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ce400fac0 con 0x7f5cf4149910 2024-03-23T03:50:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.525+0000 7f5cee000640 1 -- 172.21.15.102:0/4235357991 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce400fe30 con 0x7f5cf4149910 2024-03-23T03:50:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.525+0000 7f5cfb200640 1 -- 172.21.15.102:0/4235357991 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cf413ade0 con 0x7f5cf4149910 2024-03-23T03:50:49.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.526+0000 7f5cfb200640 1 -- 172.21.15.102:0/4235357991 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cf40c6430 con 0x7f5cf4149910 2024-03-23T03:50:49.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.527+0000 7f5cfb200640 1 -- 172.21.15.102:0/4235357991 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cf413a960 con 0x7f5cf4149910 2024-03-23T03:50:49.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.527+0000 7f5cee000640 1 -- 172.21.15.102:0/4235357991 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5ce4015460 con 0x7f5cf4149910 2024-03-23T03:50:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.529+0000 7f5cee000640 1 --2- 172.21.15.102:0/4235357991 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5cd806ac00 0x7f5cd806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.529+0000 7f5cee000640 1 -- 172.21.15.102:0/4235357991 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5ce4089a60 con 0x7f5cf4149910 2024-03-23T03:50:49.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.529+0000 7f5cf9e00640 1 --2- 172.21.15.102:0/4235357991 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5cd806ac00 0x7f5cd806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.530+0000 7f5cf9e00640 1 --2- 172.21.15.102:0/4235357991 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5cd806ac00 0x7f5cd806d0c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8006fd0 tx=0x7f5ce8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:49.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.533+0000 7f5cee000640 1 -- 172.21.15.102:0/4235357991 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5cf413a960 con 0x7f5cf4149910 2024-03-23T03:50:49.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.726+0000 7f5cfb200640 1 -- 172.21.15.102:0/4235357991 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5cf413a960 con 0x7f5cf4149910 2024-03-23T03:50:49.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.727+0000 7f5cee000640 1 -- 172.21.15.102:0/4235357991 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16310 (secure 0 0 0) 0x7f5cf413a960 con 0x7f5cf4149910 2024-03-23T03:50:49.728 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:49.728 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":92,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:49.295737+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:50:47.750670+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-23T03:50:48.305583+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:50:49.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.730+0000 7f5cfb200640 1 -- 172.21.15.102:0/4235357991 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5cd806ac00 msgr2=0x7f5cd806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.730+0000 7f5cfb200640 1 --2- 172.21.15.102:0/4235357991 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5cd806ac00 0x7f5cd806d0c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8006fd0 tx=0x7f5ce8008040 comp rx=0 tx=0).stop 2024-03-23T03:50:49.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.730+0000 7f5cfb200640 1 -- 172.21.15.102:0/4235357991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cf4149910 msgr2=0x7f5cf413b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.730+0000 7f5cfb200640 1 --2- 172.21.15.102:0/4235357991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cf4149910 0x7f5cf413b5e0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f5ce40116d0 tx=0x7f5ce4011b90 comp rx=0 tx=0).stop 2024-03-23T03:50:49.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.730+0000 7f5cfb200640 1 -- 172.21.15.102:0/4235357991 shutdown_connections 2024-03-23T03:50:49.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.730+0000 7f5cfb200640 1 --2- 172.21.15.102:0/4235357991 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5cd806ac00 0x7f5cd806d0c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.730+0000 7f5cfb200640 1 --2- 172.21.15.102:0/4235357991 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5cf4149d70 0x7f5cf413bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.730+0000 7f5cfb200640 1 --2- 172.21.15.102:0/4235357991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5cf4149910 0x7f5cf413b5e0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.730+0000 7f5cfb200640 1 --2- 172.21.15.102:0/4235357991 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5cdc004d00 0x7f5cf40cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.730+0000 7f5cfb200640 1 -- 172.21.15.102:0/4235357991 >> 172.21.15.102:0/4235357991 conn(0x7f5cf4065df0 msgr2=0x7f5cf413f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:49.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.730+0000 7f5cfb200640 1 -- 172.21.15.102:0/4235357991 shutdown_connections 2024-03-23T03:50:49.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.730+0000 7f5cfb200640 1 -- 172.21.15.102:0/4235357991 wait complete. 2024-03-23T03:50:49.741 INFO:tasks.ceph.mds.c.smithi102.stderr:2024-03-23T03:50:49.741+0000 7facfca00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T03:50:49.749 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:50:49.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.940+0000 7fbcf3e00640 1 Processor -- start 2024-03-23T03:50:49.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.940+0000 7fbcf3e00640 1 -- start start 2024-03-23T03:50:49.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.942+0000 7fbcf3e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcec0b1db0 0x7fbcec167880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.942+0000 7fbcf3e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcec161640 con 0x7fbcec13faa0 2024-03-23T03:50:49.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.942+0000 7fbcf3e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcec0b38a0 con 0x7fbcec160800 2024-03-23T03:50:49.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.942+0000 7fbcf3e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec0b3b80 con 0x7fbcec0b1db0 2024-03-23T03:50:49.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.942+0000 7fbcf1400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fbcec13faa0 0x7fbcec160f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56836/0 (socket says 172.21.15.102:56836) 2024-03-23T03:50:49.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.942+0000 7fbcf1400640 1 -- 172.21.15.102:0/1534581959 learned_addr learned my addr 172.21.15.102:0/1534581959 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:49.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.942+0000 7fbcebe00640 1 -- 172.21.15.102:0/1534581959 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004458098 0 0) 0x7fbcec161640 con 0x7fbcec13faa0 2024-03-23T03:50:49.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.942+0000 7fbcebe00640 1 -- 172.21.15.102:0/1534581959 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbcd4003520 con 0x7fbcec13faa0 2024-03-23T03:50:49.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.942+0000 7fbcf1e00640 1 --2- 172.21.15.102:0/1534581959 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcec0b1db0 0x7fbcec167880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.943+0000 7fbcebe00640 1 -- 172.21.15.102:0/1534581959 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4035889478 0 0) 0x7fbcd4003520 con 0x7fbcec13faa0 2024-03-23T03:50:49.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.943+0000 7fbcebe00640 1 -- 172.21.15.102:0/1534581959 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcec0b1db0 msgr2=0x7fbcec167880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:49.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.943+0000 7fbcebe00640 1 --2- 172.21.15.102:0/1534581959 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcec0b1db0 0x7fbcec167880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.943+0000 7fbcebe00640 1 -- 172.21.15.102:0/1534581959 >> v1:172.21.15.173:6789/0 conn(0x7fbcec160800 legacy=0x7fbcec0b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.943+0000 7fbcebe00640 1 -- 172.21.15.102:0/1534581959 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec0b3e60 con 0x7fbcec13faa0 2024-03-23T03:50:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.943+0000 7fbcebe00640 1 -- 172.21.15.102:0/1534581959 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fbce0001670 con 0x7fbcec13faa0 2024-03-23T03:50:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.943+0000 7fbcebe00640 1 -- 172.21.15.102:0/1534581959 >> v1:172.21.15.102:6789/0 conn(0x7fbcec13faa0 legacy=0x7fbcec160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.943+0000 7fbcebe00640 1 --2- 172.21.15.102:0/1534581959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcd40039a0 0x7fbcd4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.943+0000 7fbcebe00640 1 --2- 172.21.15.102:0/1534581959 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcd40047c0 0x7fbcd4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.943+0000 7fbcebe00640 1 --2- 172.21.15.102:0/1534581959 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcd4005180 0x7fbcd400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.943+0000 7fbcebe00640 1 -- 172.21.15.102:0/1534581959 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec0b3b80 con 0x7fbcd40039a0 2024-03-23T03:50:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.943+0000 7fbcebe00640 1 -- 172.21.15.102:0/1534581959 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd400c080 con 0x7fbcd4005180 2024-03-23T03:50:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.943+0000 7fbcebe00640 1 -- 172.21.15.102:0/1534581959 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd400c240 con 0x7fbcd40047c0 2024-03-23T03:50:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.943+0000 7fbcf1e00640 1 --2- 172.21.15.102:0/1534581959 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcec0b1db0 0x7fbcec167880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:50:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.943+0000 7fbcf1400640 1 --2- 172.21.15.102:0/1534581959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcd40039a0 0x7fbcd4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.944+0000 7fbcf1400640 1 -- 172.21.15.102:0/1534581959 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcd40047c0 msgr2=0x7fbcd4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.944+0000 7fbcf1400640 1 --2- 172.21.15.102:0/1534581959 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcd40047c0 0x7fbcd4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.944+0000 7fbcf1400640 1 -- 172.21.15.102:0/1534581959 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcd4005180 msgr2=0x7fbcd400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.944+0000 7fbcf1400640 1 --2- 172.21.15.102:0/1534581959 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcd4005180 0x7fbcd400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.944+0000 7fbcf1400640 1 -- 172.21.15.102:0/1534581959 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd4003520 con 0x7fbcd40039a0 2024-03-23T03:50:49.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.945+0000 7fbcf1400640 1 --2- 172.21.15.102:0/1534581959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcd40039a0 0x7fbcd4003dc0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fbce000c0e0 tx=0x7fbcd40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218c26d2ea61960e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.945+0000 7fbcebe00640 1 -- 172.21.15.102:0/1534581959 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce0005ad0 con 0x7fbcd40039a0 2024-03-23T03:50:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.945+0000 7fbcebe00640 1 -- 172.21.15.102:0/1534581959 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce0005c30 con 0x7fbcd40039a0 2024-03-23T03:50:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.945+0000 7fbcebe00640 1 -- 172.21.15.102:0/1534581959 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce002c810 con 0x7fbcd40039a0 2024-03-23T03:50:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.945+0000 7fbcf3e00640 1 -- 172.21.15.102:0/1534581959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcd40039a0 msgr2=0x7fbcd4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.945+0000 7fbcf3e00640 1 --2- 172.21.15.102:0/1534581959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcd40039a0 0x7fbcd4003dc0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fbce000c0e0 tx=0x7fbcd40057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.945+0000 7fbcf1e00640 1 -- 172.21.15.102:0/1534581959 reap_dead start 2024-03-23T03:50:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.945+0000 7fbcf3e00640 1 -- 172.21.15.102:0/1534581959 shutdown_connections 2024-03-23T03:50:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.945+0000 7fbcf3e00640 1 -- 172.21.15.102:0/1534581959 >> 172.21.15.102:0/1534581959 conn(0x7fbcec06a3d0 msgr2=0x7fbcec0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.945+0000 7fbcf3e00640 1 -- 172.21.15.102:0/1534581959 shutdown_connections 2024-03-23T03:50:49.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.946+0000 7fbcf3e00640 1 -- 172.21.15.102:0/1534581959 wait complete. 2024-03-23T03:50:49.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.947+0000 7fbcf3e00640 1 Processor -- start 2024-03-23T03:50:49.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.947+0000 7fbcf3e00640 1 -- start start 2024-03-23T03:50:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.948+0000 7fbcf3e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcd40039a0 0x7fbcec0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.948+0000 7fbcf3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcec0cc280 0x7fbcec142a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.948+0000 7fbcf3e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcec142f70 0x7fbcec0ad720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.948+0000 7fbcf3e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec157650 con 0x7fbcec0cc280 2024-03-23T03:50:49.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.948+0000 7fbcf3e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec0cdd00 con 0x7fbcd40039a0 2024-03-23T03:50:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.948+0000 7fbcf3e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcec0cc720 con 0x7fbcec142f70 2024-03-23T03:50:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.948+0000 7fbcf0a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcec0cc280 0x7fbcec142a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.948+0000 7fbcf0a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcec0cc280 0x7fbcec142a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44856/0 (socket says 172.21.15.102:44856) 2024-03-23T03:50:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.948+0000 7fbcf0a00640 1 -- 172.21.15.102:0/2118290240 learned_addr learned my addr 172.21.15.102:0/2118290240 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.949+0000 7fbcf1400640 1 --2- 172.21.15.102:0/2118290240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcd40039a0 0x7fbcec0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.949+0000 7fbcf1e00640 1 --2- 172.21.15.102:0/2118290240 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcec142f70 0x7fbcec0ad720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.949+0000 7fbcf0a00640 1 -- 172.21.15.102:0/2118290240 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcec142f70 msgr2=0x7fbcec0ad720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.949+0000 7fbcf0a00640 1 --2- 172.21.15.102:0/2118290240 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcec142f70 0x7fbcec0ad720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.949+0000 7fbcf0a00640 1 -- 172.21.15.102:0/2118290240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcd40039a0 msgr2=0x7fbcec0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.949+0000 7fbcf0a00640 1 --2- 172.21.15.102:0/2118290240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcd40039a0 0x7fbcec0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:49.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.949+0000 7fbcf0a00640 1 -- 172.21.15.102:0/2118290240 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcec0cca00 con 0x7fbcec0cc280 2024-03-23T03:50:49.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.949+0000 7fbcf0a00640 1 --2- 172.21.15.102:0/2118290240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcec0cc280 0x7fbcec142a30 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fbcd800c500 tx=0x7fbcd800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:49.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.949+0000 7fbce9600640 1 -- 172.21.15.102:0/2118290240 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd80052d0 con 0x7fbcec0cc280 2024-03-23T03:50:49.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.950+0000 7fbce9600640 1 -- 172.21.15.102:0/2118290240 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd8005430 con 0x7fbcec0cc280 2024-03-23T03:50:49.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.950+0000 7fbce9600640 1 -- 172.21.15.102:0/2118290240 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd8012870 con 0x7fbcec0cc280 2024-03-23T03:50:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.950+0000 7fbcf3e00640 1 -- 172.21.15.102:0/2118290240 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcec0ccce0 con 0x7fbcec0cc280 2024-03-23T03:50:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.950+0000 7fbcf3e00640 1 -- 172.21.15.102:0/2118290240 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcec0cd2a0 con 0x7fbcec0cc280 2024-03-23T03:50:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.951+0000 7fbcf3e00640 1 -- 172.21.15.102:0/2118290240 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcec0cd460 con 0x7fbcec0cc280 2024-03-23T03:50:49.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.952+0000 7fbce9600640 1 -- 172.21.15.102:0/2118290240 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbcd80129d0 con 0x7fbcec0cc280 2024-03-23T03:50:49.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.952+0000 7fbce9600640 1 --2- 172.21.15.102:0/2118290240 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbccc06ac00 0x7fbccc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:49.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.953+0000 7fbce9600640 1 -- 172.21.15.102:0/2118290240 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbcd80890c0 con 0x7fbcec0cc280 2024-03-23T03:50:49.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.953+0000 7fbcf1400640 1 --2- 172.21.15.102:0/2118290240 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbccc06ac00 0x7fbccc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:49.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.953+0000 7fbcf1400640 1 --2- 172.21.15.102:0/2118290240 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbccc06ac00 0x7fbccc06d0c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fbce0034000 tx=0x7fbce0009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:49.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:49.957+0000 7fbce9600640 1 -- 172.21.15.102:0/2118290240 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbcd8054150 con 0x7fbcec0cc280 2024-03-23T03:50:50.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.136+0000 7fbcf3e00640 1 -- 172.21.15.102:0/2118290240 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcec073390 con 0x7fbcec0cc280 2024-03-23T03:50:50.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.137+0000 7fbce9600640 1 -- 172.21.15.102:0/2118290240 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+16310 (secure 0 0 0) 0x7fbcd8057770 con 0x7fbcec0cc280 2024-03-23T03:50:50.138 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:50.138 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":92,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:49.295737+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:50:47.750670+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-23T03:50:48.305583+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:50:50.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.140+0000 7fbcf3e00640 1 -- 172.21.15.102:0/2118290240 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbccc06ac00 msgr2=0x7fbccc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:50.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.140+0000 7fbcf3e00640 1 --2- 172.21.15.102:0/2118290240 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbccc06ac00 0x7fbccc06d0c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fbce0034000 tx=0x7fbce0009030 comp rx=0 tx=0).stop 2024-03-23T03:50:50.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.140+0000 7fbcf3e00640 1 -- 172.21.15.102:0/2118290240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcec0cc280 msgr2=0x7fbcec142a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:50.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.140+0000 7fbcf3e00640 1 --2- 172.21.15.102:0/2118290240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcec0cc280 0x7fbcec142a30 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fbcd800c500 tx=0x7fbcd800c9c0 comp rx=0 tx=0).stop 2024-03-23T03:50:50.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.140+0000 7fbcf3e00640 1 -- 172.21.15.102:0/2118290240 shutdown_connections 2024-03-23T03:50:50.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.140+0000 7fbcf3e00640 1 --2- 172.21.15.102:0/2118290240 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbccc06ac00 0x7fbccc06d0c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:50.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.140+0000 7fbcf3e00640 1 --2- 172.21.15.102:0/2118290240 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbcec142f70 0x7fbcec0ad720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:50.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.140+0000 7fbcf3e00640 1 --2- 172.21.15.102:0/2118290240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbcec0cc280 0x7fbcec142a30 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:50.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.140+0000 7fbcf3e00640 1 --2- 172.21.15.102:0/2118290240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbcd40039a0 0x7fbcec0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:50.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.140+0000 7fbcf3e00640 1 -- 172.21.15.102:0/2118290240 >> 172.21.15.102:0/2118290240 conn(0x7fbcec06a3d0 msgr2=0x7fbcec15d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:50.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.140+0000 7fbcf3e00640 1 -- 172.21.15.102:0/2118290240 shutdown_connections 2024-03-23T03:50:50.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.140+0000 7fbcf3e00640 1 -- 172.21.15.102:0/2118290240 wait complete. 2024-03-23T03:50:50.156 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T03:50:50.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.357+0000 7f66be000640 1 Processor -- start 2024-03-23T03:50:50.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.357+0000 7f66be000640 1 -- start start 2024-03-23T03:50:50.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.358+0000 7f66be000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f66b80cf040 0x7f66b81604c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:50.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.358+0000 7f66be000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f66b8070cd0 0x7f66b8160a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:50.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.358+0000 7f66be000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b806d390 con 0x7f66b81456a0 2024-03-23T03:50:50.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.358+0000 7f66be000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b80ad890 con 0x7f66b8070cd0 2024-03-23T03:50:50.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.358+0000 7f66be000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b8145a80 con 0x7f66b80cf040 2024-03-23T03:50:50.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.359+0000 7f66b7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f66b8070cd0 0x7f66b8160a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:50.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.359+0000 7f66b7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f66b8070cd0 0x7f66b8160a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44870/0 (socket says 172.21.15.102:44870) 2024-03-23T03:50:50.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.359+0000 7f66b7e00640 1 -- 172.21.15.102:0/2876739298 learned_addr learned my addr 172.21.15.102:0/2876739298 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:50.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.359+0000 7f66b6a00640 1 --2- 172.21.15.102:0/2876739298 >> v2:172.21.15.173:3300/0 conn(0x7f66b80cf040 0x7f66b81604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:50.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.359+0000 7f66b7e00640 1 -- 172.21.15.102:0/2876739298 >> v2:172.21.15.173:3300/0 conn(0x7f66b80cf040 msgr2=0x7f66b81604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:50.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.359+0000 7f66b7e00640 1 --2- 172.21.15.102:0/2876739298 >> v2:172.21.15.173:3300/0 conn(0x7f66b80cf040 0x7f66b81604c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:50.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.359+0000 7f66b7e00640 1 -- 172.21.15.102:0/2876739298 >> v1:172.21.15.173:6789/0 conn(0x7f66b81456a0 legacy=0x7f66b80ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:50.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.359+0000 7f66b7e00640 1 -- 172.21.15.102:0/2876739298 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b806d670 con 0x7f66b8070cd0 2024-03-23T03:50:50.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.360+0000 7f66b7e00640 1 --2- 172.21.15.102:0/2876739298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b8070cd0 0x7f66b8160a00 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f66b80ae2e0 tx=0x7f66b002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4d9a903486abe46 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:50.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.360+0000 7f66b6000640 1 -- 172.21.15.102:0/2876739298 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b002ca70 con 0x7f66b8070cd0 2024-03-23T03:50:50.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.360+0000 7f66b6000640 1 -- 172.21.15.102:0/2876739298 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66b002cbd0 con 0x7f66b8070cd0 2024-03-23T03:50:50.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.360+0000 7f66b6000640 1 -- 172.21.15.102:0/2876739298 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66b00343f0 con 0x7f66b8070cd0 2024-03-23T03:50:50.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.361+0000 7f66be000640 1 -- 172.21.15.102:0/2876739298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b8070cd0 msgr2=0x7f66b8160a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:50.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.361+0000 7f66be000640 1 --2- 172.21.15.102:0/2876739298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b8070cd0 0x7f66b8160a00 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f66b80ae2e0 tx=0x7f66b002bed0 comp rx=0 tx=0).stop 2024-03-23T03:50:50.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.361+0000 7f66be000640 1 -- 172.21.15.102:0/2876739298 shutdown_connections 2024-03-23T03:50:50.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.361+0000 7f66be000640 1 --2- 172.21.15.102:0/2876739298 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b8070cd0 0x7f66b8160a00 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:50.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.361+0000 7f66be000640 1 --2- 172.21.15.102:0/2876739298 >> v2:172.21.15.173:3300/0 conn(0x7f66b80cf040 0x7f66b81604c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:50.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.361+0000 7f66be000640 1 -- 172.21.15.102:0/2876739298 >> 172.21.15.102:0/2876739298 conn(0x7f66b806a0b0 msgr2=0x7f66b806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:50.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.361+0000 7f66be000640 1 -- 172.21.15.102:0/2876739298 shutdown_connections 2024-03-23T03:50:50.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.361+0000 7f66be000640 1 -- 172.21.15.102:0/2876739298 wait complete. 2024-03-23T03:50:50.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.363+0000 7f66be000640 1 Processor -- start 2024-03-23T03:50:50.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.363+0000 7f66be000640 1 -- start start 2024-03-23T03:50:50.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.364+0000 7f66be000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66b8070cd0 0x7f66b81f5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:50.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.364+0000 7f66be000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b80cf040 0x7f66b813bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:50.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.364+0000 7f66be000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f66b81456a0 0x7f66b813c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:50.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.364+0000 7f66be000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b815a240 con 0x7f66b80cf040 2024-03-23T03:50:50.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.364+0000 7f66be000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b815e490 con 0x7f66b8070cd0 2024-03-23T03:50:50.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.364+0000 7f66be000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b815e770 con 0x7f66b81456a0 2024-03-23T03:50:50.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.364+0000 7f66b6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b80cf040 0x7f66b813bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:50.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.364+0000 7f66b6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b80cf040 0x7f66b813bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44876/0 (socket says 172.21.15.102:44876) 2024-03-23T03:50:50.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.364+0000 7f66b6a00640 1 -- 172.21.15.102:0/3345627476 learned_addr learned my addr 172.21.15.102:0/3345627476 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:50.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.364+0000 7f66b7400640 1 --2- 172.21.15.102:0/3345627476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66b8070cd0 0x7f66b81f5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:50.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.364+0000 7f66b6a00640 1 -- 172.21.15.102:0/3345627476 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f66b81456a0 msgr2=0x7f66b813c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:50.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.364+0000 7f66b6a00640 1 --2- 172.21.15.102:0/3345627476 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f66b81456a0 0x7f66b813c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:50.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.364+0000 7f66b6a00640 1 -- 172.21.15.102:0/3345627476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66b8070cd0 msgr2=0x7f66b81f5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:50.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.364+0000 7f66b6a00640 1 --2- 172.21.15.102:0/3345627476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66b8070cd0 0x7f66b81f5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:50.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.365+0000 7f66b6a00640 1 -- 172.21.15.102:0/3345627476 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b0009b30 con 0x7f66b80cf040 2024-03-23T03:50:50.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.365+0000 7f66b6a00640 1 --2- 172.21.15.102:0/3345627476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b80cf040 0x7f66b813bd20 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f66a400c7f0 tx=0x7f66a400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:50.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.365+0000 7f66af400640 1 -- 172.21.15.102:0/3345627476 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a4004220 con 0x7f66b80cf040 2024-03-23T03:50:50.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.365+0000 7f66af400640 1 -- 172.21.15.102:0/3345627476 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a40043a0 con 0x7f66b80cf040 2024-03-23T03:50:50.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.365+0000 7f66af400640 1 -- 172.21.15.102:0/3345627476 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66a4007860 con 0x7f66b80cf040 2024-03-23T03:50:50.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.365+0000 7f66be000640 1 -- 172.21.15.102:0/3345627476 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b8159bf0 con 0x7f66b80cf040 2024-03-23T03:50:50.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.365+0000 7f66be000640 1 -- 172.21.15.102:0/3345627476 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b81f60f0 con 0x7f66b80cf040 2024-03-23T03:50:50.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.366+0000 7f66be000640 1 -- 172.21.15.102:0/3345627476 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b815a3f0 con 0x7f66b80cf040 2024-03-23T03:50:50.368 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.367+0000 7f66af400640 1 -- 172.21.15.102:0/3345627476 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f66a4007bd0 con 0x7f66b80cf040 2024-03-23T03:50:50.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.368+0000 7f66af400640 1 --2- 172.21.15.102:0/3345627476 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f669806ac00 0x7f669806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:50.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.368+0000 7f66af400640 1 -- 172.21.15.102:0/3345627476 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f66a4089400 con 0x7f66b80cf040 2024-03-23T03:50:50.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.368+0000 7f66b7400640 1 --2- 172.21.15.102:0/3345627476 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f669806ac00 0x7f669806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:50.369 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.369+0000 7f66b7400640 1 --2- 172.21.15.102:0/3345627476 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f669806ac00 0x7f669806d0c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f66b806a4e0 tx=0x7f66a8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:50.373 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.372+0000 7f66af400640 1 -- 172.21.15.102:0/3345627476 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f66b815a3f0 con 0x7f66b80cf040 2024-03-23T03:50:50.471 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:50:50.479 DEBUG:teuthology.orchestra.run.smithi173:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:50:50.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:50.607+0000 7f66be000640 1 -- 172.21.15.102:0/3345627476 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f66b8073390 con 0x7f66b80cf040 2024-03-23T03:50:51.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.310+0000 7f66af400640 1 -- 172.21.15.102:0/3345627476 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v47) v1 ==== 118+0+0 (secure 0 0 0) 0x7f66a40544a0 con 0x7f66b80cf040 2024-03-23T03:50:51.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.314+0000 7f66be000640 1 -- 172.21.15.102:0/3345627476 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f669806ac00 msgr2=0x7f669806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:51.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.314+0000 7f66be000640 1 --2- 172.21.15.102:0/3345627476 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f669806ac00 0x7f669806d0c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f66b806a4e0 tx=0x7f66a8008040 comp rx=0 tx=0).stop 2024-03-23T03:50:51.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.314+0000 7f66be000640 1 -- 172.21.15.102:0/3345627476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b80cf040 msgr2=0x7f66b813bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:51.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.314+0000 7f66be000640 1 --2- 172.21.15.102:0/3345627476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b80cf040 0x7f66b813bd20 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f66a400c7f0 tx=0x7f66a400ccb0 comp rx=0 tx=0).stop 2024-03-23T03:50:51.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.314+0000 7f66be000640 1 -- 172.21.15.102:0/3345627476 shutdown_connections 2024-03-23T03:50:51.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.314+0000 7f66be000640 1 --2- 172.21.15.102:0/3345627476 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f669806ac00 0x7f669806d0c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:51.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.314+0000 7f66be000640 1 --2- 172.21.15.102:0/3345627476 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f66b81456a0 0x7f66b813c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:51.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.314+0000 7f66be000640 1 --2- 172.21.15.102:0/3345627476 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66b80cf040 0x7f66b813bd20 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:51.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.314+0000 7f66be000640 1 --2- 172.21.15.102:0/3345627476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f66b8070cd0 0x7f66b81f5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:51.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.314+0000 7f66be000640 1 -- 172.21.15.102:0/3345627476 >> 172.21.15.102:0/3345627476 conn(0x7f66b806a0b0 msgr2=0x7f66b80b2b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:51.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.314+0000 7f66be000640 1 -- 172.21.15.102:0/3345627476 shutdown_connections 2024-03-23T03:50:51.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.314+0000 7f66be000640 1 -- 172.21.15.102:0/3345627476 wait complete. 2024-03-23T03:50:51.336 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-23T03:50:51.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T03:50:51.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.596+0000 7fe82fe00640 1 Processor -- start 2024-03-23T03:50:51.597 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.596+0000 7fe82fe00640 1 -- start start 2024-03-23T03:50:51.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.598+0000 7fe82fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe8300cf7c0 0x7fe8300cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:51.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.598+0000 7fe82fe00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe830160e90 0x7fe8300d0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:51.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.598+0000 7fe82fe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8300ccd90 con 0x7fe83015c820 2024-03-23T03:50:51.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.598+0000 7fe82fe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8300cdd70 con 0x7fe8300cf7c0 2024-03-23T03:50:51.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.598+0000 7fe82fe00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8300b7fc0 con 0x7fe830160e90 2024-03-23T03:50:51.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.598+0000 7fe827e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe8300cf7c0 0x7fe8300cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:51.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.598+0000 7fe827e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe8300cf7c0 0x7fe8300cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44880/0 (socket says 172.21.15.102:44880) 2024-03-23T03:50:51.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.598+0000 7fe827e00640 1 -- 172.21.15.102:0/4017194134 learned_addr learned my addr 172.21.15.102:0/4017194134 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:51.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.598+0000 7fe827e00640 1 -- 172.21.15.102:0/4017194134 >> v2:172.21.15.173:3300/0 conn(0x7fe830160e90 msgr2=0x7fe8300d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:51.599 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.598+0000 7fe827e00640 1 --2- 172.21.15.102:0/4017194134 >> v2:172.21.15.173:3300/0 conn(0x7fe830160e90 0x7fe8300d0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:51.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.598+0000 7fe827e00640 1 -- 172.21.15.102:0/4017194134 >> v1:172.21.15.102:6789/0 conn(0x7fe83015c820 legacy=0x7fe8300cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:51.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.598+0000 7fe827e00640 1 -- 172.21.15.102:0/4017194134 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8300b82a0 con 0x7fe8300cf7c0 2024-03-23T03:50:51.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.599+0000 7fe827e00640 1 --2- 172.21.15.102:0/4017194134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8300cf7c0 0x7fe8300cd830 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe83008b340 tx=0x7fe81802ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e67b2d73a7e0793 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:51.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.600+0000 7fe82e000640 1 -- 172.21.15.102:0/4017194134 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81802f850 con 0x7fe8300cf7c0 2024-03-23T03:50:51.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.600+0000 7fe82e000640 1 -- 172.21.15.102:0/4017194134 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81802f9b0 con 0x7fe8300cf7c0 2024-03-23T03:50:51.600 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.600+0000 7fe82e000640 1 -- 172.21.15.102:0/4017194134 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81802fd40 con 0x7fe8300cf7c0 2024-03-23T03:50:51.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.600+0000 7fe82fe00640 1 -- 172.21.15.102:0/4017194134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8300cf7c0 msgr2=0x7fe8300cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:51.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.600+0000 7fe82fe00640 1 --2- 172.21.15.102:0/4017194134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8300cf7c0 0x7fe8300cd830 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe83008b340 tx=0x7fe81802ba30 comp rx=0 tx=0).stop 2024-03-23T03:50:51.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.600+0000 7fe82fe00640 1 -- 172.21.15.102:0/4017194134 shutdown_connections 2024-03-23T03:50:51.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.600+0000 7fe82fe00640 1 --2- 172.21.15.102:0/4017194134 >> v2:172.21.15.173:3300/0 conn(0x7fe830160e90 0x7fe8300d0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:51.601 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.600+0000 7fe82fe00640 1 --2- 172.21.15.102:0/4017194134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe8300cf7c0 0x7fe8300cd830 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:51.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.600+0000 7fe82fe00640 1 -- 172.21.15.102:0/4017194134 >> 172.21.15.102:0/4017194134 conn(0x7fe83006b050 msgr2=0x7fe83006b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:51.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.600+0000 7fe82fe00640 1 -- 172.21.15.102:0/4017194134 shutdown_connections 2024-03-23T03:50:51.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.601+0000 7fe82fe00640 1 -- 172.21.15.102:0/4017194134 wait complete. 2024-03-23T03:50:51.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.602+0000 7fe82fe00640 1 Processor -- start 2024-03-23T03:50:51.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.602+0000 7fe82fe00640 1 -- start start 2024-03-23T03:50:51.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.603+0000 7fe82fe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe8300cf7c0 0x7fe8300c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:51.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.603+0000 7fe82fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe83015c820 0x7fe83015ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:51.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.603+0000 7fe82fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe830160e90 0x7fe8300c8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:51.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.603+0000 7fe82fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8300c7d00 con 0x7fe830160e90 2024-03-23T03:50:51.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.603+0000 7fe82fe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8300c6720 con 0x7fe83015c820 2024-03-23T03:50:51.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.603+0000 7fe82fe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8300c6a00 con 0x7fe8300cf7c0 2024-03-23T03:50:51.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.603+0000 7fe82f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe830160e90 0x7fe8300c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:51.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.604+0000 7fe82f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe830160e90 0x7fe8300c8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44896/0 (socket says 172.21.15.102:44896) 2024-03-23T03:50:51.604 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.604+0000 7fe82f400640 1 -- 172.21.15.102:0/4072111990 learned_addr learned my addr 172.21.15.102:0/4072111990 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:51.605 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.604+0000 7fe827e00640 1 --2- 172.21.15.102:0/4072111990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe83015c820 0x7fe83015ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:51.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.604+0000 7fe82ea00640 1 --2- 172.21.15.102:0/4072111990 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe8300cf7c0 0x7fe8300c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:51.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.604+0000 7fe82f400640 1 -- 172.21.15.102:0/4072111990 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe8300cf7c0 msgr2=0x7fe8300c77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:51.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.604+0000 7fe82f400640 1 --2- 172.21.15.102:0/4072111990 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe8300cf7c0 0x7fe8300c77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:51.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.604+0000 7fe82f400640 1 -- 172.21.15.102:0/4072111990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe83015c820 msgr2=0x7fe83015ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:51.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.604+0000 7fe82f400640 1 --2- 172.21.15.102:0/4072111990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe83015c820 0x7fe83015ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:51.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.604+0000 7fe82f400640 1 -- 172.21.15.102:0/4072111990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8180094d0 con 0x7fe830160e90 2024-03-23T03:50:51.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.604+0000 7fe82f400640 1 --2- 172.21.15.102:0/4072111990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe830160e90 0x7fe8300c8850 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fe82800b970 tx=0x7fe82800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:51.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.604+0000 7fe826a00640 1 -- 172.21.15.102:0/4072111990 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe82800cb40 con 0x7fe830160e90 2024-03-23T03:50:51.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.605+0000 7fe826a00640 1 -- 172.21.15.102:0/4072111990 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82800ccc0 con 0x7fe830160e90 2024-03-23T03:50:51.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.605+0000 7fe826a00640 1 -- 172.21.15.102:0/4072111990 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe828014680 con 0x7fe830160e90 2024-03-23T03:50:51.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.605+0000 7fe82fe00640 1 -- 172.21.15.102:0/4072111990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8300c6ce0 con 0x7fe830160e90 2024-03-23T03:50:51.606 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.605+0000 7fe82fe00640 1 -- 172.21.15.102:0/4072111990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8300c72a0 con 0x7fe830160e90 2024-03-23T03:50:51.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.606+0000 7fe82fe00640 1 -- 172.21.15.102:0/4072111990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8300c74c0 con 0x7fe830160e90 2024-03-23T03:50:51.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.607+0000 7fe826a00640 1 -- 172.21.15.102:0/4072111990 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe82801d3b0 con 0x7fe830160e90 2024-03-23T03:50:51.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.607+0000 7fe826a00640 1 --2- 172.21.15.102:0/4072111990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe80c06ac00 0x7fe80c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:51.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.608+0000 7fe826a00640 1 -- 172.21.15.102:0/4072111990 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe828089480 con 0x7fe830160e90 2024-03-23T03:50:51.608 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.608+0000 7fe82ea00640 1 --2- 172.21.15.102:0/4072111990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe80c06ac00 0x7fe80c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:51.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.609+0000 7fe82ea00640 1 --2- 172.21.15.102:0/4072111990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe80c06ac00 0x7fe80c06d0c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe820000f60 tx=0x7fe820009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:51.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.612+0000 7fe826a00640 1 -- 172.21.15.102:0/4072111990 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe828054480 con 0x7fe830160e90 2024-03-23T03:50:51.845 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:51.845+0000 7fe82fe00640 1 -- 172.21.15.102:0/4072111990 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe83004e030 con 0x7fe830160e90 2024-03-23T03:50:52.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.324+0000 7fe826a00640 1 -- 172.21.15.102:0/4072111990 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v48) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe828057aa0 con 0x7fe830160e90 2024-03-23T03:50:52.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.328+0000 7fe82fe00640 1 -- 172.21.15.102:0/4072111990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe80c06ac00 msgr2=0x7fe80c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:52.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.328+0000 7fe82fe00640 1 --2- 172.21.15.102:0/4072111990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe80c06ac00 0x7fe80c06d0c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe820000f60 tx=0x7fe820009040 comp rx=0 tx=0).stop 2024-03-23T03:50:52.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.328+0000 7fe82fe00640 1 -- 172.21.15.102:0/4072111990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe830160e90 msgr2=0x7fe8300c8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:52.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.328+0000 7fe82fe00640 1 --2- 172.21.15.102:0/4072111990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe830160e90 0x7fe8300c8850 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fe82800b970 tx=0x7fe82800be30 comp rx=0 tx=0).stop 2024-03-23T03:50:52.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.328+0000 7fe82fe00640 1 -- 172.21.15.102:0/4072111990 shutdown_connections 2024-03-23T03:50:52.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.328+0000 7fe82fe00640 1 --2- 172.21.15.102:0/4072111990 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe80c06ac00 0x7fe80c06d0c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:52.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.328+0000 7fe82fe00640 1 --2- 172.21.15.102:0/4072111990 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe830160e90 0x7fe8300c8850 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:52.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.328+0000 7fe82fe00640 1 --2- 172.21.15.102:0/4072111990 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe83015c820 0x7fe83015ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:52.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.328+0000 7fe82fe00640 1 --2- 172.21.15.102:0/4072111990 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe8300cf7c0 0x7fe8300c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:52.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.328+0000 7fe82fe00640 1 -- 172.21.15.102:0/4072111990 >> 172.21.15.102:0/4072111990 conn(0x7fe83006b050 msgr2=0x7fe83015ace0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:52.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.328+0000 7fe82fe00640 1 -- 172.21.15.102:0/4072111990 shutdown_connections 2024-03-23T03:50:52.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.328+0000 7fe82fe00640 1 -- 172.21.15.102:0/4072111990 wait complete. 2024-03-23T03:50:52.348 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T03:50:52.348 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:50:52.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.553+0000 7fd2c1c00640 1 Processor -- start 2024-03-23T03:50:52.553 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.553+0000 7fd2c1c00640 1 -- start start 2024-03-23T03:50:52.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.554+0000 7fd2c1c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd2bc0cdc00 0x7fd2bc0b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:52.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.554+0000 7fd2c1c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2bc1575b0 con 0x7fd2bc1571d0 2024-03-23T03:50:52.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.555+0000 7fd2c1c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2bc159f20 con 0x7fd2bc0cae50 2024-03-23T03:50:52.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.555+0000 7fd2c1c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2bc15a200 con 0x7fd2bc0cdc00 2024-03-23T03:50:52.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.555+0000 7fd2bbe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd2bc1571d0 0x7fd2bc15aaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:56866/0 (socket says 172.21.15.102:56866) 2024-03-23T03:50:52.555 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.555+0000 7fd2bbe00640 1 -- 172.21.15.102:0/2138184422 learned_addr learned my addr 172.21.15.102:0/2138184422 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:52.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.555+0000 7fd2bb400640 1 -- 172.21.15.102:0/2138184422 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2190055680 0 0) 0x7fd2bc1575b0 con 0x7fd2bc1571d0 2024-03-23T03:50:52.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.555+0000 7fd2bb400640 1 -- 172.21.15.102:0/2138184422 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2a0003520 con 0x7fd2bc1571d0 2024-03-23T03:50:52.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.555+0000 7fd2b3400640 1 --2- 172.21.15.102:0/2138184422 >> v2:172.21.15.173:3300/0 conn(0x7fd2bc0cdc00 0x7fd2bc0b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.555+0000 7fd2bb400640 1 -- 172.21.15.102:0/2138184422 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1219907421 0 0) 0x7fd2a0003520 con 0x7fd2bc1571d0 2024-03-23T03:50:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.555+0000 7fd2bb400640 1 -- 172.21.15.102:0/2138184422 >> v2:172.21.15.173:3300/0 conn(0x7fd2bc0cdc00 msgr2=0x7fd2bc0b35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.555+0000 7fd2bb400640 1 --2- 172.21.15.102:0/2138184422 >> v2:172.21.15.173:3300/0 conn(0x7fd2bc0cdc00 0x7fd2bc0b35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.556+0000 7fd2bb400640 1 -- 172.21.15.102:0/2138184422 >> v1:172.21.15.173:6789/0 conn(0x7fd2bc0cae50 legacy=0x7fd2bc15d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.556+0000 7fd2bb400640 1 -- 172.21.15.102:0/2138184422 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc15a4e0 con 0x7fd2bc1571d0 2024-03-23T03:50:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.556+0000 7fd2bb400640 1 -- 172.21.15.102:0/2138184422 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fd2ac001670 con 0x7fd2bc1571d0 2024-03-23T03:50:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.556+0000 7fd2bb400640 1 -- 172.21.15.102:0/2138184422 >> v1:172.21.15.102:6789/0 conn(0x7fd2bc1571d0 legacy=0x7fd2bc15aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.556+0000 7fd2bb400640 1 --2- 172.21.15.102:0/2138184422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2a00039c0 0x7fd2a0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.556+0000 7fd2bb400640 1 --2- 172.21.15.102:0/2138184422 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2a00047c0 0x7fd2a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.556+0000 7fd2bb400640 1 --2- 172.21.15.102:0/2138184422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2a00051d0 0x7fd2a000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.556+0000 7fd2bb400640 1 -- 172.21.15.102:0/2138184422 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc15a200 con 0x7fd2a00051d0 2024-03-23T03:50:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.556+0000 7fd2bb400640 1 -- 172.21.15.102:0/2138184422 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a000c100 con 0x7fd2a00039c0 2024-03-23T03:50:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.556+0000 7fd2bb400640 1 -- 172.21.15.102:0/2138184422 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a000e4d0 con 0x7fd2a00047c0 2024-03-23T03:50:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.556+0000 7fd2c1200640 1 --2- 172.21.15.102:0/2138184422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2a00051d0 0x7fd2a000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:52.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.556+0000 7fd2c1200640 1 -- 172.21.15.102:0/2138184422 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2a00047c0 msgr2=0x7fd2a0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.556+0000 7fd2c1200640 1 --2- 172.21.15.102:0/2138184422 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2a00047c0 0x7fd2a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.556+0000 7fd2c1200640 1 -- 172.21.15.102:0/2138184422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2a00039c0 msgr2=0x7fd2a0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.556+0000 7fd2bbe00640 1 --2- 172.21.15.102:0/2138184422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2a00039c0 0x7fd2a0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.557+0000 7fd2c1200640 1 --2- 172.21.15.102:0/2138184422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2a00039c0 0x7fd2a0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.557+0000 7fd2c1200640 1 -- 172.21.15.102:0/2138184422 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a0003520 con 0x7fd2a00051d0 2024-03-23T03:50:52.559 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.557+0000 7fd2bbe00640 1 --2- 172.21.15.102:0/2138184422 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2a00039c0 0x7fd2a0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:52.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.558+0000 7fd2c1200640 1 -- 172.21.15.102:0/2138184422 reap_dead start 2024-03-23T03:50:52.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.558+0000 7fd2c1200640 1 --2- 172.21.15.102:0/2138184422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2a00051d0 0x7fd2a000b980 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4004480 tx=0x7fd2a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=428434ece14d9cb0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:52.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.558+0000 7fd2bb400640 1 -- 172.21.15.102:0/2138184422 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4006b50 con 0x7fd2a00051d0 2024-03-23T03:50:52.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.558+0000 7fd2bb400640 1 -- 172.21.15.102:0/2138184422 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b4006cd0 con 0x7fd2a00051d0 2024-03-23T03:50:52.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.558+0000 7fd2bb400640 1 -- 172.21.15.102:0/2138184422 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b4007260 con 0x7fd2a00051d0 2024-03-23T03:50:52.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.558+0000 7fd2c1c00640 1 -- 172.21.15.102:0/2138184422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2a00051d0 msgr2=0x7fd2a000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:52.560 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.558+0000 7fd2c1c00640 1 --2- 172.21.15.102:0/2138184422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2a00051d0 0x7fd2a000b980 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4004480 tx=0x7fd2a00057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:52.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.559+0000 7fd2c1c00640 1 -- 172.21.15.102:0/2138184422 shutdown_connections 2024-03-23T03:50:52.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.559+0000 7fd2c1c00640 1 --2- 172.21.15.102:0/2138184422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2a00051d0 0x7fd2a000b980 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:52.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.559+0000 7fd2c1c00640 1 -- 172.21.15.102:0/2138184422 >> 172.21.15.102:0/2138184422 conn(0x7fd2bc065df0 msgr2=0x7fd2bc069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:52.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.559+0000 7fd2c1c00640 1 -- 172.21.15.102:0/2138184422 shutdown_connections 2024-03-23T03:50:52.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.559+0000 7fd2c1c00640 1 -- 172.21.15.102:0/2138184422 wait complete. 2024-03-23T03:50:52.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.560+0000 7fd2c1c00640 1 Processor -- start 2024-03-23T03:50:52.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.561+0000 7fd2c1c00640 1 -- start start 2024-03-23T03:50:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.561+0000 7fd2c1c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2a00051d0 0x7fd2bc0b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.561+0000 7fd2c1c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2bc0c33e0 0x7fd2bc0c3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.562+0000 7fd2c1c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2bc1413f0 0x7fd2bc141870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.562+0000 7fd2c1c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc141db0 con 0x7fd2a00051d0 2024-03-23T03:50:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.562+0000 7fd2c1c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc0ab950 con 0x7fd2bc0c33e0 2024-03-23T03:50:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.562+0000 7fd2c1c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc0c3e80 con 0x7fd2bc1413f0 2024-03-23T03:50:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.562+0000 7fd2bbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2a00051d0 0x7fd2bc0b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.562+0000 7fd2bbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2a00051d0 0x7fd2bc0b02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:44914/0 (socket says 172.21.15.102:44914) 2024-03-23T03:50:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.562+0000 7fd2bbe00640 1 -- 172.21.15.102:0/2676870898 learned_addr learned my addr 172.21.15.102:0/2676870898 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.562+0000 7fd2bbe00640 1 -- 172.21.15.102:0/2676870898 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2bc1413f0 msgr2=0x7fd2bc141870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.562+0000 7fd2bbe00640 1 --2- 172.21.15.102:0/2676870898 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2bc1413f0 0x7fd2bc141870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.562+0000 7fd2bbe00640 1 -- 172.21.15.102:0/2676870898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2bc0c33e0 msgr2=0x7fd2bc0c3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.562+0000 7fd2bbe00640 1 --2- 172.21.15.102:0/2676870898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2bc0c33e0 0x7fd2bc0c3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:52.563 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.562+0000 7fd2bbe00640 1 -- 172.21.15.102:0/2676870898 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc0c4160 con 0x7fd2a00051d0 2024-03-23T03:50:52.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.562+0000 7fd2bbe00640 1 --2- 172.21.15.102:0/2676870898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2a00051d0 0x7fd2bc0b02d0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00d770 tx=0x7fd2ac00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:52.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.563+0000 7fd2b8c00640 1 -- 172.21.15.102:0/2676870898 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac0074e0 con 0x7fd2a00051d0 2024-03-23T03:50:52.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.563+0000 7fd2b8c00640 1 -- 172.21.15.102:0/2676870898 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ac007640 con 0x7fd2a00051d0 2024-03-23T03:50:52.564 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.563+0000 7fd2c1c00640 1 -- 172.21.15.102:0/2676870898 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2bc0abbf0 con 0x7fd2a00051d0 2024-03-23T03:50:52.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.563+0000 7fd2b8c00640 1 -- 172.21.15.102:0/2676870898 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac0049e0 con 0x7fd2a00051d0 2024-03-23T03:50:52.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.563+0000 7fd2c1c00640 1 -- 172.21.15.102:0/2676870898 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2bc0c4440 con 0x7fd2a00051d0 2024-03-23T03:50:52.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.564+0000 7fd2c1c00640 1 -- 172.21.15.102:0/2676870898 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2bc04e030 con 0x7fd2a00051d0 2024-03-23T03:50:52.565 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.565+0000 7fd2b8c00640 1 -- 172.21.15.102:0/2676870898 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd2ac004b40 con 0x7fd2a00051d0 2024-03-23T03:50:52.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.566+0000 7fd2b8c00640 1 --2- 172.21.15.102:0/2676870898 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd29806ac30 0x7fd29806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:52.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.566+0000 7fd2b8c00640 1 -- 172.21.15.102:0/2676870898 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd2ac089200 con 0x7fd2a00051d0 2024-03-23T03:50:52.566 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.566+0000 7fd2b3400640 1 --2- 172.21.15.102:0/2676870898 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd29806ac30 0x7fd29806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:52.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.567+0000 7fd2b3400640 1 --2- 172.21.15.102:0/2676870898 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd29806ac30 0x7fd29806d0f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4006f00 tx=0x7fd2a4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:52.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.570+0000 7fd2b8c00640 1 -- 172.21.15.102:0/2676870898 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd2ac054200 con 0x7fd2a00051d0 2024-03-23T03:50:52.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.801+0000 7fd2c1c00640 1 -- 172.21.15.102:0/2676870898 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2bc15ae80 con 0x7fd2a00051d0 2024-03-23T03:50:52.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.802+0000 7fd2b8c00640 1 -- 172.21.15.102:0/2676870898 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4745 (secure 0 0 0) 0x7fd2ac057820 con 0x7fd2a00051d0 2024-03-23T03:50:52.804 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:52.804 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/755891320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":755891320},{"type":"v1","addr":"172.21.15.102:6836","nonce":755891320}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":5207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3792558756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3792558756},{"type":"v1","addr":"172.21.15.173:6835","nonce":3792558756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":5590,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1632622692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1632622692},{"type":"v1","addr":"172.21.15.173:6833","nonce":1632622692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:50:49.296593+0000","modified":"2024-03-23T03:50:52.307714+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/3452944670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":3452944670},{"type":"v1","addr":"172.21.15.102:6837","nonce":3452944670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T03:50:52.804 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 48 2024-03-23T03:50:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.805+0000 7fd2c1c00640 1 -- 172.21.15.102:0/2676870898 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd29806ac30 msgr2=0x7fd29806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.805+0000 7fd2c1c00640 1 --2- 172.21.15.102:0/2676870898 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd29806ac30 0x7fd29806d0f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4006f00 tx=0x7fd2a4008040 comp rx=0 tx=0).stop 2024-03-23T03:50:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.805+0000 7fd2c1c00640 1 -- 172.21.15.102:0/2676870898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2a00051d0 msgr2=0x7fd2bc0b02d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:52.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.805+0000 7fd2c1c00640 1 --2- 172.21.15.102:0/2676870898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2a00051d0 0x7fd2bc0b02d0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac00d770 tx=0x7fd2ac00dc30 comp rx=0 tx=0).stop 2024-03-23T03:50:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.806+0000 7fd2c1c00640 1 -- 172.21.15.102:0/2676870898 shutdown_connections 2024-03-23T03:50:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.806+0000 7fd2c1c00640 1 --2- 172.21.15.102:0/2676870898 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd29806ac30 0x7fd29806d0f0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.806+0000 7fd2c1c00640 1 --2- 172.21.15.102:0/2676870898 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2bc1413f0 0x7fd2bc141870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.806+0000 7fd2c1c00640 1 --2- 172.21.15.102:0/2676870898 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2bc0c33e0 0x7fd2bc0c3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.806+0000 7fd2c1c00640 1 --2- 172.21.15.102:0/2676870898 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2a00051d0 0x7fd2bc0b02d0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.806+0000 7fd2c1c00640 1 -- 172.21.15.102:0/2676870898 >> 172.21.15.102:0/2676870898 conn(0x7fd2bc065df0 msgr2=0x7fd2bc1420b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.806+0000 7fd2c1c00640 1 -- 172.21.15.102:0/2676870898 shutdown_connections 2024-03-23T03:50:52.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:52.806+0000 7fd2c1c00640 1 -- 172.21.15.102:0/2676870898 wait complete. 2024-03-23T03:50:52.826 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:50:53.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.034+0000 7f06f6c00640 1 Processor -- start 2024-03-23T03:50:53.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.034+0000 7f06f6c00640 1 -- start start 2024-03-23T03:50:53.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.036+0000 7f06f6c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f06f015b560 0x7f06f015b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.036+0000 7f06f6c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f015f320 con 0x7f06f00b39b0 2024-03-23T03:50:53.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.036+0000 7f06f6c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06f006c6f0 con 0x7f06f015d410 2024-03-23T03:50:53.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.036+0000 7f06f6c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f006c9d0 con 0x7f06f015b560 2024-03-23T03:50:53.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.036+0000 7f06e7e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f06f00b39b0 0x7f06f015e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55514/0 (socket says 172.21.15.102:55514) 2024-03-23T03:50:53.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.036+0000 7f06e7e00640 1 -- 172.21.15.102:0/1622505772 learned_addr learned my addr 172.21.15.102:0/1622505772 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.036+0000 7f06efe00640 1 --2- 172.21.15.102:0/1622505772 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f06f015b560 0x7f06f015b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.036+0000 7f06ef400640 1 -- 172.21.15.102:0/1622505772 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102214296 0 0) 0x7f06f015f320 con 0x7f06f00b39b0 2024-03-23T03:50:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.036+0000 7f06ef400640 1 -- 172.21.15.102:0/1622505772 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06d4003520 con 0x7f06f00b39b0 2024-03-23T03:50:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.037+0000 7f06ef400640 1 -- 172.21.15.102:0/1622505772 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1574212706 0 0) 0x7f06d4003520 con 0x7f06f00b39b0 2024-03-23T03:50:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.037+0000 7f06ef400640 1 -- 172.21.15.102:0/1622505772 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f06f015b560 msgr2=0x7f06f015b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.037+0000 7f06ef400640 1 --2- 172.21.15.102:0/1622505772 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f06f015b560 0x7f06f015b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.037+0000 7f06ef400640 1 -- 172.21.15.102:0/1622505772 >> v1:172.21.15.173:6789/0 conn(0x7f06f015d410 legacy=0x7f06f015ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:53.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.037+0000 7f06ef400640 1 -- 172.21.15.102:0/1622505772 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f006ccb0 con 0x7f06f00b39b0 2024-03-23T03:50:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.037+0000 7f06ef400640 1 -- 172.21.15.102:0/1622505772 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f06d8001670 con 0x7f06f00b39b0 2024-03-23T03:50:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.037+0000 7f06ef400640 1 -- 172.21.15.102:0/1622505772 >> v1:172.21.15.102:6789/0 conn(0x7f06f00b39b0 legacy=0x7f06f015e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.037+0000 7f06ef400640 1 --2- 172.21.15.102:0/1622505772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06d40039a0 0x7f06d4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.037+0000 7f06ef400640 1 --2- 172.21.15.102:0/1622505772 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f06f015b560 0x7f06d40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.037+0000 7f06ef400640 1 --2- 172.21.15.102:0/1622505772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06d4004d00 0x7f06d4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.037+0000 7f06ef400640 1 -- 172.21.15.102:0/1622505772 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f006c9d0 con 0x7f06d4004d00 2024-03-23T03:50:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.037+0000 7f06ef400640 1 -- 172.21.15.102:0/1622505772 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06d400ba60 con 0x7f06d40039a0 2024-03-23T03:50:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.037+0000 7f06ef400640 1 -- 172.21.15.102:0/1622505772 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06d400bc20 con 0x7f06f015b560 2024-03-23T03:50:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.037+0000 7f06f4c00640 1 --2- 172.21.15.102:0/1622505772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06d4004d00 0x7f06d4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.038+0000 7f06efe00640 1 --2- 172.21.15.102:0/1622505772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06d40039a0 0x7f06d4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.038+0000 7f06e7e00640 1 --2- 172.21.15.102:0/1622505772 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f06f015b560 0x7f06d40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.038+0000 7f06f4c00640 1 -- 172.21.15.102:0/1622505772 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f06f015b560 msgr2=0x7f06d40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:53.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.038+0000 7f06f4c00640 1 --2- 172.21.15.102:0/1622505772 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f06f015b560 0x7f06d40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.038+0000 7f06f4c00640 1 -- 172.21.15.102:0/1622505772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06d40039a0 msgr2=0x7f06d4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:53.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.038+0000 7f06f4c00640 1 --2- 172.21.15.102:0/1622505772 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06d40039a0 0x7f06d4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.038+0000 7f06f4c00640 1 -- 172.21.15.102:0/1622505772 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d4003520 con 0x7f06d4004d00 2024-03-23T03:50:53.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.039+0000 7f06f4c00640 1 --2- 172.21.15.102:0/1622505772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06d4004d00 0x7f06d4005180 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f06f015d8a0 tx=0x7f06d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffff312a95f11028 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:53.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.039+0000 7f06ef400640 1 -- 172.21.15.102:0/1622505772 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e8003b70 con 0x7f06d4004d00 2024-03-23T03:50:53.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.039+0000 7f06ef400640 1 -- 172.21.15.102:0/1622505772 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e8003cd0 con 0x7f06d4004d00 2024-03-23T03:50:53.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.039+0000 7f06ef400640 1 -- 172.21.15.102:0/1622505772 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e8005310 con 0x7f06d4004d00 2024-03-23T03:50:53.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.040+0000 7f06f6c00640 1 -- 172.21.15.102:0/1622505772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06d4004d00 msgr2=0x7f06d4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:53.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.040+0000 7f06f6c00640 1 --2- 172.21.15.102:0/1622505772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06d4004d00 0x7f06d4005180 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f06f015d8a0 tx=0x7f06d40057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.040+0000 7f06f4c00640 1 -- 172.21.15.102:0/1622505772 reap_dead start 2024-03-23T03:50:53.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.040+0000 7f06f6c00640 1 -- 172.21.15.102:0/1622505772 shutdown_connections 2024-03-23T03:50:53.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.040+0000 7f06f6c00640 1 -- 172.21.15.102:0/1622505772 >> 172.21.15.102:0/1622505772 conn(0x7f06f006a3d0 msgr2=0x7f06f00b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:53.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.040+0000 7f06f6c00640 1 -- 172.21.15.102:0/1622505772 shutdown_connections 2024-03-23T03:50:53.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.040+0000 7f06f6c00640 1 -- 172.21.15.102:0/1622505772 wait complete. 2024-03-23T03:50:53.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.042+0000 7f06f6c00640 1 Processor -- start 2024-03-23T03:50:53.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.042+0000 7f06f6c00640 1 -- start start 2024-03-23T03:50:53.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.043+0000 7f06f6c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06f013c400 0x7f06f013e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.043+0000 7f06f6c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f06f0154550 0x7f06f01549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.043+0000 7f06f6c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06f0142bd0 0x7f06f0143050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.043+0000 7f06f6c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f0143590 con 0x7f06f013c400 2024-03-23T03:50:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.043+0000 7f06f6c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f013ccc0 con 0x7f06f0142bd0 2024-03-23T03:50:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.043+0000 7f06f6c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06f013cfa0 con 0x7f06f0154550 2024-03-23T03:50:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.043+0000 7f06efe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06f013c400 0x7f06f013e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.043+0000 7f06efe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06f013c400 0x7f06f013e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57926/0 (socket says 172.21.15.102:57926) 2024-03-23T03:50:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.043+0000 7f06efe00640 1 -- 172.21.15.102:0/2274846397 learned_addr learned my addr 172.21.15.102:0/2274846397 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.043+0000 7f06efe00640 1 -- 172.21.15.102:0/2274846397 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f06f0154550 msgr2=0x7f06f01549d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.043+0000 7f06efe00640 1 --2- 172.21.15.102:0/2274846397 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f06f0154550 0x7f06f01549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.043+0000 7f06efe00640 1 -- 172.21.15.102:0/2274846397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06f0142bd0 msgr2=0x7f06f0143050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.043+0000 7f06efe00640 1 --2- 172.21.15.102:0/2274846397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06f0142bd0 0x7f06f0143050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.043+0000 7f06efe00640 1 -- 172.21.15.102:0/2274846397 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f013d280 con 0x7f06f013c400 2024-03-23T03:50:53.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.044+0000 7f06efe00640 1 --2- 172.21.15.102:0/2274846397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06f013c400 0x7f06f013e040 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f06e0011600 tx=0x7f06e0011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:53.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.044+0000 7f06ecc00640 1 -- 172.21.15.102:0/2274846397 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e0015870 con 0x7f06f013c400 2024-03-23T03:50:53.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.044+0000 7f06ecc00640 1 -- 172.21.15.102:0/2274846397 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e00159d0 con 0x7f06f013c400 2024-03-23T03:50:53.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.044+0000 7f06f6c00640 1 -- 172.21.15.102:0/2274846397 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f013d560 con 0x7f06f013c400 2024-03-23T03:50:53.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.044+0000 7f06ecc00640 1 -- 172.21.15.102:0/2274846397 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e0015d40 con 0x7f06f013c400 2024-03-23T03:50:53.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.044+0000 7f06f6c00640 1 -- 172.21.15.102:0/2274846397 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f013db20 con 0x7f06f013c400 2024-03-23T03:50:53.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.045+0000 7f06f6c00640 1 -- 172.21.15.102:0/2274846397 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f013dce0 con 0x7f06f013c400 2024-03-23T03:50:53.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.046+0000 7f06ecc00640 1 -- 172.21.15.102:0/2274846397 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f06e000f490 con 0x7f06f013c400 2024-03-23T03:50:53.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.047+0000 7f06ecc00640 1 --2- 172.21.15.102:0/2274846397 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06cc06ac30 0x7f06cc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.047+0000 7f06ecc00640 1 -- 172.21.15.102:0/2274846397 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f06e0089d40 con 0x7f06f013c400 2024-03-23T03:50:53.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.047+0000 7f06e7e00640 1 --2- 172.21.15.102:0/2274846397 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06cc06ac30 0x7f06cc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.048+0000 7f06e7e00640 1 --2- 172.21.15.102:0/2274846397 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06cc06ac30 0x7f06cc06d0f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f06d8001050 tx=0x7f06d8001b50 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:53.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.051+0000 7f06ecc00640 1 -- 172.21.15.102:0/2274846397 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f06e0054d40 con 0x7f06f013c400 2024-03-23T03:50:53.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.243+0000 7f06f6c00640 1 -- 172.21.15.102:0/2274846397 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06f006ac20 con 0x7f06f013c400 2024-03-23T03:50:53.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.245+0000 7f06ecc00640 1 -- 172.21.15.102:0/2274846397 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16297 (secure 0 0 0) 0x7f06e0058360 con 0x7f06f013c400 2024-03-23T03:50:53.249 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:53.249 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":93,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:50.300471+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:50:47.750670+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-23T03:50:48.305583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:50:53.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.248+0000 7f06f6c00640 1 -- 172.21.15.102:0/2274846397 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06cc06ac30 msgr2=0x7f06cc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:53.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.248+0000 7f06f6c00640 1 --2- 172.21.15.102:0/2274846397 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06cc06ac30 0x7f06cc06d0f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f06d8001050 tx=0x7f06d8001b50 comp rx=0 tx=0).stop 2024-03-23T03:50:53.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.248+0000 7f06f6c00640 1 -- 172.21.15.102:0/2274846397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06f013c400 msgr2=0x7f06f013e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:53.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.248+0000 7f06f6c00640 1 --2- 172.21.15.102:0/2274846397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06f013c400 0x7f06f013e040 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f06e0011600 tx=0x7f06e0011ac0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.248+0000 7f06f6c00640 1 -- 172.21.15.102:0/2274846397 shutdown_connections 2024-03-23T03:50:53.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.248+0000 7f06f6c00640 1 --2- 172.21.15.102:0/2274846397 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f06cc06ac30 0x7f06cc06d0f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.248+0000 7f06f6c00640 1 --2- 172.21.15.102:0/2274846397 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06f0142bd0 0x7f06f0143050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.248+0000 7f06f6c00640 1 --2- 172.21.15.102:0/2274846397 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f06f0154550 0x7f06f01549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.248+0000 7f06f6c00640 1 --2- 172.21.15.102:0/2274846397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06f013c400 0x7f06f013e040 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.248+0000 7f06f6c00640 1 -- 172.21.15.102:0/2274846397 >> 172.21.15.102:0/2274846397 conn(0x7f06f006a3d0 msgr2=0x7f06f00ae830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:53.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.248+0000 7f06f6c00640 1 -- 172.21.15.102:0/2274846397 shutdown_connections 2024-03-23T03:50:53.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.248+0000 7f06f6c00640 1 -- 172.21.15.102:0/2274846397 wait complete. 2024-03-23T03:50:53.265 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T03:50:53.266 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:50:53.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.462+0000 7fd9d9a00640 1 Processor -- start 2024-03-23T03:50:53.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.463+0000 7fd9d9a00640 1 -- start start 2024-03-23T03:50:53.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.464+0000 7fd9d9a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9d415cda0 0x7fd9d40cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.464+0000 7fd9d9a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9d40b4a60 0x7fd9d40b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.464+0000 7fd9d9a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9d40d1830 con 0x7fd9d40b5a30 2024-03-23T03:50:53.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.464+0000 7fd9d9a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9d40b1f20 con 0x7fd9d40b4a60 2024-03-23T03:50:53.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.464+0000 7fd9d9a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d40d0c20 con 0x7fd9d415cda0 2024-03-23T03:50:53.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.464+0000 7fd9d3400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9d40b4a60 0x7fd9d40b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.464+0000 7fd9d3400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd9d40b4a60 0x7fd9d40b5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57938/0 (socket says 172.21.15.102:57938) 2024-03-23T03:50:53.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.464+0000 7fd9d3400640 1 -- 172.21.15.102:0/2993429555 learned_addr learned my addr 172.21.15.102:0/2993429555 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:53.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.465+0000 7fd9d2a00640 1 --2- 172.21.15.102:0/2993429555 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9d415cda0 0x7fd9d40cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.465+0000 7fd9d3400640 1 -- 172.21.15.102:0/2993429555 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9d415cda0 msgr2=0x7fd9d40cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:53.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.465+0000 7fd9d3400640 1 --2- 172.21.15.102:0/2993429555 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9d415cda0 0x7fd9d40cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.465+0000 7fd9d3400640 1 -- 172.21.15.102:0/2993429555 >> v1:172.21.15.173:6789/0 conn(0x7fd9d40b5a30 legacy=0x7fd9d40b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:53.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.465+0000 7fd9d3400640 1 -- 172.21.15.102:0/2993429555 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d40d0f00 con 0x7fd9d40b4a60 2024-03-23T03:50:53.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.465+0000 7fd9d2a00640 1 --2- 172.21.15.102:0/2993429555 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9d415cda0 0x7fd9d40cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:53.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.466+0000 7fd9d3400640 1 --2- 172.21.15.102:0/2993429555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40b4a60 0x7fd9d40b5e90 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc009870 tx=0x7fd9cc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7674bdea7aefa709 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:53.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.466+0000 7fd9d1600640 1 -- 172.21.15.102:0/2993429555 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc031640 con 0x7fd9d40b4a60 2024-03-23T03:50:53.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.466+0000 7fd9d1600640 1 -- 172.21.15.102:0/2993429555 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9cc0317a0 con 0x7fd9d40b4a60 2024-03-23T03:50:53.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.466+0000 7fd9d1600640 1 -- 172.21.15.102:0/2993429555 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9cc031b30 con 0x7fd9d40b4a60 2024-03-23T03:50:53.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.467+0000 7fd9d9a00640 1 -- 172.21.15.102:0/2993429555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40b4a60 msgr2=0x7fd9d40b5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:53.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.467+0000 7fd9d9a00640 1 --2- 172.21.15.102:0/2993429555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40b4a60 0x7fd9d40b5e90 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fd9cc009870 tx=0x7fd9cc02b8f0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.467+0000 7fd9d9a00640 1 -- 172.21.15.102:0/2993429555 shutdown_connections 2024-03-23T03:50:53.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.467+0000 7fd9d9a00640 1 --2- 172.21.15.102:0/2993429555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40b4a60 0x7fd9d40b5e90 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.467+0000 7fd9d9a00640 1 --2- 172.21.15.102:0/2993429555 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9d415cda0 0x7fd9d40cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.467+0000 7fd9d9a00640 1 -- 172.21.15.102:0/2993429555 >> 172.21.15.102:0/2993429555 conn(0x7fd9d40cf350 msgr2=0x7fd9d40cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:53.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.467+0000 7fd9d9a00640 1 -- 172.21.15.102:0/2993429555 shutdown_connections 2024-03-23T03:50:53.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.467+0000 7fd9d9a00640 1 -- 172.21.15.102:0/2993429555 wait complete. 2024-03-23T03:50:53.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.469+0000 7fd9d9a00640 1 Processor -- start 2024-03-23T03:50:53.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.469+0000 7fd9d9a00640 1 -- start start 2024-03-23T03:50:53.473 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.470+0000 7fd9d9a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9d40b4a60 0x7fd9d40b0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.470+0000 7fd9d9a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40b5a30 0x7fd9d41f1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.470+0000 7fd9d9a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9d415cda0 0x7fd9d40cba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.470+0000 7fd9d9a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d415e110 con 0x7fd9d40b5a30 2024-03-23T03:50:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.470+0000 7fd9d9a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d40b4670 con 0x7fd9d40b4a60 2024-03-23T03:50:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.470+0000 7fd9d9a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9d415d240 con 0x7fd9d415cda0 2024-03-23T03:50:53.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.470+0000 7fd9d2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40b5a30 0x7fd9d41f1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.470+0000 7fd9d2000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40b5a30 0x7fd9d41f1d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57946/0 (socket says 172.21.15.102:57946) 2024-03-23T03:50:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.470+0000 7fd9d2000640 1 -- 172.21.15.102:0/437481213 learned_addr learned my addr 172.21.15.102:0/437481213 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.470+0000 7fd9d2a00640 1 --2- 172.21.15.102:0/437481213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9d40b4a60 0x7fd9d40b0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.471+0000 7fd9d3400640 1 --2- 172.21.15.102:0/437481213 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9d415cda0 0x7fd9d40cba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.471+0000 7fd9d2000640 1 -- 172.21.15.102:0/437481213 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9d415cda0 msgr2=0x7fd9d40cba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.471+0000 7fd9d2000640 1 --2- 172.21.15.102:0/437481213 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9d415cda0 0x7fd9d40cba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.471+0000 7fd9d2000640 1 -- 172.21.15.102:0/437481213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9d40b4a60 msgr2=0x7fd9d40b0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.471+0000 7fd9d2000640 1 --2- 172.21.15.102:0/437481213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9d40b4a60 0x7fd9d40b0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.471+0000 7fd9d2000640 1 -- 172.21.15.102:0/437481213 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9cc0094d0 con 0x7fd9d40b5a30 2024-03-23T03:50:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.471+0000 7fd9d3400640 1 --2- 172.21.15.102:0/437481213 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9d415cda0 0x7fd9d40cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:53.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.471+0000 7fd9d2a00640 1 --2- 172.21.15.102:0/437481213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9d40b4a60 0x7fd9d40b0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:50:53.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.471+0000 7fd9d2000640 1 --2- 172.21.15.102:0/437481213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40b5a30 0x7fd9d41f1d60 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00c7a0 tx=0x7fd9bc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:53.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.471+0000 7fd9caa00640 1 -- 172.21.15.102:0/437481213 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc003d80 con 0x7fd9d40b5a30 2024-03-23T03:50:53.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.471+0000 7fd9caa00640 1 -- 172.21.15.102:0/437481213 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9bc005070 con 0x7fd9d40b5a30 2024-03-23T03:50:53.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.471+0000 7fd9d9a00640 1 -- 172.21.15.102:0/437481213 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d415ddc0 con 0x7fd9d40b5a30 2024-03-23T03:50:53.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.471+0000 7fd9caa00640 1 -- 172.21.15.102:0/437481213 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9bc005460 con 0x7fd9d40b5a30 2024-03-23T03:50:53.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.472+0000 7fd9d9a00640 1 -- 172.21.15.102:0/437481213 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9d415d770 con 0x7fd9d40b5a30 2024-03-23T03:50:53.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.473+0000 7fd9d9a00640 1 -- 172.21.15.102:0/437481213 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9d415df50 con 0x7fd9d40b5a30 2024-03-23T03:50:53.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.474+0000 7fd9caa00640 1 -- 172.21.15.102:0/437481213 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd9bc0055c0 con 0x7fd9d40b5a30 2024-03-23T03:50:53.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.475+0000 7fd9caa00640 1 --2- 172.21.15.102:0/437481213 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9b006ac30 0x7fd9b006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.475+0000 7fd9caa00640 1 -- 172.21.15.102:0/437481213 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd9bc088e60 con 0x7fd9d40b5a30 2024-03-23T03:50:53.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.475+0000 7fd9d2a00640 1 --2- 172.21.15.102:0/437481213 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9b006ac30 0x7fd9b006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.476+0000 7fd9d2a00640 1 --2- 172.21.15.102:0/437481213 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9b006ac30 0x7fd9b006d0f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400a990 tx=0x7fd9c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:53.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.479+0000 7fd9caa00640 1 -- 172.21.15.102:0/437481213 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd9bc053e60 con 0x7fd9d40b5a30 2024-03-23T03:50:53.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.671+0000 7fd9d9a00640 1 -- 172.21.15.102:0/437481213 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd9d40ce080 con 0x7fd9b006ac30 2024-03-23T03:50:53.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.679+0000 7fd9caa00640 1 -- 172.21.15.102:0/437481213 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+272405 (secure 0 0 0) 0x7fd9c400a710 con 0x7fd9b006ac30 2024-03-23T03:50:53.683 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:53.684 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:50:53.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.684+0000 7fd9d9a00640 1 -- 172.21.15.102:0/437481213 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9b006ac30 msgr2=0x7fd9b006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:53.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.684+0000 7fd9d9a00640 1 --2- 172.21.15.102:0/437481213 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9b006ac30 0x7fd9b006d0f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd9c400a990 tx=0x7fd9c4008040 comp rx=0 tx=0).stop 2024-03-23T03:50:53.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.684+0000 7fd9d9a00640 1 -- 172.21.15.102:0/437481213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40b5a30 msgr2=0x7fd9d41f1d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:53.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.684+0000 7fd9d9a00640 1 --2- 172.21.15.102:0/437481213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40b5a30 0x7fd9d41f1d60 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc00c7a0 tx=0x7fd9bc00cc60 comp rx=0 tx=0).stop 2024-03-23T03:50:53.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.684+0000 7fd9d9a00640 1 -- 172.21.15.102:0/437481213 shutdown_connections 2024-03-23T03:50:53.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.684+0000 7fd9d9a00640 1 --2- 172.21.15.102:0/437481213 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd9b006ac30 0x7fd9b006d0f0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.684+0000 7fd9d9a00640 1 --2- 172.21.15.102:0/437481213 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd9d415cda0 0x7fd9d40cba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.684+0000 7fd9d9a00640 1 --2- 172.21.15.102:0/437481213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd9d40b5a30 0x7fd9d41f1d60 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.684+0000 7fd9d9a00640 1 --2- 172.21.15.102:0/437481213 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9d40b4a60 0x7fd9d40b0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.684+0000 7fd9d9a00640 1 -- 172.21.15.102:0/437481213 >> 172.21.15.102:0/437481213 conn(0x7fd9d40cf350 msgr2=0x7fd9d40b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:53.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.684+0000 7fd9d9a00640 1 -- 172.21.15.102:0/437481213 shutdown_connections 2024-03-23T03:50:53.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.684+0000 7fd9d9a00640 1 -- 172.21.15.102:0/437481213 wait complete. 2024-03-23T03:50:53.702 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-03-23T03:50:51.792108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":242236,"kb_used_data":9656,"kb_used_omap":12,"kb_used_meta":232435,"kb_avail":754732484,"statfs":{"total":773094113280,"available":772846063616,"internally_reserved":0,"allocated":9887744,"data_stored":5085196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":238014010},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000939"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:48.805561+0000","last_change":"2024-03-23T03:50:48.805561+0000","last_active":"2024-03-23T03:50:48.805561+0000","last_peered":"2024-03-23T03:50:48.805561+0000","last_clean":"2024-03-23T03:50:48.805561+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:48.805561+0000","last_undegraded":"2024-03-23T03:50:48.805561+0000","last_fullsized":"2024-03-23T03:50:48.805561+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T03:50:47.800987+0000","last_change":"2024-03-23T03:50:47.800987+0000","last_active":"2024-03-23T03:50:47.800987+0000","last_peered":"2024-03-23T03:50:47.800987+0000","last_clean":"2024-03-23T03:50:47.800987+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:50:47.800987+0000","last_undegraded":"2024-03-23T03:50:47.800987+0000","last_fullsized":"2024-03-23T03:50:47.800987+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":217,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-03-23T03:50:40.718443+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:50:40.718443+0000","last_peered":"2024-03-23T03:50:40.718443+0000","last_clean":"2024-03-23T03:50:40.718443+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:50:40.718443+0000","last_undegraded":"2024-03-23T03:50:40.718443+0000","last_fullsized":"2024-03-23T03:50:40.718443+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542196,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28656,"kb_used_data":1636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343184,"statfs":{"total":96636764160,"available":96607420416,"internally_reserved":0,"allocated":1675264,"data_stored":1075702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.079,"15min":1.069},"min":{"1min":0.68200000000000005,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.573,"5min":1.635,"15min":1.635},"last":1.4510000000000001},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0449999999999999,"15min":1.048},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.417,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":0.872}]},{"osd":1,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0549999999999999,"15min":1.044},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.544,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.085,"5min":1.2330000000000001,"15min":1.3009999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.3999999999999999,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":1.1599999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.139,"15min":1.1559999999999999},"min":{"1min":0.76400000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.7909999999999999,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.583},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.345,"15min":1.421},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.4630000000000001,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.212}]},{"osd":3,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.355,"15min":1.4370000000000001},"min":{"1min":0.873,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.397,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.27,"5min":1.3380000000000001,"15min":1.3580000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":1.5840000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.6259999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.266,"15min":1.2669999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.6299999999999999,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.296},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.357,"15min":1.375},"min":{"1min":0.77900000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0619999999999998,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.778}]},{"osd":5,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.3999999999999999,"15min":1.4219999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.6539999999999999,"5min":2.645,"15min":2.645},"last":1.542},{"interface":"front","average":{"1min":1.339,"5min":1.4370000000000001,"15min":1.4530000000000001},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":1.7050000000000001,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.504}]},{"osd":6,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.5229999999999999,"15min":1.5589999999999999},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":1.8200000000000001,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.415,"5min":1.4930000000000001,"15min":1.5209999999999999},"min":{"1min":1.1319999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":2.1509999999999998,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.9399999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542194,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32180,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1081344,"data_stored":485334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.0309999999999999,"15min":1.04},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.3979999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.7330000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0429999999999999,"15min":1.038},"min":{"1min":0.72099999999999997,"5min":0.62,"15min":0.62},"max":{"1min":1.6120000000000001,"5min":1.946,"15min":1.946},"last":0.85199999999999998}]},{"osd":1,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.23},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.129,"15min":1.1559999999999999},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.4810000000000001,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.4890000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.075,"15min":1.069},"min":{"1min":0.73699999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.5449999999999999,"5min":2.044,"15min":2.044},"last":1.151},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1950000000000001,"15min":1.218},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":1.423}]},{"osd":3,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2629999999999999,"15min":1.2849999999999999},"min":{"1min":0.76800000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.4910000000000001,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2549999999999999,"15min":1.2649999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.6319999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.0740000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3759999999999999,"15min":1.3839999999999999},"min":{"1min":0.67700000000000005,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.73,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.363,"15min":1.383},"min":{"1min":0.63400000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.762,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.3660000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.415,"15min":1.4359999999999999},"min":{"1min":0.71099999999999997,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.6919999999999999,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.794},{"interface":"front","average":{"1min":1.456,"5min":1.5249999999999999,"15min":1.5469999999999999},"min":{"1min":0.79700000000000004,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8149999999999999,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.8200000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.4059999999999999,"15min":1.4119999999999999},"min":{"1min":0.77600000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.802,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.548},{"interface":"front","average":{"1min":1.423,"5min":1.5129999999999999,"15min":1.5409999999999999},"min":{"1min":0.84099999999999997,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.74,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.7010000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542194,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28088,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343752,"statfs":{"total":96636764160,"available":96608002048,"internally_reserved":0,"allocated":1093632,"data_stored":490781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.1759999999999999,"15min":1.214},"min":{"1min":0.67900000000000005,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.6779999999999999,"5min":1.806,"15min":1.806},"last":1.375},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.157,"15min":1.181},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.554,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.1930000000000001}]},{"osd":1,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.81699999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.5489999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.004},{"interface":"front","average":{"1min":1.026,"5min":1.1599999999999999,"15min":1.1919999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4219999999999999,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.4079999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.298,"15min":1.3540000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5429999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.5900000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3340000000000001,"15min":1.3779999999999999},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.772,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.458}]},{"osd":3,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.21,"15min":1.2470000000000001},"min":{"1min":0.70599999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.653,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.393,"15min":1.452},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.6779999999999999,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.5069999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.3420000000000001,"15min":1.4119999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.671,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.071},{"interface":"front","average":{"1min":1.282,"5min":1.4119999999999999,"15min":1.444},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.77,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":0.83699999999999997}]},{"osd":6,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.458,"15min":1.514},"min":{"1min":0.76700000000000002,"5min":0.63,"15min":0.63},"max":{"1min":1.873,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.288},{"interface":"front","average":{"1min":1.464,"5min":1.5940000000000001,"15min":1.6559999999999999},"min":{"1min":0.86199999999999999,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.956,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.536}]},{"osd":7,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.538,"15min":1.607},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.673,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.6200000000000001,"15min":1.696},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.855,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.6930000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542195,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32192,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339648,"statfs":{"total":96636764160,"available":96603799552,"internally_reserved":0,"allocated":1093632,"data_stored":490781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.002,"15min":0.97999999999999998},"min":{"1min":0.67800000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.0149999999999999,"15min":1.012},"min":{"1min":0.81200000000000006,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3069999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.95399999999999996}]},{"osd":1,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.0700000000000001,"15min":1.05},"min":{"1min":0.74299999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.704,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.147,"5min":1.0880000000000001,"15min":1.0640000000000001},"min":{"1min":0.50600000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.0589999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2949999999999999,"15min":1.325},"min":{"1min":0.81599999999999995,"5min":0.625,"15min":0.625},"max":{"1min":1.569,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":1.159},{"interface":"front","average":{"1min":1.29,"5min":1.3100000000000001,"15min":1.3080000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.8680000000000001,"5min":2.181,"15min":2.181},"last":0.92800000000000005}]},{"osd":3,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1339999999999999,"15min":1.1299999999999999},"min":{"1min":0.73199999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.26,"15min":1.278},"min":{"1min":0.82899999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.5720000000000001,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.0409999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.278,"15min":1.274},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":0.745},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.298,"15min":1.2829999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.2349999999999999,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":0.89000000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.415,"15min":1.4239999999999999},"min":{"1min":1.143,"5min":0.748,"15min":0.748},"max":{"1min":1.833,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.478,"5min":1.425,"15min":1.4039999999999999},"min":{"1min":0.745,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":0.997}]},{"osd":7,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.385,"15min":1.371},"min":{"1min":1.0880000000000001,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":2.3479999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.4970000000000001,"15min":1.5009999999999999},"min":{"1min":1.056,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.974,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.101}]}]},{"osd":3,"up_from":14,"seq":60129542195,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28084,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343756,"statfs":{"total":96636764160,"available":96608006144,"internally_reserved":0,"allocated":1081344,"data_stored":485334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.83899999999999997,"15min":0.86099999999999999},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.256,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.446,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.58499999999999996}]},{"osd":1,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.92800000000000005,"15min":0.97199999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.997,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.626},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.034,"15min":1.0720000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.712,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.52400000000000002}]},{"osd":2,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.006,"15min":1.052},"min":{"1min":0.435,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.613,"5min":1.823,"15min":1.823},"last":0.47099999999999997},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.1319999999999999,"15min":1.153},"min":{"1min":0.48199999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.6020000000000001,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":0.67500000000000004}]},{"osd":4,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.2310000000000001,"15min":1.2450000000000001},"min":{"1min":0.67600000000000005,"5min":0.625,"15min":0.625},"max":{"1min":1.3859999999999999,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":0.501},{"interface":"front","average":{"1min":1.151,"5min":1.377,"15min":1.4370000000000001},"min":{"1min":0.77100000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.4770000000000001,"5min":4.819,"15min":4.819},"last":0.85899999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.603,"15min":1.4470000000000001},"min":{"1min":0.75,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.0299999999999998,"5min":30.228000000000002,"15min":30.228000000000002},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.3320000000000001,"15min":1.355},"min":{"1min":0.91400000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.637,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":0.54800000000000004}]},{"osd":6,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.448,"15min":1.52},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.5860000000000001,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3999999999999999,"15min":1.397},"min":{"1min":0.90100000000000002,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.1429999999999998,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":0.56799999999999995}]},{"osd":7,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.395,"15min":1.4219999999999999},"min":{"1min":0.81200000000000006,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.71,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":0.65600000000000003},{"interface":"front","average":{"1min":1.298,"5min":1.577,"15min":1.601},"min":{"1min":0.73299999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.629,"5min":10.663,"15min":10.663},"last":0.94599999999999995}]}]},{"osd":2,"up_from":14,"seq":60129542195,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32192,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339648,"statfs":{"total":96636764160,"available":96603799552,"internally_reserved":0,"allocated":1093632,"data_stored":490781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.84999999999999998,"15min":0.88100000000000001},"min":{"1min":0.318,"5min":0.307,"15min":0.307},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":0.745},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.88700000000000001,"15min":0.94299999999999995},"min":{"1min":0.53500000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.23,"5min":2.226,"15min":2.226},"last":0.79000000000000004}]},{"osd":1,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.96899999999999997,"15min":0.995},"min":{"1min":0.621,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.5109999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.88,"15min":0.91700000000000004},"min":{"1min":0.53700000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.282,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.1080000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.98499999999999999,"15min":1.026},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.4750000000000001,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0549999999999999,"15min":1.085},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3380000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.143}]},{"osd":4,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.2090000000000001,"15min":1.236},"min":{"1min":0.96399999999999997,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.488,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.087,"5min":1.179,"15min":1.214},"min":{"1min":0.66300000000000003,"5min":0.627,"15min":0.627},"max":{"1min":1.47,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.3109999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.3360000000000001,"15min":1.3759999999999999},"min":{"1min":0.76200000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.304,"15min":1.349},"min":{"1min":0.75,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.5880000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.224}]},{"osd":6,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.268,"15min":1.3400000000000001},"min":{"1min":0.82199999999999995,"5min":0.753,"15min":0.753},"max":{"1min":1.5109999999999999,"5min":2.456,"15min":2.456},"last":1.359},{"interface":"front","average":{"1min":1.359,"5min":1.4390000000000001,"15min":1.4890000000000001},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.2849999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.335,"15min":1.379},"min":{"1min":0.82699999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.6870000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.171},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.3460000000000001,"15min":1.4099999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5900000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.2509999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542194,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343760,"statfs":{"total":96636764160,"available":96608010240,"internally_reserved":0,"allocated":1081344,"data_stored":485334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.81599999999999995,"15min":0.87},"min":{"1min":0.40200000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.1819999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.82999999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.53800000000000003}]},{"osd":2,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.92800000000000005,"15min":0.96599999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.4570000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.94899999999999995,"15min":0.98599999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.4750000000000001,"5min":1.625,"15min":1.625},"last":0.58899999999999997}]},{"osd":3,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.117,"15min":1.117},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.7929999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.99399999999999999,"15min":1.0189999999999999},"min":{"1min":0.57399999999999995,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.5600000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.86299999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.232,"15min":1.292},"min":{"1min":0.69199999999999995,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.752,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.2749999999999999,"15min":1.3149999999999999},"min":{"1min":0.82599999999999996,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.575,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.69899999999999995}]},{"osd":5,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.2729999999999999,"15min":1.329},"min":{"1min":0.82399999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.7470000000000001,"5min":3.593,"15min":3.593},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.276,"5min":1.3700000000000001,"15min":1.4399999999999999},"min":{"1min":0.76200000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6719999999999999,"5min":2.359,"15min":2.359},"last":0.72799999999999998}]},{"osd":6,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.349,"15min":1.405},"min":{"1min":0.79300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.1709999999999998,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.147,"5min":1.302,"15min":1.3580000000000001},"min":{"1min":0.77800000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.696,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":0.91900000000000004}]},{"osd":7,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.3759999999999999,"15min":1.4379999999999999},"min":{"1min":0.81100000000000005,"5min":0.627,"15min":0.627},"max":{"1min":1.7649999999999999,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":0.997},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.4219999999999999,"15min":1.506},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.7569999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":0.76600000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542194,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32764,"kb_used_data":1648,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339076,"statfs":{"total":96636764160,"available":96603213824,"internally_reserved":0,"allocated":1687552,"data_stored":1081149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.90000000000000002,"15min":0.94699999999999995},"min":{"1min":0.435,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.1599999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":1.478},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.98899999999999999},"min":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.3640000000000001,"5min":1.506,"15min":1.506},"last":1.0489999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.95799999999999996,"15min":1.0009999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1279999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.1160000000000001},{"interface":"front","average":{"1min":0.91900000000000004,"5min":1.0640000000000001,"15min":1.105},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.4350000000000001,"5min":1.901,"15min":1.901},"last":0.89500000000000002}]},{"osd":3,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.97499999999999998,"15min":1.01},"min":{"1min":0.48999999999999999,"5min":0.433,"15min":0.433},"max":{"1min":1.5189999999999999,"5min":1.972,"15min":1.972},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.103,"15min":1.159},"min":{"1min":0.51400000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3180000000000001,"5min":2.298,"15min":2.298},"last":1.004}]},{"osd":4,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.304,"15min":1.3420000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.599,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":1.7090000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.252,"15min":1.28},"min":{"1min":0.83099999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.4990000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.4199999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.3340000000000001,"15min":1.3779999999999999},"min":{"1min":0.878,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.4450000000000001,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.571},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.403,"15min":1.4470000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.49,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.8700000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.46,"15min":1.538},"min":{"1min":0.85799999999999998,"5min":0.747,"15min":0.747},"max":{"1min":1.5940000000000001,"5min":3.032,"15min":3.032},"last":1.978},{"interface":"front","average":{"1min":1.28,"5min":1.4419999999999999,"15min":1.498},"min":{"1min":0.98199999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.5740000000000001,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.9079999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.46,"15min":1.5129999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.7110000000000001,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.8220000000000001},{"interface":"front","average":{"1min":1.329,"5min":1.3939999999999999,"15min":1.4299999999999999},"min":{"1min":0.871,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.008,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.3580000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:50:53.707 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:50:53.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.949+0000 7f1042800640 1 Processor -- start 2024-03-23T03:50:53.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.949+0000 7f1042800640 1 -- start start 2024-03-23T03:50:53.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.951+0000 7f1042800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f103c15cd70 0x7f103c0d0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.951+0000 7f1042800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f103c0b4a50 0x7f103c0b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.951+0000 7f1042800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103c0d17d0 con 0x7f103c0d1330 2024-03-23T03:50:53.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.951+0000 7f1042800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f103c15d150 con 0x7f103c15cd70 2024-03-23T03:50:53.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.951+0000 7f1042800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f103c0d1ab0 con 0x7f103c0b4a50 2024-03-23T03:50:53.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.951+0000 7f103b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f103c15cd70 0x7f103c0d0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.951+0000 7f103b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f103c15cd70 0x7f103c0d0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57958/0 (socket says 172.21.15.102:57958) 2024-03-23T03:50:53.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.951+0000 7f103b400640 1 -- 172.21.15.102:0/1112836666 learned_addr learned my addr 172.21.15.102:0/1112836666 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:53.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.951+0000 7f103be00640 1 --2- 172.21.15.102:0/1112836666 >> v2:172.21.15.173:3300/0 conn(0x7f103c0b4a50 0x7f103c0b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.952+0000 7f103b400640 1 -- 172.21.15.102:0/1112836666 >> v2:172.21.15.173:3300/0 conn(0x7f103c0b4a50 msgr2=0x7f103c0b5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:53.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.952+0000 7f103b400640 1 --2- 172.21.15.102:0/1112836666 >> v2:172.21.15.173:3300/0 conn(0x7f103c0b4a50 0x7f103c0b5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.952+0000 7f103b400640 1 -- 172.21.15.102:0/1112836666 >> v1:172.21.15.102:6789/0 conn(0x7f103c0d1330 legacy=0x7f103c0b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:53.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.952+0000 7f103b400640 1 -- 172.21.15.102:0/1112836666 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c0d1d90 con 0x7f103c15cd70 2024-03-23T03:50:53.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.953+0000 7f103b400640 1 --2- 172.21.15.102:0/1112836666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f103c15cd70 0x7f103c0d0df0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f103c06cbb0 tx=0x7f102c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdfcded2f121c5c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:53.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.953+0000 7f103aa00640 1 -- 172.21.15.102:0/1112836666 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c031870 con 0x7f103c15cd70 2024-03-23T03:50:53.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.953+0000 7f103aa00640 1 -- 172.21.15.102:0/1112836666 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102c0319d0 con 0x7f103c15cd70 2024-03-23T03:50:53.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.953+0000 7f103aa00640 1 -- 172.21.15.102:0/1112836666 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c031d60 con 0x7f103c15cd70 2024-03-23T03:50:53.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.953+0000 7f1042800640 1 -- 172.21.15.102:0/1112836666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f103c15cd70 msgr2=0x7f103c0d0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:53.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.953+0000 7f1042800640 1 --2- 172.21.15.102:0/1112836666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f103c15cd70 0x7f103c0d0df0 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f103c06cbb0 tx=0x7f102c02b910 comp rx=0 tx=0).stop 2024-03-23T03:50:53.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.954+0000 7f1042800640 1 -- 172.21.15.102:0/1112836666 shutdown_connections 2024-03-23T03:50:53.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.954+0000 7f1042800640 1 --2- 172.21.15.102:0/1112836666 >> v2:172.21.15.173:3300/0 conn(0x7f103c0b4a50 0x7f103c0b5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.954+0000 7f1042800640 1 --2- 172.21.15.102:0/1112836666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f103c15cd70 0x7f103c0d0df0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.954+0000 7f1042800640 1 -- 172.21.15.102:0/1112836666 >> 172.21.15.102:0/1112836666 conn(0x7f103c0cf340 msgr2=0x7f103c0cf750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:53.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.954+0000 7f1042800640 1 -- 172.21.15.102:0/1112836666 shutdown_connections 2024-03-23T03:50:53.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.954+0000 7f1042800640 1 -- 172.21.15.102:0/1112836666 wait complete. 2024-03-23T03:50:53.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.955+0000 7f1042800640 1 Processor -- start 2024-03-23T03:50:53.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.955+0000 7f1042800640 1 -- start start 2024-03-23T03:50:53.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.956+0000 7f1042800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f103c0b4a50 0x7f103c152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.956+0000 7f1042800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f103c0d1330 0x7f103c1f5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.956+0000 7f1042800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f103c15cd70 0x7f103c1f5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.956+0000 7f1042800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103c158fb0 con 0x7f103c0d1330 2024-03-23T03:50:53.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.956+0000 7f1042800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103c1526b0 con 0x7f103c15cd70 2024-03-23T03:50:53.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.956+0000 7f1042800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f103c156540 con 0x7f103c0b4a50 2024-03-23T03:50:53.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.957+0000 7f1032a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f103c0d1330 0x7f103c1f5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.957+0000 7f1032a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f103c0d1330 0x7f103c1f5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57974/0 (socket says 172.21.15.102:57974) 2024-03-23T03:50:53.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.957+0000 7f1032a00640 1 -- 172.21.15.102:0/3785860511 learned_addr learned my addr 172.21.15.102:0/3785860511 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:53.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.957+0000 7f103b400640 1 --2- 172.21.15.102:0/3785860511 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f103c0b4a50 0x7f103c152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.957+0000 7f103be00640 1 --2- 172.21.15.102:0/3785860511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f103c15cd70 0x7f103c1f5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.957+0000 7f1032a00640 1 -- 172.21.15.102:0/3785860511 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f103c0b4a50 msgr2=0x7f103c152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:53.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.957+0000 7f1032a00640 1 --2- 172.21.15.102:0/3785860511 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f103c0b4a50 0x7f103c152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.957+0000 7f1032a00640 1 -- 172.21.15.102:0/3785860511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f103c15cd70 msgr2=0x7f103c1f5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:53.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.957+0000 7f1032a00640 1 --2- 172.21.15.102:0/3785860511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f103c15cd70 0x7f103c1f5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:53.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.957+0000 7f1032a00640 1 -- 172.21.15.102:0/3785860511 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c0094d0 con 0x7f103c0d1330 2024-03-23T03:50:53.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.957+0000 7f1032a00640 1 --2- 172.21.15.102:0/3785860511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f103c0d1330 0x7f103c1f5790 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f102400c400 tx=0x7f102400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:53.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.958+0000 7f1033e00640 1 -- 172.21.15.102:0/3785860511 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10240052d0 con 0x7f103c0d1330 2024-03-23T03:50:53.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.958+0000 7f1033e00640 1 -- 172.21.15.102:0/3785860511 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1024005430 con 0x7f103c0d1330 2024-03-23T03:50:53.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.958+0000 7f1033e00640 1 -- 172.21.15.102:0/3785860511 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1024003830 con 0x7f103c0d1330 2024-03-23T03:50:53.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.958+0000 7f1042800640 1 -- 172.21.15.102:0/3785860511 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f103c156820 con 0x7f103c0d1330 2024-03-23T03:50:53.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.958+0000 7f1042800640 1 -- 172.21.15.102:0/3785860511 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103c158c40 con 0x7f103c0d1330 2024-03-23T03:50:53.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.959+0000 7f1042800640 1 -- 172.21.15.102:0/3785860511 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f103c073390 con 0x7f103c0d1330 2024-03-23T03:50:53.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.960+0000 7f1033e00640 1 -- 172.21.15.102:0/3785860511 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1024003990 con 0x7f103c0d1330 2024-03-23T03:50:53.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.961+0000 7f1033e00640 1 --2- 172.21.15.102:0/3785860511 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f101806ac00 0x7f101806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:53.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.961+0000 7f1033e00640 1 -- 172.21.15.102:0/3785860511 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1024088b40 con 0x7f103c0d1330 2024-03-23T03:50:53.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.961+0000 7f103b400640 1 --2- 172.21.15.102:0/3785860511 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f101806ac00 0x7f101806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:53.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.962+0000 7f103b400640 1 --2- 172.21.15.102:0/3785860511 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f101806ac00 0x7f101806d0c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f102c036000 tx=0x7f102c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:53.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:53.965+0000 7f1033e00640 1 -- 172.21.15.102:0/3785860511 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1024053b40 con 0x7f103c0d1330 2024-03-23T03:50:54.155 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.154+0000 7f1042800640 1 -- 172.21.15.102:0/3785860511 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f103c0d2350 con 0x7f101806ac00 2024-03-23T03:50:54.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.165+0000 7f1033e00640 1 -- 172.21.15.102:0/3785860511 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287190 (secure 0 0 0) 0x7f102c044400 con 0x7f101806ac00 2024-03-23T03:50:54.169 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:54.171 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:50:54.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.172+0000 7f1042800640 1 -- 172.21.15.102:0/3785860511 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f101806ac00 msgr2=0x7f101806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:54.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.172+0000 7f1042800640 1 --2- 172.21.15.102:0/3785860511 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f101806ac00 0x7f101806d0c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f102c036000 tx=0x7f102c03c040 comp rx=0 tx=0).stop 2024-03-23T03:50:54.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.172+0000 7f1042800640 1 -- 172.21.15.102:0/3785860511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f103c0d1330 msgr2=0x7f103c1f5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:54.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.172+0000 7f1042800640 1 --2- 172.21.15.102:0/3785860511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f103c0d1330 0x7f103c1f5790 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f102400c400 tx=0x7f102400c8c0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.172+0000 7f1042800640 1 -- 172.21.15.102:0/3785860511 shutdown_connections 2024-03-23T03:50:54.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.172+0000 7f1042800640 1 --2- 172.21.15.102:0/3785860511 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f101806ac00 0x7f101806d0c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.172+0000 7f1042800640 1 --2- 172.21.15.102:0/3785860511 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f103c15cd70 0x7f103c1f5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.172+0000 7f1042800640 1 --2- 172.21.15.102:0/3785860511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f103c0d1330 0x7f103c1f5790 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.172+0000 7f1042800640 1 --2- 172.21.15.102:0/3785860511 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f103c0b4a50 0x7f103c152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.172+0000 7f1042800640 1 -- 172.21.15.102:0/3785860511 >> 172.21.15.102:0/3785860511 conn(0x7f103c0cf340 msgr2=0x7f103c0b2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:54.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.173+0000 7f1042800640 1 -- 172.21.15.102:0/3785860511 shutdown_connections 2024-03-23T03:50:54.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.173+0000 7f1042800640 1 -- 172.21.15.102:0/3785860511 wait complete. 2024-03-23T03:50:54.191 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-03-23T03:50:53.792642+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":140,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":250904,"kb_used_data":10104,"kb_used_omap":12,"kb_used_meta":240627,"kb_avail":754723816,"statfs":{"total":773094113280,"available":772837187584,"internally_reserved":0,"allocated":10346496,"data_stored":5296804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":246402618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":43,"apply_latency_ms":43,"commit_latency_ns":43000000,"apply_latency_ns":43000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.985548"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.312311+0000","last_change":"2024-03-23T03:50:49.338674+0000","last_active":"2024-03-23T03:50:50.312311+0000","last_peered":"2024-03-23T03:50:50.312311+0000","last_clean":"2024-03-23T03:50:50.312311+0000","last_became_active":"2024-03-23T03:50:49.338428+0000","last_became_peered":"2024-03-23T03:50:49.338428+0000","last_unstale":"2024-03-23T03:50:50.312311+0000","last_undegraded":"2024-03-23T03:50:50.312311+0000","last_fullsized":"2024-03-23T03:50:50.312311+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:04:39.959954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.913461+0000","last_change":"2024-03-23T03:50:49.345282+0000","last_active":"2024-03-23T03:50:50.913461+0000","last_peered":"2024-03-23T03:50:50.913461+0000","last_clean":"2024-03-23T03:50:50.913461+0000","last_became_active":"2024-03-23T03:50:49.344810+0000","last_became_peered":"2024-03-23T03:50:49.344810+0000","last_unstale":"2024-03-23T03:50:50.913461+0000","last_undegraded":"2024-03-23T03:50:50.913461+0000","last_fullsized":"2024-03-23T03:50:50.913461+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:31:49.474231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.311971+0000","last_change":"2024-03-23T03:50:49.350085+0000","last_active":"2024-03-23T03:50:50.311971+0000","last_peered":"2024-03-23T03:50:50.311971+0000","last_clean":"2024-03-23T03:50:50.311971+0000","last_became_active":"2024-03-23T03:50:49.349819+0000","last_became_peered":"2024-03-23T03:50:49.349819+0000","last_unstale":"2024-03-23T03:50:50.311971+0000","last_undegraded":"2024-03-23T03:50:50.311971+0000","last_fullsized":"2024-03-23T03:50:50.311971+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:43:39.758040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.561074+0000","last_change":"2024-03-23T03:50:49.317927+0000","last_active":"2024-03-23T03:50:50.561074+0000","last_peered":"2024-03-23T03:50:50.561074+0000","last_clean":"2024-03-23T03:50:50.561074+0000","last_became_active":"2024-03-23T03:50:49.317361+0000","last_became_peered":"2024-03-23T03:50:49.317361+0000","last_unstale":"2024-03-23T03:50:50.561074+0000","last_undegraded":"2024-03-23T03:50:50.561074+0000","last_fullsized":"2024-03-23T03:50:50.561074+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:47:38.500387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.348116+0000","last_change":"2024-03-23T03:50:49.345422+0000","last_active":"2024-03-23T03:50:49.348116+0000","last_peered":"2024-03-23T03:50:49.348116+0000","last_clean":"2024-03-23T03:50:49.348116+0000","last_became_active":"2024-03-23T03:50:49.345125+0000","last_became_peered":"2024-03-23T03:50:49.345125+0000","last_unstale":"2024-03-23T03:50:49.348116+0000","last_undegraded":"2024-03-23T03:50:49.348116+0000","last_fullsized":"2024-03-23T03:50:49.348116+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:13:59.703234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.342452+0000","last_change":"2024-03-23T03:50:49.338862+0000","last_active":"2024-03-23T03:50:49.342452+0000","last_peered":"2024-03-23T03:50:49.342452+0000","last_clean":"2024-03-23T03:50:49.342452+0000","last_became_active":"2024-03-23T03:50:49.338335+0000","last_became_peered":"2024-03-23T03:50:49.338335+0000","last_unstale":"2024-03-23T03:50:49.342452+0000","last_undegraded":"2024-03-23T03:50:49.342452+0000","last_fullsized":"2024-03-23T03:50:49.342452+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:38.947196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.312523+0000","last_change":"2024-03-23T03:50:49.342925+0000","last_active":"2024-03-23T03:50:50.312523+0000","last_peered":"2024-03-23T03:50:50.312523+0000","last_clean":"2024-03-23T03:50:50.312523+0000","last_became_active":"2024-03-23T03:50:49.342615+0000","last_became_peered":"2024-03-23T03:50:49.342615+0000","last_unstale":"2024-03-23T03:50:50.312523+0000","last_undegraded":"2024-03-23T03:50:50.312523+0000","last_fullsized":"2024-03-23T03:50:50.312523+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:54:28.049512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.560793+0000","last_change":"2024-03-23T03:50:49.334136+0000","last_active":"2024-03-23T03:50:50.560793+0000","last_peered":"2024-03-23T03:50:50.560793+0000","last_clean":"2024-03-23T03:50:50.560793+0000","last_became_active":"2024-03-23T03:50:49.333735+0000","last_became_peered":"2024-03-23T03:50:49.333735+0000","last_unstale":"2024-03-23T03:50:50.560793+0000","last_undegraded":"2024-03-23T03:50:50.560793+0000","last_fullsized":"2024-03-23T03:50:50.560793+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:40:47.728994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.353054+0000","last_change":"2024-03-23T03:50:49.345966+0000","last_active":"2024-03-23T03:50:49.353054+0000","last_peered":"2024-03-23T03:50:49.353054+0000","last_clean":"2024-03-23T03:50:49.353054+0000","last_became_active":"2024-03-23T03:50:49.345581+0000","last_became_peered":"2024-03-23T03:50:49.345581+0000","last_unstale":"2024-03-23T03:50:49.353054+0000","last_undegraded":"2024-03-23T03:50:49.353054+0000","last_fullsized":"2024-03-23T03:50:49.353054+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:08:00.174486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.338226+0000","last_change":"2024-03-23T03:50:49.328844+0000","last_active":"2024-03-23T03:50:49.338226+0000","last_peered":"2024-03-23T03:50:49.338226+0000","last_clean":"2024-03-23T03:50:49.338226+0000","last_became_active":"2024-03-23T03:50:49.328584+0000","last_became_peered":"2024-03-23T03:50:49.328584+0000","last_unstale":"2024-03-23T03:50:49.338226+0000","last_undegraded":"2024-03-23T03:50:49.338226+0000","last_fullsized":"2024-03-23T03:50:49.338226+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:58:56.723966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.354119+0000","last_change":"2024-03-23T03:50:49.350796+0000","last_active":"2024-03-23T03:50:49.354119+0000","last_peered":"2024-03-23T03:50:49.354119+0000","last_clean":"2024-03-23T03:50:49.354119+0000","last_became_active":"2024-03-23T03:50:49.348198+0000","last_became_peered":"2024-03-23T03:50:49.348198+0000","last_unstale":"2024-03-23T03:50:49.354119+0000","last_undegraded":"2024-03-23T03:50:49.354119+0000","last_fullsized":"2024-03-23T03:50:49.354119+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:07:47.177228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.332842+0000","last_change":"2024-03-23T03:50:49.326855+0000","last_active":"2024-03-23T03:50:49.332842+0000","last_peered":"2024-03-23T03:50:49.332842+0000","last_clean":"2024-03-23T03:50:49.332842+0000","last_became_active":"2024-03-23T03:50:49.326616+0000","last_became_peered":"2024-03-23T03:50:49.326616+0000","last_unstale":"2024-03-23T03:50:49.332842+0000","last_undegraded":"2024-03-23T03:50:49.332842+0000","last_fullsized":"2024-03-23T03:50:49.332842+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:16:20.376385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.561049+0000","last_change":"2024-03-23T03:50:49.334942+0000","last_active":"2024-03-23T03:50:50.561049+0000","last_peered":"2024-03-23T03:50:50.561049+0000","last_clean":"2024-03-23T03:50:50.561049+0000","last_became_active":"2024-03-23T03:50:49.334643+0000","last_became_peered":"2024-03-23T03:50:49.334643+0000","last_unstale":"2024-03-23T03:50:50.561049+0000","last_undegraded":"2024-03-23T03:50:50.561049+0000","last_fullsized":"2024-03-23T03:50:50.561049+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:44.576808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.350162+0000","last_change":"2024-03-23T03:50:49.340510+0000","last_active":"2024-03-23T03:50:49.350162+0000","last_peered":"2024-03-23T03:50:49.350162+0000","last_clean":"2024-03-23T03:50:49.350162+0000","last_became_active":"2024-03-23T03:50:49.340043+0000","last_became_peered":"2024-03-23T03:50:49.340043+0000","last_unstale":"2024-03-23T03:50:49.350162+0000","last_undegraded":"2024-03-23T03:50:49.350162+0000","last_fullsized":"2024-03-23T03:50:49.350162+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:31:51.073893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.560841+0000","last_change":"2024-03-23T03:50:49.326011+0000","last_active":"2024-03-23T03:50:50.560841+0000","last_peered":"2024-03-23T03:50:50.560841+0000","last_clean":"2024-03-23T03:50:50.560841+0000","last_became_active":"2024-03-23T03:50:49.325746+0000","last_became_peered":"2024-03-23T03:50:49.325746+0000","last_unstale":"2024-03-23T03:50:50.560841+0000","last_undegraded":"2024-03-23T03:50:50.560841+0000","last_fullsized":"2024-03-23T03:50:50.560841+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:31:49.474231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.335868+0000","last_change":"2024-03-23T03:50:49.329659+0000","last_active":"2024-03-23T03:50:49.335868+0000","last_peered":"2024-03-23T03:50:49.335868+0000","last_clean":"2024-03-23T03:50:49.335868+0000","last_became_active":"2024-03-23T03:50:49.329389+0000","last_became_peered":"2024-03-23T03:50:49.329389+0000","last_unstale":"2024-03-23T03:50:49.335868+0000","last_undegraded":"2024-03-23T03:50:49.335868+0000","last_fullsized":"2024-03-23T03:50:49.335868+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:28.763494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.335388+0000","last_change":"2024-03-23T03:50:49.329024+0000","last_active":"2024-03-23T03:50:49.335388+0000","last_peered":"2024-03-23T03:50:49.335388+0000","last_clean":"2024-03-23T03:50:49.335388+0000","last_became_active":"2024-03-23T03:50:49.328774+0000","last_became_peered":"2024-03-23T03:50:49.328774+0000","last_unstale":"2024-03-23T03:50:49.335388+0000","last_undegraded":"2024-03-23T03:50:49.335388+0000","last_fullsized":"2024-03-23T03:50:49.335388+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:31:03.571521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.312105+0000","last_change":"2024-03-23T03:50:49.344267+0000","last_active":"2024-03-23T03:50:50.312105+0000","last_peered":"2024-03-23T03:50:50.312105+0000","last_clean":"2024-03-23T03:50:50.312105+0000","last_became_active":"2024-03-23T03:50:49.343965+0000","last_became_peered":"2024-03-23T03:50:49.343965+0000","last_unstale":"2024-03-23T03:50:50.312105+0000","last_undegraded":"2024-03-23T03:50:50.312105+0000","last_fullsized":"2024-03-23T03:50:50.312105+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:56:14.113291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.354134+0000","last_change":"2024-03-23T03:50:49.350843+0000","last_active":"2024-03-23T03:50:49.354134+0000","last_peered":"2024-03-23T03:50:49.354134+0000","last_clean":"2024-03-23T03:50:49.354134+0000","last_became_active":"2024-03-23T03:50:49.347882+0000","last_became_peered":"2024-03-23T03:50:49.347882+0000","last_unstale":"2024-03-23T03:50:49.354134+0000","last_undegraded":"2024-03-23T03:50:49.354134+0000","last_fullsized":"2024-03-23T03:50:49.354134+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:00:01.037161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.561173+0000","last_change":"2024-03-23T03:50:49.327152+0000","last_active":"2024-03-23T03:50:50.561173+0000","last_peered":"2024-03-23T03:50:50.561173+0000","last_clean":"2024-03-23T03:50:50.561173+0000","last_became_active":"2024-03-23T03:50:49.326867+0000","last_became_peered":"2024-03-23T03:50:49.326867+0000","last_unstale":"2024-03-23T03:50:50.561173+0000","last_undegraded":"2024-03-23T03:50:50.561173+0000","last_fullsized":"2024-03-23T03:50:50.561173+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:58:41.231872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.353959+0000","last_change":"2024-03-23T03:50:49.347793+0000","last_active":"2024-03-23T03:50:49.353959+0000","last_peered":"2024-03-23T03:50:49.353959+0000","last_clean":"2024-03-23T03:50:49.353959+0000","last_became_active":"2024-03-23T03:50:49.347411+0000","last_became_peered":"2024-03-23T03:50:49.347411+0000","last_unstale":"2024-03-23T03:50:49.353959+0000","last_undegraded":"2024-03-23T03:50:49.353959+0000","last_fullsized":"2024-03-23T03:50:49.353959+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:04:37.814004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.350143+0000","last_change":"2024-03-23T03:50:49.338764+0000","last_active":"2024-03-23T03:50:49.350143+0000","last_peered":"2024-03-23T03:50:49.350143+0000","last_clean":"2024-03-23T03:50:49.350143+0000","last_became_active":"2024-03-23T03:50:49.338351+0000","last_became_peered":"2024-03-23T03:50:49.338351+0000","last_unstale":"2024-03-23T03:50:49.350143+0000","last_undegraded":"2024-03-23T03:50:49.350143+0000","last_fullsized":"2024-03-23T03:50:49.350143+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:37:15.819663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.321932+0000","last_change":"2024-03-23T03:50:49.316814+0000","last_active":"2024-03-23T03:50:49.321932+0000","last_peered":"2024-03-23T03:50:49.321932+0000","last_clean":"2024-03-23T03:50:49.321932+0000","last_became_active":"2024-03-23T03:50:49.315936+0000","last_became_peered":"2024-03-23T03:50:49.315936+0000","last_unstale":"2024-03-23T03:50:49.321932+0000","last_undegraded":"2024-03-23T03:50:49.321932+0000","last_fullsized":"2024-03-23T03:50:49.321932+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:46:59.735716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.311649+0000","last_change":"2024-03-23T03:50:49.350397+0000","last_active":"2024-03-23T03:50:50.311649+0000","last_peered":"2024-03-23T03:50:50.311649+0000","last_clean":"2024-03-23T03:50:50.311649+0000","last_became_active":"2024-03-23T03:50:49.350081+0000","last_became_peered":"2024-03-23T03:50:49.350081+0000","last_unstale":"2024-03-23T03:50:50.311649+0000","last_undegraded":"2024-03-23T03:50:50.311649+0000","last_fullsized":"2024-03-23T03:50:50.311649+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:14:41.036541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.913363+0000","last_change":"2024-03-23T03:50:49.343211+0000","last_active":"2024-03-23T03:50:50.913363+0000","last_peered":"2024-03-23T03:50:50.913363+0000","last_clean":"2024-03-23T03:50:50.913363+0000","last_became_active":"2024-03-23T03:50:49.342951+0000","last_became_peered":"2024-03-23T03:50:49.342951+0000","last_unstale":"2024-03-23T03:50:50.913363+0000","last_undegraded":"2024-03-23T03:50:50.913363+0000","last_fullsized":"2024-03-23T03:50:50.913363+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:00:13.715170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.354048+0000","last_change":"2024-03-23T03:50:49.348746+0000","last_active":"2024-03-23T03:50:49.354048+0000","last_peered":"2024-03-23T03:50:49.354048+0000","last_clean":"2024-03-23T03:50:49.354048+0000","last_became_active":"2024-03-23T03:50:49.348490+0000","last_became_peered":"2024-03-23T03:50:49.348490+0000","last_unstale":"2024-03-23T03:50:49.354048+0000","last_undegraded":"2024-03-23T03:50:49.354048+0000","last_fullsized":"2024-03-23T03:50:49.354048+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:32:08.354910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.338382+0000","last_change":"2024-03-23T03:50:49.331415+0000","last_active":"2024-03-23T03:50:49.338382+0000","last_peered":"2024-03-23T03:50:49.338382+0000","last_clean":"2024-03-23T03:50:49.338382+0000","last_became_active":"2024-03-23T03:50:49.331172+0000","last_became_peered":"2024-03-23T03:50:49.331172+0000","last_unstale":"2024-03-23T03:50:49.338382+0000","last_undegraded":"2024-03-23T03:50:49.338382+0000","last_fullsized":"2024-03-23T03:50:49.338382+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:05:01.968663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.561505+0000","last_change":"2024-03-23T03:50:49.347830+0000","last_active":"2024-03-23T03:50:50.561505+0000","last_peered":"2024-03-23T03:50:50.561505+0000","last_clean":"2024-03-23T03:50:50.561505+0000","last_became_active":"2024-03-23T03:50:49.347557+0000","last_became_peered":"2024-03-23T03:50:49.347557+0000","last_unstale":"2024-03-23T03:50:50.561505+0000","last_undegraded":"2024-03-23T03:50:50.561505+0000","last_fullsized":"2024-03-23T03:50:50.561505+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:50:03.455763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.312870+0000","last_change":"2024-03-23T03:50:49.342791+0000","last_active":"2024-03-23T03:50:50.312870+0000","last_peered":"2024-03-23T03:50:50.312870+0000","last_clean":"2024-03-23T03:50:50.312870+0000","last_became_active":"2024-03-23T03:50:49.342056+0000","last_became_peered":"2024-03-23T03:50:49.342056+0000","last_unstale":"2024-03-23T03:50:50.312870+0000","last_undegraded":"2024-03-23T03:50:50.312870+0000","last_fullsized":"2024-03-23T03:50:50.312870+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:00:13.715170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.350178+0000","last_change":"2024-03-23T03:50:49.344831+0000","last_active":"2024-03-23T03:50:49.350178+0000","last_peered":"2024-03-23T03:50:49.350178+0000","last_clean":"2024-03-23T03:50:49.350178+0000","last_became_active":"2024-03-23T03:50:49.342428+0000","last_became_peered":"2024-03-23T03:50:49.342428+0000","last_unstale":"2024-03-23T03:50:49.350178+0000","last_undegraded":"2024-03-23T03:50:49.350178+0000","last_fullsized":"2024-03-23T03:50:49.350178+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:17:08.009072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.912701+0000","last_change":"2024-03-23T03:50:49.337737+0000","last_active":"2024-03-23T03:50:50.912701+0000","last_peered":"2024-03-23T03:50:50.912701+0000","last_clean":"2024-03-23T03:50:50.912701+0000","last_became_active":"2024-03-23T03:50:49.337465+0000","last_became_peered":"2024-03-23T03:50:49.337465+0000","last_unstale":"2024-03-23T03:50:50.912701+0000","last_undegraded":"2024-03-23T03:50:50.912701+0000","last_fullsized":"2024-03-23T03:50:50.912701+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:36:19.333343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.335887+0000","last_change":"2024-03-23T03:50:49.330302+0000","last_active":"2024-03-23T03:50:49.335887+0000","last_peered":"2024-03-23T03:50:49.335887+0000","last_clean":"2024-03-23T03:50:49.335887+0000","last_became_active":"2024-03-23T03:50:49.330077+0000","last_became_peered":"2024-03-23T03:50:49.330077+0000","last_unstale":"2024-03-23T03:50:49.335887+0000","last_undegraded":"2024-03-23T03:50:49.335887+0000","last_fullsized":"2024-03-23T03:50:49.335887+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:43:39.758040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.356573+0000","last_change":"2024-03-23T03:50:49.350273+0000","last_active":"2024-03-23T03:50:49.356573+0000","last_peered":"2024-03-23T03:50:49.356573+0000","last_clean":"2024-03-23T03:50:49.356573+0000","last_became_active":"2024-03-23T03:50:49.349651+0000","last_became_peered":"2024-03-23T03:50:49.349651+0000","last_unstale":"2024-03-23T03:50:49.356573+0000","last_undegraded":"2024-03-23T03:50:49.356573+0000","last_fullsized":"2024-03-23T03:50:49.356573+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:00:01.916671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"92'4","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.349027+0000","last_change":"2024-03-23T03:50:48.855297+0000","last_active":"2024-03-23T03:50:49.349027+0000","last_peered":"2024-03-23T03:50:49.349027+0000","last_clean":"2024-03-23T03:50:49.349027+0000","last_became_active":"2024-03-23T03:50:48.855076+0000","last_became_peered":"2024-03-23T03:50:48.855076+0000","last_unstale":"2024-03-23T03:50:49.349027+0000","last_undegraded":"2024-03-23T03:50:49.349027+0000","last_fullsized":"2024-03-23T03:50:49.349027+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:55:13.738261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.332494+0000","last_change":"2024-03-23T03:50:49.322892+0000","last_active":"2024-03-23T03:50:49.332494+0000","last_peered":"2024-03-23T03:50:49.332494+0000","last_clean":"2024-03-23T03:50:49.332494+0000","last_became_active":"2024-03-23T03:50:49.322641+0000","last_became_peered":"2024-03-23T03:50:49.322641+0000","last_unstale":"2024-03-23T03:50:49.332494+0000","last_undegraded":"2024-03-23T03:50:49.332494+0000","last_fullsized":"2024-03-23T03:50:49.332494+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:01:29.654781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"92'1","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.348296+0000","last_change":"2024-03-23T03:50:48.847594+0000","last_active":"2024-03-23T03:50:49.348296+0000","last_peered":"2024-03-23T03:50:49.348296+0000","last_clean":"2024-03-23T03:50:49.348296+0000","last_became_active":"2024-03-23T03:50:48.847249+0000","last_became_peered":"2024-03-23T03:50:48.847249+0000","last_unstale":"2024-03-23T03:50:49.348296+0000","last_undegraded":"2024-03-23T03:50:49.348296+0000","last_fullsized":"2024-03-23T03:50:49.348296+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:25:05.475515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.313114+0000","last_change":"2024-03-23T03:50:49.339055+0000","last_active":"2024-03-23T03:50:50.313114+0000","last_peered":"2024-03-23T03:50:50.313114+0000","last_clean":"2024-03-23T03:50:50.313114+0000","last_became_active":"2024-03-23T03:50:49.338717+0000","last_became_peered":"2024-03-23T03:50:49.338717+0000","last_unstale":"2024-03-23T03:50:50.313114+0000","last_undegraded":"2024-03-23T03:50:50.313114+0000","last_fullsized":"2024-03-23T03:50:50.313114+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:36:19.333343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"92'2","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.313315+0000","last_change":"2024-03-23T03:50:48.853062+0000","last_active":"2024-03-23T03:50:50.313315+0000","last_peered":"2024-03-23T03:50:50.313315+0000","last_clean":"2024-03-23T03:50:50.313315+0000","last_became_active":"2024-03-23T03:50:48.852512+0000","last_became_peered":"2024-03-23T03:50:48.852512+0000","last_unstale":"2024-03-23T03:50:50.313315+0000","last_undegraded":"2024-03-23T03:50:50.313315+0000","last_fullsized":"2024-03-23T03:50:50.313315+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:53:21.634284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.335370+0000","last_change":"2024-03-23T03:50:49.325391+0000","last_active":"2024-03-23T03:50:49.335370+0000","last_peered":"2024-03-23T03:50:49.335370+0000","last_clean":"2024-03-23T03:50:49.335370+0000","last_became_active":"2024-03-23T03:50:49.325027+0000","last_became_peered":"2024-03-23T03:50:49.325027+0000","last_unstale":"2024-03-23T03:50:49.335370+0000","last_undegraded":"2024-03-23T03:50:49.335370+0000","last_fullsized":"2024-03-23T03:50:49.335370+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:32:41.989270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"92'2","reported_seq":12,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.339571+0000","last_change":"2024-03-23T03:50:48.851375+0000","last_active":"2024-03-23T03:50:49.339571+0000","last_peered":"2024-03-23T03:50:49.339571+0000","last_clean":"2024-03-23T03:50:49.339571+0000","last_became_active":"2024-03-23T03:50:48.851025+0000","last_became_peered":"2024-03-23T03:50:48.851025+0000","last_unstale":"2024-03-23T03:50:49.339571+0000","last_undegraded":"2024-03-23T03:50:49.339571+0000","last_fullsized":"2024-03-23T03:50:49.339571+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:32:00.919390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.560819+0000","last_change":"2024-03-23T03:50:49.325849+0000","last_active":"2024-03-23T03:50:50.560819+0000","last_peered":"2024-03-23T03:50:50.560819+0000","last_clean":"2024-03-23T03:50:50.560819+0000","last_became_active":"2024-03-23T03:50:49.325588+0000","last_became_peered":"2024-03-23T03:50:49.325588+0000","last_unstale":"2024-03-23T03:50:50.560819+0000","last_undegraded":"2024-03-23T03:50:50.560819+0000","last_fullsized":"2024-03-23T03:50:50.560819+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:55:31.916538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"92'1","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.347758+0000","last_change":"2024-03-23T03:50:48.847098+0000","last_active":"2024-03-23T03:50:49.347758+0000","last_peered":"2024-03-23T03:50:49.347758+0000","last_clean":"2024-03-23T03:50:49.347758+0000","last_became_active":"2024-03-23T03:50:48.846786+0000","last_became_peered":"2024-03-23T03:50:48.846786+0000","last_unstale":"2024-03-23T03:50:49.347758+0000","last_undegraded":"2024-03-23T03:50:49.347758+0000","last_fullsized":"2024-03-23T03:50:49.347758+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:55:13.738261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.343866+0000","last_change":"2024-03-23T03:50:49.339233+0000","last_active":"2024-03-23T03:50:49.343866+0000","last_peered":"2024-03-23T03:50:49.343866+0000","last_clean":"2024-03-23T03:50:49.343866+0000","last_became_active":"2024-03-23T03:50:49.338960+0000","last_became_peered":"2024-03-23T03:50:49.338960+0000","last_unstale":"2024-03-23T03:50:49.343866+0000","last_undegraded":"2024-03-23T03:50:49.343866+0000","last_fullsized":"2024-03-23T03:50:49.343866+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:59:34.917369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"92'1","reported_seq":12,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.561507+0000","last_change":"2024-03-23T03:50:48.851750+0000","last_active":"2024-03-23T03:50:50.561507+0000","last_peered":"2024-03-23T03:50:50.561507+0000","last_clean":"2024-03-23T03:50:50.561507+0000","last_became_active":"2024-03-23T03:50:48.851278+0000","last_became_peered":"2024-03-23T03:50:48.851278+0000","last_unstale":"2024-03-23T03:50:50.561507+0000","last_undegraded":"2024-03-23T03:50:50.561507+0000","last_fullsized":"2024-03-23T03:50:50.561507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:54:10.680930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.342547+0000","last_change":"2024-03-23T03:50:49.339190+0000","last_active":"2024-03-23T03:50:49.342547+0000","last_peered":"2024-03-23T03:50:49.342547+0000","last_clean":"2024-03-23T03:50:49.342547+0000","last_became_active":"2024-03-23T03:50:49.338917+0000","last_became_peered":"2024-03-23T03:50:49.338917+0000","last_unstale":"2024-03-23T03:50:49.342547+0000","last_undegraded":"2024-03-23T03:50:49.342547+0000","last_fullsized":"2024-03-23T03:50:49.342547+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:55:24.534741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.311922+0000","last_change":"2024-03-23T03:50:48.842347+0000","last_active":"2024-03-23T03:50:50.311922+0000","last_peered":"2024-03-23T03:50:50.311922+0000","last_clean":"2024-03-23T03:50:50.311922+0000","last_became_active":"2024-03-23T03:50:48.841927+0000","last_became_peered":"2024-03-23T03:50:48.841927+0000","last_unstale":"2024-03-23T03:50:50.311922+0000","last_undegraded":"2024-03-23T03:50:50.311922+0000","last_fullsized":"2024-03-23T03:50:50.311922+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:09:27.204049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.561269+0000","last_change":"2024-03-23T03:50:49.316660+0000","last_active":"2024-03-23T03:50:50.561269+0000","last_peered":"2024-03-23T03:50:50.561269+0000","last_clean":"2024-03-23T03:50:50.561269+0000","last_became_active":"2024-03-23T03:50:49.316390+0000","last_became_peered":"2024-03-23T03:50:49.316390+0000","last_unstale":"2024-03-23T03:50:50.561269+0000","last_undegraded":"2024-03-23T03:50:50.561269+0000","last_fullsized":"2024-03-23T03:50:50.561269+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:36:47.994716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.312487+0000","last_change":"2024-03-23T03:50:48.852394+0000","last_active":"2024-03-23T03:50:50.312487+0000","last_peered":"2024-03-23T03:50:50.312487+0000","last_clean":"2024-03-23T03:50:50.312487+0000","last_became_active":"2024-03-23T03:50:48.852092+0000","last_became_peered":"2024-03-23T03:50:48.852092+0000","last_unstale":"2024-03-23T03:50:50.312487+0000","last_undegraded":"2024-03-23T03:50:50.312487+0000","last_fullsized":"2024-03-23T03:50:50.312487+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:24:39.928516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.353971+0000","last_change":"2024-03-23T03:50:49.348123+0000","last_active":"2024-03-23T03:50:49.353971+0000","last_peered":"2024-03-23T03:50:49.353971+0000","last_clean":"2024-03-23T03:50:49.353971+0000","last_became_active":"2024-03-23T03:50:49.347793+0000","last_became_peered":"2024-03-23T03:50:49.347793+0000","last_unstale":"2024-03-23T03:50:49.353971+0000","last_undegraded":"2024-03-23T03:50:49.353971+0000","last_fullsized":"2024-03-23T03:50:49.353971+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:12:31.306540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.913524+0000","last_change":"2024-03-23T03:50:48.848019+0000","last_active":"2024-03-23T03:50:50.913524+0000","last_peered":"2024-03-23T03:50:50.913524+0000","last_clean":"2024-03-23T03:50:50.913524+0000","last_became_active":"2024-03-23T03:50:48.847757+0000","last_became_peered":"2024-03-23T03:50:48.847757+0000","last_unstale":"2024-03-23T03:50:50.913524+0000","last_undegraded":"2024-03-23T03:50:50.913524+0000","last_fullsized":"2024-03-23T03:50:50.913524+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:24:39.928516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.313152+0000","last_change":"2024-03-23T03:50:49.350294+0000","last_active":"2024-03-23T03:50:50.313152+0000","last_peered":"2024-03-23T03:50:50.313152+0000","last_clean":"2024-03-23T03:50:50.313152+0000","last_became_active":"2024-03-23T03:50:49.350036+0000","last_became_peered":"2024-03-23T03:50:49.350036+0000","last_unstale":"2024-03-23T03:50:50.313152+0000","last_undegraded":"2024-03-23T03:50:50.313152+0000","last_fullsized":"2024-03-23T03:50:50.313152+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:20:18.585071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.312219+0000","last_change":"2024-03-23T03:50:48.841260+0000","last_active":"2024-03-23T03:50:50.312219+0000","last_peered":"2024-03-23T03:50:50.312219+0000","last_clean":"2024-03-23T03:50:50.312219+0000","last_became_active":"2024-03-23T03:50:48.840952+0000","last_became_peered":"2024-03-23T03:50:48.840952+0000","last_unstale":"2024-03-23T03:50:50.312219+0000","last_undegraded":"2024-03-23T03:50:50.312219+0000","last_fullsized":"2024-03-23T03:50:50.312219+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:21:36.970873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.312267+0000","last_change":"2024-03-23T03:50:49.339889+0000","last_active":"2024-03-23T03:50:50.312267+0000","last_peered":"2024-03-23T03:50:50.312267+0000","last_clean":"2024-03-23T03:50:50.312267+0000","last_became_active":"2024-03-23T03:50:49.337959+0000","last_became_peered":"2024-03-23T03:50:49.337959+0000","last_unstale":"2024-03-23T03:50:50.312267+0000","last_undegraded":"2024-03-23T03:50:50.312267+0000","last_fullsized":"2024-03-23T03:50:50.312267+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:55:28.763494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"92'3","reported_seq":13,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.351400+0000","last_change":"2024-03-23T03:50:48.853424+0000","last_active":"2024-03-23T03:50:49.351400+0000","last_peered":"2024-03-23T03:50:49.351400+0000","last_clean":"2024-03-23T03:50:49.351400+0000","last_became_active":"2024-03-23T03:50:48.853039+0000","last_became_peered":"2024-03-23T03:50:48.853039+0000","last_unstale":"2024-03-23T03:50:49.351400+0000","last_undegraded":"2024-03-23T03:50:49.351400+0000","last_fullsized":"2024-03-23T03:50:49.351400+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:26.405998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.561131+0000","last_change":"2024-03-23T03:50:49.344478+0000","last_active":"2024-03-23T03:50:50.561131+0000","last_peered":"2024-03-23T03:50:50.561131+0000","last_clean":"2024-03-23T03:50:50.561131+0000","last_became_active":"2024-03-23T03:50:49.344076+0000","last_became_peered":"2024-03-23T03:50:49.344076+0000","last_unstale":"2024-03-23T03:50:50.561131+0000","last_undegraded":"2024-03-23T03:50:50.561131+0000","last_fullsized":"2024-03-23T03:50:50.561131+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:40:22.342235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"92'6","reported_seq":16,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.357045+0000","last_change":"2024-03-23T03:50:48.845558+0000","last_active":"2024-03-23T03:50:49.357045+0000","last_peered":"2024-03-23T03:50:49.357045+0000","last_clean":"2024-03-23T03:50:49.357045+0000","last_became_active":"2024-03-23T03:50:48.845283+0000","last_became_peered":"2024-03-23T03:50:48.845283+0000","last_unstale":"2024-03-23T03:50:49.357045+0000","last_undegraded":"2024-03-23T03:50:49.357045+0000","last_fullsized":"2024-03-23T03:50:49.357045+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:05:59.862274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.560921+0000","last_change":"2024-03-23T03:50:49.334148+0000","last_active":"2024-03-23T03:50:50.560921+0000","last_peered":"2024-03-23T03:50:50.560921+0000","last_clean":"2024-03-23T03:50:50.560921+0000","last_became_active":"2024-03-23T03:50:49.332551+0000","last_became_peered":"2024-03-23T03:50:49.332551+0000","last_unstale":"2024-03-23T03:50:50.560921+0000","last_undegraded":"2024-03-23T03:50:50.560921+0000","last_fullsized":"2024-03-23T03:50:50.560921+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:50:36.732234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"92'2","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.312967+0000","last_change":"2024-03-23T03:50:48.853259+0000","last_active":"2024-03-23T03:50:50.312967+0000","last_peered":"2024-03-23T03:50:50.312967+0000","last_clean":"2024-03-23T03:50:50.312967+0000","last_became_active":"2024-03-23T03:50:48.852916+0000","last_became_peered":"2024-03-23T03:50:48.852916+0000","last_unstale":"2024-03-23T03:50:50.312967+0000","last_undegraded":"2024-03-23T03:50:50.312967+0000","last_fullsized":"2024-03-23T03:50:50.312967+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:54:10.680930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.348052+0000","last_change":"2024-03-23T03:50:49.341182+0000","last_active":"2024-03-23T03:50:49.348052+0000","last_peered":"2024-03-23T03:50:49.348052+0000","last_clean":"2024-03-23T03:50:49.348052+0000","last_became_active":"2024-03-23T03:50:49.340906+0000","last_became_peered":"2024-03-23T03:50:49.340906+0000","last_unstale":"2024-03-23T03:50:49.348052+0000","last_undegraded":"2024-03-23T03:50:49.348052+0000","last_fullsized":"2024-03-23T03:50:49.348052+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:35:16.594407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.316765+0000","last_change":"2024-03-23T03:50:48.850242+0000","last_active":"2024-03-23T03:50:49.316765+0000","last_peered":"2024-03-23T03:50:49.316765+0000","last_clean":"2024-03-23T03:50:49.316765+0000","last_became_active":"2024-03-23T03:50:48.849973+0000","last_became_peered":"2024-03-23T03:50:48.849973+0000","last_unstale":"2024-03-23T03:50:49.316765+0000","last_undegraded":"2024-03-23T03:50:49.316765+0000","last_fullsized":"2024-03-23T03:50:49.316765+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:16:57.359997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.341247+0000","last_change":"2024-03-23T03:50:49.334964+0000","last_active":"2024-03-23T03:50:49.341247+0000","last_peered":"2024-03-23T03:50:49.341247+0000","last_clean":"2024-03-23T03:50:49.341247+0000","last_became_active":"2024-03-23T03:50:49.334737+0000","last_became_peered":"2024-03-23T03:50:49.334737+0000","last_unstale":"2024-03-23T03:50:49.341247+0000","last_undegraded":"2024-03-23T03:50:49.341247+0000","last_fullsized":"2024-03-23T03:50:49.341247+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:19.378087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.561114+0000","last_change":"2024-03-23T03:50:48.843169+0000","last_active":"2024-03-23T03:50:50.561114+0000","last_peered":"2024-03-23T03:50:50.561114+0000","last_clean":"2024-03-23T03:50:50.561114+0000","last_became_active":"2024-03-23T03:50:48.842871+0000","last_became_peered":"2024-03-23T03:50:48.842871+0000","last_unstale":"2024-03-23T03:50:50.561114+0000","last_undegraded":"2024-03-23T03:50:50.561114+0000","last_fullsized":"2024-03-23T03:50:50.561114+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:08:20.059038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.355636+0000","last_change":"2024-03-23T03:50:49.349118+0000","last_active":"2024-03-23T03:50:49.355636+0000","last_peered":"2024-03-23T03:50:49.355636+0000","last_clean":"2024-03-23T03:50:49.355636+0000","last_became_active":"2024-03-23T03:50:49.348450+0000","last_became_peered":"2024-03-23T03:50:49.348450+0000","last_unstale":"2024-03-23T03:50:49.355636+0000","last_undegraded":"2024-03-23T03:50:49.355636+0000","last_fullsized":"2024-03-23T03:50:49.355636+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:41:46.897501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"92'8","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.560918+0000","last_change":"2024-03-23T03:50:48.851098+0000","last_active":"2024-03-23T03:50:50.560918+0000","last_peered":"2024-03-23T03:50:50.560918+0000","last_clean":"2024-03-23T03:50:50.560918+0000","last_became_active":"2024-03-23T03:50:48.850783+0000","last_became_peered":"2024-03-23T03:50:48.850783+0000","last_unstale":"2024-03-23T03:50:50.560918+0000","last_undegraded":"2024-03-23T03:50:50.560918+0000","last_fullsized":"2024-03-23T03:50:50.560918+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:14:08.573818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.350981+0000","last_change":"2024-03-23T03:50:49.348325+0000","last_active":"2024-03-23T03:50:49.350981+0000","last_peered":"2024-03-23T03:50:49.350981+0000","last_clean":"2024-03-23T03:50:49.350981+0000","last_became_active":"2024-03-23T03:50:49.347913+0000","last_became_peered":"2024-03-23T03:50:49.347913+0000","last_unstale":"2024-03-23T03:50:49.350981+0000","last_undegraded":"2024-03-23T03:50:49.350981+0000","last_fullsized":"2024-03-23T03:50:49.350981+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:54:15.427936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"92'1","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.346995+0000","last_change":"2024-03-23T03:50:48.846648+0000","last_active":"2024-03-23T03:50:49.346995+0000","last_peered":"2024-03-23T03:50:49.346995+0000","last_clean":"2024-03-23T03:50:49.346995+0000","last_became_active":"2024-03-23T03:50:48.845261+0000","last_became_peered":"2024-03-23T03:50:48.845261+0000","last_unstale":"2024-03-23T03:50:49.346995+0000","last_undegraded":"2024-03-23T03:50:49.346995+0000","last_fullsized":"2024-03-23T03:50:49.346995+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:32:30.327508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.352398+0000","last_change":"2024-03-23T03:50:49.345281+0000","last_active":"2024-03-23T03:50:49.352398+0000","last_peered":"2024-03-23T03:50:49.352398+0000","last_clean":"2024-03-23T03:50:49.352398+0000","last_became_active":"2024-03-23T03:50:49.344979+0000","last_became_peered":"2024-03-23T03:50:49.344979+0000","last_unstale":"2024-03-23T03:50:49.352398+0000","last_undegraded":"2024-03-23T03:50:49.352398+0000","last_fullsized":"2024-03-23T03:50:49.352398+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:12:31.306540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"92'2","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.313385+0000","last_change":"2024-03-23T03:50:48.844421+0000","last_active":"2024-03-23T03:50:50.313385+0000","last_peered":"2024-03-23T03:50:50.313385+0000","last_clean":"2024-03-23T03:50:50.313385+0000","last_became_active":"2024-03-23T03:50:48.843232+0000","last_became_peered":"2024-03-23T03:50:48.843232+0000","last_unstale":"2024-03-23T03:50:50.313385+0000","last_undegraded":"2024-03-23T03:50:50.313385+0000","last_fullsized":"2024-03-23T03:50:50.313385+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:08:20.059038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.354027+0000","last_change":"2024-03-23T03:50:49.348203+0000","last_active":"2024-03-23T03:50:49.354027+0000","last_peered":"2024-03-23T03:50:49.354027+0000","last_clean":"2024-03-23T03:50:49.354027+0000","last_became_active":"2024-03-23T03:50:49.346774+0000","last_became_peered":"2024-03-23T03:50:49.346774+0000","last_unstale":"2024-03-23T03:50:49.354027+0000","last_undegraded":"2024-03-23T03:50:49.354027+0000","last_fullsized":"2024-03-23T03:50:49.354027+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:20.511033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"92'1","reported_seq":11,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.351414+0000","last_change":"2024-03-23T03:50:48.845719+0000","last_active":"2024-03-23T03:50:49.351414+0000","last_peered":"2024-03-23T03:50:49.351414+0000","last_clean":"2024-03-23T03:50:49.351414+0000","last_became_active":"2024-03-23T03:50:48.845422+0000","last_became_peered":"2024-03-23T03:50:48.845422+0000","last_unstale":"2024-03-23T03:50:49.351414+0000","last_undegraded":"2024-03-23T03:50:49.351414+0000","last_fullsized":"2024-03-23T03:50:49.351414+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:32:30.327508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.337102+0000","last_change":"2024-03-23T03:50:49.333935+0000","last_active":"2024-03-23T03:50:49.337102+0000","last_peered":"2024-03-23T03:50:49.337102+0000","last_clean":"2024-03-23T03:50:49.337102+0000","last_became_active":"2024-03-23T03:50:49.333309+0000","last_became_peered":"2024-03-23T03:50:49.333309+0000","last_unstale":"2024-03-23T03:50:49.337102+0000","last_undegraded":"2024-03-23T03:50:49.337102+0000","last_fullsized":"2024-03-23T03:50:49.337102+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:33:52.357680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"92'2","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.913000+0000","last_change":"2024-03-23T03:50:48.842870+0000","last_active":"2024-03-23T03:50:50.913000+0000","last_peered":"2024-03-23T03:50:50.913000+0000","last_clean":"2024-03-23T03:50:50.913000+0000","last_became_active":"2024-03-23T03:50:48.840575+0000","last_became_peered":"2024-03-23T03:50:48.840575+0000","last_unstale":"2024-03-23T03:50:50.913000+0000","last_undegraded":"2024-03-23T03:50:50.913000+0000","last_fullsized":"2024-03-23T03:50:50.913000+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:08:20.059038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.912782+0000","last_change":"2024-03-23T03:50:49.338167+0000","last_active":"2024-03-23T03:50:50.912782+0000","last_peered":"2024-03-23T03:50:50.912782+0000","last_clean":"2024-03-23T03:50:50.912782+0000","last_became_active":"2024-03-23T03:50:49.337851+0000","last_became_peered":"2024-03-23T03:50:49.337851+0000","last_unstale":"2024-03-23T03:50:50.912782+0000","last_undegraded":"2024-03-23T03:50:50.912782+0000","last_fullsized":"2024-03-23T03:50:50.912782+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:54:28.049512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.315354+0000","last_change":"2024-03-23T03:50:48.850053+0000","last_active":"2024-03-23T03:50:49.315354+0000","last_peered":"2024-03-23T03:50:49.315354+0000","last_clean":"2024-03-23T03:50:49.315354+0000","last_became_active":"2024-03-23T03:50:48.849625+0000","last_became_peered":"2024-03-23T03:50:48.849625+0000","last_unstale":"2024-03-23T03:50:49.315354+0000","last_undegraded":"2024-03-23T03:50:49.315354+0000","last_fullsized":"2024-03-23T03:50:49.315354+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:55:05.598199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.912869+0000","last_change":"2024-03-23T03:50:49.336359+0000","last_active":"2024-03-23T03:50:50.912869+0000","last_peered":"2024-03-23T03:50:50.912869+0000","last_clean":"2024-03-23T03:50:50.912869+0000","last_became_active":"2024-03-23T03:50:49.336024+0000","last_became_peered":"2024-03-23T03:50:49.336024+0000","last_unstale":"2024-03-23T03:50:50.912869+0000","last_undegraded":"2024-03-23T03:50:50.912869+0000","last_fullsized":"2024-03-23T03:50:50.912869+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:20:18.585071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.912955+0000","last_change":"2024-03-23T03:50:48.838476+0000","last_active":"2024-03-23T03:50:50.912955+0000","last_peered":"2024-03-23T03:50:50.912955+0000","last_clean":"2024-03-23T03:50:50.912955+0000","last_became_active":"2024-03-23T03:50:48.838116+0000","last_became_peered":"2024-03-23T03:50:48.838116+0000","last_unstale":"2024-03-23T03:50:50.912955+0000","last_undegraded":"2024-03-23T03:50:50.912955+0000","last_fullsized":"2024-03-23T03:50:50.912955+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:14:08.573818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.330672+0000","last_change":"2024-03-23T03:50:49.322198+0000","last_active":"2024-03-23T03:50:49.330672+0000","last_peered":"2024-03-23T03:50:49.330672+0000","last_clean":"2024-03-23T03:50:49.330672+0000","last_became_active":"2024-03-23T03:50:49.321933+0000","last_became_peered":"2024-03-23T03:50:49.321933+0000","last_unstale":"2024-03-23T03:50:49.330672+0000","last_undegraded":"2024-03-23T03:50:49.330672+0000","last_fullsized":"2024-03-23T03:50:49.330672+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:56:35.624720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.316507+0000","last_change":"2024-03-23T03:50:48.848822+0000","last_active":"2024-03-23T03:50:49.316507+0000","last_peered":"2024-03-23T03:50:49.316507+0000","last_clean":"2024-03-23T03:50:49.316507+0000","last_became_active":"2024-03-23T03:50:48.847281+0000","last_became_peered":"2024-03-23T03:50:48.847281+0000","last_unstale":"2024-03-23T03:50:49.316507+0000","last_undegraded":"2024-03-23T03:50:49.316507+0000","last_fullsized":"2024-03-23T03:50:49.316507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:28:51.556186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.342332+0000","last_change":"2024-03-23T03:50:49.333673+0000","last_active":"2024-03-23T03:50:49.342332+0000","last_peered":"2024-03-23T03:50:49.342332+0000","last_clean":"2024-03-23T03:50:49.342332+0000","last_became_active":"2024-03-23T03:50:49.333389+0000","last_became_peered":"2024-03-23T03:50:49.333389+0000","last_unstale":"2024-03-23T03:50:49.342332+0000","last_undegraded":"2024-03-23T03:50:49.342332+0000","last_fullsized":"2024-03-23T03:50:49.342332+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:41:45.508345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"92'2","reported_seq":13,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.313442+0000","last_change":"2024-03-23T03:50:48.850373+0000","last_active":"2024-03-23T03:50:50.313442+0000","last_peered":"2024-03-23T03:50:50.313442+0000","last_clean":"2024-03-23T03:50:50.313442+0000","last_became_active":"2024-03-23T03:50:48.850091+0000","last_became_peered":"2024-03-23T03:50:48.850091+0000","last_unstale":"2024-03-23T03:50:50.313442+0000","last_undegraded":"2024-03-23T03:50:50.313442+0000","last_fullsized":"2024-03-23T03:50:50.313442+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:28:17.670259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.356532+0000","last_change":"2024-03-23T03:50:49.348860+0000","last_active":"2024-03-23T03:50:49.356532+0000","last_peered":"2024-03-23T03:50:49.356532+0000","last_clean":"2024-03-23T03:50:49.356532+0000","last_became_active":"2024-03-23T03:50:49.348544+0000","last_became_peered":"2024-03-23T03:50:49.348544+0000","last_unstale":"2024-03-23T03:50:49.356532+0000","last_undegraded":"2024-03-23T03:50:49.356532+0000","last_fullsized":"2024-03-23T03:50:49.356532+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:14:41.036541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.339736+0000","last_change":"2024-03-23T03:50:48.848726+0000","last_active":"2024-03-23T03:50:49.339736+0000","last_peered":"2024-03-23T03:50:49.339736+0000","last_clean":"2024-03-23T03:50:49.339736+0000","last_became_active":"2024-03-23T03:50:48.848205+0000","last_became_peered":"2024-03-23T03:50:48.848205+0000","last_unstale":"2024-03-23T03:50:49.339736+0000","last_undegraded":"2024-03-23T03:50:49.339736+0000","last_fullsized":"2024-03-23T03:50:49.339736+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:32:00.919390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.331111+0000","last_change":"2024-03-23T03:50:49.322409+0000","last_active":"2024-03-23T03:50:49.331111+0000","last_peered":"2024-03-23T03:50:49.331111+0000","last_clean":"2024-03-23T03:50:49.331111+0000","last_became_active":"2024-03-23T03:50:49.322181+0000","last_became_peered":"2024-03-23T03:50:49.322181+0000","last_unstale":"2024-03-23T03:50:49.331111+0000","last_undegraded":"2024-03-23T03:50:49.331111+0000","last_fullsized":"2024-03-23T03:50:49.331111+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:04:39.959954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.337705+0000","last_change":"2024-03-23T03:50:48.840992+0000","last_active":"2024-03-23T03:50:49.337705+0000","last_peered":"2024-03-23T03:50:49.337705+0000","last_clean":"2024-03-23T03:50:49.337705+0000","last_became_active":"2024-03-23T03:50:48.840734+0000","last_became_peered":"2024-03-23T03:50:48.840734+0000","last_unstale":"2024-03-23T03:50:49.337705+0000","last_undegraded":"2024-03-23T03:50:49.337705+0000","last_fullsized":"2024-03-23T03:50:49.337705+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:03:39.391624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.913593+0000","last_change":"2024-03-23T03:50:49.351892+0000","last_active":"2024-03-23T03:50:50.913593+0000","last_peered":"2024-03-23T03:50:50.913593+0000","last_clean":"2024-03-23T03:50:50.913593+0000","last_became_active":"2024-03-23T03:50:49.350724+0000","last_became_peered":"2024-03-23T03:50:49.350724+0000","last_unstale":"2024-03-23T03:50:50.913593+0000","last_undegraded":"2024-03-23T03:50:50.913593+0000","last_fullsized":"2024-03-23T03:50:50.913593+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:55:31.916538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.313128+0000","last_change":"2024-03-23T03:50:48.851518+0000","last_active":"2024-03-23T03:50:49.313128+0000","last_peered":"2024-03-23T03:50:49.313128+0000","last_clean":"2024-03-23T03:50:49.313128+0000","last_became_active":"2024-03-23T03:50:48.851162+0000","last_became_peered":"2024-03-23T03:50:48.851162+0000","last_unstale":"2024-03-23T03:50:49.313128+0000","last_undegraded":"2024-03-23T03:50:49.313128+0000","last_fullsized":"2024-03-23T03:50:49.313128+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:26.405998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.325853+0000","last_change":"2024-03-23T03:50:49.323915+0000","last_active":"2024-03-23T03:50:49.325853+0000","last_peered":"2024-03-23T03:50:49.325853+0000","last_clean":"2024-03-23T03:50:49.325853+0000","last_became_active":"2024-03-23T03:50:49.323671+0000","last_became_peered":"2024-03-23T03:50:49.323671+0000","last_unstale":"2024-03-23T03:50:49.325853+0000","last_undegraded":"2024-03-23T03:50:49.325853+0000","last_fullsized":"2024-03-23T03:50:49.325853+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:21:58.644077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.315217+0000","last_change":"2024-03-23T03:50:48.849379+0000","last_active":"2024-03-23T03:50:49.315217+0000","last_peered":"2024-03-23T03:50:49.315217+0000","last_clean":"2024-03-23T03:50:49.315217+0000","last_became_active":"2024-03-23T03:50:48.849119+0000","last_became_peered":"2024-03-23T03:50:48.849119+0000","last_unstale":"2024-03-23T03:50:49.315217+0000","last_undegraded":"2024-03-23T03:50:49.315217+0000","last_fullsized":"2024-03-23T03:50:49.315217+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:16:35.273758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.311718+0000","last_change":"2024-03-23T03:50:49.334236+0000","last_active":"2024-03-23T03:50:50.311718+0000","last_peered":"2024-03-23T03:50:50.311718+0000","last_clean":"2024-03-23T03:50:50.311718+0000","last_became_active":"2024-03-23T03:50:49.333980+0000","last_became_peered":"2024-03-23T03:50:49.333980+0000","last_unstale":"2024-03-23T03:50:50.311718+0000","last_undegraded":"2024-03-23T03:50:50.311718+0000","last_fullsized":"2024-03-23T03:50:50.311718+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:43:51.831367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.912991+0000","last_change":"2024-03-23T03:50:48.841469+0000","last_active":"2024-03-23T03:50:50.912991+0000","last_peered":"2024-03-23T03:50:50.912991+0000","last_clean":"2024-03-23T03:50:50.912991+0000","last_became_active":"2024-03-23T03:50:48.841160+0000","last_became_peered":"2024-03-23T03:50:48.841160+0000","last_unstale":"2024-03-23T03:50:50.912991+0000","last_undegraded":"2024-03-23T03:50:50.912991+0000","last_fullsized":"2024-03-23T03:50:50.912991+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:54:10.680930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.332963+0000","last_change":"2024-03-23T03:50:49.325327+0000","last_active":"2024-03-23T03:50:49.332963+0000","last_peered":"2024-03-23T03:50:49.332963+0000","last_clean":"2024-03-23T03:50:49.332963+0000","last_became_active":"2024-03-23T03:50:49.324794+0000","last_became_peered":"2024-03-23T03:50:49.324794+0000","last_unstale":"2024-03-23T03:50:49.332963+0000","last_undegraded":"2024-03-23T03:50:49.332963+0000","last_fullsized":"2024-03-23T03:50:49.332963+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:29:44.983051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":10,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.310738+0000","last_change":"2024-03-23T03:50:48.851590+0000","last_active":"2024-03-23T03:50:49.310738+0000","last_peered":"2024-03-23T03:50:49.310738+0000","last_clean":"2024-03-23T03:50:49.310738+0000","last_became_active":"2024-03-23T03:50:48.851317+0000","last_became_peered":"2024-03-23T03:50:48.851317+0000","last_unstale":"2024-03-23T03:50:49.310738+0000","last_undegraded":"2024-03-23T03:50:49.310738+0000","last_fullsized":"2024-03-23T03:50:49.310738+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:29:08.608786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":9,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-23T03:50:49.335771+0000","last_change":"2024-03-23T03:50:49.329640+0000","last_active":"2024-03-23T03:50:49.335771+0000","last_peered":"2024-03-23T03:50:49.335771+0000","last_clean":"2024-03-23T03:50:49.335771+0000","last_became_active":"2024-03-23T03:50:49.329356+0000","last_became_peered":"2024-03-23T03:50:49.329356+0000","last_unstale":"2024-03-23T03:50:49.335771+0000","last_undegraded":"2024-03-23T03:50:49.335771+0000","last_fullsized":"2024-03-23T03:50:49.335771+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:10:17.396462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.312507+0000","last_change":"2024-03-23T03:50:48.844024+0000","last_active":"2024-03-23T03:50:50.312507+0000","last_peered":"2024-03-23T03:50:50.312507+0000","last_clean":"2024-03-23T03:50:50.312507+0000","last_became_active":"2024-03-23T03:50:48.843760+0000","last_became_peered":"2024-03-23T03:50:48.843760+0000","last_unstale":"2024-03-23T03:50:50.312507+0000","last_undegraded":"2024-03-23T03:50:50.312507+0000","last_fullsized":"2024-03-23T03:50:50.312507+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:14:08.573818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":10,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.313260+0000","last_change":"2024-03-23T03:50:49.342021+0000","last_active":"2024-03-23T03:50:50.313260+0000","last_peered":"2024-03-23T03:50:50.313260+0000","last_clean":"2024-03-23T03:50:50.313260+0000","last_became_active":"2024-03-23T03:50:49.341746+0000","last_became_peered":"2024-03-23T03:50:49.341746+0000","last_unstale":"2024-03-23T03:50:50.313260+0000","last_undegraded":"2024-03-23T03:50:50.313260+0000","last_fullsized":"2024-03-23T03:50:50.313260+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:48.805561+0000","last_clean_scrub_stamp":"2024-03-23T03:50:48.805561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:31:49.474231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.561455+0000","last_change":"2024-03-23T03:50:48.842760+0000","last_active":"2024-03-23T03:50:50.561455+0000","last_peered":"2024-03-23T03:50:50.561455+0000","last_clean":"2024-03-23T03:50:50.561455+0000","last_became_active":"2024-03-23T03:50:48.842451+0000","last_became_peered":"2024-03-23T03:50:48.842451+0000","last_unstale":"2024-03-23T03:50:50.561455+0000","last_undegraded":"2024-03-23T03:50:50.561455+0000","last_fullsized":"2024-03-23T03:50:50.561455+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":90,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:50:47.800987+0000","last_clean_scrub_stamp":"2024-03-23T03:50:47.800987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:53:21.634284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":223,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-23T03:50:50.312704+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:50:50.312704+0000","last_peered":"2024-03-23T03:50:50.312704+0000","last_clean":"2024-03-23T03:50:50.312704+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:50:50.312704+0000","last_undegraded":"2024-03-23T03:50:50.312704+0000","last_fullsized":"2024-03-23T03:50:50.312704+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542197,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32828,"kb_used_data":1704,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339012,"statfs":{"total":96636764160,"available":96603148288,"internally_reserved":0,"allocated":1744896,"data_stored":1107652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.079,"15min":1.069},"min":{"1min":0.68200000000000005,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.573,"5min":1.635,"15min":1.635},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.0449999999999999,"15min":1.048},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.417,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.0900000000000001}]},{"osd":1,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0549999999999999,"15min":1.044},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.544,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.085,"5min":1.2330000000000001,"15min":1.3009999999999999},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.3999999999999999,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":0.93799999999999994}]},{"osd":2,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1240000000000001,"5min":1.139,"15min":1.1559999999999999},"min":{"1min":0.76400000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.7909999999999999,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.345,"15min":1.421},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.4630000000000001,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.506}]},{"osd":3,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.355,"15min":1.4370000000000001},"min":{"1min":0.873,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.397,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.22},{"interface":"front","average":{"1min":1.27,"5min":1.3380000000000001,"15min":1.3580000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":1.5840000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.677}]},{"osd":4,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.266,"15min":1.2669999999999999},"min":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"max":{"1min":1.6299999999999999,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.25},{"interface":"front","average":{"1min":1.2669999999999999,"5min":1.357,"15min":1.375},"min":{"1min":0.77900000000000003,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.0619999999999998,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.3140000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.3999999999999999,"15min":1.4219999999999999},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.6539999999999999,"5min":2.645,"15min":2.645},"last":1.7310000000000001},{"interface":"front","average":{"1min":1.339,"5min":1.4370000000000001,"15min":1.4530000000000001},"min":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":1.7050000000000001,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.3660000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.5229999999999999,"15min":1.5589999999999999},"min":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"max":{"1min":1.8200000000000001,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.4390000000000001},{"interface":"front","average":{"1min":1.415,"5min":1.4930000000000001,"15min":1.5209999999999999},"min":{"1min":1.1319999999999999,"5min":0.92400000000000004,"15min":0.92400000000000004},"max":{"1min":2.1509999999999998,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.47}]}]},{"osd":6,"up_from":14,"seq":60129542195,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32252,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339588,"statfs":{"total":96636764160,"available":96603738112,"internally_reserved":0,"allocated":1150976,"data_stored":512776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.0309999999999999,"15min":1.04},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.3979999999999999,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.167},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0429999999999999,"15min":1.038},"min":{"1min":0.72099999999999997,"5min":0.62,"15min":0.62},"max":{"1min":1.6120000000000001,"5min":1.946,"15min":1.946},"last":1.4239999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.71799999999999997,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.623,"5min":1.623,"15min":1.623},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.129,"15min":1.1559999999999999},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.4810000000000001,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.268}]},{"osd":2,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.075,"15min":1.069},"min":{"1min":0.73699999999999999,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.5449999999999999,"5min":2.044,"15min":2.044},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1950000000000001,"15min":1.218},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.681,"5min":1.681,"15min":1.681},"last":1.0820000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2629999999999999,"15min":1.2849999999999999},"min":{"1min":0.76800000000000002,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.4910000000000001,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2549999999999999,"15min":1.2649999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.6319999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.446}]},{"osd":4,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3759999999999999,"15min":1.3839999999999999},"min":{"1min":0.67700000000000005,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.73,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.371},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.363,"15min":1.383},"min":{"1min":0.63400000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.762,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.5329999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.415,"15min":1.4359999999999999},"min":{"1min":0.71099999999999997,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.6919999999999999,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.456,"5min":1.5249999999999999,"15min":1.5469999999999999},"min":{"1min":0.79700000000000004,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8149999999999999,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.5669999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:49:54 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.4059999999999999,"15min":1.4119999999999999},"min":{"1min":0.77600000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.802,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.5129999999999999,"15min":1.5409999999999999},"min":{"1min":0.84099999999999997,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.74,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.2969999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542195,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28144,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343696,"statfs":{"total":96636764160,"available":96607944704,"internally_reserved":0,"allocated":1142784,"data_stored":517228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.1759999999999999,"15min":1.214},"min":{"1min":0.67900000000000005,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.6779999999999999,"5min":1.806,"15min":1.806},"last":1.095},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.157,"15min":1.181},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.554,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.855}]},{"osd":1,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.1040000000000001,"15min":1.1040000000000001},"min":{"1min":0.81699999999999995,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.5489999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.026,"5min":1.1599999999999999,"15min":1.1919999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4219999999999999,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.6419999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.298,"15min":1.3540000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5429999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.8939999999999999},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.3340000000000001,"15min":1.3779999999999999},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.772,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.2609999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.21,"15min":1.2470000000000001},"min":{"1min":0.70599999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.653,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.6739999999999999},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.393,"15min":1.452},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.6779999999999999,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.724}]},{"osd":4,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.3420000000000001,"15min":1.4119999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.671,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.488},{"interface":"front","average":{"1min":1.282,"5min":1.4119999999999999,"15min":1.444},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.77,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.5549999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.458,"15min":1.514},"min":{"1min":0.76700000000000002,"5min":0.63,"15min":0.63},"max":{"1min":1.873,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.5940000000000001,"15min":1.6559999999999999},"min":{"1min":0.86199999999999999,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.956,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.528}]},{"osd":7,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.335,"5min":1.538,"15min":1.607},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.673,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.9259999999999999},{"interface":"front","average":{"1min":1.411,"5min":1.6200000000000001,"15min":1.696},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.855,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.4530000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542196,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32236,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339604,"statfs":{"total":96636764160,"available":96603754496,"internally_reserved":0,"allocated":1138688,"data_stored":510768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.002,"15min":0.97999999999999998},"min":{"1min":0.67800000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.7789999999999999,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.03,"5min":1.0149999999999999,"15min":1.012},"min":{"1min":0.81200000000000006,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3069999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.83699999999999997}]},{"osd":1,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.0700000000000001,"15min":1.05},"min":{"1min":0.74299999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.704,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.147,"5min":1.0880000000000001,"15min":1.0640000000000001},"min":{"1min":0.50600000000000001,"5min":0.496,"15min":0.496},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.2450000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.2949999999999999,"15min":1.325},"min":{"1min":0.81599999999999995,"5min":0.625,"15min":0.625},"max":{"1min":1.569,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.29,"5min":1.3100000000000001,"15min":1.3080000000000001},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.8680000000000001,"5min":2.181,"15min":2.181},"last":1.3700000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.1930000000000001,"5min":1.1339999999999999,"15min":1.1299999999999999},"min":{"1min":0.73199999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.821,"5min":1.821,"15min":1.821},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.26,"15min":1.278},"min":{"1min":0.82899999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.5720000000000001,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.3129999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.278,"15min":1.274},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.929,"5min":1.929,"15min":1.929},"last":1.49},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.298,"15min":1.2829999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.2349999999999999,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.4119999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.415,"15min":1.4239999999999999},"min":{"1min":1.143,"5min":0.748,"15min":0.748},"max":{"1min":1.833,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.478,"5min":1.425,"15min":1.4039999999999999},"min":{"1min":0.745,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":1.5609999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:49:50 2024","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.385,"15min":1.371},"min":{"1min":1.0880000000000001,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":2.3479999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.4970000000000001,"15min":1.5009999999999999},"min":{"1min":1.056,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.974,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.6870000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542196,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32240,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339600,"statfs":{"total":96636764160,"available":96603750400,"internally_reserved":0,"allocated":1142784,"data_stored":517228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.83899999999999997,"15min":0.86099999999999999},"min":{"1min":0.37,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.256,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.92500000000000004,"15min":0.92500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.446,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.66800000000000004}]},{"osd":1,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.92800000000000005,"15min":0.97199999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.997,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.034,"15min":1.0720000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.712,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.73799999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":1.006,"15min":1.052},"min":{"1min":0.435,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.613,"5min":1.823,"15min":1.823},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.1319999999999999,"15min":1.153},"min":{"1min":0.48199999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.6020000000000001,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":0.83899999999999997}]},{"osd":4,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1000000000000001,"5min":1.2310000000000001,"15min":1.2450000000000001},"min":{"1min":0.67600000000000005,"5min":0.625,"15min":0.625},"max":{"1min":1.3859999999999999,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.151,"5min":1.377,"15min":1.4370000000000001},"min":{"1min":0.77100000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.4770000000000001,"5min":4.819,"15min":4.819},"last":1.681}]},{"osd":5,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.603,"15min":1.4470000000000001},"min":{"1min":0.75,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.0299999999999998,"5min":30.228000000000002,"15min":30.228000000000002},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.3320000000000001,"15min":1.355},"min":{"1min":0.91400000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.637,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":1.4370000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.448,"15min":1.52},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.5860000000000001,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.5389999999999999},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.3999999999999999,"15min":1.397},"min":{"1min":0.90100000000000002,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.1429999999999998,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":1.2729999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.395,"15min":1.4219999999999999},"min":{"1min":0.81200000000000006,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.71,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":1.579},{"interface":"front","average":{"1min":1.298,"5min":1.577,"15min":1.601},"min":{"1min":0.73299999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.629,"5min":10.663,"15min":10.663},"last":1.6359999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542196,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32228,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339612,"statfs":{"total":96636764160,"available":96603762688,"internally_reserved":0,"allocated":1130496,"data_stored":510608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.80100000000000005,"5min":0.84999999999999998,"15min":0.88100000000000001},"min":{"1min":0.318,"5min":0.307,"15min":0.307},"max":{"1min":1.456,"5min":1.456,"15min":1.456},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.88700000000000001,"15min":0.94299999999999995},"min":{"1min":0.53500000000000003,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.23,"5min":2.226,"15min":2.226},"last":1.333}]},{"osd":1,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.96899999999999997,"15min":0.995},"min":{"1min":0.621,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.5109999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":1.377},{"interface":"front","average":{"1min":0.82999999999999996,"5min":0.88,"15min":0.91700000000000004},"min":{"1min":0.53700000000000003,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.282,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.73199999999999998}]},{"osd":3,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.98499999999999999,"15min":1.026},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.4750000000000001,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.0549999999999999,"15min":1.085},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.3380000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":1.2130000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.2090000000000001,"15min":1.236},"min":{"1min":0.96399999999999997,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.488,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.444},{"interface":"front","average":{"1min":1.087,"5min":1.179,"15min":1.214},"min":{"1min":0.66300000000000003,"5min":0.627,"15min":0.627},"max":{"1min":1.47,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.0800000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.3360000000000001,"15min":1.3759999999999999},"min":{"1min":0.76200000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.304,"15min":1.349},"min":{"1min":0.75,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.5880000000000001,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.2609999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.268,"15min":1.3400000000000001},"min":{"1min":0.82199999999999995,"5min":0.753,"15min":0.753},"max":{"1min":1.5109999999999999,"5min":2.456,"15min":2.456},"last":1.125},{"interface":"front","average":{"1min":1.359,"5min":1.4390000000000001,"15min":1.4890000000000001},"min":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.3939999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:49:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.335,"15min":1.379},"min":{"1min":0.82699999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.6870000000000001,"5min":1.8120000000000001,"15min":1.8120000000000001},"last":1.498},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.3460000000000001,"15min":1.4099999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5900000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.421}]}]},{"osd":1,"up_from":14,"seq":60129542195,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28148,"kb_used_data":1120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343692,"statfs":{"total":96636764160,"available":96607940608,"internally_reserved":0,"allocated":1146880,"data_stored":517298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.81599999999999995,"15min":0.87},"min":{"1min":0.40200000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.1819999999999999,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.82999999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.56499999999999995}]},{"osd":2,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.92800000000000005,"15min":0.96599999999999997},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.4570000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.94899999999999995,"15min":0.98599999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.4750000000000001,"5min":1.625,"15min":1.625},"last":0.93000000000000005}]},{"osd":3,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.117,"15min":1.117},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.7929999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.99399999999999999,"15min":1.0189999999999999},"min":{"1min":0.57399999999999995,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.5600000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.621}]},{"osd":4,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.232,"15min":1.292},"min":{"1min":0.69199999999999995,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.752,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.2749999999999999,"15min":1.3149999999999999},"min":{"1min":0.82599999999999996,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.575,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.0309999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1259999999999999,"5min":1.2729999999999999,"15min":1.329},"min":{"1min":0.82399999999999995,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.7470000000000001,"5min":3.593,"15min":3.593},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.276,"5min":1.3700000000000001,"15min":1.4399999999999999},"min":{"1min":0.76200000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6719999999999999,"5min":2.359,"15min":2.359},"last":0.96099999999999997}]},{"osd":6,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.349,"15min":1.405},"min":{"1min":0.79300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":2.1709999999999998,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.147,"5min":1.302,"15min":1.3580000000000001},"min":{"1min":0.77800000000000002,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.696,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":0.89800000000000002}]},{"osd":7,"last update":"Sat Mar 23 03:49:55 2024","interfaces":[{"interface":"back","average":{"1min":1.218,"5min":1.3759999999999999,"15min":1.4379999999999999},"min":{"1min":0.81100000000000005,"5min":0.627,"15min":0.627},"max":{"1min":1.7649999999999999,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.4219999999999999,"15min":1.506},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.7569999999999999,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.28}]}]},{"osd":0,"up_from":14,"seq":60129542195,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32828,"kb_used_data":1708,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339012,"statfs":{"total":96636764160,"available":96603148288,"internally_reserved":0,"allocated":1748992,"data_stored":1103246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.90000000000000002,"15min":0.94699999999999995},"min":{"1min":0.435,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.1599999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.98899999999999999},"min":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":1.3640000000000001,"5min":1.506,"15min":1.506},"last":0.81699999999999995}]},{"osd":2,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.95799999999999996,"15min":1.0009999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.1279999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.91900000000000004,"5min":1.0640000000000001,"15min":1.105},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.4350000000000001,"5min":1.901,"15min":1.901},"last":1.3200000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.97499999999999998,"15min":1.01},"min":{"1min":0.48999999999999999,"5min":0.433,"15min":0.433},"max":{"1min":1.5189999999999999,"5min":1.972,"15min":1.972},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.99199999999999999,"5min":1.103,"15min":1.159},"min":{"1min":0.51400000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.3180000000000001,"5min":2.298,"15min":2.298},"last":1.2949999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.304,"15min":1.3420000000000001},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.599,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":1.647},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.252,"15min":1.28},"min":{"1min":0.83099999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.4990000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.095}]},{"osd":5,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.3340000000000001,"15min":1.3779999999999999},"min":{"1min":0.878,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.4450000000000001,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.2929999999999999,"5min":1.403,"15min":1.4470000000000001},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.49,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.4450000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.46,"15min":1.538},"min":{"1min":0.85799999999999998,"5min":0.747,"15min":0.747},"max":{"1min":1.5940000000000001,"5min":3.032,"15min":3.032},"last":1.5169999999999999},{"interface":"front","average":{"1min":1.28,"5min":1.4419999999999999,"15min":1.498},"min":{"1min":0.98199999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.5740000000000001,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":2.0459999999999998}]},{"osd":7,"last update":"Sat Mar 23 03:49:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.46,"15min":1.5129999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.7110000000000001,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.9339999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.3939999999999999,"15min":1.4299999999999999},"min":{"1min":0.871,"5min":0.85299999999999998,"15min":0.85299999999999998},"max":{"1min":2.008,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.873}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:50:54.195 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T03:50:54.195 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-23T03:50:54.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.458+0000 7ff450a00640 1 Processor -- start 2024-03-23T03:50:54.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.459+0000 7ff450a00640 1 -- start start 2024-03-23T03:50:54.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.460+0000 7ff450a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff44c0add30 0x7ff44c0ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:54.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.460+0000 7ff450a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff44c06ce90 0x7ff44c06d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:54.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.460+0000 7ff450a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff44c0cf250 con 0x7ff44c0ce930 2024-03-23T03:50:54.461 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.460+0000 7ff450a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff44c0ae1c0 con 0x7ff44c0add30 2024-03-23T03:50:54.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.460+0000 7ff450a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c06c760 con 0x7ff44c06ce90 2024-03-23T03:50:54.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.461+0000 7ff449e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff44c0ce930 0x7ff44c0cd3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55542/0 (socket says 172.21.15.102:55542) 2024-03-23T03:50:54.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.461+0000 7ff449e00640 1 -- 172.21.15.102:0/3125871131 learned_addr learned my addr 172.21.15.102:0/3125871131 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:54.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.461+0000 7ff441400640 1 --2- 172.21.15.102:0/3125871131 >> v2:172.21.15.102:3300/0 conn(0x7ff44c0add30 0x7ff44c0ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:54.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.461+0000 7ff44a800640 1 --2- 172.21.15.102:0/3125871131 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff44c06ce90 0x7ff44c06d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.461+0000 7ff449400640 1 -- 172.21.15.102:0/3125871131 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082383267 0 0) 0x7ff44c0cf250 con 0x7ff44c0ce930 2024-03-23T03:50:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.461+0000 7ff449400640 1 -- 172.21.15.102:0/3125871131 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff430003520 con 0x7ff44c0ce930 2024-03-23T03:50:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.461+0000 7ff441400640 1 -- 172.21.15.102:0/3125871131 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff44c06ce90 msgr2=0x7ff44c06d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.461+0000 7ff441400640 1 --2- 172.21.15.102:0/3125871131 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff44c06ce90 0x7ff44c06d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.461+0000 7ff441400640 1 -- 172.21.15.102:0/3125871131 >> v1:172.21.15.102:6789/0 conn(0x7ff44c0ce930 legacy=0x7ff44c0cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:54.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.461+0000 7ff441400640 1 -- 172.21.15.102:0/3125871131 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44c0cdba0 con 0x7ff44c0add30 2024-03-23T03:50:54.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.463+0000 7ff441400640 1 --2- 172.21.15.102:0/3125871131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff44c0add30 0x7ff44c0ced10 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7ff4340097a0 tx=0x7ff43402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24be6b8610114fb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:54.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.463+0000 7ff449400640 1 -- 172.21.15.102:0/3125871131 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff434031830 con 0x7ff44c0add30 2024-03-23T03:50:54.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.463+0000 7ff449400640 1 -- 172.21.15.102:0/3125871131 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff434031990 con 0x7ff44c0add30 2024-03-23T03:50:54.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.463+0000 7ff449400640 1 -- 172.21.15.102:0/3125871131 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff434031d20 con 0x7ff44c0add30 2024-03-23T03:50:54.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.463+0000 7ff450a00640 1 -- 172.21.15.102:0/3125871131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff44c0add30 msgr2=0x7ff44c0ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:54.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.463+0000 7ff450a00640 1 --2- 172.21.15.102:0/3125871131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff44c0add30 0x7ff44c0ced10 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7ff4340097a0 tx=0x7ff43402b8d0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.464+0000 7ff450a00640 1 -- 172.21.15.102:0/3125871131 shutdown_connections 2024-03-23T03:50:54.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.464+0000 7ff450a00640 1 --2- 172.21.15.102:0/3125871131 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff44c06ce90 0x7ff44c06d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.464+0000 7ff450a00640 1 --2- 172.21.15.102:0/3125871131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff44c0add30 0x7ff44c0ced10 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.464+0000 7ff450a00640 1 -- 172.21.15.102:0/3125871131 >> 172.21.15.102:0/3125871131 conn(0x7ff44c06a0b0 msgr2=0x7ff44c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:54.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.464+0000 7ff450a00640 1 -- 172.21.15.102:0/3125871131 shutdown_connections 2024-03-23T03:50:54.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.464+0000 7ff450a00640 1 -- 172.21.15.102:0/3125871131 wait complete. 2024-03-23T03:50:54.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.465+0000 7ff450a00640 1 Processor -- start 2024-03-23T03:50:54.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.466+0000 7ff450a00640 1 -- start start 2024-03-23T03:50:54.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.466+0000 7ff450a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff44c06ce90 0x7ff44c148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:54.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.467+0000 7ff450a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff44c0add30 0x7ff44c0b1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:54.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.467+0000 7ff450a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff44c0ce930 0x7ff44c15cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:54.467 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.467+0000 7ff450a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c148750 con 0x7ff44c0add30 2024-03-23T03:50:54.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.467+0000 7ff450a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c0acc70 con 0x7ff44c06ce90 2024-03-23T03:50:54.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.467+0000 7ff450a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff44c1f5a60 con 0x7ff44c0ce930 2024-03-23T03:50:54.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.467+0000 7ff441400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff44c0add30 0x7ff44c0b1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:54.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.467+0000 7ff441400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff44c0add30 0x7ff44c0b1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57980/0 (socket says 172.21.15.102:57980) 2024-03-23T03:50:54.468 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.467+0000 7ff441400640 1 -- 172.21.15.102:0/3319528653 learned_addr learned my addr 172.21.15.102:0/3319528653 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:54.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.467+0000 7ff44a800640 1 --2- 172.21.15.102:0/3319528653 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff44c0ce930 0x7ff44c15cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:54.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.467+0000 7ff449e00640 1 --2- 172.21.15.102:0/3319528653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff44c06ce90 0x7ff44c148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:54.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.467+0000 7ff441400640 1 -- 172.21.15.102:0/3319528653 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff44c0ce930 msgr2=0x7ff44c15cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:54.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.467+0000 7ff441400640 1 --2- 172.21.15.102:0/3319528653 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff44c0ce930 0x7ff44c15cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.467+0000 7ff441400640 1 -- 172.21.15.102:0/3319528653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff44c06ce90 msgr2=0x7ff44c148210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:54.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.467+0000 7ff441400640 1 --2- 172.21.15.102:0/3319528653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff44c06ce90 0x7ff44c148210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.467+0000 7ff441400640 1 -- 172.21.15.102:0/3319528653 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff434009400 con 0x7ff44c0add30 2024-03-23T03:50:54.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.467+0000 7ff449e00640 1 --2- 172.21.15.102:0/3319528653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff44c06ce90 0x7ff44c148210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:54.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.468+0000 7ff441400640 1 --2- 172.21.15.102:0/3319528653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff44c0add30 0x7ff44c0b1770 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7ff43402bee0 tx=0x7ff4340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:54.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.468+0000 7ff442a00640 1 -- 172.21.15.102:0/3319528653 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43403a6f0 con 0x7ff44c0add30 2024-03-23T03:50:54.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.468+0000 7ff442a00640 1 -- 172.21.15.102:0/3319528653 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff43403a870 con 0x7ff44c0add30 2024-03-23T03:50:54.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.468+0000 7ff442a00640 1 -- 172.21.15.102:0/3319528653 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43403ac20 con 0x7ff44c0add30 2024-03-23T03:50:54.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.468+0000 7ff450a00640 1 -- 172.21.15.102:0/3319528653 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44c1f5d40 con 0x7ff44c0add30 2024-03-23T03:50:54.470 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.468+0000 7ff450a00640 1 -- 172.21.15.102:0/3319528653 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff44c1f6300 con 0x7ff44c0add30 2024-03-23T03:50:54.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.469+0000 7ff450a00640 1 -- 172.21.15.102:0/3319528653 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff44c073390 con 0x7ff44c0add30 2024-03-23T03:50:54.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.470+0000 7ff442a00640 1 -- 172.21.15.102:0/3319528653 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff434030380 con 0x7ff44c0add30 2024-03-23T03:50:54.471 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.471+0000 7ff442a00640 1 --2- 172.21.15.102:0/3319528653 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff42806ac00 0x7ff42806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:54.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.471+0000 7ff442a00640 1 -- 172.21.15.102:0/3319528653 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff4340aadf0 con 0x7ff44c0add30 2024-03-23T03:50:54.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.471+0000 7ff449e00640 1 --2- 172.21.15.102:0/3319528653 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff42806ac00 0x7ff42806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:54.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.472+0000 7ff449e00640 1 --2- 172.21.15.102:0/3319528653 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff42806ac00 0x7ff42806d0c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0011e0 tx=0x7ff43c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:54.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.476+0000 7ff442a00640 1 -- 172.21.15.102:0/3319528653 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff434075fa0 con 0x7ff44c0add30 2024-03-23T03:50:54.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.742+0000 7ff450a00640 1 -- 172.21.15.102:0/3319528653 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff44c0ceff0 con 0x7ff44c0add30 2024-03-23T03:50:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.750+0000 7ff442a00640 1 -- 172.21.15.102:0/3319528653 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+0 (secure 0 0 0) 0x7ff4340350e0 con 0x7ff44c0add30 2024-03-23T03:50:54.750 INFO:teuthology.orchestra.run.smithi102.stderr:updated caps for client.0 2024-03-23T03:50:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.753+0000 7ff450a00640 1 -- 172.21.15.102:0/3319528653 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff42806ac00 msgr2=0x7ff42806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.753+0000 7ff450a00640 1 --2- 172.21.15.102:0/3319528653 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff42806ac00 0x7ff42806d0c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff43c0011e0 tx=0x7ff43c009040 comp rx=0 tx=0).stop 2024-03-23T03:50:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.753+0000 7ff450a00640 1 -- 172.21.15.102:0/3319528653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff44c0add30 msgr2=0x7ff44c0b1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.753+0000 7ff450a00640 1 --2- 172.21.15.102:0/3319528653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff44c0add30 0x7ff44c0b1770 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7ff43402bee0 tx=0x7ff4340046c0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.753+0000 7ff450a00640 1 -- 172.21.15.102:0/3319528653 shutdown_connections 2024-03-23T03:50:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.753+0000 7ff450a00640 1 --2- 172.21.15.102:0/3319528653 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff42806ac00 0x7ff42806d0c0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.753+0000 7ff450a00640 1 --2- 172.21.15.102:0/3319528653 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff44c0ce930 0x7ff44c15cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.753+0000 7ff450a00640 1 --2- 172.21.15.102:0/3319528653 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff44c0add30 0x7ff44c0b1770 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.753+0000 7ff450a00640 1 --2- 172.21.15.102:0/3319528653 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff44c06ce90 0x7ff44c148210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.753+0000 7ff450a00640 1 -- 172.21.15.102:0/3319528653 >> 172.21.15.102:0/3319528653 conn(0x7ff44c06a0b0 msgr2=0x7ff44c0b0b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.753+0000 7ff450a00640 1 -- 172.21.15.102:0/3319528653 shutdown_connections 2024-03-23T03:50:54.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.753+0000 7ff450a00640 1 -- 172.21.15.102:0/3319528653 wait complete. 2024-03-23T03:50:54.774 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:50:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.977+0000 7f82cbe00640 1 Processor -- start 2024-03-23T03:50:54.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.978+0000 7f82cbe00640 1 -- start start 2024-03-23T03:50:54.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.979+0000 7f82cbe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82c40b5a30 0x7f82c40b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:54.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.979+0000 7f82cbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f82c40b4350 0x7f82c40b47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:54.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.979+0000 7f82cbe00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82c40d1830 con 0x7f82c415cda0 2024-03-23T03:50:54.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.979+0000 7f82cbe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82c40b4d20 con 0x7f82c40b4350 2024-03-23T03:50:54.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.979+0000 7f82cbe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c40b1f20 con 0x7f82c40b5a30 2024-03-23T03:50:54.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.979+0000 7f82c9e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f82c40b4350 0x7f82c40b47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:54.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.979+0000 7f82c9e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f82c40b4350 0x7f82c40b47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57990/0 (socket says 172.21.15.102:57990) 2024-03-23T03:50:54.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.979+0000 7f82c9e00640 1 -- 172.21.15.102:0/2660753640 learned_addr learned my addr 172.21.15.102:0/2660753640 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:54.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.980+0000 7f82c9e00640 1 -- 172.21.15.102:0/2660753640 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82c40b5a30 msgr2=0x7f82c40b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:54.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.980+0000 7f82c8a00640 1 --2- 172.21.15.102:0/2660753640 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82c40b5a30 0x7f82c40b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:54.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.980+0000 7f82c9e00640 1 --2- 172.21.15.102:0/2660753640 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82c40b5a30 0x7f82c40b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.980+0000 7f82c9e00640 1 -- 172.21.15.102:0/2660753640 >> v1:172.21.15.173:6789/0 conn(0x7f82c415cda0 legacy=0x7f82c40cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:54.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.980+0000 7f82c9e00640 1 -- 172.21.15.102:0/2660753640 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c40d0c20 con 0x7f82c40b4350 2024-03-23T03:50:54.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.980+0000 7f82c8a00640 1 --2- 172.21.15.102:0/2660753640 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82c40b5a30 0x7f82c40b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:54.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.981+0000 7f82c9e00640 1 --2- 172.21.15.102:0/2660753640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82c40b4350 0x7f82c40b47e0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f82bc009e50 tx=0x7f82bc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb056cd83647c1a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:54.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.981+0000 7f82c3e00640 1 -- 172.21.15.102:0/2660753640 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc02ca70 con 0x7f82c40b4350 2024-03-23T03:50:54.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.981+0000 7f82c3e00640 1 -- 172.21.15.102:0/2660753640 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82bc02cbd0 con 0x7f82c40b4350 2024-03-23T03:50:54.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.981+0000 7f82c3e00640 1 -- 172.21.15.102:0/2660753640 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc0343f0 con 0x7f82c40b4350 2024-03-23T03:50:54.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.982+0000 7f82cbe00640 1 -- 172.21.15.102:0/2660753640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82c40b4350 msgr2=0x7f82c40b47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:54.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.982+0000 7f82cbe00640 1 --2- 172.21.15.102:0/2660753640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82c40b4350 0x7f82c40b47e0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f82bc009e50 tx=0x7f82bc02bed0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.982+0000 7f82cbe00640 1 -- 172.21.15.102:0/2660753640 shutdown_connections 2024-03-23T03:50:54.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.982+0000 7f82cbe00640 1 --2- 172.21.15.102:0/2660753640 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82c40b4350 0x7f82c40b47e0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.982+0000 7f82cbe00640 1 --2- 172.21.15.102:0/2660753640 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82c40b5a30 0x7f82c40b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.982+0000 7f82cbe00640 1 -- 172.21.15.102:0/2660753640 >> 172.21.15.102:0/2660753640 conn(0x7f82c40cf350 msgr2=0x7f82c40cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:54.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.982+0000 7f82cbe00640 1 -- 172.21.15.102:0/2660753640 shutdown_connections 2024-03-23T03:50:54.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.982+0000 7f82cbe00640 1 -- 172.21.15.102:0/2660753640 wait complete. 2024-03-23T03:50:54.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.984+0000 7f82cbe00640 1 Processor -- start 2024-03-23T03:50:54.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.984+0000 7f82cbe00640 1 -- start start 2024-03-23T03:50:54.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.985+0000 7f82cbe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82c40b4350 0x7f82c4157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:54.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.985+0000 7f82cbe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82c40b5a30 0x7f82c415d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:54.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.985+0000 7f82cbe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82c415cda0 0x7f82c40caf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:54.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.985+0000 7f82cbe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c40c9eb0 con 0x7f82c415cda0 2024-03-23T03:50:54.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.985+0000 7f82cbe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c41576a0 con 0x7f82c40b5a30 2024-03-23T03:50:54.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.985+0000 7f82cbe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82c40cb9c0 con 0x7f82c40b4350 2024-03-23T03:50:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.985+0000 7f82c9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82c415cda0 0x7f82c40caf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.985+0000 7f82c9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82c415cda0 0x7f82c40caf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:57994/0 (socket says 172.21.15.102:57994) 2024-03-23T03:50:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.985+0000 7f82c9e00640 1 -- 172.21.15.102:0/2913563711 learned_addr learned my addr 172.21.15.102:0/2913563711 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.985+0000 7f82c8a00640 1 --2- 172.21.15.102:0/2913563711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82c40b5a30 0x7f82c415d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.985+0000 7f82c9400640 1 --2- 172.21.15.102:0/2913563711 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82c40b4350 0x7f82c4157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.985+0000 7f82c9e00640 1 -- 172.21.15.102:0/2913563711 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82c40b4350 msgr2=0x7f82c4157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.985+0000 7f82c9e00640 1 --2- 172.21.15.102:0/2913563711 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82c40b4350 0x7f82c4157160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.985+0000 7f82c9e00640 1 -- 172.21.15.102:0/2913563711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82c40b5a30 msgr2=0x7f82c415d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:54.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.985+0000 7f82c9e00640 1 --2- 172.21.15.102:0/2913563711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82c40b5a30 0x7f82c415d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:54.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.985+0000 7f82c9e00640 1 -- 172.21.15.102:0/2913563711 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc009b30 con 0x7f82c415cda0 2024-03-23T03:50:54.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.986+0000 7f82c9e00640 1 --2- 172.21.15.102:0/2913563711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82c415cda0 0x7f82c40caf70 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f82bc036040 tx=0x7f82bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:54.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.986+0000 7f82c1600640 1 -- 172.21.15.102:0/2913563711 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc03e450 con 0x7f82c415cda0 2024-03-23T03:50:54.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.986+0000 7f82c1600640 1 -- 172.21.15.102:0/2913563711 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82bc03e5d0 con 0x7f82c415cda0 2024-03-23T03:50:54.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.986+0000 7f82cbe00640 1 -- 172.21.15.102:0/2913563711 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c40cbca0 con 0x7f82c415cda0 2024-03-23T03:50:54.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.986+0000 7f82c1600640 1 -- 172.21.15.102:0/2913563711 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc03e980 con 0x7f82c415cda0 2024-03-23T03:50:54.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.986+0000 7f82cbe00640 1 -- 172.21.15.102:0/2913563711 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c40c6eb0 con 0x7f82c415cda0 2024-03-23T03:50:54.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.987+0000 7f82cbe00640 1 -- 172.21.15.102:0/2913563711 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82c40c99a0 con 0x7f82c415cda0 2024-03-23T03:50:54.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.988+0000 7f82c1600640 1 -- 172.21.15.102:0/2913563711 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f82bc03eae0 con 0x7f82c415cda0 2024-03-23T03:50:54.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.989+0000 7f82c1600640 1 --2- 172.21.15.102:0/2913563711 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82a406ac00 0x7f82a406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:54.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.989+0000 7f82c1600640 1 -- 172.21.15.102:0/2913563711 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f82bc0ac330 con 0x7f82c415cda0 2024-03-23T03:50:54.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.989+0000 7f82c9400640 1 --2- 172.21.15.102:0/2913563711 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82a406ac00 0x7f82a406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:54.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.990+0000 7f82c9400640 1 --2- 172.21.15.102:0/2913563711 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82a406ac00 0x7f82a406d0c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f82b4006fd0 tx=0x7f82b4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:54.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:54.994+0000 7f82c1600640 1 -- 172.21.15.102:0/2913563711 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f82c40c99a0 con 0x7f82c415cda0 2024-03-23T03:50:55.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.229+0000 7f82cbe00640 1 -- 172.21.15.102:0/2913563711 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82c40c7040 con 0x7f82c415cda0 2024-03-23T03:50:55.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.230+0000 7f82c1600640 1 -- 172.21.15.102:0/2913563711 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f82c40c7040 con 0x7f82c415cda0 2024-03-23T03:50:55.231 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:50:55.231 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/755891320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":755891320},{"type":"v1","addr":"172.21.15.102:6836","nonce":755891320}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":5207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3792558756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3792558756},{"type":"v1","addr":"172.21.15.173:6835","nonce":3792558756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":5590,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1632622692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1632622692},{"type":"v1","addr":"172.21.15.173:6833","nonce":1632622692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:50:49.296593+0000","modified":"2024-03-23T03:50:52.307714+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/3452944670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":3452944670},{"type":"v1","addr":"172.21.15.102:6837","nonce":3452944670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T03:50:55.231 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 48 2024-03-23T03:50:55.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.232+0000 7f82cbe00640 1 -- 172.21.15.102:0/2913563711 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82a406ac00 msgr2=0x7f82a406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.232+0000 7f82cbe00640 1 --2- 172.21.15.102:0/2913563711 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82a406ac00 0x7f82a406d0c0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f82b4006fd0 tx=0x7f82b4008040 comp rx=0 tx=0).stop 2024-03-23T03:50:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.232+0000 7f82cbe00640 1 -- 172.21.15.102:0/2913563711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82c415cda0 msgr2=0x7f82c40caf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.232+0000 7f82cbe00640 1 --2- 172.21.15.102:0/2913563711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82c415cda0 0x7f82c40caf70 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f82bc036040 tx=0x7f82bc004a40 comp rx=0 tx=0).stop 2024-03-23T03:50:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.232+0000 7f82cbe00640 1 -- 172.21.15.102:0/2913563711 shutdown_connections 2024-03-23T03:50:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.232+0000 7f82cbe00640 1 --2- 172.21.15.102:0/2913563711 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82a406ac00 0x7f82a406d0c0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.232+0000 7f82cbe00640 1 --2- 172.21.15.102:0/2913563711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82c415cda0 0x7f82c40caf70 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.232+0000 7f82cbe00640 1 --2- 172.21.15.102:0/2913563711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82c40b5a30 0x7f82c415d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.232+0000 7f82cbe00640 1 --2- 172.21.15.102:0/2913563711 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82c40b4350 0x7f82c4157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.233+0000 7f82cbe00640 1 -- 172.21.15.102:0/2913563711 >> 172.21.15.102:0/2913563711 conn(0x7f82c40cf350 msgr2=0x7f82c40cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.233+0000 7f82cbe00640 1 -- 172.21.15.102:0/2913563711 shutdown_connections 2024-03-23T03:50:55.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.233+0000 7f82cbe00640 1 -- 172.21.15.102:0/2913563711 wait complete. 2024-03-23T03:50:55.654 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 48, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T03:50:49.296593+0000', 'modified': '2024-03-23T03:50:52.307714+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6828}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6828': {'gid': 6828, 'name': 'c', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.102:6837/3452944670', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6835', 'nonce': 3452944670}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 3452944670}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T03:50:55.655 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T03:50:55.655 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-23T03:50:55.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.873+0000 7f1e12000640 1 Processor -- start 2024-03-23T03:50:55.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.873+0000 7f1e12000640 1 -- start start 2024-03-23T03:50:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.875+0000 7f1e12000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1e0c160dd0 0x7f1e0c0ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.875+0000 7f1e12000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0c0cf2c0 con 0x7f1e0c13fa80 2024-03-23T03:50:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.875+0000 7f1e12000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0c06cdc0 con 0x7f1e0c0cee70 2024-03-23T03:50:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.875+0000 7f1e12000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e0c06d060 con 0x7f1e0c160dd0 2024-03-23T03:50:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.875+0000 7f1e0b400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1e0c13fa80 0x7f1e0c13ff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55544/0 (socket says 172.21.15.102:55544) 2024-03-23T03:50:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.875+0000 7f1e0b400640 1 -- 172.21.15.102:0/1388078222 learned_addr learned my addr 172.21.15.102:0/1388078222 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.876+0000 7f1e0aa00640 1 --2- 172.21.15.102:0/1388078222 >> v2:172.21.15.173:3300/0 conn(0x7f1e0c160dd0 0x7f1e0c0ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.876+0000 7f1e0a000640 1 -- 172.21.15.102:0/1388078222 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635546018 0 0) 0x7f1dfc001110 con 0x7f1e0c13fa80 2024-03-23T03:50:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.876+0000 7f1e0a000640 1 -- 172.21.15.102:0/1388078222 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1df0003520 con 0x7f1e0c13fa80 2024-03-23T03:50:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.876+0000 7f1e0a000640 1 -- 172.21.15.102:0/1388078222 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2389096514 0 0) 0x7f1df0003520 con 0x7f1e0c13fa80 2024-03-23T03:50:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.876+0000 7f1e0a000640 1 -- 172.21.15.102:0/1388078222 >> v2:172.21.15.173:3300/0 conn(0x7f1e0c160dd0 msgr2=0x7f1e0c0ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.876+0000 7f1e0a000640 1 --2- 172.21.15.102:0/1388078222 >> v2:172.21.15.173:3300/0 conn(0x7f1e0c160dd0 0x7f1e0c0ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.876+0000 7f1e0a000640 1 -- 172.21.15.102:0/1388078222 >> v1:172.21.15.173:6789/0 conn(0x7f1e0c0cee70 legacy=0x7f1e0c1422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.876+0000 7f1e0a000640 1 -- 172.21.15.102:0/1388078222 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0c06d340 con 0x7f1e0c13fa80 2024-03-23T03:50:55.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.876+0000 7f1e0a000640 1 -- 172.21.15.102:0/1388078222 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f1dfc001800 con 0x7f1e0c13fa80 2024-03-23T03:50:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.876+0000 7f1e0a000640 1 -- 172.21.15.102:0/1388078222 >> v1:172.21.15.102:6789/0 conn(0x7f1e0c13fa80 legacy=0x7f1e0c13ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.876+0000 7f1e0a000640 1 --2- 172.21.15.102:0/1388078222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df00039c0 0x7f1df0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.876+0000 7f1e0a000640 1 --2- 172.21.15.102:0/1388078222 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1df00047c0 0x7f1df0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.876+0000 7f1e0a000640 1 --2- 172.21.15.102:0/1388078222 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1df00051d0 0x7f1df000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.876+0000 7f1e0a000640 1 -- 172.21.15.102:0/1388078222 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c06d060 con 0x7f1df00039c0 2024-03-23T03:50:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.876+0000 7f1e0a000640 1 -- 172.21.15.102:0/1388078222 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df000c100 con 0x7f1df00051d0 2024-03-23T03:50:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.876+0000 7f1e0a000640 1 -- 172.21.15.102:0/1388078222 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df000c2c0 con 0x7f1df00047c0 2024-03-23T03:50:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.877+0000 7f1e0b400640 1 --2- 172.21.15.102:0/1388078222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df00039c0 0x7f1df0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.877+0000 7f1e0aa00640 1 --2- 172.21.15.102:0/1388078222 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1df00047c0 0x7f1df0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.877+0000 7f1e0be00640 1 --2- 172.21.15.102:0/1388078222 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1df00051d0 0x7f1df000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.877+0000 7f1e0b400640 1 -- 172.21.15.102:0/1388078222 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1df00047c0 msgr2=0x7f1df0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.877+0000 7f1e0b400640 1 --2- 172.21.15.102:0/1388078222 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1df00047c0 0x7f1df0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.877+0000 7f1e0b400640 1 -- 172.21.15.102:0/1388078222 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1df00051d0 msgr2=0x7f1df000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.877+0000 7f1e0b400640 1 --2- 172.21.15.102:0/1388078222 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1df00051d0 0x7f1df000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.877+0000 7f1e0b400640 1 -- 172.21.15.102:0/1388078222 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df0003520 con 0x7f1df00039c0 2024-03-23T03:50:55.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.877+0000 7f1e0be00640 1 -- 172.21.15.102:0/1388078222 reap_dead start 2024-03-23T03:50:55.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.878+0000 7f1e0b400640 1 --2- 172.21.15.102:0/1388078222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df00039c0 0x7f1df0003dc0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc00c0e0 tx=0x7f1df00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc9ba04b6837660 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.879+0000 7f1e0a000640 1 -- 172.21.15.102:0/1388078222 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc005ad0 con 0x7f1df00039c0 2024-03-23T03:50:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.879+0000 7f1e0a000640 1 -- 172.21.15.102:0/1388078222 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dfc005c30 con 0x7f1df00039c0 2024-03-23T03:50:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.879+0000 7f1e0a000640 1 -- 172.21.15.102:0/1388078222 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc02c7d0 con 0x7f1df00039c0 2024-03-23T03:50:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.879+0000 7f1e12000640 1 -- 172.21.15.102:0/1388078222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df00039c0 msgr2=0x7f1df0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.879+0000 7f1e12000640 1 --2- 172.21.15.102:0/1388078222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df00039c0 0x7f1df0003dc0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc00c0e0 tx=0x7f1df00057f0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.879+0000 7f1e12000640 1 -- 172.21.15.102:0/1388078222 shutdown_connections 2024-03-23T03:50:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.879+0000 7f1e12000640 1 --2- 172.21.15.102:0/1388078222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df00039c0 0x7f1df0003dc0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.879+0000 7f1e12000640 1 -- 172.21.15.102:0/1388078222 >> 172.21.15.102:0/1388078222 conn(0x7f1e0c06a0b0 msgr2=0x7f1e0c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.879+0000 7f1e12000640 1 -- 172.21.15.102:0/1388078222 shutdown_connections 2024-03-23T03:50:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.879+0000 7f1e12000640 1 -- 172.21.15.102:0/1388078222 wait complete. 2024-03-23T03:50:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.881+0000 7f1e12000640 1 Processor -- start 2024-03-23T03:50:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.881+0000 7f1e12000640 1 -- start start 2024-03-23T03:50:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.882+0000 7f1e12000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df00039c0 0x7f1e0c157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.882+0000 7f1e12000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e0c158ba0 0x7f1e0c151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.882+0000 7f1e12000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1e0c155330 0x7f1e0c15f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.882+0000 7f1e12000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c1578b0 con 0x7f1df00039c0 2024-03-23T03:50:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.882+0000 7f1e12000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c152360 con 0x7f1e0c158ba0 2024-03-23T03:50:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.882+0000 7f1e12000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0c157b00 con 0x7f1e0c155330 2024-03-23T03:50:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.882+0000 7f1e0b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df00039c0 0x7f1e0c157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.882+0000 7f1e0b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df00039c0 0x7f1e0c157370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58010/0 (socket says 172.21.15.102:58010) 2024-03-23T03:50:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.882+0000 7f1e0b400640 1 -- 172.21.15.102:0/1835154313 learned_addr learned my addr 172.21.15.102:0/1835154313 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.883+0000 7f1e0aa00640 1 --2- 172.21.15.102:0/1835154313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e0c158ba0 0x7f1e0c151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:55.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.883+0000 7f1e0b400640 1 -- 172.21.15.102:0/1835154313 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1e0c155330 msgr2=0x7f1e0c15f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:55.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.883+0000 7f1e0b400640 1 --2- 172.21.15.102:0/1835154313 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1e0c155330 0x7f1e0c15f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.883+0000 7f1e0b400640 1 -- 172.21.15.102:0/1835154313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e0c158ba0 msgr2=0x7f1e0c151e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:55.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.883+0000 7f1e0b400640 1 --2- 172.21.15.102:0/1835154313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e0c158ba0 0x7f1e0c151e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.883+0000 7f1e0b400640 1 -- 172.21.15.102:0/1835154313 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0c157de0 con 0x7f1df00039c0 2024-03-23T03:50:55.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.883+0000 7f1e0b400640 1 --2- 172.21.15.102:0/1835154313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df00039c0 0x7f1e0c157370 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc00c0e0 tx=0x7f1dfc02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:55.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.883+0000 7f1e03400640 1 -- 172.21.15.102:0/1835154313 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc03c040 con 0x7f1df00039c0 2024-03-23T03:50:55.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.883+0000 7f1e03400640 1 -- 172.21.15.102:0/1835154313 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dfc02cbe0 con 0x7f1df00039c0 2024-03-23T03:50:55.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.883+0000 7f1e03400640 1 -- 172.21.15.102:0/1835154313 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc007070 con 0x7f1df00039c0 2024-03-23T03:50:55.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.883+0000 7f1e12000640 1 -- 172.21.15.102:0/1835154313 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e0c1580c0 con 0x7f1df00039c0 2024-03-23T03:50:55.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.884+0000 7f1e12000640 1 -- 172.21.15.102:0/1835154313 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0c158680 con 0x7f1df00039c0 2024-03-23T03:50:55.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.885+0000 7f1e03400640 1 -- 172.21.15.102:0/1835154313 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1dfc032070 con 0x7f1df00039c0 2024-03-23T03:50:55.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.886+0000 7f1e03400640 1 --2- 172.21.15.102:0/1835154313 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1dec06ac30 0x7f1dec06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.887+0000 7f1e03400640 1 -- 172.21.15.102:0/1835154313 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1dfc0aa4b0 con 0x7f1df00039c0 2024-03-23T03:50:55.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.887+0000 7f1e0aa00640 1 --2- 172.21.15.102:0/1835154313 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1dec06ac30 0x7f1dec06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:55.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.887+0000 7f1e14c72740 1 Processor -- start 2024-03-23T03:50:55.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.887+0000 7f1e0aa00640 1 --2- 172.21.15.102:0/1835154313 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1dec06ac30 0x7f1dec06d0f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c1547f0 tx=0x7f1df8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:55.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.888+0000 7f1e14c72740 1 -- start start 2024-03-23T03:50:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.888+0000 7f1e14c72740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55926b762d00 0x55926b735ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.888+0000 7f1e14c72740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55926b7630e0 0x55926b765a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.888+0000 7f1e14c72740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b7361e0 0x55926b765fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.888+0000 7f1e14c72740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55926b7112c0 con 0x55926b7361e0 2024-03-23T03:50:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.888+0000 7f1e14c72740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55926b6a68f0 con 0x55926b762d00 2024-03-23T03:50:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.888+0000 7f1e14c72740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x55926b6a4080 con 0x55926b7630e0 2024-03-23T03:50:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.889+0000 7f1e0be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b7361e0 0x55926b765fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.889+0000 7f1e0be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b7361e0 0x55926b765fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58018/0 (socket says 172.21.15.102:58018) 2024-03-23T03:50:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.889+0000 7f1e0be00640 1 -- 172.21.15.102:0/128323369 learned_addr learned my addr 172.21.15.102:0/128323369 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.889+0000 7f1e0be00640 1 -- 172.21.15.102:0/128323369 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55926b7630e0 msgr2=0x55926b765a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.889+0000 7f1e0be00640 1 --2- 172.21.15.102:0/128323369 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55926b7630e0 0x55926b765a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.889+0000 7f1e0b400640 1 --2- 172.21.15.102:0/128323369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55926b762d00 0x55926b735ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:55.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.889+0000 7f1e0be00640 1 -- 172.21.15.102:0/128323369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55926b762d00 msgr2=0x55926b735ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:55.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.889+0000 7f1e0be00640 1 --2- 172.21.15.102:0/128323369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55926b762d00 0x55926b735ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.889+0000 7f1e0be00640 1 -- 172.21.15.102:0/128323369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55926b66aa10 con 0x55926b7361e0 2024-03-23T03:50:55.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.889+0000 7f1e0be00640 1 --2- 172.21.15.102:0/128323369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b7361e0 0x55926b765fd0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c0cf540 tx=0x7f1e040054c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98ac18fd328b59ed server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.890+0000 7f1df7e00640 1 -- 172.21.15.102:0/128323369 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e04003df0 con 0x55926b7361e0 2024-03-23T03:50:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.890+0000 7f1df7e00640 1 -- 172.21.15.102:0/128323369 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e04013420 con 0x55926b7361e0 2024-03-23T03:50:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.890+0000 7f1df7e00640 1 -- 172.21.15.102:0/128323369 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e040137d0 con 0x55926b7361e0 2024-03-23T03:50:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.890+0000 7f1e14c72740 1 -- 172.21.15.102:0/128323369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b7361e0 msgr2=0x55926b765fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.890+0000 7f1e14c72740 1 --2- 172.21.15.102:0/128323369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b7361e0 0x55926b765fd0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c0cf540 tx=0x7f1e040054c0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.890+0000 7f1e14c72740 1 -- 172.21.15.102:0/128323369 shutdown_connections 2024-03-23T03:50:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.890+0000 7f1e14c72740 1 --2- 172.21.15.102:0/128323369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b7361e0 0x55926b765fd0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.890+0000 7f1e14c72740 1 --2- 172.21.15.102:0/128323369 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55926b7630e0 0x55926b765a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.890+0000 7f1e14c72740 1 --2- 172.21.15.102:0/128323369 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55926b762d00 0x55926b735ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.890+0000 7f1e14c72740 1 -- 172.21.15.102:0/128323369 >> 172.21.15.102:0/128323369 conn(0x55926b737f10 msgr2=0x55926b780a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:55.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.890+0000 7f1e14c72740 1 -- 172.21.15.102:0/128323369 shutdown_connections 2024-03-23T03:50:55.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.890+0000 7f1e14c72740 1 -- 172.21.15.102:0/128323369 wait complete. 2024-03-23T03:50:55.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.891+0000 7f1e14c72740 1 -- start start 2024-03-23T03:50:55.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.891+0000 7f1e14c72740 1 Processor -- start 2024-03-23T03:50:55.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.892+0000 7f1e14c72740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b762d00 0x55926b761360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.892+0000 7f1e14c72740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55926b7630e0 0x55926b7618a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.892+0000 7f1e14c72740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55926b778ce0 0x55926b790310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.892+0000 7f1e14c72740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55926b69f970 con 0x55926b762d00 2024-03-23T03:50:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.892+0000 7f1e14c72740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55926b6410a0 con 0x55926b7630e0 2024-03-23T03:50:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.892+0000 7f1e14c72740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x55926b640d40 con 0x55926b778ce0 2024-03-23T03:50:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.892+0000 7f1e14c72740 10 client.0 _ll_register_callbacks cb 0x55926b698fe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T03:50:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.893+0000 7f1e0b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b762d00 0x55926b761360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.893+0000 7f1e0b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b762d00 0x55926b761360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58026/0 (socket says 172.21.15.102:58026) 2024-03-23T03:50:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.893+0000 7f1e0b400640 1 -- 172.21.15.102:0/1718782623 learned_addr learned my addr 172.21.15.102:0/1718782623 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.893+0000 7f1e0b400640 1 -- 172.21.15.102:0/1718782623 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55926b778ce0 msgr2=0x55926b790310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.893+0000 7f1e0b400640 1 --2- 172.21.15.102:0/1718782623 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55926b778ce0 0x55926b790310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.893+0000 7f1e0b400640 1 -- 172.21.15.102:0/1718782623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55926b7630e0 msgr2=0x55926b7618a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.893+0000 7f1e0b400640 1 --2- 172.21.15.102:0/1718782623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55926b7630e0 0x55926b7618a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.893+0000 7f1e0b400640 1 -- 172.21.15.102:0/1718782623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0400f040 con 0x55926b762d00 2024-03-23T03:50:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.893+0000 7f1e0b400640 1 --2- 172.21.15.102:0/1718782623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b762d00 0x55926b761360 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0758d0 tx=0x7f1dfc078df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c270e767feeab8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.893+0000 7f1df4c00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-23T03:50:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.893+0000 7f1df4c00640 1 -- 172.21.15.102:0/1718782623 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc0aa0b0 con 0x55926b762d00 2024-03-23T03:50:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.893+0000 7f1df4c00640 1 -- 172.21.15.102:0/1718782623 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1dfc0aa210 con 0x55926b762d00 2024-03-23T03:50:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.893+0000 7f1e14c72740 1 -- 172.21.15.102:0/1718782623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55926b6c9b10 con 0x55926b762d00 2024-03-23T03:50:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.894+0000 7f1df4c00640 1 -- 172.21.15.102:0/1718782623 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1dfc081280 con 0x55926b762d00 2024-03-23T03:50:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.894+0000 7f1e14c72740 10 client.6972 fetch_fsmap learned FSMap version 48 2024-03-23T03:50:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.894+0000 7f1e14c72740 1 -- 172.21.15.102:0/1718782623 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x55926b69dff0 con 0x55926b762d00 2024-03-23T03:50:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.894+0000 7f1df4c00640 1 -- 172.21.15.102:0/1718782623 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55926b6c9b10 con 0x55926b762d00 2024-03-23T03:50:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.894+0000 7f1df4c00640 1 -- 172.21.15.102:0/1718782623 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 48) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1dfc0a8250 con 0x55926b762d00 2024-03-23T03:50:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.894+0000 7f1e14c72740 10 client.6972 fetch_fsmap finished waiting for FSMap version 48 2024-03-23T03:50:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.894+0000 7f1e14c72740 10 client.6972 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-23T03:50:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.894+0000 7f1e14c72740 20 client.6972 populate_metadata read hostname 'smithi102' 2024-03-23T03:50:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.894+0000 7f1e14c72740 1 --2- 172.21.15.102:0/1718782623 >> [v2:172.21.15.102:6835/3452944670,v1:172.21.15.102:6837/3452944670] conn(0x55926b77a4a0 0x55926b767ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.894+0000 7f1e14c72740 4 client.6972 mds_command: new command op to 6828 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T03:50:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.894+0000 7f1e14c72740 1 -- 172.21.15.102:0/1718782623 --> [v2:172.21.15.102:6835/3452944670,v1:172.21.15.102:6837/3452944670] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55926b3933c0 con 0x55926b77a4a0 2024-03-23T03:50:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.894+0000 7f1e0aa00640 1 --2- 172.21.15.102:0/1718782623 >> [v2:172.21.15.102:6835/3452944670,v1:172.21.15.102:6837/3452944670] conn(0x55926b77a4a0 0x55926b767ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.895+0000 7f1e0aa00640 1 --2- 172.21.15.102:0/1718782623 >> [v2:172.21.15.102:6835/3452944670,v1:172.21.15.102:6837/3452944670] conn(0x55926b77a4a0 0x55926b767ad0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbbbfecc37bdbe7f server_cookie=91bd21bee8f2df3f in_seq=0 out_seq=0 2024-03-23T03:50:55.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.895+0000 7f1df4c00640 10 client.6972 ms_handle_connect on v2:172.21.15.102:6835/3452944670 2024-03-23T03:50:55.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.896+0000 7f1df4c00640 1 -- 172.21.15.102:0/1718782623 <== mds.0 v2:172.21.15.102:6835/3452944670 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x55926b3933c0 con 0x55926b77a4a0 2024-03-23T03:50:55.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.896+0000 7f1df4c00640 10 client.6972 handle_command_reply: tid=0 2024-03-23T03:50:55.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.896+0000 7f1e14c72740 1 client.6972 shutdown 2024-03-23T03:50:55.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.896+0000 7f1e14c72740 1 -- 172.21.15.102:0/1718782623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b762d00 msgr2=0x55926b761360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:55.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.896+0000 7f1e14c72740 1 --2- 172.21.15.102:0/1718782623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b762d00 0x55926b761360 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0758d0 tx=0x7f1dfc078df0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.896+0000 7f1e14c72740 1 -- 172.21.15.102:0/1718782623 shutdown_connections 2024-03-23T03:50:55.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.896+0000 7f1e14c72740 1 --2- 172.21.15.102:0/1718782623 >> [v2:172.21.15.102:6835/3452944670,v1:172.21.15.102:6837/3452944670] conn(0x55926b77a4a0 0x55926b767ad0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.896+0000 7f1e14c72740 1 --2- 172.21.15.102:0/1718782623 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55926b778ce0 0x55926b790310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.896+0000 7f1e14c72740 1 --2- 172.21.15.102:0/1718782623 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55926b7630e0 0x55926b7618a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.896+0000 7f1e14c72740 1 --2- 172.21.15.102:0/1718782623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b762d00 0x55926b761360 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.896+0000 7f1df4c00640 0 client.6972 ms_handle_reset on v2:172.21.15.102:6835/3452944670 2024-03-23T03:50:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.896+0000 7f1e14c72740 1 -- 172.21.15.102:0/1718782623 >> 172.21.15.102:0/1718782623 conn(0x55926b7361e0 msgr2=0x55926b765a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.896+0000 7f1e14c72740 1 -- 172.21.15.102:0/1718782623 shutdown_connections 2024-03-23T03:50:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.896+0000 7f1e14c72740 1 -- 172.21.15.102:0/1718782623 wait complete. 2024-03-23T03:50:55.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.896+0000 7f1e14c72740 20 client.6972 trim_cache size 0 max 16384 2024-03-23T03:50:55.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.912+0000 7f1e14c72740 1 Processor -- start 2024-03-23T03:50:55.913 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.913+0000 7f1e14c72740 1 -- start start 2024-03-23T03:50:55.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.913+0000 7f1e14c72740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b762d00 0x55926b7313c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.913+0000 7f1e14c72740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55926b7630e0 0x55926b731900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.913+0000 7f1e14c72740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55926b778ce0 0x55926b768a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.914 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.913+0000 7f1e14c72740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55926b406f20 con 0x55926b762d00 2024-03-23T03:50:55.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.913+0000 7f1e14c72740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55926b7813f0 con 0x55926b778ce0 2024-03-23T03:50:55.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.913+0000 7f1e14c72740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x55926b765fc0 con 0x55926b7630e0 2024-03-23T03:50:55.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.914+0000 7f1e0b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b762d00 0x55926b7313c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:55.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.914+0000 7f1e0b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b762d00 0x55926b7313c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58042/0 (socket says 172.21.15.102:58042) 2024-03-23T03:50:55.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.914+0000 7f1e0b400640 1 -- 172.21.15.102:0/3680849146 learned_addr learned my addr 172.21.15.102:0/3680849146 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:55.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.914+0000 7f1e0b400640 1 -- 172.21.15.102:0/3680849146 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55926b7630e0 msgr2=0x55926b731900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:55.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.914+0000 7f1e0b400640 1 --2- 172.21.15.102:0/3680849146 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55926b7630e0 0x55926b731900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.914+0000 7f1e0b400640 1 -- 172.21.15.102:0/3680849146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55926b778ce0 msgr2=0x55926b768a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:55.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.914+0000 7f1e0b400640 1 --2- 172.21.15.102:0/3680849146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55926b778ce0 0x55926b768a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.916 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.914+0000 7f1e0b400640 1 -- 172.21.15.102:0/3680849146 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55926b6d37c0 con 0x55926b762d00 2024-03-23T03:50:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.914+0000 7f1e0b400640 1 --2- 172.21.15.102:0/3680849146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b762d00 0x55926b7313c0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0811e0 tx=0x7f1dfc07c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f417e621c9afb858 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.915+0000 7f1e00c00640 1 -- 172.21.15.102:0/3680849146 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc075ae0 con 0x55926b762d00 2024-03-23T03:50:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.915+0000 7f1e00c00640 1 -- 172.21.15.102:0/3680849146 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dfc075c40 con 0x55926b762d00 2024-03-23T03:50:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.915+0000 7f1e00c00640 1 -- 172.21.15.102:0/3680849146 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc0aa2f0 con 0x55926b762d00 2024-03-23T03:50:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.915+0000 7f1e14c72740 1 -- 172.21.15.102:0/3680849146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b762d00 msgr2=0x55926b7313c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.915+0000 7f1e14c72740 1 --2- 172.21.15.102:0/3680849146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b762d00 0x55926b7313c0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc0811e0 tx=0x7f1dfc07c000 comp rx=0 tx=0).stop 2024-03-23T03:50:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.915+0000 7f1e14c72740 1 -- 172.21.15.102:0/3680849146 shutdown_connections 2024-03-23T03:50:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.915+0000 7f1e14c72740 1 --2- 172.21.15.102:0/3680849146 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55926b778ce0 0x55926b768a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.915+0000 7f1e14c72740 1 --2- 172.21.15.102:0/3680849146 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55926b7630e0 0x55926b731900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.915+0000 7f1e14c72740 1 --2- 172.21.15.102:0/3680849146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b762d00 0x55926b7313c0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.915+0000 7f1e14c72740 1 -- 172.21.15.102:0/3680849146 >> 172.21.15.102:0/3680849146 conn(0x55926b7361e0 msgr2=0x55926b776420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.915+0000 7f1e14c72740 1 -- 172.21.15.102:0/3680849146 shutdown_connections 2024-03-23T03:50:55.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.915+0000 7f1e14c72740 1 -- 172.21.15.102:0/3680849146 wait complete. 2024-03-23T03:50:55.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.916+0000 7f1e14c72740 1 -- start start 2024-03-23T03:50:55.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.916+0000 7f1e14c72740 1 Processor -- start 2024-03-23T03:50:55.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.917+0000 7f1e14c72740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55926b7630e0 0x55926b776420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.920 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.917+0000 7f1e14c72740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b778ce0 0x55926b776960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.917+0000 7f1e14c72740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55926b7dc6a0 0x55926b78ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.917+0000 7f1e14c72740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55926b739d50 con 0x55926b778ce0 2024-03-23T03:50:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.917+0000 7f1e14c72740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x55926b7e80d0 con 0x55926b7dc6a0 2024-03-23T03:50:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.917+0000 7f1e14c72740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x55926b760400 con 0x55926b7630e0 2024-03-23T03:50:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.917+0000 7f1e14c72740 10 client.0 _ll_register_callbacks cb 0x7f1dfc078f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T03:50:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.918+0000 7f1e0aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b778ce0 0x55926b776960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.918+0000 7f1e0aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b778ce0 0x55926b776960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58048/0 (socket says 172.21.15.102:58048) 2024-03-23T03:50:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.918+0000 7f1e0aa00640 1 -- 172.21.15.102:0/3394233347 learned_addr learned my addr 172.21.15.102:0/3394233347 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.918+0000 7f1e0b400640 1 --2- 172.21.15.102:0/3394233347 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55926b7630e0 0x55926b776420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.918+0000 7f1e0aa00640 1 -- 172.21.15.102:0/3394233347 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55926b7630e0 msgr2=0x55926b776420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.918+0000 7f1e0aa00640 1 --2- 172.21.15.102:0/3394233347 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55926b7630e0 0x55926b776420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.918+0000 7f1e0aa00640 1 -- 172.21.15.102:0/3394233347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55926b7dc6a0 msgr2=0x55926b78ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.918+0000 7f1e0aa00640 1 --2- 172.21.15.102:0/3394233347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55926b7dc6a0 0x55926b78ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.918+0000 7f1e0aa00640 1 -- 172.21.15.102:0/3394233347 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0400f040 con 0x55926b778ce0 2024-03-23T03:50:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.918+0000 7f1e0aa00640 1 --2- 172.21.15.102:0/3394233347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b778ce0 0x55926b776960 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c1f5940 tx=0x7f1df8010fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e35ec8e33ca55e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.918+0000 7f1df4c00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-23T03:50:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.918+0000 7f1df4c00640 1 -- 172.21.15.102:0/3394233347 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df8024270 con 0x55926b778ce0 2024-03-23T03:50:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.919+0000 7f1df4c00640 1 -- 172.21.15.102:0/3394233347 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1df80243d0 con 0x55926b778ce0 2024-03-23T03:50:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.919+0000 7f1e14c72740 1 -- 172.21.15.102:0/3394233347 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1e04013420 con 0x55926b778ce0 2024-03-23T03:50:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.919+0000 7f1df4c00640 1 -- 172.21.15.102:0/3394233347 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1df802d420 con 0x55926b778ce0 2024-03-23T03:50:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.919+0000 7f1e14c72740 10 client.6978 fetch_fsmap learned FSMap version 48 2024-03-23T03:50:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.919+0000 7f1e14c72740 1 -- 172.21.15.102:0/3394233347 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x55926b6402b0 con 0x55926b778ce0 2024-03-23T03:50:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.919+0000 7f1df4c00640 1 -- 172.21.15.102:0/3394233347 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e04013420 con 0x55926b778ce0 2024-03-23T03:50:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.919+0000 7f1df4c00640 1 -- 172.21.15.102:0/3394233347 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 48) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1df802d580 con 0x55926b778ce0 2024-03-23T03:50:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.919+0000 7f1e14c72740 10 client.6978 fetch_fsmap finished waiting for FSMap version 48 2024-03-23T03:50:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.919+0000 7f1e14c72740 10 client.6978 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-23T03:50:55.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.919+0000 7f1e14c72740 20 client.6978 populate_metadata read hostname 'smithi102' 2024-03-23T03:50:55.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.919+0000 7f1e14c72740 1 --2- 172.21.15.102:0/3394233347 >> [v2:172.21.15.102:6835/3452944670,v1:172.21.15.102:6837/3452944670] conn(0x55926b78f950 0x55926b7e6d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:55.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.919+0000 7f1e14c72740 4 client.6978 mds_command: new command op to 6828 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-23T03:50:55.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.919+0000 7f1e14c72740 1 -- 172.21.15.102:0/3394233347 --> [v2:172.21.15.102:6835/3452944670,v1:172.21.15.102:6837/3452944670] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55926b3a46c0 con 0x55926b78f950 2024-03-23T03:50:55.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.920+0000 7f1e0b400640 1 --2- 172.21.15.102:0/3394233347 >> [v2:172.21.15.102:6835/3452944670,v1:172.21.15.102:6837/3452944670] conn(0x55926b78f950 0x55926b7e6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:55.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.920+0000 7f1e0b400640 1 --2- 172.21.15.102:0/3394233347 >> [v2:172.21.15.102:6835/3452944670,v1:172.21.15.102:6837/3452944670] conn(0x55926b78f950 0x55926b7e6d90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c59151dcf8e91c07 server_cookie=1c28c664014ee371 in_seq=0 out_seq=0 2024-03-23T03:50:55.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.920+0000 7f1df4c00640 10 client.6978 ms_handle_connect on v2:172.21.15.102:6835/3452944670 2024-03-23T03:50:55.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.920+0000 7f1df4c00640 1 -- 172.21.15.102:0/3394233347 <== mds.0 v2:172.21.15.102:6835/3452944670 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55926b3a46c0 con 0x55926b78f950 2024-03-23T03:50:55.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.920+0000 7f1df4c00640 10 client.6978 handle_command_reply: tid=0 2024-03-23T03:50:55.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.920+0000 7f1e14c72740 1 client.6978 shutdown 2024-03-23T03:50:55.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.920+0000 7f1e14c72740 1 -- 172.21.15.102:0/3394233347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b778ce0 msgr2=0x55926b776960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:55.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.920+0000 7f1e14c72740 1 --2- 172.21.15.102:0/3394233347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b778ce0 0x55926b776960 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c1f5940 tx=0x7f1df8010fd0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.921+0000 7f1e14c72740 1 -- 172.21.15.102:0/3394233347 shutdown_connections 2024-03-23T03:50:55.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.921+0000 7f1e14c72740 1 --2- 172.21.15.102:0/3394233347 >> [v2:172.21.15.102:6835/3452944670,v1:172.21.15.102:6837/3452944670] conn(0x55926b78f950 0x55926b7e6d90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.921+0000 7f1e14c72740 1 --2- 172.21.15.102:0/3394233347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x55926b7dc6a0 0x55926b78ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.921+0000 7f1e14c72740 1 --2- 172.21.15.102:0/3394233347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55926b778ce0 0x55926b776960 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.921+0000 7f1e14c72740 1 --2- 172.21.15.102:0/3394233347 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x55926b7630e0 0x55926b776420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.921+0000 7f1df4c00640 0 client.6978 ms_handle_reset on v2:172.21.15.102:6835/3452944670 2024-03-23T03:50:55.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.921+0000 7f1e14c72740 1 -- 172.21.15.102:0/3394233347 >> 172.21.15.102:0/3394233347 conn(0x55926b7361e0 msgr2=0x55926b765590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:55.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.921+0000 7f1e14c72740 1 -- 172.21.15.102:0/3394233347 shutdown_connections 2024-03-23T03:50:55.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.921+0000 7f1e14c72740 1 -- 172.21.15.102:0/3394233347 wait complete. 2024-03-23T03:50:55.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.921+0000 7f1e14c72740 20 client.6978 trim_cache size 0 max 16384 2024-03-23T03:50:55.925 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:50:55.925 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "776b2820-05f2-4e50-8d40-18dd76c5f6a0", 2024-03-23T03:50:55.925 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 0, 2024-03-23T03:50:55.925 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 6828, 2024-03-23T03:50:55.926 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-03-23T03:50:55.926 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-03-23T03:50:55.926 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-03-23T03:50:55.926 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 6.6199615109999996, 2024-03-23T03:50:55.926 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 48, 2024-03-23T03:50:55.926 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 93, 2024-03-23T03:50:55.926 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 93, 2024-03-23T03:50:55.926 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 11.539643549999999 2024-03-23T03:50:55.926 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:50:55.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.921+0000 7f1e12000640 1 -- 172.21.15.102:0/1835154313 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1dec06ac30 msgr2=0x7f1dec06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:55.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.921+0000 7f1e12000640 1 --2- 172.21.15.102:0/1835154313 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1dec06ac30 0x7f1dec06d0f0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c1547f0 tx=0x7f1df8008040 comp rx=0 tx=0).stop 2024-03-23T03:50:55.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.921+0000 7f1e12000640 1 -- 172.21.15.102:0/1835154313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df00039c0 msgr2=0x7f1e0c157370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:55.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.921+0000 7f1e12000640 1 --2- 172.21.15.102:0/1835154313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df00039c0 0x7f1e0c157370 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc00c0e0 tx=0x7f1dfc02e070 comp rx=0 tx=0).stop 2024-03-23T03:50:55.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.922+0000 7f1e12000640 1 -- 172.21.15.102:0/1835154313 shutdown_connections 2024-03-23T03:50:55.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.922+0000 7f1e12000640 1 --2- 172.21.15.102:0/1835154313 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1dec06ac30 0x7f1dec06d0f0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.922+0000 7f1e12000640 1 --2- 172.21.15.102:0/1835154313 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1e0c155330 0x7f1e0c15f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.922+0000 7f1e12000640 1 --2- 172.21.15.102:0/1835154313 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1e0c158ba0 0x7f1e0c151e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.922+0000 7f1e12000640 1 --2- 172.21.15.102:0/1835154313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1df00039c0 0x7f1e0c157370 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:55.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.922+0000 7f1e12000640 1 -- 172.21.15.102:0/1835154313 >> 172.21.15.102:0/1835154313 conn(0x7f1e0c06a0b0 msgr2=0x7f1e0c0b2d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:55.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.922+0000 7f1e12000640 1 -- 172.21.15.102:0/1835154313 shutdown_connections 2024-03-23T03:50:55.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:55.922+0000 7f1e12000640 1 -- 172.21.15.102:0/1835154313 wait complete. 2024-03-23T03:50:55.940 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:50:56.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.147+0000 7f6d48800640 1 Processor -- start 2024-03-23T03:50:56.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.147+0000 7f6d48800640 1 -- start start 2024-03-23T03:50:56.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.149+0000 7f6d48800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6d4015b560 0x7f6d4015b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:56.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.149+0000 7f6d48800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6d400b39b0 0x7f6d400b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:56.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.149+0000 7f6d48800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d4006c6f0 con 0x7f6d4015e500 2024-03-23T03:50:56.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.149+0000 7f6d48800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d400b4370 con 0x7f6d400b39b0 2024-03-23T03:50:56.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.149+0000 7f6d48800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4006c9d0 con 0x7f6d4015b560 2024-03-23T03:50:56.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.149+0000 7f6d45400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6d400b39b0 0x7f6d400b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:56.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.149+0000 7f6d45400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6d400b39b0 0x7f6d400b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58056/0 (socket says 172.21.15.102:58056) 2024-03-23T03:50:56.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.149+0000 7f6d45400640 1 -- 172.21.15.102:0/3419163329 learned_addr learned my addr 172.21.15.102:0/3419163329 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:56.150 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.149+0000 7f6d45e00640 1 --2- 172.21.15.102:0/3419163329 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6d4015b560 0x7f6d4015b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:56.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.150+0000 7f6d45400640 1 -- 172.21.15.102:0/3419163329 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6d4015b560 msgr2=0x7f6d4015b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:56.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.150+0000 7f6d45400640 1 --2- 172.21.15.102:0/3419163329 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6d4015b560 0x7f6d4015b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:56.151 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.150+0000 7f6d45400640 1 -- 172.21.15.102:0/3419163329 >> v1:172.21.15.102:6789/0 conn(0x7f6d4015e500 legacy=0x7f6d4015e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:56.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.150+0000 7f6d45400640 1 -- 172.21.15.102:0/3419163329 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4006ccb0 con 0x7f6d400b39b0 2024-03-23T03:50:56.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.151+0000 7f6d45400640 1 --2- 172.21.15.102:0/3419163329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d400b39b0 0x7f6d400b3e30 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f6d30009870 tx=0x7f6d3002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=251ae4277c86d347 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:56.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.151+0000 7f6d44a00640 1 -- 172.21.15.102:0/3419163329 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30030840 con 0x7f6d400b39b0 2024-03-23T03:50:56.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.151+0000 7f6d44a00640 1 -- 172.21.15.102:0/3419163329 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d300309a0 con 0x7f6d400b39b0 2024-03-23T03:50:56.152 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.151+0000 7f6d44a00640 1 -- 172.21.15.102:0/3419163329 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30030d30 con 0x7f6d400b39b0 2024-03-23T03:50:56.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.152+0000 7f6d48800640 1 -- 172.21.15.102:0/3419163329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d400b39b0 msgr2=0x7f6d400b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:56.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.152+0000 7f6d48800640 1 --2- 172.21.15.102:0/3419163329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d400b39b0 0x7f6d400b3e30 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f6d30009870 tx=0x7f6d3002b980 comp rx=0 tx=0).stop 2024-03-23T03:50:56.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.152+0000 7f6d48800640 1 -- 172.21.15.102:0/3419163329 shutdown_connections 2024-03-23T03:50:56.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.152+0000 7f6d48800640 1 --2- 172.21.15.102:0/3419163329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d400b39b0 0x7f6d400b3e30 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:56.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.152+0000 7f6d48800640 1 --2- 172.21.15.102:0/3419163329 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6d4015b560 0x7f6d4015b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:56.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.152+0000 7f6d48800640 1 -- 172.21.15.102:0/3419163329 >> 172.21.15.102:0/3419163329 conn(0x7f6d4006a3d0 msgr2=0x7f6d400b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:56.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.152+0000 7f6d48800640 1 -- 172.21.15.102:0/3419163329 shutdown_connections 2024-03-23T03:50:56.153 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.152+0000 7f6d48800640 1 -- 172.21.15.102:0/3419163329 wait complete. 2024-03-23T03:50:56.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.153+0000 7f6d48800640 1 Processor -- start 2024-03-23T03:50:56.154 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.154+0000 7f6d48800640 1 -- start start 2024-03-23T03:50:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.154+0000 7f6d48800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6d400b39b0 0x7f6d4013c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.154+0000 7f6d48800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d4015b560 0x7f6d40159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.154+0000 7f6d48800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d4015e500 0x7f6d40159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.154+0000 7f6d48800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d400b2590 con 0x7f6d4015b560 2024-03-23T03:50:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.154+0000 7f6d48800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d400b2870 con 0x7f6d4015e500 2024-03-23T03:50:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.154+0000 7f6d48800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d400b2b50 con 0x7f6d400b39b0 2024-03-23T03:50:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.155+0000 7f6d45400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d4015b560 0x7f6d40159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.155+0000 7f6d45400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d4015b560 0x7f6d40159930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58072/0 (socket says 172.21.15.102:58072) 2024-03-23T03:50:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.155+0000 7f6d45400640 1 -- 172.21.15.102:0/1673369795 learned_addr learned my addr 172.21.15.102:0/1673369795 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.155+0000 7f6d45e00640 1 --2- 172.21.15.102:0/1673369795 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6d400b39b0 0x7f6d4013c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.155+0000 7f6d45400640 1 -- 172.21.15.102:0/1673369795 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6d400b39b0 msgr2=0x7f6d4013c160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:56.156 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.155+0000 7f6d45400640 1 --2- 172.21.15.102:0/1673369795 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6d400b39b0 0x7f6d4013c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:56.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.155+0000 7f6d45400640 1 -- 172.21.15.102:0/1673369795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d4015e500 msgr2=0x7f6d40159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:56.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.155+0000 7f6d45400640 1 --2- 172.21.15.102:0/1673369795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d4015e500 0x7f6d40159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:56.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.155+0000 7f6d45400640 1 -- 172.21.15.102:0/1673369795 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d300094d0 con 0x7f6d4015b560 2024-03-23T03:50:56.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.155+0000 7f6d45400640 1 --2- 172.21.15.102:0/1673369795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d4015b560 0x7f6d40159930 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f6d30000c00 tx=0x7f6d30004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:56.157 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.156+0000 7f6d3a000640 1 -- 172.21.15.102:0/1673369795 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30039c90 con 0x7f6d4015b560 2024-03-23T03:50:56.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.156+0000 7f6d3a000640 1 -- 172.21.15.102:0/1673369795 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d30039df0 con 0x7f6d4015b560 2024-03-23T03:50:56.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.156+0000 7f6d3a000640 1 -- 172.21.15.102:0/1673369795 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d30043640 con 0x7f6d4015b560 2024-03-23T03:50:56.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.156+0000 7f6d48800640 1 -- 172.21.15.102:0/1673369795 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d40152160 con 0x7f6d4015b560 2024-03-23T03:50:56.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.156+0000 7f6d48800640 1 -- 172.21.15.102:0/1673369795 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d400b3200 con 0x7f6d4015b560 2024-03-23T03:50:56.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.157+0000 7f6d48800640 1 -- 172.21.15.102:0/1673369795 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d40151fa0 con 0x7f6d4015b560 2024-03-23T03:50:56.158 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.158+0000 7f6d3a000640 1 -- 172.21.15.102:0/1673369795 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6d300437a0 con 0x7f6d4015b560 2024-03-23T03:50:56.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.159+0000 7f6d3a000640 1 --2- 172.21.15.102:0/1673369795 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6d2406ac00 0x7f6d2406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:56.159 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.159+0000 7f6d3a000640 1 -- 172.21.15.102:0/1673369795 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6d3003e070 con 0x7f6d4015b560 2024-03-23T03:50:56.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.159+0000 7f6d45e00640 1 --2- 172.21.15.102:0/1673369795 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6d2406ac00 0x7f6d2406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:56.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.160+0000 7f6d45e00640 1 --2- 172.21.15.102:0/1673369795 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6d2406ac00 0x7f6d2406d0c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6d34006fd0 tx=0x7f6d34008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:56.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.163+0000 7f6d3a000640 1 -- 172.21.15.102:0/1673369795 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6d30077620 con 0x7f6d4015b560 2024-03-23T03:50:56.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.400+0000 7f6d48800640 1 -- 172.21.15.102:0/1673369795 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6d400b34b0 con 0x7f6d4015b560 2024-03-23T03:50:56.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.401+0000 7f6d3a000640 1 -- 172.21.15.102:0/1673369795 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v48) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6d3007ac40 con 0x7f6d4015b560 2024-03-23T03:50:56.402 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T03:50:56.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.404+0000 7f6d48800640 1 -- 172.21.15.102:0/1673369795 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6d2406ac00 msgr2=0x7f6d2406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:56.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.404+0000 7f6d48800640 1 --2- 172.21.15.102:0/1673369795 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6d2406ac00 0x7f6d2406d0c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6d34006fd0 tx=0x7f6d34008040 comp rx=0 tx=0).stop 2024-03-23T03:50:56.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.404+0000 7f6d48800640 1 -- 172.21.15.102:0/1673369795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d4015b560 msgr2=0x7f6d40159930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:56.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.404+0000 7f6d48800640 1 --2- 172.21.15.102:0/1673369795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d4015b560 0x7f6d40159930 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f6d30000c00 tx=0x7f6d30004730 comp rx=0 tx=0).stop 2024-03-23T03:50:56.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.404+0000 7f6d48800640 1 -- 172.21.15.102:0/1673369795 shutdown_connections 2024-03-23T03:50:56.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.404+0000 7f6d48800640 1 --2- 172.21.15.102:0/1673369795 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6d2406ac00 0x7f6d2406d0c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:56.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.404+0000 7f6d48800640 1 --2- 172.21.15.102:0/1673369795 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6d4015e500 0x7f6d40159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:56.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.404+0000 7f6d48800640 1 --2- 172.21.15.102:0/1673369795 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6d4015b560 0x7f6d40159930 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:56.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.404+0000 7f6d48800640 1 --2- 172.21.15.102:0/1673369795 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6d400b39b0 0x7f6d4013c160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:56.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.404+0000 7f6d48800640 1 -- 172.21.15.102:0/1673369795 >> 172.21.15.102:0/1673369795 conn(0x7f6d4006a3d0 msgr2=0x7f6d400cb110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:56.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.404+0000 7f6d48800640 1 -- 172.21.15.102:0/1673369795 shutdown_connections 2024-03-23T03:50:56.406 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:56.404+0000 7f6d48800640 1 -- 172.21.15.102:0/1673369795 wait complete. 2024-03-23T03:50:56.424 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-23T03:50:56.424 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:50:56.424 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2024-03-23T03:50:56.424 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T03:50:56.424 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T03:50:56.424 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-23T03:50:56.424 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-23T03:50:56.424 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:50:56.424 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:50:56.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:56.424 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2024-03-23T03:50:56.473 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:50:56.473 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:50:56.473 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:50:56.473 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:56.473 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-03-23T03:50:56.473 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:56.473 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:50:56.473 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:56.473 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f0 2024-03-23T03:50:56.473 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:50:56.473 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:56.474 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f1 2024-03-23T03:50:56.474 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:50:56.474 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:56.474 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:50:56.474 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:56.474 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:50:56.474 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 42414sec preferred_lft 42414sec 2024-03-23T03:50:56.474 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-03-23T03:50:56.474 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:56.474 INFO:teuthology.orchestra.run.smithi102.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:50:56.474 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether aa:17:e3:39:b0:56 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:56.474 INFO:teuthology.orchestra.run.smithi102.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T03:50:56.474 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:56.474 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::5cce:1fff:fe41:5a05/64 scope link 2024-03-23T03:50:56.474 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:56.475 INFO:teuthology.orchestra.run.smithi102.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:50:56.475 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether aa:17:e3:39:b0:56 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T03:50:56.475 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::a817:e3ff:fe39:b056/64 scope link 2024-03-23T03:50:56.475 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:56.475 INFO:teuthology.orchestra.run.smithi102.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:50:56.475 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether ce:9b:95:fb:de:c5 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T03:50:56.475 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::cc9b:95ff:fefb:dec5/64 scope link 2024-03-23T03:50:56.475 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:56.476 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:56.476 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:50:56.562 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-23T03:50:56.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:56.567 DEBUG:teuthology.orchestra.run.smithi102:> route 2024-03-23T03:50:56.619 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2024-03-23T03:50:56.619 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:50:56.619 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:50:56.619 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:50:56.620 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:50:56.620 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:56.620 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:50:56.620 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:50:56.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:50:56.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:50:56.620 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:50:56.620 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:50:56.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:56.802 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-23T03:50:56.851 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T03:50:56.852 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T03:50:56.852 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:50:56.853 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:56.853 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.0 up 2024-03-23T03:50:56.937 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:50:56.937 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T03:50:56.990 INFO:teuthology.orchestra.run.smithi102.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:50:56.990 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:50:56.990 DEBUG:teuthology.orchestra.run.smithi102:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T03:50:57.039 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-23T03:50:57.040 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:57.040 DEBUG:teuthology.orchestra.run.smithi102:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:50:57.177 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:50:57.177 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:50:57.177 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:50:57.199 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:50:57.199 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:50:57.199 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:50:57.199 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:50:57.199 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:50:57.225 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:50:57.225 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:50:57.226 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:50:57.226 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:50:57.226 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:50:57.226 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790" 2024-03-23T03:50:57.226 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/ 2024-03-23T03:50:57.226 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=776b2820-05f2-4e50-8d40-18dd76c5f6a0" will pass to kernel 2024-03-23T03:50:57.228 INFO:teuthology.orchestra.run.smithi102.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-23T03:50:57.228 INFO:teuthology.orchestra.run.smithi102.stdout: You just mounted a file system that supports labels which does not 2024-03-23T03:50:57.228 INFO:teuthology.orchestra.run.smithi102.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T03:50:57.229 INFO:teuthology.orchestra.run.smithi102.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T03:50:57.229 INFO:teuthology.orchestra.run.smithi102.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T03:50:57.233 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T03:50:57.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:57.233 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T03:50:57.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:57.315 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:50:57.376 INFO:teuthology.orchestra.run.smithi102.stdout:client6987 2024-03-23T03:50:57.377 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:50:57.377 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:57.425 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:50:57.670 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.670+0000 7f6ed3200640 1 Processor -- start 2024-03-23T03:50:57.671 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.670+0000 7f6ed3200640 1 -- start start 2024-03-23T03:50:57.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.672+0000 7f6ed3200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6ed4143460 0x7f6ed415aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:57.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.672+0000 7f6ed3200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ed4143930 0x7f6ed40cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:57.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.672+0000 7f6ed3200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f6ed40cdc50 0x7f6ed415e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:57.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.672+0000 7f6ed3200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ed40cc6b0 con 0x7f6ed4143460 2024-03-23T03:50:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.672+0000 7f6ed3200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ed40cc950 con 0x7f6ed40cdc50 2024-03-23T03:50:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.672+0000 7f6ed3200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed40ccc30 con 0x7f6ed4143930 2024-03-23T03:50:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.672+0000 7f6ed1e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6ed4143460 0x7f6ed415aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.672+0000 7f6ed1e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6ed4143460 0x7f6ed415aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58086/0 (socket says 172.21.15.102:58086) 2024-03-23T03:50:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.672+0000 7f6ed1e00640 1 -- 172.21.15.102:0/4071336586 learned_addr learned my addr 172.21.15.102:0/4071336586 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.673+0000 7f6ed1e00640 1 -- 172.21.15.102:0/4071336586 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ed4143930 msgr2=0x7f6ed40cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.673+0000 7f6ed1e00640 1 --2- 172.21.15.102:0/4071336586 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ed4143930 0x7f6ed40cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.673+0000 7f6ed1e00640 1 -- 172.21.15.102:0/4071336586 >> v2:172.21.15.173:3300/0 conn(0x7f6ed40cdc50 msgr2=0x7f6ed415e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.673+0000 7f6ed1e00640 1 --2- 172.21.15.102:0/4071336586 >> v2:172.21.15.173:3300/0 conn(0x7f6ed40cdc50 0x7f6ed415e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:57.674 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.673+0000 7f6ed1e00640 1 -- 172.21.15.102:0/4071336586 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed40ccf10 con 0x7f6ed4143460 2024-03-23T03:50:57.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.674+0000 7f6ed1e00640 1 --2- 172.21.15.102:0/4071336586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed4143460 0x7f6ed415aba0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f6ed40cc190 tx=0x7f6ec402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a2736704032c181 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.674+0000 7f6ed0a00640 1 -- 172.21.15.102:0/4071336586 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4030850 con 0x7f6ed4143460 2024-03-23T03:50:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.674+0000 7f6ed0a00640 1 -- 172.21.15.102:0/4071336586 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec40309b0 con 0x7f6ed4143460 2024-03-23T03:50:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.674+0000 7f6ed0a00640 1 -- 172.21.15.102:0/4071336586 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec4030d40 con 0x7f6ed4143460 2024-03-23T03:50:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.675+0000 7f6ed3200640 1 -- 172.21.15.102:0/4071336586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed4143460 msgr2=0x7f6ed415aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.675+0000 7f6ed3200640 1 --2- 172.21.15.102:0/4071336586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed4143460 0x7f6ed415aba0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f6ed40cc190 tx=0x7f6ec402b990 comp rx=0 tx=0).stop 2024-03-23T03:50:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.675+0000 7f6ed3200640 1 -- 172.21.15.102:0/4071336586 shutdown_connections 2024-03-23T03:50:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.675+0000 7f6ed3200640 1 --2- 172.21.15.102:0/4071336586 >> v2:172.21.15.173:3300/0 conn(0x7f6ed40cdc50 0x7f6ed415e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.675+0000 7f6ed3200640 1 --2- 172.21.15.102:0/4071336586 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ed4143930 0x7f6ed40cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.675+0000 7f6ed3200640 1 --2- 172.21.15.102:0/4071336586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed4143460 0x7f6ed415aba0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.675+0000 7f6ed3200640 1 -- 172.21.15.102:0/4071336586 >> 172.21.15.102:0/4071336586 conn(0x7f6ed4065df0 msgr2=0x7f6ed4069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.675+0000 7f6ed3200640 1 -- 172.21.15.102:0/4071336586 shutdown_connections 2024-03-23T03:50:57.676 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.675+0000 7f6ed3200640 1 -- 172.21.15.102:0/4071336586 wait complete. 2024-03-23T03:50:57.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.677+0000 7f6ed3200640 1 Processor -- start 2024-03-23T03:50:57.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.677+0000 7f6ed3200640 1 -- start start 2024-03-23T03:50:57.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.678+0000 7f6ed3200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ed40cdc50 0x7f6ed40b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:57.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.678+0000 7f6ed3200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ed4143460 0x7f6ed4135de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:57.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.678+0000 7f6ed3200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed4143930 0x7f6ed41381b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:57.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.678+0000 7f6ed3200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed4136830 con 0x7f6ed4143930 2024-03-23T03:50:57.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.678+0000 7f6ed3200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed4136b10 con 0x7f6ed40cdc50 2024-03-23T03:50:57.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.678+0000 7f6ed3200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed4136df0 con 0x7f6ed4143460 2024-03-23T03:50:57.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.678+0000 7f6ed2800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed4143930 0x7f6ed41381b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:57.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.678+0000 7f6ed2800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed4143930 0x7f6ed41381b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58092/0 (socket says 172.21.15.102:58092) 2024-03-23T03:50:57.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.678+0000 7f6ed2800640 1 -- 172.21.15.102:0/2091727915 learned_addr learned my addr 172.21.15.102:0/2091727915 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:57.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.678+0000 7f6ed1e00640 1 --2- 172.21.15.102:0/2091727915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ed40cdc50 0x7f6ed40b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:57.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.678+0000 7f6ed1400640 1 --2- 172.21.15.102:0/2091727915 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ed4143460 0x7f6ed4135de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:57.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.678+0000 7f6ed2800640 1 -- 172.21.15.102:0/2091727915 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ed4143460 msgr2=0x7f6ed4135de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:57.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.679+0000 7f6ed2800640 1 --2- 172.21.15.102:0/2091727915 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ed4143460 0x7f6ed4135de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:57.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.679+0000 7f6ed2800640 1 -- 172.21.15.102:0/2091727915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ed40cdc50 msgr2=0x7f6ed40b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:57.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.679+0000 7f6ed2800640 1 --2- 172.21.15.102:0/2091727915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ed40cdc50 0x7f6ed40b10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:57.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.679+0000 7f6ed2800640 1 -- 172.21.15.102:0/2091727915 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec40094d0 con 0x7f6ed4143930 2024-03-23T03:50:57.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.679+0000 7f6ed1e00640 1 --2- 172.21.15.102:0/2091727915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ed40cdc50 0x7f6ed40b10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:57.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.679+0000 7f6ed1400640 1 --2- 172.21.15.102:0/2091727915 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ed4143460 0x7f6ed4135de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:50:57.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.679+0000 7f6ed2800640 1 --2- 172.21.15.102:0/2091727915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed4143930 0x7f6ed41381b0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc00b970 tx=0x7f6ecc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:57.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.679+0000 7f6eca000640 1 -- 172.21.15.102:0/2091727915 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc00cb40 con 0x7f6ed4143930 2024-03-23T03:50:57.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.679+0000 7f6eca000640 1 -- 172.21.15.102:0/2091727915 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ecc00ccc0 con 0x7f6ed4143930 2024-03-23T03:50:57.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.679+0000 7f6eca000640 1 -- 172.21.15.102:0/2091727915 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc014680 con 0x7f6ed4143930 2024-03-23T03:50:57.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.679+0000 7f6ed3200640 1 -- 172.21.15.102:0/2091727915 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed41370d0 con 0x7f6ed4143930 2024-03-23T03:50:57.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.679+0000 7f6ed3200640 1 -- 172.21.15.102:0/2091727915 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed41f0010 con 0x7f6ed4143930 2024-03-23T03:50:57.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.681+0000 7f6ed3200640 1 -- 172.21.15.102:0/2091727915 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed41375f0 con 0x7f6ed4143930 2024-03-23T03:50:57.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.681+0000 7f6eca000640 1 -- 172.21.15.102:0/2091727915 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6ecc01d3b0 con 0x7f6ed4143930 2024-03-23T03:50:57.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.682+0000 7f6eca000640 1 --2- 172.21.15.102:0/2091727915 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6eb406ac00 0x7f6eb406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:57.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.682+0000 7f6eca000640 1 -- 172.21.15.102:0/2091727915 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6ecc089380 con 0x7f6ed4143930 2024-03-23T03:50:57.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.683+0000 7f6ed1e00640 1 --2- 172.21.15.102:0/2091727915 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6eb406ac00 0x7f6eb406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:57.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.683+0000 7f6ed1e00640 1 --2- 172.21.15.102:0/2091727915 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6eb406ac00 0x7f6eb406d0c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4036000 tx=0x7f6ec403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:57.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.687+0000 7f6eca000640 1 -- 172.21.15.102:0/2091727915 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f6ed41375f0 con 0x7f6ed4143930 2024-03-23T03:50:57.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.921+0000 7f6ed3200640 1 -- 172.21.15.102:0/2091727915 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6ed41375f0 con 0x7f6ed4143930 2024-03-23T03:50:57.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.922+0000 7f6eca000640 1 -- 172.21.15.102:0/2091727915 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v48) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6ed41375f0 con 0x7f6ed4143930 2024-03-23T03:50:57.923 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T03:50:57.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.924+0000 7f6ed3200640 1 -- 172.21.15.102:0/2091727915 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6eb406ac00 msgr2=0x7f6eb406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:57.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.924+0000 7f6ed3200640 1 --2- 172.21.15.102:0/2091727915 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6eb406ac00 0x7f6eb406d0c0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4036000 tx=0x7f6ec403b040 comp rx=0 tx=0).stop 2024-03-23T03:50:57.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.924+0000 7f6ed3200640 1 -- 172.21.15.102:0/2091727915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed4143930 msgr2=0x7f6ed41381b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:57.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.924+0000 7f6ed3200640 1 --2- 172.21.15.102:0/2091727915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed4143930 0x7f6ed41381b0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc00b970 tx=0x7f6ecc00be30 comp rx=0 tx=0).stop 2024-03-23T03:50:57.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.924+0000 7f6ed3200640 1 -- 172.21.15.102:0/2091727915 shutdown_connections 2024-03-23T03:50:57.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.924+0000 7f6ed3200640 1 --2- 172.21.15.102:0/2091727915 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6eb406ac00 0x7f6eb406d0c0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:57.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.924+0000 7f6ed3200640 1 --2- 172.21.15.102:0/2091727915 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ed4143930 0x7f6ed41381b0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:57.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.924+0000 7f6ed3200640 1 --2- 172.21.15.102:0/2091727915 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6ed4143460 0x7f6ed4135de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:57.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.924+0000 7f6ed3200640 1 --2- 172.21.15.102:0/2091727915 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6ed40cdc50 0x7f6ed40b10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:57.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.924+0000 7f6ed3200640 1 -- 172.21.15.102:0/2091727915 >> 172.21.15.102:0/2091727915 conn(0x7f6ed4065df0 msgr2=0x7f6ed4140380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:57.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.924+0000 7f6ed3200640 1 -- 172.21.15.102:0/2091727915 shutdown_connections 2024-03-23T03:50:57.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:57.924+0000 7f6ed3200640 1 -- 172.21.15.102:0/2091727915 wait complete. 2024-03-23T03:50:57.943 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-23T03:50:57.943 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:50:57.943 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi173.front.sepia.ceph.com 2024-03-23T03:50:57.943 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-23T03:50:57.943 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-23T03:50:57.943 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-23T03:50:57.943 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-23T03:50:57.943 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:50:57.943 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:50:57.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:57.944 DEBUG:teuthology.orchestra.run.smithi173:> ip addr 2024-03-23T03:50:58.003 INFO:teuthology.orchestra.run.smithi173.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:50:58.003 INFO:teuthology.orchestra.run.smithi173.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:50:58.003 INFO:teuthology.orchestra.run.smithi173.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:50:58.003 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:58.003 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 ::1/128 scope host 2024-03-23T03:50:58.003 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:58.004 INFO:teuthology.orchestra.run.smithi173.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:50:58.004 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6a brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:58.004 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f0 2024-03-23T03:50:58.004 INFO:teuthology.orchestra.run.smithi173.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:50:58.004 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6b brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:58.004 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f1 2024-03-23T03:50:58.004 INFO:teuthology.orchestra.run.smithi173.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:50:58.004 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:58 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:58.004 INFO:teuthology.orchestra.run.smithi173.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:50:58.004 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:59 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:58.004 INFO:teuthology.orchestra.run.smithi173.stdout: inet 172.21.15.173/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:50:58.004 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft 42384sec preferred_lft 42384sec 2024-03-23T03:50:58.004 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::ec4:7aff:fe88:7259/64 scope link 2024-03-23T03:50:58.004 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:58.004 INFO:teuthology.orchestra.run.smithi173.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:50:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 52:69:f2:bf:6d:02 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:50:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T03:50:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::6894:e9ff:fe5d:1492/64 scope link 2024-03-23T03:50:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:58.005 INFO:teuthology.orchestra.run.smithi173.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:50:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether a2:27:ac:42:68:ce brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T03:50:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::a027:acff:fe42:68ce/64 scope link 2024-03-23T03:50:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:58.005 INFO:teuthology.orchestra.run.smithi173.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:50:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 52:69:f2:bf:6d:02 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-23T03:50:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::5069:f2ff:febf:6d02/64 scope link 2024-03-23T03:50:58.005 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:50:58.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:58.006 DEBUG:teuthology.orchestra.run.smithi173:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:50:58.094 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-03-23T03:50:58.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:58.099 DEBUG:teuthology.orchestra.run.smithi173:> route 2024-03-23T03:50:58.153 INFO:teuthology.orchestra.run.smithi173.stdout:Kernel IP routing table 2024-03-23T03:50:58.153 INFO:teuthology.orchestra.run.smithi173.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:50:58.154 INFO:teuthology.orchestra.run.smithi173.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:50:58.154 INFO:teuthology.orchestra.run.smithi173.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:50:58.154 INFO:teuthology.orchestra.run.smithi173.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:50:58.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:58.155 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-03-23T03:50:58.155 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-03-23T03:50:58.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:50:58.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:50:58.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:50:58.155 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-03-23T03:50:58.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:58.333 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list 2024-03-23T03:50:58.379 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-23T03:50:58.379 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-23T03:50:58.380 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:50:58.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:58.381 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link set brx.0 up 2024-03-23T03:50:58.465 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:50:58.465 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-23T03:50:58.517 INFO:teuthology.orchestra.run.smithi173.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:50:58.517 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:50:58.518 DEBUG:teuthology.orchestra.run.smithi173:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-23T03:50:58.567 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-23T03:50:58.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:58.567 DEBUG:teuthology.orchestra.run.smithi173:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:50:58.702 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:50:58.703 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:50:58.703 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:50:58.723 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:50:58.723 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:50:58.723 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:50:58.724 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:50:58.724 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:50:58.750 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:50:58.750 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:50:58.750 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:50:58.750 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:50:58.750 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:50:58.750 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790" 2024-03-23T03:50:58.750 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/ 2024-03-23T03:50:58.750 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=776b2820-05f2-4e50-8d40-18dd76c5f6a0" will pass to kernel 2024-03-23T03:50:58.752 INFO:teuthology.orchestra.run.smithi173.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-23T03:50:58.752 INFO:teuthology.orchestra.run.smithi173.stdout: You just mounted a file system that supports labels which does not 2024-03-23T03:50:58.752 INFO:teuthology.orchestra.run.smithi173.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T03:50:58.752 INFO:teuthology.orchestra.run.smithi173.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T03:50:58.752 INFO:teuthology.orchestra.run.smithi173.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T03:50:58.755 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T03:50:58.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:58.755 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-23T03:50:58.836 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:58.836 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:50:58.889 INFO:teuthology.orchestra.run.smithi173.stdout:client5653 2024-03-23T03:50:58.889 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:50:58.889 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:58.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:58.940 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir) 2024-03-23T03:50:59.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:59.008 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_files ./subdir) 2024-03-23T03:50:59.064 INFO:teuthology.orchestra.run.smithi102.stderr:./subdir: ceph.quota.max_files: No such attribute 2024-03-23T03:50:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:50:59.065 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:59.065 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec getfattr --only-values -n ceph.quota.max_files ./subdir) 2024-03-23T03:50:59.128 INFO:teuthology.orchestra.run.smithi173.stderr:./subdir: ceph.quota.max_files: No such attribute 2024-03-23T03:50:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:50:59.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:59.129 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_files -v 10 ./subdir) 2024-03-23T03:50:59.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:59.189 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.quota.max_files ./subdir) 2024-03-23T03:50:59.241 INFO:teuthology.orchestra.run.smithi102.stdout:10 2024-03-23T03:50:59.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:50:59.241 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec getfattr --only-values -n ceph.quota.max_files ./subdir) 2024-03-23T03:50:59.300 INFO:teuthology.orchestra.run.smithi173.stdout:10 2024-03-23T03:50:59.300 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T03:50:59.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:50:59.416 DEBUG:teuthology.orchestra.run.smithi173:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:50:59.530 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T03:50:59.530 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:50:59.530 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:59.587 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:50:59.587 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:50:59.636 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:50:59.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.891+0000 7f93a5c00640 1 Processor -- start 2024-03-23T03:50:59.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.891+0000 7f93a5c00640 1 -- start start 2024-03-23T03:50:59.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.893+0000 7f93a5c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f93a00b39b0 0x7f93a00b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:59.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.893+0000 7f93a5c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a006c6f0 con 0x7f93a015b560 2024-03-23T03:50:59.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.893+0000 7f93a5c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93a006c9d0 con 0x7f93a015c400 2024-03-23T03:50:59.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.893+0000 7f93a5c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93a00b4370 con 0x7f93a00b39b0 2024-03-23T03:50:59.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.893+0000 7f939e000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f93a00b39b0 0x7f93a00b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:59.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.893+0000 7f939ea00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f93a015b560 0x7f93a015b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55550/0 (socket says 172.21.15.102:55550) 2024-03-23T03:50:59.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.893+0000 7f939ea00640 1 -- 172.21.15.102:0/4177924245 learned_addr learned my addr 172.21.15.102:0/4177924245 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:50:59.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.894+0000 7f939e000640 1 -- 172.21.15.102:0/4177924245 >> v1:172.21.15.173:6789/0 conn(0x7f93a015c400 legacy=0x7f93a015c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:50:59.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.894+0000 7f939e000640 1 -- 172.21.15.102:0/4177924245 >> v1:172.21.15.102:6789/0 conn(0x7f93a015b560 legacy=0x7f93a015b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:59.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.894+0000 7f939e000640 1 -- 172.21.15.102:0/4177924245 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a006ccb0 con 0x7f93a00b39b0 2024-03-23T03:50:59.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.895+0000 7f939e000640 1 --2- 172.21.15.102:0/4177924245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93a00b39b0 0x7f93a00b3e30 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f938c0097a0 tx=0x7f938c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e171762f3a49eff8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:59.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.895+0000 7f939d600640 1 -- 172.21.15.102:0/4177924245 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c031830 con 0x7f93a00b39b0 2024-03-23T03:50:59.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.895+0000 7f939d600640 1 -- 172.21.15.102:0/4177924245 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938c031990 con 0x7f93a00b39b0 2024-03-23T03:50:59.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.895+0000 7f939d600640 1 -- 172.21.15.102:0/4177924245 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c031d20 con 0x7f93a00b39b0 2024-03-23T03:50:59.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.896+0000 7f93a5c00640 1 -- 172.21.15.102:0/4177924245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93a00b39b0 msgr2=0x7f93a00b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:59.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.896+0000 7f93a5c00640 1 --2- 172.21.15.102:0/4177924245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93a00b39b0 0x7f93a00b3e30 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f938c0097a0 tx=0x7f938c02b8d0 comp rx=0 tx=0).stop 2024-03-23T03:50:59.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.896+0000 7f93a5c00640 1 -- 172.21.15.102:0/4177924245 shutdown_connections 2024-03-23T03:50:59.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.896+0000 7f93a5c00640 1 --2- 172.21.15.102:0/4177924245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93a00b39b0 0x7f93a00b3e30 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:59.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.896+0000 7f93a5c00640 1 -- 172.21.15.102:0/4177924245 >> 172.21.15.102:0/4177924245 conn(0x7f93a006a3d0 msgr2=0x7f93a00b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:50:59.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.896+0000 7f93a5c00640 1 -- 172.21.15.102:0/4177924245 shutdown_connections 2024-03-23T03:50:59.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.896+0000 7f93a5c00640 1 -- 172.21.15.102:0/4177924245 wait complete. 2024-03-23T03:50:59.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.898+0000 7f93a5c00640 1 Processor -- start 2024-03-23T03:50:59.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.898+0000 7f93a5c00640 1 -- start start 2024-03-23T03:50:59.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.899+0000 7f93a5c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93a00b39b0 0x7f93a014c160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:59.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.899+0000 7f93a5c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f93a015b560 0x7f93a0147420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:59.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.899+0000 7f93a5c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93a015c400 0x7f93a0159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:59.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.899+0000 7f93a5c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a0153260 con 0x7f93a00b39b0 2024-03-23T03:50:59.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.899+0000 7f93a5c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a0153540 con 0x7f93a015c400 2024-03-23T03:50:59.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.899+0000 7f93a5c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93a0153820 con 0x7f93a015b560 2024-03-23T03:50:59.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.899+0000 7f939ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93a00b39b0 0x7f93a014c160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:59.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.899+0000 7f939ea00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93a00b39b0 0x7f93a014c160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58110/0 (socket says 172.21.15.102:58110) 2024-03-23T03:50:59.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.899+0000 7f939ea00640 1 -- 172.21.15.102:0/2635487830 learned_addr learned my addr 172.21.15.102:0/2635487830 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:50:59.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.900+0000 7f939e000640 1 --2- 172.21.15.102:0/2635487830 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f93a015b560 0x7f93a0147420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:59.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.900+0000 7f939f400640 1 --2- 172.21.15.102:0/2635487830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93a015c400 0x7f93a0159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:59.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.900+0000 7f939ea00640 1 -- 172.21.15.102:0/2635487830 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f93a015b560 msgr2=0x7f93a0147420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:59.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.900+0000 7f939ea00640 1 --2- 172.21.15.102:0/2635487830 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f93a015b560 0x7f93a0147420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:59.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.900+0000 7f939ea00640 1 -- 172.21.15.102:0/2635487830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93a015c400 msgr2=0x7f93a0159a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:50:59.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.900+0000 7f939ea00640 1 --2- 172.21.15.102:0/2635487830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93a015c400 0x7f93a0159a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:50:59.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.900+0000 7f939ea00640 1 -- 172.21.15.102:0/2635487830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f938c009400 con 0x7f93a00b39b0 2024-03-23T03:50:59.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.900+0000 7f939ea00640 1 --2- 172.21.15.102:0/2635487830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93a00b39b0 0x7f93a014c160 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f939000c4a0 tx=0x7f939000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:59.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.900+0000 7f9396a00640 1 -- 172.21.15.102:0/2635487830 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9390005ce0 con 0x7f93a00b39b0 2024-03-23T03:50:59.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.900+0000 7f9396a00640 1 -- 172.21.15.102:0/2635487830 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9390005e40 con 0x7f93a00b39b0 2024-03-23T03:50:59.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.900+0000 7f93a5c00640 1 -- 172.21.15.102:0/2635487830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a0153b00 con 0x7f93a00b39b0 2024-03-23T03:50:59.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.900+0000 7f9396a00640 1 -- 172.21.15.102:0/2635487830 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9390007270 con 0x7f93a00b39b0 2024-03-23T03:50:59.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.901+0000 7f93a5c00640 1 -- 172.21.15.102:0/2635487830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a014c6a0 con 0x7f93a00b39b0 2024-03-23T03:50:59.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.902+0000 7f93a5c00640 1 -- 172.21.15.102:0/2635487830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a006ac20 con 0x7f93a00b39b0 2024-03-23T03:50:59.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.903+0000 7f9396a00640 1 -- 172.21.15.102:0/2635487830 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9390007490 con 0x7f93a00b39b0 2024-03-23T03:50:59.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.903+0000 7f9396a00640 1 --2- 172.21.15.102:0/2635487830 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f938006ac00 0x7f938006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:50:59.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.903+0000 7f9396a00640 1 -- 172.21.15.102:0/2635487830 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f93900889e0 con 0x7f93a00b39b0 2024-03-23T03:50:59.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.904+0000 7f939e000640 1 --2- 172.21.15.102:0/2635487830 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f938006ac00 0x7f938006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:50:59.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.904+0000 7f939e000640 1 --2- 172.21.15.102:0/2635487830 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f938006ac00 0x7f938006d0c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f938c036000 tx=0x7f938c008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:50:59.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:50:59.908+0000 7f9396a00640 1 -- 172.21.15.102:0/2635487830 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9390053a90 con 0x7f93a00b39b0 2024-03-23T03:51:00.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.107+0000 7f93a5c00640 1 -- 172.21.15.102:0/2635487830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93a0073390 con 0x7f93a00b39b0 2024-03-23T03:51:00.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.110+0000 7f9396a00640 1 -- 172.21.15.102:0/2635487830 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16297 (secure 0 0 0) 0x7f9390017020 con 0x7f93a00b39b0 2024-03-23T03:51:00.110 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:00.110 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":93,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:50.300471+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:50:47.750670+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-23T03:50:48.305583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:51:00.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.112+0000 7f93a5c00640 1 -- 172.21.15.102:0/2635487830 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f938006ac00 msgr2=0x7f938006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:00.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.112+0000 7f93a5c00640 1 --2- 172.21.15.102:0/2635487830 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f938006ac00 0x7f938006d0c0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f938c036000 tx=0x7f938c008030 comp rx=0 tx=0).stop 2024-03-23T03:51:00.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.112+0000 7f93a5c00640 1 -- 172.21.15.102:0/2635487830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93a00b39b0 msgr2=0x7f93a014c160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.112+0000 7f93a5c00640 1 --2- 172.21.15.102:0/2635487830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93a00b39b0 0x7f93a014c160 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f939000c4a0 tx=0x7f939000c960 comp rx=0 tx=0).stop 2024-03-23T03:51:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.112+0000 7f93a5c00640 1 -- 172.21.15.102:0/2635487830 shutdown_connections 2024-03-23T03:51:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.112+0000 7f93a5c00640 1 --2- 172.21.15.102:0/2635487830 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f938006ac00 0x7f938006d0c0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.112+0000 7f93a5c00640 1 --2- 172.21.15.102:0/2635487830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93a015c400 0x7f93a0159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.112+0000 7f93a5c00640 1 --2- 172.21.15.102:0/2635487830 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f93a015b560 0x7f93a0147420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.112+0000 7f93a5c00640 1 --2- 172.21.15.102:0/2635487830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93a00b39b0 0x7f93a014c160 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.112+0000 7f93a5c00640 1 -- 172.21.15.102:0/2635487830 >> 172.21.15.102:0/2635487830 conn(0x7f93a006a3d0 msgr2=0x7f93a00cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.113+0000 7f93a5c00640 1 -- 172.21.15.102:0/2635487830 shutdown_connections 2024-03-23T03:51:00.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.113+0000 7f93a5c00640 1 -- 172.21.15.102:0/2635487830 wait complete. 2024-03-23T03:51:00.132 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:50:47.750670+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:50:48.305583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:51:00.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:51:00.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.345+0000 7f2e3c800640 1 Processor -- start 2024-03-23T03:51:00.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.345+0000 7f2e3c800640 1 -- start start 2024-03-23T03:51:00.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.347+0000 7f2e3c800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f2e340cd740 0x7f2e340b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:00.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.347+0000 7f2e3c800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2e340b5aa0 0x7f2e340b5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:00.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.347+0000 7f2e3c800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e340d20c0 con 0x7f2e340d0d40 2024-03-23T03:51:00.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.347+0000 7f2e3c800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e340d14b0 con 0x7f2e340b5aa0 2024-03-23T03:51:00.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.347+0000 7f2e3c800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e340d1790 con 0x7f2e340cd740 2024-03-23T03:51:00.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.347+0000 7f2e3a800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2e340b5aa0 0x7f2e340b5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:00.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.348+0000 7f2e3a800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2e340b5aa0 0x7f2e340b5f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58120/0 (socket says 172.21.15.102:58120) 2024-03-23T03:51:00.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.348+0000 7f2e3a800640 1 -- 172.21.15.102:0/3803505619 learned_addr learned my addr 172.21.15.102:0/3803505619 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:00.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.348+0000 7f2e39e00640 1 --2- 172.21.15.102:0/3803505619 >> v2:172.21.15.173:3300/0 conn(0x7f2e340cd740 0x7f2e340b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:00.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.348+0000 7f2e3a800640 1 -- 172.21.15.102:0/3803505619 >> v2:172.21.15.173:3300/0 conn(0x7f2e340cd740 msgr2=0x7f2e340b84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:00.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.348+0000 7f2e3a800640 1 --2- 172.21.15.102:0/3803505619 >> v2:172.21.15.173:3300/0 conn(0x7f2e340cd740 0x7f2e340b84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.348+0000 7f2e3a800640 1 -- 172.21.15.102:0/3803505619 >> v1:172.21.15.173:6789/0 conn(0x7f2e340d0d40 legacy=0x7f2e340cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:00.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.348+0000 7f2e3a800640 1 -- 172.21.15.102:0/3803505619 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e340d1a70 con 0x7f2e340b5aa0 2024-03-23T03:51:00.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.349+0000 7f2e3a800640 1 --2- 172.21.15.102:0/3803505619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e340b5aa0 0x7f2e340b5f30 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f2e3408a100 tx=0x7f2e3002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d841295d5b043bd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:00.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.350+0000 7f2e38a00640 1 -- 172.21.15.102:0/3803505619 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3002cb20 con 0x7f2e340b5aa0 2024-03-23T03:51:00.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.350+0000 7f2e38a00640 1 -- 172.21.15.102:0/3803505619 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3002cc80 con 0x7f2e340b5aa0 2024-03-23T03:51:00.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.350+0000 7f2e38a00640 1 -- 172.21.15.102:0/3803505619 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e300343f0 con 0x7f2e340b5aa0 2024-03-23T03:51:00.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.350+0000 7f2e3c800640 1 -- 172.21.15.102:0/3803505619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e340b5aa0 msgr2=0x7f2e340b5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:00.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.350+0000 7f2e3c800640 1 --2- 172.21.15.102:0/3803505619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e340b5aa0 0x7f2e340b5f30 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f2e3408a100 tx=0x7f2e3002bf80 comp rx=0 tx=0).stop 2024-03-23T03:51:00.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.351+0000 7f2e3c800640 1 -- 172.21.15.102:0/3803505619 shutdown_connections 2024-03-23T03:51:00.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.351+0000 7f2e3c800640 1 --2- 172.21.15.102:0/3803505619 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e340b5aa0 0x7f2e340b5f30 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.351+0000 7f2e3c800640 1 --2- 172.21.15.102:0/3803505619 >> v2:172.21.15.173:3300/0 conn(0x7f2e340cd740 0x7f2e340b84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.351+0000 7f2e3c800640 1 -- 172.21.15.102:0/3803505619 >> 172.21.15.102:0/3803505619 conn(0x7f2e340cf5d0 msgr2=0x7f2e340cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:00.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.351+0000 7f2e3c800640 1 -- 172.21.15.102:0/3803505619 shutdown_connections 2024-03-23T03:51:00.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.351+0000 7f2e3c800640 1 -- 172.21.15.102:0/3803505619 wait complete. 2024-03-23T03:51:00.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.353+0000 7f2e3c800640 1 Processor -- start 2024-03-23T03:51:00.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.353+0000 7f2e3c800640 1 -- start start 2024-03-23T03:51:00.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.354+0000 7f2e3c800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e340b5aa0 0x7f2e340caa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:00.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.354+0000 7f2e3c800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2e340cd740 0x7f2e34156820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:00.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.354+0000 7f2e3c800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e340d0d40 0x7f2e34156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:00.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.354+0000 7f2e3c800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340cb4e0 con 0x7f2e340b5aa0 2024-03-23T03:51:00.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.354+0000 7f2e3c800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340cb7c0 con 0x7f2e340d0d40 2024-03-23T03:51:00.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.354+0000 7f2e3c800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e340cbaa0 con 0x7f2e340cd740 2024-03-23T03:51:00.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.354+0000 7f2e39e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e340b5aa0 0x7f2e340caa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.354+0000 7f2e39e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e340b5aa0 0x7f2e340caa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58136/0 (socket says 172.21.15.102:58136) 2024-03-23T03:51:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.354+0000 7f2e39e00640 1 -- 172.21.15.102:0/3013898285 learned_addr learned my addr 172.21.15.102:0/3013898285 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.355+0000 7f2e39400640 1 --2- 172.21.15.102:0/3013898285 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2e340cd740 0x7f2e34156820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.355+0000 7f2e3a800640 1 --2- 172.21.15.102:0/3013898285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e340d0d40 0x7f2e34156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.355+0000 7f2e39e00640 1 -- 172.21.15.102:0/3013898285 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2e340cd740 msgr2=0x7f2e34156820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.355+0000 7f2e39e00640 1 --2- 172.21.15.102:0/3013898285 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2e340cd740 0x7f2e34156820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.355+0000 7f2e39e00640 1 -- 172.21.15.102:0/3013898285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e340d0d40 msgr2=0x7f2e34156d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.355+0000 7f2e39e00640 1 --2- 172.21.15.102:0/3013898285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e340d0d40 0x7f2e34156d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.355+0000 7f2e39e00640 1 -- 172.21.15.102:0/3013898285 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e30009b30 con 0x7f2e340b5aa0 2024-03-23T03:51:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.355+0000 7f2e39400640 1 --2- 172.21.15.102:0/3013898285 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2e340cd740 0x7f2e34156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.355+0000 7f2e3a800640 1 --2- 172.21.15.102:0/3013898285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e340d0d40 0x7f2e34156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.355+0000 7f2e39e00640 1 --2- 172.21.15.102:0/3013898285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e340b5aa0 0x7f2e340caa90 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800e9b0 tx=0x7f2e2800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.355+0000 7f2e2e000640 1 -- 172.21.15.102:0/3013898285 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e28009670 con 0x7f2e340b5aa0 2024-03-23T03:51:00.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.355+0000 7f2e2e000640 1 -- 172.21.15.102:0/3013898285 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2800c9a0 con 0x7f2e340b5aa0 2024-03-23T03:51:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.355+0000 7f2e2e000640 1 -- 172.21.15.102:0/3013898285 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2800cd50 con 0x7f2e340b5aa0 2024-03-23T03:51:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.356+0000 7f2e3c800640 1 -- 172.21.15.102:0/3013898285 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e340cbd80 con 0x7f2e340b5aa0 2024-03-23T03:51:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.356+0000 7f2e3c800640 1 -- 172.21.15.102:0/3013898285 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e341f1f60 con 0x7f2e340b5aa0 2024-03-23T03:51:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.357+0000 7f2e3c800640 1 -- 172.21.15.102:0/3013898285 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e34073390 con 0x7f2e340b5aa0 2024-03-23T03:51:00.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.358+0000 7f2e2e000640 1 -- 172.21.15.102:0/3013898285 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2e28013470 con 0x7f2e340b5aa0 2024-03-23T03:51:00.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.359+0000 7f2e2e000640 1 --2- 172.21.15.102:0/3013898285 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2e1806ac30 0x7f2e1806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:00.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.359+0000 7f2e2e000640 1 -- 172.21.15.102:0/3013898285 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2e280896f0 con 0x7f2e340b5aa0 2024-03-23T03:51:00.358 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.359+0000 7f2e39400640 1 --2- 172.21.15.102:0/3013898285 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2e1806ac30 0x7f2e1806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:00.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.360+0000 7f2e39400640 1 --2- 172.21.15.102:0/3013898285 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2e1806ac30 0x7f2e1806d0f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2e340cc700 tx=0x7f2e24009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:00.362 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.363+0000 7f2e2e000640 1 -- 172.21.15.102:0/3013898285 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2e28054780 con 0x7f2e340b5aa0 2024-03-23T03:51:00.557 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.558+0000 7f2e3c800640 1 -- 172.21.15.102:0/3013898285 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2e340b20b0 con 0x7f2e340b5aa0 2024-03-23T03:51:00.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.559+0000 7f2e2e000640 1 -- 172.21.15.102:0/3013898285 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2e28087e60 con 0x7f2e340b5aa0 2024-03-23T03:51:00.558 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:51:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.561+0000 7f2e3c800640 1 -- 172.21.15.102:0/3013898285 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2e1806ac30 msgr2=0x7f2e1806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.561+0000 7f2e3c800640 1 --2- 172.21.15.102:0/3013898285 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2e1806ac30 0x7f2e1806d0f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2e340cc700 tx=0x7f2e24009040 comp rx=0 tx=0).stop 2024-03-23T03:51:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.561+0000 7f2e3c800640 1 -- 172.21.15.102:0/3013898285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e340b5aa0 msgr2=0x7f2e340caa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.561+0000 7f2e3c800640 1 --2- 172.21.15.102:0/3013898285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e340b5aa0 0x7f2e340caa90 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f2e2800e9b0 tx=0x7f2e2800ee70 comp rx=0 tx=0).stop 2024-03-23T03:51:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.561+0000 7f2e3c800640 1 -- 172.21.15.102:0/3013898285 shutdown_connections 2024-03-23T03:51:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.561+0000 7f2e3c800640 1 --2- 172.21.15.102:0/3013898285 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2e1806ac30 0x7f2e1806d0f0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.561+0000 7f2e3c800640 1 --2- 172.21.15.102:0/3013898285 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2e340d0d40 0x7f2e34156d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.561 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.561+0000 7f2e3c800640 1 --2- 172.21.15.102:0/3013898285 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2e340cd740 0x7f2e34156820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.561+0000 7f2e3c800640 1 --2- 172.21.15.102:0/3013898285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e340b5aa0 0x7f2e340caa90 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.561+0000 7f2e3c800640 1 -- 172.21.15.102:0/3013898285 >> 172.21.15.102:0/3013898285 conn(0x7f2e340cf5d0 msgr2=0x7f2e3415ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.561+0000 7f2e3c800640 1 -- 172.21.15.102:0/3013898285 shutdown_connections 2024-03-23T03:51:00.562 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.561+0000 7f2e3c800640 1 -- 172.21.15.102:0/3013898285 wait complete. 2024-03-23T03:51:00.579 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:51:00.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.787+0000 7fba17a00640 1 Processor -- start 2024-03-23T03:51:00.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.787+0000 7fba17a00640 1 -- start start 2024-03-23T03:51:00.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.788+0000 7fba17a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fba100705b0 0x7fba100ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:00.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.788+0000 7fba17a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba100caec0 con 0x7fba10158db0 2024-03-23T03:51:00.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.789+0000 7fba17a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba100cba90 con 0x7fba10070150 2024-03-23T03:51:00.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.789+0000 7fba17a00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba10159190 con 0x7fba100705b0 2024-03-23T03:51:00.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.789+0000 7fba15000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fba10158db0 0x7fba1015d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55564/0 (socket says 172.21.15.102:55564) 2024-03-23T03:51:00.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.789+0000 7fba15000640 1 -- 172.21.15.102:0/2211129691 learned_addr learned my addr 172.21.15.102:0/2211129691 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:00.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.789+0000 7fba0f400640 1 -- 172.21.15.102:0/2211129691 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954158535 0 0) 0x7fba100caec0 con 0x7fba10158db0 2024-03-23T03:51:00.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.789+0000 7fba0f400640 1 -- 172.21.15.102:0/2211129691 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f8003520 con 0x7fba10158db0 2024-03-23T03:51:00.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.789+0000 7fba15a00640 1 --2- 172.21.15.102:0/2211129691 >> v2:172.21.15.173:3300/0 conn(0x7fba100705b0 0x7fba100ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:00.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.790+0000 7fba0f400640 1 -- 172.21.15.102:0/2211129691 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 237129455 0 0) 0x7fb9f8003520 con 0x7fba10158db0 2024-03-23T03:51:00.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.790+0000 7fba0f400640 1 -- 172.21.15.102:0/2211129691 >> v2:172.21.15.173:3300/0 conn(0x7fba100705b0 msgr2=0x7fba100ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:00.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.790+0000 7fba0f400640 1 --2- 172.21.15.102:0/2211129691 >> v2:172.21.15.173:3300/0 conn(0x7fba100705b0 0x7fba100ccf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.790+0000 7fba0f400640 1 -- 172.21.15.102:0/2211129691 >> v1:172.21.15.173:6789/0 conn(0x7fba10070150 legacy=0x7fba100cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:00.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.790+0000 7fba0f400640 1 -- 172.21.15.102:0/2211129691 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba100cabe0 con 0x7fba10158db0 2024-03-23T03:51:00.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.790+0000 7fba0f400640 1 -- 172.21.15.102:0/2211129691 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fba04001670 con 0x7fba10158db0 2024-03-23T03:51:00.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.790+0000 7fba0f400640 1 -- 172.21.15.102:0/2211129691 >> v1:172.21.15.102:6789/0 conn(0x7fba10158db0 legacy=0x7fba1015d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:00.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.790+0000 7fba0f400640 1 --2- 172.21.15.102:0/2211129691 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb9f8003970 0x7fb9f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:00.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.790+0000 7fba0f400640 1 --2- 172.21.15.102:0/2211129691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9f80047c0 0x7fb9f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:00.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.790+0000 7fba0f400640 1 --2- 172.21.15.102:0/2211129691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f8005180 0x7fb9f800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:00.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.790+0000 7fba0f400640 1 -- 172.21.15.102:0/2211129691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba10159190 con 0x7fb9f80047c0 2024-03-23T03:51:00.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.790+0000 7fba0f400640 1 -- 172.21.15.102:0/2211129691 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f800c0d0 con 0x7fb9f8005180 2024-03-23T03:51:00.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.790+0000 7fba0f400640 1 -- 172.21.15.102:0/2211129691 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f800c290 con 0x7fb9f8003970 2024-03-23T03:51:00.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.790+0000 7fba0fe00640 1 --2- 172.21.15.102:0/2211129691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9f80047c0 0x7fb9f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:00.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.790+0000 7fba15a00640 1 --2- 172.21.15.102:0/2211129691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f8005180 0x7fb9f800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:00.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.790+0000 7fba15000640 1 --2- 172.21.15.102:0/2211129691 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb9f8003970 0x7fb9f8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:00.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.790+0000 7fba0fe00640 1 -- 172.21.15.102:0/2211129691 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb9f8003970 msgr2=0x7fb9f8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:00.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.791+0000 7fba0fe00640 1 --2- 172.21.15.102:0/2211129691 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb9f8003970 0x7fb9f8003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.791+0000 7fba0fe00640 1 -- 172.21.15.102:0/2211129691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f8005180 msgr2=0x7fb9f800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:00.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.791+0000 7fba0fe00640 1 --2- 172.21.15.102:0/2211129691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f8005180 0x7fb9f800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.791+0000 7fba0fe00640 1 -- 172.21.15.102:0/2211129691 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f8003520 con 0x7fb9f80047c0 2024-03-23T03:51:00.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.791+0000 7fba15a00640 1 --2- 172.21.15.102:0/2211129691 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f8005180 0x7fb9f800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:00.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.791+0000 7fba15a00640 1 -- 172.21.15.102:0/2211129691 reap_dead start 2024-03-23T03:51:00.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.792+0000 7fba0fe00640 1 --2- 172.21.15.102:0/2211129691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9f80047c0 0x7fb9f8004c40 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc001750 tx=0x7fb9f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97460f8e14388fc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:00.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.792+0000 7fba0f400640 1 -- 172.21.15.102:0/2211129691 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc0052d0 con 0x7fb9f80047c0 2024-03-23T03:51:00.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.792+0000 7fba0f400640 1 -- 172.21.15.102:0/2211129691 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9fc02e050 con 0x7fb9f80047c0 2024-03-23T03:51:00.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.792+0000 7fba0f400640 1 -- 172.21.15.102:0/2211129691 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc003830 con 0x7fb9f80047c0 2024-03-23T03:51:00.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.792+0000 7fba17a00640 1 -- 172.21.15.102:0/2211129691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9f80047c0 msgr2=0x7fb9f8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:00.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.792+0000 7fba17a00640 1 --2- 172.21.15.102:0/2211129691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9f80047c0 0x7fb9f8004c40 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc001750 tx=0x7fb9f80057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.792+0000 7fba17a00640 1 -- 172.21.15.102:0/2211129691 shutdown_connections 2024-03-23T03:51:00.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.792+0000 7fba17a00640 1 --2- 172.21.15.102:0/2211129691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb9f80047c0 0x7fb9f8004c40 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.792+0000 7fba17a00640 1 -- 172.21.15.102:0/2211129691 >> 172.21.15.102:0/2211129691 conn(0x7fba1006a0b0 msgr2=0x7fba1006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:00.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.793+0000 7fba17a00640 1 -- 172.21.15.102:0/2211129691 shutdown_connections 2024-03-23T03:51:00.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.793+0000 7fba17a00640 1 -- 172.21.15.102:0/2211129691 wait complete. 2024-03-23T03:51:00.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.794+0000 7fba17a00640 1 Processor -- start 2024-03-23T03:51:00.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.794+0000 7fba17a00640 1 -- start start 2024-03-23T03:51:00.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.795+0000 7fba17a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f80047c0 0x7fba1015af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:00.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.795+0000 7fba17a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fba10168e90 0x7fba100b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:00.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.795+0000 7fba17a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba10169310 0x7fba1013eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:00.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.795+0000 7fba17a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba100b2ce0 con 0x7fba10169310 2024-03-23T03:51:00.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.795+0000 7fba17a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba100b2fc0 con 0x7fb9f80047c0 2024-03-23T03:51:00.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.795+0000 7fba17a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1015a730 con 0x7fba10168e90 2024-03-23T03:51:00.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.796+0000 7fba15a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba10169310 0x7fba1013eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:00.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.796+0000 7fba15a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba10169310 0x7fba1013eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58154/0 (socket says 172.21.15.102:58154) 2024-03-23T03:51:00.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.796+0000 7fba15a00640 1 -- 172.21.15.102:0/2301777380 learned_addr learned my addr 172.21.15.102:0/2301777380 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:00.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.796+0000 7fba15000640 1 --2- 172.21.15.102:0/2301777380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f80047c0 0x7fba1015af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:00.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.796+0000 7fba0fe00640 1 --2- 172.21.15.102:0/2301777380 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fba10168e90 0x7fba100b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:00.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.796+0000 7fba15a00640 1 -- 172.21.15.102:0/2301777380 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fba10168e90 msgr2=0x7fba100b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:00.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.796+0000 7fba15a00640 1 --2- 172.21.15.102:0/2301777380 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fba10168e90 0x7fba100b3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.796+0000 7fba15a00640 1 -- 172.21.15.102:0/2301777380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f80047c0 msgr2=0x7fba1015af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:00.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.796+0000 7fba15a00640 1 --2- 172.21.15.102:0/2301777380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f80047c0 0x7fba1015af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:00.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.796+0000 7fba15a00640 1 -- 172.21.15.102:0/2301777380 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1015aa10 con 0x7fba10169310 2024-03-23T03:51:00.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.797+0000 7fba15a00640 1 --2- 172.21.15.102:0/2301777380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba10169310 0x7fba1013eae0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fba1006db20 tx=0x7fba08009cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:00.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.797+0000 7fba0cc00640 1 -- 172.21.15.102:0/2301777380 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08007c70 con 0x7fba10169310 2024-03-23T03:51:00.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.797+0000 7fba0cc00640 1 -- 172.21.15.102:0/2301777380 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba08007df0 con 0x7fba10169310 2024-03-23T03:51:00.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.797+0000 7fba0cc00640 1 -- 172.21.15.102:0/2301777380 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08014640 con 0x7fba10169310 2024-03-23T03:51:00.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.797+0000 7fba17a00640 1 -- 172.21.15.102:0/2301777380 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba101402f0 con 0x7fba10169310 2024-03-23T03:51:00.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.797+0000 7fba17a00640 1 -- 172.21.15.102:0/2301777380 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba1015b470 con 0x7fba10169310 2024-03-23T03:51:00.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.798+0000 7fba17a00640 1 -- 172.21.15.102:0/2301777380 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba100ce150 con 0x7fba10169310 2024-03-23T03:51:00.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.800+0000 7fba0cc00640 1 -- 172.21.15.102:0/2301777380 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fba0801d3b0 con 0x7fba10169310 2024-03-23T03:51:00.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.800+0000 7fba0cc00640 1 --2- 172.21.15.102:0/2301777380 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb9f006ac00 0x7fb9f006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:00.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.801+0000 7fba0cc00640 1 -- 172.21.15.102:0/2301777380 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fba080892e0 con 0x7fba10169310 2024-03-23T03:51:00.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.801+0000 7fba15000640 1 --2- 172.21.15.102:0/2301777380 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb9f006ac00 0x7fb9f006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:00.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.801+0000 7fba15000640 1 --2- 172.21.15.102:0/2301777380 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb9f006ac00 0x7fb9f006d0c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fba04001200 tx=0x7fba04009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:00.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.805+0000 7fba0cc00640 1 -- 172.21.15.102:0/2301777380 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fba08054380 con 0x7fba10169310 2024-03-23T03:51:00.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.998+0000 7fba17a00640 1 -- 172.21.15.102:0/2301777380 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fba10073390 con 0x7fba10169310 2024-03-23T03:51:00.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:00.999+0000 7fba0cc00640 1 -- 172.21.15.102:0/2301777380 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7fba080579a0 con 0x7fba10169310 2024-03-23T03:51:00.999 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:51:01.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.001+0000 7fba17a00640 1 -- 172.21.15.102:0/2301777380 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb9f006ac00 msgr2=0x7fb9f006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:01.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.001+0000 7fba17a00640 1 --2- 172.21.15.102:0/2301777380 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb9f006ac00 0x7fb9f006d0c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fba04001200 tx=0x7fba04009040 comp rx=0 tx=0).stop 2024-03-23T03:51:01.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.001+0000 7fba17a00640 1 -- 172.21.15.102:0/2301777380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba10169310 msgr2=0x7fba1013eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:01.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.001+0000 7fba17a00640 1 --2- 172.21.15.102:0/2301777380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba10169310 0x7fba1013eae0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fba1006db20 tx=0x7fba08009cd0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.001+0000 7fba17a00640 1 -- 172.21.15.102:0/2301777380 shutdown_connections 2024-03-23T03:51:01.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.002+0000 7fba17a00640 1 --2- 172.21.15.102:0/2301777380 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb9f006ac00 0x7fb9f006d0c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.002+0000 7fba17a00640 1 --2- 172.21.15.102:0/2301777380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fba10169310 0x7fba1013eae0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.002+0000 7fba17a00640 1 --2- 172.21.15.102:0/2301777380 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fba10168e90 0x7fba100b3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.002+0000 7fba17a00640 1 --2- 172.21.15.102:0/2301777380 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f80047c0 0x7fba1015af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.002+0000 7fba17a00640 1 -- 172.21.15.102:0/2301777380 >> 172.21.15.102:0/2301777380 conn(0x7fba1006a0b0 msgr2=0x7fba100cc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:01.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.002+0000 7fba17a00640 1 -- 172.21.15.102:0/2301777380 shutdown_connections 2024-03-23T03:51:01.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.002+0000 7fba17a00640 1 -- 172.21.15.102:0/2301777380 wait complete. 2024-03-23T03:51:01.021 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:51:01.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.230+0000 7f6417c00640 1 Processor -- start 2024-03-23T03:51:01.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.230+0000 7f6417c00640 1 -- start start 2024-03-23T03:51:01.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.232+0000 7f6417c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f641006ce90 0x7f641006d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:01.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.232+0000 7f6417c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64100cdba0 con 0x7f64100add30 2024-03-23T03:51:01.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.232+0000 7f6417c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64100cde80 con 0x7f64100ce930 2024-03-23T03:51:01.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.232+0000 7f6417c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64100ae1c0 con 0x7f641006ce90 2024-03-23T03:51:01.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.232+0000 7f640fe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f64100add30 0x7f64100ced10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55578/0 (socket says 172.21.15.102:55578) 2024-03-23T03:51:01.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.232+0000 7f640fe00640 1 -- 172.21.15.102:0/3565216195 learned_addr learned my addr 172.21.15.102:0/3565216195 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:01.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.232+0000 7f6415c00640 1 --2- 172.21.15.102:0/3565216195 >> v2:172.21.15.102:3300/0 conn(0x7f641006ce90 0x7f641006d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:01.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.233+0000 7f640f400640 1 -- 172.21.15.102:0/3565216195 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260901188 0 0) 0x7f64100cdba0 con 0x7f64100add30 2024-03-23T03:51:01.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.233+0000 7f6415c00640 1 -- 172.21.15.102:0/3565216195 >> v1:172.21.15.173:6789/0 conn(0x7f64100ce930 legacy=0x7f64100cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:01.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.233+0000 7f6415c00640 1 -- 172.21.15.102:0/3565216195 >> v1:172.21.15.102:6789/0 conn(0x7f64100add30 legacy=0x7f64100ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:01.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.233+0000 7f6415c00640 1 -- 172.21.15.102:0/3565216195 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64100ce160 con 0x7f641006ce90 2024-03-23T03:51:01.234 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.234+0000 7f6415c00640 1 --2- 172.21.15.102:0/3565216195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f641006ce90 0x7f641006d300 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f64080097a0 tx=0x7f640802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d4c092b00ac5931 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:01.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.234+0000 7f640f400640 1 -- 172.21.15.102:0/3565216195 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6408031830 con 0x7f641006ce90 2024-03-23T03:51:01.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.234+0000 7f640f400640 1 -- 172.21.15.102:0/3565216195 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6408031990 con 0x7f641006ce90 2024-03-23T03:51:01.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.234+0000 7f640f400640 1 -- 172.21.15.102:0/3565216195 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6408031d20 con 0x7f641006ce90 2024-03-23T03:51:01.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.235+0000 7f6417c00640 1 -- 172.21.15.102:0/3565216195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f641006ce90 msgr2=0x7f641006d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:01.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.235+0000 7f6417c00640 1 --2- 172.21.15.102:0/3565216195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f641006ce90 0x7f641006d300 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f64080097a0 tx=0x7f640802b8d0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.235+0000 7f6417c00640 1 -- 172.21.15.102:0/3565216195 shutdown_connections 2024-03-23T03:51:01.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.235+0000 7f6417c00640 1 --2- 172.21.15.102:0/3565216195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f641006ce90 0x7f641006d300 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.235+0000 7f6417c00640 1 -- 172.21.15.102:0/3565216195 >> 172.21.15.102:0/3565216195 conn(0x7f641006a0b0 msgr2=0x7f641006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:01.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.235+0000 7f6417c00640 1 -- 172.21.15.102:0/3565216195 shutdown_connections 2024-03-23T03:51:01.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.235+0000 7f6417c00640 1 -- 172.21.15.102:0/3565216195 wait complete. 2024-03-23T03:51:01.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.237+0000 7f6417c00640 1 Processor -- start 2024-03-23T03:51:01.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.237+0000 7f6417c00640 1 -- start start 2024-03-23T03:51:01.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.238+0000 7f6417c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f641006ce90 0x7f6410154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:01.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.238+0000 7f6417c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64100add30 0x7f64100ae310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:01.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.238+0000 7f6417c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64100ce930 0x7f64100afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:01.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.238+0000 7f6417c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64100aec10 con 0x7f64100add30 2024-03-23T03:51:01.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.238+0000 7f6417c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64100aeef0 con 0x7f64100ce930 2024-03-23T03:51:01.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.238+0000 7f6417c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64100af1d0 con 0x7f641006ce90 2024-03-23T03:51:01.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.238+0000 7f640fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64100add30 0x7f64100ae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:01.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.238+0000 7f640fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64100add30 0x7f64100ae310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58172/0 (socket says 172.21.15.102:58172) 2024-03-23T03:51:01.239 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.238+0000 7f640fe00640 1 -- 172.21.15.102:0/2357839888 learned_addr learned my addr 172.21.15.102:0/2357839888 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:01.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.239+0000 7f6415200640 1 --2- 172.21.15.102:0/2357839888 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f641006ce90 0x7f6410154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:01.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.239+0000 7f640fe00640 1 -- 172.21.15.102:0/2357839888 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f641006ce90 msgr2=0x7f6410154320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:01.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.239+0000 7f6415c00640 1 --2- 172.21.15.102:0/2357839888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64100ce930 0x7f64100afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:01.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.239+0000 7f640fe00640 1 --2- 172.21.15.102:0/2357839888 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f641006ce90 0x7f6410154320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.239+0000 7f640fe00640 1 -- 172.21.15.102:0/2357839888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64100ce930 msgr2=0x7f64100afcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:01.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.239+0000 7f640fe00640 1 --2- 172.21.15.102:0/2357839888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64100ce930 0x7f64100afcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.239+0000 7f640fe00640 1 -- 172.21.15.102:0/2357839888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6408009400 con 0x7f64100add30 2024-03-23T03:51:01.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.239+0000 7f6415200640 1 --2- 172.21.15.102:0/2357839888 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f641006ce90 0x7f6410154320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:01.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.239+0000 7f640fe00640 1 --2- 172.21.15.102:0/2357839888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64100add30 0x7f64100ae310 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f640000c890 tx=0x7f640000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:01.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.239+0000 7f640cc00640 1 -- 172.21.15.102:0/2357839888 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640000a640 con 0x7f64100add30 2024-03-23T03:51:01.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.239+0000 7f640cc00640 1 -- 172.21.15.102:0/2357839888 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640000a7c0 con 0x7f64100add30 2024-03-23T03:51:01.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.239+0000 7f6417c00640 1 -- 172.21.15.102:0/2357839888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64100af4b0 con 0x7f64100add30 2024-03-23T03:51:01.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.239+0000 7f640cc00640 1 -- 172.21.15.102:0/2357839888 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640000ab70 con 0x7f64100add30 2024-03-23T03:51:01.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.239+0000 7f6417c00640 1 -- 172.21.15.102:0/2357839888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64100ae850 con 0x7f64100add30 2024-03-23T03:51:01.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.241+0000 7f6417c00640 1 -- 172.21.15.102:0/2357839888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64100af9d0 con 0x7f64100add30 2024-03-23T03:51:01.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.242+0000 7f640cc00640 1 -- 172.21.15.102:0/2357839888 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f640000acd0 con 0x7f64100add30 2024-03-23T03:51:01.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.242+0000 7f640cc00640 1 --2- 172.21.15.102:0/2357839888 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f63ec06ac00 0x7f63ec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:01.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.243+0000 7f640cc00640 1 -- 172.21.15.102:0/2357839888 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6400088d70 con 0x7f64100add30 2024-03-23T03:51:01.243 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.243+0000 7f6415200640 1 --2- 172.21.15.102:0/2357839888 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f63ec06ac00 0x7f63ec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:01.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.243+0000 7f6415200640 1 --2- 172.21.15.102:0/2357839888 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f63ec06ac00 0x7f63ec06d0c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f63f800a830 tx=0x7f63f8008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:01.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.247+0000 7f640cc00640 1 -- 172.21.15.102:0/2357839888 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f64100af9d0 con 0x7f64100add30 2024-03-23T03:51:01.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.441+0000 7f6417c00640 1 -- 172.21.15.102:0/2357839888 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f64100af9d0 con 0x7f64100add30 2024-03-23T03:51:01.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.442+0000 7f640cc00640 1 -- 172.21.15.102:0/2357839888 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f64100af9d0 con 0x7f64100add30 2024-03-23T03:51:01.443 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:51:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.445+0000 7f6417c00640 1 -- 172.21.15.102:0/2357839888 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f63ec06ac00 msgr2=0x7f63ec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.445+0000 7f6417c00640 1 --2- 172.21.15.102:0/2357839888 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f63ec06ac00 0x7f63ec06d0c0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f63f800a830 tx=0x7f63f8008040 comp rx=0 tx=0).stop 2024-03-23T03:51:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.445+0000 7f6417c00640 1 -- 172.21.15.102:0/2357839888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64100add30 msgr2=0x7f64100ae310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.445+0000 7f6417c00640 1 --2- 172.21.15.102:0/2357839888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64100add30 0x7f64100ae310 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f640000c890 tx=0x7f640000cd50 comp rx=0 tx=0).stop 2024-03-23T03:51:01.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.445+0000 7f6417c00640 1 -- 172.21.15.102:0/2357839888 shutdown_connections 2024-03-23T03:51:01.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.445+0000 7f6417c00640 1 --2- 172.21.15.102:0/2357839888 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f63ec06ac00 0x7f63ec06d0c0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.445+0000 7f6417c00640 1 --2- 172.21.15.102:0/2357839888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64100ce930 0x7f64100afcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.445+0000 7f6417c00640 1 --2- 172.21.15.102:0/2357839888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64100add30 0x7f64100ae310 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.445+0000 7f6417c00640 1 --2- 172.21.15.102:0/2357839888 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f641006ce90 0x7f6410154320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.445+0000 7f6417c00640 1 -- 172.21.15.102:0/2357839888 >> 172.21.15.102:0/2357839888 conn(0x7f641006a0b0 msgr2=0x7f641015e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:01.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.445+0000 7f6417c00640 1 -- 172.21.15.102:0/2357839888 shutdown_connections 2024-03-23T03:51:01.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.446+0000 7f6417c00640 1 -- 172.21.15.102:0/2357839888 wait complete. 2024-03-23T03:51:01.464 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:51:01.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.674+0000 7f457ec00640 1 Processor -- start 2024-03-23T03:51:01.675 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.674+0000 7f457ec00640 1 -- start start 2024-03-23T03:51:01.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.676+0000 7f457ec00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f45780ce930 0x7f45780ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:01.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.676+0000 7f457ec00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f457806d2c0 0x7f45780cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:01.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.676+0000 7f457ec00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45780cf250 con 0x7f45780cb660 2024-03-23T03:51:01.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.676+0000 7f457ec00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f457815bae0 con 0x7f45780ce930 2024-03-23T03:51:01.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.676+0000 7f457ec00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457815ae60 con 0x7f457806d2c0 2024-03-23T03:51:01.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.676+0000 7f4577e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f45780ce930 0x7f45780ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:01.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.676+0000 7f4577e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f45780ce930 0x7f45780ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58184/0 (socket says 172.21.15.102:58184) 2024-03-23T03:51:01.677 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.676+0000 7f4577e00640 1 -- 172.21.15.102:0/1247626984 learned_addr learned my addr 172.21.15.102:0/1247626984 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:01.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.677+0000 7f4577e00640 1 -- 172.21.15.102:0/1247626984 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f457806d2c0 msgr2=0x7f45780cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:01.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.677+0000 7f457cc00640 1 --2- 172.21.15.102:0/1247626984 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f457806d2c0 0x7f45780cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:01.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.677+0000 7f4577e00640 1 --2- 172.21.15.102:0/1247626984 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f457806d2c0 0x7f45780cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.677+0000 7f4577e00640 1 -- 172.21.15.102:0/1247626984 >> v1:172.21.15.173:6789/0 conn(0x7f45780cb660 legacy=0x7f4578070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:01.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.677+0000 7f4577e00640 1 -- 172.21.15.102:0/1247626984 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457815b140 con 0x7f45780ce930 2024-03-23T03:51:01.678 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.677+0000 7f457cc00640 1 --2- 172.21.15.102:0/1247626984 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f457806d2c0 0x7f45780cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:01.679 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.678+0000 7f4577e00640 1 --2- 172.21.15.102:0/1247626984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45780ce930 0x7f45780ced10 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f4568009870 tx=0x7f456802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad3026f033dad036 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:01.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.678+0000 7f4577400640 1 -- 172.21.15.102:0/1247626984 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568031640 con 0x7f45780ce930 2024-03-23T03:51:01.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.678+0000 7f4577400640 1 -- 172.21.15.102:0/1247626984 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45680317a0 con 0x7f45780ce930 2024-03-23T03:51:01.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.678+0000 7f4577400640 1 -- 172.21.15.102:0/1247626984 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568031b30 con 0x7f45780ce930 2024-03-23T03:51:01.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.679+0000 7f457ec00640 1 -- 172.21.15.102:0/1247626984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45780ce930 msgr2=0x7f45780ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:01.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.679+0000 7f457ec00640 1 --2- 172.21.15.102:0/1247626984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45780ce930 0x7f45780ced10 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f4568009870 tx=0x7f456802b8f0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.679+0000 7f457ec00640 1 -- 172.21.15.102:0/1247626984 shutdown_connections 2024-03-23T03:51:01.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.679+0000 7f457ec00640 1 --2- 172.21.15.102:0/1247626984 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f457806d2c0 0x7f45780cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.679+0000 7f457ec00640 1 --2- 172.21.15.102:0/1247626984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45780ce930 0x7f45780ced10 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.679+0000 7f457ec00640 1 -- 172.21.15.102:0/1247626984 >> 172.21.15.102:0/1247626984 conn(0x7f457806a0b0 msgr2=0x7f457806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:01.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.679+0000 7f457ec00640 1 -- 172.21.15.102:0/1247626984 shutdown_connections 2024-03-23T03:51:01.680 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.679+0000 7f457ec00640 1 -- 172.21.15.102:0/1247626984 wait complete. 2024-03-23T03:51:01.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.681+0000 7f457ec00640 1 Processor -- start 2024-03-23T03:51:01.681 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.681+0000 7f457ec00640 1 -- start start 2024-03-23T03:51:01.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.682+0000 7f457ec00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f457806d2c0 0x7f45780b0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:01.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.682+0000 7f457ec00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45780cb660 0x7f457814bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:01.683 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.682+0000 7f457ec00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45780ce930 0x7f457814c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.682+0000 7f457ec00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4578152b10 con 0x7f45780cb660 2024-03-23T03:51:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.682+0000 7f457ec00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4578152df0 con 0x7f45780ce930 2024-03-23T03:51:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.682+0000 7f457ec00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45781530d0 con 0x7f457806d2c0 2024-03-23T03:51:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.682+0000 7f456fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45780cb660 0x7f457814bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.682+0000 7f456fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45780cb660 0x7f457814bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58196/0 (socket says 172.21.15.102:58196) 2024-03-23T03:51:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.682+0000 7f456fe00640 1 -- 172.21.15.102:0/958874978 learned_addr learned my addr 172.21.15.102:0/958874978 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.682+0000 7f4577e00640 1 --2- 172.21.15.102:0/958874978 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f457806d2c0 0x7f45780b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.682+0000 7f457cc00640 1 --2- 172.21.15.102:0/958874978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45780ce930 0x7f457814c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.682+0000 7f456fe00640 1 -- 172.21.15.102:0/958874978 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f457806d2c0 msgr2=0x7f45780b0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.682+0000 7f456fe00640 1 --2- 172.21.15.102:0/958874978 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f457806d2c0 0x7f45780b0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.682+0000 7f456fe00640 1 -- 172.21.15.102:0/958874978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45780ce930 msgr2=0x7f457814c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.682+0000 7f456fe00640 1 --2- 172.21.15.102:0/958874978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45780ce930 0x7f457814c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.682+0000 7f456fe00640 1 -- 172.21.15.102:0/958874978 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45680094d0 con 0x7f45780cb660 2024-03-23T03:51:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.683+0000 7f456fe00640 1 --2- 172.21.15.102:0/958874978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45780cb660 0x7f457814bdb0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f456000c430 tx=0x7f456000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.683+0000 7f4574c00640 1 -- 172.21.15.102:0/958874978 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4560011790 con 0x7f45780cb660 2024-03-23T03:51:01.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.683+0000 7f4574c00640 1 -- 172.21.15.102:0/958874978 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45600118f0 con 0x7f45780cb660 2024-03-23T03:51:01.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.683+0000 7f457ec00640 1 -- 172.21.15.102:0/958874978 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4578155280 con 0x7f45780cb660 2024-03-23T03:51:01.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.683+0000 7f4574c00640 1 -- 172.21.15.102:0/958874978 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4560011ce0 con 0x7f45780cb660 2024-03-23T03:51:01.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.683+0000 7f457ec00640 1 -- 172.21.15.102:0/958874978 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4578153690 con 0x7f45780cb660 2024-03-23T03:51:01.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.684+0000 7f457ec00640 1 -- 172.21.15.102:0/958874978 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4578073390 con 0x7f45780cb660 2024-03-23T03:51:01.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.685+0000 7f4574c00640 1 -- 172.21.15.102:0/958874978 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f456001c460 con 0x7f45780cb660 2024-03-23T03:51:01.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.686+0000 7f4574c00640 1 --2- 172.21.15.102:0/958874978 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f455406ac00 0x7f455406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:01.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.686+0000 7f4574c00640 1 -- 172.21.15.102:0/958874978 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4560089440 con 0x7f45780cb660 2024-03-23T03:51:01.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.686+0000 7f4577e00640 1 --2- 172.21.15.102:0/958874978 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f455406ac00 0x7f455406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:01.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.687+0000 7f4577e00640 1 --2- 172.21.15.102:0/958874978 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f455406ac00 0x7f455406d0c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4568036000 tx=0x7f456803c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:01.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.691+0000 7f4574c00640 1 -- 172.21.15.102:0/958874978 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f45600544d0 con 0x7f45780cb660 2024-03-23T03:51:01.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.903+0000 7f457ec00640 1 -- 172.21.15.102:0/958874978 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4578153850 con 0x7f45780cb660 2024-03-23T03:51:01.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.904+0000 7f4574c00640 1 -- 172.21.15.102:0/958874978 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v93) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4560057af0 con 0x7f45780cb660 2024-03-23T03:51:01.905 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:51:01.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.907+0000 7f457ec00640 1 -- 172.21.15.102:0/958874978 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f455406ac00 msgr2=0x7f455406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:01.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.907+0000 7f457ec00640 1 --2- 172.21.15.102:0/958874978 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f455406ac00 0x7f455406d0c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4568036000 tx=0x7f456803c040 comp rx=0 tx=0).stop 2024-03-23T03:51:01.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.907+0000 7f457ec00640 1 -- 172.21.15.102:0/958874978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45780cb660 msgr2=0x7f457814bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:01.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.907+0000 7f457ec00640 1 --2- 172.21.15.102:0/958874978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45780cb660 0x7f457814bdb0 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f456000c430 tx=0x7f456000c8f0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.907+0000 7f457ec00640 1 -- 172.21.15.102:0/958874978 shutdown_connections 2024-03-23T03:51:01.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.907+0000 7f457ec00640 1 --2- 172.21.15.102:0/958874978 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f455406ac00 0x7f455406d0c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.907+0000 7f457ec00640 1 --2- 172.21.15.102:0/958874978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45780ce930 0x7f457814c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.907+0000 7f457ec00640 1 --2- 172.21.15.102:0/958874978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45780cb660 0x7f457814bdb0 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.907+0000 7f457ec00640 1 --2- 172.21.15.102:0/958874978 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f457806d2c0 0x7f45780b0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:01.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.907+0000 7f457ec00640 1 -- 172.21.15.102:0/958874978 >> 172.21.15.102:0/958874978 conn(0x7f457806a0b0 msgr2=0x7f4578159ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:01.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.907+0000 7f457ec00640 1 -- 172.21.15.102:0/958874978 shutdown_connections 2024-03-23T03:51:01.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:01.907+0000 7f457ec00640 1 -- 172.21.15.102:0/958874978 wait complete. 2024-03-23T03:51:01.926 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-23T03:51:01.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:01.926 DEBUG:teuthology.orchestra.run.smithi102:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-23T03:51:02.031 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:51:02.031 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:02.031 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T03:51:02.085 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:51:02.085 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:02.142 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:51:02.142 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:02.193 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:51:02.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.410+0000 7f5c73200640 1 Processor -- start 2024-03-23T03:51:02.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.410+0000 7f5c73200640 1 -- start start 2024-03-23T03:51:02.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.411+0000 7f5c73200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5c74070670 0x7f5c7413eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:02.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.411+0000 7f5c73200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f5c7415cae0 0x7f5c7415cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:02.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.411+0000 7f5c73200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c740b2f50 con 0x7f5c74070290 2024-03-23T03:51:02.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.411+0000 7f5c73200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c740b2620 con 0x7f5c74070670 2024-03-23T03:51:02.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.411+0000 7f5c73200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c740b2900 con 0x7f5c7415cae0 2024-03-23T03:51:02.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.412+0000 7f5c69400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5c74070670 0x7f5c7413eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:02.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.412+0000 7f5c69400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5c74070670 0x7f5c7413eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58210/0 (socket says 172.21.15.102:58210) 2024-03-23T03:51:02.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.412+0000 7f5c69400640 1 -- 172.21.15.102:0/150987903 learned_addr learned my addr 172.21.15.102:0/150987903 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:02.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.412+0000 7f5c72800640 1 --2- 172.21.15.102:0/150987903 >> v2:172.21.15.173:3300/0 conn(0x7f5c7415cae0 0x7f5c7415cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:02.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.412+0000 7f5c69400640 1 -- 172.21.15.102:0/150987903 >> v2:172.21.15.173:3300/0 conn(0x7f5c7415cae0 msgr2=0x7f5c7415cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:02.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.412+0000 7f5c69400640 1 --2- 172.21.15.102:0/150987903 >> v2:172.21.15.173:3300/0 conn(0x7f5c7415cae0 0x7f5c7415cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:02.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.412+0000 7f5c69400640 1 -- 172.21.15.102:0/150987903 >> v1:172.21.15.173:6789/0 conn(0x7f5c74070290 legacy=0x7f5c740b33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:02.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.412+0000 7f5c69400640 1 -- 172.21.15.102:0/150987903 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c740b2be0 con 0x7f5c74070670 2024-03-23T03:51:02.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.413+0000 7f5c69400640 1 --2- 172.21.15.102:0/150987903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c74070670 0x7f5c7413eb40 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f5c740b1500 tx=0x7f5c5c02df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e3e90ca822f43c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:02.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.413+0000 7f5c71400640 1 -- 172.21.15.102:0/150987903 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c02ecb0 con 0x7f5c74070670 2024-03-23T03:51:02.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.414+0000 7f5c71400640 1 -- 172.21.15.102:0/150987903 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c5c02ee10 con 0x7f5c74070670 2024-03-23T03:51:02.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.414+0000 7f5c71400640 1 -- 172.21.15.102:0/150987903 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c036660 con 0x7f5c74070670 2024-03-23T03:51:02.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.414+0000 7f5c73200640 1 -- 172.21.15.102:0/150987903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c74070670 msgr2=0x7f5c7413eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:02.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.414+0000 7f5c73200640 1 --2- 172.21.15.102:0/150987903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c74070670 0x7f5c7413eb40 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f5c740b1500 tx=0x7f5c5c02df00 comp rx=0 tx=0).stop 2024-03-23T03:51:02.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.414+0000 7f5c73200640 1 -- 172.21.15.102:0/150987903 shutdown_connections 2024-03-23T03:51:02.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.414+0000 7f5c73200640 1 --2- 172.21.15.102:0/150987903 >> v2:172.21.15.173:3300/0 conn(0x7f5c7415cae0 0x7f5c7415cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:02.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.414+0000 7f5c73200640 1 --2- 172.21.15.102:0/150987903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c74070670 0x7f5c7413eb40 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:02.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.414+0000 7f5c73200640 1 -- 172.21.15.102:0/150987903 >> 172.21.15.102:0/150987903 conn(0x7f5c74065df0 msgr2=0x7f5c74069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:02.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.414+0000 7f5c73200640 1 -- 172.21.15.102:0/150987903 shutdown_connections 2024-03-23T03:51:02.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.414+0000 7f5c73200640 1 -- 172.21.15.102:0/150987903 wait complete. 2024-03-23T03:51:02.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.416+0000 7f5c73200640 1 Processor -- start 2024-03-23T03:51:02.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.416+0000 7f5c73200640 1 -- start start 2024-03-23T03:51:02.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.417+0000 7f5c73200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c74070290 0x7f5c7413c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:02.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.417+0000 7f5c73200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5c74070670 0x7f5c741ede90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:02.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.417+0000 7f5c73200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c7415cae0 0x7f5c7406d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:02.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.417+0000 7f5c73200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c740ad3a0 con 0x7f5c7415cae0 2024-03-23T03:51:02.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.417+0000 7f5c73200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c74154c30 con 0x7f5c74070290 2024-03-23T03:51:02.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.417+0000 7f5c73200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c74154f10 con 0x7f5c74070670 2024-03-23T03:51:02.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.417+0000 7f5c72800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c7415cae0 0x7f5c7406d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:02.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.417+0000 7f5c72800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c7415cae0 0x7f5c7406d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58212/0 (socket says 172.21.15.102:58212) 2024-03-23T03:51:02.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.417+0000 7f5c72800640 1 -- 172.21.15.102:0/3073477188 learned_addr learned my addr 172.21.15.102:0/3073477188 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:02.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.418+0000 7f5c71e00640 1 --2- 172.21.15.102:0/3073477188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c74070290 0x7f5c7413c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:02.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.418+0000 7f5c72800640 1 -- 172.21.15.102:0/3073477188 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5c74070670 msgr2=0x7f5c741ede90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:02.419 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.418+0000 7f5c72800640 1 --2- 172.21.15.102:0/3073477188 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5c74070670 0x7f5c741ede90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.418+0000 7f5c72800640 1 -- 172.21.15.102:0/3073477188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c74070290 msgr2=0x7f5c7413c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.418+0000 7f5c72800640 1 --2- 172.21.15.102:0/3073477188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c74070290 0x7f5c7413c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.418+0000 7f5c72800640 1 -- 172.21.15.102:0/3073477188 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c00a670 con 0x7f5c7415cae0 2024-03-23T03:51:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.418+0000 7f5c71e00640 1 --2- 172.21.15.102:0/3073477188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c74070290 0x7f5c7413c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.418+0000 7f5c72800640 1 --2- 172.21.15.102:0/3073477188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c7415cae0 0x7f5c7406d660 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00cd70 tx=0x7f5c6c00a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.418+0000 7f5c6aa00640 1 -- 172.21.15.102:0/3073477188 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c003b90 con 0x7f5c7415cae0 2024-03-23T03:51:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.418+0000 7f5c6aa00640 1 -- 172.21.15.102:0/3073477188 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c003cf0 con 0x7f5c7415cae0 2024-03-23T03:51:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.418+0000 7f5c6aa00640 1 -- 172.21.15.102:0/3073477188 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c004240 con 0x7f5c7415cae0 2024-03-23T03:51:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.418+0000 7f5c73200640 1 -- 172.21.15.102:0/3073477188 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c741551f0 con 0x7f5c7415cae0 2024-03-23T03:51:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.418+0000 7f5c73200640 1 -- 172.21.15.102:0/3073477188 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c741557b0 con 0x7f5c7415cae0 2024-03-23T03:51:02.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.420+0000 7f5c73200640 1 -- 172.21.15.102:0/3073477188 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c741559d0 con 0x7f5c7415cae0 2024-03-23T03:51:02.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.421+0000 7f5c6aa00640 1 -- 172.21.15.102:0/3073477188 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5c6c0043a0 con 0x7f5c7415cae0 2024-03-23T03:51:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.421+0000 7f5c6aa00640 1 --2- 172.21.15.102:0/3073477188 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5c5006ac30 0x7f5c5006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.422+0000 7f5c6aa00640 1 -- 172.21.15.102:0/3073477188 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5c6c012070 con 0x7f5c7415cae0 2024-03-23T03:51:02.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.422+0000 7f5c71e00640 1 --2- 172.21.15.102:0/3073477188 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5c5006ac30 0x7f5c5006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:02.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.422+0000 7f5c71e00640 1 --2- 172.21.15.102:0/3073477188 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5c5006ac30 0x7f5c5006d0f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f5c7415be10 tx=0x7f5c64008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:02.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.426+0000 7f5c6aa00640 1 -- 172.21.15.102:0/3073477188 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c6c054f50 con 0x7f5c7415cae0 2024-03-23T03:51:02.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.617+0000 7f5c73200640 1 -- 172.21.15.102:0/3073477188 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c7404e030 con 0x7f5c7415cae0 2024-03-23T03:51:02.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.619+0000 7f5c6aa00640 1 -- 172.21.15.102:0/3073477188 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16297 (secure 0 0 0) 0x7f5c6c058570 con 0x7f5c7415cae0 2024-03-23T03:51:02.620 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:02.620 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":93,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:50.300471+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:50:47.750670+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-23T03:50:48.305583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:51:02.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.622+0000 7f5c73200640 1 -- 172.21.15.102:0/3073477188 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5c5006ac30 msgr2=0x7f5c5006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:02.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.622+0000 7f5c73200640 1 --2- 172.21.15.102:0/3073477188 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5c5006ac30 0x7f5c5006d0f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f5c7415be10 tx=0x7f5c64008040 comp rx=0 tx=0).stop 2024-03-23T03:51:02.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.622+0000 7f5c73200640 1 -- 172.21.15.102:0/3073477188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c7415cae0 msgr2=0x7f5c7406d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:02.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.622+0000 7f5c73200640 1 --2- 172.21.15.102:0/3073477188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c7415cae0 0x7f5c7406d660 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c00cd70 tx=0x7f5c6c00a660 comp rx=0 tx=0).stop 2024-03-23T03:51:02.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.622+0000 7f5c73200640 1 -- 172.21.15.102:0/3073477188 shutdown_connections 2024-03-23T03:51:02.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.622+0000 7f5c73200640 1 --2- 172.21.15.102:0/3073477188 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5c5006ac30 0x7f5c5006d0f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:02.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.622+0000 7f5c73200640 1 --2- 172.21.15.102:0/3073477188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c7415cae0 0x7f5c7406d660 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:02.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.622+0000 7f5c73200640 1 --2- 172.21.15.102:0/3073477188 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5c74070670 0x7f5c741ede90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:02.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.622+0000 7f5c73200640 1 --2- 172.21.15.102:0/3073477188 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c74070290 0x7f5c7413c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:02.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.622+0000 7f5c73200640 1 -- 172.21.15.102:0/3073477188 >> 172.21.15.102:0/3073477188 conn(0x7f5c74065df0 msgr2=0x7f5c7413cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:02.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.622+0000 7f5c73200640 1 -- 172.21.15.102:0/3073477188 shutdown_connections 2024-03-23T03:51:02.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.622+0000 7f5c73200640 1 -- 172.21.15.102:0/3073477188 wait complete. 2024-03-23T03:51:02.641 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:50:47.750670+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:50:48.305583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:51:02.642 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:51:02.851 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.851+0000 7fb31c600640 1 Processor -- start 2024-03-23T03:51:02.852 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.851+0000 7fb31c600640 1 -- start start 2024-03-23T03:51:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.852+0000 7fb31c600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb314159830 0x7fb3140cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.853+0000 7fb31c600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb3140ca7b0 0x7fb3140cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.853+0000 7fb31c600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3140ce930 0x7fb3140cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.853+0000 7fb31c600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb31406de50 con 0x7fb3140ca7b0 2024-03-23T03:51:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.853+0000 7fb31c600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3140cf300 con 0x7fb314159830 2024-03-23T03:51:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.853+0000 7fb31c600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb314159c10 con 0x7fb3140ce930 2024-03-23T03:51:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.853+0000 7fb319200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb3140ca7b0 0x7fb3140cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:02.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.853+0000 7fb319200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb3140ca7b0 0x7fb3140cdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34152/0 (socket says 172.21.15.102:34152) 2024-03-23T03:51:02.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.853+0000 7fb319200640 1 -- 172.21.15.102:0/3411088137 learned_addr learned my addr 172.21.15.102:0/3411088137 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:02.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.853+0000 7fb31a600640 1 --2- 172.21.15.102:0/3411088137 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3140ce930 0x7fb3140cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:02.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.853+0000 7fb319c00640 1 --2- 172.21.15.102:0/3411088137 >> v2:172.21.15.173:3300/0 conn(0x7fb314159830 0x7fb3140cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:02.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.853+0000 7fb319200640 1 -- 172.21.15.102:0/3411088137 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3140ce930 msgr2=0x7fb3140cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:02.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.854+0000 7fb319200640 1 --2- 172.21.15.102:0/3411088137 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3140ce930 0x7fb3140cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:02.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.854+0000 7fb319200640 1 -- 172.21.15.102:0/3411088137 >> v2:172.21.15.173:3300/0 conn(0x7fb314159830 msgr2=0x7fb3140cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:02.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.854+0000 7fb319200640 1 --2- 172.21.15.102:0/3411088137 >> v2:172.21.15.173:3300/0 conn(0x7fb314159830 0x7fb3140cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:02.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.854+0000 7fb319200640 1 -- 172.21.15.102:0/3411088137 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3140cd840 con 0x7fb3140ca7b0 2024-03-23T03:51:02.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.854+0000 7fb319c00640 1 --2- 172.21.15.102:0/3411088137 >> v2:172.21.15.173:3300/0 conn(0x7fb314159830 0x7fb3140cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:02.855 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.854+0000 7fb31a600640 1 --2- 172.21.15.102:0/3411088137 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3140ce930 0x7fb3140cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:02.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.855+0000 7fb319200640 1 --2- 172.21.15.102:0/3411088137 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3140ca7b0 0x7fb3140cdcd0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fb31406e4e0 tx=0x7fb30002e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc7aaedbc2f42fd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:02.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.855+0000 7fb313e00640 1 -- 172.21.15.102:0/3411088137 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30002ed10 con 0x7fb3140ca7b0 2024-03-23T03:51:02.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.855+0000 7fb313e00640 1 -- 172.21.15.102:0/3411088137 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30002ee70 con 0x7fb3140ca7b0 2024-03-23T03:51:02.856 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.855+0000 7fb313e00640 1 -- 172.21.15.102:0/3411088137 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3000365f0 con 0x7fb3140ca7b0 2024-03-23T03:51:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.856+0000 7fb31c600640 1 -- 172.21.15.102:0/3411088137 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3140ca7b0 msgr2=0x7fb3140cdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.856+0000 7fb31c600640 1 --2- 172.21.15.102:0/3411088137 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3140ca7b0 0x7fb3140cdcd0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fb31406e4e0 tx=0x7fb30002e170 comp rx=0 tx=0).stop 2024-03-23T03:51:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.856+0000 7fb31c600640 1 -- 172.21.15.102:0/3411088137 shutdown_connections 2024-03-23T03:51:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.856+0000 7fb31c600640 1 --2- 172.21.15.102:0/3411088137 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3140ce930 0x7fb3140cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.856+0000 7fb31c600640 1 --2- 172.21.15.102:0/3411088137 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3140ca7b0 0x7fb3140cdcd0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.856+0000 7fb31c600640 1 --2- 172.21.15.102:0/3411088137 >> v2:172.21.15.173:3300/0 conn(0x7fb314159830 0x7fb3140cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.856+0000 7fb31c600640 1 -- 172.21.15.102:0/3411088137 >> 172.21.15.102:0/3411088137 conn(0x7fb31406a0b0 msgr2=0x7fb31406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.856+0000 7fb31c600640 1 -- 172.21.15.102:0/3411088137 shutdown_connections 2024-03-23T03:51:02.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.856+0000 7fb31c600640 1 -- 172.21.15.102:0/3411088137 wait complete. 2024-03-23T03:51:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.858+0000 7fb31c600640 1 Processor -- start 2024-03-23T03:51:02.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.858+0000 7fb31c600640 1 -- start start 2024-03-23T03:51:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.859+0000 7fb31c600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3140ca7b0 0x7fb31415a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.859+0000 7fb31c600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3140ce930 0x7fb3140c9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.859+0000 7fb31c600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb314159830 0x7fb3140c3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.859+0000 7fb31c600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3140ca3d0 con 0x7fb3140ca7b0 2024-03-23T03:51:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.859+0000 7fb31c600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3141480d0 con 0x7fb314159830 2024-03-23T03:51:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.859+0000 7fb31c600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3141483b0 con 0x7fb3140ce930 2024-03-23T03:51:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.859+0000 7fb319c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3140ca7b0 0x7fb31415a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.859+0000 7fb319c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3140ca7b0 0x7fb31415a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34164/0 (socket says 172.21.15.102:34164) 2024-03-23T03:51:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.859+0000 7fb319c00640 1 -- 172.21.15.102:0/3134168231 learned_addr learned my addr 172.21.15.102:0/3134168231 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:02.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.860+0000 7fb31a600640 1 --2- 172.21.15.102:0/3134168231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb314159830 0x7fb3140c3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.860+0000 7fb319200640 1 --2- 172.21.15.102:0/3134168231 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3140ce930 0x7fb3140c9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.860+0000 7fb319c00640 1 -- 172.21.15.102:0/3134168231 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3140ce930 msgr2=0x7fb3140c9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.860+0000 7fb319c00640 1 --2- 172.21.15.102:0/3134168231 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3140ce930 0x7fb3140c9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.860+0000 7fb319c00640 1 -- 172.21.15.102:0/3134168231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb314159830 msgr2=0x7fb3140c3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.860+0000 7fb319c00640 1 --2- 172.21.15.102:0/3134168231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb314159830 0x7fb3140c3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.860+0000 7fb319c00640 1 -- 172.21.15.102:0/3134168231 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb300009b30 con 0x7fb3140ca7b0 2024-03-23T03:51:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.860+0000 7fb319c00640 1 --2- 172.21.15.102:0/3134168231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3140ca7b0 0x7fb31415a5a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00c7f0 tx=0x7fb30c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.860+0000 7fb311600640 1 -- 172.21.15.102:0/3134168231 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c007820 con 0x7fb3140ca7b0 2024-03-23T03:51:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.860+0000 7fb311600640 1 -- 172.21.15.102:0/3134168231 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30c007980 con 0x7fb3140ca7b0 2024-03-23T03:51:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.860+0000 7fb311600640 1 -- 172.21.15.102:0/3134168231 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30c007cf0 con 0x7fb3140ca7b0 2024-03-23T03:51:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.860+0000 7fb31c600640 1 -- 172.21.15.102:0/3134168231 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb314148690 con 0x7fb3140ca7b0 2024-03-23T03:51:02.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.861+0000 7fb31c600640 1 -- 172.21.15.102:0/3134168231 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb314148c50 con 0x7fb3140ca7b0 2024-03-23T03:51:02.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.862+0000 7fb31c600640 1 -- 172.21.15.102:0/3134168231 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb314073390 con 0x7fb3140ca7b0 2024-03-23T03:51:02.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.863+0000 7fb311600640 1 -- 172.21.15.102:0/3134168231 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb30c0113f0 con 0x7fb3140ca7b0 2024-03-23T03:51:02.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.864+0000 7fb311600640 1 --2- 172.21.15.102:0/3134168231 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2f406ac00 0x7fb2f406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.864+0000 7fb311600640 1 -- 172.21.15.102:0/3134168231 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb30c089790 con 0x7fb3140ca7b0 2024-03-23T03:51:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.864+0000 7fb319200640 1 --2- 172.21.15.102:0/3134168231 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2f406ac00 0x7fb2f406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:02.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.865+0000 7fb319200640 1 --2- 172.21.15.102:0/3134168231 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2f406ac00 0x7fb2f406d0c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb300037000 tx=0x7fb30000d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:02.869 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:02.869+0000 7fb311600640 1 -- 172.21.15.102:0/3134168231 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb30c054790 con 0x7fb3140ca7b0 2024-03-23T03:51:03.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.064+0000 7fb31c600640 1 -- 172.21.15.102:0/3134168231 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb3140adf00 con 0x7fb3140ca7b0 2024-03-23T03:51:03.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.065+0000 7fb311600640 1 -- 172.21.15.102:0/3134168231 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb30c057db0 con 0x7fb3140ca7b0 2024-03-23T03:51:03.066 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:51:03.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.068+0000 7fb31c600640 1 -- 172.21.15.102:0/3134168231 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2f406ac00 msgr2=0x7fb2f406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:03.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.068+0000 7fb31c600640 1 --2- 172.21.15.102:0/3134168231 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2f406ac00 0x7fb2f406d0c0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb300037000 tx=0x7fb30000d040 comp rx=0 tx=0).stop 2024-03-23T03:51:03.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.068+0000 7fb31c600640 1 -- 172.21.15.102:0/3134168231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3140ca7b0 msgr2=0x7fb31415a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:03.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.068+0000 7fb31c600640 1 --2- 172.21.15.102:0/3134168231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3140ca7b0 0x7fb31415a5a0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fb30c00c7f0 tx=0x7fb30c00ccb0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.068+0000 7fb31c600640 1 -- 172.21.15.102:0/3134168231 shutdown_connections 2024-03-23T03:51:03.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.068+0000 7fb31c600640 1 --2- 172.21.15.102:0/3134168231 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2f406ac00 0x7fb2f406d0c0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.068+0000 7fb31c600640 1 --2- 172.21.15.102:0/3134168231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb314159830 0x7fb3140c3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.068+0000 7fb31c600640 1 --2- 172.21.15.102:0/3134168231 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3140ce930 0x7fb3140c9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.068+0000 7fb31c600640 1 --2- 172.21.15.102:0/3134168231 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3140ca7b0 0x7fb31415a5a0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.068+0000 7fb31c600640 1 -- 172.21.15.102:0/3134168231 >> 172.21.15.102:0/3134168231 conn(0x7fb31406a0b0 msgr2=0x7fb314154c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:03.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.069+0000 7fb31c600640 1 -- 172.21.15.102:0/3134168231 shutdown_connections 2024-03-23T03:51:03.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.069+0000 7fb31c600640 1 -- 172.21.15.102:0/3134168231 wait complete. 2024-03-23T03:51:03.088 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:51:03.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.289+0000 7f9417800640 1 Processor -- start 2024-03-23T03:51:03.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.289+0000 7f9417800640 1 -- start start 2024-03-23T03:51:03.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.291+0000 7f9417800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f9410160d20 0x7f9410164920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:03.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.291+0000 7f9417800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f94100ca7b0 0x7f94100ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:03.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.291+0000 7f9417800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9410165830 con 0x7f94100cee70 2024-03-23T03:51:03.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.291+0000 7f9417800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94100cf2e0 con 0x7f94100ca7b0 2024-03-23T03:51:03.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.291+0000 7f9417800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9410164f90 con 0x7f9410160d20 2024-03-23T03:51:03.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.291+0000 7f940fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f94100ca7b0 0x7f94100ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:03.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.291+0000 7f940fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f94100ca7b0 0x7f94100ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34170/0 (socket says 172.21.15.102:34170) 2024-03-23T03:51:03.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.291+0000 7f940fe00640 1 -- 172.21.15.102:0/1749851195 learned_addr learned my addr 172.21.15.102:0/1749851195 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:03.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.291+0000 7f9414e00640 1 --2- 172.21.15.102:0/1749851195 >> v2:172.21.15.173:3300/0 conn(0x7f9410160d20 0x7f9410164920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:03.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.291+0000 7f940fe00640 1 -- 172.21.15.102:0/1749851195 >> v2:172.21.15.173:3300/0 conn(0x7f9410160d20 msgr2=0x7f9410164920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:03.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.291+0000 7f940fe00640 1 --2- 172.21.15.102:0/1749851195 >> v2:172.21.15.173:3300/0 conn(0x7f9410160d20 0x7f9410164920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.291+0000 7f940fe00640 1 -- 172.21.15.102:0/1749851195 >> v1:172.21.15.102:6789/0 conn(0x7f94100cee70 legacy=0x7f9410070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:03.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.291+0000 7f940fe00640 1 -- 172.21.15.102:0/1749851195 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9410165270 con 0x7f94100ca7b0 2024-03-23T03:51:03.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.293+0000 7f940fe00640 1 --2- 172.21.15.102:0/1749851195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94100ca7b0 0x7f94100ce930 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f94100cf4c0 tx=0x7f93fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d49aa090979a92 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:03.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.293+0000 7f940f400640 1 -- 172.21.15.102:0/1749851195 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc031870 con 0x7f94100ca7b0 2024-03-23T03:51:03.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.293+0000 7f940f400640 1 -- 172.21.15.102:0/1749851195 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc0319d0 con 0x7f94100ca7b0 2024-03-23T03:51:03.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.293+0000 7f940f400640 1 -- 172.21.15.102:0/1749851195 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc031d60 con 0x7f94100ca7b0 2024-03-23T03:51:03.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.293+0000 7f9417800640 1 -- 172.21.15.102:0/1749851195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94100ca7b0 msgr2=0x7f94100ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:03.294 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.293+0000 7f9417800640 1 --2- 172.21.15.102:0/1749851195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94100ca7b0 0x7f94100ce930 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f94100cf4c0 tx=0x7f93fc02b910 comp rx=0 tx=0).stop 2024-03-23T03:51:03.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.294+0000 7f9417800640 1 -- 172.21.15.102:0/1749851195 shutdown_connections 2024-03-23T03:51:03.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.294+0000 7f9417800640 1 --2- 172.21.15.102:0/1749851195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f94100ca7b0 0x7f94100ce930 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.294+0000 7f9417800640 1 --2- 172.21.15.102:0/1749851195 >> v2:172.21.15.173:3300/0 conn(0x7f9410160d20 0x7f9410164920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.294+0000 7f9417800640 1 -- 172.21.15.102:0/1749851195 >> 172.21.15.102:0/1749851195 conn(0x7f941006a0b0 msgr2=0x7f941006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:03.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.294+0000 7f9417800640 1 -- 172.21.15.102:0/1749851195 shutdown_connections 2024-03-23T03:51:03.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.294+0000 7f9417800640 1 -- 172.21.15.102:0/1749851195 wait complete. 2024-03-23T03:51:03.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.295+0000 7f9417800640 1 Processor -- start 2024-03-23T03:51:03.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.296+0000 7f9417800640 1 -- start start 2024-03-23T03:51:03.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.296+0000 7f9417800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94100ca7b0 0x7f941014ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:03.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.297+0000 7f9417800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94100cee70 0x7f941014c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:03.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.297+0000 7f9417800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9410160d20 0x7f9410147a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:03.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.297+0000 7f9417800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f941014c8a0 con 0x7f9410160d20 2024-03-23T03:51:03.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.297+0000 7f9417800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9410147f40 con 0x7f94100ca7b0 2024-03-23T03:51:03.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.297+0000 7f9417800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94100cb570 con 0x7f94100cee70 2024-03-23T03:51:03.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.297+0000 7f9415800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9410160d20 0x7f9410147a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:03.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.297+0000 7f9415800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9410160d20 0x7f9410147a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34182/0 (socket says 172.21.15.102:34182) 2024-03-23T03:51:03.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.297+0000 7f9415800640 1 -- 172.21.15.102:0/1555901195 learned_addr learned my addr 172.21.15.102:0/1555901195 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:03.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.297+0000 7f9414e00640 1 --2- 172.21.15.102:0/1555901195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94100ca7b0 0x7f941014ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:03.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.297+0000 7f940fe00640 1 --2- 172.21.15.102:0/1555901195 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94100cee70 0x7f941014c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:03.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.297+0000 7f9415800640 1 -- 172.21.15.102:0/1555901195 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94100cee70 msgr2=0x7f941014c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:03.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.297+0000 7f9415800640 1 --2- 172.21.15.102:0/1555901195 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94100cee70 0x7f941014c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.297+0000 7f9415800640 1 -- 172.21.15.102:0/1555901195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94100ca7b0 msgr2=0x7f941014ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:03.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.297+0000 7f9415800640 1 --2- 172.21.15.102:0/1555901195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94100ca7b0 0x7f941014ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.297+0000 7f9415800640 1 -- 172.21.15.102:0/1555901195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93fc0094d0 con 0x7f9410160d20 2024-03-23T03:51:03.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.297+0000 7f9414e00640 1 --2- 172.21.15.102:0/1555901195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94100ca7b0 0x7f941014ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:03.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.298+0000 7f9415800640 1 --2- 172.21.15.102:0/1555901195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9410160d20 0x7f9410147a00 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f940800c6b0 tx=0x7f940800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:03.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.298+0000 7f940cc00640 1 -- 172.21.15.102:0/1555901195 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408003b70 con 0x7f9410160d20 2024-03-23T03:51:03.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.298+0000 7f940cc00640 1 -- 172.21.15.102:0/1555901195 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9408003cf0 con 0x7f9410160d20 2024-03-23T03:51:03.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.298+0000 7f940cc00640 1 -- 172.21.15.102:0/1555901195 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9408005310 con 0x7f9410160d20 2024-03-23T03:51:03.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.298+0000 7f9417800640 1 -- 172.21.15.102:0/1555901195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94100cb850 con 0x7f9410160d20 2024-03-23T03:51:03.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.298+0000 7f9417800640 1 -- 172.21.15.102:0/1555901195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94100cbe10 con 0x7f9410160d20 2024-03-23T03:51:03.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.299+0000 7f9417800640 1 -- 172.21.15.102:0/1555901195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9410073390 con 0x7f9410160d20 2024-03-23T03:51:03.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.300+0000 7f940cc00640 1 -- 172.21.15.102:0/1555901195 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9408005470 con 0x7f9410160d20 2024-03-23T03:51:03.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.301+0000 7f940cc00640 1 --2- 172.21.15.102:0/1555901195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93f006ac00 0x7f93f006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:03.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.301+0000 7f940cc00640 1 -- 172.21.15.102:0/1555901195 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f94080892c0 con 0x7f9410160d20 2024-03-23T03:51:03.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.301+0000 7f9414e00640 1 --2- 172.21.15.102:0/1555901195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93f006ac00 0x7f93f006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:03.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.302+0000 7f9414e00640 1 --2- 172.21.15.102:0/1555901195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93f006ac00 0x7f93f006d0c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9404006fd0 tx=0x7f9404008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:03.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.305+0000 7f940cc00640 1 -- 172.21.15.102:0/1555901195 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f94080542c0 con 0x7f9410160d20 2024-03-23T03:51:03.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.497+0000 7f9417800640 1 -- 172.21.15.102:0/1555901195 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f94100b37c0 con 0x7f9410160d20 2024-03-23T03:51:03.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.498+0000 7f940cc00640 1 -- 172.21.15.102:0/1555901195 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9408003e60 con 0x7f9410160d20 2024-03-23T03:51:03.499 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:51:03.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.501+0000 7f9417800640 1 -- 172.21.15.102:0/1555901195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93f006ac00 msgr2=0x7f93f006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:03.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.501+0000 7f9417800640 1 --2- 172.21.15.102:0/1555901195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93f006ac00 0x7f93f006d0c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9404006fd0 tx=0x7f9404008040 comp rx=0 tx=0).stop 2024-03-23T03:51:03.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.501+0000 7f9417800640 1 -- 172.21.15.102:0/1555901195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9410160d20 msgr2=0x7f9410147a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:03.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.501+0000 7f9417800640 1 --2- 172.21.15.102:0/1555901195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9410160d20 0x7f9410147a00 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f940800c6b0 tx=0x7f940800cb70 comp rx=0 tx=0).stop 2024-03-23T03:51:03.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.501+0000 7f9417800640 1 -- 172.21.15.102:0/1555901195 shutdown_connections 2024-03-23T03:51:03.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.501+0000 7f9417800640 1 --2- 172.21.15.102:0/1555901195 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f93f006ac00 0x7f93f006d0c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.501+0000 7f9417800640 1 --2- 172.21.15.102:0/1555901195 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9410160d20 0x7f9410147a00 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.501+0000 7f9417800640 1 --2- 172.21.15.102:0/1555901195 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f94100cee70 0x7f941014c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.501+0000 7f9417800640 1 --2- 172.21.15.102:0/1555901195 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f94100ca7b0 0x7f941014ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.501+0000 7f9417800640 1 -- 172.21.15.102:0/1555901195 >> 172.21.15.102:0/1555901195 conn(0x7f941006a0b0 msgr2=0x7f9410153940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:03.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.501+0000 7f9417800640 1 -- 172.21.15.102:0/1555901195 shutdown_connections 2024-03-23T03:51:03.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.501+0000 7f9417800640 1 -- 172.21.15.102:0/1555901195 wait complete. 2024-03-23T03:51:03.521 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:51:03.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.725+0000 7fb15d400640 1 Processor -- start 2024-03-23T03:51:03.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.725+0000 7fb15d400640 1 -- start start 2024-03-23T03:51:03.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.727+0000 7fb15d400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb1580ce930 0x7fb1580add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:03.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.727+0000 7fb15d400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb1580cd3e0 0x7fb1580ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:03.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.727+0000 7fb15d400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1580cf250 con 0x7fb15806ce90 2024-03-23T03:51:03.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.727+0000 7fb15d400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb15806c760 con 0x7fb1580ce930 2024-03-23T03:51:03.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.727+0000 7fb15d400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1580b42d0 con 0x7fb1580cd3e0 2024-03-23T03:51:03.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.727+0000 7fb156800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb1580ce930 0x7fb1580add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:03.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.727+0000 7fb155e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb1580cd3e0 0x7fb1580ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:03.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.728+0000 7fb156800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb1580ce930 0x7fb1580add30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:39064/0 (socket says 172.21.15.102:39064) 2024-03-23T03:51:03.728 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.728+0000 7fb156800640 1 -- 172.21.15.102:0/3710458058 learned_addr learned my addr 172.21.15.102:0/3710458058 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:03.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.728+0000 7fb155e00640 1 -- 172.21.15.102:0/3710458058 >> v2:172.21.15.173:3300/0 conn(0x7fb1580ce930 msgr2=0x7fb1580add30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:03.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.728+0000 7fb155e00640 1 --2- 172.21.15.102:0/3710458058 >> v2:172.21.15.173:3300/0 conn(0x7fb1580ce930 0x7fb1580add30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.728+0000 7fb155e00640 1 -- 172.21.15.102:0/3710458058 >> v1:172.21.15.173:6789/0 conn(0x7fb15806ce90 legacy=0x7fb15815e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:03.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.728+0000 7fb155e00640 1 -- 172.21.15.102:0/3710458058 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1580cdba0 con 0x7fb1580cd3e0 2024-03-23T03:51:03.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.728+0000 7fb156800640 1 --2- 172.21.15.102:0/3710458058 >> v2:172.21.15.173:3300/0 conn(0x7fb1580ce930 0x7fb1580add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:51:03.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.730+0000 7fb155e00640 1 --2- 172.21.15.102:0/3710458058 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb1580cd3e0 0x7fb1580ced10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb1440097a0 tx=0x7fb14402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97964c75ba9838a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:03.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.731+0000 7fb155400640 1 -- 172.21.15.102:0/3710458058 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb144031830 con 0x7fb1580cd3e0 2024-03-23T03:51:03.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.731+0000 7fb155400640 1 -- 172.21.15.102:0/3710458058 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb144031990 con 0x7fb1580cd3e0 2024-03-23T03:51:03.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.731+0000 7fb155400640 1 -- 172.21.15.102:0/3710458058 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb144031d20 con 0x7fb1580cd3e0 2024-03-23T03:51:03.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.732+0000 7fb15d400640 1 -- 172.21.15.102:0/3710458058 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb1580cd3e0 msgr2=0x7fb1580ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:03.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.732+0000 7fb15d400640 1 --2- 172.21.15.102:0/3710458058 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb1580cd3e0 0x7fb1580ced10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb1440097a0 tx=0x7fb14402b8d0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.732+0000 7fb15d400640 1 -- 172.21.15.102:0/3710458058 shutdown_connections 2024-03-23T03:51:03.732 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.732+0000 7fb15d400640 1 --2- 172.21.15.102:0/3710458058 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb1580cd3e0 0x7fb1580ced10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.732+0000 7fb15d400640 1 --2- 172.21.15.102:0/3710458058 >> v2:172.21.15.173:3300/0 conn(0x7fb1580ce930 0x7fb1580add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.732+0000 7fb15d400640 1 -- 172.21.15.102:0/3710458058 >> 172.21.15.102:0/3710458058 conn(0x7fb15806a0b0 msgr2=0x7fb15806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:03.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.732+0000 7fb15d400640 1 -- 172.21.15.102:0/3710458058 shutdown_connections 2024-03-23T03:51:03.733 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.732+0000 7fb15d400640 1 -- 172.21.15.102:0/3710458058 wait complete. 2024-03-23T03:51:03.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.734+0000 7fb15d400640 1 Processor -- start 2024-03-23T03:51:03.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.734+0000 7fb15d400640 1 -- start start 2024-03-23T03:51:03.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.735+0000 7fb15d400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb15806ce90 0x7fb15814b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:03.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.735+0000 7fb15d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1580cd3e0 0x7fb1580c3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:03.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.735+0000 7fb15d400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1580ce930 0x7fb15814bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:03.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.735+0000 7fb15d400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1580c4970 con 0x7fb1580cd3e0 2024-03-23T03:51:03.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.735+0000 7fb15d400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1580ac900 con 0x7fb1580ce930 2024-03-23T03:51:03.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.735+0000 7fb15d400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1580acb50 con 0x7fb15806ce90 2024-03-23T03:51:03.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.736+0000 7fb155e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1580cd3e0 0x7fb1580c3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:03.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.736+0000 7fb155e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1580cd3e0 0x7fb1580c3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34184/0 (socket says 172.21.15.102:34184) 2024-03-23T03:51:03.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.736+0000 7fb155e00640 1 -- 172.21.15.102:0/203127789 learned_addr learned my addr 172.21.15.102:0/203127789 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:03.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.736+0000 7fb156800640 1 --2- 172.21.15.102:0/203127789 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb15806ce90 0x7fb15814b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:03.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.736+0000 7fb157200640 1 --2- 172.21.15.102:0/203127789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1580ce930 0x7fb15814bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:03.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.736+0000 7fb155e00640 1 -- 172.21.15.102:0/203127789 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb15806ce90 msgr2=0x7fb15814b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:03.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.736+0000 7fb155e00640 1 --2- 172.21.15.102:0/203127789 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb15806ce90 0x7fb15814b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.736+0000 7fb155e00640 1 -- 172.21.15.102:0/203127789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1580ce930 msgr2=0x7fb15814bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:03.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.736+0000 7fb155e00640 1 --2- 172.21.15.102:0/203127789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1580ce930 0x7fb15814bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.736+0000 7fb155e00640 1 -- 172.21.15.102:0/203127789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1480094d0 con 0x7fb1580cd3e0 2024-03-23T03:51:03.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.736+0000 7fb157200640 1 --2- 172.21.15.102:0/203127789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1580ce930 0x7fb15814bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:03.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.737+0000 7fb155e00640 1 --2- 172.21.15.102:0/203127789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1580cd3e0 0x7fb1580c3530 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fb14402bee0 tx=0x7fb1440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:03.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.737+0000 7fb14ea00640 1 -- 172.21.15.102:0/203127789 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb14403b9f0 con 0x7fb1580cd3e0 2024-03-23T03:51:03.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.737+0000 7fb14ea00640 1 -- 172.21.15.102:0/203127789 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14403bb70 con 0x7fb1580cd3e0 2024-03-23T03:51:03.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.737+0000 7fb14ea00640 1 -- 172.21.15.102:0/203127789 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1440453f0 con 0x7fb1580cd3e0 2024-03-23T03:51:03.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.737+0000 7fb15d400640 1 -- 172.21.15.102:0/203127789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb144009400 con 0x7fb1580cd3e0 2024-03-23T03:51:03.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.737+0000 7fb15d400640 1 -- 172.21.15.102:0/203127789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1580ad110 con 0x7fb1580cd3e0 2024-03-23T03:51:03.739 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.738+0000 7fb15d400640 1 -- 172.21.15.102:0/203127789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1580ad300 con 0x7fb1580cd3e0 2024-03-23T03:51:03.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.742+0000 7fb14ea00640 1 -- 172.21.15.102:0/203127789 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb14404e450 con 0x7fb1580cd3e0 2024-03-23T03:51:03.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.743+0000 7fb14ea00640 1 --2- 172.21.15.102:0/203127789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb13806ac00 0x7fb13806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:03.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.743+0000 7fb14ea00640 1 -- 172.21.15.102:0/203127789 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb1440aba90 con 0x7fb1580cd3e0 2024-03-23T03:51:03.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.743+0000 7fb156800640 1 --2- 172.21.15.102:0/203127789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb13806ac00 0x7fb13806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:03.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.744+0000 7fb156800640 1 --2- 172.21.15.102:0/203127789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb13806ac00 0x7fb13806d0c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb148009cf0 tx=0x7fb148008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:03.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.745+0000 7fb14ea00640 1 -- 172.21.15.102:0/203127789 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb144076b10 con 0x7fb1580cd3e0 2024-03-23T03:51:03.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.939+0000 7fb15d400640 1 -- 172.21.15.102:0/203127789 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb158073390 con 0x7fb1580cd3e0 2024-03-23T03:51:03.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.941+0000 7fb14ea00640 1 -- 172.21.15.102:0/203127789 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb14407a130 con 0x7fb1580cd3e0 2024-03-23T03:51:03.941 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:51:03.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.943+0000 7fb15d400640 1 -- 172.21.15.102:0/203127789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb13806ac00 msgr2=0x7fb13806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:03.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.943+0000 7fb15d400640 1 --2- 172.21.15.102:0/203127789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb13806ac00 0x7fb13806d0c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fb148009cf0 tx=0x7fb148008040 comp rx=0 tx=0).stop 2024-03-23T03:51:03.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.943+0000 7fb15d400640 1 -- 172.21.15.102:0/203127789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1580cd3e0 msgr2=0x7fb1580c3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:03.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.943+0000 7fb15d400640 1 --2- 172.21.15.102:0/203127789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1580cd3e0 0x7fb1580c3530 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fb14402bee0 tx=0x7fb1440046c0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.943+0000 7fb15d400640 1 -- 172.21.15.102:0/203127789 shutdown_connections 2024-03-23T03:51:03.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.943+0000 7fb15d400640 1 --2- 172.21.15.102:0/203127789 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb13806ac00 0x7fb13806d0c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.943+0000 7fb15d400640 1 --2- 172.21.15.102:0/203127789 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb1580ce930 0x7fb15814bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.943+0000 7fb15d400640 1 --2- 172.21.15.102:0/203127789 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1580cd3e0 0x7fb1580c3530 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.943+0000 7fb15d400640 1 --2- 172.21.15.102:0/203127789 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb15806ce90 0x7fb15814b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:03.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.943+0000 7fb15d400640 1 -- 172.21.15.102:0/203127789 >> 172.21.15.102:0/203127789 conn(0x7fb15806a0b0 msgr2=0x7fb15806a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:03.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.943+0000 7fb15d400640 1 -- 172.21.15.102:0/203127789 shutdown_connections 2024-03-23T03:51:03.945 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:03.943+0000 7fb15d400640 1 -- 172.21.15.102:0/203127789 wait complete. 2024-03-23T03:51:03.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:51:04.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.159+0000 7f919b200640 1 Processor -- start 2024-03-23T03:51:04.160 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.159+0000 7f919b200640 1 -- start start 2024-03-23T03:51:04.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.161+0000 7f919b200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f919413fa80 0x7f919413ff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:04.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.161+0000 7f919b200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f91940cf040 0x7f91941422d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:04.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.161+0000 7f919b200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91941404a0 con 0x7f9194160dd0 2024-03-23T03:51:04.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.161+0000 7f919b200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f919406cdc0 con 0x7f919413fa80 2024-03-23T03:51:04.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.161+0000 7f919b200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f919406d060 con 0x7f91940cf040 2024-03-23T03:51:04.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.161+0000 7f9193e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f919413fa80 0x7f919413ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:04.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.161+0000 7f9193e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f919413fa80 0x7f919413ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34194/0 (socket says 172.21.15.102:34194) 2024-03-23T03:51:04.162 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.161+0000 7f9193e00640 1 -- 172.21.15.102:0/2340298021 learned_addr learned my addr 172.21.15.102:0/2340298021 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:04.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.162+0000 7f9193e00640 1 -- 172.21.15.102:0/2340298021 >> v2:172.21.15.173:3300/0 conn(0x7f91940cf040 msgr2=0x7f91941422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:04.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.162+0000 7f9193e00640 1 --2- 172.21.15.102:0/2340298021 >> v2:172.21.15.173:3300/0 conn(0x7f91940cf040 0x7f91941422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:04.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.162+0000 7f9193e00640 1 -- 172.21.15.102:0/2340298021 >> v1:172.21.15.173:6789/0 conn(0x7f9194160dd0 legacy=0x7f91940ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:04.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.162+0000 7f9193e00640 1 -- 172.21.15.102:0/2340298021 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919406d340 con 0x7f919413fa80 2024-03-23T03:51:04.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.163+0000 7f9193e00640 1 --2- 172.21.15.102:0/2340298021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f919413fa80 0x7f919413ff60 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f919406c850 tx=0x7f918402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc672beae14f5516 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:04.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.163+0000 7f9192a00640 1 -- 172.21.15.102:0/2340298021 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184031870 con 0x7f919413fa80 2024-03-23T03:51:04.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.163+0000 7f9192a00640 1 -- 172.21.15.102:0/2340298021 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91840319d0 con 0x7f919413fa80 2024-03-23T03:51:04.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.163+0000 7f9192a00640 1 -- 172.21.15.102:0/2340298021 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9184031d60 con 0x7f919413fa80 2024-03-23T03:51:04.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.164+0000 7f919b200640 1 -- 172.21.15.102:0/2340298021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f919413fa80 msgr2=0x7f919413ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.164+0000 7f919b200640 1 --2- 172.21.15.102:0/2340298021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f919413fa80 0x7f919413ff60 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f919406c850 tx=0x7f918402b910 comp rx=0 tx=0).stop 2024-03-23T03:51:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.164+0000 7f919b200640 1 -- 172.21.15.102:0/2340298021 shutdown_connections 2024-03-23T03:51:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.164+0000 7f919b200640 1 --2- 172.21.15.102:0/2340298021 >> v2:172.21.15.173:3300/0 conn(0x7f91940cf040 0x7f91941422d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.164+0000 7f919b200640 1 --2- 172.21.15.102:0/2340298021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f919413fa80 0x7f919413ff60 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.164+0000 7f919b200640 1 -- 172.21.15.102:0/2340298021 >> 172.21.15.102:0/2340298021 conn(0x7f919406a0b0 msgr2=0x7f919406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.164+0000 7f919b200640 1 -- 172.21.15.102:0/2340298021 shutdown_connections 2024-03-23T03:51:04.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.164+0000 7f919b200640 1 -- 172.21.15.102:0/2340298021 wait complete. 2024-03-23T03:51:04.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.165+0000 7f919b200640 1 Processor -- start 2024-03-23T03:51:04.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.166+0000 7f919b200640 1 -- start start 2024-03-23T03:51:04.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.166+0000 7f919b200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91940cf040 0x7f91941e6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:04.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.166+0000 7f919b200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f919413fa80 0x7f9194155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:04.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.166+0000 7f919b200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9194160dd0 0x7f91941e88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:04.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.166+0000 7f919b200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91940b11d0 con 0x7f9194160dd0 2024-03-23T03:51:04.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.166+0000 7f919b200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91940ac460 con 0x7f919413fa80 2024-03-23T03:51:04.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.166+0000 7f919b200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9194155cc0 con 0x7f91940cf040 2024-03-23T03:51:04.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.167+0000 7f9199200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9194160dd0 0x7f91941e88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:04.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.167+0000 7f9199200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9194160dd0 0x7f91941e88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34210/0 (socket says 172.21.15.102:34210) 2024-03-23T03:51:04.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.167+0000 7f9199200640 1 -- 172.21.15.102:0/3364767071 learned_addr learned my addr 172.21.15.102:0/3364767071 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:04.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.167+0000 7f9193e00640 1 --2- 172.21.15.102:0/3364767071 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91940cf040 0x7f91941e6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:04.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.167+0000 7f9193400640 1 --2- 172.21.15.102:0/3364767071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f919413fa80 0x7f9194155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:04.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.167+0000 7f9199200640 1 -- 172.21.15.102:0/3364767071 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91940cf040 msgr2=0x7f91941e6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:04.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.167+0000 7f9199200640 1 --2- 172.21.15.102:0/3364767071 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91940cf040 0x7f91941e6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:04.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.167+0000 7f9199200640 1 -- 172.21.15.102:0/3364767071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f919413fa80 msgr2=0x7f9194155780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:04.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.167+0000 7f9199200640 1 --2- 172.21.15.102:0/3364767071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f919413fa80 0x7f9194155780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:04.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.167+0000 7f9199200640 1 -- 172.21.15.102:0/3364767071 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91840094d0 con 0x7f9194160dd0 2024-03-23T03:51:04.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.167+0000 7f9199200640 1 --2- 172.21.15.102:0/3364767071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9194160dd0 0x7f91941e88d0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f918c00b3f0 tx=0x7f918c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:04.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.168+0000 7f918be00640 1 -- 172.21.15.102:0/3364767071 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918c011830 con 0x7f9194160dd0 2024-03-23T03:51:04.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.168+0000 7f918be00640 1 -- 172.21.15.102:0/3364767071 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918c0119b0 con 0x7f9194160dd0 2024-03-23T03:51:04.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.168+0000 7f918be00640 1 -- 172.21.15.102:0/3364767071 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918c011d60 con 0x7f9194160dd0 2024-03-23T03:51:04.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.168+0000 7f919b200640 1 -- 172.21.15.102:0/3364767071 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91940ac180 con 0x7f9194160dd0 2024-03-23T03:51:04.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.168+0000 7f919b200640 1 -- 172.21.15.102:0/3364767071 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91940b0ef0 con 0x7f9194160dd0 2024-03-23T03:51:04.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.169+0000 7f919b200640 1 -- 172.21.15.102:0/3364767071 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9194160600 con 0x7f9194160dd0 2024-03-23T03:51:04.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.170+0000 7f918be00640 1 -- 172.21.15.102:0/3364767071 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f918c01a400 con 0x7f9194160dd0 2024-03-23T03:51:04.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.170+0000 7f918be00640 1 --2- 172.21.15.102:0/3364767071 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f917006ac00 0x7f917006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:04.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.171+0000 7f918be00640 1 -- 172.21.15.102:0/3364767071 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f918c089610 con 0x7f9194160dd0 2024-03-23T03:51:04.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.171+0000 7f9193e00640 1 --2- 172.21.15.102:0/3364767071 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f917006ac00 0x7f917006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:04.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.171+0000 7f9193e00640 1 --2- 172.21.15.102:0/3364767071 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f917006ac00 0x7f917006d0c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9184036000 tx=0x7f918403c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:04.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.175+0000 7f918be00640 1 -- 172.21.15.102:0/3364767071 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f918c0546a0 con 0x7f9194160dd0 2024-03-23T03:51:04.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.382+0000 7f919b200640 1 -- 172.21.15.102:0/3364767071 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9194073390 con 0x7f9194160dd0 2024-03-23T03:51:04.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.383+0000 7f918be00640 1 -- 172.21.15.102:0/3364767071 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v93) v1 ==== 80+0+0 (secure 0 0 0) 0x7f918c057cc0 con 0x7f9194160dd0 2024-03-23T03:51:04.384 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.386+0000 7f919b200640 1 -- 172.21.15.102:0/3364767071 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f917006ac00 msgr2=0x7f917006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.386+0000 7f919b200640 1 --2- 172.21.15.102:0/3364767071 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f917006ac00 0x7f917006d0c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f9184036000 tx=0x7f918403c040 comp rx=0 tx=0).stop 2024-03-23T03:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.386+0000 7f919b200640 1 -- 172.21.15.102:0/3364767071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9194160dd0 msgr2=0x7f91941e88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.386+0000 7f919b200640 1 --2- 172.21.15.102:0/3364767071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9194160dd0 0x7f91941e88d0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f918c00b3f0 tx=0x7f918c00b8b0 comp rx=0 tx=0).stop 2024-03-23T03:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.386+0000 7f919b200640 1 -- 172.21.15.102:0/3364767071 shutdown_connections 2024-03-23T03:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.386+0000 7f919b200640 1 --2- 172.21.15.102:0/3364767071 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f917006ac00 0x7f917006d0c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.386+0000 7f919b200640 1 --2- 172.21.15.102:0/3364767071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9194160dd0 0x7f91941e88d0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.386+0000 7f919b200640 1 --2- 172.21.15.102:0/3364767071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f919413fa80 0x7f9194155780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.386+0000 7f919b200640 1 --2- 172.21.15.102:0/3364767071 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f91940cf040 0x7f91941e6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.386+0000 7f919b200640 1 -- 172.21.15.102:0/3364767071 >> 172.21.15.102:0/3364767071 conn(0x7f919406a0b0 msgr2=0x7f919415f930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.386+0000 7f919b200640 1 -- 172.21.15.102:0/3364767071 shutdown_connections 2024-03-23T03:51:04.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.386+0000 7f919b200640 1 -- 172.21.15.102:0/3364767071 wait complete. 2024-03-23T03:51:04.406 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-23T03:51:04.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:04.406 DEBUG:teuthology.orchestra.run.smithi173:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-03-23T03:51:04.515 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:51:04.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:04.516 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T03:51:04.563 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:51:04.563 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:04.618 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:51:04.618 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:51:04.618 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:04.674 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:51:04.674 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:51:04.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.889+0000 7fed2fc00640 1 Processor -- start 2024-03-23T03:51:04.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.889+0000 7fed2fc00640 1 -- start start 2024-03-23T03:51:04.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.890+0000 7fed2fc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fed2815b560 0x7fed2815b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:04.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.890+0000 7fed2fc00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fed280b39b0 0x7fed280b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:04.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.890+0000 7fed2fc00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2806c6f0 con 0x7fed2815e500 2024-03-23T03:51:04.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.890+0000 7fed2fc00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed280b4370 con 0x7fed280b39b0 2024-03-23T03:51:04.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.890+0000 7fed2fc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed2806c9d0 con 0x7fed2815b560 2024-03-23T03:51:04.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.891+0000 7fed2dc00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fed2815e500 0x7fed2815e990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60544/0 (socket says 172.21.15.102:60544) 2024-03-23T03:51:04.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.891+0000 7fed2dc00640 1 -- 172.21.15.102:0/3433462561 learned_addr learned my addr 172.21.15.102:0/3433462561 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:04.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.891+0000 7fed1fe00640 1 --2- 172.21.15.102:0/3433462561 >> v2:172.21.15.173:3300/0 conn(0x7fed280b39b0 0x7fed280b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:04.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.891+0000 7fed27e00640 1 -- 172.21.15.102:0/3433462561 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4212327892 0 0) 0x7fed2806c6f0 con 0x7fed2815e500 2024-03-23T03:51:04.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.891+0000 7fed27e00640 1 -- 172.21.15.102:0/3433462561 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed0c003520 con 0x7fed2815e500 2024-03-23T03:51:04.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.891+0000 7fed2d200640 1 --2- 172.21.15.102:0/3433462561 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fed2815b560 0x7fed2815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:04.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.891+0000 7fed27e00640 1 -- 172.21.15.102:0/3433462561 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2951956051 0 0) 0x7fed0c003520 con 0x7fed2815e500 2024-03-23T03:51:04.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.891+0000 7fed27e00640 1 -- 172.21.15.102:0/3433462561 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fed2815b560 msgr2=0x7fed2815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:04.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.891+0000 7fed27e00640 1 --2- 172.21.15.102:0/3433462561 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fed2815b560 0x7fed2815b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:04.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.891+0000 7fed27e00640 1 -- 172.21.15.102:0/3433462561 >> v2:172.21.15.173:3300/0 conn(0x7fed280b39b0 msgr2=0x7fed280b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:04.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.891+0000 7fed27e00640 1 --2- 172.21.15.102:0/3433462561 >> v2:172.21.15.173:3300/0 conn(0x7fed280b39b0 0x7fed280b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:04.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.891+0000 7fed27e00640 1 -- 172.21.15.102:0/3433462561 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2806ccb0 con 0x7fed2815e500 2024-03-23T03:51:04.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.891+0000 7fed27e00640 1 -- 172.21.15.102:0/3433462561 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fed200019f0 con 0x7fed2815e500 2024-03-23T03:51:04.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.891+0000 7fed27e00640 1 -- 172.21.15.102:0/3433462561 >> v1:172.21.15.102:6789/0 conn(0x7fed2815e500 legacy=0x7fed2815e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:04.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.891+0000 7fed27e00640 1 --2- 172.21.15.102:0/3433462561 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fed2815b560 0x7fed0c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:04.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.891+0000 7fed27e00640 1 --2- 172.21.15.102:0/3433462561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c003fa0 0x7fed0c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:04.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.892+0000 7fed27e00640 1 --2- 172.21.15.102:0/3433462561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed0c004d00 0x7fed0c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:04.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.892+0000 7fed27e00640 1 -- 172.21.15.102:0/3433462561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed280b4370 con 0x7fed0c003fa0 2024-03-23T03:51:04.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.892+0000 7fed27e00640 1 -- 172.21.15.102:0/3433462561 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2806c9d0 con 0x7fed0c004d00 2024-03-23T03:51:04.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.892+0000 7fed27e00640 1 -- 172.21.15.102:0/3433462561 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c00ba80 con 0x7fed2815b560 2024-03-23T03:51:04.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.892+0000 7fed1fe00640 1 --2- 172.21.15.102:0/3433462561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c003fa0 0x7fed0c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:04.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.892+0000 7fed2d200640 1 --2- 172.21.15.102:0/3433462561 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fed2815b560 0x7fed0c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:04.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.892+0000 7fed2dc00640 1 --2- 172.21.15.102:0/3433462561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed0c004d00 0x7fed0c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:04.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.892+0000 7fed1fe00640 1 -- 172.21.15.102:0/3433462561 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fed2815b560 msgr2=0x7fed0c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:04.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.892+0000 7fed1fe00640 1 --2- 172.21.15.102:0/3433462561 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fed2815b560 0x7fed0c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:04.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.892+0000 7fed1fe00640 1 -- 172.21.15.102:0/3433462561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed0c004d00 msgr2=0x7fed0c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:04.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.892+0000 7fed1fe00640 1 --2- 172.21.15.102:0/3433462561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed0c004d00 0x7fed0c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:04.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.892+0000 7fed1fe00640 1 -- 172.21.15.102:0/3433462561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c003520 con 0x7fed0c003fa0 2024-03-23T03:51:04.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.894+0000 7fed1fe00640 1 --2- 172.21.15.102:0/3433462561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c003fa0 0x7fed0c0047c0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fed1000ba30 tx=0x7fed0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35c4551f8250740e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:04.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.894+0000 7fed27e00640 1 -- 172.21.15.102:0/3433462561 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed10032680 con 0x7fed0c003fa0 2024-03-23T03:51:04.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.894+0000 7fed27e00640 1 -- 172.21.15.102:0/3433462561 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed100327e0 con 0x7fed0c003fa0 2024-03-23T03:51:04.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.894+0000 7fed27e00640 1 -- 172.21.15.102:0/3433462561 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed10032b70 con 0x7fed0c003fa0 2024-03-23T03:51:04.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.894+0000 7fed2fc00640 1 -- 172.21.15.102:0/3433462561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c003fa0 msgr2=0x7fed0c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:04.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.894+0000 7fed2fc00640 1 --2- 172.21.15.102:0/3433462561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c003fa0 0x7fed0c0047c0 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fed1000ba30 tx=0x7fed0c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:04.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.894+0000 7fed2dc00640 1 -- 172.21.15.102:0/3433462561 reap_dead start 2024-03-23T03:51:04.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.895+0000 7fed2fc00640 1 -- 172.21.15.102:0/3433462561 shutdown_connections 2024-03-23T03:51:04.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.895+0000 7fed2fc00640 1 -- 172.21.15.102:0/3433462561 >> 172.21.15.102:0/3433462561 conn(0x7fed2806a3d0 msgr2=0x7fed280b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:04.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.895+0000 7fed2fc00640 1 -- 172.21.15.102:0/3433462561 shutdown_connections 2024-03-23T03:51:04.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.895+0000 7fed2fc00640 1 -- 172.21.15.102:0/3433462561 wait complete. 2024-03-23T03:51:04.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.896+0000 7fed2fc00640 1 Processor -- start 2024-03-23T03:51:04.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.896+0000 7fed2fc00640 1 -- start start 2024-03-23T03:51:04.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.897+0000 7fed2fc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c003fa0 0x7fed281f6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:04.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.897+0000 7fed2fc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fed2812e790 0x7fed2813d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:04.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.897+0000 7fed2fc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed281f1ab0 0x7fed281f1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:04.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.897+0000 7fed2fc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed281f67b0 con 0x7fed0c003fa0 2024-03-23T03:51:04.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.897+0000 7fed2fc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2812ed00 con 0x7fed281f1ab0 2024-03-23T03:51:04.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.897+0000 7fed2fc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed2812efe0 con 0x7fed2812e790 2024-03-23T03:51:04.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.898+0000 7fed2d200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c003fa0 0x7fed281f6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:04.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.898+0000 7fed2d200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c003fa0 0x7fed281f6270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34240/0 (socket says 172.21.15.102:34240) 2024-03-23T03:51:04.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.898+0000 7fed2d200640 1 -- 172.21.15.102:0/151428559 learned_addr learned my addr 172.21.15.102:0/151428559 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:04.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.898+0000 7fed2d200640 1 -- 172.21.15.102:0/151428559 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fed2812e790 msgr2=0x7fed2813d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:04.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.898+0000 7fed2d200640 1 --2- 172.21.15.102:0/151428559 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fed2812e790 0x7fed2813d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:04.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.898+0000 7fed2d200640 1 -- 172.21.15.102:0/151428559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed281f1ab0 msgr2=0x7fed281f1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:04.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.898+0000 7fed2d200640 1 --2- 172.21.15.102:0/151428559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed281f1ab0 0x7fed281f1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:04.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.898+0000 7fed2d200640 1 -- 172.21.15.102:0/151428559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2812f2c0 con 0x7fed0c003fa0 2024-03-23T03:51:04.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.898+0000 7fed2d200640 1 --2- 172.21.15.102:0/151428559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c003fa0 0x7fed281f6270 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fed1800f580 tx=0x7fed1800fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:04.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.898+0000 7fed25600640 1 -- 172.21.15.102:0/151428559 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed18014870 con 0x7fed0c003fa0 2024-03-23T03:51:04.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.898+0000 7fed25600640 1 -- 172.21.15.102:0/151428559 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed180149d0 con 0x7fed0c003fa0 2024-03-23T03:51:04.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.898+0000 7fed25600640 1 -- 172.21.15.102:0/151428559 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed18014d80 con 0x7fed0c003fa0 2024-03-23T03:51:04.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.899+0000 7fed2fc00640 1 -- 172.21.15.102:0/151428559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed2812f5a0 con 0x7fed0c003fa0 2024-03-23T03:51:04.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.899+0000 7fed2fc00640 1 -- 172.21.15.102:0/151428559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed280b29e0 con 0x7fed0c003fa0 2024-03-23T03:51:04.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.900+0000 7fed2fc00640 1 -- 172.21.15.102:0/151428559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed2806ac20 con 0x7fed0c003fa0 2024-03-23T03:51:04.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.901+0000 7fed25600640 1 -- 172.21.15.102:0/151428559 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fed18015410 con 0x7fed0c003fa0 2024-03-23T03:51:04.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.901+0000 7fed25600640 1 --2- 172.21.15.102:0/151428559 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fed0406ac00 0x7fed0406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:04.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.902+0000 7fed25600640 1 -- 172.21.15.102:0/151428559 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fed18089ec0 con 0x7fed0c003fa0 2024-03-23T03:51:04.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.902+0000 7fed1fe00640 1 --2- 172.21.15.102:0/151428559 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fed0406ac00 0x7fed0406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:04.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.902+0000 7fed1fe00640 1 --2- 172.21.15.102:0/151428559 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fed0406ac00 0x7fed0406d0c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fed1000ba00 tx=0x7fed10032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:04.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:04.906+0000 7fed25600640 1 -- 172.21.15.102:0/151428559 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fed18054ed0 con 0x7fed0c003fa0 2024-03-23T03:51:05.122 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.121+0000 7fed2fc00640 1 -- 172.21.15.102:0/151428559 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed281f2480 con 0x7fed0c003fa0 2024-03-23T03:51:05.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.122+0000 7fed25600640 1 -- 172.21.15.102:0/151428559 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4745 (secure 0 0 0) 0x7fed180584f0 con 0x7fed0c003fa0 2024-03-23T03:51:05.123 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:05.123 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/755891320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":755891320},{"type":"v1","addr":"172.21.15.102:6836","nonce":755891320}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":5207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3792558756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3792558756},{"type":"v1","addr":"172.21.15.173:6835","nonce":3792558756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":5590,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1632622692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1632622692},{"type":"v1","addr":"172.21.15.173:6833","nonce":1632622692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:50:49.296593+0000","modified":"2024-03-23T03:50:52.307714+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/3452944670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":3452944670},{"type":"v1","addr":"172.21.15.102:6837","nonce":3452944670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T03:51:05.123 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 48 2024-03-23T03:51:05.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.125+0000 7fed2fc00640 1 -- 172.21.15.102:0/151428559 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fed0406ac00 msgr2=0x7fed0406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:05.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.125+0000 7fed2fc00640 1 --2- 172.21.15.102:0/151428559 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fed0406ac00 0x7fed0406d0c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fed1000ba00 tx=0x7fed10032000 comp rx=0 tx=0).stop 2024-03-23T03:51:05.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.125+0000 7fed2fc00640 1 -- 172.21.15.102:0/151428559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c003fa0 msgr2=0x7fed281f6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:05.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.125+0000 7fed2fc00640 1 --2- 172.21.15.102:0/151428559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c003fa0 0x7fed281f6270 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fed1800f580 tx=0x7fed1800fa40 comp rx=0 tx=0).stop 2024-03-23T03:51:05.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.125+0000 7fed2fc00640 1 -- 172.21.15.102:0/151428559 shutdown_connections 2024-03-23T03:51:05.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.125+0000 7fed2fc00640 1 --2- 172.21.15.102:0/151428559 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fed0406ac00 0x7fed0406d0c0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.126 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.125+0000 7fed2fc00640 1 --2- 172.21.15.102:0/151428559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fed281f1ab0 0x7fed281f1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.125+0000 7fed2fc00640 1 --2- 172.21.15.102:0/151428559 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fed2812e790 0x7fed2813d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.125+0000 7fed2fc00640 1 --2- 172.21.15.102:0/151428559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c003fa0 0x7fed281f6270 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.125+0000 7fed2fc00640 1 -- 172.21.15.102:0/151428559 >> 172.21.15.102:0/151428559 conn(0x7fed2806a3d0 msgr2=0x7fed281434e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:05.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.125+0000 7fed2fc00640 1 -- 172.21.15.102:0/151428559 shutdown_connections 2024-03-23T03:51:05.127 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.125+0000 7fed2fc00640 1 -- 172.21.15.102:0/151428559 wait complete. 2024-03-23T03:51:05.145 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:51:05.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.353+0000 7f81bb200640 1 Processor -- start 2024-03-23T03:51:05.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.353+0000 7f81bb200640 1 -- start start 2024-03-23T03:51:05.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.355+0000 7f81bb200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f81bc06aca0 0x7f81bc0cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:05.356 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.355+0000 7f81bb200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f81bc15a800 0x7f81bc06b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.355+0000 7f81bb200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f81bc071350 0x7f81bc06c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.355+0000 7f81bb200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81bc0cd9e0 con 0x7f81bc071350 2024-03-23T03:51:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.355+0000 7f81bb200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81bc071b10 con 0x7f81bc06aca0 2024-03-23T03:51:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.355+0000 7f81bb200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc0ce260 con 0x7f81bc15a800 2024-03-23T03:51:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.355+0000 7f81ba800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f81bc071350 0x7f81bc06c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.356+0000 7f81ba800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f81bc071350 0x7f81bc06c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34246/0 (socket says 172.21.15.102:34246) 2024-03-23T03:51:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.356+0000 7f81ba800640 1 -- 172.21.15.102:0/2344996347 learned_addr learned my addr 172.21.15.102:0/2344996347 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.356+0000 7f81b9e00640 1 --2- 172.21.15.102:0/2344996347 >> v2:172.21.15.173:3300/0 conn(0x7f81bc06aca0 0x7f81bc0cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.356+0000 7f81b9400640 1 --2- 172.21.15.102:0/2344996347 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f81bc15a800 0x7f81bc06b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.356+0000 7f81ba800640 1 -- 172.21.15.102:0/2344996347 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f81bc15a800 msgr2=0x7f81bc06b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.356+0000 7f81ba800640 1 --2- 172.21.15.102:0/2344996347 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f81bc15a800 0x7f81bc06b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.356+0000 7f81ba800640 1 -- 172.21.15.102:0/2344996347 >> v2:172.21.15.173:3300/0 conn(0x7f81bc06aca0 msgr2=0x7f81bc0cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.356+0000 7f81ba800640 1 --2- 172.21.15.102:0/2344996347 >> v2:172.21.15.173:3300/0 conn(0x7f81bc06aca0 0x7f81bc0cc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.356+0000 7f81ba800640 1 -- 172.21.15.102:0/2344996347 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81bc06b5c0 con 0x7f81bc071350 2024-03-23T03:51:05.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.357+0000 7f81ba800640 1 --2- 172.21.15.102:0/2344996347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81bc071350 0x7f81bc06c7f0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0b1670 tx=0x7f81b402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb558286e8859d5f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:05.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.357+0000 7f81b8a00640 1 -- 172.21.15.102:0/2344996347 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b402ca70 con 0x7f81bc071350 2024-03-23T03:51:05.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.357+0000 7f81b8a00640 1 -- 172.21.15.102:0/2344996347 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81b402cbd0 con 0x7f81bc071350 2024-03-23T03:51:05.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.357+0000 7f81b8a00640 1 -- 172.21.15.102:0/2344996347 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b40343f0 con 0x7f81bc071350 2024-03-23T03:51:05.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.358+0000 7f81bb200640 1 -- 172.21.15.102:0/2344996347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81bc071350 msgr2=0x7f81bc06c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:05.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.358+0000 7f81bb200640 1 --2- 172.21.15.102:0/2344996347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81bc071350 0x7f81bc06c7f0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f81bc0b1670 tx=0x7f81b402bed0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.358+0000 7f81bb200640 1 -- 172.21.15.102:0/2344996347 shutdown_connections 2024-03-23T03:51:05.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.358+0000 7f81bb200640 1 --2- 172.21.15.102:0/2344996347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81bc071350 0x7f81bc06c7f0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.358+0000 7f81bb200640 1 --2- 172.21.15.102:0/2344996347 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f81bc15a800 0x7f81bc06b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.358+0000 7f81bb200640 1 --2- 172.21.15.102:0/2344996347 >> v2:172.21.15.173:3300/0 conn(0x7f81bc06aca0 0x7f81bc0cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.359 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.358+0000 7f81bb200640 1 -- 172.21.15.102:0/2344996347 >> 172.21.15.102:0/2344996347 conn(0x7f81bc070860 msgr2=0x7f81bc070c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:05.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.358+0000 7f81bb200640 1 -- 172.21.15.102:0/2344996347 shutdown_connections 2024-03-23T03:51:05.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.358+0000 7f81bb200640 1 -- 172.21.15.102:0/2344996347 wait complete. 2024-03-23T03:51:05.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.360+0000 7f81bb200640 1 Processor -- start 2024-03-23T03:51:05.360 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.360+0000 7f81bb200640 1 -- start start 2024-03-23T03:51:05.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.361+0000 7f81bb200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81bc06aca0 0x7f81bc15b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:05.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.361+0000 7f81bb200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81bc071350 0x7f81bc154160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:05.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.361+0000 7f81bb200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f81bc15abf0 0x7f81bc14b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:05.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.361+0000 7f81bb200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc152d40 con 0x7f81bc071350 2024-03-23T03:51:05.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.361+0000 7f81bb200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc15bae0 con 0x7f81bc06aca0 2024-03-23T03:51:05.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.361+0000 7f81bb200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81bc15d410 con 0x7f81bc15abf0 2024-03-23T03:51:05.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.361+0000 7f81b9400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81bc071350 0x7f81bc154160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:05.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.361+0000 7f81b9400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81bc071350 0x7f81bc154160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34248/0 (socket says 172.21.15.102:34248) 2024-03-23T03:51:05.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.361+0000 7f81b9400640 1 -- 172.21.15.102:0/3790432595 learned_addr learned my addr 172.21.15.102:0/3790432595 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:05.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.361+0000 7f81b9e00640 1 --2- 172.21.15.102:0/3790432595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81bc06aca0 0x7f81bc15b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:05.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.361+0000 7f81b9400640 1 -- 172.21.15.102:0/3790432595 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f81bc15abf0 msgr2=0x7f81bc14b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:05.363 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.361+0000 7f81ba800640 1 --2- 172.21.15.102:0/3790432595 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f81bc15abf0 0x7f81bc14b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:05.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.362+0000 7f81b9400640 1 --2- 172.21.15.102:0/3790432595 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f81bc15abf0 0x7f81bc14b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.362+0000 7f81b9400640 1 -- 172.21.15.102:0/3790432595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81bc06aca0 msgr2=0x7f81bc15b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:05.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.362+0000 7f81b9400640 1 --2- 172.21.15.102:0/3790432595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81bc06aca0 0x7f81bc15b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.362+0000 7f81b9400640 1 -- 172.21.15.102:0/3790432595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81b4009b30 con 0x7f81bc071350 2024-03-23T03:51:05.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.362+0000 7f81b9e00640 1 --2- 172.21.15.102:0/3790432595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81bc06aca0 0x7f81bc15b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:05.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.362+0000 7f81ba800640 1 --2- 172.21.15.102:0/3790432595 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f81bc15abf0 0x7f81bc14b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:05.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.362+0000 7f81b9400640 1 --2- 172.21.15.102:0/3790432595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81bc071350 0x7f81bc154160 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f81a800d800 tx=0x7f81a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:05.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.362+0000 7f81b2000640 1 -- 172.21.15.102:0/3790432595 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a800b6c0 con 0x7f81bc071350 2024-03-23T03:51:05.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.362+0000 7f81b2000640 1 -- 172.21.15.102:0/3790432595 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a800b820 con 0x7f81bc071350 2024-03-23T03:51:05.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.362+0000 7f81b2000640 1 -- 172.21.15.102:0/3790432595 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a800bbb0 con 0x7f81bc071350 2024-03-23T03:51:05.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.362+0000 7f81bb200640 1 -- 172.21.15.102:0/3790432595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81bc15d6b0 con 0x7f81bc071350 2024-03-23T03:51:05.364 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.362+0000 7f81bb200640 1 -- 172.21.15.102:0/3790432595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81bc15bcf0 con 0x7f81bc071350 2024-03-23T03:51:05.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.363+0000 7f81bb200640 1 -- 172.21.15.102:0/3790432595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81bc073e40 con 0x7f81bc071350 2024-03-23T03:51:05.365 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.365+0000 7f81b2000640 1 -- 172.21.15.102:0/3790432595 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f81a800bd10 con 0x7f81bc071350 2024-03-23T03:51:05.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.365+0000 7f81b2000640 1 --2- 172.21.15.102:0/3790432595 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f819c06ac00 0x7f819c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:05.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.366+0000 7f81b2000640 1 -- 172.21.15.102:0/3790432595 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f81a8089ae0 con 0x7f81bc071350 2024-03-23T03:51:05.366 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.366+0000 7f81b9e00640 1 --2- 172.21.15.102:0/3790432595 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f819c06ac00 0x7f819c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:05.367 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.366+0000 7f81b9e00640 1 --2- 172.21.15.102:0/3790432595 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f819c06ac00 0x7f819c06d0c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f81ac006fd0 tx=0x7f81ac008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:05.370 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.370+0000 7f81b2000640 1 -- 172.21.15.102:0/3790432595 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f81a8054ae0 con 0x7f81bc071350 2024-03-23T03:51:05.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.568+0000 7f81bb200640 1 -- 172.21.15.102:0/3790432595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81bc04e030 con 0x7f81bc071350 2024-03-23T03:51:05.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.570+0000 7f81b2000640 1 -- 172.21.15.102:0/3790432595 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16297 (secure 0 0 0) 0x7f81a8058100 con 0x7f81bc071350 2024-03-23T03:51:05.571 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:05.571 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":93,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:50.300471+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:50:47.750670+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-23T03:50:48.305583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:51:05.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.573+0000 7f81bb200640 1 -- 172.21.15.102:0/3790432595 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f819c06ac00 msgr2=0x7f819c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:05.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.573+0000 7f81bb200640 1 --2- 172.21.15.102:0/3790432595 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f819c06ac00 0x7f819c06d0c0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f81ac006fd0 tx=0x7f81ac008040 comp rx=0 tx=0).stop 2024-03-23T03:51:05.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.573+0000 7f81bb200640 1 -- 172.21.15.102:0/3790432595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81bc071350 msgr2=0x7f81bc154160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:05.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.573+0000 7f81bb200640 1 --2- 172.21.15.102:0/3790432595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81bc071350 0x7f81bc154160 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f81a800d800 tx=0x7f81a800dcc0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.573+0000 7f81bb200640 1 -- 172.21.15.102:0/3790432595 shutdown_connections 2024-03-23T03:51:05.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.573+0000 7f81bb200640 1 --2- 172.21.15.102:0/3790432595 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f819c06ac00 0x7f819c06d0c0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.573+0000 7f81bb200640 1 --2- 172.21.15.102:0/3790432595 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f81bc15abf0 0x7f81bc14b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.573+0000 7f81bb200640 1 --2- 172.21.15.102:0/3790432595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81bc071350 0x7f81bc154160 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.573+0000 7f81bb200640 1 --2- 172.21.15.102:0/3790432595 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81bc06aca0 0x7f81bc15b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.573+0000 7f81bb200640 1 -- 172.21.15.102:0/3790432595 >> 172.21.15.102:0/3790432595 conn(0x7f81bc070860 msgr2=0x7f81bc0cbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:05.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.573+0000 7f81bb200640 1 -- 172.21.15.102:0/3790432595 shutdown_connections 2024-03-23T03:51:05.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.573+0000 7f81bb200640 1 -- 172.21.15.102:0/3790432595 wait complete. 2024-03-23T03:51:05.591 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:50:47.750670+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '92', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:50:48.305583+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '93', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:51:05.591 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:51:05.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.796+0000 7fcee6a00640 1 Processor -- start 2024-03-23T03:51:05.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.796+0000 7fcee6a00640 1 -- start start 2024-03-23T03:51:05.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.798+0000 7fcee6a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcee00cd740 0x7fcee00b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:05.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.798+0000 7fcee6a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fcee00d0d40 0x7fcee00cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:05.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.798+0000 7fcee6a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcee006b0a0 0x7fcee006b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:05.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.798+0000 7fcee6a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee00d20c0 con 0x7fcee006b0a0 2024-03-23T03:51:05.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.798+0000 7fcee6a00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee00ce060 con 0x7fcee00d0d40 2024-03-23T03:51:05.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.798+0000 7fcee6a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00d14b0 con 0x7fcee00cd740 2024-03-23T03:51:05.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.798+0000 7fcee4a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcee006b0a0 0x7fcee006b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:05.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.798+0000 7fcee4a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcee006b0a0 0x7fcee006b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34258/0 (socket says 172.21.15.102:34258) 2024-03-23T03:51:05.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.798+0000 7fcee4a00640 1 -- 172.21.15.102:0/1773353688 learned_addr learned my addr 172.21.15.102:0/1773353688 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:05.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.798+0000 7fcedfe00640 1 --2- 172.21.15.102:0/1773353688 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcee00cd740 0x7fcee00b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:05.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.799+0000 7fced7e00640 1 --2- 172.21.15.102:0/1773353688 >> v2:172.21.15.173:3300/0 conn(0x7fcee00d0d40 0x7fcee00cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:05.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.799+0000 7fcee4a00640 1 -- 172.21.15.102:0/1773353688 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcee00cd740 msgr2=0x7fcee00b84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:05.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.799+0000 7fcee4a00640 1 --2- 172.21.15.102:0/1773353688 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcee00cd740 0x7fcee00b84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.799+0000 7fcee4a00640 1 -- 172.21.15.102:0/1773353688 >> v2:172.21.15.173:3300/0 conn(0x7fcee00d0d40 msgr2=0x7fcee00cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:05.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.799+0000 7fcee4a00640 1 --2- 172.21.15.102:0/1773353688 >> v2:172.21.15.173:3300/0 conn(0x7fcee00d0d40 0x7fcee00cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.799+0000 7fcee4a00640 1 -- 172.21.15.102:0/1773353688 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee00d1790 con 0x7fcee006b0a0 2024-03-23T03:51:05.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.799+0000 7fced7e00640 1 --2- 172.21.15.102:0/1773353688 >> v2:172.21.15.173:3300/0 conn(0x7fcee00d0d40 0x7fcee00cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:05.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.800+0000 7fcee4a00640 1 --2- 172.21.15.102:0/1773353688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee006b0a0 0x7fcee006b530 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fcee008acc0 tx=0x7fced802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=850ea54274cf20c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:05.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.800+0000 7fcedf400640 1 -- 172.21.15.102:0/1773353688 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced802ca70 con 0x7fcee006b0a0 2024-03-23T03:51:05.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.800+0000 7fcedf400640 1 -- 172.21.15.102:0/1773353688 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced802cbd0 con 0x7fcee006b0a0 2024-03-23T03:51:05.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.800+0000 7fcedf400640 1 -- 172.21.15.102:0/1773353688 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced80343f0 con 0x7fcee006b0a0 2024-03-23T03:51:05.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.801+0000 7fcee6a00640 1 -- 172.21.15.102:0/1773353688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee006b0a0 msgr2=0x7fcee006b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:05.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.801+0000 7fcee6a00640 1 --2- 172.21.15.102:0/1773353688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee006b0a0 0x7fcee006b530 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fcee008acc0 tx=0x7fced802bed0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.801+0000 7fcee6a00640 1 -- 172.21.15.102:0/1773353688 shutdown_connections 2024-03-23T03:51:05.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.801+0000 7fcee6a00640 1 --2- 172.21.15.102:0/1773353688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee006b0a0 0x7fcee006b530 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.801+0000 7fcee6a00640 1 --2- 172.21.15.102:0/1773353688 >> v2:172.21.15.173:3300/0 conn(0x7fcee00d0d40 0x7fcee00cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.801+0000 7fcee6a00640 1 --2- 172.21.15.102:0/1773353688 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcee00cd740 0x7fcee00b84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.801+0000 7fcee6a00640 1 -- 172.21.15.102:0/1773353688 >> 172.21.15.102:0/1773353688 conn(0x7fcee00cf5d0 msgr2=0x7fcee00cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:05.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.801+0000 7fcee6a00640 1 -- 172.21.15.102:0/1773353688 shutdown_connections 2024-03-23T03:51:05.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.801+0000 7fcee6a00640 1 -- 172.21.15.102:0/1773353688 wait complete. 2024-03-23T03:51:05.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.802+0000 7fcee6a00640 1 Processor -- start 2024-03-23T03:51:05.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.803+0000 7fcee6a00640 1 -- start start 2024-03-23T03:51:05.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.803+0000 7fcee6a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcee006b0a0 0x7fcee0146240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:05.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.803+0000 7fcee6a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcee00cd740 0x7fcee0146780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:05.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.803+0000 7fcee6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee00d0d40 0x7fcee01598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:05.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.803+0000 7fcee6a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee01441b0 con 0x7fcee00d0d40 2024-03-23T03:51:05.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.803+0000 7fcee6a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee01476a0 con 0x7fcee006b0a0 2024-03-23T03:51:05.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.803+0000 7fcee6a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00cb3c0 con 0x7fcee00cd740 2024-03-23T03:51:05.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.804+0000 7fcee4a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee00d0d40 0x7fcee01598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:05.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.804+0000 7fcee4a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee00d0d40 0x7fcee01598c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34260/0 (socket says 172.21.15.102:34260) 2024-03-23T03:51:05.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.804+0000 7fcee4a00640 1 -- 172.21.15.102:0/1107106701 learned_addr learned my addr 172.21.15.102:0/1107106701 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:05.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.804+0000 7fcedfe00640 1 --2- 172.21.15.102:0/1107106701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcee006b0a0 0x7fcee0146240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:05.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.804+0000 7fcee4a00640 1 -- 172.21.15.102:0/1107106701 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcee00cd740 msgr2=0x7fcee0146780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:05.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.804+0000 7fcee4a00640 1 --2- 172.21.15.102:0/1107106701 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcee00cd740 0x7fcee0146780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.804+0000 7fcee4a00640 1 -- 172.21.15.102:0/1107106701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcee006b0a0 msgr2=0x7fcee0146240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:05.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.804+0000 7fcee4a00640 1 --2- 172.21.15.102:0/1107106701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcee006b0a0 0x7fcee0146240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:05.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.804+0000 7fcee4a00640 1 -- 172.21.15.102:0/1107106701 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced8009b30 con 0x7fcee00d0d40 2024-03-23T03:51:05.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.804+0000 7fcedfe00640 1 --2- 172.21.15.102:0/1107106701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcee006b0a0 0x7fcee0146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:05.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.805+0000 7fcee4a00640 1 --2- 172.21.15.102:0/1107106701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee00d0d40 0x7fcee01598c0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fced8009f40 tx=0x7fced8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:05.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.805+0000 7fcedcc00640 1 -- 172.21.15.102:0/1107106701 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced8009510 con 0x7fcee00d0d40 2024-03-23T03:51:05.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.805+0000 7fcedcc00640 1 -- 172.21.15.102:0/1107106701 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced803e450 con 0x7fcee00d0d40 2024-03-23T03:51:05.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.805+0000 7fcee6a00640 1 -- 172.21.15.102:0/1107106701 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee0144400 con 0x7fcee00d0d40 2024-03-23T03:51:05.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.805+0000 7fcedcc00640 1 -- 172.21.15.102:0/1107106701 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced803e800 con 0x7fcee00d0d40 2024-03-23T03:51:05.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.805+0000 7fcee6a00640 1 -- 172.21.15.102:0/1107106701 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee01449c0 con 0x7fcee00d0d40 2024-03-23T03:51:05.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.806+0000 7fcee6a00640 1 -- 172.21.15.102:0/1107106701 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee0073390 con 0x7fcee00d0d40 2024-03-23T03:51:05.808 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.807+0000 7fcedcc00640 1 -- 172.21.15.102:0/1107106701 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fced803e960 con 0x7fcee00d0d40 2024-03-23T03:51:05.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.809+0000 7fcedcc00640 1 --2- 172.21.15.102:0/1107106701 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcebc06ac00 0x7fcebc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:05.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.809+0000 7fcedcc00640 1 -- 172.21.15.102:0/1107106701 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fced80ac2c0 con 0x7fcee00d0d40 2024-03-23T03:51:05.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.809+0000 7fcedfe00640 1 --2- 172.21.15.102:0/1107106701 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcebc06ac00 0x7fcebc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:05.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.809+0000 7fcedfe00640 1 --2- 172.21.15.102:0/1107106701 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcebc06ac00 0x7fcebc06d0c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fced0006fd0 tx=0x7fced0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:05.815 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:05.814+0000 7fcedcc00640 1 -- 172.21.15.102:0/1107106701 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fced80772c0 con 0x7fcee00d0d40 2024-03-23T03:51:06.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.010+0000 7fcee6a00640 1 -- 172.21.15.102:0/1107106701 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcee00b20b0 con 0x7fcee00d0d40 2024-03-23T03:51:06.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.011+0000 7fcedcc00640 1 -- 172.21.15.102:0/1107106701 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fced807a8e0 con 0x7fcee00d0d40 2024-03-23T03:51:06.012 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:51:06.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.013+0000 7fcee6a00640 1 -- 172.21.15.102:0/1107106701 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcebc06ac00 msgr2=0x7fcebc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:06.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.013+0000 7fcee6a00640 1 --2- 172.21.15.102:0/1107106701 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcebc06ac00 0x7fcebc06d0c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fced0006fd0 tx=0x7fced0008040 comp rx=0 tx=0).stop 2024-03-23T03:51:06.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.013+0000 7fcee6a00640 1 -- 172.21.15.102:0/1107106701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee00d0d40 msgr2=0x7fcee01598c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:06.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.013+0000 7fcee6a00640 1 --2- 172.21.15.102:0/1107106701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee00d0d40 0x7fcee01598c0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fced8009f40 tx=0x7fced8004a40 comp rx=0 tx=0).stop 2024-03-23T03:51:06.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.014+0000 7fcee6a00640 1 -- 172.21.15.102:0/1107106701 shutdown_connections 2024-03-23T03:51:06.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.014+0000 7fcee6a00640 1 --2- 172.21.15.102:0/1107106701 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcebc06ac00 0x7fcebc06d0c0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.014+0000 7fcee6a00640 1 --2- 172.21.15.102:0/1107106701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcee00d0d40 0x7fcee01598c0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.014+0000 7fcee6a00640 1 --2- 172.21.15.102:0/1107106701 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcee00cd740 0x7fcee0146780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.014+0000 7fcee6a00640 1 --2- 172.21.15.102:0/1107106701 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcee006b0a0 0x7fcee0146240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.014+0000 7fcee6a00640 1 -- 172.21.15.102:0/1107106701 >> 172.21.15.102:0/1107106701 conn(0x7fcee00cf5d0 msgr2=0x7fcee00b1380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:06.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.014+0000 7fcee6a00640 1 -- 172.21.15.102:0/1107106701 shutdown_connections 2024-03-23T03:51:06.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.014+0000 7fcee6a00640 1 -- 172.21.15.102:0/1107106701 wait complete. 2024-03-23T03:51:06.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:51:06.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.242+0000 7f76abe00640 1 Processor -- start 2024-03-23T03:51:06.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.242+0000 7f76abe00640 1 -- start start 2024-03-23T03:51:06.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.243+0000 7f76abe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f76ac0cd270 0x7f76ac098a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:06.244 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.243+0000 7f76abe00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f76ac0cd650 0x7f76ac15d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:06.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.243+0000 7f76abe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76ac098f80 0x7f76ac15ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:06.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.243+0000 7f76abe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76ac099ae0 con 0x7f76ac0cd270 2024-03-23T03:51:06.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.243+0000 7f76abe00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76ac0cb100 con 0x7f76ac0cd650 2024-03-23T03:51:06.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.243+0000 7f76abe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac099520 con 0x7f76ac098f80 2024-03-23T03:51:06.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.243+0000 7f76aaa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f76ac0cd270 0x7f76ac098a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:06.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.243+0000 7f76aaa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f76ac0cd270 0x7f76ac098a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34262/0 (socket says 172.21.15.102:34262) 2024-03-23T03:51:06.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.243+0000 7f76aaa00640 1 -- 172.21.15.102:0/2994259474 learned_addr learned my addr 172.21.15.102:0/2994259474 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:06.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.244+0000 7f76aa000640 1 --2- 172.21.15.102:0/2994259474 >> v2:172.21.15.173:3300/0 conn(0x7f76ac0cd650 0x7f76ac15d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:06.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.244+0000 7f76ab400640 1 --2- 172.21.15.102:0/2994259474 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76ac098f80 0x7f76ac15ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:06.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.244+0000 7f76aaa00640 1 -- 172.21.15.102:0/2994259474 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76ac098f80 msgr2=0x7f76ac15ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:06.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.244+0000 7f76aaa00640 1 --2- 172.21.15.102:0/2994259474 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76ac098f80 0x7f76ac15ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.244+0000 7f76aaa00640 1 -- 172.21.15.102:0/2994259474 >> v2:172.21.15.173:3300/0 conn(0x7f76ac0cd650 msgr2=0x7f76ac15d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:06.245 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.244+0000 7f76aaa00640 1 --2- 172.21.15.102:0/2994259474 >> v2:172.21.15.173:3300/0 conn(0x7f76ac0cd650 0x7f76ac15d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.244+0000 7f76aaa00640 1 -- 172.21.15.102:0/2994259474 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ac099800 con 0x7f76ac0cd270 2024-03-23T03:51:06.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.244+0000 7f76aa000640 1 --2- 172.21.15.102:0/2994259474 >> v2:172.21.15.173:3300/0 conn(0x7f76ac0cd650 0x7f76ac15d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:06.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.245+0000 7f76aaa00640 1 --2- 172.21.15.102:0/2994259474 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76ac0cd270 0x7f76ac098a40 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f76ac0cdac0 tx=0x7f769c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e077d95036ef31e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:06.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.245+0000 7f76a9600640 1 -- 172.21.15.102:0/2994259474 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769c031870 con 0x7f76ac0cd270 2024-03-23T03:51:06.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.245+0000 7f76a9600640 1 -- 172.21.15.102:0/2994259474 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769c0319d0 con 0x7f76ac0cd270 2024-03-23T03:51:06.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.245+0000 7f76a9600640 1 -- 172.21.15.102:0/2994259474 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769c031d60 con 0x7f76ac0cd270 2024-03-23T03:51:06.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.246+0000 7f76abe00640 1 -- 172.21.15.102:0/2994259474 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76ac0cd270 msgr2=0x7f76ac098a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:06.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.246+0000 7f76abe00640 1 --2- 172.21.15.102:0/2994259474 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76ac0cd270 0x7f76ac098a40 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f76ac0cdac0 tx=0x7f769c02b910 comp rx=0 tx=0).stop 2024-03-23T03:51:06.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.246+0000 7f76abe00640 1 -- 172.21.15.102:0/2994259474 shutdown_connections 2024-03-23T03:51:06.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.246+0000 7f76abe00640 1 --2- 172.21.15.102:0/2994259474 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76ac098f80 0x7f76ac15ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.246+0000 7f76abe00640 1 --2- 172.21.15.102:0/2994259474 >> v2:172.21.15.173:3300/0 conn(0x7f76ac0cd650 0x7f76ac15d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.246+0000 7f76abe00640 1 --2- 172.21.15.102:0/2994259474 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76ac0cd270 0x7f76ac098a40 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.246+0000 7f76abe00640 1 -- 172.21.15.102:0/2994259474 >> 172.21.15.102:0/2994259474 conn(0x7f76ac0d05f0 msgr2=0x7f76ac0d0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:06.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.246+0000 7f76abe00640 1 -- 172.21.15.102:0/2994259474 shutdown_connections 2024-03-23T03:51:06.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.246+0000 7f76abe00640 1 -- 172.21.15.102:0/2994259474 wait complete. 2024-03-23T03:51:06.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.248+0000 7f76abe00640 1 Processor -- start 2024-03-23T03:51:06.248 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.248+0000 7f76abe00640 1 -- start start 2024-03-23T03:51:06.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.249+0000 7f76abe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76ac098f80 0x7f76ac0b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:06.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.249+0000 7f76abe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76ac0cd270 0x7f76ac150c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:06.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.249+0000 7f76abe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76ac0cd650 0x7f76ac1e45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:06.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.249+0000 7f76abe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac152b70 con 0x7f76ac098f80 2024-03-23T03:51:06.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.249+0000 7f76abe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac152e50 con 0x7f76ac0cd270 2024-03-23T03:51:06.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.249+0000 7f76abe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac0c7840 con 0x7f76ac0cd650 2024-03-23T03:51:06.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.249+0000 7f76aaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76ac098f80 0x7f76ac0b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:06.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.249+0000 7f76aaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76ac098f80 0x7f76ac0b3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34264/0 (socket says 172.21.15.102:34264) 2024-03-23T03:51:06.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.249+0000 7f76aaa00640 1 -- 172.21.15.102:0/2166437790 learned_addr learned my addr 172.21.15.102:0/2166437790 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:06.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.250+0000 7f76aa000640 1 --2- 172.21.15.102:0/2166437790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76ac0cd270 0x7f76ac150c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:06.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.250+0000 7f76aaa00640 1 -- 172.21.15.102:0/2166437790 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76ac0cd650 msgr2=0x7f76ac1e45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:06.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.250+0000 7f76aaa00640 1 --2- 172.21.15.102:0/2166437790 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76ac0cd650 0x7f76ac1e45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.250+0000 7f76aaa00640 1 -- 172.21.15.102:0/2166437790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76ac0cd270 msgr2=0x7f76ac150c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:06.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.250+0000 7f76aaa00640 1 --2- 172.21.15.102:0/2166437790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76ac0cd270 0x7f76ac150c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.250+0000 7f76aaa00640 1 -- 172.21.15.102:0/2166437790 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f769c0094d0 con 0x7f76ac098f80 2024-03-23T03:51:06.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.250+0000 7f76aa000640 1 --2- 172.21.15.102:0/2166437790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76ac0cd270 0x7f76ac150c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:06.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.250+0000 7f76aaa00640 1 --2- 172.21.15.102:0/2166437790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76ac098f80 0x7f76ac0b3b80 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f769c02b950 tx=0x7f769c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:06.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.250+0000 7f76a2a00640 1 -- 172.21.15.102:0/2166437790 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769c031860 con 0x7f76ac098f80 2024-03-23T03:51:06.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.250+0000 7f76a2a00640 1 -- 172.21.15.102:0/2166437790 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769c0443f0 con 0x7f76ac098f80 2024-03-23T03:51:06.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.250+0000 7f76a2a00640 1 -- 172.21.15.102:0/2166437790 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769c0447a0 con 0x7f76ac098f80 2024-03-23T03:51:06.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.250+0000 7f76abe00640 1 -- 172.21.15.102:0/2166437790 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ac0c7b20 con 0x7f76ac098f80 2024-03-23T03:51:06.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.250+0000 7f76abe00640 1 -- 172.21.15.102:0/2166437790 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ac1e4af0 con 0x7f76ac098f80 2024-03-23T03:51:06.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.252+0000 7f76abe00640 1 -- 172.21.15.102:0/2166437790 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76ac0c76a0 con 0x7f76ac098f80 2024-03-23T03:51:06.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.252+0000 7f76a2a00640 1 -- 172.21.15.102:0/2166437790 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f769c04d450 con 0x7f76ac098f80 2024-03-23T03:51:06.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.253+0000 7f76a2a00640 1 --2- 172.21.15.102:0/2166437790 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f768c06ac00 0x7f768c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:06.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.253+0000 7f76a2a00640 1 -- 172.21.15.102:0/2166437790 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f769c02d040 con 0x7f76ac098f80 2024-03-23T03:51:06.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.254+0000 7f76aa000640 1 --2- 172.21.15.102:0/2166437790 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f768c06ac00 0x7f768c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:06.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.254+0000 7f76aa000640 1 --2- 172.21.15.102:0/2166437790 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f768c06ac00 0x7f768c06d0c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f7698006fd0 tx=0x7f7698008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:06.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.258+0000 7f76a2a00640 1 -- 172.21.15.102:0/2166437790 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f76ac0c76a0 con 0x7f76ac098f80 2024-03-23T03:51:06.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.451+0000 7f76abe00640 1 -- 172.21.15.102:0/2166437790 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f76ac0c76a0 con 0x7f76ac098f80 2024-03-23T03:51:06.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.451+0000 7f76a2a00640 1 -- 172.21.15.102:0/2166437790 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f76ac0c76a0 con 0x7f76ac098f80 2024-03-23T03:51:06.452 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:51:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.454+0000 7f76abe00640 1 -- 172.21.15.102:0/2166437790 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f768c06ac00 msgr2=0x7f768c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.454+0000 7f76abe00640 1 --2- 172.21.15.102:0/2166437790 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f768c06ac00 0x7f768c06d0c0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f7698006fd0 tx=0x7f7698008040 comp rx=0 tx=0).stop 2024-03-23T03:51:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.454+0000 7f76abe00640 1 -- 172.21.15.102:0/2166437790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76ac098f80 msgr2=0x7f76ac0b3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.454+0000 7f76abe00640 1 --2- 172.21.15.102:0/2166437790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76ac098f80 0x7f76ac0b3b80 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f769c02b950 tx=0x7f769c0046c0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.454+0000 7f76abe00640 1 -- 172.21.15.102:0/2166437790 shutdown_connections 2024-03-23T03:51:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.454+0000 7f76abe00640 1 --2- 172.21.15.102:0/2166437790 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f768c06ac00 0x7f768c06d0c0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.454+0000 7f76abe00640 1 --2- 172.21.15.102:0/2166437790 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f76ac0cd650 0x7f76ac1e45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.454+0000 7f76abe00640 1 --2- 172.21.15.102:0/2166437790 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76ac0cd270 0x7f76ac150c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.454+0000 7f76abe00640 1 --2- 172.21.15.102:0/2166437790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76ac098f80 0x7f76ac0b3b80 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.454+0000 7f76abe00640 1 -- 172.21.15.102:0/2166437790 >> 172.21.15.102:0/2166437790 conn(0x7f76ac0d05f0 msgr2=0x7f76ac0cc150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:06.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.454+0000 7f76abe00640 1 -- 172.21.15.102:0/2166437790 shutdown_connections 2024-03-23T03:51:06.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.454+0000 7f76abe00640 1 -- 172.21.15.102:0/2166437790 wait complete. 2024-03-23T03:51:06.473 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:51:06.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.683+0000 7f59ade00640 1 Processor -- start 2024-03-23T03:51:06.684 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.683+0000 7f59ade00640 1 -- start start 2024-03-23T03:51:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.684+0000 7f59ade00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f59a80ca7b0 0x7f59a80cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.685+0000 7f59ade00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a8159c10 con 0x7f59a80ce930 2024-03-23T03:51:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.685+0000 7f59ade00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a80cd840 con 0x7f59a8159830 2024-03-23T03:51:06.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.685+0000 7f59ade00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a806de50 con 0x7f59a80ca7b0 2024-03-23T03:51:06.687 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.685+0000 7f59a7400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f59a80ce930 0x7f59a80b18d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60550/0 (socket says 172.21.15.102:60550) 2024-03-23T03:51:06.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.685+0000 7f59a7400640 1 -- 172.21.15.102:0/2725452483 learned_addr learned my addr 172.21.15.102:0/2725452483 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:06.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.685+0000 7f599fe00640 1 --2- 172.21.15.102:0/2725452483 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f59a80ca7b0 0x7f59a80cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:06.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.685+0000 7f59a6000640 1 -- 172.21.15.102:0/2725452483 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401399744 0 0) 0x7f59a8159c10 con 0x7f59a80ce930 2024-03-23T03:51:06.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.686+0000 7f59a6000640 1 -- 172.21.15.102:0/2725452483 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f598c003520 con 0x7f59a80ce930 2024-03-23T03:51:06.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.686+0000 7f59a6000640 1 -- 172.21.15.102:0/2725452483 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1195459886 0 0) 0x7f598c003520 con 0x7f59a80ce930 2024-03-23T03:51:06.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.686+0000 7f59a6000640 1 -- 172.21.15.102:0/2725452483 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f59a80ca7b0 msgr2=0x7f59a80cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:06.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.686+0000 7f59a6000640 1 --2- 172.21.15.102:0/2725452483 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f59a80ca7b0 0x7f59a80cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.686+0000 7f59a6000640 1 -- 172.21.15.102:0/2725452483 >> v1:172.21.15.173:6789/0 conn(0x7f59a8159830 legacy=0x7f59a80cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:06.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.686+0000 7f59a6000640 1 -- 172.21.15.102:0/2725452483 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a80ccc30 con 0x7f59a80ce930 2024-03-23T03:51:06.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.686+0000 7f59a6000640 1 -- 172.21.15.102:0/2725452483 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f59a00019f0 con 0x7f59a80ce930 2024-03-23T03:51:06.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.686+0000 7f59a6000640 1 -- 172.21.15.102:0/2725452483 >> v1:172.21.15.102:6789/0 conn(0x7f59a80ce930 legacy=0x7f59a80b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:06.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.686+0000 7f59a6000640 1 --2- 172.21.15.102:0/2725452483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f598c0039a0 0x7f598c003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:06.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.686+0000 7f59a6000640 1 --2- 172.21.15.102:0/2725452483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f598c004350 0x7f598c004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:06.688 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.686+0000 7f59a6000640 1 --2- 172.21.15.102:0/2725452483 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f59a80ca7b0 0x7f598c005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.686+0000 7f59a6000640 1 -- 172.21.15.102:0/2725452483 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c00db90 con 0x7f598c0039a0 2024-03-23T03:51:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.686+0000 7f59a6000640 1 -- 172.21.15.102:0/2725452483 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c00dcd0 con 0x7f598c004350 2024-03-23T03:51:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.686+0000 7f59a6000640 1 -- 172.21.15.102:0/2725452483 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598c00de90 con 0x7f59a80ca7b0 2024-03-23T03:51:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.686+0000 7f59a6a00640 1 --2- 172.21.15.102:0/2725452483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f598c0039a0 0x7f598c003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.687+0000 7f599fe00640 1 --2- 172.21.15.102:0/2725452483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f598c004350 0x7f598c004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.687+0000 7f59a6a00640 1 -- 172.21.15.102:0/2725452483 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f59a80ca7b0 msgr2=0x7f598c005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.687+0000 7f59a6a00640 1 --2- 172.21.15.102:0/2725452483 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f59a80ca7b0 0x7f598c005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.687+0000 7f59a6a00640 1 -- 172.21.15.102:0/2725452483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f598c004350 msgr2=0x7f598c004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.687+0000 7f59a6a00640 1 --2- 172.21.15.102:0/2725452483 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f598c004350 0x7f598c004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.687+0000 7f59a6a00640 1 -- 172.21.15.102:0/2725452483 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c003520 con 0x7f598c0039a0 2024-03-23T03:51:06.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.688+0000 7f59a6a00640 1 --2- 172.21.15.102:0/2725452483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f598c0039a0 0x7f598c003d80 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f5998001860 tx=0x7f598c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3fc5691273ffeb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:06.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.688+0000 7f59a6000640 1 -- 172.21.15.102:0/2725452483 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998033630 con 0x7f598c0039a0 2024-03-23T03:51:06.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.688+0000 7f59a6000640 1 -- 172.21.15.102:0/2725452483 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59980337b0 con 0x7f598c0039a0 2024-03-23T03:51:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.688+0000 7f59a6000640 1 -- 172.21.15.102:0/2725452483 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998033b60 con 0x7f598c0039a0 2024-03-23T03:51:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.689+0000 7f59ade00640 1 -- 172.21.15.102:0/2725452483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f598c0039a0 msgr2=0x7f598c003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.689+0000 7f59ade00640 1 --2- 172.21.15.102:0/2725452483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f598c0039a0 0x7f598c003d80 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f5998001860 tx=0x7f598c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.689+0000 7f59a7400640 1 -- 172.21.15.102:0/2725452483 reap_dead start 2024-03-23T03:51:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.689+0000 7f59ade00640 1 -- 172.21.15.102:0/2725452483 shutdown_connections 2024-03-23T03:51:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.689+0000 7f59ade00640 1 -- 172.21.15.102:0/2725452483 >> 172.21.15.102:0/2725452483 conn(0x7f59a806a0b0 msgr2=0x7f59a806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.689+0000 7f59ade00640 1 -- 172.21.15.102:0/2725452483 shutdown_connections 2024-03-23T03:51:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.689+0000 7f59ade00640 1 -- 172.21.15.102:0/2725452483 wait complete. 2024-03-23T03:51:06.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.691+0000 7f59ade00640 1 Processor -- start 2024-03-23T03:51:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.691+0000 7f59ade00640 1 -- start start 2024-03-23T03:51:06.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.692+0000 7f59ade00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f598c0039a0 0x7f59a815c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:06.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.692+0000 7f59ade00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f59a814f610 0x7f59a8159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:06.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.692+0000 7f59ade00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a8152580 0x7f59a80707b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:06.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.692+0000 7f59ade00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a80b3410 con 0x7f598c0039a0 2024-03-23T03:51:06.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.692+0000 7f59ade00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a80ad140 con 0x7f59a8152580 2024-03-23T03:51:06.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.692+0000 7f59ade00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a8156bf0 con 0x7f59a814f610 2024-03-23T03:51:06.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.692+0000 7f59a6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f598c0039a0 0x7f59a815c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:06.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.692+0000 7f59a6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f598c0039a0 0x7f59a815c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34286/0 (socket says 172.21.15.102:34286) 2024-03-23T03:51:06.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.692+0000 7f59a6a00640 1 -- 172.21.15.102:0/2170754612 learned_addr learned my addr 172.21.15.102:0/2170754612 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:06.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.692+0000 7f59a7400640 1 --2- 172.21.15.102:0/2170754612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a8152580 0x7f59a80707b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:06.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.692+0000 7f59a6a00640 1 -- 172.21.15.102:0/2170754612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f59a814f610 msgr2=0x7f59a8159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:06.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.693+0000 7f599fe00640 1 --2- 172.21.15.102:0/2170754612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f59a814f610 0x7f59a8159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:06.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.693+0000 7f59a6a00640 1 --2- 172.21.15.102:0/2170754612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f59a814f610 0x7f59a8159ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.693+0000 7f59a6a00640 1 -- 172.21.15.102:0/2170754612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a8152580 msgr2=0x7f59a80707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:06.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.693+0000 7f59a6a00640 1 --2- 172.21.15.102:0/2170754612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a8152580 0x7f59a80707b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.693+0000 7f59a6a00640 1 -- 172.21.15.102:0/2170754612 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a8152310 con 0x7f598c0039a0 2024-03-23T03:51:06.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.693+0000 7f59a7400640 1 --2- 172.21.15.102:0/2170754612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a8152580 0x7f59a80707b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:06.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.693+0000 7f59a6a00640 1 --2- 172.21.15.102:0/2170754612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f598c0039a0 0x7f59a815c8a0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f599802bdf0 tx=0x7f5998001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:06.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.693+0000 7f599f400640 1 -- 172.21.15.102:0/2170754612 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59980035f0 con 0x7f598c0039a0 2024-03-23T03:51:06.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.693+0000 7f599f400640 1 -- 172.21.15.102:0/2170754612 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5998003750 con 0x7f598c0039a0 2024-03-23T03:51:06.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.693+0000 7f599f400640 1 -- 172.21.15.102:0/2170754612 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998003ac0 con 0x7f598c0039a0 2024-03-23T03:51:06.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.693+0000 7f59ade00640 1 -- 172.21.15.102:0/2170754612 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a80ad390 con 0x7f598c0039a0 2024-03-23T03:51:06.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.693+0000 7f59ade00640 1 -- 172.21.15.102:0/2170754612 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a8156880 con 0x7f598c0039a0 2024-03-23T03:51:06.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.694+0000 7f59ade00640 1 -- 172.21.15.102:0/2170754612 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a8156a40 con 0x7f598c0039a0 2024-03-23T03:51:06.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.695+0000 7f599f400640 1 -- 172.21.15.102:0/2170754612 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5998003c20 con 0x7f598c0039a0 2024-03-23T03:51:06.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.697+0000 7f599f400640 1 --2- 172.21.15.102:0/2170754612 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f598406ac30 0x7f598406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:06.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.697+0000 7f599f400640 1 -- 172.21.15.102:0/2170754612 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f59980ab330 con 0x7f598c0039a0 2024-03-23T03:51:06.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.697+0000 7f599fe00640 1 --2- 172.21.15.102:0/2170754612 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f598406ac30 0x7f598406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:06.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.698+0000 7f599fe00640 1 --2- 172.21.15.102:0/2170754612 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f598406ac30 0x7f598406d0f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f59a80b3010 tx=0x7f5990008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:06.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.701+0000 7f599f400640 1 -- 172.21.15.102:0/2170754612 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f59980763d0 con 0x7f598c0039a0 2024-03-23T03:51:06.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.903+0000 7f59ade00640 1 -- 172.21.15.102:0/2170754612 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f59a8073390 con 0x7f598c0039a0 2024-03-23T03:51:06.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.904+0000 7f599f400640 1 -- 172.21.15.102:0/2170754612 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f59980799f0 con 0x7f598c0039a0 2024-03-23T03:51:06.905 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:51:06.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.907+0000 7f59ade00640 1 -- 172.21.15.102:0/2170754612 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f598406ac30 msgr2=0x7f598406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:06.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.907+0000 7f59ade00640 1 --2- 172.21.15.102:0/2170754612 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f598406ac30 0x7f598406d0f0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f59a80b3010 tx=0x7f5990008040 comp rx=0 tx=0).stop 2024-03-23T03:51:06.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.907+0000 7f59ade00640 1 -- 172.21.15.102:0/2170754612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f598c0039a0 msgr2=0x7f59a815c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:06.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.907+0000 7f59ade00640 1 --2- 172.21.15.102:0/2170754612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f598c0039a0 0x7f59a815c8a0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f599802bdf0 tx=0x7f5998001230 comp rx=0 tx=0).stop 2024-03-23T03:51:06.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.908+0000 7f59ade00640 1 -- 172.21.15.102:0/2170754612 shutdown_connections 2024-03-23T03:51:06.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.908+0000 7f59ade00640 1 --2- 172.21.15.102:0/2170754612 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f598406ac30 0x7f598406d0f0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.908+0000 7f59ade00640 1 --2- 172.21.15.102:0/2170754612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a8152580 0x7f59a80707b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.908+0000 7f59ade00640 1 --2- 172.21.15.102:0/2170754612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f59a814f610 0x7f59a8159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.908+0000 7f59ade00640 1 --2- 172.21.15.102:0/2170754612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f598c0039a0 0x7f59a815c8a0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:06.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.908+0000 7f59ade00640 1 -- 172.21.15.102:0/2170754612 >> 172.21.15.102:0/2170754612 conn(0x7f59a806a0b0 msgr2=0x7f59a8155590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:06.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.908+0000 7f59ade00640 1 -- 172.21.15.102:0/2170754612 shutdown_connections 2024-03-23T03:51:06.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:06.908+0000 7f59ade00640 1 -- 172.21.15.102:0/2170754612 wait complete. 2024-03-23T03:51:06.927 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:51:07.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.130+0000 7fb7c2a00640 1 Processor -- start 2024-03-23T03:51:07.130 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.130+0000 7fb7c2a00640 1 -- start start 2024-03-23T03:51:07.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.131+0000 7fb7c2a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb7bc0b39b0 0x7fb7bc0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:07.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.131+0000 7fb7c2a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb7bc15c400 0x7fb7bc15c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:07.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.131+0000 7fb7c2a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7bc06c6f0 con 0x7fb7bc15b560 2024-03-23T03:51:07.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.131+0000 7fb7c2a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7bc0b4370 con 0x7fb7bc15c400 2024-03-23T03:51:07.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.131+0000 7fb7c2a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc06c9d0 con 0x7fb7bc0b39b0 2024-03-23T03:51:07.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.132+0000 7fb7c0a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb7bc15c400 0x7fb7bc15c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:07.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.132+0000 7fb7c0a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb7bc15c400 0x7fb7bc15c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34298/0 (socket says 172.21.15.102:34298) 2024-03-23T03:51:07.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.132+0000 7fb7c0a00640 1 -- 172.21.15.102:0/81862718 learned_addr learned my addr 172.21.15.102:0/81862718 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:07.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.132+0000 7fb7b3400640 1 --2- 172.21.15.102:0/81862718 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb7bc0b39b0 0x7fb7bc0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:07.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.132+0000 7fb7c0a00640 1 -- 172.21.15.102:0/81862718 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb7bc0b39b0 msgr2=0x7fb7bc0b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:07.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.132+0000 7fb7c0a00640 1 --2- 172.21.15.102:0/81862718 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb7bc0b39b0 0x7fb7bc0b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.133 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.132+0000 7fb7c0a00640 1 -- 172.21.15.102:0/81862718 >> v1:172.21.15.173:6789/0 conn(0x7fb7bc15b560 legacy=0x7fb7bc15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:07.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.132+0000 7fb7c0a00640 1 -- 172.21.15.102:0/81862718 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7bc06ccb0 con 0x7fb7bc15c400 2024-03-23T03:51:07.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.133+0000 7fb7c0a00640 1 --2- 172.21.15.102:0/81862718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7bc15c400 0x7fb7bc15c890 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009e50 tx=0x7fb7b402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e35c60a1fe2a902e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:07.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.134+0000 7fb7bb400640 1 -- 172.21.15.102:0/81862718 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b402ca70 con 0x7fb7bc15c400 2024-03-23T03:51:07.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.134+0000 7fb7bb400640 1 -- 172.21.15.102:0/81862718 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b402cbd0 con 0x7fb7bc15c400 2024-03-23T03:51:07.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.134+0000 7fb7bb400640 1 -- 172.21.15.102:0/81862718 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b40343f0 con 0x7fb7bc15c400 2024-03-23T03:51:07.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.134+0000 7fb7c2a00640 1 -- 172.21.15.102:0/81862718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7bc15c400 msgr2=0x7fb7bc15c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:07.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.134+0000 7fb7c2a00640 1 --2- 172.21.15.102:0/81862718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7bc15c400 0x7fb7bc15c890 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009e50 tx=0x7fb7b402bed0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.135+0000 7fb7c2a00640 1 -- 172.21.15.102:0/81862718 shutdown_connections 2024-03-23T03:51:07.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.135+0000 7fb7c2a00640 1 --2- 172.21.15.102:0/81862718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7bc15c400 0x7fb7bc15c890 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.135+0000 7fb7c2a00640 1 --2- 172.21.15.102:0/81862718 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb7bc0b39b0 0x7fb7bc0b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.135+0000 7fb7c2a00640 1 -- 172.21.15.102:0/81862718 >> 172.21.15.102:0/81862718 conn(0x7fb7bc06a3d0 msgr2=0x7fb7bc0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:07.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.135+0000 7fb7c2a00640 1 -- 172.21.15.102:0/81862718 shutdown_connections 2024-03-23T03:51:07.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.135+0000 7fb7c2a00640 1 -- 172.21.15.102:0/81862718 wait complete. 2024-03-23T03:51:07.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.136+0000 7fb7c2a00640 1 Processor -- start 2024-03-23T03:51:07.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.136+0000 7fb7c2a00640 1 -- start start 2024-03-23T03:51:07.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.137+0000 7fb7c2a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7bc0b39b0 0x7fb7bc148b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:07.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.137+0000 7fb7c2a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb7bc15b560 0x7fb7bc0afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:07.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.137+0000 7fb7c2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7bc15c400 0x7fb7bc0cc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:07.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.137+0000 7fb7c2a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc153040 con 0x7fb7bc15c400 2024-03-23T03:51:07.139 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.137+0000 7fb7c2a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc1f21b0 con 0x7fb7bc0b39b0 2024-03-23T03:51:07.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.137+0000 7fb7c2a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7bc152430 con 0x7fb7bc15b560 2024-03-23T03:51:07.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.138+0000 7fb7c0a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7bc15c400 0x7fb7bc0cc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:07.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.138+0000 7fb7c0a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7bc15c400 0x7fb7bc0cc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34306/0 (socket says 172.21.15.102:34306) 2024-03-23T03:51:07.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.138+0000 7fb7c0a00640 1 -- 172.21.15.102:0/2255667406 learned_addr learned my addr 172.21.15.102:0/2255667406 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:07.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.138+0000 7fb7bbe00640 1 --2- 172.21.15.102:0/2255667406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7bc0b39b0 0x7fb7bc148b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:07.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.138+0000 7fb7b3400640 1 --2- 172.21.15.102:0/2255667406 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb7bc15b560 0x7fb7bc0afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:07.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.138+0000 7fb7c0a00640 1 -- 172.21.15.102:0/2255667406 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb7bc15b560 msgr2=0x7fb7bc0afd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:07.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.138+0000 7fb7c0a00640 1 --2- 172.21.15.102:0/2255667406 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb7bc15b560 0x7fb7bc0afd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.138+0000 7fb7c0a00640 1 -- 172.21.15.102:0/2255667406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7bc0b39b0 msgr2=0x7fb7bc148b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:07.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.138+0000 7fb7c0a00640 1 --2- 172.21.15.102:0/2255667406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7bc0b39b0 0x7fb7bc148b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.138+0000 7fb7c0a00640 1 -- 172.21.15.102:0/2255667406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b4009b30 con 0x7fb7bc15c400 2024-03-23T03:51:07.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.138+0000 7fb7bbe00640 1 --2- 172.21.15.102:0/2255667406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7bc0b39b0 0x7fb7bc148b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:07.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.138+0000 7fb7b3400640 1 --2- 172.21.15.102:0/2255667406 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb7bc15b560 0x7fb7bc0afd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:07.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.138+0000 7fb7c0a00640 1 --2- 172.21.15.102:0/2255667406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7bc15c400 0x7fb7bc0cc0d0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009e50 tx=0x7fb7b4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:07.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.139+0000 7fb7b8c00640 1 -- 172.21.15.102:0/2255667406 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b4009510 con 0x7fb7bc15c400 2024-03-23T03:51:07.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.139+0000 7fb7b8c00640 1 -- 172.21.15.102:0/2255667406 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7b403e450 con 0x7fb7bc15c400 2024-03-23T03:51:07.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.139+0000 7fb7b8c00640 1 -- 172.21.15.102:0/2255667406 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7b403e800 con 0x7fb7bc15c400 2024-03-23T03:51:07.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.139+0000 7fb7c2a00640 1 -- 172.21.15.102:0/2255667406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7bc152710 con 0x7fb7bc15c400 2024-03-23T03:51:07.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.139+0000 7fb7c2a00640 1 -- 172.21.15.102:0/2255667406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7bc152cd0 con 0x7fb7bc15c400 2024-03-23T03:51:07.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.140+0000 7fb7c2a00640 1 -- 172.21.15.102:0/2255667406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7bc149110 con 0x7fb7bc15c400 2024-03-23T03:51:07.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.141+0000 7fb7b8c00640 1 -- 172.21.15.102:0/2255667406 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb7b403e960 con 0x7fb7bc15c400 2024-03-23T03:51:07.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.142+0000 7fb7b8c00640 1 --2- 172.21.15.102:0/2255667406 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb79806ac30 0x7fb79806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:07.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.142+0000 7fb7b8c00640 1 -- 172.21.15.102:0/2255667406 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb7b40ac280 con 0x7fb7bc15c400 2024-03-23T03:51:07.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.142+0000 7fb7bbe00640 1 --2- 172.21.15.102:0/2255667406 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb79806ac30 0x7fb79806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:07.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.143+0000 7fb7bbe00640 1 --2- 172.21.15.102:0/2255667406 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb79806ac30 0x7fb79806d0f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac003eb0 tx=0x7fb7ac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:07.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.147+0000 7fb7b8c00640 1 -- 172.21.15.102:0/2255667406 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb7bc149110 con 0x7fb7bc15c400 2024-03-23T03:51:07.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.379+0000 7fb7c2a00640 1 -- 172.21.15.102:0/2255667406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7bc149110 con 0x7fb7bc15c400 2024-03-23T03:51:07.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.380+0000 7fb7b8c00640 1 -- 172.21.15.102:0/2255667406 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4745 (secure 0 0 0) 0x7fb7bc149110 con 0x7fb7bc15c400 2024-03-23T03:51:07.380 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:07.381 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/755891320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":755891320},{"type":"v1","addr":"172.21.15.102:6836","nonce":755891320}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":5207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3792558756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3792558756},{"type":"v1","addr":"172.21.15.173:6835","nonce":3792558756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":5590,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1632622692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1632622692},{"type":"v1","addr":"172.21.15.173:6833","nonce":1632622692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:50:49.296593+0000","modified":"2024-03-23T03:50:52.307714+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/3452944670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":3452944670},{"type":"v1","addr":"172.21.15.102:6837","nonce":3452944670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T03:51:07.381 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 48 2024-03-23T03:51:07.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.382+0000 7fb7c2a00640 1 -- 172.21.15.102:0/2255667406 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb79806ac30 msgr2=0x7fb79806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:07.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.382+0000 7fb7c2a00640 1 --2- 172.21.15.102:0/2255667406 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb79806ac30 0x7fb79806d0f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb7ac003eb0 tx=0x7fb7ac009040 comp rx=0 tx=0).stop 2024-03-23T03:51:07.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.382+0000 7fb7c2a00640 1 -- 172.21.15.102:0/2255667406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7bc15c400 msgr2=0x7fb7bc0cc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:07.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.382+0000 7fb7c2a00640 1 --2- 172.21.15.102:0/2255667406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7bc15c400 0x7fb7bc0cc0d0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7fb7b4009e50 tx=0x7fb7b4004a40 comp rx=0 tx=0).stop 2024-03-23T03:51:07.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.382+0000 7fb7c2a00640 1 -- 172.21.15.102:0/2255667406 shutdown_connections 2024-03-23T03:51:07.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.382+0000 7fb7c2a00640 1 --2- 172.21.15.102:0/2255667406 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb79806ac30 0x7fb79806d0f0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.382+0000 7fb7c2a00640 1 --2- 172.21.15.102:0/2255667406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7bc15c400 0x7fb7bc0cc0d0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.382+0000 7fb7c2a00640 1 --2- 172.21.15.102:0/2255667406 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb7bc15b560 0x7fb7bc0afd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.382+0000 7fb7c2a00640 1 --2- 172.21.15.102:0/2255667406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7bc0b39b0 0x7fb7bc148b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.382+0000 7fb7c2a00640 1 -- 172.21.15.102:0/2255667406 >> 172.21.15.102:0/2255667406 conn(0x7fb7bc06a3d0 msgr2=0x7fb7bc15c810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:07.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.382+0000 7fb7c2a00640 1 -- 172.21.15.102:0/2255667406 shutdown_connections 2024-03-23T03:51:07.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.382+0000 7fb7c2a00640 1 -- 172.21.15.102:0/2255667406 wait complete. 2024-03-23T03:51:07.401 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:51:07.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.606+0000 7f03fb800640 1 Processor -- start 2024-03-23T03:51:07.607 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.607+0000 7f03fb800640 1 -- start start 2024-03-23T03:51:07.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.608+0000 7f03fb800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f03f415b560 0x7f03f415b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:07.609 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.608+0000 7f03fb800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f03f40b39b0 0x7f03f40b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:07.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.608+0000 7f03fb800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03f406c6f0 con 0x7f03f415e500 2024-03-23T03:51:07.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.608+0000 7f03fb800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f40b4370 con 0x7f03f40b39b0 2024-03-23T03:51:07.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.608+0000 7f03fb800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03f406c9d0 con 0x7f03f415b560 2024-03-23T03:51:07.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.608+0000 7f03ebe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f03f40b39b0 0x7f03f40b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:07.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.608+0000 7f03ebe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f03f40b39b0 0x7f03f40b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34318/0 (socket says 172.21.15.102:34318) 2024-03-23T03:51:07.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.608+0000 7f03ebe00640 1 -- 172.21.15.102:0/3380048569 learned_addr learned my addr 172.21.15.102:0/3380048569 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:07.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.609+0000 7f03f8e00640 1 --2- 172.21.15.102:0/3380048569 >> v2:172.21.15.173:3300/0 conn(0x7f03f415b560 0x7f03f415b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:07.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.609+0000 7f03ebe00640 1 -- 172.21.15.102:0/3380048569 >> v2:172.21.15.173:3300/0 conn(0x7f03f415b560 msgr2=0x7f03f415b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:07.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.609+0000 7f03ebe00640 1 --2- 172.21.15.102:0/3380048569 >> v2:172.21.15.173:3300/0 conn(0x7f03f415b560 0x7f03f415b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.609+0000 7f03ebe00640 1 -- 172.21.15.102:0/3380048569 >> v1:172.21.15.102:6789/0 conn(0x7f03f415e500 legacy=0x7f03f415e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:07.610 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.609+0000 7f03ebe00640 1 -- 172.21.15.102:0/3380048569 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f406ccb0 con 0x7f03f40b39b0 2024-03-23T03:51:07.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.610+0000 7f03ebe00640 1 --2- 172.21.15.102:0/3380048569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03f40b39b0 0x7f03f40b3e30 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f03f4071470 tx=0x7f03e402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be763db78fc5cfbf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:07.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.610+0000 7f03f3e00640 1 -- 172.21.15.102:0/3380048569 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e4030870 con 0x7f03f40b39b0 2024-03-23T03:51:07.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.611+0000 7f03f3e00640 1 -- 172.21.15.102:0/3380048569 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e40309d0 con 0x7f03f40b39b0 2024-03-23T03:51:07.611 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.611+0000 7f03f3e00640 1 -- 172.21.15.102:0/3380048569 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e4030d60 con 0x7f03f40b39b0 2024-03-23T03:51:07.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.611+0000 7f03fb800640 1 -- 172.21.15.102:0/3380048569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03f40b39b0 msgr2=0x7f03f40b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:07.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.611+0000 7f03fb800640 1 --2- 172.21.15.102:0/3380048569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03f40b39b0 0x7f03f40b3e30 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f03f4071470 tx=0x7f03e402b9b0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.611+0000 7f03fb800640 1 -- 172.21.15.102:0/3380048569 shutdown_connections 2024-03-23T03:51:07.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.611+0000 7f03fb800640 1 --2- 172.21.15.102:0/3380048569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03f40b39b0 0x7f03f40b3e30 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.611+0000 7f03fb800640 1 --2- 172.21.15.102:0/3380048569 >> v2:172.21.15.173:3300/0 conn(0x7f03f415b560 0x7f03f415b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.611+0000 7f03fb800640 1 -- 172.21.15.102:0/3380048569 >> 172.21.15.102:0/3380048569 conn(0x7f03f406a3d0 msgr2=0x7f03f40b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:07.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.611+0000 7f03fb800640 1 -- 172.21.15.102:0/3380048569 shutdown_connections 2024-03-23T03:51:07.612 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.611+0000 7f03fb800640 1 -- 172.21.15.102:0/3380048569 wait complete. 2024-03-23T03:51:07.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.613+0000 7f03fb800640 1 Processor -- start 2024-03-23T03:51:07.613 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.613+0000 7f03fb800640 1 -- start start 2024-03-23T03:51:07.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.614+0000 7f03fb800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03f40b39b0 0x7f03f41f30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:07.615 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.614+0000 7f03fb800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03f415b560 0x7f03f40b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.614+0000 7f03fb800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f03f415e500 0x7f03f40ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.614+0000 7f03fb800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f4155890 con 0x7f03f415b560 2024-03-23T03:51:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.614+0000 7f03fb800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03f41f2c10 con 0x7f03f40b39b0 2024-03-23T03:51:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.614+0000 7f03fb800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f40b05f0 con 0x7f03f415e500 2024-03-23T03:51:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.614+0000 7f03ebe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03f415b560 0x7f03f40b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.614+0000 7f03ebe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03f415b560 0x7f03f40b00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34332/0 (socket says 172.21.15.102:34332) 2024-03-23T03:51:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.614+0000 7f03ebe00640 1 -- 172.21.15.102:0/4228961004 learned_addr learned my addr 172.21.15.102:0/4228961004 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.615+0000 7f03f9800640 1 --2- 172.21.15.102:0/4228961004 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f03f415e500 0x7f03f40ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.615+0000 7f03f8e00640 1 --2- 172.21.15.102:0/4228961004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03f40b39b0 0x7f03f41f30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.615+0000 7f03ebe00640 1 -- 172.21.15.102:0/4228961004 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f03f415e500 msgr2=0x7f03f40ace30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.615+0000 7f03ebe00640 1 --2- 172.21.15.102:0/4228961004 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f03f415e500 0x7f03f40ace30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.615+0000 7f03ebe00640 1 -- 172.21.15.102:0/4228961004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03f40b39b0 msgr2=0x7f03f41f30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:07.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.615+0000 7f03ebe00640 1 --2- 172.21.15.102:0/4228961004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03f40b39b0 0x7f03f41f30a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.615+0000 7f03ebe00640 1 -- 172.21.15.102:0/4228961004 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e40094d0 con 0x7f03f415b560 2024-03-23T03:51:07.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.615+0000 7f03f8e00640 1 --2- 172.21.15.102:0/4228961004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03f40b39b0 0x7f03f41f30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:07.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.615+0000 7f03ebe00640 1 --2- 172.21.15.102:0/4228961004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03f415b560 0x7f03f40b00b0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f03e402b9f0 tx=0x7f03e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:07.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.615+0000 7f03f1600640 1 -- 172.21.15.102:0/4228961004 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e4039d60 con 0x7f03f415b560 2024-03-23T03:51:07.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.615+0000 7f03f1600640 1 -- 172.21.15.102:0/4228961004 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03e4043400 con 0x7f03f415b560 2024-03-23T03:51:07.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.616+0000 7f03f1600640 1 -- 172.21.15.102:0/4228961004 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03e40437b0 con 0x7f03f415b560 2024-03-23T03:51:07.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.616+0000 7f03fb800640 1 -- 172.21.15.102:0/4228961004 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03f41f6680 con 0x7f03f415b560 2024-03-23T03:51:07.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.616+0000 7f03fb800640 1 -- 172.21.15.102:0/4228961004 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03f41f25c0 con 0x7f03f415b560 2024-03-23T03:51:07.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.617+0000 7f03fb800640 1 -- 172.21.15.102:0/4228961004 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03f41f2da0 con 0x7f03f415b560 2024-03-23T03:51:07.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.618+0000 7f03f1600640 1 -- 172.21.15.102:0/4228961004 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f03e4043910 con 0x7f03f415b560 2024-03-23T03:51:07.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.619+0000 7f03f1600640 1 --2- 172.21.15.102:0/4228961004 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f03d006ac30 0x7f03d006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:07.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.619+0000 7f03f1600640 1 -- 172.21.15.102:0/4228961004 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f03e403e070 con 0x7f03f415b560 2024-03-23T03:51:07.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.619+0000 7f03f8e00640 1 --2- 172.21.15.102:0/4228961004 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f03d006ac30 0x7f03d006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:07.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.620+0000 7f03f8e00640 1 --2- 172.21.15.102:0/4228961004 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f03d006ac30 0x7f03d006d0f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f03dc006fd0 tx=0x7f03dc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:07.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.624+0000 7f03f1600640 1 -- 172.21.15.102:0/4228961004 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f03e4077720 con 0x7f03f415b560 2024-03-23T03:51:07.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.817+0000 7f03fb800640 1 -- 172.21.15.102:0/4228961004 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03f406ac20 con 0x7f03f415b560 2024-03-23T03:51:07.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.819+0000 7f03f1600640 1 -- 172.21.15.102:0/4228961004 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16297 (secure 0 0 0) 0x7f03e407ad40 con 0x7f03f415b560 2024-03-23T03:51:07.820 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:07.820 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":93,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:50.300471+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:50:47.750670+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-23T03:50:48.305583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:51:07.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.822+0000 7f03fb800640 1 -- 172.21.15.102:0/4228961004 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f03d006ac30 msgr2=0x7f03d006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:07.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.822+0000 7f03fb800640 1 --2- 172.21.15.102:0/4228961004 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f03d006ac30 0x7f03d006d0f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f03dc006fd0 tx=0x7f03dc008040 comp rx=0 tx=0).stop 2024-03-23T03:51:07.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.822+0000 7f03fb800640 1 -- 172.21.15.102:0/4228961004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03f415b560 msgr2=0x7f03f40b00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:07.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.822+0000 7f03fb800640 1 --2- 172.21.15.102:0/4228961004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03f415b560 0x7f03f40b00b0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f03e402b9f0 tx=0x7f03e4004730 comp rx=0 tx=0).stop 2024-03-23T03:51:07.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.822+0000 7f03fb800640 1 -- 172.21.15.102:0/4228961004 shutdown_connections 2024-03-23T03:51:07.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.822+0000 7f03fb800640 1 --2- 172.21.15.102:0/4228961004 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f03d006ac30 0x7f03d006d0f0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.822+0000 7f03fb800640 1 --2- 172.21.15.102:0/4228961004 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f03f415e500 0x7f03f40ace30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.822+0000 7f03fb800640 1 --2- 172.21.15.102:0/4228961004 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f03f415b560 0x7f03f40b00b0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.822+0000 7f03fb800640 1 --2- 172.21.15.102:0/4228961004 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f03f40b39b0 0x7f03f41f30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:07.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.822+0000 7f03fb800640 1 -- 172.21.15.102:0/4228961004 >> 172.21.15.102:0/4228961004 conn(0x7f03f406a3d0 msgr2=0x7f03f4159d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:07.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.822+0000 7f03fb800640 1 -- 172.21.15.102:0/4228961004 shutdown_connections 2024-03-23T03:51:07.823 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:07.822+0000 7f03fb800640 1 -- 172.21.15.102:0/4228961004 wait complete. 2024-03-23T03:51:07.842 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T03:51:07.842 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:51:08.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.045+0000 7f3807e00640 1 Processor -- start 2024-03-23T03:51:08.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.045+0000 7f3807e00640 1 -- start start 2024-03-23T03:51:08.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.046+0000 7f3807e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f38080cb560 0x7f38080b35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:08.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.046+0000 7f3807e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38081575b0 con 0x7f38081571d0 2024-03-23T03:51:08.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.046+0000 7f3807e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3808159f20 con 0x7f38080cdc00 2024-03-23T03:51:08.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.046+0000 7f3807e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f380815a200 con 0x7f38080cb560 2024-03-23T03:51:08.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.047+0000 7f3806a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f38081571d0 0x7f380815aaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60564/0 (socket says 172.21.15.102:60564) 2024-03-23T03:51:08.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.047+0000 7f3806a00640 1 -- 172.21.15.102:0/9964258 learned_addr learned my addr 172.21.15.102:0/9964258 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:08.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.047+0000 7f3807400640 1 --2- 172.21.15.102:0/9964258 >> v2:172.21.15.102:3300/0 conn(0x7f38080cb560 0x7f38080b35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:08.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.047+0000 7f3805600640 1 -- 172.21.15.102:0/9964258 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632803692 0 0) 0x7f38081575b0 con 0x7f38081571d0 2024-03-23T03:51:08.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.047+0000 7f3805600640 1 -- 172.21.15.102:0/9964258 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ec003520 con 0x7f38081571d0 2024-03-23T03:51:08.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.047+0000 7f3807400640 1 -- 172.21.15.102:0/9964258 >> v1:172.21.15.173:6789/0 conn(0x7f38080cdc00 legacy=0x7f38080cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:08.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.047+0000 7f3807400640 1 -- 172.21.15.102:0/9964258 >> v1:172.21.15.102:6789/0 conn(0x7f38081571d0 legacy=0x7f380815aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:08.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.047+0000 7f3807400640 1 -- 172.21.15.102:0/9964258 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f380815a4e0 con 0x7f38080cb560 2024-03-23T03:51:08.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.048+0000 7f3807400640 1 --2- 172.21.15.102:0/9964258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080cb560 0x7f38080b35d0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f3800009d90 tx=0x7f380002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef279e8ae798bca1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:08.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.049+0000 7f3805600640 1 -- 172.21.15.102:0/9964258 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380002ccf0 con 0x7f38080cb560 2024-03-23T03:51:08.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.049+0000 7f3805600640 1 -- 172.21.15.102:0/9964258 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380002ce50 con 0x7f38080cb560 2024-03-23T03:51:08.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.049+0000 7f3805600640 1 -- 172.21.15.102:0/9964258 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3800034660 con 0x7f38080cb560 2024-03-23T03:51:08.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.049+0000 7f3807e00640 1 -- 172.21.15.102:0/9964258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080cb560 msgr2=0x7f38080b35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:08.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.049+0000 7f3807e00640 1 --2- 172.21.15.102:0/9964258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080cb560 0x7f38080b35d0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f3800009d90 tx=0x7f380002bf40 comp rx=0 tx=0).stop 2024-03-23T03:51:08.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.049+0000 7f3807e00640 1 -- 172.21.15.102:0/9964258 shutdown_connections 2024-03-23T03:51:08.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.049+0000 7f3807e00640 1 --2- 172.21.15.102:0/9964258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080cb560 0x7f38080b35d0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.049+0000 7f3807e00640 1 -- 172.21.15.102:0/9964258 >> 172.21.15.102:0/9964258 conn(0x7f3808065df0 msgr2=0x7f3808069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:08.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.049+0000 7f3807e00640 1 -- 172.21.15.102:0/9964258 shutdown_connections 2024-03-23T03:51:08.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.050+0000 7f3807e00640 1 -- 172.21.15.102:0/9964258 wait complete. 2024-03-23T03:51:08.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.051+0000 7f3807e00640 1 Processor -- start 2024-03-23T03:51:08.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.051+0000 7f3807e00640 1 -- start start 2024-03-23T03:51:08.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.052+0000 7f3807e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38080cb560 0x7f3808151a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:08.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.052+0000 7f3807e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080cdc00 0x7f3808155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:08.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.052+0000 7f3807e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38081571d0 0x7f38080ae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:08.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.052+0000 7f3807e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38081565e0 con 0x7f38080cdc00 2024-03-23T03:51:08.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.052+0000 7f3807e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3808156020 con 0x7f38080cb560 2024-03-23T03:51:08.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.052+0000 7f3807e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3808156300 con 0x7f38081571d0 2024-03-23T03:51:08.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.053+0000 7f3806000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080cdc00 0x7f3808155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:08.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.053+0000 7f3806000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080cdc00 0x7f3808155600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34344/0 (socket says 172.21.15.102:34344) 2024-03-23T03:51:08.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.053+0000 7f3806000640 1 -- 172.21.15.102:0/3315040280 learned_addr learned my addr 172.21.15.102:0/3315040280 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:08.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.053+0000 7f3806a00640 1 --2- 172.21.15.102:0/3315040280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38080cb560 0x7f3808151a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:08.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.053+0000 7f3807400640 1 --2- 172.21.15.102:0/3315040280 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38081571d0 0x7f38080ae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:08.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.053+0000 7f3806000640 1 -- 172.21.15.102:0/3315040280 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38081571d0 msgr2=0x7f38080ae0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:08.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.053+0000 7f3806000640 1 --2- 172.21.15.102:0/3315040280 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38081571d0 0x7f38080ae0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.053+0000 7f3806000640 1 -- 172.21.15.102:0/3315040280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38080cb560 msgr2=0x7f3808151a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:08.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.053+0000 7f3806000640 1 --2- 172.21.15.102:0/3315040280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38080cb560 0x7f3808151a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.053+0000 7f3806000640 1 -- 172.21.15.102:0/3315040280 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3800009a70 con 0x7f38080cdc00 2024-03-23T03:51:08.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.053+0000 7f3806000640 1 --2- 172.21.15.102:0/3315040280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080cdc00 0x7f3808155600 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f37f400c450 tx=0x7f37f400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:08.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.054+0000 7f37ff400640 1 -- 172.21.15.102:0/3315040280 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f40052d0 con 0x7f38080cdc00 2024-03-23T03:51:08.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.054+0000 7f37ff400640 1 -- 172.21.15.102:0/3315040280 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37f4005430 con 0x7f38080cdc00 2024-03-23T03:51:08.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.054+0000 7f3807e00640 1 -- 172.21.15.102:0/3315040280 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38080af5f0 con 0x7f38080cdc00 2024-03-23T03:51:08.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.054+0000 7f37ff400640 1 -- 172.21.15.102:0/3315040280 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37f4012870 con 0x7f38080cdc00 2024-03-23T03:51:08.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.054+0000 7f3807e00640 1 -- 172.21.15.102:0/3315040280 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3808151f90 con 0x7f38080cdc00 2024-03-23T03:51:08.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.055+0000 7f3807e00640 1 -- 172.21.15.102:0/3315040280 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38080afb10 con 0x7f38080cdc00 2024-03-23T03:51:08.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.056+0000 7f37ff400640 1 -- 172.21.15.102:0/3315040280 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f37f40129d0 con 0x7f38080cdc00 2024-03-23T03:51:08.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.057+0000 7f37ff400640 1 --2- 172.21.15.102:0/3315040280 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f37e806ac00 0x7f37e806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:08.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.057+0000 7f37ff400640 1 -- 172.21.15.102:0/3315040280 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f37f4088ff0 con 0x7f38080cdc00 2024-03-23T03:51:08.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.057+0000 7f3806a00640 1 --2- 172.21.15.102:0/3315040280 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f37e806ac00 0x7f37e806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:08.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.058+0000 7f3806a00640 1 --2- 172.21.15.102:0/3315040280 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f37e806ac00 0x7f37e806d0c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f37f80011e0 tx=0x7f37f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:08.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.061+0000 7f37ff400640 1 -- 172.21.15.102:0/3315040280 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f37f40bc580 con 0x7f38080cdc00 2024-03-23T03:51:08.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.291+0000 7f3807e00640 1 -- 172.21.15.102:0/3315040280 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38081521b0 con 0x7f38080cdc00 2024-03-23T03:51:08.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.292+0000 7f37ff400640 1 -- 172.21.15.102:0/3315040280 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f38080afb10 con 0x7f38080cdc00 2024-03-23T03:51:08.292 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:08.292 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/755891320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":755891320},{"type":"v1","addr":"172.21.15.102:6836","nonce":755891320}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":5207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3792558756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3792558756},{"type":"v1","addr":"172.21.15.173:6835","nonce":3792558756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":5590,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1632622692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1632622692},{"type":"v1","addr":"172.21.15.173:6833","nonce":1632622692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:50:49.296593+0000","modified":"2024-03-23T03:50:52.307714+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/3452944670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":3452944670},{"type":"v1","addr":"172.21.15.102:6837","nonce":3452944670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T03:51:08.293 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 48 2024-03-23T03:51:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.294+0000 7f3807e00640 1 -- 172.21.15.102:0/3315040280 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f37e806ac00 msgr2=0x7f37e806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.294+0000 7f3807e00640 1 --2- 172.21.15.102:0/3315040280 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f37e806ac00 0x7f37e806d0c0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f37f80011e0 tx=0x7f37f8009040 comp rx=0 tx=0).stop 2024-03-23T03:51:08.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.294+0000 7f3807e00640 1 -- 172.21.15.102:0/3315040280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080cdc00 msgr2=0x7f3808155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:08.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.294+0000 7f3807e00640 1 --2- 172.21.15.102:0/3315040280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080cdc00 0x7f3808155600 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f37f400c450 tx=0x7f37f400c910 comp rx=0 tx=0).stop 2024-03-23T03:51:08.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.295+0000 7f3807e00640 1 -- 172.21.15.102:0/3315040280 shutdown_connections 2024-03-23T03:51:08.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.295+0000 7f3807e00640 1 --2- 172.21.15.102:0/3315040280 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f37e806ac00 0x7f37e806d0c0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.295+0000 7f3807e00640 1 --2- 172.21.15.102:0/3315040280 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38081571d0 0x7f38080ae0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.295+0000 7f3807e00640 1 --2- 172.21.15.102:0/3315040280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080cdc00 0x7f3808155600 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.295+0000 7f3807e00640 1 --2- 172.21.15.102:0/3315040280 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38080cb560 0x7f3808151a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.295+0000 7f3807e00640 1 -- 172.21.15.102:0/3315040280 >> 172.21.15.102:0/3315040280 conn(0x7f3808065df0 msgr2=0x7f380815bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:08.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.295+0000 7f3807e00640 1 -- 172.21.15.102:0/3315040280 shutdown_connections 2024-03-23T03:51:08.296 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.295+0000 7f3807e00640 1 -- 172.21.15.102:0/3315040280 wait complete. 2024-03-23T03:51:08.315 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:51:08.521 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.521+0000 7f82d0a00640 1 Processor -- start 2024-03-23T03:51:08.522 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.521+0000 7f82d0a00640 1 -- start start 2024-03-23T03:51:08.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.522+0000 7f82d0a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82cc0b39b0 0x7f82cc0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:08.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.522+0000 7f82d0a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82cc06c6f0 con 0x7f82cc15b560 2024-03-23T03:51:08.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.522+0000 7f82d0a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82cc06c9d0 con 0x7f82cc15c400 2024-03-23T03:51:08.527 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.522+0000 7f82d0a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc0b4370 con 0x7f82cc0b39b0 2024-03-23T03:51:08.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.523+0000 7f82c9e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f82cc15b560 0x7f82cc15b940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60570/0 (socket says 172.21.15.102:60570) 2024-03-23T03:51:08.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.523+0000 7f82c9e00640 1 -- 172.21.15.102:0/3158975384 learned_addr learned my addr 172.21.15.102:0/3158975384 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:08.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.523+0000 7f82c3e00640 1 --2- 172.21.15.102:0/3158975384 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82cc0b39b0 0x7f82cc0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:08.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.523+0000 7f82c9400640 1 -- 172.21.15.102:0/3158975384 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323934472 0 0) 0x7f82cc06c6f0 con 0x7f82cc15b560 2024-03-23T03:51:08.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.523+0000 7f82c9400640 1 -- 172.21.15.102:0/3158975384 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b0003520 con 0x7f82cc15b560 2024-03-23T03:51:08.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.523+0000 7f82c9400640 1 -- 172.21.15.102:0/3158975384 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1656694757 0 0) 0x7f82b0003520 con 0x7f82cc15b560 2024-03-23T03:51:08.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.523+0000 7f82c9400640 1 -- 172.21.15.102:0/3158975384 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82cc0b39b0 msgr2=0x7f82cc0b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:08.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.523+0000 7f82c9400640 1 --2- 172.21.15.102:0/3158975384 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82cc0b39b0 0x7f82cc0b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.523+0000 7f82c9400640 1 -- 172.21.15.102:0/3158975384 >> v1:172.21.15.173:6789/0 conn(0x7f82cc15c400 legacy=0x7f82cc15c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:08.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.523+0000 7f82c9400640 1 -- 172.21.15.102:0/3158975384 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82cc06ccb0 con 0x7f82cc15b560 2024-03-23T03:51:08.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.523+0000 7f82c9400640 1 -- 172.21.15.102:0/3158975384 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f82bc001670 con 0x7f82cc15b560 2024-03-23T03:51:08.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.523+0000 7f82c9400640 1 -- 172.21.15.102:0/3158975384 >> v1:172.21.15.102:6789/0 conn(0x7f82cc15b560 legacy=0x7f82cc15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:08.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.524+0000 7f82c9400640 1 --2- 172.21.15.102:0/3158975384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82b00039a0 0x7f82b0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:08.528 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.524+0000 7f82c9400640 1 --2- 172.21.15.102:0/3158975384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b00047c0 0x7f82b0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:08.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.524+0000 7f82c9400640 1 --2- 172.21.15.102:0/3158975384 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82cc0b39b0 0x7f82b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:08.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.524+0000 7f82c9400640 1 -- 172.21.15.102:0/3158975384 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc0b4370 con 0x7f82b00047c0 2024-03-23T03:51:08.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.524+0000 7f82c9400640 1 -- 172.21.15.102:0/3158975384 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b000bab0 con 0x7f82b00039a0 2024-03-23T03:51:08.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.524+0000 7f82c9400640 1 -- 172.21.15.102:0/3158975384 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b000bc70 con 0x7f82cc0b39b0 2024-03-23T03:51:08.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.524+0000 7f82c3e00640 1 --2- 172.21.15.102:0/3158975384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b00047c0 0x7f82b0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:08.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.524+0000 7f82c9e00640 1 --2- 172.21.15.102:0/3158975384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82b00039a0 0x7f82b0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:08.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.524+0000 7f82c3e00640 1 -- 172.21.15.102:0/3158975384 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82cc0b39b0 msgr2=0x7f82b00051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:08.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.524+0000 7f82c3e00640 1 --2- 172.21.15.102:0/3158975384 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82cc0b39b0 0x7f82b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.524+0000 7f82c3e00640 1 -- 172.21.15.102:0/3158975384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82b00039a0 msgr2=0x7f82b0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:08.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.524+0000 7f82c3e00640 1 --2- 172.21.15.102:0/3158975384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82b00039a0 0x7f82b0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.524+0000 7f82c3e00640 1 -- 172.21.15.102:0/3158975384 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b0003520 con 0x7f82b00047c0 2024-03-23T03:51:08.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.524+0000 7f82c9e00640 1 --2- 172.21.15.102:0/3158975384 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82b00039a0 0x7f82b0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:08.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.525+0000 7f82c3e00640 1 --2- 172.21.15.102:0/3158975384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b00047c0 0x7f82b0004c40 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f82b400ba30 tx=0x7f82b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2dfc6e817a53b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:08.529 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.525+0000 7f82c9400640 1 -- 172.21.15.102:0/3158975384 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b4032680 con 0x7f82b00047c0 2024-03-23T03:51:08.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.526+0000 7f82c9400640 1 -- 172.21.15.102:0/3158975384 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b40327e0 con 0x7f82b00047c0 2024-03-23T03:51:08.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.526+0000 7f82c9400640 1 -- 172.21.15.102:0/3158975384 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b4032b70 con 0x7f82b00047c0 2024-03-23T03:51:08.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.526+0000 7f82d0a00640 1 -- 172.21.15.102:0/3158975384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b00047c0 msgr2=0x7f82b0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:08.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.526+0000 7f82d0a00640 1 --2- 172.21.15.102:0/3158975384 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b00047c0 0x7f82b0004c40 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f82b400ba30 tx=0x7f82b00057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.526+0000 7f82ca800640 1 -- 172.21.15.102:0/3158975384 reap_dead start 2024-03-23T03:51:08.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.526+0000 7f82d0a00640 1 -- 172.21.15.102:0/3158975384 shutdown_connections 2024-03-23T03:51:08.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.526+0000 7f82d0a00640 1 -- 172.21.15.102:0/3158975384 >> 172.21.15.102:0/3158975384 conn(0x7f82cc06a3d0 msgr2=0x7f82cc0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:08.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.526+0000 7f82d0a00640 1 -- 172.21.15.102:0/3158975384 shutdown_connections 2024-03-23T03:51:08.530 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.526+0000 7f82d0a00640 1 -- 172.21.15.102:0/3158975384 wait complete. 2024-03-23T03:51:08.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.528+0000 7f82d0a00640 1 Processor -- start 2024-03-23T03:51:08.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.528+0000 7f82d0a00640 1 -- start start 2024-03-23T03:51:08.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.529+0000 7f82d0a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b00047c0 0x7f82cc1edaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:08.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.529+0000 7f82d0a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82cc145050 0x7f82cc157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:08.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.529+0000 7f82d0a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82cc0cc630 0x7f82cc0ccac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:08.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.529+0000 7f82d0a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc0cd000 con 0x7f82b00047c0 2024-03-23T03:51:08.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.529+0000 7f82d0a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc152040 con 0x7f82cc0cc630 2024-03-23T03:51:08.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.529+0000 7f82d0a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc1572d0 con 0x7f82cc145050 2024-03-23T03:51:08.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.529+0000 7f82c9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b00047c0 0x7f82cc1edaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:08.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.529+0000 7f82c9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b00047c0 0x7f82cc1edaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34348/0 (socket says 172.21.15.102:34348) 2024-03-23T03:51:08.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.529+0000 7f82c9e00640 1 -- 172.21.15.102:0/1120643456 learned_addr learned my addr 172.21.15.102:0/1120643456 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:08.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.530+0000 7f82ca800640 1 --2- 172.21.15.102:0/1120643456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82cc0cc630 0x7f82cc0ccac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:08.532 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.530+0000 7f82c9e00640 1 -- 172.21.15.102:0/1120643456 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82cc145050 msgr2=0x7f82cc157c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:08.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.530+0000 7f82c9e00640 1 --2- 172.21.15.102:0/1120643456 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82cc145050 0x7f82cc157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.530+0000 7f82c9e00640 1 -- 172.21.15.102:0/1120643456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82cc0cc630 msgr2=0x7f82cc0ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:08.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.530+0000 7f82c9e00640 1 --2- 172.21.15.102:0/1120643456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82cc0cc630 0x7f82cc0ccac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.530+0000 7f82c9e00640 1 -- 172.21.15.102:0/1120643456 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82cc1575b0 con 0x7f82b00047c0 2024-03-23T03:51:08.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.530+0000 7f82c9e00640 1 --2- 172.21.15.102:0/1120643456 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b00047c0 0x7f82cc1edaa0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f82bc00d8e0 tx=0x7f82bc00dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:08.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.530+0000 7f82c2000640 1 -- 172.21.15.102:0/1120643456 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc0074e0 con 0x7f82b00047c0 2024-03-23T03:51:08.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.530+0000 7f82c2000640 1 -- 172.21.15.102:0/1120643456 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82bc007640 con 0x7f82b00047c0 2024-03-23T03:51:08.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.530+0000 7f82c2000640 1 -- 172.21.15.102:0/1120643456 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc0049e0 con 0x7f82b00047c0 2024-03-23T03:51:08.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.530+0000 7f82d0a00640 1 -- 172.21.15.102:0/1120643456 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82cc1522e0 con 0x7f82b00047c0 2024-03-23T03:51:08.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.531+0000 7f82d0a00640 1 -- 172.21.15.102:0/1120643456 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82cc156f50 con 0x7f82b00047c0 2024-03-23T03:51:08.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.532+0000 7f82d0a00640 1 -- 172.21.15.102:0/1120643456 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82cc157a70 con 0x7f82b00047c0 2024-03-23T03:51:08.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.533+0000 7f82c2000640 1 -- 172.21.15.102:0/1120643456 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f82bc004b40 con 0x7f82b00047c0 2024-03-23T03:51:08.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.533+0000 7f82c2000640 1 --2- 172.21.15.102:0/1120643456 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82a806ac00 0x7f82a806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:08.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.534+0000 7f82c2000640 1 -- 172.21.15.102:0/1120643456 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f82bc088ec0 con 0x7f82b00047c0 2024-03-23T03:51:08.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.534+0000 7f82c3e00640 1 --2- 172.21.15.102:0/1120643456 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82a806ac00 0x7f82a806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:08.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.534+0000 7f82c3e00640 1 --2- 172.21.15.102:0/1120643456 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82a806ac00 0x7f82a806d0c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f82b400ba00 tx=0x7f82b4032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:08.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.538+0000 7f82c2000640 1 -- 172.21.15.102:0/1120643456 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f82bc057a90 con 0x7f82b00047c0 2024-03-23T03:51:08.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.770+0000 7f82d0a00640 1 -- 172.21.15.102:0/1120643456 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82cc06ac20 con 0x7f82b00047c0 2024-03-23T03:51:08.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.773+0000 7f82c2000640 1 -- 172.21.15.102:0/1120643456 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f82bc019020 con 0x7f82b00047c0 2024-03-23T03:51:08.774 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:08.774 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":48,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/755891320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":755891320},{"type":"v1","addr":"172.21.15.102:6836","nonce":755891320}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":5207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3792558756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3792558756},{"type":"v1","addr":"172.21.15.173:6835","nonce":3792558756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":5590,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1632622692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1632622692},{"type":"v1","addr":"172.21.15.173:6833","nonce":1632622692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:50:49.296593+0000","modified":"2024-03-23T03:50:52.307714+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6828},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6828":{"gid":6828,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/3452944670","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":3452944670},{"type":"v1","addr":"172.21.15.102:6837","nonce":3452944670}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-03-23T03:51:08.774 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 48 2024-03-23T03:51:08.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.776+0000 7f82d0a00640 1 -- 172.21.15.102:0/1120643456 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82a806ac00 msgr2=0x7f82a806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:08.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.776+0000 7f82d0a00640 1 --2- 172.21.15.102:0/1120643456 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82a806ac00 0x7f82a806d0c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f82b400ba00 tx=0x7f82b4032000 comp rx=0 tx=0).stop 2024-03-23T03:51:08.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.776+0000 7f82d0a00640 1 -- 172.21.15.102:0/1120643456 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b00047c0 msgr2=0x7f82cc1edaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:08.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.776+0000 7f82d0a00640 1 --2- 172.21.15.102:0/1120643456 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b00047c0 0x7f82cc1edaa0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f82bc00d8e0 tx=0x7f82bc00dda0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.776+0000 7f82d0a00640 1 -- 172.21.15.102:0/1120643456 shutdown_connections 2024-03-23T03:51:08.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.776+0000 7f82d0a00640 1 --2- 172.21.15.102:0/1120643456 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f82a806ac00 0x7f82a806d0c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.776+0000 7f82d0a00640 1 --2- 172.21.15.102:0/1120643456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82cc0cc630 0x7f82cc0ccac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.776+0000 7f82d0a00640 1 --2- 172.21.15.102:0/1120643456 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f82cc145050 0x7f82cc157c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.776+0000 7f82d0a00640 1 --2- 172.21.15.102:0/1120643456 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82b00047c0 0x7f82cc1edaa0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.777+0000 7f82d0a00640 1 -- 172.21.15.102:0/1120643456 >> 172.21.15.102:0/1120643456 conn(0x7f82cc06a3d0 msgr2=0x7f82cc0cd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:08.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.777+0000 7f82d0a00640 1 -- 172.21.15.102:0/1120643456 shutdown_connections 2024-03-23T03:51:08.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.777+0000 7f82d0a00640 1 -- 172.21.15.102:0/1120643456 wait complete. 2024-03-23T03:51:08.795 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:51:08.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.994+0000 7ff5cb000640 1 Processor -- start 2024-03-23T03:51:08.995 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.994+0000 7ff5cb000640 1 -- start start 2024-03-23T03:51:08.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.995+0000 7ff5cb000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ff5c415b560 0x7ff5c415b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:08.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.995+0000 7ff5cb000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff5c40b39b0 0x7ff5c40b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:08.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.995+0000 7ff5cb000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c406c6f0 con 0x7ff5c415e500 2024-03-23T03:51:08.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.995+0000 7ff5cb000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5c40b4370 con 0x7ff5c415b560 2024-03-23T03:51:08.996 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.995+0000 7ff5cb000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c406c9d0 con 0x7ff5c40b39b0 2024-03-23T03:51:08.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.996+0000 7ff5c9000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff5c415e500 0x7ff5c415e990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60580/0 (socket says 172.21.15.102:60580) 2024-03-23T03:51:08.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.996+0000 7ff5c9000640 1 -- 172.21.15.102:0/1909599486 learned_addr learned my addr 172.21.15.102:0/1909599486 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.996+0000 7ff5c3e00640 1 --2- 172.21.15.102:0/1909599486 >> v2:172.21.15.173:3300/0 conn(0x7ff5c415b560 0x7ff5c415b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.996+0000 7ff5c3400640 1 --2- 172.21.15.102:0/1909599486 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff5c40b39b0 0x7ff5c40b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.996+0000 7ff5c2a00640 1 -- 172.21.15.102:0/1909599486 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726450757 0 0) 0x7ff5c406c6f0 con 0x7ff5c415e500 2024-03-23T03:51:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.996+0000 7ff5c2a00640 1 -- 172.21.15.102:0/1909599486 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5a8003520 con 0x7ff5c415e500 2024-03-23T03:51:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.997+0000 7ff5c2a00640 1 -- 172.21.15.102:0/1909599486 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1879559914 0 0) 0x7ff5a8003520 con 0x7ff5c415e500 2024-03-23T03:51:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.997+0000 7ff5c2a00640 1 -- 172.21.15.102:0/1909599486 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff5c40b39b0 msgr2=0x7ff5c40b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.997+0000 7ff5c2a00640 1 --2- 172.21.15.102:0/1909599486 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff5c40b39b0 0x7ff5c40b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.997+0000 7ff5c2a00640 1 -- 172.21.15.102:0/1909599486 >> v2:172.21.15.173:3300/0 conn(0x7ff5c415b560 msgr2=0x7ff5c415b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.997+0000 7ff5c2a00640 1 --2- 172.21.15.102:0/1909599486 >> v2:172.21.15.173:3300/0 conn(0x7ff5c415b560 0x7ff5c415b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.997+0000 7ff5c2a00640 1 -- 172.21.15.102:0/1909599486 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c406ccb0 con 0x7ff5c415e500 2024-03-23T03:51:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.997+0000 7ff5c2a00640 1 -- 172.21.15.102:0/1909599486 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7ff5bc001670 con 0x7ff5c415e500 2024-03-23T03:51:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.997+0000 7ff5c2a00640 1 -- 172.21.15.102:0/1909599486 >> v1:172.21.15.102:6789/0 conn(0x7ff5c415e500 legacy=0x7ff5c415e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.997+0000 7ff5c2a00640 1 --2- 172.21.15.102:0/1909599486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8003970 0x7ff5a8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:08.999 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.997+0000 7ff5c2a00640 1 --2- 172.21.15.102:0/1909599486 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff5c40b39b0 0x7ff5a80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.997+0000 7ff5c2a00640 1 --2- 172.21.15.102:0/1909599486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5a8004d00 0x7ff5a8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.997+0000 7ff5c2a00640 1 -- 172.21.15.102:0/1909599486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c40b4370 con 0x7ff5a8003970 2024-03-23T03:51:09.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.997+0000 7ff5c2a00640 1 -- 172.21.15.102:0/1909599486 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c406c9d0 con 0x7ff5a8004d00 2024-03-23T03:51:09.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.997+0000 7ff5c2a00640 1 -- 172.21.15.102:0/1909599486 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a800b980 con 0x7ff5c40b39b0 2024-03-23T03:51:09.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.997+0000 7ff5c3e00640 1 --2- 172.21.15.102:0/1909599486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8003970 0x7ff5a8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:09.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.998+0000 7ff5c3e00640 1 -- 172.21.15.102:0/1909599486 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff5c40b39b0 msgr2=0x7ff5a80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:09.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.998+0000 7ff5c3e00640 1 --2- 172.21.15.102:0/1909599486 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff5c40b39b0 0x7ff5a80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.998+0000 7ff5c3e00640 1 -- 172.21.15.102:0/1909599486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5a8004d00 msgr2=0x7ff5a8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:09.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.998+0000 7ff5c3e00640 1 --2- 172.21.15.102:0/1909599486 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5a8004d00 0x7ff5a8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.000 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.998+0000 7ff5c3e00640 1 -- 172.21.15.102:0/1909599486 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a8003520 con 0x7ff5a8003970 2024-03-23T03:51:09.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.999+0000 7ff5c3e00640 1 --2- 172.21.15.102:0/1909599486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8003970 0x7ff5a8003d90 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4009fc0 tx=0x7ff5a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a015cdeb6eb51ca server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:09.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.999+0000 7ff5c2a00640 1 -- 172.21.15.102:0/1909599486 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b402cc60 con 0x7ff5a8003970 2024-03-23T03:51:09.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.999+0000 7ff5c2a00640 1 -- 172.21.15.102:0/1909599486 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b400f040 con 0x7ff5a8003970 2024-03-23T03:51:09.001 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.999+0000 7ff5c2a00640 1 -- 172.21.15.102:0/1909599486 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b4034440 con 0x7ff5a8003970 2024-03-23T03:51:09.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.999+0000 7ff5cb000640 1 -- 172.21.15.102:0/1909599486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8003970 msgr2=0x7ff5a8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:09.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.999+0000 7ff5cb000640 1 --2- 172.21.15.102:0/1909599486 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8003970 0x7ff5a8003d90 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4009fc0 tx=0x7ff5a80057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:08.999+0000 7ff5c9000640 1 -- 172.21.15.102:0/1909599486 reap_dead start 2024-03-23T03:51:09.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.000+0000 7ff5cb000640 1 -- 172.21.15.102:0/1909599486 shutdown_connections 2024-03-23T03:51:09.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.000+0000 7ff5cb000640 1 -- 172.21.15.102:0/1909599486 >> 172.21.15.102:0/1909599486 conn(0x7ff5c406a3d0 msgr2=0x7ff5c40b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:09.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.000+0000 7ff5cb000640 1 -- 172.21.15.102:0/1909599486 shutdown_connections 2024-03-23T03:51:09.002 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.000+0000 7ff5cb000640 1 -- 172.21.15.102:0/1909599486 wait complete. 2024-03-23T03:51:09.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.002+0000 7ff5cb000640 1 Processor -- start 2024-03-23T03:51:09.003 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.002+0000 7ff5cb000640 1 -- start start 2024-03-23T03:51:09.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.003+0000 7ff5cb000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5a8003970 0x7ff5c413f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.003+0000 7ff5cb000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5c4142080 0x7ff5c4142500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.003+0000 7ff5cb000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff5c4159640 0x7ff5c4159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.003+0000 7ff5cb000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c4140030 con 0x7ff5c4142080 2024-03-23T03:51:09.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.003+0000 7ff5cb000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c415a000 con 0x7ff5a8003970 2024-03-23T03:51:09.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.003+0000 7ff5cb000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c413fdb0 con 0x7ff5c4159640 2024-03-23T03:51:09.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.003+0000 7ff5c3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5c4142080 0x7ff5c4142500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:09.004 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.003+0000 7ff5c3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5c4142080 0x7ff5c4142500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34366/0 (socket says 172.21.15.102:34366) 2024-03-23T03:51:09.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.003+0000 7ff5c3400640 1 -- 172.21.15.102:0/1225872362 learned_addr learned my addr 172.21.15.102:0/1225872362 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:09.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.004+0000 7ff5c3400640 1 -- 172.21.15.102:0/1225872362 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff5c4159640 msgr2=0x7ff5c4159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:09.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.004+0000 7ff5c3400640 1 --2- 172.21.15.102:0/1225872362 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff5c4159640 0x7ff5c4159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.004+0000 7ff5c3400640 1 -- 172.21.15.102:0/1225872362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5a8003970 msgr2=0x7ff5c413f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:09.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.004+0000 7ff5c3400640 1 --2- 172.21.15.102:0/1225872362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5a8003970 0x7ff5c413f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.004+0000 7ff5c3400640 1 -- 172.21.15.102:0/1225872362 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c4142b80 con 0x7ff5c4142080 2024-03-23T03:51:09.005 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.004+0000 7ff5c3400640 1 --2- 172.21.15.102:0/1225872362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5c4142080 0x7ff5c4142500 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7ff5b000e7f0 tx=0x7ff5b000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:09.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.004+0000 7ff5bbe00640 1 -- 172.21.15.102:0/1225872362 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b000c8e0 con 0x7ff5c4142080 2024-03-23T03:51:09.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.004+0000 7ff5bbe00640 1 -- 172.21.15.102:0/1225872362 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b000ca40 con 0x7ff5c4142080 2024-03-23T03:51:09.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.004+0000 7ff5bbe00640 1 -- 172.21.15.102:0/1225872362 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b000cdd0 con 0x7ff5c4142080 2024-03-23T03:51:09.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.004+0000 7ff5cb000640 1 -- 172.21.15.102:0/1225872362 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c4142e60 con 0x7ff5c4142080 2024-03-23T03:51:09.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.004+0000 7ff5cb000640 1 -- 172.21.15.102:0/1225872362 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c4140520 con 0x7ff5c4142080 2024-03-23T03:51:09.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.005+0000 7ff5cb000640 1 -- 172.21.15.102:0/1225872362 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c4143050 con 0x7ff5c4142080 2024-03-23T03:51:09.007 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.006+0000 7ff5bbe00640 1 -- 172.21.15.102:0/1225872362 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff5b0013470 con 0x7ff5c4142080 2024-03-23T03:51:09.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.007+0000 7ff5bbe00640 1 --2- 172.21.15.102:0/1225872362 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff5a406ac00 0x7ff5a406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.007+0000 7ff5bbe00640 1 -- 172.21.15.102:0/1225872362 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff5b00895c0 con 0x7ff5c4142080 2024-03-23T03:51:09.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.007+0000 7ff5c3e00640 1 --2- 172.21.15.102:0/1225872362 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff5a406ac00 0x7ff5a406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:09.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.008+0000 7ff5c3e00640 1 --2- 172.21.15.102:0/1225872362 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff5a406ac00 0x7ff5a406d0c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4036000 tx=0x7ff5b402e070 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:09.012 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.012+0000 7ff5bbe00640 1 -- 172.21.15.102:0/1225872362 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff5c4143050 con 0x7ff5c4142080 2024-03-23T03:51:09.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.209+0000 7ff5cb000640 1 -- 172.21.15.102:0/1225872362 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff5c4143050 con 0x7ff5c4142080 2024-03-23T03:51:09.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.211+0000 7ff5bbe00640 1 -- 172.21.15.102:0/1225872362 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+16297 (secure 0 0 0) 0x7ff5c4143050 con 0x7ff5c4142080 2024-03-23T03:51:09.212 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:09.213 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":93,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:50:50.300471+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:50:47.750670+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"92","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-03-23T03:50:48.305583+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"93","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:51:09.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.214+0000 7ff5cb000640 1 -- 172.21.15.102:0/1225872362 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff5a406ac00 msgr2=0x7ff5a406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:09.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.214+0000 7ff5cb000640 1 --2- 172.21.15.102:0/1225872362 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff5a406ac00 0x7ff5a406d0c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff5b4036000 tx=0x7ff5b402e070 comp rx=0 tx=0).stop 2024-03-23T03:51:09.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.214+0000 7ff5cb000640 1 -- 172.21.15.102:0/1225872362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5c4142080 msgr2=0x7ff5c4142500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:09.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.214+0000 7ff5cb000640 1 --2- 172.21.15.102:0/1225872362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5c4142080 0x7ff5c4142500 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7ff5b000e7f0 tx=0x7ff5b000ecb0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.214+0000 7ff5cb000640 1 -- 172.21.15.102:0/1225872362 shutdown_connections 2024-03-23T03:51:09.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.214+0000 7ff5cb000640 1 --2- 172.21.15.102:0/1225872362 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff5a406ac00 0x7ff5a406d0c0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.214+0000 7ff5cb000640 1 --2- 172.21.15.102:0/1225872362 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff5c4159640 0x7ff5c4159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.214+0000 7ff5cb000640 1 --2- 172.21.15.102:0/1225872362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5c4142080 0x7ff5c4142500 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.214+0000 7ff5cb000640 1 --2- 172.21.15.102:0/1225872362 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff5a8003970 0x7ff5c413f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.214+0000 7ff5cb000640 1 -- 172.21.15.102:0/1225872362 >> 172.21.15.102:0/1225872362 conn(0x7ff5c406a3d0 msgr2=0x7ff5c40b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:09.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.214+0000 7ff5cb000640 1 -- 172.21.15.102:0/1225872362 shutdown_connections 2024-03-23T03:51:09.216 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.214+0000 7ff5cb000640 1 -- 172.21.15.102:0/1225872362 wait complete. 2024-03-23T03:51:09.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T03:51:09.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.451+0000 7f8667e00640 1 Processor -- start 2024-03-23T03:51:09.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.451+0000 7f8667e00640 1 -- start start 2024-03-23T03:51:09.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.452+0000 7f8667e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8660159690 0x7f86600ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.452+0000 7f8667e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f86600cc330 0x7f86600b18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.452+0000 7f8667e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f866015e010 con 0x7f86600b1e10 2024-03-23T03:51:09.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.452+0000 7f8667e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86600cc7f0 con 0x7f8660159690 2024-03-23T03:51:09.453 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.452+0000 7f8667e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86600ccad0 con 0x7f86600cc330 2024-03-23T03:51:09.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.453+0000 7f8665400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8660159690 0x7f86600ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:09.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.453+0000 7f8665400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8660159690 0x7f86600ca7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34370/0 (socket says 172.21.15.102:34370) 2024-03-23T03:51:09.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.453+0000 7f8665400640 1 -- 172.21.15.102:0/1734807913 learned_addr learned my addr 172.21.15.102:0/1734807913 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:09.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.453+0000 7f8665e00640 1 --1- 172.21.15.102:0/1734807913 >> v1:172.21.15.102:6789/0 conn(0x7f86600b1e10 0x7f86600cd890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60594/0 (socket says 172.21.15.102:60594) 2024-03-23T03:51:09.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.453+0000 7f8664a00640 1 --2- 172.21.15.102:0/1734807913 >> v2:172.21.15.173:3300/0 conn(0x7f86600cc330 0x7f86600b18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:09.454 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.454+0000 7f8665400640 1 -- 172.21.15.102:0/1734807913 >> v2:172.21.15.173:3300/0 conn(0x7f86600cc330 msgr2=0x7f86600b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:09.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.454+0000 7f8665400640 1 --2- 172.21.15.102:0/1734807913 >> v2:172.21.15.173:3300/0 conn(0x7f86600cc330 0x7f86600b18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.454+0000 7f8665400640 1 -- 172.21.15.102:0/1734807913 >> v1:172.21.15.102:6789/0 conn(0x7f86600b1e10 legacy=0x7f86600cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:09.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.454+0000 7f8665400640 1 -- 172.21.15.102:0/1734807913 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86600ccdb0 con 0x7f8660159690 2024-03-23T03:51:09.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.455+0000 7f8665400640 1 --2- 172.21.15.102:0/1734807913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8660159690 0x7f86600ca7b0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f86600cb520 tx=0x7f865402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb9b00368c1067 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:09.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.455+0000 7f865fe00640 1 -- 172.21.15.102:0/1734807913 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8654030860 con 0x7f8660159690 2024-03-23T03:51:09.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.455+0000 7f865fe00640 1 -- 172.21.15.102:0/1734807913 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86540309c0 con 0x7f8660159690 2024-03-23T03:51:09.456 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.455+0000 7f865fe00640 1 -- 172.21.15.102:0/1734807913 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8654030d50 con 0x7f8660159690 2024-03-23T03:51:09.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.456+0000 7f8667e00640 1 -- 172.21.15.102:0/1734807913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8660159690 msgr2=0x7f86600ca7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:09.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.456+0000 7f8667e00640 1 --2- 172.21.15.102:0/1734807913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8660159690 0x7f86600ca7b0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f86600cb520 tx=0x7f865402b9a0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.456+0000 7f8667e00640 1 -- 172.21.15.102:0/1734807913 shutdown_connections 2024-03-23T03:51:09.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.456+0000 7f8667e00640 1 --2- 172.21.15.102:0/1734807913 >> v2:172.21.15.173:3300/0 conn(0x7f86600cc330 0x7f86600b18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.456+0000 7f8667e00640 1 --2- 172.21.15.102:0/1734807913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8660159690 0x7f86600ca7b0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.456+0000 7f8667e00640 1 -- 172.21.15.102:0/1734807913 >> 172.21.15.102:0/1734807913 conn(0x7f866006a0b0 msgr2=0x7f866006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:09.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.456+0000 7f8667e00640 1 -- 172.21.15.102:0/1734807913 shutdown_connections 2024-03-23T03:51:09.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.456+0000 7f8667e00640 1 -- 172.21.15.102:0/1734807913 wait complete. 2024-03-23T03:51:09.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.458+0000 7f8667e00640 1 Processor -- start 2024-03-23T03:51:09.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.459+0000 7f8667e00640 1 -- start start 2024-03-23T03:51:09.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.460+0000 7f8667e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86600b1e10 0x7f866014ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.460+0000 7f8667e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86600cc330 0x7f8660148b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.460+0000 7f8667e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8660159690 0x7f8660147370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.460+0000 7f8667e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866014b5c0 con 0x7f86600cc330 2024-03-23T03:51:09.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.460+0000 7f8667e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866014a9b0 con 0x7f86600b1e10 2024-03-23T03:51:09.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.460+0000 7f8667e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866014ac90 con 0x7f8660159690 2024-03-23T03:51:09.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.460+0000 7f8664a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86600cc330 0x7f8660148b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:09.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.460+0000 7f8664a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86600cc330 0x7f8660148b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34386/0 (socket says 172.21.15.102:34386) 2024-03-23T03:51:09.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.460+0000 7f8664a00640 1 -- 172.21.15.102:0/1058290581 learned_addr learned my addr 172.21.15.102:0/1058290581 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:09.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.461+0000 7f8665400640 1 --2- 172.21.15.102:0/1058290581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86600b1e10 0x7f866014ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:09.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.461+0000 7f8664a00640 1 -- 172.21.15.102:0/1058290581 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8660159690 msgr2=0x7f8660147370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:09.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.461+0000 7f8664a00640 1 --2- 172.21.15.102:0/1058290581 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8660159690 0x7f8660147370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.461+0000 7f8664a00640 1 -- 172.21.15.102:0/1058290581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86600b1e10 msgr2=0x7f866014ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:09.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.461+0000 7f8664a00640 1 --2- 172.21.15.102:0/1058290581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86600b1e10 0x7f866014ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.461+0000 7f8664a00640 1 -- 172.21.15.102:0/1058290581 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86540094d0 con 0x7f86600cc330 2024-03-23T03:51:09.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.461+0000 7f8664a00640 1 --2- 172.21.15.102:0/1058290581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86600cc330 0x7f8660148b30 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f864c00d800 tx=0x7f864c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:09.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.461+0000 7f865d600640 1 -- 172.21.15.102:0/1058290581 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c00b6c0 con 0x7f86600cc330 2024-03-23T03:51:09.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.461+0000 7f865d600640 1 -- 172.21.15.102:0/1058290581 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f864c00b820 con 0x7f86600cc330 2024-03-23T03:51:09.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.461+0000 7f865d600640 1 -- 172.21.15.102:0/1058290581 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864c00bbb0 con 0x7f86600cc330 2024-03-23T03:51:09.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.461+0000 7f8667e00640 1 -- 172.21.15.102:0/1058290581 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866014af70 con 0x7f86600cc330 2024-03-23T03:51:09.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.462+0000 7f8667e00640 1 -- 172.21.15.102:0/1058290581 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86601483a0 con 0x7f86600cc330 2024-03-23T03:51:09.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.463+0000 7f8667e00640 1 -- 172.21.15.102:0/1058290581 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8660148530 con 0x7f86600cc330 2024-03-23T03:51:09.464 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.463+0000 7f865d600640 1 -- 172.21.15.102:0/1058290581 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f864c00bd10 con 0x7f86600cc330 2024-03-23T03:51:09.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.464+0000 7f865d600640 1 --2- 172.21.15.102:0/1058290581 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f864006ac00 0x7f864006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.464+0000 7f865d600640 1 -- 172.21.15.102:0/1058290581 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f864c089a80 con 0x7f86600cc330 2024-03-23T03:51:09.465 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.465+0000 7f8665400640 1 --2- 172.21.15.102:0/1058290581 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f864006ac00 0x7f864006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:09.466 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.465+0000 7f8665400640 1 --2- 172.21.15.102:0/1058290581 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f864006ac00 0x7f864006d0c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8654036000 tx=0x7f865403b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:09.469 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.469+0000 7f865d600640 1 -- 172.21.15.102:0/1058290581 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f864c054a80 con 0x7f86600cc330 2024-03-23T03:51:09.707 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.707+0000 7f8667e00640 1 -- 172.21.15.102:0/1058290581 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f8660073390 con 0x7f86600cc330 2024-03-23T03:51:09.709 INFO:tasks.ceph.mon.a.smithi102.stderr:2024-03-23T03:51:09.708+0000 7f5aab200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T03:51:09.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.720+0000 7f865d600640 1 -- 172.21.15.102:0/1058290581 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v49) v1 ==== 161+0+0 (secure 0 0 0) 0x7f864c0580a0 con 0x7f86600cc330 2024-03-23T03:51:09.721 INFO:teuthology.orchestra.run.smithi102.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T03:51:09.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.723+0000 7f8667e00640 1 -- 172.21.15.102:0/1058290581 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f864006ac00 msgr2=0x7f864006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:09.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.723+0000 7f8667e00640 1 --2- 172.21.15.102:0/1058290581 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f864006ac00 0x7f864006d0c0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8654036000 tx=0x7f865403b040 comp rx=0 tx=0).stop 2024-03-23T03:51:09.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.723+0000 7f8667e00640 1 -- 172.21.15.102:0/1058290581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86600cc330 msgr2=0x7f8660148b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:09.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.723+0000 7f8667e00640 1 --2- 172.21.15.102:0/1058290581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86600cc330 0x7f8660148b30 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f864c00d800 tx=0x7f864c00dcc0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.723+0000 7f8667e00640 1 -- 172.21.15.102:0/1058290581 shutdown_connections 2024-03-23T03:51:09.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.723+0000 7f8667e00640 1 --2- 172.21.15.102:0/1058290581 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f864006ac00 0x7f864006d0c0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.723+0000 7f8667e00640 1 --2- 172.21.15.102:0/1058290581 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8660159690 0x7f8660147370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.723+0000 7f8667e00640 1 --2- 172.21.15.102:0/1058290581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86600cc330 0x7f8660148b30 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.723+0000 7f8667e00640 1 --2- 172.21.15.102:0/1058290581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86600b1e10 0x7f866014ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.723+0000 7f8667e00640 1 -- 172.21.15.102:0/1058290581 >> 172.21.15.102:0/1058290581 conn(0x7f866006a0b0 msgr2=0x7f86600b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:09.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.723+0000 7f8667e00640 1 -- 172.21.15.102:0/1058290581 shutdown_connections 2024-03-23T03:51:09.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.724+0000 7f8667e00640 1 -- 172.21.15.102:0/1058290581 wait complete. 2024-03-23T03:51:09.732 INFO:tasks.ceph.mds.c.smithi102.stderr: -3295> 2024-03-23T03:50:49.741+0000 7facfca00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T03:51:09.742 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T03:51:09.791 INFO:tasks.ceph.mds.c.smithi102.stdout:starting mds.c at 2024-03-23T03:51:09.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.928+0000 7f1b89c00640 1 Processor -- start 2024-03-23T03:51:09.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.928+0000 7f1b89c00640 1 -- start start 2024-03-23T03:51:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.929+0000 7f1b89c00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1b8415eb80 0x7f1b8406aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.929+0000 7f1b89c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8406f120 con 0x7f1b8406b1e0 2024-03-23T03:51:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.929+0000 7f1b89c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b84071700 con 0x7f1b84154ca0 2024-03-23T03:51:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.929+0000 7f1b89c00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b8406b670 con 0x7f1b8415eb80 2024-03-23T03:51:09.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.930+0000 7f1b89200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1b8406b1e0 0x7f1b8415aab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60602/0 (socket says 172.21.15.102:60602) 2024-03-23T03:51:09.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.930+0000 7f1b89200640 1 -- 172.21.15.102:0/2142054707 learned_addr learned my addr 172.21.15.102:0/2142054707 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:09.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.930+0000 7f1b83400640 1 --2- 172.21.15.102:0/2142054707 >> v2:172.21.15.173:3300/0 conn(0x7f1b8415eb80 0x7f1b8406aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:09.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.930+0000 7f1b82a00640 1 -- 172.21.15.102:0/2142054707 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579238854 0 0) 0x7f1b8406f120 con 0x7f1b8406b1e0 2024-03-23T03:51:09.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.930+0000 7f1b82a00640 1 -- 172.21.15.102:0/2142054707 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b68003520 con 0x7f1b8406b1e0 2024-03-23T03:51:09.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.931+0000 7f1b82a00640 1 -- 172.21.15.102:0/2142054707 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1551046054 0 0) 0x7f1b68003520 con 0x7f1b8406b1e0 2024-03-23T03:51:09.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.931+0000 7f1b82a00640 1 -- 172.21.15.102:0/2142054707 >> v2:172.21.15.173:3300/0 conn(0x7f1b8415eb80 msgr2=0x7f1b8406aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:09.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.931+0000 7f1b82a00640 1 --2- 172.21.15.102:0/2142054707 >> v2:172.21.15.173:3300/0 conn(0x7f1b8415eb80 0x7f1b8406aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.931+0000 7f1b82a00640 1 -- 172.21.15.102:0/2142054707 >> v1:172.21.15.173:6789/0 conn(0x7f1b84154ca0 legacy=0x7f1b840adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:09.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.931+0000 7f1b82a00640 1 -- 172.21.15.102:0/2142054707 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b840719e0 con 0x7f1b8406b1e0 2024-03-23T03:51:09.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.931+0000 7f1b82a00640 1 -- 172.21.15.102:0/2142054707 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f1b7c0019f0 con 0x7f1b8406b1e0 2024-03-23T03:51:09.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.931+0000 7f1b82a00640 1 -- 172.21.15.102:0/2142054707 >> v1:172.21.15.102:6789/0 conn(0x7f1b8406b1e0 legacy=0x7f1b8415aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:09.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.931+0000 7f1b82a00640 1 --2- 172.21.15.102:0/2142054707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b680039c0 0x7f1b68003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.933 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.931+0000 7f1b82a00640 1 --2- 172.21.15.102:0/2142054707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b680047c0 0x7f1b68004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.931+0000 7f1b82a00640 1 --2- 172.21.15.102:0/2142054707 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b68005180 0x7f1b6800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.931+0000 7f1b82a00640 1 -- 172.21.15.102:0/2142054707 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8406b670 con 0x7f1b680039c0 2024-03-23T03:51:09.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.931+0000 7f1b82a00640 1 -- 172.21.15.102:0/2142054707 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6800c080 con 0x7f1b680047c0 2024-03-23T03:51:09.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.931+0000 7f1b82a00640 1 -- 172.21.15.102:0/2142054707 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6800c240 con 0x7f1b68005180 2024-03-23T03:51:09.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.931+0000 7f1b83e00640 1 --2- 172.21.15.102:0/2142054707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b680039c0 0x7f1b68003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:09.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.931+0000 7f1b83e00640 1 -- 172.21.15.102:0/2142054707 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b68005180 msgr2=0x7f1b6800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:09.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.932+0000 7f1b83e00640 1 --2- 172.21.15.102:0/2142054707 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b68005180 0x7f1b6800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.932+0000 7f1b83e00640 1 -- 172.21.15.102:0/2142054707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b680047c0 msgr2=0x7f1b68004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:09.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.932+0000 7f1b83e00640 1 --2- 172.21.15.102:0/2142054707 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b680047c0 0x7f1b68004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.932+0000 7f1b83e00640 1 -- 172.21.15.102:0/2142054707 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b68003520 con 0x7f1b680039c0 2024-03-23T03:51:09.934 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.932+0000 7f1b89200640 1 -- 172.21.15.102:0/2142054707 reap_dead start 2024-03-23T03:51:09.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.933+0000 7f1b83e00640 1 --2- 172.21.15.102:0/2142054707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b680039c0 0x7f1b68003de0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f1b74001750 tx=0x7f1b680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6bbc07c45c7d00f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:09.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.933+0000 7f1b82a00640 1 -- 172.21.15.102:0/2142054707 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b740052d0 con 0x7f1b680039c0 2024-03-23T03:51:09.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.933+0000 7f1b82a00640 1 -- 172.21.15.102:0/2142054707 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7402e050 con 0x7f1b680039c0 2024-03-23T03:51:09.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.933+0000 7f1b82a00640 1 -- 172.21.15.102:0/2142054707 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b74003830 con 0x7f1b680039c0 2024-03-23T03:51:09.936 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.934+0000 7f1b89c00640 1 -- 172.21.15.102:0/2142054707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b680039c0 msgr2=0x7f1b68003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:09.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.934+0000 7f1b89c00640 1 --2- 172.21.15.102:0/2142054707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b680039c0 0x7f1b68003de0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f1b74001750 tx=0x7f1b680057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.934+0000 7f1b89c00640 1 -- 172.21.15.102:0/2142054707 shutdown_connections 2024-03-23T03:51:09.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.934+0000 7f1b89c00640 1 --2- 172.21.15.102:0/2142054707 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b680039c0 0x7f1b68003de0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.934+0000 7f1b89c00640 1 -- 172.21.15.102:0/2142054707 >> 172.21.15.102:0/2142054707 conn(0x7f1b84070800 msgr2=0x7f1b84070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:09.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.934+0000 7f1b89c00640 1 -- 172.21.15.102:0/2142054707 shutdown_connections 2024-03-23T03:51:09.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.934+0000 7f1b89c00640 1 -- 172.21.15.102:0/2142054707 wait complete. 2024-03-23T03:51:09.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.936+0000 7f1b89c00640 1 Processor -- start 2024-03-23T03:51:09.937 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.936+0000 7f1b89c00640 1 -- start start 2024-03-23T03:51:09.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.937+0000 7f1b89c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b680039c0 0x7f1b840af900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.937+0000 7f1b89c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b84150ea0 0x7f1b8413b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.937+0000 7f1b89c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b841468d0 0x7f1b84146d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.937+0000 7f1b89c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b84147290 con 0x7f1b841468d0 2024-03-23T03:51:09.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.937+0000 7f1b89c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b84151660 con 0x7f1b84150ea0 2024-03-23T03:51:09.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.937+0000 7f1b89c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b84151940 con 0x7f1b680039c0 2024-03-23T03:51:09.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.937+0000 7f1b89200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b841468d0 0x7f1b84146d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:09.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.937+0000 7f1b89200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b841468d0 0x7f1b84146d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34424/0 (socket says 172.21.15.102:34424) 2024-03-23T03:51:09.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.937+0000 7f1b89200640 1 -- 172.21.15.102:0/3716143046 learned_addr learned my addr 172.21.15.102:0/3716143046 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:09.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.937+0000 7f1b83e00640 1 --2- 172.21.15.102:0/3716143046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b680039c0 0x7f1b840af900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:09.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.937+0000 7f1b83400640 1 --2- 172.21.15.102:0/3716143046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b84150ea0 0x7f1b8413b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:09.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.937+0000 7f1b89200640 1 -- 172.21.15.102:0/3716143046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b680039c0 msgr2=0x7f1b840af900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:09.940 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.937+0000 7f1b89200640 1 --2- 172.21.15.102:0/3716143046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b680039c0 0x7f1b840af900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.937+0000 7f1b89200640 1 -- 172.21.15.102:0/3716143046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b84150ea0 msgr2=0x7f1b8413b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:09.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.937+0000 7f1b89200640 1 --2- 172.21.15.102:0/3716143046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b84150ea0 0x7f1b8413b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:09.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.937+0000 7f1b89200640 1 -- 172.21.15.102:0/3716143046 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b84151c20 con 0x7f1b841468d0 2024-03-23T03:51:09.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.938+0000 7f1b89200640 1 --2- 172.21.15.102:0/3716143046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b841468d0 0x7f1b84146d50 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f1b8415ef60 tx=0x7f1b7c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:09.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.938+0000 7f1b7be00640 1 -- 172.21.15.102:0/3716143046 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c00ce50 con 0x7f1b841468d0 2024-03-23T03:51:09.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.938+0000 7f1b7be00640 1 -- 172.21.15.102:0/3716143046 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b7c00a610 con 0x7f1b841468d0 2024-03-23T03:51:09.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.938+0000 7f1b7be00640 1 -- 172.21.15.102:0/3716143046 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b7c00a9c0 con 0x7f1b841468d0 2024-03-23T03:51:09.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.938+0000 7f1b89c00640 1 -- 172.21.15.102:0/3716143046 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b84151f00 con 0x7f1b841468d0 2024-03-23T03:51:09.941 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.938+0000 7f1b89c00640 1 -- 172.21.15.102:0/3716143046 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b841512e0 con 0x7f1b841468d0 2024-03-23T03:51:09.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.939+0000 7f1b89c00640 1 -- 172.21.15.102:0/3716143046 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b84152420 con 0x7f1b841468d0 2024-03-23T03:51:09.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.940+0000 7f1b7be00640 1 -- 172.21.15.102:0/3716143046 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1b7c00ab20 con 0x7f1b841468d0 2024-03-23T03:51:09.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.941+0000 7f1b7be00640 1 --2- 172.21.15.102:0/3716143046 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b6406ac30 0x7f1b6406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:09.942 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.941+0000 7f1b7be00640 1 -- 172.21.15.102:0/3716143046 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f1b7c088d50 con 0x7f1b841468d0 2024-03-23T03:51:09.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.942+0000 7f1b83e00640 1 --2- 172.21.15.102:0/3716143046 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b6406ac30 0x7f1b6406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:09.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.942+0000 7f1b83e00640 1 --2- 172.21.15.102:0/3716143046 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b6406ac30 0x7f1b6406d0f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f1b74033000 tx=0x7f1b74009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:09.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:09.946+0000 7f1b7be00640 1 -- 172.21.15.102:0/3716143046 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1b84152420 con 0x7f1b841468d0 2024-03-23T03:51:10.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:10.179+0000 7f1b89c00640 1 -- 172.21.15.102:0/3716143046 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1b84152420 con 0x7f1b841468d0 2024-03-23T03:51:10.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:10.789+0000 7f1b7be00640 1 -- 172.21.15.102:0/3716143046 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v50) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1b84152420 con 0x7f1b841468d0 2024-03-23T03:51:10.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:10.793+0000 7f1b89c00640 1 -- 172.21.15.102:0/3716143046 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b6406ac30 msgr2=0x7f1b6406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:10.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:10.793+0000 7f1b89c00640 1 --2- 172.21.15.102:0/3716143046 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b6406ac30 0x7f1b6406d0f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f1b74033000 tx=0x7f1b74009030 comp rx=0 tx=0).stop 2024-03-23T03:51:10.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:10.793+0000 7f1b89c00640 1 -- 172.21.15.102:0/3716143046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b841468d0 msgr2=0x7f1b84146d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:10.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:10.793+0000 7f1b89c00640 1 --2- 172.21.15.102:0/3716143046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b841468d0 0x7f1b84146d50 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f1b8415ef60 tx=0x7f1b7c00cc00 comp rx=0 tx=0).stop 2024-03-23T03:51:10.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:10.793+0000 7f1b89c00640 1 -- 172.21.15.102:0/3716143046 shutdown_connections 2024-03-23T03:51:10.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:10.793+0000 7f1b89c00640 1 --2- 172.21.15.102:0/3716143046 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b6406ac30 0x7f1b6406d0f0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:10.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:10.793+0000 7f1b89c00640 1 --2- 172.21.15.102:0/3716143046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b841468d0 0x7f1b84146d50 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:10.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:10.793+0000 7f1b89c00640 1 --2- 172.21.15.102:0/3716143046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b84150ea0 0x7f1b8413b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:10.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:10.793+0000 7f1b89c00640 1 --2- 172.21.15.102:0/3716143046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b680039c0 0x7f1b840af900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:10.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:10.793+0000 7f1b89c00640 1 -- 172.21.15.102:0/3716143046 >> 172.21.15.102:0/3716143046 conn(0x7f1b84070800 msgr2=0x7f1b840cd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:10.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:10.794+0000 7f1b89c00640 1 -- 172.21.15.102:0/3716143046 shutdown_connections 2024-03-23T03:51:10.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:10.794+0000 7f1b89c00640 1 -- 172.21.15.102:0/3716143046 wait complete. 2024-03-23T03:51:10.814 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T03:51:11.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.020+0000 7f1598e00640 1 Processor -- start 2024-03-23T03:51:11.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.020+0000 7f1598e00640 1 -- start start 2024-03-23T03:51:11.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.021+0000 7f1598e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f15940ca7b0 0x7f15940cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:11.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.021+0000 7f1598e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940ce930 0x7f15940cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:11.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.021+0000 7f1598e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1594159c10 con 0x7f1594159830 2024-03-23T03:51:11.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.022+0000 7f1598e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f159406de50 con 0x7f15940ca7b0 2024-03-23T03:51:11.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.022+0000 7f1598e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15940cf300 con 0x7f15940ce930 2024-03-23T03:51:11.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.022+0000 7f1591e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1594159830 0x7f15940cc3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60604/0 (socket says 172.21.15.102:60604) 2024-03-23T03:51:11.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.022+0000 7f1591e00640 1 -- 172.21.15.102:0/2766064590 learned_addr learned my addr 172.21.15.102:0/2766064590 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:11.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.022+0000 7f1589400640 1 --2- 172.21.15.102:0/2766064590 >> v2:172.21.15.102:3300/0 conn(0x7f15940ca7b0 0x7f15940cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:11.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.022+0000 7f1589400640 1 -- 172.21.15.102:0/2766064590 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940ce930 msgr2=0x7f15940cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:11.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.022+0000 7f1589400640 1 --2- 172.21.15.102:0/2766064590 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940ce930 0x7f15940cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:11.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.022+0000 7f1589400640 1 -- 172.21.15.102:0/2766064590 >> v1:172.21.15.102:6789/0 conn(0x7f1594159830 legacy=0x7f15940cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:11.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.022+0000 7f1589400640 1 -- 172.21.15.102:0/2766064590 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15940cd840 con 0x7f15940ca7b0 2024-03-23T03:51:11.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.023+0000 7f1589400640 1 --2- 172.21.15.102:0/2766064590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ca7b0 0x7f15940cdcd0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f157c0097a0 tx=0x7f157c02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e68368216048bb1f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:11.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.024+0000 7f1591400640 1 -- 172.21.15.102:0/2766064590 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157c030800 con 0x7f15940ca7b0 2024-03-23T03:51:11.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.024+0000 7f1591400640 1 -- 172.21.15.102:0/2766064590 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157c030960 con 0x7f15940ca7b0 2024-03-23T03:51:11.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.024+0000 7f1591400640 1 -- 172.21.15.102:0/2766064590 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157c030cf0 con 0x7f15940ca7b0 2024-03-23T03:51:11.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.024+0000 7f1598e00640 1 -- 172.21.15.102:0/2766064590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ca7b0 msgr2=0x7f15940cdcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:11.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.024+0000 7f1598e00640 1 --2- 172.21.15.102:0/2766064590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ca7b0 0x7f15940cdcd0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f157c0097a0 tx=0x7f157c02b940 comp rx=0 tx=0).stop 2024-03-23T03:51:11.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.024+0000 7f1598e00640 1 -- 172.21.15.102:0/2766064590 shutdown_connections 2024-03-23T03:51:11.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.024+0000 7f1598e00640 1 --2- 172.21.15.102:0/2766064590 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940ce930 0x7f15940cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:11.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.024+0000 7f1598e00640 1 --2- 172.21.15.102:0/2766064590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ca7b0 0x7f15940cdcd0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:11.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.024+0000 7f1598e00640 1 -- 172.21.15.102:0/2766064590 >> 172.21.15.102:0/2766064590 conn(0x7f159406a0b0 msgr2=0x7f159406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:11.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.025+0000 7f1598e00640 1 -- 172.21.15.102:0/2766064590 shutdown_connections 2024-03-23T03:51:11.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.025+0000 7f1598e00640 1 -- 172.21.15.102:0/2766064590 wait complete. 2024-03-23T03:51:11.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.026+0000 7f1598e00640 1 Processor -- start 2024-03-23T03:51:11.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.027+0000 7f1598e00640 1 -- start start 2024-03-23T03:51:11.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.027+0000 7f1598e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940ca7b0 0x7f15941490e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:11.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.027+0000 7f1598e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ce930 0x7f15940afbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:11.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.027+0000 7f1598e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1594159830 0x7f15940ad410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:11.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.028+0000 7f1598e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1594151340 con 0x7f15940ce930 2024-03-23T03:51:11.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.028+0000 7f1598e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1594159fa0 con 0x7f1594159830 2024-03-23T03:51:11.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.028+0000 7f1598e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15941507c0 con 0x7f15940ca7b0 2024-03-23T03:51:11.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.028+0000 7f1589400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ce930 0x7f15940afbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:11.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.028+0000 7f1589400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ce930 0x7f15940afbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34452/0 (socket says 172.21.15.102:34452) 2024-03-23T03:51:11.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.028+0000 7f1589400640 1 -- 172.21.15.102:0/3113819581 learned_addr learned my addr 172.21.15.102:0/3113819581 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:11.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.028+0000 7f1589400640 1 -- 172.21.15.102:0/3113819581 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940ca7b0 msgr2=0x7f15941490e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:11.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.028+0000 7f1591e00640 1 --2- 172.21.15.102:0/3113819581 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940ca7b0 0x7f15941490e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:11.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.028+0000 7f1592800640 1 --2- 172.21.15.102:0/3113819581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1594159830 0x7f15940ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:11.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.028+0000 7f1589400640 1 --2- 172.21.15.102:0/3113819581 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940ca7b0 0x7f15941490e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:11.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.028+0000 7f1589400640 1 -- 172.21.15.102:0/3113819581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1594159830 msgr2=0x7f15940ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:11.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.028+0000 7f1589400640 1 --2- 172.21.15.102:0/3113819581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1594159830 0x7f15940ad410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:11.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.028+0000 7f1589400640 1 -- 172.21.15.102:0/3113819581 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157c009400 con 0x7f15940ce930 2024-03-23T03:51:11.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.028+0000 7f1592800640 1 --2- 172.21.15.102:0/3113819581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1594159830 0x7f15940ad410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:11.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.029+0000 7f1589400640 1 --2- 172.21.15.102:0/3113819581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ce930 0x7f15940afbe0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f157c02b980 tx=0x7f157c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:11.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.029+0000 7f158aa00640 1 -- 172.21.15.102:0/3113819581 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157c03e040 con 0x7f15940ce930 2024-03-23T03:51:11.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.029+0000 7f158aa00640 1 -- 172.21.15.102:0/3113819581 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157c039ce0 con 0x7f15940ce930 2024-03-23T03:51:11.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.029+0000 7f158aa00640 1 -- 172.21.15.102:0/3113819581 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f157c044630 con 0x7f15940ce930 2024-03-23T03:51:11.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.029+0000 7f1598e00640 1 -- 172.21.15.102:0/3113819581 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1594150aa0 con 0x7f15940ce930 2024-03-23T03:51:11.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.029+0000 7f1598e00640 1 -- 172.21.15.102:0/3113819581 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1594151060 con 0x7f15940ce930 2024-03-23T03:51:11.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.030+0000 7f1598e00640 1 -- 172.21.15.102:0/3113819581 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15940ccef0 con 0x7f15940ce930 2024-03-23T03:51:11.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.031+0000 7f158aa00640 1 -- 172.21.15.102:0/3113819581 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f157c04d3b0 con 0x7f15940ce930 2024-03-23T03:51:11.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.032+0000 7f158aa00640 1 --2- 172.21.15.102:0/3113819581 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f157006ac00 0x7f157006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:11.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.032+0000 7f158aa00640 1 -- 172.21.15.102:0/3113819581 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f157c0abd90 con 0x7f15940ce930 2024-03-23T03:51:11.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.032+0000 7f1591e00640 1 --2- 172.21.15.102:0/3113819581 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f157006ac00 0x7f157006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:11.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.033+0000 7f1591e00640 1 --2- 172.21.15.102:0/3113819581 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f157006ac00 0x7f157006d0c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1584000f60 tx=0x7f1584009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:11.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.036+0000 7f158aa00640 1 -- 172.21.15.102:0/3113819581 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f15940ccef0 con 0x7f15940ce930 2024-03-23T03:51:11.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.229+0000 7f1598e00640 1 -- 172.21.15.102:0/3113819581 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f15940ccef0 con 0x7f15940ce930 2024-03-23T03:51:11.793 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.792+0000 7f158aa00640 1 -- 172.21.15.102:0/3113819581 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v95) v1 ==== 181+0+0 (secure 0 0 0) 0x7f15940ccef0 con 0x7f15940ce930 2024-03-23T03:51:11.793 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' removed 2024-03-23T03:51:11.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.797+0000 7f1598e00640 1 -- 172.21.15.102:0/3113819581 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f157006ac00 msgr2=0x7f157006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:11.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.797+0000 7f1598e00640 1 --2- 172.21.15.102:0/3113819581 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f157006ac00 0x7f157006d0c0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1584000f60 tx=0x7f1584009040 comp rx=0 tx=0).stop 2024-03-23T03:51:11.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.797+0000 7f1598e00640 1 -- 172.21.15.102:0/3113819581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ce930 msgr2=0x7f15940afbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:11.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.797+0000 7f1598e00640 1 --2- 172.21.15.102:0/3113819581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ce930 0x7f15940afbe0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f157c02b980 tx=0x7f157c004730 comp rx=0 tx=0).stop 2024-03-23T03:51:11.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.797+0000 7f1598e00640 1 -- 172.21.15.102:0/3113819581 shutdown_connections 2024-03-23T03:51:11.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.797+0000 7f1598e00640 1 --2- 172.21.15.102:0/3113819581 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f157006ac00 0x7f157006d0c0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:11.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.797+0000 7f1598e00640 1 --2- 172.21.15.102:0/3113819581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1594159830 0x7f15940ad410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:11.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.797+0000 7f1598e00640 1 --2- 172.21.15.102:0/3113819581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15940ce930 0x7f15940afbe0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:11.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.797+0000 7f1598e00640 1 --2- 172.21.15.102:0/3113819581 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f15940ca7b0 0x7f15941490e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:11.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.797+0000 7f1598e00640 1 -- 172.21.15.102:0/3113819581 >> 172.21.15.102:0/3113819581 conn(0x7f159406a0b0 msgr2=0x7f1594158b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:11.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.798+0000 7f1598e00640 1 -- 172.21.15.102:0/3113819581 shutdown_connections 2024-03-23T03:51:11.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:11.798+0000 7f1598e00640 1 -- 172.21.15.102:0/3113819581 wait complete. 2024-03-23T03:51:11.822 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T03:51:12.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.028+0000 7fc8bb400640 1 Processor -- start 2024-03-23T03:51:12.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.029+0000 7fc8bb400640 1 -- start start 2024-03-23T03:51:12.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.030+0000 7fc8bb400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8b40add30 0x7fc8b40ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:12.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.030+0000 7fc8bb400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc8b406ce90 0x7fc8b406d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:12.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.030+0000 7fc8bb400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8b40cf250 con 0x7fc8b40ce930 2024-03-23T03:51:12.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.030+0000 7fc8bb400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8b406c760 con 0x7fc8b406ce90 2024-03-23T03:51:12.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.030+0000 7fc8bb400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b40b42d0 con 0x7fc8b40add30 2024-03-23T03:51:12.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.031+0000 7fc8b9400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc8b406ce90 0x7fc8b406d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:12.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.031+0000 7fc8b9400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc8b406ce90 0x7fc8b406d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34454/0 (socket says 172.21.15.102:34454) 2024-03-23T03:51:12.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.031+0000 7fc8b9400640 1 -- 172.21.15.102:0/329694866 learned_addr learned my addr 172.21.15.102:0/329694866 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:12.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.031+0000 7fc8b3e00640 1 --2- 172.21.15.102:0/329694866 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8b40add30 0x7fc8b40ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:12.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.031+0000 7fc8b9400640 1 -- 172.21.15.102:0/329694866 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8b40add30 msgr2=0x7fc8b40ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:12.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.031+0000 7fc8b9400640 1 --2- 172.21.15.102:0/329694866 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8b40add30 0x7fc8b40ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:12.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.031+0000 7fc8b9400640 1 -- 172.21.15.102:0/329694866 >> v1:172.21.15.173:6789/0 conn(0x7fc8b40ce930 legacy=0x7fc8b40cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:12.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.031+0000 7fc8b9400640 1 -- 172.21.15.102:0/329694866 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b40cdba0 con 0x7fc8b406ce90 2024-03-23T03:51:12.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.032+0000 7fc8b9400640 1 --2- 172.21.15.102:0/329694866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8b406ce90 0x7fc8b406d320 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac009870 tx=0x7fc8ac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd1f25721b649de server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:12.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.033+0000 7fc8b3400640 1 -- 172.21.15.102:0/329694866 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac031640 con 0x7fc8b406ce90 2024-03-23T03:51:12.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.033+0000 7fc8b3400640 1 -- 172.21.15.102:0/329694866 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8ac0317a0 con 0x7fc8b406ce90 2024-03-23T03:51:12.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.033+0000 7fc8b3400640 1 -- 172.21.15.102:0/329694866 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8ac031b30 con 0x7fc8b406ce90 2024-03-23T03:51:12.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.033+0000 7fc8bb400640 1 -- 172.21.15.102:0/329694866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8b406ce90 msgr2=0x7fc8b406d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:12.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.033+0000 7fc8bb400640 1 --2- 172.21.15.102:0/329694866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8b406ce90 0x7fc8b406d320 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac009870 tx=0x7fc8ac02b8f0 comp rx=0 tx=0).stop 2024-03-23T03:51:12.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.033+0000 7fc8bb400640 1 -- 172.21.15.102:0/329694866 shutdown_connections 2024-03-23T03:51:12.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.033+0000 7fc8bb400640 1 --2- 172.21.15.102:0/329694866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8b406ce90 0x7fc8b406d320 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:12.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.033+0000 7fc8bb400640 1 --2- 172.21.15.102:0/329694866 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8b40add30 0x7fc8b40ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:12.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.033+0000 7fc8bb400640 1 -- 172.21.15.102:0/329694866 >> 172.21.15.102:0/329694866 conn(0x7fc8b406a0b0 msgr2=0x7fc8b406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:12.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.033+0000 7fc8bb400640 1 -- 172.21.15.102:0/329694866 shutdown_connections 2024-03-23T03:51:12.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.034+0000 7fc8bb400640 1 -- 172.21.15.102:0/329694866 wait complete. 2024-03-23T03:51:12.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.035+0000 7fc8bb400640 1 Processor -- start 2024-03-23T03:51:12.036 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.035+0000 7fc8bb400640 1 -- start start 2024-03-23T03:51:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.036+0000 7fc8bb400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8b406ce90 0x7fc8b4153f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:12.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.036+0000 7fc8bb400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8b40add30 0x7fc8b4158000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.036+0000 7fc8bb400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8b40ce930 0x7fc8b40af360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.036+0000 7fc8bb400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b41590f0 con 0x7fc8b40add30 2024-03-23T03:51:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.036+0000 7fc8bb400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b4154450 con 0x7fc8b40ce930 2024-03-23T03:51:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.036+0000 7fc8bb400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b4158850 con 0x7fc8b406ce90 2024-03-23T03:51:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.036+0000 7fc8b3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8b40add30 0x7fc8b4158000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.036+0000 7fc8b3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8b40add30 0x7fc8b4158000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:34462/0 (socket says 172.21.15.102:34462) 2024-03-23T03:51:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.037+0000 7fc8b3e00640 1 -- 172.21.15.102:0/1331703891 learned_addr learned my addr 172.21.15.102:0/1331703891 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.037+0000 7fc8b9400640 1 --2- 172.21.15.102:0/1331703891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8b40ce930 0x7fc8b40af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.037+0000 7fc8b8a00640 1 --2- 172.21.15.102:0/1331703891 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8b406ce90 0x7fc8b4153f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.037+0000 7fc8b3e00640 1 -- 172.21.15.102:0/1331703891 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8b406ce90 msgr2=0x7fc8b4153f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.037+0000 7fc8b3e00640 1 --2- 172.21.15.102:0/1331703891 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8b406ce90 0x7fc8b4153f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.037+0000 7fc8b3e00640 1 -- 172.21.15.102:0/1331703891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8b40ce930 msgr2=0x7fc8b40af360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:12.039 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.037+0000 7fc8b3e00640 1 --2- 172.21.15.102:0/1331703891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8b40ce930 0x7fc8b40af360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.037+0000 7fc8b3e00640 1 -- 172.21.15.102:0/1331703891 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ac0094d0 con 0x7fc8b40add30 2024-03-23T03:51:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.037+0000 7fc8b9400640 1 --2- 172.21.15.102:0/1331703891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8b40ce930 0x7fc8b40af360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.037+0000 7fc8b8a00640 1 --2- 172.21.15.102:0/1331703891 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8b406ce90 0x7fc8b4153f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.037+0000 7fc8b3e00640 1 --2- 172.21.15.102:0/1331703891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8b40add30 0x7fc8b4158000 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fc89c007ef0 tx=0x7fc89c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.038+0000 7fc8b0c00640 1 -- 172.21.15.102:0/1331703891 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89c014620 con 0x7fc8b40add30 2024-03-23T03:51:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.038+0000 7fc8b0c00640 1 -- 172.21.15.102:0/1331703891 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc89c014780 con 0x7fc8b40add30 2024-03-23T03:51:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.038+0000 7fc8bb400640 1 -- 172.21.15.102:0/1331703891 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b4158b30 con 0x7fc8b40add30 2024-03-23T03:51:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.038+0000 7fc8b0c00640 1 -- 172.21.15.102:0/1331703891 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc89c014b30 con 0x7fc8b40add30 2024-03-23T03:51:12.040 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.038+0000 7fc8bb400640 1 -- 172.21.15.102:0/1331703891 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b4154c20 con 0x7fc8b40add30 2024-03-23T03:51:12.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.039+0000 7fc8bb400640 1 -- 172.21.15.102:0/1331703891 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b4154a80 con 0x7fc8b40add30 2024-03-23T03:51:12.041 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.040+0000 7fc8b0c00640 1 -- 172.21.15.102:0/1331703891 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc89c014c90 con 0x7fc8b40add30 2024-03-23T03:51:12.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.041+0000 7fc8b0c00640 1 --2- 172.21.15.102:0/1331703891 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc89006ac00 0x7fc89006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:12.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.041+0000 7fc8b0c00640 1 -- 172.21.15.102:0/1331703891 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5677+0+0 (secure 0 0 0) 0x7fc89c08aa60 con 0x7fc8b40add30 2024-03-23T03:51:12.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.041+0000 7fc8b8a00640 1 --2- 172.21.15.102:0/1331703891 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc89006ac00 0x7fc89006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:12.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.042+0000 7fc8b8a00640 1 --2- 172.21.15.102:0/1331703891 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc89006ac00 0x7fc89006d0c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40ad3e0 tx=0x7fc8a4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:12.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.044+0000 7fc8b0c00640 1 -- 172.21.15.102:0/1331703891 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc89c055c40 con 0x7fc8b40add30 2024-03-23T03:51:12.237 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.236+0000 7fc8bb400640 1 -- 172.21.15.102:0/1331703891 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc8b4154a60 con 0x7fc8b40add30 2024-03-23T03:51:12.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.825+0000 7fc8b0c00640 1 -- 172.21.15.102:0/1331703891 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v96) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc89c059260 con 0x7fc8b40add30 2024-03-23T03:51:12.826 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' removed 2024-03-23T03:51:12.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.830+0000 7fc8bb400640 1 -- 172.21.15.102:0/1331703891 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc89006ac00 msgr2=0x7fc89006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:12.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.830+0000 7fc8bb400640 1 --2- 172.21.15.102:0/1331703891 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc89006ac00 0x7fc89006d0c0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fc8b40ad3e0 tx=0x7fc8a4008040 comp rx=0 tx=0).stop 2024-03-23T03:51:12.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.830+0000 7fc8bb400640 1 -- 172.21.15.102:0/1331703891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8b40add30 msgr2=0x7fc8b4158000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:12.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.830+0000 7fc8bb400640 1 --2- 172.21.15.102:0/1331703891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8b40add30 0x7fc8b4158000 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fc89c007ef0 tx=0x7fc89c00e4b0 comp rx=0 tx=0).stop 2024-03-23T03:51:12.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.831+0000 7fc8bb400640 1 -- 172.21.15.102:0/1331703891 shutdown_connections 2024-03-23T03:51:12.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.831+0000 7fc8bb400640 1 --2- 172.21.15.102:0/1331703891 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc89006ac00 0x7fc89006d0c0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:12.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.831+0000 7fc8bb400640 1 --2- 172.21.15.102:0/1331703891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc8b40ce930 0x7fc8b40af360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:12.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.831+0000 7fc8bb400640 1 --2- 172.21.15.102:0/1331703891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8b40add30 0x7fc8b4158000 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:12.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.831+0000 7fc8bb400640 1 --2- 172.21.15.102:0/1331703891 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8b406ce90 0x7fc8b4153f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:12.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.831+0000 7fc8bb400640 1 -- 172.21.15.102:0/1331703891 >> 172.21.15.102:0/1331703891 conn(0x7fc8b406a0b0 msgr2=0x7fc8b40cd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:12.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.832+0000 7fc8bb400640 1 -- 172.21.15.102:0/1331703891 shutdown_connections 2024-03-23T03:51:12.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:12.832+0000 7fc8bb400640 1 -- 172.21.15.102:0/1331703891 wait complete. 2024-03-23T03:51:12.855 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-23T03:51:13.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.041+0000 7fd068000640 1 Processor -- start 2024-03-23T03:51:13.042 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.041+0000 7fd068000640 1 -- start start 2024-03-23T03:51:13.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.043+0000 7fd068000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd060070cd0 0x7fd060160bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:13.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.043+0000 7fd068000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd06006d390 con 0x7fd0600cf040 2024-03-23T03:51:13.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.043+0000 7fd068000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd06006d670 con 0x7fd0601456a0 2024-03-23T03:51:13.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.043+0000 7fd068000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0600ad890 con 0x7fd060070cd0 2024-03-23T03:51:13.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.043+0000 7fd064c00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd0600cf040 0x7fd0601604c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60612/0 (socket says 172.21.15.102:60612) 2024-03-23T03:51:13.044 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.043+0000 7fd064c00640 1 -- 172.21.15.102:0/3253438141 learned_addr learned my addr 172.21.15.102:0/3253438141 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:13.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.044+0000 7fd066000640 1 --2- 172.21.15.102:0/3253438141 >> v2:172.21.15.173:3300/0 conn(0x7fd060070cd0 0x7fd060160bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:13.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.044+0000 7fd05fe00640 1 -- 172.21.15.102:0/3253438141 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438630203 0 0) 0x7fd06006d390 con 0x7fd0600cf040 2024-03-23T03:51:13.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.044+0000 7fd05fe00640 1 -- 172.21.15.102:0/3253438141 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd048003520 con 0x7fd0600cf040 2024-03-23T03:51:13.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.044+0000 7fd05fe00640 1 -- 172.21.15.102:0/3253438141 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2476954645 0 0) 0x7fd048003520 con 0x7fd0600cf040 2024-03-23T03:51:13.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.044+0000 7fd05fe00640 1 -- 172.21.15.102:0/3253438141 >> v2:172.21.15.173:3300/0 conn(0x7fd060070cd0 msgr2=0x7fd060160bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:13.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.044+0000 7fd05fe00640 1 --2- 172.21.15.102:0/3253438141 >> v2:172.21.15.173:3300/0 conn(0x7fd060070cd0 0x7fd060160bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.044+0000 7fd05fe00640 1 -- 172.21.15.102:0/3253438141 >> v1:172.21.15.173:6789/0 conn(0x7fd0601456a0 legacy=0x7fd0600ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:13.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.044+0000 7fd05fe00640 1 -- 172.21.15.102:0/3253438141 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0600701f0 con 0x7fd0600cf040 2024-03-23T03:51:13.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.044+0000 7fd05fe00640 1 -- 172.21.15.102:0/3253438141 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fd04c001670 con 0x7fd0600cf040 2024-03-23T03:51:13.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.044+0000 7fd05fe00640 1 -- 172.21.15.102:0/3253438141 >> v1:172.21.15.102:6789/0 conn(0x7fd0600cf040 legacy=0x7fd0601604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:13.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.044+0000 7fd05fe00640 1 --2- 172.21.15.102:0/3253438141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd048003970 0x7fd048003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:13.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.044+0000 7fd05fe00640 1 --2- 172.21.15.102:0/3253438141 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0480047c0 0x7fd048004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:13.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.044+0000 7fd05fe00640 1 --2- 172.21.15.102:0/3253438141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd048005180 0x7fd04800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:13.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.044+0000 7fd05fe00640 1 -- 172.21.15.102:0/3253438141 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0600ad890 con 0x7fd048003970 2024-03-23T03:51:13.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.044+0000 7fd05fe00640 1 -- 172.21.15.102:0/3253438141 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04800c0d0 con 0x7fd048005180 2024-03-23T03:51:13.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.044+0000 7fd05fe00640 1 -- 172.21.15.102:0/3253438141 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04800c290 con 0x7fd0480047c0 2024-03-23T03:51:13.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.044+0000 7fd066000640 1 --2- 172.21.15.102:0/3253438141 >> v2:172.21.15.173:3300/0 conn(0x7fd060070cd0 0x7fd060160bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:51:13.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.045+0000 7fd065600640 1 --2- 172.21.15.102:0/3253438141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd048003970 0x7fd048003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:13.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.045+0000 7fd064c00640 1 --2- 172.21.15.102:0/3253438141 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0480047c0 0x7fd048004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:13.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.045+0000 7fd065600640 1 -- 172.21.15.102:0/3253438141 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0480047c0 msgr2=0x7fd048004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:13.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.045+0000 7fd065600640 1 --2- 172.21.15.102:0/3253438141 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd0480047c0 0x7fd048004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.045+0000 7fd065600640 1 -- 172.21.15.102:0/3253438141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd048005180 msgr2=0x7fd04800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:13.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.045+0000 7fd065600640 1 --2- 172.21.15.102:0/3253438141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd048005180 0x7fd04800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.045+0000 7fd065600640 1 -- 172.21.15.102:0/3253438141 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd048003520 con 0x7fd048003970 2024-03-23T03:51:13.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.045+0000 7fd066000640 1 -- 172.21.15.102:0/3253438141 reap_dead start 2024-03-23T03:51:13.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.046+0000 7fd065600640 1 --2- 172.21.15.102:0/3253438141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd048003970 0x7fd048003d90 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fd054001750 tx=0x7fd0480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7257fcb01708bd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:13.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.046+0000 7fd05fe00640 1 -- 172.21.15.102:0/3253438141 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0540052d0 con 0x7fd048003970 2024-03-23T03:51:13.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.046+0000 7fd05fe00640 1 -- 172.21.15.102:0/3253438141 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05402e050 con 0x7fd048003970 2024-03-23T03:51:13.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.046+0000 7fd05fe00640 1 -- 172.21.15.102:0/3253438141 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd054003830 con 0x7fd048003970 2024-03-23T03:51:13.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.047+0000 7fd068000640 1 -- 172.21.15.102:0/3253438141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd048003970 msgr2=0x7fd048003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:13.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.047+0000 7fd068000640 1 --2- 172.21.15.102:0/3253438141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd048003970 0x7fd048003d90 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fd054001750 tx=0x7fd0480057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.047+0000 7fd068000640 1 -- 172.21.15.102:0/3253438141 shutdown_connections 2024-03-23T03:51:13.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.047+0000 7fd068000640 1 --2- 172.21.15.102:0/3253438141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd048003970 0x7fd048003d90 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.047+0000 7fd068000640 1 -- 172.21.15.102:0/3253438141 >> 172.21.15.102:0/3253438141 conn(0x7fd06006a0b0 msgr2=0x7fd06006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:13.049 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.047+0000 7fd068000640 1 -- 172.21.15.102:0/3253438141 shutdown_connections 2024-03-23T03:51:13.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.047+0000 7fd068000640 1 -- 172.21.15.102:0/3253438141 wait complete. 2024-03-23T03:51:13.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.049+0000 7fd068000640 1 Processor -- start 2024-03-23T03:51:13.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.049+0000 7fd068000640 1 -- start start 2024-03-23T03:51:13.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.050+0000 7fd068000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd048003970 0x7fd060143770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:13.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.050+0000 7fd068000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd060135580 0x7fd0600c4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:13.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.050+0000 7fd068000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd06015eee0 0x7fd06015f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:13.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.050+0000 7fd068000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd060143cb0 con 0x7fd048003970 2024-03-23T03:51:13.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.050+0000 7fd068000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd06015f8a0 con 0x7fd060135580 2024-03-23T03:51:13.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.050+0000 7fd068000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd060144ab0 con 0x7fd06015eee0 2024-03-23T03:51:13.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.050+0000 7fd065600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd048003970 0x7fd060143770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:13.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.050+0000 7fd065600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd048003970 0x7fd060143770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52996/0 (socket says 172.21.15.102:52996) 2024-03-23T03:51:13.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.050+0000 7fd065600640 1 -- 172.21.15.102:0/988393101 learned_addr learned my addr 172.21.15.102:0/988393101 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:13.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.050+0000 7fd065600640 1 -- 172.21.15.102:0/988393101 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd06015eee0 msgr2=0x7fd06015f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:13.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.050+0000 7fd065600640 1 --2- 172.21.15.102:0/988393101 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd06015eee0 0x7fd06015f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.050+0000 7fd065600640 1 -- 172.21.15.102:0/988393101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd060135580 msgr2=0x7fd0600c4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:13.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.050+0000 7fd064c00640 1 --2- 172.21.15.102:0/988393101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd060135580 0x7fd0600c4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:13.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.051+0000 7fd065600640 1 --2- 172.21.15.102:0/988393101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd060135580 0x7fd0600c4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.051+0000 7fd065600640 1 -- 172.21.15.102:0/988393101 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0600c48c0 con 0x7fd048003970 2024-03-23T03:51:13.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.051+0000 7fd065600640 1 --2- 172.21.15.102:0/988393101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd048003970 0x7fd060143770 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fd05402bc30 tx=0x7fd054001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:13.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.051+0000 7fd05d600640 1 -- 172.21.15.102:0/988393101 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd054007a30 con 0x7fd048003970 2024-03-23T03:51:13.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.051+0000 7fd05d600640 1 -- 172.21.15.102:0/988393101 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd054007640 con 0x7fd048003970 2024-03-23T03:51:13.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.051+0000 7fd05d600640 1 -- 172.21.15.102:0/988393101 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd054006be0 con 0x7fd048003970 2024-03-23T03:51:13.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.051+0000 7fd068000640 1 -- 172.21.15.102:0/988393101 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd060143ea0 con 0x7fd048003970 2024-03-23T03:51:13.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.051+0000 7fd068000640 1 -- 172.21.15.102:0/988393101 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd060144460 con 0x7fd048003970 2024-03-23T03:51:13.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.053+0000 7fd068000640 1 -- 172.21.15.102:0/988393101 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd060073390 con 0x7fd048003970 2024-03-23T03:51:13.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.053+0000 7fd05d600640 1 -- 172.21.15.102:0/988393101 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd054006d40 con 0x7fd048003970 2024-03-23T03:51:13.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.054+0000 7fd05d600640 1 --2- 172.21.15.102:0/988393101 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd04006ac00 0x7fd04006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:13.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.054+0000 7fd05d600640 1 -- 172.21.15.102:0/988393101 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fd0540aadc0 con 0x7fd048003970 2024-03-23T03:51:13.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.055+0000 7fd064c00640 1 --2- 172.21.15.102:0/988393101 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd04006ac00 0x7fd04006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:13.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.055+0000 7fd064c00640 1 --2- 172.21.15.102:0/988393101 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd04006ac00 0x7fd04006d0c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fd060161190 tx=0x7fd04c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:13.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.059+0000 7fd05d600640 1 -- 172.21.15.102:0/988393101 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd054076030 con 0x7fd048003970 2024-03-23T03:51:13.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.255+0000 7fd068000640 1 -- 172.21.15.102:0/988393101 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fd060146600 con 0x7fd048003970 2024-03-23T03:51:13.262 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.261+0000 7fd05d600640 1 -- 172.21.15.102:0/988393101 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd0540410a0 con 0x7fd048003970 2024-03-23T03:51:13.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.263+0000 7fd068000640 1 -- 172.21.15.102:0/988393101 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd04006ac00 msgr2=0x7fd04006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:13.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.263+0000 7fd068000640 1 --2- 172.21.15.102:0/988393101 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd04006ac00 0x7fd04006d0c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fd060161190 tx=0x7fd04c009040 comp rx=0 tx=0).stop 2024-03-23T03:51:13.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.263+0000 7fd068000640 1 -- 172.21.15.102:0/988393101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd048003970 msgr2=0x7fd060143770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:13.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.263+0000 7fd068000640 1 --2- 172.21.15.102:0/988393101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd048003970 0x7fd060143770 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fd05402bc30 tx=0x7fd054001120 comp rx=0 tx=0).stop 2024-03-23T03:51:13.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.263+0000 7fd068000640 1 -- 172.21.15.102:0/988393101 shutdown_connections 2024-03-23T03:51:13.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.263+0000 7fd068000640 1 --2- 172.21.15.102:0/988393101 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd04006ac00 0x7fd04006d0c0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.263+0000 7fd068000640 1 --2- 172.21.15.102:0/988393101 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd06015eee0 0x7fd06015f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.263+0000 7fd068000640 1 --2- 172.21.15.102:0/988393101 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd060135580 0x7fd0600c4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.263+0000 7fd068000640 1 --2- 172.21.15.102:0/988393101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd048003970 0x7fd060143770 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.263+0000 7fd068000640 1 -- 172.21.15.102:0/988393101 >> 172.21.15.102:0/988393101 conn(0x7fd06006a0b0 msgr2=0x7fd0600b2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:13.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.264+0000 7fd068000640 1 -- 172.21.15.102:0/988393101 shutdown_connections 2024-03-23T03:51:13.266 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.264+0000 7fd068000640 1 -- 172.21.15.102:0/988393101 wait complete. 2024-03-23T03:51:13.284 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr' 2024-03-23T03:51:13.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.501+0000 7f5592600640 1 Processor -- start 2024-03-23T03:51:13.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.501+0000 7f5592600640 1 -- start start 2024-03-23T03:51:13.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.502+0000 7f5592600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f558c15b560 0x7f558c15b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:13.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.502+0000 7f5592600640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558c15f320 con 0x7f558c15d410 2024-03-23T03:51:13.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.502+0000 7f5592600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f558c06c6f0 con 0x7f558c0b39b0 2024-03-23T03:51:13.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.502+0000 7f5592600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f558c06c9d0 con 0x7f558c15b560 2024-03-23T03:51:13.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.503+0000 7f558b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f558c15b560 0x7f558c15b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:13.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.503+0000 7f558b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f558c15b560 0x7f558c15b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53004/0 (socket says 172.21.15.102:53004) 2024-03-23T03:51:13.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.503+0000 7f558b400640 1 -- 172.21.15.102:0/3739164331 learned_addr learned my addr 172.21.15.102:0/3739164331 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:13.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.503+0000 7f558be00640 1 --1- 172.21.15.102:0/3739164331 >> v1:172.21.15.102:6789/0 conn(0x7f558c15d410 0x7f558c15ec10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60622/0 (socket says 172.21.15.102:60622) 2024-03-23T03:51:13.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.503+0000 7f558b400640 1 -- 172.21.15.102:0/3739164331 >> v1:172.21.15.173:6789/0 conn(0x7f558c0b39b0 legacy=0x7f558c15e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:13.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.503+0000 7f558b400640 1 -- 172.21.15.102:0/3739164331 >> v1:172.21.15.102:6789/0 conn(0x7f558c15d410 legacy=0x7f558c15ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:13.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.503+0000 7f558b400640 1 -- 172.21.15.102:0/3739164331 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f558c06ccb0 con 0x7f558c15b560 2024-03-23T03:51:13.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.504+0000 7f558b400640 1 --2- 172.21.15.102:0/3739164331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f558c15b560 0x7f558c15b940 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f557c009870 tx=0x7f557c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3719bbda18c563e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:13.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.505+0000 7f558aa00640 1 -- 172.21.15.102:0/3739164331 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c031870 con 0x7f558c15b560 2024-03-23T03:51:13.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.505+0000 7f558aa00640 1 -- 172.21.15.102:0/3739164331 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557c0319d0 con 0x7f558c15b560 2024-03-23T03:51:13.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.505+0000 7f558aa00640 1 -- 172.21.15.102:0/3739164331 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557c031d60 con 0x7f558c15b560 2024-03-23T03:51:13.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.505+0000 7f5592600640 1 -- 172.21.15.102:0/3739164331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f558c15b560 msgr2=0x7f558c15b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:13.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.505+0000 7f5592600640 1 --2- 172.21.15.102:0/3739164331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f558c15b560 0x7f558c15b940 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f557c009870 tx=0x7f557c02b910 comp rx=0 tx=0).stop 2024-03-23T03:51:13.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.505+0000 7f5592600640 1 -- 172.21.15.102:0/3739164331 shutdown_connections 2024-03-23T03:51:13.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.505+0000 7f5592600640 1 --2- 172.21.15.102:0/3739164331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f558c15b560 0x7f558c15b940 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.505+0000 7f5592600640 1 -- 172.21.15.102:0/3739164331 >> 172.21.15.102:0/3739164331 conn(0x7f558c06a3d0 msgr2=0x7f558c0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:13.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.505+0000 7f5592600640 1 -- 172.21.15.102:0/3739164331 shutdown_connections 2024-03-23T03:51:13.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.506+0000 7f5592600640 1 -- 172.21.15.102:0/3739164331 wait complete. 2024-03-23T03:51:13.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.507+0000 7f5592600640 1 Processor -- start 2024-03-23T03:51:13.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.507+0000 7f5592600640 1 -- start start 2024-03-23T03:51:13.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.508+0000 7f5592600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f558c0b39b0 0x7f558c140170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:13.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.508+0000 7f5592600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f558c15b560 0x7f558c0acad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:13.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.508+0000 7f5592600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f558c15d410 0x7f558c0ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:13.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.508+0000 7f5592600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c1427e0 con 0x7f558c15b560 2024-03-23T03:51:13.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.508+0000 7f5592600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558c141c60 con 0x7f558c0b39b0 2024-03-23T03:51:13.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.508+0000 7f5592600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f558c141f40 con 0x7f558c15d410 2024-03-23T03:51:13.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.509+0000 7f5582a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f558c15b560 0x7f558c0acad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:13.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.509+0000 7f5582a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f558c15b560 0x7f558c0acad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53010/0 (socket says 172.21.15.102:53010) 2024-03-23T03:51:13.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.509+0000 7f5582a00640 1 -- 172.21.15.102:0/3814346742 learned_addr learned my addr 172.21.15.102:0/3814346742 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:13.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.509+0000 7f558b400640 1 --2- 172.21.15.102:0/3814346742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f558c0b39b0 0x7f558c140170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:13.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.509+0000 7f558be00640 1 --2- 172.21.15.102:0/3814346742 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f558c15d410 0x7f558c0ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:13.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.509+0000 7f5582a00640 1 -- 172.21.15.102:0/3814346742 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f558c15d410 msgr2=0x7f558c0ad010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:13.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.509+0000 7f5582a00640 1 --2- 172.21.15.102:0/3814346742 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f558c15d410 0x7f558c0ad010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.511 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.509+0000 7f5582a00640 1 -- 172.21.15.102:0/3814346742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f558c0b39b0 msgr2=0x7f558c140170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:13.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.509+0000 7f5582a00640 1 --2- 172.21.15.102:0/3814346742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f558c0b39b0 0x7f558c140170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.509+0000 7f5582a00640 1 -- 172.21.15.102:0/3814346742 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f557c0094d0 con 0x7f558c15b560 2024-03-23T03:51:13.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.509+0000 7f558b400640 1 --2- 172.21.15.102:0/3814346742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f558c0b39b0 0x7f558c140170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:13.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.510+0000 7f5582a00640 1 --2- 172.21.15.102:0/3814346742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f558c15b560 0x7f558c0acad0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f557400c430 tx=0x7f557400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:13.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.510+0000 7f5583e00640 1 -- 172.21.15.102:0/3814346742 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5574011790 con 0x7f558c15b560 2024-03-23T03:51:13.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.510+0000 7f5583e00640 1 -- 172.21.15.102:0/3814346742 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55740118f0 con 0x7f558c15b560 2024-03-23T03:51:13.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.510+0000 7f5592600640 1 -- 172.21.15.102:0/3814346742 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f558c142220 con 0x7f558c15b560 2024-03-23T03:51:13.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.510+0000 7f5583e00640 1 -- 172.21.15.102:0/3814346742 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5574011ce0 con 0x7f558c15b560 2024-03-23T03:51:13.512 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.510+0000 7f5592600640 1 -- 172.21.15.102:0/3814346742 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558c140f30 con 0x7f558c15b560 2024-03-23T03:51:13.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.511+0000 7f5592600640 1 -- 172.21.15.102:0/3814346742 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f558c06ac20 con 0x7f558c15b560 2024-03-23T03:51:13.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.512+0000 7f5583e00640 1 -- 172.21.15.102:0/3814346742 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f557401c460 con 0x7f558c15b560 2024-03-23T03:51:13.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.513+0000 7f5583e00640 1 --2- 172.21.15.102:0/3814346742 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f556806ac00 0x7f556806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:13.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.513+0000 7f5583e00640 1 -- 172.21.15.102:0/3814346742 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f5574089160 con 0x7f558c15b560 2024-03-23T03:51:13.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.514+0000 7f558b400640 1 --2- 172.21.15.102:0/3814346742 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f556806ac00 0x7f556806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:13.515 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.514+0000 7f558b400640 1 --2- 172.21.15.102:0/3814346742 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f556806ac00 0x7f556806d0c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f557c036000 tx=0x7f557c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:13.518 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.518+0000 7f5583e00640 1 -- 172.21.15.102:0/3814346742 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f55740544f0 con 0x7f558c15b560 2024-03-23T03:51:13.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.777+0000 7f5592600640 1 -- 172.21.15.102:0/3814346742 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr"]} v 0) v1 -- 0x7f558c073390 con 0x7f558c15b560 2024-03-23T03:51:13.818 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.818+0000 7f5583e00640 1 -- 172.21.15.102:0/3814346742 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_getfattr"]}]=0 v256) v1 ==== 140+0+0 (secure 0 0 0) 0x7f5574057b10 con 0x7f558c15b560 2024-03-23T03:51:13.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.820+0000 7f5592600640 1 -- 172.21.15.102:0/3814346742 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f556806ac00 msgr2=0x7f556806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:13.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.820+0000 7f5592600640 1 --2- 172.21.15.102:0/3814346742 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f556806ac00 0x7f556806d0c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f557c036000 tx=0x7f557c03c040 comp rx=0 tx=0).stop 2024-03-23T03:51:13.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.820+0000 7f5592600640 1 -- 172.21.15.102:0/3814346742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f558c15b560 msgr2=0x7f558c0acad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:13.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.820+0000 7f5592600640 1 --2- 172.21.15.102:0/3814346742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f558c15b560 0x7f558c0acad0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f557400c430 tx=0x7f557400c8f0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.820+0000 7f5592600640 1 -- 172.21.15.102:0/3814346742 shutdown_connections 2024-03-23T03:51:13.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.820+0000 7f5592600640 1 --2- 172.21.15.102:0/3814346742 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f556806ac00 0x7f556806d0c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.820+0000 7f5592600640 1 --2- 172.21.15.102:0/3814346742 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f558c15d410 0x7f558c0ad010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.820+0000 7f5592600640 1 --2- 172.21.15.102:0/3814346742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f558c15b560 0x7f558c0acad0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.820+0000 7f5592600640 1 --2- 172.21.15.102:0/3814346742 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f558c0b39b0 0x7f558c140170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:13.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.820+0000 7f5592600640 1 -- 172.21.15.102:0/3814346742 >> 172.21.15.102:0/3814346742 conn(0x7f558c06a3d0 msgr2=0x7f558c151480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:13.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.821+0000 7f5592600640 1 -- 172.21.15.102:0/3814346742 shutdown_connections 2024-03-23T03:51:13.822 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:13.821+0000 7f5592600640 1 -- 172.21.15.102:0/3814346742 wait complete. 2024-03-23T03:51:13.840 INFO:tasks.cephfs_test_runner:That quota changes made from one client are visible to another ... ok 2024-03-23T03:51:13.840 INFO:tasks.cephfs_test_runner:Starting test: test_remote_update_write (tasks.cephfs.test_quota.TestQuota) That when a client modifies the quota on a directory used 2024-03-23T03:51:13.840 INFO:tasks.cephfs_test_runner:test_remote_update_write (tasks.cephfs.test_quota.TestQuota) 2024-03-23T03:51:13.840 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_write' 2024-03-23T03:51:14.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.050+0000 7f90d6200640 1 Processor -- start 2024-03-23T03:51:14.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.050+0000 7f90d6200640 1 -- start start 2024-03-23T03:51:14.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.051+0000 7f90d6200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f90d015b560 0x7f90d015b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:14.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.051+0000 7f90d6200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f90d015d410 0x7f90d00b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:14.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.051+0000 7f90d6200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90d006c6f0 con 0x7f90d00b39b0 2024-03-23T03:51:14.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.051+0000 7f90d6200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90d00b4370 con 0x7f90d015d410 2024-03-23T03:51:14.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.051+0000 7f90d6200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d006c9d0 con 0x7f90d015b560 2024-03-23T03:51:14.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.052+0000 7f90cfe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f90d015d410 0x7f90d00b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:14.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.052+0000 7f90c6a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f90d00b39b0 0x7f90d015e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60626/0 (socket says 172.21.15.102:60626) 2024-03-23T03:51:14.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.052+0000 7f90cfe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f90d015d410 0x7f90d00b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53024/0 (socket says 172.21.15.102:53024) 2024-03-23T03:51:14.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.052+0000 7f90c6a00640 1 -- 172.21.15.102:0/3768317774 learned_addr learned my addr 172.21.15.102:0/3768317774 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:14.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.052+0000 7f90cf400640 1 --2- 172.21.15.102:0/3768317774 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f90d015b560 0x7f90d015b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:14.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.052+0000 7f90cfe00640 1 -- 172.21.15.102:0/3768317774 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f90d015b560 msgr2=0x7f90d015b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:14.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.052+0000 7f90cfe00640 1 --2- 172.21.15.102:0/3768317774 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f90d015b560 0x7f90d015b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:14.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.052+0000 7f90cfe00640 1 -- 172.21.15.102:0/3768317774 >> v1:172.21.15.102:6789/0 conn(0x7f90d00b39b0 legacy=0x7f90d015e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:14.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.052+0000 7f90cfe00640 1 -- 172.21.15.102:0/3768317774 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d006ccb0 con 0x7f90d015d410 2024-03-23T03:51:14.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.053+0000 7f90cfe00640 1 --2- 172.21.15.102:0/3768317774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90d015d410 0x7f90d00b3e30 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009d90 tx=0x7f90c802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef4418a8825121c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:14.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.054+0000 7f90cea00640 1 -- 172.21.15.102:0/3768317774 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c802cc60 con 0x7f90d015d410 2024-03-23T03:51:14.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.054+0000 7f90cea00640 1 -- 172.21.15.102:0/3768317774 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c802cdc0 con 0x7f90d015d410 2024-03-23T03:51:14.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.054+0000 7f90cea00640 1 -- 172.21.15.102:0/3768317774 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c8034660 con 0x7f90d015d410 2024-03-23T03:51:14.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.054+0000 7f90d6200640 1 -- 172.21.15.102:0/3768317774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90d015d410 msgr2=0x7f90d00b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:14.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.054+0000 7f90d6200640 1 --2- 172.21.15.102:0/3768317774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90d015d410 0x7f90d00b3e30 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009d90 tx=0x7f90c802beb0 comp rx=0 tx=0).stop 2024-03-23T03:51:14.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.054+0000 7f90d6200640 1 -- 172.21.15.102:0/3768317774 shutdown_connections 2024-03-23T03:51:14.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.054+0000 7f90d6200640 1 --2- 172.21.15.102:0/3768317774 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90d015d410 0x7f90d00b3e30 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:14.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.054+0000 7f90d6200640 1 --2- 172.21.15.102:0/3768317774 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f90d015b560 0x7f90d015b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:14.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.054+0000 7f90d6200640 1 -- 172.21.15.102:0/3768317774 >> 172.21.15.102:0/3768317774 conn(0x7f90d006a3d0 msgr2=0x7f90d00b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:14.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.054+0000 7f90d6200640 1 -- 172.21.15.102:0/3768317774 shutdown_connections 2024-03-23T03:51:14.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.054+0000 7f90d6200640 1 -- 172.21.15.102:0/3768317774 wait complete. 2024-03-23T03:51:14.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.056+0000 7f90d6200640 1 Processor -- start 2024-03-23T03:51:14.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.056+0000 7f90d6200640 1 -- start start 2024-03-23T03:51:14.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.057+0000 7f90d6200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f90d00b39b0 0x7f90d01ef970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.057+0000 7f90d6200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90d015b560 0x7f90d013f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.057+0000 7f90d6200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90d015d410 0x7f90d013f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.057+0000 7f90d6200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d0152160 con 0x7f90d015d410 2024-03-23T03:51:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.057+0000 7f90d6200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90d013d3a0 con 0x7f90d015b560 2024-03-23T03:51:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.057+0000 7f90d6200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90d013d680 con 0x7f90d00b39b0 2024-03-23T03:51:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.057+0000 7f90cfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90d015d410 0x7f90d013f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.057+0000 7f90cfe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90d015d410 0x7f90d013f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53034/0 (socket says 172.21.15.102:53034) 2024-03-23T03:51:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.057+0000 7f90cfe00640 1 -- 172.21.15.102:0/1051675225 learned_addr learned my addr 172.21.15.102:0/1051675225 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.058+0000 7f90cf400640 1 --2- 172.21.15.102:0/1051675225 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f90d00b39b0 0x7f90d01ef970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.058+0000 7f90cfe00640 1 -- 172.21.15.102:0/1051675225 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f90d00b39b0 msgr2=0x7f90d01ef970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.058+0000 7f90cfe00640 1 --2- 172.21.15.102:0/1051675225 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f90d00b39b0 0x7f90d01ef970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.058+0000 7f90c6a00640 1 --2- 172.21.15.102:0/1051675225 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90d015b560 0x7f90d013f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:14.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.058+0000 7f90cfe00640 1 -- 172.21.15.102:0/1051675225 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90d015b560 msgr2=0x7f90d013f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:14.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.058+0000 7f90cfe00640 1 --2- 172.21.15.102:0/1051675225 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90d015b560 0x7f90d013f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:14.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.058+0000 7f90cfe00640 1 -- 172.21.15.102:0/1051675225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90c8009a70 con 0x7f90d015d410 2024-03-23T03:51:14.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.058+0000 7f90cf400640 1 --2- 172.21.15.102:0/1051675225 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f90d00b39b0 0x7f90d01ef970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:14.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.058+0000 7f90cfe00640 1 --2- 172.21.15.102:0/1051675225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90d015d410 0x7f90d013f650 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009d90 tx=0x7f90c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:14.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.058+0000 7f90c7e00640 1 -- 172.21.15.102:0/1051675225 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c8040650 con 0x7f90d015d410 2024-03-23T03:51:14.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.058+0000 7f90c7e00640 1 -- 172.21.15.102:0/1051675225 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90c80407d0 con 0x7f90d015d410 2024-03-23T03:51:14.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.058+0000 7f90c7e00640 1 -- 172.21.15.102:0/1051675225 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90c8040b80 con 0x7f90d015d410 2024-03-23T03:51:14.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.058+0000 7f90d6200640 1 -- 172.21.15.102:0/1051675225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90d013d960 con 0x7f90d015d410 2024-03-23T03:51:14.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.059+0000 7f90d6200640 1 -- 172.21.15.102:0/1051675225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90d013d070 con 0x7f90d015d410 2024-03-23T03:51:14.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.060+0000 7f90d6200640 1 -- 172.21.15.102:0/1051675225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90d006ac20 con 0x7f90d015d410 2024-03-23T03:51:14.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.060+0000 7f90c7e00640 1 -- 172.21.15.102:0/1051675225 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f90c8040ce0 con 0x7f90d015d410 2024-03-23T03:51:14.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.061+0000 7f90c7e00640 1 --2- 172.21.15.102:0/1051675225 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f90ac06ac00 0x7f90ac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:14.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.061+0000 7f90c7e00640 1 -- 172.21.15.102:0/1051675225 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f90c80abfe0 con 0x7f90d015d410 2024-03-23T03:51:14.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.062+0000 7f90cf400640 1 --2- 172.21.15.102:0/1051675225 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f90ac06ac00 0x7f90ac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:14.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.062+0000 7f90cf400640 1 --2- 172.21.15.102:0/1051675225 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f90ac06ac00 0x7f90ac06d0c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f90c0006fd0 tx=0x7f90c0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:14.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.066+0000 7f90c7e00640 1 -- 172.21.15.102:0/1051675225 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f90c8077490 con 0x7f90d015d410 2024-03-23T03:51:14.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.312+0000 7f90d6200640 1 -- 172.21.15.102:0/1051675225 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_write"]} v 0) v1 -- 0x7f90d00cab90 con 0x7f90d015d410 2024-03-23T03:51:14.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.820+0000 7f90c7e00640 1 -- 172.21.15.102:0/1051675225 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_quota.TestQuota.test_remote_update_write"]}]=0 v257) v1 ==== 140+0+0 (secure 0 0 0) 0x7f90c807aab0 con 0x7f90d015d410 2024-03-23T03:51:14.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.824+0000 7f90d6200640 1 -- 172.21.15.102:0/1051675225 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f90ac06ac00 msgr2=0x7f90ac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:14.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.824+0000 7f90d6200640 1 --2- 172.21.15.102:0/1051675225 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f90ac06ac00 0x7f90ac06d0c0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f90c0006fd0 tx=0x7f90c0008040 comp rx=0 tx=0).stop 2024-03-23T03:51:14.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.824+0000 7f90d6200640 1 -- 172.21.15.102:0/1051675225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90d015d410 msgr2=0x7f90d013f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:14.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.824+0000 7f90d6200640 1 --2- 172.21.15.102:0/1051675225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90d015d410 0x7f90d013f650 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f90c8009d90 tx=0x7f90c8004a40 comp rx=0 tx=0).stop 2024-03-23T03:51:14.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.824+0000 7f90d6200640 1 -- 172.21.15.102:0/1051675225 shutdown_connections 2024-03-23T03:51:14.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.824+0000 7f90d6200640 1 --2- 172.21.15.102:0/1051675225 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f90ac06ac00 0x7f90ac06d0c0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:14.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.824+0000 7f90d6200640 1 --2- 172.21.15.102:0/1051675225 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90d015d410 0x7f90d013f650 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:14.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.824+0000 7f90d6200640 1 --2- 172.21.15.102:0/1051675225 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f90d015b560 0x7f90d013f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:14.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.824+0000 7f90d6200640 1 --2- 172.21.15.102:0/1051675225 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f90d00b39b0 0x7f90d01ef970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:14.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.824+0000 7f90d6200640 1 -- 172.21.15.102:0/1051675225 >> 172.21.15.102:0/1051675225 conn(0x7f90d006a3d0 msgr2=0x7f90d00cc200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:14.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.824+0000 7f90d6200640 1 -- 172.21.15.102:0/1051675225 shutdown_connections 2024-03-23T03:51:14.825 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:14.824+0000 7f90d6200640 1 -- 172.21.15.102:0/1051675225 wait complete. 2024-03-23T03:51:14.845 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-23T03:51:15.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.053+0000 7f4666a00640 1 Processor -- start 2024-03-23T03:51:15.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.053+0000 7f4666a00640 1 -- start start 2024-03-23T03:51:15.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.054+0000 7f4666a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46600b2030 0x7f4660069c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:15.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.054+0000 7f4666a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f466006a1d0 0x7f466006ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:15.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.054+0000 7f4666a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46600b33b0 con 0x7f46600b1c50 2024-03-23T03:51:15.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.054+0000 7f4666a00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46600b2f90 con 0x7f466006a1d0 2024-03-23T03:51:15.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.054+0000 7f4666a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f466006a660 con 0x7f46600b2030 2024-03-23T03:51:15.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.054+0000 7f4665600640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f46600b1c50 0x7f46600ca300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60632/0 (socket says 172.21.15.102:60632) 2024-03-23T03:51:15.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.054+0000 7f4665600640 1 -- 172.21.15.102:0/3533104388 learned_addr learned my addr 172.21.15.102:0/3533104388 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:15.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f465fe00640 1 -- 172.21.15.102:0/3533104388 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875276564 0 0) 0x7f46600b33b0 con 0x7f46600b1c50 2024-03-23T03:51:15.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f465fe00640 1 -- 172.21.15.102:0/3533104388 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4644003520 con 0x7f46600b1c50 2024-03-23T03:51:15.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f4666000640 1 --2- 172.21.15.102:0/3533104388 >> v2:172.21.15.173:3300/0 conn(0x7f466006a1d0 0x7f466006ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:15.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f4664c00640 1 --2- 172.21.15.102:0/3533104388 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46600b2030 0x7f4660069c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:15.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f465fe00640 1 -- 172.21.15.102:0/3533104388 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2911975264 0 0) 0x7f4644003520 con 0x7f46600b1c50 2024-03-23T03:51:15.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f465fe00640 1 -- 172.21.15.102:0/3533104388 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46600b2030 msgr2=0x7f4660069c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:15.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f465fe00640 1 --2- 172.21.15.102:0/3533104388 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46600b2030 0x7f4660069c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:15.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f465fe00640 1 -- 172.21.15.102:0/3533104388 >> v2:172.21.15.173:3300/0 conn(0x7f466006a1d0 msgr2=0x7f466006ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:15.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f465fe00640 1 --2- 172.21.15.102:0/3533104388 >> v2:172.21.15.173:3300/0 conn(0x7f466006a1d0 0x7f466006ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:15.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f465fe00640 1 -- 172.21.15.102:0/3533104388 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4660159410 con 0x7f46600b1c50 2024-03-23T03:51:15.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f465fe00640 1 -- 172.21.15.102:0/3533104388 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f4650001670 con 0x7f46600b1c50 2024-03-23T03:51:15.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f465fe00640 1 -- 172.21.15.102:0/3533104388 >> v1:172.21.15.102:6789/0 conn(0x7f46600b1c50 legacy=0x7f46600ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:15.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f465fe00640 1 --2- 172.21.15.102:0/3533104388 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46600b2030 0x7f4644003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:15.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f465fe00640 1 --2- 172.21.15.102:0/3533104388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4644003fa0 0x7f46440047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:15.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f465fe00640 1 --2- 172.21.15.102:0/3533104388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4644004d00 0x7f4644005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:15.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f465fe00640 1 -- 172.21.15.102:0/3533104388 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46600b2f90 con 0x7f4644004d00 2024-03-23T03:51:15.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f465fe00640 1 -- 172.21.15.102:0/3533104388 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466006a660 con 0x7f4644003fa0 2024-03-23T03:51:15.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.055+0000 7f465fe00640 1 -- 172.21.15.102:0/3533104388 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464400bb40 con 0x7f46600b2030 2024-03-23T03:51:15.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.056+0000 7f4666000640 1 --2- 172.21.15.102:0/3533104388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4644004d00 0x7f4644005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:15.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.056+0000 7f4665600640 1 --2- 172.21.15.102:0/3533104388 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46600b2030 0x7f4644003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:15.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.056+0000 7f4664c00640 1 --2- 172.21.15.102:0/3533104388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4644003fa0 0x7f46440047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:15.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.056+0000 7f4666000640 1 -- 172.21.15.102:0/3533104388 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46600b2030 msgr2=0x7f4644003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:15.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.056+0000 7f4666000640 1 --2- 172.21.15.102:0/3533104388 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46600b2030 0x7f4644003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:15.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.056+0000 7f4666000640 1 -- 172.21.15.102:0/3533104388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4644003fa0 msgr2=0x7f46440047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:15.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.056+0000 7f4666000640 1 --2- 172.21.15.102:0/3533104388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4644003fa0 0x7f46440047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:15.059 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.056+0000 7f4666000640 1 -- 172.21.15.102:0/3533104388 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4644003520 con 0x7f4644004d00 2024-03-23T03:51:15.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.056+0000 7f4665600640 1 --2- 172.21.15.102:0/3533104388 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46600b2030 0x7f4644003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:15.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.057+0000 7f4666000640 1 --2- 172.21.15.102:0/3533104388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4644004d00 0x7f4644005180 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f465800ba30 tx=0x7f46440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=698889195fa1625e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:15.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.057+0000 7f465fe00640 1 -- 172.21.15.102:0/3533104388 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4658032680 con 0x7f4644004d00 2024-03-23T03:51:15.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.057+0000 7f465fe00640 1 -- 172.21.15.102:0/3533104388 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46580327e0 con 0x7f4644004d00 2024-03-23T03:51:15.060 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.057+0000 7f465fe00640 1 -- 172.21.15.102:0/3533104388 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4658032b70 con 0x7f4644004d00 2024-03-23T03:51:15.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.058+0000 7f4666a00640 1 -- 172.21.15.102:0/3533104388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4644004d00 msgr2=0x7f4644005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:15.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.058+0000 7f4666a00640 1 --2- 172.21.15.102:0/3533104388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4644004d00 0x7f4644005180 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f465800ba30 tx=0x7f46440057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:15.062 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.058+0000 7f4666000640 1 -- 172.21.15.102:0/3533104388 reap_dead start 2024-03-23T03:51:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.058+0000 7f4666a00640 1 -- 172.21.15.102:0/3533104388 shutdown_connections 2024-03-23T03:51:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.058+0000 7f4666a00640 1 -- 172.21.15.102:0/3533104388 >> 172.21.15.102:0/3533104388 conn(0x7f46600b01b0 msgr2=0x7f46600b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.058+0000 7f4666a00640 1 -- 172.21.15.102:0/3533104388 shutdown_connections 2024-03-23T03:51:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.058+0000 7f4666a00640 1 -- 172.21.15.102:0/3533104388 wait complete. 2024-03-23T03:51:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.059+0000 7f4666a00640 1 Processor -- start 2024-03-23T03:51:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.060+0000 7f4666a00640 1 -- start start 2024-03-23T03:51:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.060+0000 7f4666a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4644004d00 0x7f4660097a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.060+0000 7f4666a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46601514b0 0x7f4660152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.060+0000 7f4666a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4660157bc0 0x7f4660158040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.060+0000 7f4666a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4660097f40 con 0x7f46601514b0 2024-03-23T03:51:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.060+0000 7f4666a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4660158580 con 0x7f4644004d00 2024-03-23T03:51:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.060+0000 7f4666a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4660150520 con 0x7f4660157bc0 2024-03-23T03:51:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.061+0000 7f4664c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46601514b0 0x7f4660152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:15.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.061+0000 7f4664c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46601514b0 0x7f4660152440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53052/0 (socket says 172.21.15.102:53052) 2024-03-23T03:51:15.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.061+0000 7f4664c00640 1 -- 172.21.15.102:0/2907135665 learned_addr learned my addr 172.21.15.102:0/2907135665 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:15.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.061+0000 7f4664c00640 1 -- 172.21.15.102:0/2907135665 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4660157bc0 msgr2=0x7f4660158040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:15.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.061+0000 7f4664c00640 1 --2- 172.21.15.102:0/2907135665 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4660157bc0 0x7f4660158040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:15.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.061+0000 7f4664c00640 1 -- 172.21.15.102:0/2907135665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4644004d00 msgr2=0x7f4660097a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:15.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.061+0000 7f4664c00640 1 --2- 172.21.15.102:0/2907135665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4644004d00 0x7f4660097a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:15.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.061+0000 7f4664c00640 1 -- 172.21.15.102:0/2907135665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4660150800 con 0x7f46601514b0 2024-03-23T03:51:15.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.061+0000 7f4664c00640 1 --2- 172.21.15.102:0/2907135665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46601514b0 0x7f4660152440 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f464c007e60 tx=0x7f464c00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:15.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.061+0000 7f465d600640 1 -- 172.21.15.102:0/2907135665 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c0145f0 con 0x7f46601514b0 2024-03-23T03:51:15.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.061+0000 7f465d600640 1 -- 172.21.15.102:0/2907135665 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464c014750 con 0x7f46601514b0 2024-03-23T03:51:15.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.061+0000 7f465d600640 1 -- 172.21.15.102:0/2907135665 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c014ae0 con 0x7f46601514b0 2024-03-23T03:51:15.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.061+0000 7f4666a00640 1 -- 172.21.15.102:0/2907135665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4660150ae0 con 0x7f46601514b0 2024-03-23T03:51:15.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.062+0000 7f4666a00640 1 -- 172.21.15.102:0/2907135665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46601501f0 con 0x7f46601514b0 2024-03-23T03:51:15.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.063+0000 7f4666a00640 1 -- 172.21.15.102:0/2907135665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f466004e030 con 0x7f46601514b0 2024-03-23T03:51:15.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.064+0000 7f465d600640 1 -- 172.21.15.102:0/2907135665 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f464c014c40 con 0x7f46601514b0 2024-03-23T03:51:15.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.064+0000 7f465d600640 1 --2- 172.21.15.102:0/2907135665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f464006ac30 0x7f464006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:15.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.064+0000 7f465d600640 1 -- 172.21.15.102:0/2907135665 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f464c089f20 con 0x7f46601514b0 2024-03-23T03:51:15.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.065+0000 7f4665600640 1 --2- 172.21.15.102:0/2907135665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f464006ac30 0x7f464006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:15.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.065+0000 7f4665600640 1 --2- 172.21.15.102:0/2907135665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f464006ac30 0x7f464006d0f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4650000ff0 tx=0x7f4650009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:15.069 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.069+0000 7f465d600640 1 -- 172.21.15.102:0/2907135665 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f464c055210 con 0x7f46601514b0 2024-03-23T03:51:15.246 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.246+0000 7f4666a00640 1 -- 172.21.15.102:0/2907135665 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f46600b2470 con 0x7f46601514b0 2024-03-23T03:51:15.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.252+0000 7f465d600640 1 -- 172.21.15.102:0/2907135665 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f464c058830 con 0x7f46601514b0 2024-03-23T03:51:15.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.255+0000 7f4666a00640 1 -- 172.21.15.102:0/2907135665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f464006ac30 msgr2=0x7f464006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:15.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.255+0000 7f4666a00640 1 --2- 172.21.15.102:0/2907135665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f464006ac30 0x7f464006d0f0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4650000ff0 tx=0x7f4650009040 comp rx=0 tx=0).stop 2024-03-23T03:51:15.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.255+0000 7f4666a00640 1 -- 172.21.15.102:0/2907135665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46601514b0 msgr2=0x7f4660152440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:15.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.255+0000 7f4666a00640 1 --2- 172.21.15.102:0/2907135665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46601514b0 0x7f4660152440 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f464c007e60 tx=0x7f464c00f520 comp rx=0 tx=0).stop 2024-03-23T03:51:15.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.255+0000 7f4666a00640 1 -- 172.21.15.102:0/2907135665 shutdown_connections 2024-03-23T03:51:15.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.255+0000 7f4666a00640 1 --2- 172.21.15.102:0/2907135665 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f464006ac30 0x7f464006d0f0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:15.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.255+0000 7f4666a00640 1 --2- 172.21.15.102:0/2907135665 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4660157bc0 0x7f4660158040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:15.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.255+0000 7f4666a00640 1 --2- 172.21.15.102:0/2907135665 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46601514b0 0x7f4660152440 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:15.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.255+0000 7f4666a00640 1 --2- 172.21.15.102:0/2907135665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4644004d00 0x7f4660097a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:15.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.255+0000 7f4666a00640 1 -- 172.21.15.102:0/2907135665 >> 172.21.15.102:0/2907135665 conn(0x7f46600b01b0 msgr2=0x7f466006fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:15.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.255+0000 7f4666a00640 1 -- 172.21.15.102:0/2907135665 shutdown_connections 2024-03-23T03:51:15.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.255+0000 7f4666a00640 1 -- 172.21.15.102:0/2907135665 wait complete. 2024-03-23T03:51:15.272 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T03:51:15.272 DEBUG:teuthology.orchestra.run.smithi102:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:51:15.384 DEBUG:teuthology.orchestra.run.smithi173:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:51:15.505 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T03:51:15.506 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:51:15.506 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:15.562 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:51:15.563 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:51:15.563 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:15.620 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:51:15.620 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:51:15.620 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:15.677 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:51:15.677 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:51:15.677 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:15.733 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:51:15.733 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-23T03:51:15.733 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:51:15.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.945+0000 7fb0e8e00640 1 Processor -- start 2024-03-23T03:51:15.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.945+0000 7fb0e8e00640 1 -- start start 2024-03-23T03:51:15.947 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.947+0000 7fb0e8e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0e4156e30 0x7fb0e406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:15.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.947+0000 7fb0e8e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb0e40b3310 0x7fb0e40cae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:15.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.947+0000 7fb0e8e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e40cc8c0 con 0x7fb0e40cb370 2024-03-23T03:51:15.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.947+0000 7fb0e8e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e40cb7e0 con 0x7fb0e40b3310 2024-03-23T03:51:15.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.947+0000 7fb0e8e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e406d630 con 0x7fb0e4156e30 2024-03-23T03:51:15.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.947+0000 7fb0dbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb0e40b3310 0x7fb0e40cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:15.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.948+0000 7fb0dbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb0e40b3310 0x7fb0e40cae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53054/0 (socket says 172.21.15.102:53054) 2024-03-23T03:51:15.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.948+0000 7fb0dbe00640 1 -- 172.21.15.102:0/3937889872 learned_addr learned my addr 172.21.15.102:0/3937889872 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:15.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.948+0000 7fb0e3400640 1 --2- 172.21.15.102:0/3937889872 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0e4156e30 0x7fb0e406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:15.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.948+0000 7fb0dbe00640 1 -- 172.21.15.102:0/3937889872 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0e4156e30 msgr2=0x7fb0e406d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:15.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.948+0000 7fb0dbe00640 1 --2- 172.21.15.102:0/3937889872 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0e4156e30 0x7fb0e406d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:15.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.948+0000 7fb0dbe00640 1 -- 172.21.15.102:0/3937889872 >> v1:172.21.15.173:6789/0 conn(0x7fb0e40cb370 legacy=0x7fb0e4070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:15.949 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.948+0000 7fb0dbe00640 1 -- 172.21.15.102:0/3937889872 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e415ce60 con 0x7fb0e40b3310 2024-03-23T03:51:15.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.949+0000 7fb0dbe00640 1 --2- 172.21.15.102:0/3937889872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0e40b3310 0x7fb0e40cae30 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009870 tx=0x7fb0cc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24b22851c6395c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:15.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.950+0000 7fb0e2a00640 1 -- 172.21.15.102:0/3937889872 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc030840 con 0x7fb0e40b3310 2024-03-23T03:51:15.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.950+0000 7fb0e2a00640 1 -- 172.21.15.102:0/3937889872 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0cc0309a0 con 0x7fb0e40b3310 2024-03-23T03:51:15.950 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.950+0000 7fb0e2a00640 1 -- 172.21.15.102:0/3937889872 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc030d30 con 0x7fb0e40b3310 2024-03-23T03:51:15.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.950+0000 7fb0e8e00640 1 -- 172.21.15.102:0/3937889872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0e40b3310 msgr2=0x7fb0e40cae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:15.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.950+0000 7fb0e8e00640 1 --2- 172.21.15.102:0/3937889872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0e40b3310 0x7fb0e40cae30 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009870 tx=0x7fb0cc02b980 comp rx=0 tx=0).stop 2024-03-23T03:51:15.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.951+0000 7fb0e8e00640 1 -- 172.21.15.102:0/3937889872 shutdown_connections 2024-03-23T03:51:15.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.951+0000 7fb0e8e00640 1 --2- 172.21.15.102:0/3937889872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0e40b3310 0x7fb0e40cae30 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:15.951 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.951+0000 7fb0e8e00640 1 --2- 172.21.15.102:0/3937889872 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0e4156e30 0x7fb0e406d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:15.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.951+0000 7fb0e8e00640 1 -- 172.21.15.102:0/3937889872 >> 172.21.15.102:0/3937889872 conn(0x7fb0e4065df0 msgr2=0x7fb0e4069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:15.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.951+0000 7fb0e8e00640 1 -- 172.21.15.102:0/3937889872 shutdown_connections 2024-03-23T03:51:15.952 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.951+0000 7fb0e8e00640 1 -- 172.21.15.102:0/3937889872 wait complete. 2024-03-23T03:51:15.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.952+0000 7fb0e8e00640 1 Processor -- start 2024-03-23T03:51:15.953 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.952+0000 7fb0e8e00640 1 -- start start 2024-03-23T03:51:15.954 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.953+0000 7fb0e8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0e40b3310 0x7fb0e40af7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:15.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.953+0000 7fb0e8e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e40cb370 0x7fb0e41515f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:15.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.953+0000 7fb0e8e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0e4156e30 0x7fb0e40b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:15.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.953+0000 7fb0e8e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e41488b0 con 0x7fb0e40b3310 2024-03-23T03:51:15.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.953+0000 7fb0e8e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e4151b30 con 0x7fb0e40cb370 2024-03-23T03:51:15.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.953+0000 7fb0e8e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e40afdb0 con 0x7fb0e4156e30 2024-03-23T03:51:15.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.954+0000 7fb0e3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0e40b3310 0x7fb0e40af7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:15.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.954+0000 7fb0e3400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0e40b3310 0x7fb0e40af7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53064/0 (socket says 172.21.15.102:53064) 2024-03-23T03:51:15.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.954+0000 7fb0e3400640 1 -- 172.21.15.102:0/164632045 learned_addr learned my addr 172.21.15.102:0/164632045 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:15.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.954+0000 7fb0dbe00640 1 --2- 172.21.15.102:0/164632045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e40cb370 0x7fb0e41515f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:15.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.954+0000 7fb0e3400640 1 -- 172.21.15.102:0/164632045 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0e4156e30 msgr2=0x7fb0e40b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:15.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.954+0000 7fb0e3400640 1 --2- 172.21.15.102:0/164632045 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0e4156e30 0x7fb0e40b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:15.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.954+0000 7fb0e3400640 1 -- 172.21.15.102:0/164632045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e40cb370 msgr2=0x7fb0e41515f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:15.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.954+0000 7fb0e3400640 1 --2- 172.21.15.102:0/164632045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e40cb370 0x7fb0e41515f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:15.955 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.954+0000 7fb0e3400640 1 -- 172.21.15.102:0/164632045 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0cc0094d0 con 0x7fb0e40b3310 2024-03-23T03:51:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.954+0000 7fb0e3400640 1 --2- 172.21.15.102:0/164632045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0e40b3310 0x7fb0e40af7d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fb0d400d5f0 tx=0x7fb0d400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.954+0000 7fb0db400640 1 -- 172.21.15.102:0/164632045 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d40125f0 con 0x7fb0e40b3310 2024-03-23T03:51:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.955+0000 7fb0db400640 1 -- 172.21.15.102:0/164632045 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d4012750 con 0x7fb0e40b3310 2024-03-23T03:51:15.956 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.955+0000 7fb0db400640 1 -- 172.21.15.102:0/164632045 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4012b00 con 0x7fb0e40b3310 2024-03-23T03:51:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.955+0000 7fb0e8e00640 1 -- 172.21.15.102:0/164632045 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e40b0090 con 0x7fb0e40b3310 2024-03-23T03:51:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.955+0000 7fb0e8e00640 1 -- 172.21.15.102:0/164632045 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e40b0650 con 0x7fb0e40b3310 2024-03-23T03:51:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.956+0000 7fb0e8e00640 1 -- 172.21.15.102:0/164632045 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e404e030 con 0x7fb0e40b3310 2024-03-23T03:51:15.957 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.956+0000 7fb0db400640 1 -- 172.21.15.102:0/164632045 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb0d4012c60 con 0x7fb0e40b3310 2024-03-23T03:51:15.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.957+0000 7fb0db400640 1 --2- 172.21.15.102:0/164632045 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb0c006ac00 0x7fb0c006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:15.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.957+0000 7fb0db400640 1 -- 172.21.15.102:0/164632045 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fb0d4089720 con 0x7fb0e40b3310 2024-03-23T03:51:15.958 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.958+0000 7fb0dbe00640 1 --2- 172.21.15.102:0/164632045 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb0c006ac00 0x7fb0c006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:15.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.958+0000 7fb0dbe00640 1 --2- 172.21.15.102:0/164632045 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb0c006ac00 0x7fb0c006d0c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc036000 tx=0x7fb0cc03b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:15.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:15.962+0000 7fb0db400640 1 -- 172.21.15.102:0/164632045 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb0d4054a20 con 0x7fb0e40b3310 2024-03-23T03:51:16.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.194+0000 7fb0e8e00640 1 -- 172.21.15.102:0/164632045 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0e4069dc0 con 0x7fb0e40b3310 2024-03-23T03:51:16.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.195+0000 7fb0db400640 1 -- 172.21.15.102:0/164632045 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+3538 (secure 0 0 0) 0x7fb0d4058040 con 0x7fb0e40b3310 2024-03-23T03:51:16.196 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:16.196 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":50,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5195,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6836/755891320","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":755891320},{"type":"v1","addr":"172.21.15.102:6836","nonce":755891320}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":5207,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/3792558756","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":3792558756},{"type":"v1","addr":"172.21.15.173:6835","nonce":3792558756}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":5590,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/1632622692","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":1632622692},{"type":"v1","addr":"172.21.15.173:6833","nonce":1632622692}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":44},{"gid":7137,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/4220954821","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6835","nonce":4220954821},{"type":"v1","addr":"172.21.15.102:6837","nonce":4220954821}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":50}],"filesystems":[]} 2024-03-23T03:51:16.196 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 50 2024-03-23T03:51:16.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.198+0000 7fb0e8e00640 1 -- 172.21.15.102:0/164632045 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb0c006ac00 msgr2=0x7fb0c006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:16.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.198+0000 7fb0e8e00640 1 --2- 172.21.15.102:0/164632045 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb0c006ac00 0x7fb0c006d0c0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc036000 tx=0x7fb0cc03b040 comp rx=0 tx=0).stop 2024-03-23T03:51:16.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.198+0000 7fb0e8e00640 1 -- 172.21.15.102:0/164632045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0e40b3310 msgr2=0x7fb0e40af7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:16.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.198+0000 7fb0e8e00640 1 --2- 172.21.15.102:0/164632045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0e40b3310 0x7fb0e40af7d0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fb0d400d5f0 tx=0x7fb0d400dab0 comp rx=0 tx=0).stop 2024-03-23T03:51:16.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.198+0000 7fb0e8e00640 1 -- 172.21.15.102:0/164632045 shutdown_connections 2024-03-23T03:51:16.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.198+0000 7fb0e8e00640 1 --2- 172.21.15.102:0/164632045 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb0c006ac00 0x7fb0c006d0c0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:16.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.198+0000 7fb0e8e00640 1 --2- 172.21.15.102:0/164632045 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb0e4156e30 0x7fb0e40b0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:16.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.198+0000 7fb0e8e00640 1 --2- 172.21.15.102:0/164632045 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb0e40cb370 0x7fb0e41515f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:16.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.198+0000 7fb0e8e00640 1 --2- 172.21.15.102:0/164632045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb0e40b3310 0x7fb0e40af7d0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:16.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.198+0000 7fb0e8e00640 1 -- 172.21.15.102:0/164632045 >> 172.21.15.102:0/164632045 conn(0x7fb0e4065df0 msgr2=0x7fb0e40cdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:16.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.199+0000 7fb0e8e00640 1 -- 172.21.15.102:0/164632045 shutdown_connections 2024-03-23T03:51:16.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.199+0000 7fb0e8e00640 1 -- 172.21.15.102:0/164632045 wait complete. 2024-03-23T03:51:16.219 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T03:51:16.219 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-23T03:51:16.219 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T03:51:16.219 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:51:16.219 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T03:51:16.219 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-23T03:51:16.220 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T03:51:16.220 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:51:16.220 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T03:51:16.220 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-23T03:51:16.220 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T03:51:16.220 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:51:16.220 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T03:51:16.221 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-23T03:51:16.221 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T03:51:16.221 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:51:16.241 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T03:51:16.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T03:51:16.242 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T03:51:16.242 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T03:51:16.243 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T03:51:16.243 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T03:51:16.248 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T03:51:16.248 DEBUG:teuthology.orchestra.run.smithi173:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T03:51:16.288 INFO:tasks.ceph.mds.c:Started 2024-03-23T03:51:16.289 DEBUG:teuthology.parallel:result is None 2024-03-23T03:51:16.293 INFO:tasks.ceph.mds.d:Started 2024-03-23T03:51:16.293 DEBUG:teuthology.parallel:result is None 2024-03-23T03:51:16.294 INFO:tasks.ceph.mds.a:Started 2024-03-23T03:51:16.294 DEBUG:teuthology.parallel:result is None 2024-03-23T03:51:16.300 INFO:tasks.ceph.mds.b:Started 2024-03-23T03:51:16.300 DEBUG:teuthology.parallel:result is None 2024-03-23T03:51:16.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-23T03:51:16.438 INFO:tasks.ceph.mds.c.smithi102.stdout:starting mds.c at 2024-03-23T03:51:16.448 INFO:tasks.ceph.mds.a.smithi102.stdout:starting mds.a at 2024-03-23T03:51:16.453 INFO:tasks.ceph.mds.d.smithi173.stdout:starting mds.d at 2024-03-23T03:51:16.457 INFO:tasks.ceph.mds.b.smithi173.stdout:starting mds.b at 2024-03-23T03:51:16.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.492+0000 7f332d000640 1 Processor -- start 2024-03-23T03:51:16.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.492+0000 7f332d000640 1 -- start start 2024-03-23T03:51:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.493+0000 7f332d000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3328160dd0 0x7f33280ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.493+0000 7f332d000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f33280cee70 0x7f33281422d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.493+0000 7f332d000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33280cf2c0 con 0x7f332813fa80 2024-03-23T03:51:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.493+0000 7f332d000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f332806cdc0 con 0x7f3328160dd0 2024-03-23T03:51:16.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.493+0000 7f332d000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f332806d060 con 0x7f33280cee70 2024-03-23T03:51:16.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.494+0000 7f3325400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3328160dd0 0x7f33280ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:16.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.494+0000 7f3325400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3328160dd0 0x7f33280ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53092/0 (socket says 172.21.15.102:53092) 2024-03-23T03:51:16.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.494+0000 7f3325400640 1 -- 172.21.15.102:0/729108194 learned_addr learned my addr 172.21.15.102:0/729108194 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:16.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.494+0000 7f3326800640 1 --2- 172.21.15.102:0/729108194 >> v2:172.21.15.173:3300/0 conn(0x7f33280cee70 0x7f33281422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:16.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.494+0000 7f3325400640 1 -- 172.21.15.102:0/729108194 >> v2:172.21.15.173:3300/0 conn(0x7f33280cee70 msgr2=0x7f33281422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:16.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.494+0000 7f3325400640 1 --2- 172.21.15.102:0/729108194 >> v2:172.21.15.173:3300/0 conn(0x7f33280cee70 0x7f33281422d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:16.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.494+0000 7f3325400640 1 -- 172.21.15.102:0/729108194 >> v1:172.21.15.173:6789/0 conn(0x7f332813fa80 legacy=0x7f332813ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:16.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.494+0000 7f3325400640 1 -- 172.21.15.102:0/729108194 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332806d340 con 0x7f3328160dd0 2024-03-23T03:51:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.495+0000 7f3325400640 1 --2- 172.21.15.102:0/729108194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3328160dd0 0x7f33280ce930 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f332806c850 tx=0x7f331402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=705abf3aea18338c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.496+0000 7f3324a00640 1 -- 172.21.15.102:0/729108194 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3314030860 con 0x7f3328160dd0 2024-03-23T03:51:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.496+0000 7f3324a00640 1 -- 172.21.15.102:0/729108194 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33140309c0 con 0x7f3328160dd0 2024-03-23T03:51:16.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.496+0000 7f3324a00640 1 -- 172.21.15.102:0/729108194 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3314030d50 con 0x7f3328160dd0 2024-03-23T03:51:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.496+0000 7f332d000640 1 -- 172.21.15.102:0/729108194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3328160dd0 msgr2=0x7f33280ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.496+0000 7f332d000640 1 --2- 172.21.15.102:0/729108194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3328160dd0 0x7f33280ce930 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f332806c850 tx=0x7f331402b9a0 comp rx=0 tx=0).stop 2024-03-23T03:51:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.496+0000 7f332d000640 1 -- 172.21.15.102:0/729108194 shutdown_connections 2024-03-23T03:51:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.496+0000 7f332d000640 1 --2- 172.21.15.102:0/729108194 >> v2:172.21.15.173:3300/0 conn(0x7f33280cee70 0x7f33281422d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.496+0000 7f332d000640 1 --2- 172.21.15.102:0/729108194 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3328160dd0 0x7f33280ce930 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.496+0000 7f332d000640 1 -- 172.21.15.102:0/729108194 >> 172.21.15.102:0/729108194 conn(0x7f332806a0b0 msgr2=0x7f332806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.496+0000 7f332d000640 1 -- 172.21.15.102:0/729108194 shutdown_connections 2024-03-23T03:51:16.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.496+0000 7f332d000640 1 -- 172.21.15.102:0/729108194 wait complete. 2024-03-23T03:51:16.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.498+0000 7f332d000640 1 Processor -- start 2024-03-23T03:51:16.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.498+0000 7f332d000640 1 -- start start 2024-03-23T03:51:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.499+0000 7f332d000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33280cee70 0x7f3328139b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.499+0000 7f332d000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f332813fa80 0x7f3328156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.499+0000 7f332d000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3328160dd0 0x7f33281e7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.499+0000 7f332d000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3328138b60 con 0x7f3328160dd0 2024-03-23T03:51:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.499+0000 7f332d000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33281396e0 con 0x7f33280cee70 2024-03-23T03:51:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.499+0000 7f332d000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33280b0ff0 con 0x7f332813fa80 2024-03-23T03:51:16.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.499+0000 7f3326800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3328160dd0 0x7f33281e7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:16.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.499+0000 7f3326800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3328160dd0 0x7f33281e7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53094/0 (socket says 172.21.15.102:53094) 2024-03-23T03:51:16.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.499+0000 7f3326800640 1 -- 172.21.15.102:0/1139777489 learned_addr learned my addr 172.21.15.102:0/1139777489 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:16.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.499+0000 7f3325400640 1 --2- 172.21.15.102:0/1139777489 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f332813fa80 0x7f3328156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:16.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.499+0000 7f3325e00640 1 --2- 172.21.15.102:0/1139777489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33280cee70 0x7f3328139b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:16.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.500+0000 7f3326800640 1 -- 172.21.15.102:0/1139777489 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f332813fa80 msgr2=0x7f3328156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:16.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.500+0000 7f3326800640 1 --2- 172.21.15.102:0/1139777489 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f332813fa80 0x7f3328156770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:16.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.500+0000 7f3326800640 1 -- 172.21.15.102:0/1139777489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33280cee70 msgr2=0x7f3328139b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:16.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.500+0000 7f3326800640 1 --2- 172.21.15.102:0/1139777489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33280cee70 0x7f3328139b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:16.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.500+0000 7f3326800640 1 -- 172.21.15.102:0/1139777489 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33140094d0 con 0x7f3328160dd0 2024-03-23T03:51:16.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.500+0000 7f3326800640 1 --2- 172.21.15.102:0/1139777489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3328160dd0 0x7f33281e7480 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f332000efc0 tx=0x7f332000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:16.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.500+0000 7f331e000640 1 -- 172.21.15.102:0/1139777489 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3320014480 con 0x7f3328160dd0 2024-03-23T03:51:16.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.500+0000 7f331e000640 1 -- 172.21.15.102:0/1139777489 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f332000f040 con 0x7f3328160dd0 2024-03-23T03:51:16.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.500+0000 7f331e000640 1 -- 172.21.15.102:0/1139777489 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3320014820 con 0x7f3328160dd0 2024-03-23T03:51:16.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.501+0000 7f332d000640 1 -- 172.21.15.102:0/1139777489 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3328157290 con 0x7f3328160dd0 2024-03-23T03:51:16.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.501+0000 7f332d000640 1 -- 172.21.15.102:0/1139777489 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3328139020 con 0x7f3328160dd0 2024-03-23T03:51:16.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.502+0000 7f332d000640 1 -- 172.21.15.102:0/1139777489 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3328139820 con 0x7f3328160dd0 2024-03-23T03:51:16.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.503+0000 7f331e000640 1 -- 172.21.15.102:0/1139777489 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3320014980 con 0x7f3328160dd0 2024-03-23T03:51:16.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.504+0000 7f331e000640 1 --2- 172.21.15.102:0/1139777489 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f330806ac00 0x7f330806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:16.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.504+0000 7f331e000640 1 -- 172.21.15.102:0/1139777489 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f3320089c10 con 0x7f3328160dd0 2024-03-23T03:51:16.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.504+0000 7f3325e00640 1 --2- 172.21.15.102:0/1139777489 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f330806ac00 0x7f330806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:16.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.505+0000 7f3325e00640 1 --2- 172.21.15.102:0/1139777489 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f330806ac00 0x7f330806d0c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3318000f60 tx=0x7f3318009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:16.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.509+0000 7f331e000640 1 -- 172.21.15.102:0/1139777489 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3328139820 con 0x7f3328160dd0 2024-03-23T03:51:16.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:16.717+0000 7f332d000640 1 -- 172.21.15.102:0/1139777489 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f3328139820 con 0x7f3328160dd0 2024-03-23T03:51:17.486 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.485+0000 7f331e000640 1 -- 172.21.15.102:0/1139777489 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v101) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3328139820 con 0x7f3328160dd0 2024-03-23T03:51:17.487 INFO:teuthology.orchestra.run.smithi102.stderr: removed all blocklist entries 2024-03-23T03:51:17.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.489+0000 7f332d000640 1 -- 172.21.15.102:0/1139777489 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f330806ac00 msgr2=0x7f330806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:17.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.489+0000 7f332d000640 1 --2- 172.21.15.102:0/1139777489 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f330806ac00 0x7f330806d0c0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f3318000f60 tx=0x7f3318009040 comp rx=0 tx=0).stop 2024-03-23T03:51:17.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.489+0000 7f332d000640 1 -- 172.21.15.102:0/1139777489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3328160dd0 msgr2=0x7f33281e7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:17.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.489+0000 7f332d000640 1 --2- 172.21.15.102:0/1139777489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3328160dd0 0x7f33281e7480 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f332000efc0 tx=0x7f332000c490 comp rx=0 tx=0).stop 2024-03-23T03:51:17.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.489+0000 7f332d000640 1 -- 172.21.15.102:0/1139777489 shutdown_connections 2024-03-23T03:51:17.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.489+0000 7f332d000640 1 --2- 172.21.15.102:0/1139777489 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f330806ac00 0x7f330806d0c0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:17.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.489+0000 7f332d000640 1 --2- 172.21.15.102:0/1139777489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3328160dd0 0x7f33281e7480 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:17.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.489+0000 7f332d000640 1 --2- 172.21.15.102:0/1139777489 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f332813fa80 0x7f3328156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:17.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.489+0000 7f332d000640 1 --2- 172.21.15.102:0/1139777489 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33280cee70 0x7f3328139b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:17.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.489+0000 7f332d000640 1 -- 172.21.15.102:0/1139777489 >> 172.21.15.102:0/1139777489 conn(0x7f332806a0b0 msgr2=0x7f332815e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:17.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.490+0000 7f332d000640 1 -- 172.21.15.102:0/1139777489 shutdown_connections 2024-03-23T03:51:17.491 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.490+0000 7f332d000640 1 -- 172.21.15.102:0/1139777489 wait complete. 2024-03-23T03:51:17.509 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-23T03:51:17.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.713+0000 7fca42200640 1 Processor -- start 2024-03-23T03:51:17.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.713+0000 7fca42200640 1 -- start start 2024-03-23T03:51:17.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.715+0000 7fca42200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fca3c0cbab0 0x7fca3c0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:17.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.715+0000 7fca42200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca3c06d390 0x7fca3c06d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:17.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.715+0000 7fca42200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3c0b16a0 con 0x7fca3c0adbc0 2024-03-23T03:51:17.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.715+0000 7fca42200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca3c06dd60 con 0x7fca3c0cbab0 2024-03-23T03:51:17.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.715+0000 7fca42200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c0cbfb0 con 0x7fca3c06d390 2024-03-23T03:51:17.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.715+0000 7fca3b400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fca3c0adbc0 0x7fca3c0b41c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60642/0 (socket says 172.21.15.102:60642) 2024-03-23T03:51:17.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.715+0000 7fca3b400640 1 -- 172.21.15.102:0/3435884669 learned_addr learned my addr 172.21.15.102:0/3435884669 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:17.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.715+0000 7fca3aa00640 1 --2- 172.21.15.102:0/3435884669 >> v2:172.21.15.102:3300/0 conn(0x7fca3c0cbab0 0x7fca3c0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:17.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.716+0000 7fca3aa00640 1 -- 172.21.15.102:0/3435884669 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca3c06d390 msgr2=0x7fca3c06d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:17.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.716+0000 7fca3be00640 1 --2- 172.21.15.102:0/3435884669 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca3c06d390 0x7fca3c06d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:17.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.716+0000 7fca3aa00640 1 --2- 172.21.15.102:0/3435884669 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca3c06d390 0x7fca3c06d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.716+0000 7fca3aa00640 1 -- 172.21.15.102:0/3435884669 >> v1:172.21.15.102:6789/0 conn(0x7fca3c0adbc0 legacy=0x7fca3c0b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.716+0000 7fca3a000640 1 -- 172.21.15.102:0/3435884669 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020508939 0 0) 0x7fca3c0b16a0 con 0x7fca3c0adbc0 2024-03-23T03:51:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.716+0000 7fca3aa00640 1 -- 172.21.15.102:0/3435884669 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c0cc290 con 0x7fca3c0cbab0 2024-03-23T03:51:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.717+0000 7fca3aa00640 1 --2- 172.21.15.102:0/3435884669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c0cbab0 0x7fca3c0b20f0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fca240097a0 tx=0x7fca2402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e385697d898f724b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.717+0000 7fca3a000640 1 -- 172.21.15.102:0/3435884669 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2402e070 con 0x7fca3c0cbab0 2024-03-23T03:51:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.717+0000 7fca3a000640 1 -- 172.21.15.102:0/3435884669 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca240315a0 con 0x7fca3c0cbab0 2024-03-23T03:51:17.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.717+0000 7fca3a000640 1 -- 172.21.15.102:0/3435884669 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca24031930 con 0x7fca3c0cbab0 2024-03-23T03:51:17.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.718+0000 7fca42200640 1 -- 172.21.15.102:0/3435884669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c0cbab0 msgr2=0x7fca3c0b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:17.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.718+0000 7fca42200640 1 --2- 172.21.15.102:0/3435884669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c0cbab0 0x7fca3c0b20f0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fca240097a0 tx=0x7fca2402b840 comp rx=0 tx=0).stop 2024-03-23T03:51:17.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.718+0000 7fca42200640 1 -- 172.21.15.102:0/3435884669 shutdown_connections 2024-03-23T03:51:17.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.718+0000 7fca42200640 1 --2- 172.21.15.102:0/3435884669 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca3c06d390 0x7fca3c06d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:17.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.718+0000 7fca42200640 1 --2- 172.21.15.102:0/3435884669 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c0cbab0 0x7fca3c0b20f0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:17.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.718+0000 7fca42200640 1 -- 172.21.15.102:0/3435884669 >> 172.21.15.102:0/3435884669 conn(0x7fca3c06a3d0 msgr2=0x7fca3c0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:17.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.718+0000 7fca42200640 1 -- 172.21.15.102:0/3435884669 shutdown_connections 2024-03-23T03:51:17.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.718+0000 7fca42200640 1 -- 172.21.15.102:0/3435884669 wait complete. 2024-03-23T03:51:17.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.719+0000 7fca42200640 1 Processor -- start 2024-03-23T03:51:17.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.720+0000 7fca42200640 1 -- start start 2024-03-23T03:51:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.721+0000 7fca42200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c06d390 0x7fca3c0aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.721+0000 7fca42200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca3c0adbc0 0x7fca3c0ac790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.721+0000 7fca42200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca3c0cbab0 0x7fca3c13bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.721+0000 7fca42200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c14ef30 con 0x7fca3c06d390 2024-03-23T03:51:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.721+0000 7fca42200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c0ad140 con 0x7fca3c0adbc0 2024-03-23T03:51:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.721+0000 7fca42200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3c15a000 con 0x7fca3c0cbab0 2024-03-23T03:51:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.721+0000 7fca3b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c06d390 0x7fca3c0aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.721+0000 7fca3b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c06d390 0x7fca3c0aebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53114/0 (socket says 172.21.15.102:53114) 2024-03-23T03:51:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.721+0000 7fca3b400640 1 -- 172.21.15.102:0/2314781495 learned_addr learned my addr 172.21.15.102:0/2314781495 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.721+0000 7fca3b400640 1 -- 172.21.15.102:0/2314781495 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca3c0cbab0 msgr2=0x7fca3c13bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:17.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.721+0000 7fca3b400640 1 --2- 172.21.15.102:0/2314781495 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca3c0cbab0 0x7fca3c13bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:17.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.721+0000 7fca3b400640 1 -- 172.21.15.102:0/2314781495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca3c0adbc0 msgr2=0x7fca3c0ac790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:17.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.721+0000 7fca3b400640 1 --2- 172.21.15.102:0/2314781495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca3c0adbc0 0x7fca3c0ac790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:17.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.721+0000 7fca3b400640 1 -- 172.21.15.102:0/2314781495 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca24009400 con 0x7fca3c06d390 2024-03-23T03:51:17.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.721+0000 7fca3b400640 1 --2- 172.21.15.102:0/2314781495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c06d390 0x7fca3c0aebb0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fca2c00c890 tx=0x7fca2c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:17.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.722+0000 7fca33400640 1 -- 172.21.15.102:0/2314781495 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c00a640 con 0x7fca3c06d390 2024-03-23T03:51:17.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.722+0000 7fca33400640 1 -- 172.21.15.102:0/2314781495 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2c00a7c0 con 0x7fca3c06d390 2024-03-23T03:51:17.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.722+0000 7fca33400640 1 -- 172.21.15.102:0/2314781495 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c00ab70 con 0x7fca3c06d390 2024-03-23T03:51:17.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.722+0000 7fca42200640 1 -- 172.21.15.102:0/2314781495 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca3c15a2e0 con 0x7fca3c06d390 2024-03-23T03:51:17.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.722+0000 7fca42200640 1 -- 172.21.15.102:0/2314781495 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca3c0ad420 con 0x7fca3c06d390 2024-03-23T03:51:17.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.723+0000 7fca42200640 1 -- 172.21.15.102:0/2314781495 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca3c15a800 con 0x7fca3c06d390 2024-03-23T03:51:17.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.724+0000 7fca33400640 1 -- 172.21.15.102:0/2314781495 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fca2c00acd0 con 0x7fca3c06d390 2024-03-23T03:51:17.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.724+0000 7fca33400640 1 --2- 172.21.15.102:0/2314781495 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fca1806ac00 0x7fca1806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:17.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.725+0000 7fca33400640 1 -- 172.21.15.102:0/2314781495 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fca2c088690 con 0x7fca3c06d390 2024-03-23T03:51:17.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.725+0000 7fca3aa00640 1 --2- 172.21.15.102:0/2314781495 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fca1806ac00 0x7fca1806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:17.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.725+0000 7fca3aa00640 1 --2- 172.21.15.102:0/2314781495 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fca1806ac00 0x7fca1806d0c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fca24036000 tx=0x7fca24030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:17.729 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.729+0000 7fca33400640 1 -- 172.21.15.102:0/2314781495 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fca3c15a800 con 0x7fca3c06d390 2024-03-23T03:51:17.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.975+0000 7fca42200640 1 -- 172.21.15.102:0/2314781495 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fca3c15a800 con 0x7fca3c06d390 2024-03-23T03:51:17.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.976+0000 7fca33400640 1 -- 172.21.15.102:0/2314781495 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fca3c15a800 con 0x7fca3c06d390 2024-03-23T03:51:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:17.977 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:51:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: "auth_dump": [ 2024-03-23T03:51:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.a", 2024-03-23T03:51:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5loBTZFRAA7/tTYyZwrDmwKwiU+dtCEw==", 2024-03-23T03:51:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:51:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:51:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:51:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:51:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.977 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.b", 2024-03-23T03:51:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5lJKKlNxAAj5+j99K79IhbeEm7/qC+wg==", 2024-03-23T03:51:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:51:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:51:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:51:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:51:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.c", 2024-03-23T03:51:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5l0b92JhAA5rvs+mdgmrQX/9lI3GW2rg==", 2024-03-23T03:51:17.978 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:51:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:51:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:51:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:51:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mds.d", 2024-03-23T03:51:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5ltjNMDBAAD1h5Nlv2Cp7mJt+DMlBtzw==", 2024-03-23T03:51:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:51:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:51:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:51:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:51:17.979 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.0", 2024-03-23T03:51:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB2UP5lPW6uORAAejiQReOF95oFliFmk/vdQQ==", 2024-03-23T03:51:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:51:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:51:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:51:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.1", 2024-03-23T03:51:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB6UP5lFbbUBxAAIl0ZEYtJ+/oi3W6a4OwMhg==", 2024-03-23T03:51:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:51:17.980 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:51:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:51:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.2", 2024-03-23T03:51:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQB9UP5l2LaQIBAAKJv7jWJAjlUJrm7+KTXi6Q==", 2024-03-23T03:51:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:51:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:51:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:51:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.3", 2024-03-23T03:51:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCAUP5lxjWsKhAAzVO8CXUd+4LLvv4f/5xMOg==", 2024-03-23T03:51:17.981 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:51:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:51:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:51:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.4", 2024-03-23T03:51:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCHUP5lHcd2JRAAwID1TPb+Z9ViH51B0Mv/kQ==", 2024-03-23T03:51:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:51:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:51:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:51:17.982 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.983 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.983 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.983 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.5", 2024-03-23T03:51:17.983 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCLUP5lApp3AxAADslbuzTe/bf5FML1ckE05w==", 2024-03-23T03:51:17.983 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:51:17.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:51:17.983 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:51:17.983 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.983 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.983 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.983 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.6", 2024-03-23T03:51:17.983 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCOUP5lsi7VGxAArhWvXQ1VzJ8oSiOXzq8VsQ==", 2024-03-23T03:51:17.983 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:51:17.983 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:51:17.983 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:51:17.984 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.984 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.984 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.984 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "osd.7", 2024-03-23T03:51:17.984 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQCRUP5lu182JxAA5LaDXfrBkOXEUNAJXFDCJQ==", 2024-03-23T03:51:17.984 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.984 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow profile osd", 2024-03-23T03:51:17.984 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile osd", 2024-03-23T03:51:17.984 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:51:17.984 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.984 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.984 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.984 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.0", 2024-03-23T03:51:17.984 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lxTfoGRAA3WIQ8p1rGSAyJkLZJMk3dA==", 2024-03-23T03:51:17.984 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:51:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow r", 2024-03-23T03:51:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-03-23T03:51:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.1", 2024-03-23T03:51:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lneSHLhAAwn6ULOajHXrOM650HUcLbw==", 2024-03-23T03:51:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:51:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:51:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:51:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:51:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.985 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.2", 2024-03-23T03:51:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5lOIklJBAAhx2d0Xy/j1PG0qsvC3nMqA==", 2024-03-23T03:51:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:51:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:51:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:51:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:51:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.3", 2024-03-23T03:51:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQByUP5l+7BVORAA7PBYS6mkgLA8b4sLCAYgBA==", 2024-03-23T03:51:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow", 2024-03-23T03:51:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow r", 2024-03-23T03:51:17.986 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow rw", 2024-03-23T03:51:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow rwx" 2024-03-23T03:51:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "client.admin", 2024-03-23T03:51:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBwUP5ldlDgBxAAOiqmhC5GulxELun8OiR4lg==", 2024-03-23T03:51:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:51:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mgr": "allow *", 2024-03-23T03:51:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow *", 2024-03-23T03:51:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:51:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mgr.x", 2024-03-23T03:51:17.987 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBxUP5lQXojChAAzepTDh0Pk3fuBZYsFt/+DQ==", 2024-03-23T03:51:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:51:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile mgr", 2024-03-23T03:51:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:51:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-23T03:51:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: "entity": "mgr.y", 2024-03-23T03:51:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: "key": "AQBwUP5lK59iORAACuBF0JaQ8epyu3Mq3slzjg==", 2024-03-23T03:51:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": { 2024-03-23T03:51:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": "allow *", 2024-03-23T03:51:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: "mon": "allow profile mgr", 2024-03-23T03:51:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: "osd": "allow *" 2024-03-23T03:51:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:17.988 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-03-23T03:51:17.988 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:51:17.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.979+0000 7fca42200640 1 -- 172.21.15.102:0/2314781495 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fca1806ac00 msgr2=0x7fca1806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:17.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.979+0000 7fca42200640 1 --2- 172.21.15.102:0/2314781495 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fca1806ac00 0x7fca1806d0c0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fca24036000 tx=0x7fca24030000 comp rx=0 tx=0).stop 2024-03-23T03:51:17.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.979+0000 7fca42200640 1 -- 172.21.15.102:0/2314781495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c06d390 msgr2=0x7fca3c0aebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:17.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.979+0000 7fca42200640 1 --2- 172.21.15.102:0/2314781495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c06d390 0x7fca3c0aebb0 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fca2c00c890 tx=0x7fca2c00cd50 comp rx=0 tx=0).stop 2024-03-23T03:51:17.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.979+0000 7fca42200640 1 -- 172.21.15.102:0/2314781495 shutdown_connections 2024-03-23T03:51:17.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.979+0000 7fca42200640 1 --2- 172.21.15.102:0/2314781495 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fca1806ac00 0x7fca1806d0c0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:17.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.979+0000 7fca42200640 1 --2- 172.21.15.102:0/2314781495 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fca3c0cbab0 0x7fca3c13bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:17.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.979+0000 7fca42200640 1 --2- 172.21.15.102:0/2314781495 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fca3c0adbc0 0x7fca3c0ac790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:17.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.979+0000 7fca42200640 1 --2- 172.21.15.102:0/2314781495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca3c06d390 0x7fca3c0aebb0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:17.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.979+0000 7fca42200640 1 -- 172.21.15.102:0/2314781495 >> 172.21.15.102:0/2314781495 conn(0x7fca3c06a3d0 msgr2=0x7fca3c0726a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:17.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.979+0000 7fca42200640 1 -- 172.21.15.102:0/2314781495 shutdown_connections 2024-03-23T03:51:17.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:17.979+0000 7fca42200640 1 -- 172.21.15.102:0/2314781495 wait complete. 2024-03-23T03:51:17.998 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:51:18.198 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.198+0000 7fb70d600640 1 Processor -- start 2024-03-23T03:51:18.199 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.198+0000 7fb70d600640 1 -- start start 2024-03-23T03:51:18.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.199+0000 7fb70d600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb7080b37f0 0x7fb7080cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:18.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.199+0000 7fb70d600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb7080ccec0 0x7fb70806c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:18.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.199+0000 7fb70d600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb70806da70 con 0x7fb70814aa40 2024-03-23T03:51:18.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.199+0000 7fb70d600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb70806d7d0 con 0x7fb7080b37f0 2024-03-23T03:51:18.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.199+0000 7fb70d600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7080b1d00 con 0x7fb7080ccec0 2024-03-23T03:51:18.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.200+0000 7fb6fe000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb7080b37f0 0x7fb7080cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:18.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.200+0000 7fb6fe000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb7080b37f0 0x7fb7080cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:45550/0 (socket says 172.21.15.102:45550) 2024-03-23T03:51:18.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.200+0000 7fb6fe000640 1 -- 172.21.15.102:0/2057269864 learned_addr learned my addr 172.21.15.102:0/2057269864 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:18.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.200+0000 7fb707400640 1 --2- 172.21.15.102:0/2057269864 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb7080ccec0 0x7fb70806c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:18.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.201+0000 7fb6fe000640 1 -- 172.21.15.102:0/2057269864 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb7080ccec0 msgr2=0x7fb70806c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:18.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.201+0000 7fb6fe000640 1 --2- 172.21.15.102:0/2057269864 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb7080ccec0 0x7fb70806c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.201+0000 7fb6fe000640 1 -- 172.21.15.102:0/2057269864 >> v1:172.21.15.173:6789/0 conn(0x7fb70814aa40 legacy=0x7fb7080cc7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:18.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.201+0000 7fb6fe000640 1 -- 172.21.15.102:0/2057269864 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7080b1fe0 con 0x7fb7080b37f0 2024-03-23T03:51:18.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.202+0000 7fb6fe000640 1 --2- 172.21.15.102:0/2057269864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7080b37f0 0x7fb7080cd7a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0009870 tx=0x7fb6f002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba5f97cd4a52d644 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:18.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.202+0000 7fb706000640 1 -- 172.21.15.102:0/2057269864 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0030840 con 0x7fb7080b37f0 2024-03-23T03:51:18.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.202+0000 7fb706000640 1 -- 172.21.15.102:0/2057269864 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f00309a0 con 0x7fb7080b37f0 2024-03-23T03:51:18.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.202+0000 7fb706000640 1 -- 172.21.15.102:0/2057269864 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0030d30 con 0x7fb7080b37f0 2024-03-23T03:51:18.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.203+0000 7fb70d600640 1 -- 172.21.15.102:0/2057269864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7080b37f0 msgr2=0x7fb7080cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:18.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.203+0000 7fb70d600640 1 --2- 172.21.15.102:0/2057269864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7080b37f0 0x7fb7080cd7a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0009870 tx=0x7fb6f002b980 comp rx=0 tx=0).stop 2024-03-23T03:51:18.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.203+0000 7fb70d600640 1 -- 172.21.15.102:0/2057269864 shutdown_connections 2024-03-23T03:51:18.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.203+0000 7fb70d600640 1 --2- 172.21.15.102:0/2057269864 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb7080ccec0 0x7fb70806c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.203+0000 7fb70d600640 1 --2- 172.21.15.102:0/2057269864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7080b37f0 0x7fb7080cd7a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.203+0000 7fb70d600640 1 -- 172.21.15.102:0/2057269864 >> 172.21.15.102:0/2057269864 conn(0x7fb70806a3d0 msgr2=0x7fb7080b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:18.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.203+0000 7fb70d600640 1 -- 172.21.15.102:0/2057269864 shutdown_connections 2024-03-23T03:51:18.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.203+0000 7fb70d600640 1 -- 172.21.15.102:0/2057269864 wait complete. 2024-03-23T03:51:18.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.205+0000 7fb70d600640 1 Processor -- start 2024-03-23T03:51:18.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.205+0000 7fb70d600640 1 -- start start 2024-03-23T03:51:18.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.206+0000 7fb70d600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7080b37f0 0x7fb7081f5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:18.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.206+0000 7fb70d600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7080ccec0 0x7fb7080a9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:18.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.206+0000 7fb70d600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb70814aa40 0x7fb708148680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:18.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.206+0000 7fb70d600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7081f64c0 con 0x7fb7080ccec0 2024-03-23T03:51:18.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.206+0000 7fb70d600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7081638f0 con 0x7fb7080b37f0 2024-03-23T03:51:18.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.206+0000 7fb70d600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb708162580 con 0x7fb70814aa40 2024-03-23T03:51:18.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.206+0000 7fb6fe000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7080ccec0 0x7fb7080a9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:18.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.206+0000 7fb6fe000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7080ccec0 0x7fb7080a9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53118/0 (socket says 172.21.15.102:53118) 2024-03-23T03:51:18.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.206+0000 7fb6fe000640 1 -- 172.21.15.102:0/350117627 learned_addr learned my addr 172.21.15.102:0/350117627 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:18.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.206+0000 7fb706a00640 1 --2- 172.21.15.102:0/350117627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7080b37f0 0x7fb7081f5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:18.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.206+0000 7fb6fe000640 1 -- 172.21.15.102:0/350117627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb70814aa40 msgr2=0x7fb708148680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:18.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.206+0000 7fb6fe000640 1 --2- 172.21.15.102:0/350117627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb70814aa40 0x7fb708148680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.206+0000 7fb6fe000640 1 -- 172.21.15.102:0/350117627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7080b37f0 msgr2=0x7fb7081f5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:18.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.206+0000 7fb6fe000640 1 --2- 172.21.15.102:0/350117627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7080b37f0 0x7fb7081f5f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.206+0000 7fb6fe000640 1 -- 172.21.15.102:0/350117627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f00094d0 con 0x7fb7080ccec0 2024-03-23T03:51:18.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.207+0000 7fb6fe000640 1 --2- 172.21.15.102:0/350117627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7080ccec0 0x7fb7080a9740 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0000c00 tx=0x7fb6f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:18.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.207+0000 7fb6ff400640 1 -- 172.21.15.102:0/350117627 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0039c10 con 0x7fb7080ccec0 2024-03-23T03:51:18.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.207+0000 7fb6ff400640 1 -- 172.21.15.102:0/350117627 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f0039d70 con 0x7fb7080ccec0 2024-03-23T03:51:18.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.207+0000 7fb6ff400640 1 -- 172.21.15.102:0/350117627 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0043640 con 0x7fb7080ccec0 2024-03-23T03:51:18.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.207+0000 7fb70d600640 1 -- 172.21.15.102:0/350117627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb708162860 con 0x7fb7080ccec0 2024-03-23T03:51:18.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.207+0000 7fb70d600640 1 -- 172.21.15.102:0/350117627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb708162e20 con 0x7fb7080ccec0 2024-03-23T03:51:18.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.208+0000 7fb70d600640 1 -- 172.21.15.102:0/350117627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70806ac20 con 0x7fb7080ccec0 2024-03-23T03:51:18.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.209+0000 7fb6ff400640 1 -- 172.21.15.102:0/350117627 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb6f00437a0 con 0x7fb7080ccec0 2024-03-23T03:51:18.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.210+0000 7fb6ff400640 1 --2- 172.21.15.102:0/350117627 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6e406ac00 0x7fb6e406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:18.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.210+0000 7fb6ff400640 1 -- 172.21.15.102:0/350117627 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb6f003e070 con 0x7fb7080ccec0 2024-03-23T03:51:18.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.210+0000 7fb706a00640 1 --2- 172.21.15.102:0/350117627 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6e406ac00 0x7fb6e406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:18.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.211+0000 7fb706a00640 1 --2- 172.21.15.102:0/350117627 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6e406ac00 0x7fb6e406d0c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8000f60 tx=0x7fb6f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:18.215 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.214+0000 7fb6ff400640 1 -- 172.21.15.102:0/350117627 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb6f00774a0 con 0x7fb7080ccec0 2024-03-23T03:51:18.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.407+0000 7fb70d600640 1 -- 172.21.15.102:0/350117627 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb708073390 con 0x7fb7080ccec0 2024-03-23T03:51:18.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.409+0000 7fb6ff400640 1 -- 172.21.15.102:0/350117627 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+12644 (secure 0 0 0) 0x7fb6f007aac0 con 0x7fb7080ccec0 2024-03-23T03:51:18.410 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:18.410 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":101,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:51:17.476867+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":91,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:51:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.411+0000 7fb70d600640 1 -- 172.21.15.102:0/350117627 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6e406ac00 msgr2=0x7fb6e406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.411+0000 7fb70d600640 1 --2- 172.21.15.102:0/350117627 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6e406ac00 0x7fb6e406d0c0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb6f8000f60 tx=0x7fb6f8009040 comp rx=0 tx=0).stop 2024-03-23T03:51:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.412+0000 7fb70d600640 1 -- 172.21.15.102:0/350117627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7080ccec0 msgr2=0x7fb7080a9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.412+0000 7fb70d600640 1 --2- 172.21.15.102:0/350117627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7080ccec0 0x7fb7080a9740 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0000c00 tx=0x7fb6f0004730 comp rx=0 tx=0).stop 2024-03-23T03:51:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.412+0000 7fb70d600640 1 -- 172.21.15.102:0/350117627 shutdown_connections 2024-03-23T03:51:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.412+0000 7fb70d600640 1 --2- 172.21.15.102:0/350117627 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb6e406ac00 0x7fb6e406d0c0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.412+0000 7fb70d600640 1 --2- 172.21.15.102:0/350117627 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb70814aa40 0x7fb708148680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.412+0000 7fb70d600640 1 --2- 172.21.15.102:0/350117627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7080ccec0 0x7fb7080a9740 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.412+0000 7fb70d600640 1 --2- 172.21.15.102:0/350117627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb7080b37f0 0x7fb7081f5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.412+0000 7fb70d600640 1 -- 172.21.15.102:0/350117627 >> 172.21.15.102:0/350117627 conn(0x7fb70806a3d0 msgr2=0x7fb7080cc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:18.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.412+0000 7fb70d600640 1 -- 172.21.15.102:0/350117627 shutdown_connections 2024-03-23T03:51:18.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.412+0000 7fb70d600640 1 -- 172.21.15.102:0/350117627 wait complete. 2024-03-23T03:51:18.431 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:51:18.432 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:51:18.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.643+0000 7f929da00640 1 Processor -- start 2024-03-23T03:51:18.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.644+0000 7f929da00640 1 -- start start 2024-03-23T03:51:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.645+0000 7f929da00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9298159830 0x7f92980cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.645+0000 7f929da00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f92980ca7b0 0x7f92980cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.645+0000 7f929da00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92980ce930 0x7f92980cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.645+0000 7f929da00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f929806de50 con 0x7f9298159830 2024-03-23T03:51:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.645+0000 7f929da00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92980cf300 con 0x7f92980ca7b0 2024-03-23T03:51:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.645+0000 7f929da00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9298159c10 con 0x7f92980ce930 2024-03-23T03:51:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.646+0000 7f9296a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9298159830 0x7f92980cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.646+0000 7f9296a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9298159830 0x7f92980cc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53126/0 (socket says 172.21.15.102:53126) 2024-03-23T03:51:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.646+0000 7f9296a00640 1 -- 172.21.15.102:0/528607006 learned_addr learned my addr 172.21.15.102:0/528607006 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.646+0000 7f9296a00640 1 -- 172.21.15.102:0/528607006 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92980ce930 msgr2=0x7f92980cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.646+0000 7f9296a00640 1 --2- 172.21.15.102:0/528607006 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92980ce930 0x7f92980cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.646+0000 7f928e000640 1 --2- 172.21.15.102:0/528607006 >> v2:172.21.15.173:3300/0 conn(0x7f92980ca7b0 0x7f92980cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.646+0000 7f9296a00640 1 -- 172.21.15.102:0/528607006 >> v2:172.21.15.173:3300/0 conn(0x7f92980ca7b0 msgr2=0x7f92980cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.646+0000 7f9296a00640 1 --2- 172.21.15.102:0/528607006 >> v2:172.21.15.173:3300/0 conn(0x7f92980ca7b0 0x7f92980cdcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.646+0000 7f9296a00640 1 -- 172.21.15.102:0/528607006 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92980cd840 con 0x7f9298159830 2024-03-23T03:51:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.647+0000 7f9296a00640 1 --2- 172.21.15.102:0/528607006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9298159830 0x7f92980cc3c0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f92980ad950 tx=0x7f928802e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8322b8aaacc256f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.648+0000 7f9296000640 1 -- 172.21.15.102:0/528607006 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9288036440 con 0x7f9298159830 2024-03-23T03:51:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.648+0000 7f9296000640 1 -- 172.21.15.102:0/528607006 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92880365a0 con 0x7f9298159830 2024-03-23T03:51:18.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.648+0000 7f9296000640 1 -- 172.21.15.102:0/528607006 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9288036930 con 0x7f9298159830 2024-03-23T03:51:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.648+0000 7f929da00640 1 -- 172.21.15.102:0/528607006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9298159830 msgr2=0x7f92980cc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.648+0000 7f929da00640 1 --2- 172.21.15.102:0/528607006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9298159830 0x7f92980cc3c0 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f92980ad950 tx=0x7f928802e170 comp rx=0 tx=0).stop 2024-03-23T03:51:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.648+0000 7f929da00640 1 -- 172.21.15.102:0/528607006 shutdown_connections 2024-03-23T03:51:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.648+0000 7f929da00640 1 --2- 172.21.15.102:0/528607006 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92980ce930 0x7f92980cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.648+0000 7f929da00640 1 --2- 172.21.15.102:0/528607006 >> v2:172.21.15.173:3300/0 conn(0x7f92980ca7b0 0x7f92980cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.648+0000 7f929da00640 1 --2- 172.21.15.102:0/528607006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9298159830 0x7f92980cc3c0 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.648+0000 7f929da00640 1 -- 172.21.15.102:0/528607006 >> 172.21.15.102:0/528607006 conn(0x7f929806a0b0 msgr2=0x7f929806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.649+0000 7f929da00640 1 -- 172.21.15.102:0/528607006 shutdown_connections 2024-03-23T03:51:18.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.649+0000 7f929da00640 1 -- 172.21.15.102:0/528607006 wait complete. 2024-03-23T03:51:18.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.650+0000 7f929da00640 1 Processor -- start 2024-03-23T03:51:18.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.650+0000 7f929da00640 1 -- start start 2024-03-23T03:51:18.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.651+0000 7f929da00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92980ca7b0 0x7f92981342a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:18.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.651+0000 7f929da00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92980ce930 0x7f92980b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:18.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.651+0000 7f929da00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9298159830 0x7f92980b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:18.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.651+0000 7f929da00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92980ac2c0 con 0x7f92980ce930 2024-03-23T03:51:18.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.651+0000 7f929da00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9298137240 con 0x7f9298159830 2024-03-23T03:51:18.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.651+0000 7f929da00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92981529c0 con 0x7f92980ca7b0 2024-03-23T03:51:18.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.652+0000 7f928e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92980ce930 0x7f92980b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:18.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.652+0000 7f928e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92980ce930 0x7f92980b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53132/0 (socket says 172.21.15.102:53132) 2024-03-23T03:51:18.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.652+0000 7f928e000640 1 -- 172.21.15.102:0/3852466714 learned_addr learned my addr 172.21.15.102:0/3852466714 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:18.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.652+0000 7f9296a00640 1 --2- 172.21.15.102:0/3852466714 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92980ca7b0 0x7f92981342a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:18.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.652+0000 7f9297400640 1 --2- 172.21.15.102:0/3852466714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9298159830 0x7f92980b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:18.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.652+0000 7f928e000640 1 -- 172.21.15.102:0/3852466714 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92980ca7b0 msgr2=0x7f92981342a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:18.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.652+0000 7f928e000640 1 --2- 172.21.15.102:0/3852466714 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92980ca7b0 0x7f92981342a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.652+0000 7f928e000640 1 -- 172.21.15.102:0/3852466714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9298159830 msgr2=0x7f92980b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:18.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.652+0000 7f928e000640 1 --2- 172.21.15.102:0/3852466714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9298159830 0x7f92980b0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.652+0000 7f928e000640 1 -- 172.21.15.102:0/3852466714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f928800b770 con 0x7f92980ce930 2024-03-23T03:51:18.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.652+0000 7f9296a00640 1 --2- 172.21.15.102:0/3852466714 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92980ca7b0 0x7f92981342a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:18.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.653+0000 7f928e000640 1 --2- 172.21.15.102:0/3852466714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92980ce930 0x7f92980b0870 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f928000c7f0 tx=0x7f928000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:18.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.653+0000 7f928f400640 1 -- 172.21.15.102:0/3852466714 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9280004220 con 0x7f92980ce930 2024-03-23T03:51:18.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.653+0000 7f928f400640 1 -- 172.21.15.102:0/3852466714 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92800043a0 con 0x7f92980ce930 2024-03-23T03:51:18.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.653+0000 7f928f400640 1 -- 172.21.15.102:0/3852466714 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9280007860 con 0x7f92980ce930 2024-03-23T03:51:18.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.653+0000 7f929da00640 1 -- 172.21.15.102:0/3852466714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9298152ca0 con 0x7f92980ce930 2024-03-23T03:51:18.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.653+0000 7f929da00640 1 -- 172.21.15.102:0/3852466714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92981347e0 con 0x7f92980ce930 2024-03-23T03:51:18.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.654+0000 7f929da00640 1 -- 172.21.15.102:0/3852466714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92980ccef0 con 0x7f92980ce930 2024-03-23T03:51:18.655 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.655+0000 7f928f400640 1 -- 172.21.15.102:0/3852466714 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9280007bd0 con 0x7f92980ce930 2024-03-23T03:51:18.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.656+0000 7f928f400640 1 --2- 172.21.15.102:0/3852466714 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f927406ac00 0x7f927406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:18.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.656+0000 7f928f400640 1 -- 172.21.15.102:0/3852466714 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f92800890b0 con 0x7f92980ce930 2024-03-23T03:51:18.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.656+0000 7f9296a00640 1 --2- 172.21.15.102:0/3852466714 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f927406ac00 0x7f927406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:18.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.657+0000 7f9296a00640 1 --2- 172.21.15.102:0/3852466714 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f927406ac00 0x7f927406d0c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9288038000 tx=0x7f9288008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:18.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.661+0000 7f928f400640 1 -- 172.21.15.102:0/3852466714 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f92980ccef0 con 0x7f92980ce930 2024-03-23T03:51:18.853 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.853+0000 7f929da00640 1 -- 172.21.15.102:0/3852466714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f92980ccef0 con 0x7f92980ce930 2024-03-23T03:51:18.854 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.854+0000 7f928f400640 1 -- 172.21.15.102:0/3852466714 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) v1 ==== 93+0+10 (secure 0 0 0) 0x7f92980ccef0 con 0x7f92980ce930 2024-03-23T03:51:18.854 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:51:18.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.856+0000 7f929da00640 1 -- 172.21.15.102:0/3852466714 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f927406ac00 msgr2=0x7f927406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:18.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.856+0000 7f929da00640 1 --2- 172.21.15.102:0/3852466714 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f927406ac00 0x7f927406d0c0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f9288038000 tx=0x7f9288008040 comp rx=0 tx=0).stop 2024-03-23T03:51:18.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.856+0000 7f929da00640 1 -- 172.21.15.102:0/3852466714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92980ce930 msgr2=0x7f92980b0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:18.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.856+0000 7f929da00640 1 --2- 172.21.15.102:0/3852466714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92980ce930 0x7f92980b0870 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f928000c7f0 tx=0x7f928000ccb0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.856+0000 7f929da00640 1 -- 172.21.15.102:0/3852466714 shutdown_connections 2024-03-23T03:51:18.857 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.856+0000 7f929da00640 1 --2- 172.21.15.102:0/3852466714 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f927406ac00 0x7f927406d0c0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.856+0000 7f929da00640 1 --2- 172.21.15.102:0/3852466714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9298159830 0x7f92980b0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.856+0000 7f929da00640 1 --2- 172.21.15.102:0/3852466714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f92980ce930 0x7f92980b0870 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.856+0000 7f929da00640 1 --2- 172.21.15.102:0/3852466714 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92980ca7b0 0x7f92981342a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:18.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.856+0000 7f929da00640 1 -- 172.21.15.102:0/3852466714 >> 172.21.15.102:0/3852466714 conn(0x7f929806a0b0 msgr2=0x7f9298070130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:18.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.857+0000 7f929da00640 1 -- 172.21.15.102:0/3852466714 shutdown_connections 2024-03-23T03:51:18.858 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:18.857+0000 7f929da00640 1 -- 172.21.15.102:0/3852466714 wait complete. 2024-03-23T03:51:18.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T03:51:19.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.085+0000 7fb38c000640 1 Processor -- start 2024-03-23T03:51:19.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.085+0000 7fb38c000640 1 -- start start 2024-03-23T03:51:19.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.086+0000 7fb38c000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb38415b560 0x7fb38415b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:19.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.087+0000 7fb38c000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb3840b39b0 0x7fb3840b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:19.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.087+0000 7fb38c000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb38406c6f0 con 0x7fb38415e500 2024-03-23T03:51:19.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.087+0000 7fb38c000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3840b4370 con 0x7fb3840b39b0 2024-03-23T03:51:19.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.087+0000 7fb38c000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb38406c9d0 con 0x7fb38415b560 2024-03-23T03:51:19.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.087+0000 7fb388c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb3840b39b0 0x7fb3840b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:19.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.087+0000 7fb388c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fb3840b39b0 0x7fb3840b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53140/0 (socket says 172.21.15.102:53140) 2024-03-23T03:51:19.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.087+0000 7fb388c00640 1 -- 172.21.15.102:0/3914676767 learned_addr learned my addr 172.21.15.102:0/3914676767 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:19.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.087+0000 7fb38a000640 1 --1- 172.21.15.102:0/3914676767 >> v1:172.21.15.102:6789/0 conn(0x7fb38415e500 0x7fb38415e990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60658/0 (socket says 172.21.15.102:60658) 2024-03-23T03:51:19.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.087+0000 7fb388c00640 1 -- 172.21.15.102:0/3914676767 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb38415b560 msgr2=0x7fb38415b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:19.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.087+0000 7fb383e00640 1 -- 172.21.15.102:0/3914676767 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866536757 0 0) 0x7fb38406c6f0 con 0x7fb38415e500 2024-03-23T03:51:19.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.087+0000 7fb389600640 1 --2- 172.21.15.102:0/3914676767 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb38415b560 0x7fb38415b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:19.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.088+0000 7fb388c00640 1 --2- 172.21.15.102:0/3914676767 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb38415b560 0x7fb38415b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:19.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.088+0000 7fb388c00640 1 -- 172.21.15.102:0/3914676767 >> v1:172.21.15.102:6789/0 conn(0x7fb38415e500 legacy=0x7fb38415e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:19.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.088+0000 7fb388c00640 1 -- 172.21.15.102:0/3914676767 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38406ccb0 con 0x7fb3840b39b0 2024-03-23T03:51:19.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.088+0000 7fb389600640 1 --2- 172.21.15.102:0/3914676767 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb38415b560 0x7fb38415b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:19.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.089+0000 7fb388c00640 1 --2- 172.21.15.102:0/3914676767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3840b39b0 0x7fb3840b3e30 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fb370009870 tx=0x7fb37002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd2c89bb1983d80 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:19.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.089+0000 7fb383e00640 1 -- 172.21.15.102:0/3914676767 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb370030840 con 0x7fb3840b39b0 2024-03-23T03:51:19.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.089+0000 7fb383e00640 1 -- 172.21.15.102:0/3914676767 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3700309a0 con 0x7fb3840b39b0 2024-03-23T03:51:19.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.089+0000 7fb383e00640 1 -- 172.21.15.102:0/3914676767 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb370030d30 con 0x7fb3840b39b0 2024-03-23T03:51:19.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.090+0000 7fb38c000640 1 -- 172.21.15.102:0/3914676767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3840b39b0 msgr2=0x7fb3840b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:19.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.090+0000 7fb38c000640 1 --2- 172.21.15.102:0/3914676767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3840b39b0 0x7fb3840b3e30 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fb370009870 tx=0x7fb37002b980 comp rx=0 tx=0).stop 2024-03-23T03:51:19.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.090+0000 7fb38c000640 1 -- 172.21.15.102:0/3914676767 shutdown_connections 2024-03-23T03:51:19.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.090+0000 7fb38c000640 1 --2- 172.21.15.102:0/3914676767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3840b39b0 0x7fb3840b3e30 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:19.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.090+0000 7fb38c000640 1 --2- 172.21.15.102:0/3914676767 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb38415b560 0x7fb38415b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:19.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.090+0000 7fb38c000640 1 -- 172.21.15.102:0/3914676767 >> 172.21.15.102:0/3914676767 conn(0x7fb38406a3d0 msgr2=0x7fb3840b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:19.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.090+0000 7fb38c000640 1 -- 172.21.15.102:0/3914676767 shutdown_connections 2024-03-23T03:51:19.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.090+0000 7fb38c000640 1 -- 172.21.15.102:0/3914676767 wait complete. 2024-03-23T03:51:19.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.092+0000 7fb38c000640 1 Processor -- start 2024-03-23T03:51:19.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.092+0000 7fb38c000640 1 -- start start 2024-03-23T03:51:19.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.093+0000 7fb38c000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3840b39b0 0x7fb3841f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:19.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.093+0000 7fb38c000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb38415b560 0x7fb384159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:19.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.093+0000 7fb38c000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb38415e500 0x7fb384159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:19.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.093+0000 7fb38c000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3841536a0 con 0x7fb3840b39b0 2024-03-23T03:51:19.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.093+0000 7fb38c000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb384153980 con 0x7fb38415b560 2024-03-23T03:51:19.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.093+0000 7fb38c000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3840b2190 con 0x7fb38415e500 2024-03-23T03:51:19.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.093+0000 7fb389600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3840b39b0 0x7fb3841f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:19.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.093+0000 7fb389600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3840b39b0 0x7fb3841f37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53156/0 (socket says 172.21.15.102:53156) 2024-03-23T03:51:19.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.093+0000 7fb389600640 1 -- 172.21.15.102:0/3283950824 learned_addr learned my addr 172.21.15.102:0/3283950824 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:19.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.093+0000 7fb389600640 1 -- 172.21.15.102:0/3283950824 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb38415e500 msgr2=0x7fb384159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:19.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.093+0000 7fb389600640 1 --2- 172.21.15.102:0/3283950824 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb38415e500 0x7fb384159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:19.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.093+0000 7fb388c00640 1 --2- 172.21.15.102:0/3283950824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb38415b560 0x7fb384159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:19.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.093+0000 7fb389600640 1 -- 172.21.15.102:0/3283950824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb38415b560 msgr2=0x7fb384159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:19.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.093+0000 7fb389600640 1 --2- 172.21.15.102:0/3283950824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb38415b560 0x7fb384159930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:19.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.093+0000 7fb389600640 1 -- 172.21.15.102:0/3283950824 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3700094d0 con 0x7fb3840b39b0 2024-03-23T03:51:19.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.094+0000 7fb389600640 1 --2- 172.21.15.102:0/3283950824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3840b39b0 0x7fb3841f37a0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fb37800c7f0 tx=0x7fb37800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:19.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.094+0000 7fb381600640 1 -- 172.21.15.102:0/3283950824 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb378007a30 con 0x7fb3840b39b0 2024-03-23T03:51:19.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.094+0000 7fb381600640 1 -- 172.21.15.102:0/3283950824 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb378007b90 con 0x7fb3840b39b0 2024-03-23T03:51:19.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.094+0000 7fb381600640 1 -- 172.21.15.102:0/3283950824 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb37800ce40 con 0x7fb3840b39b0 2024-03-23T03:51:19.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.094+0000 7fb38c000640 1 -- 172.21.15.102:0/3283950824 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3840b2470 con 0x7fb3840b39b0 2024-03-23T03:51:19.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.094+0000 7fb38c000640 1 -- 172.21.15.102:0/3283950824 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3841f3ce0 con 0x7fb3840b39b0 2024-03-23T03:51:19.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.095+0000 7fb38c000640 1 -- 172.21.15.102:0/3283950824 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3840b1ff0 con 0x7fb3840b39b0 2024-03-23T03:51:19.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.096+0000 7fb381600640 1 -- 172.21.15.102:0/3283950824 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb3780113f0 con 0x7fb3840b39b0 2024-03-23T03:51:19.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.097+0000 7fb381600640 1 --2- 172.21.15.102:0/3283950824 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb36406ac30 0x7fb36406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:19.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.097+0000 7fb381600640 1 -- 172.21.15.102:0/3283950824 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb3780893c0 con 0x7fb3840b39b0 2024-03-23T03:51:19.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.097+0000 7fb388c00640 1 --2- 172.21.15.102:0/3283950824 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb36406ac30 0x7fb36406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:19.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.098+0000 7fb388c00640 1 --2- 172.21.15.102:0/3283950824 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb36406ac30 0x7fb36406d0f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb370036000 tx=0x7fb37003b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:19.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.101+0000 7fb381600640 1 -- 172.21.15.102:0/3283950824 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb3840b1ff0 con 0x7fb3840b39b0 2024-03-23T03:51:19.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.297+0000 7fb38c000640 1 -- 172.21.15.102:0/3283950824 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb3840b2990 con 0x7fb3840b39b0 2024-03-23T03:51:19.298 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.298+0000 7fb381600640 1 -- 172.21.15.102:0/3283950824 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v101) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb37801f060 con 0x7fb3840b39b0 2024-03-23T03:51:19.299 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:19.299 INFO:teuthology.orchestra.run.smithi102.stdout:[ 2024-03-23T03:51:19.299 INFO:teuthology.orchestra.run.smithi102.stdout: { 2024-03-23T03:51:19.299 INFO:teuthology.orchestra.run.smithi102.stdout: "poolnum": 1, 2024-03-23T03:51:19.299 INFO:teuthology.orchestra.run.smithi102.stdout: "poolname": ".mgr" 2024-03-23T03:51:19.299 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-23T03:51:19.299 INFO:teuthology.orchestra.run.smithi102.stdout:] 2024-03-23T03:51:19.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.301+0000 7fb38c000640 1 -- 172.21.15.102:0/3283950824 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb36406ac30 msgr2=0x7fb36406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:19.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.301+0000 7fb38c000640 1 --2- 172.21.15.102:0/3283950824 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb36406ac30 0x7fb36406d0f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fb370036000 tx=0x7fb37003b040 comp rx=0 tx=0).stop 2024-03-23T03:51:19.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.301+0000 7fb38c000640 1 -- 172.21.15.102:0/3283950824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3840b39b0 msgr2=0x7fb3841f37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:19.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.301+0000 7fb38c000640 1 --2- 172.21.15.102:0/3283950824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3840b39b0 0x7fb3841f37a0 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fb37800c7f0 tx=0x7fb37800cbf0 comp rx=0 tx=0).stop 2024-03-23T03:51:19.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.301+0000 7fb38c000640 1 -- 172.21.15.102:0/3283950824 shutdown_connections 2024-03-23T03:51:19.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.301+0000 7fb38c000640 1 --2- 172.21.15.102:0/3283950824 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb36406ac30 0x7fb36406d0f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:19.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.301+0000 7fb38c000640 1 --2- 172.21.15.102:0/3283950824 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb38415e500 0x7fb384159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:19.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.301+0000 7fb38c000640 1 --2- 172.21.15.102:0/3283950824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb38415b560 0x7fb384159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:19.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.301+0000 7fb38c000640 1 --2- 172.21.15.102:0/3283950824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3840b39b0 0x7fb3841f37a0 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:19.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.301+0000 7fb38c000640 1 -- 172.21.15.102:0/3283950824 >> 172.21.15.102:0/3283950824 conn(0x7fb38406a3d0 msgr2=0x7fb384142af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:19.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.301+0000 7fb38c000640 1 -- 172.21.15.102:0/3283950824 shutdown_connections 2024-03-23T03:51:19.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.301+0000 7fb38c000640 1 -- 172.21.15.102:0/3283950824 wait complete. 2024-03-23T03:51:19.319 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T03:51:19.320 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T03:51:19.533 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.533+0000 7f11ec600640 1 Processor -- start 2024-03-23T03:51:19.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.533+0000 7f11ec600640 1 -- start start 2024-03-23T03:51:19.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.534+0000 7f11ec600640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f11e4160fc0 0x7f11e406ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:19.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.534+0000 7f11ec600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f11e406b3c0 0x7f11e40c3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:19.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.534+0000 7f11ec600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f11e40c3ba0 0x7f11e40b0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:19.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.534+0000 7f11ec600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e40c3ff0 con 0x7f11e40c3ba0 2024-03-23T03:51:19.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.534+0000 7f11ec600640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e41613a0 con 0x7f11e4160fc0 2024-03-23T03:51:19.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.534+0000 7f11ec600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e4098d90 con 0x7f11e406b3c0 2024-03-23T03:51:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.535+0000 7f11ea600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f11e40c3ba0 0x7f11e40b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.535+0000 7f11ea600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f11e40c3ba0 0x7f11e40b0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53172/0 (socket says 172.21.15.102:53172) 2024-03-23T03:51:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.535+0000 7f11ea600640 1 -- 172.21.15.102:0/2474531309 learned_addr learned my addr 172.21.15.102:0/2474531309 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.535+0000 7f11e9c00640 1 --2- 172.21.15.102:0/2474531309 >> v2:172.21.15.173:3300/0 conn(0x7f11e4160fc0 0x7f11e406ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.535+0000 7f11e9200640 1 --2- 172.21.15.102:0/2474531309 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f11e406b3c0 0x7f11e40c3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.535+0000 7f11ea600640 1 -- 172.21.15.102:0/2474531309 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f11e406b3c0 msgr2=0x7f11e40c3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.535+0000 7f11ea600640 1 --2- 172.21.15.102:0/2474531309 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f11e406b3c0 0x7f11e40c3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.535+0000 7f11ea600640 1 -- 172.21.15.102:0/2474531309 >> v2:172.21.15.173:3300/0 conn(0x7f11e4160fc0 msgr2=0x7f11e406ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:19.536 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.535+0000 7f11ea600640 1 --2- 172.21.15.102:0/2474531309 >> v2:172.21.15.173:3300/0 conn(0x7f11e4160fc0 0x7f11e406ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.535+0000 7f11ea600640 1 -- 172.21.15.102:0/2474531309 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e4099070 con 0x7f11e40c3ba0 2024-03-23T03:51:19.537 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.535+0000 7f11e9c00640 1 --2- 172.21.15.102:0/2474531309 >> v2:172.21.15.173:3300/0 conn(0x7f11e4160fc0 0x7f11e406ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.536+0000 7f11ea600640 1 --2- 172.21.15.102:0/2474531309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11e40c3ba0 0x7f11e40b0070 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f11e4071640 tx=0x7f11dc02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83cf48272bd92305 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.537+0000 7f11e3e00640 1 -- 172.21.15.102:0/2474531309 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11dc031880 con 0x7f11e40c3ba0 2024-03-23T03:51:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.537+0000 7f11e3e00640 1 -- 172.21.15.102:0/2474531309 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11dc0319e0 con 0x7f11e40c3ba0 2024-03-23T03:51:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.537+0000 7f11e3e00640 1 -- 172.21.15.102:0/2474531309 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11dc031d70 con 0x7f11e40c3ba0 2024-03-23T03:51:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.537+0000 7f11ec600640 1 -- 172.21.15.102:0/2474531309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11e40c3ba0 msgr2=0x7f11e40b0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.537+0000 7f11ec600640 1 --2- 172.21.15.102:0/2474531309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11e40c3ba0 0x7f11e40b0070 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f11e4071640 tx=0x7f11dc02b920 comp rx=0 tx=0).stop 2024-03-23T03:51:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.537+0000 7f11ec600640 1 -- 172.21.15.102:0/2474531309 shutdown_connections 2024-03-23T03:51:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.537+0000 7f11ec600640 1 --2- 172.21.15.102:0/2474531309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11e40c3ba0 0x7f11e40b0070 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.537+0000 7f11ec600640 1 --2- 172.21.15.102:0/2474531309 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f11e406b3c0 0x7f11e40c3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:19.538 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.537+0000 7f11ec600640 1 --2- 172.21.15.102:0/2474531309 >> v2:172.21.15.173:3300/0 conn(0x7f11e4160fc0 0x7f11e406ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:19.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.537+0000 7f11ec600640 1 -- 172.21.15.102:0/2474531309 >> 172.21.15.102:0/2474531309 conn(0x7f11e40b2180 msgr2=0x7f11e40b2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:19.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.537+0000 7f11ec600640 1 -- 172.21.15.102:0/2474531309 shutdown_connections 2024-03-23T03:51:19.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.537+0000 7f11ec600640 1 -- 172.21.15.102:0/2474531309 wait complete. 2024-03-23T03:51:19.539 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.539+0000 7f11ec600640 1 Processor -- start 2024-03-23T03:51:19.540 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.539+0000 7f11ec600640 1 -- start start 2024-03-23T03:51:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.540+0000 7f11ec600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11e406b3c0 0x7f11e40b4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.540+0000 7f11ec600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11e40c3ba0 0x7f11e41e7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.540+0000 7f11ec600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f11e4160fc0 0x7f11e415fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.540+0000 7f11ec600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e40b4b20 con 0x7f11e40c3ba0 2024-03-23T03:51:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.540+0000 7f11ec600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e41e8a50 con 0x7f11e406b3c0 2024-03-23T03:51:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.540+0000 7f11ec600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e41e8770 con 0x7f11e4160fc0 2024-03-23T03:51:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.540+0000 7f11e9200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11e40c3ba0 0x7f11e41e7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.541+0000 7f11e9200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11e40c3ba0 0x7f11e41e7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53186/0 (socket says 172.21.15.102:53186) 2024-03-23T03:51:19.541 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.541+0000 7f11e9200640 1 -- 172.21.15.102:0/3092736841 learned_addr learned my addr 172.21.15.102:0/3092736841 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:19.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.541+0000 7f11e9c00640 1 --2- 172.21.15.102:0/3092736841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11e406b3c0 0x7f11e40b4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:19.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.541+0000 7f11ea600640 1 --2- 172.21.15.102:0/3092736841 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f11e4160fc0 0x7f11e415fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:19.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.541+0000 7f11e9200640 1 -- 172.21.15.102:0/3092736841 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f11e4160fc0 msgr2=0x7f11e415fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:19.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.541+0000 7f11e9200640 1 --2- 172.21.15.102:0/3092736841 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f11e4160fc0 0x7f11e415fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:19.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.541+0000 7f11e9200640 1 -- 172.21.15.102:0/3092736841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11e406b3c0 msgr2=0x7f11e40b4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:19.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.541+0000 7f11e9200640 1 --2- 172.21.15.102:0/3092736841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11e406b3c0 0x7f11e40b4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:19.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.541+0000 7f11e9200640 1 -- 172.21.15.102:0/3092736841 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc0094d0 con 0x7f11e40c3ba0 2024-03-23T03:51:19.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.541+0000 7f11e9c00640 1 --2- 172.21.15.102:0/3092736841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11e406b3c0 0x7f11e40b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:19.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.541+0000 7f11e9200640 1 --2- 172.21.15.102:0/3092736841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11e40c3ba0 0x7f11e41e7e20 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f11d000c7f0 tx=0x7f11d000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:19.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.541+0000 7f11e1600640 1 -- 172.21.15.102:0/3092736841 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d0004220 con 0x7f11e40c3ba0 2024-03-23T03:51:19.543 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.541+0000 7f11e1600640 1 -- 172.21.15.102:0/3092736841 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d0004380 con 0x7f11e40c3ba0 2024-03-23T03:51:19.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.541+0000 7f11e1600640 1 -- 172.21.15.102:0/3092736841 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d0007860 con 0x7f11e40c3ba0 2024-03-23T03:51:19.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.542+0000 7f11ec600640 1 -- 172.21.15.102:0/3092736841 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e40b4460 con 0x7f11e40c3ba0 2024-03-23T03:51:19.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.542+0000 7f11ec600640 1 -- 172.21.15.102:0/3092736841 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e40b5480 con 0x7f11e40c3ba0 2024-03-23T03:51:19.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.543+0000 7f11ec600640 1 -- 172.21.15.102:0/3092736841 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e40995f0 con 0x7f11e40c3ba0 2024-03-23T03:51:19.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.544+0000 7f11e1600640 1 -- 172.21.15.102:0/3092736841 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f11d0007bd0 con 0x7f11e40c3ba0 2024-03-23T03:51:19.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.544+0000 7f11e1600640 1 --2- 172.21.15.102:0/3092736841 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f11c406ac00 0x7f11c406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:19.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.545+0000 7f11e1600640 1 -- 172.21.15.102:0/3092736841 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f11d0089150 con 0x7f11e40c3ba0 2024-03-23T03:51:19.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.545+0000 7f11e9c00640 1 --2- 172.21.15.102:0/3092736841 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f11c406ac00 0x7f11c406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:19.545 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.545+0000 7f11e9c00640 1 --2- 172.21.15.102:0/3092736841 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f11c406ac00 0x7f11c406d0c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f11d8007f40 tx=0x7f11d8007490 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:19.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.551+0000 7f11e1600640 1 -- 172.21.15.102:0/3092736841 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f11d00544a0 con 0x7f11e40c3ba0 2024-03-23T03:51:19.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:19.745+0000 7f11ec600640 1 -- 172.21.15.102:0/3092736841 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f11e4066bb0 con 0x7f11e40c3ba0 2024-03-23T03:51:20.604 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:51:20.612 DEBUG:teuthology.orchestra.run.smithi173:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:51:20.737 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:20.736+0000 7f11e1600640 1 -- 172.21.15.102:0/3092736841 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v102) v1 ==== 138+0+0 (secure 0 0 0) 0x7f11d0057ac0 con 0x7f11e40c3ba0 2024-03-23T03:51:20.737 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' created 2024-03-23T03:51:20.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:20.740+0000 7f11ec600640 1 -- 172.21.15.102:0/3092736841 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f11c406ac00 msgr2=0x7f11c406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:20.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:20.740+0000 7f11ec600640 1 --2- 172.21.15.102:0/3092736841 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f11c406ac00 0x7f11c406d0c0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f11d8007f40 tx=0x7f11d8007490 comp rx=0 tx=0).stop 2024-03-23T03:51:20.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:20.740+0000 7f11ec600640 1 -- 172.21.15.102:0/3092736841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11e40c3ba0 msgr2=0x7f11e41e7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:20.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:20.740+0000 7f11ec600640 1 --2- 172.21.15.102:0/3092736841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11e40c3ba0 0x7f11e41e7e20 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f11d000c7f0 tx=0x7f11d000ccb0 comp rx=0 tx=0).stop 2024-03-23T03:51:20.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:20.740+0000 7f11ec600640 1 -- 172.21.15.102:0/3092736841 shutdown_connections 2024-03-23T03:51:20.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:20.740+0000 7f11ec600640 1 --2- 172.21.15.102:0/3092736841 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f11c406ac00 0x7f11c406d0c0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:20.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:20.740+0000 7f11ec600640 1 --2- 172.21.15.102:0/3092736841 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f11e4160fc0 0x7f11e415fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:20.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:20.740+0000 7f11ec600640 1 --2- 172.21.15.102:0/3092736841 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11e40c3ba0 0x7f11e41e7e20 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:20.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:20.740+0000 7f11ec600640 1 --2- 172.21.15.102:0/3092736841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f11e406b3c0 0x7f11e40b4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:20.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:20.740+0000 7f11ec600640 1 -- 172.21.15.102:0/3092736841 >> 172.21.15.102:0/3092736841 conn(0x7f11e40b2180 msgr2=0x7f11e41eb6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:20.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:20.740+0000 7f11ec600640 1 -- 172.21.15.102:0/3092736841 shutdown_connections 2024-03-23T03:51:20.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:20.740+0000 7f11ec600640 1 -- 172.21.15.102:0/3092736841 wait complete. 2024-03-23T03:51:20.766 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T03:51:21.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.019+0000 7ff9bc200640 1 Processor -- start 2024-03-23T03:51:21.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.019+0000 7ff9bc200640 1 -- start start 2024-03-23T03:51:21.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.021+0000 7ff9bc200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff9b40add30 0x7ff9b40ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:21.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.021+0000 7ff9bc200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff9b406ce90 0x7ff9b406d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:21.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.021+0000 7ff9bc200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b40cf250 con 0x7ff9b40ce930 2024-03-23T03:51:21.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.021+0000 7ff9bc200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b40ae1c0 con 0x7ff9b40add30 2024-03-23T03:51:21.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.021+0000 7ff9bc200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b406c760 con 0x7ff9b406ce90 2024-03-23T03:51:21.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.021+0000 7ff9b8e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff9b40add30 0x7ff9b40ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:21.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.021+0000 7ff9b9800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff9b40ce930 0x7ff9b40cd3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60672/0 (socket says 172.21.15.102:60672) 2024-03-23T03:51:21.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.021+0000 7ff9b9800640 1 -- 172.21.15.102:0/1406413385 learned_addr learned my addr 172.21.15.102:0/1406413385 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:21.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.022+0000 7ff9ba200640 1 --2- 172.21.15.102:0/1406413385 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff9b406ce90 0x7ff9b406d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:21.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.022+0000 7ff9b8e00640 1 -- 172.21.15.102:0/1406413385 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff9b406ce90 msgr2=0x7ff9b406d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:21.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.022+0000 7ff9b8e00640 1 --2- 172.21.15.102:0/1406413385 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff9b406ce90 0x7ff9b406d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:21.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.022+0000 7ff9b3e00640 1 -- 172.21.15.102:0/1406413385 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120324108 0 0) 0x7ff9b40cf250 con 0x7ff9b40ce930 2024-03-23T03:51:21.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.022+0000 7ff9b8e00640 1 -- 172.21.15.102:0/1406413385 >> v1:172.21.15.102:6789/0 conn(0x7ff9b40ce930 legacy=0x7ff9b40cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:21.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.022+0000 7ff9b8e00640 1 -- 172.21.15.102:0/1406413385 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b40cdba0 con 0x7ff9b40add30 2024-03-23T03:51:21.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.022+0000 7ff9ba200640 1 --2- 172.21.15.102:0/1406413385 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff9b406ce90 0x7ff9b406d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:21.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.023+0000 7ff9b8e00640 1 --2- 172.21.15.102:0/1406413385 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9b40add30 0x7ff9b40ced10 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7ff9a00097a0 tx=0x7ff9a002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e34a0e4698d3940 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:21.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.023+0000 7ff9b3e00640 1 -- 172.21.15.102:0/1406413385 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a002d040 con 0x7ff9b40add30 2024-03-23T03:51:21.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.023+0000 7ff9b3e00640 1 -- 172.21.15.102:0/1406413385 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a0031810 con 0x7ff9b40add30 2024-03-23T03:51:21.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.023+0000 7ff9b3e00640 1 -- 172.21.15.102:0/1406413385 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a0031ba0 con 0x7ff9b40add30 2024-03-23T03:51:21.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.024+0000 7ff9bc200640 1 -- 172.21.15.102:0/1406413385 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9b40add30 msgr2=0x7ff9b40ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:21.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.024+0000 7ff9bc200640 1 --2- 172.21.15.102:0/1406413385 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9b40add30 0x7ff9b40ced10 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7ff9a00097a0 tx=0x7ff9a002b8b0 comp rx=0 tx=0).stop 2024-03-23T03:51:21.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.024+0000 7ff9bc200640 1 -- 172.21.15.102:0/1406413385 shutdown_connections 2024-03-23T03:51:21.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.024+0000 7ff9bc200640 1 --2- 172.21.15.102:0/1406413385 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff9b406ce90 0x7ff9b406d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:21.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.024+0000 7ff9bc200640 1 --2- 172.21.15.102:0/1406413385 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9b40add30 0x7ff9b40ced10 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:21.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.024+0000 7ff9bc200640 1 -- 172.21.15.102:0/1406413385 >> 172.21.15.102:0/1406413385 conn(0x7ff9b406a0b0 msgr2=0x7ff9b406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:21.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.024+0000 7ff9bc200640 1 -- 172.21.15.102:0/1406413385 shutdown_connections 2024-03-23T03:51:21.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.024+0000 7ff9bc200640 1 -- 172.21.15.102:0/1406413385 wait complete. 2024-03-23T03:51:21.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.026+0000 7ff9bc200640 1 Processor -- start 2024-03-23T03:51:21.026 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.026+0000 7ff9bc200640 1 -- start start 2024-03-23T03:51:21.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.027+0000 7ff9bc200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff9b406ce90 0x7ff9b41f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:21.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.027+0000 7ff9bc200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9b40add30 0x7ff9b4154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:21.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.027+0000 7ff9bc200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9b40ce930 0x7ff9b41501e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:21.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.027+0000 7ff9bc200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b4150720 con 0x7ff9b40ce930 2024-03-23T03:51:21.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.027+0000 7ff9bc200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b415e770 con 0x7ff9b40add30 2024-03-23T03:51:21.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.027+0000 7ff9bc200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b415ea50 con 0x7ff9b406ce90 2024-03-23T03:51:21.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.027+0000 7ff9ba200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9b40ce930 0x7ff9b41501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:21.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.027+0000 7ff9ba200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9b40ce930 0x7ff9b41501e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53208/0 (socket says 172.21.15.102:53208) 2024-03-23T03:51:21.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.027+0000 7ff9ba200640 1 -- 172.21.15.102:0/3994262612 learned_addr learned my addr 172.21.15.102:0/3994262612 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:21.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.027+0000 7ff9b9800640 1 --2- 172.21.15.102:0/3994262612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff9b406ce90 0x7ff9b41f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:21.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.028+0000 7ff9b8e00640 1 --2- 172.21.15.102:0/3994262612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9b40add30 0x7ff9b4154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:21.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.028+0000 7ff9ba200640 1 -- 172.21.15.102:0/3994262612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff9b406ce90 msgr2=0x7ff9b41f2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:21.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.028+0000 7ff9ba200640 1 --2- 172.21.15.102:0/3994262612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff9b406ce90 0x7ff9b41f2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:21.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.028+0000 7ff9ba200640 1 -- 172.21.15.102:0/3994262612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9b40add30 msgr2=0x7ff9b4154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:21.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.028+0000 7ff9ba200640 1 --2- 172.21.15.102:0/3994262612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9b40add30 0x7ff9b4154d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:21.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.028+0000 7ff9ba200640 1 -- 172.21.15.102:0/3994262612 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a0009400 con 0x7ff9b40ce930 2024-03-23T03:51:21.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.028+0000 7ff9ba200640 1 --2- 172.21.15.102:0/3994262612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9b40ce930 0x7ff9b41501e0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00cef0 tx=0x7ff9ac00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:21.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.028+0000 7ff9b1600640 1 -- 172.21.15.102:0/3994262612 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac00d840 con 0x7ff9b40ce930 2024-03-23T03:51:21.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.028+0000 7ff9b1600640 1 -- 172.21.15.102:0/3994262612 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9ac00d9c0 con 0x7ff9b40ce930 2024-03-23T03:51:21.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.028+0000 7ff9b1600640 1 -- 172.21.15.102:0/3994262612 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac00dd70 con 0x7ff9b40ce930 2024-03-23T03:51:21.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.028+0000 7ff9bc200640 1 -- 172.21.15.102:0/3994262612 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b41f23c0 con 0x7ff9b40ce930 2024-03-23T03:51:21.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.029+0000 7ff9bc200640 1 -- 172.21.15.102:0/3994262612 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b41f2980 con 0x7ff9b40ce930 2024-03-23T03:51:21.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.030+0000 7ff9bc200640 1 -- 172.21.15.102:0/3994262612 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b41f2b70 con 0x7ff9b40ce930 2024-03-23T03:51:21.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.031+0000 7ff9b1600640 1 -- 172.21.15.102:0/3994262612 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff9ac01e3b0 con 0x7ff9b40ce930 2024-03-23T03:51:21.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.031+0000 7ff9b1600640 1 --2- 172.21.15.102:0/3994262612 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff99406ac30 0x7ff99406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:21.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.032+0000 7ff9b1600640 1 -- 172.21.15.102:0/3994262612 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5547+0+0 (secure 0 0 0) 0x7ff9ac089d50 con 0x7ff9b40ce930 2024-03-23T03:51:21.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.032+0000 7ff9b9800640 1 --2- 172.21.15.102:0/3994262612 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff99406ac30 0x7ff99406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:21.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.032+0000 7ff9b9800640 1 --2- 172.21.15.102:0/3994262612 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff99406ac30 0x7ff99406d0f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8000f60 tx=0x7ff9a8007c00 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:21.037 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.037+0000 7ff9b1600640 1 -- 172.21.15.102:0/3994262612 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff9ac055080 con 0x7ff9b40ce930 2024-03-23T03:51:21.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.231+0000 7ff9bc200640 1 -- 172.21.15.102:0/3994262612 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff9b4073390 con 0x7ff9b40ce930 2024-03-23T03:51:21.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.754+0000 7ff9b1600640 1 -- 172.21.15.102:0/3994262612 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v103) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff9ac01a020 con 0x7ff9b40ce930 2024-03-23T03:51:21.755 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' created 2024-03-23T03:51:21.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.759+0000 7ff9bc200640 1 -- 172.21.15.102:0/3994262612 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff99406ac30 msgr2=0x7ff99406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:21.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.759+0000 7ff9bc200640 1 --2- 172.21.15.102:0/3994262612 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff99406ac30 0x7ff99406d0f0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7ff9a8000f60 tx=0x7ff9a8007c00 comp rx=0 tx=0).stop 2024-03-23T03:51:21.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.759+0000 7ff9bc200640 1 -- 172.21.15.102:0/3994262612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9b40ce930 msgr2=0x7ff9b41501e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:21.760 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.759+0000 7ff9bc200640 1 --2- 172.21.15.102:0/3994262612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9b40ce930 0x7ff9b41501e0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac00cef0 tx=0x7ff9ac00a710 comp rx=0 tx=0).stop 2024-03-23T03:51:21.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.760+0000 7ff9bc200640 1 -- 172.21.15.102:0/3994262612 shutdown_connections 2024-03-23T03:51:21.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.760+0000 7ff9bc200640 1 --2- 172.21.15.102:0/3994262612 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff99406ac30 0x7ff99406d0f0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:21.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.760+0000 7ff9bc200640 1 --2- 172.21.15.102:0/3994262612 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9b40ce930 0x7ff9b41501e0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:21.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.760+0000 7ff9bc200640 1 --2- 172.21.15.102:0/3994262612 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff9b40add30 0x7ff9b4154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:21.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.760+0000 7ff9bc200640 1 --2- 172.21.15.102:0/3994262612 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff9b406ce90 0x7ff9b41f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:21.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.760+0000 7ff9bc200640 1 -- 172.21.15.102:0/3994262612 >> 172.21.15.102:0/3994262612 conn(0x7ff9b406a0b0 msgr2=0x7ff9b415b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:21.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.761+0000 7ff9bc200640 1 -- 172.21.15.102:0/3994262612 shutdown_connections 2024-03-23T03:51:21.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.761+0000 7ff9bc200640 1 -- 172.21.15.102:0/3994262612 wait complete. 2024-03-23T03:51:21.794 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T03:51:21.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.976+0000 7f982be00640 1 Processor -- start 2024-03-23T03:51:21.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.977+0000 7f982be00640 1 -- start start 2024-03-23T03:51:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.978+0000 7f982be00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f982c0cce70 0x7f982c164f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.978+0000 7f982be00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c1621d0 con 0x7f982c0cb420 2024-03-23T03:51:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.978+0000 7f982be00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982c0cd370 con 0x7f982c160bd0 2024-03-23T03:51:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.978+0000 7f982be00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f982c0c9a90 con 0x7f982c0cce70 2024-03-23T03:51:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.978+0000 7f982b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f982c0cce70 0x7f982c164f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:21.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.978+0000 7f982a000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f982c0cb420 0x7f982c164880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60684/0 (socket says 172.21.15.102:60684) 2024-03-23T03:51:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.978+0000 7f982b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f982c0cce70 0x7f982c164f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53212/0 (socket says 172.21.15.102:53212) 2024-03-23T03:51:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.978+0000 7f982a000640 1 -- 172.21.15.102:0/4151773552 learned_addr learned my addr 172.21.15.102:0/4151773552 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.979+0000 7f982b400640 1 -- 172.21.15.102:0/4151773552 >> v1:172.21.15.173:6789/0 conn(0x7f982c160bd0 legacy=0x7f982c0cad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.979+0000 7f982b400640 1 -- 172.21.15.102:0/4151773552 >> v1:172.21.15.102:6789/0 conn(0x7f982c0cb420 legacy=0x7f982c164880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.979+0000 7f982b400640 1 -- 172.21.15.102:0/4151773552 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c161410 con 0x7f982c0cce70 2024-03-23T03:51:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.980+0000 7f982b400640 1 --2- 172.21.15.102:0/4151773552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f982c0cce70 0x7f982c164f90 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f9824009d90 tx=0x7f982402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8ef2952b74aea6a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:21.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.980+0000 7f9829600640 1 -- 172.21.15.102:0/4151773552 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982402cc60 con 0x7f982c0cce70 2024-03-23T03:51:21.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.980+0000 7f9829600640 1 -- 172.21.15.102:0/4151773552 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982402cdc0 con 0x7f982c0cce70 2024-03-23T03:51:21.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.980+0000 7f9829600640 1 -- 172.21.15.102:0/4151773552 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9824034660 con 0x7f982c0cce70 2024-03-23T03:51:21.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.980+0000 7f982be00640 1 -- 172.21.15.102:0/4151773552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f982c0cce70 msgr2=0x7f982c164f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:21.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.980+0000 7f982be00640 1 --2- 172.21.15.102:0/4151773552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f982c0cce70 0x7f982c164f90 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f9824009d90 tx=0x7f982402beb0 comp rx=0 tx=0).stop 2024-03-23T03:51:21.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.981+0000 7f982be00640 1 -- 172.21.15.102:0/4151773552 shutdown_connections 2024-03-23T03:51:21.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.981+0000 7f982be00640 1 --2- 172.21.15.102:0/4151773552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f982c0cce70 0x7f982c164f90 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:21.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.981+0000 7f982be00640 1 -- 172.21.15.102:0/4151773552 >> 172.21.15.102:0/4151773552 conn(0x7f982c0b01b0 msgr2=0x7f982c0b05c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:21.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.981+0000 7f982be00640 1 -- 172.21.15.102:0/4151773552 shutdown_connections 2024-03-23T03:51:21.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.981+0000 7f982be00640 1 -- 172.21.15.102:0/4151773552 wait complete. 2024-03-23T03:51:21.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.982+0000 7f982be00640 1 Processor -- start 2024-03-23T03:51:21.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.982+0000 7f982be00640 1 -- start start 2024-03-23T03:51:21.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.983+0000 7f982be00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f982c0cb420 0x7f982c0713c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:21.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.983+0000 7f982be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f982c0cce70 0x7f982c15f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:21.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.983+0000 7f982be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f982c160bd0 0x7f982c15f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:21.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.983+0000 7f982be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c148290 con 0x7f982c0cce70 2024-03-23T03:51:21.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.983+0000 7f982be00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982c160060 con 0x7f982c160bd0 2024-03-23T03:51:21.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.983+0000 7f982be00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982c1578e0 con 0x7f982c0cb420 2024-03-23T03:51:21.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.984+0000 7f982a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f982c0cce70 0x7f982c15f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:21.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.984+0000 7f982a000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f982c0cce70 0x7f982c15f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53218/0 (socket says 172.21.15.102:53218) 2024-03-23T03:51:21.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.984+0000 7f982a000640 1 -- 172.21.15.102:0/2438839830 learned_addr learned my addr 172.21.15.102:0/2438839830 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:21.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.984+0000 7f982aa00640 1 --2- 172.21.15.102:0/2438839830 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f982c0cb420 0x7f982c0713c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:21.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.984+0000 7f982b400640 1 --2- 172.21.15.102:0/2438839830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f982c160bd0 0x7f982c15f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:21.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.984+0000 7f982a000640 1 -- 172.21.15.102:0/2438839830 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f982c0cb420 msgr2=0x7f982c0713c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:21.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.984+0000 7f982a000640 1 --2- 172.21.15.102:0/2438839830 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f982c0cb420 0x7f982c0713c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:21.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.984+0000 7f982a000640 1 -- 172.21.15.102:0/2438839830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f982c160bd0 msgr2=0x7f982c15f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:21.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.984+0000 7f982a000640 1 --2- 172.21.15.102:0/2438839830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f982c160bd0 0x7f982c15f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:21.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.984+0000 7f982a000640 1 -- 172.21.15.102:0/2438839830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9824009a70 con 0x7f982c0cce70 2024-03-23T03:51:21.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.984+0000 7f982a000640 1 --2- 172.21.15.102:0/2438839830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f982c0cce70 0x7f982c15f210 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f981800c480 tx=0x7f981800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:21.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.985+0000 7f9822a00640 1 -- 172.21.15.102:0/2438839830 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818005ce0 con 0x7f982c0cce70 2024-03-23T03:51:21.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.985+0000 7f9822a00640 1 -- 172.21.15.102:0/2438839830 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9818005e40 con 0x7f982c0cce70 2024-03-23T03:51:21.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.985+0000 7f982be00640 1 -- 172.21.15.102:0/2438839830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f982c15fd80 con 0x7f982c0cce70 2024-03-23T03:51:21.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.985+0000 7f9822a00640 1 -- 172.21.15.102:0/2438839830 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98180072f0 con 0x7f982c0cce70 2024-03-23T03:51:21.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.985+0000 7f982be00640 1 -- 172.21.15.102:0/2438839830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982c14a9c0 con 0x7f982c0cce70 2024-03-23T03:51:21.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.987+0000 7f9822a00640 1 -- 172.21.15.102:0/2438839830 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9818007510 con 0x7f982c0cce70 2024-03-23T03:51:21.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.987+0000 7f982be00640 1 -- 172.21.15.102:0/2438839830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982c04e030 con 0x7f982c0cce70 2024-03-23T03:51:21.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.988+0000 7f9822a00640 1 --2- 172.21.15.102:0/2438839830 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f980c06ac00 0x7f980c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:21.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.988+0000 7f9822a00640 1 -- 172.21.15.102:0/2438839830 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f98180888f0 con 0x7f982c0cce70 2024-03-23T03:51:21.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.988+0000 7f982aa00640 1 --2- 172.21.15.102:0/2438839830 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f980c06ac00 0x7f980c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:21.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.989+0000 7f982aa00640 1 --2- 172.21.15.102:0/2438839830 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f980c06ac00 0x7f980c06d0c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f981c003d90 tx=0x7f981c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:21.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:21.993+0000 7f9822a00640 1 -- 172.21.15.102:0/2438839830 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9818053a90 con 0x7f982c0cce70 2024-03-23T03:51:22.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.226+0000 7f982be00640 1 -- 172.21.15.102:0/2438839830 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f982c0ccc50 con 0x7f982c0cce70 2024-03-23T03:51:22.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.236+0000 7f9822a00640 1 -- 172.21.15.102:0/2438839830 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v52) v1 ==== 174+0+0 (secure 0 0 0) 0x7f9818017020 con 0x7f982c0cce70 2024-03-23T03:51:22.236 INFO:teuthology.orchestra.run.smithi102.stderr:new fs with metadata pool 15 and data pool 16 2024-03-23T03:51:22.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.240+0000 7f982be00640 1 -- 172.21.15.102:0/2438839830 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f980c06ac00 msgr2=0x7f980c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:22.240 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.240+0000 7f982be00640 1 --2- 172.21.15.102:0/2438839830 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f980c06ac00 0x7f980c06d0c0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f981c003d90 tx=0x7f981c009040 comp rx=0 tx=0).stop 2024-03-23T03:51:22.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.240+0000 7f982be00640 1 -- 172.21.15.102:0/2438839830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f982c0cce70 msgr2=0x7f982c15f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:22.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.240+0000 7f982be00640 1 --2- 172.21.15.102:0/2438839830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f982c0cce70 0x7f982c15f210 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f981800c480 tx=0x7f981800c940 comp rx=0 tx=0).stop 2024-03-23T03:51:22.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.240+0000 7f982be00640 1 -- 172.21.15.102:0/2438839830 shutdown_connections 2024-03-23T03:51:22.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.240+0000 7f982be00640 1 --2- 172.21.15.102:0/2438839830 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f980c06ac00 0x7f980c06d0c0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.240+0000 7f982be00640 1 --2- 172.21.15.102:0/2438839830 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f982c160bd0 0x7f982c15f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.240+0000 7f982be00640 1 --2- 172.21.15.102:0/2438839830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f982c0cce70 0x7f982c15f210 secure :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f981800c480 tx=0x7f981800c940 comp rx=0 tx=0).stop 2024-03-23T03:51:22.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.240+0000 7f982be00640 1 --2- 172.21.15.102:0/2438839830 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f982c0cb420 0x7f982c0713c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.241 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.240+0000 7f982be00640 1 -- 172.21.15.102:0/2438839830 >> 172.21.15.102:0/2438839830 conn(0x7f982c0b01b0 msgr2=0x7f982c0cd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:22.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.241+0000 7f982be00640 1 -- 172.21.15.102:0/2438839830 shutdown_connections 2024-03-23T03:51:22.242 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.241+0000 7f982be00640 1 -- 172.21.15.102:0/2438839830 wait complete. 2024-03-23T03:51:22.267 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:51:22.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.438+0000 7fc492000640 1 Processor -- start 2024-03-23T03:51:22.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.438+0000 7fc492000640 1 -- start start 2024-03-23T03:51:22.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.439+0000 7fc492000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc48c15b560 0x7fc48c15b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:22.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.439+0000 7fc492000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc48c15d410 0x7fc48c0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:22.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.439+0000 7fc492000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc48c06c6f0 con 0x7fc48c0b39b0 2024-03-23T03:51:22.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.439+0000 7fc492000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc48c0b4370 con 0x7fc48c15d410 2024-03-23T03:51:22.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.439+0000 7fc492000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c06c9d0 con 0x7fc48c15b560 2024-03-23T03:51:22.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.440+0000 7fc482a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc48c0b39b0 0x7fc48c15e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:60696/0 (socket says 172.21.15.102:60696) 2024-03-23T03:51:22.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.440+0000 7fc482a00640 1 -- 172.21.15.102:0/1415974205 learned_addr learned my addr 172.21.15.102:0/1415974205 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:22.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.440+0000 7fc48be00640 1 --2- 172.21.15.102:0/1415974205 >> v2:172.21.15.173:3300/0 conn(0x7fc48c15d410 0x7fc48c0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:22.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.440+0000 7fc48b400640 1 --2- 172.21.15.102:0/1415974205 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc48c15b560 0x7fc48c15b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:22.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.440+0000 7fc48aa00640 1 -- 172.21.15.102:0/1415974205 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851106949 0 0) 0x7fc48c06c6f0 con 0x7fc48c0b39b0 2024-03-23T03:51:22.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.440+0000 7fc48aa00640 1 -- 172.21.15.102:0/1415974205 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc46c003520 con 0x7fc48c0b39b0 2024-03-23T03:51:22.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.440+0000 7fc48aa00640 1 -- 172.21.15.102:0/1415974205 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 158360407 0 0) 0x7fc46c003520 con 0x7fc48c0b39b0 2024-03-23T03:51:22.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.440+0000 7fc48aa00640 1 -- 172.21.15.102:0/1415974205 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc48c15b560 msgr2=0x7fc48c15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:22.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.440+0000 7fc48aa00640 1 --2- 172.21.15.102:0/1415974205 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc48c15b560 0x7fc48c15b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.440+0000 7fc48aa00640 1 -- 172.21.15.102:0/1415974205 >> v2:172.21.15.173:3300/0 conn(0x7fc48c15d410 msgr2=0x7fc48c0b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:22.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.440+0000 7fc48aa00640 1 --2- 172.21.15.102:0/1415974205 >> v2:172.21.15.173:3300/0 conn(0x7fc48c15d410 0x7fc48c0b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.440+0000 7fc48aa00640 1 -- 172.21.15.102:0/1415974205 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48c06ccb0 con 0x7fc48c0b39b0 2024-03-23T03:51:22.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.441+0000 7fc48aa00640 1 -- 172.21.15.102:0/1415974205 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fc474001670 con 0x7fc48c0b39b0 2024-03-23T03:51:22.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.441+0000 7fc48aa00640 1 -- 172.21.15.102:0/1415974205 >> v1:172.21.15.102:6789/0 conn(0x7fc48c0b39b0 legacy=0x7fc48c15e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:22.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.441+0000 7fc48aa00640 1 --2- 172.21.15.102:0/1415974205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc46c003970 0x7fc46c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:22.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.441+0000 7fc48aa00640 1 --2- 172.21.15.102:0/1415974205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc46c0047c0 0x7fc46c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:22.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.441+0000 7fc48aa00640 1 --2- 172.21.15.102:0/1415974205 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc48c15b560 0x7fc46c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:22.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.441+0000 7fc48aa00640 1 -- 172.21.15.102:0/1415974205 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c0b4370 con 0x7fc46c003970 2024-03-23T03:51:22.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.441+0000 7fc48aa00640 1 -- 172.21.15.102:0/1415974205 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c06c9d0 con 0x7fc46c0047c0 2024-03-23T03:51:22.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.441+0000 7fc48aa00640 1 -- 172.21.15.102:0/1415974205 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c00ba80 con 0x7fc48c15b560 2024-03-23T03:51:22.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.441+0000 7fc48b400640 1 --2- 172.21.15.102:0/1415974205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc46c003970 0x7fc46c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:22.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.441+0000 7fc48b400640 1 -- 172.21.15.102:0/1415974205 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc48c15b560 msgr2=0x7fc46c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:22.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.441+0000 7fc48b400640 1 --2- 172.21.15.102:0/1415974205 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc48c15b560 0x7fc46c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.441+0000 7fc48b400640 1 -- 172.21.15.102:0/1415974205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc46c0047c0 msgr2=0x7fc46c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:22.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.441+0000 7fc48b400640 1 --2- 172.21.15.102:0/1415974205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc46c0047c0 0x7fc46c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.441+0000 7fc48b400640 1 -- 172.21.15.102:0/1415974205 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46c003520 con 0x7fc46c003970 2024-03-23T03:51:22.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.442+0000 7fc48b400640 1 --2- 172.21.15.102:0/1415974205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc46c003970 0x7fc46c003d90 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fc47c00ba00 tx=0x7fc46c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dfddc73b331865c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:22.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.443+0000 7fc48aa00640 1 -- 172.21.15.102:0/1415974205 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47c032630 con 0x7fc46c003970 2024-03-23T03:51:22.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.443+0000 7fc48aa00640 1 -- 172.21.15.102:0/1415974205 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47c032790 con 0x7fc46c003970 2024-03-23T03:51:22.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.443+0000 7fc48aa00640 1 -- 172.21.15.102:0/1415974205 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47c032b20 con 0x7fc46c003970 2024-03-23T03:51:22.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.443+0000 7fc492000640 1 -- 172.21.15.102:0/1415974205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc46c003970 msgr2=0x7fc46c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:22.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.443+0000 7fc492000640 1 --2- 172.21.15.102:0/1415974205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc46c003970 0x7fc46c003d90 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7fc47c00ba00 tx=0x7fc46c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.443+0000 7fc48be00640 1 -- 172.21.15.102:0/1415974205 reap_dead start 2024-03-23T03:51:22.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.443+0000 7fc492000640 1 -- 172.21.15.102:0/1415974205 shutdown_connections 2024-03-23T03:51:22.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.443+0000 7fc492000640 1 -- 172.21.15.102:0/1415974205 >> 172.21.15.102:0/1415974205 conn(0x7fc48c06a3d0 msgr2=0x7fc48c0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:22.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.443+0000 7fc492000640 1 -- 172.21.15.102:0/1415974205 shutdown_connections 2024-03-23T03:51:22.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.443+0000 7fc492000640 1 -- 172.21.15.102:0/1415974205 wait complete. 2024-03-23T03:51:22.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.445+0000 7fc492000640 1 Processor -- start 2024-03-23T03:51:22.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.445+0000 7fc492000640 1 -- start start 2024-03-23T03:51:22.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.446+0000 7fc492000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc46c003970 0x7fc48c1f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:22.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.446+0000 7fc492000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc48c153820 0x7fc48c153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:22.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.446+0000 7fc492000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc48c0b2d90 0x7fc48c0b3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:22.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.446+0000 7fc492000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c13b750 con 0x7fc46c003970 2024-03-23T03:51:22.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.446+0000 7fc492000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c0b3750 con 0x7fc48c153820 2024-03-23T03:51:22.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.446+0000 7fc492000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc48c1f6840 con 0x7fc48c0b2d90 2024-03-23T03:51:22.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.446+0000 7fc48b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc46c003970 0x7fc48c1f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:22.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.446+0000 7fc48b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc46c003970 0x7fc48c1f1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:53244/0 (socket says 172.21.15.102:53244) 2024-03-23T03:51:22.447 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.446+0000 7fc48b400640 1 -- 172.21.15.102:0/2117421561 learned_addr learned my addr 172.21.15.102:0/2117421561 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:22.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.447+0000 7fc48b400640 1 -- 172.21.15.102:0/2117421561 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc48c0b2d90 msgr2=0x7fc48c0b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:22.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.447+0000 7fc48b400640 1 --2- 172.21.15.102:0/2117421561 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc48c0b2d90 0x7fc48c0b3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.447+0000 7fc482a00640 1 --2- 172.21.15.102:0/2117421561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc48c153820 0x7fc48c153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.447+0000 7fc48b400640 1 -- 172.21.15.102:0/2117421561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc48c153820 msgr2=0x7fc48c153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.447+0000 7fc48b400640 1 --2- 172.21.15.102:0/2117421561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc48c153820 0x7fc48c153ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.447+0000 7fc48b400640 1 -- 172.21.15.102:0/2117421561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48c142960 con 0x7fc46c003970 2024-03-23T03:51:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.447+0000 7fc48b400640 1 --2- 172.21.15.102:0/2117421561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc46c003970 0x7fc48c1f1e50 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fc47c02beb0 tx=0x7fc47c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.447+0000 7fc483e00640 1 -- 172.21.15.102:0/2117421561 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47c03ca80 con 0x7fc46c003970 2024-03-23T03:51:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.447+0000 7fc483e00640 1 -- 172.21.15.102:0/2117421561 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47c03cc00 con 0x7fc46c003970 2024-03-23T03:51:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.447+0000 7fc483e00640 1 -- 172.21.15.102:0/2117421561 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47c030400 con 0x7fc46c003970 2024-03-23T03:51:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.447+0000 7fc492000640 1 -- 172.21.15.102:0/2117421561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc48c1f5a80 con 0x7fc46c003970 2024-03-23T03:51:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.448+0000 7fc492000640 1 -- 172.21.15.102:0/2117421561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc48c1f6040 con 0x7fc46c003970 2024-03-23T03:51:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.449+0000 7fc492000640 1 -- 172.21.15.102:0/2117421561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc48c06ac20 con 0x7fc46c003970 2024-03-23T03:51:22.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.450+0000 7fc483e00640 1 -- 172.21.15.102:0/2117421561 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc47c030560 con 0x7fc46c003970 2024-03-23T03:51:22.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.450+0000 7fc483e00640 1 --2- 172.21.15.102:0/2117421561 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc46806ac30 0x7fc46806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:22.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.450+0000 7fc483e00640 1 -- 172.21.15.102:0/2117421561 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc47c0ab850 con 0x7fc46c003970 2024-03-23T03:51:22.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.451+0000 7fc482a00640 1 --2- 172.21.15.102:0/2117421561 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc46806ac30 0x7fc46806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:22.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.451+0000 7fc482a00640 1 --2- 172.21.15.102:0/2117421561 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc46806ac30 0x7fc46806d0f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc48c15cf70 tx=0x7fc474009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:22.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.455+0000 7fc483e00640 1 -- 172.21.15.102:0/2117421561 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc47c076a30 con 0x7fc46c003970 2024-03-23T03:51:22.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.644+0000 7fc492000640 1 -- 172.21.15.102:0/2117421561 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc48c073390 con 0x7fc46c003970 2024-03-23T03:51:22.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.646+0000 7fc483e00640 1 -- 172.21.15.102:0/2117421561 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+16309 (secure 0 0 0) 0x7fc47c036020 con 0x7fc46c003970 2024-03-23T03:51:22.647 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:22.647 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":104,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:51:22.228018+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:51:19.796618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-23T03:51:21.283447+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:51:22.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.648+0000 7fc492000640 1 -- 172.21.15.102:0/2117421561 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc46806ac30 msgr2=0x7fc46806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:22.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.648+0000 7fc492000640 1 --2- 172.21.15.102:0/2117421561 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc46806ac30 0x7fc46806d0f0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fc48c15cf70 tx=0x7fc474009040 comp rx=0 tx=0).stop 2024-03-23T03:51:22.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.648+0000 7fc492000640 1 -- 172.21.15.102:0/2117421561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc46c003970 msgr2=0x7fc48c1f1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:22.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.648+0000 7fc492000640 1 --2- 172.21.15.102:0/2117421561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc46c003970 0x7fc48c1f1e50 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7fc47c02beb0 tx=0x7fc47c004730 comp rx=0 tx=0).stop 2024-03-23T03:51:22.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.649+0000 7fc492000640 1 -- 172.21.15.102:0/2117421561 shutdown_connections 2024-03-23T03:51:22.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.649+0000 7fc492000640 1 --2- 172.21.15.102:0/2117421561 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc46806ac30 0x7fc46806d0f0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.649+0000 7fc492000640 1 --2- 172.21.15.102:0/2117421561 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc48c0b2d90 0x7fc48c0b3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.649+0000 7fc492000640 1 --2- 172.21.15.102:0/2117421561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc48c153820 0x7fc48c153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.649+0000 7fc492000640 1 --2- 172.21.15.102:0/2117421561 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc46c003970 0x7fc48c1f1e50 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.649+0000 7fc492000640 1 -- 172.21.15.102:0/2117421561 >> 172.21.15.102:0/2117421561 conn(0x7fc48c06a3d0 msgr2=0x7fc48c159930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:22.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.649+0000 7fc492000640 1 -- 172.21.15.102:0/2117421561 shutdown_connections 2024-03-23T03:51:22.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.649+0000 7fc492000640 1 -- 172.21.15.102:0/2117421561 wait complete. 2024-03-23T03:51:22.666 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:51:22.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.861+0000 7fd2d3800640 1 Processor -- start 2024-03-23T03:51:22.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.861+0000 7fd2d3800640 1 -- start start 2024-03-23T03:51:22.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.863+0000 7fd2d3800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd2cc15b560 0x7fd2cc15b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:22.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.863+0000 7fd2d3800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2cc15d410 0x7fd2cc0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:22.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.863+0000 7fd2d3800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2cc06c6f0 con 0x7fd2cc0b39b0 2024-03-23T03:51:22.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.863+0000 7fd2d3800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2cc0b4370 con 0x7fd2cc15b560 2024-03-23T03:51:22.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.863+0000 7fd2d3800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc06c9d0 con 0x7fd2cc15d410 2024-03-23T03:51:22.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.863+0000 7fd2d0e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd2cc15b560 0x7fd2cc15b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:22.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.863+0000 7fd2cbe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fd2cc0b39b0 0x7fd2cc15e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38552/0 (socket says 172.21.15.102:38552) 2024-03-23T03:51:22.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.863+0000 7fd2cbe00640 1 -- 172.21.15.102:0/2276532556 learned_addr learned my addr 172.21.15.102:0/2276532556 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:22.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.864+0000 7fd2d1800640 1 --2- 172.21.15.102:0/2276532556 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2cc15d410 0x7fd2cc0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:22.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.864+0000 7fd2cb400640 1 -- 172.21.15.102:0/2276532556 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827726461 0 0) 0x7fd2cc06c6f0 con 0x7fd2cc0b39b0 2024-03-23T03:51:22.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.864+0000 7fd2cb400640 1 -- 172.21.15.102:0/2276532556 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2b4003520 con 0x7fd2cc0b39b0 2024-03-23T03:51:22.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.864+0000 7fd2d0e00640 1 -- 172.21.15.102:0/2276532556 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2cc15d410 msgr2=0x7fd2cc0b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:22.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.864+0000 7fd2d0e00640 1 --2- 172.21.15.102:0/2276532556 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2cc15d410 0x7fd2cc0b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.864+0000 7fd2d0e00640 1 -- 172.21.15.102:0/2276532556 >> v1:172.21.15.102:6789/0 conn(0x7fd2cc0b39b0 legacy=0x7fd2cc15e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:22.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.864+0000 7fd2d0e00640 1 -- 172.21.15.102:0/2276532556 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2cc06ccb0 con 0x7fd2cc15b560 2024-03-23T03:51:22.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.865+0000 7fd2d0e00640 1 --2- 172.21.15.102:0/2276532556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2cc15b560 0x7fd2cc15b940 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fd2c00097a0 tx=0x7fd2c002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29cdebbef5299064 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:22.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.865+0000 7fd2cb400640 1 -- 172.21.15.102:0/2276532556 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c002d040 con 0x7fd2cc15b560 2024-03-23T03:51:22.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.865+0000 7fd2cb400640 1 -- 172.21.15.102:0/2276532556 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c0031810 con 0x7fd2cc15b560 2024-03-23T03:51:22.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.865+0000 7fd2cb400640 1 -- 172.21.15.102:0/2276532556 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0031ba0 con 0x7fd2cc15b560 2024-03-23T03:51:22.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.866+0000 7fd2d3800640 1 -- 172.21.15.102:0/2276532556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2cc15b560 msgr2=0x7fd2cc15b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:22.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.866+0000 7fd2d3800640 1 --2- 172.21.15.102:0/2276532556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2cc15b560 0x7fd2cc15b940 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7fd2c00097a0 tx=0x7fd2c002b8b0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.866+0000 7fd2d3800640 1 -- 172.21.15.102:0/2276532556 shutdown_connections 2024-03-23T03:51:22.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.866+0000 7fd2d3800640 1 --2- 172.21.15.102:0/2276532556 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2cc15d410 0x7fd2cc0b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.866+0000 7fd2d3800640 1 --2- 172.21.15.102:0/2276532556 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2cc15b560 0x7fd2cc15b940 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.866+0000 7fd2d3800640 1 -- 172.21.15.102:0/2276532556 >> 172.21.15.102:0/2276532556 conn(0x7fd2cc06a3d0 msgr2=0x7fd2cc0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:22.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.866+0000 7fd2d3800640 1 -- 172.21.15.102:0/2276532556 shutdown_connections 2024-03-23T03:51:22.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.866+0000 7fd2d3800640 1 -- 172.21.15.102:0/2276532556 wait complete. 2024-03-23T03:51:22.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.867+0000 7fd2d3800640 1 Processor -- start 2024-03-23T03:51:22.868 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.868+0000 7fd2d3800640 1 -- start start 2024-03-23T03:51:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.868+0000 7fd2d3800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2cc0b39b0 0x7fd2cc1548a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.868+0000 7fd2d3800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2cc15b560 0x7fd2cc157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.869+0000 7fd2d3800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2cc15d410 0x7fd2cc0acec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.869+0000 7fd2d3800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc156ca0 con 0x7fd2cc0b39b0 2024-03-23T03:51:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.869+0000 7fd2d3800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc1463d0 con 0x7fd2cc15d410 2024-03-23T03:51:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.869+0000 7fd2d3800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2cc152b50 con 0x7fd2cc15b560 2024-03-23T03:51:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.869+0000 7fd2d0e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2cc0b39b0 0x7fd2cc1548a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.869+0000 7fd2d0e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2cc0b39b0 0x7fd2cc1548a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42478/0 (socket says 172.21.15.102:42478) 2024-03-23T03:51:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.869+0000 7fd2d0e00640 1 -- 172.21.15.102:0/2474339214 learned_addr learned my addr 172.21.15.102:0/2474339214 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.869+0000 7fd2d0e00640 1 -- 172.21.15.102:0/2474339214 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2cc15b560 msgr2=0x7fd2cc157a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.869+0000 7fd2d0e00640 1 --2- 172.21.15.102:0/2474339214 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2cc15b560 0x7fd2cc157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.869+0000 7fd2d0e00640 1 -- 172.21.15.102:0/2474339214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2cc15d410 msgr2=0x7fd2cc0acec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:22.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.869+0000 7fd2d0e00640 1 --2- 172.21.15.102:0/2474339214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2cc15d410 0x7fd2cc0acec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:22.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.869+0000 7fd2d0e00640 1 -- 172.21.15.102:0/2474339214 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c0009400 con 0x7fd2cc0b39b0 2024-03-23T03:51:22.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.869+0000 7fd2d0e00640 1 --2- 172.21.15.102:0/2474339214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2cc0b39b0 0x7fd2cc1548a0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fd2c00098c0 tx=0x7fd2c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:22.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.870+0000 7fd2c8c00640 1 -- 172.21.15.102:0/2474339214 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0006e10 con 0x7fd2cc0b39b0 2024-03-23T03:51:22.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.870+0000 7fd2c8c00640 1 -- 172.21.15.102:0/2474339214 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c003ad60 con 0x7fd2cc0b39b0 2024-03-23T03:51:22.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.870+0000 7fd2c8c00640 1 -- 172.21.15.102:0/2474339214 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0044620 con 0x7fd2cc0b39b0 2024-03-23T03:51:22.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.870+0000 7fd2d3800640 1 -- 172.21.15.102:0/2474339214 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2cc154de0 con 0x7fd2cc0b39b0 2024-03-23T03:51:22.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.870+0000 7fd2d3800640 1 -- 172.21.15.102:0/2474339214 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2cc152df0 con 0x7fd2cc0b39b0 2024-03-23T03:51:22.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.871+0000 7fd2d3800640 1 -- 172.21.15.102:0/2474339214 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2cc1529b0 con 0x7fd2cc0b39b0 2024-03-23T03:51:22.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.872+0000 7fd2c8c00640 1 -- 172.21.15.102:0/2474339214 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd2c004d440 con 0x7fd2cc0b39b0 2024-03-23T03:51:22.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.873+0000 7fd2c8c00640 1 --2- 172.21.15.102:0/2474339214 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd2ac06ac00 0x7fd2ac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:22.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.873+0000 7fd2c8c00640 1 -- 172.21.15.102:0/2474339214 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd2c00ab810 con 0x7fd2cc0b39b0 2024-03-23T03:51:22.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.873+0000 7fd2cbe00640 1 --2- 172.21.15.102:0/2474339214 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd2ac06ac00 0x7fd2ac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:22.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.874+0000 7fd2cbe00640 1 --2- 172.21.15.102:0/2474339214 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd2ac06ac00 0x7fd2ac06d0c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8001610 tx=0x7fd2b8007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:22.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:22.878+0000 7fd2c8c00640 1 -- 172.21.15.102:0/2474339214 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd2cc1529b0 con 0x7fd2cc0b39b0 2024-03-23T03:51:23.072 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.072+0000 7fd2d3800640 1 -- 172.21.15.102:0/2474339214 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2cc1529b0 con 0x7fd2cc0b39b0 2024-03-23T03:51:23.074 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.074+0000 7fd2c8c00640 1 -- 172.21.15.102:0/2474339214 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+16309 (secure 0 0 0) 0x7fd2cc1529b0 con 0x7fd2cc0b39b0 2024-03-23T03:51:23.075 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:23.075 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":104,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:51:22.228018+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:51:19.796618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-23T03:51:21.283447+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:51:23.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.076+0000 7fd2d3800640 1 -- 172.21.15.102:0/2474339214 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd2ac06ac00 msgr2=0x7fd2ac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:23.077 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.076+0000 7fd2d3800640 1 --2- 172.21.15.102:0/2474339214 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd2ac06ac00 0x7fd2ac06d0c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd2b8001610 tx=0x7fd2b8007930 comp rx=0 tx=0).stop 2024-03-23T03:51:23.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.076+0000 7fd2d3800640 1 -- 172.21.15.102:0/2474339214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2cc0b39b0 msgr2=0x7fd2cc1548a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:23.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.076+0000 7fd2d3800640 1 --2- 172.21.15.102:0/2474339214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2cc0b39b0 0x7fd2cc1548a0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7fd2c00098c0 tx=0x7fd2c00046c0 comp rx=0 tx=0).stop 2024-03-23T03:51:23.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.077+0000 7fd2d3800640 1 -- 172.21.15.102:0/2474339214 shutdown_connections 2024-03-23T03:51:23.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.077+0000 7fd2d3800640 1 --2- 172.21.15.102:0/2474339214 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd2ac06ac00 0x7fd2ac06d0c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:23.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.077+0000 7fd2d3800640 1 --2- 172.21.15.102:0/2474339214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd2cc15d410 0x7fd2cc0acec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:23.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.077+0000 7fd2d3800640 1 --2- 172.21.15.102:0/2474339214 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd2cc15b560 0x7fd2cc157a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:23.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.077+0000 7fd2d3800640 1 --2- 172.21.15.102:0/2474339214 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd2cc0b39b0 0x7fd2cc1548a0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:23.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.077+0000 7fd2d3800640 1 -- 172.21.15.102:0/2474339214 >> 172.21.15.102:0/2474339214 conn(0x7fd2cc06a3d0 msgr2=0x7fd2cc0cd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:23.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.077+0000 7fd2d3800640 1 -- 172.21.15.102:0/2474339214 shutdown_connections 2024-03-23T03:51:23.078 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.077+0000 7fd2d3800640 1 -- 172.21.15.102:0/2474339214 wait complete. 2024-03-23T03:51:23.095 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T03:51:23.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.310+0000 7f8760000640 1 Processor -- start 2024-03-23T03:51:23.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.310+0000 7f8760000640 1 -- start start 2024-03-23T03:51:23.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.311+0000 7f8760000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87581456a0 0x7f87580ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:23.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.311+0000 7f8760000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f8758070cd0 0x7f87580cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:23.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.311+0000 7f8760000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8758145c20 0x7f87580ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:23.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.311+0000 7f8760000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87580ad890 con 0x7f8758145c20 2024-03-23T03:51:23.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.311+0000 7f8760000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8758145a80 con 0x7f8758070cd0 2024-03-23T03:51:23.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.312+0000 7f8760000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f875806d390 con 0x7f87581456a0 2024-03-23T03:51:23.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.312+0000 7f875e000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8758145c20 0x7f87580ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:23.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.312+0000 7f875e000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f8758145c20 0x7f87580ab8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42492/0 (socket says 172.21.15.102:42492) 2024-03-23T03:51:23.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.312+0000 7f875e000640 1 -- 172.21.15.102:0/966743604 learned_addr learned my addr 172.21.15.102:0/966743604 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:23.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.312+0000 7f875d600640 1 --2- 172.21.15.102:0/966743604 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87581456a0 0x7f87580ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:23.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.312+0000 7f875e000640 1 -- 172.21.15.102:0/966743604 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87581456a0 msgr2=0x7f87580ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:23.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.312+0000 7f875e000640 1 --2- 172.21.15.102:0/966743604 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87581456a0 0x7f87580ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:23.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.312+0000 7f875cc00640 1 --2- 172.21.15.102:0/966743604 >> v2:172.21.15.173:3300/0 conn(0x7f8758070cd0 0x7f87580cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:23.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.312+0000 7f875e000640 1 -- 172.21.15.102:0/966743604 >> v2:172.21.15.173:3300/0 conn(0x7f8758070cd0 msgr2=0x7f87580cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:23.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.312+0000 7f875e000640 1 --2- 172.21.15.102:0/966743604 >> v2:172.21.15.173:3300/0 conn(0x7f8758070cd0 0x7f87580cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:23.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.312+0000 7f875e000640 1 -- 172.21.15.102:0/966743604 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875806d670 con 0x7f8758145c20 2024-03-23T03:51:23.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.314+0000 7f875e000640 1 --2- 172.21.15.102:0/966743604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8758145c20 0x7f87580ab8b0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f87580cf570 tx=0x7f875002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f18ae1ebe3daa84 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:23.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.314+0000 7f8757e00640 1 -- 172.21.15.102:0/966743604 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875002ca70 con 0x7f8758145c20 2024-03-23T03:51:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.314+0000 7f8757e00640 1 -- 172.21.15.102:0/966743604 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875002cbd0 con 0x7f8758145c20 2024-03-23T03:51:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.314+0000 7f8757e00640 1 -- 172.21.15.102:0/966743604 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87500343f0 con 0x7f8758145c20 2024-03-23T03:51:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.315+0000 7f8760000640 1 -- 172.21.15.102:0/966743604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8758145c20 msgr2=0x7f87580ab8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.315+0000 7f8760000640 1 --2- 172.21.15.102:0/966743604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8758145c20 0x7f87580ab8b0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f87580cf570 tx=0x7f875002bed0 comp rx=0 tx=0).stop 2024-03-23T03:51:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.315+0000 7f8760000640 1 -- 172.21.15.102:0/966743604 shutdown_connections 2024-03-23T03:51:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.315+0000 7f8760000640 1 --2- 172.21.15.102:0/966743604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8758145c20 0x7f87580ab8b0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.315+0000 7f8760000640 1 --2- 172.21.15.102:0/966743604 >> v2:172.21.15.173:3300/0 conn(0x7f8758070cd0 0x7f87580cee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.315+0000 7f8760000640 1 --2- 172.21.15.102:0/966743604 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87581456a0 0x7f87580ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.315+0000 7f8760000640 1 -- 172.21.15.102:0/966743604 >> 172.21.15.102:0/966743604 conn(0x7f875806a0b0 msgr2=0x7f875806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.315+0000 7f8760000640 1 -- 172.21.15.102:0/966743604 shutdown_connections 2024-03-23T03:51:23.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.315+0000 7f8760000640 1 -- 172.21.15.102:0/966743604 wait complete. 2024-03-23T03:51:23.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.317+0000 7f8760000640 1 Processor -- start 2024-03-23T03:51:23.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.317+0000 7f8760000640 1 -- start start 2024-03-23T03:51:23.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.318+0000 7f8760000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8758070cd0 0x7f875813ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:23.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.318+0000 7f8760000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87581456a0 0x7f87581429b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:23.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.318+0000 7f8760000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8758145c20 0x7f87581446d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:23.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.318+0000 7f8760000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875815eba0 con 0x7f8758145c20 2024-03-23T03:51:23.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.318+0000 7f8760000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875815ee80 con 0x7f8758070cd0 2024-03-23T03:51:23.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.318+0000 7f8760000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8758141ef0 con 0x7f87581456a0 2024-03-23T03:51:23.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.318+0000 7f875e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8758145c20 0x7f87581446d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:23.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.318+0000 7f875e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8758145c20 0x7f87581446d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42502/0 (socket says 172.21.15.102:42502) 2024-03-23T03:51:23.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.318+0000 7f875e000640 1 -- 172.21.15.102:0/1818436930 learned_addr learned my addr 172.21.15.102:0/1818436930 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:23.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.318+0000 7f875d600640 1 --2- 172.21.15.102:0/1818436930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8758070cd0 0x7f875813ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:23.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.318+0000 7f875cc00640 1 --2- 172.21.15.102:0/1818436930 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87581456a0 0x7f87581429b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:23.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.318+0000 7f875e000640 1 -- 172.21.15.102:0/1818436930 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87581456a0 msgr2=0x7f87581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:23.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.318+0000 7f875e000640 1 --2- 172.21.15.102:0/1818436930 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87581456a0 0x7f87581429b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:23.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.318+0000 7f875e000640 1 -- 172.21.15.102:0/1818436930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8758070cd0 msgr2=0x7f875813ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:23.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.318+0000 7f875e000640 1 --2- 172.21.15.102:0/1818436930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8758070cd0 0x7f875813ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:23.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.318+0000 7f875e000640 1 -- 172.21.15.102:0/1818436930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8750009b30 con 0x7f8758145c20 2024-03-23T03:51:23.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.319+0000 7f875e000640 1 --2- 172.21.15.102:0/1818436930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8758145c20 0x7f87581446d0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f8750036040 tx=0x7f8750004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:23.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.319+0000 7f8755600640 1 -- 172.21.15.102:0/1818436930 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8750040670 con 0x7f8758145c20 2024-03-23T03:51:23.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.319+0000 7f8755600640 1 -- 172.21.15.102:0/1818436930 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87500407d0 con 0x7f8758145c20 2024-03-23T03:51:23.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.319+0000 7f8755600640 1 -- 172.21.15.102:0/1818436930 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8750040b80 con 0x7f8758145c20 2024-03-23T03:51:23.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.319+0000 7f8760000640 1 -- 172.21.15.102:0/1818436930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87581421d0 con 0x7f8758145c20 2024-03-23T03:51:23.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.319+0000 7f8760000640 1 -- 172.21.15.102:0/1818436930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f875815e790 con 0x7f8758145c20 2024-03-23T03:51:23.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.320+0000 7f8760000640 1 -- 172.21.15.102:0/1818436930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f875815e920 con 0x7f8758145c20 2024-03-23T03:51:23.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.321+0000 7f8755600640 1 -- 172.21.15.102:0/1818436930 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8750040ce0 con 0x7f8758145c20 2024-03-23T03:51:23.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.322+0000 7f8755600640 1 --2- 172.21.15.102:0/1818436930 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f873806ac00 0x7f873806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:23.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.322+0000 7f8755600640 1 -- 172.21.15.102:0/1818436930 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f87500ac770 con 0x7f8758145c20 2024-03-23T03:51:23.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.322+0000 7f875d600640 1 --2- 172.21.15.102:0/1818436930 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f873806ac00 0x7f873806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:23.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.323+0000 7f875d600640 1 --2- 172.21.15.102:0/1818436930 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f873806ac00 0x7f873806d0c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f874c00a8f0 tx=0x7f874c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:23.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.326+0000 7f8755600640 1 -- 172.21.15.102:0/1818436930 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8750077770 con 0x7f8758145c20 2024-03-23T03:51:23.558 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:23.558+0000 7f8760000640 1 -- 172.21.15.102:0/1818436930 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8758073390 con 0x7f8758145c20 2024-03-23T03:51:24.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.249+0000 7f8755600640 1 -- 172.21.15.102:0/1818436930 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v55) v1 ==== 118+0+0 (secure 0 0 0) 0x7f875007ad90 con 0x7f8758145c20 2024-03-23T03:51:24.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.252+0000 7f8760000640 1 -- 172.21.15.102:0/1818436930 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f873806ac00 msgr2=0x7f873806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:24.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.252+0000 7f8760000640 1 --2- 172.21.15.102:0/1818436930 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f873806ac00 0x7f873806d0c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f874c00a8f0 tx=0x7f874c008040 comp rx=0 tx=0).stop 2024-03-23T03:51:24.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.252+0000 7f8760000640 1 -- 172.21.15.102:0/1818436930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8758145c20 msgr2=0x7f87581446d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:24.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.252+0000 7f8760000640 1 --2- 172.21.15.102:0/1818436930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8758145c20 0x7f87581446d0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f8750036040 tx=0x7f8750004a40 comp rx=0 tx=0).stop 2024-03-23T03:51:24.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.253+0000 7f8760000640 1 -- 172.21.15.102:0/1818436930 shutdown_connections 2024-03-23T03:51:24.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.253+0000 7f8760000640 1 --2- 172.21.15.102:0/1818436930 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f873806ac00 0x7f873806d0c0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:24.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.253+0000 7f8760000640 1 --2- 172.21.15.102:0/1818436930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8758145c20 0x7f87581446d0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:24.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.253+0000 7f8760000640 1 --2- 172.21.15.102:0/1818436930 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f87581456a0 0x7f87581429b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:24.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.253+0000 7f8760000640 1 --2- 172.21.15.102:0/1818436930 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8758070cd0 0x7f875813ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:24.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.253+0000 7f8760000640 1 -- 172.21.15.102:0/1818436930 >> 172.21.15.102:0/1818436930 conn(0x7f875806a0b0 msgr2=0x7f87580b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:24.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.253+0000 7f8760000640 1 -- 172.21.15.102:0/1818436930 shutdown_connections 2024-03-23T03:51:24.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.253+0000 7f8760000640 1 -- 172.21.15.102:0/1818436930 wait complete. 2024-03-23T03:51:24.274 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-23T03:51:24.274 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T03:51:24.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.488+0000 7f8ed3800640 1 Processor -- start 2024-03-23T03:51:24.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.488+0000 7f8ed3800640 1 -- start start 2024-03-23T03:51:24.492 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.490+0000 7f8ed3800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f8ecc0ce930 0x7f8ecc0cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:24.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.490+0000 7f8ed3800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc159c10 con 0x7f8ecc0ca7b0 2024-03-23T03:51:24.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.490+0000 7f8ed3800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ecc0cd840 con 0x7f8ecc159830 2024-03-23T03:51:24.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.490+0000 7f8ed3800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ecc06de50 con 0x7f8ecc0ce930 2024-03-23T03:51:24.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.490+0000 7f8ecbe00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f8ecc0ca7b0 0x7f8ecc0cdcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38566/0 (socket says 172.21.15.102:38566) 2024-03-23T03:51:24.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.490+0000 7f8ecbe00640 1 -- 172.21.15.102:0/1776880342 learned_addr learned my addr 172.21.15.102:0/1776880342 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:24.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.490+0000 7f8ed1800640 1 --2- 172.21.15.102:0/1776880342 >> v2:172.21.15.173:3300/0 conn(0x7f8ecc0ce930 0x7f8ecc0cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:24.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.490+0000 7f8ecb400640 1 -- 172.21.15.102:0/1776880342 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620632447 0 0) 0x7f8ecc159c10 con 0x7f8ecc0ca7b0 2024-03-23T03:51:24.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.490+0000 7f8ecb400640 1 -- 172.21.15.102:0/1776880342 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eb4003520 con 0x7f8ecc0ca7b0 2024-03-23T03:51:24.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.491+0000 7f8ecb400640 1 -- 172.21.15.102:0/1776880342 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 310468955 0 0) 0x7f8eb4003520 con 0x7f8ecc0ca7b0 2024-03-23T03:51:24.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.491+0000 7f8ecb400640 1 -- 172.21.15.102:0/1776880342 >> v2:172.21.15.173:3300/0 conn(0x7f8ecc0ce930 msgr2=0x7f8ecc0cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:24.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.491+0000 7f8ecb400640 1 --2- 172.21.15.102:0/1776880342 >> v2:172.21.15.173:3300/0 conn(0x7f8ecc0ce930 0x7f8ecc0cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:24.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.491+0000 7f8ecb400640 1 -- 172.21.15.102:0/1776880342 >> v1:172.21.15.173:6789/0 conn(0x7f8ecc159830 legacy=0x7f8ecc0cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:24.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.491+0000 7f8ecb400640 1 -- 172.21.15.102:0/1776880342 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc0ccc30 con 0x7f8ecc0ca7b0 2024-03-23T03:51:24.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.491+0000 7f8ecb400640 1 -- 172.21.15.102:0/1776880342 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f8eb8001670 con 0x7f8ecc0ca7b0 2024-03-23T03:51:24.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.491+0000 7f8ecb400640 1 -- 172.21.15.102:0/1776880342 >> v1:172.21.15.102:6789/0 conn(0x7f8ecc0ca7b0 legacy=0x7f8ecc0cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:24.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.491+0000 7f8ecb400640 1 --2- 172.21.15.102:0/1776880342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8eb40039f0 0x7f8eb4003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:24.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.491+0000 7f8ecb400640 1 --2- 172.21.15.102:0/1776880342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb40047c0 0x7f8eb4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:24.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.491+0000 7f8ecb400640 1 --2- 172.21.15.102:0/1776880342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8eb4005180 0x7f8eb400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:24.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.491+0000 7f8ecb400640 1 -- 172.21.15.102:0/1776880342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb400e260 con 0x7f8eb40047c0 2024-03-23T03:51:24.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.491+0000 7f8ecb400640 1 -- 172.21.15.102:0/1776880342 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb400e3d0 con 0x7f8eb40039f0 2024-03-23T03:51:24.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.491+0000 7f8ecb400640 1 -- 172.21.15.102:0/1776880342 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb400e590 con 0x7f8eb4005180 2024-03-23T03:51:24.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.491+0000 7f8ecbe00640 1 --2- 172.21.15.102:0/1776880342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb40047c0 0x7f8eb4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:24.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.491+0000 7f8ed0e00640 1 --2- 172.21.15.102:0/1776880342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8eb40039f0 0x7f8eb4003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:24.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.492+0000 7f8ed1800640 1 --2- 172.21.15.102:0/1776880342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8eb4005180 0x7f8eb400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:24.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.492+0000 7f8ecbe00640 1 -- 172.21.15.102:0/1776880342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8eb4005180 msgr2=0x7f8eb400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:24.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.492+0000 7f8ecbe00640 1 --2- 172.21.15.102:0/1776880342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8eb4005180 0x7f8eb400db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:24.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.492+0000 7f8ecbe00640 1 -- 172.21.15.102:0/1776880342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8eb40039f0 msgr2=0x7f8eb4003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:24.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.492+0000 7f8ecbe00640 1 --2- 172.21.15.102:0/1776880342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8eb40039f0 0x7f8eb4003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:24.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.492+0000 7f8ecbe00640 1 -- 172.21.15.102:0/1776880342 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb4003520 con 0x7f8eb40047c0 2024-03-23T03:51:24.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.492+0000 7f8ed1800640 1 --2- 172.21.15.102:0/1776880342 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8eb4005180 0x7f8eb400db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:24.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.492+0000 7f8ed1800640 1 -- 172.21.15.102:0/1776880342 reap_dead start 2024-03-23T03:51:24.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.493+0000 7f8ecbe00640 1 --2- 172.21.15.102:0/1776880342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb40047c0 0x7f8eb4004c40 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8007fa0 tx=0x7f8eb40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c60dfbfa67ff618e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:24.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.493+0000 7f8ecb400640 1 -- 172.21.15.102:0/1776880342 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb8005ad0 con 0x7f8eb40047c0 2024-03-23T03:51:24.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.493+0000 7f8ecb400640 1 -- 172.21.15.102:0/1776880342 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb8005c50 con 0x7f8eb40047c0 2024-03-23T03:51:24.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.493+0000 7f8ecb400640 1 -- 172.21.15.102:0/1776880342 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb8007070 con 0x7f8eb40047c0 2024-03-23T03:51:24.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.494+0000 7f8ed3800640 1 -- 172.21.15.102:0/1776880342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb40047c0 msgr2=0x7f8eb4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:24.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.494+0000 7f8ed3800640 1 --2- 172.21.15.102:0/1776880342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb40047c0 0x7f8eb4004c40 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8007fa0 tx=0x7f8eb40057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:24.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.494+0000 7f8ed3800640 1 -- 172.21.15.102:0/1776880342 shutdown_connections 2024-03-23T03:51:24.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.494+0000 7f8ed3800640 1 --2- 172.21.15.102:0/1776880342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb40047c0 0x7f8eb4004c40 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:24.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.494+0000 7f8ed3800640 1 -- 172.21.15.102:0/1776880342 >> 172.21.15.102:0/1776880342 conn(0x7f8ecc06a0b0 msgr2=0x7f8ecc06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:24.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.494+0000 7f8ed3800640 1 -- 172.21.15.102:0/1776880342 shutdown_connections 2024-03-23T03:51:24.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.494+0000 7f8ed3800640 1 -- 172.21.15.102:0/1776880342 wait complete. 2024-03-23T03:51:24.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.495+0000 7f8ed3800640 1 Processor -- start 2024-03-23T03:51:24.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.496+0000 7f8ed3800640 1 -- start start 2024-03-23T03:51:24.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.497+0000 7f8ed3800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb40047c0 0x7f8ecc140200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:24.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.497+0000 7f8ed3800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ecc15bae0 0x7f8ecc15bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:24.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.497+0000 7f8ed3800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8ecc0b3a90 0x7f8ecc0b3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:24.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.497+0000 7f8ed3800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc0a76b0 con 0x7f8eb40047c0 2024-03-23T03:51:24.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.497+0000 7f8ed3800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc0b4450 con 0x7f8ecc15bae0 2024-03-23T03:51:24.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.497+0000 7f8ed3800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ecc1438a0 con 0x7f8ecc0b3a90 2024-03-23T03:51:24.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.497+0000 7f8ed0e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb40047c0 0x7f8ecc140200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:24.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.497+0000 7f8ed0e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb40047c0 0x7f8ecc140200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42528/0 (socket says 172.21.15.102:42528) 2024-03-23T03:51:24.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.497+0000 7f8ed0e00640 1 -- 172.21.15.102:0/597019130 learned_addr learned my addr 172.21.15.102:0/597019130 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:24.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.497+0000 7f8ed0e00640 1 -- 172.21.15.102:0/597019130 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8ecc0b3a90 msgr2=0x7f8ecc0b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:24.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.497+0000 7f8ed0e00640 1 --2- 172.21.15.102:0/597019130 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8ecc0b3a90 0x7f8ecc0b3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:24.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.497+0000 7f8ecbe00640 1 --2- 172.21.15.102:0/597019130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ecc15bae0 0x7f8ecc15bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:24.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.497+0000 7f8ed0e00640 1 -- 172.21.15.102:0/597019130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ecc15bae0 msgr2=0x7f8ecc15bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:24.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.497+0000 7f8ed0e00640 1 --2- 172.21.15.102:0/597019130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ecc15bae0 0x7f8ecc15bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:24.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.497+0000 7f8ed0e00640 1 -- 172.21.15.102:0/597019130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ecc1429b0 con 0x7f8eb40047c0 2024-03-23T03:51:24.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.498+0000 7f8ed0e00640 1 --2- 172.21.15.102:0/597019130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb40047c0 0x7f8ecc140200 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000e3c0 tx=0x7f8ec000e880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:24.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.498+0000 7f8ec8c00640 1 -- 172.21.15.102:0/597019130 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec00052e0 con 0x7f8eb40047c0 2024-03-23T03:51:24.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.498+0000 7f8ec8c00640 1 -- 172.21.15.102:0/597019130 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec0005440 con 0x7f8eb40047c0 2024-03-23T03:51:24.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.498+0000 7f8ec8c00640 1 -- 172.21.15.102:0/597019130 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec0003810 con 0x7f8eb40047c0 2024-03-23T03:51:24.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.498+0000 7f8ed3800640 1 -- 172.21.15.102:0/597019130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ecc142c90 con 0x7f8eb40047c0 2024-03-23T03:51:24.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.498+0000 7f8ed3800640 1 -- 172.21.15.102:0/597019130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ecc143250 con 0x7f8eb40047c0 2024-03-23T03:51:24.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.499+0000 7f8ed3800640 1 -- 172.21.15.102:0/597019130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ecc073390 con 0x7f8eb40047c0 2024-03-23T03:51:24.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.500+0000 7f8ec8c00640 1 -- 172.21.15.102:0/597019130 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f8ec0003970 con 0x7f8eb40047c0 2024-03-23T03:51:24.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.501+0000 7f8ec8c00640 1 --2- 172.21.15.102:0/597019130 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8eac06ac00 0x7f8eac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:24.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.501+0000 7f8ec8c00640 1 -- 172.21.15.102:0/597019130 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8ec0088a70 con 0x7f8eb40047c0 2024-03-23T03:51:24.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.501+0000 7f8ecbe00640 1 --2- 172.21.15.102:0/597019130 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8eac06ac00 0x7f8eac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:24.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.502+0000 7f8ecbe00640 1 --2- 172.21.15.102:0/597019130 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8eac06ac00 0x7f8eac06d0c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8034000 tx=0x7f8eb8007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:24.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.505+0000 7f8ec8c00640 1 -- 172.21.15.102:0/597019130 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f8ec0053c50 con 0x7f8eb40047c0 2024-03-23T03:51:24.738 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:24.738+0000 7f8ed3800640 1 -- 172.21.15.102:0/597019130 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8ecc15b670 con 0x7f8eb40047c0 2024-03-23T03:51:25.731 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.730+0000 7f8ec8c00640 1 -- 172.21.15.102:0/597019130 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v56) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8ec0019020 con 0x7f8eb40047c0 2024-03-23T03:51:25.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.734+0000 7f8ed3800640 1 -- 172.21.15.102:0/597019130 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8eac06ac00 msgr2=0x7f8eac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:25.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.734+0000 7f8ed3800640 1 --2- 172.21.15.102:0/597019130 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8eac06ac00 0x7f8eac06d0c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8eb8034000 tx=0x7f8eb8007000 comp rx=0 tx=0).stop 2024-03-23T03:51:25.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.734+0000 7f8ed3800640 1 -- 172.21.15.102:0/597019130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb40047c0 msgr2=0x7f8ecc140200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:25.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.734+0000 7f8ed3800640 1 --2- 172.21.15.102:0/597019130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb40047c0 0x7f8ecc140200 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f8ec000e3c0 tx=0x7f8ec000e880 comp rx=0 tx=0).stop 2024-03-23T03:51:25.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.734+0000 7f8ed3800640 1 -- 172.21.15.102:0/597019130 shutdown_connections 2024-03-23T03:51:25.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.734+0000 7f8ed3800640 1 --2- 172.21.15.102:0/597019130 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f8eac06ac00 0x7f8eac06d0c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:25.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.734+0000 7f8ed3800640 1 --2- 172.21.15.102:0/597019130 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f8ecc0b3a90 0x7f8ecc0b3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:25.735 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.734+0000 7f8ed3800640 1 --2- 172.21.15.102:0/597019130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8ecc15bae0 0x7f8ecc15bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:25.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.734+0000 7f8ed3800640 1 --2- 172.21.15.102:0/597019130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8eb40047c0 0x7f8ecc140200 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:25.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.734+0000 7f8ed3800640 1 -- 172.21.15.102:0/597019130 >> 172.21.15.102:0/597019130 conn(0x7f8ecc06a0b0 msgr2=0x7f8ecc0b2da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:25.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.734+0000 7f8ed3800640 1 -- 172.21.15.102:0/597019130 shutdown_connections 2024-03-23T03:51:25.736 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.734+0000 7f8ed3800640 1 -- 172.21.15.102:0/597019130 wait complete. 2024-03-23T03:51:25.755 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T03:51:25.755 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:51:25.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.958+0000 7fc798800640 1 Processor -- start 2024-03-23T03:51:25.959 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.958+0000 7fc798800640 1 -- start start 2024-03-23T03:51:25.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.959+0000 7fc798800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc79015cd70 0x7fc7900d0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:25.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.959+0000 7fc798800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc7900b4a50 0x7fc7900b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:25.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.959+0000 7fc798800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7900d17d0 con 0x7fc7900d1330 2024-03-23T03:51:25.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.959+0000 7fc798800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc79015d150 con 0x7fc7900b4a50 2024-03-23T03:51:25.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.959+0000 7fc798800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7900d1ab0 con 0x7fc79015cd70 2024-03-23T03:51:25.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.959+0000 7fc796800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc7900b4a50 0x7fc7900b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:25.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.960+0000 7fc796800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc7900b4a50 0x7fc7900b5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42542/0 (socket says 172.21.15.102:42542) 2024-03-23T03:51:25.960 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.960+0000 7fc796800640 1 -- 172.21.15.102:0/2926262760 learned_addr learned my addr 172.21.15.102:0/2926262760 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:25.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.960+0000 7fc796800640 1 -- 172.21.15.102:0/2926262760 >> v2:172.21.15.173:3300/0 conn(0x7fc79015cd70 msgr2=0x7fc7900d0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:25.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.960+0000 7fc796800640 1 --2- 172.21.15.102:0/2926262760 >> v2:172.21.15.173:3300/0 conn(0x7fc79015cd70 0x7fc7900d0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:25.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.960+0000 7fc796800640 1 -- 172.21.15.102:0/2926262760 >> v1:172.21.15.173:6789/0 conn(0x7fc7900d1330 legacy=0x7fc7900b4340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:25.961 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.960+0000 7fc796800640 1 -- 172.21.15.102:0/2926262760 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7900d1d90 con 0x7fc7900b4a50 2024-03-23T03:51:25.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.961+0000 7fc796800640 1 --2- 172.21.15.102:0/2926262760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7900b4a50 0x7fc7900b5ba0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fc79008b520 tx=0x7fc78c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d52a380a38f5a8e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:25.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.961+0000 7fc794a00640 1 -- 172.21.15.102:0/2926262760 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c02c9f0 con 0x7fc7900b4a50 2024-03-23T03:51:25.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.961+0000 7fc794a00640 1 -- 172.21.15.102:0/2926262760 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc78c02cb50 con 0x7fc7900b4a50 2024-03-23T03:51:25.962 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.961+0000 7fc794a00640 1 -- 172.21.15.102:0/2926262760 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc78c0343f0 con 0x7fc7900b4a50 2024-03-23T03:51:25.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.962+0000 7fc798800640 1 -- 172.21.15.102:0/2926262760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7900b4a50 msgr2=0x7fc7900b5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:25.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.962+0000 7fc798800640 1 --2- 172.21.15.102:0/2926262760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7900b4a50 0x7fc7900b5ba0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fc79008b520 tx=0x7fc78c02be50 comp rx=0 tx=0).stop 2024-03-23T03:51:25.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.962+0000 7fc798800640 1 -- 172.21.15.102:0/2926262760 shutdown_connections 2024-03-23T03:51:25.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.962+0000 7fc798800640 1 --2- 172.21.15.102:0/2926262760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7900b4a50 0x7fc7900b5ba0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:25.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.962+0000 7fc798800640 1 --2- 172.21.15.102:0/2926262760 >> v2:172.21.15.173:3300/0 conn(0x7fc79015cd70 0x7fc7900d0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:25.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.962+0000 7fc798800640 1 -- 172.21.15.102:0/2926262760 >> 172.21.15.102:0/2926262760 conn(0x7fc7900cf340 msgr2=0x7fc7900cf750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:25.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.962+0000 7fc798800640 1 -- 172.21.15.102:0/2926262760 shutdown_connections 2024-03-23T03:51:25.963 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.962+0000 7fc798800640 1 -- 172.21.15.102:0/2926262760 wait complete. 2024-03-23T03:51:25.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.964+0000 7fc798800640 1 Processor -- start 2024-03-23T03:51:25.964 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.964+0000 7fc798800640 1 -- start start 2024-03-23T03:51:25.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.965+0000 7fc798800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7900b4a50 0x7fc79015fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:25.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.965+0000 7fc798800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7900d1330 0x7fc790153790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:25.965 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.965+0000 7fc798800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc79015cd70 0x7fc7900b72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:25.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.965+0000 7fc798800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7900ca880 con 0x7fc7900d1330 2024-03-23T03:51:25.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.965+0000 7fc798800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7901600a0 con 0x7fc79015cd70 2024-03-23T03:51:25.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.965+0000 7fc798800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7900c9990 con 0x7fc7900b4a50 2024-03-23T03:51:25.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.965+0000 7fc795400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7900d1330 0x7fc790153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:25.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.965+0000 7fc795400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7900d1330 0x7fc790153790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42558/0 (socket says 172.21.15.102:42558) 2024-03-23T03:51:25.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.965+0000 7fc795400640 1 -- 172.21.15.102:0/3333505832 learned_addr learned my addr 172.21.15.102:0/3333505832 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:25.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.965+0000 7fc795400640 1 -- 172.21.15.102:0/3333505832 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7900b4a50 msgr2=0x7fc79015fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:25.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.965+0000 7fc795400640 1 --2- 172.21.15.102:0/3333505832 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7900b4a50 0x7fc79015fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:25.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.965+0000 7fc795400640 1 -- 172.21.15.102:0/3333505832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc79015cd70 msgr2=0x7fc7900b72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:25.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.965+0000 7fc795400640 1 --2- 172.21.15.102:0/3333505832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc79015cd70 0x7fc7900b72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:25.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.965+0000 7fc795400640 1 -- 172.21.15.102:0/3333505832 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc78c009b30 con 0x7fc7900d1330 2024-03-23T03:51:25.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.966+0000 7fc795400640 1 --2- 172.21.15.102:0/3333505832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7900d1330 0x7fc790153790 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fc77c00c400 tx=0x7fc77c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:25.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.966+0000 7fc78a000640 1 -- 172.21.15.102:0/3333505832 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c011790 con 0x7fc7900d1330 2024-03-23T03:51:25.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.966+0000 7fc78a000640 1 -- 172.21.15.102:0/3333505832 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc77c011930 con 0x7fc7900d1330 2024-03-23T03:51:25.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.966+0000 7fc798800640 1 -- 172.21.15.102:0/3333505832 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7900c9c70 con 0x7fc7900d1330 2024-03-23T03:51:25.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.966+0000 7fc78a000640 1 -- 172.21.15.102:0/3333505832 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c011ce0 con 0x7fc7900d1330 2024-03-23T03:51:25.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.966+0000 7fc798800640 1 -- 172.21.15.102:0/3333505832 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7900ca230 con 0x7fc7900d1330 2024-03-23T03:51:25.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.967+0000 7fc798800640 1 -- 172.21.15.102:0/3333505832 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc790073390 con 0x7fc7900d1330 2024-03-23T03:51:25.968 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.968+0000 7fc78a000640 1 -- 172.21.15.102:0/3333505832 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc77c01c460 con 0x7fc7900d1330 2024-03-23T03:51:25.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.969+0000 7fc78a000640 1 --2- 172.21.15.102:0/3333505832 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc77006ac00 0x7fc77006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:25.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.969+0000 7fc78a000640 1 -- 172.21.15.102:0/3333505832 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc77c0894d0 con 0x7fc7900d1330 2024-03-23T03:51:25.969 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.969+0000 7fc795e00640 1 --2- 172.21.15.102:0/3333505832 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc77006ac00 0x7fc77006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:25.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.970+0000 7fc795e00640 1 --2- 172.21.15.102:0/3333505832 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc77006ac00 0x7fc77006d0c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc784009670 tx=0x7fc784006bf0 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:25.974 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:25.973+0000 7fc78a000640 1 -- 172.21.15.102:0/3333505832 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc77c0544d0 con 0x7fc7900d1330 2024-03-23T03:51:26.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.208+0000 7fc798800640 1 -- 172.21.15.102:0/3333505832 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7900b6fc0 con 0x7fc7900d1330 2024-03-23T03:51:26.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.209+0000 7fc78a000640 1 -- 172.21.15.102:0/3333505832 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fc77c057af0 con 0x7fc7900d1330 2024-03-23T03:51:26.210 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:26.210 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2017025570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2017025570},{"type":"v1","addr":"172.21.15.173:6835","nonce":2017025570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/173893445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":173893445},{"type":"v1","addr":"172.21.15.102:6835","nonce":173893445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/255507815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":255507815},{"type":"v1","addr":"172.21.15.173:6833","nonce":255507815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:51:22.228808+0000","modified":"2024-03-23T03:51:25.724445+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7212":{"gid":7212,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/883705787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":883705787},{"type":"v1","addr":"172.21.15.102:6837","nonce":883705787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T03:51:26.210 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 56 2024-03-23T03:51:26.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.212+0000 7fc798800640 1 -- 172.21.15.102:0/3333505832 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc77006ac00 msgr2=0x7fc77006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:26.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.212+0000 7fc798800640 1 --2- 172.21.15.102:0/3333505832 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc77006ac00 0x7fc77006d0c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc784009670 tx=0x7fc784006bf0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.212+0000 7fc798800640 1 -- 172.21.15.102:0/3333505832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7900d1330 msgr2=0x7fc790153790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:26.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.212+0000 7fc798800640 1 --2- 172.21.15.102:0/3333505832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7900d1330 0x7fc790153790 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7fc77c00c400 tx=0x7fc77c00c8c0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.212+0000 7fc798800640 1 -- 172.21.15.102:0/3333505832 shutdown_connections 2024-03-23T03:51:26.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.212+0000 7fc798800640 1 --2- 172.21.15.102:0/3333505832 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc77006ac00 0x7fc77006d0c0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.212+0000 7fc798800640 1 --2- 172.21.15.102:0/3333505832 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc79015cd70 0x7fc7900b72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.212+0000 7fc798800640 1 --2- 172.21.15.102:0/3333505832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7900d1330 0x7fc790153790 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.212+0000 7fc798800640 1 --2- 172.21.15.102:0/3333505832 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7900b4a50 0x7fc79015fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.212+0000 7fc798800640 1 -- 172.21.15.102:0/3333505832 >> 172.21.15.102:0/3333505832 conn(0x7fc7900cf340 msgr2=0x7fc7900b34b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:26.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.212+0000 7fc798800640 1 -- 172.21.15.102:0/3333505832 shutdown_connections 2024-03-23T03:51:26.214 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.213+0000 7fc798800640 1 -- 172.21.15.102:0/3333505832 wait complete. 2024-03-23T03:51:26.234 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:51:26.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.439+0000 7fc86fc00640 1 Processor -- start 2024-03-23T03:51:26.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.439+0000 7fc86fc00640 1 -- start start 2024-03-23T03:51:26.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.440+0000 7fc86fc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc86815b560 0x7fc86815b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:26.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.440+0000 7fc86fc00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fc8680b39b0 0x7fc8680b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:26.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.440+0000 7fc86fc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc86815e500 0x7fc86815e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:26.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.440+0000 7fc86fc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8680b4370 con 0x7fc86815e500 2024-03-23T03:51:26.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.440+0000 7fc86fc00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc86806c6f0 con 0x7fc8680b39b0 2024-03-23T03:51:26.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.440+0000 7fc86fc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86806c9d0 con 0x7fc86815b560 2024-03-23T03:51:26.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.441+0000 7fc86dc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc86815e500 0x7fc86815e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:26.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.441+0000 7fc86dc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fc86815e500 0x7fc86815e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42572/0 (socket says 172.21.15.102:42572) 2024-03-23T03:51:26.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.441+0000 7fc86dc00640 1 -- 172.21.15.102:0/2091047611 learned_addr learned my addr 172.21.15.102:0/2091047611 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:26.443 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.441+0000 7fc867e00640 1 --2- 172.21.15.102:0/2091047611 >> v2:172.21.15.173:3300/0 conn(0x7fc8680b39b0 0x7fc8680b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:26.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.441+0000 7fc86d200640 1 --2- 172.21.15.102:0/2091047611 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc86815b560 0x7fc86815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:26.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.441+0000 7fc86dc00640 1 -- 172.21.15.102:0/2091047611 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc86815b560 msgr2=0x7fc86815b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:26.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.441+0000 7fc86dc00640 1 --2- 172.21.15.102:0/2091047611 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc86815b560 0x7fc86815b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.441+0000 7fc86dc00640 1 -- 172.21.15.102:0/2091047611 >> v2:172.21.15.173:3300/0 conn(0x7fc8680b39b0 msgr2=0x7fc8680b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:26.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.441+0000 7fc86dc00640 1 --2- 172.21.15.102:0/2091047611 >> v2:172.21.15.173:3300/0 conn(0x7fc8680b39b0 0x7fc8680b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.441+0000 7fc86dc00640 1 -- 172.21.15.102:0/2091047611 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86806ccb0 con 0x7fc86815e500 2024-03-23T03:51:26.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.441+0000 7fc86d200640 1 --2- 172.21.15.102:0/2091047611 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc86815b560 0x7fc86815b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:26.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.442+0000 7fc86dc00640 1 --2- 172.21.15.102:0/2091047611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc86815e500 0x7fc86815e990 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fc868071510 tx=0x7fc86002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cee9db527ad80214 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:26.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.442+0000 7fc867400640 1 -- 172.21.15.102:0/2091047611 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86002ca80 con 0x7fc86815e500 2024-03-23T03:51:26.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.443+0000 7fc867400640 1 -- 172.21.15.102:0/2091047611 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86002cbe0 con 0x7fc86815e500 2024-03-23T03:51:26.445 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.443+0000 7fc867400640 1 -- 172.21.15.102:0/2091047611 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8600343f0 con 0x7fc86815e500 2024-03-23T03:51:26.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.443+0000 7fc86fc00640 1 -- 172.21.15.102:0/2091047611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc86815e500 msgr2=0x7fc86815e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:26.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.443+0000 7fc86fc00640 1 --2- 172.21.15.102:0/2091047611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc86815e500 0x7fc86815e990 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7fc868071510 tx=0x7fc86002bee0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.443+0000 7fc86fc00640 1 -- 172.21.15.102:0/2091047611 shutdown_connections 2024-03-23T03:51:26.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.443+0000 7fc86fc00640 1 --2- 172.21.15.102:0/2091047611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc86815e500 0x7fc86815e990 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.443+0000 7fc86fc00640 1 --2- 172.21.15.102:0/2091047611 >> v2:172.21.15.173:3300/0 conn(0x7fc8680b39b0 0x7fc8680b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.443+0000 7fc86fc00640 1 --2- 172.21.15.102:0/2091047611 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc86815b560 0x7fc86815b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.443+0000 7fc86fc00640 1 -- 172.21.15.102:0/2091047611 >> 172.21.15.102:0/2091047611 conn(0x7fc86806a3d0 msgr2=0x7fc8680b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:26.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.443+0000 7fc86fc00640 1 -- 172.21.15.102:0/2091047611 shutdown_connections 2024-03-23T03:51:26.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.443+0000 7fc86fc00640 1 -- 172.21.15.102:0/2091047611 wait complete. 2024-03-23T03:51:26.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.445+0000 7fc86fc00640 1 Processor -- start 2024-03-23T03:51:26.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.445+0000 7fc86fc00640 1 -- start start 2024-03-23T03:51:26.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.446+0000 7fc86fc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8680b39b0 0x7fc8681f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:26.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.446+0000 7fc86fc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc86815b560 0x7fc868159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:26.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.446+0000 7fc86fc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc86815e500 0x7fc8680accc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:26.448 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.446+0000 7fc86fc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8681f31f0 con 0x7fc86815e500 2024-03-23T03:51:26.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.446+0000 7fc86fc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8681f3900 con 0x7fc86815b560 2024-03-23T03:51:26.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.446+0000 7fc86fc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8681f2c30 con 0x7fc8680b39b0 2024-03-23T03:51:26.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.446+0000 7fc86dc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc86815e500 0x7fc8680accc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:26.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.446+0000 7fc86dc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc86815e500 0x7fc8680accc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42576/0 (socket says 172.21.15.102:42576) 2024-03-23T03:51:26.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.446+0000 7fc86dc00640 1 -- 172.21.15.102:0/2749671588 learned_addr learned my addr 172.21.15.102:0/2749671588 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:26.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.447+0000 7fc867e00640 1 --2- 172.21.15.102:0/2749671588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc86815b560 0x7fc868159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:26.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.447+0000 7fc86dc00640 1 -- 172.21.15.102:0/2749671588 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8680b39b0 msgr2=0x7fc8681f2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:26.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.447+0000 7fc86d200640 1 --2- 172.21.15.102:0/2749671588 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8680b39b0 0x7fc8681f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:26.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.447+0000 7fc86dc00640 1 --2- 172.21.15.102:0/2749671588 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8680b39b0 0x7fc8681f2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.447+0000 7fc86dc00640 1 -- 172.21.15.102:0/2749671588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc86815b560 msgr2=0x7fc868159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:26.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.447+0000 7fc86dc00640 1 --2- 172.21.15.102:0/2749671588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc86815b560 0x7fc868159b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.447+0000 7fc86dc00640 1 -- 172.21.15.102:0/2749671588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc860009b30 con 0x7fc86815e500 2024-03-23T03:51:26.449 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.447+0000 7fc867e00640 1 --2- 172.21.15.102:0/2749671588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc86815b560 0x7fc868159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:26.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.447+0000 7fc86dc00640 1 --2- 172.21.15.102:0/2749671588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc86815e500 0x7fc8680accc0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fc860036040 tx=0x7fc860004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:26.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.447+0000 7fc864c00640 1 -- 172.21.15.102:0/2749671588 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc860009510 con 0x7fc86815e500 2024-03-23T03:51:26.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.447+0000 7fc864c00640 1 -- 172.21.15.102:0/2749671588 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc86003e450 con 0x7fc86815e500 2024-03-23T03:51:26.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.447+0000 7fc86fc00640 1 -- 172.21.15.102:0/2749671588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8681f2f10 con 0x7fc86815e500 2024-03-23T03:51:26.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.447+0000 7fc864c00640 1 -- 172.21.15.102:0/2749671588 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc86003e800 con 0x7fc86815e500 2024-03-23T03:51:26.450 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.448+0000 7fc86fc00640 1 -- 172.21.15.102:0/2749671588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8681f2790 con 0x7fc86815e500 2024-03-23T03:51:26.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.449+0000 7fc86fc00640 1 -- 172.21.15.102:0/2749671588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8681f3a40 con 0x7fc86815e500 2024-03-23T03:51:26.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.449+0000 7fc864c00640 1 -- 172.21.15.102:0/2749671588 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc86002cd50 con 0x7fc86815e500 2024-03-23T03:51:26.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.450+0000 7fc864c00640 1 --2- 172.21.15.102:0/2749671588 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc84406ac00 0x7fc84406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:26.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.450+0000 7fc864c00640 1 -- 172.21.15.102:0/2749671588 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc8600abf70 con 0x7fc86815e500 2024-03-23T03:51:26.451 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.450+0000 7fc86d200640 1 --2- 172.21.15.102:0/2749671588 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc84406ac00 0x7fc84406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:26.452 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.451+0000 7fc86d200640 1 --2- 172.21.15.102:0/2749671588 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc84406ac00 0x7fc84406d0c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc8681515d0 tx=0x7fc858008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:26.455 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.455+0000 7fc864c00640 1 -- 172.21.15.102:0/2749671588 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc8681f3a40 con 0x7fc86815e500 2024-03-23T03:51:26.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.656+0000 7fc86fc00640 1 -- 172.21.15.102:0/2749671588 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8681f3a40 con 0x7fc86815e500 2024-03-23T03:51:26.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.658+0000 7fc864c00640 1 -- 172.21.15.102:0/2749671588 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+16296 (secure 0 0 0) 0x7fc8681f3a40 con 0x7fc86815e500 2024-03-23T03:51:26.659 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:26.659 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":105,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:51:23.233227+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:51:19.796618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-23T03:51:21.283447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:51:26.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.660+0000 7fc86fc00640 1 -- 172.21.15.102:0/2749671588 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc84406ac00 msgr2=0x7fc84406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:26.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.660+0000 7fc86fc00640 1 --2- 172.21.15.102:0/2749671588 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc84406ac00 0x7fc84406d0c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fc8681515d0 tx=0x7fc858008040 comp rx=0 tx=0).stop 2024-03-23T03:51:26.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.660+0000 7fc86fc00640 1 -- 172.21.15.102:0/2749671588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc86815e500 msgr2=0x7fc8680accc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:26.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.660+0000 7fc86fc00640 1 --2- 172.21.15.102:0/2749671588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc86815e500 0x7fc8680accc0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7fc860036040 tx=0x7fc860004a40 comp rx=0 tx=0).stop 2024-03-23T03:51:26.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.661+0000 7fc86fc00640 1 -- 172.21.15.102:0/2749671588 shutdown_connections 2024-03-23T03:51:26.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.661+0000 7fc86fc00640 1 --2- 172.21.15.102:0/2749671588 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc84406ac00 0x7fc84406d0c0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.661+0000 7fc86fc00640 1 --2- 172.21.15.102:0/2749671588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc86815e500 0x7fc8680accc0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.661+0000 7fc86fc00640 1 --2- 172.21.15.102:0/2749671588 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc86815b560 0x7fc868159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.661+0000 7fc86fc00640 1 --2- 172.21.15.102:0/2749671588 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc8680b39b0 0x7fc8681f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.661+0000 7fc86fc00640 1 -- 172.21.15.102:0/2749671588 >> 172.21.15.102:0/2749671588 conn(0x7fc86806a3d0 msgr2=0x7fc8680ae310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:26.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.661+0000 7fc86fc00640 1 -- 172.21.15.102:0/2749671588 shutdown_connections 2024-03-23T03:51:26.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.661+0000 7fc86fc00640 1 -- 172.21.15.102:0/2749671588 wait complete. 2024-03-23T03:51:26.681 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T03:51:26.682 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:51:26.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.886+0000 7f5c7ac00640 1 Processor -- start 2024-03-23T03:51:26.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.886+0000 7f5c7ac00640 1 -- start start 2024-03-23T03:51:26.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.888+0000 7f5c7ac00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f5c74160800 0x7f5c740b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:26.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.888+0000 7f5c7ac00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c740b2030 con 0x7f5c7413faa0 2024-03-23T03:51:26.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.888+0000 7f5c7ac00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c74161640 con 0x7f5c740b1be0 2024-03-23T03:51:26.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.888+0000 7f5c7ac00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c740b38a0 con 0x7f5c74160800 2024-03-23T03:51:26.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.888+0000 7f5c73e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f5c7413faa0 0x7f5c74160f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38574/0 (socket says 172.21.15.102:38574) 2024-03-23T03:51:26.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.888+0000 7f5c73e00640 1 -- 172.21.15.102:0/810063050 learned_addr learned my addr 172.21.15.102:0/810063050 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:26.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.888+0000 7f5c73400640 1 --2- 172.21.15.102:0/810063050 >> v2:172.21.15.102:3300/0 conn(0x7f5c74160800 0x7f5c740b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:26.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.888+0000 7f5c72a00640 1 -- 172.21.15.102:0/810063050 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507116984 0 0) 0x7f5c64001110 con 0x7f5c7413faa0 2024-03-23T03:51:26.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.889+0000 7f5c72a00640 1 -- 172.21.15.102:0/810063050 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c58003520 con 0x7f5c7413faa0 2024-03-23T03:51:26.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.889+0000 7f5c73400640 1 -- 172.21.15.102:0/810063050 >> v1:172.21.15.173:6789/0 conn(0x7f5c740b1be0 legacy=0x7f5c7406c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:26.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.889+0000 7f5c73400640 1 -- 172.21.15.102:0/810063050 >> v1:172.21.15.102:6789/0 conn(0x7f5c7413faa0 legacy=0x7f5c74160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:26.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.889+0000 7f5c73400640 1 -- 172.21.15.102:0/810063050 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c740b3b80 con 0x7f5c74160800 2024-03-23T03:51:26.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.890+0000 7f5c73400640 1 --2- 172.21.15.102:0/810063050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c74160800 0x7f5c740b16a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f5c600097d0 tx=0x7f5c6002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c336077293711753 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:26.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.890+0000 7f5c72a00640 1 -- 172.21.15.102:0/810063050 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c60031830 con 0x7f5c74160800 2024-03-23T03:51:26.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.890+0000 7f5c72a00640 1 -- 172.21.15.102:0/810063050 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c60031990 con 0x7f5c74160800 2024-03-23T03:51:26.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.890+0000 7f5c72a00640 1 -- 172.21.15.102:0/810063050 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c60031d20 con 0x7f5c74160800 2024-03-23T03:51:26.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.891+0000 7f5c7ac00640 1 -- 172.21.15.102:0/810063050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c74160800 msgr2=0x7f5c740b16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:26.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.891+0000 7f5c7ac00640 1 --2- 172.21.15.102:0/810063050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c74160800 0x7f5c740b16a0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f5c600097d0 tx=0x7f5c6002b8d0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.891+0000 7f5c7ac00640 1 -- 172.21.15.102:0/810063050 shutdown_connections 2024-03-23T03:51:26.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.891+0000 7f5c7ac00640 1 --2- 172.21.15.102:0/810063050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c74160800 0x7f5c740b16a0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.891+0000 7f5c7ac00640 1 -- 172.21.15.102:0/810063050 >> 172.21.15.102:0/810063050 conn(0x7f5c7406a3d0 msgr2=0x7f5c740b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:26.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.891+0000 7f5c7ac00640 1 -- 172.21.15.102:0/810063050 shutdown_connections 2024-03-23T03:51:26.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.891+0000 7f5c7ac00640 1 -- 172.21.15.102:0/810063050 wait complete. 2024-03-23T03:51:26.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.893+0000 7f5c7ac00640 1 Processor -- start 2024-03-23T03:51:26.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.893+0000 7f5c7ac00640 1 -- start start 2024-03-23T03:51:26.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.894+0000 7f5c7ac00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c740b1be0 0x7f5c740cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.894+0000 7f5c7ac00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5c7413faa0 0x7f5c74153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.894+0000 7f5c7ac00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c74160800 0x7f5c7413dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.894+0000 7f5c7ac00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c740cc410 con 0x7f5c740b1be0 2024-03-23T03:51:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.894+0000 7f5c7ac00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c740cc6f0 con 0x7f5c74160800 2024-03-23T03:51:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.894+0000 7f5c7ac00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c740cc9d0 con 0x7f5c7413faa0 2024-03-23T03:51:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.894+0000 7f5c73e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c740b1be0 0x7f5c740cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.894+0000 7f5c73e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c740b1be0 0x7f5c740cd4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42590/0 (socket says 172.21.15.102:42590) 2024-03-23T03:51:26.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.894+0000 7f5c73e00640 1 -- 172.21.15.102:0/2395549591 learned_addr learned my addr 172.21.15.102:0/2395549591 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.894+0000 7f5c78c00640 1 --2- 172.21.15.102:0/2395549591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c74160800 0x7f5c7413dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.895+0000 7f5c73400640 1 --2- 172.21.15.102:0/2395549591 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5c7413faa0 0x7f5c74153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.895+0000 7f5c73e00640 1 -- 172.21.15.102:0/2395549591 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5c7413faa0 msgr2=0x7f5c74153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:26.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.895+0000 7f5c73e00640 1 --2- 172.21.15.102:0/2395549591 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5c7413faa0 0x7f5c74153460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.895+0000 7f5c73e00640 1 -- 172.21.15.102:0/2395549591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c74160800 msgr2=0x7f5c7413dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:26.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.895+0000 7f5c73e00640 1 --2- 172.21.15.102:0/2395549591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c74160800 0x7f5c7413dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:26.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.895+0000 7f5c73e00640 1 -- 172.21.15.102:0/2395549591 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c60009400 con 0x7f5c740b1be0 2024-03-23T03:51:26.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.895+0000 7f5c73e00640 1 --2- 172.21.15.102:0/2395549591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c740b1be0 0x7f5c740cd4b0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400c700 tx=0x7f5c6400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:26.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.895+0000 7f5c6be00640 1 -- 172.21.15.102:0/2395549591 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c64005ab0 con 0x7f5c740b1be0 2024-03-23T03:51:26.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.895+0000 7f5c6be00640 1 -- 172.21.15.102:0/2395549591 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c64005c10 con 0x7f5c740b1be0 2024-03-23T03:51:26.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.895+0000 7f5c6be00640 1 -- 172.21.15.102:0/2395549591 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c64001d50 con 0x7f5c740b1be0 2024-03-23T03:51:26.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.895+0000 7f5c7ac00640 1 -- 172.21.15.102:0/2395549591 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c740cccb0 con 0x7f5c740b1be0 2024-03-23T03:51:26.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.896+0000 7f5c7ac00640 1 -- 172.21.15.102:0/2395549591 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c741539a0 con 0x7f5c740b1be0 2024-03-23T03:51:26.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.897+0000 7f5c7ac00640 1 -- 172.21.15.102:0/2395549591 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c740cd1d0 con 0x7f5c740b1be0 2024-03-23T03:51:26.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.898+0000 7f5c6be00640 1 -- 172.21.15.102:0/2395549591 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f5c64007070 con 0x7f5c740b1be0 2024-03-23T03:51:26.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.898+0000 7f5c6be00640 1 --2- 172.21.15.102:0/2395549591 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5c5406ac50 0x7f5c5406d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:26.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.899+0000 7f5c6be00640 1 -- 172.21.15.102:0/2395549591 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5c64088870 con 0x7f5c740b1be0 2024-03-23T03:51:26.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.899+0000 7f5c73400640 1 --2- 172.21.15.102:0/2395549591 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5c5406ac50 0x7f5c5406d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:26.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.900+0000 7f5c73400640 1 --2- 172.21.15.102:0/2395549591 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5c5406ac50 0x7f5c5406d110 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5c60036000 tx=0x7f5c6003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:26.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:26.903+0000 7f5c6be00640 1 -- 172.21.15.102:0/2395549591 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f5c740cd1d0 con 0x7f5c740b1be0 2024-03-23T03:51:27.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.095+0000 7f5c7ac00640 1 -- 172.21.15.102:0/2395549591 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5c740b4420 con 0x7f5c5406ac50 2024-03-23T03:51:27.113 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.113+0000 7f5c6be00640 1 -- 172.21.15.102:0/2395549591 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284760 (secure 0 0 0) 0x7f5c60044400 con 0x7f5c5406ac50 2024-03-23T03:51:27.117 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:27.119 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:51:27.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.118+0000 7f5c7ac00640 1 -- 172.21.15.102:0/2395549591 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5c5406ac50 msgr2=0x7f5c5406d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:27.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.118+0000 7f5c7ac00640 1 --2- 172.21.15.102:0/2395549591 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5c5406ac50 0x7f5c5406d110 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f5c60036000 tx=0x7f5c6003c040 comp rx=0 tx=0).stop 2024-03-23T03:51:27.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.118+0000 7f5c7ac00640 1 -- 172.21.15.102:0/2395549591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c740b1be0 msgr2=0x7f5c740cd4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:27.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.118+0000 7f5c7ac00640 1 --2- 172.21.15.102:0/2395549591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c740b1be0 0x7f5c740cd4b0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400c700 tx=0x7f5c6400cbc0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.119+0000 7f5c7ac00640 1 -- 172.21.15.102:0/2395549591 shutdown_connections 2024-03-23T03:51:27.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.119+0000 7f5c7ac00640 1 --2- 172.21.15.102:0/2395549591 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f5c5406ac50 0x7f5c5406d110 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.119+0000 7f5c7ac00640 1 --2- 172.21.15.102:0/2395549591 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c74160800 0x7f5c7413dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.119+0000 7f5c7ac00640 1 --2- 172.21.15.102:0/2395549591 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f5c7413faa0 0x7f5c74153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.119+0000 7f5c7ac00640 1 --2- 172.21.15.102:0/2395549591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c740b1be0 0x7f5c740cd4b0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.119+0000 7f5c7ac00640 1 -- 172.21.15.102:0/2395549591 >> 172.21.15.102:0/2395549591 conn(0x7f5c7406a3d0 msgr2=0x7f5c740ad720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:27.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.119+0000 7f5c7ac00640 1 -- 172.21.15.102:0/2395549591 shutdown_connections 2024-03-23T03:51:27.121 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.119+0000 7f5c7ac00640 1 -- 172.21.15.102:0/2395549591 wait complete. 2024-03-23T03:51:27.137 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-03-23T03:51:25.801531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":251752,"kb_used_data":11016,"kb_used_omap":12,"kb_used_meta":240627,"kb_avail":754722968,"statfs":{"total":773094113280,"available":772836319232,"internally_reserved":0,"allocated":11280384,"data_stored":5734920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":246402618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.067702"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.774604+0000","last_change":"2024-03-23T03:51:21.774604+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.774604+0000","last_undegraded":"2024-03-23T03:51:21.774604+0000","last_fullsized":"2024-03-23T03:51:21.774604+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:24:42.534444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.784556+0000","last_change":"2024-03-23T03:51:21.784556+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.784556+0000","last_undegraded":"2024-03-23T03:51:21.784556+0000","last_fullsized":"2024-03-23T03:51:21.784556+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:11:35.553745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.768447+0000","last_change":"2024-03-23T03:51:21.768447+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.768447+0000","last_undegraded":"2024-03-23T03:51:21.768447+0000","last_fullsized":"2024-03-23T03:51:21.768447+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:20:25.913448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.758852+0000","last_change":"2024-03-23T03:51:21.758852+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.758852+0000","last_undegraded":"2024-03-23T03:51:21.758852+0000","last_fullsized":"2024-03-23T03:51:21.758852+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:36:30.827292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.768312+0000","last_change":"2024-03-23T03:51:21.768312+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.768312+0000","last_undegraded":"2024-03-23T03:51:21.768312+0000","last_fullsized":"2024-03-23T03:51:21.768312+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:46:33.274045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.775572+0000","last_change":"2024-03-23T03:51:21.775572+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.775572+0000","last_undegraded":"2024-03-23T03:51:21.775572+0000","last_fullsized":"2024-03-23T03:51:21.775572+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:15:44.577151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.775304+0000","last_change":"2024-03-23T03:51:21.775304+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.775304+0000","last_undegraded":"2024-03-23T03:51:21.775304+0000","last_fullsized":"2024-03-23T03:51:21.775304+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:19.362177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.767403+0000","last_change":"2024-03-23T03:51:21.767403+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.767403+0000","last_undegraded":"2024-03-23T03:51:21.767403+0000","last_fullsized":"2024-03-23T03:51:21.767403+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:23:32.227733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.781582+0000","last_change":"2024-03-23T03:51:21.781582+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.781582+0000","last_undegraded":"2024-03-23T03:51:21.781582+0000","last_fullsized":"2024-03-23T03:51:21.781582+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:54.013665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.764834+0000","last_change":"2024-03-23T03:51:21.764834+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.764834+0000","last_undegraded":"2024-03-23T03:51:21.764834+0000","last_fullsized":"2024-03-23T03:51:21.764834+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:58:33.997748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.763235+0000","last_change":"2024-03-23T03:51:21.763235+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.763235+0000","last_undegraded":"2024-03-23T03:51:21.763235+0000","last_fullsized":"2024-03-23T03:51:21.763235+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:10:28.823831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.769131+0000","last_change":"2024-03-23T03:51:21.769131+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.769131+0000","last_undegraded":"2024-03-23T03:51:21.769131+0000","last_fullsized":"2024-03-23T03:51:21.769131+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:07.649845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.757932+0000","last_change":"2024-03-23T03:51:21.757932+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.757932+0000","last_undegraded":"2024-03-23T03:51:21.757932+0000","last_fullsized":"2024-03-23T03:51:21.757932+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:35:05.132187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.773754+0000","last_change":"2024-03-23T03:51:21.773754+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.773754+0000","last_undegraded":"2024-03-23T03:51:21.773754+0000","last_fullsized":"2024-03-23T03:51:21.773754+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:30:04.249708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.777084+0000","last_change":"2024-03-23T03:51:21.777084+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.777084+0000","last_undegraded":"2024-03-23T03:51:21.777084+0000","last_fullsized":"2024-03-23T03:51:21.777084+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:17:09.963708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.772977+0000","last_change":"2024-03-23T03:51:21.772977+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.772977+0000","last_undegraded":"2024-03-23T03:51:21.772977+0000","last_fullsized":"2024-03-23T03:51:21.772977+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:04:21.319327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.774158+0000","last_change":"2024-03-23T03:51:21.774158+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.774158+0000","last_undegraded":"2024-03-23T03:51:21.774158+0000","last_fullsized":"2024-03-23T03:51:21.774158+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:55.608125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.783911+0000","last_change":"2024-03-23T03:51:21.783911+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.783911+0000","last_undegraded":"2024-03-23T03:51:21.783911+0000","last_fullsized":"2024-03-23T03:51:21.783911+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:03:27.282303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.778317+0000","last_change":"2024-03-23T03:51:21.778317+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.778317+0000","last_undegraded":"2024-03-23T03:51:21.778317+0000","last_fullsized":"2024-03-23T03:51:21.778317+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:48:26.593847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.765961+0000","last_change":"2024-03-23T03:51:21.765961+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.765961+0000","last_undegraded":"2024-03-23T03:51:21.765961+0000","last_fullsized":"2024-03-23T03:51:21.765961+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:13:39.429313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.767195+0000","last_change":"2024-03-23T03:51:21.767195+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.767195+0000","last_undegraded":"2024-03-23T03:51:21.767195+0000","last_fullsized":"2024-03-23T03:51:21.767195+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:20:51.607176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.764682+0000","last_change":"2024-03-23T03:51:21.764682+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.764682+0000","last_undegraded":"2024-03-23T03:51:21.764682+0000","last_fullsized":"2024-03-23T03:51:21.764682+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:04.683164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.765012+0000","last_change":"2024-03-23T03:51:21.765012+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.765012+0000","last_undegraded":"2024-03-23T03:51:21.765012+0000","last_fullsized":"2024-03-23T03:51:21.765012+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:46:32.018385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.765000+0000","last_change":"2024-03-23T03:51:21.765000+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.765000+0000","last_undegraded":"2024-03-23T03:51:21.765000+0000","last_fullsized":"2024-03-23T03:51:21.765000+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:47.178608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.770079+0000","last_change":"2024-03-23T03:51:21.770079+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.770079+0000","last_undegraded":"2024-03-23T03:51:21.770079+0000","last_fullsized":"2024-03-23T03:51:21.770079+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:58:40.479780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.766309+0000","last_change":"2024-03-23T03:51:21.766309+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.766309+0000","last_undegraded":"2024-03-23T03:51:21.766309+0000","last_fullsized":"2024-03-23T03:51:21.766309+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:58:05.924179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.761725+0000","last_change":"2024-03-23T03:51:21.761725+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.761725+0000","last_undegraded":"2024-03-23T03:51:21.761725+0000","last_fullsized":"2024-03-23T03:51:21.761725+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:20.563242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.776967+0000","last_change":"2024-03-23T03:51:21.776967+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.776967+0000","last_undegraded":"2024-03-23T03:51:21.776967+0000","last_fullsized":"2024-03-23T03:51:21.776967+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:11:23.842491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.766688+0000","last_change":"2024-03-23T03:51:21.766688+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.766688+0000","last_undegraded":"2024-03-23T03:51:21.766688+0000","last_fullsized":"2024-03-23T03:51:21.766688+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:46.517427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.769700+0000","last_change":"2024-03-23T03:51:21.769700+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.769700+0000","last_undegraded":"2024-03-23T03:51:21.769700+0000","last_fullsized":"2024-03-23T03:51:21.769700+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:36:38.929407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.774265+0000","last_change":"2024-03-23T03:51:21.774265+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.774265+0000","last_undegraded":"2024-03-23T03:51:21.774265+0000","last_fullsized":"2024-03-23T03:51:21.774265+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:25:25.635704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.766878+0000","last_change":"2024-03-23T03:51:21.766878+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.766878+0000","last_undegraded":"2024-03-23T03:51:21.766878+0000","last_fullsized":"2024-03-23T03:51:21.766878+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:19:58.455715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.764169+0000","last_change":"2024-03-23T03:51:21.764169+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.764169+0000","last_undegraded":"2024-03-23T03:51:21.764169+0000","last_fullsized":"2024-03-23T03:51:21.764169+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:57:56.460821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.793055+0000","last_change":"2024-03-23T03:51:21.789123+0000","last_active":"2024-03-23T03:51:21.793055+0000","last_peered":"2024-03-23T03:51:21.793055+0000","last_clean":"2024-03-23T03:51:21.793055+0000","last_became_active":"2024-03-23T03:51:21.788839+0000","last_became_peered":"2024-03-23T03:51:21.788839+0000","last_unstale":"2024-03-23T03:51:21.793055+0000","last_undegraded":"2024-03-23T03:51:21.793055+0000","last_fullsized":"2024-03-23T03:51:21.793055+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:54:15.206699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.765433+0000","last_change":"2024-03-23T03:51:21.765433+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.765433+0000","last_undegraded":"2024-03-23T03:51:21.765433+0000","last_fullsized":"2024-03-23T03:51:21.765433+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:04.438866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.791287+0000","last_change":"2024-03-23T03:51:21.785555+0000","last_active":"2024-03-23T03:51:21.791287+0000","last_peered":"2024-03-23T03:51:21.791287+0000","last_clean":"2024-03-23T03:51:21.791287+0000","last_became_active":"2024-03-23T03:51:21.785232+0000","last_became_peered":"2024-03-23T03:51:21.785232+0000","last_unstale":"2024-03-23T03:51:21.791287+0000","last_undegraded":"2024-03-23T03:51:21.791287+0000","last_fullsized":"2024-03-23T03:51:21.791287+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:27:33.193844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.768476+0000","last_change":"2024-03-23T03:51:21.768476+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.768476+0000","last_undegraded":"2024-03-23T03:51:21.768476+0000","last_fullsized":"2024-03-23T03:51:21.768476+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:27:17.201821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.791576+0000","last_change":"2024-03-23T03:51:21.783666+0000","last_active":"2024-03-23T03:51:21.791576+0000","last_peered":"2024-03-23T03:51:21.791576+0000","last_clean":"2024-03-23T03:51:21.791576+0000","last_became_active":"2024-03-23T03:51:21.783377+0000","last_became_peered":"2024-03-23T03:51:21.783377+0000","last_unstale":"2024-03-23T03:51:21.791576+0000","last_undegraded":"2024-03-23T03:51:21.791576+0000","last_fullsized":"2024-03-23T03:51:21.791576+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:42:08.794820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.761765+0000","last_change":"2024-03-23T03:51:21.761765+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.761765+0000","last_undegraded":"2024-03-23T03:51:21.761765+0000","last_fullsized":"2024-03-23T03:51:21.761765+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:17:09.963708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.795666+0000","last_change":"2024-03-23T03:51:21.792017+0000","last_active":"2024-03-23T03:51:21.795666+0000","last_peered":"2024-03-23T03:51:21.795666+0000","last_clean":"2024-03-23T03:51:21.795666+0000","last_became_active":"2024-03-23T03:51:21.791763+0000","last_became_peered":"2024-03-23T03:51:21.791763+0000","last_unstale":"2024-03-23T03:51:21.795666+0000","last_undegraded":"2024-03-23T03:51:21.795666+0000","last_fullsized":"2024-03-23T03:51:21.795666+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:05:09.454442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.757766+0000","last_change":"2024-03-23T03:51:21.757766+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.757766+0000","last_undegraded":"2024-03-23T03:51:21.757766+0000","last_fullsized":"2024-03-23T03:51:21.757766+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:38:27.787294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.794129+0000","last_change":"2024-03-23T03:51:21.792106+0000","last_active":"2024-03-23T03:51:21.794129+0000","last_peered":"2024-03-23T03:51:21.794129+0000","last_clean":"2024-03-23T03:51:21.794129+0000","last_became_active":"2024-03-23T03:51:21.791295+0000","last_became_peered":"2024-03-23T03:51:21.791295+0000","last_unstale":"2024-03-23T03:51:21.794129+0000","last_undegraded":"2024-03-23T03:51:21.794129+0000","last_fullsized":"2024-03-23T03:51:21.794129+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:00:35.520733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.768896+0000","last_change":"2024-03-23T03:51:21.768896+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.768896+0000","last_undegraded":"2024-03-23T03:51:21.768896+0000","last_fullsized":"2024-03-23T03:51:21.768896+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:37:27.534087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.785068+0000","last_change":"2024-03-23T03:51:21.780198+0000","last_active":"2024-03-23T03:51:21.785068+0000","last_peered":"2024-03-23T03:51:21.785068+0000","last_clean":"2024-03-23T03:51:21.785068+0000","last_became_active":"2024-03-23T03:51:21.779256+0000","last_became_peered":"2024-03-23T03:51:21.779256+0000","last_unstale":"2024-03-23T03:51:21.785068+0000","last_undegraded":"2024-03-23T03:51:21.785068+0000","last_fullsized":"2024-03-23T03:51:21.785068+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:00:33.834214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.781854+0000","last_change":"2024-03-23T03:51:21.781854+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.781854+0000","last_undegraded":"2024-03-23T03:51:21.781854+0000","last_fullsized":"2024-03-23T03:51:21.781854+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:50:44.800098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.791010+0000","last_change":"2024-03-23T03:51:21.783223+0000","last_active":"2024-03-23T03:51:21.791010+0000","last_peered":"2024-03-23T03:51:21.791010+0000","last_clean":"2024-03-23T03:51:21.791010+0000","last_became_active":"2024-03-23T03:51:21.782799+0000","last_became_peered":"2024-03-23T03:51:21.782799+0000","last_unstale":"2024-03-23T03:51:21.791010+0000","last_undegraded":"2024-03-23T03:51:21.791010+0000","last_fullsized":"2024-03-23T03:51:21.791010+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:56:44.890019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.759573+0000","last_change":"2024-03-23T03:51:21.759573+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.759573+0000","last_undegraded":"2024-03-23T03:51:21.759573+0000","last_fullsized":"2024-03-23T03:51:21.759573+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:31:57.940022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.782389+0000","last_change":"2024-03-23T03:51:21.776001+0000","last_active":"2024-03-23T03:51:21.782389+0000","last_peered":"2024-03-23T03:51:21.782389+0000","last_clean":"2024-03-23T03:51:21.782389+0000","last_became_active":"2024-03-23T03:51:21.775726+0000","last_became_peered":"2024-03-23T03:51:21.775726+0000","last_unstale":"2024-03-23T03:51:21.782389+0000","last_undegraded":"2024-03-23T03:51:21.782389+0000","last_fullsized":"2024-03-23T03:51:21.782389+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:07:52.649065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.766180+0000","last_change":"2024-03-23T03:51:21.766180+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.766180+0000","last_undegraded":"2024-03-23T03:51:21.766180+0000","last_fullsized":"2024-03-23T03:51:21.766180+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:55:30.397347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.791658+0000","last_change":"2024-03-23T03:51:21.786526+0000","last_active":"2024-03-23T03:51:21.791658+0000","last_peered":"2024-03-23T03:51:21.791658+0000","last_clean":"2024-03-23T03:51:21.791658+0000","last_became_active":"2024-03-23T03:51:21.783040+0000","last_became_peered":"2024-03-23T03:51:21.783040+0000","last_unstale":"2024-03-23T03:51:21.791658+0000","last_undegraded":"2024-03-23T03:51:21.791658+0000","last_fullsized":"2024-03-23T03:51:21.791658+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:53.003075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.764341+0000","last_change":"2024-03-23T03:51:21.764341+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.764341+0000","last_undegraded":"2024-03-23T03:51:21.764341+0000","last_fullsized":"2024-03-23T03:51:21.764341+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:42:06.329388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.792942+0000","last_change":"2024-03-23T03:51:21.787895+0000","last_active":"2024-03-23T03:51:21.792942+0000","last_peered":"2024-03-23T03:51:21.792942+0000","last_clean":"2024-03-23T03:51:21.792942+0000","last_became_active":"2024-03-23T03:51:21.787627+0000","last_became_peered":"2024-03-23T03:51:21.787627+0000","last_unstale":"2024-03-23T03:51:21.792942+0000","last_undegraded":"2024-03-23T03:51:21.792942+0000","last_fullsized":"2024-03-23T03:51:21.792942+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:31:02.266262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.765914+0000","last_change":"2024-03-23T03:51:21.765914+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.765914+0000","last_undegraded":"2024-03-23T03:51:21.765914+0000","last_fullsized":"2024-03-23T03:51:21.765914+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:34:16.439779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.793407+0000","last_change":"2024-03-23T03:51:21.786687+0000","last_active":"2024-03-23T03:51:21.793407+0000","last_peered":"2024-03-23T03:51:21.793407+0000","last_clean":"2024-03-23T03:51:21.793407+0000","last_became_active":"2024-03-23T03:51:21.786374+0000","last_became_peered":"2024-03-23T03:51:21.786374+0000","last_unstale":"2024-03-23T03:51:21.793407+0000","last_undegraded":"2024-03-23T03:51:21.793407+0000","last_fullsized":"2024-03-23T03:51:21.793407+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:09:59.406767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.766747+0000","last_change":"2024-03-23T03:51:21.766747+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.766747+0000","last_undegraded":"2024-03-23T03:51:21.766747+0000","last_fullsized":"2024-03-23T03:51:21.766747+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:19:44.510925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.792849+0000","last_change":"2024-03-23T03:51:21.789690+0000","last_active":"2024-03-23T03:51:21.792849+0000","last_peered":"2024-03-23T03:51:21.792849+0000","last_clean":"2024-03-23T03:51:21.792849+0000","last_became_active":"2024-03-23T03:51:21.789409+0000","last_became_peered":"2024-03-23T03:51:21.789409+0000","last_unstale":"2024-03-23T03:51:21.792849+0000","last_undegraded":"2024-03-23T03:51:21.792849+0000","last_fullsized":"2024-03-23T03:51:21.792849+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:03:22.934141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.764042+0000","last_change":"2024-03-23T03:51:21.764042+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.764042+0000","last_undegraded":"2024-03-23T03:51:21.764042+0000","last_fullsized":"2024-03-23T03:51:21.764042+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:04:22.944014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.782681+0000","last_change":"2024-03-23T03:51:21.778247+0000","last_active":"2024-03-23T03:51:21.782681+0000","last_peered":"2024-03-23T03:51:21.782681+0000","last_clean":"2024-03-23T03:51:21.782681+0000","last_became_active":"2024-03-23T03:51:21.777903+0000","last_became_peered":"2024-03-23T03:51:21.777903+0000","last_unstale":"2024-03-23T03:51:21.782681+0000","last_undegraded":"2024-03-23T03:51:21.782681+0000","last_fullsized":"2024-03-23T03:51:21.782681+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:30:03.239118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.775918+0000","last_change":"2024-03-23T03:51:21.775918+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.775918+0000","last_undegraded":"2024-03-23T03:51:21.775918+0000","last_fullsized":"2024-03-23T03:51:21.775918+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:06:14.520077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.786533+0000","last_change":"2024-03-23T03:51:21.780251+0000","last_active":"2024-03-23T03:51:21.786533+0000","last_peered":"2024-03-23T03:51:21.786533+0000","last_clean":"2024-03-23T03:51:21.786533+0000","last_became_active":"2024-03-23T03:51:21.779927+0000","last_became_peered":"2024-03-23T03:51:21.779927+0000","last_unstale":"2024-03-23T03:51:21.786533+0000","last_undegraded":"2024-03-23T03:51:21.786533+0000","last_fullsized":"2024-03-23T03:51:21.786533+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:28:51.270476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.768567+0000","last_change":"2024-03-23T03:51:21.768567+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.768567+0000","last_undegraded":"2024-03-23T03:51:21.768567+0000","last_fullsized":"2024-03-23T03:51:21.768567+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:53.513182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.793062+0000","last_change":"2024-03-23T03:51:21.790264+0000","last_active":"2024-03-23T03:51:21.793062+0000","last_peered":"2024-03-23T03:51:21.793062+0000","last_clean":"2024-03-23T03:51:21.793062+0000","last_became_active":"2024-03-23T03:51:21.789990+0000","last_became_peered":"2024-03-23T03:51:21.789990+0000","last_unstale":"2024-03-23T03:51:21.793062+0000","last_undegraded":"2024-03-23T03:51:21.793062+0000","last_fullsized":"2024-03-23T03:51:21.793062+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:17:42.968056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.773680+0000","last_change":"2024-03-23T03:51:21.773680+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.773680+0000","last_undegraded":"2024-03-23T03:51:21.773680+0000","last_fullsized":"2024-03-23T03:51:21.773680+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:52:35.913771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.789713+0000","last_change":"2024-03-23T03:51:21.783083+0000","last_active":"2024-03-23T03:51:21.789713+0000","last_peered":"2024-03-23T03:51:21.789713+0000","last_clean":"2024-03-23T03:51:21.789713+0000","last_became_active":"2024-03-23T03:51:21.782771+0000","last_became_peered":"2024-03-23T03:51:21.782771+0000","last_unstale":"2024-03-23T03:51:21.789713+0000","last_undegraded":"2024-03-23T03:51:21.789713+0000","last_fullsized":"2024-03-23T03:51:21.789713+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:11:22.831901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.766375+0000","last_change":"2024-03-23T03:51:21.766375+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.766375+0000","last_undegraded":"2024-03-23T03:51:21.766375+0000","last_fullsized":"2024-03-23T03:51:21.766375+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:35:48.145231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.795163+0000","last_change":"2024-03-23T03:51:21.793220+0000","last_active":"2024-03-23T03:51:21.795163+0000","last_peered":"2024-03-23T03:51:21.795163+0000","last_clean":"2024-03-23T03:51:21.795163+0000","last_became_active":"2024-03-23T03:51:21.792937+0000","last_became_peered":"2024-03-23T03:51:21.792937+0000","last_unstale":"2024-03-23T03:51:21.795163+0000","last_undegraded":"2024-03-23T03:51:21.795163+0000","last_fullsized":"2024-03-23T03:51:21.795163+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:52.502591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.773976+0000","last_change":"2024-03-23T03:51:21.773976+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.773976+0000","last_undegraded":"2024-03-23T03:51:21.773976+0000","last_fullsized":"2024-03-23T03:51:21.773976+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:31:10.713669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.794067+0000","last_change":"2024-03-23T03:51:21.792048+0000","last_active":"2024-03-23T03:51:21.794067+0000","last_peered":"2024-03-23T03:51:21.794067+0000","last_clean":"2024-03-23T03:51:21.794067+0000","last_became_active":"2024-03-23T03:51:21.791713+0000","last_became_peered":"2024-03-23T03:51:21.791713+0000","last_unstale":"2024-03-23T03:51:21.794067+0000","last_undegraded":"2024-03-23T03:51:21.794067+0000","last_fullsized":"2024-03-23T03:51:21.794067+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:59:01.379564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.767517+0000","last_change":"2024-03-23T03:51:21.767517+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.767517+0000","last_undegraded":"2024-03-23T03:51:21.767517+0000","last_fullsized":"2024-03-23T03:51:21.767517+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:59:59.276905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.791584+0000","last_change":"2024-03-23T03:51:21.789004+0000","last_active":"2024-03-23T03:51:21.791584+0000","last_peered":"2024-03-23T03:51:21.791584+0000","last_clean":"2024-03-23T03:51:21.791584+0000","last_became_active":"2024-03-23T03:51:21.788647+0000","last_became_peered":"2024-03-23T03:51:21.788647+0000","last_unstale":"2024-03-23T03:51:21.791584+0000","last_undegraded":"2024-03-23T03:51:21.791584+0000","last_fullsized":"2024-03-23T03:51:21.791584+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:59:01.379564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.767902+0000","last_change":"2024-03-23T03:51:21.767902+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.767902+0000","last_undegraded":"2024-03-23T03:51:21.767902+0000","last_fullsized":"2024-03-23T03:51:21.767902+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:15:49.588458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.788658+0000","last_change":"2024-03-23T03:51:21.782473+0000","last_active":"2024-03-23T03:51:21.788658+0000","last_peered":"2024-03-23T03:51:21.788658+0000","last_clean":"2024-03-23T03:51:21.788658+0000","last_became_active":"2024-03-23T03:51:21.781740+0000","last_became_peered":"2024-03-23T03:51:21.781740+0000","last_unstale":"2024-03-23T03:51:21.788658+0000","last_undegraded":"2024-03-23T03:51:21.788658+0000","last_fullsized":"2024-03-23T03:51:21.788658+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:30:22.629109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.769342+0000","last_change":"2024-03-23T03:51:21.769342+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.769342+0000","last_undegraded":"2024-03-23T03:51:21.769342+0000","last_fullsized":"2024-03-23T03:51:21.769342+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:18:12.413501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.785499+0000","last_change":"2024-03-23T03:51:21.782262+0000","last_active":"2024-03-23T03:51:21.785499+0000","last_peered":"2024-03-23T03:51:21.785499+0000","last_clean":"2024-03-23T03:51:21.785499+0000","last_became_active":"2024-03-23T03:51:21.781940+0000","last_became_peered":"2024-03-23T03:51:21.781940+0000","last_unstale":"2024-03-23T03:51:21.785499+0000","last_undegraded":"2024-03-23T03:51:21.785499+0000","last_fullsized":"2024-03-23T03:51:21.785499+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:49.654183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.759451+0000","last_change":"2024-03-23T03:51:21.759451+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.759451+0000","last_undegraded":"2024-03-23T03:51:21.759451+0000","last_fullsized":"2024-03-23T03:51:21.759451+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:06:50.564546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.795104+0000","last_change":"2024-03-23T03:51:21.792924+0000","last_active":"2024-03-23T03:51:21.795104+0000","last_peered":"2024-03-23T03:51:21.795104+0000","last_clean":"2024-03-23T03:51:21.795104+0000","last_became_active":"2024-03-23T03:51:21.790283+0000","last_became_peered":"2024-03-23T03:51:21.790283+0000","last_unstale":"2024-03-23T03:51:21.795104+0000","last_undegraded":"2024-03-23T03:51:21.795104+0000","last_fullsized":"2024-03-23T03:51:21.795104+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:19:57.445124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.769014+0000","last_change":"2024-03-23T03:51:21.769014+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.769014+0000","last_undegraded":"2024-03-23T03:51:21.769014+0000","last_fullsized":"2024-03-23T03:51:21.769014+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:35.027397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.790727+0000","last_change":"2024-03-23T03:51:21.784479+0000","last_active":"2024-03-23T03:51:21.790727+0000","last_peered":"2024-03-23T03:51:21.790727+0000","last_clean":"2024-03-23T03:51:21.790727+0000","last_became_active":"2024-03-23T03:51:21.784166+0000","last_became_peered":"2024-03-23T03:51:21.784166+0000","last_unstale":"2024-03-23T03:51:21.790727+0000","last_undegraded":"2024-03-23T03:51:21.790727+0000","last_fullsized":"2024-03-23T03:51:21.790727+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:35:04.474544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.766611+0000","last_change":"2024-03-23T03:51:21.766611+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.766611+0000","last_undegraded":"2024-03-23T03:51:21.766611+0000","last_fullsized":"2024-03-23T03:51:21.766611+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:08:03.409470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.782622+0000","last_change":"2024-03-23T03:51:21.776941+0000","last_active":"2024-03-23T03:51:21.782622+0000","last_peered":"2024-03-23T03:51:21.782622+0000","last_clean":"2024-03-23T03:51:21.782622+0000","last_became_active":"2024-03-23T03:51:21.776643+0000","last_became_peered":"2024-03-23T03:51:21.776643+0000","last_unstale":"2024-03-23T03:51:21.782622+0000","last_undegraded":"2024-03-23T03:51:21.782622+0000","last_fullsized":"2024-03-23T03:51:21.782622+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:17:08.953118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.771267+0000","last_change":"2024-03-23T03:51:21.771267+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.771267+0000","last_undegraded":"2024-03-23T03:51:21.771267+0000","last_fullsized":"2024-03-23T03:51:21.771267+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:46:03.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.790974+0000","last_change":"2024-03-23T03:51:21.785341+0000","last_active":"2024-03-23T03:51:21.790974+0000","last_peered":"2024-03-23T03:51:21.790974+0000","last_clean":"2024-03-23T03:51:21.790974+0000","last_became_active":"2024-03-23T03:51:21.784654+0000","last_became_peered":"2024-03-23T03:51:21.784654+0000","last_unstale":"2024-03-23T03:51:21.790974+0000","last_undegraded":"2024-03-23T03:51:21.790974+0000","last_fullsized":"2024-03-23T03:51:21.790974+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:36.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.758321+0000","last_change":"2024-03-23T03:51:21.758321+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.758321+0000","last_undegraded":"2024-03-23T03:51:21.758321+0000","last_fullsized":"2024-03-23T03:51:21.758321+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:27:15.422967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.792705+0000","last_change":"2024-03-23T03:51:21.788631+0000","last_active":"2024-03-23T03:51:21.792705+0000","last_peered":"2024-03-23T03:51:21.792705+0000","last_clean":"2024-03-23T03:51:21.792705+0000","last_became_active":"2024-03-23T03:51:21.788145+0000","last_became_peered":"2024-03-23T03:51:21.788145+0000","last_unstale":"2024-03-23T03:51:21.792705+0000","last_undegraded":"2024-03-23T03:51:21.792705+0000","last_fullsized":"2024-03-23T03:51:21.792705+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:45.506837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.764213+0000","last_change":"2024-03-23T03:51:21.764213+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.764213+0000","last_undegraded":"2024-03-23T03:51:21.764213+0000","last_fullsized":"2024-03-23T03:51:21.764213+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:30:04.249708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.791253+0000","last_change":"2024-03-23T03:51:21.788546+0000","last_active":"2024-03-23T03:51:21.791253+0000","last_peered":"2024-03-23T03:51:21.791253+0000","last_clean":"2024-03-23T03:51:21.791253+0000","last_became_active":"2024-03-23T03:51:21.788282+0000","last_became_peered":"2024-03-23T03:51:21.788282+0000","last_unstale":"2024-03-23T03:51:21.791253+0000","last_undegraded":"2024-03-23T03:51:21.791253+0000","last_fullsized":"2024-03-23T03:51:21.791253+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:04:20.308736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.765746+0000","last_change":"2024-03-23T03:51:21.765746+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.765746+0000","last_undegraded":"2024-03-23T03:51:21.765746+0000","last_fullsized":"2024-03-23T03:51:21.765746+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:05.542126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.795287+0000","last_change":"2024-03-23T03:51:21.793737+0000","last_active":"2024-03-23T03:51:21.795287+0000","last_peered":"2024-03-23T03:51:21.795287+0000","last_clean":"2024-03-23T03:51:21.795287+0000","last_became_active":"2024-03-23T03:51:21.791575+0000","last_became_peered":"2024-03-23T03:51:21.791575+0000","last_unstale":"2024-03-23T03:51:21.795287+0000","last_undegraded":"2024-03-23T03:51:21.795287+0000","last_fullsized":"2024-03-23T03:51:21.795287+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:54:20.064900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.767710+0000","last_change":"2024-03-23T03:51:21.767710+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.767710+0000","last_undegraded":"2024-03-23T03:51:21.767710+0000","last_fullsized":"2024-03-23T03:51:21.767710+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:45:36.655749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.791188+0000","last_change":"2024-03-23T03:51:21.787337+0000","last_active":"2024-03-23T03:51:21.791188+0000","last_peered":"2024-03-23T03:51:21.791188+0000","last_clean":"2024-03-23T03:51:21.791188+0000","last_became_active":"2024-03-23T03:51:21.787076+0000","last_became_peered":"2024-03-23T03:51:21.787076+0000","last_unstale":"2024-03-23T03:51:21.791188+0000","last_undegraded":"2024-03-23T03:51:21.791188+0000","last_fullsized":"2024-03-23T03:51:21.791188+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:09:48.708095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.776317+0000","last_change":"2024-03-23T03:51:21.776317+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.776317+0000","last_undegraded":"2024-03-23T03:51:21.776317+0000","last_fullsized":"2024-03-23T03:51:21.776317+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:05:10.465032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.794483+0000","last_change":"2024-03-23T03:51:21.789939+0000","last_active":"2024-03-23T03:51:21.794483+0000","last_peered":"2024-03-23T03:51:21.794483+0000","last_clean":"2024-03-23T03:51:21.794483+0000","last_became_active":"2024-03-23T03:51:21.788412+0000","last_became_peered":"2024-03-23T03:51:21.788412+0000","last_unstale":"2024-03-23T03:51:21.794483+0000","last_undegraded":"2024-03-23T03:51:21.794483+0000","last_fullsized":"2024-03-23T03:51:21.794483+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:35:47.134641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.763474+0000","last_change":"2024-03-23T03:51:21.763474+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.763474+0000","last_undegraded":"2024-03-23T03:51:21.763474+0000","last_fullsized":"2024-03-23T03:51:21.763474+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:05:03.516080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.795576+0000","last_change":"2024-03-23T03:51:21.791827+0000","last_active":"2024-03-23T03:51:21.795576+0000","last_peered":"2024-03-23T03:51:21.795576+0000","last_clean":"2024-03-23T03:51:21.795576+0000","last_became_active":"2024-03-23T03:51:21.791445+0000","last_became_peered":"2024-03-23T03:51:21.791445+0000","last_unstale":"2024-03-23T03:51:21.795576+0000","last_undegraded":"2024-03-23T03:51:21.795576+0000","last_fullsized":"2024-03-23T03:51:21.795576+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:58.150428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.762191+0000","last_change":"2024-03-23T03:51:21.762191+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.762191+0000","last_undegraded":"2024-03-23T03:51:21.762191+0000","last_fullsized":"2024-03-23T03:51:21.762191+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:04:21.319327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.782798+0000","last_change":"2024-03-23T03:51:21.779521+0000","last_active":"2024-03-23T03:51:21.782798+0000","last_peered":"2024-03-23T03:51:21.782798+0000","last_clean":"2024-03-23T03:51:21.782798+0000","last_became_active":"2024-03-23T03:51:21.779111+0000","last_became_peered":"2024-03-23T03:51:21.779111+0000","last_unstale":"2024-03-23T03:51:21.782798+0000","last_undegraded":"2024-03-23T03:51:21.782798+0000","last_fullsized":"2024-03-23T03:51:21.782798+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:29:39.956286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":228,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-23T03:51:20.741124+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:51:20.741124+0000","last_peered":"2024-03-23T03:51:20.741124+0000","last_clean":"2024-03-23T03:51:20.741124+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:51:20.741124+0000","last_undegraded":"2024-03-23T03:51:20.741124+0000","last_fullsized":"2024-03-23T03:51:20.741124+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542203,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1855488,"data_stored":1159641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.087,"15min":1.0720000000000001},"min":{"1min":0.65700000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5209999999999999,"5min":1.635,"15min":1.635},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.042,"15min":1.0469999999999999},"min":{"1min":0.749,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.393,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.49}]},{"osd":1,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.081,"15min":1.0529999999999999},"min":{"1min":0.71499999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6120000000000001,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.077},{"interface":"front","average":{"1min":1.03,"5min":1.1719999999999999,"15min":1.2809999999999999},"min":{"1min":0.79000000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.321,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":0.98399999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.125,"15min":1.151},"min":{"1min":0.81299999999999994,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.583,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.028},{"interface":"front","average":{"1min":1.167,"5min":1.2869999999999999,"15min":1.401},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.506,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.5369999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.298,"15min":1.4179999999999999},"min":{"1min":0.71999999999999997,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.4219999999999999,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.125},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.325,"15min":1.353},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.677,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.421}]},{"osd":4,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.254,"15min":1.2629999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.8220000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.605},{"interface":"front","average":{"1min":1.333,"5min":1.347,"15min":1.3720000000000001},"min":{"1min":0.67500000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.778,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.4610000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.389,"15min":1.4179999999999999},"min":{"1min":0.81299999999999994,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.7310000000000001,"5min":2.645,"15min":2.645},"last":1.206},{"interface":"front","average":{"1min":1.361,"5min":1.417,"15min":1.446},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.7649999999999999,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.661}]},{"osd":6,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.504,"15min":1.552},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.9279999999999999,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.708},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.474,"15min":1.5149999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9399999999999999,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.347}]}]},{"osd":6,"up_from":14,"seq":60129542201,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32348,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339492,"statfs":{"total":96636764160,"available":96603639808,"internally_reserved":0,"allocated":1261568,"data_stored":569273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.0429999999999999,"15min":1.044},"min":{"1min":0.69699999999999995,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.05,"15min":1.04},"min":{"1min":0.66100000000000003,"5min":0.62,"15min":0.62},"max":{"1min":1.5409999999999999,"5min":1.946,"15min":1.946},"last":1.385}]},{"osd":1,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.087,"15min":1.075},"min":{"1min":0.745,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.196,"5min":1.1339999999999999,"15min":1.1579999999999999},"min":{"1min":0.80500000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.5369999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.88800000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.083,"15min":1.071},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.4350000000000001,"5min":2.044,"15min":2.044},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.133,"5min":1.1759999999999999,"15min":1.212},"min":{"1min":0.85499999999999998,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.6599999999999999,"5min":1.681,"15min":1.681},"last":1.028}]},{"osd":3,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.2370000000000001,"15min":1.276},"min":{"1min":0.88100000000000001,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.786,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.262},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.2330000000000001,"15min":1.2569999999999999},"min":{"1min":0.90100000000000002,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.446,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.157}]},{"osd":4,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3779999999999999,"15min":1.385},"min":{"1min":1.1930000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.6679999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.431,"5min":1.371,"15min":1.3859999999999999},"min":{"1min":1.234,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.639,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.5429999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.403,"15min":1.4319999999999999},"min":{"1min":1.0669999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.794,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.474,"5min":1.508,"15min":1.542},"min":{"1min":1.091,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8200000000000001,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.2350000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.4059999999999999,"15min":1.4119999999999999},"min":{"1min":0.95999999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.0760000000000001,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.482,"15min":1.5309999999999999},"min":{"1min":1.125,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.8660000000000001,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.1830000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1261568,"data_stored":569273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1439999999999999,"15min":1.2030000000000001},"min":{"1min":0.72999999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4830000000000001,"5min":1.806,"15min":1.806},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.113,"5min":1.1399999999999999,"15min":1.1759999999999999},"min":{"1min":0.76500000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.855,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.872}]},{"osd":1,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0940000000000001,"15min":1.1000000000000001},"min":{"1min":0.80100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.3520000000000001,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.165,"15min":1.1930000000000001},"min":{"1min":0.84799999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.663,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.94399999999999995}]},{"osd":2,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.274,"15min":1.3460000000000001},"min":{"1min":0.70499999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8939999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.2989999999999999,"15min":1.3660000000000001},"min":{"1min":0.875,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.579,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.22}]},{"osd":3,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.2,"15min":1.244},"min":{"1min":0.85399999999999998,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.7050000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.375,"15min":1.446},"min":{"1min":0.997,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.964,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.1739999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3049999999999999,"15min":1.399},"min":{"1min":0.65700000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6100000000000001,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.528},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3859999999999999,"15min":1.4359999999999999},"min":{"1min":0.83699999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8069999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.415}]},{"osd":6,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.4159999999999999,"15min":1.5},"min":{"1min":0.83399999999999996,"5min":0.63,"15min":0.63},"max":{"1min":1.5840000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.48},{"interface":"front","average":{"1min":1.423,"5min":1.5409999999999999,"15min":1.639},"min":{"1min":1.1950000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.6359999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.5880000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4910000000000001,"15min":1.591},"min":{"1min":1.095,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.9259999999999999,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.623},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5840000000000001,"15min":1.6839999999999999},"min":{"1min":1.272,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.056,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.833}]}]},{"osd":4,"up_from":14,"seq":60129542202,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32348,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339492,"statfs":{"total":96636764160,"available":96603639808,"internally_reserved":0,"allocated":1261568,"data_stored":569273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.004,"15min":0.98099999999999998},"min":{"1min":0.624,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.47,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.995,"15min":1.0049999999999999},"min":{"1min":0.56100000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3400000000000001,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.82699999999999996}]},{"osd":1,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0680000000000001,"15min":1.05},"min":{"1min":0.65100000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.5,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0880000000000001,"15min":1.0640000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.5209999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.93600000000000005}]},{"osd":2,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.2470000000000001,"15min":1.3089999999999999},"min":{"1min":0.70399999999999996,"5min":0.625,"15min":0.625},"max":{"1min":1.452,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.2789999999999999,"15min":1.2969999999999999},"min":{"1min":0.59899999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6859999999999999,"5min":2.181,"15min":2.181},"last":1.1459999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.1160000000000001,"15min":1.1240000000000001},"min":{"1min":0.61899999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.4419999999999999,"5min":1.821,"15min":1.821},"last":1.006},{"interface":"front","average":{"1min":1.141,"5min":1.2310000000000001,"15min":1.268},"min":{"1min":0.73799999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.599,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.0529999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.2370000000000001,"15min":1.26},"min":{"1min":0.63800000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.49,"5min":1.929,"15min":1.929},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.2709999999999999,"15min":1.274},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.45,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.1120000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.371,"15min":1.409},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6719999999999999,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.4139999999999999,"15min":1.3999999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.8100000000000001,"5min":2.016,"15min":2.016},"last":1.319}]},{"osd":7,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.361,"15min":1.363},"min":{"1min":0.68200000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.5569999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.452},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.456,"15min":1.4870000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.716,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.363}]}]},{"osd":3,"up_from":14,"seq":60129542202,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32348,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339492,"statfs":{"total":96636764160,"available":96603639808,"internally_reserved":0,"allocated":1261568,"data_stored":569273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.81999999999999995,"15min":0.85499999999999998},"min":{"1min":0.35499999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.4419999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.92200000000000004,"15min":0.92400000000000004},"min":{"1min":0.502,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.538,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.56000000000000005}]},{"osd":1,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.91500000000000004,"15min":0.96799999999999997},"min":{"1min":0.58099999999999996,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.9990000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":1.034,"5min":1.0229999999999999,"15min":1.069},"min":{"1min":0.52000000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.651,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.002,"15min":1.0509999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.5710000000000002,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1240000000000001,"15min":1.1499999999999999},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.7649999999999999,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":0.68600000000000005}]},{"osd":4,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.2410000000000001,"15min":1.248},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":4.0380000000000003,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.351,"15min":1.429},"min":{"1min":0.77000000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.3450000000000002,"5min":4.819,"15min":4.819},"last":0.90700000000000003}]},{"osd":5,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.6040000000000001,"15min":1.4470000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.2679999999999998,"5min":30.228000000000002,"15min":30.228000000000002},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.337,"15min":1.357},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":4.3630000000000004,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":0.96399999999999997}]},{"osd":6,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4119999999999999,"15min":1.508},"min":{"1min":0.83999999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.6589999999999998,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4199999999999999,"15min":1.4039999999999999},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.4039999999999999,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":0.93899999999999995}]},{"osd":7,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.389,"15min":1.4199999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.0880000000000001,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.542,"15min":1.5900000000000001},"min":{"1min":0.80500000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.4780000000000002,"5min":10.663,"15min":10.663},"last":0.97799999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542202,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32348,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339492,"statfs":{"total":96636764160,"available":96603639808,"internally_reserved":0,"allocated":1261568,"data_stored":569273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83799999999999997,"15min":0.877},"min":{"1min":0.52100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":1.4259999999999999,"5min":1.456,"15min":1.456},"last":1.1899999999999999},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.86399999999999999,"15min":0.93600000000000005},"min":{"1min":0.39500000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.4690000000000001,"5min":2.226,"15min":2.226},"last":0.751}]},{"osd":1,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.93899999999999995,"15min":0.98499999999999999},"min":{"1min":0.46800000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.377,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.85899999999999999,"15min":0.91000000000000003},"min":{"1min":0.57599999999999996,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.2210000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.159}]},{"osd":3,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.96099999999999997,"15min":1.018},"min":{"1min":0.47499999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.403,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.032,"5min":1.042,"15min":1.0800000000000001},"min":{"1min":0.61099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.92300000000000004}]},{"osd":4,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.194,"15min":1.2310000000000001},"min":{"1min":0.82399999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.7270000000000001,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.157,"15min":1.2070000000000001},"min":{"1min":0.83799999999999997,"5min":0.627,"15min":0.627},"max":{"1min":1.4950000000000001,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.421}]},{"osd":5,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.3100000000000001,"15min":1.367},"min":{"1min":0.76900000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.6259999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.2749999999999999,"15min":1.339},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.819,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.077}]},{"osd":6,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.232,"15min":1.3280000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6479999999999999,"5min":2.456,"15min":2.456},"last":1.256},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3999999999999999,"15min":1.476},"min":{"1min":0.85299999999999998,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.097,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.282}]},{"osd":7,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3069999999999999,"15min":1.369},"min":{"1min":0.79800000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.302,"15min":1.395},"min":{"1min":0.78700000000000003,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5860000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.335}]}]},{"osd":1,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1261568,"data_stored":569273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79100000000000004,"15min":0.86099999999999999},"min":{"1min":0.41599999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.1240000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81799999999999995,"15min":0.82999999999999996},"min":{"1min":0.435,"5min":0.374,"15min":0.374},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.59299999999999997}]},{"osd":2,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.92200000000000004,"15min":0.96399999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.4430000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.92400000000000004,"15min":0.97799999999999998},"min":{"1min":0.496,"5min":0.443,"15min":0.443},"max":{"1min":1.2210000000000001,"5min":1.625,"15min":1.625},"last":0.66500000000000004}]},{"osd":3,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.103,"15min":1.113},"min":{"1min":0.68400000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.6559999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.875,"5min":0.96199999999999997,"15min":1.008},"min":{"1min":0.59499999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.6180000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.89100000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.179,"15min":1.2749999999999999},"min":{"1min":0.67700000000000005,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.5069999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.748},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.246,"15min":1.3049999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6120000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.0569999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.248,"15min":1.3200000000000001},"min":{"1min":0.79500000000000004,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.95,"5min":3.593,"15min":3.593},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.264,"5min":1.327,"15min":1.4259999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.671,"5min":2.359,"15min":2.359},"last":0.69899999999999995}]},{"osd":6,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.3089999999999999,"15min":1.3919999999999999},"min":{"1min":0.84199999999999997,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6200000000000001,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.206},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.288,"15min":1.3540000000000001},"min":{"1min":0.89100000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.0209999999999999,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.0940000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.359,"15min":1.4330000000000001},"min":{"1min":0.99299999999999999,"5min":0.627,"15min":0.627},"max":{"1min":2.0840000000000001,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.242},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.363,"15min":1.486},"min":{"1min":0.76600000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.673,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":0.84399999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542201,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1855488,"data_stored":1159641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.86399999999999999,"15min":0.93500000000000005},"min":{"1min":0.48399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.478,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.94799999999999995,"15min":0.98199999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.456,"5min":1.506,"15min":1.506},"last":0.61899999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.92700000000000005,"15min":0.99099999999999999},"min":{"1min":0.52900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.4399999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.0349999999999999,"15min":1.095},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.601,"5min":1.901,"15min":1.901},"last":0.55700000000000005}]},{"osd":3,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.93300000000000005,"15min":0.996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.474,"5min":1.972,"15min":1.972},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0669999999999999,"15min":1.147},"min":{"1min":0.60299999999999998,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.597,"5min":2.298,"15min":2.298},"last":0.70699999999999996}]},{"osd":4,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.288,"15min":1.3360000000000001},"min":{"1min":0.72799999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.7090000000000001,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.2230000000000001,"15min":1.27},"min":{"1min":0.80600000000000005,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.8380000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.95499999999999996}]},{"osd":5,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3100000000000001,"15min":1.3700000000000001},"min":{"1min":0.89500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.948,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.377,"15min":1.4379999999999999},"min":{"1min":0.83499999999999996,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.8700000000000001,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.036}]},{"osd":6,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.4099999999999999,"15min":1.5209999999999999},"min":{"1min":0.80000000000000004,"5min":0.747,"15min":0.747},"max":{"1min":1.978,"5min":3.032,"15min":3.032},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.411,"15min":1.4870000000000001},"min":{"1min":0.84699999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.1349999999999998,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":0.98099999999999998}]},{"osd":7,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4390000000000001,"15min":1.506},"min":{"1min":0.90500000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1850000000000001,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.099},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3620000000000001,"15min":1.419},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.873,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.077}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:51:27.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:51:27.268 INFO:tasks.ceph.mds.a.smithi102.stderr:2024-03-23T03:51:27.267+0000 7f2694a00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T03:51:27.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.392+0000 7f7fe3a00640 1 Processor -- start 2024-03-23T03:51:27.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.392+0000 7f7fe3a00640 1 -- start start 2024-03-23T03:51:27.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.393+0000 7f7fe3a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7fdc15b560 0x7f7fdc15b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:27.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.393+0000 7f7fe3a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7fdc0b39b0 0x7f7fdc0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:27.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.393+0000 7f7fe3a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fdc06c6f0 con 0x7f7fdc15e500 2024-03-23T03:51:27.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.394+0000 7f7fe3a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fdc0b4370 con 0x7f7fdc0b39b0 2024-03-23T03:51:27.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.394+0000 7f7fe3a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc06c9d0 con 0x7f7fdc15b560 2024-03-23T03:51:27.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.394+0000 7f7fdbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7fdc0b39b0 0x7f7fdc0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:27.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.394+0000 7f7fdbe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f7fdc0b39b0 0x7f7fdc0b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42596/0 (socket says 172.21.15.102:42596) 2024-03-23T03:51:27.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.394+0000 7f7fdbe00640 1 -- 172.21.15.102:0/4290970559 learned_addr learned my addr 172.21.15.102:0/4290970559 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:27.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.394+0000 7f7fe1000640 1 --2- 172.21.15.102:0/4290970559 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7fdc15b560 0x7f7fdc15b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:27.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.394+0000 7f7fdbe00640 1 -- 172.21.15.102:0/4290970559 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7fdc15b560 msgr2=0x7f7fdc15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:27.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.394+0000 7f7fdbe00640 1 --2- 172.21.15.102:0/4290970559 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7fdc15b560 0x7f7fdc15b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.394+0000 7f7fdbe00640 1 -- 172.21.15.102:0/4290970559 >> v1:172.21.15.173:6789/0 conn(0x7f7fdc15e500 legacy=0x7f7fdc15e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:27.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.394+0000 7f7fdbe00640 1 -- 172.21.15.102:0/4290970559 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc06ccb0 con 0x7f7fdc0b39b0 2024-03-23T03:51:27.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.394+0000 7f7fe1000640 1 --2- 172.21.15.102:0/4290970559 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7fdc15b560 0x7f7fdc15b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:27.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.396+0000 7f7fdbe00640 1 --2- 172.21.15.102:0/4290970559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fdc0b39b0 0x7f7fdc0b3e30 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009870 tx=0x7f7fc802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87371cf9c9c462dc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:27.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.396+0000 7f7fdb400640 1 -- 172.21.15.102:0/4290970559 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8031640 con 0x7f7fdc0b39b0 2024-03-23T03:51:27.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.396+0000 7f7fdb400640 1 -- 172.21.15.102:0/4290970559 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc80317a0 con 0x7f7fdc0b39b0 2024-03-23T03:51:27.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.396+0000 7f7fdb400640 1 -- 172.21.15.102:0/4290970559 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc8031b30 con 0x7f7fdc0b39b0 2024-03-23T03:51:27.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.396+0000 7f7fe3a00640 1 -- 172.21.15.102:0/4290970559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fdc0b39b0 msgr2=0x7f7fdc0b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:27.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.396+0000 7f7fe3a00640 1 --2- 172.21.15.102:0/4290970559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fdc0b39b0 0x7f7fdc0b3e30 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8009870 tx=0x7f7fc802b8f0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.397+0000 7f7fe3a00640 1 -- 172.21.15.102:0/4290970559 shutdown_connections 2024-03-23T03:51:27.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.397+0000 7f7fe3a00640 1 --2- 172.21.15.102:0/4290970559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fdc0b39b0 0x7f7fdc0b3e30 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.397+0000 7f7fe3a00640 1 --2- 172.21.15.102:0/4290970559 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7fdc15b560 0x7f7fdc15b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.397+0000 7f7fe3a00640 1 -- 172.21.15.102:0/4290970559 >> 172.21.15.102:0/4290970559 conn(0x7f7fdc06a3d0 msgr2=0x7f7fdc0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:27.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.397+0000 7f7fe3a00640 1 -- 172.21.15.102:0/4290970559 shutdown_connections 2024-03-23T03:51:27.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.397+0000 7f7fe3a00640 1 -- 172.21.15.102:0/4290970559 wait complete. 2024-03-23T03:51:27.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.398+0000 7f7fe3a00640 1 Processor -- start 2024-03-23T03:51:27.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.399+0000 7f7fe3a00640 1 -- start start 2024-03-23T03:51:27.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.399+0000 7f7fe3a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fdc0b39b0 0x7f7fdc0a8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:27.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.399+0000 7f7fe3a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7fdc15b560 0x7f7fdc141a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:27.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.399+0000 7f7fe3a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fdc15e500 0x7f7fdc140880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:27.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.399+0000 7f7fe3a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc0a8db0 con 0x7f7fdc0b39b0 2024-03-23T03:51:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.399+0000 7f7fe3a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc141fc0 con 0x7f7fdc15e500 2024-03-23T03:51:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.399+0000 7f7fe3a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc151fa0 con 0x7f7fdc15b560 2024-03-23T03:51:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.400+0000 7f7fe1000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fdc0b39b0 0x7f7fdc0a8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.400+0000 7f7fe1000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fdc0b39b0 0x7f7fdc0a8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42608/0 (socket says 172.21.15.102:42608) 2024-03-23T03:51:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.400+0000 7f7fe1000640 1 -- 172.21.15.102:0/3115866257 learned_addr learned my addr 172.21.15.102:0/3115866257 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.400+0000 7f7fe1000640 1 -- 172.21.15.102:0/3115866257 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7fdc15b560 msgr2=0x7f7fdc141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.400+0000 7f7fe1000640 1 --2- 172.21.15.102:0/3115866257 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7fdc15b560 0x7f7fdc141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.400+0000 7f7fe1000640 1 -- 172.21.15.102:0/3115866257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fdc15e500 msgr2=0x7f7fdc140880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.400+0000 7f7fe1000640 1 --2- 172.21.15.102:0/3115866257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fdc15e500 0x7f7fdc140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.400+0000 7f7fe1000640 1 -- 172.21.15.102:0/3115866257 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc80094d0 con 0x7f7fdc0b39b0 2024-03-23T03:51:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.400+0000 7f7fe1000640 1 --2- 172.21.15.102:0/3115866257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fdc0b39b0 0x7f7fdc0a8870 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000c7f0 tx=0x7f7fd000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.401+0000 7f7fd8c00640 1 -- 172.21.15.102:0/3115866257 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd0007a30 con 0x7f7fdc0b39b0 2024-03-23T03:51:27.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.401+0000 7f7fd8c00640 1 -- 172.21.15.102:0/3115866257 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd0007b90 con 0x7f7fdc0b39b0 2024-03-23T03:51:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.401+0000 7f7fd8c00640 1 -- 172.21.15.102:0/3115866257 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd000ce40 con 0x7f7fdc0b39b0 2024-03-23T03:51:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.401+0000 7f7fe3a00640 1 -- 172.21.15.102:0/3115866257 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fdc1421f0 con 0x7f7fdc0b39b0 2024-03-23T03:51:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.401+0000 7f7fe3a00640 1 -- 172.21.15.102:0/3115866257 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fdc1524d0 con 0x7f7fdc0b39b0 2024-03-23T03:51:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.402+0000 7f7fe3a00640 1 -- 172.21.15.102:0/3115866257 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fdc06ac20 con 0x7f7fdc0b39b0 2024-03-23T03:51:27.403 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.403+0000 7f7fd8c00640 1 -- 172.21.15.102:0/3115866257 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7fd00113f0 con 0x7f7fdc0b39b0 2024-03-23T03:51:27.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.404+0000 7f7fd8c00640 1 --2- 172.21.15.102:0/3115866257 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7fbc06ac00 0x7f7fbc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:27.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.404+0000 7f7fd8c00640 1 -- 172.21.15.102:0/3115866257 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7fd0089660 con 0x7f7fdc0b39b0 2024-03-23T03:51:27.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.404+0000 7f7fdbe00640 1 --2- 172.21.15.102:0/3115866257 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7fbc06ac00 0x7f7fbc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:27.405 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.405+0000 7f7fdbe00640 1 --2- 172.21.15.102:0/3115866257 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7fbc06ac00 0x7f7fbc06d0c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8036000 tx=0x7f7fc8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:27.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.408+0000 7f7fd8c00640 1 -- 172.21.15.102:0/3115866257 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7fd00546f0 con 0x7f7fdc0b39b0 2024-03-23T03:51:27.598 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.597+0000 7f7fe3a00640 1 -- 172.21.15.102:0/3115866257 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7fdc06dd80 con 0x7f7fbc06ac00 2024-03-23T03:51:27.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.620+0000 7f7fd8c00640 1 -- 172.21.15.102:0/3115866257 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+284760 (secure 0 0 0) 0x7f7fc803a420 con 0x7f7fbc06ac00 2024-03-23T03:51:27.622 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:27.625 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:51:27.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.625+0000 7f7fe3a00640 1 -- 172.21.15.102:0/3115866257 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7fbc06ac00 msgr2=0x7f7fbc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:27.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.625+0000 7f7fe3a00640 1 --2- 172.21.15.102:0/3115866257 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7fbc06ac00 0x7f7fbc06d0c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8036000 tx=0x7f7fc8030000 comp rx=0 tx=0).stop 2024-03-23T03:51:27.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.625+0000 7f7fe3a00640 1 -- 172.21.15.102:0/3115866257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fdc0b39b0 msgr2=0x7f7fdc0a8870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:27.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.625+0000 7f7fe3a00640 1 --2- 172.21.15.102:0/3115866257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fdc0b39b0 0x7f7fdc0a8870 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f7fd000c7f0 tx=0x7f7fd000cbf0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.625+0000 7f7fe3a00640 1 -- 172.21.15.102:0/3115866257 shutdown_connections 2024-03-23T03:51:27.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.625+0000 7f7fe3a00640 1 --2- 172.21.15.102:0/3115866257 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7fbc06ac00 0x7f7fbc06d0c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.625+0000 7f7fe3a00640 1 --2- 172.21.15.102:0/3115866257 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fdc15e500 0x7f7fdc140880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.625+0000 7f7fe3a00640 1 --2- 172.21.15.102:0/3115866257 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7fdc15b560 0x7f7fdc141a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.625+0000 7f7fe3a00640 1 --2- 172.21.15.102:0/3115866257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fdc0b39b0 0x7f7fdc0a8870 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.625+0000 7f7fe3a00640 1 -- 172.21.15.102:0/3115866257 >> 172.21.15.102:0/3115866257 conn(0x7f7fdc06a3d0 msgr2=0x7f7fdc15e950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:27.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.626+0000 7f7fe3a00640 1 -- 172.21.15.102:0/3115866257 shutdown_connections 2024-03-23T03:51:27.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.626+0000 7f7fe3a00640 1 -- 172.21.15.102:0/3115866257 wait complete. 2024-03-23T03:51:27.645 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-03-23T03:51:25.801531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":22,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":251752,"kb_used_data":11016,"kb_used_omap":12,"kb_used_meta":240627,"kb_avail":754722968,"statfs":{"total":773094113280,"available":772836319232,"internally_reserved":0,"allocated":11280384,"data_stored":5734920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":246402618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.067702"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.774604+0000","last_change":"2024-03-23T03:51:21.774604+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.774604+0000","last_undegraded":"2024-03-23T03:51:21.774604+0000","last_fullsized":"2024-03-23T03:51:21.774604+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:24:42.534444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.784556+0000","last_change":"2024-03-23T03:51:21.784556+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.784556+0000","last_undegraded":"2024-03-23T03:51:21.784556+0000","last_fullsized":"2024-03-23T03:51:21.784556+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:11:35.553745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.768447+0000","last_change":"2024-03-23T03:51:21.768447+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.768447+0000","last_undegraded":"2024-03-23T03:51:21.768447+0000","last_fullsized":"2024-03-23T03:51:21.768447+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:20:25.913448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.758852+0000","last_change":"2024-03-23T03:51:21.758852+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.758852+0000","last_undegraded":"2024-03-23T03:51:21.758852+0000","last_fullsized":"2024-03-23T03:51:21.758852+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:36:30.827292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.768312+0000","last_change":"2024-03-23T03:51:21.768312+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.768312+0000","last_undegraded":"2024-03-23T03:51:21.768312+0000","last_fullsized":"2024-03-23T03:51:21.768312+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:46:33.274045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.775572+0000","last_change":"2024-03-23T03:51:21.775572+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.775572+0000","last_undegraded":"2024-03-23T03:51:21.775572+0000","last_fullsized":"2024-03-23T03:51:21.775572+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:15:44.577151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.775304+0000","last_change":"2024-03-23T03:51:21.775304+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.775304+0000","last_undegraded":"2024-03-23T03:51:21.775304+0000","last_fullsized":"2024-03-23T03:51:21.775304+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:19.362177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.767403+0000","last_change":"2024-03-23T03:51:21.767403+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.767403+0000","last_undegraded":"2024-03-23T03:51:21.767403+0000","last_fullsized":"2024-03-23T03:51:21.767403+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:23:32.227733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.781582+0000","last_change":"2024-03-23T03:51:21.781582+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.781582+0000","last_undegraded":"2024-03-23T03:51:21.781582+0000","last_fullsized":"2024-03-23T03:51:21.781582+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:54.013665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.764834+0000","last_change":"2024-03-23T03:51:21.764834+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.764834+0000","last_undegraded":"2024-03-23T03:51:21.764834+0000","last_fullsized":"2024-03-23T03:51:21.764834+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:58:33.997748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.763235+0000","last_change":"2024-03-23T03:51:21.763235+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.763235+0000","last_undegraded":"2024-03-23T03:51:21.763235+0000","last_fullsized":"2024-03-23T03:51:21.763235+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:10:28.823831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.769131+0000","last_change":"2024-03-23T03:51:21.769131+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.769131+0000","last_undegraded":"2024-03-23T03:51:21.769131+0000","last_fullsized":"2024-03-23T03:51:21.769131+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:07.649845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.757932+0000","last_change":"2024-03-23T03:51:21.757932+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.757932+0000","last_undegraded":"2024-03-23T03:51:21.757932+0000","last_fullsized":"2024-03-23T03:51:21.757932+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:35:05.132187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.773754+0000","last_change":"2024-03-23T03:51:21.773754+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.773754+0000","last_undegraded":"2024-03-23T03:51:21.773754+0000","last_fullsized":"2024-03-23T03:51:21.773754+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:30:04.249708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.777084+0000","last_change":"2024-03-23T03:51:21.777084+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.777084+0000","last_undegraded":"2024-03-23T03:51:21.777084+0000","last_fullsized":"2024-03-23T03:51:21.777084+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:17:09.963708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.772977+0000","last_change":"2024-03-23T03:51:21.772977+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.772977+0000","last_undegraded":"2024-03-23T03:51:21.772977+0000","last_fullsized":"2024-03-23T03:51:21.772977+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:04:21.319327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.774158+0000","last_change":"2024-03-23T03:51:21.774158+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.774158+0000","last_undegraded":"2024-03-23T03:51:21.774158+0000","last_fullsized":"2024-03-23T03:51:21.774158+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:55.608125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.783911+0000","last_change":"2024-03-23T03:51:21.783911+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.783911+0000","last_undegraded":"2024-03-23T03:51:21.783911+0000","last_fullsized":"2024-03-23T03:51:21.783911+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:03:27.282303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.778317+0000","last_change":"2024-03-23T03:51:21.778317+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.778317+0000","last_undegraded":"2024-03-23T03:51:21.778317+0000","last_fullsized":"2024-03-23T03:51:21.778317+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:48:26.593847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.765961+0000","last_change":"2024-03-23T03:51:21.765961+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.765961+0000","last_undegraded":"2024-03-23T03:51:21.765961+0000","last_fullsized":"2024-03-23T03:51:21.765961+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:13:39.429313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.767195+0000","last_change":"2024-03-23T03:51:21.767195+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.767195+0000","last_undegraded":"2024-03-23T03:51:21.767195+0000","last_fullsized":"2024-03-23T03:51:21.767195+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:20:51.607176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.764682+0000","last_change":"2024-03-23T03:51:21.764682+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.764682+0000","last_undegraded":"2024-03-23T03:51:21.764682+0000","last_fullsized":"2024-03-23T03:51:21.764682+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:04.683164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.765012+0000","last_change":"2024-03-23T03:51:21.765012+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.765012+0000","last_undegraded":"2024-03-23T03:51:21.765012+0000","last_fullsized":"2024-03-23T03:51:21.765012+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:46:32.018385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.765000+0000","last_change":"2024-03-23T03:51:21.765000+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.765000+0000","last_undegraded":"2024-03-23T03:51:21.765000+0000","last_fullsized":"2024-03-23T03:51:21.765000+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:47.178608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.770079+0000","last_change":"2024-03-23T03:51:21.770079+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.770079+0000","last_undegraded":"2024-03-23T03:51:21.770079+0000","last_fullsized":"2024-03-23T03:51:21.770079+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:58:40.479780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.766309+0000","last_change":"2024-03-23T03:51:21.766309+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.766309+0000","last_undegraded":"2024-03-23T03:51:21.766309+0000","last_fullsized":"2024-03-23T03:51:21.766309+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:58:05.924179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.761725+0000","last_change":"2024-03-23T03:51:21.761725+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.761725+0000","last_undegraded":"2024-03-23T03:51:21.761725+0000","last_fullsized":"2024-03-23T03:51:21.761725+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:20.563242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.776967+0000","last_change":"2024-03-23T03:51:21.776967+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.776967+0000","last_undegraded":"2024-03-23T03:51:21.776967+0000","last_fullsized":"2024-03-23T03:51:21.776967+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:11:23.842491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.766688+0000","last_change":"2024-03-23T03:51:21.766688+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.766688+0000","last_undegraded":"2024-03-23T03:51:21.766688+0000","last_fullsized":"2024-03-23T03:51:21.766688+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:46.517427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.769700+0000","last_change":"2024-03-23T03:51:21.769700+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.769700+0000","last_undegraded":"2024-03-23T03:51:21.769700+0000","last_fullsized":"2024-03-23T03:51:21.769700+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:36:38.929407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.774265+0000","last_change":"2024-03-23T03:51:21.774265+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.774265+0000","last_undegraded":"2024-03-23T03:51:21.774265+0000","last_fullsized":"2024-03-23T03:51:21.774265+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:25:25.635704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.766878+0000","last_change":"2024-03-23T03:51:21.766878+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.766878+0000","last_undegraded":"2024-03-23T03:51:21.766878+0000","last_fullsized":"2024-03-23T03:51:21.766878+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:19:58.455715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.764169+0000","last_change":"2024-03-23T03:51:21.764169+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.764169+0000","last_undegraded":"2024-03-23T03:51:21.764169+0000","last_fullsized":"2024-03-23T03:51:21.764169+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:57:56.460821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.793055+0000","last_change":"2024-03-23T03:51:21.789123+0000","last_active":"2024-03-23T03:51:21.793055+0000","last_peered":"2024-03-23T03:51:21.793055+0000","last_clean":"2024-03-23T03:51:21.793055+0000","last_became_active":"2024-03-23T03:51:21.788839+0000","last_became_peered":"2024-03-23T03:51:21.788839+0000","last_unstale":"2024-03-23T03:51:21.793055+0000","last_undegraded":"2024-03-23T03:51:21.793055+0000","last_fullsized":"2024-03-23T03:51:21.793055+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:54:15.206699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.765433+0000","last_change":"2024-03-23T03:51:21.765433+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.765433+0000","last_undegraded":"2024-03-23T03:51:21.765433+0000","last_fullsized":"2024-03-23T03:51:21.765433+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:04.438866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.791287+0000","last_change":"2024-03-23T03:51:21.785555+0000","last_active":"2024-03-23T03:51:21.791287+0000","last_peered":"2024-03-23T03:51:21.791287+0000","last_clean":"2024-03-23T03:51:21.791287+0000","last_became_active":"2024-03-23T03:51:21.785232+0000","last_became_peered":"2024-03-23T03:51:21.785232+0000","last_unstale":"2024-03-23T03:51:21.791287+0000","last_undegraded":"2024-03-23T03:51:21.791287+0000","last_fullsized":"2024-03-23T03:51:21.791287+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:27:33.193844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.768476+0000","last_change":"2024-03-23T03:51:21.768476+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.768476+0000","last_undegraded":"2024-03-23T03:51:21.768476+0000","last_fullsized":"2024-03-23T03:51:21.768476+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:27:17.201821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.791576+0000","last_change":"2024-03-23T03:51:21.783666+0000","last_active":"2024-03-23T03:51:21.791576+0000","last_peered":"2024-03-23T03:51:21.791576+0000","last_clean":"2024-03-23T03:51:21.791576+0000","last_became_active":"2024-03-23T03:51:21.783377+0000","last_became_peered":"2024-03-23T03:51:21.783377+0000","last_unstale":"2024-03-23T03:51:21.791576+0000","last_undegraded":"2024-03-23T03:51:21.791576+0000","last_fullsized":"2024-03-23T03:51:21.791576+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:42:08.794820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.761765+0000","last_change":"2024-03-23T03:51:21.761765+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.761765+0000","last_undegraded":"2024-03-23T03:51:21.761765+0000","last_fullsized":"2024-03-23T03:51:21.761765+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:17:09.963708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.795666+0000","last_change":"2024-03-23T03:51:21.792017+0000","last_active":"2024-03-23T03:51:21.795666+0000","last_peered":"2024-03-23T03:51:21.795666+0000","last_clean":"2024-03-23T03:51:21.795666+0000","last_became_active":"2024-03-23T03:51:21.791763+0000","last_became_peered":"2024-03-23T03:51:21.791763+0000","last_unstale":"2024-03-23T03:51:21.795666+0000","last_undegraded":"2024-03-23T03:51:21.795666+0000","last_fullsized":"2024-03-23T03:51:21.795666+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:05:09.454442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.757766+0000","last_change":"2024-03-23T03:51:21.757766+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.757766+0000","last_undegraded":"2024-03-23T03:51:21.757766+0000","last_fullsized":"2024-03-23T03:51:21.757766+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:38:27.787294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.794129+0000","last_change":"2024-03-23T03:51:21.792106+0000","last_active":"2024-03-23T03:51:21.794129+0000","last_peered":"2024-03-23T03:51:21.794129+0000","last_clean":"2024-03-23T03:51:21.794129+0000","last_became_active":"2024-03-23T03:51:21.791295+0000","last_became_peered":"2024-03-23T03:51:21.791295+0000","last_unstale":"2024-03-23T03:51:21.794129+0000","last_undegraded":"2024-03-23T03:51:21.794129+0000","last_fullsized":"2024-03-23T03:51:21.794129+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:00:35.520733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.768896+0000","last_change":"2024-03-23T03:51:21.768896+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.768896+0000","last_undegraded":"2024-03-23T03:51:21.768896+0000","last_fullsized":"2024-03-23T03:51:21.768896+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:37:27.534087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.785068+0000","last_change":"2024-03-23T03:51:21.780198+0000","last_active":"2024-03-23T03:51:21.785068+0000","last_peered":"2024-03-23T03:51:21.785068+0000","last_clean":"2024-03-23T03:51:21.785068+0000","last_became_active":"2024-03-23T03:51:21.779256+0000","last_became_peered":"2024-03-23T03:51:21.779256+0000","last_unstale":"2024-03-23T03:51:21.785068+0000","last_undegraded":"2024-03-23T03:51:21.785068+0000","last_fullsized":"2024-03-23T03:51:21.785068+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:00:33.834214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.781854+0000","last_change":"2024-03-23T03:51:21.781854+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.781854+0000","last_undegraded":"2024-03-23T03:51:21.781854+0000","last_fullsized":"2024-03-23T03:51:21.781854+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:50:44.800098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.791010+0000","last_change":"2024-03-23T03:51:21.783223+0000","last_active":"2024-03-23T03:51:21.791010+0000","last_peered":"2024-03-23T03:51:21.791010+0000","last_clean":"2024-03-23T03:51:21.791010+0000","last_became_active":"2024-03-23T03:51:21.782799+0000","last_became_peered":"2024-03-23T03:51:21.782799+0000","last_unstale":"2024-03-23T03:51:21.791010+0000","last_undegraded":"2024-03-23T03:51:21.791010+0000","last_fullsized":"2024-03-23T03:51:21.791010+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:56:44.890019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.759573+0000","last_change":"2024-03-23T03:51:21.759573+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.759573+0000","last_undegraded":"2024-03-23T03:51:21.759573+0000","last_fullsized":"2024-03-23T03:51:21.759573+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:31:57.940022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.782389+0000","last_change":"2024-03-23T03:51:21.776001+0000","last_active":"2024-03-23T03:51:21.782389+0000","last_peered":"2024-03-23T03:51:21.782389+0000","last_clean":"2024-03-23T03:51:21.782389+0000","last_became_active":"2024-03-23T03:51:21.775726+0000","last_became_peered":"2024-03-23T03:51:21.775726+0000","last_unstale":"2024-03-23T03:51:21.782389+0000","last_undegraded":"2024-03-23T03:51:21.782389+0000","last_fullsized":"2024-03-23T03:51:21.782389+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:07:52.649065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.766180+0000","last_change":"2024-03-23T03:51:21.766180+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.766180+0000","last_undegraded":"2024-03-23T03:51:21.766180+0000","last_fullsized":"2024-03-23T03:51:21.766180+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:55:30.397347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.791658+0000","last_change":"2024-03-23T03:51:21.786526+0000","last_active":"2024-03-23T03:51:21.791658+0000","last_peered":"2024-03-23T03:51:21.791658+0000","last_clean":"2024-03-23T03:51:21.791658+0000","last_became_active":"2024-03-23T03:51:21.783040+0000","last_became_peered":"2024-03-23T03:51:21.783040+0000","last_unstale":"2024-03-23T03:51:21.791658+0000","last_undegraded":"2024-03-23T03:51:21.791658+0000","last_fullsized":"2024-03-23T03:51:21.791658+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:53.003075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.764341+0000","last_change":"2024-03-23T03:51:21.764341+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.764341+0000","last_undegraded":"2024-03-23T03:51:21.764341+0000","last_fullsized":"2024-03-23T03:51:21.764341+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:42:06.329388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.792942+0000","last_change":"2024-03-23T03:51:21.787895+0000","last_active":"2024-03-23T03:51:21.792942+0000","last_peered":"2024-03-23T03:51:21.792942+0000","last_clean":"2024-03-23T03:51:21.792942+0000","last_became_active":"2024-03-23T03:51:21.787627+0000","last_became_peered":"2024-03-23T03:51:21.787627+0000","last_unstale":"2024-03-23T03:51:21.792942+0000","last_undegraded":"2024-03-23T03:51:21.792942+0000","last_fullsized":"2024-03-23T03:51:21.792942+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:31:02.266262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.765914+0000","last_change":"2024-03-23T03:51:21.765914+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.765914+0000","last_undegraded":"2024-03-23T03:51:21.765914+0000","last_fullsized":"2024-03-23T03:51:21.765914+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:34:16.439779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.793407+0000","last_change":"2024-03-23T03:51:21.786687+0000","last_active":"2024-03-23T03:51:21.793407+0000","last_peered":"2024-03-23T03:51:21.793407+0000","last_clean":"2024-03-23T03:51:21.793407+0000","last_became_active":"2024-03-23T03:51:21.786374+0000","last_became_peered":"2024-03-23T03:51:21.786374+0000","last_unstale":"2024-03-23T03:51:21.793407+0000","last_undegraded":"2024-03-23T03:51:21.793407+0000","last_fullsized":"2024-03-23T03:51:21.793407+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:09:59.406767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.766747+0000","last_change":"2024-03-23T03:51:21.766747+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.766747+0000","last_undegraded":"2024-03-23T03:51:21.766747+0000","last_fullsized":"2024-03-23T03:51:21.766747+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:19:44.510925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.792849+0000","last_change":"2024-03-23T03:51:21.789690+0000","last_active":"2024-03-23T03:51:21.792849+0000","last_peered":"2024-03-23T03:51:21.792849+0000","last_clean":"2024-03-23T03:51:21.792849+0000","last_became_active":"2024-03-23T03:51:21.789409+0000","last_became_peered":"2024-03-23T03:51:21.789409+0000","last_unstale":"2024-03-23T03:51:21.792849+0000","last_undegraded":"2024-03-23T03:51:21.792849+0000","last_fullsized":"2024-03-23T03:51:21.792849+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:03:22.934141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.764042+0000","last_change":"2024-03-23T03:51:21.764042+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.764042+0000","last_undegraded":"2024-03-23T03:51:21.764042+0000","last_fullsized":"2024-03-23T03:51:21.764042+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:04:22.944014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.782681+0000","last_change":"2024-03-23T03:51:21.778247+0000","last_active":"2024-03-23T03:51:21.782681+0000","last_peered":"2024-03-23T03:51:21.782681+0000","last_clean":"2024-03-23T03:51:21.782681+0000","last_became_active":"2024-03-23T03:51:21.777903+0000","last_became_peered":"2024-03-23T03:51:21.777903+0000","last_unstale":"2024-03-23T03:51:21.782681+0000","last_undegraded":"2024-03-23T03:51:21.782681+0000","last_fullsized":"2024-03-23T03:51:21.782681+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:30:03.239118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.775918+0000","last_change":"2024-03-23T03:51:21.775918+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.775918+0000","last_undegraded":"2024-03-23T03:51:21.775918+0000","last_fullsized":"2024-03-23T03:51:21.775918+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:06:14.520077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.786533+0000","last_change":"2024-03-23T03:51:21.780251+0000","last_active":"2024-03-23T03:51:21.786533+0000","last_peered":"2024-03-23T03:51:21.786533+0000","last_clean":"2024-03-23T03:51:21.786533+0000","last_became_active":"2024-03-23T03:51:21.779927+0000","last_became_peered":"2024-03-23T03:51:21.779927+0000","last_unstale":"2024-03-23T03:51:21.786533+0000","last_undegraded":"2024-03-23T03:51:21.786533+0000","last_fullsized":"2024-03-23T03:51:21.786533+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:28:51.270476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.768567+0000","last_change":"2024-03-23T03:51:21.768567+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.768567+0000","last_undegraded":"2024-03-23T03:51:21.768567+0000","last_fullsized":"2024-03-23T03:51:21.768567+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:53.513182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.793062+0000","last_change":"2024-03-23T03:51:21.790264+0000","last_active":"2024-03-23T03:51:21.793062+0000","last_peered":"2024-03-23T03:51:21.793062+0000","last_clean":"2024-03-23T03:51:21.793062+0000","last_became_active":"2024-03-23T03:51:21.789990+0000","last_became_peered":"2024-03-23T03:51:21.789990+0000","last_unstale":"2024-03-23T03:51:21.793062+0000","last_undegraded":"2024-03-23T03:51:21.793062+0000","last_fullsized":"2024-03-23T03:51:21.793062+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:17:42.968056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.773680+0000","last_change":"2024-03-23T03:51:21.773680+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.773680+0000","last_undegraded":"2024-03-23T03:51:21.773680+0000","last_fullsized":"2024-03-23T03:51:21.773680+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:52:35.913771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.789713+0000","last_change":"2024-03-23T03:51:21.783083+0000","last_active":"2024-03-23T03:51:21.789713+0000","last_peered":"2024-03-23T03:51:21.789713+0000","last_clean":"2024-03-23T03:51:21.789713+0000","last_became_active":"2024-03-23T03:51:21.782771+0000","last_became_peered":"2024-03-23T03:51:21.782771+0000","last_unstale":"2024-03-23T03:51:21.789713+0000","last_undegraded":"2024-03-23T03:51:21.789713+0000","last_fullsized":"2024-03-23T03:51:21.789713+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:11:22.831901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.766375+0000","last_change":"2024-03-23T03:51:21.766375+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.766375+0000","last_undegraded":"2024-03-23T03:51:21.766375+0000","last_fullsized":"2024-03-23T03:51:21.766375+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:35:48.145231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.795163+0000","last_change":"2024-03-23T03:51:21.793220+0000","last_active":"2024-03-23T03:51:21.795163+0000","last_peered":"2024-03-23T03:51:21.795163+0000","last_clean":"2024-03-23T03:51:21.795163+0000","last_became_active":"2024-03-23T03:51:21.792937+0000","last_became_peered":"2024-03-23T03:51:21.792937+0000","last_unstale":"2024-03-23T03:51:21.795163+0000","last_undegraded":"2024-03-23T03:51:21.795163+0000","last_fullsized":"2024-03-23T03:51:21.795163+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:52.502591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.773976+0000","last_change":"2024-03-23T03:51:21.773976+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.773976+0000","last_undegraded":"2024-03-23T03:51:21.773976+0000","last_fullsized":"2024-03-23T03:51:21.773976+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:31:10.713669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.794067+0000","last_change":"2024-03-23T03:51:21.792048+0000","last_active":"2024-03-23T03:51:21.794067+0000","last_peered":"2024-03-23T03:51:21.794067+0000","last_clean":"2024-03-23T03:51:21.794067+0000","last_became_active":"2024-03-23T03:51:21.791713+0000","last_became_peered":"2024-03-23T03:51:21.791713+0000","last_unstale":"2024-03-23T03:51:21.794067+0000","last_undegraded":"2024-03-23T03:51:21.794067+0000","last_fullsized":"2024-03-23T03:51:21.794067+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:59:01.379564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.767517+0000","last_change":"2024-03-23T03:51:21.767517+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.767517+0000","last_undegraded":"2024-03-23T03:51:21.767517+0000","last_fullsized":"2024-03-23T03:51:21.767517+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:59:59.276905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.791584+0000","last_change":"2024-03-23T03:51:21.789004+0000","last_active":"2024-03-23T03:51:21.791584+0000","last_peered":"2024-03-23T03:51:21.791584+0000","last_clean":"2024-03-23T03:51:21.791584+0000","last_became_active":"2024-03-23T03:51:21.788647+0000","last_became_peered":"2024-03-23T03:51:21.788647+0000","last_unstale":"2024-03-23T03:51:21.791584+0000","last_undegraded":"2024-03-23T03:51:21.791584+0000","last_fullsized":"2024-03-23T03:51:21.791584+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:59:01.379564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.767902+0000","last_change":"2024-03-23T03:51:21.767902+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.767902+0000","last_undegraded":"2024-03-23T03:51:21.767902+0000","last_fullsized":"2024-03-23T03:51:21.767902+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:15:49.588458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.788658+0000","last_change":"2024-03-23T03:51:21.782473+0000","last_active":"2024-03-23T03:51:21.788658+0000","last_peered":"2024-03-23T03:51:21.788658+0000","last_clean":"2024-03-23T03:51:21.788658+0000","last_became_active":"2024-03-23T03:51:21.781740+0000","last_became_peered":"2024-03-23T03:51:21.781740+0000","last_unstale":"2024-03-23T03:51:21.788658+0000","last_undegraded":"2024-03-23T03:51:21.788658+0000","last_fullsized":"2024-03-23T03:51:21.788658+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:30:22.629109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.769342+0000","last_change":"2024-03-23T03:51:21.769342+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.769342+0000","last_undegraded":"2024-03-23T03:51:21.769342+0000","last_fullsized":"2024-03-23T03:51:21.769342+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:18:12.413501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.785499+0000","last_change":"2024-03-23T03:51:21.782262+0000","last_active":"2024-03-23T03:51:21.785499+0000","last_peered":"2024-03-23T03:51:21.785499+0000","last_clean":"2024-03-23T03:51:21.785499+0000","last_became_active":"2024-03-23T03:51:21.781940+0000","last_became_peered":"2024-03-23T03:51:21.781940+0000","last_unstale":"2024-03-23T03:51:21.785499+0000","last_undegraded":"2024-03-23T03:51:21.785499+0000","last_fullsized":"2024-03-23T03:51:21.785499+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:49.654183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.759451+0000","last_change":"2024-03-23T03:51:21.759451+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.759451+0000","last_undegraded":"2024-03-23T03:51:21.759451+0000","last_fullsized":"2024-03-23T03:51:21.759451+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:06:50.564546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.795104+0000","last_change":"2024-03-23T03:51:21.792924+0000","last_active":"2024-03-23T03:51:21.795104+0000","last_peered":"2024-03-23T03:51:21.795104+0000","last_clean":"2024-03-23T03:51:21.795104+0000","last_became_active":"2024-03-23T03:51:21.790283+0000","last_became_peered":"2024-03-23T03:51:21.790283+0000","last_unstale":"2024-03-23T03:51:21.795104+0000","last_undegraded":"2024-03-23T03:51:21.795104+0000","last_fullsized":"2024-03-23T03:51:21.795104+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:19:57.445124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.769014+0000","last_change":"2024-03-23T03:51:21.769014+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.769014+0000","last_undegraded":"2024-03-23T03:51:21.769014+0000","last_fullsized":"2024-03-23T03:51:21.769014+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:35.027397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.790727+0000","last_change":"2024-03-23T03:51:21.784479+0000","last_active":"2024-03-23T03:51:21.790727+0000","last_peered":"2024-03-23T03:51:21.790727+0000","last_clean":"2024-03-23T03:51:21.790727+0000","last_became_active":"2024-03-23T03:51:21.784166+0000","last_became_peered":"2024-03-23T03:51:21.784166+0000","last_unstale":"2024-03-23T03:51:21.790727+0000","last_undegraded":"2024-03-23T03:51:21.790727+0000","last_fullsized":"2024-03-23T03:51:21.790727+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:35:04.474544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.766611+0000","last_change":"2024-03-23T03:51:21.766611+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.766611+0000","last_undegraded":"2024-03-23T03:51:21.766611+0000","last_fullsized":"2024-03-23T03:51:21.766611+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:08:03.409470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.782622+0000","last_change":"2024-03-23T03:51:21.776941+0000","last_active":"2024-03-23T03:51:21.782622+0000","last_peered":"2024-03-23T03:51:21.782622+0000","last_clean":"2024-03-23T03:51:21.782622+0000","last_became_active":"2024-03-23T03:51:21.776643+0000","last_became_peered":"2024-03-23T03:51:21.776643+0000","last_unstale":"2024-03-23T03:51:21.782622+0000","last_undegraded":"2024-03-23T03:51:21.782622+0000","last_fullsized":"2024-03-23T03:51:21.782622+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:17:08.953118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.771267+0000","last_change":"2024-03-23T03:51:21.771267+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.771267+0000","last_undegraded":"2024-03-23T03:51:21.771267+0000","last_fullsized":"2024-03-23T03:51:21.771267+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:46:03.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.790974+0000","last_change":"2024-03-23T03:51:21.785341+0000","last_active":"2024-03-23T03:51:21.790974+0000","last_peered":"2024-03-23T03:51:21.790974+0000","last_clean":"2024-03-23T03:51:21.790974+0000","last_became_active":"2024-03-23T03:51:21.784654+0000","last_became_peered":"2024-03-23T03:51:21.784654+0000","last_unstale":"2024-03-23T03:51:21.790974+0000","last_undegraded":"2024-03-23T03:51:21.790974+0000","last_fullsized":"2024-03-23T03:51:21.790974+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:36.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.758321+0000","last_change":"2024-03-23T03:51:21.758321+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.758321+0000","last_undegraded":"2024-03-23T03:51:21.758321+0000","last_fullsized":"2024-03-23T03:51:21.758321+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:27:15.422967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.792705+0000","last_change":"2024-03-23T03:51:21.788631+0000","last_active":"2024-03-23T03:51:21.792705+0000","last_peered":"2024-03-23T03:51:21.792705+0000","last_clean":"2024-03-23T03:51:21.792705+0000","last_became_active":"2024-03-23T03:51:21.788145+0000","last_became_peered":"2024-03-23T03:51:21.788145+0000","last_unstale":"2024-03-23T03:51:21.792705+0000","last_undegraded":"2024-03-23T03:51:21.792705+0000","last_fullsized":"2024-03-23T03:51:21.792705+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:45.506837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.764213+0000","last_change":"2024-03-23T03:51:21.764213+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.764213+0000","last_undegraded":"2024-03-23T03:51:21.764213+0000","last_fullsized":"2024-03-23T03:51:21.764213+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:30:04.249708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.791253+0000","last_change":"2024-03-23T03:51:21.788546+0000","last_active":"2024-03-23T03:51:21.791253+0000","last_peered":"2024-03-23T03:51:21.791253+0000","last_clean":"2024-03-23T03:51:21.791253+0000","last_became_active":"2024-03-23T03:51:21.788282+0000","last_became_peered":"2024-03-23T03:51:21.788282+0000","last_unstale":"2024-03-23T03:51:21.791253+0000","last_undegraded":"2024-03-23T03:51:21.791253+0000","last_fullsized":"2024-03-23T03:51:21.791253+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:04:20.308736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.765746+0000","last_change":"2024-03-23T03:51:21.765746+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.765746+0000","last_undegraded":"2024-03-23T03:51:21.765746+0000","last_fullsized":"2024-03-23T03:51:21.765746+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:05.542126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.795287+0000","last_change":"2024-03-23T03:51:21.793737+0000","last_active":"2024-03-23T03:51:21.795287+0000","last_peered":"2024-03-23T03:51:21.795287+0000","last_clean":"2024-03-23T03:51:21.795287+0000","last_became_active":"2024-03-23T03:51:21.791575+0000","last_became_peered":"2024-03-23T03:51:21.791575+0000","last_unstale":"2024-03-23T03:51:21.795287+0000","last_undegraded":"2024-03-23T03:51:21.795287+0000","last_fullsized":"2024-03-23T03:51:21.795287+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:54:20.064900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.767710+0000","last_change":"2024-03-23T03:51:21.767710+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.767710+0000","last_undegraded":"2024-03-23T03:51:21.767710+0000","last_fullsized":"2024-03-23T03:51:21.767710+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:45:36.655749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.791188+0000","last_change":"2024-03-23T03:51:21.787337+0000","last_active":"2024-03-23T03:51:21.791188+0000","last_peered":"2024-03-23T03:51:21.791188+0000","last_clean":"2024-03-23T03:51:21.791188+0000","last_became_active":"2024-03-23T03:51:21.787076+0000","last_became_peered":"2024-03-23T03:51:21.787076+0000","last_unstale":"2024-03-23T03:51:21.791188+0000","last_undegraded":"2024-03-23T03:51:21.791188+0000","last_fullsized":"2024-03-23T03:51:21.791188+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:09:48.708095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.776317+0000","last_change":"2024-03-23T03:51:21.776317+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.776317+0000","last_undegraded":"2024-03-23T03:51:21.776317+0000","last_fullsized":"2024-03-23T03:51:21.776317+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:05:10.465032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.794483+0000","last_change":"2024-03-23T03:51:21.789939+0000","last_active":"2024-03-23T03:51:21.794483+0000","last_peered":"2024-03-23T03:51:21.794483+0000","last_clean":"2024-03-23T03:51:21.794483+0000","last_became_active":"2024-03-23T03:51:21.788412+0000","last_became_peered":"2024-03-23T03:51:21.788412+0000","last_unstale":"2024-03-23T03:51:21.794483+0000","last_undegraded":"2024-03-23T03:51:21.794483+0000","last_fullsized":"2024-03-23T03:51:21.794483+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:35:47.134641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.763474+0000","last_change":"2024-03-23T03:51:21.763474+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.763474+0000","last_undegraded":"2024-03-23T03:51:21.763474+0000","last_fullsized":"2024-03-23T03:51:21.763474+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:05:03.516080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.795576+0000","last_change":"2024-03-23T03:51:21.791827+0000","last_active":"2024-03-23T03:51:21.795576+0000","last_peered":"2024-03-23T03:51:21.795576+0000","last_clean":"2024-03-23T03:51:21.795576+0000","last_became_active":"2024-03-23T03:51:21.791445+0000","last_became_peered":"2024-03-23T03:51:21.791445+0000","last_unstale":"2024-03-23T03:51:21.795576+0000","last_undegraded":"2024-03-23T03:51:21.795576+0000","last_fullsized":"2024-03-23T03:51:21.795576+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:58.150428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":1,"reported_epoch":103,"state":"creating+peering","last_fresh":"2024-03-23T03:51:21.762191+0000","last_change":"2024-03-23T03:51:21.762191+0000","last_active":"2024-03-23T03:51:21.733694+0000","last_peered":"2024-03-23T03:51:21.733694+0000","last_clean":"2024-03-23T03:51:21.733694+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T03:51:21.762191+0000","last_undegraded":"2024-03-23T03:51:21.762191+0000","last_fullsized":"2024-03-23T03:51:21.762191+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:04:21.319327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":9,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-03-23T03:51:21.782798+0000","last_change":"2024-03-23T03:51:21.779521+0000","last_active":"2024-03-23T03:51:21.782798+0000","last_peered":"2024-03-23T03:51:21.782798+0000","last_clean":"2024-03-23T03:51:21.782798+0000","last_became_active":"2024-03-23T03:51:21.779111+0000","last_became_peered":"2024-03-23T03:51:21.779111+0000","last_unstale":"2024-03-23T03:51:21.782798+0000","last_undegraded":"2024-03-23T03:51:21.782798+0000","last_fullsized":"2024-03-23T03:51:21.782798+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:29:39.956286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":228,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-23T03:51:20.741124+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:51:20.741124+0000","last_peered":"2024-03-23T03:51:20.741124+0000","last_clean":"2024-03-23T03:51:20.741124+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:51:20.741124+0000","last_undegraded":"2024-03-23T03:51:20.741124+0000","last_fullsized":"2024-03-23T03:51:20.741124+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542203,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1855488,"data_stored":1159641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.087,"15min":1.0720000000000001},"min":{"1min":0.65700000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5209999999999999,"5min":1.635,"15min":1.635},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.042,"15min":1.0469999999999999},"min":{"1min":0.749,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.393,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.49}]},{"osd":1,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.081,"15min":1.0529999999999999},"min":{"1min":0.71499999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6120000000000001,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.077},{"interface":"front","average":{"1min":1.03,"5min":1.1719999999999999,"15min":1.2809999999999999},"min":{"1min":0.79000000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.321,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":0.98399999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.125,"15min":1.151},"min":{"1min":0.81299999999999994,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.583,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.028},{"interface":"front","average":{"1min":1.167,"5min":1.2869999999999999,"15min":1.401},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.506,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.5369999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.298,"15min":1.4179999999999999},"min":{"1min":0.71999999999999997,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.4219999999999999,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.125},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.325,"15min":1.353},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.677,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.421}]},{"osd":4,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.254,"15min":1.2629999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.8220000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.605},{"interface":"front","average":{"1min":1.333,"5min":1.347,"15min":1.3720000000000001},"min":{"1min":0.67500000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.778,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.4610000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.389,"15min":1.4179999999999999},"min":{"1min":0.81299999999999994,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.7310000000000001,"5min":2.645,"15min":2.645},"last":1.206},{"interface":"front","average":{"1min":1.361,"5min":1.417,"15min":1.446},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.7649999999999999,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.661}]},{"osd":6,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.504,"15min":1.552},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.9279999999999999,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.708},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.474,"15min":1.5149999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9399999999999999,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.347}]}]},{"osd":6,"up_from":14,"seq":60129542201,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32348,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339492,"statfs":{"total":96636764160,"available":96603639808,"internally_reserved":0,"allocated":1261568,"data_stored":569273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.0429999999999999,"15min":1.044},"min":{"1min":0.69699999999999995,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.05,"15min":1.04},"min":{"1min":0.66100000000000003,"5min":0.62,"15min":0.62},"max":{"1min":1.5409999999999999,"5min":1.946,"15min":1.946},"last":1.385}]},{"osd":1,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.087,"15min":1.075},"min":{"1min":0.745,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.196,"5min":1.1339999999999999,"15min":1.1579999999999999},"min":{"1min":0.80500000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.5369999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.88800000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.083,"15min":1.071},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.4350000000000001,"5min":2.044,"15min":2.044},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.133,"5min":1.1759999999999999,"15min":1.212},"min":{"1min":0.85499999999999998,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.6599999999999999,"5min":1.681,"15min":1.681},"last":1.028}]},{"osd":3,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.2370000000000001,"15min":1.276},"min":{"1min":0.88100000000000001,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.786,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.262},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.2330000000000001,"15min":1.2569999999999999},"min":{"1min":0.90100000000000002,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.446,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.157}]},{"osd":4,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3779999999999999,"15min":1.385},"min":{"1min":1.1930000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.6679999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.431,"5min":1.371,"15min":1.3859999999999999},"min":{"1min":1.234,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.639,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.5429999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.403,"15min":1.4319999999999999},"min":{"1min":1.0669999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.794,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.474,"5min":1.508,"15min":1.542},"min":{"1min":1.091,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8200000000000001,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.2350000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.4059999999999999,"15min":1.4119999999999999},"min":{"1min":0.95999999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.0760000000000001,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.482,"15min":1.5309999999999999},"min":{"1min":1.125,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.8660000000000001,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.1830000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1261568,"data_stored":569273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1439999999999999,"15min":1.2030000000000001},"min":{"1min":0.72999999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4830000000000001,"5min":1.806,"15min":1.806},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.113,"5min":1.1399999999999999,"15min":1.1759999999999999},"min":{"1min":0.76500000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.855,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.872}]},{"osd":1,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0940000000000001,"15min":1.1000000000000001},"min":{"1min":0.80100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.3520000000000001,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.232,"5min":1.165,"15min":1.1930000000000001},"min":{"1min":0.84799999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.663,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.94399999999999995}]},{"osd":2,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.274,"15min":1.3460000000000001},"min":{"1min":0.70499999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8939999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.226,"5min":1.2989999999999999,"15min":1.3660000000000001},"min":{"1min":0.875,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.579,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.22}]},{"osd":3,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.2,"15min":1.244},"min":{"1min":0.85399999999999998,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.7050000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.375,"15min":1.446},"min":{"1min":0.997,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.964,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.1739999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3049999999999999,"15min":1.399},"min":{"1min":0.65700000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6100000000000001,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.528},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3859999999999999,"15min":1.4359999999999999},"min":{"1min":0.83699999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8069999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.415}]},{"osd":6,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.4159999999999999,"15min":1.5},"min":{"1min":0.83399999999999996,"5min":0.63,"15min":0.63},"max":{"1min":1.5840000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.48},{"interface":"front","average":{"1min":1.423,"5min":1.5409999999999999,"15min":1.639},"min":{"1min":1.1950000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.6359999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.5880000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4910000000000001,"15min":1.591},"min":{"1min":1.095,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.9259999999999999,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.623},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5840000000000001,"15min":1.6839999999999999},"min":{"1min":1.272,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.056,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.833}]}]},{"osd":4,"up_from":14,"seq":60129542202,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32348,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339492,"statfs":{"total":96636764160,"available":96603639808,"internally_reserved":0,"allocated":1261568,"data_stored":569273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.004,"15min":0.98099999999999998},"min":{"1min":0.624,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.47,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.995,"15min":1.0049999999999999},"min":{"1min":0.56100000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3400000000000001,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.82699999999999996}]},{"osd":1,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0680000000000001,"15min":1.05},"min":{"1min":0.65100000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.5,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0880000000000001,"15min":1.0640000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.5209999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.93600000000000005}]},{"osd":2,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.2470000000000001,"15min":1.3089999999999999},"min":{"1min":0.70399999999999996,"5min":0.625,"15min":0.625},"max":{"1min":1.452,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.2789999999999999,"15min":1.2969999999999999},"min":{"1min":0.59899999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6859999999999999,"5min":2.181,"15min":2.181},"last":1.1459999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.1160000000000001,"15min":1.1240000000000001},"min":{"1min":0.61899999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.4419999999999999,"5min":1.821,"15min":1.821},"last":1.006},{"interface":"front","average":{"1min":1.141,"5min":1.2310000000000001,"15min":1.268},"min":{"1min":0.73799999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.599,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.0529999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.2370000000000001,"15min":1.26},"min":{"1min":0.63800000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.49,"5min":1.929,"15min":1.929},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.2709999999999999,"15min":1.274},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.45,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.1120000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.371,"15min":1.409},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6719999999999999,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.1830000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.4139999999999999,"15min":1.3999999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.8100000000000001,"5min":2.016,"15min":2.016},"last":1.319}]},{"osd":7,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.361,"15min":1.363},"min":{"1min":0.68200000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.5569999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.452},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.456,"15min":1.4870000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.716,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.363}]}]},{"osd":3,"up_from":14,"seq":60129542202,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32348,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339492,"statfs":{"total":96636764160,"available":96603639808,"internally_reserved":0,"allocated":1261568,"data_stored":569273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.81999999999999995,"15min":0.85499999999999998},"min":{"1min":0.35499999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.4419999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.92200000000000004,"15min":0.92400000000000004},"min":{"1min":0.502,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.538,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.56000000000000005}]},{"osd":1,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.91500000000000004,"15min":0.96799999999999997},"min":{"1min":0.58099999999999996,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.9990000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.61199999999999999},{"interface":"front","average":{"1min":1.034,"5min":1.0229999999999999,"15min":1.069},"min":{"1min":0.52000000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.651,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.002,"15min":1.0509999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.5710000000000002,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":0.82099999999999995},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1240000000000001,"15min":1.1499999999999999},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.7649999999999999,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":0.68600000000000005}]},{"osd":4,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.2410000000000001,"15min":1.248},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":4.0380000000000003,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.337,"5min":1.351,"15min":1.429},"min":{"1min":0.77000000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.3450000000000002,"5min":4.819,"15min":4.819},"last":0.90700000000000003}]},{"osd":5,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.6040000000000001,"15min":1.4470000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.2679999999999998,"5min":30.228000000000002,"15min":30.228000000000002},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.337,"15min":1.357},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":4.3630000000000004,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":0.96399999999999997}]},{"osd":6,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4119999999999999,"15min":1.508},"min":{"1min":0.83999999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.6589999999999998,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4199999999999999,"15min":1.4039999999999999},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.4039999999999999,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":0.93899999999999995}]},{"osd":7,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.389,"15min":1.4199999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.0880000000000001,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.542,"15min":1.5900000000000001},"min":{"1min":0.80500000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.4780000000000002,"5min":10.663,"15min":10.663},"last":0.97799999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542202,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32348,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339492,"statfs":{"total":96636764160,"available":96603639808,"internally_reserved":0,"allocated":1261568,"data_stored":569273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83799999999999997,"15min":0.877},"min":{"1min":0.52100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":1.4259999999999999,"5min":1.456,"15min":1.456},"last":1.1899999999999999},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.86399999999999999,"15min":0.93600000000000005},"min":{"1min":0.39500000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.4690000000000001,"5min":2.226,"15min":2.226},"last":0.751}]},{"osd":1,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.93899999999999995,"15min":0.98499999999999999},"min":{"1min":0.46800000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.377,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.85899999999999999,"15min":0.91000000000000003},"min":{"1min":0.57599999999999996,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.2210000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.159}]},{"osd":3,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.96099999999999997,"15min":1.018},"min":{"1min":0.47499999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.403,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.032,"5min":1.042,"15min":1.0800000000000001},"min":{"1min":0.61099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.92300000000000004}]},{"osd":4,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.194,"15min":1.2310000000000001},"min":{"1min":0.82399999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.7270000000000001,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.157,"15min":1.2070000000000001},"min":{"1min":0.83799999999999997,"5min":0.627,"15min":0.627},"max":{"1min":1.4950000000000001,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.421}]},{"osd":5,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.3100000000000001,"15min":1.367},"min":{"1min":0.76900000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.6259999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.2749999999999999,"15min":1.339},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.819,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.077}]},{"osd":6,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.232,"15min":1.3280000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6479999999999999,"5min":2.456,"15min":2.456},"last":1.256},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3999999999999999,"15min":1.476},"min":{"1min":0.85299999999999998,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.097,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.282}]},{"osd":7,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3069999999999999,"15min":1.369},"min":{"1min":0.79800000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.302,"15min":1.395},"min":{"1min":0.78700000000000003,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5860000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.335}]}]},{"osd":1,"up_from":14,"seq":60129542201,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28252,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343588,"statfs":{"total":96636764160,"available":96607834112,"internally_reserved":0,"allocated":1261568,"data_stored":569273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79100000000000004,"15min":0.86099999999999999},"min":{"1min":0.41599999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.1240000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81799999999999995,"15min":0.82999999999999996},"min":{"1min":0.435,"5min":0.374,"15min":0.374},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.59299999999999997}]},{"osd":2,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.92200000000000004,"15min":0.96399999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.4430000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.92400000000000004,"15min":0.97799999999999998},"min":{"1min":0.496,"5min":0.443,"15min":0.443},"max":{"1min":1.2210000000000001,"5min":1.625,"15min":1.625},"last":0.66500000000000004}]},{"osd":3,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.103,"15min":1.113},"min":{"1min":0.68400000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.6559999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.875,"5min":0.96199999999999997,"15min":1.008},"min":{"1min":0.59499999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.6180000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.89100000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.179,"15min":1.2749999999999999},"min":{"1min":0.67700000000000005,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.5069999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.748},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.246,"15min":1.3049999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6120000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.0569999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.248,"15min":1.3200000000000001},"min":{"1min":0.79500000000000004,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.95,"5min":3.593,"15min":3.593},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.264,"5min":1.327,"15min":1.4259999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.671,"5min":2.359,"15min":2.359},"last":0.69899999999999995}]},{"osd":6,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.3089999999999999,"15min":1.3919999999999999},"min":{"1min":0.84199999999999997,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6200000000000001,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.206},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.288,"15min":1.3540000000000001},"min":{"1min":0.89100000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.0209999999999999,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.0940000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.359,"15min":1.4330000000000001},"min":{"1min":0.99299999999999999,"5min":0.627,"15min":0.627},"max":{"1min":2.0840000000000001,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.242},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.363,"15min":1.486},"min":{"1min":0.76600000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.673,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":0.84399999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542201,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32928,"kb_used_data":1812,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338912,"statfs":{"total":96636764160,"available":96603045888,"internally_reserved":0,"allocated":1855488,"data_stored":1159641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.86399999999999999,"15min":0.93500000000000005},"min":{"1min":0.48399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.478,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.94799999999999995,"15min":0.98199999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.456,"5min":1.506,"15min":1.506},"last":0.61899999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.92700000000000005,"15min":0.99099999999999999},"min":{"1min":0.52900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.4399999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.0349999999999999,"15min":1.095},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.601,"5min":1.901,"15min":1.901},"last":0.55700000000000005}]},{"osd":3,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.93300000000000005,"15min":0.996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.474,"5min":1.972,"15min":1.972},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0669999999999999,"15min":1.147},"min":{"1min":0.60299999999999998,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.597,"5min":2.298,"15min":2.298},"last":0.70699999999999996}]},{"osd":4,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.288,"15min":1.3360000000000001},"min":{"1min":0.72799999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.7090000000000001,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.2230000000000001,"15min":1.27},"min":{"1min":0.80600000000000005,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.8380000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":0.95499999999999996}]},{"osd":5,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3100000000000001,"15min":1.3700000000000001},"min":{"1min":0.89500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.948,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.377,"15min":1.4379999999999999},"min":{"1min":0.83499999999999996,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.8700000000000001,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.036}]},{"osd":6,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.4099999999999999,"15min":1.5209999999999999},"min":{"1min":0.80000000000000004,"5min":0.747,"15min":0.747},"max":{"1min":1.978,"5min":3.032,"15min":3.032},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.411,"15min":1.4870000000000001},"min":{"1min":0.84699999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.1349999999999998,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":0.98099999999999998}]},{"osd":7,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4390000000000001,"15min":1.506},"min":{"1min":0.90500000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1850000000000001,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.099},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3620000000000001,"15min":1.419},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.873,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.077}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:51:27.650 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:51:27.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.895+0000 7fabb0400640 1 Processor -- start 2024-03-23T03:51:27.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.895+0000 7fabb0400640 1 -- start start 2024-03-23T03:51:27.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.897+0000 7fabb0400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faba815b560 0x7faba815b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:27.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.897+0000 7fabb0400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba815f320 con 0x7faba815d410 2024-03-23T03:51:27.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.897+0000 7fabb0400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faba806c6f0 con 0x7faba80b39b0 2024-03-23T03:51:27.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.897+0000 7fabb0400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faba806c9d0 con 0x7faba815b560 2024-03-23T03:51:27.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.897+0000 7fabada00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faba815b560 0x7faba815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:27.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.897+0000 7fabada00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7faba815b560 0x7faba815b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42614/0 (socket says 172.21.15.102:42614) 2024-03-23T03:51:27.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.897+0000 7fabada00640 1 -- 172.21.15.102:0/1390335312 learned_addr learned my addr 172.21.15.102:0/1390335312 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.897+0000 7fabada00640 1 -- 172.21.15.102:0/1390335312 >> v1:172.21.15.173:6789/0 conn(0x7faba80b39b0 legacy=0x7faba815e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.897+0000 7fabada00640 1 -- 172.21.15.102:0/1390335312 >> v1:172.21.15.102:6789/0 conn(0x7faba815d410 legacy=0x7faba815ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.898+0000 7fabada00640 1 -- 172.21.15.102:0/1390335312 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba806ccb0 con 0x7faba815b560 2024-03-23T03:51:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.898+0000 7faba7e00640 1 -- 172.21.15.102:0/1390335312 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623614192 0 0) 0x7faba815f320 con 0x7faba815d410 2024-03-23T03:51:27.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.899+0000 7fabada00640 1 --2- 172.21.15.102:0/1390335312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faba815b560 0x7faba815b940 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fab9c009870 tx=0x7fab9c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad0eed7029d3b114 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:27.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.899+0000 7faba7e00640 1 -- 172.21.15.102:0/1390335312 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c031860 con 0x7faba815b560 2024-03-23T03:51:27.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.899+0000 7faba7e00640 1 -- 172.21.15.102:0/1390335312 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9c0319c0 con 0x7faba815b560 2024-03-23T03:51:27.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.899+0000 7faba7e00640 1 -- 172.21.15.102:0/1390335312 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c031d50 con 0x7faba815b560 2024-03-23T03:51:27.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.900+0000 7fabb0400640 1 -- 172.21.15.102:0/1390335312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faba815b560 msgr2=0x7faba815b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:27.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.900+0000 7fabb0400640 1 --2- 172.21.15.102:0/1390335312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faba815b560 0x7faba815b940 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fab9c009870 tx=0x7fab9c02b900 comp rx=0 tx=0).stop 2024-03-23T03:51:27.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.900+0000 7fabb0400640 1 -- 172.21.15.102:0/1390335312 shutdown_connections 2024-03-23T03:51:27.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.900+0000 7fabb0400640 1 --2- 172.21.15.102:0/1390335312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faba815b560 0x7faba815b940 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.900+0000 7fabb0400640 1 -- 172.21.15.102:0/1390335312 >> 172.21.15.102:0/1390335312 conn(0x7faba806a3d0 msgr2=0x7faba80b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:27.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.900+0000 7fabb0400640 1 -- 172.21.15.102:0/1390335312 shutdown_connections 2024-03-23T03:51:27.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.900+0000 7fabb0400640 1 -- 172.21.15.102:0/1390335312 wait complete. 2024-03-23T03:51:27.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.902+0000 7fabb0400640 1 Processor -- start 2024-03-23T03:51:27.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.902+0000 7fabb0400640 1 -- start start 2024-03-23T03:51:27.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.903+0000 7fabb0400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faba80b39b0 0x7faba814c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:27.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.903+0000 7fabb0400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faba815b560 0x7faba80afaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:27.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.903+0000 7fabb0400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faba815d410 0x7faba8155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:27.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.903+0000 7fabb0400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba8152d00 con 0x7faba80b39b0 2024-03-23T03:51:27.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.903+0000 7fabb0400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faba8150930 con 0x7faba815b560 2024-03-23T03:51:27.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.903+0000 7fabb0400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faba80b1490 con 0x7faba815d410 2024-03-23T03:51:27.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.903+0000 7fabada00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faba80b39b0 0x7faba814c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:27.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.903+0000 7fabada00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faba80b39b0 0x7faba814c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42630/0 (socket says 172.21.15.102:42630) 2024-03-23T03:51:27.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.903+0000 7fabada00640 1 -- 172.21.15.102:0/104823432 learned_addr learned my addr 172.21.15.102:0/104823432 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:27.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.904+0000 7fabada00640 1 -- 172.21.15.102:0/104823432 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faba815d410 msgr2=0x7faba8155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:27.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.904+0000 7fabad000640 1 --2- 172.21.15.102:0/104823432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faba815b560 0x7faba80afaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:27.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.904+0000 7fabae400640 1 --2- 172.21.15.102:0/104823432 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faba815d410 0x7faba8155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:27.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.904+0000 7fabada00640 1 --2- 172.21.15.102:0/104823432 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faba815d410 0x7faba8155540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.904+0000 7fabada00640 1 -- 172.21.15.102:0/104823432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faba815b560 msgr2=0x7faba80afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:27.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.904+0000 7fabada00640 1 --2- 172.21.15.102:0/104823432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faba815b560 0x7faba80afaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:27.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.904+0000 7fabada00640 1 -- 172.21.15.102:0/104823432 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9c0094d0 con 0x7faba80b39b0 2024-03-23T03:51:27.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.904+0000 7fabad000640 1 --2- 172.21.15.102:0/104823432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faba815b560 0x7faba80afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:27.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.904+0000 7fabada00640 1 --2- 172.21.15.102:0/104823432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faba80b39b0 0x7faba814c3d0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fab9c009870 tx=0x7fab9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:27.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.904+0000 7faba5600640 1 -- 172.21.15.102:0/104823432 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c031810 con 0x7faba80b39b0 2024-03-23T03:51:27.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.904+0000 7faba5600640 1 -- 172.21.15.102:0/104823432 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab9c0443f0 con 0x7faba80b39b0 2024-03-23T03:51:27.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.904+0000 7fabb0400640 1 -- 172.21.15.102:0/104823432 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faba8150bd0 con 0x7faba80b39b0 2024-03-23T03:51:27.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.904+0000 7faba5600640 1 -- 172.21.15.102:0/104823432 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab9c0447a0 con 0x7faba80b39b0 2024-03-23T03:51:27.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.904+0000 7fabb0400640 1 -- 172.21.15.102:0/104823432 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faba8151190 con 0x7faba80b39b0 2024-03-23T03:51:27.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.905+0000 7fabb0400640 1 -- 172.21.15.102:0/104823432 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba806ac20 con 0x7faba80b39b0 2024-03-23T03:51:27.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.907+0000 7faba5600640 1 -- 172.21.15.102:0/104823432 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fab9c04d450 con 0x7faba80b39b0 2024-03-23T03:51:27.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.907+0000 7faba5600640 1 --2- 172.21.15.102:0/104823432 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab8806ac00 0x7fab8806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:27.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.908+0000 7faba5600640 1 -- 172.21.15.102:0/104823432 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fab9c0abc00 con 0x7faba80b39b0 2024-03-23T03:51:27.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.908+0000 7fabad000640 1 --2- 172.21.15.102:0/104823432 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab8806ac00 0x7fab8806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:27.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.908+0000 7fabad000640 1 --2- 172.21.15.102:0/104823432 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab8806ac00 0x7fab8806d0c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fab94007fd0 tx=0x7fab94009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:27.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:27.912+0000 7faba5600640 1 -- 172.21.15.102:0/104823432 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fab9c0b2080 con 0x7faba80b39b0 2024-03-23T03:51:28.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.101+0000 7fabb0400640 1 -- 172.21.15.102:0/104823432 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faba806dd80 con 0x7fab8806ac00 2024-03-23T03:51:28.110 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.110+0000 7faba5600640 1 -- 172.21.15.102:0/104823432 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287806 (secure 0 0 0) 0x7fab94001370 con 0x7fab8806ac00 2024-03-23T03:51:28.114 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:28.115 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:51:28.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.115+0000 7fabb0400640 1 -- 172.21.15.102:0/104823432 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab8806ac00 msgr2=0x7fab8806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:28.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.115+0000 7fabb0400640 1 --2- 172.21.15.102:0/104823432 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab8806ac00 0x7fab8806d0c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fab94007fd0 tx=0x7fab94009040 comp rx=0 tx=0).stop 2024-03-23T03:51:28.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.115+0000 7fabb0400640 1 -- 172.21.15.102:0/104823432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faba80b39b0 msgr2=0x7faba814c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:28.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.115+0000 7fabb0400640 1 --2- 172.21.15.102:0/104823432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faba80b39b0 0x7faba814c3d0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fab9c009870 tx=0x7fab9c0046c0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.115+0000 7fabb0400640 1 -- 172.21.15.102:0/104823432 shutdown_connections 2024-03-23T03:51:28.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.115+0000 7fabb0400640 1 --2- 172.21.15.102:0/104823432 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fab8806ac00 0x7fab8806d0c0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.115+0000 7fabb0400640 1 --2- 172.21.15.102:0/104823432 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7faba815d410 0x7faba8155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.115+0000 7fabb0400640 1 --2- 172.21.15.102:0/104823432 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faba815b560 0x7faba80afaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.115+0000 7fabb0400640 1 --2- 172.21.15.102:0/104823432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faba80b39b0 0x7faba814c3d0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.115+0000 7fabb0400640 1 -- 172.21.15.102:0/104823432 >> 172.21.15.102:0/104823432 conn(0x7faba806a3d0 msgr2=0x7faba815a840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:28.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.115+0000 7fabb0400640 1 -- 172.21.15.102:0/104823432 shutdown_connections 2024-03-23T03:51:28.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.115+0000 7fabb0400640 1 -- 172.21.15.102:0/104823432 wait complete. 2024-03-23T03:51:28.136 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-03-23T03:51:27.801978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252144,"kb_used_data":11368,"kb_used_omap":12,"kb_used_meta":240627,"kb_avail":754722576,"statfs":{"total":773094113280,"available":772835917824,"internally_reserved":0,"allocated":11640832,"data_stored":5902124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":246402618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.050539"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.267572+0000","last_change":"2024-03-23T03:51:22.263565+0000","last_active":"2024-03-23T03:51:22.267572+0000","last_peered":"2024-03-23T03:51:22.267572+0000","last_clean":"2024-03-23T03:51:22.267572+0000","last_became_active":"2024-03-23T03:51:22.263323+0000","last_became_peered":"2024-03-23T03:51:22.263323+0000","last_unstale":"2024-03-23T03:51:22.267572+0000","last_undegraded":"2024-03-23T03:51:22.267572+0000","last_fullsized":"2024-03-23T03:51:22.267572+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:24:42.534444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.288428+0000","last_change":"2024-03-23T03:51:22.286197+0000","last_active":"2024-03-23T03:51:22.288428+0000","last_peered":"2024-03-23T03:51:22.288428+0000","last_clean":"2024-03-23T03:51:22.288428+0000","last_became_active":"2024-03-23T03:51:22.280588+0000","last_became_peered":"2024-03-23T03:51:22.280588+0000","last_unstale":"2024-03-23T03:51:22.288428+0000","last_undegraded":"2024-03-23T03:51:22.288428+0000","last_fullsized":"2024-03-23T03:51:22.288428+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:11:35.553745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.286379+0000","last_change":"2024-03-23T03:51:22.281125+0000","last_active":"2024-03-23T03:51:22.286379+0000","last_peered":"2024-03-23T03:51:22.286379+0000","last_clean":"2024-03-23T03:51:22.286379+0000","last_became_active":"2024-03-23T03:51:22.280105+0000","last_became_peered":"2024-03-23T03:51:22.280105+0000","last_unstale":"2024-03-23T03:51:22.286379+0000","last_undegraded":"2024-03-23T03:51:22.286379+0000","last_fullsized":"2024-03-23T03:51:22.286379+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:20:25.913448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.280149+0000","last_change":"2024-03-23T03:51:22.278175+0000","last_active":"2024-03-23T03:51:22.280149+0000","last_peered":"2024-03-23T03:51:22.280149+0000","last_clean":"2024-03-23T03:51:22.280149+0000","last_became_active":"2024-03-23T03:51:22.277895+0000","last_became_peered":"2024-03-23T03:51:22.277895+0000","last_unstale":"2024-03-23T03:51:22.280149+0000","last_undegraded":"2024-03-23T03:51:22.280149+0000","last_fullsized":"2024-03-23T03:51:22.280149+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:36:30.827292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.283501+0000","last_change":"2024-03-23T03:51:22.279802+0000","last_active":"2024-03-23T03:51:22.283501+0000","last_peered":"2024-03-23T03:51:22.283501+0000","last_clean":"2024-03-23T03:51:22.283501+0000","last_became_active":"2024-03-23T03:51:22.278955+0000","last_became_peered":"2024-03-23T03:51:22.278955+0000","last_unstale":"2024-03-23T03:51:22.283501+0000","last_undegraded":"2024-03-23T03:51:22.283501+0000","last_fullsized":"2024-03-23T03:51:22.283501+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:46:33.274045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.287694+0000","last_change":"2024-03-23T03:51:22.279603+0000","last_active":"2024-03-23T03:51:22.287694+0000","last_peered":"2024-03-23T03:51:22.287694+0000","last_clean":"2024-03-23T03:51:22.287694+0000","last_became_active":"2024-03-23T03:51:22.279288+0000","last_became_peered":"2024-03-23T03:51:22.279288+0000","last_unstale":"2024-03-23T03:51:22.287694+0000","last_undegraded":"2024-03-23T03:51:22.287694+0000","last_fullsized":"2024-03-23T03:51:22.287694+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:15:44.577151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.288339+0000","last_change":"2024-03-23T03:51:22.279845+0000","last_active":"2024-03-23T03:51:22.288339+0000","last_peered":"2024-03-23T03:51:22.288339+0000","last_clean":"2024-03-23T03:51:22.288339+0000","last_became_active":"2024-03-23T03:51:22.279530+0000","last_became_peered":"2024-03-23T03:51:22.279530+0000","last_unstale":"2024-03-23T03:51:22.288339+0000","last_undegraded":"2024-03-23T03:51:22.288339+0000","last_fullsized":"2024-03-23T03:51:22.288339+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:19.362177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.274843+0000","last_change":"2024-03-23T03:51:22.267328+0000","last_active":"2024-03-23T03:51:22.274843+0000","last_peered":"2024-03-23T03:51:22.274843+0000","last_clean":"2024-03-23T03:51:22.274843+0000","last_became_active":"2024-03-23T03:51:22.267081+0000","last_became_peered":"2024-03-23T03:51:22.267081+0000","last_unstale":"2024-03-23T03:51:22.274843+0000","last_undegraded":"2024-03-23T03:51:22.274843+0000","last_fullsized":"2024-03-23T03:51:22.274843+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:23:32.227733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.278415+0000","last_change":"2024-03-23T03:51:22.275332+0000","last_active":"2024-03-23T03:51:22.278415+0000","last_peered":"2024-03-23T03:51:22.278415+0000","last_clean":"2024-03-23T03:51:22.278415+0000","last_became_active":"2024-03-23T03:51:22.274890+0000","last_became_peered":"2024-03-23T03:51:22.274890+0000","last_unstale":"2024-03-23T03:51:22.278415+0000","last_undegraded":"2024-03-23T03:51:22.278415+0000","last_fullsized":"2024-03-23T03:51:22.278415+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:54.013665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.278612+0000","last_change":"2024-03-23T03:51:22.273679+0000","last_active":"2024-03-23T03:51:22.278612+0000","last_peered":"2024-03-23T03:51:22.278612+0000","last_clean":"2024-03-23T03:51:22.278612+0000","last_became_active":"2024-03-23T03:51:22.273411+0000","last_became_peered":"2024-03-23T03:51:22.273411+0000","last_unstale":"2024-03-23T03:51:22.278612+0000","last_undegraded":"2024-03-23T03:51:22.278612+0000","last_fullsized":"2024-03-23T03:51:22.278612+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:58:33.997748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.284851+0000","last_change":"2024-03-23T03:51:22.278541+0000","last_active":"2024-03-23T03:51:22.284851+0000","last_peered":"2024-03-23T03:51:22.284851+0000","last_clean":"2024-03-23T03:51:22.284851+0000","last_became_active":"2024-03-23T03:51:22.278274+0000","last_became_peered":"2024-03-23T03:51:22.278274+0000","last_unstale":"2024-03-23T03:51:22.284851+0000","last_undegraded":"2024-03-23T03:51:22.284851+0000","last_fullsized":"2024-03-23T03:51:22.284851+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:10:28.823831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.282176+0000","last_change":"2024-03-23T03:51:22.277885+0000","last_active":"2024-03-23T03:51:22.282176+0000","last_peered":"2024-03-23T03:51:22.282176+0000","last_clean":"2024-03-23T03:51:22.282176+0000","last_became_active":"2024-03-23T03:51:22.277209+0000","last_became_peered":"2024-03-23T03:51:22.277209+0000","last_unstale":"2024-03-23T03:51:22.282176+0000","last_undegraded":"2024-03-23T03:51:22.282176+0000","last_fullsized":"2024-03-23T03:51:22.282176+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:07.649845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.265283+0000","last_change":"2024-03-23T03:51:22.261164+0000","last_active":"2024-03-23T03:51:22.265283+0000","last_peered":"2024-03-23T03:51:22.265283+0000","last_clean":"2024-03-23T03:51:22.265283+0000","last_became_active":"2024-03-23T03:51:22.260899+0000","last_became_peered":"2024-03-23T03:51:22.260899+0000","last_unstale":"2024-03-23T03:51:22.265283+0000","last_undegraded":"2024-03-23T03:51:22.265283+0000","last_fullsized":"2024-03-23T03:51:22.265283+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:35:05.132187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.272780+0000","last_change":"2024-03-23T03:51:22.266968+0000","last_active":"2024-03-23T03:51:22.272780+0000","last_peered":"2024-03-23T03:51:22.272780+0000","last_clean":"2024-03-23T03:51:22.272780+0000","last_became_active":"2024-03-23T03:51:22.266453+0000","last_became_peered":"2024-03-23T03:51:22.266453+0000","last_unstale":"2024-03-23T03:51:22.272780+0000","last_undegraded":"2024-03-23T03:51:22.272780+0000","last_fullsized":"2024-03-23T03:51:22.272780+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:30:04.249708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.281242+0000","last_change":"2024-03-23T03:51:22.271984+0000","last_active":"2024-03-23T03:51:22.281242+0000","last_peered":"2024-03-23T03:51:22.281242+0000","last_clean":"2024-03-23T03:51:22.281242+0000","last_became_active":"2024-03-23T03:51:22.271663+0000","last_became_peered":"2024-03-23T03:51:22.271663+0000","last_unstale":"2024-03-23T03:51:22.281242+0000","last_undegraded":"2024-03-23T03:51:22.281242+0000","last_fullsized":"2024-03-23T03:51:22.281242+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:17:09.963708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.281899+0000","last_change":"2024-03-23T03:51:22.277918+0000","last_active":"2024-03-23T03:51:22.281899+0000","last_peered":"2024-03-23T03:51:22.281899+0000","last_clean":"2024-03-23T03:51:22.281899+0000","last_became_active":"2024-03-23T03:51:22.277642+0000","last_became_peered":"2024-03-23T03:51:22.277642+0000","last_unstale":"2024-03-23T03:51:22.281899+0000","last_undegraded":"2024-03-23T03:51:22.281899+0000","last_fullsized":"2024-03-23T03:51:22.281899+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:04:21.319327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.286384+0000","last_change":"2024-03-23T03:51:22.281023+0000","last_active":"2024-03-23T03:51:22.286384+0000","last_peered":"2024-03-23T03:51:22.286384+0000","last_clean":"2024-03-23T03:51:22.286384+0000","last_became_active":"2024-03-23T03:51:22.280871+0000","last_became_peered":"2024-03-23T03:51:22.280871+0000","last_unstale":"2024-03-23T03:51:22.286384+0000","last_undegraded":"2024-03-23T03:51:22.286384+0000","last_fullsized":"2024-03-23T03:51:22.286384+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:55.608125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.288408+0000","last_change":"2024-03-23T03:51:22.285885+0000","last_active":"2024-03-23T03:51:22.288408+0000","last_peered":"2024-03-23T03:51:22.288408+0000","last_clean":"2024-03-23T03:51:22.288408+0000","last_became_active":"2024-03-23T03:51:22.280190+0000","last_became_peered":"2024-03-23T03:51:22.280190+0000","last_unstale":"2024-03-23T03:51:22.288408+0000","last_undegraded":"2024-03-23T03:51:22.288408+0000","last_fullsized":"2024-03-23T03:51:22.288408+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:03:27.282303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.279574+0000","last_change":"2024-03-23T03:51:22.276946+0000","last_active":"2024-03-23T03:51:22.279574+0000","last_peered":"2024-03-23T03:51:22.279574+0000","last_clean":"2024-03-23T03:51:22.279574+0000","last_became_active":"2024-03-23T03:51:22.276662+0000","last_became_peered":"2024-03-23T03:51:22.276662+0000","last_unstale":"2024-03-23T03:51:22.279574+0000","last_undegraded":"2024-03-23T03:51:22.279574+0000","last_fullsized":"2024-03-23T03:51:22.279574+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:48:26.593847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.282198+0000","last_change":"2024-03-23T03:51:22.279151+0000","last_active":"2024-03-23T03:51:22.282198+0000","last_peered":"2024-03-23T03:51:22.282198+0000","last_clean":"2024-03-23T03:51:22.282198+0000","last_became_active":"2024-03-23T03:51:22.276916+0000","last_became_peered":"2024-03-23T03:51:22.276916+0000","last_unstale":"2024-03-23T03:51:22.282198+0000","last_undegraded":"2024-03-23T03:51:22.282198+0000","last_fullsized":"2024-03-23T03:51:22.282198+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:13:39.429313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.276047+0000","last_change":"2024-03-23T03:51:22.273964+0000","last_active":"2024-03-23T03:51:22.276047+0000","last_peered":"2024-03-23T03:51:22.276047+0000","last_clean":"2024-03-23T03:51:22.276047+0000","last_became_active":"2024-03-23T03:51:22.273497+0000","last_became_peered":"2024-03-23T03:51:22.273497+0000","last_unstale":"2024-03-23T03:51:22.276047+0000","last_undegraded":"2024-03-23T03:51:22.276047+0000","last_fullsized":"2024-03-23T03:51:22.276047+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:20:51.607176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.267969+0000","last_change":"2024-03-23T03:51:22.262943+0000","last_active":"2024-03-23T03:51:22.267969+0000","last_peered":"2024-03-23T03:51:22.267969+0000","last_clean":"2024-03-23T03:51:22.267969+0000","last_became_active":"2024-03-23T03:51:22.262449+0000","last_became_peered":"2024-03-23T03:51:22.262449+0000","last_unstale":"2024-03-23T03:51:22.267969+0000","last_undegraded":"2024-03-23T03:51:22.267969+0000","last_fullsized":"2024-03-23T03:51:22.267969+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:04.683164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.272306+0000","last_change":"2024-03-23T03:51:22.270863+0000","last_active":"2024-03-23T03:51:22.272306+0000","last_peered":"2024-03-23T03:51:22.272306+0000","last_clean":"2024-03-23T03:51:22.272306+0000","last_became_active":"2024-03-23T03:51:22.270601+0000","last_became_peered":"2024-03-23T03:51:22.270601+0000","last_unstale":"2024-03-23T03:51:22.272306+0000","last_undegraded":"2024-03-23T03:51:22.272306+0000","last_fullsized":"2024-03-23T03:51:22.272306+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:46:32.018385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.267987+0000","last_change":"2024-03-23T03:51:22.262151+0000","last_active":"2024-03-23T03:51:22.267987+0000","last_peered":"2024-03-23T03:51:22.267987+0000","last_clean":"2024-03-23T03:51:22.267987+0000","last_became_active":"2024-03-23T03:51:22.261931+0000","last_became_peered":"2024-03-23T03:51:22.261931+0000","last_unstale":"2024-03-23T03:51:22.267987+0000","last_undegraded":"2024-03-23T03:51:22.267987+0000","last_fullsized":"2024-03-23T03:51:22.267987+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:47.178608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.288713+0000","last_change":"2024-03-23T03:51:22.286444+0000","last_active":"2024-03-23T03:51:22.288713+0000","last_peered":"2024-03-23T03:51:22.288713+0000","last_clean":"2024-03-23T03:51:22.288713+0000","last_became_active":"2024-03-23T03:51:22.285470+0000","last_became_peered":"2024-03-23T03:51:22.285470+0000","last_unstale":"2024-03-23T03:51:22.288713+0000","last_undegraded":"2024-03-23T03:51:22.288713+0000","last_fullsized":"2024-03-23T03:51:22.288713+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:58:40.479780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.284517+0000","last_change":"2024-03-23T03:51:22.280518+0000","last_active":"2024-03-23T03:51:22.284517+0000","last_peered":"2024-03-23T03:51:22.284517+0000","last_clean":"2024-03-23T03:51:22.284517+0000","last_became_active":"2024-03-23T03:51:22.280191+0000","last_became_peered":"2024-03-23T03:51:22.280191+0000","last_unstale":"2024-03-23T03:51:22.284517+0000","last_undegraded":"2024-03-23T03:51:22.284517+0000","last_fullsized":"2024-03-23T03:51:22.284517+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:58:05.924179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.283921+0000","last_change":"2024-03-23T03:51:22.272661+0000","last_active":"2024-03-23T03:51:22.283921+0000","last_peered":"2024-03-23T03:51:22.283921+0000","last_clean":"2024-03-23T03:51:22.283921+0000","last_became_active":"2024-03-23T03:51:22.272238+0000","last_became_peered":"2024-03-23T03:51:22.272238+0000","last_unstale":"2024-03-23T03:51:22.283921+0000","last_undegraded":"2024-03-23T03:51:22.283921+0000","last_fullsized":"2024-03-23T03:51:22.283921+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:20.563242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.278988+0000","last_change":"2024-03-23T03:51:22.274596+0000","last_active":"2024-03-23T03:51:22.278988+0000","last_peered":"2024-03-23T03:51:22.278988+0000","last_clean":"2024-03-23T03:51:22.278988+0000","last_became_active":"2024-03-23T03:51:22.274326+0000","last_became_peered":"2024-03-23T03:51:22.274326+0000","last_unstale":"2024-03-23T03:51:22.278988+0000","last_undegraded":"2024-03-23T03:51:22.278988+0000","last_fullsized":"2024-03-23T03:51:22.278988+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:11:23.842491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.275911+0000","last_change":"2024-03-23T03:51:22.273810+0000","last_active":"2024-03-23T03:51:22.275911+0000","last_peered":"2024-03-23T03:51:22.275911+0000","last_clean":"2024-03-23T03:51:22.275911+0000","last_became_active":"2024-03-23T03:51:22.273091+0000","last_became_peered":"2024-03-23T03:51:22.273091+0000","last_unstale":"2024-03-23T03:51:22.275911+0000","last_undegraded":"2024-03-23T03:51:22.275911+0000","last_fullsized":"2024-03-23T03:51:22.275911+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:46.517427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.287367+0000","last_change":"2024-03-23T03:51:22.283425+0000","last_active":"2024-03-23T03:51:22.287367+0000","last_peered":"2024-03-23T03:51:22.287367+0000","last_clean":"2024-03-23T03:51:22.287367+0000","last_became_active":"2024-03-23T03:51:22.283140+0000","last_became_peered":"2024-03-23T03:51:22.283140+0000","last_unstale":"2024-03-23T03:51:22.287367+0000","last_undegraded":"2024-03-23T03:51:22.287367+0000","last_fullsized":"2024-03-23T03:51:22.287367+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:36:38.929407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.274309+0000","last_change":"2024-03-23T03:51:22.269314+0000","last_active":"2024-03-23T03:51:22.274309+0000","last_peered":"2024-03-23T03:51:22.274309+0000","last_clean":"2024-03-23T03:51:22.274309+0000","last_became_active":"2024-03-23T03:51:22.268912+0000","last_became_peered":"2024-03-23T03:51:22.268912+0000","last_unstale":"2024-03-23T03:51:22.274309+0000","last_undegraded":"2024-03-23T03:51:22.274309+0000","last_fullsized":"2024-03-23T03:51:22.274309+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:25:25.635704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.286480+0000","last_change":"2024-03-23T03:51:22.282218+0000","last_active":"2024-03-23T03:51:22.286480+0000","last_peered":"2024-03-23T03:51:22.286480+0000","last_clean":"2024-03-23T03:51:22.286480+0000","last_became_active":"2024-03-23T03:51:22.281934+0000","last_became_peered":"2024-03-23T03:51:22.281934+0000","last_unstale":"2024-03-23T03:51:22.286480+0000","last_undegraded":"2024-03-23T03:51:22.286480+0000","last_fullsized":"2024-03-23T03:51:22.286480+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:19:58.455715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.279440+0000","last_change":"2024-03-23T03:51:22.276935+0000","last_active":"2024-03-23T03:51:22.279440+0000","last_peered":"2024-03-23T03:51:22.279440+0000","last_clean":"2024-03-23T03:51:22.279440+0000","last_became_active":"2024-03-23T03:51:22.276687+0000","last_became_peered":"2024-03-23T03:51:22.276687+0000","last_unstale":"2024-03-23T03:51:22.279440+0000","last_undegraded":"2024-03-23T03:51:22.279440+0000","last_fullsized":"2024-03-23T03:51:22.279440+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:57:56.460821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"104'4","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.274281+0000","last_change":"2024-03-23T03:51:21.789123+0000","last_active":"2024-03-23T03:51:22.274281+0000","last_peered":"2024-03-23T03:51:22.274281+0000","last_clean":"2024-03-23T03:51:22.274281+0000","last_became_active":"2024-03-23T03:51:21.788839+0000","last_became_peered":"2024-03-23T03:51:21.788839+0000","last_unstale":"2024-03-23T03:51:22.274281+0000","last_undegraded":"2024-03-23T03:51:22.274281+0000","last_fullsized":"2024-03-23T03:51:22.274281+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:54:15.206699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.261655+0000","last_change":"2024-03-23T03:51:22.258376+0000","last_active":"2024-03-23T03:51:22.261655+0000","last_peered":"2024-03-23T03:51:22.261655+0000","last_clean":"2024-03-23T03:51:22.261655+0000","last_became_active":"2024-03-23T03:51:22.257856+0000","last_became_peered":"2024-03-23T03:51:22.257856+0000","last_unstale":"2024-03-23T03:51:22.261655+0000","last_undegraded":"2024-03-23T03:51:22.261655+0000","last_fullsized":"2024-03-23T03:51:22.261655+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:04.438866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"104'1","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.276956+0000","last_change":"2024-03-23T03:51:21.785555+0000","last_active":"2024-03-23T03:51:22.276956+0000","last_peered":"2024-03-23T03:51:22.276956+0000","last_clean":"2024-03-23T03:51:22.276956+0000","last_became_active":"2024-03-23T03:51:21.785232+0000","last_became_peered":"2024-03-23T03:51:21.785232+0000","last_unstale":"2024-03-23T03:51:22.276956+0000","last_undegraded":"2024-03-23T03:51:22.276956+0000","last_fullsized":"2024-03-23T03:51:22.276956+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:27:33.193844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.282442+0000","last_change":"2024-03-23T03:51:22.278002+0000","last_active":"2024-03-23T03:51:22.282442+0000","last_peered":"2024-03-23T03:51:22.282442+0000","last_clean":"2024-03-23T03:51:22.282442+0000","last_became_active":"2024-03-23T03:51:22.277586+0000","last_became_peered":"2024-03-23T03:51:22.277586+0000","last_unstale":"2024-03-23T03:51:22.282442+0000","last_undegraded":"2024-03-23T03:51:22.282442+0000","last_fullsized":"2024-03-23T03:51:22.282442+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:27:17.201821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.285705+0000","last_change":"2024-03-23T03:51:21.783666+0000","last_active":"2024-03-23T03:51:22.285705+0000","last_peered":"2024-03-23T03:51:22.285705+0000","last_clean":"2024-03-23T03:51:22.285705+0000","last_became_active":"2024-03-23T03:51:21.783377+0000","last_became_peered":"2024-03-23T03:51:21.783377+0000","last_unstale":"2024-03-23T03:51:22.285705+0000","last_undegraded":"2024-03-23T03:51:22.285705+0000","last_fullsized":"2024-03-23T03:51:22.285705+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:42:08.794820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.287243+0000","last_change":"2024-03-23T03:51:22.284472+0000","last_active":"2024-03-23T03:51:22.287243+0000","last_peered":"2024-03-23T03:51:22.287243+0000","last_clean":"2024-03-23T03:51:22.287243+0000","last_became_active":"2024-03-23T03:51:22.284222+0000","last_became_peered":"2024-03-23T03:51:22.284222+0000","last_unstale":"2024-03-23T03:51:22.287243+0000","last_undegraded":"2024-03-23T03:51:22.287243+0000","last_fullsized":"2024-03-23T03:51:22.287243+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:17:09.963708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.253628+0000","last_change":"2024-03-23T03:51:21.792017+0000","last_active":"2024-03-23T03:51:22.253628+0000","last_peered":"2024-03-23T03:51:22.253628+0000","last_clean":"2024-03-23T03:51:22.253628+0000","last_became_active":"2024-03-23T03:51:21.791763+0000","last_became_peered":"2024-03-23T03:51:21.791763+0000","last_unstale":"2024-03-23T03:51:22.253628+0000","last_undegraded":"2024-03-23T03:51:22.253628+0000","last_fullsized":"2024-03-23T03:51:22.253628+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:05:09.454442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.260943+0000","last_change":"2024-03-23T03:51:22.255429+0000","last_active":"2024-03-23T03:51:22.260943+0000","last_peered":"2024-03-23T03:51:22.260943+0000","last_clean":"2024-03-23T03:51:22.260943+0000","last_became_active":"2024-03-23T03:51:22.253694+0000","last_became_peered":"2024-03-23T03:51:22.253694+0000","last_unstale":"2024-03-23T03:51:22.260943+0000","last_undegraded":"2024-03-23T03:51:22.260943+0000","last_fullsized":"2024-03-23T03:51:22.260943+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:38:27.787294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"104'1","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.279394+0000","last_change":"2024-03-23T03:51:21.792106+0000","last_active":"2024-03-23T03:51:22.279394+0000","last_peered":"2024-03-23T03:51:22.279394+0000","last_clean":"2024-03-23T03:51:22.279394+0000","last_became_active":"2024-03-23T03:51:21.791295+0000","last_became_peered":"2024-03-23T03:51:21.791295+0000","last_unstale":"2024-03-23T03:51:22.279394+0000","last_undegraded":"2024-03-23T03:51:22.279394+0000","last_fullsized":"2024-03-23T03:51:22.279394+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:00:35.520733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.279590+0000","last_change":"2024-03-23T03:51:22.277153+0000","last_active":"2024-03-23T03:51:22.279590+0000","last_peered":"2024-03-23T03:51:22.279590+0000","last_clean":"2024-03-23T03:51:22.279590+0000","last_became_active":"2024-03-23T03:51:22.276655+0000","last_became_peered":"2024-03-23T03:51:22.276655+0000","last_unstale":"2024-03-23T03:51:22.279590+0000","last_undegraded":"2024-03-23T03:51:22.279590+0000","last_fullsized":"2024-03-23T03:51:22.279590+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:37:27.534087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"104'1","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.284168+0000","last_change":"2024-03-23T03:51:21.780198+0000","last_active":"2024-03-23T03:51:22.284168+0000","last_peered":"2024-03-23T03:51:22.284168+0000","last_clean":"2024-03-23T03:51:22.284168+0000","last_became_active":"2024-03-23T03:51:21.779256+0000","last_became_peered":"2024-03-23T03:51:21.779256+0000","last_unstale":"2024-03-23T03:51:22.284168+0000","last_undegraded":"2024-03-23T03:51:22.284168+0000","last_fullsized":"2024-03-23T03:51:22.284168+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:00:33.834214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.276291+0000","last_change":"2024-03-23T03:51:22.273524+0000","last_active":"2024-03-23T03:51:22.276291+0000","last_peered":"2024-03-23T03:51:22.276291+0000","last_clean":"2024-03-23T03:51:22.276291+0000","last_became_active":"2024-03-23T03:51:22.270458+0000","last_became_peered":"2024-03-23T03:51:22.270458+0000","last_unstale":"2024-03-23T03:51:22.276291+0000","last_undegraded":"2024-03-23T03:51:22.276291+0000","last_fullsized":"2024-03-23T03:51:22.276291+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:50:44.800098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.269859+0000","last_change":"2024-03-23T03:51:21.783223+0000","last_active":"2024-03-23T03:51:22.269859+0000","last_peered":"2024-03-23T03:51:22.269859+0000","last_clean":"2024-03-23T03:51:22.269859+0000","last_became_active":"2024-03-23T03:51:21.782799+0000","last_became_peered":"2024-03-23T03:51:21.782799+0000","last_unstale":"2024-03-23T03:51:22.269859+0000","last_undegraded":"2024-03-23T03:51:22.269859+0000","last_fullsized":"2024-03-23T03:51:22.269859+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:56:44.890019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.272586+0000","last_change":"2024-03-23T03:51:22.263756+0000","last_active":"2024-03-23T03:51:22.272586+0000","last_peered":"2024-03-23T03:51:22.272586+0000","last_clean":"2024-03-23T03:51:22.272586+0000","last_became_active":"2024-03-23T03:51:22.263510+0000","last_became_peered":"2024-03-23T03:51:22.263510+0000","last_unstale":"2024-03-23T03:51:22.272586+0000","last_undegraded":"2024-03-23T03:51:22.272586+0000","last_fullsized":"2024-03-23T03:51:22.272586+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:31:57.940022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.246134+0000","last_change":"2024-03-23T03:51:21.776001+0000","last_active":"2024-03-23T03:51:22.246134+0000","last_peered":"2024-03-23T03:51:22.246134+0000","last_clean":"2024-03-23T03:51:22.246134+0000","last_became_active":"2024-03-23T03:51:21.775726+0000","last_became_peered":"2024-03-23T03:51:21.775726+0000","last_unstale":"2024-03-23T03:51:22.246134+0000","last_undegraded":"2024-03-23T03:51:22.246134+0000","last_fullsized":"2024-03-23T03:51:22.246134+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:07:52.649065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.279542+0000","last_change":"2024-03-23T03:51:22.277230+0000","last_active":"2024-03-23T03:51:22.279542+0000","last_peered":"2024-03-23T03:51:22.279542+0000","last_clean":"2024-03-23T03:51:22.279542+0000","last_became_active":"2024-03-23T03:51:22.276977+0000","last_became_peered":"2024-03-23T03:51:22.276977+0000","last_unstale":"2024-03-23T03:51:22.279542+0000","last_undegraded":"2024-03-23T03:51:22.279542+0000","last_fullsized":"2024-03-23T03:51:22.279542+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:55:30.397347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.273372+0000","last_change":"2024-03-23T03:51:21.786526+0000","last_active":"2024-03-23T03:51:22.273372+0000","last_peered":"2024-03-23T03:51:22.273372+0000","last_clean":"2024-03-23T03:51:22.273372+0000","last_became_active":"2024-03-23T03:51:21.783040+0000","last_became_peered":"2024-03-23T03:51:21.783040+0000","last_unstale":"2024-03-23T03:51:22.273372+0000","last_undegraded":"2024-03-23T03:51:22.273372+0000","last_fullsized":"2024-03-23T03:51:22.273372+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:53.003075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.264940+0000","last_change":"2024-03-23T03:51:22.258821+0000","last_active":"2024-03-23T03:51:22.264940+0000","last_peered":"2024-03-23T03:51:22.264940+0000","last_clean":"2024-03-23T03:51:22.264940+0000","last_became_active":"2024-03-23T03:51:22.258581+0000","last_became_peered":"2024-03-23T03:51:22.258581+0000","last_unstale":"2024-03-23T03:51:22.264940+0000","last_undegraded":"2024-03-23T03:51:22.264940+0000","last_fullsized":"2024-03-23T03:51:22.264940+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:42:06.329388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.259334+0000","last_change":"2024-03-23T03:51:21.787895+0000","last_active":"2024-03-23T03:51:22.259334+0000","last_peered":"2024-03-23T03:51:22.259334+0000","last_clean":"2024-03-23T03:51:22.259334+0000","last_became_active":"2024-03-23T03:51:21.787627+0000","last_became_peered":"2024-03-23T03:51:21.787627+0000","last_unstale":"2024-03-23T03:51:22.259334+0000","last_undegraded":"2024-03-23T03:51:22.259334+0000","last_fullsized":"2024-03-23T03:51:22.259334+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:31:02.266262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.269389+0000","last_change":"2024-03-23T03:51:22.266962+0000","last_active":"2024-03-23T03:51:22.269389+0000","last_peered":"2024-03-23T03:51:22.269389+0000","last_clean":"2024-03-23T03:51:22.269389+0000","last_became_active":"2024-03-23T03:51:22.266486+0000","last_became_peered":"2024-03-23T03:51:22.266486+0000","last_unstale":"2024-03-23T03:51:22.269389+0000","last_undegraded":"2024-03-23T03:51:22.269389+0000","last_fullsized":"2024-03-23T03:51:22.269389+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:34:16.439779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"104'3","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.275377+0000","last_change":"2024-03-23T03:51:21.786687+0000","last_active":"2024-03-23T03:51:22.275377+0000","last_peered":"2024-03-23T03:51:22.275377+0000","last_clean":"2024-03-23T03:51:22.275377+0000","last_became_active":"2024-03-23T03:51:21.786374+0000","last_became_peered":"2024-03-23T03:51:21.786374+0000","last_unstale":"2024-03-23T03:51:22.275377+0000","last_undegraded":"2024-03-23T03:51:22.275377+0000","last_fullsized":"2024-03-23T03:51:22.275377+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:09:59.406767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.265284+0000","last_change":"2024-03-23T03:51:22.259503+0000","last_active":"2024-03-23T03:51:22.265284+0000","last_peered":"2024-03-23T03:51:22.265284+0000","last_clean":"2024-03-23T03:51:22.265284+0000","last_became_active":"2024-03-23T03:51:22.256943+0000","last_became_peered":"2024-03-23T03:51:22.256943+0000","last_unstale":"2024-03-23T03:51:22.265284+0000","last_undegraded":"2024-03-23T03:51:22.265284+0000","last_fullsized":"2024-03-23T03:51:22.265284+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:19:44.510925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"104'6","reported_seq":17,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.289168+0000","last_change":"2024-03-23T03:51:21.789690+0000","last_active":"2024-03-23T03:51:22.289168+0000","last_peered":"2024-03-23T03:51:22.289168+0000","last_clean":"2024-03-23T03:51:22.289168+0000","last_became_active":"2024-03-23T03:51:21.789409+0000","last_became_peered":"2024-03-23T03:51:21.789409+0000","last_unstale":"2024-03-23T03:51:22.289168+0000","last_undegraded":"2024-03-23T03:51:22.289168+0000","last_fullsized":"2024-03-23T03:51:22.289168+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:03:22.934141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.283171+0000","last_change":"2024-03-23T03:51:22.271629+0000","last_active":"2024-03-23T03:51:22.283171+0000","last_peered":"2024-03-23T03:51:22.283171+0000","last_clean":"2024-03-23T03:51:22.283171+0000","last_became_active":"2024-03-23T03:51:22.271350+0000","last_became_peered":"2024-03-23T03:51:22.271350+0000","last_unstale":"2024-03-23T03:51:22.283171+0000","last_undegraded":"2024-03-23T03:51:22.283171+0000","last_fullsized":"2024-03-23T03:51:22.283171+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:04:22.944014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.256406+0000","last_change":"2024-03-23T03:51:21.778247+0000","last_active":"2024-03-23T03:51:22.256406+0000","last_peered":"2024-03-23T03:51:22.256406+0000","last_clean":"2024-03-23T03:51:22.256406+0000","last_became_active":"2024-03-23T03:51:21.777903+0000","last_became_peered":"2024-03-23T03:51:21.777903+0000","last_unstale":"2024-03-23T03:51:22.256406+0000","last_undegraded":"2024-03-23T03:51:22.256406+0000","last_fullsized":"2024-03-23T03:51:22.256406+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:30:03.239118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.274568+0000","last_change":"2024-03-23T03:51:22.265915+0000","last_active":"2024-03-23T03:51:22.274568+0000","last_peered":"2024-03-23T03:51:22.274568+0000","last_clean":"2024-03-23T03:51:22.274568+0000","last_became_active":"2024-03-23T03:51:22.265640+0000","last_became_peered":"2024-03-23T03:51:22.265640+0000","last_unstale":"2024-03-23T03:51:22.274568+0000","last_undegraded":"2024-03-23T03:51:22.274568+0000","last_fullsized":"2024-03-23T03:51:22.274568+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:06:14.520077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.246653+0000","last_change":"2024-03-23T03:51:21.780251+0000","last_active":"2024-03-23T03:51:22.246653+0000","last_peered":"2024-03-23T03:51:22.246653+0000","last_clean":"2024-03-23T03:51:22.246653+0000","last_became_active":"2024-03-23T03:51:21.779927+0000","last_became_peered":"2024-03-23T03:51:21.779927+0000","last_unstale":"2024-03-23T03:51:22.246653+0000","last_undegraded":"2024-03-23T03:51:22.246653+0000","last_fullsized":"2024-03-23T03:51:22.246653+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:28:51.270476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.288871+0000","last_change":"2024-03-23T03:51:22.285115+0000","last_active":"2024-03-23T03:51:22.288871+0000","last_peered":"2024-03-23T03:51:22.288871+0000","last_clean":"2024-03-23T03:51:22.288871+0000","last_became_active":"2024-03-23T03:51:22.284771+0000","last_became_peered":"2024-03-23T03:51:22.284771+0000","last_unstale":"2024-03-23T03:51:22.288871+0000","last_undegraded":"2024-03-23T03:51:22.288871+0000","last_fullsized":"2024-03-23T03:51:22.288871+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:53.513182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.249815+0000","last_change":"2024-03-23T03:51:21.790264+0000","last_active":"2024-03-23T03:51:22.249815+0000","last_peered":"2024-03-23T03:51:22.249815+0000","last_clean":"2024-03-23T03:51:22.249815+0000","last_became_active":"2024-03-23T03:51:21.789990+0000","last_became_peered":"2024-03-23T03:51:21.789990+0000","last_unstale":"2024-03-23T03:51:22.249815+0000","last_undegraded":"2024-03-23T03:51:22.249815+0000","last_fullsized":"2024-03-23T03:51:22.249815+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:17:42.968056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.286366+0000","last_change":"2024-03-23T03:51:22.280933+0000","last_active":"2024-03-23T03:51:22.286366+0000","last_peered":"2024-03-23T03:51:22.286366+0000","last_clean":"2024-03-23T03:51:22.286366+0000","last_became_active":"2024-03-23T03:51:22.280679+0000","last_became_peered":"2024-03-23T03:51:22.280679+0000","last_unstale":"2024-03-23T03:51:22.286366+0000","last_undegraded":"2024-03-23T03:51:22.286366+0000","last_fullsized":"2024-03-23T03:51:22.286366+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:52:35.913771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"104'8","reported_seq":19,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.287974+0000","last_change":"2024-03-23T03:51:21.783083+0000","last_active":"2024-03-23T03:51:22.287974+0000","last_peered":"2024-03-23T03:51:22.287974+0000","last_clean":"2024-03-23T03:51:22.287974+0000","last_became_active":"2024-03-23T03:51:21.782771+0000","last_became_peered":"2024-03-23T03:51:21.782771+0000","last_unstale":"2024-03-23T03:51:22.287974+0000","last_undegraded":"2024-03-23T03:51:22.287974+0000","last_fullsized":"2024-03-23T03:51:22.287974+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:11:22.831901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.274824+0000","last_change":"2024-03-23T03:51:22.263700+0000","last_active":"2024-03-23T03:51:22.274824+0000","last_peered":"2024-03-23T03:51:22.274824+0000","last_clean":"2024-03-23T03:51:22.274824+0000","last_became_active":"2024-03-23T03:51:22.263452+0000","last_became_peered":"2024-03-23T03:51:22.263452+0000","last_unstale":"2024-03-23T03:51:22.274824+0000","last_undegraded":"2024-03-23T03:51:22.274824+0000","last_fullsized":"2024-03-23T03:51:22.274824+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:35:48.145231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"104'1","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.286170+0000","last_change":"2024-03-23T03:51:21.793220+0000","last_active":"2024-03-23T03:51:22.286170+0000","last_peered":"2024-03-23T03:51:22.286170+0000","last_clean":"2024-03-23T03:51:22.286170+0000","last_became_active":"2024-03-23T03:51:21.792937+0000","last_became_peered":"2024-03-23T03:51:21.792937+0000","last_unstale":"2024-03-23T03:51:22.286170+0000","last_undegraded":"2024-03-23T03:51:22.286170+0000","last_fullsized":"2024-03-23T03:51:22.286170+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:52.502591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.276080+0000","last_change":"2024-03-23T03:51:22.271578+0000","last_active":"2024-03-23T03:51:22.276080+0000","last_peered":"2024-03-23T03:51:22.276080+0000","last_clean":"2024-03-23T03:51:22.276080+0000","last_became_active":"2024-03-23T03:51:22.271281+0000","last_became_peered":"2024-03-23T03:51:22.271281+0000","last_unstale":"2024-03-23T03:51:22.276080+0000","last_undegraded":"2024-03-23T03:51:22.276080+0000","last_fullsized":"2024-03-23T03:51:22.276080+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:31:10.713669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"104'1","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.277479+0000","last_change":"2024-03-23T03:51:21.792048+0000","last_active":"2024-03-23T03:51:22.277479+0000","last_peered":"2024-03-23T03:51:22.277479+0000","last_clean":"2024-03-23T03:51:22.277479+0000","last_became_active":"2024-03-23T03:51:21.791713+0000","last_became_peered":"2024-03-23T03:51:21.791713+0000","last_unstale":"2024-03-23T03:51:22.277479+0000","last_undegraded":"2024-03-23T03:51:22.277479+0000","last_fullsized":"2024-03-23T03:51:22.277479+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:59:01.379564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.278624+0000","last_change":"2024-03-23T03:51:22.269905+0000","last_active":"2024-03-23T03:51:22.278624+0000","last_peered":"2024-03-23T03:51:22.278624+0000","last_clean":"2024-03-23T03:51:22.278624+0000","last_became_active":"2024-03-23T03:51:22.269683+0000","last_became_peered":"2024-03-23T03:51:22.269683+0000","last_unstale":"2024-03-23T03:51:22.278624+0000","last_undegraded":"2024-03-23T03:51:22.278624+0000","last_fullsized":"2024-03-23T03:51:22.278624+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:59:59.276905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.249880+0000","last_change":"2024-03-23T03:51:21.789004+0000","last_active":"2024-03-23T03:51:22.249880+0000","last_peered":"2024-03-23T03:51:22.249880+0000","last_clean":"2024-03-23T03:51:22.249880+0000","last_became_active":"2024-03-23T03:51:21.788647+0000","last_became_peered":"2024-03-23T03:51:21.788647+0000","last_unstale":"2024-03-23T03:51:22.249880+0000","last_undegraded":"2024-03-23T03:51:22.249880+0000","last_fullsized":"2024-03-23T03:51:22.249880+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:59:01.379564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.267588+0000","last_change":"2024-03-23T03:51:22.263556+0000","last_active":"2024-03-23T03:51:22.267588+0000","last_peered":"2024-03-23T03:51:22.267588+0000","last_clean":"2024-03-23T03:51:22.267588+0000","last_became_active":"2024-03-23T03:51:22.263345+0000","last_became_peered":"2024-03-23T03:51:22.263345+0000","last_unstale":"2024-03-23T03:51:22.267588+0000","last_undegraded":"2024-03-23T03:51:22.267588+0000","last_fullsized":"2024-03-23T03:51:22.267588+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:15:49.588458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.263002+0000","last_change":"2024-03-23T03:51:21.782473+0000","last_active":"2024-03-23T03:51:22.263002+0000","last_peered":"2024-03-23T03:51:22.263002+0000","last_clean":"2024-03-23T03:51:22.263002+0000","last_became_active":"2024-03-23T03:51:21.781740+0000","last_became_peered":"2024-03-23T03:51:21.781740+0000","last_unstale":"2024-03-23T03:51:22.263002+0000","last_undegraded":"2024-03-23T03:51:22.263002+0000","last_fullsized":"2024-03-23T03:51:22.263002+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:30:22.629109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.270955+0000","last_change":"2024-03-23T03:51:22.269215+0000","last_active":"2024-03-23T03:51:22.270955+0000","last_peered":"2024-03-23T03:51:22.270955+0000","last_clean":"2024-03-23T03:51:22.270955+0000","last_became_active":"2024-03-23T03:51:22.268957+0000","last_became_peered":"2024-03-23T03:51:22.268957+0000","last_unstale":"2024-03-23T03:51:22.270955+0000","last_undegraded":"2024-03-23T03:51:22.270955+0000","last_fullsized":"2024-03-23T03:51:22.270955+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:18:12.413501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.250313+0000","last_change":"2024-03-23T03:51:21.782262+0000","last_active":"2024-03-23T03:51:22.250313+0000","last_peered":"2024-03-23T03:51:22.250313+0000","last_clean":"2024-03-23T03:51:22.250313+0000","last_became_active":"2024-03-23T03:51:21.781940+0000","last_became_peered":"2024-03-23T03:51:21.781940+0000","last_unstale":"2024-03-23T03:51:22.250313+0000","last_undegraded":"2024-03-23T03:51:22.250313+0000","last_fullsized":"2024-03-23T03:51:22.250313+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:49.654183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.275604+0000","last_change":"2024-03-23T03:51:22.273117+0000","last_active":"2024-03-23T03:51:22.275604+0000","last_peered":"2024-03-23T03:51:22.275604+0000","last_clean":"2024-03-23T03:51:22.275604+0000","last_became_active":"2024-03-23T03:51:22.272861+0000","last_became_peered":"2024-03-23T03:51:22.272861+0000","last_unstale":"2024-03-23T03:51:22.275604+0000","last_undegraded":"2024-03-23T03:51:22.275604+0000","last_fullsized":"2024-03-23T03:51:22.275604+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:06:50.564546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.263230+0000","last_change":"2024-03-23T03:51:21.792924+0000","last_active":"2024-03-23T03:51:22.263230+0000","last_peered":"2024-03-23T03:51:22.263230+0000","last_clean":"2024-03-23T03:51:22.263230+0000","last_became_active":"2024-03-23T03:51:21.790283+0000","last_became_peered":"2024-03-23T03:51:21.790283+0000","last_unstale":"2024-03-23T03:51:22.263230+0000","last_undegraded":"2024-03-23T03:51:22.263230+0000","last_fullsized":"2024-03-23T03:51:22.263230+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:19:57.445124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.269326+0000","last_change":"2024-03-23T03:51:22.265543+0000","last_active":"2024-03-23T03:51:22.269326+0000","last_peered":"2024-03-23T03:51:22.269326+0000","last_clean":"2024-03-23T03:51:22.269326+0000","last_became_active":"2024-03-23T03:51:22.265310+0000","last_became_peered":"2024-03-23T03:51:22.265310+0000","last_unstale":"2024-03-23T03:51:22.269326+0000","last_undegraded":"2024-03-23T03:51:22.269326+0000","last_fullsized":"2024-03-23T03:51:22.269326+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:35.027397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.249976+0000","last_change":"2024-03-23T03:51:21.784479+0000","last_active":"2024-03-23T03:51:22.249976+0000","last_peered":"2024-03-23T03:51:22.249976+0000","last_clean":"2024-03-23T03:51:22.249976+0000","last_became_active":"2024-03-23T03:51:21.784166+0000","last_became_peered":"2024-03-23T03:51:21.784166+0000","last_unstale":"2024-03-23T03:51:22.249976+0000","last_undegraded":"2024-03-23T03:51:22.249976+0000","last_fullsized":"2024-03-23T03:51:22.249976+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:35:04.474544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.267498+0000","last_change":"2024-03-23T03:51:22.263572+0000","last_active":"2024-03-23T03:51:22.267498+0000","last_peered":"2024-03-23T03:51:22.267498+0000","last_clean":"2024-03-23T03:51:22.267498+0000","last_became_active":"2024-03-23T03:51:22.263304+0000","last_became_peered":"2024-03-23T03:51:22.263304+0000","last_unstale":"2024-03-23T03:51:22.267498+0000","last_undegraded":"2024-03-23T03:51:22.267498+0000","last_fullsized":"2024-03-23T03:51:22.267498+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:08:03.409470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.256089+0000","last_change":"2024-03-23T03:51:21.776941+0000","last_active":"2024-03-23T03:51:22.256089+0000","last_peered":"2024-03-23T03:51:22.256089+0000","last_clean":"2024-03-23T03:51:22.256089+0000","last_became_active":"2024-03-23T03:51:21.776643+0000","last_became_peered":"2024-03-23T03:51:21.776643+0000","last_unstale":"2024-03-23T03:51:22.256089+0000","last_undegraded":"2024-03-23T03:51:22.256089+0000","last_fullsized":"2024-03-23T03:51:22.256089+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:17:08.953118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.282153+0000","last_change":"2024-03-23T03:51:22.278810+0000","last_active":"2024-03-23T03:51:22.282153+0000","last_peered":"2024-03-23T03:51:22.282153+0000","last_clean":"2024-03-23T03:51:22.282153+0000","last_became_active":"2024-03-23T03:51:22.278506+0000","last_became_peered":"2024-03-23T03:51:22.278506+0000","last_unstale":"2024-03-23T03:51:22.282153+0000","last_undegraded":"2024-03-23T03:51:22.282153+0000","last_fullsized":"2024-03-23T03:51:22.282153+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:46:03.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"105'1","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T03:51:23.268762+0000","last_change":"2024-03-23T03:51:21.785341+0000","last_active":"2024-03-23T03:51:23.268762+0000","last_peered":"2024-03-23T03:51:23.268762+0000","last_clean":"2024-03-23T03:51:23.268762+0000","last_became_active":"2024-03-23T03:51:21.784654+0000","last_became_peered":"2024-03-23T03:51:21.784654+0000","last_unstale":"2024-03-23T03:51:23.268762+0000","last_undegraded":"2024-03-23T03:51:23.268762+0000","last_fullsized":"2024-03-23T03:51:23.268762+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:36.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.272683+0000","last_change":"2024-03-23T03:51:22.265659+0000","last_active":"2024-03-23T03:51:22.272683+0000","last_peered":"2024-03-23T03:51:22.272683+0000","last_clean":"2024-03-23T03:51:22.272683+0000","last_became_active":"2024-03-23T03:51:22.265380+0000","last_became_peered":"2024-03-23T03:51:22.265380+0000","last_unstale":"2024-03-23T03:51:22.272683+0000","last_undegraded":"2024-03-23T03:51:22.272683+0000","last_fullsized":"2024-03-23T03:51:22.272683+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:27:15.422967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.257431+0000","last_change":"2024-03-23T03:51:21.788631+0000","last_active":"2024-03-23T03:51:22.257431+0000","last_peered":"2024-03-23T03:51:22.257431+0000","last_clean":"2024-03-23T03:51:22.257431+0000","last_became_active":"2024-03-23T03:51:21.788145+0000","last_became_peered":"2024-03-23T03:51:21.788145+0000","last_unstale":"2024-03-23T03:51:22.257431+0000","last_undegraded":"2024-03-23T03:51:22.257431+0000","last_fullsized":"2024-03-23T03:51:22.257431+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:45.506837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.265867+0000","last_change":"2024-03-23T03:51:22.259619+0000","last_active":"2024-03-23T03:51:22.265867+0000","last_peered":"2024-03-23T03:51:22.265867+0000","last_clean":"2024-03-23T03:51:22.265867+0000","last_became_active":"2024-03-23T03:51:22.259270+0000","last_became_peered":"2024-03-23T03:51:22.259270+0000","last_unstale":"2024-03-23T03:51:22.265867+0000","last_undegraded":"2024-03-23T03:51:22.265867+0000","last_fullsized":"2024-03-23T03:51:22.265867+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:30:04.249708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.241639+0000","last_change":"2024-03-23T03:51:21.788546+0000","last_active":"2024-03-23T03:51:22.241639+0000","last_peered":"2024-03-23T03:51:22.241639+0000","last_clean":"2024-03-23T03:51:22.241639+0000","last_became_active":"2024-03-23T03:51:21.788282+0000","last_became_peered":"2024-03-23T03:51:21.788282+0000","last_unstale":"2024-03-23T03:51:22.241639+0000","last_undegraded":"2024-03-23T03:51:22.241639+0000","last_fullsized":"2024-03-23T03:51:22.241639+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:04:20.308736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.261946+0000","last_change":"2024-03-23T03:51:22.258385+0000","last_active":"2024-03-23T03:51:22.261946+0000","last_peered":"2024-03-23T03:51:22.261946+0000","last_clean":"2024-03-23T03:51:22.261946+0000","last_became_active":"2024-03-23T03:51:22.258120+0000","last_became_peered":"2024-03-23T03:51:22.258120+0000","last_unstale":"2024-03-23T03:51:22.261946+0000","last_undegraded":"2024-03-23T03:51:22.261946+0000","last_fullsized":"2024-03-23T03:51:22.261946+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:05.542126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.265163+0000","last_change":"2024-03-23T03:51:21.793737+0000","last_active":"2024-03-23T03:51:22.265163+0000","last_peered":"2024-03-23T03:51:22.265163+0000","last_clean":"2024-03-23T03:51:22.265163+0000","last_became_active":"2024-03-23T03:51:21.791575+0000","last_became_peered":"2024-03-23T03:51:21.791575+0000","last_unstale":"2024-03-23T03:51:22.265163+0000","last_undegraded":"2024-03-23T03:51:22.265163+0000","last_fullsized":"2024-03-23T03:51:22.265163+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:54:20.064900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.269225+0000","last_change":"2024-03-23T03:51:22.266599+0000","last_active":"2024-03-23T03:51:22.269225+0000","last_peered":"2024-03-23T03:51:22.269225+0000","last_clean":"2024-03-23T03:51:22.269225+0000","last_became_active":"2024-03-23T03:51:22.266281+0000","last_became_peered":"2024-03-23T03:51:22.266281+0000","last_unstale":"2024-03-23T03:51:22.269225+0000","last_undegraded":"2024-03-23T03:51:22.269225+0000","last_fullsized":"2024-03-23T03:51:22.269225+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:45:36.655749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.250446+0000","last_change":"2024-03-23T03:51:21.787337+0000","last_active":"2024-03-23T03:51:22.250446+0000","last_peered":"2024-03-23T03:51:22.250446+0000","last_clean":"2024-03-23T03:51:22.250446+0000","last_became_active":"2024-03-23T03:51:21.787076+0000","last_became_peered":"2024-03-23T03:51:21.787076+0000","last_unstale":"2024-03-23T03:51:22.250446+0000","last_undegraded":"2024-03-23T03:51:22.250446+0000","last_fullsized":"2024-03-23T03:51:22.250446+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:09:48.708095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.282566+0000","last_change":"2024-03-23T03:51:22.280399+0000","last_active":"2024-03-23T03:51:22.282566+0000","last_peered":"2024-03-23T03:51:22.282566+0000","last_clean":"2024-03-23T03:51:22.282566+0000","last_became_active":"2024-03-23T03:51:22.280129+0000","last_became_peered":"2024-03-23T03:51:22.280129+0000","last_unstale":"2024-03-23T03:51:22.282566+0000","last_undegraded":"2024-03-23T03:51:22.282566+0000","last_fullsized":"2024-03-23T03:51:22.282566+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:05:10.465032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.263361+0000","last_change":"2024-03-23T03:51:21.789939+0000","last_active":"2024-03-23T03:51:22.263361+0000","last_peered":"2024-03-23T03:51:22.263361+0000","last_clean":"2024-03-23T03:51:22.263361+0000","last_became_active":"2024-03-23T03:51:21.788412+0000","last_became_peered":"2024-03-23T03:51:21.788412+0000","last_unstale":"2024-03-23T03:51:22.263361+0000","last_undegraded":"2024-03-23T03:51:22.263361+0000","last_fullsized":"2024-03-23T03:51:22.263361+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:35:47.134641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.283863+0000","last_change":"2024-03-23T03:51:22.272329+0000","last_active":"2024-03-23T03:51:22.283863+0000","last_peered":"2024-03-23T03:51:22.283863+0000","last_clean":"2024-03-23T03:51:22.283863+0000","last_became_active":"2024-03-23T03:51:22.271506+0000","last_became_peered":"2024-03-23T03:51:22.271506+0000","last_unstale":"2024-03-23T03:51:22.283863+0000","last_undegraded":"2024-03-23T03:51:22.283863+0000","last_fullsized":"2024-03-23T03:51:22.283863+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:05:03.516080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.249914+0000","last_change":"2024-03-23T03:51:21.791827+0000","last_active":"2024-03-23T03:51:22.249914+0000","last_peered":"2024-03-23T03:51:22.249914+0000","last_clean":"2024-03-23T03:51:22.249914+0000","last_became_active":"2024-03-23T03:51:21.791445+0000","last_became_peered":"2024-03-23T03:51:21.791445+0000","last_unstale":"2024-03-23T03:51:22.249914+0000","last_undegraded":"2024-03-23T03:51:22.249914+0000","last_fullsized":"2024-03-23T03:51:22.249914+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:58.150428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.275945+0000","last_change":"2024-03-23T03:51:22.270121+0000","last_active":"2024-03-23T03:51:22.275945+0000","last_peered":"2024-03-23T03:51:22.275945+0000","last_clean":"2024-03-23T03:51:22.275945+0000","last_became_active":"2024-03-23T03:51:22.269806+0000","last_became_peered":"2024-03-23T03:51:22.269806+0000","last_unstale":"2024-03-23T03:51:22.275945+0000","last_undegraded":"2024-03-23T03:51:22.275945+0000","last_fullsized":"2024-03-23T03:51:22.275945+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:04:21.319327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.257587+0000","last_change":"2024-03-23T03:51:21.779521+0000","last_active":"2024-03-23T03:51:22.257587+0000","last_peered":"2024-03-23T03:51:22.257587+0000","last_clean":"2024-03-23T03:51:22.257587+0000","last_became_active":"2024-03-23T03:51:21.779111+0000","last_became_peered":"2024-03-23T03:51:21.779111+0000","last_unstale":"2024-03-23T03:51:22.257587+0000","last_undegraded":"2024-03-23T03:51:22.257587+0000","last_fullsized":"2024-03-23T03:51:22.257587+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:29:39.956286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":230,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.260723+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:51:22.260723+0000","last_peered":"2024-03-23T03:51:22.260723+0000","last_clean":"2024-03-23T03:51:22.260723+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:51:22.260723+0000","last_undegraded":"2024-03-23T03:51:22.260723+0000","last_fullsized":"2024-03-23T03:51:22.260723+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542204,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32980,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338860,"statfs":{"total":96636764160,"available":96602992640,"internally_reserved":0,"allocated":1900544,"data_stored":1180111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.087,"15min":1.0720000000000001},"min":{"1min":0.65700000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5209999999999999,"5min":1.635,"15min":1.635},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.042,"15min":1.0469999999999999},"min":{"1min":0.749,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.393,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.0049999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.081,"15min":1.0529999999999999},"min":{"1min":0.71499999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6120000000000001,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.147},{"interface":"front","average":{"1min":1.03,"5min":1.1719999999999999,"15min":1.2809999999999999},"min":{"1min":0.79000000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.321,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":1.5109999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.125,"15min":1.151},"min":{"1min":0.81299999999999994,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.583,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.171},{"interface":"front","average":{"1min":1.167,"5min":1.2869999999999999,"15min":1.401},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.506,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.4990000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.298,"15min":1.4179999999999999},"min":{"1min":0.71999999999999997,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.4219999999999999,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.325,"15min":1.353},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.677,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.452}]},{"osd":4,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.254,"15min":1.2629999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.8220000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.272},{"interface":"front","average":{"1min":1.333,"5min":1.347,"15min":1.3720000000000001},"min":{"1min":0.67500000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.778,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":0.92900000000000005}]},{"osd":5,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.389,"15min":1.4179999999999999},"min":{"1min":0.81299999999999994,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.7310000000000001,"5min":2.645,"15min":2.645},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.361,"5min":1.417,"15min":1.446},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.7649999999999999,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.119}]},{"osd":6,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.504,"15min":1.552},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.9279999999999999,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.421},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.474,"15min":1.5149999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9399999999999999,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.321}]}]},{"osd":6,"up_from":14,"seq":60129542202,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32404,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339436,"statfs":{"total":96636764160,"available":96603582464,"internally_reserved":0,"allocated":1318912,"data_stored":591751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.0429999999999999,"15min":1.044},"min":{"1min":0.69699999999999995,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.05,"15min":1.04},"min":{"1min":0.66100000000000003,"5min":0.62,"15min":0.62},"max":{"1min":1.5409999999999999,"5min":1.946,"15min":1.946},"last":1.0580000000000001}]},{"osd":1,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.087,"15min":1.075},"min":{"1min":0.745,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":1.204},{"interface":"front","average":{"1min":1.196,"5min":1.1339999999999999,"15min":1.1579999999999999},"min":{"1min":0.80500000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.5369999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.1439999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.083,"15min":1.071},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.4350000000000001,"5min":2.044,"15min":2.044},"last":1.327},{"interface":"front","average":{"1min":1.133,"5min":1.1759999999999999,"15min":1.212},"min":{"1min":0.85499999999999998,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.6599999999999999,"5min":1.681,"15min":1.681},"last":1.109}]},{"osd":3,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.2370000000000001,"15min":1.276},"min":{"1min":0.88100000000000001,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.786,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.2330000000000001,"15min":1.2569999999999999},"min":{"1min":0.90100000000000002,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.446,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.256}]},{"osd":4,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3779999999999999,"15min":1.385},"min":{"1min":1.1930000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.6679999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.431,"5min":1.371,"15min":1.3859999999999999},"min":{"1min":1.234,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.639,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.52}]},{"osd":5,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.403,"15min":1.4319999999999999},"min":{"1min":1.0669999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.794,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.474,"5min":1.508,"15min":1.542},"min":{"1min":1.091,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8200000000000001,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.5669999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.4059999999999999,"15min":1.4119999999999999},"min":{"1min":0.95999999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.0760000000000001,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.482,"15min":1.5309999999999999},"min":{"1min":1.125,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.8660000000000001,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.373}]}]},{"osd":5,"up_from":14,"seq":60129542202,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1298432,"data_stored":589583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1439999999999999,"15min":1.2030000000000001},"min":{"1min":0.72999999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4830000000000001,"5min":1.806,"15min":1.806},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.113,"5min":1.1399999999999999,"15min":1.1759999999999999},"min":{"1min":0.76500000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.855,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.93300000000000005}]},{"osd":1,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0940000000000001,"15min":1.1000000000000001},"min":{"1min":0.80100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.3520000000000001,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.232,"5min":1.165,"15min":1.1930000000000001},"min":{"1min":0.84799999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.663,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.167}]},{"osd":2,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.274,"15min":1.3460000000000001},"min":{"1min":0.70499999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8939999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.226,"5min":1.2989999999999999,"15min":1.3660000000000001},"min":{"1min":0.875,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.579,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.88400000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.2,"15min":1.244},"min":{"1min":0.85399999999999998,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.7050000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.375,"15min":1.446},"min":{"1min":0.997,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.964,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.3340000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3049999999999999,"15min":1.399},"min":{"1min":0.65700000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6100000000000001,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.038},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3859999999999999,"15min":1.4359999999999999},"min":{"1min":0.83699999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8069999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.4370000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.4159999999999999,"15min":1.5},"min":{"1min":0.83399999999999996,"5min":0.63,"15min":0.63},"max":{"1min":1.5840000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.5409999999999999,"15min":1.639},"min":{"1min":1.1950000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.6359999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.3049999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4910000000000001,"15min":1.591},"min":{"1min":1.095,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.9259999999999999,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.268},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5840000000000001,"15min":1.6839999999999999},"min":{"1min":1.272,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.056,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.3939999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542203,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32392,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339448,"statfs":{"total":96636764160,"available":96603594752,"internally_reserved":0,"allocated":1306624,"data_stored":589675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.004,"15min":0.98099999999999998},"min":{"1min":0.624,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.47,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.995,"15min":1.0049999999999999},"min":{"1min":0.56100000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3400000000000001,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.84999999999999998}]},{"osd":1,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0680000000000001,"15min":1.05},"min":{"1min":0.65100000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.5,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.226},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0880000000000001,"15min":1.0640000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.5209999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.272}]},{"osd":2,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.2470000000000001,"15min":1.3089999999999999},"min":{"1min":0.70399999999999996,"5min":0.625,"15min":0.625},"max":{"1min":1.452,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.2789999999999999,"15min":1.2969999999999999},"min":{"1min":0.59899999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6859999999999999,"5min":2.181,"15min":2.181},"last":1.4510000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.1160000000000001,"15min":1.1240000000000001},"min":{"1min":0.61899999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.4419999999999999,"5min":1.821,"15min":1.821},"last":1.097},{"interface":"front","average":{"1min":1.141,"5min":1.2310000000000001,"15min":1.268},"min":{"1min":0.73799999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.599,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.0529999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.2370000000000001,"15min":1.26},"min":{"1min":0.63800000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.49,"5min":1.929,"15min":1.929},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.2709999999999999,"15min":1.274},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.45,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.3959999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.371,"15min":1.409},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6719999999999999,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.4139999999999999,"15min":1.3999999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.8100000000000001,"5min":2.016,"15min":2.016},"last":1.5580000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.361,"15min":1.363},"min":{"1min":0.68200000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.5569999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.456,"15min":1.4870000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.716,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.494}]}]},{"osd":3,"up_from":14,"seq":60129542203,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1302528,"data_stored":590165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.81999999999999995,"15min":0.85499999999999998},"min":{"1min":0.35499999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.4419999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":1.48},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.92200000000000004,"15min":0.92400000000000004},"min":{"1min":0.502,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.538,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":1.0129999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.91500000000000004,"15min":0.96799999999999997},"min":{"1min":0.58099999999999996,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.9990000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.0229999999999999,"15min":1.069},"min":{"1min":0.52000000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.651,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.77000000000000002}]},{"osd":2,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.002,"15min":1.0509999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.5710000000000002,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1240000000000001,"15min":1.1499999999999999},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.7649999999999999,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":1.2170000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.2410000000000001,"15min":1.248},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":4.0380000000000003,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.337,"5min":1.351,"15min":1.429},"min":{"1min":0.77000000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.3450000000000002,"5min":4.819,"15min":4.819},"last":1.1899999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.6040000000000001,"15min":1.4470000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.2679999999999998,"5min":30.228000000000002,"15min":30.228000000000002},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.337,"15min":1.357},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":4.3630000000000004,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":1.115}]},{"osd":6,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4119999999999999,"15min":1.508},"min":{"1min":0.83999999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.6589999999999998,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.302},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4199999999999999,"15min":1.4039999999999999},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.4039999999999999,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":1.409}]},{"osd":7,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.389,"15min":1.4199999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.0880000000000001,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":1.264},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.542,"15min":1.5900000000000001},"min":{"1min":0.80500000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.4780000000000002,"5min":10.663,"15min":10.663},"last":1.377}]}]},{"osd":2,"up_from":14,"seq":60129542203,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1302528,"data_stored":589617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83799999999999997,"15min":0.877},"min":{"1min":0.52100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":1.4259999999999999,"5min":1.456,"15min":1.456},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.86399999999999999,"15min":0.93600000000000005},"min":{"1min":0.39500000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.4690000000000001,"5min":2.226,"15min":2.226},"last":0.65500000000000003}]},{"osd":1,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.93899999999999995,"15min":0.98499999999999999},"min":{"1min":0.46800000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.377,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.85899999999999999,"15min":0.91000000000000003},"min":{"1min":0.57599999999999996,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.2210000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.59399999999999997}]},{"osd":3,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.96099999999999997,"15min":1.018},"min":{"1min":0.47499999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.403,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.032,"5min":1.042,"15min":1.0800000000000001},"min":{"1min":0.61099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.63300000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.194,"15min":1.2310000000000001},"min":{"1min":0.82399999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.7270000000000001,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.157,"15min":1.2070000000000001},"min":{"1min":0.83799999999999997,"5min":0.627,"15min":0.627},"max":{"1min":1.4950000000000001,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.1950000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.3100000000000001,"15min":1.367},"min":{"1min":0.76900000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.6259999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.2749999999999999,"15min":1.339},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.819,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.0669999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.232,"15min":1.3280000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6479999999999999,"5min":2.456,"15min":2.456},"last":1.272},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3999999999999999,"15min":1.476},"min":{"1min":0.85299999999999998,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.097,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.0820000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3069999999999999,"15min":1.369},"min":{"1min":0.79800000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.171},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.302,"15min":1.395},"min":{"1min":0.78700000000000003,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5860000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.3700000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542202,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1302528,"data_stored":589617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79100000000000004,"15min":0.86099999999999999},"min":{"1min":0.41599999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.1240000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81799999999999995,"15min":0.82999999999999996},"min":{"1min":0.435,"5min":0.374,"15min":0.374},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.69199999999999995}]},{"osd":2,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.92200000000000004,"15min":0.96399999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.4430000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.75},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.92400000000000004,"15min":0.97799999999999998},"min":{"1min":0.496,"5min":0.443,"15min":0.443},"max":{"1min":1.2210000000000001,"5min":1.625,"15min":1.625},"last":0.89500000000000002}]},{"osd":3,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.103,"15min":1.113},"min":{"1min":0.68400000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.6559999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":0.875,"5min":0.96199999999999997,"15min":1.008},"min":{"1min":0.59499999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.6180000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.94499999999999995}]},{"osd":4,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.179,"15min":1.2749999999999999},"min":{"1min":0.67700000000000005,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.5069999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.246,"15min":1.3049999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6120000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.165}]},{"osd":5,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.248,"15min":1.3200000000000001},"min":{"1min":0.79500000000000004,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.95,"5min":3.593,"15min":3.593},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.264,"5min":1.327,"15min":1.4259999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.671,"5min":2.359,"15min":2.359},"last":1.4039999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.3089999999999999,"15min":1.3919999999999999},"min":{"1min":0.84199999999999997,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6200000000000001,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.357},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.288,"15min":1.3540000000000001},"min":{"1min":0.89100000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.0209999999999999,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.1399999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.359,"15min":1.4330000000000001},"min":{"1min":0.99299999999999999,"5min":0.627,"15min":0.627},"max":{"1min":2.0840000000000001,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.363,"15min":1.486},"min":{"1min":0.76600000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.673,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.4530000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542202,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32988,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338852,"statfs":{"total":96636764160,"available":96602984448,"internally_reserved":0,"allocated":1908736,"data_stored":1181605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.86399999999999999,"15min":0.93500000000000005},"min":{"1min":0.48399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.478,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.94799999999999995,"15min":0.98199999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.456,"5min":1.506,"15min":1.506},"last":1.2709999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.92700000000000005,"15min":0.99099999999999999},"min":{"1min":0.52900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.4399999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.0349999999999999,"15min":1.095},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.601,"5min":1.901,"15min":1.901},"last":0.95999999999999996}]},{"osd":3,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.93300000000000005,"15min":0.996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.474,"5min":1.972,"15min":1.972},"last":0.878},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0669999999999999,"15min":1.147},"min":{"1min":0.60299999999999998,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.597,"5min":2.298,"15min":2.298},"last":1.0449999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.288,"15min":1.3360000000000001},"min":{"1min":0.72799999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.7090000000000001,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":1.367},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.2230000000000001,"15min":1.27},"min":{"1min":0.80600000000000005,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.8380000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.1100000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3100000000000001,"15min":1.3700000000000001},"min":{"1min":0.89500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.948,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.573},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.377,"15min":1.4379999999999999},"min":{"1min":0.83499999999999996,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.8700000000000001,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.5429999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.4099999999999999,"15min":1.5209999999999999},"min":{"1min":0.80000000000000004,"5min":0.747,"15min":0.747},"max":{"1min":1.978,"5min":3.032,"15min":3.032},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.411,"15min":1.4870000000000001},"min":{"1min":0.84699999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.1349999999999998,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.1919999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4390000000000001,"15min":1.506},"min":{"1min":0.90500000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1850000000000001,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.45},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3620000000000001,"15min":1.419},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.873,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.0049999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:51:28.141 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-23T03:51:28.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.384+0000 7f645ac00640 1 Processor -- start 2024-03-23T03:51:28.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.385+0000 7f645ac00640 1 -- start start 2024-03-23T03:51:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.386+0000 7f645ac00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f645415cab0 0x7f64540b8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.386+0000 7f645ac00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f64540ceeb0 0x7f64540cf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.386+0000 7f645ac00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f64540b4120 0x7f64540b45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.386+0000 7f645ac00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64540cf850 con 0x7f64540ceeb0 2024-03-23T03:51:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.386+0000 7f645ac00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f645415ce90 con 0x7f64540b4120 2024-03-23T03:51:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.386+0000 7f645ac00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64540cddf0 con 0x7f645415cab0 2024-03-23T03:51:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.387+0000 7f6458e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f64540ceeb0 0x7f64540cf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.387+0000 7f6458e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f64540ceeb0 0x7f64540cf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42632/0 (socket says 172.21.15.102:42632) 2024-03-23T03:51:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.387+0000 7f6458e00640 1 -- 172.21.15.102:0/3815128514 learned_addr learned my addr 172.21.15.102:0/3815128514 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.387+0000 7f6459800640 1 --2- 172.21.15.102:0/3815128514 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f645415cab0 0x7f64540b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.387+0000 7f645a200640 1 --2- 172.21.15.102:0/3815128514 >> v2:172.21.15.173:3300/0 conn(0x7f64540b4120 0x7f64540b45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.387+0000 7f6458e00640 1 -- 172.21.15.102:0/3815128514 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f645415cab0 msgr2=0x7f64540b8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:28.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.387+0000 7f6458e00640 1 --2- 172.21.15.102:0/3815128514 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f645415cab0 0x7f64540b8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.387+0000 7f6458e00640 1 -- 172.21.15.102:0/3815128514 >> v2:172.21.15.173:3300/0 conn(0x7f64540b4120 msgr2=0x7f64540b45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:28.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.387+0000 7f6458e00640 1 --2- 172.21.15.102:0/3815128514 >> v2:172.21.15.173:3300/0 conn(0x7f64540b4120 0x7f64540b45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.387+0000 7f6458e00640 1 -- 172.21.15.102:0/3815128514 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64540cd570 con 0x7f64540ceeb0 2024-03-23T03:51:28.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.388+0000 7f6458e00640 1 --2- 172.21.15.102:0/3815128514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64540ceeb0 0x7f64540cf310 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f64540b20b0 tx=0x7f644002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3ef969bfd571909 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:28.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.389+0000 7f6453e00640 1 -- 172.21.15.102:0/3815128514 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644002ca70 con 0x7f64540ceeb0 2024-03-23T03:51:28.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.389+0000 7f6453e00640 1 -- 172.21.15.102:0/3815128514 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644002cbd0 con 0x7f64540ceeb0 2024-03-23T03:51:28.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.389+0000 7f6453e00640 1 -- 172.21.15.102:0/3815128514 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64400343f0 con 0x7f64540ceeb0 2024-03-23T03:51:28.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.389+0000 7f645ac00640 1 -- 172.21.15.102:0/3815128514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64540ceeb0 msgr2=0x7f64540cf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:28.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.389+0000 7f645ac00640 1 --2- 172.21.15.102:0/3815128514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64540ceeb0 0x7f64540cf310 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f64540b20b0 tx=0x7f644002bed0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.389+0000 7f645ac00640 1 -- 172.21.15.102:0/3815128514 shutdown_connections 2024-03-23T03:51:28.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.389+0000 7f645ac00640 1 --2- 172.21.15.102:0/3815128514 >> v2:172.21.15.173:3300/0 conn(0x7f64540b4120 0x7f64540b45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.389+0000 7f645ac00640 1 --2- 172.21.15.102:0/3815128514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64540ceeb0 0x7f64540cf310 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.389+0000 7f645ac00640 1 --2- 172.21.15.102:0/3815128514 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f645415cab0 0x7f64540b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.389+0000 7f645ac00640 1 -- 172.21.15.102:0/3815128514 >> 172.21.15.102:0/3815128514 conn(0x7f645406ae90 msgr2=0x7f645406b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:28.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.389+0000 7f645ac00640 1 -- 172.21.15.102:0/3815128514 shutdown_connections 2024-03-23T03:51:28.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.389+0000 7f645ac00640 1 -- 172.21.15.102:0/3815128514 wait complete. 2024-03-23T03:51:28.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.391+0000 7f645ac00640 1 Processor -- start 2024-03-23T03:51:28.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.391+0000 7f645ac00640 1 -- start start 2024-03-23T03:51:28.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.392+0000 7f645ac00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64540b4120 0x7f6454154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:28.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.392+0000 7f645ac00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f645415cab0 0x7f64541568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:28.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.392+0000 7f645ac00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f645415e350 0x7f6454156e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:28.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.392+0000 7f645ac00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645415d060 con 0x7f645415e350 2024-03-23T03:51:28.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.392+0000 7f645ac00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f645415d2b0 con 0x7f64540b4120 2024-03-23T03:51:28.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.392+0000 7f645ac00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f645415d590 con 0x7f645415cab0 2024-03-23T03:51:28.394 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.392+0000 7f645a200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f645415e350 0x7f6454156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.392+0000 7f645a200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f645415e350 0x7f6454156e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42648/0 (socket says 172.21.15.102:42648) 2024-03-23T03:51:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.392+0000 7f645a200640 1 -- 172.21.15.102:0/3752064274 learned_addr learned my addr 172.21.15.102:0/3752064274 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.393+0000 7f6459800640 1 --2- 172.21.15.102:0/3752064274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64540b4120 0x7f6454154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.393+0000 7f6458e00640 1 --2- 172.21.15.102:0/3752064274 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f645415cab0 0x7f64541568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.393+0000 7f645a200640 1 -- 172.21.15.102:0/3752064274 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f645415cab0 msgr2=0x7f64541568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.393+0000 7f645a200640 1 --2- 172.21.15.102:0/3752064274 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f645415cab0 0x7f64541568d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.393+0000 7f645a200640 1 -- 172.21.15.102:0/3752064274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64540b4120 msgr2=0x7f6454154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.393+0000 7f645a200640 1 --2- 172.21.15.102:0/3752064274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64540b4120 0x7f6454154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.393+0000 7f645a200640 1 -- 172.21.15.102:0/3752064274 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6440009b30 con 0x7f645415e350 2024-03-23T03:51:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.393+0000 7f645a200640 1 --2- 172.21.15.102:0/3752064274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f645415e350 0x7f6454156e10 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f644c00d800 tx=0x7f644c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.393+0000 7f6451600640 1 -- 172.21.15.102:0/3752064274 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c0096c0 con 0x7f645415e350 2024-03-23T03:51:28.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.393+0000 7f6451600640 1 -- 172.21.15.102:0/3752064274 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644c009840 con 0x7f645415e350 2024-03-23T03:51:28.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.393+0000 7f6451600640 1 -- 172.21.15.102:0/3752064274 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644c009bf0 con 0x7f645415e350 2024-03-23T03:51:28.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.394+0000 7f645ac00640 1 -- 172.21.15.102:0/3752064274 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645415d870 con 0x7f645415e350 2024-03-23T03:51:28.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.394+0000 7f645ac00640 1 -- 172.21.15.102:0/3752064274 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f645415de30 con 0x7f645415e350 2024-03-23T03:51:28.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.395+0000 7f645ac00640 1 -- 172.21.15.102:0/3752064274 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f645404e030 con 0x7f645415e350 2024-03-23T03:51:28.396 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.396+0000 7f6451600640 1 -- 172.21.15.102:0/3752064274 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f644c009d50 con 0x7f645415e350 2024-03-23T03:51:28.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.397+0000 7f6451600640 1 --2- 172.21.15.102:0/3752064274 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f643406ac00 0x7f643406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:28.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.397+0000 7f6451600640 1 -- 172.21.15.102:0/3752064274 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f644c0899b0 con 0x7f645415e350 2024-03-23T03:51:28.397 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.397+0000 7f6459800640 1 --2- 172.21.15.102:0/3752064274 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f643406ac00 0x7f643406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:28.398 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.398+0000 7f6459800640 1 --2- 172.21.15.102:0/3752064274 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f643406ac00 0x7f643406d0c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6444006fd0 tx=0x7f6444008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:28.402 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.402+0000 7f6451600640 1 -- 172.21.15.102:0/3752064274 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f644c0549b0 con 0x7f645415e350 2024-03-23T03:51:28.668 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.668+0000 7f645ac00640 1 -- 172.21.15.102:0/3752064274 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f645415e020 con 0x7f645415e350 2024-03-23T03:51:28.669 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.669+0000 7f6451600640 1 -- 172.21.15.102:0/3752064274 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1432 (secure 0 0 0) 0x7f644c0185b0 con 0x7f645415e350 2024-03-23T03:51:28.669 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:28.669 INFO:teuthology.orchestra.run.smithi102.stdout:{"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":293,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":105,"num_osds":8,"num_up_osds":8,"osd_up_since":1711165606,"num_in_osds":8,"osd_in_since":1711165598,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":97}],"num_pgs":97,"num_pools":3,"num_objects":24,"data_bytes":592730,"bytes_used":258195456,"bytes_avail":772835917824,"bytes_total":773094113280,"write_bytes_sec":2369,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":56,"id":7,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":7,"rank":0,"name":"a","status":"up:active","gid":7212}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-03-23T03:51:09.797893+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-23T03:51:28.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.671+0000 7f645ac00640 1 -- 172.21.15.102:0/3752064274 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f643406ac00 msgr2=0x7f643406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:28.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.671+0000 7f645ac00640 1 --2- 172.21.15.102:0/3752064274 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f643406ac00 0x7f643406d0c0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6444006fd0 tx=0x7f6444008040 comp rx=0 tx=0).stop 2024-03-23T03:51:28.672 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.671+0000 7f645ac00640 1 -- 172.21.15.102:0/3752064274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f645415e350 msgr2=0x7f6454156e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:28.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.671+0000 7f645ac00640 1 --2- 172.21.15.102:0/3752064274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f645415e350 0x7f6454156e10 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f644c00d800 tx=0x7f644c00dcc0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.672+0000 7f645ac00640 1 -- 172.21.15.102:0/3752064274 shutdown_connections 2024-03-23T03:51:28.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.672+0000 7f645ac00640 1 --2- 172.21.15.102:0/3752064274 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f643406ac00 0x7f643406d0c0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.672+0000 7f645ac00640 1 --2- 172.21.15.102:0/3752064274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f645415e350 0x7f6454156e10 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.672+0000 7f645ac00640 1 --2- 172.21.15.102:0/3752064274 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f645415cab0 0x7f64541568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.672+0000 7f645ac00640 1 --2- 172.21.15.102:0/3752064274 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64540b4120 0x7f6454154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.672+0000 7f645ac00640 1 -- 172.21.15.102:0/3752064274 >> 172.21.15.102:0/3752064274 conn(0x7f645406ae90 msgr2=0x7f64540b1500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:28.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.672+0000 7f645ac00640 1 -- 172.21.15.102:0/3752064274 shutdown_connections 2024-03-23T03:51:28.673 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.672+0000 7f645ac00640 1 -- 172.21.15.102:0/3752064274 wait complete. 2024-03-23T03:51:28.690 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-23T03:51:28.690 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:51:28.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.898+0000 7fcd25c00640 1 Processor -- start 2024-03-23T03:51:28.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.899+0000 7fcd25c00640 1 -- start start 2024-03-23T03:51:28.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.900+0000 7fcd25c00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcd2015d350 0x7fcd20163fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:28.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.900+0000 7fcd25c00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd200ce440 con 0x7fcd2013f200 2024-03-23T03:51:28.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.900+0000 7fcd25c00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd200cdba0 con 0x7fcd200cd6f0 2024-03-23T03:51:28.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.900+0000 7fcd25c00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd200cde80 con 0x7fcd2015d350 2024-03-23T03:51:28.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.900+0000 7fcd1ea00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fcd2013f200 0x7fcd200ccfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:38580/0 (socket says 172.21.15.102:38580) 2024-03-23T03:51:28.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.900+0000 7fcd1ea00640 1 -- 172.21.15.102:0/509726394 learned_addr learned my addr 172.21.15.102:0/509726394 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:28.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.900+0000 7fcd1f400640 1 --2- 172.21.15.102:0/509726394 >> v2:172.21.15.102:3300/0 conn(0x7fcd2015d350 0x7fcd20163fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:28.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.901+0000 7fcd1d600640 1 -- 172.21.15.102:0/509726394 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908555250 0 0) 0x7fcd200ce440 con 0x7fcd2013f200 2024-03-23T03:51:28.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.901+0000 7fcd1d600640 1 -- 172.21.15.102:0/509726394 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd04003520 con 0x7fcd2013f200 2024-03-23T03:51:28.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.901+0000 7fcd1f400640 1 -- 172.21.15.102:0/509726394 >> v1:172.21.15.173:6789/0 conn(0x7fcd200cd6f0 legacy=0x7fcd2015cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:28.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.901+0000 7fcd1f400640 1 -- 172.21.15.102:0/509726394 >> v1:172.21.15.102:6789/0 conn(0x7fcd2013f200 legacy=0x7fcd200ccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:28.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.901+0000 7fcd1f400640 1 -- 172.21.15.102:0/509726394 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd200ce160 con 0x7fcd2015d350 2024-03-23T03:51:28.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.902+0000 7fcd1f400640 1 --2- 172.21.15.102:0/509726394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd2015d350 0x7fcd20163fd0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fcd18009dc0 tx=0x7fcd1802be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81bbc08c93cfdebb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:28.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.902+0000 7fcd1d600640 1 -- 172.21.15.102:0/509726394 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1802cbb0 con 0x7fcd2015d350 2024-03-23T03:51:28.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.902+0000 7fcd1d600640 1 -- 172.21.15.102:0/509726394 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1802cd10 con 0x7fcd2015d350 2024-03-23T03:51:28.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.902+0000 7fcd1d600640 1 -- 172.21.15.102:0/509726394 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd18034660 con 0x7fcd2015d350 2024-03-23T03:51:28.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.903+0000 7fcd25c00640 1 -- 172.21.15.102:0/509726394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd2015d350 msgr2=0x7fcd20163fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:28.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.903+0000 7fcd25c00640 1 --2- 172.21.15.102:0/509726394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd2015d350 0x7fcd20163fd0 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fcd18009dc0 tx=0x7fcd1802be00 comp rx=0 tx=0).stop 2024-03-23T03:51:28.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.903+0000 7fcd25c00640 1 -- 172.21.15.102:0/509726394 shutdown_connections 2024-03-23T03:51:28.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.903+0000 7fcd25c00640 1 --2- 172.21.15.102:0/509726394 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd2015d350 0x7fcd20163fd0 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.903+0000 7fcd25c00640 1 -- 172.21.15.102:0/509726394 >> 172.21.15.102:0/509726394 conn(0x7fcd2006a3d0 msgr2=0x7fcd200b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:28.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.903+0000 7fcd25c00640 1 -- 172.21.15.102:0/509726394 shutdown_connections 2024-03-23T03:51:28.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.903+0000 7fcd25c00640 1 -- 172.21.15.102:0/509726394 wait complete. 2024-03-23T03:51:28.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.905+0000 7fcd25c00640 1 Processor -- start 2024-03-23T03:51:28.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.905+0000 7fcd25c00640 1 -- start start 2024-03-23T03:51:28.906 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.906+0000 7fcd25c00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd200cd6f0 0x7fcd200af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:28.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.906+0000 7fcd25c00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcd2013f200 0x7fcd2014f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:28.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.906+0000 7fcd25c00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd2015d350 0x7fcd2013b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:28.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.906+0000 7fcd25c00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd200af8d0 con 0x7fcd2015d350 2024-03-23T03:51:28.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.906+0000 7fcd25c00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd200afbb0 con 0x7fcd200cd6f0 2024-03-23T03:51:28.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.906+0000 7fcd25c00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd200afe90 con 0x7fcd2013f200 2024-03-23T03:51:28.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.906+0000 7fcd1f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd2015d350 0x7fcd2013b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:28.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.906+0000 7fcd1f400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd2015d350 0x7fcd2013b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42674/0 (socket says 172.21.15.102:42674) 2024-03-23T03:51:28.907 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.906+0000 7fcd1f400640 1 -- 172.21.15.102:0/3417093043 learned_addr learned my addr 172.21.15.102:0/3417093043 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:28.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.906+0000 7fcd1f400640 1 -- 172.21.15.102:0/3417093043 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcd2013f200 msgr2=0x7fcd2014f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:28.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.906+0000 7fcd1f400640 1 --2- 172.21.15.102:0/3417093043 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcd2013f200 0x7fcd2014f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.906+0000 7fcd1f400640 1 -- 172.21.15.102:0/3417093043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd200cd6f0 msgr2=0x7fcd200af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:28.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.906+0000 7fcd1f400640 1 --2- 172.21.15.102:0/3417093043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd200cd6f0 0x7fcd200af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:28.908 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.906+0000 7fcd1f400640 1 -- 172.21.15.102:0/3417093043 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd18009a70 con 0x7fcd2015d350 2024-03-23T03:51:28.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.907+0000 7fcd1f400640 1 --2- 172.21.15.102:0/3417093043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd2015d350 0x7fcd2013b650 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fcd18037040 tx=0x7fcd18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:28.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.907+0000 7fcd16a00640 1 -- 172.21.15.102:0/3417093043 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1803e610 con 0x7fcd2015d350 2024-03-23T03:51:28.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.907+0000 7fcd16a00640 1 -- 172.21.15.102:0/3417093043 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1803e770 con 0x7fcd2015d350 2024-03-23T03:51:28.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.907+0000 7fcd25c00640 1 -- 172.21.15.102:0/3417093043 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd200b0170 con 0x7fcd2015d350 2024-03-23T03:51:28.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.907+0000 7fcd16a00640 1 -- 172.21.15.102:0/3417093043 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1803eb00 con 0x7fcd2015d350 2024-03-23T03:51:28.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.907+0000 7fcd25c00640 1 -- 172.21.15.102:0/3417093043 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd200af5c0 con 0x7fcd2015d350 2024-03-23T03:51:28.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.909+0000 7fcd25c00640 1 -- 172.21.15.102:0/3417093043 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd200670d0 con 0x7fcd2015d350 2024-03-23T03:51:28.910 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.909+0000 7fcd16a00640 1 -- 172.21.15.102:0/3417093043 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcd1803b050 con 0x7fcd2015d350 2024-03-23T03:51:28.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.911+0000 7fcd16a00640 1 --2- 172.21.15.102:0/3417093043 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcd0006ac00 0x7fcd0006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:28.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.911+0000 7fcd16a00640 1 -- 172.21.15.102:0/3417093043 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fcd180abd50 con 0x7fcd2015d350 2024-03-23T03:51:28.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.911+0000 7fcd1ea00640 1 --2- 172.21.15.102:0/3417093043 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcd0006ac00 0x7fcd0006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:28.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.911+0000 7fcd1ea00640 1 --2- 172.21.15.102:0/3417093043 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcd0006ac00 0x7fcd0006d0c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fcd10001610 tx=0x7fcd10007930 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:28.915 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:28.915+0000 7fcd16a00640 1 -- 172.21.15.102:0/3417093043 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcd180af030 con 0x7fcd2015d350 2024-03-23T03:51:29.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:29.107+0000 7fcd25c00640 1 -- 172.21.15.102:0/3417093043 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcd2013f5e0 con 0x7fcd0006ac00 2024-03-23T03:51:29.129 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:29.129+0000 7fcd16a00640 1 -- 172.21.15.102:0/3417093043 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287806 (secure 0 0 0) 0x7fcd100058b0 con 0x7fcd0006ac00 2024-03-23T03:51:29.133 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:29.135 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:51:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:29.134+0000 7fcd25c00640 1 -- 172.21.15.102:0/3417093043 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcd0006ac00 msgr2=0x7fcd0006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:29.134+0000 7fcd25c00640 1 --2- 172.21.15.102:0/3417093043 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcd0006ac00 0x7fcd0006d0c0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fcd10001610 tx=0x7fcd10007930 comp rx=0 tx=0).stop 2024-03-23T03:51:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:29.134+0000 7fcd25c00640 1 -- 172.21.15.102:0/3417093043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd2015d350 msgr2=0x7fcd2013b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:29.134+0000 7fcd25c00640 1 --2- 172.21.15.102:0/3417093043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd2015d350 0x7fcd2013b650 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fcd18037040 tx=0x7fcd18004a40 comp rx=0 tx=0).stop 2024-03-23T03:51:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:29.134+0000 7fcd25c00640 1 -- 172.21.15.102:0/3417093043 shutdown_connections 2024-03-23T03:51:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:29.134+0000 7fcd25c00640 1 --2- 172.21.15.102:0/3417093043 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcd0006ac00 0x7fcd0006d0c0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:29.134+0000 7fcd25c00640 1 --2- 172.21.15.102:0/3417093043 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd2015d350 0x7fcd2013b650 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:29.134+0000 7fcd25c00640 1 --2- 172.21.15.102:0/3417093043 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcd2013f200 0x7fcd2014f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:29.134+0000 7fcd25c00640 1 --2- 172.21.15.102:0/3417093043 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcd200cd6f0 0x7fcd200af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:29.134+0000 7fcd25c00640 1 -- 172.21.15.102:0/3417093043 >> 172.21.15.102:0/3417093043 conn(0x7fcd2006a3d0 msgr2=0x7fcd200b41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:29.135+0000 7fcd25c00640 1 -- 172.21.15.102:0/3417093043 shutdown_connections 2024-03-23T03:51:29.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:29.135+0000 7fcd25c00640 1 -- 172.21.15.102:0/3417093043 wait complete. 2024-03-23T03:51:29.154 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2024-03-23T03:51:27.801978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252144,"kb_used_data":11368,"kb_used_omap":12,"kb_used_meta":240627,"kb_avail":754722576,"statfs":{"total":773094113280,"available":772835917824,"internally_reserved":0,"allocated":11640832,"data_stored":5902124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":246402618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.050539"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.267572+0000","last_change":"2024-03-23T03:51:22.263565+0000","last_active":"2024-03-23T03:51:22.267572+0000","last_peered":"2024-03-23T03:51:22.267572+0000","last_clean":"2024-03-23T03:51:22.267572+0000","last_became_active":"2024-03-23T03:51:22.263323+0000","last_became_peered":"2024-03-23T03:51:22.263323+0000","last_unstale":"2024-03-23T03:51:22.267572+0000","last_undegraded":"2024-03-23T03:51:22.267572+0000","last_fullsized":"2024-03-23T03:51:22.267572+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:24:42.534444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.288428+0000","last_change":"2024-03-23T03:51:22.286197+0000","last_active":"2024-03-23T03:51:22.288428+0000","last_peered":"2024-03-23T03:51:22.288428+0000","last_clean":"2024-03-23T03:51:22.288428+0000","last_became_active":"2024-03-23T03:51:22.280588+0000","last_became_peered":"2024-03-23T03:51:22.280588+0000","last_unstale":"2024-03-23T03:51:22.288428+0000","last_undegraded":"2024-03-23T03:51:22.288428+0000","last_fullsized":"2024-03-23T03:51:22.288428+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:11:35.553745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.286379+0000","last_change":"2024-03-23T03:51:22.281125+0000","last_active":"2024-03-23T03:51:22.286379+0000","last_peered":"2024-03-23T03:51:22.286379+0000","last_clean":"2024-03-23T03:51:22.286379+0000","last_became_active":"2024-03-23T03:51:22.280105+0000","last_became_peered":"2024-03-23T03:51:22.280105+0000","last_unstale":"2024-03-23T03:51:22.286379+0000","last_undegraded":"2024-03-23T03:51:22.286379+0000","last_fullsized":"2024-03-23T03:51:22.286379+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:20:25.913448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.280149+0000","last_change":"2024-03-23T03:51:22.278175+0000","last_active":"2024-03-23T03:51:22.280149+0000","last_peered":"2024-03-23T03:51:22.280149+0000","last_clean":"2024-03-23T03:51:22.280149+0000","last_became_active":"2024-03-23T03:51:22.277895+0000","last_became_peered":"2024-03-23T03:51:22.277895+0000","last_unstale":"2024-03-23T03:51:22.280149+0000","last_undegraded":"2024-03-23T03:51:22.280149+0000","last_fullsized":"2024-03-23T03:51:22.280149+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:36:30.827292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.283501+0000","last_change":"2024-03-23T03:51:22.279802+0000","last_active":"2024-03-23T03:51:22.283501+0000","last_peered":"2024-03-23T03:51:22.283501+0000","last_clean":"2024-03-23T03:51:22.283501+0000","last_became_active":"2024-03-23T03:51:22.278955+0000","last_became_peered":"2024-03-23T03:51:22.278955+0000","last_unstale":"2024-03-23T03:51:22.283501+0000","last_undegraded":"2024-03-23T03:51:22.283501+0000","last_fullsized":"2024-03-23T03:51:22.283501+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:46:33.274045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.287694+0000","last_change":"2024-03-23T03:51:22.279603+0000","last_active":"2024-03-23T03:51:22.287694+0000","last_peered":"2024-03-23T03:51:22.287694+0000","last_clean":"2024-03-23T03:51:22.287694+0000","last_became_active":"2024-03-23T03:51:22.279288+0000","last_became_peered":"2024-03-23T03:51:22.279288+0000","last_unstale":"2024-03-23T03:51:22.287694+0000","last_undegraded":"2024-03-23T03:51:22.287694+0000","last_fullsized":"2024-03-23T03:51:22.287694+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:15:44.577151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.288339+0000","last_change":"2024-03-23T03:51:22.279845+0000","last_active":"2024-03-23T03:51:22.288339+0000","last_peered":"2024-03-23T03:51:22.288339+0000","last_clean":"2024-03-23T03:51:22.288339+0000","last_became_active":"2024-03-23T03:51:22.279530+0000","last_became_peered":"2024-03-23T03:51:22.279530+0000","last_unstale":"2024-03-23T03:51:22.288339+0000","last_undegraded":"2024-03-23T03:51:22.288339+0000","last_fullsized":"2024-03-23T03:51:22.288339+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:19.362177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.274843+0000","last_change":"2024-03-23T03:51:22.267328+0000","last_active":"2024-03-23T03:51:22.274843+0000","last_peered":"2024-03-23T03:51:22.274843+0000","last_clean":"2024-03-23T03:51:22.274843+0000","last_became_active":"2024-03-23T03:51:22.267081+0000","last_became_peered":"2024-03-23T03:51:22.267081+0000","last_unstale":"2024-03-23T03:51:22.274843+0000","last_undegraded":"2024-03-23T03:51:22.274843+0000","last_fullsized":"2024-03-23T03:51:22.274843+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:23:32.227733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.278415+0000","last_change":"2024-03-23T03:51:22.275332+0000","last_active":"2024-03-23T03:51:22.278415+0000","last_peered":"2024-03-23T03:51:22.278415+0000","last_clean":"2024-03-23T03:51:22.278415+0000","last_became_active":"2024-03-23T03:51:22.274890+0000","last_became_peered":"2024-03-23T03:51:22.274890+0000","last_unstale":"2024-03-23T03:51:22.278415+0000","last_undegraded":"2024-03-23T03:51:22.278415+0000","last_fullsized":"2024-03-23T03:51:22.278415+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:54.013665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.278612+0000","last_change":"2024-03-23T03:51:22.273679+0000","last_active":"2024-03-23T03:51:22.278612+0000","last_peered":"2024-03-23T03:51:22.278612+0000","last_clean":"2024-03-23T03:51:22.278612+0000","last_became_active":"2024-03-23T03:51:22.273411+0000","last_became_peered":"2024-03-23T03:51:22.273411+0000","last_unstale":"2024-03-23T03:51:22.278612+0000","last_undegraded":"2024-03-23T03:51:22.278612+0000","last_fullsized":"2024-03-23T03:51:22.278612+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:58:33.997748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.284851+0000","last_change":"2024-03-23T03:51:22.278541+0000","last_active":"2024-03-23T03:51:22.284851+0000","last_peered":"2024-03-23T03:51:22.284851+0000","last_clean":"2024-03-23T03:51:22.284851+0000","last_became_active":"2024-03-23T03:51:22.278274+0000","last_became_peered":"2024-03-23T03:51:22.278274+0000","last_unstale":"2024-03-23T03:51:22.284851+0000","last_undegraded":"2024-03-23T03:51:22.284851+0000","last_fullsized":"2024-03-23T03:51:22.284851+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:10:28.823831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.282176+0000","last_change":"2024-03-23T03:51:22.277885+0000","last_active":"2024-03-23T03:51:22.282176+0000","last_peered":"2024-03-23T03:51:22.282176+0000","last_clean":"2024-03-23T03:51:22.282176+0000","last_became_active":"2024-03-23T03:51:22.277209+0000","last_became_peered":"2024-03-23T03:51:22.277209+0000","last_unstale":"2024-03-23T03:51:22.282176+0000","last_undegraded":"2024-03-23T03:51:22.282176+0000","last_fullsized":"2024-03-23T03:51:22.282176+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:07.649845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.265283+0000","last_change":"2024-03-23T03:51:22.261164+0000","last_active":"2024-03-23T03:51:22.265283+0000","last_peered":"2024-03-23T03:51:22.265283+0000","last_clean":"2024-03-23T03:51:22.265283+0000","last_became_active":"2024-03-23T03:51:22.260899+0000","last_became_peered":"2024-03-23T03:51:22.260899+0000","last_unstale":"2024-03-23T03:51:22.265283+0000","last_undegraded":"2024-03-23T03:51:22.265283+0000","last_fullsized":"2024-03-23T03:51:22.265283+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:35:05.132187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.272780+0000","last_change":"2024-03-23T03:51:22.266968+0000","last_active":"2024-03-23T03:51:22.272780+0000","last_peered":"2024-03-23T03:51:22.272780+0000","last_clean":"2024-03-23T03:51:22.272780+0000","last_became_active":"2024-03-23T03:51:22.266453+0000","last_became_peered":"2024-03-23T03:51:22.266453+0000","last_unstale":"2024-03-23T03:51:22.272780+0000","last_undegraded":"2024-03-23T03:51:22.272780+0000","last_fullsized":"2024-03-23T03:51:22.272780+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:30:04.249708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.281242+0000","last_change":"2024-03-23T03:51:22.271984+0000","last_active":"2024-03-23T03:51:22.281242+0000","last_peered":"2024-03-23T03:51:22.281242+0000","last_clean":"2024-03-23T03:51:22.281242+0000","last_became_active":"2024-03-23T03:51:22.271663+0000","last_became_peered":"2024-03-23T03:51:22.271663+0000","last_unstale":"2024-03-23T03:51:22.281242+0000","last_undegraded":"2024-03-23T03:51:22.281242+0000","last_fullsized":"2024-03-23T03:51:22.281242+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:17:09.963708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.281899+0000","last_change":"2024-03-23T03:51:22.277918+0000","last_active":"2024-03-23T03:51:22.281899+0000","last_peered":"2024-03-23T03:51:22.281899+0000","last_clean":"2024-03-23T03:51:22.281899+0000","last_became_active":"2024-03-23T03:51:22.277642+0000","last_became_peered":"2024-03-23T03:51:22.277642+0000","last_unstale":"2024-03-23T03:51:22.281899+0000","last_undegraded":"2024-03-23T03:51:22.281899+0000","last_fullsized":"2024-03-23T03:51:22.281899+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:04:21.319327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.286384+0000","last_change":"2024-03-23T03:51:22.281023+0000","last_active":"2024-03-23T03:51:22.286384+0000","last_peered":"2024-03-23T03:51:22.286384+0000","last_clean":"2024-03-23T03:51:22.286384+0000","last_became_active":"2024-03-23T03:51:22.280871+0000","last_became_peered":"2024-03-23T03:51:22.280871+0000","last_unstale":"2024-03-23T03:51:22.286384+0000","last_undegraded":"2024-03-23T03:51:22.286384+0000","last_fullsized":"2024-03-23T03:51:22.286384+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:55.608125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.288408+0000","last_change":"2024-03-23T03:51:22.285885+0000","last_active":"2024-03-23T03:51:22.288408+0000","last_peered":"2024-03-23T03:51:22.288408+0000","last_clean":"2024-03-23T03:51:22.288408+0000","last_became_active":"2024-03-23T03:51:22.280190+0000","last_became_peered":"2024-03-23T03:51:22.280190+0000","last_unstale":"2024-03-23T03:51:22.288408+0000","last_undegraded":"2024-03-23T03:51:22.288408+0000","last_fullsized":"2024-03-23T03:51:22.288408+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:03:27.282303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.279574+0000","last_change":"2024-03-23T03:51:22.276946+0000","last_active":"2024-03-23T03:51:22.279574+0000","last_peered":"2024-03-23T03:51:22.279574+0000","last_clean":"2024-03-23T03:51:22.279574+0000","last_became_active":"2024-03-23T03:51:22.276662+0000","last_became_peered":"2024-03-23T03:51:22.276662+0000","last_unstale":"2024-03-23T03:51:22.279574+0000","last_undegraded":"2024-03-23T03:51:22.279574+0000","last_fullsized":"2024-03-23T03:51:22.279574+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:48:26.593847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.282198+0000","last_change":"2024-03-23T03:51:22.279151+0000","last_active":"2024-03-23T03:51:22.282198+0000","last_peered":"2024-03-23T03:51:22.282198+0000","last_clean":"2024-03-23T03:51:22.282198+0000","last_became_active":"2024-03-23T03:51:22.276916+0000","last_became_peered":"2024-03-23T03:51:22.276916+0000","last_unstale":"2024-03-23T03:51:22.282198+0000","last_undegraded":"2024-03-23T03:51:22.282198+0000","last_fullsized":"2024-03-23T03:51:22.282198+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:13:39.429313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.276047+0000","last_change":"2024-03-23T03:51:22.273964+0000","last_active":"2024-03-23T03:51:22.276047+0000","last_peered":"2024-03-23T03:51:22.276047+0000","last_clean":"2024-03-23T03:51:22.276047+0000","last_became_active":"2024-03-23T03:51:22.273497+0000","last_became_peered":"2024-03-23T03:51:22.273497+0000","last_unstale":"2024-03-23T03:51:22.276047+0000","last_undegraded":"2024-03-23T03:51:22.276047+0000","last_fullsized":"2024-03-23T03:51:22.276047+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:20:51.607176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.267969+0000","last_change":"2024-03-23T03:51:22.262943+0000","last_active":"2024-03-23T03:51:22.267969+0000","last_peered":"2024-03-23T03:51:22.267969+0000","last_clean":"2024-03-23T03:51:22.267969+0000","last_became_active":"2024-03-23T03:51:22.262449+0000","last_became_peered":"2024-03-23T03:51:22.262449+0000","last_unstale":"2024-03-23T03:51:22.267969+0000","last_undegraded":"2024-03-23T03:51:22.267969+0000","last_fullsized":"2024-03-23T03:51:22.267969+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:04.683164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.272306+0000","last_change":"2024-03-23T03:51:22.270863+0000","last_active":"2024-03-23T03:51:22.272306+0000","last_peered":"2024-03-23T03:51:22.272306+0000","last_clean":"2024-03-23T03:51:22.272306+0000","last_became_active":"2024-03-23T03:51:22.270601+0000","last_became_peered":"2024-03-23T03:51:22.270601+0000","last_unstale":"2024-03-23T03:51:22.272306+0000","last_undegraded":"2024-03-23T03:51:22.272306+0000","last_fullsized":"2024-03-23T03:51:22.272306+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:46:32.018385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.267987+0000","last_change":"2024-03-23T03:51:22.262151+0000","last_active":"2024-03-23T03:51:22.267987+0000","last_peered":"2024-03-23T03:51:22.267987+0000","last_clean":"2024-03-23T03:51:22.267987+0000","last_became_active":"2024-03-23T03:51:22.261931+0000","last_became_peered":"2024-03-23T03:51:22.261931+0000","last_unstale":"2024-03-23T03:51:22.267987+0000","last_undegraded":"2024-03-23T03:51:22.267987+0000","last_fullsized":"2024-03-23T03:51:22.267987+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:47.178608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.288713+0000","last_change":"2024-03-23T03:51:22.286444+0000","last_active":"2024-03-23T03:51:22.288713+0000","last_peered":"2024-03-23T03:51:22.288713+0000","last_clean":"2024-03-23T03:51:22.288713+0000","last_became_active":"2024-03-23T03:51:22.285470+0000","last_became_peered":"2024-03-23T03:51:22.285470+0000","last_unstale":"2024-03-23T03:51:22.288713+0000","last_undegraded":"2024-03-23T03:51:22.288713+0000","last_fullsized":"2024-03-23T03:51:22.288713+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:58:40.479780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.284517+0000","last_change":"2024-03-23T03:51:22.280518+0000","last_active":"2024-03-23T03:51:22.284517+0000","last_peered":"2024-03-23T03:51:22.284517+0000","last_clean":"2024-03-23T03:51:22.284517+0000","last_became_active":"2024-03-23T03:51:22.280191+0000","last_became_peered":"2024-03-23T03:51:22.280191+0000","last_unstale":"2024-03-23T03:51:22.284517+0000","last_undegraded":"2024-03-23T03:51:22.284517+0000","last_fullsized":"2024-03-23T03:51:22.284517+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:58:05.924179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.283921+0000","last_change":"2024-03-23T03:51:22.272661+0000","last_active":"2024-03-23T03:51:22.283921+0000","last_peered":"2024-03-23T03:51:22.283921+0000","last_clean":"2024-03-23T03:51:22.283921+0000","last_became_active":"2024-03-23T03:51:22.272238+0000","last_became_peered":"2024-03-23T03:51:22.272238+0000","last_unstale":"2024-03-23T03:51:22.283921+0000","last_undegraded":"2024-03-23T03:51:22.283921+0000","last_fullsized":"2024-03-23T03:51:22.283921+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:20.563242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.278988+0000","last_change":"2024-03-23T03:51:22.274596+0000","last_active":"2024-03-23T03:51:22.278988+0000","last_peered":"2024-03-23T03:51:22.278988+0000","last_clean":"2024-03-23T03:51:22.278988+0000","last_became_active":"2024-03-23T03:51:22.274326+0000","last_became_peered":"2024-03-23T03:51:22.274326+0000","last_unstale":"2024-03-23T03:51:22.278988+0000","last_undegraded":"2024-03-23T03:51:22.278988+0000","last_fullsized":"2024-03-23T03:51:22.278988+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:11:23.842491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.275911+0000","last_change":"2024-03-23T03:51:22.273810+0000","last_active":"2024-03-23T03:51:22.275911+0000","last_peered":"2024-03-23T03:51:22.275911+0000","last_clean":"2024-03-23T03:51:22.275911+0000","last_became_active":"2024-03-23T03:51:22.273091+0000","last_became_peered":"2024-03-23T03:51:22.273091+0000","last_unstale":"2024-03-23T03:51:22.275911+0000","last_undegraded":"2024-03-23T03:51:22.275911+0000","last_fullsized":"2024-03-23T03:51:22.275911+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:46.517427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.287367+0000","last_change":"2024-03-23T03:51:22.283425+0000","last_active":"2024-03-23T03:51:22.287367+0000","last_peered":"2024-03-23T03:51:22.287367+0000","last_clean":"2024-03-23T03:51:22.287367+0000","last_became_active":"2024-03-23T03:51:22.283140+0000","last_became_peered":"2024-03-23T03:51:22.283140+0000","last_unstale":"2024-03-23T03:51:22.287367+0000","last_undegraded":"2024-03-23T03:51:22.287367+0000","last_fullsized":"2024-03-23T03:51:22.287367+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:36:38.929407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.274309+0000","last_change":"2024-03-23T03:51:22.269314+0000","last_active":"2024-03-23T03:51:22.274309+0000","last_peered":"2024-03-23T03:51:22.274309+0000","last_clean":"2024-03-23T03:51:22.274309+0000","last_became_active":"2024-03-23T03:51:22.268912+0000","last_became_peered":"2024-03-23T03:51:22.268912+0000","last_unstale":"2024-03-23T03:51:22.274309+0000","last_undegraded":"2024-03-23T03:51:22.274309+0000","last_fullsized":"2024-03-23T03:51:22.274309+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:25:25.635704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.286480+0000","last_change":"2024-03-23T03:51:22.282218+0000","last_active":"2024-03-23T03:51:22.286480+0000","last_peered":"2024-03-23T03:51:22.286480+0000","last_clean":"2024-03-23T03:51:22.286480+0000","last_became_active":"2024-03-23T03:51:22.281934+0000","last_became_peered":"2024-03-23T03:51:22.281934+0000","last_unstale":"2024-03-23T03:51:22.286480+0000","last_undegraded":"2024-03-23T03:51:22.286480+0000","last_fullsized":"2024-03-23T03:51:22.286480+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:19:58.455715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.279440+0000","last_change":"2024-03-23T03:51:22.276935+0000","last_active":"2024-03-23T03:51:22.279440+0000","last_peered":"2024-03-23T03:51:22.279440+0000","last_clean":"2024-03-23T03:51:22.279440+0000","last_became_active":"2024-03-23T03:51:22.276687+0000","last_became_peered":"2024-03-23T03:51:22.276687+0000","last_unstale":"2024-03-23T03:51:22.279440+0000","last_undegraded":"2024-03-23T03:51:22.279440+0000","last_fullsized":"2024-03-23T03:51:22.279440+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:57:56.460821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"104'4","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.274281+0000","last_change":"2024-03-23T03:51:21.789123+0000","last_active":"2024-03-23T03:51:22.274281+0000","last_peered":"2024-03-23T03:51:22.274281+0000","last_clean":"2024-03-23T03:51:22.274281+0000","last_became_active":"2024-03-23T03:51:21.788839+0000","last_became_peered":"2024-03-23T03:51:21.788839+0000","last_unstale":"2024-03-23T03:51:22.274281+0000","last_undegraded":"2024-03-23T03:51:22.274281+0000","last_fullsized":"2024-03-23T03:51:22.274281+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:54:15.206699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.261655+0000","last_change":"2024-03-23T03:51:22.258376+0000","last_active":"2024-03-23T03:51:22.261655+0000","last_peered":"2024-03-23T03:51:22.261655+0000","last_clean":"2024-03-23T03:51:22.261655+0000","last_became_active":"2024-03-23T03:51:22.257856+0000","last_became_peered":"2024-03-23T03:51:22.257856+0000","last_unstale":"2024-03-23T03:51:22.261655+0000","last_undegraded":"2024-03-23T03:51:22.261655+0000","last_fullsized":"2024-03-23T03:51:22.261655+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:04.438866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"104'1","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.276956+0000","last_change":"2024-03-23T03:51:21.785555+0000","last_active":"2024-03-23T03:51:22.276956+0000","last_peered":"2024-03-23T03:51:22.276956+0000","last_clean":"2024-03-23T03:51:22.276956+0000","last_became_active":"2024-03-23T03:51:21.785232+0000","last_became_peered":"2024-03-23T03:51:21.785232+0000","last_unstale":"2024-03-23T03:51:22.276956+0000","last_undegraded":"2024-03-23T03:51:22.276956+0000","last_fullsized":"2024-03-23T03:51:22.276956+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:27:33.193844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.282442+0000","last_change":"2024-03-23T03:51:22.278002+0000","last_active":"2024-03-23T03:51:22.282442+0000","last_peered":"2024-03-23T03:51:22.282442+0000","last_clean":"2024-03-23T03:51:22.282442+0000","last_became_active":"2024-03-23T03:51:22.277586+0000","last_became_peered":"2024-03-23T03:51:22.277586+0000","last_unstale":"2024-03-23T03:51:22.282442+0000","last_undegraded":"2024-03-23T03:51:22.282442+0000","last_fullsized":"2024-03-23T03:51:22.282442+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:27:17.201821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.285705+0000","last_change":"2024-03-23T03:51:21.783666+0000","last_active":"2024-03-23T03:51:22.285705+0000","last_peered":"2024-03-23T03:51:22.285705+0000","last_clean":"2024-03-23T03:51:22.285705+0000","last_became_active":"2024-03-23T03:51:21.783377+0000","last_became_peered":"2024-03-23T03:51:21.783377+0000","last_unstale":"2024-03-23T03:51:22.285705+0000","last_undegraded":"2024-03-23T03:51:22.285705+0000","last_fullsized":"2024-03-23T03:51:22.285705+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:42:08.794820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.287243+0000","last_change":"2024-03-23T03:51:22.284472+0000","last_active":"2024-03-23T03:51:22.287243+0000","last_peered":"2024-03-23T03:51:22.287243+0000","last_clean":"2024-03-23T03:51:22.287243+0000","last_became_active":"2024-03-23T03:51:22.284222+0000","last_became_peered":"2024-03-23T03:51:22.284222+0000","last_unstale":"2024-03-23T03:51:22.287243+0000","last_undegraded":"2024-03-23T03:51:22.287243+0000","last_fullsized":"2024-03-23T03:51:22.287243+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:17:09.963708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.253628+0000","last_change":"2024-03-23T03:51:21.792017+0000","last_active":"2024-03-23T03:51:22.253628+0000","last_peered":"2024-03-23T03:51:22.253628+0000","last_clean":"2024-03-23T03:51:22.253628+0000","last_became_active":"2024-03-23T03:51:21.791763+0000","last_became_peered":"2024-03-23T03:51:21.791763+0000","last_unstale":"2024-03-23T03:51:22.253628+0000","last_undegraded":"2024-03-23T03:51:22.253628+0000","last_fullsized":"2024-03-23T03:51:22.253628+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:05:09.454442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.260943+0000","last_change":"2024-03-23T03:51:22.255429+0000","last_active":"2024-03-23T03:51:22.260943+0000","last_peered":"2024-03-23T03:51:22.260943+0000","last_clean":"2024-03-23T03:51:22.260943+0000","last_became_active":"2024-03-23T03:51:22.253694+0000","last_became_peered":"2024-03-23T03:51:22.253694+0000","last_unstale":"2024-03-23T03:51:22.260943+0000","last_undegraded":"2024-03-23T03:51:22.260943+0000","last_fullsized":"2024-03-23T03:51:22.260943+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:38:27.787294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"104'1","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.279394+0000","last_change":"2024-03-23T03:51:21.792106+0000","last_active":"2024-03-23T03:51:22.279394+0000","last_peered":"2024-03-23T03:51:22.279394+0000","last_clean":"2024-03-23T03:51:22.279394+0000","last_became_active":"2024-03-23T03:51:21.791295+0000","last_became_peered":"2024-03-23T03:51:21.791295+0000","last_unstale":"2024-03-23T03:51:22.279394+0000","last_undegraded":"2024-03-23T03:51:22.279394+0000","last_fullsized":"2024-03-23T03:51:22.279394+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:00:35.520733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.279590+0000","last_change":"2024-03-23T03:51:22.277153+0000","last_active":"2024-03-23T03:51:22.279590+0000","last_peered":"2024-03-23T03:51:22.279590+0000","last_clean":"2024-03-23T03:51:22.279590+0000","last_became_active":"2024-03-23T03:51:22.276655+0000","last_became_peered":"2024-03-23T03:51:22.276655+0000","last_unstale":"2024-03-23T03:51:22.279590+0000","last_undegraded":"2024-03-23T03:51:22.279590+0000","last_fullsized":"2024-03-23T03:51:22.279590+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:37:27.534087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"104'1","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.284168+0000","last_change":"2024-03-23T03:51:21.780198+0000","last_active":"2024-03-23T03:51:22.284168+0000","last_peered":"2024-03-23T03:51:22.284168+0000","last_clean":"2024-03-23T03:51:22.284168+0000","last_became_active":"2024-03-23T03:51:21.779256+0000","last_became_peered":"2024-03-23T03:51:21.779256+0000","last_unstale":"2024-03-23T03:51:22.284168+0000","last_undegraded":"2024-03-23T03:51:22.284168+0000","last_fullsized":"2024-03-23T03:51:22.284168+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:00:33.834214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.276291+0000","last_change":"2024-03-23T03:51:22.273524+0000","last_active":"2024-03-23T03:51:22.276291+0000","last_peered":"2024-03-23T03:51:22.276291+0000","last_clean":"2024-03-23T03:51:22.276291+0000","last_became_active":"2024-03-23T03:51:22.270458+0000","last_became_peered":"2024-03-23T03:51:22.270458+0000","last_unstale":"2024-03-23T03:51:22.276291+0000","last_undegraded":"2024-03-23T03:51:22.276291+0000","last_fullsized":"2024-03-23T03:51:22.276291+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:50:44.800098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.269859+0000","last_change":"2024-03-23T03:51:21.783223+0000","last_active":"2024-03-23T03:51:22.269859+0000","last_peered":"2024-03-23T03:51:22.269859+0000","last_clean":"2024-03-23T03:51:22.269859+0000","last_became_active":"2024-03-23T03:51:21.782799+0000","last_became_peered":"2024-03-23T03:51:21.782799+0000","last_unstale":"2024-03-23T03:51:22.269859+0000","last_undegraded":"2024-03-23T03:51:22.269859+0000","last_fullsized":"2024-03-23T03:51:22.269859+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:56:44.890019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.272586+0000","last_change":"2024-03-23T03:51:22.263756+0000","last_active":"2024-03-23T03:51:22.272586+0000","last_peered":"2024-03-23T03:51:22.272586+0000","last_clean":"2024-03-23T03:51:22.272586+0000","last_became_active":"2024-03-23T03:51:22.263510+0000","last_became_peered":"2024-03-23T03:51:22.263510+0000","last_unstale":"2024-03-23T03:51:22.272586+0000","last_undegraded":"2024-03-23T03:51:22.272586+0000","last_fullsized":"2024-03-23T03:51:22.272586+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:31:57.940022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.246134+0000","last_change":"2024-03-23T03:51:21.776001+0000","last_active":"2024-03-23T03:51:22.246134+0000","last_peered":"2024-03-23T03:51:22.246134+0000","last_clean":"2024-03-23T03:51:22.246134+0000","last_became_active":"2024-03-23T03:51:21.775726+0000","last_became_peered":"2024-03-23T03:51:21.775726+0000","last_unstale":"2024-03-23T03:51:22.246134+0000","last_undegraded":"2024-03-23T03:51:22.246134+0000","last_fullsized":"2024-03-23T03:51:22.246134+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:07:52.649065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.279542+0000","last_change":"2024-03-23T03:51:22.277230+0000","last_active":"2024-03-23T03:51:22.279542+0000","last_peered":"2024-03-23T03:51:22.279542+0000","last_clean":"2024-03-23T03:51:22.279542+0000","last_became_active":"2024-03-23T03:51:22.276977+0000","last_became_peered":"2024-03-23T03:51:22.276977+0000","last_unstale":"2024-03-23T03:51:22.279542+0000","last_undegraded":"2024-03-23T03:51:22.279542+0000","last_fullsized":"2024-03-23T03:51:22.279542+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:55:30.397347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.273372+0000","last_change":"2024-03-23T03:51:21.786526+0000","last_active":"2024-03-23T03:51:22.273372+0000","last_peered":"2024-03-23T03:51:22.273372+0000","last_clean":"2024-03-23T03:51:22.273372+0000","last_became_active":"2024-03-23T03:51:21.783040+0000","last_became_peered":"2024-03-23T03:51:21.783040+0000","last_unstale":"2024-03-23T03:51:22.273372+0000","last_undegraded":"2024-03-23T03:51:22.273372+0000","last_fullsized":"2024-03-23T03:51:22.273372+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:53.003075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.264940+0000","last_change":"2024-03-23T03:51:22.258821+0000","last_active":"2024-03-23T03:51:22.264940+0000","last_peered":"2024-03-23T03:51:22.264940+0000","last_clean":"2024-03-23T03:51:22.264940+0000","last_became_active":"2024-03-23T03:51:22.258581+0000","last_became_peered":"2024-03-23T03:51:22.258581+0000","last_unstale":"2024-03-23T03:51:22.264940+0000","last_undegraded":"2024-03-23T03:51:22.264940+0000","last_fullsized":"2024-03-23T03:51:22.264940+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:42:06.329388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.259334+0000","last_change":"2024-03-23T03:51:21.787895+0000","last_active":"2024-03-23T03:51:22.259334+0000","last_peered":"2024-03-23T03:51:22.259334+0000","last_clean":"2024-03-23T03:51:22.259334+0000","last_became_active":"2024-03-23T03:51:21.787627+0000","last_became_peered":"2024-03-23T03:51:21.787627+0000","last_unstale":"2024-03-23T03:51:22.259334+0000","last_undegraded":"2024-03-23T03:51:22.259334+0000","last_fullsized":"2024-03-23T03:51:22.259334+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:31:02.266262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.269389+0000","last_change":"2024-03-23T03:51:22.266962+0000","last_active":"2024-03-23T03:51:22.269389+0000","last_peered":"2024-03-23T03:51:22.269389+0000","last_clean":"2024-03-23T03:51:22.269389+0000","last_became_active":"2024-03-23T03:51:22.266486+0000","last_became_peered":"2024-03-23T03:51:22.266486+0000","last_unstale":"2024-03-23T03:51:22.269389+0000","last_undegraded":"2024-03-23T03:51:22.269389+0000","last_fullsized":"2024-03-23T03:51:22.269389+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:34:16.439779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"104'3","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.275377+0000","last_change":"2024-03-23T03:51:21.786687+0000","last_active":"2024-03-23T03:51:22.275377+0000","last_peered":"2024-03-23T03:51:22.275377+0000","last_clean":"2024-03-23T03:51:22.275377+0000","last_became_active":"2024-03-23T03:51:21.786374+0000","last_became_peered":"2024-03-23T03:51:21.786374+0000","last_unstale":"2024-03-23T03:51:22.275377+0000","last_undegraded":"2024-03-23T03:51:22.275377+0000","last_fullsized":"2024-03-23T03:51:22.275377+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:09:59.406767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.265284+0000","last_change":"2024-03-23T03:51:22.259503+0000","last_active":"2024-03-23T03:51:22.265284+0000","last_peered":"2024-03-23T03:51:22.265284+0000","last_clean":"2024-03-23T03:51:22.265284+0000","last_became_active":"2024-03-23T03:51:22.256943+0000","last_became_peered":"2024-03-23T03:51:22.256943+0000","last_unstale":"2024-03-23T03:51:22.265284+0000","last_undegraded":"2024-03-23T03:51:22.265284+0000","last_fullsized":"2024-03-23T03:51:22.265284+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:19:44.510925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"104'6","reported_seq":17,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.289168+0000","last_change":"2024-03-23T03:51:21.789690+0000","last_active":"2024-03-23T03:51:22.289168+0000","last_peered":"2024-03-23T03:51:22.289168+0000","last_clean":"2024-03-23T03:51:22.289168+0000","last_became_active":"2024-03-23T03:51:21.789409+0000","last_became_peered":"2024-03-23T03:51:21.789409+0000","last_unstale":"2024-03-23T03:51:22.289168+0000","last_undegraded":"2024-03-23T03:51:22.289168+0000","last_fullsized":"2024-03-23T03:51:22.289168+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:03:22.934141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.283171+0000","last_change":"2024-03-23T03:51:22.271629+0000","last_active":"2024-03-23T03:51:22.283171+0000","last_peered":"2024-03-23T03:51:22.283171+0000","last_clean":"2024-03-23T03:51:22.283171+0000","last_became_active":"2024-03-23T03:51:22.271350+0000","last_became_peered":"2024-03-23T03:51:22.271350+0000","last_unstale":"2024-03-23T03:51:22.283171+0000","last_undegraded":"2024-03-23T03:51:22.283171+0000","last_fullsized":"2024-03-23T03:51:22.283171+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:04:22.944014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.256406+0000","last_change":"2024-03-23T03:51:21.778247+0000","last_active":"2024-03-23T03:51:22.256406+0000","last_peered":"2024-03-23T03:51:22.256406+0000","last_clean":"2024-03-23T03:51:22.256406+0000","last_became_active":"2024-03-23T03:51:21.777903+0000","last_became_peered":"2024-03-23T03:51:21.777903+0000","last_unstale":"2024-03-23T03:51:22.256406+0000","last_undegraded":"2024-03-23T03:51:22.256406+0000","last_fullsized":"2024-03-23T03:51:22.256406+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:30:03.239118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.274568+0000","last_change":"2024-03-23T03:51:22.265915+0000","last_active":"2024-03-23T03:51:22.274568+0000","last_peered":"2024-03-23T03:51:22.274568+0000","last_clean":"2024-03-23T03:51:22.274568+0000","last_became_active":"2024-03-23T03:51:22.265640+0000","last_became_peered":"2024-03-23T03:51:22.265640+0000","last_unstale":"2024-03-23T03:51:22.274568+0000","last_undegraded":"2024-03-23T03:51:22.274568+0000","last_fullsized":"2024-03-23T03:51:22.274568+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:06:14.520077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.246653+0000","last_change":"2024-03-23T03:51:21.780251+0000","last_active":"2024-03-23T03:51:22.246653+0000","last_peered":"2024-03-23T03:51:22.246653+0000","last_clean":"2024-03-23T03:51:22.246653+0000","last_became_active":"2024-03-23T03:51:21.779927+0000","last_became_peered":"2024-03-23T03:51:21.779927+0000","last_unstale":"2024-03-23T03:51:22.246653+0000","last_undegraded":"2024-03-23T03:51:22.246653+0000","last_fullsized":"2024-03-23T03:51:22.246653+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:28:51.270476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.288871+0000","last_change":"2024-03-23T03:51:22.285115+0000","last_active":"2024-03-23T03:51:22.288871+0000","last_peered":"2024-03-23T03:51:22.288871+0000","last_clean":"2024-03-23T03:51:22.288871+0000","last_became_active":"2024-03-23T03:51:22.284771+0000","last_became_peered":"2024-03-23T03:51:22.284771+0000","last_unstale":"2024-03-23T03:51:22.288871+0000","last_undegraded":"2024-03-23T03:51:22.288871+0000","last_fullsized":"2024-03-23T03:51:22.288871+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:53.513182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.249815+0000","last_change":"2024-03-23T03:51:21.790264+0000","last_active":"2024-03-23T03:51:22.249815+0000","last_peered":"2024-03-23T03:51:22.249815+0000","last_clean":"2024-03-23T03:51:22.249815+0000","last_became_active":"2024-03-23T03:51:21.789990+0000","last_became_peered":"2024-03-23T03:51:21.789990+0000","last_unstale":"2024-03-23T03:51:22.249815+0000","last_undegraded":"2024-03-23T03:51:22.249815+0000","last_fullsized":"2024-03-23T03:51:22.249815+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:17:42.968056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.286366+0000","last_change":"2024-03-23T03:51:22.280933+0000","last_active":"2024-03-23T03:51:22.286366+0000","last_peered":"2024-03-23T03:51:22.286366+0000","last_clean":"2024-03-23T03:51:22.286366+0000","last_became_active":"2024-03-23T03:51:22.280679+0000","last_became_peered":"2024-03-23T03:51:22.280679+0000","last_unstale":"2024-03-23T03:51:22.286366+0000","last_undegraded":"2024-03-23T03:51:22.286366+0000","last_fullsized":"2024-03-23T03:51:22.286366+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:52:35.913771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"104'8","reported_seq":19,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.287974+0000","last_change":"2024-03-23T03:51:21.783083+0000","last_active":"2024-03-23T03:51:22.287974+0000","last_peered":"2024-03-23T03:51:22.287974+0000","last_clean":"2024-03-23T03:51:22.287974+0000","last_became_active":"2024-03-23T03:51:21.782771+0000","last_became_peered":"2024-03-23T03:51:21.782771+0000","last_unstale":"2024-03-23T03:51:22.287974+0000","last_undegraded":"2024-03-23T03:51:22.287974+0000","last_fullsized":"2024-03-23T03:51:22.287974+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:11:22.831901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.274824+0000","last_change":"2024-03-23T03:51:22.263700+0000","last_active":"2024-03-23T03:51:22.274824+0000","last_peered":"2024-03-23T03:51:22.274824+0000","last_clean":"2024-03-23T03:51:22.274824+0000","last_became_active":"2024-03-23T03:51:22.263452+0000","last_became_peered":"2024-03-23T03:51:22.263452+0000","last_unstale":"2024-03-23T03:51:22.274824+0000","last_undegraded":"2024-03-23T03:51:22.274824+0000","last_fullsized":"2024-03-23T03:51:22.274824+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:35:48.145231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"104'1","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.286170+0000","last_change":"2024-03-23T03:51:21.793220+0000","last_active":"2024-03-23T03:51:22.286170+0000","last_peered":"2024-03-23T03:51:22.286170+0000","last_clean":"2024-03-23T03:51:22.286170+0000","last_became_active":"2024-03-23T03:51:21.792937+0000","last_became_peered":"2024-03-23T03:51:21.792937+0000","last_unstale":"2024-03-23T03:51:22.286170+0000","last_undegraded":"2024-03-23T03:51:22.286170+0000","last_fullsized":"2024-03-23T03:51:22.286170+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:52.502591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.276080+0000","last_change":"2024-03-23T03:51:22.271578+0000","last_active":"2024-03-23T03:51:22.276080+0000","last_peered":"2024-03-23T03:51:22.276080+0000","last_clean":"2024-03-23T03:51:22.276080+0000","last_became_active":"2024-03-23T03:51:22.271281+0000","last_became_peered":"2024-03-23T03:51:22.271281+0000","last_unstale":"2024-03-23T03:51:22.276080+0000","last_undegraded":"2024-03-23T03:51:22.276080+0000","last_fullsized":"2024-03-23T03:51:22.276080+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:31:10.713669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"104'1","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.277479+0000","last_change":"2024-03-23T03:51:21.792048+0000","last_active":"2024-03-23T03:51:22.277479+0000","last_peered":"2024-03-23T03:51:22.277479+0000","last_clean":"2024-03-23T03:51:22.277479+0000","last_became_active":"2024-03-23T03:51:21.791713+0000","last_became_peered":"2024-03-23T03:51:21.791713+0000","last_unstale":"2024-03-23T03:51:22.277479+0000","last_undegraded":"2024-03-23T03:51:22.277479+0000","last_fullsized":"2024-03-23T03:51:22.277479+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:59:01.379564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.278624+0000","last_change":"2024-03-23T03:51:22.269905+0000","last_active":"2024-03-23T03:51:22.278624+0000","last_peered":"2024-03-23T03:51:22.278624+0000","last_clean":"2024-03-23T03:51:22.278624+0000","last_became_active":"2024-03-23T03:51:22.269683+0000","last_became_peered":"2024-03-23T03:51:22.269683+0000","last_unstale":"2024-03-23T03:51:22.278624+0000","last_undegraded":"2024-03-23T03:51:22.278624+0000","last_fullsized":"2024-03-23T03:51:22.278624+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:59:59.276905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.249880+0000","last_change":"2024-03-23T03:51:21.789004+0000","last_active":"2024-03-23T03:51:22.249880+0000","last_peered":"2024-03-23T03:51:22.249880+0000","last_clean":"2024-03-23T03:51:22.249880+0000","last_became_active":"2024-03-23T03:51:21.788647+0000","last_became_peered":"2024-03-23T03:51:21.788647+0000","last_unstale":"2024-03-23T03:51:22.249880+0000","last_undegraded":"2024-03-23T03:51:22.249880+0000","last_fullsized":"2024-03-23T03:51:22.249880+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:59:01.379564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.267588+0000","last_change":"2024-03-23T03:51:22.263556+0000","last_active":"2024-03-23T03:51:22.267588+0000","last_peered":"2024-03-23T03:51:22.267588+0000","last_clean":"2024-03-23T03:51:22.267588+0000","last_became_active":"2024-03-23T03:51:22.263345+0000","last_became_peered":"2024-03-23T03:51:22.263345+0000","last_unstale":"2024-03-23T03:51:22.267588+0000","last_undegraded":"2024-03-23T03:51:22.267588+0000","last_fullsized":"2024-03-23T03:51:22.267588+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:15:49.588458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.263002+0000","last_change":"2024-03-23T03:51:21.782473+0000","last_active":"2024-03-23T03:51:22.263002+0000","last_peered":"2024-03-23T03:51:22.263002+0000","last_clean":"2024-03-23T03:51:22.263002+0000","last_became_active":"2024-03-23T03:51:21.781740+0000","last_became_peered":"2024-03-23T03:51:21.781740+0000","last_unstale":"2024-03-23T03:51:22.263002+0000","last_undegraded":"2024-03-23T03:51:22.263002+0000","last_fullsized":"2024-03-23T03:51:22.263002+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:30:22.629109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.270955+0000","last_change":"2024-03-23T03:51:22.269215+0000","last_active":"2024-03-23T03:51:22.270955+0000","last_peered":"2024-03-23T03:51:22.270955+0000","last_clean":"2024-03-23T03:51:22.270955+0000","last_became_active":"2024-03-23T03:51:22.268957+0000","last_became_peered":"2024-03-23T03:51:22.268957+0000","last_unstale":"2024-03-23T03:51:22.270955+0000","last_undegraded":"2024-03-23T03:51:22.270955+0000","last_fullsized":"2024-03-23T03:51:22.270955+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:18:12.413501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.250313+0000","last_change":"2024-03-23T03:51:21.782262+0000","last_active":"2024-03-23T03:51:22.250313+0000","last_peered":"2024-03-23T03:51:22.250313+0000","last_clean":"2024-03-23T03:51:22.250313+0000","last_became_active":"2024-03-23T03:51:21.781940+0000","last_became_peered":"2024-03-23T03:51:21.781940+0000","last_unstale":"2024-03-23T03:51:22.250313+0000","last_undegraded":"2024-03-23T03:51:22.250313+0000","last_fullsized":"2024-03-23T03:51:22.250313+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:49.654183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.275604+0000","last_change":"2024-03-23T03:51:22.273117+0000","last_active":"2024-03-23T03:51:22.275604+0000","last_peered":"2024-03-23T03:51:22.275604+0000","last_clean":"2024-03-23T03:51:22.275604+0000","last_became_active":"2024-03-23T03:51:22.272861+0000","last_became_peered":"2024-03-23T03:51:22.272861+0000","last_unstale":"2024-03-23T03:51:22.275604+0000","last_undegraded":"2024-03-23T03:51:22.275604+0000","last_fullsized":"2024-03-23T03:51:22.275604+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:06:50.564546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.263230+0000","last_change":"2024-03-23T03:51:21.792924+0000","last_active":"2024-03-23T03:51:22.263230+0000","last_peered":"2024-03-23T03:51:22.263230+0000","last_clean":"2024-03-23T03:51:22.263230+0000","last_became_active":"2024-03-23T03:51:21.790283+0000","last_became_peered":"2024-03-23T03:51:21.790283+0000","last_unstale":"2024-03-23T03:51:22.263230+0000","last_undegraded":"2024-03-23T03:51:22.263230+0000","last_fullsized":"2024-03-23T03:51:22.263230+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:19:57.445124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.269326+0000","last_change":"2024-03-23T03:51:22.265543+0000","last_active":"2024-03-23T03:51:22.269326+0000","last_peered":"2024-03-23T03:51:22.269326+0000","last_clean":"2024-03-23T03:51:22.269326+0000","last_became_active":"2024-03-23T03:51:22.265310+0000","last_became_peered":"2024-03-23T03:51:22.265310+0000","last_unstale":"2024-03-23T03:51:22.269326+0000","last_undegraded":"2024-03-23T03:51:22.269326+0000","last_fullsized":"2024-03-23T03:51:22.269326+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:35.027397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.249976+0000","last_change":"2024-03-23T03:51:21.784479+0000","last_active":"2024-03-23T03:51:22.249976+0000","last_peered":"2024-03-23T03:51:22.249976+0000","last_clean":"2024-03-23T03:51:22.249976+0000","last_became_active":"2024-03-23T03:51:21.784166+0000","last_became_peered":"2024-03-23T03:51:21.784166+0000","last_unstale":"2024-03-23T03:51:22.249976+0000","last_undegraded":"2024-03-23T03:51:22.249976+0000","last_fullsized":"2024-03-23T03:51:22.249976+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:35:04.474544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.267498+0000","last_change":"2024-03-23T03:51:22.263572+0000","last_active":"2024-03-23T03:51:22.267498+0000","last_peered":"2024-03-23T03:51:22.267498+0000","last_clean":"2024-03-23T03:51:22.267498+0000","last_became_active":"2024-03-23T03:51:22.263304+0000","last_became_peered":"2024-03-23T03:51:22.263304+0000","last_unstale":"2024-03-23T03:51:22.267498+0000","last_undegraded":"2024-03-23T03:51:22.267498+0000","last_fullsized":"2024-03-23T03:51:22.267498+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:08:03.409470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.256089+0000","last_change":"2024-03-23T03:51:21.776941+0000","last_active":"2024-03-23T03:51:22.256089+0000","last_peered":"2024-03-23T03:51:22.256089+0000","last_clean":"2024-03-23T03:51:22.256089+0000","last_became_active":"2024-03-23T03:51:21.776643+0000","last_became_peered":"2024-03-23T03:51:21.776643+0000","last_unstale":"2024-03-23T03:51:22.256089+0000","last_undegraded":"2024-03-23T03:51:22.256089+0000","last_fullsized":"2024-03-23T03:51:22.256089+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:17:08.953118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.282153+0000","last_change":"2024-03-23T03:51:22.278810+0000","last_active":"2024-03-23T03:51:22.282153+0000","last_peered":"2024-03-23T03:51:22.282153+0000","last_clean":"2024-03-23T03:51:22.282153+0000","last_became_active":"2024-03-23T03:51:22.278506+0000","last_became_peered":"2024-03-23T03:51:22.278506+0000","last_unstale":"2024-03-23T03:51:22.282153+0000","last_undegraded":"2024-03-23T03:51:22.282153+0000","last_fullsized":"2024-03-23T03:51:22.282153+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:46:03.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"105'1","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T03:51:23.268762+0000","last_change":"2024-03-23T03:51:21.785341+0000","last_active":"2024-03-23T03:51:23.268762+0000","last_peered":"2024-03-23T03:51:23.268762+0000","last_clean":"2024-03-23T03:51:23.268762+0000","last_became_active":"2024-03-23T03:51:21.784654+0000","last_became_peered":"2024-03-23T03:51:21.784654+0000","last_unstale":"2024-03-23T03:51:23.268762+0000","last_undegraded":"2024-03-23T03:51:23.268762+0000","last_fullsized":"2024-03-23T03:51:23.268762+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:36.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.272683+0000","last_change":"2024-03-23T03:51:22.265659+0000","last_active":"2024-03-23T03:51:22.272683+0000","last_peered":"2024-03-23T03:51:22.272683+0000","last_clean":"2024-03-23T03:51:22.272683+0000","last_became_active":"2024-03-23T03:51:22.265380+0000","last_became_peered":"2024-03-23T03:51:22.265380+0000","last_unstale":"2024-03-23T03:51:22.272683+0000","last_undegraded":"2024-03-23T03:51:22.272683+0000","last_fullsized":"2024-03-23T03:51:22.272683+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:27:15.422967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.257431+0000","last_change":"2024-03-23T03:51:21.788631+0000","last_active":"2024-03-23T03:51:22.257431+0000","last_peered":"2024-03-23T03:51:22.257431+0000","last_clean":"2024-03-23T03:51:22.257431+0000","last_became_active":"2024-03-23T03:51:21.788145+0000","last_became_peered":"2024-03-23T03:51:21.788145+0000","last_unstale":"2024-03-23T03:51:22.257431+0000","last_undegraded":"2024-03-23T03:51:22.257431+0000","last_fullsized":"2024-03-23T03:51:22.257431+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:45.506837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.265867+0000","last_change":"2024-03-23T03:51:22.259619+0000","last_active":"2024-03-23T03:51:22.265867+0000","last_peered":"2024-03-23T03:51:22.265867+0000","last_clean":"2024-03-23T03:51:22.265867+0000","last_became_active":"2024-03-23T03:51:22.259270+0000","last_became_peered":"2024-03-23T03:51:22.259270+0000","last_unstale":"2024-03-23T03:51:22.265867+0000","last_undegraded":"2024-03-23T03:51:22.265867+0000","last_fullsized":"2024-03-23T03:51:22.265867+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:30:04.249708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.241639+0000","last_change":"2024-03-23T03:51:21.788546+0000","last_active":"2024-03-23T03:51:22.241639+0000","last_peered":"2024-03-23T03:51:22.241639+0000","last_clean":"2024-03-23T03:51:22.241639+0000","last_became_active":"2024-03-23T03:51:21.788282+0000","last_became_peered":"2024-03-23T03:51:21.788282+0000","last_unstale":"2024-03-23T03:51:22.241639+0000","last_undegraded":"2024-03-23T03:51:22.241639+0000","last_fullsized":"2024-03-23T03:51:22.241639+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:04:20.308736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.261946+0000","last_change":"2024-03-23T03:51:22.258385+0000","last_active":"2024-03-23T03:51:22.261946+0000","last_peered":"2024-03-23T03:51:22.261946+0000","last_clean":"2024-03-23T03:51:22.261946+0000","last_became_active":"2024-03-23T03:51:22.258120+0000","last_became_peered":"2024-03-23T03:51:22.258120+0000","last_unstale":"2024-03-23T03:51:22.261946+0000","last_undegraded":"2024-03-23T03:51:22.261946+0000","last_fullsized":"2024-03-23T03:51:22.261946+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:05.542126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.265163+0000","last_change":"2024-03-23T03:51:21.793737+0000","last_active":"2024-03-23T03:51:22.265163+0000","last_peered":"2024-03-23T03:51:22.265163+0000","last_clean":"2024-03-23T03:51:22.265163+0000","last_became_active":"2024-03-23T03:51:21.791575+0000","last_became_peered":"2024-03-23T03:51:21.791575+0000","last_unstale":"2024-03-23T03:51:22.265163+0000","last_undegraded":"2024-03-23T03:51:22.265163+0000","last_fullsized":"2024-03-23T03:51:22.265163+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:54:20.064900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.269225+0000","last_change":"2024-03-23T03:51:22.266599+0000","last_active":"2024-03-23T03:51:22.269225+0000","last_peered":"2024-03-23T03:51:22.269225+0000","last_clean":"2024-03-23T03:51:22.269225+0000","last_became_active":"2024-03-23T03:51:22.266281+0000","last_became_peered":"2024-03-23T03:51:22.266281+0000","last_unstale":"2024-03-23T03:51:22.269225+0000","last_undegraded":"2024-03-23T03:51:22.269225+0000","last_fullsized":"2024-03-23T03:51:22.269225+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:45:36.655749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.250446+0000","last_change":"2024-03-23T03:51:21.787337+0000","last_active":"2024-03-23T03:51:22.250446+0000","last_peered":"2024-03-23T03:51:22.250446+0000","last_clean":"2024-03-23T03:51:22.250446+0000","last_became_active":"2024-03-23T03:51:21.787076+0000","last_became_peered":"2024-03-23T03:51:21.787076+0000","last_unstale":"2024-03-23T03:51:22.250446+0000","last_undegraded":"2024-03-23T03:51:22.250446+0000","last_fullsized":"2024-03-23T03:51:22.250446+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:09:48.708095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.282566+0000","last_change":"2024-03-23T03:51:22.280399+0000","last_active":"2024-03-23T03:51:22.282566+0000","last_peered":"2024-03-23T03:51:22.282566+0000","last_clean":"2024-03-23T03:51:22.282566+0000","last_became_active":"2024-03-23T03:51:22.280129+0000","last_became_peered":"2024-03-23T03:51:22.280129+0000","last_unstale":"2024-03-23T03:51:22.282566+0000","last_undegraded":"2024-03-23T03:51:22.282566+0000","last_fullsized":"2024-03-23T03:51:22.282566+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:05:10.465032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.263361+0000","last_change":"2024-03-23T03:51:21.789939+0000","last_active":"2024-03-23T03:51:22.263361+0000","last_peered":"2024-03-23T03:51:22.263361+0000","last_clean":"2024-03-23T03:51:22.263361+0000","last_became_active":"2024-03-23T03:51:21.788412+0000","last_became_peered":"2024-03-23T03:51:21.788412+0000","last_unstale":"2024-03-23T03:51:22.263361+0000","last_undegraded":"2024-03-23T03:51:22.263361+0000","last_fullsized":"2024-03-23T03:51:22.263361+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:35:47.134641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.283863+0000","last_change":"2024-03-23T03:51:22.272329+0000","last_active":"2024-03-23T03:51:22.283863+0000","last_peered":"2024-03-23T03:51:22.283863+0000","last_clean":"2024-03-23T03:51:22.283863+0000","last_became_active":"2024-03-23T03:51:22.271506+0000","last_became_peered":"2024-03-23T03:51:22.271506+0000","last_unstale":"2024-03-23T03:51:22.283863+0000","last_undegraded":"2024-03-23T03:51:22.283863+0000","last_fullsized":"2024-03-23T03:51:22.283863+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:05:03.516080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.249914+0000","last_change":"2024-03-23T03:51:21.791827+0000","last_active":"2024-03-23T03:51:22.249914+0000","last_peered":"2024-03-23T03:51:22.249914+0000","last_clean":"2024-03-23T03:51:22.249914+0000","last_became_active":"2024-03-23T03:51:21.791445+0000","last_became_peered":"2024-03-23T03:51:21.791445+0000","last_unstale":"2024-03-23T03:51:22.249914+0000","last_undegraded":"2024-03-23T03:51:22.249914+0000","last_fullsized":"2024-03-23T03:51:22.249914+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:58.150428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.275945+0000","last_change":"2024-03-23T03:51:22.270121+0000","last_active":"2024-03-23T03:51:22.275945+0000","last_peered":"2024-03-23T03:51:22.275945+0000","last_clean":"2024-03-23T03:51:22.275945+0000","last_became_active":"2024-03-23T03:51:22.269806+0000","last_became_peered":"2024-03-23T03:51:22.269806+0000","last_unstale":"2024-03-23T03:51:22.275945+0000","last_undegraded":"2024-03-23T03:51:22.275945+0000","last_fullsized":"2024-03-23T03:51:22.275945+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:04:21.319327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.257587+0000","last_change":"2024-03-23T03:51:21.779521+0000","last_active":"2024-03-23T03:51:22.257587+0000","last_peered":"2024-03-23T03:51:22.257587+0000","last_clean":"2024-03-23T03:51:22.257587+0000","last_became_active":"2024-03-23T03:51:21.779111+0000","last_became_peered":"2024-03-23T03:51:21.779111+0000","last_unstale":"2024-03-23T03:51:22.257587+0000","last_undegraded":"2024-03-23T03:51:22.257587+0000","last_fullsized":"2024-03-23T03:51:22.257587+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:29:39.956286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":230,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.260723+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:51:22.260723+0000","last_peered":"2024-03-23T03:51:22.260723+0000","last_clean":"2024-03-23T03:51:22.260723+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:51:22.260723+0000","last_undegraded":"2024-03-23T03:51:22.260723+0000","last_fullsized":"2024-03-23T03:51:22.260723+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542204,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32980,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338860,"statfs":{"total":96636764160,"available":96602992640,"internally_reserved":0,"allocated":1900544,"data_stored":1180111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.087,"15min":1.0720000000000001},"min":{"1min":0.65700000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5209999999999999,"5min":1.635,"15min":1.635},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.042,"15min":1.0469999999999999},"min":{"1min":0.749,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.393,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.0049999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.081,"15min":1.0529999999999999},"min":{"1min":0.71499999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6120000000000001,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.147},{"interface":"front","average":{"1min":1.03,"5min":1.1719999999999999,"15min":1.2809999999999999},"min":{"1min":0.79000000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.321,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":1.5109999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.125,"15min":1.151},"min":{"1min":0.81299999999999994,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.583,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.171},{"interface":"front","average":{"1min":1.167,"5min":1.2869999999999999,"15min":1.401},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.506,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.4990000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.298,"15min":1.4179999999999999},"min":{"1min":0.71999999999999997,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.4219999999999999,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.325,"15min":1.353},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.677,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.452}]},{"osd":4,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.254,"15min":1.2629999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.8220000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.272},{"interface":"front","average":{"1min":1.333,"5min":1.347,"15min":1.3720000000000001},"min":{"1min":0.67500000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.778,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":0.92900000000000005}]},{"osd":5,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.389,"15min":1.4179999999999999},"min":{"1min":0.81299999999999994,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.7310000000000001,"5min":2.645,"15min":2.645},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.361,"5min":1.417,"15min":1.446},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.7649999999999999,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.119}]},{"osd":6,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.504,"15min":1.552},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.9279999999999999,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.421},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.474,"15min":1.5149999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9399999999999999,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.321}]}]},{"osd":6,"up_from":14,"seq":60129542202,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32404,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339436,"statfs":{"total":96636764160,"available":96603582464,"internally_reserved":0,"allocated":1318912,"data_stored":591751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.0429999999999999,"15min":1.044},"min":{"1min":0.69699999999999995,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.05,"15min":1.04},"min":{"1min":0.66100000000000003,"5min":0.62,"15min":0.62},"max":{"1min":1.5409999999999999,"5min":1.946,"15min":1.946},"last":1.0580000000000001}]},{"osd":1,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.087,"15min":1.075},"min":{"1min":0.745,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":1.204},{"interface":"front","average":{"1min":1.196,"5min":1.1339999999999999,"15min":1.1579999999999999},"min":{"1min":0.80500000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.5369999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.1439999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.083,"15min":1.071},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.4350000000000001,"5min":2.044,"15min":2.044},"last":1.327},{"interface":"front","average":{"1min":1.133,"5min":1.1759999999999999,"15min":1.212},"min":{"1min":0.85499999999999998,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.6599999999999999,"5min":1.681,"15min":1.681},"last":1.109}]},{"osd":3,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.2370000000000001,"15min":1.276},"min":{"1min":0.88100000000000001,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.786,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.2330000000000001,"15min":1.2569999999999999},"min":{"1min":0.90100000000000002,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.446,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.256}]},{"osd":4,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3779999999999999,"15min":1.385},"min":{"1min":1.1930000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.6679999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.431,"5min":1.371,"15min":1.3859999999999999},"min":{"1min":1.234,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.639,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.52}]},{"osd":5,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.403,"15min":1.4319999999999999},"min":{"1min":1.0669999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.794,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.474,"5min":1.508,"15min":1.542},"min":{"1min":1.091,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8200000000000001,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.5669999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.4059999999999999,"15min":1.4119999999999999},"min":{"1min":0.95999999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.0760000000000001,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.482,"15min":1.5309999999999999},"min":{"1min":1.125,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.8660000000000001,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.373}]}]},{"osd":5,"up_from":14,"seq":60129542202,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1298432,"data_stored":589583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1439999999999999,"15min":1.2030000000000001},"min":{"1min":0.72999999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4830000000000001,"5min":1.806,"15min":1.806},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.113,"5min":1.1399999999999999,"15min":1.1759999999999999},"min":{"1min":0.76500000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.855,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.93300000000000005}]},{"osd":1,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0940000000000001,"15min":1.1000000000000001},"min":{"1min":0.80100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.3520000000000001,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.232,"5min":1.165,"15min":1.1930000000000001},"min":{"1min":0.84799999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.663,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.167}]},{"osd":2,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.274,"15min":1.3460000000000001},"min":{"1min":0.70499999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8939999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.226,"5min":1.2989999999999999,"15min":1.3660000000000001},"min":{"1min":0.875,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.579,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.88400000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.2,"15min":1.244},"min":{"1min":0.85399999999999998,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.7050000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.375,"15min":1.446},"min":{"1min":0.997,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.964,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.3340000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3049999999999999,"15min":1.399},"min":{"1min":0.65700000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6100000000000001,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.038},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3859999999999999,"15min":1.4359999999999999},"min":{"1min":0.83699999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8069999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.4370000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.4159999999999999,"15min":1.5},"min":{"1min":0.83399999999999996,"5min":0.63,"15min":0.63},"max":{"1min":1.5840000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.5409999999999999,"15min":1.639},"min":{"1min":1.1950000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.6359999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.3049999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4910000000000001,"15min":1.591},"min":{"1min":1.095,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.9259999999999999,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.268},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5840000000000001,"15min":1.6839999999999999},"min":{"1min":1.272,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.056,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.3939999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542203,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32392,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339448,"statfs":{"total":96636764160,"available":96603594752,"internally_reserved":0,"allocated":1306624,"data_stored":589675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.004,"15min":0.98099999999999998},"min":{"1min":0.624,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.47,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.995,"15min":1.0049999999999999},"min":{"1min":0.56100000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3400000000000001,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.84999999999999998}]},{"osd":1,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0680000000000001,"15min":1.05},"min":{"1min":0.65100000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.5,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.226},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0880000000000001,"15min":1.0640000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.5209999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.272}]},{"osd":2,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.2470000000000001,"15min":1.3089999999999999},"min":{"1min":0.70399999999999996,"5min":0.625,"15min":0.625},"max":{"1min":1.452,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.2789999999999999,"15min":1.2969999999999999},"min":{"1min":0.59899999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6859999999999999,"5min":2.181,"15min":2.181},"last":1.4510000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.1160000000000001,"15min":1.1240000000000001},"min":{"1min":0.61899999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.4419999999999999,"5min":1.821,"15min":1.821},"last":1.097},{"interface":"front","average":{"1min":1.141,"5min":1.2310000000000001,"15min":1.268},"min":{"1min":0.73799999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.599,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.0529999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.2370000000000001,"15min":1.26},"min":{"1min":0.63800000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.49,"5min":1.929,"15min":1.929},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.2709999999999999,"15min":1.274},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.45,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.3959999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.371,"15min":1.409},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6719999999999999,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.4139999999999999,"15min":1.3999999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.8100000000000001,"5min":2.016,"15min":2.016},"last":1.5580000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.361,"15min":1.363},"min":{"1min":0.68200000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.5569999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.456,"15min":1.4870000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.716,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.494}]}]},{"osd":3,"up_from":14,"seq":60129542203,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1302528,"data_stored":590165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.81999999999999995,"15min":0.85499999999999998},"min":{"1min":0.35499999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.4419999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":1.48},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.92200000000000004,"15min":0.92400000000000004},"min":{"1min":0.502,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.538,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":1.0129999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.91500000000000004,"15min":0.96799999999999997},"min":{"1min":0.58099999999999996,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.9990000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.0229999999999999,"15min":1.069},"min":{"1min":0.52000000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.651,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.77000000000000002}]},{"osd":2,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.002,"15min":1.0509999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.5710000000000002,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1240000000000001,"15min":1.1499999999999999},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.7649999999999999,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":1.2170000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.2410000000000001,"15min":1.248},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":4.0380000000000003,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.337,"5min":1.351,"15min":1.429},"min":{"1min":0.77000000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.3450000000000002,"5min":4.819,"15min":4.819},"last":1.1899999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.6040000000000001,"15min":1.4470000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.2679999999999998,"5min":30.228000000000002,"15min":30.228000000000002},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.337,"15min":1.357},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":4.3630000000000004,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":1.115}]},{"osd":6,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4119999999999999,"15min":1.508},"min":{"1min":0.83999999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.6589999999999998,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.302},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4199999999999999,"15min":1.4039999999999999},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.4039999999999999,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":1.409}]},{"osd":7,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.389,"15min":1.4199999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.0880000000000001,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":1.264},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.542,"15min":1.5900000000000001},"min":{"1min":0.80500000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.4780000000000002,"5min":10.663,"15min":10.663},"last":1.377}]}]},{"osd":2,"up_from":14,"seq":60129542203,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1302528,"data_stored":589617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83799999999999997,"15min":0.877},"min":{"1min":0.52100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":1.4259999999999999,"5min":1.456,"15min":1.456},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.86399999999999999,"15min":0.93600000000000005},"min":{"1min":0.39500000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.4690000000000001,"5min":2.226,"15min":2.226},"last":0.65500000000000003}]},{"osd":1,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.93899999999999995,"15min":0.98499999999999999},"min":{"1min":0.46800000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.377,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.85899999999999999,"15min":0.91000000000000003},"min":{"1min":0.57599999999999996,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.2210000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.59399999999999997}]},{"osd":3,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.96099999999999997,"15min":1.018},"min":{"1min":0.47499999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.403,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.032,"5min":1.042,"15min":1.0800000000000001},"min":{"1min":0.61099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.63300000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.194,"15min":1.2310000000000001},"min":{"1min":0.82399999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.7270000000000001,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.157,"15min":1.2070000000000001},"min":{"1min":0.83799999999999997,"5min":0.627,"15min":0.627},"max":{"1min":1.4950000000000001,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.1950000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.3100000000000001,"15min":1.367},"min":{"1min":0.76900000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.6259999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.2749999999999999,"15min":1.339},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.819,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.0669999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.232,"15min":1.3280000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6479999999999999,"5min":2.456,"15min":2.456},"last":1.272},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3999999999999999,"15min":1.476},"min":{"1min":0.85299999999999998,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.097,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.0820000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3069999999999999,"15min":1.369},"min":{"1min":0.79800000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.171},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.302,"15min":1.395},"min":{"1min":0.78700000000000003,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5860000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.3700000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542202,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1302528,"data_stored":589617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79100000000000004,"15min":0.86099999999999999},"min":{"1min":0.41599999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.1240000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81799999999999995,"15min":0.82999999999999996},"min":{"1min":0.435,"5min":0.374,"15min":0.374},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.69199999999999995}]},{"osd":2,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.92200000000000004,"15min":0.96399999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.4430000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.75},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.92400000000000004,"15min":0.97799999999999998},"min":{"1min":0.496,"5min":0.443,"15min":0.443},"max":{"1min":1.2210000000000001,"5min":1.625,"15min":1.625},"last":0.89500000000000002}]},{"osd":3,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.103,"15min":1.113},"min":{"1min":0.68400000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.6559999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":0.875,"5min":0.96199999999999997,"15min":1.008},"min":{"1min":0.59499999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.6180000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.94499999999999995}]},{"osd":4,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.179,"15min":1.2749999999999999},"min":{"1min":0.67700000000000005,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.5069999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.246,"15min":1.3049999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6120000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.165}]},{"osd":5,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.248,"15min":1.3200000000000001},"min":{"1min":0.79500000000000004,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.95,"5min":3.593,"15min":3.593},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.264,"5min":1.327,"15min":1.4259999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.671,"5min":2.359,"15min":2.359},"last":1.4039999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.3089999999999999,"15min":1.3919999999999999},"min":{"1min":0.84199999999999997,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6200000000000001,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.357},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.288,"15min":1.3540000000000001},"min":{"1min":0.89100000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.0209999999999999,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.1399999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.359,"15min":1.4330000000000001},"min":{"1min":0.99299999999999999,"5min":0.627,"15min":0.627},"max":{"1min":2.0840000000000001,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.363,"15min":1.486},"min":{"1min":0.76600000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.673,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.4530000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542202,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32988,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338852,"statfs":{"total":96636764160,"available":96602984448,"internally_reserved":0,"allocated":1908736,"data_stored":1181605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.86399999999999999,"15min":0.93500000000000005},"min":{"1min":0.48399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.478,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.94799999999999995,"15min":0.98199999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.456,"5min":1.506,"15min":1.506},"last":1.2709999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.92700000000000005,"15min":0.99099999999999999},"min":{"1min":0.52900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.4399999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.0349999999999999,"15min":1.095},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.601,"5min":1.901,"15min":1.901},"last":0.95999999999999996}]},{"osd":3,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.93300000000000005,"15min":0.996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.474,"5min":1.972,"15min":1.972},"last":0.878},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0669999999999999,"15min":1.147},"min":{"1min":0.60299999999999998,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.597,"5min":2.298,"15min":2.298},"last":1.0449999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.288,"15min":1.3360000000000001},"min":{"1min":0.72799999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.7090000000000001,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":1.367},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.2230000000000001,"15min":1.27},"min":{"1min":0.80600000000000005,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.8380000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.1100000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3100000000000001,"15min":1.3700000000000001},"min":{"1min":0.89500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.948,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.573},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.377,"15min":1.4379999999999999},"min":{"1min":0.83499999999999996,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.8700000000000001,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.5429999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.4099999999999999,"15min":1.5209999999999999},"min":{"1min":0.80000000000000004,"5min":0.747,"15min":0.747},"max":{"1min":1.978,"5min":3.032,"15min":3.032},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.411,"15min":1.4870000000000001},"min":{"1min":0.84699999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.1349999999999998,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.1919999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4390000000000001,"15min":1.506},"min":{"1min":0.90500000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1850000000000001,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.45},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3620000000000001,"15min":1.419},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.873,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.0049999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:51:32.159 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:51:32.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.373+0000 7fe706a00640 1 Processor -- start 2024-03-23T03:51:32.374 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.373+0000 7fe706a00640 1 -- start start 2024-03-23T03:51:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.375+0000 7fe706a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7000b39b0 0x7fe7000b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.375+0000 7fe706a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe70015c400 0x7fe70015c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.375+0000 7fe706a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70006c6f0 con 0x7fe70015b560 2024-03-23T03:51:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.375+0000 7fe706a00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7000b4370 con 0x7fe70015c400 2024-03-23T03:51:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.375+0000 7fe706a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe70006c9d0 con 0x7fe7000b39b0 2024-03-23T03:51:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.375+0000 7fe704a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe70015c400 0x7fe70015c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.375+0000 7fe6ff400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7000b39b0 0x7fe7000b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.375+0000 7fe704a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe70015c400 0x7fe70015c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:51942/0 (socket says 172.21.15.102:51942) 2024-03-23T03:51:32.376 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.375+0000 7fe704a00640 1 -- 172.21.15.102:0/2075211768 learned_addr learned my addr 172.21.15.102:0/2075211768 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:32.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.376+0000 7fe704a00640 1 -- 172.21.15.102:0/2075211768 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7000b39b0 msgr2=0x7fe7000b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:32.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.376+0000 7fe704a00640 1 --2- 172.21.15.102:0/2075211768 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7000b39b0 0x7fe7000b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:32.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.376+0000 7fe704a00640 1 -- 172.21.15.102:0/2075211768 >> v1:172.21.15.173:6789/0 conn(0x7fe70015b560 legacy=0x7fe70015b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:32.377 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.376+0000 7fe704a00640 1 -- 172.21.15.102:0/2075211768 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe70006ccb0 con 0x7fe70015c400 2024-03-23T03:51:32.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.378+0000 7fe704a00640 1 --2- 172.21.15.102:0/2075211768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe70015c400 0x7fe70015c890 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8009e50 tx=0x7fe6f802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9244d4488088d809 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:32.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.378+0000 7fe6fea00640 1 -- 172.21.15.102:0/2075211768 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f802caf0 con 0x7fe70015c400 2024-03-23T03:51:32.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.378+0000 7fe6fea00640 1 -- 172.21.15.102:0/2075211768 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f802cc50 con 0x7fe70015c400 2024-03-23T03:51:32.378 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.378+0000 7fe6fea00640 1 -- 172.21.15.102:0/2075211768 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f80343f0 con 0x7fe70015c400 2024-03-23T03:51:32.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.378+0000 7fe706a00640 1 -- 172.21.15.102:0/2075211768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe70015c400 msgr2=0x7fe70015c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:32.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.378+0000 7fe706a00640 1 --2- 172.21.15.102:0/2075211768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe70015c400 0x7fe70015c890 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8009e50 tx=0x7fe6f802bf50 comp rx=0 tx=0).stop 2024-03-23T03:51:32.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.379+0000 7fe706a00640 1 -- 172.21.15.102:0/2075211768 shutdown_connections 2024-03-23T03:51:32.379 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.379+0000 7fe706a00640 1 --2- 172.21.15.102:0/2075211768 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe70015c400 0x7fe70015c890 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:32.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.379+0000 7fe706a00640 1 --2- 172.21.15.102:0/2075211768 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe7000b39b0 0x7fe7000b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:32.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.379+0000 7fe706a00640 1 -- 172.21.15.102:0/2075211768 >> 172.21.15.102:0/2075211768 conn(0x7fe70006a3d0 msgr2=0x7fe7000b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:32.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.379+0000 7fe706a00640 1 -- 172.21.15.102:0/2075211768 shutdown_connections 2024-03-23T03:51:32.380 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.379+0000 7fe706a00640 1 -- 172.21.15.102:0/2075211768 wait complete. 2024-03-23T03:51:32.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.381+0000 7fe706a00640 1 Processor -- start 2024-03-23T03:51:32.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.381+0000 7fe706a00640 1 -- start start 2024-03-23T03:51:32.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.382+0000 7fe706a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7000b39b0 0x7fe7001f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:32.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.382+0000 7fe706a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe70015b560 0x7fe70014b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:32.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.382+0000 7fe706a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe70015c400 0x7fe700151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:32.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.382+0000 7fe706a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70014b670 con 0x7fe70015b560 2024-03-23T03:51:32.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.382+0000 7fe706a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7001524e0 con 0x7fe7000b39b0 2024-03-23T03:51:32.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.382+0000 7fe706a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7001f68d0 con 0x7fe70015c400 2024-03-23T03:51:32.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.382+0000 7fe6ff400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe70015b560 0x7fe70014b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:32.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.382+0000 7fe6ff400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe70015b560 0x7fe70014b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:42684/0 (socket says 172.21.15.102:42684) 2024-03-23T03:51:32.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.382+0000 7fe6ff400640 1 -- 172.21.15.102:0/1043634136 learned_addr learned my addr 172.21.15.102:0/1043634136 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:32.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.382+0000 7fe6ffe00640 1 --2- 172.21.15.102:0/1043634136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7000b39b0 0x7fe7001f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:32.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.382+0000 7fe704a00640 1 --2- 172.21.15.102:0/1043634136 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe70015c400 0x7fe700151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:32.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.382+0000 7fe6ff400640 1 -- 172.21.15.102:0/1043634136 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe70015c400 msgr2=0x7fe700151fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:32.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.382+0000 7fe6ff400640 1 --2- 172.21.15.102:0/1043634136 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe70015c400 0x7fe700151fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:32.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.382+0000 7fe6ff400640 1 -- 172.21.15.102:0/1043634136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7000b39b0 msgr2=0x7fe7001f5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:32.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.382+0000 7fe6ff400640 1 --2- 172.21.15.102:0/1043634136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7000b39b0 0x7fe7001f5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:32.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.382+0000 7fe6ff400640 1 -- 172.21.15.102:0/1043634136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f8009b30 con 0x7fe70015b560 2024-03-23T03:51:32.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.383+0000 7fe6ff400640 1 --2- 172.21.15.102:0/1043634136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe70015b560 0x7fe70014b130 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec00e870 tx=0x7fe6ec00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:32.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.383+0000 7fe6f7e00640 1 -- 172.21.15.102:0/1043634136 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec0094f0 con 0x7fe70015b560 2024-03-23T03:51:32.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.383+0000 7fe6f7e00640 1 -- 172.21.15.102:0/1043634136 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ec00c940 con 0x7fe70015b560 2024-03-23T03:51:32.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.383+0000 7fe6f7e00640 1 -- 172.21.15.102:0/1043634136 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6ec00ccf0 con 0x7fe70015b560 2024-03-23T03:51:32.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.383+0000 7fe706a00640 1 -- 172.21.15.102:0/1043634136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7001f6030 con 0x7fe70015b560 2024-03-23T03:51:32.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.383+0000 7fe706a00640 1 -- 172.21.15.102:0/1043634136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7001f65f0 con 0x7fe70015b560 2024-03-23T03:51:32.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.384+0000 7fe706a00640 1 -- 172.21.15.102:0/1043634136 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe70006ac20 con 0x7fe70015b560 2024-03-23T03:51:32.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.385+0000 7fe6f7e00640 1 -- 172.21.15.102:0/1043634136 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe6ec013470 con 0x7fe70015b560 2024-03-23T03:51:32.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.386+0000 7fe6f7e00640 1 --2- 172.21.15.102:0/1043634136 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe6e006ac00 0x7fe6e006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:32.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.386+0000 7fe6f7e00640 1 -- 172.21.15.102:0/1043634136 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe6ec0894f0 con 0x7fe70015b560 2024-03-23T03:51:32.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.386+0000 7fe6ffe00640 1 --2- 172.21.15.102:0/1043634136 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe6e006ac00 0x7fe6e006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:32.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.387+0000 7fe6ffe00640 1 --2- 172.21.15.102:0/1043634136 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe6e006ac00 0x7fe6e006d0c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0000f60 tx=0x7fe6f0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:32.391 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.391+0000 7fe6f7e00640 1 -- 172.21.15.102:0/1043634136 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe6ec054620 con 0x7fe70015b560 2024-03-23T03:51:32.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.582+0000 7fe706a00640 1 -- 172.21.15.102:0/1043634136 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe700073390 con 0x7fe6e006ac00 2024-03-23T03:51:32.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.594+0000 7fe6f7e00640 1 -- 172.21.15.102:0/1043634136 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287806 (secure 0 0 0) 0x7fe700073390 con 0x7fe6e006ac00 2024-03-23T03:51:32.597 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:32.600 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:51:32.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.601+0000 7fe706a00640 1 -- 172.21.15.102:0/1043634136 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe6e006ac00 msgr2=0x7fe6e006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:32.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.601+0000 7fe706a00640 1 --2- 172.21.15.102:0/1043634136 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe6e006ac00 0x7fe6e006d0c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fe6f0000f60 tx=0x7fe6f0009040 comp rx=0 tx=0).stop 2024-03-23T03:51:32.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.601+0000 7fe706a00640 1 -- 172.21.15.102:0/1043634136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe70015b560 msgr2=0x7fe70014b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:32.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.601+0000 7fe706a00640 1 --2- 172.21.15.102:0/1043634136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe70015b560 0x7fe70014b130 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fe6ec00e870 tx=0x7fe6ec00ed30 comp rx=0 tx=0).stop 2024-03-23T03:51:32.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.601+0000 7fe706a00640 1 -- 172.21.15.102:0/1043634136 shutdown_connections 2024-03-23T03:51:32.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.601+0000 7fe706a00640 1 --2- 172.21.15.102:0/1043634136 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe6e006ac00 0x7fe6e006d0c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:32.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.601+0000 7fe706a00640 1 --2- 172.21.15.102:0/1043634136 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe70015c400 0x7fe700151fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:32.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.601+0000 7fe706a00640 1 --2- 172.21.15.102:0/1043634136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe70015b560 0x7fe70014b130 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:32.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.601+0000 7fe706a00640 1 --2- 172.21.15.102:0/1043634136 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe7000b39b0 0x7fe7001f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:32.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.601+0000 7fe706a00640 1 -- 172.21.15.102:0/1043634136 >> 172.21.15.102:0/1043634136 conn(0x7fe70006a3d0 msgr2=0x7fe7001593d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:32.602 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.601+0000 7fe706a00640 1 -- 172.21.15.102:0/1043634136 shutdown_connections 2024-03-23T03:51:32.603 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.601+0000 7fe706a00640 1 -- 172.21.15.102:0/1043634136 wait complete. 2024-03-23T03:51:32.620 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":241,"stamp":"2024-03-23T03:51:31.803679+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":252144,"kb_used_data":11368,"kb_used_omap":12,"kb_used_meta":240627,"kb_avail":754722576,"statfs":{"total":773094113280,"available":772835917824,"internally_reserved":0,"allocated":11640832,"data_stored":5902124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":246402618},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.568910"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.267572+0000","last_change":"2024-03-23T03:51:22.263565+0000","last_active":"2024-03-23T03:51:22.267572+0000","last_peered":"2024-03-23T03:51:22.267572+0000","last_clean":"2024-03-23T03:51:22.267572+0000","last_became_active":"2024-03-23T03:51:22.263323+0000","last_became_peered":"2024-03-23T03:51:22.263323+0000","last_unstale":"2024-03-23T03:51:22.267572+0000","last_undegraded":"2024-03-23T03:51:22.267572+0000","last_fullsized":"2024-03-23T03:51:22.267572+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:24:42.534444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.288428+0000","last_change":"2024-03-23T03:51:22.286197+0000","last_active":"2024-03-23T03:51:22.288428+0000","last_peered":"2024-03-23T03:51:22.288428+0000","last_clean":"2024-03-23T03:51:22.288428+0000","last_became_active":"2024-03-23T03:51:22.280588+0000","last_became_peered":"2024-03-23T03:51:22.280588+0000","last_unstale":"2024-03-23T03:51:22.288428+0000","last_undegraded":"2024-03-23T03:51:22.288428+0000","last_fullsized":"2024-03-23T03:51:22.288428+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:11:35.553745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.286379+0000","last_change":"2024-03-23T03:51:22.281125+0000","last_active":"2024-03-23T03:51:22.286379+0000","last_peered":"2024-03-23T03:51:22.286379+0000","last_clean":"2024-03-23T03:51:22.286379+0000","last_became_active":"2024-03-23T03:51:22.280105+0000","last_became_peered":"2024-03-23T03:51:22.280105+0000","last_unstale":"2024-03-23T03:51:22.286379+0000","last_undegraded":"2024-03-23T03:51:22.286379+0000","last_fullsized":"2024-03-23T03:51:22.286379+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:20:25.913448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.280149+0000","last_change":"2024-03-23T03:51:22.278175+0000","last_active":"2024-03-23T03:51:22.280149+0000","last_peered":"2024-03-23T03:51:22.280149+0000","last_clean":"2024-03-23T03:51:22.280149+0000","last_became_active":"2024-03-23T03:51:22.277895+0000","last_became_peered":"2024-03-23T03:51:22.277895+0000","last_unstale":"2024-03-23T03:51:22.280149+0000","last_undegraded":"2024-03-23T03:51:22.280149+0000","last_fullsized":"2024-03-23T03:51:22.280149+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:36:30.827292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.283501+0000","last_change":"2024-03-23T03:51:22.279802+0000","last_active":"2024-03-23T03:51:22.283501+0000","last_peered":"2024-03-23T03:51:22.283501+0000","last_clean":"2024-03-23T03:51:22.283501+0000","last_became_active":"2024-03-23T03:51:22.278955+0000","last_became_peered":"2024-03-23T03:51:22.278955+0000","last_unstale":"2024-03-23T03:51:22.283501+0000","last_undegraded":"2024-03-23T03:51:22.283501+0000","last_fullsized":"2024-03-23T03:51:22.283501+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:46:33.274045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.287694+0000","last_change":"2024-03-23T03:51:22.279603+0000","last_active":"2024-03-23T03:51:22.287694+0000","last_peered":"2024-03-23T03:51:22.287694+0000","last_clean":"2024-03-23T03:51:22.287694+0000","last_became_active":"2024-03-23T03:51:22.279288+0000","last_became_peered":"2024-03-23T03:51:22.279288+0000","last_unstale":"2024-03-23T03:51:22.287694+0000","last_undegraded":"2024-03-23T03:51:22.287694+0000","last_fullsized":"2024-03-23T03:51:22.287694+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:15:44.577151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.288339+0000","last_change":"2024-03-23T03:51:22.279845+0000","last_active":"2024-03-23T03:51:22.288339+0000","last_peered":"2024-03-23T03:51:22.288339+0000","last_clean":"2024-03-23T03:51:22.288339+0000","last_became_active":"2024-03-23T03:51:22.279530+0000","last_became_peered":"2024-03-23T03:51:22.279530+0000","last_unstale":"2024-03-23T03:51:22.288339+0000","last_undegraded":"2024-03-23T03:51:22.288339+0000","last_fullsized":"2024-03-23T03:51:22.288339+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:19.362177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.274843+0000","last_change":"2024-03-23T03:51:22.267328+0000","last_active":"2024-03-23T03:51:22.274843+0000","last_peered":"2024-03-23T03:51:22.274843+0000","last_clean":"2024-03-23T03:51:22.274843+0000","last_became_active":"2024-03-23T03:51:22.267081+0000","last_became_peered":"2024-03-23T03:51:22.267081+0000","last_unstale":"2024-03-23T03:51:22.274843+0000","last_undegraded":"2024-03-23T03:51:22.274843+0000","last_fullsized":"2024-03-23T03:51:22.274843+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:23:32.227733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.278415+0000","last_change":"2024-03-23T03:51:22.275332+0000","last_active":"2024-03-23T03:51:22.278415+0000","last_peered":"2024-03-23T03:51:22.278415+0000","last_clean":"2024-03-23T03:51:22.278415+0000","last_became_active":"2024-03-23T03:51:22.274890+0000","last_became_peered":"2024-03-23T03:51:22.274890+0000","last_unstale":"2024-03-23T03:51:22.278415+0000","last_undegraded":"2024-03-23T03:51:22.278415+0000","last_fullsized":"2024-03-23T03:51:22.278415+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:54.013665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.278612+0000","last_change":"2024-03-23T03:51:22.273679+0000","last_active":"2024-03-23T03:51:22.278612+0000","last_peered":"2024-03-23T03:51:22.278612+0000","last_clean":"2024-03-23T03:51:22.278612+0000","last_became_active":"2024-03-23T03:51:22.273411+0000","last_became_peered":"2024-03-23T03:51:22.273411+0000","last_unstale":"2024-03-23T03:51:22.278612+0000","last_undegraded":"2024-03-23T03:51:22.278612+0000","last_fullsized":"2024-03-23T03:51:22.278612+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:58:33.997748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.284851+0000","last_change":"2024-03-23T03:51:22.278541+0000","last_active":"2024-03-23T03:51:22.284851+0000","last_peered":"2024-03-23T03:51:22.284851+0000","last_clean":"2024-03-23T03:51:22.284851+0000","last_became_active":"2024-03-23T03:51:22.278274+0000","last_became_peered":"2024-03-23T03:51:22.278274+0000","last_unstale":"2024-03-23T03:51:22.284851+0000","last_undegraded":"2024-03-23T03:51:22.284851+0000","last_fullsized":"2024-03-23T03:51:22.284851+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:10:28.823831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.282176+0000","last_change":"2024-03-23T03:51:22.277885+0000","last_active":"2024-03-23T03:51:22.282176+0000","last_peered":"2024-03-23T03:51:22.282176+0000","last_clean":"2024-03-23T03:51:22.282176+0000","last_became_active":"2024-03-23T03:51:22.277209+0000","last_became_peered":"2024-03-23T03:51:22.277209+0000","last_unstale":"2024-03-23T03:51:22.282176+0000","last_undegraded":"2024-03-23T03:51:22.282176+0000","last_fullsized":"2024-03-23T03:51:22.282176+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:07.649845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.265283+0000","last_change":"2024-03-23T03:51:22.261164+0000","last_active":"2024-03-23T03:51:22.265283+0000","last_peered":"2024-03-23T03:51:22.265283+0000","last_clean":"2024-03-23T03:51:22.265283+0000","last_became_active":"2024-03-23T03:51:22.260899+0000","last_became_peered":"2024-03-23T03:51:22.260899+0000","last_unstale":"2024-03-23T03:51:22.265283+0000","last_undegraded":"2024-03-23T03:51:22.265283+0000","last_fullsized":"2024-03-23T03:51:22.265283+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:35:05.132187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.272780+0000","last_change":"2024-03-23T03:51:22.266968+0000","last_active":"2024-03-23T03:51:22.272780+0000","last_peered":"2024-03-23T03:51:22.272780+0000","last_clean":"2024-03-23T03:51:22.272780+0000","last_became_active":"2024-03-23T03:51:22.266453+0000","last_became_peered":"2024-03-23T03:51:22.266453+0000","last_unstale":"2024-03-23T03:51:22.272780+0000","last_undegraded":"2024-03-23T03:51:22.272780+0000","last_fullsized":"2024-03-23T03:51:22.272780+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:30:04.249708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.281242+0000","last_change":"2024-03-23T03:51:22.271984+0000","last_active":"2024-03-23T03:51:22.281242+0000","last_peered":"2024-03-23T03:51:22.281242+0000","last_clean":"2024-03-23T03:51:22.281242+0000","last_became_active":"2024-03-23T03:51:22.271663+0000","last_became_peered":"2024-03-23T03:51:22.271663+0000","last_unstale":"2024-03-23T03:51:22.281242+0000","last_undegraded":"2024-03-23T03:51:22.281242+0000","last_fullsized":"2024-03-23T03:51:22.281242+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:17:09.963708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.281899+0000","last_change":"2024-03-23T03:51:22.277918+0000","last_active":"2024-03-23T03:51:22.281899+0000","last_peered":"2024-03-23T03:51:22.281899+0000","last_clean":"2024-03-23T03:51:22.281899+0000","last_became_active":"2024-03-23T03:51:22.277642+0000","last_became_peered":"2024-03-23T03:51:22.277642+0000","last_unstale":"2024-03-23T03:51:22.281899+0000","last_undegraded":"2024-03-23T03:51:22.281899+0000","last_fullsized":"2024-03-23T03:51:22.281899+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:04:21.319327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.286384+0000","last_change":"2024-03-23T03:51:22.281023+0000","last_active":"2024-03-23T03:51:22.286384+0000","last_peered":"2024-03-23T03:51:22.286384+0000","last_clean":"2024-03-23T03:51:22.286384+0000","last_became_active":"2024-03-23T03:51:22.280871+0000","last_became_peered":"2024-03-23T03:51:22.280871+0000","last_unstale":"2024-03-23T03:51:22.286384+0000","last_undegraded":"2024-03-23T03:51:22.286384+0000","last_fullsized":"2024-03-23T03:51:22.286384+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:20:55.608125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.288408+0000","last_change":"2024-03-23T03:51:22.285885+0000","last_active":"2024-03-23T03:51:22.288408+0000","last_peered":"2024-03-23T03:51:22.288408+0000","last_clean":"2024-03-23T03:51:22.288408+0000","last_became_active":"2024-03-23T03:51:22.280190+0000","last_became_peered":"2024-03-23T03:51:22.280190+0000","last_unstale":"2024-03-23T03:51:22.288408+0000","last_undegraded":"2024-03-23T03:51:22.288408+0000","last_fullsized":"2024-03-23T03:51:22.288408+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:03:27.282303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.279574+0000","last_change":"2024-03-23T03:51:22.276946+0000","last_active":"2024-03-23T03:51:22.279574+0000","last_peered":"2024-03-23T03:51:22.279574+0000","last_clean":"2024-03-23T03:51:22.279574+0000","last_became_active":"2024-03-23T03:51:22.276662+0000","last_became_peered":"2024-03-23T03:51:22.276662+0000","last_unstale":"2024-03-23T03:51:22.279574+0000","last_undegraded":"2024-03-23T03:51:22.279574+0000","last_fullsized":"2024-03-23T03:51:22.279574+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:48:26.593847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.282198+0000","last_change":"2024-03-23T03:51:22.279151+0000","last_active":"2024-03-23T03:51:22.282198+0000","last_peered":"2024-03-23T03:51:22.282198+0000","last_clean":"2024-03-23T03:51:22.282198+0000","last_became_active":"2024-03-23T03:51:22.276916+0000","last_became_peered":"2024-03-23T03:51:22.276916+0000","last_unstale":"2024-03-23T03:51:22.282198+0000","last_undegraded":"2024-03-23T03:51:22.282198+0000","last_fullsized":"2024-03-23T03:51:22.282198+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:13:39.429313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.276047+0000","last_change":"2024-03-23T03:51:22.273964+0000","last_active":"2024-03-23T03:51:22.276047+0000","last_peered":"2024-03-23T03:51:22.276047+0000","last_clean":"2024-03-23T03:51:22.276047+0000","last_became_active":"2024-03-23T03:51:22.273497+0000","last_became_peered":"2024-03-23T03:51:22.273497+0000","last_unstale":"2024-03-23T03:51:22.276047+0000","last_undegraded":"2024-03-23T03:51:22.276047+0000","last_fullsized":"2024-03-23T03:51:22.276047+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:20:51.607176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.267969+0000","last_change":"2024-03-23T03:51:22.262943+0000","last_active":"2024-03-23T03:51:22.267969+0000","last_peered":"2024-03-23T03:51:22.267969+0000","last_clean":"2024-03-23T03:51:22.267969+0000","last_became_active":"2024-03-23T03:51:22.262449+0000","last_became_peered":"2024-03-23T03:51:22.262449+0000","last_unstale":"2024-03-23T03:51:22.267969+0000","last_undegraded":"2024-03-23T03:51:22.267969+0000","last_fullsized":"2024-03-23T03:51:22.267969+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:27:04.683164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.272306+0000","last_change":"2024-03-23T03:51:22.270863+0000","last_active":"2024-03-23T03:51:22.272306+0000","last_peered":"2024-03-23T03:51:22.272306+0000","last_clean":"2024-03-23T03:51:22.272306+0000","last_became_active":"2024-03-23T03:51:22.270601+0000","last_became_peered":"2024-03-23T03:51:22.270601+0000","last_unstale":"2024-03-23T03:51:22.272306+0000","last_undegraded":"2024-03-23T03:51:22.272306+0000","last_fullsized":"2024-03-23T03:51:22.272306+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:46:32.018385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.267987+0000","last_change":"2024-03-23T03:51:22.262151+0000","last_active":"2024-03-23T03:51:22.267987+0000","last_peered":"2024-03-23T03:51:22.267987+0000","last_clean":"2024-03-23T03:51:22.267987+0000","last_became_active":"2024-03-23T03:51:22.261931+0000","last_became_peered":"2024-03-23T03:51:22.261931+0000","last_unstale":"2024-03-23T03:51:22.267987+0000","last_undegraded":"2024-03-23T03:51:22.267987+0000","last_fullsized":"2024-03-23T03:51:22.267987+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:47.178608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.288713+0000","last_change":"2024-03-23T03:51:22.286444+0000","last_active":"2024-03-23T03:51:22.288713+0000","last_peered":"2024-03-23T03:51:22.288713+0000","last_clean":"2024-03-23T03:51:22.288713+0000","last_became_active":"2024-03-23T03:51:22.285470+0000","last_became_peered":"2024-03-23T03:51:22.285470+0000","last_unstale":"2024-03-23T03:51:22.288713+0000","last_undegraded":"2024-03-23T03:51:22.288713+0000","last_fullsized":"2024-03-23T03:51:22.288713+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:58:40.479780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.284517+0000","last_change":"2024-03-23T03:51:22.280518+0000","last_active":"2024-03-23T03:51:22.284517+0000","last_peered":"2024-03-23T03:51:22.284517+0000","last_clean":"2024-03-23T03:51:22.284517+0000","last_became_active":"2024-03-23T03:51:22.280191+0000","last_became_peered":"2024-03-23T03:51:22.280191+0000","last_unstale":"2024-03-23T03:51:22.284517+0000","last_undegraded":"2024-03-23T03:51:22.284517+0000","last_fullsized":"2024-03-23T03:51:22.284517+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T03:58:05.924179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.283921+0000","last_change":"2024-03-23T03:51:22.272661+0000","last_active":"2024-03-23T03:51:22.283921+0000","last_peered":"2024-03-23T03:51:22.283921+0000","last_clean":"2024-03-23T03:51:22.283921+0000","last_became_active":"2024-03-23T03:51:22.272238+0000","last_became_peered":"2024-03-23T03:51:22.272238+0000","last_unstale":"2024-03-23T03:51:22.283921+0000","last_undegraded":"2024-03-23T03:51:22.283921+0000","last_fullsized":"2024-03-23T03:51:22.283921+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:20.563242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.278988+0000","last_change":"2024-03-23T03:51:22.274596+0000","last_active":"2024-03-23T03:51:22.278988+0000","last_peered":"2024-03-23T03:51:22.278988+0000","last_clean":"2024-03-23T03:51:22.278988+0000","last_became_active":"2024-03-23T03:51:22.274326+0000","last_became_peered":"2024-03-23T03:51:22.274326+0000","last_unstale":"2024-03-23T03:51:22.278988+0000","last_undegraded":"2024-03-23T03:51:22.278988+0000","last_fullsized":"2024-03-23T03:51:22.278988+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:11:23.842491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.275911+0000","last_change":"2024-03-23T03:51:22.273810+0000","last_active":"2024-03-23T03:51:22.275911+0000","last_peered":"2024-03-23T03:51:22.275911+0000","last_clean":"2024-03-23T03:51:22.275911+0000","last_became_active":"2024-03-23T03:51:22.273091+0000","last_became_peered":"2024-03-23T03:51:22.273091+0000","last_unstale":"2024-03-23T03:51:22.275911+0000","last_undegraded":"2024-03-23T03:51:22.275911+0000","last_fullsized":"2024-03-23T03:51:22.275911+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:46.517427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.287367+0000","last_change":"2024-03-23T03:51:22.283425+0000","last_active":"2024-03-23T03:51:22.287367+0000","last_peered":"2024-03-23T03:51:22.287367+0000","last_clean":"2024-03-23T03:51:22.287367+0000","last_became_active":"2024-03-23T03:51:22.283140+0000","last_became_peered":"2024-03-23T03:51:22.283140+0000","last_unstale":"2024-03-23T03:51:22.287367+0000","last_undegraded":"2024-03-23T03:51:22.287367+0000","last_fullsized":"2024-03-23T03:51:22.287367+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:36:38.929407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.274309+0000","last_change":"2024-03-23T03:51:22.269314+0000","last_active":"2024-03-23T03:51:22.274309+0000","last_peered":"2024-03-23T03:51:22.274309+0000","last_clean":"2024-03-23T03:51:22.274309+0000","last_became_active":"2024-03-23T03:51:22.268912+0000","last_became_peered":"2024-03-23T03:51:22.268912+0000","last_unstale":"2024-03-23T03:51:22.274309+0000","last_undegraded":"2024-03-23T03:51:22.274309+0000","last_fullsized":"2024-03-23T03:51:22.274309+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:25:25.635704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.286480+0000","last_change":"2024-03-23T03:51:22.282218+0000","last_active":"2024-03-23T03:51:22.286480+0000","last_peered":"2024-03-23T03:51:22.286480+0000","last_clean":"2024-03-23T03:51:22.286480+0000","last_became_active":"2024-03-23T03:51:22.281934+0000","last_became_peered":"2024-03-23T03:51:22.281934+0000","last_unstale":"2024-03-23T03:51:22.286480+0000","last_undegraded":"2024-03-23T03:51:22.286480+0000","last_fullsized":"2024-03-23T03:51:22.286480+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:19:58.455715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.279440+0000","last_change":"2024-03-23T03:51:22.276935+0000","last_active":"2024-03-23T03:51:22.279440+0000","last_peered":"2024-03-23T03:51:22.279440+0000","last_clean":"2024-03-23T03:51:22.279440+0000","last_became_active":"2024-03-23T03:51:22.276687+0000","last_became_peered":"2024-03-23T03:51:22.276687+0000","last_unstale":"2024-03-23T03:51:22.279440+0000","last_undegraded":"2024-03-23T03:51:22.279440+0000","last_fullsized":"2024-03-23T03:51:22.279440+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:57:56.460821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"104'4","reported_seq":15,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.274281+0000","last_change":"2024-03-23T03:51:21.789123+0000","last_active":"2024-03-23T03:51:22.274281+0000","last_peered":"2024-03-23T03:51:22.274281+0000","last_clean":"2024-03-23T03:51:22.274281+0000","last_became_active":"2024-03-23T03:51:21.788839+0000","last_became_peered":"2024-03-23T03:51:21.788839+0000","last_unstale":"2024-03-23T03:51:22.274281+0000","last_undegraded":"2024-03-23T03:51:22.274281+0000","last_fullsized":"2024-03-23T03:51:22.274281+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:54:15.206699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.261655+0000","last_change":"2024-03-23T03:51:22.258376+0000","last_active":"2024-03-23T03:51:22.261655+0000","last_peered":"2024-03-23T03:51:22.261655+0000","last_clean":"2024-03-23T03:51:22.261655+0000","last_became_active":"2024-03-23T03:51:22.257856+0000","last_became_peered":"2024-03-23T03:51:22.257856+0000","last_unstale":"2024-03-23T03:51:22.261655+0000","last_undegraded":"2024-03-23T03:51:22.261655+0000","last_fullsized":"2024-03-23T03:51:22.261655+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:39:04.438866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"104'1","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.276956+0000","last_change":"2024-03-23T03:51:21.785555+0000","last_active":"2024-03-23T03:51:22.276956+0000","last_peered":"2024-03-23T03:51:22.276956+0000","last_clean":"2024-03-23T03:51:22.276956+0000","last_became_active":"2024-03-23T03:51:21.785232+0000","last_became_peered":"2024-03-23T03:51:21.785232+0000","last_unstale":"2024-03-23T03:51:22.276956+0000","last_undegraded":"2024-03-23T03:51:22.276956+0000","last_fullsized":"2024-03-23T03:51:22.276956+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:27:33.193844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.282442+0000","last_change":"2024-03-23T03:51:22.278002+0000","last_active":"2024-03-23T03:51:22.282442+0000","last_peered":"2024-03-23T03:51:22.282442+0000","last_clean":"2024-03-23T03:51:22.282442+0000","last_became_active":"2024-03-23T03:51:22.277586+0000","last_became_peered":"2024-03-23T03:51:22.277586+0000","last_unstale":"2024-03-23T03:51:22.282442+0000","last_undegraded":"2024-03-23T03:51:22.282442+0000","last_fullsized":"2024-03-23T03:51:22.282442+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:27:17.201821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.285705+0000","last_change":"2024-03-23T03:51:21.783666+0000","last_active":"2024-03-23T03:51:22.285705+0000","last_peered":"2024-03-23T03:51:22.285705+0000","last_clean":"2024-03-23T03:51:22.285705+0000","last_became_active":"2024-03-23T03:51:21.783377+0000","last_became_peered":"2024-03-23T03:51:21.783377+0000","last_unstale":"2024-03-23T03:51:22.285705+0000","last_undegraded":"2024-03-23T03:51:22.285705+0000","last_fullsized":"2024-03-23T03:51:22.285705+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:42:08.794820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.287243+0000","last_change":"2024-03-23T03:51:22.284472+0000","last_active":"2024-03-23T03:51:22.287243+0000","last_peered":"2024-03-23T03:51:22.287243+0000","last_clean":"2024-03-23T03:51:22.287243+0000","last_became_active":"2024-03-23T03:51:22.284222+0000","last_became_peered":"2024-03-23T03:51:22.284222+0000","last_unstale":"2024-03-23T03:51:22.287243+0000","last_undegraded":"2024-03-23T03:51:22.287243+0000","last_fullsized":"2024-03-23T03:51:22.287243+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:17:09.963708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.253628+0000","last_change":"2024-03-23T03:51:21.792017+0000","last_active":"2024-03-23T03:51:22.253628+0000","last_peered":"2024-03-23T03:51:22.253628+0000","last_clean":"2024-03-23T03:51:22.253628+0000","last_became_active":"2024-03-23T03:51:21.791763+0000","last_became_peered":"2024-03-23T03:51:21.791763+0000","last_unstale":"2024-03-23T03:51:22.253628+0000","last_undegraded":"2024-03-23T03:51:22.253628+0000","last_fullsized":"2024-03-23T03:51:22.253628+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:05:09.454442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.260943+0000","last_change":"2024-03-23T03:51:22.255429+0000","last_active":"2024-03-23T03:51:22.260943+0000","last_peered":"2024-03-23T03:51:22.260943+0000","last_clean":"2024-03-23T03:51:22.260943+0000","last_became_active":"2024-03-23T03:51:22.253694+0000","last_became_peered":"2024-03-23T03:51:22.253694+0000","last_unstale":"2024-03-23T03:51:22.260943+0000","last_undegraded":"2024-03-23T03:51:22.260943+0000","last_fullsized":"2024-03-23T03:51:22.260943+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:38:27.787294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"104'1","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.279394+0000","last_change":"2024-03-23T03:51:21.792106+0000","last_active":"2024-03-23T03:51:22.279394+0000","last_peered":"2024-03-23T03:51:22.279394+0000","last_clean":"2024-03-23T03:51:22.279394+0000","last_became_active":"2024-03-23T03:51:21.791295+0000","last_became_peered":"2024-03-23T03:51:21.791295+0000","last_unstale":"2024-03-23T03:51:22.279394+0000","last_undegraded":"2024-03-23T03:51:22.279394+0000","last_fullsized":"2024-03-23T03:51:22.279394+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T13:00:35.520733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.279590+0000","last_change":"2024-03-23T03:51:22.277153+0000","last_active":"2024-03-23T03:51:22.279590+0000","last_peered":"2024-03-23T03:51:22.279590+0000","last_clean":"2024-03-23T03:51:22.279590+0000","last_became_active":"2024-03-23T03:51:22.276655+0000","last_became_peered":"2024-03-23T03:51:22.276655+0000","last_unstale":"2024-03-23T03:51:22.279590+0000","last_undegraded":"2024-03-23T03:51:22.279590+0000","last_fullsized":"2024-03-23T03:51:22.279590+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:37:27.534087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"104'1","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.284168+0000","last_change":"2024-03-23T03:51:21.780198+0000","last_active":"2024-03-23T03:51:22.284168+0000","last_peered":"2024-03-23T03:51:22.284168+0000","last_clean":"2024-03-23T03:51:22.284168+0000","last_became_active":"2024-03-23T03:51:21.779256+0000","last_became_peered":"2024-03-23T03:51:21.779256+0000","last_unstale":"2024-03-23T03:51:22.284168+0000","last_undegraded":"2024-03-23T03:51:22.284168+0000","last_fullsized":"2024-03-23T03:51:22.284168+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:00:33.834214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.276291+0000","last_change":"2024-03-23T03:51:22.273524+0000","last_active":"2024-03-23T03:51:22.276291+0000","last_peered":"2024-03-23T03:51:22.276291+0000","last_clean":"2024-03-23T03:51:22.276291+0000","last_became_active":"2024-03-23T03:51:22.270458+0000","last_became_peered":"2024-03-23T03:51:22.270458+0000","last_unstale":"2024-03-23T03:51:22.276291+0000","last_undegraded":"2024-03-23T03:51:22.276291+0000","last_fullsized":"2024-03-23T03:51:22.276291+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:50:44.800098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.269859+0000","last_change":"2024-03-23T03:51:21.783223+0000","last_active":"2024-03-23T03:51:22.269859+0000","last_peered":"2024-03-23T03:51:22.269859+0000","last_clean":"2024-03-23T03:51:22.269859+0000","last_became_active":"2024-03-23T03:51:21.782799+0000","last_became_peered":"2024-03-23T03:51:21.782799+0000","last_unstale":"2024-03-23T03:51:22.269859+0000","last_undegraded":"2024-03-23T03:51:22.269859+0000","last_fullsized":"2024-03-23T03:51:22.269859+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:56:44.890019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.272586+0000","last_change":"2024-03-23T03:51:22.263756+0000","last_active":"2024-03-23T03:51:22.272586+0000","last_peered":"2024-03-23T03:51:22.272586+0000","last_clean":"2024-03-23T03:51:22.272586+0000","last_became_active":"2024-03-23T03:51:22.263510+0000","last_became_peered":"2024-03-23T03:51:22.263510+0000","last_unstale":"2024-03-23T03:51:22.272586+0000","last_undegraded":"2024-03-23T03:51:22.272586+0000","last_fullsized":"2024-03-23T03:51:22.272586+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:31:57.940022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.246134+0000","last_change":"2024-03-23T03:51:21.776001+0000","last_active":"2024-03-23T03:51:22.246134+0000","last_peered":"2024-03-23T03:51:22.246134+0000","last_clean":"2024-03-23T03:51:22.246134+0000","last_became_active":"2024-03-23T03:51:21.775726+0000","last_became_peered":"2024-03-23T03:51:21.775726+0000","last_unstale":"2024-03-23T03:51:22.246134+0000","last_undegraded":"2024-03-23T03:51:22.246134+0000","last_fullsized":"2024-03-23T03:51:22.246134+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:07:52.649065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.279542+0000","last_change":"2024-03-23T03:51:22.277230+0000","last_active":"2024-03-23T03:51:22.279542+0000","last_peered":"2024-03-23T03:51:22.279542+0000","last_clean":"2024-03-23T03:51:22.279542+0000","last_became_active":"2024-03-23T03:51:22.276977+0000","last_became_peered":"2024-03-23T03:51:22.276977+0000","last_unstale":"2024-03-23T03:51:22.279542+0000","last_undegraded":"2024-03-23T03:51:22.279542+0000","last_fullsized":"2024-03-23T03:51:22.279542+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:55:30.397347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.273372+0000","last_change":"2024-03-23T03:51:21.786526+0000","last_active":"2024-03-23T03:51:22.273372+0000","last_peered":"2024-03-23T03:51:22.273372+0000","last_clean":"2024-03-23T03:51:22.273372+0000","last_became_active":"2024-03-23T03:51:21.783040+0000","last_became_peered":"2024-03-23T03:51:21.783040+0000","last_unstale":"2024-03-23T03:51:22.273372+0000","last_undegraded":"2024-03-23T03:51:22.273372+0000","last_fullsized":"2024-03-23T03:51:22.273372+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:53.003075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.264940+0000","last_change":"2024-03-23T03:51:22.258821+0000","last_active":"2024-03-23T03:51:22.264940+0000","last_peered":"2024-03-23T03:51:22.264940+0000","last_clean":"2024-03-23T03:51:22.264940+0000","last_became_active":"2024-03-23T03:51:22.258581+0000","last_became_peered":"2024-03-23T03:51:22.258581+0000","last_unstale":"2024-03-23T03:51:22.264940+0000","last_undegraded":"2024-03-23T03:51:22.264940+0000","last_fullsized":"2024-03-23T03:51:22.264940+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:42:06.329388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.259334+0000","last_change":"2024-03-23T03:51:21.787895+0000","last_active":"2024-03-23T03:51:22.259334+0000","last_peered":"2024-03-23T03:51:22.259334+0000","last_clean":"2024-03-23T03:51:22.259334+0000","last_became_active":"2024-03-23T03:51:21.787627+0000","last_became_peered":"2024-03-23T03:51:21.787627+0000","last_unstale":"2024-03-23T03:51:22.259334+0000","last_undegraded":"2024-03-23T03:51:22.259334+0000","last_fullsized":"2024-03-23T03:51:22.259334+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:31:02.266262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.269389+0000","last_change":"2024-03-23T03:51:22.266962+0000","last_active":"2024-03-23T03:51:22.269389+0000","last_peered":"2024-03-23T03:51:22.269389+0000","last_clean":"2024-03-23T03:51:22.269389+0000","last_became_active":"2024-03-23T03:51:22.266486+0000","last_became_peered":"2024-03-23T03:51:22.266486+0000","last_unstale":"2024-03-23T03:51:22.269389+0000","last_undegraded":"2024-03-23T03:51:22.269389+0000","last_fullsized":"2024-03-23T03:51:22.269389+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:34:16.439779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"104'3","reported_seq":14,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.275377+0000","last_change":"2024-03-23T03:51:21.786687+0000","last_active":"2024-03-23T03:51:22.275377+0000","last_peered":"2024-03-23T03:51:22.275377+0000","last_clean":"2024-03-23T03:51:22.275377+0000","last_became_active":"2024-03-23T03:51:21.786374+0000","last_became_peered":"2024-03-23T03:51:21.786374+0000","last_unstale":"2024-03-23T03:51:22.275377+0000","last_undegraded":"2024-03-23T03:51:22.275377+0000","last_fullsized":"2024-03-23T03:51:22.275377+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:09:59.406767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.265284+0000","last_change":"2024-03-23T03:51:22.259503+0000","last_active":"2024-03-23T03:51:22.265284+0000","last_peered":"2024-03-23T03:51:22.265284+0000","last_clean":"2024-03-23T03:51:22.265284+0000","last_became_active":"2024-03-23T03:51:22.256943+0000","last_became_peered":"2024-03-23T03:51:22.256943+0000","last_unstale":"2024-03-23T03:51:22.265284+0000","last_undegraded":"2024-03-23T03:51:22.265284+0000","last_fullsized":"2024-03-23T03:51:22.265284+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:19:44.510925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"104'6","reported_seq":17,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.289168+0000","last_change":"2024-03-23T03:51:21.789690+0000","last_active":"2024-03-23T03:51:22.289168+0000","last_peered":"2024-03-23T03:51:22.289168+0000","last_clean":"2024-03-23T03:51:22.289168+0000","last_became_active":"2024-03-23T03:51:21.789409+0000","last_became_peered":"2024-03-23T03:51:21.789409+0000","last_unstale":"2024-03-23T03:51:22.289168+0000","last_undegraded":"2024-03-23T03:51:22.289168+0000","last_fullsized":"2024-03-23T03:51:22.289168+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:03:22.934141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.283171+0000","last_change":"2024-03-23T03:51:22.271629+0000","last_active":"2024-03-23T03:51:22.283171+0000","last_peered":"2024-03-23T03:51:22.283171+0000","last_clean":"2024-03-23T03:51:22.283171+0000","last_became_active":"2024-03-23T03:51:22.271350+0000","last_became_peered":"2024-03-23T03:51:22.271350+0000","last_unstale":"2024-03-23T03:51:22.283171+0000","last_undegraded":"2024-03-23T03:51:22.283171+0000","last_fullsized":"2024-03-23T03:51:22.283171+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:04:22.944014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.256406+0000","last_change":"2024-03-23T03:51:21.778247+0000","last_active":"2024-03-23T03:51:22.256406+0000","last_peered":"2024-03-23T03:51:22.256406+0000","last_clean":"2024-03-23T03:51:22.256406+0000","last_became_active":"2024-03-23T03:51:21.777903+0000","last_became_peered":"2024-03-23T03:51:21.777903+0000","last_unstale":"2024-03-23T03:51:22.256406+0000","last_undegraded":"2024-03-23T03:51:22.256406+0000","last_fullsized":"2024-03-23T03:51:22.256406+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:30:03.239118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.274568+0000","last_change":"2024-03-23T03:51:22.265915+0000","last_active":"2024-03-23T03:51:22.274568+0000","last_peered":"2024-03-23T03:51:22.274568+0000","last_clean":"2024-03-23T03:51:22.274568+0000","last_became_active":"2024-03-23T03:51:22.265640+0000","last_became_peered":"2024-03-23T03:51:22.265640+0000","last_unstale":"2024-03-23T03:51:22.274568+0000","last_undegraded":"2024-03-23T03:51:22.274568+0000","last_fullsized":"2024-03-23T03:51:22.274568+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T06:06:14.520077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.246653+0000","last_change":"2024-03-23T03:51:21.780251+0000","last_active":"2024-03-23T03:51:22.246653+0000","last_peered":"2024-03-23T03:51:22.246653+0000","last_clean":"2024-03-23T03:51:22.246653+0000","last_became_active":"2024-03-23T03:51:21.779927+0000","last_became_peered":"2024-03-23T03:51:21.779927+0000","last_unstale":"2024-03-23T03:51:22.246653+0000","last_undegraded":"2024-03-23T03:51:22.246653+0000","last_fullsized":"2024-03-23T03:51:22.246653+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:28:51.270476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.288871+0000","last_change":"2024-03-23T03:51:22.285115+0000","last_active":"2024-03-23T03:51:22.288871+0000","last_peered":"2024-03-23T03:51:22.288871+0000","last_clean":"2024-03-23T03:51:22.288871+0000","last_became_active":"2024-03-23T03:51:22.284771+0000","last_became_peered":"2024-03-23T03:51:22.284771+0000","last_unstale":"2024-03-23T03:51:22.288871+0000","last_undegraded":"2024-03-23T03:51:22.288871+0000","last_fullsized":"2024-03-23T03:51:22.288871+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:53.513182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.249815+0000","last_change":"2024-03-23T03:51:21.790264+0000","last_active":"2024-03-23T03:51:22.249815+0000","last_peered":"2024-03-23T03:51:22.249815+0000","last_clean":"2024-03-23T03:51:22.249815+0000","last_became_active":"2024-03-23T03:51:21.789990+0000","last_became_peered":"2024-03-23T03:51:21.789990+0000","last_unstale":"2024-03-23T03:51:22.249815+0000","last_undegraded":"2024-03-23T03:51:22.249815+0000","last_fullsized":"2024-03-23T03:51:22.249815+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:17:42.968056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.286366+0000","last_change":"2024-03-23T03:51:22.280933+0000","last_active":"2024-03-23T03:51:22.286366+0000","last_peered":"2024-03-23T03:51:22.286366+0000","last_clean":"2024-03-23T03:51:22.286366+0000","last_became_active":"2024-03-23T03:51:22.280679+0000","last_became_peered":"2024-03-23T03:51:22.280679+0000","last_unstale":"2024-03-23T03:51:22.286366+0000","last_undegraded":"2024-03-23T03:51:22.286366+0000","last_fullsized":"2024-03-23T03:51:22.286366+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:52:35.913771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"104'8","reported_seq":19,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.287974+0000","last_change":"2024-03-23T03:51:21.783083+0000","last_active":"2024-03-23T03:51:22.287974+0000","last_peered":"2024-03-23T03:51:22.287974+0000","last_clean":"2024-03-23T03:51:22.287974+0000","last_became_active":"2024-03-23T03:51:21.782771+0000","last_became_peered":"2024-03-23T03:51:21.782771+0000","last_unstale":"2024-03-23T03:51:22.287974+0000","last_undegraded":"2024-03-23T03:51:22.287974+0000","last_fullsized":"2024-03-23T03:51:22.287974+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:11:22.831901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.274824+0000","last_change":"2024-03-23T03:51:22.263700+0000","last_active":"2024-03-23T03:51:22.274824+0000","last_peered":"2024-03-23T03:51:22.274824+0000","last_clean":"2024-03-23T03:51:22.274824+0000","last_became_active":"2024-03-23T03:51:22.263452+0000","last_became_peered":"2024-03-23T03:51:22.263452+0000","last_unstale":"2024-03-23T03:51:22.274824+0000","last_undegraded":"2024-03-23T03:51:22.274824+0000","last_fullsized":"2024-03-23T03:51:22.274824+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:35:48.145231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"104'1","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.286170+0000","last_change":"2024-03-23T03:51:21.793220+0000","last_active":"2024-03-23T03:51:22.286170+0000","last_peered":"2024-03-23T03:51:22.286170+0000","last_clean":"2024-03-23T03:51:22.286170+0000","last_became_active":"2024-03-23T03:51:21.792937+0000","last_became_peered":"2024-03-23T03:51:21.792937+0000","last_unstale":"2024-03-23T03:51:22.286170+0000","last_undegraded":"2024-03-23T03:51:22.286170+0000","last_fullsized":"2024-03-23T03:51:22.286170+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:52.502591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.276080+0000","last_change":"2024-03-23T03:51:22.271578+0000","last_active":"2024-03-23T03:51:22.276080+0000","last_peered":"2024-03-23T03:51:22.276080+0000","last_clean":"2024-03-23T03:51:22.276080+0000","last_became_active":"2024-03-23T03:51:22.271281+0000","last_became_peered":"2024-03-23T03:51:22.271281+0000","last_unstale":"2024-03-23T03:51:22.276080+0000","last_undegraded":"2024-03-23T03:51:22.276080+0000","last_fullsized":"2024-03-23T03:51:22.276080+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:31:10.713669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"104'1","reported_seq":12,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.277479+0000","last_change":"2024-03-23T03:51:21.792048+0000","last_active":"2024-03-23T03:51:22.277479+0000","last_peered":"2024-03-23T03:51:22.277479+0000","last_clean":"2024-03-23T03:51:22.277479+0000","last_became_active":"2024-03-23T03:51:21.791713+0000","last_became_peered":"2024-03-23T03:51:21.791713+0000","last_unstale":"2024-03-23T03:51:22.277479+0000","last_undegraded":"2024-03-23T03:51:22.277479+0000","last_fullsized":"2024-03-23T03:51:22.277479+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:59:01.379564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.278624+0000","last_change":"2024-03-23T03:51:22.269905+0000","last_active":"2024-03-23T03:51:22.278624+0000","last_peered":"2024-03-23T03:51:22.278624+0000","last_clean":"2024-03-23T03:51:22.278624+0000","last_became_active":"2024-03-23T03:51:22.269683+0000","last_became_peered":"2024-03-23T03:51:22.269683+0000","last_unstale":"2024-03-23T03:51:22.278624+0000","last_undegraded":"2024-03-23T03:51:22.278624+0000","last_fullsized":"2024-03-23T03:51:22.278624+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:59:59.276905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.249880+0000","last_change":"2024-03-23T03:51:21.789004+0000","last_active":"2024-03-23T03:51:22.249880+0000","last_peered":"2024-03-23T03:51:22.249880+0000","last_clean":"2024-03-23T03:51:22.249880+0000","last_became_active":"2024-03-23T03:51:21.788647+0000","last_became_peered":"2024-03-23T03:51:21.788647+0000","last_unstale":"2024-03-23T03:51:22.249880+0000","last_undegraded":"2024-03-23T03:51:22.249880+0000","last_fullsized":"2024-03-23T03:51:22.249880+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:59:01.379564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.267588+0000","last_change":"2024-03-23T03:51:22.263556+0000","last_active":"2024-03-23T03:51:22.267588+0000","last_peered":"2024-03-23T03:51:22.267588+0000","last_clean":"2024-03-23T03:51:22.267588+0000","last_became_active":"2024-03-23T03:51:22.263345+0000","last_became_peered":"2024-03-23T03:51:22.263345+0000","last_unstale":"2024-03-23T03:51:22.267588+0000","last_undegraded":"2024-03-23T03:51:22.267588+0000","last_fullsized":"2024-03-23T03:51:22.267588+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:15:49.588458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.263002+0000","last_change":"2024-03-23T03:51:21.782473+0000","last_active":"2024-03-23T03:51:22.263002+0000","last_peered":"2024-03-23T03:51:22.263002+0000","last_clean":"2024-03-23T03:51:22.263002+0000","last_became_active":"2024-03-23T03:51:21.781740+0000","last_became_peered":"2024-03-23T03:51:21.781740+0000","last_unstale":"2024-03-23T03:51:22.263002+0000","last_undegraded":"2024-03-23T03:51:22.263002+0000","last_fullsized":"2024-03-23T03:51:22.263002+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:30:22.629109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.270955+0000","last_change":"2024-03-23T03:51:22.269215+0000","last_active":"2024-03-23T03:51:22.270955+0000","last_peered":"2024-03-23T03:51:22.270955+0000","last_clean":"2024-03-23T03:51:22.270955+0000","last_became_active":"2024-03-23T03:51:22.268957+0000","last_became_peered":"2024-03-23T03:51:22.268957+0000","last_unstale":"2024-03-23T03:51:22.270955+0000","last_undegraded":"2024-03-23T03:51:22.270955+0000","last_fullsized":"2024-03-23T03:51:22.270955+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:18:12.413501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.250313+0000","last_change":"2024-03-23T03:51:21.782262+0000","last_active":"2024-03-23T03:51:22.250313+0000","last_peered":"2024-03-23T03:51:22.250313+0000","last_clean":"2024-03-23T03:51:22.250313+0000","last_became_active":"2024-03-23T03:51:21.781940+0000","last_became_peered":"2024-03-23T03:51:21.781940+0000","last_unstale":"2024-03-23T03:51:22.250313+0000","last_undegraded":"2024-03-23T03:51:22.250313+0000","last_fullsized":"2024-03-23T03:51:22.250313+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:50:49.654183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.275604+0000","last_change":"2024-03-23T03:51:22.273117+0000","last_active":"2024-03-23T03:51:22.275604+0000","last_peered":"2024-03-23T03:51:22.275604+0000","last_clean":"2024-03-23T03:51:22.275604+0000","last_became_active":"2024-03-23T03:51:22.272861+0000","last_became_peered":"2024-03-23T03:51:22.272861+0000","last_unstale":"2024-03-23T03:51:22.275604+0000","last_undegraded":"2024-03-23T03:51:22.275604+0000","last_fullsized":"2024-03-23T03:51:22.275604+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T09:06:50.564546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.263230+0000","last_change":"2024-03-23T03:51:21.792924+0000","last_active":"2024-03-23T03:51:22.263230+0000","last_peered":"2024-03-23T03:51:22.263230+0000","last_clean":"2024-03-23T03:51:22.263230+0000","last_became_active":"2024-03-23T03:51:21.790283+0000","last_became_peered":"2024-03-23T03:51:21.790283+0000","last_unstale":"2024-03-23T03:51:22.263230+0000","last_undegraded":"2024-03-23T03:51:22.263230+0000","last_fullsized":"2024-03-23T03:51:22.263230+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:19:57.445124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.269326+0000","last_change":"2024-03-23T03:51:22.265543+0000","last_active":"2024-03-23T03:51:22.269326+0000","last_peered":"2024-03-23T03:51:22.269326+0000","last_clean":"2024-03-23T03:51:22.269326+0000","last_became_active":"2024-03-23T03:51:22.265310+0000","last_became_peered":"2024-03-23T03:51:22.265310+0000","last_unstale":"2024-03-23T03:51:22.269326+0000","last_undegraded":"2024-03-23T03:51:22.269326+0000","last_fullsized":"2024-03-23T03:51:22.269326+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:02:35.027397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.249976+0000","last_change":"2024-03-23T03:51:21.784479+0000","last_active":"2024-03-23T03:51:22.249976+0000","last_peered":"2024-03-23T03:51:22.249976+0000","last_clean":"2024-03-23T03:51:22.249976+0000","last_became_active":"2024-03-23T03:51:21.784166+0000","last_became_peered":"2024-03-23T03:51:21.784166+0000","last_unstale":"2024-03-23T03:51:22.249976+0000","last_undegraded":"2024-03-23T03:51:22.249976+0000","last_fullsized":"2024-03-23T03:51:22.249976+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:35:04.474544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.267498+0000","last_change":"2024-03-23T03:51:22.263572+0000","last_active":"2024-03-23T03:51:22.267498+0000","last_peered":"2024-03-23T03:51:22.267498+0000","last_clean":"2024-03-23T03:51:22.267498+0000","last_became_active":"2024-03-23T03:51:22.263304+0000","last_became_peered":"2024-03-23T03:51:22.263304+0000","last_unstale":"2024-03-23T03:51:22.267498+0000","last_undegraded":"2024-03-23T03:51:22.267498+0000","last_fullsized":"2024-03-23T03:51:22.267498+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:08:03.409470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.256089+0000","last_change":"2024-03-23T03:51:21.776941+0000","last_active":"2024-03-23T03:51:22.256089+0000","last_peered":"2024-03-23T03:51:22.256089+0000","last_clean":"2024-03-23T03:51:22.256089+0000","last_became_active":"2024-03-23T03:51:21.776643+0000","last_became_peered":"2024-03-23T03:51:21.776643+0000","last_unstale":"2024-03-23T03:51:22.256089+0000","last_undegraded":"2024-03-23T03:51:22.256089+0000","last_fullsized":"2024-03-23T03:51:22.256089+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:17:08.953118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.282153+0000","last_change":"2024-03-23T03:51:22.278810+0000","last_active":"2024-03-23T03:51:22.282153+0000","last_peered":"2024-03-23T03:51:22.282153+0000","last_clean":"2024-03-23T03:51:22.282153+0000","last_became_active":"2024-03-23T03:51:22.278506+0000","last_became_peered":"2024-03-23T03:51:22.278506+0000","last_unstale":"2024-03-23T03:51:22.282153+0000","last_undegraded":"2024-03-23T03:51:22.282153+0000","last_fullsized":"2024-03-23T03:51:22.282153+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:46:03.713161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"105'1","reported_seq":12,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-23T03:51:23.268762+0000","last_change":"2024-03-23T03:51:21.785341+0000","last_active":"2024-03-23T03:51:23.268762+0000","last_peered":"2024-03-23T03:51:23.268762+0000","last_clean":"2024-03-23T03:51:23.268762+0000","last_became_active":"2024-03-23T03:51:21.784654+0000","last_became_peered":"2024-03-23T03:51:21.784654+0000","last_unstale":"2024-03-23T03:51:23.268762+0000","last_undegraded":"2024-03-23T03:51:23.268762+0000","last_fullsized":"2024-03-23T03:51:23.268762+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:59:36.375421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.272683+0000","last_change":"2024-03-23T03:51:22.265659+0000","last_active":"2024-03-23T03:51:22.272683+0000","last_peered":"2024-03-23T03:51:22.272683+0000","last_clean":"2024-03-23T03:51:22.272683+0000","last_became_active":"2024-03-23T03:51:22.265380+0000","last_became_peered":"2024-03-23T03:51:22.265380+0000","last_unstale":"2024-03-23T03:51:22.272683+0000","last_undegraded":"2024-03-23T03:51:22.272683+0000","last_fullsized":"2024-03-23T03:51:22.272683+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:27:15.422967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.257431+0000","last_change":"2024-03-23T03:51:21.788631+0000","last_active":"2024-03-23T03:51:22.257431+0000","last_peered":"2024-03-23T03:51:22.257431+0000","last_clean":"2024-03-23T03:51:22.257431+0000","last_became_active":"2024-03-23T03:51:21.788145+0000","last_became_peered":"2024-03-23T03:51:21.788145+0000","last_unstale":"2024-03-23T03:51:22.257431+0000","last_undegraded":"2024-03-23T03:51:22.257431+0000","last_fullsized":"2024-03-23T03:51:22.257431+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:00:45.506837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.265867+0000","last_change":"2024-03-23T03:51:22.259619+0000","last_active":"2024-03-23T03:51:22.265867+0000","last_peered":"2024-03-23T03:51:22.265867+0000","last_clean":"2024-03-23T03:51:22.265867+0000","last_became_active":"2024-03-23T03:51:22.259270+0000","last_became_peered":"2024-03-23T03:51:22.259270+0000","last_unstale":"2024-03-23T03:51:22.265867+0000","last_undegraded":"2024-03-23T03:51:22.265867+0000","last_fullsized":"2024-03-23T03:51:22.265867+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:30:04.249708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.241639+0000","last_change":"2024-03-23T03:51:21.788546+0000","last_active":"2024-03-23T03:51:22.241639+0000","last_peered":"2024-03-23T03:51:22.241639+0000","last_clean":"2024-03-23T03:51:22.241639+0000","last_became_active":"2024-03-23T03:51:21.788282+0000","last_became_peered":"2024-03-23T03:51:21.788282+0000","last_unstale":"2024-03-23T03:51:22.241639+0000","last_undegraded":"2024-03-23T03:51:22.241639+0000","last_fullsized":"2024-03-23T03:51:22.241639+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:04:20.308736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.261946+0000","last_change":"2024-03-23T03:51:22.258385+0000","last_active":"2024-03-23T03:51:22.261946+0000","last_peered":"2024-03-23T03:51:22.261946+0000","last_clean":"2024-03-23T03:51:22.261946+0000","last_became_active":"2024-03-23T03:51:22.258120+0000","last_became_peered":"2024-03-23T03:51:22.258120+0000","last_unstale":"2024-03-23T03:51:22.261946+0000","last_undegraded":"2024-03-23T03:51:22.261946+0000","last_fullsized":"2024-03-23T03:51:22.261946+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:11:05.542126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.265163+0000","last_change":"2024-03-23T03:51:21.793737+0000","last_active":"2024-03-23T03:51:22.265163+0000","last_peered":"2024-03-23T03:51:22.265163+0000","last_clean":"2024-03-23T03:51:22.265163+0000","last_became_active":"2024-03-23T03:51:21.791575+0000","last_became_peered":"2024-03-23T03:51:21.791575+0000","last_unstale":"2024-03-23T03:51:22.265163+0000","last_undegraded":"2024-03-23T03:51:22.265163+0000","last_fullsized":"2024-03-23T03:51:22.265163+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:54:20.064900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.269225+0000","last_change":"2024-03-23T03:51:22.266599+0000","last_active":"2024-03-23T03:51:22.269225+0000","last_peered":"2024-03-23T03:51:22.269225+0000","last_clean":"2024-03-23T03:51:22.269225+0000","last_became_active":"2024-03-23T03:51:22.266281+0000","last_became_peered":"2024-03-23T03:51:22.266281+0000","last_unstale":"2024-03-23T03:51:22.269225+0000","last_undegraded":"2024-03-23T03:51:22.269225+0000","last_fullsized":"2024-03-23T03:51:22.269225+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:45:36.655749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.250446+0000","last_change":"2024-03-23T03:51:21.787337+0000","last_active":"2024-03-23T03:51:22.250446+0000","last_peered":"2024-03-23T03:51:22.250446+0000","last_clean":"2024-03-23T03:51:22.250446+0000","last_became_active":"2024-03-23T03:51:21.787076+0000","last_became_peered":"2024-03-23T03:51:21.787076+0000","last_unstale":"2024-03-23T03:51:22.250446+0000","last_undegraded":"2024-03-23T03:51:22.250446+0000","last_fullsized":"2024-03-23T03:51:22.250446+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T08:09:48.708095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.282566+0000","last_change":"2024-03-23T03:51:22.280399+0000","last_active":"2024-03-23T03:51:22.282566+0000","last_peered":"2024-03-23T03:51:22.282566+0000","last_clean":"2024-03-23T03:51:22.282566+0000","last_became_active":"2024-03-23T03:51:22.280129+0000","last_became_peered":"2024-03-23T03:51:22.280129+0000","last_unstale":"2024-03-23T03:51:22.282566+0000","last_undegraded":"2024-03-23T03:51:22.282566+0000","last_fullsized":"2024-03-23T03:51:22.282566+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T07:05:10.465032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.263361+0000","last_change":"2024-03-23T03:51:21.789939+0000","last_active":"2024-03-23T03:51:22.263361+0000","last_peered":"2024-03-23T03:51:22.263361+0000","last_clean":"2024-03-23T03:51:22.263361+0000","last_became_active":"2024-03-23T03:51:21.788412+0000","last_became_peered":"2024-03-23T03:51:21.788412+0000","last_unstale":"2024-03-23T03:51:22.263361+0000","last_undegraded":"2024-03-23T03:51:22.263361+0000","last_fullsized":"2024-03-23T03:51:22.263361+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:35:47.134641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.283863+0000","last_change":"2024-03-23T03:51:22.272329+0000","last_active":"2024-03-23T03:51:22.283863+0000","last_peered":"2024-03-23T03:51:22.283863+0000","last_clean":"2024-03-23T03:51:22.283863+0000","last_became_active":"2024-03-23T03:51:22.271506+0000","last_became_peered":"2024-03-23T03:51:22.271506+0000","last_unstale":"2024-03-23T03:51:22.283863+0000","last_undegraded":"2024-03-23T03:51:22.283863+0000","last_fullsized":"2024-03-23T03:51:22.283863+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:05:03.516080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.249914+0000","last_change":"2024-03-23T03:51:21.791827+0000","last_active":"2024-03-23T03:51:22.249914+0000","last_peered":"2024-03-23T03:51:22.249914+0000","last_clean":"2024-03-23T03:51:22.249914+0000","last_became_active":"2024-03-23T03:51:21.791445+0000","last_became_peered":"2024-03-23T03:51:21.791445+0000","last_unstale":"2024-03-23T03:51:22.249914+0000","last_undegraded":"2024-03-23T03:51:22.249914+0000","last_fullsized":"2024-03-23T03:51:22.249914+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:58.150428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":10,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.275945+0000","last_change":"2024-03-23T03:51:22.270121+0000","last_active":"2024-03-23T03:51:22.275945+0000","last_peered":"2024-03-23T03:51:22.275945+0000","last_clean":"2024-03-23T03:51:22.275945+0000","last_became_active":"2024-03-23T03:51:22.269806+0000","last_became_peered":"2024-03-23T03:51:22.269806+0000","last_unstale":"2024-03-23T03:51:22.275945+0000","last_undegraded":"2024-03-23T03:51:22.275945+0000","last_fullsized":"2024-03-23T03:51:22.275945+0000","mapping_epoch":103,"log_start":"0'0","ondisk_log_start":"0'0","created":103,"last_epoch_clean":104,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:21.733694+0000","last_clean_scrub_stamp":"2024-03-23T03:51:21.733694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T04:04:21.319327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"104'2","reported_seq":13,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.257587+0000","last_change":"2024-03-23T03:51:21.779521+0000","last_active":"2024-03-23T03:51:22.257587+0000","last_peered":"2024-03-23T03:51:22.257587+0000","last_clean":"2024-03-23T03:51:22.257587+0000","last_became_active":"2024-03-23T03:51:21.779111+0000","last_became_peered":"2024-03-23T03:51:21.779111+0000","last_unstale":"2024-03-23T03:51:22.257587+0000","last_undegraded":"2024-03-23T03:51:22.257587+0000","last_fullsized":"2024-03-23T03:51:22.257587+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":102,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:51:20.723103+0000","last_clean_scrub_stamp":"2024-03-23T03:51:20.723103+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T05:29:39.956286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"19'117","reported_seq":230,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-03-23T03:51:22.260723+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:51:22.260723+0000","last_peered":"2024-03-23T03:51:22.260723+0000","last_clean":"2024-03-23T03:51:22.260723+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:51:22.260723+0000","last_undegraded":"2024-03-23T03:51:22.260723+0000","last_fullsized":"2024-03-23T03:51:22.260723+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542204,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32980,"kb_used_data":1856,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338860,"statfs":{"total":96636764160,"available":96602992640,"internally_reserved":0,"allocated":1900544,"data_stored":1180111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.087,"15min":1.0720000000000001},"min":{"1min":0.65700000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5209999999999999,"5min":1.635,"15min":1.635},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.042,"15min":1.0469999999999999},"min":{"1min":0.749,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.393,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.0049999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.081,"15min":1.0529999999999999},"min":{"1min":0.71499999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6120000000000001,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.147},{"interface":"front","average":{"1min":1.03,"5min":1.1719999999999999,"15min":1.2809999999999999},"min":{"1min":0.79000000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.321,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":1.5109999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.125,"15min":1.151},"min":{"1min":0.81299999999999994,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.583,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.171},{"interface":"front","average":{"1min":1.167,"5min":1.2869999999999999,"15min":1.401},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.506,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.4990000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.298,"15min":1.4179999999999999},"min":{"1min":0.71999999999999997,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.4219999999999999,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.325,"15min":1.353},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.677,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.452}]},{"osd":4,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.254,"15min":1.2629999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.8220000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.272},{"interface":"front","average":{"1min":1.333,"5min":1.347,"15min":1.3720000000000001},"min":{"1min":0.67500000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.778,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":0.92900000000000005}]},{"osd":5,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.389,"15min":1.4179999999999999},"min":{"1min":0.81299999999999994,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.7310000000000001,"5min":2.645,"15min":2.645},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.361,"5min":1.417,"15min":1.446},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.7649999999999999,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.119}]},{"osd":6,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.504,"15min":1.552},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.9279999999999999,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.421},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.474,"15min":1.5149999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9399999999999999,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.321}]}]},{"osd":6,"up_from":14,"seq":60129542202,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32404,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339436,"statfs":{"total":96636764160,"available":96603582464,"internally_reserved":0,"allocated":1318912,"data_stored":591751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.0429999999999999,"15min":1.044},"min":{"1min":0.69699999999999995,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.05,"15min":1.04},"min":{"1min":0.66100000000000003,"5min":0.62,"15min":0.62},"max":{"1min":1.5409999999999999,"5min":1.946,"15min":1.946},"last":1.0580000000000001}]},{"osd":1,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.087,"15min":1.075},"min":{"1min":0.745,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":1.204},{"interface":"front","average":{"1min":1.196,"5min":1.1339999999999999,"15min":1.1579999999999999},"min":{"1min":0.80500000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.5369999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.1439999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.083,"15min":1.071},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.4350000000000001,"5min":2.044,"15min":2.044},"last":1.327},{"interface":"front","average":{"1min":1.133,"5min":1.1759999999999999,"15min":1.212},"min":{"1min":0.85499999999999998,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.6599999999999999,"5min":1.681,"15min":1.681},"last":1.109}]},{"osd":3,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.2370000000000001,"15min":1.276},"min":{"1min":0.88100000000000001,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.786,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.85999999999999999},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.2330000000000001,"15min":1.2569999999999999},"min":{"1min":0.90100000000000002,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.446,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.256}]},{"osd":4,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3779999999999999,"15min":1.385},"min":{"1min":1.1930000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.6679999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.431,"5min":1.371,"15min":1.3859999999999999},"min":{"1min":1.234,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.639,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.52}]},{"osd":5,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.403,"15min":1.4319999999999999},"min":{"1min":1.0669999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.794,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.474,"5min":1.508,"15min":1.542},"min":{"1min":1.091,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8200000000000001,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.5669999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.4059999999999999,"15min":1.4119999999999999},"min":{"1min":0.95999999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.0760000000000001,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.4550000000000001},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.482,"15min":1.5309999999999999},"min":{"1min":1.125,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.8660000000000001,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.373}]}]},{"osd":5,"up_from":14,"seq":60129542202,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28296,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343544,"statfs":{"total":96636764160,"available":96607789056,"internally_reserved":0,"allocated":1298432,"data_stored":589583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1439999999999999,"15min":1.2030000000000001},"min":{"1min":0.72999999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4830000000000001,"5min":1.806,"15min":1.806},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.113,"5min":1.1399999999999999,"15min":1.1759999999999999},"min":{"1min":0.76500000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.855,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":0.93300000000000005}]},{"osd":1,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0940000000000001,"15min":1.1000000000000001},"min":{"1min":0.80100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.3520000000000001,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.232,"5min":1.165,"15min":1.1930000000000001},"min":{"1min":0.84799999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.663,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.167}]},{"osd":2,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.274,"15min":1.3460000000000001},"min":{"1min":0.70499999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8939999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.226,"5min":1.2989999999999999,"15min":1.3660000000000001},"min":{"1min":0.875,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.579,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.88400000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.2,"15min":1.244},"min":{"1min":0.85399999999999998,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.7050000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.375,"15min":1.446},"min":{"1min":0.997,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.964,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.3340000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3049999999999999,"15min":1.399},"min":{"1min":0.65700000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6100000000000001,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.038},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3859999999999999,"15min":1.4359999999999999},"min":{"1min":0.83699999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8069999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.4370000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.4159999999999999,"15min":1.5},"min":{"1min":0.83399999999999996,"5min":0.63,"15min":0.63},"max":{"1min":1.5840000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.5409999999999999,"15min":1.639},"min":{"1min":1.1950000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.6359999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.3049999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4910000000000001,"15min":1.591},"min":{"1min":1.095,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.9259999999999999,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.268},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5840000000000001,"15min":1.6839999999999999},"min":{"1min":1.272,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.056,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.3939999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542203,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32392,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339448,"statfs":{"total":96636764160,"available":96603594752,"internally_reserved":0,"allocated":1306624,"data_stored":589675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.004,"15min":0.98099999999999998},"min":{"1min":0.624,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.47,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.995,"15min":1.0049999999999999},"min":{"1min":0.56100000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3400000000000001,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.84999999999999998}]},{"osd":1,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0680000000000001,"15min":1.05},"min":{"1min":0.65100000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.5,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.226},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0880000000000001,"15min":1.0640000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.5209999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.272}]},{"osd":2,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.2470000000000001,"15min":1.3089999999999999},"min":{"1min":0.70399999999999996,"5min":0.625,"15min":0.625},"max":{"1min":1.452,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.2789999999999999,"15min":1.2969999999999999},"min":{"1min":0.59899999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6859999999999999,"5min":2.181,"15min":2.181},"last":1.4510000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.1160000000000001,"15min":1.1240000000000001},"min":{"1min":0.61899999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.4419999999999999,"5min":1.821,"15min":1.821},"last":1.097},{"interface":"front","average":{"1min":1.141,"5min":1.2310000000000001,"15min":1.268},"min":{"1min":0.73799999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.599,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.0529999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.2370000000000001,"15min":1.26},"min":{"1min":0.63800000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.49,"5min":1.929,"15min":1.929},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.2709999999999999,"15min":1.274},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.45,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.3959999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.371,"15min":1.409},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6719999999999999,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.4139999999999999,"15min":1.3999999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.8100000000000001,"5min":2.016,"15min":2.016},"last":1.5580000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.361,"15min":1.363},"min":{"1min":0.68200000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.5569999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.456,"15min":1.4870000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.716,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.494}]}]},{"osd":3,"up_from":14,"seq":60129542203,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32396,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339444,"statfs":{"total":96636764160,"available":96603590656,"internally_reserved":0,"allocated":1302528,"data_stored":590165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.81999999999999995,"15min":0.85499999999999998},"min":{"1min":0.35499999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.4419999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":1.48},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.92200000000000004,"15min":0.92400000000000004},"min":{"1min":0.502,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.538,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":1.0129999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.91500000000000004,"15min":0.96799999999999997},"min":{"1min":0.58099999999999996,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.9990000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.0229999999999999,"15min":1.069},"min":{"1min":0.52000000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.651,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.77000000000000002}]},{"osd":2,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.002,"15min":1.0509999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.5710000000000002,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1240000000000001,"15min":1.1499999999999999},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.7649999999999999,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":1.2170000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.2410000000000001,"15min":1.248},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":4.0380000000000003,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":1.3420000000000001},{"interface":"front","average":{"1min":1.337,"5min":1.351,"15min":1.429},"min":{"1min":0.77000000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.3450000000000002,"5min":4.819,"15min":4.819},"last":1.1899999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.6040000000000001,"15min":1.4470000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.2679999999999998,"5min":30.228000000000002,"15min":30.228000000000002},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.337,"15min":1.357},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":4.3630000000000004,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":1.115}]},{"osd":6,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4119999999999999,"15min":1.508},"min":{"1min":0.83999999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.6589999999999998,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.302},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4199999999999999,"15min":1.4039999999999999},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.4039999999999999,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":1.409}]},{"osd":7,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.389,"15min":1.4199999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.0880000000000001,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":1.264},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.542,"15min":1.5900000000000001},"min":{"1min":0.80500000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.4780000000000002,"5min":10.663,"15min":10.663},"last":1.377}]}]},{"osd":2,"up_from":14,"seq":60129542203,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32388,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339452,"statfs":{"total":96636764160,"available":96603598848,"internally_reserved":0,"allocated":1302528,"data_stored":589617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83799999999999997,"15min":0.877},"min":{"1min":0.52100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":1.4259999999999999,"5min":1.456,"15min":1.456},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.86399999999999999,"15min":0.93600000000000005},"min":{"1min":0.39500000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.4690000000000001,"5min":2.226,"15min":2.226},"last":0.65500000000000003}]},{"osd":1,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.93899999999999995,"15min":0.98499999999999999},"min":{"1min":0.46800000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.377,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.85899999999999999,"15min":0.91000000000000003},"min":{"1min":0.57599999999999996,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.2210000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.59399999999999997}]},{"osd":3,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.96099999999999997,"15min":1.018},"min":{"1min":0.47499999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.403,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.032,"5min":1.042,"15min":1.0800000000000001},"min":{"1min":0.61099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.63300000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.194,"15min":1.2310000000000001},"min":{"1min":0.82399999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.7270000000000001,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.157,"15min":1.2070000000000001},"min":{"1min":0.83799999999999997,"5min":0.627,"15min":0.627},"max":{"1min":1.4950000000000001,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.1950000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.3100000000000001,"15min":1.367},"min":{"1min":0.76900000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.6259999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.2749999999999999,"15min":1.339},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.819,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.0669999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.232,"15min":1.3280000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6479999999999999,"5min":2.456,"15min":2.456},"last":1.272},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3999999999999999,"15min":1.476},"min":{"1min":0.85299999999999998,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.097,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.0820000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3069999999999999,"15min":1.369},"min":{"1min":0.79800000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.171},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.302,"15min":1.395},"min":{"1min":0.78700000000000003,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5860000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.3700000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542202,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28300,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343540,"statfs":{"total":96636764160,"available":96607784960,"internally_reserved":0,"allocated":1302528,"data_stored":589617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79100000000000004,"15min":0.86099999999999999},"min":{"1min":0.41599999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.1240000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.0800000000000001},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81799999999999995,"15min":0.82999999999999996},"min":{"1min":0.435,"5min":0.374,"15min":0.374},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.69199999999999995}]},{"osd":2,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.92200000000000004,"15min":0.96399999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.4430000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.75},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.92400000000000004,"15min":0.97799999999999998},"min":{"1min":0.496,"5min":0.443,"15min":0.443},"max":{"1min":1.2210000000000001,"5min":1.625,"15min":1.625},"last":0.89500000000000002}]},{"osd":3,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.103,"15min":1.113},"min":{"1min":0.68400000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.6559999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":0.875,"5min":0.96199999999999997,"15min":1.008},"min":{"1min":0.59499999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.6180000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.94499999999999995}]},{"osd":4,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.179,"15min":1.2749999999999999},"min":{"1min":0.67700000000000005,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.5069999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.99299999999999999},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.246,"15min":1.3049999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6120000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.165}]},{"osd":5,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.248,"15min":1.3200000000000001},"min":{"1min":0.79500000000000004,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.95,"5min":3.593,"15min":3.593},"last":1.0580000000000001},{"interface":"front","average":{"1min":1.264,"5min":1.327,"15min":1.4259999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.671,"5min":2.359,"15min":2.359},"last":1.4039999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.3089999999999999,"15min":1.3919999999999999},"min":{"1min":0.84199999999999997,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6200000000000001,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.357},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.288,"15min":1.3540000000000001},"min":{"1min":0.89100000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.0209999999999999,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.1399999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.359,"15min":1.4330000000000001},"min":{"1min":0.99299999999999999,"5min":0.627,"15min":0.627},"max":{"1min":2.0840000000000001,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.363,"15min":1.486},"min":{"1min":0.76600000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.673,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.4530000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542202,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32988,"kb_used_data":1864,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338852,"statfs":{"total":96636764160,"available":96602984448,"internally_reserved":0,"allocated":1908736,"data_stored":1181605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.86399999999999999,"15min":0.93500000000000005},"min":{"1min":0.48399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.478,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.94799999999999995,"15min":0.98199999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.456,"5min":1.506,"15min":1.506},"last":1.2709999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.92700000000000005,"15min":0.99099999999999999},"min":{"1min":0.52900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.4399999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.0349999999999999,"15min":1.095},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.601,"5min":1.901,"15min":1.901},"last":0.95999999999999996}]},{"osd":3,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.93300000000000005,"15min":0.996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.474,"5min":1.972,"15min":1.972},"last":0.878},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0669999999999999,"15min":1.147},"min":{"1min":0.60299999999999998,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.597,"5min":2.298,"15min":2.298},"last":1.0449999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.288,"15min":1.3360000000000001},"min":{"1min":0.72799999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.7090000000000001,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":1.367},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.2230000000000001,"15min":1.27},"min":{"1min":0.80600000000000005,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.8380000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.1100000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3100000000000001,"15min":1.3700000000000001},"min":{"1min":0.89500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.948,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.573},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.377,"15min":1.4379999999999999},"min":{"1min":0.83499999999999996,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.8700000000000001,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.5429999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.4099999999999999,"15min":1.5209999999999999},"min":{"1min":0.80000000000000004,"5min":0.747,"15min":0.747},"max":{"1min":1.978,"5min":3.032,"15min":3.032},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.371,"5min":1.411,"15min":1.4870000000000001},"min":{"1min":0.84699999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.1349999999999998,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.1919999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4390000000000001,"15min":1.506},"min":{"1min":0.90500000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1850000000000001,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.45},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3620000000000001,"15min":1.419},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.873,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.0049999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:51:32.625 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T03:51:32.625 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-23T03:51:32.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.884+0000 7f7d6e800640 1 Processor -- start 2024-03-23T03:51:32.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.884+0000 7f7d6e800640 1 -- start start 2024-03-23T03:51:32.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.885+0000 7f7d6e800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d6815a780 0x7f7d68099a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:32.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.885+0000 7f7d6e800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f7d6815ab60 0x7f7d680cbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:32.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.885+0000 7f7d6e800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d680ce560 con 0x7f7d680cc1f0 2024-03-23T03:51:32.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.885+0000 7f7d6e800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d68099f90 con 0x7f7d6815ab60 2024-03-23T03:51:32.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.885+0000 7f7d6e800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d68098a40 con 0x7f7d6815a780 2024-03-23T03:51:32.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.886+0000 7f7d67e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7d680cc1f0 0x7f7d680cec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50032/0 (socket says 172.21.15.102:50032) 2024-03-23T03:51:32.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.886+0000 7f7d67e00640 1 -- 172.21.15.102:0/4028980455 learned_addr learned my addr 172.21.15.102:0/4028980455 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:32.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.886+0000 7f7d66a00640 1 --2- 172.21.15.102:0/4028980455 >> v2:172.21.15.173:3300/0 conn(0x7f7d6815ab60 0x7f7d680cbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:32.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.886+0000 7f7d67400640 1 --2- 172.21.15.102:0/4028980455 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d6815a780 0x7f7d68099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:32.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.886+0000 7f7d66000640 1 -- 172.21.15.102:0/4028980455 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223104329 0 0) 0x7f7d680ce560 con 0x7f7d680cc1f0 2024-03-23T03:51:32.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.886+0000 7f7d66000640 1 -- 172.21.15.102:0/4028980455 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d48003520 con 0x7f7d680cc1f0 2024-03-23T03:51:32.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.886+0000 7f7d66000640 1 -- 172.21.15.102:0/4028980455 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3284356692 0 0) 0x7f7d48003520 con 0x7f7d680cc1f0 2024-03-23T03:51:32.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.886+0000 7f7d66000640 1 -- 172.21.15.102:0/4028980455 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d6815a780 msgr2=0x7f7d68099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:32.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.886+0000 7f7d66000640 1 --2- 172.21.15.102:0/4028980455 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d6815a780 0x7f7d68099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:32.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.886+0000 7f7d66000640 1 -- 172.21.15.102:0/4028980455 >> v2:172.21.15.173:3300/0 conn(0x7f7d6815ab60 msgr2=0x7f7d680cbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:32.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.886+0000 7f7d66000640 1 --2- 172.21.15.102:0/4028980455 >> v2:172.21.15.173:3300/0 conn(0x7f7d6815ab60 0x7f7d680cbcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:32.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.887+0000 7f7d66000640 1 -- 172.21.15.102:0/4028980455 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d68098c90 con 0x7f7d680cc1f0 2024-03-23T03:51:32.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.887+0000 7f7d66000640 1 -- 172.21.15.102:0/4028980455 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f7d600019f0 con 0x7f7d680cc1f0 2024-03-23T03:51:32.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.887+0000 7f7d66000640 1 -- 172.21.15.102:0/4028980455 >> v1:172.21.15.102:6789/0 conn(0x7f7d680cc1f0 legacy=0x7f7d680cec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:32.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.887+0000 7f7d66000640 1 --2- 172.21.15.102:0/4028980455 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d6815a780 0x7f7d48003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:32.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.887+0000 7f7d66000640 1 --2- 172.21.15.102:0/4028980455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d48003ef0 0x7f7d480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:32.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.887+0000 7f7d66000640 1 --2- 172.21.15.102:0/4028980455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d48004d00 0x7f7d48005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:32.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.887+0000 7f7d66000640 1 -- 172.21.15.102:0/4028980455 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d68099f90 con 0x7f7d48004d00 2024-03-23T03:51:32.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.887+0000 7f7d66000640 1 -- 172.21.15.102:0/4028980455 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d68098a40 con 0x7f7d48003ef0 2024-03-23T03:51:32.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.887+0000 7f7d66000640 1 -- 172.21.15.102:0/4028980455 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4800b980 con 0x7f7d6815a780 2024-03-23T03:51:32.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.887+0000 7f7d67e00640 1 --2- 172.21.15.102:0/4028980455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d48004d00 0x7f7d48005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:32.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.887+0000 7f7d66a00640 1 --2- 172.21.15.102:0/4028980455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d48003ef0 0x7f7d480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:32.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.887+0000 7f7d67400640 1 --2- 172.21.15.102:0/4028980455 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d6815a780 0x7f7d48003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:32.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.887+0000 7f7d67e00640 1 -- 172.21.15.102:0/4028980455 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d6815a780 msgr2=0x7f7d48003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:32.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.887+0000 7f7d67e00640 1 --2- 172.21.15.102:0/4028980455 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d6815a780 0x7f7d48003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:32.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.887+0000 7f7d67e00640 1 -- 172.21.15.102:0/4028980455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d48003ef0 msgr2=0x7f7d480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:32.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.887+0000 7f7d67e00640 1 --2- 172.21.15.102:0/4028980455 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d48003ef0 0x7f7d480047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:32.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.887+0000 7f7d67e00640 1 -- 172.21.15.102:0/4028980455 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d48003520 con 0x7f7d48004d00 2024-03-23T03:51:32.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.889+0000 7f7d67e00640 1 --2- 172.21.15.102:0/4028980455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d48004d00 0x7f7d48005180 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000a8d0 tx=0x7f7d480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=316700f67d4520ba server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:32.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.889+0000 7f7d66000640 1 -- 172.21.15.102:0/4028980455 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d60034630 con 0x7f7d48004d00 2024-03-23T03:51:32.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.889+0000 7f7d66000640 1 -- 172.21.15.102:0/4028980455 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6002f070 con 0x7f7d48004d00 2024-03-23T03:51:32.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.889+0000 7f7d66000640 1 -- 172.21.15.102:0/4028980455 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d600349d0 con 0x7f7d48004d00 2024-03-23T03:51:32.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.890+0000 7f7d6e800640 1 -- 172.21.15.102:0/4028980455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d48004d00 msgr2=0x7f7d48005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:32.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.890+0000 7f7d6e800640 1 --2- 172.21.15.102:0/4028980455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d48004d00 0x7f7d48005180 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f7d6000a8d0 tx=0x7f7d480057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:32.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.890+0000 7f7d67e00640 1 -- 172.21.15.102:0/4028980455 reap_dead start 2024-03-23T03:51:32.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.890+0000 7f7d6e800640 1 -- 172.21.15.102:0/4028980455 shutdown_connections 2024-03-23T03:51:32.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.890+0000 7f7d6e800640 1 -- 172.21.15.102:0/4028980455 >> 172.21.15.102:0/4028980455 conn(0x7f7d680b2180 msgr2=0x7f7d680b2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:32.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.890+0000 7f7d6e800640 1 -- 172.21.15.102:0/4028980455 shutdown_connections 2024-03-23T03:51:32.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.890+0000 7f7d6e800640 1 -- 172.21.15.102:0/4028980455 wait complete. 2024-03-23T03:51:32.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.892+0000 7f7d6e800640 1 Processor -- start 2024-03-23T03:51:32.893 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.892+0000 7f7d6e800640 1 -- start start 2024-03-23T03:51:32.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.893+0000 7f7d6e800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d48004d00 0x7f7d6806ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:32.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.893+0000 7f7d6e800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d680c9040 0x7f7d6814f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:32.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.893+0000 7f7d6e800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d680c94c0 0x7f7d6814fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:32.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.893+0000 7f7d6e800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6814ff70 con 0x7f7d48004d00 2024-03-23T03:51:32.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.893+0000 7f7d6e800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6806b4a0 con 0x7f7d680c9040 2024-03-23T03:51:32.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.893+0000 7f7d6e800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6806b780 con 0x7f7d680c94c0 2024-03-23T03:51:32.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.893+0000 7f7d67400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d48004d00 0x7f7d6806ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:32.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.893+0000 7f7d67400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d48004d00 0x7f7d6806ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48330/0 (socket says 172.21.15.102:48330) 2024-03-23T03:51:32.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.893+0000 7f7d67400640 1 -- 172.21.15.102:0/4012796348 learned_addr learned my addr 172.21.15.102:0/4012796348 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:32.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.893+0000 7f7d67400640 1 -- 172.21.15.102:0/4012796348 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d680c94c0 msgr2=0x7f7d6814fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:32.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.893+0000 7f7d67400640 1 --2- 172.21.15.102:0/4012796348 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d680c94c0 0x7f7d6814fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:32.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.893+0000 7f7d67400640 1 -- 172.21.15.102:0/4012796348 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d680c9040 msgr2=0x7f7d6814f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:32.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.893+0000 7f7d67400640 1 --2- 172.21.15.102:0/4012796348 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d680c9040 0x7f7d6814f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:32.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.893+0000 7f7d67400640 1 -- 172.21.15.102:0/4012796348 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6806ba60 con 0x7f7d48004d00 2024-03-23T03:51:32.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.894+0000 7f7d67400640 1 --2- 172.21.15.102:0/4012796348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d48004d00 0x7f7d6806ada0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f7d58010860 tx=0x7f7d58010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:32.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.894+0000 7f7d5f400640 1 -- 172.21.15.102:0/4012796348 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d58009610 con 0x7f7d48004d00 2024-03-23T03:51:32.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.894+0000 7f7d5f400640 1 -- 172.21.15.102:0/4012796348 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5800eae0 con 0x7f7d48004d00 2024-03-23T03:51:32.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.894+0000 7f7d5f400640 1 -- 172.21.15.102:0/4012796348 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d5800ee70 con 0x7f7d48004d00 2024-03-23T03:51:32.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.894+0000 7f7d6e800640 1 -- 172.21.15.102:0/4012796348 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d6806bd40 con 0x7f7d48004d00 2024-03-23T03:51:32.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.894+0000 7f7d6e800640 1 -- 172.21.15.102:0/4012796348 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d681f2580 con 0x7f7d48004d00 2024-03-23T03:51:32.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.896+0000 7f7d6e800640 1 -- 172.21.15.102:0/4012796348 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d68066bb0 con 0x7f7d48004d00 2024-03-23T03:51:32.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.896+0000 7f7d5f400640 1 -- 172.21.15.102:0/4012796348 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7d5801c400 con 0x7f7d48004d00 2024-03-23T03:51:32.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.897+0000 7f7d5f400640 1 --2- 172.21.15.102:0/4012796348 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d4406ac30 0x7f7d4406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:32.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.897+0000 7f7d5f400640 1 -- 172.21.15.102:0/4012796348 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7d58089950 con 0x7f7d48004d00 2024-03-23T03:51:32.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.897+0000 7f7d66a00640 1 --2- 172.21.15.102:0/4012796348 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d4406ac30 0x7f7d4406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:32.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.898+0000 7f7d66a00640 1 --2- 172.21.15.102:0/4012796348 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d4406ac30 0x7f7d4406d0f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7d680674c0 tx=0x7f7d50008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:32.902 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:32.902+0000 7f7d5f400640 1 -- 172.21.15.102:0/4012796348 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7d580549f0 con 0x7f7d48004d00 2024-03-23T03:51:33.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.163+0000 7f7d6e800640 1 -- 172.21.15.102:0/4012796348 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f7d680b2ea0 con 0x7f7d48004d00 2024-03-23T03:51:33.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.171+0000 7f7d5f400640 1 -- 172.21.15.102:0/4012796348 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+0 (secure 0 0 0) 0x7f7d58058010 con 0x7f7d48004d00 2024-03-23T03:51:33.172 INFO:teuthology.orchestra.run.smithi102.stderr:updated caps for client.0 2024-03-23T03:51:33.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.174+0000 7f7d6e800640 1 -- 172.21.15.102:0/4012796348 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d4406ac30 msgr2=0x7f7d4406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:33.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.174+0000 7f7d6e800640 1 --2- 172.21.15.102:0/4012796348 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d4406ac30 0x7f7d4406d0f0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7d680674c0 tx=0x7f7d50008040 comp rx=0 tx=0).stop 2024-03-23T03:51:33.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.174+0000 7f7d6e800640 1 -- 172.21.15.102:0/4012796348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d48004d00 msgr2=0x7f7d6806ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:33.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.174+0000 7f7d6e800640 1 --2- 172.21.15.102:0/4012796348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d48004d00 0x7f7d6806ada0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f7d58010860 tx=0x7f7d58010d20 comp rx=0 tx=0).stop 2024-03-23T03:51:33.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.174+0000 7f7d6e800640 1 -- 172.21.15.102:0/4012796348 shutdown_connections 2024-03-23T03:51:33.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.174+0000 7f7d6e800640 1 --2- 172.21.15.102:0/4012796348 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7d4406ac30 0x7f7d4406d0f0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:33.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.174+0000 7f7d6e800640 1 --2- 172.21.15.102:0/4012796348 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7d680c94c0 0x7f7d6814fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:33.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.174+0000 7f7d6e800640 1 --2- 172.21.15.102:0/4012796348 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7d680c9040 0x7f7d6814f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:33.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.174+0000 7f7d6e800640 1 --2- 172.21.15.102:0/4012796348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7d48004d00 0x7f7d6806ada0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:33.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.174+0000 7f7d6e800640 1 -- 172.21.15.102:0/4012796348 >> 172.21.15.102:0/4012796348 conn(0x7f7d680b2180 msgr2=0x7f7d6815b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:33.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.174+0000 7f7d6e800640 1 -- 172.21.15.102:0/4012796348 shutdown_connections 2024-03-23T03:51:33.176 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.174+0000 7f7d6e800640 1 -- 172.21.15.102:0/4012796348 wait complete. 2024-03-23T03:51:33.194 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:51:33.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.400+0000 7fb316e00640 1 Processor -- start 2024-03-23T03:51:33.401 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.401+0000 7fb316e00640 1 -- start start 2024-03-23T03:51:33.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.402+0000 7fb316e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb31015cda0 0x7fb3100cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:33.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.402+0000 7fb316e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3100b4a60 0x7fb3100b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:33.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.402+0000 7fb316e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3100d1830 con 0x7fb3100b5a30 2024-03-23T03:51:33.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.402+0000 7fb316e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3100b1f20 con 0x7fb31015cda0 2024-03-23T03:51:33.407 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.402+0000 7fb316e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3100d0c20 con 0x7fb3100b4a60 2024-03-23T03:51:33.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.403+0000 7fb30fe00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb31015cda0 0x7fb3100cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:33.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.403+0000 7fb30fe00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fb31015cda0 0x7fb3100cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:35550/0 (socket says 172.21.15.102:35550) 2024-03-23T03:51:33.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.403+0000 7fb30fe00640 1 -- 172.21.15.102:0/3284164203 learned_addr learned my addr 172.21.15.102:0/3284164203 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:33.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.403+0000 7fb314e00640 1 --2- 172.21.15.102:0/3284164203 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3100b4a60 0x7fb3100b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:33.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.404+0000 7fb314e00640 1 -- 172.21.15.102:0/3284164203 >> v2:172.21.15.173:3300/0 conn(0x7fb31015cda0 msgr2=0x7fb3100cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:33.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.404+0000 7fb314e00640 1 --2- 172.21.15.102:0/3284164203 >> v2:172.21.15.173:3300/0 conn(0x7fb31015cda0 0x7fb3100cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:33.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.404+0000 7fb314e00640 1 -- 172.21.15.102:0/3284164203 >> v1:172.21.15.173:6789/0 conn(0x7fb3100b5a30 legacy=0x7fb3100b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:33.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.404+0000 7fb314e00640 1 -- 172.21.15.102:0/3284164203 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3100d0f00 con 0x7fb3100b4a60 2024-03-23T03:51:33.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.405+0000 7fb314e00640 1 --2- 172.21.15.102:0/3284164203 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3100b4a60 0x7fb3100b5e90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb308009d90 tx=0x7fb30802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bafbed4cf7641c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:33.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.406+0000 7fb30ea00640 1 -- 172.21.15.102:0/3284164203 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30802cc50 con 0x7fb3100b4a60 2024-03-23T03:51:33.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.406+0000 7fb30ea00640 1 -- 172.21.15.102:0/3284164203 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30802cdb0 con 0x7fb3100b4a60 2024-03-23T03:51:33.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.406+0000 7fb30ea00640 1 -- 172.21.15.102:0/3284164203 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308034660 con 0x7fb3100b4a60 2024-03-23T03:51:33.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.406+0000 7fb316e00640 1 -- 172.21.15.102:0/3284164203 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3100b4a60 msgr2=0x7fb3100b5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:33.408 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.406+0000 7fb316e00640 1 --2- 172.21.15.102:0/3284164203 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3100b4a60 0x7fb3100b5e90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb308009d90 tx=0x7fb30802bea0 comp rx=0 tx=0).stop 2024-03-23T03:51:33.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.406+0000 7fb316e00640 1 -- 172.21.15.102:0/3284164203 shutdown_connections 2024-03-23T03:51:33.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.406+0000 7fb316e00640 1 --2- 172.21.15.102:0/3284164203 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb3100b4a60 0x7fb3100b5e90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:33.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.406+0000 7fb316e00640 1 --2- 172.21.15.102:0/3284164203 >> v2:172.21.15.173:3300/0 conn(0x7fb31015cda0 0x7fb3100cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:33.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.406+0000 7fb316e00640 1 -- 172.21.15.102:0/3284164203 >> 172.21.15.102:0/3284164203 conn(0x7fb3100cf350 msgr2=0x7fb3100cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:33.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.407+0000 7fb316e00640 1 -- 172.21.15.102:0/3284164203 shutdown_connections 2024-03-23T03:51:33.409 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.407+0000 7fb316e00640 1 -- 172.21.15.102:0/3284164203 wait complete. 2024-03-23T03:51:33.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.408+0000 7fb316e00640 1 Processor -- start 2024-03-23T03:51:33.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.408+0000 7fb316e00640 1 -- start start 2024-03-23T03:51:33.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.409+0000 7fb316e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3100b4a60 0x7fb310156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:33.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.409+0000 7fb316e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3100b5a30 0x7fb31006b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:33.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.409+0000 7fb316e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb31015cda0 0x7fb310155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:33.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.409+0000 7fb316e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3101f5e10 con 0x7fb3100b4a60 2024-03-23T03:51:33.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.409+0000 7fb316e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3101f5500 con 0x7fb3100b5a30 2024-03-23T03:51:33.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.409+0000 7fb316e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb310157860 con 0x7fb31015cda0 2024-03-23T03:51:33.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.410+0000 7fb30fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3100b4a60 0x7fb310156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:33.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.410+0000 7fb30fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3100b4a60 0x7fb310156f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48344/0 (socket says 172.21.15.102:48344) 2024-03-23T03:51:33.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.410+0000 7fb30fe00640 1 -- 172.21.15.102:0/891484205 learned_addr learned my addr 172.21.15.102:0/891484205 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:33.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.410+0000 7fb314e00640 1 --2- 172.21.15.102:0/891484205 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb31015cda0 0x7fb310155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:33.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.410+0000 7fb30fe00640 1 -- 172.21.15.102:0/891484205 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb31015cda0 msgr2=0x7fb310155800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:33.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.410+0000 7fb30f400640 1 --2- 172.21.15.102:0/891484205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3100b5a30 0x7fb31006b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:33.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.410+0000 7fb30fe00640 1 --2- 172.21.15.102:0/891484205 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb31015cda0 0x7fb310155800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:33.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.410+0000 7fb30fe00640 1 -- 172.21.15.102:0/891484205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3100b5a30 msgr2=0x7fb31006b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:33.412 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.410+0000 7fb30fe00640 1 --2- 172.21.15.102:0/891484205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3100b5a30 0x7fb31006b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:33.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.410+0000 7fb30fe00640 1 -- 172.21.15.102:0/891484205 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb308009a70 con 0x7fb3100b4a60 2024-03-23T03:51:33.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.410+0000 7fb314e00640 1 --2- 172.21.15.102:0/891484205 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb31015cda0 0x7fb310155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:33.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.410+0000 7fb30f400640 1 --2- 172.21.15.102:0/891484205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3100b5a30 0x7fb31006b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:33.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.410+0000 7fb30fe00640 1 --2- 172.21.15.102:0/891484205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3100b4a60 0x7fb310156f60 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fb30000e870 tx=0x7fb30000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:33.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.411+0000 7fb307e00640 1 -- 172.21.15.102:0/891484205 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30000c8e0 con 0x7fb3100b4a60 2024-03-23T03:51:33.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.411+0000 7fb307e00640 1 -- 172.21.15.102:0/891484205 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30000ca40 con 0x7fb3100b4a60 2024-03-23T03:51:33.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.411+0000 7fb307e00640 1 -- 172.21.15.102:0/891484205 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30000cdf0 con 0x7fb3100b4a60 2024-03-23T03:51:33.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.411+0000 7fb316e00640 1 -- 172.21.15.102:0/891484205 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3101f5750 con 0x7fb3100b4a60 2024-03-23T03:51:33.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.411+0000 7fb316e00640 1 -- 172.21.15.102:0/891484205 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3101f6230 con 0x7fb3100b4a60 2024-03-23T03:51:33.413 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.412+0000 7fb316e00640 1 -- 172.21.15.102:0/891484205 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3101f6420 con 0x7fb3100b4a60 2024-03-23T03:51:33.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.413+0000 7fb307e00640 1 -- 172.21.15.102:0/891484205 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fb300013470 con 0x7fb3100b4a60 2024-03-23T03:51:33.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.414+0000 7fb307e00640 1 --2- 172.21.15.102:0/891484205 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2ec06ac00 0x7fb2ec06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:33.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.414+0000 7fb307e00640 1 -- 172.21.15.102:0/891484205 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb3000895a0 con 0x7fb3100b4a60 2024-03-23T03:51:33.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.414+0000 7fb30f400640 1 --2- 172.21.15.102:0/891484205 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2ec06ac00 0x7fb2ec06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:33.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.415+0000 7fb30f400640 1 --2- 172.21.15.102:0/891484205 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2ec06ac00 0x7fb2ec06d0c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb310156a90 tx=0x7fb2f8009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:33.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.418+0000 7fb307e00640 1 -- 172.21.15.102:0/891484205 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fb300054710 con 0x7fb3100b4a60 2024-03-23T03:51:33.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.652+0000 7fb316e00640 1 -- 172.21.15.102:0/891484205 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb310073390 con 0x7fb3100b4a60 2024-03-23T03:51:33.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.653+0000 7fb307e00640 1 -- 172.21.15.102:0/891484205 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fb300057d30 con 0x7fb3100b4a60 2024-03-23T03:51:33.654 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:33.654 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2017025570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2017025570},{"type":"v1","addr":"172.21.15.173:6835","nonce":2017025570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/173893445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":173893445},{"type":"v1","addr":"172.21.15.102:6835","nonce":173893445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/255507815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":255507815},{"type":"v1","addr":"172.21.15.173:6833","nonce":255507815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:51:22.228808+0000","modified":"2024-03-23T03:51:25.724445+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7212":{"gid":7212,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/883705787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":883705787},{"type":"v1","addr":"172.21.15.102:6837","nonce":883705787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T03:51:33.654 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 56 2024-03-23T03:51:33.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.656+0000 7fb316e00640 1 -- 172.21.15.102:0/891484205 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2ec06ac00 msgr2=0x7fb2ec06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:33.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.656+0000 7fb316e00640 1 --2- 172.21.15.102:0/891484205 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2ec06ac00 0x7fb2ec06d0c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fb310156a90 tx=0x7fb2f8009040 comp rx=0 tx=0).stop 2024-03-23T03:51:33.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.656+0000 7fb316e00640 1 -- 172.21.15.102:0/891484205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3100b4a60 msgr2=0x7fb310156f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:33.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.656+0000 7fb316e00640 1 --2- 172.21.15.102:0/891484205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3100b4a60 0x7fb310156f60 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fb30000e870 tx=0x7fb30000ed30 comp rx=0 tx=0).stop 2024-03-23T03:51:33.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.656+0000 7fb316e00640 1 -- 172.21.15.102:0/891484205 shutdown_connections 2024-03-23T03:51:33.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.656+0000 7fb316e00640 1 --2- 172.21.15.102:0/891484205 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fb2ec06ac00 0x7fb2ec06d0c0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:33.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.656+0000 7fb316e00640 1 --2- 172.21.15.102:0/891484205 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fb31015cda0 0x7fb310155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:33.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.656+0000 7fb316e00640 1 --2- 172.21.15.102:0/891484205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3100b5a30 0x7fb31006b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:33.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.656+0000 7fb316e00640 1 --2- 172.21.15.102:0/891484205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3100b4a60 0x7fb310156f60 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:33.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.656+0000 7fb316e00640 1 -- 172.21.15.102:0/891484205 >> 172.21.15.102:0/891484205 conn(0x7fb3100cf350 msgr2=0x7fb3100cb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:33.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.656+0000 7fb316e00640 1 -- 172.21.15.102:0/891484205 shutdown_connections 2024-03-23T03:51:33.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:33.656+0000 7fb316e00640 1 -- 172.21.15.102:0/891484205 wait complete. 2024-03-23T03:51:34.078 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T03:51:22.228808+0000', 'modified': '2024-03-23T03:51:25.724445+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7212}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7212': {'gid': 7212, 'name': 'a', 'rank': 0, 'incarnation': 52, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.102:6837/883705787', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 883705787}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 883705787}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-23T03:51:34.079 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T03:51:34.079 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-23T03:51:34.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.299+0000 7f1460200640 1 Processor -- start 2024-03-23T03:51:34.300 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.299+0000 7f1460200640 1 -- start start 2024-03-23T03:51:34.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.301+0000 7f1460200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14580cd6b0 0x7f14580b8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.301+0000 7f1460200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f14580cda90 0x7f14580d1370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.301+0000 7f1460200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f145806b840 con 0x7f145815cd90 2024-03-23T03:51:34.301 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.301+0000 7f1460200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f145806c0e0 con 0x7f14580cda90 2024-03-23T03:51:34.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.301+0000 7f1460200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f145806bb20 con 0x7f14580cd6b0 2024-03-23T03:51:34.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.301+0000 7f145d800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14580cd6b0 0x7f14580b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.301+0000 7f145e200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f14580cda90 0x7f14580d1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.302+0000 7f145d800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14580cd6b0 0x7f14580b8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:52464/0 (socket says 172.21.15.102:52464) 2024-03-23T03:51:34.302 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.302+0000 7f145d800640 1 -- 172.21.15.102:0/2412881804 learned_addr learned my addr 172.21.15.102:0/2412881804 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:34.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.302+0000 7f145e200640 1 -- 172.21.15.102:0/2412881804 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14580cd6b0 msgr2=0x7f14580b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:34.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.302+0000 7f145e200640 1 --2- 172.21.15.102:0/2412881804 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14580cd6b0 0x7f14580b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.302+0000 7f145e200640 1 -- 172.21.15.102:0/2412881804 >> v1:172.21.15.173:6789/0 conn(0x7f145815cd90 legacy=0x7f145806b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:34.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.302+0000 7f145e200640 1 -- 172.21.15.102:0/2412881804 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f145806be00 con 0x7f14580cda90 2024-03-23T03:51:34.303 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.302+0000 7f145d800640 1 --2- 172.21.15.102:0/2412881804 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14580cd6b0 0x7f14580b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:51:34.304 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.304+0000 7f145e200640 1 --2- 172.21.15.102:0/2412881804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14580cda90 0x7f14580d1370 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f14500097a0 tx=0x7f145002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e17223b71348b149 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:34.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.304+0000 7f1457e00640 1 -- 172.21.15.102:0/2412881804 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1450031830 con 0x7f14580cda90 2024-03-23T03:51:34.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.304+0000 7f1457e00640 1 -- 172.21.15.102:0/2412881804 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1450031990 con 0x7f14580cda90 2024-03-23T03:51:34.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.304+0000 7f1457e00640 1 -- 172.21.15.102:0/2412881804 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1450031d20 con 0x7f14580cda90 2024-03-23T03:51:34.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.305+0000 7f1460200640 1 -- 172.21.15.102:0/2412881804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14580cda90 msgr2=0x7f14580d1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:34.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.305+0000 7f1460200640 1 --2- 172.21.15.102:0/2412881804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14580cda90 0x7f14580d1370 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f14500097a0 tx=0x7f145002b8d0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.305+0000 7f1460200640 1 -- 172.21.15.102:0/2412881804 shutdown_connections 2024-03-23T03:51:34.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.305+0000 7f1460200640 1 --2- 172.21.15.102:0/2412881804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14580cda90 0x7f14580d1370 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.305+0000 7f1460200640 1 --2- 172.21.15.102:0/2412881804 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14580cd6b0 0x7f14580b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.305+0000 7f1460200640 1 -- 172.21.15.102:0/2412881804 >> 172.21.15.102:0/2412881804 conn(0x7f14580cf780 msgr2=0x7f14580cfb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:34.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.305+0000 7f1460200640 1 -- 172.21.15.102:0/2412881804 shutdown_connections 2024-03-23T03:51:34.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.305+0000 7f1460200640 1 -- 172.21.15.102:0/2412881804 wait complete. 2024-03-23T03:51:34.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.307+0000 7f1460200640 1 Processor -- start 2024-03-23T03:51:34.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.307+0000 7f1460200640 1 -- start start 2024-03-23T03:51:34.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.308+0000 7f1460200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14580cd6b0 0x7f1458151480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.308+0000 7f1460200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14580cda90 0x7f14581e7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.308+0000 7f1460200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f145815cd90 0x7f14580ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.308+0000 7f1460200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14581519c0 con 0x7f14580cd6b0 2024-03-23T03:51:34.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.308+0000 7f1460200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14580c9c50 con 0x7f145815cd90 2024-03-23T03:51:34.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.308+0000 7f1460200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14581e64c0 con 0x7f14580cda90 2024-03-23T03:51:34.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.308+0000 7f145d800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14580cd6b0 0x7f1458151480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.308+0000 7f145d800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14580cd6b0 0x7f1458151480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48356/0 (socket says 172.21.15.102:48356) 2024-03-23T03:51:34.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.308+0000 7f145d800640 1 -- 172.21.15.102:0/899242399 learned_addr learned my addr 172.21.15.102:0/899242399 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:34.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.309+0000 7f145e200640 1 --2- 172.21.15.102:0/899242399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f145815cd90 0x7f14580ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.309+0000 7f145d800640 1 -- 172.21.15.102:0/899242399 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14580cda90 msgr2=0x7f14581e7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:34.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.309+0000 7f145ce00640 1 --2- 172.21.15.102:0/899242399 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14580cda90 0x7f14581e7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.309+0000 7f145d800640 1 --2- 172.21.15.102:0/899242399 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14580cda90 0x7f14581e7840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.309+0000 7f145d800640 1 -- 172.21.15.102:0/899242399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f145815cd90 msgr2=0x7f14580ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:34.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.309+0000 7f145d800640 1 --2- 172.21.15.102:0/899242399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f145815cd90 0x7f14580ca8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.309+0000 7f145d800640 1 -- 172.21.15.102:0/899242399 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144c0094d0 con 0x7f14580cd6b0 2024-03-23T03:51:34.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.309+0000 7f145e200640 1 --2- 172.21.15.102:0/899242399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f145815cd90 0x7f14580ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:34.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.309+0000 7f145ce00640 1 --2- 172.21.15.102:0/899242399 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14580cda90 0x7f14581e7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:34.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.309+0000 7f145d800640 1 --2- 172.21.15.102:0/899242399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14580cd6b0 0x7f1458151480 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f144c00e9d0 tx=0x7f144c00ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:34.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.309+0000 7f1455600640 1 -- 172.21.15.102:0/899242399 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c00ca60 con 0x7f14580cd6b0 2024-03-23T03:51:34.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.309+0000 7f1455600640 1 -- 172.21.15.102:0/899242399 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144c00cbc0 con 0x7f14580cd6b0 2024-03-23T03:51:34.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.309+0000 7f1460200640 1 -- 172.21.15.102:0/899242399 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1450009400 con 0x7f14580cd6b0 2024-03-23T03:51:34.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.309+0000 7f1455600640 1 -- 172.21.15.102:0/899242399 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c013470 con 0x7f14580cd6b0 2024-03-23T03:51:34.311 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.310+0000 7f1460200640 1 -- 172.21.15.102:0/899242399 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14581e6a80 con 0x7f14580cd6b0 2024-03-23T03:51:34.312 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.312+0000 7f1455600640 1 -- 172.21.15.102:0/899242399 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f144c0135d0 con 0x7f14580cd6b0 2024-03-23T03:51:34.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.313+0000 7f1455600640 1 --2- 172.21.15.102:0/899242399 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f143c06ac00 0x7f143c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.314+0000 7f1455600640 1 -- 172.21.15.102:0/899242399 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f144c0898c0 con 0x7f14580cd6b0 2024-03-23T03:51:34.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.314+0000 7f145ce00640 1 --2- 172.21.15.102:0/899242399 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f143c06ac00 0x7f143c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.314+0000 7f1462f7e740 1 Processor -- start 2024-03-23T03:51:34.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.314+0000 7f1462f7e740 1 -- start start 2024-03-23T03:51:34.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.314+0000 7f145ce00640 1 --2- 172.21.15.102:0/899242399 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f143c06ac00 0x7f143c06d0c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f14581611b0 tx=0x7f1448009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:34.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.315+0000 7f1462f7e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e287ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.315+0000 7f1462f7e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2b50e0 0x557b8e2b7a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.315+0000 7f1462f7e740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2881e0 0x557b8e2b7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.315+0000 7f1462f7e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b8e2632c0 con 0x557b8e2b4d00 2024-03-23T03:51:34.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.315+0000 7f1462f7e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b8e1f88f0 con 0x557b8e2b50e0 2024-03-23T03:51:34.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.315+0000 7f1462f7e740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x557b8e1f6080 con 0x557b8e2881e0 2024-03-23T03:51:34.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.315+0000 7f145d800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e287ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.315+0000 7f145d800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e287ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48362/0 (socket says 172.21.15.102:48362) 2024-03-23T03:51:34.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.315+0000 7f145d800640 1 -- 172.21.15.102:0/2287731488 learned_addr learned my addr 172.21.15.102:0/2287731488 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:34.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.316+0000 7f145e200640 1 --2- 172.21.15.102:0/2287731488 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2881e0 0x557b8e2b7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.316+0000 7f145d800640 1 -- 172.21.15.102:0/2287731488 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2881e0 msgr2=0x557b8e2b7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:34.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.316+0000 7f145ce00640 1 --2- 172.21.15.102:0/2287731488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2b50e0 0x557b8e2b7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.316+0000 7f145d800640 1 --2- 172.21.15.102:0/2287731488 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2881e0 0x557b8e2b7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.316+0000 7f145d800640 1 -- 172.21.15.102:0/2287731488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2b50e0 msgr2=0x557b8e2b7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:34.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.316+0000 7f145d800640 1 --2- 172.21.15.102:0/2287731488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2b50e0 0x557b8e2b7a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.316+0000 7f145d800640 1 -- 172.21.15.102:0/2287731488 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b8e1bca10 con 0x557b8e2b4d00 2024-03-23T03:51:34.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.316+0000 7f145e200640 1 --2- 172.21.15.102:0/2287731488 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2881e0 0x557b8e2b7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:34.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.316+0000 7f145ce00640 1 --2- 172.21.15.102:0/2287731488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2b50e0 0x557b8e2b7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:34.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.316+0000 7f145d800640 1 --2- 172.21.15.102:0/2287731488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e287ca0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f144c0603c0 tx=0x7f144c058cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5a52115e1e11b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:34.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.316+0000 7f1446000640 1 -- 172.21.15.102:0/2287731488 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c05c120 con 0x557b8e2b4d00 2024-03-23T03:51:34.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.316+0000 7f1446000640 1 -- 172.21.15.102:0/2287731488 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144c0608f0 con 0x557b8e2b4d00 2024-03-23T03:51:34.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.316+0000 7f1446000640 1 -- 172.21.15.102:0/2287731488 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c054c10 con 0x557b8e2b4d00 2024-03-23T03:51:34.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.317+0000 7f1462f7e740 1 -- 172.21.15.102:0/2287731488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 msgr2=0x557b8e287ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:34.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.317+0000 7f1462f7e740 1 --2- 172.21.15.102:0/2287731488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e287ca0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f144c0603c0 tx=0x7f144c058cc0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.317+0000 7f1462f7e740 1 -- 172.21.15.102:0/2287731488 shutdown_connections 2024-03-23T03:51:34.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.317+0000 7f1462f7e740 1 --2- 172.21.15.102:0/2287731488 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2881e0 0x557b8e2b7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.317+0000 7f1462f7e740 1 --2- 172.21.15.102:0/2287731488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2b50e0 0x557b8e2b7a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.317+0000 7f1462f7e740 1 --2- 172.21.15.102:0/2287731488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e287ca0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.317+0000 7f1462f7e740 1 -- 172.21.15.102:0/2287731488 >> 172.21.15.102:0/2287731488 conn(0x557b8e289f10 msgr2=0x557b8e2d2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:34.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.317+0000 7f1462f7e740 1 -- 172.21.15.102:0/2287731488 shutdown_connections 2024-03-23T03:51:34.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.317+0000 7f1462f7e740 1 -- 172.21.15.102:0/2287731488 wait complete. 2024-03-23T03:51:34.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.318+0000 7f1462f7e740 1 -- start start 2024-03-23T03:51:34.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.318+0000 7f1462f7e740 1 Processor -- start 2024-03-23T03:51:34.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.319+0000 7f1462f7e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2b4d00 0x557b8e2d2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.319+0000 7f1462f7e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b50e0 0x557b8e2d2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.319+0000 7f1462f7e740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2a2fb0 0x557b8e2cfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.319+0000 7f1462f7e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b8e1f1970 con 0x557b8e2b50e0 2024-03-23T03:51:34.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.319+0000 7f1462f7e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b8e1930a0 con 0x557b8e2b4d00 2024-03-23T03:51:34.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.319+0000 7f1462f7e740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x557b8e192d40 con 0x557b8e2a2fb0 2024-03-23T03:51:34.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.319+0000 7f1462f7e740 10 client.0 _ll_register_callbacks cb 0x557b8e1eafe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T03:51:34.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.319+0000 7f145ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b50e0 0x557b8e2d2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.320+0000 7f145ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b50e0 0x557b8e2d2d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48366/0 (socket says 172.21.15.102:48366) 2024-03-23T03:51:34.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.320+0000 7f145ce00640 1 -- 172.21.15.102:0/2023255046 learned_addr learned my addr 172.21.15.102:0/2023255046 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:34.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.320+0000 7f145d800640 1 --2- 172.21.15.102:0/2023255046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2b4d00 0x557b8e2d2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.320+0000 7f145ce00640 1 -- 172.21.15.102:0/2023255046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2a2fb0 msgr2=0x557b8e2cfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:34.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.320+0000 7f145ce00640 1 --2- 172.21.15.102:0/2023255046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2a2fb0 0x557b8e2cfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.320+0000 7f145ce00640 1 -- 172.21.15.102:0/2023255046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2b4d00 msgr2=0x557b8e2d2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:34.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.320+0000 7f145ce00640 1 --2- 172.21.15.102:0/2023255046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2b4d00 0x557b8e2d2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.320+0000 7f145ce00640 1 -- 172.21.15.102:0/2023255046 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14581e6a80 con 0x557b8e2b50e0 2024-03-23T03:51:34.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.320+0000 7f145d800640 1 --2- 172.21.15.102:0/2023255046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2b4d00 0x557b8e2d2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:34.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.320+0000 7f145ce00640 1 --2- 172.21.15.102:0/2023255046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b50e0 0x557b8e2d2d80 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f145806b770 tx=0x7f1448001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c11d6f2be28e6e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:34.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.320+0000 7f143aa00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-23T03:51:34.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.320+0000 7f143aa00640 1 -- 172.21.15.102:0/2023255046 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14480127e0 con 0x557b8e2b50e0 2024-03-23T03:51:34.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.321+0000 7f143aa00640 1 -- 172.21.15.102:0/2023255046 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1448012940 con 0x557b8e2b50e0 2024-03-23T03:51:34.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.321+0000 7f1462f7e740 1 -- 172.21.15.102:0/2023255046 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557b8e21bb10 con 0x557b8e2b50e0 2024-03-23T03:51:34.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.321+0000 7f143aa00640 1 -- 172.21.15.102:0/2023255046 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1448014e70 con 0x557b8e2b50e0 2024-03-23T03:51:34.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.321+0000 7f1462f7e740 10 client.7374 fetch_fsmap learned FSMap version 56 2024-03-23T03:51:34.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.321+0000 7f1462f7e740 1 -- 172.21.15.102:0/2023255046 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x557b8e1efff0 con 0x557b8e2b50e0 2024-03-23T03:51:34.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.321+0000 7f143aa00640 1 -- 172.21.15.102:0/2023255046 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557b8e21bb10 con 0x557b8e2b50e0 2024-03-23T03:51:34.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.321+0000 7f143aa00640 1 -- 172.21.15.102:0/2023255046 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 56) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1448012b00 con 0x557b8e2b50e0 2024-03-23T03:51:34.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.321+0000 7f1462f7e740 10 client.7374 fetch_fsmap finished waiting for FSMap version 56 2024-03-23T03:51:34.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.321+0000 7f1462f7e740 10 client.7374 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T03:51:34.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.321+0000 7f1462f7e740 20 client.7374 populate_metadata read hostname 'smithi102' 2024-03-23T03:51:34.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.321+0000 7f1462f7e740 1 --2- 172.21.15.102:0/2023255046 >> [v2:172.21.15.102:6836/883705787,v1:172.21.15.102:6837/883705787] conn(0x557b8e2bb3d0 0x557b8e2a5010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.321+0000 7f1462f7e740 4 client.7374 mds_command: new command op to 7212 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T03:51:34.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.321+0000 7f1462f7e740 1 -- 172.21.15.102:0/2023255046 --> [v2:172.21.15.102:6836/883705787,v1:172.21.15.102:6837/883705787] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557b8dee53c0 con 0x557b8e2bb3d0 2024-03-23T03:51:34.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.322+0000 7f145d800640 1 --2- 172.21.15.102:0/2023255046 >> [v2:172.21.15.102:6836/883705787,v1:172.21.15.102:6837/883705787] conn(0x557b8e2bb3d0 0x557b8e2a5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.322+0000 7f145d800640 1 --2- 172.21.15.102:0/2023255046 >> [v2:172.21.15.102:6836/883705787,v1:172.21.15.102:6837/883705787] conn(0x557b8e2bb3d0 0x557b8e2a5010 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93b80c68618f72c6 server_cookie=63e65f554361bc49 in_seq=0 out_seq=0 2024-03-23T03:51:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.322+0000 7f143aa00640 10 client.7374 ms_handle_connect on v2:172.21.15.102:6836/883705787 2024-03-23T03:51:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.323+0000 7f143aa00640 1 -- 172.21.15.102:0/2023255046 <== mds.0 v2:172.21.15.102:6836/883705787 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17819 (crc 0 0 0) 0x557b8dee53c0 con 0x557b8e2bb3d0 2024-03-23T03:51:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.323+0000 7f143aa00640 10 client.7374 handle_command_reply: tid=0 2024-03-23T03:51:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.323+0000 7f1462f7e740 1 client.7374 shutdown 2024-03-23T03:51:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.323+0000 7f1462f7e740 1 -- 172.21.15.102:0/2023255046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b50e0 msgr2=0x557b8e2d2d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.323+0000 7f1462f7e740 1 --2- 172.21.15.102:0/2023255046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b50e0 0x557b8e2d2d80 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f145806b770 tx=0x7f1448001250 comp rx=0 tx=0).stop 2024-03-23T03:51:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.323+0000 7f1462f7e740 1 -- 172.21.15.102:0/2023255046 shutdown_connections 2024-03-23T03:51:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.323+0000 7f1462f7e740 1 --2- 172.21.15.102:0/2023255046 >> [v2:172.21.15.102:6836/883705787,v1:172.21.15.102:6837/883705787] conn(0x557b8e2bb3d0 0x557b8e2a5010 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.323+0000 7f1462f7e740 1 --2- 172.21.15.102:0/2023255046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2a2fb0 0x557b8e2cfcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.323+0000 7f1462f7e740 1 --2- 172.21.15.102:0/2023255046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b50e0 0x557b8e2d2d80 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.323+0000 7f143aa00640 0 client.7374 ms_handle_reset on v2:172.21.15.102:6836/883705787 2024-03-23T03:51:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.323+0000 7f1462f7e740 1 --2- 172.21.15.102:0/2023255046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2b4d00 0x557b8e2d2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.323+0000 7f1462f7e740 1 -- 172.21.15.102:0/2023255046 >> 172.21.15.102:0/2023255046 conn(0x557b8e2881e0 msgr2=0x557b8e2b7a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:34.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.323+0000 7f1462f7e740 1 -- 172.21.15.102:0/2023255046 shutdown_connections 2024-03-23T03:51:34.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.323+0000 7f1462f7e740 1 -- 172.21.15.102:0/2023255046 wait complete. 2024-03-23T03:51:34.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.324+0000 7f1462f7e740 20 client.7374 trim_cache size 0 max 16384 2024-03-23T03:51:34.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.340+0000 7f1462f7e740 1 Processor -- start 2024-03-23T03:51:34.340 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.340+0000 7f1462f7e740 1 -- start start 2024-03-23T03:51:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.341+0000 7f1462f7e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2a2fb0 0x557b8e2cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.341+0000 7f1462f7e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e28b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.341+0000 7f1462f7e740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2b50e0 0x557b8e28bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.341+0000 7f1462f7e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b8df58f20 con 0x557b8e2b4d00 2024-03-23T03:51:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.341+0000 7f1462f7e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b8e2bc050 con 0x557b8e2a2fb0 2024-03-23T03:51:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.341+0000 7f1462f7e740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x557b8e2cfb80 con 0x557b8e2b50e0 2024-03-23T03:51:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.341+0000 7f145ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e28b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.341+0000 7f145ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e28b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48376/0 (socket says 172.21.15.102:48376) 2024-03-23T03:51:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.341+0000 7f145ce00640 1 -- 172.21.15.102:0/3015810349 learned_addr learned my addr 172.21.15.102:0/3015810349 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.341+0000 7f145ce00640 1 -- 172.21.15.102:0/3015810349 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2b50e0 msgr2=0x557b8e28bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.341+0000 7f145ce00640 1 --2- 172.21.15.102:0/3015810349 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2b50e0 0x557b8e28bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.342 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.341+0000 7f145ce00640 1 -- 172.21.15.102:0/3015810349 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2a2fb0 msgr2=0x557b8e2cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:34.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.341+0000 7f145ce00640 1 --2- 172.21.15.102:0/3015810349 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2a2fb0 0x557b8e2cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.341+0000 7f145ce00640 1 -- 172.21.15.102:0/3015810349 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b8e2257c0 con 0x557b8e2b4d00 2024-03-23T03:51:34.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.342+0000 7f145ce00640 1 --2- 172.21.15.102:0/3015810349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e28b6d0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f1448001290 tx=0x7f144801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db5b397516e397c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:34.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.342+0000 7f1446a00640 1 -- 172.21.15.102:0/3015810349 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448006d50 con 0x557b8e2b4d00 2024-03-23T03:51:34.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.342+0000 7f1446a00640 1 -- 172.21.15.102:0/3015810349 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144801a050 con 0x557b8e2b4d00 2024-03-23T03:51:34.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.342+0000 7f1446a00640 1 -- 172.21.15.102:0/3015810349 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448012800 con 0x557b8e2b4d00 2024-03-23T03:51:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.342+0000 7f1462f7e740 1 -- 172.21.15.102:0/3015810349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 msgr2=0x557b8e28b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.342+0000 7f1462f7e740 1 --2- 172.21.15.102:0/3015810349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e28b6d0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f1448001290 tx=0x7f144801d000 comp rx=0 tx=0).stop 2024-03-23T03:51:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.342+0000 7f1462f7e740 1 -- 172.21.15.102:0/3015810349 shutdown_connections 2024-03-23T03:51:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.342+0000 7f1462f7e740 1 --2- 172.21.15.102:0/3015810349 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2b50e0 0x557b8e28bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.342+0000 7f1462f7e740 1 --2- 172.21.15.102:0/3015810349 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e28b6d0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.342+0000 7f1462f7e740 1 --2- 172.21.15.102:0/3015810349 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2a2fb0 0x557b8e2cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.342+0000 7f1462f7e740 1 -- 172.21.15.102:0/3015810349 >> 172.21.15.102:0/3015810349 conn(0x557b8e2881e0 msgr2=0x557b8e283660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.343+0000 7f1462f7e740 1 -- 172.21.15.102:0/3015810349 shutdown_connections 2024-03-23T03:51:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.343+0000 7f1462f7e740 1 -- 172.21.15.102:0/3015810349 wait complete. 2024-03-23T03:51:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.343+0000 7f1462f7e740 1 -- start start 2024-03-23T03:51:34.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.343+0000 7f1462f7e740 1 Processor -- start 2024-03-23T03:51:34.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.344+0000 7f1462f7e740 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2a2fb0 0x557b8e3380f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.344+0000 7f1462f7e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e2a1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.344+0000 7f1462f7e740 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2b50e0 0x557b8e33c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.344+0000 7f1462f7e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b8e33bd50 con 0x557b8e2b4d00 2024-03-23T03:51:34.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.344+0000 7f1462f7e740 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b8e2a1860 con 0x557b8e2b50e0 2024-03-23T03:51:34.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.344+0000 7f1462f7e740 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x557b8e2d1010 con 0x557b8e2a2fb0 2024-03-23T03:51:34.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.344+0000 7f1462f7e740 10 client.0 _ll_register_callbacks cb 0x557b8e28e060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T03:51:34.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.345+0000 7f145ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e2a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.345+0000 7f145ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e2a1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48390/0 (socket says 172.21.15.102:48390) 2024-03-23T03:51:34.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.345+0000 7f145ce00640 1 -- 172.21.15.102:0/361732104 learned_addr learned my addr 172.21.15.102:0/361732104 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:34.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.345+0000 7f145e200640 1 --2- 172.21.15.102:0/361732104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2b50e0 0x557b8e33c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.345+0000 7f145ce00640 1 -- 172.21.15.102:0/361732104 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2a2fb0 msgr2=0x557b8e3380f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:34.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.345+0000 7f145ce00640 1 --2- 172.21.15.102:0/361732104 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2a2fb0 0x557b8e3380f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.345+0000 7f145ce00640 1 -- 172.21.15.102:0/361732104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2b50e0 msgr2=0x557b8e33c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:34.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.345+0000 7f145ce00640 1 --2- 172.21.15.102:0/361732104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2b50e0 0x557b8e33c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.345+0000 7f145ce00640 1 -- 172.21.15.102:0/361732104 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1448014e70 con 0x557b8e2b4d00 2024-03-23T03:51:34.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.345+0000 7f145ce00640 1 --2- 172.21.15.102:0/361732104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e2a1110 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f1448007000 tx=0x7f1448012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a07eefb9596238d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:34.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.345+0000 7f143aa00640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-23T03:51:34.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.346+0000 7f143aa00640 1 -- 172.21.15.102:0/361732104 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14480129f0 con 0x557b8e2b4d00 2024-03-23T03:51:34.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.346+0000 7f143aa00640 1 -- 172.21.15.102:0/361732104 <== mon.0 v2:172.21.15.102:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1448012b50 con 0x557b8e2b4d00 2024-03-23T03:51:34.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.346+0000 7f1462f7e740 1 -- 172.21.15.102:0/361732104 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f144c0608f0 con 0x557b8e2b4d00 2024-03-23T03:51:34.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.346+0000 7f143aa00640 1 -- 172.21.15.102:0/361732104 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f144801f410 con 0x557b8e2b4d00 2024-03-23T03:51:34.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.346+0000 7f1462f7e740 10 client.7380 fetch_fsmap learned FSMap version 56 2024-03-23T03:51:34.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.346+0000 7f1462f7e740 1 -- 172.21.15.102:0/361732104 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x557b8e1922b0 con 0x557b8e2b4d00 2024-03-23T03:51:34.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.346+0000 7f143aa00640 1 -- 172.21.15.102:0/361732104 <== mon.0 v2:172.21.15.102:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144c0608f0 con 0x557b8e2b4d00 2024-03-23T03:51:34.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.346+0000 7f143aa00640 1 -- 172.21.15.102:0/361732104 <== mon.0 v2:172.21.15.102:3300/0 5 ==== fsmap(e 56) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f144801f5f0 con 0x557b8e2b4d00 2024-03-23T03:51:34.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.346+0000 7f1462f7e740 10 client.7380 fetch_fsmap finished waiting for FSMap version 56 2024-03-23T03:51:34.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.346+0000 7f1462f7e740 10 client.7380 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T03:51:34.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.346+0000 7f1462f7e740 20 client.7380 populate_metadata read hostname 'smithi102' 2024-03-23T03:51:34.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.346+0000 7f1462f7e740 1 --2- 172.21.15.102:0/361732104 >> [v2:172.21.15.102:6836/883705787,v1:172.21.15.102:6837/883705787] conn(0x557b8e33c6b0 0x557b8e28b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.346+0000 7f1462f7e740 4 client.7380 mds_command: new command op to 7212 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-23T03:51:34.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.346+0000 7f1462f7e740 1 -- 172.21.15.102:0/361732104 --> [v2:172.21.15.102:6836/883705787,v1:172.21.15.102:6837/883705787] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557b8def66c0 con 0x557b8e33c6b0 2024-03-23T03:51:34.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.347+0000 7f145d800640 1 --2- 172.21.15.102:0/361732104 >> [v2:172.21.15.102:6836/883705787,v1:172.21.15.102:6837/883705787] conn(0x557b8e33c6b0 0x557b8e28b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.347+0000 7f145d800640 1 --2- 172.21.15.102:0/361732104 >> [v2:172.21.15.102:6836/883705787,v1:172.21.15.102:6837/883705787] conn(0x557b8e33c6b0 0x557b8e28b6d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b3b6a9a86ba109bd server_cookie=9c1ca6fa49bcdf96 in_seq=0 out_seq=0 2024-03-23T03:51:34.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.347+0000 7f143aa00640 10 client.7380 ms_handle_connect on v2:172.21.15.102:6836/883705787 2024-03-23T03:51:34.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.347+0000 7f143aa00640 1 -- 172.21.15.102:0/361732104 <== mds.0 v2:172.21.15.102:6836/883705787 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x557b8def66c0 con 0x557b8e33c6b0 2024-03-23T03:51:34.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.347+0000 7f143aa00640 10 client.7380 handle_command_reply: tid=0 2024-03-23T03:51:34.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.347+0000 7f1462f7e740 1 client.7380 shutdown 2024-03-23T03:51:34.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.348+0000 7f1462f7e740 1 -- 172.21.15.102:0/361732104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 msgr2=0x557b8e2a1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:34.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.348+0000 7f1462f7e740 1 --2- 172.21.15.102:0/361732104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e2a1110 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f1448007000 tx=0x7f1448012000 comp rx=0 tx=0).stop 2024-03-23T03:51:34.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.348+0000 7f1462f7e740 1 -- 172.21.15.102:0/361732104 shutdown_connections 2024-03-23T03:51:34.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.348+0000 7f1462f7e740 1 --2- 172.21.15.102:0/361732104 >> [v2:172.21.15.102:6836/883705787,v1:172.21.15.102:6837/883705787] conn(0x557b8e33c6b0 0x557b8e28b6d0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.348+0000 7f1462f7e740 1 --2- 172.21.15.102:0/361732104 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x557b8e2b50e0 0x557b8e33c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.348+0000 7f1462f7e740 1 --2- 172.21.15.102:0/361732104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557b8e2b4d00 0x557b8e2a1110 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.348+0000 7f1462f7e740 1 --2- 172.21.15.102:0/361732104 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x557b8e2a2fb0 0x557b8e3380f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.348+0000 7f143aa00640 0 client.7380 ms_handle_reset on v2:172.21.15.102:6836/883705787 2024-03-23T03:51:34.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.348+0000 7f1462f7e740 1 -- 172.21.15.102:0/361732104 >> 172.21.15.102:0/361732104 conn(0x557b8e2881e0 msgr2=0x557b8e2d5100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:34.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.348+0000 7f1462f7e740 1 -- 172.21.15.102:0/361732104 shutdown_connections 2024-03-23T03:51:34.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.348+0000 7f1462f7e740 1 -- 172.21.15.102:0/361732104 wait complete. 2024-03-23T03:51:34.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.348+0000 7f1462f7e740 20 client.7380 trim_cache size 0 max 16384 2024-03-23T03:51:34.353 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:51:34.353 INFO:teuthology.orchestra.run.smithi102.stdout: "cluster_fsid": "776b2820-05f2-4e50-8d40-18dd76c5f6a0", 2024-03-23T03:51:34.353 INFO:teuthology.orchestra.run.smithi102.stdout: "whoami": 0, 2024-03-23T03:51:34.353 INFO:teuthology.orchestra.run.smithi102.stdout: "id": 7212, 2024-03-23T03:51:34.353 INFO:teuthology.orchestra.run.smithi102.stdout: "want_state": "up:active", 2024-03-23T03:51:34.353 INFO:teuthology.orchestra.run.smithi102.stdout: "state": "up:active", 2024-03-23T03:51:34.353 INFO:teuthology.orchestra.run.smithi102.stdout: "fs_name": "cephfs", 2024-03-23T03:51:34.353 INFO:teuthology.orchestra.run.smithi102.stdout: "rank_uptime": 12.115061952, 2024-03-23T03:51:34.353 INFO:teuthology.orchestra.run.smithi102.stdout: "mdsmap_epoch": 56, 2024-03-23T03:51:34.353 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch": 105, 2024-03-23T03:51:34.353 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_epoch_barrier": 105, 2024-03-23T03:51:34.353 INFO:teuthology.orchestra.run.smithi102.stdout: "uptime": 17.893384526999998 2024-03-23T03:51:34.354 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:51:34.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.349+0000 7f1460200640 1 -- 172.21.15.102:0/899242399 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f143c06ac00 msgr2=0x7f143c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:34.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.349+0000 7f1460200640 1 --2- 172.21.15.102:0/899242399 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f143c06ac00 0x7f143c06d0c0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f14581611b0 tx=0x7f1448009040 comp rx=0 tx=0).stop 2024-03-23T03:51:34.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.349+0000 7f1460200640 1 -- 172.21.15.102:0/899242399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14580cd6b0 msgr2=0x7f1458151480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:34.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.349+0000 7f1460200640 1 --2- 172.21.15.102:0/899242399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14580cd6b0 0x7f1458151480 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f144c00e9d0 tx=0x7f144c00ee90 comp rx=0 tx=0).stop 2024-03-23T03:51:34.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.349+0000 7f1460200640 1 -- 172.21.15.102:0/899242399 shutdown_connections 2024-03-23T03:51:34.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.349+0000 7f1460200640 1 --2- 172.21.15.102:0/899242399 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f143c06ac00 0x7f143c06d0c0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.349+0000 7f1460200640 1 --2- 172.21.15.102:0/899242399 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f145815cd90 0x7f14580ca8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.349+0000 7f1460200640 1 --2- 172.21.15.102:0/899242399 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14580cda90 0x7f14581e7840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.349+0000 7f1460200640 1 --2- 172.21.15.102:0/899242399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14580cd6b0 0x7f1458151480 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.349+0000 7f1460200640 1 -- 172.21.15.102:0/899242399 >> 172.21.15.102:0/899242399 conn(0x7f14580cf780 msgr2=0x7f14580cfb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:34.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.349+0000 7f1460200640 1 -- 172.21.15.102:0/899242399 shutdown_connections 2024-03-23T03:51:34.355 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.349+0000 7f1460200640 1 -- 172.21.15.102:0/899242399 wait complete. 2024-03-23T03:51:34.366 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:51:34.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.566+0000 7f3b54200640 1 Processor -- start 2024-03-23T03:51:34.567 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.566+0000 7f3b54200640 1 -- start start 2024-03-23T03:51:34.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.567+0000 7f3b54200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3b4c0adbc0 0x7f3b4c0b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.568 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.568+0000 7f3b54200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f3b4c06d390 0x7f3b4c06d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.568+0000 7f3b54200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b4c0b16a0 con 0x7f3b4c0cbab0 2024-03-23T03:51:34.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.568+0000 7f3b54200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b4c06dd40 con 0x7f3b4c0adbc0 2024-03-23T03:51:34.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.568+0000 7f3b54200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b4c0b4700 con 0x7f3b4c06d390 2024-03-23T03:51:34.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.568+0000 7f3b51800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3b4c0adbc0 0x7f3b4c0b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.568+0000 7f3b51800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f3b4c0adbc0 0x7f3b4c0b41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48404/0 (socket says 172.21.15.102:48404) 2024-03-23T03:51:34.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.568+0000 7f3b51800640 1 -- 172.21.15.102:0/753727143 learned_addr learned my addr 172.21.15.102:0/753727143 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:34.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.568+0000 7f3b51800640 1 -- 172.21.15.102:0/753727143 >> v2:172.21.15.173:3300/0 conn(0x7f3b4c06d390 msgr2=0x7f3b4c06d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:34.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.568+0000 7f3b52200640 1 --2- 172.21.15.102:0/753727143 >> v2:172.21.15.173:3300/0 conn(0x7f3b4c06d390 0x7f3b4c06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.568+0000 7f3b51800640 1 --2- 172.21.15.102:0/753727143 >> v2:172.21.15.173:3300/0 conn(0x7f3b4c06d390 0x7f3b4c06d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.568+0000 7f3b51800640 1 -- 172.21.15.102:0/753727143 >> v1:172.21.15.173:6789/0 conn(0x7f3b4c0cbab0 legacy=0x7f3b4c0b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:34.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.568+0000 7f3b51800640 1 -- 172.21.15.102:0/753727143 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4c0cbfb0 con 0x7f3b4c0adbc0 2024-03-23T03:51:34.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.569+0000 7f3b51800640 1 --2- 172.21.15.102:0/753727143 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b4c0adbc0 0x7f3b4c0b41c0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c0ccf20 tx=0x7f3b4002e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23a694c88c6638 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:34.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.570+0000 7f3b4be00640 1 -- 172.21.15.102:0/753727143 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4002ee00 con 0x7f3b4c0adbc0 2024-03-23T03:51:34.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.570+0000 7f3b4be00640 1 -- 172.21.15.102:0/753727143 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b40036440 con 0x7f3b4c0adbc0 2024-03-23T03:51:34.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.570+0000 7f3b4be00640 1 -- 172.21.15.102:0/753727143 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b400367d0 con 0x7f3b4c0adbc0 2024-03-23T03:51:34.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.570+0000 7f3b54200640 1 -- 172.21.15.102:0/753727143 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b4c0adbc0 msgr2=0x7f3b4c0b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:34.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.570+0000 7f3b54200640 1 --2- 172.21.15.102:0/753727143 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b4c0adbc0 0x7f3b4c0b41c0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c0ccf20 tx=0x7f3b4002e050 comp rx=0 tx=0).stop 2024-03-23T03:51:34.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.571+0000 7f3b54200640 1 -- 172.21.15.102:0/753727143 shutdown_connections 2024-03-23T03:51:34.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.571+0000 7f3b54200640 1 --2- 172.21.15.102:0/753727143 >> v2:172.21.15.173:3300/0 conn(0x7f3b4c06d390 0x7f3b4c06d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.571+0000 7f3b54200640 1 --2- 172.21.15.102:0/753727143 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b4c0adbc0 0x7f3b4c0b41c0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.571+0000 7f3b54200640 1 -- 172.21.15.102:0/753727143 >> 172.21.15.102:0/753727143 conn(0x7f3b4c06a3d0 msgr2=0x7f3b4c0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:34.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.571+0000 7f3b54200640 1 -- 172.21.15.102:0/753727143 shutdown_connections 2024-03-23T03:51:34.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.571+0000 7f3b54200640 1 -- 172.21.15.102:0/753727143 wait complete. 2024-03-23T03:51:34.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.572+0000 7f3b54200640 1 Processor -- start 2024-03-23T03:51:34.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.573+0000 7f3b54200640 1 -- start start 2024-03-23T03:51:34.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.573+0000 7f3b54200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b4c06d390 0x7f3b4c13b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.574+0000 7f3b54200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b4c0adbc0 0x7f3b4c1590f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.574+0000 7f3b54200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b4c0cbab0 0x7f3b4c0b2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.574+0000 7f3b54200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c13c220 con 0x7f3b4c0cbab0 2024-03-23T03:51:34.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.574+0000 7f3b54200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c13c500 con 0x7f3b4c0adbc0 2024-03-23T03:51:34.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.574+0000 7f3b54200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b4c139080 con 0x7f3b4c06d390 2024-03-23T03:51:34.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.574+0000 7f3b52200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b4c0cbab0 0x7f3b4c0b2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.574+0000 7f3b52200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b4c0cbab0 0x7f3b4c0b2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48414/0 (socket says 172.21.15.102:48414) 2024-03-23T03:51:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.574+0000 7f3b52200640 1 -- 172.21.15.102:0/3030109410 learned_addr learned my addr 172.21.15.102:0/3030109410 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.574+0000 7f3b52200640 1 -- 172.21.15.102:0/3030109410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b4c06d390 msgr2=0x7f3b4c13b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.574+0000 7f3b51800640 1 --2- 172.21.15.102:0/3030109410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b4c06d390 0x7f3b4c13b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.574+0000 7f3b50e00640 1 --2- 172.21.15.102:0/3030109410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b4c0adbc0 0x7f3b4c1590f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.574+0000 7f3b52200640 1 --2- 172.21.15.102:0/3030109410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b4c06d390 0x7f3b4c13b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.574+0000 7f3b52200640 1 -- 172.21.15.102:0/3030109410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b4c0adbc0 msgr2=0x7f3b4c1590f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.574+0000 7f3b52200640 1 --2- 172.21.15.102:0/3030109410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b4c0adbc0 0x7f3b4c1590f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.574+0000 7f3b52200640 1 -- 172.21.15.102:0/3030109410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b4000b770 con 0x7f3b4c0cbab0 2024-03-23T03:51:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.574+0000 7f3b50e00640 1 --2- 172.21.15.102:0/3030109410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b4c0adbc0 0x7f3b4c1590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:34.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.575+0000 7f3b52200640 1 --2- 172.21.15.102:0/3030109410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b4c0cbab0 0x7f3b4c0b2aa0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400cc30 tx=0x7f3b4400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:34.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.575+0000 7f3b49600640 1 -- 172.21.15.102:0/3030109410 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b4400ce40 con 0x7f3b4c0cbab0 2024-03-23T03:51:34.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.575+0000 7f3b49600640 1 -- 172.21.15.102:0/3030109410 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b44003990 con 0x7f3b4c0cbab0 2024-03-23T03:51:34.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.575+0000 7f3b54200640 1 -- 172.21.15.102:0/3030109410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b4c139360 con 0x7f3b4c0cbab0 2024-03-23T03:51:34.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.575+0000 7f3b49600640 1 -- 172.21.15.102:0/3030109410 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b44003d20 con 0x7f3b4c0cbab0 2024-03-23T03:51:34.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.575+0000 7f3b54200640 1 -- 172.21.15.102:0/3030109410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b4c159630 con 0x7f3b4c0cbab0 2024-03-23T03:51:34.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.576+0000 7f3b54200640 1 -- 172.21.15.102:0/3030109410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b4c06ac20 con 0x7f3b4c0cbab0 2024-03-23T03:51:34.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.577+0000 7f3b49600640 1 -- 172.21.15.102:0/3030109410 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3b44003e80 con 0x7f3b4c0cbab0 2024-03-23T03:51:34.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.578+0000 7f3b49600640 1 --2- 172.21.15.102:0/3030109410 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b2c06ac30 0x7f3b2c06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:34.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.578+0000 7f3b49600640 1 -- 172.21.15.102:0/3030109410 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3b44089910 con 0x7f3b4c0cbab0 2024-03-23T03:51:34.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.578+0000 7f3b51800640 1 --2- 172.21.15.102:0/3030109410 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b2c06ac30 0x7f3b2c06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:34.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.579+0000 7f3b51800640 1 --2- 172.21.15.102:0/3030109410 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b2c06ac30 0x7f3b2c06d0f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3b40038000 tx=0x7f3b40032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:34.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.582+0000 7f3b49600640 1 -- 172.21.15.102:0/3030109410 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3b44054880 con 0x7f3b4c0cbab0 2024-03-23T03:51:34.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.819+0000 7f3b54200640 1 -- 172.21.15.102:0/3030109410 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3b4c139550 con 0x7f3b4c0cbab0 2024-03-23T03:51:34.821 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.820+0000 7f3b49600640 1 -- 172.21.15.102:0/3030109410 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v56) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3b4401d030 con 0x7f3b4c0cbab0 2024-03-23T03:51:34.821 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T03:51:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.823+0000 7f3b54200640 1 -- 172.21.15.102:0/3030109410 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b2c06ac30 msgr2=0x7f3b2c06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.823+0000 7f3b54200640 1 --2- 172.21.15.102:0/3030109410 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b2c06ac30 0x7f3b2c06d0f0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f3b40038000 tx=0x7f3b40032000 comp rx=0 tx=0).stop 2024-03-23T03:51:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.823+0000 7f3b54200640 1 -- 172.21.15.102:0/3030109410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b4c0cbab0 msgr2=0x7f3b4c0b2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.823+0000 7f3b54200640 1 --2- 172.21.15.102:0/3030109410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b4c0cbab0 0x7f3b4c0b2aa0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f3b4400cc30 tx=0x7f3b4400a5c0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.823+0000 7f3b54200640 1 -- 172.21.15.102:0/3030109410 shutdown_connections 2024-03-23T03:51:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.823+0000 7f3b54200640 1 --2- 172.21.15.102:0/3030109410 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3b2c06ac30 0x7f3b2c06d0f0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.823+0000 7f3b54200640 1 --2- 172.21.15.102:0/3030109410 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b4c0cbab0 0x7f3b4c0b2aa0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.823+0000 7f3b54200640 1 --2- 172.21.15.102:0/3030109410 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3b4c0adbc0 0x7f3b4c1590f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.823+0000 7f3b54200640 1 --2- 172.21.15.102:0/3030109410 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3b4c06d390 0x7f3b4c13b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.823+0000 7f3b54200640 1 -- 172.21.15.102:0/3030109410 >> 172.21.15.102:0/3030109410 conn(0x7f3b4c06a3d0 msgr2=0x7f3b4c0b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.823+0000 7f3b54200640 1 -- 172.21.15.102:0/3030109410 shutdown_connections 2024-03-23T03:51:34.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:34.823+0000 7f3b54200640 1 -- 172.21.15.102:0/3030109410 wait complete. 2024-03-23T03:51:34.842 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-23T03:51:34.842 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:51:34.842 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi102.front.sepia.ceph.com 2024-03-23T03:51:34.842 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T03:51:34.842 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T03:51:34.842 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-23T03:51:34.842 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-23T03:51:34.842 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:51:34.842 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:51:34.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:34.843 DEBUG:teuthology.orchestra.run.smithi102:> ip addr 2024-03-23T03:51:34.893 INFO:teuthology.orchestra.run.smithi102.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:51:34.893 INFO:teuthology.orchestra.run.smithi102.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:51:34.893 INFO:teuthology.orchestra.run.smithi102.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:51:34.893 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:51:34.893 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 ::1/128 scope host 2024-03-23T03:51:34.893 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:51:34.893 INFO:teuthology.orchestra.run.smithi102.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:51:34.893 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dc brd ff:ff:ff:ff:ff:ff 2024-03-23T03:51:34.893 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f0 2024-03-23T03:51:34.893 INFO:teuthology.orchestra.run.smithi102.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:51:34.894 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:d8:d9:dd brd ff:ff:ff:ff:ff:ff 2024-03-23T03:51:34.894 INFO:teuthology.orchestra.run.smithi102.stdout: altname enp7s0f1 2024-03-23T03:51:34.894 INFO:teuthology.orchestra.run.smithi102.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:51:34.894 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:68 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:51:34.894 INFO:teuthology.orchestra.run.smithi102.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:51:34.894 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether 0c:c4:7a:8f:d7:69 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:51:34.894 INFO:teuthology.orchestra.run.smithi102.stdout: inet 172.21.15.102/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:51:34.894 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft 42376sec preferred_lft 42376sec 2024-03-23T03:51:34.894 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::ec4:7aff:fe8f:d769/64 scope link 2024-03-23T03:51:34.894 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:51:34.894 INFO:teuthology.orchestra.run.smithi102.stdout:8: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:51:34.894 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether aa:17:e3:39:b0:56 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:51:34.894 INFO:teuthology.orchestra.run.smithi102.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T03:51:34.894 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:51:34.895 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::5cce:1fff:fe41:5a05/64 scope link 2024-03-23T03:51:34.895 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:51:34.895 INFO:teuthology.orchestra.run.smithi102.stdout:9: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:51:34.895 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether aa:17:e3:39:b0:56 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T03:51:34.895 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::a817:e3ff:fe39:b056/64 scope link 2024-03-23T03:51:34.895 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:51:34.895 INFO:teuthology.orchestra.run.smithi102.stdout:10: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:51:34.895 INFO:teuthology.orchestra.run.smithi102.stdout: link/ether ce:9b:95:fb:de:c5 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T03:51:34.895 INFO:teuthology.orchestra.run.smithi102.stdout: inet6 fe80::cc9b:95ff:fefb:dec5/64 scope link 2024-03-23T03:51:34.895 INFO:teuthology.orchestra.run.smithi102.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:51:34.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:34.896 DEBUG:teuthology.orchestra.run.smithi102:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:51:34.978 INFO:teuthology.orchestra.run.smithi102.stdout:1 2024-03-23T03:51:34.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:34.982 DEBUG:teuthology.orchestra.run.smithi102:> route 2024-03-23T03:51:35.029 INFO:teuthology.orchestra.run.smithi102.stdout:Kernel IP routing table 2024-03-23T03:51:35.029 INFO:teuthology.orchestra.run.smithi102.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:51:35.029 INFO:teuthology.orchestra.run.smithi102.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:51:35.029 INFO:teuthology.orchestra.run.smithi102.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:51:35.030 INFO:teuthology.orchestra.run.smithi102.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:51:35.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:35.030 DEBUG:teuthology.orchestra.run.smithi102:> (cd / && exec bash -c ' 2024-03-23T03:51:35.030 DEBUG:teuthology.orchestra.run.smithi102:> set -e 2024-03-23T03:51:35.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:51:35.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:51:35.030 DEBUG:teuthology.orchestra.run.smithi102:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:51:35.030 DEBUG:teuthology.orchestra.run.smithi102:> ') 2024-03-23T03:51:35.214 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:35.214 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-23T03:51:35.262 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T03:51:35.263 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T03:51:35.263 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:51:35.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:35.263 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link set brx.0 up 2024-03-23T03:51:35.349 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:51:35.349 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T03:51:35.400 INFO:teuthology.orchestra.run.smithi102.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T03:51:35.400 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:51:35.401 DEBUG:teuthology.orchestra.run.smithi102:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T03:51:35.452 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-23T03:51:35.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:35.452 DEBUG:teuthology.orchestra.run.smithi102:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:51:35.590 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:51:35.590 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:51:35.590 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:51:35.611 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:51:35.611 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:51:35.611 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:51:35.611 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:51:35.611 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:51:35.637 INFO:teuthology.orchestra.run.smithi102.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-23T03:51:35.637 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-23T03:51:35.637 INFO:teuthology.orchestra.run.smithi102.stdout:invalid new device string format 2024-03-23T03:51:35.637 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:51:35.637 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:51:35.637 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790" 2024-03-23T03:51:35.638 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/ 2024-03-23T03:51:35.638 INFO:teuthology.orchestra.run.smithi102.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=776b2820-05f2-4e50-8d40-18dd76c5f6a0" will pass to kernel 2024-03-23T03:51:35.640 INFO:teuthology.orchestra.run.smithi102.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-23T03:51:35.640 INFO:teuthology.orchestra.run.smithi102.stdout: You just mounted a file system that supports labels which does not 2024-03-23T03:51:35.640 INFO:teuthology.orchestra.run.smithi102.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T03:51:35.640 INFO:teuthology.orchestra.run.smithi102.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T03:51:35.640 INFO:teuthology.orchestra.run.smithi102.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T03:51:35.644 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T03:51:35.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:35.644 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T03:51:35.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:35.722 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:51:35.785 INFO:teuthology.orchestra.run.smithi102.stdout:client5806 2024-03-23T03:51:35.785 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:51:35.785 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:35.832 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T03:51:36.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.083+0000 7f33c6e00640 1 Processor -- start 2024-03-23T03:51:36.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.084+0000 7f33c6e00640 1 -- start start 2024-03-23T03:51:36.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.085+0000 7f33c6e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f33c015cda0 0x7f33c00cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:36.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.085+0000 7f33c6e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f33c00b5a30 0x7f33c00b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:36.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.085+0000 7f33c6e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c00d1830 con 0x7f33c00b4350 2024-03-23T03:51:36.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.085+0000 7f33c6e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c00b1f20 con 0x7f33c015cda0 2024-03-23T03:51:36.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.085+0000 7f33c6e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c00d0c20 con 0x7f33c00b5a30 2024-03-23T03:51:36.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.085+0000 7f33bfe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f33c015cda0 0x7f33c00cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:36.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.085+0000 7f33bfe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f33c015cda0 0x7f33c00cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48428/0 (socket says 172.21.15.102:48428) 2024-03-23T03:51:36.086 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.086+0000 7f33bfe00640 1 -- 172.21.15.102:0/2228801896 learned_addr learned my addr 172.21.15.102:0/2228801896 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:36.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.086+0000 7f33bf400640 1 --2- 172.21.15.102:0/2228801896 >> v2:172.21.15.173:3300/0 conn(0x7f33c00b5a30 0x7f33c00b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:36.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.086+0000 7f33bfe00640 1 -- 172.21.15.102:0/2228801896 >> v2:172.21.15.173:3300/0 conn(0x7f33c00b5a30 msgr2=0x7f33c00b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:36.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.086+0000 7f33bfe00640 1 --2- 172.21.15.102:0/2228801896 >> v2:172.21.15.173:3300/0 conn(0x7f33c00b5a30 0x7f33c00b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:36.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.086+0000 7f33bfe00640 1 -- 172.21.15.102:0/2228801896 >> v1:172.21.15.173:6789/0 conn(0x7f33c00b4350 legacy=0x7f33c00d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:36.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.086+0000 7f33bfe00640 1 -- 172.21.15.102:0/2228801896 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c00d0f00 con 0x7f33c015cda0 2024-03-23T03:51:36.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.086+0000 7f33bf400640 1 --2- 172.21.15.102:0/2228801896 >> v2:172.21.15.173:3300/0 conn(0x7f33c00b5a30 0x7f33c00b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:36.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.087+0000 7f33bfe00640 1 --2- 172.21.15.102:0/2228801896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c015cda0 0x7f33c00cdb40 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f33c00cf010 tx=0x7f33b002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe188b1b827675d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:36.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.087+0000 7f33bea00640 1 -- 172.21.15.102:0/2228801896 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b0031870 con 0x7f33c015cda0 2024-03-23T03:51:36.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.087+0000 7f33bea00640 1 -- 172.21.15.102:0/2228801896 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b00319d0 con 0x7f33c015cda0 2024-03-23T03:51:36.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.087+0000 7f33bea00640 1 -- 172.21.15.102:0/2228801896 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b0031d60 con 0x7f33c015cda0 2024-03-23T03:51:36.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.088+0000 7f33c6e00640 1 -- 172.21.15.102:0/2228801896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c015cda0 msgr2=0x7f33c00cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:36.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.088+0000 7f33c6e00640 1 --2- 172.21.15.102:0/2228801896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c015cda0 0x7f33c00cdb40 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f33c00cf010 tx=0x7f33b002b910 comp rx=0 tx=0).stop 2024-03-23T03:51:36.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.088+0000 7f33c6e00640 1 -- 172.21.15.102:0/2228801896 shutdown_connections 2024-03-23T03:51:36.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.088+0000 7f33c6e00640 1 --2- 172.21.15.102:0/2228801896 >> v2:172.21.15.173:3300/0 conn(0x7f33c00b5a30 0x7f33c00b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:36.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.088+0000 7f33c6e00640 1 --2- 172.21.15.102:0/2228801896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c015cda0 0x7f33c00cdb40 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:36.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.088+0000 7f33c6e00640 1 -- 172.21.15.102:0/2228801896 >> 172.21.15.102:0/2228801896 conn(0x7f33c00cf350 msgr2=0x7f33c00cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:36.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.088+0000 7f33c6e00640 1 -- 172.21.15.102:0/2228801896 shutdown_connections 2024-03-23T03:51:36.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.088+0000 7f33c6e00640 1 -- 172.21.15.102:0/2228801896 wait complete. 2024-03-23T03:51:36.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.090+0000 7f33c6e00640 1 Processor -- start 2024-03-23T03:51:36.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.091+0000 7f33c6e00640 1 -- start start 2024-03-23T03:51:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.092+0000 7f33c6e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33c00b4350 0x7f33c01f19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.092+0000 7f33c6e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c00b5a30 0x7f33c00d2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.092+0000 7f33c6e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f33c015cda0 0x7f33c015f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.092+0000 7f33c6e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c00c7f90 con 0x7f33c00b5a30 2024-03-23T03:51:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.092+0000 7f33c6e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c00c8270 con 0x7f33c00b4350 2024-03-23T03:51:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.092+0000 7f33c6e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c00c8550 con 0x7f33c015cda0 2024-03-23T03:51:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.092+0000 7f33bf400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c00b5a30 0x7f33c00d2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.092+0000 7f33bf400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c00b5a30 0x7f33c00d2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48436/0 (socket says 172.21.15.102:48436) 2024-03-23T03:51:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.092+0000 7f33bf400640 1 -- 172.21.15.102:0/338242491 learned_addr learned my addr 172.21.15.102:0/338242491 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:36.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.092+0000 7f33bfe00640 1 --2- 172.21.15.102:0/338242491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33c00b4350 0x7f33c01f19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.092+0000 7f33bf400640 1 -- 172.21.15.102:0/338242491 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f33c015cda0 msgr2=0x7f33c015f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.092+0000 7f33c4e00640 1 --2- 172.21.15.102:0/338242491 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f33c015cda0 0x7f33c015f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.093+0000 7f33bf400640 1 --2- 172.21.15.102:0/338242491 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f33c015cda0 0x7f33c015f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.093+0000 7f33bf400640 1 -- 172.21.15.102:0/338242491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33c00b4350 msgr2=0x7f33c01f19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.093+0000 7f33bf400640 1 --2- 172.21.15.102:0/338242491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33c00b4350 0x7f33c01f19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.093+0000 7f33bf400640 1 -- 172.21.15.102:0/338242491 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b00094d0 con 0x7f33c00b5a30 2024-03-23T03:51:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.093+0000 7f33bfe00640 1 --2- 172.21.15.102:0/338242491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33c00b4350 0x7f33c01f19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.093+0000 7f33bf400640 1 --2- 172.21.15.102:0/338242491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c00b5a30 0x7f33c00d2f60 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f33a800c7f0 tx=0x7f33a800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.093+0000 7f33b7e00640 1 -- 172.21.15.102:0/338242491 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a8007a30 con 0x7f33c00b5a30 2024-03-23T03:51:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.093+0000 7f33b7e00640 1 -- 172.21.15.102:0/338242491 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33a8007b90 con 0x7f33c00b5a30 2024-03-23T03:51:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.093+0000 7f33b7e00640 1 -- 172.21.15.102:0/338242491 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33a800ce40 con 0x7f33c00b5a30 2024-03-23T03:51:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.093+0000 7f33c6e00640 1 -- 172.21.15.102:0/338242491 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c00c8830 con 0x7f33c00b5a30 2024-03-23T03:51:36.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.093+0000 7f33c6e00640 1 -- 172.21.15.102:0/338242491 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c01f1ee0 con 0x7f33c00b5a30 2024-03-23T03:51:36.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.095+0000 7f33c6e00640 1 -- 172.21.15.102:0/338242491 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c00c8d50 con 0x7f33c00b5a30 2024-03-23T03:51:36.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.096+0000 7f33b7e00640 1 -- 172.21.15.102:0/338242491 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f33a80113f0 con 0x7f33c00b5a30 2024-03-23T03:51:36.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.096+0000 7f33b7e00640 1 --2- 172.21.15.102:0/338242491 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f339c06ac00 0x7f339c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:36.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.097+0000 7f33b7e00640 1 -- 172.21.15.102:0/338242491 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f33a8089690 con 0x7f33c00b5a30 2024-03-23T03:51:36.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.097+0000 7f33bfe00640 1 --2- 172.21.15.102:0/338242491 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f339c06ac00 0x7f339c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:36.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.097+0000 7f33bfe00640 1 --2- 172.21.15.102:0/338242491 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f339c06ac00 0x7f339c06d0c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f33b0036000 tx=0x7f33b003c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:36.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.101+0000 7f33b7e00640 1 -- 172.21.15.102:0/338242491 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f33c00c8d50 con 0x7f33c00b5a30 2024-03-23T03:51:36.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.334+0000 7f33c6e00640 1 -- 172.21.15.102:0/338242491 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f33c00c8d50 con 0x7f33c00b5a30 2024-03-23T03:51:36.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.335+0000 7f33b7e00640 1 -- 172.21.15.102:0/338242491 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v56) v1 ==== 53+0+73 (secure 0 0 0) 0x7f33c00c8d50 con 0x7f33c00b5a30 2024-03-23T03:51:36.335 INFO:teuthology.orchestra.run.smithi102.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T03:51:36.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.337+0000 7f33c6e00640 1 -- 172.21.15.102:0/338242491 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f339c06ac00 msgr2=0x7f339c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:36.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.337+0000 7f33c6e00640 1 --2- 172.21.15.102:0/338242491 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f339c06ac00 0x7f339c06d0c0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f33b0036000 tx=0x7f33b003c040 comp rx=0 tx=0).stop 2024-03-23T03:51:36.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.337+0000 7f33c6e00640 1 -- 172.21.15.102:0/338242491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c00b5a30 msgr2=0x7f33c00d2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:36.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.337+0000 7f33c6e00640 1 --2- 172.21.15.102:0/338242491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c00b5a30 0x7f33c00d2f60 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f33a800c7f0 tx=0x7f33a800cbf0 comp rx=0 tx=0).stop 2024-03-23T03:51:36.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.337+0000 7f33c6e00640 1 -- 172.21.15.102:0/338242491 shutdown_connections 2024-03-23T03:51:36.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.337+0000 7f33c6e00640 1 --2- 172.21.15.102:0/338242491 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f339c06ac00 0x7f339c06d0c0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:36.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.337+0000 7f33c6e00640 1 --2- 172.21.15.102:0/338242491 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f33c015cda0 0x7f33c015f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:36.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.337+0000 7f33c6e00640 1 --2- 172.21.15.102:0/338242491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f33c00b5a30 0x7f33c00d2f60 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:36.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.337+0000 7f33c6e00640 1 --2- 172.21.15.102:0/338242491 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33c00b4350 0x7f33c01f19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:36.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.337+0000 7f33c6e00640 1 -- 172.21.15.102:0/338242491 >> 172.21.15.102:0/338242491 conn(0x7f33c00cf350 msgr2=0x7f33c00cb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:36.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.338+0000 7f33c6e00640 1 -- 172.21.15.102:0/338242491 shutdown_connections 2024-03-23T03:51:36.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:36.338+0000 7f33c6e00640 1 -- 172.21.15.102:0/338242491 wait complete. 2024-03-23T03:51:36.356 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-23T03:51:36.356 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T03:51:36.356 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi173.front.sepia.ceph.com 2024-03-23T03:51:36.356 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-23T03:51:36.357 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-23T03:51:36.357 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-23T03:51:36.357 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2024-03-23T03:51:36.357 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T03:51:36.357 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:51:36.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:36.357 DEBUG:teuthology.orchestra.run.smithi173:> ip addr 2024-03-23T03:51:36.415 INFO:teuthology.orchestra.run.smithi173.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T03:51:36.415 INFO:teuthology.orchestra.run.smithi173.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T03:51:36.415 INFO:teuthology.orchestra.run.smithi173.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T03:51:36.415 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:51:36.415 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 ::1/128 scope host 2024-03-23T03:51:36.415 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:51:36.415 INFO:teuthology.orchestra.run.smithi173.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:51:36.415 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6a brd ff:ff:ff:ff:ff:ff 2024-03-23T03:51:36.415 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f0 2024-03-23T03:51:36.416 INFO:teuthology.orchestra.run.smithi173.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:51:36.416 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:d9:38:6b brd ff:ff:ff:ff:ff:ff 2024-03-23T03:51:36.416 INFO:teuthology.orchestra.run.smithi173.stdout: altname enp7s0f1 2024-03-23T03:51:36.416 INFO:teuthology.orchestra.run.smithi173.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T03:51:36.416 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:58 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:51:36.416 INFO:teuthology.orchestra.run.smithi173.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T03:51:36.416 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 0c:c4:7a:88:72:59 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:51:36.416 INFO:teuthology.orchestra.run.smithi173.stdout: inet 172.21.15.173/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T03:51:36.416 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft 42345sec preferred_lft 42345sec 2024-03-23T03:51:36.416 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::ec4:7aff:fe88:7259/64 scope link 2024-03-23T03:51:36.416 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:51:36.416 INFO:teuthology.orchestra.run.smithi173.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T03:51:36.416 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 52:69:f2:bf:6d:02 brd ff:ff:ff:ff:ff:ff 2024-03-23T03:51:36.416 INFO:teuthology.orchestra.run.smithi173.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T03:51:36.417 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:51:36.417 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::6894:e9ff:fe5d:1492/64 scope link 2024-03-23T03:51:36.417 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:51:36.417 INFO:teuthology.orchestra.run.smithi173.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:51:36.417 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether a2:27:ac:42:68:ce brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T03:51:36.417 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::a027:acff:fe42:68ce/64 scope link 2024-03-23T03:51:36.417 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:51:36.417 INFO:teuthology.orchestra.run.smithi173.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T03:51:36.417 INFO:teuthology.orchestra.run.smithi173.stdout: link/ether 52:69:f2:bf:6d:02 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-23T03:51:36.417 INFO:teuthology.orchestra.run.smithi173.stdout: inet6 fe80::5069:f2ff:febf:6d02/64 scope link 2024-03-23T03:51:36.417 INFO:teuthology.orchestra.run.smithi173.stdout: valid_lft forever preferred_lft forever 2024-03-23T03:51:36.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:36.418 DEBUG:teuthology.orchestra.run.smithi173:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T03:51:36.506 INFO:teuthology.orchestra.run.smithi173.stdout:1 2024-03-23T03:51:36.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:36.510 DEBUG:teuthology.orchestra.run.smithi173:> route 2024-03-23T03:51:36.564 INFO:teuthology.orchestra.run.smithi173.stdout:Kernel IP routing table 2024-03-23T03:51:36.564 INFO:teuthology.orchestra.run.smithi173.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T03:51:36.564 INFO:teuthology.orchestra.run.smithi173.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T03:51:36.564 INFO:teuthology.orchestra.run.smithi173.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T03:51:36.564 INFO:teuthology.orchestra.run.smithi173.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T03:51:36.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:36.565 DEBUG:teuthology.orchestra.run.smithi173:> (cd / && exec bash -c ' 2024-03-23T03:51:36.565 DEBUG:teuthology.orchestra.run.smithi173:> set -e 2024-03-23T03:51:36.565 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T03:51:36.565 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T03:51:36.565 DEBUG:teuthology.orchestra.run.smithi173:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T03:51:36.565 DEBUG:teuthology.orchestra.run.smithi173:> ') 2024-03-23T03:51:36.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:36.749 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list 2024-03-23T03:51:36.799 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-23T03:51:36.799 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-23T03:51:36.800 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:51:36.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:36.800 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link set brx.0 up 2024-03-23T03:51:36.877 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:51:36.877 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-23T03:51:36.926 INFO:teuthology.orchestra.run.smithi173.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-23T03:51:36.926 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T03:51:36.926 DEBUG:teuthology.orchestra.run.smithi173:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-23T03:51:36.973 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-23T03:51:36.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:36.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:51:37.105 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:51:37.105 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:51:37.105 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:51:37.127 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:51:37.127 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:51:37.127 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:51:37.127 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:51:37.127 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:51:37.153 INFO:teuthology.orchestra.run.smithi173.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-23T03:51:37.153 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-23T03:51:37.153 INFO:teuthology.orchestra.run.smithi173.stdout:invalid new device string format 2024-03-23T03:51:37.153 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:3300,172.21.15.173:3300,172.21.15.173:3301" 2024-03-23T03:51:37.153 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-23T03:51:37.153 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: resolved to: "172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790" 2024-03-23T03:51:37.153 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.102:6789,172.21.15.173:6789,172.21.15.173:6790:/ 2024-03-23T03:51:37.153 INFO:teuthology.orchestra.run.smithi173.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=776b2820-05f2-4e50-8d40-18dd76c5f6a0" will pass to kernel 2024-03-23T03:51:37.155 INFO:teuthology.orchestra.run.smithi173.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-23T03:51:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: You just mounted a file system that supports labels which does not 2024-03-23T03:51:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-23T03:51:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: applications will generate AVC messages and not be allowed access to 2024-03-23T03:51:37.155 INFO:teuthology.orchestra.run.smithi173.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-23T03:51:37.158 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-23T03:51:37.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:37.159 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-23T03:51:37.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:37.233 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-23T03:51:37.285 INFO:teuthology.orchestra.run.smithi173.stdout:client5348 2024-03-23T03:51:37.286 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:51:37.286 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:37.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:37.336 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir_files) 2024-03-23T03:51:37.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:37.402 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir_data) 2024-03-23T03:51:37.460 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:37.460 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_files -v 100 ./subdir_files) 2024-03-23T03:51:37.523 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:37.523 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 104857600 ./subdir_data) 2024-03-23T03:51:37.584 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:51:37.584 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:37.642 DEBUG:teuthology.orchestra.run.smithi173:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T03:51:37.642 DEBUG:teuthology.orchestra.run.smithi173:> import os 2024-03-23T03:51:37.642 DEBUG:teuthology.orchestra.run.smithi173:> import uuid 2024-03-23T03:51:37.642 DEBUG:teuthology.orchestra.run.smithi173:> 2024-03-23T03:51:37.642 DEBUG:teuthology.orchestra.run.smithi173:> n = 20 2024-03-23T03:51:37.642 DEBUG:teuthology.orchestra.run.smithi173:> create_hard_links = False 2024-03-23T03:51:37.642 DEBUG:teuthology.orchestra.run.smithi173:> if 0 > 0: 2024-03-23T03:51:37.642 DEBUG:teuthology.orchestra.run.smithi173:> create_hard_links = True 2024-03-23T03:51:37.643 DEBUG:teuthology.orchestra.run.smithi173:> path = "/home/ubuntu/cephtest/mnt.1/subdir_files/file" 2024-03-23T03:51:37.643 DEBUG:teuthology.orchestra.run.smithi173:> 2024-03-23T03:51:37.643 DEBUG:teuthology.orchestra.run.smithi173:> dpath = os.path.dirname(path) 2024-03-23T03:51:37.643 DEBUG:teuthology.orchestra.run.smithi173:> fnameprefix = os.path.basename(path) 2024-03-23T03:51:37.643 DEBUG:teuthology.orchestra.run.smithi173:> os.makedirs(dpath, exist_ok=True) 2024-03-23T03:51:37.643 DEBUG:teuthology.orchestra.run.smithi173:> 2024-03-23T03:51:37.643 DEBUG:teuthology.orchestra.run.smithi173:> try: 2024-03-23T03:51:37.643 DEBUG:teuthology.orchestra.run.smithi173:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-23T03:51:37.643 DEBUG:teuthology.orchestra.run.smithi173:> 2024-03-23T03:51:37.643 DEBUG:teuthology.orchestra.run.smithi173:> for i in range(n): 2024-03-23T03:51:37.643 DEBUG:teuthology.orchestra.run.smithi173:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-23T03:51:37.643 DEBUG:teuthology.orchestra.run.smithi173:> with open(fpath, '"'"'w'"'"') as f: 2024-03-23T03:51:37.643 DEBUG:teuthology.orchestra.run.smithi173:> f.write(f"{i}") 2024-03-23T03:51:37.643 DEBUG:teuthology.orchestra.run.smithi173:> if False: 2024-03-23T03:51:37.644 DEBUG:teuthology.orchestra.run.smithi173:> f.flush() 2024-03-23T03:51:37.644 DEBUG:teuthology.orchestra.run.smithi173:> os.fsync(f.fileno()) 2024-03-23T03:51:37.644 DEBUG:teuthology.orchestra.run.smithi173:> if False: 2024-03-23T03:51:37.644 DEBUG:teuthology.orchestra.run.smithi173:> os.unlink(fpath) 2024-03-23T03:51:37.644 DEBUG:teuthology.orchestra.run.smithi173:> if False: 2024-03-23T03:51:37.644 DEBUG:teuthology.orchestra.run.smithi173:> os.fsync(dirfd) 2024-03-23T03:51:37.644 DEBUG:teuthology.orchestra.run.smithi173:> if create_hard_links: 2024-03-23T03:51:37.644 DEBUG:teuthology.orchestra.run.smithi173:> for j in range(0): 2024-03-23T03:51:37.644 DEBUG:teuthology.orchestra.run.smithi173:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-23T03:51:37.644 DEBUG:teuthology.orchestra.run.smithi173:> if False: 2024-03-23T03:51:37.644 DEBUG:teuthology.orchestra.run.smithi173:> os.fsync(dirfd) 2024-03-23T03:51:37.644 DEBUG:teuthology.orchestra.run.smithi173:> finally: 2024-03-23T03:51:37.644 DEBUG:teuthology.orchestra.run.smithi173:> os.close(dirfd) 2024-03-23T03:51:37.644 DEBUG:teuthology.orchestra.run.smithi173:> ' 2024-03-23T03:51:37.963 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:51:37.963 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:38.059 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:38.059 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec dd if=/dev/urandom of=subdir_data/file bs=1M conv=fdatasync count=20 seek=0) 2024-03-23T03:51:38.364 INFO:teuthology.orchestra.run.smithi173.stderr:20+0 records in 2024-03-23T03:51:38.364 INFO:teuthology.orchestra.run.smithi173.stderr:20+0 records out 2024-03-23T03:51:38.364 INFO:teuthology.orchestra.run.smithi173.stderr:20971520 bytes (21 MB, 20 MiB) copied, 0.254097 s, 82.5 MB/s 2024-03-23T03:51:38.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:38.366 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_files -v 10 ./subdir_files) 2024-03-23T03:51:38.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:38.423 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.quota.max_bytes -v 1048576 ./subdir_data) 2024-03-23T03:51:38.475 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:51:38.476 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:38.533 DEBUG:teuthology.orchestra.run.smithi173:> adjust-ulimits daemon-helper kill python3 -c ' 2024-03-23T03:51:38.533 DEBUG:teuthology.orchestra.run.smithi173:> import os 2024-03-23T03:51:38.533 DEBUG:teuthology.orchestra.run.smithi173:> import uuid 2024-03-23T03:51:38.533 DEBUG:teuthology.orchestra.run.smithi173:> 2024-03-23T03:51:38.533 DEBUG:teuthology.orchestra.run.smithi173:> n = 40 2024-03-23T03:51:38.533 DEBUG:teuthology.orchestra.run.smithi173:> create_hard_links = False 2024-03-23T03:51:38.533 DEBUG:teuthology.orchestra.run.smithi173:> if 0 > 0: 2024-03-23T03:51:38.533 DEBUG:teuthology.orchestra.run.smithi173:> create_hard_links = True 2024-03-23T03:51:38.534 DEBUG:teuthology.orchestra.run.smithi173:> path = "/home/ubuntu/cephtest/mnt.1/subdir_files/file" 2024-03-23T03:51:38.534 DEBUG:teuthology.orchestra.run.smithi173:> 2024-03-23T03:51:38.534 DEBUG:teuthology.orchestra.run.smithi173:> dpath = os.path.dirname(path) 2024-03-23T03:51:38.534 DEBUG:teuthology.orchestra.run.smithi173:> fnameprefix = os.path.basename(path) 2024-03-23T03:51:38.534 DEBUG:teuthology.orchestra.run.smithi173:> os.makedirs(dpath, exist_ok=True) 2024-03-23T03:51:38.534 DEBUG:teuthology.orchestra.run.smithi173:> 2024-03-23T03:51:38.534 DEBUG:teuthology.orchestra.run.smithi173:> try: 2024-03-23T03:51:38.534 DEBUG:teuthology.orchestra.run.smithi173:> dirfd = os.open(dpath, os.O_DIRECTORY) 2024-03-23T03:51:38.534 DEBUG:teuthology.orchestra.run.smithi173:> 2024-03-23T03:51:38.534 DEBUG:teuthology.orchestra.run.smithi173:> for i in range(n): 2024-03-23T03:51:38.534 DEBUG:teuthology.orchestra.run.smithi173:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2024-03-23T03:51:38.534 DEBUG:teuthology.orchestra.run.smithi173:> with open(fpath, '"'"'w'"'"') as f: 2024-03-23T03:51:38.534 DEBUG:teuthology.orchestra.run.smithi173:> f.write(f"{i}") 2024-03-23T03:51:38.534 DEBUG:teuthology.orchestra.run.smithi173:> if False: 2024-03-23T03:51:38.534 DEBUG:teuthology.orchestra.run.smithi173:> f.flush() 2024-03-23T03:51:38.534 DEBUG:teuthology.orchestra.run.smithi173:> os.fsync(f.fileno()) 2024-03-23T03:51:38.535 DEBUG:teuthology.orchestra.run.smithi173:> if False: 2024-03-23T03:51:38.535 DEBUG:teuthology.orchestra.run.smithi173:> os.unlink(fpath) 2024-03-23T03:51:38.535 DEBUG:teuthology.orchestra.run.smithi173:> if False: 2024-03-23T03:51:38.535 DEBUG:teuthology.orchestra.run.smithi173:> os.fsync(dirfd) 2024-03-23T03:51:38.535 DEBUG:teuthology.orchestra.run.smithi173:> if create_hard_links: 2024-03-23T03:51:38.535 DEBUG:teuthology.orchestra.run.smithi173:> for j in range(0): 2024-03-23T03:51:38.535 DEBUG:teuthology.orchestra.run.smithi173:> os.system(f"ln {fpath} {dpath}/{fnameprefix}_{i}_{uuid.uuid4()}") 2024-03-23T03:51:38.535 DEBUG:teuthology.orchestra.run.smithi173:> if False: 2024-03-23T03:51:38.535 DEBUG:teuthology.orchestra.run.smithi173:> os.fsync(dirfd) 2024-03-23T03:51:38.535 DEBUG:teuthology.orchestra.run.smithi173:> finally: 2024-03-23T03:51:38.535 DEBUG:teuthology.orchestra.run.smithi173:> os.close(dirfd) 2024-03-23T03:51:38.535 DEBUG:teuthology.orchestra.run.smithi173:> ' 2024-03-23T03:51:39.013 INFO:teuthology.orchestra.run.smithi173.stderr:Traceback (most recent call last): 2024-03-23T03:51:39.013 INFO:teuthology.orchestra.run.smithi173.stderr: File "", line 20, in 2024-03-23T03:51:39.013 INFO:teuthology.orchestra.run.smithi173.stderr:OSError: [Errno 122] Disk quota exceeded: '/home/ubuntu/cephtest/mnt.1/subdir_files/file_20' 2024-03-23T03:51:39.048 INFO:teuthology.orchestra.run.smithi173.stderr:daemon-helper: command failed with exit status 1 2024-03-23T03:51:39.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:51:39.059 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:51:39.059 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:39.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:39.156 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest/mnt.1 && exec dd if=/dev/urandom of=subdir_data/file bs=1M conv=fdatasync count=40 seek=0) 2024-03-23T03:51:39.240 INFO:teuthology.orchestra.run.smithi173.stderr:dd: error writing 'subdir_data/file': Disk quota exceeded 2024-03-23T03:51:39.240 INFO:teuthology.orchestra.run.smithi173.stderr:2+0 records in 2024-03-23T03:51:39.240 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-23T03:51:39.240 INFO:teuthology.orchestra.run.smithi173.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158216 s, 66.3 MB/s 2024-03-23T03:51:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:51:39.242 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T03:51:39.242 DEBUG:teuthology.orchestra.run.smithi102:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:51:39.358 DEBUG:teuthology.orchestra.run.smithi173:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T03:51:39.476 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T03:51:39.476 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:51:39.476 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:39.532 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:51:39.533 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:39.586 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:51:39.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.794+0000 7fe9a0200640 1 Processor -- start 2024-03-23T03:51:39.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.795+0000 7fe9a0200640 1 -- start start 2024-03-23T03:51:39.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.796+0000 7fe9a0200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe9980ccb10 0x7fe9980ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:39.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.796+0000 7fe9a0200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe998071870 con 0x7fe998144460 2024-03-23T03:51:39.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.796+0000 7fe9a0200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe99815b2f0 con 0x7fe9980cec20 2024-03-23T03:51:39.797 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.796+0000 7fe9a0200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9980cd500 con 0x7fe9980ccb10 2024-03-23T03:51:39.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.797+0000 7fe99ce00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe9980ccb10 0x7fe9980ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:39.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.797+0000 7fe99ce00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe9980ccb10 0x7fe9980ccfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48438/0 (socket says 172.21.15.102:48438) 2024-03-23T03:51:39.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.797+0000 7fe99ce00640 1 -- 172.21.15.102:0/3430183219 learned_addr learned my addr 172.21.15.102:0/3430183219 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:39.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.797+0000 7fe99ce00640 1 -- 172.21.15.102:0/3430183219 >> v1:172.21.15.173:6789/0 conn(0x7fe9980cec20 legacy=0x7fe9980cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:39.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.797+0000 7fe99ce00640 1 -- 172.21.15.102:0/3430183219 >> v1:172.21.15.102:6789/0 conn(0x7fe998144460 legacy=0x7fe9980cc070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:39.798 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.797+0000 7fe99ce00640 1 -- 172.21.15.102:0/3430183219 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99815b5d0 con 0x7fe9980ccb10 2024-03-23T03:51:39.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.799+0000 7fe99ce00640 1 --2- 172.21.15.102:0/3430183219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9980ccb10 0x7fe9980ccfc0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fe988009e50 tx=0x7fe98802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ac65b679e1ebc4f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:39.799 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.799+0000 7fe997e00640 1 -- 172.21.15.102:0/3430183219 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98802ca80 con 0x7fe9980ccb10 2024-03-23T03:51:39.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.799+0000 7fe997e00640 1 -- 172.21.15.102:0/3430183219 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98802cbe0 con 0x7fe9980ccb10 2024-03-23T03:51:39.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.799+0000 7fe997e00640 1 -- 172.21.15.102:0/3430183219 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9880343f0 con 0x7fe9980ccb10 2024-03-23T03:51:39.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.800+0000 7fe9a0200640 1 -- 172.21.15.102:0/3430183219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9980ccb10 msgr2=0x7fe9980ccfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:39.800 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.800+0000 7fe9a0200640 1 --2- 172.21.15.102:0/3430183219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9980ccb10 0x7fe9980ccfc0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7fe988009e50 tx=0x7fe98802bee0 comp rx=0 tx=0).stop 2024-03-23T03:51:39.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.800+0000 7fe9a0200640 1 -- 172.21.15.102:0/3430183219 shutdown_connections 2024-03-23T03:51:39.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.800+0000 7fe9a0200640 1 --2- 172.21.15.102:0/3430183219 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9980ccb10 0x7fe9980ccfc0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:39.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.800+0000 7fe9a0200640 1 -- 172.21.15.102:0/3430183219 >> 172.21.15.102:0/3430183219 conn(0x7fe99806d3a0 msgr2=0x7fe99806d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:39.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.800+0000 7fe9a0200640 1 -- 172.21.15.102:0/3430183219 shutdown_connections 2024-03-23T03:51:39.801 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.800+0000 7fe9a0200640 1 -- 172.21.15.102:0/3430183219 wait complete. 2024-03-23T03:51:39.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.802+0000 7fe9a0200640 1 Processor -- start 2024-03-23T03:51:39.802 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.802+0000 7fe9a0200640 1 -- start start 2024-03-23T03:51:39.803 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.803+0000 7fe9a0200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe9980ccb10 0x7fe99813d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:39.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.803+0000 7fe9a0200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9980cec20 0x7fe9980b1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:39.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.803+0000 7fe9a0200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe998144460 0x7fe99813f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:39.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.803+0000 7fe9a0200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9981f5e20 con 0x7fe9980cec20 2024-03-23T03:51:39.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.803+0000 7fe9a0200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9981f60e0 con 0x7fe998144460 2024-03-23T03:51:39.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.803+0000 7fe9a0200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9981f63c0 con 0x7fe9980ccb10 2024-03-23T03:51:39.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.803+0000 7fe99ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9980cec20 0x7fe9980b1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:39.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.803+0000 7fe99ce00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9980cec20 0x7fe9980b1a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48454/0 (socket says 172.21.15.102:48454) 2024-03-23T03:51:39.804 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.803+0000 7fe99ce00640 1 -- 172.21.15.102:0/4014249478 learned_addr learned my addr 172.21.15.102:0/4014249478 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:39.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.803+0000 7fe99d800640 1 --2- 172.21.15.102:0/4014249478 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe9980ccb10 0x7fe99813d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:39.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.803+0000 7fe99e200640 1 --2- 172.21.15.102:0/4014249478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe998144460 0x7fe99813f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:39.805 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.804+0000 7fe99ce00640 1 -- 172.21.15.102:0/4014249478 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe9980ccb10 msgr2=0x7fe99813d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:39.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.804+0000 7fe99ce00640 1 --2- 172.21.15.102:0/4014249478 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe9980ccb10 0x7fe99813d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:39.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.804+0000 7fe99ce00640 1 -- 172.21.15.102:0/4014249478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe998144460 msgr2=0x7fe99813f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:39.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.804+0000 7fe99ce00640 1 --2- 172.21.15.102:0/4014249478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe998144460 0x7fe99813f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:39.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.804+0000 7fe99ce00640 1 -- 172.21.15.102:0/4014249478 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe988009b30 con 0x7fe9980cec20 2024-03-23T03:51:39.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.804+0000 7fe99e200640 1 --2- 172.21.15.102:0/4014249478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe998144460 0x7fe99813f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:39.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.804+0000 7fe99ce00640 1 --2- 172.21.15.102:0/4014249478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9980cec20 0x7fe9980b1a50 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fe988009e50 tx=0x7fe988004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:39.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.804+0000 7fe995600640 1 -- 172.21.15.102:0/4014249478 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98803e450 con 0x7fe9980cec20 2024-03-23T03:51:39.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.804+0000 7fe995600640 1 -- 172.21.15.102:0/4014249478 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98803e5f0 con 0x7fe9980cec20 2024-03-23T03:51:39.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.804+0000 7fe995600640 1 -- 172.21.15.102:0/4014249478 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98803e9a0 con 0x7fe9980cec20 2024-03-23T03:51:39.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.804+0000 7fe9a0200640 1 -- 172.21.15.102:0/4014249478 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9981f66a0 con 0x7fe9980cec20 2024-03-23T03:51:39.806 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.805+0000 7fe9a0200640 1 -- 172.21.15.102:0/4014249478 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe99815a7a0 con 0x7fe9980cec20 2024-03-23T03:51:39.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.806+0000 7fe9a0200640 1 -- 172.21.15.102:0/4014249478 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9981f6b60 con 0x7fe9980cec20 2024-03-23T03:51:39.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.807+0000 7fe995600640 1 -- 172.21.15.102:0/4014249478 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe98802cd50 con 0x7fe9980cec20 2024-03-23T03:51:39.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.808+0000 7fe995600640 1 --2- 172.21.15.102:0/4014249478 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe97c06ac00 0x7fe97c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:39.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.809+0000 7fe995600640 1 -- 172.21.15.102:0/4014249478 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe9880ac100 con 0x7fe9980cec20 2024-03-23T03:51:39.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.809+0000 7fe99d800640 1 --2- 172.21.15.102:0/4014249478 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe97c06ac00 0x7fe97c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:39.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.809+0000 7fe99d800640 1 --2- 172.21.15.102:0/4014249478 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe97c06ac00 0x7fe97c06d0c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe98c000f60 tx=0x7fe98c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:39.813 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:39.813+0000 7fe995600640 1 -- 172.21.15.102:0/4014249478 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe9880772c0 con 0x7fe9980cec20 2024-03-23T03:51:40.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.008+0000 7fe9a0200640 1 -- 172.21.15.102:0/4014249478 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe998066bb0 con 0x7fe9980cec20 2024-03-23T03:51:40.011 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.011+0000 7fe995600640 1 -- 172.21.15.102:0/4014249478 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+16296 (secure 0 0 0) 0x7fe98807a8e0 con 0x7fe9980cec20 2024-03-23T03:51:40.012 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:40.012 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":105,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:51:23.233227+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:51:19.796618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-23T03:51:21.283447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:51:40.014 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.013+0000 7fe9a0200640 1 -- 172.21.15.102:0/4014249478 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe97c06ac00 msgr2=0x7fe97c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.013+0000 7fe9a0200640 1 --2- 172.21.15.102:0/4014249478 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe97c06ac00 0x7fe97c06d0c0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fe98c000f60 tx=0x7fe98c009040 comp rx=0 tx=0).stop 2024-03-23T03:51:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.013+0000 7fe9a0200640 1 -- 172.21.15.102:0/4014249478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9980cec20 msgr2=0x7fe9980b1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.013+0000 7fe9a0200640 1 --2- 172.21.15.102:0/4014249478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9980cec20 0x7fe9980b1a50 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7fe988009e50 tx=0x7fe988004a40 comp rx=0 tx=0).stop 2024-03-23T03:51:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.014+0000 7fe9a0200640 1 -- 172.21.15.102:0/4014249478 shutdown_connections 2024-03-23T03:51:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.014+0000 7fe9a0200640 1 --2- 172.21.15.102:0/4014249478 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe97c06ac00 0x7fe97c06d0c0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.014+0000 7fe9a0200640 1 --2- 172.21.15.102:0/4014249478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe998144460 0x7fe99813f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.014+0000 7fe9a0200640 1 --2- 172.21.15.102:0/4014249478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9980cec20 0x7fe9980b1a50 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.014+0000 7fe9a0200640 1 --2- 172.21.15.102:0/4014249478 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe9980ccb10 0x7fe99813d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.014+0000 7fe9a0200640 1 -- 172.21.15.102:0/4014249478 >> 172.21.15.102:0/4014249478 conn(0x7fe99806d3a0 msgr2=0x7fe9980cc070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.014+0000 7fe9a0200640 1 -- 172.21.15.102:0/4014249478 shutdown_connections 2024-03-23T03:51:40.015 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.014+0000 7fe9a0200640 1 -- 172.21.15.102:0/4014249478 wait complete. 2024-03-23T03:51:40.034 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:51:19.796618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:51:21.283447+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:51:40.034 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:51:40.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.246+0000 7f712e000640 1 Processor -- start 2024-03-23T03:51:40.247 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.246+0000 7f712e000640 1 -- start start 2024-03-23T03:51:40.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.248+0000 7f712e000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f712815cd10 0x7f71280cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:40.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.248+0000 7f712e000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71280d1f30 0x7f71280b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:40.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.248+0000 7f712e000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71281617d0 con 0x7f71280b47d0 2024-03-23T03:51:40.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.248+0000 7f712e000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71280b5150 con 0x7f712815cd10 2024-03-23T03:51:40.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.248+0000 7f712e000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7128160c50 con 0x7f71280d1f30 2024-03-23T03:51:40.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.248+0000 7f7126a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f71280b47d0 0x7f712806b030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50036/0 (socket says 172.21.15.102:50036) 2024-03-23T03:51:40.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.248+0000 7f7126a00640 1 -- 172.21.15.102:0/855659046 learned_addr learned my addr 172.21.15.102:0/855659046 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:40.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.249+0000 7f7127e00640 1 --2- 172.21.15.102:0/855659046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71280d1f30 0x7f71280b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:40.249 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.249+0000 7f7127400640 1 --2- 172.21.15.102:0/855659046 >> v2:172.21.15.173:3300/0 conn(0x7f712815cd10 0x7f71280cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:40.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.249+0000 7f7126000640 1 -- 172.21.15.102:0/855659046 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942071495 0 0) 0x7f71281617d0 con 0x7f71280b47d0 2024-03-23T03:51:40.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.249+0000 7f7126000640 1 -- 172.21.15.102:0/855659046 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f710c003520 con 0x7f71280b47d0 2024-03-23T03:51:40.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.249+0000 7f7126000640 1 -- 172.21.15.102:0/855659046 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2702899947 0 0) 0x7f710c003520 con 0x7f71280b47d0 2024-03-23T03:51:40.251 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.249+0000 7f7126000640 1 -- 172.21.15.102:0/855659046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71280d1f30 msgr2=0x7f71280b4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.249+0000 7f7126000640 1 --2- 172.21.15.102:0/855659046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71280d1f30 0x7f71280b4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.249+0000 7f7126000640 1 -- 172.21.15.102:0/855659046 >> v2:172.21.15.173:3300/0 conn(0x7f712815cd10 msgr2=0x7f71280cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.249+0000 7f7126000640 1 --2- 172.21.15.102:0/855659046 >> v2:172.21.15.173:3300/0 conn(0x7f712815cd10 0x7f71280cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.249+0000 7f7126000640 1 -- 172.21.15.102:0/855659046 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7128160f30 con 0x7f71280b47d0 2024-03-23T03:51:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.249+0000 7f7126000640 1 -- 172.21.15.102:0/855659046 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f7114001670 con 0x7f71280b47d0 2024-03-23T03:51:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.249+0000 7f7126000640 1 -- 172.21.15.102:0/855659046 >> v1:172.21.15.102:6789/0 conn(0x7f71280b47d0 legacy=0x7f712806b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.249+0000 7f7126000640 1 --2- 172.21.15.102:0/855659046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f710c003970 0x7f710c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.250+0000 7f7126000640 1 --2- 172.21.15.102:0/855659046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f710c0047c0 0x7f710c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.250+0000 7f7126000640 1 --2- 172.21.15.102:0/855659046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71280d1f30 0x7f710c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.250+0000 7f7126000640 1 -- 172.21.15.102:0/855659046 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71280b5150 con 0x7f710c003970 2024-03-23T03:51:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.250+0000 7f7126000640 1 -- 172.21.15.102:0/855659046 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7128160c50 con 0x7f710c0047c0 2024-03-23T03:51:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.250+0000 7f7126000640 1 -- 172.21.15.102:0/855659046 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f710c00b980 con 0x7f71280d1f30 2024-03-23T03:51:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.250+0000 7f7127400640 1 --2- 172.21.15.102:0/855659046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f710c003970 0x7f710c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:40.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.250+0000 7f7127e00640 1 --2- 172.21.15.102:0/855659046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71280d1f30 0x7f710c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:40.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.250+0000 7f7127400640 1 -- 172.21.15.102:0/855659046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71280d1f30 msgr2=0x7f710c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:40.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.250+0000 7f7127400640 1 --2- 172.21.15.102:0/855659046 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71280d1f30 0x7f710c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.250+0000 7f7127400640 1 -- 172.21.15.102:0/855659046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f710c0047c0 msgr2=0x7f710c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:40.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.250+0000 7f7127400640 1 --2- 172.21.15.102:0/855659046 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f710c0047c0 0x7f710c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.250+0000 7f7127400640 1 -- 172.21.15.102:0/855659046 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710c003520 con 0x7f710c003970 2024-03-23T03:51:40.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.251+0000 7f7127400640 1 --2- 172.21.15.102:0/855659046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f710c003970 0x7f710c003d90 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f7118009fc0 tx=0x7f710c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c0e2eb15e29c73 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:40.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.251+0000 7f7126000640 1 -- 172.21.15.102:0/855659046 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f711802cc60 con 0x7f710c003970 2024-03-23T03:51:40.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.252+0000 7f7126000640 1 -- 172.21.15.102:0/855659046 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711800f040 con 0x7f710c003970 2024-03-23T03:51:40.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.252+0000 7f7126000640 1 -- 172.21.15.102:0/855659046 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7118034440 con 0x7f710c003970 2024-03-23T03:51:40.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.252+0000 7f712e000640 1 -- 172.21.15.102:0/855659046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f710c003970 msgr2=0x7f710c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:40.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.252+0000 7f712e000640 1 --2- 172.21.15.102:0/855659046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f710c003970 0x7f710c003d90 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f7118009fc0 tx=0x7f710c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.252+0000 7f7127e00640 1 -- 172.21.15.102:0/855659046 reap_dead start 2024-03-23T03:51:40.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.252+0000 7f712e000640 1 -- 172.21.15.102:0/855659046 shutdown_connections 2024-03-23T03:51:40.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.252+0000 7f712e000640 1 -- 172.21.15.102:0/855659046 >> 172.21.15.102:0/855659046 conn(0x7f71280cf830 msgr2=0x7f71280cfc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:40.254 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.252+0000 7f712e000640 1 -- 172.21.15.102:0/855659046 shutdown_connections 2024-03-23T03:51:40.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.252+0000 7f712e000640 1 -- 172.21.15.102:0/855659046 wait complete. 2024-03-23T03:51:40.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.254+0000 7f712e000640 1 Processor -- start 2024-03-23T03:51:40.255 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.254+0000 7f712e000640 1 -- start start 2024-03-23T03:51:40.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.255+0000 7f712e000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f710c003970 0x7f71280ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:40.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.255+0000 7f712e000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71280c8210 0x7f71281f4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:40.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.255+0000 7f712e000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71281f2300 0x7f71281f2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:40.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.255+0000 7f712e000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71280ca9b0 con 0x7f71280c8210 2024-03-23T03:51:40.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.255+0000 7f712e000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71280cac80 con 0x7f710c003970 2024-03-23T03:51:40.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.255+0000 7f712e000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71280caf60 con 0x7f71281f2300 2024-03-23T03:51:40.257 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.255+0000 7f7126a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71280c8210 0x7f71281f4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.255+0000 7f7126a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71280c8210 0x7f71281f4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48464/0 (socket says 172.21.15.102:48464) 2024-03-23T03:51:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.255+0000 7f7126a00640 1 -- 172.21.15.102:0/2928948976 learned_addr learned my addr 172.21.15.102:0/2928948976 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.256+0000 7f7127e00640 1 --2- 172.21.15.102:0/2928948976 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71281f2300 0x7f71281f2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.256+0000 7f7127400640 1 --2- 172.21.15.102:0/2928948976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f710c003970 0x7f71280ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.256+0000 7f7126a00640 1 -- 172.21.15.102:0/2928948976 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71281f2300 msgr2=0x7f71281f2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.256+0000 7f7126a00640 1 --2- 172.21.15.102:0/2928948976 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71281f2300 0x7f71281f2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.256+0000 7f7126a00640 1 -- 172.21.15.102:0/2928948976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f710c003970 msgr2=0x7f71280ca470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.256+0000 7f7126a00640 1 --2- 172.21.15.102:0/2928948976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f710c003970 0x7f71280ca470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.256+0000 7f7126a00640 1 -- 172.21.15.102:0/2928948976 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71280cb240 con 0x7f71280c8210 2024-03-23T03:51:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.256+0000 7f7127e00640 1 --2- 172.21.15.102:0/2928948976 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71281f2300 0x7f71281f2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.256+0000 7f7126a00640 1 --2- 172.21.15.102:0/2928948976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71280c8210 0x7f71281f4e10 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f711400c770 tx=0x7f711400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.256+0000 7f711f400640 1 -- 172.21.15.102:0/2928948976 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71140074e0 con 0x7f71280c8210 2024-03-23T03:51:40.258 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.256+0000 7f711f400640 1 -- 172.21.15.102:0/2928948976 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7114007660 con 0x7f71280c8210 2024-03-23T03:51:40.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.256+0000 7f711f400640 1 -- 172.21.15.102:0/2928948976 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7114004a20 con 0x7f71280c8210 2024-03-23T03:51:40.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.256+0000 7f712e000640 1 -- 172.21.15.102:0/2928948976 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71280cb520 con 0x7f71280c8210 2024-03-23T03:51:40.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.257+0000 7f712e000640 1 -- 172.21.15.102:0/2928948976 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71281f5350 con 0x7f71280c8210 2024-03-23T03:51:40.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.258+0000 7f712e000640 1 -- 172.21.15.102:0/2928948976 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71280cba40 con 0x7f71280c8210 2024-03-23T03:51:40.259 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.259+0000 7f711f400640 1 -- 172.21.15.102:0/2928948976 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7114004b80 con 0x7f71280c8210 2024-03-23T03:51:40.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.259+0000 7f711f400640 1 --2- 172.21.15.102:0/2928948976 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f710806ac00 0x7f710806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:40.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.260+0000 7f711f400640 1 -- 172.21.15.102:0/2928948976 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7114088a80 con 0x7f71280c8210 2024-03-23T03:51:40.260 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.260+0000 7f7127400640 1 --2- 172.21.15.102:0/2928948976 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f710806ac00 0x7f710806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:40.261 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.260+0000 7f7127400640 1 --2- 172.21.15.102:0/2928948976 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f710806ac00 0x7f710806d0c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7118036000 tx=0x7f711802e000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:40.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.264+0000 7f711f400640 1 -- 172.21.15.102:0/2928948976 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f71280cba40 con 0x7f71280c8210 2024-03-23T03:51:40.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.458+0000 7f712e000640 1 -- 172.21.15.102:0/2928948976 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f71280cba40 con 0x7f71280c8210 2024-03-23T03:51:40.459 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.459+0000 7f711f400640 1 -- 172.21.15.102:0/2928948976 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) v1 ==== 93+0+10 (secure 0 0 0) 0x7f71280cba40 con 0x7f71280c8210 2024-03-23T03:51:40.460 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:51:40.462 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.461+0000 7f712e000640 1 -- 172.21.15.102:0/2928948976 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f710806ac00 msgr2=0x7f710806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:40.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.461+0000 7f712e000640 1 --2- 172.21.15.102:0/2928948976 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f710806ac00 0x7f710806d0c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f7118036000 tx=0x7f711802e000 comp rx=0 tx=0).stop 2024-03-23T03:51:40.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.461+0000 7f712e000640 1 -- 172.21.15.102:0/2928948976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71280c8210 msgr2=0x7f71281f4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:40.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.461+0000 7f712e000640 1 --2- 172.21.15.102:0/2928948976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71280c8210 0x7f71281f4e10 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f711400c770 tx=0x7f711400cc30 comp rx=0 tx=0).stop 2024-03-23T03:51:40.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.461+0000 7f712e000640 1 -- 172.21.15.102:0/2928948976 shutdown_connections 2024-03-23T03:51:40.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.461+0000 7f712e000640 1 --2- 172.21.15.102:0/2928948976 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f710806ac00 0x7f710806d0c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.461+0000 7f712e000640 1 --2- 172.21.15.102:0/2928948976 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71281f2300 0x7f71281f2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.461+0000 7f712e000640 1 --2- 172.21.15.102:0/2928948976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71280c8210 0x7f71281f4e10 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.461+0000 7f712e000640 1 --2- 172.21.15.102:0/2928948976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f710c003970 0x7f71280ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.461+0000 7f712e000640 1 -- 172.21.15.102:0/2928948976 >> 172.21.15.102:0/2928948976 conn(0x7f71280cf830 msgr2=0x7f71280b3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:40.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.462+0000 7f712e000640 1 -- 172.21.15.102:0/2928948976 shutdown_connections 2024-03-23T03:51:40.463 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.462+0000 7f712e000640 1 -- 172.21.15.102:0/2928948976 wait complete. 2024-03-23T03:51:40.481 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:51:40.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.688+0000 7f7dc9800640 1 Processor -- start 2024-03-23T03:51:40.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.689+0000 7f7dc9800640 1 -- start start 2024-03-23T03:51:40.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.690+0000 7f7dc9800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f7dc41456a0 0x7f7dc40ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:40.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.690+0000 7f7dc9800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dc4145c20 0x7f7dc40cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:40.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.690+0000 7f7dc9800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc406d390 con 0x7f7dc4070cd0 2024-03-23T03:51:40.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.690+0000 7f7dc9800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc40ad890 con 0x7f7dc41456a0 2024-03-23T03:51:40.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.690+0000 7f7dc9800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc4145a80 con 0x7f7dc4145c20 2024-03-23T03:51:40.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.691+0000 7f7dc3400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dc4145c20 0x7f7dc40cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:40.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.691+0000 7f7dc3400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dc4145c20 0x7f7dc40cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:52556/0 (socket says 172.21.15.102:52556) 2024-03-23T03:51:40.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.691+0000 7f7dc3400640 1 -- 172.21.15.102:0/163176632 learned_addr learned my addr 172.21.15.102:0/163176632 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:40.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.691+0000 7f7dc2a00640 1 --2- 172.21.15.102:0/163176632 >> v2:172.21.15.173:3300/0 conn(0x7f7dc41456a0 0x7f7dc40ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:40.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.692+0000 7f7dc3400640 1 -- 172.21.15.102:0/163176632 >> v2:172.21.15.173:3300/0 conn(0x7f7dc41456a0 msgr2=0x7f7dc40ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:40.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.692+0000 7f7dc3400640 1 --2- 172.21.15.102:0/163176632 >> v2:172.21.15.173:3300/0 conn(0x7f7dc41456a0 0x7f7dc40ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.692+0000 7f7dc3400640 1 -- 172.21.15.102:0/163176632 >> v1:172.21.15.173:6789/0 conn(0x7f7dc4070cd0 legacy=0x7f7dc40ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:40.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.692+0000 7f7dc2000640 1 -- 172.21.15.102:0/163176632 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837656444 0 0) 0x7f7dc406d390 con 0x7f7dc4070cd0 2024-03-23T03:51:40.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.692+0000 7f7dc3400640 1 -- 172.21.15.102:0/163176632 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc406d670 con 0x7f7dc4145c20 2024-03-23T03:51:40.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.693+0000 7f7dc3400640 1 --2- 172.21.15.102:0/163176632 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dc4145c20 0x7f7dc40cee70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc009e50 tx=0x7f7dbc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=770394393be40772 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:40.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.693+0000 7f7dc2000640 1 -- 172.21.15.102:0/163176632 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc02cb00 con 0x7f7dc4145c20 2024-03-23T03:51:40.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.693+0000 7f7dc2000640 1 -- 172.21.15.102:0/163176632 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dbc02cc60 con 0x7f7dc4145c20 2024-03-23T03:51:40.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.693+0000 7f7dc2000640 1 -- 172.21.15.102:0/163176632 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dbc0343f0 con 0x7f7dc4145c20 2024-03-23T03:51:40.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.694+0000 7f7dc9800640 1 -- 172.21.15.102:0/163176632 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dc4145c20 msgr2=0x7f7dc40cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:40.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.694+0000 7f7dc9800640 1 --2- 172.21.15.102:0/163176632 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dc4145c20 0x7f7dc40cee70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7dbc009e50 tx=0x7f7dbc02bf60 comp rx=0 tx=0).stop 2024-03-23T03:51:40.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.694+0000 7f7dc9800640 1 -- 172.21.15.102:0/163176632 shutdown_connections 2024-03-23T03:51:40.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.694+0000 7f7dc9800640 1 --2- 172.21.15.102:0/163176632 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dc4145c20 0x7f7dc40cee70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.694+0000 7f7dc9800640 1 --2- 172.21.15.102:0/163176632 >> v2:172.21.15.173:3300/0 conn(0x7f7dc41456a0 0x7f7dc40ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.694+0000 7f7dc9800640 1 -- 172.21.15.102:0/163176632 >> 172.21.15.102:0/163176632 conn(0x7f7dc406a0b0 msgr2=0x7f7dc406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:40.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.694+0000 7f7dc9800640 1 -- 172.21.15.102:0/163176632 shutdown_connections 2024-03-23T03:51:40.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.694+0000 7f7dc9800640 1 -- 172.21.15.102:0/163176632 wait complete. 2024-03-23T03:51:40.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.696+0000 7f7dc9800640 1 Processor -- start 2024-03-23T03:51:40.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.696+0000 7f7dc9800640 1 -- start start 2024-03-23T03:51:40.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.697+0000 7f7dc9800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dc4070cd0 0x7f7dc41338f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:40.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.697+0000 7f7dc9800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7dc41456a0 0x7f7dc415e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:40.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.697+0000 7f7dc9800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dc4145c20 0x7f7dc415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:40.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.697+0000 7f7dc9800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc413f3e0 con 0x7f7dc4070cd0 2024-03-23T03:51:40.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.697+0000 7f7dc9800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc4140220 con 0x7f7dc41456a0 2024-03-23T03:51:40.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.697+0000 7f7dc9800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc413f610 con 0x7f7dc4145c20 2024-03-23T03:51:40.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.697+0000 7f7dc2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dc4070cd0 0x7f7dc41338f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:40.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.697+0000 7f7dc2a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dc4070cd0 0x7f7dc41338f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48478/0 (socket says 172.21.15.102:48478) 2024-03-23T03:51:40.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.697+0000 7f7dc2a00640 1 -- 172.21.15.102:0/136211544 learned_addr learned my addr 172.21.15.102:0/136211544 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:40.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.698+0000 7f7dc3400640 1 --2- 172.21.15.102:0/136211544 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dc4145c20 0x7f7dc415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:40.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.698+0000 7f7dbbe00640 1 --2- 172.21.15.102:0/136211544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7dc41456a0 0x7f7dc415e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:40.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.698+0000 7f7dc2a00640 1 -- 172.21.15.102:0/136211544 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dc4145c20 msgr2=0x7f7dc415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:40.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.698+0000 7f7dc2a00640 1 --2- 172.21.15.102:0/136211544 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dc4145c20 0x7f7dc415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.698+0000 7f7dc2a00640 1 -- 172.21.15.102:0/136211544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7dc41456a0 msgr2=0x7f7dc415e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:40.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.698+0000 7f7dc2a00640 1 --2- 172.21.15.102:0/136211544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7dc41456a0 0x7f7dc415e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.698+0000 7f7dc2a00640 1 -- 172.21.15.102:0/136211544 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dbc009b30 con 0x7f7dc4070cd0 2024-03-23T03:51:40.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.698+0000 7f7dc2a00640 1 --2- 172.21.15.102:0/136211544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dc4070cd0 0x7f7dc41338f0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f7db400e9b0 tx=0x7f7db400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:40.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.698+0000 7f7dbaa00640 1 -- 172.21.15.102:0/136211544 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db4009670 con 0x7f7dc4070cd0 2024-03-23T03:51:40.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.698+0000 7f7dbaa00640 1 -- 172.21.15.102:0/136211544 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db400c9a0 con 0x7f7dc4070cd0 2024-03-23T03:51:40.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.698+0000 7f7dc9800640 1 -- 172.21.15.102:0/136211544 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc413f8f0 con 0x7f7dc4070cd0 2024-03-23T03:51:40.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.698+0000 7f7dbaa00640 1 -- 172.21.15.102:0/136211544 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db400cd50 con 0x7f7dc4070cd0 2024-03-23T03:51:40.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.699+0000 7f7dc9800640 1 -- 172.21.15.102:0/136211544 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc413feb0 con 0x7f7dc4070cd0 2024-03-23T03:51:40.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.700+0000 7f7dc9800640 1 -- 172.21.15.102:0/136211544 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc4073390 con 0x7f7dc4070cd0 2024-03-23T03:51:40.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.701+0000 7f7dbaa00640 1 -- 172.21.15.102:0/136211544 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7db4013470 con 0x7f7dc4070cd0 2024-03-23T03:51:40.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.701+0000 7f7dbaa00640 1 --2- 172.21.15.102:0/136211544 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7da006ac00 0x7f7da006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:40.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.702+0000 7f7dbaa00640 1 -- 172.21.15.102:0/136211544 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7db40897c0 con 0x7f7dc4070cd0 2024-03-23T03:51:40.702 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.702+0000 7f7dbbe00640 1 --2- 172.21.15.102:0/136211544 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7da006ac00 0x7f7da006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:40.703 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.702+0000 7f7dbbe00640 1 --2- 172.21.15.102:0/136211544 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7da006ac00 0x7f7da006d0c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4130a40 tx=0x7f7dac009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:40.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.706+0000 7f7dbaa00640 1 -- 172.21.15.102:0/136211544 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7db40547c0 con 0x7f7dc4070cd0 2024-03-23T03:51:40.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.900+0000 7f7dc9800640 1 -- 172.21.15.102:0/136211544 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7dc40af610 con 0x7f7dc4070cd0 2024-03-23T03:51:40.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.901+0000 7f7dbaa00640 1 -- 172.21.15.102:0/136211544 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v105) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7db4057de0 con 0x7f7dc4070cd0 2024-03-23T03:51:40.901 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:51:40.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.903+0000 7f7dc9800640 1 -- 172.21.15.102:0/136211544 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7da006ac00 msgr2=0x7f7da006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:40.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.903+0000 7f7dc9800640 1 --2- 172.21.15.102:0/136211544 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7da006ac00 0x7f7da006d0c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4130a40 tx=0x7f7dac009040 comp rx=0 tx=0).stop 2024-03-23T03:51:40.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.903+0000 7f7dc9800640 1 -- 172.21.15.102:0/136211544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dc4070cd0 msgr2=0x7f7dc41338f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:40.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.903+0000 7f7dc9800640 1 --2- 172.21.15.102:0/136211544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dc4070cd0 0x7f7dc41338f0 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f7db400e9b0 tx=0x7f7db400ee70 comp rx=0 tx=0).stop 2024-03-23T03:51:40.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.903+0000 7f7dc9800640 1 -- 172.21.15.102:0/136211544 shutdown_connections 2024-03-23T03:51:40.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.903+0000 7f7dc9800640 1 --2- 172.21.15.102:0/136211544 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7da006ac00 0x7f7da006d0c0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.903+0000 7f7dc9800640 1 --2- 172.21.15.102:0/136211544 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7dc4145c20 0x7f7dc415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.903+0000 7f7dc9800640 1 --2- 172.21.15.102:0/136211544 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7dc41456a0 0x7f7dc415e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.903+0000 7f7dc9800640 1 --2- 172.21.15.102:0/136211544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7dc4070cd0 0x7f7dc41338f0 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:40.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.903+0000 7f7dc9800640 1 -- 172.21.15.102:0/136211544 >> 172.21.15.102:0/136211544 conn(0x7f7dc406a0b0 msgr2=0x7f7dc4160450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:40.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.903+0000 7f7dc9800640 1 -- 172.21.15.102:0/136211544 shutdown_connections 2024-03-23T03:51:40.905 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:40.903+0000 7f7dc9800640 1 -- 172.21.15.102:0/136211544 wait complete. 2024-03-23T03:51:40.922 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:51:41.131 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.131+0000 7f9fbd000640 1 Processor -- start 2024-03-23T03:51:41.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.131+0000 7f9fbd000640 1 -- start start 2024-03-23T03:51:41.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.132+0000 7f9fbd000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9fb806e240 0x7f9fb8156580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:41.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.132+0000 7f9fbd000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f9fb8159830 0x7f9fb80705e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:41.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.132+0000 7f9fbd000640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb8159c40 con 0x7f9fb8070150 2024-03-23T03:51:41.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.132+0000 7f9fbd000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fb8070b20 con 0x7f9fb806e240 2024-03-23T03:51:41.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.132+0000 7f9fbd000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fb8159e90 con 0x7f9fb8159830 2024-03-23T03:51:41.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.133+0000 7f9fb5e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9fb806e240 0x7f9fb8156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:41.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.133+0000 7f9fb5e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f9fb806e240 0x7f9fb8156580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48486/0 (socket says 172.21.15.102:48486) 2024-03-23T03:51:41.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.133+0000 7f9fb5e00640 1 -- 172.21.15.102:0/1798226364 learned_addr learned my addr 172.21.15.102:0/1798226364 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:41.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.133+0000 7f9fb5400640 1 --1- 172.21.15.102:0/1798226364 >> v1:172.21.15.102:6789/0 conn(0x7f9fb8070150 0x7f9fb8159120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:50046/0 (socket says 172.21.15.102:50046) 2024-03-23T03:51:41.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.133+0000 7f9fb5e00640 1 -- 172.21.15.102:0/1798226364 >> v2:172.21.15.173:3300/0 conn(0x7f9fb8159830 msgr2=0x7f9fb80705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:41.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.133+0000 7f9fb5e00640 1 --2- 172.21.15.102:0/1798226364 >> v2:172.21.15.173:3300/0 conn(0x7f9fb8159830 0x7f9fb80705e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.133+0000 7f9fb5e00640 1 -- 172.21.15.102:0/1798226364 >> v1:172.21.15.102:6789/0 conn(0x7f9fb8070150 legacy=0x7f9fb8159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:41.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.133+0000 7f9fb5e00640 1 -- 172.21.15.102:0/1798226364 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb80b1660 con 0x7f9fb806e240 2024-03-23T03:51:41.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.134+0000 7f9fb5e00640 1 --2- 172.21.15.102:0/1798226364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fb806e240 0x7f9fb8156580 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f9fb80ce4c0 tx=0x7f9fa802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=901dd1916ade67ee server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:41.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.135+0000 7f9fb4a00640 1 -- 172.21.15.102:0/1798226364 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa802ca80 con 0x7f9fb806e240 2024-03-23T03:51:41.135 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.135+0000 7f9fb4a00640 1 -- 172.21.15.102:0/1798226364 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fa802cbe0 con 0x7f9fb806e240 2024-03-23T03:51:41.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.135+0000 7f9fb4a00640 1 -- 172.21.15.102:0/1798226364 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fa80343f0 con 0x7f9fb806e240 2024-03-23T03:51:41.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.135+0000 7f9fbd000640 1 -- 172.21.15.102:0/1798226364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fb806e240 msgr2=0x7f9fb8156580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:41.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.135+0000 7f9fbd000640 1 --2- 172.21.15.102:0/1798226364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fb806e240 0x7f9fb8156580 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f9fb80ce4c0 tx=0x7f9fa802bee0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.136+0000 7f9fbd000640 1 -- 172.21.15.102:0/1798226364 shutdown_connections 2024-03-23T03:51:41.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.136+0000 7f9fbd000640 1 --2- 172.21.15.102:0/1798226364 >> v2:172.21.15.173:3300/0 conn(0x7f9fb8159830 0x7f9fb80705e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.136+0000 7f9fbd000640 1 --2- 172.21.15.102:0/1798226364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fb806e240 0x7f9fb8156580 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.136+0000 7f9fbd000640 1 -- 172.21.15.102:0/1798226364 >> 172.21.15.102:0/1798226364 conn(0x7f9fb806a0b0 msgr2=0x7f9fb806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:41.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.136+0000 7f9fbd000640 1 -- 172.21.15.102:0/1798226364 shutdown_connections 2024-03-23T03:51:41.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.136+0000 7f9fbd000640 1 -- 172.21.15.102:0/1798226364 wait complete. 2024-03-23T03:51:41.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.138+0000 7f9fbd000640 1 Processor -- start 2024-03-23T03:51:41.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.138+0000 7f9fbd000640 1 -- start start 2024-03-23T03:51:41.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.139+0000 7f9fbd000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fb806e240 0x7f9fb814b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:41.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.139+0000 7f9fbd000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9fb8070150 0x7f9fb8158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:41.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.139+0000 7f9fbd000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fb8159830 0x7f9fb81585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:41.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.139+0000 7f9fbd000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb814c220 con 0x7f9fb8159830 2024-03-23T03:51:41.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.139+0000 7f9fbd000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb81e3a90 con 0x7f9fb806e240 2024-03-23T03:51:41.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.139+0000 7f9fbd000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb81e3d70 con 0x7f9fb8070150 2024-03-23T03:51:41.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.139+0000 7f9fb6800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fb8159830 0x7f9fb81585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:41.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.139+0000 7f9fb6800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fb8159830 0x7f9fb81585b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48492/0 (socket says 172.21.15.102:48492) 2024-03-23T03:51:41.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.139+0000 7f9fb6800640 1 -- 172.21.15.102:0/1323260462 learned_addr learned my addr 172.21.15.102:0/1323260462 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:41.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.140+0000 7f9fb5e00640 1 --2- 172.21.15.102:0/1323260462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fb806e240 0x7f9fb814b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:41.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.140+0000 7f9fb5400640 1 --2- 172.21.15.102:0/1323260462 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9fb8070150 0x7f9fb8158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:41.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.140+0000 7f9fb6800640 1 -- 172.21.15.102:0/1323260462 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9fb8070150 msgr2=0x7f9fb8158070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:41.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.140+0000 7f9fb6800640 1 --2- 172.21.15.102:0/1323260462 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9fb8070150 0x7f9fb8158070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.140+0000 7f9fb6800640 1 -- 172.21.15.102:0/1323260462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fb806e240 msgr2=0x7f9fb814b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:41.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.140+0000 7f9fb6800640 1 --2- 172.21.15.102:0/1323260462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fb806e240 0x7f9fb814b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.140+0000 7f9fb6800640 1 -- 172.21.15.102:0/1323260462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa8009b30 con 0x7f9fb8159830 2024-03-23T03:51:41.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.140+0000 7f9fb5400640 1 --2- 172.21.15.102:0/1323260462 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9fb8070150 0x7f9fb8158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:41.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.140+0000 7f9fb6800640 1 --2- 172.21.15.102:0/1323260462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fb8159830 0x7f9fb81585b0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f9fb000b410 tx=0x7f9fb000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:41.142 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.141+0000 7f9fae000640 1 -- 172.21.15.102:0/1323260462 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb0011830 con 0x7f9fb8159830 2024-03-23T03:51:41.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.141+0000 7f9fae000640 1 -- 172.21.15.102:0/1323260462 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb00119b0 con 0x7f9fb8159830 2024-03-23T03:51:41.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.141+0000 7f9fae000640 1 -- 172.21.15.102:0/1323260462 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fb0011d60 con 0x7f9fb8159830 2024-03-23T03:51:41.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.141+0000 7f9fbd000640 1 -- 172.21.15.102:0/1323260462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb81e4050 con 0x7f9fb8159830 2024-03-23T03:51:41.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.141+0000 7f9fbd000640 1 -- 172.21.15.102:0/1323260462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb80c7940 con 0x7f9fb8159830 2024-03-23T03:51:41.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.142+0000 7f9fbd000640 1 -- 172.21.15.102:0/1323260462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb81e4570 con 0x7f9fb8159830 2024-03-23T03:51:41.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.143+0000 7f9fae000640 1 -- 172.21.15.102:0/1323260462 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9fb001a400 con 0x7f9fb8159830 2024-03-23T03:51:41.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.144+0000 7f9fae000640 1 --2- 172.21.15.102:0/1323260462 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f9f9806ac00 0x7f9f9806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:41.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.144+0000 7f9fae000640 1 -- 172.21.15.102:0/1323260462 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9fb0089640 con 0x7f9fb8159830 2024-03-23T03:51:41.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.144+0000 7f9fb5e00640 1 --2- 172.21.15.102:0/1323260462 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f9f9806ac00 0x7f9f9806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:41.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.144+0000 7f9fb5e00640 1 --2- 172.21.15.102:0/1323260462 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f9f9806ac00 0x7f9f9806d0c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8035000 tx=0x7f9fa8030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:41.148 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.148+0000 7f9fae000640 1 -- 172.21.15.102:0/1323260462 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9fb81e4570 con 0x7f9fb8159830 2024-03-23T03:51:41.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.343+0000 7f9fbd000640 1 -- 172.21.15.102:0/1323260462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9fb81e4570 con 0x7f9fb8159830 2024-03-23T03:51:41.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.344+0000 7f9fae000640 1 -- 172.21.15.102:0/1323260462 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v105) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9fb81e4570 con 0x7f9fb8159830 2024-03-23T03:51:41.344 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:51:41.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.346+0000 7f9fbd000640 1 -- 172.21.15.102:0/1323260462 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f9f9806ac00 msgr2=0x7f9f9806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:41.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.346+0000 7f9fbd000640 1 --2- 172.21.15.102:0/1323260462 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f9f9806ac00 0x7f9f9806d0c0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f9fa8035000 tx=0x7f9fa8030000 comp rx=0 tx=0).stop 2024-03-23T03:51:41.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.346+0000 7f9fbd000640 1 -- 172.21.15.102:0/1323260462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fb8159830 msgr2=0x7f9fb81585b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:41.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.346+0000 7f9fbd000640 1 --2- 172.21.15.102:0/1323260462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fb8159830 0x7f9fb81585b0 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f9fb000b410 tx=0x7f9fb000b8d0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.346+0000 7f9fbd000640 1 -- 172.21.15.102:0/1323260462 shutdown_connections 2024-03-23T03:51:41.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.346+0000 7f9fbd000640 1 --2- 172.21.15.102:0/1323260462 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f9f9806ac00 0x7f9f9806d0c0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.346+0000 7f9fbd000640 1 --2- 172.21.15.102:0/1323260462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9fb8159830 0x7f9fb81585b0 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.346+0000 7f9fbd000640 1 --2- 172.21.15.102:0/1323260462 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9fb8070150 0x7f9fb8158070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.346+0000 7f9fbd000640 1 --2- 172.21.15.102:0/1323260462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9fb806e240 0x7f9fb814b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.346+0000 7f9fbd000640 1 -- 172.21.15.102:0/1323260462 >> 172.21.15.102:0/1323260462 conn(0x7f9fb806a0b0 msgr2=0x7f9fb80cbe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:41.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.346+0000 7f9fbd000640 1 -- 172.21.15.102:0/1323260462 shutdown_connections 2024-03-23T03:51:41.348 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.346+0000 7f9fbd000640 1 -- 172.21.15.102:0/1323260462 wait complete. 2024-03-23T03:51:41.364 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:51:41.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.574+0000 7f64ef200640 1 Processor -- start 2024-03-23T03:51:41.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.575+0000 7f64ef200640 1 -- start start 2024-03-23T03:51:41.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.576+0000 7f64ef200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f64f015c710 0x7f64f015caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:41.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.576+0000 7f64ef200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f64f0070730 0x7f64f00cc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:41.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.576+0000 7f64ef200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f015d100 con 0x7f64f00cda00 2024-03-23T03:51:41.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.576+0000 7f64ef200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f015d3e0 con 0x7f64f0070730 2024-03-23T03:51:41.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.576+0000 7f64ef200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f015d6c0 con 0x7f64f015c710 2024-03-23T03:51:41.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.577+0000 7f64ee800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f64f0070730 0x7f64f00cc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:41.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.577+0000 7f64ee800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f64f0070730 0x7f64f00cc5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48500/0 (socket says 172.21.15.102:48500) 2024-03-23T03:51:41.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.577+0000 7f64ee800640 1 -- 172.21.15.102:0/1884022793 learned_addr learned my addr 172.21.15.102:0/1884022793 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:41.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.577+0000 7f64ede00640 1 --2- 172.21.15.102:0/1884022793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f64f015c710 0x7f64f015caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:41.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.577+0000 7f64ee800640 1 -- 172.21.15.102:0/1884022793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f64f015c710 msgr2=0x7f64f015caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:41.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.577+0000 7f64ee800640 1 --2- 172.21.15.102:0/1884022793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f64f015c710 0x7f64f015caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.577+0000 7f64ee800640 1 -- 172.21.15.102:0/1884022793 >> v1:172.21.15.173:6789/0 conn(0x7f64f00cda00 legacy=0x7f64f0070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:41.578 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.577+0000 7f64ee800640 1 -- 172.21.15.102:0/1884022793 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f015d9a0 con 0x7f64f0070730 2024-03-23T03:51:41.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.578+0000 7f64ee800640 1 --2- 172.21.15.102:0/1884022793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f0070730 0x7f64f00cc5a0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f64e8009e50 tx=0x7f64e802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f094a733a07d54 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:41.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.579+0000 7f64ed400640 1 -- 172.21.15.102:0/1884022793 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e802ca70 con 0x7f64f0070730 2024-03-23T03:51:41.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.579+0000 7f64ed400640 1 -- 172.21.15.102:0/1884022793 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e802cbd0 con 0x7f64f0070730 2024-03-23T03:51:41.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.579+0000 7f64ed400640 1 -- 172.21.15.102:0/1884022793 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e80343f0 con 0x7f64f0070730 2024-03-23T03:51:41.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.579+0000 7f64ef200640 1 -- 172.21.15.102:0/1884022793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f0070730 msgr2=0x7f64f00cc5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:41.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.579+0000 7f64ef200640 1 --2- 172.21.15.102:0/1884022793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f0070730 0x7f64f00cc5a0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f64e8009e50 tx=0x7f64e802bed0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.579+0000 7f64ef200640 1 -- 172.21.15.102:0/1884022793 shutdown_connections 2024-03-23T03:51:41.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.579+0000 7f64ef200640 1 --2- 172.21.15.102:0/1884022793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f0070730 0x7f64f00cc5a0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.579+0000 7f64ef200640 1 --2- 172.21.15.102:0/1884022793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f64f015c710 0x7f64f015caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.579+0000 7f64ef200640 1 -- 172.21.15.102:0/1884022793 >> 172.21.15.102:0/1884022793 conn(0x7f64f0065e20 msgr2=0x7f64f0069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:41.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.579+0000 7f64ef200640 1 -- 172.21.15.102:0/1884022793 shutdown_connections 2024-03-23T03:51:41.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.579+0000 7f64ef200640 1 -- 172.21.15.102:0/1884022793 wait complete. 2024-03-23T03:51:41.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.581+0000 7f64ef200640 1 Processor -- start 2024-03-23T03:51:41.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.581+0000 7f64ef200640 1 -- start start 2024-03-23T03:51:41.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.582+0000 7f64ef200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f64f0070730 0x7f64f014f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:41.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.582+0000 7f64ef200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64f00cda00 0x7f64f015b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:41.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.582+0000 7f64ef200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f015c710 0x7f64f0157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:41.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.582+0000 7f64ef200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f01f55e0 con 0x7f64f015c710 2024-03-23T03:51:41.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.582+0000 7f64ef200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f01f58c0 con 0x7f64f00cda00 2024-03-23T03:51:41.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.582+0000 7f64ef200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f01f5ba0 con 0x7f64f0070730 2024-03-23T03:51:41.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.583+0000 7f64ee800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f015c710 0x7f64f0157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:41.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.583+0000 7f64ee800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f015c710 0x7f64f0157d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48504/0 (socket says 172.21.15.102:48504) 2024-03-23T03:51:41.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.583+0000 7f64ee800640 1 -- 172.21.15.102:0/2134425793 learned_addr learned my addr 172.21.15.102:0/2134425793 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:41.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.583+0000 7f64ede00640 1 --2- 172.21.15.102:0/2134425793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f64f0070730 0x7f64f014f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:41.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.583+0000 7f64e7e00640 1 --2- 172.21.15.102:0/2134425793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64f00cda00 0x7f64f015b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:41.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.583+0000 7f64ee800640 1 -- 172.21.15.102:0/2134425793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f64f0070730 msgr2=0x7f64f014f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:41.584 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.583+0000 7f64ee800640 1 --2- 172.21.15.102:0/2134425793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f64f0070730 0x7f64f014f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.583+0000 7f64ee800640 1 -- 172.21.15.102:0/2134425793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64f00cda00 msgr2=0x7f64f015b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:41.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.583+0000 7f64ee800640 1 --2- 172.21.15.102:0/2134425793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64f00cda00 0x7f64f015b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.583+0000 7f64ee800640 1 -- 172.21.15.102:0/2134425793 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e8009b30 con 0x7f64f015c710 2024-03-23T03:51:41.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.583+0000 7f64ede00640 1 --2- 172.21.15.102:0/2134425793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f64f0070730 0x7f64f014f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:51:41.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.584+0000 7f64ee800640 1 --2- 172.21.15.102:0/2134425793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f015c710 0x7f64f0157d70 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f64e8009e50 tx=0x7f64e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:41.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.584+0000 7f64e6000640 1 -- 172.21.15.102:0/2134425793 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e8009510 con 0x7f64f015c710 2024-03-23T03:51:41.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.584+0000 7f64e6000640 1 -- 172.21.15.102:0/2134425793 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e803e450 con 0x7f64f015c710 2024-03-23T03:51:41.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.584+0000 7f64e6000640 1 -- 172.21.15.102:0/2134425793 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e803e800 con 0x7f64f015c710 2024-03-23T03:51:41.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.584+0000 7f64ef200640 1 -- 172.21.15.102:0/2134425793 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f01f5e80 con 0x7f64f015c710 2024-03-23T03:51:41.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.584+0000 7f64ef200640 1 -- 172.21.15.102:0/2134425793 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f006cd10 con 0x7f64f015c710 2024-03-23T03:51:41.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.585+0000 7f64ef200640 1 -- 172.21.15.102:0/2134425793 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f006cea0 con 0x7f64f015c710 2024-03-23T03:51:41.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.587+0000 7f64e6000640 1 -- 172.21.15.102:0/2134425793 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f64e803e960 con 0x7f64f015c710 2024-03-23T03:51:41.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.588+0000 7f64e6000640 1 --2- 172.21.15.102:0/2134425793 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f64cc06ac30 0x7f64cc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:41.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.588+0000 7f64e6000640 1 -- 172.21.15.102:0/2134425793 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f64e80ac200 con 0x7f64f015c710 2024-03-23T03:51:41.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.588+0000 7f64ede00640 1 --2- 172.21.15.102:0/2134425793 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f64cc06ac30 0x7f64cc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:41.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.589+0000 7f64ede00640 1 --2- 172.21.15.102:0/2134425793 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f64cc06ac30 0x7f64cc06d0f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f64f0066bd0 tx=0x7f64e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:41.593 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.593+0000 7f64e6000640 1 -- 172.21.15.102:0/2134425793 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f64f006cea0 con 0x7f64f015c710 2024-03-23T03:51:41.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.804+0000 7f64ef200640 1 -- 172.21.15.102:0/2134425793 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f64f006cea0 con 0x7f64f015c710 2024-03-23T03:51:41.807 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.805+0000 7f64e6000640 1 -- 172.21.15.102:0/2134425793 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v105) v1 ==== 80+0+0 (secure 0 0 0) 0x7f64f006cea0 con 0x7f64f015c710 2024-03-23T03:51:41.807 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:51:41.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.808+0000 7f64ef200640 1 -- 172.21.15.102:0/2134425793 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f64cc06ac30 msgr2=0x7f64cc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:41.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.808+0000 7f64ef200640 1 --2- 172.21.15.102:0/2134425793 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f64cc06ac30 0x7f64cc06d0f0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f64f0066bd0 tx=0x7f64e0008040 comp rx=0 tx=0).stop 2024-03-23T03:51:41.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.808+0000 7f64ef200640 1 -- 172.21.15.102:0/2134425793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f015c710 msgr2=0x7f64f0157d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:41.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.808+0000 7f64ef200640 1 --2- 172.21.15.102:0/2134425793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f015c710 0x7f64f0157d70 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f64e8009e50 tx=0x7f64e8004a40 comp rx=0 tx=0).stop 2024-03-23T03:51:41.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.808+0000 7f64ef200640 1 -- 172.21.15.102:0/2134425793 shutdown_connections 2024-03-23T03:51:41.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.808+0000 7f64ef200640 1 --2- 172.21.15.102:0/2134425793 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f64cc06ac30 0x7f64cc06d0f0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.808+0000 7f64ef200640 1 --2- 172.21.15.102:0/2134425793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f64f015c710 0x7f64f0157d70 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.808+0000 7f64ef200640 1 --2- 172.21.15.102:0/2134425793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f64f00cda00 0x7f64f015b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.808+0000 7f64ef200640 1 --2- 172.21.15.102:0/2134425793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f64f0070730 0x7f64f014f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:41.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.808+0000 7f64ef200640 1 -- 172.21.15.102:0/2134425793 >> 172.21.15.102:0/2134425793 conn(0x7f64f0065e20 msgr2=0x7f64f00ae060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:41.809 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.808+0000 7f64ef200640 1 -- 172.21.15.102:0/2134425793 shutdown_connections 2024-03-23T03:51:41.810 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:41.808+0000 7f64ef200640 1 -- 172.21.15.102:0/2134425793 wait complete. 2024-03-23T03:51:41.826 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-23T03:51:41.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:41.826 DEBUG:teuthology.orchestra.run.smithi102:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-23T03:51:41.929 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:51:41.929 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:41.929 DEBUG:teuthology.orchestra.run.smithi102:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T03:51:41.978 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:51:41.978 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:42.036 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:51:42.036 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:42.092 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:51:42.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.304+0000 7f78dee00640 1 Processor -- start 2024-03-23T03:51:42.305 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.304+0000 7f78dee00640 1 -- start start 2024-03-23T03:51:42.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.305+0000 7f78dee00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78d8143930 0x7f78d80cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:42.306 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.305+0000 7f78dee00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f78d80cdc50 0x7f78d8070020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.305+0000 7f78dee00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d815b2b0 con 0x7f78d8143460 2024-03-23T03:51:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.305+0000 7f78dee00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d80cc6b0 con 0x7f78d80cdc50 2024-03-23T03:51:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.305+0000 7f78dee00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d80cc950 con 0x7f78d8143930 2024-03-23T03:51:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.305+0000 7f78de400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f78d80cdc50 0x7f78d8070020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.306+0000 7f78de400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f78d80cdc50 0x7f78d8070020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48508/0 (socket says 172.21.15.102:48508) 2024-03-23T03:51:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.306+0000 7f78de400640 1 -- 172.21.15.102:0/956002780 learned_addr learned my addr 172.21.15.102:0/956002780 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.306+0000 7f78dd000640 1 --2- 172.21.15.102:0/956002780 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78d8143930 0x7f78d80cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.306+0000 7f78de400640 1 -- 172.21.15.102:0/956002780 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78d8143930 msgr2=0x7f78d80cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.306+0000 7f78de400640 1 --2- 172.21.15.102:0/956002780 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78d8143930 0x7f78d80cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.306+0000 7f78de400640 1 -- 172.21.15.102:0/956002780 >> v1:172.21.15.102:6789/0 conn(0x7f78d8143460 legacy=0x7f78d815aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.306+0000 7f78de400640 1 -- 172.21.15.102:0/956002780 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d80ccc30 con 0x7f78d80cdc50 2024-03-23T03:51:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.306+0000 7f78dd000640 1 --2- 172.21.15.102:0/956002780 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78d8143930 0x7f78d80cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:42.307 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.306+0000 7f78d7e00640 1 -- 172.21.15.102:0/956002780 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576228090 0 0) 0x7f78d815b2b0 con 0x7f78d8143460 2024-03-23T03:51:42.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.307+0000 7f78de400640 1 --2- 172.21.15.102:0/956002780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78d80cdc50 0x7f78d8070020 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f78d0009870 tx=0x7f78d002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5dde92f807c7427 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:42.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.307+0000 7f78d7e00640 1 -- 172.21.15.102:0/956002780 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d002f830 con 0x7f78d80cdc50 2024-03-23T03:51:42.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.307+0000 7f78d7e00640 1 -- 172.21.15.102:0/956002780 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78d002f990 con 0x7f78d80cdc50 2024-03-23T03:51:42.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.307+0000 7f78d7e00640 1 -- 172.21.15.102:0/956002780 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78d002fd20 con 0x7f78d80cdc50 2024-03-23T03:51:42.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.308+0000 7f78dee00640 1 -- 172.21.15.102:0/956002780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78d80cdc50 msgr2=0x7f78d8070020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:42.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.308+0000 7f78dee00640 1 --2- 172.21.15.102:0/956002780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78d80cdc50 0x7f78d8070020 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f78d0009870 tx=0x7f78d002ba10 comp rx=0 tx=0).stop 2024-03-23T03:51:42.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.308+0000 7f78dee00640 1 -- 172.21.15.102:0/956002780 shutdown_connections 2024-03-23T03:51:42.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.308+0000 7f78dee00640 1 --2- 172.21.15.102:0/956002780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78d80cdc50 0x7f78d8070020 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.308+0000 7f78dee00640 1 --2- 172.21.15.102:0/956002780 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78d8143930 0x7f78d80cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.308+0000 7f78dee00640 1 -- 172.21.15.102:0/956002780 >> 172.21.15.102:0/956002780 conn(0x7f78d8065df0 msgr2=0x7f78d8069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:42.309 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.308+0000 7f78dee00640 1 -- 172.21.15.102:0/956002780 shutdown_connections 2024-03-23T03:51:42.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.308+0000 7f78dee00640 1 -- 172.21.15.102:0/956002780 wait complete. 2024-03-23T03:51:42.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.310+0000 7f78dee00640 1 Processor -- start 2024-03-23T03:51:42.310 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.310+0000 7f78dee00640 1 -- start start 2024-03-23T03:51:42.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.311+0000 7f78dee00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78d80cdc50 0x7f78d80b0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:42.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.311+0000 7f78dee00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78d8143460 0x7f78d813d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:42.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.311+0000 7f78dee00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78d8143930 0x7f78d813da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:42.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.311+0000 7f78dee00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d80b00c0 con 0x7f78d80cdc50 2024-03-23T03:51:42.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.311+0000 7f78dee00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d80af4b0 con 0x7f78d8143930 2024-03-23T03:51:42.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.311+0000 7f78dee00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d80af790 con 0x7f78d8143460 2024-03-23T03:51:42.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.311+0000 7f78dda00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78d80cdc50 0x7f78d80b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:42.313 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.311+0000 7f78dda00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78d80cdc50 0x7f78d80b0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48522/0 (socket says 172.21.15.102:48522) 2024-03-23T03:51:42.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.311+0000 7f78dda00640 1 -- 172.21.15.102:0/2721221788 learned_addr learned my addr 172.21.15.102:0/2721221788 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:42.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.312+0000 7f78dd000640 1 --2- 172.21.15.102:0/2721221788 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78d8143460 0x7f78d813d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:42.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.312+0000 7f78dda00640 1 -- 172.21.15.102:0/2721221788 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78d8143460 msgr2=0x7f78d813d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:42.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.312+0000 7f78de400640 1 --2- 172.21.15.102:0/2721221788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78d8143930 0x7f78d813da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:42.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.312+0000 7f78dda00640 1 --2- 172.21.15.102:0/2721221788 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78d8143460 0x7f78d813d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.312+0000 7f78dda00640 1 -- 172.21.15.102:0/2721221788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78d8143930 msgr2=0x7f78d813da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:42.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.312+0000 7f78dda00640 1 --2- 172.21.15.102:0/2721221788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78d8143930 0x7f78d813da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.312+0000 7f78dda00640 1 -- 172.21.15.102:0/2721221788 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d00094d0 con 0x7f78d80cdc50 2024-03-23T03:51:42.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.312+0000 7f78dd000640 1 --2- 172.21.15.102:0/2721221788 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78d8143460 0x7f78d813d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:42.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.312+0000 7f78dda00640 1 --2- 172.21.15.102:0/2721221788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78d80cdc50 0x7f78d80b0880 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f78cc00c890 tx=0x7f78cc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:42.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.312+0000 7f78d5600640 1 -- 172.21.15.102:0/2721221788 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78cc003da0 con 0x7f78d80cdc50 2024-03-23T03:51:42.314 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.312+0000 7f78d5600640 1 -- 172.21.15.102:0/2721221788 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78cc005070 con 0x7f78d80cdc50 2024-03-23T03:51:42.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.312+0000 7f78d5600640 1 -- 172.21.15.102:0/2721221788 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78cc005420 con 0x7f78d80cdc50 2024-03-23T03:51:42.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.312+0000 7f78dee00640 1 -- 172.21.15.102:0/2721221788 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d80afa70 con 0x7f78d80cdc50 2024-03-23T03:51:42.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.312+0000 7f78dee00640 1 -- 172.21.15.102:0/2721221788 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d80af130 con 0x7f78d80cdc50 2024-03-23T03:51:42.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.314+0000 7f78dee00640 1 -- 172.21.15.102:0/2721221788 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d804e030 con 0x7f78d80cdc50 2024-03-23T03:51:42.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.315+0000 7f78d5600640 1 -- 172.21.15.102:0/2721221788 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f78cc005580 con 0x7f78d80cdc50 2024-03-23T03:51:42.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.316+0000 7f78d5600640 1 --2- 172.21.15.102:0/2721221788 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78b806ac00 0x7f78b806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:42.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.316+0000 7f78d5600640 1 -- 172.21.15.102:0/2721221788 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f78cc088e10 con 0x7f78d80cdc50 2024-03-23T03:51:42.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.316+0000 7f78dd000640 1 --2- 172.21.15.102:0/2721221788 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78b806ac00 0x7f78b806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:42.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.317+0000 7f78dd000640 1 --2- 172.21.15.102:0/2721221788 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78b806ac00 0x7f78b806d0c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f78c4006fd0 tx=0x7f78c4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:42.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.320+0000 7f78d5600640 1 -- 172.21.15.102:0/2721221788 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f78cc08c030 con 0x7f78d80cdc50 2024-03-23T03:51:42.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.513+0000 7f78dee00640 1 -- 172.21.15.102:0/2721221788 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78d8141c60 con 0x7f78d80cdc50 2024-03-23T03:51:42.516 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.516+0000 7f78d5600640 1 -- 172.21.15.102:0/2721221788 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+16296 (secure 0 0 0) 0x7f78cc017070 con 0x7f78d80cdc50 2024-03-23T03:51:42.517 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:42.517 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":105,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:51:23.233227+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:51:19.796618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-23T03:51:21.283447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:51:42.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.518+0000 7f78dee00640 1 -- 172.21.15.102:0/2721221788 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78b806ac00 msgr2=0x7f78b806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:42.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.518+0000 7f78dee00640 1 --2- 172.21.15.102:0/2721221788 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78b806ac00 0x7f78b806d0c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f78c4006fd0 tx=0x7f78c4008040 comp rx=0 tx=0).stop 2024-03-23T03:51:42.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.518+0000 7f78dee00640 1 -- 172.21.15.102:0/2721221788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78d80cdc50 msgr2=0x7f78d80b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:42.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.518+0000 7f78dee00640 1 --2- 172.21.15.102:0/2721221788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78d80cdc50 0x7f78d80b0880 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f78cc00c890 tx=0x7f78cc00cd50 comp rx=0 tx=0).stop 2024-03-23T03:51:42.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.518+0000 7f78dee00640 1 -- 172.21.15.102:0/2721221788 shutdown_connections 2024-03-23T03:51:42.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.518+0000 7f78dee00640 1 --2- 172.21.15.102:0/2721221788 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f78b806ac00 0x7f78b806d0c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.518+0000 7f78dee00640 1 --2- 172.21.15.102:0/2721221788 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78d8143930 0x7f78d813da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.518+0000 7f78dee00640 1 --2- 172.21.15.102:0/2721221788 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f78d8143460 0x7f78d813d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.518+0000 7f78dee00640 1 --2- 172.21.15.102:0/2721221788 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f78d80cdc50 0x7f78d80b0880 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.519 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.518+0000 7f78dee00640 1 -- 172.21.15.102:0/2721221788 >> 172.21.15.102:0/2721221788 conn(0x7f78d8065df0 msgr2=0x7f78d813e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:42.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.518+0000 7f78dee00640 1 -- 172.21.15.102:0/2721221788 shutdown_connections 2024-03-23T03:51:42.520 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.519+0000 7f78dee00640 1 -- 172.21.15.102:0/2721221788 wait complete. 2024-03-23T03:51:42.538 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:51:19.796618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:51:21.283447+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:51:42.539 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:51:42.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.751+0000 7fcf2ca00640 1 Processor -- start 2024-03-23T03:51:42.752 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.751+0000 7fcf2ca00640 1 -- start start 2024-03-23T03:51:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.752+0000 7fcf2ca00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcf281588b0 0x7fcf28070150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.753+0000 7fcf2ca00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf2815cc00 0x7fcf28070690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.753+0000 7fcf2ca00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf2815d220 con 0x7fcf28158c90 2024-03-23T03:51:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.753+0000 7fcf2ca00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2815d040 con 0x7fcf281588b0 2024-03-23T03:51:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.753+0000 7fcf2ca00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf280b25d0 con 0x7fcf2815cc00 2024-03-23T03:51:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.753+0000 7fcf25e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcf281588b0 0x7fcf28070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.753+0000 7fcf25e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fcf281588b0 0x7fcf28070150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48536/0 (socket says 172.21.15.102:48536) 2024-03-23T03:51:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.753+0000 7fcf25e00640 1 -- 172.21.15.102:0/377330218 learned_addr learned my addr 172.21.15.102:0/377330218 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.753+0000 7fcf25e00640 1 -- 172.21.15.102:0/377330218 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf2815cc00 msgr2=0x7fcf28070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.753+0000 7fcf25e00640 1 --2- 172.21.15.102:0/377330218 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf2815cc00 0x7fcf28070690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.753+0000 7fcf25e00640 1 -- 172.21.15.102:0/377330218 >> v1:172.21.15.173:6789/0 conn(0x7fcf28158c90 legacy=0x7fcf2815c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:42.754 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.753+0000 7fcf25e00640 1 -- 172.21.15.102:0/377330218 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf281591c0 con 0x7fcf281588b0 2024-03-23T03:51:42.755 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.754+0000 7fcf25e00640 1 --2- 172.21.15.102:0/377330218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf281588b0 0x7fcf28070150 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fcf280ce470 tx=0x7fcf1802b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=303a9edef03ebe3b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:42.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.757+0000 7fcf24a00640 1 -- 172.21.15.102:0/377330218 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18031710 con 0x7fcf281588b0 2024-03-23T03:51:42.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.757+0000 7fcf24a00640 1 -- 172.21.15.102:0/377330218 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1802f050 con 0x7fcf281588b0 2024-03-23T03:51:42.758 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.757+0000 7fcf24a00640 1 -- 172.21.15.102:0/377330218 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18031a90 con 0x7fcf281588b0 2024-03-23T03:51:42.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.758+0000 7fcf2ca00640 1 -- 172.21.15.102:0/377330218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf281588b0 msgr2=0x7fcf28070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:42.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.758+0000 7fcf2ca00640 1 --2- 172.21.15.102:0/377330218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf281588b0 0x7fcf28070150 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7fcf280ce470 tx=0x7fcf1802b7b0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.758+0000 7fcf2ca00640 1 -- 172.21.15.102:0/377330218 shutdown_connections 2024-03-23T03:51:42.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.758+0000 7fcf2ca00640 1 --2- 172.21.15.102:0/377330218 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf2815cc00 0x7fcf28070690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.758+0000 7fcf2ca00640 1 --2- 172.21.15.102:0/377330218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf281588b0 0x7fcf28070150 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.758+0000 7fcf2ca00640 1 -- 172.21.15.102:0/377330218 >> 172.21.15.102:0/377330218 conn(0x7fcf28069c90 msgr2=0x7fcf2806a070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:42.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.758+0000 7fcf2ca00640 1 -- 172.21.15.102:0/377330218 shutdown_connections 2024-03-23T03:51:42.759 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.759+0000 7fcf2ca00640 1 -- 172.21.15.102:0/377330218 wait complete. 2024-03-23T03:51:42.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.760+0000 7fcf2ca00640 1 Processor -- start 2024-03-23T03:51:42.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.760+0000 7fcf2ca00640 1 -- start start 2024-03-23T03:51:42.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.761+0000 7fcf2ca00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf281588b0 0x7fcf2814da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:42.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.761+0000 7fcf2ca00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf28158c90 0x7fcf28146790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:42.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.761+0000 7fcf2ca00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf2815cc00 0x7fcf2814a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:42.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.761+0000 7fcf2ca00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf28158140 con 0x7fcf2815cc00 2024-03-23T03:51:42.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.761+0000 7fcf2ca00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2814aa40 con 0x7fcf281588b0 2024-03-23T03:51:42.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.761+0000 7fcf2ca00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf28157b80 con 0x7fcf28158c90 2024-03-23T03:51:42.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.762+0000 7fcf26800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf2815cc00 0x7fcf2814a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:42.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.762+0000 7fcf26800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf2815cc00 0x7fcf2814a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48540/0 (socket says 172.21.15.102:48540) 2024-03-23T03:51:42.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.762+0000 7fcf26800640 1 -- 172.21.15.102:0/3802027671 learned_addr learned my addr 172.21.15.102:0/3802027671 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:42.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.762+0000 7fcf25e00640 1 --2- 172.21.15.102:0/3802027671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf281588b0 0x7fcf2814da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:42.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.762+0000 7fcf26800640 1 -- 172.21.15.102:0/3802027671 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf28158c90 msgr2=0x7fcf28146790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:42.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.762+0000 7fcf26800640 1 --2- 172.21.15.102:0/3802027671 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf28158c90 0x7fcf28146790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.762+0000 7fcf26800640 1 -- 172.21.15.102:0/3802027671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf281588b0 msgr2=0x7fcf2814da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:42.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.762+0000 7fcf26800640 1 --2- 172.21.15.102:0/3802027671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf281588b0 0x7fcf2814da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.762+0000 7fcf26800640 1 -- 172.21.15.102:0/3802027671 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf180094d0 con 0x7fcf2815cc00 2024-03-23T03:51:42.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.762+0000 7fcf25e00640 1 --2- 172.21.15.102:0/3802027671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf281588b0 0x7fcf2814da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:42.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.763+0000 7fcf26800640 1 --2- 172.21.15.102:0/3802027671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf2815cc00 0x7fcf2814a500 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fcf2000ba10 tx=0x7fcf2000bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:42.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.763+0000 7fcf1e000640 1 -- 172.21.15.102:0/3802027671 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf2000cac0 con 0x7fcf2815cc00 2024-03-23T03:51:42.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.763+0000 7fcf1e000640 1 -- 172.21.15.102:0/3802027671 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2000cc40 con 0x7fcf2815cc00 2024-03-23T03:51:42.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.763+0000 7fcf1e000640 1 -- 172.21.15.102:0/3802027671 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf20014440 con 0x7fcf2815cc00 2024-03-23T03:51:42.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.763+0000 7fcf2ca00640 1 -- 172.21.15.102:0/3802027671 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf28159e20 con 0x7fcf2815cc00 2024-03-23T03:51:42.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.763+0000 7fcf2ca00640 1 -- 172.21.15.102:0/3802027671 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf28165b50 con 0x7fcf2815cc00 2024-03-23T03:51:42.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.764+0000 7fcf2ca00640 1 -- 172.21.15.102:0/3802027671 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2806a070 con 0x7fcf2815cc00 2024-03-23T03:51:42.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.765+0000 7fcf1e000640 1 -- 172.21.15.102:0/3802027671 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fcf2001d3b0 con 0x7fcf2815cc00 2024-03-23T03:51:42.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.766+0000 7fcf1e000640 1 --2- 172.21.15.102:0/3802027671 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcf0806ac00 0x7fcf0806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:42.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.766+0000 7fcf1e000640 1 -- 172.21.15.102:0/3802027671 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6116+0+0 (secure 0 0 0) 0x7fcf20089310 con 0x7fcf2815cc00 2024-03-23T03:51:42.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.766+0000 7fcf25e00640 1 --2- 172.21.15.102:0/3802027671 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcf0806ac00 0x7fcf0806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:42.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.767+0000 7fcf25e00640 1 --2- 172.21.15.102:0/3802027671 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcf0806ac00 0x7fcf0806d0c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fcf18035000 tx=0x7fcf18008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:42.771 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.771+0000 7fcf1e000640 1 -- 172.21.15.102:0/3802027671 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fcf20057e70 con 0x7fcf2815cc00 2024-03-23T03:51:42.966 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.965+0000 7fcf2ca00640 1 -- 172.21.15.102:0/3802027671 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcf28073390 con 0x7fcf2815cc00 2024-03-23T03:51:42.967 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.966+0000 7fcf1e000640 1 -- 172.21.15.102:0/3802027671 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf20057a00 con 0x7fcf2815cc00 2024-03-23T03:51:42.967 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:51:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.969+0000 7fcf2ca00640 1 -- 172.21.15.102:0/3802027671 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcf0806ac00 msgr2=0x7fcf0806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.969+0000 7fcf2ca00640 1 --2- 172.21.15.102:0/3802027671 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcf0806ac00 0x7fcf0806d0c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fcf18035000 tx=0x7fcf18008030 comp rx=0 tx=0).stop 2024-03-23T03:51:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.969+0000 7fcf2ca00640 1 -- 172.21.15.102:0/3802027671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf2815cc00 msgr2=0x7fcf2814a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.969+0000 7fcf2ca00640 1 --2- 172.21.15.102:0/3802027671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf2815cc00 0x7fcf2814a500 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7fcf2000ba10 tx=0x7fcf2000bed0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.969+0000 7fcf2ca00640 1 -- 172.21.15.102:0/3802027671 shutdown_connections 2024-03-23T03:51:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.969+0000 7fcf2ca00640 1 --2- 172.21.15.102:0/3802027671 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fcf0806ac00 0x7fcf0806d0c0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.969+0000 7fcf2ca00640 1 --2- 172.21.15.102:0/3802027671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf2815cc00 0x7fcf2814a500 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.970 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.969+0000 7fcf2ca00640 1 --2- 172.21.15.102:0/3802027671 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fcf28158c90 0x7fcf28146790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.969+0000 7fcf2ca00640 1 --2- 172.21.15.102:0/3802027671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcf281588b0 0x7fcf2814da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:42.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.969+0000 7fcf2ca00640 1 -- 172.21.15.102:0/3802027671 >> 172.21.15.102:0/3802027671 conn(0x7fcf28069c90 msgr2=0x7fcf2815b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:42.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.969+0000 7fcf2ca00640 1 -- 172.21.15.102:0/3802027671 shutdown_connections 2024-03-23T03:51:42.971 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:42.969+0000 7fcf2ca00640 1 -- 172.21.15.102:0/3802027671 wait complete. 2024-03-23T03:51:42.987 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:51:43.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.195+0000 7fec0be00640 1 Processor -- start 2024-03-23T03:51:43.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.196+0000 7fec0be00640 1 -- start start 2024-03-23T03:51:43.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.197+0000 7fec0be00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec0c15c820 0x7fec0c0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:43.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.197+0000 7fec0be00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec0c0ccd90 con 0x7fec0c0cf7c0 2024-03-23T03:51:43.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.197+0000 7fec0be00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec0c0b7fc0 con 0x7fec0c160e90 2024-03-23T03:51:43.197 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.197+0000 7fec0be00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c0cdd70 con 0x7fec0c15c820 2024-03-23T03:51:43.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.197+0000 7fec0b400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fec0c0cf7c0 0x7fec0c0d0560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55020/0 (socket says 172.21.15.102:55020) 2024-03-23T03:51:43.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.197+0000 7fec0b400640 1 -- 172.21.15.102:0/3529880855 learned_addr learned my addr 172.21.15.102:0/3529880855 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:43.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.198+0000 7fec0aa00640 1 --2- 172.21.15.102:0/3529880855 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec0c15c820 0x7fec0c0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:43.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.198+0000 7fec09600640 1 -- 172.21.15.102:0/3529880855 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024563632 0 0) 0x7fec0c0ccd90 con 0x7fec0c0cf7c0 2024-03-23T03:51:43.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.198+0000 7fec09600640 1 -- 172.21.15.102:0/3529880855 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febf0003520 con 0x7fec0c0cf7c0 2024-03-23T03:51:43.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.198+0000 7fec09600640 1 -- 172.21.15.102:0/3529880855 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3440197765 0 0) 0x7febf0003520 con 0x7fec0c0cf7c0 2024-03-23T03:51:43.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.198+0000 7fec09600640 1 -- 172.21.15.102:0/3529880855 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec0c15c820 msgr2=0x7fec0c0cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:43.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.198+0000 7fec09600640 1 --2- 172.21.15.102:0/3529880855 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec0c15c820 0x7fec0c0cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.198+0000 7fec09600640 1 -- 172.21.15.102:0/3529880855 >> v1:172.21.15.173:6789/0 conn(0x7fec0c160e90 legacy=0x7fec0c0cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.198+0000 7fec09600640 1 -- 172.21.15.102:0/3529880855 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c0b82a0 con 0x7fec0c0cf7c0 2024-03-23T03:51:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.198+0000 7fec09600640 1 -- 172.21.15.102:0/3529880855 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fec040019f0 con 0x7fec0c0cf7c0 2024-03-23T03:51:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.198+0000 7fec09600640 1 -- 172.21.15.102:0/3529880855 >> v1:172.21.15.102:6789/0 conn(0x7fec0c0cf7c0 legacy=0x7fec0c0d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.198+0000 7fec09600640 1 --2- 172.21.15.102:0/3529880855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febf00039a0 0x7febf0003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.198+0000 7fec09600640 1 --2- 172.21.15.102:0/3529880855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf0004350 0x7febf0004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.198+0000 7fec09600640 1 --2- 172.21.15.102:0/3529880855 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec0c15c820 0x7febf0005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.199+0000 7fec09600640 1 -- 172.21.15.102:0/3529880855 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c0cdd70 con 0x7febf0004350 2024-03-23T03:51:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.199+0000 7fec09600640 1 -- 172.21.15.102:0/3529880855 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf000b980 con 0x7febf00039a0 2024-03-23T03:51:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.199+0000 7fec09600640 1 -- 172.21.15.102:0/3529880855 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf000bb40 con 0x7fec0c15c820 2024-03-23T03:51:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.199+0000 7fec0a000640 1 --2- 172.21.15.102:0/3529880855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf0004350 0x7febf0004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.199+0000 7fec0aa00640 1 --2- 172.21.15.102:0/3529880855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febf00039a0 0x7febf0003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.199+0000 7fec0a000640 1 -- 172.21.15.102:0/3529880855 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec0c15c820 msgr2=0x7febf0005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.199+0000 7fec0a000640 1 --2- 172.21.15.102:0/3529880855 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec0c15c820 0x7febf0005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.201 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.199+0000 7fec0a000640 1 -- 172.21.15.102:0/3529880855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febf00039a0 msgr2=0x7febf0003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:43.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.199+0000 7fec0a000640 1 --2- 172.21.15.102:0/3529880855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febf00039a0 0x7febf0003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.199+0000 7fec0a000640 1 -- 172.21.15.102:0/3529880855 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf0003520 con 0x7febf0004350 2024-03-23T03:51:43.202 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.199+0000 7fec0aa00640 1 --2- 172.21.15.102:0/3529880855 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7febf00039a0 0x7febf0003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:43.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.200+0000 7fec0a000640 1 --2- 172.21.15.102:0/3529880855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf0004350 0x7febf0004820 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7febf8001860 tx=0x7febf00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db90e80c9eb9b06 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:43.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.200+0000 7fec09600640 1 -- 172.21.15.102:0/3529880855 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8033410 con 0x7febf0004350 2024-03-23T03:51:43.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.201+0000 7fec09600640 1 -- 172.21.15.102:0/3529880855 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf8033590 con 0x7febf0004350 2024-03-23T03:51:43.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.201+0000 7fec09600640 1 -- 172.21.15.102:0/3529880855 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8033940 con 0x7febf0004350 2024-03-23T03:51:43.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.201+0000 7fec0be00640 1 -- 172.21.15.102:0/3529880855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf0004350 msgr2=0x7febf0004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:43.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.201+0000 7fec0be00640 1 --2- 172.21.15.102:0/3529880855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf0004350 0x7febf0004820 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7febf8001860 tx=0x7febf00057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.201+0000 7fec0b400640 1 -- 172.21.15.102:0/3529880855 reap_dead start 2024-03-23T03:51:43.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.201+0000 7fec0be00640 1 -- 172.21.15.102:0/3529880855 shutdown_connections 2024-03-23T03:51:43.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.201+0000 7fec0be00640 1 -- 172.21.15.102:0/3529880855 >> 172.21.15.102:0/3529880855 conn(0x7fec0c06b050 msgr2=0x7fec0c06b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:43.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.201+0000 7fec0be00640 1 -- 172.21.15.102:0/3529880855 shutdown_connections 2024-03-23T03:51:43.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.201+0000 7fec0be00640 1 -- 172.21.15.102:0/3529880855 wait complete. 2024-03-23T03:51:43.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.203+0000 7fec0be00640 1 Processor -- start 2024-03-23T03:51:43.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.203+0000 7fec0be00640 1 -- start start 2024-03-23T03:51:43.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.204+0000 7fec0be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf0004350 0x7fec0c1f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:43.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.204+0000 7fec0be00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec0c15db60 0x7fec0c0cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:43.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.204+0000 7fec0be00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec0c15dfe0 0x7fec0c0cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:43.206 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.204+0000 7fec0be00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c15f780 con 0x7febf0004350 2024-03-23T03:51:43.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.204+0000 7fec0be00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c15cda0 con 0x7fec0c15db60 2024-03-23T03:51:43.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.204+0000 7fec0be00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0c15d080 con 0x7fec0c15dfe0 2024-03-23T03:51:43.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.204+0000 7fec0aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf0004350 0x7fec0c1f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:43.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.205+0000 7fec0aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf0004350 0x7fec0c1f1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48824/0 (socket says 172.21.15.102:48824) 2024-03-23T03:51:43.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.205+0000 7fec0aa00640 1 -- 172.21.15.102:0/3093870729 learned_addr learned my addr 172.21.15.102:0/3093870729 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:43.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.205+0000 7fec0aa00640 1 -- 172.21.15.102:0/3093870729 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec0c15dfe0 msgr2=0x7fec0c0cbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:43.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.205+0000 7fec0aa00640 1 --2- 172.21.15.102:0/3093870729 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec0c15dfe0 0x7fec0c0cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.205+0000 7fec0a000640 1 --2- 172.21.15.102:0/3093870729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec0c15db60 0x7fec0c0cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:43.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.205+0000 7fec0aa00640 1 -- 172.21.15.102:0/3093870729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec0c15db60 msgr2=0x7fec0c0cb990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:43.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.205+0000 7fec0aa00640 1 --2- 172.21.15.102:0/3093870729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec0c15db60 0x7fec0c0cb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.205+0000 7fec0aa00640 1 -- 172.21.15.102:0/3093870729 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c15d360 con 0x7febf0004350 2024-03-23T03:51:43.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.205+0000 7fec0aa00640 1 --2- 172.21.15.102:0/3093870729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf0004350 0x7fec0c1f1af0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7febfc0108b0 tx=0x7febfc010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:43.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.205+0000 7fec02a00640 1 -- 172.21.15.102:0/3093870729 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc00eb60 con 0x7febf0004350 2024-03-23T03:51:43.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.206+0000 7fec02a00640 1 -- 172.21.15.102:0/3093870729 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febfc00ecc0 con 0x7febf0004350 2024-03-23T03:51:43.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.206+0000 7fec0be00640 1 -- 172.21.15.102:0/3093870729 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec0c15d640 con 0x7febf0004350 2024-03-23T03:51:43.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.206+0000 7fec02a00640 1 -- 172.21.15.102:0/3093870729 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc01c400 con 0x7febf0004350 2024-03-23T03:51:43.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.206+0000 7fec0be00640 1 -- 172.21.15.102:0/3093870729 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec0c15ea50 con 0x7febf0004350 2024-03-23T03:51:43.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.207+0000 7fec0be00640 1 -- 172.21.15.102:0/3093870729 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec0c04e030 con 0x7febf0004350 2024-03-23T03:51:43.208 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.208+0000 7fec02a00640 1 -- 172.21.15.102:0/3093870729 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7febfc01c560 con 0x7febf0004350 2024-03-23T03:51:43.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.209+0000 7fec02a00640 1 --2- 172.21.15.102:0/3093870729 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7febec06ac30 0x7febec06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:43.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.209+0000 7fec02a00640 1 -- 172.21.15.102:0/3093870729 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6116+0+0 (secure 0 0 0) 0x7febfc089b90 con 0x7febf0004350 2024-03-23T03:51:43.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.209+0000 7fec0a000640 1 --2- 172.21.15.102:0/3093870729 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7febec06ac30 0x7febec06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:43.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.210+0000 7fec0a000640 1 --2- 172.21.15.102:0/3093870729 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7febec06ac30 0x7febec06d0f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7febf8001860 tx=0x7febf8009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:43.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.213+0000 7fec02a00640 1 -- 172.21.15.102:0/3093870729 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7febfc054be0 con 0x7febf0004350 2024-03-23T03:51:43.410 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.410+0000 7fec0be00640 1 -- 172.21.15.102:0/3093870729 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fec0c0d15a0 con 0x7febf0004350 2024-03-23T03:51:43.411 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.411+0000 7fec02a00640 1 -- 172.21.15.102:0/3093870729 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) v1 ==== 104+0+11 (secure 0 0 0) 0x7febfc058200 con 0x7febf0004350 2024-03-23T03:51:43.411 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:51:43.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.413+0000 7fec0be00640 1 -- 172.21.15.102:0/3093870729 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7febec06ac30 msgr2=0x7febec06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:43.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.413+0000 7fec0be00640 1 --2- 172.21.15.102:0/3093870729 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7febec06ac30 0x7febec06d0f0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7febf8001860 tx=0x7febf8009030 comp rx=0 tx=0).stop 2024-03-23T03:51:43.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.413+0000 7fec0be00640 1 -- 172.21.15.102:0/3093870729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf0004350 msgr2=0x7fec0c1f1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:43.414 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.413+0000 7fec0be00640 1 --2- 172.21.15.102:0/3093870729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf0004350 0x7fec0c1f1af0 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7febfc0108b0 tx=0x7febfc010d70 comp rx=0 tx=0).stop 2024-03-23T03:51:43.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.413+0000 7fec0be00640 1 -- 172.21.15.102:0/3093870729 shutdown_connections 2024-03-23T03:51:43.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.413+0000 7fec0be00640 1 --2- 172.21.15.102:0/3093870729 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7febec06ac30 0x7febec06d0f0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.413+0000 7fec0be00640 1 --2- 172.21.15.102:0/3093870729 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fec0c15dfe0 0x7fec0c0cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.413+0000 7fec0be00640 1 --2- 172.21.15.102:0/3093870729 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fec0c15db60 0x7fec0c0cb990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.413+0000 7fec0be00640 1 --2- 172.21.15.102:0/3093870729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febf0004350 0x7fec0c1f1af0 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.413+0000 7fec0be00640 1 -- 172.21.15.102:0/3093870729 >> 172.21.15.102:0/3093870729 conn(0x7fec0c06b050 msgr2=0x7fec0c1603f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:43.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.414+0000 7fec0be00640 1 -- 172.21.15.102:0/3093870729 shutdown_connections 2024-03-23T03:51:43.415 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.414+0000 7fec0be00640 1 -- 172.21.15.102:0/3093870729 wait complete. 2024-03-23T03:51:43.432 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:51:43.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.646+0000 7f7c6ac00640 1 Processor -- start 2024-03-23T03:51:43.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.646+0000 7f7c6ac00640 1 -- start start 2024-03-23T03:51:43.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.648+0000 7f7c6ac00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c64158db0 0x7f7c6415d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:43.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.648+0000 7f7c6ac00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c6415d870 con 0x7f7c640cd820 2024-03-23T03:51:43.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.648+0000 7f7c6ac00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c640caec0 con 0x7f7c64070150 2024-03-23T03:51:43.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.648+0000 7f7c6ac00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6406d0b0 con 0x7f7c64158db0 2024-03-23T03:51:43.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.648+0000 7f7c68c00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f7c640cd820 0x7f7c6406c9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55032/0 (socket says 172.21.15.102:55032) 2024-03-23T03:51:43.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.648+0000 7f7c68c00640 1 -- 172.21.15.102:0/3378344530 learned_addr learned my addr 172.21.15.102:0/3378344530 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:43.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.648+0000 7f7c63e00640 1 --2- 172.21.15.102:0/3378344530 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c64158db0 0x7f7c6415d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:43.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.648+0000 7f7c62a00640 1 -- 172.21.15.102:0/3378344530 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3030105142 0 0) 0x7f7c6415d870 con 0x7f7c640cd820 2024-03-23T03:51:43.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.649+0000 7f7c62a00640 1 -- 172.21.15.102:0/3378344530 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c44003440 con 0x7f7c640cd820 2024-03-23T03:51:43.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.649+0000 7f7c62a00640 1 -- 172.21.15.102:0/3378344530 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 730098503 0 0) 0x7f7c44003440 con 0x7f7c640cd820 2024-03-23T03:51:43.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.649+0000 7f7c62a00640 1 -- 172.21.15.102:0/3378344530 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c64158db0 msgr2=0x7f7c6415d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:43.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.649+0000 7f7c62a00640 1 --2- 172.21.15.102:0/3378344530 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c64158db0 0x7f7c6415d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.649+0000 7f7c62a00640 1 -- 172.21.15.102:0/3378344530 >> v1:172.21.15.173:6789/0 conn(0x7f7c64070150 legacy=0x7f7c6406c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:43.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.649+0000 7f7c62a00640 1 -- 172.21.15.102:0/3378344530 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c640cba90 con 0x7f7c640cd820 2024-03-23T03:51:43.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.649+0000 7f7c62a00640 1 -- 172.21.15.102:0/3378344530 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f7c5c001670 con 0x7f7c640cd820 2024-03-23T03:51:43.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.649+0000 7f7c62a00640 1 -- 172.21.15.102:0/3378344530 >> v1:172.21.15.102:6789/0 conn(0x7f7c640cd820 legacy=0x7f7c6406c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:43.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.649+0000 7f7c62a00640 1 --2- 172.21.15.102:0/3378344530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44003a90 0x7f7c44003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:43.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.649+0000 7f7c62a00640 1 --2- 172.21.15.102:0/3378344530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c44008850 0x7f7c4400ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:43.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.649+0000 7f7c62a00640 1 --2- 172.21.15.102:0/3378344530 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c64158db0 0x7f7c4400d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:43.651 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.649+0000 7f7c62a00640 1 -- 172.21.15.102:0/3378344530 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6406d0b0 con 0x7f7c44003a90 2024-03-23T03:51:43.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.649+0000 7f7c62a00640 1 -- 172.21.15.102:0/3378344530 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4400dc10 con 0x7f7c44008850 2024-03-23T03:51:43.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.649+0000 7f7c62a00640 1 -- 172.21.15.102:0/3378344530 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4400ddd0 con 0x7f7c64158db0 2024-03-23T03:51:43.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.649+0000 7f7c63e00640 1 --2- 172.21.15.102:0/3378344530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44003a90 0x7f7c44003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:43.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.650+0000 7f7c63e00640 1 -- 172.21.15.102:0/3378344530 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c64158db0 msgr2=0x7f7c4400d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:43.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.650+0000 7f7c63e00640 1 --2- 172.21.15.102:0/3378344530 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c64158db0 0x7f7c4400d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.650+0000 7f7c63400640 1 --2- 172.21.15.102:0/3378344530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c44008850 0x7f7c4400ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:43.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.650+0000 7f7c63e00640 1 -- 172.21.15.102:0/3378344530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c44008850 msgr2=0x7f7c4400ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:43.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.650+0000 7f7c63e00640 1 --2- 172.21.15.102:0/3378344530 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c44008850 0x7f7c4400ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.650+0000 7f7c63e00640 1 -- 172.21.15.102:0/3378344530 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c44003440 con 0x7f7c44003a90 2024-03-23T03:51:43.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.651+0000 7f7c63e00640 1 --2- 172.21.15.102:0/3378344530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44003a90 0x7f7c44003eb0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009ac0 tx=0x7f7c440036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb91f67de990ffe server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:43.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.651+0000 7f7c62a00640 1 -- 172.21.15.102:0/3378344530 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5402c710 con 0x7f7c44003a90 2024-03-23T03:51:43.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.651+0000 7f7c62a00640 1 -- 172.21.15.102:0/3378344530 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5402c870 con 0x7f7c44003a90 2024-03-23T03:51:43.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.651+0000 7f7c62a00640 1 -- 172.21.15.102:0/3378344530 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c5402cc00 con 0x7f7c44003a90 2024-03-23T03:51:43.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.652+0000 7f7c6ac00640 1 -- 172.21.15.102:0/3378344530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44003a90 msgr2=0x7f7c44003eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:43.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.652+0000 7f7c6ac00640 1 --2- 172.21.15.102:0/3378344530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44003a90 0x7f7c44003eb0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009ac0 tx=0x7f7c440036f0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.652+0000 7f7c68c00640 1 -- 172.21.15.102:0/3378344530 reap_dead start 2024-03-23T03:51:43.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.652+0000 7f7c6ac00640 1 -- 172.21.15.102:0/3378344530 shutdown_connections 2024-03-23T03:51:43.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.652+0000 7f7c6ac00640 1 -- 172.21.15.102:0/3378344530 >> 172.21.15.102:0/3378344530 conn(0x7f7c6406a0b0 msgr2=0x7f7c6406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:43.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.652+0000 7f7c6ac00640 1 -- 172.21.15.102:0/3378344530 shutdown_connections 2024-03-23T03:51:43.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.652+0000 7f7c6ac00640 1 -- 172.21.15.102:0/3378344530 wait complete. 2024-03-23T03:51:43.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.654+0000 7f7c6ac00640 1 Processor -- start 2024-03-23T03:51:43.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.654+0000 7f7c6ac00640 1 -- start start 2024-03-23T03:51:43.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.655+0000 7f7c6ac00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44003a90 0x7f7c6413d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:43.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.655+0000 7f7c6ac00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c6414fdf0 0x7f7c6414c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:43.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.655+0000 7f7c6ac00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c64150250 0x7f7c6414c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:43.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.655+0000 7f7c6ac00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c64153470 con 0x7f7c44003a90 2024-03-23T03:51:43.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.655+0000 7f7c6ac00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c64152fe0 con 0x7f7c6414fdf0 2024-03-23T03:51:43.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.655+0000 7f7c6ac00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c641523d0 con 0x7f7c64150250 2024-03-23T03:51:43.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.655+0000 7f7c63e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44003a90 0x7f7c6413d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:43.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.655+0000 7f7c63e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44003a90 0x7f7c6413d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48842/0 (socket says 172.21.15.102:48842) 2024-03-23T03:51:43.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.655+0000 7f7c63e00640 1 -- 172.21.15.102:0/3886219836 learned_addr learned my addr 172.21.15.102:0/3886219836 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:43.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.655+0000 7f7c63e00640 1 -- 172.21.15.102:0/3886219836 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c64150250 msgr2=0x7f7c6414c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:43.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.655+0000 7f7c63e00640 1 --2- 172.21.15.102:0/3886219836 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c64150250 0x7f7c6414c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.655+0000 7f7c63e00640 1 -- 172.21.15.102:0/3886219836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c6414fdf0 msgr2=0x7f7c6414c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:43.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.655+0000 7f7c63e00640 1 --2- 172.21.15.102:0/3886219836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c6414fdf0 0x7f7c6414c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.655+0000 7f7c63e00640 1 -- 172.21.15.102:0/3886219836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c641526b0 con 0x7f7c44003a90 2024-03-23T03:51:43.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.656+0000 7f7c63e00640 1 --2- 172.21.15.102:0/3886219836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44003a90 0x7f7c6413d140 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f7c5402c2b0 tx=0x7f7c54004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:43.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.656+0000 7f7c5be00640 1 -- 172.21.15.102:0/3886219836 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54003810 con 0x7f7c44003a90 2024-03-23T03:51:43.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.656+0000 7f7c5be00640 1 -- 172.21.15.102:0/3886219836 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c5402de90 con 0x7f7c44003a90 2024-03-23T03:51:43.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.656+0000 7f7c5be00640 1 -- 172.21.15.102:0/3886219836 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c54003b70 con 0x7f7c44003a90 2024-03-23T03:51:43.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.656+0000 7f7c6ac00640 1 -- 172.21.15.102:0/3886219836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c64152990 con 0x7f7c44003a90 2024-03-23T03:51:43.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.656+0000 7f7c6ac00640 1 -- 172.21.15.102:0/3886219836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c6413d680 con 0x7f7c44003a90 2024-03-23T03:51:43.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.657+0000 7f7c6ac00640 1 -- 172.21.15.102:0/3886219836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c64073390 con 0x7f7c44003a90 2024-03-23T03:51:43.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.658+0000 7f7c5be00640 1 -- 172.21.15.102:0/3886219836 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7c54003cd0 con 0x7f7c44003a90 2024-03-23T03:51:43.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.659+0000 7f7c5be00640 1 --2- 172.21.15.102:0/3886219836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7c4006ac00 0x7f7c4006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:43.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.659+0000 7f7c5be00640 1 -- 172.21.15.102:0/3886219836 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f7c540ab410 con 0x7f7c44003a90 2024-03-23T03:51:43.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.659+0000 7f7c63400640 1 --2- 172.21.15.102:0/3886219836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7c4006ac00 0x7f7c4006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:43.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.660+0000 7f7c63400640 1 --2- 172.21.15.102:0/3886219836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7c4006ac00 0x7f7c4006d0c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7c64154ed0 tx=0x7f7c4c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:43.663 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.663+0000 7f7c5be00640 1 -- 172.21.15.102:0/3886219836 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7c54076440 con 0x7f7c44003a90 2024-03-23T03:51:43.863 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.862+0000 7f7c6ac00640 1 -- 172.21.15.102:0/3886219836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7c6415c690 con 0x7f7c44003a90 2024-03-23T03:51:43.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.863+0000 7f7c5be00640 1 -- 172.21.15.102:0/3886219836 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7c54079a60 con 0x7f7c44003a90 2024-03-23T03:51:43.864 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:51:43.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.866+0000 7f7c6ac00640 1 -- 172.21.15.102:0/3886219836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7c4006ac00 msgr2=0x7f7c4006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:43.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.866+0000 7f7c6ac00640 1 --2- 172.21.15.102:0/3886219836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7c4006ac00 0x7f7c4006d0c0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f7c64154ed0 tx=0x7f7c4c009040 comp rx=0 tx=0).stop 2024-03-23T03:51:43.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.866+0000 7f7c6ac00640 1 -- 172.21.15.102:0/3886219836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44003a90 msgr2=0x7f7c6413d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:43.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.866+0000 7f7c6ac00640 1 --2- 172.21.15.102:0/3886219836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44003a90 0x7f7c6413d140 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f7c5402c2b0 tx=0x7f7c54004730 comp rx=0 tx=0).stop 2024-03-23T03:51:43.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.866+0000 7f7c6ac00640 1 -- 172.21.15.102:0/3886219836 shutdown_connections 2024-03-23T03:51:43.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.866+0000 7f7c6ac00640 1 --2- 172.21.15.102:0/3886219836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f7c4006ac00 0x7f7c4006d0c0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.866+0000 7f7c6ac00640 1 --2- 172.21.15.102:0/3886219836 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7c64150250 0x7f7c6414c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.866+0000 7f7c6ac00640 1 --2- 172.21.15.102:0/3886219836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c6414fdf0 0x7f7c6414c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.866+0000 7f7c6ac00640 1 --2- 172.21.15.102:0/3886219836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c44003a90 0x7f7c6413d140 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:43.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.866+0000 7f7c6ac00640 1 -- 172.21.15.102:0/3886219836 >> 172.21.15.102:0/3886219836 conn(0x7f7c6406a0b0 msgr2=0x7f7c640b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:43.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.866+0000 7f7c6ac00640 1 -- 172.21.15.102:0/3886219836 shutdown_connections 2024-03-23T03:51:43.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:43.866+0000 7f7c6ac00640 1 -- 172.21.15.102:0/3886219836 wait complete. 2024-03-23T03:51:43.887 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T03:51:44.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.094+0000 7f4613e00640 1 Processor -- start 2024-03-23T03:51:44.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.094+0000 7f4613e00640 1 -- start start 2024-03-23T03:51:44.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.096+0000 7f4613e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f46140cf7c0 0x7f46140cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:44.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.096+0000 7f4613e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46140ccd90 con 0x7f461415c820 2024-03-23T03:51:44.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.096+0000 7f4613e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46140b7fc0 con 0x7f4614160e90 2024-03-23T03:51:44.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.096+0000 7f4613e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46140cdd70 con 0x7f46140cf7c0 2024-03-23T03:51:44.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.096+0000 7f4612a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f461415c820 0x7f46140cf0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55036/0 (socket says 172.21.15.102:55036) 2024-03-23T03:51:44.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.096+0000 7f4612a00640 1 -- 172.21.15.102:0/2060862038 learned_addr learned my addr 172.21.15.102:0/2060862038 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:44.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.097+0000 7f4612000640 1 --2- 172.21.15.102:0/2060862038 >> v2:172.21.15.173:3300/0 conn(0x7f46140cf7c0 0x7f46140cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:44.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.097+0000 7f4611600640 1 -- 172.21.15.102:0/2060862038 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962974555 0 0) 0x7f46140ccd90 con 0x7f461415c820 2024-03-23T03:51:44.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.097+0000 7f4611600640 1 -- 172.21.15.102:0/2060862038 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45f8003520 con 0x7f461415c820 2024-03-23T03:51:44.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.097+0000 7f4611600640 1 -- 172.21.15.102:0/2060862038 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 887367570 0 0) 0x7f45f8003520 con 0x7f461415c820 2024-03-23T03:51:44.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.097+0000 7f4611600640 1 -- 172.21.15.102:0/2060862038 >> v2:172.21.15.173:3300/0 conn(0x7f46140cf7c0 msgr2=0x7f46140cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:44.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.097+0000 7f4611600640 1 --2- 172.21.15.102:0/2060862038 >> v2:172.21.15.173:3300/0 conn(0x7f46140cf7c0 0x7f46140cd830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:44.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.097+0000 7f4611600640 1 -- 172.21.15.102:0/2060862038 >> v1:172.21.15.173:6789/0 conn(0x7f4614160e90 legacy=0x7f46140d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:44.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.097+0000 7f4611600640 1 -- 172.21.15.102:0/2060862038 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46140b82a0 con 0x7f461415c820 2024-03-23T03:51:44.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.097+0000 7f4611600640 1 -- 172.21.15.102:0/2060862038 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f4604001670 con 0x7f461415c820 2024-03-23T03:51:44.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.097+0000 7f4611600640 1 -- 172.21.15.102:0/2060862038 >> v1:172.21.15.102:6789/0 conn(0x7f461415c820 legacy=0x7f46140cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:44.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.097+0000 7f4611600640 1 --2- 172.21.15.102:0/2060862038 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45f8003970 0x7f45f8003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:44.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.097+0000 7f4611600640 1 --2- 172.21.15.102:0/2060862038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45f80047c0 0x7f45f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:44.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.097+0000 7f4611600640 1 --2- 172.21.15.102:0/2060862038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45f8005180 0x7f45f800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:44.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.097+0000 7f4611600640 1 -- 172.21.15.102:0/2060862038 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46140cdd70 con 0x7f45f8005180 2024-03-23T03:51:44.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.097+0000 7f4611600640 1 -- 172.21.15.102:0/2060862038 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f800c000 con 0x7f45f80047c0 2024-03-23T03:51:44.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.097+0000 7f4611600640 1 -- 172.21.15.102:0/2060862038 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f800c1c0 con 0x7f45f8003970 2024-03-23T03:51:44.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.098+0000 7f4613400640 1 --2- 172.21.15.102:0/2060862038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45f8005180 0x7f45f800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:44.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.098+0000 7f4612a00640 1 --2- 172.21.15.102:0/2060862038 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45f8003970 0x7f45f8003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:44.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.098+0000 7f4613400640 1 -- 172.21.15.102:0/2060862038 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45f8003970 msgr2=0x7f45f8003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:44.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.098+0000 7f4613400640 1 --2- 172.21.15.102:0/2060862038 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45f8003970 0x7f45f8003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:44.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.098+0000 7f4613400640 1 -- 172.21.15.102:0/2060862038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45f80047c0 msgr2=0x7f45f8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:44.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.098+0000 7f4613400640 1 --2- 172.21.15.102:0/2060862038 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f45f80047c0 0x7f45f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:44.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.098+0000 7f4613400640 1 -- 172.21.15.102:0/2060862038 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f8003520 con 0x7f45f8005180 2024-03-23T03:51:44.100 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.098+0000 7f4612a00640 1 --2- 172.21.15.102:0/2060862038 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45f8003970 0x7f45f8003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:44.101 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.099+0000 7f4613400640 1 -- 172.21.15.102:0/2060862038 reap_dead start 2024-03-23T03:51:44.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.099+0000 7f4613400640 1 --2- 172.21.15.102:0/2060862038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45f8005180 0x7f45f800b980 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f460c004480 tx=0x7f45f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f503379778afcb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:44.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.099+0000 7f4611600640 1 -- 172.21.15.102:0/2060862038 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c030400 con 0x7f45f8005180 2024-03-23T03:51:44.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.099+0000 7f4611600640 1 -- 172.21.15.102:0/2060862038 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460c030560 con 0x7f45f8005180 2024-03-23T03:51:44.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.099+0000 7f4611600640 1 -- 172.21.15.102:0/2060862038 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c0308f0 con 0x7f45f8005180 2024-03-23T03:51:44.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.100+0000 7f4613e00640 1 -- 172.21.15.102:0/2060862038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45f8005180 msgr2=0x7f45f800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:44.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.100+0000 7f4613e00640 1 --2- 172.21.15.102:0/2060862038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45f8005180 0x7f45f800b980 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f460c004480 tx=0x7f45f80057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:44.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.100+0000 7f4613e00640 1 -- 172.21.15.102:0/2060862038 shutdown_connections 2024-03-23T03:51:44.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.100+0000 7f4613e00640 1 --2- 172.21.15.102:0/2060862038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45f8005180 0x7f45f800b980 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:44.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.100+0000 7f4613e00640 1 -- 172.21.15.102:0/2060862038 >> 172.21.15.102:0/2060862038 conn(0x7f461406b070 msgr2=0x7f461406b480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:44.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.100+0000 7f4613e00640 1 -- 172.21.15.102:0/2060862038 shutdown_connections 2024-03-23T03:51:44.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.100+0000 7f4613e00640 1 -- 172.21.15.102:0/2060862038 wait complete. 2024-03-23T03:51:44.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.102+0000 7f4613e00640 1 Processor -- start 2024-03-23T03:51:44.103 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.102+0000 7f4613e00640 1 -- start start 2024-03-23T03:51:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.103+0000 7f4613e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45f8005180 0x7f4614149ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.103+0000 7f4613e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46140b1c40 0x7f46140b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.103+0000 7f4613e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4614156490 0x7f4614156940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.103+0000 7f4613e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4614156e80 con 0x7f4614156490 2024-03-23T03:51:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.103+0000 7f4613e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46140b3a60 con 0x7f46140b1c40 2024-03-23T03:51:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.103+0000 7f4613e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46140b2d30 con 0x7f45f8005180 2024-03-23T03:51:44.105 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.103+0000 7f4613400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4614156490 0x7f4614156940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.103+0000 7f4613400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4614156490 0x7f4614156940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48870/0 (socket says 172.21.15.102:48870) 2024-03-23T03:51:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.103+0000 7f4613400640 1 -- 172.21.15.102:0/4103506829 learned_addr learned my addr 172.21.15.102:0/4103506829 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.104+0000 7f4612a00640 1 --2- 172.21.15.102:0/4103506829 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45f8005180 0x7f4614149ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.104+0000 7f4612000640 1 --2- 172.21.15.102:0/4103506829 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46140b1c40 0x7f46140b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.104+0000 7f4613400640 1 -- 172.21.15.102:0/4103506829 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45f8005180 msgr2=0x7f4614149ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.104+0000 7f4613400640 1 --2- 172.21.15.102:0/4103506829 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45f8005180 0x7f4614149ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.104+0000 7f4613400640 1 -- 172.21.15.102:0/4103506829 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46140b1c40 msgr2=0x7f46140b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.104+0000 7f4613400640 1 --2- 172.21.15.102:0/4103506829 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46140b1c40 0x7f46140b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.104+0000 7f4613400640 1 -- 172.21.15.102:0/4103506829 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461414a430 con 0x7f4614156490 2024-03-23T03:51:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.104+0000 7f4612000640 1 --2- 172.21.15.102:0/4103506829 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46140b1c40 0x7f46140b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:51:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.104+0000 7f4613400640 1 --2- 172.21.15.102:0/4103506829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4614156490 0x7f4614156940 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f460c000f80 tx=0x7f460c003ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.105+0000 7f460b400640 1 -- 172.21.15.102:0/4103506829 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c006b50 con 0x7f4614156490 2024-03-23T03:51:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.105+0000 7f460b400640 1 -- 172.21.15.102:0/4103506829 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460c006cb0 con 0x7f4614156490 2024-03-23T03:51:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.105+0000 7f460b400640 1 -- 172.21.15.102:0/4103506829 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c039880 con 0x7f4614156490 2024-03-23T03:51:44.106 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.105+0000 7f4613e00640 1 -- 172.21.15.102:0/4103506829 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46140b2120 con 0x7f4614156490 2024-03-23T03:51:44.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.105+0000 7f4613e00640 1 -- 172.21.15.102:0/4103506829 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46140b26e0 con 0x7f4614156490 2024-03-23T03:51:44.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.106+0000 7f4613e00640 1 -- 172.21.15.102:0/4103506829 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4614156e80 con 0x7f4614156490 2024-03-23T03:51:44.107 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.107+0000 7f460b400640 1 -- 172.21.15.102:0/4103506829 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f460c0399e0 con 0x7f4614156490 2024-03-23T03:51:44.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.108+0000 7f460b400640 1 --2- 172.21.15.102:0/4103506829 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f45f406ac30 0x7f45f406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:44.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.108+0000 7f460b400640 1 -- 172.21.15.102:0/4103506829 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f460c0ab020 con 0x7f4614156490 2024-03-23T03:51:44.108 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.108+0000 7f4612a00640 1 --2- 172.21.15.102:0/4103506829 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f45f406ac30 0x7f45f406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:44.109 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.109+0000 7f4612a00640 1 --2- 172.21.15.102:0/4103506829 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f45f406ac30 0x7f45f406d0f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f46040017a0 tx=0x7f4604009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:44.112 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.112+0000 7f460b400640 1 -- 172.21.15.102:0/4103506829 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4614156e80 con 0x7f4614156490 2024-03-23T03:51:44.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.317+0000 7f4613e00640 1 -- 172.21.15.102:0/4103506829 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4614156e80 con 0x7f4614156490 2024-03-23T03:51:44.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.318+0000 7f460b400640 1 -- 172.21.15.102:0/4103506829 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v107) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4614156e80 con 0x7f4614156490 2024-03-23T03:51:44.319 INFO:teuthology.orchestra.run.smithi102.stderr:listed 0 entries 2024-03-23T03:51:44.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.320+0000 7f4613e00640 1 -- 172.21.15.102:0/4103506829 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f45f406ac30 msgr2=0x7f45f406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:44.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.320+0000 7f4613e00640 1 --2- 172.21.15.102:0/4103506829 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f45f406ac30 0x7f45f406d0f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f46040017a0 tx=0x7f4604009040 comp rx=0 tx=0).stop 2024-03-23T03:51:44.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.320+0000 7f4613e00640 1 -- 172.21.15.102:0/4103506829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4614156490 msgr2=0x7f4614156940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:44.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.320+0000 7f4613e00640 1 --2- 172.21.15.102:0/4103506829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4614156490 0x7f4614156940 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f460c000f80 tx=0x7f460c003ae0 comp rx=0 tx=0).stop 2024-03-23T03:51:44.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.320+0000 7f4613e00640 1 -- 172.21.15.102:0/4103506829 shutdown_connections 2024-03-23T03:51:44.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.320+0000 7f4613e00640 1 --2- 172.21.15.102:0/4103506829 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f45f406ac30 0x7f45f406d0f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:44.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.321+0000 7f4613e00640 1 --2- 172.21.15.102:0/4103506829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4614156490 0x7f4614156940 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:44.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.321+0000 7f4613e00640 1 --2- 172.21.15.102:0/4103506829 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46140b1c40 0x7f46140b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:44.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.321+0000 7f4613e00640 1 --2- 172.21.15.102:0/4103506829 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45f8005180 0x7f4614149ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:44.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.321+0000 7f4613e00640 1 -- 172.21.15.102:0/4103506829 >> 172.21.15.102:0/4103506829 conn(0x7f461406b070 msgr2=0x7f46140d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:44.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.321+0000 7f4613e00640 1 -- 172.21.15.102:0/4103506829 shutdown_connections 2024-03-23T03:51:44.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.321+0000 7f4613e00640 1 -- 172.21.15.102:0/4103506829 wait complete. 2024-03-23T03:51:44.340 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2024-03-23T03:51:44.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:44.340 DEBUG:teuthology.orchestra.run.smithi173:> sudo umount /home/ubuntu/cephtest/mnt.1 2024-03-23T03:51:44.547 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:51:44.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:44.547 DEBUG:teuthology.orchestra.run.smithi173:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T03:51:44.598 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:51:44.598 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:44.637 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:51:44.637 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:51:44.637 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:44.678 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:51:44.678 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:51:44.867 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.867+0000 7f3467000640 1 Processor -- start 2024-03-23T03:51:44.870 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.867+0000 7f3467000640 1 -- start start 2024-03-23T03:51:44.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.868+0000 7f3467000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f34600cbab0 0x7f34600b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:44.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.868+0000 7f3467000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f346006d390 0x7f346006d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:44.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.868+0000 7f3467000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34600b16a0 con 0x7f34600adbc0 2024-03-23T03:51:44.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.868+0000 7f3467000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f346006dd60 con 0x7f346006d390 2024-03-23T03:51:44.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.868+0000 7f3467000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34600cbfb0 con 0x7f34600cbab0 2024-03-23T03:51:44.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.869+0000 7f3465000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f346006d390 0x7f346006d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:44.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.869+0000 7f3465000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f346006d390 0x7f346006d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48880/0 (socket says 172.21.15.102:48880) 2024-03-23T03:51:44.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.869+0000 7f3465000640 1 -- 172.21.15.102:0/2599554714 learned_addr learned my addr 172.21.15.102:0/2599554714 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:44.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.869+0000 7f3465000640 1 -- 172.21.15.102:0/2599554714 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f34600cbab0 msgr2=0x7f34600b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:44.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.869+0000 7f3457400640 1 --2- 172.21.15.102:0/2599554714 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f34600cbab0 0x7f34600b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:44.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.869+0000 7f3465000640 1 --2- 172.21.15.102:0/2599554714 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f34600cbab0 0x7f34600b20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:44.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.869+0000 7f3465000640 1 -- 172.21.15.102:0/2599554714 >> v1:172.21.15.173:6789/0 conn(0x7f34600adbc0 legacy=0x7f34600b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:44.871 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.869+0000 7f3465000640 1 -- 172.21.15.102:0/2599554714 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34600cc290 con 0x7f346006d390 2024-03-23T03:51:44.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.870+0000 7f3465000640 1 --2- 172.21.15.102:0/2599554714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f346006d390 0x7f346006d820 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f3458009e50 tx=0x7f345802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=322f035b104f4014 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:44.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.871+0000 7f345f400640 1 -- 172.21.15.102:0/2599554714 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345802c9e0 con 0x7f346006d390 2024-03-23T03:51:44.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.871+0000 7f345f400640 1 -- 172.21.15.102:0/2599554714 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345802cb40 con 0x7f346006d390 2024-03-23T03:51:44.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.871+0000 7f345f400640 1 -- 172.21.15.102:0/2599554714 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34580343f0 con 0x7f346006d390 2024-03-23T03:51:44.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.871+0000 7f3467000640 1 -- 172.21.15.102:0/2599554714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f346006d390 msgr2=0x7f346006d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:44.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.871+0000 7f3467000640 1 --2- 172.21.15.102:0/2599554714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f346006d390 0x7f346006d820 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f3458009e50 tx=0x7f345802be40 comp rx=0 tx=0).stop 2024-03-23T03:51:44.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.871+0000 7f3467000640 1 -- 172.21.15.102:0/2599554714 shutdown_connections 2024-03-23T03:51:44.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.871+0000 7f3467000640 1 --2- 172.21.15.102:0/2599554714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f346006d390 0x7f346006d820 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:44.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.871+0000 7f3467000640 1 --2- 172.21.15.102:0/2599554714 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f34600cbab0 0x7f34600b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:44.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.871+0000 7f3467000640 1 -- 172.21.15.102:0/2599554714 >> 172.21.15.102:0/2599554714 conn(0x7f346006a3d0 msgr2=0x7f34600b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:44.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.871+0000 7f3467000640 1 -- 172.21.15.102:0/2599554714 shutdown_connections 2024-03-23T03:51:44.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.871+0000 7f3467000640 1 -- 172.21.15.102:0/2599554714 wait complete. 2024-03-23T03:51:44.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.873+0000 7f3467000640 1 Processor -- start 2024-03-23T03:51:44.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.873+0000 7f3467000640 1 -- start start 2024-03-23T03:51:44.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.874+0000 7f3467000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f346006d390 0x7f34601381f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:44.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.874+0000 7f3467000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f34600adbc0 0x7f34600c4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:44.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.874+0000 7f3467000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34600cbab0 0x7f34600ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:44.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.874+0000 7f3467000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34600c3610 con 0x7f34600cbab0 2024-03-23T03:51:44.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.874+0000 7f3467000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346015cc40 con 0x7f346006d390 2024-03-23T03:51:44.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.874+0000 7f3467000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34600c3820 con 0x7f34600adbc0 2024-03-23T03:51:44.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.874+0000 7f3465000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34600cbab0 0x7f34600ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:44.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.874+0000 7f3465000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34600cbab0 0x7f34600ab8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48886/0 (socket says 172.21.15.102:48886) 2024-03-23T03:51:44.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.874+0000 7f3465000640 1 -- 172.21.15.102:0/4227904587 learned_addr learned my addr 172.21.15.102:0/4227904587 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:44.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.875+0000 7f3457400640 1 --2- 172.21.15.102:0/4227904587 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f34600adbc0 0x7f34600c4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:44.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.875+0000 7f345fe00640 1 --2- 172.21.15.102:0/4227904587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f346006d390 0x7f34601381f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:44.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.875+0000 7f3465000640 1 -- 172.21.15.102:0/4227904587 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f34600adbc0 msgr2=0x7f34600c4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:44.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.875+0000 7f3465000640 1 --2- 172.21.15.102:0/4227904587 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f34600adbc0 0x7f34600c4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:44.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.875+0000 7f3465000640 1 -- 172.21.15.102:0/4227904587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f346006d390 msgr2=0x7f34601381f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:44.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.875+0000 7f3465000640 1 --2- 172.21.15.102:0/4227904587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f346006d390 0x7f34601381f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:44.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.875+0000 7f3465000640 1 -- 172.21.15.102:0/4227904587 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3458009b30 con 0x7f34600cbab0 2024-03-23T03:51:44.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.875+0000 7f3465000640 1 --2- 172.21.15.102:0/4227904587 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34600cbab0 0x7f34600ab8b0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f3458036040 tx=0x7f3458004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:44.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.875+0000 7f345cc00640 1 -- 172.21.15.102:0/4227904587 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345803e450 con 0x7f34600cbab0 2024-03-23T03:51:44.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.875+0000 7f345cc00640 1 -- 172.21.15.102:0/4227904587 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f345803e5d0 con 0x7f34600cbab0 2024-03-23T03:51:44.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.875+0000 7f3467000640 1 -- 172.21.15.102:0/4227904587 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34600c3b00 con 0x7f34600cbab0 2024-03-23T03:51:44.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.875+0000 7f345cc00640 1 -- 172.21.15.102:0/4227904587 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f345803e980 con 0x7f34600cbab0 2024-03-23T03:51:44.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.876+0000 7f3467000640 1 -- 172.21.15.102:0/4227904587 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3460138730 con 0x7f34600cbab0 2024-03-23T03:51:44.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.877+0000 7f3467000640 1 -- 172.21.15.102:0/4227904587 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346006ac20 con 0x7f34600cbab0 2024-03-23T03:51:44.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.878+0000 7f345cc00640 1 -- 172.21.15.102:0/4227904587 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f345803eae0 con 0x7f34600cbab0 2024-03-23T03:51:44.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.879+0000 7f345cc00640 1 --2- 172.21.15.102:0/4227904587 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f343c06ac00 0x7f343c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:44.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.879+0000 7f345cc00640 1 -- 172.21.15.102:0/4227904587 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f34580ac470 con 0x7f34600cbab0 2024-03-23T03:51:44.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.879+0000 7f345fe00640 1 --2- 172.21.15.102:0/4227904587 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f343c06ac00 0x7f343c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:44.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.880+0000 7f345fe00640 1 --2- 172.21.15.102:0/4227904587 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f343c06ac00 0x7f343c06d0c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3450003d90 tx=0x7f3450009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:44.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:44.884+0000 7f345cc00640 1 -- 172.21.15.102:0/4227904587 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f34580774c0 con 0x7f34600cbab0 2024-03-23T03:51:45.118 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.118+0000 7f3467000640 1 -- 172.21.15.102:0/4227904587 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3460073390 con 0x7f34600cbab0 2024-03-23T03:51:45.120 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.120+0000 7f345cc00640 1 -- 172.21.15.102:0/4227904587 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f345807aae0 con 0x7f34600cbab0 2024-03-23T03:51:45.120 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:45.121 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2017025570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2017025570},{"type":"v1","addr":"172.21.15.173:6835","nonce":2017025570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/173893445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":173893445},{"type":"v1","addr":"172.21.15.102:6835","nonce":173893445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/255507815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":255507815},{"type":"v1","addr":"172.21.15.173:6833","nonce":255507815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:51:22.228808+0000","modified":"2024-03-23T03:51:25.724445+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7212":{"gid":7212,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/883705787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":883705787},{"type":"v1","addr":"172.21.15.102:6837","nonce":883705787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T03:51:45.121 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 56 2024-03-23T03:51:45.123 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.122+0000 7f3467000640 1 -- 172.21.15.102:0/4227904587 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f343c06ac00 msgr2=0x7f343c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.122+0000 7f3467000640 1 --2- 172.21.15.102:0/4227904587 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f343c06ac00 0x7f343c06d0c0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3450003d90 tx=0x7f3450009040 comp rx=0 tx=0).stop 2024-03-23T03:51:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.122+0000 7f3467000640 1 -- 172.21.15.102:0/4227904587 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34600cbab0 msgr2=0x7f34600ab8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.122+0000 7f3467000640 1 --2- 172.21.15.102:0/4227904587 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34600cbab0 0x7f34600ab8b0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f3458036040 tx=0x7f3458004a40 comp rx=0 tx=0).stop 2024-03-23T03:51:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.123+0000 7f3467000640 1 -- 172.21.15.102:0/4227904587 shutdown_connections 2024-03-23T03:51:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.123+0000 7f3467000640 1 --2- 172.21.15.102:0/4227904587 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f343c06ac00 0x7f343c06d0c0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.123+0000 7f3467000640 1 --2- 172.21.15.102:0/4227904587 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34600cbab0 0x7f34600ab8b0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.123+0000 7f3467000640 1 --2- 172.21.15.102:0/4227904587 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f34600adbc0 0x7f34600c4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.123+0000 7f3467000640 1 --2- 172.21.15.102:0/4227904587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f346006d390 0x7f34601381f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.123+0000 7f3467000640 1 -- 172.21.15.102:0/4227904587 >> 172.21.15.102:0/4227904587 conn(0x7f346006a3d0 msgr2=0x7f34600b3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.123+0000 7f3467000640 1 -- 172.21.15.102:0/4227904587 shutdown_connections 2024-03-23T03:51:45.124 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.123+0000 7f3467000640 1 -- 172.21.15.102:0/4227904587 wait complete. 2024-03-23T03:51:45.142 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:51:45.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.340+0000 7ff0cf000640 1 Processor -- start 2024-03-23T03:51:45.341 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.341+0000 7ff0cf000640 1 -- start start 2024-03-23T03:51:45.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.342+0000 7ff0cf000640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff0c815cda0 0x7ff0c80cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:45.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.342+0000 7ff0cf000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ff0c80b5a30 0x7ff0c80b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:45.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.342+0000 7ff0cf000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c80d1830 con 0x7ff0c80b4350 2024-03-23T03:51:45.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.342+0000 7ff0cf000640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c80b1f20 con 0x7ff0c815cda0 2024-03-23T03:51:45.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.342+0000 7ff0cf000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0c80d0c20 con 0x7ff0c80b5a30 2024-03-23T03:51:45.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.342+0000 7ff0c7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff0c815cda0 0x7ff0c80cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:45.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.342+0000 7ff0c7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff0c815cda0 0x7ff0c80cdb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48892/0 (socket says 172.21.15.102:48892) 2024-03-23T03:51:45.343 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.342+0000 7ff0c7e00640 1 -- 172.21.15.102:0/1275112822 learned_addr learned my addr 172.21.15.102:0/1275112822 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:45.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.343+0000 7ff0c7400640 1 --2- 172.21.15.102:0/1275112822 >> v2:172.21.15.173:3300/0 conn(0x7ff0c80b5a30 0x7ff0c80b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:45.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.343+0000 7ff0c7e00640 1 -- 172.21.15.102:0/1275112822 >> v2:172.21.15.173:3300/0 conn(0x7ff0c80b5a30 msgr2=0x7ff0c80b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:45.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.343+0000 7ff0c7e00640 1 --2- 172.21.15.102:0/1275112822 >> v2:172.21.15.173:3300/0 conn(0x7ff0c80b5a30 0x7ff0c80b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.343+0000 7ff0c7e00640 1 -- 172.21.15.102:0/1275112822 >> v1:172.21.15.173:6789/0 conn(0x7ff0c80b4350 legacy=0x7ff0c80d1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:45.344 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.343+0000 7ff0c7e00640 1 -- 172.21.15.102:0/1275112822 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c80d0f00 con 0x7ff0c815cda0 2024-03-23T03:51:45.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.344+0000 7ff0c7e00640 1 --2- 172.21.15.102:0/1275112822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c815cda0 0x7ff0c80cdb40 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff0c80cf010 tx=0x7ff0b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45440eef5951ebbb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:45.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.344+0000 7ff0c6a00640 1 -- 172.21.15.102:0/1275112822 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8031870 con 0x7ff0c815cda0 2024-03-23T03:51:45.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.344+0000 7ff0c6a00640 1 -- 172.21.15.102:0/1275112822 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b80319d0 con 0x7ff0c815cda0 2024-03-23T03:51:45.345 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.344+0000 7ff0c6a00640 1 -- 172.21.15.102:0/1275112822 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8031d60 con 0x7ff0c815cda0 2024-03-23T03:51:45.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.345+0000 7ff0cf000640 1 -- 172.21.15.102:0/1275112822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c815cda0 msgr2=0x7ff0c80cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:45.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.345+0000 7ff0cf000640 1 --2- 172.21.15.102:0/1275112822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c815cda0 0x7ff0c80cdb40 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7ff0c80cf010 tx=0x7ff0b802b910 comp rx=0 tx=0).stop 2024-03-23T03:51:45.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.345+0000 7ff0cf000640 1 -- 172.21.15.102:0/1275112822 shutdown_connections 2024-03-23T03:51:45.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.345+0000 7ff0cf000640 1 --2- 172.21.15.102:0/1275112822 >> v2:172.21.15.173:3300/0 conn(0x7ff0c80b5a30 0x7ff0c80b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.345+0000 7ff0cf000640 1 --2- 172.21.15.102:0/1275112822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c815cda0 0x7ff0c80cdb40 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.345+0000 7ff0cf000640 1 -- 172.21.15.102:0/1275112822 >> 172.21.15.102:0/1275112822 conn(0x7ff0c80cf350 msgr2=0x7ff0c80cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:45.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.345+0000 7ff0cf000640 1 -- 172.21.15.102:0/1275112822 shutdown_connections 2024-03-23T03:51:45.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.345+0000 7ff0cf000640 1 -- 172.21.15.102:0/1275112822 wait complete. 2024-03-23T03:51:45.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.347+0000 7ff0cf000640 1 Processor -- start 2024-03-23T03:51:45.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.347+0000 7ff0cf000640 1 -- start start 2024-03-23T03:51:45.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.348+0000 7ff0cf000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c80b4350 0x7ff0c8158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:45.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.348+0000 7ff0cf000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0c80b5a30 0x7ff0c8158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:45.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.348+0000 7ff0cf000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff0c815cda0 0x7ff0c815dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:45.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.348+0000 7ff0cf000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c8158c90 con 0x7ff0c80b4350 2024-03-23T03:51:45.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.348+0000 7ff0cf000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c815e7a0 con 0x7ff0c80b5a30 2024-03-23T03:51:45.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.348+0000 7ff0cf000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c80c7580 con 0x7ff0c815cda0 2024-03-23T03:51:45.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.348+0000 7ff0c7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c80b4350 0x7ff0c8158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:45.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.348+0000 7ff0c7e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c80b4350 0x7ff0c8158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48898/0 (socket says 172.21.15.102:48898) 2024-03-23T03:51:45.349 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.348+0000 7ff0c7e00640 1 -- 172.21.15.102:0/2033071596 learned_addr learned my addr 172.21.15.102:0/2033071596 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:45.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.349+0000 7ff0c7400640 1 --2- 172.21.15.102:0/2033071596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0c80b5a30 0x7ff0c8158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:45.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.349+0000 7ff0c7e00640 1 -- 172.21.15.102:0/2033071596 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff0c815cda0 msgr2=0x7ff0c815dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:45.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.349+0000 7ff0cd000640 1 --2- 172.21.15.102:0/2033071596 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff0c815cda0 0x7ff0c815dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:45.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.349+0000 7ff0c7e00640 1 --2- 172.21.15.102:0/2033071596 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff0c815cda0 0x7ff0c815dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.349+0000 7ff0c7e00640 1 -- 172.21.15.102:0/2033071596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0c80b5a30 msgr2=0x7ff0c8158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:45.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.349+0000 7ff0c7e00640 1 --2- 172.21.15.102:0/2033071596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0c80b5a30 0x7ff0c8158e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.349+0000 7ff0c7e00640 1 -- 172.21.15.102:0/2033071596 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b80094d0 con 0x7ff0c80b4350 2024-03-23T03:51:45.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.349+0000 7ff0c7400640 1 --2- 172.21.15.102:0/2033071596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0c80b5a30 0x7ff0c8158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:45.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.349+0000 7ff0c7e00640 1 --2- 172.21.15.102:0/2033071596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c80b4350 0x7ff0c8158750 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8000c00 tx=0x7ff0b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:45.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.349+0000 7ff0bfe00640 1 -- 172.21.15.102:0/2033071596 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b803ad30 con 0x7ff0c80b4350 2024-03-23T03:51:45.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.349+0000 7ff0bfe00640 1 -- 172.21.15.102:0/2033071596 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0b803ae90 con 0x7ff0c80b4350 2024-03-23T03:51:45.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.349+0000 7ff0cf000640 1 -- 172.21.15.102:0/2033071596 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c80d34c0 con 0x7ff0c80b4350 2024-03-23T03:51:45.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.350+0000 7ff0bfe00640 1 -- 172.21.15.102:0/2033071596 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0b8044630 con 0x7ff0c80b4350 2024-03-23T03:51:45.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.350+0000 7ff0cf000640 1 -- 172.21.15.102:0/2033071596 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c80c7b00 con 0x7ff0c80b4350 2024-03-23T03:51:45.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.351+0000 7ff0cf000640 1 -- 172.21.15.102:0/2033071596 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c8073390 con 0x7ff0c80b4350 2024-03-23T03:51:45.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.352+0000 7ff0bfe00640 1 -- 172.21.15.102:0/2033071596 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff0b804d450 con 0x7ff0c80b4350 2024-03-23T03:51:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.352+0000 7ff0bfe00640 1 --2- 172.21.15.102:0/2033071596 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff0a806ac80 0x7ff0a806d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.353+0000 7ff0bfe00640 1 -- 172.21.15.102:0/2033071596 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6116+0+0 (secure 0 0 0) 0x7ff0b802d040 con 0x7ff0c80b4350 2024-03-23T03:51:45.353 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.353+0000 7ff0c7400640 1 --2- 172.21.15.102:0/2033071596 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff0a806ac80 0x7ff0a806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:45.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.353+0000 7ff0c7400640 1 --2- 172.21.15.102:0/2033071596 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff0a806ac80 0x7ff0a806d140 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8065e00 tx=0x7ff0b400b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:45.357 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.357+0000 7ff0bfe00640 1 -- 172.21.15.102:0/2033071596 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ff0b8076b30 con 0x7ff0c80b4350 2024-03-23T03:51:45.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.543+0000 7ff0cf000640 1 -- 172.21.15.102:0/2033071596 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0c80cc460 con 0x7ff0c80b4350 2024-03-23T03:51:45.547 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.546+0000 7ff0bfe00640 1 -- 172.21.15.102:0/2033071596 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16501 (secure 0 0 0) 0x7ff0b807a150 con 0x7ff0c80b4350 2024-03-23T03:51:45.547 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:45.547 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":107,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:51:43.745674+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:51:19.796618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-23T03:51:21.283447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.8","mappings":[{"from":0,"to":7}]},{"pgid":"16.39","mappings":[{"from":1,"to":4}]},{"pgid":"16.3d","mappings":[{"from":5,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:51:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.549+0000 7ff0cf000640 1 -- 172.21.15.102:0/2033071596 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff0a806ac80 msgr2=0x7ff0a806d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.549+0000 7ff0cf000640 1 --2- 172.21.15.102:0/2033071596 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff0a806ac80 0x7ff0a806d140 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8065e00 tx=0x7ff0b400b040 comp rx=0 tx=0).stop 2024-03-23T03:51:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.549+0000 7ff0cf000640 1 -- 172.21.15.102:0/2033071596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c80b4350 msgr2=0x7ff0c8158750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.549+0000 7ff0cf000640 1 --2- 172.21.15.102:0/2033071596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c80b4350 0x7ff0c8158750 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8000c00 tx=0x7ff0b80046c0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.549+0000 7ff0cf000640 1 -- 172.21.15.102:0/2033071596 shutdown_connections 2024-03-23T03:51:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.549+0000 7ff0cf000640 1 --2- 172.21.15.102:0/2033071596 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff0a806ac80 0x7ff0a806d140 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.549+0000 7ff0cf000640 1 --2- 172.21.15.102:0/2033071596 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff0c815cda0 0x7ff0c815dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.549+0000 7ff0cf000640 1 --2- 172.21.15.102:0/2033071596 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff0c80b5a30 0x7ff0c8158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.550 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.549+0000 7ff0cf000640 1 --2- 172.21.15.102:0/2033071596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0c80b4350 0x7ff0c8158750 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.549+0000 7ff0cf000640 1 -- 172.21.15.102:0/2033071596 >> 172.21.15.102:0/2033071596 conn(0x7ff0c80cf350 msgr2=0x7ff0c80cb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:45.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.549+0000 7ff0cf000640 1 -- 172.21.15.102:0/2033071596 shutdown_connections 2024-03-23T03:51:45.551 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.549+0000 7ff0cf000640 1 -- 172.21.15.102:0/2033071596 wait complete. 2024-03-23T03:51:45.568 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T03:46:47.784997+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T03:51:19.796618+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T03:51:21.283447+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T03:51:45.568 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T03:51:45.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.773+0000 7f36c9e00640 1 Processor -- start 2024-03-23T03:51:45.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.773+0000 7f36c9e00640 1 -- start start 2024-03-23T03:51:45.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.774+0000 7f36c9e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f36c415c860 0x7f36c4071350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:45.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.774+0000 7f36c9e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c415cc40 con 0x7f36c406b3b0 2024-03-23T03:51:45.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.774+0000 7f36c9e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c406c990 con 0x7f36c40abb20 2024-03-23T03:51:45.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.774+0000 7f36c9e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c4071ba0 con 0x7f36c415c860 2024-03-23T03:51:45.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.775+0000 7f36c9400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f36c406b3b0 0x7f36c406d750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55042/0 (socket says 172.21.15.102:55042) 2024-03-23T03:51:45.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.775+0000 7f36c9400640 1 -- 172.21.15.102:0/2588791036 learned_addr learned my addr 172.21.15.102:0/2588791036 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:45.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.775+0000 7f36c8a00640 1 --2- 172.21.15.102:0/2588791036 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f36c415c860 0x7f36c4071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:45.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.776+0000 7f36c8a00640 1 -- 172.21.15.102:0/2588791036 >> v1:172.21.15.173:6789/0 conn(0x7f36c40abb20 legacy=0x7f36c406aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:45.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.776+0000 7f36c8a00640 1 -- 172.21.15.102:0/2588791036 >> v1:172.21.15.102:6789/0 conn(0x7f36c406b3b0 legacy=0x7f36c406d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:45.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.776+0000 7f36c8a00640 1 -- 172.21.15.102:0/2588791036 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c406cc70 con 0x7f36c415c860 2024-03-23T03:51:45.778 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.778+0000 7f36c8a00640 1 --2- 172.21.15.102:0/2588791036 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f36c415c860 0x7f36c4071350 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f36b4009dc0 tx=0x7f36b402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50658c79b43e20af server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:45.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.780+0000 7f36c3e00640 1 -- 172.21.15.102:0/2588791036 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b402ccd0 con 0x7f36c415c860 2024-03-23T03:51:45.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.781+0000 7f36c3e00640 1 -- 172.21.15.102:0/2588791036 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36b402ce30 con 0x7f36c415c860 2024-03-23T03:51:45.781 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.781+0000 7f36c3e00640 1 -- 172.21.15.102:0/2588791036 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36b4034660 con 0x7f36c415c860 2024-03-23T03:51:45.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.781+0000 7f36c9e00640 1 -- 172.21.15.102:0/2588791036 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f36c415c860 msgr2=0x7f36c4071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:45.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.781+0000 7f36c9e00640 1 --2- 172.21.15.102:0/2588791036 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f36c415c860 0x7f36c4071350 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f36b4009dc0 tx=0x7f36b402bf20 comp rx=0 tx=0).stop 2024-03-23T03:51:45.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.781+0000 7f36c9e00640 1 -- 172.21.15.102:0/2588791036 shutdown_connections 2024-03-23T03:51:45.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.781+0000 7f36c9e00640 1 --2- 172.21.15.102:0/2588791036 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f36c415c860 0x7f36c4071350 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.781+0000 7f36c9e00640 1 -- 172.21.15.102:0/2588791036 >> 172.21.15.102:0/2588791036 conn(0x7f36c4070860 msgr2=0x7f36c4070c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:45.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.782+0000 7f36c9e00640 1 -- 172.21.15.102:0/2588791036 shutdown_connections 2024-03-23T03:51:45.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.782+0000 7f36c9e00640 1 -- 172.21.15.102:0/2588791036 wait complete. 2024-03-23T03:51:45.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.783+0000 7f36c9e00640 1 Processor -- start 2024-03-23T03:51:45.784 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.783+0000 7f36c9e00640 1 -- start start 2024-03-23T03:51:45.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.784+0000 7f36c9e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f36c406b3b0 0x7f36c40ae770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:45.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.784+0000 7f36c9e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36c40abb20 0x7f36c41571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:45.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.784+0000 7f36c9e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36c415c860 0x7f36c4157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:45.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.784+0000 7f36c9e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c4157d80 con 0x7f36c415c860 2024-03-23T03:51:45.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.784+0000 7f36c9e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c4158060 con 0x7f36c40abb20 2024-03-23T03:51:45.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.784+0000 7f36c9e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c4158340 con 0x7f36c406b3b0 2024-03-23T03:51:45.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.785+0000 7f36c9400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36c415c860 0x7f36c4157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:45.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.785+0000 7f36c9400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36c415c860 0x7f36c4157720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48912/0 (socket says 172.21.15.102:48912) 2024-03-23T03:51:45.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.785+0000 7f36c9400640 1 -- 172.21.15.102:0/684536036 learned_addr learned my addr 172.21.15.102:0/684536036 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:45.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.785+0000 7f36c8a00640 1 --2- 172.21.15.102:0/684536036 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f36c406b3b0 0x7f36c40ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:45.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.785+0000 7f36bbe00640 1 --2- 172.21.15.102:0/684536036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36c40abb20 0x7f36c41571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:45.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.785+0000 7f36c9400640 1 -- 172.21.15.102:0/684536036 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f36c406b3b0 msgr2=0x7f36c40ae770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:45.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.785+0000 7f36c9400640 1 --2- 172.21.15.102:0/684536036 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f36c406b3b0 0x7f36c40ae770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.785+0000 7f36c9400640 1 -- 172.21.15.102:0/684536036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36c40abb20 msgr2=0x7f36c41571e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:45.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.785+0000 7f36c9400640 1 --2- 172.21.15.102:0/684536036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36c40abb20 0x7f36c41571e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.785+0000 7f36c9400640 1 -- 172.21.15.102:0/684536036 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b4009a70 con 0x7f36c415c860 2024-03-23T03:51:45.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.785+0000 7f36c9400640 1 --2- 172.21.15.102:0/684536036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36c415c860 0x7f36c4157720 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f36bc00c3c0 tx=0x7f36bc00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:45.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.786+0000 7f36c1600640 1 -- 172.21.15.102:0/684536036 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc005ce0 con 0x7f36c415c860 2024-03-23T03:51:45.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.786+0000 7f36c1600640 1 -- 172.21.15.102:0/684536036 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36bc005e40 con 0x7f36c415c860 2024-03-23T03:51:45.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.786+0000 7f36c1600640 1 -- 172.21.15.102:0/684536036 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36bc0072d0 con 0x7f36c415c860 2024-03-23T03:51:45.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.786+0000 7f36c9e00640 1 -- 172.21.15.102:0/684536036 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c4158620 con 0x7f36c415c860 2024-03-23T03:51:45.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.786+0000 7f36c9e00640 1 -- 172.21.15.102:0/684536036 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c4147c60 con 0x7f36c415c860 2024-03-23T03:51:45.788 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.787+0000 7f36c9e00640 1 -- 172.21.15.102:0/684536036 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c4073e40 con 0x7f36c415c860 2024-03-23T03:51:45.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.788+0000 7f36c1600640 1 -- 172.21.15.102:0/684536036 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f36bc0074f0 con 0x7f36c415c860 2024-03-23T03:51:45.789 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.789+0000 7f36c1600640 1 --2- 172.21.15.102:0/684536036 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f36a006ac00 0x7f36a006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:45.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.789+0000 7f36c1600640 1 -- 172.21.15.102:0/684536036 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f36bc088990 con 0x7f36c415c860 2024-03-23T03:51:45.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.789+0000 7f36c8a00640 1 --2- 172.21.15.102:0/684536036 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f36a006ac00 0x7f36a006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:45.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.790+0000 7f36c8a00640 1 --2- 172.21.15.102:0/684536036 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f36a006ac00 0x7f36a006d0c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f36b4036000 tx=0x7f36b4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:45.794 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.794+0000 7f36c1600640 1 -- 172.21.15.102:0/684536036 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f36bc053a80 con 0x7f36c415c860 2024-03-23T03:51:45.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.975+0000 7f36c9e00640 1 -- 172.21.15.102:0/684536036 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f36c404e030 con 0x7f36c415c860 2024-03-23T03:51:45.976 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.976+0000 7f36c1600640 1 -- 172.21.15.102:0/684536036 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7f36bc0170a0 con 0x7f36c415c860 2024-03-23T03:51:45.976 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 1 2024-03-23T03:51:45.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.978+0000 7f36c9e00640 1 -- 172.21.15.102:0/684536036 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f36a006ac00 msgr2=0x7f36a006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:45.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.978+0000 7f36c9e00640 1 --2- 172.21.15.102:0/684536036 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f36a006ac00 0x7f36a006d0c0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f36b4036000 tx=0x7f36b4030000 comp rx=0 tx=0).stop 2024-03-23T03:51:45.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.978+0000 7f36c9e00640 1 -- 172.21.15.102:0/684536036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36c415c860 msgr2=0x7f36c4157720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:45.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.978+0000 7f36c9e00640 1 --2- 172.21.15.102:0/684536036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36c415c860 0x7f36c4157720 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f36bc00c3c0 tx=0x7f36bc00c7c0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.978+0000 7f36c9e00640 1 -- 172.21.15.102:0/684536036 shutdown_connections 2024-03-23T03:51:45.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.978+0000 7f36c9e00640 1 --2- 172.21.15.102:0/684536036 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f36a006ac00 0x7f36a006d0c0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.978+0000 7f36c9e00640 1 --2- 172.21.15.102:0/684536036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36c415c860 0x7f36c4157720 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.978+0000 7f36c9e00640 1 --2- 172.21.15.102:0/684536036 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f36c40abb20 0x7f36c41571e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.978+0000 7f36c9e00640 1 --2- 172.21.15.102:0/684536036 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f36c406b3b0 0x7f36c40ae770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:45.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.978+0000 7f36c9e00640 1 -- 172.21.15.102:0/684536036 >> 172.21.15.102:0/684536036 conn(0x7f36c4070860 msgr2=0x7f36c415f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:45.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.978+0000 7f36c9e00640 1 -- 172.21.15.102:0/684536036 shutdown_connections 2024-03-23T03:51:45.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:45.978+0000 7f36c9e00640 1 -- 172.21.15.102:0/684536036 wait complete. 2024-03-23T03:51:45.995 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T03:51:46.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.199+0000 7f56eaa00640 1 Processor -- start 2024-03-23T03:51:46.200 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.199+0000 7f56eaa00640 1 -- start start 2024-03-23T03:51:46.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.201+0000 7f56eaa00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f56e41456a0 0x7f56e40ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:46.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.201+0000 7f56eaa00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e40ac6d0 con 0x7f56e4145c20 2024-03-23T03:51:46.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.201+0000 7f56eaa00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e406d390 con 0x7f56e4070cd0 2024-03-23T03:51:46.203 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.201+0000 7f56eaa00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e40ad890 con 0x7f56e41456a0 2024-03-23T03:51:46.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.201+0000 7f56e8a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f56e4145c20 0x7f56e40abfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55048/0 (socket says 172.21.15.102:55048) 2024-03-23T03:51:46.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.201+0000 7f56e8a00640 1 -- 172.21.15.102:0/3800113116 learned_addr learned my addr 172.21.15.102:0/3800113116 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:46.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.201+0000 7f56e3e00640 1 --2- 172.21.15.102:0/3800113116 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f56e41456a0 0x7f56e40ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:46.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.201+0000 7f56e3400640 1 -- 172.21.15.102:0/3800113116 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288554276 0 0) 0x7f56e40ac6d0 con 0x7f56e4145c20 2024-03-23T03:51:46.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.202+0000 7f56e3400640 1 -- 172.21.15.102:0/3800113116 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56c8003520 con 0x7f56e4145c20 2024-03-23T03:51:46.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.202+0000 7f56e3400640 1 -- 172.21.15.102:0/3800113116 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1314129763 0 0) 0x7f56c8003520 con 0x7f56e4145c20 2024-03-23T03:51:46.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.202+0000 7f56e3400640 1 -- 172.21.15.102:0/3800113116 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f56e41456a0 msgr2=0x7f56e40ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:46.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.202+0000 7f56e3400640 1 --2- 172.21.15.102:0/3800113116 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f56e41456a0 0x7f56e40ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.202+0000 7f56e3400640 1 -- 172.21.15.102:0/3800113116 >> v1:172.21.15.173:6789/0 conn(0x7f56e4070cd0 legacy=0x7f56e40ab8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:46.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.202+0000 7f56e3400640 1 -- 172.21.15.102:0/3800113116 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e406d670 con 0x7f56e4145c20 2024-03-23T03:51:46.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.202+0000 7f56e3400640 1 -- 172.21.15.102:0/3800113116 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f56dc0019f0 con 0x7f56e4145c20 2024-03-23T03:51:46.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.202+0000 7f56e3400640 1 -- 172.21.15.102:0/3800113116 >> v1:172.21.15.102:6789/0 conn(0x7f56e4145c20 legacy=0x7f56e40abfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:46.204 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.202+0000 7f56e3400640 1 --2- 172.21.15.102:0/3800113116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56c80039a0 0x7f56c8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:46.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.202+0000 7f56e3400640 1 --2- 172.21.15.102:0/3800113116 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f56e41456a0 0x7f56c80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:46.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.202+0000 7f56e3400640 1 --2- 172.21.15.102:0/3800113116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56c8004d00 0x7f56c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:46.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.202+0000 7f56e3400640 1 -- 172.21.15.102:0/3800113116 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e40ad890 con 0x7f56c80039a0 2024-03-23T03:51:46.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.202+0000 7f56e3400640 1 -- 172.21.15.102:0/3800113116 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c800ba30 con 0x7f56c8004d00 2024-03-23T03:51:46.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.202+0000 7f56e3400640 1 -- 172.21.15.102:0/3800113116 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c800bbf0 con 0x7f56e41456a0 2024-03-23T03:51:46.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.202+0000 7f56e3e00640 1 --2- 172.21.15.102:0/3800113116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56c80039a0 0x7f56c8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:46.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.203+0000 7f56e3e00640 1 -- 172.21.15.102:0/3800113116 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f56e41456a0 msgr2=0x7f56c80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:46.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.203+0000 7f56e3e00640 1 --2- 172.21.15.102:0/3800113116 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f56e41456a0 0x7f56c80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.203+0000 7f56e3e00640 1 -- 172.21.15.102:0/3800113116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56c8004d00 msgr2=0x7f56c8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:46.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.203+0000 7f56e3e00640 1 --2- 172.21.15.102:0/3800113116 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56c8004d00 0x7f56c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.205 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.203+0000 7f56e3e00640 1 -- 172.21.15.102:0/3800113116 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c8003520 con 0x7f56c80039a0 2024-03-23T03:51:46.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.204+0000 7f56e3e00640 1 --2- 172.21.15.102:0/3800113116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56c80039a0 0x7f56c8003dc0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009a90 tx=0x7f56c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=610976a3e474a83 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:46.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.204+0000 7f56e3400640 1 -- 172.21.15.102:0/3800113116 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d402c6b0 con 0x7f56c80039a0 2024-03-23T03:51:46.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.204+0000 7f56e3400640 1 -- 172.21.15.102:0/3800113116 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d402c810 con 0x7f56c80039a0 2024-03-23T03:51:46.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.204+0000 7f56e3400640 1 -- 172.21.15.102:0/3800113116 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d402cba0 con 0x7f56c80039a0 2024-03-23T03:51:46.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.205+0000 7f56eaa00640 1 -- 172.21.15.102:0/3800113116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56c80039a0 msgr2=0x7f56c8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:46.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.205+0000 7f56eaa00640 1 --2- 172.21.15.102:0/3800113116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56c80039a0 0x7f56c8003dc0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009a90 tx=0x7f56c80057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.205+0000 7f56e8a00640 1 -- 172.21.15.102:0/3800113116 reap_dead start 2024-03-23T03:51:46.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.205+0000 7f56eaa00640 1 -- 172.21.15.102:0/3800113116 shutdown_connections 2024-03-23T03:51:46.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.205+0000 7f56eaa00640 1 -- 172.21.15.102:0/3800113116 >> 172.21.15.102:0/3800113116 conn(0x7f56e406a0b0 msgr2=0x7f56e406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:46.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.205+0000 7f56eaa00640 1 -- 172.21.15.102:0/3800113116 shutdown_connections 2024-03-23T03:51:46.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.205+0000 7f56eaa00640 1 -- 172.21.15.102:0/3800113116 wait complete. 2024-03-23T03:51:46.207 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.206+0000 7f56eaa00640 1 Processor -- start 2024-03-23T03:51:46.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.207+0000 7f56eaa00640 1 -- start start 2024-03-23T03:51:46.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.207+0000 7f56eaa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56c80039a0 0x7f56e415f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:46.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.207+0000 7f56eaa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56e4143470 0x7f56e4141a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:46.209 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.207+0000 7f56eaa00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f56e413eef0 0x7f56e415bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:46.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.207+0000 7f56eaa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e41f3b00 con 0x7f56c80039a0 2024-03-23T03:51:46.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.207+0000 7f56eaa00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e4141fb0 con 0x7f56e4143470 2024-03-23T03:51:46.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.207+0000 7f56eaa00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e41f2f80 con 0x7f56e413eef0 2024-03-23T03:51:46.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.208+0000 7f56e3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56c80039a0 0x7f56e415f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:46.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.208+0000 7f56e3e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56c80039a0 0x7f56e415f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48936/0 (socket says 172.21.15.102:48936) 2024-03-23T03:51:46.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.208+0000 7f56e3e00640 1 -- 172.21.15.102:0/2846463412 learned_addr learned my addr 172.21.15.102:0/2846463412 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:46.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.208+0000 7f56e3e00640 1 -- 172.21.15.102:0/2846463412 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f56e413eef0 msgr2=0x7f56e415bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:46.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.208+0000 7f56e3e00640 1 --2- 172.21.15.102:0/2846463412 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f56e413eef0 0x7f56e415bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.208+0000 7f56e3e00640 1 -- 172.21.15.102:0/2846463412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56e4143470 msgr2=0x7f56e4141a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:46.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.208+0000 7f56e3e00640 1 --2- 172.21.15.102:0/2846463412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56e4143470 0x7f56e4141a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.208+0000 7f56e3e00640 1 -- 172.21.15.102:0/2846463412 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e41f3260 con 0x7f56c80039a0 2024-03-23T03:51:46.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.208+0000 7f56e3e00640 1 --2- 172.21.15.102:0/2846463412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56c80039a0 0x7f56e415f9c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f56d402c250 tx=0x7f56d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:46.210 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.209+0000 7f56e0c00640 1 -- 172.21.15.102:0/2846463412 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d4003810 con 0x7f56c80039a0 2024-03-23T03:51:46.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.209+0000 7f56e0c00640 1 -- 172.21.15.102:0/2846463412 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d4003970 con 0x7f56c80039a0 2024-03-23T03:51:46.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.209+0000 7f56e0c00640 1 -- 172.21.15.102:0/2846463412 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d4003d00 con 0x7f56c80039a0 2024-03-23T03:51:46.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.209+0000 7f56eaa00640 1 -- 172.21.15.102:0/2846463412 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e41f3540 con 0x7f56c80039a0 2024-03-23T03:51:46.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.209+0000 7f56eaa00640 1 -- 172.21.15.102:0/2846463412 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e41f2b30 con 0x7f56c80039a0 2024-03-23T03:51:46.211 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.210+0000 7f56eaa00640 1 -- 172.21.15.102:0/2846463412 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e41f3a60 con 0x7f56c80039a0 2024-03-23T03:51:46.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.211+0000 7f56e0c00640 1 -- 172.21.15.102:0/2846463412 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f56d4034440 con 0x7f56c80039a0 2024-03-23T03:51:46.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.212+0000 7f56e0c00640 1 --2- 172.21.15.102:0/2846463412 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f56c006ac00 0x7f56c006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:46.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.212+0000 7f56e0c00640 1 -- 172.21.15.102:0/2846463412 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f56d40ab6c0 con 0x7f56c80039a0 2024-03-23T03:51:46.212 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.212+0000 7f56db400640 1 --2- 172.21.15.102:0/2846463412 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f56c006ac00 0x7f56c006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:46.213 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.213+0000 7f56db400640 1 --2- 172.21.15.102:0/2846463412 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f56c006ac00 0x7f56c006d0c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f56e4143320 tx=0x7f56cc007990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:46.217 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.216+0000 7f56e0c00640 1 -- 172.21.15.102:0/2846463412 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f56d40767a0 con 0x7f56c80039a0 2024-03-23T03:51:46.416 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.415+0000 7f56eaa00640 1 -- 172.21.15.102:0/2846463412 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f56e4073390 con 0x7f56c80039a0 2024-03-23T03:51:46.417 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.416+0000 7f56e0c00640 1 -- 172.21.15.102:0/2846463412 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v107) v1 ==== 104+0+11 (secure 0 0 0) 0x7f56d40044e0 con 0x7f56c80039a0 2024-03-23T03:51:46.417 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 32 2024-03-23T03:51:46.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.419+0000 7f56eaa00640 1 -- 172.21.15.102:0/2846463412 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f56c006ac00 msgr2=0x7f56c006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:46.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.419+0000 7f56eaa00640 1 --2- 172.21.15.102:0/2846463412 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f56c006ac00 0x7f56c006d0c0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f56e4143320 tx=0x7f56cc007990 comp rx=0 tx=0).stop 2024-03-23T03:51:46.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.419+0000 7f56eaa00640 1 -- 172.21.15.102:0/2846463412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56c80039a0 msgr2=0x7f56e415f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:46.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.419+0000 7f56eaa00640 1 --2- 172.21.15.102:0/2846463412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56c80039a0 0x7f56e415f9c0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f56d402c250 tx=0x7f56d4004730 comp rx=0 tx=0).stop 2024-03-23T03:51:46.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.419+0000 7f56eaa00640 1 -- 172.21.15.102:0/2846463412 shutdown_connections 2024-03-23T03:51:46.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.419+0000 7f56eaa00640 1 --2- 172.21.15.102:0/2846463412 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f56c006ac00 0x7f56c006d0c0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.419+0000 7f56eaa00640 1 --2- 172.21.15.102:0/2846463412 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f56e413eef0 0x7f56e415bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.419+0000 7f56eaa00640 1 --2- 172.21.15.102:0/2846463412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f56e4143470 0x7f56e4141a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.419+0000 7f56eaa00640 1 --2- 172.21.15.102:0/2846463412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f56c80039a0 0x7f56e415f9c0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.419+0000 7f56eaa00640 1 -- 172.21.15.102:0/2846463412 >> 172.21.15.102:0/2846463412 conn(0x7f56e406a0b0 msgr2=0x7f56e40b3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:46.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.419+0000 7f56eaa00640 1 -- 172.21.15.102:0/2846463412 shutdown_connections 2024-03-23T03:51:46.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.419+0000 7f56eaa00640 1 -- 172.21.15.102:0/2846463412 wait complete. 2024-03-23T03:51:46.439 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T03:51:46.648 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.648+0000 7f4441400640 1 Processor -- start 2024-03-23T03:51:46.649 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.648+0000 7f4441400640 1 -- start start 2024-03-23T03:51:46.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.649+0000 7f4441400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f443c143930 0x7f443c0cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:46.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.649+0000 7f4441400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f443c15b2b0 con 0x7f443c0cdc50 2024-03-23T03:51:46.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.649+0000 7f4441400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f443c0cc6b0 con 0x7f443c143460 2024-03-23T03:51:46.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.649+0000 7f4441400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0cc950 con 0x7f443c143930 2024-03-23T03:51:46.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.650+0000 7f4440a00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f443c0cdc50 0x7f443c070020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55054/0 (socket says 172.21.15.102:55054) 2024-03-23T03:51:46.650 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.650+0000 7f4440a00640 1 -- 172.21.15.102:0/1738234433 learned_addr learned my addr 172.21.15.102:0/1738234433 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:46.652 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.650+0000 7f4433400640 1 --2- 172.21.15.102:0/1738234433 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f443c143930 0x7f443c0cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.650+0000 7f443b400640 1 -- 172.21.15.102:0/1738234433 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418643930 0 0) 0x7f443c15b2b0 con 0x7f443c0cdc50 2024-03-23T03:51:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.650+0000 7f443b400640 1 -- 172.21.15.102:0/1738234433 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4420003520 con 0x7f443c0cdc50 2024-03-23T03:51:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.651+0000 7f443b400640 1 -- 172.21.15.102:0/1738234433 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3960636181 0 0) 0x7f4420003520 con 0x7f443c0cdc50 2024-03-23T03:51:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.651+0000 7f443b400640 1 -- 172.21.15.102:0/1738234433 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f443c143930 msgr2=0x7f443c0cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.651+0000 7f443b400640 1 --2- 172.21.15.102:0/1738234433 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f443c143930 0x7f443c0cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.651+0000 7f443b400640 1 -- 172.21.15.102:0/1738234433 >> v1:172.21.15.173:6789/0 conn(0x7f443c143460 legacy=0x7f443c15aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.651+0000 7f443b400640 1 -- 172.21.15.102:0/1738234433 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c0ccc30 con 0x7f443c0cdc50 2024-03-23T03:51:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.651+0000 7f443b400640 1 -- 172.21.15.102:0/1738234433 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f44340019f0 con 0x7f443c0cdc50 2024-03-23T03:51:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.651+0000 7f443b400640 1 -- 172.21.15.102:0/1738234433 >> v1:172.21.15.102:6789/0 conn(0x7f443c0cdc50 legacy=0x7f443c070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.651+0000 7f443b400640 1 --2- 172.21.15.102:0/1738234433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44200039a0 0x7f4420003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.651+0000 7f443b400640 1 --2- 172.21.15.102:0/1738234433 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44200047c0 0x7f4420004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.651+0000 7f443b400640 1 --2- 172.21.15.102:0/1738234433 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f443c143930 0x7f4420005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.651+0000 7f443b400640 1 -- 172.21.15.102:0/1738234433 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0cc950 con 0x7f44200039a0 2024-03-23T03:51:46.653 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.651+0000 7f443b400640 1 -- 172.21.15.102:0/1738234433 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442000bac0 con 0x7f44200047c0 2024-03-23T03:51:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.651+0000 7f443b400640 1 -- 172.21.15.102:0/1738234433 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f442000bc80 con 0x7f443c143930 2024-03-23T03:51:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.651+0000 7f443be00640 1 --2- 172.21.15.102:0/1738234433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44200039a0 0x7f4420003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.651+0000 7f4433400640 1 --2- 172.21.15.102:0/1738234433 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44200047c0 0x7f4420004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.651+0000 7f4440a00640 1 --2- 172.21.15.102:0/1738234433 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f443c143930 0x7f4420005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.652+0000 7f443be00640 1 -- 172.21.15.102:0/1738234433 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f443c143930 msgr2=0x7f4420005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.652+0000 7f443be00640 1 --2- 172.21.15.102:0/1738234433 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f443c143930 0x7f4420005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.652+0000 7f443be00640 1 -- 172.21.15.102:0/1738234433 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44200047c0 msgr2=0x7f4420004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.652+0000 7f443be00640 1 --2- 172.21.15.102:0/1738234433 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44200047c0 0x7f4420004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.652+0000 7f443be00640 1 -- 172.21.15.102:0/1738234433 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4420003520 con 0x7f44200039a0 2024-03-23T03:51:46.654 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.652+0000 7f4440a00640 1 --2- 172.21.15.102:0/1738234433 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f443c143930 0x7f4420005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:46.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.653+0000 7f443be00640 1 --2- 172.21.15.102:0/1738234433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44200039a0 0x7f4420003dc0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f442c001720 tx=0x7f44200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7ab96bac5bec12e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:46.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.653+0000 7f443b400640 1 -- 172.21.15.102:0/1738234433 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c0052d0 con 0x7f44200039a0 2024-03-23T03:51:46.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.653+0000 7f443b400640 1 -- 172.21.15.102:0/1738234433 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442c02e050 con 0x7f44200039a0 2024-03-23T03:51:46.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.653+0000 7f443b400640 1 -- 172.21.15.102:0/1738234433 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c0035f0 con 0x7f44200039a0 2024-03-23T03:51:46.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.653+0000 7f4441400640 1 -- 172.21.15.102:0/1738234433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44200039a0 msgr2=0x7f4420003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:46.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.653+0000 7f4441400640 1 --2- 172.21.15.102:0/1738234433 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44200039a0 0x7f4420003dc0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f442c001720 tx=0x7f44200057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.653+0000 7f4440a00640 1 -- 172.21.15.102:0/1738234433 reap_dead start 2024-03-23T03:51:46.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.653+0000 7f4441400640 1 -- 172.21.15.102:0/1738234433 shutdown_connections 2024-03-23T03:51:46.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.653+0000 7f4441400640 1 -- 172.21.15.102:0/1738234433 >> 172.21.15.102:0/1738234433 conn(0x7f443c065df0 msgr2=0x7f443c069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:46.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.653+0000 7f4441400640 1 -- 172.21.15.102:0/1738234433 shutdown_connections 2024-03-23T03:51:46.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.653+0000 7f4441400640 1 -- 172.21.15.102:0/1738234433 wait complete. 2024-03-23T03:51:46.656 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.655+0000 7f4441400640 1 Processor -- start 2024-03-23T03:51:46.657 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.655+0000 7f4441400640 1 -- start start 2024-03-23T03:51:46.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.656+0000 7f4441400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44200039a0 0x7f443c1f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:46.658 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.656+0000 7f4441400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f443c0ac370 0x7f443c1f0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:46.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.656+0000 7f4441400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f443c0c33e0 0x7f443c0c3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:46.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.656+0000 7f4441400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0c3da0 con 0x7f44200039a0 2024-03-23T03:51:46.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.656+0000 7f4441400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0ad310 con 0x7f443c0c33e0 2024-03-23T03:51:46.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.656+0000 7f4441400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443c12fae0 con 0x7f443c0ac370 2024-03-23T03:51:46.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.656+0000 7f443be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44200039a0 0x7f443c1f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:46.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.657+0000 7f443be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44200039a0 0x7f443c1f3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48962/0 (socket says 172.21.15.102:48962) 2024-03-23T03:51:46.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.657+0000 7f443be00640 1 -- 172.21.15.102:0/2975074520 learned_addr learned my addr 172.21.15.102:0/2975074520 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:46.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.657+0000 7f4440a00640 1 --2- 172.21.15.102:0/2975074520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f443c0c33e0 0x7f443c0c3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:46.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.657+0000 7f4433400640 1 --2- 172.21.15.102:0/2975074520 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f443c0ac370 0x7f443c1f0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:46.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.657+0000 7f443be00640 1 -- 172.21.15.102:0/2975074520 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f443c0ac370 msgr2=0x7f443c1f0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:46.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.657+0000 7f443be00640 1 --2- 172.21.15.102:0/2975074520 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f443c0ac370 0x7f443c1f0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.657+0000 7f443be00640 1 -- 172.21.15.102:0/2975074520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f443c0c33e0 msgr2=0x7f443c0c3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:46.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.657+0000 7f443be00640 1 --2- 172.21.15.102:0/2975074520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f443c0c33e0 0x7f443c0c3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.659 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.657+0000 7f443be00640 1 -- 172.21.15.102:0/2975074520 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c0ac7f0 con 0x7f44200039a0 2024-03-23T03:51:46.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.657+0000 7f443be00640 1 --2- 172.21.15.102:0/2975074520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44200039a0 0x7f443c1f3a10 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f442c005040 tx=0x7f442c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:46.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.658+0000 7f4438c00640 1 -- 172.21.15.102:0/2975074520 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c0076c0 con 0x7f44200039a0 2024-03-23T03:51:46.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.658+0000 7f4438c00640 1 -- 172.21.15.102:0/2975074520 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442c007a30 con 0x7f44200039a0 2024-03-23T03:51:46.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.658+0000 7f4441400640 1 -- 172.21.15.102:0/2975074520 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f443c0aca70 con 0x7f44200039a0 2024-03-23T03:51:46.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.658+0000 7f4438c00640 1 -- 172.21.15.102:0/2975074520 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c006be0 con 0x7f44200039a0 2024-03-23T03:51:46.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.658+0000 7f4441400640 1 -- 172.21.15.102:0/2975074520 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443c0ad030 con 0x7f44200039a0 2024-03-23T03:51:46.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.659+0000 7f4441400640 1 -- 172.21.15.102:0/2975074520 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f443c12f8b0 con 0x7f44200039a0 2024-03-23T03:51:46.660 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.660+0000 7f4438c00640 1 -- 172.21.15.102:0/2975074520 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f442c006d40 con 0x7f44200039a0 2024-03-23T03:51:46.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.661+0000 7f4438c00640 1 --2- 172.21.15.102:0/2975074520 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f441806ac00 0x7f441806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:46.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.661+0000 7f4438c00640 1 -- 172.21.15.102:0/2975074520 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f442c0ab1a0 con 0x7f44200039a0 2024-03-23T03:51:46.661 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.661+0000 7f4433400640 1 --2- 172.21.15.102:0/2975074520 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f441806ac00 0x7f441806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:46.662 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.661+0000 7f4433400640 1 --2- 172.21.15.102:0/2975074520 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f441806ac00 0x7f441806d0c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f443c13fbe0 tx=0x7f4424008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:46.665 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.665+0000 7f4438c00640 1 -- 172.21.15.102:0/2975074520 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f443c12f8b0 con 0x7f44200039a0 2024-03-23T03:51:46.861 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.860+0000 7f4441400640 1 -- 172.21.15.102:0/2975074520 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f443c12f8b0 con 0x7f44200039a0 2024-03-23T03:51:46.862 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.861+0000 7f4438c00640 1 -- 172.21.15.102:0/2975074520 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v107) v1 ==== 100+0+11 (secure 0 0 0) 0x7f443c12f8b0 con 0x7f44200039a0 2024-03-23T03:51:46.862 INFO:teuthology.orchestra.run.smithi102.stdout:pg_num: 64 2024-03-23T03:51:46.864 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.864+0000 7f4441400640 1 -- 172.21.15.102:0/2975074520 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f441806ac00 msgr2=0x7f441806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:46.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.864+0000 7f4441400640 1 --2- 172.21.15.102:0/2975074520 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f441806ac00 0x7f441806d0c0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f443c13fbe0 tx=0x7f4424008040 comp rx=0 tx=0).stop 2024-03-23T03:51:46.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.864+0000 7f4441400640 1 -- 172.21.15.102:0/2975074520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44200039a0 msgr2=0x7f443c1f3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:46.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.864+0000 7f4441400640 1 --2- 172.21.15.102:0/2975074520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44200039a0 0x7f443c1f3a10 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f442c005040 tx=0x7f442c001120 comp rx=0 tx=0).stop 2024-03-23T03:51:46.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.864+0000 7f4441400640 1 -- 172.21.15.102:0/2975074520 shutdown_connections 2024-03-23T03:51:46.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.864+0000 7f4441400640 1 --2- 172.21.15.102:0/2975074520 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f441806ac00 0x7f441806d0c0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.865 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.864+0000 7f4441400640 1 --2- 172.21.15.102:0/2975074520 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f443c0c33e0 0x7f443c0c3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.864+0000 7f4441400640 1 --2- 172.21.15.102:0/2975074520 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f443c0ac370 0x7f443c1f0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.864+0000 7f4441400640 1 --2- 172.21.15.102:0/2975074520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44200039a0 0x7f443c1f3a10 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:46.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.864+0000 7f4441400640 1 -- 172.21.15.102:0/2975074520 >> 172.21.15.102:0/2975074520 conn(0x7f443c065df0 msgr2=0x7f443c0b1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:46.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.865+0000 7f4441400640 1 -- 172.21.15.102:0/2975074520 shutdown_connections 2024-03-23T03:51:46.866 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:46.865+0000 7f4441400640 1 -- 172.21.15.102:0/2975074520 wait complete. 2024-03-23T03:51:46.883 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:51:47.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.086+0000 7f418fe00640 1 Processor -- start 2024-03-23T03:51:47.087 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.086+0000 7f418fe00640 1 -- start start 2024-03-23T03:51:47.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.087+0000 7f418fe00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f418815b560 0x7f418815b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:47.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.087+0000 7f418fe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418815f320 con 0x7f41880b39b0 2024-03-23T03:51:47.088 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.088+0000 7f418fe00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f418806c6f0 con 0x7f418815d410 2024-03-23T03:51:47.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.088+0000 7f418fe00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f418806c9d0 con 0x7f418815b560 2024-03-23T03:51:47.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.088+0000 7f418d400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f418815b560 0x7f418815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:47.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.088+0000 7f418d400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f418815b560 0x7f418815b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48976/0 (socket says 172.21.15.102:48976) 2024-03-23T03:51:47.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.088+0000 7f418ca00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f41880b39b0 0x7f418815e500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:55056/0 (socket says 172.21.15.102:55056) 2024-03-23T03:51:47.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.088+0000 7f418d400640 1 -- 172.21.15.102:0/3438487361 learned_addr learned my addr 172.21.15.102:0/3438487361 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:47.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.088+0000 7f418d400640 1 -- 172.21.15.102:0/3438487361 >> v1:172.21.15.173:6789/0 conn(0x7f418815d410 legacy=0x7f418815ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:47.089 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.088+0000 7f418d400640 1 -- 172.21.15.102:0/3438487361 >> v1:172.21.15.102:6789/0 conn(0x7f41880b39b0 legacy=0x7f418815e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:47.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.088+0000 7f418d400640 1 -- 172.21.15.102:0/3438487361 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f418806ccb0 con 0x7f418815b560 2024-03-23T03:51:47.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.088+0000 7f4187e00640 1 -- 172.21.15.102:0/3438487361 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646592605 0 0) 0x7f418815f320 con 0x7f41880b39b0 2024-03-23T03:51:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.090+0000 7f418d400640 1 --2- 172.21.15.102:0/3438487361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f418815b560 0x7f418815b940 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f417c009870 tx=0x7f417c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b889b8ffd3e09f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.090+0000 7f4187e00640 1 -- 172.21.15.102:0/3438487361 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c031870 con 0x7f418815b560 2024-03-23T03:51:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.090+0000 7f4187e00640 1 -- 172.21.15.102:0/3438487361 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c0319d0 con 0x7f418815b560 2024-03-23T03:51:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.090+0000 7f4187e00640 1 -- 172.21.15.102:0/3438487361 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c031d60 con 0x7f418815b560 2024-03-23T03:51:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.090+0000 7f418fe00640 1 -- 172.21.15.102:0/3438487361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f418815b560 msgr2=0x7f418815b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.090+0000 7f418fe00640 1 --2- 172.21.15.102:0/3438487361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f418815b560 0x7f418815b940 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f417c009870 tx=0x7f417c02b910 comp rx=0 tx=0).stop 2024-03-23T03:51:47.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.091+0000 7f418fe00640 1 -- 172.21.15.102:0/3438487361 shutdown_connections 2024-03-23T03:51:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.091+0000 7f418fe00640 1 --2- 172.21.15.102:0/3438487361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f418815b560 0x7f418815b940 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.091+0000 7f418fe00640 1 -- 172.21.15.102:0/3438487361 >> 172.21.15.102:0/3438487361 conn(0x7f418806a3d0 msgr2=0x7f41880b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.091+0000 7f418fe00640 1 -- 172.21.15.102:0/3438487361 shutdown_connections 2024-03-23T03:51:47.092 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.091+0000 7f418fe00640 1 -- 172.21.15.102:0/3438487361 wait complete. 2024-03-23T03:51:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.092+0000 7f418fe00640 1 Processor -- start 2024-03-23T03:51:47.093 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.092+0000 7f418fe00640 1 -- start start 2024-03-23T03:51:47.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.093+0000 7f418fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41880b39b0 0x7f4188142f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.093+0000 7f418fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f418815b560 0x7f41881558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.093+0000 7f418fe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f418815d410 0x7f41880cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.093+0000 7f418fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41881434b0 con 0x7f41880b39b0 2024-03-23T03:51:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.093+0000 7f418fe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4188152140 con 0x7f418815b560 2024-03-23T03:51:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.093+0000 7f418fe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4188152420 con 0x7f418815d410 2024-03-23T03:51:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.094+0000 7f418d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41880b39b0 0x7f4188142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.094+0000 7f418d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41880b39b0 0x7f4188142f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48984/0 (socket says 172.21.15.102:48984) 2024-03-23T03:51:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.094+0000 7f418d400640 1 -- 172.21.15.102:0/2538904160 learned_addr learned my addr 172.21.15.102:0/2538904160 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.094+0000 7f418d400640 1 -- 172.21.15.102:0/2538904160 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f418815d410 msgr2=0x7f41880cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.094+0000 7f418d400640 1 --2- 172.21.15.102:0/2538904160 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f418815d410 0x7f41880cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.094+0000 7f418d400640 1 -- 172.21.15.102:0/2538904160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f418815b560 msgr2=0x7f41881558e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.094+0000 7f418ca00640 1 --2- 172.21.15.102:0/2538904160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f418815b560 0x7f41881558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.094+0000 7f418d400640 1 --2- 172.21.15.102:0/2538904160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f418815b560 0x7f41881558e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.094+0000 7f418d400640 1 -- 172.21.15.102:0/2538904160 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417c0094d0 con 0x7f41880b39b0 2024-03-23T03:51:47.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.094+0000 7f418d400640 1 --2- 172.21.15.102:0/2538904160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41880b39b0 0x7f4188142f70 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f417c009870 tx=0x7f417c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.095+0000 7f4185600640 1 -- 172.21.15.102:0/2538904160 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c03ad30 con 0x7f41880b39b0 2024-03-23T03:51:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.095+0000 7f4185600640 1 -- 172.21.15.102:0/2538904160 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c03ae90 con 0x7f41880b39b0 2024-03-23T03:51:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.095+0000 7f418fe00640 1 -- 172.21.15.102:0/2538904160 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4188152700 con 0x7f41880b39b0 2024-03-23T03:51:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.095+0000 7f4185600640 1 -- 172.21.15.102:0/2538904160 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c044630 con 0x7f41880b39b0 2024-03-23T03:51:47.096 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.095+0000 7f418fe00640 1 -- 172.21.15.102:0/2538904160 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4188152cc0 con 0x7f41880b39b0 2024-03-23T03:51:47.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.096+0000 7f418fe00640 1 -- 172.21.15.102:0/2538904160 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418806ac20 con 0x7f41880b39b0 2024-03-23T03:51:47.097 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.097+0000 7f4185600640 1 -- 172.21.15.102:0/2538904160 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f417c04d450 con 0x7f41880b39b0 2024-03-23T03:51:47.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.098+0000 7f4185600640 1 --2- 172.21.15.102:0/2538904160 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f416806ac30 0x7f416806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:47.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.098+0000 7f4185600640 1 -- 172.21.15.102:0/2538904160 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f417c02d040 con 0x7f41880b39b0 2024-03-23T03:51:47.098 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.098+0000 7f418ca00640 1 --2- 172.21.15.102:0/2538904160 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f416806ac30 0x7f416806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:47.099 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.098+0000 7f418ca00640 1 --2- 172.21.15.102:0/2538904160 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f416806ac30 0x7f416806d0f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4174007fd0 tx=0x7f4174009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:47.102 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.102+0000 7f4185600640 1 -- 172.21.15.102:0/2538904160 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f417c0ade60 con 0x7f41880b39b0 2024-03-23T03:51:47.346 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.345+0000 7f418fe00640 1 -- 172.21.15.102:0/2538904160 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4188152eb0 con 0x7f41880b39b0 2024-03-23T03:51:47.347 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.347+0000 7f4185600640 1 -- 172.21.15.102:0/2538904160 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f417c076c40 con 0x7f41880b39b0 2024-03-23T03:51:47.348 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:47.348 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2017025570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2017025570},{"type":"v1","addr":"172.21.15.173:6835","nonce":2017025570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/173893445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":173893445},{"type":"v1","addr":"172.21.15.102:6835","nonce":173893445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/255507815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":255507815},{"type":"v1","addr":"172.21.15.173:6833","nonce":255507815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:51:22.228808+0000","modified":"2024-03-23T03:51:25.724445+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7212":{"gid":7212,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/883705787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":883705787},{"type":"v1","addr":"172.21.15.102:6837","nonce":883705787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T03:51:47.348 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 56 2024-03-23T03:51:47.350 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.349+0000 7f418fe00640 1 -- 172.21.15.102:0/2538904160 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f416806ac30 msgr2=0x7f416806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:47.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.349+0000 7f418fe00640 1 --2- 172.21.15.102:0/2538904160 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f416806ac30 0x7f416806d0f0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4174007fd0 tx=0x7f4174009040 comp rx=0 tx=0).stop 2024-03-23T03:51:47.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.349+0000 7f418fe00640 1 -- 172.21.15.102:0/2538904160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41880b39b0 msgr2=0x7f4188142f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:47.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.349+0000 7f418fe00640 1 --2- 172.21.15.102:0/2538904160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41880b39b0 0x7f4188142f70 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f417c009870 tx=0x7f417c0046c0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.350+0000 7f418fe00640 1 -- 172.21.15.102:0/2538904160 shutdown_connections 2024-03-23T03:51:47.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.350+0000 7f418fe00640 1 --2- 172.21.15.102:0/2538904160 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f416806ac30 0x7f416806d0f0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.350+0000 7f418fe00640 1 --2- 172.21.15.102:0/2538904160 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f418815d410 0x7f41880cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.350+0000 7f418fe00640 1 --2- 172.21.15.102:0/2538904160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f418815b560 0x7f41881558e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.350+0000 7f418fe00640 1 --2- 172.21.15.102:0/2538904160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f41880b39b0 0x7f4188142f70 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.350+0000 7f418fe00640 1 -- 172.21.15.102:0/2538904160 >> 172.21.15.102:0/2538904160 conn(0x7f418806a3d0 msgr2=0x7f41880b2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:47.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.350+0000 7f418fe00640 1 -- 172.21.15.102:0/2538904160 shutdown_connections 2024-03-23T03:51:47.351 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.350+0000 7f418fe00640 1 -- 172.21.15.102:0/2538904160 wait complete. 2024-03-23T03:51:47.370 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:51:47.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.576+0000 7f08d2e00640 1 Processor -- start 2024-03-23T03:51:47.577 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.576+0000 7f08d2e00640 1 -- start start 2024-03-23T03:51:47.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.578+0000 7f08d2e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f08cc0b39b0 0x7f08cc0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:47.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.578+0000 7f08d2e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f08cc15c400 0x7f08cc15c830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:47.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.578+0000 7f08d2e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08cc06c6f0 con 0x7f08cc15b560 2024-03-23T03:51:47.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.578+0000 7f08d2e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08cc0b4370 con 0x7f08cc15c400 2024-03-23T03:51:47.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.578+0000 7f08d2e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08cc06c9d0 con 0x7f08cc0b39b0 2024-03-23T03:51:47.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.578+0000 7f08d0e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f08cc15c400 0x7f08cc15c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:47.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.578+0000 7f08d0e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f08cc15c400 0x7f08cc15c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48988/0 (socket says 172.21.15.102:48988) 2024-03-23T03:51:47.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.578+0000 7f08d0e00640 1 -- 172.21.15.102:0/847295411 learned_addr learned my addr 172.21.15.102:0/847295411 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:47.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.579+0000 7f08cb400640 1 --2- 172.21.15.102:0/847295411 >> v2:172.21.15.173:3300/0 conn(0x7f08cc0b39b0 0x7f08cc0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:47.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.579+0000 7f08d0e00640 1 -- 172.21.15.102:0/847295411 >> v2:172.21.15.173:3300/0 conn(0x7f08cc0b39b0 msgr2=0x7f08cc0b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:47.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.579+0000 7f08d0e00640 1 --2- 172.21.15.102:0/847295411 >> v2:172.21.15.173:3300/0 conn(0x7f08cc0b39b0 0x7f08cc0b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.579+0000 7f08d0e00640 1 -- 172.21.15.102:0/847295411 >> v1:172.21.15.173:6789/0 conn(0x7f08cc15b560 legacy=0x7f08cc15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:47.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.579+0000 7f08d0e00640 1 -- 172.21.15.102:0/847295411 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08cc06ccb0 con 0x7f08cc15c400 2024-03-23T03:51:47.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.580+0000 7f08d0e00640 1 --2- 172.21.15.102:0/847295411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08cc15c400 0x7f08cc15c830 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f08cc0ce180 tx=0x7f08c402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83bdf3c048a8571 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:47.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.580+0000 7f08caa00640 1 -- 172.21.15.102:0/847295411 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c402ca80 con 0x7f08cc15c400 2024-03-23T03:51:47.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.580+0000 7f08caa00640 1 -- 172.21.15.102:0/847295411 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c402cbe0 con 0x7f08cc15c400 2024-03-23T03:51:47.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.580+0000 7f08caa00640 1 -- 172.21.15.102:0/847295411 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c40343f0 con 0x7f08cc15c400 2024-03-23T03:51:47.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.581+0000 7f08d2e00640 1 -- 172.21.15.102:0/847295411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08cc15c400 msgr2=0x7f08cc15c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:47.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.581+0000 7f08d2e00640 1 --2- 172.21.15.102:0/847295411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08cc15c400 0x7f08cc15c830 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f08cc0ce180 tx=0x7f08c402bee0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.581+0000 7f08d2e00640 1 -- 172.21.15.102:0/847295411 shutdown_connections 2024-03-23T03:51:47.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.581+0000 7f08d2e00640 1 --2- 172.21.15.102:0/847295411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08cc15c400 0x7f08cc15c830 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.581+0000 7f08d2e00640 1 --2- 172.21.15.102:0/847295411 >> v2:172.21.15.173:3300/0 conn(0x7f08cc0b39b0 0x7f08cc0b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.581+0000 7f08d2e00640 1 -- 172.21.15.102:0/847295411 >> 172.21.15.102:0/847295411 conn(0x7f08cc06a3d0 msgr2=0x7f08cc0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:47.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.581+0000 7f08d2e00640 1 -- 172.21.15.102:0/847295411 shutdown_connections 2024-03-23T03:51:47.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.581+0000 7f08d2e00640 1 -- 172.21.15.102:0/847295411 wait complete. 2024-03-23T03:51:47.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.584+0000 7f08d2e00640 1 Processor -- start 2024-03-23T03:51:47.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.584+0000 7f08d2e00640 1 -- start start 2024-03-23T03:51:47.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.584+0000 7f08d2e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f08cc0b39b0 0x7f08cc156a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:47.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.584+0000 7f08d2e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08cc15b560 0x7f08cc155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:47.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.584+0000 7f08d2e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08cc15c400 0x7f08cc1407d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:47.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.584+0000 7f08d2e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc140d10 con 0x7f08cc15c400 2024-03-23T03:51:47.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.584+0000 7f08d2e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc152140 con 0x7f08cc15b560 2024-03-23T03:51:47.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.584+0000 7f08d2e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08cc152420 con 0x7f08cc0b39b0 2024-03-23T03:51:47.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.584+0000 7f08d0e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08cc15c400 0x7f08cc1407d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:47.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.584+0000 7f08d0e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08cc15c400 0x7f08cc1407d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48992/0 (socket says 172.21.15.102:48992) 2024-03-23T03:51:47.585 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.584+0000 7f08d0e00640 1 -- 172.21.15.102:0/526136702 learned_addr learned my addr 172.21.15.102:0/526136702 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:47.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.584+0000 7f08d0e00640 1 -- 172.21.15.102:0/526136702 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f08cc0b39b0 msgr2=0x7f08cc156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:47.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.584+0000 7f08d0e00640 1 --2- 172.21.15.102:0/526136702 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f08cc0b39b0 0x7f08cc156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.585+0000 7f08d0e00640 1 -- 172.21.15.102:0/526136702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08cc15b560 msgr2=0x7f08cc155520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:47.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.585+0000 7f08d0e00640 1 --2- 172.21.15.102:0/526136702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08cc15b560 0x7f08cc155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.585+0000 7f08d0e00640 1 -- 172.21.15.102:0/526136702 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08c4009b30 con 0x7f08cc15c400 2024-03-23T03:51:47.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.585+0000 7f08d0e00640 1 --2- 172.21.15.102:0/526136702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08cc15c400 0x7f08cc1407d0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f08c4036040 tx=0x7f08c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:47.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.585+0000 7f08c3e00640 1 -- 172.21.15.102:0/526136702 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c4009510 con 0x7f08cc15c400 2024-03-23T03:51:47.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.585+0000 7f08c3e00640 1 -- 172.21.15.102:0/526136702 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08c403e450 con 0x7f08cc15c400 2024-03-23T03:51:47.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.585+0000 7f08d2e00640 1 -- 172.21.15.102:0/526136702 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08cc152700 con 0x7f08cc15c400 2024-03-23T03:51:47.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.585+0000 7f08c3e00640 1 -- 172.21.15.102:0/526136702 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08c403e7e0 con 0x7f08cc15c400 2024-03-23T03:51:47.587 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.585+0000 7f08d2e00640 1 -- 172.21.15.102:0/526136702 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08cc152cc0 con 0x7f08cc15c400 2024-03-23T03:51:47.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.587+0000 7f08d2e00640 1 -- 172.21.15.102:0/526136702 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08cc06ac20 con 0x7f08cc15c400 2024-03-23T03:51:47.588 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.587+0000 7f08c3e00640 1 -- 172.21.15.102:0/526136702 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f08c402cd50 con 0x7f08cc15c400 2024-03-23T03:51:47.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.589+0000 7f08c3e00640 1 --2- 172.21.15.102:0/526136702 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f08ac06ac00 0x7f08ac06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:47.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.589+0000 7f08c3e00640 1 -- 172.21.15.102:0/526136702 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f08c40ac060 con 0x7f08cc15c400 2024-03-23T03:51:47.589 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.589+0000 7f08cbe00640 1 --2- 172.21.15.102:0/526136702 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f08ac06ac00 0x7f08ac06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:47.590 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.590+0000 7f08cbe00640 1 --2- 172.21.15.102:0/526136702 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f08ac06ac00 0x7f08ac06d0c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f08bc001560 tx=0x7f08bc007990 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:47.594 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.593+0000 7f08c3e00640 1 -- 172.21.15.102:0/526136702 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f08c40770a0 con 0x7f08cc15c400 2024-03-23T03:51:47.790 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.789+0000 7f08d2e00640 1 -- 172.21.15.102:0/526136702 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08cc152eb0 con 0x7f08cc15c400 2024-03-23T03:51:47.792 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.791+0000 7f08c3e00640 1 -- 172.21.15.102:0/526136702 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16501 (secure 0 0 0) 0x7f08c407a6c0 con 0x7f08cc15c400 2024-03-23T03:51:47.792 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:47.793 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":107,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:51:43.745674+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:51:19.796618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-23T03:51:21.283447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.8","mappings":[{"from":0,"to":7}]},{"pgid":"16.39","mappings":[{"from":1,"to":4}]},{"pgid":"16.3d","mappings":[{"from":5,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:51:47.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.794+0000 7f08d2e00640 1 -- 172.21.15.102:0/526136702 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f08ac06ac00 msgr2=0x7f08ac06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:47.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.794+0000 7f08d2e00640 1 --2- 172.21.15.102:0/526136702 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f08ac06ac00 0x7f08ac06d0c0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f08bc001560 tx=0x7f08bc007990 comp rx=0 tx=0).stop 2024-03-23T03:51:47.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.794+0000 7f08d2e00640 1 -- 172.21.15.102:0/526136702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08cc15c400 msgr2=0x7f08cc1407d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:47.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.794+0000 7f08d2e00640 1 --2- 172.21.15.102:0/526136702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08cc15c400 0x7f08cc1407d0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f08c4036040 tx=0x7f08c4004a40 comp rx=0 tx=0).stop 2024-03-23T03:51:47.795 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.794+0000 7f08d2e00640 1 -- 172.21.15.102:0/526136702 shutdown_connections 2024-03-23T03:51:47.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.794+0000 7f08d2e00640 1 --2- 172.21.15.102:0/526136702 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f08ac06ac00 0x7f08ac06d0c0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.795+0000 7f08d2e00640 1 --2- 172.21.15.102:0/526136702 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f08cc15c400 0x7f08cc1407d0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.795+0000 7f08d2e00640 1 --2- 172.21.15.102:0/526136702 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08cc15b560 0x7f08cc155520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.795+0000 7f08d2e00640 1 --2- 172.21.15.102:0/526136702 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f08cc0b39b0 0x7f08cc156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:47.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.795+0000 7f08d2e00640 1 -- 172.21.15.102:0/526136702 >> 172.21.15.102:0/526136702 conn(0x7f08cc06a3d0 msgr2=0x7f08cc15a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:47.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.795+0000 7f08d2e00640 1 -- 172.21.15.102:0/526136702 shutdown_connections 2024-03-23T03:51:47.796 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:47.795+0000 7f08d2e00640 1 -- 172.21.15.102:0/526136702 wait complete. 2024-03-23T03:51:47.815 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T03:51:47.816 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:51:48.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.017+0000 7efdfdc00640 1 Processor -- start 2024-03-23T03:51:48.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.017+0000 7efdfdc00640 1 -- start start 2024-03-23T03:51:48.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.018+0000 7efdfdc00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7efdf80adbc0 0x7efdf80b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.018+0000 7efdfdc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdf806d390 0x7efdf806d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.018+0000 7efdfdc00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf80b16a0 con 0x7efdf80cbab0 2024-03-23T03:51:48.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.019+0000 7efdfdc00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf806dd40 con 0x7efdf80adbc0 2024-03-23T03:51:48.019 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.019+0000 7efdfdc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf80b4700 con 0x7efdf806d390 2024-03-23T03:51:48.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.019+0000 7efdf6a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7efdf80adbc0 0x7efdf80b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.019+0000 7efdee000640 1 --1- >> v1:172.21.15.173:6789/0 conn(0x7efdf80cbab0 0x7efdf80b20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.173:6789/0 says I am v1:172.21.15.102:55886/0 (socket says 172.21.15.102:55886) 2024-03-23T03:51:48.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.019+0000 7efdf7400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdf806d390 0x7efdf806d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.021+0000 7efdee000640 1 -- 172.21.15.102:0/3712430793 learned_addr learned my addr 172.21.15.102:0/3712430793 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:48.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.021+0000 7efdf6a00640 1 -- 172.21.15.102:0/3712430793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdf806d390 msgr2=0x7efdf806d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:48.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.021+0000 7efdf6a00640 1 --2- 172.21.15.102:0/3712430793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdf806d390 0x7efdf806d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.021+0000 7efdf6a00640 1 -- 172.21.15.102:0/3712430793 >> v1:172.21.15.173:6789/0 conn(0x7efdf80cbab0 legacy=0x7efdf80b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:48.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.021+0000 7efdf6a00640 1 -- 172.21.15.102:0/3712430793 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf80cbfb0 con 0x7efdf80adbc0 2024-03-23T03:51:48.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.022+0000 7efdf6a00640 1 --2- 172.21.15.102:0/3712430793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf80adbc0 0x7efdf80b41c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7efde80097a0 tx=0x7efde802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80d7218254578547 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:48.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.022+0000 7efdf6000640 1 -- 172.21.15.102:0/3712430793 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde8031780 con 0x7efdf80adbc0 2024-03-23T03:51:48.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.022+0000 7efdf6000640 1 -- 172.21.15.102:0/3712430793 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde80318e0 con 0x7efdf80adbc0 2024-03-23T03:51:48.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.022+0000 7efdf6000640 1 -- 172.21.15.102:0/3712430793 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde8031c70 con 0x7efdf80adbc0 2024-03-23T03:51:48.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.022+0000 7efdfdc00640 1 -- 172.21.15.102:0/3712430793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf80adbc0 msgr2=0x7efdf80b41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:48.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.022+0000 7efdfdc00640 1 --2- 172.21.15.102:0/3712430793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf80adbc0 0x7efdf80b41c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7efde80097a0 tx=0x7efde802b820 comp rx=0 tx=0).stop 2024-03-23T03:51:48.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.023+0000 7efdfdc00640 1 -- 172.21.15.102:0/3712430793 shutdown_connections 2024-03-23T03:51:48.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.023+0000 7efdfdc00640 1 --2- 172.21.15.102:0/3712430793 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdf806d390 0x7efdf806d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.023+0000 7efdfdc00640 1 --2- 172.21.15.102:0/3712430793 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf80adbc0 0x7efdf80b41c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.023+0000 7efdfdc00640 1 -- 172.21.15.102:0/3712430793 >> 172.21.15.102:0/3712430793 conn(0x7efdf806a3d0 msgr2=0x7efdf80b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:48.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.023+0000 7efdfdc00640 1 -- 172.21.15.102:0/3712430793 shutdown_connections 2024-03-23T03:51:48.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.023+0000 7efdfdc00640 1 -- 172.21.15.102:0/3712430793 wait complete. 2024-03-23T03:51:48.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.024+0000 7efdfdc00640 1 Processor -- start 2024-03-23T03:51:48.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.025+0000 7efdfdc00640 1 -- start start 2024-03-23T03:51:48.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.025+0000 7efdfdc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdf806d390 0x7efdf8140690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.025+0000 7efdfdc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdf80adbc0 0x7efdf80ab850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.025+0000 7efdfdc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf80cbab0 0x7efdf81f38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.025+0000 7efdfdc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf80a7c00 con 0x7efdf80adbc0 2024-03-23T03:51:48.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.025+0000 7efdfdc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf813fb00 con 0x7efdf80cbab0 2024-03-23T03:51:48.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.025+0000 7efdfdc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf81f4480 con 0x7efdf806d390 2024-03-23T03:51:48.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.026+0000 7efdee000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdf80adbc0 0x7efdf80ab850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.026+0000 7efdee000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdf80adbc0 0x7efdf80ab850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48998/0 (socket says 172.21.15.102:48998) 2024-03-23T03:51:48.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.026+0000 7efdee000640 1 -- 172.21.15.102:0/2587098763 learned_addr learned my addr 172.21.15.102:0/2587098763 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:48.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.026+0000 7efdf6a00640 1 --2- 172.21.15.102:0/2587098763 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdf806d390 0x7efdf8140690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.026+0000 7efdf7400640 1 --2- 172.21.15.102:0/2587098763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf80cbab0 0x7efdf81f38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.026+0000 7efdee000640 1 -- 172.21.15.102:0/2587098763 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdf806d390 msgr2=0x7efdf8140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:48.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.026+0000 7efdee000640 1 --2- 172.21.15.102:0/2587098763 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdf806d390 0x7efdf8140690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.026+0000 7efdee000640 1 -- 172.21.15.102:0/2587098763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf80cbab0 msgr2=0x7efdf81f38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:48.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.026+0000 7efdee000640 1 --2- 172.21.15.102:0/2587098763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf80cbab0 0x7efdf81f38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.026+0000 7efdee000640 1 -- 172.21.15.102:0/2587098763 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde8009400 con 0x7efdf80adbc0 2024-03-23T03:51:48.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.027+0000 7efdee000640 1 --2- 172.21.15.102:0/2587098763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdf80adbc0 0x7efdf80ab850 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7efde000c500 tx=0x7efde000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:48.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.027+0000 7efdef400640 1 -- 172.21.15.102:0/2587098763 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde0005ce0 con 0x7efdf80adbc0 2024-03-23T03:51:48.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.027+0000 7efdef400640 1 -- 172.21.15.102:0/2587098763 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde0005e40 con 0x7efdf80adbc0 2024-03-23T03:51:48.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.027+0000 7efdfdc00640 1 -- 172.21.15.102:0/2587098763 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf81f4760 con 0x7efdf80adbc0 2024-03-23T03:51:48.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.027+0000 7efdef400640 1 -- 172.21.15.102:0/2587098763 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde00072f0 con 0x7efdf80adbc0 2024-03-23T03:51:48.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.027+0000 7efdfdc00640 1 -- 172.21.15.102:0/2587098763 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf80a7820 con 0x7efdf80adbc0 2024-03-23T03:51:48.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.028+0000 7efdfdc00640 1 -- 172.21.15.102:0/2587098763 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf80a7d40 con 0x7efdf80adbc0 2024-03-23T03:51:48.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.029+0000 7efdef400640 1 -- 172.21.15.102:0/2587098763 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7efde0007450 con 0x7efdf80adbc0 2024-03-23T03:51:48.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.030+0000 7efdef400640 1 --2- 172.21.15.102:0/2587098763 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efdd406ac00 0x7efdd406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.030+0000 7efdef400640 1 -- 172.21.15.102:0/2587098763 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6116+0+0 (secure 0 0 0) 0x7efde00889d0 con 0x7efdf80adbc0 2024-03-23T03:51:48.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.030+0000 7efdf6a00640 1 --2- 172.21.15.102:0/2587098763 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efdd406ac00 0x7efdd406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.031 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.031+0000 7efdf6a00640 1 --2- 172.21.15.102:0/2587098763 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efdd406ac00 0x7efdd406d0c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7efde8036000 tx=0x7efde803d040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:48.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.035+0000 7efdef400640 1 -- 172.21.15.102:0/2587098763 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7efdf80a7d40 con 0x7efdf80adbc0 2024-03-23T03:51:48.264 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.264+0000 7efdfdc00640 1 -- 172.21.15.102:0/2587098763 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdf80a7d40 con 0x7efdf80adbc0 2024-03-23T03:51:48.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.266+0000 7efdef400640 1 -- 172.21.15.102:0/2587098763 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4739 (secure 0 0 0) 0x7efdf80a7d40 con 0x7efdf80adbc0 2024-03-23T03:51:48.265 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:48.266 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2017025570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2017025570},{"type":"v1","addr":"172.21.15.173:6835","nonce":2017025570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/173893445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":173893445},{"type":"v1","addr":"172.21.15.102:6835","nonce":173893445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/255507815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":255507815},{"type":"v1","addr":"172.21.15.173:6833","nonce":255507815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:51:22.228808+0000","modified":"2024-03-23T03:51:25.724445+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7212":{"gid":7212,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/883705787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":883705787},{"type":"v1","addr":"172.21.15.102:6837","nonce":883705787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T03:51:48.266 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 56 2024-03-23T03:51:48.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.268+0000 7efdfdc00640 1 -- 172.21.15.102:0/2587098763 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efdd406ac00 msgr2=0x7efdd406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:48.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.268+0000 7efdfdc00640 1 --2- 172.21.15.102:0/2587098763 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efdd406ac00 0x7efdd406d0c0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7efde8036000 tx=0x7efde803d040 comp rx=0 tx=0).stop 2024-03-23T03:51:48.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.268+0000 7efdfdc00640 1 -- 172.21.15.102:0/2587098763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdf80adbc0 msgr2=0x7efdf80ab850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:48.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.268+0000 7efdfdc00640 1 --2- 172.21.15.102:0/2587098763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdf80adbc0 0x7efdf80ab850 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7efde000c500 tx=0x7efde000c9c0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.268 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.269+0000 7efdfdc00640 1 -- 172.21.15.102:0/2587098763 shutdown_connections 2024-03-23T03:51:48.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.269+0000 7efdfdc00640 1 --2- 172.21.15.102:0/2587098763 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7efdd406ac00 0x7efdd406d0c0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.269+0000 7efdfdc00640 1 --2- 172.21.15.102:0/2587098763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7efdf80cbab0 0x7efdf81f38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.269+0000 7efdfdc00640 1 --2- 172.21.15.102:0/2587098763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdf80adbc0 0x7efdf80ab850 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.269+0000 7efdfdc00640 1 --2- 172.21.15.102:0/2587098763 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7efdf806d390 0x7efdf8140690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.269+0000 7efdfdc00640 1 -- 172.21.15.102:0/2587098763 >> 172.21.15.102:0/2587098763 conn(0x7efdf806a3d0 msgr2=0x7efdf815d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:48.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.269+0000 7efdfdc00640 1 -- 172.21.15.102:0/2587098763 shutdown_connections 2024-03-23T03:51:48.269 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.269+0000 7efdfdc00640 1 -- 172.21.15.102:0/2587098763 wait complete. 2024-03-23T03:51:48.287 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:51:48.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.496+0000 7f9f7fe00640 1 Processor -- start 2024-03-23T03:51:48.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.496+0000 7f9f7fe00640 1 -- start start 2024-03-23T03:51:48.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.498+0000 7f9f7fe00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f9f7815b560 0x7f9f7815b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.498+0000 7f9f7fe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9f7815d410 0x7f9f780b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.498+0000 7f9f7fe00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f7806c6f0 con 0x7f9f780b39b0 2024-03-23T03:51:48.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.498+0000 7f9f7fe00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f780b4370 con 0x7f9f7815b560 2024-03-23T03:51:48.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.498+0000 7f9f7fe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7806c9d0 con 0x7f9f7815d410 2024-03-23T03:51:48.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.498+0000 7f9f7de00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9f7815d410 0x7f9f780b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.498+0000 7f9f7de00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9f7815d410 0x7f9f780b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:39290/0 (socket says 172.21.15.102:39290) 2024-03-23T03:51:48.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.498+0000 7f9f7d400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f9f7815b560 0x7f9f7815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.498+0000 7f9f7de00640 1 -- 172.21.15.102:0/1218941910 learned_addr learned my addr 172.21.15.102:0/1218941910 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:48.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.499+0000 7f9f7d400640 1 -- 172.21.15.102:0/1218941910 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9f7815d410 msgr2=0x7f9f780b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:48.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.499+0000 7f9f7d400640 1 --2- 172.21.15.102:0/1218941910 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9f7815d410 0x7f9f780b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.499+0000 7f9f7d400640 1 -- 172.21.15.102:0/1218941910 >> v1:172.21.15.173:6789/0 conn(0x7f9f780b39b0 legacy=0x7f9f7815e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:48.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.499+0000 7f9f7d400640 1 -- 172.21.15.102:0/1218941910 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f7806ccb0 con 0x7f9f7815b560 2024-03-23T03:51:48.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.501+0000 7f9f7d400640 1 --2- 172.21.15.102:0/1218941910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f7815b560 0x7f9f7815b940 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c009d90 tx=0x7f9f6c02beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fcdc3801c86bded server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:48.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.501+0000 7f9f77e00640 1 -- 172.21.15.102:0/1218941910 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c02cc60 con 0x7f9f7815b560 2024-03-23T03:51:48.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.501+0000 7f9f77e00640 1 -- 172.21.15.102:0/1218941910 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f6c02cdc0 con 0x7f9f7815b560 2024-03-23T03:51:48.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.501+0000 7f9f77e00640 1 -- 172.21.15.102:0/1218941910 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c034660 con 0x7f9f7815b560 2024-03-23T03:51:48.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.502+0000 7f9f7fe00640 1 -- 172.21.15.102:0/1218941910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f7815b560 msgr2=0x7f9f7815b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:48.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.502+0000 7f9f7fe00640 1 --2- 172.21.15.102:0/1218941910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f7815b560 0x7f9f7815b940 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c009d90 tx=0x7f9f6c02beb0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.502+0000 7f9f7fe00640 1 -- 172.21.15.102:0/1218941910 shutdown_connections 2024-03-23T03:51:48.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.502+0000 7f9f7fe00640 1 --2- 172.21.15.102:0/1218941910 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9f7815d410 0x7f9f780b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.502+0000 7f9f7fe00640 1 --2- 172.21.15.102:0/1218941910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f7815b560 0x7f9f7815b940 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.502+0000 7f9f7fe00640 1 -- 172.21.15.102:0/1218941910 >> 172.21.15.102:0/1218941910 conn(0x7f9f7806a3d0 msgr2=0x7f9f780b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:48.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.502+0000 7f9f7fe00640 1 -- 172.21.15.102:0/1218941910 shutdown_connections 2024-03-23T03:51:48.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.502+0000 7f9f7fe00640 1 -- 172.21.15.102:0/1218941910 wait complete. 2024-03-23T03:51:48.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.504+0000 7f9f7fe00640 1 Processor -- start 2024-03-23T03:51:48.503 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.504+0000 7f9f7fe00640 1 -- start start 2024-03-23T03:51:48.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.505+0000 7f9f7fe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9f780b39b0 0x7f9f78154940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.505+0000 7f9f7fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f7815b560 0x7f9f7813fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.505+0000 7f9f7fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f7815d410 0x7f9f78151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.505+0000 7f9f7fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f78154e80 con 0x7f9f7815d410 2024-03-23T03:51:48.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.505+0000 7f9f7fe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f78140410 con 0x7f9f7815b560 2024-03-23T03:51:48.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.505+0000 7f9f7fe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f78158170 con 0x7f9f780b39b0 2024-03-23T03:51:48.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.506+0000 7f9f7de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f7815d410 0x7f9f78151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.506+0000 7f9f7de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f7815d410 0x7f9f78151fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49000/0 (socket says 172.21.15.102:49000) 2024-03-23T03:51:48.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.506+0000 7f9f7de00640 1 -- 172.21.15.102:0/2333296761 learned_addr learned my addr 172.21.15.102:0/2333296761 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:48.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.506+0000 7f9f7d400640 1 --2- 172.21.15.102:0/2333296761 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9f780b39b0 0x7f9f78154940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.506+0000 7f9f7ca00640 1 --2- 172.21.15.102:0/2333296761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f7815b560 0x7f9f7813fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.506+0000 7f9f7de00640 1 -- 172.21.15.102:0/2333296761 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9f780b39b0 msgr2=0x7f9f78154940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:48.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.506+0000 7f9f7de00640 1 --2- 172.21.15.102:0/2333296761 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9f780b39b0 0x7f9f78154940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.506+0000 7f9f7de00640 1 -- 172.21.15.102:0/2333296761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f7815b560 msgr2=0x7f9f7813fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:48.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.506+0000 7f9f7de00640 1 --2- 172.21.15.102:0/2333296761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f7815b560 0x7f9f7813fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.506+0000 7f9f7de00640 1 -- 172.21.15.102:0/2333296761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6c009a70 con 0x7f9f7815d410 2024-03-23T03:51:48.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.506+0000 7f9f7de00640 1 --2- 172.21.15.102:0/2333296761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f7815d410 0x7f9f78151fa0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000e870 tx=0x7f9f7000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:48.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.507+0000 7f9f75600640 1 -- 172.21.15.102:0/2333296761 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7000ee60 con 0x7f9f7815d410 2024-03-23T03:51:48.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.507+0000 7f9f75600640 1 -- 172.21.15.102:0/2333296761 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7000c940 con 0x7f9f7815d410 2024-03-23T03:51:48.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.507+0000 7f9f75600640 1 -- 172.21.15.102:0/2333296761 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7000ccf0 con 0x7f9f7815d410 2024-03-23T03:51:48.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.507+0000 7f9f7fe00640 1 -- 172.21.15.102:0/2333296761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f78158450 con 0x7f9f7815d410 2024-03-23T03:51:48.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.507+0000 7f9f7fe00640 1 -- 172.21.15.102:0/2333296761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f78140730 con 0x7f9f7815d410 2024-03-23T03:51:48.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.508+0000 7f9f7fe00640 1 -- 172.21.15.102:0/2333296761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f7806ac20 con 0x7f9f7815d410 2024-03-23T03:51:48.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.509+0000 7f9f75600640 1 -- 172.21.15.102:0/2333296761 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f9f70013470 con 0x7f9f7815d410 2024-03-23T03:51:48.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.510+0000 7f9f75600640 1 --2- 172.21.15.102:0/2333296761 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f9f5806ac50 0x7f9f5806d110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.510+0000 7f9f75600640 1 -- 172.21.15.102:0/2333296761 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f9f700894e0 con 0x7f9f7815d410 2024-03-23T03:51:48.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.511+0000 7f9f7d400640 1 --2- 172.21.15.102:0/2333296761 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f9f5806ac50 0x7f9f5806d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.511+0000 7f9f7d400640 1 --2- 172.21.15.102:0/2333296761 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f9f5806ac50 0x7f9f5806d110 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c036000 tx=0x7f9f6c030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:48.514 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.515+0000 7f9f75600640 1 -- 172.21.15.102:0/2333296761 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f9f70054600 con 0x7f9f7815d410 2024-03-23T03:51:48.746 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.747+0000 7f9f7fe00640 1 -- 172.21.15.102:0/2333296761 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f78073390 con 0x7f9f7815d410 2024-03-23T03:51:48.747 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.748+0000 7f9f75600640 1 -- 172.21.15.102:0/2333296761 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f9f70057c20 con 0x7f9f7815d410 2024-03-23T03:51:48.747 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:48.748 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":56,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2017025570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2017025570},{"type":"v1","addr":"172.21.15.173:6835","nonce":2017025570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/173893445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":173893445},{"type":"v1","addr":"172.21.15.102:6835","nonce":173893445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/255507815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":255507815},{"type":"v1","addr":"172.21.15.173:6833","nonce":255507815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T03:51:22.228808+0000","modified":"2024-03-23T03:51:25.724445+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7212":{"gid":7212,"name":"a","rank":0,"incarnation":52,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6837/883705787","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":883705787},{"type":"v1","addr":"172.21.15.102:6837","nonce":883705787}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-03-23T03:51:48.748 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 56 2024-03-23T03:51:48.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.750+0000 7f9f7fe00640 1 -- 172.21.15.102:0/2333296761 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f9f5806ac50 msgr2=0x7f9f5806d110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:48.750 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.750+0000 7f9f7fe00640 1 --2- 172.21.15.102:0/2333296761 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f9f5806ac50 0x7f9f5806d110 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c036000 tx=0x7f9f6c030000 comp rx=0 tx=0).stop 2024-03-23T03:51:48.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.750+0000 7f9f7fe00640 1 -- 172.21.15.102:0/2333296761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f7815d410 msgr2=0x7f9f78151fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:48.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.750+0000 7f9f7fe00640 1 --2- 172.21.15.102:0/2333296761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f7815d410 0x7f9f78151fa0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f9f7000e870 tx=0x7f9f7000ed30 comp rx=0 tx=0).stop 2024-03-23T03:51:48.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.751+0000 7f9f7fe00640 1 -- 172.21.15.102:0/2333296761 shutdown_connections 2024-03-23T03:51:48.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.751+0000 7f9f7fe00640 1 --2- 172.21.15.102:0/2333296761 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f9f5806ac50 0x7f9f5806d110 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.751+0000 7f9f7fe00640 1 --2- 172.21.15.102:0/2333296761 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9f7815d410 0x7f9f78151fa0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.751+0000 7f9f7fe00640 1 --2- 172.21.15.102:0/2333296761 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f7815b560 0x7f9f7813fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.751+0000 7f9f7fe00640 1 --2- 172.21.15.102:0/2333296761 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f9f780b39b0 0x7f9f78154940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.751+0000 7f9f7fe00640 1 -- 172.21.15.102:0/2333296761 >> 172.21.15.102:0/2333296761 conn(0x7f9f7806a3d0 msgr2=0x7f9f780ae310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:48.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.751+0000 7f9f7fe00640 1 -- 172.21.15.102:0/2333296761 shutdown_connections 2024-03-23T03:51:48.751 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.751+0000 7f9f7fe00640 1 -- 172.21.15.102:0/2333296761 wait complete. 2024-03-23T03:51:48.769 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:51:48.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.981+0000 7f3bf1000640 1 Processor -- start 2024-03-23T03:51:48.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.981+0000 7f3bf1000640 1 -- start start 2024-03-23T03:51:48.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.982+0000 7f3bf1000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3bec0b39b0 0x7f3bec0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.982+0000 7f3bf1000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f3bec15c400 0x7f3bec15c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.982+0000 7f3bf1000640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bec06c6f0 con 0x7f3bec15b560 2024-03-23T03:51:48.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.982+0000 7f3bf1000640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bec0b4370 con 0x7f3bec15c400 2024-03-23T03:51:48.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.982+0000 7f3bf1000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec06c9d0 con 0x7f3bec0b39b0 2024-03-23T03:51:48.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.983+0000 7f3bea800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f3bec15c400 0x7f3bec15c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.983+0000 7f3be1400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3bec0b39b0 0x7f3bec0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.983+0000 7f3bea800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f3bec15c400 0x7f3bec15c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:38226/0 (socket says 172.21.15.102:38226) 2024-03-23T03:51:48.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.983+0000 7f3bea800640 1 -- 172.21.15.102:0/1848776773 learned_addr learned my addr 172.21.15.102:0/1848776773 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:48.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.984+0000 7f3bea800640 1 -- 172.21.15.102:0/1848776773 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3bec0b39b0 msgr2=0x7f3bec0b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:48.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.984+0000 7f3bea800640 1 --2- 172.21.15.102:0/1848776773 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3bec0b39b0 0x7f3bec0b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.984+0000 7f3bea800640 1 -- 172.21.15.102:0/1848776773 >> v1:172.21.15.173:6789/0 conn(0x7f3bec15b560 legacy=0x7f3bec15b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:48.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.984+0000 7f3bea800640 1 -- 172.21.15.102:0/1848776773 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bec06ccb0 con 0x7f3bec15c400 2024-03-23T03:51:48.984 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.985+0000 7f3bea800640 1 --2- 172.21.15.102:0/1848776773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bec15c400 0x7f3bec15c890 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009e50 tx=0x7f3be402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c631c4a4c2266ef server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:48.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.985+0000 7f3be9400640 1 -- 172.21.15.102:0/1848776773 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be402cb00 con 0x7f3bec15c400 2024-03-23T03:51:48.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.986+0000 7f3be9400640 1 -- 172.21.15.102:0/1848776773 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be402cc60 con 0x7f3bec15c400 2024-03-23T03:51:48.985 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.986+0000 7f3be9400640 1 -- 172.21.15.102:0/1848776773 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be40343f0 con 0x7f3bec15c400 2024-03-23T03:51:48.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.986+0000 7f3bf1000640 1 -- 172.21.15.102:0/1848776773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bec15c400 msgr2=0x7f3bec15c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:48.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.986+0000 7f3bf1000640 1 --2- 172.21.15.102:0/1848776773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bec15c400 0x7f3bec15c890 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009e50 tx=0x7f3be402bf60 comp rx=0 tx=0).stop 2024-03-23T03:51:48.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.986+0000 7f3bf1000640 1 -- 172.21.15.102:0/1848776773 shutdown_connections 2024-03-23T03:51:48.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.986+0000 7f3bf1000640 1 --2- 172.21.15.102:0/1848776773 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bec15c400 0x7f3bec15c890 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.986+0000 7f3bf1000640 1 --2- 172.21.15.102:0/1848776773 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3bec0b39b0 0x7f3bec0b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.986+0000 7f3bf1000640 1 -- 172.21.15.102:0/1848776773 >> 172.21.15.102:0/1848776773 conn(0x7f3bec06a3d0 msgr2=0x7f3bec0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:48.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.986+0000 7f3bf1000640 1 -- 172.21.15.102:0/1848776773 shutdown_connections 2024-03-23T03:51:48.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.986+0000 7f3bf1000640 1 -- 172.21.15.102:0/1848776773 wait complete. 2024-03-23T03:51:48.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.988+0000 7f3bf1000640 1 Processor -- start 2024-03-23T03:51:48.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.988+0000 7f3bf1000640 1 -- start start 2024-03-23T03:51:48.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.989+0000 7f3bf1000640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bec0b39b0 0x7f3bec151220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.989+0000 7f3bf1000640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3bec15b560 0x7f3bec1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.989+0000 7f3bf1000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bec15c400 0x7f3bec159ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.989+0000 7f3bf1000640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec156be0 con 0x7f3bec15c400 2024-03-23T03:51:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.989+0000 7f3bf1000640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec1535f0 con 0x7f3bec0b39b0 2024-03-23T03:51:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.989+0000 7f3bf1000640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec152d50 con 0x7f3bec15b560 2024-03-23T03:51:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.990+0000 7f3bea800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bec15c400 0x7f3bec159ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.990+0000 7f3bea800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bec15c400 0x7f3bec159ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49014/0 (socket says 172.21.15.102:49014) 2024-03-23T03:51:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.990+0000 7f3bea800640 1 -- 172.21.15.102:0/4214528658 learned_addr learned my addr 172.21.15.102:0/4214528658 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.990+0000 7f3be9e00640 1 --2- 172.21.15.102:0/4214528658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bec0b39b0 0x7f3bec151220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.990+0000 7f3be1400640 1 --2- 172.21.15.102:0/4214528658 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3bec15b560 0x7f3bec1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.990+0000 7f3bea800640 1 -- 172.21.15.102:0/4214528658 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3bec15b560 msgr2=0x7f3bec1545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.990+0000 7f3bea800640 1 --2- 172.21.15.102:0/4214528658 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3bec15b560 0x7f3bec1545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.990+0000 7f3bea800640 1 -- 172.21.15.102:0/4214528658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bec0b39b0 msgr2=0x7f3bec151220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:48.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.990+0000 7f3bea800640 1 --2- 172.21.15.102:0/4214528658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bec0b39b0 0x7f3bec151220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:48.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.990+0000 7f3bea800640 1 -- 172.21.15.102:0/4214528658 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be4009b30 con 0x7f3bec15c400 2024-03-23T03:51:48.991 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.990+0000 7f3bea800640 1 --2- 172.21.15.102:0/4214528658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bec15c400 0x7f3bec159ba0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009e50 tx=0x7f3be4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.991+0000 7f3be2a00640 1 -- 172.21.15.102:0/4214528658 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be4009510 con 0x7f3bec15c400 2024-03-23T03:51:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.991+0000 7f3be2a00640 1 -- 172.21.15.102:0/4214528658 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3be403e450 con 0x7f3bec15c400 2024-03-23T03:51:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.991+0000 7f3bf1000640 1 -- 172.21.15.102:0/4214528658 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bec153030 con 0x7f3bec15c400 2024-03-23T03:51:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.991+0000 7f3be2a00640 1 -- 172.21.15.102:0/4214528658 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3be403e800 con 0x7f3bec15c400 2024-03-23T03:51:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.991+0000 7f3bf1000640 1 -- 172.21.15.102:0/4214528658 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bec151760 con 0x7f3bec15c400 2024-03-23T03:51:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.992+0000 7f3bf1000640 1 -- 172.21.15.102:0/4214528658 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bec156940 con 0x7f3bec15c400 2024-03-23T03:51:48.992 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.993+0000 7f3be2a00640 1 -- 172.21.15.102:0/4214528658 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3be403e960 con 0x7f3bec15c400 2024-03-23T03:51:48.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.994+0000 7f3be2a00640 1 --2- 172.21.15.102:0/4214528658 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3bc806ac00 0x7f3bc806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:48.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.994+0000 7f3be2a00640 1 -- 172.21.15.102:0/4214528658 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f3be40ac1b0 con 0x7f3bec15c400 2024-03-23T03:51:48.993 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.994+0000 7f3be9e00640 1 --2- 172.21.15.102:0/4214528658 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3bc806ac00 0x7f3bc806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:48.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.995+0000 7f3be9e00640 1 --2- 172.21.15.102:0/4214528658 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3bc806ac00 0x7f3bc806d0c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3bec1555d0 tx=0x7f3bdc009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:48.998 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:48.998+0000 7f3be2a00640 1 -- 172.21.15.102:0/4214528658 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3bec156940 con 0x7f3bec15c400 2024-03-23T03:51:49.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.190+0000 7f3bf1000640 1 -- 172.21.15.102:0/4214528658 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bec156940 con 0x7f3bec15c400 2024-03-23T03:51:49.192 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.192+0000 7f3be2a00640 1 -- 172.21.15.102:0/4214528658 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+16501 (secure 0 0 0) 0x7f3bec156940 con 0x7f3bec15c400 2024-03-23T03:51:49.193 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:49.193 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":107,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:51:43.745674+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-03-23T03:51:19.796618+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-03-23T03:51:21.283447+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.8","mappings":[{"from":0,"to":7}]},{"pgid":"16.39","mappings":[{"from":1,"to":4}]},{"pgid":"16.3d","mappings":[{"from":5,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:51:49.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.195+0000 7f3bf1000640 1 -- 172.21.15.102:0/4214528658 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3bc806ac00 msgr2=0x7f3bc806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:49.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.195+0000 7f3bf1000640 1 --2- 172.21.15.102:0/4214528658 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3bc806ac00 0x7f3bc806d0c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3bec1555d0 tx=0x7f3bdc009040 comp rx=0 tx=0).stop 2024-03-23T03:51:49.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.195+0000 7f3bf1000640 1 -- 172.21.15.102:0/4214528658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bec15c400 msgr2=0x7f3bec159ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:49.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.195+0000 7f3bf1000640 1 --2- 172.21.15.102:0/4214528658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bec15c400 0x7f3bec159ba0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f3be4009e50 tx=0x7f3be4004a40 comp rx=0 tx=0).stop 2024-03-23T03:51:49.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.195+0000 7f3bf1000640 1 -- 172.21.15.102:0/4214528658 shutdown_connections 2024-03-23T03:51:49.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.195+0000 7f3bf1000640 1 --2- 172.21.15.102:0/4214528658 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f3bc806ac00 0x7f3bc806d0c0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.195+0000 7f3bf1000640 1 --2- 172.21.15.102:0/4214528658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bec15c400 0x7f3bec159ba0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.195 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.195+0000 7f3bf1000640 1 --2- 172.21.15.102:0/4214528658 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3bec15b560 0x7f3bec1545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.195+0000 7f3bf1000640 1 --2- 172.21.15.102:0/4214528658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3bec0b39b0 0x7f3bec151220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.195+0000 7f3bf1000640 1 -- 172.21.15.102:0/4214528658 >> 172.21.15.102:0/4214528658 conn(0x7f3bec06a3d0 msgr2=0x7f3bec1593d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:49.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.195+0000 7f3bf1000640 1 -- 172.21.15.102:0/4214528658 shutdown_connections 2024-03-23T03:51:49.196 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.195+0000 7f3bf1000640 1 -- 172.21.15.102:0/4214528658 wait complete. 2024-03-23T03:51:49.213 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T03:51:49.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.418+0000 7f38ee600640 1 Processor -- start 2024-03-23T03:51:49.418 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.418+0000 7f38ee600640 1 -- start start 2024-03-23T03:51:49.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.419+0000 7f38ee600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38e8159690 0x7f38e80ca7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:49.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.419+0000 7f38ee600640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f38e80cd890 0x7f38e80cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:49.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.419+0000 7f38ee600640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38e815e010 con 0x7f38e80cc330 2024-03-23T03:51:49.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.420+0000 7f38ee600640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38e80cc7f0 con 0x7f38e80cd890 2024-03-23T03:51:49.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.420+0000 7f38ee600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e80ccad0 con 0x7f38e8159690 2024-03-23T03:51:49.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.420+0000 7f38e7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f38e80cd890 0x7f38e80cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:49.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.420+0000 7f38e7e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f38e80cd890 0x7f38e80cdce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49020/0 (socket says 172.21.15.102:49020) 2024-03-23T03:51:49.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.420+0000 7f38e7e00640 1 -- 172.21.15.102:0/572450872 learned_addr learned my addr 172.21.15.102:0/572450872 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:49.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.420+0000 7f38e7400640 1 --2- 172.21.15.102:0/572450872 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38e8159690 0x7f38e80ca7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:49.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.420+0000 7f38e7e00640 1 -- 172.21.15.102:0/572450872 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38e8159690 msgr2=0x7f38e80ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:49.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.420+0000 7f38e7e00640 1 --2- 172.21.15.102:0/572450872 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38e8159690 0x7f38e80ca7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.420+0000 7f38e7e00640 1 -- 172.21.15.102:0/572450872 >> v1:172.21.15.173:6789/0 conn(0x7f38e80cc330 legacy=0x7f38e80b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:49.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.420+0000 7f38e7e00640 1 -- 172.21.15.102:0/572450872 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e80ccdb0 con 0x7f38e80cd890 2024-03-23T03:51:49.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.421+0000 7f38e7e00640 1 --2- 172.21.15.102:0/572450872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38e80cd890 0x7f38e80cdce0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f38e0009e50 tx=0x7f38e002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7c016f5c94a0351 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:49.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.422+0000 7f38e6000640 1 -- 172.21.15.102:0/572450872 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e002ca70 con 0x7f38e80cd890 2024-03-23T03:51:49.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.422+0000 7f38e6000640 1 -- 172.21.15.102:0/572450872 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38e002cbd0 con 0x7f38e80cd890 2024-03-23T03:51:49.422 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.422+0000 7f38e6000640 1 -- 172.21.15.102:0/572450872 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38e00343f0 con 0x7f38e80cd890 2024-03-23T03:51:49.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.422+0000 7f38ee600640 1 -- 172.21.15.102:0/572450872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38e80cd890 msgr2=0x7f38e80cdce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:49.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.422+0000 7f38ee600640 1 --2- 172.21.15.102:0/572450872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38e80cd890 0x7f38e80cdce0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f38e0009e50 tx=0x7f38e002bed0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.423+0000 7f38ee600640 1 -- 172.21.15.102:0/572450872 shutdown_connections 2024-03-23T03:51:49.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.423+0000 7f38ee600640 1 --2- 172.21.15.102:0/572450872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38e80cd890 0x7f38e80cdce0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.423+0000 7f38ee600640 1 --2- 172.21.15.102:0/572450872 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38e8159690 0x7f38e80ca7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.423+0000 7f38ee600640 1 -- 172.21.15.102:0/572450872 >> 172.21.15.102:0/572450872 conn(0x7f38e806a0b0 msgr2=0x7f38e806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:49.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.423+0000 7f38ee600640 1 -- 172.21.15.102:0/572450872 shutdown_connections 2024-03-23T03:51:49.423 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.423+0000 7f38ee600640 1 -- 172.21.15.102:0/572450872 wait complete. 2024-03-23T03:51:49.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.424+0000 7f38ee600640 1 Processor -- start 2024-03-23T03:51:49.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.424+0000 7f38ee600640 1 -- start start 2024-03-23T03:51:49.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.425+0000 7f38ee600640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38e80cc330 0x7f38e8148e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:49.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.425+0000 7f38ee600640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38e80cd890 0x7f38e80ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:49.426 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.425+0000 7f38ee600640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38e8159690 0x7f38e80b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:49.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.425+0000 7f38ee600640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e80b2de0 con 0x7f38e80cc330 2024-03-23T03:51:49.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.425+0000 7f38ee600640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e80b21f0 con 0x7f38e8159690 2024-03-23T03:51:49.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.425+0000 7f38ee600640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e80b2440 con 0x7f38e80cd890 2024-03-23T03:51:49.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.426+0000 7f38e7400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38e80cc330 0x7f38e8148e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:49.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.426+0000 7f38e7400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38e80cc330 0x7f38e8148e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49028/0 (socket says 172.21.15.102:49028) 2024-03-23T03:51:49.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.426+0000 7f38e7400640 1 -- 172.21.15.102:0/3818096716 learned_addr learned my addr 172.21.15.102:0/3818096716 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:49.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.426+0000 7f38e6a00640 1 --2- 172.21.15.102:0/3818096716 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38e80cd890 0x7f38e80ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:49.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.426+0000 7f38e7e00640 1 --2- 172.21.15.102:0/3818096716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38e8159690 0x7f38e80b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:49.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.426+0000 7f38e7400640 1 -- 172.21.15.102:0/3818096716 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38e80cd890 msgr2=0x7f38e80ae1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:49.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.426+0000 7f38e7400640 1 --2- 172.21.15.102:0/3818096716 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38e80cd890 0x7f38e80ae1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.426+0000 7f38e7400640 1 -- 172.21.15.102:0/3818096716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38e8159690 msgr2=0x7f38e80b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:49.427 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.426+0000 7f38e7400640 1 --2- 172.21.15.102:0/3818096716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38e8159690 0x7f38e80b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.426+0000 7f38e7400640 1 -- 172.21.15.102:0/3818096716 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e0009b30 con 0x7f38e80cc330 2024-03-23T03:51:49.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.426+0000 7f38e7e00640 1 --2- 172.21.15.102:0/3818096716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38e8159690 0x7f38e80b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:49.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.426+0000 7f38e7400640 1 --2- 172.21.15.102:0/3818096716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38e80cc330 0x7f38e8148e40 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f38d800d5f0 tx=0x7f38d800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:49.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.427+0000 7f38df400640 1 -- 172.21.15.102:0/3818096716 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d80125f0 con 0x7f38e80cc330 2024-03-23T03:51:49.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.427+0000 7f38df400640 1 -- 172.21.15.102:0/3818096716 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d8012750 con 0x7f38e80cc330 2024-03-23T03:51:49.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.427+0000 7f38df400640 1 -- 172.21.15.102:0/3818096716 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d8012b00 con 0x7f38e80cc330 2024-03-23T03:51:49.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.427+0000 7f38ee600640 1 -- 172.21.15.102:0/3818096716 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38e80b2720 con 0x7f38e80cc330 2024-03-23T03:51:49.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.427+0000 7f38ee600640 1 -- 172.21.15.102:0/3818096716 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e81495b0 con 0x7f38e80cc330 2024-03-23T03:51:49.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.428+0000 7f38ee600640 1 -- 172.21.15.102:0/3818096716 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e80b3050 con 0x7f38e80cc330 2024-03-23T03:51:49.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.429+0000 7f38df400640 1 -- 172.21.15.102:0/3818096716 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f38d8012c60 con 0x7f38e80cc330 2024-03-23T03:51:49.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.429+0000 7f38df400640 1 --2- 172.21.15.102:0/3818096716 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f38c806ac00 0x7f38c806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:49.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.430+0000 7f38df400640 1 -- 172.21.15.102:0/3818096716 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f38d8089920 con 0x7f38e80cc330 2024-03-23T03:51:49.429 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.430+0000 7f38e6a00640 1 --2- 172.21.15.102:0/3818096716 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f38c806ac00 0x7f38c806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:49.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.430+0000 7f38e6a00640 1 --2- 172.21.15.102:0/3818096716 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f38c806ac00 0x7f38c806d0c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f38e80ac0d0 tx=0x7f38d4009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:49.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.434+0000 7f38df400640 1 -- 172.21.15.102:0/3818096716 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f38d80549f0 con 0x7f38e80cc330 2024-03-23T03:51:49.667 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.667+0000 7f38ee600640 1 -- 172.21.15.102:0/3818096716 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f38e8073390 con 0x7f38e80cc330 2024-03-23T03:51:49.669 INFO:tasks.ceph.mon.a.smithi102.stderr:2024-03-23T03:51:49.669+0000 7f5aab200640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T03:51:49.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.681+0000 7f38df400640 1 -- 172.21.15.102:0/3818096716 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v57) v1 ==== 161+0+0 (secure 0 0 0) 0x7f38d8058010 con 0x7f38e80cc330 2024-03-23T03:51:49.682 INFO:teuthology.orchestra.run.smithi102.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T03:51:49.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.684+0000 7f38ee600640 1 -- 172.21.15.102:0/3818096716 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f38c806ac00 msgr2=0x7f38c806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:49.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.684+0000 7f38ee600640 1 --2- 172.21.15.102:0/3818096716 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f38c806ac00 0x7f38c806d0c0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f38e80ac0d0 tx=0x7f38d4009040 comp rx=0 tx=0).stop 2024-03-23T03:51:49.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.684+0000 7f38ee600640 1 -- 172.21.15.102:0/3818096716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38e80cc330 msgr2=0x7f38e8148e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:49.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.684+0000 7f38ee600640 1 --2- 172.21.15.102:0/3818096716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38e80cc330 0x7f38e8148e40 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f38d800d5f0 tx=0x7f38d800dab0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.684+0000 7f38ee600640 1 -- 172.21.15.102:0/3818096716 shutdown_connections 2024-03-23T03:51:49.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.684+0000 7f38ee600640 1 --2- 172.21.15.102:0/3818096716 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f38c806ac00 0x7f38c806d0c0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.684+0000 7f38ee600640 1 --2- 172.21.15.102:0/3818096716 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38e8159690 0x7f38e80b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.684+0000 7f38ee600640 1 --2- 172.21.15.102:0/3818096716 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38e80cd890 0x7f38e80ae1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.684+0000 7f38ee600640 1 --2- 172.21.15.102:0/3818096716 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38e80cc330 0x7f38e8148e40 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.684+0000 7f38ee600640 1 -- 172.21.15.102:0/3818096716 >> 172.21.15.102:0/3818096716 conn(0x7f38e806a0b0 msgr2=0x7f38e80ac810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:49.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.685+0000 7f38ee600640 1 -- 172.21.15.102:0/3818096716 shutdown_connections 2024-03-23T03:51:49.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.685+0000 7f38ee600640 1 -- 172.21.15.102:0/3818096716 wait complete. 2024-03-23T03:51:49.703 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T03:51:49.754 INFO:tasks.ceph.mds.a.smithi102.stdout:starting mds.a at 2024-03-23T03:51:49.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.885+0000 7fe04cc00640 1 Processor -- start 2024-03-23T03:51:49.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.886+0000 7fe04cc00640 1 -- start start 2024-03-23T03:51:49.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.887+0000 7fe04cc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe0480cd6b0 0x7fe04806baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:49.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.887+0000 7fe04cc00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe04815cd10 0x7fe0480cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:49.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.887+0000 7fe04cc00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fe0480b1b60 0x7fe0480d1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:49.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.887+0000 7fe04cc00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe04806b0a0 con 0x7fe04815cd10 2024-03-23T03:51:49.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.887+0000 7fe04cc00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0480cdfd0 con 0x7fe0480b1b60 2024-03-23T03:51:49.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.887+0000 7fe04cc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0480d2330 con 0x7fe0480cd6b0 2024-03-23T03:51:49.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.887+0000 7fe045400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe04815cd10 0x7fe0480cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:49.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.887+0000 7fe045400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fe04815cd10 0x7fe0480cda90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49060/0 (socket says 172.21.15.102:49060) 2024-03-23T03:51:49.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.887+0000 7fe045400640 1 -- 172.21.15.102:0/3880961730 learned_addr learned my addr 172.21.15.102:0/3880961730 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:49.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.887+0000 7fe045400640 1 -- 172.21.15.102:0/3880961730 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe0480cd6b0 msgr2=0x7fe04806baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:49.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.887+0000 7fe045400640 1 --2- 172.21.15.102:0/3880961730 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe0480cd6b0 0x7fe04806baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.887+0000 7fe045400640 1 -- 172.21.15.102:0/3880961730 >> v2:172.21.15.173:3300/0 conn(0x7fe0480b1b60 msgr2=0x7fe0480d1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:49.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.887+0000 7fe046800640 1 --2- 172.21.15.102:0/3880961730 >> v2:172.21.15.173:3300/0 conn(0x7fe0480b1b60 0x7fe0480d1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:49.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.888+0000 7fe045400640 1 --2- 172.21.15.102:0/3880961730 >> v2:172.21.15.173:3300/0 conn(0x7fe0480b1b60 0x7fe0480d1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.888+0000 7fe045400640 1 -- 172.21.15.102:0/3880961730 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0480d2610 con 0x7fe04815cd10 2024-03-23T03:51:49.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.889+0000 7fe045400640 1 --2- 172.21.15.102:0/3880961730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe04815cd10 0x7fe0480cda90 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fe04808b2c0 tx=0x7fe03402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c293f96b776b73a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:49.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.889+0000 7fe044a00640 1 -- 172.21.15.102:0/3880961730 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe034031860 con 0x7fe04815cd10 2024-03-23T03:51:49.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.889+0000 7fe044a00640 1 -- 172.21.15.102:0/3880961730 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0340319c0 con 0x7fe04815cd10 2024-03-23T03:51:49.890 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.889+0000 7fe044a00640 1 -- 172.21.15.102:0/3880961730 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe034031d50 con 0x7fe04815cd10 2024-03-23T03:51:49.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.889+0000 7fe04cc00640 1 -- 172.21.15.102:0/3880961730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe04815cd10 msgr2=0x7fe0480cda90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:49.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.889+0000 7fe04cc00640 1 --2- 172.21.15.102:0/3880961730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe04815cd10 0x7fe0480cda90 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7fe04808b2c0 tx=0x7fe03402b900 comp rx=0 tx=0).stop 2024-03-23T03:51:49.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.890+0000 7fe04cc00640 1 -- 172.21.15.102:0/3880961730 shutdown_connections 2024-03-23T03:51:49.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.890+0000 7fe04cc00640 1 --2- 172.21.15.102:0/3880961730 >> v2:172.21.15.173:3300/0 conn(0x7fe0480b1b60 0x7fe0480d1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.890+0000 7fe04cc00640 1 --2- 172.21.15.102:0/3880961730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe04815cd10 0x7fe0480cda90 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.890+0000 7fe04cc00640 1 --2- 172.21.15.102:0/3880961730 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe0480cd6b0 0x7fe04806baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.890+0000 7fe04cc00640 1 -- 172.21.15.102:0/3880961730 >> 172.21.15.102:0/3880961730 conn(0x7fe0480cf5d0 msgr2=0x7fe0480cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:49.891 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.890+0000 7fe04cc00640 1 -- 172.21.15.102:0/3880961730 shutdown_connections 2024-03-23T03:51:49.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.890+0000 7fe04cc00640 1 -- 172.21.15.102:0/3880961730 wait complete. 2024-03-23T03:51:49.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.891+0000 7fe04cc00640 1 Processor -- start 2024-03-23T03:51:49.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.891+0000 7fe04cc00640 1 -- start start 2024-03-23T03:51:49.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.892+0000 7fe04cc00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe0480b1b60 0x7fe0480c8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:49.894 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.892+0000 7fe04cc00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0480cd6b0 0x7fe0480c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:49.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.892+0000 7fe04cc00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe04815cd10 0x7fe0480cabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:49.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.892+0000 7fe04cc00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe048145d20 con 0x7fe04815cd10 2024-03-23T03:51:49.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.892+0000 7fe04cc00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04815d4d0 con 0x7fe0480cd6b0 2024-03-23T03:51:49.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.892+0000 7fe04cc00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe04815d7b0 con 0x7fe0480b1b60 2024-03-23T03:51:49.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.893+0000 7fe046800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe04815cd10 0x7fe0480cabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:49.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.893+0000 7fe046800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe04815cd10 0x7fe0480cabc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49062/0 (socket says 172.21.15.102:49062) 2024-03-23T03:51:49.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.893+0000 7fe046800640 1 -- 172.21.15.102:0/3294268637 learned_addr learned my addr 172.21.15.102:0/3294268637 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:49.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.893+0000 7fe045e00640 1 --2- 172.21.15.102:0/3294268637 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe0480b1b60 0x7fe0480c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:49.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.893+0000 7fe045400640 1 --2- 172.21.15.102:0/3294268637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0480cd6b0 0x7fe0480c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:49.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.893+0000 7fe046800640 1 -- 172.21.15.102:0/3294268637 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe0480b1b60 msgr2=0x7fe0480c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:49.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.893+0000 7fe046800640 1 --2- 172.21.15.102:0/3294268637 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe0480b1b60 0x7fe0480c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.893+0000 7fe046800640 1 -- 172.21.15.102:0/3294268637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0480cd6b0 msgr2=0x7fe0480c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:49.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.893+0000 7fe046800640 1 --2- 172.21.15.102:0/3294268637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0480cd6b0 0x7fe0480c8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:49.895 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.893+0000 7fe046800640 1 -- 172.21.15.102:0/3294268637 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0340094d0 con 0x7fe04815cd10 2024-03-23T03:51:49.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.893+0000 7fe045e00640 1 --2- 172.21.15.102:0/3294268637 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe0480b1b60 0x7fe0480c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:49.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.893+0000 7fe046800640 1 --2- 172.21.15.102:0/3294268637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe04815cd10 0x7fe0480cabc0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fe04000cd70 tx=0x7fe04000a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:49.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.894+0000 7fe03e000640 1 -- 172.21.15.102:0/3294268637 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe040003b90 con 0x7fe04815cd10 2024-03-23T03:51:49.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.894+0000 7fe03e000640 1 -- 172.21.15.102:0/3294268637 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe040003cf0 con 0x7fe04815cd10 2024-03-23T03:51:49.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.894+0000 7fe04cc00640 1 -- 172.21.15.102:0/3294268637 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe048145660 con 0x7fe04815cd10 2024-03-23T03:51:49.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.894+0000 7fe03e000640 1 -- 172.21.15.102:0/3294268637 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe040004240 con 0x7fe04815cd10 2024-03-23T03:51:49.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.894+0000 7fe04cc00640 1 -- 172.21.15.102:0/3294268637 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe04815d110 con 0x7fe04815cd10 2024-03-23T03:51:49.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.895+0000 7fe04cc00640 1 -- 172.21.15.102:0/3294268637 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe048145e60 con 0x7fe04815cd10 2024-03-23T03:51:49.896 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.896+0000 7fe03e000640 1 -- 172.21.15.102:0/3294268637 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fe0400043a0 con 0x7fe04815cd10 2024-03-23T03:51:49.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.896+0000 7fe03e000640 1 --2- 172.21.15.102:0/3294268637 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe02806ac00 0x7fe02806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:49.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.897+0000 7fe03e000640 1 -- 172.21.15.102:0/3294268637 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6202+0+0 (secure 0 0 0) 0x7fe040012070 con 0x7fe04815cd10 2024-03-23T03:51:49.897 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.897+0000 7fe045e00640 1 --2- 172.21.15.102:0/3294268637 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe02806ac00 0x7fe02806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:49.898 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.897+0000 7fe045e00640 1 --2- 172.21.15.102:0/3294268637 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe02806ac00 0x7fe02806d0c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe038006fd0 tx=0x7fe038008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:49.901 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:49.901+0000 7fe03e000640 1 -- 172.21.15.102:0/3294268637 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fe048145e60 con 0x7fe04815cd10 2024-03-23T03:51:50.117 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:50.117+0000 7fe04cc00640 1 -- 172.21.15.102:0/3294268637 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe048145e60 con 0x7fe04815cd10 2024-03-23T03:51:50.740 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:50.739+0000 7fe03e000640 1 -- 172.21.15.102:0/3294268637 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v58) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe048145e60 con 0x7fe04815cd10 2024-03-23T03:51:50.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:50.743+0000 7fe04cc00640 1 -- 172.21.15.102:0/3294268637 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe02806ac00 msgr2=0x7fe02806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:50.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:50.743+0000 7fe04cc00640 1 --2- 172.21.15.102:0/3294268637 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe02806ac00 0x7fe02806d0c0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe038006fd0 tx=0x7fe038008040 comp rx=0 tx=0).stop 2024-03-23T03:51:50.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:50.743+0000 7fe04cc00640 1 -- 172.21.15.102:0/3294268637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe04815cd10 msgr2=0x7fe0480cabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:50.743 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:50.743+0000 7fe04cc00640 1 --2- 172.21.15.102:0/3294268637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe04815cd10 0x7fe0480cabc0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7fe04000cd70 tx=0x7fe04000a660 comp rx=0 tx=0).stop 2024-03-23T03:51:50.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:50.743+0000 7fe04cc00640 1 -- 172.21.15.102:0/3294268637 shutdown_connections 2024-03-23T03:51:50.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:50.743+0000 7fe04cc00640 1 --2- 172.21.15.102:0/3294268637 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fe02806ac00 0x7fe02806d0c0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:50.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:50.743+0000 7fe04cc00640 1 --2- 172.21.15.102:0/3294268637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe04815cd10 0x7fe0480cabc0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:50.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:50.743+0000 7fe04cc00640 1 --2- 172.21.15.102:0/3294268637 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe0480cd6b0 0x7fe0480c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:50.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:50.743+0000 7fe04cc00640 1 --2- 172.21.15.102:0/3294268637 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fe0480b1b60 0x7fe0480c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:50.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:50.743+0000 7fe04cc00640 1 -- 172.21.15.102:0/3294268637 >> 172.21.15.102:0/3294268637 conn(0x7fe0480cf5d0 msgr2=0x7fe0480b6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:50.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:50.743+0000 7fe04cc00640 1 -- 172.21.15.102:0/3294268637 shutdown_connections 2024-03-23T03:51:50.745 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:50.743+0000 7fe04cc00640 1 -- 172.21.15.102:0/3294268637 wait complete. 2024-03-23T03:51:50.757 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:51:50.764 DEBUG:teuthology.orchestra.run.smithi173:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:51:50.765 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T03:51:51.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.017+0000 7fbd3aa00640 1 Processor -- start 2024-03-23T03:51:51.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.017+0000 7fbd3aa00640 1 -- start start 2024-03-23T03:51:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.018+0000 7fbd3aa00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fbd340cd740 0x7fbd340b84b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.018+0000 7fbd3aa00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd340d0d40 0x7fbd340cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.018+0000 7fbd3aa00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbd3406b0a0 0x7fbd3406b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.018+0000 7fbd3aa00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd340d20c0 con 0x7fbd3406b0a0 2024-03-23T03:51:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.018+0000 7fbd3aa00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd340ce060 con 0x7fbd340cd740 2024-03-23T03:51:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.018+0000 7fbd3aa00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd340d14b0 con 0x7fbd340d0d40 2024-03-23T03:51:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.019+0000 7fbd38a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbd3406b0a0 0x7fbd3406b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.019+0000 7fbd38a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fbd3406b0a0 0x7fbd3406b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49074/0 (socket says 172.21.15.102:49074) 2024-03-23T03:51:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.019+0000 7fbd38a00640 1 -- 172.21.15.102:0/686454826 learned_addr learned my addr 172.21.15.102:0/686454826 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.019+0000 7fbd38a00640 1 -- 172.21.15.102:0/686454826 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd340d0d40 msgr2=0x7fbd340cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.019+0000 7fbd38a00640 1 --2- 172.21.15.102:0/686454826 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd340d0d40 0x7fbd340cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.019+0000 7fbd38a00640 1 -- 172.21.15.102:0/686454826 >> v2:172.21.15.173:3300/0 conn(0x7fbd340cd740 msgr2=0x7fbd340b84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.019+0000 7fbd38a00640 1 --2- 172.21.15.102:0/686454826 >> v2:172.21.15.173:3300/0 conn(0x7fbd340cd740 0x7fbd340b84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:51.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.019+0000 7fbd38a00640 1 -- 172.21.15.102:0/686454826 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd340d1790 con 0x7fbd3406b0a0 2024-03-23T03:51:51.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.020+0000 7fbd38a00640 1 --2- 172.21.15.102:0/686454826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd3406b0a0 0x7fbd3406b530 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fbd3408a100 tx=0x7fbd2c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa6bbef605e0e81d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:51.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.021+0000 7fbd32a00640 1 -- 172.21.15.102:0/686454826 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c02ca80 con 0x7fbd3406b0a0 2024-03-23T03:51:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.021+0000 7fbd32a00640 1 -- 172.21.15.102:0/686454826 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2c02cbe0 con 0x7fbd3406b0a0 2024-03-23T03:51:51.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.021+0000 7fbd32a00640 1 -- 172.21.15.102:0/686454826 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c0343f0 con 0x7fbd3406b0a0 2024-03-23T03:51:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.021+0000 7fbd3aa00640 1 -- 172.21.15.102:0/686454826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd3406b0a0 msgr2=0x7fbd3406b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.021+0000 7fbd3aa00640 1 --2- 172.21.15.102:0/686454826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd3406b0a0 0x7fbd3406b530 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fbd3408a100 tx=0x7fbd2c02bee0 comp rx=0 tx=0).stop 2024-03-23T03:51:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.022+0000 7fbd3aa00640 1 -- 172.21.15.102:0/686454826 shutdown_connections 2024-03-23T03:51:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.022+0000 7fbd3aa00640 1 --2- 172.21.15.102:0/686454826 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd3406b0a0 0x7fbd3406b530 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.022+0000 7fbd3aa00640 1 --2- 172.21.15.102:0/686454826 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd340d0d40 0x7fbd340cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.022+0000 7fbd3aa00640 1 --2- 172.21.15.102:0/686454826 >> v2:172.21.15.173:3300/0 conn(0x7fbd340cd740 0x7fbd340b84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.022+0000 7fbd3aa00640 1 -- 172.21.15.102:0/686454826 >> 172.21.15.102:0/686454826 conn(0x7fbd340cf5d0 msgr2=0x7fbd340cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.022+0000 7fbd3aa00640 1 -- 172.21.15.102:0/686454826 shutdown_connections 2024-03-23T03:51:51.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.022+0000 7fbd3aa00640 1 -- 172.21.15.102:0/686454826 wait complete. 2024-03-23T03:51:51.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.024+0000 7fbd3aa00640 1 Processor -- start 2024-03-23T03:51:51.024 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.024+0000 7fbd3aa00640 1 -- start start 2024-03-23T03:51:51.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.025+0000 7fbd3aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd3406b0a0 0x7fbd34151760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:51.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.025+0000 7fbd3aa00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd340cd740 0x7fbd340cb4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:51.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.025+0000 7fbd3aa00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd340d0d40 0x7fbd340ab4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:51.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.025+0000 7fbd3aa00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd340a9610 con 0x7fbd3406b0a0 2024-03-23T03:51:51.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.025+0000 7fbd3aa00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd34151ca0 con 0x7fbd340d0d40 2024-03-23T03:51:51.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.025+0000 7fbd3aa00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd340a9270 con 0x7fbd340cd740 2024-03-23T03:51:51.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.025+0000 7fbd33e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd3406b0a0 0x7fbd34151760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:51.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.025+0000 7fbd33e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd3406b0a0 0x7fbd34151760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49078/0 (socket says 172.21.15.102:49078) 2024-03-23T03:51:51.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.025+0000 7fbd33e00640 1 -- 172.21.15.102:0/3089012513 learned_addr learned my addr 172.21.15.102:0/3089012513 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:51.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.026+0000 7fbd33400640 1 --2- 172.21.15.102:0/3089012513 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd340cd740 0x7fbd340cb4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:51.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.026+0000 7fbd38a00640 1 --2- 172.21.15.102:0/3089012513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd340d0d40 0x7fbd340ab4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:51.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.026+0000 7fbd33e00640 1 -- 172.21.15.102:0/3089012513 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd340cd740 msgr2=0x7fbd340cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:51.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.026+0000 7fbd33e00640 1 --2- 172.21.15.102:0/3089012513 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd340cd740 0x7fbd340cb4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:51.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.026+0000 7fbd33e00640 1 -- 172.21.15.102:0/3089012513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd340d0d40 msgr2=0x7fbd340ab4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:51.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.026+0000 7fbd33e00640 1 --2- 172.21.15.102:0/3089012513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd340d0d40 0x7fbd340ab4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:51.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.026+0000 7fbd33e00640 1 -- 172.21.15.102:0/3089012513 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2c009b30 con 0x7fbd3406b0a0 2024-03-23T03:51:51.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.026+0000 7fbd33400640 1 --2- 172.21.15.102:0/3089012513 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd340cd740 0x7fbd340cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:51.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.026+0000 7fbd33e00640 1 --2- 172.21.15.102:0/3089012513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd3406b0a0 0x7fbd34151760 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fbd2400b4a0 tx=0x7fbd2400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:51.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.026+0000 7fbd2be00640 1 -- 172.21.15.102:0/3089012513 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd24010610 con 0x7fbd3406b0a0 2024-03-23T03:51:51.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.026+0000 7fbd2be00640 1 -- 172.21.15.102:0/3089012513 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd24010770 con 0x7fbd3406b0a0 2024-03-23T03:51:51.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.026+0000 7fbd2be00640 1 -- 172.21.15.102:0/3089012513 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd24010b20 con 0x7fbd3406b0a0 2024-03-23T03:51:51.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.026+0000 7fbd3aa00640 1 -- 172.21.15.102:0/3089012513 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd340cbcf0 con 0x7fbd3406b0a0 2024-03-23T03:51:51.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.026+0000 7fbd3aa00640 1 -- 172.21.15.102:0/3089012513 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd340cc2b0 con 0x7fbd3406b0a0 2024-03-23T03:51:51.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.027+0000 7fbd3aa00640 1 -- 172.21.15.102:0/3089012513 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd34073390 con 0x7fbd3406b0a0 2024-03-23T03:51:51.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.028+0000 7fbd2be00640 1 -- 172.21.15.102:0/3089012513 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fbd24010c80 con 0x7fbd3406b0a0 2024-03-23T03:51:51.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.029+0000 7fbd2be00640 1 --2- 172.21.15.102:0/3089012513 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd1406ac00 0x7fbd1406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:51.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.029+0000 7fbd2be00640 1 -- 172.21.15.102:0/3089012513 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 6202+0+0 (secure 0 0 0) 0x7fbd240891b0 con 0x7fbd3406b0a0 2024-03-23T03:51:51.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.029+0000 7fbd33400640 1 --2- 172.21.15.102:0/3089012513 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd1406ac00 0x7fbd1406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:51.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.030+0000 7fbd33400640 1 --2- 172.21.15.102:0/3089012513 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd1406ac00 0x7fbd1406d0c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fbd340cca40 tx=0x7fbd20008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:51.034 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.034+0000 7fbd2be00640 1 -- 172.21.15.102:0/3089012513 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fbd240541a0 con 0x7fbd3406b0a0 2024-03-23T03:51:51.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.224+0000 7fbd3aa00640 1 -- 172.21.15.102:0/3089012513 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbd340a93b0 con 0x7fbd3406b0a0 2024-03-23T03:51:51.820 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.819+0000 7fbd2be00640 1 -- 172.21.15.102:0/3089012513 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v109) v1 ==== 181+0+0 (secure 0 0 0) 0x7fbd240577c0 con 0x7fbd3406b0a0 2024-03-23T03:51:51.820 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_metadata' removed 2024-03-23T03:51:51.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.824+0000 7fbd3aa00640 1 -- 172.21.15.102:0/3089012513 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd1406ac00 msgr2=0x7fbd1406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:51.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.824+0000 7fbd3aa00640 1 --2- 172.21.15.102:0/3089012513 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd1406ac00 0x7fbd1406d0c0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fbd340cca40 tx=0x7fbd20008040 comp rx=0 tx=0).stop 2024-03-23T03:51:51.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.824+0000 7fbd3aa00640 1 -- 172.21.15.102:0/3089012513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd3406b0a0 msgr2=0x7fbd34151760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:51.824 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.824+0000 7fbd3aa00640 1 --2- 172.21.15.102:0/3089012513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd3406b0a0 0x7fbd34151760 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fbd2400b4a0 tx=0x7fbd2400b960 comp rx=0 tx=0).stop 2024-03-23T03:51:51.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.824+0000 7fbd3aa00640 1 -- 172.21.15.102:0/3089012513 shutdown_connections 2024-03-23T03:51:51.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.824+0000 7fbd3aa00640 1 --2- 172.21.15.102:0/3089012513 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fbd1406ac00 0x7fbd1406d0c0 secure :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fbd340cca40 tx=0x7fbd20008040 comp rx=0 tx=0).stop 2024-03-23T03:51:51.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.824+0000 7fbd3aa00640 1 --2- 172.21.15.102:0/3089012513 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbd340d0d40 0x7fbd340ab4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:51.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.824+0000 7fbd3aa00640 1 --2- 172.21.15.102:0/3089012513 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fbd340cd740 0x7fbd340cb4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:51.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.824+0000 7fbd3aa00640 1 --2- 172.21.15.102:0/3089012513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd3406b0a0 0x7fbd34151760 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:51.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.824+0000 7fbd3aa00640 1 -- 172.21.15.102:0/3089012513 >> 172.21.15.102:0/3089012513 conn(0x7fbd340cf5d0 msgr2=0x7fbd340b64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:51.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.826+0000 7fbd3aa00640 1 -- 172.21.15.102:0/3089012513 shutdown_connections 2024-03-23T03:51:51.827 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:51.827+0000 7fbd3aa00640 1 -- 172.21.15.102:0/3089012513 wait complete. 2024-03-23T03:51:51.852 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T03:51:52.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.044+0000 7f4744a00640 1 Processor -- start 2024-03-23T03:51:52.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.044+0000 7f4744a00640 1 -- start start 2024-03-23T03:51:52.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.046+0000 7f4744a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4740159830 0x7f47400cc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:52.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.046+0000 7f4744a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f47400ce930 0x7f47400cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:52.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.046+0000 7f4744a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4740159c10 con 0x7f47400ca7b0 2024-03-23T03:51:52.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.046+0000 7f4744a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f474006de50 con 0x7f47400ce930 2024-03-23T03:51:52.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.046+0000 7f4744a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47400cf300 con 0x7f4740159830 2024-03-23T03:51:52.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.046+0000 7f473e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f47400ce930 0x7f47400cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:52.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.046+0000 7f473e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f47400ce930 0x7f47400cedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49092/0 (socket says 172.21.15.102:49092) 2024-03-23T03:51:52.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.046+0000 7f473e800640 1 -- 172.21.15.102:0/458867960 learned_addr learned my addr 172.21.15.102:0/458867960 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:52.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.047+0000 7f473de00640 1 --2- 172.21.15.102:0/458867960 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4740159830 0x7f47400cc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:52.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.047+0000 7f473e800640 1 -- 172.21.15.102:0/458867960 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4740159830 msgr2=0x7f47400cc3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:52.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.047+0000 7f473e800640 1 --2- 172.21.15.102:0/458867960 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4740159830 0x7f47400cc3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:52.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.047+0000 7f473e800640 1 -- 172.21.15.102:0/458867960 >> v1:172.21.15.173:6789/0 conn(0x7f47400ca7b0 legacy=0x7f47400cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:52.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.047+0000 7f473e800640 1 -- 172.21.15.102:0/458867960 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47400cd840 con 0x7f47400ce930 2024-03-23T03:51:52.048 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.047+0000 7f473de00640 1 --2- 172.21.15.102:0/458867960 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4740159830 0x7f47400cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:52.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.048+0000 7f473e800640 1 --2- 172.21.15.102:0/458867960 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47400ce930 0x7f47400cedc0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f4738009e50 tx=0x7f473802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=929825e0bb5f77f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:52.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.048+0000 7f473d400640 1 -- 172.21.15.102:0/458867960 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473802cb00 con 0x7f47400ce930 2024-03-23T03:51:52.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.048+0000 7f473d400640 1 -- 172.21.15.102:0/458867960 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473802cc60 con 0x7f47400ce930 2024-03-23T03:51:52.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.048+0000 7f473d400640 1 -- 172.21.15.102:0/458867960 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47380343f0 con 0x7f47400ce930 2024-03-23T03:51:52.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.049+0000 7f4744a00640 1 -- 172.21.15.102:0/458867960 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47400ce930 msgr2=0x7f47400cedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:52.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.049+0000 7f4744a00640 1 --2- 172.21.15.102:0/458867960 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47400ce930 0x7f47400cedc0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f4738009e50 tx=0x7f473802bf60 comp rx=0 tx=0).stop 2024-03-23T03:51:52.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.049+0000 7f4744a00640 1 -- 172.21.15.102:0/458867960 shutdown_connections 2024-03-23T03:51:52.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.049+0000 7f4744a00640 1 --2- 172.21.15.102:0/458867960 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47400ce930 0x7f47400cedc0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:52.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.049+0000 7f4744a00640 1 --2- 172.21.15.102:0/458867960 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4740159830 0x7f47400cc3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:52.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.049+0000 7f4744a00640 1 -- 172.21.15.102:0/458867960 >> 172.21.15.102:0/458867960 conn(0x7f474006a0b0 msgr2=0x7f474006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:52.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.049+0000 7f4744a00640 1 -- 172.21.15.102:0/458867960 shutdown_connections 2024-03-23T03:51:52.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.049+0000 7f4744a00640 1 -- 172.21.15.102:0/458867960 wait complete. 2024-03-23T03:51:52.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.050+0000 7f4744a00640 1 Processor -- start 2024-03-23T03:51:52.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.051+0000 7f4744a00640 1 -- start start 2024-03-23T03:51:52.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.051+0000 7f4744a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47400ca7b0 0x7f47401407e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:52.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.051+0000 7f4744a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f47400ce930 0x7f47401563a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:52.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.052+0000 7f4744a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4740159830 0x7f4740155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:52.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.052+0000 7f4744a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474014bbc0 con 0x7f47400ca7b0 2024-03-23T03:51:52.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.052+0000 7f4744a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474014b600 con 0x7f4740159830 2024-03-23T03:51:52.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.052+0000 7f4744a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f474014b8e0 con 0x7f47400ce930 2024-03-23T03:51:52.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.052+0000 7f473de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47400ca7b0 0x7f47401407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:52.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.052+0000 7f473de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47400ca7b0 0x7f47401407e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:49106/0 (socket says 172.21.15.102:49106) 2024-03-23T03:51:52.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.052+0000 7f473de00640 1 -- 172.21.15.102:0/2635275299 learned_addr learned my addr 172.21.15.102:0/2635275299 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:52.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.052+0000 7f473e800640 1 --2- 172.21.15.102:0/2635275299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4740159830 0x7f4740155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:52.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.052+0000 7f473de00640 1 -- 172.21.15.102:0/2635275299 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f47400ce930 msgr2=0x7f47401563a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:52.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.052+0000 7f4737e00640 1 --2- 172.21.15.102:0/2635275299 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f47400ce930 0x7f47401563a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:52.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.052+0000 7f473de00640 1 --2- 172.21.15.102:0/2635275299 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f47400ce930 0x7f47401563a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:52.054 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.052+0000 7f473de00640 1 -- 172.21.15.102:0/2635275299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4740159830 msgr2=0x7f4740155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:52.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.052+0000 7f473de00640 1 --2- 172.21.15.102:0/2635275299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4740159830 0x7f4740155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:52.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.052+0000 7f473de00640 1 -- 172.21.15.102:0/2635275299 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4738009b30 con 0x7f47400ca7b0 2024-03-23T03:51:52.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.052+0000 7f473e800640 1 --2- 172.21.15.102:0/2635275299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4740159830 0x7f4740155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:52.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.053+0000 7f473de00640 1 --2- 172.21.15.102:0/2635275299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47400ca7b0 0x7f47401407e0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f473000c7f0 tx=0x7f473000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:52.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.053+0000 7f4736000640 1 -- 172.21.15.102:0/2635275299 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4730007820 con 0x7f47400ca7b0 2024-03-23T03:51:52.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.053+0000 7f4736000640 1 -- 172.21.15.102:0/2635275299 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4730007980 con 0x7f47400ca7b0 2024-03-23T03:51:52.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.053+0000 7f4736000640 1 -- 172.21.15.102:0/2635275299 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4730007cf0 con 0x7f47400ca7b0 2024-03-23T03:51:52.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.053+0000 7f4744a00640 1 -- 172.21.15.102:0/2635275299 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47400ae4b0 con 0x7f47400ca7b0 2024-03-23T03:51:52.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.053+0000 7f4744a00640 1 -- 172.21.15.102:0/2635275299 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4740140d20 con 0x7f47400ca7b0 2024-03-23T03:51:52.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.054+0000 7f4744a00640 1 -- 172.21.15.102:0/2635275299 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47400ae310 con 0x7f47400ca7b0 2024-03-23T03:51:52.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.055+0000 7f4736000640 1 -- 172.21.15.102:0/2635275299 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f47300113f0 con 0x7f47400ca7b0 2024-03-23T03:51:52.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.057+0000 7f4736000640 1 --2- 172.21.15.102:0/2635275299 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f471c06ac00 0x7f471c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:52.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.057+0000 7f4736000640 1 -- 172.21.15.102:0/2635275299 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5764+0+0 (secure 0 0 0) 0x7f4730089620 con 0x7f47400ca7b0 2024-03-23T03:51:52.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.057+0000 7f4737e00640 1 --2- 172.21.15.102:0/2635275299 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f471c06ac00 0x7f471c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:52.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.058+0000 7f4737e00640 1 --2- 172.21.15.102:0/2635275299 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f471c06ac00 0x7f471c06d0c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f474006d5d0 tx=0x7f4728009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:52.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.061+0000 7f4736000640 1 -- 172.21.15.102:0/2635275299 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f47400ae310 con 0x7f47400ca7b0 2024-03-23T03:51:52.256 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.255+0000 7f4744a00640 1 -- 172.21.15.102:0/2635275299 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f47400ae310 con 0x7f47400ca7b0 2024-03-23T03:51:52.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.829+0000 7f4736000640 1 -- 172.21.15.102:0/2635275299 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v110) v1 ==== 169+0+0 (secure 0 0 0) 0x7f47400ae310 con 0x7f47400ca7b0 2024-03-23T03:51:52.831 INFO:teuthology.orchestra.run.smithi102.stderr:pool 'cephfs_data' removed 2024-03-23T03:51:52.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.836+0000 7f4744a00640 1 -- 172.21.15.102:0/2635275299 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f471c06ac00 msgr2=0x7f471c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:52.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.837+0000 7f4744a00640 1 --2- 172.21.15.102:0/2635275299 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f471c06ac00 0x7f471c06d0c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f474006d5d0 tx=0x7f4728009040 comp rx=0 tx=0).stop 2024-03-23T03:51:52.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.837+0000 7f4744a00640 1 -- 172.21.15.102:0/2635275299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47400ca7b0 msgr2=0x7f47401407e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:52.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.837+0000 7f4744a00640 1 --2- 172.21.15.102:0/2635275299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47400ca7b0 0x7f47401407e0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f473000c7f0 tx=0x7f473000ccb0 comp rx=0 tx=0).stop 2024-03-23T03:51:52.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.837+0000 7f4744a00640 1 -- 172.21.15.102:0/2635275299 shutdown_connections 2024-03-23T03:51:52.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.837+0000 7f4744a00640 1 --2- 172.21.15.102:0/2635275299 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f471c06ac00 0x7f471c06d0c0 secure :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f474006d5d0 tx=0x7f4728009040 comp rx=0 tx=0).stop 2024-03-23T03:51:52.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.837+0000 7f4744a00640 1 --2- 172.21.15.102:0/2635275299 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4740159830 0x7f4740155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:52.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.837+0000 7f4744a00640 1 --2- 172.21.15.102:0/2635275299 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f47400ce930 0x7f47401563a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:52.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.837+0000 7f4744a00640 1 --2- 172.21.15.102:0/2635275299 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47400ca7b0 0x7f47401407e0 secure :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f473000c7f0 tx=0x7f473000ccb0 comp rx=0 tx=0).stop 2024-03-23T03:51:52.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.837+0000 7f4744a00640 1 -- 172.21.15.102:0/2635275299 >> 172.21.15.102:0/2635275299 conn(0x7f474006a0b0 msgr2=0x7f4740150a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:52.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.840+0000 7f4744a00640 1 -- 172.21.15.102:0/2635275299 shutdown_connections 2024-03-23T03:51:52.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:52.840+0000 7f4744a00640 1 -- 172.21.15.102:0/2635275299 wait complete. 2024-03-23T03:51:52.863 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-23T03:51:53.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.052+0000 7f2bc0a00640 1 Processor -- start 2024-03-23T03:51:53.053 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.053+0000 7f2bc0a00640 1 -- start start 2024-03-23T03:51:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.054+0000 7f2bc0a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f2bbc0cf7c0 0x7f2bbc0cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.054+0000 7f2bc0a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc0ccd90 con 0x7f2bbc15c820 2024-03-23T03:51:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.054+0000 7f2bc0a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bbc0b7fc0 con 0x7f2bbc160e90 2024-03-23T03:51:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.054+0000 7f2bc0a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bbc0cdd70 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.054+0000 7f2bba800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f2bbc15c820 0x7f2bbc0cf0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54242/0 (socket says 172.21.15.102:54242) 2024-03-23T03:51:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.054+0000 7f2bba800640 1 -- 172.21.15.102:0/1163810215 learned_addr learned my addr 172.21.15.102:0/1163810215 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:53.055 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.054+0000 7f2bb1e00640 1 --2- 172.21.15.102:0/1163810215 >> v2:172.21.15.102:3300/0 conn(0x7f2bbc0cf7c0 0x7f2bbc0cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:53.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.055+0000 7f2bb1e00640 1 -- 172.21.15.102:0/1163810215 >> v1:172.21.15.173:6789/0 conn(0x7f2bbc160e90 legacy=0x7f2bbc0d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:53.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.055+0000 7f2bb1e00640 1 -- 172.21.15.102:0/1163810215 >> v1:172.21.15.102:6789/0 conn(0x7f2bbc15c820 legacy=0x7f2bbc0cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:53.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.055+0000 7f2bb1e00640 1 -- 172.21.15.102:0/1163810215 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc0b82a0 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.056 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.056+0000 7f2bb1e00640 1 --2- 172.21.15.102:0/1163810215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bbc0cf7c0 0x7f2bbc0cd830 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40097a0 tx=0x7f2ba402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982b0355ff26607d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:53.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.056+0000 7f2bb9e00640 1 -- 172.21.15.102:0/1163810215 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4030610 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.056+0000 7f2bb9e00640 1 -- 172.21.15.102:0/1163810215 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba4030770 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.057 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.056+0000 7f2bb9e00640 1 -- 172.21.15.102:0/1163810215 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba4030b00 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.057+0000 7f2bc0a00640 1 -- 172.21.15.102:0/1163810215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bbc0cf7c0 msgr2=0x7f2bbc0cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:53.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.057+0000 7f2bc0a00640 1 --2- 172.21.15.102:0/1163810215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bbc0cf7c0 0x7f2bbc0cd830 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f2ba40097a0 tx=0x7f2ba402b960 comp rx=0 tx=0).stop 2024-03-23T03:51:53.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.057+0000 7f2bc0a00640 1 -- 172.21.15.102:0/1163810215 shutdown_connections 2024-03-23T03:51:53.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.057+0000 7f2bc0a00640 1 --2- 172.21.15.102:0/1163810215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bbc0cf7c0 0x7f2bbc0cd830 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.057+0000 7f2bc0a00640 1 -- 172.21.15.102:0/1163810215 >> 172.21.15.102:0/1163810215 conn(0x7f2bbc06b050 msgr2=0x7f2bbc06b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:53.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.057+0000 7f2bc0a00640 1 -- 172.21.15.102:0/1163810215 shutdown_connections 2024-03-23T03:51:53.058 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.057+0000 7f2bc0a00640 1 -- 172.21.15.102:0/1163810215 wait complete. 2024-03-23T03:51:53.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.059+0000 7f2bc0a00640 1 Processor -- start 2024-03-23T03:51:53.061 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.060+0000 7f2bc0a00640 1 -- start start 2024-03-23T03:51:53.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.061+0000 7f2bc0a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bbc0cf7c0 0x7f2bbc1e6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:53.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.061+0000 7f2bc0a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2bbc15c820 0x7f2bbc0c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:53.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.061+0000 7f2bc0a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2bbc160e90 0x7f2bbc154880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:53.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.061+0000 7f2bc0a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc1e7530 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.061+0000 7f2bc0a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc1e7810 con 0x7f2bbc160e90 2024-03-23T03:51:53.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.061+0000 7f2bc0a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bbc1e7af0 con 0x7f2bbc15c820 2024-03-23T03:51:53.063 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.061+0000 7f2bba800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bbc0cf7c0 0x7f2bbc1e6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.061+0000 7f2bba800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bbc0cf7c0 0x7f2bbc1e6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43484/0 (socket says 172.21.15.102:43484) 2024-03-23T03:51:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.061+0000 7f2bba800640 1 -- 172.21.15.102:0/2506730493 learned_addr learned my addr 172.21.15.102:0/2506730493 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.061+0000 7f2bba800640 1 -- 172.21.15.102:0/2506730493 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2bbc15c820 msgr2=0x7f2bbc0c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.061+0000 7f2bba800640 1 --2- 172.21.15.102:0/2506730493 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2bbc15c820 0x7f2bbc0c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.061+0000 7f2bba800640 1 -- 172.21.15.102:0/2506730493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2bbc160e90 msgr2=0x7f2bbc154880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.061+0000 7f2bba800640 1 --2- 172.21.15.102:0/2506730493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2bbc160e90 0x7f2bbc154880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.061+0000 7f2bba800640 1 -- 172.21.15.102:0/2506730493 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba4009400 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.062+0000 7f2bba800640 1 --2- 172.21.15.102:0/2506730493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bbc0cf7c0 0x7f2bbc1e6c30 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00c4a0 tx=0x7f2bac00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.062+0000 7f2bb3400640 1 -- 172.21.15.102:0/2506730493 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bac005ce0 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.062+0000 7f2bb3400640 1 -- 172.21.15.102:0/2506730493 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bac005e40 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.062+0000 7f2bb3400640 1 -- 172.21.15.102:0/2506730493 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bac007270 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.062+0000 7f2bc0a00640 1 -- 172.21.15.102:0/2506730493 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bbc0b2c10 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.064 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.062+0000 7f2bc0a00640 1 -- 172.21.15.102:0/2506730493 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bbc1e7170 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.063+0000 7f2bc0a00640 1 -- 172.21.15.102:0/2506730493 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bbc0b2e00 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.065 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.064+0000 7f2bb3400640 1 -- 172.21.15.102:0/2506730493 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f2bac007490 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.065+0000 7f2bb3400640 1 --2- 172.21.15.102:0/2506730493 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2b9806ac00 0x7f2b9806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:53.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.066+0000 7f2bb3400640 1 -- 172.21.15.102:0/2506730493 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f2bac088560 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.066 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.066+0000 7f2bb1e00640 1 --2- 172.21.15.102:0/2506730493 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2b9806ac00 0x7f2b9806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:53.067 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.066+0000 7f2bb1e00640 1 --2- 172.21.15.102:0/2506730493 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2b9806ac00 0x7f2b9806d0c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4036000 tx=0x7f2ba4030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:53.070 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.070+0000 7f2bb3400640 1 -- 172.21.15.102:0/2506730493 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f2bac0574d0 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.265 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.265+0000 7f2bc0a00640 1 -- 172.21.15.102:0/2506730493 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f2bbc04e030 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.278+0000 7f2bb3400640 1 -- 172.21.15.102:0/2506730493 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2bac057060 con 0x7f2bbc0cf7c0 2024-03-23T03:51:53.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.282+0000 7f2bc0a00640 1 -- 172.21.15.102:0/2506730493 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2b9806ac00 msgr2=0x7f2b9806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:53.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.282+0000 7f2bc0a00640 1 --2- 172.21.15.102:0/2506730493 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2b9806ac00 0x7f2b9806d0c0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2ba4036000 tx=0x7f2ba4030000 comp rx=0 tx=0).stop 2024-03-23T03:51:53.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.282+0000 7f2bc0a00640 1 -- 172.21.15.102:0/2506730493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bbc0cf7c0 msgr2=0x7f2bbc1e6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:53.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.282+0000 7f2bc0a00640 1 --2- 172.21.15.102:0/2506730493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bbc0cf7c0 0x7f2bbc1e6c30 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7f2bac00c4a0 tx=0x7f2bac00c960 comp rx=0 tx=0).stop 2024-03-23T03:51:53.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.282+0000 7f2bc0a00640 1 -- 172.21.15.102:0/2506730493 shutdown_connections 2024-03-23T03:51:53.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.282+0000 7f2bc0a00640 1 --2- 172.21.15.102:0/2506730493 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f2b9806ac00 0x7f2b9806d0c0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.282+0000 7f2bc0a00640 1 --2- 172.21.15.102:0/2506730493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2bbc160e90 0x7f2bbc154880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.282+0000 7f2bc0a00640 1 --2- 172.21.15.102:0/2506730493 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f2bbc15c820 0x7f2bbc0c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.282+0000 7f2bc0a00640 1 --2- 172.21.15.102:0/2506730493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bbc0cf7c0 0x7f2bbc1e6c30 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.282+0000 7f2bc0a00640 1 -- 172.21.15.102:0/2506730493 >> 172.21.15.102:0/2506730493 conn(0x7f2bbc06b050 msgr2=0x7f2bbc06b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:53.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.282+0000 7f2bc0a00640 1 -- 172.21.15.102:0/2506730493 shutdown_connections 2024-03-23T03:51:53.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.282+0000 7f2bc0a00640 1 -- 172.21.15.102:0/2506730493 wait complete. 2024-03-23T03:51:53.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_write' 2024-03-23T03:51:53.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.495+0000 7f4e50e00640 1 Processor -- start 2024-03-23T03:51:53.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.496+0000 7f4e50e00640 1 -- start start 2024-03-23T03:51:53.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.497+0000 7f4e50e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4e4c13faa0 0x7f4e4c160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:53.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.497+0000 7f4e50e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f4e4c160800 0x7f4e4c0b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.497+0000 7f4e50e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4e4c0b1be0 0x7f4e4c1422b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.497+0000 7f4e50e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e4c0b2030 con 0x7f4e4c0b1be0 2024-03-23T03:51:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.497+0000 7f4e50e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e4c0b38a0 con 0x7f4e4c160800 2024-03-23T03:51:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.497+0000 7f4e50e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c0b3b80 con 0x7f4e4c13faa0 2024-03-23T03:51:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.498+0000 7f4e4a800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4e4c0b1be0 0x7f4e4c1422b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.498+0000 7f4e4a800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4e4c0b1be0 0x7f4e4c1422b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43498/0 (socket says 172.21.15.102:43498) 2024-03-23T03:51:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.498+0000 7f4e4a800640 1 -- 172.21.15.102:0/823920891 learned_addr learned my addr 172.21.15.102:0/823920891 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.498+0000 7f4e49e00640 1 --2- 172.21.15.102:0/823920891 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4e4c13faa0 0x7f4e4c160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.498+0000 7f4e41400640 1 --2- 172.21.15.102:0/823920891 >> v2:172.21.15.173:3300/0 conn(0x7f4e4c160800 0x7f4e4c0b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.498+0000 7f4e4a800640 1 -- 172.21.15.102:0/823920891 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4e4c13faa0 msgr2=0x7f4e4c160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.498+0000 7f4e4a800640 1 --2- 172.21.15.102:0/823920891 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4e4c13faa0 0x7f4e4c160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.498+0000 7f4e4a800640 1 -- 172.21.15.102:0/823920891 >> v2:172.21.15.173:3300/0 conn(0x7f4e4c160800 msgr2=0x7f4e4c0b16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.498+0000 7f4e4a800640 1 --2- 172.21.15.102:0/823920891 >> v2:172.21.15.173:3300/0 conn(0x7f4e4c160800 0x7f4e4c0b16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.498+0000 7f4e4a800640 1 -- 172.21.15.102:0/823920891 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4c0b3e60 con 0x7f4e4c0b1be0 2024-03-23T03:51:53.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.499+0000 7f4e4a800640 1 --2- 172.21.15.102:0/823920891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e4c0b1be0 0x7f4e4c1422b0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c071470 tx=0x7f4e4402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb9b31ce855cd500 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:53.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.499+0000 7f4e49400640 1 -- 172.21.15.102:0/823920891 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e4402ca80 con 0x7f4e4c0b1be0 2024-03-23T03:51:53.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.500+0000 7f4e49400640 1 -- 172.21.15.102:0/823920891 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e4402cbe0 con 0x7f4e4c0b1be0 2024-03-23T03:51:53.500 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.500+0000 7f4e49400640 1 -- 172.21.15.102:0/823920891 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e440343f0 con 0x7f4e4c0b1be0 2024-03-23T03:51:53.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.500+0000 7f4e50e00640 1 -- 172.21.15.102:0/823920891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e4c0b1be0 msgr2=0x7f4e4c1422b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:53.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.500+0000 7f4e50e00640 1 --2- 172.21.15.102:0/823920891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e4c0b1be0 0x7f4e4c1422b0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f4e4c071470 tx=0x7f4e4402bee0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.500+0000 7f4e50e00640 1 -- 172.21.15.102:0/823920891 shutdown_connections 2024-03-23T03:51:53.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.500+0000 7f4e50e00640 1 --2- 172.21.15.102:0/823920891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e4c0b1be0 0x7f4e4c1422b0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.500+0000 7f4e50e00640 1 --2- 172.21.15.102:0/823920891 >> v2:172.21.15.173:3300/0 conn(0x7f4e4c160800 0x7f4e4c0b16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.500+0000 7f4e50e00640 1 --2- 172.21.15.102:0/823920891 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4e4c13faa0 0x7f4e4c160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.500+0000 7f4e50e00640 1 -- 172.21.15.102:0/823920891 >> 172.21.15.102:0/823920891 conn(0x7f4e4c06a3d0 msgr2=0x7f4e4c0b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:53.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.500+0000 7f4e50e00640 1 -- 172.21.15.102:0/823920891 shutdown_connections 2024-03-23T03:51:53.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.500+0000 7f4e50e00640 1 -- 172.21.15.102:0/823920891 wait complete. 2024-03-23T03:51:53.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.502+0000 7f4e50e00640 1 Processor -- start 2024-03-23T03:51:53.502 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.502+0000 7f4e50e00640 1 -- start start 2024-03-23T03:51:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.503+0000 7f4e50e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4e4c0b1be0 0x7f4e4c13cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.503+0000 7f4e50e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e4c13faa0 0x7f4e4c1409a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.503+0000 7f4e50e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e4c160800 0x7f4e4c142450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.503+0000 7f4e50e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c1422d0 con 0x7f4e4c13faa0 2024-03-23T03:51:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.503+0000 7f4e50e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c13d490 con 0x7f4e4c160800 2024-03-23T03:51:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.503+0000 7f4e50e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4c141ac0 con 0x7f4e4c0b1be0 2024-03-23T03:51:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.503+0000 7f4e41400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e4c13faa0 0x7f4e4c1409a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.503+0000 7f4e41400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e4c13faa0 0x7f4e4c1409a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43512/0 (socket says 172.21.15.102:43512) 2024-03-23T03:51:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.503+0000 7f4e41400640 1 -- 172.21.15.102:0/3857016379 learned_addr learned my addr 172.21.15.102:0/3857016379 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:53.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.504+0000 7f4e49e00640 1 --2- 172.21.15.102:0/3857016379 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4e4c0b1be0 0x7f4e4c13cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:53.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.504+0000 7f4e41400640 1 -- 172.21.15.102:0/3857016379 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4e4c0b1be0 msgr2=0x7f4e4c13cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:53.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.504+0000 7f4e4a800640 1 --2- 172.21.15.102:0/3857016379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e4c160800 0x7f4e4c142450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:53.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.504+0000 7f4e41400640 1 --2- 172.21.15.102:0/3857016379 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4e4c0b1be0 0x7f4e4c13cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.504+0000 7f4e41400640 1 -- 172.21.15.102:0/3857016379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e4c160800 msgr2=0x7f4e4c142450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:53.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.504+0000 7f4e41400640 1 --2- 172.21.15.102:0/3857016379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e4c160800 0x7f4e4c142450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.504+0000 7f4e41400640 1 -- 172.21.15.102:0/3857016379 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e44009b30 con 0x7f4e4c13faa0 2024-03-23T03:51:53.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.504+0000 7f4e49e00640 1 --2- 172.21.15.102:0/3857016379 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4e4c0b1be0 0x7f4e4c13cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:53.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.504+0000 7f4e41400640 1 --2- 172.21.15.102:0/3857016379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e4c13faa0 0x7f4e4c1409a0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400c7f0 tx=0x7f4e3400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:53.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.504+0000 7f4e42a00640 1 -- 172.21.15.102:0/3857016379 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e340077c0 con 0x7f4e4c13faa0 2024-03-23T03:51:53.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.504+0000 7f4e42a00640 1 -- 172.21.15.102:0/3857016379 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e34007920 con 0x7f4e4c13faa0 2024-03-23T03:51:53.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.504+0000 7f4e42a00640 1 -- 172.21.15.102:0/3857016379 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e34007c90 con 0x7f4e4c13faa0 2024-03-23T03:51:53.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.504+0000 7f4e50e00640 1 -- 172.21.15.102:0/3857016379 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4c140510 con 0x7f4e4c13faa0 2024-03-23T03:51:53.506 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.505+0000 7f4e50e00640 1 -- 172.21.15.102:0/3857016379 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e4c141ff0 con 0x7f4e4c13faa0 2024-03-23T03:51:53.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.506+0000 7f4e50e00640 1 -- 172.21.15.102:0/3857016379 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e4c1406a0 con 0x7f4e4c13faa0 2024-03-23T03:51:53.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.507+0000 7f4e42a00640 1 -- 172.21.15.102:0/3857016379 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4e340113f0 con 0x7f4e4c13faa0 2024-03-23T03:51:53.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.508+0000 7f4e42a00640 1 --2- 172.21.15.102:0/3857016379 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4e2806ac30 0x7f4e2806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:53.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.508+0000 7f4e42a00640 1 -- 172.21.15.102:0/3857016379 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f4e34088f70 con 0x7f4e4c13faa0 2024-03-23T03:51:53.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.508+0000 7f4e49e00640 1 --2- 172.21.15.102:0/3857016379 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4e2806ac30 0x7f4e2806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:53.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.509+0000 7f4e49e00640 1 --2- 172.21.15.102:0/3857016379 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4e2806ac30 0x7f4e2806d0f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c006fd0 tx=0x7f4e3c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:53.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.513+0000 7f4e42a00640 1 -- 172.21.15.102:0/3857016379 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f4e340bc580 con 0x7f4e4c13faa0 2024-03-23T03:51:53.777 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.777+0000 7f4e50e00640 1 -- 172.21.15.102:0/3857016379 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_write"]} v 0) v1 -- 0x7f4e4c073390 con 0x7f4e4c13faa0 2024-03-23T03:51:53.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.833+0000 7f4e42a00640 1 -- 172.21.15.102:0/3857016379 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_quota.TestQuota.test_remote_update_write"]}]=0 v290) v1 ==== 137+0+0 (secure 0 0 0) 0x7f4e34054310 con 0x7f4e4c13faa0 2024-03-23T03:51:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.838+0000 7f4e50e00640 1 -- 172.21.15.102:0/3857016379 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4e2806ac30 msgr2=0x7f4e2806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.838+0000 7f4e50e00640 1 --2- 172.21.15.102:0/3857016379 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4e2806ac30 0x7f4e2806d0f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c006fd0 tx=0x7f4e3c008040 comp rx=0 tx=0).stop 2024-03-23T03:51:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.838+0000 7f4e50e00640 1 -- 172.21.15.102:0/3857016379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e4c13faa0 msgr2=0x7f4e4c1409a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.838+0000 7f4e50e00640 1 --2- 172.21.15.102:0/3857016379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e4c13faa0 0x7f4e4c1409a0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400c7f0 tx=0x7f4e3400ccb0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.838+0000 7f4e50e00640 1 -- 172.21.15.102:0/3857016379 shutdown_connections 2024-03-23T03:51:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.838+0000 7f4e50e00640 1 --2- 172.21.15.102:0/3857016379 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4e2806ac30 0x7f4e2806d0f0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.838+0000 7f4e50e00640 1 --2- 172.21.15.102:0/3857016379 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e4c160800 0x7f4e4c142450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.838+0000 7f4e50e00640 1 --2- 172.21.15.102:0/3857016379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4e4c13faa0 0x7f4e4c1409a0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.838+0000 7f4e50e00640 1 --2- 172.21.15.102:0/3857016379 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4e4c0b1be0 0x7f4e4c13cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.838+0000 7f4e50e00640 1 -- 172.21.15.102:0/3857016379 >> 172.21.15.102:0/3857016379 conn(0x7f4e4c06a3d0 msgr2=0x7f4e4c15fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:53.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.838+0000 7f4e50e00640 1 -- 172.21.15.102:0/3857016379 shutdown_connections 2024-03-23T03:51:53.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:53.838+0000 7f4e50e00640 1 -- 172.21.15.102:0/3857016379 wait complete. 2024-03-23T03:51:53.857 INFO:tasks.cephfs_test_runner:That when a client modifies the quota on a directory used ... ok 2024-03-23T03:51:53.857 INFO:tasks.cephfs_test_runner: 2024-03-23T03:51:53.857 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-23T03:51:53.857 INFO:tasks.cephfs_test_runner:Ran 6 tests in 242.514s 2024-03-23T03:51:53.857 INFO:tasks.cephfs_test_runner: 2024-03-23T03:51:53.857 INFO:tasks.cephfs_test_runner:OK 2024-03-23T03:51:53.858 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-23T03:51:53.868 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-23T03:51:53.875 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-23T03:51:53.875 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:51:53.875 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:53.927 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:51:53.928 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-23T03:51:53.928 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:53.978 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053428,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249180k,nr_inodes=812295,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498364k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T03:51:53.979 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:51:53.979 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:54.038 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:51:54.038 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-23T03:51:54.038 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T03:51:54.087 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053432,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249184k,nr_inodes=812296,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498368k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T03:51:54.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:54.088 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-23T03:51:54.145 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T03:51:54.145 INFO:teuthology.orchestra.run.smithi102.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T03:51:54.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:54.146 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T03:51:54.241 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:54.242 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T03:51:54.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:54.328 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link delete ceph-brx 2024-03-23T03:51:54.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:54.452 DEBUG:teuthology.orchestra.run.smithi102:> ip netns list 2024-03-23T03:51:54.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:54.499 DEBUG:teuthology.orchestra.run.smithi102:> sudo ip link delete ceph-brx 2024-03-23T03:51:54.636 INFO:teuthology.orchestra.run.smithi102.stderr:Cannot find device "ceph-brx" 2024-03-23T03:51:54.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:51:54.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:54.642 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list 2024-03-23T03:51:54.700 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-23T03:51:54.700 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-23T03:51:54.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:54.701 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T03:51:54.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:54.797 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-23T03:51:54.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:54.880 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link delete ceph-brx 2024-03-23T03:51:55.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:55.011 DEBUG:teuthology.orchestra.run.smithi173:> ip netns list 2024-03-23T03:51:55.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T03:51:55.056 DEBUG:teuthology.orchestra.run.smithi173:> sudo ip link delete ceph-brx 2024-03-23T03:51:55.196 INFO:teuthology.orchestra.run.smithi173.stderr:Cannot find device "ceph-brx" 2024-03-23T03:51:55.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:51:55.200 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-23T03:51:55.210 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-23T03:51:55.210 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:51:55.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.427+0000 7f71f5400640 1 Processor -- start 2024-03-23T03:51:55.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.427+0000 7f71f5400640 1 -- start start 2024-03-23T03:51:55.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.429+0000 7f71f5400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71f00b39b0 0x7f71f00b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:55.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.429+0000 7f71f5400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f71f015c400 0x7f71f015c890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:55.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.429+0000 7f71f5400640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f006c6f0 con 0x7f71f015b560 2024-03-23T03:51:55.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.429+0000 7f71f5400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f00b4370 con 0x7f71f015c400 2024-03-23T03:51:55.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.429+0000 7f71f5400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f006c9d0 con 0x7f71f00b39b0 2024-03-23T03:51:55.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.430+0000 7f71ef200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f71f015c400 0x7f71f015c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:55.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.430+0000 7f71ede00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71f00b39b0 0x7f71f00b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:55.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.430+0000 7f71ef200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f71f015c400 0x7f71f015c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:53228/0 (socket says 172.21.15.102:53228) 2024-03-23T03:51:55.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.430+0000 7f71ede00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71f00b39b0 0x7f71f00b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:46204/0 (socket says 172.21.15.102:46204) 2024-03-23T03:51:55.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.430+0000 7f71ef200640 1 -- 172.21.15.102:0/2111801966 learned_addr learned my addr 172.21.15.102:0/2111801966 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:55.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.431+0000 7f71ef200640 1 -- 172.21.15.102:0/2111801966 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71f00b39b0 msgr2=0x7f71f00b3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:55.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.431+0000 7f71ef200640 1 --2- 172.21.15.102:0/2111801966 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71f00b39b0 0x7f71f00b3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:55.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.431+0000 7f71ef200640 1 -- 172.21.15.102:0/2111801966 >> v1:172.21.15.173:6789/0 conn(0x7f71f015b560 legacy=0x7f71f015b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:55.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.431+0000 7f71ef200640 1 -- 172.21.15.102:0/2111801966 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f006ccb0 con 0x7f71f015c400 2024-03-23T03:51:55.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.431+0000 7f71ede00640 1 --2- 172.21.15.102:0/2111801966 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71f00b39b0 0x7f71f00b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:51:55.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.432+0000 7f71ef200640 1 --2- 172.21.15.102:0/2111801966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f71f015c400 0x7f71f015c890 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f71e8009870 tx=0x7f71e802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec9f3144dc3e51cc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:55.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.432+0000 7f71ed400640 1 -- 172.21.15.102:0/2111801966 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e8030840 con 0x7f71f015c400 2024-03-23T03:51:55.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.433+0000 7f71ed400640 1 -- 172.21.15.102:0/2111801966 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e80309a0 con 0x7f71f015c400 2024-03-23T03:51:55.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.433+0000 7f71ed400640 1 -- 172.21.15.102:0/2111801966 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e8030d30 con 0x7f71f015c400 2024-03-23T03:51:55.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.433+0000 7f71f5400640 1 -- 172.21.15.102:0/2111801966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f71f015c400 msgr2=0x7f71f015c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:55.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.433+0000 7f71f5400640 1 --2- 172.21.15.102:0/2111801966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f71f015c400 0x7f71f015c890 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f71e8009870 tx=0x7f71e802b980 comp rx=0 tx=0).stop 2024-03-23T03:51:55.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.433+0000 7f71f5400640 1 -- 172.21.15.102:0/2111801966 shutdown_connections 2024-03-23T03:51:55.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.433+0000 7f71f5400640 1 --2- 172.21.15.102:0/2111801966 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f71f015c400 0x7f71f015c890 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:55.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.433+0000 7f71f5400640 1 --2- 172.21.15.102:0/2111801966 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71f00b39b0 0x7f71f00b3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:55.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.433+0000 7f71f5400640 1 -- 172.21.15.102:0/2111801966 >> 172.21.15.102:0/2111801966 conn(0x7f71f006a3d0 msgr2=0x7f71f00b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:55.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.433+0000 7f71f5400640 1 -- 172.21.15.102:0/2111801966 shutdown_connections 2024-03-23T03:51:55.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.433+0000 7f71f5400640 1 -- 172.21.15.102:0/2111801966 wait complete. 2024-03-23T03:51:55.435 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.435+0000 7f71f5400640 1 Processor -- start 2024-03-23T03:51:55.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.435+0000 7f71f5400640 1 -- start start 2024-03-23T03:51:55.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.436+0000 7f71f5400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71f00b39b0 0x7f71f013e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.436+0000 7f71f5400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71f015b560 0x7f71f00af360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.436+0000 7f71f5400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f71f015c400 0x7f71f00af8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.436+0000 7f71f5400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f013cbb0 con 0x7f71f00b39b0 2024-03-23T03:51:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.436+0000 7f71f5400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f013fd30 con 0x7f71f015c400 2024-03-23T03:51:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.436+0000 7f71f5400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f0153e80 con 0x7f71f015b560 2024-03-23T03:51:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.436+0000 7f71ee800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71f00b39b0 0x7f71f013e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.437+0000 7f71ee800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71f00b39b0 0x7f71f013e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43520/0 (socket says 172.21.15.102:43520) 2024-03-23T03:51:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.437+0000 7f71ee800640 1 -- 172.21.15.102:0/3001973138 learned_addr learned my addr 172.21.15.102:0/3001973138 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.437+0000 7f71ede00640 1 --2- 172.21.15.102:0/3001973138 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71f015b560 0x7f71f00af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.437+0000 7f71ef200640 1 --2- 172.21.15.102:0/3001973138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f71f015c400 0x7f71f00af8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.437+0000 7f71ee800640 1 -- 172.21.15.102:0/3001973138 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71f015b560 msgr2=0x7f71f00af360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.437+0000 7f71ee800640 1 --2- 172.21.15.102:0/3001973138 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71f015b560 0x7f71f00af360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:55.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.437+0000 7f71ee800640 1 -- 172.21.15.102:0/3001973138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f71f015c400 msgr2=0x7f71f00af8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:55.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.437+0000 7f71ee800640 1 --2- 172.21.15.102:0/3001973138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f71f015c400 0x7f71f00af8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:55.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.437+0000 7f71ee800640 1 -- 172.21.15.102:0/3001973138 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71dc009400 con 0x7f71f00b39b0 2024-03-23T03:51:55.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.437+0000 7f71ef200640 1 --2- 172.21.15.102:0/3001973138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f71f015c400 0x7f71f00af8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:55.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.437+0000 7f71ee800640 1 --2- 172.21.15.102:0/3001973138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71f00b39b0 0x7f71f013e980 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f71e000c870 tx=0x7f71e000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.437+0000 7f71e6a00640 1 -- 172.21.15.102:0/3001973138 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e0003b70 con 0x7f71f00b39b0 2024-03-23T03:51:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.438+0000 7f71e6a00640 1 -- 172.21.15.102:0/3001973138 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e0003cf0 con 0x7f71f00b39b0 2024-03-23T03:51:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.438+0000 7f71e6a00640 1 -- 172.21.15.102:0/3001973138 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e000aa90 con 0x7f71f00b39b0 2024-03-23T03:51:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.438+0000 7f71f5400640 1 -- 172.21.15.102:0/3001973138 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e80094d0 con 0x7f71f00b39b0 2024-03-23T03:51:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.438+0000 7f71f5400640 1 -- 172.21.15.102:0/3001973138 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f0154440 con 0x7f71f00b39b0 2024-03-23T03:51:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.439+0000 7f71f5400640 1 -- 172.21.15.102:0/3001973138 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f006ac20 con 0x7f71f00b39b0 2024-03-23T03:51:55.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.440+0000 7f71e6a00640 1 -- 172.21.15.102:0/3001973138 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f71e000abf0 con 0x7f71f00b39b0 2024-03-23T03:51:55.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.441+0000 7f71e6a00640 1 --2- 172.21.15.102:0/3001973138 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f71d006ac00 0x7f71d006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:55.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.441+0000 7f71e6a00640 1 -- 172.21.15.102:0/3001973138 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f71e0089110 con 0x7f71f00b39b0 2024-03-23T03:51:55.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.441+0000 7f71ede00640 1 --2- 172.21.15.102:0/3001973138 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f71d006ac00 0x7f71d006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:55.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.442+0000 7f71ede00640 1 --2- 172.21.15.102:0/3001973138 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f71d006ac00 0x7f71d006d0c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f71f0066d20 tx=0x7f71dc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:55.446 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.446+0000 7f71e6a00640 1 -- 172.21.15.102:0/3001973138 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f71e0054410 con 0x7f71f00b39b0 2024-03-23T03:51:55.637 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.636+0000 7f71f5400640 1 -- 172.21.15.102:0/3001973138 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f71f0073390 con 0x7f71d006ac00 2024-03-23T03:51:55.640 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.640+0000 7f71e6a00640 1 -- 172.21.15.102:0/3001973138 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48848 (secure 0 0 0) 0x7f71f0073390 con 0x7f71d006ac00 2024-03-23T03:51:55.641 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:55.641 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:51:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.642+0000 7f71f5400640 1 -- 172.21.15.102:0/3001973138 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f71d006ac00 msgr2=0x7f71d006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.642+0000 7f71f5400640 1 --2- 172.21.15.102:0/3001973138 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f71d006ac00 0x7f71d006d0c0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f71f0066d20 tx=0x7f71dc008040 comp rx=0 tx=0).stop 2024-03-23T03:51:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.642+0000 7f71f5400640 1 -- 172.21.15.102:0/3001973138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71f00b39b0 msgr2=0x7f71f013e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.642+0000 7f71f5400640 1 --2- 172.21.15.102:0/3001973138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71f00b39b0 0x7f71f013e980 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f71e000c870 tx=0x7f71e000cd30 comp rx=0 tx=0).stop 2024-03-23T03:51:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.643+0000 7f71f5400640 1 -- 172.21.15.102:0/3001973138 shutdown_connections 2024-03-23T03:51:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.643+0000 7f71f5400640 1 --2- 172.21.15.102:0/3001973138 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f71d006ac00 0x7f71d006d0c0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.643+0000 7f71f5400640 1 --2- 172.21.15.102:0/3001973138 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f71f015c400 0x7f71f00af8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.643+0000 7f71f5400640 1 --2- 172.21.15.102:0/3001973138 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f71f015b560 0x7f71f00af360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.643+0000 7f71f5400640 1 --2- 172.21.15.102:0/3001973138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71f00b39b0 0x7f71f013e980 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.643+0000 7f71f5400640 1 -- 172.21.15.102:0/3001973138 >> 172.21.15.102:0/3001973138 conn(0x7f71f006a3d0 msgr2=0x7f71f0158f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.643+0000 7f71f5400640 1 -- 172.21.15.102:0/3001973138 shutdown_connections 2024-03-23T03:51:55.644 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.643+0000 7f71f5400640 1 -- 172.21.15.102:0/3001973138 wait complete. 2024-03-23T03:51:55.662 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-03-23T03:51:53.810488+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259280,"kb_used_data":14448,"kb_used_omap":12,"kb_used_meta":244723,"kb_avail":754715440,"statfs":{"total":773094113280,"available":772828610560,"internally_reserved":0,"allocated":14794752,"data_stored":8755518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":250596922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1352135,"num_objects":-47,"num_object_clones":0,"num_object_copies":-94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-166,"num_write_kb":-21904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002148"},"pg_stats":[{"pgid":"1.0","version":"19'117","reported_seq":233,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-23T03:51:43.763852+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:51:43.763852+0000","last_peered":"2024-03-23T03:51:43.763852+0000","last_clean":"2024-03-23T03:51:43.763852+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:51:43.763852+0000","last_undegraded":"2024-03-23T03:51:43.763852+0000","last_fullsized":"2024-03-23T03:51:43.763852+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542209,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1970176,"data_stored":1199353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.087,"15min":1.0720000000000001},"min":{"1min":0.65700000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5209999999999999,"5min":1.635,"15min":1.635},"last":1.478},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.042,"15min":1.0469999999999999},"min":{"1min":0.749,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.393,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.431}]},{"osd":1,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.081,"15min":1.0529999999999999},"min":{"1min":0.71499999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6120000000000001,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.6579999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.1719999999999999,"15min":1.2809999999999999},"min":{"1min":0.79000000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.321,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":1.21}]},{"osd":2,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.125,"15min":1.151},"min":{"1min":0.81299999999999994,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.583,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.544},{"interface":"front","average":{"1min":1.167,"5min":1.2869999999999999,"15min":1.401},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.506,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.276}]},{"osd":3,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.298,"15min":1.4179999999999999},"min":{"1min":0.71999999999999997,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.4219999999999999,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.325,"15min":1.353},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.677,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.5840000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.254,"15min":1.2629999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.8220000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.51},{"interface":"front","average":{"1min":1.333,"5min":1.347,"15min":1.3720000000000001},"min":{"1min":0.67500000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.778,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.3520000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.389,"15min":1.4179999999999999},"min":{"1min":0.81299999999999994,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.7310000000000001,"5min":2.645,"15min":2.645},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.417,"15min":1.446},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.7649999999999999,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.319}]},{"osd":6,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.504,"15min":1.552},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.9279999999999999,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.474,"15min":1.5149999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9399999999999999,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.452}]}]},{"osd":6,"up_from":14,"seq":60129542207,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33780,"kb_used_data":2664,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338060,"statfs":{"total":96636764160,"available":96602173440,"internally_reserved":0,"allocated":2727936,"data_stored":1960735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.0429999999999999,"15min":1.044},"min":{"1min":0.69699999999999995,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.05,"15min":1.04},"min":{"1min":0.66100000000000003,"5min":0.62,"15min":0.62},"max":{"1min":1.5409999999999999,"5min":1.946,"15min":1.946},"last":0.96299999999999997}]},{"osd":1,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.087,"15min":1.075},"min":{"1min":0.745,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":1.5509999999999999},{"interface":"front","average":{"1min":1.196,"5min":1.1339999999999999,"15min":1.1579999999999999},"min":{"1min":0.80500000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.5369999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.181}]},{"osd":2,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.083,"15min":1.071},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.4350000000000001,"5min":2.044,"15min":2.044},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.133,"5min":1.1759999999999999,"15min":1.212},"min":{"1min":0.85499999999999998,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.6599999999999999,"5min":1.681,"15min":1.681},"last":1.4450000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.2370000000000001,"15min":1.276},"min":{"1min":0.88100000000000001,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.786,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.2330000000000001,"15min":1.2569999999999999},"min":{"1min":0.90100000000000002,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.446,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.085}]},{"osd":4,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3779999999999999,"15min":1.385},"min":{"1min":1.1930000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.6679999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.431,"5min":1.371,"15min":1.3859999999999999},"min":{"1min":1.234,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.639,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.6140000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.403,"15min":1.4319999999999999},"min":{"1min":1.0669999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.794,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.474,"5min":1.508,"15min":1.542},"min":{"1min":1.091,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8200000000000001,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.48}]},{"osd":7,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.4059999999999999,"15min":1.4119999999999999},"min":{"1min":0.95999999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.0760000000000001,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.482,"15min":1.5309999999999999},"min":{"1min":1.125,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.8660000000000001,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.667}]}]},{"osd":5,"up_from":14,"seq":60129542207,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28340,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343500,"statfs":{"total":96636764160,"available":96607744000,"internally_reserved":0,"allocated":1351680,"data_stored":608821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1439999999999999,"15min":1.2030000000000001},"min":{"1min":0.72999999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4830000000000001,"5min":1.806,"15min":1.806},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.113,"5min":1.1399999999999999,"15min":1.1759999999999999},"min":{"1min":0.76500000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.855,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.0649999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0940000000000001,"15min":1.1000000000000001},"min":{"1min":0.80100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.3520000000000001,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.232,"5min":1.165,"15min":1.1930000000000001},"min":{"1min":0.84799999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.663,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.101}]},{"osd":2,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.274,"15min":1.3460000000000001},"min":{"1min":0.70499999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8939999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.268},{"interface":"front","average":{"1min":1.226,"5min":1.2989999999999999,"15min":1.3660000000000001},"min":{"1min":0.875,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.579,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.028}]},{"osd":3,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.2,"15min":1.244},"min":{"1min":0.85399999999999998,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.7050000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.375,"15min":1.446},"min":{"1min":0.997,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.964,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.4570000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3049999999999999,"15min":1.399},"min":{"1min":0.65700000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6100000000000001,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3859999999999999,"15min":1.4359999999999999},"min":{"1min":0.83699999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8069999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":0.98299999999999998}]},{"osd":6,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.4159999999999999,"15min":1.5},"min":{"1min":0.83399999999999996,"5min":0.63,"15min":0.63},"max":{"1min":1.5840000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.5409999999999999,"15min":1.639},"min":{"1min":1.1950000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.6359999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.3120000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4910000000000001,"15min":1.591},"min":{"1min":1.095,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.9259999999999999,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5840000000000001,"15min":1.6839999999999999},"min":{"1min":1.272,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.056,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.375}]}]},{"osd":4,"up_from":14,"seq":60129542208,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1376256,"data_stored":608919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.004,"15min":0.98099999999999998},"min":{"1min":0.624,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.47,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.995,"15min":1.0049999999999999},"min":{"1min":0.56100000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3400000000000001,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.365}]},{"osd":1,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0680000000000001,"15min":1.05},"min":{"1min":0.65100000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.5,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.234},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0880000000000001,"15min":1.0640000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.5209999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.304}]},{"osd":2,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.2470000000000001,"15min":1.3089999999999999},"min":{"1min":0.70399999999999996,"5min":0.625,"15min":0.625},"max":{"1min":1.452,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.2789999999999999,"15min":1.2969999999999999},"min":{"1min":0.59899999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6859999999999999,"5min":2.181,"15min":2.181},"last":1.4119999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.1160000000000001,"15min":1.1240000000000001},"min":{"1min":0.61899999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.4419999999999999,"5min":1.821,"15min":1.821},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.141,"5min":1.2310000000000001,"15min":1.268},"min":{"1min":0.73799999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.599,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.0820000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.2370000000000001,"15min":1.26},"min":{"1min":0.63800000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.49,"5min":1.929,"15min":1.929},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.2709999999999999,"15min":1.274},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.45,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.5089999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.371,"15min":1.409},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6719999999999999,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.345},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.4139999999999999,"15min":1.3999999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.8100000000000001,"5min":2.016,"15min":2.016},"last":1.617}]},{"osd":7,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.361,"15min":1.363},"min":{"1min":0.68200000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.5569999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.456,"15min":1.4870000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.716,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.538}]}]},{"osd":3,"up_from":14,"seq":60129542208,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32432,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339408,"statfs":{"total":96636764160,"available":96603553792,"internally_reserved":0,"allocated":1347584,"data_stored":609398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.81999999999999995,"15min":0.85499999999999998},"min":{"1min":0.35499999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.4419999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.92200000000000004,"15min":0.92400000000000004},"min":{"1min":0.502,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.538,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.41399999999999998}]},{"osd":1,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.91500000000000004,"15min":0.96799999999999997},"min":{"1min":0.58099999999999996,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.9990000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.034,"5min":1.0229999999999999,"15min":1.069},"min":{"1min":0.52000000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.651,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.54300000000000004}]},{"osd":2,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.002,"15min":1.0509999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.5710000000000002,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1240000000000001,"15min":1.1499999999999999},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.7649999999999999,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":0.65300000000000002}]},{"osd":4,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.2410000000000001,"15min":1.248},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":4.0380000000000003,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.337,"5min":1.351,"15min":1.429},"min":{"1min":0.77000000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.3450000000000002,"5min":4.819,"15min":4.819},"last":0.84999999999999998}]},{"osd":5,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.6040000000000001,"15min":1.4470000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.2679999999999998,"5min":30.228000000000002,"15min":30.228000000000002},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.337,"15min":1.357},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":4.3630000000000004,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":0.64300000000000002}]},{"osd":6,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4119999999999999,"15min":1.508},"min":{"1min":0.83999999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.6589999999999998,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.137},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4199999999999999,"15min":1.4039999999999999},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.4039999999999999,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":0.872}]},{"osd":7,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.389,"15min":1.4199999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.0880000000000001,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.542,"15min":1.5900000000000001},"min":{"1min":0.80500000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.4780000000000002,"5min":10.663,"15min":10.663},"last":0.76100000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542208,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1355776,"data_stored":608853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83799999999999997,"15min":0.877},"min":{"1min":0.52100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":1.4259999999999999,"5min":1.456,"15min":1.456},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.86399999999999999,"15min":0.93600000000000005},"min":{"1min":0.39500000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.4690000000000001,"5min":2.226,"15min":2.226},"last":0.77600000000000002}]},{"osd":1,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.93899999999999995,"15min":0.98499999999999999},"min":{"1min":0.46800000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.377,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.85899999999999999,"15min":0.91000000000000003},"min":{"1min":0.57599999999999996,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.2210000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.52400000000000002}]},{"osd":3,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.96099999999999997,"15min":1.018},"min":{"1min":0.47499999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.403,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.032,"5min":1.042,"15min":1.0800000000000001},"min":{"1min":0.61099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.95099999999999996}]},{"osd":4,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.194,"15min":1.2310000000000001},"min":{"1min":0.82399999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.7270000000000001,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.157,"15min":1.2070000000000001},"min":{"1min":0.83799999999999997,"5min":0.627,"15min":0.627},"max":{"1min":1.4950000000000001,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.276}]},{"osd":5,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.3100000000000001,"15min":1.367},"min":{"1min":0.76900000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.6259999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.2749999999999999,"15min":1.339},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.819,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.99199999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.232,"15min":1.3280000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6479999999999999,"5min":2.456,"15min":2.456},"last":1.242},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3999999999999999,"15min":1.476},"min":{"1min":0.85299999999999998,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.097,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.0329999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3069999999999999,"15min":1.369},"min":{"1min":0.79800000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.302,"15min":1.395},"min":{"1min":0.78700000000000003,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5860000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.0129999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542207,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33468,"kb_used_data":2352,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338372,"statfs":{"total":96636764160,"available":96602492928,"internally_reserved":0,"allocated":2408448,"data_stored":1657432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79100000000000004,"15min":0.86099999999999999},"min":{"1min":0.41599999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.1240000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81799999999999995,"15min":0.82999999999999996},"min":{"1min":0.435,"5min":0.374,"15min":0.374},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.0469999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.92200000000000004,"15min":0.96399999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.4430000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.087},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.92400000000000004,"15min":0.97799999999999998},"min":{"1min":0.496,"5min":0.443,"15min":0.443},"max":{"1min":1.2210000000000001,"5min":1.625,"15min":1.625},"last":0.89700000000000002}]},{"osd":3,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.103,"15min":1.113},"min":{"1min":0.68400000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.6559999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.875,"5min":0.96199999999999997,"15min":1.008},"min":{"1min":0.59499999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.6180000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.64900000000000002}]},{"osd":4,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.179,"15min":1.2749999999999999},"min":{"1min":0.67700000000000005,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.5069999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.246,"15min":1.3049999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6120000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.133}]},{"osd":5,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.248,"15min":1.3200000000000001},"min":{"1min":0.79500000000000004,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.95,"5min":3.593,"15min":3.593},"last":1.27},{"interface":"front","average":{"1min":1.264,"5min":1.327,"15min":1.4259999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.671,"5min":2.359,"15min":2.359},"last":1.4850000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.3089999999999999,"15min":1.3919999999999999},"min":{"1min":0.84199999999999997,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6200000000000001,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.288,"15min":1.3540000000000001},"min":{"1min":0.89100000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.0209999999999999,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.4199999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.359,"15min":1.4330000000000001},"min":{"1min":0.99299999999999999,"5min":0.627,"15min":0.627},"max":{"1min":2.0840000000000001,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.363,"15min":1.486},"min":{"1min":0.76600000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.673,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.613}]}]},{"osd":0,"up_from":14,"seq":60129542207,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33320,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338520,"statfs":{"total":96636764160,"available":96602644480,"internally_reserved":0,"allocated":2256896,"data_stored":1502007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.86399999999999999,"15min":0.93500000000000005},"min":{"1min":0.48399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.478,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.94799999999999995,"15min":0.98199999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.456,"5min":1.506,"15min":1.506},"last":1.643}]},{"osd":2,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.92700000000000005,"15min":0.99099999999999999},"min":{"1min":0.52900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.4399999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.321},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.0349999999999999,"15min":1.095},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.601,"5min":1.901,"15min":1.901},"last":1.1359999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.93300000000000005,"15min":0.996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.474,"5min":1.972,"15min":1.972},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0669999999999999,"15min":1.147},"min":{"1min":0.60299999999999998,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.597,"5min":2.298,"15min":2.298},"last":1.409}]},{"osd":4,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.288,"15min":1.3360000000000001},"min":{"1min":0.72799999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.7090000000000001,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":1.708},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.2230000000000001,"15min":1.27},"min":{"1min":0.80600000000000005,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.8380000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.5229999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3100000000000001,"15min":1.3700000000000001},"min":{"1min":0.89500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.948,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.377,"15min":1.4379999999999999},"min":{"1min":0.83499999999999996,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.8700000000000001,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.4870000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.4099999999999999,"15min":1.5209999999999999},"min":{"1min":0.80000000000000004,"5min":0.747,"15min":0.747},"max":{"1min":1.978,"5min":3.032,"15min":3.032},"last":1.8009999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.411,"15min":1.4870000000000001},"min":{"1min":0.84699999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.1349999999999998,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.748}]},{"osd":7,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4390000000000001,"15min":1.506},"min":{"1min":0.90500000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1850000000000001,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.8540000000000001},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3620000000000001,"15min":1.419},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.873,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.2190000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:51:55.664 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:51:55.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.871+0000 7f0586a00640 1 Processor -- start 2024-03-23T03:51:55.872 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.871+0000 7f0586a00640 1 -- start start 2024-03-23T03:51:55.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.872+0000 7f0586a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f058006bec0 0x7f058006afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:55.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.872+0000 7f0586a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05800b1200 0x7f0580098ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:55.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.873+0000 7f0586a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05800cf310 con 0x7f05800b0ae0 2024-03-23T03:51:55.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.873+0000 7f0586a00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0580098a40 con 0x7f058006bec0 2024-03-23T03:51:55.873 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.873+0000 7f0586a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05800b19f0 con 0x7f05800b1200 2024-03-23T03:51:55.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.873+0000 7f0584a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05800b1200 0x7f0580098ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:55.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.873+0000 7f0584a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05800b1200 0x7f0580098ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:46234/0 (socket says 172.21.15.102:46234) 2024-03-23T03:51:55.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.873+0000 7f0584a00640 1 -- 172.21.15.102:0/3133440851 learned_addr learned my addr 172.21.15.102:0/3133440851 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:55.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.873+0000 7f0577400640 1 --2- 172.21.15.102:0/3133440851 >> v2:172.21.15.173:3300/0 conn(0x7f058006bec0 0x7f058006afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:55.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.874+0000 7f0584a00640 1 -- 172.21.15.102:0/3133440851 >> v2:172.21.15.173:3300/0 conn(0x7f058006bec0 msgr2=0x7f058006afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:55.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.874+0000 7f0584a00640 1 --2- 172.21.15.102:0/3133440851 >> v2:172.21.15.173:3300/0 conn(0x7f058006bec0 0x7f058006afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:55.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.874+0000 7f0584a00640 1 -- 172.21.15.102:0/3133440851 >> v1:172.21.15.173:6789/0 conn(0x7f05800b0ae0 legacy=0x7f058006b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:55.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.874+0000 7f0584a00640 1 -- 172.21.15.102:0/3133440851 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05800cf560 con 0x7f05800b1200 2024-03-23T03:51:55.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.875+0000 7f0584a00640 1 --2- 172.21.15.102:0/3133440851 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05800b1200 0x7f0580098ed0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0578009e50 tx=0x7f057802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf3f046e877ee673 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.876+0000 7f057f400640 1 -- 172.21.15.102:0/3133440851 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057802ca70 con 0x7f05800b1200 2024-03-23T03:51:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.876+0000 7f057f400640 1 -- 172.21.15.102:0/3133440851 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057802cbd0 con 0x7f05800b1200 2024-03-23T03:51:55.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.876+0000 7f057f400640 1 -- 172.21.15.102:0/3133440851 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05780343f0 con 0x7f05800b1200 2024-03-23T03:51:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.876+0000 7f0586a00640 1 -- 172.21.15.102:0/3133440851 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05800b1200 msgr2=0x7f0580098ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.876+0000 7f0586a00640 1 --2- 172.21.15.102:0/3133440851 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05800b1200 0x7f0580098ed0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f0578009e50 tx=0x7f057802bed0 comp rx=0 tx=0).stop 2024-03-23T03:51:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.876+0000 7f0586a00640 1 -- 172.21.15.102:0/3133440851 shutdown_connections 2024-03-23T03:51:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.876+0000 7f0586a00640 1 --2- 172.21.15.102:0/3133440851 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f05800b1200 0x7f0580098ed0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.876+0000 7f0586a00640 1 --2- 172.21.15.102:0/3133440851 >> v2:172.21.15.173:3300/0 conn(0x7f058006bec0 0x7f058006afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.876+0000 7f0586a00640 1 -- 172.21.15.102:0/3133440851 >> 172.21.15.102:0/3133440851 conn(0x7f05800b2160 msgr2=0x7f05800b2570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.877+0000 7f0586a00640 1 -- 172.21.15.102:0/3133440851 shutdown_connections 2024-03-23T03:51:55.877 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.877+0000 7f0586a00640 1 -- 172.21.15.102:0/3133440851 wait complete. 2024-03-23T03:51:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.878+0000 7f0586a00640 1 Processor -- start 2024-03-23T03:51:55.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.878+0000 7f0586a00640 1 -- start start 2024-03-23T03:51:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.879+0000 7f0586a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f058006bec0 0x7f05800b6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.879+0000 7f0586a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05800b0ae0 0x7f0580156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.879+0000 7f0586a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05800b1200 0x7f05800c5370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.879+0000 7f0586a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0580157440 con 0x7f05800b0ae0 2024-03-23T03:51:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.879+0000 7f0586a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05800c5bc0 con 0x7f05800b1200 2024-03-23T03:51:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.879+0000 7f0586a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05800c5ea0 con 0x7f058006bec0 2024-03-23T03:51:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.880+0000 7f0577400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05800b0ae0 0x7f0580156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.880+0000 7f0577400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05800b0ae0 0x7f0580156f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43528/0 (socket says 172.21.15.102:43528) 2024-03-23T03:51:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.880+0000 7f0577400640 1 -- 172.21.15.102:0/3844420932 learned_addr learned my addr 172.21.15.102:0/3844420932 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.880+0000 7f057fe00640 1 --2- 172.21.15.102:0/3844420932 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f058006bec0 0x7f05800b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.880+0000 7f0584a00640 1 --2- 172.21.15.102:0/3844420932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05800b1200 0x7f05800c5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.880+0000 7f0577400640 1 -- 172.21.15.102:0/3844420932 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f058006bec0 msgr2=0x7f05800b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:55.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.880+0000 7f0577400640 1 --2- 172.21.15.102:0/3844420932 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f058006bec0 0x7f05800b6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.880+0000 7f0577400640 1 -- 172.21.15.102:0/3844420932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05800b1200 msgr2=0x7f05800c5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.880+0000 7f0577400640 1 --2- 172.21.15.102:0/3844420932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05800b1200 0x7f05800c5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.880+0000 7f0577400640 1 -- 172.21.15.102:0/3844420932 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0578009b30 con 0x7f05800b0ae0 2024-03-23T03:51:55.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.880+0000 7f0577400640 1 --2- 172.21.15.102:0/3844420932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05800b0ae0 0x7f0580156f00 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f056800e870 tx=0x7f056800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:55.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.881+0000 7f057cc00640 1 -- 172.21.15.102:0/3844420932 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05680094f0 con 0x7f05800b0ae0 2024-03-23T03:51:55.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.881+0000 7f057cc00640 1 -- 172.21.15.102:0/3844420932 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056800c940 con 0x7f05800b0ae0 2024-03-23T03:51:55.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.881+0000 7f0586a00640 1 -- 172.21.15.102:0/3844420932 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05800c6180 con 0x7f05800b0ae0 2024-03-23T03:51:55.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.881+0000 7f057cc00640 1 -- 172.21.15.102:0/3844420932 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056800ccf0 con 0x7f05800b0ae0 2024-03-23T03:51:55.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.881+0000 7f0586a00640 1 -- 172.21.15.102:0/3844420932 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05800c6740 con 0x7f05800b0ae0 2024-03-23T03:51:55.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.882+0000 7f0586a00640 1 -- 172.21.15.102:0/3844420932 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0580073390 con 0x7f05800b0ae0 2024-03-23T03:51:55.883 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.883+0000 7f057cc00640 1 -- 172.21.15.102:0/3844420932 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0568013470 con 0x7f05800b0ae0 2024-03-23T03:51:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.884+0000 7f057cc00640 1 --2- 172.21.15.102:0/3844420932 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f055c06ac00 0x7f055c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.884+0000 7f057cc00640 1 -- 172.21.15.102:0/3844420932 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f05680891e0 con 0x7f05800b0ae0 2024-03-23T03:51:55.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.884+0000 7f057fe00640 1 --2- 172.21.15.102:0/3844420932 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f055c06ac00 0x7f055c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:55.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.885+0000 7f057fe00640 1 --2- 172.21.15.102:0/3844420932 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f055c06ac00 0x7f055c06d0c0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f0570000f60 tx=0x7f0570009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:55.889 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:55.888+0000 7f057cc00640 1 -- 172.21.15.102:0/3844420932 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f0568054610 con 0x7f05800b0ae0 2024-03-23T03:51:56.079 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.078+0000 7f0586a00640 1 -- 172.21.15.102:0/3844420932 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f05800cacc0 con 0x7f055c06ac00 2024-03-23T03:51:56.081 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.080+0000 7f057cc00640 1 -- 172.21.15.102:0/3844420932 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48848 (secure 0 0 0) 0x7f05700050d0 con 0x7f055c06ac00 2024-03-23T03:51:56.082 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:56.082 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:51:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.082+0000 7f0586a00640 1 -- 172.21.15.102:0/3844420932 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f055c06ac00 msgr2=0x7f055c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.082+0000 7f0586a00640 1 --2- 172.21.15.102:0/3844420932 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f055c06ac00 0x7f055c06d0c0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f0570000f60 tx=0x7f0570009040 comp rx=0 tx=0).stop 2024-03-23T03:51:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.082+0000 7f0586a00640 1 -- 172.21.15.102:0/3844420932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05800b0ae0 msgr2=0x7f0580156f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.082+0000 7f0586a00640 1 --2- 172.21.15.102:0/3844420932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05800b0ae0 0x7f0580156f00 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f056800e870 tx=0x7f056800ed30 comp rx=0 tx=0).stop 2024-03-23T03:51:56.084 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.083+0000 7f0586a00640 1 -- 172.21.15.102:0/3844420932 shutdown_connections 2024-03-23T03:51:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.083+0000 7f0586a00640 1 --2- 172.21.15.102:0/3844420932 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f055c06ac00 0x7f055c06d0c0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.083+0000 7f0586a00640 1 --2- 172.21.15.102:0/3844420932 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f05800b1200 0x7f05800c5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.083+0000 7f0586a00640 1 --2- 172.21.15.102:0/3844420932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05800b0ae0 0x7f0580156f00 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.083+0000 7f0586a00640 1 --2- 172.21.15.102:0/3844420932 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f058006bec0 0x7f05800b6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.083+0000 7f0586a00640 1 -- 172.21.15.102:0/3844420932 >> 172.21.15.102:0/3844420932 conn(0x7f05800b2160 msgr2=0x7f058006f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.083+0000 7f0586a00640 1 -- 172.21.15.102:0/3844420932 shutdown_connections 2024-03-23T03:51:56.085 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.083+0000 7f0586a00640 1 -- 172.21.15.102:0/3844420932 wait complete. 2024-03-23T03:51:56.102 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-03-23T03:51:55.810841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259280,"kb_used_data":14448,"kb_used_omap":12,"kb_used_meta":244723,"kb_avail":754715440,"statfs":{"total":773094113280,"available":772828610560,"internally_reserved":0,"allocated":14794752,"data_stored":8755518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":250596922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1352135,"num_objects":-47,"num_object_clones":0,"num_object_copies":-94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-166,"num_write_kb":-21904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.137981"},"pg_stats":[{"pgid":"1.0","version":"19'117","reported_seq":233,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-23T03:51:43.763852+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:51:43.763852+0000","last_peered":"2024-03-23T03:51:43.763852+0000","last_clean":"2024-03-23T03:51:43.763852+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:51:43.763852+0000","last_undegraded":"2024-03-23T03:51:43.763852+0000","last_fullsized":"2024-03-23T03:51:43.763852+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542209,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1970176,"data_stored":1199353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.087,"15min":1.0720000000000001},"min":{"1min":0.65700000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5209999999999999,"5min":1.635,"15min":1.635},"last":1.478},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.042,"15min":1.0469999999999999},"min":{"1min":0.749,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.393,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.431}]},{"osd":1,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.081,"15min":1.0529999999999999},"min":{"1min":0.71499999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6120000000000001,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.6579999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.1719999999999999,"15min":1.2809999999999999},"min":{"1min":0.79000000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.321,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":1.21}]},{"osd":2,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.125,"15min":1.151},"min":{"1min":0.81299999999999994,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.583,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.544},{"interface":"front","average":{"1min":1.167,"5min":1.2869999999999999,"15min":1.401},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.506,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.276}]},{"osd":3,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.298,"15min":1.4179999999999999},"min":{"1min":0.71999999999999997,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.4219999999999999,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.325,"15min":1.353},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.677,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.5840000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.254,"15min":1.2629999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.8220000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.51},{"interface":"front","average":{"1min":1.333,"5min":1.347,"15min":1.3720000000000001},"min":{"1min":0.67500000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.778,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.3520000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.389,"15min":1.4179999999999999},"min":{"1min":0.81299999999999994,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.7310000000000001,"5min":2.645,"15min":2.645},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.417,"15min":1.446},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.7649999999999999,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.319}]},{"osd":6,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.504,"15min":1.552},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.9279999999999999,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.474,"15min":1.5149999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9399999999999999,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.452}]}]},{"osd":6,"up_from":14,"seq":60129542207,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33780,"kb_used_data":2664,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338060,"statfs":{"total":96636764160,"available":96602173440,"internally_reserved":0,"allocated":2727936,"data_stored":1960735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.0429999999999999,"15min":1.044},"min":{"1min":0.69699999999999995,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.05,"15min":1.04},"min":{"1min":0.66100000000000003,"5min":0.62,"15min":0.62},"max":{"1min":1.5409999999999999,"5min":1.946,"15min":1.946},"last":0.96299999999999997}]},{"osd":1,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.087,"15min":1.075},"min":{"1min":0.745,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":1.5509999999999999},{"interface":"front","average":{"1min":1.196,"5min":1.1339999999999999,"15min":1.1579999999999999},"min":{"1min":0.80500000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.5369999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.181}]},{"osd":2,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.083,"15min":1.071},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.4350000000000001,"5min":2.044,"15min":2.044},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.133,"5min":1.1759999999999999,"15min":1.212},"min":{"1min":0.85499999999999998,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.6599999999999999,"5min":1.681,"15min":1.681},"last":1.4450000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.2370000000000001,"15min":1.276},"min":{"1min":0.88100000000000001,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.786,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.2330000000000001,"15min":1.2569999999999999},"min":{"1min":0.90100000000000002,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.446,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.085}]},{"osd":4,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3779999999999999,"15min":1.385},"min":{"1min":1.1930000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.6679999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.431,"5min":1.371,"15min":1.3859999999999999},"min":{"1min":1.234,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.639,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.6140000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.403,"15min":1.4319999999999999},"min":{"1min":1.0669999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.794,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.474,"5min":1.508,"15min":1.542},"min":{"1min":1.091,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8200000000000001,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.48}]},{"osd":7,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.4059999999999999,"15min":1.4119999999999999},"min":{"1min":0.95999999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.0760000000000001,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.482,"15min":1.5309999999999999},"min":{"1min":1.125,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.8660000000000001,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.667}]}]},{"osd":5,"up_from":14,"seq":60129542207,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28340,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343500,"statfs":{"total":96636764160,"available":96607744000,"internally_reserved":0,"allocated":1351680,"data_stored":608821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1439999999999999,"15min":1.2030000000000001},"min":{"1min":0.72999999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4830000000000001,"5min":1.806,"15min":1.806},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.113,"5min":1.1399999999999999,"15min":1.1759999999999999},"min":{"1min":0.76500000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.855,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.0649999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0940000000000001,"15min":1.1000000000000001},"min":{"1min":0.80100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.3520000000000001,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.232,"5min":1.165,"15min":1.1930000000000001},"min":{"1min":0.84799999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.663,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.101}]},{"osd":2,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.274,"15min":1.3460000000000001},"min":{"1min":0.70499999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8939999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.268},{"interface":"front","average":{"1min":1.226,"5min":1.2989999999999999,"15min":1.3660000000000001},"min":{"1min":0.875,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.579,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.028}]},{"osd":3,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.2,"15min":1.244},"min":{"1min":0.85399999999999998,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.7050000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.375,"15min":1.446},"min":{"1min":0.997,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.964,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.4570000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3049999999999999,"15min":1.399},"min":{"1min":0.65700000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6100000000000001,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3859999999999999,"15min":1.4359999999999999},"min":{"1min":0.83699999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8069999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":0.98299999999999998}]},{"osd":6,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.4159999999999999,"15min":1.5},"min":{"1min":0.83399999999999996,"5min":0.63,"15min":0.63},"max":{"1min":1.5840000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.5409999999999999,"15min":1.639},"min":{"1min":1.1950000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.6359999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.3120000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4910000000000001,"15min":1.591},"min":{"1min":1.095,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.9259999999999999,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5840000000000001,"15min":1.6839999999999999},"min":{"1min":1.272,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.056,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.375}]}]},{"osd":4,"up_from":14,"seq":60129542208,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1376256,"data_stored":608919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.004,"15min":0.98099999999999998},"min":{"1min":0.624,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.47,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.995,"15min":1.0049999999999999},"min":{"1min":0.56100000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3400000000000001,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.365}]},{"osd":1,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0680000000000001,"15min":1.05},"min":{"1min":0.65100000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.5,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.234},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0880000000000001,"15min":1.0640000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.5209999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.304}]},{"osd":2,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.2470000000000001,"15min":1.3089999999999999},"min":{"1min":0.70399999999999996,"5min":0.625,"15min":0.625},"max":{"1min":1.452,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.2789999999999999,"15min":1.2969999999999999},"min":{"1min":0.59899999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6859999999999999,"5min":2.181,"15min":2.181},"last":1.4119999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.1160000000000001,"15min":1.1240000000000001},"min":{"1min":0.61899999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.4419999999999999,"5min":1.821,"15min":1.821},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.141,"5min":1.2310000000000001,"15min":1.268},"min":{"1min":0.73799999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.599,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.0820000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.2370000000000001,"15min":1.26},"min":{"1min":0.63800000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.49,"5min":1.929,"15min":1.929},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.2709999999999999,"15min":1.274},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.45,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.5089999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.371,"15min":1.409},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6719999999999999,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.345},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.4139999999999999,"15min":1.3999999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.8100000000000001,"5min":2.016,"15min":2.016},"last":1.617}]},{"osd":7,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.361,"15min":1.363},"min":{"1min":0.68200000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.5569999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.456,"15min":1.4870000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.716,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.538}]}]},{"osd":3,"up_from":14,"seq":60129542208,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32432,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339408,"statfs":{"total":96636764160,"available":96603553792,"internally_reserved":0,"allocated":1347584,"data_stored":609398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.81999999999999995,"15min":0.85499999999999998},"min":{"1min":0.35499999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.4419999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.92200000000000004,"15min":0.92400000000000004},"min":{"1min":0.502,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.538,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.41399999999999998}]},{"osd":1,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.91500000000000004,"15min":0.96799999999999997},"min":{"1min":0.58099999999999996,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.9990000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.034,"5min":1.0229999999999999,"15min":1.069},"min":{"1min":0.52000000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.651,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.54300000000000004}]},{"osd":2,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.002,"15min":1.0509999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.5710000000000002,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1240000000000001,"15min":1.1499999999999999},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.7649999999999999,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":0.65300000000000002}]},{"osd":4,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.2410000000000001,"15min":1.248},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":4.0380000000000003,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.337,"5min":1.351,"15min":1.429},"min":{"1min":0.77000000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.3450000000000002,"5min":4.819,"15min":4.819},"last":0.84999999999999998}]},{"osd":5,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.6040000000000001,"15min":1.4470000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.2679999999999998,"5min":30.228000000000002,"15min":30.228000000000002},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.337,"15min":1.357},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":4.3630000000000004,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":0.64300000000000002}]},{"osd":6,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4119999999999999,"15min":1.508},"min":{"1min":0.83999999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.6589999999999998,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.137},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4199999999999999,"15min":1.4039999999999999},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.4039999999999999,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":0.872}]},{"osd":7,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.389,"15min":1.4199999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.0880000000000001,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.542,"15min":1.5900000000000001},"min":{"1min":0.80500000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.4780000000000002,"5min":10.663,"15min":10.663},"last":0.76100000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542208,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1355776,"data_stored":608853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83799999999999997,"15min":0.877},"min":{"1min":0.52100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":1.4259999999999999,"5min":1.456,"15min":1.456},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.86399999999999999,"15min":0.93600000000000005},"min":{"1min":0.39500000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.4690000000000001,"5min":2.226,"15min":2.226},"last":0.77600000000000002}]},{"osd":1,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.93899999999999995,"15min":0.98499999999999999},"min":{"1min":0.46800000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.377,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.85899999999999999,"15min":0.91000000000000003},"min":{"1min":0.57599999999999996,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.2210000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.52400000000000002}]},{"osd":3,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.96099999999999997,"15min":1.018},"min":{"1min":0.47499999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.403,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.032,"5min":1.042,"15min":1.0800000000000001},"min":{"1min":0.61099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.95099999999999996}]},{"osd":4,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.194,"15min":1.2310000000000001},"min":{"1min":0.82399999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.7270000000000001,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.157,"15min":1.2070000000000001},"min":{"1min":0.83799999999999997,"5min":0.627,"15min":0.627},"max":{"1min":1.4950000000000001,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.276}]},{"osd":5,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.3100000000000001,"15min":1.367},"min":{"1min":0.76900000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.6259999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.2749999999999999,"15min":1.339},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.819,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.99199999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.232,"15min":1.3280000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6479999999999999,"5min":2.456,"15min":2.456},"last":1.242},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3999999999999999,"15min":1.476},"min":{"1min":0.85299999999999998,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.097,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.0329999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3069999999999999,"15min":1.369},"min":{"1min":0.79800000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.302,"15min":1.395},"min":{"1min":0.78700000000000003,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5860000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.0129999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542207,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33468,"kb_used_data":2352,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338372,"statfs":{"total":96636764160,"available":96602492928,"internally_reserved":0,"allocated":2408448,"data_stored":1657432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79100000000000004,"15min":0.86099999999999999},"min":{"1min":0.41599999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.1240000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81799999999999995,"15min":0.82999999999999996},"min":{"1min":0.435,"5min":0.374,"15min":0.374},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.0469999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.92200000000000004,"15min":0.96399999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.4430000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.087},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.92400000000000004,"15min":0.97799999999999998},"min":{"1min":0.496,"5min":0.443,"15min":0.443},"max":{"1min":1.2210000000000001,"5min":1.625,"15min":1.625},"last":0.89700000000000002}]},{"osd":3,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.103,"15min":1.113},"min":{"1min":0.68400000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.6559999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.875,"5min":0.96199999999999997,"15min":1.008},"min":{"1min":0.59499999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.6180000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.64900000000000002}]},{"osd":4,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.179,"15min":1.2749999999999999},"min":{"1min":0.67700000000000005,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.5069999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.246,"15min":1.3049999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6120000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.133}]},{"osd":5,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.248,"15min":1.3200000000000001},"min":{"1min":0.79500000000000004,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.95,"5min":3.593,"15min":3.593},"last":1.27},{"interface":"front","average":{"1min":1.264,"5min":1.327,"15min":1.4259999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.671,"5min":2.359,"15min":2.359},"last":1.4850000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.3089999999999999,"15min":1.3919999999999999},"min":{"1min":0.84199999999999997,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6200000000000001,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.288,"15min":1.3540000000000001},"min":{"1min":0.89100000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.0209999999999999,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.4199999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.359,"15min":1.4330000000000001},"min":{"1min":0.99299999999999999,"5min":0.627,"15min":0.627},"max":{"1min":2.0840000000000001,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.363,"15min":1.486},"min":{"1min":0.76600000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.673,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.613}]}]},{"osd":0,"up_from":14,"seq":60129542207,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33320,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338520,"statfs":{"total":96636764160,"available":96602644480,"internally_reserved":0,"allocated":2256896,"data_stored":1502007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.86399999999999999,"15min":0.93500000000000005},"min":{"1min":0.48399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.478,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.94799999999999995,"15min":0.98199999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.456,"5min":1.506,"15min":1.506},"last":1.643}]},{"osd":2,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.92700000000000005,"15min":0.99099999999999999},"min":{"1min":0.52900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.4399999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.321},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.0349999999999999,"15min":1.095},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.601,"5min":1.901,"15min":1.901},"last":1.1359999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.93300000000000005,"15min":0.996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.474,"5min":1.972,"15min":1.972},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0669999999999999,"15min":1.147},"min":{"1min":0.60299999999999998,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.597,"5min":2.298,"15min":2.298},"last":1.409}]},{"osd":4,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.288,"15min":1.3360000000000001},"min":{"1min":0.72799999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.7090000000000001,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":1.708},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.2230000000000001,"15min":1.27},"min":{"1min":0.80600000000000005,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.8380000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.5229999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3100000000000001,"15min":1.3700000000000001},"min":{"1min":0.89500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.948,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.377,"15min":1.4379999999999999},"min":{"1min":0.83499999999999996,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.8700000000000001,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.4870000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.4099999999999999,"15min":1.5209999999999999},"min":{"1min":0.80000000000000004,"5min":0.747,"15min":0.747},"max":{"1min":1.978,"5min":3.032,"15min":3.032},"last":1.8009999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.411,"15min":1.4870000000000001},"min":{"1min":0.84699999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.1349999999999998,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.748}]},{"osd":7,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4390000000000001,"15min":1.506},"min":{"1min":0.90500000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1850000000000001,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.8540000000000001},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3620000000000001,"15min":1.419},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.873,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.2190000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:51:56.103 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-23T03:51:56.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:51:56.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.314+0000 7f7937e00640 1 Processor -- start 2024-03-23T03:51:56.315 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.314+0000 7f7937e00640 1 -- start start 2024-03-23T03:51:56.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.316+0000 7f7937e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7938143460 0x7f793815aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:56.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.316+0000 7f7937e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f7938143930 0x7f79380cd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:56.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.316+0000 7f7937e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79380cc6b0 con 0x7f79380cdc50 2024-03-23T03:51:56.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.316+0000 7f7937e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79380cc950 con 0x7f7938143930 2024-03-23T03:51:56.316 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.316+0000 7f7937e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79380ccc30 con 0x7f7938143460 2024-03-23T03:51:56.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.316+0000 7f7936000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f7938143930 0x7f79380cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:56.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.316+0000 7f7936a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7938143460 0x7f793815aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:56.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.317+0000 7f7936000640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f7938143930 0x7f79380cd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:53284/0 (socket says 172.21.15.102:53284) 2024-03-23T03:51:56.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.317+0000 7f7936a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7938143460 0x7f793815aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3301/0 says I am v2:172.21.15.102:46260/0 (socket says 172.21.15.102:46260) 2024-03-23T03:51:56.317 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.317+0000 7f7936000640 1 -- 172.21.15.102:0/2126072297 learned_addr learned my addr 172.21.15.102:0/2126072297 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:56.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.317+0000 7f7936a00640 1 -- 172.21.15.102:0/2126072297 >> v2:172.21.15.173:3300/0 conn(0x7f7938143930 msgr2=0x7f79380cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:56.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.317+0000 7f7936a00640 1 --2- 172.21.15.102:0/2126072297 >> v2:172.21.15.173:3300/0 conn(0x7f7938143930 0x7f79380cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.317+0000 7f7936a00640 1 -- 172.21.15.102:0/2126072297 >> v1:172.21.15.173:6789/0 conn(0x7f79380cdc50 legacy=0x7f793815e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:56.318 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.317+0000 7f7935600640 1 -- 172.21.15.102:0/2126072297 <== mon.1 v1:172.21.15.173:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849398911 0 0) 0x7f79380cc6b0 con 0x7f79380cdc50 2024-03-23T03:51:56.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.317+0000 7f7936a00640 1 -- 172.21.15.102:0/2126072297 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79380ccf10 con 0x7f7938143460 2024-03-23T03:51:56.319 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.317+0000 7f7936000640 1 --2- 172.21.15.102:0/2126072297 >> v2:172.21.15.173:3300/0 conn(0x7f7938143930 0x7f79380cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:51:56.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.319+0000 7f7936a00640 1 --2- 172.21.15.102:0/2126072297 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7938143460 0x7f793815aba0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f79280097a0 tx=0x7f792802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2531bba43365d2db server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:56.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.320+0000 7f7935600640 1 -- 172.21.15.102:0/2126072297 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7928030800 con 0x7f7938143460 2024-03-23T03:51:56.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.320+0000 7f7935600640 1 -- 172.21.15.102:0/2126072297 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7928030960 con 0x7f7938143460 2024-03-23T03:51:56.320 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.320+0000 7f7935600640 1 -- 172.21.15.102:0/2126072297 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7928030cf0 con 0x7f7938143460 2024-03-23T03:51:56.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.320+0000 7f7937e00640 1 -- 172.21.15.102:0/2126072297 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7938143460 msgr2=0x7f793815aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:56.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.320+0000 7f7937e00640 1 --2- 172.21.15.102:0/2126072297 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7938143460 0x7f793815aba0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f79280097a0 tx=0x7f792802b940 comp rx=0 tx=0).stop 2024-03-23T03:51:56.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.321+0000 7f7937e00640 1 -- 172.21.15.102:0/2126072297 shutdown_connections 2024-03-23T03:51:56.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.321+0000 7f7937e00640 1 --2- 172.21.15.102:0/2126072297 >> v2:172.21.15.173:3300/0 conn(0x7f7938143930 0x7f79380cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.321 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.321+0000 7f7937e00640 1 --2- 172.21.15.102:0/2126072297 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7938143460 0x7f793815aba0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.321+0000 7f7937e00640 1 -- 172.21.15.102:0/2126072297 >> 172.21.15.102:0/2126072297 conn(0x7f7938065df0 msgr2=0x7f7938069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:56.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.321+0000 7f7937e00640 1 -- 172.21.15.102:0/2126072297 shutdown_connections 2024-03-23T03:51:56.322 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.321+0000 7f7937e00640 1 -- 172.21.15.102:0/2126072297 wait complete. 2024-03-23T03:51:56.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.322+0000 7f7937e00640 1 Processor -- start 2024-03-23T03:51:56.323 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.322+0000 7f7937e00640 1 -- start start 2024-03-23T03:51:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.323+0000 7f7937e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79380cdc50 0x7f79380ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.323+0000 7f7937e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7938143460 0x7f793814d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.323+0000 7f7937e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7938143930 0x7f79380b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.323+0000 7f7937e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79380ae620 con 0x7f7938143460 2024-03-23T03:51:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.323+0000 7f7937e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79381391e0 con 0x7f79380cdc50 2024-03-23T03:51:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.323+0000 7f7937e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79380afdc0 con 0x7f7938143930 2024-03-23T03:51:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.324+0000 7f7936000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7938143460 0x7f793814d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.324+0000 7f7936000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7938143460 0x7f793814d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43540/0 (socket says 172.21.15.102:43540) 2024-03-23T03:51:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.324+0000 7f7936000640 1 -- 172.21.15.102:0/31420964 learned_addr learned my addr 172.21.15.102:0/31420964 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.324+0000 7f7936a00640 1 --2- 172.21.15.102:0/31420964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79380cdc50 0x7f79380ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:56.325 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.324+0000 7f7936000640 1 -- 172.21.15.102:0/31420964 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7938143930 msgr2=0x7f79380b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:56.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.324+0000 7f7936000640 1 --2- 172.21.15.102:0/31420964 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7938143930 0x7f79380b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.324+0000 7f7936000640 1 -- 172.21.15.102:0/31420964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79380cdc50 msgr2=0x7f79380ae0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:56.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.324+0000 7f7936000640 1 --2- 172.21.15.102:0/31420964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79380cdc50 0x7f79380ae0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.324+0000 7f7936000640 1 -- 172.21.15.102:0/31420964 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79240094d0 con 0x7f7938143460 2024-03-23T03:51:56.326 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.325+0000 7f7936000640 1 --2- 172.21.15.102:0/31420964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7938143460 0x7f793814d0a0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f792400e9b0 tx=0x7f792400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:56.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.325+0000 7f792f400640 1 -- 172.21.15.102:0/31420964 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792400c9a0 con 0x7f7938143460 2024-03-23T03:51:56.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.325+0000 7f792f400640 1 -- 172.21.15.102:0/31420964 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f792400cb00 con 0x7f7938143460 2024-03-23T03:51:56.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.325+0000 7f7937e00640 1 -- 172.21.15.102:0/31420964 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7928009400 con 0x7f7938143460 2024-03-23T03:51:56.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.325+0000 7f792f400640 1 -- 172.21.15.102:0/31420964 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792400ce70 con 0x7f7938143460 2024-03-23T03:51:56.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.325+0000 7f7937e00640 1 -- 172.21.15.102:0/31420964 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79380b0380 con 0x7f7938143460 2024-03-23T03:51:56.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.326+0000 7f7937e00640 1 -- 172.21.15.102:0/31420964 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f793804e030 con 0x7f7938143460 2024-03-23T03:51:56.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.327+0000 7f792f400640 1 -- 172.21.15.102:0/31420964 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f7924013470 con 0x7f7938143460 2024-03-23T03:51:56.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.328+0000 7f792f400640 1 --2- 172.21.15.102:0/31420964 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f791806ac00 0x7f791806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:56.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.328+0000 7f792f400640 1 -- 172.21.15.102:0/31420964 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f79240897c0 con 0x7f7938143460 2024-03-23T03:51:56.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.328+0000 7f7936a00640 1 --2- 172.21.15.102:0/31420964 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f791806ac00 0x7f791806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:56.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.329+0000 7f7936a00640 1 --2- 172.21.15.102:0/31420964 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f791806ac00 0x7f791806d0c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f7928036000 tx=0x7f792803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:56.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.333+0000 7f792f400640 1 -- 172.21.15.102:0/31420964 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f7924054b60 con 0x7f7938143460 2024-03-23T03:51:56.525 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.524+0000 7f7937e00640 1 -- 172.21.15.102:0/31420964 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7938144080 con 0x7f791806ac00 2024-03-23T03:51:56.531 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.530+0000 7f792f400640 1 -- 172.21.15.102:0/31420964 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48848 (secure 0 0 0) 0x7f7928030b50 con 0x7f791806ac00 2024-03-23T03:51:56.532 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:56.532 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:51:56.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.533+0000 7f7937e00640 1 -- 172.21.15.102:0/31420964 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f791806ac00 msgr2=0x7f791806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:56.534 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.533+0000 7f7937e00640 1 --2- 172.21.15.102:0/31420964 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f791806ac00 0x7f791806d0c0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f7928036000 tx=0x7f792803b040 comp rx=0 tx=0).stop 2024-03-23T03:51:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.533+0000 7f7937e00640 1 -- 172.21.15.102:0/31420964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7938143460 msgr2=0x7f793814d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.533+0000 7f7937e00640 1 --2- 172.21.15.102:0/31420964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7938143460 0x7f793814d0a0 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7f792400e9b0 tx=0x7f792400ee70 comp rx=0 tx=0).stop 2024-03-23T03:51:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.533+0000 7f7937e00640 1 -- 172.21.15.102:0/31420964 shutdown_connections 2024-03-23T03:51:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.533+0000 7f7937e00640 1 --2- 172.21.15.102:0/31420964 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f791806ac00 0x7f791806d0c0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.533+0000 7f7937e00640 1 --2- 172.21.15.102:0/31420964 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f7938143930 0x7f79380b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.533+0000 7f7937e00640 1 --2- 172.21.15.102:0/31420964 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7938143460 0x7f793814d0a0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.533+0000 7f7937e00640 1 --2- 172.21.15.102:0/31420964 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f79380cdc50 0x7f79380ae0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.533+0000 7f7937e00640 1 -- 172.21.15.102:0/31420964 >> 172.21.15.102:0/31420964 conn(0x7f7938065df0 msgr2=0x7f793813f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.533+0000 7f7937e00640 1 -- 172.21.15.102:0/31420964 shutdown_connections 2024-03-23T03:51:56.535 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.533+0000 7f7937e00640 1 -- 172.21.15.102:0/31420964 wait complete. 2024-03-23T03:51:56.554 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-03-23T03:51:55.810841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":259280,"kb_used_data":14448,"kb_used_omap":12,"kb_used_meta":244723,"kb_avail":754715440,"statfs":{"total":773094113280,"available":772828610560,"internally_reserved":0,"allocated":14794752,"data_stored":8755518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":250596922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1352135,"num_objects":-47,"num_object_clones":0,"num_object_copies":-94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-47,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-166,"num_write_kb":-21904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.137981"},"pg_stats":[{"pgid":"1.0","version":"19'117","reported_seq":233,"reported_epoch":107,"state":"active+clean","last_fresh":"2024-03-23T03:51:43.763852+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:51:43.763852+0000","last_peered":"2024-03-23T03:51:43.763852+0000","last_clean":"2024-03-23T03:51:43.763852+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:51:43.763852+0000","last_undegraded":"2024-03-23T03:51:43.763852+0000","last_fullsized":"2024-03-23T03:51:43.763852+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542209,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1970176,"data_stored":1199353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.087,"15min":1.0720000000000001},"min":{"1min":0.65700000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.5209999999999999,"5min":1.635,"15min":1.635},"last":1.478},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.042,"15min":1.0469999999999999},"min":{"1min":0.749,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.393,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.431}]},{"osd":1,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.081,"15min":1.0529999999999999},"min":{"1min":0.71499999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6120000000000001,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.6579999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.1719999999999999,"15min":1.2809999999999999},"min":{"1min":0.79000000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.321,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":1.21}]},{"osd":2,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0940000000000001,"5min":1.125,"15min":1.151},"min":{"1min":0.81299999999999994,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.583,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.544},{"interface":"front","average":{"1min":1.167,"5min":1.2869999999999999,"15min":1.401},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.506,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.276}]},{"osd":3,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.298,"15min":1.4179999999999999},"min":{"1min":0.71999999999999997,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.4219999999999999,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.3009999999999999,"5min":1.325,"15min":1.353},"min":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.677,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.5840000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.254,"15min":1.2629999999999999},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":1.8220000000000001,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.51},{"interface":"front","average":{"1min":1.333,"5min":1.347,"15min":1.3720000000000001},"min":{"1min":0.67500000000000004,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.778,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.3520000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.389,"15min":1.4179999999999999},"min":{"1min":0.81299999999999994,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.7310000000000001,"5min":2.645,"15min":2.645},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.361,"5min":1.417,"15min":1.446},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.7649999999999999,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.319}]},{"osd":6,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.504,"15min":1.552},"min":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.9279999999999999,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.474,"15min":1.5149999999999999},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9399999999999999,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.452}]}]},{"osd":6,"up_from":14,"seq":60129542207,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33780,"kb_used_data":2664,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338060,"statfs":{"total":96636764160,"available":96602173440,"internally_reserved":0,"allocated":2727936,"data_stored":1960735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1040000000000001,"5min":1.0429999999999999,"15min":1.044},"min":{"1min":0.69699999999999995,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.7330000000000001,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.05,"15min":1.04},"min":{"1min":0.66100000000000003,"5min":0.62,"15min":0.62},"max":{"1min":1.5409999999999999,"5min":1.946,"15min":1.946},"last":0.96299999999999997}]},{"osd":1,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.087,"15min":1.075},"min":{"1min":0.745,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.6839999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":1.5509999999999999},{"interface":"front","average":{"1min":1.196,"5min":1.1339999999999999,"15min":1.1579999999999999},"min":{"1min":0.80500000000000005,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.5369999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.181}]},{"osd":2,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.083,"15min":1.071},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.4350000000000001,"5min":2.044,"15min":2.044},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.133,"5min":1.1759999999999999,"15min":1.212},"min":{"1min":0.85499999999999998,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.6599999999999999,"5min":1.681,"15min":1.681},"last":1.4450000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.2370000000000001,"15min":1.276},"min":{"1min":0.88100000000000001,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.786,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.2330000000000001,"15min":1.2569999999999999},"min":{"1min":0.90100000000000002,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.446,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.085}]},{"osd":4,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.3779999999999999,"15min":1.385},"min":{"1min":1.1930000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.6679999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.431,"5min":1.371,"15min":1.3859999999999999},"min":{"1min":1.234,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.639,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.6140000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.403,"15min":1.4319999999999999},"min":{"1min":1.0669999999999999,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.794,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.474,"5min":1.508,"15min":1.542},"min":{"1min":1.091,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.8200000000000001,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.48}]},{"osd":7,"last update":"Sat Mar 23 03:50:55 2024","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.4059999999999999,"15min":1.4119999999999999},"min":{"1min":0.95999999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":2.0760000000000001,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.482,"15min":1.5309999999999999},"min":{"1min":1.125,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.8660000000000001,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.667}]}]},{"osd":5,"up_from":14,"seq":60129542207,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28340,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343500,"statfs":{"total":96636764160,"available":96607744000,"internally_reserved":0,"allocated":1351680,"data_stored":608821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.1439999999999999,"15min":1.2030000000000001},"min":{"1min":0.72999999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.4830000000000001,"5min":1.806,"15min":1.806},"last":0.91600000000000004},{"interface":"front","average":{"1min":1.113,"5min":1.1399999999999999,"15min":1.1759999999999999},"min":{"1min":0.76500000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.855,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.0649999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.054,"5min":1.0940000000000001,"15min":1.1000000000000001},"min":{"1min":0.80100000000000005,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.3520000000000001,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":0.72899999999999998},{"interface":"front","average":{"1min":1.232,"5min":1.165,"15min":1.1930000000000001},"min":{"1min":0.84799999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.663,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.101}]},{"osd":2,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.274,"15min":1.3460000000000001},"min":{"1min":0.70499999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.8939999999999999,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.268},{"interface":"front","average":{"1min":1.226,"5min":1.2989999999999999,"15min":1.3660000000000001},"min":{"1min":0.875,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.579,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.028}]},{"osd":3,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.2,"15min":1.244},"min":{"1min":0.85399999999999998,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.7050000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.375,"15min":1.446},"min":{"1min":0.997,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.964,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.4570000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3049999999999999,"15min":1.399},"min":{"1min":0.65700000000000003,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6100000000000001,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3859999999999999,"15min":1.4359999999999999},"min":{"1min":0.83699999999999997,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.8069999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":0.98299999999999998}]},{"osd":6,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.4159999999999999,"15min":1.5},"min":{"1min":0.83399999999999996,"5min":0.63,"15min":0.63},"max":{"1min":1.5840000000000001,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.5409999999999999,"15min":1.639},"min":{"1min":1.1950000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":1.6359999999999999,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.3120000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:50:53 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4910000000000001,"15min":1.591},"min":{"1min":1.095,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.9259999999999999,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5840000000000001,"15min":1.6839999999999999},"min":{"1min":1.272,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.056,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.375}]}]},{"osd":4,"up_from":14,"seq":60129542208,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1376256,"data_stored":608919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.98399999999999999,"5min":1.004,"15min":0.98099999999999998},"min":{"1min":0.624,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.47,"5min":1.7789999999999999,"15min":1.7789999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.995,"15min":1.0049999999999999},"min":{"1min":0.56100000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3400000000000001,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.365}]},{"osd":1,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0680000000000001,"15min":1.05},"min":{"1min":0.65100000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.5,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.234},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0880000000000001,"15min":1.0640000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.5209999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.304}]},{"osd":2,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.2470000000000001,"15min":1.3089999999999999},"min":{"1min":0.70399999999999996,"5min":0.625,"15min":0.625},"max":{"1min":1.452,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.151,"5min":1.2789999999999999,"15min":1.2969999999999999},"min":{"1min":0.59899999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6859999999999999,"5min":2.181,"15min":2.181},"last":1.4119999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.1160000000000001,"15min":1.1240000000000001},"min":{"1min":0.61899999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.4419999999999999,"5min":1.821,"15min":1.821},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.141,"5min":1.2310000000000001,"15min":1.268},"min":{"1min":0.73799999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.599,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.0820000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.2370000000000001,"15min":1.26},"min":{"1min":0.63800000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.49,"5min":1.929,"15min":1.929},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.2709999999999999,"15min":1.274},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.45,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.5089999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.371,"15min":1.409},"min":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6719999999999999,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.345},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.4139999999999999,"15min":1.3999999999999999},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.8100000000000001,"5min":2.016,"15min":2.016},"last":1.617}]},{"osd":7,"last update":"Sat Mar 23 03:50:54 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.361,"15min":1.363},"min":{"1min":0.68200000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.5569999999999999,"5min":2.3479999999999999,"15min":2.3479999999999999},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.2989999999999999,"5min":1.456,"15min":1.4870000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.716,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.538}]}]},{"osd":3,"up_from":14,"seq":60129542208,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32432,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339408,"statfs":{"total":96636764160,"available":96603553792,"internally_reserved":0,"allocated":1347584,"data_stored":609398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.81999999999999995,"15min":0.85499999999999998},"min":{"1min":0.35499999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.4419999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.92200000000000004,"15min":0.92400000000000004},"min":{"1min":0.502,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.538,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.41399999999999998}]},{"osd":1,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.91500000000000004,"15min":0.96799999999999997},"min":{"1min":0.58099999999999996,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.9990000000000001,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.034,"5min":1.0229999999999999,"15min":1.069},"min":{"1min":0.52000000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.651,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.54300000000000004}]},{"osd":2,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.002,"15min":1.0509999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":2.5710000000000002,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1240000000000001,"15min":1.1499999999999999},"min":{"1min":0.48599999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.7649999999999999,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":0.65300000000000002}]},{"osd":4,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.2410000000000001,"15min":1.248},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":4.0380000000000003,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.337,"5min":1.351,"15min":1.429},"min":{"1min":0.77000000000000002,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.3450000000000002,"5min":4.819,"15min":4.819},"last":0.84999999999999998}]},{"osd":5,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.6040000000000001,"15min":1.4470000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":2.2679999999999998,"5min":30.228000000000002,"15min":30.228000000000002},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.337,"15min":1.357},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":4.3630000000000004,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":0.64300000000000002}]},{"osd":6,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.4119999999999999,"15min":1.508},"min":{"1min":0.83999999999999997,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.6589999999999998,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.137},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.4199999999999999,"15min":1.4039999999999999},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.4039999999999999,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":0.872}]},{"osd":7,"last update":"Sat Mar 23 03:50:57 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.389,"15min":1.4199999999999999},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":2.0880000000000001,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.4390000000000001,"5min":1.542,"15min":1.5900000000000001},"min":{"1min":0.80500000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.4780000000000002,"5min":10.663,"15min":10.663},"last":0.76100000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542208,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32440,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339400,"statfs":{"total":96636764160,"available":96603545600,"internally_reserved":0,"allocated":1355776,"data_stored":608853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83799999999999997,"15min":0.877},"min":{"1min":0.52100000000000002,"5min":0.307,"15min":0.307},"max":{"1min":1.4259999999999999,"5min":1.456,"15min":1.456},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.86399999999999999,"15min":0.93600000000000005},"min":{"1min":0.39500000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.4690000000000001,"5min":2.226,"15min":2.226},"last":0.77600000000000002}]},{"osd":1,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.93899999999999995,"15min":0.98499999999999999},"min":{"1min":0.46800000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.377,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.85899999999999999,"15min":0.91000000000000003},"min":{"1min":0.57599999999999996,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.2210000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.52400000000000002}]},{"osd":3,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.96099999999999997,"15min":1.018},"min":{"1min":0.47499999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.403,"5min":1.6419999999999999,"15min":1.6419999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.032,"5min":1.042,"15min":1.0800000000000001},"min":{"1min":0.61099999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.605,"5min":1.605,"15min":1.605},"last":0.95099999999999996}]},{"osd":4,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.194,"15min":1.2310000000000001},"min":{"1min":0.82399999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.7270000000000001,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.157,"15min":1.2070000000000001},"min":{"1min":0.83799999999999997,"5min":0.627,"15min":0.627},"max":{"1min":1.4950000000000001,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.276}]},{"osd":5,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.3100000000000001,"15min":1.367},"min":{"1min":0.76900000000000002,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.6259999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.228,"5min":1.2749999999999999,"15min":1.339},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.819,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":0.99199999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.232,"15min":1.3280000000000001},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.6479999999999999,"5min":2.456,"15min":2.456},"last":1.242},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.3999999999999999,"15min":1.476},"min":{"1min":0.85299999999999998,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.097,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.0329999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:59 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.3069999999999999,"15min":1.369},"min":{"1min":0.79800000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.302,"15min":1.395},"min":{"1min":0.78700000000000003,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5860000000000001,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.0129999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542207,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33468,"kb_used_data":2352,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338372,"statfs":{"total":96636764160,"available":96602492928,"internally_reserved":0,"allocated":2408448,"data_stored":1657432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.79100000000000004,"15min":0.86099999999999999},"min":{"1min":0.41599999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.1240000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.81799999999999995,"15min":0.82999999999999996},"min":{"1min":0.435,"5min":0.374,"15min":0.374},"max":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":1.0469999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.92200000000000004,"15min":0.96399999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.4430000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.087},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.92400000000000004,"15min":0.97799999999999998},"min":{"1min":0.496,"5min":0.443,"15min":0.443},"max":{"1min":1.2210000000000001,"5min":1.625,"15min":1.625},"last":0.89700000000000002}]},{"osd":3,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.046,"5min":1.103,"15min":1.113},"min":{"1min":0.68400000000000005,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.6559999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.875,"5min":0.96199999999999997,"15min":1.008},"min":{"1min":0.59499999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.6180000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.64900000000000002}]},{"osd":4,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.179,"15min":1.2749999999999999},"min":{"1min":0.67700000000000005,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.5069999999999999,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.246,"15min":1.3049999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6120000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.133}]},{"osd":5,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.248,"15min":1.3200000000000001},"min":{"1min":0.79500000000000004,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.95,"5min":3.593,"15min":3.593},"last":1.27},{"interface":"front","average":{"1min":1.264,"5min":1.327,"15min":1.4259999999999999},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.671,"5min":2.359,"15min":2.359},"last":1.4850000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.3089999999999999,"15min":1.3919999999999999},"min":{"1min":0.84199999999999997,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.6200000000000001,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.3839999999999999},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.288,"15min":1.3540000000000001},"min":{"1min":0.89100000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.0209999999999999,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.4199999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:50:56 2024","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.359,"15min":1.4330000000000001},"min":{"1min":0.99299999999999999,"5min":0.627,"15min":0.627},"max":{"1min":2.0840000000000001,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.363,"15min":1.486},"min":{"1min":0.76600000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.673,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.613}]}]},{"osd":0,"up_from":14,"seq":60129542207,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33320,"kb_used_data":2204,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338520,"statfs":{"total":96636764160,"available":96602644480,"internally_reserved":0,"allocated":2256896,"data_stored":1502007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.86399999999999999,"15min":0.93500000000000005},"min":{"1min":0.48399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.478,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.94799999999999995,"15min":0.98199999999999998},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.456,"5min":1.506,"15min":1.506},"last":1.643}]},{"osd":2,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.92700000000000005,"15min":0.99099999999999999},"min":{"1min":0.52900000000000003,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.4399999999999999,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.321},{"interface":"front","average":{"1min":0.98399999999999999,"5min":1.0349999999999999,"15min":1.095},"min":{"1min":0.45300000000000001,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.601,"5min":1.901,"15min":1.901},"last":1.1359999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.93300000000000005,"15min":0.996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.474,"5min":1.972,"15min":1.972},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.0049999999999999,"5min":1.0669999999999999,"15min":1.147},"min":{"1min":0.60299999999999998,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.597,"5min":2.298,"15min":2.298},"last":1.409}]},{"osd":4,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.288,"15min":1.3360000000000001},"min":{"1min":0.72799999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.7090000000000001,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":1.708},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.2230000000000001,"15min":1.27},"min":{"1min":0.80600000000000005,"5min":0.76800000000000002,"15min":0.76800000000000002},"max":{"1min":1.8380000000000001,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.5229999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3100000000000001,"15min":1.3700000000000001},"min":{"1min":0.89500000000000002,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.948,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.377,"15min":1.4379999999999999},"min":{"1min":0.83499999999999996,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.8700000000000001,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.4870000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.4099999999999999,"15min":1.5209999999999999},"min":{"1min":0.80000000000000004,"5min":0.747,"15min":0.747},"max":{"1min":1.978,"5min":3.032,"15min":3.032},"last":1.8009999999999999},{"interface":"front","average":{"1min":1.371,"5min":1.411,"15min":1.4870000000000001},"min":{"1min":0.84699999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.1349999999999998,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.748}]},{"osd":7,"last update":"Sat Mar 23 03:50:51 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.4390000000000001,"15min":1.506},"min":{"1min":0.90500000000000003,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.1850000000000001,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.8540000000000001},{"interface":"front","average":{"1min":1.2909999999999999,"5min":1.3620000000000001,"15min":1.419},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.873,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.2190000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:51:56.555 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T03:51:56.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.760+0000 7fd522e00640 1 Processor -- start 2024-03-23T03:51:56.761 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.760+0000 7fd522e00640 1 -- start start 2024-03-23T03:51:56.762 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.762+0000 7fd522e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd51c15cda0 0x7fd51c0cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:56.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.762+0000 7fd522e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd51c0b4a60 0x7fd51c0b5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:56.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.762+0000 7fd522e00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd51c0d1830 con 0x7fd51c0b5a30 2024-03-23T03:51:56.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.762+0000 7fd522e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd51c0b1f20 con 0x7fd51c0b4a60 2024-03-23T03:51:56.763 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.762+0000 7fd522e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd51c0d0c20 con 0x7fd51c15cda0 2024-03-23T03:51:56.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.763+0000 7fd520e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd51c0b4a60 0x7fd51c0b5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:56.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.763+0000 7fd51be00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd51c15cda0 0x7fd51c0cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:56.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.763+0000 7fd520e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd51c0b4a60 0x7fd51c0b5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:53308/0 (socket says 172.21.15.102:53308) 2024-03-23T03:51:56.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.763+0000 7fd520e00640 1 -- 172.21.15.102:0/1804554803 learned_addr learned my addr 172.21.15.102:0/1804554803 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:56.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.763+0000 7fd51be00640 1 -- 172.21.15.102:0/1804554803 >> v2:172.21.15.173:3300/0 conn(0x7fd51c0b4a60 msgr2=0x7fd51c0b5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:56.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.763+0000 7fd51be00640 1 --2- 172.21.15.102:0/1804554803 >> v2:172.21.15.173:3300/0 conn(0x7fd51c0b4a60 0x7fd51c0b5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.763+0000 7fd51be00640 1 -- 172.21.15.102:0/1804554803 >> v1:172.21.15.173:6789/0 conn(0x7fd51c0b5a30 legacy=0x7fd51c0b4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:56.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.763+0000 7fd51be00640 1 -- 172.21.15.102:0/1804554803 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51c0d0f00 con 0x7fd51c15cda0 2024-03-23T03:51:56.764 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.763+0000 7fd520e00640 1 --2- 172.21.15.102:0/1804554803 >> v2:172.21.15.173:3300/0 conn(0x7fd51c0b4a60 0x7fd51c0b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T03:51:56.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.765+0000 7fd51be00640 1 --2- 172.21.15.102:0/1804554803 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd51c15cda0 0x7fd51c0cdb40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0097a0 tx=0x7fd50c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c9364c68f2e0a06 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:56.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.765+0000 7fd51b400640 1 -- 172.21.15.102:0/1804554803 <== mon.2 v2:172.21.15.173:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c02d040 con 0x7fd51c15cda0 2024-03-23T03:51:56.765 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.765+0000 7fd51b400640 1 -- 172.21.15.102:0/1804554803 <== mon.2 v2:172.21.15.173:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd50c031810 con 0x7fd51c15cda0 2024-03-23T03:51:56.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.765+0000 7fd51b400640 1 -- 172.21.15.102:0/1804554803 <== mon.2 v2:172.21.15.173:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c031ba0 con 0x7fd51c15cda0 2024-03-23T03:51:56.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.766+0000 7fd522e00640 1 -- 172.21.15.102:0/1804554803 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd51c15cda0 msgr2=0x7fd51c0cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:56.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.766+0000 7fd522e00640 1 --2- 172.21.15.102:0/1804554803 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd51c15cda0 0x7fd51c0cdb40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0097a0 tx=0x7fd50c02b8b0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.766+0000 7fd522e00640 1 -- 172.21.15.102:0/1804554803 shutdown_connections 2024-03-23T03:51:56.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.766+0000 7fd522e00640 1 --2- 172.21.15.102:0/1804554803 >> v2:172.21.15.173:3300/0 conn(0x7fd51c0b4a60 0x7fd51c0b5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.766 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.766+0000 7fd522e00640 1 --2- 172.21.15.102:0/1804554803 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd51c15cda0 0x7fd51c0cdb40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.766+0000 7fd522e00640 1 -- 172.21.15.102:0/1804554803 >> 172.21.15.102:0/1804554803 conn(0x7fd51c0cf350 msgr2=0x7fd51c0cf760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:56.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.766+0000 7fd522e00640 1 -- 172.21.15.102:0/1804554803 shutdown_connections 2024-03-23T03:51:56.767 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.766+0000 7fd522e00640 1 -- 172.21.15.102:0/1804554803 wait complete. 2024-03-23T03:51:56.768 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.768+0000 7fd522e00640 1 Processor -- start 2024-03-23T03:51:56.769 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.768+0000 7fd522e00640 1 -- start start 2024-03-23T03:51:56.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.769+0000 7fd522e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd51c0b4a60 0x7fd51c1f1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:56.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.769+0000 7fd522e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd51c0b5a30 0x7fd51c0d1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:56.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.770+0000 7fd522e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd51c15cda0 0x7fd51c0d2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:56.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.770+0000 7fd522e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51c1f22a0 con 0x7fd51c0b5a30 2024-03-23T03:51:56.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.770+0000 7fd522e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51c14b840 con 0x7fd51c0b4a60 2024-03-23T03:51:56.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.770+0000 7fd522e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd51c14bb20 con 0x7fd51c15cda0 2024-03-23T03:51:56.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.770+0000 7fd513400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd51c0b5a30 0x7fd51c0d1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:56.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.770+0000 7fd513400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd51c0b5a30 0x7fd51c0d1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43554/0 (socket says 172.21.15.102:43554) 2024-03-23T03:51:56.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.770+0000 7fd513400640 1 -- 172.21.15.102:0/1491276929 learned_addr learned my addr 172.21.15.102:0/1491276929 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:56.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.770+0000 7fd51be00640 1 --2- 172.21.15.102:0/1491276929 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd51c0b4a60 0x7fd51c1f1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:56.772 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.770+0000 7fd520e00640 1 --2- 172.21.15.102:0/1491276929 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd51c15cda0 0x7fd51c0d2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:56.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.770+0000 7fd513400640 1 -- 172.21.15.102:0/1491276929 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd51c15cda0 msgr2=0x7fd51c0d2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:56.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.770+0000 7fd513400640 1 --2- 172.21.15.102:0/1491276929 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd51c15cda0 0x7fd51c0d2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.770+0000 7fd513400640 1 -- 172.21.15.102:0/1491276929 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd51c0b4a60 msgr2=0x7fd51c1f1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:56.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.770+0000 7fd513400640 1 --2- 172.21.15.102:0/1491276929 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd51c0b4a60 0x7fd51c1f1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.770+0000 7fd513400640 1 -- 172.21.15.102:0/1491276929 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd514009b30 con 0x7fd51c0b5a30 2024-03-23T03:51:56.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.771+0000 7fd513400640 1 --2- 172.21.15.102:0/1491276929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd51c0b5a30 0x7fd51c0d1cc0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fd50400c450 tx=0x7fd50400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:56.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.771+0000 7fd518c00640 1 -- 172.21.15.102:0/1491276929 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5040052d0 con 0x7fd51c0b5a30 2024-03-23T03:51:56.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.771+0000 7fd518c00640 1 -- 172.21.15.102:0/1491276929 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd504005430 con 0x7fd51c0b5a30 2024-03-23T03:51:56.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.771+0000 7fd518c00640 1 -- 172.21.15.102:0/1491276929 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd504012870 con 0x7fd51c0b5a30 2024-03-23T03:51:56.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.771+0000 7fd522e00640 1 -- 172.21.15.102:0/1491276929 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd50c009400 con 0x7fd51c0b5a30 2024-03-23T03:51:56.773 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.771+0000 7fd522e00640 1 -- 172.21.15.102:0/1491276929 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd51c14c0e0 con 0x7fd51c0b5a30 2024-03-23T03:51:56.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.772+0000 7fd522e00640 1 -- 172.21.15.102:0/1491276929 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd51c073390 con 0x7fd51c0b5a30 2024-03-23T03:51:56.774 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.773+0000 7fd518c00640 1 -- 172.21.15.102:0/1491276929 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd5040129d0 con 0x7fd51c0b5a30 2024-03-23T03:51:56.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.774+0000 7fd518c00640 1 --2- 172.21.15.102:0/1491276929 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd4f806ac80 0x7fd4f806d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:56.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.775+0000 7fd518c00640 1 -- 172.21.15.102:0/1491276929 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fd504088ba0 con 0x7fd51c0b5a30 2024-03-23T03:51:56.775 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.775+0000 7fd51be00640 1 --2- 172.21.15.102:0/1491276929 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd4f806ac80 0x7fd4f806d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:56.776 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.776+0000 7fd51be00640 1 --2- 172.21.15.102:0/1491276929 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd4f806ac80 0x7fd4f806d140 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd50c036000 tx=0x7fd50c03c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:56.779 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.779+0000 7fd518c00640 1 -- 172.21.15.102:0/1491276929 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd504054020 con 0x7fd51c0b5a30 2024-03-23T03:51:56.973 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.972+0000 7fd522e00640 1 -- 172.21.15.102:0/1491276929 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd51c0c6160 con 0x7fd51c0b5a30 2024-03-23T03:51:56.975 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.974+0000 7fd518c00640 1 -- 172.21.15.102:0/1491276929 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+12781 (secure 0 0 0) 0x7fd504057640 con 0x7fd51c0b5a30 2024-03-23T03:51:56.975 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:51:56.976 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":110,"fsid":"776b2820-05f2-4e50-8d40-18dd76c5f6a0","created":"2024-03-23T03:46:34.849248+0000","modified":"2024-03-23T03:51:52.816382+0000","last_up_change":"2024-03-23T03:46:46.328969+0000","last_in_change":"2024-03-23T03:46:38.365403+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T03:46:47.784997+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2c69740f-05ae-4bab-abe9-08e4e449e501","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":37351},{"type":"v1","addr":"172.21.15.102:6817","nonce":37351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":37351},{"type":"v1","addr":"172.21.15.102:6819","nonce":37351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":37351},{"type":"v1","addr":"172.21.15.102:6823","nonce":37351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":37351},{"type":"v1","addr":"172.21.15.102:6821","nonce":37351}]},"public_addr":"172.21.15.102:6817/37351","cluster_addr":"172.21.15.102:6819/37351","heartbeat_back_addr":"172.21.15.102:6823/37351","heartbeat_front_addr":"172.21.15.102:6821/37351","state":["exists","up"]},{"osd":1,"uuid":"e23064a9-e519-4076-a8c3-a36a6e595694","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":37352},{"type":"v1","addr":"172.21.15.102:6825","nonce":37352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":37352},{"type":"v1","addr":"172.21.15.102:6827","nonce":37352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":37352},{"type":"v1","addr":"172.21.15.102:6831","nonce":37352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":37352},{"type":"v1","addr":"172.21.15.102:6829","nonce":37352}]},"public_addr":"172.21.15.102:6825/37352","cluster_addr":"172.21.15.102:6827/37352","heartbeat_back_addr":"172.21.15.102:6831/37352","heartbeat_front_addr":"172.21.15.102:6829/37352","state":["exists","up"]},{"osd":2,"uuid":"d5c691e5-4b59-4203-9483-09d0a433e855","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":37350},{"type":"v1","addr":"172.21.15.102:6801","nonce":37350}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":37350},{"type":"v1","addr":"172.21.15.102:6803","nonce":37350}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":37350},{"type":"v1","addr":"172.21.15.102:6807","nonce":37350}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":37350},{"type":"v1","addr":"172.21.15.102:6805","nonce":37350}]},"public_addr":"172.21.15.102:6801/37350","cluster_addr":"172.21.15.102:6803/37350","heartbeat_back_addr":"172.21.15.102:6807/37350","heartbeat_front_addr":"172.21.15.102:6805/37350","state":["exists","up"]},{"osd":3,"uuid":"69470c30-bacd-4c78-a774-0a17ad1608b1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":37353},{"type":"v1","addr":"172.21.15.102:6809","nonce":37353}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":37353},{"type":"v1","addr":"172.21.15.102:6811","nonce":37353}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":37353},{"type":"v1","addr":"172.21.15.102:6815","nonce":37353}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":37353},{"type":"v1","addr":"172.21.15.102:6813","nonce":37353}]},"public_addr":"172.21.15.102:6809/37353","cluster_addr":"172.21.15.102:6811/37353","heartbeat_back_addr":"172.21.15.102:6815/37353","heartbeat_front_addr":"172.21.15.102:6813/37353","state":["exists","up"]},{"osd":4,"uuid":"939c44ed-ffc0-4066-b4a8-667943081e0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":37509},{"type":"v1","addr":"172.21.15.173:6817","nonce":37509}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6819","nonce":37509},{"type":"v1","addr":"172.21.15.173:6820","nonce":37509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6825","nonce":37509},{"type":"v1","addr":"172.21.15.173:6827","nonce":37509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":37509},{"type":"v1","addr":"172.21.15.173:6823","nonce":37509}]},"public_addr":"172.21.15.173:6817/37509","cluster_addr":"172.21.15.173:6820/37509","heartbeat_back_addr":"172.21.15.173:6827/37509","heartbeat_front_addr":"172.21.15.173:6823/37509","state":["exists","up"]},{"osd":5,"uuid":"7f78f3b3-7b77-45fc-b705-6637c50bf18e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":103,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":37507},{"type":"v1","addr":"172.21.15.173:6801","nonce":37507}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":37507},{"type":"v1","addr":"172.21.15.173:6803","nonce":37507}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":37507},{"type":"v1","addr":"172.21.15.173:6807","nonce":37507}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":37507},{"type":"v1","addr":"172.21.15.173:6805","nonce":37507}]},"public_addr":"172.21.15.173:6801/37507","cluster_addr":"172.21.15.173:6803/37507","heartbeat_back_addr":"172.21.15.173:6807/37507","heartbeat_front_addr":"172.21.15.173:6805/37507","state":["exists","up"]},{"osd":6,"uuid":"300271a9-e4bd-41cc-a4b5-7ad4e466f6c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":37508},{"type":"v1","addr":"172.21.15.173:6809","nonce":37508}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":37508},{"type":"v1","addr":"172.21.15.173:6811","nonce":37508}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":37508},{"type":"v1","addr":"172.21.15.173:6815","nonce":37508}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":37508},{"type":"v1","addr":"172.21.15.173:6813","nonce":37508}]},"public_addr":"172.21.15.173:6809/37508","cluster_addr":"172.21.15.173:6811/37508","heartbeat_back_addr":"172.21.15.173:6815/37508","heartbeat_front_addr":"172.21.15.173:6813/37508","state":["exists","up"]},{"osd":7,"uuid":"d242af13-3439-4b6b-8c84-08a5a27e639c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":106,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":37510},{"type":"v1","addr":"172.21.15.173:6821","nonce":37510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6824","nonce":37510},{"type":"v1","addr":"172.21.15.173:6826","nonce":37510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6830","nonce":37510},{"type":"v1","addr":"172.21.15.173:6831","nonce":37510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6828","nonce":37510},{"type":"v1","addr":"172.21.15.173:6829","nonce":37510}]},"public_addr":"172.21.15.173:6821/37510","cluster_addr":"172.21.15.173:6826/37510","heartbeat_back_addr":"172.21.15.173:6831/37510","heartbeat_front_addr":"172.21.15.173:6829/37510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.709006+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.670522+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.938191+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.674655+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.990953+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.681561+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:43.928103+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T03:46:45.421901+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.102:6837/883705787":"2024-03-24T03:51:49.668429+0000","172.21.15.102:6836/883705787":"2024-03-24T03:51:49.668429+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T03:51:56.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.977+0000 7fd522e00640 1 -- 172.21.15.102:0/1491276929 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd4f806ac80 msgr2=0x7fd4f806d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:56.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.977+0000 7fd522e00640 1 --2- 172.21.15.102:0/1491276929 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd4f806ac80 0x7fd4f806d140 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fd50c036000 tx=0x7fd50c03c040 comp rx=0 tx=0).stop 2024-03-23T03:51:56.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.977+0000 7fd522e00640 1 -- 172.21.15.102:0/1491276929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd51c0b5a30 msgr2=0x7fd51c0d1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:56.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.977+0000 7fd522e00640 1 --2- 172.21.15.102:0/1491276929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd51c0b5a30 0x7fd51c0d1cc0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fd50400c450 tx=0x7fd50400c910 comp rx=0 tx=0).stop 2024-03-23T03:51:56.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.978+0000 7fd522e00640 1 -- 172.21.15.102:0/1491276929 shutdown_connections 2024-03-23T03:51:56.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.978+0000 7fd522e00640 1 --2- 172.21.15.102:0/1491276929 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd4f806ac80 0x7fd4f806d140 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.978+0000 7fd522e00640 1 --2- 172.21.15.102:0/1491276929 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd51c15cda0 0x7fd51c0d2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.978+0000 7fd522e00640 1 --2- 172.21.15.102:0/1491276929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd51c0b5a30 0x7fd51c0d1cc0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.978+0000 7fd522e00640 1 --2- 172.21.15.102:0/1491276929 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd51c0b4a60 0x7fd51c1f1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:56.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.978+0000 7fd522e00640 1 -- 172.21.15.102:0/1491276929 >> 172.21.15.102:0/1491276929 conn(0x7fd51c0cf350 msgr2=0x7fd51c0b2fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:56.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.978+0000 7fd522e00640 1 -- 172.21.15.102:0/1491276929 shutdown_connections 2024-03-23T03:51:56.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:56.978+0000 7fd522e00640 1 -- 172.21.15.102:0/1491276929 wait complete. 2024-03-23T03:51:57.999 INFO:tasks.ceph:Scrubbing osd.0 2024-03-23T03:51:57.999 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-23T03:51:58.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.222+0000 7f1479e00640 1 Processor -- start 2024-03-23T03:51:58.223 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.222+0000 7f1479e00640 1 -- start start 2024-03-23T03:51:58.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.223+0000 7f1479e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f14740ce930 0x7f14740add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.223+0000 7f1479e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14740cd3e0 0x7f14740ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.224+0000 7f1479e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f147406ce90 0x7f147406d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.224+0000 7f1479e00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f147406c760 con 0x7f147406ce90 2024-03-23T03:51:58.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.224+0000 7f1479e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14740b42d0 con 0x7f14740ce930 2024-03-23T03:51:58.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.224+0000 7f1479e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14740cf250 con 0x7f14740cd3e0 2024-03-23T03:51:58.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.224+0000 7f1473400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f147406ce90 0x7f147406d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.224+0000 7f1473400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f147406ce90 0x7f147406d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43566/0 (socket says 172.21.15.102:43566) 2024-03-23T03:51:58.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.224+0000 7f1473400640 1 -- 172.21.15.102:0/2521002265 learned_addr learned my addr 172.21.15.102:0/2521002265 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:58.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.224+0000 7f1472a00640 1 --2- 172.21.15.102:0/2521002265 >> v2:172.21.15.173:3300/0 conn(0x7f14740ce930 0x7f14740add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.224+0000 7f1472000640 1 --2- 172.21.15.102:0/2521002265 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14740cd3e0 0x7f14740ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.224+0000 7f1473400640 1 -- 172.21.15.102:0/2521002265 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14740cd3e0 msgr2=0x7f14740ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:58.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.224+0000 7f1473400640 1 --2- 172.21.15.102:0/2521002265 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14740cd3e0 0x7f14740ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.224+0000 7f1473400640 1 -- 172.21.15.102:0/2521002265 >> v2:172.21.15.173:3300/0 conn(0x7f14740ce930 msgr2=0x7f14740add30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:58.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.224+0000 7f1473400640 1 --2- 172.21.15.102:0/2521002265 >> v2:172.21.15.173:3300/0 conn(0x7f14740ce930 0x7f14740add30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.224+0000 7f1473400640 1 -- 172.21.15.102:0/2521002265 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14740cdba0 con 0x7f147406ce90 2024-03-23T03:51:58.226 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.224+0000 7f1472a00640 1 --2- 172.21.15.102:0/2521002265 >> v2:172.21.15.173:3300/0 conn(0x7f14740ce930 0x7f14740add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:58.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.226+0000 7f1473400640 1 --2- 172.21.15.102:0/2521002265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f147406ce90 0x7f147406d320 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f14740cf540 tx=0x7f146c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb24887a6a6983c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:58.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.226+0000 7f1471600640 1 -- 172.21.15.102:0/2521002265 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c02ca70 con 0x7f147406ce90 2024-03-23T03:51:58.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.226+0000 7f1471600640 1 -- 172.21.15.102:0/2521002265 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146c02cbd0 con 0x7f147406ce90 2024-03-23T03:51:58.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.226+0000 7f1471600640 1 -- 172.21.15.102:0/2521002265 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c0343f0 con 0x7f147406ce90 2024-03-23T03:51:58.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.226+0000 7f1479e00640 1 -- 172.21.15.102:0/2521002265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f147406ce90 msgr2=0x7f147406d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.227 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.226+0000 7f1479e00640 1 --2- 172.21.15.102:0/2521002265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f147406ce90 0x7f147406d320 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f14740cf540 tx=0x7f146c02bed0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.226+0000 7f1479e00640 1 -- 172.21.15.102:0/2521002265 shutdown_connections 2024-03-23T03:51:58.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.226+0000 7f1479e00640 1 --2- 172.21.15.102:0/2521002265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f147406ce90 0x7f147406d320 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.226+0000 7f1479e00640 1 --2- 172.21.15.102:0/2521002265 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14740cd3e0 0x7f14740ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.226+0000 7f1479e00640 1 --2- 172.21.15.102:0/2521002265 >> v2:172.21.15.173:3300/0 conn(0x7f14740ce930 0x7f14740add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.226+0000 7f1479e00640 1 -- 172.21.15.102:0/2521002265 >> 172.21.15.102:0/2521002265 conn(0x7f147406a0b0 msgr2=0x7f147406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:58.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.227+0000 7f1479e00640 1 -- 172.21.15.102:0/2521002265 shutdown_connections 2024-03-23T03:51:58.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.227+0000 7f1479e00640 1 -- 172.21.15.102:0/2521002265 wait complete. 2024-03-23T03:51:58.228 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.228+0000 7f1479e00640 1 Processor -- start 2024-03-23T03:51:58.229 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.228+0000 7f1479e00640 1 -- start start 2024-03-23T03:51:58.230 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.229+0000 7f1479e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f147406ce90 0x7f1474153180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.229+0000 7f1479e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14740cd3e0 0x7f14740cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.229+0000 7f1479e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14740ce930 0x7f14740cc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.229+0000 7f1479e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14741536c0 con 0x7f147406ce90 2024-03-23T03:51:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.229+0000 7f1479e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14741556b0 con 0x7f14740cd3e0 2024-03-23T03:51:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.229+0000 7f1479e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1474158130 con 0x7f14740ce930 2024-03-23T03:51:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.230+0000 7f1472a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f147406ce90 0x7f1474153180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.230+0000 7f1472a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f147406ce90 0x7f1474153180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43568/0 (socket says 172.21.15.102:43568) 2024-03-23T03:51:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.230+0000 7f1472a00640 1 -- 172.21.15.102:0/356700488 learned_addr learned my addr 172.21.15.102:0/356700488 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.230+0000 7f1473400640 1 --2- 172.21.15.102:0/356700488 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14740ce930 0x7f14740cc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.230+0000 7f1472000640 1 --2- 172.21.15.102:0/356700488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14740cd3e0 0x7f14740cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.230+0000 7f1472a00640 1 -- 172.21.15.102:0/356700488 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14740ce930 msgr2=0x7f14740cc4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.230+0000 7f1472a00640 1 --2- 172.21.15.102:0/356700488 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f14740ce930 0x7f14740cc4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.230+0000 7f1472a00640 1 -- 172.21.15.102:0/356700488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14740cd3e0 msgr2=0x7f14740cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.231 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.230+0000 7f1472a00640 1 --2- 172.21.15.102:0/356700488 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14740cd3e0 0x7f14740cbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.230+0000 7f1472a00640 1 -- 172.21.15.102:0/356700488 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c009b30 con 0x7f147406ce90 2024-03-23T03:51:58.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.231+0000 7f1472a00640 1 --2- 172.21.15.102:0/356700488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f147406ce90 0x7f1474153180 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f146400c7f0 tx=0x7f146400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:58.232 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.231+0000 7f146b400640 1 -- 172.21.15.102:0/356700488 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1464007a30 con 0x7f147406ce90 2024-03-23T03:51:58.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.231+0000 7f146b400640 1 -- 172.21.15.102:0/356700488 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1464007b90 con 0x7f147406ce90 2024-03-23T03:51:58.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.231+0000 7f146b400640 1 -- 172.21.15.102:0/356700488 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146400ce40 con 0x7f147406ce90 2024-03-23T03:51:58.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.231+0000 7f1479e00640 1 -- 172.21.15.102:0/356700488 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1474158410 con 0x7f147406ce90 2024-03-23T03:51:58.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.231+0000 7f1479e00640 1 -- 172.21.15.102:0/356700488 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1474155990 con 0x7f147406ce90 2024-03-23T03:51:58.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.232+0000 7f1479e00640 1 -- 172.21.15.102:0/356700488 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f147406c970 con 0x7f147406ce90 2024-03-23T03:51:58.233 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.233+0000 7f146b400640 1 -- 172.21.15.102:0/356700488 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f14640113f0 con 0x7f147406ce90 2024-03-23T03:51:58.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.234+0000 7f146b400640 1 --2- 172.21.15.102:0/356700488 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f145406ac30 0x7f145406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.234+0000 7f146b400640 1 -- 172.21.15.102:0/356700488 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f1464089840 con 0x7f147406ce90 2024-03-23T03:51:58.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.234+0000 7f146b400640 1 --2- 172.21.15.102:0/356700488 >> [v2:172.21.15.102:6816/37351,v1:172.21.15.102:6817/37351] conn(0x7f1454070f30 0x7f1454073370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.234+0000 7f146b400640 1 -- 172.21.15.102:0/356700488 --> [v2:172.21.15.102:6816/37351,v1:172.21.15.102:6817/37351] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1454073a40 con 0x7f1454070f30 2024-03-23T03:51:58.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.234+0000 7f146b400640 1 -- 172.21.15.102:0/356700488 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1464089bb0 con 0x7f147406ce90 2024-03-23T03:51:58.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.234+0000 7f1472000640 1 --2- 172.21.15.102:0/356700488 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f145406ac30 0x7f145406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.235 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.235+0000 7f1473400640 1 --2- 172.21.15.102:0/356700488 >> [v2:172.21.15.102:6816/37351,v1:172.21.15.102:6817/37351] conn(0x7f1454070f30 0x7f1454073370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.235+0000 7f1472000640 1 --2- 172.21.15.102:0/356700488 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f145406ac30 0x7f145406d0f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1460006fd0 tx=0x7f1460008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:58.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.235+0000 7f1473400640 1 --2- 172.21.15.102:0/356700488 >> [v2:172.21.15.102:6816/37351,v1:172.21.15.102:6817/37351] conn(0x7f1454070f30 0x7f1454073370 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:58.236 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.236+0000 7f146b400640 1 -- 172.21.15.102:0/356700488 <== osd.0 v2:172.21.15.102:6816/37351 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f1454073a40 con 0x7f1454070f30 2024-03-23T03:51:58.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.250+0000 7f1479e00640 1 -- 172.21.15.102:0/356700488 --> [v2:172.21.15.102:6816/37351,v1:172.21.15.102:6817/37351] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f14740ce160 con 0x7f1454070f30 2024-03-23T03:51:58.250 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.250+0000 7f146b400640 1 -- 172.21.15.102:0/356700488 <== osd.0 v2:172.21.15.102:6816/37351 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f146c009510 con 0x7f1454070f30 2024-03-23T03:51:58.251 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:51:58.251 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T03:51:58.251 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:51:58.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.250+0000 7f1479e00640 1 -- 172.21.15.102:0/356700488 >> [v2:172.21.15.102:6816/37351,v1:172.21.15.102:6817/37351] conn(0x7f1454070f30 msgr2=0x7f1454073370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.250+0000 7f1479e00640 1 --2- 172.21.15.102:0/356700488 >> [v2:172.21.15.102:6816/37351,v1:172.21.15.102:6817/37351] conn(0x7f1454070f30 0x7f1454073370 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.250+0000 7f1479e00640 1 -- 172.21.15.102:0/356700488 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f145406ac30 msgr2=0x7f145406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.250+0000 7f1479e00640 1 --2- 172.21.15.102:0/356700488 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f145406ac30 0x7f145406d0f0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f1460006fd0 tx=0x7f1460008040 comp rx=0 tx=0).stop 2024-03-23T03:51:58.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.250+0000 7f1479e00640 1 -- 172.21.15.102:0/356700488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f147406ce90 msgr2=0x7f1474153180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.250+0000 7f1479e00640 1 --2- 172.21.15.102:0/356700488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f147406ce90 0x7f1474153180 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f146400c7f0 tx=0x7f146400cbf0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.250+0000 7f1473400640 1 -- 172.21.15.102:0/356700488 reap_dead start 2024-03-23T03:51:58.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.251+0000 7f1479e00640 1 -- 172.21.15.102:0/356700488 shutdown_connections 2024-03-23T03:51:58.252 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.251+0000 7f1479e00640 1 -- 172.21.15.102:0/356700488 >> 172.21.15.102:0/356700488 conn(0x7f147406a0b0 msgr2=0x7f147415b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:58.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.251+0000 7f1479e00640 1 -- 172.21.15.102:0/356700488 shutdown_connections 2024-03-23T03:51:58.253 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.251+0000 7f1479e00640 1 -- 172.21.15.102:0/356700488 wait complete. 2024-03-23T03:51:58.268 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-23T03:51:58.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.471+0000 7fd864a00640 1 Processor -- start 2024-03-23T03:51:58.472 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.471+0000 7fd864a00640 1 -- start start 2024-03-23T03:51:58.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.473+0000 7fd864a00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fd8600cc330 0x7fd8600b18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.473+0000 7fd864a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd8600b1e10 0x7fd8600cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.473+0000 7fd864a00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd86015e010 con 0x7fd860159690 2024-03-23T03:51:58.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.473+0000 7fd864a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8600caec0 con 0x7fd8600b1e10 2024-03-23T03:51:58.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.473+0000 7fd864a00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8600cc7f0 con 0x7fd8600cc330 2024-03-23T03:51:58.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.473+0000 7fd85e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd8600b1e10 0x7fd8600cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.474+0000 7fd85e800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7fd8600b1e10 0x7fd8600cd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43574/0 (socket says 172.21.15.102:43574) 2024-03-23T03:51:58.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.474+0000 7fd85e800640 1 -- 172.21.15.102:0/2822254482 learned_addr learned my addr 172.21.15.102:0/2822254482 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:58.474 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.474+0000 7fd85d400640 1 --2- 172.21.15.102:0/2822254482 >> v2:172.21.15.173:3300/0 conn(0x7fd8600cc330 0x7fd8600b18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.474+0000 7fd85e800640 1 -- 172.21.15.102:0/2822254482 >> v2:172.21.15.173:3300/0 conn(0x7fd8600cc330 msgr2=0x7fd8600b18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:58.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.474+0000 7fd85e800640 1 --2- 172.21.15.102:0/2822254482 >> v2:172.21.15.173:3300/0 conn(0x7fd8600cc330 0x7fd8600b18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.474+0000 7fd85e800640 1 -- 172.21.15.102:0/2822254482 >> v1:172.21.15.173:6789/0 conn(0x7fd860159690 legacy=0x7fd8600ca7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.475 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.474+0000 7fd85e800640 1 -- 172.21.15.102:0/2822254482 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8600ccad0 con 0x7fd8600b1e10 2024-03-23T03:51:58.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.475+0000 7fd85e800640 1 --2- 172.21.15.102:0/2822254482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8600b1e10 0x7fd8600cd890 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fd8600cb520 tx=0x7fd85802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632875175d9e4b9a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:58.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.476+0000 7fd85ca00640 1 -- 172.21.15.102:0/2822254482 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85802cb20 con 0x7fd8600b1e10 2024-03-23T03:51:58.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.476+0000 7fd85ca00640 1 -- 172.21.15.102:0/2822254482 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd85802cc80 con 0x7fd8600b1e10 2024-03-23T03:51:58.476 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.476+0000 7fd85ca00640 1 -- 172.21.15.102:0/2822254482 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8580343f0 con 0x7fd8600b1e10 2024-03-23T03:51:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.476+0000 7fd864a00640 1 -- 172.21.15.102:0/2822254482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8600b1e10 msgr2=0x7fd8600cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.476+0000 7fd864a00640 1 --2- 172.21.15.102:0/2822254482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8600b1e10 0x7fd8600cd890 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fd8600cb520 tx=0x7fd85802bf80 comp rx=0 tx=0).stop 2024-03-23T03:51:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.476+0000 7fd864a00640 1 -- 172.21.15.102:0/2822254482 shutdown_connections 2024-03-23T03:51:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.476+0000 7fd864a00640 1 --2- 172.21.15.102:0/2822254482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8600b1e10 0x7fd8600cd890 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.476+0000 7fd864a00640 1 --2- 172.21.15.102:0/2822254482 >> v2:172.21.15.173:3300/0 conn(0x7fd8600cc330 0x7fd8600b18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.477 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.476+0000 7fd864a00640 1 -- 172.21.15.102:0/2822254482 >> 172.21.15.102:0/2822254482 conn(0x7fd86006a0b0 msgr2=0x7fd86006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:58.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.477+0000 7fd864a00640 1 -- 172.21.15.102:0/2822254482 shutdown_connections 2024-03-23T03:51:58.478 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.477+0000 7fd864a00640 1 -- 172.21.15.102:0/2822254482 wait complete. 2024-03-23T03:51:58.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.478+0000 7fd864a00640 1 Processor -- start 2024-03-23T03:51:58.479 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.478+0000 7fd864a00640 1 -- start start 2024-03-23T03:51:58.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.479+0000 7fd864a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8600b1e10 0x7fd86014b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.479+0000 7fd864a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd8600cc330 0x7fd8600af580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.479+0000 7fd864a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd860159690 0x7fd8600afac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.481 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.479+0000 7fd864a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd860155f70 con 0x7fd8600b1e10 2024-03-23T03:51:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.479+0000 7fd864a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd860156620 con 0x7fd860159690 2024-03-23T03:51:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.479+0000 7fd864a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8600af0a0 con 0x7fd8600cc330 2024-03-23T03:51:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.480+0000 7fd85de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8600b1e10 0x7fd86014b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.480+0000 7fd85de00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8600b1e10 0x7fd86014b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43584/0 (socket says 172.21.15.102:43584) 2024-03-23T03:51:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.480+0000 7fd85de00640 1 -- 172.21.15.102:0/1833750836 learned_addr learned my addr 172.21.15.102:0/1833750836 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.480+0000 7fd85de00640 1 -- 172.21.15.102:0/1833750836 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd8600cc330 msgr2=0x7fd8600af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.480+0000 7fd85de00640 1 --2- 172.21.15.102:0/1833750836 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd8600cc330 0x7fd8600af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.480+0000 7fd85de00640 1 -- 172.21.15.102:0/1833750836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd860159690 msgr2=0x7fd8600afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.480+0000 7fd85de00640 1 --2- 172.21.15.102:0/1833750836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd860159690 0x7fd8600afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.480+0000 7fd85de00640 1 -- 172.21.15.102:0/1833750836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd858009b30 con 0x7fd8600b1e10 2024-03-23T03:51:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.480+0000 7fd85de00640 1 --2- 172.21.15.102:0/1833750836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8600b1e10 0x7fd86014b2b0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fd85000c400 tx=0x7fd85000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.481+0000 7fd856000640 1 -- 172.21.15.102:0/1833750836 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8500052d0 con 0x7fd8600b1e10 2024-03-23T03:51:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.481+0000 7fd864a00640 1 -- 172.21.15.102:0/1833750836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8601568c0 con 0x7fd8600b1e10 2024-03-23T03:51:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.481+0000 7fd856000640 1 -- 172.21.15.102:0/1833750836 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd850005430 con 0x7fd8600b1e10 2024-03-23T03:51:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.481+0000 7fd856000640 1 -- 172.21.15.102:0/1833750836 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd850003830 con 0x7fd8600b1e10 2024-03-23T03:51:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.481+0000 7fd864a00640 1 -- 172.21.15.102:0/1833750836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd860156ba0 con 0x7fd8600b1e10 2024-03-23T03:51:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.482+0000 7fd864a00640 1 -- 172.21.15.102:0/1833750836 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8600af260 con 0x7fd8600b1e10 2024-03-23T03:51:58.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.483+0000 7fd856000640 1 -- 172.21.15.102:0/1833750836 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fd850003990 con 0x7fd8600b1e10 2024-03-23T03:51:58.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.484+0000 7fd856000640 1 --2- 172.21.15.102:0/1833750836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd84006ac00 0x7fd84006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.484+0000 7fd856000640 1 -- 172.21.15.102:0/1833750836 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fd8500887a0 con 0x7fd8600b1e10 2024-03-23T03:51:58.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.484+0000 7fd85d400640 1 --2- 172.21.15.102:0/1833750836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd84006ac00 0x7fd84006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.485 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.485+0000 7fd85d400640 1 --2- 172.21.15.102:0/1833750836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd84006ac00 0x7fd84006d0c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd84c006a90 tx=0x7fd84c006a20 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:58.489 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.488+0000 7fd856000640 1 -- 172.21.15.102:0/1833750836 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fd850053ac0 con 0x7fd8600b1e10 2024-03-23T03:51:58.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.681+0000 7fd864a00640 1 -- 172.21.15.102:0/1833750836 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8600cd370 con 0x7fd84006ac00 2024-03-23T03:51:58.682 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.682+0000 7fd856000640 1 -- 172.21.15.102:0/1833750836 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd84c00e860 con 0x7fd84006ac00 2024-03-23T03:51:58.683 INFO:teuthology.orchestra.run.smithi102.stderr:instructed osd(s) 0 to deep-scrub 2024-03-23T03:51:58.685 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.684+0000 7fd864a00640 1 -- 172.21.15.102:0/1833750836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd84006ac00 msgr2=0x7fd84006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.684+0000 7fd864a00640 1 --2- 172.21.15.102:0/1833750836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd84006ac00 0x7fd84006d0c0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fd84c006a90 tx=0x7fd84c006a20 comp rx=0 tx=0).stop 2024-03-23T03:51:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.684+0000 7fd864a00640 1 -- 172.21.15.102:0/1833750836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8600b1e10 msgr2=0x7fd86014b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.684+0000 7fd864a00640 1 --2- 172.21.15.102:0/1833750836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8600b1e10 0x7fd86014b2b0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fd85000c400 tx=0x7fd85000c8c0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.685+0000 7fd864a00640 1 -- 172.21.15.102:0/1833750836 shutdown_connections 2024-03-23T03:51:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.685+0000 7fd864a00640 1 --2- 172.21.15.102:0/1833750836 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fd84006ac00 0x7fd84006d0c0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.685+0000 7fd864a00640 1 --2- 172.21.15.102:0/1833750836 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd860159690 0x7fd8600afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.685+0000 7fd864a00640 1 --2- 172.21.15.102:0/1833750836 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fd8600cc330 0x7fd8600af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.685+0000 7fd864a00640 1 --2- 172.21.15.102:0/1833750836 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8600b1e10 0x7fd86014b2b0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.685+0000 7fd864a00640 1 -- 172.21.15.102:0/1833750836 >> 172.21.15.102:0/1833750836 conn(0x7fd86006a0b0 msgr2=0x7fd86006a490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.685+0000 7fd864a00640 1 -- 172.21.15.102:0/1833750836 shutdown_connections 2024-03-23T03:51:58.686 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.685+0000 7fd864a00640 1 -- 172.21.15.102:0/1833750836 wait complete. 2024-03-23T03:51:58.703 INFO:tasks.ceph:Scrubbing osd.1 2024-03-23T03:51:58.703 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-23T03:51:58.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.916+0000 7f6e7f200640 1 Processor -- start 2024-03-23T03:51:58.917 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.917+0000 7f6e7f200640 1 -- start start 2024-03-23T03:51:58.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.918+0000 7f6e7f200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e800cf7c0 0x7f6e800cd830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.918+0000 7f6e7f200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e800ccd90 con 0x7f6e8015c820 2024-03-23T03:51:58.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.918+0000 7f6e7f200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e800b7fc0 con 0x7f6e80160e90 2024-03-23T03:51:58.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.918+0000 7f6e7f200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e800cdd70 con 0x7f6e800cf7c0 2024-03-23T03:51:58.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.918+0000 7f6e7de00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f6e8015c820 0x7f6e800cf0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54244/0 (socket says 172.21.15.102:54244) 2024-03-23T03:51:58.919 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.918+0000 7f6e7de00640 1 -- 172.21.15.102:0/161681824 learned_addr learned my addr 172.21.15.102:0/161681824 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:58.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.919+0000 7f6e7ca00640 1 -- 172.21.15.102:0/161681824 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923439274 0 0) 0x7f6e800ccd90 con 0x7f6e8015c820 2024-03-23T03:51:58.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.919+0000 7f6e7ca00640 1 -- 172.21.15.102:0/161681824 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e64003520 con 0x7f6e8015c820 2024-03-23T03:51:58.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.919+0000 7f6e7d400640 1 --2- 172.21.15.102:0/161681824 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e800cf7c0 0x7f6e800cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.919+0000 7f6e7ca00640 1 -- 172.21.15.102:0/161681824 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1357171615 0 0) 0x7f6e64003520 con 0x7f6e8015c820 2024-03-23T03:51:58.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.919+0000 7f6e7ca00640 1 -- 172.21.15.102:0/161681824 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e800cf7c0 msgr2=0x7f6e800cd830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:58.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.919+0000 7f6e7ca00640 1 --2- 172.21.15.102:0/161681824 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e800cf7c0 0x7f6e800cd830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.919+0000 7f6e7ca00640 1 -- 172.21.15.102:0/161681824 >> v1:172.21.15.173:6789/0 conn(0x7f6e80160e90 legacy=0x7f6e800d0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:58.921 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.919+0000 7f6e7ca00640 1 -- 172.21.15.102:0/161681824 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e800b82a0 con 0x7f6e8015c820 2024-03-23T03:51:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.919+0000 7f6e7ca00640 1 -- 172.21.15.102:0/161681824 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f6e70001670 con 0x7f6e8015c820 2024-03-23T03:51:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.919+0000 7f6e7ca00640 1 -- 172.21.15.102:0/161681824 >> v1:172.21.15.102:6789/0 conn(0x7f6e8015c820 legacy=0x7f6e800cf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.919+0000 7f6e7ca00640 1 --2- 172.21.15.102:0/161681824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e640039a0 0x7f6e64003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.919+0000 7f6e7ca00640 1 --2- 172.21.15.102:0/161681824 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e800cf7c0 0x7f6e640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.919+0000 7f6e7ca00640 1 --2- 172.21.15.102:0/161681824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e64004d70 0x7f6e640051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.919+0000 7f6e7ca00640 1 -- 172.21.15.102:0/161681824 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e800cdd70 con 0x7f6e64004d70 2024-03-23T03:51:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.919+0000 7f6e7ca00640 1 -- 172.21.15.102:0/161681824 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6400bc70 con 0x7f6e640039a0 2024-03-23T03:51:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.919+0000 7f6e7ca00640 1 -- 172.21.15.102:0/161681824 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6400be30 con 0x7f6e800cf7c0 2024-03-23T03:51:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.920+0000 7f6e7e800640 1 --2- 172.21.15.102:0/161681824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e64004d70 0x7f6e640051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.920+0000 7f6e7d400640 1 --2- 172.21.15.102:0/161681824 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e800cf7c0 0x7f6e640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.920+0000 7f6e7de00640 1 --2- 172.21.15.102:0/161681824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e640039a0 0x7f6e64003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.920+0000 7f6e7e800640 1 -- 172.21.15.102:0/161681824 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e800cf7c0 msgr2=0x7f6e640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.920+0000 7f6e7e800640 1 --2- 172.21.15.102:0/161681824 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e800cf7c0 0x7f6e640047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.922 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.920+0000 7f6e7e800640 1 -- 172.21.15.102:0/161681824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e640039a0 msgr2=0x7f6e64003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:58.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.920+0000 7f6e7e800640 1 --2- 172.21.15.102:0/161681824 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e640039a0 0x7f6e64003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.923 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.920+0000 7f6e7e800640 1 -- 172.21.15.102:0/161681824 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e64003520 con 0x7f6e64004d70 2024-03-23T03:51:58.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.921+0000 7f6e7e800640 1 --2- 172.21.15.102:0/161681824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e64004d70 0x7f6e640051f0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f6e780044b0 tx=0x7f6e640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e12d33dd38c379c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:58.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.922+0000 7f6e7ca00640 1 -- 172.21.15.102:0/161681824 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78031610 con 0x7f6e64004d70 2024-03-23T03:51:58.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.922+0000 7f6e7ca00640 1 -- 172.21.15.102:0/161681824 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e7802d040 con 0x7f6e64004d70 2024-03-23T03:51:58.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.922+0000 7f6e7ca00640 1 -- 172.21.15.102:0/161681824 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e780319b0 con 0x7f6e64004d70 2024-03-23T03:51:58.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.922+0000 7f6e7f200640 1 -- 172.21.15.102:0/161681824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e64004d70 msgr2=0x7f6e640051f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.922+0000 7f6e7f200640 1 --2- 172.21.15.102:0/161681824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e64004d70 0x7f6e640051f0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f6e780044b0 tx=0x7f6e640057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.922+0000 7f6e7e800640 1 -- 172.21.15.102:0/161681824 reap_dead start 2024-03-23T03:51:58.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.922+0000 7f6e7f200640 1 -- 172.21.15.102:0/161681824 shutdown_connections 2024-03-23T03:51:58.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.922+0000 7f6e7f200640 1 -- 172.21.15.102:0/161681824 >> 172.21.15.102:0/161681824 conn(0x7f6e8006b050 msgr2=0x7f6e8006b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:58.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.922+0000 7f6e7f200640 1 -- 172.21.15.102:0/161681824 shutdown_connections 2024-03-23T03:51:58.924 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.922+0000 7f6e7f200640 1 -- 172.21.15.102:0/161681824 wait complete. 2024-03-23T03:51:58.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.924+0000 7f6e7f200640 1 Processor -- start 2024-03-23T03:51:58.925 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.924+0000 7f6e7f200640 1 -- start start 2024-03-23T03:51:58.926 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.925+0000 7f6e7f200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e64004d70 0x7f6e801f1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.925+0000 7f6e7f200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e800b7010 0x7f6e800b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.925+0000 7f6e7f200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e800b36f0 0x7f6e800b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.925+0000 7f6e7f200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e800b40b0 con 0x7f6e800b7010 2024-03-23T03:51:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.925+0000 7f6e7f200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8015d1c0 con 0x7f6e800b36f0 2024-03-23T03:51:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.925+0000 7f6e7f200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e800b33d0 con 0x7f6e64004d70 2024-03-23T03:51:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.925+0000 7f6e7d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e800b7010 0x7f6e800b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.926+0000 7f6e7d400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e800b7010 0x7f6e800b2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43602/0 (socket says 172.21.15.102:43602) 2024-03-23T03:51:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.926+0000 7f6e7d400640 1 -- 172.21.15.102:0/1919273722 learned_addr learned my addr 172.21.15.102:0/1919273722 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.926+0000 7f6e7e800640 1 --2- 172.21.15.102:0/1919273722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e800b36f0 0x7f6e800b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.926+0000 7f6e7de00640 1 --2- 172.21.15.102:0/1919273722 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e64004d70 0x7f6e801f1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.926+0000 7f6e7d400640 1 -- 172.21.15.102:0/1919273722 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e64004d70 msgr2=0x7f6e801f1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.926+0000 7f6e7d400640 1 --2- 172.21.15.102:0/1919273722 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f6e64004d70 0x7f6e801f1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.927 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.926+0000 7f6e7d400640 1 -- 172.21.15.102:0/1919273722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e800b36f0 msgr2=0x7f6e800b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.926+0000 7f6e7d400640 1 --2- 172.21.15.102:0/1919273722 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6e800b36f0 0x7f6e800b3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.926+0000 7f6e7d400640 1 -- 172.21.15.102:0/1919273722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8015da40 con 0x7f6e800b7010 2024-03-23T03:51:58.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.926+0000 7f6e7d400640 1 --2- 172.21.15.102:0/1919273722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e800b7010 0x7f6e800b2a10 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800e840 tx=0x7f6e6800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:58.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.927+0000 7f6e76000640 1 -- 172.21.15.102:0/1919273722 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6800ce20 con 0x7f6e800b7010 2024-03-23T03:51:58.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.927+0000 7f6e76000640 1 -- 172.21.15.102:0/1919273722 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6800a920 con 0x7f6e800b7010 2024-03-23T03:51:58.928 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.927+0000 7f6e76000640 1 -- 172.21.15.102:0/1919273722 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6800acd0 con 0x7f6e800b7010 2024-03-23T03:51:58.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.927+0000 7f6e7f200640 1 -- 172.21.15.102:0/1919273722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e800b7490 con 0x7f6e800b7010 2024-03-23T03:51:58.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.927+0000 7f6e7f200640 1 -- 172.21.15.102:0/1919273722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e8015d3f0 con 0x7f6e800b7010 2024-03-23T03:51:58.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.928+0000 7f6e7f200640 1 -- 172.21.15.102:0/1919273722 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6e800b8580 con 0x7f6e800b7010 2024-03-23T03:51:58.929 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.929+0000 7f6e76000640 1 -- 172.21.15.102:0/1919273722 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6e68015480 con 0x7f6e800b7010 2024-03-23T03:51:58.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.930+0000 7f6e76000640 1 --2- 172.21.15.102:0/1919273722 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6e5c06ac00 0x7f6e5c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.930+0000 7f6e76000640 1 -- 172.21.15.102:0/1919273722 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f6e68089a00 con 0x7f6e800b7010 2024-03-23T03:51:58.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.930+0000 7f6e7de00640 1 --2- 172.21.15.102:0/1919273722 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6e5c06ac00 0x7f6e5c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.930+0000 7f6e76000640 1 --2- 172.21.15.102:0/1919273722 >> [v2:172.21.15.102:6824/37352,v1:172.21.15.102:6825/37352] conn(0x7f6e5c070f00 0x7f6e5c073340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:58.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.930+0000 7f6e76000640 1 -- 172.21.15.102:0/1919273722 --> [v2:172.21.15.102:6824/37352,v1:172.21.15.102:6825/37352] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6e5c073a10 con 0x7f6e5c070f00 2024-03-23T03:51:58.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.930+0000 7f6e76000640 1 -- 172.21.15.102:0/1919273722 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6e6808e2c0 con 0x7f6e800b7010 2024-03-23T03:51:58.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.930+0000 7f6e7e800640 1 --2- 172.21.15.102:0/1919273722 >> [v2:172.21.15.102:6824/37352,v1:172.21.15.102:6825/37352] conn(0x7f6e5c070f00 0x7f6e5c073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:58.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.930+0000 7f6e7de00640 1 --2- 172.21.15.102:0/1919273722 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6e5c06ac00 0x7f6e5c06d0c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6e70001eb0 tx=0x7f6e70001d80 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:58.931 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.931+0000 7f6e7e800640 1 --2- 172.21.15.102:0/1919273722 >> [v2:172.21.15.102:6824/37352,v1:172.21.15.102:6825/37352] conn(0x7f6e5c070f00 0x7f6e5c073340 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:58.932 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.932+0000 7f6e76000640 1 -- 172.21.15.102:0/1919273722 <== osd.1 v2:172.21.15.102:6824/37352 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6e5c073a10 con 0x7f6e5c070f00 2024-03-23T03:51:58.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.946+0000 7f6e7f200640 1 -- 172.21.15.102:0/1919273722 --> [v2:172.21.15.102:6824/37352,v1:172.21.15.102:6825/37352] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6e800cd310 con 0x7f6e5c070f00 2024-03-23T03:51:58.946 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.946+0000 7f6e76000640 1 -- 172.21.15.102:0/1919273722 <== osd.1 v2:172.21.15.102:6824/37352 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6e7803b430 con 0x7f6e5c070f00 2024-03-23T03:51:58.947 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:51:58.947 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T03:51:58.947 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:51:58.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.946+0000 7f6e7f200640 1 -- 172.21.15.102:0/1919273722 >> [v2:172.21.15.102:6824/37352,v1:172.21.15.102:6825/37352] conn(0x7f6e5c070f00 msgr2=0x7f6e5c073340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.946+0000 7f6e7f200640 1 --2- 172.21.15.102:0/1919273722 >> [v2:172.21.15.102:6824/37352,v1:172.21.15.102:6825/37352] conn(0x7f6e5c070f00 0x7f6e5c073340 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:58.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.947+0000 7f6e7f200640 1 -- 172.21.15.102:0/1919273722 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6e5c06ac00 msgr2=0x7f6e5c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.947+0000 7f6e7f200640 1 --2- 172.21.15.102:0/1919273722 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f6e5c06ac00 0x7f6e5c06d0c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6e70001eb0 tx=0x7f6e70001d80 comp rx=0 tx=0).stop 2024-03-23T03:51:58.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.947+0000 7f6e7f200640 1 -- 172.21.15.102:0/1919273722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e800b7010 msgr2=0x7f6e800b2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:58.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.947+0000 7f6e7f200640 1 --2- 172.21.15.102:0/1919273722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e800b7010 0x7f6e800b2a10 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800e840 tx=0x7f6e6800ed00 comp rx=0 tx=0).stop 2024-03-23T03:51:58.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.947+0000 7f6e7e800640 1 -- 172.21.15.102:0/1919273722 reap_dead start 2024-03-23T03:51:58.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.947+0000 7f6e7f200640 1 -- 172.21.15.102:0/1919273722 shutdown_connections 2024-03-23T03:51:58.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.947+0000 7f6e7f200640 1 -- 172.21.15.102:0/1919273722 >> 172.21.15.102:0/1919273722 conn(0x7f6e8006b050 msgr2=0x7f6e800d0560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:58.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.947+0000 7f6e7f200640 1 -- 172.21.15.102:0/1919273722 shutdown_connections 2024-03-23T03:51:58.948 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:58.947+0000 7f6e7f200640 1 -- 172.21.15.102:0/1919273722 wait complete. 2024-03-23T03:51:58.965 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-23T03:51:59.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.172+0000 7ffa4fe00640 1 Processor -- start 2024-03-23T03:51:59.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.172+0000 7ffa4fe00640 1 -- start start 2024-03-23T03:51:59.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.174+0000 7ffa4fe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa50140cf0 0x7ffa500b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.174+0000 7ffa4fe00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7ffa500cb480 0x7ffa50141410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.174+0000 7ffa4fe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa501410d0 con 0x7ffa500b23e0 2024-03-23T03:51:59.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.174+0000 7ffa4fe00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa500cd1b0 con 0x7ffa500cb480 2024-03-23T03:51:59.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.174+0000 7ffa4fe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa500cd490 con 0x7ffa50140cf0 2024-03-23T03:51:59.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.174+0000 7ffa46000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ffa500b23e0 0x7ffa500cad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54258/0 (socket says 172.21.15.102:54258) 2024-03-23T03:51:59.175 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.174+0000 7ffa46000640 1 -- 172.21.15.102:0/3637481097 learned_addr learned my addr 172.21.15.102:0/3637481097 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:59.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.174+0000 7ffa4e000640 1 -- 172.21.15.102:0/3637481097 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581328843 0 0) 0x7ffa501410d0 con 0x7ffa500b23e0 2024-03-23T03:51:59.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.174+0000 7ffa4ea00640 1 --2- 172.21.15.102:0/3637481097 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa50140cf0 0x7ffa500b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.174+0000 7ffa4f400640 1 --2- 172.21.15.102:0/3637481097 >> v2:172.21.15.173:3300/0 conn(0x7ffa500cb480 0x7ffa50141410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.174+0000 7ffa4e000640 1 -- 172.21.15.102:0/3637481097 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa34003520 con 0x7ffa500b23e0 2024-03-23T03:51:59.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.175+0000 7ffa4e000640 1 -- 172.21.15.102:0/3637481097 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3180012765 0 0) 0x7ffa34003520 con 0x7ffa500b23e0 2024-03-23T03:51:59.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.175+0000 7ffa4e000640 1 -- 172.21.15.102:0/3637481097 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa50140cf0 msgr2=0x7ffa500b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:59.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.175+0000 7ffa4e000640 1 --2- 172.21.15.102:0/3637481097 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa50140cf0 0x7ffa500b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.175+0000 7ffa4e000640 1 -- 172.21.15.102:0/3637481097 >> v2:172.21.15.173:3300/0 conn(0x7ffa500cb480 msgr2=0x7ffa50141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:59.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.175+0000 7ffa4e000640 1 --2- 172.21.15.102:0/3637481097 >> v2:172.21.15.173:3300/0 conn(0x7ffa500cb480 0x7ffa50141410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.175+0000 7ffa4e000640 1 -- 172.21.15.102:0/3637481097 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa500cd770 con 0x7ffa500b23e0 2024-03-23T03:51:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.175+0000 7ffa4e000640 1 -- 172.21.15.102:0/3637481097 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7ffa38001670 con 0x7ffa500b23e0 2024-03-23T03:51:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.175+0000 7ffa4f400640 1 --2- 172.21.15.102:0/3637481097 >> v2:172.21.15.173:3300/0 conn(0x7ffa500cb480 0x7ffa50141410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:51:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.175+0000 7ffa4e000640 1 -- 172.21.15.102:0/3637481097 >> v1:172.21.15.102:6789/0 conn(0x7ffa500b23e0 legacy=0x7ffa500cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.175+0000 7ffa4e000640 1 --2- 172.21.15.102:0/3637481097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa34003970 0x7ffa34003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.175+0000 7ffa4e000640 1 --2- 172.21.15.102:0/3637481097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa340047c0 0x7ffa34004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.175+0000 7ffa4e000640 1 --2- 172.21.15.102:0/3637481097 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa50140cf0 0x7ffa34005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.175+0000 7ffa4e000640 1 -- 172.21.15.102:0/3637481097 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa500cd1b0 con 0x7ffa340047c0 2024-03-23T03:51:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.175+0000 7ffa4e000640 1 -- 172.21.15.102:0/3637481097 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa500cd490 con 0x7ffa34003970 2024-03-23T03:51:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.175+0000 7ffa4e000640 1 -- 172.21.15.102:0/3637481097 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa3400ba80 con 0x7ffa50140cf0 2024-03-23T03:51:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.175+0000 7ffa46000640 1 --2- 172.21.15.102:0/3637481097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa340047c0 0x7ffa34004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.176+0000 7ffa46000640 1 -- 172.21.15.102:0/3637481097 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa50140cf0 msgr2=0x7ffa34005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.176+0000 7ffa4ea00640 1 --2- 172.21.15.102:0/3637481097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa34003970 0x7ffa34003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.176+0000 7ffa4f400640 1 --2- 172.21.15.102:0/3637481097 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa50140cf0 0x7ffa34005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.176+0000 7ffa46000640 1 --2- 172.21.15.102:0/3637481097 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa50140cf0 0x7ffa34005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.176+0000 7ffa46000640 1 -- 172.21.15.102:0/3637481097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa34003970 msgr2=0x7ffa34003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:59.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.176+0000 7ffa46000640 1 --2- 172.21.15.102:0/3637481097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa34003970 0x7ffa34003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.176+0000 7ffa46000640 1 -- 172.21.15.102:0/3637481097 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa34003520 con 0x7ffa340047c0 2024-03-23T03:51:59.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.177+0000 7ffa46000640 1 --2- 172.21.15.102:0/3637481097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa340047c0 0x7ffa34004c40 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7ffa3800c0e0 tx=0x7ffa340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f7a68babd24a90 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:59.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.177+0000 7ffa4e000640 1 -- 172.21.15.102:0/3637481097 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa38005ad0 con 0x7ffa340047c0 2024-03-23T03:51:59.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.177+0000 7ffa4e000640 1 -- 172.21.15.102:0/3637481097 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa38005c50 con 0x7ffa340047c0 2024-03-23T03:51:59.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.177+0000 7ffa4e000640 1 -- 172.21.15.102:0/3637481097 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3802c7e0 con 0x7ffa340047c0 2024-03-23T03:51:59.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.177+0000 7ffa4fe00640 1 -- 172.21.15.102:0/3637481097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa340047c0 msgr2=0x7ffa34004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.177+0000 7ffa4fe00640 1 --2- 172.21.15.102:0/3637481097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa340047c0 0x7ffa34004c40 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7ffa3800c0e0 tx=0x7ffa340057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.177+0000 7ffa4f400640 1 -- 172.21.15.102:0/3637481097 reap_dead start 2024-03-23T03:51:59.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.178+0000 7ffa4fe00640 1 -- 172.21.15.102:0/3637481097 shutdown_connections 2024-03-23T03:51:59.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.178+0000 7ffa4fe00640 1 -- 172.21.15.102:0/3637481097 >> 172.21.15.102:0/3637481097 conn(0x7ffa50065e20 msgr2=0x7ffa50069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:59.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.178+0000 7ffa4fe00640 1 -- 172.21.15.102:0/3637481097 shutdown_connections 2024-03-23T03:51:59.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.178+0000 7ffa4fe00640 1 -- 172.21.15.102:0/3637481097 wait complete. 2024-03-23T03:51:59.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.179+0000 7ffa4fe00640 1 Processor -- start 2024-03-23T03:51:59.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.180+0000 7ffa4fe00640 1 -- start start 2024-03-23T03:51:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.180+0000 7ffa4fe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa340047c0 0x7ffa501584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.180+0000 7ffa4fe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa500ad330 0x7ffa500ab920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.180+0000 7ffa4fe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa5006c040 0x7ffa5006c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.180+0000 7ffa4fe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa5006ca00 con 0x7ffa500ad330 2024-03-23T03:51:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.180+0000 7ffa4fe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa50158a20 con 0x7ffa5006c040 2024-03-23T03:51:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.181+0000 7ffa4fe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa500ac290 con 0x7ffa340047c0 2024-03-23T03:51:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.181+0000 7ffa46000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa500ad330 0x7ffa500ab920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.181+0000 7ffa46000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa500ad330 0x7ffa500ab920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43614/0 (socket says 172.21.15.102:43614) 2024-03-23T03:51:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.181+0000 7ffa46000640 1 -- 172.21.15.102:0/1424642671 learned_addr learned my addr 172.21.15.102:0/1424642671 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.181+0000 7ffa46000640 1 -- 172.21.15.102:0/1424642671 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa340047c0 msgr2=0x7ffa501584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.181+0000 7ffa46000640 1 --2- 172.21.15.102:0/1424642671 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa340047c0 0x7ffa501584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.181+0000 7ffa4f400640 1 --2- 172.21.15.102:0/1424642671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa5006c040 0x7ffa5006c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.181+0000 7ffa46000640 1 -- 172.21.15.102:0/1424642671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa5006c040 msgr2=0x7ffa5006c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.183 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.181+0000 7ffa46000640 1 --2- 172.21.15.102:0/1424642671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa5006c040 0x7ffa5006c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.181+0000 7ffa46000640 1 -- 172.21.15.102:0/1424642671 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa500ac570 con 0x7ffa500ad330 2024-03-23T03:51:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.182+0000 7ffa46000640 1 --2- 172.21.15.102:0/1424642671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa500ad330 0x7ffa500ab920 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7ffa3802c380 tx=0x7ffa38001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.182+0000 7ffa47400640 1 -- 172.21.15.102:0/1424642671 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa3803c040 con 0x7ffa500ad330 2024-03-23T03:51:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.182+0000 7ffa47400640 1 -- 172.21.15.102:0/1424642671 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa3802dc80 con 0x7ffa500ad330 2024-03-23T03:51:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.182+0000 7ffa4fe00640 1 -- 172.21.15.102:0/1424642671 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa500ac850 con 0x7ffa500ad330 2024-03-23T03:51:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.182+0000 7ffa47400640 1 -- 172.21.15.102:0/1424642671 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa38007070 con 0x7ffa500ad330 2024-03-23T03:51:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.182+0000 7ffa4fe00640 1 -- 172.21.15.102:0/1424642671 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa500ace10 con 0x7ffa500ad330 2024-03-23T03:51:59.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.183+0000 7ffa4fe00640 1 -- 172.21.15.102:0/1424642671 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa5004e030 con 0x7ffa500ad330 2024-03-23T03:51:59.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.184+0000 7ffa47400640 1 -- 172.21.15.102:0/1424642671 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ffa38032070 con 0x7ffa500ad330 2024-03-23T03:51:59.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.185+0000 7ffa47400640 1 --2- 172.21.15.102:0/1424642671 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa2c06ac00 0x7ffa2c06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.185 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.185+0000 7ffa47400640 1 -- 172.21.15.102:0/1424642671 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7ffa380aaca0 con 0x7ffa500ad330 2024-03-23T03:51:59.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.185+0000 7ffa4ea00640 1 --2- 172.21.15.102:0/1424642671 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa2c06ac00 0x7ffa2c06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.186+0000 7ffa4ea00640 1 --2- 172.21.15.102:0/1424642671 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa2c06ac00 0x7ffa2c06d0c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ffa40006f00 tx=0x7ffa40006e10 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:59.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.190+0000 7ffa47400640 1 -- 172.21.15.102:0/1424642671 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7ffa38075fa0 con 0x7ffa500ad330 2024-03-23T03:51:59.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.384+0000 7ffa4fe00640 1 -- 172.21.15.102:0/1424642671 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7ffa500cdd30 con 0x7ffa2c06ac00 2024-03-23T03:51:59.385 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.385+0000 7ffa47400640 1 -- 172.21.15.102:0/1424642671 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ffa40009420 con 0x7ffa2c06ac00 2024-03-23T03:51:59.386 INFO:teuthology.orchestra.run.smithi102.stderr:instructed osd(s) 1 to deep-scrub 2024-03-23T03:51:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.387+0000 7ffa4fe00640 1 -- 172.21.15.102:0/1424642671 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa2c06ac00 msgr2=0x7ffa2c06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.388+0000 7ffa4fe00640 1 --2- 172.21.15.102:0/1424642671 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa2c06ac00 0x7ffa2c06d0c0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ffa40006f00 tx=0x7ffa40006e10 comp rx=0 tx=0).stop 2024-03-23T03:51:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.388+0000 7ffa4fe00640 1 -- 172.21.15.102:0/1424642671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa500ad330 msgr2=0x7ffa500ab920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.388+0000 7ffa4fe00640 1 --2- 172.21.15.102:0/1424642671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa500ad330 0x7ffa500ab920 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7ffa3802c380 tx=0x7ffa38001930 comp rx=0 tx=0).stop 2024-03-23T03:51:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.388+0000 7ffa4fe00640 1 -- 172.21.15.102:0/1424642671 shutdown_connections 2024-03-23T03:51:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.388+0000 7ffa4fe00640 1 --2- 172.21.15.102:0/1424642671 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ffa2c06ac00 0x7ffa2c06d0c0 secure :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ffa40006f00 tx=0x7ffa40006e10 comp rx=0 tx=0).stop 2024-03-23T03:51:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.388+0000 7ffa4fe00640 1 --2- 172.21.15.102:0/1424642671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffa5006c040 0x7ffa5006c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.388+0000 7ffa4fe00640 1 --2- 172.21.15.102:0/1424642671 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa500ad330 0x7ffa500ab920 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.388+0000 7ffa4fe00640 1 --2- 172.21.15.102:0/1424642671 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ffa340047c0 0x7ffa501584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.388+0000 7ffa4fe00640 1 -- 172.21.15.102:0/1424642671 >> 172.21.15.102:0/1424642671 conn(0x7ffa50065e20 msgr2=0x7ffa5006a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:59.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.388+0000 7ffa4fe00640 1 -- 172.21.15.102:0/1424642671 shutdown_connections 2024-03-23T03:51:59.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.388+0000 7ffa4fe00640 1 -- 172.21.15.102:0/1424642671 wait complete. 2024-03-23T03:51:59.410 INFO:tasks.ceph:Scrubbing osd.2 2024-03-23T03:51:59.410 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-23T03:51:59.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.615+0000 7f3828200640 1 Processor -- start 2024-03-23T03:51:59.616 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.615+0000 7f3828200640 1 -- start start 2024-03-23T03:51:59.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.617+0000 7f3828200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3820070cd0 0x7f3820160bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.617+0000 7f3828200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f382006d390 con 0x7f38201456a0 2024-03-23T03:51:59.617 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.617+0000 7f3828200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f382006d670 con 0x7f38200cf040 2024-03-23T03:51:59.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.617+0000 7f3828200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38200ad890 con 0x7f3820070cd0 2024-03-23T03:51:59.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.617+0000 7f3825800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f38201456a0 0x7f38200ce930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54266/0 (socket says 172.21.15.102:54266) 2024-03-23T03:51:59.618 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.617+0000 7f3825800640 1 -- 172.21.15.102:0/3020112316 learned_addr learned my addr 172.21.15.102:0/3020112316 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:59.619 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.617+0000 7f3826200640 1 --2- 172.21.15.102:0/3020112316 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3820070cd0 0x7f3820160bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.618+0000 7f381fe00640 1 -- 172.21.15.102:0/3020112316 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180244301 0 0) 0x7f382006d390 con 0x7f38201456a0 2024-03-23T03:51:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.618+0000 7f381fe00640 1 -- 172.21.15.102:0/3020112316 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3808003520 con 0x7f38201456a0 2024-03-23T03:51:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.618+0000 7f381fe00640 1 -- 172.21.15.102:0/3020112316 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2120744720 0 0) 0x7f3808003520 con 0x7f38201456a0 2024-03-23T03:51:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.618+0000 7f381fe00640 1 -- 172.21.15.102:0/3020112316 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3820070cd0 msgr2=0x7f3820160bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.618+0000 7f381fe00640 1 --2- 172.21.15.102:0/3020112316 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3820070cd0 0x7f3820160bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.618+0000 7f381fe00640 1 -- 172.21.15.102:0/3020112316 >> v1:172.21.15.173:6789/0 conn(0x7f38200cf040 legacy=0x7f38201604c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.618+0000 7f381fe00640 1 -- 172.21.15.102:0/3020112316 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38200701f0 con 0x7f38201456a0 2024-03-23T03:51:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.618+0000 7f381fe00640 1 -- 172.21.15.102:0/3020112316 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f3814001690 con 0x7f38201456a0 2024-03-23T03:51:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.618+0000 7f381fe00640 1 -- 172.21.15.102:0/3020112316 >> v1:172.21.15.102:6789/0 conn(0x7f38201456a0 legacy=0x7f38200ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.618+0000 7f381fe00640 1 --2- 172.21.15.102:0/3020112316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080039a0 0x7f3808003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.618+0000 7f381fe00640 1 --2- 172.21.15.102:0/3020112316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38080047c0 0x7f3808004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.618+0000 7f381fe00640 1 --2- 172.21.15.102:0/3020112316 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3820070cd0 0x7f3808005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.618+0000 7f381fe00640 1 -- 172.21.15.102:0/3020112316 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38200ad890 con 0x7f38080039a0 2024-03-23T03:51:59.620 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.618+0000 7f381fe00640 1 -- 172.21.15.102:0/3020112316 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380800ba60 con 0x7f38080047c0 2024-03-23T03:51:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.618+0000 7f381fe00640 1 -- 172.21.15.102:0/3020112316 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f380800bc20 con 0x7f3820070cd0 2024-03-23T03:51:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.619+0000 7f3825800640 1 --2- 172.21.15.102:0/3020112316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080039a0 0x7f3808003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.619+0000 7f3824e00640 1 --2- 172.21.15.102:0/3020112316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38080047c0 0x7f3808004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.619+0000 7f3825800640 1 -- 172.21.15.102:0/3020112316 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3820070cd0 msgr2=0x7f3808005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.619+0000 7f3825800640 1 --2- 172.21.15.102:0/3020112316 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f3820070cd0 0x7f3808005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.619+0000 7f3825800640 1 -- 172.21.15.102:0/3020112316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38080047c0 msgr2=0x7f3808004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.619+0000 7f3825800640 1 --2- 172.21.15.102:0/3020112316 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f38080047c0 0x7f3808004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.621 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.619+0000 7f3825800640 1 -- 172.21.15.102:0/3020112316 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3808003520 con 0x7f38080039a0 2024-03-23T03:51:59.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.620+0000 7f3825800640 1 --2- 172.21.15.102:0/3020112316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080039a0 0x7f3808003dc0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f38201615c0 tx=0x7f38080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8de0ccce0bcafa9f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:59.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.621+0000 7f381fe00640 1 -- 172.21.15.102:0/3020112316 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3814005ad0 con 0x7f38080039a0 2024-03-23T03:51:59.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.621+0000 7f381fe00640 1 -- 172.21.15.102:0/3020112316 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3814005c30 con 0x7f38080039a0 2024-03-23T03:51:59.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.621+0000 7f381fe00640 1 -- 172.21.15.102:0/3020112316 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381402c790 con 0x7f38080039a0 2024-03-23T03:51:59.622 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.621+0000 7f3828200640 1 -- 172.21.15.102:0/3020112316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080039a0 msgr2=0x7f3808003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.621+0000 7f3828200640 1 --2- 172.21.15.102:0/3020112316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080039a0 0x7f3808003dc0 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f38201615c0 tx=0x7f38080057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.621+0000 7f3826200640 1 -- 172.21.15.102:0/3020112316 reap_dead start 2024-03-23T03:51:59.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.622+0000 7f3828200640 1 -- 172.21.15.102:0/3020112316 shutdown_connections 2024-03-23T03:51:59.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.622+0000 7f3828200640 1 -- 172.21.15.102:0/3020112316 >> 172.21.15.102:0/3020112316 conn(0x7f382006a0b0 msgr2=0x7f382006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:59.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.622+0000 7f3828200640 1 -- 172.21.15.102:0/3020112316 shutdown_connections 2024-03-23T03:51:59.623 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.622+0000 7f3828200640 1 -- 172.21.15.102:0/3020112316 wait complete. 2024-03-23T03:51:59.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.623+0000 7f3828200640 1 Processor -- start 2024-03-23T03:51:59.624 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.623+0000 7f3828200640 1 -- start start 2024-03-23T03:51:59.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.624+0000 7f3828200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080039a0 0x7f38201428e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.624+0000 7f3828200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f382013d240 0x7f3820140070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.624+0000 7f3828200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38201405b0 0x7f3820143180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.624+0000 7f3828200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3820140a30 con 0x7f38080039a0 2024-03-23T03:51:59.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.624+0000 7f3828200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3820142e20 con 0x7f382013d240 2024-03-23T03:51:59.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.624+0000 7f3828200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3820141840 con 0x7f38201405b0 2024-03-23T03:51:59.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.625+0000 7f3825800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080039a0 0x7f38201428e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.625+0000 7f3825800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080039a0 0x7f38201428e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43634/0 (socket says 172.21.15.102:43634) 2024-03-23T03:51:59.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.625+0000 7f3825800640 1 -- 172.21.15.102:0/2234058992 learned_addr learned my addr 172.21.15.102:0/2234058992 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:59.626 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.625+0000 7f3826200640 1 --2- 172.21.15.102:0/2234058992 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38201405b0 0x7f3820143180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.625+0000 7f3824e00640 1 --2- 172.21.15.102:0/2234058992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f382013d240 0x7f3820140070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.625+0000 7f3825800640 1 -- 172.21.15.102:0/2234058992 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38201405b0 msgr2=0x7f3820143180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.625+0000 7f3825800640 1 --2- 172.21.15.102:0/2234058992 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f38201405b0 0x7f3820143180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.625+0000 7f3825800640 1 -- 172.21.15.102:0/2234058992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f382013d240 msgr2=0x7f3820140070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.625+0000 7f3825800640 1 --2- 172.21.15.102:0/2234058992 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f382013d240 0x7f3820140070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.625+0000 7f3825800640 1 -- 172.21.15.102:0/2234058992 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3820141b20 con 0x7f38080039a0 2024-03-23T03:51:59.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.625+0000 7f3825800640 1 --2- 172.21.15.102:0/2234058992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080039a0 0x7f38201428e0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f381402c330 tx=0x7f3814001950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:59.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.626+0000 7f381d600640 1 -- 172.21.15.102:0/2234058992 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381402cba0 con 0x7f38080039a0 2024-03-23T03:51:59.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.626+0000 7f381d600640 1 -- 172.21.15.102:0/2234058992 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f381402cd00 con 0x7f38080039a0 2024-03-23T03:51:59.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.626+0000 7f381d600640 1 -- 172.21.15.102:0/2234058992 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3814007290 con 0x7f38080039a0 2024-03-23T03:51:59.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.626+0000 7f3828200640 1 -- 172.21.15.102:0/2234058992 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3820141e00 con 0x7f38080039a0 2024-03-23T03:51:59.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.626+0000 7f3828200640 1 -- 172.21.15.102:0/2234058992 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38201423c0 con 0x7f38080039a0 2024-03-23T03:51:59.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.627+0000 7f3828200640 1 -- 172.21.15.102:0/2234058992 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f38200707b0 con 0x7f38080039a0 2024-03-23T03:51:59.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.628+0000 7f381d600640 1 -- 172.21.15.102:0/2234058992 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f3814032070 con 0x7f38080039a0 2024-03-23T03:51:59.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.629+0000 7f381d600640 1 --2- 172.21.15.102:0/2234058992 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f380006ac00 0x7f380006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.629+0000 7f381d600640 1 -- 172.21.15.102:0/2234058992 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f38140aa530 con 0x7f38080039a0 2024-03-23T03:51:59.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.629+0000 7f381d600640 1 --2- 172.21.15.102:0/2234058992 >> [v2:172.21.15.102:6800/37350,v1:172.21.15.102:6801/37350] conn(0x7f3800070f00 0x7f3800073340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.629+0000 7f381d600640 1 -- 172.21.15.102:0/2234058992 --> [v2:172.21.15.102:6800/37350,v1:172.21.15.102:6801/37350] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3800073a10 con 0x7f3800070f00 2024-03-23T03:51:59.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.629+0000 7f3824e00640 1 --2- 172.21.15.102:0/2234058992 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f380006ac00 0x7f380006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.629+0000 7f381d600640 1 -- 172.21.15.102:0/2234058992 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38140acbf0 con 0x7f38080039a0 2024-03-23T03:51:59.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.629+0000 7f3826200640 1 --2- 172.21.15.102:0/2234058992 >> [v2:172.21.15.102:6800/37350,v1:172.21.15.102:6801/37350] conn(0x7f3800070f00 0x7f3800073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.630+0000 7f3824e00640 1 --2- 172.21.15.102:0/2234058992 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f380006ac00 0x7f380006d0c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f380c007fd0 tx=0x7f380c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:59.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.630+0000 7f3826200640 1 --2- 172.21.15.102:0/2234058992 >> [v2:172.21.15.102:6800/37350,v1:172.21.15.102:6801/37350] conn(0x7f3800070f00 0x7f3800073340 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:59.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.631+0000 7f381d600640 1 -- 172.21.15.102:0/2234058992 <== osd.2 v2:172.21.15.102:6800/37350 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f3800073a10 con 0x7f3800070f00 2024-03-23T03:51:59.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.644+0000 7f3828200640 1 -- 172.21.15.102:0/2234058992 --> [v2:172.21.15.102:6800/37350,v1:172.21.15.102:6801/37350] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3820145a80 con 0x7f3800070f00 2024-03-23T03:51:59.645 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.645+0000 7f381d600640 1 -- 172.21.15.102:0/2234058992 <== osd.2 v2:172.21.15.102:6800/37350 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f3818011040 con 0x7f3800070f00 2024-03-23T03:51:59.645 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:51:59.645 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T03:51:59.645 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:51:59.646 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.645+0000 7f3828200640 1 -- 172.21.15.102:0/2234058992 >> [v2:172.21.15.102:6800/37350,v1:172.21.15.102:6801/37350] conn(0x7f3800070f00 msgr2=0x7f3800073340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.645+0000 7f3828200640 1 --2- 172.21.15.102:0/2234058992 >> [v2:172.21.15.102:6800/37350,v1:172.21.15.102:6801/37350] conn(0x7f3800070f00 0x7f3800073340 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.645+0000 7f3828200640 1 -- 172.21.15.102:0/2234058992 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f380006ac00 msgr2=0x7f380006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.645+0000 7f3828200640 1 --2- 172.21.15.102:0/2234058992 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f380006ac00 0x7f380006d0c0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f380c007fd0 tx=0x7f380c009040 comp rx=0 tx=0).stop 2024-03-23T03:51:59.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.645+0000 7f3828200640 1 -- 172.21.15.102:0/2234058992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080039a0 msgr2=0x7f38201428e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.645+0000 7f3828200640 1 --2- 172.21.15.102:0/2234058992 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f38080039a0 0x7f38201428e0 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f381402c330 tx=0x7f3814001950 comp rx=0 tx=0).stop 2024-03-23T03:51:59.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.645+0000 7f3826200640 1 -- 172.21.15.102:0/2234058992 reap_dead start 2024-03-23T03:51:59.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.646+0000 7f3828200640 1 -- 172.21.15.102:0/2234058992 shutdown_connections 2024-03-23T03:51:59.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.646+0000 7f3828200640 1 -- 172.21.15.102:0/2234058992 >> 172.21.15.102:0/2234058992 conn(0x7f382006a0b0 msgr2=0x7f38200ce900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:59.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.646+0000 7f3828200640 1 -- 172.21.15.102:0/2234058992 shutdown_connections 2024-03-23T03:51:59.647 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.646+0000 7f3828200640 1 -- 172.21.15.102:0/2234058992 wait complete. 2024-03-23T03:51:59.665 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-23T03:51:59.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.873+0000 7fa1b3200640 1 Processor -- start 2024-03-23T03:51:59.874 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.873+0000 7fa1b3200640 1 -- start start 2024-03-23T03:51:59.875 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.875+0000 7fa1b3200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7fa1ac158c00 0x7fa1ac06d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.875+0000 7fa1b3200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac06c3e0 con 0x7fa1ac070170 2024-03-23T03:51:59.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.875+0000 7fa1b3200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1ac06c6c0 con 0x7fa1ac0ca960 2024-03-23T03:51:59.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.875+0000 7fa1b3200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1ac06d9c0 con 0x7fa1ac158c00 2024-03-23T03:51:59.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.875+0000 7fa1b1200640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fa1ac070170 0x7fa1ac15a490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54280/0 (socket says 172.21.15.102:54280) 2024-03-23T03:51:59.876 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.875+0000 7fa1b1200640 1 -- 172.21.15.102:0/3343720152 learned_addr learned my addr 172.21.15.102:0/3343720152 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:51:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.875+0000 7fa1aaa00640 1 -- 172.21.15.102:0/3343720152 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592541254 0 0) 0x7fa1ac06c3e0 con 0x7fa1ac070170 2024-03-23T03:51:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.875+0000 7fa1abe00640 1 --2- 172.21.15.102:0/3343720152 >> v2:172.21.15.173:3300/0 conn(0x7fa1ac158c00 0x7fa1ac06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.875+0000 7fa1aaa00640 1 -- 172.21.15.102:0/3343720152 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa190003520 con 0x7fa1ac070170 2024-03-23T03:51:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.876+0000 7fa1aaa00640 1 -- 172.21.15.102:0/3343720152 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2799035427 0 0) 0x7fa190003520 con 0x7fa1ac070170 2024-03-23T03:51:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.876+0000 7fa1aaa00640 1 -- 172.21.15.102:0/3343720152 >> v2:172.21.15.173:3300/0 conn(0x7fa1ac158c00 msgr2=0x7fa1ac06d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.876+0000 7fa1aaa00640 1 --2- 172.21.15.102:0/3343720152 >> v2:172.21.15.173:3300/0 conn(0x7fa1ac158c00 0x7fa1ac06d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.876+0000 7fa1aaa00640 1 -- 172.21.15.102:0/3343720152 >> v1:172.21.15.173:6789/0 conn(0x7fa1ac0ca960 legacy=0x7fa1ac0ada50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.876+0000 7fa1aaa00640 1 -- 172.21.15.102:0/3343720152 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac06c9a0 con 0x7fa1ac070170 2024-03-23T03:51:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.876+0000 7fa1aaa00640 1 -- 172.21.15.102:0/3343720152 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fa1a4001670 con 0x7fa1ac070170 2024-03-23T03:51:59.878 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.876+0000 7fa1aaa00640 1 -- 172.21.15.102:0/3343720152 >> v1:172.21.15.102:6789/0 conn(0x7fa1ac070170 legacy=0x7fa1ac15a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.876+0000 7fa1aaa00640 1 --2- 172.21.15.102:0/3343720152 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa1900039c0 0x7fa190003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.876+0000 7fa1aaa00640 1 --2- 172.21.15.102:0/3343720152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1900047c0 0x7fa190004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.876+0000 7fa1aaa00640 1 --2- 172.21.15.102:0/3343720152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1900052a0 0x7fa19000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.876+0000 7fa1aaa00640 1 -- 172.21.15.102:0/3343720152 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac06d9c0 con 0x7fa1900047c0 2024-03-23T03:51:59.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.876+0000 7fa1aaa00640 1 -- 172.21.15.102:0/3343720152 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa19000c1a0 con 0x7fa1900052a0 2024-03-23T03:51:59.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.876+0000 7fa1aaa00640 1 -- 172.21.15.102:0/3343720152 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa19000c360 con 0x7fa1900039c0 2024-03-23T03:51:59.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.876+0000 7fa1ab400640 1 --2- 172.21.15.102:0/3343720152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1900047c0 0x7fa190004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.877+0000 7fa1b1200640 1 --2- 172.21.15.102:0/3343720152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1900052a0 0x7fa19000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.877+0000 7fa1abe00640 1 --2- 172.21.15.102:0/3343720152 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa1900039c0 0x7fa190003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.877+0000 7fa1ab400640 1 -- 172.21.15.102:0/3343720152 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa1900039c0 msgr2=0x7fa190003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:59.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.877+0000 7fa1ab400640 1 --2- 172.21.15.102:0/3343720152 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa1900039c0 0x7fa190003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.877+0000 7fa1ab400640 1 -- 172.21.15.102:0/3343720152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1900052a0 msgr2=0x7fa19000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:51:59.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.877+0000 7fa1ab400640 1 --2- 172.21.15.102:0/3343720152 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1900052a0 0x7fa19000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.879 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.877+0000 7fa1ab400640 1 -- 172.21.15.102:0/3343720152 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa190003520 con 0x7fa1900047c0 2024-03-23T03:51:59.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.877+0000 7fa1b1200640 1 -- 172.21.15.102:0/3343720152 reap_dead start 2024-03-23T03:51:59.880 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.877+0000 7fa1abe00640 1 --2- 172.21.15.102:0/3343720152 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa1900039c0 0x7fa190003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:51:59.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.878+0000 7fa1ab400640 1 --2- 172.21.15.102:0/3343720152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1900047c0 0x7fa190004c40 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fa194001750 tx=0x7fa1900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93ff2419573bdf39 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:59.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.878+0000 7fa1aaa00640 1 -- 172.21.15.102:0/3343720152 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1940052d0 con 0x7fa1900047c0 2024-03-23T03:51:59.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.878+0000 7fa1aaa00640 1 -- 172.21.15.102:0/3343720152 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19402e050 con 0x7fa1900047c0 2024-03-23T03:51:59.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.878+0000 7fa1aaa00640 1 -- 172.21.15.102:0/3343720152 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa194003830 con 0x7fa1900047c0 2024-03-23T03:51:59.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.879+0000 7fa1b3200640 1 -- 172.21.15.102:0/3343720152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1900047c0 msgr2=0x7fa190004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.879+0000 7fa1b3200640 1 --2- 172.21.15.102:0/3343720152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1900047c0 0x7fa190004c40 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fa194001750 tx=0x7fa1900057f0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.879+0000 7fa1b3200640 1 -- 172.21.15.102:0/3343720152 shutdown_connections 2024-03-23T03:51:59.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.879+0000 7fa1b3200640 1 --2- 172.21.15.102:0/3343720152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1900047c0 0x7fa190004c40 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.879+0000 7fa1b3200640 1 -- 172.21.15.102:0/3343720152 >> 172.21.15.102:0/3343720152 conn(0x7fa1ac06a0b0 msgr2=0x7fa1ac06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:51:59.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.879+0000 7fa1b3200640 1 -- 172.21.15.102:0/3343720152 shutdown_connections 2024-03-23T03:51:59.881 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.879+0000 7fa1b3200640 1 -- 172.21.15.102:0/3343720152 wait complete. 2024-03-23T03:51:59.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.881+0000 7fa1b3200640 1 Processor -- start 2024-03-23T03:51:59.882 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.881+0000 7fa1b3200640 1 -- start start 2024-03-23T03:51:59.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.882+0000 7fa1b3200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa1900047c0 0x7fa1ac1475c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.882+0000 7fa1b3200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1ac1573d0 0x7fa1ac166cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.882+0000 7fa1b3200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1ac157850 0x7fa1ac1671f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.882+0000 7fa1b3200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac13e150 con 0x7fa1ac157850 2024-03-23T03:51:59.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.882+0000 7fa1b3200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac167730 con 0x7fa1ac1573d0 2024-03-23T03:51:59.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.882+0000 7fa1b3200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1ac13dcc0 con 0x7fa1900047c0 2024-03-23T03:51:59.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.883+0000 7fa1b1200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1ac157850 0x7fa1ac1671f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.883+0000 7fa1b1200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1ac157850 0x7fa1ac1671f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43654/0 (socket says 172.21.15.102:43654) 2024-03-23T03:51:59.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.883+0000 7fa1b1200640 1 -- 172.21.15.102:0/1058168747 learned_addr learned my addr 172.21.15.102:0/1058168747 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:51:59.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.883+0000 7fa1abe00640 1 --2- 172.21.15.102:0/1058168747 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa1900047c0 0x7fa1ac1475c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.883+0000 7fa1ab400640 1 --2- 172.21.15.102:0/1058168747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1ac1573d0 0x7fa1ac166cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.883+0000 7fa1b1200640 1 -- 172.21.15.102:0/1058168747 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa1900047c0 msgr2=0x7fa1ac1475c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.883+0000 7fa1b1200640 1 --2- 172.21.15.102:0/1058168747 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa1900047c0 0x7fa1ac1475c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.884 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.883+0000 7fa1b1200640 1 -- 172.21.15.102:0/1058168747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1ac1573d0 msgr2=0x7fa1ac166cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:51:59.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.883+0000 7fa1b1200640 1 --2- 172.21.15.102:0/1058168747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1ac1573d0 0x7fa1ac166cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:51:59.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.883+0000 7fa1b1200640 1 -- 172.21.15.102:0/1058168747 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1ac13d0b0 con 0x7fa1ac157850 2024-03-23T03:51:59.885 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.883+0000 7fa1b1200640 1 --2- 172.21.15.102:0/1058168747 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1ac157850 0x7fa1ac1671f0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0b3f80 tx=0x7fa1a400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.884+0000 7fa1a3e00640 1 -- 172.21.15.102:0/1058168747 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a400eed0 con 0x7fa1ac157850 2024-03-23T03:51:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.884+0000 7fa1a3e00640 1 -- 172.21.15.102:0/1058168747 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a400c660 con 0x7fa1ac157850 2024-03-23T03:51:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.884+0000 7fa1a3e00640 1 -- 172.21.15.102:0/1058168747 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a400c9f0 con 0x7fa1ac157850 2024-03-23T03:51:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.884+0000 7fa1b3200640 1 -- 172.21.15.102:0/1058168747 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1ac13d390 con 0x7fa1ac157850 2024-03-23T03:51:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.884+0000 7fa1b3200640 1 -- 172.21.15.102:0/1058168747 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1ac13d950 con 0x7fa1ac157850 2024-03-23T03:51:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.885+0000 7fa1b3200640 1 -- 172.21.15.102:0/1058168747 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1ac073390 con 0x7fa1ac157850 2024-03-23T03:51:59.886 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.886+0000 7fa1a3e00640 1 -- 172.21.15.102:0/1058168747 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fa1a400cb50 con 0x7fa1ac157850 2024-03-23T03:51:59.887 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.887+0000 7fa1a3e00640 1 --2- 172.21.15.102:0/1058168747 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa18806ac30 0x7fa18806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:51:59.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.887+0000 7fa1a3e00640 1 -- 172.21.15.102:0/1058168747 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fa1a4088960 con 0x7fa1ac157850 2024-03-23T03:51:59.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.887+0000 7fa1abe00640 1 --2- 172.21.15.102:0/1058168747 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa18806ac30 0x7fa18806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:51:59.888 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.888+0000 7fa1abe00640 1 --2- 172.21.15.102:0/1058168747 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa18806ac30 0x7fa18806d0f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac1481c0 tx=0x7fa19c00b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:51:59.892 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:51:59.892+0000 7fa1a3e00640 1 -- 172.21.15.102:0/1058168747 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fa1a4053c60 con 0x7fa1ac157850 2024-03-23T03:52:00.090 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.090+0000 7fa1b3200640 1 -- 172.21.15.102:0/1058168747 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1ac06cf60 con 0x7fa18806ac30 2024-03-23T03:52:00.091 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.091+0000 7fa1a3e00640 1 -- 172.21.15.102:0/1058168747 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa19c00f050 con 0x7fa18806ac30 2024-03-23T03:52:00.091 INFO:teuthology.orchestra.run.smithi102.stderr:instructed osd(s) 2 to deep-scrub 2024-03-23T03:52:00.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.093+0000 7fa1b3200640 1 -- 172.21.15.102:0/1058168747 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa18806ac30 msgr2=0x7fa18806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:00.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.093+0000 7fa1b3200640 1 --2- 172.21.15.102:0/1058168747 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa18806ac30 0x7fa18806d0f0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac1481c0 tx=0x7fa19c00b040 comp rx=0 tx=0).stop 2024-03-23T03:52:00.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.093+0000 7fa1b3200640 1 -- 172.21.15.102:0/1058168747 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1ac157850 msgr2=0x7fa1ac1671f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:00.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.093+0000 7fa1b3200640 1 --2- 172.21.15.102:0/1058168747 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1ac157850 0x7fa1ac1671f0 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac0b3f80 tx=0x7fa1a400eb90 comp rx=0 tx=0).stop 2024-03-23T03:52:00.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.093+0000 7fa1b3200640 1 -- 172.21.15.102:0/1058168747 shutdown_connections 2024-03-23T03:52:00.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.093+0000 7fa1b3200640 1 --2- 172.21.15.102:0/1058168747 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fa18806ac30 0x7fa18806d0f0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.094 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.093+0000 7fa1b3200640 1 --2- 172.21.15.102:0/1058168747 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa1ac157850 0x7fa1ac1671f0 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.093+0000 7fa1b3200640 1 --2- 172.21.15.102:0/1058168747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa1ac1573d0 0x7fa1ac166cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.093+0000 7fa1b3200640 1 --2- 172.21.15.102:0/1058168747 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fa1900047c0 0x7fa1ac1475c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.093+0000 7fa1b3200640 1 -- 172.21.15.102:0/1058168747 >> 172.21.15.102:0/1058168747 conn(0x7fa1ac06a0b0 msgr2=0x7fa1ac0b3770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:00.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.093+0000 7fa1b3200640 1 -- 172.21.15.102:0/1058168747 shutdown_connections 2024-03-23T03:52:00.095 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.093+0000 7fa1b3200640 1 -- 172.21.15.102:0/1058168747 wait complete. 2024-03-23T03:52:00.112 INFO:tasks.ceph:Scrubbing osd.3 2024-03-23T03:52:00.113 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-23T03:52:00.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.323+0000 7f6743a00640 1 Processor -- start 2024-03-23T03:52:00.324 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.324+0000 7f6743a00640 1 -- start start 2024-03-23T03:52:00.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.325+0000 7f6743a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f673c0ce930 0x7f673c0add30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:00.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.325+0000 7f6743a00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f673c06ce90 0x7f673c06d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:00.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.325+0000 7f6743a00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673c0cdba0 con 0x7f673c0cd3e0 2024-03-23T03:52:00.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.325+0000 7f6743a00640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673c06c760 con 0x7f673c06ce90 2024-03-23T03:52:00.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.325+0000 7f6743a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673c0b42d0 con 0x7f673c0ce930 2024-03-23T03:52:00.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.325+0000 7f6733e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f673c0cd3e0 0x7f673c0ced10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54286/0 (socket says 172.21.15.102:54286) 2024-03-23T03:52:00.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.325+0000 7f6733e00640 1 -- 172.21.15.102:0/2169602955 learned_addr learned my addr 172.21.15.102:0/2169602955 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:52:00.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.325+0000 7f6741a00640 1 --2- 172.21.15.102:0/2169602955 >> v2:172.21.15.102:3300/0 conn(0x7f673c06ce90 0x7f673c06d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:00.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.326+0000 7f6741000640 1 --2- 172.21.15.102:0/2169602955 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f673c0ce930 0x7f673c0add30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:00.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.326+0000 7f673be00640 1 -- 172.21.15.102:0/2169602955 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827896635 0 0) 0x7f673c0cdba0 con 0x7f673c0cd3e0 2024-03-23T03:52:00.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.326+0000 7f6741a00640 1 -- 172.21.15.102:0/2169602955 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f673c0ce930 msgr2=0x7f673c0add30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:00.327 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.326+0000 7f6741a00640 1 --2- 172.21.15.102:0/2169602955 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f673c0ce930 0x7f673c0add30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.326+0000 7f6741a00640 1 -- 172.21.15.102:0/2169602955 >> v1:172.21.15.102:6789/0 conn(0x7f673c0cd3e0 legacy=0x7f673c0ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:00.328 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.326+0000 7f6741a00640 1 -- 172.21.15.102:0/2169602955 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c0cde80 con 0x7f673c06ce90 2024-03-23T03:52:00.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.327+0000 7f6741a00640 1 --2- 172.21.15.102:0/2169602955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f673c06ce90 0x7f673c06d320 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f6734009d90 tx=0x7f673402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b8e00e0617fe2c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:00.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.327+0000 7f673be00640 1 -- 172.21.15.102:0/2169602955 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673402cc60 con 0x7f673c06ce90 2024-03-23T03:52:00.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.327+0000 7f673be00640 1 -- 172.21.15.102:0/2169602955 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f673402cdc0 con 0x7f673c06ce90 2024-03-23T03:52:00.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.327+0000 7f673be00640 1 -- 172.21.15.102:0/2169602955 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6734034660 con 0x7f673c06ce90 2024-03-23T03:52:00.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.328+0000 7f6743a00640 1 -- 172.21.15.102:0/2169602955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f673c06ce90 msgr2=0x7f673c06d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:00.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.328+0000 7f6743a00640 1 --2- 172.21.15.102:0/2169602955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f673c06ce90 0x7f673c06d320 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f6734009d90 tx=0x7f673402beb0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.328+0000 7f6743a00640 1 -- 172.21.15.102:0/2169602955 shutdown_connections 2024-03-23T03:52:00.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.328+0000 7f6743a00640 1 --2- 172.21.15.102:0/2169602955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f673c06ce90 0x7f673c06d320 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.328+0000 7f6743a00640 1 --2- 172.21.15.102:0/2169602955 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f673c0ce930 0x7f673c0add30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.328+0000 7f6743a00640 1 -- 172.21.15.102:0/2169602955 >> 172.21.15.102:0/2169602955 conn(0x7f673c06a0b0 msgr2=0x7f673c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:00.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.328+0000 7f6743a00640 1 -- 172.21.15.102:0/2169602955 shutdown_connections 2024-03-23T03:52:00.329 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.328+0000 7f6743a00640 1 -- 172.21.15.102:0/2169602955 wait complete. 2024-03-23T03:52:00.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.330+0000 7f6743a00640 1 Processor -- start 2024-03-23T03:52:00.330 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.330+0000 7f6743a00640 1 -- start start 2024-03-23T03:52:00.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.331+0000 7f6743a00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f673c06ce90 0x7f673c1569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:00.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.331+0000 7f6743a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f673c0cd3e0 0x7f673c0b06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:00.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.331+0000 7f6743a00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c0ce930 0x7f673c152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:00.332 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.331+0000 7f6743a00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c156ef0 con 0x7f673c0cd3e0 2024-03-23T03:52:00.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.331+0000 7f6743a00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c157190 con 0x7f673c0ce930 2024-03-23T03:52:00.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.331+0000 7f6743a00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673c157470 con 0x7f673c06ce90 2024-03-23T03:52:00.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.331+0000 7f6733e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f673c0cd3e0 0x7f673c0b06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:00.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.331+0000 7f6733e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f673c0cd3e0 0x7f673c0b06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43674/0 (socket says 172.21.15.102:43674) 2024-03-23T03:52:00.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.331+0000 7f6733e00640 1 -- 172.21.15.102:0/2870886141 learned_addr learned my addr 172.21.15.102:0/2870886141 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:00.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.332+0000 7f6741000640 1 --2- 172.21.15.102:0/2870886141 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f673c06ce90 0x7f673c1569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:00.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.332+0000 7f6741a00640 1 --2- 172.21.15.102:0/2870886141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c0ce930 0x7f673c152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:00.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.332+0000 7f6733e00640 1 -- 172.21.15.102:0/2870886141 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f673c06ce90 msgr2=0x7f673c1569b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:00.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.332+0000 7f6733e00640 1 --2- 172.21.15.102:0/2870886141 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f673c06ce90 0x7f673c1569b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.332+0000 7f6733e00640 1 -- 172.21.15.102:0/2870886141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c0ce930 msgr2=0x7f673c152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:00.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.332+0000 7f6733e00640 1 --2- 172.21.15.102:0/2870886141 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f673c0ce930 0x7f673c152040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.332+0000 7f6733e00640 1 -- 172.21.15.102:0/2870886141 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6734009a70 con 0x7f673c0cd3e0 2024-03-23T03:52:00.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.332+0000 7f6733e00640 1 --2- 172.21.15.102:0/2870886141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f673c0cd3e0 0x7f673c0b06a0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f672400c4d0 tx=0x7f672400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:00.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.332+0000 7f6739600640 1 -- 172.21.15.102:0/2870886141 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67240074e0 con 0x7f673c0cd3e0 2024-03-23T03:52:00.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.332+0000 7f6739600640 1 -- 172.21.15.102:0/2870886141 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6724007660 con 0x7f673c0cd3e0 2024-03-23T03:52:00.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.332+0000 7f6739600640 1 -- 172.21.15.102:0/2870886141 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724004a20 con 0x7f673c0cd3e0 2024-03-23T03:52:00.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.332+0000 7f6743a00640 1 -- 172.21.15.102:0/2870886141 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673c157750 con 0x7f673c0cd3e0 2024-03-23T03:52:00.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.333+0000 7f6743a00640 1 -- 172.21.15.102:0/2870886141 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673c0b0be0 con 0x7f673c0cd3e0 2024-03-23T03:52:00.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.334+0000 7f6743a00640 1 -- 172.21.15.102:0/2870886141 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f673c15df90 con 0x7f673c0cd3e0 2024-03-23T03:52:00.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.335+0000 7f6739600640 1 -- 172.21.15.102:0/2870886141 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f6724004b80 con 0x7f673c0cd3e0 2024-03-23T03:52:00.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.336+0000 7f6739600640 1 --2- 172.21.15.102:0/2870886141 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f671806ac00 0x7f671806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:00.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.336+0000 7f6739600640 1 -- 172.21.15.102:0/2870886141 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f6724088440 con 0x7f673c0cd3e0 2024-03-23T03:52:00.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.336+0000 7f6741000640 1 --2- 172.21.15.102:0/2870886141 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f671806ac00 0x7f671806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:00.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.336+0000 7f6739600640 1 --2- 172.21.15.102:0/2870886141 >> [v2:172.21.15.102:6808/37353,v1:172.21.15.102:6809/37353] conn(0x7f6718070f00 0x7f6718073340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:00.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.336+0000 7f6739600640 1 -- 172.21.15.102:0/2870886141 --> [v2:172.21.15.102:6808/37353,v1:172.21.15.102:6809/37353] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6718073a10 con 0x7f6718070f00 2024-03-23T03:52:00.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.336+0000 7f6739600640 1 -- 172.21.15.102:0/2870886141 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f672408aa20 con 0x7f673c0cd3e0 2024-03-23T03:52:00.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.336+0000 7f6741a00640 1 --2- 172.21.15.102:0/2870886141 >> [v2:172.21.15.102:6808/37353,v1:172.21.15.102:6809/37353] conn(0x7f6718070f00 0x7f6718073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:00.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.337+0000 7f6741000640 1 --2- 172.21.15.102:0/2870886141 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f671806ac00 0x7f671806d0c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f672c006fd0 tx=0x7f672c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:00.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.337+0000 7f6741a00640 1 --2- 172.21.15.102:0/2870886141 >> [v2:172.21.15.102:6808/37353,v1:172.21.15.102:6809/37353] conn(0x7f6718070f00 0x7f6718073340 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:00.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.338+0000 7f6739600640 1 -- 172.21.15.102:0/2870886141 <== osd.3 v2:172.21.15.102:6808/37353 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f6718073a10 con 0x7f6718070f00 2024-03-23T03:52:00.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.351+0000 7f6743a00640 1 -- 172.21.15.102:0/2870886141 --> [v2:172.21.15.102:6808/37353,v1:172.21.15.102:6809/37353] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f673c06c970 con 0x7f6718070f00 2024-03-23T03:52:00.352 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.352+0000 7f6739600640 1 -- 172.21.15.102:0/2870886141 <== osd.3 v2:172.21.15.102:6808/37353 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f6734009450 con 0x7f6718070f00 2024-03-23T03:52:00.352 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:52:00.353 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T03:52:00.353 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:52:00.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.352+0000 7f6743a00640 1 -- 172.21.15.102:0/2870886141 >> [v2:172.21.15.102:6808/37353,v1:172.21.15.102:6809/37353] conn(0x7f6718070f00 msgr2=0x7f6718073340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:00.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.352+0000 7f6743a00640 1 --2- 172.21.15.102:0/2870886141 >> [v2:172.21.15.102:6808/37353,v1:172.21.15.102:6809/37353] conn(0x7f6718070f00 0x7f6718073340 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.352+0000 7f6743a00640 1 -- 172.21.15.102:0/2870886141 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f671806ac00 msgr2=0x7f671806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:00.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.352+0000 7f6743a00640 1 --2- 172.21.15.102:0/2870886141 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f671806ac00 0x7f671806d0c0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f672c006fd0 tx=0x7f672c008040 comp rx=0 tx=0).stop 2024-03-23T03:52:00.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.352+0000 7f6743a00640 1 -- 172.21.15.102:0/2870886141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f673c0cd3e0 msgr2=0x7f673c0b06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:00.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.352+0000 7f6743a00640 1 --2- 172.21.15.102:0/2870886141 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f673c0cd3e0 0x7f673c0b06a0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7f672400c4d0 tx=0x7f672400c990 comp rx=0 tx=0).stop 2024-03-23T03:52:00.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.352+0000 7f6741a00640 1 -- 172.21.15.102:0/2870886141 reap_dead start 2024-03-23T03:52:00.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.353+0000 7f6743a00640 1 -- 172.21.15.102:0/2870886141 shutdown_connections 2024-03-23T03:52:00.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.353+0000 7f6743a00640 1 -- 172.21.15.102:0/2870886141 >> 172.21.15.102:0/2870886141 conn(0x7f673c06a0b0 msgr2=0x7f673c0cbc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:00.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.353+0000 7f6743a00640 1 -- 172.21.15.102:0/2870886141 shutdown_connections 2024-03-23T03:52:00.354 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.353+0000 7f6743a00640 1 -- 172.21.15.102:0/2870886141 wait complete. 2024-03-23T03:52:00.369 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-23T03:52:00.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.568+0000 7f344c800640 1 Processor -- start 2024-03-23T03:52:00.569 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.568+0000 7f344c800640 1 -- start start 2024-03-23T03:52:00.570 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.570+0000 7f344c800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f34440ce930 0x7f34440ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:00.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.570+0000 7f344c800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f344406d2c0 0x7f34440cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:00.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.570+0000 7f344c800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34440cf250 con 0x7f34440cb660 2024-03-23T03:52:00.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.570+0000 7f344c800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f344415bae0 con 0x7f34440ce930 2024-03-23T03:52:00.571 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.570+0000 7f344c800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f344415ae60 con 0x7f344406d2c0 2024-03-23T03:52:00.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.571+0000 7f3449e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f34440ce930 0x7f34440ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:00.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.571+0000 7f344a800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f344406d2c0 0x7f34440cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:00.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.571+0000 7f3449e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f34440ce930 0x7f34440ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.102:53468/0 (socket says 172.21.15.102:53468) 2024-03-23T03:52:00.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.571+0000 7f3449e00640 1 -- 172.21.15.102:0/2517398818 learned_addr learned my addr 172.21.15.102:0/2517398818 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:00.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.571+0000 7f3449e00640 1 -- 172.21.15.102:0/2517398818 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f344406d2c0 msgr2=0x7f34440cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:00.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.571+0000 7f3449e00640 1 --2- 172.21.15.102:0/2517398818 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f344406d2c0 0x7f34440cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.571+0000 7f3449e00640 1 -- 172.21.15.102:0/2517398818 >> v1:172.21.15.173:6789/0 conn(0x7f34440cb660 legacy=0x7f3444070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:00.572 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.571+0000 7f3449e00640 1 -- 172.21.15.102:0/2517398818 --> v2:172.21.15.173:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f344415b140 con 0x7f34440ce930 2024-03-23T03:52:00.573 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.573+0000 7f3449e00640 1 --2- 172.21.15.102:0/2517398818 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34440ce930 0x7f34440ced10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3438009870 tx=0x7f343802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a93154f826c0326d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:00.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.574+0000 7f3448a00640 1 -- 172.21.15.102:0/2517398818 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438030840 con 0x7f34440ce930 2024-03-23T03:52:00.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.574+0000 7f3448a00640 1 -- 172.21.15.102:0/2517398818 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34380309a0 con 0x7f34440ce930 2024-03-23T03:52:00.574 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.574+0000 7f3448a00640 1 -- 172.21.15.102:0/2517398818 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438030d30 con 0x7f34440ce930 2024-03-23T03:52:00.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.574+0000 7f344c800640 1 -- 172.21.15.102:0/2517398818 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34440ce930 msgr2=0x7f34440ced10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:00.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.574+0000 7f344c800640 1 --2- 172.21.15.102:0/2517398818 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34440ce930 0x7f34440ced10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f3438009870 tx=0x7f343802b980 comp rx=0 tx=0).stop 2024-03-23T03:52:00.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.574+0000 7f344c800640 1 -- 172.21.15.102:0/2517398818 shutdown_connections 2024-03-23T03:52:00.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.574+0000 7f344c800640 1 --2- 172.21.15.102:0/2517398818 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f344406d2c0 0x7f34440cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.574+0000 7f344c800640 1 --2- 172.21.15.102:0/2517398818 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34440ce930 0x7f34440ced10 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.574+0000 7f344c800640 1 -- 172.21.15.102:0/2517398818 >> 172.21.15.102:0/2517398818 conn(0x7f344406a0b0 msgr2=0x7f344406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:00.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.574+0000 7f344c800640 1 -- 172.21.15.102:0/2517398818 shutdown_connections 2024-03-23T03:52:00.575 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.574+0000 7f344c800640 1 -- 172.21.15.102:0/2517398818 wait complete. 2024-03-23T03:52:00.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.576+0000 7f344c800640 1 Processor -- start 2024-03-23T03:52:00.576 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.576+0000 7f344c800640 1 -- start start 2024-03-23T03:52:00.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.577+0000 7f344c800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f344406d2c0 0x7f34441578d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:00.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.577+0000 7f344c800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f34440cb660 0x7f344414b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:00.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.577+0000 7f344c800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34440ce930 0x7f34440b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:00.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.577+0000 7f344c800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344414bf10 con 0x7f344406d2c0 2024-03-23T03:52:00.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.577+0000 7f344c800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34440b0530 con 0x7f34440ce930 2024-03-23T03:52:00.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.577+0000 7f344c800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34440aff70 con 0x7f34440cb660 2024-03-23T03:52:00.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.578+0000 7f3449e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f344406d2c0 0x7f34441578d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:00.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.578+0000 7f3449e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f344406d2c0 0x7f34441578d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43676/0 (socket says 172.21.15.102:43676) 2024-03-23T03:52:00.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.578+0000 7f3449e00640 1 -- 172.21.15.102:0/933412099 learned_addr learned my addr 172.21.15.102:0/933412099 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:00.579 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.578+0000 7f3449400640 1 --2- 172.21.15.102:0/933412099 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f34440cb660 0x7f344414b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:00.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.578+0000 7f344a800640 1 --2- 172.21.15.102:0/933412099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34440ce930 0x7f34440b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:00.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.578+0000 7f3449e00640 1 -- 172.21.15.102:0/933412099 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f34440cb660 msgr2=0x7f344414b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:00.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.578+0000 7f3449e00640 1 --2- 172.21.15.102:0/933412099 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f34440cb660 0x7f344414b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.578+0000 7f3449e00640 1 -- 172.21.15.102:0/933412099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34440ce930 msgr2=0x7f34440b0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:00.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.578+0000 7f3449e00640 1 --2- 172.21.15.102:0/933412099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34440ce930 0x7f34440b0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.578+0000 7f3449e00640 1 -- 172.21.15.102:0/933412099 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34380094d0 con 0x7f344406d2c0 2024-03-23T03:52:00.580 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.578+0000 7f3449e00640 1 --2- 172.21.15.102:0/933412099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f344406d2c0 0x7f34441578d0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f3438000c00 tx=0x7f3438004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:00.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.579+0000 7f343e000640 1 -- 172.21.15.102:0/933412099 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438039c90 con 0x7f344406d2c0 2024-03-23T03:52:00.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.579+0000 7f343e000640 1 -- 172.21.15.102:0/933412099 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3438039df0 con 0x7f344406d2c0 2024-03-23T03:52:00.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.579+0000 7f343e000640 1 -- 172.21.15.102:0/933412099 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3438043640 con 0x7f344406d2c0 2024-03-23T03:52:00.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.579+0000 7f344c800640 1 -- 172.21.15.102:0/933412099 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3444155280 con 0x7f344406d2c0 2024-03-23T03:52:00.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.579+0000 7f344c800640 1 -- 172.21.15.102:0/933412099 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34441570d0 con 0x7f344406d2c0 2024-03-23T03:52:00.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.580+0000 7f344c800640 1 -- 172.21.15.102:0/933412099 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3444073390 con 0x7f344406d2c0 2024-03-23T03:52:00.581 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.581+0000 7f343e000640 1 -- 172.21.15.102:0/933412099 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f34380437a0 con 0x7f344406d2c0 2024-03-23T03:52:00.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.582+0000 7f343e000640 1 --2- 172.21.15.102:0/933412099 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f342806ac00 0x7f342806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:00.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.582+0000 7f343e000640 1 -- 172.21.15.102:0/933412099 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f343803e070 con 0x7f344406d2c0 2024-03-23T03:52:00.582 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.582+0000 7f3449400640 1 --2- 172.21.15.102:0/933412099 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f342806ac00 0x7f342806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:00.583 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.583+0000 7f3449400640 1 --2- 172.21.15.102:0/933412099 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f342806ac00 0x7f342806d0c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f34440caba0 tx=0x7f3434009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:00.586 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.586+0000 7f343e000640 1 -- 172.21.15.102:0/933412099 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f3438077520 con 0x7f344406d2c0 2024-03-23T03:52:00.782 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.781+0000 7f344c800640 1 -- 172.21.15.102:0/933412099 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f3444155100 con 0x7f342806ac00 2024-03-23T03:52:00.783 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.782+0000 7f343e000640 1 -- 172.21.15.102:0/933412099 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3434001c70 con 0x7f342806ac00 2024-03-23T03:52:00.783 INFO:teuthology.orchestra.run.smithi102.stderr:instructed osd(s) 3 to deep-scrub 2024-03-23T03:52:00.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.785+0000 7f344c800640 1 -- 172.21.15.102:0/933412099 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f342806ac00 msgr2=0x7f342806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:00.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.785+0000 7f344c800640 1 --2- 172.21.15.102:0/933412099 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f342806ac00 0x7f342806d0c0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f34440caba0 tx=0x7f3434009040 comp rx=0 tx=0).stop 2024-03-23T03:52:00.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.785+0000 7f344c800640 1 -- 172.21.15.102:0/933412099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f344406d2c0 msgr2=0x7f34441578d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:00.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.785+0000 7f344c800640 1 --2- 172.21.15.102:0/933412099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f344406d2c0 0x7f34441578d0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f3438000c00 tx=0x7f3438004730 comp rx=0 tx=0).stop 2024-03-23T03:52:00.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.785+0000 7f344c800640 1 -- 172.21.15.102:0/933412099 shutdown_connections 2024-03-23T03:52:00.786 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.785+0000 7f344c800640 1 --2- 172.21.15.102:0/933412099 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f342806ac00 0x7f342806d0c0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.785+0000 7f344c800640 1 --2- 172.21.15.102:0/933412099 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f34440ce930 0x7f34440b0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.785+0000 7f344c800640 1 --2- 172.21.15.102:0/933412099 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f34440cb660 0x7f344414b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.785+0000 7f344c800640 1 --2- 172.21.15.102:0/933412099 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f344406d2c0 0x7f34441578d0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:00.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.785+0000 7f344c800640 1 -- 172.21.15.102:0/933412099 >> 172.21.15.102:0/933412099 conn(0x7f344406a0b0 msgr2=0x7f344415bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:00.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.785+0000 7f344c800640 1 -- 172.21.15.102:0/933412099 shutdown_connections 2024-03-23T03:52:00.787 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:00.785+0000 7f344c800640 1 -- 172.21.15.102:0/933412099 wait complete. 2024-03-23T03:52:00.804 INFO:tasks.ceph:Scrubbing osd.4 2024-03-23T03:52:00.804 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-23T03:52:01.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.017+0000 7f4bbc200640 1 Processor -- start 2024-03-23T03:52:01.018 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.017+0000 7f4bbc200640 1 -- start start 2024-03-23T03:52:01.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.019+0000 7f4bbc200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4bb4158db0 0x7f4bb415d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.019+0000 7f4bbc200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb415d870 con 0x7f4bb40ce930 2024-03-23T03:52:01.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.019+0000 7f4bbc200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb40caec0 con 0x7f4bb4070860 2024-03-23T03:52:01.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.019+0000 7f4bbc200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bb40cba90 con 0x7f4bb4158db0 2024-03-23T03:52:01.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.019+0000 7f4bb9800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4bb4158db0 0x7f4bb415d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.019+0000 7f4bb9800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f4bb4158db0 0x7f4bb415d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43688/0 (socket says 172.21.15.102:43688) 2024-03-23T03:52:01.020 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.019+0000 7f4bb9800640 1 -- 172.21.15.102:0/1094725729 learned_addr learned my addr 172.21.15.102:0/1094725729 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:01.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.020+0000 7f4bb9800640 1 -- 172.21.15.102:0/1094725729 >> v1:172.21.15.173:6789/0 conn(0x7f4bb4070860 legacy=0x7f4bb4163aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:01.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.020+0000 7f4bb9800640 1 -- 172.21.15.102:0/1094725729 >> v1:172.21.15.102:6789/0 conn(0x7f4bb40ce930 legacy=0x7f4bb4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.021 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.020+0000 7f4bb9800640 1 -- 172.21.15.102:0/1094725729 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb4159190 con 0x7f4bb4158db0 2024-03-23T03:52:01.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.021+0000 7f4bb9800640 1 --2- 172.21.15.102:0/1094725729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bb4158db0 0x7f4bb415d330 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8009870 tx=0x7f4ba802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2287bd8174fbae server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:01.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.021+0000 7f4bb3e00640 1 -- 172.21.15.102:0/1094725729 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8030860 con 0x7f4bb4158db0 2024-03-23T03:52:01.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.021+0000 7f4bb3e00640 1 -- 172.21.15.102:0/1094725729 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba80309c0 con 0x7f4bb4158db0 2024-03-23T03:52:01.022 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.021+0000 7f4bb3e00640 1 -- 172.21.15.102:0/1094725729 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8030d50 con 0x7f4bb4158db0 2024-03-23T03:52:01.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.022+0000 7f4bbc200640 1 -- 172.21.15.102:0/1094725729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bb4158db0 msgr2=0x7f4bb415d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.022+0000 7f4bbc200640 1 --2- 172.21.15.102:0/1094725729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bb4158db0 0x7f4bb415d330 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8009870 tx=0x7f4ba802b9a0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.022+0000 7f4bbc200640 1 -- 172.21.15.102:0/1094725729 shutdown_connections 2024-03-23T03:52:01.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.022+0000 7f4bbc200640 1 --2- 172.21.15.102:0/1094725729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bb4158db0 0x7f4bb415d330 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.022+0000 7f4bbc200640 1 -- 172.21.15.102:0/1094725729 >> 172.21.15.102:0/1094725729 conn(0x7f4bb406a0b0 msgr2=0x7f4bb406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:01.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.022+0000 7f4bbc200640 1 -- 172.21.15.102:0/1094725729 shutdown_connections 2024-03-23T03:52:01.023 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.022+0000 7f4bbc200640 1 -- 172.21.15.102:0/1094725729 wait complete. 2024-03-23T03:52:01.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.024+0000 7f4bbc200640 1 Processor -- start 2024-03-23T03:52:01.025 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.025+0000 7f4bbc200640 1 -- start start 2024-03-23T03:52:01.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.026+0000 7f4bbc200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4bb4070860 0x7f4bb413b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.026+0000 7f4bbc200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bb40ce930 0x7f4bb414e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.026+0000 7f4bbc200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bb4158db0 0x7f4bb414e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.026+0000 7f4bbc200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb41498d0 con 0x7f4bb40ce930 2024-03-23T03:52:01.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.026+0000 7f4bbc200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb414e1b0 con 0x7f4bb4158db0 2024-03-23T03:52:01.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.026+0000 7f4bbc200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb414ded0 con 0x7f4bb4070860 2024-03-23T03:52:01.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.026+0000 7f4bb8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bb40ce930 0x7f4bb414e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.027 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.026+0000 7f4bb8e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bb40ce930 0x7f4bb414e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43704/0 (socket says 172.21.15.102:43704) 2024-03-23T03:52:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.026+0000 7f4bb8e00640 1 -- 172.21.15.102:0/1283686532 learned_addr learned my addr 172.21.15.102:0/1283686532 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.026+0000 7f4bba200640 1 --2- 172.21.15.102:0/1283686532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bb4158db0 0x7f4bb414e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.026+0000 7f4bb8e00640 1 -- 172.21.15.102:0/1283686532 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4bb4070860 msgr2=0x7f4bb413b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.026+0000 7f4bb8e00640 1 --2- 172.21.15.102:0/1283686532 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4bb4070860 0x7f4bb413b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.026+0000 7f4bb8e00640 1 -- 172.21.15.102:0/1283686532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bb4158db0 msgr2=0x7f4bb414e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.026+0000 7f4bb8e00640 1 --2- 172.21.15.102:0/1283686532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bb4158db0 0x7f4bb414e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.026+0000 7f4bb8e00640 1 -- 172.21.15.102:0/1283686532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba80094d0 con 0x7f4bb40ce930 2024-03-23T03:52:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.027+0000 7f4bba200640 1 --2- 172.21.15.102:0/1283686532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4bb4158db0 0x7f4bb414e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:52:01.028 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.027+0000 7f4bb8e00640 1 --2- 172.21.15.102:0/1283686532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bb40ce930 0x7f4bb414e420 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400c450 tx=0x7f4ba400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:01.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.027+0000 7f4bb1600640 1 -- 172.21.15.102:0/1283686532 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba40052d0 con 0x7f4bb40ce930 2024-03-23T03:52:01.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.027+0000 7f4bb1600640 1 -- 172.21.15.102:0/1283686532 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ba4005430 con 0x7f4bb40ce930 2024-03-23T03:52:01.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.027+0000 7f4bb1600640 1 -- 172.21.15.102:0/1283686532 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba4012870 con 0x7f4bb40ce930 2024-03-23T03:52:01.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.027+0000 7f4bbc200640 1 -- 172.21.15.102:0/1283686532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb4149210 con 0x7f4bb40ce930 2024-03-23T03:52:01.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.027+0000 7f4bbc200640 1 -- 172.21.15.102:0/1283686532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb413b560 con 0x7f4bb40ce930 2024-03-23T03:52:01.029 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.028+0000 7f4bbc200640 1 -- 172.21.15.102:0/1283686532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4bb40cc010 con 0x7f4bb40ce930 2024-03-23T03:52:01.030 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.029+0000 7f4bb1600640 1 -- 172.21.15.102:0/1283686532 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4ba40129d0 con 0x7f4bb40ce930 2024-03-23T03:52:01.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.031+0000 7f4bb1600640 1 --2- 172.21.15.102:0/1283686532 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4b9806ac30 0x7f4b9806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.031+0000 7f4bb1600640 1 -- 172.21.15.102:0/1283686532 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f4ba4088cf0 con 0x7f4bb40ce930 2024-03-23T03:52:01.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.031+0000 7f4bb9800640 1 --2- 172.21.15.102:0/1283686532 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4b9806ac30 0x7f4b9806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.031+0000 7f4bb1600640 1 --2- 172.21.15.102:0/1283686532 >> [v2:172.21.15.173:6816/37509,v1:172.21.15.173:6817/37509] conn(0x7f4b98070f30 0x7f4b98073370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.031+0000 7f4bb1600640 1 -- 172.21.15.102:0/1283686532 --> [v2:172.21.15.173:6816/37509,v1:172.21.15.173:6817/37509] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b98073a40 con 0x7f4b98070f30 2024-03-23T03:52:01.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.031+0000 7f4bb1600640 1 -- 172.21.15.102:0/1283686532 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ba408b250 con 0x7f4bb40ce930 2024-03-23T03:52:01.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.032+0000 7f4bb9800640 1 --2- 172.21.15.102:0/1283686532 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4b9806ac30 0x7f4b9806d0f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8036000 tx=0x7f4ba803b040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:01.032 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.032+0000 7f4bba200640 1 --2- 172.21.15.102:0/1283686532 >> [v2:172.21.15.173:6816/37509,v1:172.21.15.173:6817/37509] conn(0x7f4b98070f30 0x7f4b98073370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.033 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.033+0000 7f4bba200640 1 --2- 172.21.15.102:0/1283686532 >> [v2:172.21.15.173:6816/37509,v1:172.21.15.173:6817/37509] conn(0x7f4b98070f30 0x7f4b98073370 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:01.035 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.035+0000 7f4bb1600640 1 -- 172.21.15.102:0/1283686532 <== osd.4 v2:172.21.15.173:6816/37509 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4b98073a40 con 0x7f4b98070f30 2024-03-23T03:52:01.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.048+0000 7f4bbc200640 1 -- 172.21.15.102:0/1283686532 --> [v2:172.21.15.173:6816/37509,v1:172.21.15.173:6817/37509] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4bb40cbd30 con 0x7f4b98070f30 2024-03-23T03:52:01.050 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.050+0000 7f4bb1600640 1 -- 172.21.15.102:0/1283686532 <== osd.4 v2:172.21.15.173:6816/37509 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4bac0019d0 con 0x7f4b98070f30 2024-03-23T03:52:01.050 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:52:01.050 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T03:52:01.051 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:52:01.051 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.050+0000 7f4bbc200640 1 -- 172.21.15.102:0/1283686532 >> [v2:172.21.15.173:6816/37509,v1:172.21.15.173:6817/37509] conn(0x7f4b98070f30 msgr2=0x7f4b98073370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.050+0000 7f4bbc200640 1 --2- 172.21.15.102:0/1283686532 >> [v2:172.21.15.173:6816/37509,v1:172.21.15.173:6817/37509] conn(0x7f4b98070f30 0x7f4b98073370 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.050+0000 7f4bbc200640 1 -- 172.21.15.102:0/1283686532 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4b9806ac30 msgr2=0x7f4b9806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.050+0000 7f4bbc200640 1 --2- 172.21.15.102:0/1283686532 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4b9806ac30 0x7f4b9806d0f0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8036000 tx=0x7f4ba803b040 comp rx=0 tx=0).stop 2024-03-23T03:52:01.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.050+0000 7f4bbc200640 1 -- 172.21.15.102:0/1283686532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bb40ce930 msgr2=0x7f4bb414e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.050+0000 7f4bbc200640 1 --2- 172.21.15.102:0/1283686532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4bb40ce930 0x7f4bb414e420 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400c450 tx=0x7f4ba400c910 comp rx=0 tx=0).stop 2024-03-23T03:52:01.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.050+0000 7f4bba200640 1 -- 172.21.15.102:0/1283686532 reap_dead start 2024-03-23T03:52:01.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.050+0000 7f4bbc200640 1 -- 172.21.15.102:0/1283686532 shutdown_connections 2024-03-23T03:52:01.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.050+0000 7f4bbc200640 1 -- 172.21.15.102:0/1283686532 >> 172.21.15.102:0/1283686532 conn(0x7f4bb406a0b0 msgr2=0x7f4bb4151ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:01.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.051+0000 7f4bbc200640 1 -- 172.21.15.102:0/1283686532 shutdown_connections 2024-03-23T03:52:01.052 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.051+0000 7f4bbc200640 1 -- 172.21.15.102:0/1283686532 wait complete. 2024-03-23T03:52:01.067 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-23T03:52:01.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.272+0000 7f1babe00640 1 Processor -- start 2024-03-23T03:52:01.273 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.272+0000 7f1babe00640 1 -- start start 2024-03-23T03:52:01.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.274+0000 7f1babe00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f1ba4160dd0 0x7f1ba40ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.274+0000 7f1babe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ba40cee70 0x7f1ba41422d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.274+0000 7f1babe00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba406cdc0 con 0x7f1ba413fa80 2024-03-23T03:52:01.274 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.274+0000 7f1babe00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba40cf300 con 0x7f1ba4160dd0 2024-03-23T03:52:01.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.274+0000 7f1babe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba406d060 con 0x7f1ba40cee70 2024-03-23T03:52:01.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.274+0000 7f1ba9400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f1ba413fa80 0x7f1ba413ff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54304/0 (socket says 172.21.15.102:54304) 2024-03-23T03:52:01.275 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.274+0000 7f1ba9400640 1 -- 172.21.15.102:0/4125621715 learned_addr learned my addr 172.21.15.102:0/4125621715 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:52:01.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.274+0000 7f1ba9e00640 1 --2- 172.21.15.102:0/4125621715 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ba40cee70 0x7f1ba41422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.274+0000 7f1ba8a00640 1 --2- 172.21.15.102:0/4125621715 >> v2:172.21.15.173:3300/0 conn(0x7f1ba4160dd0 0x7f1ba40ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.275+0000 7f1ba3e00640 1 -- 172.21.15.102:0/4125621715 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025262754 0 0) 0x7f1ba406cdc0 con 0x7f1ba413fa80 2024-03-23T03:52:01.276 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.275+0000 7f1ba3e00640 1 -- 172.21.15.102:0/4125621715 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b8c003520 con 0x7f1ba413fa80 2024-03-23T03:52:01.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.275+0000 7f1ba3e00640 1 -- 172.21.15.102:0/4125621715 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3594425974 0 0) 0x7f1b8c003520 con 0x7f1ba413fa80 2024-03-23T03:52:01.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.275+0000 7f1ba3e00640 1 -- 172.21.15.102:0/4125621715 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ba40cee70 msgr2=0x7f1ba41422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:01.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.275+0000 7f1ba3e00640 1 --2- 172.21.15.102:0/4125621715 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ba40cee70 0x7f1ba41422d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.275+0000 7f1ba3e00640 1 -- 172.21.15.102:0/4125621715 >> v2:172.21.15.173:3300/0 conn(0x7f1ba4160dd0 msgr2=0x7f1ba40ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:01.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.275+0000 7f1ba3e00640 1 --2- 172.21.15.102:0/4125621715 >> v2:172.21.15.173:3300/0 conn(0x7f1ba4160dd0 0x7f1ba40ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.275+0000 7f1ba3e00640 1 -- 172.21.15.102:0/4125621715 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba406d340 con 0x7f1ba413fa80 2024-03-23T03:52:01.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.275+0000 7f1ba3e00640 1 -- 172.21.15.102:0/4125621715 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f1b98001670 con 0x7f1ba413fa80 2024-03-23T03:52:01.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.275+0000 7f1ba3e00640 1 -- 172.21.15.102:0/4125621715 >> v1:172.21.15.102:6789/0 conn(0x7f1ba413fa80 legacy=0x7f1ba413ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.275+0000 7f1ba3e00640 1 --2- 172.21.15.102:0/4125621715 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ba40cee70 0x7f1b8c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.275+0000 7f1ba3e00640 1 --2- 172.21.15.102:0/4125621715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b8c003f40 0x7f1b8c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.275+0000 7f1ba3e00640 1 --2- 172.21.15.102:0/4125621715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b8c004d00 0x7f1b8c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.275+0000 7f1ba3e00640 1 -- 172.21.15.102:0/4125621715 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba40cf300 con 0x7f1b8c003f40 2024-03-23T03:52:01.277 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.275+0000 7f1ba3e00640 1 -- 172.21.15.102:0/4125621715 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba406d060 con 0x7f1b8c004d00 2024-03-23T03:52:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.275+0000 7f1ba3e00640 1 -- 172.21.15.102:0/4125621715 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c00b980 con 0x7f1ba40cee70 2024-03-23T03:52:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.275+0000 7f1ba8a00640 1 --2- 172.21.15.102:0/4125621715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b8c003f40 0x7f1b8c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.276+0000 7f1ba9e00640 1 --2- 172.21.15.102:0/4125621715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b8c004d00 0x7f1b8c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.276+0000 7f1ba8a00640 1 -- 172.21.15.102:0/4125621715 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ba40cee70 msgr2=0x7f1b8c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.276+0000 7f1ba8a00640 1 --2- 172.21.15.102:0/4125621715 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1ba40cee70 0x7f1b8c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.276+0000 7f1ba8a00640 1 -- 172.21.15.102:0/4125621715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b8c004d00 msgr2=0x7f1b8c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.276+0000 7f1ba8a00640 1 --2- 172.21.15.102:0/4125621715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b8c004d00 0x7f1b8c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.278 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.276+0000 7f1ba8a00640 1 -- 172.21.15.102:0/4125621715 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c003520 con 0x7f1b8c003f40 2024-03-23T03:52:01.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.277+0000 7f1ba8a00640 1 --2- 172.21.15.102:0/4125621715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b8c003f40 0x7f1b8c0047c0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f1b90009af0 tx=0x7f1b8c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bd87c27f41e126e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:01.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.277+0000 7f1ba3e00640 1 -- 172.21.15.102:0/4125621715 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9002c710 con 0x7f1b8c003f40 2024-03-23T03:52:01.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.277+0000 7f1ba3e00640 1 -- 172.21.15.102:0/4125621715 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9002c870 con 0x7f1b8c003f40 2024-03-23T03:52:01.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.277+0000 7f1ba3e00640 1 -- 172.21.15.102:0/4125621715 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9002cc00 con 0x7f1b8c003f40 2024-03-23T03:52:01.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.278+0000 7f1babe00640 1 -- 172.21.15.102:0/4125621715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b8c003f40 msgr2=0x7f1b8c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.278+0000 7f1babe00640 1 --2- 172.21.15.102:0/4125621715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b8c003f40 0x7f1b8c0047c0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f1b90009af0 tx=0x7f1b8c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.278+0000 7f1ba9e00640 1 -- 172.21.15.102:0/4125621715 reap_dead start 2024-03-23T03:52:01.279 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.278+0000 7f1babe00640 1 -- 172.21.15.102:0/4125621715 shutdown_connections 2024-03-23T03:52:01.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.278+0000 7f1babe00640 1 -- 172.21.15.102:0/4125621715 >> 172.21.15.102:0/4125621715 conn(0x7f1ba406a0b0 msgr2=0x7f1ba406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:01.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.278+0000 7f1babe00640 1 -- 172.21.15.102:0/4125621715 shutdown_connections 2024-03-23T03:52:01.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.278+0000 7f1babe00640 1 -- 172.21.15.102:0/4125621715 wait complete. 2024-03-23T03:52:01.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.280+0000 7f1babe00640 1 Processor -- start 2024-03-23T03:52:01.280 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.280+0000 7f1babe00640 1 -- start start 2024-03-23T03:52:01.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.281+0000 7f1babe00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b8c003f40 0x7f1ba41f3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.281+0000 7f1babe00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ba415fbc0 0x7f1ba40b0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.281+0000 7f1babe00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ba41f62a0 0x7f1ba41f66e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.281+0000 7f1babe00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba41f40e0 con 0x7f1ba415fbc0 2024-03-23T03:52:01.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.281+0000 7f1babe00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba41e6160 con 0x7f1ba41f62a0 2024-03-23T03:52:01.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.281+0000 7f1babe00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba41e63b0 con 0x7f1b8c003f40 2024-03-23T03:52:01.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.281+0000 7f1ba8a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ba415fbc0 0x7f1ba40b0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.281+0000 7f1ba8a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ba415fbc0 0x7f1ba40b0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43722/0 (socket says 172.21.15.102:43722) 2024-03-23T03:52:01.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.281+0000 7f1ba8a00640 1 -- 172.21.15.102:0/418714782 learned_addr learned my addr 172.21.15.102:0/418714782 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:01.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.281+0000 7f1ba9400640 1 --2- 172.21.15.102:0/418714782 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b8c003f40 0x7f1ba41f3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.281+0000 7f1ba9e00640 1 --2- 172.21.15.102:0/418714782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ba41f62a0 0x7f1ba41f66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.281+0000 7f1ba8a00640 1 -- 172.21.15.102:0/418714782 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b8c003f40 msgr2=0x7f1ba41f3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.281+0000 7f1ba8a00640 1 --2- 172.21.15.102:0/418714782 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b8c003f40 0x7f1ba41f3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.281+0000 7f1ba8a00640 1 -- 172.21.15.102:0/418714782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ba41f62a0 msgr2=0x7f1ba41f66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.281+0000 7f1ba8a00640 1 --2- 172.21.15.102:0/418714782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ba41f62a0 0x7f1ba41f66e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.281+0000 7f1ba8a00640 1 -- 172.21.15.102:0/418714782 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba41e6690 con 0x7f1ba415fbc0 2024-03-23T03:52:01.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.282+0000 7f1ba8a00640 1 --2- 172.21.15.102:0/418714782 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ba415fbc0 0x7f1ba40b0360 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f1b90037040 tx=0x7f1b90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:01.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.282+0000 7f1ba1600640 1 -- 172.21.15.102:0/418714782 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b90003810 con 0x7f1ba415fbc0 2024-03-23T03:52:01.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.282+0000 7f1ba1600640 1 -- 172.21.15.102:0/418714782 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9002de90 con 0x7f1ba415fbc0 2024-03-23T03:52:01.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.282+0000 7f1ba1600640 1 -- 172.21.15.102:0/418714782 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b90003b70 con 0x7f1ba415fbc0 2024-03-23T03:52:01.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.282+0000 7f1babe00640 1 -- 172.21.15.102:0/418714782 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba41e6970 con 0x7f1ba415fbc0 2024-03-23T03:52:01.284 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.282+0000 7f1babe00640 1 -- 172.21.15.102:0/418714782 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba41e6f30 con 0x7f1ba415fbc0 2024-03-23T03:52:01.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.284+0000 7f1babe00640 1 -- 172.21.15.102:0/418714782 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba4160600 con 0x7f1ba415fbc0 2024-03-23T03:52:01.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.284+0000 7f1ba1600640 1 -- 172.21.15.102:0/418714782 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f1b90003cd0 con 0x7f1ba415fbc0 2024-03-23T03:52:01.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.286+0000 7f1ba1600640 1 --2- 172.21.15.102:0/418714782 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b8406ac00 0x7f1b8406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.286+0000 7f1ba1600640 1 -- 172.21.15.102:0/418714782 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f1b900ab2e0 con 0x7f1ba415fbc0 2024-03-23T03:52:01.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.286+0000 7f1ba9400640 1 --2- 172.21.15.102:0/418714782 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b8406ac00 0x7f1b8406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.287+0000 7f1ba9400640 1 --2- 172.21.15.102:0/418714782 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b8406ac00 0x7f1b8406d0c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1b98001860 tx=0x7f1b98009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:01.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.290+0000 7f1ba1600640 1 -- 172.21.15.102:0/418714782 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f1b90076700 con 0x7f1ba415fbc0 2024-03-23T03:52:01.483 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.482+0000 7f1babe00640 1 -- 172.21.15.102:0/418714782 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ba406dbe0 con 0x7f1b8406ac00 2024-03-23T03:52:01.484 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.483+0000 7f1ba1600640 1 -- 172.21.15.102:0/418714782 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1b98001c50 con 0x7f1b8406ac00 2024-03-23T03:52:01.484 INFO:teuthology.orchestra.run.smithi102.stderr:instructed osd(s) 4 to deep-scrub 2024-03-23T03:52:01.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.486+0000 7f1babe00640 1 -- 172.21.15.102:0/418714782 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b8406ac00 msgr2=0x7f1b8406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.486+0000 7f1babe00640 1 --2- 172.21.15.102:0/418714782 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b8406ac00 0x7f1b8406d0c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f1b98001860 tx=0x7f1b98009040 comp rx=0 tx=0).stop 2024-03-23T03:52:01.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.486+0000 7f1babe00640 1 -- 172.21.15.102:0/418714782 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ba415fbc0 msgr2=0x7f1ba40b0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.486+0000 7f1babe00640 1 --2- 172.21.15.102:0/418714782 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ba415fbc0 0x7f1ba40b0360 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7f1b90037040 tx=0x7f1b90004730 comp rx=0 tx=0).stop 2024-03-23T03:52:01.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.486+0000 7f1babe00640 1 -- 172.21.15.102:0/418714782 shutdown_connections 2024-03-23T03:52:01.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.486+0000 7f1babe00640 1 --2- 172.21.15.102:0/418714782 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f1b8406ac00 0x7f1b8406d0c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.486+0000 7f1babe00640 1 --2- 172.21.15.102:0/418714782 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ba41f62a0 0x7f1ba41f66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.486+0000 7f1babe00640 1 --2- 172.21.15.102:0/418714782 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ba415fbc0 0x7f1ba40b0360 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.486+0000 7f1babe00640 1 --2- 172.21.15.102:0/418714782 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f1b8c003f40 0x7f1ba41f3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.487 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.486+0000 7f1babe00640 1 -- 172.21.15.102:0/418714782 >> 172.21.15.102:0/418714782 conn(0x7f1ba406a0b0 msgr2=0x7f1ba40b2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:01.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.486+0000 7f1babe00640 1 -- 172.21.15.102:0/418714782 shutdown_connections 2024-03-23T03:52:01.488 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.486+0000 7f1babe00640 1 -- 172.21.15.102:0/418714782 wait complete. 2024-03-23T03:52:01.505 INFO:tasks.ceph:Scrubbing osd.5 2024-03-23T03:52:01.505 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-23T03:52:01.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.709+0000 7f4f92200640 1 Processor -- start 2024-03-23T03:52:01.710 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.710+0000 7f4f92200640 1 -- start start 2024-03-23T03:52:01.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.711+0000 7f4f92200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f4f8c0b84b0 0x7f4f8c0d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.711+0000 7f4f92200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c0d1280 con 0x7f4f8c0b5aa0 2024-03-23T03:52:01.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.711+0000 7f4f92200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f8c0d20c0 con 0x7f4f8c0cd740 2024-03-23T03:52:01.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.711+0000 7f4f92200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f8c0b8940 con 0x7f4f8c0b84b0 2024-03-23T03:52:01.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.711+0000 7f4f8be00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f4f8c0b5aa0 0x7f4f8c0b5f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54318/0 (socket says 172.21.15.102:54318) 2024-03-23T03:52:01.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.711+0000 7f4f8be00640 1 -- 172.21.15.102:0/47939 learned_addr learned my addr 172.21.15.102:0/47939 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:52:01.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.712+0000 7f4f8aa00640 1 --2- 172.21.15.102:0/47939 >> v2:172.21.15.173:3300/0 conn(0x7f4f8c0b84b0 0x7f4f8c0d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.712+0000 7f4f8a000640 1 -- 172.21.15.102:0/47939 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827150123 0 0) 0x7f4f8c0d1280 con 0x7f4f8c0b5aa0 2024-03-23T03:52:01.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.712+0000 7f4f8a000640 1 -- 172.21.15.102:0/47939 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f6c003520 con 0x7f4f8c0b5aa0 2024-03-23T03:52:01.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.712+0000 7f4f8a000640 1 -- 172.21.15.102:0/47939 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 306323756 0 0) 0x7f4f6c003520 con 0x7f4f8c0b5aa0 2024-03-23T03:52:01.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.712+0000 7f4f8a000640 1 -- 172.21.15.102:0/47939 >> v2:172.21.15.173:3300/0 conn(0x7f4f8c0b84b0 msgr2=0x7f4f8c0d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:01.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.712+0000 7f4f8a000640 1 --2- 172.21.15.102:0/47939 >> v2:172.21.15.173:3300/0 conn(0x7f4f8c0b84b0 0x7f4f8c0d0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.712+0000 7f4f8a000640 1 -- 172.21.15.102:0/47939 >> v1:172.21.15.173:6789/0 conn(0x7f4f8c0cd740 legacy=0x7f4f8c0cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.712+0000 7f4f8a000640 1 -- 172.21.15.102:0/47939 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c0d14b0 con 0x7f4f8c0b5aa0 2024-03-23T03:52:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.712+0000 7f4f8a000640 1 -- 172.21.15.102:0/47939 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f4f840019f0 con 0x7f4f8c0b5aa0 2024-03-23T03:52:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.712+0000 7f4f8a000640 1 -- 172.21.15.102:0/47939 >> v1:172.21.15.102:6789/0 conn(0x7f4f8c0b5aa0 legacy=0x7f4f8c0b5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.713+0000 7f4f8a000640 1 --2- 172.21.15.102:0/47939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f6c003970 0x7f4f6c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.713+0000 7f4f8a000640 1 --2- 172.21.15.102:0/47939 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f6c0047c0 0x7f4f6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.713+0000 7f4f8a000640 1 --2- 172.21.15.102:0/47939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f6c005180 0x7f4f6c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.713+0000 7f4f8a000640 1 -- 172.21.15.102:0/47939 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c0b8940 con 0x7f4f6c003970 2024-03-23T03:52:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.713+0000 7f4f8a000640 1 -- 172.21.15.102:0/47939 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c00c020 con 0x7f4f6c005180 2024-03-23T03:52:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.713+0000 7f4f8a000640 1 -- 172.21.15.102:0/47939 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c00c1e0 con 0x7f4f6c0047c0 2024-03-23T03:52:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.713+0000 7f4f8b400640 1 --2- 172.21.15.102:0/47939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f6c003970 0x7f4f6c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.713+0000 7f4f8be00640 1 --2- 172.21.15.102:0/47939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f6c005180 0x7f4f6c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.713+0000 7f4f8b400640 1 -- 172.21.15.102:0/47939 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f6c0047c0 msgr2=0x7f4f6c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.713+0000 7f4f8b400640 1 --2- 172.21.15.102:0/47939 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f6c0047c0 0x7f4f6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.713+0000 7f4f8b400640 1 -- 172.21.15.102:0/47939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f6c005180 msgr2=0x7f4f6c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:01.715 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.713+0000 7f4f8b400640 1 --2- 172.21.15.102:0/47939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f6c005180 0x7f4f6c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.713+0000 7f4f8b400640 1 -- 172.21.15.102:0/47939 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6c003520 con 0x7f4f6c003970 2024-03-23T03:52:01.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.713+0000 7f4f8be00640 1 -- 172.21.15.102:0/47939 reap_dead start 2024-03-23T03:52:01.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.714+0000 7f4f8b400640 1 --2- 172.21.15.102:0/47939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f6c003970 0x7f4f6c003d90 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c001750 tx=0x7f4f6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afbdeab6fc043f5b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:01.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.715+0000 7f4f8a000640 1 -- 172.21.15.102:0/47939 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c0052d0 con 0x7f4f6c003970 2024-03-23T03:52:01.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.715+0000 7f4f8a000640 1 -- 172.21.15.102:0/47939 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7c02e050 con 0x7f4f6c003970 2024-03-23T03:52:01.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.715+0000 7f4f8a000640 1 -- 172.21.15.102:0/47939 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c003830 con 0x7f4f6c003970 2024-03-23T03:52:01.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.715+0000 7f4f92200640 1 -- 172.21.15.102:0/47939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f6c003970 msgr2=0x7f4f6c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.715+0000 7f4f92200640 1 --2- 172.21.15.102:0/47939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f6c003970 0x7f4f6c003d90 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c001750 tx=0x7f4f6c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.715+0000 7f4f92200640 1 -- 172.21.15.102:0/47939 shutdown_connections 2024-03-23T03:52:01.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.715+0000 7f4f92200640 1 --2- 172.21.15.102:0/47939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f6c003970 0x7f4f6c003d90 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.715+0000 7f4f92200640 1 -- 172.21.15.102:0/47939 >> 172.21.15.102:0/47939 conn(0x7f4f8c0cf5d0 msgr2=0x7f4f8c0cf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:01.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.715+0000 7f4f92200640 1 -- 172.21.15.102:0/47939 shutdown_connections 2024-03-23T03:52:01.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.715+0000 7f4f92200640 1 -- 172.21.15.102:0/47939 wait complete. 2024-03-23T03:52:01.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.717+0000 7f4f92200640 1 Processor -- start 2024-03-23T03:52:01.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.717+0000 7f4f92200640 1 -- start start 2024-03-23T03:52:01.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.718+0000 7f4f92200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f6c003970 0x7f4f8c14fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.718+0000 7f4f92200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f8c14d340 0x7f4f8c150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.718+0000 7f4f92200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f8c150d70 0x7f4f8c14e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.718+0000 7f4f92200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c1531b0 con 0x7f4f8c14d340 2024-03-23T03:52:01.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.718+0000 7f4f92200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c1502d0 con 0x7f4f6c003970 2024-03-23T03:52:01.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.718+0000 7f4f92200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8c14d800 con 0x7f4f8c150d70 2024-03-23T03:52:01.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.719+0000 7f4f8aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f8c14d340 0x7f4f8c150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.719+0000 7f4f8aa00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f8c14d340 0x7f4f8c150830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43740/0 (socket says 172.21.15.102:43740) 2024-03-23T03:52:01.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.719+0000 7f4f8aa00640 1 -- 172.21.15.102:0/2545872143 learned_addr learned my addr 172.21.15.102:0/2545872143 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:01.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.719+0000 7f4f8aa00640 1 -- 172.21.15.102:0/2545872143 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f8c150d70 msgr2=0x7f4f8c14e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.719+0000 7f4f8aa00640 1 --2- 172.21.15.102:0/2545872143 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4f8c150d70 0x7f4f8c14e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.719+0000 7f4f8aa00640 1 -- 172.21.15.102:0/2545872143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f6c003970 msgr2=0x7f4f8c14fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.719+0000 7f4f8b400640 1 --2- 172.21.15.102:0/2545872143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f6c003970 0x7f4f8c14fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.719+0000 7f4f8aa00640 1 --2- 172.21.15.102:0/2545872143 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4f6c003970 0x7f4f8c14fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.719+0000 7f4f8aa00640 1 -- 172.21.15.102:0/2545872143 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c14dae0 con 0x7f4f8c14d340 2024-03-23T03:52:01.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.719+0000 7f4f8aa00640 1 --2- 172.21.15.102:0/2545872143 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f8c14d340 0x7f4f8c150830 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400e770 tx=0x7f4f7400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:01.721 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.719+0000 7f4f83400640 1 -- 172.21.15.102:0/2545872143 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7400c730 con 0x7f4f8c14d340 2024-03-23T03:52:01.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.720+0000 7f4f83400640 1 -- 172.21.15.102:0/2545872143 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7400c890 con 0x7f4f8c14d340 2024-03-23T03:52:01.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.720+0000 7f4f83400640 1 -- 172.21.15.102:0/2545872143 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7400cc40 con 0x7f4f8c14d340 2024-03-23T03:52:01.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.720+0000 7f4f92200640 1 -- 172.21.15.102:0/2545872143 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f8c14ddc0 con 0x7f4f8c14d340 2024-03-23T03:52:01.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.720+0000 7f4f92200640 1 -- 172.21.15.102:0/2545872143 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f8c14e380 con 0x7f4f8c14d340 2024-03-23T03:52:01.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.721+0000 7f4f92200640 1 -- 172.21.15.102:0/2545872143 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4f8c0d1d50 con 0x7f4f8c14d340 2024-03-23T03:52:01.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.722+0000 7f4f83400640 1 -- 172.21.15.102:0/2545872143 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f4f74013470 con 0x7f4f8c14d340 2024-03-23T03:52:01.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.723+0000 7f4f83400640 1 --2- 172.21.15.102:0/2545872143 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f6806ac00 0x7f4f6806d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.723+0000 7f4f83400640 1 -- 172.21.15.102:0/2545872143 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f4f74089450 con 0x7f4f8c14d340 2024-03-23T03:52:01.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.723+0000 7f4f83400640 1 --2- 172.21.15.102:0/2545872143 >> [v2:172.21.15.173:6800/37507,v1:172.21.15.173:6801/37507] conn(0x7f4f68070f00 0x7f4f68073340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.723+0000 7f4f83400640 1 -- 172.21.15.102:0/2545872143 --> [v2:172.21.15.173:6800/37507,v1:172.21.15.173:6801/37507] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f68073a10 con 0x7f4f68070f00 2024-03-23T03:52:01.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.723+0000 7f4f8b400640 1 --2- 172.21.15.102:0/2545872143 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f6806ac00 0x7f4f6806d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.723+0000 7f4f83400640 1 -- 172.21.15.102:0/2545872143 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f7408b9d0 con 0x7f4f8c14d340 2024-03-23T03:52:01.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.724+0000 7f4f8be00640 1 --2- 172.21.15.102:0/2545872143 >> [v2:172.21.15.173:6800/37507,v1:172.21.15.173:6801/37507] conn(0x7f4f68070f00 0x7f4f68073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.724+0000 7f4f8b400640 1 --2- 172.21.15.102:0/2545872143 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f6806ac00 0x7f4f6806d0c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c033000 tx=0x7f4f7c009030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:01.725 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.725+0000 7f4f8be00640 1 --2- 172.21.15.102:0/2545872143 >> [v2:172.21.15.173:6800/37507,v1:172.21.15.173:6801/37507] conn(0x7f4f68070f00 0x7f4f68073340 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:01.727 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.727+0000 7f4f83400640 1 -- 172.21.15.102:0/2545872143 <== osd.5 v2:172.21.15.173:6800/37507 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f4f68073a10 con 0x7f4f68070f00 2024-03-23T03:52:01.741 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.741+0000 7f4f92200640 1 -- 172.21.15.102:0/2545872143 --> [v2:172.21.15.173:6800/37507,v1:172.21.15.173:6801/37507] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4f8c0d1a70 con 0x7f4f68070f00 2024-03-23T03:52:01.742 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.742+0000 7f4f83400640 1 -- 172.21.15.102:0/2545872143 <== osd.5 v2:172.21.15.173:6800/37507 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4f84001df0 con 0x7f4f68070f00 2024-03-23T03:52:01.743 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:52:01.743 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T03:52:01.743 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:52:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.742+0000 7f4f92200640 1 -- 172.21.15.102:0/2545872143 >> [v2:172.21.15.173:6800/37507,v1:172.21.15.173:6801/37507] conn(0x7f4f68070f00 msgr2=0x7f4f68073340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.742+0000 7f4f92200640 1 --2- 172.21.15.102:0/2545872143 >> [v2:172.21.15.173:6800/37507,v1:172.21.15.173:6801/37507] conn(0x7f4f68070f00 0x7f4f68073340 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.742+0000 7f4f92200640 1 -- 172.21.15.102:0/2545872143 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f6806ac00 msgr2=0x7f4f6806d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.742+0000 7f4f92200640 1 --2- 172.21.15.102:0/2545872143 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f4f6806ac00 0x7f4f6806d0c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c033000 tx=0x7f4f7c009030 comp rx=0 tx=0).stop 2024-03-23T03:52:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.742+0000 7f4f92200640 1 -- 172.21.15.102:0/2545872143 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f8c14d340 msgr2=0x7f4f8c150830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.742+0000 7f4f92200640 1 --2- 172.21.15.102:0/2545872143 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f8c14d340 0x7f4f8c150830 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400e770 tx=0x7f4f7400ec30 comp rx=0 tx=0).stop 2024-03-23T03:52:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.742+0000 7f4f8be00640 1 -- 172.21.15.102:0/2545872143 reap_dead start 2024-03-23T03:52:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.743+0000 7f4f92200640 1 -- 172.21.15.102:0/2545872143 shutdown_connections 2024-03-23T03:52:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.743+0000 7f4f92200640 1 -- 172.21.15.102:0/2545872143 >> 172.21.15.102:0/2545872143 conn(0x7f4f8c0cf5d0 msgr2=0x7f4f8c0b4540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.743+0000 7f4f92200640 1 -- 172.21.15.102:0/2545872143 shutdown_connections 2024-03-23T03:52:01.744 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.743+0000 7f4f92200640 1 -- 172.21.15.102:0/2545872143 wait complete. 2024-03-23T03:52:01.761 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-23T03:52:01.977 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.977+0000 7f620b400640 1 Processor -- start 2024-03-23T03:52:01.978 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.977+0000 7f620b400640 1 -- start start 2024-03-23T03:52:01.979 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.978+0000 7f620b400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f62040ce930 0x7f62040ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.978+0000 7f620b400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f62040cb660 0x7f62040cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.978+0000 7f620b400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6204070170 0x7f6204070600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.978+0000 7f620b400640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f620415bae0 con 0x7f6204070170 2024-03-23T03:52:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.978+0000 7f620b400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62040cf250 con 0x7f62040cb660 2024-03-23T03:52:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.978+0000 7f620b400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f620415ae60 con 0x7f62040ce930 2024-03-23T03:52:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.979+0000 7f6209400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6204070170 0x7f6204070600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.979+0000 7f6209400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f6204070170 0x7f6204070600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43746/0 (socket says 172.21.15.102:43746) 2024-03-23T03:52:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.979+0000 7f6209400640 1 -- 172.21.15.102:0/1903964575 learned_addr learned my addr 172.21.15.102:0/1903964575 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.979+0000 7f6208a00640 1 --2- 172.21.15.102:0/1903964575 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f62040ce930 0x7f62040ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.979+0000 7f61fbe00640 1 --2- 172.21.15.102:0/1903964575 >> v2:172.21.15.173:3300/0 conn(0x7f62040cb660 0x7f62040cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.979+0000 7f6209400640 1 -- 172.21.15.102:0/1903964575 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f62040ce930 msgr2=0x7f62040ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.979+0000 7f6209400640 1 --2- 172.21.15.102:0/1903964575 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f62040ce930 0x7f62040ced10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.979+0000 7f6209400640 1 -- 172.21.15.102:0/1903964575 >> v2:172.21.15.173:3300/0 conn(0x7f62040cb660 msgr2=0x7f62040cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:01.980 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.979+0000 7f6209400640 1 --2- 172.21.15.102:0/1903964575 >> v2:172.21.15.173:3300/0 conn(0x7f62040cb660 0x7f62040cbae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.981 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.979+0000 7f6209400640 1 -- 172.21.15.102:0/1903964575 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620415b140 con 0x7f6204070170 2024-03-23T03:52:01.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.980+0000 7f6209400640 1 --2- 172.21.15.102:0/1903964575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204070170 0x7f6204070600 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f62040b4880 tx=0x7f61fc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91bec54d401e11f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:01.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.980+0000 7f6203e00640 1 -- 172.21.15.102:0/1903964575 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc02ca70 con 0x7f6204070170 2024-03-23T03:52:01.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.980+0000 7f6203e00640 1 -- 172.21.15.102:0/1903964575 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61fc02cbd0 con 0x7f6204070170 2024-03-23T03:52:01.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.980+0000 7f6203e00640 1 -- 172.21.15.102:0/1903964575 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61fc0343f0 con 0x7f6204070170 2024-03-23T03:52:01.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.981+0000 7f620b400640 1 -- 172.21.15.102:0/1903964575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204070170 msgr2=0x7f6204070600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.981+0000 7f620b400640 1 --2- 172.21.15.102:0/1903964575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204070170 0x7f6204070600 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f62040b4880 tx=0x7f61fc02bed0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.981+0000 7f620b400640 1 -- 172.21.15.102:0/1903964575 shutdown_connections 2024-03-23T03:52:01.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.981+0000 7f620b400640 1 --2- 172.21.15.102:0/1903964575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204070170 0x7f6204070600 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.981+0000 7f620b400640 1 --2- 172.21.15.102:0/1903964575 >> v2:172.21.15.173:3300/0 conn(0x7f62040cb660 0x7f62040cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.982 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.981+0000 7f620b400640 1 --2- 172.21.15.102:0/1903964575 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f62040ce930 0x7f62040ced10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.981+0000 7f620b400640 1 -- 172.21.15.102:0/1903964575 >> 172.21.15.102:0/1903964575 conn(0x7f620406a0b0 msgr2=0x7f620406a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:01.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.981+0000 7f620b400640 1 -- 172.21.15.102:0/1903964575 shutdown_connections 2024-03-23T03:52:01.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.981+0000 7f620b400640 1 -- 172.21.15.102:0/1903964575 wait complete. 2024-03-23T03:52:01.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.983+0000 7f620b400640 1 Processor -- start 2024-03-23T03:52:01.983 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.983+0000 7f620b400640 1 -- start start 2024-03-23T03:52:01.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.984+0000 7f620b400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204070170 0x7f6204153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.984+0000 7f620b400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f62040cb660 0x7f620414fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.984+0000 7f620b400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62040ce930 0x7f6204150330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.984+0000 7f620b400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62040b0d50 con 0x7f6204070170 2024-03-23T03:52:01.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.984+0000 7f620b400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62041578a0 con 0x7f62040ce930 2024-03-23T03:52:01.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.984+0000 7f620b400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6204157b80 con 0x7f62040cb660 2024-03-23T03:52:01.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.984+0000 7f6208a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204070170 0x7f6204153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.986 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.984+0000 7f6208a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204070170 0x7f6204153150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43754/0 (socket says 172.21.15.102:43754) 2024-03-23T03:52:01.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.984+0000 7f6208a00640 1 -- 172.21.15.102:0/2058226532 learned_addr learned my addr 172.21.15.102:0/2058226532 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:01.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.985+0000 7f6209400640 1 --2- 172.21.15.102:0/2058226532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62040ce930 0x7f6204150330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.985+0000 7f61fbe00640 1 --2- 172.21.15.102:0/2058226532 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f62040cb660 0x7f620414fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.985+0000 7f6208a00640 1 -- 172.21.15.102:0/2058226532 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f62040cb660 msgr2=0x7f620414fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.985+0000 7f6208a00640 1 --2- 172.21.15.102:0/2058226532 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f62040cb660 0x7f620414fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.985+0000 7f6208a00640 1 -- 172.21.15.102:0/2058226532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62040ce930 msgr2=0x7f6204150330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:01.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.985+0000 7f6208a00640 1 --2- 172.21.15.102:0/2058226532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62040ce930 0x7f6204150330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:01.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.985+0000 7f6208a00640 1 -- 172.21.15.102:0/2058226532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc009b30 con 0x7f6204070170 2024-03-23T03:52:01.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.985+0000 7f6209400640 1 --2- 172.21.15.102:0/2058226532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62040ce930 0x7f6204150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:52:01.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.985+0000 7f61fbe00640 1 --2- 172.21.15.102:0/2058226532 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f62040cb660 0x7f620414fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:52:01.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.985+0000 7f6208a00640 1 --2- 172.21.15.102:0/2058226532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204070170 0x7f6204153150 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00c7f0 tx=0x7f61ec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:01.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.985+0000 7f6201600640 1 -- 172.21.15.102:0/2058226532 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ec004220 con 0x7f6204070170 2024-03-23T03:52:01.987 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.985+0000 7f6201600640 1 -- 172.21.15.102:0/2058226532 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ec0043a0 con 0x7f6204070170 2024-03-23T03:52:01.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.986+0000 7f620b400640 1 -- 172.21.15.102:0/2058226532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6204155280 con 0x7f6204070170 2024-03-23T03:52:01.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.986+0000 7f6201600640 1 -- 172.21.15.102:0/2058226532 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ec007860 con 0x7f6204070170 2024-03-23T03:52:01.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.986+0000 7f620b400640 1 -- 172.21.15.102:0/2058226532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6204158140 con 0x7f6204070170 2024-03-23T03:52:01.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.987+0000 7f620b400640 1 -- 172.21.15.102:0/2058226532 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6204073390 con 0x7f6204070170 2024-03-23T03:52:01.988 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.988+0000 7f6201600640 1 -- 172.21.15.102:0/2058226532 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f61ec007bd0 con 0x7f6204070170 2024-03-23T03:52:01.989 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.989+0000 7f6201600640 1 --2- 172.21.15.102:0/2058226532 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f61e006ac00 0x7f61e006d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:01.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.989+0000 7f6201600640 1 -- 172.21.15.102:0/2058226532 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f61ec089150 con 0x7f6204070170 2024-03-23T03:52:01.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.989+0000 7f61fbe00640 1 --2- 172.21.15.102:0/2058226532 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f61e006ac00 0x7f61e006d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:01.990 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.990+0000 7f61fbe00640 1 --2- 172.21.15.102:0/2058226532 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f61e006ac00 0x7f61e006d0c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f62040ae9f0 tx=0x7f61f4008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:01.994 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:01.994+0000 7f6201600640 1 -- 172.21.15.102:0/2058226532 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f61ec0544e0 con 0x7f6204070170 2024-03-23T03:52:02.186 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.185+0000 7f620b400640 1 -- 172.21.15.102:0/2058226532 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f6204155100 con 0x7f61e006ac00 2024-03-23T03:52:02.187 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.186+0000 7f6201600640 1 -- 172.21.15.102:0/2058226532 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f61f400c040 con 0x7f61e006ac00 2024-03-23T03:52:02.187 INFO:teuthology.orchestra.run.smithi102.stderr:instructed osd(s) 5 to deep-scrub 2024-03-23T03:52:02.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.189+0000 7f620b400640 1 -- 172.21.15.102:0/2058226532 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f61e006ac00 msgr2=0x7f61e006d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:02.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.189+0000 7f620b400640 1 --2- 172.21.15.102:0/2058226532 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f61e006ac00 0x7f61e006d0c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f62040ae9f0 tx=0x7f61f4008040 comp rx=0 tx=0).stop 2024-03-23T03:52:02.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.189+0000 7f620b400640 1 -- 172.21.15.102:0/2058226532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204070170 msgr2=0x7f6204153150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:02.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.189+0000 7f620b400640 1 --2- 172.21.15.102:0/2058226532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204070170 0x7f6204153150 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f61ec00c7f0 tx=0x7f61ec00ccb0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.189+0000 7f620b400640 1 -- 172.21.15.102:0/2058226532 shutdown_connections 2024-03-23T03:52:02.190 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.189+0000 7f620b400640 1 --2- 172.21.15.102:0/2058226532 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f61e006ac00 0x7f61e006d0c0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.189+0000 7f620b400640 1 --2- 172.21.15.102:0/2058226532 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f62040ce930 0x7f6204150330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.189+0000 7f620b400640 1 --2- 172.21.15.102:0/2058226532 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f62040cb660 0x7f620414fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.189+0000 7f620b400640 1 --2- 172.21.15.102:0/2058226532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6204070170 0x7f6204153150 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.189+0000 7f620b400640 1 -- 172.21.15.102:0/2058226532 >> 172.21.15.102:0/2058226532 conn(0x7f620406a0b0 msgr2=0x7f620415c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:02.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.189+0000 7f620b400640 1 -- 172.21.15.102:0/2058226532 shutdown_connections 2024-03-23T03:52:02.191 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.189+0000 7f620b400640 1 -- 172.21.15.102:0/2058226532 wait complete. 2024-03-23T03:52:02.209 INFO:tasks.ceph:Scrubbing osd.6 2024-03-23T03:52:02.209 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-23T03:52:02.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.427+0000 7ff51e200640 1 Processor -- start 2024-03-23T03:52:02.428 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.427+0000 7ff51e200640 1 -- start start 2024-03-23T03:52:02.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.429+0000 7ff51e200640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff51806ce90 0x7ff51806d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:02.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.429+0000 7ff51e200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5180cdba0 con 0x7ff5180ce930 2024-03-23T03:52:02.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.429+0000 7ff51e200640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5180cde80 con 0x7ff5180add30 2024-03-23T03:52:02.430 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.429+0000 7ff51e200640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5180ae1c0 con 0x7ff51806ce90 2024-03-23T03:52:02.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.430+0000 7ff517e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff51806ce90 0x7ff51806d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:02.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.430+0000 7ff517400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7ff5180ce930 0x7ff5180cd3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:54330/0 (socket says 172.21.15.102:54330) 2024-03-23T03:52:02.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.430+0000 7ff517e00640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7ff51806ce90 0x7ff51806d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43758/0 (socket says 172.21.15.102:43758) 2024-03-23T03:52:02.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.430+0000 7ff517e00640 1 -- 172.21.15.102:0/800238741 learned_addr learned my addr 172.21.15.102:0/800238741 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:02.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.430+0000 7ff517e00640 1 -- 172.21.15.102:0/800238741 >> v1:172.21.15.173:6789/0 conn(0x7ff5180add30 legacy=0x7ff5180ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:02.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.430+0000 7ff517e00640 1 -- 172.21.15.102:0/800238741 >> v1:172.21.15.102:6789/0 conn(0x7ff5180ce930 legacy=0x7ff5180cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:02.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.430+0000 7ff517e00640 1 -- 172.21.15.102:0/800238741 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5180ce160 con 0x7ff51806ce90 2024-03-23T03:52:02.431 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.431+0000 7ff517e00640 1 --2- 172.21.15.102:0/800238741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff51806ce90 0x7ff51806d300 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7ff510009e50 tx=0x7ff51002bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=114123c5d222fc96 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:02.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.431+0000 7ff516a00640 1 -- 172.21.15.102:0/800238741 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51002cb90 con 0x7ff51806ce90 2024-03-23T03:52:02.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.432+0000 7ff516a00640 1 -- 172.21.15.102:0/800238741 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51002ccf0 con 0x7ff51806ce90 2024-03-23T03:52:02.432 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.432+0000 7ff516a00640 1 -- 172.21.15.102:0/800238741 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff510034610 con 0x7ff51806ce90 2024-03-23T03:52:02.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.432+0000 7ff51e200640 1 -- 172.21.15.102:0/800238741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff51806ce90 msgr2=0x7ff51806d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:02.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.432+0000 7ff51e200640 1 --2- 172.21.15.102:0/800238741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff51806ce90 0x7ff51806d300 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7ff510009e50 tx=0x7ff51002bff0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.432+0000 7ff51e200640 1 -- 172.21.15.102:0/800238741 shutdown_connections 2024-03-23T03:52:02.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.432+0000 7ff51e200640 1 --2- 172.21.15.102:0/800238741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff51806ce90 0x7ff51806d300 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.432+0000 7ff51e200640 1 -- 172.21.15.102:0/800238741 >> 172.21.15.102:0/800238741 conn(0x7ff51806a0b0 msgr2=0x7ff51806a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:02.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.432+0000 7ff51e200640 1 -- 172.21.15.102:0/800238741 shutdown_connections 2024-03-23T03:52:02.433 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.432+0000 7ff51e200640 1 -- 172.21.15.102:0/800238741 wait complete. 2024-03-23T03:52:02.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.434+0000 7ff51e200640 1 Processor -- start 2024-03-23T03:52:02.434 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.434+0000 7ff51e200640 1 -- start start 2024-03-23T03:52:02.436 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.435+0000 7ff51e200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff51806ce90 0x7ff518147270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:02.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.435+0000 7ff51e200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff5180add30 0x7ff518151800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:02.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.435+0000 7ff51e200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5180ce930 0x7ff518156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:02.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.435+0000 7ff51e200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff518157190 con 0x7ff5180ce930 2024-03-23T03:52:02.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.435+0000 7ff51e200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5181f1980 con 0x7ff51806ce90 2024-03-23T03:52:02.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.435+0000 7ff51e200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5181f2500 con 0x7ff5180add30 2024-03-23T03:52:02.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.435+0000 7ff517e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5180ce930 0x7ff518156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:02.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.435+0000 7ff517e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5180ce930 0x7ff518156c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43766/0 (socket says 172.21.15.102:43766) 2024-03-23T03:52:02.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.435+0000 7ff517e00640 1 -- 172.21.15.102:0/2115464714 learned_addr learned my addr 172.21.15.102:0/2115464714 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:02.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.436+0000 7ff517400640 1 --2- 172.21.15.102:0/2115464714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff51806ce90 0x7ff518147270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:02.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.436+0000 7ff50ea00640 1 --2- 172.21.15.102:0/2115464714 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff5180add30 0x7ff518151800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:02.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.436+0000 7ff517e00640 1 -- 172.21.15.102:0/2115464714 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff5180add30 msgr2=0x7ff518151800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:02.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.436+0000 7ff517e00640 1 --2- 172.21.15.102:0/2115464714 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7ff5180add30 0x7ff518151800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.436+0000 7ff517e00640 1 -- 172.21.15.102:0/2115464714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff51806ce90 msgr2=0x7ff518147270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:02.437 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.436+0000 7ff517e00640 1 --2- 172.21.15.102:0/2115464714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff51806ce90 0x7ff518147270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.436+0000 7ff517e00640 1 -- 172.21.15.102:0/2115464714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff510009b30 con 0x7ff5180ce930 2024-03-23T03:52:02.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.436+0000 7ff517400640 1 --2- 172.21.15.102:0/2115464714 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff51806ce90 0x7ff518147270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:52:02.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.436+0000 7ff517e00640 1 --2- 172.21.15.102:0/2115464714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5180ce930 0x7ff518156c50 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7ff510036040 tx=0x7ff510004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:02.438 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.436+0000 7ff50fe00640 1 -- 172.21.15.102:0/2115464714 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff510009510 con 0x7ff5180ce930 2024-03-23T03:52:02.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.436+0000 7ff50fe00640 1 -- 172.21.15.102:0/2115464714 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51003e450 con 0x7ff5180ce930 2024-03-23T03:52:02.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.436+0000 7ff50fe00640 1 -- 172.21.15.102:0/2115464714 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51003e800 con 0x7ff5180ce930 2024-03-23T03:52:02.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.437+0000 7ff51e200640 1 -- 172.21.15.102:0/2115464714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5181f1bd0 con 0x7ff5180ce930 2024-03-23T03:52:02.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.437+0000 7ff51e200640 1 -- 172.21.15.102:0/2115464714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5181f2190 con 0x7ff5180ce930 2024-03-23T03:52:02.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.437+0000 7ff51e200640 1 -- 172.21.15.102:0/2115464714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff51815df90 con 0x7ff5180ce930 2024-03-23T03:52:02.439 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.439+0000 7ff50fe00640 1 -- 172.21.15.102:0/2115464714 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7ff51003e960 con 0x7ff5180ce930 2024-03-23T03:52:02.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.439+0000 7ff50fe00640 1 --2- 172.21.15.102:0/2115464714 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff4f406ac00 0x7ff4f406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:02.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.440+0000 7ff50fe00640 1 -- 172.21.15.102:0/2115464714 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7ff5100abeb0 con 0x7ff5180ce930 2024-03-23T03:52:02.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.440+0000 7ff50fe00640 1 --2- 172.21.15.102:0/2115464714 >> [v2:172.21.15.173:6808/37508,v1:172.21.15.173:6809/37508] conn(0x7ff4f4070f00 0x7ff4f4073340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:02.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.440+0000 7ff517400640 1 --2- 172.21.15.102:0/2115464714 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff4f406ac00 0x7ff4f406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:02.440 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.440+0000 7ff50fe00640 1 -- 172.21.15.102:0/2115464714 --> [v2:172.21.15.173:6808/37508,v1:172.21.15.173:6809/37508] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4f4073a10 con 0x7ff4f4070f00 2024-03-23T03:52:02.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.440+0000 7ff50fe00640 1 -- 172.21.15.102:0/2115464714 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5100ae490 con 0x7ff5180ce930 2024-03-23T03:52:02.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.440+0000 7ff517400640 1 --2- 172.21.15.102:0/2115464714 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff4f406ac00 0x7ff4f406d0c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff508000f60 tx=0x7ff508009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:02.441 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.440+0000 7ff50ea00640 1 --2- 172.21.15.102:0/2115464714 >> [v2:172.21.15.173:6808/37508,v1:172.21.15.173:6809/37508] conn(0x7ff4f4070f00 0x7ff4f4073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:02.442 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.441+0000 7ff50ea00640 1 --2- 172.21.15.102:0/2115464714 >> [v2:172.21.15.173:6808/37508,v1:172.21.15.173:6809/37508] conn(0x7ff4f4070f00 0x7ff4f4073340 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:02.444 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.443+0000 7ff50fe00640 1 -- 172.21.15.102:0/2115464714 <== osd.6 v2:172.21.15.173:6808/37508 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7ff4f4073a10 con 0x7ff4f4070f00 2024-03-23T03:52:02.457 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.457+0000 7ff51e200640 1 -- 172.21.15.102:0/2115464714 --> [v2:172.21.15.173:6808/37508,v1:172.21.15.173:6809/37508] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff5180b42d0 con 0x7ff4f4070f00 2024-03-23T03:52:02.458 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.458+0000 7ff50fe00640 1 -- 172.21.15.102:0/2115464714 <== osd.6 v2:172.21.15.173:6808/37508 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff500001770 con 0x7ff4f4070f00 2024-03-23T03:52:02.459 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:52:02.459 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T03:52:02.459 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:52:02.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.458+0000 7ff51e200640 1 -- 172.21.15.102:0/2115464714 >> [v2:172.21.15.173:6808/37508,v1:172.21.15.173:6809/37508] conn(0x7ff4f4070f00 msgr2=0x7ff4f4073340 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:02.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.458+0000 7ff51e200640 1 --2- 172.21.15.102:0/2115464714 >> [v2:172.21.15.173:6808/37508,v1:172.21.15.173:6809/37508] conn(0x7ff4f4070f00 0x7ff4f4073340 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.458+0000 7ff51e200640 1 -- 172.21.15.102:0/2115464714 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff4f406ac00 msgr2=0x7ff4f406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:02.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.458+0000 7ff51e200640 1 --2- 172.21.15.102:0/2115464714 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7ff4f406ac00 0x7ff4f406d0c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff508000f60 tx=0x7ff508009040 comp rx=0 tx=0).stop 2024-03-23T03:52:02.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.458+0000 7ff51e200640 1 -- 172.21.15.102:0/2115464714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5180ce930 msgr2=0x7ff518156c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:02.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.458+0000 7ff51e200640 1 --2- 172.21.15.102:0/2115464714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5180ce930 0x7ff518156c50 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7ff510036040 tx=0x7ff510004a40 comp rx=0 tx=0).stop 2024-03-23T03:52:02.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.458+0000 7ff517e00640 1 -- 172.21.15.102:0/2115464714 reap_dead start 2024-03-23T03:52:02.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.459+0000 7ff51e200640 1 -- 172.21.15.102:0/2115464714 shutdown_connections 2024-03-23T03:52:02.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.459+0000 7ff51e200640 1 -- 172.21.15.102:0/2115464714 >> 172.21.15.102:0/2115464714 conn(0x7ff51806a0b0 msgr2=0x7ff51815c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:02.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.459+0000 7ff51e200640 1 -- 172.21.15.102:0/2115464714 shutdown_connections 2024-03-23T03:52:02.460 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.459+0000 7ff51e200640 1 -- 172.21.15.102:0/2115464714 wait complete. 2024-03-23T03:52:02.480 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-23T03:52:02.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.688+0000 7f46fd800640 1 Processor -- start 2024-03-23T03:52:02.689 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.688+0000 7f46fd800640 1 -- start start 2024-03-23T03:52:02.690 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.690+0000 7f46fd800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f46f80b4b20 0x7f46f80d0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:02.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.690+0000 7f46fd800640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f46f80b8470 0x7f46f80b5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:02.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.690+0000 7f46fd800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46f80d1460 con 0x7f46f815cba0 2024-03-23T03:52:02.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.690+0000 7f46fd800640 1 -- --> v2:172.21.15.102:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46f815cf80 con 0x7f46f80b8470 2024-03-23T03:52:02.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.690+0000 7f46fd800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46f80d1740 con 0x7f46f80b4b20 2024-03-23T03:52:02.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.690+0000 7f46f7400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f46f80b8470 0x7f46f80b5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:02.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.690+0000 7f46f7400640 1 --2- >> v2:172.21.15.102:3300/0 conn(0x7f46f80b8470 0x7f46f80b5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43772/0 (socket says 172.21.15.102:43772) 2024-03-23T03:52:02.691 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.690+0000 7f46f7400640 1 -- 172.21.15.102:0/3808076793 learned_addr learned my addr 172.21.15.102:0/3808076793 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:02.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.691+0000 7f46ee000640 1 --2- 172.21.15.102:0/3808076793 >> v2:172.21.15.173:3300/0 conn(0x7f46f80b4b20 0x7f46f80d0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:02.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.691+0000 7f46f7400640 1 -- 172.21.15.102:0/3808076793 >> v2:172.21.15.173:3300/0 conn(0x7f46f80b4b20 msgr2=0x7f46f80d0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:02.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.691+0000 7f46f7400640 1 --2- 172.21.15.102:0/3808076793 >> v2:172.21.15.173:3300/0 conn(0x7f46f80b4b20 0x7f46f80d0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.691+0000 7f46f7400640 1 -- 172.21.15.102:0/3808076793 >> v1:172.21.15.173:6789/0 conn(0x7f46f815cba0 legacy=0x7f46f80b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:02.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.691+0000 7f46f7400640 1 -- 172.21.15.102:0/3808076793 --> v2:172.21.15.102:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f80d1a20 con 0x7f46f80b8470 2024-03-23T03:52:02.692 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.692+0000 7f46f7400640 1 --2- 172.21.15.102:0/3808076793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f80b8470 0x7f46f80b5ac0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f46f80ccf70 tx=0x7f46f002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d602f44c5fe12fa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:02.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.692+0000 7f46f6000640 1 -- 172.21.15.102:0/3808076793 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f0030860 con 0x7f46f80b8470 2024-03-23T03:52:02.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.693+0000 7f46f6000640 1 -- 172.21.15.102:0/3808076793 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f00309c0 con 0x7f46f80b8470 2024-03-23T03:52:02.693 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.693+0000 7f46f6000640 1 -- 172.21.15.102:0/3808076793 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f0030d50 con 0x7f46f80b8470 2024-03-23T03:52:02.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.693+0000 7f46fd800640 1 -- 172.21.15.102:0/3808076793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f80b8470 msgr2=0x7f46f80b5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:02.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.693+0000 7f46fd800640 1 --2- 172.21.15.102:0/3808076793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f80b8470 0x7f46f80b5ac0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f46f80ccf70 tx=0x7f46f002b9a0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.693+0000 7f46fd800640 1 -- 172.21.15.102:0/3808076793 shutdown_connections 2024-03-23T03:52:02.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.693+0000 7f46fd800640 1 --2- 172.21.15.102:0/3808076793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f80b8470 0x7f46f80b5ac0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.693+0000 7f46fd800640 1 --2- 172.21.15.102:0/3808076793 >> v2:172.21.15.173:3300/0 conn(0x7f46f80b4b20 0x7f46f80d0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.693+0000 7f46fd800640 1 -- 172.21.15.102:0/3808076793 >> 172.21.15.102:0/3808076793 conn(0x7f46f80cf5f0 msgr2=0x7f46f80cfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:02.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.693+0000 7f46fd800640 1 -- 172.21.15.102:0/3808076793 shutdown_connections 2024-03-23T03:52:02.694 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.693+0000 7f46fd800640 1 -- 172.21.15.102:0/3808076793 wait complete. 2024-03-23T03:52:02.695 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.695+0000 7f46fd800640 1 Processor -- start 2024-03-23T03:52:02.696 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.695+0000 7f46fd800640 1 -- start start 2024-03-23T03:52:02.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.696+0000 7f46fd800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f80b4b20 0x7f46f815ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:02.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.696+0000 7f46fd800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46f80b8470 0x7f46f80c6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:02.697 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.696+0000 7f46fd800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46f815cba0 0x7f46f80cba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:02.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.696+0000 7f46fd800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f80c7890 con 0x7f46f80b4b20 2024-03-23T03:52:02.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.696+0000 7f46fd800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46f80c7b70 con 0x7f46f815cba0 2024-03-23T03:52:02.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.696+0000 7f46fd800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46f80c7e50 con 0x7f46f80b8470 2024-03-23T03:52:02.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.696+0000 7f46f6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f80b4b20 0x7f46f815ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:02.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.696+0000 7f46f6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f80b4b20 0x7f46f815ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:43774/0 (socket says 172.21.15.102:43774) 2024-03-23T03:52:02.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.697+0000 7f46f6a00640 1 -- 172.21.15.102:0/1732021395 learned_addr learned my addr 172.21.15.102:0/1732021395 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:02.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.697+0000 7f46f6a00640 1 -- 172.21.15.102:0/1732021395 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46f80b8470 msgr2=0x7f46f80c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:02.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.697+0000 7f46f6a00640 1 --2- 172.21.15.102:0/1732021395 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46f80b8470 0x7f46f80c6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.697+0000 7f46f6a00640 1 -- 172.21.15.102:0/1732021395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46f815cba0 msgr2=0x7f46f80cba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:02.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.697+0000 7f46f6a00640 1 --2- 172.21.15.102:0/1732021395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46f815cba0 0x7f46f80cba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.697+0000 7f46f6a00640 1 -- 172.21.15.102:0/1732021395 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f00094d0 con 0x7f46f80b4b20 2024-03-23T03:52:02.698 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.697+0000 7f46f6a00640 1 --2- 172.21.15.102:0/1732021395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f80b4b20 0x7f46f815ac80 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f46e800c870 tx=0x7f46e800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:02.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.697+0000 7f46ef400640 1 -- 172.21.15.102:0/1732021395 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e8003b70 con 0x7f46f80b4b20 2024-03-23T03:52:02.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.697+0000 7f46ef400640 1 -- 172.21.15.102:0/1732021395 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e8003cf0 con 0x7f46f80b4b20 2024-03-23T03:52:02.699 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.697+0000 7f46ef400640 1 -- 172.21.15.102:0/1732021395 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e800aa90 con 0x7f46f80b4b20 2024-03-23T03:52:02.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.697+0000 7f46fd800640 1 -- 172.21.15.102:0/1732021395 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46f80c8130 con 0x7f46f80b4b20 2024-03-23T03:52:02.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.698+0000 7f46fd800640 1 -- 172.21.15.102:0/1732021395 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46f80c86f0 con 0x7f46f80b4b20 2024-03-23T03:52:02.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.699+0000 7f46fd800640 1 -- 172.21.15.102:0/1732021395 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46f8073390 con 0x7f46f80b4b20 2024-03-23T03:52:02.700 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.699+0000 7f46ef400640 1 -- 172.21.15.102:0/1732021395 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f46e800abf0 con 0x7f46f80b4b20 2024-03-23T03:52:02.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.700+0000 7f46ef400640 1 --2- 172.21.15.102:0/1732021395 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f46d406ac00 0x7f46d406d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:02.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.700+0000 7f46ef400640 1 -- 172.21.15.102:0/1732021395 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f46e80890b0 con 0x7f46f80b4b20 2024-03-23T03:52:02.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.701+0000 7f46ee000640 1 --2- 172.21.15.102:0/1732021395 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f46d406ac00 0x7f46d406d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:02.701 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.701+0000 7f46ee000640 1 --2- 172.21.15.102:0/1732021395 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f46d406ac00 0x7f46d406d0c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f46e0006fd0 tx=0x7f46e0008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:02.705 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.705+0000 7f46ef400640 1 -- 172.21.15.102:0/1732021395 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f46e8054450 con 0x7f46f80b4b20 2024-03-23T03:52:02.899 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.899+0000 7f46fd800640 1 -- 172.21.15.102:0/1732021395 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f46f80b88a0 con 0x7f46d406ac00 2024-03-23T03:52:02.900 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.900+0000 7f46ef400640 1 -- 172.21.15.102:0/1732021395 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f46e00090c0 con 0x7f46d406ac00 2024-03-23T03:52:02.900 INFO:teuthology.orchestra.run.smithi102.stderr:instructed osd(s) 6 to deep-scrub 2024-03-23T03:52:02.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.902+0000 7f46fd800640 1 -- 172.21.15.102:0/1732021395 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f46d406ac00 msgr2=0x7f46d406d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:02.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.902+0000 7f46fd800640 1 --2- 172.21.15.102:0/1732021395 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f46d406ac00 0x7f46d406d0c0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f46e0006fd0 tx=0x7f46e0008040 comp rx=0 tx=0).stop 2024-03-23T03:52:02.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.902+0000 7f46fd800640 1 -- 172.21.15.102:0/1732021395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f80b4b20 msgr2=0x7f46f815ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:02.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.902+0000 7f46fd800640 1 --2- 172.21.15.102:0/1732021395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f80b4b20 0x7f46f815ac80 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f46e800c870 tx=0x7f46e800cd30 comp rx=0 tx=0).stop 2024-03-23T03:52:02.903 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.902+0000 7f46fd800640 1 -- 172.21.15.102:0/1732021395 shutdown_connections 2024-03-23T03:52:02.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.902+0000 7f46fd800640 1 --2- 172.21.15.102:0/1732021395 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f46d406ac00 0x7f46d406d0c0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.902+0000 7f46fd800640 1 --2- 172.21.15.102:0/1732021395 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46f815cba0 0x7f46f80cba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.902+0000 7f46fd800640 1 --2- 172.21.15.102:0/1732021395 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f46f80b8470 0x7f46f80c6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.902+0000 7f46fd800640 1 --2- 172.21.15.102:0/1732021395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f46f80b4b20 0x7f46f815ac80 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:02.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.902+0000 7f46fd800640 1 -- 172.21.15.102:0/1732021395 >> 172.21.15.102:0/1732021395 conn(0x7f46f80cf5f0 msgr2=0x7f46f80cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:02.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.903+0000 7f46fd800640 1 -- 172.21.15.102:0/1732021395 shutdown_connections 2024-03-23T03:52:02.904 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:02.903+0000 7f46fd800640 1 -- 172.21.15.102:0/1732021395 wait complete. 2024-03-23T03:52:02.922 INFO:tasks.ceph:Scrubbing osd.7 2024-03-23T03:52:02.922 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-23T03:52:03.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.131+0000 7f0d42800640 1 Processor -- start 2024-03-23T03:52:03.132 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.132+0000 7f0d42800640 1 -- start start 2024-03-23T03:52:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.133+0000 7f0d42800640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f0d3c0add30 0x7f0d3c0ced10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.133+0000 7f0d42800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0d3c06ce90 0x7f0d3c06d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.133+0000 7f0d42800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d3c0cf250 con 0x7f0d3c0ce930 2024-03-23T03:52:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.133+0000 7f0d42800640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d3c0ae1c0 con 0x7f0d3c0add30 2024-03-23T03:52:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.133+0000 7f0d42800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c06c760 con 0x7f0d3c06ce90 2024-03-23T03:52:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.133+0000 7f0d3b400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f0d3c0ce930 0x7f0d3c0cd3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33982/0 (socket says 172.21.15.102:33982) 2024-03-23T03:52:03.134 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.134+0000 7f0d3b400640 1 -- 172.21.15.102:0/1444701296 learned_addr learned my addr 172.21.15.102:0/1444701296 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:52:03.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.134+0000 7f0d3aa00640 1 -- 172.21.15.102:0/1444701296 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968276424 0 0) 0x7f0d3c0cf250 con 0x7f0d3c0ce930 2024-03-23T03:52:03.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.134+0000 7f0d3be00640 1 --2- 172.21.15.102:0/1444701296 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0d3c06ce90 0x7f0d3c06d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.136 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.134+0000 7f0d3aa00640 1 -- 172.21.15.102:0/1444701296 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d20003520 con 0x7f0d3c0ce930 2024-03-23T03:52:03.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.134+0000 7f0d32a00640 1 --2- 172.21.15.102:0/1444701296 >> v2:172.21.15.173:3300/0 conn(0x7f0d3c0add30 0x7f0d3c0ced10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.134+0000 7f0d3aa00640 1 -- 172.21.15.102:0/1444701296 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 956023882 0 0) 0x7f0d20003520 con 0x7f0d3c0ce930 2024-03-23T03:52:03.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.134+0000 7f0d3aa00640 1 -- 172.21.15.102:0/1444701296 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0d3c06ce90 msgr2=0x7f0d3c06d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:03.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.134+0000 7f0d3aa00640 1 --2- 172.21.15.102:0/1444701296 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0d3c06ce90 0x7f0d3c06d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.134+0000 7f0d3aa00640 1 -- 172.21.15.102:0/1444701296 >> v2:172.21.15.173:3300/0 conn(0x7f0d3c0add30 msgr2=0x7f0d3c0ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:03.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.134+0000 7f0d3aa00640 1 --2- 172.21.15.102:0/1444701296 >> v2:172.21.15.173:3300/0 conn(0x7f0d3c0add30 0x7f0d3c0ced10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.134+0000 7f0d3aa00640 1 -- 172.21.15.102:0/1444701296 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3c0cdba0 con 0x7f0d3c0ce930 2024-03-23T03:52:03.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.135+0000 7f0d3aa00640 1 -- 172.21.15.102:0/1444701296 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f0d2c001670 con 0x7f0d3c0ce930 2024-03-23T03:52:03.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.135+0000 7f0d3aa00640 1 -- 172.21.15.102:0/1444701296 >> v1:172.21.15.102:6789/0 conn(0x7f0d3c0ce930 legacy=0x7f0d3c0cd3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.135+0000 7f0d3aa00640 1 --2- 172.21.15.102:0/1444701296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d20003970 0x7f0d20003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.135+0000 7f0d3aa00640 1 --2- 172.21.15.102:0/1444701296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d200047c0 0x7f0d20004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.135+0000 7f0d3aa00640 1 --2- 172.21.15.102:0/1444701296 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0d3c06ce90 0x7f0d20005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.137 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.135+0000 7f0d3aa00640 1 -- 172.21.15.102:0/1444701296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c06c760 con 0x7f0d20003970 2024-03-23T03:52:03.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.135+0000 7f0d3aa00640 1 -- 172.21.15.102:0/1444701296 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2000b980 con 0x7f0d200047c0 2024-03-23T03:52:03.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.135+0000 7f0d3aa00640 1 -- 172.21.15.102:0/1444701296 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2000bac0 con 0x7f0d3c06ce90 2024-03-23T03:52:03.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.135+0000 7f0d3b400640 1 --2- 172.21.15.102:0/1444701296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d20003970 0x7f0d20003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.135+0000 7f0d3b400640 1 -- 172.21.15.102:0/1444701296 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0d3c06ce90 msgr2=0x7f0d20005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:03.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.135+0000 7f0d3b400640 1 --2- 172.21.15.102:0/1444701296 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0d3c06ce90 0x7f0d20005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.135+0000 7f0d3b400640 1 -- 172.21.15.102:0/1444701296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d200047c0 msgr2=0x7f0d20004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:03.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.135+0000 7f0d32a00640 1 --2- 172.21.15.102:0/1444701296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d200047c0 0x7f0d20004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.135+0000 7f0d3b400640 1 --2- 172.21.15.102:0/1444701296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d200047c0 0x7f0d20004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.135+0000 7f0d3b400640 1 -- 172.21.15.102:0/1444701296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d20003520 con 0x7f0d20003970 2024-03-23T03:52:03.138 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.135+0000 7f0d32a00640 1 --2- 172.21.15.102:0/1444701296 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d200047c0 0x7f0d20004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:52:03.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.137+0000 7f0d3b400640 1 --2- 172.21.15.102:0/1444701296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d20003970 0x7f0d20003d90 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00c290 tx=0x7f0d200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9474f1d7c191d443 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:03.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.137+0000 7f0d3aa00640 1 -- 172.21.15.102:0/1444701296 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c005ad0 con 0x7f0d20003970 2024-03-23T03:52:03.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.137+0000 7f0d3aa00640 1 -- 172.21.15.102:0/1444701296 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d2c005c30 con 0x7f0d20003970 2024-03-23T03:52:03.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.137+0000 7f0d3aa00640 1 -- 172.21.15.102:0/1444701296 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c02c8d0 con 0x7f0d20003970 2024-03-23T03:52:03.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.137+0000 7f0d42800640 1 -- 172.21.15.102:0/1444701296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d20003970 msgr2=0x7f0d20003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.137+0000 7f0d42800640 1 --2- 172.21.15.102:0/1444701296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d20003970 0x7f0d20003d90 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c00c290 tx=0x7f0d200057f0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.137+0000 7f0d3be00640 1 -- 172.21.15.102:0/1444701296 reap_dead start 2024-03-23T03:52:03.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.137+0000 7f0d42800640 1 -- 172.21.15.102:0/1444701296 shutdown_connections 2024-03-23T03:52:03.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.137+0000 7f0d42800640 1 -- 172.21.15.102:0/1444701296 >> 172.21.15.102:0/1444701296 conn(0x7f0d3c06a0b0 msgr2=0x7f0d3c06a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:03.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.138+0000 7f0d42800640 1 -- 172.21.15.102:0/1444701296 shutdown_connections 2024-03-23T03:52:03.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.138+0000 7f0d42800640 1 -- 172.21.15.102:0/1444701296 wait complete. 2024-03-23T03:52:03.140 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.139+0000 7f0d42800640 1 Processor -- start 2024-03-23T03:52:03.141 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.139+0000 7f0d42800640 1 -- start start 2024-03-23T03:52:03.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.140+0000 7f0d42800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d20003970 0x7f0d3c14c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.140+0000 7f0d42800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0d3c1591c0 0x7f0d3c15b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.140+0000 7f0d42800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d3c15d450 0x7f0d3c15b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.140+0000 7f0d42800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c0ad370 con 0x7f0d3c15d450 2024-03-23T03:52:03.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.140+0000 7f0d42800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c152fd0 con 0x7f0d20003970 2024-03-23T03:52:03.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.140+0000 7f0d42800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c1508b0 con 0x7f0d3c1591c0 2024-03-23T03:52:03.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.141+0000 7f0d3be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d3c15d450 0x7f0d3c15b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.141+0000 7f0d3be00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d3c15d450 0x7f0d3c15b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:32988/0 (socket says 172.21.15.102:32988) 2024-03-23T03:52:03.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.141+0000 7f0d3be00640 1 -- 172.21.15.102:0/1992872406 learned_addr learned my addr 172.21.15.102:0/1992872406 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:03.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.141+0000 7f0d3b400640 1 --2- 172.21.15.102:0/1992872406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d20003970 0x7f0d3c14c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.141+0000 7f0d32a00640 1 --2- 172.21.15.102:0/1992872406 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0d3c1591c0 0x7f0d3c15b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.141+0000 7f0d3be00640 1 -- 172.21.15.102:0/1992872406 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0d3c1591c0 msgr2=0x7f0d3c15b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.143 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.141+0000 7f0d3be00640 1 --2- 172.21.15.102:0/1992872406 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f0d3c1591c0 0x7f0d3c15b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.141+0000 7f0d3be00640 1 -- 172.21.15.102:0/1992872406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d20003970 msgr2=0x7f0d3c14c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.141+0000 7f0d3be00640 1 --2- 172.21.15.102:0/1992872406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d20003970 0x7f0d3c14c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.141+0000 7f0d3be00640 1 -- 172.21.15.102:0/1992872406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3c151210 con 0x7f0d3c15d450 2024-03-23T03:52:03.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.141+0000 7f0d3b400640 1 --2- 172.21.15.102:0/1992872406 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0d20003970 0x7f0d3c14c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:52:03.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.141+0000 7f0d3be00640 1 --2- 172.21.15.102:0/1992872406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d3c15d450 0x7f0d3c15b970 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c06fde0 tx=0x7f0d3400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:03.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.142+0000 7f0d33e00640 1 -- 172.21.15.102:0/1992872406 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d34010040 con 0x7f0d3c15d450 2024-03-23T03:52:03.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.142+0000 7f0d33e00640 1 -- 172.21.15.102:0/1992872406 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3400bae0 con 0x7f0d3c15d450 2024-03-23T03:52:03.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.142+0000 7f0d33e00640 1 -- 172.21.15.102:0/1992872406 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3400be70 con 0x7f0d3c15d450 2024-03-23T03:52:03.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.142+0000 7f0d42800640 1 -- 172.21.15.102:0/1992872406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d3c0ad090 con 0x7f0d3c15d450 2024-03-23T03:52:03.144 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.142+0000 7f0d42800640 1 -- 172.21.15.102:0/1992872406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d3c1504d0 con 0x7f0d3c15d450 2024-03-23T03:52:03.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.143+0000 7f0d42800640 1 -- 172.21.15.102:0/1992872406 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0d3c0ce160 con 0x7f0d3c15d450 2024-03-23T03:52:03.145 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.144+0000 7f0d33e00640 1 -- 172.21.15.102:0/1992872406 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f0d3401d400 con 0x7f0d3c15d450 2024-03-23T03:52:03.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.145+0000 7f0d33e00640 1 --2- 172.21.15.102:0/1992872406 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0d1806ac30 0x7f0d1806d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.145+0000 7f0d33e00640 1 -- 172.21.15.102:0/1992872406 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f0d3408a0b0 con 0x7f0d3c15d450 2024-03-23T03:52:03.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.145+0000 7f0d3b400640 1 --2- 172.21.15.102:0/1992872406 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0d1806ac30 0x7f0d1806d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.145+0000 7f0d33e00640 1 --2- 172.21.15.102:0/1992872406 >> [v2:172.21.15.173:6818/37510,v1:172.21.15.173:6821/37510] conn(0x7f0d18070f30 0x7f0d18073370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.145+0000 7f0d33e00640 1 -- 172.21.15.102:0/1992872406 --> [v2:172.21.15.173:6818/37510,v1:172.21.15.173:6821/37510] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d18073a40 con 0x7f0d18070f30 2024-03-23T03:52:03.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.145+0000 7f0d33e00640 1 -- 172.21.15.102:0/1992872406 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=110) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d3408a420 con 0x7f0d3c15d450 2024-03-23T03:52:03.146 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.146+0000 7f0d3b400640 1 --2- 172.21.15.102:0/1992872406 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0d1806ac30 0x7f0d1806d0f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c034000 tx=0x7f0d2c007000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:03.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.146+0000 7f0d32a00640 1 --2- 172.21.15.102:0/1992872406 >> [v2:172.21.15.173:6818/37510,v1:172.21.15.173:6821/37510] conn(0x7f0d18070f30 0x7f0d18073370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.147 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.147+0000 7f0d32a00640 1 --2- 172.21.15.102:0/1992872406 >> [v2:172.21.15.173:6818/37510,v1:172.21.15.173:6821/37510] conn(0x7f0d18070f30 0x7f0d18073370 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:03.149 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.149+0000 7f0d33e00640 1 -- 172.21.15.102:0/1992872406 <== osd.7 v2:172.21.15.173:6818/37510 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25254 (crc 0 0 0) 0x7f0d18073a40 con 0x7f0d18070f30 2024-03-23T03:52:03.163 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.163+0000 7f0d42800640 1 -- 172.21.15.102:0/1992872406 --> [v2:172.21.15.173:6818/37510,v1:172.21.15.173:6821/37510] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0d3c152c20 con 0x7f0d18070f30 2024-03-23T03:52:03.164 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.164+0000 7f0d33e00640 1 -- 172.21.15.102:0/1992872406 <== osd.7 v2:172.21.15.173:6818/37510 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f0d3c152c20 con 0x7f0d18070f30 2024-03-23T03:52:03.164 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-23T03:52:03.165 INFO:teuthology.orchestra.run.smithi102.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T03:52:03.165 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-23T03:52:03.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.164+0000 7f0d42800640 1 -- 172.21.15.102:0/1992872406 >> [v2:172.21.15.173:6818/37510,v1:172.21.15.173:6821/37510] conn(0x7f0d18070f30 msgr2=0x7f0d18073370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.164+0000 7f0d42800640 1 --2- 172.21.15.102:0/1992872406 >> [v2:172.21.15.173:6818/37510,v1:172.21.15.173:6821/37510] conn(0x7f0d18070f30 0x7f0d18073370 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.164+0000 7f0d42800640 1 -- 172.21.15.102:0/1992872406 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0d1806ac30 msgr2=0x7f0d1806d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.164+0000 7f0d42800640 1 --2- 172.21.15.102:0/1992872406 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f0d1806ac30 0x7f0d1806d0f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c034000 tx=0x7f0d2c007000 comp rx=0 tx=0).stop 2024-03-23T03:52:03.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.164+0000 7f0d42800640 1 -- 172.21.15.102:0/1992872406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d3c15d450 msgr2=0x7f0d3c15b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.164+0000 7f0d42800640 1 --2- 172.21.15.102:0/1992872406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d3c15d450 0x7f0d3c15b970 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c06fde0 tx=0x7f0d3400b480 comp rx=0 tx=0).stop 2024-03-23T03:52:03.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.164+0000 7f0d3be00640 1 -- 172.21.15.102:0/1992872406 reap_dead start 2024-03-23T03:52:03.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.165+0000 7f0d42800640 1 -- 172.21.15.102:0/1992872406 shutdown_connections 2024-03-23T03:52:03.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.165+0000 7f0d42800640 1 -- 172.21.15.102:0/1992872406 >> 172.21.15.102:0/1992872406 conn(0x7f0d3c06a0b0 msgr2=0x7f0d3c0cbc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:03.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.165+0000 7f0d42800640 1 -- 172.21.15.102:0/1992872406 shutdown_connections 2024-03-23T03:52:03.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.165+0000 7f0d42800640 1 -- 172.21.15.102:0/1992872406 wait complete. 2024-03-23T03:52:03.182 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-23T03:52:03.381 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.381+0000 7fc747800640 1 Processor -- start 2024-03-23T03:52:03.382 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.381+0000 7fc747800640 1 -- start start 2024-03-23T03:52:03.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.382+0000 7fc747800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7400cb660 0x7fc7400cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.383+0000 7fc747800640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc74015ae60 con 0x7fc74006c2b0 2024-03-23T03:52:03.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.383+0000 7fc747800640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc74015b140 con 0x7fc7400ce930 2024-03-23T03:52:03.383 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.383+0000 7fc747800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc74015bae0 con 0x7fc7400cb660 2024-03-23T03:52:03.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.383+0000 7fc745800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7fc74006c2b0 0x7fc74006c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33990/0 (socket says 172.21.15.102:33990) 2024-03-23T03:52:03.384 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.383+0000 7fc745800640 1 -- 172.21.15.102:0/2814593604 learned_addr learned my addr 172.21.15.102:0/2814593604 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:52:03.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.383+0000 7fc73fe00640 1 --2- 172.21.15.102:0/2814593604 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7400cb660 0x7fc7400cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.383+0000 7fc73f400640 1 -- 172.21.15.102:0/2814593604 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871827366 0 0) 0x7fc74015ae60 con 0x7fc74006c2b0 2024-03-23T03:52:03.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.383+0000 7fc73f400640 1 -- 172.21.15.102:0/2814593604 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc728003520 con 0x7fc74006c2b0 2024-03-23T03:52:03.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.384+0000 7fc73f400640 1 -- 172.21.15.102:0/2814593604 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 547110479 0 0) 0x7fc728003520 con 0x7fc74006c2b0 2024-03-23T03:52:03.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.384+0000 7fc73f400640 1 -- 172.21.15.102:0/2814593604 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7400cb660 msgr2=0x7fc7400cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:03.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.384+0000 7fc73f400640 1 --2- 172.21.15.102:0/2814593604 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7400cb660 0x7fc7400cbae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.384+0000 7fc73f400640 1 -- 172.21.15.102:0/2814593604 >> v1:172.21.15.173:6789/0 conn(0x7fc7400ce930 legacy=0x7fc7400ced10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.384+0000 7fc73f400640 1 -- 172.21.15.102:0/2814593604 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74015b420 con 0x7fc74006c2b0 2024-03-23T03:52:03.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.384+0000 7fc73f400640 1 -- 172.21.15.102:0/2814593604 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7fc7380019f0 con 0x7fc74006c2b0 2024-03-23T03:52:03.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.384+0000 7fc73f400640 1 -- 172.21.15.102:0/2814593604 >> v1:172.21.15.102:6789/0 conn(0x7fc74006c2b0 legacy=0x7fc74006c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.386 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.384+0000 7fc73f400640 1 --2- 172.21.15.102:0/2814593604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7280039a0 0x7fc728003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.384+0000 7fc73f400640 1 --2- 172.21.15.102:0/2814593604 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7400cb660 0x7fc7280047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.384+0000 7fc73f400640 1 --2- 172.21.15.102:0/2814593604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc728004d00 0x7fc728005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.384+0000 7fc73f400640 1 -- 172.21.15.102:0/2814593604 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74015bae0 con 0x7fc728004d00 2024-03-23T03:52:03.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.384+0000 7fc73f400640 1 -- 172.21.15.102:0/2814593604 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72800ba60 con 0x7fc7280039a0 2024-03-23T03:52:03.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.384+0000 7fc73f400640 1 -- 172.21.15.102:0/2814593604 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc72800bc20 con 0x7fc7400cb660 2024-03-23T03:52:03.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.384+0000 7fc745800640 1 --2- 172.21.15.102:0/2814593604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc728004d00 0x7fc728005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.385+0000 7fc744e00640 1 --2- 172.21.15.102:0/2814593604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7280039a0 0x7fc728003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.385+0000 7fc73fe00640 1 --2- 172.21.15.102:0/2814593604 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7400cb660 0x7fc7280047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.385+0000 7fc745800640 1 -- 172.21.15.102:0/2814593604 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7400cb660 msgr2=0x7fc7280047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:03.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.385+0000 7fc745800640 1 --2- 172.21.15.102:0/2814593604 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7400cb660 0x7fc7280047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.385+0000 7fc745800640 1 -- 172.21.15.102:0/2814593604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7280039a0 msgr2=0x7fc728003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:03.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.385+0000 7fc745800640 1 --2- 172.21.15.102:0/2814593604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7280039a0 0x7fc728003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.385+0000 7fc745800640 1 -- 172.21.15.102:0/2814593604 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc728003520 con 0x7fc728004d00 2024-03-23T03:52:03.387 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.385+0000 7fc744e00640 1 --2- 172.21.15.102:0/2814593604 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7280039a0 0x7fc728003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:52:03.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.386+0000 7fc745800640 1 --2- 172.21.15.102:0/2814593604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc728004d00 0x7fc728005180 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fc738002230 tx=0x7fc7280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed70f35b6f4d1f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:03.388 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.386+0000 7fc73f400640 1 -- 172.21.15.102:0/2814593604 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc738034630 con 0x7fc728004d00 2024-03-23T03:52:03.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.386+0000 7fc73f400640 1 -- 172.21.15.102:0/2814593604 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc73802f070 con 0x7fc728004d00 2024-03-23T03:52:03.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.386+0000 7fc73f400640 1 -- 172.21.15.102:0/2814593604 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7380349f0 con 0x7fc728004d00 2024-03-23T03:52:03.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.387+0000 7fc747800640 1 -- 172.21.15.102:0/2814593604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc728004d00 msgr2=0x7fc728005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.387+0000 7fc747800640 1 --2- 172.21.15.102:0/2814593604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc728004d00 0x7fc728005180 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fc738002230 tx=0x7fc7280057f0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.387+0000 7fc745800640 1 -- 172.21.15.102:0/2814593604 reap_dead start 2024-03-23T03:52:03.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.387+0000 7fc747800640 1 -- 172.21.15.102:0/2814593604 shutdown_connections 2024-03-23T03:52:03.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.387+0000 7fc747800640 1 -- 172.21.15.102:0/2814593604 >> 172.21.15.102:0/2814593604 conn(0x7fc74006a0b0 msgr2=0x7fc74006a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:03.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.387+0000 7fc747800640 1 -- 172.21.15.102:0/2814593604 shutdown_connections 2024-03-23T03:52:03.389 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.387+0000 7fc747800640 1 -- 172.21.15.102:0/2814593604 wait complete. 2024-03-23T03:52:03.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.389+0000 7fc747800640 1 Processor -- start 2024-03-23T03:52:03.390 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.389+0000 7fc747800640 1 -- start start 2024-03-23T03:52:03.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.390+0000 7fc747800640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc728004d00 0x7fc740145130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.390+0000 7fc747800640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7400b2300 0x7fc7400b2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.390+0000 7fc747800640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7401f4a80 0x7fc7401f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.390+0000 7fc747800640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7400b2cc0 con 0x7fc728004d00 2024-03-23T03:52:03.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.390+0000 7fc747800640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7400b01d0 con 0x7fc7400b2300 2024-03-23T03:52:03.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.390+0000 7fc747800640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7400b1030 con 0x7fc7401f4a80 2024-03-23T03:52:03.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.390+0000 7fc744e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc728004d00 0x7fc740145130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.390+0000 7fc744e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc728004d00 0x7fc740145130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33006/0 (socket says 172.21.15.102:33006) 2024-03-23T03:52:03.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.390+0000 7fc744e00640 1 -- 172.21.15.102:0/3718762644 learned_addr learned my addr 172.21.15.102:0/3718762644 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:03.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.390+0000 7fc744e00640 1 -- 172.21.15.102:0/3718762644 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7401f4a80 msgr2=0x7fc7401f4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.390+0000 7fc744e00640 1 --2- 172.21.15.102:0/3718762644 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7401f4a80 0x7fc7401f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.390+0000 7fc744e00640 1 -- 172.21.15.102:0/3718762644 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7400b2300 msgr2=0x7fc7400b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.390+0000 7fc744e00640 1 --2- 172.21.15.102:0/3718762644 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7400b2300 0x7fc7400b2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.392 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.390+0000 7fc744e00640 1 -- 172.21.15.102:0/3718762644 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc740155280 con 0x7fc728004d00 2024-03-23T03:52:03.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.391+0000 7fc744e00640 1 --2- 172.21.15.102:0/3718762644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc728004d00 0x7fc740145130 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fc73400e550 tx=0x7fc73400ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:03.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.391+0000 7fc73cc00640 1 -- 172.21.15.102:0/3718762644 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7340053a0 con 0x7fc728004d00 2024-03-23T03:52:03.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.391+0000 7fc73cc00640 1 -- 172.21.15.102:0/3718762644 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7340035f0 con 0x7fc728004d00 2024-03-23T03:52:03.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.391+0000 7fc747800640 1 -- 172.21.15.102:0/3718762644 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7400b0420 con 0x7fc728004d00 2024-03-23T03:52:03.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.391+0000 7fc73cc00640 1 -- 172.21.15.102:0/3718762644 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc734003980 con 0x7fc728004d00 2024-03-23T03:52:03.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.391+0000 7fc747800640 1 -- 172.21.15.102:0/3718762644 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7400b09e0 con 0x7fc728004d00 2024-03-23T03:52:03.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.392+0000 7fc747800640 1 -- 172.21.15.102:0/3718762644 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc740073390 con 0x7fc728004d00 2024-03-23T03:52:03.393 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.393+0000 7fc73cc00640 1 -- 172.21.15.102:0/3718762644 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7fc734003ae0 con 0x7fc728004d00 2024-03-23T03:52:03.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.394+0000 7fc73cc00640 1 --2- 172.21.15.102:0/3718762644 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc72006ac30 0x7fc72006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.394+0000 7fc73cc00640 1 -- 172.21.15.102:0/3718762644 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fc734088e90 con 0x7fc728004d00 2024-03-23T03:52:03.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.394+0000 7fc73fe00640 1 --2- 172.21.15.102:0/3718762644 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc72006ac30 0x7fc72006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.395 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.395+0000 7fc73fe00640 1 --2- 172.21.15.102:0/3718762644 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc72006ac30 0x7fc72006d0f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc7400af750 tx=0x7fc72c008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:03.399 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.398+0000 7fc73cc00640 1 -- 172.21.15.102:0/3718762644 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7fc7340542c0 con 0x7fc728004d00 2024-03-23T03:52:03.591 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.591+0000 7fc747800640 1 -- 172.21.15.102:0/3718762644 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fc740155100 con 0x7fc72006ac30 2024-03-23T03:52:03.592 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.592+0000 7fc73cc00640 1 -- 172.21.15.102:0/3718762644 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc72c011860 con 0x7fc72006ac30 2024-03-23T03:52:03.592 INFO:teuthology.orchestra.run.smithi102.stderr:instructed osd(s) 7 to deep-scrub 2024-03-23T03:52:03.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.594+0000 7fc747800640 1 -- 172.21.15.102:0/3718762644 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc72006ac30 msgr2=0x7fc72006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.594+0000 7fc747800640 1 --2- 172.21.15.102:0/3718762644 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc72006ac30 0x7fc72006d0f0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fc7400af750 tx=0x7fc72c008040 comp rx=0 tx=0).stop 2024-03-23T03:52:03.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.594+0000 7fc747800640 1 -- 172.21.15.102:0/3718762644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc728004d00 msgr2=0x7fc740145130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.595 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.594+0000 7fc747800640 1 --2- 172.21.15.102:0/3718762644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc728004d00 0x7fc740145130 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fc73400e550 tx=0x7fc73400ea10 comp rx=0 tx=0).stop 2024-03-23T03:52:03.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.594+0000 7fc747800640 1 -- 172.21.15.102:0/3718762644 shutdown_connections 2024-03-23T03:52:03.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.594+0000 7fc747800640 1 --2- 172.21.15.102:0/3718762644 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7fc72006ac30 0x7fc72006d0f0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.594+0000 7fc747800640 1 --2- 172.21.15.102:0/3718762644 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7fc7401f4a80 0x7fc7401f4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.594+0000 7fc747800640 1 --2- 172.21.15.102:0/3718762644 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc7400b2300 0x7fc7400b2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.594+0000 7fc747800640 1 --2- 172.21.15.102:0/3718762644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc728004d00 0x7fc740145130 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.594+0000 7fc747800640 1 -- 172.21.15.102:0/3718762644 >> 172.21.15.102:0/3718762644 conn(0x7fc74006a0b0 msgr2=0x7fc74015a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:03.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.595+0000 7fc747800640 1 -- 172.21.15.102:0/3718762644 shutdown_connections 2024-03-23T03:52:03.596 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.595+0000 7fc747800640 1 -- 172.21.15.102:0/3718762644 wait complete. 2024-03-23T03:52:03.614 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:52:03.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.826+0000 7f92bda00640 1 Processor -- start 2024-03-23T03:52:03.826 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.826+0000 7f92bda00640 1 -- start start 2024-03-23T03:52:03.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.827+0000 7f92bda00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f92b80adbc0 0x7f92b80b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.827+0000 7f92bda00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b80b16a0 con 0x7f92b80cbab0 2024-03-23T03:52:03.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.827+0000 7f92bda00640 1 -- --> v1:172.21.15.173:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b80b4700 con 0x7f92b806d390 2024-03-23T03:52:03.828 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.827+0000 7f92bda00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92b80cbfb0 con 0x7f92b80adbc0 2024-03-23T03:52:03.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.828+0000 7f92b6000640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f92b80cbab0 0x7f92b80b20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:33998/0 (socket says 172.21.15.102:33998) 2024-03-23T03:52:03.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.828+0000 7f92b6000640 1 -- 172.21.15.102:0/346144862 learned_addr learned my addr 172.21.15.102:0/346144862 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:52:03.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.828+0000 7f92b6a00640 1 --2- 172.21.15.102:0/346144862 >> v2:172.21.15.173:3300/0 conn(0x7f92b80adbc0 0x7f92b80b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.828+0000 7f92b5600640 1 -- 172.21.15.102:0/346144862 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557545586 0 0) 0x7f92b80b16a0 con 0x7f92b80cbab0 2024-03-23T03:52:03.830 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.828+0000 7f92b5600640 1 -- 172.21.15.102:0/346144862 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f929c003520 con 0x7f92b80cbab0 2024-03-23T03:52:03.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.828+0000 7f92b5600640 1 -- 172.21.15.102:0/346144862 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3131360031 0 0) 0x7f929c003520 con 0x7f92b80cbab0 2024-03-23T03:52:03.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.829+0000 7f92b5600640 1 -- 172.21.15.102:0/346144862 >> v2:172.21.15.173:3300/0 conn(0x7f92b80adbc0 msgr2=0x7f92b80b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:03.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.829+0000 7f92b5600640 1 --2- 172.21.15.102:0/346144862 >> v2:172.21.15.173:3300/0 conn(0x7f92b80adbc0 0x7f92b80b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.829+0000 7f92b5600640 1 -- 172.21.15.102:0/346144862 >> v1:172.21.15.173:6789/0 conn(0x7f92b806d390 legacy=0x7f92b80b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:03.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.829+0000 7f92b5600640 1 -- 172.21.15.102:0/346144862 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b80cc290 con 0x7f92b80cbab0 2024-03-23T03:52:03.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.829+0000 7f92b5600640 1 -- 172.21.15.102:0/346144862 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f92a0001670 con 0x7f92b80cbab0 2024-03-23T03:52:03.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.829+0000 7f92b5600640 1 -- 172.21.15.102:0/346144862 >> v1:172.21.15.102:6789/0 conn(0x7f92b80cbab0 legacy=0x7f92b80b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.829+0000 7f92b5600640 1 --2- 172.21.15.102:0/346144862 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f929c0039c0 0x7f929c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.829+0000 7f92b5600640 1 --2- 172.21.15.102:0/346144862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f929c0047c0 0x7f929c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.829+0000 7f92b5600640 1 --2- 172.21.15.102:0/346144862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f929c0051d0 0x7f929c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.829+0000 7f92b5600640 1 -- 172.21.15.102:0/346144862 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b80cbfb0 con 0x7f929c0047c0 2024-03-23T03:52:03.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.829+0000 7f92b5600640 1 -- 172.21.15.102:0/346144862 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c00e310 con 0x7f929c0051d0 2024-03-23T03:52:03.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.829+0000 7f92b5600640 1 -- 172.21.15.102:0/346144862 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f929c00e4d0 con 0x7f929c0039c0 2024-03-23T03:52:03.831 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.829+0000 7f92b6000640 1 --2- 172.21.15.102:0/346144862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f929c0047c0 0x7f929c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.829+0000 7f92b6000640 1 -- 172.21.15.102:0/346144862 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f929c0039c0 msgr2=0x7f929c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:03.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.830+0000 7f92b6000640 1 --2- 172.21.15.102:0/346144862 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f929c0039c0 0x7f929c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.830+0000 7f92b6000640 1 -- 172.21.15.102:0/346144862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f929c0051d0 msgr2=0x7f929c00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:03.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.830+0000 7f92b6000640 1 --2- 172.21.15.102:0/346144862 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f929c0051d0 0x7f929c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.830+0000 7f92b6000640 1 -- 172.21.15.102:0/346144862 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c003520 con 0x7f929c0047c0 2024-03-23T03:52:03.832 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.830+0000 7f92b7400640 1 -- 172.21.15.102:0/346144862 reap_dead start 2024-03-23T03:52:03.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.831+0000 7f92b6000640 1 --2- 172.21.15.102:0/346144862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f929c0047c0 0x7f929c004c40 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f92a000c160 tx=0x7f929c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5466222e2904c877 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:03.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.831+0000 7f92b5600640 1 -- 172.21.15.102:0/346144862 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a0005ad0 con 0x7f929c0047c0 2024-03-23T03:52:03.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.831+0000 7f92b5600640 1 -- 172.21.15.102:0/346144862 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a0005c30 con 0x7f929c0047c0 2024-03-23T03:52:03.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.831+0000 7f92b5600640 1 -- 172.21.15.102:0/346144862 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a002c7e0 con 0x7f929c0047c0 2024-03-23T03:52:03.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.831+0000 7f92bda00640 1 -- 172.21.15.102:0/346144862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f929c0047c0 msgr2=0x7f929c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.831+0000 7f92bda00640 1 --2- 172.21.15.102:0/346144862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f929c0047c0 0x7f929c004c40 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f92a000c160 tx=0x7f929c0057f0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.832+0000 7f92bda00640 1 -- 172.21.15.102:0/346144862 shutdown_connections 2024-03-23T03:52:03.833 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.832+0000 7f92bda00640 1 --2- 172.21.15.102:0/346144862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f929c0047c0 0x7f929c004c40 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.832+0000 7f92bda00640 1 -- 172.21.15.102:0/346144862 >> 172.21.15.102:0/346144862 conn(0x7f92b806a3d0 msgr2=0x7f92b80b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:03.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.832+0000 7f92bda00640 1 -- 172.21.15.102:0/346144862 shutdown_connections 2024-03-23T03:52:03.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.832+0000 7f92bda00640 1 -- 172.21.15.102:0/346144862 wait complete. 2024-03-23T03:52:03.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.833+0000 7f92bda00640 1 Processor -- start 2024-03-23T03:52:03.834 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.834+0000 7f92bda00640 1 -- start start 2024-03-23T03:52:03.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.834+0000 7f92bda00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f929c0047c0 0x7f92b8140b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.835+0000 7f92bda00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92b8141050 0x7f92b815d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.835+0000 7f92bda00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92b81595f0 0x7f92b815d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.835+0000 7f92bda00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b813fa70 con 0x7f929c0047c0 2024-03-23T03:52:03.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.835+0000 7f92bda00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92b813fd50 con 0x7f92b81595f0 2024-03-23T03:52:03.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.835+0000 7f92bda00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92b8140030 con 0x7f92b8141050 2024-03-23T03:52:03.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.835+0000 7f92b6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f929c0047c0 0x7f92b8140b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.835+0000 7f92b6a00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f929c0047c0 0x7f92b8140b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:33036/0 (socket says 172.21.15.102:33036) 2024-03-23T03:52:03.836 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.835+0000 7f92b6a00640 1 -- 172.21.15.102:0/4192281466 learned_addr learned my addr 172.21.15.102:0/4192281466 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:03.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.835+0000 7f92b7400640 1 --2- 172.21.15.102:0/4192281466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92b81595f0 0x7f92b815d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.835+0000 7f92b6a00640 1 -- 172.21.15.102:0/4192281466 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92b8141050 msgr2=0x7f92b815d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.835+0000 7f92b6a00640 1 --2- 172.21.15.102:0/4192281466 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92b8141050 0x7f92b815d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.835+0000 7f92b6a00640 1 -- 172.21.15.102:0/4192281466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92b81595f0 msgr2=0x7f92b815d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:03.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.835+0000 7f92b6a00640 1 --2- 172.21.15.102:0/4192281466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92b81595f0 0x7f92b815d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:03.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.835+0000 7f92b6a00640 1 -- 172.21.15.102:0/4192281466 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b8140310 con 0x7f929c0047c0 2024-03-23T03:52:03.837 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.836+0000 7f92b6a00640 1 --2- 172.21.15.102:0/4192281466 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f929c0047c0 0x7f92b8140b10 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f92a800e7f0 tx=0x7f92a800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:03.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.836+0000 7f92aea00640 1 -- 172.21.15.102:0/4192281466 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a800c730 con 0x7f929c0047c0 2024-03-23T03:52:03.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.836+0000 7f92aea00640 1 -- 172.21.15.102:0/4192281466 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a800c8b0 con 0x7f929c0047c0 2024-03-23T03:52:03.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.836+0000 7f92aea00640 1 -- 172.21.15.102:0/4192281466 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a800cca0 con 0x7f929c0047c0 2024-03-23T03:52:03.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.836+0000 7f92bda00640 1 -- 172.21.15.102:0/4192281466 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b81405f0 con 0x7f929c0047c0 2024-03-23T03:52:03.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.836+0000 7f92bda00640 1 -- 172.21.15.102:0/4192281466 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b813ae70 con 0x7f929c0047c0 2024-03-23T03:52:03.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.837+0000 7f92bda00640 1 -- 172.21.15.102:0/4192281466 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b806ac20 con 0x7f929c0047c0 2024-03-23T03:52:03.838 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.838+0000 7f92aea00640 1 -- 172.21.15.102:0/4192281466 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f92a8013470 con 0x7f929c0047c0 2024-03-23T03:52:03.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.839+0000 7f92aea00640 1 --2- 172.21.15.102:0/4192281466 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f929406ac30 0x7f929406d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:03.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.839+0000 7f92aea00640 1 -- 172.21.15.102:0/4192281466 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f92a8089220 con 0x7f929c0047c0 2024-03-23T03:52:03.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.839+0000 7f92b6000640 1 --2- 172.21.15.102:0/4192281466 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f929406ac30 0x7f929406d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:03.840 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.840+0000 7f92b6000640 1 --2- 172.21.15.102:0/4192281466 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f929406ac30 0x7f929406d0f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f92a0034000 tx=0x7f92a0039040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:03.844 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:03.843+0000 7f92aea00640 1 -- 172.21.15.102:0/4192281466 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f92a8054650 con 0x7f929c0047c0 2024-03-23T03:52:04.038 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:04.037+0000 7f92bda00640 1 -- 172.21.15.102:0/4192281466 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92b813fec0 con 0x7f929406ac30 2024-03-23T03:52:04.043 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:04.042+0000 7f92aea00640 1 -- 172.21.15.102:0/4192281466 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48622 (secure 0 0 0) 0x7f92b813fec0 con 0x7f929406ac30 2024-03-23T03:52:04.044 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:52:04.044 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:52:04.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:04.045+0000 7f92bda00640 1 -- 172.21.15.102:0/4192281466 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f929406ac30 msgr2=0x7f929406d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:04.045 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:04.045+0000 7f92bda00640 1 --2- 172.21.15.102:0/4192281466 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f929406ac30 0x7f929406d0f0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f92a0034000 tx=0x7f92a0039040 comp rx=0 tx=0).stop 2024-03-23T03:52:04.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:04.045+0000 7f92bda00640 1 -- 172.21.15.102:0/4192281466 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f929c0047c0 msgr2=0x7f92b8140b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:04.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:04.045+0000 7f92bda00640 1 --2- 172.21.15.102:0/4192281466 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f929c0047c0 0x7f92b8140b10 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f92a800e7f0 tx=0x7f92a800ecb0 comp rx=0 tx=0).stop 2024-03-23T03:52:04.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:04.045+0000 7f92bda00640 1 -- 172.21.15.102:0/4192281466 shutdown_connections 2024-03-23T03:52:04.046 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:04.045+0000 7f92bda00640 1 --2- 172.21.15.102:0/4192281466 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f929406ac30 0x7f929406d0f0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:04.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:04.045+0000 7f92bda00640 1 --2- 172.21.15.102:0/4192281466 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f92b81595f0 0x7f92b815d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:04.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:04.045+0000 7f92bda00640 1 --2- 172.21.15.102:0/4192281466 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f92b8141050 0x7f92b815d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:04.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:04.045+0000 7f92bda00640 1 --2- 172.21.15.102:0/4192281466 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f929c0047c0 0x7f92b8140b10 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:04.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:04.045+0000 7f92bda00640 1 -- 172.21.15.102:0/4192281466 >> 172.21.15.102:0/4192281466 conn(0x7f92b806a3d0 msgr2=0x7f92b80b33d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:04.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:04.046+0000 7f92bda00640 1 -- 172.21.15.102:0/4192281466 shutdown_connections 2024-03-23T03:52:04.047 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:04.046+0000 7f92bda00640 1 -- 172.21.15.102:0/4192281466 wait complete. 2024-03-23T03:52:04.063 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2024-03-23T03:52:03.812900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":256684,"kb_used_data":11848,"kb_used_omap":12,"kb_used_meta":244723,"kb_avail":754718036,"statfs":{"total":773094113280,"available":772831268864,"internally_reserved":0,"allocated":12132352,"data_stored":6150016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":250596922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.985670"},"pg_stats":[{"pgid":"1.0","version":"19'117","reported_seq":234,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-23T03:51:50.981540+0000","last_change":"2024-03-23T03:46:49.735885+0000","last_active":"2024-03-23T03:51:50.981540+0000","last_peered":"2024-03-23T03:51:50.981540+0000","last_clean":"2024-03-23T03:51:50.981540+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:51:50.981540+0000","last_undegraded":"2024-03-23T03:51:50.981540+0000","last_fullsized":"2024-03-23T03:51:50.981540+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T03:46:48.337680+0000","last_clean_scrub_stamp":"2024-03-23T03:46:48.337680+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T10:40:55.871173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542211,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1961984,"data_stored":1211528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0740000000000001,"15min":1.0680000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.478,"5min":1.635,"15min":1.635},"last":1.121},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.0569999999999999,"15min":1.052},"min":{"1min":0.76100000000000001,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":1.3009999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:52:00 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.091,"15min":1.056},"min":{"1min":0.70899999999999996,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6579999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.137,"15min":1.2689999999999999},"min":{"1min":0.80000000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.5109999999999999,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":1.4159999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:52:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.111,"15min":1.147},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.6279999999999999,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.224},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.2310000000000001,"15min":1.383},"min":{"1min":0.79500000000000004,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.631,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.0549999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:52:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.242,"15min":1.3999999999999999},"min":{"1min":0.81999999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.5269999999999999,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.286},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.3089999999999999,"15min":1.3480000000000001},"min":{"1min":0.90500000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.7370000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.95699999999999996}]},{"osd":4,"last update":"Sat Mar 23 03:52:00 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2649999999999999,"15min":1.2669999999999999},"min":{"1min":0.97399999999999998,"5min":0.62,"15min":0.62},"max":{"1min":1.641,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3360000000000001,"15min":1.3680000000000001},"min":{"1min":0.92900000000000005,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.722,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.1499999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:52:00 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.367,"15min":1.411},"min":{"1min":0.98499999999999999,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.8,"5min":2.645,"15min":2.645},"last":1.2450000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.403,"15min":1.4410000000000001},"min":{"1min":0.84699999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.75,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.573}]},{"osd":6,"last update":"Sat Mar 23 03:52:00 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.49,"15min":1.548},"min":{"1min":1.0649999999999999,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.772,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.454,"15min":1.508},"min":{"1min":1.0940000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.8049999999999999,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.3360000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1368064,"data_stored":621160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:51:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0369999999999999,"15min":1.042},"min":{"1min":0.63300000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.671,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.016,"5min":1.046,"15min":1.0389999999999999},"min":{"1min":0.69299999999999995,"5min":0.62,"15min":0.62},"max":{"1min":1.486,"5min":1.946,"15min":1.946},"last":0.83599999999999997}]},{"osd":1,"last update":"Sat Mar 23 03:51:58 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.081,"15min":1.073},"min":{"1min":0.70799999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5509999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.107,"15min":1.149},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.4330000000000001,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.60199999999999998}]},{"osd":2,"last update":"Sat Mar 23 03:51:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.083,"15min":1.071},"min":{"1min":0.72499999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.4359999999999999,"5min":2.044,"15min":2.044},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.113,"5min":1.1519999999999999,"15min":1.204},"min":{"1min":0.72299999999999998,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.4450000000000001,"5min":1.681,"15min":1.681},"last":1.181}]},{"osd":3,"last update":"Sat Mar 23 03:51:58 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.222,"15min":1.2709999999999999},"min":{"1min":0.85999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.611,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.296},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.218,"15min":1.252},"min":{"1min":0.88,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.5169999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.131}]},{"osd":4,"last update":"Sat Mar 23 03:51:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3779999999999999,"15min":1.385},"min":{"1min":0.74299999999999999,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.796,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.373,"5min":1.3660000000000001,"15min":1.3839999999999999},"min":{"1min":0.72799999999999998,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.661,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.3460000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:51:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.375,"15min":1.423},"min":{"1min":0.63800000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.7,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.407},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4890000000000001,"15min":1.5349999999999999},"min":{"1min":0.78800000000000003,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.728,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.655}]},{"osd":7,"last update":"Sat Mar 23 03:51:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3959999999999999,"15min":1.4079999999999999},"min":{"1min":0.66500000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.579,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.4670000000000001},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.4490000000000001,"15min":1.52},"min":{"1min":0.80000000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.7609999999999999,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.496}]}]},{"osd":5,"up_from":14,"seq":60129542209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28356,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343484,"statfs":{"total":96636764160,"available":96607727616,"internally_reserved":0,"allocated":1368064,"data_stored":621160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.087,"15min":1.1839999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.355,"5min":1.806,"15min":1.806},"last":1.355},{"interface":"front","average":{"1min":1.085,"5min":1.119,"15min":1.169},"min":{"1min":0.63200000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.4299999999999999,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.4299999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.079,"15min":1.095},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.4139999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.111,"5min":1.1459999999999999,"15min":1.1870000000000001},"min":{"1min":0.66100000000000003,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.706,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":1.706}]},{"osd":2,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.2190000000000001,"15min":1.327},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.4710000000000001,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.089,"5min":1.2370000000000001,"15min":1.345},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.5760000000000001,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.2909999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.171,"15min":1.234},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.379,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.325},{"interface":"front","average":{"1min":1.216,"5min":1.321,"15min":1.4279999999999999},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":1.5700000000000001,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.2450000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.2410000000000001,"15min":1.3779999999999999},"min":{"1min":0.65100000000000002,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6000000000000001,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.206,"5min":1.335,"15min":1.419},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.6259999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.528}]},{"osd":6,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.3440000000000001,"15min":1.476},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.5029999999999999,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.409},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.4610000000000001,"15min":1.6120000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.667,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.4550000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.4219999999999999,"15min":1.5680000000000001},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.7529999999999999,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.5},{"interface":"front","average":{"1min":1.413,"5min":1.52,"15min":1.6619999999999999},"min":{"1min":0.74299999999999999,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.833,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.625}]}]},{"osd":4,"up_from":14,"seq":60129542210,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1368064,"data_stored":621160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.028,"15min":0.98899999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0029999999999999,"15min":1.008},"min":{"1min":0.65900000000000003,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.952,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.92700000000000005}]},{"osd":1,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.0529999999999999,"15min":1.0449999999999999},"min":{"1min":0.64800000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.282,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.109,"15min":1.071},"min":{"1min":0.61899999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.5820000000000001,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":1.6579999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.22,"15min":1.3},"min":{"1min":0.79300000000000004,"5min":0.625,"15min":0.625},"max":{"1min":1.55,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":0.876},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.268,"15min":1.294},"min":{"1min":0.84699999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6679999999999999,"5min":2.181,"15min":2.181},"last":1.782}]},{"osd":3,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.109,"15min":1.1220000000000001},"min":{"1min":0.63,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.5069999999999999,"5min":1.821,"15min":1.821},"last":1.165},{"interface":"front","average":{"1min":1.252,"5min":1.224,"15min":1.266},"min":{"1min":0.83599999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.6870000000000001,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.3360000000000001}]},{"osd":5,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.252,"15min":1.2649999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.7450000000000001,"5min":1.929,"15min":1.929},"last":1.901},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.274,"15min":1.2749999999999999},"min":{"1min":0.51100000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.532,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.252}]},{"osd":6,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.357,"15min":1.405},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.847,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.619},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.4359999999999999,"15min":1.4079999999999999},"min":{"1min":0.67400000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":2.8879999999999999,"5min":2.8879999999999999,"15min":2.8879999999999999},"last":1.986}]},{"osd":7,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.379,"15min":1.369},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.6560000000000001,"5min":2.6560000000000001,"15min":2.6560000000000001},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.516,"5min":1.4590000000000001,"15min":1.488},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.5859999999999999,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.9399999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542210,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1368064,"data_stored":621160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.78800000000000003,"15min":0.84399999999999997},"min":{"1min":0.33100000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.48,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.89900000000000002,"15min":0.91700000000000004},"min":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.3799999999999999,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.36099999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.874,"15min":0.95399999999999996},"min":{"1min":0.41799999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.52,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.96699999999999997,"15min":1.05},"min":{"1min":0.46200000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.323,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":0.46400000000000002}]},{"osd":2,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.95999999999999996,"15min":1.0369999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.413,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.90800000000000003,"5min":1.073,"15min":1.133},"min":{"1min":0.443,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.4550000000000001,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":0.502}]},{"osd":4,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.2050000000000001,"15min":1.236},"min":{"1min":0.66600000000000004,"5min":0.501,"15min":0.501},"max":{"1min":1.5569999999999999,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.28,"15min":1.405},"min":{"1min":0.78200000000000003,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.478,"5min":4.819,"15min":4.819},"last":0.999}]},{"osd":5,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.5660000000000001,"15min":1.4339999999999999},"min":{"1min":0.77700000000000002,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.613,"5min":30.228000000000002,"15min":30.228000000000002},"last":0.77800000000000002},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.286,"15min":1.3400000000000001},"min":{"1min":0.64300000000000002,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.4470000000000001,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":0.751}]},{"osd":6,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.347,"15min":1.4870000000000001},"min":{"1min":0.80600000000000005,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.873,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.91200000000000003},{"interface":"front","average":{"1min":1.25,"5min":1.391,"15min":1.3939999999999999},"min":{"1min":0.872,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.7310000000000001,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":0.97799999999999998}]},{"osd":7,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.3400000000000001,"15min":1.4039999999999999},"min":{"1min":0.69799999999999995,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.649,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.4650000000000001,"15min":1.5640000000000001},"min":{"1min":0.76100000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.8069999999999999,"5min":10.663,"15min":10.663},"last":0.95899999999999996}]}]},{"osd":2,"up_from":14,"seq":60129542210,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1368064,"data_stored":621160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.81299999999999994,"15min":0.86899999999999999},"min":{"1min":0.39600000000000002,"5min":0.307,"15min":0.307},"max":{"1min":1.1899999999999999,"5min":1.456,"15min":1.456},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83599999999999997,"15min":0.92600000000000005},"min":{"1min":0.435,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.4059999999999999,"5min":2.226,"15min":2.226},"last":0.61499999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.92000000000000004,"15min":0.97899999999999998},"min":{"1min":0.52200000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.2070000000000001,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.84299999999999997,"15min":0.90500000000000003},"min":{"1min":0.499,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.4790000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.69899999999999995}]},{"osd":3,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":0.95899999999999996,"15min":1.0169999999999999},"min":{"1min":0.53700000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.018,"5min":1.0249999999999999,"15min":1.075},"min":{"1min":0.57799999999999996,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":0.75600000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.1659999999999999,"15min":1.222},"min":{"1min":0.69499999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.5509999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.226,"5min":1.1559999999999999,"15min":1.206},"min":{"1min":0.77300000000000002,"5min":0.627,"15min":0.627},"max":{"1min":2.4409999999999998,"5min":2.4409999999999998,"15min":2.4409999999999998},"last":2.4409999999999998}]},{"osd":5,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.29,"15min":1.3600000000000001},"min":{"1min":0.78900000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.0259999999999998,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.135},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.232,"15min":1.325},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.4079999999999999,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.212}]},{"osd":6,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.1970000000000001,"15min":1.3160000000000001},"min":{"1min":0.68100000000000005,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.55,"5min":2.456,"15min":2.456},"last":1.375},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3660000000000001,"15min":1.4650000000000001},"min":{"1min":0.90900000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.1299999999999999,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.236}]},{"osd":7,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.2849999999999999,"15min":1.3620000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.5389999999999999,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.25},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.2749999999999999,"15min":1.3859999999999999},"min":{"1min":0.74299999999999999,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.724,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.323}]}]},{"osd":1,"up_from":14,"seq":60129542209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32456,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1368064,"data_stored":621160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.76700000000000002,"15min":0.85299999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.0800000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.80900000000000005,"15min":0.82699999999999996},"min":{"1min":0.51800000000000002,"5min":0.374,"15min":0.374},"max":{"1min":1.115,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.85499999999999998}]},{"osd":2,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.90100000000000002,"15min":0.95699999999999996},"min":{"1min":0.39800000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.3160000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":1.139},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.90600000000000003,"15min":0.97199999999999998},"min":{"1min":0.52800000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.327,"5min":1.625,"15min":1.625},"last":1.0860000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.109,"15min":1.115},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.6839999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.95199999999999996,"15min":1.0049999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.7370000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":0.75600000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.1559999999999999,"15min":1.2669999999999999},"min":{"1min":0.748,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2310000000000001,"15min":1.3},"min":{"1min":0.81599999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6120000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":0.96799999999999997}]},{"osd":5,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2230000000000001,"15min":1.3120000000000001},"min":{"1min":0.98499999999999999,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.5289999999999999,"5min":3.593,"15min":3.593},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.304,"15min":1.4179999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":2.097,"5min":2.359,"15min":2.359},"last":1.8500000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.278,"15min":1.3819999999999999},"min":{"1min":0.79100000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.8180000000000001,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.214},{"interface":"front","average":{"1min":1.248,"5min":1.2609999999999999,"15min":1.345},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.641,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.335}]},{"osd":7,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.333,"15min":1.4239999999999999},"min":{"1min":0.89100000000000001,"5min":0.627,"15min":0.627},"max":{"1min":1.9670000000000001,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3260000000000001,"15min":1.474},"min":{"1min":0.83599999999999997,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.2810000000000001,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.7370000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542209,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33032,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338808,"statfs":{"total":96636764160,"available":96602939392,"internally_reserved":0,"allocated":1961984,"data_stored":1211528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.84699999999999998,"15min":0.92900000000000005},"min":{"1min":0.373,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.3779999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.93600000000000005,"15min":0.97799999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":0.86799999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.89000000000000001,"15min":0.97799999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.468,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.95999999999999996,"15min":1.0700000000000001},"min":{"1min":0.40100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.141,"5min":1.901,"15min":1.901},"last":0.72899999999999998}]},{"osd":3,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.89900000000000002,"15min":0.98499999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.4399999999999999,"5min":1.972,"15min":1.972},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.91800000000000004,"5min":1.0129999999999999,"15min":1.129},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.6020000000000001,"5min":2.298,"15min":2.298},"last":1.04}]},{"osd":4,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.2450000000000001,"15min":1.3220000000000001},"min":{"1min":0.83699999999999997,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.708,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.079,"5min":1.1799999999999999,"15min":1.256},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.5229999999999999,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.3069999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.27,"15min":1.3560000000000001},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.6679999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":0.999},{"interface":"front","average":{"1min":1.234,"5min":1.3300000000000001,"15min":1.4219999999999999},"min":{"1min":0.85399999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.722,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.3520000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.347,"15min":1.5},"min":{"1min":0.77600000000000002,"5min":0.747,"15min":0.747},"max":{"1min":1.998,"5min":3.032,"15min":3.032},"last":1.2},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.349,"15min":1.4670000000000001},"min":{"1min":0.90300000000000002,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.7629999999999999,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.155}]},{"osd":7,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.385,"15min":1.488},"min":{"1min":0.82799999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.8540000000000001,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.329},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.3089999999999999,"15min":1.401},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.1509999999999998,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.0940000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:52:04.067 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-23T03:46:48.337680+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=3, tm_min=46, tm_sec=48, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=3, tm_min=51, tm_sec=56, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T03:52:04.067 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-23T03:52:20.893 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:52:20.901 DEBUG:teuthology.orchestra.run.smithi173:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:52:24.067 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T03:52:24.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.280+0000 7f4528400640 1 Processor -- start 2024-03-23T03:52:24.281 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.280+0000 7f4528400640 1 -- start start 2024-03-23T03:52:24.282 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.282+0000 7f4528400640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f45200adbc0 0x7f45200b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.282+0000 7f4528400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45200cbab0 0x7f45200b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.282+0000 7f4528400640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45200b16a0 con 0x7f452006d390 2024-03-23T03:52:24.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.282+0000 7f4528400640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45200b4700 con 0x7f45200adbc0 2024-03-23T03:52:24.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.282+0000 7f4528400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45200cbfb0 con 0x7f45200cbab0 2024-03-23T03:52:24.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.282+0000 7f4526400640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f452006d390 0x7f45200b2aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37262/0 (socket says 172.21.15.102:37262) 2024-03-23T03:52:24.283 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.282+0000 7f4526400640 1 -- 172.21.15.102:0/518915294 learned_addr learned my addr 172.21.15.102:0/518915294 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:52:24.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f4525000640 1 --2- 172.21.15.102:0/518915294 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45200cbab0 0x7f45200b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:24.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f4525a00640 1 --2- 172.21.15.102:0/518915294 >> v2:172.21.15.173:3300/0 conn(0x7f45200adbc0 0x7f45200b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:24.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f451fe00640 1 -- 172.21.15.102:0/518915294 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860357774 0 0) 0x7f45200b16a0 con 0x7f452006d390 2024-03-23T03:52:24.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f451fe00640 1 -- 172.21.15.102:0/518915294 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4508003520 con 0x7f452006d390 2024-03-23T03:52:24.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f451fe00640 1 -- 172.21.15.102:0/518915294 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1608774005 0 0) 0x7f4508003520 con 0x7f452006d390 2024-03-23T03:52:24.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f451fe00640 1 -- 172.21.15.102:0/518915294 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45200cbab0 msgr2=0x7f45200b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:24.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f451fe00640 1 --2- 172.21.15.102:0/518915294 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45200cbab0 0x7f45200b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f451fe00640 1 -- 172.21.15.102:0/518915294 >> v2:172.21.15.173:3300/0 conn(0x7f45200adbc0 msgr2=0x7f45200b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:24.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f451fe00640 1 --2- 172.21.15.102:0/518915294 >> v2:172.21.15.173:3300/0 conn(0x7f45200adbc0 0x7f45200b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f451fe00640 1 -- 172.21.15.102:0/518915294 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45200cc290 con 0x7f452006d390 2024-03-23T03:52:24.285 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f451fe00640 1 -- 172.21.15.102:0/518915294 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f45180019f0 con 0x7f452006d390 2024-03-23T03:52:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f451fe00640 1 -- 172.21.15.102:0/518915294 >> v1:172.21.15.102:6789/0 conn(0x7f452006d390 legacy=0x7f45200b2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f451fe00640 1 --2- 172.21.15.102:0/518915294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4508003970 0x7f4508003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f451fe00640 1 --2- 172.21.15.102:0/518915294 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45200cbab0 0x7f45080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f451fe00640 1 --2- 172.21.15.102:0/518915294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4508004d00 0x7f4508005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f451fe00640 1 -- 172.21.15.102:0/518915294 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45200b4700 con 0x7f4508003970 2024-03-23T03:52:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f451fe00640 1 -- 172.21.15.102:0/518915294 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45200cbfb0 con 0x7f4508004d00 2024-03-23T03:52:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.283+0000 7f451fe00640 1 -- 172.21.15.102:0/518915294 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f450800ba80 con 0x7f45200cbab0 2024-03-23T03:52:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.284+0000 7f4525a00640 1 --2- 172.21.15.102:0/518915294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4508003970 0x7f4508003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.284+0000 7f4525000640 1 --2- 172.21.15.102:0/518915294 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45200cbab0 0x7f45080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.284+0000 7f4525a00640 1 -- 172.21.15.102:0/518915294 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45200cbab0 msgr2=0x7f45080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.284+0000 7f4525a00640 1 --2- 172.21.15.102:0/518915294 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f45200cbab0 0x7f45080047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.284+0000 7f4525a00640 1 -- 172.21.15.102:0/518915294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4508004d00 msgr2=0x7f4508005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.284+0000 7f4526400640 1 --2- 172.21.15.102:0/518915294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4508004d00 0x7f4508005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.284+0000 7f4525a00640 1 --2- 172.21.15.102:0/518915294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4508004d00 0x7f4508005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.286 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.284+0000 7f4525a00640 1 -- 172.21.15.102:0/518915294 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4508003520 con 0x7f4508003970 2024-03-23T03:52:24.287 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.284+0000 7f4526400640 1 --2- 172.21.15.102:0/518915294 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4508004d00 0x7f4508005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T03:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.285+0000 7f4525a00640 1 --2- 172.21.15.102:0/518915294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4508003970 0x7f4508003d90 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f451400ba60 tx=0x7f45080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0dbee6e2acc8a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.286+0000 7f451fe00640 1 -- 172.21.15.102:0/518915294 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4514032680 con 0x7f4508003970 2024-03-23T03:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.286+0000 7f451fe00640 1 -- 172.21.15.102:0/518915294 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45140327e0 con 0x7f4508003970 2024-03-23T03:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.286+0000 7f451fe00640 1 -- 172.21.15.102:0/518915294 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4514032b70 con 0x7f4508003970 2024-03-23T03:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.286+0000 7f4528400640 1 -- 172.21.15.102:0/518915294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4508003970 msgr2=0x7f4508003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.286+0000 7f4528400640 1 --2- 172.21.15.102:0/518915294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4508003970 0x7f4508003d90 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f451400ba60 tx=0x7f45080057f0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.286+0000 7f4526400640 1 -- 172.21.15.102:0/518915294 reap_dead start 2024-03-23T03:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.286+0000 7f4528400640 1 -- 172.21.15.102:0/518915294 shutdown_connections 2024-03-23T03:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.286+0000 7f4528400640 1 -- 172.21.15.102:0/518915294 >> 172.21.15.102:0/518915294 conn(0x7f452006a3d0 msgr2=0x7f45200b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.286+0000 7f4528400640 1 -- 172.21.15.102:0/518915294 shutdown_connections 2024-03-23T03:52:24.288 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.286+0000 7f4528400640 1 -- 172.21.15.102:0/518915294 wait complete. 2024-03-23T03:52:24.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.288+0000 7f4528400640 1 Processor -- start 2024-03-23T03:52:24.289 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.288+0000 7f4528400640 1 -- start start 2024-03-23T03:52:24.290 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.289+0000 7f4528400640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4508003970 0x7f452013c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.289+0000 7f4528400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f452013d160 0x7f4520140080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.289+0000 7f4528400640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f452013e210 0x7f452013e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.289+0000 7f4528400640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f452013ebd0 con 0x7f452013d160 2024-03-23T03:52:24.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.289+0000 7f4528400640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45201409d0 con 0x7f452013e210 2024-03-23T03:52:24.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.289+0000 7f4528400640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4520140cb0 con 0x7f4508003970 2024-03-23T03:52:24.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.290+0000 7f4525000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f452013d160 0x7f4520140080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:24.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.290+0000 7f4525000640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f452013d160 0x7f4520140080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48502/0 (socket says 172.21.15.102:48502) 2024-03-23T03:52:24.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.290+0000 7f4525000640 1 -- 172.21.15.102:0/3670757172 learned_addr learned my addr 172.21.15.102:0/3670757172 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:24.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.290+0000 7f4525000640 1 -- 172.21.15.102:0/3670757172 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4508003970 msgr2=0x7f452013c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:24.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.290+0000 7f4525000640 1 --2- 172.21.15.102:0/3670757172 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4508003970 0x7f452013c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.290+0000 7f4526400640 1 --2- 172.21.15.102:0/3670757172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f452013e210 0x7f452013e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:24.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.290+0000 7f4525000640 1 -- 172.21.15.102:0/3670757172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f452013e210 msgr2=0x7f452013e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:24.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.290+0000 7f4525000640 1 --2- 172.21.15.102:0/3670757172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f452013e210 0x7f452013e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.291 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.290+0000 7f4525000640 1 -- 172.21.15.102:0/3670757172 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4520140f90 con 0x7f452013d160 2024-03-23T03:52:24.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.290+0000 7f4525000640 1 --2- 172.21.15.102:0/3670757172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f452013d160 0x7f4520140080 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f450c011600 tx=0x7f450c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:24.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.291+0000 7f451d600640 1 -- 172.21.15.102:0/3670757172 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c00f920 con 0x7f452013d160 2024-03-23T03:52:24.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.291+0000 7f451d600640 1 -- 172.21.15.102:0/3670757172 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450c00fa80 con 0x7f452013d160 2024-03-23T03:52:24.292 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.291+0000 7f451d600640 1 -- 172.21.15.102:0/3670757172 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c00fdf0 con 0x7f452013d160 2024-03-23T03:52:24.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.291+0000 7f4528400640 1 -- 172.21.15.102:0/3670757172 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4520141270 con 0x7f452013d160 2024-03-23T03:52:24.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.291+0000 7f4528400640 1 -- 172.21.15.102:0/3670757172 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45201405c0 con 0x7f452013d160 2024-03-23T03:52:24.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.292+0000 7f4528400640 1 -- 172.21.15.102:0/3670757172 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f452006ac20 con 0x7f452013d160 2024-03-23T03:52:24.293 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.293+0000 7f451d600640 1 -- 172.21.15.102:0/3670757172 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f450c015460 con 0x7f452013d160 2024-03-23T03:52:24.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.294+0000 7f451d600640 1 --2- 172.21.15.102:0/3670757172 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f450006ac30 0x7f450006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.294+0000 7f451d600640 1 -- 172.21.15.102:0/3670757172 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f450c089780 con 0x7f452013d160 2024-03-23T03:52:24.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.294+0000 7f4525a00640 1 --2- 172.21.15.102:0/3670757172 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f450006ac30 0x7f450006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:24.295 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.295+0000 7f4525a00640 1 --2- 172.21.15.102:0/3670757172 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f450006ac30 0x7f450006d0f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f451400ba30 tx=0x7f4514032000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:24.299 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.299+0000 7f451d600640 1 -- 172.21.15.102:0/3670757172 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f450c0549f0 con 0x7f452013d160 2024-03-23T03:52:24.490 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.490+0000 7f4528400640 1 -- 172.21.15.102:0/3670757172 --> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4520073390 con 0x7f450006ac30 2024-03-23T03:52:24.493 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.492+0000 7f451d600640 1 -- 172.21.15.102:0/3670757172 <== mgr.4104 v2:172.21.15.102:6832/37097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48560 (secure 0 0 0) 0x7f4520073390 con 0x7f450006ac30 2024-03-23T03:52:24.493 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:52:24.494 INFO:teuthology.orchestra.run.smithi102.stderr:dumped all 2024-03-23T03:52:24.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.495+0000 7f4528400640 1 -- 172.21.15.102:0/3670757172 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f450006ac30 msgr2=0x7f450006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:24.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.495+0000 7f4528400640 1 --2- 172.21.15.102:0/3670757172 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f450006ac30 0x7f450006d0f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f451400ba30 tx=0x7f4514032000 comp rx=0 tx=0).stop 2024-03-23T03:52:24.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.495+0000 7f4528400640 1 -- 172.21.15.102:0/3670757172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f452013d160 msgr2=0x7f4520140080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:24.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.495+0000 7f4528400640 1 --2- 172.21.15.102:0/3670757172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f452013d160 0x7f4520140080 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f450c011600 tx=0x7f450c011ac0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.495+0000 7f4528400640 1 -- 172.21.15.102:0/3670757172 shutdown_connections 2024-03-23T03:52:24.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.495+0000 7f4528400640 1 --2- 172.21.15.102:0/3670757172 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f450006ac30 0x7f450006d0f0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.495+0000 7f4528400640 1 --2- 172.21.15.102:0/3670757172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f452013e210 0x7f452013e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.495+0000 7f4528400640 1 --2- 172.21.15.102:0/3670757172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f452013d160 0x7f4520140080 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.495+0000 7f4528400640 1 --2- 172.21.15.102:0/3670757172 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f4508003970 0x7f452013c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.495+0000 7f4528400640 1 -- 172.21.15.102:0/3670757172 >> 172.21.15.102:0/3670757172 conn(0x7f452006a3d0 msgr2=0x7f45200b3660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:24.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.495+0000 7f4528400640 1 -- 172.21.15.102:0/3670757172 shutdown_connections 2024-03-23T03:52:24.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.495+0000 7f4528400640 1 -- 172.21.15.102:0/3670757172 wait complete. 2024-03-23T03:52:24.512 INFO:teuthology.orchestra.run.smithi102.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-03-23T03:52:23.818585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":256676,"kb_used_data":11848,"kb_used_omap":12,"kb_used_meta":244723,"kb_avail":754718044,"statfs":{"total":773094113280,"available":772831277056,"internally_reserved":0,"allocated":12132352,"data_stored":6150016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12742,"internal_metadata":250596922},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003452"},"pg_stats":[{"pgid":"1.0","version":"110'119","reported_seq":248,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-03-23T03:52:05.241776+0000","last_change":"2024-03-23T03:52:05.241993+0000","last_active":"2024-03-23T03:52:05.241776+0000","last_peered":"2024-03-23T03:52:05.241776+0000","last_clean":"2024-03-23T03:52:05.241776+0000","last_became_active":"2024-03-23T03:46:49.733047+0000","last_became_peered":"2024-03-23T03:46:49.733047+0000","last_unstale":"2024-03-23T03:52:05.241776+0000","last_undegraded":"2024-03-23T03:52:05.241776+0000","last_fullsized":"2024-03-23T03:52:05.241776+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"110'119","last_scrub_stamp":"2024-03-23T03:52:05.241723+0000","last_deep_scrub":"110'119","last_deep_scrub_stamp":"2024-03-23T03:52:05.241723+0000","last_clean_scrub_stamp":"2024-03-23T03:52:05.241723+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T11:33:25.484093+0000","scrub_duration":0.011216267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542215,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1961984,"data_stored":1211528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:52:00 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0740000000000001,"15min":1.0680000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.478,"5min":1.635,"15min":1.635},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.0569999999999999,"15min":1.052},"min":{"1min":0.76100000000000001,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":1.419}]},{"osd":1,"last update":"Sat Mar 23 03:52:00 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.091,"15min":1.056},"min":{"1min":0.70899999999999996,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.6579999999999999,"5min":1.8959999999999999,"15min":1.8959999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.137,"15min":1.2689999999999999},"min":{"1min":0.80000000000000004,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.5109999999999999,"5min":4.9109999999999996,"15min":4.9109999999999996},"last":1.468}]},{"osd":2,"last update":"Sat Mar 23 03:52:00 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.111,"15min":1.147},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":1.6279999999999999,"5min":1.9690000000000001,"15min":1.9690000000000001},"last":1.387},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.2310000000000001,"15min":1.383},"min":{"1min":0.79500000000000004,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.631,"5min":4.8310000000000004,"15min":4.8310000000000004},"last":1.1850000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:52:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.242,"15min":1.3999999999999999},"min":{"1min":0.81999999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.5269999999999999,"5min":4.8639999999999999,"15min":4.8639999999999999},"last":1.246},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.3089999999999999,"15min":1.3480000000000001},"min":{"1min":0.90500000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"max":{"1min":1.7370000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":1.292}]},{"osd":4,"last update":"Sat Mar 23 03:52:00 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2649999999999999,"15min":1.2669999999999999},"min":{"1min":0.97399999999999998,"5min":0.62,"15min":0.62},"max":{"1min":1.641,"5min":2.2730000000000001,"15min":2.2730000000000001},"last":1.335},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.3360000000000001,"15min":1.3680000000000001},"min":{"1min":0.92900000000000005,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.722,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.135}]},{"osd":5,"last update":"Sat Mar 23 03:52:00 2024","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.367,"15min":1.411},"min":{"1min":0.98499999999999999,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.8,"5min":2.645,"15min":2.645},"last":1.367},{"interface":"front","average":{"1min":1.395,"5min":1.403,"15min":1.4410000000000001},"min":{"1min":0.84699999999999998,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.75,"5min":2.5089999999999999,"15min":2.5089999999999999},"last":1.51}]},{"osd":6,"last update":"Sat Mar 23 03:52:00 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.49,"15min":1.548},"min":{"1min":1.0649999999999999,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.772,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.554},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.454,"15min":1.508},"min":{"1min":1.0940000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.8049999999999999,"5min":2.7170000000000001,"15min":2.7170000000000001},"last":1.4790000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1368064,"data_stored":621160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:51:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0369999999999999,"15min":1.042},"min":{"1min":0.63300000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.671,"5min":1.7330000000000001,"15min":1.7330000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.016,"5min":1.046,"15min":1.0389999999999999},"min":{"1min":0.69299999999999995,"5min":0.62,"15min":0.62},"max":{"1min":1.486,"5min":1.946,"15min":1.946},"last":1.246}]},{"osd":1,"last update":"Sat Mar 23 03:51:58 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.081,"15min":1.073},"min":{"1min":0.70799999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.5509999999999999,"5min":1.6839999999999999,"15min":1.6839999999999999},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.107,"15min":1.149},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.4330000000000001,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.214}]},{"osd":2,"last update":"Sat Mar 23 03:51:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.083,"15min":1.071},"min":{"1min":0.72499999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.4359999999999999,"5min":2.044,"15min":2.044},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.113,"5min":1.1519999999999999,"15min":1.204},"min":{"1min":0.72299999999999998,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.4450000000000001,"5min":1.681,"15min":1.681},"last":1.151}]},{"osd":3,"last update":"Sat Mar 23 03:51:58 2024","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.222,"15min":1.2709999999999999},"min":{"1min":0.85999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.611,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.218,"15min":1.252},"min":{"1min":0.88,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":1.5169999999999999,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.96399999999999997}]},{"osd":4,"last update":"Sat Mar 23 03:51:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.3779999999999999,"15min":1.385},"min":{"1min":0.74299999999999999,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.796,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.373,"5min":1.3660000000000001,"15min":1.3839999999999999},"min":{"1min":0.72799999999999998,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.661,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.3799999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:51:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.375,"15min":1.423},"min":{"1min":0.63800000000000001,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.7,"5min":2.4340000000000002,"15min":2.4340000000000002},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.4890000000000001,"15min":1.5349999999999999},"min":{"1min":0.78800000000000003,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.728,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.4490000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:51:58 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3959999999999999,"15min":1.4079999999999999},"min":{"1min":0.66500000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.579,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.4490000000000001,"15min":1.52},"min":{"1min":0.80000000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.7609999999999999,"5min":2.5870000000000002,"15min":2.5870000000000002},"last":1.5900000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":28356,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343484,"statfs":{"total":96636764160,"available":96607727616,"internally_reserved":0,"allocated":1368064,"data_stored":621160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.087,"15min":1.1839999999999999},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.355,"5min":1.806,"15min":1.806},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.085,"5min":1.119,"15min":1.169},"min":{"1min":0.63200000000000001,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.4299999999999999,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.3089999999999999}]},{"osd":1,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.0329999999999999,"5min":1.079,"15min":1.095},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.4139999999999999,"5min":1.7669999999999999,"15min":1.7669999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.111,"5min":1.1459999999999999,"15min":1.1870000000000001},"min":{"1min":0.66100000000000003,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.706,"5min":1.8620000000000001,"15min":1.8620000000000001},"last":0.98899999999999999}]},{"osd":2,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.2190000000000001,"15min":1.327},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.4710000000000001,"5min":2.9199999999999999,"15min":2.9199999999999999},"last":1.236},{"interface":"front","average":{"1min":1.089,"5min":1.2370000000000001,"15min":1.345},"min":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.5760000000000001,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":1.018}]},{"osd":3,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.171,"15min":1.234},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.379,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.367},{"interface":"front","average":{"1min":1.216,"5min":1.321,"15min":1.4279999999999999},"min":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"max":{"1min":1.5700000000000001,"5min":2.9969999999999999,"15min":2.9969999999999999},"last":1.397}]},{"osd":4,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.129,"5min":1.2410000000000001,"15min":1.3779999999999999},"min":{"1min":0.65100000000000002,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.6000000000000001,"5min":2.0489999999999999,"15min":2.0489999999999999},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.206,"5min":1.335,"15min":1.419},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.6259999999999999,"5min":2.1419999999999999,"15min":2.1419999999999999},"last":1.0780000000000001}]},{"osd":6,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.3440000000000001,"15min":1.476},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.5029999999999999,"5min":2.4950000000000001,"15min":2.4950000000000001},"last":1.208},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.4610000000000001,"15min":1.6120000000000001},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.667,"5min":2.6299999999999999,"15min":2.6299999999999999},"last":1.331}]},{"osd":7,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.4219999999999999,"15min":1.5680000000000001},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.7529999999999999,"5min":2.3090000000000002,"15min":2.3090000000000002},"last":1.431},{"interface":"front","average":{"1min":1.413,"5min":1.52,"15min":1.6619999999999999},"min":{"1min":0.74299999999999999,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.833,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":1.4570000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1368064,"data_stored":621160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.087,"5min":1.028,"15min":0.98899999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.6970000000000001,"5min":2.6970000000000001,"15min":2.6970000000000001},"last":1.093},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0029999999999999,"15min":1.008},"min":{"1min":0.65900000000000003,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.952,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.874}]},{"osd":1,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":1.0529999999999999,"15min":1.0449999999999999},"min":{"1min":0.64800000000000002,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.282,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.109,"15min":1.071},"min":{"1min":0.61899999999999999,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.5820000000000001,"5min":1.8029999999999999,"15min":1.8029999999999999},"last":0.92900000000000005}]},{"osd":2,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.22,"15min":1.3},"min":{"1min":0.79300000000000004,"5min":0.625,"15min":0.625},"max":{"1min":1.55,"5min":3.0859999999999999,"15min":3.0859999999999999},"last":1.405},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.268,"15min":1.294},"min":{"1min":0.84699999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6679999999999999,"5min":2.181,"15min":2.181},"last":1.3680000000000001}]},{"osd":3,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.109,"15min":1.1220000000000001},"min":{"1min":0.63,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.5069999999999999,"5min":1.821,"15min":1.821},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.252,"5min":1.224,"15min":1.266},"min":{"1min":0.83599999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.6870000000000001,"5min":3.1749999999999998,"15min":3.1749999999999998},"last":1.0529999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.252,"15min":1.2649999999999999},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.7450000000000001,"5min":1.929,"15min":1.929},"last":1.234},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.274,"15min":1.2749999999999999},"min":{"1min":0.51100000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.532,"5min":2.8639999999999999,"15min":2.8639999999999999},"last":1.1679999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.359,"5min":1.357,"15min":1.405},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.847,"5min":3.2570000000000001,"15min":3.2570000000000001},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.5029999999999999,"5min":1.4359999999999999,"15min":1.4079999999999999},"min":{"1min":0.67400000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":2.8879999999999999,"5min":2.8879999999999999,"15min":2.8879999999999999},"last":1.4870000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.379,"15min":1.369},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":2.6560000000000001,"5min":2.6560000000000001,"15min":2.6560000000000001},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.516,"5min":1.4590000000000001,"15min":1.488},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":2.5859999999999999,"5min":2.9420000000000002,"15min":2.9420000000000002},"last":1.6950000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1368064,"data_stored":621160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.78800000000000003,"15min":0.84399999999999997},"min":{"1min":0.33100000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.48,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.89900000000000002,"15min":0.91700000000000004},"min":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.3799999999999999,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":1.335}]},{"osd":1,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.79400000000000004,"5min":0.874,"15min":0.95399999999999996},"min":{"1min":0.41799999999999998,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.52,"5min":3.1850000000000001,"15min":3.1850000000000001},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.96699999999999997,"15min":1.05},"min":{"1min":0.46200000000000002,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.323,"5min":2.7450000000000001,"15min":2.7450000000000001},"last":1.006}]},{"osd":2,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.95999999999999996,"15min":1.0369999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.413,"5min":2.5710000000000002,"15min":2.5710000000000002},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.90800000000000003,"5min":1.073,"15min":1.133},"min":{"1min":0.443,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.4550000000000001,"5min":4.8010000000000002,"15min":4.8010000000000002},"last":1.2}]},{"osd":4,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.2050000000000001,"15min":1.236},"min":{"1min":0.66600000000000004,"5min":0.501,"15min":0.501},"max":{"1min":1.5569999999999999,"5min":5.9210000000000003,"15min":5.9210000000000003},"last":1.0569999999999999},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.28,"15min":1.405},"min":{"1min":0.78200000000000003,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.478,"5min":4.819,"15min":4.819},"last":1.3999999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.5660000000000001,"15min":1.4339999999999999},"min":{"1min":0.77700000000000002,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.613,"5min":30.228000000000002,"15min":30.228000000000002},"last":1.3149999999999999},{"interface":"front","average":{"1min":1.1100000000000001,"5min":1.286,"15min":1.3400000000000001},"min":{"1min":0.64300000000000002,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.4470000000000001,"5min":5.8920000000000003,"15min":5.8920000000000003},"last":1.1579999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.347,"15min":1.4870000000000001},"min":{"1min":0.80600000000000005,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.873,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":1.099},{"interface":"front","average":{"1min":1.25,"5min":1.391,"15min":1.3939999999999999},"min":{"1min":0.872,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.7310000000000001,"5min":4.7610000000000001,"15min":4.7610000000000001},"last":1.3740000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.3400000000000001,"15min":1.4039999999999999},"min":{"1min":0.69799999999999995,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":1.649,"5min":5.9589999999999996,"15min":5.9589999999999996},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.4650000000000001,"15min":1.5640000000000001},"min":{"1min":0.76100000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.8069999999999999,"5min":10.663,"15min":10.663},"last":1.272}]}]},{"osd":2,"up_from":14,"seq":60129542214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32452,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1368064,"data_stored":621160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.81299999999999994,"15min":0.86899999999999999},"min":{"1min":0.39600000000000002,"5min":0.307,"15min":0.307},"max":{"1min":1.1899999999999999,"5min":1.456,"15min":1.456},"last":1.2709999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.83599999999999997,"15min":0.92600000000000005},"min":{"1min":0.435,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.4059999999999999,"5min":2.226,"15min":2.226},"last":0.72099999999999997}]},{"osd":1,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.91500000000000004,"5min":0.92000000000000004,"15min":0.97899999999999998},"min":{"1min":0.52200000000000002,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.2070000000000001,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.84299999999999997,"15min":0.90500000000000003},"min":{"1min":0.499,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.4790000000000001,"5min":2.1739999999999999,"15min":2.1739999999999999},"last":0.95399999999999996}]},{"osd":3,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":0.95899999999999996,"15min":1.0169999999999999},"min":{"1min":0.53700000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.444},{"interface":"front","average":{"1min":1.018,"5min":1.0249999999999999,"15min":1.075},"min":{"1min":0.57799999999999996,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.665,"5min":1.665,"15min":1.665},"last":1.5800000000000001}]},{"osd":4,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.1659999999999999,"15min":1.222},"min":{"1min":0.69499999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.5509999999999999,"5min":1.9119999999999999,"15min":1.9119999999999999},"last":1.085},{"interface":"front","average":{"1min":1.226,"5min":1.1559999999999999,"15min":1.206},"min":{"1min":0.77300000000000002,"5min":0.627,"15min":0.627},"max":{"1min":2.4409999999999998,"5min":2.4409999999999998,"15min":2.4409999999999998},"last":1.5289999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.29,"15min":1.3600000000000001},"min":{"1min":0.78900000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":2.0259999999999998,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.78},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.232,"15min":1.325},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.4079999999999999,"5min":1.8500000000000001,"15min":1.8500000000000001},"last":1.196}]},{"osd":6,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.1970000000000001,"15min":1.3160000000000001},"min":{"1min":0.68100000000000005,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.55,"5min":2.456,"15min":2.456},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.3660000000000001,"15min":1.4650000000000001},"min":{"1min":0.90900000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"max":{"1min":2.1299999999999999,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.6910000000000001}]},{"osd":7,"last update":"Sat Mar 23 03:51:59 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.2849999999999999,"15min":1.3620000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.5389999999999999,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":1.151},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.2749999999999999,"15min":1.3859999999999999},"min":{"1min":0.74299999999999999,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":1.724,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":1.4019999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32456,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1368064,"data_stored":621160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.76700000000000002,"15min":0.85299999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":1.0800000000000001,"5min":2.2749999999999999,"15min":2.2749999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.80900000000000005,"15min":0.82699999999999996},"min":{"1min":0.51800000000000002,"5min":0.374,"15min":0.374},"max":{"1min":1.115,"5min":1.5600000000000001,"15min":1.5600000000000001},"last":0.83099999999999996}]},{"osd":2,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.90100000000000002,"15min":0.95699999999999996},"min":{"1min":0.39800000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.3160000000000001,"5min":1.6990000000000001,"15min":1.6990000000000001},"last":0.996},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.90600000000000003,"15min":0.97199999999999998},"min":{"1min":0.52800000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.327,"5min":1.625,"15min":1.625},"last":1.073}]},{"osd":3,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.109,"15min":1.115},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.6839999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.282},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.95199999999999996,"15min":1.0049999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.7370000000000001,"5min":3.2410000000000001,"15min":3.2410000000000001},"last":1.4079999999999999}]},{"osd":4,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.1559999999999999,"15min":1.2669999999999999},"min":{"1min":0.748,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":1.123},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.2310000000000001,"15min":1.3},"min":{"1min":0.81599999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6120000000000001,"5min":2.3100000000000001,"15min":2.3100000000000001},"last":1.196}]},{"osd":5,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.2230000000000001,"15min":1.3120000000000001},"min":{"1min":0.98499999999999999,"5min":0.58299999999999996,"15min":0.58299999999999996},"max":{"1min":1.5289999999999999,"5min":3.593,"15min":3.593},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.304,"15min":1.4179999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":2.097,"5min":2.359,"15min":2.359},"last":1.5529999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.278,"15min":1.3819999999999999},"min":{"1min":0.79100000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.8180000000000001,"5min":2.4319999999999999,"15min":2.4319999999999999},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.248,"5min":1.2609999999999999,"15min":1.345},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.641,"5min":3.7429999999999999,"15min":3.7429999999999999},"last":1.502}]},{"osd":7,"last update":"Sat Mar 23 03:51:57 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.333,"15min":1.4239999999999999},"min":{"1min":0.89100000000000001,"5min":0.627,"15min":0.627},"max":{"1min":1.9670000000000001,"5min":3.8839999999999999,"15min":3.8839999999999999},"last":1.597},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3260000000000001,"15min":1.474},"min":{"1min":0.83599999999999997,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":2.2810000000000001,"5min":3.3079999999999998,"15min":3.3079999999999998},"last":1.3959999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542213,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33028,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338812,"statfs":{"total":96636764160,"available":96602943488,"internally_reserved":0,"allocated":1961984,"data_stored":1211528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.84699999999999998,"15min":0.92900000000000005},"min":{"1min":0.373,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.3779999999999999,"5min":1.5009999999999999,"15min":1.5009999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.93799999999999994,"5min":0.93600000000000005,"15min":0.97799999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.643,"5min":1.643,"15min":1.643},"last":1.2490000000000001}]},{"osd":2,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.89000000000000001,"15min":0.97799999999999998},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.468,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.95999999999999996,"15min":1.0700000000000001},"min":{"1min":0.40100000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.141,"5min":1.901,"15min":1.901},"last":0.99199999999999999}]},{"osd":3,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.89900000000000002,"15min":0.98499999999999999},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.4399999999999999,"5min":1.972,"15min":1.972},"last":1.111},{"interface":"front","average":{"1min":0.91800000000000004,"5min":1.0129999999999999,"15min":1.129},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.6020000000000001,"5min":2.298,"15min":2.298},"last":1.052}]},{"osd":4,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.2450000000000001,"15min":1.3220000000000001},"min":{"1min":0.83699999999999997,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.708,"5min":2.4940000000000002,"15min":2.4940000000000002},"last":1.496},{"interface":"front","average":{"1min":1.079,"5min":1.1799999999999999,"15min":1.256},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.5229999999999999,"5min":2.2400000000000002,"15min":2.2400000000000002},"last":1.2949999999999999}]},{"osd":5,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.27,"15min":1.3560000000000001},"min":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.6679999999999999,"5min":2.3799999999999999,"15min":2.3799999999999999},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.234,"5min":1.3300000000000001,"15min":1.4219999999999999},"min":{"1min":0.85399999999999998,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.722,"5min":2.9449999999999998,"15min":2.9449999999999998},"last":1.3919999999999999}]},{"osd":6,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.347,"15min":1.5},"min":{"1min":0.77600000000000002,"5min":0.747,"15min":0.747},"max":{"1min":1.998,"5min":3.032,"15min":3.032},"last":1.327},{"interface":"front","average":{"1min":1.2170000000000001,"5min":1.349,"15min":1.4670000000000001},"min":{"1min":0.90300000000000002,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.7629999999999999,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":1.4159999999999999}]},{"osd":7,"last update":"Sat Mar 23 03:51:54 2024","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.385,"15min":1.488},"min":{"1min":0.82799999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.8540000000000001,"5min":2.5249999999999999,"15min":2.5249999999999999},"last":1.363},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.3089999999999999,"15min":1.401},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"max":{"1min":2.1509999999999998,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":1.4470000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T03:52:24.514 DEBUG:teuthology.orchestra.run.smithi102:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-23T03:52:24.712 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.712+0000 7f74e5200640 1 Processor -- start 2024-03-23T03:52:24.713 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.712+0000 7f74e5200640 1 -- start start 2024-03-23T03:52:24.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.713+0000 7f74e5200640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f74e0098a40 0x7f74e0098e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.713+0000 7f74e5200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f74e00cf120 0x7f74e015e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.713+0000 7f74e5200640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e0099360 con 0x7f74e015eb50 2024-03-23T03:52:24.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.713+0000 7f74e5200640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74e00cb5d0 con 0x7f74e0098a40 2024-03-23T03:52:24.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.713+0000 7f74e5200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e00cb8b0 con 0x7f74e00cf120 2024-03-23T03:52:24.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.713+0000 7f74de800640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f74e015eb50 0x7f74e00b34e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37276/0 (socket says 172.21.15.102:37276) 2024-03-23T03:52:24.714 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.713+0000 7f74de800640 1 -- 172.21.15.102:0/3392625968 learned_addr learned my addr 172.21.15.102:0/3392625968 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:52:24.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.714+0000 7f74dd400640 1 --2- 172.21.15.102:0/3392625968 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f74e00cf120 0x7f74e015e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:24.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.714+0000 7f74dde00640 1 --2- 172.21.15.102:0/3392625968 >> v2:172.21.15.173:3300/0 conn(0x7f74e0098a40 0x7f74e0098e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:24.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.714+0000 7f74dca00640 1 -- 172.21.15.102:0/3392625968 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754091892 0 0) 0x7f74e0099360 con 0x7f74e015eb50 2024-03-23T03:52:24.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.714+0000 7f74dca00640 1 -- 172.21.15.102:0/3392625968 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c4003520 con 0x7f74e015eb50 2024-03-23T03:52:24.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.714+0000 7f74dca00640 1 -- 172.21.15.102:0/3392625968 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 517230963 0 0) 0x7f74c4003520 con 0x7f74e015eb50 2024-03-23T03:52:24.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.714+0000 7f74dca00640 1 -- 172.21.15.102:0/3392625968 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f74e00cf120 msgr2=0x7f74e015e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:24.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.714+0000 7f74dca00640 1 --2- 172.21.15.102:0/3392625968 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f74e00cf120 0x7f74e015e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.714+0000 7f74dca00640 1 -- 172.21.15.102:0/3392625968 >> v2:172.21.15.173:3300/0 conn(0x7f74e0098a40 msgr2=0x7f74e0098e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:24.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.714+0000 7f74dca00640 1 --2- 172.21.15.102:0/3392625968 >> v2:172.21.15.173:3300/0 conn(0x7f74e0098a40 0x7f74e0098e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.716 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.714+0000 7f74dca00640 1 -- 172.21.15.102:0/3392625968 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e00cbb90 con 0x7f74e015eb50 2024-03-23T03:52:24.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.714+0000 7f74dca00640 1 -- 172.21.15.102:0/3392625968 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f74d80019f0 con 0x7f74e015eb50 2024-03-23T03:52:24.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.714+0000 7f74dca00640 1 -- 172.21.15.102:0/3392625968 >> v1:172.21.15.102:6789/0 conn(0x7f74e015eb50 legacy=0x7f74e00b34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:24.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.715+0000 7f74dca00640 1 --2- 172.21.15.102:0/3392625968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74c4003970 0x7f74c4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.715+0000 7f74dca00640 1 --2- 172.21.15.102:0/3392625968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74c40047c0 0x7f74c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.715+0000 7f74dca00640 1 --2- 172.21.15.102:0/3392625968 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f74e00cf120 0x7f74c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.715+0000 7f74dca00640 1 -- 172.21.15.102:0/3392625968 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e00cb5d0 con 0x7f74c4003970 2024-03-23T03:52:24.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.715+0000 7f74dca00640 1 -- 172.21.15.102:0/3392625968 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e00cb8b0 con 0x7f74c40047c0 2024-03-23T03:52:24.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.715+0000 7f74dca00640 1 -- 172.21.15.102:0/3392625968 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c400b980 con 0x7f74e00cf120 2024-03-23T03:52:24.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.715+0000 7f74dde00640 1 --2- 172.21.15.102:0/3392625968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74c4003970 0x7f74c4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:24.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.715+0000 7f74dde00640 1 -- 172.21.15.102:0/3392625968 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f74e00cf120 msgr2=0x7f74c4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:24.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.715+0000 7f74dde00640 1 --2- 172.21.15.102:0/3392625968 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f74e00cf120 0x7f74c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.715+0000 7f74dde00640 1 -- 172.21.15.102:0/3392625968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74c40047c0 msgr2=0x7f74c4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:24.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.715+0000 7f74dd400640 1 --2- 172.21.15.102:0/3392625968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74c40047c0 0x7f74c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:24.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.715+0000 7f74dde00640 1 --2- 172.21.15.102:0/3392625968 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74c40047c0 0x7f74c4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.715+0000 7f74dde00640 1 -- 172.21.15.102:0/3392625968 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c4003520 con 0x7f74c4003970 2024-03-23T03:52:24.718 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.716+0000 7f74dde00640 1 --2- 172.21.15.102:0/3392625968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74c4003970 0x7f74c4003d90 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f74d0009ac0 tx=0x7f74c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec3d7337bbef3f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:24.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.717+0000 7f74dca00640 1 -- 172.21.15.102:0/3392625968 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d002c710 con 0x7f74c4003970 2024-03-23T03:52:24.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.717+0000 7f74dca00640 1 -- 172.21.15.102:0/3392625968 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d002c870 con 0x7f74c4003970 2024-03-23T03:52:24.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.717+0000 7f74dca00640 1 -- 172.21.15.102:0/3392625968 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d002cc00 con 0x7f74c4003970 2024-03-23T03:52:24.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.717+0000 7f74e5200640 1 -- 172.21.15.102:0/3392625968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74c4003970 msgr2=0x7f74c4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:24.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.717+0000 7f74e5200640 1 --2- 172.21.15.102:0/3392625968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74c4003970 0x7f74c4003d90 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f74d0009ac0 tx=0x7f74c40057f0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.717+0000 7f74de800640 1 -- 172.21.15.102:0/3392625968 reap_dead start 2024-03-23T03:52:24.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.717+0000 7f74e5200640 1 -- 172.21.15.102:0/3392625968 shutdown_connections 2024-03-23T03:52:24.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.717+0000 7f74e5200640 1 -- 172.21.15.102:0/3392625968 >> 172.21.15.102:0/3392625968 conn(0x7f74e00b2180 msgr2=0x7f74e00b2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:24.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.717+0000 7f74e5200640 1 -- 172.21.15.102:0/3392625968 shutdown_connections 2024-03-23T03:52:24.719 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.718+0000 7f74e5200640 1 -- 172.21.15.102:0/3392625968 wait complete. 2024-03-23T03:52:24.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.719+0000 7f74e5200640 1 Processor -- start 2024-03-23T03:52:24.720 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.719+0000 7f74e5200640 1 -- start start 2024-03-23T03:52:24.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.720+0000 7f74e5200640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74c4003970 0x7f74e0140240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.720+0000 7f74e5200640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74e00c3d10 0x7f74e0142c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.720+0000 7f74e5200640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f74e00c4830 0x7f74e00b4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.720+0000 7f74e5200640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e01ee7a0 con 0x7f74e00c3d10 2024-03-23T03:52:24.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.720+0000 7f74e5200640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e00c62d0 con 0x7f74c4003970 2024-03-23T03:52:24.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.720+0000 7f74e5200640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e00c65b0 con 0x7f74e00c4830 2024-03-23T03:52:24.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.721+0000 7f74dd400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74e00c3d10 0x7f74e0142c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:24.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.721+0000 7f74dd400640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74e00c3d10 0x7f74e0142c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48516/0 (socket says 172.21.15.102:48516) 2024-03-23T03:52:24.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.721+0000 7f74dd400640 1 -- 172.21.15.102:0/748697055 learned_addr learned my addr 172.21.15.102:0/748697055 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:24.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.721+0000 7f74dde00640 1 --2- 172.21.15.102:0/748697055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74c4003970 0x7f74e0140240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:24.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.721+0000 7f74de800640 1 --2- 172.21.15.102:0/748697055 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f74e00c4830 0x7f74e00b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:24.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.721+0000 7f74dd400640 1 -- 172.21.15.102:0/748697055 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f74e00c4830 msgr2=0x7f74e00b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:24.722 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.721+0000 7f74dd400640 1 --2- 172.21.15.102:0/748697055 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f74e00c4830 0x7f74e00b4660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.721+0000 7f74dd400640 1 -- 172.21.15.102:0/748697055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74c4003970 msgr2=0x7f74e0140240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:24.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.721+0000 7f74dd400640 1 --2- 172.21.15.102:0/748697055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74c4003970 0x7f74e0140240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.721+0000 7f74dd400640 1 -- 172.21.15.102:0/748697055 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e00c6890 con 0x7f74e00c3d10 2024-03-23T03:52:24.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.721+0000 7f74dd400640 1 --2- 172.21.15.102:0/748697055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74e00c3d10 0x7f74e0142c80 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f74c8010860 tx=0x7f74c8010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:24.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.722+0000 7f74d6000640 1 -- 172.21.15.102:0/748697055 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c8009610 con 0x7f74e00c3d10 2024-03-23T03:52:24.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.722+0000 7f74d6000640 1 -- 172.21.15.102:0/748697055 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c800eb60 con 0x7f74e00c3d10 2024-03-23T03:52:24.723 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.722+0000 7f74d6000640 1 -- 172.21.15.102:0/748697055 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c801c400 con 0x7f74e00c3d10 2024-03-23T03:52:24.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.722+0000 7f74e5200640 1 -- 172.21.15.102:0/748697055 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e00c6b70 con 0x7f74e00c3d10 2024-03-23T03:52:24.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.722+0000 7f74e5200640 1 -- 172.21.15.102:0/748697055 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e01431c0 con 0x7f74e00c3d10 2024-03-23T03:52:24.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.723+0000 7f74e5200640 1 -- 172.21.15.102:0/748697055 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e00b1f80 con 0x7f74e00c3d10 2024-03-23T03:52:24.724 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.724+0000 7f74d6000640 1 -- 172.21.15.102:0/748697055 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f74c801c560 con 0x7f74e00c3d10 2024-03-23T03:52:24.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.725+0000 7f74d6000640 1 --2- 172.21.15.102:0/748697055 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f74bc06ac30 0x7f74bc06d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:24.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.725+0000 7f74d6000640 1 -- 172.21.15.102:0/748697055 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f74c8089700 con 0x7f74e00c3d10 2024-03-23T03:52:24.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.725+0000 7f74dde00640 1 --2- 172.21.15.102:0/748697055 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f74bc06ac30 0x7f74bc06d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:24.726 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.726+0000 7f74dde00640 1 --2- 172.21.15.102:0/748697055 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f74bc06ac30 0x7f74bc06d0f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f74d0036000 tx=0x7f74d0030000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:24.730 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.730+0000 7f74d6000640 1 -- 172.21.15.102:0/748697055 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f74c8054b40 con 0x7f74e00c3d10 2024-03-23T03:52:24.930 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.929+0000 7f74e5200640 1 -- 172.21.15.102:0/748697055 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f74e00cfc70 con 0x7f74e00c3d10 2024-03-23T03:52:24.938 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.938+0000 7f74d6000640 1 -- 172.21.15.102:0/748697055 <== mon.0 v2:172.21.15.102:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f74c8054b40 con 0x7f74e00c3d10 2024-03-23T03:52:24.939 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.938+0000 7f74d6000640 1 -- 172.21.15.102:0/748697055 <== mon.0 v2:172.21.15.102:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v22)=0 v22) v1 ==== 123+0+0 (secure 0 0 0) 0x7f74c8058160 con 0x7f74e00c3d10 2024-03-23T03:52:24.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.942+0000 7f74e5200640 1 -- 172.21.15.102:0/748697055 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f74bc06ac30 msgr2=0x7f74bc06d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:24.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.942+0000 7f74e5200640 1 --2- 172.21.15.102:0/748697055 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f74bc06ac30 0x7f74bc06d0f0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f74d0036000 tx=0x7f74d0030000 comp rx=0 tx=0).stop 2024-03-23T03:52:24.943 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.942+0000 7f74e5200640 1 -- 172.21.15.102:0/748697055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74e00c3d10 msgr2=0x7f74e0142c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:24.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.942+0000 7f74e5200640 1 --2- 172.21.15.102:0/748697055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74e00c3d10 0x7f74e0142c80 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f74c8010860 tx=0x7f74c8010d20 comp rx=0 tx=0).stop 2024-03-23T03:52:24.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.943+0000 7f74e5200640 1 -- 172.21.15.102:0/748697055 shutdown_connections 2024-03-23T03:52:24.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.943+0000 7f74e5200640 1 --2- 172.21.15.102:0/748697055 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f74bc06ac30 0x7f74bc06d0f0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.943+0000 7f74e5200640 1 --2- 172.21.15.102:0/748697055 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f74e00c4830 0x7f74e00b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.943+0000 7f74e5200640 1 --2- 172.21.15.102:0/748697055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74e00c3d10 0x7f74e0142c80 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.943+0000 7f74e5200640 1 --2- 172.21.15.102:0/748697055 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f74c4003970 0x7f74e0140240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:24.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.943+0000 7f74e5200640 1 -- 172.21.15.102:0/748697055 >> 172.21.15.102:0/748697055 conn(0x7f74e00b2180 msgr2=0x7f74e006bbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:24.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.943+0000 7f74e5200640 1 -- 172.21.15.102:0/748697055 shutdown_connections 2024-03-23T03:52:24.944 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:24.943+0000 7f74e5200640 1 -- 172.21.15.102:0/748697055 wait complete. 2024-03-23T03:52:24.961 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T03:52:24.961 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T03:52:25.165 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.165+0000 7f51f2e00640 1 Processor -- start 2024-03-23T03:52:25.166 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.165+0000 7f51f2e00640 1 -- start start 2024-03-23T03:52:25.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.166+0000 7f51f2e00640 1 --2- >> v2:172.21.15.173:3300/0 conn(0x7f51ec15ab20 0x7f51ec071d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:25.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.167+0000 7f51f2e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f51ec06f4f0 0x7f51ec069c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:25.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.167+0000 7f51f2e00640 1 -- --> v1:172.21.15.102:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51ec0722d0 con 0x7f51ec06a1d0 2024-03-23T03:52:25.167 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.167+0000 7f51f2e00640 1 -- --> v2:172.21.15.173:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51ec06a660 con 0x7f51ec15ab20 2024-03-23T03:52:25.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.167+0000 7f51f2e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec0d0440 con 0x7f51ec06f4f0 2024-03-23T03:52:25.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.167+0000 7f51f0e00640 1 --1- >> v1:172.21.15.102:6789/0 conn(0x7f51ec06a1d0 0x7f51ec15e6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.102:6789/0 says I am v1:172.21.15.102:37286/0 (socket says 172.21.15.102:37286) 2024-03-23T03:52:25.168 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.167+0000 7f51f0e00640 1 -- 172.21.15.102:0/465093778 learned_addr learned my addr 172.21.15.102:0/465093778 (peer_addr_for_me v1:172.21.15.102:0/0) 2024-03-23T03:52:25.169 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.167+0000 7f51ebe00640 1 --2- 172.21.15.102:0/465093778 >> v2:172.21.15.173:3300/0 conn(0x7f51ec15ab20 0x7f51ec071d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:25.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.167+0000 7f51eb400640 1 --2- 172.21.15.102:0/465093778 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f51ec06f4f0 0x7f51ec069c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:25.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.167+0000 7f51eaa00640 1 -- 172.21.15.102:0/465093778 <== mon.0 v1:172.21.15.102:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906877923 0 0) 0x7f51e4001490 con 0x7f51ec06a1d0 2024-03-23T03:52:25.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.168+0000 7f51eaa00640 1 -- 172.21.15.102:0/465093778 --> v1:172.21.15.102:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51d0003520 con 0x7f51ec06a1d0 2024-03-23T03:52:25.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.168+0000 7f51eaa00640 1 -- 172.21.15.102:0/465093778 <== mon.0 v1:172.21.15.102:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1274477154 0 0) 0x7f51d0003520 con 0x7f51ec06a1d0 2024-03-23T03:52:25.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.168+0000 7f51eaa00640 1 -- 172.21.15.102:0/465093778 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f51ec06f4f0 msgr2=0x7f51ec069c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:25.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.168+0000 7f51eaa00640 1 --2- 172.21.15.102:0/465093778 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f51ec06f4f0 0x7f51ec069c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:25.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.168+0000 7f51eaa00640 1 -- 172.21.15.102:0/465093778 >> v2:172.21.15.173:3300/0 conn(0x7f51ec15ab20 msgr2=0x7f51ec071d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:25.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.168+0000 7f51eaa00640 1 --2- 172.21.15.102:0/465093778 >> v2:172.21.15.173:3300/0 conn(0x7f51ec15ab20 0x7f51ec071d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:25.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.168+0000 7f51eaa00640 1 -- 172.21.15.102:0/465093778 --> v1:172.21.15.102:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ec0cf3a0 con 0x7f51ec06a1d0 2024-03-23T03:52:25.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.168+0000 7f51eaa00640 1 -- 172.21.15.102:0/465093778 <== mon.0 v1:172.21.15.102:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1700659169 0 0) 0x7f51e4001b80 con 0x7f51ec06a1d0 2024-03-23T03:52:25.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.168+0000 7f51ebe00640 1 --2- 172.21.15.102:0/465093778 >> v2:172.21.15.173:3300/0 conn(0x7f51ec15ab20 0x7f51ec071d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T03:52:25.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.168+0000 7f51eaa00640 1 -- 172.21.15.102:0/465093778 >> v1:172.21.15.102:6789/0 conn(0x7f51ec06a1d0 legacy=0x7f51ec15e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:25.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.168+0000 7f51eaa00640 1 --2- 172.21.15.102:0/465093778 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f51ec06f4f0 0x7f51d00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:25.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.168+0000 7f51eaa00640 1 --2- 172.21.15.102:0/465093778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51d0003f20 0x7f51d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:25.170 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.168+0000 7f51eaa00640 1 --2- 172.21.15.102:0/465093778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51d0004d00 0x7f51d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:25.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.168+0000 7f51eaa00640 1 -- 172.21.15.102:0/465093778 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec06a660 con 0x7f51d0003f20 2024-03-23T03:52:25.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.168+0000 7f51eaa00640 1 -- 172.21.15.102:0/465093778 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec0d0440 con 0x7f51d0004d00 2024-03-23T03:52:25.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.168+0000 7f51eaa00640 1 -- 172.21.15.102:0/465093778 --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51d000ba00 con 0x7f51ec06f4f0 2024-03-23T03:52:25.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.168+0000 7f51eb400640 1 --2- 172.21.15.102:0/465093778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51d0003f20 0x7f51d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:25.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.169+0000 7f51ebe00640 1 --2- 172.21.15.102:0/465093778 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f51ec06f4f0 0x7f51d00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:25.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.169+0000 7f51f0e00640 1 --2- 172.21.15.102:0/465093778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51d0004d00 0x7f51d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:25.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.169+0000 7f51eb400640 1 -- 172.21.15.102:0/465093778 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f51ec06f4f0 msgr2=0x7f51d00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:25.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.169+0000 7f51eb400640 1 --2- 172.21.15.102:0/465093778 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f51ec06f4f0 0x7f51d00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:25.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.169+0000 7f51eb400640 1 -- 172.21.15.102:0/465093778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51d0004d00 msgr2=0x7f51d0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T03:52:25.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.169+0000 7f51eb400640 1 --2- 172.21.15.102:0/465093778 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51d0004d00 0x7f51d0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:25.171 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.169+0000 7f51eb400640 1 -- 172.21.15.102:0/465093778 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51d0003520 con 0x7f51d0003f20 2024-03-23T03:52:25.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.170+0000 7f51eb400640 1 --2- 172.21.15.102:0/465093778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51d0003f20 0x7f51d00047c0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f51d8009ac0 tx=0x7f51d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f35e68a4a8da38d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:25.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.170+0000 7f51eaa00640 1 -- 172.21.15.102:0/465093778 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d802c6b0 con 0x7f51d0003f20 2024-03-23T03:52:25.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.170+0000 7f51eaa00640 1 -- 172.21.15.102:0/465093778 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f51d802c810 con 0x7f51d0003f20 2024-03-23T03:52:25.172 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.170+0000 7f51eaa00640 1 -- 172.21.15.102:0/465093778 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51d802cbc0 con 0x7f51d0003f20 2024-03-23T03:52:25.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.171+0000 7f51f2e00640 1 -- 172.21.15.102:0/465093778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51d0003f20 msgr2=0x7f51d00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:25.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.171+0000 7f51f2e00640 1 --2- 172.21.15.102:0/465093778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51d0003f20 0x7f51d00047c0 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f51d8009ac0 tx=0x7f51d00057f0 comp rx=0 tx=0).stop 2024-03-23T03:52:25.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.171+0000 7f51f0e00640 1 -- 172.21.15.102:0/465093778 reap_dead start 2024-03-23T03:52:25.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.171+0000 7f51f2e00640 1 -- 172.21.15.102:0/465093778 shutdown_connections 2024-03-23T03:52:25.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.171+0000 7f51f2e00640 1 -- 172.21.15.102:0/465093778 >> 172.21.15.102:0/465093778 conn(0x7f51ec06d380 msgr2=0x7f51ec06d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:25.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.171+0000 7f51f2e00640 1 -- 172.21.15.102:0/465093778 shutdown_connections 2024-03-23T03:52:25.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.171+0000 7f51f2e00640 1 -- 172.21.15.102:0/465093778 wait complete. 2024-03-23T03:52:25.173 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.173+0000 7f51f2e00640 1 Processor -- start 2024-03-23T03:52:25.174 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.173+0000 7f51f2e00640 1 -- start start 2024-03-23T03:52:25.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.174+0000 7f51f2e00640 1 --2- >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f51d0003f20 0x7f51ec1e8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:25.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.174+0000 7f51f2e00640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51ec1493c0 0x7f51ec0a7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:25.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.175+0000 7f51f2e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51ec0aa580 0x7f51ec0aaa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:25.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.175+0000 7f51f2e00640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec0aaf50 con 0x7f51ec0aa580 2024-03-23T03:52:25.177 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.175+0000 7f51f2e00640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec0ade00 con 0x7f51ec1493c0 2024-03-23T03:52:25.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.175+0000 7f51f2e00640 1 -- --> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51ec156a00 con 0x7f51d0003f20 2024-03-23T03:52:25.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.175+0000 7f51f0e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51ec0aa580 0x7f51ec0aaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:25.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.175+0000 7f51f0e00640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51ec0aa580 0x7f51ec0aaa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48538/0 (socket says 172.21.15.102:48538) 2024-03-23T03:52:25.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.175+0000 7f51f0e00640 1 -- 172.21.15.102:0/999405846 learned_addr learned my addr 172.21.15.102:0/999405846 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-23T03:52:25.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.175+0000 7f51ebe00640 1 --2- 172.21.15.102:0/999405846 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f51d0003f20 0x7f51ec1e8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:25.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.175+0000 7f51eb400640 1 --2- 172.21.15.102:0/999405846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51ec1493c0 0x7f51ec0a7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:25.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.175+0000 7f51f0e00640 1 -- 172.21.15.102:0/999405846 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f51d0003f20 msgr2=0x7f51ec1e8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:25.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.175+0000 7f51f0e00640 1 --2- 172.21.15.102:0/999405846 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f51d0003f20 0x7f51ec1e8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:25.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.175+0000 7f51f0e00640 1 -- 172.21.15.102:0/999405846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51ec1493c0 msgr2=0x7f51ec0a7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:25.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.175+0000 7f51f0e00640 1 --2- 172.21.15.102:0/999405846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51ec1493c0 0x7f51ec0a7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:25.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.175+0000 7f51f0e00640 1 -- 172.21.15.102:0/999405846 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51ec0a80d0 con 0x7f51ec0aa580 2024-03-23T03:52:25.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.176+0000 7f51f0e00640 1 --2- 172.21.15.102:0/999405846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51ec0aa580 0x7f51ec0aaa10 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f51ec0d0680 tx=0x7f51e4007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:25.178 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.176+0000 7f51e3e00640 1 -- 172.21.15.102:0/999405846 <== mon.0 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e400ea20 con 0x7f51ec0aa580 2024-03-23T03:52:25.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.176+0000 7f51e3e00640 1 -- 172.21.15.102:0/999405846 <== mon.0 v2:172.21.15.102:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f51e400eba0 con 0x7f51ec0aa580 2024-03-23T03:52:25.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.176+0000 7f51f2e00640 1 -- 172.21.15.102:0/999405846 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51ec0adb20 con 0x7f51ec0aa580 2024-03-23T03:52:25.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.176+0000 7f51e3e00640 1 -- 172.21.15.102:0/999405846 <== mon.0 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e4013420 con 0x7f51ec0aa580 2024-03-23T03:52:25.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.176+0000 7f51f2e00640 1 -- 172.21.15.102:0/999405846 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51ec1e7e20 con 0x7f51ec0aa580 2024-03-23T03:52:25.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.177+0000 7f51f2e00640 1 -- 172.21.15.102:0/999405846 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51ec066bb0 con 0x7f51ec0aa580 2024-03-23T03:52:25.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.178+0000 7f51e3e00640 1 -- 172.21.15.102:0/999405846 <== mon.0 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 5) v1 ==== 89137+0+0 (secure 0 0 0) 0x7f51e4013580 con 0x7f51ec0aa580 2024-03-23T03:52:25.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.179+0000 7f51e3e00640 1 --2- 172.21.15.102:0/999405846 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f51cc06ac00 0x7f51cc06d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T03:52:25.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.179+0000 7f51e3e00640 1 -- 172.21.15.102:0/999405846 <== mon.0 v2:172.21.15.102:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f51e4089400 con 0x7f51ec0aa580 2024-03-23T03:52:25.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.179+0000 7f51ebe00640 1 --2- 172.21.15.102:0/999405846 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f51cc06ac00 0x7f51cc06d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T03:52:25.180 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.180+0000 7f51ebe00640 1 --2- 172.21.15.102:0/999405846 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f51cc06ac00 0x7f51cc06d0c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f51ec0b2540 tx=0x7f51dc008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T03:52:25.184 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.184+0000 7f51e3e00640 1 -- 172.21.15.102:0/999405846 <== mon.0 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+181614 (secure 0 0 0) 0x7f51e4054670 con 0x7f51ec0aa580 2024-03-23T03:52:25.420 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.419+0000 7f51f2e00640 1 -- 172.21.15.102:0/999405846 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51ec0cd2b0 con 0x7f51ec0aa580 2024-03-23T03:52:25.421 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.420+0000 7f51e3e00640 1 -- 172.21.15.102:0/999405846 <== mon.0 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f51e4057c90 con 0x7f51ec0aa580 2024-03-23T03:52:25.421 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-23T03:52:25.421 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":58,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6835/2017025570","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6834","nonce":2017025570},{"type":"v1","addr":"172.21.15.173:6835","nonce":2017025570}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5725,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/173893445","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":173893445},{"type":"v1","addr":"172.21.15.102:6835","nonce":173893445}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":5740,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.173:6833/255507815","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6832","nonce":255507815},{"type":"v1","addr":"172.21.15.173:6833","nonce":255507815}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51},{"gid":7521,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/1507784449","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":1507784449},{"type":"v1","addr":"172.21.15.102:6837","nonce":1507784449}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58}],"filesystems":[]} 2024-03-23T03:52:25.421 INFO:teuthology.orchestra.run.smithi102.stderr:dumped fsmap epoch 58 2024-03-23T03:52:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.423+0000 7f51f2e00640 1 -- 172.21.15.102:0/999405846 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f51cc06ac00 msgr2=0x7f51cc06d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.423+0000 7f51f2e00640 1 --2- 172.21.15.102:0/999405846 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f51cc06ac00 0x7f51cc06d0c0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f51ec0b2540 tx=0x7f51dc008040 comp rx=0 tx=0).stop 2024-03-23T03:52:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.423+0000 7f51f2e00640 1 -- 172.21.15.102:0/999405846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51ec0aa580 msgr2=0x7f51ec0aaa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T03:52:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.423+0000 7f51f2e00640 1 --2- 172.21.15.102:0/999405846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51ec0aa580 0x7f51ec0aaa10 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f51ec0d0680 tx=0x7f51e4007bc0 comp rx=0 tx=0).stop 2024-03-23T03:52:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.423+0000 7f51f2e00640 1 -- 172.21.15.102:0/999405846 shutdown_connections 2024-03-23T03:52:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.423+0000 7f51f2e00640 1 --2- 172.21.15.102:0/999405846 >> [v2:172.21.15.102:6832/37097,v1:172.21.15.102:6833/37097] conn(0x7f51cc06ac00 0x7f51cc06d0c0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.423+0000 7f51f2e00640 1 --2- 172.21.15.102:0/999405846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f51ec0aa580 0x7f51ec0aaa10 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.423+0000 7f51f2e00640 1 --2- 172.21.15.102:0/999405846 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f51ec1493c0 0x7f51ec0a7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.423+0000 7f51f2e00640 1 --2- 172.21.15.102:0/999405846 >> [v2:172.21.15.173:3301/0,v1:172.21.15.173:6790/0] conn(0x7f51d0003f20 0x7f51ec1e8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T03:52:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.423+0000 7f51f2e00640 1 -- 172.21.15.102:0/999405846 >> 172.21.15.102:0/999405846 conn(0x7f51ec06d380 msgr2=0x7f51ec157330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T03:52:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.423+0000 7f51f2e00640 1 -- 172.21.15.102:0/999405846 shutdown_connections 2024-03-23T03:52:25.424 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-23T03:52:25.423+0000 7f51f2e00640 1 -- 172.21.15.102:0/999405846 wait complete. 2024-03-23T03:52:25.443 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-23T03:52:25.443 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-23T03:52:25.443 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T03:52:25.443 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:52:25.473 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T03:52:25.473 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T03:52:25.473 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:52:25.535 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T03:52:25.536 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T03:52:25.536 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:52:25.562 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T03:52:25.563 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T03:52:25.563 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:52:25.627 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T03:52:25.627 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-23T03:52:25.627 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-23T03:52:25.627 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:52:25.672 INFO:tasks.ceph.osd.0:Stopped 2024-03-23T03:52:25.672 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-23T03:52:25.672 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:52:25.753 INFO:tasks.ceph.osd.1:Stopped 2024-03-23T03:52:25.753 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-23T03:52:25.753 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:52:25.836 INFO:tasks.ceph.osd.2:Stopped 2024-03-23T03:52:25.836 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-23T03:52:25.837 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:52:25.914 INFO:tasks.ceph.osd.3:Stopped 2024-03-23T03:52:25.914 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-23T03:52:25.914 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:52:25.972 INFO:tasks.ceph.osd.4:Stopped 2024-03-23T03:52:25.972 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-23T03:52:25.972 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:52:26.052 INFO:tasks.ceph.osd.5:Stopped 2024-03-23T03:52:26.052 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-23T03:52:26.052 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:52:26.112 INFO:tasks.ceph.osd.6:Stopped 2024-03-23T03:52:26.113 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-23T03:52:26.113 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:52:26.171 INFO:tasks.ceph.osd.7:Stopped 2024-03-23T03:52:26.171 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-23T03:52:26.171 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-23T03:52:26.171 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:52:26.253 INFO:tasks.ceph.mgr.y:Stopped 2024-03-23T03:52:26.253 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-23T03:52:26.253 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:52:26.345 INFO:tasks.ceph.mgr.x:Stopped 2024-03-23T03:52:26.346 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-23T03:52:26.346 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-23T03:52:26.346 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:52:26.399 INFO:tasks.ceph.mon.a:Stopped 2024-03-23T03:52:26.399 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-23T03:52:26.399 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:52:26.431 INFO:tasks.ceph.mon.b:Stopped 2024-03-23T03:52:26.431 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-23T03:52:26.431 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T03:52:26.472 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-03-23T03:52:26.510 INFO:tasks.ceph.mon.c:Stopped 2024-03-23T03:52:26.511 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-23T03:52:26.511 DEBUG:teuthology.orchestra.run.smithi102:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-23T03:52:26.614 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:52:26.614 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-23T03:52:26.865 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:52:26.865 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-23T03:52:27.024 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:52:27.025 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-23T03:52:27.165 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:52:27.165 DEBUG:teuthology.orchestra.run.smithi102:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-23T03:52:27.309 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:52:27.309 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-23T03:52:27.580 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:52:27.580 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-23T03:52:27.727 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:52:27.727 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-23T03:52:27.902 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:52:27.902 DEBUG:teuthology.orchestra.run.smithi173:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-23T03:52:28.061 INFO:tasks.ceph:Archiving mon data... 2024-03-23T03:52:28.062 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617203/data/mon.a.tgz 2024-03-23T03:52:28.062 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-03-23T03:52:28.119 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.FeBdndgJ7G 2024-03-23T03:52:28.120 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.FeBdndgJ7G 2024-03-23T03:52:28.357 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.FeBdndgJ7G 2024-03-23T03:52:28.507 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.FeBdndgJ7G is 507KB 2024-03-23T03:52:28.617 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.FeBdndgJ7G 2024-03-23T03:52:28.671 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617203/data/mon.b.tgz 2024-03-23T03:52:28.671 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-03-23T03:52:28.729 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.3k5S7lZj6Q 2024-03-23T03:52:28.729 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.3k5S7lZj6Q 2024-03-23T03:52:28.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.3k5S7lZj6Q 2024-03-23T03:52:29.111 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.3k5S7lZj6Q is 645KB 2024-03-23T03:52:29.223 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.3k5S7lZj6Q 2024-03-23T03:52:29.278 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617203/data/mon.c.tgz 2024-03-23T03:52:29.278 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-03-23T03:52:29.329 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.wW7cUkyh90 2024-03-23T03:52:29.329 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.wW7cUkyh90 2024-03-23T03:52:29.566 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.wW7cUkyh90 2024-03-23T03:52:29.720 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.wW7cUkyh90 is 645KB 2024-03-23T03:52:29.824 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.wW7cUkyh90 2024-03-23T03:52:29.880 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-23T03:52:29.880 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-23T03:52:29.888 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-23T03:52:29.983 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-03-23T03:52:30.013 DEBUG:teuthology.orchestra.run.smithi102:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-03-23T03:52:30.020 DEBUG:teuthology.orchestra.run.smithi173:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-03-23T03:52:30.107 INFO:teuthology.orchestra.run.smithi102.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-03-23T03:52:30.125 INFO:teuthology.orchestra.run.smithi173.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-03-23T03:52:30.138 INFO:tasks.ceph:Archiving crash dumps... 2024-03-23T03:52:30.140 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617203/remote/smithi102/crash 2024-03-23T03:52:30.141 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-23T03:52:30.227 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617203/remote/smithi173/crash 2024-03-23T03:52:30.228 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-23T03:52:30.323 INFO:tasks.ceph:Shutting down logrotate 2024-03-23T03:52:30.324 DEBUG:teuthology.orchestra.run.smithi102:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:52:30.331 DEBUG:teuthology.orchestra.run.smithi173:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T03:52:30.459 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-23T03:52:30.549 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-23T03:52:30.647 INFO:tasks.ceph:Compressing logs... 2024-03-23T03:52:30.648 DEBUG:teuthology.orchestra.run.smithi102:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T03:52:30.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T03:52:31.472 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-03-23T03:52:36.269 INFO:tasks.ceph:Archiving logs... 2024-03-23T03:52:36.270 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617203/remote/smithi102/log 2024-03-23T03:52:36.271 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-23T03:52:37.373 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617203/remote/smithi173/log 2024-03-23T03:52:37.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-23T03:52:38.048 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-23T03:52:38.057 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-23T03:52:38.057 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-23T03:52:38.065 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-23T03:52:38.212 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-23T03:52:38.221 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-23T03:52:38.221 DEBUG:teuthology.orchestra.run.smithi102:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T03:52:38.230 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T03:52:38.275 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-03-23T03:52:38.284 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-03-23T03:52:38.285 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T03:52:38.285 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-03-23T03:52:38.285 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 5 +594us[ +595us] +/- 102ms 2024-03-23T03:52:38.285 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 5 -1044us[-1043us] +/- 73ms 2024-03-23T03:52:38.285 INFO:teuthology.orchestra.run.smithi102.stdout:^* hv03.front.sepia.ceph.com 3 6 377 5 +646us[ +646us] +/- 73ms 2024-03-23T03:52:38.285 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T03:52:38.294 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T03:52:38.294 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-03-23T03:52:38.294 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 4 +623us[ +623us] +/- 102ms 2024-03-23T03:52:38.294 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -1047us[-1037us] +/- 73ms 2024-03-23T03:52:38.294 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +668us[ +668us] +/- 73ms 2024-03-23T03:52:38.294 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T03:52:38.295 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-23T03:52:38.304 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-23T03:52:38.305 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-23T03:52:38.314 DEBUG:teuthology.orchestra.run.smithi102:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-23T03:52:38.554 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-23T03:52:38.773 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-23T03:52:38.874 INFO:teuthology.orchestra.run.smithi102.stdout:type=AVC msg=audit(1711165077.576:198): avc: denied { checkpoint_restore } for pid=1159 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1 2024-03-23T03:52:38.875 DEBUG:teuthology.task.selinux:ubuntu@smithi102.front.sepia.ceph.com has 1 denials 2024-03-23T03:52:38.876 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-23T03:52:38.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:52:38.976 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 166, in teardown self.get_new_denials() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 215, in get_new_denials raise SELinuxError(node=remote, teuthology.exceptions.SELinuxError: SELinux denials found on ubuntu@smithi102.front.sepia.ceph.com: ['type=AVC msg=audit(1711165077.576:198): avc: denied { checkpoint_restore } for pid=1159 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-03-23T03:52:38.977 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-23T03:52:38.986 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-23T03:52:38.994 INFO:teuthology.task.internal:Duration was 883.037287 seconds 2024-03-23T03:52:38.995 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-23T03:52:39.003 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-23T03:52:39.003 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-23T03:52:39.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-23T03:52:39.157 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T03:52:39.174 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T03:52:39.477 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-23T03:52:39.477 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi102.front.sepia.ceph.com 2024-03-23T03:52:39.478 DEBUG:teuthology.orchestra.run.smithi102:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-23T03:52:39.536 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-03-23T03:52:39.536 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-23T03:52:39.607 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-23T03:52:39.607 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T03:52:39.615 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T03:52:39.706 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-23T03:52:39.707 DEBUG:teuthology.orchestra.run.smithi102:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-23T03:52:39.711 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-23T03:52:39.904 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-23T03:52:39.914 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-23T03:52:39.914 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-23T03:52:40.008 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-23T03:52:40.105 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-23T03:52:40.116 DEBUG:teuthology.orchestra.run.smithi102:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-23T03:52:40.124 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-23T03:52:40.226 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = core 2024-03-23T03:52:40.227 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-03-23T03:52:40.283 DEBUG:teuthology.orchestra.run.smithi102:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-23T03:52:40.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:52:40.331 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-23T03:52:40.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T03:52:40.384 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-23T03:52:40.394 INFO:teuthology.task.internal:Transferring archived files... 2024-03-23T03:52:40.395 DEBUG:teuthology.misc:Transferring archived files from smithi102:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617203/remote/smithi102 2024-03-23T03:52:40.395 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-23T03:52:40.502 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617203/remote/smithi173 2024-03-23T03:52:40.502 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-23T03:52:40.609 INFO:teuthology.task.internal:Removing archive directory... 2024-03-23T03:52:40.610 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-23T03:52:40.617 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-23T03:52:40.672 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-23T03:52:40.691 INFO:teuthology.task.internal:Not uploading archives. 2024-03-23T03:52:40.691 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-23T03:52:40.701 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-23T03:52:40.701 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-23T03:52:40.707 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-23T03:52:40.755 INFO:teuthology.orchestra.run.smithi102.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 23 03:52 /home/ubuntu/cephtest 2024-03-23T03:52:40.764 INFO:teuthology.orchestra.run.smithi173.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 23 03:52 /home/ubuntu/cephtest 2024-03-23T03:52:40.767 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-23T03:52:40.779 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-23T03:52:40.790 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: SELinuxError: SELinux denials found on ubuntu@smithi102.front.sepia.ceph.com: ['type=AVC msg=audit(1711165077.576:198): avc: denied { checkpoint_restore } for pid=1159 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'] 2024-03-23T03:52:40.791 INFO:teuthology.nuke:Checking targets against current locks 2024-03-23T03:52:40.816 DEBUG:teuthology.nuke:shortname: smithi102 2024-03-23T03:52:40.817 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T03:52:40.838 DEBUG:teuthology.nuke:shortname: smithi173 2024-03-23T03:52:40.838 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T03:52:40.844 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617203', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 03:28:43.692644', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMTizH0wUqZ4U/b89MXw+0NgcpY4R8KU/gvKM8U+SnddorpI5l/0MjELps6rL3KkXG2xlUc9NiXI5iVz1sRTpFs='} 2024-03-23T03:52:40.861 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-03-22_13:15:24-fs-wip-yuri11-testing-2024-03-21-0851-reef-distro-default-smithi/7617203', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 03:28:43.690835', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDYfGpry3ANlMxAACapxS822P0GSJelsL/67tVqmYt77USJcr2GPvIQKVoKwjDDWC8XLqyAmDUKqDa+hhPpnnWg='} 2024-03-23T03:52:40.891 INFO:teuthology.orchestra.console.smithi102:Power off 2024-03-23T03:52:40.891 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-23T03:52:40.916 DEBUG:teuthology.orchestra.console.smithi102:power off output: Chassis Power Control: Down/Off 2024-03-23T03:52:40.916 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T03:52:40.930 INFO:teuthology.orchestra.console.smithi173:Power off 2024-03-23T03:52:40.931 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-23T03:52:40.940 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-03-23T03:52:40.952 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off 2024-03-23T03:52:40.952 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T03:52:40.974 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-03-23T03:52:44.941 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T03:52:44.974 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T03:52:45.052 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-03-23T03:52:45.086 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-03-23T03:52:49.056 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T03:52:49.086 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T03:52:49.166 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is off 2024-03-23T03:52:49.195 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off 2024-03-23T03:52:49.268 INFO:teuthology.orchestra.console.smithi102:Power off completed 2024-03-23T03:52:49.296 INFO:teuthology.orchestra.console.smithi173:Power off completed 2024-03-23T03:52:49.402 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 client.1-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} subvol_versions/create_subvol_version_v2 tasks/quota} duration: 883.0372867584229 failure_reason: 'SELinux denials found on ubuntu@smithi102.front.sepia.ceph.com: [''type=AVC msg=audit(1711165077.576:198): avc: denied { checkpoint_restore } for pid=1159 comm="agetty" capability=40 scontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tcontext=system_u:system_r:getty_t:s0-s0:c0.c1023 tclass=capability2 permissive=1'']' flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2024-03-23T03:52:49.402 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-23T03:52:49.496 INFO:teuthology.run:FAIL